Objects do not gain speed as they fall over. If you drive a car into a wall the wall will gradually tip over as opposed to fall over, often clipping with the car.
Also shooting a wall causes a large piece of the wall to fall over very slowly after taking no physical damage, whereas you would expect the wall to take damage and crumble.