Stand near a ladder and click the middle mouse button once the symbol for climbing down is shown.
Most of the times this has happen, I have been moving about a few seconds before clicking "climb down" so I am not sure if it is caused by moving before climbing down or if it is caused by being about a meter or so away from the actual ladder.
Your character should proceed with standing up, running off the edge and falling down. If the height is survivable. You will teleport back onto the ladder and can then either chose to climb down or back up. If it's a non-survivable height, you'll just die.