I'm happy to say this didn't block me and force me to suicide since jumping off things is so good now, very nice. The ladder to the watchtower shown on the picture (first one NW of the southern entrance) is allowing you to enter from the bottom but not to leave it there. Character will just repeat climbing anim at the 'bottom' when trying to descend.
I cannot see this working for anyone differently, they will all have to jump until the ladder is nudged into place.
For the smooth ladder cause!
There's actually four watchtowers around the perimeter there that are placed too low into the ground so that you cannot climb down enough to exit the ladder.
There's also one that is placed too high so that you cannot enter the ladder from standing in front of it, though you can get on the concrete slab from the side and then climb the ladder.