it seems as if the "access" spots are somewhat misplaced so that while you are still climbing up to the lower level, the symbol to "ascend" already appears for the next level.
the moment you arrive on the lower level you have no possible way to get the "ascend" symbol again, leaving you only with the option to descend again.
https://imgur.com/a/KitnY
images show the relevant chimney and the issue.