It's like when looking through binoculars you can't look through the side of the scope, and change direction is done by turning you whole head, which normally can on be turned about 90 degrees to each side, same goes for up and down. This would also make NV-goggles less OP when worn by AI and make stealth more simple, just saying.