If you have the binoculars out and change stance you will automatically switch to the rifle before executing the stance-change and then switch back to the binoculars when you have changed the stance.
On top of this, you can't run with binoculars in your hand.
Binoculars should behave more like a weapon so that you take them out with B but enter the "scope-mode" with mouse button 2 (default key for aiming), just like with rifles.
Hopefully this issue will addressed before the retail release.
{F17210} {F17211}