When using items with holding a 2nd gun in your had the character will always drop it in the ground. It can get lost (dropped into geometry, etc) especially if you're backed into a corner or being too close to a wall (for climbing a ladder), but it will always be dropped.

The bigger issue here is that when you have 2 rifles and you log out/in it's possible to sprint/turbo because there's nothing in your hands.