As always, I'll leave a short video to make the explanation easier below.
If you get into the inventory, and move a backpack to your hands slot, there is a weird and noticeable framedrop.
In the video below you can see how stable my frames are when looking around. But as soon as I open the inventory, and move my backpack into my hands, FPS go down from 60-65 to 37-39.
I also experenced this with an empty Assault Backpack, so looks like it is independent of what is inside of it.
Video with magnified Steam FPS counter: