Try changing to a hardcore server and try a normal one afterwards. this fixed it, but I don't know how, but your explanation exactly matches what I was dealing with.

Try joining a hardcore server and load up a normal afterwards, see if it solves your issue. I did this after someone told me it worked! I just don't know why anyone figure out what is causing this?

Changing to a hardcore server then back to a normal one fixed this issue for me.
I think it has to do with something in your inventory.