Please note this issue can also cause other players to not be able to interact with their inventory, as the servers seem to choke on the tracking of that inventory. see issue 0005854

A question: Do you or any in your group stack backpacks?
my friend stacked 9 backpacks, which resulted in similar symptoms.
see issue 0005854

Yes, my friend and I were having the same exact issue, and when he unstacked his backpacks, it resolved all the issues we were having. see issue # 0005854.

I could play just fine, then when my friend would join the server, I and he would be unable to interact with the environment or our inventory at all.

Seems We've fixed it!
Turns out the friend had stacked about 9 backpacks together, which caused the servers to choke and not correctly track any info. When my friend would join a server, I would be unable to interact with my inventory, hotbar, and I couldn't even open doors.

I can vouch that this issue caused massive server problems. see issue 0005854 and 0006605