If I join a server, my desync will go away with in a few seconds. However, whenever my friend joins into my server, my desync and his desync will shoot up. Once one of us leaves, it will go down. We do not play from the same house, or same connection, we are in the same town however.