I shut down every program on my pc and now it looks like it was actually my screen recording software (Dxtory) that made the game crash for some reason.
I did multiple tests with and without the recording software and indeed it looks like it always crashes when it's on, but doesn't when it's off.
Sorry I wasted your time with this problem that seems to be non-game related after all.

I haven't tried private or experimental servers. And my connection is fine when it happens, also I can join back to the server. It's happened multiple times when I play with friends who are in the same server and they have no problems.