I've also been experiencing this issue sometimes it allows the other person to play moderator but about 30 minutes later the server just becomes unplayable. And other times when the player goes moderator everyone get's 5fps. I've had this happen when i was a playing and i've got a GTX titan it managed to cripple that down to 5fps.