Page MenuHomeFeedback Tracker

Reproducible High Pop Server Lag Report
Assigned, NormalPublic

Description

TLDR: Everytime SyncEvents::OnSyncEvent() is called with case ESyncEvent.PlayerList (Player List Sync) on a high pop server, it makes every player rubber band for other clients.

You can see the effect of this by timing the player lag, and the debug ui at the bottom left (specifically the RPC) in this youtube video. The first parameter is just a counter to count the amount of times that rpc has been sent to my client. The second parameter is the event id to prove that it's specifically the player list sync. https://www.youtube.com/watch?v=GGF1_2nUZis

This was literally like 5 minutes of gameplay on a 120 pop server edited down. Some of the lag spikes hit harder than others, but it was pretty much consistent every single time that number went up, you could see the effect on moving players.

This is the code for the actual programmers reading this to understand my debug tool and maybe be able to pinpoint the problem:

//The only change here is the invoker
modded class SyncEvents
{
	static ref ScriptInvoker OnRPCDebug = new ScriptInvoker();
	
	override static void RegisterEvents() //no change, needed for override Event_OnRPC to work 
	{
		DayZGame dz_game = DayZGame.Cast( GetGame() );
		
		dz_game.Event_OnRPC.Insert(Event_OnRPC);
		Print("SyncEvents -> RegisterEvents");
	}

	override static void Event_OnRPC(PlayerIdentity sender, Object target, int rpc_type, ParamsReadContext ctx)
	{
		if (rpc_type == ERPCs.RPC_SYNC_EVENT && GetGame() && GetGame().IsMultiplayer() && GetGame().IsClient())
		{			
			Param2<ESyncEvent, ref SyncData> event_data = new Param2<ESyncEvent, ref SyncData>(-1, null);
			
			if (ctx.Read(event_data))
			{
				OnSyncEvent(event_data.param1, event_data.param2, target);
				OnRPCDebug.Invoke(rpc_type, event_data.param1); //handling the ESyncEvent primarily, rpc_type isn't used in this example
			}
		}
	}
}
//debug hud stuff
modded class MissionGameplay
{
	protected ref DebugHud m_DebugHud;

	override void OnMissionStart()
	{
		super.OnMissionStart();

		m_DebugHud = new DebugHud();
		SyncEvents.OnRPCDebug.Insert(m_DebugHud.OnDebugRPC); //updating the hud
	}
}

class DebugHud: ScriptView
{
	int RPCCounter = 0;
	TextWidget line0, line1, line2, line3, line4;

	void OnDebugRPC(int rpc, int data)
	{
		RPCCounter++;
		line2.SetText(string.Format("RPC: %1 (%2)", RPCCounter, data)); //RPC counter + Event ID on HUD
	}

	override string GetLayoutFile()
	{
		return "RearmedServer\\GUI\\layouts\\debug.layout";
	}
}

This is a major issue that has been plaguing the game for SO very long, and I'm so grateful I could finally have this visualization to assist in getting it fixed. I apologize though, as I do not know anything more about what the issue may be, but I have the tools here available in order to reliably reproduce the issue, and I am more than welcome to work together with a developer on testing and deploying a fix for this if needed.

My personal guess is that it's related to the player list being large, so the sync is more heavy that it causes these hiccups. But it could be anything related to some other code being executed as the same time as the sync.

Details

Severity
Major
Resolution
Open
Reproducibility
Always
Operating System
Windows 10
Category
General
Steps To Reproduce

Use code provided on a high pop server (I cannot confirm the effect of the lag on low pop servers, this was only tested on 120 pop, and I assume it's because of the larger player list)

Event Timeline

designful updated the task description. (Show Details)
designful updated the task description. (Show Details)Nov 22 2023, 8:34 AM
designful updated the task description. (Show Details)
Lad added a subscriber: Lad.Nov 22 2023, 8:44 AM
lava76 added a subscriber: lava76.Nov 22 2023, 10:53 AM
lava76 added a comment.EditedNov 22 2023, 10:56 AM

The reason for this seems to be that the player list is sent to all clients whenever a player respawns - at this point that is wasteful though, because the playerlist hasn't changed. The underlying issue is that ClientNewEventTypeID fires for new players (where sending the playerlist makes sense since it has actually changed) as well as respawning players (here, the playerlist hasn't changed).

Adding on to what @lava76 mentioned.

* Order of Events firing:
* When player respawns                            : ClientPrepareEventTypeID ► ClientNewEventTypeID ► ClientNewReadyEventTypeID
* When player joins (no saved character)          : ClientPrepareEventTypeID ► ClientNewEventTypeID ► ClientNewReadyEventTypeID
* When player joins                               : ClientPrepareEventTypeID ► ClientReadyEventTypeID
* When player logs out                            : ClientDisconnectedEventTypeID
* When player cancels logout                      : LogoutCancelEventTypeID

Reducing the player list sync at some events when not needed would be a good starting point.

Geez changed the task status from New to Assigned.Nov 22 2023, 12:51 PM

Adding on to what @lava76 mentioned.

* Order of Events firing:
* When player respawns                            : ClientPrepareEventTypeID ► ClientNewEventTypeID ► ClientNewReadyEventTypeID

There is one event before that, ClientRespawnEventTypeID, so it is possible to distinguish respawn from new player joining.

Performance is looking a lot better after this update. Thanks for the hard work team!