Page MenuHomeFeedback Tracker

CLE is causing massive fps dips at high entity counts
Assigned, NormalPublic

Description

Hello, for the past few months we've been trying to debug what is the cause of our fps dips, as they occur exactly every 20-30 seconds, and results in nearly 30-40 fps being consumed by it.

We've been able to identify that it is CLE as the CELoop takes 25 seconds to complete, as well as disabling it removes the dips.
Here is a visualization of the dips during peak hours.

This specific server has a very high entity count (AU1)

Most of these entities are in cargo of storage, not parentless.
The exception are base objects, which make up a small % of the entity count


We spawn these with flag ECE_NOLIFETIME, and delete them with our own method (proven to have no effect on this)
In my mind this would make CLE do less work when looping through them.

I've tried looking into a solution with the tools provided, but class CEApi is extremely limited, and I honestly just don't really know the extent at how the CLE works, if it's still looping through items in cargo, or if the base objects could be ignored by CLE by some way.

Any info would be greatly appreciated, thanks.

Also as Dan pointed out by matching our CE Logs to Grafana timestamps, you can see our logs say
16:22:16.504 players: 59, loot: 191108, infected: 190, animals: 48
191108 is an accurate ItemBase count for that server, so I guess the problem just stands that it's looping through all items, many of which are items that do not require any updating as theyre sitting in cargo. I should mention while we're on that topic also, that we have Food Decay and Temperature updating of items disabled.

Details

Severity
Major
Resolution
Open
Reproducibility
Always
Operating System
Windows 7
Category
General
Additional Information


Low pop server, high entity count (server fps is limited to 100)


Another low pop server, lower entity count (still high, it is EU1 in graph above)

Just for clarity, our Entity Counter is this simple code:

int TotalLoadCount;

modded class ItemBase
{			
	void ItemBase()
	{
		TotalLoadCount++;
	}

	void ~ItemBase()
	{
		TotalLoadCount--;
	}
}

Here also is a playercount and server fps overlayed on one another

Here is a similar graph, except with 70k entity count, instead of 260k.


The dips are still present, no worries, but the extremity of them isn't comparable, only fluctuating 5-10 fps each ce loop.

Event Timeline

designful edited Additional Information. (Show Details)Jun 10 2024, 12:07 PM
designful edited Additional Information. (Show Details)
AvatarDan added a subscriber: AvatarDan.EditedJun 10 2024, 1:49 PM

Zooming in on one of these drips and comparing to CE logging, can see the following
Things to note, player pop is around 60~ players here. So the server is about half full, we have AI turned down as well, 190 infected, 48 animals.

The two red lines are the timestamps as to when the CE loop is logged/ending.

16:21:47.688 *** CE Loop took 26.9 (sec)...
16:21:47.688 players: 57, loot: 190805, infected: 190, animals: 48

16:22:16.504 *** CE Loop took 26.9 (sec)...
16:22:16.504 players: 59, loot: 191108, infected: 190, animals: 48

16:22:05 Server FPS = 47 frames (The lowest point in the picture)
This is about halfway through the CE loop, right before logs from [CE][LootRespawner] (PRIDummy) :: [RESPAWN CANDIDATE] start.

My theory is at the start of the CE loop, it scans all entities on the server, which causes server FPS to plumet.

This example isnt very problematic, because 47~ server fps is still fine, but there are times where the server fps spikes from a 60~ down to a 2-6. Which causes severe lag during those 5-10 seconds that fps is low


Can see the lowpoint of this graph showing 6 serverFPS

Geez added a subscriber: Geez.Jun 11 2024, 11:07 AM
This comment was removed by Geez.
This comment was removed by designful.
designful edited Additional Information. (Show Details)Jun 11 2024, 11:17 AM
designful edited Additional Information. (Show Details)Jun 11 2024, 11:24 AM
Geez added a comment.Jun 11 2024, 11:29 AM

Thank you. There was a miscommunication within the team and the storage from your server will not be needed. Apologies for any inconvenience.
Regards,
Geez

Geez changed the task status from New to Assigned.Jun 11 2024, 11:29 AM
designful edited Additional Information. (Show Details)Jun 11 2024, 11:32 AM
designful updated the task description. (Show Details)Jun 11 2024, 11:49 AM
ham added a subscriber: ham.Jun 17 2024, 10:25 PM