Page MenuHomeFeedback Tracker

Potential memory leak
Closed, ResolvedPublic

Description

Figure 1 shows the memory usage and playercount over a period of 24 hours.
As you can see sometimes the memory usage suddenly increases by several hundred megabytes which is probably caused by a memory leak.
This problem exists since the beginning (v0.9.5.x).


The memory usage was tracked by an external program and represents the value that is shown in the task manager or ressource monitor (figure 2).
Using logStats as start parameter or script command System.MemoryAllocationKB() will result in significant lower values somehow.
This is what System.MemoryAllocationKB() returned when figure 2 was made:
-> MemoryAllocationKB: 1.915 MB, MemoryAllocationCount: 1.768.820

Which mods were used:

  • Team ELAN - Main
  • Team ELAN - Assets
  • Vergys Custom Clothing (no impact if mod is deactivated)
  • Beetle 1967 (no impact if mod is deactivated)

Which OS is used:

  • Windows Server 2016
  • Windows Server 2019

Meanwhile i am pretty sure that its not a leak in our mod itself. We are checking every player/vehicle and the gamemod including all components via GetSizeOf().
There is no increase in memory usage so i am guessing its coming out of the engine.

If this problem did not exists on official servers these are the biggest differences to the vanila mods:

  • Heavy use of the rest api
  • Heavy use of JsonSaveContext/JsonLoadContext
  • Many player inventory operations (Deleting, adding items)

If there a logs or any data needed, let me know.

Details

Severity
Major
Resolution
Open
Reproducibility
Always
Operating System
Windows 7
Category
General

Event Timeline

RamBob created this task.Dec 12 2022, 7:22 AM
Geez changed the task status from New to Assigned.Dec 12 2022, 10:06 AM

Update


Since the last updates the servers are way more stable, so it was possible to isolate the problem a bit and generate a diagram without other crashes.
Therefore figure 3 shows players and memory usage over a periode of 12 hours with high load.

What figure 3 shows:

  • The memory usage increases sharply in one frame.
  • The first noticeable spike is every time around 80 minutes after restart.
  • From there on the spike increases everytime. The increase in memory is roughly twice that of the last spike.
  • The pattern of the memory usage is almost the same in every period. (There are other users on the Arma Discord that made the same experience without using our modfiles. Screenshot by bacon: https://i.imgur.com/3LaAh03.jpeg)

What we figuered out so far:

  • On our other servers with a playerlimit of 32,42 players the spikes are significant lower and the servers can run for 6 or more hours.
  • When we messed up and placed some NPCs with an inhereted prefab from Character_FIA_base.et the spikes were significant higher.
  • When we messed more up and made the RPL Component of the NPCs static and disabled streaming the spikes were way more higher.
  • When we replaced the NPCs the spikes were significant lower.

Conclusion
I could imagine that the issue has something to do with one component of the prefab, respectively one of the prefabs Character_FIA_base.et is inhereted from. The problem seems to occurs only if the entity is streamed to other players.
During testing when i placed 50 streamable "Character_FIA_base.et" entities on an loneley island far away from other players streaming range nothing happened.
Doing the same, but placing them near Montignac results in significant higher memory spikes.
I also created prefabs where some components were deactivated to have a more detailed look into it and check the guess, but had no time yet to check it on the server.

If you need more data or infos that would help you, let me know.

It seems like today's update 0.9.7.91 has fixed the issue.

Geez added a subscriber: Geez.May 23 2023, 2:03 PM

Hello RamBob.
Do you have any updates for the current version?

Heloo Geez,

in terms of memory usage the current version 0.9.8.64 is doing pretty good even on higher load.
The problem is fixed entirely since the mentioned version 0.9.7.91.

Geez added a comment.May 24 2023, 11:01 AM

Thank you for the update!

Geez closed this task as Resolved.May 24 2023, 11:24 AM
Geez claimed this task.