Page MenuHomeFeedback Tracker

Jenkins out of memory bug
New, NormalPublic

Description

Please provide the following when creating the Ticket:

Game Version number: 1.4.0.48
Modded?: Yes
If modded, please list the mods:

		"mods": [
			{
                "modId": "620D47D4EE359AE9",
                "name": "LoadoutAllowances"
            },
			{
                "modId": "64F71D076BF8AF49",
                "name": "LoadoutAllowancesExtended"
            },
			{
                "modId": "5EE51BF388E35B44",
                "name": "MedicalNotifications"
            },
			{
				"modId": "5E92F5A4A1B75A75",
				"name": "PVEPlayerMarkers"
			},
			{
				"modId": "6088A3044B7ECBFD",
				"name": "keepGunOnUncon"
			},
			{
				"modId": "5C8AD2767D87626B",
				"name": "DisableFF"
			},
			{
				"modId": "5C9758250C8C56F1",
				"name": "BonActionAnimations"
			},
			{
				"modId": "5AAAC70D754245DD",
				"name": "AdminTools"
			},
			{
				"modId": "612F512CD4CB21D5",
				"name": "EarPlugs"
			},
			{
				"modId": "64F73E541C398A57",
				"name": "RISLaserAttachmentsEdits"
			},
			{
				"modId": "64F752CC69E4A5CD",
				"name": "M16A1Edits"
			},
			{
				"modId": "615028AE56FCFB67",
				"name": "M82Edits"
			},
			{
                "modId": "62753004B26ED2CB",
                "name": "RTUS"
            },
			{
                "modId": "632C289569A58A82",
                "name": "ORT4ChillFam"
            },
			{
				"modId": "60B134EF0216CF4D",
				"name": "Early2000sUSMilitary"
			},
			{
				"modId": "631C5C826EC11C4B",
				"name": "Early2000sFix"
			},
			{
				"modId": "64D4E4ACDE3EBEDB",
				"name": "ChillFamGSEdits"
			},
			{
				"modId": "64F757D06B59D80B",
				"name": "frogSkinCamoEdits"
			},
			{
				"modId": "64F755EA66FFBC4F",
				"name": "tigerStripeCamoEdits"
			},
			{
				"modId": "60C4CE4888FF4621",
				"name": "ACE_Core"
			},
			{
				"modId": "60EAEA0389DB3CC2",
				"name": "ACE_Trenches"
			},
			{
				"modId": "60C53A9372ED3964",
				"name": "ACE_Compass"
			},
			{
				"modId": "6214CEA5DA92BFF3",
				"name": "ACE_MedicalPrototypes"
			},
			{
				"modId": "606C369BAC3F6CC3",
				"name": "ACE_Finger"
			},
			{
				"modId": "5DBD560C5148E1DA",
				"name": "ACE_Carrying"
			},
			{
				"modId": "611CB1D409001EB0",
				"name": "ACE_MagRepack"
			},
			{
				"modId": "5EB744C5F42E0800",
				"name": "ACE_Chopping"
			},
			{
				"modId": "61226BB18D360BDD",
				"name": "ACE_TacticalLadderPrototype"
			},
			{
				"modId": "62F802951CC8A37E",
				"name": "ACE_Tactical_Periscope"
			},
			{
				"modId": "64475DC102F2BDA4",
				"name": "ACE_Radio"
			},
			{
                "modId": "5CCC20B09E94E467",
                "name": "GatorsCarPack"
            },
			{
                "modId": "63191C8F6FDAE613",
                "name": "VTFRallyChamp4Conflict"
            },
			{
				"modId": "629B2BA37EFFD577",
				"name": "WCS_Armaments"
			},
			{
				"modId": "62CCD69DD17E4F2F",
				"name": "AKI_Core"
			},
			{
				"modId": "6273146ADFE8241D",
				"name": "AH6M"
			},
			{
				"modId": "64476E44FA28A9F7",
				"name": "AH6MForConflict"
			},
			{
				"modId": "5E389BB9F58B79A6",
				"name": "SpaceCore"
			},
			{
				"modId": "5E5C154FEE1094BB",
				"name": "M113"
			},
			{
				"modId": "5D1880C4AD410C14",
				"name": "M1 Abrams"
			},
			{
				"modId": "63120AE07E6C0966",
				"name": "M2A2Bradley"
			},
			{
				"modId": "5E524E4FEECCA92B",
				"name": "BMP-1_IFV"
			},
			{
                "modId": "5E0AB16BEB16D6A4",
                "name": "T72"
            },
			{
                "modId": "5B383D4CB27E0D54",
                "name": "BMP3"
            },
			{
                "modId": "62D15D0025AE021B",
                "name": "ZSU23Shilka"
            },
			{
				"modId": "646D53066E86C9F1",
				"name": "RHIB"
			},
			{
                "modId": "631452D162C86B0B",
                "name": "CivVehicles4Conflict"
            },
			{
                "modId": "64F841CAD0832625",
                "name": "DeployableBike"
            },
			{
				"modId": "621D3771875C1D3D",
				"name": "carRadio4All"
			},
			{
                "modId": "61BA4EB5C886D396",
                "name": "WCS_VehicleLock"
            },
			{
				"modId": "5B0D1E4380971EBD",
				"name": "CSISquadInterface"
			},
			{
                "modId": "613B4DD4A91CA144",
                "name": "LMSuppression"
            },
			{
                "modId": "64F6F01A683C6AF8",
                "name": "CivExtract"
            },
			{
				"modId": "620C243361C6167F",
				"name": "CivClothingInUSArsenal"
			},
			{
				"modId": "6319100F430C4407",
				"name": "ColoredFlashlights"
			},
			{
				"modId": "5DD2FE4D7F7A3A6F",
				"name": "FireExtinguisher"
			},
			{
				"modId": "62BE25F97D19ED65",
				"name": "DynamicMines"
			},
			{
                "modId": "64CF13E064FE8D9D",
                "name": "ChillFamInvEdits"
            },
			{
                "modId": "61BD6595183FCEBD",
                "name": "BigChungusBolt"
            },
			{
                "modId": "65136B5364BB5831",
                "name": "ChillFamBCBGEdits"
            },
			{
                "modId": "61B514B96692C049",
                "name": "ConflictPVERemixedVanilla2.0"
            },
			{
                "modId": "628729E87E79DA7F",
                "name": "LinearConflictPVE"
            },
			{
				"modId": "613629A17AFFEBFE",
				"name": "NoAILivingQuarters"
			},
			{
                "modId": "62A17A5C6163FE2E",
                "name": "FactionFix"
            },
			{
                "modId": "65CC19B695D6831A",
                "name": "CPL",
				"version": "0.1.1"
            },
			{
                "modId": "65CE571C7CB8ED1C",
                "name": "CPLResetter"
            },
			{
				"modId": "62E46BB768990AA2",
				"name": "SlowXPGain"
			},
			{
				"modId": "653A2B685DB88308",
				"name": "PersistentRankResetter"
			},
			{
                "modId": "65B4F2295017BBE7",
                "name": "RemoveConflictCallsigns"
            },
			{
				"modId": "62113AA293414DB9",
				"name": "BandageFix"
			},
			{
                "modId": "65C2215B66767AB2",
                "name": "CharacterSelector4Conflict"
            }
		]

Issue Description:
Three times in the past month I have restarted one of my server instances that uses -loadSessionSave, and once everything is loaded and it enters GAME mode, the server FPS slowly drops from 60 (max) to less than 1. Then I get the following error in console.log:

13:45:14.741 WORLD : UpdateEntities
13:45:14.741 WORLD : FixedFrame
13:45:14.741 ENGINE (F): Crashed
13:45:14.765 ENGINE (E): Application crashed! Generated memory dump: C:\Games\SteamCMD\steamapps\common\Arma Reforger Server3\ServerProfile3/logs/logs_2025-07-14_13-44-21\ArmaReforgerServer3_2025-07-14_13-45-14.mdmp
13:45:14.947 RPL : Reducing puzzle difficulty level to 5.
13:45:14:947 CRASH REPORT : Crash reporter started!
13:45:14:947 CRASH REPORT : Started without dialog window! Automatically send Report.

Once this happens, I get the following error in my console log of another server instance on the same machine:

13:45:15.172 ENGINE (F): Crashed
13:45:15.215 ENGINE (E): Application crashed! Generated memory dump: C:\Games\SteamCMD\steamapps\common\Arma Reforger Server\ServerProfile/logs/logs_2025-07-13_15-29-36\ArmaReforgerServer_2025-07-14_13-45-15.mdmp
13:45:15.386 ENGINE (E): Cannot run crashreport process...

Once all this happens, I start getting windows pop up errors that say:
"ArmaReforgerServer3.exe - Application Error
The instruction at 0x00007FF8CFC51AF0 referenced memory at 0x0000000000000020. The memory could not be written. Click on OK to terminate the program"

then on another instance:
"ArmaReforgerServer2.exe - System Error
Exception Processing Message 0xc0000005 - Unexpected parameters"

and on the 3rd instance:
"ArmaReforgerServer.exe - System Error
Exception Processing Message 0xc0000005 - Unexpected parameters"

So when I can log into my dedicated server to click ok on all these errors, all instances are shut down and attempt to restart. This whole process will keep repeating until I delete the save file for the instance that had the dropping server fps (in this case, server 3).

If I dont delete the save file and try to restart, I get this in server 3's console log:

14:12:24.746 ENGINE (F): Crashed
14:12:24.763 ENGINE (E): Application crashed! Generated memory dump: C:\Games\SteamCMD\steamapps\common\Arma Reforger Server3\ServerProfile3/logs/logs_2025-07-14_14-11-36\ArmaReforgerServer3_2025-07-14_14-12-24.mdmp
14:12:24.764 ENGINE (E): Cannot run crashreport process...

Then a few seconds later on the first instance I get:

14:12:51.308 MEMORY (E): C:\jenkins\workspace\continuous_branches_stable_1.4.0\ARGamecode\Enfusion\Enfusion\src\system\enf_memory.cpp(545) Out of memory when requested 1064959 Type: General
14:12:51.308 MEMORY (E): Free sysMem 12858288 Kbytes
14:12:51.308 MEMORY (E): TotalIndices 10769 Kbytes
14:12:51.309 MEMORY (E): TotalVertices 1198 Kbytes
14:12:51.309 MEMORY (E): TotalCBBytes 626 Kbytes
14:12:51.309 MEMORY (E): ObjectIndices 0 Kbytes
14:12:51.309 MEMORY (E): RenderTargets 0 Kbytes
14:12:51.309 MEMORY (E): UAV(pure) 0 Kbytes
14:12:51.310 MEMORY (E): ObjectVertices 0 Kbytes
14:12:51.310 MEMORY (E): SysTextures 0 Kbytes
14:12:51.310 MEMORY (E): VRAMTextures 0 Kbytes
14:12:51.310 MEMORY (E): EntityData 0 Kbytes
14:12:51.310 MEMORY (E): Particles 45 Kbytes

this spams the logs about every 0.010 seconds until it does:

14:13:02.452 ENGINE (F): Crashed
14:13:02.475 ENGINE (E): Application crashed! Generated memory dump: C:\Games\SteamCMD\steamapps\common\Arma Reforger Server\ServerProfile/logs/logs_2025-07-14_14-11-26\ArmaReforgerServer_2025-07-14_14-13-02.mdmp
14:13:02.478 ENGINE (E): Cannot run crashreport process...
14:18:01.341 ENGINE (F): Application hangs (force crash) 304 s

During this time, server 3 never crashes and does let players try to join but all will timeout. Example:

14:12:56.807 RPL : ServerImpl event: authenticating (identity=0x00000000, address=97.90.171.83:64964)
14:13:26.812 RPL : ServerImpl event: authentication timeout (identity=0x00000000)
14:13:26.812 NETWORK (W): RplAuthBackend::OnFailure called (identity=0x00000000, group=1, reason=3)

Details

Severity
Crash
Resolution
Open
Reproducibility
Unable To Reproduce
Operating System
Windows 10 x64
Operating System Version
Windows Server 2019 Standard Version: 1809 OS Build: 17763.7558
Category
General
Steps To Reproduce

Unable to reproduce

Additional Information

This seems like a very serious error that should have a high priority to get fixed! I dont understand why one instance would crash all other instances on the same machine. It must be a serious bug in the software to do that!
The fix Ive found for this is to delete the save files on server 3, then all other servers will be able to start and run normally. That being said, it's my best guess that it has something to do with the save file. Maybe the server is shutting down in the middle of a save, therefore corrupting the file? That still doesnt explain why it would affect all other instances on the same machine though. This is highly frustrating! Today my servers were down for about 5 hours while I was at work.

Other than this error, server stability has improved GREATLY recently and I have had my server 3 stay up for days without a restart needed, so great job on that!

Event Timeline

Gramps created this task.Tue, Jul 15, 4:24 AM