Page MenuHomePhabricator

Server running out of memory - Stack Overflow
Assigned, WishlistPublic

Description

After updating the server to 1.54.x with SteamCMD, we checked the steam_appid and validated the ArmA 3 files. We are running a modded ArmA 3 server (with custom map which is based on Jackson_County / AppleGate)

We started our server with our batch file including following start parameters:

-port=2302 "-Bepath=C:\ArmA3\A3Master\battleye" "-config=C:\ArmA3\A3Master\config.cfg" "-cfg=C:\ArmA3\A3Master\basic.cfg" "-profiles=C:\ArmA3\A3Master\TADST\LSRL" -name=LSRL "-servermod=@ASM;@life_server;@CSE_Server;@extDB;" "-mod=@LRL;@task_force_radio;@CSE;" -world=empty -nosplash -noSound -noPause -exThreads=7 -autoinit -malloc=system -loadMissionToMemory

The server start-period started without error - as before in 1.52. After *15-20* minutes starting the server scripts didn't get executed as fast as usual. Players FPS dropped down from *45* to *5* and lower until the client of the player has crashed. About *10* minutes later the server has ran out of memory (error-box on the windows server).

I tested it with/-out -malloc and -loadMissionToMemory paramters.
Didn't work either - after *30* minutes of running the server it crashed with 'Out-Of-Memory' again.

RPT-Files got deleted automatically as we downgraded to 1.52.x.

  • I'm going to upload an RPT-Files in *2-5* days **
  • I added a partial part of RPT log down below in 'Additional Information' **

If you need our mission-file and/or mod-files leave a message, I'm going to add you on skype then. {F27483} {F27484} {F27485} {F27486}

Details

Legacy ID
3681695413
Severity
None
Resolution
Open
Reproducibility
Always
Category
Server
Steps To Reproduce
  1. Update server to 1.54.x with SteamCMD
  2. Check steam_appid.txt and validate ArmA 3 files
  3. Start server with start parameters as mentioned above and our mod- and mission-files.
  4. Play on it with *10-20* players
  5. After ~*25* minutes scripts won't be executed as fast as usual - the scripts get executed VERY slowly!
  6. Game-Client (of player) crashes
  7. 10 minutes of waiting the server is going to run out-of-memory
Additional Information

Before the 1.54.x update NO issues where located on our servers!
Everything was working fine until the update.
We cannot wait for an update of the mod-files and running our servers without mod-files is not makeable (because our community is based on the mod-files).


► Our server specifications:
RAM: 32 GB RAM
CPU: Intel Xeon E3-1271 v3 (3.60GHz)

We are using extDB-Addon ( on arma 3 server ) for connection to our MySQL server and a SSD hard-disk, of course!


► Just a partial part of the RPT log I sent to my buddy after server crash:

20:33:15 Virtual memory total 4095 MB (4294836224 B)
20:33:15 Virtual memory free 89 MB (93667328 B)
20:33:15 Physical memory free 25655 MB (26901499904 B)
20:33:15 Page file free 29644 MB (31084879872 B)
20:33:15 Process working set 3489 MB (3658833920 B)
20:33:15 Process page file used 3768 MB (3951542272 B)
20:33:15 Longest free VM region: 12451840 B
20:33:15 VM busy 4201234432 B (reserved 134545408 B, committed 4066689024 B, mapped 52789248 B), free 93601792 B
20:33:15 Small mapped regions: 17, size 81920 B
20:34:20 NetServer::finishDestroyPlayer(1180082922): DESTROY immediately after CREATE, both cancelled
20:34:29 NetServer::finishDestroyPlayer(115444701): DESTROY immediately after CREATE, both cancelled
20:35:06 NetServer::finishDestroyPlayer(1081893126): DESTROY immediately after CREATE, both cancelled
20:35:12 Exception code: C00000FD STACK_OVERFLOW at 1E3C69ED

Event Timeline

MrMatze edited Steps To Reproduce. (Show Details)Dec 2 2015, 9:44 PM
MrMatze edited Additional Information. (Show Details)
MrMatze set Category to Server.
MrMatze set Reproducibility to Always.
MrMatze set Severity to None.
MrMatze set Resolution to Open.
MrMatze set Legacy ID to 3681695413.May 8 2016, 1:11 PM
MrMatze edited a custom field.
onefox added a subscriber: onefox.May 8 2016, 1:11 PM
Adam added a comment.Dec 3 2015, 9:18 AM

Hey, thanks for reporting the issue. Is there a way how to directly contact you? I would like to reproduce the issue. Thanks!

onefox added a comment.Dec 3 2015, 9:35 AM

On our server we got also the ryan zombie mod running. Some links that could be usefull:
http://zombiebugs.site40.net/mantisbt/view.php?id=37
http://www.exilemod.com/topic/8093-server-slowly-dies-after-90-minutes/#comment-39979

Multiple source say that it can be reproduced with the 3.0 ryan zombie mod

We have the same problehm the update was -.-

2015/12/02, 19:06:12 Server: Object info 2:7701 not found during Changing Owner
2015/12/02, 19:06:13 Virtual memory total 4095 MB (4294836224 B)
2015/12/02, 19:06:13 Virtual memory free 1587 MB (1664966656 B)
2015/12/02, 19:06:13 Physical memory free 11840 MB (12415201280 B)
2015/12/02, 19:06:13 Page file free 11377 MB (11930284032 B)
2015/12/02, 19:06:13 Process working set 1977 MB (2073137152 B)
2015/12/02, 19:06:13 Process page file used 2269 MB (2379784192 B)
2015/12/02, 19:06:13 Longest free VM region: 1596932096 B
2015/12/02, 19:06:13 VM busy 2629935104 B (reserved 145432576 B, committed 2484502528 B, mapped 50765824 B), free 1664901120 B
2015/12/02, 19:06:13 Small mapped regions: 17, size 86016 B
2015/12/02, 19:08:36 NetServer::finishDestroyPlayer(1218848854): DESTROY immediately after CREATE, both cancelled
2015/12/02, 19:10:15 NetServer::finishDestroyPlayer(1647142698): DESTROY immediately after CREATE, both cancelled
2015/12/02, 19:10:16 NetServer::finishDestroyPlayer(150599454): DESTROY immediately after CREATE, both cancelled
2015/12/02, 19:10:26 NetServer::finishDestroyPlayer(1202330601): DESTROY immediately after CREATE, both cancelled
2015/12/02, 19:11:10 NetServer::finishDestroyPlayer(1251724135): DESTROY immediately after CREATE, both cancelled

onefox added a comment.Dec 3 2015, 8:41 PM

The problem is still there with the 1.54 hotfix. Here is another hint:

"ANY custom units or uniform packs should be temporarily disabled or you should rollback to 1.52, the reasons behind this are that whilst the mods "will run" on 1.54 they won't be working correctly given that there is a new hitpoint system that has been introduced, any custom uniform or unit/character pack will simply be broken, if using uniform its players, if using character packs for AI or zombies as an example the units will be incredibly OP and won't take damage effectively, this cannot be fixed by us (the community) and will require the mod authors to update the various addons, TRYK, RyanZombies etc are examples of these addons.

Was this on vanilla Exile or on modded whilst actually shooting at custom units or units wearing the custom clothing? Those are the only 2 circumstances that will cause the issues noted, hitpoints are setup as "dots" if you like inside the model where the game picks up bullet collisions, so its possible that you simply may have just shot them in the right place but this would not be the case 100% of the time in-action and inaccurately shooting.

HERE is a example of I believe the old system (credits to breaking point for the image source), I don't know what exactly has been changed in the newer versions but the units, clothing etc from current (pre 1.54) mods will be using this older system, if they upped hitpoint count then players wearing vanilla uniform etc will die a lot easier than those wearing the older clothing.

If you still wish to run custom units/clothing and are suffering a "out of memory" errors after a unspecified period of time, removing it from the server till it has been updated is all I can suggest, if the time it happens is guaranteed and is after an hour or 2 try cutting restart times down to before the crash" from KamikazeXeX

http://www.exilemod.com/topic/8074-collective-list-of-154-temp-fixes-till-exile-0936-updated/

onefox added a comment.Dec 5 2015, 3:08 AM

I added a dump from my local pc.
I can reproduce the mem leek easily with:

Just run this and everytime the server eats more mem. See arma3server_mem_leek.zip

for "_i" from 0 to 500 step 1 do {
_group = creategroup east;

_civil = _group createUnit ["B_Soldier_F", [175.769, 8030.45, 0], [], 0, "FORM"];

_civil setvariable ["civilrole", "civil", true];
_civil setvariable ["destination", [175.769, 8030.45, 0], false];
_civil setvariable ["wcprotected", true, false];
_civil disableAI "FSM";
_civil disableAI "TARGET";
_civil disableAI "AUTOTARGET";
_civil setBehaviour "CARELESS";
_civil disableConversation true;
_civil setCaptive true;

deletevehicle _civil;
deletegroup _group;
};

or do i miss to clean something?

BL1P added a subscriber: BL1P.May 8 2016, 1:11 PM
BL1P added a comment.Dec 5 2015, 8:35 AM

Hello we at our group are still getting the out of Memory error.

Link to the rpt and bidmp of last nights crash in .rar :-
https://mega.nz/#!K1MFhB7B!mAAuZDq5tj7JuBMhl9ucoIZBQKc-rQCer56RZk6VDqE

We are playing modded but not Zombies or anything weird like that.

Uploaded crash reports from to different roots with -memram changes and memoryalloctor changes but everything fails... server randomly crash and thats so silly....

MrMatze added a subscriber: MrMatze.May 8 2016, 1:11 PM

Attached 2 log-file packages of arma 3 client (while I was playing on updated server and fps dropped down until crash).

  • Already sent it to Adam via Skype.
BL1P added a comment.Dec 8 2015, 1:09 PM

Crashed alot over the weekend... just so you know :)
Currently our servers are down until a fix comes along. Saves on disappointment when the persistent mission crashes.

We are having the same issues on our 100Slot server. I hope this is being heard, its rough for our gaming community. I pray for a hotfix!

This crash is being plagued by alot of Altis Life Servers within the Australian Altis Life Community. Myself at OzzyGaming we are having this issue consistantly every 30 minutes. At most our server can stay up for about 2 hours. Before the 1.54 update the server could stay up for days without needing a restart.

Tonight My client is now even crashing with this error.

Within our community we have a tag saying when ever anyone breaks anything on the server. #BIbrokeIt has been past around a lot lately.

seems like it is still happening with the new 1.54.133701

s1k added a subscriber: s1k.May 8 2016, 1:11 PM
s1k added a comment.Dec 13 2015, 12:13 AM

Tried the latest RC build. We might actually be crashing even sooner now....
We have even crashed 30 minutes in, a few times(on tbb4)..

*worth mentioning*
Our server hadn't crashed in MONTHS prior to this update....
(wish others would list helpful data like this to compare against <3)

  • MALLOC = EVERY _bi malloc - even custom ones, primarily tbb4malloc_bi though.
  • extDB Version = extDB(35).
  • Game Mode/Version = Altis Life 3.x heavily modified.
  • Using bis/life_fnc_mp OR remoteExec = still using life_fnc_mp/BIS_fnc_MP.
  • Headless Client = no, removed.

Here's a link to our dump files/rpt, we were online for an hour only 50-60 players =(...
http://feedback.arma3.com/file_download.php?file_id=13315&type=bug

tried ryanzombies 3.3 and 1.54 still crashing :(

Hey Everyone.

I have been doing quite a bit of testing with this issue, and so everyone is aware I host OzzyGaming.com Altis Life server. We have been plagued with these crashes. On average from 10 minutes to an hour at best.

I have done quite abit of research into what has been happening and One thing stands out, Windows.

As such I wanted to confirm my findings by setting up a server hosted under linux. So I purchased a VPS with Ubuntu pre-installed and installed my server on that. And the results were shocking to say the least.

Not one crash at all. Desync just as you'd expect from a low grade VPS server but that server survived without a crash for upto 7 hours where it was restarted normally.

This issue has been around since late July. Not sure why but there was evidence of this issue back then. So Bohemia Interactive this has something to do with the way ARMA is trying to store data to memory under a Windows Based environment.

Also anyone experiencing this issue. grab a VPS server with what you afford and or use HyperV to run ARMA under Ubuntu or other linux based OS

Uploaded crashes with performance v6 (Crash_withV6.rar).

hirschi added a subscriber: hirschi.May 8 2016, 1:11 PM

Crashed again ..
This Time with Perf.BIN V6
1.54.133788
and this time with extDB2

https://www.dropbox.com/s/z0vu7sbf9ve14k2/DieLiga_ServerCrash_4.zip?dl=0

@hirschi

What OS are you using?

If that is Windows, seriously get a VPS with Ubuntu pre-installed, I promise you will not have any crashes.

Win Server 2012 R2 :-( .. it´s Hard worke to change the OS ... Maybe we try it ... last week i changed to extDB 65 .. it was hard work too ... Hope BI get a HotFix out before the gos to X-Mas holidays

But Thx... Mybe we try it out

Yeah I'm running windows server 2012 r2 as well. I'm not suggesting to change the OS. Just get a VPS to run ARMA 3 under Ubuntu. I am using this until Bohemia Interactive fixes the windows binaries.

Added another one with the performance exe v7.
The server is using less than 1 gb mem now (i improved some of my scripts)

different Crashes with V6 ... Sorry ,, don´t have time to report every Crash

https://www.dropbox.com/s/c2w25250k24j1ar/DieLiga_ServerCrash_5.zip?dl=0

maybe an update for you guys since the issue is still open.

im on windows server 2012 and had this issue for a while during december and my server was crashing like once a hour or something. When trying lots of thing to fix it i change 2 major things and it stop crashing, its been 48 hour now and still up. (Auto reboot each 6h)

What i change:
old version of extDB -> extDB2 release 70
change all my BIS_fnc_MP (and life_fnc_MP because i have an Altis Life server) to remoteExec and remoteExecCall

i dont really know if it fix the crash but at least it give me enought time so the server can reboot by itself every 6 hour

Out of memory crashes on RC build of EDEN running Dedicated Server :(

BL1P added a comment.Feb 13 2016, 7:47 AM

Our comunity has stopped playing arma because of this server bug