When running modded community servers with a high player count around 80+, server crashes at random times, the more players on, the more frequent it seems.
Server always crashes after saving players:
16:43:47.405 Saved 123 players ... 16:43:47.405 ENGINE (F): Crashed ======================================================= ------------------------------------------------------- Time: 16:43:47 Exception code: C0000005 ACCESS_VIOLATION at 3023672E
It seems after a fresh wipe, it will take some time until the crashes occour.
Hey! Deer Geez or whoever from BI will be reviewing this, I want to add logs of our server crashes. They look very similar to Funatics log.
What I've noticed that it is related to players DB save (RPT log ending).
Please, may I ask You to investigate ASAP cause this crash happens 2-3 times a hour. We've done a lot of tests on live server with wiping, disabling mods etc but still are not able to figure out a reason :(
80 players seems to be a magic bottleneck for community servers, so my guess its when players gear up and have all slots in use filled with stuff, it crashes above 80 players. Our Server now runs at 80 slots straight without a crash for 12h+
Same issue on our server, after changing to 80 slots no more crashes for a few hours now
Same problem
21:40:27.76 Saved 102 players ... 21:40:27.76 ENGINE (F): Crashed ======================================================= ------------------------------------------------------- Time: 21:40:27 Exception code: C0000005 ACCESS_VIOLATION at 00000000
The addresses are different, but the crash occurs at the moment of saving the players
21:57:21.292 Saved 103 players ... 21:57:21.292 ENGINE (F): Crashed ======================================================= ------------------------------------------------------- Time: 21:57:21 Exception code: C0000005 ACCESS_VIOLATION at E329672E
@Geez Please review this, for us running stable for 48h with 80 player cap, now also crashes at 80 players. I am in contact with other owners of big DayZ communities, and most have the same problem running different mods.
19:44:03.179 CHAR_DEBUG - SAVE 19:44:03.179 player John Q (dpnid = 1015906532) 19:44:03.179 Identity: playerID = 516 , charID = 516 19:44:03.179 Person : playerID = 516 , charID = 516 19:44:03.180 Saved 80 players ... 19:44:03.180 ENGINE (F): Crashed ======================================================= ------------------------------------------------------- Time: 19:44:03 Exception code: C0000005 ACCESS_VIOLATION at 3023672E Allocator: system graphics: No resolution: 160x120x32 Addons:
Greetings, we can also confirm this happening on our end as well. I was discussing this in a conversation with Funatic and I wrote to him this:
if we look at a healthy log file we should see this line
Saved XXX players ...
if the transaction commit to the sqlite db is completing normally we may be ruling that out, so i started look at what a healthy log should do after that completes. that should look something like this
20:39:08.950 CHAR_DEBUG - SAVE
20:39:08.950 player XYZ (dpnid = 36222217)
20:39:08.950 Identity: playerID = 1 , charID = 1
20:39:08.950 Person : playerID = 1 , charID = 1
20:39:08.950 Saved 1 players ...
20:39:09.247 [CE][LootRespawner] (PRIDummy) :: Item [12] causing search overtime: "PileOfWoodenPlanks"
20:39:11.721 Average server FPS: 16792.50 (measured interval: 10 s)
is this crashing instead at the next CE instruction?
10:50:38.932 CHAR_DEBUG - SAVE
10:50:38.932 player Valkyrie (dpnid = 65192228)
10:50:38.932 Identity: playerID = 346 , charID = 346
10:50:38.932 Person : playerID = 346 , charID = 346
10:50:38.932 CHAR_DEBUG - SAVE
10:50:38.932 player Octavia2469 (dpnid = 116410330)
10:50:38.932 Identity: playerID = 79 , charID = 79
10:50:38.932 Person : playerID = 79 , charID = 79
10:50:38.948 CHAR_DEBUG - SAVE
10:50:38.948 player Demon (dpnid = 93518874)
10:50:38.948 Identity: playerID = 618 , charID = 618
10:50:38.948 Person : playerID = 618 , charID = 618
10:50:38.948 CHAR_DEBUG - SAVE
10:50:38.948 player BustedBalls (dpnid = 114209891)
10:50:38.948 Identity: playerID = 689 , charID = 689
10:50:38.948 Person : playerID = 689 , charID = 689
10:50:38.948 CHAR_DEBUG - SAVE
10:50:38.948 player Genghis Christ (dpnid = 266205655)
10:50:38.948 Identity: playerID = 310 , charID = 310
10:50:38.948 Person : playerID = 310 , charID = 310
10:50:38.948 Saved 58 players ...
10:50:38.948 ENGINE (F): Crashed
Time: 10:50:39
Exception code: C0000005 ACCESS_VIOLATION at 20A0672E
Allocator: system
graphics: No{F4470034}
resolution: 160x120x32
16:19:31.655 Person : playerID = 196 , charID = 196
16:19:31.655 CHAR_DEBUG - SAVE
16:19:31.655 player JumpingToaster (dpnid = 72389220)
16:19:31.655 Identity: playerID = 862 , charID = 862
16:19:31.655 Person : playerID = 862 , charID = 862
16:19:31.655 CHAR_DEBUG - SAVE
16:19:31.655 player Wood (dpnid = 122066320)
16:19:31.655 Identity: playerID = 246 , charID = 246
16:19:31.655 Person : playerID = 246 , charID = 246
16:19:31.655 CHAR_DEBUG - SAVE
16:19:31.655 player Sm0kie (dpnid = 57333431)
16:19:31.655 Identity: playerID = 821 , charID = 821
16:19:31.655 Person : playerID = 821 , charID = 821
16:19:31.655 CHAR_DEBUG - SAVE
16:19:31.655 player Rompers (dpnid = 60323848)
16:19:31.655 Identity: playerID = 1146 , charID = 1146
16:19:31.655 Person : playerID = 1146 , charID = 1146
16:19:31.655 CHAR_DEBUG - SAVE
16:19:31.655 player QBGOTTEM (dpnid = 269190192)
16:19:31.655 Identity: playerID = 752 , charID = 752
16:19:31.655 Person : playerID = 752 , charID = 752
16:19:31.655 Saved 60 players ...
16:19:31.655 ENGINE (F): Crashed
Time: 16:19:31
Exception code: C0000005 ACCESS_VIOLATION at DFA1672E
Allocator: system
graphics: No
resolution: 160x120x32
Addons:
the slot size that is safe to use without crashing repeatedly - is continuing to go lower and lower to ensure the server can stay online
18:49:00.248 Person : playerID = 598 , charID = 598
18:49:00.248 CHAR_DEBUG - SAVE
18:49:00.248 player piggyoinkoink (dpnid = 65228239)
18:49:00.248 Identity: playerID = 278 , charID = 278
18:49:00.248 Person : playerID = 278 , charID = 278
18:49:00.248 CHAR_DEBUG - SAVE
18:49:00.248 player Undead (dpnid = 761643611)
18:49:00.248 Identity: playerID = 735 , charID = 735
18:49:00.248 Person : playerID = 735 , charID = 735
18:49:00.248 CHAR_DEBUG - SAVE
18:49:00.248 player RedBeerd (dpnid = 48545731)
18:49:00.248 Identity: playerID = 609 , charID = 609
18:49:00.248 Person : playerID = 609 , charID = 609
18:49:00.248 CHAR_DEBUG - SAVE
18:49:00.248 player Exotic (dpnid = 58380383)
18:49:00.248 Identity: playerID = 329 , charID = 329
18:49:00.248 Person : playerID = 329 , charID = 329
18:49:00.248 Saved 60 players ...
18:49:00.248 ENGINE (F): Crashed
Time: 18:49:00
Exception code: C0000005 ACCESS_VIOLATION at DFA1672E
Allocator: system
graphics: No
resolution: 160x120x32{F4470388}
20:40:28.608 CHAR_DEBUG - SAVE
20:40:28.608 player plex (dpnid = 1390242254)
20:40:28.608 Identity: playerID = 815 , charID = 815
20:40:28.608 Person : playerID = 815 , charID = 815
20:40:28.624 CHAR_DEBUG - SAVE
20:40:28.624 player TripinRay (dpnid = 47835795)
20:40:28.624 Identity: playerID = 319 , charID = 319
20:40:28.624 Person : playerID = 319 , charID = 319
20:40:28.624 CHAR_DEBUG - SAVE
20:40:28.624 player MaceCrastur (dpnid = 83051651)
20:40:28.624 Identity: playerID = 164 , charID = 164
20:40:28.624 Person : playerID = 164 , charID = 164
20:40:28.624 Saved 60 players ...
20:40:28.624 ENGINE (F): Crashed
Time: 20:40:28
Exception code: C0000005 ACCESS_VIOLATION at DFA1672E
Allocator: system
graphics: No
resolution: 160x120x32{F4470516}
I tried to force crash my current stable server with 70 slots by joining/staying with a character that has over 80 kb on items in inventory. Was not able to force crash.
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player rubber duckey (dpnid = 60757437)
10:41:05.392 Identity: playerID = 900 , charID = 900
10:41:05.392 Person : playerID = 900 , charID = 900
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player Diredusk (dpnid = 50120201)
10:41:05.392 Identity: playerID = 130 , charID = 130
10:41:05.392 Person : playerID = 130 , charID = 130
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player Alucard (dpnid = 36509411)
10:41:05.392 Identity: playerID = 614 , charID = 614
10:41:05.392 Person : playerID = 614 , charID = 614
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player sam (dpnid = 370882558)
10:41:05.392 Identity: playerID = 57 , charID = 57
10:41:05.392 Person : playerID = 57 , charID = 57
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player Bru (dpnid = 34747476)
10:41:05.392 Identity: playerID = 704 , charID = 704
10:41:05.392 Person : playerID = 704 , charID = 704
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player AcidicSoup (dpnid = 406056662)
10:41:05.392 Identity: playerID = 886 , charID = 886
10:41:05.392 Person : playerID = 886 , charID = 886
10:41:05.392 CHAR_DEBUG - SAVE
10:41:05.392 player Nick (dpnid = 84158099)
10:41:05.392 Identity: playerID = 320 , charID = 320
10:41:05.392 Person : playerID = 320 , charID = 320
10:41:05.407 CHAR_DEBUG - SAVE
10:41:05.407 player Moonshadow (dpnid = 245376825)
10:41:05.407 Identity: playerID = 871 , charID = 871
10:41:05.407 Person : playerID = 871 , charID = 871
10:41:05.407 CHAR_DEBUG - SAVE
10:41:05.407 player Kristofarian (dpnid = 72177107)
10:41:05.407 Identity: playerID = 380 , charID = 380
10:41:05.407 Person : playerID = 380 , charID = 380
10:41:05.407 Saved 60 players ...
10:41:05.407 ENGINE (F): Crashed
Time: 10:41:05
Exception code: C0000005 ACCESS_VIOLATION at 20A0672E
Allocator: system
graphics: No
resolution: 160x120x32{F4471323}
14:54:47.360 player Alucard (dpnid = 839640907)
14:54:47.360 Identity: playerID = 614 , charID = 614
14:54:47.360 Person : playerID = 614 , charID = 614
14:54:47.375 CHAR_DEBUG - SAVE
14:54:47.375 player Racodi (dpnid = 89452828)
14:54:47.375 Identity: playerID = 212 , charID = 212
14:54:47.375 Person : playerID = 212 , charID = 212
14:54:47.375 CHAR_DEBUG - SAVE
14:54:47.375 player XxX_Astonish_XxX (dpnid = 1731879700)
14:54:47.375 Identity: playerID = 309 , charID = 309
14:54:47.375 Person : playerID = 309 , charID = 309
14:54:47.375 CHAR_DEBUG - SAVE
14:54:47.375 player Solow357 (dpnid = 1292482467)
14:54:47.375 Identity: playerID = 428 , charID = 428
14:54:47.375 Person : playerID = 428 , charID = 428
14:54:47.375 CHAR_DEBUG - SAVE
14:54:47.375 player Jlynx (dpnid = 84333513)
14:54:47.375 Identity: playerID = 427 , charID = 427
14:54:47.375 Person : playerID = 427 , charID = 427
14:54:47.375 CHAR_DEBUG - SAVE
14:54:47.375 player lilshark1 (dpnid = 59024189)
14:54:47.375 Identity: playerID = 347 , charID = 347
14:54:47.375 Person : playerID = 347 , charID = 347
14:54:47.375 CHAR_DEBUG - SAVE
14:54:47.375 player moons (dpnid = 54595680)
14:54:47.375 Identity: playerID = 942 , charID = 942
14:54:47.375 Person : playerID = 942 , charID = 942
14:54:47.391 CHAR_DEBUG - SAVE
14:54:47.391 player Swan (dpnid = 80989548)
14:54:47.391 Identity: playerID = 356 , charID = 356
14:54:47.391 Person : playerID = 356 , charID = 356
14:54:47.391 Saved 60 players ...
14:54:47.391 ENGINE (F): Crashed
Time: 14:54:47
Exception code: C0000005 ACCESS_VIOLATION at 20A0672E
Allocator: system
graphics: No
resolution: 160x120x32{F4472033}
16:48:30.227 Identity: playerID = 156 , charID = 156
16:48:30.227 Person : playerID = 156 , charID = 156
16:48:30.227 CHAR_DEBUG - SAVE
16:48:30.227 player White Lightning (dpnid = 1270600052)
16:48:30.227 Identity: playerID = 220 , charID = 220
16:48:30.227 Person : playerID = 220 , charID = 220
16:48:30.227 CHAR_DEBUG - SAVE
16:48:30.227 player AshDerDrache (dpnid = 527569724)
16:48:30.227 Identity: playerID = 1179 , charID = 1179
16:48:30.227 Person : playerID = 1179 , charID = 1179
16:48:30.227 CHAR_DEBUG - SAVE
16:48:30.227 player Chewrang (dpnid = 42922530)
16:48:30.227 Identity: playerID = 106 , charID = 106
16:48:30.227 Person : playerID = 106 , charID = 106
16:48:30.227 CHAR_DEBUG - SAVE
16:48:30.227 player todd (dpnid = 82641920)
16:48:30.227 Identity: playerID = 601 , charID = 601
16:48:30.227 Person : playerID = 601 , charID = 601
16:48:30.227 CHAR_DEBUG - SAVE
16:48:30.227 player RedBeerd (dpnid = 42967535)
16:48:30.227 Identity: playerID = 609 , charID = 609
16:48:30.227 Person : playerID = 609 , charID = 609
16:48:30.227 CHAR_DEBUG - SAVE
16:48:30.227 player rqc88 (dpnid = 54639675)
16:48:30.227 Identity: playerID = 67 , charID = 67
16:48:30.227 Person : playerID = 67 , charID = 67
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player Wood (dpnid = 120708610)
16:48:30.242 Identity: playerID = 246 , charID = 246
16:48:30.242 Person : playerID = 246 , charID = 246
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player IggysPiggy (dpnid = 936257010)
16:48:30.242 Identity: playerID = 754 , charID = 754
16:48:30.242 Person : playerID = 754 , charID = 754
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player GeekZ (dpnid = 46110089)
16:48:30.242 Identity: playerID = 231 , charID = 231
16:48:30.242 Person : playerID = 231 , charID = 231
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player Titus (dpnid = 109761635)
16:48:30.242 Identity: playerID = 344 , charID = 344
16:48:30.242 Person : playerID = 344 , charID = 344
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player MICH43L (dpnid = 41389629)
16:48:30.242 Identity: playerID = 17 , charID = 17
16:48:30.242 Person : playerID = 17 , charID = 17
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player BassMonster173 (dpnid = 77756243)
16:48:30.242 Identity: playerID = 249 , charID = 249
16:48:30.242 Person : playerID = 249 , charID = 249
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player TacoTactical96 (dpnid = 140429850)
16:48:30.242 Identity: playerID = 1305 , charID = 1305
16:48:30.242 Person : playerID = 1305 , charID = 1305
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player Capn Moe (dpnid = 330272874)
16:48:30.242 Identity: playerID = 261 , charID = 261
16:48:30.242 Person : playerID = 261 , charID = 261
16:48:30.242 CHAR_DEBUG - SAVE
16:48:30.242 player Spitfire (dpnid = 41164199)
16:48:30.242 Identity: playerID = 72 , charID = 72
16:48:30.242 Person : playerID = 72 , charID = 72
16:48:30.258 CHAR_DEBUG - SAVE
16:48:30.258 player sMuRk (dpnid = 57932778)
16:48:30.258 Identity: playerID = 866 , charID = 866
16:48:30.258 Person : playerID = 866 , charID = 866
16:48:30.258 Saved 60 players ...
16:48:30.258 ENGINE (F): Crashed
Time: 16:48:30
Exception code: C0000005 ACCESS_VIOLATION at DFA1672E
Allocator: system
graphics: No
resolution: 160x120x32
17:34:57.939 CHAR_DEBUG - SAVE
17:34:57.939 player sMuRk (dpnid = 58208794)
17:34:57.939 Identity: playerID = 866 , charID = 866
17:34:57.939 Person : playerID = 866 , charID = 866
17:34:57.939 CHAR_DEBUG - SAVE
17:34:57.939 player Anthony (dpnid = 118376109)
17:34:57.939 Identity: playerID = 81 , charID = 81
17:34:57.939 Person : playerID = 81 , charID = 81
17:34:57.939 CHAR_DEBUG - SAVE
17:34:57.939 player BassMonster173 (dpnid = 470627420)
17:34:57.939 Identity: playerID = 249 , charID = 249
17:34:57.939 Person : playerID = 249 , charID = 249
17:34:57.939 CHAR_DEBUG - SAVE
17:34:57.939 player MAINSHOTS (dpnid = 66849115)
17:34:57.939 Identity: playerID = 582 , charID = 582
17:34:57.939 Person : playerID = 582 , charID = 582
17:34:57.955 CHAR_DEBUG - SAVE
17:34:57.955 player AshDerDrache (dpnid = 640071104)
17:34:57.955 Identity: playerID = 1179 , charID = 1179
17:34:57.955 Person : playerID = 1179 , charID = 1179
17:34:57.955 CHAR_DEBUG - SAVE
17:34:57.955 player ButtLipps (dpnid = 47900934)
17:34:57.955 Identity: playerID = 1086 , charID = 1086
17:34:57.955 Person : playerID = 1086 , charID = 1086
17:34:57.955 CHAR_DEBUG - SAVE
17:34:57.955 player GeekZ (dpnid = 48916506)
17:34:57.955 Identity: playerID = 231 , charID = 231
17:34:57.955 Person : playerID = 231 , charID = 231
17:34:57.955 Saved 60 players ...
17:34:57.955 ENGINE (F): Crashed
Time: 17:34:58
Exception code: C0000005 ACCESS_VIOLATION at DFA1672E
Allocator: system
graphics: No
resolution: 160x120x32
At lease all errors at his server are occurring at same memory addresses:
Exception code: C0000005 ACCESS_VIOLATION at 20A0672E
Exception code: C0000005 ACCESS_VIOLATION at DFA1672E
As well as Funatic's logs are showing same faulty address for his two different crashes:
Exception code: C0000005 ACCESS_VIOLATION at 3023672E
And those are somewhat similar to me with other faulty addresses from different servers provided at this topic:
Exception code: C0000005 ACCESS_VIOLATION at E329672E
Exception code: C0000005 ACCESS_VIOLATION at 3D6E672E
So my conclusion is that at the same single server this error is occurring at the same spot every time.
And at different servers the similarity is a ####672E in address.
so far its also my conclution that the crashes increase while time is progressing on players. players.db wipe allows us to raise slots again for some time
I can confirm this is an issue.
This has been occurring ever since the update. At first I thought this was an issue specifically with my server, but I have looked around numerous discords and spoke with many different server owners and they are facing the same issue.
Our mods aren't similar at all either. And the mods we do have similar, are used by the entire modding community. (E.g. Admin Tools, CF, Dabs Framework).
I have lowered the population to 80, and so have other servers and we are no longer crashing.
19:07:39.143 Saved 87 players ... 19:07:39.143 ENGINE (F): Crashed Time: 19:07:39 Exception code: C0000005 ACCESS_VIOLATION at CD0F672E Allocator: system graphics: No resolution: 160x120x32
19:24:42.559 Saved 84 players ... 19:24:42.559 ENGINE (F): Crashed Time: 19:24:42 Exception code: C0000005 ACCESS_VIOLATION at CD0F672E Allocator: system graphics: No resolution: 160x120x32
19:44:06.292 Saved 83 players ... 19:44:06.292 ENGINE (F): Crashed Time: 19:44:06 Exception code: C0000005 ACCESS_VIOLATION at CD0F672E Allocator: system graphics: No resolution: 160x120x32
As for all of you wiping your servers and testing inventories, it has nothing to do with it. Don't wipe and kill your servers, I run a server that hasn't wiped for 2.5 years and I am facing this exact issue.
@Geez I have heard that you guys are looking into this, for which I am super grateful. But could you please tell us if you have any suspicion yet?
Since the server crash always happens on player save, could it have to do with too many items on them?
I noticed that at the time I'm commenting, a dayz update came out, would this be the hotfix planned for the fix? @Geez
@Geez Thank you and the whole dev team to look into this, also on behalf of oll other owners i would just say "thank you!"
Can you confirm if this fix has already been released? Because my server is still crashing with even fewer players after a recent DayZ update on 10/31 in Brazil
If this hotfix has not yet been released, could you please update this post here to let us know that this hotfix has been released?
I can confirm that the hotfix was released and server runs fine and not crashing anymore while saving player data. @Machineblack you might have not the problem that is the same as in ticket?
Your server reaches more than 110 players and doesn't crash?
Mine after the update they released (source steamdb):
Date Day Time Patch Title BuildID
31 October 2024 Thu 13:39 No title 16252220
Now it crashes with 70 players, and after more than 100 it crashes even more
Did you do anything, or just update your server?
@Geez Hello friend, any update on the error : Exception code: C0000005 ACCESS_VIOLATION ???
18:40:16.266 ENGINE (F): Crashed
Time: 18:40:16
Exception code: C0000005 ACCESS_VIOLATION at 807AF287
Allocator: system
graphics: No
resolution: 160x120x32