We are experimenting regular server crashes (usually between 1 and 20 minutes after mission start) during our missions with more than 130 players.
We use the latest PERF binaries (linux x64) 2.10.149973 and stable branche one.
Description
Description
Details
Details
- Severity
- Crash
- Resolution
- Open
- Reproducibility
- Unable To Reproduce
- Operating System
- Linux x64
- Operating System Version
- Linux Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-122-generic x86_64)
- Category
- Server
Steps To Reproduce
Start the server, have the players slotted in, go on map, start the mission.
Usually, the crash happens when a player connects (or disconnects) to (or from) the multiplayer session, not the server itself
Additional Information
Server was always completly restarted before the mission
Just before the crash, the server rpt show 4 to 20 messages : Array index out of range
Mods used during last crash
- http://steamcommunity.com/sharedfiles/filedetails/?id=893349825
- http://steamcommunity.com/sharedfiles/filedetails/?id=1515851169
- http://steamcommunity.com/sharedfiles/filedetails/?id=893339590
- http://steamcommunity.com/sharedfiles/filedetails/?id=1515845502
- http://steamcommunity.com/sharedfiles/filedetails/?id=1673456286
- http://steamcommunity.com/sharedfiles/filedetails/?id=463939057
- http://steamcommunity.com/sharedfiles/filedetails/?id=773131200
- http://steamcommunity.com/sharedfiles/filedetails/?id=884966711
- http://steamcommunity.com/sharedfiles/filedetails/?id=2174495332
- http://steamcommunity.com/sharedfiles/filedetails/?id=773125288
- http://steamcommunity.com/sharedfiles/filedetails/?id=866772689
- http://steamcommunity.com/sharedfiles/filedetails/?id=713709341
- http://steamcommunity.com/sharedfiles/filedetails/?id=615007497
- http://steamcommunity.com/sharedfiles/filedetails/?id=639837898
- http://steamcommunity.com/sharedfiles/filedetails/?id=730310357
- http://steamcommunity.com/sharedfiles/filedetails/?id=648172507
- http://steamcommunity.com/sharedfiles/filedetails/?id=820924072
- http://steamcommunity.com/sharedfiles/filedetails/?id=1314910827
- http://steamcommunity.com/sharedfiles/filedetails/?id=1889104923
- http://steamcommunity.com/sharedfiles/filedetails/?id=450814997
- http://steamcommunity.com/sharedfiles/filedetails/?id=583496184
- http://steamcommunity.com/sharedfiles/filedetails/?id=708665067
- http://steamcommunity.com/sharedfiles/filedetails/?id=333310405
- http://steamcommunity.com/sharedfiles/filedetails/?id=825179978
- http://steamcommunity.com/sharedfiles/filedetails/?id=1224892496
- http://steamcommunity.com/sharedfiles/filedetails/?id=906748433
- http://steamcommunity.com/sharedfiles/filedetails/?id=779520435
- http://steamcommunity.com/sharedfiles/filedetails/?id=667352839
- http://steamcommunity.com/sharedfiles/filedetails/?id=1541041923
- http://steamcommunity.com/sharedfiles/filedetails/?id=667454606
- http://steamcommunity.com/sharedfiles/filedetails/?id=1399779368
- http://steamcommunity.com/sharedfiles/filedetails/?id=2688071833
- http://steamcommunity.com/sharedfiles/filedetails/?id=843425103
- http://steamcommunity.com/sharedfiles/filedetails/?id=843593391
- http://steamcommunity.com/sharedfiles/filedetails/?id=843632231
- http://steamcommunity.com/sharedfiles/filedetails/?id=843577117
- http://steamcommunity.com/sharedfiles/filedetails/?id=1661066023
- http://steamcommunity.com/sharedfiles/filedetails/?id=770418577
- http://steamcommunity.com/sharedfiles/filedetails/?id=324952672
- http://steamcommunity.com/sharedfiles/filedetails/?id=894678801
- http://steamcommunity.com/sharedfiles/filedetails/?id=2141020863
- http://steamcommunity.com/sharedfiles/filedetails/?id=1703187116
Crash dump:
Mission file:
Server config: https://www.hetzner.com/dedicated-rootserver/ax51-nvme
After restarting the server (after the crash), we can usually play the mission without any issue.
EDIT 2022-10-07: New server crashes with 140+ players. No issue reported with 10-15 players...
Crashdump and RPT available here:
EDIT 2022-10-13: New server crashes with 130+ players.On stable branch.
Syslog error:
Oct 13 21:44:42 OFCRA-HETZNER kernel: [818989.907108] traps: NetSndMsg[229324] general protection fault ip:18bdf11 sp:7f5d2296ca38 error:0 in arma3server_x64[40e000+1e62000]
Crashdump available here:
EDIT 2022-10-27: New server crashes with 140+ players.On perf branch (v6)
Crashdump available here:
EDIT 2022-12-12: New server crashes with ~80 players.On perf branch (v10)
Just before the crash, the server rpt shows many messages : Array index out of range
Crashdump available here:
Dec 8 22:07:21 OFCRA-HETZNER kernel: [3744372.111292] traps: arma3server_x64[3885982] general protection fault ip:1645d65 sp:7ffeaa08ece0 error:0 in arma3server_x64[40e000+1e3f000]
Event Timeline
Comment Actions
core-2022-10-27_21_25.rar
Unclear, I think its not showing me correct crash location
core-2022-10-13.rar
Uh, same. Surely I*'m doing something wrong.
0x00000000018bdf11 no, no I'm not that matches that log entry.
Something tried to create a light, I'd say somethig particle effect. And while assigning its position it crashed. But I don't see how that could go wrong, it seems to all be valid.
Okey actually I just looked at the code and thought "wtf?! why?" so.. its weird definitely but still shouldn't crash there. And the "wtf" doesn't match the crash segfault. Well I fixed the wtf thing but probably not gonna change anything.
core-2022-10-06.rar
Okey there is a efficiency bug in code here, that when fixed will not run the code that "crashed" here.
Segfault while comparing a string with itself, that shouldn't be possible to segfault...
These crashes just don't seem to make any sense.
core-2022-09-29.rar
These all seem so random
This looks like memory corruption... certainly interesting. A single bitflip tho? 🤔
Just before the crash, the server rpt show 4 to 20 messages : Array index out of range
I couldn't see any of this