Page MenuHomeFeedback Tracker

Server crash - 130+ players - Linux x64
New, NormalPublic

Description

We are experimenting regular server crashes (usually between 1 and 20 minutes after mission start) during our missions with more than 130 players.
We use the latest PERF binaries (linux x64) 2.10.149973 and stable branche one.

Details

Severity
Crash
Resolution
Open
Reproducibility
Unable To Reproduce
Operating System
Linux x64
Operating System Version
Linux Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-122-generic x86_64)
Category
Server
Steps To Reproduce

Start the server, have the players slotted in, go on map, start the mission.
Usually, the crash happens when a player connects (or disconnects) to (or from) the multiplayer session, not the server itself

Additional Information

Server was always completly restarted before the mission

Just before the crash, the server rpt show 4 to 20 messages : Array index out of range

Mods used during last crash

Crash dump:

Mission file:

Server config: https://www.hetzner.com/dedicated-rootserver/ax51-nvme

After restarting the server (after the crash), we can usually play the mission without any issue.

EDIT 2022-10-07: New server crashes with 140+ players. No issue reported with 10-15 players...
Crashdump and RPT available here:

EDIT 2022-10-13: New server crashes with 130+ players.On stable branch.
Syslog error:

Oct 13 21:44:42 OFCRA-HETZNER kernel: [818989.907108] traps: NetSndMsg[229324] general protection fault ip:18bdf11 sp:7f5d2296ca38 error:0 in arma3server_x64[40e000+1e62000]

Crashdump available here:

EDIT 2022-10-27: New server crashes with 140+ players.On perf branch (v6)
Crashdump available here:

Event Timeline

Manchot created this task.Oct 3 2022, 11:02 AM
Manchot edited Steps To Reproduce. (Show Details)Oct 3 2022, 11:22 AM
ilbinek added a subscriber: ilbinek.Oct 7 2022, 9:56 AM
Manchot edited Additional Information. (Show Details)Oct 7 2022, 10:43 AM
Manchot updated the task description. (Show Details)Oct 14 2022, 1:14 PM
Manchot edited Additional Information. (Show Details)
Manchot edited Additional Information. (Show Details)Oct 27 2022, 10:49 PM
dedmen added a subscriber: dedmen.Mon, Nov 14, 12:15 PM

core-2022-10-27_21_25.rar

Unclear, I think its not showing me correct crash location

core-2022-10-13.rar

Uh, same. Surely I*'m doing something wrong.
0x00000000018bdf11 no, no I'm not that matches that log entry.

Something tried to create a light, I'd say somethig particle effect. And while assigning its position it crashed. But I don't see how that could go wrong, it seems to all be valid.

Okey actually I just looked at the code and thought "wtf?! why?" so.. its weird definitely but still shouldn't crash there. And the "wtf" doesn't match the crash segfault. Well I fixed the wtf thing but probably not gonna change anything.

core-2022-10-06.rar

Okey there is a efficiency bug in code here, that when fixed will not run the code that "crashed" here.
Segfault while comparing a string with itself, that shouldn't be possible to segfault...

These crashes just don't seem to make any sense.

core-2022-09-29.rar
These all seem so random
This looks like memory corruption... certainly interesting. A single bitflip tho? 🤔

Just before the crash, the server rpt show 4 to 20 messages : Array index out of range

I couldn't see any of this