Page MenuHomeFeedback Tracker
Create Task
Maniphest T155407

Arma 3 Dedicated server lag spike
Closed, ResolvedPublic
Actions

Description

Arma 3 dedicated server lags out completly and as a result every player connected gets kicked for exceeding the MaxDesync or MaxPing limit set in the server.cfg when one is set, BattlEye: Client Not Responding if MaxDesync and MaxPing aren't set.

Sometimes the players can reconnect after that but sometimes they can't even reconnect and the server has to be rebooted manually.

19:21:42 Server load: FPS 33, memory used: 2819 MB, out: 7835 Kbps, in: 710 Kbps, NG:0, G:5546, BE-NG:0, BE-G:0, Players: 97 (L:1, R:0, B:0, G:95, D:1)
19:21:47 Server load: FPS 29, memory used: 2819 MB, out: 10455 Kbps, in: 769 Kbps, NG:0, G:18330, BE-NG:0, BE-G:0, Players: 97 (L:1, R:0, B:0, G:95, D:1)
19:21:52 Player Mouleshote disconnected.
19:21:52 BattlEye Server: Player #333 Mouleshote disconnected
19:21:52 Server load: FPS 29, memory used: 2820 MB, out: 9640 Kbps, in: 688 Kbps, NG:16544, G:48830, BE-NG:0, BE-G:0, Players: 96 (L:0, R:0, B:0, G:95, D:1)
19:21:57 Server load: FPS 21, memory used: 2828 MB, out: 18149 Kbps, in: 793 Kbps, NG:0, G:12596, BE-NG:0, BE-G:0, Players: 96 (L:0, R:0, B:0, G:95, D:1)
19:21:58 [BLW] Thibaut Boulawa uses modified data file
19:22:00 BattlEye Server: Player #333 [BLW] Thibaut Boulawa (77.195.110.71:2304) connected
19:22:00 Player [BLW] Thibaut Boulawa connecting.
19:22:02 Server load: FPS 19, memory used: 2830 MB, out: 21290 Kbps, in: 825 Kbps, NG:0, G:20180, BE-NG:0, BE-G:0, Players: 97 (L:1, R:0, B:0, G:95, D:1)
19:22:02 BattlEye Server: Player #333 [BLW] Thibaut Boulawa - BE GUID: afc7bec04654ff98923aabdc25517981
19:22:02 Player [BLW] Thibaut Boulawa connected (id=76561198355646624).
19:22:02 BattlEye Server: Verified GUID (afc7bec04654ff98923aabdc25517981) of player #333 [BLW] Thibaut Boulawa
19:22:07 Server load: FPS 14, memory used: 2839 MB, out: 27873 Kbps, in: 945 Kbps, NG:15792, G:23782, BE-NG:0, BE-G:0, Players: 97 (L:1, R:0, B:0, G:95, D:1)
19:22:12 Server load: FPS 4, memory used: 2857 MB, out: 35350 Kbps, in: 1066 Kbps, NG:0, G:263363, BE-NG:9, BE-G:0, Players: 97 (L:1, R:0, B:0, G:95, D:1)
19:22:28 Server load: FPS 0, memory used: 3081 MB, out: 15744 Kbps, in: 1154 Kbps, NG:260192, G:3995115, BE-NG:54, BE-G:0, Players: 97 (L:0, R:1, B:0, G:95, D:1)
19:22:33 Paul Renner uses modified data file
19:22:33 Server load: FPS 0, memory used: 2939 MB, out: 121094 Kbps, in: 491 Kbps, NG:85164, G:2571835, BE-NG:63, BE-G:0, Players: 98 (L:1, R:1, B:0, G:95, D:1)

Console logs show ServerFPS go to 33FPS to 0 in 40 secs with no particular activity on the server at the moment.

As a result, every player on the server gets kicked:

19:23:08 Kicking player name='[S.K.B]Elpolo elchapo' ID=1058031782, steamID=76561198202629844. Ping too high (834/ 300 ms)
19:23:08 Kicking player name='[MG] SmilleRR Akrapovic' ID=269983191, steamID=76561198154605798. Desync too high (256/ 150 )
19:23:08 Kicking player name='[L.A.D]Capone Tinel' ID=1844009206, steamID=76561198063403303. Ping too high (888/ 300 ms)
19:23:08 Kicking player name='[BLW] Clement Boulawa' ID=1207587198, steamID=76561198126595972. Desync too high (192/ 150 )
19:23:08 Kicking player name='[SG-AS] Lavoiserie J.' ID=114708905, steamID=76561198021743123. Ping too high (674/ 300 ms)
19:23:08 Kicking player name='[BLW] Thibaut Boulawa' ID=1363587942, steamID=76561198355646624. Ping too high (690/ 300 ms)

RPT logs show millions of these:

2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending
2020/11/27, 19:22:10 Server: Network message 716a7a4 is pending

Network usage on the server when it happens:

2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------
2020/11/27, 19:22:50  Server - Sent
2020/11/27, 19:22:50 Type                                     |          msg |     msg/s |            B |          B/s | avg. B
2020/11/27, 19:22:50 Type_284                                 |     14370238 | 263175.13 |    406468016 |   5797667.00 |   28.3
2020/11/27, 19:22:50 Type_23                                  |     21534790 |   3776.52 |  11215604430 |   1966863.00 |  520.8
2020/11/27, 19:22:50 Type_127                                 |     39464461 |   4407.97 |   2670095516 |    314986.38 |   67.7
2020/11/27, 19:22:50 Type_133                                 |     18428256 |   2654.07 |   1085718197 |    156587.30 |   58.9
2020/11/27, 19:22:50 Type_10                                  |       602491 |     47.55 |    886243745 |    110145.27 | 1471.0
2020/11/27, 19:22:50 Type_453                                 |      8106433 |    599.53 |    599016621 |     44364.98 |   73.9
2020/11/27, 19:22:50 Type_326                                 |     44979709 |   2464.95 |    794207623 |     44228.51 |   17.7
2020/11/27, 19:22:50 Type_21                                  |       808270 |     73.44 |    404769063 |     36777.25 |  500.8
2020/11/27, 19:22:50 Type_92                                  |     21168416 |    492.77 |   1091413841 |     25600.99 |   51.6
2020/11/27, 19:22:50 Type_462                                 |       824823 |     91.73 |    190471295 |     21904.84 |  230.9
2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------
2020/11/27, 19:22:50  Server - Received
2020/11/27, 19:22:50 Type                                     |          msg |     msg/s |            B |          B/s | avg. B
2020/11/27, 19:22:50 Type_284                                 |       119058 |   2766.33 |      2690689 |     49828.37 |   22.6
2020/11/27, 19:22:50 Type_127                                 |      3093842 |    319.34 |    261831192 |     28227.24 |   84.6
2020/11/27, 19:22:50 Type_133                                 |      2130462 |    271.59 |    143972258 |     18427.99 |   67.6
2020/11/27, 19:22:50 Type_163                                 |      2771971 |    291.05 |     72948146 |      7868.89 |   26.3
2020/11/27, 19:22:50 Type_255                                 |       234195 |     19.49 |     91168579 |      7473.51 |  389.3
2020/11/27, 19:22:50 Type_453                                 |      1049115 |     83.47 |     87916083 |      7035.39 |   83.8
2020/11/27, 19:22:50 Type_326                                 |      1680595 |    179.89 |     27182128 |      3020.43 |   16.2
2020/11/27, 19:22:50 Type_120                                 |       229950 |     30.79 |     18568600 |      2572.90 |   80.8
2020/11/27, 19:22:50 Type_92                                  |      2562080 |     43.68 |    139579871 |      2379.65 |   54.5
2020/11/27, 19:22:50 Type_126                                 |       126183 |     13.76 |     14616398 |      1608.02 |  115.8
2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------

Client network diag:

2020/11/27, 19:22:50 Client
2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------
2020/11/27, 19:22:50 Raw statistics 18575.9
2020/11/27, 19:22:50                                              player |     sent msg |   sent msg/s |       sent B |     sent B/s |      rec msg |    rec msg/s |        rec B |      rec B/s |       DS min |       DS max |       DS avg |       BW min |       BW max |       BW avg |
2020/11/27, 19:22:50                                               Total |            0 |         0.00 |            0 |         0.00 |            0 |         0.00 |            0 |         0.00 |         0.00 |         0.00 |         0.00 |
2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------
2020/11/27, 19:22:50  Client - Sent
2020/11/27, 19:22:50 Type                                     |          msg |     msg/s |            B |          B/s | avg. B
2020/11/27, 19:22:50 Type_373                                 |       600851 |     91.06 |    251913665 |     15227.54 |  419.3
2020/11/27, 19:22:50 Type_92                                  |       681503 |     45.38 |     35253708 |      2350.83 |   51.7
2020/11/27, 19:22:50 Type_120                                 |       780290 |     34.22 |     51715682 |      2287.48 |   66.3
2020/11/27, 19:22:50 Type_348                                 |         1673 |      0.10 |     15531693 |      2216.61 | 9283.7
2020/11/27, 19:22:50 Type_10                                  |         8761 |      0.47 |     11905107 |      1122.55 | 1358.9
2020/11/27, 19:22:50 Type_127                                 |       275981 |      9.73 |     17640057 |       622.72 |   63.9
2020/11/27, 19:22:50 Type_379                                 |        13351 |      3.20 |      1035567 |       341.96 |   77.6
2020/11/27, 19:22:50 Type_160                                 |       165609 |      7.36 |      4485277 |       200.02 |   27.1
2020/11/27, 19:22:50 Type_193                                 |        49091 |      2.10 |      2945460 |       125.95 |   60.0
2020/11/27, 19:22:50 Type_91                                  |        43781 |      3.27 |      1634715 |       122.62 |   37.3
2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------
2020/11/27, 19:22:50  Client - Received
2020/11/27, 19:22:50 Type                                     |          msg |     msg/s |            B |          B/s | avg. B
2020/11/27, 19:22:50 Type_284                                 |       137190 |   2766.66 |      3737188 |     60893.47 |   27.2
2020/11/27, 19:22:50 Type_127                                 |      2814611 |    134.05 |    190356808 |      9526.09 |   67.6
2020/11/27, 19:22:50 Type_133                                 |      2056635 |     99.73 |    121130520 |      5883.81 |   58.9
2020/11/27, 19:22:50 Type_453                                 |      1008141 |     30.52 |     74444555 |      2258.56 |   73.8
2020/11/27, 19:22:50 Type_92                                  |      2448985 |     41.55 |    126192201 |      2160.27 |   51.5
2020/11/27, 19:22:50 Type_462                                 |        35909 |      3.30 |      8709463 |       906.50 |  242.5
2020/11/27, 19:22:50 Type_120                                 |       116527 |     12.66 |      8127547 |       898.01 |   69.7
2020/11/27, 19:22:50 Type_326                                 |       720413 |     38.25 |     12646855 |       686.26 |   17.6
2020/11/27, 19:22:50 Type_372                                 |        38820 |      2.27 |     38421288 |       662.40 |  989.7
2020/11/27, 19:22:50 Type_181                                 |       111068 |     13.89 |      4442430 |       560.11 |   40.0
2020/11/27, 19:22:50 -----------------------------------------------------------------------------------------------------------------------------------------------

Server basic.cfg:

MinBandwidth=400000000;
MaxBandwidth=500000000;
MaxMsgSend=768;
MaxSizeGuaranteed=830;
MaxSizeNonguaranteed=224;
MinErrorToSend=0.004;
MinErrorToSendNear=0.04;
MaxCustomFileSize=0;

Server.cfg :

hostName = "[FR] Altis Life | Altis4life.net | ts.altis4life.net";
password = "";
passwordAdmin = "********";
serverCommandPassword = "***********";
logFile = "";

motd[] = {};
motdInterval = 3;

maxPlayers = 140;
kickduplicate = 0;
verifySignatures = 2;
allowedFilePatching = 0;
requiredSecureId = 0;
allowedLoadFileExtensions[] = {"hpp","sqs","sqf","fsm","cpp","paa","txt","xml","inc","ext","sqm","ods","fxy","lip","csv","kb","bik","bikb","html","htm","biedi"};
allowedPreprocessFileExtensions[] = {"hpp","sqs","sqf","fsm","cpp","paa","txt","xml","inc","ext","sqm","ods","fxy","lip","csv","kb","bik","bikb","html","htm","biedi"};
allowedHTMLLoadExtensions[] = {"htm","html","xml","txt"};

disconnectTimeout = 15;
maxdesync = 150;
maxpacketloss = 50;
maxping = 300;
kickClientsOnSlowNetwork[] = { 1, 1, 1, 1};
lobbyIdleTimeout = 100;

voteMissionPlayers = 0;
voteThreshold = 1;

disableVoN = 0;
vonCodecQuality = 30;
vonCodec = 1;

persistent = 1;
timeStampFormat = "full";
BattlEye = 1;
battleyeLicense=1;

doubleIdDetected = "";
onUserConnected = "";
onUserDisconnected = "";
onDifferentData = "";
regularCheck = "";
onUnsignedData = "kick (_this select 0)";
onhackedData = "kick (_this select 0)";

class Missions
{
	class Mission_1
	{
		template = "A4L_2.3.9.Altis";
		difficulty = "Custom";
	};

};

I've tried millions of things and none of them worked, i am pretty desperate at this point.

It seems to be happening more when more people are connected to the server, although we've had 6 hours uptime with 100+ players without it happening and we've had 10 minutes uptime with 40 players and a freeze.

full logs available here: download

Details

Severity
Major
Resolution
Open
Reproducibility
Random
Operating System
Windows 10 x64
Category
Dedicated Server

Event Timeline

FrancescoSkuya updated the task description. (Show Details)Nov 27 2020, 9:49 PM
FrancescoSkuya updated the task description. (Show Details)
FrancescoSkuya updated the task description. (Show Details)
FrancescoSkuya added a comment.EditedNov 29 2020, 10:21 PM

New lag spike happenned, same logs appear, and i see a pattern: The Type_284 packets go up and crazy, and this happens only when the lag spikes happens.
Got information frow Dwarden that those packets are setVariable with public flag.

screenshot

Normal logs with 80+ players connected with no lags spike happening for comparison:

screenshot

ToxaBes added a subscriber: ToxaBes.EditedDec 1 2020, 12:11 PM

@FrancescoSkuya please look at my ticket -> https://feedback.bistudio.com/T155335
Issue is similar but in my case it was Type_10 (public variables).

Soultion is check all strings in your mission for "setVariable [param, value, true]" (param will be some string and value could be anything).
Some value have complex type (object, array or array of arrays). You need to rewrite it to using simple types (string, number or boolean).

Example:

  1. if you use some object as value replace it with object network id with BIS_fnc_netId and then you could get object via this id with BIS_fnc_objectFromNetId.
  2. if you use some group as value replace it with BIS_fnc_netId / BIS_fnc_groupFromNetId functions.
  3. if you use arrays or arrays of arrays you need rewite logic for avoid it. I found what using simple array with 3 elements like coordinates [x,y,z] is ok but all bigger could lead to lag spikes.
dedmen closed this task as Resolved.Dec 16 2020, 12:16 PM
dedmen claimed this task.
dedmen added a subscriber: dedmen.

If you don't want to overwhelm your bandwidth and get your players to disconnect, don't overwhelm your bandwidth by spamming setVariable with public flag everywhere

© Bohemia Interactive a.s. Bohemia Interactive® is a registered trademark of Bohemia Interactive a.s. All rights reserved. · Privacy Policy · Terms and Conditions