Page MenuHomeFeedback Tracker

Noticeable Server FPS issues 1.10
Assigned, NormalPublic

Description

Server FPS seems to be extremely volatile now. Much lower than before. I can upload logs for evidence.

What I’ve done is disabled food decay and the worldwettemp in the globals, reverted weather settings on my init.c and cleared my events.bin and this somewhat as helped with server FPS, but is still volatile. Many modded servers are reporting this issue, along with vanilla servers.

1.09 I could boot up with 14k-16k frames, player joins it would drop to 6-7k

Now in 1.10 it seems the server can’t even boot starting up with 11k-12k but it’s volatile. Sometimes it dips extremely low 3k-6k and will jump back to 12k and eventually even back out to 15-16k without a player even present, when player joins FPS gets volatile, 4K-6k. Eventually leveling out to 6k with 1 player.

Now my live servers. After restart players swarm in to join in, and FPS will even dip to 2k-3k when players joining up, this did not happen at all in 1.09. 1.09 was smooth sailing always stable FPS.

Now really populated servers are suffering, saying that are experiencing massive server FPS issues. Some reporting 5 FPS with 100 players, where previously they were achieving 100-200 FPS with 100 players. There is a clear problem with 1.10. If we can have some insight to this, as is it even a problem? Or has the new food decay, bonestat, and weather really this draining on server performance?

Details

Severity
Major
Resolution
Open
Reproducibility
Always
Operating System
Windows 10 x64
Category
Performance
Steps To Reproduce

Compare 1.09 serverconsole logs to 1.10 logs

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
thad0ctor added a comment.EditedNov 22 2020, 5:38 PM

This is a huge problem on official servers right now too. If you join any relatively populated server there is incredible rubber banding, delays opening doors, delays accessing inventory, flying cars, teleporting zombies. It’s really bad. The game is unplayable.

Why are so many of these food decay and environmental calculations being handled so often and on the server side? I can’t imagine how much more overhead and network traffic this takes up.

OP, bump up the severity. I wouldn’t say this is minor. Last night was the first night in ages my crew got off to play something else. This is going to hurt the game.

Nelyo added a subscriber: Nelyo.Nov 22 2020, 6:17 PM

This should not be a minor issue. It's just unplayable.

sanguine00 added a subscriber: sanguine00.EditedNov 22 2020, 6:55 PM

DayOne is seeing a 60-70% decrease in server FPS as of the 1.10 patch. During normal operation it is playable, but there is no headroom for the frequent performance dips which result in the typical issues (doors and inventory lagging, players stuttering, awkward zombie combat, desync).

mdc added a comment.Nov 22 2020, 9:55 PM

As reported by the OP, a significant contributor to this issue is the Food Decay / Cooling system. This is *not*, however, the only factor. Disabling it will definitely prolong the time it takes for these issues to be felt, but we've had to switch from a 6 hour restart cycle to a 3 hour one - as well as disabling the new systems - in order to maintain a playable environment.

After approximately 3 hours, server FPS is down from ~300 to ~20, with a player count in the mid-30s. Still combing through the code for a change that would have this much impact, but the only major changes I've found so far (aside from the decay & cooling systems) are the addition of more network sync'd player variables for bleed states. As always, I can't rule out an engine-side cause, so searching may be pointless, but until this is acknowledged there isn't much else we can do.

FutureSixx changed Severity from Minor to Major.Nov 22 2020, 10:51 PM
FutureSixx added a comment.EditedNov 22 2020, 11:00 PM

Well after about week of 1.10
I'll average about 25-30 players and see a constant average 1.5k-2k FPS with that many players.
I've ran modded servers since 1.06, and I will say I'm use to being able to push 3k-3.5k with that many players. I would say I'm very experienced when it comes to running servers.
When I ran ApocalypticSixx ( 1.06-1.09 ), I did 70 slot servers, and ran 2 instances on each machine I had. I had 4 servers total, 2 machines.
I would average 500-1k FPS with 70 players on each instance. So 140 players on 1 machine.

Its just crazy to see 1.10 update, and me getting 1.5-2k with just 30 players!

PS I changed the ticket to "Major"

mdc added a comment.Nov 23 2020, 5:32 AM

Further update:

Dipped under 1 FPS during peak time tonight. Pushed a restart, and removed vehicles (approximately 30), and have not yet seen any significant degradation of performance. One suspected cause is now network broadcast of vehicle health states, possibly implemented engine-side to facilitate car repair in 1.10. Previously this was absent and was instead handled by a replication of TransferValues by vehicle repair mods. Currently sitting at ~800 FPS after 1 and a half hours, whereas previously would've been at approximately 200.

I’ve done a vehicle wipe 2 days after 1.10 release, had about 40 vehicles, and I saw no difference in server FPS. After combing through 1.10, learning everything it’s introduced, I feel this is just the side affect of its new features. hopefully in future updates they can optimize these new features, because this is a very negative side affect we are experiencing.

Spaggie added a subscriber: Spaggie.EditedNov 23 2020, 7:56 AM

We're seeing it as well ('vanilla' PC Servers)

And I'm hoping this has been put in somewhere from our console hosting friends:
https://twitter.com/bantamgreg1/status/1330434752378179585 thread

Can verify for a modded server. We should be running at 5k FPS, but running at 1k-2kFPS, with multiple reports of flying & bouncing cars.

Geez changed the task status from New to Assigned.Nov 23 2020, 9:41 AM
Beavis added a subscriber: Beavis.EditedNov 23 2020, 9:58 AM

I am getting it too, seems to be a general 1.10 problem. Server FPS fluctuating violently, and even if it is still over 2k on average you will see players rubberbanding, door and other interactions getting delayed and cars being horrible to drive with constant rubberbanding. Sometimes the car is jerked sideways, sometimes it changes direction up to 45 degrees without you even touching the keyboard. Results are of course disastrous if you are travelling at speed.

I did not observe this on Experimental.

hegysk added a subscriber: hegysk.Nov 23 2020, 10:01 AM

"our" medium populated official server was absolutlely unplayable yesterday we also quit DayZ for time being, I think it was bit better on friday and saturday - it was bad but i dont think as bad. Yesterday it was just 0.37 experience.
I played on some community servers, I have to say you guys running those are doing actually very good job, compared to official I felt like I joined csgo server.

mdc added a comment.EditedNov 23 2020, 10:26 AM

I’ve done a vehicle wipe 2 days after 1.10 release, had about 40 vehicles, and I saw no difference in server FPS. After combing through 1.10, learning everything it’s introduced, I feel this is just the side affect of its new features. hopefully in future updates they can optimize these new features, because this is a very negative side affect we are experiencing.

It's definitely a combination of factors, but as part of tests last night I just ripped out all of the 1.10 changes and ran with what essentially amounted to 1.09 \scripts\, and the issue *still* presented itself. It wouldn't entirely surprise me if we hit the same pop tonight and we *still* have issues, but so far it seems to be resolved with the vehicle wipe. Do you have *any* vehicles currently? If so, I'd recommend disabling spawns (ours are provided through RP, and thus spawns were already disabled) and wiping vehicles.bin again, and seeing if you still notice the same issues.

Ultimately the more data we can gather on this the better, because by this point I am 99.9999% certain that this is an engine issue, and not a script one; which means we're shit out of luck when it comes to fixing it.

Sski added a comment.EditedNov 23 2020, 1:55 PM

Is there any official update or acknowledgement of these server lag issues? This is a game breaking issue and you are not providing your customers with any information. Thx

Edit: in regards to PC. I see the console players update on Twitter

The ticket getting 'Assigned' means it's being passed in for investigation

roeh added a subscriber: roeh.Nov 23 2020, 2:43 PM

Just a general observation: on official servers (medium Pop) on Xbox the lag seems to be somewhat manageable if not non-existent for around an hour or so after server reset.

Bohemia, perhaps you can accelerate server reset times to every 60-120 minutes while this issue gets resolved?

Thank you.

Can also confirm, less fps this patch.

Hello everyone, this chat comment was seen in a stream

15:00 DayZ: Note: We are working on Hotfixes to bring back same performance as 1.09

Thanks

Empty server, its unstable

Empty server, its unstable

puts a delay of 10 to 15sec

return us with feedback

Piling on here with a snippet from my server ... the game is virtually unplayable after the update. Players rubber banding constantly, vehicles flying off the map, etc.

6:37:01.332 Average server FPS: 1733.00 (measured interval: 1 s)
6:37:01.332 Used memory: 2911069 KB
6:37:01.332 Players: 1 in total
6:37:02.332 Average server FPS: 1591.00 (measured interval: 1 s)
6:37:02.332 Used memory: 2911008 KB
6:37:02.332 Players: 1 in total
6:37:03.332 Average server FPS: 1724.00 (measured interval: 1 s)
6:37:03.332 Used memory: 2911336 KB
6:37:03.332 Players: 1 in total
6:37:04.332 Average server FPS: 1827.00 (measured interval: 1 s)
6:37:04.332 Used memory: 2911388 KB
6:37:04.332 Players: 1 in total
6:37:05.334 Average server FPS: 1641.36 (measured interval: 1 s)
6:37:05.334 Used memory: 2911389 KB
6:37:05.334 Players: 1 in total
6:37:06.333 Average server FPS: 1609.00 (measured interval: 1 s)
6:37:06.333 Used memory: 2911110 KB
6:37:06.333 Players: 1 in total
6:37:07.333 Average server FPS: 1684.00 (measured interval: 1 s)
6:37:07.333 Used memory: 2911114 KB
6:37:07.333 Players: 1 in total
6:37:08.336 Average server FPS: 1757.73 (measured interval: 1 s)
6:37:08.336 Used memory: 2911114 KB
6:37:08.336 Players: 1 in total
6:37:09.336 Average server FPS: 1799.00 (measured interval: 1 s)
6:37:09.336 Used memory: 2910801 KB
6:37:09.336 Players: 1 in total
6:37:10.336 Average server FPS: 1663.00 (measured interval: 1 s)
6:37:10.336 Used memory: 2910805 KB
6:37:10.336 Players: 1 in total
6:37:11.336 Average server FPS: 1879.00 (measured interval: 1 s)
6:37:11.336 Used memory: 2910805 KB
6:37:11.336 Players: 1 in total
6:37:12.336 Average server FPS: 1844.00 (measured interval: 1 s)
6:37:12.336 Used memory: 2910810 KB
6:37:12.336 Players: 1 in total

Why it has normal priority, i think it should be urgent, shouldn't it?

Geez added a subscriber: FixServerRubberbanding.
Geez added a subscriber: Geez.

Hello everyone and thank you for all your feedback and the information provided.
We have indeed found several issues which could cause the problems with performance as you describe and they are being looked into internally as our top priority and we will attempt to resolve this as soon as possible.
Regards,
Geez

Thanks for the update

Happy Hunting

mdc added a comment.Nov 25 2020, 8:51 PM

Hello everyone and thank you for all your feedback and the information provided.
We have indeed found several issues which could cause the problems with performance as you describe and they are being looked into internally as our top priority and we will attempt to resolve this as soon as possible.
Regards,
Geez

Thanks for the update. Can you confirm, for my own sanity, that at least one of the issues is indeed with network broadcast of variables associated with vehicles?

I think I am done playing this game until they are able sort out these issues. I hate traveling on foot and no matter how slow or cautious I drive, the car always ends up spiraling out of control. I've crashed 3 times now and 2 of those times I was killed and lost a lot of gear and items. If I had known they were going to release a broken update that makes some major features of the game completely unusable I would have never spent $50 on this game or recommended it to 3 other friends.

@Spaggie sound just like a simp.

Spaggie added a comment.EditedNov 29 2020, 7:05 AM

Everyone else on this ticket kept it as factual and emotionless as possible while working to provide collaborative confirmation

And you're still here whining and being childish

Looking forward to the update :)

miko added a comment.Nov 29 2020, 11:07 AM

Everyone else on this ticket kept it as factual and emotionless as possible while working to provide collaborative confirmation

And you're still here whining and being childish

Is he? releasing a broken patch means it was not properly tested. this shows lack of respect to the dayz community . already 10 days and there are no fixes.
It is not a free game.Patches remove problems ,they dont add ones. Its understandable to have few bugs when you join a game like DayZ.but adding NEW ones that is another thing mate.
So do not call the people that are complaining childish. It is a survival game that when you die you go back to zero. Hours of gaming disappear from one bug that was not there before.

And thus the ticket goes off the rails and descends into the normal quagmire usually apparent on Twitter

Any updates for us? Is there any reasonable ETA? Can you share whats root cause, maybe atleast community server owners could react by adjusting config if possible.
We keep playing although it is how it is now - we observed that sometimes official servers run pretty well, sometimes its not really playable anymore, regardless of time or player count, thats weird. Also the performance appearently fluctuates sometimes server runs smoothly for a minute or two then IT goes downhill again for some time.

Spaggie added a comment.EditedNov 29 2020, 1:52 PM

Hi

They ran 2 playtests last week, which feels like they're testing around the issue, including turning off wet clothes (which is strange), the patch notes for that are here:

https://forums.dayz.com/topic/249348-experimental-update-110-changelog/?page=20&tab=comments#comment-2464443

They asked for people to come help stress test the server (this FPS seems amplified by higher populations

For example: https://twitter.com/DayZ/status/1332284855997050881

And some fold did what they could to help (I think I played maybe 3 hours across the two playtests and tried to enthuse some folk: https://twitter.com/Spaggie/status/1332007677401116674 across the two days

But it's tough to get the numbers

I think these 2 lines are interesting :

CHANGED

Improved server performance by optimizing the variable synchronization
Improved server performance by optimizing the OnCEUpdate call on each item

I would say that the playtests felt smooth as a player but the server never really populated up so I don't know if they got the insights they needed

The hotfix to solve the huge performance drain caused by 1.10 is to be expected next week. This was told on twitter @hegysk, @miko.
They are working very hard on solving it, complaining is not going to help.

Spaggie thank's for information provided, it helps a lot just to know things are moving.

I played on exp playtest too and it felt good. I would feel better if it would be properly tested with full server running atleast for a day but yeah, it's something. Official servers are extremely laggy even with 15-20 ppl though. I imagine it can be really hard to simulate organic live server with stashes and bases if items ingame have big impact on this issue but I hope for the best.

Geez added a comment.Nov 29 2020, 11:56 PM

Hello everyone.
There have been some fixes applied on the current experimental version already.
Can you please let us know if there are any major issues while playing the experimental version?
Regards,
Geez

Random disconnecting from official servers is related to this issue?

hegysk added a comment.Dec 1 2020, 8:33 AM

I think it is. The disconnects are hand in hand with server performance. The worse it is (rubberbanding, delays, desync) the more frequent disconnects (kicks) are.
Yesterday I scored record with ~3 kicks within 1 minute of playtime, Idk what was going on yesterday on official but it was really bad, desync was so strong I couldn't sprint with Epi pen because my stamina drained before server registered and refreshed my stamina (epi). Hell I am way more hyped for this hotfix than entire major patch. I really hope BI can find a way how to properly stress test their patches before release, it feels like they are limited to few volunteers on exp branch and that's it - not enough. I am all about to help testing such things but really I didn't see any requests for help from BI on socials and if there is a place like dev forum, I don't dig this deep.

Looks like we're going to stable guys

Good Luck everyone <3

https://twitter.com/DayZ/status/1333717078243733505

FutureSixx added a comment.EditedDec 1 2020, 7:06 PM

Btw. This hotfix did nothing.
I'd go as far as saying, it made it worse.

The problem was not solved!

Problem was not solved with update , pls fix this

hegysk added a comment.Dec 1 2020, 9:22 PM

For me servers are now absolutely brilliant. 0 kicks over 4 hours, no noticable delay/desync. Good work!

I think we're feeling better across ours, but we've had some ... attention ... this evening that's clouded opinion

So far so good on my server. I can fly helicopters without significant hitching/rubber banding and vehicles appear to be staying planted on the ground. I do have a low pop server, so YMMV.

Nelyo added a comment.EditedDec 2 2020, 11:49 PM

I share the predominant opinion here. We still have a crucial server performance problem. My indicator is the following: Bait a zombie to move him close to a wall. Afterwards, from all angles the zombie stuck. Either a bug in the pathfinding (or collision detection) overloads the physic thread or the physic thread is overloaded by something else and prevents this calculation. I play on Spaggie's Vanilla community server

Hello @Geez

I wonder if you'd add @Nelyo above to:

T145768: Zombie Pathing

I still have lags (not often) and disconnecting without reason (every 1 or 2 hours) on server DayZ EU - DE 1297, while DE 1390 was ok after hotfix.

@Vodovodnik
I have the same issue on official NL server.

This comment was removed by tratata123.

I find there to be generally less lag and rubber banding but this hot fix seems to have created more desync glitches, especially with knives, after interacting with dead bodies and after getting cut with barbed wire

Lags on full populated official EU servers are still here.

hegysk added a comment.Dec 7 2020, 9:44 AM

After a week of playing i can say it is so much better indeed. Most of the time you have no problems but when it comes to pvp combat ocassionaly, you are doomed, unless target is standing still you are not going to hit it by intent, you are going to hit by lucky roll or not going to hit at all. I would love to see some further optimizaton.

Lex added a subscriber: Lex.Dec 7 2020, 3:34 PM
hegysk added a comment.Sun, Jan 3, 7:01 PM

First of all - HNY to all of you guys.

Just wanted to know if there is something new up with this ticket. I absolutely feel ppl hopped on DayZ train and I love it. What I really don't love is - it pretty much killed playing experience on Official and I am afraid ppl will again pass on the game due to this.

More factual feedback - official servers are choking hard when 55+ players, around 50 is playable-ish but above that it really shuts down most gameplay possibilities. Zedz are mostly stuck, players are rubberbanding and getting kicked. Every client-server interaction is delayed by second(s). Basically the same issue as before hotfix but the treshold moved a tad up (with player count). I love full official servers but I am afraid it will just fall off in a week or two in this state and you can't blame anyone.

I also believe this fix helped a lot community driven servers where I believe they got much more hardware performance headroom but for official it's still very bad in prime. Also, we might be a minority playing on officials, however it's most likely newcommers will start (maybe also end right away) with official and first impression should be very importnant.

Hope you guys had a nice time off and good luck with optimization of DayZ :)

PS: any word back from you guys is welcome it really helps keep our hope up :)

mamulated added a comment.EditedSun, Jan 3, 7:55 PM

First of all - HNY to all of you guys.

Just wanted to know if there is something new up with this ticket. I absolutely feel ppl hopped on DayZ train and I love it. What I really don't love is - it pretty much killed playing experience on Official and I am afraid ppl will again pass on the game due to this.

More factual feedback - official servers are choking hard when 55+ players, around 50 is playable-ish but above that it really shuts down most gameplay possibilities. Zedz are mostly stuck, players are rubberbanding and getting kicked. Every client-server interaction is delayed by second(s). Basically the same issue as before hotfix but the treshold moved a tad up (with player count). I love full official servers but I am afraid it will just fall off in a week or two in this state and you can't blame anyone.

I also believe this fix helped a lot community driven servers where I believe they got much more hardware performance headroom but for official it's still very bad in prime. Also, we might be a minority playing on officials, however it's most likely newcommers will start (maybe also end right away) with official and first impression should be very importnant.

Hope you guys had a nice time off and good luck with optimization of DayZ :)

PS: any word back from you guys is welcome it really helps keep our hope up :)

I can confirm what hegysk mentions. Above a certain threshold of players the game is still unplayable, with continuous rubber-banding, zoom delay and doors opening after 1-2 seconds. The other day the lag was so bad that when I entered a vehicle without a back wheel, I was immediately launched into space and died. I mostly play on NL-DE servers, don't know if that matters.

hegysk added a comment.EditedTue, Jan 12, 10:39 AM

No answer status update for last 10 days, you could atleast let us know what's going on, what's planned, if cause is identified and some ETAs on hotfix. Official servers are in unplayable state as a multiplayer game. I measured official server delay and if I am correct, server is running around 0.8 ticrate (fps). Action delay is ~1.5s in high pop. As much as we'd love to play officials it's simply not bearable anymore.

Is there any other ticket opened for this issue or it's just this one and ppl gave up again already. We'd love to come back to official as soon as some progress is made or atleast some light at the end of this tunnel.

Geez added a comment.Wed, Jan 13, 4:18 PM

No answer status update for last 10 days, you could atleast let us know what's going on, what's planned, if cause is identified and some ETAs on hotfix. Official servers are in unplayable state as a multiplayer game. I measured official server delay and if I am correct, server is running around 0.8 ticrate (fps). Action delay is ~1.5s in high pop. As much as we'd love to play officials it's simply not bearable anymore.

Is there any other ticket opened for this issue or it's just this one and ppl gave up again already. We'd love to come back to official as soon as some progress is made or atleast some light at the end of this tunnel.

The issue is indeed being worked on. We are monitoring the performance internally and also on the current stable to improve the performance as much as we can.
Regards,
Geez

Thanks, Geez, for updating us.