Page MenuHomeFeedback Tracker

Sound caused FPS drop.
Closed, ResolvedPublic

Description

This is marked as minor just because it only happens after a few hours. But when it happens fps drop to 5-10 and never recover.. So when it happens it's a major issue.

It's not easilly reproducible and profiling build report doesn't hold much more info than sound->ssAdv being very heavy. I've identified the function that has the highest execution time while that happens. See Additional info.

FPS immediatly recovers when you tab out. Because sounds stops being processed then.. So I think this is also known as the Alt-Tab bug.

This bug has only recently appeared. I can't remember it happen <1.66.

I'll try to get a full profiler capture from VS or vTune next time this happens.

Details

Severity
Minor
Resolution
Open
Reproducibility
Random
Operating System
Windows 8 x64
Category
Sound
Steps To Reproduce

None.. Happens randomly after some hours of play

Additional Information

Dedmen - 04/29/2017
Sound bug... prof build wSimu->sound->ssAdv taking >90ms per frame. No looping sounds or anything abnormal to hear. Getting worse with time. Cannot reproduce reliably. Happens randomly and rarely
Tabbing out brings fps instantly back up to 50 because sounds stop playing
The function causing that is profv13 arma3.exe+0x00571600 (WaveXA2::IsTerminated)
It is calling a sndLH function but sndLH doesn't actually appear in the profiling capture.
According to the purpose of the function... It seems to think that sound isn't terminated for some reason.
Call comes from ssAdv so it's either 2D or 3D env sounds.

Event Timeline

dedmen created this task.May 9 2017, 1:13 PM
dedmen updated the task description. (Show Details)
dedmen edited Additional Information. (Show Details)May 9 2017, 6:17 PM
dedmen edited Additional Information. (Show Details)May 9 2017, 6:43 PM
dedmen updated the task description. (Show Details)
dedmen updated the task description. (Show Details)May 9 2017, 6:46 PM
Dwarden added a subscriber: Dwarden.May 9 2017, 7:03 PM
dedmen edited Additional Information. (Show Details)May 9 2017, 7:39 PM
dedmen added a comment.EditedMay 9 2017, 7:54 PM

Okey I just reversed the SoundScene class and have access to the sound tables. It's very likely some special sound causing that.. Now I'll just have to wait till I get that sound bug again then I can provide more data.

Note to myself.
Breakpoint
AbstractWave Constructor
CC {(char*)((*((int*)eax))+8)}
WaveXA2::IsTerminated
IT {(char*)((*((int*)(ecx+12)))+8)}

repro / more data is indeed needed
also check RPT logs if no samples are missing or similar errors ...

dedmen added a comment.EditedMay 13 2017, 1:04 AM

I have it right now. Profiler is running. RPT till now is attached.


This is interesting:

1:02:11 Sound: Error: File: .ogg not found !!!
1:02:35 Sound: Error: File: .ogg not found !!!

Bug started appearing at about 0:50
Here is profiler log from VS2017 running Profiling v13 build of Arma

Here is the log of files calling WaveXA2::IsTerminated. At the beginning I'm tabbed out and at 40 fps. I annotated when I tabbed back in game. FPS plummited to 5 and you can clearly see that something is wrong in the log.

diag_captureFrame output:

I just teleported myself a few dozen kilometers outside of the map. I'm in the middle of nowhere but I can still hear a helicopter sound ingame. Though that might be unrelated to this.. I lost Internet connection and reconnected and since then I had that helicopter sound... That's that already known sound bug thingy I'm sure.
Tabbing out 70fps. Tabbing in 8 fps.
Here is another IsTerminated log. You can clearly see when I tab back in.


The problematic sounds seem to come from Dubbing_Radio_F_EXP.
I don't know why they are even playing.
I am running https://github.com/acemod/ACE3/tree/master/addons/noradio
Which should mute every Radio message.
As you can see here: http://image.prntscr.com/image/960db38d1a1e4c1996b48b5463301664.png
There are about 5000 sounds in that array. That number doesn't increase when I go ingame.
I just set the itemcount in that array (Array::_n) to 0 and my fps ingame shot up from 6 to 70 again. And it's fixed now.
And now I dropped into 3fps bug.

Again I'm running 1.68.141.262 profv13.

Still no real repro for this. Happens only after a few hours of play. But when it happens it worsens quickly.. First you only drop down to 20 fps.. And 10 minutes later you are at 8.

question, still issue with 1.70 ?

Couldn't try to reproduce yet. As it only appears after a couple hours of turbulent Play..

dedmen added a comment.EditedMay 25 2017, 10:57 PM

Yes. I'm having it right now on profv1.
1.70.141.792

I couldn't attach any Profiler. I tried VS2015, VS2017 and vTune. They all failed to attach.
So I can't provide a Profiler report but I'm quite sure it will be the same as last time.

Here again the log of calls to WaveXA2::IsTerminated

You can again clearly see when I'm tabbing back in. This time It's RHS Radio messages. As our Enemies in this mission are RHS units. Before it was CSAT Radio messages as your enemies were CSAT.

So it seems to generally be related to Radio messages.
Btw my Radio volume is set to 0 and these messages in there are from Enemy units. And so far I wasn't ever in hearable range of enemies. So I couldn't even hear these sounds.

And the time where the FPS started to noticably lower we were already back in FOB with only Human Players in a 2km radius. Our players are Nato Units not Russian RHS. So they also shouldn't be emitting these Radio messages.

Just happened again on 1.74.142.560 Don't have any logs.

Appeared in https://github.com/Wyqer/kp_liberation/issues/243
Might very well be in some way related to the Liberation mission. It mosly happened to me on there.
Considering the sound bug is a huge flood of unit radio sounds... It might be connected to some sort of AI caching or spawning/despawning.

Liberation is not the most stable mission i know in terms of memmory leaks on Client and server side

nigel added a subscriber: nigel.EditedFeb 13 2018, 7:41 AM

https://streamable.com/ij7t9
https://streamable.com/urade

1 instance of this happening in recent days.

1.80.143869

dedmen added a comment.EditedFeb 14 2018, 4:14 PM

We need some diag command to dump the current content of the sound scene's _globalSounds and get the reason why they are not being terminated.

WaveXA2::IsTerminated is running repeatedly but for whatever reason the sound that should be terminated isn't terminated.

This problem builds up very slowly like a memory leak. But as you can see in nigel's comment it's really not insignificant at all.
This is probably one of the major causes of long term fps reduction. Getting this fixed could be a big thing.

Just happened again on 1.80.144071 while playing Liberation mission. It seems like playing that mission makes it most easily reproducible for me.

Copy pasta from Discord:

Okey I found two potential causes. Second one seems most likely to me. Every soldier can be saying one thing at a time. Which makes sense.
In the soldiers simulation cycle it checks if the unit is dead and if he is currently talking. And if so deinitialize the sound and delete it.
If the soldier now dies on the server. And the simulation stops running on the client. The client's soldier simulation will never see that the unit is dead and the sounds needs to be freed. Thus the sound stays alive. And theoretically that's a Memory Leak. Not per definition as there are references to that memory. But there are no references capable of deleting that sound.
It should be checked if the unit is currently talking in his killedEH or destructor. And if so the sound should be cleared. That should fix this.

Cre8or added a subscriber: Cre8or.Jun 18 2018, 8:49 AM
Wulf added a comment.Jul 12 2018, 5:11 PM

Fixed in the current Dev branch of the game. High probability that it will be in the next update.

Confirmed fixed. Please mark this ticket as fixed.

Groove_C added a subscriber: Groove_C.EditedDec 25 2018, 8:09 AM

No mods used.
No script errors.
With database.
With 2 HCs.

Dead units, destroyed vehicles and items left on the map are cleaned regularly.

We restart our server daily at 5 am.
In the morning and during the day the server can be full (32 players) and server FPS remains @ 30-40+ without problems.
Server FPS drops progressively.
Before 11-12 pm server FPS is already under 20.
After 11-12 pm server FPS drops under 10, or exactly 3-5-7 and stays forever, until restart.

Checked yesterday - still the same situation.

I know several other admins of other servers that have this same problem, but much sooner than we do and the only way to "repair" this is to restart servers more than once a day, like every 4-5 hours for Domination servers with no mods used, also with database, but without HCs and everything also cleaned regularly.

https://forums.bohemia.net/forums/topic/160288-arma-3-stable-server-184-performance-binary-feedback/?page=92&tab=comments#comment-3319073

This was the snapshot from our Evolution mission.
https://gist.github.com/winseros/edecd8d5f2cbde7b28c4db4fbbf945b9

@Groove_C DIfferent issue. This exact bug is confirmed fixed.

Wulf closed this task as Resolved.Jan 2 2019, 2:06 PM
Wulf claimed this task.

Thank you for letting know dedmen. Closing.

dedmen set Ref Ticket to AIII-49250.Aug 30 2020, 7:16 AM