Thanks for the reply.

After you replied, I confirmed that as well. I then started eliminating parameters and I was finally able to get it to work. Turns out that the server executable behaves differently when started by FireDaemon. If I start the server with FireDaemon, the netlog doesn't get created or written to, but if I start it with a batch file or straight from cmd, it works just fine.

After that, I figured it had to be a permissions issue since FireDaemon starts services as LocalSystem by default. Once I added it to run it as the Administrator account, it logged the netlog just fine.

Are there any plans of allowing that to be moved in the future? I would much rather have it in the server folder somewhere. Especially since boxes with multiple servers are going to be hard to determine which is which..... Especially for ARMACon, that I'm working on. It's going to be impossible to determine which file to use since the files are distinguished by PID, which can't be determined by FTP or even HDD access for those running locally.....

Maybe you can do what the RPT file does and use the profile that is being used as to where it should put those files.

Either way, thanks for your help on this one.

The latest 1.52 update moved it's location on Windows to %AppData%, but it's not being written there either.

I'm about to open a tracker issue for it. They must have broken it when they updated it.

EDIT: Tracker: http://feedback.arma3.com/view.php?id=26155

90% of the time I am still seeing at least ONCE per day. Sometimes it's after the server has been up for 45 minutes, and sometimes it's 3 and a half hours.

I can find no common denominator. The only thing that is odd and consistent is that it never creates MDMP.

I can't get Windows to catch the user-mode dump still either.

This was the "expired" issue that was never resolved, assisted with, or debugged in any way.

http://feedback.arma3.com/view.php?id=23743

I don't consider that a valid debugging step and I will not be able to test that, and additionally, of course it's going to work.

I can even run the Altis Life file just fine with no one on it and it runs just fine and never crashes.

Thanks Xeno - my points exactly.

If you create an engine that is flexible and made for customization, then you need to handle exceptions correctly so that the modders knows what went wrong and how to fix it.

I am not disagreeing that the code written could be causing the issue, but how am I to determine what code is causing the issue when your engine isn't handling the exception correctly?

For the sake of proving absolutely nothing, I ran a server on the box (default map with no customization) since my last post, with a tool restarting it every 12 hours and it never crashed. So, as was already obvious it is something in the custom code that is causing the issue (but we already knew that), but that doesn't help us (content creators and server admins) figure out what is causing the issue because it's YOUR engine that isn't handling the exception correctly. What's more, your code is "handling" it enough to prevent windows from seeing the failure (hence no user-mode dumps and no event log entries) but not enough to do its job.

I would like to be able to figure out what code is causing the issue, but you, the creator of the engine, provide me no valid way of doing so.

I poked around in the hex editor in the bidmp files and can see the error
that it "thinks" is causing the issue which is:
"Out of memory (requested 4203 KB).

footprint 536870912 KB.
pages 81920 KB.

B, mapped 50040832 B), free 60440576 B", but as I said before, the server is on a box with 32 GB of RAM (https://paronity.com/i/6Z7g3.png) and its the only thing running. It's RAM consumption stays around the 110MB-169MB range. (https://paronity.com/i/1E2l4.png)

The only thing on the box is this particular server (and the MySQL instance for it) - and the server has 32 GB of RAM.

Most of the time, the ARMA server idles around 110-200 MB of RAM (with the 3rd party malloc) and 1.4GB-1.8GB (with the default malloc) depending on player count, object count, ect....

The system NEVER gets above 25% RAM usage (if we are doing testing with another server, or something else with the box at that point in time). Point being, we aren't even coming close to running out of RAM.

https://paronity.com/i/8i8I3.png

Altis is a vanilla map......

Your software isn't handling an exception the way it should. That is a fact. If it were, we would be getting MDMP files. It's irrelevant if I'm running a mod or not.

Just close this issue out. It's apparent that it's going no where. Thanks.

So there is nothing more that I can do to debug this issue?

Just extDB for database connectivity.

It's not windows. If it were windows, there would be events being thrown and a memory dump would be created (because I have user-mode dumps enabled for that executable).

Memory related crash is possible, but you, as the software that is running are responsible for managing those memory calls and handling them should they fail.

Is there anything I can do to help further debug the issue on your end? I'm an operating system level debugger for a living and have a lot of experience in this area, I just need to know what I can do with your software to do so. I am doing everything possible from the OS level, and am getting no where, which leads me to believe it's on ARMA side of things.

Bump.....

Still seeing the crashes on my end. MDMP files are not being created still, either....

Yup. Twice yesterday, and once today so far.

It has crashed several times, and the frequency seems to be more after the last push this week. It's never throwing an mdmp file.

We did finally get one today that had all three. It's attached and here as well: https://paronity.com/i/crash_9_default_malloc.zip (RPT, MDMP, and BIDMP)

Is there anything else we need to make some progress on this one? Is there anything to note about it only giving MDMP files like 5% of the time? Anything I can do to "influence" them to be caught correctly? I know that exceptions (especially memory), are a slippery slope, just trying to make progress. :)

Another one. No MDMP or user-dump on this one:

https://paronity.com/i/crash_6_default_malloc.zip

How long should I expect before I get some sort of movement on this? Should I keep uploading these files? Are they helpful at all? Any tips on what I can do to expedite the process would be great. Thanks!

Got a crash totay that actually seems to have been caught "correctly". There was an mdmp file in it this time and the RPT reports an access violation.

Here is the zip with RPT, BIDMP, and MDMP: http://paronity.com/i/dump_default_malloc_with_mdmp.zip

I have no idea what your MDMP file contains, so just in case it's helpful, the user-mode dump ALSO threw and I have a full dump should it be necessary. Also, just in case, here is the WinDBG anaylyze output for starters.I know details are scarce without symbols, but I figured I'd throw it in just in case. :)

• *
• Exception Analysis *
• *

• ERROR: Symbol file could not be found. Defaulted to export symbols for tbb4malloc_bi.dll -
• ERROR: Symbol file could not be found. Defaulted to export symbols for PhysX3Common_x86.dll -
• ERROR: Symbol file could not be found. Defaulted to export symbols for tier0_s.dll -
• ERROR: Symbol file could not be found. Defaulted to export symbols for steamclient.dll -
• WARNING: Unable to verify checksum for extdb.dll
• ERROR: Symbol file could not be found. Defaulted to export symbols for extdb.dll -

FAULTING_IP:
arma3server+db603a
00ee603a f3a4 rep movs byte ptr es:[edi],byte ptr [esi]

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00ee603a (arma3server+0x00db603a)

 ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000

NumberParameters: 2

Parameter[0]: 00000000
Parameter[1]: 00000000

Attempt to read from address 00000000

CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000003 edi=00000003
eip=7744d2ec esp=01bea95c ebp=01beaadc iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00200206
ntdll!NtWaitForMultipleObjects+0xc:
7744d2ec c21400 ret 14h

DEFAULT_BUCKET_ID: NULL_POINTER_READ

PROCESS_NAME: arma3server.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1: 00000000

EXCEPTION_PARAMETER2: 00000000

READ_ADDRESS: 00000000

FOLLOWUP_IP:
arma3server+db603a
00ee603a f3a4 rep movs byte ptr es:[edi],byte ptr [esi]

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

APP: arma3server.exe

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) x86fre

FAULTING_THREAD: 00000f70

PRIMARY_PROBLEM_CLASS: NULL_POINTER_READ

BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_READ

LAST_CONTROL_TRANSFER: from 00e96585 to 00ee603a

STACK_TEXT:

STACK_COMMAND: ~0s; .ecxr ; kb

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: arma3server+db603a

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: arma3server

IMAGE_NAME: arma3server.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 55253f19

FAILURE_BUCKET_ID: NULL_POINTER_READ_c0000005_arma3server.exe!Unknown

BUCKET_ID: APPLICATION_FAULT_NULL_POINTER_READ_arma3server+db603a

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:null_pointer_read_c0000005_arma3server.exe!unknown

FAILURE_ID_HASH: {785b27d8-09bd-7a57-4840-60685f9f0af0}

Followup: MachineOwner

If you need the full dump uploaded, just let me know.

Again,

https://paronity.com/i/crash_7_default_malloc.zip

No MDMP or User-mode.

mdmp files are never created.

I have yet to see an mdmp file produced by any of these crashes, either in the profile folder, nor have any OS dumps been created by my user-mode dumps being enabled.

Ok - turned on default (and inferior :P) allocator, here is the latest RPT and bidmp file. Again, no mdmp file is being created. Also, no user-mode dump.

https://paronity.com/i/crash5_default_malloc.zip

I am still seeing this issue on our servers as well.

EDIT: Turns out steamcmd didn't update the hotfix for some strange reason. I had to run it 3 times to get it to grab it then it works as before.

In summation, the latest hotfix works with the expected password.

The solution posted by uncle_fedor was also the solution for me as well. Apparently, they broke the updater?

You can get them from here, if you need them (i know its unlikely to trust files from here, but they are if you want to).

https://paronity.com/i/anims_f_kart.pbo
https://paronity.com/i/cargoposes_f_heli.ebo

See, I'm not getting any errors about any animation files. I'm getting a TON of these errors in my CLIENT rpt though

11:42:33 control[CA_ServerDetailExpansion]: Unexpected control type [0]

Server RPT is as clean as a whistle.

I see this issue periodically on my live server (which indicates it only becomes and issue when it's under load). Our test server can run for days without the issue (for the most part).

Another couple notes that I neglected to mention.

Every time it crashes, it seems to be somewhere in the malloc code that is causing an issue (based on looking at the dumps in WinDBG). This has led me to try all the different mallocs that are available and every single one has the same behavior after a random amount of time.

Additionally, I have User-Mode dumps on for the arma server process, so if a full memory dump would be helpful, just let me know, as I have the last 4 or 5 saved to my disk and I could get one or all of them to you.

Thanks!

I do still see this randomly. I have't set up a sand box to test the details of it grain by grain (I will work on doing that shortly), but could this have anything to do with the setObjectTextureGlobal bug? (http://feedback.arma3.com/view.php?id=18748)

I have been seeing this issue as well. It seems like it actually is becoming more prominent and annoying though.

This would make a ton of mods that are being worked on MUCH simpler. Would love to see this.

No official response on something so simple? Why has this not been addressed in some manner yet? Seems so simple...

I've been using preprocessFileLineNumbers as a workaround, but it just seems like this should be working.

Have seen this on the quad in a few separate occasions. This one, the guy on the front had a broken leg, and when he got on, remained in the broken leg position.

http://cl.ly/NNjq

I'm sure you would feel the same way if you bought something but didn't get all the features it promised.

I did buy the supporter pack, and I don't have my badge, but it's not a bug with the game....

Should probably post on the forums then. This feedback portal should be for game issues only.

This is a limitation of Windows and how the game people make the game. Most games will not do this properly, unless they do not bind to windows default. Most games bind to whatever windows default is during launch. If they do that, there is no way to change what the output is that is being used while the game is running.

It's because the servers are highly unoptimized at this time. They can't keep the server framerate up. They are degrading at a very rapid rate. http://feedback.arma3.com/view.php?id=129

I can confirm that this does not happen to everyone. I minimize and change screen mode a good bit and have not seen this particular issue yet.

Do you use more than one monitor?

2700K
GTX670

You could use Afterburner/Precision to watch your temps too. See how hot they are actually getting. Is it a Kepler chip (GTX600 series)? What card are you using?

I fail to see how your card overheating is the games problem? It's using your card, which makes it heat up. :)

This is probably the bug where lowering your weapon can cause you to walk until you vault: http://feedback.arma3.com/view.php?id=1016

While I agree with you about it being a limitation of the engine itself (makes sense), I don't agree that it would be too expensive to do. Shadow casting is not as heavy as many of the other rendering methods already at play in the engine.

The point of people reporting these things, is to see if anything can be done about it. This is one of those things that would be great if it could be addressed.

It would be nice if this was fixed. Was always frustrating to see a flashlight through an entire city....

We were seeing this behavior on fresh spawned vehicles. If you get in , move the vehicle, then you should be able to take the item(s).

I have seen this issue on a couple of the fences that are through out the map as well. No need to vault, you can just crouch right through.

That would be problematic if you were typing and rolled into a middle of a field because of it. Boom! :)

Instability on your card is most likely the issue, not the game. The game _does_ have some crashes happening, but they would not be directly linked to afterburner. I have my GPU OC'ed with it just fine, and I don't see any issues.

This happened in ARMA 2 as well. It happens from building and stuff as well as they continue to "emit" damage from them after the destruction animation has completed.

It makes a little sense that you see higher usage CAPTNCAPS since your hardware is slightly more dated than most in this thread.

This could be attributed to the server. The server FPS can make your max FPS go way down, which would thus make your hardware (which is capable of much more), not need to work that hard to hit it. The server has a lot of influence over the FPS that it's clients have.

Agreed. I hate the current grenade animation.

This happened in ARMA2 as well. Would be nice to see this get fixed though.

I have seen this more often than not. They seem to do it more frequently then they did in ARMA 2.

In many of the co-op missions that we have done, I am seeing similar frames than that of those I Was getting in ARMA 2 missions. I think they are about the same if not better already ans I suspect that they will continue to get enhanced.

I haven't seen this issue and I have done vehicle shooting a good bit already. They are probably going to need/want your DxDiag info.

I see this behavior as well, even on VERY simple missions. The server we are running it on typically can run two wasteland servers completely full (80 slots) without issue for up to 12 hours.

15-20 minutes into a 40 person game shows that the server FPS is down to ~1-5 (if lucky).

Ok. Made our server start catching dumps as well. Here is a dump that crashed and caused all 40 clients that were connected to crash with the same dump type:

http://cl.ly/NMse

EDIT by millhauz

I have seen this on several PC's. I have also seen it so that when the "server" crashes, it triggers all clients to crash as well. I have enabled local mode dumps. If I catch it again, I will upload one.

Full crash dump. Have 3 more if you need them.

http://cl.ly/NLf1

EDIT : Stop copypasting txt files to Notes. Millhauz