Issue started occurring when relocating a server to a new machine and keeping the configs from the old one. RSYNC was used. I suspect keypair mismatch.

After discussing this in the dayz discord, we are pretty sure that this bug and
https://feedback.bistudio.com/T182075
and
https://feedback.bistudio.com/T93061

are the same issue.
Whenever one of the issues mentioned in the bugs above happens to one of my players, I have the rsa errors in the log:

Feb 28 19:16:23 game002 hashima-server[984309]: Error decrypting message: error:0406506C:rsa routines:rsa_ossl_private_decrypt:data greater than mod len
Feb 28 19:16:23 game002 hashima-server[984309]: Error decrypting message: error:0406506C:rsa routines:rsa_ossl_private_decrypt:data greater than mod len
Feb 28 19:16:23 game002 hashima-server[984309]: Error decrypting message: error:0406506C:rsa routines:rsa_ossl_private_decrypt:data greater than mod len
Feb 28 19:16:34 game002 hashima-server[984309]: Error decrypting message: error:04099079:rsa routines:RSA_padding_check_PKCS1_OAEP_mgf1:oaep decoding error
Feb 28 19:16:34 game002 hashima-server[984309]: Error decrypting message: error:04099079:rsa routines:RSA_padding_check_PKCS1_OAEP_mgf1:oaep decoding error

It affects AMD and Intel CPUs of different generations, so I doubt its a CPU crypto instruction issue.

What I've seen while poking into the dayzserver executable was, that its parts (guess the statically linked libraries) were compiled with a mix of gcc from ubuntu 16 and 18. Both releases are not even supported anymore!

Looking at what is used from the system:

➜  _data ldd DayZServer
	linux-vdso.so.1 (0x00007ffe66dc5000)
	libsteam_api.so => /home/dayz-hashima/.local/share/containers/storage/volumes/hashima_serverfiles/_data/./libsteam_api.so (0x00007fccd789c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fccd788e000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fccd7889000)
	libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fccd787d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fccd7878000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fccd7600000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fccd7521000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fccd7856000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fccd7340000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fccd78ec000)
➜  _data ldd /home/dayz-hashima/.local/share/containers/storage/volumes/hashima_serverfiles/_data/./libsteam_api.so
	linux-vdso.so.1 (0x00007ffe6bfb0000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f206aa10000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f206aa6e000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f206aa0b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f206a82a000)
➜  _data ldd battleye/beserver_x64.so
	linux-vdso.so.1 (0x00007ffda2ba8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa0ac81f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa0acc8a000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa0acc85000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa0acc9a000)
➜  _data

we can see the obvious libraries - but if its the case that the binary was built using Ubuntu 18, there might be slight differences in the libc and libm behaviour (maybe even other libs) and the crypto support from the kernel.

I'm trying to get a tcpdump to see if there is something awkward in the traffic.
If there is anything you need to know and I can help with, let me know.

In T175939#2576945, @rarnold wrote:

Issue started occurring when relocating a server to a new machine and keeping the configs from the old one. RSYNC was used. I suspect keypair mismatch.

@rarnold - which OS did you use before?

I no longer have the logs unfortunately. But if I remember correctly it was a migration from debian 11 to fedora server 39. I know this is not useful for you without logs.

I tried Debian 11 in a podman container, that didn't make a difference.

Might be kernel related though.

I've also tried OPENSSL_ia32cap=~0x200000200000000 to really make sure its not a mix of hardware / too old openssl - but the issue stays the same.

Mar 01 17:30:59 game002 hashima-server[1326254]: Error decrypting message: error:0406506C:rsa routines:rsa_ossl_private_decrypt:data greater than mod len
Mar 01 17:30:59 game002 hashima-server[1326254]: Error decrypting message: error:04099079:rsa routines:RSA_padding_check_PKCS1_OAEP_mgf1:oaep decoding error

A mix of these two errors always shows up when somebody tries to login and gets stuck as described in the two mentioned bugs. Its happening rather often, so a soonish fix would be appreciated.

I also have a traffic dump I can share privately.

@Geez @dedmen it might make sense to merge https://feedback.bistudio.com/T182075 and https://feedback.bistudio.com/T93061 into this bug, or at least link them - they are definitely related.

So I can actually reproduce this:

• go to an edge or a corner that is in the air / above water.
• logout
• login again

So far it seems to affect buildings or other "hard" structures, logging out on a powerline is just fine.
I'll attach a map and photo from hashima, and then I'll try to reproduce it on an official map.

In game photo will follow

Also I came up (actually with the help of chatgpt) with a script (its a hack, yes) that replaces the player's location in players.db. After changing the location the login works as expected.

➜  storage_1 update_player_location players.db FOOBAR-TGf360-GRI2hlG3Bs1M0vxstrr5ZC45OVsc0= 1308 73 1552
Found position at offset 2: (2479.455810546875, 10.036487579345703, 2746.058349609375)
Updated player FOOBAR-TGf360-GRI2hlG3Bs1M0vxstrr5ZC45OVsc0= to new position: (1308.0, 73.0, 1552.0)
➜  storage_1

The script is attached for those with the same problem.

The RSA messages are known, but they are just noise and don't actually cause any issues.
I tried to fix them but couldn't figure it out and couldn't track down why they happen, its something related to bisign validation, I know our input data into openssl is correct (or rather, the same as on windows) but it just doesn't like it.

The issue described by the comments here are unrelated to this ticket.

I can validate the issues are unrelated.

I managed to stop the RSA messages, unfortunately I can't quite remember how as it was a long time ago.
I do recall reading somewhere about definition variances between windows and linux with openssl, I'm running gwenhywfar-tools in my dayz server container and I'm fairly sure that's what fixed the RSA messages but don't quote me...

The other bugs (Players unable to interact, "zombie" players on server that aren't connected etc) do still happen to me and as mentioned I don't see any RSA errors in the logs.

Sorry I can't give more detail, I didn't record my container changes properly back then :/
But hope this helps somehow!