This is the problem with ADM GPU's. Sometimes they work great and other times they do not, and Nvidia simply works. Good luck getting this resolved since the official stance from AMD is that "This bug has been fixed with driver version 22.5.1" but the AMD forums (and me included) are of another opinion. Use the official bug report tool in Adrenaline and pray that AMD gives a crap, I'm preparing my report as I'm writing this. I have the exact same problem (on my 6700xt) as you have and there are many more AMD users in the same situation. If you want to read more stories like yours then google 'dayz and amd'.

Same struggles.
RX 6700 XT
i7-8700k
32gb ram
game installed on m.2 ssd
1000w psu