So after some more Googling, I found out about the audio sources bug. I dropped my audio to 16bit, 44100Hz, and played for over three hours with absolutely no issues. Could be the same with the others?
EDIT: So on the latest RC branch, I bumped the audio up as high as it could go and had no issues during my play time. Can't say if this also solved the issue on Altis Life, but in SP, for me, it's fixed. For now.