I have had around 2 occasions of this happening. I will be in Xbox Live Party chat prior to joining a server. Once joined the server, I came across a player who was talking to me via the game chat. I switched to game chat through the party settings and tried talking back using down on the d-pad. He couldn't hear me and continued asking if I had a mic. I assumed the fault was because I was still technically in a party, so I left the Xbox Live Party and tried talking in game again, but still it did not work.

I have previously talked in-game on a few occasions before, and they all worked, but I was never in a party before. I feel like because I joined a server whilst in an Xbox Live Party, this prevented me from using in game voice chat even if I leave the party in the same server.