When I have my NVGs on and then swap equipment (clothes, helmet, etc), it turns them off, although when looking at my player model they still appear to be activated.
To solve you have to turn them off then on again, which isn't that big of an issue, but it gets annoying fast.