If you try to combine M4A1 ammo, 5.56mm, it is capped to 30 shells in your inventory. However if you load them into a 40 shell magazine, then empty out the magazine, it is combined into 40 shells in your inventory.
It appears that the max cap set on combining the shells in your inventory, directly, is set at 30 and should be 40. That or when you unload the 40 shell clip that it should separate into 30 shells and 10 shells respectively.