I had a pristine pistol suppressor (never fired out of it) that became ruined very quickly only after few zombie hits.
My pants became damaged because of the zombies, and the suppressor got ruined.

This makes it extremely punishing if I don't want to have suppressor attached to my weapons all the time and I think it's wrong.

I noticed this behaviour applies to all suppressors so long as they are in your inventory and not on the gun.
No other item gets this damaged this quickly if they are in your inventory.