Okay, I know the title is a bit weird, but please bear with me.
While crafting arrows, you combine 1 feather with 1 sharpened stick. That's ok, works as intended. But when you have only 2 feathers left (they have to be stacked to produce the problem) and try to craft an arrow, both the feathers are consumed.
When you have more than 2 feathers stacked, or if you have only 1, it works as intended: 1 feather per arrow. I guess it is a small bug in the coding but still a bug and an easy one to fix.