I think pressing the inventory button a second time when it is already open should close it again, quite simply. Right now it is pretty difficult to close intuitively unless you reach for the escape button.
Right now pressing the inventory button several times when it is already open apparently does something because "All" is highlighted, then on multiple presses it is un-highlighted and then nothing happens.