I saw I was being able to dismantle my fence's frames and panels from the exterior side of the wall and not from the interior side of the wall. This is a game-breaking bug. I actually can destroy the walls and I did. They couldn't be destroyed from the interior, only from the exterior with any tools that can destroy walls.

EDIT: Just realized it is actually a feature, which I kind of not disagree nor agree with. It is a very easy way to destroy a base. Therefore, I must make my base somewhere deep inside a forest. Gotcha, sorry for any inconveniences.