I have the same problem, damned garden plot in the way of my base foundation. I’ve posted on all the forums and haven’t gotten any help this feedback site is useless...

I wish I can upvote this maybe then we’d get some help

It would be great to have the option to flatten the garden plot away again with the shovel - just like we can bury corpses, infected or fireplaces.
That would also get rid of garden plots being used to grief bases so others can't build anymore.

I read numerous reddit/forum posts with questions about this I really believe this is an easy to fix issue with huge impact on basebuilding. This alone can keep your base absolutely exposed for example, we built an enterance to our base, afterwards we put a garden plot behind it (clipping fences a bit). Later on, we found some metal sheets so we dismantled wooden walls to replace and you can't build anymore. And you really don't have any other option than wait who knows how long (please, if someone competent is seeing this comment - let me know despawn mechanics of plots if it's affected by flagpole and if it's affected by players moving over it - thanks!).