On the other hand, often the fence made sense before the building next to it got demolished decades ago and nobody remembers that there even was a building.

Ideally you'd want to discover that the building was there, then confidently tear down that fence, but having a fence that nobody dares to touch because the building has been forgotten and we can't find why the fence was even there isn't great either.