You're ignoring the fact that many people's perception of what is "a safe speed" is not at all reliable. (Compare: what percentage of people would claim to be better-than-average drivers? https://www.smithlawco.com/blog/2017/december/do-most-driver...)
We have speed limits because trusting people to "go a safe speed" doesn't work, in general.
I don’t think anybody is arguing speed limits should be abolished. At least not in any general sense. But, it’s plain enough to see that speed limits on some roads are wrong.
I-495 in VA is a prime example. It’s posted at 55mph and traffic regularly flows faster (or slower, during rush hour). It really needs variable limits based on traffic volume instead of a dumb 55.
We have speed limits because trusting people to "go a safe speed" doesn't work, in general.