In other countries this is often dealth with through zoning. You can buy land but you can't do whatever you want with it. I assume you have that in the US too, but maybe not as restrictive?
An additional issue is that there are lots of different administrative units making the rules, so a particular set of restrictive rules won't apply 1 mile away.
