What about my backyard? What about the roof of my house? My neighbors can't see all of that area due to one reason or another (fences, trees, lack of windows on their house, etc. etc.) As many others have already noted, just because you can, doesn't mean you should.
Well, this case en.wikipedia.org/wiki/Florida_v._Riley pertains to your question. Some of the dissenting opinions raise the same questions you do, but there's a reason I had to call them 'dissenting opinions'.
If indeed these drones are using Dirtboxes and are indiscriminately tracking people based on cell phone signatures, then I would posit that the combination of [1] and [2], and ongoing issues like [3], make what law enforcement is doing Baltimore disgustingly illegal and oppressive. Targeted and specific aerial imagery and inspection is a totally different ballpark than what these drones are doing.
