Both were always going to be kind of inevitable as soon as the technology would get there. Rather than debating how to stop this (which is mostly futile and requires all of us to be nice, which we just aren't), the more urgent debate is how to adapt to this being the reality.
Related to this is the notion of ubiquitous surveillance. Where basically anywhere you go, there is going to be active surveillance literally everywhere and AIs filtering and digging through that constantly. That's already the case in a lot of our public spaces in densely populated areas. But imagine that just being everywhere and virtually inescapable (barring Faraday cages, tin foil hats, etc.).
The most feasible way to limit the downsides of that kind of surveillance is a combination of legislation regulating this, and counter surveillance to ensure any would be illegal surveillance has a high chance of being observed and thus punished. You do this by making the technology widely available but regulating its use. People would still try to get around it but the price of getting caught abusing the tech would be jail. And with surveillance being inescapable, you'd never be certain nobody is watching you misbehaving. The beauty of mass, multilateral surveillance is that you wouldn't ever be sure nobody is not watching you abuse your privileges.
Of course, the reality of states adopting this and monopolizing this is already resulting in 1984 like scenarios in e.g. China, North Korea, and elsewhere.
> ...the more urgent debate is how to adapt to this being the reality.
Start building more offline community. Building things that are outside the reach of AI because they're in places you entirely control, and start discouraging (or actively evicting...) cell phones from those spaces. Don't build digital-first ways of interacting.
Might work, might not. If someone keeps their cell phone silenced in their pocket, unless you're strip searching you won't know it's there. Does the customer have some app on it listening to the environment and using some kind of voice identification to figure out who's there. Do you have smart TVs up on the walls at this place, because hell, they're probably monitoring you too.
And that's only for cell phones. We are coming to the age where there is no such thing as an inanimate object. Anything could end up being a spying device feeding data back to some corporation.
> Does the customer have some app on it listening to the environment and using some kind of voice identification to figure out who's there.
This is no different from "So-and-so joined the group, but is secretly an FBI informer!" sort of problems, in practice. It's fairly low on my list of things to be concerned about, but as offline groups grow and are then, of course, talked about by a compliant media as "Your neighbor's firepit nights could be plotting terrorist activities because they don't have cell phones!" when prompted, it's a thing to be aware of.
Though you don't need a strip search. A decent NLJD (non-linear junction detector) or thermal imager should do it if you cared.
I'm more interested in creating (re-creating?) the norms where, when you're in a group of people interacting in person, cell phones are off, out of earshot. It's possibly a bit more paranoid than needed, but the path of consumer tech is certainly in that direction, and even non-technical people are creeped out by things like "I talked to a friend about this, and now I'm seeing ads for it..." - it may be just noticing it since you talked about it recently (buy a green car, suddenly everyone drives green cars), or you may be predictable in ways that the advertising companies have figured out, but it's not a hard sell to get quite a few people to believe that their phones are listening. And, hell, I sure can't prove they aren't listening.
> Do you have smart TVs up on the walls at this place...
I mean, I don't. But, yes, those are a concern too.
And, yes. Literally everything can be listening. It's quite a concern, and I think the only sane solution, at this point, is to reject just about all of that more and more. Desktop computers without microphones, cell phones that can be powered off, and flat out turning off wireless on a regular basis (the papers on "identifying where and what everyone is doing in a house by their impacts on a wifi signal" remain disturbing reads).
I really don't have any answers. The past 30 years of tech have led to a place I do not like, and I am not at all comfortable with. But it's now the default way that a lot of our society interacts, and it's going to be a hard sell to change that. I just do what I can within my bounds, and I've noticed that while I don't feel my position has changed substantially in the past decade or so (if anything, I've gotten further out of the center and over to the slightly paranoid edge of the bell curve), it's a lot more crowded where I stand, and there are certain issues where I'm rather surprisingly in the center of the bell curve as of late.
Good luck building things with out leaving an ai reachable paper trail. You'd have to grow your own trees, mine your own iron and coal, refine your own plastic from your own oil field.
Sounds fun to me and my social group. We not-quite-joke about the coming backyard refineries. I'm working on the charcoal production at the moment (not a joke, I have some small retorts in weekly production, though I'm mostly aiming for biochar production instead of fuel charcoal production).
Realistically, though, if all you have to work with are my general flows of materials in and out, I'm a lot less worried than if you have, say, details of home audio, my social media postings, etc (nothing I say here is inconsistent with my blog, which is quite public). And there are many things I don't say in these environments.
> Building things that are outside the reach of AI because they're in places you entirely control
This sounds great in principle, but I'd say "outside the reach of AI" is a much higher bar than one would naively think. You don't merely need to avoid its physical nervous system (digital perception/control), but rather prevent its incentives leaking in from outside interaction. All the while there is a strong attractor to just give in to the "AI" because it's advantageous. Essentially regardless of how you set up a space, humans themselves become agents of AI.
There are strong parallels between "AI" and centralizing debt-fueled command-capitalism which we've been suffering for several decades at least. And I haven't seen any shining successes at constraining the power of the latter.
Oh, I'm aware it's a high bar. Like most people here, I've worked my life in tech, and I'm in the deeper weeds of it.
But I don't see an alternative unless, as you note, one just gives into the "flow" of the AI, app based, "social" media, advertising and manipulation driven ecosystem that is now the default.
I'm aware I'm proposing resisting exactly that, and that it's an uphill battle, but the tradeoff is retaining your own mind, your own ability to think, and to not be "influenced" by a wide range of things chosen by other people to cross your attention in very effective ways.
And I'm willing to work out some of what works in that space, and to share it with others.
> Both were always going to be kind of inevitable as soon as the technology would get there
This is my take on everything sci-fi or futuristic. Once a human conceives something, its existence is essentially guaranteed as soon as we figure out how to do it.
Its demise is also inevitable, so it would be a matter of being wise in figuring out how long it takes us to see/feel the downsides, or how long until we (or it) build something "better".
Related to this is the notion of ubiquitous surveillance. Where basically anywhere you go, there is going to be active surveillance literally everywhere and AIs filtering and digging through that constantly. That's already the case in a lot of our public spaces in densely populated areas. But imagine that just being everywhere and virtually inescapable (barring Faraday cages, tin foil hats, etc.).
The most feasible way to limit the downsides of that kind of surveillance is a combination of legislation regulating this, and counter surveillance to ensure any would be illegal surveillance has a high chance of being observed and thus punished. You do this by making the technology widely available but regulating its use. People would still try to get around it but the price of getting caught abusing the tech would be jail. And with surveillance being inescapable, you'd never be certain nobody is watching you misbehaving. The beauty of mass, multilateral surveillance is that you wouldn't ever be sure nobody is not watching you abuse your privileges.
Of course, the reality of states adopting this and monopolizing this is already resulting in 1984 like scenarios in e.g. China, North Korea, and elsewhere.