My guess is that Google doesn't care (not that they ever did but now less than they did before). The simple math is "Google makes no money by indexing your website." The easiest way in the past to get reliably indexed by Google was to put Google ads on your web site. Then they have skin in the game so to speak. "Long tail" websites, those that are very niche, not especially linked outside of a few folks, have only ever been there to impress people using Google into believing they index everything (which they don't, and frankly haven't for at least the last 10 years).
Over the years, as all of the ways that people who hosted a "search engine", exploited the "digital exhaust" of the people who used it, and sold that information for profit, have been revealed, well regulations and laws have been enacted to reduce or eliminate the egregious (and usually more profitable) ones. That and advertisers tired of paying for "engagement" and seeing few sales or conversions, the economics of running a search engine as this high margin, make money hand over fist, days are slowly winding down.
That Bing has (for now at least), continued to index these sites, will just accelerate Google's decline.
When I was helping to run Blekko, people would look at our curated results and be super impressed at how much better they were than Bing or Google results, but then they would search for their cousin's minecraft blog and wouldn't find it and lament that they couldn't possibly use it as their "daily driver" because if it didn't have their cousin's website in it, how could they know what they weren't seeing? Blekko tried really hard to make the argument that if you made a list of the minecraft blogs you followed as a Blekko user we'd index all of them, and if everyone did that for their favorite stuff then the index would fill up with good web sites and not be riddled with junk. But sadly we couldn't get them to internalize that and they weren't willing to create an account to have a better web experience. Perhaps we were just too early but still it is a weird thing.
I think it's important to realize that search engines both bring websites to people and people to websites. The genius part of Google's position in the market is that they both sell ads, and then direct people to the places where the ads are sold. This is also a major contributor to their search engine spam problem; they can't well penalize ads without undermining their core business model.
Although I don't think any search engine has ever indexed or will ever index everything. That doesn't really make sense. Realistically I think maybe 1% of the documents online are ever going to be a good search result for any query ever.
Internet search is all about being judicious about what you index.
>>> I think it's important to realize that search engines both bring websites to people and people to websites.
I don't disagree with this, I was simply pointing out that Google makes no money by sending people to a web site unless the owner of that web site has bought an advertisement and the person involved has clicked on that ad. Organic search, like "good explanation for approximation theory" should have something like this: https://xn--2-umb.com/22/approximation/ as one of its top results but it doesn't even make it to the first page.
Over the years, as all of the ways that people who hosted a "search engine", exploited the "digital exhaust" of the people who used it, and sold that information for profit, have been revealed, well regulations and laws have been enacted to reduce or eliminate the egregious (and usually more profitable) ones. That and advertisers tired of paying for "engagement" and seeing few sales or conversions, the economics of running a search engine as this high margin, make money hand over fist, days are slowly winding down.
That Bing has (for now at least), continued to index these sites, will just accelerate Google's decline.
When I was helping to run Blekko, people would look at our curated results and be super impressed at how much better they were than Bing or Google results, but then they would search for their cousin's minecraft blog and wouldn't find it and lament that they couldn't possibly use it as their "daily driver" because if it didn't have their cousin's website in it, how could they know what they weren't seeing? Blekko tried really hard to make the argument that if you made a list of the minecraft blogs you followed as a Blekko user we'd index all of them, and if everyone did that for their favorite stuff then the index would fill up with good web sites and not be riddled with junk. But sadly we couldn't get them to internalize that and they weren't willing to create an account to have a better web experience. Perhaps we were just too early but still it is a weird thing.