What you say is true, in that context. I guess it depends on what the group is we're talking about when we're talking what "average" is. He could become an "average" consumer in the context of a vegetarian; always eating what every other vegetarian is eating because he's never offered an alternative.
As for your example with the programming languages, I would find it annoying if it only always displayed results for what it "thinks" I'm looking for. I'd rather define what language I need the information for. Otherwise, if I get accustomed to the system then it would impede me from researching a new language because I'll have to go on a crusade to convince it that I do indeed want information about a new subject. Then it learns my new subject but what if I want to go back to getting information about the old subject? Now I have to convince it that, yes, I do want the information I wanted before that I didn't want later that I want again now.
Google reverses your solution. By default, you use less words to get what it thinks you want. If you type more words, you get what you actually want. So 99% of the time, I get the Java results. If I want a Ruby result, I type "ruby" in the query dialog and now all the results are relevant.
Most people are not power users and only type a single word into the search box. Learning means that Google gives even novice users great results.
No, you repeated my solution. I stated I would rather it be that I have to type in "ruby", just as you state. You are ignoring what I said about getting accustomed to the system and forgetting to put in the "ruby" part. Then I have to repeat the search. Then Google suddenly thinks I'm a ruby developer. Then I have to convince it yet again for another example. It never ends.
Google trying to learn what it "thinks" people want creates the filter bubble, which is the point of the discussion. It will not always give the most relevant result on the topic but what it "thinks" is the most relevant to you; but it could be wrong, a lot.
Plus, has anybody discussed how this stuff might affect traffic for websites in general? If Google is sending different people with the same search query to completely different websites then being the most relevant on a topic might not be as helpful anymore.
As for your example with the programming languages, I would find it annoying if it only always displayed results for what it "thinks" I'm looking for. I'd rather define what language I need the information for. Otherwise, if I get accustomed to the system then it would impede me from researching a new language because I'll have to go on a crusade to convince it that I do indeed want information about a new subject. Then it learns my new subject but what if I want to go back to getting information about the old subject? Now I have to convince it that, yes, I do want the information I wanted before that I didn't want later that I want again now.
At least give us the option to turn it off.
I don't know where you got scary from though.