I think the conflicting point you missed is that "right tool for the job" also implies "right tool". If you don't think that probabilistic output counts as "query response" then LLMs are the "wrong tool" for any text query engine. If a database engine returned the right answer only X% of the time, you would say the database engine is faulty and find another. LLMs are probabilistic algorithms that by their very nature cannot hit 100% accuracy. (Especially as you get into the specifics of things like the lossy "compression" mechanics of tokenization and vectorization. The training set isn't 100% represented without some loss, either.) That doesn't seem like a good fit for a "query engine" tool in a database sense to some of us.
In practice they seem to work well for that at a surface level, most of the time. The complaint is not that LLMs are not a tool for the job of "fancy text query engine", the complaint is that at scale and in the long run, LLMs are not a good tool for that.
For lots of jobs of “text querying” they do good enough of a job to be on par with humans (which are not infallible either).
And there are applications where you don’t have/wouldn’t pay another human, and the job that an AI does for mere cents is good enough most of the times. Like doing an analysis on a legacy codebase. I’ll read and verify, but running that “query” then saved me a lot of time.
Not everything needs to be deterministic to be of value.
I agree, they can be "practical tools for the job", that's where I ended my comment. The disagreement seems to be that "practical tool for the job" is the same as "right tool for the job". A hammer can be a practical tool for the job of screwing a nail into a wall (once, at least) but few would call it the right tool for that job. An LLM can be a practical tool for a text query (at least as a first pass, at least with review and a grain of sand), but if you need reliability or repeatability or the ability to send results directly to a customer without a human in the loop it may not be the right tool for the job.
There's obviously a value in practical tools, deterministic or not. It's just worth making the distinction that a practical tool is not always fit for purpose as the "right" tool if you really are seeking the (most) right tool for the job.
And he's right. LLMs are fancy text query engines and work very well as such.
The problem is when people try to shoehorn everything into LLMs. That's a disaster yet being perused vigorously by some.