Suppose you have some natural language analysis question like “how many users clicked on x product?”. In the demo, there’s always a ‘clicks’ table and the LLM just gets it right. In the real world, that product has multiple links on multiple platforms some of which represent a prefetch and also that product really comes in multiple skus so do you want all of them bla bla bla. It would already be helpful without AI to better document how a table works but it doesn’t happen and there are already better ways to document tables than natural language which nobody uses so I doubt businesses are just going to suddenly be great at documentation. The LLM will get the wrong analytic answer because it doesn’t know how anything works, which makes the analytic tools not very useful.
So figuring out and reasoning through that mess is pretty much the specific thing that LLMs are good at, I'd recommend actually trying the newer models if you're still operating on the assumption that these are at GPT 3 level. Random (probably cherry picked) twitter thread but it seems to be doing fine for the type of stuff that makes up 99% of spreadsheet use as far back as april 2023: https://twitter.com/emollick/status/1652170706312896512
That real world example also doesn't seem representative of the bulk of real world use of spreadsheets. Even in the announcement they're focusing on simple pivot tables and creating presentations. I don't see any reason why this wouldn't be able to handle asking questions on a data dump from quickbooks or the like. I get the feeling that you might be operating in a very unusual context, like in tech or finance, where you might even have professional data analysts who are hired to work with data. That's probably the biggest market revenue wise for this stuff but in terms of the number of users, spreadsheets are used EVERYWHERE
I don't agree that they are good at it, but even if they are I don't think the models even have the information needed to reason, they just have a very short abbreviation. Even in that very simple example the model is screwing up, running regressions determined entirely by one outlier and not even copying the R^2 to the text correctly. So now someone has to go back and fix the model's work.
Spreadsheets ARE everywhere, but they're even harder to connect back to the source of the data because their proximal source is some e-mail message. There just isn't enough information in the sheet to tell the model what the data actually means.
