After spending the last few years doing deep dives into how these systems work, what they are doing and the math behind them. NO.
Any time I see an AI SAFETY paper I am reminded of the phrase "Never get high on your own supply". Simply put these systems are NOT dynamic, they can not modify based on experience, they lack reflection. The moment that we realize what these systems are (were NOT on the path to AI, or AGI here folks) and start leaning into what they are good at rather than try to make them something else is the point where we get useful tools, and research aimed at building usable products.
The math no one is talking about: If we had to pay full price for these products, no one would use them. Moores law is dead, IPC has hit a ceiling. Unless we move into exotic cooling we simply can't push more power into chips.
Hardware advancement is NOT going to save the emerging industry, and I'm not seeing the papers on efficiency or effectiveness at smaller scales come out to make the accounting work.
"Full price"? LLM inference is currently profitable. If you don't even know that, the entire extent of your "expertise" is just you being full of shit.
>Simply put these systems are NOT dynamic, they can not modify based on experience, they lack reflection.
We already have many, many, many attempts to put LLMs towards the task of self-modification - and some of them can be used to extract meaningful capability improvements. I expect more advances to come - online learning is extremely desirable, and a lot of people are working on it.
I wish I could hammer one thing through the skull of every "AI SAFETY ISNT REAL" moron: if you only start thinking about AI safety after AI becomes capable of causing an extinction level safety incident, it's going to be a little too late.
> I wish I could hammer one thing through the skull of every "AI SAFETY ISNT REAL" moron: if you only start thinking about AI safety after AI becomes capable of causing an extinction level safety incident, it's going to be a little too late.
How about waiting till after "AI" becomes capable of doing... anything even remotely resembling that, or displaying anything like actual volition?
"AI safety" consists of the same thing all industrial safety does: not putting a nondeterministic process in charge of life- or safety-critical systems, and only putting other automated systems in charge with appropriate interlocks, redundancy, and failsafes. It's the exact same thing it was when everybody was doing "machine learning" (and before that, "intelligent systems", and before that some other buzzword that anthropomorphized machines...) and not being cultishly weird about statistical text generators. It's the kind of thing OSHA, NTSB and the FAA (among others) do every day, not some semi-mystical religion built around detecting intent in a thing that can't actually intend anything.
If you want actual "AI safety", fund public safety agencies like NHTSA and the CPSC, not weird Silicon Valley cults.
> How about waiting till after "AI" becomes capable of doing... anything even remotely resembling that
I think it would pretty unfortunate to wait until AI is capable of doing something that "remotely resembles" causing an extinction event before acting.
> , or displaying anything like actual volition?
Define "volition" and explain how modern LLMs + agent scaffolding systems don't have it.
What people currently refer to as "generative AI" is statistical output generation. It cannot do anything but statistically generate output. You can, should you so choose, feed its output to a system with actual operational capabilities -- and people are of course starting to do this with LLMs, in the form of MCPs (and other things before the MCP concept came along), but that's not new. Automation systems (including automation systems with feedback and machine-learning capabilities) have been put in control of various things for decades. (Sometimes people even referred to them in anthropomorphic terms, despite them being relatively simple.) Designing those systems and their interconnects to not do dangerous things is basic safety engineering. It's not a special discipline that is new or unique to working with LLMs, and all the messianic mysticism around "AI safety" is just obscuring (at this point, one presumes intentionally) that basic fact. Just as with those earlier automation and control systems, if you actually hook up a statistical text generator to an operational mechanism, you should put safeguards on the mechanism to stop it from doing (or design it to inherently lack the ability to do) costly or risky things, much as you might have a throttle limiter on a machine where overspeed commanded by computer control would be damaging -- but not because the control system has "misaligned values".
Nobody talks about a malfunctioning thermostat that makes a room too cold being "misaligned with human values" or a miscalibrated thermometer exhibiting "deception", even though both of those can carry very real risks to, or mislead, humans depending on what they control or relying on them being accurate. (Just ask the 737 MAX engineers about software taking improper actions based on faulty inputs -- the MAX's MCAS was not malicious, it was poorly-engineered.)
As to the last point, the burden of proof is not to prove a nonliving thing does not have mind or will -- it's the other way around. People without a programming background back in the day also regularly described ELIZA as "insightful" or "friendly" or other such anthropomorphic attributes, but nobody with even rudimentary knowledge of how it worked said "well, prove ELIZA isn't exhibiting free will".
Christopher Strachey's commentary on the ability of the computers of his day to do things like write simple "love letters" seems almost tailor-made for the current LLM hype:
"...with no explanation of the way in which they work, these programs can very easily give the impression that computers can 'think.' They are, of course, the most spectacular examples and ones which are easily understood by laymen. As a consequence they get much more publicity -- and generally very inaccurate publicity at that -- than perhaps they deserve."
LLMs are already capable of complex behavior. They are capable of goal-oriented behavior. And they are already capable of carrying out the staples of instrumental convergence - such as goal guarding or instrumental self-preservation.
We also keep training LLMs to work with greater autonomy, on longer timescales, and tackle more complex goals.
Whether LLMs are "actually thinking" or have "volition" is pointless pseudo-philosophical bickering. What's real and measurable is that they are extremely complex and extremely capable - and both metrics are expected to increase.
If you expect an advanced AI to pose the same risks as a faulty thermostat, you're delusional.
It depends a lot on which LLMs you're talking about, and what kind of usage. See e.g. the recent post about how "Anthropic is bleeding out": https://news.ycombinator.com/item?id=44534291
Ignore the hype in the headline, the point is that there's good evidence that inference in many circumstances isn't profitable.
> In simpler terms, CCusage is a relatively-accurate barometer of how much you are costing Anthropic at any given time, with the understanding that its costs may (we truly have no idea) be lower than the API prices they charge, though I add that based on how Anthropic is expected to lose $3 billion billion this year (that’s after revenue!) there’s a chance that it’s actually losing money on every API call.
So he's using their API prices as a proxy for token costs, doesn't actually know the actual inference prices, and ... that's your "good evidence?" This big sentence with all these "We don't knows?"
Well, that and the $3 billion expected loss after revenue.
Does this idea upset you for some reason? Other people have analyzed this and come to similar conclusions, I just picked that one because it's the most recent example I've seen.
Feel free to look to a source that explains how LLM Internet is mostly profitable at this point, taking training costs into account. But I suspect you might have a hard time finding evidence of that.
Any time I see an AI SAFETY paper I am reminded of the phrase "Never get high on your own supply". Simply put these systems are NOT dynamic, they can not modify based on experience, they lack reflection. The moment that we realize what these systems are (were NOT on the path to AI, or AGI here folks) and start leaning into what they are good at rather than try to make them something else is the point where we get useful tools, and research aimed at building usable products.
The math no one is talking about: If we had to pay full price for these products, no one would use them. Moores law is dead, IPC has hit a ceiling. Unless we move into exotic cooling we simply can't push more power into chips.
Hardware advancement is NOT going to save the emerging industry, and I'm not seeing the papers on efficiency or effectiveness at smaller scales come out to make the accounting work.