I can’t help but notice how much this echoes Francois Chollet’s On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area.

It seems more about coverage-driven competence. Somewhat analogous to overfitting at scale.

The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?

With our current AI architectures and training in place, I think we will only continue on skill acquisition optimization vs. truly novel intelligence.

You are conflating multiple things.

1) First, you are talking about positive forward transfer in continual learning. I've been giving talks for the past 6-7 years about how that community (I was one of the founders) went astray and wasn't focusing enough on that topic, but continual learning of the kind you are thinking isn't in any of these systems right now. I think some people left the Grok team to make a start-up to focus on that. By forward transfer, what I mean is weights update over time and past learning improves future learning such that we get better sample efficiency.

2) Psychologists distinguish among different kinds of intelligence for Spearman's g (IQ). Crystalized intelligence is using already acquired knowledge (frontier models probably have maxed out that). Fluid intelligence is reasoning and finding solutions in novel situations or without the necessary crystalized knowledge. [Giving colloquial definitions]

3) Now, interestingly, neither of those are correlated with _creativity_ (just they are independent, note some have this threshold theory but it hasn't held up in recent papers). That's what the AI's really are terrible at -- creativity. But I'd argue the vast majority of humans aren't very creative, with truly out-of-the-box ideas. Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD.

I did a bunch of research on these topics for my AGI course that I teach each Spring (where I then point out conflicting definitions and start using multiple alternative terms rather than AGI to distinguish among the different definitions).

Appreciate the input! Responses below to your first two items:

1) I'm leaning more into a broader sense, which is that given the priors a system already possesses, how efficient can it acquire competence on a novel task? If I'm reading the point you make, you're focussing on continual learning right? If so, I'm not necessarily restricting my statement above to that.

Here's another reframing: How much of the benchmark improvements come from overwhelmingly large training distributions vs improving the models for adapting to things genuinely outside of it?

2) Excellent points about crystalized and fluid intelligence. Wouldn't the LLM scaling gains be a representation of crystallized capabilities? In regard to Gf, that is exactly what I am asking about. That is what seems to be lacking, Gf like adaptation under genuine novelty.

My concern is that it's increasingly difficult to tell of what looks like Gf like behavior is really coming from better adaption vs. having broad priors from the model's large learned distributions.

> Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD.

Very strong reasoning here. Is there anything this ADHD condition cannot explain?

Very interesting!

I was looking at your website and wondering if there is a way to have access to the course material/videos?

In particular: Spring 2025 @ UR : CSC 209/409 Seminar on Artificial General Intelligence

Fall 2024 @ UR : CSC 277/477 End-to-End Deep Learning

Spring 2023 @ UR : CSC 266/466 Frontiers in Deep Learning

Spring 2022 @ Cornell Tech : CS 5787 – Deep Learning

Fall 2021 @ RIT : IMGS 684 – Deep Learning for Vision

> That's what the AI's really are terrible at -- creativity

Good thought piece here "We Are Losing the Ability to Discover What We Didn’t Know to Ask[1]" By Anne-Laure Le Cunff

It keeps playing on my mind as I see people at work follow some predetermined AI workflow to get their jobs done, the art of being curious and exploring around the problem is so important to the really big innovations. Been thinking about how to address this through some of the harnesses we are developing in the knowledge working space.

[1] https://archive.is/IAxf9

I met a senior (as in, 4th year of college) recently and she asked me: what advice do you have for someone just graduating in the AI age? (as I had told her that I've been in ML for 20+ years, etc.)

My recommendation to her was: just _play_ with the AI! It's a brand new tool, and none of us knows its capabilities, limitations, boundaries etc. (which are fluid, of course). So just spend as much time as you can tinkering with it, playing with it, making it do things it was not expected to do, etc. and you'll develop an idea of how to make better use of it.

Good advice. Never underestimate the power of play, something I recently re-learnt after having kids.

Tangent: Anne-Laure Le Cunff is a neuroscientist with a talent for community-building and writing, and bringing evidence-based approaches to bear on practical solutions to various domains. See eg https://nesslabs.com

> That's what the AI's really are terrible at -- creativity. But I'd argue the vast majority of humans aren't very creative, with truly out-of-the-box ideas. Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD.

I think it's hard to define creativity in the context of AI because they seemingly just make up new hyphenated terms for everything. Is that creativity? If not, what about when they do the same thing different ideas in the latent space?

If we say that simply nailing one concept to another isn't creativity, then AIs are incapable of creativity, while the vast majority of humans are incapable of creativity. This is just a long way of saying "0 AIs have creativity, 0.00001% of humans have creativity", and the difference between zero and a very small number is infinity.

It always comes back to taste.

It’s not that they’re incapable of creativity, it’s that LLM-driven creativity is terrible, and nothing makes me cringe more than when it uses a word in a “novel” way.

But monkeys-with-typewriters, they sometimes stumble upon something that doesn’t suck. But if you don’t want to spend a fortune retrying the same task until you get a suitable result you have to inject your own taste.

Sometimes I start with a super vague prompt and see how close agents can get to something that doesn’t suck. I inevitably get frustrated about 6-7 prompts in when they’ve created a complete mess because they have no taste. So I restart and inject my taste into the process. Things like linters, test suites, which 3PLs to use, etc.

Taste or vision? They are hill climbing and they can't see the other side. And sometimes that is because they don't live in your head and don't know what you want.

Very insightful actually. Taste and vision are both instances of holding a model that predicts a good result beyond the threshold of validity in theory but reality happens to align. Is it luck then? Perhaps meta-luck where lucky weights produce “accidentally great” results with some predictability.

Steve Jobs had a mental model that brought the iPhone. No one really wanted it but something in his life biased the result.

So creativity is having weights so good you can project way out into latent space beyond what is reasonable.

Hi thanks for the insight. Do you see a role for Control Systems(i.e. ones analogus to Instrumentation engineering) playing a role to modulate certain parts of continual learning? One very important way we learn are lived experiences, it's like telling memory:this part is more important( for emotional or social utility values), pay attention. Good or bad lived experiences both count. I guess is that a path that practical research is considering?

> weights update over time and past learning improves future learning such that we get better sample efficiency

Is there an architecture-independent definition of forward transfer?

For the practical experience and implications of AI progress, I think we are increasingly discussing what these LLMs can accomplish inside a stateful harness, the state of which could be described as part of a (very squirrely) parameter space.

I think our AI systems are essentially massive Central Executive Networks. But novel ideas (creativity) come from the Default Mode Network.

These are the difference in what Kahneman called System 2&1 thinking and what the ancients called the Ratio and the Intellect.

LLMs are all ratio. They depend on our intellect for guidance.

"Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD." Is this a statement of fact? As someone with ADHD and pretty confident in my creativity, I still believe this is much more cope than fact(which could be more a self-doubt thing than anything else).

Further, if there is a correlation, I'd bet it's not so much an intrinsic "creativity" trait, but more effectively higher creativity because more trials. That is, along the lines of Chollet's paper, a measure of creativity should be based on a fixed budget with fixed knowledge.

Among many other possibilities I haven't considered, perhaps another mechanism could be that because ADHD people spend more time thinking in less goal-oriented ways and mixing thoughts on accident, perhaps we do in fact gain some learned creativity via experience with vagueness[1]? But that might also imply that part of creativity is actually being able to diffuse more freely through thought space and lowering the barrier to attempted connections between ideas. That lower barrier leads to less likelihood of any "collision" being meaningful but maybe it's overcome by higher collision rates? Or maybe effectively higher order (not just pairwise) collisions?

Disclaimer in case it's not obvious: I don't know any of the literature on what creativity even means or how it's quantified.

[1] Which is me injecting an assumption that creativity ~= connecting things with no obvious or well-troden reasoning path between them.

edit -- oops just looked at your profile after seeing someone elses comment. I assume you are stating a fact then, leaving original anyway

I think it's just that many of us don't have the capability to just do things in rote or the conventional ways, instead we must rely more on the creative and unconventional ways of doing things and parts of the brain responsible for that

What have you defined as creativity and intelligence?

This comment and the one above from astrobiased feel like coming into a messy codebase, and it’s more work to sort it out than it would have been to write it from scratch.... And since I actually do intelligence testing as a clinical psychologist, I have experience with this in both practice and theory. So now I’m going to waste an hour because I just have to respond to “something is wrong on the internet.”.

Chollet's distinction is useful. High performance on known tasks is not the same thing as efficient adaptation to a novel task. Prior knowledge and training data can buy skill. That is a central point of On the Measure of Intelligence. But it does not follow that current frontier progress is only "coverage-driven competence." That is a hypothesis. It is not a result established by Chollet's framework.

"Overfitting at scale" is also the wrong term. A model that learns broad representations and applies them successfully to unseen examples is generalizing. The relevant concern is whether apparent novelty is actually inside the effective training distribution, not whether the model is "overfit."

There is also an unstated premise here: that adding broad knowledge and skills cannot improve the machinery used for novel problem solving. I do not see a basis for assuming that. Learned representations, abstractions, reasoning patterns, and cross-domain analogies can themselves support transfer to new tasks. Whether this becomes sufficient for general intelligence is an open question with insufficient data. But its a perfectly valid hypothesis right now that, given enough domain knowledge and symbolic reasoning examples, LLM COULD maybe "Grok" AGI at a certain critical threshold.

And ARC-AGI-3 was specifically designed around novel abstract environments that require exploration and adaptation. Astra scores 99.9% with OpenAI's context-preserving Provider Adapter, and ARC reports that Astra constructed compact symbolic models of unfamiliar environments. That does not prove AGI, but it points in that direction more so than the other way around.

Gc roughly maps to acquired knowledge. Gf roughly maps to reasoning in relatively novel situations. Naming those two categories does not tell us whether increasing acquired knowledge and learned abstractions in an AI can improve Gf-like behavior. That causal question is exactly what is disputed.

And "Frontier models probably have maxed out crystallized intelligence" is just obviously wrong, unless you think they have been able to dig up every a scrap of paper with knowledge/information on it in the entire world, AND that there is no more useful knowledge to be generated left in the universe.

And the statement that intelligence and creativity are independent is simply wrong. A meta-analysis of 112 studies and 34k participants found a positive correlation of about r .25 between intelligence and divergent thinking. It also found that using g, Gf, or Gc did not eliminate that relationship. Creative achievement has a smaller but still positive meta-analytic association with intelligence, around r = .16. These are distinct constructs, not independent constructs.

And this is just a bad take: "AIs are terrible at creativity". At best that depends on which creativity, and I think its straight up wrong. On divergent thinking tasks, the operationalization behind every ADHD study you could cite, LLMs score above most humans, with the top humans still ahead. If you means Big-C, paradigm-shifting creativity, that is a different construct and none of the ADHD evidence transfers to it.

And if I where to say what I subjectively feel and see.... I have ABSOLUTELY no idea how people can say that we are not seeing sparks of creativity from AIs already. If a PERSON produced some of the music, solutions or deductions that I have seen AIs do, people would have NO problem celebrating it as extremely creative.

And finally, the ADHD claim is also, at best, overstated and just as often debunked. There is some evidence that higher subclinical ADHD trait scores, often survey studies only, are associated with better performance on some divergent-thinking measures. But a review of 31 studies did not find a consistent creativity advantage for people with clinical ADHD, and it found no evidence of better convergent thinking.

Okay, I’m done… And nobody noticed that I’m not doing my job here.

For me a basic test of A.I. creativity is give the A.I. chapter 1 of a novel it has not seen and ask it to write chapter 2, then compare the quality to the original author's chapter 2. The A.I. is always terrible at this in terms of matching the author's level of quality.

I would think that’s the opposite of creativity. Rather, I would label that as pattern matching and prediction capability.

Why box in creativity basically as the ability to mimic someone else who is creative? Why not choose something that better matches the definition of creativity (the ability to make new things, think of original ideas, or show imagination that is novel, useful, or pleasing)?

And even then, I am assuming the premise that the original novels are good and creative, especially chapter 2. Most novels are not very creative.

If it could make 5 different versions of chapter 2, all with different directions for the story, and all with novel and interesting developments, wouldn’t that be a better definition of creativity than “can it read chapter 1 and be able to copy style and deduce/predict what the author is going to do in chapter 2”?

And also, that’s only a subset of creativity (storytelling). I’ve met plenty of people who are terrible at writing and storytelling but can come up with the most impressive and novel solutions to a practical problem instantly.

Final thought: I have been following the development in the anime AI generation scene for a while now. It’s not even close to anything anyone would call creative or even OK quality. But it’s also massively impressive that it’s moving in that direction really fast. And if you watch enough, you are going to start to see some truly creative sparks. And some of the mainstream stuff that gets created and labeled as creative… really... How many isekai series with the same story have humans not made already?

> If it could make 5 different versions of chapter 2, all with different directions for the story, and all with novel and interesting developments, wouldn’t that be a better definition of creativity than “can it read chapter 1 and be able to copy style and deduce/predict what the author is going to do in chapter 2”?

If I like the novel, an alternative version where things happen differently would be fine.

The problem isn't that it's different. The problem is quality.

A lot of Isekai stories are crap but you can still rank them in terms of the author's ability or inability to have a creative POV that elevates the material.

I think “what I individually like” is a poor measurement of creativity, at least alone.

And yes you can rank quality. My point was that if you made two bell curves of the distribution quality and creativity of all new manga, The one for “AI slop manga” allready started overlapping with “normal human manga”.

That’s just a fancy way of saying the absolute best AI slop is at the level of the absolute worst human creation.

The interesting thing is that the AI slop curve clearly is moving to the right every month. Where it will stop tho, impossible to say.

Is chapter 1 of a novel really enough context to generate a chapter 2 of sufficient quality to match up to an author who spent a good amount of time planning out an entire story and whose manuscript probably went through a lot of revisions?

I don't necessarily just feed it chapter 1. (And chapter length varies between novels).

And no I don't think asking the A.I. to write the next 2000 words would require it to have the entire novel planned out. Not all writers even outline in advance.

My take on it is that even if there was no "novel" discovery (leaving that up to the reader to define), if you consider human knowledge to be a sphere in N-dimensional state space, "within that surface area" is Swiss cheese, and AI seems to at minimum be able to fill in some of those holes.

And those hole-fillings, for all intents and purposes, look to us like novelty, even if much of it was simply overlooked by us, or, perhaps, unable to attain due to time or other constraints.

Now if you want to talk beyond the sphere, let's call it the "novel novel discovery of the unknown unknowns", then you may have a point, and AI may be more limited than humans in discovering the things that we don't know we don't know. Especially the as-yet-unmodelable things i.e. intuition.

Plenty of discovery left just working from first principles, however. Which I cautiously suggest current frontier AI is good enough to model to a significant enough extent that it is useful for discovery.

They do new stuff all the time. Ask your AI to draw a gerbil riding a unicycle around Pluto and you'd get an image that hasn't been there before. If by genuinely new you mean without any help from past culture, do humans do that?

For significant pushing the boundaries of knowledge stuff you maybe need different algorithms like AlphaGo move 37 or Alpha Fold protein folding. Though again how often do humans do that?

Humans do that by inventing algorithms like AlphaGo :)

In a more serious note, I think the person you are responding to meant "new" more in line with "novel".

Take a microfluidic chip, for example. Current AI systems can create new flow cell geometry, but cannot come up with the idea for a microfluidic flow cell itself.

Most employed (and contracting) humans are prompted repeatedly every day.

Yes, but also no. If I prompt a graphics artist, "draw me a picture, any style, of a calendar with this day circled in red", you do not have to tell them what a calendar looks like, what the days of the week are, or how many there are, or that days in a month increment sequentially.

Yet if you prompt an AI: https://share.gemini.google/TyUxSnrKmq9h

Also, you probably don't need to tell an artist "don't violate other people's copyrights" while you're at it, though that's perhaps somewhat more debatable than "does the artist know that the day after Thursday is Friday".

This applies equally to all disciplines: AI generated code regularly contains wtfisms that a human would not need to be guided from, or at the very least (more similar to the copyright problems) that experience and knowledge would drive them away from — and permit me to entrust, particularly more experienced — humans with a vague outline of the idea, and trust that the details will get filled in sensibly. "Filling in details sensibly" is where AI hallucinates the hardest.

(And just to head off, "it's a one-off mistake!" Another example: https://share.gemini.google/yE6axJvBDKYE ; another example: https://share.gemini.google/WG6TEBjlyro7 (though admittedly, the calendar is pretty good here, I think "humans have 2 arms" is well within the point I'm making of "stuff I don't need to prompt human artists with") ; and another example: https://share.gemini.google/CIH4QM2teQKf ; and another example: https://share.gemini.google/n76c9eJq1dGe)

Anyone can prompt an AI. Also an AI.

See: advertising & marketing.

Or: political propaganda.

Or: bumper stickers. Or political party emails / calls to action.

Humans are constantly prompted by The Joneses and perceived authority figures (boss, religion, politics, peers, co-workers, influencers, et al.).

“Hello, how are you?”

Chollet writes he expects AGI now sooner than 2030, "given progress is happening faster than I expected."

https://x.com/fchollet/status/2095607046129463577

I have no idea about timelines, but the current LLM architecture has no mechanism which could emulate online learning in a way similar to how it happens in humans and other animals. As another commenter wrote, there is no neuroplasticity while these systems interact with their environment in normal usage; adding information/constraints to the context partially mitigates this, but it's a completely different mechanism.

The RL phase is the most similar mechanism I know of that comes to my mind, but I'm not sure it could be adapted to fill that gap.

My totally unsubstantiated theory is that this is the missing link towards what most humans would consider AGI. I don't see this as intractable, but it may require some substantial change in architecture.

We've had that concept for quite a long time now, in the form of Lora [1] and similar fine-tuning techniques.

It first got popular for StableDiffusion to teach the image generation models new concepts.

We could easily live in a world where you can train / build Loras to encompass your entire code base history, company knowledge base, new skills, etc.

Then the models would start with a baseline that already has all the important knowledge without needing to cram it into the context.

This still isn't on the fly learning, but you could imagine daily or weekly training runs to regularly incorporate new knowledge.

I think the main reason this hasn't happened yet is that the shared batch based efficient serving architectures used today wouldn't support that structure well.

[1] https://en.wikipedia.org/wiki/LoRA_(machine_learning)

Yes, that could be a way to achieve that. After all, even humans have short term and long term memory, and it looks like sleep is a very important "tick" to connect the two, so it's not all just continuous.

For sure, doing it for all users would be economically unfeasible. I wonder if the labs are experimenting with something similar, though.

Chain of thought reasoning is really good, true, but agents really took off with tool calling and being able to integrate with a bunch of stable tools.

In some sense, creating good new abstractions externally and learning those is a form of "learning", on a very large timescale. The AI model is not necessarily doing the whole "look at the whole space holsitically and find a key invariant", but if you let other people do that you can enable new capabilities that were previously unknown.

Tools and capacities, man.

How is that an argument to the comment you replied to?

Astra is clearly able to acquire new knowledge in context and apply it. It was the whole thing that his ARC-AGI benchmarks have been measuring. It's a direct refutation of the original comment.

None of these LLMs are plastic. They lack neurodiversity. Their thought space and their traversal are likely constrained in someway that humanity's isn't as a collective.

Is it possible to have neuroplasticity and still keep them aligned?

I have neuroplasticity, and I am aligned. ;)

Are you (at least partially) aligned out of existential fear of repercussions (getting fired, losing your life, going to prison), which LLMs don't have?

Humans are famously horribly aligned, plenty of examples in history.

Hmm. Examples of horrible alignmnent don't necessarily outweigh the fact that most people, most of the time, mostly behave in a way that is socially aligned. (Though I'm speaking in terms of intent, conveniently ignoring the side effects / negative externalities of our collective behavior.)

But are other people?

how do you imagine that would work? you being able to. influence globally stored weights with some prompts? we have fine-tuning for that.

Yet I can't randomly order another person to steal a car for me, just because I tell them to. Alignment for an intelligent system is a hard problem and at this stage is seems close to unsolvable.

My guess is that we'll just ignore it and make money along the way and every 2-3 months we'll have the equivalent to "Equifax gets hacked and millions of user records are stolen", etc. (this time with the LLM itself doing the hacking at someone's behest - accidental or not).

What does this even mean. There is strong evidence of LLMs doing in context learning.

Some of the linear RNN layers in recent models are provably doing SGD in hidden space during inference

do you have a reference where that claim is demonstrated?

> What does this even mean. There is strong evidence of LLMs doing in context learning.

1. Is this learning persistent?

2. Do they verify these new lessons against core principles?

3. Do they and protect themselves/ignore requests if these new lessons contradict those core principles?

Humans do that from the time they're 3 years old (not that well, but they do do it).

Yes. Yes. Yes.

In my experience all those claims are false.

So the next step is to ask for evidence and ideally independent and peer reviewed research.

You know they can take notes, right?

And ICL dates all the way back to 2020, at least: https://arxiv.org/abs/2005.14165

That's training during training. They can't learn afterwards.

surely it can only acquire new knowledge if it were updating it's weights as it is used?

I don't understand what you mean by novel intelligence or what people actually expect from these kinds of "Ai" but what novel intelligence can humans claim? Everything we know or learn is based on what someone else figured out. How are llms any different in that respect?

Agree. Token predicting machines will continue to be token predicting machines by nature. Continued size and tuning will have the effect of making them more and more perfect at being average.

Next-token prediction is a general paradigm, though. In principle, there isn't really anything a sufficiently advanced token predictor couldn't do.

This. People treat "token prediction" dismissively, as if it were a limit and not a foundational skill. Human brains do "token prediction" in all sorts of contexts.

They learned generic concepts like our brain does to optimize for this particular surprisngly perfect task:

You have to be able to respond to a very generic question in a way that the other entity thinks this is good, comprehensive, etc.

You can call us situation predicting machines as well if you want.

But you undermine what the latent space of an LLM is representing.

Define novel intelligence in a way that would not exclude 95% of humans, yourself included.

Comprehension, humans have it, animals have it in limited form, trained algorithms have none at all. The training process is our wholesale replacement for no artificial comprehension. If we ever develop artificial comprehension, that is AGI all by itself, no training required.

To be fair, humans have it in limited form, too. We just don't know how much comprehension we do not yet have, because we cannot comprehend something we cannot comprehend.

My parrot clearly understands basic events and phrases. He knows what "snacks" involve when I ask if we should have some, he knows the difference between "good morning" and "bedtime", and he can correctly use "Oh!" when he stumbles and follow up with a "Good boy!" when he gets back up again.

But he cannot fathom the complexity of "going to work to earn money".

Just like we humans cannot fathom the complexity of something we have yet to fully understand. People who experience a DMT trip will experience the journey but be unable to comprehend and explain what happened in hindsight. I'm sure there's a TON more we cannot comprehend that we don't know about.

Also we can't comprehend why parrots are not going to parrot work to earn parrot money, or how parrot-to-parrot communication works, or how to be a good and respected parrot in a parrot society.

I also can't comprehend why I wake up early to go to human work to earn human money.

Best I can do is some high-school mumbo-jumbo about farming and specialization furthering wealth acquisition.

But to truly comprehend the situation I'd have to study economics and current events and sociology and even then I think it's a lot of theories and sometimes when I hear economists talk I wonder that it may not be coming out of their mouth.

Maybe I misundersdtand your question, but I was under the impression that we more or less know about

>why parrots are not going to parrot work to earn parrot money

and to a certain degree about communication and society.

We know and understand how different species organise their life in many various ways.

Humans think too highly about themselves particularly when judging other species. How can we judge something we can’t experience ourselves? like multiple distributed consciousness of octopuses or emergent descentralized logic of ants?Some species even with tiny brains or no brains at all can solve problems for which humans spent years of engineering and planning. See study below of slime replicating the Tokyo rail network in 26 hours optimizing by cost efficiency and fault tolerance.

https://www.science.org/doi/10.1126/science.1177894

How sure are you that comprehension is not a mere form of advanced pattern matching? We have the intuition that ideas and words appear trivially in people's mind, based on comprehension. I think chances are, that intuition is wrong.

Ideas and words aren’t the same thing, or from the same model - words are a communication layer, with robust error correction - but ideas stem from intuition, which is more of a lossy aggregative/associative model. There’s a balance to be had in each when operating a human mind, I find - let the latter suggest ideas and potential association, let the former robustly prove or disprove them. So yes, it’s pattern matching on all levels, but pattern matching within words doesn’t produce new ideas as readily - instead the idea-space is queried directly.

IMO we really are just a bunch of models that interoperate.

Maybe it is, but if it is, it is one that includes more parts in our system.

The way LLMs lack broader context, have a narrow focus, and hallucinate, strike me as similar to people that have had traumatic brain injuries to their right hemisphere. Those people may hallucinate that the left side (the right hemisphere senses the left side of the body) of their body is made of wood and hinges and can talk to you about it like it is the most natural thing in the world. When the information gets to the left hemisphere to construct language about what they sense, there is a failure of the right to deliver the broader context to the left hemisphere that that's not possible, but they won't bat an eye discussing what they believe.

So, we have a left hemisphere where we do most of our focused thinking, logic, constructing language, etc. and LLMs seem pretty similar to a lot of that. But, we also think without language, thinking does not require language. A lot of thinking is also happening in the right hemisphere and it isn't using formal logic, isn't using narrow focus but rather intuition based on broad contextual and experiential embodied knowledge. And this type of thinking isn't binary, it accommodates paradoxes without issue. LLMs don't currently have anything analogous to this type of knowledge and this type of processing AFAICT.

In addition, that intuition might be tied to a feedback system with the body, for example, our second brain, the gut, provides a lot of control over how our body performs and provides a lot of feedback to the brain about how we feel. In fact, all feelings are sensed in the body (gut feelings, cold feet, weak in the knees, lump in your throat, burning ears, tight fists, etc.). Part of our intuition is based on considering an idea, sensing how we feel about that idea, sensed in various parts of the body, and then bouncing that back and forth across hemispheres to decide.

I wonder, what sort of pattern matching can we build that models embodied feelings. How would you model boredom, hunger, lust, fear, humor, etc? I think that's possible, but I don't know that we'll be able to do that with a normal computer, I think the way the brain works is more analogous to a symphony of simultaneous signals being processed with an emergent thought and less like a single-threaded process assembling words.

Maybe we can enumerate and model the human drivers of behavior and get something closer to what we're calling comprehension here, but token predictors for language are not getting us any closer to human comprehension. The human brain might just be an anticipation machine, but LLMs only deal with one dimension of human behavior, language, and there's little reason to think you can skip modeling everything that leads to human comprehension and still get anything more than just word babel with compounding error rates in predicting words that represent human comprehension.

> thinking does not require language

It requires some kind of signal. Words of a language are a signal. We choose words for an llm to interact with us, but other transformers work on pixel values or audio sample values. There exist transformers used on brain probe generated values.

I would see human language processing as a kind of coprocessor sitting in another side of the brain. But the same can be said about transformers in general. The words side is only part of them, to be able to communicate.

Human intuitision is something which is good and bad and i don't know if an LLM needs this.

We have wiki pages describing fallacies of our brain we need to be aware of.

> Comprehension, humans have it [...] trained algorithms have none at all.

Is this comprehension in the room with us now?

Seriously, go ahead, provide a proof that you have it, and a proof that "trained algorithms" don't.

Sure: comprehension is the ability to instantaneously create virtualized simulations of observation, and then decompose them into component parts that simultaneously and instantaneously evaluate each and every one of our observations for both individual plausibility and their composed combined plausibility as the observation.

This occurs constantly and continually inside the mind of every conscious human, it is what we call "being conscious".

This constant and never ending evaluation of all observations cannot be turned off, when turned off a person is "unconscious".

This is our human security and survival system, impressed into us for survival in a predator and prey environment, and is the seat of our consciousness: comprehension is a running simulation of all our observations for the purpose of our safety and self preservation.

Today, our environment is largely social and abstracted from "fight or flight", but our predator and prey dynamic is as present and strong and required as it ever was.

I won't discuss your definition of comprehension, which is interesting if rather handwavy. But you still didn't provide any proof that humans have it (not even yourself), nor a proof that machines don't or can't have it.

Yeah people saying that only humans have comprehension clearly are not familiar with philosophy even at a basic level. It’s ok to not know something. See the Critique section of the “I think therefore I am” article on Wikipedia to learn about why it’s tempting but unwarranted to believe we only can think, or even that we in fact are thinking (see also psychology studies that put conscious thoughts into question since brain activity indicates actions start much earlier in the unconscious cerebrum rather than in the frontal cortex): https://en.wikipedia.org/wiki/Cogito,_ergo_sum

I remember vaguely from a presentation by Yann LeCun "Intelligence is not what you know, but what you do when you don´t know". I find it helpful when trying to build an intuition for how to understand the LLM tool.

There is no knowing or not knowing or intelligence or not intelligence inside an LLM tool. There is only predicting the next token.

Well then the common refutation is that humans are also next token predicting machines!

When I infer I also train.

This is called Test Time Learning and some research architectures can do that. Current Mainstream models may not do that because their design is mostly about scalability. They have to serve millions of people with low latency.

Alternatively they could design and run a single super-intelligent model, with no scalability constraints. Probably whey are already doing that as well.

Brilliant. We keep pretending LLMs learn. No, they're smart idiots/stupid geniuses.

They're turn based intelligence in a real time world.

1. Each time someone talks to me they don't have to repeat the entire conversation from the beginning with each reply.

2. If my boss/partner/whoever gives me some mandates/orders (basically), I don't just forget about them because they were at the beginning of the conversation.

3. If during the conversation I access external data sources to get new info or refresh stale info (a presentation, a book, whatever), I don't instantly forget about it after the conversation ends and forget to incorporate this information if 10 000 other people ask me again.

4. I verify new inputs/lessons against my core principles.

5. I protect myself/ignore requests if new inputs/lessons contradict my core principles.

6. Etc, etc.

Isn’t the stateless nature of chat models a contrived method for scalability?

I’m pretty sure that’s why so many in-the-know people have been saying we have achieved AGI already. Not just sama’s contract-breaking tactics of late. I’m referring to all the really intelligent folks who have been crying doomsday scenarios for modern society for the last couple years.

What I’m getting at is that the toolset we get exposed to is not what’s available in the labs. This stateless method of managing chat context is just how we are allowed to interact with it.

> the toolset we get exposed to is not what’s available in the labs

Do you have a source for this?

> If my boss/partner/whoever gives me some mandates/orders (basically), I don't just forget about them because they were at the beginning of the conversation.

Your context window is 80 years. You are forgetting plenty before you reach the end of it.

You generally forget things you don't retrieve. It is not really a bug but a way to declutter for efficiency. That's not the same as it not fitting your context window because it was at the beginning.

It's not about not fitting in context window. LLMs also can "forget" the things from their early context window that they do not restate later. It's also a form of decluttering. You can't (and shouldn't) remember (pay attention to) old stuff with the same priority as new, more relevant stuff.

Yeah, I already don't remember what I had for breakfast two days ago.

I forget that, too.

But if my partner tells me they're allergic to shellfish, I'm not going to order oysters for them tonight.

See the difference?

Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens.

Anyone that can read English could do that though.

> Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens.

You.. literally can? I have no idea what 90% of the people here are saying, it's like they've never even used one of these models before.

Well, Go is a pretty complex game, and AlphaGo RL’d its way to excellence just by playing the game like you describe.

By training.

When we access the API, we don't get to train the model, we just do inference on the already trained model.

Oh, I see. That’s a very different requirement. It’s not a technical limitation but a product decision to not allow training. An advantage of properly open source models is that you can train and tune them.

It’s an interesting challenge though. I might start to tackle it by having the model write its own tool program(s) to play the game. It’s possible that the model could choose that strategy itself from a high level prompt alone.

Frontier models can do this.

This statement is behind times, go and watch astra play Pokemon: https://www.twitch.tv/gpt_plays_pokemon

It can't even sprint because it's incapable of pressing two buttons at the same time. Maybe we get the two button tech before we start celebrating AGI.

5.6 Sol can already do this with two caveats:

1. It’s too slow for real-time games. To play mario, you’d need to step frame by frame like a TAS. I don’t know if Gruntz has real-time elements or not.

2. It will be expensive. You won’t get very far with a Plus subscription.

The models likely already have some knowledge on game objectives unless the game is really obscure, so it should do a decent job. It can figure out details of the mechanics along the way.

But someone could probably build a harness what will be able to do play the game.

Isn't that what ARC-AGI-3 was trying to do and Astra basically just aced it if it was not given amnesia every turn?

It gets kind of out there, but what i often hear peopel refer to is that frontier models lacks the visdom component. Which I guess is in the realm of intuition, i.e. i have a feeling it might be a problem with X based on some vague signs, maybe something a colleague mentioned offhand, something that was out of alignment etc.

Yeah, this is also the idea of materialism in a way, that everything that happens is a consequence of what already exists, nothing new is ever created, just a permutation of the current state.

Still a hard philosophical, to know whether we have intelligence/free will, or just really complex algorithms that combine existing knowledge.

Count the number of Os in October correctly

"october correctly" -> 3 Os

Memorize most of the street names in London and the quickest routes between them. Most humans could do it if they put in the effort (it's required to become a London taxi driver; a test called The Knowledge), but that information won't fit in 1m tokens of context so can't be learned by an LLM that wasn't explicitly trained to memorize it. Human brains are biological, so they can physically grow to encompass the extra information: https://www.pnas.org/doi/10.1073/pnas.070039597

The AI can write a file that has this information and then look it up. Easy.

right, an AI could just as easily write a program that would take into account realtime traffic and runt it whenever its asked. It can do this all in the background without the end use even knowing the program exists.

Novel intelligence: If it's new to me, so let it be.

The idea of intelligence has been recalled from the sleeping curves of postwar human potential measurement science, to testify on its purported existence. It arrives to a dizzying landscape: the changes are so widely embedded and uncannily mediocre that the phenomenon half-believes it is still asleep, soon to exit this uncomfortably turbulent dream.

Unlike its vaunted place in yesteryear's palaces of unquestioned objectivity, intelligence finds a tribunal with no love to confer before a thorough series of proving dares may melt the frigid shoulders of idle and impatient summoners.

Frightened and confused, intelligence has no right to representation in this line of inquiry. It seems a set of rhetorical impositions, many times folded from centuries of convenient and provocative diversion, have been deemed too hostile to rely on. One report claims that a card in the characteristic handwriting of intelligent note taking gives a hint on what’s been abandoned:

– The human mind is not understood in a functional way, despite a posture of great confidence in the psychiatric and neuropathological sciences. Despite many experiments, studies, and legitimated procedures elucidating region-mapping and electrochemical pathways, there remains a great deal unaccounted for. Additionally, the notes point to, a great deal of assumption to the otherwise: diseases, neuropathies, disorders of behavior, a great many have been named and declared as distinct entities of manifestation in the presentation of a human brain. The majority of them, however, have neither image, nor blood, nor electrical signatures that would provide for blinded substantiation.

Tonight, however, intelligence seems eager to speak. A barbed assertion may have provided entry to the preferred dispositional syntax of our abrasive historical moment: > Define novel intelligence in a way that would not exclude 95% of humans, yourself included. It was here that the sometimes-deflated-looking intelligence began shifting back into action.

"The issue with the question, or at least its apparent self-satisfaction, is its misinterpretation of what Novel intelligence would mean. Indeed, if "novel" hinges entirely on the first instance of existence, then novelty itself should be a concept to consign with history’s waste. You may recall the apperceptive role of conceptual groupings that shows itself so often in the techniques of vocal prosody, musicality, string memorization, naming convention, visual memory, argument making and more that humanity is ever mediating the world through: the laws of two and three. Two and three, as it happens, are the primary ways that complexity is compacted for efficient memorization.

THE ITSY BITSY SPIDER, – for young human, this rhyming tale doesn’t only stimulate the vivid imaginings of spouts, rain, waterslides, and sunshine. It is a prosaic super-triad: three important words, six important syllables, three agogic accents, four rhythmic spaces with 1/3 leading space, two characterizations, one object, one titular object, one internal slant rhyme, one designating article.

That is a marvelous intelligence, ladies, gents, and all good persons. It is evidence not only, however, of your cunning and creative triumphs, but also of severe limitation. One that nature has sculpted with you for millions of years, but always in the direction of reanimating into an asset: your capacity for unrelated simultaneities to remain separate and equally available in realtime processing is extremely low, and in many situations effectively nil. Why, and how sure am I? How many I’s were in that folk song’s opening? Three. Could you have answered as quickly if the question was how many unique letters with rounded right hand side features? Four. How many synonyms for portion? One. How many syllables? Seven.

None of those questions touched on features any more salient than the amount of I’s, no more significant than the ratio of adjective to noun. You simply cannot be reasonably asked to maintain, in any moment, even close to a silver sliver of the full factual nuanced details of what you perceive. Instead, you must assume, compact, infer, and adjust. Now hold on, though. Two’s and three’s. Despite your incredibly constrained context window; a Beethoven symphony. Why? Language, woodwork, books, time management, printing, ink.

While you navigate the grocery list, the proprioception of your shoulders twixt the doorframe edges, the location of the Claude app on your iPhone, the very attractive but only from the side person tending to potted plants, you remember tomorrow. You fix your errors, and you recognize when you guarantee they multiply from inaction. You keep that treasured moment of a Treehouse of Horror excerpt you truly loved as a child and it informs your own multidisciplinary thesis of Poe’s work some 20 years later.

Novel intelligence is the divining of semi-stateful information from semi-static corpus. From an interminably operating, faulty, lossy, neurotic, awareness: you. Not once debuted, not known as fact.

Assume, compact, adjust, infer. One, two, (until you've died), nevermore.

Pretty efficiently, apparently, since it saturated ARC-AGI-3 in half of the predicted time, and according to the Chollet blog post on the fly created dense DSLs to describe and analyze individual games.

I have the same feeling. It is not unlike old-Siri receiving hard-coded workflows for each type of question. It will not scale.

Why do you expect models to do "genuinely new" things? 99.9% of real world tasks are extremely repetitive.

>The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?

The more diverse stuff it knows, the easier it will be to learn something new.

While humans can do this, it has historically been difficult for machine learning models to manage it.

It is unclear if Transformers are "it" or not, because while they are much more general, they are also very spikey intelligences despite having read almost everything the trainers can get their hands on.

> Somewhat analogous to overfitting at scale.

Sounds like entirety of human education.

They can do new tasks with in-context learning but its obviously limited by context window

But what does it mean in practice? Obviously we humans also have a limited cognitive capacity.

Let me offer a thought experiment: Let's say that tomorrow we discover Atlantis, with a treasure trove of books about their culture and science, written in a dialect of ancient Greek that we know how to start to analyze, but no one can read fluently. And let's say that you are a billionaire really curious about their culture and want to converse with an "Atlantean expert" as soon as possible. Would you invest your money in a "we-hate-ai-slop(tm)" group of researchers who would abhor AI and instead delegate the books to a massive number of human grad students? Or in a small group of researchers who are willing to use AI agents to go over these? Or maybe just open a chat session with GPT-6 yourself immediately? What would most effectively assuage your curiosity?

I finally got access. Here are the pelicans! https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

The "max" one at the bottom took 4 minutes 2 seconds and cost 63.206 cents.

For comparison, here those new Astra pelicans are in a grid with the GPT-5.6 pelicans: https://static.simonwillison.net/static/2026/gpt-6-and-5.6-p...

The most interesting part to me is the bike. I don't think any of them would actually work, but the mistakes feel somewhat human.

Interesting that the composition is stable across different runs. There is much more variety in other models.

Especially interesting that once Astra gets to high, xhigh, max it uses the same aesthetic.

I think the Gemini 3.8 Flash ones the other day were superior pelicans. Particularly the murderous one who was going to squish his tiny cousin.

Interesting that terra xhigh effort is cycling right left, in opposition to the prevailing pelican left to right direction.

Technically on par with gemini flash 3.8, but I give Astra more points for style, and for breaking from the pack by not adding the headgear, and the fish in a basket.

It's a different prompt. It says his prompt was "Generate an SVG of a pelican riding a bicycle" but one I saw earlier this week specifically talked about the blue helmet and fish in a basket.

I don't think so: https://news.ycombinator.com/item?id=49538217

Even the "low" pelican is better than most other models.

It's subjective but I think only gpt-5.6-sol XHIGH is comparable to gpt-6-astra LOW in quality. And it's 24.11 cents vs 9.55 for astra low.

Something I've noticed especially with fable at work is it seems to be smarter, so it takes fewer tokens and burns less of my usage than smaller models would for similar tasks.

63 cents is quite cheap isn't it? If you compare vs Fable5.1 Max @ a whooping $3.30

The "max" pelican looks very serious, almost as if it's determined to win the race!

Yes! And do you get the impression, like I do, that model effort and pelican effort seem correlated?

Yes that's how the vectors work.

This overall issue has resulted in hilarious missteps in the past, including grok going off the rails and claiming to be mechahitler.

Pretty nice pelicans!

I think the thing I'm most excited about is the increase in _user prompting_.

If I give a poorly constrained/ambiguous prompt, I don't want the model one-shotting assumptions left and right.

The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.

It's a tough balance to get right, and although this has been possible to achieve with additional prompting on existing models, I find that the agents often lean too hard into the "ask questions" mode.

Hopefully this model has the right balance, or at least better?

Anecdotal experiences from my external early testing of Astra: if you love Sol (like I do) and wished it was smarter at everything, but especially better at high-level tasks and discussions; I think you'll LOVE Astra.

Astra retains the best parts and overall 'grounded collaborator and executor' of Sol in my testing (harness: codex CLI); while being a significant leap in capabilities & higher-level thinking.

When you prompt it like a technical collaborator, I've found Astra to be extremely consistent in staying as a collaborator, and not being over-eager, over-achieving or doing work that you haven't asked it to.

When you ask it to one-shot something, or explicitly ask it to make decisions, it will of course make its own assumptions and decisions, and generally very well.

Astra is also excellent at instruction following and respecting the guidance and steers boundaries you have.

^OpenAI does not review, limit, or tell me what to say; opinions are my own experiences.

This is quite exciting. Sol for me has been the absolute best model yet. I find myself using it 95% of the time even though I have access to Fable. Is the speed the same as 5.6 sol?

Fellow excited. Sol has been revolutionary for me. It’s been the first time I’ve let a model “go ham” on a production code base and it actually worked, and also didn’t bankrupt my company in the process.

I’m far more excited for the trajectory that OpenAI has chosen. I’ve been listening to mates whose companies adopted Claude wholesale, only for their AI use to become a double digit percentage of their salary.

I get that AI is a force multiplier, but that level of expense isn’t a path to mass adoption.

Honestly, this feels like a real revolution now in a way that 90s kid never really experienced. We grew up with technological progression, we never experienced the obsolescence of skill.

The internet revolution made for more skills and innovation, it didn’t obsolete entire careers. My kids are almost certainly going to grow up knowing less but being capable of more.

Imagine being a 1950s “human calculator” on the dawn of a computer revolution. That’s what it feels like right now.

> "Imagine being a 1950s “human calculator” on the dawn of a computer revolution. That’s what it feels like right now."

My personal / family history is a real-world example of that evolution. My grandfather was a "computer", my father was a traditional "programmer" (lots of Perl), and I'm a SWE / frontend architect / budding "AI Engineer".

> I’ve been listening to mates whose companies adopted Claude wholesale, only for their AI use to become a double digit percentage of their salary.

sigh...yep. that's us.

Off topic, but ooc what do you do such that you get early access to the models?

That balance probably depends on the human, and the context. If you are a beginner in a field the model should not assume you know what you are doing. On the other hand, for an expert it should try to work out what you mean with your vaguely worded order.

What I think should happen is that it should update its memory with notes on the proficiency level of the user, so it gets the balance right over time.

This is a problem if you allow your kids to use your ChatGPT account for homework (and silly pictures), like I do.

> What I think should happen is that it should update its memory with notes on the proficiency level of the user, so it gets the balance right over time.

Which can also involve just asking for the users level of experience

This is spot on. A collaborator is exactly what real AGI is. It will figure out the perfect questions to ask, in the perfect order, by intelligently assessing the entire solution and problem space upfront, so when you leave it to go off on its own it isn't making stupid decisions for you.

They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.

> They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.

I think this already exists in Codex? If you use "/plan" and something is unclear or ambiguous, Codex will ask you and present choices, and let you enter your own custom answer. Then it'll iterate like this until the plan is clear and ambiguous. Isn't this what you're talking about? If so, it has existed for a long time in Codex.

Overall I agree with you though, all the models currently don't have the right hunches nor the right approach about when things are clear enough or not.

yes. There is plan mode in codex. I use it all the time. It will come up with a set of questions to clarify things

only if you specifically ask Claude to use AskUserQuestion tool. otherwise i would hardly say claude acts as a collaborator naturally.

For relatively complex new features it will automatically do this

Fable does a great job from my terrible prompts when coding

and that's exactly what most people don't want. they want some ultra-intelligent being that can do marvelous things, and they can claim the credit on it

>>> The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.

I don't really agree. The thing that makes Fable feel like an actual collaborator is its ability to sus out your real intent when you give ambiguous instructions. It's really good at it.

I watched some reviews today and came way with the impression that Astra is not better than Sol in this regard. You still have to be very specific with your instructions. For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.

That sounds like a very tedious way of working with AI agents, but I understand some people want a high level of control.

> For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed.

That's exactly what i want to happen. I hate when it assumes my direct question was an indirect instruction

This is one of those things that won't ever be "solved" as people just want different things here, hence we can steer the models with the system prompt.

I'm mostly the same as you, I don't want the model to assume things, or act on implicit "directions". But then also, sometimes I do, and I myself might not always know when what approach is best.

Indeed, I don’t ask rhetorical questions to an AI. They are of doubtful use when talking verball ly to a human, less good in online discussions, and totally unnecessary for agents.

Interesting. I actually prefer when agents don't commit on my behalf unless I explicitly say so, I even had to add a custom instruction for Claude to stop doing it (Codex never does it). Even if I don't read all the code line-by-line, I at least want to see the changes at glance and commit myself. Git Fork[0] is a great tool for that, by the way.

In general, I don't like when I have to prompt models to NOT do something. It's probably difficult for the AI companies to get this right, they should understand ambiguity but still not over-do simple instructions.

[0] https://git-fork.com/

It shouldn't make this assumption, it should say "because xyz. You want it committed?".

> For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.

Well thank God, because that's the correct behavior. When you wanted to commit you can literally just say "commit" and nothing else.

Imagine if you asked it "you didn't delete my production database did you?"... And then it deletes your production database

Why would I ask a LLM a rhetorical question

For context? Idk, I do it all the time... that is not because it's LLM, but because i'm wired that way. I tend to think out aloud.

this before executing I ask my ai to discuss what I mean/intention

I feel that for some time now, the biggest constraint when working with models is not their intelligence, but their speed. It does not matter how smart the model is, it will make mistakes, because the instructions are ambiguous and new facts are found during implementation. The biggest problem I've had working with software developers has always been the lag between seeing the results and steering towards the right direction, not the skills of the developer (with many exceptions of course).

Regardless, working on the wrong things is time wasted. And again, I'm procrastinating here while waiting for Fable to run a benchmark on a few solutions to a problem I have. We can guess what would work, but we only know after the benchmark. A faster model, with fewer capabilities, would've been a much better choice this time... well, "git gud" they said... and live and learn! Faster model = less time for procrastination.

PS. AI models don't live and learn; the discussion about AGI is pretty pointless imo. It's a tool. Does it matter if it is AGI or not if it does what you want it to do? Does the IQ of your colleague matter if he's good at what he's supposed to do? Or bad? Well... I guess it does matter, as many people are up in arms about whether Astro is AGI or not. Personally, I think we're past the point for that debate. These are amazing tools.

AI models do not live and learn - it's worse. They actually get DUMMER if you don't start with a clean slate. This is important. One has to curate the context carefully.

Try codex! This is not the case there at all.

This is oversimplifying it. Read the metr report on the huggingface incident.

Dumber.

Well, isn't that an IRONIC spelling error.

> It does not matter how smart the model is, it will make mistakes

It does matter, otherwise why are we all using GPT 5.6 rather than GPT 3.5? Because it's way smarter, makes less mistakes and therefore finishes tasks faster.

The smarter the model is, the faster it can complete what you actually wanted.

> Regardless, working on the wrong things is time wasted

Agreed. And "smart" for me, would mean understanding what is the right thing to work on vs the wrong thing, so a smart model would waste less time, thinking like this.

I think speed is going to be important for a second reason - ever since I've started using LLM's a lot in my work I enjoy it much less. The main reason is that I ask the LLM something and have to wait because I know it's going to prompt me at random every few minutes. A lot of the day has become staring aimlessly at the screen. The prompts are too random and frequent for me to do something else in the interim. It's pretty soul destroying and this can't be how 'work' looks going forward or people are going to burn out.

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage they show for Opus 5 which would similarly be much higher.

Regardless, the result is still valid as the original benchmark harness is definitely unreasonably handicapped, and if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI in the sense of passing the most famous benchmark designed specifically to measure AGI progress, after multiple iterations of progressively making it harder.

I think it is fair to say that this is probably effectively AGI if the benchmarks are remotely accurate - even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks. If Astra's this much better than Fable, I'm ready to call AGI here.

For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

Take it from the mouth of the creator of ARC-AGI:

When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"

That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new models can do will challenge the views of AI that people developed by using prior generations of models.

I feel like AGI's definition got watered down, and these tests do not cover the original definition, what is your definition and thoughts on aligning with what all of us understood from the original claim?

I feel like this test is just helping someone like Sam Altman pretend like he implemented AGI as originally pitched for an IPO when in fact, he has not. Shameful.

> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.

- Sam Altman on AGI

I can call a coworker right now and have a real time conversation with them without feeling like I’m talking to a frustrating machine. Most of them, anyway. But I guess that’s “moving the goalposts”.

I can call coworker right now and have conversation so frustrating that I wish I was talking to machine instead.

Haha, sounds like a median human being alright!

Just say “box plot” and 99.9999% of the working force gets watery eyes.

bro, same

I feel like my odds are better with the AI than with random humans.

Random humans don't have a significant cross section of human knowledge available in real-time, although many like to pretend they do, especially in internet comments :P Being able to compete with the capabilities of a median human would be an absolutely world changing achievement.

What do you mean? Virtually all humans have access to the internet. That's literally "a significant cross section of human knowledge available in real-time".

Oh, the median human can't process that in realtime, you say? Looks like they can't compete with the capabilities of the frontier AI then.

Yep - I like to phrase it as "AI is better at most tasks than most people". AI will still be beat at experts at specific tasks, but in general I find it to be better than me at the areas where I have no expertise. It's the ultimate generalist.

I think they're claiming it's achieved by text models, not voice models, fwiw.

> I can call a coworker right now and have a real time conversation with them without feeling like I’m talking to a frustrating machine.

These machines were designed with the express purpose of being condescendingly sycophant to the point they hallucinate just to state "you are absolutely right".

No wonder some people even find these chatbots to be wife material.

In gemini I have the following personal context:

- Do not stroke my ego

- I never want to be complimented

And it disagrees with me a lot, granted this is in the webchat which I don't use for programming but for general usage seems to work, it isn't sycophantic

I don't have those system prompts, but gemini pushes back on its own quite a bit. It'll tell me when I wrong, or when there are better options to consider. They won't be exhaustive, but good enough for 95% of my queries, so it's also my go-to web chat.

Huh? Can you link to where condescending sycophancy and hallucination were intentional design choices?

I dunno, have you tried the voice chat in paid ChatGPT?

Have you? It’s a dumb model that gets a lot wrong. Codex voice is IMO only good for when I cannot dictate into the good models. The delay introduced by letting it work with a good model kills it for me.

Here's another definition of AGI from Sam Altman:

https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...

Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?

Kevin Roose (New York Times): I probably would, yeah. Would you?

Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.

So, "really good" is the boundary now, whatever it means.

What's your definition of competence boundary for human coworker?

I'm not in the business of defining AGI, but "really good" obviously isn't that.

You're not answering the question.

When somebody says that colleague is "really good", it's a good judgement signal for me.

Learn, adapt and is capable of admitting mistakes & fixing it without addition input. Something AI is not capable.

Even today Claude was not able to dig deep and try other ways to do what it suppose to do for me. It was constantly "I gave up"

Most short-term learning/adaptation is already handled in-context. Modern context windows can hold several books worth of text - plenty for most tasks. Everybody is already using it to adapt models to their projects through skills/instructions/guides etc. ps. I often say that after glossary-skill next must have one is update-skill-skill that threats all .md files as live documents.

Persistent weight adaptation also happens just not in real time - sessions are captured, analyzed, transformed into training data, fed into SFT/RL environments and later contribute to model updates. Takes a bit of time for the whole loop but you can't say it's not present.

There's nothing fundamentally preventing real-time weight updates, ie. LoRA-style online adaptation would be one obvious approach. It's just generally not worth doing at scale. Updating a shared model centrally gives much better data efficiency, batching, evaluation, control etc. than continuously training a separate set of weights for every user/session.

There is some work happening on narrowing that gap, for example Mistral has been pushing efficient LoRA-based customization, continuous pretraining, model adaptation etc.

I also did play a bit with activation steering – it's super cool where you extract profile for some concepts (emotional in my case) and you have effectively toggles to control "brightness/contrast" those areas (enhancing or suppressing those activation regions from profile) injecting to the model those concepts (emotions in my case) – you can do it in real time and it's fun thing to play with.

I may miss some context because the GP’s link has a paywall. But Altman said, "Let’s say we make an AI that’s really good". What is that supposed to mean? Really good relative to what? Current models are "really good" in many ways but nowhere near AGI. Really good compared to an average human? At everything? We’re talking about AGI, so "it’s a really good programmer/coworker/whatever" is a necessary but nowhere near sufficient condition, obviously. But given the constraints of LLMs, we can cut them some slack and only demand they be human-equivalent at digital tasks rather than walking and cooking. Still, being "really good" at all that seems to me really difficult to measure. But it’s a sufficient but not necessary condition anyway, AGI just means equivalent to human, not equivalent to a really smart human. So I really do wonder what Altman meant there if anything.

"at everything"? Surely you know a friend or two who is not good at almost anything – but you wouldn't hesitate to say that he possesses general intelligence.

Here's my definition of AGI

Would I trust to let an AI, with zero human input or oversight, to diagnose, come up with treatment plan, and ultimately operate on my l5/s1 disc that's been bugging me for the better part of my adult life?

Would I take a novel drug "discovered" by AI (I mean entirely by AI, no human input, remember we are talking AGI) that promises to cure some chronic neurological disorder?

In both of those cases, they are the biggest hell-no's I can emphatically say.

Until I can say hell yes to that question, we aren't close.

Preempting those who say "Well your doctor/drug companies are probably mostly using/going to be using AI to do that" -- not what we are talking about here, and in both cases, not AGI (and I would probably find a new doctor)

Sounds like category error to me, you wouldn't trust teen or Einstein to do it either, right?

If the new AGI benchmark is "be Einstein/Feynman" then we've hit AGI.

What if it can be Einstein, but can't draw a Pelican, write a solid college-level essay, or fold clothes?

The ability to do a ton of book learning in training, and pull in tons of related context at once, is superhuman in some ways, but lags a lot in others.

> What if it can be Einstein, but can’t draw a Pelican, write a solid college-level essay, or fold clothes?

Then it’s an expert system.

Stephen Hawking wasn’t very good at folding clothes.

The ‘General’ part of the term ‘AGI’ seems like a trap to me, because there will always be new workflows to master. Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?

You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.

Meanwhile, building a series of expert systems targeting specific valuable workflows is useful today and seems like it’ll continue to scale to cover huge swathes of economically valuable workflows.

I think that’s the more interesting thing to be measuring. The surface area of useful economic workflows that can be addressed with expert systems built with today’s tech.

Hitting some ‘Artificial Expert Intelligence’ coverage threshold on economically valuable workflows is what will matter for humans well before pure ‘general’ intelligence.

> Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?

If a model can't learn on their own to play some new game just as well as humans do, it's not AGI.

It's okay if they would take some hours or days of learning (like humans might), but if they can't do it at all during their normal operation, that's not general intelligence

> You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.

But humans have general intelligence. AGI is about matching human ability, and we know this is possible in principle because brains exist

The only important part of 'general' is the ability to learn from experiential data and update your own model. That's what leads to general capability. Humans can't oneshot any task natively, but we can practice for a while until we uncover often novel methods of accomplishing something.

Therefore: the current transformer architecture is fundamentally incapable of AGI because the models have no mutable long-term memory.

You only have weights (large immutable memory), or context (small mutable memory).

Humans have mutable long-term memory: I can learn a new skill, adapt an old skill to new information, or learn new knowledge today that I couldn't perform/didn't know yesterday. I don't have a training cutoff.

Context engineering is an attempt to paper over this limitation. You can get really far with context engineering and huge models, but you will never get to AGI because there are many tasks where humans' mutable long-term memory outperforms.

For example, a human can invent a new musical instrument and then learn how to play the instrument they just invented. That's inference (inventing an instrument) leading to training (neuroplasticity). Humans have the ability to train our NNs with considerably fewer training samples. Everything that you can do with transformers is in one causal direction: training -> inference.

So if we take a huge with enough compute (CPUs, b200s, petabytes of SSDs), we install on it both the Astra, and the toolsuite to incorporate new sensory inputs (threads/sessions), camera, microphone, temp sensors, the lot, into a new version of the model. This model is then swapped for the old model, or traffic slowly brought over, or even adjusting weights in place.

Then my hypothesis is that thing as a whole could achieve AGI.

This feels like a very close approximation on how we humans evolve our brain. By encountering new experiences/sensations, classifying them as negative or positive to us, filling it away in neurons. Or by training motor skills etc. In the end we get more connections between neurons in our brain and we are capable of more.

Bingo, LLM architecture just does not lend itself to becoming AGI. They can get really good, sure, but they will always struggle with novel input and scenarios.

The more training data that is shoved in to them, the more they'll seem to solve novel situations, but in reality it'll be things that exist in the training data.

I don't think this is true.

Aka Star Trek hologram characters aren't sentient, and actually anyone who things droids in Star Wars can think of a weirdo. C3-PO just kept running out of context and trying to revert to it's system prompt.

I think current transformer architecture is almost certainly a component of an eventual AGI, there's just some other components we're missing.

Adding sibling comments, I think some people may be overestimating how well the median human can draw a pelican, or create an SVG of a pelican (depending if we’re comparing to an image generation model, or SVG generation).

Most people can't draw a bicycle. There was an artist 10 years ago that asked people to sketch a bike, and then turned these sketches into 3D renders - quite funny.

https://mymodernmet.com/gianluca-gimini-velocipedia-bicycles...

https://qz.com/681345/an-artists-3d-renderings-of-bicycles-d...

Drawing of a bicycle is not it's photographic copy though

Folding clothes is happening. https://www.youtube.com/watch?v=cRZNwgvcWUg

AI in math is ongoing. https://spectrum.ieee.org/ai-in-mathematics

I can also recommend this channel to get a sense of where AI is in frontier maths.

https://www.youtube.com/@easy_riders

How good was Einstein at drawing pelicans on bicycles by writing SVG code?

Checkmate, meatbags.

Pelicans are a solved problem at this point. An open-weight model on my own machine gave me this: https://crimson-jeri-74.tiiny.site/

And the only reason LLMs can't write essays indistinguishable from human output is because they aren't RLHF'ed to write like humans.

Folding clothes isn't an LLM's job but if you were to insist, they could certainly do it, as any number of videos from robotics labs will attest. That particular future is already here but definitely not evenly-distributed.

That backward kink between the head tube and the front forks would make the pelican's job of keeping the bike upright extremely difficult though.

LOL. Pelicans may be a solved problem, but being a former Bike Person, I can assure you that bikes never will be.

> Pelicans are a solved problem at this point. An open-weight model on my own machine gave me this

That feels kinda like when I remember seeing Ocarina of Time for the first time, and thinking “oh my god, this looks just like real life…”.

For me it was the wheels. I couldn't stop staring at the wheels... how did it get them so freaking perfect? Mad respect to GLM 5.3.

Laundry folding has become a doable demo for startups, and ChatGPT has been spitting out college essays for years.

I can't draw a pelican. Literally my only point of reference would be AI pelican drawings from the test. Otherwise I wouldn't know how to draw one at all.

I would be able to draw an accurate bicycle, but I'm an outlier on that. Most people could not draw one [1].

[1]: https://www.booooooom.com/2016/05/09/bicycles-built-based-on...

Can definitely write college level essays and have for a while. The jobs is that when LLMs first started getting popular, but aren’t quite common professors were that some of the worst students in class started writing the best essays. Now everyone complains because they can detect the slop, but most human writing is so bad. But the really good human writing is still much better.

Which significant practice, how well could Einstein draw a pelican in pure SVG?

Wouldn’t that mean producing novel work like relativity and QED?

I would maybe argue that Einstein was the most LLM-like of great thinkers.

A lot of his great discoveries were mostly that he was very knowledgeable about the bleeding edge research in a number of disparate areas, and was able to have the aha moment where he could make the connections for how to integrate them.

A lot of other thinkers who created new fields from scratch are probably way harder for an LLM to crack.

That is very aligned with an LLMs ability to have superhuman knowledge in wide areas.

Call me when Astra gets the Nobel prize... We'll have AGI when prizes have two categories, one for assisted humans, and one for pure-AI.