Declaring victory without a definition simply muddies the waters
By Gary Marcus — via Substack
A couple hours ago Jensen Huang declared that the race to AGI is over.
Unfortunately, Huang gave no evidence and no definitions, which feels to me like an effort at a takeover of a scientific question by corporate fiat.
I would urge him to read agidefinition.AI by Hendrycks, Yoshua Bengio & many others (including myself), and to consider how Astra is doing on the kinds of examples I laid out in my 10 -item bet with Brundage. Autoformalization may finally be in reach, and maybe (?) reliable coding; I doubt that Astra will have hit any of the other eight.
By conventional definitions, Astra still falls short.
Declaring victory without a definition simply muddies the waters.
§
Aside from the lack of definitions, I would expect that if Astra really were AGI, it would be a quantum leap ahead of its competitors. Instead, many see it as not much more than on a par with Fable 5.1 in real-world applications:
My current view is that GPT 6 Astra is not meaningfully better than Fable 5.1 for my personal work, but that using both side-by-side is nonetheless very helpful and additive.
I have been using GPT 6 Astra and Fable 5.1 a bunch over the past two days, largely for policy analysis,…
— Peter Wildeford🇺🇸🚀 (@peterwildeford) · on X
And one well-respected set of benchmarks suggests that Astra is a genuine improvement but not significantly off-trend.
When real AGI arrives, we won’t need to squint our eyes.
And we won’t need Jensen’s approval, either. The results, at that point, will speak for themselves.
Update: One reader pointed to the ARC-AGI test as a criterion. Here’s what the inventor of ARC said about this:
Many of you will ask, “if it saturates ARC 3, is it AGI?”
We’re not making this claim. All we know about the system so far are its benchmark scores.
When we launched ARC 3, and in every presentation we made about it, we were very insistent on one thing: solving it is not proof …
— François Chollet (@fchollet) · on X
Update two: Although Jensen congratulated OpenAI today, about six months he already declared victory, pre-Astra, with respect to an earlier model.
