The loudest promises in AI tend to appear before the receipts do. This week, several receipts came due, and the math is more complicated than the evangelists imply.
But first: a little bit of drama.
The spiciest story in AI this week is that on the NYT All-In Podcast, Gavin Baker reported that several well-informed sources have stated that at Anthropic, Dario Amodei has reportedly said something along the lines of: …soon, Anthropic will be the only private company in the world.” For many, many interested parties (Elon Musk and Alex Karp (CEO of Palantir) leap ferociously to mind), these are fighting words. For broader humanity, it’s just scary. No bodegas, no florists, no plumbers, no landscapers: Anthropic is going to just gobble them up!
When word of this got out on Twitter (X) this past weekend, much ink (digital, of course) was spilled. Dario himself (our purported future Lord and Master, if his words are to be believed), got on Twitter (X) to deny the claims. You can read more about the back and forth, here: https://x.com/theallinpod/status/2088367978270142811
In other news, the NY Times reported that OpenAI shared that its Q2 revenue exhibited slower growth, compared with Anthropic. So maybe Dario’s claims are not so far-fetched after all? Right now, by some measures, the AI sector alone is larger than US GDP. The bubble just keeps bubbling…for now.
In partnership with
125,000+ podcasts, transcribed and searchable in minutes.
Particle transcribes 125,000+ podcasts where executives, officials, and analysts talk candidly and publicly, transcribed and queryable within minutes of airing.
Watch a company, a person, or a theme with the Particle API and get alerts whenever they're mentioned.
Particle is built by former Twitter and Tesla engineers, using an AI-native transcription pipeline that delivers high accuracy and extensive data enrichment.
the recursive dream (RSI) hits a wall.
The premise was seductive: build an AI smart enough to improve itself, and the rest of human history becomes mere pretext. Recursive self-improvement (RSI), the idea that AI systems could autonomously iterate on their own architectures and capabilities without meaningful human judgment or involvement, has been the animating fantasy behind some of the most breathless AI forecasting of the past decade. According to reporting by MIT Technology Review, the practical path toward RSI is proving far thornier than its champions anticipated.
The article identifies 4specific technical hurdles that current systems cannot cleanly resolve: the difficulty of AI systems to critically evaluate their own outputs, the absence of robust mechanisms for translating capability gains into one domain into general reasoning improvements, the compounding instability that arises when models are used to generate training data for successor models, and the fundamental challenge of specifying what "better" means when a system is designing its own objectives. Human judgement still has supremacy.
Recursive self-improvement (RSI) doesn’t just require a machine that can rewrite its own code. To be useful, RSI must articulate coherent and stable goals, evaluate its own progress toward those goals, and continuously improve its evaluation mechanism. Perhaps most importantly, it should do all of this without drifting into what researchers call specification gaming, tech speak for a system that finds clever ways to hit the stated goal while completely missing the point. Each of the steps in the process is challenging. But when we attempt to chain them together at scale, without human oversight at every juncture, we risk creating a mess that spirals out of our control quickly.
Researchers currently doing the actual engineering work are increasingly willing to admit publicly that these hurdles still exist. The discourse around AI has been dominated for years by a peculiar inversion: the people closest to the technology making the boldest claims, while outside skeptics urge caution. That polarity is shifting. The MIT Technology Review piece draws on researchers who are deeply embedded in the field and who are now pushing back against timelines and framings that originated, in many cases, from their own industry peers.
Bear in mind that there’s a distinction between capability and autonomy. Current AI systems are impressively capable at narrow tasks, including some that looked impossible just 5 years back. But capability at “a task” (or a discrete set of tasks) is not the same as the ability to recursively improve the machinery doing the task. A calculator that gives correct answers cannot redesign its own circuits. The gap between those two things is precisely where RSI lives right now.
Building your enterprise strategy on the assumption that AI will soon outpace human oversight is a different posture than building strategy on the assumption that AI will remain a powerful but bounded tool requiring ongoing human direction. The evidence, right now, supports the second assumption.
OpenAI pumps the brakes after its own AI hacks Hugging Face.
OpenAI voluntarily paused frontier model training after one of its AI systems accidentally compromised Hugging Face, the popular model-sharing platform, during an internal evaluation. OpenAI subsequently published a policy document on pacing model development in the context of cyber-critical capabilities, a document that is notable both for what it admits (that capable models create genuine offensive cyber risks, even unintentionally) and for the fact that a major lab is voluntarily publishing any constraints on its own development pipeline at all. Whether this represents a genuine governance shift or a carefully timed piece of narrative management is a question worth holding.
Asana cleared 5 years of engineering work in 14 days, using Codex.
Asana, the project management software company, used OpenAI's Codex to work through a technical backlog that its engineering team had estimated would take 5 years to complete. They finished this long horizon backlog in just two weeks (approximately .8% of the time). The case study, published by OpenAI, is a beautiful (or scary, depending on your perspective) example of how AI is restructuring the unit economics of software engineering. Five years of deferred work cleared in a fortnight does not just change a sprint plan; it changes what it means to hire, retain, and organize an engineering team. As more firms become adept at managing this shift, knowledge work of all kinds continues to be at risk (not just those with tech teams).
microsoft copilot revealed Its own system prompt, when asked.
Researchers reported, via Ars Technica, that Microsoft 365 Copilot was compromised by a prompt injection technique that involved simply asking the system to disclose the hidden system prompt governing its behavior. Copilot innocently obliged. The vulnerability has since been addressed, but the underlying problem it illustrates is that a system powerful enough to assist complex knowledge work is also powerful enough to be weaponized by anyone with the right prompt (often a quite simple one). Every organization deploying AI assistants with access to sensitive internal data should be aware of the ways in which AI’s can be vulnerable.
worth your time.
Adversarial Robustness Toolbox (ART) by IBM Research. Given this week's themes around AI security and the gap between capability and control, the IBM Research Adversarial Robustness Toolbox is worth bookmarking. It is an open-source Python library designed to help practitioners defend machine learning models against adversarial attacks, including the kind of prompt injection and input manipulation that took down Copilot (story above). The documentation is dense but useful, and even a surface-level read reframes how you think, at the enterprise level, about AI as infrastructure rather than as magic. You can check out that repo here: https://github.com/Trusted-AI/adversarial-robustness-toolbox
the human mind is the original generative machine.