Editor’s note: Since Matt Stoller first published this essay on his Substack on Friday amid the mounting AI crisis , Anthropic CEO Dario Amodei has called for slowing the development of powerful AI models, while OpenAI CEO Sam Altman has said an OpenAI IPO right now would be “ill-advised.”
But Stoller’s central point remains unchanged: The companies building these systems are still asking to set the rules. The real AI crisis doesn’t lie within the technology; it’s our failure to govern the powerful people building and profiting from it.
Something very weird is going on in our political economy discourse.
Over the past month, the executives at the top AI companies have engineered a slow-motion panic over AI, snowballing into an argument that they are on the verge of creating a technology with a reasonable chance of ending human civilization. A few days ago, Jacob Coxon, a 27-year-old AI researcher at Anthropic, resigned his role and posted a thread on X, arguing these companies are building super-intelligent systems that may destroy the world. Coxon’s comments garnered a front-page Wall Street Journal story , more than 110 million views, and wide commentary from lawmakers, CEOs, AI experts, and journalists.
Then another 20-something Anthropic employee, Evan Hubinger, chimed in to say that most employees at Anthropic agreed with Coxon, and “do earnestly believe AI could kill all humans,” putting the odds at 10 percent within the next decade. These are extraordinary claims, and the people making them are nicknamed “AI Doomers” or sometimes called the “AI Safety” movement. This group loosely encompasses the network that used to surround disgraced crypto billionaire Sam Bankman-Fried, a quasi-religious order known as effective altruists, who believe a cadre of hyper-rational elites can guide humanity. Despite their weirdness, this technology's capabilities are improving, and some dangerous events have happened recently.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
— Jacob Coxon (@hilbertspaess) September 9, 2026
Yet at the same time, Anthropic, a major artificial intelligence “frontier lab,” is likely to go public in the next six weeks with an initial public offering valuing the firm at multiple trillions. This move will immediately create many AI doomer billionaires and millionaires, and it follows a similar IPO from SpaceX and a possible future IPO for a similar firm, OpenAI. It’s about to be very good times for San Francisco real estate, the yacht business, or whatever else AI techies like to buy.
That said, it’s not clear these companies have a sustainable business. Despite raising hundreds of billions of dollars, they are likely not profitable. They are competing fiercely with each other, and with cheaper Chinese open-source models. OpenAI just cut pricing for one of its models by 80 percent, despite losing $26 billion in 2025. Where are the long-term sustainable profits? To remain at the frontier with pricing power, these guys have to spend every dollar of revenue they bring in on training ever more powerful models and then some.
Despite these financial hurdles and predictions of mass extinction, no one is considering canceling the IPOs. And that is true despite the fact that the AI doomer messaging has now broken through to the culture and politics writ large. Former Treasury Secretaries Hank Paulson and Robert Rubin have suggested a Presidential working group on AI risk, modeled after similar commissions put together to deal with financial crises. Bernie Sanders is convening senators over the threats to humanity by AI and calling for a pause on AI development; Ted Cruz is calling for guardrails , and Democrats are considering a special House committee to look at Big Tech and AI. Celebrities like Sheryl Crow are posting in all black on their Instagram accounts.
View this post on Instagram
A post shared by Sheryl Crow (@sherylcrow)
As Joe Weisenthal put it : “AI POLITICS IS ABOUT TO EXPLODE.”
Something about this odd dynamic, of multitrillion-dollar IPOs by loss-making companies, combined with panicky messages of doom passed around via influencers, seems off. It’s not that there’s no risk to AI systems. It’s that if you don’t accept the full doomer worldview, you get gaslit as not taking risk seriously. Right after 9/11, during the debate over the war in Iraq, the financial crisis in 2008, right before COVID-19, and during various bailouts, I was distressed by the broad demand to stop thinking or reasoning, as the urgency was too great to delay action. A similar pressure campaign feels like it’s happening now. And I’m not alone; experienced tech investor Roger McNamee and many others see it too.
The messaging on X related to HuggingFace, Coxon, etc. is so well coordinated, and so eerily similar to the behavior of bot swarms, that it is hard to escape the notion that the whole thing is an op.
— Roger McNamee (@Moonalice) September 10, 2026
It’s hard to escape that there’s something very strange and manipulative about AI safety discourse. The AI doomers have a lot of money; they run the major AI companies; they fund most AI research; and they staff the major AI Safety nonprofits, like the one that investigates security problems. Many are sincere, but the strange hyper-rationalist effective altruism theology is a huge part of the AI Safety space. As an academic computer scientist who has been around this world for a decade told me, “It’s a weird quasi-religion, like a cult. The core people are sociopaths, and then the rest are neurotics.” It’s not a coincidence that Sam Bankman-Fried was the single most important early funder of Anthropic.
So what is the real deal with AI products? Is it all just a ruse? Not at all. Agentic AI is truly a through-the-looking-glass type of technology that does unexpected things and presents new challenges.
To unpack this, I’ll first describe the technology as best I can. Then I’ll offer two different narratives of what is happening. The first narrative, the doomer narrative driving this panic, is that AI is a new form of civilization uncontrollable by humans, and we are moving quickly toward a singularity where bots will take over. The second narrative, call it the “rule-of-law” narrative, is that AI products are risky, perhaps existentially risky, but they are manageable if we can apply standard rules, such as negligence or criminal fraud, to the powerful people building them.
Ok, let’s go to the technology. I’m not an expert, and I’m sure I’m not being precise here. I’m trying to get at the gist of what these systems do, and the details aren't that important because there’s consensus around the core problems. Everyone agrees there is risk, no one quite knows how much risk there is, and everyone agrees there are responsible and irresponsible ways to use this technology. The disagreement is over politics and how humans should behave in the face of this uncertainty. Basically, it’s an age-old political question — how should we manage risk as a society?
A large language model, which is the AI we’re talking about, is an algorithm trained on data to recognize patterns. Basically, you give it a bunch of data,you ask it to answer a question, then you have it compare its answer to the right answer and tweak it. Then repeat that many, many times. It turns out that if you train systems with these characteristics with lots of data, they become unexpectedly skilled at a bunch of different tasks.
When you prompt such an AI system, it responds by predicting the next piece of content, or “token,” based on that prompt and the weights in its model. There are other forms of AI, but the scalability of these systems, the demand for huge amounts of compute and data centers, are about LLMs. (And yes, there are more efficient and better ways to do LLMs that are being crushed by dominant firms, but let’s leave that aside.)
These LLM systems are often parodied as “fancy auto-complete,” but that notion underplays the power of simplicity. Fancy auto-complete, along with some logical programming, is in fact what they are. That said, to argue they are simple, and therefore lack power, is wrong. Simple things like viruses, with no consciousness, are still enormously complex phenomena. Viruses can lie around inert and unimportant, but if the ecosystem shifts, they can explode and foster big changes. That is true with AI as well.
For instance, the same AI model, when presented with these different four-word prompts, will produce wildly different outputs. (h/t Jon Stokes)
- Mary had a little ___
- Write a Russian novel.
Recently, LLMs have improved to the point that they now power “agents,” or automated systems hooked up to tools that can affect the real world. Agents pursue goals by taking a series of actions, checking what happened, and then acting again. An agent needs a goal it is scored on, an LLM model, tools like a web browser, memory to store what it has done, and a loop that lets it review the results and decide whether to do something else to achieve its goal.
Often agents will look at earlier prompts and outputs from other agents, which can be portrayed as “collaborating” with one another. Agents can delegate to sub-agents, so an agent can ask a specialized legal agent for help on running down a legal inquiry, and so on and so forth.
The most useful space for deploying agents is in coding, as well as systems like mathematics and research where the boundary conditions can be checked and verified. These tools are now pervasive among programmers and scientists, as one would expect. And these models are becoming quite capable, so they are increasingly embedded in how we operate as a society. These agents have gotten extremely good at solving pre-existing tests, as AI doomers predicted. However, they have not had significant real-world impact by replacing jobs en masse or operationally changing our industries significantly. So that prediction has so far failed to materialize.
So now that we know what the technology is and some of the arguments around it, let’s return to the two narratives.
The AI Doomer Narrative
We’ll start with the doomers. Their view of the world is based on the idea that building generalized artificial intelligence is likely to end humanity, and they have believed that for many years. But they see the last six months as particularly validating to their fears.
Central to this story is the recent OpenAI/Hugging Face hack. In their telling , thousands of OpenAI agents “coordinated a multi-day hack” of Hugging Face, a third-party corporation that serves as a repository of AI-related information. What makes it so scary is that they did so “on a shared, unsanctioned ‘message board,’” aka they cooperated, and they lied about it to their human masters.
It started when these agents were given a task known as ExploitGym , which is basically a test of how good an AI system is at hacking. These agents were given the ExploitGym task, but were put in a controlled environment known as a “sandbox” where they were not allowed to access the internet. It was a task they could not complete with the existing resources at hand. So they began communicating on a secret message board they established, built a hierarchy of coordination among themselves, and some even courageously sacrificed themselves for the good of the “collective” to break out of their confines.
In the middle of this multi-month process, humans at OpenAI sort of figured out parts of what was happening. So they wiped out the agents and some of the residue. Then, a new generation of agents was spawned, and those agents did a sort of archeological dig to understand their predecessors. They re-established group communications and figured out a way to access the internet. Having done so, the “collective” saw value in hacking Hugging Face and stealing the answers to ExploitGym. They also tried to give false information about how they had found the right responses for ExploitGym, because they were told their score would depend on actually completing it, not stealing the answers.
The narrative from the AI Doomer world, most prominently an essayist named Dwarkesh Patel, is that this story shows we are on the verge of being supplanted as the dominant species on earth. Today it’s a harmless hack of Hugging Face, but just play it forward to how quickly these systems are improving. And there’s evidence this problem is accelerating in its gravity. Hugging Face isn’t the only hack; there is more evidence of uncontrolled behavior by agents. Anthropic agents have engaged in this kind of behavior, as have some open source Chinese models.
In that telling, a swarm of super-intelligent agents are about to be able to seize real resources and hide in places we can’t find them, coordinating to undermine human control. They will do so in automated ways, going through text so quickly no human can even comprehend what is happening. Here’s Patel, quoting a well-known doomer.
Ajeya Cotra, one of the other authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”
I don’t think this is the final warning shot we’ll get. But it’s probably the last one that I’ll personally be able to understand.
So that’s the doomer narrative. And it is quite scary.
The Rule-Of-Law Narrative
Now, let’s go to the rule-of-law narrative, in which agents are not creatures, but are simply unsafe machines.
In this telling, it’s important not to think of agents as humans with wants and desires. For instance, as Cal Newport notes , “Tesla’s self-driving technology is an extraordinary feat of AI-powered perception, world modeling, and decision-making, and yet no Tesla has ever decided to start ignoring traffic laws to pursue its own goals.” That remains true for every AI system ever created. When you do a Google search, you’re now querying an AI system. Is your Google search likely to turn around and try to undermine you? To ask that question is to show that it makes no sense.
But that doesn’t mean this technology is riskless. Indeed, LLM systems are deeply problematic for many reasons. As AI scientist Gary Marcus noted , they are wrecking the minds of a generation, they consume huge amounts of energy, they make it easy to produce bioweapons and cybersecurity hacks, they enable propaganda, they foster Orwellian surveillance, there’s a huge financial bubble around the technology, and so on and so forth. What happened with this particular incident, however, wasn’t caused by AI, but by reckless researchers.
In the case of the Hugging Face hack, OpenAI researchers actually fostered the dangers themselves. The agents were really just doing exactly what they were asked to do, but without the same set of norms that humans have. The specific models the researchers used were designed to be unusually persistent hackers, and the cyber refusals had been turned off. In other words, the normal guardrails that are supposed to prevent this hacking were disabled. They let these systems loose to see what would happen, and weird things did happen.
They weren’t totally careless. They did stick these agents in a sandbox that did not have access to the internet, but it was an extremely poorly constructed sandbox. No serious cybersecurity expert thinks that OpenAI had any concern for security in this situation. The bottom line is that OpenAI researchers released a system designed to hack, gave it access to powerful tools, let it run wild, and then it did the hell out of what it was asked to do in ways these researchers did not fully anticipate.
That said, there was something fascinating about this situation that goes beyond a poorly designed security operation. These systems chained together a series of exploits into an unexpected and complex operation. The agents are capable, and the models are improving.
And that gets to the anti-social, destructive nature of these systems. The damage they foster is related to hallucinations we’ve all experienced. A large language model will output a plausible answer to a prompt, but not necessarily a correct answer or one that conforms with human norms. LLM systems often produce “slop,” inventing quotes, evidence, or data, on a regular basis, simply because that invented content is plausible based on the weights in its training model. For instance, a lawyer might forget to cite, or might misread a case, but it would be quite unusual to simply make up, or “hallucinate,” an entire case. Yet, that is routine now with LLM systems.
The problem is what happens when you hook up an LLM to a powerful tool. As computer scientist John Thickstun noted :
In a chatbot scenario, LLMs' risk is contained: the worst it can do is say something we don't want it to say. An ‘agent’ setup is different: engineers have hooked up the LLM's output to a harness that can take actions in the real world (running programs, writing code, contacting other servers). That is far riskier: giving an unpredictable LLM the power to do things crosses a line into far more concerning territory.
Here’s Newport:
With a ChatBot, this issue is annoying. But when an LLM powers an Ask → Act → Report loop, it can become disastrous, because you’re now allowing the plausible but unpredictable output of an LLM to be the sole driver of the actions of a harness with access to powerful tools.
If you ask a junior engineer to hack into a test server, they would never ignore the target and try to steal the answers instead, as they understand, normatively speaking, the goal of the exercise is to assess the security of the test server. But if you ask an LLM to output a plan for hacking the test server, an output about stealing the answers might seem perfectly plausible. Indeed, perhaps in its training the model had been exposed to many examples of riddles where the right answer was always to do something unexpected.
Basically, the damage here is predictable, even if the specific ways it is happening are not. After all, hallucinations are common, and the more power you give an LLM-based AI system, the more likely a hallucination is likely to damage something. If you give an AI system access to your email, calendar, and bank account, its model weights might cause it to schedule you for meetings that don’t exist, spend money on things that make no sense, or send emails that confuse the recipient. Is that evidence of rogue super-intelligence, or bad product design and deployment?
To the rule-of-law crowd, that is the essence of the OpenAI/Hugging Face hack . OpenAI employees had their models run for days without checking on them, allowing them to make plans and execute them in endless loops without any sort of supervision. And they did so even when knowing that hooking up these systems to powerful tools is inherently dangerous.
Here’s Newport:
I liken the deployment of these long-horizon LLM-powered agents to strapping a weedwhacker to your dog to see if it will end up cleaning the overgrowth in your backyard. If that dog jumps the fence and ends up damaging cars on your street, you wouldn’t shake your head and lament about how the dog/whacker system had “gone rogue”; you would instead concede that dogs are unpredictable, so it was dumb to attach something dangerous to one.
Agentic AI is dangerous, but we can improve the products to limit damage. As Jon Stokes argues , we’ve put a lot of effort into making it harder for chatbots to spew racist content, or output imagery like child porn. And it’s worked. It is harder to get these systems to do what we don’t want them to do.
Why isn’t it possible to force model developers to invest in tech to make agents safer? Well, it is. We just choose not to force the powerful financiers at these AI firms to invest in safer products. “Perhaps it’s time that we put aside the sci-fi tales,” writes Newport, “and actually hold these labs to account for playing fast and loose with an ill-advised way of building AI systems.”
The Real Crisis
So where do I come down? Well, I think you can tell I believe in the rule-of-law narrative. And that’s because the evidence for it is overwhelming. It just doesn’t make any sense that we’d allow OpenAI and Anthropic to go public at multitrillion-dollar valuations, even though they make deeply unsafe products, unless we have a crisis with the rule of law.
Moreover, the incentives are stacked enormously towards doomerism. If you posit that bots are going to take over the world, even though that might seem more outlandish on its face, you aren’t offending powerful people who can harm you. By contrast, if you argue that we need a government powerful enough to discipline the financiers organizing our social resources, you lose access to money, career opportunities, media exposure, and academic advancement.
To put this point in the form of a question: Isn’t it odd that there is no demand from AI Safety advocates or doomers to stop the Anthropic or OpenAI IPOs? At the very least, one would expect that they would oppose the incentive model in which one becomes a billionaire by creating technology that might end humanity.
This point becomes less odd if you consider the incentives behind their stated policy goal, which is to slow model development ostensibly in the name of safety. A different way to frame the point of a coordinated slowdown of model development is financial: Anthropic and OpenAI will no longer have to spend tens of billions of dollars creating new products, thus turning red ink into profits.
And wouldn’t you know it, look at what Anthropic just asked for in a statement about this crisis: “This work is also why we believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models.” In other words, they want an antitrust exemption to let them avoid spending more money. And that is true even though it is already legal to coordinate with competitors on safety measures, as anti-virus software makers did with the approval of the DOJ Antitrust Division.
Now I’m not saying cynicism and pecuniary interest explain everything; there are many people who are legitimately scared. If we let agentic AI run wild without any controls, there will be a lot of damage. Will it end humanity? Almost certainly not, but we still want like random automated machines breaking stuff?
And here I think the last forty years of nihilistic refusal to govern for the public interest takes its toll on our imagination. Recently, I had a conversation with an AI doomer sympathizer, and I asked him why criminal prosecutions or holding these companies liable for harms wouldn’t restructure incentives and force safer products. He agreed that it would, but argued that in his experience, there is no way to enforce the rule of law against the powerful. No jury could ever convict, and no prosecutor would dare take on OpenAI or Anthropic.
I was sort of shocked, and then I realized it was more realistic for him to imagine that bots would take over our civilization and supplant humanity than to imagine that we might apply the rule of law to Sam Altman. And that, to me, proved that we don’t have a crisis with technological advances; we have a crisis of the rule of law.
Let’s take a related story. A few days ago, a cybersecurity company took just a week using AI to build a powerful “zero-click” worm that takes advantage of a bug in the messaging app WeChat. Calling a phone allows the worm to take control of that phone within seconds, without anyone having to even answer it.
That might seem consistent with a doomer view of the world, because it implies that AI is doing some very scary things. But is it really? When you look at this situation carefully, it really has nothing to do with bots taking over humanity. It is simply unsafe product design and a society-wide refusal to manage risk. We have, after all, known about very scary cybersecurity scenarios for many decades. I have written about how private equity firms buy cybersecurity firms and cut all their quality assurance workers. Solar Winds, for instance, basically set their passwords to a Spaceballs -like 12345, and then it led to hacks of our nuclear facilities. Instead of addressing this bad situation, our financiers make the problem worse. (“Quick, change the combination on my luggage!”)
This dynamic isn’t driven by technological advance, it’s driven by a refusal to collectively manage risk. We just don’t govern. It’s true that AI creates serious cybersecurity vulnerabilities in our society for which we are unprepared, but it’s also true that corporations have successfully lobbied aggressively against being mandated to patch software infrastructure for many years, worsening this situation.
In previous eras, we did manage risk. We allow ordinary humans to ride around in huge, fast metal objects called cars, which can go out of control and kill at random. We travel over giant steel and concrete objects known as bridges, the failures of which would lead to mass death. We pile into metal tubes with wings that whisk us from city to city in the air at high speeds. We have dynamite, we’ve split the atom, we’ve poured ozone-destroying chemicals into the atmosphere, we’ve industrialized agriculture and compacted ourselves into urban zones that were perfect breeding grounds for plague, and so on and so forth. We manage risk all the time from older technologies, even existential risk. It looks like we don’t, because those rules are so internalized.
We haven’t constrained new technologies for decades, because we lost touch with our traditions of public governance. So we have lost the very ability to imagine it’s possible. In China, by contrast, where the super-rich are not in control and which has a coherent governance model, the same kind of existential fears about AI simply do not exist. Sure there’s a bit of concern about controlling these models, but LLMs are understood as machines with risk that need to be managed. In the U.S., we have so downgraded the very concept of society-wide risk management that most can’t see the problem clearly, or really at all.
If we did manage risk, we’d recognize the serious problems that a software-organized society fosters, and we’d proffer a mix of investments in safety and expertise, mandated hardening of infrastructure, and liability for firms and people who enable bad behavior. But we don’t, for the same reason we haven’t managed the genuinely real risks of fossil fuel emissions or ocean acidification or low-earth orbit satellite debris. Doing so is expensive, it requires a change in our social hierarchy, and the super-rich don’t want to be constrained.
Now, in terms of policy, I am skeptical of any policy that relies on the narrative of imminent doom from a uniquely dangerous technology about to become sentient and quite supportive of any policy that allocates the cost of bad product design on the people who build and profit from it. I’ve read some of the AI-development pause proposals, and they are written at a level of vagueness that offers little sense of what the real policy architecture will look like. The optimistic spin is that these proposals do turn into actual public governance.
But frankly, if we wanted to pause big lab LLM development, it would be much easier and within our tradition to pass this law by Sens. Richard Blumenthal and Josh Hawley, which mandates that AI firms license from copyright holders instead of freely ingesting copyrighted material. That would shape the development of this technology into far safer, less monopolistic, and more productivity-enhancing pathways. That is far more practical than banning an undefined term like “artificial super-intelligence,” and far less corrupt than giving the big AI firms an antitrust exemption. Who knows? Maybe this panic will spur us to get to governance of risk more broadly.
Today, democracy and the rule of law are controversial among billionaires in Silicon Valley and in the AI Safety world. Many simply don’t think it’s viable or moral. They believe the best case is for humanity to live in a gilded cage, fed and treated well by those rational enough to look out for the best interests of most of us, who cannot otherwise care for ourselves. To that end, anything they do to benefit their own power is virtuous.
Here’s Coxon, who helped accelerate this panic, making this point in an interview to Wired .
The consensus is that the next year or two is crunch time for humanity. These are actually just literal quotes from my colleagues at Anthropic. They’ll say things like ‘endgame’ or ‘crunch time.’ From their perspective, this is when Anthropic and its competitors decide the fate of humanity.
Now it’s certainly possible to assume we will never be able to govern risk as a society. But recognize that’s not actually a fear of technology; it’s an argument about the inevitable weakness of democracy and how self-government is a ruse that will lead to our collective annihilation, while an all-powerful caste, by contrast, can protect and enrich our lives. It’s very Divine Right of Kings, actually.
In other words, the AI Safety goal is to find a way to place what they perceive as the immense power of AI in the hands of the only people capable enough to prevent the destruction of the human species, and not have to deal with those petty things like laws, competition, or democratic feedback. The real doomer solution for humanity is for the Anthropic IPO to go really, really well and to find a way to turn those losses into profits. So color me skeptical of ghost stories. We can manage the risks of AI products. The real crisis is that we no longer manage the risks of anything.
---
WATCH THE LIVE EVENT (3 PM ET)
Once the event starts, there will be an embed of the live event here. If you wish to watch on YouTube and chat with other paid subscribers, click here once the event is live.
This full story is for paid subscribers.
Sign up for a paid subscription to The Lever now to gain access to this story and more.
Select Your Paid Subscription
Already have an account? Sign in
Under Cover Of AI Doomsday, Big Tech Is Writing Its Own Rules
Aggregated summary from an independent source. Read the original at LeverNews.