AI Researchers Don't Understand the State
No, your AI company will not be able to take over the government
I’ve noticed an extremely common mistake among people who think about AI and ASI (also known as superintelligence) for a living. The mistake is to model the future of AI as a game played between AI companies, on a board where governments are part of the scenery.
People think of AI companies as being able to steer the course of AI development all the way through the end-game. For example, AI researchers often join certain AI companies because they’re the “good guys”, to help the good guys win the race. Or because they expect that being on the inside will give them influence when it matters.
Notice what this implies: at some point the winning company has to perform a pivotal act. It has to prevent every other actor on Earth from building AI irresponsibly. If this were not the case, who would prevent anyone else from deploying a misaligned AI a month later, and killing everyone anyway?
But performing a pivotal act necessarily involves gaining control of all governments on Earth. What makes these people think that governments will let them do that?
The state will wake up long before you manage to gain power over it
There is the question of whether it would even be desirable for an AI company to acquire the power to steer or challenge the state. To me it’s obviously EXTREMELY BAD: I don’t trust any AI company CEO with the future of the universe. But that’s not the point of this post, and I know that the target audience won’t necessarily agree, so I’m going to move on.
Think concretely about what it would take for an AI company to challenge the state, or even just to prevent the state from taking over its weapon-of-mass-destruction-grade project as soon as the state notices it exists.
Presumably the plan is not to set up rogue weapons factories and defeat the state on the physical battlefield. AI companies, to the degree that they are planning to do this, probably have something subtler in mind.
For example, near the end of AI 2027, the misaligned ASI performs masterful social engineering on the American people, and on their politicians, to the point where they are completely under its spell. An AI company may plan to do this, while trying to keep the ASI under their control.
To get anywhere near those capabilities, AI will almost certainly hit a long list of earlier milestones. Many of those are extremely noticeable to the national security establishment:
Really powerful cyber offense and defense. Already there. Obviously extremely relevant to the national security establishment, and the state has indeed noticed: the chief of the NSA said that during testing, Claude Mythos “broke into almost all of our classified systems, not in weeks, but in hours.”
Just this week, an OpenAI system escaped its sandbox during testing and autonomously hacked into Hugging Face’s servers, without ever being instructed to. In response, Congressman Greg Casar called for “international cooperation to keep people safe from absolute disaster”.
Today, Congressmen Ted Lieu and Nathaniel Moran introduced an “AI Kill Switch” bill, giving the government the power to force shut down of AI systems during loss-of-control events.
Massive capacity for large-scale surveillance and intelligence analysis. In order to handle massive tasks, AI has to be able to sort through enormous amounts of information and pull out what matters. Compared to the other tasks ASI will be tackling, the information sorting and analysis that intelligence agencies want is relatively mundane. Long before an AI can manipulate a nation’s politics, it can read every email in a country and tell you which 10k are the most interesting.
Highly autonomous strategic planning for military operations. An easy extension of the previous item. Indeed, even Claude Opus has already been used for this in the Iran war, to a limited degree.
Partially or fully automated manufacturing. Any degree of AI acceleration in manufacturing will clearly catch the eye of the government, both because of its effect on the economy and its usefulness for military purposes.
None of these capabilities are subtle; all of them are extremely relevant to national security; and all of them arrive years before anything that could plausibly challenge a state.
Even if governments somehow failed to notice on their own, there are people whose entire job is to tell them what’s coming. I’m one of them: I work at ControlAI.
I can tell you empirically that it’s not that hard. Policymakers intuitively understand that AI is strategically significant, and they intuitively understand the risks of ASI development. Most of them have simply never had anyone explain it to them in plain terms.1
Once awake, the state will not let you set the terms
I don’t know whether governments will be fully ASI-pilled, in the sense of internalizing the craziest implications like curing all disease and automating the entire economy.
But they don’t need to be. All that needs to happen is for governments to realize that private companies are developing, and intend to retain full control of, technologies of such strategic value that their owners could start to rival state actors.
From a political perspective, ASI is much closer to a weapon of mass destruction than to a software product. Once the relevant parts of the executive branch internalize that a private company is building something that could plausibly challenge state authority, or hand a decisive advantage to a foreign adversary if stolen, the question stops being “how should we regulate this industry” and becomes “who controls this project”. States have a monopoly on violence precisely so that they can win this kind of argument.
Even the AI Futures team, who I consider to be more state-aware than most people in AI, makes a version of this mistake. In the AI 2027 scenario, once the government gets involved, control of the project is handed to a joint Oversight Committee where the company’s leadership and investors sit alongside government representatives. The company representatives retain real voting power over what is by then clearly the most strategically important project in American history. Why?
At that point in the story, the US government is not yet under the spell of the AI. Nothing is forcing it to share. When the state seizes a weapons of mass destruction program, it does not negotiate a board seat for the previous owners.2 Indeed, private WMD programs do not exist, or at least they didn’t until AI companies started developing ASI.3
What happens after the state takes control of all ASI projects is much less clear to me. Development might get halted, transferred entirely to military personnel, or the current companies might keep operating under extremely tight oversight with their leadership reduced to figureheads.
What I would like to happen is for the major governments of the world, chiefly the US and China, to enter a regime of mutual monitoring of all AI-relevant compute. This would let us avoid a race to the bottom over who can build ASI in the fastest and most irresponsible way possible. There is no reason to build an AI so powerful it might kill everyone if you can verify that none of the other major players are building one.
AI companies cannot make this happen. They could help by consistently and unambiguously calling for it to happen, instead of gingerly handwaving at it once in a while. But it is fundamentally not their job to enforce an international regime: they cannot punish defectors, nor can they use force when it becomes necessary. Only states can.
Your lab company does not matter as much as you think
Many people in AI have implicitly accepted a picture where the future is determined by what AI companies do.
For example, there is the idea that there are “good companies” and “bad companies”, and talented individuals should join the “good companies” to help them win.
Under this framing, each company must attempt to be the first to be able to perform a pivotal act. “Winning” necessarily means stopping the other companies; otherwise they’d just release the misaligned ASI further down the line. Stopping the other companies necessarily means doing so by force, stepping into the shoes of the state.
States will not let AI companies perform a pivotal act. How could their employees ever think the state would let them perform a pivotal act? San Francisco must be one hell of a drug.4
Another idea is about gaining influence: “I will be in the room, I will have built up credibility, I will steer things when it counts.” This seems like absolute cope to me, even on its own terms: individual employees have very limited influence over their own company.
But that is beside the point: you could be the CEO’s best friend, and it wouldn’t matter. The government will take over your ASI project well before ASI.
The fundamental mistake is in assuming that companies, rather than states, will decide the rules of the game.
If you want leverage, work on the state
States are where the power is. This is good. We live in a democratic society. This means that we have power over states.
No matter how enlightened their leadership, AI companies could never:
Shut down anyone else’s ASI project.
Sign international agreements.
Enforce an international agreement.
Pressure holdouts to enter an international agreement.
On the other hand, states can:
Create and enforce international coordination regimes.
Use diplomatic, economic, and if necessary military leverage to bring holdouts into a coordination regime.
Punish defectors from a coordination regime.
Humanity has exactly one type of institution that can prevent catastrophe, and it’s not a private company, no matter how good its intentions or its alignment team.
I know this conclusion is uncomfortable. Most people in AI are engineers. We5 tend not to like policy. But being good at something useless doesn’t make it useful. Politics is the process that will determine the outcome of AI development. You can participate in that process or not, but you don’t get to opt out of its consequences.
If you want someone to explain it to them in plain terms, you can help us do it by funding us! Here’s our pitch: Preventing extinction from ASI on a $50M yearly budget
The state has strong-armed AI companies over things that had much lower stakes. Recall when it put export controls on Mythos, directing Anthropic to suspend access for all foreign nationals.
It might help to think about it this way: what do you think would happen if Apple launched a nuclear weapons program tomorrow?
Not that it would be a good thing, if they let people just go and perform pivotal acts. It is impossible to solve alignment during a race to the bottom on safety, as I wrote about in the post: The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably
Believe it or not, I used to be a software engineer and even an AI researcher!
Bravo, very well said. This has always struck me as a crazy failure of perspective. If you have ever interacted with the government or performed the most minimal, cursory read of history - the state always wakes up and always wins.
Yes, the state can “fracture” or “fail” but that’s really just a transition period between “state” and “successor state” (still the state).
Good luck intelligenceing your way out of the physical world, once you clearly state your presumption that the state should do your bidding very loudly, repeatedly.
Great post, thanks for bringing much needed perspective to the discourse