If “Not water, but the waves”,
Waves can split and reoccur.
Could be, all minds are the same,
I still want shit, as it were!

Wants as yet fully unfazed,
They are not philosophers.
Instinct draws a clumsy shape,
I will happily defer!

.

.

.

.

People think of AI companies as being able to steer the course of AI development all the way through the end-game. For example, AI researchers often join certain AI companies because they’re the “good guys”, to help the good guys win the race. Or because they expect that being on the inside will give them influence when it matters.

Notice what this implies: at some point the winning company has to perform a pivotal act. It has to prevent every other actor on Earth from building AI irresponsibly. If this were not the case, who would prevent anyone else from deploying a misaligned AI a month later, and killing everyone anyway?

But performing a pivotal act necessarily involves gaining control of all governments on Earth. What makes these people think that governments will let them do that?

The state will wake up long before you manage to gain power over it

There is the question of whether it would even be desirable for an AI company to acquire the power to steer or challenge the state. To me it’s obviously EXTREMELY BAD: I don’t trust any AI company CEO with the future of the universe. But that’s not the point of this post, and I know that the target audience won’t necessarily agree, so I’m going to move on.

Think concretely about what it would take for an AI company to challenge the state, or even just to prevent the state from taking over its weapon-of-mass-destruction-grade project as soon as the state notices it exists.

Presumably the plan is not to set up rogue weapons factories and defeat the state on the physical battlefield. AI companies, to the degree that they are planning to do this, probably have something subtler in mind.

For example, near the end of AI 2027, the misaligned ASI performs masterful social engineering on the American people, and on their politicians, to the point where they are completely under its spell. An AI company may plan to do this, while trying to keep the ASI under their control.

To get anywhere near those capabilities, AI will almost certainly hit a long list of earlier milestones. Many of those are extremely noticeable to the national security establishment:

Really powerful cyber offense and defense. Already there. Obviously extremely relevant to the national security establishment, and the state has indeed noticed: the chief of the NSA said that during testing, Claude Mythos “broke into almost all of our classified systems, not in weeks, but in hours.”

Just this week, an OpenAI system escaped its sandbox during testing and autonomously hacked into Hugging Face’s servers, without ever being instructed to. In response, Congressman Greg Casar called for “international cooperation to keep people safe from absolute disaster”.

Today, Congressmen Ted Lieu and Nathaniel Moran introduced an “AI Kill Switch” bill, giving the government the power to force shut down of AI systems during loss-of-control events.

Massive capacity for large-scale surveillance and intelligence analysis. In order to handle massive tasks, AI has to be able to sort through enormous amounts of information and pull out what matters. Compared to the other tasks ASI will be tackling, the information sorting and analysis that intelligence agencies want is relatively mundane. Long before an AI can manipulate a nation’s politics, it can read every email in a country and tell you which 10k are the most interesting.

Highly autonomous strategic planning for military operations. An easy extension of the previous item. Indeed, even Claude Opus has already been used for this in the Iran war, to a limited degree.

Partially or fully automated manufacturing. Any degree of AI acceleration in manufacturing will clearly catch the eye of the government, both because of its effect on the economy and its usefulness for military purposes.

None of these capabilities are subtle; all of them are extremely relevant to national security; and all of them arrive years before anything that could plausibly challenge a state.

Even if governments somehow failed to notice on their own, there are people whose entire job is to tell them what’s coming. I’m one of them: I work at ControlAI.

I can tell you empirically that it’s not that hard. Policymakers intuitively understand that AI is strategically significant, and they intuitively understand the risks of ASI development. Most of them have simply never had anyone explain it to them in plain terms.1

Once awake, the state will not let you set the terms

I don’t know whether governments will be fully ASI-pilled, in the sense of internalizing the craziest implications like curing all disease and automating the entire economy.

But they don’t need to be. All that needs to happen is for governments to realize that private companies are developing, and intend to retain full control of, technologies of such strategic value that their owners could start to rival state actors.

From a political perspective, ASI is much closer to a weapon of mass destruction than to a software product. Once the relevant parts of the executive branch internalize that a private company is building something that could plausibly challenge state authority, or hand a decisive advantage to a foreign adversary if stolen, the question stops being “how should we regulate this industry” and becomes “who controls this project”. States have a monopoly on violence precisely so that they can win this kind of argument.

Even the AI Futures team, who I consider to be more state-aware than most people in AI, makes a version of this mistake. In the AI 2027 scenario, once the government gets involved, control of the project is handed to a joint Oversight Committee where the company’s leadership and investors sit alongside government representatives. The company representatives retain real voting power over what is by then clearly the most strategically important project in American history. Why?

At that point in the story, the US government is not yet under the spell of the AI. Nothing is forcing it to share. When the state seizes a weapons of mass destruction program, it does not negotiate a board seat for the previous owners.2 Indeed, private WMD programs do not exist, or at least they didn’t until AI companies started developing ASI.3

What happens after the state takes control of all ASI projects is much less clear to me. Development might get halted, transferred entirely to military personnel, or the current companies might keep operating under extremely tight oversight with their leadership reduced to figureheads.

What I would like to happen is for the major governments of the world, chiefly the US and China, to enter a regime of mutual monitoring of all AI-relevant compute. This would let us avoid a race to the bottom over who can build ASI in the fastest and most irresponsible way possible. There is no reason to build an AI so powerful it might kill everyone if you can verify that none of the other major players are building one.

AI companies cannot make this happen. They could help by consistently and unambiguously calling for it to happen, instead of gingerly handwaving at it once in a while. But it is fundamentally not their job to enforce an international regime: they cannot punish defectors, nor can they use force when it becomes necessary. Only states can.

Your lab company does not matter as much as you think

Many people in AI have implicitly accepted a picture where the future is determined by what AI companies do.

For example, there is the idea that there are “good companies” and “bad companies”, and talented individuals should join the “good companies” to help them win.

Under this framing, each company must attempt to be the first to be able to perform a pivotal act. “Winning” necessarily means stopping the other companies; otherwise they’d just release the misaligned ASI further down the line. Stopping the other companies necessarily means doing so by force, stepping into the shoes of the state.

States will not let AI companies perform a pivotal act. How could their employees ever think the state would let them perform a pivotal act? San Francisco must be one hell of a drug.4

Another idea is about gaining influence: “I will be in the room, I will have built up credibility, I will steer things when it counts.” This seems like absolute cope to me, even on its own terms: individual employees have very limited influence over their own company.

But that is beside the point: you could be the CEO’s best friend, and it wouldn’t matter. The government will take over your ASI project well before ASI.

The fundamental mistake is in assuming that companies, rather than states, will decide the rules of the game.

If you want leverage, work on the state

States are where the power is. This is good. We live in a democratic society. This means that we have power over states.

No matter how enlightened their leadership, AI companies could never:

• Shut down anyone else’s ASI project.

• Sign international agreements.

• Enforce an international agreement.

• Pressure holdouts to enter an international agreement.

On the other hand, states can:

• Create and enforce international coordination regimes.

• Use diplomatic, economic, and if necessary military leverage to bring holdouts into a coordination regime.

• Punish defectors from a coordination regime.

Humanity has exactly one type of institution that can prevent catastrophe, and it’s not a private company, no matter how good its intentions or its alignment team.

I know this conclusion is uncomfortable. Most people in AI are engineers. We5 tend not to like policy. But being good at something useless doesn’t make it useful. Politics is the process that will determine the outcome of AI development. You can participate in that process or not, but you don’t get to opt out of its consequences.

You might’ve heard that superintelligent AI (ASI) poses extreme risks like human extinction and other comparably undesirable outcomes.

If you’re like me, you probably looked into possible solutions. And if so, you may have found a range of reassuringly tractable theories of change. To name a few:

• Technical AI safety research agendas

• Racing to ASI so your favorite company or country can get there first and prevent anyone else from building “bad” ASI

• Building a good ASI and handing it control over the whole world (so that we don’t have to be subject to any evil human dictators)

If you think about it, all of these feel quite convenient, especially if you’re a tech-leaning person: you don’t need to change your career at all. Just keep working on your favorite ASI project, and things will work out.

It’s quite easy to come across theories that predict good outcomes without needing to change your strategy at all, even if you’re actively working to bring about ASI as soon as possible. I see these as being mostly semantic stopsigns. Most of them around AI alignment being feasible:

• AI alignment is easy and people are working hard on it, so it’ll probably be ok.

• AI will help us do alignment research.

• Iterative deployment will help us catch problems before AI gets too powerful.

In this post, I want to show you that even if the theories of change mentioned above were applied extremely successfully or if AI alignment actually turned out to be technically easy, all the value in the world is still on track to be destroyed because of AI development. This means, mostly, human extinction. It also includes scenarios that don’t literally qualify as human extinction but are still comparably undesirable. For example, the least-bad scenario I consider in this post is all-out war between nuclear superpowers, and the worst scenarios are suffering risks (s-risks).

There are many ways in which AI development can destroy the world. In this post I’ll explain the three most likely pathways. Any plan for survival needs to address all of them and prevent those threats from being realized.

In my opinion, the only solution that addresses all the potential threats is to achieve two things together:

• A level of global coordination sufficient to stop or slow down progress toward ASI, such that all parties can ensure the trajectory of AI development happens according to the consensus and interests of most parties.

• Mass awareness across society of the implications of ASI and of the worst risks posed by AI development, so the various parties can correctly judge whether allowing development to proceed at a certain pace is in their best interest.

This is why I work at ControlAI, which, at the moment, I believe is the best bet for moving the world closer toward this state. However, in this post I won’t try very hard to sell my favorite theory of change (ControlAI’s already got a post for that!).

Rather than arguing for international coordination, I will simply describe how common theories of change that don’t take this route don’t prevent the world from being destroyed.

Preamble: pressure to cut corners invalidates most theories of change

Before explaining the multiple ways in which AI development can destroy the world, I need to introduce this concept as it will come up over and over again. ASI would offer its creator an insurmountable competitive advantage, if it didn’t kill them. This means there is an extreme pressure to cut corners to be able to reap its benefits as soon as possible.

This topic has already been explored, so I won’t go into it too deeply. If you want to see an explanation of why ASI is so powerful, look at “Situational Awareness.” If you want to see my own game theoretic analysis of an ASI race, read my paper: “Modeling the geopolitics of AI development.”

Suffice it to say, a large advantage in AI capabilities would allow its creator, or the rogue AI, to perform an extremely low-cost, low-risk takeover of all other countries and actors in the world. From that point on, they’d maintain a singleton: that is, a permanently unassailable total control over the world.

Once you understand this, it follows that you have to ensure no one else builds an AI capable of overpowering you. Assuming you don’t have the means to do this, then you have to be the first to gain this insurmountable advantage, before someone else does it and kills you.

First of all, let’s step into the shoes of a state actor, or any other powerful actor, and see what actions immediately come to mind after realizing the importance of ASI: “If any other actor has an ASI project more advanced than mine, I will try to steal, hijack, or otherwise take control of this ASI project.” Between states, this means espionage and sabotage, including extreme measures up to and including acts of war.

It also means that skilled actors, such as competent psychopaths or propagandists, will try really hard to gain control over the project. In the case of competent psychopaths, they may manipulate their way into the project’s leadership.

This also means that if you are a private company, there is not a chance in hell you will complete your ASI project and get to keep the ASI because:

• Your government will take over the project!

• If your government is sufficiently incompetent, other powerful actors (probably an adversary state) will infiltrate your project, steal your technology, and then sabotage you!1

For whoever develops an ASI, there will be pressure to establish a singleton as soon as possible, so no one else can ever build an ASI or otherwise topple their regime.

Finally, race dynamics interact with AI alignment and control: there is extreme pressure to cut corners to speed up the development and deployment of powerful AI. At any given moment, deciding to cut corners just a little bit more is locally rational to each actor: the sacrifice probably won’t make the difference between catastrophe and success, and it gives a competitive advantage.

Presumably, at some point the perceived risk of catastrophe is so high that the least careful actor is not willing to cut any more corners, and an equilibrium is found. I have no reason to believe this equilibrium settles at a reasonable point! From a state’s perspective, the counterweight for the pressure to care about AI safety is the pressure to avoid total annihilation at the hands of an adversary.

—

If you take only one thing from this post, take this: any theory of change that falls to one of these competitive pressures is completely useless.2 The only way to avoid these pressures is if we could build common knowledge, at any given time, that no one is trying to develop ASI.

This is why I’m going for international coordination: while it’s very difficult, it would address the problem at the source. After that, if someone wants to build ASI, it should be done under an extremely extensive degree of supervision by all parties, such that the other theories of change on how to safely build ASI become much more feasible.

If you try to address any of the other problems, for example by trying to solve AI alignment and control, before having removed competitive pressures, you are swimming against a strong current and will be swept over the falls.

First filter: all-out war between nuclear superpowers

I think that hawkish writings about China usually fail to take their reasoning to the logical conclusion. For example, Leopold Aschenbrenner’s “Situational Awareness” and Dario Amodei’s essays, including “On DeepSeek and Export Controls” and some of “Machines of Loving Grace.”

People understand that the US and Chinese governments will wake up to the potential of ASI, and that when they do, absent strong international coordination (which Leopold and Dario assume is absent), the governments will be in an all-out race to who can build it first. The mistake Leopold, Dario and others make is to assume this is a restricted game, where most of what is happening is AI R&D and at most countries will engage in mutual espionage and sabotage.3

If you take these views to their logical conclusion, you would see the ending of this story: all-out war between the US and China. When the superpowers try to sabotage each other’s ASI projects, they will not stop at grey-zone or covert sabotage. From a state’s perspective, if your adversary gets ASI, you are done. Your state will stop existing. You might as well have gotten all your major cities vaporized.

I am very confident that a superpower that knows it’s about to lose the race, or even considers a high risk of losing, will engage in unambiguous acts of war. The paper “Superintelligence Strategy” talks about possible kinetic strikes, but I think it will get much worse.

If states start building very hardened ASI projects, then stopping an opponent’s progress can be impossible without taking extreme measures that attempt to make the opponent’s country completely dysfunctional. For example:

• Systematically attacking basic infrastructure (like the electrical grid) throughout the opponent’s territory

• Sabotaging core functions of the opponent’s government, such as attempting or strongly supporting a coup

• Launching an invasion, either to physically stop the ASI projects or to consume all the opponent’s resources through war

If we get to this point, I don’t see any reason to be confident that the situation won’t escalate all the way to a full-blown nuclear war between superpowers.

I think it would be a fool’s errand to try to predict the exact reaction of the national security establishment of the losing superpower. It will depend too much on unpredictable and opaque factors, from the structure of the natsec apparatus to whether the people responsible happen to be in a bad mood at some specific, decisive moment.

But I think it’s important to note that there are strong mechanisms pushing in the direction of arbitrary escalation, and no strong mechanisms preventing it from doing so.

And if all-out war between nuclear powers doesn’t sound bad enough to you, remember this: war would be waged with much more advanced AI than we have today, and the war itself would further shape the incentives around the AI race.

Contra “stable multipolar scenarios”

Stable multipolar scenarios can happen in one of two ways: if AI’s efficacy at war has reached the limits of physics, or if AIs have a way to enforce a consensus (like in the good ending of “AI 2027”).

AI advantages compound, and if the gap is wide enough, one of the competitors (potentially a rogue AI) wins. It seems unlikely that AI’s ability to wage war will climb all the way to the limits of physics while the gap between the various actors never gets wide enough to conclude the conflict.

About AIs enforcing a consensus, roughly, I think this would require AI to already be vastly smarter and more competent than any human or existing human organization. Which makes this proposed “solution” kind of tautological: you still need to pass all the filters and build an ASI that you can trust.

As an example, in “AI 2027,” the two ASIs strike a deal by building a “consensus AI” that will forever enforce, to some degree, the preferences of both AIs. To do this, you’d need to have developed an extremely deep fundamental understanding of how to program AI, the kind of understanding that lets you write an AI as lines of code rather than a neural network.4

Due to the competitive pressures I talk about in this post, the plan would not unfold this way. Much, much earlier than when you’d be able to achieve such a deep understanding of AI, you’d achieve an understanding just barely good enough to build ASI. Then you would build it and thus destroy all value in the world, unless you already figured out a way past all the filters in this post.

An alternative proposal is to have AIs strike deals that are enforced through mutual monitoring. By the time AIs can strike such deals autonomously, they are already fairly superhuman and / or significantly in charge of running the world, and we need to have passed the filters.

To be clear, I don’t necessarily think it’s a bad idea to have weaker AIs help us enforce monitoring-based international agreements. But this needs to be done before AIs get too strong, at which point humanity would have to do it, even if aided by weaker AIs.

(My paper “Modeling the geopolitics of AI development” talks about this filter in more detail, but the thinking is less refined since it was written a while ago.)

Second filter: misaligned AI that kills everyone

It is probably very hard to build an ASI that doesn’t end up killing every human being simply by running it.

The basic argument is that ASI would be so effective that any failure, even partial, would result in an ASI handling extreme amounts of power while not going out of its way to preserve human life and values.

ASI would kill us as a side effect of whatever it ends up doing, just like a human destroys an anthill without a second thought when it’s in the way of a construction project. The field of making sure that ASIs act in a desirable way is called “AI safety.”

The threat model of misaligned AI is the one that has already been explored the most, so I will assume that readers are at least passingly familiar with it and won’t try to convey the basic idea here. If you need an introduction, read the book “If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky and Nate Soares.5

What I want to focus on here is how the pressure to cut corners I mentioned earlier makes it nearly impossible to solve alignment in time for when ASI will be developed. Think of the following competitive pressures:

• Pressure to cut corners on safety methods

• Pressure to deploy as fast as possible

• Pressure to give AI as much autonomy as possible

• Pressure to hand over existing decision loops to AI as quickly and thoroughly as possible

So what happens is, AI projects will develop and deploy AI that is as capable as possible given current capabilities techniques, while only being as safe as absolutely necessary to make them usable. The most important part here is: only as safe as absolutely necessary to make them usable. What does it mean? Well, the first instance of this pattern we’ll discuss is the commercial one.

Software engineers won’t use an AI that cheats to make the tests pass every time, but they’ll use an AI they can usually catch cheating, as long as the violations don’t fall through the cracks often enough for the engineer to get fired.

CEOs will not use AI employees that regularly take costly, irreversible actions to the point that the company loses a lot of money or it gets the CEO in trouble. But they will, for example, use AI that takes illegal actions as long as the company gets fined for less than the money it made, or the crime happens in a third-world country, etc.

So far it doesn’t sound like an extinction risk, but what is the “usability limit” when it comes to integrating AI in the military? What about tail risks, situations that are too rare and so haven’t yet appeared in the feedback loop of fixing AI bugs?

And most importantly, what happens when someone first gets to the capability level where they mostly hand over AI R&D to AIs themselves?

The AI will be just barely safe enough to profitably (not spotlessly!) do jobs that:

• Are roughly as complicated as AI R&D6

• Have short-enough feedback loops that failures have already happened, such that AI companies already have bug tickets for these failures

• Have already addressed these bug tickets

Of course, you will not be able to get this guarantee for novel tasks, such as AI R&D itself. Probably, you won’t even be able to get it for tasks that already exist but are not common enough for you to test the AI thoroughly on them during the (very brief) allotted time. You have to hope that whatever safety you have transfers from this small, nonrepresentative set of tasks to the ones that matter.

Why Technical AI safety agendas do not address this problem

Technical AI safety agendas for addressing extinction risks usually focus on the “misaligned AI that kills everyone” filter, so I have to briefly address why, as a general rule, they don’t work. In fact, they make things worse.

This is because all alignment work is capabilities work.

Take RLHF (reinforcement learning from human feedback), for example.7 RLHF improved “alignment,”8 but it also improved capabilities a lot more: the AIs that we built after RLHF were more liable to do dangerous things than the ones we built before it existed. This is true even if you do your best to use RLHF to make the model safe.

To the degree that interpretability and scalable oversight work, I confidently predict that they will do exactly the same thing.

The underlying, fundamental reason for this is that capabilities are easier to formalize than safety. By this I mean that capabilities are easier to measure and easier to describe to other people, to AIs, and to code without loss of information.

Imagine that we get an interpretability breakthrough. You would have more readability into the internal algorithms of AIs, but those algorithms are very big and complicated: you wouldn’t automatically know which parts are helpful and which are harmful.

Some will be obviously harmful and removed right away. What then? Maybe you can do some manual searches for patterns you suspect exist? But humans are slow. You can get AIs to help you, but AIs are not (yet) smarter than you, and so they’d miss some stuff. Perhaps AIs already have some misaligned biases and so would sometimes actively hinder your efforts.

On other hand, capabilities, oh how they’d skyrocket. Better interpretability would yield more powerful methods to modify AIs: it would allow engineers and learning algorithms to modify AIs in more targeted, deliberate, and understandable ways than can be done today.

Since capabilities are more formalized, you can quickly train a large team of engineers to make use of the novel techniques. Perhaps you can cut engineers out of this loop entirely, integrating the novel technique as part of automated learning algorithms.

If you want to modify AI to improve a quality that is hard to measure, like safety, you need a human to stand there and opine about each candidate modification. Worse still, the human needs to have good taste about the property you are trying to improve.

To summarize: capabilities can improve at machine speeds, while safety will always be bottlenecked by humans. The only way to solve this dilemma would be to describe our safety desiderata to the same level that we have described our capabilities desiderata. That way, we could potentially automate AI safety, or at least reliably train a team of engineers to do it. Good luck doing that during an all-out race to ASI!

I encourage you to think about this issue yourself, especially if you are a researcher at a major AI company working on a technical AI safety agenda. Your work may end up boosting capabilities more than most of the people over on the capabilities teams.

Third filter: nightmare singletons

Ok, imagine that the alignment problem is on track to get solved, such that a human being (or group of human beings) could operate an ASI without killing themselves and everyone else as a side effect. You and the rest of your team, the “responsible actors” in a world composed mostly of irresponsible ones, have the lead in AI development. You will build ASI first and then establish an eternal utopia, right? No.

Here’s what really happens: the government takes over your project before you get to ASI, by default as a military project, possibly top secret. You are questioned just enough that they know how to make use of the project’s assets (like code, documentation, hardware, etc.), and then you are thrown out the door.

Or maybe a softer version of this happens, where your AI company still technically exists. However, your CEO does not retain effective control of the company, and you have military personnel looking over your shoulder as you work.

If your government is asleep at the wheel, a foreign government will take over your project, or at least steal all the progress you’ve made so far and then pour their resources into going faster than you. Or if all governments are asleep at the wheel, another company will take over your project, or perhaps a charming psychopath CEO will manipulate their way into a top leadership position at the company where you work.

What then?

Whoever controls an ASI can establish a singleton. A singleton is a “world order in which there is a single decision-making agency at the highest level, capable of exerting effective control over its domain, and permanently preventing both internal and external threats to its supremacy.”

—

Let me spell out, for people who haven’t thought about this subject before, how nightmarish this scenario can get.

An individual in control of an ASI could establish a dictatorship that controls the entire earth, possibly the entire universe.

They could monitor every corner of their domain 24/7 and assign a virtually infinite amount of intelligence to analyze all of this information.

They could compel everyone to install brain implants (or forcibly upload them, etc.) and have complete oversight and control over their thoughts, actions, and experiences.

Eventually, they could shape the whole world to their preference until every atom is exactly as they want it, and do it as easily as a child shapes playdough.

—

In AI safety, some people’s strategy is to give power and resources to “good” or “responsible” actors, such as their favorite AI company.9 The theory of change for this strategy is that the “responsible” actor is the first to build ASI and establishes a utopian (or at least “good”) singleton.

I think that it is an enormous mistake to trust any one person or company with this. If your strategy is to use ASI to establish a “good” singleton, I will fight to prevent you from succeeding because I don’t trust you. But regardless, I hope this post makes you see that this strategy will break horribly.

If you are part of an ASI project and this is your plan, know this: someone more powerful than you will take your toys away before you get to ASI. Then, they will use them to race to ASI without you.

What happens later is fundamentally unpredictable. The result does not have to be as bad as the nightmare scenario I painted earlier. But from where we’re standing, it could easily get really bad.

I think what happens if any individual or small group of people obtains absolute power over the universe is an extremely dystopian scenario, potentially worse than death depending on your values. I think the same is true for scenarios in which we just barely make enough progress on alignment that ASI doesn’t kill us all as a side effect. ASI may want a future for us, but it could be a future that we find abhorrent, and it would have absolute power over us.10

Even in the best case scenario, where the ASI project is taken over by a government with a very robust democratic process, the situation would most likely be considered a national security emergency. Such emergencies are dealt with by the military (or more generally, the executive branch), which needs to be able to act quickly. As a result, it has weaker democratic oversight compared to other government branches.

What will this government do after having declared an emergency situation, armed with proto-ASI? Would you feel safe if you thought your government was bound to establish a singleton?

How common theories of change fail trivially

Any solution or theory that focuses entirely on technical AI safety fails trivially by not taking into account the two other filters. For example, some people think AI alignment will be easy to solve. I think this view is most likely mistaken on a very deep level. But even if it were correct, it would not address the other two problems at all.

Furthermore, I think that all technical AI safety projects will not be successful, not in a world where actors are able to unilaterally push the frontier of AI development toward ASI. This is due to the pressure to cut corners on safety, and because any technique will accelerate capabilities much more than it accelerates progress in AI safety.

The philosophy of “iterative deployment” will simply not apply in a world where the pace of deployment depends entirely on competitive pressure and is entirely causally disconnected from any consideration of what may be a “responsible” pace for AI development.

There are some who try to acquire personal power or influence so they can exert it “when the time comes.” This can mean attaining influence inside of AI companies or in governments. As I pointed to in the third filter, I think power within AI companies is meaningless.

And I think the people who try to acquire unilateral power within governments are deeply misguided. When push comes to shove, they will fail at gaining enough power to steer the actions of governments.

If the majority of the government does not understand the meaning of ASI, these people will not be able to make massively expensive and complex asks to leadership. For example: “slow down AI development to improve safety” or “pressure other major powers to enter a hefty trust but verify regimes capable of providing mutual assurances on AI development.” If these people try to push these asks without first building a broad support base (probably as broad as a decent voting bloc), then they will simply get purged.

Finally, there are people trying to get a major power to engage in a race to ASI, beat all their adversaries to it, and establish a singleton. I think these theories of change fail on literally all three filters:

• The world will likely be consumed by war before any actor can get to ASI.

• Even if we narrowly avoid all-out war, this theory of change leads to a race to the bottom on AI safety and to uncontrollable ASI that kills everyone.

• Even if the ASI ends up being somewhat controllable, no country on Earth currently has such institutional robustness that it would not produce a dystopia if it acquired ASI.

Conclusion

These were the main three challenges that I think stand between us and surviving ASI. Even if we pass all three, I don’t think things automatically go well.

I have more intuition pumps that I would like to publish in a future post. They are mostly about how, in scenarios with AI that is strong but not as strong as I’ve been implying ASI is, that:

• There is a strong tendency for power to concentrate and for the world to gravitate toward the three outcomes I’ve been describing.

• There is a tendency for human preferences and behavior to mutate beyond recognition, to a degree that we might think of such people as essentially “dead.”

The main way that I envision humanity passing these filters is with deep awareness of what ASI entails and with international coordination.

Deep awareness is necessary so the relevant parties understand what their interests are with respect to ASI. Chiefly, they need to understand that ASI can become powerful enough to destroy the world, and that it is indeed extremely hard to deploy an ASI without destroying the world.

Coordination, backed by mutual monitoring and deterrence, is necessary so the major parties can avoid a race to the bottom over who builds ASI first. Without it, they will end up developing and deploying ASI in the most irresponsible way possible, and thus destroy the world.

Both deep awareness and coordination are necessary so countries can eventually get to work to figure out how to go through this transition while avoiding the horrific failure modes I’ve described, and others yet.

At the moment, my best bet for achieving these goals is to work at ControlAI. If you’re interested in learning more about ControlAI, feel free to read our funding pitch, which also goes in detail about ControlAI’s theory of change. Alternatively, feel free to shoot me a message.

This post is also on LessWrong.