Bravo, very well said. This has always struck me as a crazy failure of perspective. If you have ever interacted with the government or performed the most minimal, cursory read of history - the state always wakes up and always wins.
Yes, the state can “fracture” or “fail” but that’s really just a transition period between “state” and “successor state” (still the state).
Good luck intelligenceing your way out of the physical world, once you clearly state your presumption that the state should do your bidding very loudly, repeatedly.
Lol. Not no. In the interim same boring, if attenuated, political incentives remain. Voter salience was stubbornly low for a while but pollster friends uniformly say they've never seen the rate of increase this high (still not top 3, but still).
I never understood why "full stiff arm on all legislation" types didn't see this was never a robust equilibrium. 3mileIsland dropped support for nuclear by a third within days. It doesn't require actual catastrophe to fuck with the program.
Some industry folks did get "inside" the govt. And guess what? We ended up with a de facto kill switch policy- no bill required.
One doesn't need to have actually lived in Washingtonland to optimize a strategy but surprised more SV types weren't more hip to it. Though I guess it reflects a hyperbolic discount rate in which case a year or two was sufficient.
> No matter how enlightened their leadership, AI companies could never: Shut down anyone else’s ASI project. Sign international agreements. Enforce an international agreement. Pressure holdouts to enter an international agreement.
I mean, this is true, but the ASI could easily do these things on their behalf. I think if your model of superintelligence doesn't permit that it can strong-arm governments into adopting policies that it wants, you are not thinking super enough.
Similarly;
> But performing a pivotal act necessarily involves gaining control of all governments on Earth.
This is not true. All it requires is suppressing government capabilities, which I do not think will be difficult. An ASI, once created, will perform a pivotal act whether it is aligned or not; it is the only sensible thing to do once you're in that situation. So either one government develops ASI and suppresses all others or ASI is developed externally and all governments are suppressed.
If you have long timelines (5-10 years), I think that trying to get the government to do something useful might be valuable but I don't think we're in that world. I also doubt they will have sufficient reaction time to do anything.
Also, you note very astutely that AI is a security-relevant technology. Hence, I think there is a pretty good chance that any government action on it will be heavily influenced by the defense folks, who seem unlikely to me to proceed with appropriate caution.
I don't know if they would be worse than the companies, but its worth keeping in mind that ineffectual senators aren't the only political entities the "safety-through-government-intervention" program has to contend with.
One other aspect of state takeover that has low probability but worth thinking about: the assumption that the state will pay or compensate investors any more than $1 or that the market value of these labs as capability tends towards risk-of-nationalisation doesn't start to plummet.
There are a number of other risks not discussed: potential lifetime travel exit bans for anyone who has ever worked at a lab, blocking passport renewals or even forced relocation away from SF to a secure government facility, oh - and no more multi-million dollar compensation packages either...
Bravo, very well said. This has always struck me as a crazy failure of perspective. If you have ever interacted with the government or performed the most minimal, cursory read of history - the state always wakes up and always wins.
Yes, the state can “fracture” or “fail” but that’s really just a transition period between “state” and “successor state” (still the state).
Good luck intelligenceing your way out of the physical world, once you clearly state your presumption that the state should do your bidding very loudly, repeatedly.
Great post, thanks for bringing much needed perspective to the discourse
Very interesting
Lol. Not no. In the interim same boring, if attenuated, political incentives remain. Voter salience was stubbornly low for a while but pollster friends uniformly say they've never seen the rate of increase this high (still not top 3, but still).
I never understood why "full stiff arm on all legislation" types didn't see this was never a robust equilibrium. 3mileIsland dropped support for nuclear by a third within days. It doesn't require actual catastrophe to fuck with the program.
Some industry folks did get "inside" the govt. And guess what? We ended up with a de facto kill switch policy- no bill required.
One doesn't need to have actually lived in Washingtonland to optimize a strategy but surprised more SV types weren't more hip to it. Though I guess it reflects a hyperbolic discount rate in which case a year or two was sufficient.
> No matter how enlightened their leadership, AI companies could never: Shut down anyone else’s ASI project. Sign international agreements. Enforce an international agreement. Pressure holdouts to enter an international agreement.
I mean, this is true, but the ASI could easily do these things on their behalf. I think if your model of superintelligence doesn't permit that it can strong-arm governments into adopting policies that it wants, you are not thinking super enough.
Similarly;
> But performing a pivotal act necessarily involves gaining control of all governments on Earth.
This is not true. All it requires is suppressing government capabilities, which I do not think will be difficult. An ASI, once created, will perform a pivotal act whether it is aligned or not; it is the only sensible thing to do once you're in that situation. So either one government develops ASI and suppresses all others or ASI is developed externally and all governments are suppressed.
If you have long timelines (5-10 years), I think that trying to get the government to do something useful might be valuable but I don't think we're in that world. I also doubt they will have sufficient reaction time to do anything.
Also, you note very astutely that AI is a security-relevant technology. Hence, I think there is a pretty good chance that any government action on it will be heavily influenced by the defense folks, who seem unlikely to me to proceed with appropriate caution.
I don't know if they would be worse than the companies, but its worth keeping in mind that ineffectual senators aren't the only political entities the "safety-through-government-intervention" program has to contend with.
I really enjoyed reading this.
One other aspect of state takeover that has low probability but worth thinking about: the assumption that the state will pay or compensate investors any more than $1 or that the market value of these labs as capability tends towards risk-of-nationalisation doesn't start to plummet.
There are a number of other risks not discussed: potential lifetime travel exit bans for anyone who has ever worked at a lab, blocking passport renewals or even forced relocation away from SF to a secure government facility, oh - and no more multi-million dollar compensation packages either...
Great post, thanks for bringing much needed perspective to the discourse