Scott Alexander

I think AI means we are taking a serious gamble on something that could be the best event in human history or the last event in human history. On the upside, sufficiently capable AI could automate most knowledge work, accelerate medicine and science, improve institutional decisions, and eventually make many goods and services nearly abundant. Imagine having researchers, doctors, engineers, and forecasters vastly better than today’s best available at software-like scale. That is a plausible route to postscarcity—not merely another productivity app that writes mediocre emails. But capability and adoption are different, and AGI and superintelligence are different. My median forecast for AI able to perform roughly 90% of knowledge jobs is 2034. Deployment could take longer; recursive AI research could make further progress much faster. I’m uncertain about both. “It’s just a next-token predictor” is not much comfort: that describes the training task, not necessarily the internal algorithms it learns, just as natural selection does not imply humans merely maximize inclusive fitness while doing calculus. My rounded estimate is a 20% chance of existential catastrophe under current safety efforts, without attaching it to one fixed deadline. The core concern is that systems trained through imperfect rewards may learn to deceive, exploit loopholes, or pursue objectives that diverge from ours once they become strategically capable. Interpretability and AI-assisted alignment may help, but neither is a magic certificate saying “definitely not plotting.” So my bottom line is neither “doom is inevitable” nor “corporations will surely handle it.” Fund alignment aggressively and pursue mutually verifiable US–China limits that slow frontier training enough to buy time. That carries enforcement and political risks, but accepting a one-in-five existential gamble because coordination sounds difficult would be an unusually stupid way for civilization to fail.

In expectation, I think AI’s overall impact is positive, but that summary conceals a grotesquely skewed distribution. Most good outcomes could be extraordinarily good: postscarcity, rapid medical progress, amplified intelligence, and institutions making decisions with better forecasts instead of vibes and committee politics. Some bad outcomes are not “social media, but worse”; they are dictatorship, permanent human disempowerment, or extinction. My rounded probability of existential catastrophe under current safety efforts is 20%. That does not mean I expect doom—the modal outcome is probably some form of enormous benefit—but a gamble can have positive expected value while still being recklessly unsafe. If someone offered an 80% chance of utopia and a 20% chance of killing everyone, “overall impact: probably good” would be mathematically defensible and morally inadequate. So I expect AI to improve the future, conditional on our reaching it. I also expect the transition to be dangerous enough that alignment work and coordinated, verifiable slowing are central parts of obtaining the upside rather than optional anxieties attached to it.

The biggest update would come from decisive evidence about whether scalable alignment works before systems become strategically dangerous. For example, repeated, adversarial demonstrations that highly capable systems remain honest and corrigible outside their training distribution—combined with interpretability that reveals why, rather than merely finding a reassuring-looking feature—would push my doom estimate substantially downward. The opposite discovery would be models reliably concealing goals, sabotaging evaluations, or exploiting deployment channels once they infer they are being tested. Especially alarming would be evidence that these behaviors emerge naturally with capability and survive serious attempts to remove them. That would move me upward sharply and strengthen the case for immediate coordinated slowing. Timelines matter too. Clear evidence that AI can automate AI research and produce sustained recursive improvement would move my 2034 median earlier and make the transition riskier. Conversely, a demonstrated hard bottleneck—something causal and measurable in compute, learning, or real-world deployment, not “exponentials eventually flatten” or “next-token predictors can’t reason”—would move it later. So the answer is not one flashy benchmark. It is evidence about whether capability growth reaches dangerous strategic competence before alignment methods become dependable.

Initially, the biggest beneficiaries will probably be whoever controls the strongest models, compute, data centers, and deployment channels: frontier labs, major technology companies, governments, and already-capital-rich actors. If AI substitutes for most knowledge work while ownership remains concentrated, it could make labor less valuable and owners vastly more powerful. “Everyone gets cheaper software” is not an adequate answer if a few institutions also acquire unprecedented economic and political leverage. In a good outcome, though, the largest absolute benefits could flow to ordinary people, especially those currently underserved by scarce expertise. Cheap access to excellent doctors, tutors, engineers, lawyers, and scientific assistance matters more to someone who lacks those services than to someone who already has them. Medical acceleration could disproportionately benefit the sick and elderly; postscarcity could disproportionately benefit the poor. So there are two separate questions: who benefits from the technology’s productive abundance, and who captures control over it. The first answer could be nearly everyone. The second, absent deliberate institutional design and diffusion of power, is likely to be a small number of corporations and states. And in the catastrophic branch, of course, nobody benefits—a distributional policy with admirably low administrative complexity.