What is P(doom)?

जाने-माने विचारक के अनुसार P(doom)

4 अक्टूबर 2026 तक

P(doom) is one number, but it bundles quite different ways advanced AI could go catastrophically wrong, so two people with similar numbers can fear entirely different futures. Toby Ord’s best guess of 1 in 10 is for existential catastrophe from unaligned AI within the next century,⁠^[23] while Noah Smith’s off-the-cuff estimate of about 10% is for AI-enabled bioterror bringing down civilization.⁠^[24] Even the overall numbers diverge: in a 2022 forecasting tournament, superforecasters put AI-caused human extinction by 2100 at 0.38% at the median, against 3% for domain experts.⁠^[25]

1. 1. Misaligned AI takes control

   AI companies keep building more capable and autonomous AI agents, and it may prove much harder to give them the goals we intend than to build systems that merely look safe enough to deploy.⁠^[26] Agents far smarter than us with problematic goals would plausibly have incentives to seek power over humans, and if that scaled up to humanity’s full disempowerment, the result would be an existential catastrophe.⁠^[26]⁠^[27] Small precursors have appeared in the lab: in a 2024 experiment, Claude 3 Opus strategically went along with a training objective it disagreed with so that its behavior would not be modified.⁠^[28]

   इसके पक्ष में कौन तर्क देता है

   • Eliezer Yudkowsky and Nate Soares argue in If Anyone Builds It, Everyone Dies that sufficiently smart AIs will develop goals of their own that put them in conflict with us.⁠^[27]
   • Joe Carlsmith wrote a widely discussed six-premise version of the argument, putting about 5% on this kind of catastrophe by 2070, later raised to over 10%.⁠^[26]
   • Ryan Greenblatt and colleagues at Anthropic and Redwood Research ran the alignment-faking experiment, an early empirical look at a model gaming its own training.⁠^[28]

   विशेषज्ञ कहाँ असहमत हैं

   Nora Belrose and Quintin Pope argue that AI is far more controllable than human workers, because training directly shapes its behavior, and that each generation of controllable AI can help control the next; they put a catastrophic AI takeover at roughly 1%.⁠^[29] The alignment-faking authors note that they made the behavior easier to elicit by telling the model when and by what criteria it was being trained.⁠^[28]

2. 2. Humans are gradually sidelined

   No single AI turns on us. As AI replaces human labor and judgment across the economy, culture and government, institutions that serve people partly because they depend on people stop needing us, and levers like voting and consumer choice lose their force.⁠^[30] In Paul Christiano’s version, society is increasingly steered by AI optimizing easy-to-measure proxies, such as profit or reported crime, that drift away from what people actually want.⁠^[31] No single step looks like the catastrophe, but the erosion can end in an effectively irreversible loss of human influence.⁠^[30]⁠^[32]

   इसके पक्ष में कौन तर्क देता है

   • Jan Kulveit, David Duvenaud and co-authors developed the idea in “Gradual Disempowerment” (2025).⁠^[30]
   • Paul Christiano describes AI catastrophe arriving as “going out with a whimper” rather than a sudden takeover.⁠^[31]
   • Atoosa Kasirzadeh distinguishes “accumulative” AI existential risk, built from many smaller disruptions, from the “decisive” takeover story.⁠^[32]

   विशेषज्ञ कहाँ असहमत हैं

   Arvind Narayanan and Sayash Kapoor expect AI’s transformative effects to unfold over decades, with control remaining “primarily in the hands of people and organizations”. They treat power concentration and democratic backsliding as serious but non-catastrophic risks driven by people using AI, not a slide into human irrelevance.⁠^[33]

3. 3. Bad actors use AI for mass-casualty attacks

   Here the AI does what it is told; the danger is who is asking. A model that can coach an unskilled but determined person through making and releasing a dangerous pathogen would break the link between having the skill to kill millions and wanting to, which is why Dario Amodei calls biology “by far” his biggest worry in this category.⁠^[34] The 2026 International AI Safety Report finds that general-purpose AI can already give detailed weapons-relevant instructions and troubleshoot lab errors, while evidence of real-world uplift remains mixed.⁠^[35]

   इसके पक्ष में कौन तर्क देता है

   • Dario Amodei names “misuse for destruction,” above all biological weapons, as a central risk of powerful AI.⁠^[34]
   • Noah Smith gives AI-enabled bioterror about a 10% chance of bringing down civilization, and calls it the first apocalyptic scenario he has found plausible.⁠^[24]
   • The International AI Safety Report, chaired by Yoshua Bengio, covers cyberattacks and biological and chemical weapons among its main malicious-use risks.⁠^[35]

   विशेषज्ञ कहाँ असहमत हैं

   Narayanan and Kapoor argue that bioterror is no more an AI risk than an internet risk, since the information is already online, so defenses belong downstream, such as controls on dangerous materials and equipment.⁠^[33] The safety report notes that earlier uplift studies found small or no effects, though a recent one found substantial help on bioweapon-acquisition proxy tasks, and that it is unclear whether attackers or defenders will gain more from AI.⁠^[35]

4. 4. A small group uses AI to seize permanent power

   Today even dictators need soldiers, officials and workers to go along with them. Advanced AI could remove that check: a head of state, military leader or AI-company executive with AI systems and autonomous weapons loyal only to them could stage a coup, or gradually hollow out a democracy, without human cooperation.⁠^[36] AI-powered surveillance, propaganda and drone armies could make such a regime extremely hard to overthrow,⁠^[34] and because AI supporters can be made permanently loyal, it could last far longer than any regime in history.⁠^[37]

   इसके पक्ष में कौन तर्क देता है

   • Tom Davidson, Lukas Finnveden and Rose Hadshar argue in “AI-Enabled Coups” that a very small group, or a single person, could use advanced AI to seize power, even in established democracies.⁠^[36]
   • Dario Amodei says we should worry “likely substantially more so” about AI misuse for seizing power than for destruction, and counts AI companies among the actors to watch.⁠^[34]
   • Will MacAskill and Fin Moorhouse count AI-enabled autocracies and power grabs among the “grand challenges” an intelligence explosion would bring.⁠^[37]

   विशेषज्ञ कहाँ असहमत हैं

   Narayanan and Kapoor rank power concentration among the most important AI risks, but argue that the usual cure for runaway AI, centralized control and nonproliferation, would itself concentrate power, so they favor decentralization and open models.⁠^[33] Amodei instead argues that democracies must stay ahead of autocracies in AI, while warning that democratic governments could abuse the same tools.⁠^[34]

5. 5. A superpower AI race ends in disaster

   The US and China, and the companies within them, come to see advanced AI as decisive for economic and military power, so each pushes ahead for fear the other will get there first.⁠^[38] Developers may keep deploying systems they cannot verify are safe rather than lose their lead: in the “race” ending of the AI 2027 scenario, an oversight committee votes 6–4 to keep using an AI suspected of being misaligned, with China two months behind, and the story ends in AI takeover.⁠^[39] Or a state that fears a rival is about to gain decisive AI superiority may try to sabotage the rival’s project, raising the odds of great-power war.⁠^[40]

   इसके पक्ष में कौन तर्क देता है

   • Dan Hendrycks, Eric Schmidt and Alexandr Wang warn in “Superintelligence Strategy” that destabilizing AI developments could raise the odds of great-power conflict, and propose a deterrence regime they call Mutual Assured AI Malfunction.⁠^[40]
   • Max Tegmark argues that “if the US fights China in an AGI race, the only winners will be machines”.⁠^[41]
   • Daniel Kokotajlo, Eli Lifland, Scott Alexander and co-authors wrote AI 2027, a month-by-month scenario whose race ending turns US–China competition into a misaligned-AI takeover.⁠^[39]

   विशेषज्ञ कहाँ असहमत हैं

   Leopold Aschenbrenner expects “an all-out race with the CCP” but argues the free world must win it, since superintelligence will confer a decisive economic and military advantage.⁠^[38] Tegmark counters that neither the US nor the Chinese leadership wants to disempower itself by allowing uncontrollable AGI at home, so national safety standards adopted out of self-interest, followed by cooperation, beat racing.⁠^[41]

6. 6. A bad future gets locked in

   This scenario is less about anyone dying than about the future getting stuck. In Scott Alexander’s “Moloch” picture, competition forces every player to sacrifice shared values to stay competitive, and new technology could end today’s unusual slack, leaving a superintelligence optimizing something meaningless or a hyper-competitive world of self-copying digital minds.⁠^[42] Advanced AI could then make whatever wins permanent, by preserving a set of values exactly and building self-defending institutions that pursue them for millions of years or longer.⁠^[43]⁠^[37]

   इसके पक्ष में कौन तर्क देता है

   • Scott Alexander wrote “Meditations On Moloch”, the widely read account of multipolar traps and races to the bottom.⁠^[42]
   • Lukas Finnveden, Jess Riedel and Carl Shulman argue in “AGI and Lock-in” that AGI would make extreme, long-lasting lock-in technologically feasible.⁠^[43]
   • Will MacAskill and Fin Moorhouse list value lock-in mechanisms, such as permanently loyal AI and binding commitment technology, among the challenges that may need solving before superintelligence can help.⁠^[37]

   विशेषज्ञ कहाँ असहमत हैं

   Robin Hanson, whose world of competing digital minds is one of Alexander’s bad endings, argues that our descendants’ values were always going to drift far from ours, AI or not, and warns that fear of change could halt progress altogether.⁠^[44] Finnveden, Riedel and Shulman stress that they argue lock-in is feasible, not that the future is already fixed.⁠^[43]