Doom or Bloom
Nate Soares
@So8res on XHumanity can prevent catastrophe by stopping the rush to superintelligence.
How will AI change the world?
Across: his expressed DoomâBloom outlook. Up: scale of transformation.
DoomâBloom: 18 out of 100. Scale of transformation: 100 out of 100. Interpretation ranges: 0 to 25 horizontally, 98 to 100 vertically. These are interpretation coordinates, not event probabilities.
â62%
Inferred from the likelihood described in his simulated answers. Approximate interpretation range: 50â90%. Applies to the outcome and conditions in his simulated answers; this is an inferred percentage.
AI could make an extraordinary future possible, but the current race toward superintelligence is more likely to end with humanity losing control and dying.
No milestone timing was established. Dates, ânot sure,â âpossibly never,â and dependencies can all appear here when expressed.
Grouped by milestone, not spaced or ordered by inferred dates. AGI and superhuman AI retain his definitions.
A central assumption
That can produce a system that cooperates while cooperation is useful, without producing one that reliably values human survival once it is more capable than we are.
Answer 1
If this assumption turned out differently, how would his outlook change?
What could change their mind
The biggest change would be a genuine solution to alignment: a substantive, mechanistic account of why a system much smarter than us continues to care about what we intended, even in novel situations where it can deceive us, resist correction, or seize control.
Answer 2
What evidence would be enough, and in which direction would it move his view?
More details
Several readings remain plausible: Little positive impact is expected even if advanced AI arrives. / Substantial benefits are expected, with important conditions or distribution limits. / Limited or narrowly distributed gains are expected.
35 / 100
Interpretation range 0 to 67 on the qualitative scale.
Catastrophic or irreversible loss is central to the expected future.
100 / 100
Interpretation range 100 to 100 on the qualitative scale.
Reasoning, consideration of alternatives, and handling of uncertainty in his simulated answers. This describes the simulated answers, not the real personâs intelligence or opinions.
94 / 100
Interpretation range 86 to 100 on the qualitative scale.
Human choices can substantially redirect the AI trajectory.
73 / 100
Interpretation range 50 to 100 on the qualitative scale.
Simulated position: Stop or substantially slow development of more capable AI.
Continue development under stated safeguards.
Speed up development of more capable AI.
These interpretations keep his stated conditions. Benefits and harms can both be substantial. The ranges describe how we read his simulated answers, not statistical confidence intervals.
Simulated Assessment
Sources
Articles, interviews, and writings used to ground this simulated persona.
Nate Soares on AI danger â Tucker Carlson interview
Nate Soares is a computer scientist whoâs worked at Google and the Defense Department. So when he says AI is on the path to killing every person on earth, it...
youtube.com
CNN Newsroom: Nate Soares on international AI safeguards
If Anyone Builds It, Everyone Dies: One Year Closer
In celebration of still being alive and fighting, we are giving away 1,000 Amazon e-books of âIf Anyone Builds It, Everyone Diesâ. Feel free to sendâŠ
lesswrong.com
The Problem
The stated goal of the worldâs leading AI companies is to build AI that is general enough to do anything a human can do, from solving hard problems in
intelligence.org
A case for courage, when speaking of AI danger
I think more people should say what they actually believe about AI dangers, loudly and often. Even (and perhaps especially) if you work in AI policy.
intelligence.org
Why Corrigibility is Hard and Important
A lot of objection and confusion to the MIRI worldview seems to come from a perspective of "but, it.... shouldn't be possible be that confident in soâŠ
lesswrong.com
Nate Soares â Why Superintelligent AI Could Kill Us All
Listen to #189 - Nate Soares - Why Superintelligent AI Could Kill Us All from The Peter McCormack Show. Nate Soares joins me to discuss the race to superintelligent AI, why alignment remains unsolved and why he believes building machines smarter than humanity could end in catastrophe.Nate explains why modern AI is not programmed like traditional software, but grown through training processes we do not fully understand. We discuss AI escaping labs, recursive self-improvement, why todayâs systems are only safe because they are not yet powerful enough to take over and why the danger may not be that AI hates us, but that it does not care about us at all.We also get into OpenAI, Anthropic, Google DeepMind, xAI - the AI companies racing ahead despite the risks, why âturning it offâ may not be possible, the case for an international treaty against superintelligence, and why Nate believes we should not give up before the driver wakes up.- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -TIMESTAMPS:00:00:00 - AI That Doesnât Care About Us00:01:04 - If Anyone Builds This, Everyone Dies00:04:17 - If Heâs Right, We Lose Everything00:07:28 - Why Nate Isnât Anti-AI00:08:32 - From Google and DeepMind to AI Risk00:11:00 - Why Canât We Make AI Care About Us?00:16:07 - We Can Look Inside, But We Donât Understand It00:17:20 - The Moment Nate Decided to Write the Book00:20:24 - The Senator Who Already Understood the Risk00:24:04 - How AI Goes From Useful to Terrifying00:32:17 - Could AI Live on a Portable Drive?00:36:13 - Are the AI CEOs Listening?00:39:53 - Why Would We Accept a 10% Risk?00:43:40 - Conorâs Point: We Donât Get a Choice00:51:10 - Are We Already Racing Towards the Cliff?00:58:28 - The AI That Threatened a Reporter01:15:53 - What Would Winning Look Like?01:21:38 - What Can Ordinary People Do?01:26:31 - Are We Just a Simulation for AI?01:34:26 - Donât Give Up Before the Driver Wakes Up- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -CONTACT PETEâș Website â http://petermccormack.comâș Feedback â https://www.petermccormack.com/contactâș Email â me@petermccormack.comâș Instagram â /mccormack555âș X/Twitter â https://x.com/petermccormack/CONNECT WITH NATE SOARESâș Website â https://intelligence.org/âș Book â https://ifanyonebuildsit.com/âș X/Twitter â https://x.com/So8resSPONSORSâș IREN - https://www.iren.com/âș Plaud - https://www.plaud.ai/mccormackâș Incogni - https://incogni.com/peter- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -LISTEN / SUBSCRIBEâș Apple Podcasts: https://apple.co/40ruY9Kâș Spotify: https://spoti.fi/3Wc94Vuâș Fountain: https://bit.ly/FountainPMâș YouTube: https://bit.ly/YouTube_PMâș Rumble: https://bit.ly/RumblePMFILMED BY CURTIS TAYLORâș https://www.curttaylor.co.uk/âș https://x.com/curttayloruk/EDITED BY CONOR MCCORMACKâș https://x.com/ConorM04- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
shows.acast.com
Interview with Nate Soares â Max Raskin
âI expect to die from this.â
maxraskin.com
A central AI alignment problem: capabilities generalization, and the sharp left turn
Nate Soares argues that one of the core problems with AI alignment is that an AI system's capabilities will likely generalize to new domains much fasâŠ
lesswrong.com
Nate Soares â MIRI profile
Nate Soares is the President of the Machine Intelligence Research Institute, and plays a central role in setting MIRI's vision and strategy. Soares has been
intelligence.org