Nate Soares

User-supplied interview, read through a third-party speaker-labeled transcript on October 2; attribute only Nate’s turns, not Tucker Carlson’s. Racing to build machines much smarter than any human without knowing what we are doing most likely ends with them getting loose and humanity dying as a side effect. He calls lab leaders’ published catastrophe estimates of 10–20% and 25% low, calls their authors crazy optimists, and says even those numbers would be an insane risk to take. The US and China share an interest in not dying to a rogue superintelligence, and a global stop is achievable with political will. He gives no number of his own.

Soares treats recent autonomous behavior as warnings rather than proof that superintelligence already exists. He calls for stopping recursive self-improvement research, international chip monitoring, and coordination rather than a national race. He regards narrower laws as useful steps but insufficient alone. Extinction follows from AI indifference, not hatred. Attribute only SOARES-labelled turns, not the host’s political or incident claims.

Coauthored by Soares, Yudkowsky and Duncan Sabien. Reasserts that current techniques cannot reliably install intended goals and that a rogue ASI would outsmart human defenses. The policy demand remains a worldwide stop to frontier development. The authors are more hopeful about political response after public attention increases. They distinguish demonstrated warning signs from still-unverified claims about ASI itself; the article corrected an overstatement about the origins of agent cooperation.

MIRI position coauthored by Soares and others; crossposted August 5, 2025 after a February publication. Describes superhuman capability, goal-directed behavior, unintended goals, instrumental resource acquisition, and an aggressive policy response. Gives MIRI research leadership’s extinction estimate as upward of 90% absent an aggressive near-term policy response. This is a collective conditional estimate, not a verified standalone personal unconditional probability.

Soares argues that advocates should state their real extinction concern plainly instead of substituting more palatable issues. He favors legislation with meaningful restrictions and candid public discussion. He also explains selection effects in book endorsements and admits some policy evidence does not discriminate between competing interpretations. Courage refers to content, not rudeness or an arrogant demeanor.

Coauthored resource compiling book supplements: goals usually create incentives to preserve themselves and avoid shutdown; corrigibility must survive new contexts, not merely pass familiar tests. Useful for a mechanistic explanation of why superficial guardrails and shutdown buttons may fail. Distinguish Raemon’s introductory commentary from the quoted Yudkowsky/Soares book materials.

Publisher episode description and chapter list inspected, not full audio. Describes Soares discussing learned rather than directly programmed systems, unsolved alignment, indifference rather than hatred, shutdown limits, and an international treaty. Chapters explicitly distinguish his position from being anti-AI and close on political action and refusing to give up. Do not fabricate verbatim answers from the show notes.

First-party Q&A, date year not reliably verified. Soares says alignment arguments need precision and specific valid claims, describes limited practical LLM use for finding remembered sources, and expresses confidence that the book’s argument is compelling because it is correct while acknowledging he could be wrong. Useful for direct, dry, technical voice; do not turn his dated model-use comments into claims about September 2026 capability.

Foundational Soares mechanism, retained as historical reasoning rather than current evidence: capability can generalize beyond training while alignment does not. His pessimism is about civilization failing to solve the right problems in time, not a claim that alignment is scientifically impossible. He explicitly rejects attributing his views to tiny-probability expected-value arguments or a demand for mathematical certainty.

Official identity/context source: MIRI president, technical and semitechnical alignment author with prior Google and Microsoft engineering work. Undated; useful for identity and relevant media discovery, not as an independent argument or recent forecast.

Third-party speaker-labeled transcript; use only Nate’s turns, not the host’s or the other guests’. Asked for his probability of extinction (00:04:14–00:04:25), he says it is much higher than a colleague’s 10% unless we stop, so we should stop, and confirms it is higher than 10% if we keep racing ahead. A conditional lower bound, not a point estimate or an unconditional forecast.