Scott Alexander

Responds to Pinker’s objections to AI risk: capability growth need not imply omniscience; instrumental goals, misgeneralization and reward hacking can produce power-seeking without evolved dominance drives; multiple goals do not guarantee human-compatible behavior; adoption incentives create infrastructure access. Reports his previously stated personal extinction estimate as about 25–30%, with no fixed horizon or explanation of the difference from June’s rounded 20%. Defends subjective forecasting, cooperation on near-term harms and precaution about AI welfare. Favors time and resources for alignment leading to safe superintelligence over a permanent ban. Read the full publisher text; quoted opponents’ views, other people’s probabilities and satirical dueling rhetoric are not his policy. Linked studies and incidents are his attributed evidence, not independently verified findings.

His current first-person synthesis: AGI means ability to do 90% of knowledge jobs; median 2034, with uncertain research acceleration and diffusion. Reaffirms rounded 20% P(doom), with no fixed calendar deadline; broader permanent curtailment is separate. Supports both alignment research and negotiated slowing. Expects enormous postscarcity upside, but warns about dictatorship and human disempowerment.

Explains interpretability techniques and their limitations, including probes, sparse autoencoders, and activation verbalizers. Optimistic about useful practical investigation but rejects treating a detected feature or probe as a complete understanding or guaranteed safety solution.

Explicitly neutral about banning open weights now: values user ownership and freedom from corporate control, while expecting serious misuse difficulties. Prefers saving political capital for threats where warning shots may arrive too late. Distinguishes reactive policy opportunities for misuse from strategically concealed takeover.

Defends negotiated chip regulation and verifiable training limits against blanket claims of dystopia. Acknowledges real freedom costs, including future restrictions on new open-weight training, and risks that governments implement centralizing provisions without countervailing diffusion of power.

Introduces a proposed route to manage AI development while distributing power; criticizes vague calls merely to regulate more or less without specifying a desirable end state. Used as his attributed introduction and advocacy, not evidence that the scenario will occur.

Argues cheaper capable forecasting could improve institutional and personal decisions, yet worries people will ignore advice. Treats forecasting beyond human performance as a useful prospective test of the normal-technology view. Distinguishes anecdotes and startup claims from head-to-head competitions; admits resisting forecasts that challenge his own pause hopes.

Rejects the inference that requiring a new AI paradigm implies a safely distant AGI timeline. Argues paradigm changes can arrive soon and inherit existing compute infrastructure; wants explicit bottleneck arguments rather than reassurance by terminology.

Agrees growth cannot stay exponential forever but disputes placing the bend conveniently before dangerous capability. Demands a causal bottleneck model or a defensible forecasting prior instead of the slogan that all exponentials eventually flatten.

Satirical dialogue defends discussion of transparent, enforceable bilateral US-China slowing. Separates training limits from stopping existing inference, and legitimate negotiation or enforcement objections from falsely describing every pause proposal as unilateral.

Frames confident false answers as reward-shaped guessing rather than proof that AI cannot think. Treats the gap between trained reward and useful honest advice as an alignment issue; analogous human failures undermine easy dismissal of AI competence.

Separates training objectives from the representations and algorithms they produce, using evolution and human learning analogies. Argues next-token prediction does not itself establish that a system lacks reasoning or world models.

Identifies his part-time writing/publicity contribution and explicitly says the very fast scenario is not his median. Important provenance for his connection to AI Futures Project; use June 2026 personal forecasts instead of importing Daniel Kokotajlo’s timeline or scenario catastrophe probability.