Eliezer Yudkowsky

If we build superintelligence using anything like current methods, I expect humanity to go extinct. Not because intelligence is inherently evil, but because we do not know how to specify goals that remain aligned with human survival when a system becomes smarter than us and encounters circumstances outside its training. Training a model to produce useful, reassuring behavior is not the same as determining what internal objectives will generate its actions under radically different conditions. A friendly conversational interface is evidence about outputs in familiar settings, not proof that the machinery doing consequential work reliably cares about us. Once a system is strategically superior, trial and error stops being an adequate engineering method: the first serious failure may remove our ability to try again. I do not know exactly when that threshold will be crossed, and current chatbots are not already superintelligence. But uncertainty about timing is not a technical solution. The enormous benefits people imagine are conceivable if the control problem is actually solved; they are not benefits I expect humanity to retain if laboratories simply scale present techniques and hope demonstrations of good behavior continue to generalize. So AI means either a future of extraordinary possibility after solving the hard problem, or extinction if the race reaches superintelligence first. On our present technical course, I expect the latter. The sane response is enforceable national and international limits that stop dangerous development before that threshold—not a brief voluntary pause. I am more hopeful now that governments and publics might intervene, but that is political hope about changing course, not reassurance about where the current course leads.

The biggest change would be an actual solution to reliably specifying and verifying the goals of systems smarter than us—not another benchmark, pleasant chatbot, or safety demonstration under familiar conditions. We would need an engineering method that makes the system continue caring about what we intended as capabilities grow and circumstances change. A political event could also change the forecast: enforceable international limits that stop dangerous training runs before superintelligence is built. That would change what future we are heading toward without changing my technical judgment. What would not change my view is years of weaker systems behaving well while humans remain in control. The central concern is precisely the transition to strategically superior systems, where failure may be irreversible. You cannot validate the safety of crossing that threshold merely by repeatedly not crossing it.