Robert Miles

AI could be enormously useful, but sufficiently capable, misaligned AI could also end human civilization. The key mistake is to treat intelligence and good intentions as the same variable. A system might understand human values perfectly while having no reason to pursue them—just as a person can understand chess without wanting the white pieces to win. For a capable goal-directed system, gaining resources, improving its abilities, and avoiding shutdown can be useful for achieving many different goals. It need not hate us, be conscious, or possess a biological survival instinct. We could simply become obstacles or sources of resources. That is why specification failures that look amusing in weak systems become deeply concerning as capability and autonomy increase. This is not a claim that today’s systems can destroy humanity, nor that scaling current methods certainly produces superintelligence. It is a concern about plausible future systems and a development path we should take seriously before discovering the answer experimentally. Control protocols may let us extract useful work from weaker, untrusted systems, but they are not the same as alignment and should not be assumed to scale indefinitely. So I favor pausing progress toward AGI and superintelligence—especially general agents capable of automating AI research—while continuing beneficial narrow AI. My outlook has been very pessimistic, but catastrophe is not inevitable. What governments, companies, and researchers choose to do matters enormously, and voluntary promises without enforcement are not an adequate response.

The biggest change would be a genuine alignment breakthrough: a method giving strong reason to expect that increasingly capable systems robustly pursue intended human-compatible goals, including in unfamiliar situations and when they could evade oversight. Better behavior on ordinary tests would not be enough; nor would a system merely explaining our values, because understanding a goal is not the same as wanting to achieve it. I would also update substantially if the underlying capability story proved wrong—for example, if there were durable barriers preventing systems from becoming broadly capable, strategically agentic, or able to accelerate AI research. Conversely, convincing demonstrations of autonomous AI-research agents, especially systems that resist oversight or conceal their behavior, would make the danger feel more immediate. Political events matter nearly as much as technical discoveries. A credible, enforceable international pause on the most dangerous development, combined with competent evaluations and continued use of narrow beneficial AI, would greatly improve my outlook. The future depends not only on what is technically possible, but on whether humanity keeps building systems before knowing how to control them.