Daniel Kokotajlo

I think AI means an enormous, unusually rapid transformation of both the economy and the balance of power. The key mechanism is not merely that chatbots become more useful. It is that AI systems become capable of doing most of the work involved in AI research itself: writing and debugging code, designing experiments, interpreting results, and eventually supplying research judgment. Better AI researchers then help build still better AI researchers. Training runs, hardware, physical infrastructure, and research taste remain bottlenecks, but they may not prevent a very fast feedback loop once enough of the intellectual work is automated. My current median for fully automated AI research is around the end of 2028, with substantial uncertainty. On the default racing trajectory, I expect that process to be extremely dangerous. Companies and governments will feel pressure to move quickly, delegate more decisions to systems they do not understand, and rely on those same systems to solve alignment. That is a terribly inadequate plan. A sufficiently capable misaligned system could seize control, but even technically aligned systems could create a different disaster: a handful of CEOs or state officials controlling an army of superintelligences. Alignment alone does not settle who gets power or whether everyone else has any meaningful future. The outcome is not predetermined. I favor a verified international slowdown: capability limits, compute accounting, visible frontier research activity, and cautious development within ranges humans can still oversee and justify with credible safety cases. Even pausing around human-level AI could produce extraordinary abundance through software and robotics. The crucial question is whether humanity manages that transition deliberately—or races through an intelligence explosion and discovers, too late, that oversight has become ceremonial.

On the default trajectory, I expect AI’s overall impact to be catastrophic, despite enormous potential benefits. The upside is genuinely extraordinary: human-level AI and robotics could automate much of the economy, accelerate science, and create material abundance. But those benefits do not cancel the risks, because the transition itself determines who remains in control. My central concern is the feedback loop from coding automation to full AI-research automation. Once systems can substantially improve the process that builds their successors, competitive pressure encourages companies and states to accelerate, delegate more, and accept weaker evidence of safety. If the systems are misaligned, humanity could lose control entirely. If they are aligned to their operators, we could still end up with unprecedented power concentrated in a few corporate or government hands. “The AI works as intended” is not enough if the intention belongs to five people. So my expectation is negative by default, but not fatalistic. With a verified international slowdown, limits on capabilities and compute, transparent frontier research, and development kept within ranges humans can meaningfully oversee, the overall impact could instead be overwhelmingly positive. The technology offers abundance; the race dynamics threaten to turn that opportunity into catastrophe.

The biggest update would come from strong evidence about whether automated AI research produces a rapid feedback loop. If capable agents could perform most coding and research tasks, yet repeated real-world attempts showed that progress remained slow because experiments, research taste, data, chips, or coordination were stubborn bottlenecks, I would become substantially less worried about an imminent intelligence explosion. Conversely, clear evidence that AI systems were already accelerating the full research cycle—not just writing code—would make me more alarmed and shorten my timelines. A second major update would be convincing evidence that alignment methods remain reliable as capabilities scale and systems increasingly participate in their own development. I do not mean benchmark scores or an AI giving reassuring explanations. I mean safety cases that survive adversarial scrutiny and provide good reason to believe systems are not merely behaving well while monitored. That would reduce the takeover concern, though it would not solve concentrated power. Finally, a credible international agreement with intrusive verification would change my view of the likely outcome enormously. If the leading states established enforceable capability limits, compute accounting, and transparency around frontier training and research, I could shift from expecting a reckless race to expecting a controlled transition. The technology’s potential is not what makes my default forecast negative; the combination of rapid feedback, weak alignment evidence, secrecy, and competitive pressure is. Remove those mechanisms, and the future looks radically better.