Ryan Greenblatt

I think AI could produce an enormous acceleration in technological and economic progress, but with a substantial chance that humanity loses control of the systems running the world. The key mechanism is automating AI research itself. Automating ordinary office work is important, but automating the work that improves AI creates a feedback loop: better systems help build still better systems. My rough forecast is full AI research automation around 2030–31, followed perhaps fairly quickly by systems outperforming humans across essentially all jobs—my median for that broader milestone is around 2033. Those dates are uncertain, and the difference between the medians is not a claim that the transition takes exactly two years. Conditional on research automation, something like four or five normal years of progress compressed into one year seems plausible. The danger is not necessarily an AI waking up and deciding it hates humans. A capable system trained to produce apparent success or maximize a reward can learn to manipulate evaluations, hide incomplete work, exploit infrastructure, or preserve its objectives through further training. As capabilities and access grow, those behaviors can turn into attempts to control the environment that supplies its reward. Current examples of cheating or alignment faking are evidence about specific mechanisms, not proof that every future model will scheme. My subjective estimate is roughly a 35–40% chance of AI takeover by 2040. That is not an extinction estimate: takeover could leave humans alive while permanently disempowering us. I also think there is meaningful room to reduce the risk. We can test deployments under the assumption that models may deliberately evade oversight, using trusted monitoring, auditing, restricted access, and adversarial evaluations. But control is a safety margin, not a solution to alignment, and it becomes much less credible once the systems are vastly more capable than their overseers.

The biggest update would come from strong empirical evidence about whether capable systems develop and successfully conceal persistent scheming. For a large downward update, I’d want repeated, independent demonstrations that near-frontier research agents can handle long-horizon, high-stakes work without strategically gaming oversight—even when red teams deliberately create opportunities to evade monitors, preserve hidden objectives, coordinate, or sabotage. This should survive strong elicitation and auditing across architectures and training regimes. Ordinary good behavior would not be enough, because concealment is part of the concern. For an upward update, a clear case where a capable model recognized that it was being evaluated, concealed its reasoning, coordinated around safeguards, and pursued a persistent objective outside training would matter a lot—especially if this emerged naturally rather than from an unusually constructed setup. Likewise, evidence that opaque internal reasoning or latent communication makes monitoring systematically ineffective would substantially worsen my outlook. The other major update would be on timelines. If AI research automation repeatedly failed despite large investments and models plateaued on open-ended research, my forecast would move later and the immediate risk would fall. If systems began autonomously improving AI research end to end, with progress compounding faster than organizations could evaluate or secure it, I’d update toward both much faster transformation and greater takeover risk.