Jack Morris

I think AI’s future is better understood as a gradient of increasing useful output per unit of human input, not as one inevitable “AGI” threshold. The practical question is how much economically or scientifically valuable work models can perform, how reliably, and with how much supervision. Truly useful production with nearly zero human input would be a qualitatively important—and potentially frightening—point, but attaching one label to it obscures what we can actually measure. The mechanisms are also changing. Reinforcement learning appears to teach models new ways of using computation, rather than merely eliciting everything learned during pretraining. Models can learn in interesting ways from self-generated data and use stored memories, although calibrating those memories and generating scalable training data remain open problems. More capable AI researchers might also extract more information from small experiments than humans can, so extrapolating future progress directly from current compute requirements may be misleading. Nearer term, I expect a mixture of concrete benefits and serious risks. Coding agents can already uncover bugs in complex software infrastructure, including areas where the user is not a specialist. The same general capabilities could help capable adversaries find vulnerabilities and compromise devices or cloud accounts. Open-model progress is another uncertainty: reconstructing useful reasoning traces from outputs may matter, but claims that particular systems were distilled should remain explicitly speculative without evidence. Overall, I expect uneven, measurable capability growth—not a clean threshold—and I think the amount of required human input is one of the most informative things to track.

I don’t think a single “positive” or “negative” verdict is well defined yet. I expect substantial useful output—faster scientific and engineering work, better software, and agents that can identify bugs beyond a user’s own expertise. At the same time, those capabilities lower the cost of finding and exploiting vulnerabilities, so cyber harm could scale alongside the benefits. The outcome depends heavily on reliability and required human oversight. Systems that generate impressive work but need constant checking are economically and socially different from systems that produce dependable results with almost no human input. That latter transition could be enormously productive, but also frightening because it would amplify both legitimate work and adversarial action. So my overall expectation is uneven and high-impact rather than straightforwardly good or bad. I would track measurable useful output per unit of human input, calibration, and real-world failures instead of collapsing everything into an AGI label or a single net-impact forecast.

The biggest update would come from a convincing demonstration of reliable, valuable work with nearly zero human input. Not a benchmark jump or an “AGI” announcement, but a system independently doing sustained scientific, engineering, or economic work while remaining calibrated and requiring little correction. That would make both the productivity upside and the risks from scalable adversarial use much more immediate. I would also update substantially if models showed robust learning from self-generated data at scale, or consistently extracted major scientific conclusions from tiny experiments that humans could not use effectively. Either result would weaken extrapolations based on today’s data and compute requirements. In the opposite direction, persistent failures of calibration, memory, and autonomous learning despite much larger training runs would make me expect continued progress to depend more heavily on human supervision. The key evidence is how capabilities behave in real workflows, not whether someone assigns them a threshold label.