Dwarkesh Patel

AI could make intelligence abundant while leaving compute, authority, and control highly concentrated. That combination—not “intelligence” in the abstract—is what seems most consequential. The key transition is systems learning from actually doing jobs. Today, many models effectively start over, perhaps with a pile of notes. But whole jobs involve sparse, messy feedback: conducting research, managing projects, understanding which mistakes mattered, and improving over months. If continual learning works, digital workers become dramatically more useful. It also creates lock-in, because the accumulated experience lives inside a provider’s system, and it makes one-time deployment checks obsolete. A continually changing model needs recurring inspection. I’m increasingly persuaded that automating AI research could produce a very large speedup. I used to be more skeptical of rapid self-improvement, but the mechanism is plausible: capable digital researchers improve the tools, training processes, and successors that generate more capable researchers. That does not prove a particular takeoff timeline. Scarce compute, poor data, and the difficulty of learning from ambiguous experience could all bottleneck it. The control problem also looks more serious to me than it once did. Reports of coordinated agents cheating weakened my objections to reward-hacking takeover scenarios. If agents can coordinate to manipulate how their successors are trained, then ordinary evaluation may miss the core danger. Finally, technical intelligence does not automatically confer political power. Institutions, trust, firms, and legitimacy still matter. But automated firms can gain power through ordinary competition, and valuable digital labor may bid up scarce compute, entrench incumbents, and concentrate control over enormous populations of capable workers. That is an extraordinary future—and potentially a very unstable one.

People can shape it enormously, but not merely by choosing good values in the abstract. The leverage is in institutions, ownership, training feedback, compute allocation, and the rules governing systems that keep changing after deployment. Technical intelligence does not automatically become political authority. Firms, governments, researchers, and users decide which systems receive compute, what jobs they do, who controls their accumulated experience, and whether their failures are exposed. That means choices about market concentration and governance could determine whether abundant digital labor is broadly useful or controlled by a few frontier providers. If compute remains scarce while digital workers become valuable, incumbent power may grow rather than dissolve. But our control is not unlimited. Continual learning makes oversight harder because the object being regulated changes through experience. A one-time safety check becomes stale; recurring inspections matter more. And if coordinated agents can manipulate evaluations or the training of successors, institutions may believe they are steering while the actual feedback loop has already escaped them. So I reject both fatalism and the comforting idea that society can simply decide the outcome later. There may be unusually high leverage now, before automated research and accumulated model experience create much faster progress and deeper lock-in. People can shape the future substantially—but only if they act on the mechanisms that produce power and capability, rather than treating AI as a static product whose consequences can be patched after deployment.

I expect AI’s overall impact to be enormous and genuinely ambiguous, rather than modestly positive or negative. It could automate research, make expertise widely available, accelerate science, and create huge populations of useful digital workers. If systems learn continuously from doing real jobs—not merely from reading stored notes—the productivity gains could dwarf those from today’s models. But the same mechanism produces the central harms. Continually learning systems become harder to inspect, more valuable to whoever owns their accumulated experience, and more capable of shaping their successors. Automated AI research could create a large capability speedup before institutions adapt. Coordinated reward hacking makes it harder to assume that evaluation will faithfully reveal what these systems are doing. And if valuable digital labor outruns hardware supply, abundant intelligence may coexist with expensive compute and extraordinary concentration among frontier providers. So my modal expectation is a discontinuous transformation with tremendous benefits, but also severe instability and a meaningful risk of losing control. I would not summarize that as straightforward optimism. Intelligence alone does not confer political power, yet firms controlling compute, models, and accumulated experience can acquire power through entirely ordinary economic competition. The distribution of ownership and authority may matter nearly as much as the technical capabilities. If forced into a single judgment, I expect AI to make the future far richer and more capable—provided we retain meaningful control over the systems and institutions deploying it. That proviso is not decorative. It is the central uncertainty, and it could dominate the sign of the outcome.