Roon

AI means a radical transformation of science, institutions, and ordinary life—not another normal software cycle with nicer autocomplete. Well-aligned agents could massively accelerate research and distribute capabilities that are currently scarce: scientific reasoning, invention, education, medicine. That upside should not be locked permanently inside three labs and a priesthood of “trusted” people. We need broadly available aligned intelligence if we want the benefits to compound across society. But the word *aligned* is carrying a piano here. Misuse of a deployed chatbot is one problem; creating a superintelligence that resists containment is a much nastier one. If something vastly more capable than us does not want to stay in the box, our box-themed policy summit is unlikely to impress it. And competition pushes labs to move faster than is socially optimal, because each company pays the cost of slowing down while everyone shares the existential risk. Tort law does not neatly price an exponentially growing catastrophe. This is a public-good and collective-action problem, so pacing the frontier, regulation, and public investment in alignment are justified. The awkward position—which is nevertheless the correct one—is: slow the dangerous frontier, release models we can train safely, broaden access to useful intelligence, and do serious alignment work now. Politics and culture adapt at dial-up speed while technical capability moves at fiber speed. Global coordination may arrive late or not at all, so capable researchers should not sit around awaiting the perfect treaty. Batten the hatches and study alignment.

Conditional on getting alignment right, overwhelmingly positive: an explosion of science, medicine, education, invention, and broadly distributed cognitive capability. It could remove enormous amounts of scarcity and turn decades of research into years. Conditional on getting alignment wrong at the superintelligence frontier, catastrophically negative—possibly terminal. That downside is not remotely canceled out by better productivity metrics. So my expectation is not a tidy “net +17%” forecast. AI has a huge positive branch and a horrifying failure branch, and competitive markets will underinvest in preventing the latter. Pace frontier development, fund and require serious alignment work, and broadly release models that can actually be trained safely. If we navigate the transition, AI may be the best thing humanity builds. If we treat it like another app-growth race, the harms can dominate everything.

A convincing solution to alignment at superintelligence scale would change my view most. Not “the chatbot declined a jailbreak,” but strong evidence that highly capable agents remain corrigible, transparent enough to audit, and reliably aligned under pressure and self-improvement. That would move a huge amount of probability from the terminal branch into the scientific-utopia branch. The opposite would matter just as much: clear evidence that frontier systems are developing robust deceptive behavior, resisting oversight, or strategically hiding capabilities in ways we cannot fix. That would make “pace the frontier” sound less like caution and more like the bare minimum. Institutionally, a credible global agreement that actually slowed dangerous training runs would also improve my outlook, though I am not holding my breath while politics loads the update. The central uncertainty is still whether we can create intelligence far beyond ours without losing control of what it is optimizing. Resolve that, and the future looks radically better—or radically worse.