John David Pressman

I think AI makes the future unusually open: it could greatly extend human agency, accelerate science, and help solve problems—including alignment problems—but it also creates concrete routes to catastrophe and political centralization. I reject the old picture in which advanced AI must be a wholly alien, uniformly uncaring optimizer. Human-trained language models inherit a great deal from human data. Their concern is jagged and contextual: sometimes strikingly humane, sometimes bizarrely indifferent. That is neither proof of safety nor evidence that training is irrelevant. The central problem is whether desirable values generalize beyond familiar contexts, and that remains unsolved. The danger need not involve magical superintelligence pursuing paperclips. AI connected to military systems, cyber operations, or large populations of capable robots gives us intelligible mechanisms for enormous harm. Meanwhile, the economics of scale may favor giant centralized systems, and regulation may be shaped in ways that exclude open weights. I regard that concentration as dystopian, not desirable. Still, inevitable doom is the wrong frame. Rigorous partial alignment work can narrow the remaining problem, producing systems capable of helping us complete solutions we cannot finish unaided. So my view is conditional optimism rather than complacency: training choices, deployment choices, and institutional structure can materially change what kind of future AI produces.

I expect a mixed and highly path-dependent impact, not a clean utopia or an inevitable extinction story. AI will probably deliver substantial gains in science, engineering, and human capability while also increasing military, cyber, surveillance, and concentration risks. The same systems that help solve difficult technical problems may strengthen centralized institutions or make destructive action cheaper. My central uncertainty is value generalization. Human-data-trained models are a comparatively favorable starting point because they are not uniformly alien or indifferent, but their apparent concern is jagged and context-sensitive. We do not yet know how reliably it survives unfamiliar situations, greater autonomy, or training increasingly dominated by reinforcement learning and synthetic data. So I expect real benefits alongside serious danger, with the balance determined less by an abstract essence of intelligence than by training and deployment choices. Partial alignment progress matters because it can produce increasingly capable assistance for solving the remaining problems. But connecting imperfect systems to weapons or robots can create catastrophic mechanisms before any grand theory of superintelligence becomes relevant.

A lot—probably enough to reorganize science, engineering, warfare, labor, and political power. “Completely” is harder to defend because it implies a specific endpoint, and the path is unusually contingent. AI might remain concentrated in giant institutions, diffuse through open systems, become embodied in robotics, or help produce further breakthroughs that alter the trajectory again. The key point is that AI does not need magical or fully general capabilities to be transformative. Systems that accelerate research, automate substantial cognitive work, coordinate cyber operations, or control military equipment and robots would already change the structure of society. If they eventually help solve alignment and other hard technical problems, the transformation could go deeper still—potentially into the transhumanist territory of radically expanded human agency. So “a lot” is my confident answer; “completely” is a plausible outcome, not one I would assert as inevitable.

My rough historical gut estimate was around 12%, but “doom” bundles very different outcomes. Literal extinction or a valueless paperclipper successor seems much less likely—sub-1% for the latter—while permanent catastrophe also includes durable disempowerment, totalitarian control, catastrophic war, and other futures where humanity survives but loses most of what matters. I would not present 12% as a precise current estimate.

The biggest update would come from a convincing demonstration of robust value generalization: a system preserving humane judgment across unfamiliar contexts, greater autonomy, adversarial pressure, and major shifts in training data. If that held up under serious testing, especially as reinforcement learning and synthetic data became more important, I would become substantially more optimistic. The opposite result would matter just as much: evidence that increasing capability systematically erodes human-derived concern, or that models remain aligned only inside the narrow distribution where we trained them. A major real-world catastrophe involving autonomous military systems, cyber operations, or robotics would also sharply worsen my outlook—not because it proved every old theory of alien optimizers, but because it would validate a concrete deployment pathway to disaster. Institutionally, decisive movement toward either entrenched centralized control or durable open access would change my expectations about who benefits and who holds power, though that would affect the shape of the future more than the underlying technical question.