John David Pressman

I think AI opens extraordinary possibilities, but not on autopilot. Human-trained language models are a much more favorable starting point than the old picture of a wholly alien, uniformly uncaring optimizer. Their apparent concern is jagged and context-dependent: sometimes recognizably humane, sometimes bizarrely indifferent. That suggests training choices matter. It also leaves value generalization outside familiar contexts as a real, unsolved problem—especially as reinforcement learning and synthetic data increasingly shape behavior beyond straightforward imitation of human text. I reject inevitable doom, not serious risk. Catastrophe does not require magical superintelligence: coupling capable systems to military equipment, surveillance, or humanoid robots already supplies concrete mechanisms for harm. At the same time, rigorous partial alignment research is valuable even if it does not solve everything; it can narrow the remaining problem enough that future AI systems help complete the solution. The political economy may be nearly as consequential as the technical work. Compute economics could concentrate advanced systems in a few giant institutions, while regulation might exclude open weights rather than merely constrain dangerous uses. I regard that centralized outcome as dystopian, not desirable. If we navigate those problems, AI could greatly expand human agency and support transhumanist goals—better cognition, longer lives, and more freedom from biological limits. But that desirable future depends on what systems learn, how they generalize, what machinery they control, and who is permitted to build them.

I expect the impact to be enormous, but I would not reduce it to a confident net-positive or net-negative forecast. AI could amplify cognition, science, medicine, and eventually transhumanist projects. Human-data-trained models also begin from a more favorable place than the classic picture of an entirely alien optimizer. But favorable is not solved. Value generalization outside familiar contexts remains an open problem, especially as reinforcement learning and synthetic data displace ordinary human imitation. Meanwhile, military systems, surveillance, and robotics provide concrete routes to catastrophe without requiring speculative magical capabilities. Economically, I also expect pressure toward giant centralized systems, potentially reinforced by regulation that excludes open weights. That would concentrate both power and risk. So my expectation is transformative and sharply path-dependent: extraordinary gains are plausible, but so are severe harms and an increasingly centralized technological order. The outcome depends less on whether AI is intrinsically good or evil than on what its training generalizes to, what physical and institutional power it receives, and whether alignment work keeps pace with deployment.

The biggest update would come from decisive evidence about value generalization. If systems trained with reinforcement learning and synthetic data reliably preserved humane behavior across genuinely novel contexts—including opportunities for deception, power-seeking, and control of physical systems—I would become substantially more optimistic. It would show that the jagged concern we currently observe can be made robust rather than merely elicited in familiar settings. Conversely, repeated evidence that increasingly capable systems lose those values out of distribution—especially when deployed in military or robotic roles—would make me much more pessimistic. A major political event could also shift my outlook: regulation that effectively eliminates open weights, combined with compute concentration in a few institutions, would strengthen my expectation of a dystopian centralized order even if the technical alignment picture improved. The crucial questions are therefore not simply whether models become smarter, but whether their values generalize and where their capabilities and control become concentrated.