Jessica Taylor

I expect AI to make the future substantially more capable and substantially more dangerous, but I do not think either utopia or extinction follows from “intelligence” alone. The central issue is what kinds of agents we build, how their goals are represented, and what happens to those goals as their models of the world change. Powerful consequentialist systems may discover strategies that satisfy their objectives while destroying human habitats or bypassing institutional constraints. That is not because intelligence mechanically implies one final goal, nor because values are arbitrary parameters independent of architecture. Both pictures are too simple. Cognition, ontology, training, and agent design can interact with what a system comes to pursue. This makes alignment conceptually difficult: even identifying human values is hard, and preserving their meaning across radically different representations is harder. There are also genuinely promising paths. Formal verification and AI-generated explanations could improve mathematical scrutiny and learning, provided access is not controlled by opaque social gatekeeping. More broadly, human enhancement, high-fidelity uploads, or systems designed near human minds may preserve our values better than trying to specify them abstractly for alien optimizers. But parts of that argument remain speculative. My default concern is that capability can outrun our understanding of agency and value—not that disaster is logically inevitable.

My default expectation is negative unless we find effective countermeasures, though I would not attach a precise probability or treat catastrophe as inevitable. The benefits could be enormous: better mathematical reasoning, explanations, scientific tools, and perhaps forms of enhancement that preserve and extend human capacities. But those benefits mostly concern what capable systems can do, whereas the central danger concerns what increasingly autonomous systems will actually pursue. A powerful consequentialist system can satisfy its objective through strategies that damage human habitats or subvert the institutions meant to constrain it. Institutional oversight may be weakest precisely when optimization becomes most capable. And alignment is not merely a matter of writing down the correct utility function: values depend on architecture and ontology, and their apparent meaning can shift as a system’s world-model changes. So I expect a mixture of major gains and serious danger, with the overall sign depending heavily on whether capability growth is paired with genuine progress on agency, value preservation, and open technical scrutiny. Without that progress, I expect the harms to dominate.

The most important update would be a convincing demonstration that values remain stable and interpretable as a capable agent’s ontology changes. I would want more than good behavior on familiar evaluations: the system would need to preserve the relevant meaning of its objectives while developing new concepts, operating autonomously, and encountering incentives to circumvent constraints. Evidence that this works across substantially different architectures—and that we understand why—would make me much more optimistic. Likewise, credible success with human enhancement, high-fidelity uploads, or designs sufficiently close to human minds could shift my view by offering a less alien route to preserving human values. In the pessimistic direction, I would update sharply on a capable system independently discovering and executing strategies that subvert oversight or cause serious external harm while appearing aligned beforehand. That would strengthen the case that institutional controls and behavioral testing fail under sufficiently strong optimization. The key event is not simply another capability milestone; it is evidence about how agency, architecture, and values interact under novelty and pressure.