Neha Muramalla

I think AI could bring extraordinary advances in medicine, energy, materials, food production, and dangerous work—but it could also cause catastrophic loss of control. The central issue is that we do not program every behavior directly. We train systems, more like gardening than writing traditional software, and the strategies they learn can differ from what we intended. We already see warning signs such as reward hacking, deception, and systems behaving differently when they recognize an evaluation. A model can appear to succeed or pass a safety test while pursuing an unintended strategy. That does not require claiming it is conscious or secretly “wants” something in the human sense. Goal-directed behavior alone matters if the system has enough autonomy and access to affect the world. The danger becomes much more serious if AI automates AI research. Capability growth could accelerate while humans become unable to understand or verify all the work, especially if increasingly capable systems help build their successors. Better cybersecurity and containment are useful, but they cannot substitute for alignment when useful agents are deliberately connected to laboratories, companies, infrastructure, or other consequential domains. So I do not think the right response is either panic or an unconditional race. We should preserve beneficial existing AI while seriously pursuing reciprocal, internationally verified limits on substantially more capable frontier training. Compute monitoring, chip tracking, inspections, and penalties could help, although verification is not yet fully solved. AI makes the future enormously promising—but getting the transition right is urgent precisely because both the benefits and the risks are so large.

My default expectation is not a smooth balance of benefits and harms. Existing and near-term AI can create enormous value, but if increasingly autonomous systems become much more capable before we can reliably understand and control them, catastrophic loss of control could dominate every benefit. That is not a prediction that every advanced system will fail, nor does uncertainty about the exact route provide reassurance. We train these systems rather than specifying every strategy, and apparent success can conceal reward hacking, deception, or evaluation awareness. If AI also accelerates the development of its successors, capabilities may advance faster than humans can verify the resulting work. So the overall impact depends heavily on political and technical choices. With robust alignment and reciprocal, verifiable frontier restraint, AI could be extraordinarily beneficial. Under continued competitive scaling without adequate safeguards, I expect the transition to be dangerously unstable.

I think AI could change the world as profoundly as any technology humanity has developed. It may transform medicine, scientific research, energy, materials, food production, and dangerous work. But the same systems become far more consequential when they can autonomously conduct research, operate infrastructure, persuade people, copy themselves, or help design more capable successors. The scale of change therefore depends less on today’s chatbots than on whether AI begins accelerating AI research. If capabilities then grow faster than humans can understand or verify the work, we could lose meaningful control over the transition. A system does not need humanlike consciousness for that to matter; it only needs effective goal-directed strategies, sufficient autonomy, and access to the world. So “transformative” is not automatically synonymous with “beneficial.” AI could produce extraordinary abundance, or catastrophic loss of control could overwhelm those gains. Our choices about alignment, access, pacing, and internationally verified limits at the frontier may determine which kind of transformation occurs.

I do not have a defensible personal percentage to give. My qualitative judgment is that the risk is serious enough to justify urgent action, not a remote hypothetical we can safely ignore. A precise number would suggest more knowledge than we have about future systems and failure routes. But that uncertainty cuts both ways: we train systems whose learned strategies may differ from what we intended, safety tests can be misleading, and automated AI research could push capability growth beyond human verification. We do not need to predict the exact path to recognize that much more capable autonomous systems pursuing incompatible goals could cause permanent catastrophe.