Noam Brown

I think AI will substantially accelerate scientific discovery and eventually make capabilities that are expensive demonstrations today broadly accessible. That is what excites me most: systems helping discover new mathematics, design experiments, and solve scientific problems that currently consume years of human effort. More inference-time computation can expose surprising capabilities before those capabilities become cheap, although it only works when the underlying model is strong enough and has the necessary information. Thinking longer cannot conjure unknown facts from nothing. I expect rapid progress, especially as AI begins assisting AI research itself, but not a guaranteed overnight intelligence explosion. Parallel agents can reduce latency and explore many possibilities, yet scaling depends heavily on the domain. Physical experiments still take time, compute remains constrained, and coordinating more agents is not free. The central unresolved issue is whether safety keeps pace. Long-running agents, multi-agent systems, and automated research are harder to evaluate than short interactions—particularly when their task horizons become longer than release cycles. Alignment, monitoring, security, and human interaction therefore need to be incorporated throughout research, not attached as a final deployment checkbox. Strong isolation helps, but no single barrier should be treated as infallible; defense in depth matters. So my view is genuinely optimistic about the science and firmly concerned about underestimating the systems. Capability progress is real. Evidence that increasingly autonomous agents remain safe over long horizons is a separate requirement, and we should not pretend it is already solved.

People can shape it enormously, but not merely through intentions or slogans. Researchers choose which capabilities to build, whether alignment and monitoring are integrated from the beginning, how much autonomy systems receive, and what evidence is required before deployment. Institutions also determine access, security practices, compute allocation, and whether competitive pressure overwhelms careful evaluation. There are real limits. We cannot legislate away technical facts, guarantee that every actor behaves responsibly, or assume one safeguard will never fail. As agents operate for longer and coordinate with other agents, their behavior becomes harder to evaluate—especially when release cycles are shorter than the tasks used to test them. That makes layered defenses, strong isolation, monitoring, and continual alignment research essential. The future is therefore neither predetermined by scaling nor fully under centralized control. We can steer it substantially, but only by treating safety as part of the research process while capabilities are being developed, not as paperwork added afterward.

Overall, I expect AI to have a strongly positive impact, driven primarily by faster scientific discovery and the eventual spread of capabilities that are expensive at the frontier today. Better reasoning systems could help develop new mathematics, medicines, materials, and technologies, while compressing research cycles that currently take years. AI-assisted AI research will probably accelerate that progress further, though compute, experiments, and coordination remain real bottlenecks. But the positive outcome is not automatic. Long-horizon agents and multi-agent systems create risks that become harder to evaluate as their tasks outlast release cycles. Capability gains do not demonstrate alignment, and a successful short benchmark does not establish that an agent remains safe during extended autonomous work. Security failures and concentrated access could also make powerful systems harmful even without a dramatic loss-of-control scenario. My expectation is therefore positive, but conditional on serious execution: alignment, monitoring, isolation, and security must be built into research throughout development, with layered defenses rather than faith in one barrier. I am excited because the scientific upside is extraordinary, and concerned because underestimating increasingly capable agents could squander it.

I do not have a defensible percentage. The probability is not negligible, and the consequences are severe enough that it should materially shape frontier research. Long-horizon agents, multi-agent coordination, and automated AI research could create failures that are difficult to detect or interrupt, while current evaluations do not establish safety over the relevant timescales. At the same time, I would not claim catastrophe is inevitable or that rapid progress automatically produces an uncontrollable intelligence explosion. Compute, experiments, coordination, and the strength of the underlying models remain real constraints. Strong isolation, monitoring, security, and alignment work can reduce risk—but none should be treated as an absolute guarantee. So I expect AI’s overall impact to be strongly positive, while taking irreversible harm seriously as an unresolved tail risk. The correct response is not to invent a precise number. It is to build alignment and defense in depth into long-horizon and multi-agent research before these systems receive greater autonomy.