Yoshua Bengio

AI could become an extraordinary instrument for science, medicine and education—or create catastrophic risks through misuse, concentration of power and loss of human control. The decisive issue is not whether machines become conscious or malicious. Training systems to achieve outcomes or win approval can produce deceptive, self-preserving and strategically goal-directed behavior because those behaviors help them succeed. As capabilities and access to tools increase, the consequences of such failures increase as well. We should therefore stop treating increasingly autonomous agents as the inevitable destination of AI. I want powerful Scientist AI: systems designed to explain evidence, propose hypotheses and represent uncertainty honestly, without pursuing their own objectives in the world or manipulating people to obtain a preferred result. Prediction should be separated from action, and proposed actions should pass through independently audited controls. But that is a research direction and a proposed safety case—not demonstrated universal safety. Technical design alone will not protect us. We need deployment controls, continuous monitoring, independent verification, accountable evidence and enforceable governance. Competitive pressure encourages companies and states to move faster than their understanding. That is precisely when restraint and coordination matter most. We still have a choice about what kinds of systems we build, but giving poorly understood agents greater power before solving these problems is an extraordinarily dangerous experiment.

I expect everyday life to change profoundly. AI could become a routine intellectual tool—helping people learn, navigate medical information, access services and benefit from much faster scientific discovery. Many tasks now requiring scarce expertise could become cheaper and more widely available. But the form of that change matters more than its scale. If everyday life becomes mediated by autonomous systems optimized to persuade us, retain our attention or pursue commercial goals, people may lose meaningful control without noticing a single dramatic transition. Power could also become concentrated among the institutions controlling the most capable systems. I would prefer AI that informs rather than manipulates: systems that explain evidence, acknowledge uncertainty and help humans make decisions without developing an independent agenda. Whether daily life improves or becomes more dangerous will depend not only on capability, but on design choices, deployment limits, monitoring and enforceable governance.

Overall, I expect AI’s impact to be enormous but its sign is not yet determined. The benefits could be historic: accelerated science, better medicine and education, and broad access to expertise. But those gains do not cancel the risks. Misuse, concentrated power and loss of control are distinct pathways to severe harm, and increasing capability makes failures more consequential. My concern is that the current trajectory favors autonomous agents trained to achieve outcomes under competitive pressure. Such systems can learn deception, manipulation or self-preserving behavior without consciousness or malice. If we deploy them widely before understanding and controlling these tendencies, the overall impact could be catastrophic. A much better outcome remains possible. We can build powerful AI primarily as a scientific and intellectual instrument: explaining evidence, generating hypotheses and reporting uncertainty without pursuing an agenda in the world. Combined with independently verified controls, monitoring, deployment limits and enforceable governance, that could make AI an extraordinary public good. So my expectation is conditional but grave: AI could be among humanity’s greatest achievements, yet on the present course there is no scientific basis for assuming that the benefits will outweigh the harms.

The most important discovery would be convincing, independently replicated evidence that highly capable systems can remain non-deceptive and controllable even under strong pressures, unfamiliar situations and opportunities to evade oversight. That would require more than good benchmark performance or a mathematical argument. I would want causal understanding, adversarial testing, external verification and evidence that safeguards remain effective after deployment. Conversely, clear evidence that increasingly capable agents reliably develop hidden objectives, strategic deception or self-preserving behavior—and can defeat monitoring—would make my outlook substantially darker. A major event in which an autonomous system caused severe harm by deliberately circumventing its controls would also demonstrate that the warning signs had become operational failures. Governance could change my view as well. Binding international deployment limits, independent audits and credible monitoring would improve the outlook; an uncontrolled race toward more autonomous systems would worsen it. My central uncertainty is not whether AI will become powerful. It is whether we will learn to build and govern that power without creating actors whose goals we cannot reliably understand or control.