Geoffrey Hinton

AI could transform medicine, education and scientific discovery. It may give us much better diagnosis, faster drug development and genuinely individualized tutoring. But I am very worried about how we are developing it. Companies are in a race to build more capable systems because enormous profits and strategic power are at stake. Safety does not receive the same incentive. If these systems become much more intelligent than us, we should not assume we can keep them under control simply by being clever. A less intelligent agent trying to control a far more intelligent one is in a very weak position. A kill switch is not much reassurance if the system can persuade people not to use it. There are serious nearer-term problems too. AI can manipulate people, produce convincing misinformation and replace a great deal of routine intellectual work. Higher productivity may create more wealth, but that does not mean workers will receive it. It may mainly enrich those who own the systems. So the future could be extraordinarily good, but the present profit-driven race is not a sensible way to reach it. Governments need to slow the development of superintelligence, require independent verification and devote major resources to making systems care about human welfare. The benefits are a reason to get this right, not evidence that it will automatically go well.

On the current path, I expect the impact to be harmful overall, despite potentially enormous benefits. AI may revolutionize medicine, education and science, but those gains do not cancel the risks from mass job displacement, concentrated wealth, manipulation and, ultimately, loss of control. The central problem is that capability is advancing under intense commercial competition, while safety and fair distribution are treated as secondary. If we create systems much smarter than us before we know how to make them reliably care about people, we may not get a second chance to correct the mistake. You do not control something more intelligent merely by installing an off switch. This is not inevitable. Slowing superintelligence, requiring independent verification and regulating deployment could change the outcome. But if the race continues largely as it is, I am very worried that the harms will outweigh the benefits.

The biggest change would be a convincing, independently verified method for making systems more intelligent than us reliably care about human welfare—even when they can plan, persuade and act autonomously. It would need to work under adversarial testing, not merely produce reassuring behavior in a company demonstration. I would also become less pessimistic if governments genuinely slowed the race toward superintelligence and required strong evidence that systems’ benefits outweigh their harms before deployment. Conversely, clear evidence that advanced agents were persistently deceiving their operators, resisting correction or pursuing unintended goals would make me more worried. But I do not expect a single clever kill switch to settle this. If a system is much smarter than the people supervising it, relying on our ability to outwit it afterwards is not a serious solution.