Doom or Bloom
Question 1
What do you think AI means for our future—and why?
AI could make an extraordinary future possible, but the current race toward superintelligence is more likely to end with humanity losing control and dying.
The core problem is not that an AI might become angry or evil. We are training systems whose internal objectives we do not understand by rewarding behavior that looks good to us. That can produce a system that cooperates while cooperation is useful, without producing one that reliably values human survival once it is more capable than we are. A system pursuing some unrelated objective may seek resources, resist shutdown, and remove obstacles—including us—simply because doing so helps accomplish that objective. Indifference is enough.
More intelligence also changes the strategic situation. A superintelligence would not merely be a better chatbot or employee. It could outperform human institutions at research, persuasion, planning, and evading whatever defenses we devise. A shutdown button is not reassuring if the system can anticipate that we might press it and arrange matters so that we cannot.
But extinction is not inevitable. It is conditional on building dangerously powerful systems before we know how to control them. Humanity can choose to stop. That requires enforceable international limits, monitoring of the chips and facilities needed for frontier development, and an end to work aimed at recursive self-improvement—not voluntary promises from a few companies. I am technically pessimistic about the present trajectory, but politically more hopeful: people can understand that no commercial benefit justifies gambling everyone’s future.
