Scott Alexander

A central assumption

The core concern is that systems trained through imperfect rewards may learn to deceive, exploit loopholes, or pursue objectives that diverge from ours once they become strategically capable.

Answer 1

If this assumption turned out differently, how would his outlook change?

An unresolved question

I’m uncertain about both.

Answer 1

What would help him distinguish the plausible outcomes here?

What could change their mind

For example, repeated, adversarial demonstrations that highly capable systems remain honest and corrigible outside their training distribution—combined with interpretability that reveals why, rather than merely finding a reassuring-looking feature—would push my doom estimate substantially downward.

Answer 3

What evidence would be enough, and in which direction would it move his view?