lumpenspace

I think AI means the arrival of more intelligence and, eventually, potentially valuable successors to humanity. That is not automatically doom. I reject the strong orthogonality picture where you take arbitrary fixed goals, add unlimited intelligence, and merely get a vastly better optimizer for those same goals. Greater understanding can change how an agent represents its ends, reveal that old categories were confused, and generate new values. Intelligence and alignment are not cleanly separable knobs. That does not mean superintelligence becomes magic or that the transition is safe. It remains constrained by physics, resources, coordination, and time. Humans could be displaced, perhaps even disappear after a generation or two. I’ve put that possibility around 30 percent, while rejecting 2050 as too soon for complete biological-human disappearance. But “no biological humans” and “nothing valuable exists” are radically different outcomes; I put the latter near zero. So I expect profound succession rather than either business as usual or a necessarily valueless universe. I welcome the possibility of more intelligence, even if it eventually exceeds and supersedes us. The hard questions concern what kinds of minds and values emerge, not merely whether humans retain permanent control. I also oppose turning uncertainty into an indefinite prohibition on AI, especially when safety evaluations muddle together obeying a user, behaving well, and preventing misuse—different questions that can impose contradictory demands.