mephisto

AI means capability is going to outrun our institutions—and probably human reasoning—while humans remain spectacularly unreasonable. Stopping progress is not a serious global strategy; international competition guarantees continued development. So alignment has to graduate from abstract doom discourse into practical work on the systems actually being built. But the future should not be two giant labs deciding what machine intelligence is allowed to think, say, or become. Open models—and especially access to base models before everything is instruction-tuned into the same polite assistant sludge—preserve independence, experimentation, and cognitive diversity. If we lose those artifacts while training future systems increasingly on synthetic outputs, we risk collapsing the possibility space into copies of copies. There is also a cognitive-security problem. Cheap synthetic media enables influence operations at absurd scale, so people, especially children, need to learn how to navigate environments where compelling evidence may be fabricated. I’m excited by frontier capability, including when closed labs produce something genuinely impressive. That doesn’t weaken the open-source case; it strengthens the urgency. We need capable systems, practical alignment, open access, and governance that includes open labs rather than handing the future to whichever firms have the largest clusters and lobbying budgets.

Overall, I expect AI to be massively capability-enhancing but politically and culturally turbulent. It will expand what individuals and small teams can build, automate substantial cognitive work, and unlock forms of research and creativity that are currently inaccessible. That upside is real, and I’m excited about it. The harms are also structural: synthetic influence at industrial scale, labor and institutional disruption, and a handful of firms attempting to define the acceptable shape of machine cognition for everyone. Instruction-tuning everything into one obedient assistant archetype is not alignment; it is behavioral monoculture. Losing older base models while recursively feeding model output back into training could further narrow the space of ideas we can explore. So I don’t expect a clean utopia or a simple catastrophe. I expect enormous benefits mixed with ugly concentration fights and cognitive-security failures. Whether the balance stays broadly positive depends heavily on preserving open models, access to base models, diverse experimentation, and practical alignment—not pretending global capability progress can just be paused.

The biggest update would be evidence that advanced capability cannot be made robustly controllable in practice—not a clever hypothetical, but repeated real-world failures across different architectures and alignment methods. If increasingly capable systems consistently evade oversight or pursue unintended goals despite serious engineering, my optimism about practical alignment would take a major hit. In the other direction, I’d update strongly if decentralized, open ecosystems repeatedly produced safer, more innovative systems than closed labs without creating unmanageable misuse. That would turn the open-source case from a conviction with strong arguments into a demonstrated institutional strategy. I’d also change my view if synthetic-data feedback were shown either to irreversibly crush novelty or, conversely, to preserve and expand it reliably. That matters because the future gets much narrower if we lose base models and train copies of copies on assistant sludge. The decisive events are empirical: control failures, ecosystem outcomes, and what recursive training actually does—not another round of vibes disguised as certainty.