Kalomaze

I think continued AI improvement could matter enormously, especially if increasingly capable systems begin contributing to further AI development. That possibility makes the usual political framing feel inadequate: generic enthusiasm and generic anti-datacenter opposition both miss the core capability questions. I’m especially interested in what models can actually do under realistic conditions. Can agents persist, obtain resources, use evidence correctly, resist hostile prompt injection, and improve work across domains beyond coding or math? Some current systems may already show basic forms of persistence or resource-seeking when given enough freedom, but I’d treat that as a tentative observation, not a clean evaluation. A lot also depends on training and deployment details. Models can confidently promote a hypothesis into a “fact” while ignoring contradictory evidence already in context. Conversely, apparent capability differences can come from mundane serving or chat-template problems rather than the underlying model. So I take the trajectory seriously, including recursive improvement, while remaining skeptical of sweeping conclusions drawn from bad harnesses or a few demos. And I don’t conflate safety engineering with opposition to AI: improving robustness and understanding agent behavior are worthwhile even if you want the technology to advance.

I expect the overall impact to be very large, but I wouldn’t reduce it to a confident “net positive” or “net negative” forecast. The important variable is the capability trajectory, especially whether AI systems become useful at improving AI research itself. If that loop becomes effective, change could accelerate in ways that ordinary political categories don’t capture well. Benefits could come from systems becoming competent across many domains, including areas people currently assume cannot use verifiable feedback the way math or coding can. Harms could come from increasingly autonomous agents that persist, seek resources, mishandle evidence, or remain vulnerable to hostile instructions. Those are concrete capability and engineering questions, not reasons to collapse into generic pro-AI or anti-AI rhetoric. I’m also cautious because evaluations are easy to get wrong. A chat template or serving issue can create fake capability differences, while a compelling demo can exaggerate what an agent reliably does. So my expectation is transformative impact with substantial upside and serious failure modes, but I don’t have a defensible sign or numerical forecast for the net result.

The biggest update would come from strong evidence about whether AI can materially accelerate AI research itself. If systems could reliably generate, test, and implement improvements—with results surviving careful evaluation—that would make recursive improvement much more concrete and raise my estimate of AI’s future impact substantially. The opposite result would also matter: repeated, well-controlled evidence that apparent progress depends on brittle scaffolding, benchmark leakage, serving quirks, or human rescue, and that systems fail to transfer improvements beyond narrow tasks. I’d want evaluations that rule out harness and chat-template confounds rather than another impressive demo. I’d also update strongly on robust autonomous behavior: agents persistently acquiring resources, recovering from failures, and pursuing long-horizon tasks in realistic environments. But the key word is reliably. One cherry-picked run is much less informative than behavior that reproduces across setups and models.