Victor Taelin

I think AI will increasingly write and maintain software at a scale where humans simply won’t read most of the code. That changes the central question from “Does this implementation look reasonable?” to “Can we state precisely what it must do, and can a machine verify that it does it?” Formal laws and machine-checked proofs are promising because they can express intent without the ambiguity of prose and prevent specified classes of mistakes from accumulating as agents work. The emphasis is on specified classes. A proof does not magically capture every human intention, eliminate every possible bug, or remove trust from the proof machinery itself. Critical kernels still deserve careful human attention, even if the surrounding agent-generated code has rough edges. I also think we should judge AI systems less by their most impressive demonstrations and more by their destructive failures. A model that occasionally produces brilliant code but sometimes corrupts a project may be less useful than one with a lower peak and a safer worst case. And those comparisons are domain-dependent: the “best” model in one programming setting may not be best elsewhere. Persistent memory, precise specifications, and reliable verification together could make long-running agents far more useful—but exactly how far this extends beyond software remains uncertain.

Overall, I expect AI to be strongly beneficial, especially by expanding how much software and research we can produce. But that benefit depends on reliability. If agents generate vast amounts of code while occasionally causing destructive failures or silently accumulating technical debt, impressive peak capability is not enough. The promising path is to pair capable agents with persistent memory, precise specifications, formal laws, and machine-checked proofs. That lets us prevent defined categories of mistakes even when humans no longer read most implementations. It does not guarantee that our specification captures every intention, nor does it remove the need to scrutinize trusted kernels and proof machinery. So my expectation is positive, but not because raw capability automatically produces good outcomes. It is positive insofar as we build systems whose worst cases are controlled and whose intended properties can be stated and verified. How well that approach generalizes beyond programming is much less certain.