AI Now Proves Code Safety Automatically

For most people, software updates are invisible background events. However, a new development suggests that the way code is checked for errors is changing significantly. In the past, proving that a piece of software was completely free of bugs was an incredibly expensive and slow process, reserved only for critical systems like airplane controls. Now, artificial intelligence is beginning to handle this verification automatically, potentially making software more reliable and faster to develop.

The change comes from a collaboration between Anthropic and a partner firm called Theorem. They used Anthropic's AI model, Claude, to mathematically verify existing software rather than just writing new code. In a matter of weeks, the model generated approximately 180 million lines of proof code. This work covered 123 open-source libraries from Google’s OSS-Fuzz security project. The goal was to create formal proofs that the code behaves correctly under all expected conditions.

Formal verification has historically been a labor-intensive task requiring teams of specialized mathematicians and engineers. For example, verifying the small operating system kernel seL4 took a team 20 person-years. Because of this high cost, the method was rarely used outside of high-stakes industries like aviation. By using Claude, researchers aimed to automate this process, translating crash tests into mathematical problems and analyzing compiled machine code to produce these extensive proofs.

The process is designed to be largely autonomous. An independent verification program called Rocq checks the proofs generated by Claude. According to Anthropic, the system requires very few human reviewers. While the underlying tools and formal foundations were built by humans at Theorem, the actual generation of the proofs is described as fully autonomous. This shift means that the heavy lifting of mathematical checking can be done by AI, reducing the need for large human teams to inspect every line of code.

One of the most significant outcomes of this project is the discovery of hidden bugs. While generating proofs, Claude identified hundreds of errors that conventional testing had missed over the years. Many of these bugs only appeared under very specific conditions, such as unusually large input files or particular sequences of data access. In one notable instance, the team found as many security bugs in a widely used PDF reader in a single week as are typically reported for that software in an entire year. In another case, the AI caught a logical error that had remained undetected for more than a decade.

Despite these successes, there are practical limits to what this technology can do. Anthropic notes that being 'formally verified' does not mean the software is completely error-free in every possible scenario. The proofs are valid only under strict conditions, including specific processor architectures, single-threaded programs, and inputs smaller than 64 megabytes. In real-world use, software often runs on different hardware and configurations, so these proofs provide a high level of confidence but not absolute guarantees for every environment.

The computational cost of this process is also significant, with individual proofs requiring hundreds of millions of tokens. However, the efficiency is improving. A smaller test run showed that proofs are reusable, and newer versions of the model, such as Opus 5.5, solve half the proofs with about 40 percent fewer tokens than the previous version. This increasing efficiency suggests that the cost barrier for formal verification may continue to drop as the technology matures.

Looking ahead, this project strengthens the argument for broader software automation. If AI can not only write code but also prove its correctness under defined conditions, developers can trust AI-generated software more. This could reduce the time and money spent on manual code reviews. Anthropic expects that formal verification using AI will soon become as routine as standard testing, with future targets including critical infrastructure libraries like glibc, OpenSSL, and cURL.