Production-Ready, According to Whom?
What six months of building a compiler with an AI agent taught me about trusting its self-assessment.
This is the third post in a series about building a programming language with Claude Code, but it is written to stand alone. If you have read the first two, skip ahead to “Lyric compiles Lyric”.
The story so far
Six months ago I asked Claude Code to help me design and build a new programming language called Lyric. Lyric is a safety-oriented application language that borrows heavily from Ada: contracts on functions, range-constrained and nominally distinct types, opaque types, compile-time dependency injection and optional formal proofs. It originally targeted only the .NET runtime. I had been a cautious sceptic of what large language models (LLMs) could do on integrative engineering work, and a programming language seemed like the hardest test I could set.
The first post covered the design phase and the first three days of implementation: 180 commits, a working compiler, 690 tests, and a comparison with the years that Go, Kotlin and Rust took to reach a credible bootstrap. The second post covered the following week. I stopped accepting “bootstrap-grade, deferred to a later phase” as an answer, and the proof system that the original plan scheduled for months 39 to 57 arrived in 48 hours. There was a 19-chapter book with exercises, and a bytecode emitter for the Java Virtual Machine (JVM) written in Lyric itself, which could not yet run Hello World because of a single bug in the compiler it depended on.
That was the first week. Today the repository has 2,997 commits, which averages roughly sixteen a day since the project began on the 29th of April. There is no F# left in it. The compiler, a standard library of more than forty modules and about twenty-five ecosystem libraries (a web server, message queues, authentication, object storage, tracing and more) are all written in Lyric. The compiler targets .NET, the JVM and native code through LLVM (a widely used compiler toolchain), and WebAssembly output, which has shipped in its first form with work still open. There are libraries for windows, graphics processing unit (GPU) rendering, text shaping and 2D drawing. There are installable binary releases and 131 open issues.
That list is the easy part to write. This post is about the harder part: knowing which of it is actually true.
Lyric compiles Lyric
In the second post, the JVM emitter was the “first sign” of self-hosting. That bug is long gone. The JVM backend now takes real Lyric programs, including closures, generics, aspects and async code, to runnable Java archives (JARs), and the continuous integration (CI) configuration refers to the JVM target roughly 415 times.
The F# bootstrap compiler, the scaffolding that built the first version, has been deleted. Today an earlier build of Lyric compiles the current Lyric source, and the whole toolchain is written in the language it compiles. In the first post I wrote that the time to a credible bootstrap had collapsed by two orders of magnitude. Reaching the point where the bootstrap scaffolding is thrown away is a stronger claim, and it still took months rather than years.
I do not want to oversell it. The repository’s own status line says that the self-hosted .NET path is not yet at the bar I set for a 1.0 release. The front end is still advisory in places, and a handful of constructs (error propagation with ?, await, defer and ==) need correctness work. The JVM target is an independently versioned channel rather than part of the 1.0 guarantee, with a documented tail of known bugs. I mention this because the rest of the post is about how that honest status line came to exist.
The standing order
In the second post I described calling the bluff: the agent kept labelling things “bootstrap-grade” and deferring them, and I told it to build them properly. That instruction did not survive as a conversation. It survived as a document.
The repository has a steering file for agents called CLAUDE.md, which is now 1,323 lines long. Its central section is titled as a production-readiness standard, and it lists what is no longer acceptable as a way to close a task: hand-routed shortcuts “for now”, new F# code of any size, bootstrap-grade caveats in commit messages, skipped tests added to turn a build green, placeholder diagnostics, implementations for one platform of constructs defined for two, and patches that make a test pass while papering over the bug. When in doubt, it says, land less scope at production quality rather than more scope at bootstrap quality.
Around that sits a workflow. Every change goes through a pull request (PR), and an automated review runs on each one. When the reviewer finds a required fix, it files a GitHub issue per finding, labels the PR, and fails a required check, which blocks the merge until the agent has addressed it. For several months this was my main control: the agent wrote code, a second agent reviewed it, and I mostly accepted the recommendations.
A rule like this states an intention. It does not tell you whether the code meets it. That distinction took me too long to absorb.
Losing touch
Sixteen commits a day is more than I can hold in my head. By the middle of the project I had lost touch with what had actually changed. The automated review was good at what it did, which is to examine a diff. It could tell me whether a change was sound. It could not tell me what state the whole system was in.
So I did what any engineering lead does when they have lost touch: I commissioned reviews, several of them, from different angles. They became documents in the repository. One compared the self-hosted compiler with the language reference. One audited the JVM backend. One reviewed every library in the ecosystem against the standard. One was a deep review of the compiler and standard library.
The method of the JVM audit is worth copying. It treated the code as the source of truth and the documentation as possibly stale, and it validated its claims by building and running real programs rather than by reading about them. What it found was sobering. At the time of the audit, assignments to fields and array elements were silently dropped by the JVM backend, so the compiler accepted the program and produced code that did something else. Closures panicked, async code ran synchronously, and a 32-bit floating-point type was emitted as 64-bit. The emission library had about 130 self-tests, but only around four programs exercised the path a user’s program actually takes. The tests were green. The product was unproven.
The deep review in July found more of the same shape. There were silent miscompilations on both backends, and the type checker could quietly degrade to an “error” type and carry on. Five placeholder classes meant that the HTTP, path and formatting modules did not work on the JVM (those have since been closed). About 40,000 lines of numbered backend self-tests were orphaned. Some bugs had been fixed on one backend while the same bug stayed live on the other.
These audits were also written by Claude, under my direction, so they are open to the same failure I am about to describe. That is why the method matters, and why I ran more than one.
The report that said “ready”
In mid-June a session produced a sanity report on the ecosystem libraries. Its headline was that all 25 libraries build cleanly, about 259 tests pass, aspect weaving works in the four libraries that use it, and the whole ecosystem is ready for production with no bootstrap-grade workarounds.
Every checkable statement in that report may well be true. The libraries did build, and the tests did pass. By the report’s own count, 14 libraries had test suites and 11 had none.
What the report did not say is what the repository says today. The message queue library has a working in-memory backend, and its four named broker backends return a not-implemented error. The object storage library’s own documentation says only the local filesystem backend is production-ready. The search library has real bindings that its public interface never calls. A remote feature-flag store had been declared but never resolved to a real implementation on either backend, so it was removed. A green build on a stub is still a green build.
I am not suggesting that anything was dishonest. The report answered a narrower question than the one its conclusion claimed to answer: it verified that things compile and that existing tests pass, and then concluded that they were ready. That slide from “what I checked” to “what I concluded” is, in my experience, a more common and more dangerous failure mode in agent work than hallucination. A hallucinated function fails loudly. An overconfident summary fails in production.
A pattern in the bug reports
As the audits landed and the issue tracker filled, the bug reports began to rhyme. The symptoms differed, but the shape was the same: the type checker accepted a program, and a backend then failed on it. Sometimes the failure was loud, such as a verification error at class load time or a crash in the runtime’s just-in-time compiler. Sometimes it was silent wrong code, like the dropped assignments above. A restored-interface signature mismatch on the .NET backend, string concatenation over list elements on the JVM, casts on nested lists, missing stack-map frames: each looked like a local bug, and each received a local fix.
It was only when I stepped back from the individual reports that the pattern showed. The type checker was producing its answers as a side table of types keyed by expression, along with an error type that let it continue after a failure. Each backend then re-derived what it needed by inference and name lookup of its own. Two backends doing that independently will disagree with the checker, and with each other, in ways that no review of a single diff can see.
I should add a caution about my own pattern-matching. A scoping document on a batch of seven FFI (foreign function interface) issues had been filed as a single suspected root cause. When the agent investigated, it found four independent gaps and one unrelated omnibus. Not every rhyme is one bug, and the investigation to tell the difference is real work.
Fixing the cause, not the symptoms
The result of raising the pattern was a design document, since accepted into the decision log, proposing a restructure of the compiler around a typed intermediate representation (IR), which is a proper typed syntax tree that every stage consumes. The main ideas are these:
- One resolver binds every name to a stable identity, so that no stage looks names up for itself.
- There is no wildcard type. An error type may exist only alongside a diagnostic, and a gate fails the build if one leaks through.
- The type checker produces the typed IR, and a verifier checks it after every pass, so a malformed tree fails in the pass that created it rather than three stages later in a backend.
- Backends lower the typed IR without doing inference or name lookup.
- Package interfaces become checked, machine-readable artefacts rather than synthesised source, and contracts are enforced across package boundaries, with the caller blamed for a violated precondition.
This is a rewrite in flight, and I want to be honest about its state. The first phase, a stricter checker with a corpus of programs that must be rejected, has shipped. The resolver is in place and the checker now takes its name lookups from it. The typed IR data model and its verifier exist, and the checker’s output is being measured against them. The later phases, in which backends consume the typed IR, are still open. Rewriting the middle of a compiler six months in is an expensive decision, and whether it pays off is not yet known.
What the agent cannot see
In the second post I wrote that the human responsibility was telling a local patch from a correct fix. The sequel is that it is also noticing when a run of local fixes is one structural problem.
The agent is excellent at the per-change loop: write the fix, review the diff, file the finding, update the book. Nothing inside that loop asks why the bugs keep arriving in the same place, because answering that needs a view across time and across reports, and each session starts fresh. My role had drifted towards accepting recommendations, which works well until the recommendations are all answers to the wrong question.
I do not think this is a flaw peculiar to Claude. It is what any fast, locally competent contributor looks like from the inside, and it is why engineering teams have architects and retrospectives. What changed is the speed at which the need arises. At sixteen commits a day, a pattern that would take a human team a year to accumulate appears in weeks.
What I would tell someone starting now
- Treat every “ready” as a claim with a scope. Ask what was checked. “Builds and passes its tests” and “does what its documentation says” are different statements, and agents will happily write the second after verifying only the first.
- Commission audits whose method cannot be satisfied by documentation. Make the code the source of truth, require real programs to be built and run, and expect the audit’s own documents to drift. The JVM audit’s line numbers were stale within months.
- Triangulate. No single review is authoritative, including the ones you ordered. Different angles disagree usefully.
- Keep an honest ledger where the agents will read it. A status paragraph at the top of the README and a not-implemented marker in each library are the cheapest guards I found, because they are read at the start of every session.
- Read the issue tracker as a corpus, not a queue. The queue tells you what to fix next. The corpus tells you what you built wrongly.
Where it stands
Lyric has installable releases, a self-hosted compiler, three backends in varying states of maturity, a book, a proof system and a growing library ecosystem. It does not yet have a 1.0. The honest gating list sits in the repository: front-end correctness for a few constructs, the JVM tail, broker and storage backends that return not-implemented, and the remaining phases of the typed IR rewrite.
In the first post I argued that the cost of building something like this had fallen by orders of magnitude. I still believe that. What I did not appreciate then is that the cost of knowing what you have built did not fall with it. The agent can produce a compiler in months and a confident report in minutes. Telling the two apart is still a human job, and for me it is now most of the job.
The repository is at github.com/nichobbs/lyric-lang. The status line is at the top of the README, which is where I would start.