Code wrong, Corpus wrong or both – Corpus 2/3
AI
When we automatically compared the business description and the code against each other, we found issues that no test would have caught.
Text: Arttu Ylärakkola
Corpus is a directory of Markdown files in a project's version control. They describe the system at a business level: vocabulary, roles, processes, and decisions. In my previous post, I explained why fast code generates cognitive debt.
When the same repo contains both the code and a plain-language description of what the code is supposed to do, AI can compare them to each other. And when it compares them, it finds contradictions.
Three types of contradictions
The code and the Corpus should tell the same story about the system. If the AI detects a difference between them, one of them is wrong. Or both. Every discrepancy we found fell into one of three types.
The first type is routine maintenance. Someone changed the behaviour, but the Corpus was left unupdated. The fix is quick, but important: if the Corpus remains incorrect, the AI will perform the next task based on a false premise.
In the second type, the rule is documented and the business relies on it, but during some refactoring, the code stopped following the rule. The tests pass because no one wrote a test for that exact thing. Reading the code won't reveal this bug, because the code looks fine. The fault only becomes apparent when you know what the code is supposed to do.
In the third type, both the code and the Corpus are wrong, but it is rarely a matter of carelessness. Typically, the change was made in a situation where no one had agreed on what the change was meant to entail. The developer interpreted the ticket one way, a different interpretation was documented in the Corpus, and the business had a third one in mind. The developer interpreted the ticket one way, a different interpretation was documented in the Corpus, and the business had a third one in mind.
This cannot be resolved by coding, because it concerns a business decision that has not been made. The contradiction is the very first moment this gap becomes visible. Without the comparison, it would only have surfaced once a customer called to complain.
What about when the AI is wrong?
Not all automatically discovered discrepancies hit the mark. The AI sometimes misinterprets things, and some of the alerts are false alarms.
Even so, alerts very frequently uncover something that genuinely needs fixing in the code. An alert directs you to look at a spot where, as it turns out, something else is amiss. That's why the Corpus isn't maintained for its own sake. The goal isn't polished documentation, but rather what the contradictions reveal about the system.
Fixing the Corpus takes very little time, but every fix eliminates a false assumption that the AI had been operating on. You can fix it once or pay for it individually with every task.
Contradictions can also be prevented in advance. When a pull request that alters behaviour includes a Corpus update, the review contains two diffs: the code and the description. In the diff of the description, a change that is technically correct but wrong in business terms is easy to spot, and it can be read by anyone familiar with the business.
Try it yourself!
We created an example repo at github.com/rakettitiede/corpus. Discrepancies between the code and the Corpus have been intentionally left in its codebase. It includes an example of an AI skill, /corpus-check, which reports what is wrong: the code, the Corpus, or both.
The repo also includes a script that creates a copy of it without the Corpus and other things the AI could use as a cheat sheet. So you can ask the same questions twice and compare how the AI responds with and without the Corpus.
The example is, of course, just an example. Real systems are far more complex, and that's precisely where the Corpus provides the greatest benefit.
Where is this heading?
Currently, the checks are run on the developer's machine during a commit or push. A natural next step would be a bot similar to Dependabot, which monitors the repo and opens a pull request whenever the code and Corpus drift apart.
One thing is clear in advance: if the bot becomes too noisy, no one will read its pull requests. More important than complete coverage is ensuring that the few hits are genuinely significant.
In the final part of the series, I'll explain what is worth including in the Corpus and what is not.
Read all parts of the Corpus series:
1/3: Technically flawless, practically wrong: why fast code generates cognitive debt
2/3: Code wrong, Corpus wrong, or both? Three contradictions that reveal more than tests (oh wait, you just read that one!)
3/3: Corpus in your own repo: what to include in the directory and what not to
What is Corpus?
Corpus is a directory of Markdown files in a project's version control. They describe the system at a business level: vocabulary, user roles and their motivations, process flows, and decisions made. The files can be read by a developer, a business stakeholder, or an AI, and they are always updated whenever the system's behaviour changes.
The code tells the AI how the system works. Corpus tells it why.
Corpus is neither a product nor is it licensed. It is simply a version-controlled directory of text files, meaning it works with any language, tool, or methodology. We don't claim to have invented it. We naturally drifted into this way of working in our own projects and gave it the name Corpus.
Example repo: github.com/rakettitiede/corpus