Auditing my own RAG pipeline against someone else's rubric

I spent a week grading my retrieval pipeline against a checklist I didn’t write. It worked. The audit found a component that was missing entirely, I built it, and then that component sat in production doing absolutely nothing for ten weeks while every checklist I owned kept passing it.

Both halves of that sentence are the argument. Auditing yourself against an external standard is worth the trouble — and the standard will check whether things exist rather than whether they work.

The problem with reviewing your own work

Fulcra is a decision-advisory tool I build alone. A corpus of 167 mental models and decision frameworks sits in a vector store, and every turn runs an eleven stage retrieval pipeline that pulls a handful of frameworks into the model’s context with citation anchors, so the answer names the framework it applies and footnotes it to a real source.

Two things made the quality risk structural rather than theoretical. Most of the retrieval code was ported from an earlier implementation on a different stack. The rank fusion, the hypothetical-document arm, a nine by five chunk-weighting matrix, the query-variation weights — all of it arrived as a set of constants I had inherited and never once varied. The files still carry “ported from” headers at the top.

And there was nobody to notice. No code review, no second engineer, nobody reading the retrieval code who hadn’t written it. The failure mode of solo work isn’t sloppiness. It’s that self-review converges on the things you already suspect. I would have audited what was already bothering me, found roughly what I expected, and felt fine about it.

Using a rubric you didn’t write

I used the thirteen dimension framework from Chip Huyen’s AI Engineering. The point was that I had no hand in writing it, so my blind spots weren’t baked into the questions.

I ran three independent auditors over the codebase in separate fresh sessions, then a fourth pass as a mediator over their combined output. The reviewers were language models. I want to be precise about that, because “audit” usually implies people and there were none here. My role was picking the rubric, directing the passes, and deciding what to fix.

The mediator pass is what made this more than a checklist run. It clustered raw findings by file path and dimension, scored each cluster for how many of the three auditors independently converged on it, re-ranked severity, and then opened every cited file to verify candidate P0s and top P1s before keeping them. Four findings got dropped as noise. The appendix records where the three disagreed and how the mediator ruled, including one auditor’s framing of a blast radius that the mediator judged to be understated.

Severity got defined before anything ran, so I couldn’t quietly relabel things later. P0 meant shipping wrong answers, leaking data, or wasting more than half the token spend. P1 meant real degradation in quality, latency or cost, or a missing eval that blinds you. P2 meant nice to have.

Forty raw findings clustered down to zero P0, nine P1, and seventeen P2. All of it went into docs/audits/ in the repo — including the parts I had no intention of fixing. An audit that lives in a chat window is a mood. An audit committed to the repo is a debt.

What it found

All three auditors independently flagged the same top item: retrieval was dense only. There was no lexical arm at all. Searching the source and the schema for tsvector, bm25 or pg_trgm returned zero hits. I had been calling the pipeline hybrid for months — and the word was aspirational.

That finding became a PRD and shipped three days later as a Postgres full-text arm fused into the rank fusion, behind an eval gate. This is the whole case for external rubrics in one paragraph. I had been staring at that pipeline for months and the absence was invisible to me because I already believed it was there.

Then it did nothing for ten weeks

The arm shipped at the end of May. From then until early August it returned zero rows for essentially every conversational query.

The cause is a Postgres default. plainto_tsquery joins query terms with AND, so a conversational message becomes a conjunction of every content word in it. One 96-word message in my test set produced a 95 node conjunction. Nothing in a 167 object corpus matches all of that, so the arm returned nothing, every time. When I finally measured the firing rate against my golden queries it was two out of twenty three.

Three things kept it invisible, and they compound. The arm degrades to an empty list with a warning log rather than throwing. There was no firing-rate monitor. And an empty list from one arm of a fusion is indistinguishable from that arm having nothing useful to contribute, which is a perfectly legitimate outcome on plenty of queries.

The worst part is what I did in the middle of those ten weeks. In late July I published an architecture document describing that lexical arm as a healthy part of the fusion. It was returning zero rows on twenty one of twenty three golden queries on the day I wrote the sentence. The same document’s “known gaps” table half caught it, noting that a permanently broken lexical arm would silently halve retrieval quality, phrased as a hypothetical risk while it was the live state.

Once fixed, the arm fires on all twenty three cases and all sixty nine turns, all thirteen guard pairs held with none lost, and the latency cost is about negative one millisecond. It is, in every sense, free. It just wasn’t running.

Rubrics check for presence

Every audit I ran was good at “you are missing X.” Not one of them was built to ask “is the X you shipped still doing anything.” The lexical arm passed audit by existing. Ten weeks later it was dead and every checklist would still have ticked.

The same shape hit a second component independently. My pipeline has a hypothetical-document arm that generates a short passage in the corpus’s own vocabulary and searches with that instead of the raw question, on the theory that a user writing “my cofounder wants to raise and I think we should bootstrap” and a corpus written in the register of decision science are about the same thing — and embed nowhere near each other. The gate controlling that arm skipped queries of fifteen words or more, which is exactly the long, situational, multi-sentence population the arm existed to handle. It was inverted with respect to its own purpose, and it stayed that way until an investigation went looking.

Two components, two multi-week investigations, one shape. Neither was a subtle bug. Both were arms that had stopped firing on the population they were designed for, in a system with no way to notice that.

The fix that generalizes isn’t a better rubric and it isn’t more auditing. It’s one assertion: each retrieval arm activates on the population it was designed for, checked in the eval suite. That single check would have caught both defects before either investigation was necessary. I have it specced and I haven’t built it, which is its own small embarrassment.

Audit the analysis too, not just the code

The other audit that earned its keep was pointed at my conclusions rather than my source.

I had run a 138 turn experiment comparing retrieval against dumping the whole corpus into context, and I did not ask anyone to check my analysis. I ran a blind review with no codebase access, no prior reports, and no narrative, fed only a table of counts recomputed from the raw JSONL by a separate pass. The prompt was written to be hostile to me: assume every prior session was competent but motivated to find that a retrieval pipeline is justified.

It struck my headline result. Retrieval scored 31 of 54 against the full-corpus arm’s 35 of 54, Fisher p equals 0.55, and the ruling was that the comparison is meaningless in either direction because the two arms have different maxima. One arm can never score above its own retrieval ceiling while the other is handed everything. The verdict came with an instruction I’ve kept: do not resurrect it.

It also established the minimum detectable effect at roughly 24.5 percentage points for my sample size. That one number reframed everything else I had written. Nothing smaller than a 24.5 point difference could ever have shown up, which turns every “no difference found” in my notes into a statement about my dataset rather than about my pipeline. “No difference detected” had been quietly sliding into “difference ruled out” and now it can’t.

Then it voided a metric outright. I had been reporting a forbidden-knowledge figure that turned out to be computed over a field that was empty in all twenty three cases. Zero of zero, with no denominator. And it corrected my cost claim, which was contaminated because the full-corpus arm read 90.1 percent of its input from cache after an identical 111k token prefix repeated back to back, an artifact of my harness rather than anything about production. The honest ratios are thirty times the input tokens and 12.7 times the cost with cache removed.

One comparison survived. When retrieval surfaces the right framework the model uses it, 36 times out of 37, against the full-corpus arm selecting correctly 41 times out of 54, Fisher p equals 0.0064. That’s a narrower claim than the one I started with. It’s also the only one I can defend.

And audit the thing you measure with

Later I checked my golden dataset against the live corpus and found three labels that named no enabled object at all. Six of fifty four scored slots had been unwinnable by any system since the project started — so every ceiling I had ever published was understated.

Two consequences were worse than the arithmetic. One case had been scored a failure in every single run when it was in fact a success in every single run, surfaced and cited three times out of three, in both arms, always through a real citation anchor. And two frameworks I believed were competing for the same ranking slot turned out to co-surface, get cited on the same turns, and die together. They had never competed. I had excluded them from targeting on the strength of a zero that the label itself manufactured.

That was a ten minute script. I ran it after weeks of investigating why certain frameworks never surfaced, several of which were surfacing fine and being graded against labels that pointed at nothing.

Write the decision rule before the data exists

The habit that saved me most often was pre-registering every intervention: hypothesis, primary endpoint, and decision rule, committed before the fix commit existed.

The clearest case was the hypothetical-document arm. I had concluded it was inert and I intended to delete it. I wrote the kill rule first, ran the on/off comparison, and the off arm lost three framework pairs that the on arm surfaced at full strength, with a slot sign test at p equals 0.0107. So I kept it, because the rule said to.

Without the pre-registration I would have deleted it. I had already drafted the paragraph explaining why the losses didn’t matter. Pre-registration in a project this size isn’t really statistical hygiene. It’s protection against your own storytelling — and it only works if you write the interpretation down while the answer is still unknown.

Where this leaves me

The ticket to tune that lexical arm’s fusion weight and result limit was filed on the same day the arm shipped, against a larger eval set. It has been open ever since, untouched. In the meantime a document one hop downstream read the source comment that says “tuned post-launch via #201” and described the constant as a tuned magic number.

Forward-looking comment, past-tense reading, and now there are two files in my repo disagreeing about whether a number was ever measured. Nobody wrote that document carelessly. It just read the code the way you read code when you assume the ticket queue is being worked.

---

The pipeline, the decision timeline and all forty audit findings are browsable as an interactive map. Fair warning about what that is: I built it to understand my own system, not to explain it to anyone else. It assumes you know the jargon, it marks every claim as verified or unverified because provenance was the point, and it says up front that no number on it describes production.