your rag eval number is hiding regressions

after the graph-vs-vector benchmark, i kept running into the same small annoyance: every retrieval eval tool i touched wanted to give me a number. precision, recall, some blended score. but the question i actually had, every single time i changed something, was never “what’s my score.” it was “i just changed my chunking / embedding model / top-k / reranker — what got better, what got worse, and for which queries?”

a number can’t answer that. a diff can. so i built ragdiff, a minimal open-source CLI that treats the comparison as the primary object, not the score.

the design bet: one protocol, no lock-in

every RAG eval framework i tried wants to own your stack — your embedding pipeline, your vector store, your config format. ragdiff owns nothing. the entire integration surface is one structural protocol:

class Retriever(Protocol):
    def retrieve(self, query: str, k: int) -> list[str]: ...

anything with that method — a qdrant wrapper, a hybrid pipeline, a plain function in a class — can be scored or diffed. no base classes, no vendor SDK. you point the CLI at two zero-argument factory callables (mymodule:config_a, mymodule:config_b) and a golden set (JSONL of {"query": ..., "relevant_ids": [...]}), and it does the rest.

the metrics are the boring, standard four — precision@k, recall@k, MRR, hit rate — implemented as pure functions that only ever see one query at a time. aggregation is the runner’s job. that separation is what makes the diff possible: the per-query results exist as first-class data before any averaging happens, so the diff engine compares query-by-query and sorts the movers, instead of subtracting two averages and calling it insight.

the demo diff makes the argument for me

the repo ships with a toy corpus and two example configs — one retrieves over raw text, the other normalizes (lowercase, strip punctuation) before matching. here’s the actual output of diffing them at k=3:

read the aggregate row and normalization looks like a clean win: precision up, recall up, hit rate up. ship it.

except MRR went down. and the per-query movers table says why:

1 improved, 3 regressed, 8 unchanged. the headline gain comes almost entirely from one query that normalization rescued outright. meanwhile three queries got quietly worse — the relevant doc is still found (recall and hit rate can’t see the problem) but it’s ranked lower (MRR can, barely, as a −0.069 smudge). if your users see the top result first, three of twelve queries just got a worse experience, inside a change that every aggregate metric except one calls an improvement.

that’s the entire thesis of the tool, demonstrated by its own toy example: aggregate metrics answer “is it better on average.” per-query diffs answer “who did i just hurt.” the second question is the one that matters before you ship a retrieval change.

small decisions i’d defend

• the diff refuses to run on mismatched golden sets. if report A and report B don’t cover the identical query set, it raises instead of silently diffing the intersection. a diff over different denominators is a lie with a table around it.
• “net score delta” is only a sort key. the combined per-query signal (precision Δ + recall Δ + rr Δ) exists to order the movers table, not to be a metric. the moment a blended number starts looking like a score, someone will optimize it.
• no framework, on purpose. the whole thing is ~1k lines, one runtime dependency (rich for the tables), 16 tests. the eval harness being small enough to read end-to-end is a feature — the kestrel benchmark taught me that the harness is where the wrong numbers come from, and you can’t audit what you can’t read in a sitting.

where it goes next

adapters for real vector stores (qdrant, pgvector) so diffing two live configs is one command, and a --json output mode so the diff can gate CI — fail the build if any query regresses past a threshold, not just if the average moves. the average moving is exactly the signal i no longer trust.

code is on github. MIT, uv pip install -e ., and the demo diff above reproduces with one command from the README.