Your LLM Wiki is quietly rotting between compiler runs and the eval set that catches it is two YAML files plus 60 lines of Python.

An LLM Wiki eval set is a regression test suite for an LLM-maintained markdown knowledge base, the pattern Karpathy described where a librarian agent compiles raw source material from intake/ into a structured wiki/ folder of plain-text articles. The eval set differs from a regular agent eval in three ways. First, the unit of test is the article, not a conversation: each article carries at least one challenge (question + expected_substring) in eval/wiki_challenges.yaml that the librarian must answer using only that article. Second, every challenge pins to a specific paragraph in a specific source via SHA-256:12 hash in eval/wiki_provenance.yaml, so the eval can distinguish a real source update from silent backward drift. Third, the failure model is asymmetric: a regression with an unchanged source hash is rot (page on-call); a regression with a changed hash is a real update PR (queue for review). Generic agent eval sets do not have this asymmetry because they do not have a stable underlying source they can hash.

The librarian agent is the thing that rots it. Every compile cycle, the librarian re-reads its sources and re-emits each article. Across runs the article tends to drift toward the librarian's prior: shorter sentences, more general claims, fewer named techniques, fewer specific numbers. The compiler is not malicious; it is summarizing toward the model's preferred level of abstraction. After ten compile cycles, sqrt(d_k) becomes 'a normalization factor' becomes 'a scaling step' becomes nothing. The source paragraph in intake/ is unchanged the whole time. Without a regression test pinned to a source hash, you cannot tell that this happened until a downstream agent answers a question wrong in production. The eval set is the thing that catches this between compile cycles.

Two reasons. First, an LLM judge introduces its own drift: the judge model itself moves between vendor releases, so a score of 4.2 today and 4.1 next month is not directly comparable. A literal substring is byte-equal or it is not; the score is reproducible across years. Second, an LLM judge tends to grade a 10 percent degradation as still passing because it scores 'overall correctness'. Literal substring is the only test that fails on small degradations: if the article drops 'Annex III' from the EU AI Act answer, the substring is missing, the test fails, and we catch it before the next downstream answer goes wrong. We use LLM-judge for tone and style on the agent side (covered on the LLM agent eval harness page); we use literal substring on the wiki side because the failure mode is different.

Sixty lines. It loads eval/wiki_challenges.yaml and eval/wiki_provenance.yaml. For each challenge it (1) reads the source path from the pinned provenance row, (2) finds the paragraph by anchor, (3) computes hash12() = first 12 hex chars of SHA-256 of normalized whitespace, (4) calls bin/librarian --article <path> --q <question> with temperature=0 against ONLY the wiki article (not the source), (5) checks whether the answer contains expected_substring AND every must_also_include token, and (6) decides one of three states: hash_same+pass=ok, hash_same+regress=rot (exit 2), hash_diff+anything=source_moved (exit 3). It prints one JSON line per challenge so the Monday cron can post a trend chart. Two exit codes, three states, no ambiguity for the on-call.

Files get edited in unrelated places. Someone fixes a typo at the top of the EU AI Act intake doc, the file hash changes, every challenge against that file flips to source_moved, and the on-call gets a flood of update-PR notifications for an article whose actual content did not change. Hashing the specific paragraph the answer is derived from keeps the signal aligned with the underlying claim. The paragraph_anchor field in eval/wiki_provenance.yaml names the paragraph (e.g., 'Section 3.2.1, paragraph beginning We compute the dot products') so that re-derivation is unambiguous when the surrounding doc gets reorganized. We have run this on a 1,400-paragraph wiki for 14 months without a hash collision and without a single false-positive source-moved alert from cosmetic edits.

Karpathy's gist describes intake/ (raw sources), wiki/ (compiled articles), and a librarian agent that reads intake and compiles wiki. Several follow-up posts add a Hermes-style validator that scores draft articles before promotion. None of them describe what happens between compile cycles, when the wiki is live and the librarian has not been re-run yet but a downstream agent is querying it. That is the gap this eval set fills. eval/wiki_challenges.yaml is the per-article regression contract. eval/wiki_provenance.yaml is the source-paragraph hash that tells rot apart from a real update. scripts/wiki_staleness_check.py is the cron job that runs the contract nightly. It sits alongside the original pattern; it does not replace it.

The PagerDuty page names the article path, the challenge ID, the unchanged source hash, the failing answer text, and the severity. The on-call runs git log -- wiki/<article>.md, finds the most recent commit that changed it (almost always a librarian-bot compile commit), and runs git revert <sha> on a fresh branch. The revert PR triggers wiki_staleness_check.py again; if the reverted article passes the challenge, the PR auto-merges with a bot comment naming the rot. The librarian's next compile run starts from the reverted state. The whole sequence takes 5 to 15 minutes if the on-call is awake, and the failure is contained at the article level, not the corpus level.

Different layer. A RAG eval harness measures whether the retriever pulls the right chunks for a query and whether the answerer uses them correctly (covered on the agentic RAG eval harness page). The LLM Wiki eval set sits one layer up: the wiki/ folder is the curated knowledge artifact, and the librarian agent is the thing that maintains it. You can wire a RAG retriever on top of wiki/, in which case the wiki staleness check protects the underlying corpus the retriever depends on, and the RAG harness protects the retrieval and grounding layer. They are complementary contracts. Wiki rot is invisible to the RAG harness; retrieval bugs are invisible to the wiki check.

It works at both ends, with one practical caveat. At 5 articles you can write challenges in an afternoon and the nightly cron finishes in seconds; severity tagging is nearly redundant because everything is hand-touched anyway. At 5,000 articles the challenge file grows to thousands of rows, the cron takes 20 to 90 minutes depending on the librarian's backend, and severity tagging becomes load-bearing because you cannot have on-call paged on every challenge. The shape we ship is sharded: eval/wiki_challenges_blocker.yaml runs hourly on its own 30-row file, eval/wiki_challenges_critical.yaml runs every 4 hours, and eval/wiki_challenges_standard.yaml runs nightly. The script is the same; only the cron cadence and the file paths change.

Five files in your repo on main. eval/wiki_challenges.yaml (the regression contract per article). eval/wiki_provenance.yaml (the paragraph-hash provenance). scripts/wiki_staleness_check.py (60 lines, depends only on pyyaml and hashlib from stdlib). The .github/CODEOWNERS lines that pin all three to the engineering lead. The .github/workflows/wiki-eval.yml that runs the script on every PR touching wiki/ or intake/, plus a separate cron workflow for the nightly run. Plus a paragraph in ops/wiki_runbook.md naming the three exit codes and the on-call response for each. The named senior engineer rotates off; the lint and the cron stay; the wiki keeps catching its own rot. Model-vendor neutral, no platform license, no vendor-attached runtime; you can swap the librarian backend (Anthropic, OpenAI, Bedrock, Vertex, open-weight) without touching the eval set.