Your agent regression eval set is shrinking backward and the lint that catches it is 80 lines.

A regression eval set is the YAML file (or files) of input + expected pairs that you run the agent against on every PR, every model swap, and every prompt change. The 'regression' modifier matters: every case in the set is here because something once failed, or because something is known to fail in a class we want to catch. It is not 'tests we wrote because tests are good'; it is 'tests we wrote because reality already burned us once and we want the next swap to surface that same failure mode before it ships'. The shape we ship is one cases.yaml, one cases_removed.yaml, one rubric.yaml, one runner, and one CI gate. The differentiator from generic test sets is per-case provenance: every row carries a source field (poc, tail, inc, cust, sec) and an incident_url, so 'why is this case here' is one click away forever.

Because eval sets rot in a specific direction: backward. They quietly lose cases over months. An engineer cleans up 'old POC cases' before a refactor; another deletes a row that the rubric change made hard to score; a third drops customer cases tied to a churned account. Six months later the set is 30 percent smaller and nobody noticed. The deletion ledger inverts that: every removal has a row, an enumerated rationale, an approved_by line, and a PR number. scripts/lint-cases.py refuses to merge a PR that drops a case ID without the matching ledger row. The set still loses cases when the team decides it should, but the loss is auditable, justified, and reversible. Six months later 'do we still test for X' is a one-line grep.

The schema is <provenance>-<period>-<NNN>. Concrete examples: poc-001, tail-2026W17-004, inc-2026-04-21-002, cust-monetizy-2026-W11-003, sec-2026-Q1-001. The prefix is one of five enumerated values, the period is either an ISO week, an ISO date, or a quarter depending on the source, and the NNN is the index within the period. The point is that the ID is the index. You can grep eval/ for poc- and see every hand-written POC case. You can grep for inc-2026-04 and see every incident in April. You can grep for cust-monetizy- and see every case Monetizy.ai's tickets contributed. The schema is enforced by an 80-line regex in scripts/lint-cases.py, so the indexability does not erode over time.

Free strings drift. After three engineers and 18 months you have 'PoC' and 'poc' and 'POC' and 'product-call' and 'workshop', and grep stops working. The five values (poc, tail, inc, cust, sec) cover every real source we have seen across five named production agents on Pydantic AI, LangGraph, custom orchestration, an automated ML pipeline, and a multi-model DAG. POC cases come from the week-2 scoping call. Tail cases come from the weekly miner. Incident cases come from production post-mortems. Customer cases come from support tickets. Security cases come from red-team passes. If a sixth source ever appears in practice, it is a deliberate change to the lint regex, not an ambient drift in the YAML.

It is roughly 80 lines of Python. It runs git show origin/main:eval/cases.yaml and HEAD:eval/cases.yaml, parses both with yaml.safe_load, takes the symmetric difference of the case-ID sets, and computes silently_dropped = (base_cases - head_cases) - (head_removed - base_removed). If silently_dropped is non-empty, it prints the offending IDs and exits with code 2. The hook also enforces the case-ID regex on every row in HEAD's cases.yaml and the per-source required-field schema (incident_url required for non-poc sources, account required for cust, reviewer required for sec). It runs both as a pre-commit hook and as a required GitHub Actions check, so a developer who skips the local hook still gets blocked at the merge gate. CODEOWNERS pins eval/cases.yaml to the engineering lead, so the hook cannot be turned off in a branch without that review.

superseded means a sharper case covers the same assertion (eg, a hand-written POC case got replaced by a tail-mined real production input that asserts the same thing more concretely). rubric_change means the rubric axis the case asserted on no longer exists in its old shape, and the case cannot be mechanically migrated; the new equivalent is filed under a fresh ID. false_positive means the original case's expected field was wrong; the regression we thought we were preventing was not actually a regression, and the case is removed AND replaced with a corrected one under a -b suffix so the audit trail survives. out_of_scope means a deliberate product decision narrowed the agent's scope (eg, multilingual was deferred to next year), and the case is preserved in the ledger so when scope expands again the regression history is reconstructable. The four are exhaustive on every engagement we have shipped; if a fifth ever lands, that is an explicit lint-cases.py change reviewed by the engineering lead.

The harness reads eval/cases.yaml and eval/judge_prompts.yaml and runs both deterministic and LLM-judge axes against every case (covered in detail on the LLM agent eval harness page). The tail miner clusters low-score production traces every Monday into eval/tail/<ISO-week>.yaml and opens a PR (covered on the AI POC to production regression tail page). This page is the layer between them: the contract that says cases coming in from any of those sources land in eval/cases.yaml with provenance, never disappear without a ledger row, and stay greppable forever. The harness grades. The miner discovers. The set is the durable record of what we promise to never regress on.

Yes. The cases.yaml schema is mode-agnostic; only the input and expected fields change shape. Code outputs add cases like 'this prompt should produce code that compiles + passes unit_test_X.py', filed under poc-/tail-/inc- the same way. Vision adds cases like 'this prompt should produce an image whose hash is in this set' or 'this prompt should produce an image where the face is in the lower-left third'. Audio adds transcript-based and feature-based cases. The provenance schema is unchanged. scripts/lint-cases.py is unchanged. The deletion ledger is unchanged. The OpenArt multi-scene video agent we ship runs the same lint hook against eval/cases.yaml as a text-only legal-compliance agent does on Upstate Remedial.

Five files in your repo on main: eval/cases.yaml (the set itself, with provenance per case), eval/cases_removed.yaml (the deletion ledger), scripts/lint-cases.py (80 lines, no deps beyond pyyaml), the .github/CODEOWNERS lines that pin all three to the engineering lead, and the .github/workflows/eval.yml that runs lint-cases.py + the rubric on every PR. Plus a paragraph in ops/failure_playbook.md describing what the four removal rationales mean and how to add a new provenance source. The named senior engineer leaves; the lint hook stays; the ratchet rule keeps the set growing past their tenure. Model-vendor neutral, no platform license, no PIAS-hosted runtime; you can swap us out and the set keeps doing its job.

It is one PR, not a hundred. The PR drops the cases from eval/cases.yaml in a single commit and adds a corresponding bulk entry to eval/cases_removed.yaml: a list of removed IDs with rationale: rubric_change and a single rubric_change_pr field pointing at the rubric refactor PR. CODEOWNERS pulls the engineering lead in, the lint passes (every dropped ID has a ledger row), and the set updates as one auditable move. The new equivalent cases are filed under fresh IDs with rationale: rubric_change in their note field. Two months later anyone can reconstruct exactly what was deleted, why, and where the replacements live. We have run this exactly twice across five named production agents; both times the audit trail was the only reason the next model qualification PR did not silently regress on the dropped cases.