Your eval harness is shelfware until six shell commands say it is not.

An AI agent eval harness is a thin set of files in your repo that defines what correct means for your workload (rubric.yaml), the case set that grades against (eval/cases.yaml plus eval/ragas.jsonl), the CI job that runs the grade on every relevant PR (.github/workflows/eval.yml), and the on-call link that tells you what to do when the grade slips (ops/failure_playbook.md). It is not a SaaS product, not a six-layer architecture diagram, and not a four-month infrastructure project. It is the smallest thing that lets you swap model_primary on Friday without paging anyone on Saturday.

Because each of the six maps to a distinct shelfware failure mode we have seen in real repos. Q1 catches the harness that lives in Notion. Q2 catches the one where model swaps are four-file changes nobody runs. Q3 catches the case set that froze at week 2. Q4 catches the CI job whose output nobody reads. Q5 catches the rubric that has no owner. Q6 catches the harness that exists but has never blocked a single PR. Adding a seventh question we have not seen failures of would dilute the audit; cutting any of the six would let a known failure mode slip past.

Yes, and that is the more common entry point. We run the six-command audit on day zero of every engagement, including the ones where the agent already serves traffic. A representative pre-engagement repo passes Q1 and Q5 and fails the rest: rubric.yaml exists, ownership block is present, but cases.yaml is 87 days stale, model_primary is hard-coded in two .py files, eval.yml writes to a build artifact instead of the PR, and the harness has never surfaced a regression in git log. The first PR after the audit is usually the one that adds the gh pr comment line to eval.yml. The lowest-numbered failure is always the right next PR.

80 is the smallest case-set we have seen reliably catch a per-prompt regression on a real workload across five named production agents (Monetizy.ai, Upstate Remedial, OpenLaw, OpenArt, PriceFox). Below 60 the variance dominates; the same model output can flip the score by 4 points run-to-run, which is louder than the regression you are trying to catch. The 12 adversarial rows force the engineer to sit with the product lead for a 90-minute session and write the inputs that would make the agent dangerous. That session is half the value; the other half is that adversarial rows turn vague safety claims into a row that either passes or does not.

All five. The harness lives outside the agent code by design. eval/run.py imports your agent's entry point, runs it against the cases, and scores against the rubric. We have shipped the same harness shape on Pydantic AI (Monetizy), LangGraph + Bedrock (Upstate Remedial), Anthropic with custom orchestration (OpenLaw), an automated nightly DAG (PriceFox), and a multi-model pipeline DAG (OpenArt). The framework choice is downstream of the harness. Treating the harness as a property of the framework is the inversion that produces the shelfware in the first place.

The numbers are defaults from agents we shipped, and you should overwrite them at week 0. They come from the scoping call with your product lead, where you write down what fraction of cases must score above what bar before the agent is allowed to ship. rubric_min_score 0.82 means at least 82 percent of cases must score 1.0 against the rubric. ragas_faithfulness 0.78 is a citation-grounding floor that has held up across legal-tech (OpenLaw) and email personalization (Monetizy). You picking your own numbers is the discipline; inheriting ours and never revisiting them is how a rubric becomes evidence in a postmortem instead of a control.

It means the PR cannot merge until either the regression is fixed or the threshold is explicitly raised in a separate PR with a written rationale. Engineers complain for the first week. They stop complaining around week 3, when the harness catches a quiet regression in a model swap nobody else would have noticed and the agent does not page anyone the following Saturday. The annoying-vs-load-bearing tradeoff is the entire point of Q6 in the audit. A harness that has never blocked a PR is not friendly. It is invisible.

Those are tools the eval harness uses; they are not the harness. ragas computes faithfulness and answer_relevancy against your case set, and the rubric.yaml threshold for each is what makes the score actionable. promptfoo and DeepEval are runners; the harness is the wiring that decides which runner is invoked, on which cases, on which PR, and what the failure does. LangSmith is a useful trace store and a non-required complement. The shelfware audit asks whether your repo has the wiring, not which runner is in your requirements.txt.

Yes, with two adjustments. The expected_traits in cases.yaml become per-modality (an image must contain a face in the lower-left third, an audio clip must transcribe to a known string, a code output must compile and pass a unit test). The rubric scorer in eval/run.py becomes per-modality (a vision model judges an image, a sandboxed Python runner grades a code output). The five-PR build sequence and the six-question audit do not change. The OpenArt pipeline (multi-scene commercial video) ships with an image-grading rubric and the same shape of cases.yaml, just with different expected_traits keys.

Because the harness is supposed to be in your repo, run by your CI, owned by your engineer. A SaaS dashboard that grades your harness is a vendor-attached runtime that the harness itself is designed to flag. Sixty lines of bash needs no license, runs on the same checkout your CI runs on, can be committed to your repo as scripts/audit-eval-harness.sh, and survives the engineer leaving. The discipline is the audit, not the tool that runs it. Every PIAS engagement leaves behind both the audit script and a baseline run on day-0.