The leaderboard stopped predicting production at 88 percent. Your rubric is the gate now.

Cheapest is usually a fine starting point, but only after your client-specific rubric has run on at least 100 representative cases of your real workload. The 88 to 93 band is the headline mean across thousands of academic questions. Your traffic is not academic questions. We have seen cheaper models lose by 4 to 6 points on tail intents and high-stakes slices while looking equivalent on the leaderboard, and we have seen them win for the same reason. The aggregate score does not predict it. Run the client rubric on the cheap model, run it on the expensive model, and let the slice deltas decide. If they are within noise, ship the cheap one and record both sets of numbers in rubric.yaml so the next swap has a baseline.

Public benchmarks are useful for shortlisting and for trend lines on raw model capability. They are not useful for deciding whether a specific model produces correct medical, legal, or financial output on your prompts at your latency budget. The honest split is two layers, not one. Keep MMLU-Pro and Arena Elo as the funnel from twenty candidates to four. Then run a client-specific rubric on those four against 100 to 200 cases mined from your production traces. The shipping decision belongs to layer two. We treat layer one as informational in rubric.yaml so the headline is visible in PR review without ever blocking a merge.

Public benchmarks usually report one pass per question. Real agents make eight, ten, or fifty calls before completing a task and the variance compounds. A model that scores 60 percent on a single pass against your eval can drop to 25 percent across eight consecutive runs of the same case, and that is the number that ships. We set runs_per_case to 8 in rubric.yaml and the gate reads worst-of-eight, not the mean. The leaderboards do not measure this. A model that ranks first on a single-pass benchmark and second on a worst-of-eight evaluation is a model that will look great in evaluation and break in production. Worst-of-eight is the metric the on-call engineer cares about.

rubric.yaml lives at the root of an eval directory on the client's main branch (we use eval/rubric.yaml or rubric.yaml at root depending on existing layout). Two GitHub Actions workflows read it. .github/workflows/pilot-gate.yml runs on every PR that touches the agent code or the rubric, and on a Monday 09:00 UTC cron. .github/workflows/production-gate.yml runs only on a promotion PR with the title prefix promote: <agent> pilot -> production and verifies the seven-day trailing nightly history. Both gates read rubric.yaml directly. There is no separate config in a vendor system, no Notion page, no spreadsheet that has to be reconciled.

The Monday 09:00 UTC cron in pilot-gate.yml runs the eval set against the agent in production. If the weighted score on the client_slices block drops below rubric_min_score, a refund-signal job posts the failing snapshot to a billing webhook configured at billing_refund_webhook in rubric.yaml. The webhook pauses the open invoice. The same job opens a GitHub issue against the engagement owner with labels refund-triggered and billing-paused. No human triggers anything; the gate is mechanical. If the next Monday is green, billing resumes with a 7-day prorated credit. The clause is wired to the client slice, not to the aggregate benchmark, because aggregate movement does not map to user-visible regressions.

Yes, but they are narrower than they look. Three real cases. (1) The model you are using has been formally deprecated and the only path forward is the next checkpoint from the same provider; the rubric is then a confirmation, not a decision. (2) A new modality opens up (long-context, vision, audio) and your client rubric has no cases that exercise it; the leaderboard is the only signal until you grow the corpus. (3) Cost or latency moves an order of magnitude and your rubric is comfortably above the threshold; the headline number tells you which candidate to grade first. Outside those, defaulting to the leaderboard is the failure mode this whole rubric is built to prevent.

No, but treat it as informational. Independent audits in 2025 to 2026 confirmed that MMLU, GSM8K, and ARC-Challenge questions appear verbatim or near-verbatim in training corpora for several major model families, and the contamination on MMLU has been estimated to inflate scores by 8 to 15 points for some models. That is exactly why aggregate_informational has weight 0.0 in our rubric.yaml shape. We still record the score so a reviewer can see whether the candidate is in the same ballpark as the incumbent on academic knowledge, but the gate ignores it. MMLU-Pro, GPQA, and the harder successors are slightly more useful for shortlisting because contamination is lower, but the same rule applies: informational only.

100 to 200 graded cases is the working minimum across every engagement we have shipped. Below 80 cases the per-slice signal is too noisy to set a regression threshold tighter than 5 points, which is wider than most real differences between candidates. At 200 you can set per-slice thresholds at 2 to 3 points and trust them. We split the cases roughly: 70 percent mined from production traces, weighted to actual intent distribution; 20 percent from incidents and escalations; 10 percent adversarial cases curated by hand. Cases are versioned in eval/cases.yaml. The corpus grows by 10 to 20 cases per week through a tail-mining workflow and the gate reads the current corpus on every run.

They render at the top of the per-PR report, labelled informational, with a one-line note that they did not factor into the gate decision. We keep them visible for two reasons. First, reviewers expect to see them; hiding them creates more friction than recording them at weight 0.0. Second, a sustained trend across many candidates (for example, every new candidate is ten points worse on HumanEval) is a signal worth investigating outside the gate, even if it does not block any single merge. The slice numbers go below the aggregate block in the report so the eye lands on the gate-relevant data first.

Run the aggregate score for shortlisting from a public source you trust (artificialanalysis.ai, lmsys.org for Arena Elo, MMLU-Pro from huggingface). Pick three or four candidates. For each, run the client rubric in a single CI job that scores all candidates against the same rubric.yaml in parallel. The output is one shipping report with four columns and the per-slice deltas between every pair. Total wall-clock is 30 to 60 minutes for 200 cases and four models on most production setups. The bake-off ends when one candidate clears every threshold and the others miss at least one. We have a separate guide on how the bake-off methodology slices head, tail, and high-stakes cases, linked at the bottom.