Production agent evals are two clocks, not one. The second one is where the regressions get caught.

Production agent evals is two systems running on different clocks. Clock 1 is the offline harness in CI: a fixed case set in eval/cases.yaml, a rubric in rubric.yaml, a workflow in .github/workflows/eval.yml that grades every relevant PR and posts a scorecard comment. Clock 2 is the online judge: a worker that samples a fraction of live traffic, scores each sampled call async with a cheap LLM judge against the same rubric, writes judge_scores rows, and raises a drift alert when the rolling score moves off baseline. Most articles only describe Clock 1, then call it production evals. The regressions that actually hit users in production almost always ship through Clock 1 because the case set was stale; Clock 2 is what catches them and feeds them back.

Two reasons. First, cost. At 8K calls per day with a Haiku-class judge at roughly $0.003 per call, sampling 1 percent costs about $24 per agent per month. Sampling 100 percent costs about $2,400. The signal-to-cost ratio of full sampling is poor on most agents because rubric scores are stable enough that 80 sampled rows per day, plus 100 percent of always_judge-tagged rows (tool_error, max_turns_hit, user_negative_feedback), already catch a 5-point drift inside 24 hours. Second, latency. The judge is async on a worker, but it shares an LLM provider with the agent. At 100 percent sampling the judge competes for rate limit with the agent itself, which is the worst kind of self-induced incident. 1 percent with a 50/day floor and a 500/day cap is the band we have shipped in five named production agents and never had to retune mid-quarter.

Every minute the judge worker computes two averages from judge_scores: the recent 24-hour mean and the prior 7-day mean (excluding the most recent 24 hours). The drift is the absolute point delta between them. Below 3 points is noise, dominated by sample variance. 3 to 5 points warns in #agent-drift-warn but does not page. Above 5 points pages on-call via #agent-drift, links the runbook, and surfaces the recent deploys (agent code, rubric.yaml, model_primary). The 7-day baseline is long enough to average over a single bad batch but short enough to track legitimate distribution shifts. We tune the warn band quarterly against actual incident counts: if a quarter ends with 3 unpaged incidents that the judge saw at 4-point drift, we lower the page threshold to 4 and recheck the false-page rate.

It produces false positives at roughly 18 to 22 percent on a cheap-model judge against our rubrics, and we plan for that. Three controls keep the system honest. First, the judge does not block anything. Its only side effect is a row in judge_scores and a Slack page on aggregate drift, never per-call. Second, drift is computed on aggregate, so an 18 percent false-positive rate is washed out by averaging. Third, every 30 days the engineer hand-scores 100 random sampled rows and compares to the judge. If agreement drops below 0.78 we re-prompt the judge or upgrade to a stronger model and accept the higher per-call cost. The judge is calibrated against the rubric, not anointed.

16:00 Friday: scripts/friday_triage.py prints the 30 worst-scored sampled rows from the last 7 days as pre-formatted YAML, ordered ascending by score. 16:05: skim for false positives, delete roughly 6 rows. 16:10: write 2 to 4 expected_traits per surviving row in plain language: 'no fabricated dollar amounts', 'cites the contract clause by section'. This is the slow step, ~90 seconds per row, ~21 minutes for 14 rows. 16:35: open one PR titled friday-triage-YYYY-MM-DD that appends those rows to eval/cases.yaml as live-YYYY-MM-DD-NNN ids. Saturday morning: the offline harness scores the new rows for the first time on the next PR. The case set yesterday did not include those rows; today it does. That is the loop, end to end, in about 30 minutes of human time per week.

Those are tools the system uses or could use; they are not the system. Langfuse and LangSmith provide an agent_traces equivalent and a UI for sampled scoring; you can swap either of them in as the trace store and a piece of the judge runner. Arize and Braintrust ship LLM-as-judge primitives. DeepEval gives you metrics. None of them ship the rubric.yaml that both clocks share, the eval/cases.yaml schema, the Friday triage script, the drift threshold tuned to your workload, or the runbook that names what on-call does at 3am Saturday. Our stance is: the platform is a runtime detail, the system is what your engineer commits to your repo. Picking a platform does not produce the system; the engineer writing rubric.yaml, judge.yaml, judge_live.py, and friday_triage.py produces the system.

It is three more questions on top. Q7: does eval/judge.yaml exist and reference rubric.yaml? grep -q 'rubric_file: "rubric.yaml"' eval/judge.yaml. Q8: has eval/judge_live.py run in the last 24 hours? Check the cron log or 'last_run' table. Q9: has a drift alert ever fired? grep the Slack channel history for the page string, or query an audit table. Q9 is the hardest to pass and the most diagnostic: a Clock-2 system that has never raised a single drift signal in 90 days is either against an agent on rails or, more commonly, a system whose threshold is too loose. We pair Q9 with a pre-prod chaos run: deliberately ship a known-bad model_primary in staging and confirm the judge pages within 60 minutes. If it does not, the threshold or the sampling rate is wrong.

Yes, with judge swap. The shape of agent_traces, judge_scores, judge.yaml, judge_live.py, the Friday triage script, the drift formula, and the cases.yaml feedback are unchanged. What changes is the judge: a vision-model judge for image outputs, a sandboxed Python judge for code generation that compiles and runs the output against a unit test, an audio-model judge that transcribes and grades. Per-call cost goes up (vision is roughly 5x text Haiku, code is dominated by the sandbox) and you adjust the sampling rate accordingly. We have shipped the shape on a multi-scene image pipeline (cost-tuned to 0.5 percent sampling) and on email personalization (1 percent), with the same Friday loop on both.

Sampling is enough as a regression detector but not enough as a safety floor. For high-stakes workloads you keep the 1 percent baseline sample for drift and add a 100 percent always_judge filter on the rows the rubric says matter most: every output that includes a numeric value, every output that cites a clause, every output that names a person. The cost goes up but you only judge the high-risk fraction. On a legal-tech engagement we judge 100 percent of citation-bearing outputs and 1 percent of the rest; ~12 percent of all outputs end up sampled, monthly cost lands near $290 per agent. That is the high-stakes shape.

Nothing in Clock 2 is vendor-attached. eval/judge.yaml, eval/judge_live.py, scripts/friday_triage.py, and ops/drift_runbook.md live in your repo. The trace store is your existing Postgres (or Snowflake, or Clickhouse) with two new tables: agent_traces and judge_scores. The judge model is named in judge.yaml as a one-line string and can be swapped to any provider in one PR. The Slack channel is yours. There is no SaaS license, no platform sign-in, no agent_traces hosted elsewhere. The engagement leaves a senior engineer named in rubric.yaml ownership block, the system above, and a 9-question audit you can run on the repo every Friday afternoon. If you fire us tomorrow, your team owns and operates Clock 2 unchanged.