Your POC passed at 0.86. The regression tail is a weekly pull request you have not written yet.

It is the set of production inputs that your POC eval did not cover and that your agent now handles worse than its rubric would pass. The POC was evaluated on 80 or so hand-written cases. Production sees thousands of prompts per day with a much wider surface: different phrasing, combined features, adversarial drift, upstream data drift, and rare-but-high-cost regulatory cases. The regression tail is the portion of that traffic where the rubric, if you could run it live, would score below your own threshold. The PIAS leave-behind makes that scoring a weekly PR instead of an annual postmortem.

Two reasons. First, real-time eval on every trace is expensive and noisy; you end up with an alert-fatigued on-call channel and no signal. Second, the unit that matters for regression testing is the cluster, not the single trace. A single low-score span could be a user typo. Nine similar low-score spans clustered by intent are a real rubric gap. The weekly cadence is how long it takes to accumulate enough spans to cluster meaningfully on most mid-scale production agents (tens of thousands of spans per week). On higher-volume agents like PriceFox we run it daily on the last 24 hours; on smaller agents monthly is fine. The PR shape does not change, only the window.

Yes, by design. Weekly mined cases land at eval/tail/<ISO-week>.yaml (for example, eval/tail/2026-W17.yaml). The mining script is agents/tail_miner.py. The cron-triggered workflow is .github/workflows/tail-mine.yml with cron: "0 9 * * 1" (Monday 09:00 UTC). Promoted cases fold into eval/cases.yaml alongside the original POC cases. Rejected cases stay in eval/tail/ with a rejection_reason field. All four artifacts are in the repo the engagement hands over on day 42, not in a dashboard we own.

Three triggers, any of which is sufficient. (1) rubric_score below 0.82 when the trace is scored against the rubric that lives on main. (2) ragas_faithfulness below 0.78 if the agent is retrieval-augmented. (3) explicit negative user feedback (thumbs down, regeneration, user-provided correction) regardless of score. The thresholds match exactly the thresholds that gated the week-2 prototype and that gate every new-model qualification PR. Using different thresholds for the tail and the rubric is the most common way this loop degrades over time; we keep them synchronized.

Three mechanisms. (1) Clustering by intent embedding collapses similar spans into one case; a cluster of 50 paraphrases of the same gap is one PR row, not 50. (2) Cost-weighting: regulatory- and safety-labeled spans are weighted 10x so rare but high-cost clusters outrank high-volume low-cost noise when the miner picks its top N. (3) A per-run cap of 20 clusters, so even a bad week is a reviewable PR. If you hit the cap two weeks in a row, that is itself a signal that the rubric or orchestration needs rework, and an escalation is triggered automatically.

The tail-mining loop and the qualification loop share the same eval/cases.yaml. The week-2 gate establishes the initial case set and the thresholds. The week-6 leave-behind ships both tail_miner.py and the qualification PR shape. Every case the tail miner promotes becomes a permanent row, so the next time model_primary changes in rubric.yaml (new Anthropic, OpenAI, Bedrock, or Vertex release), the new model is evaluated against the tail too, not only the original 80 POC cases. A model that passes the POC but regresses on the accumulated tail cannot merge. This is how the rubric grows into a moat.

Yes. The tail miner reads from an adapter interface, not a fixed vendor. The adapters we ship on the leave-behind cover OTEL collectors, Langsmith, Arize, and a plain Postgres table with input, output, labels, user_feedback columns. The minimum viable trace store is a Postgres table the agent writes to on every request. If you have nothing today, week 1 of the engagement sets up that table and writes the adapter; the tail miner starts being useful from week 3 onward with modest sample sizes, and hits its stride around week 6 when the cron begins firing on production traffic.

Three things. (1) Read the dry-run scorecard in the PR body: how many of the mined cases the current model_primary already handles. That sets expectations; usually 10 to 20 percent pass already, which is a healthy signal that clustering is finding real gaps and not random variance. (2) Open each cluster, read the representative input and the auto-written expected-output, and edit the expected field if the auto-writer missed a nuance. This takes about a minute per cluster on a mature engagement. (3) Mark cases as promoted (fold into eval/cases.yaml) or rejected (stay in eval/tail/ with a rejection_reason). A typical review is 15 to 25 minutes on a typical week.

That is the loudest honest signal an AI agent in production gives you. It means the rubric is missing a whole capability your users are exercising, or the orchestration itself has a systemic gap (wrong retriever, wrong tool, wrong decomposition), or the underlying model is drifting on your workload. The escalation path is automatic: after two consecutive weeks where the cluster count goes up, agents.tail_miner escalate opens a GitHub issue assigned to the engagement owner and triggers a 30-minute scoping call. At that point we are not editing the tail file; we are rewriting part of the rubric or the agent graph, with a named engineer and a new week-0 style memo.

No to all three. The five shipped agents on the leave-behind use different primaries (Anthropic direct, OpenAI direct, Bedrock, Vertex, multi-provider inference), different orchestration (Pydantic AI, LangGraph, custom DAG), and either ragas or a homegrown rubric scorer. The tail-mining loop is the same file shape on every one. model_primary is a string in rubric.yaml, the trace adapter is a class in agents/tail_miner.py, and the workflow file is the same YAML. Model-vendor neutrality is part of the leave-behind contract.