A new LLM release is a three-file pull request, not a six-week review.

It means one and only one thing on a PIAS leave-behind: can the PR that flips model_primary in rubric.yaml clear the same thresholds the agent was accepted under in week 2, on the case set that was captured during the week-0 scoping call. The case set is eval/cases.yaml on main, the thresholds are rubric_min_score 0.82 and ragas 0.78, and the workflow is .github/workflows/eval.yml. If the PR clears it, the release is production-ready for your workload. If it does not, it is not, regardless of what the provider changelog says.

Because the case set that matters is yours, not theirs. A vendor platform benchmark can only score against cases the vendor curated. The week-0 rubric was written by the engineer with your product team, capturing the failure modes that matter for your workload: a wrong outbound is a regulatory incident on Upstate Remedial, a broken scene is a retried render on OpenArt, a miscited case is a red-team failure on OpenLaw. A platform that reports MMLU going up 2 points cannot tell you any of that. A CI job running eval/cases.yaml on your keys can.

One line in rubric.yaml (model_primary), one short diff in ops/failure_playbook.md to promote the old primary to fallback, and the scorecard artifact that GitHub Actions attaches to the PR. Nothing in agents/*.py changes on a pure model swap. If the orchestration code changes, it is not a model qualification, it is a feature PR with a model change bolted on, and we split it. A disciplined three-file boundary is what makes the audit trail readable three releases later.

The per-case regression check in eval.yml fails the build if any case drops more than 3 points versus the previous model_primary. On a smaller regression, the scorecard surfaces it and the PR requires a second reviewer plus a note in the allowlist file explaining why this case is acceptable (for example, the regressed case is a known adversarial prompt the new model is less brittle on in a different way). A tolerated regression is a documented one, with an author and a date. Untracked regressions are how small drift turns into the next incident.

A gateway routes traffic at runtime. It does not score your model against your rubric. You still need an eval harness to decide whether to send traffic through the gateway to a new model at all, and most teams do not have one, which is why they default to "wait and see". The PIAS leave-behind inverts it: the eval harness is the gate, the gateway (if you run one) is downstream of the merged PR. We are neutral on whether you run LiteLLM, route directly to provider SDKs, or use Bedrock / Vertex routing. The qualification job is the same.

Yes, and a good chunk of our engagements start there. Week 0 is a read of your existing agent: named failure modes with no fallback, nodes with no eval signal, production traces with no scoring. Week 1 is the first PR that writes eval/cases.yaml from the last 30 days of real traffic plus the adversarial rows the rubric needs. Week 2 is the gate against that case set. From week 6 forward, every new LLM release is a three-file PR against the same file set. We do not require you to replace your orchestration. We require a case set we can score against.

We have shipped with Anthropic, OpenAI, Bedrock (Anthropic + Mistral), Vertex (Gemini), Azure OpenAI, and on OpenArt we run a multi-provider inference fleet the client operates. The five production agents use different primaries and different fallbacks. The rubric file makes model_primary a string, not an opinion, and the failure playbook names the fallback explicitly. Because the qualification job does not care which SDK the provider ships, a new release from any provider uses the same three-file PR.

On a typical case set (80 cases plus ragas dataset), the CI job runs between 6 and 14 minutes on a standard GitHub Actions runner. The human side — branch, edit, push, read the scorecard — is a few minutes. From provider release announcement to a reviewed merge is usually the same afternoon, sometimes the same hour if the reviewer is online. On larger case sets (PriceFox runs on the last 24 hours of real traffic) the job takes longer and runs on a self-hosted runner with matrix parallelism, but the PR shape is identical.

The scoping call produces three artifacts: the week-0 rubric memo (a one-page document naming the problem, the failure modes, the rubric thresholds, and the case set shape), a first cut of eval/cases.yaml (the 10 to 20 cases that define "correct" for your workload, written by the engineer with your product lead), and a named fallback policy (what happens when the primary is unavailable or regressed). Those three artifacts become the spine of every future LLM qualification. Every new release is scored against them. That is why the engagement is six weeks, not a quarter: the point of the six weeks is to earn the right to evaluate every future release in an afternoon.