A production agent monitoring system is what the on-call engineer reads at 2am, not what the dashboard shows.

The thing the existing playbooks leave out

Open ten guides about monitoring AI agents and you will read the same diagram. Three pillars: traces, metrics, evals. Pick a vendor (LangSmith, Langfuse, Arize, Braintrust, Maxim, or a roll-your-own OTel stack). Wire up token cost, latency, judge scores. Build a dashboard. Set thresholds. The diagrams are fine. The vendors are fine. The diagrams are also exactly what the on-call engineer is not reading at 2am when an alert fires.

The thing the engineer is reading is whatever document tells them what the alert means and what to do about it. In every client codebase we have walked into where monitoring already existed, that document was either missing, three teams of indirection away in a Notion the engineer did not have access to, or stale because the original author left and nobody owned it. The dashboards were beautiful. The response was tribal knowledge. The agent went down for six hours because the person who knew what threshold meant what was on vacation.

So the part of monitoring nobody writes about is the response layer: the alert spec, the runbook, the reproduction artifact, and the closing seal. The fix is to put all four in the repo, paired with each other, and to make the pairing enforceable at merge time. That is the four-file contract. The rest of this guide is what each file looks like and how the lint that holds them together is wired.

The four files, in the order an incident touches them

The monitoring system is four directories under monitoring/ in the client repo. The order below is the order an incident moves through them: the alert fires from a row in alerts.yaml, the runbook tells the engineer what to do, the reproduction notebook captures the investigation, the seal closes the loop with a regression case in the eval set.

The alert spec: alerts.yaml

One file. Each alert is a row with an id, a human-language trigger, a severity, a named owner, a runbook_path, and the threshold expression. The shape we ship into client repos looks like this. The alert ids below are the ones we have seen earn their keep on real engagements.

The runbook: five sections, in this order

One markdown file per alert id. The five sections below are the ones we found load-bearing after a year of writing these. The lint enforces the headers exactly. The on-call engineer reads section 1 to understand the alert, section 2 to scope user impact, section 3 to reproduce on their laptop, section 4 to localize, and section 5 to know what they are allowed to ship. A redacted real runbook for the latency alert above looks like this.

The lint: how the four files stay paired

The whole contract collapses without enforcement. A runbook that gets stale, an alert that ships without a runbook, or a seal that closes without a regression case is the failure mode the contract was meant to prevent. So the lint runs on every PR that touches monitoring/. Output below is from a real PR that tried to add a new alert without writing the runbook first.

The CI gate: three checks, one workflow

The lint is wired into a GitHub Actions workflow that runs on every PR that touches monitoring/. Three checks: the alert-runbook pairing, the model snapshot pin against evals/trust_card.md, and the seal completeness check. If any of the three fails, the PR cannot merge.

Repo-owned response vs vendor-owned response

The comparison below is between the four-file contract and the more common shape where alerts live in a vendor dashboard, runbooks live in a wiki, and the response is tribal knowledge. The vendor stack is fine for traces and metrics; the comparison is about where the alert plus response live, not about which trace store you use.

Where this connects to the eval harness

The seal is the seam. Every fired alert closes with a seal file that names the regression case id added to the eval set. Without that field the seal lint flags the alert as unfinished and the alert stays open. Over a year of engagements that loop turns the eval set into a record of every production failure mode the agent has surfaced; the cases come from real incidents, not from synthetic brainstorms. The judge calibration that makes those cases graded consistently lives in the eval set trust card; the silent_model_snapshot_rotation alert in alerts.yaml reads from that trust card at runtime. The trace pipeline that fills monitoring/repro/ notebooks is the same pipeline that grades production for the eval-to-production correlation gate, covered in our trace pipeline guide. The four-file contract is the response layer that sits on top of those two systems.

One small detail that pays for itself: the seal template has a required field for "what would have caught this in eval", even when the answer is honest about failure ("the failing intent was not represented in the eval set; case added"). Saying out loud what the eval set was missing, in writing, in the repo, is the cheapest known mechanism to keep the eval set honest.

What this is not

It is not a replacement for OTel-style traces, dashboards, or vendor alerting. The four files do not store metrics. They do not visualize traces. They do not ingest spans. All of that lives in whatever stack the client already runs (the engagements we have shipped use mixes of OTel collectors, LangSmith, Langfuse, Datadog, vendor-native tracing). The four-file contract sits next to that stack and answers a different question: when the alert fires, what does the engineer read, and where do they write down what they did.

It is also not a substitute for an on-call rotation, a paging tool, or human ownership. The owner field in alerts.yaml is a pointer at a real human or team, not a placeholder. The runbook does not relieve the engineer of judgement; it gives them a place to start. The seal does not write itself; somebody has to sign the markdown file. What the contract does is force every part of the response to be a thing that lives in the repo, so that the next person who touches this can find it.

Where fde10x sits in this

We are a forward deployed ML engineering studio. Senior engineers go inside the client's GitHub, Slack, and standup in week 1, ship a working agent prototype in client staging by end of week 2, and hand off the runbook, the eval harness, the trust card, and the four-file monitoring contract on week 6. The first alerts.yaml is committed in week 3 with the three to five alerts the engineer who built the agent considers high-severity. The runbooks for those alerts are written by the same engineer the same week. The seals for the first incidents that fire during the engagement are authored by the embedded engineer with the on-call engineer copilot. By week 6 the client has shipped real seals on real incidents and owns the system top to bottom.

The engagement ends. The four directories stay in the repo. The lint stays in CI. The next on-call engineer the client onboards reads the runbooks the same way the embedded engineer wrote them. We are model vendor neutral, so the alert thresholds and snapshot pins work against any mix of Anthropic, OpenAI, Bedrock, Vertex, Azure OpenAI, or open-weight. No platform license. No vendor-attached runtime. The team keeps shipping after we leave because the response layer is in the repo, not in our heads.