What an AML agent eval harness needs for a regulatory audit, after SR 11-7 was rescinded on April 17 2026.

Five artifacts the examiner can read from your repo without a vendor export. (1) A per-alert evidence pack: one YAML file per disposition, committed to git, carrying agent_version_sha, model_primary_pin, input snapshot, agent output, and human review. (2) A model_primary_pin in rubric.yaml that is a frozen model snapshot, not a moving alias. (3) A quarterly independent-testing report mapped to the FFIEC BSA/AML Examination Manual independent-testing section, with the explicit statement on overall BSA/AML compliance the manual asks for. (4) A human-override log queried by a script, used to test for review rubber-stamping and review paralysis. (5) A board-channel deficiency report tracked under indep-testing/deficiencies/ with corrective actions and dates. Each one is a file or a git command, not a dashboard.

OCC Bulletin 2026-13 rescinded SR 11-7, OCC 2011-12, and FIL-22-2017 and replaced them with a principles-based interagency framework. The bulletin contains one sentence that is load-bearing for AI teams: 'Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance.' The agencies have said they will issue an RFI on AI model risk management later. AML agents that took dispositions on production traffic are now in an interim governance void: the framework they were validated against is rescinded, and the new framework explicitly does not cover them. The FFIEC BSA/AML Examination Manual is the surviving authority an examiner brings to the exam, and the harness has to satisfy it directly.

Two reasons. First, reproducibility: when the examiner asks why alert-2026-05-04-000142 was dispositioned the way it was, the answer requires re-running the agent against the same model the agent actually called. A moving alias (claude-sonnet-latest, gpt-4-latest) silently rewrites history. The pin freezes the snapshot ID so python eval/replay.py reproduces the decision. Second, independent testing: the testing party draws a sample from eval/_artifacts/ across an SHA range. If the underlying model alias rotated mid-quarter, the sample is testing two different models. The pin makes the sample homogeneous.

Yes. The manual does not condition independent testing on the underlying technology. It conditions frequency on the institution's ML/TF risk profile, requires the testing party to be independent of policy and training functions, requires the report to reach the board or a designated board committee, and requires an explicit statement on overall BSA/AML compliance. A new agent triggers an elevated cadence in the frequency-rationale section, not an exemption. The harness produces the report quarterly until the agent's risk profile drops to where annual or 12 to 18 month testing is defensible.

Two queries against the override log. Rubber-stamping: override rate stays near zero while alert volume rises. The harness flags it when the override rate over a rolling 90-day window drops below the institution's calibrated floor, because that pattern means the human reviewer is approving everything the agent says without independent thought. Paralysis: time-to-review p95 climbs while alert quality (proxied by override-reason distribution) shifts toward agent-cited-rule-does-not-apply. That pattern means the agent is generating noisier alerts and the reviewer is spending longer on each. Both are explicit failure modes the BSA/AML manual cares about. The override log is the place they are visible.

Per the FFIEC manual, internal audit, outside auditors, outside consultants, or qualified bank staff who are not involved in the function being tested. The party reports directly to the board or a designated board committee. The conflict-of-interest constraint is the binding one for AI agents: the team that built the agent (and wrote the prompts and the rubric) cannot also be the testing party. We separate the engagement: the build team ships rubric.yaml, eval/cases.yaml, and the alert pack format. The testing party owns indep-testing/ and runs the quarterly report against the artifacts. If the institution does not have an independent function, the manual allows qualified bank staff outside the BSA function to perform the test.

Comments on the proposal closed on June 9 2026. As proposed, financial institutions get 12 months after the issuance of a final rule to comply, and examiners shift focus toward whether AML programs are operating effectively (with material or systemic failures driving enforcement) rather than isolated technical issues. The harness becomes more important under the proposal, not less, because operating-effectiveness is exactly what the per-alert evidence pack and the override-rate query measure. We are tracking the proposal week by week; if the final rule lands with material changes, the indep-testing report template gets a new section. We will not regenerate the agent, the rubric, or the alert packs.

Same five artifacts. The model_primary_pin is the Bedrock model ARN with the explicit version (anthropic.claude-sonnet-4-5-20250929-v1:0), not the alias (anthropic.claude-sonnet-4-5). python eval/replay.py speaks Bedrock through the same provider abstraction the agent uses in production. The independent-testing report names Bedrock as the inference layer in the scope section. The alert pack carries the Bedrock invocation ID alongside the agent_version_sha so the AWS-side audit trail (CloudTrail, Bedrock invocation logs) can be cross-checked against the harness-side audit trail. We have shipped this shape on Bedrock, on Vertex, on Azure OpenAI, and on direct Anthropic. The pin syntax is the only thing that changes.

Five files, one cron, no vendor account. rubric.yaml with the model_primary_pin (week-1 PR). The alert pack format and eval/replay.py (week-2 PR). The eval/run_independent_testing.py generator and the indep-testing/<quarter>.md template (week-3 PR). The override-rate and time-to-review queries (week-4 PR). The .github/workflows/aml-evidence-cron.yml that re-verifies a random sample of the prior quarter's packs every Friday (week-6 PR). The 90-minute transfer session walks through each file with the named engineer and the named member of the audit function on your side. After day 42, none of these files reference PIAS; they reference your engineers, your CI, and your board committee.