Two perception strategies, one numeric clause in your rubric.yaml that decides which one a case runs on.

Accessibility tree first, screenshot only as a fallback. The token math drives the default: a clean accessibility tree snapshot is roughly 4,000 tokens for a typical page, and the equivalent screenshot at a useful detail tier is roughly 50,000. The reliability math reinforces it: public benchmarks put DOM driven stacks (Playwright plus Claude, Stagehand, Browserbase) at 89 to 92 percent task success on common workflows, and vision native stacks (Anthropic Computer Use, OpenAI CUA) at 75 to 78 percent, a 12 to 17 point gap. Use screenshot only for canvas only apps, badly marked up forms, anti bot screens, and image content inside iframes. The right architectural answer is hybrid: ax_tree primary, vision fallback per eval case, with the choice scored as a numeric field in your rubric and enforced by a CI gate.

A page that has 200 interactive nodes serialises into about 4,000 tokens of role, name, value, and hierarchy text. The same page rendered as an image at a detail tier good enough for the model to read body copy is roughly 50,000 input tokens, and that count grows with viewport size and density. On a 20 step workflow the difference is the difference between an 80,000 token run and a 1 million token run. The model also skips image encode, decode, and the vision branch on every step, which is why DOM driven agents are 3 to 5x faster per step in practice.

Four real cases. (1) Canvas only UIs: a charting library, a whiteboard, a 3D viewer. There are no semantic nodes for the model to reason over. (2) Forms that fail a WCAG audit so badly the tree is empty or full of role=presentation elements; common in vendor portals. (3) Anti bot pages that strip or randomise role and id attributes between loads. (4) Image content inside an iframe the page intentionally protects from scraping. Outside those four, screenshot is paying 12x in tokens and 12 to 17 percentage points in reliability for no benefit.

We do not write it in a code comment. We add a perception_mode field to every row in eval/cases.yaml, with allowed values ax_tree_only, hybrid, and screenshot_only. Each screenshot_only case must carry a perception_reason from a closed list (canvas_only_ui, anti_bot_dom_stripped, iframe_image_payload, wcag_unmarked_form). pilot-gate.yml is a required check that refuses to merge a flip from ax_tree_only to screenshot_only without a paired token_budget edit, and refuses to land any screenshot_only case unless the runbook has a vision-fallback section that names the selector and the failure signature. A new engineer in week 7 can git blame the row and read why the case is on vision, without asking the original author.

It is one section in rubric.yaml and one block in pilot-gate.yml, both files we already publish under the standard six file leave-behind (rubric.yaml, eval/cases.yaml, .github/workflows/pilot-gate.yml, .github/workflows/production-gate.yml, runbook/<agent>.md, flags/<agent>.yaml). On a UI driving agent, the perception block is part of week 1 scoping. The first PR usually adds the perception_mode field, the budgets table, and the lint-rubric action. Week 2 the eval threshold and the perception consistency rule both run on the Monday 09:00 UTC snapshot, and the refund clause fires on either of them missing.

Yes. Playwright MCP captures accessibility snapshots rather than screenshots by default, which is why it works with non vision models and is faster and cheaper to run at scale. Browser-use captures both representations and lets the model reason over each, which is the reference implementation of the hybrid pattern. Stagehand sits in the same family. The split between the DOM driven family and the vision only family (Anthropic Computer Use, OpenAI CUA) is the architectural fault line that produces the 12 to 17 point reliability gap.

Often it does not, which is the case the vision fallback exists for. A real partner portal we worked behind randomises ids and strips role attributes on every page load, so the accessibility tree comes back as roles=presentation only. The case sits in eval/cases.yaml with perception_mode=screenshot_only and perception_reason=anti_bot_dom_stripped. The runbook section names the iframe selector and the failure signature (tool call returns element_not_actionable five steps in a row). When the vendor ships a layout pass on the second Tuesday of the month, on call opens a tracking PR labeled vendor-drift the next day. The case stays on screenshot. We do not try to flip it back, because the roles will not return.

Three numbers in rubric.yaml under perception.budgets. ax_tree_only sits at max_input_tokens_per_step around 6,000, which is enough headroom for a noisy page plus the conversation. hybrid sits around 60,000, which is one screenshot at a useful detail tier plus the tree. screenshot_only sits around 80,000, which allows two screenshots in a single step for diff style reasoning. The CI gate asserts that case.token_budget matches the budget for the case's perception_mode. If a developer wants a higher budget on a single case, they edit the case row, not the global default. The audit trail is in git, not in Slack.

A vision-fallback section that lists each screenshot_only case, its selector, the reason it is on vision, and the failure signature an on call engineer should look for. The point is that a different engineer in week 7, on call for an alert, can read the runbook cold and decide whether the regression is in the agent (flip a case, regenerate fixtures) or in the upstream UI (open a vendor-drift PR, do not flip the case). Without that section, every regression triage costs an hour pinging the original author. With it, the median triage on these cases drops to about 15 minutes.

fde10x is the forward deployed engineering practice that ships these gates, this rubric, and this runbook into your repo. We do not sell Stagehand or Computer Use; we use whichever stack your eval rubric chooses, on whichever model provider you prefer (Anthropic, OpenAI, Bedrock, Vertex, Azure OpenAI, or open weight). What we leave behind is the operational glue that makes the perception choice durable: the rubric file, the lint-rubric action, the pilot-gate workflow, and the runbook structure. Two named senior engineers ship those into your main branch in two to six weeks, and the agent keeps shipping after we leave because no part of the runtime depends on us.