# Agents pass 52% of Chrome's web checks: your guidance plan

Chrome's evaluation found frontier coding agents pass 52% of its web platform checks unaided and 91% with Modern Web Guidance installed, while Stripe's benchmark of 14 agent setups found only 29.4% of runs met every mandatory criterion. The gaps cluster in accessibility, security headers and payment details, which gives Swedish teams under Lag (2023:254) a concrete reason to install vendor guidance and add strict checks to the pipeline.

- Published: 2026-11-22 · Category: AI & Machine Learning · Tags: AI Coding Agents, Modern Web Guidance, Stripe, Accessibility, European Accessibility Act, Claude Code, Content Security Policy, Azure Static Web Apps, Evals
- Author: Technspire AB, Stockholm (https://technspire.com)
- Canonical: https://technspire.com/en/blog/agents-pass-52-percent-of-chromes-web-checks-your-guidance-plan

52%. That is the average share of Chrome's web platform checks that frontier coding agents passed without help, according to the Chrome team's evaluation published on 11 November 2026 alongside Modern Web Guidance 1.0. With the guidance installed, the same agents passed 91%. Eight days later Stripe published its own benchmark of 14 model and harness pairs building payment integrations. The average run met 78.1% of the graded criteria, yet only 29.4% of runs met every mandatory one. Two vendors measured two different domains and arrived at the same finding: agent-written code works in the demo and misses the last details, and vendor-authored guidance loaded into the agent closes part of that gap. For a Swedish team shipping consumer web services with Claude Code, Codex or Copilot, those missing details land in two places with legal weight: accessibility and payments.

## What Chrome measured

Modern Web Guidance is a set of agent skills that carry web platform best practices and browser compatibility data. The source repository says it is stewarded by the Chrome, Edge, Firefox and MDN teams, is licensed under Apache-2.0, and covers 262 web features across 204 use cases. Each guide defines expectations that must hold if its advice was followed. The eval sends one prompt to a guided agent and to an unguided one, then checks both outputs against those expectations.

Philip Walton's post reports the averages (52% unguided, 91% guided) for "frontier coding agents (as of Nov 9, 2026)". It does not name the models in the text or give per-topic pass rates; those sit on the project's eval dashboard. What the post does give is a ranked list of the ten areas where unguided agents fail most.

| Area | What unguided agents write | What the platform already offers |
| --- | --- | --- |
| Focus management | Manual focus traps, aria-modal, no focus restoration, weak focus outlines | A modal dialog element, the inert attribute, roving tabindex |
| Reduced motion | Global smooth scrolling, unguarded scroll-driven animations, a blanket 0.01ms reset | Targeted prefers-reduced-motion rules that keep essential transitions |
| Form validation and autofill | Blur-based JavaScript validation, placeholder-only hints, aria-invalid out of sync | :user-invalid, aria-describedby hints placed above the input |
| Swipe and touch | pointermove handlers with custom physics, no keyboard path | CSS Scroll Snap, scroll-driven animations |
| Dialogs and popovers | Fixed overlays, click-outside handlers, overflow:hidden scroll locks | popover, @starting-style, the dialog element |
| Dark mode | No color-scheme meta tag, binary toggle with no return to the OS default | color-scheme, light-dark() |
| Responsiveness (INP) | setTimeout queues, display:none view switching | scheduler.yield(), scheduler.postTask(), content-visibility |
| Layout shift (CLS) | Video without width and height, static vh units, anti-flicker snippets | aspect-ratio with object-fit, dvh and svh, render-blocking hints |
| DOM XSS | Hand-rolled sanitizers, pass-through Trusted Types policies, host-allowlist CSPs | setHTML(), hash or nonce CSPs with 'strict-dynamic' |
| Passkeys | Hand-rolled Base64URL conversion, no capability check before showing passkey UI | parseCreationOptionsFromJSON(), getClientCapabilities() |

Read the middle column again. None of it is broken code. A manual focus trap works. A setTimeout queue works. These are the patterns that dominated tutorials and repositories for a decade, so they dominate the training data, and the agent reproduces them with confidence. The platform moved on and the corpus did not. One line in the security section deserves separate attention: the post says the models did not add a Content Security Policy or cross-origin isolation headers unless the prompt asked for them.

## What Stripe measured

Stripe's benchmark, published on 19 November, is more detailed about method. It contains 69 public scenarios: 24 for one-time payments, 26 for subscriptions and 19 for Connect. Each agent ran every scenario twice with the Stripe agent plugin and twice without, which gives 276 runs per agent. A simulated developer held a hidden set of needs and answered the agent's questions. A judge model then graded each criterion, labelled MUST or SHOULD in the RFC 2119 sense, against the transcript and the resulting workspace.

All 14 setups scored higher with the plugin, by 2.7 to 7.9 percentage points on the average score. The share of API-call parameters that referenced outdated options fell from 7% to 1.3%. Strict grading, where a run passes only if every MUST criterion passes, is where the numbers get uncomfortable.

| Model (harness) | Average score, no plugin to plugin | Strict pass, no plugin to plugin | Cost per run with plugin |
| --- | --- | --- | --- |
| GPT-6 Astra (Codex) | 85.5% to 90.0% | 41.9% to 52.9% | $3.63 |
| GPT-6.1 Sol (Codex) | 85.6% to 89.6% | 39.0% to 50.7% | $0.61 |
| Opus 5.5 (Claude Code) | 83.5% to 86.2% | 35.3% to 36.0% | $3.15 |
| Fable 5.2 (Claude Code) | 81.1% to 85.2% | 28.4% to 43.2% | $5.35 |
| Sonnet 5.5 (Claude Code) | 78.0% to 82.6% | 29.4% to 33.1% | $1.16 |
| Haiku 5.5 (Claude Code) | 73.5% to 76.2% | 23.5% to 30.9% | $0.09 |
| GPT-6 Luna (Codex) | 72.0% to 79.1% | 17.3% to 29.1% | $0.07 |
| Gemini 4 Argon (OpenCode) | 74.3% to 77.6% | 20.6% to 31.6% | $3.86 |

The table shows eight of Stripe's 14 rows; the rest are in the source. The best strict result in the whole benchmark is 52.9%. Put differently, the strongest setup Stripe tested, with Stripe's own guidance loaded, still missed a mandatory criterion in nearly half its runs. Stripe's explanation is that many models build "almost correct implementations" and struggle with the final details.

Two examples from the write-up show what a final detail looks like. Without the plugin, one agent hardcoded `payment_method_types: ['card']` where the current pattern is `automatic_payment_methods: { enabled: true }`. In a fraud-review scenario, another checked only that the PaymentIntent status was succeeded before fulfilling, which Stripe says "can allow orders to ship after refund or dispute". The first mistake costs conversion. The second costs inventory.

### The cost math: price per strict pass

Stripe publishes cost per run and strict pass rate side by side, so you can divide one by the other. The result below is our arithmetic, not Stripe's. It answers a narrow question: how many dollars of agent time does one fully correct integration cost if you keep rerunning until it passes? It assumes reruns are independent, which they are not quite, and it ignores the human time spent discovering which runs failed.

| Model, with plugin | Cost per run | Strict pass rate | Cost per strict pass |
| --- | --- | --- | --- |
| GPT-6 Luna | $0.07 | 29.1% | $0.24 |
| Haiku 5.5 | $0.09 | 30.9% | $0.29 |
| GPT-6.1 Sol | $0.61 | 50.7% | $1.20 |
| Sonnet 5.5 | $1.16 | 33.1% | $3.50 |
| GPT-6 Astra | $3.63 | 52.9% | $6.86 |
| Opus 5.5 | $3.15 | 36.0% | $8.75 |
| Fable 5.2 | $5.35 | 43.2% | $12.38 |

Every figure in the right-hand column is small next to an hour of a developer's time. That is the real reading of this table. Token spend is a rounding error; the expensive part is the review that tells a passing run from a failing one, because a failing run looks finished. Budget for verification, and choose the model on strict pass rate before price. Stripe also cautions that models rank differently across payments, subscriptions and Connect, so test on your own scenario type before standardising.

## Why the fix is guidance and where it stops

Both vendors ship the same remedy: skills the agent loads on demand, written by the people who own the platform. Stripe's plugin adds MCP tools as well, including a documentation search and an implementation planner. The tool-usage data shows what changes in practice. Pooled across agents, web searches per run fell from 1.26 to 0.05 and page fetches from 4.84 to 1.78. The agent stops guessing from stale search results and reads the current answer from the source. Our earlier piece on [Spotify's Claude Code context pattern](/en/blog/spotify-cut-claude-code-tokens-90-copy-the-pattern) covers the same mechanism from the token-cost side.

Now the caveats, because both studies have them.

- **Vendors graded their own guidance.** Chrome's expectations are derived from the guides, so a guided agent is being tested on the material it was just handed. A jump from 52% to 91% shows that agents follow the guidance. It does not show that 91% of the output is correct by any independent standard.
- **Stripe's judge is a model.** Stripe ran a second judge from another vendor as a bias probe and reports uniform movement, with the second judge more lenient. Stripe also states that runtime behaviour is not verified yet: a well-designed integration that does not run can score full marks.
- **Testing dropped.** With the plugin, agents wrote about 40% fewer lines of tests and ran tests in 42% of evals against 53% without. Guidance made the agents more confident and less inclined to check their work.
- **A plugin is a supply chain.** You are installing third-party instructions and tools into a process that can edit your repository. After the [agent-framework package compromise](/en/blog/microsofts-agent-framework-malware-your-azure-audit) earlier this month, treat it like any other dependency.

## The Swedish and EU angle: accessibility law and local payment methods

The first three rows of Chrome's list are accessibility failures. In Sweden that has been a compliance matter since 28 June 2025, when Lag (2023:254) om vissa produkters och tjänsters tillgänglighet entered into force and implemented the European Accessibility Act. The law covers consumer e-commerce services and consumer banking services, among others. Section 10 exempts services provided by micro-enterprises, defined as fewer than ten employees with turnover or balance sheet total of at most EUR 2 million. For everyone else, section 39 sets the sanction fee at no less than SEK 10,000 and no more than SEK 10,000,000.

A checkout modal that does not return focus when it closes, an error message that is not tied to its input, a focus outline below 3:1 contrast: these are the defects an accessibility audit writes up, and Chrome's data says an unguided agent produces them by default. If your team has moved a large share of front-end work to agents since mid-2025, your accessibility regression risk has gone up with it, and the code reviews probably did not catch it because the code looks conventional.

The payments side has a local twist. A hardcoded card-only payment method list hides every other method you have enabled in the Stripe Dashboard. In a market where shoppers expect Swish and Klarna at checkout, that is a conversion problem created by a single outdated parameter. The fulfilment example matters for the same reason it matters anywhere: goods shipped against a payment that is later reversed are a loss. Teams building agent-initiated purchases should also read our note on [how agents pay in the EU today](/en/blog/chatgpt-wallet-skips-eu-how-agents-pay-here-today).

## A rollout plan for a team on Azure

### 1. Install the guidance, pinned and reviewed

Modern Web Guidance installs through an interactive wizard and needs Node.js 22 or later. The documented command uses the latest tag. For a team repository, resolve that to a specific version, commit the generated skill files and review changes to them in pull requests like any other code.

```
# Documented install (interactive wizard, Node.js 22+)
npx modern-web-guidance@latest install

# Query a guide directly
npx modern-web-guidance@latest search "animate a dialog modal backdrop"

# Scan an existing codebase for legacy patterns
npx modern-web-guidance@latest legacy-scan .
```

For Stripe, the documented commands are per agent:

```
# Claude Code
claude plugin install stripe@anthropic-plugin-directory

# Codex
codex plugin add stripe@openai-curated

# Or let the Stripe CLI detect your agents
stripe agent setup
```

Stripe's documentation says the plugin gives the agent access to live account data. Connect it to a sandbox with a restricted key while the agent is writing code. An agent that is building a checkout has no reason to read production customers.

### 2. Run the legacy scan on what agents already shipped

The legacy-scan command exists for the backlog. Run it against the front-end code merged since your team adopted coding agents and triage the output against the table above. Start with focus handling and forms in any consumer flow covered by the accessibility law, then the security items. We have not yet verified the scanner's exit-code behaviour, so run it as a report first and decide later whether it can gate a build.

### 3. Put the checks agents skip into the pipeline

Guidance raises the pass rate. It does not make it 100%, and Stripe's data shows guided agents test less. So the verification has to live somewhere the agent cannot skip it. In an Azure DevOps or GitHub Actions pipeline, that means an automated accessibility pass (axe-core through Playwright is the common choice) on the consumer flows, Lighthouse budgets for INP and CLS, and an integration test against Stripe's sandbox that exercises refund and dispute paths before fulfilment. Automated accessibility tooling catches only part of what a manual audit finds, so keep keyboard-only walkthroughs of checkout and login in the release checklist.

### 4. Set the security headers at the platform, not in the prompt

Chrome found that agents omit a Content Security Policy unless asked. Do not rely on remembering to ask. On Azure Static Web Apps, set it once in staticwebapp.config.json so every response carries it regardless of who or what wrote the page:

```
{
  "globalHeaders": {
    "Content-Security-Policy": "script-src 'sha256-REPLACE_WITH_YOUR_HASH' 'strict-dynamic'; object-src 'none'; base-uri 'none'; require-trusted-types-for 'script'",
    "Cross-Origin-Opener-Policy": "same-origin",
    "X-Content-Type-Options": "nosniff"
  }
}
```

Static headers suit hash-based policies. A server-rendered Next.js app on App Service or Container Apps should generate a per-request nonce in middleware instead. Start in report-only mode, read the violation reports for a week, then enforce. Azure Front Door's rules engine can add the same headers at the edge for apps you cannot change quickly.

### 5. Write the MUST list for your own domain

The most transferable idea in Stripe's method is the split between MUST and SHOULD criteria, graded separately, with a strict pass that requires every MUST. Your own platform has the same kind of rules: every outbound call uses managed identity, every table has row-level security, every consumer form has a programmatic error association. Write ten of them down as a skill file in the repository, and write the matching check. Stripe's eval framework is public on GitHub if you want a starting structure. Vendor guidance covers the vendor's platform; nobody else is going to write the guidance for yours.

## Checklist

1. Install Modern Web Guidance in every front-end repository where agents write code. Pin the version and commit the skill files.
2. Install the Stripe plugin where relevant, connected to a sandbox with a restricted key.
3. Run legacy-scan on agent-era code and triage focus, forms and CSP findings first.
4. Add axe-core and Core Web Vitals budgets to the pipeline for consumer flows covered by Lag (2023:254).
5. Test refund and dispute paths before fulfilment in the Stripe sandbox.
6. Set CSP and isolation headers at the hosting layer.
7. Replace any hardcoded payment_method_types list with automatic payment methods and confirm Swish and Klarna appear.
8. Write your own MUST criteria and a strict check for each.
9. Rerun a sample of agent tasks with and without guidance on your codebase, and keep the numbers. Vendor benchmarks tell you the direction; only your own tell you the size.

## Sources

- [Chrome for Developers: Top 10 things AI gets wrong in web dev (11 November 2026)](https://developer.chrome.com/blog/top-10-things-ai-gets-wrong-in-web-dev)
- [GoogleChrome/modern-web-guidance-src on GitHub](https://github.com/GoogleChrome/modern-web-guidance-src)
- [Stripe: Measuring real-world coding agent performance (19 November 2026)](https://stripe.dev/blog/stripe-integration-evals-methodology)
- [Stripe documentation: Agent plugins for Stripe](https://docs.stripe.com/agents/plugin)
- [stripe/forge: benchmark and eval framework](https://github.com/stripe/forge)
- [Lag (2023:254) om vissa produkters och tjänsters tillgänglighet](https://www.riksdagen.se/sv/dokument-och-lagar/dokument/svensk-forfattningssamling/lag-2023254-om-vissa-produkters-och-tjansters_sfs-2023-254/)

---

Technspire AB builds AI agents, Azure OpenAI solutions, and production web platforms for Swedish and EU enterprises. Book a call: https://calendly.com/technspire · hello@technspire.com · More articles: https://technspire.com/en/blog · Site overview for agents: https://technspire.com/llms.txt
