Computer use agents for legacy UIs: Gemini Flash vs Azure

Google shipped computer use as a built-in tool inside Gemini 3.5 Flash yesterday, 24 June 2026, in public preview. Until now computer use lived in a standalone model; it is now a native capability of the main Flash model, available through the Gemini API and the Gemini Enterprise Agent Platform. Google's framing is direct: agents that can see, reason and take action across browser, mobile and desktop environments, aimed at long-horizon enterprise automation such as continuous software testing and knowledge work. If you run an Azure-first shop in Sweden, this is not a product you will adopt tomorrow. It is, however, the clearest price-and-capability signal yet for a question most enterprise IT departments carry on their backlog: can an AI agent finally automate the legacy UIs that have no API, and should it replace the RPA licenses you renew every year? Azure AI Foundry has had its own answers in preview since 2025, and the two platforms have taken visibly different architectural bets. Comparing them is useful even if you never call a Google endpoint.

What Google actually shipped on 24 June

The concrete changes in the announcement:

Native tool, not a separate model. Computer use is now invoked as a built-in tool of Gemini 3.5 Flash rather than a dedicated computer-use model. That placement matters: Flash is Google's fast, low-cost tier, which signals an intent to make per-step agent actions cheap enough for high-volume automation.
Three environment classes. The agent operates across browser, mobile and desktop environments, not browsers alone. Google names continuous software testing, application analysis, documentation auditing and autonomous task completion as target workloads.
Availability. Public preview through the Gemini API and the Gemini Enterprise Agent Platform, with a hosted demo environment from Browserbase and a reference implementation on GitHub.
Benchmark position. The launch materials include an OSWorld benchmark chart; coverage of the launch reports a score of 78.4% on OSWorld-Verified. Treat single-benchmark numbers as directional, but OSWorld is the closest thing the field has to a standard test for desktop-level computer use.
Safety controls. Google describes targeted adversarial training against prompt injection, plus two optional enterprise safeguards: requiring explicit user confirmation for sensitive or irreversible actions, and automatically stopping a task when an indirect prompt injection is identified. Google itself recommends combining these with sandboxing, human verification and access controls.

Two things were absent from the announcement and matter for planning. There is no published pricing specific to the computer-use tool, so any cost model you build today rests on measurement, not a rate card. And public preview means the usual caveats: no SLA, and behaviour that can change before general availability.

What an Azure-first team already has in Foundry

Azure AI Foundry Agent Service has been building the same category since 2025, split across two tools with different architectures, plus a new management layer.

The Computer Use tool: vision-based, like Gemini

Foundry's Computer Use tool has been in preview in the Agent Service since September 2025, with feature parity with the Azure OpenAI Responses API. It is backed by the computer-use-preview model (version 2025-03-11), which receives screenshots of a computer together with instructions and responds with actions: moving and clicking the pointer, entering text, navigating. This is the same architectural family as Gemini's tool. The model sees pixels, decides an action, the harness executes it, and a new screenshot comes back. Anything a human can see on screen is in scope, including desktop applications that no DOM-based tool can reach.

The Browser Automation tool: DOM-based, a different bet

Foundry's Browser Automation tool, in preview since August 2025, takes the opposite approach. It connects the Agent Service to Playwright Workspaces provisioned in your own Azure subscription, which launch isolated, sandboxed browser sessions inside your Azure boundary. Instead of screenshots, the model reads the page structure: the DOM and accessibility tree. Microsoft's description is that the agent reasons "about web elements by their roles and labels, not just pixels." It handles form fills, filters, reservations and multi-step web processes, and every browser session runs on compute you own, in the region you chose.

Toolboxes: the management layer arrived this month

At Build 2026 on 2 June, Microsoft put Toolboxes into public preview: a single managed endpoint for every tool type, where you configure tools once, point any MCP client at one URL, and let Foundry handle authentication, lifecycle and governance. The same announcement put hosted agents on a path to general availability within 30 days, with each session running in its own sandbox with dedicated compute, memory and filesystem. For teams that want computer-use or browser agents wired into an existing agent estate with central auth and observability, this is the integration surface to plan around.

Vision agents vs DOM agents: the split that decides your project

The Gemini announcement makes the industry's fork explicit. There are now two mature architectures for UI automation with LLMs, and they fail in different ways.

Vision-based agents (Gemini 3.5 Flash computer use, Foundry's Computer Use tool) operate on what is rendered. Their reach is their advantage: a 2004-era Java Swing ERP client, a Citrix-published session, a mobile app, a terminal emulator wrapped in a Windows form. If a human temp worker could operate it by looking at the screen, a vision agent can attempt it. The costs follow from the same property. Every step is a screenshot round-trip through a multimodal model, so latency and token consumption scale with step count. Element targeting is probabilistic: the model estimates where to click, and a changed resolution, theme or overlapping dialog can turn a reliable flow into a flaky one. Long-horizon tasks compound per-step error rates.

DOM-based agents (Foundry Browser Automation on Playwright) operate on structure. Targeting an element by role and label is far more precise than estimating pixel coordinates, sessions replay more deterministically, and the payload sent to the model is text rather than images. The boundary is hard, though: no DOM, no agent. Desktop applications, Citrix sessions and anything canvas-rendered are out of scope. Web applications with pathological markup, the kind where every element is an unlabeled div, degrade the approach exactly where legacy systems are worst.

Classic RPA (UiPath, Blue Prism-style selector recording, Power Automate desktop flows) still exists as the third option, and its trade-off is unchanged: deterministic and cheap per execution, expensive and slow to maintain because every UI change breaks a recorded selector. LLM agents invert that: resilient to cosmetic change, but nondeterministic per run.

A decision framework for the RPA-replacement question

Rule zero: exhaust the API first. Every hour spent making an agent click through a UI is an hour not spent asking the vendor for the integration endpoint that probably exists. UI automation of any kind is the fallback, chosen only when the system genuinely exposes no programmatic surface, or when the licensing cost of the API tier exceeds the automation budget.

When the fallback is genuinely needed, walk these questions in order:

1. Is the target a browser application with a real DOM? If yes, start with Foundry's Browser Automation tool. You get precise element targeting, sessions inside your own Azure subscription and region, and lower per-step model cost. Reserve vision agents for pages where the DOM is unusable.
2. Is the target a desktop app, Citrix session or mobile UI? Then a vision agent is the only LLM option. Compare Foundry's Computer Use tool against the new Gemini capability on your actual workflows; OSWorld scores do not transfer reliably to your specific ERP screens.
3. Is the workflow high-volume and fully stable? Thousands of identical executions per day on a UI that has not changed in years is still classic RPA territory. Per-run LLM cost and nondeterminism both argue against an agent here.
4. Is the workflow low-volume, variable, or judgment-laden? This is where computer-use agents earn their place: the long tail of processes that never justified an RPA build because development cost exceeded the manual effort saved.
5. What appears on screen during the task? If the UI displays personal data, the vision architecture means that data travels to the model endpoint inside every screenshot. That single fact drives most of the EU analysis below.
6. Can the task tolerate a confirmation gate? Both Google's safeguards and sound engineering practice point the same way: irreversible actions (payments, deletions, submissions to authorities) need a human confirmation step in the loop. If the business case only works fully unattended, the risk assessment gets much harder.

Cost modelling without a rate card

Since neither the Gemini computer-use tool nor the Foundry previews come with automation-specific pricing you can plug into a spreadsheet, build the model from drivers you can measure in a one-week pilot:

Steps per task. Instrument the agent loop and count model round-trips per completed workflow. Vision agents typically need one multimodal call per action, so a 40-click workflow means at least 40 screenshot-bearing requests.
Tokens per step. Screenshots are the dominant token cost in vision agents; DOM snapshots are usually cheaper. Log actual token counts rather than estimating.
Success and retry rate. A task that succeeds 90% of the time costs more than 1.11 times the happy path, because failures burn steps before they fail and humans must triage them.
Human review overhead. Confirmation gates and exception handling are staff time. Price them in from the start, since they are also your compliance story.
The comparison baseline. Against RPA, the honest baseline is license cost plus the maintenance hours your team spends repairing broken selectors per year. Against manual work, it is the fully loaded hourly cost of the people doing the task today.

Google's choice to put computer use in the Flash tier is a statement that per-step economics are meant to work at scale. Verify that claim on your own workload before believing it, and run the same measurement against the Foundry tools so the comparison is like for like.

The Swedish and EU angle: screenshots are personal data in transit

Data residency and GDPR. A vision agent automating your legacy HR or invoicing UI ships screenshots of that UI to a model endpoint many times per task, and those screenshots contain whatever the screen showed: names, personnummer, salaries, health-adjacent case data. That makes the model provider a processor for everything visible on screen, so Article 28 terms, Article 30 records and your transfer analysis must cover the screenshot stream itself. Here the architectural split becomes a compliance split. Foundry's Browser Automation tool runs its Playwright sessions inside your own Azure subscription in your chosen EU region and sends structured text to the model, which is materially easier to document than a stream of full-screen images. For any vision agent, from either vendor, pin down where inference runs and whether screenshots are retained for abuse monitoring or training before the pilot touches production data, and reduce exposure mechanically: mask sensitive fields, crop capture regions, use test data during development.

The AI Act clock. As of this week, the next milestone is 2 August 2026, when obligations for high-risk systems under Annex III begin to apply. UI automation as such is not high-risk, but the classification follows the decision being automated, not the clicking mechanism. An agent that operates a legacy UI to process credit applications or recruitment decisions is participating in a high-risk context regardless of whose model drives it. Google's confirmation-gate safeguard, and its Foundry equivalents built with human approval steps, map directly onto the human-oversight expectations you will need to evidence.

Security and NIS2 posture. A computer-use agent logs into systems with credentials and acts with a user's authority, which makes it a service account with a very wide blast radius. Treat it that way: dedicated agent identities with least privilege, session recording for audit, and network egress controls on the sandbox. Indirect prompt injection deserves specific attention, since a malicious string rendered anywhere on screen becomes model input. Google shipping automatic task termination on detected injection as a launch feature tells you the vendors consider this attack practical, not theoretical. For organisations in NIS2 scope, agent identities and their monitoring belong in the same access-control documentation as every other privileged account.

Procurement reality. For Swedish public-sector and regulated buyers, preview status is itself a finding: no SLA, mutable behaviour, and in Gemini's case no published tool pricing. Pilots are defensible now; production commitments belong after general availability, with the data-flow documentation above attached to the upphandling file.

Direct takeaways

Gemini 3.5 Flash now has native computer use in public preview: browser, mobile and desktop, via the Gemini API and Gemini Enterprise Agent Platform, with adversarial training and optional confirmation and injection-stop safeguards.
Azure-first teams already have both architectures in Foundry Agent Service previews: the vision-based Computer Use tool (since September 2025) and the DOM-based Browser Automation tool on Playwright Workspaces in your own subscription (since August 2025), with Toolboxes as the new managed tool endpoint since Build 2026.
Choose by target surface: real DOM means DOM-based automation first; desktop, Citrix and mobile mean vision agents; high-volume stable flows still favour classic RPA; always exhaust APIs before automating any UI.
Build cost models from measured steps per task, tokens per step and success rates in a pilot, because no automation-specific rate card exists yet on either side.
Treat screenshots as personal data in transit: processor terms, residency, retention and field masking before production, and dedicated least-privilege identities for every agent.
Watch the 2 August 2026 AI Act milestone: classification follows the decision your agent automates, not the tool that clicks the buttons.