Gemini 3.5 Flash Can Now Click Your Browser. Agents Just

By Lazar Milicevic · Published June 24, 2026 · 8 min read

Google shipped computer use in Gemini Flash and the demos look clean. The reality when you wire it into a real SMB workflow is messier — 20-30% failure rates on multi-step flows, modals it can't see, and login walls that kill the session. Here's what works, what doesn't, and the one supervised pattern I'm shipping for clients this month.

What "computer use" actually means at the API level

Computer use is a tool-calling loop where the model receives a screenshot, returns a structured action (click, type, scroll, key), your runtime executes it, and you send back the next screenshot. It's not magic — it's a vision model wrapped in a perception-action cycle. The shift with Flash isn't capability, it's price per step.

A single multi-step browser task — say, logging into a supplier portal, filtering an orders table, and exporting a CSV — typically runs 15 to 40 steps. Each step costs one screenshot input (roughly 1,000-1,500 image tokens at standard resolution) plus a short reasoning output. According to Anthropic's published computer use pricing, Claude Sonnet runs around $3/M input and $15/M output tokens. Gemini Flash sits at roughly $0.30/M input and $2.50/M output per Google's Gemini API pricing page. That's a ~10x difference on inputs, which is where computer use bleeds tokens.

Concrete math from a portal export task I ran last week:

Flash uses slightly more steps because it makes weaker first-pass decisions and re-tries. But at one-tenth the input price, you can afford the retries and the supervisor pattern that catches its mistakes. This is the whole reason this release matters more than another frontier model release would.

The failure modes nobody puts in the demo videos

In demos, the model logs in, clicks the right tab, fills the form. In production it does this 70-80% of the time. The other 20-30% of runs fail in specific, predictable ways, and you need to know them before you ship anything client-facing.

What I've personally watched go wrong across maybe 200 client test runs:

Where it breaks

• Modal blindness. A cookie banner or "session expiring" modal pops up, model can't see the underlying button it wants, clicks through the modal, types into the wrong field.
• Confident hallucinated clicks. Returns coordinates for a button that isn't on screen, runtime clicks empty space, model thinks the action succeeded and proceeds. This is the dangerous one — silent failure that reports success.
• Login walls and 2FA. Anything with SMS or TOTP breaks the loop. You need a pre-authenticated session cookie or a human-in-the-loop pause.
• Pagination and infinite scroll. Model scrolls, sees similar content, gets confused about whether it's progressing. Loops or stops early.
• Date pickers and dropdowns. Custom React date pickers are still the boss fight. Native <select> works fine; anything fancier is a coin flip.

A 2024 Stanford HAI evaluation of agent benchmarks noted that web agents routinely report inflated success rates because benchmarks don't test for silent failure — the agent says "done" but the actual side effect (email sent, record updated) never happened. That matches what I see. Don't trust the model's own "success" output. Verify the downstream state.

The practical rule: any step that mutates data (submit, pay, delete, send) needs an external verification check. Hit the resulting page, parse for a confirmation number, fail loud if it's missing.

The supervised workflow pattern that actually ships

Don't build autonomous agents. Build a supervised loop where the model does the boring clicking and a human signs off on the result. This is the only pattern I'm shipping to paying clients right now, and it's the one you should ship this week.

The architecture:

1. Trigger (cron, webhook, button)
   ↓
2. Spin up isolated Chromium in Docker
   ↓
3. Load pre-authenticated session cookies (NOT live login)
   ↓
4. Run Gemini Flash computer use loop against task SOP
   ↓
5. Capture: final screenshot + extracted data + action log
   ↓
6. Post to Slack / email with [APPROVE] [REJECT] buttons
   ↓
7. On approve → execute downstream side effect
   On reject → flag for human, log failure mode

The key trick: the model never directly causes the side effect. It gathers data and proposes an action. A human (or a deterministic script with strict validation) executes the final mutation. That cuts your risk surface to zero on financial and customer-facing tasks.

A minimal Python skeleton:

from google import genai
from playwright.sync_api import sync_playwright
import base64

client = genai.Client(api_key=GEMINI_KEY)

def run_supervised_task(sop_text: str, start_url: str):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        ctx = browser.new_context(storage_state="auth.json")
        page = ctx.new_page()
        page.goto(start_url)

        action_log = []
        for step in range(40):  # hard ceiling
            screenshot = page.screenshot()
            response = client.models.generate_content(
                model="gemini-flash-latest",
                contents=[
                    {"text": f"SOP:\n{sop_text}\n\nReturn next action as JSON."},
                    {"inline_data": {"mime_type": "image/png",
                                     "data": base64.b64encode(screenshot)}}
                ],
                config={"tools": [{"computer_use": {}}]}
            )
            action = parse_action(response)
            action_log.append(action)
            if action["type"] == "done":
                break
            execute(page, action)

        final_state = extract_result(page)
        browser.close()
        return final_state, action_log

Note what's missing: there's no pay(), no submit_order(), no send_email() inside the loop. The model produces a proposed final state. A separate, deterministic step (after human approval) does the mutation.

Where this kills existing tooling — and where it doesn't

Computer use kills the use case for hand-written Playwright/Selenium scripts on internal back-office portals. The kind of script you wrote in 2023 to scrape a vendor dashboard every Monday — that's now a 50-line wrapper around a model call. According to McKinsey's 2024 State of AI report, about 60% of mid-market companies cite "lack of API access in legacy tools" as a top blocker to automation. That entire category just got cheaper to address.

What it does NOT kill:

• Real APIs. If your tool has a clean REST API, use it. It's 100x faster, 1000x cheaper, deterministic, and doesn't need supervision. Computer use is for tools where the API is missing, gated, or insufficient.
• n8n / Zapier for connected SaaS. If both endpoints have integrations, classic workflow tools win on cost and reliability.
• High-volume scraping. If you're hitting 10,000 pages a day, write the scraper. Computer use at any model price doesn't pencil out at that volume.
• Anything sub-second. Each step is 2-5 seconds of model latency. A 25-step workflow takes a minute or more. Fine for weekly reports, useless for live UX.

A practical decision matrix:

A concrete first project: weekly report aggregation

The cleanest place to start is a multi-dashboard report pull. Most SMBs have 3-6 SaaS tools where someone logs in every Monday, exports a CSV, and pastes numbers into a Google Sheet. That's the wedge.

Pick three dashboards. Write the SOP as you'd write it for a new hire:

# Weekly Revenue Pull SOP
1. Go to stripe.com/dashboard. If logged out, stop and flag.
2. Click "Reports" in left nav.
3. Click "Gross volume".
4. Set date range to "Last 7 days".
5. Read the total in the top card. Record as STRIPE_TOTAL.
6. Go to dashboard.shopify.com.
7. Click "Analytics" → "Reports" → "Total sales".
8. Set date range to "Last 7 days".
9. Read "Total sales" number. Record as SHOPIFY_TOTAL.
10. Output JSON: {stripe: STRIPE_TOTAL, shopify: SHOPIFY_TOTAL, run_date: TODAY}

Feed that exact document to Flash. Run it in a sandboxed browser with pre-loaded cookies. Have it Slack you the JSON with a screenshot of each final page. You spend 90 seconds verifying the screenshots match the numbers and approve. The 45-minute Monday morning task is now 2 minutes.

This is boring on purpose. It's also the only kind of project where I've seen sub-5% production failure rates over 8+ weeks of runs.

Why bizflowai.io helps with this

The pattern above — supervised browser automation against tools without APIs — is exactly what we build at bizflowai.io for clients whose stack is half-modern SaaS and half-legacy portals. The hard part isn't the model call; it's the boring infrastructure: session cookie rotation, headless browser fleets, Slack approval queues, downstream verification, failure logging, and choosing which tasks are even safe to automate. We've already built that scaffolding, so a new workflow ships in days instead of weeks. If you've got two or three weekly tasks where someone manually clicks through a portal, those are the right first projects.

The honest take on what to do next week

Don't wait for "agents to be ready." They won't be, in the autonomous sense, for years. But the supervised version is ready right now at a price that pencils out for any task that costs you more than 30 minutes of human time per week.

The plan, in order:

• Pick one weekly browser task that touches no payments and no customer data.
• Write the SOP in plain English, the way you'd onboard an assistant.
• Stand up Flash computer use in a Docker'd Chromium with pre-auth cookies.
• Pipe the result to Slack with explicit approve/reject.
• Run it for four weeks. Log every failure. Tune the SOP, not the model.

That's the actual game. Not autonomy, not AGI, not no-code wrappers. Supervised automation on the long tail of business software that refused to be automated last year. The price just dropped enough to make it worth building.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.