Browser Automation: Giving Your Agent Hands on the Web

Why a Browser, Not Just Search and Extract

The Web toolset from Lesson 2 (web_search, web_extract) handles a lot, but it cannot click a button, fill out a login form, or navigate through a multi-step flow. Browser automation gives your agent an actual browser session it can interact with the way a person would -- navigating, clicking, typing, scrolling -- for tasks that genuinely require interaction rather than just reading.

The Available Backends

• Browserbase -- cloud-managed browsers with anti-bot tooling and residential proxies, for sites that actively try to block automation
• Browser Use -- an alternative cloud browser provider, accessed via a REST API
• Firecrawl -- cloud browsers with built-in scraping capabilities baked in
• Camofox -- a local, self-hosted Firefox fork with fingerprint spoofing, for when you want anti-detection without paying for a cloud service
• Local Chrome/Brave/Edge via CDP -- attach directly to a browser you are already running on your own machine
• Local Chromium -- the default mode, using the bundled agent-browser CLI, no separate account or service needed

The default (Local Chromium) needs nothing from you beyond the install -- no signup, no key. The cloud backends are an upgrade you opt into, not a requirement to get browser automation working at all.

Setting Up a Cloud Provider

If you do want a cloud backend, each option needs its own account and credentials, created directly on that provider's own site (the Browserbase dashboard, the Browser Use dashboard, the Firecrawl dashboard), then added to ~/.hermes/.env:

BROWSERBASE_API_KEY=...
BROWSERBASE_PROJECT_ID=...

BROWSER_USE_API_KEY=...

FIRECRAWL_API_KEY=...

If you are already a Nous subscriber, you can skip managing separate credentials entirely and use the Tool Gateway via hermes setup --portal instead.

Hybrid Mode: The Best of Both Worlds

If you are using a cloud backend but also need to reach things on your own local network -- an internal dashboard at localhost or a device at 192.168.x.x, for instance -- hybrid mode automatically routes private URLs to a local Chromium sidecar while keeping public URLs on your configured cloud provider. You do not need to manually decide which backend handles which request.

What the Agent Can Actually Do

• browser_navigate -- load a URL
• browser_snapshot -- get an accessibility tree with interactive element IDs
• browser_click / browser_type -- interact with elements using those ref IDs
• browser_scroll / browser_press -- page navigation and key presses
• browser_vision -- take a screenshot and have it analyzed visually
• browser_console -- access browser logs and execute JavaScript
• browser_cdp -- raw Chrome DevTools Protocol passthrough, for anything the higher-level tools do not cover
• browser_dialog -- handle native JavaScript dialogs (alerts, confirms)
• browser_get_images -- list images present on the page

Accessibility Trees, Not Pixel Coordinates

This is the key architectural choice that makes browser automation reliable for an LLM: pages are represented as accessibility trees -- text-based structural snapshots -- rather than as raw pixel images the model has to interpret visually every time. Interactive elements receive stable reference IDs like @e1, and the agent interacts with those IDs rather than guessing screen coordinates. As the documentation puts it directly, "text-based interaction relies on accessibility tree, not pixel coordinates" -- this is far more robust to layout shifts and rendering differences than coordinate-based clicking would be.

browser_vision exists as a fallback for the cases where the accessibility tree alone is not enough -- a CAPTCHA, a canvas-based UI, or anything else that is fundamentally visual rather than structural.

Cleanup

Browser sessions clean up automatically after a period of inactivity -- you do not need to manually close sessions you have finished with, though being mindful of how many you leave open concurrently is still good practice on cloud backends where each session may carry a cost.

The next lesson covers two more capabilities your agent can use visually: analyzing images you give it, and generating images of its own.