Local control

Let a cloud agent drive Chrome or the whole desktop on your own machine, with your logins, files, and apps.

With local control, the agent runs in H’s cloud and drives a browser or desktop on your own machine. The agent and session lifecycle stays the same. Local control ships in the hai-agents Python SDK only.

Local control is not local mode: here only the browser or desktop is on your machine, while in local mode (Client.local()) the agent itself runs there too.

	Browser	Desktop
The agent drives	Chrome on your machine	Your real mouse, keyboard, and screen (macOS, Windows, Linux)
Use it for	Sites you are signed in to, pages on your local network	Native apps, the file system, multi-window flows
Environment	{"kind": "web", "host": "user_device"}	{"kind": "desktop", "host": "user_device"}
Install extra	hai-agents[browser]	hai-agents[desktop]
CLI server	hai local browser	hai local desktop

How it works

A local environment is a normal environment with host: "user_device". When a session starts, the SDK opens a connection inside your Python process. The connection receives the agent’s actions, carries them out on your machine, and ties this session to this machine. Observe and steer, changes, and Agent View work as for a cloud run.

• One at a time. A Python process drives one local browser (or desktop) at a time. A new session on it takes it over and cancels the earlier session.
• Tied to your process. The connection closes when your Python process exits, and the sessions it served are cancelled (Chrome stays open). If the connection drops mid-session, the session is cancelled too.
• Cloud-only fields. Vaults and browser profiles apply to cloud browsers only. Your local Chrome’s own logins fill that role.

Auto-connect scope

You start the session with	Connection
An agent spec or object passed to run_session, start_session, or create_session	The SDK connects your machine for you.
A registered agent referenced by name, or from the web app or another machine	Serve the machine with hai local browser or hai local desktop, then set the printed session_id on the user_device environment (see the CLI tabs below).
HAI_AUTO_BRIDGE=0 set	Auto-connect is off. Serve the machine yourself, as above.

Install

The drivers come from the hai-drivers package, as optional extras. Add cli for the hai command.

Install

pip install "hai-agents[cli,browser]"   # local browser
pip install "hai-agents[cli,desktop]"   # local desktop

Run a session

Create an agent whose environment sets host to user_device, then run a session with it. In Python there is nothing else to set up. From the CLI, serve your machine and point the agent at the session_id it prints (the CLI tab reuses the agent created in the Python tab).

Browser

CLI

# Serve the browser and leave it running; it prints the session_id it serves.
hai local browser

# In another terminal, route the agent's browser environment here.
hai run "Open news.ycombinator.com and summarize the top story" \
  --agent local-web \
  -o 'agent.environments[kind=web].session_id=<printed id>'

hai local browser flags: --session-id (generated when omitted) and --debug-port (default 9222).

The Chrome it drives. The SDK attaches to a Chrome with remote debugging open on port 9222. If none is running, it launches one with its own profile in ~/.hai/chrome-profile. That profile persists across runs: sign in once and the agent finds you signed in next time. The SDK never touches your everyday Chrome profile (Chrome does not allow remote debugging on it).

To drive a different Chrome, start it yourself before the session. Chrome only opens the debugging port with --user-data-dir, so pick a dedicated directory and keep it for the logins you want the agent to have.

macOS

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
  --remote-debugging-port=9222 --user-data-dir="$HOME/chrome-for-agents"

Desktop

CLI

# Serve the desktop and leave it running; it prints the session_id it serves.
hai local desktop

# In another terminal, route the agent's desktop environment here.
hai run "Open Notes and write a short summary of today's standup" \
  --agent local-desktop \
  -o 'agent.environments[kind=desktop].session_id=<printed id>'

hai local desktop downscales and recompresses screenshots before upload. Its flags:

Flag	Default	Meaning
--session-id	generated	Session id to serve.
--max-width	1920	Cap screenshot width in pixels. 0 disables.
--max-height	0	Cap screenshot height in pixels. 0 disables.
--image-format	jpeg	png, jpeg, or webp.
--quality	85	Encoding quality (1 to 100) for jpeg and webp.

Double-Esc kill switch. In an interactive terminal, hai local desktop also stops when you press Esc twice fast. On macOS, grant your terminal Input Monitoring. On Linux, it needs pynput and an X11 session (Wayland has no global key listener). If it cannot arm, it says so: use hai local stop.

OS permissions (macOS). The program running the agent (your terminal, or the app that launches Python) needs two grants in System Settings > Privacy & Security. Your first session triggers the prompts and stops with an error until you grant both. Grant them, restart the program, and run again.

• Accessibility, to move the mouse and type.
• Screen Recording, to read the screen.

On Windows and Linux there is nothing to grant. Run the program in a normal desktop session so it can reach the display.

Related: Browser configuration, Observe and steer, Local mode.