Core concepts

The conventions Holo is trained on. Skip one and accuracy drops.

The loop

Pick an output format

Agent-loop format	Holo4	Holo3.1	Holo3
Function calling	Use this	Use this
Structured outputs		Use this	Use this

Single calls such as element localization take structured outputs on every model, and function calling on every model except Holo3.

Pick one loop format and stay in it. The formats differ only in how you declare tools and read the output. The rest of this page applies to both.

Answer tool

Holo ends a task by calling a tool named answer, with its report in the content argument. Register it with your action tools, in both formats, and stop the loop when it is called. A step with no tool call is not a stop: append the message and keep looping.

Reasoning and request settings

Each call returns a thinking trace in message.reasoning and the action in content (or tool_calls). Holo is trained to plan before each step, so keep thinking on in agent loops:

Python

extra_body={"chat_template_kwargs": {"enable_thinking": True}}

Task	Thinking	temperature
Agent loops	On	0.8
Single calls: element localization, document OCR	Off	0.0

Past reasoning is dropped between turns, so anything the model must remember goes in content. When you append the assistant message, push only the parsed output, never the reasoning.

Prompt caching

When a prompt starts with the same tokens as a recent request to the same model, such as a fixed system prompt or the growing history of an agent loop, the server reuses its work on that part. There is nothing to enable. The reused part is reported as usage.prompt_tokens_details.cached_tokens and billed at the cached input price. The rest of the prompt and all output are billed at the listed rates. Each model’s cached input price is in Models and in GET /v1/models as pricing.input_cache_read.

Caching is best effort. Keep the stable part of the prompt at the start, and expect the count to vary between requests.

Coordinates in [0, 1000]

Send a screenshot at any size. Holo returns integer coordinates in [0, 1000], normalized to that image, origin top-left. (932, 880) is 93.2% across and 88% down: pixel (1342, 792) on a 1440 × 900 screenshot. Scale back with its dimensions:

Python

abs_x = int((x / 1000) * screenshot.width)
abs_y = int((y / 1000) * screenshot.height)

Send and scale against the same image bytes: any resize, crop, or DPI mismatch misclicks. Pick one pixel unit (CSS or device) and keep it end to end.

Observations

An observation is what the agent sees after it acts. Agents without a screen (a code runner, or MCP tools) need nothing extra: the task is the first user message and each tool result goes back as a tool result. The rest of this section is for screen agents.

Each step, send the current screenshot as a user message wrapped in <observation> tags:

Python

messages.append({"role": "user", "content": [
    {"type": "text", "text": "<observation>\n"},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "text", "text": "\n</observation>"},
]})

Image budget

Keep at most the last 3 screenshots in context: more degrades accuracy. Replace older ones with a text placeholder and keep their <observation> wrapper. Call this helper after appending each observation. The function-calling and structured-output loops use it.

Python

def trim_to_last_n_images(messages, n=3):
    seen = 0
    for msg in reversed(messages):
        if msg["role"] != "user" or not isinstance(msg["content"], list):
            continue
        for chunk in msg["content"]:
            if chunk.get("type") != "image_url":
                continue
            seen += 1
            if seen > n:
                chunk["type"] = "text"
                chunk["text"] = "[screenshot evicted]"
                chunk.pop("image_url", None)

Common pitfalls

Symptom	Likely cause
Clicks land off target	Coordinates not scaled to the screenshot, or the screenshot was resized or cropped between send and execute
Clicks land far off on a Retina display	CSS pixels mixed with device pixels
Wrong element picked	Ambiguous element description (“the button”). Name the label, position, and neighbors (“the blue Save button below the form”)
Single call slow or verbose	Thinking left on. Single calls need enable_thinking=False and temperature=0.0
Model loops, forgets earlier facts	Durable facts not carried in content (or in note for structured outputs), or old <observation> wrappers dropped instead of stripped to text
Context window fills up	Image budget not enforced
Reasoning leaks into the action	<think>...</think> written into content instead of read from message.reasoning
Quality collapses after one bad step	Raw model output replayed in history instead of the parsed result
Run stops early, or never ends	Loop exits on an empty tool call instead of on answer, or no answer tool registered
Function calling: tool calls come back as plain text	tool_choice not required, or the system prompt has conflicting tool-format examples
Function calling: tool result ignored	Sent as a user message instead of a tool message with a matching tool_call_id
Structured outputs: free-form text	structured_outputs.json missing, or the schema lacks a Literal tool_name
Structured outputs: tool result has no effect	Sent as a tool message instead of a user message with a <tool_output> wrapper
Structured outputs: reliability drops	Schema in structured_outputs but missing from the <output_format> block, or the reverse