Structured outputs

The Holo agent loop as one constrained JSON object per step, for Holo3.

holo3-122b-a10b has no tools, so its loop returns one constrained JSON object per step. Holo3.1 takes this format too. Pick an output format shows which model takes which, and the rest of Core concepts applies here too.

Step format

The tool call is one field of the JSON object:

{
  "note": "Submit succeeded; receipt URL is /orders/8421.",
  "thought": "Recording the receipt URL before navigating away.",
  "tool_call": {"tool_name": "click", "element": "Continue button at the bottom right", "x": 932, "y": 880}
}

note is the model’s durable memory: facts from the current screen that later steps need (URLs, IDs, partial answers), null when nothing is new. tool_call is flat: tool_name sits next to the arguments, not in a nested args object.

Define the schema

Define each tool as a Pydantic model with a Literal tool_name, and use their union as the schema. The decoder emits exactly one variant, and you dispatch on tool_name.

Python

from typing import Literal
from pydantic import BaseModel, Field

class ClickArgs(BaseModel):
    """Click at (x, y) coordinates"""
    tool_name: Literal["click"]
    element: str = Field(description="Detailed description of the target UI element to click on")
    x: int = Field(description="X coordinate as integer in [0, 1000]")
    y: int = Field(description="Y coordinate as integer in [0, 1000]")

class WriteArgs(BaseModel):
    """Type text into the currently focused element without clicking first"""
    tool_name: Literal["write"]
    content: str = Field(description="Content to write")
    press_enter: bool = Field(default=False, description="Whether to press Enter after typing")

class AnswerArgs(BaseModel):
    """Provide a final answer"""
    tool_name: Literal["answer"]
    content: str = Field(description="The answer content")

class Step(BaseModel):
    note: str | None = Field(default=None, description="Task-relevant information from the previous observation. Empty if nothing new.")
    thought: str = Field(description="Reasoning about next steps")
    tool_call: ClickArgs | WriteArgs | AnswerArgs

In TypeScript, write the same JSON Schema by hand: a oneOf of tool objects, each with tool_name: { const: "..." }.

Send the schema twice: in extra_body={"structured_outputs": {"json": schema}}, and appended to the system prompt in an <output_format> block. The model was trained with both, so dropping either hurts reliability. Do not use tools or tool_choice: the action arrives in content, not in tool_calls.

Loop

Tool results go back as user messages wrapped in <tool_output tool="...">, not as tool messages.

Plug in your own SYSTEM_PROMPT, observation() (the <observation> message from Core concepts), and execute(tool_call).

Python

import json

schema = Step.model_json_schema()
system = SYSTEM_PROMPT + f"\n\n<output_format>\n```json\n{json.dumps(schema)}\n```\n</output_format>"

def run(task: str, max_steps: int = 20) -> str:
    messages = [{"role": "system", "content": system}, {"role": "user", "content": task}]
    for _ in range(max_steps):
        messages.append(observation())
        trim_to_last_n_images(messages, n=3)
        resp = client.chat.completions.create(
            model="holo3-122b-a10b",
            messages=messages,
            temperature=0.8,
            extra_body={"structured_outputs": {"json": schema}},
        )
        step = Step.model_validate_json(resp.choices[0].message.content)
        messages.append({"role": "assistant", "content": step.model_dump_json()})
        if step.tool_call.tool_name == "answer":
            return step.tool_call.content
        result = execute(step.tool_call)
        messages.append({"role": "user", "content": f'<tool_output tool="{step.tool_call.tool_name}">\n{result}\n</tool_output>'})
    return "Step budget exhausted"