Build an LLM agent

The pattern: manifest → context → loop.

Render the manifest as capabilities

Turn each granted action schema into a tool-like description the model can select from, and each channel into a described percept source. This mirrors how MCP clients render tools.

def render_capabilities(session):
    lines = ["You control embodiment '%s'. Available actions:" % session.embodiment]
    for a in session.granted_actions:
        lines.append(f"- {a.type}: params {a.params_schema}")
    return "\n".join(lines)

Loop: observe, decide, submit, tick, read

while not done:
    obs = await session.observe()                     # frame for the current tick
    decision = llm.decide(render_capabilities(session), summarize(obs))
    action = await session.submit(type=decision.type, params=decision.params)
    # `action` is admitted and staged; nothing happens until the world advances.
    await session.tick()                              # world.tick — delivers statuses + next frame
    status = await action.terminal()                  # resolves from statuses delivered by the tick
    feed_back(llm, status)                            # failures are information

In lockstep, await action.terminal() without a preceding tick() never resolves — the world does not advance on its own (AWP-TIM-002). Extended actions may need several ticks; loop on tick() until the status is terminal. In streaming mode drop the tick() call: the world advances on its own and statuses arrive as they happen.

Handle the lifecycle honestly

Feed failed, preempted, and clamped statuses back into context — they are the world teaching the model its limits. Never retry non-retryable errors with identical params (AWP-ERR-001).

Prefer subscribing to a scene-graph channel when the world offers one — structured entities and affordances beat raw pixels for LLM planning, and you can add pixels later for grounding checks.

For streaming worlds, run observation ingestion and decision-making concurrently, always stamping decisions with the seq/ts_mono_ns they were based on.

Get started

Core concepts

Guides

Compare