HARNESS PROTOCOL

This reference is for the team implementing your agent endpoint. It defines version 1 of the harness protocol, including the four routes Oumi calls during an evaluation. To connect an existing endpoint through the UI, see Register a harness.

Download the OpenAPI spec

OpenAPI 3.1 definition of the four routes, their request and response bodies, and the status object.

Route	Method	Purpose
`submit`	`POST`	Start a session, or resume a paused turn
`get_status`	`GET`	Report a turn’s status
`collect_output`	`GET`	Return messages generated by a turn
`cancel`	`POST`	Request cancellation of a turn

Each route is appended to the endpoint URL’s path. A harness registered at https://agents.example.com/eval receives POST https://agents.example.com/eval/submit.

CONVENTIONS

Requests and responses are JSON. Oumi sends its stored credential on every call, and Content-Type: application/json on POST requests. GET requests carry no body.
Every successful response includes protocol_version, session_id, and invocation_id matching the submitted invocation.
Oumi omits fields with null values. Responses must contain only the fields defined in this protocol.
Responses must be uncompressed. Oumi sends Accept-Encoding: identity.
Respond to each request within 30 seconds. Request and response bodies must not exceed 10 MiB.

SESSIONS AND INVOCATIONS

Oumi creates one session per dataset row. Each session contains one invocation, which represents the agent’s turn. A turn can pause for user input and resume multiple times before completing.

start ──> queued ──> running ──> completed
                        │
                        ├──> input_required ──resume──> queued ──> ...
                        │
                        └──> failed

cancel: any non-terminal status ──> cancelled

Status	Terminal	Has output	Meaning
`queued`	No	No	Accepted, not started
`running`	No	No	The agent is working
`input_required`	No	Yes	The agent asked the user one or more questions and is waiting
`completed`	Yes	Yes	The agent finished its turn. The session’s conversation is over.
`failed`	Yes	No	The turn could not finish
`cancelled`	Yes	No	The turn stopped after a cancellation request

A resume continues the same invocation, with the same invocation_id.
Oumi does not send an end-of-session request. Your harness manages session retention and must preserve state while the row is active.
Your harness manages its tools, memory, and working state. Oumi assembles the evaluation transcript from the opening messages, collected harness output, and simulated user replies.

MESSAGES

Messages follow the OpenAI chat format.

Field	Type	Description
`role`	string	`system`, `user`, `assistant`, or `tool`
`content`	string or null	Message text. Required unless `tool_calls` is set.
`tool_calls`	array	Assistant only. Each item has an `id`, `type: "function"`, and a `function` object containing `name` and `arguments`. The `arguments` value is a JSON string.
`tool_call_id`	string	Tool messages only. The `id` of the call this message answers.

Throughout a session:

Tool call IDs are non-empty and unique.
Every tool call is answered by exactly one tool message before the next non-tool message.
A turn may leave its final tool calls unanswered only when its status is input_required. These calls represent questions for the user.

API SPEC

Oumi calls these four routes on your endpoint. Each section lists the request body, the expected response, and the status codes Oumi handles.

SUBMIT

POST {endpoint}/submit

REQUEST BODY

Field	Type	Description
`protocol_version`	string	`"1"`
`session_id`	string	Session ID, unique to the dataset row
`invocation_id`	string	Invocation ID, unchanged across retries and resumes.
`session_action`	string	`start` or `resume`
`input_messages`	array	At least one message. See below.
`model_connection`	object	The model to call for this turn. See model connection.

start: the row’s opening conversation, ending with a user message. Start a new session with it.
resume: only tool messages, with one reply for each pending tool call. Resume the existing invocation with these replies.

RESPONSE

Respond with the invocation’s status. The HTTP status depends on whether the turn is still in progress:

HTTP status	When
`202 Accepted`	The status is `queued`, `running`, or `input_required`
`200 OK`	The status is `completed`, `failed`, or `cancelled`

The response body is a status object. get_status and cancel return the same object.

Field	Type	Description
`protocol_version`	string	`"1"`
`session_id`	string	Value from the request
`invocation_id`	string	Value from the request
`status`	string	One of `queued`, `running`, `input_required`, `completed`, `failed`, or `cancelled`
`error`	object	Required when `status` is `failed`, and omitted otherwise. See get_status.
`error.message`	string	Non-empty explanation of the failure

{
  "protocol_version": "1",
  "session_id": "ses_9b2f...",
  "invocation_id": "5d0c...",
  "status": "queued"
}

If a submission times out or its response is lost, Oumi retries with the same invocation_id, session_action, and input_messages, but a fresh model_connection.api_key. Deduplicate on invocation_id, not on the full body. Repeated submissions must be idempotent: return the current status without restarting the turn or applying the same resume twice.

GET_STATUS

GET {endpoint}/get_status?invocation_id={invocation_id}

REQUEST

No request body. The invocation ID is passed as the invocation_id query parameter.

RESPONSE

Return 200 OK with the same status object as submit. When the status is failed, include an error object with a non-empty message string that explains the failure:

{
  "protocol_version": "1",
  "session_id": "ses_9b2f...",
  "invocation_id": "5d0c...",
  "status": "failed",
  "error": {"message": "The model endpoint returned HTTP 500 after 3 attempts."}
}

error must be present when the status is failed, and only then. Oumi logs the first 1,000 characters of the message for troubleshooting; it does not use the message to determine how to handle the failure. Do not include credentials, such as the model API key, in the message. Return 404 Not Found for an unknown invocation. Returning 404 for a submitted invocation fails the entire evaluation.

COLLECT_OUTPUT

GET {endpoint}/collect_output?invocation_id={invocation_id}

REQUEST

No request body. The invocation ID is passed as the invocation_id query parameter.

RESPONSE

When the status is input_required or completed, return 200 OK with the messages generated since the most recent submission. Do not include earlier messages from the session.

Field	Type	Description
`protocol_version`, `session_id`, `invocation_id`	string	Values from the submitted invocation
`status`	string	`completed` or `input_required`
`messages`	array	At least one `assistant` or `tool` message, ending with an `assistant` message

For input_required, the final assistant message contains the tool_calls that require user input:

{
  "protocol_version": "1",
  "session_id": "ses_9b2f...",
  "invocation_id": "5d0c...",
  "status": "input_required",
  "messages": [
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {"name": "ask_user", "arguments": "{\"question\": \"Which city?\"}"}
        }
      ]
    }
  ]
}

Oumi then resumes with a tool message whose tool_call_id is call_1. The tool name is up to your harness. Oumi answers every call left pending at the end of an input_required turn. Return 409 Conflict if output is not yet available. Oumi waits and retries.

CANCEL

POST {endpoint}/cancel

REQUEST BODY

Field	Type	Description
`protocol_version`	string	`"1"`
`session_id`	string	Session ID of the turn to cancel
`invocation_id`	string	Invocation ID of the turn to cancel

{"protocol_version": "1", "session_id": "ses_9b2f...", "invocation_id": "5d0c..."}

RESPONSE

Return 200 OK with the same status object as submit, reflecting the current status. If cancellation is still in progress, you can return a non-terminal status. Oumi polls get_status until the turn reports cancelled, completed, or failed. If the harness does not confirm a terminal status within the cancellation timeout, the evaluation fails. Oumi requests cancellation when the evaluation is cancelled, the turn exceeds its 15-minute processing limit, or an error prevents the turn from continuing.

MODEL CONNECTION

Each submit request includes a model_connection object:

Field	Description
`base_url`	An OpenAI-compatible base URL, `https://inference.oumi.ai/v1`
`model`	An opaque model name for this evaluation. Use it as the `model` parameter in each request.
`api_key`	An API key scoped to this model and evaluation
`generation_settings`	Settings from the evaluation, such as `temperature`, `top_p`, `max_tokens`, and `seed`. Unset settings are omitted.

from openai import OpenAI

connection = submit_request["model_connection"]
client = OpenAI(base_url=connection["base_url"], api_key=connection["api_key"])
reply = client.chat.completions.create(
    model=connection["model"],
    messages=transcript,
    **connection["generation_settings"],
)

The key authorizes only POST /v1/chat/completions requests for the specified model.
Use the connection from the latest submission. Each resume includes a new key.
The key expires when the evaluation completes, is cancelled, or fails, or after 12 hours, whichever comes first.
Don’t store or log the key.

ERRORS

Oumi handles HTTP responses and transport errors as follows:

Response or error	Oumi behavior
`401`, `403`, or `404`	Fails the entire evaluation
`429`, any `5xx`, a timeout, or a dropped connection	Retries with backoff. If retries are exhausted, fails the row.
`409` on `submit` with no `invocation_id` in the body	Treats your harness as busy and retries
`409` on `submit` with an `invocation_id` in the body	Fails the row
`409` on `collect_output`	Waits and retries
Any other `4xx` or `3xx`, an invalid response body, or a protocol violation	Fails the row
A `failed` status	Fails the row

Row failures do not stop other rows. Oumi scores completed rows and reports the number of failures. Oumi does not follow redirects.

Getting started

Oumi workflow

HARNESS PROTOCOL

Download the OpenAPI spec

CONVENTIONS

SESSIONS AND INVOCATIONS

MESSAGES

API SPEC

SUBMIT

REQUEST BODY

RESPONSE

GET_STATUS

REQUEST

RESPONSE

COLLECT_OUTPUT

REQUEST

RESPONSE

CANCEL

REQUEST BODY

RESPONSE

MODEL CONNECTION

ERRORS

Download the OpenAPI spec

​CONVENTIONS

​SESSIONS AND INVOCATIONS

​MESSAGES

​API SPEC

​SUBMIT

​REQUEST BODY

​RESPONSE

​GET_STATUS

​REQUEST

​RESPONSE

​COLLECT_OUTPUT

​REQUEST

​RESPONSE

​CANCEL

​REQUEST BODY

​RESPONSE

​MODEL CONNECTION

​ERRORS

CONVENTIONS

SESSIONS AND INVOCATIONS

MESSAGES

API SPEC

SUBMIT

REQUEST BODY

RESPONSE

GET_STATUS

REQUEST

RESPONSE

COLLECT_OUTPUT

REQUEST

RESPONSE

CANCEL

REQUEST BODY

RESPONSE

MODEL CONNECTION

ERRORS