> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oumi.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# HARNESS PROTOCOL

> Routes, payloads, statuses, and error handling for a Bring Your Own Harness endpoint

This reference is for the team implementing your agent endpoint. It defines version `1` of the harness protocol, including the four routes Oumi calls during an evaluation. To connect an existing endpoint through the UI, see [Register a harness](/guides/harnesses/register).

> **[Download the OpenAPI spec](/openapi/harness-protocol.yaml)**
>
> OpenAPI 3.1 definition of the four routes, their request and response bodies, and the status object.

| Route | Method | Purpose |
| - | - | - |
| `submit` | `POST` | Start a session, or resume a paused turn |
| `get_status` | `GET` | Report a turn's status |
| `collect_output` | `GET` | Return messages generated by a turn |
| `cancel` | `POST` | Request cancellation of a turn |

Each route is appended to the endpoint URL's path. A harness registered at `https://agents.example.com/eval` receives `POST https://agents.example.com/eval/submit`.

***

## CONVENTIONS

* Requests and responses are JSON. Oumi sends its stored credential on every call, and `Content-Type: application/json` on `POST` requests. `GET` requests carry no body.
* Every successful response includes `protocol_version`, `session_id`, and `invocation_id` matching the submitted invocation.
* Oumi omits fields with null values. Responses must contain only the fields defined in this protocol.
* Responses must be uncompressed. Oumi sends `Accept-Encoding: identity`.
* Respond to each request within 30 seconds. Request and response bodies must not exceed 10 MiB.

***

## SESSIONS AND INVOCATIONS

Oumi creates one **session** per dataset row. Each session contains one **invocation**, which represents the agent's turn. A turn can pause for user input and resume multiple times before completing.

```text theme={null}
start ──> queued ──> running ──> completed
                        │
                        ├──> input_required ──resume──> queued ──> ...
                        │
                        └──> failed

cancel: any non-terminal status ──> cancelled
```

| Status | Terminal | Has output | Meaning |
| - | - | - | - |
| `queued` | No | No | Accepted, not started |
| `running` | No | No | The agent is working |
| `input_required` | No | Yes | The agent asked the user one or more questions and is waiting |
| `completed` | Yes | Yes | The agent finished its turn. The session's conversation is over. |
| `failed` | Yes | No | The turn could not finish |
| `cancelled` | Yes | No | The turn stopped after a cancellation request |

* A resume continues the same invocation, with the same `invocation_id`.
* Oumi does not send an end-of-session request. Your harness manages session retention and must preserve state while the row is active.
* Your harness manages its tools, memory, and working state. Oumi assembles the evaluation transcript from the opening messages, collected harness output, and simulated user replies.

***

## MESSAGES

Messages follow the OpenAI chat format.

| Field | Type | Description |
| - | - | - |
| `role` | string | `system`, `user`, `assistant`, or `tool` |
| `content` | string or null | Message text. Required unless `tool_calls` is set. |
| `tool_calls` | array | Assistant only. Each item has an `id`, `type: "function"`, and a `function` object containing `name` and `arguments`. The `arguments` value is a JSON string. |
| `tool_call_id` | string | Tool messages only. The `id` of the call this message answers. |

Throughout a session:

* Tool call IDs are non-empty and unique.
* Every tool call is answered by exactly one `tool` message before the next non-tool message.
* A turn may leave its final tool calls unanswered only when its status is `input_required`. These calls represent questions for the user.

***

## API SPEC

Oumi calls these four routes on your endpoint. Each section lists the request body, the expected response, and the status codes Oumi handles.

### SUBMIT

`POST {endpoint}/submit`

#### REQUEST BODY

| Field | Type | Description |
| - | - | - |
| `protocol_version` | string | `"1"` |
| `session_id` | string | Session ID, unique to the dataset row |
| `invocation_id` | string | Invocation ID, unchanged across retries and resumes. |
| `session_action` | string | `start` or `resume` |
| `input_messages` | array | At least one message. See below. |
| `model_connection` | object | The model to call for this turn. See [model connection](#model-connection). |

* **`start`:** the row's opening conversation, ending with a `user` message. Start a new session with it.
* **`resume`:** only `tool` messages, with one reply for each pending tool call. Resume the existing invocation with these replies.

#### RESPONSE

Respond with the invocation's status. The HTTP status depends on whether the turn is still in progress:

| HTTP status | When |
| - | - |
| `202 Accepted` | The status is `queued`, `running`, or `input_required` |
| `200 OK` | The status is `completed`, `failed`, or `cancelled` |

The response body is a **status object**. `get_status` and `cancel` return the same object.

| Field | Type | Description |
| - | - | - |
| `protocol_version` | string | `"1"` |
| `session_id` | string | Value from the request |
| `invocation_id` | string | Value from the request |
| `status` | string | One of `queued`, `running`, `input_required`, `completed`, `failed`, or `cancelled` |
| `error` | object | Required when `status` is `failed`, and omitted otherwise. See [get\_status](#get_status). |
| `error.message` | string | Non-empty explanation of the failure |

```json theme={null}
{
  "protocol_version": "1",
  "session_id": "ses_9b2f...",
  "invocation_id": "5d0c...",
  "status": "queued"
}
```

If a submission times out or its response is lost, Oumi retries with the same `invocation_id`, `session_action`, and `input_messages`, but a fresh `model_connection.api_key`. Deduplicate on `invocation_id`, not on the full body. Repeated submissions must be idempotent: return the current status without restarting the turn or applying the same resume twice.

***

### GET\_STATUS

`GET {endpoint}/get_status?invocation_id={invocation_id}`

#### REQUEST

No request body. The invocation ID is passed as the `invocation_id` query parameter.

#### RESPONSE

Return `200 OK` with the same [status object](#submit) as `submit`. When the status is `failed`, include an `error` object with a non-empty `message` string that explains the failure:

```json theme={null}
{
  "protocol_version": "1",
  "session_id": "ses_9b2f...",
  "invocation_id": "5d0c...",
  "status": "failed",
  "error": {"message": "The model endpoint returned HTTP 500 after 3 attempts."}
}
```

`error` must be present when the status is `failed`, and only then. Oumi logs the first 1,000 characters of the message for troubleshooting; it does not use the message to determine how to handle the failure. Do not include credentials, such as the model API key, in the message.

Return `404 Not Found` for an unknown invocation. Returning `404` for a submitted invocation fails the entire evaluation.

***

### COLLECT\_OUTPUT

`GET {endpoint}/collect_output?invocation_id={invocation_id}`

#### REQUEST

No request body. The invocation ID is passed as the `invocation_id` query parameter.

#### RESPONSE

When the status is `input_required` or `completed`, return `200 OK` with the messages generated since the most recent submission. Do not include earlier messages from the session.

| Field | Type | Description |
| - | - | - |
| `protocol_version`, `session_id`, `invocation_id` | string | Values from the submitted invocation |
| `status` | string | `completed` or `input_required` |
| `messages` | array | At least one `assistant` or `tool` message, ending with an `assistant` message |

For `input_required`, the final assistant message contains the `tool_calls` that require user input:

```json theme={null}
{
  "protocol_version": "1",
  "session_id": "ses_9b2f...",
  "invocation_id": "5d0c...",
  "status": "input_required",
  "messages": [
    {
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_1",
          "type": "function",
          "function": {"name": "ask_user", "arguments": "{\"question\": \"Which city?\"}"}
        }
      ]
    }
  ]
}
```

Oumi then resumes with a `tool` message whose `tool_call_id` is `call_1`. The tool name is up to your harness. Oumi answers every call left pending at the end of an `input_required` turn.

Return `409 Conflict` if output is not yet available. Oumi waits and retries.

***

### CANCEL

`POST {endpoint}/cancel`

#### REQUEST BODY

| Field | Type | Description |
| - | - | - |
| `protocol_version` | string | `"1"` |
| `session_id` | string | Session ID of the turn to cancel |
| `invocation_id` | string | Invocation ID of the turn to cancel |

```json theme={null}
{"protocol_version": "1", "session_id": "ses_9b2f...", "invocation_id": "5d0c..."}
```

#### RESPONSE

Return `200 OK` with the same status object as `submit`, reflecting the current status. If cancellation is still in progress, you can return a non-terminal status. Oumi polls `get_status` until the turn reports `cancelled`, `completed`, or `failed`. If the harness does not confirm a terminal status within the cancellation timeout, the evaluation fails.

Oumi requests cancellation when the evaluation is cancelled, the turn exceeds its 15-minute processing limit, or an error prevents the turn from continuing.

***

## MODEL CONNECTION

Each `submit` request includes a `model_connection` object:

| Field | Description |
| - | - |
| `base_url` | An OpenAI-compatible base URL, `https://inference.oumi.ai/v1` |
| `model` | An opaque model name for this evaluation. Use it as the `model` parameter in each request. |
| `api_key` | An API key scoped to this model and evaluation |
| `generation_settings` | Settings from the evaluation, such as `temperature`, `top_p`, `max_tokens`, and `seed`. Unset settings are omitted. |

```python theme={null}
from openai import OpenAI

connection = submit_request["model_connection"]
client = OpenAI(base_url=connection["base_url"], api_key=connection["api_key"])
reply = client.chat.completions.create(
    model=connection["model"],
    messages=transcript,
    **connection["generation_settings"],
)
```

* The key authorizes only `POST /v1/chat/completions` requests for the specified `model`.
* Use the connection from the latest submission. Each resume includes a new key.
* The key expires when the evaluation completes, is cancelled, or fails, or after 12 hours, whichever comes first.
* Don't store or log the key.

***

## ERRORS

Oumi handles HTTP responses and transport errors as follows:

| Response or error | Oumi behavior |
| - | - |
| `401`, `403`, or `404` | Fails the entire evaluation |
| `429`, any `5xx`, a timeout, or a dropped connection | Retries with backoff. If retries are exhausted, fails the row. |
| `409` on `submit` with no `invocation_id` in the body | Treats your harness as busy and retries |
| `409` on `submit` with an `invocation_id` in the body | Fails the row |
| `409` on `collect_output` | Waits and retries |
| Any other `4xx` or `3xx`, an invalid response body, or a protocol violation | Fails the row |
| A `failed` status | Fails the row |

Row failures do not stop other rows. Oumi scores completed rows and reports the number of failures. Oumi does not follow redirects.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.