Streaming
stream: true on POST /v1/chat/completions answers 200 with Content-Type: text/event-stream, one data: line per chunk in OpenAI's chat.completion.chunk shape (choices[].delta with role, content, tool_calls[] by index; finish_reason on the last content chunk), then data: [DONE].
| Rule | Behavior |
|---|---|
| Usage | With stream_options.include_usage: true, one final chunk before [DONE] with choices: [] and usage. Without it, no usage chunk. |
The hips object | The headers that need the finished answer (x-hips-cost-micros, x-hips-latency-ms) cannot be sent at the head of a stream. They are omitted from the head, and the final usage chunk carries a top-level hips object: {"cost_micros", "latency_ms", "quote"}. It is sent when include_usage is true; whether it is sent without it is open Not decided yet. The other x-hips-* headers are in the head as usual. |
| Keep-alive | A comment line (: keep-alive) at an interval Not decided yet while the engine produces nothing. |
| A failure after the stream started | One data: line holding the error envelope, then the stream closes without [DONE]. OpenAI clients raise on it. The usage record shows the outcome. |
| Client disconnect | Closing the connection cancels the job on the worker. The charge for tokens already delivered is Not decided yet (proposed: charged). |
| Relay | The gateway relays chunks as the worker emits them and buffers nothing beyond one chunk. |
- Rule
- Usage
- Behavior
- With
stream_options.include_usage: true, one final chunk before[DONE]withchoices: []andusage. Without it, no usage chunk.
- Rule
- The
hipsobject - Behavior
- The headers that need the finished answer (
x-hips-cost-micros,x-hips-latency-ms) cannot be sent at the head of a stream. They are omitted from the head, and the final usage chunk carries a top-levelhipsobject:{"cost_micros", "latency_ms", "quote"}. It is sent wheninclude_usageis true; whether it is sent without it is open Not decided yet. The otherx-hips-*headers are in the head as usual.
- Rule
- Keep-alive
- Behavior
- A comment line (
: keep-alive) at an interval Not decided yet while the engine produces nothing.
- Rule
- A failure after the stream started
- Behavior
- One
data:line holding the error envelope, then the stream closes without[DONE]. OpenAI clients raise on it. The usage record shows the outcome.
- Rule
- Client disconnect
- Behavior
- Closing the connection cancels the job on the worker. The charge for tokens already delivered is Not decided yet (proposed: charged).
- Rule
- Relay
- Behavior
- The gateway relays chunks as the worker emits them and buffers nothing beyond one chunk.
Note
The cost of a streamed answer arrives in the hips object of the final usage chunk, not in x-hips-cost-micros: the cost is not known when the head is sent, and trailers do not pass the edge. Ask for stream_options.include_usage to receive it.
Close the connection. The job is canceled on the worker; an attempt interrupted after the first streamed token is never restarted transparently, because a stream cannot be resumed. A tool call streamed as delta.tool_calls by index is complete only when its finish_reason arrives.
Long-running work goes through jobs: submit, poll GET /v1/jobs/{id}, cancel with POST /v1/jobs/{id}/cancel, retrieve the result until it expires. A completion webhook is open Not decided yet.
Related sheets
- Sheet
- 12 of 13
- Detail
- E
- Updated
- 2026-10-21
- Status
- Preview