Streaming
stream: true on POST /v1/chat/completions answers 200 with Content-Type: text/event-stream, one data: line per chunk in OpenAI's chat.completion.chunk shape (choices[].delta with role, content, tool_calls[] by index; finish_reason on the last content chunk), then data: [DONE].
| Rule | Behavior |
|---|---|
| Usage | With stream_options.include_usage: true, one final chunk before [DONE] with choices: [] and usage. Without it, no usage chunk. |
The hips object | The headers that need the finished answer (x-hips-cost-micros, x-hips-latency-ms) cannot be sent at the head of a stream. They are omitted from the head, and the final usage chunk carries a top-level hips object: {"cost_micros", "latency_ms", "quote"}, then first_token_ms (from the request's receipt to the answer's first token) and gpu ({"model", "vendor", "backend"}) where they are known. gpu.model is the model of the GPU that served the answer, as the serving device reported it for that GPU, and is null only where the device reported no GPU product name; its vendor and backend are still named then. A non-streamed answer carries the same object as its last member, without first_token_ms: the engine answers it whole, so a client that wants the time to the first token streams. It is sent when include_usage is true; whether it is sent without it is open Not decided yet. The other x-hips-* headers are in the head as usual. |
| Keep-alive | A comment line (: keep-alive) at an interval Not decided yet while the engine produces nothing. |
| A failure after the stream started | One data: line holding the error envelope, then the stream closes without [DONE]. OpenAI clients raise on it. The usage record shows the outcome. |
| Client disconnect | Closing the connection cancels the job on the worker. The charge for tokens already delivered is Not decided yet (proposed: charged). |
| Relay | The gateway relays chunks as the worker emits them and buffers nothing beyond one chunk. |
- Rule
- Usage
- Behavior
- With
stream_options.include_usage: true, one final chunk before[DONE]withchoices: []andusage. Without it, no usage chunk.
- Rule
- The
hipsobject - Behavior
- The headers that need the finished answer (
x-hips-cost-micros,x-hips-latency-ms) cannot be sent at the head of a stream. They are omitted from the head, and the final usage chunk carries a top-levelhipsobject:{"cost_micros", "latency_ms", "quote"}, thenfirst_token_ms(from the request's receipt to the answer's first token) andgpu({"model", "vendor", "backend"}) where they are known.gpu.modelis the model of the GPU that served the answer, as the serving device reported it for that GPU, and isnullonly where the device reported no GPU product name; its vendor and backend are still named then. A non-streamed answer carries the same object as its last member, withoutfirst_token_ms: the engine answers it whole, so a client that wants the time to the first token streams. It is sent wheninclude_usageis true; whether it is sent without it is open Not decided yet. The otherx-hips-*headers are in the head as usual.
- Rule
- Keep-alive
- Behavior
- A comment line (
: keep-alive) at an interval Not decided yet while the engine produces nothing.
- Rule
- A failure after the stream started
- Behavior
- One
data:line holding the error envelope, then the stream closes without[DONE]. OpenAI clients raise on it. The usage record shows the outcome.
- Rule
- Client disconnect
- Behavior
- Closing the connection cancels the job on the worker. The charge for tokens already delivered is Not decided yet (proposed: charged).
- Rule
- Relay
- Behavior
- The gateway relays chunks as the worker emits them and buffers nothing beyond one chunk.
Note
The cost of a streamed answer arrives in the hips object of the final usage chunk, not in x-hips-cost-micros: the cost is not known when the head is sent, and trailers do not pass the edge. Ask for stream_options.include_usage to receive it.
Close the connection. The job is canceled on the worker; an attempt interrupted after the first streamed token is never restarted transparently, because a stream cannot be resumed. A tool call streamed as delta.tool_calls by index is complete only when its finish_reason arrives.
Long-running work goes through jobs: submit, poll GET /v1/jobs/{id}, cancel with POST /v1/jobs/{id}/cancel, retrieve the result until it expires. A completion webhook is open Not decided yet.
Related sheets
- Sheet
- 19 of 20
- Detail
- E
- Updated
- 2026-11-09