Prompt caching
Prompt caching is automatic. When a request starts with the same tokens as an earlier request in your workspace, the work on that shared prefix can be reused, and those input tokens are billed at the cached-input rate. There is nothing to switch on and no charge to write the cache.
Note
Caching is decided and not yet served. Each model's cached-input rate is set per model in the catalog.
| Token | Rate |
|---|---|
| Input, uncached | The model's input rate |
| Input, cached | Its own rate per model, pool and pricing mode: by default 10% of the input rate |
| Cache write | No charge. cache_write_tokens is reported as 0 |
| Output | The model's output rate |
| In a batch | The batch discount and the cached rate multiply: at the defaults, a cached input token in a batch costs 5% of the on-demand input rate |
- Token
- Input, uncached
- Rate
- The model's input rate
- Token
- Input, cached
- Rate
- Its own rate per model, pool and pricing mode: by default 10% of the input rate
- Token
- Cache write
- Rate
- No charge.
cache_write_tokensis reported as 0
- Token
- Output
- Rate
- The model's output rate
- Token
- In a batch
- Rate
- The batch discount and the cached rate multiply: at the defaults, a cached input token in a batch costs 5% of the on-demand input rate
HIPS bills only cache hits it can verify: in the approved pool, the lower of the engine's count and the gateway's own record of your workspace's prefixes.
Parts list: Usage fields
usage.prompt_tokensEvery input token, cached ones included, as OpenAI counts them.
usage.prompt_tokens_details.cached_tokensThe cached input tokens billed at the cached rate.
cache_write_tokensAlways 0: writing the cache is not charged.
x-hips-cost-microsThe request's cost with the cached rate applied; on a stream, the
hipsobject of the final usage chunk.
The Console's Usage and activity view splits uncached input, cached input and output per request and per batch.
- Put what stays the same first: the system prompt, tool definitions, long documents and examples.
- Put what changes last: the user's question, the current date, a request id.
- Keep the shared part byte for byte identical between requests.
- Send
prompt_cache_keyto group requests that share a prefix: it narrows the cache within your workspace and steers them to a worker that holds it.
A cache never crosses workspaces. Each workspace's cache is kept apart, narrowed further by prompt_cache_key, with a salt the gateway derives and sends itself, so a client's cache_salt is refused with a 400. Placement prefers a worker that holds your prefix, bounded by load and never over a requirement you set.
| Pool | At launch |
|---|---|
| Approved data-center pool | Cached, and billed at the cached rate |
| Personal GPUs | No cross-request cache, and no cached tokens billed: a device engine cannot yet keep one workspace's cache apart from another's. A device cache comes later, in memory only and never on disk |
- Pool
- Approved data-center pool
- At launch
- Cached, and billed at the cached rate
- Pool
- Personal GPUs
- At launch
- No cross-request cache, and no cached tokens billed: a device engine cannot yet keep one workspace's cache apart from another's. A device cache comes later, in memory only and never on disk
- Cached tokens count toward your tokens-per-minute limit.
- No cache lifetime is promised at launch: a prefix may be evicted at any time, and the next request simply pays the uncached rate for it.
Related sheets
- Sheet
- 08 of 13
- Updated
- 2026-10-21
- Status
- Preview