Prompt caching
HIPS keeps no prompt cache yet: every GPU engine that serves a request runs with its prompt cache off, so each request reads its whole prompt, every answer reports cached_tokens: 0, and every input token is billed at the model's input rate. There is nothing to switch on, and writing a cache is never charged.
| Token | Rate |
|---|---|
| Input | The model's input rate, for every input token |
| Cached input | None: no prefix is reused, so no request pays a cached-input rate |
| Cache write | No charge. cache_write_tokens is reported as 0 |
| Output | The model's output rate |
| In a batch | The batch rates, every input token at the batch input rate |
- Token
- Input
- Rate
- The model's input rate, for every input token
- Token
- Cached input
- Rate
- None: no prefix is reused, so no request pays a cached-input rate
- Token
- Cache write
- Rate
- No charge.
cache_write_tokensis reported as 0
- Token
- Output
- Rate
- The model's output rate
- Token
- In a batch
- Rate
- The batch rates, every input token at the batch input rate
Parts list: Usage fields
usage.prompt_tokensEvery input token, as OpenAI counts them.
usage.prompt_tokens_details.cached_tokens0on every answer today: no prefix is reused yet.cache_write_tokensAlways 0: writing a cache is not charged.
x-hips-cost-microsThe request's cost, every input token at the input rate; on a stream, the
hipsobject of the final usage chunk.
The Messages API reports the same counts as cache_read_input_tokens and cache_creation_input_tokens, and the Responses API as input_tokens_details.cached_tokens: each is 0 today. The Console's Usage and activity view counts cached input apart from the rest of the input, and it reads 0.
prompt_cache_key, a string of at most 512 characters: accepted, and never logged or stored. It does nothing more yet.cache_salt: refused with 400unknown_parameter. What keeps one workspace's cache apart from another's is HIPS's to derive, never the client's.prompt_cache_options,prompt_cache_breakpointandprompt_cache_retention: refused with 400unsupported_parameter, since HIPS promises no cache lifetime and takes no breakpoint.- On the Messages API,
cache_control{"type": "ephemeral"}where Anthropic places it: accepted, and it changes nothing. Attlis a 400.
How prompt caching will work is decided, for both pools, and it is not built yet:
- It will run on every GPU that serves a language model: providers' devices in the Community pool and HIPS's own servers in the Certified pool.
- A workspace's prompt prefix will be kept only in the memory of the engine on the GPU that ran the request. HIPS's software never writes it to a disk, a database or a log, and no answer is ever cached.
- It will be reused only by the same workspace, never by another workspace's request on any GPU, and
prompt_cache_keynarrows it further within the workspace. - A cache nobody has used for 30 minutes will be dropped, and nothing will reuse it after that. It will also end at once when the workspace is deleted. A busy GPU may overwrite a cache sooner, and no lifetime is promised.
- It will come first for speed, and change no price: until HIPS can verify each hit and a price for cached input is decided, every input token stays billed at the input rate.
- Every input token counts toward your tokens-per-minute limit.
- No cache lifetime is promised.
Related sheets
- Sheet
- 08 of 20
- Updated
- 2026-11-16