Batch jobs
Send many requests at once and collect the results when they are done. HIPS runs batch lines when GPU capacity is free, at 50% less than the on-demand price of the same model, and tells you when the batch is finished. The API is designed to follow OpenAI's Batch API, so that the official SDKs and the batch scripts you already have work with a HIPS key, base URL and model id; the compatibility matrix records each case once it is validated.
Note
Batches are built and not offered in production yet: POST /v1/files and POST /v1/batches answer 404 route_not_available until they are switched on. The queue of enqueued tokens per model and workspace is set per plan, and its values are still open Not decided yet.
Step 1: Write a JSONL file, one request per line, each with your own
custom_id.Step 2: Upload it with
POST /v1/filesand the purposebatch.Step 3: Create the batch with
POST /v1/batches, naming the endpoint and the completion window. The price is quoted and bound here.Step 4: HIPS runs the lines when capacity is free. Live requests always come first.
Step 5: When the batch ends, a signed webhook and a mail tell you; download the output and error files.
import jsonimport osfrom openai import OpenAI client = OpenAI( api_key=os.environ["HIPS_API_KEY"], base_url=os.environ["HIPS_BASE_URL"], # https://ai.hips.com/v1) # One request per line, each with your own custom_id.with open("requests.jsonl", "w") as f: for i, text in enumerate(["First document.", "Second document."]): f.write(json.dumps({ "custom_id": f"doc-{i}", "method": "POST", "url": "/v1/chat/completions", "body": { "model": os.environ["HIPS_MODEL"], "messages": [{"role": "user", "content": f"Summarize: {text}"}], }, }) + "\n") batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")batch = client.batches.create( input_file_id=batch_file.id, endpoint="/v1/chat/completions", completion_window="24h",)print(batch.id, batch.status)The output file holds one line per finished request, matched to your input by custom_id, in no particular order. Lines that failed, and lines still unfinished when the window closed, come back in the error file.
Python
batch = client.batches.retrieve(batch.id) if batch.status == "completed": for line in client.files.content(batch.output_file_id).text.splitlines(): record = json.loads(line) print(record["custom_id"], record["response"]["status_code"]) # Lines that failed, or expired when the window closed, are in the error file. if batch.error_file_id: print(client.files.content(batch.error_file_id).text)On https://ai.hips.com, in OpenAI's Batch API shape.
/v1/filesnot offeredUpload the JSONL input file with the purpose batch.
/v1/batchesnot offeredCreate a batch from an input file: input_file_id, endpoint, completion_window.
/v1/batches/{id}not offeredA batch with its status, its line counts and its output and error file ids.
/v1/batches/{id}/cancelnot offeredCancel a batch. Finished lines are delivered and charged; the rest are not.
/v1/batchesnot offeredThe workspace's batches, newest first.
/v1/files/{id}/contentnot offeredDownload an output or error file.
/v1/chat/completions, for language models: the oneendpointa batch takes as built. Any other is a 400unsupported_value.- Embeddings and the HIPS task API come later, as lines OpenAI's Batch API does not offer.
- One model per input file; your own model's lines follow the pools it may run in.
- A line may not name a router: a router chooses per request, and a batch runs one model. The file is refused with 400
invalid_file_formatnaming the first such line.
A batch in validating that finds any of these ends failed, each finding in errors.data with its code, message, param and line; nothing is held, run or charged. message is one fixed sentence per code, param names a field only where it is a field the request shape knows (for an unknown field, its parent), and nothing of a line's text is quoted.
| Code | When |
|---|---|
missing_required_parameter | custom_id, method, url or body absent, or a field the route requires |
duplicate_custom_id | A custom_id used on an earlier line |
invalid_value, invalid_type, unknown_parameter, unsupported_parameter, unsupported_value | A body that breaks the route's rules, a method other than POST, a streamed line |
context_length_exceeded, max_tokens_exceeded | A limit the body alone decides |
token_limit_exceeded | The batch's input tokens would take the model's queue for the workspace past its plan's value |
insufficient_credit | The prepaid balance cannot carry the held maximum |
budget_exceeded | A budget guardrail of the workspace or the credential cannot carry the batch's maximum charge; the message names the guardrail and when it resets |
- Code
missing_required_parameter- When
custom_id,method,urlorbodyabsent, or a field the route requires
- Code
duplicate_custom_id- When
- A
custom_idused on an earlier line
- Code
invalid_value,invalid_type,unknown_parameter,unsupported_parameter,unsupported_value- When
- A body that breaks the route's rules, a
methodother thanPOST, a streamed line
- Code
context_length_exceeded,max_tokens_exceeded- When
- A limit the body alone decides
- Code
token_limit_exceeded- When
- The batch's input tokens would take the model's queue for the workspace past its plan's value
- Code
insufficient_credit- When
- The prepaid balance cannot carry the held maximum
- Code
budget_exceeded- When
- A budget guardrail of the workspace or the credential cannot carry the batch's maximum charge; the message names the guardrail and when it resets
A line that never ran is in the error file with response: null and error: batch_expired when the window closed first, batch_cancelled when the batch was cancelled, attempt_failed when every attempt failed for a reason that is not yours. None of them is charged.
You choose how long HIPS may take: 24h by default, or any whole number of hours or days up to 7d. The price is the same for every window. No finish time is promised inside it: everything that finishes is delivered, and lines still unfinished when the window closes come back as expired and are not charged.
| Batch | |
|---|---|
| Discount | 50% less than the on-demand rate of the same model, pool and unit, applied to every rate a line uses: input, cached input, output and every task unit |
| Quote | Bound when POST /v1/batches accepts the batch, and held for every line through the window |
| Prompt caching | No line reuses a cached prompt prefix yet, so every input token is billed at the batch input rate (Prompt caching) |
| Balance | The batch's maximum charge (input at batch rates plus each line's output cap) is held against your prepaid balance at creation and released line by line as lines finish or expire |
| Expired lines | Not charged |
| A model without batch | A model whose price cannot carry the discount is sold without batch rather than at a smaller discount, and the catalog says so |
- Discount
- Batch
- 50% less than the on-demand rate of the same model, pool and unit, applied to every rate a line uses: input, cached input, output and every task unit
- Quote
- Batch
- Bound when
POST /v1/batchesaccepts the batch, and held for every line through the window
- Prompt caching
- Batch
- No line reuses a cached prompt prefix yet, so every input token is billed at the batch input rate (Prompt caching)
- Balance
- Batch
- The batch's maximum charge (input at batch rates plus each line's output cap) is held against your prepaid balance at creation and released line by line as lines finish or expire
- Expired lines
- Batch
- Not charged
- A model without batch
- Batch
- A model whose price cannot carry the discount is sold without batch rather than at a smaller discount, and the catalog says so
Actual prices are set per model in the catalog; the Pricing page shows the shape.
The limits match OpenAI's
| Limit | Value |
|---|---|
| Lines per input file | 50,000 |
| Input file size | 200 MB |
| Models per input file | One |
| Embedding inputs per batch | 50,000 |
| Batches created per hour | 2,000 |
| Open batches per workspace | 100 |
| Enqueued input tokens per model and workspace | Set per plan Not decided yet |
- Limit
- Lines per input file
- Value
- 50,000
- Limit
- Input file size
- Value
- 200 MB
- Limit
- Models per input file
- Value
- One
- Limit
- Embedding inputs per batch
- Value
- 50,000
- Limit
- Batches created per hour
- Value
- 2,000
- Limit
- Open batches per workspace
- Value
- 100
- Limit
- Enqueued input tokens per model and workspace
- Value
- Set per plan Not decided yet
- On every eligible idle GPU: Community pool devices the moment they come online, and spare capacity in the Certified pool.
- Live requests always come first. A device finishes the batch line it is running before it takes a live request.
- No line reuses another line's prompt prefix yet: each line is billed for every input token it sends.
- A batch line on your own model runs only where the model may run: the Certified pool, at launch.
A batch's input file is deleted once the batch ends (completed, failed, expired or cancelled), within about five minutes, unless another open batch reads it; to run it again, upload it again. Output and error files are kept 30 days and then deleted, sooner when you set a shorter expiry (output_expires_after) or delete them.
The Console's Batches screens follow every batch of the workspace: the list, and for each batch its timeline through the statuses, its usage and money (cached input counted apart, expired lines neither charged), its files as metadata with a short-lived download link to the workspace's storage, and a failed validation's findings, the line and the parameter, never a line's content. Cancel the batch stops it while it runs; the lines already finished stay delivered and charged.
Webhooks, beside them, holds the workspace's endpoints: add one with its URL and the events it receives, enable or disable it, send it a test once it is enabled, rotate its signing secret (with a recent sign-in), and read its deliveries, sending one again or every failed one.
Note
Batches are not offered yet: until they are, the batch routes answer 404 route_not_available, and the screens have no batch to show.
Related sheets
- Sheet
- 07 of 20
- Updated
- 2026-11-14