Data centers

The one-line installer picks the package for your distribution. HIPS_DATA_CENTER=1 declares the machine a server before it is linked to your account (see The data-center flag, below).

00000000000000000000000000000000000000001curl -fsSL https://hips.com/install.sh | sudo HIPS_DATA_CENTER=1 sh

On a server with no screen, link it with sudo hips-provider connect, which shows a code, a QR code and a link to approve in the HIPS Console.

For many servers, create an enrollment code in the HIPS Console under Connect many machines. The install line takes it: curl -fsSL https://hips.com/install.sh | sudo HIPS_DATA_CENTER=1 HIPS_ENROLLMENT_CODE=<code> sh. Keep the code in a file only root can read, so it is not typed where the shell's history keeps it, and run this as root:

00000000000000000000000000000000000000001curl -fsSL https://hips.com/install.sh | HIPS_DATA_CENTER=1 HIPS_ENROLLMENT_CODE="$(cat /root/hips-enrollment-code)" sh

A server already installed is linked from the same file (mode 0600 or 0400, not a link):

00000000000000000000000000000000000000001sudo hips-provider connect --enrollment-code-file /root/hips-enrollment-code

• Each server joins your account with nobody approving it, and its GPUs go through the normal checks before they take work.
• Revoking a code stops new servers joining. Servers it already connected stay connected.
• You can start the installer on thousands of servers at once. If HIPS asks them to slow down, each server waits and sends its code again, so give your deployment tool's task a timeout of at least ten minutes. If the tool stops the installer early, the server goes on trying and links by itself.

Build an image with the installer, and run hips-provider service run as the container's command:

00000000000000000000000000000000000000001FROM ubuntu:24.042RUN apt-get update \3 && apt-get install -y --no-install-recommends ca-certificates curl \4 && curl -fsSL https://hips.com/install.sh | HIPS_NO_LINK=1 HIPS_DATA_CENTER=1 sh \5 && rm -rf /var/lib/apt/lists/*6CMD ["hips-provider", "service", "run"]

• The state volume keeps the device's identity, its link to your account, its settings and the model cache, so a new container is the same device with the same models. To keep the models on a larger disk, mount that disk at /var/lib/hips-provider/models.
• The configuration volume keeps /etc/hips-provider/config.toml.
• --gpus all passes the GPUs to the container and needs the NVIDIA Container Toolkit on the host.
• --restart unless-stopped brings the container back by itself.
• --init hands a stop on to HIPS Provider and reaps ended processes.

To link the container with an enrollment code, give it the code and HIPS Provider reads it when the container starts, while the container is not yet linked. We recommend a file, since docker inspect shows an environment variable: owned by root and readable by root alone (mode 0600 or 0400), not a link, mounted at /run/secrets/hips-enrollment-code, or at another path named by HIPS_ENROLLMENT_CODE_FILE. Or give the code as the environment variable HIPS_ENROLLMENT_CODE.

00000000000000000000000000000000000000001docker run -d --name hips-provider \2  -v /root/hips-enrollment-code:/run/secrets/hips-enrollment-code:ro \3  ...

A refused code is said in the container's log, and the container keeps running unlinked. On Kubernetes, secret volumes use links, so use the variable from a secretKeyRef, or a subPath mount with mode 0400.

Every command below runs inside the container with docker exec hips-provider hips-provider ....

A server with the data-center flag is available whenever it is on and healthy. HIPS Provider reads no sign of a person using it, applies no idle or sleep rules, and keeps no daily cap. What stops its work is a pause, a maintenance window you set, its thermal limit, or a driver fault.

Declare the flag before the server is linked: with the installer's HIPS_DATA_CENTER=1, or with sudo hips-provider config set data_center true. A Linux container with no systemd and no desktop declares it by itself. Once the server is linked, only HIPS changes the flag. hips-provider status shows it.

The options combine: hips-provider pause --gpu 1 --wait --timeout 10m --reason "client job 4711".

• A server's pause lasts until it is resumed, across reboots and restarts of its container. A pause with --for or --until still ends at its time.
• HIPS sends no new work. A job already running may finish within the drain window, 120 seconds unless drain_window_s says otherwise; one that has not finished by then is stopped and reported to HIPS.
• Pausing while paused changes the end, and the reason where a new one is given. Resuming while not paused says "Not paused.". Both exit 0, so a script can run them again safely.

1. 01

   Pause with --wait: hips-provider pause --wait --timeout 10m --reason "...", with --gpu N for one GPU. When it exits 0, no HIPS work runs and the GPU's memory is free.

2. 02

   Hand the GPU over and start your client's work.

3. 03

   Resume once your client is done: hips-provider resume, with --gpu N for one GPU.

Pause with --wait before a change and resume after. In Docker, put docker exec hips-provider in front of the program. A task that should not fail the play on a slow drain can accept 75 as well: the pause is then in force, with work still finishing.

00000000000000000000000000000000000000001- name: Take the GPUs back from HIPS2  ansible.builtin.command:3    argv: [hips-provider, pause, --wait, --timeout, 10m, --reason, "{{ hips_pause_reason }}"]4  register: hips_pause5  changed_when: true6  failed_when: hips_pause.rc != 07 8# Your change goes here.9 10- name: Give the GPUs back to HIPS11  ansible.builtin.command:12    argv: [hips-provider, resume]13  register: hips_pause_end14  changed_when: "'Not paused.' not in hips_pause_end.stdout"

A prolog pauses with --wait, and an epilog resumes. A prolog that exits with anything but 0 sets the node to DRAIN, so give --timeout room for the drain window. On nodes that run several Slurm jobs at once, pause the job's own GPUs with --gpu, after checking that the indices hips-provider gpus lists match your gres.conf.

00000000000000000000000000000000000000001#!/bin/sh2# Prolog: take this node's GPUs from HIPS for the Slurm job.3exec hips-provider pause --wait --timeout 5m --reason "Slurm job ${SLURM_JOB_ID}"

When HIPS Provider runs on a node beside your cluster, the node moves between the two with a pause and a cordon: the pause before the uncordon when the cluster takes the node, and the resume after the cordon when HIPS takes it back. Where HIPS Provider runs as a pod on the node, run the same commands with kubectl exec into that pod, and set its terminationGracePeriodSeconds to the value of HIPS_STOP_TIMEOUT.

00000000000000000000000000000000000000001ssh "$NODE" sudo hips-provider pause --wait --timeout 10m --reason "cluster workloads"2kubectl uncordon "$NODE"

hips-provider status --json prints one JSON object for each server's state, its GPUs, any pause with its reason, and the work running.

For example, whether a server is paused and drained:

00000000000000000000000000000000000000001hips-provider status --json | jq '{state, pause: .status.pause}'