Slurm installation: production-ready Slurm on customer clusters in 48 hours
Slurm on Kubernetes keeps batch scheduling in Slurm while Kubernetes manages the services behind it. The queue, resource allocations, and job commands remain familiar; the queue controller and job-launching workers run in containers managed through Kubernetes. We use this arrangement on an existing Kubernetes foundation. It changes how a Slurm installation is deployed and maintained, while leaving identities, job-data access, and resource controls as explicit cluster-design responsibilities.
How we deploy Slurm on Kubernetes
Our clusterdOS cluster engine is Kubernetes-based. It is the open-source engine inside each cluster, extending to virtual machine and Slurm environments. Aranya OS, our proprietary multicluster operating system, builds on clusterdOS to federate clusters into one fleet.
For Slurm, we use a Helm chart, a package of Kubernetes deployment templates, that includes the Slinky Slurm operator. An operator is a controller that continually reconciles declared configuration with the services it manages. Slurm keeps responsibility for its batch queue and job allocations; Kubernetes manages the lifecycle of the services that provide them.
The chart supplies the Slurm service layer. Shared job-data storage, user mapping, accounting where required, and workload acceptance remain part of the customer-cluster design and handover. They are separate from the services the chart provisions automatically.
Why we run Slurm on Kubernetes
Our GPU machines are already Kubernetes nodes. The NVIDIA GPU Operator manages GPU software components, including drivers and the device plugin that exposes GPUs to Kubernetes. Node Feature Discovery, or NFD, identifies hardware capabilities; the GPU Operator uses those features to label GPU-present nodes.
Putting Slurm's services on that foundation avoids introducing a separate, statically provisioned Slurm host fleet with a second service-management path. We can declare their desired state alongside the rest of the cluster and use the same deployment workflow. This is our design choice for this operating model, rather than a universal prescription for Slurm clusters.
The division of work remains explicit:
- Kubernetes orchestrates services. It places and restarts pods, the units in which containers run, and connects them to configuration, secrets, and storage.
- Slurm schedules batch work. It queues requests, allocates resources, and launches job steps for batch and distributed-training workloads. These are still Slurm's scheduling responsibilities.
- The operator connects the two. It reconciles Kubernetes resources and uses Slurm state to make lifecycle decisions. Its Slurm control path goes through
slurmrestd, the REST service, which communicates withslurmctld, the queue and allocation controller. The operator never connects directly toslurmctld. This division follows the Slinky operator architecture.

The operator reaches Slurm through slurmrestd, never directly to slurmctld. The controller manages job allocations on slurmd workers.
One Kubernetes foundation does not mean one scheduler. It also does not automatically coordinate every GPU allocation between Kubernetes workloads and Slurm jobs. Capacity ownership, placement rules, and workload isolation remain explicit operating decisions.
How the chart becomes Slurm services
Our public Slurm chart pins Slinky 1.1.0 and Slurm 25.11.0-ubuntu24.04 images. The image version matters beyond reproducibility: the operator's controller and worker health probes depend on the built-in health server introduced in Slurm 25.11. The REST service probes whether its TCP port accepts connections instead.
The chart declares four custom resources, Kubernetes API objects that describe Slurm components. The operator turns those declarations into the underlying services and pods.
| Custom resource | What it describes | Why it matters |
|---|---|---|
Controller |
slurmctld, the queue and allocation controller, with a StatefulSet and persistent save-state storage |
The scheduler's authoritative state must survive pod replacement |
RestApi |
slurmrestd, Slurm's HTTP API, on port 6820 |
The operator's Slurm control path goes through this API, never directly to slurmctld |
NodeSet |
A set of slurmd workers, which receive and supervise job launches |
Worker placement and scaling are declared through the operator |
Token |
An operator-signed JSON Web Token, or JWT, written to a Secret with refresh enabled | Token issuance has a lifecycle separate from daemon authentication keys |
The chart adds no user-facing web portal. slurmrestd provides an HTTP REST API, a programmatic interface, rather than a browser portal. The familiar Slurm command-line tools remain our operator interface. Slurm also has the optional sview graphical interface, which this chart does not deploy.
A StatefulSet gives a service stable identity and storage associations. A persistent volume claim, or PVC, requests the storage backing the controller's saved state. Neither mechanism alone establishes backup, failover, or recovery readiness.
We deploy through GitOps: version-controlled desired state, per-cluster value injection, and ordered synchronization. Cert-manager comes first because the operator's admission webhooks, which validate or default API objects, need Transport Layer Security (TLS) certificates. The operator and its custom resource definitions must be available before the Slurm resources can be reconciled.
The operator owns the generated configuration
The operator generates slurm.conf, including managed node definitions and authentication settings. The chart's config.extra values feed the Controller's extraConf field for unmanaged keys. Adding duplicate operator-owned settings creates competing configuration rather than a useful override.
Several public values need a clear warning:
accounting.enabledandlogin.enabledare placeholders. Setting either totruedoes not deploy an accounting service or login nodes in this chart.config.partitionsandconfig.gresare documentation-only. They are not wired into the operator configuration.- A partition named
gpuis not established by those values. A partition is Slurm's logical queue and node grouping. Its existence and membership require effective configuration through the supported operator fields.
The upstream operator supports capabilities beyond what our chart connects today. A value in a file is not equivalent to a deployed service or an effective Slurm setting.
GPU-node placement is only the first step
Our default NodeSet uses DaemonSet scaling mode, whose intent is one slurmd worker on each matching node. This is an operator scaling mode that reconciles worker pods, rather than a promise of a native Kubernetes DaemonSet object. Its node selector targets nvidia.com/gpu.present=true. That ties worker placement to discovered GPU-node membership instead of a hand-maintained list of hostnames.
This is the elasticity we want: worker placement follows eligible Kubernetes nodes. It does not provision physical GPU capacity or establish that jobs can use the devices.
GPU discovery also has a narrower meaning than GPU allocation. Slurm's generic resource (GRES) mechanism describes devices such as GPUs so jobs can request them. Operator-generated discovery reduces manual inventory work, but a node label only identifies a placement candidate.
Slinky’s GPU installation requirements also call for GPU resources allocated through a device plugin or Dynamic Resource Allocation, Kubernetes' device-allocation mechanism, plus matching Slurm GRES configuration. Our public worker defaults request CPU and memory only. The label and autodiscovery mechanism therefore do not establish usable, schedulable GPU capacity on their own.
Worker sizing matters, too. The pinned worker resource implementation derives CPU and memory inventory inputs from the slurmd container's limits. Those settings cannot be treated as a tiny daemon-only budget while assuming jobs have unconstrained access to the host.
GPU access, resource ownership across schedulers, and successful allocation and execution are part of worker acceptance.
Native Slurm fundamentals that still apply
Containerizing Slurm changes service management. It leaves several familiar responsibilities intact.
Partitions, allocations, and job commands
Users still submit batch scripts with sbatch, launch job steps with srun, view queues with squeue, and view nodes and partitions with sinfo. A job allocation reserves the resources in which its steps execute. GPU requests still depend on the GRES inventory and scheduling policy actually configured.
Usage-based fair-share remains a Slurm capability for distributing priority according to configured usage and shares. It requires an operational accounting database and configured policy; it is not enabled merely because the daemons run in Kubernetes.
Controller state and job data are separate stores
The chart requests a controller save-state PVC to preserve scheduler state. This volume is separate from the shared job-data filesystem containing datasets, model checkpoints, environments, and job output.
Slurm transfers a submitted batch script, but does not move user files. The execution environment needs access to its inputs and writable output paths. Shared job-data storage, its mounts, permissions, and performance remain separate work; the chart does not provision that filesystem.

The chart requests controller save-state storage. Shared job-data storage is provisioned separately; the chart does not provision that filesystem.
Consistent identities and protected keys
Jobs still need consistent numeric user and group identities and correct filesystem permissions across submission and execution environments. A Kubernetes service account, which authorizes access to the Kubernetes API, does not supply that Linux identity mapping. The mechanism providing those identities is a separate design choice.
Our configuration uses auth/slurm and cred/slurm for shared-key authentication and job credentials. The protected slurm.key fills the role of the shared secret in MUNGE, an authentication service used by native Slurm clusters.
The pinned operator sets AuthInfo=use_client_ids. This client-identity option allows auth/slurm authentication without relying on LDAP directory or operating-system user lookups. It can pair with nss_slurm, which supplies user information on compute nodes. Replicated local accounts are therefore not a universal authentication prerequisite. The chart does not provision a complete user-mapping integration or resolve job-data ownership.
JWT authentication serves a different path. The operator manages its REST-client token, while the chart also declares a refreshing Token resource. The separate Token resource is not connected to a consumer in the public chart. It is not the operator's own short-lived REST-client token, and token refresh does not mean all keys rotate automatically. The chart includes logic intended to reuse existing authentication keys across upgrades. Preserving those keys is an upgrade and recovery responsibility.
Networking, resource controls, and accounting
Controller-to-worker communication and job-step traffic still need working network paths. Distributed applications may also need an interconnect and communication libraries suited to their workload. Pod readiness does not demonstrate those paths or their performance.
Linux resource controls still matter. Container limits, Slurm's advertised resources, process tracking, and device access have to agree. Security settings are component-specific: the controller declares non-root execution, a runtime-default seccomp system-call filter, disabled privilege escalation, and dropped capabilities. These settings do not establish worker privilege safety or tenant isolation. Hardening fields in a generic pod template are not evidence of the effective primary-container settings.
For durable usage records, account associations, and limits, slurmdbd, Slurm's accounting daemon, and a supported database remain separate services to configure and operate. The Slurm accounting architecture also determines what sacct can report. The placeholder accounting flag supplies none of that infrastructure.
Observability spans Kubernetes and Slurm
There are two different failure surfaces. Kubernetes may reject or fail to place a pod before a Slurm daemon starts. A running pod may then contain a worker that has not registered correctly with Slurm.
Kubernetes exposes the first layer through custom-resource status, events, pod specifications, logs, and storage claims. We act only on access the customer explicitly grants, with role-based permissions for each person. For these read-only observations, the authorized operator sets SLURM_NAMESPACE to the deployment's actual namespace, then sets SLURM_POD and SLURM_CONTAINER to names returned by pod and container discovery:
kubectl api-resources --api-group=slinky.slurm.net
kubectl -n "$SLURM_NAMESPACE" get controllers.slinky.slurm.net,restapis.slinky.slurm.net
kubectl -n "$SLURM_NAMESPACE" get nodesets.slinky.slurm.net,tokens.slinky.slurm.net
kubectl -n "$SLURM_NAMESPACE" get pods -o wide
kubectl -n "$SLURM_NAMESPACE" get pvc
kubectl -n "$SLURM_NAMESPACE" get events --sort-by=.metadata.creationTimestamp
kubectl -n "$SLURM_NAMESPACE" get pod "$SLURM_POD" \
-o jsonpath='{.spec.containers[*].name}'
kubectl -n "$SLURM_NAMESPACE" logs "$SLURM_POD" -c "$SLURM_CONTAINER"
These variables hold actual discovered names. They do not assume a release name, generated pod name, or container layout.
Where the selected container includes the Slurm command-line tools and has the necessary authentication, the familiar tools expose the second layer:
kubectl -n "$SLURM_NAMESPACE" exec "$SLURM_POD" -c "$SLURM_CONTAINER" -- scontrol ping
kubectl -n "$SLURM_NAMESPACE" exec "$SLURM_POD" -c "$SLURM_CONTAINER" -- sinfo -N -l
kubectl -n "$SLURM_NAMESPACE" exec "$SLURM_POD" -c "$SLURM_CONTAINER" -- scontrol show nodes
kubectl -n "$SLURM_NAMESPACE" exec "$SLURM_POD" -c "$SLURM_CONTAINER" -- scontrol show partition
kubectl -n "$SLURM_NAMESPACE" exec "$SLURM_POD" -c "$SLURM_CONTAINER" -- squeue
These are observations, not an installation acceptance result. A responsive controller, visible partitions, and registered workers answer different questions.
A production handover needs more than healthy pods
Production readiness covers the complete execution path, from reconciled services to finished workloads and recovery. The handover evidence needs to establish each boundary:
| Evidence | What it establishes | What remains separate |
|---|---|---|
| Reconciled resources and ready services | Kubernetes can run the intended components | Correct Slurm inventory and scheduling policy |
| Registered workers with intended CPU, memory, and GRES | Slurm sees the resources intended for its pool | Cross-scheduler ownership and isolation |
| An ordinary-user job that finishes and releases resources | Identity, allocation, launch, output, and cleanup work together | GPU execution and distributed-workload behavior |
| A successful GPU job and distributed job where required | The relevant device and communication paths work for those workloads | Sustained performance and capacity planning |
| Recovery and upgrade acceptance | State, keys, and operational procedures survive the defined lifecycle changes | Ongoing monitoring and incident ownership |
Submission alone is insufficient. A batch handover includes the completed job state, exit status, output in the intended filesystem, and released resources. Durable accounting requires its own configured storage path.
We deliver production-ready Slurm on customer clusters within 48 hours of the datacenter handing over clean bare metal, using clusterdOS on the customer's behalf. The GPU, CPU, and storage hardware must already be racked and ready; hardware procurement and physical installation are outside that window.
With fully managed clusters, we cover architecture, deployment, and end-to-end operations. With self-managed software, currently in private alpha, the customer operates the software. The delivery mode determines who owns these responsibilities; Slurm's identity, storage, scheduling, and recovery requirements remain the same.
Talk with our team about a workload-specific cluster and the operating responsibilities behind its Slurm deployment.