Heterogeneous clusters: an operator’s guide to mixed hardware

Adding newer graphics processing units (GPUs) to an existing cluster can extend its useful life. It also creates a placement problem: a workload that runs on one node may fail, run out of memory, or miss its latency target on another.

For datacenter and neocloud operators, the operating plan needs to connect hardware inventory to workload eligibility, maintenance capacity, and recovery. That connection starts with explicit node pools and a compatibility record for each workload.

What makes a cluster heterogeneous?

A heterogeneous cluster contains nodes with different hardware or execution capabilities. Differences can include GPU vendors or generations, processor architectures, memory capacity, network interfaces, and storage. A homogeneous cluster standardizes those capabilities across its worker nodes.

The distinction is operational. Two servers can advertise the same number of GPUs while supporting different workloads. Even identical GPU models can need separate treatment when their server topology, network adapters, or firmware differ.

In Kubernetes, the control plane maintains cluster state and coordinates scheduling. A node pool is an operator-defined group of workers with a common hardware and software contract. Several pools can share one control plane without becoming interchangeable capacity.

Our clusterdOS engine provides the open-source, Kubernetes-based operating layer inside an individual cluster, extending to virtual machines and Slurm, a scheduler for queued compute jobs. Aranya OS, our proprietary multicluster operating system, builds on this engine to federate clusters into one fleet. A mixed-hardware operating plan needs both boundaries to stay visible.

A worked mixed-generation inventory

Consider a hypothetical inference operator retaining A100 servers while adding H100 servers. Hardware counts and workload envelopes are planning assumptions.

All workers use x86-64 central processors (CPUs), Linux, and whole-GPU allocation. GPU partitioning and time-slicing are disabled, so one advertised GPU resource represents one physical GPU. PCIe is the server expansion bus connecting add-in devices; SXM is a GPU module form factor used in tightly connected accelerator systems.

Three dedicated control-plane nodes sit outside these worker counts. Model files originate in object storage and are staged into each worker’s local cache. No workload in this example requires cross-node GPU communication or remote direct memory access, which lets network devices transfer data without the usual host-memory copying path.

The software baseline is Kubernetes 1.35, Ubuntu 24.04 long-term support with the 6.8 Linux kernel family, containerd 2.1, GPU Operator 26.7.2, and NVIDIA driver 580.173.02. Containerd runs containers; the GPU Operator manages the supporting NVIDIA components. These platform families and the selected driver appear in NVIDIA’s platform support matrix. A deployment release record additionally pins exact operating-system packages, kernel build, firmware, operator components, and container digests, the immutable identifiers for image contents.

The H100 pool also needs NVIDIA Fabric Manager, the service that configures and monitors the NVSwitch fabric connecting its GPUs. That service and the GPU links belong in the pool’s release and acceptance records.

The example GPU applications use PyTorch 2.11.0 with CUDA 12.8. PyTorch supplies the tensor-computation framework; CUDA supplies NVIDIA’s GPU programming runtime and libraries. That PyTorch build combination exists, but it does not establish compatibility for application-specific extensions or models.

The inventory totals 24 GPUs: eight A100s and 16 H100s. It does not describe a 24-GPU machine or a shared memory space.

Compatibility has three separate gates

1. The host stack can expose the device

The operating system, kernel, driver, container runtime, and device plugin must form a supported combination. A device plugin advertises accelerator resources to Kubernetes so its scheduler can allocate them. Kubernetes documents GPU allocation through device plugins and limits.

A node reaching Ready proves less than an application completing GPU work. The acceptance record needs both the host’s physical device inventory and Kubernetes’ allocatable device count. Here, an A100 worker should expose four whole GPUs and an H100 worker eight.

2. The application contains compatible GPU code

Driver compatibility and application compatibility are different questions. NVIDIA supports a newer driver running an older CUDA runtime through CUDA backward compatibility. That does not make every compiled GPU function portable between generations.

A100 and H100 have compute capabilities 8.0 and 9.0 respectively, hardware feature levels that compiled GPU code targets. An extension containing only an 8.0 native binary cannot use that binary on H100. Compatible intermediate code, called PTX, can provide a forward-compilation path. NVIDIA’s Hopper compatibility guide explains these boundaries. For a dual-pool application, the release contract covers every extension and dependency, with compatible code for both targets.

3. The workload fits its operating envelope

A runnable application can still be unsuitable for a pool. Eligibility also depends on peak GPU memory, host memory, processor demand, concurrency, storage access, and latency or completion-time targets.

GPU memory must cover model weights, runtime workspace, and request state. For language-model serving, request state includes the key-value cache holding attention data. A nominal memory specification is an upper boundary; qualification uses memory available to the application and retains an explicit reserve.

Compatibility, placement, and lifecycle matrix

A Pod is Kubernetes’ one-node execution unit, containing one or more containers. The example uses these workload contracts:

• P: CPU preprocessing, requesting 16 CPU cores and 64 GiB host memory.
• E30: one-GPU embedding service with a 30 GB peak device-memory envelope and a 4 GB reserve. Its image supports both GPU generations.
• L64: one-GPU inference replica with a 64 GB peak device-memory envelope and a 4 GB reserve. Its image supports both generations, but its memory requirement excludes the A100 pool.
• H90: one-GPU application with a 20 GB device-memory envelope and 4 GB reserve, whose custom extension contains only compute-capability-9.0 native code.
• G8: one Pod requesting eight whole GPUs, 16 CPU cores, and 128 GiB host memory for a single-server job, with a 20 GB device-memory envelope plus 4 GB reserve per GPU. Its application supports both generations, but its deployment has no multinode launcher.

The one-GPU services each request eight CPU cores and 64 GiB host memory. Device-memory figures are example budgets in decimal GB. They are not model benchmarks. “Candidate” below means the static contract fits; production admission also requires the qualification gates that follow.

Workload: P

• cpu: Candidate
• a100-40: Excluded by placement policy
• h100-80: Excluded by placement policy
• Placement and lifecycle rule: CPU pool only; GPU nodes remain reserved for accelerator work.

Workload: E30

• cpu: No GPU
• a100-40: Candidate: 30 + 4 < 40 GB nominal
• h100-80: Candidate: 30 + 4 < 80 GB nominal
• Placement and lifecycle rule: A100 is the initial placement; H100 is an alternate only after its own performance qualification. Requalification covers both pools after image changes.

Workload: L64

• cpu: No GPU
• a100-40: Rejected: 64 + 4 > 40 GB
• h100-80: Candidate: 64 + 4 < 80 GB nominal
• Placement and lifecycle rule: H100-only placement. Maintenance reserve must come from H100 capacity qualified for this image.

Workload: H90

• cpu: No GPU
• a100-40: Rejected: wrong code target
• h100-80: Candidate: compatible code target
• Placement and lifecycle rule: H100-only until a release adds and qualifies A100 support; memory and performance gates still apply.

Workload: G8

• cpu: No GPU
• a100-40: Rejected: only four GPUs per node
• h100-80: Candidate: eight GPUs on one node
• Placement and lifecycle rule: An entire H100 worker is the allocation unit. Restart requires another eligible eight-GPU node or the job waits.

For G8, eight free A100 GPUs spread across two nodes cannot satisfy one eight-GPU Pod. A multinode implementation would require a different deployment contract, including process launch, communication libraries, network behavior, and recovery.

The same principle applies to mixing NVIDIA and AMD hardware: each backend needs its own compatibility and qualification record. Sharing a scheduler or model format does not establish that a single distributed job can combine the backends. Cross-vendor execution belongs to the specific framework’s supported configuration.

Encode pool eligibility in scheduling policy

GPU counts describe quantity. Labels describe which hardware is eligible. Kubernetes supports selecting GPU types through node labels and selectors.

For L64, the scheduling contract contains a hard pool selector and a whole-GPU limit:

nodeSelector:
  kubernetes.io/arch: amd64
  ops.example.com/pool: h100-80

Within the application container’s resource specification:

resources:
  requests:
    cpu: "8"
    memory: 64Gi
  limits:
    nvidia.com/gpu: 1

These are configuration fragments, not a complete application manifest. The ops.example.com/pool label is an illustrative operator-owned label. The host-memory request is separate from the workload’s GPU-memory envelope: memory: 64Gi does not reserve GPU memory. With the device-plugin allocation model, a GPU limit also supplies the GPU request when the request is omitted.

GPU pools can also carry a NoSchedule taint, a scheduling exclusion that designated GPU workloads tolerate. A toleration allows placement; it does not select a pool. A matching hard selector or required node affinity supplies that constraint. Kubernetes’ taint and toleration rules distinguish the two mechanisms. These controls organize placement; they do not establish a customer security boundary.

For E30, a required eligibility rule can allow both qualified GPU pools while a preference favors A100. For L64, H100 eligibility must stay required. A preference alone would allow a scheduler to choose a node that cannot meet the memory contract.

Qualification turns candidate capacity into usable capacity

Each admitted workload needs a result for each eligible pool, tied to its image digest, configuration, and node-stack release. A single cluster-wide “GPU healthy” flag loses the distinction the placement matrix depends on.

Gate: Device discovery

• Evidence required for a pass: Physical GPU identities and Kubernetes allocatable counts agree with the inventory, including after reboot
• Rejection or hold condition: Missing devices, unexpected sharing configuration, or incorrect labels

Gate: Execution and correctness

• Evidence required for a pass: Every application code path completes on the pool and matches its defined output tolerance
• Rejection or hold condition: Unsupported GPU code, missing libraries, numerical errors, or process crashes

Gate: Memory fit

• Evidence required for a pass: Peak usage at the contract’s maximum input size and concurrency leaves the defined reserve
• Rejection or hold condition: Out-of-memory failure, reserve exhausted, or increasing memory use across repeated requests

Gate: Performance

• Evidence required for a pass: Throughput and tail latency meet the workload’s target under representative concurrent load
• Rejection or hold condition: A running replica misses its service objective

Gate: Data and topology

• Evidence required for a pass: Cold-cache startup, model retrieval, storage permissions, and required GPU-to-GPU paths meet the contract
• Rejection or hold condition: Startup exceeds the recovery budget or expected links are absent

Gate: Recovery

• Evidence required for a pass: An eligible replacement starts, restores required state, warms up, and becomes ready within the recovery budget
• Rejection or hold condition: Replacement remains pending, cannot access data, or serves before it is ready

Hardware diagnostics complement these gates. NVIDIA’s Data Center GPU Manager provides DCGM diagnostics for GPU health and stress diagnostics; they do not replace an application’s correctness or latency requirements.

For an illustrative E30 service, a contract might require 100 requests per second per replica, p99 latency below 200 milliseconds, and recovery within five minutes. Those are example acceptance targets, with p99 meaning 99% of requests finish within the limit. They are neither observed results nor a throughput projection for either GPU.

Operational telemetry should retain pool and release identity alongside errors, latency, GPU-memory pressure, thermal throttling, memory error-correction events, and network faults. Otherwise, a fleet-wide average can hide a regression isolated to one generation. Our inference reliability guide develops the distinction between infrastructure health and service reliability.

Upgrade against eligible replacement capacity

A heterogeneous cluster needs a maintenance budget by workload class. Total free GPUs can give a dangerously reassuring answer.

Suppose ten L64 replicas require ten whole H100 GPUs. Both H100 workers are healthy, providing 16 eligible slots. During maintenance of one eight-GPU worker:

• Eight H100 slots remain.
• Eight A100 slots remain, but none qualify for L64.
• The remaining cluster has 16 physical GPUs, yet only eight eligible slots for ten replicas.
• Two replicas have no replacement capacity. Preserving ten requires at least two additional qualified H100 slots outside the maintenance node, with enough host resources and data access.

The capacity arithmetic assumes one replica per GPU and no additional placement restrictions. Lower traffic, reduced replica count, or another qualified cluster changes the plan; a larger A100 free count does not.

The upgrade sequence follows the qualification boundary:

1. Define the target release. The release record pins the host stack and application artifacts, their supported combinations, and the previous recoverable configuration.
2. Qualify one representative worker per hardware class. An A100 canary does not qualify the H100 topology. Cluster-wide operator and networking changes also need coverage of every affected pool.
3. Reserve compatible replacement capacity. The maintenance plan accounts for workload eligibility, node-sized allocations, model loading, and persistent state before eviction begins.
4. Cordon and drain the selected node. Cordon blocks new ordinary scheduling; drain evicts workloads for relocation. A PodDisruptionBudget limits voluntary disruption, but creates no replacement capacity. The standard node drain workflow respects that distinction.
5. Apply the change and repeat the gates. Device discovery, correctness, memory, performance, and recovery results belong to the new release. Production eligibility returns only after these workload gates pass. Broader rollout waits for a passing canary and the planned observation window.
6. Stop on a failed gate. The node remains unavailable for production scheduling while the previous compatible stack is restored or the fault is resolved. Automatic kernel updates cannot bypass this release process.

A separate node pool does not permit arbitrary Kubernetes versions. Control-plane and worker upgrades still follow Kubernetes’ version skew policy. GPU driver rollout is another lifecycle: the GPU Operator provides its own driver upgrade controls. Their concurrency settings must fit the eligible-capacity budget.

When node pools should become separate clusters

Pools are appropriate when workers can share a supported control plane, network and storage services, operational ownership, and security model. Separate clusters become useful when those shared assumptions break: incompatible lifecycle requirements, independent change windows, distinct administrative trust, or facility-level failure boundaries.

A second cluster in the same datacenter can separate upgrade responsibility. It still shares that facility’s power and connectivity risks. A second facility adds another infrastructure boundary, along with requirements for data availability, workload qualification, routing, and access control.

At that point, the responsibility extends beyond placing a Pod. Our multicluster operating system, Aranya OS, coordinates the fleet above the clusterdOS engines inside individual clusters. We operate it for you as fully managed clusters, covering architecture, deployment, and end-to-end operations, or you operate it yourself as self-managed software (currently in private alpha). The operating-system architecture stays the same; delivery responsibility changes.

The useful handoff is concrete: a hardware inventory, the workload eligibility matrix, release records, and a replacement-capacity budget. Talk to an engineer about turning that inventory into workload-tuned production clusters.