NVIDIA GPU Operator: Compatibility and Lifecycle Guide

A GPU driver update is a customer-capacity event. Someone must decide which nodes can leave service, which GPU workloads must keep running, and who restores capacity if the new driver fails. NVIDIA GPU Operator puts much of the GPU software stack under Kubernetes management. Adopting it starts with assigning responsibility for that stack. The useful decision is how its automation fits your node images, maintenance process, and operational commitments.

What is NVIDIA GPU Operator?

NVIDIA GPU Operator is a Kubernetes controller that installs and manages the NVIDIA software needed to make GPUs usable by container workloads. It coordinates drivers, container integration, device discovery, GPU resource registration, monitoring components, and validation. NVIDIA calls the components it manages operands. A Kubernetes DaemonSet places a component’s pod on each eligible node. NVIDIA’s Operator overview describes this software boundary.

The Operator keeps a declared GPU configuration applied to eligible nodes. It does not replace the cluster’s operating system, Kubernetes control plane, network, storage, or workload recovery design. We treat it as one managed subsystem within the larger GPU cluster architecture.

Choose the enablement path first

GPU Operator v26.7.2 supports two alternative management paths:

• ClusterPolicy with the NVIDIA device plugin: ClusterPolicy declares the GPU stack configuration. The device plugin registers GPUs with Kubernetes, conventionally as the nvidia.com/gpu extended resource.
• GPUCluster with Dynamic Resource Allocation (DRA): DRA is Kubernetes’ device-request API. This path uses GPUCluster to manage DRA components and a separate NVIDIADriver resource or host-installed driver for driver ownership.

A cluster can have either a ClusterPolicy or a GPUCluster, never both. Their managed components and validation paths differ. The examples here use ClusterPolicy and the device plugin, without DRA. NVIDIA’s path comparison defines the distinction.

Map components to operational owners

A containerized GPU driver still loads modules into the host kernel. Putting its installer in a pod does not isolate driver changes from the node. That is why ownership must cover both the Kubernetes resource and its host-side effects.

The core component roles follow the Operator software model. DCGM Exporter exposes telemetry; Prometheus or another collector stores and queries it. A running exporter alone does not establish monitoring coverage. NVIDIA’s exporter documentation separates those responsibilities.

Selected ClusterPolicy-managed components. NFD discovery and any optional managers also need explicit lifecycle owners.

The ownership boundary also matters when choosing the cluster layer. Our clusterdOS engine is the open-source engine inside each cluster, with its source repository available publicly. It is Kubernetes-based and extends to virtual-machine and Slurm batch-scheduling environments. Aranya OS, our multicluster operating system, builds on clusterdOS to federate clusters into one fleet and operate within and across them. Our fully managed clusters delivery covers architecture, deployment, and end-to-end operations. Customers can also operate our self-managed software, currently in private alpha. Those are distinct layers of responsibility; selecting GPU Operator does not select a cluster operating model or establish support for a particular configuration.

Compatibility and prerequisite matrix

Compatibility is an intersection: GPU model, CPU architecture, OS release, kernel, Kubernetes distribution, runtime, driver, and workload image must fit together. A supported GPU model alone is insufficient.

Illustrative environment: x86-64 bare-metal workers, four NVIDIA H100 GPUs connected over PCI Express (PCIe) per worker, Ubuntu 24.04 LTS with the 6.8 generic kernel family, upstream Kubernetes 1.35, containerd 2.1, and GPU Operator v26.7.2. The example uses Operator-managed conventional driver containers and whole GPUs, with MIG disabled. The example excludes NVSwitch GPU-interconnect fabrics, virtual GPU (vGPU) environments, and GPUDirect network or storage data paths. The CUDA smoke workload is NVIDIA’s CUDA 12.5 VectorAdd sample. These are example inputs and expected states, not observed deployment results.

NVIDIA’s v26.7 matrix lists H100, Ubuntu 24.04, Kubernetes 1.33–1.37, and containerd 2.0–2.3 among the applicable supported platforms. Each production release record still needs exact patch versions and the full kernel build identifier. NVIDIA’s platform matrix is the support boundary.

The table is a release gate, not a universal supported configuration. Three details deserve explicit decisions.

Pin the component bundle, not just the chart

For v26.7.2, the tagged chart defaults include driver 595.91.07, Container Toolkit v1.20.1, device plugin and GFD v0.20.1, and DCGM Exporter image 4.6.1-4.8.4-distroless. Standalone DCGM is disabled by default. A higher driver listed as supported is not automatically the default. The tagged chart values identify the reproducible starting bundle.

CUDA libraries and frameworks belong to the application image. Their compatibility requirement remains even when the Operator manages the driver. For example, CUDA 12.x applications can use newer drivers through backward compatibility, while minor-version compatibility has feature limitations. NVIDIA’s CUDA compatibility guidance distinguishes those cases. A successful GPU enumeration is not proof that a framework’s kernels execute correctly.

Make kernel updates part of the same lifecycle

Precompiled driver containers remove the need to fetch compiler tooling and kernel headers during installation. Their availability is specific to OS, kernel, and driver versions, and they have feature restrictions, including no NVIDIA vGPU or GPUDirect Storage support. NVIDIA’s precompiled-driver requirements define those limits.

The operational consequence is straightforward: an OS-image or kernel change reopens the GPU compatibility gate. A kernel rollout and a GPU-stack rollout need a shared release record, even when different teams execute them.

Approve host access and artifact access separately

Several operands require privileged containers and access to host process or interprocess communication (IPC) namespaces so they can access hardware, load modules, or restart services. These are infrastructure-level permissions. NVIDIA’s security requirements explain why they are needed.

Restricted networks introduce a separate dependency. Mirroring the chart is insufficient: operand images and conventional driver-build packages also need an accessible source. Precompiled drivers remove the package-download dependency, while image access remains necessary. NVIDIA’s air-gapped deployment guidance separates those artifact paths.

Decide how the Operator joins your existing stack

Three ownership models cover the main ClusterPolicy deployment decisions:

Existing NFD requires nfd.enabled=false for the Operator installation. These boundaries are documented in NVIDIA’s installation configuration options. Disabling management transfers responsibility; it does not remove the dependency.

Mixed OS pools introduce a further choice. ClusterPolicy driver management uses one driver type and version and does not support multiple OS versions. The separate NVIDIADriver custom resource supports selected node pools, multiple driver configurations, and multiple OS versions. A custom resource definition (CRD) adds such a resource type to Kubernetes. Enabling this mode delegates driver management to its controller while ClusterPolicy retains the other operands. NVIDIA’s driver-resource documentation defines the supported model.

The deployment input should therefore contain the pinned chart, component overrides, node-selection rules, ownership exclusions, and security and registry configuration. For the illustrative environment, it also records migManager.enabled=false. That disables MIG Manager; it does not turn off an existing MIG configuration. The example separately assumes the GPUs are already in MIG-disabled mode. Installation can then use NVIDIA’s versioned procedure without turning default settings into an accidental operations policy.

Lifecycle decisions after installation

Adding a node or replacing an image

A replacement worker must belong to an approved hardware and software pool before its GPU capacity returns to service. The release record should preserve the previous image and GPU-stack versions together. Separately recording “Kubernetes upgraded” and “driver installed” loses the combination needed for recovery.

Upgrading the Operator

NVIDIA supports upgrades within a major release or to the next major release. In the v26.7 support table, 26.7.x is supported, 26.3.x is deprecated, and 25.10.x and earlier are end of support. An old chart pin is therefore a lifecycle obligation, not a permanent stability strategy. NVIDIA’s release lifecycle sets those boundaries.

An Operator upgrade includes CRD handling and operand configuration. Helm, Kubernetes’ package manager, does not automatically upgrade existing CRDs; NVIDIA documents manual updates and a Helm-hook approach. The release plan must account for the chosen mechanism and retain intentional values. NVIDIA’s upgrade procedure covers that sequence. Reverting a chart alone is insufficient as a recovery plan for host drivers, CRDs, and runtime configuration.

Upgrading the driver

Driver changes unload and reload host kernel modules, so GPU clients must release the driver. The Operator manages this lifecycle only for containerized drivers. Its upgrade controller is enabled by default; the v26.7.2 defaults set maxParallelUpgrades: 1 and maxUnavailable: 25%. The percentage rounds up to whole nodes, so three managed nodes permit one unavailable node. Already-unavailable nodes consume that budget. The controller orchestrates changes to the declared driver version; it does not choose a newer release for you. Drain is disabled by default. Enabling drain permits a disruptive fallback that can evict non-GPU workloads too. NVIDIA’s driver-upgrade controls describe these behaviors.

We recommend a release gate with an approved maintenance window, recoverable workloads, spare capacity, and a canary pool, a small representative group of nodes. The decision to proceed rests on the canary’s acceptance evidence, including workload execution, rather than the controller merely starting its rollout.

Crossing a feature boundary

Adding NVSwitch, GPUDirect, virtualization, or another driver owner changes the dependency graph. For example, v26.7.2 documents an interaction when GPU Operator and Network Operator manage containerized drivers on the same GPU nodes with NVLink 5 or later: the NVLink Subnet Manager (nvlsm) service holds a reference to the ib_umad kernel module, which blocks the DOCA-OFED driver container’s installation. DOCA-OFED is NVIDIA’s networking-driver suite. That is a coordinated GPU/network lifecycle issue. NVIDIA’s release notes describe the affected configurations. The PCIe-only example does not exercise that dependency.

Focused Operator acceptance checks

The following commands are examples for the ClusterPolicy/device-plugin path, using namespace gpu-operator. They inspect the vendor stack rather than repeat cluster installation or general node bring-up.

kubectl get clusterpolicy
kubectl get daemonsets,pods -n gpu-operator -o wide
kubectl get events -n gpu-operator --sort-by=.lastTimestamp
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true \
  -o custom-columns='NAME:.metadata.name,GPU-CAPACITY:.status.capacity.nvidia\.com/gpu,GPU-ALLOCATABLE:.status.allocatable.nvidia\.com/gpu'

For the illustrative four-GPU workers, expected acceptance states are:

• ClusterPolicy reports ready.
• Enabled DaemonSets cover their intended nodes, with their containers ready. Completed one-shot validator pods are normal.
• Each healthy worker advertises capacity and allocatable values of 4 for nvidia.com/gpu.
• A one-GPU CUDA sample completes successfully. NVIDIA’s sample-workload verification uses VectorAdd and shows Test PASSED as its expected result.

Allocatable node resources are the scheduler-facing totals, not the number of GPUs currently free. A whole-GPU count assumption also stops applying when partitioning or sharing changes the resource model. A single sample on one worker establishes neither per-node coverage nor production throughput.

Container Device Interface (CDI), the standard describing how a runtime exposes devices inside containers, is enabled by default from GPU Operator v25.10. Standard device-plugin workloads do not universally require runtimeClassName: nvidia. Management containers that bypass device allocation have different requirements. NVIDIA’s CDI guidance distinguishes them.

For an active containerized-driver rollout, the node’s upgrade-state label exposes progress:

kubectl get nodes -l nvidia.com/gpu.present=true \
  -L nvidia.com/gpu-driver-upgrade-state

upgrade-done is the expected terminal success state for participating nodes. An absent label can mean the controller is disabled or has not processed the node; it is not a success signal. The application acceptance gate still applies after the driver transition.

Diagnose the first failing layer

A downstream pod waiting in Init often reflects an upstream driver or toolkit dependency. Restarting every operand at once obscures that relationship.

NVIDIA’s Operator troubleshooting guide maps the driver, runtime, device-health, and rollout symptoms. Preserve node identity, GPU identity, versions, events, and the first failing container’s logs before restarting components.

A ready GPU stack is an enablement milestone. Returning customer capacity also requires the workload, its network and storage paths, and its recovery behavior to meet the service objective. Our work on inference infrastructure reliability explains why node readiness and serving availability need separate gates.

Bring us your GPU inventory, node-image constraints, and workload requirements to scope cluster operations, including who owns the NVIDIA stack throughout its lifecycle.