Why GPU inference infrastructure fails in production and how to architect around it

Why generic managed Kubernetes is not sufficient for production inference

Most inference companies buy managed Kubernetes assuming any managed offering is suitable for their workload. And while Kubernetes itself is the right orchestration primitive for inference, generic managed Kubernetes is not the right product.

Teams aren't careless about this. They're being rational under constraints. GPU demand is outpacing supply, time-to-market concerns live at the forefront, and engineering capacity is limited. As many teams scale beyond their first cluster, they do account for a backup internet service provider, a secondary region, and a documented failover path. Those measures still need coordinated routing, ready capacity, and exercised recovery procedures. A backup that cannot serve the workload leaves a gap that may only surface when the primary fails.

By that point, the failure has already happened, and it falls into one of two buckets. Either something broke inside the datacenter, or the whole datacenter went down. The two get planned for as if they are the same problem. The thing is: they aren't.

The Two Ways GPU Inference Infrastructure Fails

The first is everything that can go wrong inside a functioning datacenter. Think hardware degradation, networking failures, and inference engine issues. The second is the datacenter itself going down.

Type 1: What breaks inside a GPU inference cluster

Production inference incidents rarely start with a hardware crash. They present as silent degradation: recurring timeouts at the engine layer, replicas serving requests insufficiently, or latency drift without an observable root cause.

A Microsoft study of 156 high-severity production LLM inference incidents found roughly 60% were inference engine failures, and within that category, about 40% were timeouts. The cluster is up, the orchestrator is healthy, but inference is not.

Load balancing is a quieter culprit, but responsible for more than it should be. Round-robin routing across GPU replicas forces each replica to recompute work already done elsewhere. That’s a 6,000-token system prompt reprocessed from scratch every time a request lands on the wrong replica. In a benchmark simulating a multi-tenant workload on 16 H100 GPUs, switching from round-robin to prefix-cache-aware routing cut P90 time-to-first-token from over 90 seconds to 0.54 seconds and roughly doubled throughput. More GPUs doesn't fix this. It actually scales the problem.

Networking is the layer that is rarely specified upfront. But it compounds everything else. On multi-tenant GPU clusters, the network fabric is typically shared even when the GPUs are physically dedicated, meaning the same model, same query, and same batch size, on the same cluster, produce meaningfully different p99 numbers day to day, sometimes hour to hour. P99 latency at 450ms one day and 900ms the next is not a software problem with a software fix. Most clusters get deployed with whatever networking configuration the datacenter provides by default. The default that's tuned for generic compute is rarely tuned for inference. The gap doesn't surface during setup. It surfaces under load, either as p99 variance, throughput degradation, or latency that can't be traced back to the application.

Type 2: What happens when the datacenter goes down

Type 2 failure modes take away the infrastructure the cluster depends on. The internet connection drops, power fails, or a cloud service has a regional event. Repairing an individual GPU node cannot restore service while that dependency is unavailable.

A physical datacenter, a cloud availability zone, and a region are different fault domains, meaning boundaries within which a failure can affect multiple resources. An AWS availability zone can contain multiple datacenters, and a region contains multiple zones. Continuity requires serving capacity and dependencies outside the domain that failed.

AWS had at least one documented outage every month in 2025 except the final two. The October event: a race condition in DynamoDB’s automated DNS management ran for roughly 15 hours, and took Slack, Atlassian, and Snapchat with it. 197 SaaS providers publicly attributed their own outages to it. Not only is us-east-1 the busiest AWS region in the world, but it also happens to be where the most outages were recorded in 2025.

A single datacenter inference deployment has no answer to this. The facility goes down, inference goes with it.

Why the Single Datacenter Default Makes Sense… Until It Doesn’t

Deploying on a single datacenter is often the right call early on. GPU supply is still constrained, time to market matters, and the architecture review window doesn’t clearly open.

Once teams begin to scale, they add clusters. The most common pattern is spinning up a secondary cluster in the same facility as the first: more capacity, same datacenter. The catch is, if that facility fails, every cluster inside goes down with it.

Even when teams do account for this and scale in different datacenters, cross-site recovery can remain unprepared. Independent clusters can use external routing, but traffic still needs a configured cutover path. Without cross-site routing and cutover, the other clusters keep running their own load when a facility goes down, and the affected traffic has nowhere to go.

Utilization and the cost case for federation

Making compute cheaper starts with using paid-for capacity well. Federation has both an engineering cost and a capacity cost. Routing, state, model weights, and failover behavior all have to coordinate across facilities. Existing paid headroom can reduce the incremental compute needed for failover, provided that headroom is usable by the displaced workload.

Average GPU duty-cycle utilization does not establish that capacity. It measures how much of a sampling window the GPU spends actively processing, rather than the requests or tokens a destination can serve at acceptable latency. Compatible GPUs and sufficient memory, healthy schedulable nodes, the required GPU arrangement for each serving replica, and available CPU, network, storage, and model access all constrain usable headroom.

The capacity budget must compare the surviving sites’ own demand plus redirected demand at the same time against their demonstrated throughput at the accepted latency and error objectives. Off-peak averages can hide coincident peaks. A secondary facility at similar average utilization is not necessarily able to absorb the primary’s traffic.

For illustrative arithmetic, suppose two facilities each have 100 units of compatible serving capacity demonstrated to meet the chosen latency objective. If each is serving 60 units of demand when one fails, the survivor receives 120 units and has a 20-unit deficit. At 80 units each, it receives 160 and has a 60-unit deficit. These are hypothetical serving-capacity units, not GPU-utilization percentages or an Aranya benchmark.

Ready replicas, replicas that still need to start and load weights, and GPUs that have yet to be provisioned offer different recovery times. Immediate continuity depends on sufficient pre-existing capacity outside the failed domain. AWS’s static-stability guidance warns against relying on acquiring resources during a failure.

When surviving capacity is insufficient, the design needs additional or reserved capacity, deferred or rejected requests, or explicitly relaxed service objectives. Each changes the cost or customer outcome. Federation can make existing headroom useful, but replication, routing, intersite transfer, reservations, and operations still carry costs.

Production-Grade Architecture for Each Failure Class

In-cluster reliability

Hardening an inference cluster against Type 1 failures comes down to three design layers: networking, observability, and storage. Each has to be specified for the workload. Managed Kubernetes supplies useful infrastructure-health and repair mechanisms; integrating them with inference performance and recovery remains an architectural responsibility.

Networking. The unexplained p99 variance and throughput degradation described above mostly trace back to how the fabric was (or wasn’t) configured at provisioning. That’s a fabric specification decision, not a software problem. What the right specification looks like depends on the workload, but the throughline is that serving thousands of concurrent users on shared, distributed GPU infrastructure requires high performance networking that a generic compute default doesn’t provide.

This is a necessary gap to close before provisioning begins. Based on the specific workload (inference type, model architecture, traffic pattern, etc.), networking architecture should be specified as an explicit requirement rather than accepting the datacenter's default. It's the most common source of Type 1 failures in the clusters we've operated, and the most disruptive layer to change after the fact.

We've seen networking-only changes reduce a category of recurring incidents substantially in engagements where nothing in the application stack was touched. The network is the most consequential reliability layer once it's specified correctly, and also the most consequential gap when it isn't.

Observability. Most of the failure modes above sit at the inference-engine layer, the dominant incident category in the Microsoft study. Pod status or node readiness alone does not establish that a model is serving correctly or meeting its latency objective. GPU health signals add another layer, and managed Kubernetes providers already automate parts of detection and repair:

• Amazon Elastic Kubernetes Service (EKS): Node monitoring and automatic repair are included with Auto Mode, Amazon’s managed compute mode. Other supported Linux configurations require the monitoring agent to be installed and repair enabled separately. Selected critical NVIDIA XID errors, the GPU driver’s fault codes, can trigger repair. Event-only findings do not, including some thermal and power signals. Coverage depends on compute mode, agent configuration, and the reported condition.
• Google Kubernetes Engine (GKE): For supported critical XID errors, GKE marks GPU resources unhealthy and blocks new GPU scheduling on the affected node. It recreates the node when auto-repair is enabled. The provider-managed Autopilot mode always enables auto-repair; Standard enables it by default for new node pools, but it can be disabled. Detection defaults also differ between GKE-managed GPUs and NVIDIA GPU Operator deployments, which use a separate GPU management stack. Non-critical XID errors do not trigger automated action.
• Azure Kubernetes Service (AKS): Node Problem Detector automatically reports GPU faults on pools using AKS-managed GPU drivers; pools created with --gpu-driver none are excluded. That reporting does not directly remediate GPU issues. Operators supply manual action or external automation. Separately, AKS has preconfigured, best-effort generic node auto-repair based on persistent NotReady, the Kubernetes unhealthy-node status, or missing node status. A reported GPU condition is not automatically the same trigger.

Node repair and inference availability are different outcomes. A replacement node still needs a ready replica with model weights loaded. Healthy replicas, inference-aware readiness checks, traffic routing, and enough spare serving capacity determine whether requests continue to meet throughput and latency objectives during repair. Detection can shorten an incident; it does not guarantee action before user impact or preserve in-flight requests. These mechanisms also do not, by themselves, provide continuity when the facility or another shared fault domain fails.

Storage. When a degraded node gets pulled from the serving pool, the replacement replica has to load model weights before it can serve. And the length of that window is a storage problem, not a GPU problem.

Loading a 232.8 GiB model's weights from storage into GPU memory takes 3-5 minutes with a default vLLM loader. During that entire window, the instance can’t serve a single request. Storage affinity tied to pod scheduling keeps model weights local to the nodes serving them, reducing that window. Storage cluster layouts are configured so rebalancing operations don’t compete with live inference reads.

Where the workload benefits from it, GPUDirect Storage creates a direct DMA path from NVMe SSDs to GPU VRAM, bypassing the CPU and system RAM entirely, cutting the window in which the cluster is running short a replica. Storage can also be configured based on the specific workload. Like networking, it should be specified upfront or the retrofit is painful.

Federation for Type 2 resilience

When the datacenter goes down, inference traffic needs somewhere to go. Federation coordinates traffic and workloads across clusters so that an available site can take over. Continued service depends on that destination’s capacity, readiness, dependencies, and latency to users.

Surviving a facility loss requires a serving path outside the failed domain. Federation organizes that path across clusters. Three implementation details determine whether it meets the intended recovery objective.

Pre-loaded model weights on the secondary cluster. For immediate continuity, the secondary needs loaded, ready replicas and sufficient compatible capacity before the primary fails. Cold-loading hundreds of gigabytes is a recovery option with a longer interruption, and its startup time belongs in the recovery objective.

Automated cutover with load redistribution. Routing needs to detect failure and direct traffic to healthy destinations with capacity. The cutover path and its dependencies must remain available outside the failed domain. A manual runbook adds response time, especially during a 3am facility failure.

True geographic distribution across fault domains. Sites in the same metro can share power, peering, or fiber paths. Separate locations need independently available dependencies as well as physical distance. Building in federation from day one is valuable because capacity, routing, and recovery behavior need to be designed together before an incident.

Managed cloud platforms also provide building blocks for this layer. Google previewed its multi-cluster GKE Inference Gateway to route inference across clusters and regions. AKS reliability guidance separately describes zone resilience and regional continuity. These capabilities still require a workload deployment, dependencies, routing, and capacity plan that can survive the intended failure. Node auto-repair does not guarantee sufficient surviving serving capacity or replace cross-facility capacity and cutover planning.

Implications for SLA Design and Platform Selection

The SLA you’re actually committing to

The service-level agreement (SLA) a provider can credibly commit to is bounded by the architecture underneath. A single datacenter deployment has no serving path when that facility goes down, and contract language cannot supply one. Federation can close that architectural gap when surviving sites have the capacity, ready replicas, and independent dependencies to meet the promised service objective. A control-plane SLA, a node-repair capability, and an inference-availability commitment describe different things.

What managed Kubernetes providers actually commit to makes this concrete. EKS, GKE, and AKS publish control plane SLAs from 99.5% to 99.99%, depending on tier and configuration. 99.5% uptime allows 43.8 hours of downtime per year, while 99.99% allows 52 minutes, making this a relatively large range. None of them are specified against your workload. The architecture is what determines which number a provider can credibly deliver, with the comparison below breaking down how each model stacks up.

How GPU inference deployment models compare on reliability

For teams that want workload-specific design, deployment, and end-to-end operations, we recommend our fully managed clusters delivery. Aranya OS is our proprietary multicluster operating system for operations within and across clusters. It builds on clusterdOS, our open-source, Kubernetes-based engine inside each cluster, to federate clusters into one fleet. The engine also extends to virtual-machine and Slurm batch-scheduling environments.

clusterdOS detects and diagnoses infrastructure faults, including GPU thermal events, error-correcting code (ECC) memory errors, and networking faults, and automates supported remediation to reduce downtime. That operational knowledge compounds as we operate more clusters. The Aranya entry describes our fully managed clusters delivery; customers can also operate our self-managed software (currently in private alpha). Available provider features are distinct from the application-level commitments in an engagement.

Capability: Kubernetes control plane

• Self-managed Kubernetes: You manage it
• Managed Kubernetes (EKS/GKE/AKS): Provider-managed (99.5–99.99% SLA)
• Neoclouds: 99.9% where published; several publish none
• Aranya, fully managed clusters: Aranya-managed with 99.99% SLA

Capability: GPU node detection and repair

• Self-managed Kubernetes: You configure monitoring and repair automation
• Managed Kubernetes (EKS/GKE/AKS): EKS/GKE: automated for covered faults with required configuration. AKS: GPU reporting plus separate generic node auto-repair
• Neoclouds: Provider-handled at varying depth
• Aranya, fully managed clusters: Automated detection and remediation for supported faults; serving availability also depends on replicas and capacity

Capability: Networking architecture

• Self-managed Kubernetes: You design it
• Managed Kubernetes (EKS/GKE/AKS): Data center defaults, outside SLA
• Neoclouds: Provider-defined
• Aranya, fully managed clusters: Specified by Aranya before provisioning

Capability: Storage configuration

• Self-managed Kubernetes: You design it
• Managed Kubernetes (EKS/GKE/AKS): Data center defaults, outside SLA
• Neoclouds: Provider-defined
• Aranya, fully managed clusters: Specified per workload before provisioning

Capability: Intra-cluster reliability incidents

• Self-managed Kubernetes: You own them
• Managed Kubernetes (EKS/GKE/AKS): Provider handles covered infrastructure repair; you integrate workload recovery and own uncovered incidents
• Neoclouds: Shared model
• Aranya, fully managed clusters: Part of the engagement

Capability: Multi-datacenter federation

• Self-managed Kubernetes: You build it
• Managed Kubernetes (EKS/GKE/AKS): Resilience tooling available; cross-cluster deployment, capacity, and cutover require design and configuration
• Neoclouds: Not standardized
• Aranya, fully managed clusters: Built in from day one; failover capacity and cutover designed for the workload

Capability: Time from bare metal to production-ready

• Self-managed Kubernetes: Months
• Managed Kubernetes (EKS/GKE/AKS): Days to weeks
• Neoclouds: Varies widely
• Aranya, fully managed clusters: Within 48 hours once hardware is racked and ready

Capability: Incident escalation path

• Self-managed Kubernetes: Your team
• Managed Kubernetes (EKS/GKE/AKS): Support ticket
• Neoclouds: Support ticket / shared site reliability engineering team
• Aranya, fully managed clusters: Slack channel and on-call

‍

Building for When, Not If

The market is still in a phase where speed wins. That won’t change soon, and it probably shouldn’t, because the teams shipping fast are learning the most. But infrastructure decisions compound in ways that aren’t obvious until twelve months later, when inference has become the critical path and the architecture from month one is what the team is stuck with.

Four principles guide inference-grade infrastructure, including deployments built on managed Kubernetes:

1. Networking, observability, and storage are workload-specific design decisions.
2. Type 1 and 2 failures require different architectural responses.
3. Federation is bounded by the complexity of making clusters across facilities behave as one logical fleet.
4. The SLA a provider can commit to is bounded by the architecture beneath it, and the number in the document is a ceiling not a floor.

From what we’ve seen operating this stack across leading inference companies, the teams that get this right do so before production traffic arrives, not after the first incident.

FAQ

How do you run AI inference on Kubernetes reliably in production?

In our fully managed clusters delivery, we design and operate infrastructure tailored for production AI inference, including networking, storage, GPU health monitoring, and remediation. Reliable serving also needs healthy replicas, loaded models, inference-aware readiness and routing, and capacity at the required latency. EKS and GKE can automatically detect and repair selected GPU faults in supported configurations. AKS reports GPU problems and separately provides generic node auto-repair; its GPU reporting does not directly remediate those faults. None of these node mechanisms alone establishes application availability or facility continuity.

What are the alternatives to generic managed Kubernetes for AI inference workloads?

Our fully managed clusters delivery is for teams that need workload-specific infrastructure designed and operated end-to-end. Customers that want to operate Aranya OS themselves can use our self-managed software (currently in private alpha).

The other categories are managed Kubernetes from hyperscalers, with provider- and configuration-specific GPU monitoring and repair; neoclouds, which operate GPU fleets and offer managed services at varying depth; and self-managed Kubernetes, where the team integrates and operates every layer. Available automation and the scope of operational ownership differ from an end-to-end application-availability commitment.

How do AI inference companies handle datacenter outages?

We build multi-datacenter federation into our fully managed clusters architecture from day one. For immediate continuity, traffic needs an independently available destination with loaded models and enough compatible capacity for its own demand plus redirected traffic at acceptable latency. Cloud platforms provide multi-zone and multi-region resilience mechanisms, but the application’s placement, dependencies, capacity, and cutover behavior still need to be configured. Node auto-repair is a separate, in-cluster mechanism. Federation can use existing paid headroom when that capacity budget works; it does not make failover cost-free or guarantee that two similarly utilized sites can cover each other.

How long does it take to set up a GPU Kubernetes cluster?

We get you from bare metal or VMs to a custom production-ready GPU inference cluster within 48 hours, starting with GPU, CPU, and storage hardware already racked and ready. Comparable bare-metal Kubernetes setups typically take weeks. Most of those weeks are spent on networking specification, GPU operator integration, observability tooling, and isolation testing. The speed comes from operational repetition: Aranya has built the same architecture across customers, so deployment is configuration rather than design from scratch.