Why GPU Inference Infrastructure Fails in Production and How to Architect Around It
Why generic managed Kubernetes is not sufficient for production inference
Most inference companies buy managed Kubernetes assuming any managed offering is suitable for their workload. And while Kubernetes itself is the right orchestration primitive for inference, generic managed Kubernetes is not the right product.
Teams aren't careless about this. They're being rational under constraints. GPU demand is outpacing supply, time-to-market concerns live at the forefront, and engineering capacity is limited. As many teams scale beyond their first cluster, they do account for a backup ISP, a secondary region, and a documented failover path. But ‘redundancy’ in this form is unfederated, unautomated, and untested. Each backup is itself a point of failure that won’t surface until the primary fails.
By that point, the failure has already happened, and it falls into one of two buckets. Either something broke inside the datacenter, or the whole datacenter went down. The two get planned for as if they are the same problem. The thing is: they aren't.
The Two Ways GPU Inference Infrastructure Fails
The first is everything that can go wrong inside a functioning datacenter. Think hardware degradation, networking failures, inference engine issues. The second is the datacenter itself going down.
Type 1: What breaks inside a GPU inference cluster
Production inference incidents rarely start with a hardware crash. They present as silent degradation: recurring timeouts at the engine layer, replicas serving requests insufficiently, or latency drift without an observable root cause.
A Microsoft study of 156 high-severity production LLM inference incidents found roughly 60% were inference engine failures, and within that category, about 40% were timeouts. The cluster is up, the orchestrator is healthy, but inference is not.
Load balancing is a quieter culprit, but responsible for more than it should be. Round-robin routing across GPU replicas forces each replica to recompute work already done elsewhere. That’s a 6,000-token system prompt reprocessed from scratch every time a request lands on the wrong replica. In a benchmark simulating a multi-tenant workload on 16 H100 GPUs, switching from round-robin to prefix-cache-aware routing cut P90 time-to-first-token from over 90 seconds to 0.54 seconds and roughly doubled throughput. More GPUs doesn't fix this. It actually scales the problem.
Networking is the layer that is rarely specified upfront. But it compounds everything else. On multi-tenant GPU clusters, the network fabric is typically shared even when the GPUs are physically dedicated, meaning the same model, same query, same batch size, on the same cluster, produce meaningfully different p99 numbers day to day, sometimes hour to hour. P99 latency at 450ms one day and 900ms the next is not a software problem with a software fix. Most clusters get deployed with whatever networking configuration the datacenter provides by default. The default that's tuned for generic compute is rarely tuned for inference. The gap doesn't surface during setup. It surfaces under load, either as p99 variance, throughput degradation, or latency that can't be traced back to the application.
Type 2: What happens when the datacenter goes down
Type 2 failure modes are external to the cluster. When the whole datacenter experiences a failure, it’s actually simpler than the first type. The ISP drops, power fails, or the hyperscaler has a regional event, and everything inside the cluster becomes irrelevant.
AWS had at least one documented outage every month in 2025 except the final two. The October event: a race condition in DynamoDB’s automated DNS management ran for roughly 15 hours, and took Slack, Atlassian, and Snapchat with it. 197 SaaS providers publicly attributed their own outages to it. Not only is us-east-1 the busiest AWS region in the world, but it also happens to be where the most outages were recorded in 2025.
A single datacenter inference deployment has no answer to this. The facility goes down, inference goes with it.
Why the Single Datacenter Default Makes Sense… Until It Doesn’t
Deploying on a single datacenter is often the right call early on. GPU supply is still constrained, time to market matters, and the architecture review window doesn’t clearly open.
Once teams begin to scale, they add clusters. The most common pattern is spinning up a secondary cluster in the same facility as the first: more capacity, same datacenter. The catch is, if that facility fails, every cluster inside goes down with it.
Even when teams do account for this and scale in different datacenters, federation is still an afterthought in many cases. But without it, clusters in different facilities are operationally independent. When a facility goes down, the others keep running their own load. There’s no routing mechanism and the affected traffic has nowhere to go.
Utilization and the cost case for federation
The standard objection to federation isn’t cost; it’s the engineering complexity of stitching clusters together across facilities. Routing, state, model weights, and failover behavior all have to coordinate across facilities, and most inference teams haven’t built that before. Any concerns around cost dissolve under closer inspection of utilization data.
Production inference clusters are bursty by design: short, GPU-intensive forward passes followed by idle time between requests. Continuous batching typically lifts GPU utilization out of the 15 to 30% range and into the 60 to 80% range at normal traffic patterns. Even at the top of that band, there is meaningful headroom across off-peak windows because the cluster is rarely at capacity.
A secondary datacenter at similar utilization absorbs redirected load using capacity that’s already paid for. The question stops being “can we afford redundancy” and becomes whether the spare capacity you’re already running is organized into something useful or just sitting idle.
Production-Grade Architecture for Each Failure Class
In-cluster reliability
Hardening an inference cluster against Type 1 failures comes down to three design layers: networking, observability, and storage. Each has to be specified rather than defaulted. Datacenters don’t default these in ways that match inference workloads, and generic managed Kubernetes doesn’t either.
Networking. The unexplained p99 variance and throughput degradation described above mostly trace back to how the fabric was (or wasn’t) configured at provisioning. That’s a fabric specification decision, not a software problem. What the right specification looks like depends on the workload, but the throughline is that serving thousands of concurrent users on shared, distributed GPU infrastructure requires high performance networking that a generic compute default doesn’t provide.
This is a necessary gap to close before provisioning begins. Based on the specific workload (inference type, model architecture, traffic pattern, etc.), networking architecture should be specified as an explicit requirement rather than accepting the datacenter's default. It's the most common source of Type 1 failures in the clusters we've operated, and the most disruptive layer to change after the fact.
We've seen networking-only changes reduce a category of recurring incidents substantially in engagements where nothing in the application stack was touched. The network is the most consequential reliability layer once it's specified correctly, and also the most consequential gap when it isn't.
Observability. Most of the failure modes above sit at the inference-engine layer, the dominant incident category in the Microsoft study. Standard Kubernetes health signals like pod status or node readiness don’t surface them. Production-grade observability for inference adds GPU-level degradation detection, proactive alerting on node and cluster health, and automated remediation. When a GPU node starts degrading, the right observability allows it to be caught and pulled from the serving pool before it affects throughput. The difference between catching that automatically versus catching it through a user complaint tends to be measured in hours.
Storage. When a degraded node gets pulled from the serving pool, the replacement replica has to load model weights before it can serve. And the length of that window is a storage problem, not a GPU problem.
Loading a 232.8 GiB model's weights from storage into GPU memory takes 3-5 minutes with a default vLLM loader. During that entire window, the instance can’t serve a single request. Storage affinity tied to pod scheduling keeps model weights local to the nodes serving them, reducing that window. Storage cluster layouts are configured so rebalancing operations don’t compete with live inference reads.
Where the workload benefits from it, GPUDirect Storage creates a direct DMA path from NVMe SSDs to GPU VRAM, bypassing the CPU and system RAM entirely, cutting the window in which the cluster is running short a replica. Storage can also be configured based on the specific workload. Like networking, it should be specified upfront or the retrofit is painful.
Federation for Type 2 resilience
When the datacenter goes down, inference traffic needs somewhere to go. Federation is what makes that possible: traffic reroutes to a secondary cluster, automatically, and the product keeps serving.
Cross-facility federation is the only architectural pattern that addresses Type 2 failures. Three implementation details determine whether it actually works.
Pre-loaded model weights on the secondary cluster. Cold-loading hundreds of gigabytes during an incident isn’t failover, but a planned outage. The secondary has to be serving-ready before the primary fails.
Automated cutover with load redistribution. A manual runbook doesn’t account for a 3am facility failure. Routing has to happen automatically.
True geographic distribution across fault domains. Two clusters in the same metro share too many failure modes: power, peering, fiber paths. Building in federation from day one is valuable. A cluster that was never designed to redistribute load can’t be made to do so after the fact. The architectural decision has to happen before production traffic arrives.
Earlier this year, Google previewed its multi-cluster GKE Inference Gateway, specifically citing availability risks from regional outages and the scalability limits of single cluster architectures. A hyperscaler building a dedicated capability for a problem validates that the problem is well-defined.
Implications for SLA Design and Platform Selection
The SLA you’re actually committing to
The SLA a provider can credibly commit to is bounded by the architecture underneath. A single datacenter deployment cannot underwrite a 99.99% uptime under failures at the facility level: the architecture has no mechanism to respond when the facility goes down, and contract language can’t paper over that. Federation closes it. Without federation, the gap remains regardless of what the document says.
What managed Kubernetes providers actually commit to makes this concrete. EKS, GKE, and AKS publish control plane SLAs from 99.5% to 99.99%, depending on tier and configuration. 99.5% uptime allows 43.8 hours of downtime per year, while 99.99% allows 52 minutes, making this a relatively large range. None of them are specified against your workload. The architecture is what determines which number a provider can credibly deliver, with the table below breaking down how each model stacks up.
How GPU inference deployment models compare on reliability
Building for When, Not If
The market is still in a phase where speed wins. That won’t change soon, and it probably shouldn’t, because the teams shipping fast are learning the most. But infrastructure decisions compound in ways that aren’t obvious until twelve months later, when inference has become the critical path and the architecture from month one is what the team is stuck with.
Four principles separate inference-grade infrastructure from what generic managed Kubernetes delivers:
- Networking, observability, and storage are workload-specific design decisions.
- Type 1 and 2 failures require different architectural responses.
- Federation is bounded by the complexity of making clusters across facilities behave as one logical fleet.
- The SLA a provider can commit to is bounded by the architecture beneath it, and the number in the document is a ceiling not a floor.
From what we’ve seen operating this stack across leading inference companies, the teams that get this right do so before production traffic arrives, not after the first incident.
FAQ
How do you run AI inference on Kubernetes reliably in production?
Aranya designs and operates Kubernetes infrastructure tailored for production AI inference. The reliability lives at two layers that generic managed Kubernetes services don't cover: in-cluster (GPU health monitoring, networking architecture, automated remediation) and across datacenters (federated failover for facility-level outages). Generic managed Kubernetes (EKS, GKE, AKS) manages the control plane and basic node health, but leaves GPU-specific failure modes, inference-aware scheduling, and facility-level failure to you.
What are the alternatives to generic managed Kubernetes for AI inference workloads?
Aranya is for teams that need infrastructure designed and operated end-to-end. The other categories: generic managed Kubernetes from the hyperscalers (EKS, GKE, AKS), which manages the control plane and basic node health; neoclouds, which operate their own GPU fleets and offer managed Kubernetes at varying depth, though scheduling across datacenters is rarely a standard offering; and self-managed Kubernetes, where the team owns every layer.
How do AI inference companies handle datacenter outages?
Aranya handles datacenter outages by building multi-datacenter federation into clusters from day one. When one facility fails, traffic reroutes to a secondary cluster that's already serving-ready, with model weights pre-loaded, since cold-loading hundreds of gigabytes during an incident isn't failover. Generic managed Kubernetes services (EKS, GKE, AKS) don't address this layer at all. Neoclouds treat multi-region as a customer-architected decision rather than a standard offering.
How long does it take to set up a GPU Kubernetes cluster?
Aranya gets you from bare metal or VMs to a custom production-ready GPU inference cluster within 48 hours. Comparable bare-metal Kubernetes setups typically take weeks. Most of those weeks are spent on networking specification, GPU operator integration, observability tooling, and isolation testing. The speed comes from operational repetition: Aranya has built the same architecture across customers, so deployment is configuration rather than design from scratch.
