InfiniBand vs Ethernet: choose a GPU fabric you can operate

Choosing between InfiniBand and Ethernet means assigning ownership of topology, congestion control, software compatibility, and recovery. Datacenter and neocloud operators need acceptance criteria tied to the customer's workload before they sell capacity. The useful comparison starts with the traffic GPUs exchange and ends with a configuration the team can operate.

Separate Ethernet/TCP, RoCE, and native InfiniBand

Ethernet describes a network technology rather than a single application data path. Comparing ordinary Ethernet sockets with InfiniBand leaves out an important option: Ethernet engineered for remote direct memory access, or RDMA. RDMA lets compatible adapters transfer data directly between registered memory regions, reducing copying and operating-system involvement after setup.

There are three practical candidates:

• Ethernet/TCP: Applications communicate through the host networking stack using Transmission Control Protocol. This is a candidate for independent inference replicas, service traffic, and workloads whose cross-node communication fits the available capacity and CPU budget.
• Ethernet/RoCEv2: RDMA over Converged Ethernet uses compatible network adapters and an engineered Ethernet fabric. RoCEv2 carries RDMA over User Datagram Protocol and Internet Protocol (UDP/IP), allowing routing between IP subnets. It is a separate data path from TCP sockets.
• Native InfiniBand: A fabric with native RDMA, credit-based link flow control, and a subnet manager, the service that discovers devices and programs paths through the network. Its operating model differs from an Ethernet/IP fabric.

The NVIDIA RDMA-Aware Networks manual v1.7 describes the memory-transfer model, InfiniBand's flow control, and subnet management. RoCEv2's routing extension adds IP-based routing to RoCE.

Our recommendation is to choose the data path against a workload target and an operating plan. We design workload-specific clusters around explicit hardware and networking requirements. Our clusterdOS engine is the open-source engine inside each cluster. It is Kubernetes-based. Kubernetes orchestrates containerized workloads; clusterdOS also extends to virtual-machine and Slurm batch-scheduling environments. Fabric compatibility remains a configuration-specific requirement.

InfiniBand vs Ethernet decision table

Use the workload's communication pattern to narrow the candidates. Node count alone is a poor selection rule: a small job can synchronize frequently, while a large fleet of independent replicas can exchange little GPU data between servers.

Decision factor: Workload traffic

• Ethernet/TCP: Candidate for independent inference replicas, request routing, and modest inter-node transfers. Distributed jobs still need workload qualification.
• Ethernet/RoCEv2: Candidate for communication-heavy training or distributed inference where Ethernet integration matters.
• Native InfiniBand: Candidate for tightly coupled training and high-performance computing where native RDMA and the InfiniBand operating model fit.

Decision factor: RDMA and host overhead

• Ethernet/TCP: Ordinary sockets do not establish an RDMA path. Account for host CPU use and memory copies.
• Ethernet/RoCEv2: Requires compatible RDMA adapters, drivers, and runtime support. GPU-memory transfers add their own compatibility requirements.
• Native InfiniBand: Native RDMA still requires a working adapter, driver, runtime, and GPU-memory path.

Decision factor: Latency and throughput

• Ethernet/TCP: Qualify application throughput and tail latency with the expected CPU load.
• Ethernet/RoCEv2: Qualify small-message completion time, sustained bandwidth, and congestion behavior together.
• Native InfiniBand: Qualify the same metrics. Native flow control does not establish a workload latency guarantee.

Decision factor: Topology

• Ethernet/TCP: Routed leaf-spine can serve the workload, provided shared links have sufficient capacity.
• Ethernet/RoCEv2: Size cross-rack capacity and validate multipath behavior for the selected adapters and runtime.
• Native InfiniBand: Size the fabric and select routing appropriate to its topology and software.

Decision factor: Congestion

• Ethernet/TCP: TCP's congestion response does not remove queueing or shared-link bottlenecks.
• Ethernet/RoCEv2: Requires an end-to-end congestion design, including traffic classification and the supported lossless or lossy mode.
• Native InfiniBand: Credit-based flow control protects buffers from overflow; congested paths can still stall traffic.

Decision factor: Staffing

• Ethernet/TCP: Linux/IP operations, host performance, and application observability.
• Ethernet/RoCEv2: Those skills plus RDMA, queue and buffer configuration, congestion telemetry, and GPU-adapter diagnosis.
• Native InfiniBand: InfiniBand subnet management, routing, fabric diagnostics, and GPU-adapter diagnosis.

Decision factor: Acceptance gate

• Ethernet/TCP: Workload target under co-traffic, with acceptable CPU cost and recovery behavior.
• Ethernet/RoCEv2: Intended RDMA path plus cross-rack collectives, congestion qualification, and recovery.
• Native InfiniBand: Intended RDMA path plus topology/routing qualification, collectives, and recovery.

Ethernet/TCP is a sensible starting candidate when GPU communication stays within each server. For frequent cross-server synchronization, put engineered RoCE and InfiniBand into the qualification plan. Either RDMA option can be the wrong purchase if its operating requirements exceed the team's capabilities.

Favor RoCE when retaining Ethernet/IP operations is valuable and the team can own end-to-end RDMA congestion control. Favor InfiniBand when a dedicated compute fabric fits the service and the team can own its subnet management and diagnostics. If both pass the workload targets, compare full operating cost and recovery responsibility.

Match the fabric to the traffic, not the workload label

Training needs more than a large-message bandwidth result

Collectives are coordinated operations across participating GPU processes. All-reduce combines values and returns the result to every participant. All-gather distributes each participant's contribution to the others. All-to-all exchanges different data between participants.

The mix matters. Replicated training workers often exchange gradients, while models split across GPUs exchange other model state or intermediate results. Expert-parallel models, which distribute specialist model components across GPUs, can generate substantial all-to-all traffic. A qualification plan centered only on all-reduce can miss the customer's limiting operation.

Separate bandwidth from exposed communication time. Compute can overlap some transfers; other transfers sit directly on the critical path. The commercial result is training-step time and useful work per GPU-hour, including the slow steps.

Inference can be local or communication-heavy

Independent replicas that each fit within one server have different requirements from a model split across servers. Tensor parallelism divides model calculations across GPUs and can require frequent communication during token generation. Disaggregated serving separates prompt processing from token generation and introduces transfers of cached attention state between those stages. The DistServe research makes cluster bandwidth part of how those stages are placed.

For distributed serving, measure time to first token, time between generated tokens, and throughput at the intended request mix. Include the 99th percentile, or p99, which captures the boundary of the slowest 1% of observations. An unloaded adapter latency figure cannot predict those service metrics.

Client requests and GPU-to-GPU transfers also need separate capacity budgets. Our discussion of production inference failures explains why cluster health and service health need separate attention.

Topology can erase a port-speed advantage

A leaf switch connects servers; spine switches connect the leaves. In either Ethernet or InfiniBand, the capacity leaving a leaf can constrain a distributed job even when every server port runs at its advertised rate.

Illustrative capacity calculation: eight servers with one 400 Gb/s fabric port each present 3.2 Tb/s of server-facing capacity. Four 400 Gb/s uplinks provide 1.6 Tb/s leaving that leaf, a 2:1 downlink-to-uplink ratio. If all eight servers send across those uplinks simultaneously, the uplink budget is shared. This is capacity arithmetic, not a measured throughput result; protocol overhead, routing, and traffic distribution reduce usable payload capacity further.

Keeping a job within one leaf can avoid that particular uplink bottleneck. Selling larger allocations that cross leaves changes the performance envelope. Job placement and the cluster scheduler therefore belong in the fabric design.

An acceptance result should distinguish same-leaf traffic from cross-leaf traffic and cover every intended adapter path. Two-node testing cannot establish full-cluster bisection capacity, the bandwidth available between halves of a cluster.

The host path matters too. A network interface card, or NIC, reaches GPUs through the server's PCI Express interconnect and, on some platforms, other internal links. GPUDirect RDMA allows compatible NICs to access GPU memory directly. Working host-memory RDMA does not establish that GPU-memory transfers work. NVIDIA's GPU troubleshooting guidance identifies topology, drivers, container configuration, and memory-access setup as relevant dependencies.

Congestion control is an operating responsibility

RoCE needs an explicit congestion profile

Two mechanisms often appear in RoCE deployments:

• Explicit Congestion Notification, or ECN: Switches mark packets when queues build. In RoCE, receivers send congestion notification packets (CNP) so senders can reduce their rate.
• Priority Flow Control, or PFC: A receiving device can pause a selected traffic priority on a link to protect buffers. Pausing can also delay other traffic assigned to that priority.

PFC and ECN have different jobs. A lossless profile commonly uses both; some supported RoCE configurations use ECN without PFC. For example, the Cumulus Linux 5.18 documentation distinguishes lossless PFC-plus-ECN and lossy ECN modes. That is implementation-specific guidance, not a configuration recipe for every Ethernet switch.

The design needs consistent classification, queue mapping, buffer allocation, and endpoint response across the selected hardware. Enabling PFC everywhere is not an acceptance criterion. Neither is requiring every congestion-related counter to remain zero: marking can be evidence that the control loop is working.

InfiniBand still needs congestion and failure diagnosis

Credits prevent a sender from overrunning a downstream link buffer. They do not create capacity on an oversubscribed path. Persistent waiting, imbalanced routes, link degradation, and poor placement can still reduce useful work.

The subnet manager and routing configuration need named owners, configuration backups, and an appropriate availability plan. Features such as adaptive routing, which can select among available paths, depend on the actual switch and software combination. They should never be assumed from the protocol name.

For shared infrastructure, distinguish performance controls from security boundaries. Traffic priorities and bandwidth allocation alone do not establish separation of customer workloads.

Qualify the fabric with an acceptance contract

We recommend a staged acceptance contract with a saved configuration and raw results. Each stage answers a different question; a pass at one stage does not replace the next.

Before a comparison, fix the GPU model and count, server topology, NIC model and ports per GPU, negotiated link rates, cabling, switch topology, and oversubscription. Record the exact operating-system/kernel, GPU driver, CUDA GPU software runtime, NIC firmware/driver, switch software, communication-library, and benchmark versions or commits. Include container device access and the scheduler's placement policy.

For the NVIDIA examples here, NCCL 2.32.3 is the documentation baseline. The NVIDIA Collective Communication Library, NCCL, implements GPU collective operations. Its network diagnostics cover link state, bandwidth, latency, RDMA counters, and fabric-specific diagnosis. A different installed version or GPU software stack needs its corresponding supported tooling.

Acceptance stage: 1. Inventory and host path

• Exercise and evidence: Capture GPU/NIC topology, link state and rate, configuration, and initial error counters. Exercise the intended GPU-memory path.
• Pass condition to agree before delivery: Every allocated device and path matches the specification; no unexpected downshift or unexplained physical-link errors.

Acceptance stage: 2. Point-to-point transport

• Exercise and evidence: For RDMA, use ib_write_bw and latency tools across same-leaf and cross-leaf pairs. Compare host-memory and supported GPU-memory runs. For TCP, measure throughput, latency, and CPU consumption.
• Pass condition to agree before delivery: Results meet separate small-message and large-message targets for the named hardware, paths, and traffic conditions.

Acceptance stage: 3. Collectives and correctness

• Exercise and evidence: Run nccl-tests operations matching the workload, first across two nodes, then across the full sellable allocation. Save correctness results, transport logs, and per-port traffic.
• Pass condition to agree before delivery: Zero reported correctness errors; intended transport and adapters in use; operation times within the qualified envelope.

Acceptance stage: 4. Contention and placement

• Exercise and evidence: Repeat with concurrent jobs, synchronized many-to-one bursts, checkpoint/storage traffic, and realistic cross-rack placement. Capture queue, pause, notification, retry, and drop-counter changes.
• Pass condition to agree before delivery: Workload throughput and tail latency remain within the agreed limits at the declared co-traffic load.

Acceptance stage: 5. Workload soak

• Exercise and evidence: Run the actual model and runtime at specified precision, batch size, sequence-length distribution, and concurrency. Include representative training or serving cycles.
• Pass condition to agree before delivery: Customer throughput and latency targets hold for the agreed duration, without unexplained stalls or accumulated errors.

Acceptance stage: 6. Failure and return to service

• Exercise and evidence: In a maintenance environment, remove a redundant link or path, exercise relevant control-service failover, and then restore it. Re-run the workload and record recovery.
• Pass condition to agree before delivery: Alarms, isolation, job failure/restart behavior, and return to the performance envelope match the recovery plan.

The upstream RDMA performance tools explicitly distinguish microbenchmarks from real application traffic. Their results diagnose a path; they do not substitute for workload qualification.

Make the pass criteria measurable

Illustrative acceptance contract, not a benchmark result: for a 16-node allocation with eight GPUs per node and one 400 Gb/s fabric port per GPU, require the named workload to retain at least 90% of its isolated-baseline throughput during a declared storage-transfer load. The model, GPU/NIC models, software manifest, topology, storage load, and measurement duration are fixed contract fields. For serving, add the customer's p99 first-token and inter-token limits. The 90% threshold is a negotiable example, not an industry standard or a predicted result.

Correctness errors, an unintended transport fallback, or recovery outside the agreed limit remain failures even if average throughput passes. A restored network link also does not guarantee that a running distributed job survives; connection recovery and application restart are separate obligations.

Confirm what the benchmark actually measured

NCCL can select sockets, its internal verbs transport, or external plugins. Verbs are a software interface for RDMA operations. On Linux, the internal names in NCCL 2.32.3 are Socket and IB; the latter uses generic RDMA verbs and is not proof of a native InfiniBand link. The transport selection controls make that distinction explicit. Retain logs and correlate them with the physical link mode and port traffic.

Also distinguish algbw, the operation's data size divided by elapsed time, from busbw, a collective-specific normalization. NCCL's bandwidth definitions show why these values differ. A multi-GPU result includes internal communication and is not a direct reading of one network port. Compare like-for-like operations, message sizes, participants, algorithms, and topology.

Where Ultra Ethernet fits

Ultra Ethernet introduces the Ultra Ethernet Transport, UET, alongside other networking changes. The consortium's 1.1 specification announcement describes transport and application-compatibility enhancements. Treat a proposed UET configuration as its own end-to-end qualification candidate. An Ethernet port or a standards label alone does not establish adapter, driver, runtime, and application support for that transport.

Compare operating cost and ownership

Our cost philosophy is that infrastructure should make compute cheaper, not just better. For fabric selection, the useful denominator is GPU-hours that meet the customer's performance target.

Compare the complete configuration: adapters, switches, optics, cables, spares, software/support, power, and operational staffing. Add time spent tuning or recovering the fabric and GPU capacity stranded by slow jobs. Ethernet's broader ecosystem does not guarantee a lower installed or operating cost for an engineered RDMA fabric. InfiniBand's integrated operating model does not remove staffing requirements.

Assign responsibility across three layers:

1. Physical fabric: Cabling, link health, switch configuration, routing, and congestion controls.
2. Cluster software: Drivers, device access, communication libraries, placement, telemetry, and workload recovery.
3. Service delivery: Customer acceptance targets, admission limits, and incident ownership.

Aranya OS is our proprietary multicluster operating system. It builds on clusterdOS to federate clusters into one fleet, operating both within and across clusters. We deliver it through fully managed clusters, in which we handle architecture, deployment, and end-to-end operations, and self-managed software operated by the customer (currently in private alpha). Neither delivery mode establishes support for every GPU, NIC, firmware, and fabric combination; that scope belongs in the agreed cluster specification.

The fabric decision is complete when a named configuration meets the workload target under contention, its recovery behavior is understood, and someone owns the whole path. Talk to an engineer about a workload-specific cluster specification, including the fabric requirements and cluster-layer operating responsibilities.