By Joongi Kim, CTO of Lablup
AI workloads are typically tuned on the assumption that they own the hardware outright. In multi-tenant clusters, how each workload’s devices are aligned determines both how efficiently the hardware is used and whether tenants suffer unpredictable contention.
Modern servers are split into NUMA nodes, each pairing a CPU socket with its local memory, while GPUs and NICs attach through PCIe roots under each socket. Every time data crosses between NUMA nodes or PCIe roots, it loses bandwidth. Alignment keeps devices that talk to each other on the same side of those boundaries.
The cost of getting this wrong is substantial. In a 2025 study on Kubernetes, an NCCL all_gather averaged 46.59 GB/s when each GPU was paired with the NIC on its own PCIe root, but only 29.20 GB/s when GPUs were assigned at random. In worst-case layouts presented at KubeCon NA 2025, with every device on the wrong CPU socket, NCCL bandwidth dropped to 41% of the aligned baseline, host-to-device transfers to 60%, and network throughput to 71%.
This article traces how Kubernetes, through v1.37, has evolved to support topology-aware device alignment. Dynamic Resource Allocation (DRA) lets a workload declare which devices belong together, and PodGroup lets it declare which nodes a distributed job should share. Three gaps remain: CPUs still sit outside DRA claims, every claim must spell out its own fallback, and workloads cannot see the topology they are asked to match. I propose that Kubernetes close them by making alignment a cluster-wide default set by the operator, not a requirement every workload must write for itself.
Throughout, I distinguish three roles. The infrastructure provider owns the hardware and knows its topology. The cluster operator configures Kubernetes on top of it. The workload author writes the pods and claims that run on it.
NUMA alignment for device plugin GPUs
Within a single node, device plugin GPUs can already be aligned automatically. Topology Manager, stable since v1.27, collects NUMA hints from the kubelet’s CPU, memory and device managers and picks an assignment all three can satisfy.
The cluster operator sets alignment strictness per node through topologyManagerPolicy, and can add static CPU and memory manager policies to align CPUs and memory for Guaranteed pods. best-effort prefers alignment but admits pods that cannot be aligned, restricted and single-numa-node reject them, and the default, none, makes no attempt. Every pod on the node inherits that choice.
PCIe alignment for DRA devices
DRA lets a workload pair a GPU with its NIC, but until v1.37 the only standard way to do so was too strict for most servers. A GPU and a NIC on the same PCIe root exchange data without crossing to another root or socket, a pairing the device plugin API cannot express because it reports topology per device.
DRA, generally available since v1.34, lets a claim bind devices with a constraint: a matchAttribute on resource.kubernetes.io/pcieRoot requires the GPU and NIC to report the same root, so the pod either runs aligned or stays unscheduled.
KEP-6072 found that matching on pcieRoot “excludes most GPUs”: on many servers, only one GPU shares a PCIe switch with a NIC. The numaNode attribute, standardized in v1.37, is looser, keeping the pair on the same NUMA node even when their roots differ. Either way, the workload author writes the pairing and must know which attribute the hardware can satisfy.
Fabric alignment for distributed training jobs
Across machines, alignment means keeping every pod of a distributed training job inside one high-bandwidth domain, such as a rack or an NVLink domain that joins 72 GPUs across nodes. A single pod stranded outside it slows every collective, so the whole job pays for one misplaced pod.
PodGroup lets the scheduler place several pods as one unit, and its topology constraints steer that placement by a node label such as topology.example.com/rack.
The scheduler treats each label value as a candidate domain and places the whole group in one. The constraint is a hard requirement: if no single domain can hold the group, the PodGroup stays pending rather than spilling across domains, much like single-numa-node, not best-effort. It has been alpha since v1.36, with GA targeted for v1.39.
Node labels cannot describe every cross-node resource. A multi-node NVLink domain must be set up before pods can use it, and NVIDIA’s DRA driver models this as a ComputeDomain that pods claim through DRA. PodGroup decides which nodes a job shares and DRA decides which devices it may use, but today the workload author must keep the two consistent.
Remaining gaps
Each mechanism splits responsibility differently. Device plugins let the operator set alignment policy for every pod on a node but cannot pair two devices. DRA lets a workload state the pairing precisely but leaves the decision, and its consequences, with the workload author. Three gaps follow.
CPUs outside the DRA claim
The kubelet places a pod’s CPUs while the scheduler places its DRA devices, so a claim that aligns a GPU and a NIC says nothing about where the CPUs land. A CPU DRA driver would bring CPUs into the claim, but none is stable yet, and a node running one would advertise its CPUs twice until KEP-5517, targeted for GA in v1.39, unifies the accounting.
Fallback written into every claim
The workload author must also write any fallback. Prioritized lists, generally available since v1.36, let a request list up to eight alternatives, such as a GPU constrained to the NIC’s pcieRoot followed by an unconstrained one. Every tier repeats the request, quota must cover every alternative, and each workload carries its own copy of what is really a property of the hardware. The cluster operator, who knows that hardware best, cannot state the preference once for the whole cluster.
Topology the workload cannot see
In hosted environments, only the infrastructure provider sees the full topology. Across nodes, vendor-specific labels such as nvidia.com/gpu.clique identify NVLink domains, but their meaning varies by provider. Each workload is left guessing: a claim that matches too strictly stays unschedulable, and one that matches too loosely runs misaligned.
The path forward
In a handful of releases, SIG Node and SIG Scheduling have taken device topology from something the kubelet handled implicitly to something workloads can express precisely. An AI platform can build today on Topology Manager, DRA device pairing and PodGroup placement.
The next step should be a default set once by the cluster operator, extending the choice Topology Manager offers on a single node to the entire cluster. The infrastructure provider would publish topology through standard attributes like pcieRoot and numaNode within a node and standardized keys across nodes. The operator would decide whether alignment is preferred or required, and every job would inherit that choice, writing matchAttribute only when it needs a hard guarantee.
Built on one principle — align when possible, and fail only under an explicit constraint — Kubernetes should become the bridge between the infrastructure provider and cluster operator, who know the hardware, and the workload author, who knows the workload.
KubeCon + CloudNativeCon is taking place in Salt Lake City from November 9 to 12. Register Now!
##
ABOUT THE AUTHOR
Joongi Kim is the creator of Backend.AI and the CTO of Lablup. He earned his Ph.D. in Computer Science from KAIST by creating a GPU-accelerated packet processing framework. His major areas of interest include scalable and automated backend systems, as well as their analysis and design. He’s also a big fan of open source, having contributed to projects like CPython, aio-libs, and many others.
