By Nayan Lad | Product Manager for AI Data Center | Qualcomm
Platform teams running AI inference in production often run into the same puzzle. The same model, on the same type of hardware, performs differently from one deployment to the next. Latency moves around, utilization stays below expectations, and power costs climb.
Sometimes the bottleneck is not the model, the driver, or the accelerator itself. It is where the workload was placed relative to the CPUs, memory, network interfaces, and other accelerators it depends on. Kubernetes found the resources the workload asked for, but it couldn’t see how those resources were connected.
As AI moves into production, infrastructure has to deliver more within fixed limits on power, cooling and cost. That makes performance per watt a central measure of efficiency. Hardware sets the ceiling. How close a cluster gets to it depends heavily on placement, and placement depends on something the industry hasn’t fully standardized yet: how devices describe where they sit.
Available is not the same as suitable
Traditional Kubernetes scheduling answers one question: Which node has enough of the requested resources?
For most cloud native applications, that’s enough. Stateless services scale horizontally, CPU cores are largely interchangeable, and any node with free capacity is a reasonable choice.
AI workloads break that assumption. Two accelerators that look identical in a resource inventory may sit behind different PCIe switches or belong to different NUMA domains. A model split across devices exchanges data constantly. If those devices sit on opposite sides of a NUMA boundary, every exchange crosses the CPU interconnect. The placement is valid, but the workload pays for it:
- More communication between devices
- Higher memory-access and network latency
- Lower effective accelerator utilization
- Fragmentation that leaves capacity stranded
- Less predictable performance
The workload still runs. It just takes more time and energy to do the same work.
Resource counting vs. topology awareness
| Resource-Count Scheduling | Topology-Aware Scheduling | |
| Core question | Is capacity available? | Is this capacity well connected for this workload? |
| Best fit | Stateless, loosely coupled services | Communication-heavy AI workloads |
| Strength | Simple, predictable, easy to operate | Can reduce communication overhead and fragmentation |
| Weakness | Blind to how resources are connected | Depends on accurate, consistent topology data |
Topology awareness isn’t about packing everything as tightly as possible. Overloading one domain creates contention and leaves other capacity idle. Good placement balances locality against capacity and overall utilization.
The last row of the table matters most: topology-aware scheduling is only as good as the topology data it receives.
The real problem: every device speaks a different dialect
An inference workload rarely needs just one kind of device. It needs accelerators, CPUs, memory and network interfaces, often managed by different drivers from different vendors.
If each driver describes locality in its own terms, the scheduler can’t answer a basic question: Is this accelerator actually close to this CPU group and this NIC? Each driver may report accurate information, but if the drivers can’t be compared, the scheduler can’t align them. Topology awareness then ends up as vendor-specific logic, custom scheduler extensions or manual node pinning. That approach is fragile, hard to maintain and doesn’t carry over between clusters.
This is why a shared vocabulary for device topology matters more than any single scheduling algorithm.
Kubernetes is normalizing topology
Dynamic Resource Allocation (DRA) provides a framework for describing and allocating devices, and Kubernetes is increasingly standardizing topology attributes that DRA drivers can publish
DRA lets workloads request devices through resource claims instead of simple counts. More importantly, Kubernetes now defines standard device attributes under the resource.kubernetes.io domain. These attributes describe a device’s NUMA node, its PCI bus address and its PCIe root complex. Because their meaning is consistent, the scheduler can place devices from different drivers together without knowing anything about any single driver.
In practice, a workload can ask for devices that share a locality domain, and any driver that publishes the standard attributes can take part.
The CPU DRA driver shows how this works. It discovers a node’s CPU topology, including sockets, NUMA nodes and cache groups, and can publish the standard PCIe root attribute. CPU allocations can then line up with accelerators and network interfaces managed by entirely different drivers. Accelerator drivers are adopting the same attributes, so a workload can request CPUs and accelerators in the same locality domain and get consistent behavior across different hardware.
Topology also matters at other layers. Inside a node, the kubelet’s Topology Manager aligns CPUs, memory and device-plugin resources, but it does not see DRA devices. Across nodes, Kueue’s Topology Aware Scheduling keeps a workload’s pods within the same rack or block.
The work isn’t finished. The community is still discussing more standard attributes and more expressive ways to describe relationships between devices. But the direction is clear: topology should be a platform capability, not a vendor feature.
Measure what matters
Better placement doesn’t improve performance per watt in every case. Results depend on the model, communication patterns, software stack and hardware. What consistent topology data does is let the platform remove waste it can see, and that waste can be measured.
Platform teams should benchmark representative workloads with and without topology constraints and track:
- Throughput and tail latency
- Accelerator utilization
- System-level power consumption
- Energy per request or per generated token
If topology-aware placement reduces overhead or raises utilization, the cluster does more useful work within the same power budget, without adding hardware.
A call to the ecosystem
Kubernetes was built on the idea that resources are largely interchangeable. AI infrastructure isn’t. As clusters add more varied accelerators, CPUs, memory configurations and interconnects, the platform needs more than a count of free devices. It needs to know how those devices relate to each other, described the same way regardless of who built them.
That work falls to the whole ecosystem. Driver authors can publish the standard topology attributes. Platform teams can build placement policies on them instead of on vendor-specific labels. And the community can keep extending the vocabulary as new hardware appears.
Performance per watt starts with efficient hardware. At scale, it depends on every device speaking the same topology language.
KubeCon + CloudNativeCon is taking place in Salt Lake City, Utah from November 9 to 12. Register Now!
##
ABOUT THE AUTHOR
Nayan Lad is a Product Manager for AI Data Center at Qualcomm, focused on full-stack AI infrastructure. With expertise in product management, product marketing, and go-to-market strategy, he helps enterprises scale AI adoption through efficient infrastructure, cloud-native technologies, and optimized compute resources. Nayan holds a Master of Engineering in Engineering Management from Cornell University and brings a blend of technical depth and business leadership to enterprise AI.
