By Abubakar Siddiq Ango, Senior Developer Advocate at Kubermatic
Kubernetes v1.37, released on 26 August, moved four more Dynamic Resource Allocation (DRA) features to Stable. One of them, extended resource support, lets a DRA driver serve a Pod that asks for example.com/gpu: 1 without any change to the Pod. Operators can therefore move GPU allocation to DRA gradually while existing workloads keep running unmodified.
NVIDIA announced in March that it was donating its GPU DRA driver to the CNCF, and the driver now lives in the kubernetes-sigs organization under SIG Node. AMD published the first official release of its GPU DRA driver in May.
With DRA, Kubernetes decides which device a Pod gets on a node. The platform team decides how many GPUs each team gets and what happens when two teams want the same pool.
Figure 1. What Kubernetes v1.37 standardizes, and what the platform team decides.
Choosing a unit of quota
ResourceQuota supports DRA devices. A namespace can be capped at four devices from a DeviceClass with a key such as examplegpu.deviceclass.resource.k8s.io/devices: 4. When a DeviceClass maps to an extended resource name, the quota for that name counts devices from DRA and from device plugins together, so a team’s quota stays in force while its nodes move from device plugins to DRA.
Both ResourceQuota and Kueue’s default mode count devices, so a small GPU partition and a full GPU each use one unit of quota. Kueue’s documentation notes that this does not reflect what each workload consumes. ResourceQuota also works one namespace at a time. When the quotas across namespaces add up to more than the cluster holds, namespaces can end up competing for the same hardware, and that contention is handled first come, first served. ResourceQuota has no way for one team to borrow capacity another team is not using.
A platform team can make the unit match what a team consumes in two ways. One is to publish a separate DeviceClass for each partition profile and set a device quota on each class. The other is to charge by capacity. Kueue can charge from the counters a DRA driver publishes for partitionable devices, such as the memory of a MIG slice, or from the capacity a workload requests on a shared device.
Where tenants are isolated from each other
A workload can request a shared GPU directly through DRA. How safe that sharing is depends on the mechanism underneath. Multi-Instance GPU (MIG) gives each instance its own path through the GPU’s memory system, with a defined quality of service and fault isolation between clients. Under MPS, a fatal fault in one tenant’s process reaches every other client on that GPU. Workloads that share a GPU through time-slicing get neither memory nor fault isolation. DRA also has its own way to share a device, consumable capacity, which lets unrelated Pods in different namespaces draw on one GPU. Its design document states that Pods referencing the same request share that request’s device resources without any isolation guarantee.
At release v0.5.0, the NVIDIA DRA driver’s README says GPU allocation is not yet officially supported, and the driver’s Helm chart installs with the GPU plugin turned off. Static MIG needs no feature gate. Time-slicing, MPS, dynamic MIG and consumable shares are alpha features, all off by default.
Figure 2. How GPU sharing mechanisms isolate tenants, and their status in the NVIDIA DRA driver v0.5.0.
A simple policy puts hardware partitions between teams and allows software sharing only inside a team. Dynamic MIG is still alpha, so operators create MIG partitions on each node ahead of time, which makes the mix of partition sizes a cluster offers a capacity-planning decision. Runtime limits on software-shared GPUs come from projects outside DRA, and the CNCF post Does Kubernetes DRA Replace HAMi? covers that enforcement layer in detail.
Queueing and fair share
A workload that fits under its quota can still wait a long time for a device. The Kubernetes scheduler cannot preempt a running Pod to free a DRA device, so a high-priority Pod stays Pending until the Pod holding the device finishes or someone deletes it. Gang scheduling through the Workload API reached Beta in 1.37 and ships disabled by default. Distributed training jobs that need all of their Pods at once depend on it or on the all-or-nothing admission Kueue provides.
Kueue, a SIG Scheduling subproject, adds the queueing that ResourceQuota lacks. Each team’s nominal quota sits in a ClusterQueue. Teams in the same cohort can borrow each other’s idle capacity up to a limit, and Kueue’s fair sharing uses preemption to move a cohort toward its weighted shares. Its DRA integration is Beta and on by default for device counts, extended resources and partitionable devices, while capacity-based accounting for shared devices is still alpha. Kueue checks quota when it admits a workload, before the scheduler allocates the actual device, and its documentation describes the gap between those two steps. The integration also requires claims to come from a ResourceClaimTemplate. A Pod that references an existing ResourceClaim, one of the ways several Pods share an allocated device, cannot be admitted.
The Kubernetes AI Conformance program, launched at KubeCon NA 2025, requires a certified platform to run at least one gang scheduling solution and names Kueue and Volcano as examples. Its v1.37 checklist has no requirement on quota, fair sharing or chargeback between tenants.
Metering usage
Kubernetes has no upstream primitive for charging device usage to a tenant. The kubelet PodResources API has reported DRA allocations per Pod as a Stable feature since 1.36, and NVIDIA’s DCGM exporter uses it to label GPU metrics with the claim and namespace that hold each device. ResourceQuota status gives a point-in-time device count per namespace, and Kueue reports how much quota each queue has been granted.
To bill a team, the platform team has to join those records over time, from device to claim to namespace to team, and put a price on them. The billing unit should match the quota unit. If they differ, a team’s bill and its quota measure different things.
What to decide first
DRA gives Kubernetes a standard API for devices, and 1.37 lets existing GPU workloads move onto it unchanged. Before a platform team opens a GPU cluster to several teams, it should decide whether quota counts devices or capacity, and which sharing methods it allows between teams and which only within one team. It should also decide how much idle capacity one team may borrow from another, and which metrics each team’s bill is calculated from.
KubeCon + CloudNativeCon is taking place in Salt Lake City, Utah from November 9 to 12. Register Now!
##
ABOUT THE AUTHOR
Abubakar Siddiq Ango is a Senior Developer Advocate at Kubermatic with expertise in Kubernetes, multi-cluster management, and infrastructure automation. Having driven cloud-native integrations at GitLab and now building DevRel at Kubermatic, he regularly speaks at KubeCon, HashiConf EU, and Open Source Summit on platform engineering, DevSecOps, and cloud-native operations. He is a Google Developer Expert (Cloud), CNCF, and NeoNephos Ambassador.
