By Pini Ben-Nahum, VP R&D at Zesty
GPUs are among the most expensive resources in modern infrastructure, yet production workloads leave the vast majority of their reserved capacity unused. For platform teams, the gap between allocated GPU capacity and actual usage can quickly become one of the highest costs of running AI at scale.
The traditional Kubernetes GPU allocation model often gives a workload an entire GPU, even when that workload uses only a fraction of it. As GPU costs have grown, the obvious next question is: how can multiple workloads safely share that GPU? NVIDIA built an answer to that question years ago, but by most measures, it’s still barely used.
Compute is easy to share. Memory is harder.
NVIDIA Time-Slicing addresses the compute side of the problem. It allows multiple workloads to share the same GPU by rotating their execution over time. From a Kubernetes perspective, one physical GPU can offer multiple shared accesses, allowing several workloads to use it, instead of being reserved for a single workload.
Memory is a different story. Time-sliced workloads still share the same VRAM pool, with no memory isolation between them. A memory-hungry workload can therefore consume capacity needed by its neighbors, causing allocation failures or instability in the other workloads sharing the GPU.
MIG (Multi-Instance GPU) was designed to solve this problem. It physically partitions supported GPUs into isolated GPU instances, using predefined profiles such as 1g, 2g, or 3g configurations with corresponding memory capacities. Each instance receives dedicated compute and memory-system resources, providing strong isolation and predictable performance between workloads.
However, MIG’s isolation comes with a tradeoff. Capacity is selected from a predefined set of instance profiles, and changing the size assigned to a running workload is not an in-place operation. It requires stopping or rescheduling the affected workload before deleting and recreating its MIG instance with a different profile. Kubernetes tooling can automate MIG reconfiguration, but it remains a disruptive infrastructure operation rather than a lightweight runtime adjustment.
In practice, this pushes GPU sizing toward an upfront capacity planning decision. Teams profile or estimate a workload’s needs, choose a MIG profile that can handle its expected demand, and keep that allocation until someone decides the configuration should change. When demand varies over time, this can create pockets of unused GPU capacity that cannot easily be reassigned to workloads with higher demand.
MIG therefore solves isolation much better than flexibility. For workloads with predictable resource requirements, that’s often exactly the right tradeoff. But for dynamic workloads whose memory and compute needs change over time, fixed profiles can leave substantial capacity stranded.
What’s changing: Software-defined GPU Fractioning
Things are starting to change. Run:ai, now part of NVIDIA, is taking a different approach by sharing GPU resources in software instead of relying only on fixed hardware partitions like MIG. That gives workloads much more flexibility in how much GPU capacity they get. Today, this is available through KAI, Run:ai’s open-source scheduler, with support for other schedulers expected to follow.
With software fractioning, workloads can request specific amounts or fractions of GPU memory instead of choosing from a small menu of MIG profiles. Compute can just be shared through mechanisms such as Time-Slicing or MPS, depending on the workload’s execution and performance requirements.
Unlike MIG, the resource split does not require reconfiguring the GPU’s hardware geometry. That removes a major operational constraint, since changing software-defined allocations does not require creating a different MIG layout for the physical GPU.
The gap that’s left
Fractioning, MIG, and Time-Slicing provide increasingly capable mechanisms for sharing a GPU, but they do not by themselves determine the optimal allocation, and they leave many questions unanswered: which workloads should share a GPU, and can safely colocate without creating performance interference? How much memory or compute each workload actually needs, and when those allocations should change as demand moves?
Today, determining how much GPU memory and compute a workload needs remains largely a manual profiling, configuration, and capacity-planning exercise.
What’s still missing is a real-time automation layer with accurate decisions, made as fast as the workload mix changes, without an engineer redoing that math by hand every time.
That means a system continuously watching GPU memory, compute behavior, and performance signals, then reevaluating and adjusting resource assignments as demand changes.
GPU sharing mechanisms are getting increasingly sophisticated, and they answer the question: How can a GPU be shared? The remaining optimization problem is: How should it be shared right now? That requires continuously understanding workload demand, interference, performance sensitivity, and available capacity, then turning those signals into allocation and placement decisions. Fractional GPUs provide the mechanism. The missing layer is the decision engine.
KubeCon + CloudNativeCon is taking place in Salt Lake City, Utah from November 9 to 12. Register Now!
##
ABOUT THE AUTHOR
Pini Ben Nahum is VP of R&D at Zesty, where he helps shape both product and company direction from an engineering perspective. With over 15 years in tech and a decade in leadership, he has built and scaled R&D organizations with a focus on cloud infrastructure, microservices architecture, and developer experience.
