As organizations look to run AI models on their own infrastructure, they face a growing challenge: how to manage inference costs, performance, and governance across different models and compute environments. Buoyant, the company behind Linkerd, is addressing this challenge with Kinference, an open source inference serving platform for Kubernetes.
In this VMblog interview, William Morgan, co-founder and CEO of Buoyant, discusses why self-hosted inference needs more than fast models, how Kinference helps organizations manage AI workloads, and why platform teams need control over how inference requests are handled.
++
VMblog: What motivated Buoyant to develop Kinference, and what specific challenges does it address for organizations bringing AI inference in-house?
William Morgan: We watched our customers struggle to operate hosted inference, and later we struggled ourselves. In both cases the cause was the same. When you’re building out a new capability, if you have a history of deploying reliable software at scale and looking at what makes that work and what doesn’t, is, you take a platform mindset. You don’t want to solve a one-off inference problem, you want to build an inference platform. But there was nothing in the ecosystem that allowed you to do that. There were plenty of tools for running individual models or for maxing out a set of GPUs, but nothing that was platform-focused. That’s when Kinference started to take shape.
VMblog: How does Kinference help organizations reduce inference costs while making better use of their GPU resources?
William Morgan: Cost and utilization are the exact inverse of each other when you rent or buy the hardware. If you want to reduce cost, your only option is to increase utilization, and vice versa. If you’re paying for N GPU hours, then any moment those GPUs are idle is, effectively, tokens that you’ve thrown away. So how do we help? First, we give you the tools to monitor exactly what is happening across your GPUs, at both the hardware level and at the token level. Then we give you the capability to assign clear priorities so that you can schedule batch or deferrable work during idle periods. Do the interactive work when there is a human waiting for every token; when there isn’t, do the batch work.
VMblog: Kinference supports intent-based routing. What does that mean in practice, and why does it matter?
William Morgan: Intent is just one type of semantic classification. We have a pluggable system that allows you to route and set policies based on semantic classifiers which can include intent, complexity, whether it contains PII… whatever you want. So for example you can send simpler prompts to cheaper and faster models, or you can refuse to let send requests externally if they have PII, or more.
VMblog: How does Kinference help organizations balance self-hosted inference with access to cloud-based models?
William Morgan: The reality is that most production situations are messy. You often have some inference that is hosted and some that you rely on an external provider for, for any number of reasons. Kinference can route requests to cloud providers, including frontier models, when conditions defined by the platform owner are met. That might happen when a GPU pool is saturated, when a particular tenant reaches a usage threshold, or based on the characteristics of a request. The important part is that organizations can define how that works and make it an explicit policy rather than leaving those decisions unmanaged.
VMblog: What makes Kinference practical for platform teams, and how can organizations get started?
William Morgan: We built Kinference with the same philosophy as Linkerd. It should be lightweight, easy to deploy, and something you can run without needing a team of specialists. It’s also agnostic to the model, cloud provider, and GPU type, so you’re not locked into a particular environment. We have a decade of hard-won lessons helping companies operate mission-critical applications on Kubernetes, and we’ve built those lessons into the core of how Kinference was designed.
##
