This post describes our team’s journey to AI/ML Ops (Artificial Intelligence / Machine Learning Operations) with Kubernetes at Oracle. We work with the development engineering team in the AI services organization of Oracle Cloud Infrastructure (OCI). The team specializes in delivering managed AI services with pre-trained models to business customers – things like Generative AI, Digital Assistant (chatbot), Speech, Language, Vision, and more. Optimizing how we construct, train, and serve our AI/ML models is always top of mind as we continuously strive to boost operational efficiency and minimize resource expenditure.
Our Journey to Kubernetes
We initially tried to implement the computer vision service using a platform designed for the constructing, training and serving of AI models that supported a single instance of a model per deployment without the capability to vertically scale. This meant we had to write custom code to scale models by adding new deployments as demand increased.
We had a few other challenges as well. Each deployment had its own endpoint for servicing requests, which meant that we would need a way to load-balance requests for the model. This in turn also meant that there would be multiple Domain Name Service (DNS) entries for each model that we would have to manage. We faced a few other challenges like these. This motivated us to develop our own load balancing software to distribute traffic across the various endpoints, which of course added further complexity to our intended deployment process.
Ultimately, we realized that our approach was causing in significant latency issues. In fact, the latency issues were so severe – without any elegant way to mitigate them – that it was ultimately deemed unacceptable for our operations, bringing our program to a standstill. So, it was “back to the drawing board”, as they say; we needed to find a deployment platform that could meet our requirements while also not being a burden to manage – and fast.
Necessity Leads to Experimentation
Before all of this, we had experimented with Kubernetes when we were creating a load testing tool used to stress-test our computer vision models. We recalled how it had been straightforward to set up the tool on Kubernetes, and we had found it easy to grow and shrink resources due to the automated built-in scaling. That positive experience with Kubernetes and the need for an alternate deployment solution inspired us to experiment with its use in serving our new computer vision service models.
“I’m not exaggerating when I describe our reaction: we were completely surprised and delighted by what we discovered,” said Simo Lin, Principal Member of Technical Staff. “The performance improvement with Kubernetes was striking – almost 10x improvement in throughput for serving our models! – and it scaled impressively”. In fact, this efficiency gain allowed us to abandon our previous custom-developed load balancing software in favor of the native load balancing mechanism built into Kubernetes, which also had the additional benefit of requiring just a single endpoint to serve requests per model. This significantly lowered our system’s complexity because we would no longer have to deal with the many DNS entries per model that our custom load balancing tool had demanded. Nice.
As we continued down the path of deploying our service on Kubernetes, we discussed the approach with many of our engineering colleagues who had experience with Kubernetes reassured us the right approach was to use our managed Kubernetes platform, OCI Container Engine for Kubernetes (OKE).
Kubernetes and Open-Source Ecosystem Benefits
Once we deployed our service on OKE, we began to see the benefits of the rich ecosystem of Kubernetes. As new capabilities needed to be added to our service, we were able to find a solution using Kubernetes features and related open-source solutions. Plus, because OKE was a service (i.e., managed on our behalf), much of the underlying Kubernetes administration was offloaded from our team – a benefit that we happily embraced.
Multi-tenancy on Kubernetes with Namespaces
One service requirement that came forth was a need to allow customers to bring their own models into our environment. This proved to be a challenge as we needed to make sure that each customer’s model was isolated for privacy and security. Since it would not be feasible to have a Kubernetes cluster to serve each customer’s model, we needed a way to isolate numerous discrete models on the same Kubernetes cluster.
We turned to Kubernetes namespaces to overcome this challenge. Namespaces provide the foundation for isolating different customer data and allowed for multi-tenancy for our clusters, which in turn allowed for better utilization of our compute resources. While namespaces don’t inherently offer isolation, they establish a framework for constructing further isolation mechanisms.
So, with namespaces as a base, we could separate customer network traffic with network policies and then use Istio, a service mesh, to secure all communication within the Kubernetes cluster. And for each customer model that we now deploy into a namespace, we also provide a unique DNS entry to access the model, thus providing isolated access to each one.
Scaling with Custom Application Metrics using KEDA
Another learning along our journey was realizing that the standard method utilized by Kubernetes Horizontal Pod Autoscaler (HPA) for scaling pods based on CPU and memory metrics does not meet our requirements. Specifically, when dispatching a batch job for execution, we observed that the CPU usage would surge to 100%. The HPA, relying on CPU utilization for scaling decisions, would initiate the launch of additional pods. This often led to unnecessary resource allocation since it didn’t correspond to an actual increase in the number of jobs that needed processing. So, instead, we adopted Kubernetes Event-Driven Autoscaling (KEDA) to incorporate custom metrics into our model’s scaling mechanism to scale pods to jobs awaiting processing.
Scaling Pods Down to Zero with Knative
Since our vision service models (which are on the smaller side) do not need to stay up all the time, we needed a way to spin down the number of instances of our models to zero to save on resources while the models are not in use. Ideally it would be an on-demand capability where the model is spun up only when a request comes in. Because Kubernetes HPA does not have this option natively, we employed Knative to build a Function-as-a-service on top of our Kubernetes clusters to spin up a vision model as requests come in. This approach also yielded better spin up and spin down time for our models, a welcome benefit.
Summary
The learning experience we had from deploying our computer vision on OKE has convinced us that Kubernetes is the right platform when it comes to building, training and serving AI/ML services. Since this experience we have deployed or migrated all of our AI services on to OKE including our recently launched OCI Generative AI. With Generative AI, we used OKE in all stages of the lifecycle of deploying the service as illustrated below.
This is just a small sample of the wonderful experience we’ve had in our journey with Kubernetes and AI/ML Ops. Hopefully you too can gain inspiration from our success and take a similar path with AI/ML Ops and the ever-expanding Kubernetes ecosystem.
##
ABOUT THE AUTHORS
Chiping Hwang, Technical Product Marketing, Application Development on OCI
Simo Lin, Principal Member of Technical Staff, AI services and Development
Brian Wood, Principal Product Marketing Manager, Application Development on OCI
Additional links:
- OCI Container Engine for Kubernetes
- CNCF projects:
