By Sachi Desai, Product Manager at Microsoft
Artificial intelligence (AI) and machine learning (ML) ignite the imagination, catching the attention of everyone from small businesses to large enterprises across industries like retail, finance, manufacturing, healthcare, and more. Integrated into copilots and assistants, the use of generative AI has grown immensely in recent years and is transforming digital experiences for customers and users. These intelligent applications can provide connected device experiences, transaction processing at scale, real-time fraud detection, personalized service and recommendations, and access to large amounts of information through conversational interfaces. Powering all this is a vast amount of data and ML models, like large language models (LLMs) that can make sense of the data.
You may be familiar with open-source LLMs like Llama 2, Falcon-7B, and Mistral-7B, and looking to integrate similar models into your applications. Using open-source models unlocks flexibility and customization, brings the latest technology to your fingertips, and reduces the need for vendor lock-in. However, starting an open-source AI inferencing workflow may seem intimidating to first-time users, especially when you must consider how to:
- Download/store the large model weights, ranging 200-400GB in size
- Handle long wait times to load model weights (i.e., from local or cloud storage) due to network topology constraints
- Reduce model weights and model image into a container file
- Host the model container files in a private registry
- Provision compatible compute/GPU infrastructure for large models
- Tune model deployment parameters to fit the chosen GPU hardware
- Troubleshoot GPU quota limitations by switching hardware
Deriving insights from big data with the help of AI-driven apps requires powerful compute resources scheduled via a reliable container orchestration platform. In addition to simplified infrastructure management, you’ll be running apps on containers automated with networking, scaling, application lifecycle management, and more.
Happily, Kubernetes is just the right open-source platform to scale and automate your AI/ML applications, without compromising the security of user and company data. Your long-running ML workloads will take you through the stages of data preparation, feature engineering, inferencing, and more, with each requiring varying compute resources. To tackle these fluctuating demands, Kubernetes offers auto-scaling to handle changes in CPU, GPU, and memory so that you don’t have to intervene manually through the application lifecycle.
What about your data? Wherever it comes from, you’ll be managing terabytes of data at a time for model training and testing. Cloud-based storage repositories are a secure place to maintain and access your data at high speeds, but how do you balance data management with running AI/ML workloads? Conveniently, you can integrate a cloud provider of your choice into your workstream using a Kubernetes storage plugin. To accommodate your project’s scalability needs, Kubernetes can automate the provisioning, or assigning capacity, of long-term storage for your data.
Further simplifying the AI onboarding workflow, the Kubernetes AI Toolchain Operator (KAITO) is an open-source project that automates the deployment of LLMs in a Kubernetes cluster. You can choose from a selection of popular open-source models or even bring your own containerized models. KAITO provides preset configurations that match GPU hardware in deployment, auto-provisions your resources, and hosts large model container files in a cloud container registry. It significantly reduces time to value from days or weeks to just minutes by handling tedious hardware configuration and provisioning. You don’t have to worry about the infrastructure and can focus on the results that matter for your business goals or projects!
As an open-source platform, Kubernetes is evolving every day with the help of collaborators and contributors across the world. KAITO is a great example of this and is a growing open-source initiative, bringing together contributors with their own AI/ML models and workflow improvements. You can learn more about Kubernetes and AI and see a demo of KAITO in this recorded session. Visit the Kubernetes AI Toolchain Operator GitHub repository and try our quick start tutorial to jumpstart your model inferencing today.
##
ABOUT THE AUTHOR
Sachi Desai is a Product Manager at Microsoft Azure, where she focuses on improving the end-to-end experience of running AI workloads on AKS, like infra configuration, scheduling enhancements, and similar features. Her background is in statistics and ML at Carnegie Mellon University and project management in Azure IoT.
