By Malini Bhandaru and Iris Ding of Intel
GenAI and retrieval-augmented generation (RAG) are top of mind for everyone with all the recent advances in large language, vision, and audio models. The possibilities for creativity and improving our productivity are limitless. However, while eager to embrace GenAI capabilities, enterprises are struggling to do so because of technical complexity, cost, and talent scarcity. These issues can be addressed with an open ecosystem.
The Open Platform for Enterprise AI (OPEA) vision is to “enable enterprises to develop and deploy GenAI solutions powered by an open ecosystem that delivers on security, safety, scalability, cost efficiency, and agility.” It was launched in May under the aegis of Linux Foundation AI & Data and is fully open source and open governance. OPEA has garnered over 40 industry partners and has a technical steering committee with representatives from hardware companies, software providers, system integrators, and end-users. You can learn more about OPEA and sign up for working groups, hackathons, and community day events at Opea.dev.
OPEA: The Project
OPEA provides sample GenAI pipelines for common use cases that have been tested on multiple hardware platforms. It further provides instructions on how to customize these pipelines, for instance swapping out one vector database solution with another or providing users with the flexibility to use different LLM models without having to modify code.
OPEA code lives at OPEA GitHub and currently has five sub-projects with more being added as the project grows:
- GenAIExamples is comprised of sample GenAI pipelines, or workflows, for applications such as chatbots, that are designed to simplify the processes of deploying, testing, and scaling GenAI applications.
- GenAIComps houses the individual stages in a pipeline, which fall into types like model serving guardrails, embedders, vector databases, re-rankers and more. There may be multiple options for a type, for example TGI and vLLM for model serving. Apart from open source models they leverage frameworks such as LlamaIndex and LangChain.
- GenAIInfra addresses enterprise aspects such as running at scale in Kubernetes with observability, authentication, and authorization. It also encompasses the GenAI Microservices Connector (GMC), which provides a Kubernetes Custom Resource Definition for GenAI pipelines that combines cloud native features of scalability and cost efficiency with GenAI needs. We will delve into GMC in the next section.
- GenAIEval is an umbrella that covers all things around evaluation, such as performance, trustworthiness, and enterprise readiness.
- Docs ties together documentation for the whole project. The best place to start is the Getting Started guide. ChatBots are the most popular use case of GenAI. Recognizing this, the getting started guide illustrates how to launch a ChatQnA application on a platform of your choice. It has two aspects, one loading in custom data (called Data Prep), and the other responding to use prompts after consulting any uploaded data, both accessed by a UI front-end. This is comprised of several services including embedded, retriever, re-ranker, and the LLM itself.
GenAI Microservice Connector (GMC)
GenAI workloads will likely run in a cloud, at the edge, or in the data center using a flavor of Kubernetes. Leveraging our Kubernetes background, we set out to bring the best of cloud native to GenAI workloads. GMC centers around a Kubernetes Custom Resource Definition (CRD) for GenAI pipelines, a controller for the CRD, and a router to connect the services in a GenAI pipeline. A developer doesn’t need to write any code should they want to swap out a microservice component, change a model, or leverage an external service, be it a proprietary hosted LLM or a database. They only need to edit the GenAI pipeline yaml file and redeploy it using Kubectl. Only services that are modified are redeployed, reducing both launch latency and churn. By design, GMC does not disrupt the data plane during re-configurations to ensure that requests in inflight proceed to completion, but it reaps resources as soon as possible. Further, it supports viewing the status of all pipelines.
To better support GenAI pipelines, GMC offers yaml constructs such as sequence, switch, and parallel. While sequence is obvious, switch is particularly valuable in these early days of AI when there are no clear model winners to let end users explore using different models dynamically. Switch can also be used to control whether an end user can access a particular model or database instance based on their access privileges. Parallel supports reaching out to multiple resources, executing actions in parallel, within a pipeline stage, before combining their responses. This is ideal for consulting multiple smaller expert models or launching agents in parallel.
Enterprises typically offer more than one GenAI pipeline, often with some overlap. GMC detects such common services and facilitates their sharing to reduce resource needs within a namespace. It also provides isolation across namespaces.
Finally, GMC is totally cloud native, designed to support integration with service mesh technologies such as Istio, Envoy, and Linkerd to ease integration with gateways, provide secure communication (TLS/mTLS), facilitate authentication and authorization using solutions such as KeyCloak, and easily add in AI guardrails.
OPEA offers a Helm chart to install GMC. Once GMC is installed, it can deploy and manage multiple GenAI pipelines. This downloadable ChatQnA demo video illustrates the ChatQnA application without RAG, with RAG, and finally the same RAG pipeline using a different LLM.
Getting Involved
OPEA aims to reduce GenAI enterprise adoption barriers. Test drive OPEA on your favorite cloud or on-prem infrastructure. See how fast you can stand up a proof of concept and then migrate to a production deployment using OPEA GMC on Kubernetes. Check out how you can use Kubernetes’ Node Resource Interface for platform optimizations. Consider contributing a GenAI microservice component or a new GenAI use case or cloud service provider infrastructure template. Join us on this journey to help democratize GenAI through contributions: code, use cases, documentation, benchmarks, tutorials, and blogs. We welcome your contributions!
To learn more about Kubernetes and the cloud native ecosystem, join us at KubeCon + CloudNativeCon North America, in Salt Lake City, Utah, on November 12-15, 2024.
##
ABOUT THE AUTHOR
Iris Ding, Senior Cloud Native Architect at Intel
Iris Ding is a Senior Cloud Native Architect at Intel with expertise in Kubernetes and Service Mesh technologies. She is the lead architect of OPEA GMC. Iris is also an Istio Steering Committee member and maintainer, and has been instrumental in making Istio efficient, extensible, secure, and easier to use. She is a frequent conference speaker and a program committee member for Istio Day and Istio Con.
Malini Bhandaru, Cloud Native Architect and Senior Distinguished Engineer at Intel
Dr. Malini Bhandaru is a Cloud Native Architect and Senior Distinguished Engineer at Intel. She currently serves as the TSC Chair for OPEA, an LF AI and Data sub project. Her interests are in security, confidential computing, and AI. She began her Intel career as a Xeon Server power and performance architect.
