In the ever-evolving landscape of data science, tools from Metaflow stand out. To gain deeper insights into this technology and its impact on the field, we had the privilege of sitting down with Savin Goyal, an industry expert and the CTO and co-founder of Outerbounds. As the conversation unfolded, we dive into the distinctive features of Metaflow, its ability to empower data scientists in designing robust machine learning (ML) applications, and the strong connection between Metaflow and Outerbounds.
Metaflow has been making waves in the data science community for its unique approach to workflow management and ML stack integration. Unlike traditional frameworks, Metaflow seamlessly combines the power of both simplicity and scalability. It offers a comprehensive ecosystem that enables data scientists to tackle complex ML problems with ease while maintaining an emphasis on productivity and collaboration.
Learn more and find out what’s next.
VMblog: Tell us about Metaflow and what makes it stand out in the data science world.
Savin Goyal: Our roots are at Netflix, where we were tasked with creating the infrastructure stack that powers much of ML that Netflix is involved in, be it content recommendation, demand modeling, pricing optimizations or fighting abuse. We noticed that there was a lot of friction in the process of developing, testing, scaling and deploying ML models and that was a drag on data scientists attempting to build. We sat through meetings where data scientists discussed how projects were delayed because getting the first version of the model to production took a very long time – mainly because of many mundane reasons related to software engineering.
We realized that nearly everything that data scientists wanted to do was technically already doable, but it was not easy. Data scientists love the freedom of being able to choose the best modeling approach for their project. They want to stay in control of model inputs and feature engineering logic. In many cases, data scientists are quite eager to own their own models in production, since it allows them to troubleshoot and iterate the models faster. On the other hand, very few data scientists feel strongly about the nature of the data warehouse, the compute platform that trains and scores their models, or the workflow scheduler. Preferably, from their point of view, these foundational components should “just work”. The infrastructure should allow them to exercise their freedom as data scientists but it should provide enough guardrails and scaffolding, so they don’t have to worry about software architecture too much.
We developed Metaflow to make things easier. The platform is a human-centric framework for data science, which we open sourced in collaboration with AWS in 2019 and now powers thousands of ML experiences outside of Netflix, at companies like 23&Me, Adobe, Carta, Fortum, Goldman Sachs, Intel, Sanofi and Zipline.
VMblog: How does the Metaflow ML stack empower data scientists to design ML apps?
Goyal: Data Scientists are a curious bunch and while they want to take smart risks that have the potential for high business impact, they soon find out it can be more of a challenge than they anticipated. Building apps requires a multitude of tools and vendors to tackle the various components, including security and infrastructure. There is a need to interface with a lot of different stakeholders to get the basic permissions to do anything, and even then when they get there, they are obliged to wrangle a bunch of point OSS projects for managing compute, orchestration, version control and so forth. This process is unscalable and inefficient. Metaflow was built because so much of that glue-code and “point-solutioning” was redundant and undifferentiated. The Metaflow platform allows for efficiency and scalability, giving Data Scientists the opportunity to do what they want to do – data science!
This stack empowers data scientists to design ML apps, enabling creativity in a dynamic, stable, and future-facing environment. Outerbounds gives you the knowledge of an engineer without actually having to be one. We allow enterprises to control the data that lives within their cloud, and Outerbounds manages it for them. With Metaflow and the Outerbounds Platform, you invite your teams and systems to an ML-driven future.
VMblog: How are Metaflow and Outerbounds connected?
Goyal: We founded Outerbounds in 2021 to continue developing Metaflow together with Netflix and the wider community. We’ve seen patterns emerge through our support of hundreds of organizations on their Metaflow journey. While the open-source Metaflow is common for all organizations, a great deal of variation exists on the infrastructure side. Some of this variation is expected and desirable due to genuine differences in organizations and the businesses behind them, but much of it is accidental. Ultimately, the outer bounds of common engineering concerns – security, scalability, cost-efficiency, high availability, and integrations to surrounding systems – look similar. Based on lessons we learned from these organizations – small, large, and every size in between – we set out to bake all infrastructural concerns and best practices into one platform, Outerbounds Platform.
VMblog: How can tools like Metaflow and Outerbounds be utilized to incorporate new AI techniques into an ML infrastructure stack?
Goyal: Take LLMs, for example. LLMs and other foundation models will get adopted as a powerful new model family in the ML stack – augmenting, not replacing it.
While it is easy to see how the initial excitement produces overly vivid dreams and nightmares, the long-term effects tend to get underestimated, as it is hard to visualize what will happen when new technology becomes gradually accessible to an increasingly diverse set of people and companies. We can’t foresee how exactly the new models will affect our lives in the future, but we do know what the journey there will look like. We need to make the latest technology easy to use by offering it up in a human-centric package, so that all interested organizations can start innovating with it independently, exploring a diverse landscape of use cases driven by their own domain knowledge and visions for the future.
Here are a couple of illustrations to make this clear. In the first illustration, the data scientist on the left is more concerned with modeling and less interested in the lower-level infrastructure. Infrastructure providers on the other hand, such as those of us at Outerbounds or internal platform teams, have an inverse focus. Now, LLMs have made everyone’s focus more extreme, as shown in the second diagram:
We believe that LLMs and other foundation models will get adopted as a powerful new model family in the ML stack – augmenting it, not replacing it. The new techniques will enable new use cases in some scenarios, especially when text and images are involved. In other situations, they will improve existing ML models and data science projects.
An important implication of the new AI techniques becoming a part, not a replacement, of the ML infrastructure stack is that you can continue working on modernizing your data and ML stacks with future-proof tools like open-source Metaflow. Realistically, most companies are still in the preliminary stages of their ML journey, so starting off by laying a solid foundation makes sense.
The Outerbounds Platform can provide this foundation and help businesses get started with minimal effort, providing an easy-to-use, optimized, and secure environment for all ML/AI experiments and production applications. All work towards this goal remains relevant and will pay dividends once you are ready to start adopting the latest AI techniques.
VMblog: The world is abuzz with talk of AI and ML. How can enterprises ensure they are making the best use of these technologies?
Goyal: We have seen only the early days of ML/AI adoption. Enterprises must remain agile with flexible foundations because change is constant. Science is naturally experimental, and companies, teams, and code bases constantly evolve. Tools that accelerate science need to be dynamic and flexible to be effective. This is why we are building Outerbounds to grow with you. While AI is already doing amazing things, we need more time to understand where it will take us. As with any new tech, it requires some time to mature, and there is significant trial and error. This is where flexibility comes into play. An enterprise that is open to change, understands its specific needs and where they stand to gain the most from AI, will excel.
VMblog: What’s coming next?
Goyal: We have some exciting projects to announce later this year! We continue to grow and learn with our customers using the Outerbounds platform, and organizations and teams continue using Metaflow. We often publish on our blog – this is a great resource to keep up with the fast pace of new projects and use cases being introduced. Our YouTube channel is also a great resource.
##
