Virtualization Technology News and Information
Article
RSS
How Intuit's cloud journey started with cloud migration and is ending with AIOps

By Edward Lee Distinguished Engineer, Intuit

Intuit has been on a cloud migration journey for the past 10 years and closed its last data center just this year!  While cloud adoption enabled automation and simplified infrastructure management, it did not in itself improve development velocity. Application teams still took the same amount of time to develop and deploy code and struggled with satisfying security and compliance requirements. 

In late 2017, Intuit decided to leverage containers and cloud-native technologies to accelerate development velocity. In early 2018, Intuit acquired Applatix, a venture-backed startup focused on productization of cloud-native technologies for enterprises, to build Intuit's "Modern SaaS" platform to accelerate application delivery.

The initial Modern SaaS "MVP" was a set of Kubernetes clusters designed to satisfy Intuit's security, compliance, and governance requirements. An important innovation was to give application teams access to Kubernetes namespaces rather than entire clusters. A self-service namespace management service created custom tailored namespaces for each application on-demand. Another innovation was to control all deployments to production via Argo CD, a GitOps-based continuous deployment tool. This simplified the deployment process for developers and meant that they did not need direct access to Kubernetes clusters, greatly improving security and reducing operational variability.

Today, Modern SaaS is the default application platform for all new applications and about half of all Intuit applications run on the platform. Intuit has over 200 Kubernetes clusters and thousands of applications running on Modern SaaS.

 

In the course of rolling out and evolving the Modern SaaS platform over the past three years, Intuit has undergone dramatic changes in the way it does software development. We, like many other companies, started with lift-and-shift migration to the cloud. This enabled self-service infrastructure management and eventual codification of associated operational processes. Gradually, these responsibilities transitioned from dedicated operations teams to development teams. Eventually, most organizational units at Intuit decided to embed operations into development teams and eliminated the dedicated operations teams.

With the adoption of Kubernetes, we were able to build declarative control planes with verifiable target states. This enabled us to create systems with high levels of introspection, eventually allowing more automated and deeper analysis of the operation of the system. An immediate benefit was the enabling of GitOps as a mechanism to configure and control the declarative control planes, which provided developers a higher levels of self-service and direct management of operational processes as well as better auditability of production changes.

 

In parallel to the evolution of development processes starting from cloud migration to devops and then the incorporation of GitOps, the operational capabilities of the system have also evolved to keep pace. To start with, as the infrastructure became more and more dynamic with the codification of infrastructure management, we needed a way to track and better observe what was happening in the system to properly attribute events to applications and responsible parties. In other words, it created a great need to improve the observability of our systems.

This process started with the collection of important log events, metric summaries and configuration information into a central data lake. We quickly found that data from many sources was needed to get a complete 360 view of what is happening in the system. The number and variety of data were so many, in fact, that it was not possible for a single central team to ensure a steady flow of quality data from all sources.

Instead of focusing on the collection and curation of data we decided that the central team should focus instead on building a self-service platform and providing governance to enable other teams to publish and manage the data. This is sometimes referred to as a data mesh, rather than a data lake. With a data mesh, the teams that own the data take responsibility for publishing the data and ensuring that it meets the needs of their consumers. In effect, the data becomes a product, while the platform provides the necessary capabilities and serves as a marketplace for sharing and processing capabilities for the data.

As the amount of data being shared increased, we found it necessary to implement additional governance polices to ensure easier discovery and use of the data as well as standardize keys for joining data from one source with data from other sources. This in turn led to the need for self-service analytic capabilities for enrichment, summarization, and modeling, including ML models and anomaly detection.

What started out as a platform to gather data to improve observability quickly became a platform for sharing and analyzing data to derive actionable insights to power automation and faster, better operational decision making. In fact our current focus is to better use data to quickly determine the extent and severity of user impact for outages as well as quickly identify their root causes.

To summarize, Intuit's journey to improve development velocity started with cloud migration, which led to codification of infrastructure management via DevOps. DevOps accelerated containerization and adoption of Kubernetes which led to management using declarative, verifiable target states and GitOps as the mechanism for application delivery as well as infrastructure and operations management. As infrastructure became dynamic, observability became a necessity. We started centralizing the collection of this information to gain 360 visibility to understand and attribute activity in the system. As the data accumulated, we decentralized ownership of the data by providing a self-service platform for data collection and curation. The platform quickly became the center for analytics to drive automation and decision support (AIOps).

To date, Intuit has realized dramatic improvements in development velocity by modernizing its application delivery capabilities. We are currently in the process of improving our operational processes via observability, data mesh and realizing the potential for AIOps that is being enabled by the modern application delivery platform.

##

***To learn more about containerized infrastructure and cloud native technologies, consider joining us at KubeCon + CloudNativeCon NA Virtual, November 17-20.

About the Author

Edward Lee Distinguished Engineer, Intuit

Edward Lee 

Ed is leading the migration of Intuit's applications to open, cloud-native platforms like Kubernetes. He was co-founder and CTO of Applatix, a venture-backed startup focussed on products for enabling enterprises to use Kubernetes, including the creation of the Argo Project for Kubernetes. Applatix was acquired by Intuit in early 2018.

Published Monday, November 02, 2020 7:34 AM by David Marshall
Comments
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
Calendar
<November 2020>
SuMoTuWeThFrSa
25262728293031
1234567
891011121314
15161718192021
22232425262728
293012345