Industry executives and experts share their predictions for 2024. Read them in this 16th annual VMblog.com series exclusive.
The intersection between AI and observability in 2024
By Arijit Mukherji, Distinguished Architect,
Splunk
The rapid adoption and advancements of AI in 2023 have taken
nearly all industries by storm. In healthcare,
for example, AI is driving improvements in
diagnostics and drug discovery, including enhancing cancer detection. In marketing, AI is being leveraged as a nascent creative tool to develop
new art styles and capture customer attention in ways previously impossible.
It's clear that AI is changing the way we create and respond to, well,
everything.
This also includes the enterprise landscape. As
automation becomes more integrated within organizational workflows and use of
AI generates more variety and volumes of data, business leaders are calling for
greater open standards to manage this complexity and data deluge. But, what
many may not know is in the coming year, observability will emerge as an
essential practice to effectively organize and decipher the increase in data,
complexity and microservices, all generated by AI.
As industry leaders prepare for the new year, embracing and
applying comprehensive observability solutions to their evolving AI strategies
will become essential.
The value of embracing a
human-in-the-loop approach
In 2024, CIOs and CTOs must form an understanding of how AI may
be disruptive, opportunistic, or both. They will need to develop a point of
view, and quickly. This is not just isolated to the tech industry, because AI
is continuing to disrupt a wide range of business areas from healthcare, law,
finance, travel and more in ways we haven't yet imagined.
The truth to AI is that it can't be successful if it operates in a
vacuum. The technology requires humans to develop, deploy and continually
manage it. In order to identify focus areas and sift
through available data at a faster pace, organizations should realize that
embracing a human-in-the-loop approach is best－with AI, ML and human intervention
working together in tandem.
Many enterprises are migrating their business processes to more
flexible systems to support the fluidity that AI demands, and these shifts
frequently call for increased human involvement. For organizations to carry out
a successful transition, the need for soft skills and a significant degree of
institutional knowledge about an organization, its competitor landscape and
sector are imperative. Business leaders agree that a human needs to be at the
center of the AI equation and that the technology should enhance human decision-making, not
replace it. This human at the center approach will result in more
opportunity - including an increase in job opportunities. Recently it was shown
that AI job-related posts increased more than
1,000% in the second quarter of
2023 on the global work marketplace, compared to the same time last year.
As AI adoption surges,
observability will emerge as essential to help manage the complexity
There's a feeling that generative AI and code copilots will
make us superhuman, right? Rising productivity is going to cause an explosion
in the number, scale and complexity of things that organizations will build in
the coming years. For example, organizations will need to observe and track
more things in the next 5 years - more environments, more applications, more
microservices, more code pushes and more clusters. As a result, observability
solutions will have to deal with far more variety and volume of data.
Human-driven observability of so many types of systems will not scale, and we
will increasingly look towards autonomous systems to monitor and manage
different aspects of these systems - e.g. infra monitoring, application
monitoring, data monitoring, deployment and operational monitoring etc. With so
many monitoring systems in place, open standards (open data formats, open
schemas, standard query languages etc.) will become even more important,
because that is how these different observability systems will inter-operate
and correlate data from each other.
In terms of this explosion in scale and complexity, AI will
also help us solve these new problems that it caused. While open standards will
help, they won't totally solve the cognitive overload of dealing with the wide
variety of monitoring tools. AI adoption will dramatically improve
Observability UX, and bring us closer towards "Intent Based Observability."
With intent based Observability, operators will express what they want to
achieve ("explain why this API is slow", "create and deploy a new alert to monitor
for container memory", ...) and have the system react accordingly. Expertise of
crafting queries using domain specific observability languages, or navigating
observability UIs will become less critical.
In conclusion, as we all look forward to the new year,
organizations should look into building a comprehensive, full-scale
observability practice as they continue to charge forward with AI adoption.
AI-focused leaders can expect the relationship between observability and AI to
exponentially increase and become more intertwined in 2024, with organizations
needing greater oversight and visibility as they continuously augment data
processes. It is essential that industry leaders and businesses invest in observability
solutions, now more than ever.
##
ABOUT THE AUTHOR
Arijit Mukherji,
Distinguished Architect, Splunk
Arijit is a distinguished architect at Splunk leading
architecture for Splunk's observability portfolio. Previously he was CTO at
SignalFx (acquired by Splunk), where he was instrumental in building their
observability solution from ground up. He holds a master's degree in computer
science and is the author of 10 technology patents.