Virtualization Technology News and Information
RSS 2024 Predictions: AI and automation will be the key drivers of change in 2024


Industry executives and experts share their predictions for 2024.  Read them in this 16th annual series exclusive.

AI and automation will be the key drivers of change in 2024

By Sean Knapp, founder and CEO,

AI had its iPhone moment in 2023, and moving into 2024 companies will further transform their technology stacks, reporting structures, and overall data strategies to capitalize on this opportunity. This may include setting new, top-down mandates for AI adoption, creating AI leadership roles that sit outside of IT, and driving faster adoption of automation technologies to help them create new data products faster.

Here are 5 predictions for how the enterprise data landscape will evolve in 2024:

Companies will appoint new AI leadership that will challenge the CIO

Many data engineering leaders will return from the holidays to discover new mandates from the C-suite around AI adoption. Examples include reducing opex by 20% through AI-powered automation, increasing customer satisfaction and net revenue retention by 10% using AI, and generating 10% of topline revenue from AI-based products and experiences.

Some organizations will take a page from their previous digital transformation playbook to create an entirely new role to lead this charge.We expect to see Chief AI Officers or similar titles become more common as organizations grapple with how to rapidly integrate this new technology into legacy operations. This role will sit outside of IT, and will receive budget, headcount, and political authority to make any change needed to drive faster rollout of AI.

These new AI roles may be contentious given the increasingly fractional role of the traditional CIO. It will be interesting to see if CIOs can deploy sufficient automation so that they can free up enough time to lead these AI programs, or if they will ultimately cede this territory to a newcomer in the c-suite.

Companies that lack sophisticated automation to power AI will feel the burn

As businesses hurry to implement AI to maintain their competitive edge, many will feel the effects of a disorganized data infrastructure more acutely. The effects of bad data (or no data) are compounded when the stakes are raised from simply serving up bad information on a dashboard, to potentially automating the wrong decisions and behaviors based on that data. It's only a matter of time before someone with weak data infrastructure and governance puts unleashed generative AI with disastrous results.

Generative AI becomes more factual thanks to retrieval-augmented generation (RAG)

Retrieval-augmented generation, or RAG, allows teams to feed clean business data into generative AI models to reduce hallucinations and ensure that outputs are grounded in factual information. This clean business data will be generated by traditional data pipelines that handle data extraction, cleansing, normalization, and enrichment on an organization-wide scale. 

Facing unsustainable workloads and slow hiring growth, data engineering teams turn to automation

Data engineer hiring is not keeping up with demand, and we expect this talent gap to widen in 2024. A recent report we ran found that 95% of data teams report being at or above their work capacity for the fourth year in a row. We can expect data engineers to remain more overworked than their immediate colleagues in the data team for the foreseeable future. 82% of the teams we surveyed expressed an interest in adopting automation to improve their productivity with data, with another 9% claiming they already had successful implementations in place. 

More point solutions become platform features

To help address data stack complexity, we will increasingly see capabilities currently bought as standalone products becoming features in larger platforms. Two examples are data ingestion and data orchestration. As data platforms modernize and automate further, these features will be subsumed by platform providers and the standalone products will disappear. This will be good for data teams, since our recent report found that data team leads are eager to consolidate their tech stacks while retaining the same functionality. 

Data has been driving innovation even before The Economist declared it "the world's most valuable resource" six years ago, and the rapid emergence of generative AI has compounded that trend. 2024 will be a massive year for data teams as they seek to capitalize on the newest opportunities. But they will need to evolve both their technology stacks and priorities to truly be successful.



Sean Knapp 

Sean Knapp is the founder and CEO of, the leader in data pipeline automation for building the world’s most intelligent data pipelines. With over 15 years of experience in software development, web and mobile applications, and data engineering, he is passionate about empowering data teams to accelerate data-driven insights and innovation. Before launching in 2015, Sean co-founded Ooyala, a video platform company that was acquired for over $400 million. As the CTO and Chief Product Officer, he led the product vision, engineering, and solutions for Ooyala's award-winning analytics and video platform solutions. He also played a key role in Ooyala's acquisitions of Videoplaza and Nativ, and served as a member of the board of directors. Prior to Ooyala, Sean was the frontend technical lead for Google's Web Search team, where he helped increase Google revenues by over $1 billion. Sean holds both B.S. and M.S. degrees in Computer Science from Stanford University, with a focus on Human Computer Interaction. He has also been granted multiple patents in search and video delivery systems.

Published Friday, November 24, 2023 7:37 AM by David Marshall
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<November 2023>