Industry executives and experts share their predictions for 2023.  Read them in this 15th annual VMblog.com series exclusive.

2023: The Year of Real-Time Data

By Gary Hagmueller, CEO, Arcion

Despite market turmoil and economic disruption, technical transformation proceeded unabated. In many ways, one could argue that the pace of change also increased. At the very least, the shape of the “modern technology stack” became clearer. Awareness of the benefits also rose to the level where day-to-day technology adopters, particularly within the enterprise space, became increasingly common.

As a result, it seems clear that 2023 will see the momentum of change accelerate. It will also see the culmination of many trends. In what amounts to a kind of “technology evolution,” some of the trends in the data infrastructural world will see permanent separation from the previous state of the art.

Prediction #1: Moving Data Is No Longer Just “ETL”

Among the most significant changes that will emerge is that the traditional notion of extract, transform, load (ETL) as a means of moving data from the point of generation to consumption will finally and forever disintegrate into at least four distinct categories. These distinct categories comprise:

• Streaming SaaS – The ability to move data from SaaS systems and BI consumption points has been underway for years now. This progression follows tried and true technology evolution trends in that the SaaS data is structured and often already refined. The BI consumers are generally looking for a way to aggregate SaaS data from multiple systems they already have access to into a unified dashboard.

• Enterprise “EL” – This category is becoming more popular by the day. As adopters of streaming SaaS solutions realize that the data stored in those systems represents only a small fraction of the data available to the enterprise, smart technologists have begun to demand solutions that complete the picture and leverage the data to drive breakthrough advantages. In this category, transactional data from production systems is moved in real time into BI and machine learning platforms to drive highly relevant and extremely valuable decision-making.

• Observability, Cataloging and Data QA – As the sophistication of enterprise data products evolves, the need to ensure that the source data comports with the original model or application specifications is key in eliminating unwanted surprises or broken models. Moreover, a lack of understanding about what data is available and how each field relates to other fields serves to limit the efficacy of the data products being developed. These are all interrelated, and this category will continue to cleave from the broader ETL space.

• Transformation – For those of us that have spent years in the data space, the fact that the “T” is sandwiched between “E” and “L” has always been a bit confusing. Transformation, feature generation, and other forms of “data wrangling” are generally too complicated and compute intensive to be done on the wire by a technology that is first responsible for moving data.

Prediction #2: Real-Time Data Is Going Mainstream

In addition to the disruption of the ETL space, the data world is rapidly beginning to demand data that can be consumed in real time. So in 2023, it seems clear that real-time use cases will go mainstream. The list of applications grows by the day. Use cases such as IoT, fraud, customer next action, inventory management, FX, edge computing, etc., are expanding by the day. One report finds that 80% of American consumers are more likely to purchase from a company that personalizes its sales offering. As these real-time uses mature, the demand for real-time pipelines to stream data from transactional systems will continue to grow. Look for the explosion of real-time use cases to drive lots of innovation and value in the market. Look also for real time to increase pressure for the implementation of “modern data stacks.”

Unfortunately, it’s impossible to deliver real-time personalization if your databases take hours or days to update your customer recommendation engine. A data-driven company that is focused on business success must rethink its data strategy in 2023 and embark on a data modernization journey. One of the most effective ways to modernize your data is to introduce a technology called change data capture (CDC) in your data infrastructure. A major benefit of CDC is that it provides fast, fresh and accurate data so decisions can be made with speed and precision.

Change data capture is a methodology or design pattern that identifies and captures changes in your database over time. These changes are recorded, synced near instantly, and sent either to a table in the same database, a different database, a streaming platform, or to cloud storage.

Prediction #3: Time To Optimize the Cloud Bill

Cloud bills in 2022 started to become material. In 2023, cloud total cost of ownership (TCO) will drive optimization behaviors. With economic pressures everywhere, the scrutiny of extraneous spending is becoming more pervasive. One area that is emerging is the manner in which cloud resources are consumed. While the cloud brings many positives and provides a material increase in effectiveness over self-deployed resources, it’s becoming more obvious that these usage-based models may result in sticker shock bills. The solution is to adjust usage patterns, insert controls, and “right size” deployments as well as optimize services that drive usage. Modern technologies have a strong part to play in delivering optimal utility while limiting (or reducing) excess cloud spending.

Prediction #4: Low-Code/No-Code Are Coming to the Data Infrastructure Space

2023 will see low-code/no-code technologies begin to penetrate the infrastructure space. As data teams demand more from the enterprise technology stack, there will continue to be a strong push to allow them to self-service. The challenge this presents is that it will become increasingly difficult to train data teams on all the different coding platforms and operating environments that would be required to integrate disparate data resources. The answer is clearly to reduce the need for training by having the technology abstract away the required coding. The low-code/no-code methodology is the solution: It seeks to anticipate the various uses and configurations required and presents them to a user in an intuitive and simplistic way. Done properly, this will allow data and IT teams to focus on the areas for which they can generate the most benefit without having to learn peripheral things that don’t drive value for their functions.

As cost pressures mount and competition increases for enterprises worldwide, their salvation will be real-time data and all the augmentations that come with it. Enterprises that leverage the trends and technologies mentioned here stand to grow and transform the way they are run and also derive the best value out of their technological buck. Real-time data is the catalyst for genuine transformation; the future is here and here to stay.

##

ABOUT THE AUTHOR

Gary Hagmueller, CEO of Arcion Labs, has been a leader in the tech industry for more than 20 years. With a deep focus on data infrastructure, AI, machine learning, and enterprise software, he has raised over $1.3 billion in debt and equity and played a key role in creating over $10 billion in enterprise value through two IPOs and four M&A exits. Previous to Arcion, he was CEO of CLARA Analytics, COO of Ayasdi, CFO of Zuora, and held many business and corporate development leadership roles. For more information on Arcion, visit www.arcion.io/, and follow the company on LinkedIn, YouTube and @ArcionLabs.