Industry executives and experts share their predictions for 2023. Read them in this 15th annual VMblog.com series exclusive.
Data transformation trends to be on the lookout for in 2023
By Satish Jayanthi, Co-founder and CTO of Coalesce.io
Modern data warehousing workloads operate on the scale of hundreds of thousands to even millions of database tables, and data engineering teams are typically tasked with building pipelines for the migration and maintenance of hundreds of terabytes of data. At that scale, projects can take months or even years to complete, and significant resources to maintain when organizations rely on code-first tools or platforms. Automating data transformations represents the missing link in helping data engineering teams overcome the bottlenecks of cleaning and transforming data for suitable consumption to make informed business and technical decisions.
Here are three data transformation predictions that I believe will come to the forefront of the data transformation market in the coming year.
The Rise of Data-as-a-Product
As the demand for the democratization of data continues to rise, businesses are becoming increasingly responsible for understanding and utilizing their sourced data to make informed business and technical decisions.
In 2023, data-as-a-product will reach maturity, improving both the quality and trust in data at companies. This will lead to more robust data organizations within enterprises that require an increased need for technologies and data teams/engineers that understand and subscribe to a culture of best practices in data modeling.
The Return of Data Modeling
Over the past 10 years, data modeling-a fundamental process of setting up data structures aligned to business requirements-took a backseat as companies rushed to bring products to market, often well before they had the data infrastructure they truly needed to be successful. (Yes, many of those products and companies were in the trendy AI/ML field.) During that same time, data volumes, data types, and the velocity of data exploded: businesses were producing more data, including real-time streaming data, and needed ways to process that data.
In 2023, industry veterans who spent nearly a decade calling for thoughtfulness in building fundamental data infrastructure instead of rushing to build buzzworthy products will get their “I told you so” moment. Data modeling is making a comeback, alongside the realization that without the infrastructure to deliver high-quality data, businesses will not get very far towards the promise of predictive analytics, machine learning/AI, or even making truly data-driven decisions.
The Rise and Fall of Everything-as-Code
In recent years, code-first technologies gained popularity as the “everything as code” trend allowed software engineering best practices to be applied to analytics. However, that approach also created challenges for organizations that became especially pronounced in 2021.
In 2023, as budgets likely continue to tighten, a trend will emerge towards seeking optimization and productivity. Rather than continuing to grow teams, companies that are forced to do more with less will look towards ways to automate data processes that they once did manually. That is good news for platforms and tools that enable automation, are simple to use, and free up time spent on repetitive tasks to focus instead of creating impact for the business.
Conclusion
By 2025, Gartner expects 70% of organizations will be compelled to leverage data more effectively. Now more than ever, data transformations are key for every organization to take full advantage of the enormous amounts of data being generated by modern enterprises. Our team at Coalesce is excited for the next phase of data transformations and the broader data cloud market being powered and propelled by the need to better utilize the data being collected for deeper and more accurate analysis.
##
ABOUT THE AUTHOR
As Co-Founder and CTO at Coalesce, Satish has designed and built the company’s data automation software. Prior, Satish was the Sr. Solutions Architect at WhereScape, a leading provider of data automation software, where he met his co-founder Armon.
