Industry executives and experts share their predictions for 2023. Read them in this 15th annual VMblog.com series exclusive.
Four Trends to Watch in the Data Management Space in 2023
By Eliud Polanco, President of Fluree
As data becomes more ubiquitous and critical to Business operations, we see major evolutions happening in how data is managed, regulated and maintained from a technical perspective. Below are four trends on how data management is changing, along with the subsequent implications and predictions for what this means to the industry.
1. Businesses are evolving from function-centric business models, to data-centric ones.
For the past 20 years, business IT investments were focused on increasing productivity at the function level. Optimizing the function often involved (1) decomposing large, monolithic business processes into agile components that can be delivered more efficiently through globalization and networking; (2) investments in enterprise software, and more specifically, software as a service to digitize and automate processes; and (3) implementation of micro-service IT architectures that re-define how business applications and built and deployed. Data often came as a result, or by-product, out of those investments.
We’ve reached a peak threshold of function-optimized productivity, and the new arena for competitive differentiation is out-smarting the competition, versus out-executing them. This requires putting data in the center and having all business functions be able to securely collaborate and leverage data coming from across all other functions. In this data-centric model, the data is the product, and the functions come to the data rather than the other way around. Data has to be designed to be understandable, interoperable and more securely managed compared to the traditional function-centric silos where data is saved today. This will challenge traditional governance from inside enterprises, primarily on the role of a business function as the producer and owner of data, to a steward on behalf of the company.
2. Regulators are exploring moving from passive privacy and consent management regimes to active ones.
Most global data privacy regulations have been dependent on a trust model: companies will explicitly describe how customer data will be used in a privacy policy; customers will be educated, read the privacy policy and make an informed consent decision on how their data will be used; and, companies will abide by the data privacy policy and use data in-line with what was agreed. In practice, none of the three core assumptions behind privacy regulations have operated as planned. Privacy policies bury critical and important details using legalese that protects the companies’ interest and often obfuscates intentions for how data will be used; customers don’t read privacy policies and terms and conditions agreements, and blindly provide consent for uses they may not have otherwise agreed with; and companies often go further and use data beyond what was described in the privacy policy. In fact, in most companies the technical controls for ensuring that data is used in-line with approved privacy policies is non-existent or not up to the challenge of managing customer data effectively.
With penalties and fines not being seen as enough of an impediment to stop customer data abuses, regulators are starting to experiment with new regulations that move data from a passive control model, based on trust, to an active control model, based on zero trust. This means that as technology evolves, the expectation will be that the customer has direct access and control to (1) view what customer data is being saved by a company; and (2) actively define and control how their data is being used – with direct accountability within the data access layers of the underlying technology. Instead of requesting to be forgotten, in this active consent model, the customer themselves would have the ability to explicitly hide their data to certain business functions (as long as permitted by law and policy).
3. The IT complexity of data storage, processing and analytics is reaching its breaking point.
The current methods of storing and analyzing data have typically involved data warehouses and data lakes (i.e., originally using technologies like Hadoop). With the advent of cloud computing, data lakes and warehouses have been able to move from on-premises to the Cloud to take advantage of scale economics (think Snowflake, Databricks, Azure Synapse, Amazon Athena to name a few). All of the major cloud service providers now offer a set of robust data capabilities – storage, technical meta-data management, pipelines, warehousing, data science workbenches, etc.
The challenge is that in most large enterprises, data is now being copied and proliferating into multiple disparate on-prem data marts and data lakes across multiple cloud providers. The role of the Chief Data Officer is becoming ever more complicated as they need to manage data across more and more environments in order to support cross-functional analytics. And the more data is being copied, the greater the risk of data fidelity, integrity and quality issues, along with risk of data leakage and cybertheft.
The latest technology innovations to try and manage data across all of these new environments include data fabrics, or data meshes. However, data fabrics are only exacerbating the complexity of the data estate and do not get to the root cause of why the data ecosystem is continuously expanding. In fact, we’re seeing exponential increases in capital and operating expenses as fabrics get deployed into production. New technologies will emerge which will get to the core of data reusability and dramatically change the landscape and conversation. This will be more of a return to better, smarter uses of reusable ‘Small Data’ components versus continuing to proliferate more and more ‘Big Data.’
4. Web 3.0 decentralized processes and technologies are becoming more mainstream, beyond NFTs and crypto.
Finally, we’re seeing the rise of new business models that take advantage of Web 3.0 concepts and capabilities. As a quick reminder, Web 3.0 introduces several novel ideas for diffusing content ownership from platforms to individual users, so that the users can both have more control as well as partake in the financial value of their own content. These include ideas such as distributed ledgers, cryptographic blockchains, smart contracts, token-based economics and verified credentials among others. While these concepts are typically associated with decentralized ownership for things like currencies or tokens (NFTs), in fact they are being used to power interesting new business applications and utilities. For example, rather than having one dominant company define a proprietary standard for how to exchange information among partners in a supply chain, all of the major players – from the ordering entity, to the truck, to the vessel, to the port, to the warehouse, to the train, to the freight forwarder, to the final truck for delivery can all cooperate in a participative, decentralized network that can share and communicate data in real-time. Through token economics, participants in the chain can be compensated for playing nicely together and defining and maintaining their own standards for interoperability.
We’re seeing these novel models now being focused inwards to enable business functions inside a company to collaborate more effectively in creating and sharing information. Most functions speak to each other, from a technical vantage, using traditional, Web 2.0 methods of communication (e.g., sharing data through message queues, event grids or copying and transmitting files through SFTP). The latest industry buzzword is “Web 5.0”, which is meant to reflect that companies internally will adopt Web 2.0 + Web 3.0 technologies to facilitate information exchange and develop novel methods for data governance and management with unique methods of encouraging collaborative and constructive behavior.
So what do these four trends portend when you put it all together? Ultimately we see new emerging database technologies for saving, sharing and protecting data that will address the business, regulatory and technical drivers discussed above. This includes how data is saved in relational models, to how data is analyzed in warehouses and lakes, to how data is shared in integration platforms (like message queues and event buses). These technologies will simplify the IT environment, and also enable new methods of access control, such as enabling the use of smart contracts on data to let customers actively define policies for how their data can be used. Fluree is on the leading edge of addressing customer challenges across the four dimensions described above, and is baking it into the core of its new Fluree DB and universal data store.
##
ABOUT THE AUTHOR
Eliud Polanco, President of Fluree
Eliud has over 15 years of experience in data management, analytics, business and technology strategy for financial institutions. He has served as the Head of Analytics and Big Data Strategy at such global institutions as Citigroup, HSBC, Deutsche Bank, and Scotiabank. He has lived through myriad data science challenges as a client, which led him to start ZettaLabs! He is a proud but lately disappointed Yankee fan.
