By Paul Moxon, SVP Data Architecture and Chief Evangelist at Denodo
The data lakehouse is one of the most reliable data architectures in use today, promising to combine the scalability and flexibility of data lakes – in their ability to handle practically unlimited volumes of unstructured data – with the proven capabilities of data warehouses to store and analyze structured data.
However, in this age of Artificial Intelligence (AI), the Internet of Things (IoT), and increasingly complex analytical expectations, the data lakehouse architecture has proven to have a few gaps. This is to be expected, as no one architecture can meet all current and future needs. Rather, to be fully responsive to the broadest of today’s and tomorrow’s analytics needs, organizations should employ multiple data architectures that work in synergy. Here, I’ll cover the role of logical data management platforms in working with data lakehouses to overcome their gaps and extend their capabilities.
The Gaps
To start off, the data lakehouse plays a central role as an organization’s primary data store and data processing engine but it lacks an overarching semantic layer. This means that within a data lakehouse, the semantics of different applications will not be standardized, inhibiting analytics across these applications and preventing organizations from delivering “AI-ready” data to AI and generative AI (GenAI) applications. By “AI-ready” data, I’m referring to data that is trustworthy, reliable, and delivered in real time, among other qualities determined by each organization.
Along that line, the lack of an overarching semantic layer prevents personalized data self-service. Without such a layer, business analysts and data scientists will not only be unable to analyze all of the available data in a lakehouse, but they will also struggle to even find and access the data they need, due to an inherently fragmented search and discovery process that could involve multiple searches across multiple siloed applications.
Silos within a data lakehouse also prevent organizations from establishing federated data governance capabilities across a data lakehouse, with centralized oversight of the entire data estate. Without these capabilities, organizations naturally have to struggle with regulatory compliance, fine-grained data security, and providing AI applications with governed, trusted data. Without these capabilities, organizations cannot be certain that their AI applications comply with privacy regulations and related requirements.
Note that though each of these three gaps refer to the data within a data lakehouse, these gaps are compounded when you consider that some data will always reside outside of the data lakehouse, for numerous reasons that include mergers and acquisitions activity (though temporary), data that needs to remain in a silo, due to data export regulations, or multi-cloud configurations. That is, though data lakehouses promise to be the one data repository that an organization will ever need, it is unlikely that that will prove to be true.
Finally, data lakehouses are unable to deliver real-time, unified data while controlling costs. Data is brought into a data lakehouse during scheduled loads, and this schedule dictates the extent to which data is up-to-date. Real-time synchronization is possible, but this requires perpetual replication processes, which can be costly.
Introducing Logical Data Management
Logical data management is a data management strategy that provides an alternative to the traditional approach, which relies on physically replicating disparate data into a common repository before it can be managed. Traditional approaches might rely on extract, transform, and load (ETL) processes to not only move data into a data lakehouse but also to integrate it between the multiple data silos within a data lakehouse.
In contrast, logical approaches use data virtualization to enable real-time connections to data, without requiring replication. Here’s how logical data management can overcome each of the four gaps:
1. An Overarching Semantic Layer
- Logical data management platforms enable a unified semantic layer above the data lakehouse and any supporting data sources, standardizing the semantics to improve search. Such semantic layers can contain rich metadata that clarifies the relationships between different datasets, includes documentation about usage, and provides business context so that both people and AI applications can leverage the data more effectively and gain deeper insights.
2. Personalized Data Self Service
- Logical data management platforms also enable self-service data marketplaces that let users explore the available data, perform any needed data preparations using an intuitive interface, and easily consume the data as if they were shopping at an online store. Such marketplaces can provide real-time contextualization, so as to become a trusted source of high-quality data for AI and analytics applications. They can also deliver data products that can be easily customized to meet the needs of different users, without requiring IT support.
3. Federated Data Governance with Centralized Oversight
- By enabling access to real-time, semantically unified data across disparate data sources, logical data management also enables organizations to establish a federated data governance model across the lakehouse and any other applicable data sources, a model that can nonetheless be managed centrally. This enables fine-grained access control, with global security policies that can flexibly adapt to changing regulatory and organizational requirements, without having to implement changes in multiple locations. It also enables privacy-compliant AI and analytics, to reduce risk while improving data usability and trust.
4. Real-Time, Unified Data Delivery, without Additional Cost
- Logical data management platforms can add real-time, unified data delivery capabilities to a data lakehouse without incurring additional costs in the form of time, expense, or complexity. Logical platforms deliver unified views of up-to-date data, querying original source systems in real-time only as needed for the given operational initiative, and avoiding expensive replication whenever possible. Advanced logical data management solutions can continuously optimize queries for both performance and cost-effectiveness and provide a dashboard to monitor data usage and associated costs.
The Next-Generation Data Lakehouse
Supported by a logical data management platform, data lakehouses can overcome all of the four gaps I covered here, so that they can become next-generation data lakehouses that enable faster, better insights for analytics and improved AI outcomes, all while reducing the costs of unnecessary replication.
##
ABOUT THE AUTHOR
Paul Moxon is the SVP Data Architecture and Chief Evangelist at Denodo. He works with customers to help them understand the benefits of data virtualization and advises them on how the Denodo Platform fits into their information architecture. Paul has over 30 years of experience with enterprise middleware technologies with leading software companies such as BEA Systems and Progress Software. Paul has a bachelor�s degree in Business Administration from Northumbria University in the UK.
