Photo Courtesy of The IT Press Tour
In an era where data is the new oil, companies are grappling with the challenge of efficiently storing, processing, and analyzing massive volumes of log data. Enter Hydrolix, a pioneering streaming data lake company that’s turning the economics of log data on its head. At the recent 58th IT Press Tour in Boston, Hydrolix executives Marty Kagan (CEO & Co-founder) and Antony Falco (VP Marketing) unveiled their groundbreaking approach to handling big data, promising a future where companies no longer need to compromise between data retention and cost-effectiveness.
The Genesis of Hydrolix: Born from Real-World Challenges
Hydrolix’s journey began with a problem that many in the tech industry face: the exponential growth of data and the associated costs of storing and processing it. Marty Kagan shared his experience from a previous venture, stating, “We were generating close to 23 billion records a day, and the infrastructure required to ingest and store and search that data was tremendously extensive.” This pain point became the catalyst for Hydrolix’s creation, with the vision of revolutionizing how companies handle data at scale.
Reimagining the Data Lake: Hydrolix’s Unique Approach
At its core, Hydrolix is a streaming data lake, but it’s far from conventional. The company has engineered a solution that combines the flexibility of traditional data lakes with optimizations specifically tailored for real-time log streams.
Key features include:
- Scalable Streaming Ingest: Capable of handling hundreds of terabytes of data daily
- Real-time ETL: Processing log streams on the fly
- Multi-source Integration: Combining various log sources into a single table
- SQL and Spark Query Support: Enabling both traditional and data science approaches
- Extended Hot Storage: Keeping data accessible for years, not just days
Antony Falco emphasized the significance of their approach, noting, “We want to avoid those problems and enable people to have access to all of their information. The data is very valuable, and up until now, there’s been this huge pressure on organizations to throw the data away to make it cost-effective.”
Breaking Down the Architecture: Decoupled for Optimal Performance
Hydrolix’s architecture stands out for its decoupled nature. The system is divided into three main components:
- Ingest Layer: Receives or pulls data from various sources
- Core Component: Handles indexing and querying
- Storage Layer: Utilizes cloud-compatible object storage
This decoupled design allows for independent scaling of each component, providing unparalleled flexibility and cost-efficiency. As Hydrolix explains it, you can add more query capacity if you need to be able to answer the query faster. If you have a big event coming up and you have a lot of data coming in, it will automatically upscale the ingest component.
Real-World Impact: Paramount’s Super Bowl Success
To illustrate the power of their platform, Hydrolix shared a case study involving Paramount’s handling of the Super Bowl. During this massive event, Hydrolix processed an astounding 20 million events per second in real-time. This capability not only provided Paramount with immediate insights but also allowed for long-term data retention for year-over-year analysis.
Sean McCarthy, Director of Product Management at Paramount, praised the solution: “We’ve addressed cost, performance, and scale challenges and as a result we have a much more robust platform.”
The Quesma Partnership: Bridging the Gap to Kibana
In a move that further expands Hydrolix’s ecosystem, the company announced a partnership with Quesma, a startup specializing in database migration solutions. This collaboration introduces a game-changing capability: enabling Kibana dashboards to work seamlessly with Hydrolix’s backend.
Pawel Brzoska, COO and Co-founder of Quesma, explained the significance: “We want to work together with Hydrolix to help bring more elastic customers. Save them money, save them time, save them hassle on managing large scales of data in the database.”
This integration allows organizations to leverage the familiar Kibana interface while benefiting from Hydrolix’s cost-effective and scalable backend. It’s a win-win situation that addresses the pain points of many Elasticsearch users struggling with rising costs and scalability issues.
Beyond Storage: Enabling Advanced Analytics and AI
While Hydrolix primarily positions itself as a data storage and processing solution, its implications for advanced analytics and AI are profound. The ability to retain vast amounts of raw data over extended periods opens up new possibilities for machine learning and predictive analytics.
Marty Kagan highlighted this potential, explaining there’s a need to retain data for longer and to store that raw data and not just the aggregate data. That’s driving the demand for longer data retention and more raw data retention.
This capability is particularly crucial for use cases like bot detection, fraud prevention, and cybersecurity, where historical data can provide invaluable insights.
The Road Ahead: Expanding Horizons and Partnerships
As Hydrolix looks to the future, the company is focused on growth and expanding its ecosystem. With plans to reach 140 employees by the end of the year and a growing presence in global markets like India, Japan, and Singapore, Hydrolix is poised for significant expansion.
The company is also exploring partnerships with major cloud providers beyond their existing relationship with Akamai. As Marty Kagan put it, “We think about Akamai more as a major cloud provider.” Akamai bought Linode, and they are growing their billion-dollar business now in the cloud.
Conclusion: Redefining the Economics of Big Data
Hydrolix is not just another data lake solution; it’s a paradigm shift in how companies approach log data management. By solving the fundamental challenges of scale, cost, and accessibility, Hydrolix is enabling organizations to unleash the full potential of their data without breaking the bank.
As data volumes continue to explode and the demand for real-time insights grows, solutions like Hydrolix will become increasingly critical. The company’s innovative approach, coupled with strategic partnerships like the one’s with Quesma and Akamai, positions it at the forefront of the big data revolution.
In a world where data is king, Hydrolix is providing the keys to the kingdom, allowing companies to store more, analyze faster, and innovate without limits. As they continue to expand their reach and capabilities, Hydrolix is undoubtedly a company to watch in the evolving landscape of data management and analytics.
##
