Qubole, the big data-as-a-service company, today announced it is building the industry's first autonomous data platform. This next-generation data platform includes three new products: Qubole Data Service (QDS) Community Edition, QDS Enterprise Edition and QDS Cloud Agents. The solution can intelligently automate and analyze platform usage to make data teams more effective. Qubole's new platform and products are being demonstrated at Data Platforms 2017, the industry's first conference focused exclusively on helping data teams build modern, self-service big data platforms. 

As data use continues to proliferate in the enterprise, organizations face immense challenges when it comes to effectively collecting, processing and analyzing data at scale. Gartner predicts that by 2018, 70 percent of big data projects will fail.¹ The complexity of big data coupled with a skills shortage for data engineers, data scientists and data analysts, increases the need to automate unnecessary manual effort and augment human tasks in order for organizations to scale big data efforts.

Through the launch of QDS Community Edition, Enterprise Edition and Cloud Agents, Qubole is building an autonomous data platform with intelligent automation to address the difficulty enterprises have in scaling their data teams and initiatives. The new QDS platform self-manages, self-optimizes and learns by watching platform usage. The architecture analyzes metadata (queries, clusters, users, data, etc.) generated by platform usage and applies machine learning and artificial intelligence to create alerts, insights, recommendations and autonomous agents which perform actions automatically. The QDS platform is the only solution that intelligently automates and analyzes usage to learn how to make data teams more effective.

"Despite all the promise and technological advancements, operationalizing big data efforts is still impossible for most organizations. Data teams simply can't scale to meet the growing demand for data across organizations. To address this hurdle, we are on a mission to remove the manual effort that comes with maintaining a big data infrastructure and empower data teams to focus on high-value, strategic work," said Ashish Thusoo, co-founder and CEO of Qubole. "With the release of Qubole's autonomous data platform, data teams can optimize cost, performance and reliability by intelligently automating tedious manual data management tasks - helping organizations gain the full value of their data."

QDS Enterprise Includes Alerts, Insights and Recommendations

The new QDS Enterprise Edition replaces the existing QDS solution and is the first data platform solution to offer Alerts, Insights and Recommendations (AIR) that provide intelligent, actionable information to data professionals:

●       Alerts provide proactive notifications to DataOps and data engineers to identify real-time situations that require immediate attention. 

●       Insights provide analytics to data professionals that help illuminate data and platform usage to improve performance, cost efficiency and reliability. 

●       Recommendations provide prescriptive actions that data professionals can perform to improve performance, cost efficiency and/or reliability

QDS' new machine learning infrastructure enables AIR that can be rule-based, workload aware and/or predictive and increasingly accurate by learning from user behavior over time.

"QDS AIR is a great example of how to collect and analyze data and present it very elegantly in the places where you need it most," said Santanu Dey, director of data science and engineering, Fanatics. "With Qubole, we have been able to discover and fix underlying issues with SQLs, tables, joins, partitions, and formats, among other issues, that we never knew existed."

QDS Cloud Agents Offer AI-Powered Automation

QDS Cloud Agents is the first product that can autonomously execute a range of data management tasks that are typically time-consuming, manual tasks. The QDS Cloud Agents include a range of cloud-based software agents that offload tasks from human data professionals, based on a set policy, that can reduce cost, offload manual effort and improve reliability. The initial release of QDS Cloud Agents includes the following three agents:

●       Workload Aware Auto-Scaling Agent: Optimizes cluster size precisely to workload requirements, reducing the over-provisioning typically seen with alternative solutions. The agent is workload-aware and manages scaling up and down dynamically based on actual processing load to reduce overall compute costs by up to 50 percent.

●       Spot Shopper Agent (AWS Only): Intelligently shops across AWS cloud to assemble the compute instances in the optimal combination of performance and cost. This can include AWS Spot Instances and Spot Blocks, mixed instance types, different availability zones and provisioned dynamically at run time. The Spot Shopper can reduce compute resource costs by up to 80 percent vs on-demand instances, and up to 50 percent more than commodity spot instance solutions.

●       Data Caching Agent: This agent optimizes the locality of your data for fast, interactive access speeds. Data Transporter takes into consideration which data sets are frequently accessed and intelligently moves data in the background for the best performance.

QDS Cloud Agents is an optional add-on to the QDS Enterprise edition. The individual agents are available for AWS, Microsoft and Oracle BMC clouds, with the exception of the Spot Agent.

QDS Community Provides Developers with Unlimited Access

QDS Community is a free product, providing the same functionality as QDS Enterprise. Community is free forever for up to four nodes and five clusters. This solution is ideal for developers interested in building and testing applications before moving to production. 

QDS Community, QDS Enterprise and QDS Cloud Agents are available in July 2017. Visit www.qubole.com to learn more.