Virtualization and Cloud executives share their predictions for 2016.  Read them in this 8th Annual VMblog.com series exclusive.

Contributed by Sean Ma, Director of Product Management at Trifacta

Big data makes it to the front office and transforms the value and framework of the analytics industry in 2016

The "big data" space is still maturing - continuing to define its vision while simultaneously converting aspects of that vision into reality. There is a natural progression to this development that amounts to answering the following questions in order: Can we build tools that make working with data easier, faster, etc...? Can we make these tools reliable and robust? Can we build processes around the use of these tools that safeguard their use and that provide consistent value?

By drawing analogies from other mature product spaces, these questions are answered in three phases. First, vendors build early stage products demonstrating that viable solutions exist. As early customers adopt these products, we move into the second phase where vendors make these products robust and reliable, and customers start experiencing their value. The third phase wraps these products into workflows and organizational structures that consistently deliver value and can scale to multiple teams or departments.

In 2015, vendors were largely focused on hardening their tools as they moved beyond early stage products, and big data projects were still tightly controlled and managed by IT. In 2016, the front office (business analysts, managers and executives) will adopt technology to solve specific business challenges or opportunities, the scale of which will force vendors to extend their offerings to manage end-to-end solutions. This forces vendors to develop or integrate with existing tools and provide best practices for how customers should orchestrate their use and coexistence. Based on that framework, we think 2016 will show:

Industry specific solutions will significantly increase:

• Vendors, like Cloudera, will work with ecosystem partners to develop and leverage solution "blueprints" to demonstrate actual value to their customers in specific verticals. These solution blueprints based on actual, working, successful customers will provide both IT and LOB users with the confidence to invest or expand upon their big data initiatives. We will see most of this value surface in exploratory or innovation projects. The difference is that companies will not require a back-office black-ops team of data scientists to achieve this value. As a greater number of analysts have access to the appropriate tooling to make use of big data and gain a better understanding of how to apply these tools, the next step will be to customize early prototypes for specific business initiatives.
  

Increasing the amount of deployments will force vendors to focus their efforts on building and marketing management tools:

• For projects to operationalize at big data scale, vendors can no longer hold customers' hands throughout the duration of a big data project. Once customers experience initial value, vendors will take the next step by developing tools and best practices for mass productionization of valuable big data solutions in a consistent and repeatable fashion.
  

Much of the functionality in these tools as they gain broader adoption will need to  replicate functionality in analogous tools from the enterprise data warehouse space, specifically in the metadata management and workflow orchestration. There will be several key differences with these new big data solutions compared to their traditional counterparts:

o   Ability to handle a wider variety of data both in terms of content and structure / semi-structured data.

o   The agility to handle frequent changes in that data, which is something that traditional enterprise data warehouses struggle with.

o   Exposing metadata, traditionally locked away in technical tools, to non-IT users in ways that improve data analysts usage and understanding of their data.

Front-office business stakeholders will demand self-service access to big data platforms which will drive organizations to adopt a hub-and-spoke IT model to support a wider variety of data-driven initiatives:

• Rather than having IT in one large centralized team, companies will complement a smaller centralized IT team with individual technical teams to collaborate with each line-of-business group. This allows for a two-tier support structure with universal needs serviced centrally and specialized needs serviced in-situ for speed, goodness-of-fit, consistency and efficiency. This will initially complicate the go-to-market strategies of big data vendors who need to develop marketing and sales mechanisms to target newly formed LOB-IT groups.
  

Beyond 2016? With new organizational structures and workflow management tools in place, many companies will be in a place to democratize access to their data and to reap the benefits of a data innovation funnel that starts with broad, lightweight exploration and continues to data curation and data pipeline productionalization. 

The next challenge? Data governance for Big Data in this new self-service world. 2016 will have built out all the building blocks necessary, but balancing top-down and bottom-up data governance processes within organizations will be critical to driving continued innovation from big data?

##

Author the Author

Sean Ma, Director of Product Management at Trifacta.

With over 10+ years of experience in enterprise data management software, Sean has spent the last 5 years leading Big Data products at companies such as Informatica and Trifacta. He holds a Bachelor of Science degree in Electrical Engineering and Computer Science from the University of California Berkeley.