Virtualization Technology News and Information
MapR 2018 Predictions: IoT Deployments Expose the Need for a Streams-Based Approach to Data Governance

VMblog Predictions 2018

Industry executives and experts share their predictions for 2018.  Read them in this 10th annual series exclusive.

Contributed by Mitesh Shah, Director, Product Marketing with MapR

IoT Deployments Expose the Need for a Streams-Based Approach to Data Governance

IoT and Data Governance are two terms not often used together today. This will change in 2018, as increasing numbers of mission-critical IoT deployments yield results whose sources and inputs need to be traceable. For example, it will be insufficient for models to say "I predict an equipment failure in oil rig A within the next 5 days." The model will instead need to say, "I predict an equipment failure in oil rig A within the next 5 days because of input received yesterday from sensors #1 and #3 on oil rig A combined with weather forecast data." Verifying the lineage of IoT sources and the data transformations leading to the end results are increasingly critical for organizations. At the same time, traditional methods addressing this particular data governance challenge are not well-suited to handle the crush of metadata coming from IoT deployments. Organizations will quickly realize a streams-based model to solving for lineage in an IoT world is not just a good idea, it is a requirement.

Over the past decade we have witnessed at least two trends in IoT. First, use cases are moving from experimentation to full-scale production for mission-critical deployments. Hospitals are using IoT to help with patient diagnoses and preventive healthcare needs. Automobile manufacturers are leveraging IoT extensively in self-driving car initiatives. Oil and gas companies are using IoT for predictive maintenance to reduce costs significantly and keep critical equipment operational. The second trend is the explosion in data volume. As the number and types of IoT deployments increase, so too does the sheer volume of data generated from these devices. Estimates at the low end call for nearly 25 billion IoT devices by 2020, with global IP traffic - much of it coming from IoT deployments - already passing the zettabyte scale.

Governance over these IoT deployments is critical in the years ahead. In particular, organizations need to know and verify which devices and intermediate data transformations were used to arrive at the end results - results that in some cases are being used to save lives. Traditionally, this type of lineage metadata has been stored and analyzed in RDBMS systems that cannot easily scale out. The rise of IoT showcases the need for a new architecture that better handles the onslaught of data and metadata generated, as well as better supports the publication and consumption of this data by downstream devices and analytics engines. A streams-based architecture is the obvious choice to support IoT lineage requirements going forward.

MapR-ES is an example of a global publish-subscribe event streaming system for big data and IoT. It is the first big data scale streaming system built into a converged data platform and the only big data streaming system to support infinite data persistence and global event replication reliably at IoT scale.

Organizations will realize in 2018 that their mission-critical IoT deployments warrant a streams-based model to solve for lineage and governance requirements.  MapR-ES will emerge is an ideal solution to address this need.


About the Author

Mitesh Shah 

Mitesh Shah is Director, Product Marketing with MapR and is responsible for industry solutions. Prior to MapR, Mitesh held positions in enterprise security at organizations including the Federal Reserve and Mitesh has a degree in computer science from Cornell and an MBA from The Wharton School of the University of Pennsylvania. You can contact the author at

Published Tuesday, January 02, 2018 7:28 AM by David Marshall
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<January 2018>