Virtualization Technology News and Information
MapR 2017 Predictions: Cloud and Virtualization Will Drive the Next Wave of Streaming Deployments

VMblog Predictions 2017

Virtualization and Cloud executives share their predictions for 2017.  Read them in this 9th annual series exclusive.

Contributed by Dale Kim, senior director, Industry Solutions, MapR

Cloud and Virtualization Will Drive the Next Wave of Streaming Deployments

Organizations will continue to look at a mix of public and private cloud deployments, and it's this mix that will drive significant adoption of streaming data architectures. In particular, the use of the cloud as a means of distributing global data will benefit from streaming.

In recent years the implementation of real-time processing, often in the form of event streaming, has been an extremely hot topic. The ability to analyze and process data as it is delivered is especially important in environments where reaction time to new data is critical, such as for fraud prevention, cyber threat analysis, and manufacturing quality assurance. Interest in real-time processing has grown hand-in-hand with streaming technologies like Apache Kafka and MapR Streams. These modern "publish-subscribe frameworks" are ideal for topologies that have data producers and data consumers in a many-to-many configuration.

Event streaming is now becoming more relevant for any data architecture, even those that don't necessarily have real-time requirements. Thinking about data in terms of streams (or "flows") instead of persistent data ("state") is promoting the increased use of microservices. Event-driven microservices environments are useful today because they simplify application development and deployment by breaking down large solutions into smaller, easier-to-maintain, and reusable tasks. Each of these tasks communicate in a streams-based data pipeline run by a streaming engine like the publish-subscribe systems mentioned earlier. It is relatively easy to swap in new services at each stage of a pipeline, or to add new services, making this architecture idea for agile environments.

Microservices fit well with the principles of cloud and virtualization, especially when it comes to agility, elasticity, and isolation. Whenever a change is needed in the architecture, a new task can be written and then quickly spun up in a new virtual node instance. As data and/or user load grows, bottlenecks are easier to identify and can be resolved by quickly and independently scaling out that portion of the data pipeline. And isolation is useful when testing out new tasks in an existing pipeline without impacting current operations.

As organizations deploy data centers across the world, whether physical or virtual, they need a means for delivering data from many sources to many targets. They will look to streaming technologies as the core framework for enabling their global portfolio of data centers. Streaming enables continuous, coordinated data flows, which is just one of the primary requirements of global cloud processing. Other requirements include management of a single global namespace, location awareness, global consistency, omni-directional replication, and standard APIs, but that's a discussion for a future blog post.


About the Author

Dale Kim, senior director, Industry Solutions, MapR

Dale Kim is the Sr. Director of Industry Solutions at MapR. His background includes a variety of technical and management roles at information technology companies. While his experience includes work with relational databases, much of his career pertains to non-relational data in the areas of search, content management, and NoSQL, and includes senior roles in technical marketing, sales engineering, and support engineering. Dale holds an MBA from Santa Clara University, and a BA in Computer Science from the University of California, Berkeley

Dale Kim 

Published Tuesday, December 06, 2016 7:02 AM by David Marshall
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<December 2016>