Industry executives and experts share their predictions for 2018.  Read them in this 10th annual VMblog.com series exclusive.

Contributed by Brad Surak, Chief Product and Strategy Officer, Hitachi Vantara

2018 Predictions for Big Data and IoT Systems

With the start of a new year comes new uncertainties, and as the world progresses we find ourselves looking to technology more and more for answers about the future. In a time when the "new normal" is anything but, technology can be a helpful partner to guide our decision-making when it comes to everything from driving economic growth to overcoming societal challenges such as security threats and breaches to changes in governments and global agreements.  It's more important than ever that businesses are vigilant and embrace new technological innovations and ‘what-if' scenario planning in order to protect themselves and customers. As new enterprise challenges continue to arise in 2018, IT and operational systems will need to be prepared.

As we steam ahead into the new year, here are a four predictions how technology and innovation will influence big data and IoT systems in 2018:

The rise of automated machine learning

In 2018, we'll see more Artificial Intelligence (AI) techniques pushing the boundaries between reinforcement learning, unsupervised learning and autogenerated training-especially in IoT environments where algorithms and bots will enable edge devices to learn on their own. With a shortage of talented data scientists with the advanced skillsets to label training datasets from sensors and other systems for AI, machine learning (ML) or deep learning (a messy, expensive and time-consuming process in itself), a supervised learning model is impractical. The bandwidth required to submit data from the edge, while enabling programmed, unsupervised learning on real-world data, is still expensive, but automated machine learning can reduce costs. While automated ML is more efficient, this concept raises other, more ethical questions around how algorithms make decisions and who is responsible for those decisions and their consequences.

Algorithmic transparency will start to become a competitive differentiator

Visibility into how algorithms make decisions will become increasingly important with the acceleration into automated data science. Algorithms will start playing a larger role in areas affecting people's lives and livelihood in 2018. This means more people will demand transparency on how the algorithms are created. Companies marketing their algorithmic transparency in 2018 as a key, competitive differentiator will stand out.

Beware of fake data scientists

The big data skills shortage is creating a talent pool of "fake" data scientists-better known as opportunists taking advantage of the poorly defined data scientist role, knowing only that the job comes with a big paycheck. Companies lacking the necessary hiring resources should interview candidates carefully to spot those with real potential. The "holy trinity" of data science fundamental skills to look for include: strong problem-solving skills, an advanced degree in applied mathematics or statistics, familiarity with statistical analysis technologies like Python or R along and knowledge of big data technologies like Hadoop and Spark.

Edge devices, open data and security will bring IoT and OT closer together

As the IoT ecosystem continues to advance in 2018-with things like edge devices getting smarter-IT and operational technology (OT) systems will move closer together. We can also expect to see more hardware manufacturers implementing open data policies. This shift is driven by the manufacturers' need to move to new service models and optimize hardware devices, resulting in them opening up their systems. This means an increased impact on security. As such, we'll see priorities move from data protection and access denial to securing system operations. Organizations embracing these developments will become more IoT-enabled.

Despite uncertain times ahead, we can take comfort in knowing innovations in big data and IoT will continue to advance. Businesses embracing these technologies will be better situated to adapt-and even thrive-in this ever-changing global climate.

##

About the Author

Brad Surak, Chief Product and Strategy Officer

Brad Surak joins Hitachi Vantara with over 25 years of success in digital transformation, general management, product and solution development, software engineering and operations across multiple industries and geographies. He has demonstrated success in driving global digital transformations that deliver top-line growth and productivity. Prior to joining Hitachi, Surak led the formation of GE Digital, a General Electric business, and served as chief operating officer. An "intra-preneur," Surak comes with a proven history in building successful businesses within large multinational companies such as SAP and Business Objects. He has also held executive positions with DayNine, Cambridge Technology Partners, and Ernst & Young's technology practice. Surak holds a Bachelor of Science degree in Computer Science from Purdue University and was awarded the title of Distinguished Science Alumni in 2017.