Virtualization Technology News and Information
DataStax 2017 Predictions: Why we'll all be engineers, not scientists, when it comes to data

VMblog Predictions 2017

Virtualization and Cloud executives share their predictions for 2017.  Read them in this 9th annual series exclusive.

Contributed by Patrick McFadin, Chief Evangelist, DataStax

Why we'll all be engineers, not scientists, when it comes to data

My predictions are around the kinds of jobs that IT professionals will be doing, why they will change, and how this will be reflected in what companies want to achieve in the next year.

Personally, I think the "Data Scientist" role will start to become less relevant. Why?

Previously, Data Scientists were brought in to analyse data in a reactive manner. Companies had bet on stuffing data into Hadoop and then needed help to actually make that data useful. Tasks like analysing company data and making recommendations needed specialty skills, and companies were prepared to pay for them.

In 2017, we'll see less of those kinds of roles coming up. What will replace them will be Data Engineers. The reason for this is that more data is getting embedded into applications rather than being pushed into a central data lake then analysed. The role of the Data Engineer is to proactively use data to improve existing technologies and services. As companies make more use of streaming analytics to look at data when it is being created and then recommend an action, it's important that there are people looking at these outputs.

This role is more of an engineering role, as it should focus on what the company needs to achieve and how to get this done. It's a much more goal-oriented, "fix the problem with the right tools" approach compared to delving into data to see what is there.

Alongside this, I think we'll see much more attention given to transactional data being analysed rather than operational data. Transactional data is the information created when you buy something or make a change in your behaviour. This is different to operational data, which is the collection of all your activities alongside everyone else's. Looking at these transactions as they take place needs a different approach. For cloud applications to be successful at delivering this real-time value at enterprise scale, how the app stores data must cover all three of the following:

  • Data is distributed - it can come from numerous endpoints, it can be stored across multiple locations, data centres or cloud services, and it must scale in a linear way;
  • The application must be responsive - this means low latency in performance, with no downtime or interruptions to service;
  • The service itself should be intelligent - that is, analytics and decisions should be carried out alongside the transaction, across multiple data models and mixed workloads.

The reason for this change is how companies will look at customer experience.

Personalisation is a very important part of the customer experience that companies need to ensure they think about in more detail. For example, it's a fight for survival against the likes of Amazon in the retail industry.

Customers need to get delight and pleasure from the shopping experience and end product equally. If companies aren't thinking of their customers, they will move on. If competitors are providing a better customer experience, it becomes a C-suite problem as it affects the bottom line of the business.

Now, this has to go beyond simply looking at what people bought and sending them an email a week later. This has to be in the moment. So as someone is making a purchase, what might fit their needs? What have they not considered, or what would fit really well? Doing this in real time is harder, but data engineering has the most potential to make a difference to customers and to companies as well.


About the Author

Patrick McFadin is Chief Evangelist at DataStax, supporting the company's work with developers building cloud applications at scale. Prior to DataStax, Patrick held roles in the software development and data management industries.

Patrick McFadin 

Published Thursday, December 22, 2016 9:06 AM by David Marshall
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<December 2016>