By Paul Brebner, Chief Technology Evangelist, Instaclustr

Data Gets More Complex - But Also More Wieldy

A storyline throughout 2020 will be increasingly sophisticated - and more dynamic - data platforms and applications. This in turn will enable better delivery of more demanding (and also more in-demand) load applications, such as those tapping complex geospatial data.

Here are three trends I expect to see this coming year:

1. Open source monitoring helps DevOps teams see the light

There's a big challenge right now around observability into complex distributed data applications (an example I'm well-familiar with: Cassandra and Kafka cluster monitoring, including servers, JVM, clusters, etc). But end-to-end visibility of the complete stack is increasingly (and rightfully) critical for the complete set of DevOps' responsibilities (building, testing, debugging, tuning, etc).

In 2020 I think we'll see more adoption of (and comfort with) particularly matured open source monitoring and tracing tools that can provide this much-sought visibility. Among those trending upward in 2020 in this category are Prometheus, and OpenTracing, as well as Kubernetes Operators for application monitoring.

I think we'll see that, used in conjunction with managed service providers, monitoring and management APIs give enterprise DevOps more powerful, flexible and (increasingly) proactive control and scaling of their data applications across hybrid cloud platforms. I predict this more elastic scaling of cluster resources (e.g. monitoring combined with management APIs) will bring about easier autoscaling of resources up and down to cope with changes to application workloads and will reduce costs for more enterprises throughout 2020.

2. Geospatial data makes it all about (how to leverage) location, location, location

The spotlight is increasing on geospatial data - and it's only going to get brighter in 2020. I recently attended ApacheCon and one of the hottest tracks was Geospatial, with no shortage of unique use cases of how enterprises are leveraging large-scale (and fast) geospatial data processing.

The trend is taking off as companies realize that just about everyone needs geospatial data. In some cases, geospatial location information is inherent to the data. In other datasets the information is added. But either way, the data must be processed in large quantities - and quickly - to reveal vital spatiotemporal insights vital to businesses and their end users. A good example is Uber providing continuous forecasts of future demand trends to drivers.

In the rush to tap into geospatial data, though, I expect many enterprises will find that indexing and querying geospatially-enriched data is more complex than they anticipated - and might bring about tradeoffs between speed, throughput, and accuracy. It's something I've been experimenting with throughout this year (leveraging Apache Cassandra as a way to run geospatial queries at scale - and even testing with 3D geospatial data to really put the open source database to the test).

3. Can Kubernetes get any hotter? Yes - and here's why

You don't have to count the hotel rooms booked at KubeCon this November to know that Kubernetes - and the ecosystem around it - isn't going anywhere. But one trend worth looking at is Kubernetes' maturity and mainstreamness for managing stateful applications like Cassandra and Kafka. I think by raising the base level sophistication of stateful cluster operations, more innovation is possible on top of Kubernetes. You're seeing more of this with Kubernetes frameworks, Kubernetes operators (we built one that functions as a Cassandra-as-a-Service on Kubernetes), Kubernetes Custom Resource Definitions (CRDs) and Controllers.

##

About the Author

Paul Brebner is the Chief Technology Evangelist at Instaclustr, which provides a managed service platform of open source technologies such as Apache Cassandra, Apace Spark, Elasticsearch, and Apache Kafka. He lives and works in Canberra, Australia.