Virtualization Technology News and Information
Using Apache Ignite for Applications Acceleration and Data Caching

By Denis Magda, Vice President, Developer Relations, GridGain Systems

Apache Ignite, a distributed database for processing at in-memory speeds, helps companies solve their application speed and scale challenges. Ignite is typically deployed on a cluster of distributed machines and pools the available RAM, CPUs and storage resources to create a high-performance data store. The solution offers flexible deployment, including on-premises, in a public or private cloud, or in a hybrid environment. Ignite can even be deployed on specialty mainframe processor cores, where the mainframe's allocated in-memory computing layer pools the CPU cores and RAM.

In a previous article, I introduced the three steps that most companies take in their Apache Ignite journey:

  1. Application acceleration and data caching
  2. A high-performance computing environment for compute-intensive logic or multi-step business operations
  3. A distributed database for mixed workloads that grow beyond the available in-memory capacity

In this follow-up article, the first of a new three-part series, I will take a deeper dive into the initial step of that journey, using Apache ignite for application acceleration and data caching.

Reliance on web-based and mobile applications was soaring even before the pandemic, which further accelerated adoption. Some of these applications, especially in the financial services, insurance and online gaming industries, rely on APIs to access and interact with core systems. This activity introduces latency that can slow transaction processing, impacting the customer experience. It can also impact the performance of the core systems themselves. To avoid this, system developers typically employ caching of frequently accessed data to reduce the need to access the core systems directly and to speed transaction processing. However, scaling the cache can be extremely expensive given the design of most existing infrastructure.

Apache Ignite offers a way to dramatically reduce the cost of building and scaling an application cache.

Apache Ignite as a memory cache

A memory cache is typically based on some form of relatively fast DRAM that sits between the fastest memory (RAM) and the much slower disk-based storage. While much cheaper than RAM, it is far more expensive than disk-based storage, so system designers are forced to make hard choices between speed and cost.

With Apache Ignite, the pooled, high performance data store can be used as an extremely low-latency and scalable in-memory cache for storing frequently accessed data, which is made available to a wide array of APIs, as well as to SQL queries. Applications interact with the cluster by using simple key-value requests. The cache can also be queried with SQL and updated atomically using ACID transactions. Custom computations written in languages such as Java, C#, and C++ can also be executed on the cache.

The most basic use case for Ignite as an in-memory cache is as a cache-aside deployment. The cache is deployed separately from the primary data store, and an application or change-data-capture (CDC) process becomes responsible for synchronizing the data between the two storage locations. This strategy is effective when the cached data is not updated frequently or when a temporary data lag between the primary store and the cache is not an issue.

Ignite as an in-memory data grid

Another basic strategy for Ignite is as a read-through/write-through cache (aka. an in-memory data grid), where the application layer treats Ignite as the primary data store. As applications write to and read from the cache, Ignite ensures the underlying database remains updated and is consistent with the in-memory data. This strategy is best for accelerating a disk-based database. Ignite integrates out-of-the-box with many databases and in write-through or write-behind mode can synchronize all changes to the database.

For ACID transactions, Ignite can also coordinate and commit a transaction across its in-memory cluster as well as to a relational database. The read-through capability ensures that if a record is missing from memory, the cache can read the data from an external database.

Finally, the data grid supports various APIs, such as SQL, compute, and key-value. Applications write to and read from the data grid, and the data grid propagates changes to all the underlying data stores. Applications can utilize multiple APIs for data processing, and key-value and SQL queries support request, join, and group distributed datasets. Further, compute and machine-learning APIs execute logic close to the data, eliminating expensive data shuffling over the network.

Ignite as a digital hub

Finally, using ignite as an in-memory data grid enables businesses to deploy what Gartner calls a digital integration hub (DIH). A DIH is a shared and unified in-memory layer that stores datasets originating from multiple on-premises and cloud-based sources and streaming data feeds. The DIH architecture enables multiple customer-facing applications to access a single view of the aggregated data. As changes to the data are made by the consuming applications, the DIH writes the changes back to the underlying databases. A DIH also enables businesses to perform real-time analytics across operational and historical data, a capability that is essential for fraud detection and recommendation engines, for example. With a DIH, businesses can also reduce API calls to the source applications, overcoming the limitations that many vendors impose and often saving significant money.

Using Apache ignite for application acceleration and data caching can enable businesses to move forward with their digital transformation strategies faster and at lower cost. However, this use of Ignite is just the first step in a longer Ignite journey that many companies are taking. The next article in this series will discuss using Ignite for high performance computing.




Denis Magda is a Vice President, Developer Relations, at GridGain Systems, provider of enterprise-grade in-memory computing solutions powered by the Apache Ignite distributed database. An Apache Software Foundation Member, Apache Ignite committer, Project Management Committee member, and open-source software enthusiast, Denis helps software engineers and architects develop their expertise in in-memory computing. A well-known industry speaker, Denis can often be found at conferences, workshops, and other events sharing his knowledge about Apache Ignite, distributed systems, and open-source communities.

Published Wednesday, September 08, 2021 7:30 AM by David Marshall
Filed under:
The Apache Ignite Journey: High Performance Computing : @VMblog - (Author's Link) - March 3, 2022 7:31 AM
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<September 2021>