This article will explore the concept of
Message Queues and a practical use case.
Imagine you just launched a pizza outlet and
built a web application for your store. The web application records each transaction
processed and generates a receipt.
When a customer orders pizza, a chain of
events occurs.
- Record the
transaction
- Update
Database
- Generate
receipt
- Present the
receipt and pizza to the customer
You notice that the receipt generation is
taking a long time. However, your pizza store is still new, and you only get
5-10 orders per day. You can process an order before the next customer arrives.
Because of the current low traffic, you only
attend to one customer at a time. Your web application works well, and you have
a small but mighty group of very happy customers. But then something changed.
More customers
As time goes by, your pizza store makes a name
for itself. Now, you are handling 5,000 - 10,000 orders daily. Your business is
booming. But then you notice a problem.
More customers are coming into your store, and
you spend a lot of time recording and generating the receipt. Because of these
two factors, your customers are waiting longer than they need to.
The application has reached its limits and is
overwhelmed. What do you do?
Message Queues to the rescue
You realize customers can leave their name,
the amount paid, and email. That way, the customers get their pizza immediately
and don't have to wait.
Customers get their pizza, and the receipt is
emailed to them later. This is an example of asynchronous processing, which is
fundamentally what Message Queues do - Think technologies like RabbitMQ and
LavinMQ. They buffer the message that needs to be handled, and the processor
picks up the message at its own pace.
Now, let's tie all this into software systems.
Systems usually send data to and receive data
from other systems. The interactive systems could be processes on the same
computer, modules of the same application, services on different computers, or
technology stacks.
When designing these systems, sometimes we
adopt a communication mechanism where the system:
- Sends data
to another service
- Waits for
the service to finish processing
- Gets a
response from the service
- Proceeds to
the next task
This pattern of communication is called
synchronous processing. We saw this in the pizza store example. This way of
handling requests can result in a high response time and more traffic.
Additionally, there is a high coupling between the client and the web
application.
Message Queues allow for two or more systems
to communicate asynchronously. The Message Queue sits between two or more
systems and acts as the buffer for messages. The Producer puts a message on the
Message Queue and the Consumer, picks up the message from the queue and
processes it.
Message Queues: A formal
definition
The key to understanding the concept of
Message Queues lies in understanding queues and messages.
A queue
is a data structure that arranges items in the order they arrive. The first
item in the queue will be the first item out. Often referred to as FIFO (first
in, first out).
A
message is data from the sender application added to the queue. For
example, a message added to the queue could be a request for the receiving
system to perform some task. A message can be plain text.
We can define a Message Queue as the
technology that accepts messages from the sending application(s) called the
Producer, lines the messages up in a buffer, and allows the receiver
application(s) called the Consumer to process the messages in the order they
arrive. Messages are stored until the Consumer is ready. This way of handling
messages decouples the sender application from the receiver application.
Message Queues foster asynchronous
communication between systems. In other words, when the first process puts a
message on the queue it does not wait for an immediate response. It proceeds
with its execution, and the receiving system processes the message at its own
pace.
The ability to decouple systems and have them
communicate asynchronously is one of the key advantages of Message Queues.
Let's pursue this benefit of Message Queues further.
Decoupled systems with Message
Queues
Decoupling means developing software
components that are self-reliant. In other words, software systems or
components that don't rely so much on other systems to function- so much so
that if the systems they depend on fail then they also fail.
To achieve a decoupled system, communication
needs to happen without connecting the systems directly. They can remain
autonomous and unaware of each other. This is exactly what Message Queues help
us achieve.
If one process in a decoupled system fails to
handle messages from the queue, other messages can still be added to the queue
and processed when the system has recovered.
Instead of building a system with a monolithic
architecture model, breaking it into smaller components is recommended. Then,
these smaller components communicate asynchronously using Message Queues.
Even though message queuing has a wide variety
of use cases, building a service that consists of loosely coupled smaller
components, aka microservices (opposite of monolith), is recommended to achieve
asynchronous communication with Message Queues.
This way, each component can evolve
independently, be written in different languages, and/or maintained by separate
development teams.
A Message Queue will keep your application's
processes separate. The client application can put messages in the queue and
then continue processing.
Since the consuming application takes messages
from the queue when they are able to process them, this system is easy to
maintain and scale.
Conclusion
Message Queues accept messages from Producers,
buffer these messages in a line, and allow Consumers to pick them up and
execute them at their own pace. This asynchronous communication pattern
decouples the Producer from the Consumer and, by extension, encourages the
implementation of systems that are easy to scale and fault-tolerant.
To learn more about Kubernetes and the
cloud native ecosystem, join us at KubeCon + CloudNativeCon North America,
in Salt Lake City, Utah, on November 12-15, 2024.
##
ABOUT THE AUTHOR
Nyior
Clement, Developer Advocate at 84codes
Nyior holds a BSc.
in Software Engineering from the American University of Nigeria, Yola - 2021.
Prior to joining 84codes, he worked as a Python Engineer and now, in his
current role as a Developer Advocate, Nyior combines his strengths in
Engineering, Marketing and more recently, Customer Support.