Module CSC5002—ASR6: Middleware and software architecture for distributed applications

The first concepts of producer, exchange, consumer, queue, routing key, and binding key are depicted in the following figure. The text following the figure presents these concepts.

[Extracted and adapted from the RabbitMQ tutorial page.] Producing means sending. A program that sends messages is a producer. A producer sends messages to an exchange. Consuming means receiving from a queue. A consumer is a program that mostly waits to receive messages.

Producers send messages to exchanges. An exchange is a matching and routing engine: It inspects messages (headers, and more especially what is called, in the AMQP vocabulary, a routing key), and decides how to forward these notifications to message queues (the decision being made using subscription filter's data, and more especially what is called, in the AMQP vocabulary, a binding key) that is provided by the consumer. We will see that they are several types of exchanges, i.e. different matching and routing engines.

Although messages flow through RabbitMQ and your applications, they can only be stored inside a queue. A queue is only limited by the host's memory and disk limits. It is essentially a large message buffer. Many producers can send messages that go to one queue, and many consumers can try to receive data from one queue.

Note that the producer, consumer, and broker do not have to reside on the same host. Indeed in most applications they don't. An application can be both a producer and consumer, too.

In all the JAVA code excerpts, clients, either producers or consumers, open a connection to the RabbitMQ broker (Note the use of the try-with-resources JAVA statement):

In exceptional cases—that is, not as a general rule—producers may connect to queues and submit messages directly to them, without those messages passing through an exchange. This functionality is demonstrated into Step 1 of the RabbitMQ tutorial: (Details 2)

see the Web page of Step 1, and the corresponding package mw.learn.amqp.rabbitmq.step1.

Here follows the picture of the architecture for this step.

At your convenience, and optionally , you can read these Web pages and run the corresponding code examples:

$ ./run_step1_podman.sh --xterm # or ./run_step1_podman.sh for all displays on the same terminal ... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running mw.learn.amqp.rabbitmq.step1.TestScenario b2a7598c5ba35fed8962265a5ead6ac9609f5f161cbe766dd08985c7bfdfe8ee Timeout: 60.0 seconds ... Listing queues for vhost / ... [*] Waiting for messages. To exit press CTRL+C [x] Sent 'Hello World!' [x] Received 'Hello World!' Timeout: 60.0 seconds ... Listing queues for vhost / ... name messages hello 0 rabbitmq rabbitmq

We will now comment out the last two lines, which stop and delete the container, and then rerun the ./run_step1_podman.sh command. By running the command “podman exec rabbitmq rabbitmqctl list_queues,” let’s now observe that the hello queue that was created still exists on the broker, even though there are no longer any producers or consumers. This is why we stop and delete the container as a precaution in all our scenarios.

We suggest that you import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example. Let us observe that the test is an integration test because the name of the class ends with IT, namely ScenarioIT.

Note: Generally speaking, if you encounter a “Connection refused” error when running a scenario, first check that you are using the Podman container that includes the rabbitmq_management plugin, then try increasing the value of the sleep 10 command in the ./run_step?_podman.sh shell script: At present, we assume that the broker within the container starts up in less than 10 seconds.

The authors of the RabbitMQ tutorial then present an initial list of useful queue properties to address the following questions: How can a queue be used as a work queue—that is, a collection of messages to be processed—and how can a “round-robin” distribution be implemented with multiple consumers connected to the shared queue? How can a sequence of messages be maintained in the queue until the consumer sends an acknowledgment message? How can we ensure that, even if the consumer fails, messages are not lost since they can be persisted by the broker? And how can we organize fair distribution using a consumer’s prefetch counter? All these questions are the subject of Step 2 of the RabbitMQ tutorial:

Learn the next set of AMQP concepts (round-robing dispatching, durable message and message acknowledgment, message durability with durable queue and persistent message, fair dispatching with a consumer prefetch count, consumer, and queue) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows (The code of the example is in package mw.learn.amqp.rabbitmq.step2):

$ ./run_step2_podman.sh --xterm # or ./run_step2_podman.sh for all displays on the same terminal ... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running mw.learn.amqp.rabbitmq.step2.TestScenario b2a7598c5ba35fed8962265a5ead6ac9609f5f161cbe766dd08985c7bfdfe8ee Timeout: 60.0 seconds ... [*] Waiting for messages. To exit press CTRL+C [*] Waiting for messages. To exit press CTRL+C [x] Sent 'wait one second .' [x] Sent 'wait two seconds..' [x] Sent 'wait three seconds...' [x] Sent 'wait four seconds....' [x] Sent 'wait five seconds.....' [x] Received 'wait one second .' [x] Received 'wait two seconds..' [x] Done [x] Received 'wait three seconds...' [x] Done [x] Received 'wait four seconds....' [x] Done [x] Received 'wait five seconds.....' [x] Done [x] Done rabbitmq rabbitmq

This step (Step 3) and the next two steps (Steps 4 and 5) of the RabbitMQ tutorial are the most important steps in a first understanding of event-based programming with the AMQP protocol, and more especially with RabbitMQ.

Let us start with a first type of exchange: "fanout".

Learn the next set of AMQP concepts (exchange [of type fanout], binding, and temporary queue) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

The code of the example is in package mw.learn.amqp.rabbitmq.step3.

Let us continue with a second type of exchange: “direct”.

Learn the next set of AMQP concepts (binding key, exchange of type direct, routing key) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

The code of the example is in package mw.learn.amqp.rabbitmq.step4.

Let us continue with the third and final type of exchange: "topic". This matching and routing engine is the one that fully realize the topic-based event-based paradigm.

Learn the next set of AMQP concepts (exchange of type topic, a word in a binding key, a star in a binding key, a hash in a binding key) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

The code of the example is in package mw.learn.amqp.rabbitmq.step5.

A question that often arises when programming in an event-driven style with the AMQP protocol is this: is it possible to “emulate” the synchronous RPC (Remote Procedure Call) paradigm using the asynchronous, event-driven AMQP protocol?

The answer to this question is “Yes, we can!” and it is the subject of Step 6 of the RabbitMQ tutorial:

Here follows the picture of the architecture for this step.

Learn the last set of AMQP concepts (How to realize RPC communication over topic-based DEBS with RabbitMQ) with the corresponding RabbitMQ tutorial page.

The code of the example is in package mw.learn.amqp.rabbitmq.step6;.

Differently from what is proposed in the tutorial page, in our code, we propose three different versions:

1. using the JAVA client library with the standard AMQP calls (this is the version that is presented in the tutorial page),
2. using the RabbitMQ-specific class StringRpcServer,
3. using the RabbitMQ-specific classes of the package com.rabbitmq.tools.jsonrpc.

For the sake of completeness, study the three implementations.

$ ./run_step6_1_podman.sh --xterm # or ./run_step6_1_podman.sh for all displays on the same terminal ... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running mw.learn.amqp.rabbitmq.step6.TestScenario b2a7598c5ba35fed8962265a5ead6ac9609f5f161cbe766dd08985c7bfdfe8ee Timeout: 60.0 seconds ... completed with 3 plugins. [x] Requesting fib(0) [.] Got '0' [x] Requesting fib(1) [.] Got '1' [x] Requesting fib(2) [.] Got '1' [x] Requesting fib(3) [.] Got '2' [x] Requesting fib(4) [.] Got '3' [x] Requesting fib(5) [.] Got '5' [x] Requesting fib(6) [.] Got '8' rabbitmq rabbitmq

channel.queueDeclare(QUEUE_NAME, false, false, false, null);
channel.queueDeclare(QUEUE_NAME, false, false, false, null);

int prefetchCount = 1;
channel.basicQos(prefetchCount);

String queueName = channel.queueDeclare().getQueue();