Module CSC5002—ASR6: Middleware and software architecture for distributed applications

The first concepts of producer, exchange, consumer, queue, routing key, and binding key are depicted in the following figure. The text following the figure presents these concepts.

Producing means sending. A program that sends messages is a producer. A producer sends messages to an exchange. Consuming means receiving from a queue. A consumer is a program that mostly waits to receive messages.

Producers send messages to exchanges. An exchange is a matching and routing engine: It inspects messages (headers, and more especially what is called, in the AMQP vocabulary, a routing key), and decides how to forward these notifications to message queues (the decision being made using subscription filter's data, and more especially what is called, in the AMQP vocabulary, a binding key) that is provided by the consumer. We will see that they are several types of exchanges, i.e. different matching and routing engines.

Although messages flow through RabbitMQ and your applications, they can only be stored inside a queue. A queue is only limited by the host's memory and disk limits. It is essentially a large message buffer. Many producers can send messages that go to one queue, and many consumers can try to receive data from one queue.

Note that the producer, consumer, and broker do not have to reside on the same host. Indeed in most applications they don't. An application can be both a producer and consumer, too.

In all the JAVA code excerpts, clients, either producers or consumers, open a connection to the RabbitMQ server:

Exceptionnaly, i.e. not usually, producers can connect to queues and directly deposit messages into queues, i.e. without the messages going through an exchange. This functionality is demonstrated into Step 1 of the RabbitMQ tutorial:

see the Web page of Step 1, and the corresponding directory RabbitMQ-Tutorial-Step1.

Here follows the picture of the architecture for this step.

At your convenience, and optionally , you can read these Web pages and run the corresponding code examples:

$ cd RabbitMQ-Tutorial-Step1 $ ./run.sh --xterm # or ./run.sh for all in the same terminal ... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running eu.telecomsudparis.rabbitmq.tutorial.TestScenario b2a7598c5ba35fed8962265a5ead6ac9609f5f161cbe766dd08985c7bfdfe8ee Timeout: 60.0 seconds ... Listing queues for vhost / ... [*] Waiting for messages. To exit press CTRL+C [x] Sent 'Hello World!' [x] Received 'Hello World!' Timeout: 60.0 seconds ... Listing queues for vhost / ... name messages hello 0 rabbitmq rabbitmq

We now propose to comment the last two lines, which stop and remove the container, and to execute again ./run.sh. Next, by executing the command "docker exec rabbitmq rabbitmqctl list_queues", let us now observe that the queue hello that has been created still exists in the server even if there is no more producer or consumer. This is why we preventively stop and remove the container in all our scenarios.

The code of the example is in directory src/main/java/eu/telecomsudparis/rabbitmq/tutorial and the code of the test is in directory src/test/java/eu/telecomsudparis/rabbitmq/tutorial. You can also import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example. In addition, observe that the test is an integration test because the name of the classe ends with IT, namely ScenarioIT.

NB: generally speaking, in case of problem "Connection refused" when executing a scenario, firstly, check that you use the docker container that includes the plugin rabbitmq_management, secondly, try a greater value in the sleep 10 command in shell script run.sh: currently, we assume that the brocker in the container starts in less that 10s.

Next, the authors of the RabbitMQ tutorial present a first list of interesting properties of queues in order to answer the following questions: How to use a queue as a backlog of work, i.e. of messages to treat, and to implement round-robin dispatching with several consumers attached to the shared queue? How to keep a sequence of messages in the queue until the consumer sends an acknowledgement message? How to make sure that even if the consumer dies messages are not lost as they can be persisted by the broker? And, how to organize fair dispatching with a prefetch countproducer? All these questions are the subject of Step 2 of the RabbitMQ tutorial:

Move to directory RabbitMQ-Tutorial-Step2. Learn the next set of AMQP concepts (round-robing dispatching, durable message and message acknowledgment, message durability with durable queue and persistent message, fair dispatching with a prefetch countproducer, consumer, and queue) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

$ cd ...../RabbitMQ-Tutorial-Step2 $ ./run.sh --xterm # or ./run.sh for all in the same terminal ... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running eu.telecomsudparis.rabbitmq.tutorial.TestScenario b2a7598c5ba35fed8962265a5ead6ac9609f5f161cbe766dd08985c7bfdfe8ee Timeout: 60.0 seconds ... [*] Waiting for messages. To exit press CTRL+C [*] Waiting for messages. To exit press CTRL+C [x] Sent 'wait one second .' [x] Sent 'wait two seconds..' [x] Sent 'wait three seconds...' [x] Sent 'wait four seconds....' [x] Sent 'wait five seconds.....' [x] Received 'wait one second .' [x] Received 'wait two seconds..' [x] Done [x] Received 'wait three seconds...' [x] Done [x] Received 'wait four seconds....' [x] Done [x] Received 'wait five seconds.....' [x] Done [x] Done rabbitmq rabbitmq

The code of the example is in directory src/main/java/eu/telecomsudparis/rabbitmq/tutorial and the code of the test is in directory src/test/java/eu/telecomsudparis/rabbitmq/tutorial. You can also import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example. In addition, observe that the test is an integration test because the name of the classe ends with IT, namely ScenarioIT.

This step (Step 3) and the next two steps (Steps 4 and 5) of the RabbitMQ tutorial are the most important steps in a first understanding of event-based programming with the AMQP protocol, and more especially with RabbitMQ.

Let us start with a first type of exchange: "fanout".

Move to directory RabbitMQ-Tutorial-Step3. Learn the next set of AMQP concepts (exchange [of type fanout], binding, and temporary queue) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

The code of the example is in directory src/main/java/eu/telecomsudparis/rabbitmq/tutorial and the code of the test is in directory src/test/java/eu/telecomsudparis/rabbitmq/tutorial. You can also import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example. In addition, observe that the test is an integration test because the name of the classe ends with IT, namely ScenarioIT.

Let us continue with a second type of exchange: "direct".

Move to directory RabbitMQ-Tutorial-Step4. Learn the next set of AMQP concepts (binding key, exchange of type direct, routing key) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

The code of the example is in directory src/main/java/eu/telecomsudparis/rabbitmq/tutorial and the code of the test is in directory src/test/java/eu/telecomsudparis/rabbitmq/tutorial. You can also import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example.

Let us continue with the third and final type of exchange: "topic". This matching and routing engine is the one that fully realize the topic-based event-based paradigm.

Move to directory RabbitMQ-Tutorial-Step5. Learn the next set of AMQP concepts (exchange of type topic, a word in a binding key, a star in a binding key, a hash in a binding key) with the corresponding RabbitMQ tutorial page.

Here follows the picture of the architecture for this step.

At the end of your reading of the tutorial page, you can execute the example as follows:

The code of the example is in directory src/main/java/eu/telecomsudparis/rabbitmq/tutorial and the code of the test is in directory src/test/java/eu/telecomsudparis/rabbitmq/tutorial. You can also import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example.

A typical question when event-based programming with the AMQP protocol is the following one: Is it possible to "emulate" the synchronous RPC (Remote Procedure Call) paradigm with event-based asynchronous AMQP?

The answer to this question is "Yes, we can!" and it is the subject of Step 6 of the RabbitMQ tutorial:

Here follows the picture of the architecture for this step.

Move to directory RabbitMQ-Tutorial-Step6. Learn the last set of AMQP concepts (How to realize RPC communication over topic-based DEBS with RabbitMQ) with the corresponding RabbitMQ tutorial page.

$ cd ...../RabbitMQ-Tutorial-Step6 $ ./run.sh --xterm # or ./run.sh for all in the same terminal ... ------------------------------------------------------- T E S T S ------------------------------------------------------- Running eu.telecomsudparis.rabbitmq.tutorial.TestScenario b2a7598c5ba35fed8962265a5ead6ac9609f5f161cbe766dd08985c7bfdfe8ee Timeout: 60.0 seconds ... completed with 3 plugins. [x] Requesting fib(0) [.] Got '0' [x] Requesting fib(1) [.] Got '1' [x] Requesting fib(2) [.] Got '1' [x] Requesting fib(3) [.] Got '2' [x] Requesting fib(4) [.] Got '3' [x] Requesting fib(5) [.] Got '5' [x] Requesting fib(6) [.] Got '8' rabbitmq rabbitmq

The code of the example is in directory src/main/java/eu/telecomsudparis/rabbitmq/tutorial and the code of the test is in directory src/test/java/eu/telecomsudparis/rabbitmq/tutorial. You can also import the Maven project in your favourite JAVA IDE, e.g. Eclipse, for browsing the code of the example.

Contrary to what is proposed into the tutorial page, in our code, we propose three different versions:

1. using the JAVA client library with the standard AMQP calls (this is the version that is presented in the tutorial page),
2. using the RabbitMQ-specific class StringRpcServer,
3. using the RabbitMQ-specific classes of the package com.rabbitmq.tools.jsonrpc.

For the sake of completeness, study the three implementations.