A cluster is composed of at least two RabbitMQ nodes. These nodes need to be able to communicate with each other.
I strongly advise you to always have an odd number of nodes in your cluster.
Load Balancing with HAProxy
When working with a cluster the goal is to have a highly available service.
So we need to dispatch requests on every running node of the cluster, and avoid sending request to failing nodes.
One way to achieve this is to use HAProxy. It’s a very light and very good tool when dealing with reverse proxy or load balancing.
HAProxy allows TCP connections and redirections out of the box and works well with the AMQP protocol.
With NGINX you will need to install plugins to manage AMQP connections.
The HAProxy service SHOULD NOT be run on a node of the RAbbitMQ cluster. Because if the node fails, the load balancer will fail too.
And you’ll loose the ability to load balance requests on other nodes.
To have high availability you need more than one node, and of course you need load balancing.
Let’s say that 3 nodes is a good start for a simple RAbbitMQ cluster.
And we need one more node for HAProxy.
Exchanges, Queues and messages mirroring and persistence
If we want a high level of resiliency, something important is to mirror everything we can on other nodes of the cluster.
We need to configure a few things:
durable, mirrored and persistent Exchanges
durable, mirrored and persistent Queues
persistent and mirrored messages (read/write on disk)
Now we can have consumers and producers connected with one (or more) nodes of the cluster.
Sometimes a node can be excluded and unreachable by the others (network failure for instance).
But it’s still running and receiving / consuming messages.
After a small period of time the node becomes desynchronized, and it appears a network partition.
Messages in the excluded node are not in the others and vice versa.
Test and benchmark
If you want to experiment scenarios with RabbitMQ cluster, I created a docker based project for that.