How To Cluster Rabbit-MQ

Foreword

This explanation of clustering Rabbit-MQ assumes that you’ve had some experience with Rabbit-MQ.  At least to the point of being able to get Rabbit-MQ up and running and processing messages.  For this explanation I will be using CentOS linux, other linux distributions may or may not require slight modifications to the setup process.  You will need at least 2 machines or virtual instances up and running and install Rabbit-MQ on both.

Overview

Clustering Rabbit-MQ is actually very simple once you understand what’s going on and how it actually works.  There is no need for a load balancer or any other hardware/software component and the idea is simple.  Send all messages to the master queue and let the master distribute the messages down to the slaves.

Create Short Names

First, we need to change the host name and host entries of our machines to something short.  Rabbit-MQ has trouble clustering queues will fully qualified DNS names.  We’ll need a single short word host and route.  For now, let’s use the names “master” for the master head, then “slave1”, “slave2” … “slaveN” respectively for the rest.

Set the master host name to “master”:

Next we need to set the entries in the /etc/hosts file to allow the short names to be aliased to machine or instance IPs.  Open the /etc/hosts file in your favorite editor and add the following lines:

Please note: Your particular /etc/hosts file will look different that the above. You’ll need to substitute your actual ip and domain suffix for each entry.

Make sure each slave you plan to add has an entry in the /etc/hosts file of the master. To verify your settings for each of the entries you provide, try pinging them by their short name.

If you get something like the above, you’re good to go. If not, take a good look at your settings and adjust them until you do.

Once your short names are setup in the master /etc/hosts file, copy the /etc/hosts file to every slave so that all machines have the same hosts file entries, or to be more specific, that each machine has the master and slave routes. If you’re familiar with routing, feel free to just add the missing routes.

Then for each slave update the host name.

Synchronize ERLang Cookie

Next we need to synchronize our ERlang cookie. Rabbit-MQ needs this to be the same on all machines for them to communicate properly. The file we need is located on the master at /var/lib/rabbitmq/.erlang.cookie, we’ll cat this value then update all the cookies on the slave.

Copy the value displayed by the cat.

Please notice that the file itself is storing the value without a carriage return nor a line feed. This value needs to go into the slaves the same way. Do so be executing the following command on each slave. Make sure you use the “-n” flag.

First let’s make sure we stop the rabbitmq-server on the slaves before updating the ERlang cookie. Do this on each slave server.

Next let’s update the cookie and start the service back up.

Once again substitute the “DQRRLCTUGOBCRFNPIABC” value with your actual ERlang cookie value.

Create The Cluster

Now we cluster the queues together. Starting with the master, issue the following commands:

Next we cluster the slaves to the master. For each slave execute the following commands:

These commands actually do the clustering of the slaves to the master. To verify that everything is in working order issue the following command on any master or slave instance:

Notice the lines containing nodes and running_nodes. They should list out all of the mater and slave entries. If they do not, you may have done something wrong. Go back and try executing all the steps again. Otherwise, you’re good to go. Start sending messages to the master and watch as they are distributed to each of the slave nodes.

You can always dynamically add more slave nodes. To do this, updated the /etc/hosts file of all the machines with the new entry. Copy the master ERlang cookie value to the new slave. Execute the commands to cluster the slave to the master and verify.

Troubleshooting

If you accidentally update the cookie value before you’ve stopped the service, you could get strange errors the next time you start the rabbitmq-server. If this happens just issue the following command:

This removes the mnesia storage and allows you to restart the rabbitmq-server without errors.