Migrating RabbitMQ from Self Hosted to AWS MQ

RabbitMQ is a widely used open-source message broker that helps in scaling the application by deploying a message queuing mechanism between applications. It supports multiple messaging protocols, queuing, delivery acknowledgement and flexible routing to queues.

Amazon MQ is a fully managed service that provisions and manages open source message brokers like RabbitMQ and Apache ActiveMQ. Amazon MQ supports RabbitMQ, a popular open source message broker. This enables the migration of any existing RabbitMQ message brokers to AWS without having any code changes.

Before we begin, The expectation is to have basic understanding of RabbitMQ and MQ in general.

For folks who don’t have time.

THINGS TO KNOW:

Producer / Consumer: Application that sends the messages / receives* the messages.

Queue: Buffer that stores messages

Message: Information that is sent from the producer to a consumer through RabbitMQ.

Connection: A TCP connection between your application and the RabbitMQ broker.

Exchange: Receives messages from producers and pushes them to queues depending on rules defined by the exchange type. To receive messages, a queue needs to be bound to at least one exchange.

Binding: A binding is a link between a queue and an exchange.

Routing key: A key that the exchange looks at to decide how to route the message to queues. Think of the routing key like an address for the message.*

AMQP: Advanced Message Queuing Protocol is the protocol used by RabbitMQ for messaging.

Virtual host: Provides a way to segregate applications using the same RabbitMQ instance. Different users can have different permissions to different vhost and queues and exchanges can be created, so they only exist in one vhost.

Typical Setup:

During Migration:

After the migration:

Steps:

Since your application will be live and you need to make sure there’s no message/data loss during the migration process.

RabbitMQ has an awesome inbuilt feature to support this transition. There are few plugins which support this argument.

1. Federation Plugin
2. Shovel Plugin

1. Federation Plugin:
   1. It helps to transmit messages between brokers without requiring clustering. This is useful for a number of reasons.
   2. The Federation plugin makes it easy to move consumers from Cluster X to Cluster Y, “without disrupting message consumption or losing messages”.
      1. They may have different users and virtual hosts
      2. They may run on different version of RabbitMQ

2. Shovel Plugin:
   1. This is the plugin that unidirectionally moves messages from a source to destination. Sometimes, it’s necessary to reliably and continually move messages from a source (typically a queue) in one cluster to a destination (an exchange, topic, etc) in another cluster

APPROACH:

THINGS TO KNOW:

1. Upstream Server: The system in which the original messages are published
2. Downstream server: The system through which messages are transmitted. Federated Exchange/ Queue takes places here.

For this scenario, we can go with Federation Plugin, which is widely used.

• Federated Exchange:
  • With Exchange FEderation, It is possible to distribute messages to other cluster/nodes. It means, messages published on federated exchanges, will be published to both local exchanges plus downstream server as well. It can be used when you want to consume the same message in multiple places.
  • Exchanges replicate their stream of messages from the upstream to one or more downstreams.

Federated Queues :

• The principle of federated queues is that the consumers now connected to "cluster B" will get messages published to "cluster A" as long as there are no consumers in "cluster A" (local consumers take precedence).
• This feature provides a way of balancing the load of a single logical queue across nodes/clusters. It does so by moving messages to other federation peers (nodes or clusters) when the local queue has no consumers. A federated queue lets a local consumer receive messages from an upstream queue.
• Queues move data where the consumers are, always preferring local consumers to remote ones. Federated queues are considered to be equal peers, there is no "leader/follower" relationship between them like with federated exchanges.

AWS MQ - access only "publicly exposed" Nodes:

• I was struggling a lot to make the Federation Exchange working between Upstream (Self Hosted RabbitMQ) and Downstream (AWS MQ). I have tried different options and configurations, but didn't work. Since, our current setup is private and not secure.
• Then, I came across this article [5]  - Amazon MQ broker only has direct access to resources over the public internet [2]. It means that for federation to access the upstream broker, it needs to be either publicly accessible or have a public proxy.

Broker Type Limits:

• We have to chose the instance type accordingly
  • Ref:https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/rabbitmq-defaults.html

"Secure" connection:

RabbitMQ supports providing the SSL certs. But we need access to the machine. Since the AWS MQ is a hosted service, we do not have access to the backend

Federation via AQMP:

amqps://user:password@server-name?cacertfile=/path/to/cacert.pem&certfile=/path/to/cert.pem&keyfile=/path/to/key.pem&verify=verify_peer

As mentioned earlier, say our setup is private and not secure, The options are

• We have to make it public and secure, that's the only option. Since, AMQP is on tcp/udp level and not HTTP. So, It will work with Nginx or softwares like HAProxy, etc.
• We can make some changes in configuration with respect to Secure Communication.Then, We will be able to make it public and secure to create a connection (Federation) between Upstream and Downstream servers. FYI, AMQPS:// (AMQP + SSL) is the secure version of AMQP

Federation via AQMPS:

amqps://user:password@server-name:5671