RabbitMQ

• Learning
• Benchmark
• Development with RabbitMQ
• Bash
• Python
• PHP
• dotNet
• AMQP Client
• RabbitMQ Standalone
• Install on RedHat
• Management UI
• Plugins
• Monitoring & Management
• RabbitMQ Cluster
• FAQ

Learning

Introduction

• Message Queue 簡介(以 RabbitMQ 為例) | 小信豬的原始部落 (godleon.github.io)
• RabbitMQ 基本介紹、安裝教學

其他類似的開源方案

• Kafka
  • Video: Taichung.py 2024-01 Meetup01, 聽說隔壁部門開始用kafka ??? - YouTube

Installation

• Installing on RPM-based Linux (RedHat Enterprise Linux, CentOS, Fedora, openSUSE)
• Production Checklist
• RabbitMQ Checklist For Production Environments - A Complete Guide

Cluster & Load Balancing

• RabbitMQ 中的分散式，普通 cluster 模式的構建
• [VMWARE] VMware Tanzu RabbitMQ for Kubernetes Overview
• [HUAWEI] Distributed Message Service for RabbitMQ

Development

• RabbitMQ 筆記
• Amazon MQ for RabbitMQ best practices
• Red Hat AMQ Clients

Commercial RabbitMQ

• IBM Messages for RabbitMQ
• VMware Tanzu RabbitMQ

IBM MQ vs. RabbitMQ vs. Kafka

Benchmark

• Java Tools
  • Load testing and performance measurements in RabbitMQ - CloudAMQP
• amqpc

Java Tools

Install: RabbitMQ PerfTest

Usage

• --time: Seconds, 程式運行時間
• --uri: MQ Server 連線字串
• --producers: Producer 數量
• --consumers: Consumer 數量
• --size: bytes, Message 的 size
• --rate: msg/s, Producer/Consumer 的 Msg Rate
• --consumer-rate: msg/s, Consumer 的 Msg Rate

#> alternatives --list
libnssckbi.so.x86_64    auto    /usr/lib64/pkcs11/p11-kit-trust.so
ld      auto    /usr/bin/ld.bfd
mta     auto    /usr/sbin/sendmail.postfix
ksh     auto    /bin/ksh93
java    manual  /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64/bin/java
jre_openjdk     auto    /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
jre_11  auto    /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
jre_11_openjdk  auto    /usr/lib/jvm/jre-11-openjdk-11.0.8.10-1.el7.x86_64

#> export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
#> cd rabbitmq-perf-test-2.18.0
#> bin/runjava com.rabbitmq.perf.PerfTest --help

Scenario #1

• Condition:
  • Queue: 200
  • Producer: 200
  • Consumer: 200
  • Message Size: 20KB
  • Message Rate: 20/s
  • Ack Mode: Manual
    
• Throughput:
  • Publish: 4000/s
  • Consumer: 4000/s
    
• Resource:
  • CPU (2 Core): 110 %
  • Memory (4 GB): 8.8 %

bin/runjava com.rabbitmq.perf.PerfTest --uri amqp://<username>:<password>@<rabbitmq-server-ip>:<port>/<vhost> \
--queue-pattern 'perf-test-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 \
--rate 20

Scenario #2 for the Cluster with 3 nodes

Queue Type: Classic

bin/runjava com.rabbitmq.perf.PerfTest \
--uris "amqp://<username>:<password>@<rabbitmq-server1-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server2-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server3-ip>:<port>/<vhost>" \
--queue-pattern 'perf-test-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 --rate 20

Queue Type: Quorum

bin/runjava com.rabbitmq.perf.PerfTest \
--uris "amqp://<username>:<password>@<rabbitmq-server1-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server2-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server3-ip>:<port>/<vhost>" \
--queue-args x-queue-type=quorum \
--auto-delete false \
--flag persistent \
--queue-pattern 'perf-test-quorum-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 \
--rate 6

效能優化與影響

• 效能影響因素: How to Run Benchmarks | RabbitMQ - Blog
• 13 Common RabbitMQ Mistakes and How to Avoid Them - CloudAMQP
• Cluster Sizing and Other Considerations | RabbitMQ - Blog

Development with RabbitMQ

Bash

curl

curl -u login:pass -i -H "content-type:application/json" -X POST http://localhost:15672/api/exchanges/%2Fvhost/exchange/publish \
 -d'{"properties":{},"routing_key":"","payload":"you message","payload_encoding":"string"}'

rabbitmqadmin

rabbitmqadmin uses HTTP API authentication mechanism (basic HTTP authentication).

• Command Line Tools
• Management Command Line Tool

Install

# Install the dependency
yum install python3

# Download the script from the RabbitMQ Server
wget http://<rabbitmq-server-hostname>:15672/cli/rabbitmqadmin

#
chmod 0755 rabbitmqadmin
mv rabbitmqadmin /usr/local/bin
rabbitmqadmin -h

# Verify the connection via HTTP API
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> list vhosts

Basic Operation

## 在其他 Linux 主機上測試
# 宣告 queue
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> declare queue name=my-testq durable=true

# 發佈訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> publish exchange=amq.default routing_key=my-testq payload="This is Alang"

# 接收訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> get queue=my-testq ackmode=ack_requeue_false
# 接收訊息, 格式為 tsv
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> -f tsv get queue=my-testq ackmode=ack_requeue_false
# 一次接收 5 訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> get queue=my-testq count=5 ackmode=ack_requeue_false

while read -r line; do 
  echo $line | rabbitmqadmin publish exchange=amq.default routing_key=my_queue ; 
done < messages

rabbitmqadmin publish exchange=amq.default routing_key=test payload="hello, world"

# With parallel
cat messages | parallel -j 100 \
  ./rabbitmqadmin -H $RABBITMQ_HOST \
                  -u $RABBITMQ_USERNAME \
                  -p $RABBITMQ_PASSWORD  \
                  publish exchange=amq.default \
                  routing_key=myqueue \
                  payload="{}"

Python

Tutorials

• python rabbitmq Code Example (codegrepper.com)
• Part 2.3: Getting started with RabbitMQ and Python - CloudAMQP
• 利用Python操作訊息佇列RabbitMQ的方法教程 | 程式前沿 (codertw.com)
• Python 实现 RabbitMQ 的六种工作模式（附 Python 代码）_Python_AlwaysBeta_InfoQ写作社区
• RedHat AMQ Python Client

RedHat AMQ Python Client

Install via RHN

[root@dotnetdev ~]# dnf repolist all | grep amq-client
amq-clients-2-for-rhel-8-x86_64-debug-rpms                                  disabled
amq-clients-2-for-rhel-8-x86_64-rpms                                        disabled
amq-clients-2-for-rhel-8-x86_64-source-rpms                                 disabled
amq-clients-2.9-for-rhel-8-x86_64-debug-rpms                                disabled
amq-clients-2.9-for-rhel-8-x86_64-rpms                                      disabled
amq-clients-2.9-for-rhel-8-x86_64-source-rpms                               disabled

[root@dotnetdev ~]# subscription-manager repos --enable=amq-clients-2-for-rhel-8-x86_64-rpms
Repository 'amq-clients-2-for-rhel-8-x86_64-rpms' is enabled for this system.

[root@dotnetdev ~]# dnf repolist
Updating Subscription Management repositories.
repo id                                              repo name
amq-clients-2-for-rhel-8-x86_64-rpms                 Red Hat AMQ Clients 2 for RHEL 8 x86_64 (RPMs)
rhel-8-for-x86_64-appstream-rpms                     Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms                        Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)

[root@dotnetdev ~]# yum install python3-qpid-proton python-qpid-proton-docs

PHP

Tutorials

• how to call a model from rabbitmq php consumer's callback in codeigniter? (google.com)

dotNet

Tutorials

• [GitHub] AMQP.Net Lite
• RedHat AMQ .Net Client
• C# 連線至 RabbitMQ Cluster - 使用 RabbitMQ .Net Client 及 EasyNetQ
• [GitHub] EasyNetQ

Case: Console RabbitMQ Client

ConsoleRabbitMQ.zip

Package Required:

• RabbitMQ.Client

Target Framework:

• net6.0

Platform Supported:

• Windows 64
• RedHat Linux 8 64 (net6 runtime is required)

AMQP Client

Terms & Concepts

Queue

• prefetch: Consumer 的參數。當 consumer 接收訊息是採 PUSH 模式，且發送端會有大量訊息寫入，由於 consumer 的 PUSH 模式，會一次接收大量(一次幾百或幾千)的訊息，假使 consumer 未能及時處理所有訊息，那些待處理的訊息會處於 Unacked 狀態，一旦 Unacked 的訊息量過多，主機就會終止連線。
  要解決這個問題，consumer 必須使用 prefetch 限制主機每次推送至 consumer 的訊息量。 
• Lazy Queue: Queue 的參數。對於較長(大)的訊息，可以使用這類 queue，記憶體的耗用可以改善，但會增加磁碟空間的使用率。

Consume Message

• Consumer 接收訊息的開發方式有分 Push (推送) 與 Poll (拉) 兩種方式。
  • 快速掌握RabbitMQ(四)——两种消费模式和QOS的C#实现 - 捞月亮的猴子 - 博客园 (cnblogs.com)

Publish Message

• 程式端發佈訊息，不像接收端那樣有多種客製參數與不同接收模式，唯一要注意的是主機效能與負載。多 nodes 的 Cluster 模式下，應該要將不同用戶端平均連接到不同的 nodes，達到負載均衡的效益。
• Publish Confirm:

TTL (Time to Live)

如果要讓訊息在超過所設置的時間，沒有被接收時即自動清除。這個設置稱為 TTL，在 RabbitMQ 裡 TTL 可以用在 Message 或 Queue 本身。

Message TTL

• 用途: 超過設置時間的訊息，即自動清除。
• 如果 Message 發送到多個 Queues 時，依照各自 Queue 的 TTL 的設置，訊息的清除時間可能會有不同。沒有設置 TTL 的 Queue，所屬的訊息則不會影響 。
• 參數值: Millisecond (60 seconds = 60000)

方法一: Policy

rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues

方法二: Queue 參數

• x-message-ttl: 60000

Sample codes in C#

var args = new Dictionary<string, object>();
args.Add("x-message-ttl", 60000);
model.QueueDeclare("myqueue", false, false, false, args);

Queue TTL

• 用途: 超過設置時間的 Queue，該 Queue 會自動清除。
• 參數值: Millisecond (30 mins = 1800000)

方法一: Policy

rabbitmqctl set_policy expiry ".*" '{"expires":1800000}' --apply-to queues

方法二: Queue 參數

• x-expires: 1800000

Sample codes in Java

Map<String, Object> args = new HashMap<String, Object>();
args.put("x-expires", 1800000);
channel.queueDeclare("myqueue", false, false, false, args);

rabbitmqadmin

• Management Command Line Tool — RabbitMQ

Usage

# Publish a message
rabbitmqadmin -H <rabbitmq-server-ip> -u <user-name> -p <secret> -V <virtual-server> publish exchange=amq.default routing_key=my-testq payload="This is Alang"

# Consume/Get a message
rabbitmqadmin -H <rabbitmq-server-ip> -u <user-name> -p <secret> -V <virtual-host> get queue=my-testq ackmode=ack_requeue_false

amqp-tools

A CLI tool is built-in Ubuntu.

Install

sudo apt update
sudo apt install amqp-tools

Usage

# Declare a queue
amqp-declare-queue --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" -d -q "my-testq"

# Publish a message
amqp-publish --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --routing-key="my-testq" -b "Hello,World"

# Get the messages (Poll mode)
amqp-get --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --queue="my-testq"

# Get the messages (Push mode)
amqp-consume --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --queue="my-testq" -p 2 ./show.sh

show.sh:

#!/usr/bin/env bash
read line
echo "Message: $line"
sleep 1

RabbitMQ Standalone

Opened Port if firewall is used

• Configuration

Ports Required:

• 4369: epmd, a peer discovery service used by RabbitMQ nodes and CLI tools
• 5672, 5671: used by AMQP 0-9-1 and 1.0 clients without and with TLS
• 25672: used for inter-node and CLI tools communication (Erlang distribution server port) and is allocated from a dynamic range (limited to a single port by default, computed as AMQP port + 20000). Unless external connections on these ports are really necessary (e.g. the cluster uses federation or CLI tools are used on machines outside the subnet), these ports should not be publicly exposed. See networking guide for details.
• 35672-35682: used by CLI tools (Erlang distribution client ports) for communication with nodes and is allocated from a dynamic range (computed as server distribution port + 10000 through server distribution port + 10010). See networking guide for details.
• 15672: HTTP API clients, management UI and rabbitmqadmin (only if the management plugin is enabled)
• 61613, 61614: STOMP clients without and with TLS (only if the STOMP plugin is enabled)
• 1883, 8883: MQTT clients without and with TLS, if the MQTT plugin is enabled
• 15674: STOMP-over-WebSockets clients (only if the Web STOMP plugin is enabled)
• 15675: MQTT-over-WebSockets clients (only if the Web MQTT plugin is enabled)
• 15692: Prometheus metrics (only if the Prometheus plugin is enabled)

Listener Port

Change the default port 5672

Edit: /etc/rabbitmq/rabbitmq.conf

## Networking
## ====================
##
## Related doc guide: https://rabbitmq.com/networking.html.
##
## By default, RabbitMQ will listen on all interfaces, using
## the standard (reserved) AMQP 0-9-1 and 1.0 port.
##
# listeners.tcp.default = 5672
listeners.tcp.default = 15690

Restart the RabbitMQ Service

# Using systemctl
systemctl stop rabbitmq-server
systemctl start rabbitmq-server

# Alternatively, using rabbitmqctl
rabbitmqctl stop_app
rabbitmqctl start_app

Users and Permissions

Default User Access

The broker creates a user guest with password guest. Unconfigured clients will in general use these credentials. By default, these credentials can only be used when connecting to the broker as localhost so you will need to take action before connecting from any other machine.

See the documentation on access control for information on how to create more users and delete the guest user.

Adding/Listing/Deleting Users

## Adding Users
# will prompt for password, only use this option interactively
rabbitmqctl add_user "username"

# Password is provided via standard input.
# Note that certain characters such as $, &, &, #, and so on must be escaped to avoid
# special interpretation by the shell.
echo '2a55f70a841f18b97c3a7db939b7adc9e34a0f1b' | rabbitmqctl add_user 'username'

 Password is provided as a command line argument.
# Note that certain characters such as $, &, &, #, and so on must be escaped to avoid
# special interpretation by the shell.
rabbitmqctl add_user 'username' '2a55f70a841f18b97c3a7db939b7adc9e34a0f1b'

## Listing User
rabbitmqctl list_users
rabbitmqctl list_users --formatter=json

## Deleting a user
rabbitmqctl delete_user 'username'

## Verifying a user
rabbitmqctl authenticate_user "a-username" "a-password"

Granting Permissions to a User

# First ".*" for configure permission on every entity
# Second ".*" for write permission on every entity
# Third ".*" for read permission on every entity
rabbitmqctl set_permissions -p "custom-vhost" "username" ".*" ".*" ".*"

# tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags username administrator

## Verifying the permission
# => Listing permissions for vhost "/" ...
# => user    configure   write   read
# => user2   .*  .*  .*
# => guest   .*  .*  .*
# => temp-user   .*  .*  .*
rabbitmqctl list_permissions --vhost /
rabbitmqctl list_permissions --vhost gw1

Clearing Permissions of a User in a Virtual Host

# Revokes permissions in a virtual host
rabbitmqctl clear_permissions -p "custom-vhost" "username"

Operations on Multiple Virtual Hosts

# Assumes a Linux shell.
# Grants a user permissions to all virtual hosts.
for v in $(rabbitmqctl list_vhosts --silent); do rabbitmqctl set_permissions -p $v "a-user" ".*" ".*" ".*"; done

Virtual Hosts

Creating a Virtual Host

## Using CLI Tools
rabbitmqctl add_vhost qa1

## Using HTTP API
curl -u userename:pa$sw0rD -X PUT http://rabbitmq.local:15672/api/vhosts/vh1

Deleting a Virtual Host

## Using CLI Tools
rabbitmqctl delete_vhost qa1

## Using HTTP API
curl -u userename:pa$sw0rD -X DELETE http://rabbitmq.local:15672/api/vhosts/vh1

Kernel Limits

RabbitMQ nodes are most commonly affected by the maximum open file handle limit. Default limit value on most Linux distributions is usually 1024, which is very low for a messaging broker (or generally, any data service). See Production Checklist for recommended values.

Open File Limit

With systemd (Modern Linux Distributions)

新增: /etc/systemd/system/rabbitmq-server.service.d/limits.conf

[Service]
LimitNOFILE=64000

Restart the service

systemctl daemon-reload
systemctl stop rabbitmq-server
systemctl start rabbitmq-server

Verify the change

#> ps -ef | grep rabbitmq

rabbitmq  460668  460654  0 11:43 ?        00:00:00 erl_child_setup 64000 <====

#> rabbitmqctl status

File Descriptors

Total: 236, limit: 63903
Sockets: 230, limit: 57510

TLS Connection

• Troubleshooting TLS-enabled Connections — RabbitMQ
• TLS Support — RabbitMQ

Data Directory

• File and Directory Locations — RabbitMQ

Environment Variables

Name	Description
RABBITMQ_BASE	Note: Windows only. This base directory contains sub-directories for the RabbitMQ server's database and log files. Alternatively, set RABBITMQ_MNESIA_BASE and RABBITMQ_LOG_BASE individually.
RABBITMQ_MNESIA_BASE	This base directory contains sub-directories for the RabbitMQ server's node database, message store and cluster state files, one for each node, unless RABBITMQ_MNESIA_DIR is set explicitly. It is important that effective RabbitMQ user has sufficient permissions to read, write and create files and subdirectories in this directory at any time. This variable is typically not overridden. Usually RABBITMQ_MNESIA_DIR is overridden instead.
RABBITMQ_MNESIA_DIR	The directory where this RabbitMQ node's data is stored. This s a schema database, message stores, cluster member information and other persistent node state.

Default Locations for Linux

Name	Location
RABBITMQ_MNESIA_BASE	${install_prefix}/var/lib/rabbitmq/mnesia
RABBITMQ_MNESIA_DIR	$RABBITMQ_MNESIA_BASE/$RABBITMQ_NODENAME

Default Locations for Windows

Name	Location
RABBITMQ_BASE	%APPDATA%\RabbitMQ
RABBITMQ_MNESIA_BASE	%RABBITMQ_BASE%\db
RABBITMQ_MNESIA_DIR	%RABBITMQ_MNESIA_BASE%\%RABBITMQ_NODENAME%-mnesia

Erlang Cookie

• How CLI Tools Authenticate to Nodes (and Nodes to Each Other): the Erlang Cookie

RabbitMQ nodes 與 CLI tools 使用 cookie 做通訊時的認證，cookie 檔的路徑是

•  /var/lib/rabbitmq/.erlang.cookie (used by the server)。
• $HOME/.erlang.cookie (used by the CLI tools)。
  

或者執行

rabbitmq-diagnostics erlang_cookie_sources

重點整理

• cookie 檔只能有 rabbitmq 用戶存取的權限。
• 每一個 node 必須有相同的 cookie 檔。
• cookie 檔的產生應該會在 Cluster 部署階段時發生。
• 如果 cookie 檔不存在，在 RabbitMQ 啟動時，Erlang VM 會自動隨機產生一個。

Q & A

How to Find Config File Location

1. Check the Log File

node           : rabbit@example
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/advanced.config
               : /etc/rabbitmq/rabbitmq.conf

2. The comand rabbitmq-diagnostics or rabbitmqctl

rabbitmq-diagnostics status
rabbitmq-diagnostics status -n [node name]
rabbitmqctl status

不同平台的路徑位置

Platform	Default Configuration File Directory	Example Configuration File Paths
Generic binary package	$RABBITMQ_HOME/etc/rabbitmq/	$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.conf, $RABBITMQ_HOME/etc/rabbitmq/advanced.config
Debian and Ubuntu	/etc/rabbitmq/	/etc/rabbitmq/rabbitmq.conf, /etc/rabbitmq/advanced.config
RPM-based Linux	/etc/rabbitmq/	/etc/rabbitmq/rabbitmq.conf, /etc/rabbitmq/advanced.config
Windows	%APPDATA%\RabbitMQ\	%APPDATA%\RabbitMQ\rabbitmq.conf, %APPDATA%\RabbitMQ\advanced.config
MacOS Homebrew Formula	${install_prefix}/etc/rabbitmq/, and the Homebrew cellar prefix is usually /usr/local	${install_prefix}/etc/rabbitmq/rabbitmq.conf, ${install_prefix}/etc/rabbitmq/advanced.config

Install on RedHat

Reference

• Installing on RPM-based Linux (RedHat Enterprise Linux, CentOS, Fedora, openSUSE) — RabbitMQ

Using PackageCloud Yum Repository

Quick-Install

Prerequisites: Internet connection

## Uses a PackageCloud-provided Yum repository setup script.
## Always verify what is downloaded before piping it to a privileged shell!
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash

Manual-Install

Import the signing keys required

## primary RabbitMQ signing key
rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
## modern Erlang repository
rpm --import https://packagecloud.io/rabbitmq/erlang/gpgkey
## RabbitMQ server repository
rpm --import https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey

Add Yum Repositories for RabbitMQ and Modern Erlang

# In /etc/yum.repos.d/rabbitmq.repo

##
## Zero dependency Erlang
##

[rabbitmq_erlang]
name=rabbitmq_erlang
baseurl=https://packagecloud.io/rabbitmq/erlang/el/8/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/erlang/gpgkey
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[rabbitmq_erlang-source]
name=rabbitmq_erlang-source
baseurl=https://packagecloud.io/rabbitmq/erlang/el/8/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/erlang/gpgkey
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

##
## RabbitMQ server
##

[rabbitmq_server]
name=rabbitmq_server
baseurl=https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[rabbitmq_server-source]
name=rabbitmq_server-source
baseurl=https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

Install Packages with Yum

# Update Yum package metadata
yum update
yum -q makecache --disablerepo='*' --enablerepo='rabbitmq_erlang' --enablerepo='rabbitmq_server'

## install these dependencies from standard OS repositories
yum install socat logrotate

## install RabbitMQ and zero dependency Erlang from the above repositories,
## ignoring any versions provided by the standard repositories
yum install --repo rabbitmq_erlang --repo rabbitmq_server erlang rabbitmq-server

Run RabbitMQ Server

systemctl status rabbitmq-server
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
systemctl enable rabbitmq-server

Check the status

Run: rabbitmqctl status or rabbitmq-diagnostics status

#> rabbitmq-diagnostics status

Status of node rabbit@tpeeaprmq98 ...
Runtime

OS PID: 549246
OS: Linux
Uptime (seconds): 504837
Is under maintenance?: false
RabbitMQ version: 3.10.7
Node name: rabbit@tpeeaprmq98
Erlang configuration: Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 1.1.1k  FIPS 25 Mar 2021
Erlang processes: 412 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_mqtt
 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98/quorum/rabbit@tpeeaprmq98

Config files


Log file(s)

 * /var/log/rabbitmq/rabbit@tpeeaprmq98.log
 * /var/log/rabbitmq/rabbit@tpeeaprmq98_upgrade.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1408 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 1.526 gb

reserved_unallocated: 0.0882 gb (62.67 %)
code: 0.0351 gb (24.97 %)
other_proc: 0.0201 gb (14.28 %)
other_system: 0.0139 gb (9.85 %)
other_ets: 0.0032 gb (2.26 %)
plugins: 0.0018 gb (1.29 %)
atom: 0.0014 gb (1.01 %)
binary: 0.0009 gb (0.62 %)
metrics: 0.0006 gb (0.4 %)
mgmt_db: 0.0003 gb (0.22 %)
mnesia: 0.0001 gb (0.07 %)
msg_index: 0.0001 gb (0.07 %)
queue_procs: 0.0 gb (0.03 %)
quorum_ets: 0.0 gb (0.02 %)
connection_other: 0.0 gb (0.01 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 6, limit: 63903
Sockets: 0, limit: 57510

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 5.4187 gb

Totals

Connection count: 0
Queue count: 2
Virtual host count: 3

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT

FAQ

GPG Key Error

Errors during downloading metadata for repository 'rabbitmq_server':
  - Curl error (35): SSL connect error for https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 ]

Solution: 如果外網有防火牆，必須開通 github.com，或者直接下載該 GPG key 檔，然後上傳到主機的目錄 /etc/pki/rpm-gpg，並修改 /etc/yum.repos.d/rabbitmq.repo

# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
       file:///etc/pki/rpm-gpg/rabbitmq-release-signing-key.asc
#       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc

Management UI

The RabbitMQ management plugin provides an HTTP-based API for management and monitoring of RabbitMQ nodes and clusters, along with a browser-based UI and a command line tool, rabbitmqadmin.

• https://www.rabbitmq.com/management.html
  

Getting Started

rabbitmq-plugins enable rabbitmq_management

Node restart is not required after plugin activation.

Auto Enable the plugin: https://www.rabbitmq.com/plugins.html#enabled-plugins-file

Management UI Access

The management UI can be accessed using a Web browser at http://{node-hostname}:15672/.

Access and Permissions

# create a user
rabbitmqctl add_user sysadmin ThisIsPassword
# tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags sysadmin administrator

Plugins

https://www.rabbitmq.com/plugins.html

A list of plugins available locally

rabbitmq-plugins list

Enable/Disable the plugin

rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins disable rabbitmq_management

Different Ways to Enable Plugins

NOTE: 在版本 3.10.7 不再需要手動編輯檔案，當使用指令 enable/disable 時，系統會自動編輯這個檔案。

[root@tpeeaprmq98 ~]# rabbitmq-plugins directories -s
Plugin archives directory: /usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.10.7/plugins
Plugin expansion directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-plugins-expand
Enabled plugins file: /etc/rabbitmq/enabled_plugins

Edit the file: /etc/rabbitmq/enabled_plugins

[rabbitmq_management,rabbitmq_amqp1_0,rabbitmq_mqtt].

重啟服務

systemctl stop rabbitmq-server
systemctl start rabbitmq-server

Monitoring & Management

Web UI

Overview

• Ready: 已經發佈到 Queue 的訊息量
• Unacked: 消費端已接收但尚未 Ack 的訊息量
• Total: Ready + Unacked

CLI

rabbitmq-diagnostics

Online Resource Utilization

rabbitmq-diagnostics observer

RabbitMQ Version

[root@tpeeaprmq98 ~]# rabbitmq-diagnostics server_version
Asking node rabbit@tpeeaprmq98 for its RabbitMQ version...
3.10.7

Check the listener ports

rabbitmq-diagnostics -s listeners

rabbitmqctl

List the queues

rabbitmqctl -p <vhost-name> list_queues name state durable arguments policy

rabbitmqctl -qs -p <vhost-name> list_queues name > queue_names.lst

User Management

• 其他 tag 資訊: Management Plugin — RabbitMQ

# List all users
rabbitmqctl list_users

# Create a new user
rabbitmqctl add_user "eapuser"
rabbitmqctl add_vhost "eap_server"
rabbitmqctl set_permissions -p "eap_server" "eapuser" ".*" ".*" ".*"
rabbitmqctl set_user_tags eapuser monitoring  # for web login only

# Remove tag 'monitoring' from user if needed
rabbitmqctl set_user_tags eapuser ""

# Tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags username administrator

## Verifying the permission
# => Listing permissions for vhost "/" ...
# => user    configure   write   read
# => user2   .*  .*  .*
# => guest   .*  .*  .*
# => temp-user   .*  .*  .*
rabbitmqctl list_permissions --vhost /
rabbitmqctl list_permissions --vhost gw1

# Revoke user access
rabbitmqctl delete_user 'username'

Connections

rabbitmqctl list_connections
rabbitmqctl list_connections user,peer_host,peer_port,channels,state

Recreate the virtual host

rabbitmqctl delete_vhost <my-vhost-name>
rabbltmqctl add_vhost <my-vhost-name>

Reset the RabbitMQ Node

The broker drops all virtual hosts, queues, exchanges, and non-administrative users.

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

Force Reset the RabbitMQ Node

如果 Cluster 損壞且無法恢復運作時，可以嘗試強制重設 Node。

rabbitmqctl stop_app
rabbitmqctl force_reset
rabbitmqctl start_app

Suspend all listeners and prevent new client connections

暫停 listener 後，可以使用指令 ss -ltpn 檢查所有的 listener port 是否關閉。

既有的連線不會受到影響，不過新的連線將無法建立。

# For current node
rabbitmqctl suspend_listeners

# suspends listeners on node rabbit@node2.cluster.rabbitmq.svc: it won't accept any new client connections
rabbitmqctl suspend_listeners -n rabbit@node2.cluster.rabbitmq.svc

# For current node, to resume all listeners on a node and make it accept new client connections again
rabbitmqctl resume_listeners
# resumes listeners on node rabbit@node2.cluster.rabbitmq.svc: it will accept new client connections again
rabbitmqctl resume_listeners -n rabbit@node2.cluster.rabbitmq.svc

rabbitmqadmin

Basic Operation

# List queus
rabbitmqadmin list queues
rabbitmqadmin -H <RabbitMQ-Server-IP> -u <username> -p <password> -V <vhost-name> list queues

# Add a queue with optional parameters
rabbitmqadmin declare queue name=<my-new-queu> durable=true auto_delete=true

Remove multiple queues

rabbitmqadmin -f tsv -q list queues name > q.txt
while read -r name; do rabbitmqadmin -q delete queue name="${name}"; done < q.txt

Connections

rabbitmqadmin -H <RabbitMQ-Server-IP> -u <username> -p <password> -V <vhost-name> list connections name

# Close multiple connections without any channels
rabbitmqadmin -f tsv -q connections name channels | awk -F "\t" '($2 < 1) {print $1}' | tee conn_noChannels.lst
while read -r conn;do rabbitmqadmin close connection name="${conn}"; done < conn_noChannels.lst

Monitoring

Health-Check

rabbitmq-diagnostics check_running
rabbitmq-diagnostics ping

Queue State

rabbitmqctl list_queues name state

Cluster Status

rabbitmqctl cluster_status

Monitor with Prometheus

• Monitoring with Prometheus & Grafana — RabbitMQ
• First steps | Prometheus

RabbitMQ Configuration

# Enable the plugin rabbitmq_prometheus
rabbitmq-plugins enable rabbitmq_prometheus

# To confirm that RabbitMQ now exposes metrics in Prometheus format
curl -s localhost:15692/metrics | head -n 10

Prometheus Configuration

/etc/hosts:

<node1.ip.addr>	 rmq01
<node2.ip.addr>	 rmq02
<node3.ip.addr>	 rmq03

prometheus.yml:

scrape_configs:
  - job_name: rabbitmq

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 15s
    
    static_configs:
      - targets: ['rmq01:15692', 'rmq02:15692', 'rmq03:15692']

Grafana Configuration

• RabbitMQ Overview | Grafana Labs
• Dashboard: RabbitMQ-Overview

Dashboard: RabbitMQ-Overview

• Download: https://grafana.com/grafana/dashboards/10991-rabbitmq-overview/ 
• Plugins
  • Stat (built-in)
    
  • Table (built-in)
    
  • Time series (built-in)
    

RabbitMQ Cluster

• Clustering Guide — RabbitMQ
• RabbitMQ Learning III: RabbitMQ Clustering and Load Balancing
• How to Set Up the RabbitMQ Cluster on Ubuntu/Debian Linux
• Clustering Guide (vmware.com)
• RabbitMQ 集群搭建 (with HAProxy)
• RabbitMQ集群架构模式 | guaosi的博客

For Windows only

• 如何在 Windwos 上設定 RabbitMQ Cluster

A few things to RabbitMQ Cluster

• Classic Queues 預設只會在 Cluster 其中一個 node，而訊息仍可以透過其他 node 發送或接收；如果 Queues 要抄寫至所有 nodes，必須設定為 Quorum Queues。
• Cluster 裡每一個 node 都是視為一個 peer，彼此都是相同的，沒有主次的分別。
• Cluster 的每個 node 之間使用 cookie 認證方式，cookie 檔路徑是 /var/lib/rabbitmq/.erlang.cookie 。
• 一個 Cluster 組成 的 Node 數量，應該為奇數，例如 3、5、7等。原因是這樣的個數，才能在有 node 中斷服務時，讓 Cluster 有多數決的識別並形成共識。
• 所有 Node 主機重新啟動時，如果只剩 1 個 Node 主機可正常運行，將無法正常啟動服務，直到第 2 個Node 主機恢復連線，服務才會重新恢復運作。 
• Cluster 的網路架構必須是 LAN，而不要在 WAN。原因是：同個 Cluster 的不同 Node 間的網路通訊必須保持連線，一旦有任一個 Node 失去網路連線超過 60 秒，Cluster 就會發生 Network Partition 異常，也就是 split-brain (腦裂) 事件 。這個異常可能會導致 Cluster 無法工作，因應的處理程序可以參考：Clustering and Network Partitions
  
• 同個 Cluster 的所有節點的設定，例如帳號認證、Queues、Exchanges、Routing Key以及其他設定均保持同步。
• Node Plugin 設定不會同步。

實驗節點

1. tpeeaprmq98 (node01)
2. tpeeaprmq981 (node02)
3. tpeeaprmq982 (node03)

/etc/hosts:

10.14.2.51      tpeeaprmq98
10.4.1.33       tpeeaprmq981
10.4.1.34       tpeeaprmq982

安裝 RabbitMQ

所有節點主機須完成 RabbitMQ 主程式安裝。

同步 Cookie 認證檔

不同 node 之間的通訊是以 Erlang cookie 檔做認證。

通常啟動 RabbitMQ 服務時，cookie 檔案會自動建立。可以自行變更檔案內容的複雜度，且檔案權限必須是 0600 。同一個 Cluster 的每一個 node 必須要有相同的 cookit 檔。

scp /var/lib/rabbitmq/.erlang.cookie root@tpeeaprmq981:/var/lib/rabbitmq/
scp /var/lib/rabbitmq/.erlang.cookie root@tpeeaprmq982:/var/lib/rabbitmq/

cookie 檔位置

預設路徑是 /var/lib/rabbitmq/.erlang.cookie。

rabbitmq-diagnostics erlang_cookie_sources

新增 Cluster

Detach the service of all nodes

建立新的 Cluster 之前，所有 node 必須先卸載舊的 Cluster。

# On Node01
rabbitmq-server -detached

# On Node02
rabbitmq-server -detached

# On Node03
rabbitmq-server -detached

Verify the cluster status

[root@tpeeaprmq98 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq98 ...
Basics

Cluster name: rabbit@tpeeaprmq98

Disk Nodes

rabbit@tpeeaprmq98

Running Nodes

rabbit@tpeeaprmq98

Versions

rabbit@tpeeaprmq98: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq98, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq98, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq98, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq98, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq98, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

[root@tpeeaprmq981 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq981 ...
Basics

Cluster name: rabbit@tpeeaprmq981

Disk Nodes

rabbit@tpeeaprmq981

Running Nodes

rabbit@tpeeaprmq981

Versions

rabbit@tpeeaprmq981: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq981, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq981, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq981, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq981, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq981, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

[root@tpeeaprmq982 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq982 ...
Basics

Cluster name: rabbit@tpeeaprmq982

Disk Nodes

rabbit@tpeeaprmq982

Running Nodes

rabbit@tpeeaprmq982

Versions

rabbit@tpeeaprmq982: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq982, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq982, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq982, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq982, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq982, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Creating a Cluster

將 node02 與 node03 加入到 node01。

# On Node02
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@tpeeaprmq98
rabbitmqctl start_app

# On Node03
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@tpeeaprmq98
rabbitmqctl start_app

加入 cluster 時，出現以下訊息，可以直接忽略。

[root@tpeeaprmq981 ~]# rabbitmqctl join_cluster rabbit@tpeeaprmq98
Clustering node rabbit@tpeeaprmq981 with rabbit@tpeeaprmq98

15:10:18.438 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags`:

15:10:18.438 [warning] Feature flags:   - list of previously disabled feature flags now marked as such: [:maintenance_mode_status]

15:10:18.561 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags`:

15:10:18.561 [warning] Feature flags:   - list of previously enabled feature flags now marked as such: [:maintenance_mode_status]

15:10:18.598 [error] Failed to create a tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}

15:10:18.598 [error] Failed to create a per-vhost tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}

15:10:18.599 [error] Failed to create a per-user tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}

Verify the cluster status

每個 node 的 Cluster 狀態應該都要一樣，除了 Cluster name 會顯示目前所在的 node name。

[root@tpeeaprmq98 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq98 ...
Basics

Cluster name: rabbit@tpeeaprmq98

Disk Nodes

rabbit@tpeeaprmq98
rabbit@tpeeaprmq981
rabbit@tpeeaprmq982

Running Nodes

rabbit@tpeeaprmq98
rabbit@tpeeaprmq981
rabbit@tpeeaprmq982

Versions

rabbit@tpeeaprmq98: RabbitMQ 3.10.7 on Erlang 25.0.4
rabbit@tpeeaprmq981: RabbitMQ 3.10.7 on Erlang 25.0.4
rabbit@tpeeaprmq982: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq98, status: not under maintenance
Node: rabbit@tpeeaprmq981, status: not under maintenance
Node: rabbit@tpeeaprmq982, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq98, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq98, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq98, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq98, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@tpeeaprmq981, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq981, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq981, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq981, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@tpeeaprmq982, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq982, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq982, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq982, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Node 管理

建議不要設定自動啟動，以避免重啟系統後，在沒有經過人工確認的程序，系統就自動加入原有的 Cluster，可能對 Cluster 線上服務造成影響。

systemctl disable rabbitmq-server

重啟 Node

# Recommend using systemd
systemctl stop rabbitmq-server
systemctl start rabbitmq-server

# Using rabbitmqctl + systemd
rabbitmqctl stop
systemctl start rabbitmq-server

# Using rabbitmqctl
# Stop the node
rabbitmqctl stop
# Satrt the node
rabbitmq-server -detached
# Verify if the node is awaiting schema table sync 
rabbitmq-diagnostics check_running

# Forcing node boot
rabbitmqctl force_boot

增加新 Node

NOTE: 加入到 Cluster 後，相關的認證、Queues、Exchange 以及其他設定均會同步到新 Node。

# Find out the path of Erlang Cookie file
rabbitmq-diagnostics erlang_cookie_sources
# Copy Cookie from one node of the cluster
scp /var/lib/rabbitmq/.erlang.cookie root@<new-node>:/var/lib/rabbitmq/

# Join a new node into the cluster rabbit@tpeeaprmq98
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster <cluster-name>
rabbitmqctl start_app
# Alternatively, you can join it as RAM node by following command
rabbitmqctl join_cluster <cluster-name>  --ram

移除 Node

NOTE: Node 卸載 Cluster 後，這個 Node 的相關的認證、Queues、Exchange 以及其他設定均會被清除。

# 正常卸載 node
# 在要卸載的 node 執行
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl cluster_status

# 強制移除 node
# 在 Cluster 的其他 node 執行 
rabbitmqctl forget_cluster_node <node-name>

Rebalance the queues across node

每一個 quorum queue 在任一個 node 上完成宣告後，會自動同步到每個其他的 node，並且隨機的區分為一個 Leader node，其餘則為 Follower node。

Queue 平常的工作負載以 Leader node 為主，除非遇到 Leader node 停止服務，系統就會從現有 Follower nodes 挑選其中一個成為新的 Leader node。

所有的 quorum queue 的 Leader node 應該要均衡分配到每一個 node 之間，這樣的效能就可以平均負載到所有 node。

• After restarting a node
  
• After joining a node

NOTE: 每個 queue 目前的 Leader node 可以從 Web-UI 得知。

queue 的 Leader Node 與 Channel Node 不一定會是同一個，有時會不同。

rabbitmq-queues rebalance all
rabbitmq-queues rebalance "all" --vhost-pattern "itp_server" --queue-pattern ".*"

FAQ

Q: 無法啟動 node 服務

Application rabbit exited with reason: {{could_not_write_file,"/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq982/cluster_nodes.config",enospc},{rabbit,start,[normal,[]]}}

Solution:

可能是磁碟空間使用爆了，移除目錄 /var/lib/rabbitmq/mnesia 底下 node 資料子目錄。

[root@tpeeaprmq981 ~]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     1.8G     0  1.8G   0% /dev
tmpfs                        1.8G  4.0K  1.8G   1% /dev/shm
tmpfs                        1.8G   24M  1.8G   2% /run
tmpfs                        1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv    9.0G  4.0G  5.1G  45% /
/dev/sda2                   1014M  344M  671M  34% /boot
/dev/sda1                    599M  5.8M  594M   1% /boot/efi
/dev/mapper/rootvg-mqdatalv  5.0G  5.0G   20K 100% /var/lib/rabbitmq
/dev/mapper/rootvg-homelv    507M   30M  478M   6% /home
/dev/mapper/rootvg-worktmp   507M   46M  462M   9% /worktmp
/dev/mapper/rootvg-optlv     2.0G  997M  1.1G  49% /opt
tmpfs                        364M     0  364M   0% /run/user/0
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]# du -csh /var/lib/rabbitmq/mnesia/*
204K    /var/lib/rabbitmq/mnesia/rabbit@rmq981
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rmq981-feature_flags
0       /var/lib/rabbitmq/mnesia/rabbit@rmq981-plugins-expand
300K    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98
5.0G    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981
4.0K    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags
0       /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
4.0K    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-feature_flags
0       /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-plugins-expand
5.0G    total

[root@tpeeaprmq981 ~]# rm -rf /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981
[root@tpeeaprmq981 ~]# rm /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags
rm: remove regular file '/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags'? y
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]# rm /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
rm: cannot remove '/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand': Is a directory
[root@tpeeaprmq981 ~]# rm -rf /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand

Q: 無法加入 Cluster

[error] Node rabbit@tpeeaprmq98 thinks it's clustered with node rabbit@tpeeaprmq982, but rabbit@tpeeaprmq982 disagrees

Solution:

到 node rabbit@tpeeaprmq98 執行 rabbitmqctl cluster_status ，如果有顯示 node rabbit@tpeeaprmq982 ，執行強制移除指令。

# On the node rabbit@tpeeaprmq98
rabbitmqctl forget_cluster_node rabbit@tpeeaprmq982

Q: Network partition detected

Web UI 出現告警訊息：

Node 執行 rabbitmqctl cluster_status 出現 Network Partitions

Network Partitions

Node rabbit@tpeeaprmq98 cannot communicate with rabbit@tpeeaprmq982
Node rabbit@tpeeaprmq981 cannot communicate with rabbit@tpeeaprmq982

原因：tpeeaprmq982 由於硬體或網路異常造成意外的離線，當 node 重新恢復網路連線後，Cluster 會觸發 Network Partition 事件(aka split-brain 腦裂事件)，且停止 quorum queue 的資料複寫，必須盡速完成修復。

解決方案：修復 Network Partition 異常事件

• 官方教學: Clustering and Network Partitions — RabbitMQ

重啟發生問題 Node tpeeaprmq982 的服務

rabbitmqctl stop
systemctl start rabbitmq-server