RabbitMQ
RabbitMQ是實現了進階訊息佇列協定的開源訊息代理軟體。RabbitMQ伺服器是用Erlang語言編寫的,而群集和故障轉移是構建在開放電信平台框架上的。所有主要的程式語言均有與代理介面通訊的客戶端函式庫。
- Learning
- Benchmark
- Development with RabbitMQ
- RabbitMQ Standalone
- Install on RedHat
- Management UI
- Plugins
- Monitoring & Management
- RabbitMQ Cluster
- FAQ
Learning
Introduction
其他類似的開源方案
Installation
- Installing on RPM-based Linux (RedHat Enterprise Linux, CentOS, Fedora, openSUSE)
- Production Checklist
- RabbitMQ Checklist For Production Environments - A Complete Guide
Cluster & Load Balancing
- RabbitMQ 中的分散式,普通 cluster 模式的構建
- [VMWARE] VMware Tanzu RabbitMQ for Kubernetes Overview
- [HUAWEI] Distributed Message Service for RabbitMQ
Development
Commercial RabbitMQ
IBM MQ vs. RabbitMQ vs. Kafka
Benchmark
Java Tools
Install: RabbitMQ PerfTest
Usage
- --time: Seconds, 程式運行時間
- --uri: MQ Server 連線字串
- --producers: Producer 數量
- --consumers: Consumer 數量
- --size: bytes, Message 的 size
- --rate: msg/s, Producer/Consumer 的 Msg Rate
- --consumer-rate: msg/s, Consumer 的 Msg Rate
#> alternatives --list
libnssckbi.so.x86_64 auto /usr/lib64/pkcs11/p11-kit-trust.so
ld auto /usr/bin/ld.bfd
mta auto /usr/sbin/sendmail.postfix
ksh auto /bin/ksh93
java manual /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64/bin/java
jre_openjdk auto /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
jre_11 auto /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
jre_11_openjdk auto /usr/lib/jvm/jre-11-openjdk-11.0.8.10-1.el7.x86_64
#> export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
#> cd rabbitmq-perf-test-2.18.0
#> bin/runjava com.rabbitmq.perf.PerfTest --help
Scenario #1
- Condition:
- Queue: 200
- Producer: 200
- Consumer: 200
- Message Size: 20KB
- Message Rate: 20/s
- Ack Mode: Manual
- Throughput:
- Publish: 4000/s
- Consumer: 4000/s
- Resource:
- CPU (2 Core): 110 %
- Memory (4 GB): 8.8 %
bin/runjava com.rabbitmq.perf.PerfTest --uri amqp://<username>:<password>@<rabbitmq-server-ip>:<port>/<vhost> \
--queue-pattern 'perf-test-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 \
--rate 20
Scenario #2 for the Cluster with 3 nodes
Queue Type: Classic
bin/runjava com.rabbitmq.perf.PerfTest \
--uris "amqp://<username>:<password>@<rabbitmq-server1-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server2-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server3-ip>:<port>/<vhost>" \
--queue-pattern 'perf-test-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 --rate 20
Queue Type: Quorum
bin/runjava com.rabbitmq.perf.PerfTest \
--uris "amqp://<username>:<password>@<rabbitmq-server1-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server2-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server3-ip>:<port>/<vhost>" \
--queue-args x-queue-type=quorum \
--auto-delete false \
--flag persistent \
--queue-pattern 'perf-test-quorum-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 \
--rate 6
效能優化與影響
- 效能影響因素: How to Run Benchmarks | RabbitMQ - Blog
- 13 Common RabbitMQ Mistakes and How to Avoid Them - CloudAMQP
- Cluster Sizing and Other Considerations | RabbitMQ - Blog
Development with RabbitMQ
Bash
curl
curl -u login:pass -i -H "content-type:application/json" -X POST http://localhost:15672/api/exchanges/%2Fvhost/exchange/publish \
-d'{"properties":{},"routing_key":"","payload":"you message","payload_encoding":"string"}'
rabbitmqadmin
rabbitmqadmin
uses HTTP API authentication mechanism (basic HTTP authentication).
Install
# Install the dependency
yum install python3
# Download the script from the RabbitMQ Server
wget http://<rabbitmq-server-hostname>:15672/cli/rabbitmqadmin
#
chmod 0755 rabbitmqadmin
mv rabbitmqadmin /usr/local/bin
rabbitmqadmin -h
# Verify the connection via HTTP API
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> list vhosts
Basic Operation
## 在其他 Linux 主機上測試
# 宣告 queue
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> declare queue name=my-testq durable=true
# 發佈訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> publish exchange=amq.default routing_key=my-testq payload="This is Alang"
# 接收訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> get queue=my-testq ackmode=ack_requeue_false
# 接收訊息, 格式為 tsv
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> -f tsv get queue=my-testq ackmode=ack_requeue_false
# 一次接收 5 訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> get queue=my-testq count=5 ackmode=ack_requeue_false
while read -r line; do
echo $line | rabbitmqadmin publish exchange=amq.default routing_key=my_queue ;
done < messages
rabbitmqadmin publish exchange=amq.default routing_key=test payload="hello, world"
# With parallel
cat messages | parallel -j 100 \
./rabbitmqadmin -H $RABBITMQ_HOST \
-u $RABBITMQ_USERNAME \
-p $RABBITMQ_PASSWORD \
publish exchange=amq.default \
routing_key=myqueue \
payload="{}"
Python
Tutorials
- python rabbitmq Code Example (codegrepper.com)
- Part 2.3: Getting started with RabbitMQ and Python - CloudAMQP
- 利用Python操作訊息佇列RabbitMQ的方法教程 | 程式前沿 (codertw.com)
- Python 实现 RabbitMQ 的六种工作模式(附 Python 代码)_Python_AlwaysBeta_InfoQ写作社区
- RedHat AMQ Python Client
RedHat AMQ Python Client
Install via RHN
[root@dotnetdev ~]# dnf repolist all | grep amq-client
amq-clients-2-for-rhel-8-x86_64-debug-rpms disabled
amq-clients-2-for-rhel-8-x86_64-rpms disabled
amq-clients-2-for-rhel-8-x86_64-source-rpms disabled
amq-clients-2.9-for-rhel-8-x86_64-debug-rpms disabled
amq-clients-2.9-for-rhel-8-x86_64-rpms disabled
amq-clients-2.9-for-rhel-8-x86_64-source-rpms disabled
[root@dotnetdev ~]# subscription-manager repos --enable=amq-clients-2-for-rhel-8-x86_64-rpms
Repository 'amq-clients-2-for-rhel-8-x86_64-rpms' is enabled for this system.
[root@dotnetdev ~]# dnf repolist
Updating Subscription Management repositories.
repo id repo name
amq-clients-2-for-rhel-8-x86_64-rpms Red Hat AMQ Clients 2 for RHEL 8 x86_64 (RPMs)
rhel-8-for-x86_64-appstream-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
[root@dotnetdev ~]# yum install python3-qpid-proton python-qpid-proton-docs
PHP
Tutorials
dotNet
Tutorials
- [GitHub] AMQP.Net Lite
- RedHat AMQ .Net Client
- C# 連線至 RabbitMQ Cluster - 使用 RabbitMQ .Net Client 及 EasyNetQ
- [GitHub] EasyNetQ
Case: Console RabbitMQ Client
Package Required:
- RabbitMQ.Client
Target Framework:
- net6.0
Platform Supported:
- Windows 64
- RedHat Linux 8 64 (net6 runtime is required)
AMQP Client
Terms & Concepts
Queue
- prefetch: Consumer 的參數。當 consumer 接收訊息是採 PUSH 模式,且發送端會有大量訊息寫入,由於 consumer 的 PUSH 模式,會一次接收大量(一次幾百或幾千)的訊息,假使 consumer 未能及時處理所有訊息,那些待處理的訊息會處於 Unacked 狀態,一旦 Unacked 的訊息量過多,主機就會終止連線。
要解決這個問題,consumer 必須使用 prefetch 限制主機每次推送至 consumer 的訊息量。 - Lazy Queue: Queue 的參數。對於較長(大)的訊息,可以使用這類 queue,記憶體的耗用可以改善,但會增加磁碟空間的使用率。
Consume Message
- Consumer 接收訊息的開發方式有分 Push (推送) 與 Poll (拉) 兩種方式。
Publish Message
- 程式端發佈訊息,不像接收端那樣有多種客製參數與不同接收模式,唯一要注意的是主機效能與負載。多 nodes 的 Cluster 模式下,應該要將不同用戶端平均連接到不同的 nodes,達到負載均衡的效益。
- Publish Confirm:
TTL (Time to Live)
如果要讓訊息在超過所設置的時間,沒有被接收時即自動清除。這個設置稱為 TTL,在 RabbitMQ 裡 TTL 可以用在 Message 或 Queue 本身。
Message TTL
- 用途: 超過設置時間的訊息,即自動清除。
- 如果 Message 發送到多個 Queues 時,依照各自 Queue 的 TTL 的設置,訊息的清除時間可能會有不同。沒有設置 TTL 的 Queue,所屬的訊息則不會影響 。
- 參數值: Millisecond (60 seconds = 60000)
方法一: Policy
rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues
方法二: Queue 參數
- x-message-ttl: 60000
Sample codes in C#
var args = new Dictionary<string, object>();
args.Add("x-message-ttl", 60000);
model.QueueDeclare("myqueue", false, false, false, args);
Queue TTL
- 用途: 超過設置時間的 Queue,該 Queue 會自動清除。
- 參數值: Millisecond (30 mins = 1800000)
方法一: Policy
rabbitmqctl set_policy expiry ".*" '{"expires":1800000}' --apply-to queues
方法二: Queue 參數
- x-expires: 1800000
Sample codes in Java
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-expires", 1800000);
channel.queueDeclare("myqueue", false, false, false, args);
rabbitmqadmin
Usage
# Publish a message
rabbitmqadmin -H <rabbitmq-server-ip> -u <user-name> -p <secret> -V <virtual-server> publish exchange=amq.default routing_key=my-testq payload="This is Alang"
# Consume/Get a message
rabbitmqadmin -H <rabbitmq-server-ip> -u <user-name> -p <secret> -V <virtual-host> get queue=my-testq ackmode=ack_requeue_false
amqp-tools
A CLI tool is built-in Ubuntu.
Install
sudo apt update
sudo apt install amqp-tools
Usage
# Declare a queue
amqp-declare-queue --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" -d -q "my-testq"
# Publish a message
amqp-publish --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --routing-key="my-testq" -b "Hello,World"
# Get the messages (Poll mode)
amqp-get --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --queue="my-testq"
# Get the messages (Push mode)
amqp-consume --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --queue="my-testq" -p 2 ./show.sh
show.sh:
#!/usr/bin/env bash
read line
echo "Message: $line"
sleep 1
RabbitMQ Standalone
Opened Port if firewall is used
Ports Required:
- 4369: epmd, a peer discovery service used by RabbitMQ nodes and CLI tools
- 5672, 5671: used by AMQP 0-9-1 and 1.0 clients without and with TLS
- 25672: used for inter-node and CLI tools communication (Erlang distribution server port) and is allocated from a dynamic range (limited to a single port by default, computed as AMQP port + 20000). Unless external connections on these ports are really necessary (e.g. the cluster uses federation or CLI tools are used on machines outside the subnet), these ports should not be publicly exposed. See networking guide for details.
- 35672-35682: used by CLI tools (Erlang distribution client ports) for communication with nodes and is allocated from a dynamic range (computed as server distribution port + 10000 through server distribution port + 10010). See networking guide for details.
- 15672: HTTP API clients, management UI and rabbitmqadmin (only if the management plugin is enabled)
- 61613, 61614: STOMP clients without and with TLS (only if the STOMP plugin is enabled)
- 1883, 8883: MQTT clients without and with TLS, if the MQTT plugin is enabled
- 15674: STOMP-over-WebSockets clients (only if the Web STOMP plugin is enabled)
- 15675: MQTT-over-WebSockets clients (only if the Web MQTT plugin is enabled)
- 15692: Prometheus metrics (only if the Prometheus plugin is enabled)
Listener Port
Change the default port 5672
Edit: /etc/rabbitmq/rabbitmq.conf
## Networking
## ====================
##
## Related doc guide: https://rabbitmq.com/networking.html.
##
## By default, RabbitMQ will listen on all interfaces, using
## the standard (reserved) AMQP 0-9-1 and 1.0 port.
##
# listeners.tcp.default = 5672
listeners.tcp.default = 15690
Restart the RabbitMQ Service
# Using systemctl
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
# Alternatively, using rabbitmqctl
rabbitmqctl stop_app
rabbitmqctl start_app
Users and Permissions
Default User Access
The broker creates a user guest
with password guest
. Unconfigured clients will in general use these credentials. By default, these credentials can only be used when connecting to the broker as localhost so you will need to take action before connecting from any other machine.
See the documentation on access control for information on how to create more users and delete the guest user.
Adding/Listing/Deleting Users
## Adding Users
# will prompt for password, only use this option interactively
rabbitmqctl add_user "username"
# Password is provided via standard input.
# Note that certain characters such as $, &, &, #, and so on must be escaped to avoid
# special interpretation by the shell.
echo '2a55f70a841f18b97c3a7db939b7adc9e34a0f1b' | rabbitmqctl add_user 'username'
Password is provided as a command line argument.
# Note that certain characters such as $, &, &, #, and so on must be escaped to avoid
# special interpretation by the shell.
rabbitmqctl add_user 'username' '2a55f70a841f18b97c3a7db939b7adc9e34a0f1b'
## Listing User
rabbitmqctl list_users
rabbitmqctl list_users --formatter=json
## Deleting a user
rabbitmqctl delete_user 'username'
## Verifying a user
rabbitmqctl authenticate_user "a-username" "a-password"
Granting Permissions to a User
# First ".*" for configure permission on every entity
# Second ".*" for write permission on every entity
# Third ".*" for read permission on every entity
rabbitmqctl set_permissions -p "custom-vhost" "username" ".*" ".*" ".*"
# tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags username administrator
## Verifying the permission
# => Listing permissions for vhost "/" ...
# => user configure write read
# => user2 .* .* .*
# => guest .* .* .*
# => temp-user .* .* .*
rabbitmqctl list_permissions --vhost /
rabbitmqctl list_permissions --vhost gw1
Clearing Permissions of a User in a Virtual Host
# Revokes permissions in a virtual host
rabbitmqctl clear_permissions -p "custom-vhost" "username"
Operations on Multiple Virtual Hosts
# Assumes a Linux shell.
# Grants a user permissions to all virtual hosts.
for v in $(rabbitmqctl list_vhosts --silent); do rabbitmqctl set_permissions -p $v "a-user" ".*" ".*" ".*"; done
Virtual Hosts
Creating a Virtual Host
## Using CLI Tools
rabbitmqctl add_vhost qa1
## Using HTTP API
curl -u userename:pa$sw0rD -X PUT http://rabbitmq.local:15672/api/vhosts/vh1
Deleting a Virtual Host
## Using CLI Tools
rabbitmqctl delete_vhost qa1
## Using HTTP API
curl -u userename:pa$sw0rD -X DELETE http://rabbitmq.local:15672/api/vhosts/vh1
Kernel Limits
RabbitMQ nodes are most commonly affected by the maximum open file handle limit. Default limit value on most Linux distributions is usually 1024, which is very low for a messaging broker (or generally, any data service). See Production Checklist for recommended values.
Open File Limit
With systemd (Modern Linux Distributions)
新增: /etc/systemd/system/rabbitmq-server.service.d/limits.conf
[Service]
LimitNOFILE=64000
Restart the service
systemctl daemon-reload
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
Verify the change
#> ps -ef | grep rabbitmq
rabbitmq 460668 460654 0 11:43 ? 00:00:00 erl_child_setup 64000 <====
#> rabbitmqctl status
File Descriptors
Total: 236, limit: 63903
Sockets: 230, limit: 57510
TLS Connection
Data Directory
Environment Variables
Name |
Description |
RABBITMQ_BASE | Note: Windows only. This base directory contains sub-directories for the RabbitMQ server's database and log files. Alternatively, set RABBITMQ_MNESIA_BASE and RABBITMQ_LOG_BASE individually. |
RABBITMQ_MNESIA_BASE | This base directory contains sub-directories for the RabbitMQ server's node database, message store and cluster state files, one for each node, unless RABBITMQ_MNESIA_DIR is set explicitly. It is important that effective RabbitMQ user has sufficient permissions to read, write and create files and subdirectories in this directory at any time. This variable is typically not overridden. Usually RABBITMQ_MNESIA_DIR is overridden instead. |
RABBITMQ_MNESIA_DIR | The directory where this RabbitMQ node's data is stored. This s a schema database, message stores, cluster member information and other persistent node state. |
Default Locations for Linux
Name |
Location |
RABBITMQ_MNESIA_BASE | ${install_prefix}/var/lib/rabbitmq/mnesia |
RABBITMQ_MNESIA_DIR | $RABBITMQ_MNESIA_BASE/$RABBITMQ_NODENAME |
Default Locations for Windows
Name |
Location |
RABBITMQ_BASE | %APPDATA%\RabbitMQ |
RABBITMQ_MNESIA_BASE | %RABBITMQ_BASE%\db |
RABBITMQ_MNESIA_DIR | %RABBITMQ_MNESIA_BASE%\%RABBITMQ_NODENAME%-mnesia |
Erlang Cookie
RabbitMQ nodes 與 CLI tools 使用 cookie 做通訊時的認證,cookie 檔的路徑是
-
/var/lib/rabbitmq/.erlang.cookie
(used by the server)。 $HOME/.erlang.cookie
(used by the CLI tools)。
或者執行
rabbitmq-diagnostics erlang_cookie_sources
重點整理
Q & A
How to Find Config File Location
1. Check the Log File
node : rabbit@example
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/advanced.config
: /etc/rabbitmq/rabbitmq.conf
2. The comand rabbitmq-diagnostics
or rabbitmqctl
rabbitmq-diagnostics status
rabbitmq-diagnostics status -n [node name]
rabbitmqctl status
不同平台的路徑位置
Platform | Default Configuration File Directory | Example Configuration File Paths |
Generic binary package | $RABBITMQ_HOME/etc/rabbitmq/ |
$RABBITMQ_HOME/etc/rabbitmq/rabbitmq.conf, $RABBITMQ_HOME/etc/rabbitmq/advanced.config |
Debian and Ubuntu | /etc/rabbitmq/ |
/etc/rabbitmq/rabbitmq.conf , /etc/rabbitmq/advanced.config |
RPM-based Linux | /etc/rabbitmq/ |
/etc/rabbitmq/rabbitmq.conf , /etc/rabbitmq/advanced.config |
Windows | %APPDATA%\RabbitMQ\ |
%APPDATA%\RabbitMQ\rabbitmq.conf , %APPDATA%\RabbitMQ\advanced.config |
MacOS Homebrew Formula | ${install_prefix}/etc/rabbitmq/ , and the Homebrew cellar prefix is usually /usr/local |
${install_prefix}/etc/rabbitmq/rabbitmq.conf , ${install_prefix}/etc/rabbitmq/advanced.config |
Install on RedHat
Reference
Using PackageCloud Yum Repository
Quick-Install
Prerequisites: Internet connection
## Uses a PackageCloud-provided Yum repository setup script.
## Always verify what is downloaded before piping it to a privileged shell!
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
Manual-Install
Import the signing keys required
## primary RabbitMQ signing key
rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
## modern Erlang repository
rpm --import https://packagecloud.io/rabbitmq/erlang/gpgkey
## RabbitMQ server repository
rpm --import https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
Add Yum Repositories for RabbitMQ and Modern Erlang
# In /etc/yum.repos.d/rabbitmq.repo
##
## Zero dependency Erlang
##
[rabbitmq_erlang]
name=rabbitmq_erlang
baseurl=https://packagecloud.io/rabbitmq/erlang/el/8/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/erlang/gpgkey
https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
[rabbitmq_erlang-source]
name=rabbitmq_erlang-source
baseurl=https://packagecloud.io/rabbitmq/erlang/el/8/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/erlang/gpgkey
https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
##
## RabbitMQ server
##
[rabbitmq_server]
name=rabbitmq_server
baseurl=https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
[rabbitmq_server-source]
name=rabbitmq_server-source
baseurl=https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
Install Packages with Yum
# Update Yum package metadata
yum update
yum -q makecache --disablerepo='*' --enablerepo='rabbitmq_erlang' --enablerepo='rabbitmq_server'
## install these dependencies from standard OS repositories
yum install socat logrotate
## install RabbitMQ and zero dependency Erlang from the above repositories,
## ignoring any versions provided by the standard repositories
yum install --repo rabbitmq_erlang --repo rabbitmq_server erlang rabbitmq-server
Run RabbitMQ Server
systemctl status rabbitmq-server
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
Check the status
Run: rabbitmqctl status
or rabbitmq-diagnostics status
#> rabbitmq-diagnostics status
Status of node rabbit@tpeeaprmq98 ...
Runtime
OS PID: 549246
OS: Linux
Uptime (seconds): 504837
Is under maintenance?: false
RabbitMQ version: 3.10.7
Node name: rabbit@tpeeaprmq98
Erlang configuration: Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 1.1.1k FIPS 25 Mar 2021
Erlang processes: 412 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60
Plugins
Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:
* rabbitmq_mqtt
* rabbitmq_management
* amqp_client
* rabbitmq_web_dispatch
* cowboy
* cowlib
* rabbitmq_management_agent
Data directory
Node data directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98/quorum/rabbit@tpeeaprmq98
Config files
Log file(s)
* /var/log/rabbitmq/rabbit@tpeeaprmq98.log
* /var/log/rabbitmq/rabbit@tpeeaprmq98_upgrade.log
* <stdout>
Alarms
(none)
Memory
Total memory used: 0.1408 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 1.526 gb
reserved_unallocated: 0.0882 gb (62.67 %)
code: 0.0351 gb (24.97 %)
other_proc: 0.0201 gb (14.28 %)
other_system: 0.0139 gb (9.85 %)
other_ets: 0.0032 gb (2.26 %)
plugins: 0.0018 gb (1.29 %)
atom: 0.0014 gb (1.01 %)
binary: 0.0009 gb (0.62 %)
metrics: 0.0006 gb (0.4 %)
mgmt_db: 0.0003 gb (0.22 %)
mnesia: 0.0001 gb (0.07 %)
msg_index: 0.0001 gb (0.07 %)
queue_procs: 0.0 gb (0.03 %)
quorum_ets: 0.0 gb (0.02 %)
connection_other: 0.0 gb (0.01 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
File Descriptors
Total: 6, limit: 63903
Sockets: 0, limit: 57510
Free Disk Space
Low free disk space watermark: 0.05 gb
Free disk space: 5.4187 gb
Totals
Connection count: 0
Queue count: 2
Virtual host count: 3
Listeners
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
FAQ
GPG Key Error
Errors during downloading metadata for repository 'rabbitmq_server':
- Curl error (35): SSL connect error for https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 ]
Solution: 如果外網有防火牆,必須開通 github.com,或者直接下載該 GPG key 檔,然後上傳到主機的目錄 /etc/pki/rpm-gpg
,並修改 /etc/yum.repos.d/rabbitmq.repo
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
file:///etc/pki/rpm-gpg/rabbitmq-release-signing-key.asc
# https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
Management UI
The RabbitMQ management plugin provides an HTTP-based API for management and monitoring of RabbitMQ nodes and clusters, along with a browser-based UI and a command line tool, rabbitmqadmin.
Getting Started
rabbitmq-plugins enable rabbitmq_management
Node restart is not required after plugin activation.
Auto Enable the plugin: https://www.rabbitmq.com/plugins.html#enabled-plugins-file
Management UI Access
The management UI can be accessed using a Web browser at http://{node-hostname}:15672/
.
Access and Permissions
# create a user
rabbitmqctl add_user sysadmin ThisIsPassword
# tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags sysadmin administrator
Plugins
https://www.rabbitmq.com/plugins.html
A list of plugins available locally
rabbitmq-plugins list
Enable/Disable the plugin
rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins disable rabbitmq_management
Different Ways to Enable Plugins
NOTE: 在版本 3.10.7 不再需要手動編輯檔案,當使用指令 enable/disable 時,系統會自動編輯這個檔案。
[root@tpeeaprmq98 ~]# rabbitmq-plugins directories -s
Plugin archives directory: /usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.10.7/plugins
Plugin expansion directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-plugins-expand
Enabled plugins file: /etc/rabbitmq/enabled_plugins
Edit the file: /etc/rabbitmq/enabled_plugins
[rabbitmq_management,rabbitmq_amqp1_0,rabbitmq_mqtt].
重啟服務
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
Monitoring & Management
Web UI
Overview
- Ready: 已經發佈到 Queue 的訊息量
- Unacked: 消費端已接收但尚未 Ack 的訊息量
- Total: Ready + Unacked
CLI
rabbitmq-diagnostics
Online Resource Utilization
rabbitmq-diagnostics observer
RabbitMQ Version
[root@tpeeaprmq98 ~]# rabbitmq-diagnostics server_version
Asking node rabbit@tpeeaprmq98 for its RabbitMQ version...
3.10.7
Check the listener ports
rabbitmq-diagnostics -s listeners
rabbitmqctl
List the queues
rabbitmqctl -p <vhost-name> list_queues name state durable arguments policy
rabbitmqctl -qs -p <vhost-name> list_queues name > queue_names.lst
User Management
- 其他 tag 資訊: Management Plugin — RabbitMQ
# List all users
rabbitmqctl list_users
# Create a new user
rabbitmqctl add_user "eapuser"
rabbitmqctl add_vhost "eap_server"
rabbitmqctl set_permissions -p "eap_server" "eapuser" ".*" ".*" ".*"
rabbitmqctl set_user_tags eapuser monitoring # for web login only
# Remove tag 'monitoring' from user if needed
rabbitmqctl set_user_tags eapuser ""
# Tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags username administrator
## Verifying the permission
# => Listing permissions for vhost "/" ...
# => user configure write read
# => user2 .* .* .*
# => guest .* .* .*
# => temp-user .* .* .*
rabbitmqctl list_permissions --vhost /
rabbitmqctl list_permissions --vhost gw1
# Revoke user access
rabbitmqctl delete_user 'username'
Connections
rabbitmqctl list_connections
rabbitmqctl list_connections user,peer_host,peer_port,channels,state
Recreate the virtual host
rabbitmqctl delete_vhost <my-vhost-name>
rabbltmqctl add_vhost <my-vhost-name>
Reset the RabbitMQ Node
The broker drops all virtual hosts, queues, exchanges, and non-administrative users.
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
Force Reset the RabbitMQ Node
如果 Cluster 損壞且無法恢復運作時,可以嘗試強制重設 Node。
rabbitmqctl stop_app
rabbitmqctl force_reset
rabbitmqctl start_app
Suspend all listeners and prevent new client connections
暫停 listener 後,可以使用指令
ss -ltpn
檢查所有的 listener port 是否關閉。既有的連線不會受到影響,不過新的連線將無法建立。
# For current node
rabbitmqctl suspend_listeners
# suspends listeners on node rabbit@node2.cluster.rabbitmq.svc: it won't accept any new client connections
rabbitmqctl suspend_listeners -n rabbit@node2.cluster.rabbitmq.svc
# For current node, to resume all listeners on a node and make it accept new client connections again
rabbitmqctl resume_listeners
# resumes listeners on node rabbit@node2.cluster.rabbitmq.svc: it will accept new client connections again
rabbitmqctl resume_listeners -n rabbit@node2.cluster.rabbitmq.svc
rabbitmqadmin
Basic Operation
# List queus
rabbitmqadmin list queues
rabbitmqadmin -H <RabbitMQ-Server-IP> -u <username> -p <password> -V <vhost-name> list queues
# Add a queue with optional parameters
rabbitmqadmin declare queue name=<my-new-queu> durable=true auto_delete=true
Remove multiple queues
rabbitmqadmin -f tsv -q list queues name > q.txt
while read -r name; do rabbitmqadmin -q delete queue name="${name}"; done < q.txt
Connections
rabbitmqadmin -H <RabbitMQ-Server-IP> -u <username> -p <password> -V <vhost-name> list connections name
# Close multiple connections without any channels
rabbitmqadmin -f tsv -q connections name channels | awk -F "\t" '($2 < 1) {print $1}' | tee conn_noChannels.lst
while read -r conn;do rabbitmqadmin close connection name="${conn}"; done < conn_noChannels.lst
Monitoring
Health-Check
rabbitmq-diagnostics check_running
rabbitmq-diagnostics ping
Queue State
rabbitmqctl list_queues name state
Cluster Status
rabbitmqctl cluster_status
Monitor with Prometheus
RabbitMQ Configuration
# Enable the plugin rabbitmq_prometheus
rabbitmq-plugins enable rabbitmq_prometheus
# To confirm that RabbitMQ now exposes metrics in Prometheus format
curl -s localhost:15692/metrics | head -n 10
Prometheus Configuration
/etc/hosts:
<node1.ip.addr> rmq01
<node2.ip.addr> rmq02
<node3.ip.addr> rmq03
prometheus.yml:
scrape_configs:
- job_name: rabbitmq
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 15s
static_configs:
- targets: ['rmq01:15692', 'rmq02:15692', 'rmq03:15692']
Grafana Configuration
Dashboard: RabbitMQ-Overview
- Download: https://grafana.com/grafana/dashboards/10991-rabbitmq-overview/
- Plugins
- Stat (built-in)
- Table (built-in)
- Time series (built-in)
- Stat (built-in)
RabbitMQ Cluster
- Clustering Guide — RabbitMQ
- RabbitMQ Learning III: RabbitMQ Clustering and Load Balancing
- How to Set Up the RabbitMQ Cluster on Ubuntu/Debian Linux
- Clustering Guide (vmware.com)
- RabbitMQ 集群搭建 (with HAProxy)
- RabbitMQ集群架构模式 | guaosi的博客
For Windows only
A few things to RabbitMQ Cluster
- Classic Queues 預設只會在 Cluster 其中一個 node,而訊息仍可以透過其他 node 發送或接收;如果 Queues 要抄寫至所有 nodes,必須設定為 Quorum Queues。
- Cluster 裡每一個 node 都是視為一個 peer,彼此都是相同的,沒有主次的分別。
- Cluster 的每個 node 之間使用 cookie 認證方式,cookie 檔路徑是
/var/lib/rabbitmq/.erlang.cookie
。 - 一個 Cluster 組成 的 Node 數量,應該為奇數,例如 3、5、7等。原因是這樣的個數,才能在有 node 中斷服務時,讓 Cluster 有多數決的識別並形成共識。
- 所有 Node 主機重新啟動時,如果只剩 1 個 Node 主機可正常運行,將無法正常啟動服務,直到第 2 個Node 主機恢復連線,服務才會重新恢復運作。
- Cluster 的網路架構必須是 LAN,而不要在 WAN。原因是:同個 Cluster 的不同 Node 間的網路通訊必須保持連線,一旦有任一個 Node 失去網路連線超過 60 秒,Cluster 就會發生 Network Partition 異常,也就是 split-brain (腦裂) 事件 。這個異常可能會導致 Cluster 無法工作,因應的處理程序可以參考:Clustering and Network Partitions
- 同個 Cluster 的所有節點的設定,例如帳號認證、Queues、Exchanges、Routing Key以及其他設定均保持同步。
- Node Plugin 設定不會同步。
實驗節點
- tpeeaprmq98 (node01)
- tpeeaprmq981 (node02)
- tpeeaprmq982 (node03)
/etc/hosts:
10.14.2.51 tpeeaprmq98
10.4.1.33 tpeeaprmq981
10.4.1.34 tpeeaprmq982
安裝 RabbitMQ
所有節點主機須完成 RabbitMQ 主程式安裝。
同步 Cookie 認證檔
不同 node 之間的通訊是以 Erlang cookie 檔做認證。
通常啟動 RabbitMQ 服務時,cookie 檔案會自動建立。可以自行變更檔案內容的複雜度,且檔案權限必須是 0600
。同一個 Cluster 的每一個 node 必須要有相同的 cookit 檔。
scp /var/lib/rabbitmq/.erlang.cookie root@tpeeaprmq981:/var/lib/rabbitmq/
scp /var/lib/rabbitmq/.erlang.cookie root@tpeeaprmq982:/var/lib/rabbitmq/
預設路徑是
/var/lib/rabbitmq/.erlang.cookie
。
rabbitmq-diagnostics erlang_cookie_sources
新增 Cluster
Detach the service of all nodes
建立新的 Cluster 之前,所有 node 必須先卸載舊的 Cluster。
# On Node01
rabbitmq-server -detached
# On Node02
rabbitmq-server -detached
# On Node03
rabbitmq-server -detached
Verify the cluster status
[root@tpeeaprmq98 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq98 ...
Basics
Cluster name: rabbit@tpeeaprmq98
Disk Nodes
rabbit@tpeeaprmq98
Running Nodes
rabbit@tpeeaprmq98
Versions
rabbit@tpeeaprmq98: RabbitMQ 3.10.7 on Erlang 25.0.4
Maintenance status
Node: rabbit@tpeeaprmq98, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@tpeeaprmq98, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq98, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq98, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq98, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Feature flags
Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
[root@tpeeaprmq981 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq981 ...
Basics
Cluster name: rabbit@tpeeaprmq981
Disk Nodes
rabbit@tpeeaprmq981
Running Nodes
rabbit@tpeeaprmq981
Versions
rabbit@tpeeaprmq981: RabbitMQ 3.10.7 on Erlang 25.0.4
Maintenance status
Node: rabbit@tpeeaprmq981, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@tpeeaprmq981, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq981, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq981, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq981, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Feature flags
Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
[root@tpeeaprmq982 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq982 ...
Basics
Cluster name: rabbit@tpeeaprmq982
Disk Nodes
rabbit@tpeeaprmq982
Running Nodes
rabbit@tpeeaprmq982
Versions
rabbit@tpeeaprmq982: RabbitMQ 3.10.7 on Erlang 25.0.4
Maintenance status
Node: rabbit@tpeeaprmq982, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@tpeeaprmq982, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq982, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq982, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq982, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Feature flags
Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
Creating a Cluster
將 node02 與 node03 加入到 node01。
# On Node02
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@tpeeaprmq98
rabbitmqctl start_app
# On Node03
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@tpeeaprmq98
rabbitmqctl start_app
加入 cluster 時,出現以下訊息,可以直接忽略。
[root@tpeeaprmq981 ~]# rabbitmqctl join_cluster rabbit@tpeeaprmq98
Clustering node rabbit@tpeeaprmq981 with rabbit@tpeeaprmq98
15:10:18.438 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags`:
15:10:18.438 [warning] Feature flags: - list of previously disabled feature flags now marked as such: [:maintenance_mode_status]
15:10:18.561 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags`:
15:10:18.561 [warning] Feature flags: - list of previously enabled feature flags now marked as such: [:maintenance_mode_status]
15:10:18.598 [error] Failed to create a tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}
15:10:18.598 [error] Failed to create a per-vhost tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}
15:10:18.599 [error] Failed to create a per-user tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}
Verify the cluster status
每個 node 的 Cluster 狀態應該都要一樣,除了 Cluster name 會顯示目前所在的 node name。
[root@tpeeaprmq98 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq98 ...
Basics
Cluster name: rabbit@tpeeaprmq98
Disk Nodes
rabbit@tpeeaprmq98
rabbit@tpeeaprmq981
rabbit@tpeeaprmq982
Running Nodes
rabbit@tpeeaprmq98
rabbit@tpeeaprmq981
rabbit@tpeeaprmq982
Versions
rabbit@tpeeaprmq98: RabbitMQ 3.10.7 on Erlang 25.0.4
rabbit@tpeeaprmq981: RabbitMQ 3.10.7 on Erlang 25.0.4
rabbit@tpeeaprmq982: RabbitMQ 3.10.7 on Erlang 25.0.4
Maintenance status
Node: rabbit@tpeeaprmq98, status: not under maintenance
Node: rabbit@tpeeaprmq981, status: not under maintenance
Node: rabbit@tpeeaprmq982, status: not under maintenance
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@tpeeaprmq98, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq98, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq98, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq98, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@tpeeaprmq981, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq981, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq981, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq981, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@tpeeaprmq982, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq982, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq982, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq982, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Feature flags
Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
Node 管理
建議不要設定自動啟動,以避免重啟系統後,在沒有經過人工確認的程序,系統就自動加入原有的 Cluster,可能對 Cluster 線上服務造成影響。
systemctl disable rabbitmq-server
重啟 Node
# Recommend using systemd
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
# Using rabbitmqctl + systemd
rabbitmqctl stop
systemctl start rabbitmq-server
# Using rabbitmqctl
# Stop the node
rabbitmqctl stop
# Satrt the node
rabbitmq-server -detached
# Verify if the node is awaiting schema table sync
rabbitmq-diagnostics check_running
# Forcing node boot
rabbitmqctl force_boot
增加新 Node
NOTE: 加入到 Cluster 後,相關的認證、Queues、Exchange 以及其他設定均會同步到新 Node。
# Find out the path of Erlang Cookie file
rabbitmq-diagnostics erlang_cookie_sources
# Copy Cookie from one node of the cluster
scp /var/lib/rabbitmq/.erlang.cookie root@<new-node>:/var/lib/rabbitmq/
# Join a new node into the cluster rabbit@tpeeaprmq98
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster <cluster-name>
rabbitmqctl start_app
# Alternatively, you can join it as RAM node by following command
rabbitmqctl join_cluster <cluster-name> --ram
移除 Node
NOTE: Node 卸載 Cluster 後,這個 Node 的相關的認證、Queues、Exchange 以及其他設定均會被清除。
# 正常卸載 node
# 在要卸載的 node 執行
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl cluster_status
# 強制移除 node
# 在 Cluster 的其他 node 執行
rabbitmqctl forget_cluster_node <node-name>
Rebalance the queues across node
每一個 quorum queue 在任一個 node 上完成宣告後,會自動同步到每個其他的 node,並且隨機的區分為一個 Leader node,其餘則為 Follower node。
Queue 平常的工作負載以 Leader node 為主,除非遇到 Leader node 停止服務,系統就會從現有 Follower nodes 挑選其中一個成為新的 Leader node。
所有的 quorum queue 的 Leader node 應該要均衡分配到每一個 node 之間,這樣的效能就可以平均負載到所有 node。
- After restarting a node
- After joining a node
NOTE: 每個 queue 目前的 Leader node 可以從 Web-UI 得知。
queue 的 Leader Node 與 Channel Node 不一定會是同一個,有時會不同。
rabbitmq-queues rebalance all
rabbitmq-queues rebalance "all" --vhost-pattern "itp_server" --queue-pattern ".*"
FAQ
Q: 無法啟動 node 服務
Application rabbit exited with reason: {{could_not_write_file,"/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq982/cluster_nodes.config",enospc},{rabbit,start,[normal,[]]}}
Solution:
可能是磁碟空間使用爆了,移除目錄 /var/lib/rabbitmq/mnesia 底下 node 資料子目錄。
[root@tpeeaprmq981 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 4.0K 1.8G 1% /dev/shm
tmpfs 1.8G 24M 1.8G 2% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv 9.0G 4.0G 5.1G 45% /
/dev/sda2 1014M 344M 671M 34% /boot
/dev/sda1 599M 5.8M 594M 1% /boot/efi
/dev/mapper/rootvg-mqdatalv 5.0G 5.0G 20K 100% /var/lib/rabbitmq
/dev/mapper/rootvg-homelv 507M 30M 478M 6% /home
/dev/mapper/rootvg-worktmp 507M 46M 462M 9% /worktmp
/dev/mapper/rootvg-optlv 2.0G 997M 1.1G 49% /opt
tmpfs 364M 0 364M 0% /run/user/0
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]# du -csh /var/lib/rabbitmq/mnesia/*
204K /var/lib/rabbitmq/mnesia/rabbit@rmq981
4.0K /var/lib/rabbitmq/mnesia/rabbit@rmq981-feature_flags
0 /var/lib/rabbitmq/mnesia/rabbit@rmq981-plugins-expand
300K /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98
5.0G /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981
4.0K /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags
0 /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
4.0K /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-feature_flags
0 /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-plugins-expand
5.0G total
[root@tpeeaprmq981 ~]# rm -rf /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981
[root@tpeeaprmq981 ~]# rm /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags
rm: remove regular file '/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags'? y
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]# rm /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
rm: cannot remove '/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand': Is a directory
[root@tpeeaprmq981 ~]# rm -rf /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
Q: 無法加入 Cluster
[error] Node rabbit@tpeeaprmq98 thinks it's clustered with node rabbit@tpeeaprmq982, but rabbit@tpeeaprmq982 disagrees
Solution:
到 node rabbit@tpeeaprmq98 執行 rabbitmqctl cluster_status
,如果有顯示 node rabbit@tpeeaprmq982 ,執行強制移除指令。
# On the node rabbit@tpeeaprmq98
rabbitmqctl forget_cluster_node rabbit@tpeeaprmq982
Q: Network partition detected
Web UI 出現告警訊息:
Node 執行 rabbitmqctl cluster_status
出現 Network Partitions
Network Partitions
Node rabbit@tpeeaprmq98 cannot communicate with rabbit@tpeeaprmq982
Node rabbit@tpeeaprmq981 cannot communicate with rabbit@tpeeaprmq982
原因:tpeeaprmq982 由於硬體或網路異常造成意外的離線,當 node 重新恢復網路連線後,Cluster 會觸發 Network Partition 事件(aka split-brain 腦裂事件),且停止 quorum queue 的資料複寫,必須盡速完成修復。
解決方案:修復 Network Partition 異常事件
重啟發生問題 Node tpeeaprmq982 的服務
rabbitmqctl stop
systemctl start rabbitmq-server