RabbitMQ

RabbitMQ是實現了進階訊息佇列協定的開源訊息代理軟體。RabbitMQ伺服器是用Erlang語言編寫的,而群集和故障轉移是構建在開放電信平台框架上的。所有主要的程式語言均有與代理介面通訊的客戶端函式庫。

Learning RabbitMQ

Introduction
Installation
Cluster & Load Balancing
Development
Commercial RabbitMQ

Benchmark

Java Tools

Install: RabbitMQ PerfTest

Usage

#> alternatives --list
libnssckbi.so.x86_64    auto    /usr/lib64/pkcs11/p11-kit-trust.so
ld      auto    /usr/bin/ld.bfd
mta     auto    /usr/sbin/sendmail.postfix
ksh     auto    /bin/ksh93
java    manual  /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64/bin/java
jre_openjdk     auto    /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
jre_11  auto    /usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
jre_11_openjdk  auto    /usr/lib/jvm/jre-11-openjdk-11.0.8.10-1.el7.x86_64

#> export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.8.10-1.el7.x86_64
#> cd rabbitmq-perf-test-2.18.0
#> bin/runjava com.rabbitmq.perf.PerfTest --help

Scenario #1

bin/runjava com.rabbitmq.perf.PerfTest --uri amqp://<username>:<password>@<rabbitmq-server-ip>:<port>/<vhost> \
--queue-pattern 'perf-test-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 \
--rate 20

Scenario #2 for the Cluster with 3 nodes

Queue Type: Classic

bin/runjava com.rabbitmq.perf.PerfTest \
--uris "amqp://<username>:<password>@<rabbitmq-server1-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server2-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server3-ip>:<port>/<vhost>" \
--queue-pattern 'perf-test-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 --rate 20

Queue Type: Quorum

bin/runjava com.rabbitmq.perf.PerfTest \
--uris "amqp://<username>:<password>@<rabbitmq-server1-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server2-ip>:<port>/<vhost>,amqp://<username>:<password>@<rabbitmq-server3-ip>:<port>/<vhost>" \
--queue-args x-queue-type=quorum \
--auto-delete false \
--flag persistent \
--queue-pattern 'perf-test-quorum-%d' \
--queue-pattern-from 1 \
--queue-pattern-to 200 \
--producers 200 \
--consumers 200 \
--size 20000 \
--rate 6
效能優化與影響






Development with RabbitMQ

Development with RabbitMQ

Bash

curl
curl -u login:pass -i -H "content-type:application/json" -X POST http://localhost:15672/api/exchanges/%2Fvhost/exchange/publish \
 -d'{"properties":{},"routing_key":"","payload":"you message","payload_encoding":"string"}'
rabbitmqadmin

rabbitmqadmin uses HTTP API authentication mechanism (basic HTTP authentication).

Install

# Install the dependency
yum install python3

# Download the script from the RabbitMQ Server
wget http://<rabbitmq-server-hostname>:15672/cli/rabbitmqadmin

#
chmod 0755 rabbitmqadmin
mv rabbitmqadmin /usr/local/bin
rabbitmqadmin -h

# Verify the connection via HTTP API
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> list vhosts

Basic Operation

## 在其他 Linux 主機上測試
# 宣告 queue
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> declare queue name=my-testq durable=true

# 發佈訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> publish exchange=amq.default routing_key=my-testq payload="This is Alang"

# 接收訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> get queue=my-testq ackmode=ack_requeue_false
# 接收訊息, 格式為 tsv
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> -f tsv get queue=my-testq ackmode=ack_requeue_false
# 一次接收 5 訊息
rabbitmqadmin -H <rabbitmq-server-hostname> -u <username> -p <password> -V <vhost-name> get queue=my-testq count=5 ackmode=ack_requeue_false
while read -r line; do 
  echo $line | rabbitmqadmin publish exchange=amq.default routing_key=my_queue ; 
done < messages

rabbitmqadmin publish exchange=amq.default routing_key=test payload="hello, world"

# With parallel
cat messages | parallel -j 100 \
  ./rabbitmqadmin -H $RABBITMQ_HOST \
                  -u $RABBITMQ_USERNAME \
                  -p $RABBITMQ_PASSWORD  \
                  publish exchange=amq.default \
                  routing_key=myqueue \
                  payload="{}"
Development with RabbitMQ

Python

Tutorials
RedHat AMQ Python Client

Install via RHN

[root@dotnetdev ~]# dnf repolist all | grep amq-client
amq-clients-2-for-rhel-8-x86_64-debug-rpms                                  disabled
amq-clients-2-for-rhel-8-x86_64-rpms                                        disabled
amq-clients-2-for-rhel-8-x86_64-source-rpms                                 disabled
amq-clients-2.9-for-rhel-8-x86_64-debug-rpms                                disabled
amq-clients-2.9-for-rhel-8-x86_64-rpms                                      disabled
amq-clients-2.9-for-rhel-8-x86_64-source-rpms                               disabled

[root@dotnetdev ~]# subscription-manager repos --enable=amq-clients-2-for-rhel-8-x86_64-rpms
Repository 'amq-clients-2-for-rhel-8-x86_64-rpms' is enabled for this system.

[root@dotnetdev ~]# dnf repolist
Updating Subscription Management repositories.
repo id                                              repo name
amq-clients-2-for-rhel-8-x86_64-rpms                 Red Hat AMQ Clients 2 for RHEL 8 x86_64 (RPMs)
rhel-8-for-x86_64-appstream-rpms                     Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-rpms                        Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)

[root@dotnetdev ~]# yum install python3-qpid-proton python-qpid-proton-docs
Development with RabbitMQ

PHP

Tutorials
Development with RabbitMQ

dotNet

Tutorials
Development with RabbitMQ

AMQP Client

Terms & Concepts

Queue

Consume Message

Publish Message

TTL (Time to Live)

如果要讓訊息在超過所設置的時間,沒有被接收時即自動清除。這個設置稱為 TTL,在 RabbitMQ 裡 TTL 可以用在 Message 或 Queue 本身。

Message TTL

方法一: Policy

rabbitmqctl set_policy TTL ".*" '{"message-ttl":60000}' --apply-to queues

方法二: Queue 參數

Sample codes in C#

var args = new Dictionary<string, object>();
args.Add("x-message-ttl", 60000);
model.QueueDeclare("myqueue", false, false, false, args);

Queue TTL

方法一: Policy

rabbitmqctl set_policy expiry ".*" '{"expires":1800000}' --apply-to queues

方法二: Queue 參數

Sample codes in Java

Map<String, Object> args = new HashMap<String, Object>();
args.put("x-expires", 1800000);
channel.queueDeclare("myqueue", false, false, false, args);
rabbitmqadmin

Usage

# Publish a message
rabbitmqadmin -H <rabbitmq-server-ip> -u <user-name> -p <secret> -V <virtual-server> publish exchange=amq.default routing_key=my-testq payload="This is Alang"

# Consume/Get a message
rabbitmqadmin -H <rabbitmq-server-ip> -u <user-name> -p <secret> -V <virtual-host> get queue=my-testq ackmode=ack_requeue_false
amqp-tools

A CLI tool is built-in Ubuntu.

Install

sudo apt update
sudo apt install amqp-tools

Usage

# Declare a queue
amqp-declare-queue --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" -d -q "my-testq"

# Publish a message
amqp-publish --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --routing-key="my-testq" -b "Hello,World"

# Get the messages (Poll mode)
amqp-get --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --queue="my-testq"

# Get the messages (Push mode)
amqp-consume --url="amqp://<user-name>:<secret>@<rabbitmq-server-ip>:<rabbitmq-server-port>/<virtual-server>" --queue="my-testq" -p 2 ./show.sh

show.sh:

#!/usr/bin/env bash
read line
echo "Message: $line"
sleep 1

RabbitMQ Standalone

Opened Port if firewall is used

Ports Required:

Listener Port

Change the default port 5672

Edit: /etc/rabbitmq/rabbitmq.conf

## Networking
## ====================
##
## Related doc guide: https://rabbitmq.com/networking.html.
##
## By default, RabbitMQ will listen on all interfaces, using
## the standard (reserved) AMQP 0-9-1 and 1.0 port.
##
# listeners.tcp.default = 5672
listeners.tcp.default = 15690

Restart the RabbitMQ Service

# Using systemctl
systemctl stop rabbitmq-server
systemctl start rabbitmq-server

# Alternatively, using rabbitmqctl
rabbitmqctl stop_app
rabbitmqctl start_app

Users and Permissions

Default User Access

The broker creates a user guest with password guest. Unconfigured clients will in general use these credentials. By default, these credentials can only be used when connecting to the broker as localhost so you will need to take action before connecting from any other machine.

See the documentation on access control for information on how to create more users and delete the guest user.

Adding/Listing/Deleting Users

## Adding Users
# will prompt for password, only use this option interactively
rabbitmqctl add_user "username"

# Password is provided via standard input.
# Note that certain characters such as $, &, &, #, and so on must be escaped to avoid
# special interpretation by the shell.
echo '2a55f70a841f18b97c3a7db939b7adc9e34a0f1b' | rabbitmqctl add_user 'username'

 Password is provided as a command line argument.
# Note that certain characters such as $, &, &, #, and so on must be escaped to avoid
# special interpretation by the shell.
rabbitmqctl add_user 'username' '2a55f70a841f18b97c3a7db939b7adc9e34a0f1b'

## Listing User
rabbitmqctl list_users
rabbitmqctl list_users --formatter=json

## Deleting a user
rabbitmqctl delete_user 'username'

## Verifying a user
rabbitmqctl authenticate_user "a-username" "a-password"

Granting Permissions to a User

# First ".*" for configure permission on every entity
# Second ".*" for write permission on every entity
# Third ".*" for read permission on every entity
rabbitmqctl set_permissions -p "custom-vhost" "username" ".*" ".*" ".*"

# tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags username administrator

## Verifying the permission
# => Listing permissions for vhost "/" ...
# => user    configure   write   read
# => user2   .*  .*  .*
# => guest   .*  .*  .*
# => temp-user   .*  .*  .*
rabbitmqctl list_permissions --vhost /
rabbitmqctl list_permissions --vhost gw1

Clearing Permissions of a User in a Virtual Host

# Revokes permissions in a virtual host
rabbitmqctl clear_permissions -p "custom-vhost" "username"

Operations on Multiple Virtual Hosts

# Assumes a Linux shell.
# Grants a user permissions to all virtual hosts.
for v in $(rabbitmqctl list_vhosts --silent); do rabbitmqctl set_permissions -p $v "a-user" ".*" ".*" ".*"; done

Virtual Hosts

Creating a Virtual Host

## Using CLI Tools
rabbitmqctl add_vhost qa1

## Using HTTP API
curl -u userename:pa$sw0rD -X PUT http://rabbitmq.local:15672/api/vhosts/vh1

Deleting a Virtual Host

## Using CLI Tools
rabbitmqctl delete_vhost qa1

## Using HTTP API
curl -u userename:pa$sw0rD -X DELETE http://rabbitmq.local:15672/api/vhosts/vh1

Kernel Limits

RabbitMQ nodes are most commonly affected by the maximum open file handle limit. Default limit value on most Linux distributions is usually 1024, which is very low for a messaging broker (or generally, any data service). See Production Checklist for recommended values.

Open File Limit

With systemd (Modern Linux Distributions)

新增: /etc/systemd/system/rabbitmq-server.service.d/limits.conf

[Service]
LimitNOFILE=64000

Restart the service

systemctl daemon-reload
systemctl stop rabbitmq-server
systemctl start rabbitmq-server

Verify the change

#> ps -ef | grep rabbitmq

rabbitmq  460668  460654  0 11:43 ?        00:00:00 erl_child_setup 64000 <====
#> rabbitmqctl status

File Descriptors

Total: 236, limit: 63903
Sockets: 230, limit: 57510

TLS Connection

Data Directory

Environment Variables

Name
Description
RABBITMQ_BASE Note: Windows only. This base directory contains sub-directories for the RabbitMQ server's database and log files. Alternatively, set RABBITMQ_MNESIA_BASE and RABBITMQ_LOG_BASE individually.
RABBITMQ_MNESIA_BASE This base directory contains sub-directories for the RabbitMQ server's node database, message store and cluster state files, one for each node, unless RABBITMQ_MNESIA_DIR is set explicitly. It is important that effective RabbitMQ user has sufficient permissions to read, write and create files and subdirectories in this directory at any time. This variable is typically not overridden. Usually RABBITMQ_MNESIA_DIR is overridden instead.
RABBITMQ_MNESIA_DIR The directory where this RabbitMQ node's data is stored. This s a schema database, message stores, cluster member information and other persistent node state.


Default Locations for Linux

Name
Location
RABBITMQ_MNESIA_BASE ${install_prefix}/var/lib/rabbitmq/mnesia
RABBITMQ_MNESIA_DIR $RABBITMQ_MNESIA_BASE/$RABBITMQ_NODENAME

Default Locations for Windows

Name
Location
RABBITMQ_BASE %APPDATA%\RabbitMQ
RABBITMQ_MNESIA_BASE %RABBITMQ_BASE%\db
RABBITMQ_MNESIA_DIR %RABBITMQ_MNESIA_BASE%\%RABBITMQ_NODENAME%-mnesia

Erlang Cookie

RabbitMQ nodes 與 CLI tools 使用 cookie 做通訊時的認證,cookie 檔的路徑是

或者執行

rabbitmq-diagnostics erlang_cookie_sources


重點整理


Q & A

How to Find Config File Location

1. Check the Log File

node           : rabbit@example
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/advanced.config
               : /etc/rabbitmq/rabbitmq.conf

2. The comand rabbitmq-diagnostics or rabbitmqctl

rabbitmq-diagnostics status
rabbitmq-diagnostics status -n [node name]
rabbitmqctl status

不同平台的路徑位置

Platform Default Configuration File Directory Example Configuration File Paths
Generic binary package $RABBITMQ_HOME/etc/rabbitmq/ $RABBITMQ_HOME/etc/rabbitmq/rabbitmq.conf, $RABBITMQ_HOME/etc/rabbitmq/advanced.config
Debian and Ubuntu /etc/rabbitmq/ /etc/rabbitmq/rabbitmq.conf, /etc/rabbitmq/advanced.config
RPM-based Linux /etc/rabbitmq/ /etc/rabbitmq/rabbitmq.conf, /etc/rabbitmq/advanced.config
Windows %APPDATA%\RabbitMQ\ %APPDATA%\RabbitMQ\rabbitmq.conf, %APPDATA%\RabbitMQ\advanced.config
MacOS Homebrew Formula ${install_prefix}/etc/rabbitmq/, and the Homebrew cellar prefix is usually /usr/local ${install_prefix}/etc/rabbitmq/rabbitmq.conf, ${install_prefix}/etc/rabbitmq/advanced.config


Install on RedHat

Reference

Using PackageCloud Yum Repository

Quick-Install

Prerequisites: Internet connection

## Uses a PackageCloud-provided Yum repository setup script.
## Always verify what is downloaded before piping it to a privileged shell!
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
Manual-Install

Import the signing keys required

## primary RabbitMQ signing key
rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
## modern Erlang repository
rpm --import https://packagecloud.io/rabbitmq/erlang/gpgkey
## RabbitMQ server repository
rpm --import https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey

Add Yum Repositories for RabbitMQ and Modern Erlang

# In /etc/yum.repos.d/rabbitmq.repo

##
## Zero dependency Erlang
##

[rabbitmq_erlang]
name=rabbitmq_erlang
baseurl=https://packagecloud.io/rabbitmq/erlang/el/8/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/erlang/gpgkey
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[rabbitmq_erlang-source]
name=rabbitmq_erlang-source
baseurl=https://packagecloud.io/rabbitmq/erlang/el/8/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/erlang/gpgkey
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

##
## RabbitMQ server
##

[rabbitmq_server]
name=rabbitmq_server
baseurl=https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[rabbitmq_server-source]
name=rabbitmq_server-source
baseurl=https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/SRPMS
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

Install Packages with Yum

# Update Yum package metadata
yum update
yum -q makecache --disablerepo='*' --enablerepo='rabbitmq_erlang' --enablerepo='rabbitmq_server'

## install these dependencies from standard OS repositories
yum install socat logrotate

## install RabbitMQ and zero dependency Erlang from the above repositories,
## ignoring any versions provided by the standard repositories
yum install --repo rabbitmq_erlang --repo rabbitmq_server erlang rabbitmq-server

Run RabbitMQ Server

systemctl status rabbitmq-server
systemctl stop rabbitmq-server
systemctl start rabbitmq-server
systemctl enable rabbitmq-server

Check the status

Run: rabbitmqctl status or rabbitmq-diagnostics status

#> rabbitmq-diagnostics status

Status of node rabbit@tpeeaprmq98 ...
Runtime

OS PID: 549246
OS: Linux
Uptime (seconds): 504837
Is under maintenance?: false
RabbitMQ version: 3.10.7
Node name: rabbit@tpeeaprmq98
Erlang configuration: Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 1.1.1k  FIPS 25 Mar 2021
Erlang processes: 412 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_mqtt
 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98/quorum/rabbit@tpeeaprmq98

Config files


Log file(s)

 * /var/log/rabbitmq/rabbit@tpeeaprmq98.log
 * /var/log/rabbitmq/rabbit@tpeeaprmq98_upgrade.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1408 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 1.526 gb

reserved_unallocated: 0.0882 gb (62.67 %)
code: 0.0351 gb (24.97 %)
other_proc: 0.0201 gb (14.28 %)
other_system: 0.0139 gb (9.85 %)
other_ets: 0.0032 gb (2.26 %)
plugins: 0.0018 gb (1.29 %)
atom: 0.0014 gb (1.01 %)
binary: 0.0009 gb (0.62 %)
metrics: 0.0006 gb (0.4 %)
mgmt_db: 0.0003 gb (0.22 %)
mnesia: 0.0001 gb (0.07 %)
msg_index: 0.0001 gb (0.07 %)
queue_procs: 0.0 gb (0.03 %)
quorum_ets: 0.0 gb (0.02 %)
connection_other: 0.0 gb (0.01 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 6, limit: 63903
Sockets: 0, limit: 57510

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 5.4187 gb

Totals

Connection count: 0
Queue count: 2
Virtual host count: 3

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT

FAQ

GPG Key Error

Errors during downloading metadata for repository 'rabbitmq_server':
  - Curl error (35): SSL connect error for https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 ]

Solution: 如果外網有防火牆,必須開通 github.com,或者直接下載該 GPG key 檔,然後上傳到主機的目錄 /etc/pki/rpm-gpg,並修改 /etc/yum.repos.d/rabbitmq.repo

# PackageCloud's repository key and RabbitMQ package signing key
gpgkey=https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey
       file:///etc/pki/rpm-gpg/rabbitmq-release-signing-key.asc
#       https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc


Management UI

The RabbitMQ management plugin provides an HTTP-based API for management and monitoring of RabbitMQ nodes and clusters, along with a browser-based UI and a command line tool, rabbitmqadmin.

Getting Started

rabbitmq-plugins enable rabbitmq_management

Node restart is not required after plugin activation.

Auto Enable the plugin: https://www.rabbitmq.com/plugins.html#enabled-plugins-file

Management UI Access

The management UI can be accessed using a Web browser at http://{node-hostname}:15672/.

Access and Permissions

# create a user
rabbitmqctl add_user sysadmin ThisIsPassword
# tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags sysadmin administrator



Plugins

https://www.rabbitmq.com/plugins.html

A list of plugins available locally

rabbitmq-plugins list

Enable/Disable the plugin

rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins disable rabbitmq_management
Different Ways to Enable Plugins

NOTE: 在版本 3.10.7 不再需要手動編輯檔案,當使用指令 enable/disable 時,系統會自動編輯這個檔案。

[root@tpeeaprmq98 ~]# rabbitmq-plugins directories -s
Plugin archives directory: /usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.10.7/plugins
Plugin expansion directory: /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-plugins-expand
Enabled plugins file: /etc/rabbitmq/enabled_plugins

Edit the file: /etc/rabbitmq/enabled_plugins

[rabbitmq_management,rabbitmq_amqp1_0,rabbitmq_mqtt].

重啟服務

systemctl stop rabbitmq-server
systemctl start rabbitmq-server

Monitoring & Management

Tutorials

Web UI

Overview

CLI

rabbitmq-diagnostics

Online Resource Utilization

rabbitmq-diagnostics observer

RabbitMQ Version

[root@tpeeaprmq98 ~]# rabbitmq-diagnostics server_version
Asking node rabbit@tpeeaprmq98 for its RabbitMQ version...
3.10.7
rabbitmqctl

List the queues

rabbitmqctl -p <vhost-name> list_queues name state durable arguments policy

rabbitmqctl -qs -p <vhost-name> list_queues name > queue_names.lst

User Management

# List all users
rabbitmqctl list_users

# Create a new user
rabbitmqctl add_user "username"
rabbitmqctl set_permissions -p "custom-vhost" "username" ".*" ".*" ".*"

# Tag the user with "administrator" for full management UI and HTTP API access
rabbitmqctl set_user_tags username administrator

## Verifying the permission
# => Listing permissions for vhost "/" ...
# => user    configure   write   read
# => user2   .*  .*  .*
# => guest   .*  .*  .*
# => temp-user   .*  .*  .*
rabbitmqctl list_permissions --vhost /
rabbitmqctl list_permissions --vhost gw1

Connections

rabbitmqctl list_connections

Recreate the virtual host

rabbitmqctl delete_vhost <my-vhost-name>
rabbltmqctl add_vhost <my-vhost-name>

Reset the RabbitMQ Node

The broker drops all virtual hosts, queues, exchanges, and non-administrative users.

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

Force Reset the RabbitMQ Node

如果 Cluster 損壞且無法恢復運作時,可以嘗試強制重設 Node。

rabbitmqctl stop_app
rabbitmqctl force_reset
rabbitmqctl start_app

Suspend all listeners and prevent new client connections

rabbitmqctl suspend_listeners

# suspends listeners on node rabbit@node2.cluster.rabbitmq.svc: it won't accept any new client connections
rabbitmqctl suspend_listeners -n rabbit@node2.cluster.rabbitmq.svc

# To resume all listeners on a node and make it accept new client connections again
rabbitmqctl resume_listeners
# resumes listeners on node rabbit@node2.cluster.rabbitmq.svc: it will accept new client connections again
rabbitmqctl resume_listeners -n rabbit@node2.cluster.rabbitmq.svc
rabbitmqadmin

Basic Operation

# List queus
rabbitmqadmin list queues
rabbitmqadmin -H <RabbitMQ-Server-IP> -u <username> -p <password> -V <vhost-name> list queues

# Add a queue with optional parameters
rabbitmqadmin declare queue name=<my-new-queu> durable=true auto_delete=true

Remove multiple queues

rabbitmqadmin -f tsv -q list queues name > q.txt
while read -r name; do rabbitmqadmin -q delete queue name="${name}"; done < q.txt

Monitoring

RabbitMQ Cluster

For Windows only

A few things to RabbitMQ Cluster
實驗節點
  1. tpeeaprmq98 (node01)
  2. tpeeaprmq981 (node02)
  3. tpeeaprmq982 (node03)

/etc/hosts:

10.14.2.51      tpeeaprmq98
10.4.1.33       tpeeaprmq981
10.4.1.34       tpeeaprmq982
安裝 RabbitMQ

所有節點主機須完成 RabbitMQ 主程式安裝。

scp /var/lib/rabbitmq/.erlang.cookie root@tpeeaprmq981:/var/lib/rabbitmq/
scp /var/lib/rabbitmq/.erlang.cookie root@tpeeaprmq982:/var/lib/rabbitmq/
新增 Cluster

Detach the service of all nodes

建立新的 Cluster 之前,所有 node 必須先卸載舊的 Cluster。

# On Node01
rabbitmq-server -detached

# On Node02
rabbitmq-server -detached

# On Node03
rabbitmq-server -detached

Verify the cluster status

[root@tpeeaprmq98 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq98 ...
Basics

Cluster name: rabbit@tpeeaprmq98

Disk Nodes

rabbit@tpeeaprmq98

Running Nodes

rabbit@tpeeaprmq98

Versions

rabbit@tpeeaprmq98: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq98, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq98, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq98, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq98, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq98, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: disabled
Flag: empty_basic_get_metric, state: disabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
[root@tpeeaprmq981 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq981 ...
Basics

Cluster name: rabbit@tpeeaprmq981

Disk Nodes

rabbit@tpeeaprmq981

Running Nodes

rabbit@tpeeaprmq981

Versions

rabbit@tpeeaprmq981: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq981, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq981, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq981, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq981, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq981, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
[root@tpeeaprmq982 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq982 ...
Basics

Cluster name: rabbit@tpeeaprmq982

Disk Nodes

rabbit@tpeeaprmq982

Running Nodes

rabbit@tpeeaprmq982

Versions

rabbit@tpeeaprmq982: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq982, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq982, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq982, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq982, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq982, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Creating a Cluster

將 node02 與 node03 加入到 node01。

# On Node02
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@tpeeaprmq98
rabbitmqctl start_app

# On Node03
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@tpeeaprmq98
rabbitmqctl start_app

加入 cluster 時,出現以下訊息,可以直接忽略。

[root@tpeeaprmq981 ~]# rabbitmqctl join_cluster rabbit@tpeeaprmq98
Clustering node rabbit@tpeeaprmq981 with rabbit@tpeeaprmq98

15:10:18.438 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags`:

15:10:18.438 [warning] Feature flags:   - list of previously disabled feature flags now marked as such: [:maintenance_mode_status]

15:10:18.561 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags`:

15:10:18.561 [warning] Feature flags:   - list of previously enabled feature flags now marked as such: [:maintenance_mode_status]

15:10:18.598 [error] Failed to create a tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}

15:10:18.598 [error] Failed to create a per-vhost tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}

15:10:18.599 [error] Failed to create a per-user tracked connection table for node :rabbit@tpeeaprmq981: {:node_not_running, :rabbit@tpeeaprmq981}

Verify the cluster status

每個 node 的 Cluster 狀態應該都要一樣,除了 Cluster name 會顯示目前所在的 node name。

[root@tpeeaprmq98 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@tpeeaprmq98 ...
Basics

Cluster name: rabbit@tpeeaprmq98

Disk Nodes

rabbit@tpeeaprmq98
rabbit@tpeeaprmq981
rabbit@tpeeaprmq982

Running Nodes

rabbit@tpeeaprmq98
rabbit@tpeeaprmq981
rabbit@tpeeaprmq982

Versions

rabbit@tpeeaprmq98: RabbitMQ 3.10.7 on Erlang 25.0.4
rabbit@tpeeaprmq981: RabbitMQ 3.10.7 on Erlang 25.0.4
rabbit@tpeeaprmq982: RabbitMQ 3.10.7 on Erlang 25.0.4

Maintenance status

Node: rabbit@tpeeaprmq98, status: not under maintenance
Node: rabbit@tpeeaprmq981, status: not under maintenance
Node: rabbit@tpeeaprmq982, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@tpeeaprmq98, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq98, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq98, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq98, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@tpeeaprmq981, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq981, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq981, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq981, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@tpeeaprmq982, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@tpeeaprmq982, interface: [::], port: 1883, protocol: mqtt, purpose: MQTT
Node: rabbit@tpeeaprmq982, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@tpeeaprmq982, interface: [::], port: 15690, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: stream_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled
Node 管理

重啟 Node

# Recommend using systemd
systemctl stop rabbitmq-server
systemctl start rabbitmq-server

# Using rabbitmqctl
# Stop the node
rabbitmqctl stop
# Satrt the node
rabbitmq-server -detached
# Verify if the node is awaiting schema table sync 
rabbitmq-diagnostics check_running

# Forcing node boot
rabbitmqctl force_boot

增加新 Node

NOTE: 加入到 Cluster 後,相關的認證、Queues、Exchange 以及其他設定均會同步。

移除 Node

NOTE: Node 卸載 Cluster 後,這個 Node 的相關的認證、Queues、Exchange 以及其他設定均會被清除。

# 正常卸載 node
# 在要卸載的 node 執行
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl cluster_status

# 強制移除 node
# 在 Cluster 的其他 node 執行 
rabbitmqctl forget_cluster_node <node-name>

Rebalance the queues across node

每一個 quorum queue 在任一個 node 上完成宣告後,會自動同步到每個其他的 node,並且隨機的區分為一個 Leader node,其餘則為 Follower node

Queue 平常的工作負載以 Leader node 為主,除非遇到 Leader node 停止服務,系統就會從現有 Follower nodes 挑選其中一個成為新的 Leader node。

所有的 quorum queue 的 Leader node 應該要均衡分配到每一個 node 之間,這樣的效能就可以平均負載到所有 node。

NOTE: 每個 queue 目前的 Leader node 可以從 Web-UI 得知。

queue 的 Leader Node 與 Channel Node 不一定會是同一個,有時會不同。

rabbitmq-queues rebalance all
rabbitmq-queues rebalance "all" --vhost-pattern "itp_server" --queue-pattern ".*"





FAQ

Q: 無法啟動 node 服務

Application rabbit exited with reason: {{could_not_write_file,"/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq982/cluster_nodes.config",enospc},{rabbit,start,[normal,[]]}}

Solution:

可能是磁碟空間使用爆了,移除目錄 /var/lib/rabbitmq/mnesia 底下 node 資料子目錄。

[root@tpeeaprmq981 ~]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
devtmpfs                     1.8G     0  1.8G   0% /dev
tmpfs                        1.8G  4.0K  1.8G   1% /dev/shm
tmpfs                        1.8G   24M  1.8G   2% /run
tmpfs                        1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv    9.0G  4.0G  5.1G  45% /
/dev/sda2                   1014M  344M  671M  34% /boot
/dev/sda1                    599M  5.8M  594M   1% /boot/efi
/dev/mapper/rootvg-mqdatalv  5.0G  5.0G   20K 100% /var/lib/rabbitmq
/dev/mapper/rootvg-homelv    507M   30M  478M   6% /home
/dev/mapper/rootvg-worktmp   507M   46M  462M   9% /worktmp
/dev/mapper/rootvg-optlv     2.0G  997M  1.1G  49% /opt
tmpfs                        364M     0  364M   0% /run/user/0
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]# du -csh /var/lib/rabbitmq/mnesia/*
204K    /var/lib/rabbitmq/mnesia/rabbit@rmq981
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rmq981-feature_flags
0       /var/lib/rabbitmq/mnesia/rabbit@rmq981-plugins-expand
300K    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98
5.0G    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981
4.0K    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags
0       /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
4.0K    /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-feature_flags
0       /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq98-plugins-expand
5.0G    total

[root@tpeeaprmq981 ~]# rm -rf /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981
[root@tpeeaprmq981 ~]# rm /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags
rm: remove regular file '/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-feature_flags'? y
[root@tpeeaprmq981 ~]#
[root@tpeeaprmq981 ~]# rm /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
rm: cannot remove '/var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand': Is a directory
[root@tpeeaprmq981 ~]# rm -rf /var/lib/rabbitmq/mnesia/rabbit@tpeeaprmq981-plugins-expand
Q: 無法加入 Cluster

[error] Node rabbit@tpeeaprmq98 thinks it's clustered with node rabbit@tpeeaprmq982, but rabbit@tpeeaprmq982 disagrees

Solution:

到 node rabbit@tpeeaprmq98 執行 rabbitmqctl cluster_status ,如果有顯示 node rabbit@tpeeaprmq982 ,執行強制移除指令。

# On the node rabbit@tpeeaprmq98
rabbitmqctl forget_cluster_node rabbit@tpeeaprmq982
Q: Network partition detected

Web UI 出現告警訊息:

network-partitions.png

Node 執行 rabbitmqctl cluster_status 出現 Network Partitions

Network Partitions

Node rabbit@tpeeaprmq98 cannot communicate with rabbit@tpeeaprmq982
Node rabbit@tpeeaprmq981 cannot communicate with rabbit@tpeeaprmq982

原因:tpeeaprmq982 由於硬體或網路異常造成意外的離線,當 node 重新恢復網路連線後,Cluster 會觸發 Network Partition 事件(aka split-brain 腦裂事件),且停止 quorum queue 的資料複寫,必須盡速完成修復。

解決方案:修復 Network Partition 異常事件

重啟發生問題 Node tpeeaprmq982 的服務

rabbitmqctl stop
systemctl start rabbitmq-server

network-partitions-2.png