# Prometheus

#### Installation

Download: [Download | Prometheus](https://prometheus.io/download/) ，select Operating system: linux, Architecture: amd64

```
tar xzf prometheus-2.43.0.linux-amd64.tar.gz
mv prometheus-2.43.0.linux-amd64 /opt/prometheus
```

第一次啟動

```bash
cd /opt/prometheus/
./prometheus --config.file="prometheus.yml"
```

網頁介面 (僅限本機端存取):

- http://localhost:9090/metrics  
    預設會有本機系統的效能指標
- http://localhost:9090/  
    選擇 Graph，在 Expression 輸入 `promhttp_metric_handler_requests_total` ，按下 Execute，會有圖形顯示。

#### Configuration

##### 資料儲存與清理週期  


```bash
./prometheus --config.file="prometheus.yml" \
    --storage.tsdb.path="/data/prometheus" \
    --storage.tsdb.retention.time=30d
```

- `--storage.tsdb.path`:   
    Where Prometheus writes its database. Defaults to data/.
- `--storage.tsdb.retention.time`:   
    When to remove old data. Defaults to 15d. Overrides storage.tsdb.retention if this flag is set to anything other than default.
- `--storage.tsdb.retention.size`:   
    The maximum number of bytes of storage blocks to retain. The oldest data will be removed first. Defaults to 0 or disabled. Units supported: B, KB, MB, GB, TB, PB, EB. Ex: "512MB". Based on powers-of-2, so 1KB is 1024B. Only the persistent blocks are deleted to honor this retention although WAL and m-mapped chunks are counted in the total size. So the minimum requirement for the disk is the peak space taken by the wal (the WAL and Checkpoint) and chunks\_head (m-mapped Head chunks) directory combined (peaks every 2 hours).

#### 新增服務(自動啟動)

##### RedHat 8

新增帳號與目錄

```bash
useradd -s /sbin/nologin --system prometheus
mkdir /etc/prometheus /data/prometheus
```

複製檔案

```bash
tar xvf prometheus-*.tar.gz
cd prometheus-*/
cp prometheus promtool /usr/local/bin/
cp -r prometheus.yml consoles/ console_libraries/ /etc/prometheus/

chown -R prometheus.prometheus /etc/prometheus
chmod -R 0755 /etc/prometheus
chown prometheus.prometheus /data/prometheus
```

新增設定檔: `/etc/systemd/system/prometheus.service`

```
[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecReload=/bin/kill -HUP $MAINPID
EnvironmentFile=/etc/sysconfig/prometheus
ExecStart=/usr/local/bin/prometheus $OPTIONS

[Install]
WantedBy=multi-user.target
```

新增設定檔: `/etc/sysconfig/prometheus`

```
OPTIONS="
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /data/prometheus/ \
  --storage.tsdb.retention.time=30d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
"
```

啟動服務

```bash
systemctl daemon-reload
systemctl start prometheus.service
systemctl enable prometheus.service
```

#### Monitor to Linux node

Linux Monitoring with Node Exporter

- [Node Exporter for Prometheus Dashboard EN v20201010](https://grafana.com/grafana/dashboards/11074)
- [Node Exporter Full](https://grafana.com/grafana/dashboards/1860)
- [Node Exporter Server Metrics](https://grafana.com/grafana/dashboards/405)
- [Node Exporter for Prometheus Dashboard based on 11074](https://grafana.com/grafana/dashboards/15172-node-exporter-for-prometheus-dashboard-based-on-11074/)
- [Node Exporter Quickstart and Dashboard](https://grafana.com/grafana/dashboards/13978-node-exporter-quickstart-and-dashboard/)
- \[Video\] [Setting up Prometheus and Grafana for monitoring your servers](https://www.youtube.com/watch?v=4WWW2ZLEg74)
- [MONITORING LINUX HOST METRICS WITH THE NODE EXPORTER](https://prometheus.io/docs/guides/node-exporter/)

##### On Linux target

Node Exporter Installation

Download: [Download | Prometheus](https://prometheus.io/download/#node_exporter)

```bash
tar xzf node_exporter-1.5.0.linux-amd64.tar.gz
mv node_exporter-1.5.0.linux-amd64 /opt/node_exporter
chown -R root.root /opt/node_exporter

cd /opt/node_exporter
./node_exporter
# Ctrl + C to exit
```

Set up node\_exporter as service

```bash
# Create a user
useradd -r -c "Node Exporter" -s /sbin/nologin node_exporter

# Create a service file
cat <<EOF>/etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter

[Service]
User=node_exporter
EnvironmentFile=/etc/sysconfig/node_exporter
ExecStart=/opt/node_exporter/node_exporter $OPTIONS

[Install]
WantedBy=multi-user.target
EOF
```

```bash
# Create the file /etc/sysconfig/node_exporter
echo '#OPTIONS=""' > /etc/sysconfig/node_exporter

# Start the node exporter
systemctl daemon-reload
systemctl start node_exporter.service
```

##### On Prometheus Server

prometheus.yml:

```yaml
scrape_configs:

  # Linux Nodes
  - job_name: linux

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 15s

    static_configs:
      - targets: ['linux-node-ip:9100']
```

#### Monitor to MySQL

- [MySQL Overview](https://grafana.com/grafana/dashboards/7362)
- [MySQL Server Exporter](https://github.com/prometheus/mysqld_exporter)
- \[Video\] [Setting up Prometheus and Grafana for monitoring your servers](https://www.youtube.com/watch?v=4WWW2ZLEg74)
- [Prometheus 監控Mysql伺服器及Grafana視覺化](https://iter01.com/553444.html)

#### Monitor to AIX

- [nimon working with Prometheus](https://www.ibm.com/support/pages/nimon-working-prometheus)

#### Monitor to RabbitMQ  


prometheus.yml:

```yaml
scrape_configs:

  # RabbitMQ Nodes
  - job_name: rabbitmq

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 15s

    static_configs:
      - targets: ['rmq01:15692', 'rmq02:15692', 'rmq03:15692']

```

#### Monitor to Containers

- [MONITORING DOCKER CONTAINER METRICS USING CADVISOR](https://prometheus.io/docs/guides/cadvisor/)
- \[Video\] [Node Application Monitoring with cAdvisor Prometheus and Grafana | part 1](https://www.youtube.com/watch?v=L17-EN4HcY0)
- [容器监控：cAdvisor](https://yunlzheng.gitbook.io/prometheus-book/part-ii-prometheus-jin-jie/exporter/commonly-eporter-usage/use-prometheus-monitor-container)