‹ 返回《综合实战场景》目录

🧩 第05章 · 搭建监控体系

Prometheus + 告警 + 报告

搭建监控体系

服务器跑起来后，下一步是知道它健不健康。本章用 Prometheus（127.0.0.1:9090）+ node-exporter（127.0.0.1:9100）采集主机指标，配告警规则，再用 cron 定期出监控报告。

本章目标：采集 CPU/内存/磁盘/网络等核心指标，能自动告警，并定时把健康报告发到邮箱。

1. 架构与端口

组件 端口 作用
node-exporter 9100 暴露主机指标（CPU/内存/磁盘/网络）
Prometheus 9090 抓取并存储指标、执行告警规则
alertmanager 9093 接收告警并推送（邮件/Webhook）
Grafana（可选） 3000 可视化仪表盘 
# 基础工具
sudo apt install -y curl wget

2. 安装 node-exporter

让 node-exporter 走 systemd 托管并只监听本机。

# 下载最新版（以官方 release 为准）
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
sudo tar -xzf node_exporter-*.tar.gz -C /usr/local/bin --strip-components=1
sudo mv /usr/local/bin/node_exporter /usr/local/bin/node_exporter 2>/dev/null

# /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus node exporter
After=network.target

[Service]
Type=simple
User=nobody
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now node-exporter
systemctl status node-exporter --no-pager | grep active
curl -s http://127.0.0.1:9100/metrics | grep -E '^node_cpu_seconds_total|^node_memory_MemTotal_bytes' | head -3

🔒 安全：node-exporter 一定要 --web.listen-address=127.0.0.1:9100，否则任何人都能抓你的主机指标。公网直接暴露 9100 是常见的安全事故点。

3. 安装 Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz
sudo tar -xzf prometheus-*.tar.gz -C /opt --strip-components=1
sudo mkdir -p /etc/prometheus

配置抓取目标：

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

rule_files:
  - /etc/prometheus/rules.yml

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['127.0.0.1:9100']

sudo cp /opt/prometheus-*.linux-amd64/prometheus /opt/prometheus-*.linux-amd64/promtool /usr/local/bin/
sudo cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak

# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/srv/prometheus
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now prometheus
curl -s http://127.0.0.1:9090/-/ready && echo
curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | head -c 300

⚠️ 易错：--storage.tsdb.path 需要可写且空间足够。别用一个权限不对的目录（比如 nobody 无写权限）导致 Prometheus 启动崩溃。

4. 配置告警规则

# /etc/prometheus/rules.yml
groups:
  - name: host-alerts
    rules:
      - alert: HighCPU
        expr: 100 - (avg(url:node_cpu_seconds_total{mode="idle"}) by (instance) * 100) > 90
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "CPU 高 ({{ $value }}%)"

      - alert: DiskFull
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "根分区可用空间不足 10%"

      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "主机不可达"

sudo promtool check rules /etc/prometheus/rules.yml
sudo systemctl restart prometheus
sleep 10
curl -s 'http://127.0.0.1:9090/api/v1/query?query=ALERTS' | python3 -m json.tool | head -30

⚠️ 易错：expr 写错（比如字段名不存在）会导致该告警永不触发或一直 pending。每次改规则务必 promtool check rules 校验，再 reload。

5. 安装 alertmanager 并接入邮件（可选）

wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
sudo tar -xzf alertmanager-*.tar.gz -C /opt --strip-components=1

# /etc/alertmanager/alertmanager.yml
route:
  receiver: mail
receivers:
  - name: mail
    email_configs:
      - to: 'admin@bangdream.xyz'
        from: 'noreply@bangdream.xyz'
        smarthost: 'smtp.exmail.qq.com:465'
        auth_username: 'noreply@bangdream.xyz'
        auth_password: '邮箱授权码'
        require_tls: true

在 prometheus.yml 里加上 alerting 段指向 alertmanager：

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['127.0.0.1:9093']

sudo systemctl restart prometheus
curl -s http://127.0.0.1:9093/-/ready && echo

6. cron 定时监控报告

每天生成一份汇总并写入日志（也可配 mail）。

sudo tee /usr/local/bin/health-report.sh >/dev/null <<'EOF'
#!/bin/bash
OUT=/srv/reports/health-$(date +%F).txt
mkdir -p /srv/reports
{
  echo "=== $(date) 主机健康摘要 ==="
  echo "-- CPU 平均负载 --"; uptime
  echo "-- 内存 --"; free -h
  echo "-- 磁盘 --"; df -h / | tail -1
  echo "-- 关键服务 --"
  systemctl is-active nginx prometheus node-exporter bangdream-site hbbs hbbr 2>/dev/null
} > "$OUT"
cat "$OUT"
EOF
sudo chmod +x /usr/local/bin/health-report.sh

crontab -e

# 每天 08:30 生成健康报告
30 8 * * * /usr/local/bin/health-report.sh >> /var/log/health-report.log 2>&1

7. 完成清单

# 验证点 命令 预期
1 node-exporter curl -s 127.0.0.1:9100/metrics 返回指标
2 Prometheus curl -s :9090/-/ready Prometheus is Ready
3 目标已抓取 浏览器 9090 → Status/Targets node up
4 告警规则 curl :9090/api/v1/query?query=ALERTS 有规则
5 定时报告 ls /srv/reports 当天 .txt
6 端口 9090 curl -sI http://127.0.0.1:9090/graph 200

一条命令总验证：

curl -s http://127.0.0.1:9090/-/ready && curl -s http://127.0.0.1:9100/metrics | grep -c '^node_' \
  && /usr/local/bin/health-report.sh | tail -3 \
  && curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | grep -o '"value":\[[^]]*\]'

监控体系就绪：Prometheus 持续采集指标、能自动告警、每天凌晨生成报告。下一步是给数据配一套可靠的备份与恢复演练，保证出问题时能真正救回来。

‹ 搭建远程协助服务器备份与恢复演练 ›