Projects/Dual-Channel Alert System (Email + ntfy)
Monitoring

Dual-Channel Alert System (Email + ntfy)

Automatically sends email and ntfy push notifications for disk, memory, temperature, and service anomalies, with the two channels serving as backups for each other.

中文版 ↗← Back

What Is This

This is a "dual-channel alerting system" in my Homelab: whenever there is an issue with disks, memory, CPU temperature, systemd services, container health, mounted disks, node online status, or website/tunnel availability, my phone's ntfy immediately rings, and at the same time an email is sent to my self-hosted mailbox xiejun@xiejun-server.top. The two channels back each other up, so if either one goes down, the other can still alert me.

Why I Built It

Previously I relied only on watching the monitoring dashboard page, so problems were often seen only when I actively opened it. Once a photo disk SMART alert and once an unhealthy container were both discovered after the fact. So what I wanted was "know without actively looking"—the machines come to me. I chose ntfy because it is free on mobile, has no ads, and subscribing to a topic is all it takes; I chose email because it is sufficiently independent and does not rely on the push path of the same machine.

How Responsibilities Are Divided

Alerting is split into two layers that are independent of each other:

  • Gatus (running on .223) handles the external perspective: 16 probes, website/service URLs, public availability, two Sakura tunnels, photo disk SMART, Windows host SSH, and three machines online.
  • Monitoring station (notify.py on .164) handles the internal perspective: node offline, disk ≥88%/94%, memory ≥92%/96%, CPU temperature ≥85/92°C, service startup failure, container unhealthy, and mounted disk failure.

Both layers have debouncing and cooldown: push only after 2 consecutive failures, send a "recovered" message after recovery, and repeat the same issue only once every 6 hours.

Current Status

Running. Both the Gatus→ntfy and monitoring station→ntfy links have been verified with real triggers, and the state machine self-check passed; the email side uses self-hosted docker-mailserver + Brevo free relay, and mail-tester shows DMARC/DKIM/SPF all pass.

Known Boundaries

ntfy uses the public ntfy.sh, and the topic is the password; if .164 goes down entirely, Gatus on .223 provides fallback alerting.

Features

  • Gatus 侧 16 项探测覆盖网站、公网可用性、隧道、SMART、SSH 与机器在线
  • 监控台侧覆盖节点离线、磁盘/内存/温度阈值、服务与容器健康、挂载盘失效
  • 连续失败 2 次才推送，恢复后补一条「已恢复」，避免抖动刷屏
  • 同一问题 6 小时冷却，只重复提醒一次
  • ntfy 手机推送 + 自建邮箱 SMTP 发信，双通道互为备份
  • 通道可切换：notify.json 里换成企业微信或钉钉机器人，30 秒内生效

Stack: Python · ntfy · Gatus · Docker · docker-mailserver · Brevo SMTP 中继 · systemd

I. Mobile subscription (1 minute)

  1. Install the ntfy app (Android app store / F-Droid, iOS App Store).
  2. Open the app → + in the top-right corner → Subscribe to topic → enter the topic string (a 22-byte random string, equivalent to a password, do not share it).
  3. No further action is needed after completion. Right after subscribing, you may see a few historical test messages from within the last 12 hours.

II. Deploy notify.py on the monitoring dashboard side

  1. Place notify.py (pure standard-library implementation, detection + push) in /opt/dashboard/, create notify.json in the same directory, and set permissions to 600:
   chmod 600 /opt/dashboard/notify.json
  1. In notify.json, write the topic, thresholds, and channels; the default is {"type": "ntfy"}. To switch to WeCom, change channels to webhook form; it takes effect automatically within 30 seconds without restart.
  2. Add two lines to server.py to integrate: import notify and notify.start(stats). Back up server.py before modifying it.
  3. Self-test the state machine (including the "recovery" path assertion):
   python3 /opt/dashboard/notify.py --selftest
  1. Verification: temporarily lower a threshold below the current actual value to trigger one cycle, confirm you receive both the alert and the "recovered" messages, then change it back to the official threshold and check that notify-state.json is {"problems": {}}.

III. Configure the Gatus side

  1. Edit /opt/gatus/config/config.yaml; back it up first.
  2. Add an alerting.ntfy section and fill in the topic and push URL.
  3. Add alerts: [- type: ntfy] to each endpoint that needs alerting (currently 16 items).
  4. Restart Gatus, then temporarily add an endpoint that is guaranteed to fail, confirm the phone receives the alert, and then delete it.

IV. Connect the email channel

  1. Create a sending account on the VPS using docker-mailserver (docker exec mailserver setup email add <addr> <password>).
  2. In environments where outbound port 25 is blocked, you must use an ESP relay: configure RELAY_HOST / RELAY_PORT=587 / RELAY_USER / RELAY_PASSWORD in mailserver.env; after changes, run docker compose restart mailserver.
  3. Complete domain authentication on the ESP side (the root domain must match the domain of the From address); otherwise the signature will not match.
  4. Send a test email with mail-tester and confirm spf=pass, dkim=pass, and dmarc=pass.

V. Routine maintenance

Adjust thresholds according to actual machine load; cooldown time is controlled by cooldown_sec; whenever you add a machine or a service, remember to add the corresponding item on both the Gatus and monitoring dashboard sides.