Projects/Server Lag/Freeze Incident Retrospective
Monitoring

Server Lag/Freeze Incident Retrospective

A complete retrospective of two real incidents covering symptoms, evidence collection, root causes, fixes, and preventive measures.

中文版 ↗← Back

What This Is

This is my complete postmortem archive of two real incidents in my Homelab: the total system freeze on 2026-10-16 and the server sluggishness on 2026-10-17. Both incidents occurred on the same .164 small host running a pile of services. It uses a Celeron J1900 and has only 3.8GB of RAM, while simultaneously hosting a reverse proxy, whole-home DNS, a photo album, intranet penetration, memos, filebrowser, and other services.

Why I Wrote This

When the failures occurred, users only noticed that the website would not open. The real cause was neither a network outage nor a failed disk. Instead, memory was exhausted by containers with no limits, and the entire machine was thrashing on swap on a mechanical disk. I wanted to preserve the full chain of “symptom → evidence collection → root cause → fix → prevention”: load 79, iowait 90%, vmstat b=73, and sda 90% busy yet only 5.42MB/s. These numbers are more convincing than conclusions.

What I Did

I added triple limits on memory, CPU, and IO plus a watchdog to PhotoPrism; added a rule to automatically fill in the missing memory limit for the quantitative backtesting container; changed swap to a tiered setup of zram → SSD → NAS mechanical disk; and changed the AdGuard health check so that it no longer blindly restarts containers when the system is sluggish.

Current Status

Both incidents have been handled and verified, and the project is marked as archived. The only root-cause fix not done is adding a 4GB DDR3L memory stick; the motherboard still has one empty slot.

Features

  • 完整记录两次事故的证据链：load、iowait、swap、阻塞进程数逐项留档
  • PhotoPrism 容器限内存 1200m / 1.5 核 / IO 64MB/s，超限只杀自己不拖整机
  • photoprism-guard 看门狗每 20 秒巡检，整机濒死时自动停相册保网站
  • sim-memcap 给没有内存上限的回测容器自动补 memory limit
  • swap 三级分层：zram > SSD > NAS 机械盘，换页不再抖动 HDD
  • AdGuard 健康检查 v2：高负载时只记日志，连续两轮失败才重启

Stack: Docker · cgroup v2 · systemd timer · zram / swap · PSI · PhotoPrism · AdGuard Home · Shell

Steps to reproduce this protection setup

Prerequisite: Linux + Docker + cgroup v2 (stat -fc %T /sys/fs/cgroup returns cgroup2fs).

1. Collect evidence first, don't rush to restart

When things stall, capture the scene immediately; restarting destroys the evidence:

uptime; vmstat 1 5; free -m; cat /proc/pressure/io
iostat -x 1 3
docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}'

Focus on three things: whether load and iowait are high, the b column in vmstat (number of uninterruptible IO processes), and high disk busy with extremely low throughput—that is a typical sign of swap thrashing.

2. Add hard limits to resource-heavy containers

Edit its compose.yaml (back up the original file first):

services:
  photoprism:
    mem_limit: 1200m
    memswap_limit: 1600m
    cpus: 1.5
    cpu_shares: 128
    pids_limit: 512
    logging:
      driver: json-file
      options: { max-size: 10m, max-file: "3" }

Rebuild with docker compose up -d <service>; this affects only that service.

3. Limit disk IO with cgroup v2

echo '8:0 rbps=67108864 wbps=33554432 riops=80 wiops=60' > /sys/fs/cgroup/system.slice/docker-<ID>.scope/io.max
echo 'default 20' > /sys/fs/cgroup/system.slice/docker-<ID>.scope/io.weight
echo 1000M > /sys/fs/cgroup/system.slice/docker-<ID>.scope/memory.high

The cgroup is reset after the container is rebuilt, so write a script together with a systemd timer to reapply it every 20 seconds.

4. Add a watchdog

In the same timer, check: if the full avg60 of /proc/pressure/io exceeds 50% and the 1-minute load exceeds 25, and this happens twice consecutively, docker stop the most resource-hungry container and write the status to a file under /run/. Query logs with journalctl -t <tag>.

5. Tier swap; don't put it on a mechanical drive

fallocate -l 4G /swapfile && chmod 600 /swapfile
mkswap /swapfile && swapon -p 50 /swapfile
echo '/swapfile none swap pri=50 0 0' >> /etc/fstab
systemctl daemon-reload

Priorities must ensure zram is highest, SSD in the middle, and the mechanical drive only as a fallback.

6. Raise the CPU weight of key containers

docker update --cpu-shares 2048 npm adguardhome natfrp-service

The rules are saved in the container configuration and still take effect after restart.

7. Verify

Confirm the limit values in docker inspect, whether io.max was written, and whether the file write speed inside the container is actually throttled; finally, curl the key services one by one to confirm the return codes are normal.

Keep a backup with a date suffix every time you change the configuration; to roll back, simply overwrite it back and rebuild.