Projects/Short-term Quantitative Research Platform
Tools

Short-term Quantitative Research Platform

An A-share short-term quantitative research system migrated from an old machine to .223 and is publicly accessible over the internet.

中文版 ↗← Back

What This Is

Short-term quantitative research platform (AQF Research) is a short-term A-share strategy research system I wrote myself: implemented with the Python standard library, with a built-in backtesting pipeline, running in a Docker container, with a web frontend for viewing results and checking health status. It originally ran on a small J1900 / 3.8G memory machine (.164). On 2026-10-16, four consecutive OOMs brought the entire machine down for about 1 hour, and the load average spiked to 54.41. The website, photo album, music, and monitoring dashboard were all affected. The culprit was the backtesting pipeline container having no memory limit, with each instance consuming 0.8–1.4GB.

What Was Solved

So the entire project was moved to .223, an i5-8279U / 16G machine: the program 676MB, backtesting data 1.3GB, the systemd web service, and the public entry point were all migrated, with paths unchanged (/tmp/sim symlinked to /opt/aqf-sim), and not a single line of the scripts was changed. The migration was verified in two rounds: per-file checksum comparison showed 0 differences, all 173 tests passed, and a real backtest ran successfully through the first stage (memory peak 942MB, exactly the step that was killed by OOM on the old machine). The public entry https://xiejun-server.top/quant/ remains unchanged, imperceptible to users.

Now

The service is running on .223, with auto-start on boot, MemoryMax=2G, and automatic restart on failure. The programs and data on .164 are retained as-is as a cold backup, and a rollback is possible at any time.

Features

  • 程序与 1.3GB 回测数据从 .164 整体迁移到 .223，路径不变、脚本零改动
  • 迁移前后两遍逐文件校验和比对，差异 0 条（含软链与 SQLite 完整性检查）
  • 173 个自带测试在新机器全部通过，耗时从 23.9s 降到约 10.5s
  • 用真实回测流水线验证：第一阶段退出码 0，内存峰值 942MB
  • 网页服务交给 systemd 托管，开机自启、MemoryMax=2G、崩溃 5 秒内自动拉起
  • 公网反代与监控探测（Gatus、自建监控台）切到新机器，入口 URL 不变

Stack: Python · Docker · systemd · SQLite · Nginx · Gatus · rsync

Deployment and Usage

All following operations are performed on .223.

1. Confirm service status

systemctl status aqf-research

This is a web service. It has been enabled to start at boot, with MemoryMax=2G and Restart=always configured. Page addresses: intranet http://192.168.50.223:5555/, public network https://xiejun-server.top/quant/.

2. Run the backtesting pipeline

cd /tmp/sim && bash overnight.sh

The path is unchanged: /tmp/sim is a symlink to /opt/aqf-sim, automatically recreated at boot by /etc/tmpfiles.d/aqf-sim.conf. The hardcoded /tmp/sim in scripts does not need to be changed.

3. Manually run a single container

docker run --rm -v /opt/aqf-research/current:/app:ro -v /tmp/sim:/sim -w /app \
  -e AQF_DATA_DIR=/sim/d_xxx \
  docker.m.daocloud.io/library/python:3.12-alpine \
  python -u /sim/xxx.py

The image has already been pulled locally in advance. Be sure to add a memory limit to the container; do not repeat the OOM incident.

4. Backfill data from the old machine

If .164 produces new files again, preview first and then perform the actual sync (one-way):

bash /opt/aqf-sim/同步164新文件到223.sh
bash /opt/aqf-sim/同步164新文件到223.sh --go

5. Roll back to .164

ssh nas 'systemctl enable --now aqf-research'

Then change the 4 proxy_pass entries in NPM (the /quant/ and /api/ in 10-lan-ssl.conf, and the same two in 1.conf) back to 192.168.50.164:5555, update the URLs in the Gatus configuration and the monitoring console's collect.py accordingly, and reload NPM and Gatus. The backup before the changes is at .164:/root/quant-proxy-bak-20261016-180803/.

6. Migration verification approach (reusable)

  • rsync -rcn --itemize-changes compares content rather than size; symlinks are compared separately
  • For SQLite, use integrity_check, and verify row counts in key tables
  • Run the program's built-in tests
  • Actually run a business pipeline once and confirm the exit code
  • Stop the old service to freeze the data, then compare a second time to avoid divergence between the two sides