refactor(hosts): organize repository by server profiles (hosts/raspberrypi5, hosts/llm-node) and add host-aware installer

This commit is contained in:
Lars committed 2026-10-29 14:27:49 +01:00
1 parent c53fc0a939
commit c2740c9542
44 files changed
+528 -40

No files matched your search

+69 -28
View File
@@ -1,38 +1,79 @@
# Backup-Orchestrator v0.4 – Backup & Recovery Management
# Backup-Orchestrator v0.4 – Multi-Server Backup & Recovery Management
**Status: Version 0.4.0 (Multi-Host Backup & Application Recovery Management).**
Erweitert den auf `raspberrypi5` produktiv laufenden Backup-Orchestrator v0.3 um ein integriertes Recovery Management, kryptografisch geprüfte Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Konflikt-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für die Referenzanwendung **Shinkan** auf `llm-node` (`192.168.2.144`).
Erweitert den Backup-Orchestrator v0.3 um eine **server-spezifische Repository-Struktur (`hosts/<server>/`)**, ein integriertes Recovery Management mit kryptografisch geprüften Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für **Shinkan** auf `llm-node` (`192.168.2.144`).
---
## 1. Repository-Struktur & Enthaltene Dateien
## 1. Repository-Struktur (nach Servern & gemeinsamen Modulen getrennt)
- `orchestrator.py`: Phasenbasierter Backup- & Recovery-Controller (`v0.4.0`, Python 3.11+, nur Standardbibliothek).
- `targets.json`: Multi-Host- und Recovery-Target-Definitionen (`raspberrypi5`, `llm-node`, Targets `pi-prod`, `llm-prod`, `llm-restoretest`).
- `jobs.csv`: Produktive Backup-Job-Tabelle (17 Datenbank-Exporte in Phase 10, `nas-data` in Phase 20, `system-image` in Phase 30).
- `notifications.json.example`: Beispielkonfiguration für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore (enthält keine echten Secrets).
- `systemd/`:
- `backup-orchestrator-daily`: Täglicher Runner-Wrapper (ab v0.4 ohne fragiles `systemctl disable/enable raspiBackup.timer`).
- `backup-orchestrator-daily.service` & `backup-orchestrator-daily.timer`: Systemd-Units für den täglichen Lauf um `05:30` Uhr.
- `scripts/`:
- Alle 17 Datenbank-Dump-Wrapper (`db-bookstack.sh`, `db-buyingagent-dev.sh`, `db-buyingagent-prod.sh`, `db-kairo-dev.sh`, `db-kairo-prod.sh`, `db-kansho-dev.sh`, `db-kansho-prod.sh`, `db-karatewiki-dev.sh`, `db-mitai-dev.sh`, `db-mitai-prod.sh`, `db-netbox.sh`, `db-paperless.sh`, `db-shinkan-dev.sh`, `db-shinkan-prod.sh`, `db-tandoor.sh`, `db-tandoor.sh`, `db-tsumugu-dev.sh`, `db-tsumugu-prod.sh`) sowie die generischen Dump-Engines `db-postgres-docker.sh` und `db-mariadb-docker.sh`.
- `backup-nas-data.sh` & `nas-data-job.sh`: Versionierte rsync-Hardlink-Sicherung (`/media/lars/paperless` → `/backup/data_archiv/01..31` + `last`).
- `publish-backup-set.sh` & `shinkan-prod-manifest.sh`: Atomare Erzeugung und Publikation vollständiger Backup-Sets (`<dump>.sql`, `<dump>.sql.sha256`, `<dump>.sql.manifest.json`, `<dump>.sql.media.jsonl`).
- `scripts/shinkan/` (Shinkan Recovery Contract Adapter):
- `shinkan-backup-manifest`: Erzeugt SHA256, Manifest (`schemaVersion: 1` + Shinkan Contract) und inkrementelles Media-Inventar (`media.jsonl`).
- `shinkan-restore`: 9-stufiger atomarer Application-Restore über Staging-DB (`shinkan_restore_<ts>`), Single-Transaction-Import, Schema-Migrations-Gate, Plausibilitätsprüfung, `TRUNCATE sessions`, DB-Swap (`shinkan_before_<ts>`), `/api/health/ready`-Prüfung und automatischem Rollback bei Fehlern (Exit `5`).
- `shinkan-media-diff`: Abgleich zwischen PostgreSQL (`media_assets`, `exercise_media`), Dateisystem und `media.jsonl` (`missing`, `mismatch`, `added_after_backup`, `orphan`).
- `shinkan-restore-test` & `docker-compose.restoretest.yml`: Isolierter End-to-End-Restore-Test im Projekt `shinkan-restoretest` (Ports `127.0.0.1:18003` / `127.0.0.1:13003`, Read-Only-Media `:ro`, deaktivierte externe Integrationen & Cronjobs).
- `docs/ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md`: Vollständige Ist-Inventur, Gap-Analyse und Architektur-Spezifikation für v0.4.
- `tests/`:
- `test_orchestrator.py`: Unit-Tests für Backup-Phasen, Parallelität, Timeouts, Mount-Prüfung und Notifications.
- `test_recovery.py`: Unit- und Integrationstests für Manifest-Validierung, 4-Stufen-Modell, Schema-Kompatibilitätsprüfung (`EXACT_MATCH`, `FORWARD_MIGRATE`, `INCOMPATIBLE_ROLLBACK`), Sicherheits-Gates (`pi-prod` Sperre, `--confirm`), Dry-Run-Pläne und `shinkan-backup-manifest`.
```text
backup-orchestrator/
├── orchestrator.py # Universeller Backup- & Recovery-Controller (v0.4.0)
├── install.sh # Server-spezifischer Installer (--host raspberrypi5 | llm-node)
├── targets.json # Globale Target-Registry
├── scripts/
│ ├── common/ # Gemeinsame, serverunabhängige Backup-Engines
│ │ ├── db-postgres-docker.sh
│ │ ├── db-mariadb-docker.sh
│ │ └── publish-backup-set.sh
│ └── shinkan/ # Anwendungs-Adapter für Shinkan (Recovery Contract)
│ ├── shinkan-backup-manifest
│ ├── shinkan-restore
│ ├── shinkan-restore-test
│ ├── shinkan-media-diff
│ └── docker-compose.restoretest.yml
├── hosts/ # Server-spezifische Profile, Jobs, Wrapper & Systemd-Units
│ ├── raspberrypi5/ # Produktiver Backup-Host (Debian 12 ARM64, 192.168.2.49)
│ │ ├── host.json # Mounts (/backup), Pfade (/media/lars/paperless/db-backup)
│ │ ├── jobs.csv # 17 DB-Jobs (Phase 10), nas-data (Phase 20), system-image (Phase 30)
│ │ ├── targets.json # Host-Targets (pi-prod = source_only)
│ │ ├── notifications.json.example
│ │ ├── scripts/ # Alle 17 DB-Wrapper + NAS-Skripte für raspberrypi5
│ │ └── systemd/ # backup-orchestrator-daily + .service/.timer (05:30 Uhr)
│ └── llm-node/ # Neuer Anwendungs- & Recovery-Host (Ubuntu 24.04 x86_64, 192.168.2.144)
│ ├── host.json # Mounts (/mnt/nas_backup, /srv/shinkan/prod/media)
│ ├── jobs.csv # Backup-Jobs auf llm-node (shinkan-prod-db für Post-Cutover vorbereitet)
│ ├── targets.json # Targets (llm-prod, llm-restoretest, pi-prod = source_only)
│ ├── notifications.json.example
│ ├── scripts/ # Host-spezifische Wrapper für llm-node
│ └── systemd/ # backup-orchestrator-daily + .service/.timer (05:00 Uhr)
├── docs/
│ └── ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md
└── tests/
├── test_orchestrator.py
└── test_recovery.py
```
---
## 2. CLI-Befehle (v0.4.0)
## 2. Server-spezifische Installation (`install.sh`)
### A. Backup-Steuerung (unverändert kompatibel zu v0.3)
Auf jedem Server (`raspberrypi5` oder `llm-node`) erkennt `install.sh` automatisch den Hostnamen (`hostname -s`) oder akzeptiert `--host <servername>` und installiert genau die gemeinsame Engine + die zum jeweiligen Server gehörende Konfiguration nach `/etc/backup-orchestrator/`:
```bash
# Auf raspberrypi5:
sudo ./install.sh --host raspberrypi5
# Auf llm-node:
sudo ./install.sh --host llm-node
```
Optional:
- `--force-jobs`: Überschreibt eine bereits vorhandene `/etc/backup-orchestrator/jobs.csv` mit `hosts/<host>/jobs.csv`.
- `--install-systemd`: Installiert und lädt die zum Server gehörenden Systemd-Units (`backup-orchestrator-daily.service` & `.timer`).
Auch direkt aus dem Git-Arbeitsverzeichnis können Konfigurationen jedes Servers über `--host` geprüft werden:
```bash
python3 orchestrator.py --plan --host raspberrypi5
python3 orchestrator.py --plan --host llm-node
```
---
## 3. CLI-Befehle (v0.4.0)
### A. Backup-Steuerung (kompatibel zu v0.3)
```bash
backup-orchestrator --check
@@ -73,7 +114,7 @@ backup-orchestrator --restore \
---
## 3. Exit-Codes im Recovery-Modus
## 4. Exit-Codes im Recovery-Modus
| Exit-Code | Bedeutung |
|---|---|
@@ -85,9 +126,9 @@ backup-orchestrator --restore \
---
## 4. Tests ausführen
## 5. Tests ausführen
```bash
python3 -m unittest discover -s tests -v
```
Alle Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und berühren niemals produktive Datenbanken oder NAS-Mounts.
Alle 23 Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und prüfen zusätzlich die Vollständigkeit aller Server-Profile unter `hosts/*`.
+20
View File
@@ -0,0 +1,20 @@
{
"hostname": "llm-node",
"architecture": "x86_64",
"os": "Ubuntu 24.04 LTS",
"role": "application_and_recovery_host",
"nas_mount": "/mnt/nas_backup",
"nas_source": "192.168.2.63:/volume1/Backup_LLM",
"nas_fstype": "nfs",
"local_dump_dir": "/mnt/nas_backup/shinkan/prod/db/latest",
"archive_sets_dir": "/mnt/nas_backup/shinkan/prod/db",
"media_mounts": {
"shinkan-prod": "/srv/shinkan/prod/media"
},
"conflict_services": [],
"conflict_timers": [],
"cron_users": [
"root",
"llmadmin"
]
}
+2
View File
@@ -0,0 +1,2 @@
phase,id,script,critical,timeout_seconds,enabled
10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,900,false
1 phase id script critical timeout_seconds enabled
2 10 shinkan-prod-db /etc/backup-orchestrator/scripts/db-shinkan-prod.sh true 900 false
+10
View File
@@ -0,0 +1,10 @@
{
"timeout_seconds": 3,
"webhooks": [],
"kuma": {
"overall_push_url": "",
"job_push_urls": {
"shinkan-prod-db": ""
}
}
}
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
# Backup wrapper for Shinkan Production on llm-node (post-cutover)
set -Eeuo pipefail
DEST_ROOT="/mnt/nas_backup/shinkan/prod/db"
TS="$(date -u +%Y%m%dT%H%M%SZ)"
GEN_DIR="${DEST_ROOT}/generations/${TS}"
DUMP_FILE="${GEN_DIR}/shinkan-prod.sql"
install -d -m 0750 "${GEN_DIR}"
export DB_CONTAINER="shinkan-db-prod"
export DB_USER="shinkan"
export DB_NAME="shinkan"
export OUTPUT_PATH="${DUMP_FILE}"
/etc/backup-orchestrator/scripts/db-postgres-docker.sh
# Generate SHA256, Manifest (schemaVersion: 1), and Media Inventory (.media.jsonl)
/etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \
--dump "${DUMP_FILE}" \
--env prod \
--host "llm-node" \
--compose-dir "/home/llmadmin/docker/shinkan-prod" \
--media-dir "/srv/shinkan/prod/media" \
--db-container "shinkan-db-prod" \
--db-name "shinkan" \
--db-user "shinkan"
# Atomically update 'latest' symlink after complete backup set verification
ln -sfn "generations/${TS}" "${DEST_ROOT}/latest.tmp.$$"
mv -Tf "${DEST_ROOT}/latest.tmp.$$" "${DEST_ROOT}/latest"
echo "Published atomic Shinkan backup set on llm-node: ${GEN_DIR}"
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
# Standalone helper to generate manifest + media inventory for an existing dump on llm-node
set -Eeuo pipefail
DUMP_FILE="${1:-/mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql}"
exec /etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \
--dump "${DUMP_FILE}" \
--env prod \
--host "llm-node" \
--compose-dir "/home/llmadmin/docker/shinkan-prod" \
--media-dir "/srv/shinkan/prod/media" \
--db-container "shinkan-db-prod" \
--db-name "shinkan" \
--db-user "shinkan"
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -Eeuo pipefail
export BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES
exec /usr/local/sbin/backup-orchestrator --run
@@ -0,0 +1,11 @@
[Unit]
Description=Backup Orchestrator daily run on llm-node
After=network-online.target docker.service mnt-nas_backup.mount
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup-orchestrator-daily
User=root
Group=root
TimeoutStartSec=14400
@@ -0,0 +1,10 @@
[Unit]
Description=Daily timer for Backup Orchestrator on llm-node
[Timer]
OnCalendar=*-*-* 05:00:00
Persistent=true
RandomizedDelaySec=60
[Install]
WantedBy=timers.target
+74
View File
@@ -0,0 +1,74 @@
{
"schemaVersion": 1,
"host": "llm-node",
"hosts": {
"llm-node": {
"hostname": "llm-node",
"arch": "x86_64",
"nas_mount": "/mnt/nas_backup",
"nas_source": "192.168.2.63:/volume1/Backup_LLM",
"nas_fstype": "nfs",
"local_dump_dir": "/mnt/nas_backup/shinkan/prod/db/latest",
"archive_sets_dir": "/mnt/nas_backup/shinkan/prod/db"
}
},
"targets": {
"pi-prod": {
"application": "shinkan",
"environment": "prod",
"host": "raspberrypi5",
"role": "source_only",
"allow_restore": false,
"compose_dir": "/home/lars/docker/shinkan-prod",
"compose_files": ["docker-compose.yml"],
"env_file": "/home/lars/docker/shinkan-prod/.env",
"media_dir": "/shinkan-media/prod",
"db_container": "shinkan-db-prod",
"db_name": "shinkan",
"db_user": "shinkan",
"health_url": "http://localhost:8003/api/health/ready"
},
"llm-prod": {
"application": "shinkan",
"environment": "prod",
"host": "llm-node",
"role": "production_target",
"allow_restore": true,
"require_confirmation": "shinkan@llm-node",
"compose_dir": "/home/llmadmin/docker/shinkan-prod",
"compose_files": ["docker-compose.yml", "docker-compose.llm-node.yml"],
"env_file": "/etc/shinkan/prod.env",
"media_dir": "/srv/shinkan/prod/media",
"db_container": "shinkan-db-prod",
"db_name": "shinkan",
"db_user": "shinkan",
"health_url": "http://localhost:8003/api/health/ready",
"restore_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-restore",
"media_diff_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-media-diff"
},
"llm-restoretest": {
"application": "shinkan",
"environment": "restoretest",
"host": "llm-node",
"role": "isolated_test_target",
"allow_restore": true,
"compose_dir": "/home/llmadmin/docker/shinkan-prod",
"compose_files": [
"docker-compose.yml",
"docker-compose.llm-node.yml",
"/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml"
],
"project_name": "shinkan-restoretest",
"env_file": "/etc/shinkan/prod.env",
"media_dir": "/srv/shinkan/prod/media",
"media_readonly": true,
"db_container": "shinkan-db-restoretest",
"db_name": "shinkan_restoretest",
"db_user": "shinkan",
"health_url": "http://127.0.0.1:18003/api/health/ready",
"frontend_url": "http://127.0.0.1:13003",
"restore_test_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-restore-test",
"media_diff_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-media-diff"
}
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"hostname": "raspberrypi5",
"architecture": "arm64",
"os": "Debian GNU/Linux 12 (bookworm)",
"role": "backup_primary_and_legacy_app_host",
"nas_mount": "/backup",
"nas_source": "192.168.2.63:/volume1/Backup_Paperless",
"nas_fstype": "nfs4",
"local_dump_dir": "/media/lars/paperless/db-backup",
"archive_sets_dir": "/backup/data_archiv/last/paperless/db-backup",
"media_mounts": {
"shinkan-prod": "/shinkan-media/prod",
"shinkan-dev": "/shinkan-media/dev"
},
"conflict_services": [
"raspiBackup.service"
],
"conflict_timers": [
"paperless-data-backup.timer",
"borgmatic.timer"
],
"cron_users": [
"root",
"lars"
]
}
+20
View File
@@ -0,0 +1,20 @@
phase,id,script,critical,timeout_seconds,enabled
10,paperless-db,/etc/backup-orchestrator/scripts/db-paperless.sh,true,600,true
10,tandoor-db,/etc/backup-orchestrator/scripts/db-tandoor.sh,true,600,true
10,mitai-prod-db,/etc/backup-orchestrator/scripts/db-mitai-prod.sh,true,900,true
10,mitai-dev-db,/etc/backup-orchestrator/scripts/db-mitai-dev.sh,false,900,true
10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,600,true
10,shinkan-dev-db,/etc/backup-orchestrator/scripts/db-shinkan-dev.sh,false,600,true
20,nas-data,/etc/backup-orchestrator/scripts/nas-data-job.sh,true,10800,true
30,system-image,/etc/backup-orchestrator/scripts/system-image-job.sh,true,14400,false
10,kairo-prod-db,/etc/backup-orchestrator/scripts/db-kairo-prod.sh,true,900,true
10,kairo-dev-db,/etc/backup-orchestrator/scripts/db-kairo-dev.sh,false,900,true
10,tsumugu-prod-db,/etc/backup-orchestrator/scripts/db-tsumugu-prod.sh,true,900,true
10,tsumugu-dev-db,/etc/backup-orchestrator/scripts/db-tsumugu-dev.sh,false,900,true
10,kansho-prod-db,/etc/backup-orchestrator/scripts/db-kansho-prod.sh,true,900,true
10,kansho-dev-db,/etc/backup-orchestrator/scripts/db-kansho-dev.sh,false,900,true
10,buyingagent-prod-db,/etc/backup-orchestrator/scripts/db-buyingagent-prod.sh,true,900,true
10,buyingagent-dev-db,/etc/backup-orchestrator/scripts/db-buyingagent-dev.sh,false,900,true
10,netbox-db,/etc/backup-orchestrator/scripts/db-netbox.sh,true,900,true
10,bookstack-db,/etc/backup-orchestrator/scripts/db-bookstack.sh,true,900,true
10,karatewiki-dev-db,/etc/backup-orchestrator/scripts/db-karatewiki-dev.sh,false,900,true
1 phase id script critical timeout_seconds enabled
2 10 paperless-db /etc/backup-orchestrator/scripts/db-paperless.sh true 600 true
3 10 tandoor-db /etc/backup-orchestrator/scripts/db-tandoor.sh true 600 true
4 10 mitai-prod-db /etc/backup-orchestrator/scripts/db-mitai-prod.sh true 900 true
5 10 mitai-dev-db /etc/backup-orchestrator/scripts/db-mitai-dev.sh false 900 true
6 10 shinkan-prod-db /etc/backup-orchestrator/scripts/db-shinkan-prod.sh true 600 true
7 10 shinkan-dev-db /etc/backup-orchestrator/scripts/db-shinkan-dev.sh false 600 true
8 20 nas-data /etc/backup-orchestrator/scripts/nas-data-job.sh true 10800 true
9 30 system-image /etc/backup-orchestrator/scripts/system-image-job.sh true 14400 false
10 10 kairo-prod-db /etc/backup-orchestrator/scripts/db-kairo-prod.sh true 900 true
11 10 kairo-dev-db /etc/backup-orchestrator/scripts/db-kairo-dev.sh false 900 true
12 10 tsumugu-prod-db /etc/backup-orchestrator/scripts/db-tsumugu-prod.sh true 900 true
13 10 tsumugu-dev-db /etc/backup-orchestrator/scripts/db-tsumugu-dev.sh false 900 true
14 10 kansho-prod-db /etc/backup-orchestrator/scripts/db-kansho-prod.sh true 900 true
15 10 kansho-dev-db /etc/backup-orchestrator/scripts/db-kansho-dev.sh false 900 true
16 10 buyingagent-prod-db /etc/backup-orchestrator/scripts/db-buyingagent-prod.sh true 900 true
17 10 buyingagent-dev-db /etc/backup-orchestrator/scripts/db-buyingagent-dev.sh false 900 true
18 10 netbox-db /etc/backup-orchestrator/scripts/db-netbox.sh true 900 true
19 10 bookstack-db /etc/backup-orchestrator/scripts/db-bookstack.sh true 900 true
20 10 karatewiki-dev-db /etc/backup-orchestrator/scripts/db-karatewiki-dev.sh false 900 true
@@ -0,0 +1,10 @@
{
"timeout_seconds": 3,
"webhooks": [],
"kuma": {
"overall_push_url": "",
"job_push_urls": {
"nas-data": ""
}
}
}
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
+32
View File
@@ -0,0 +1,32 @@
{
"schemaVersion": 1,
"host": "raspberrypi5",
"hosts": {
"raspberrypi5": {
"hostname": "raspberrypi5",
"arch": "arm64",
"nas_mount": "/backup",
"nas_source": "192.168.2.63:/volume1/Backup_Paperless",
"nas_fstype": "nfs4",
"local_dump_dir": "/media/lars/paperless/db-backup",
"archive_sets_dir": "/backup/data_archiv/last/paperless/db-backup"
}
},
"targets": {
"pi-prod": {
"application": "shinkan",
"environment": "prod",
"host": "raspberrypi5",
"role": "source_only",
"allow_restore": false,
"compose_dir": "/home/lars/docker/shinkan-prod",
"compose_files": ["docker-compose.yml"],
"env_file": "/home/lars/docker/shinkan-prod/.env",
"media_dir": "/shinkan-media/prod",
"db_container": "shinkan-db-prod",
"db_name": "shinkan",
"db_user": "shinkan",
"health_url": "http://localhost:8003/api/health/ready"
}
}
}
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
# Host-aware installer for Backup-Orchestrator v0.4
# Usage: sudo ./install.sh [--host <raspberrypi5|llm-node>] [--force-jobs] [--install-systemd]
set -Eeuo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
HOST_NAME="$(hostname -s 2>/dev/null || hostname)"
FORCE_JOBS=0
INSTALL_SYSTEMD=0
while [[ $# -gt 0 ]]; do
case "$1" in
--host)
HOST_NAME="$2"
shift 2
;;
--force-jobs)
FORCE_JOBS=1
shift
;;
--install-systemd)
INSTALL_SYSTEMD=1
shift
;;
-h|--help)
echo "Usage: sudo $0 [--host <hostname>] [--force-jobs] [--install-systemd]"
exit 0
;;
*)
echo "ERROR: Unknown argument: $1" >&2
exit 2
;;
esac
done
HOST_DIR="${REPO_ROOT}/hosts/${HOST_NAME}"
if [[ ! -d "${HOST_DIR}" ]]; then
echo "ERROR: Unknown host configuration '${HOST_NAME}'. Expected directory: ${HOST_DIR}" >&2
echo "Available hosts:" >&2
ls -1 "${REPO_ROOT}/hosts" >&2
exit 2
fi
echo "=== Installing Backup-Orchestrator v0.4 for host '${HOST_NAME}' ==="
# 1. Backup existing binary if present
if [[ -f /usr/local/sbin/backup-orchestrator ]]; then
cp -a /usr/local/sbin/backup-orchestrator "/usr/local/sbin/backup-orchestrator.bak.$(date +%Y%m%d%H%M%S)"
fi
install -m 0755 "${REPO_ROOT}/orchestrator.py" /usr/local/sbin/backup-orchestrator
# 2. Prepare /etc/backup-orchestrator directories
install -d -m 0750 /etc/backup-orchestrator
install -d -m 0750 /etc/backup-orchestrator/scripts
install -d -m 0750 /etc/backup-orchestrator/scripts/shinkan
install -d -m 0700 /etc/backup-orchestrator/credentials
install -d -m 0750 /var/log/backup-orchestrator
# 3. Install host-specific metadata & target registry
install -m 0640 "${HOST_DIR}/host.json" /etc/backup-orchestrator/host.json
install -m 0640 "${HOST_DIR}/targets.json" /etc/backup-orchestrator/targets.json
if [[ ! -f /etc/backup-orchestrator/jobs.csv || "${FORCE_JOBS}" -eq 1 ]]; then
install -m 0640 "${HOST_DIR}/jobs.csv" /etc/backup-orchestrator/jobs.csv
echo "Installed /etc/backup-orchestrator/jobs.csv from hosts/${HOST_NAME}/jobs.csv"
else
echo "Keeping existing /etc/backup-orchestrator/jobs.csv (use --force-jobs to overwrite)"
fi
if [[ ! -f /etc/backup-orchestrator/notifications.json ]]; then
install -m 0600 "${HOST_DIR}/notifications.json.example" /etc/backup-orchestrator/notifications.json
echo "Created initial /etc/backup-orchestrator/notifications.json (mode 0600)"
fi
# 4. Install shared common scripts + host-specific scripts + Shinkan adapters
for s in "${REPO_ROOT}/scripts/common/"*.sh; do
[[ -f "$s" ]] && install -m 0750 "$s" "/etc/backup-orchestrator/scripts/$(basename "$s")"
done
for s in "${HOST_DIR}/scripts/"*.sh; do
[[ -f "$s" ]] && install -m 0750 "$s" "/etc/backup-orchestrator/scripts/$(basename "$s")"
done
for s in "${REPO_ROOT}/scripts/shinkan/"*; do
if [[ -f "$s" ]]; then
case "$s" in
*.yml|*.yaml|*.json)
install -m 0644 "$s" "/etc/backup-orchestrator/scripts/shinkan/$(basename "$s")"
;;
*)
install -m 0755 "$s" "/etc/backup-orchestrator/scripts/shinkan/$(basename "$s")"
;;
esac
fi
done
# 5. Install daily wrapper & optional systemd units
if [[ -f "${HOST_DIR}/systemd/backup-orchestrator-daily" ]]; then
install -m 0755 "${HOST_DIR}/systemd/backup-orchestrator-daily" /usr/local/sbin/backup-orchestrator-daily
fi
if [[ "${INSTALL_SYSTEMD}" -eq 1 ]]; then
install -m 0644 "${HOST_DIR}/systemd/backup-orchestrator-daily.service" /etc/systemd/system/backup-orchestrator-daily.service
install -m 0644 "${HOST_DIR}/systemd/backup-orchestrator-daily.timer" /etc/systemd/system/backup-orchestrator-daily.timer
systemctl daemon-reload
echo "Installed systemd service and timer for ${HOST_NAME}."
fi
echo "=== Verification (--check) ==="
/usr/local/sbin/backup-orchestrator --check
+52 -12
View File
@@ -66,8 +66,15 @@ def lock_exclusive(fp):
fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
def load_host_config(path=DEFAULT_HOST_CONFIG):
def resolve_repo_host_dir(host_name=None):
host = host_name or socket.gethostname().split('.')[0]
candidate = Path(__file__).resolve().parent / 'hosts' / host
return candidate if candidate.is_dir() else None
def load_host_config(path=DEFAULT_HOST_CONFIG, host_name=None):
default = {
'hostname': host_name or socket.gethostname(),
'nas_mount': NAS_MOUNT,
'nas_source': NAS_SOURCE,
'nas_fstype': 'nfs4',
@@ -77,16 +84,37 @@ def load_host_config(path=DEFAULT_HOST_CONFIG):
'conflict_timers': ['paperless-data-backup.timer', 'borgmatic.timer'],
'cron_users': ['root', 'lars'],
}
if path and Path(path).is_file():
with Path(path).open(encoding='utf-8') as fp:
data = json.load(fp)
if isinstance(data, dict):
default.update(data)
candidates = []
if host_name:
host_dir = resolve_repo_host_dir(host_name)
if host_dir:
candidates.append(host_dir / 'host.json')
if path:
candidates.append(Path(path))
host_dir = resolve_repo_host_dir()
if host_dir:
candidates.append(host_dir / 'host.json')
for candidate in candidates:
if candidate.is_file():
with candidate.open(encoding='utf-8') as fp:
data = json.load(fp)
if isinstance(data, dict):
default.update(data)
break
return default
def load_targets_config(path=DEFAULT_TARGETS_CONFIG):
candidates = [Path(path)] if path else []
def load_targets_config(path=DEFAULT_TARGETS_CONFIG, host_name=None):
candidates = []
if host_name:
host_dir = resolve_repo_host_dir(host_name)
if host_dir:
candidates.append(host_dir / 'targets.json')
if path:
candidates.append(Path(path))
host_dir = resolve_repo_host_dir()
if host_dir:
candidates.append(host_dir / 'targets.json')
candidates.append(Path(__file__).resolve().parent / 'targets.json')
for candidate in candidates:
if candidate.is_file():
@@ -1396,6 +1424,7 @@ def main():
mode.add_argument('--restore-test', action='store_true', help='Execute isolated disposable restore test')
parser.add_argument('--json', action='store_true', help='Machine-readable JSON output')
parser.add_argument('--host', type=str, default=None, help='Select server profile from hosts/<host> (e.g. raspberrypi5, llm-node)')
parser.add_argument('--config', type=Path, default=DEFAULT_CONFIG)
parser.add_argument('--notify-config', type=Path, default=DEFAULT_NOTIFY_CONFIG)
parser.add_argument('--targets-config', type=Path, default=DEFAULT_TARGETS_CONFIG)
@@ -1419,6 +1448,17 @@ def main():
parser.add_argument('--keep', action='store_true', help='Keep isolated test environment running after --restore-test')
args = parser.parse_args()
if args.host:
host_dir = resolve_repo_host_dir(args.host)
if not host_dir:
parser.error(f'Unknown host profile: {args.host!r}')
if args.config == DEFAULT_CONFIG:
args.config = host_dir / 'jobs.csv'
if args.notify_config == DEFAULT_NOTIFY_CONFIG and (host_dir / 'notifications.json.example').is_file():
args.notify_config = host_dir / 'notifications.json.example'
if args.targets_config == DEFAULT_TARGETS_CONFIG and (host_dir / 'targets.json').is_file():
args.targets_config = host_dir / 'targets.json'
if not 1 <= args.max_parallel <= 16:
parser.error('--max-parallel must be 1..16')
json_capable = (
@@ -1434,7 +1474,7 @@ def main():
return show_status(args.log_root, as_json=args.json)
if args.list_recovery_points:
host_cfg = load_host_config()
host_cfg = load_host_config(host_name=args.host)
search_dirs = args.search_dirs or [
host_cfg['archive_sets_dir'],
host_cfg['local_dump_dir'],
@@ -1472,7 +1512,7 @@ def main():
if args.recovery_plan:
if not args.point or not args.target:
parser.error('--point and --target are required with --recovery-plan')
targets_cfg = load_targets_config(args.targets_config)
targets_cfg = load_targets_config(args.targets_config, host_name=args.host)
plan_doc = generate_recovery_plan(args.point, args.target, targets_cfg, sessions_mode=args.sessions)
if args.json:
print(json.dumps(plan_doc, ensure_ascii=False, indent=2))
@@ -1492,11 +1532,11 @@ def main():
return plan_doc['compatibility']['exit_code']
if args.restore:
targets_cfg = load_targets_config(args.targets_config)
targets_cfg = load_targets_config(args.targets_config, host_name=args.host)
return execute_recovery_command('restore', args, targets_cfg)
if args.restore_test:
targets_cfg = load_targets_config(args.targets_config)
targets_cfg = load_targets_config(args.targets_config, host_name=args.host)
return execute_recovery_command('restore-test', args, targets_cfg)
jobs = load_jobs(args.config)
File renamed without changes.
File renamed without changes.
File renamed without changes.
+29
View File
@@ -239,6 +239,35 @@ class RecoveryPointAndManifestTests(unittest.TestCase):
self.assertTrue((out_dir / 'shinkan-prod.sql.sha256').is_file())
self.assertTrue((out_dir / 'shinkan-prod.sql.media.jsonl').is_file())
def test_host_profiles_raspberrypi5_and_llm_node(self):
repo_root = Path(__file__).resolve().parents[1]
for host_name, expected_jobs in (('raspberrypi5', 19), ('llm-node', 1)):
host_dir = app.resolve_repo_host_dir(host_name)
self.assertIsNotNone(host_dir, f'Missing host dir for {host_name}')
host_cfg = app.load_host_config(host_name=host_name)
self.assertEqual(host_cfg['hostname'], host_name)
jobs = app.load_jobs(host_dir / 'jobs.csv')
self.assertEqual(len(jobs), expected_jobs)
notify = app.load_notifications(
host_dir / 'notifications.json.example',
known_ids=[j['id'] for j in jobs],
)
self.assertIn('kuma', notify)
targets = app.load_targets_config(host_name=host_name)
self.assertEqual(targets['host'], host_name)
self.assertIn('pi-prod', targets['targets'])
# Verify every job script exists in either hosts/<host>/scripts/ or scripts/common/
for job in jobs:
script_name = Path(job['script']).name
if job['id'] == 'system-image':
continue
exists_in_host = (host_dir / 'scripts' / script_name).is_file()
exists_in_common = (repo_root / 'scripts' / 'common' / script_name).is_file()
self.assertTrue(
exists_in_host or exists_in_common,
f"Missing script {script_name} for job {job['id']} on host {host_name}",
)
if __name__ == '__main__':
unittest.main()