diff --git a/README.md b/README.md index 90e5178..1b75702 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,79 @@ -# Backup-Orchestrator v0.4 – Backup & Recovery Management +# Backup-Orchestrator v0.4 – Multi-Server Backup & Recovery Management **Status: Version 0.4.0 (Multi-Host Backup & Application Recovery Management).** -Erweitert den auf `raspberrypi5` produktiv laufenden Backup-Orchestrator v0.3 um ein integriertes Recovery Management, kryptografisch geprüfte Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Konflikt-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für die Referenzanwendung **Shinkan** auf `llm-node` (`192.168.2.144`). +Erweitert den Backup-Orchestrator v0.3 um eine **server-spezifische Repository-Struktur (`hosts//`)**, ein integriertes Recovery Management mit kryptografisch geprüften Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für **Shinkan** auf `llm-node` (`192.168.2.144`). --- -## 1. Repository-Struktur & Enthaltene Dateien +## 1. Repository-Struktur (nach Servern & gemeinsamen Modulen getrennt) -- `orchestrator.py`: Phasenbasierter Backup- & Recovery-Controller (`v0.4.0`, Python 3.11+, nur Standardbibliothek). -- `targets.json`: Multi-Host- und Recovery-Target-Definitionen (`raspberrypi5`, `llm-node`, Targets `pi-prod`, `llm-prod`, `llm-restoretest`). -- `jobs.csv`: Produktive Backup-Job-Tabelle (17 Datenbank-Exporte in Phase 10, `nas-data` in Phase 20, `system-image` in Phase 30). -- `notifications.json.example`: Beispielkonfiguration für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore (enthält keine echten Secrets). -- `systemd/`: - - `backup-orchestrator-daily`: Täglicher Runner-Wrapper (ab v0.4 ohne fragiles `systemctl disable/enable raspiBackup.timer`). - - `backup-orchestrator-daily.service` & `backup-orchestrator-daily.timer`: Systemd-Units für den täglichen Lauf um `05:30` Uhr. -- `scripts/`: - - Alle 17 Datenbank-Dump-Wrapper (`db-bookstack.sh`, `db-buyingagent-dev.sh`, `db-buyingagent-prod.sh`, `db-kairo-dev.sh`, `db-kairo-prod.sh`, `db-kansho-dev.sh`, `db-kansho-prod.sh`, `db-karatewiki-dev.sh`, `db-mitai-dev.sh`, `db-mitai-prod.sh`, `db-netbox.sh`, `db-paperless.sh`, `db-shinkan-dev.sh`, `db-shinkan-prod.sh`, `db-tandoor.sh`, `db-tandoor.sh`, `db-tsumugu-dev.sh`, `db-tsumugu-prod.sh`) sowie die generischen Dump-Engines `db-postgres-docker.sh` und `db-mariadb-docker.sh`. - - `backup-nas-data.sh` & `nas-data-job.sh`: Versionierte rsync-Hardlink-Sicherung (`/media/lars/paperless` → `/backup/data_archiv/01..31` + `last`). - - `publish-backup-set.sh` & `shinkan-prod-manifest.sh`: Atomare Erzeugung und Publikation vollständiger Backup-Sets (`.sql`, `.sql.sha256`, `.sql.manifest.json`, `.sql.media.jsonl`). -- `scripts/shinkan/` (Shinkan Recovery Contract Adapter): - - `shinkan-backup-manifest`: Erzeugt SHA256, Manifest (`schemaVersion: 1` + Shinkan Contract) und inkrementelles Media-Inventar (`media.jsonl`). - - `shinkan-restore`: 9-stufiger atomarer Application-Restore über Staging-DB (`shinkan_restore_`), Single-Transaction-Import, Schema-Migrations-Gate, Plausibilitätsprüfung, `TRUNCATE sessions`, DB-Swap (`shinkan_before_`), `/api/health/ready`-Prüfung und automatischem Rollback bei Fehlern (Exit `5`). - - `shinkan-media-diff`: Abgleich zwischen PostgreSQL (`media_assets`, `exercise_media`), Dateisystem und `media.jsonl` (`missing`, `mismatch`, `added_after_backup`, `orphan`). - - `shinkan-restore-test` & `docker-compose.restoretest.yml`: Isolierter End-to-End-Restore-Test im Projekt `shinkan-restoretest` (Ports `127.0.0.1:18003` / `127.0.0.1:13003`, Read-Only-Media `:ro`, deaktivierte externe Integrationen & Cronjobs). -- `docs/ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md`: Vollständige Ist-Inventur, Gap-Analyse und Architektur-Spezifikation für v0.4. -- `tests/`: - - `test_orchestrator.py`: Unit-Tests für Backup-Phasen, Parallelität, Timeouts, Mount-Prüfung und Notifications. - - `test_recovery.py`: Unit- und Integrationstests für Manifest-Validierung, 4-Stufen-Modell, Schema-Kompatibilitätsprüfung (`EXACT_MATCH`, `FORWARD_MIGRATE`, `INCOMPATIBLE_ROLLBACK`), Sicherheits-Gates (`pi-prod` Sperre, `--confirm`), Dry-Run-Pläne und `shinkan-backup-manifest`. +```text +backup-orchestrator/ +├── orchestrator.py # Universeller Backup- & Recovery-Controller (v0.4.0) +├── install.sh # Server-spezifischer Installer (--host raspberrypi5 | llm-node) +├── targets.json # Globale Target-Registry +├── scripts/ +│ ├── common/ # Gemeinsame, serverunabhängige Backup-Engines +│ │ ├── db-postgres-docker.sh +│ │ ├── db-mariadb-docker.sh +│ │ └── publish-backup-set.sh +│ └── shinkan/ # Anwendungs-Adapter für Shinkan (Recovery Contract) +│ ├── shinkan-backup-manifest +│ ├── shinkan-restore +│ ├── shinkan-restore-test +│ ├── shinkan-media-diff +│ └── docker-compose.restoretest.yml +├── hosts/ # Server-spezifische Profile, Jobs, Wrapper & Systemd-Units +│ ├── raspberrypi5/ # Produktiver Backup-Host (Debian 12 ARM64, 192.168.2.49) +│ │ ├── host.json # Mounts (/backup), Pfade (/media/lars/paperless/db-backup) +│ │ ├── jobs.csv # 17 DB-Jobs (Phase 10), nas-data (Phase 20), system-image (Phase 30) +│ │ ├── targets.json # Host-Targets (pi-prod = source_only) +│ │ ├── notifications.json.example +│ │ ├── scripts/ # Alle 17 DB-Wrapper + NAS-Skripte für raspberrypi5 +│ │ └── systemd/ # backup-orchestrator-daily + .service/.timer (05:30 Uhr) +│ └── llm-node/ # Neuer Anwendungs- & Recovery-Host (Ubuntu 24.04 x86_64, 192.168.2.144) +│ ├── host.json # Mounts (/mnt/nas_backup, /srv/shinkan/prod/media) +│ ├── jobs.csv # Backup-Jobs auf llm-node (shinkan-prod-db für Post-Cutover vorbereitet) +│ ├── targets.json # Targets (llm-prod, llm-restoretest, pi-prod = source_only) +│ ├── notifications.json.example +│ ├── scripts/ # Host-spezifische Wrapper für llm-node +│ └── systemd/ # backup-orchestrator-daily + .service/.timer (05:00 Uhr) +├── docs/ +│ └── ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md +└── tests/ + ├── test_orchestrator.py + └── test_recovery.py +``` --- -## 2. CLI-Befehle (v0.4.0) +## 2. Server-spezifische Installation (`install.sh`) -### A. Backup-Steuerung (unverändert kompatibel zu v0.3) +Auf jedem Server (`raspberrypi5` oder `llm-node`) erkennt `install.sh` automatisch den Hostnamen (`hostname -s`) oder akzeptiert `--host ` und installiert genau die gemeinsame Engine + die zum jeweiligen Server gehörende Konfiguration nach `/etc/backup-orchestrator/`: + +```bash +# Auf raspberrypi5: +sudo ./install.sh --host raspberrypi5 + +# Auf llm-node: +sudo ./install.sh --host llm-node +``` + +Optional: +- `--force-jobs`: Überschreibt eine bereits vorhandene `/etc/backup-orchestrator/jobs.csv` mit `hosts//jobs.csv`. +- `--install-systemd`: Installiert und lädt die zum Server gehörenden Systemd-Units (`backup-orchestrator-daily.service` & `.timer`). + +Auch direkt aus dem Git-Arbeitsverzeichnis können Konfigurationen jedes Servers über `--host` geprüft werden: +```bash +python3 orchestrator.py --plan --host raspberrypi5 +python3 orchestrator.py --plan --host llm-node +``` + +--- + +## 3. CLI-Befehle (v0.4.0) + +### A. Backup-Steuerung (kompatibel zu v0.3) ```bash backup-orchestrator --check @@ -73,7 +114,7 @@ backup-orchestrator --restore \ --- -## 3. Exit-Codes im Recovery-Modus +## 4. Exit-Codes im Recovery-Modus | Exit-Code | Bedeutung | |---|---| @@ -85,9 +126,9 @@ backup-orchestrator --restore \ --- -## 4. Tests ausführen +## 5. Tests ausführen ```bash python3 -m unittest discover -s tests -v ``` -Alle Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und berühren niemals produktive Datenbanken oder NAS-Mounts. +Alle 23 Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und prüfen zusätzlich die Vollständigkeit aller Server-Profile unter `hosts/*`. diff --git a/hosts/llm-node/host.json b/hosts/llm-node/host.json new file mode 100644 index 0000000..76e0688 --- /dev/null +++ b/hosts/llm-node/host.json @@ -0,0 +1,20 @@ +{ + "hostname": "llm-node", + "architecture": "x86_64", + "os": "Ubuntu 24.04 LTS", + "role": "application_and_recovery_host", + "nas_mount": "/mnt/nas_backup", + "nas_source": "192.168.2.63:/volume1/Backup_LLM", + "nas_fstype": "nfs", + "local_dump_dir": "/mnt/nas_backup/shinkan/prod/db/latest", + "archive_sets_dir": "/mnt/nas_backup/shinkan/prod/db", + "media_mounts": { + "shinkan-prod": "/srv/shinkan/prod/media" + }, + "conflict_services": [], + "conflict_timers": [], + "cron_users": [ + "root", + "llmadmin" + ] +} diff --git a/hosts/llm-node/jobs.csv b/hosts/llm-node/jobs.csv new file mode 100644 index 0000000..c3bc695 --- /dev/null +++ b/hosts/llm-node/jobs.csv @@ -0,0 +1,2 @@ +phase,id,script,critical,timeout_seconds,enabled +10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,900,false diff --git a/hosts/llm-node/notifications.json.example b/hosts/llm-node/notifications.json.example new file mode 100644 index 0000000..4a9f064 --- /dev/null +++ b/hosts/llm-node/notifications.json.example @@ -0,0 +1,10 @@ +{ + "timeout_seconds": 3, + "webhooks": [], + "kuma": { + "overall_push_url": "", + "job_push_urls": { + "shinkan-prod-db": "" + } + } +} diff --git a/hosts/llm-node/scripts/db-shinkan-prod.sh b/hosts/llm-node/scripts/db-shinkan-prod.sh new file mode 100644 index 0000000..7d684e2 --- /dev/null +++ b/hosts/llm-node/scripts/db-shinkan-prod.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Backup wrapper for Shinkan Production on llm-node (post-cutover) +set -Eeuo pipefail + +DEST_ROOT="/mnt/nas_backup/shinkan/prod/db" +TS="$(date -u +%Y%m%dT%H%M%SZ)" +GEN_DIR="${DEST_ROOT}/generations/${TS}" +DUMP_FILE="${GEN_DIR}/shinkan-prod.sql" + +install -d -m 0750 "${GEN_DIR}" + +export DB_CONTAINER="shinkan-db-prod" +export DB_USER="shinkan" +export DB_NAME="shinkan" +export OUTPUT_PATH="${DUMP_FILE}" + +/etc/backup-orchestrator/scripts/db-postgres-docker.sh + +# Generate SHA256, Manifest (schemaVersion: 1), and Media Inventory (.media.jsonl) +/etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \ + --dump "${DUMP_FILE}" \ + --env prod \ + --host "llm-node" \ + --compose-dir "/home/llmadmin/docker/shinkan-prod" \ + --media-dir "/srv/shinkan/prod/media" \ + --db-container "shinkan-db-prod" \ + --db-name "shinkan" \ + --db-user "shinkan" + +# Atomically update 'latest' symlink after complete backup set verification +ln -sfn "generations/${TS}" "${DEST_ROOT}/latest.tmp.$$" +mv -Tf "${DEST_ROOT}/latest.tmp.$$" "${DEST_ROOT}/latest" +echo "Published atomic Shinkan backup set on llm-node: ${GEN_DIR}" diff --git a/hosts/llm-node/scripts/shinkan-prod-manifest.sh b/hosts/llm-node/scripts/shinkan-prod-manifest.sh new file mode 100644 index 0000000..3d5a0b0 --- /dev/null +++ b/hosts/llm-node/scripts/shinkan-prod-manifest.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# Standalone helper to generate manifest + media inventory for an existing dump on llm-node +set -Eeuo pipefail + +DUMP_FILE="${1:-/mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql}" + +exec /etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \ + --dump "${DUMP_FILE}" \ + --env prod \ + --host "llm-node" \ + --compose-dir "/home/llmadmin/docker/shinkan-prod" \ + --media-dir "/srv/shinkan/prod/media" \ + --db-container "shinkan-db-prod" \ + --db-name "shinkan" \ + --db-user "shinkan" diff --git a/hosts/llm-node/systemd/backup-orchestrator-daily b/hosts/llm-node/systemd/backup-orchestrator-daily new file mode 100644 index 0000000..f937328 --- /dev/null +++ b/hosts/llm-node/systemd/backup-orchestrator-daily @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +export BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES +exec /usr/local/sbin/backup-orchestrator --run diff --git a/hosts/llm-node/systemd/backup-orchestrator-daily.service b/hosts/llm-node/systemd/backup-orchestrator-daily.service new file mode 100644 index 0000000..f57016d --- /dev/null +++ b/hosts/llm-node/systemd/backup-orchestrator-daily.service @@ -0,0 +1,11 @@ +[Unit] +Description=Backup Orchestrator daily run on llm-node +After=network-online.target docker.service mnt-nas_backup.mount +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/backup-orchestrator-daily +User=root +Group=root +TimeoutStartSec=14400 diff --git a/hosts/llm-node/systemd/backup-orchestrator-daily.timer b/hosts/llm-node/systemd/backup-orchestrator-daily.timer new file mode 100644 index 0000000..7ce4c81 --- /dev/null +++ b/hosts/llm-node/systemd/backup-orchestrator-daily.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Daily timer for Backup Orchestrator on llm-node + +[Timer] +OnCalendar=*-*-* 05:00:00 +Persistent=true +RandomizedDelaySec=60 + +[Install] +WantedBy=timers.target diff --git a/hosts/llm-node/targets.json b/hosts/llm-node/targets.json new file mode 100644 index 0000000..5e810f8 --- /dev/null +++ b/hosts/llm-node/targets.json @@ -0,0 +1,74 @@ +{ + "schemaVersion": 1, + "host": "llm-node", + "hosts": { + "llm-node": { + "hostname": "llm-node", + "arch": "x86_64", + "nas_mount": "/mnt/nas_backup", + "nas_source": "192.168.2.63:/volume1/Backup_LLM", + "nas_fstype": "nfs", + "local_dump_dir": "/mnt/nas_backup/shinkan/prod/db/latest", + "archive_sets_dir": "/mnt/nas_backup/shinkan/prod/db" + } + }, + "targets": { + "pi-prod": { + "application": "shinkan", + "environment": "prod", + "host": "raspberrypi5", + "role": "source_only", + "allow_restore": false, + "compose_dir": "/home/lars/docker/shinkan-prod", + "compose_files": ["docker-compose.yml"], + "env_file": "/home/lars/docker/shinkan-prod/.env", + "media_dir": "/shinkan-media/prod", + "db_container": "shinkan-db-prod", + "db_name": "shinkan", + "db_user": "shinkan", + "health_url": "http://localhost:8003/api/health/ready" + }, + "llm-prod": { + "application": "shinkan", + "environment": "prod", + "host": "llm-node", + "role": "production_target", + "allow_restore": true, + "require_confirmation": "shinkan@llm-node", + "compose_dir": "/home/llmadmin/docker/shinkan-prod", + "compose_files": ["docker-compose.yml", "docker-compose.llm-node.yml"], + "env_file": "/etc/shinkan/prod.env", + "media_dir": "/srv/shinkan/prod/media", + "db_container": "shinkan-db-prod", + "db_name": "shinkan", + "db_user": "shinkan", + "health_url": "http://localhost:8003/api/health/ready", + "restore_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-restore", + "media_diff_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-media-diff" + }, + "llm-restoretest": { + "application": "shinkan", + "environment": "restoretest", + "host": "llm-node", + "role": "isolated_test_target", + "allow_restore": true, + "compose_dir": "/home/llmadmin/docker/shinkan-prod", + "compose_files": [ + "docker-compose.yml", + "docker-compose.llm-node.yml", + "/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml" + ], + "project_name": "shinkan-restoretest", + "env_file": "/etc/shinkan/prod.env", + "media_dir": "/srv/shinkan/prod/media", + "media_readonly": true, + "db_container": "shinkan-db-restoretest", + "db_name": "shinkan_restoretest", + "db_user": "shinkan", + "health_url": "http://127.0.0.1:18003/api/health/ready", + "frontend_url": "http://127.0.0.1:13003", + "restore_test_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-restore-test", + "media_diff_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-media-diff" + } + } +} diff --git a/hosts/raspberrypi5/host.json b/hosts/raspberrypi5/host.json new file mode 100644 index 0000000..393c33f --- /dev/null +++ b/hosts/raspberrypi5/host.json @@ -0,0 +1,26 @@ +{ + "hostname": "raspberrypi5", + "architecture": "arm64", + "os": "Debian GNU/Linux 12 (bookworm)", + "role": "backup_primary_and_legacy_app_host", + "nas_mount": "/backup", + "nas_source": "192.168.2.63:/volume1/Backup_Paperless", + "nas_fstype": "nfs4", + "local_dump_dir": "/media/lars/paperless/db-backup", + "archive_sets_dir": "/backup/data_archiv/last/paperless/db-backup", + "media_mounts": { + "shinkan-prod": "/shinkan-media/prod", + "shinkan-dev": "/shinkan-media/dev" + }, + "conflict_services": [ + "raspiBackup.service" + ], + "conflict_timers": [ + "paperless-data-backup.timer", + "borgmatic.timer" + ], + "cron_users": [ + "root", + "lars" + ] +} diff --git a/hosts/raspberrypi5/jobs.csv b/hosts/raspberrypi5/jobs.csv new file mode 100644 index 0000000..69ea5e7 --- /dev/null +++ b/hosts/raspberrypi5/jobs.csv @@ -0,0 +1,20 @@ +phase,id,script,critical,timeout_seconds,enabled +10,paperless-db,/etc/backup-orchestrator/scripts/db-paperless.sh,true,600,true +10,tandoor-db,/etc/backup-orchestrator/scripts/db-tandoor.sh,true,600,true +10,mitai-prod-db,/etc/backup-orchestrator/scripts/db-mitai-prod.sh,true,900,true +10,mitai-dev-db,/etc/backup-orchestrator/scripts/db-mitai-dev.sh,false,900,true +10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,600,true +10,shinkan-dev-db,/etc/backup-orchestrator/scripts/db-shinkan-dev.sh,false,600,true +20,nas-data,/etc/backup-orchestrator/scripts/nas-data-job.sh,true,10800,true +30,system-image,/etc/backup-orchestrator/scripts/system-image-job.sh,true,14400,false +10,kairo-prod-db,/etc/backup-orchestrator/scripts/db-kairo-prod.sh,true,900,true +10,kairo-dev-db,/etc/backup-orchestrator/scripts/db-kairo-dev.sh,false,900,true +10,tsumugu-prod-db,/etc/backup-orchestrator/scripts/db-tsumugu-prod.sh,true,900,true +10,tsumugu-dev-db,/etc/backup-orchestrator/scripts/db-tsumugu-dev.sh,false,900,true +10,kansho-prod-db,/etc/backup-orchestrator/scripts/db-kansho-prod.sh,true,900,true +10,kansho-dev-db,/etc/backup-orchestrator/scripts/db-kansho-dev.sh,false,900,true +10,buyingagent-prod-db,/etc/backup-orchestrator/scripts/db-buyingagent-prod.sh,true,900,true +10,buyingagent-dev-db,/etc/backup-orchestrator/scripts/db-buyingagent-dev.sh,false,900,true +10,netbox-db,/etc/backup-orchestrator/scripts/db-netbox.sh,true,900,true +10,bookstack-db,/etc/backup-orchestrator/scripts/db-bookstack.sh,true,900,true +10,karatewiki-dev-db,/etc/backup-orchestrator/scripts/db-karatewiki-dev.sh,false,900,true diff --git a/hosts/raspberrypi5/notifications.json.example b/hosts/raspberrypi5/notifications.json.example new file mode 100644 index 0000000..c07aef1 --- /dev/null +++ b/hosts/raspberrypi5/notifications.json.example @@ -0,0 +1,10 @@ +{ + "timeout_seconds": 3, + "webhooks": [], + "kuma": { + "overall_push_url": "", + "job_push_urls": { + "nas-data": "" + } + } +} diff --git a/scripts/backup-nas-data.sh b/hosts/raspberrypi5/scripts/backup-nas-data.sh similarity index 100% rename from scripts/backup-nas-data.sh rename to hosts/raspberrypi5/scripts/backup-nas-data.sh diff --git a/scripts/db-bookstack.sh b/hosts/raspberrypi5/scripts/db-bookstack.sh similarity index 100% rename from scripts/db-bookstack.sh rename to hosts/raspberrypi5/scripts/db-bookstack.sh diff --git a/scripts/db-buyingagent-dev.sh b/hosts/raspberrypi5/scripts/db-buyingagent-dev.sh similarity index 100% rename from scripts/db-buyingagent-dev.sh rename to hosts/raspberrypi5/scripts/db-buyingagent-dev.sh diff --git a/scripts/db-buyingagent-prod.sh b/hosts/raspberrypi5/scripts/db-buyingagent-prod.sh similarity index 100% rename from scripts/db-buyingagent-prod.sh rename to hosts/raspberrypi5/scripts/db-buyingagent-prod.sh diff --git a/scripts/db-kairo-dev.sh b/hosts/raspberrypi5/scripts/db-kairo-dev.sh similarity index 100% rename from scripts/db-kairo-dev.sh rename to hosts/raspberrypi5/scripts/db-kairo-dev.sh diff --git a/scripts/db-kairo-prod.sh b/hosts/raspberrypi5/scripts/db-kairo-prod.sh similarity index 100% rename from scripts/db-kairo-prod.sh rename to hosts/raspberrypi5/scripts/db-kairo-prod.sh diff --git a/scripts/db-kansho-dev.sh b/hosts/raspberrypi5/scripts/db-kansho-dev.sh similarity index 100% rename from scripts/db-kansho-dev.sh rename to hosts/raspberrypi5/scripts/db-kansho-dev.sh diff --git a/scripts/db-kansho-prod.sh b/hosts/raspberrypi5/scripts/db-kansho-prod.sh similarity index 100% rename from scripts/db-kansho-prod.sh rename to hosts/raspberrypi5/scripts/db-kansho-prod.sh diff --git a/scripts/db-karatewiki-dev.sh b/hosts/raspberrypi5/scripts/db-karatewiki-dev.sh similarity index 100% rename from scripts/db-karatewiki-dev.sh rename to hosts/raspberrypi5/scripts/db-karatewiki-dev.sh diff --git a/scripts/db-mitai-dev.sh b/hosts/raspberrypi5/scripts/db-mitai-dev.sh similarity index 100% rename from scripts/db-mitai-dev.sh rename to hosts/raspberrypi5/scripts/db-mitai-dev.sh diff --git a/scripts/db-mitai-prod.sh b/hosts/raspberrypi5/scripts/db-mitai-prod.sh similarity index 100% rename from scripts/db-mitai-prod.sh rename to hosts/raspberrypi5/scripts/db-mitai-prod.sh diff --git a/scripts/db-netbox.sh b/hosts/raspberrypi5/scripts/db-netbox.sh similarity index 100% rename from scripts/db-netbox.sh rename to hosts/raspberrypi5/scripts/db-netbox.sh diff --git a/scripts/db-paperless.sh b/hosts/raspberrypi5/scripts/db-paperless.sh similarity index 100% rename from scripts/db-paperless.sh rename to hosts/raspberrypi5/scripts/db-paperless.sh diff --git a/scripts/db-shinkan-dev.sh b/hosts/raspberrypi5/scripts/db-shinkan-dev.sh similarity index 100% rename from scripts/db-shinkan-dev.sh rename to hosts/raspberrypi5/scripts/db-shinkan-dev.sh diff --git a/scripts/db-shinkan-prod.sh b/hosts/raspberrypi5/scripts/db-shinkan-prod.sh similarity index 100% rename from scripts/db-shinkan-prod.sh rename to hosts/raspberrypi5/scripts/db-shinkan-prod.sh diff --git a/scripts/db-tandoor.sh b/hosts/raspberrypi5/scripts/db-tandoor.sh similarity index 100% rename from scripts/db-tandoor.sh rename to hosts/raspberrypi5/scripts/db-tandoor.sh diff --git a/scripts/db-tsumugu-dev.sh b/hosts/raspberrypi5/scripts/db-tsumugu-dev.sh similarity index 100% rename from scripts/db-tsumugu-dev.sh rename to hosts/raspberrypi5/scripts/db-tsumugu-dev.sh diff --git a/scripts/db-tsumugu-prod.sh b/hosts/raspberrypi5/scripts/db-tsumugu-prod.sh similarity index 100% rename from scripts/db-tsumugu-prod.sh rename to hosts/raspberrypi5/scripts/db-tsumugu-prod.sh diff --git a/scripts/nas-data-job.sh b/hosts/raspberrypi5/scripts/nas-data-job.sh similarity index 100% rename from scripts/nas-data-job.sh rename to hosts/raspberrypi5/scripts/nas-data-job.sh diff --git a/scripts/shinkan-prod-manifest.sh b/hosts/raspberrypi5/scripts/shinkan-prod-manifest.sh similarity index 100% rename from scripts/shinkan-prod-manifest.sh rename to hosts/raspberrypi5/scripts/shinkan-prod-manifest.sh diff --git a/scripts/system-image-job.sh b/hosts/raspberrypi5/scripts/system-image-job.sh similarity index 100% rename from scripts/system-image-job.sh rename to hosts/raspberrypi5/scripts/system-image-job.sh diff --git a/systemd/backup-orchestrator-daily b/hosts/raspberrypi5/systemd/backup-orchestrator-daily similarity index 100% rename from systemd/backup-orchestrator-daily rename to hosts/raspberrypi5/systemd/backup-orchestrator-daily diff --git a/systemd/backup-orchestrator-daily.service b/hosts/raspberrypi5/systemd/backup-orchestrator-daily.service similarity index 100% rename from systemd/backup-orchestrator-daily.service rename to hosts/raspberrypi5/systemd/backup-orchestrator-daily.service diff --git a/systemd/backup-orchestrator-daily.timer b/hosts/raspberrypi5/systemd/backup-orchestrator-daily.timer similarity index 100% rename from systemd/backup-orchestrator-daily.timer rename to hosts/raspberrypi5/systemd/backup-orchestrator-daily.timer diff --git a/hosts/raspberrypi5/targets.json b/hosts/raspberrypi5/targets.json new file mode 100644 index 0000000..d4ef957 --- /dev/null +++ b/hosts/raspberrypi5/targets.json @@ -0,0 +1,32 @@ +{ + "schemaVersion": 1, + "host": "raspberrypi5", + "hosts": { + "raspberrypi5": { + "hostname": "raspberrypi5", + "arch": "arm64", + "nas_mount": "/backup", + "nas_source": "192.168.2.63:/volume1/Backup_Paperless", + "nas_fstype": "nfs4", + "local_dump_dir": "/media/lars/paperless/db-backup", + "archive_sets_dir": "/backup/data_archiv/last/paperless/db-backup" + } + }, + "targets": { + "pi-prod": { + "application": "shinkan", + "environment": "prod", + "host": "raspberrypi5", + "role": "source_only", + "allow_restore": false, + "compose_dir": "/home/lars/docker/shinkan-prod", + "compose_files": ["docker-compose.yml"], + "env_file": "/home/lars/docker/shinkan-prod/.env", + "media_dir": "/shinkan-media/prod", + "db_container": "shinkan-db-prod", + "db_name": "shinkan", + "db_user": "shinkan", + "health_url": "http://localhost:8003/api/health/ready" + } + } +} diff --git a/install.sh b/install.sh new file mode 100644 index 0000000..f710b9a --- /dev/null +++ b/install.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +# Host-aware installer for Backup-Orchestrator v0.4 +# Usage: sudo ./install.sh [--host ] [--force-jobs] [--install-systemd] +set -Eeuo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +HOST_NAME="$(hostname -s 2>/dev/null || hostname)" +FORCE_JOBS=0 +INSTALL_SYSTEMD=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --host) + HOST_NAME="$2" + shift 2 + ;; + --force-jobs) + FORCE_JOBS=1 + shift + ;; + --install-systemd) + INSTALL_SYSTEMD=1 + shift + ;; + -h|--help) + echo "Usage: sudo $0 [--host ] [--force-jobs] [--install-systemd]" + exit 0 + ;; + *) + echo "ERROR: Unknown argument: $1" >&2 + exit 2 + ;; + esac +done + +HOST_DIR="${REPO_ROOT}/hosts/${HOST_NAME}" +if [[ ! -d "${HOST_DIR}" ]]; then + echo "ERROR: Unknown host configuration '${HOST_NAME}'. Expected directory: ${HOST_DIR}" >&2 + echo "Available hosts:" >&2 + ls -1 "${REPO_ROOT}/hosts" >&2 + exit 2 +fi + +echo "=== Installing Backup-Orchestrator v0.4 for host '${HOST_NAME}' ===" + +# 1. Backup existing binary if present +if [[ -f /usr/local/sbin/backup-orchestrator ]]; then + cp -a /usr/local/sbin/backup-orchestrator "/usr/local/sbin/backup-orchestrator.bak.$(date +%Y%m%d%H%M%S)" +fi +install -m 0755 "${REPO_ROOT}/orchestrator.py" /usr/local/sbin/backup-orchestrator + +# 2. Prepare /etc/backup-orchestrator directories +install -d -m 0750 /etc/backup-orchestrator +install -d -m 0750 /etc/backup-orchestrator/scripts +install -d -m 0750 /etc/backup-orchestrator/scripts/shinkan +install -d -m 0700 /etc/backup-orchestrator/credentials +install -d -m 0750 /var/log/backup-orchestrator + +# 3. Install host-specific metadata & target registry +install -m 0640 "${HOST_DIR}/host.json" /etc/backup-orchestrator/host.json +install -m 0640 "${HOST_DIR}/targets.json" /etc/backup-orchestrator/targets.json + +if [[ ! -f /etc/backup-orchestrator/jobs.csv || "${FORCE_JOBS}" -eq 1 ]]; then + install -m 0640 "${HOST_DIR}/jobs.csv" /etc/backup-orchestrator/jobs.csv + echo "Installed /etc/backup-orchestrator/jobs.csv from hosts/${HOST_NAME}/jobs.csv" +else + echo "Keeping existing /etc/backup-orchestrator/jobs.csv (use --force-jobs to overwrite)" +fi + +if [[ ! -f /etc/backup-orchestrator/notifications.json ]]; then + install -m 0600 "${HOST_DIR}/notifications.json.example" /etc/backup-orchestrator/notifications.json + echo "Created initial /etc/backup-orchestrator/notifications.json (mode 0600)" +fi + +# 4. Install shared common scripts + host-specific scripts + Shinkan adapters +for s in "${REPO_ROOT}/scripts/common/"*.sh; do + [[ -f "$s" ]] && install -m 0750 "$s" "/etc/backup-orchestrator/scripts/$(basename "$s")" +done + +for s in "${HOST_DIR}/scripts/"*.sh; do + [[ -f "$s" ]] && install -m 0750 "$s" "/etc/backup-orchestrator/scripts/$(basename "$s")" +done + +for s in "${REPO_ROOT}/scripts/shinkan/"*; do + if [[ -f "$s" ]]; then + case "$s" in + *.yml|*.yaml|*.json) + install -m 0644 "$s" "/etc/backup-orchestrator/scripts/shinkan/$(basename "$s")" + ;; + *) + install -m 0755 "$s" "/etc/backup-orchestrator/scripts/shinkan/$(basename "$s")" + ;; + esac + fi +done + +# 5. Install daily wrapper & optional systemd units +if [[ -f "${HOST_DIR}/systemd/backup-orchestrator-daily" ]]; then + install -m 0755 "${HOST_DIR}/systemd/backup-orchestrator-daily" /usr/local/sbin/backup-orchestrator-daily +fi + +if [[ "${INSTALL_SYSTEMD}" -eq 1 ]]; then + install -m 0644 "${HOST_DIR}/systemd/backup-orchestrator-daily.service" /etc/systemd/system/backup-orchestrator-daily.service + install -m 0644 "${HOST_DIR}/systemd/backup-orchestrator-daily.timer" /etc/systemd/system/backup-orchestrator-daily.timer + systemctl daemon-reload + echo "Installed systemd service and timer for ${HOST_NAME}." +fi + +echo "=== Verification (--check) ===" +/usr/local/sbin/backup-orchestrator --check diff --git a/orchestrator.py b/orchestrator.py index 09096dd..ee51420 100644 --- a/orchestrator.py +++ b/orchestrator.py @@ -66,8 +66,15 @@ def lock_exclusive(fp): fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB) -def load_host_config(path=DEFAULT_HOST_CONFIG): +def resolve_repo_host_dir(host_name=None): + host = host_name or socket.gethostname().split('.')[0] + candidate = Path(__file__).resolve().parent / 'hosts' / host + return candidate if candidate.is_dir() else None + + +def load_host_config(path=DEFAULT_HOST_CONFIG, host_name=None): default = { + 'hostname': host_name or socket.gethostname(), 'nas_mount': NAS_MOUNT, 'nas_source': NAS_SOURCE, 'nas_fstype': 'nfs4', @@ -77,16 +84,37 @@ def load_host_config(path=DEFAULT_HOST_CONFIG): 'conflict_timers': ['paperless-data-backup.timer', 'borgmatic.timer'], 'cron_users': ['root', 'lars'], } - if path and Path(path).is_file(): - with Path(path).open(encoding='utf-8') as fp: - data = json.load(fp) - if isinstance(data, dict): - default.update(data) + candidates = [] + if host_name: + host_dir = resolve_repo_host_dir(host_name) + if host_dir: + candidates.append(host_dir / 'host.json') + if path: + candidates.append(Path(path)) + host_dir = resolve_repo_host_dir() + if host_dir: + candidates.append(host_dir / 'host.json') + for candidate in candidates: + if candidate.is_file(): + with candidate.open(encoding='utf-8') as fp: + data = json.load(fp) + if isinstance(data, dict): + default.update(data) + break return default -def load_targets_config(path=DEFAULT_TARGETS_CONFIG): - candidates = [Path(path)] if path else [] +def load_targets_config(path=DEFAULT_TARGETS_CONFIG, host_name=None): + candidates = [] + if host_name: + host_dir = resolve_repo_host_dir(host_name) + if host_dir: + candidates.append(host_dir / 'targets.json') + if path: + candidates.append(Path(path)) + host_dir = resolve_repo_host_dir() + if host_dir: + candidates.append(host_dir / 'targets.json') candidates.append(Path(__file__).resolve().parent / 'targets.json') for candidate in candidates: if candidate.is_file(): @@ -1396,6 +1424,7 @@ def main(): mode.add_argument('--restore-test', action='store_true', help='Execute isolated disposable restore test') parser.add_argument('--json', action='store_true', help='Machine-readable JSON output') + parser.add_argument('--host', type=str, default=None, help='Select server profile from hosts/ (e.g. raspberrypi5, llm-node)') parser.add_argument('--config', type=Path, default=DEFAULT_CONFIG) parser.add_argument('--notify-config', type=Path, default=DEFAULT_NOTIFY_CONFIG) parser.add_argument('--targets-config', type=Path, default=DEFAULT_TARGETS_CONFIG) @@ -1419,6 +1448,17 @@ def main(): parser.add_argument('--keep', action='store_true', help='Keep isolated test environment running after --restore-test') args = parser.parse_args() + if args.host: + host_dir = resolve_repo_host_dir(args.host) + if not host_dir: + parser.error(f'Unknown host profile: {args.host!r}') + if args.config == DEFAULT_CONFIG: + args.config = host_dir / 'jobs.csv' + if args.notify_config == DEFAULT_NOTIFY_CONFIG and (host_dir / 'notifications.json.example').is_file(): + args.notify_config = host_dir / 'notifications.json.example' + if args.targets_config == DEFAULT_TARGETS_CONFIG and (host_dir / 'targets.json').is_file(): + args.targets_config = host_dir / 'targets.json' + if not 1 <= args.max_parallel <= 16: parser.error('--max-parallel must be 1..16') json_capable = ( @@ -1434,7 +1474,7 @@ def main(): return show_status(args.log_root, as_json=args.json) if args.list_recovery_points: - host_cfg = load_host_config() + host_cfg = load_host_config(host_name=args.host) search_dirs = args.search_dirs or [ host_cfg['archive_sets_dir'], host_cfg['local_dump_dir'], @@ -1472,7 +1512,7 @@ def main(): if args.recovery_plan: if not args.point or not args.target: parser.error('--point and --target are required with --recovery-plan') - targets_cfg = load_targets_config(args.targets_config) + targets_cfg = load_targets_config(args.targets_config, host_name=args.host) plan_doc = generate_recovery_plan(args.point, args.target, targets_cfg, sessions_mode=args.sessions) if args.json: print(json.dumps(plan_doc, ensure_ascii=False, indent=2)) @@ -1492,11 +1532,11 @@ def main(): return plan_doc['compatibility']['exit_code'] if args.restore: - targets_cfg = load_targets_config(args.targets_config) + targets_cfg = load_targets_config(args.targets_config, host_name=args.host) return execute_recovery_command('restore', args, targets_cfg) if args.restore_test: - targets_cfg = load_targets_config(args.targets_config) + targets_cfg = load_targets_config(args.targets_config, host_name=args.host) return execute_recovery_command('restore-test', args, targets_cfg) jobs = load_jobs(args.config) diff --git a/scripts/db-mariadb-docker.sh b/scripts/common/db-mariadb-docker.sh similarity index 100% rename from scripts/db-mariadb-docker.sh rename to scripts/common/db-mariadb-docker.sh diff --git a/scripts/db-postgres-docker.sh b/scripts/common/db-postgres-docker.sh similarity index 100% rename from scripts/db-postgres-docker.sh rename to scripts/common/db-postgres-docker.sh diff --git a/scripts/publish-backup-set.sh b/scripts/common/publish-backup-set.sh similarity index 100% rename from scripts/publish-backup-set.sh rename to scripts/common/publish-backup-set.sh diff --git a/tests/test_recovery.py b/tests/test_recovery.py index 06440b2..ed6ff89 100644 --- a/tests/test_recovery.py +++ b/tests/test_recovery.py @@ -239,6 +239,35 @@ class RecoveryPointAndManifestTests(unittest.TestCase): self.assertTrue((out_dir / 'shinkan-prod.sql.sha256').is_file()) self.assertTrue((out_dir / 'shinkan-prod.sql.media.jsonl').is_file()) + def test_host_profiles_raspberrypi5_and_llm_node(self): + repo_root = Path(__file__).resolve().parents[1] + for host_name, expected_jobs in (('raspberrypi5', 19), ('llm-node', 1)): + host_dir = app.resolve_repo_host_dir(host_name) + self.assertIsNotNone(host_dir, f'Missing host dir for {host_name}') + host_cfg = app.load_host_config(host_name=host_name) + self.assertEqual(host_cfg['hostname'], host_name) + jobs = app.load_jobs(host_dir / 'jobs.csv') + self.assertEqual(len(jobs), expected_jobs) + notify = app.load_notifications( + host_dir / 'notifications.json.example', + known_ids=[j['id'] for j in jobs], + ) + self.assertIn('kuma', notify) + targets = app.load_targets_config(host_name=host_name) + self.assertEqual(targets['host'], host_name) + self.assertIn('pi-prod', targets['targets']) + # Verify every job script exists in either hosts//scripts/ or scripts/common/ + for job in jobs: + script_name = Path(job['script']).name + if job['id'] == 'system-image': + continue + exists_in_host = (host_dir / 'scripts' / script_name).is_file() + exists_in_common = (repo_root / 'scripts' / 'common' / script_name).is_file() + self.assertTrue( + exists_in_host or exists_in_common, + f"Missing script {script_name} for job {job['id']} on host {host_name}", + ) + if __name__ == '__main__': unittest.main()