refactor(hosts): organize repository by server profiles (hosts/raspberrypi5, hosts/llm-node) and add host-aware installer
This commit is contained in:
1 parent
c53fc0a939
commit
c2740c9542
44 files changed
+528
-40
No files matched your search
@@ -1,38 +1,79 @@
|
||||
# Backup-Orchestrator v0.4 – Backup & Recovery Management
|
||||
# Backup-Orchestrator v0.4 – Multi-Server Backup & Recovery Management
|
||||
|
||||
**Status: Version 0.4.0 (Multi-Host Backup & Application Recovery Management).**
|
||||
Erweitert den auf `raspberrypi5` produktiv laufenden Backup-Orchestrator v0.3 um ein integriertes Recovery Management, kryptografisch geprüfte Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Konflikt-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für die Referenzanwendung **Shinkan** auf `llm-node` (`192.168.2.144`).
|
||||
Erweitert den Backup-Orchestrator v0.3 um eine **server-spezifische Repository-Struktur (`hosts/<server>/`)**, ein integriertes Recovery Management mit kryptografisch geprüften Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für **Shinkan** auf `llm-node` (`192.168.2.144`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Repository-Struktur & Enthaltene Dateien
|
||||
## 1. Repository-Struktur (nach Servern & gemeinsamen Modulen getrennt)
|
||||
|
||||
- `orchestrator.py`: Phasenbasierter Backup- & Recovery-Controller (`v0.4.0`, Python 3.11+, nur Standardbibliothek).
|
||||
- `targets.json`: Multi-Host- und Recovery-Target-Definitionen (`raspberrypi5`, `llm-node`, Targets `pi-prod`, `llm-prod`, `llm-restoretest`).
|
||||
- `jobs.csv`: Produktive Backup-Job-Tabelle (17 Datenbank-Exporte in Phase 10, `nas-data` in Phase 20, `system-image` in Phase 30).
|
||||
- `notifications.json.example`: Beispielkonfiguration für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore (enthält keine echten Secrets).
|
||||
- `systemd/`:
|
||||
- `backup-orchestrator-daily`: Täglicher Runner-Wrapper (ab v0.4 ohne fragiles `systemctl disable/enable raspiBackup.timer`).
|
||||
- `backup-orchestrator-daily.service` & `backup-orchestrator-daily.timer`: Systemd-Units für den täglichen Lauf um `05:30` Uhr.
|
||||
- `scripts/`:
|
||||
- Alle 17 Datenbank-Dump-Wrapper (`db-bookstack.sh`, `db-buyingagent-dev.sh`, `db-buyingagent-prod.sh`, `db-kairo-dev.sh`, `db-kairo-prod.sh`, `db-kansho-dev.sh`, `db-kansho-prod.sh`, `db-karatewiki-dev.sh`, `db-mitai-dev.sh`, `db-mitai-prod.sh`, `db-netbox.sh`, `db-paperless.sh`, `db-shinkan-dev.sh`, `db-shinkan-prod.sh`, `db-tandoor.sh`, `db-tandoor.sh`, `db-tsumugu-dev.sh`, `db-tsumugu-prod.sh`) sowie die generischen Dump-Engines `db-postgres-docker.sh` und `db-mariadb-docker.sh`.
|
||||
- `backup-nas-data.sh` & `nas-data-job.sh`: Versionierte rsync-Hardlink-Sicherung (`/media/lars/paperless` → `/backup/data_archiv/01..31` + `last`).
|
||||
- `publish-backup-set.sh` & `shinkan-prod-manifest.sh`: Atomare Erzeugung und Publikation vollständiger Backup-Sets (`<dump>.sql`, `<dump>.sql.sha256`, `<dump>.sql.manifest.json`, `<dump>.sql.media.jsonl`).
|
||||
- `scripts/shinkan/` (Shinkan Recovery Contract Adapter):
|
||||
- `shinkan-backup-manifest`: Erzeugt SHA256, Manifest (`schemaVersion: 1` + Shinkan Contract) und inkrementelles Media-Inventar (`media.jsonl`).
|
||||
- `shinkan-restore`: 9-stufiger atomarer Application-Restore über Staging-DB (`shinkan_restore_<ts>`), Single-Transaction-Import, Schema-Migrations-Gate, Plausibilitätsprüfung, `TRUNCATE sessions`, DB-Swap (`shinkan_before_<ts>`), `/api/health/ready`-Prüfung und automatischem Rollback bei Fehlern (Exit `5`).
|
||||
- `shinkan-media-diff`: Abgleich zwischen PostgreSQL (`media_assets`, `exercise_media`), Dateisystem und `media.jsonl` (`missing`, `mismatch`, `added_after_backup`, `orphan`).
|
||||
- `shinkan-restore-test` & `docker-compose.restoretest.yml`: Isolierter End-to-End-Restore-Test im Projekt `shinkan-restoretest` (Ports `127.0.0.1:18003` / `127.0.0.1:13003`, Read-Only-Media `:ro`, deaktivierte externe Integrationen & Cronjobs).
|
||||
- `docs/ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md`: Vollständige Ist-Inventur, Gap-Analyse und Architektur-Spezifikation für v0.4.
|
||||
- `tests/`:
|
||||
- `test_orchestrator.py`: Unit-Tests für Backup-Phasen, Parallelität, Timeouts, Mount-Prüfung und Notifications.
|
||||
- `test_recovery.py`: Unit- und Integrationstests für Manifest-Validierung, 4-Stufen-Modell, Schema-Kompatibilitätsprüfung (`EXACT_MATCH`, `FORWARD_MIGRATE`, `INCOMPATIBLE_ROLLBACK`), Sicherheits-Gates (`pi-prod` Sperre, `--confirm`), Dry-Run-Pläne und `shinkan-backup-manifest`.
|
||||
```text
|
||||
backup-orchestrator/
|
||||
├── orchestrator.py # Universeller Backup- & Recovery-Controller (v0.4.0)
|
||||
├── install.sh # Server-spezifischer Installer (--host raspberrypi5 | llm-node)
|
||||
├── targets.json # Globale Target-Registry
|
||||
├── scripts/
|
||||
│ ├── common/ # Gemeinsame, serverunabhängige Backup-Engines
|
||||
│ │ ├── db-postgres-docker.sh
|
||||
│ │ ├── db-mariadb-docker.sh
|
||||
│ │ └── publish-backup-set.sh
|
||||
│ └── shinkan/ # Anwendungs-Adapter für Shinkan (Recovery Contract)
|
||||
│ ├── shinkan-backup-manifest
|
||||
│ ├── shinkan-restore
|
||||
│ ├── shinkan-restore-test
|
||||
│ ├── shinkan-media-diff
|
||||
│ └── docker-compose.restoretest.yml
|
||||
├── hosts/ # Server-spezifische Profile, Jobs, Wrapper & Systemd-Units
|
||||
│ ├── raspberrypi5/ # Produktiver Backup-Host (Debian 12 ARM64, 192.168.2.49)
|
||||
│ │ ├── host.json # Mounts (/backup), Pfade (/media/lars/paperless/db-backup)
|
||||
│ │ ├── jobs.csv # 17 DB-Jobs (Phase 10), nas-data (Phase 20), system-image (Phase 30)
|
||||
│ │ ├── targets.json # Host-Targets (pi-prod = source_only)
|
||||
│ │ ├── notifications.json.example
|
||||
│ │ ├── scripts/ # Alle 17 DB-Wrapper + NAS-Skripte für raspberrypi5
|
||||
│ │ └── systemd/ # backup-orchestrator-daily + .service/.timer (05:30 Uhr)
|
||||
│ └── llm-node/ # Neuer Anwendungs- & Recovery-Host (Ubuntu 24.04 x86_64, 192.168.2.144)
|
||||
│ ├── host.json # Mounts (/mnt/nas_backup, /srv/shinkan/prod/media)
|
||||
│ ├── jobs.csv # Backup-Jobs auf llm-node (shinkan-prod-db für Post-Cutover vorbereitet)
|
||||
│ ├── targets.json # Targets (llm-prod, llm-restoretest, pi-prod = source_only)
|
||||
│ ├── notifications.json.example
|
||||
│ ├── scripts/ # Host-spezifische Wrapper für llm-node
|
||||
│ └── systemd/ # backup-orchestrator-daily + .service/.timer (05:00 Uhr)
|
||||
├── docs/
|
||||
│ └── ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md
|
||||
└── tests/
|
||||
├── test_orchestrator.py
|
||||
└── test_recovery.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. CLI-Befehle (v0.4.0)
|
||||
## 2. Server-spezifische Installation (`install.sh`)
|
||||
|
||||
### A. Backup-Steuerung (unverändert kompatibel zu v0.3)
|
||||
Auf jedem Server (`raspberrypi5` oder `llm-node`) erkennt `install.sh` automatisch den Hostnamen (`hostname -s`) oder akzeptiert `--host <servername>` und installiert genau die gemeinsame Engine + die zum jeweiligen Server gehörende Konfiguration nach `/etc/backup-orchestrator/`:
|
||||
|
||||
```bash
|
||||
# Auf raspberrypi5:
|
||||
sudo ./install.sh --host raspberrypi5
|
||||
|
||||
# Auf llm-node:
|
||||
sudo ./install.sh --host llm-node
|
||||
```
|
||||
|
||||
Optional:
|
||||
- `--force-jobs`: Überschreibt eine bereits vorhandene `/etc/backup-orchestrator/jobs.csv` mit `hosts/<host>/jobs.csv`.
|
||||
- `--install-systemd`: Installiert und lädt die zum Server gehörenden Systemd-Units (`backup-orchestrator-daily.service` & `.timer`).
|
||||
|
||||
Auch direkt aus dem Git-Arbeitsverzeichnis können Konfigurationen jedes Servers über `--host` geprüft werden:
|
||||
```bash
|
||||
python3 orchestrator.py --plan --host raspberrypi5
|
||||
python3 orchestrator.py --plan --host llm-node
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. CLI-Befehle (v0.4.0)
|
||||
|
||||
### A. Backup-Steuerung (kompatibel zu v0.3)
|
||||
|
||||
```bash
|
||||
backup-orchestrator --check
|
||||
@@ -73,7 +114,7 @@ backup-orchestrator --restore \
|
||||
|
||||
---
|
||||
|
||||
## 3. Exit-Codes im Recovery-Modus
|
||||
## 4. Exit-Codes im Recovery-Modus
|
||||
|
||||
| Exit-Code | Bedeutung |
|
||||
|---|---|
|
||||
@@ -85,9 +126,9 @@ backup-orchestrator --restore \
|
||||
|
||||
---
|
||||
|
||||
## 4. Tests ausführen
|
||||
## 5. Tests ausführen
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s tests -v
|
||||
```
|
||||
Alle Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und berühren niemals produktive Datenbanken oder NAS-Mounts.
|
||||
Alle 23 Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und prüfen zusätzlich die Vollständigkeit aller Server-Profile unter `hosts/*`.
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"hostname": "llm-node",
|
||||
"architecture": "x86_64",
|
||||
"os": "Ubuntu 24.04 LTS",
|
||||
"role": "application_and_recovery_host",
|
||||
"nas_mount": "/mnt/nas_backup",
|
||||
"nas_source": "192.168.2.63:/volume1/Backup_LLM",
|
||||
"nas_fstype": "nfs",
|
||||
"local_dump_dir": "/mnt/nas_backup/shinkan/prod/db/latest",
|
||||
"archive_sets_dir": "/mnt/nas_backup/shinkan/prod/db",
|
||||
"media_mounts": {
|
||||
"shinkan-prod": "/srv/shinkan/prod/media"
|
||||
},
|
||||
"conflict_services": [],
|
||||
"conflict_timers": [],
|
||||
"cron_users": [
|
||||
"root",
|
||||
"llmadmin"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
phase,id,script,critical,timeout_seconds,enabled
|
||||
10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,900,false
|
||||
|
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"timeout_seconds": 3,
|
||||
"webhooks": [],
|
||||
"kuma": {
|
||||
"overall_push_url": "",
|
||||
"job_push_urls": {
|
||||
"shinkan-prod-db": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env bash
|
||||
# Backup wrapper for Shinkan Production on llm-node (post-cutover)
|
||||
set -Eeuo pipefail
|
||||
|
||||
DEST_ROOT="/mnt/nas_backup/shinkan/prod/db"
|
||||
TS="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
GEN_DIR="${DEST_ROOT}/generations/${TS}"
|
||||
DUMP_FILE="${GEN_DIR}/shinkan-prod.sql"
|
||||
|
||||
install -d -m 0750 "${GEN_DIR}"
|
||||
|
||||
export DB_CONTAINER="shinkan-db-prod"
|
||||
export DB_USER="shinkan"
|
||||
export DB_NAME="shinkan"
|
||||
export OUTPUT_PATH="${DUMP_FILE}"
|
||||
|
||||
/etc/backup-orchestrator/scripts/db-postgres-docker.sh
|
||||
|
||||
# Generate SHA256, Manifest (schemaVersion: 1), and Media Inventory (.media.jsonl)
|
||||
/etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \
|
||||
--dump "${DUMP_FILE}" \
|
||||
--env prod \
|
||||
--host "llm-node" \
|
||||
--compose-dir "/home/llmadmin/docker/shinkan-prod" \
|
||||
--media-dir "/srv/shinkan/prod/media" \
|
||||
--db-container "shinkan-db-prod" \
|
||||
--db-name "shinkan" \
|
||||
--db-user "shinkan"
|
||||
|
||||
# Atomically update 'latest' symlink after complete backup set verification
|
||||
ln -sfn "generations/${TS}" "${DEST_ROOT}/latest.tmp.$$"
|
||||
mv -Tf "${DEST_ROOT}/latest.tmp.$$" "${DEST_ROOT}/latest"
|
||||
echo "Published atomic Shinkan backup set on llm-node: ${GEN_DIR}"
|
||||
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
# Standalone helper to generate manifest + media inventory for an existing dump on llm-node
|
||||
set -Eeuo pipefail
|
||||
|
||||
DUMP_FILE="${1:-/mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql}"
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \
|
||||
--dump "${DUMP_FILE}" \
|
||||
--env prod \
|
||||
--host "llm-node" \
|
||||
--compose-dir "/home/llmadmin/docker/shinkan-prod" \
|
||||
--media-dir "/srv/shinkan/prod/media" \
|
||||
--db-container "shinkan-db-prod" \
|
||||
--db-name "shinkan" \
|
||||
--db-user "shinkan"
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES
|
||||
exec /usr/local/sbin/backup-orchestrator --run
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Backup Orchestrator daily run on llm-node
|
||||
After=network-online.target docker.service mnt-nas_backup.mount
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/backup-orchestrator-daily
|
||||
User=root
|
||||
Group=root
|
||||
TimeoutStartSec=14400
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Daily timer for Backup Orchestrator on llm-node
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 05:00:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=60
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"host": "llm-node",
|
||||
"hosts": {
|
||||
"llm-node": {
|
||||
"hostname": "llm-node",
|
||||
"arch": "x86_64",
|
||||
"nas_mount": "/mnt/nas_backup",
|
||||
"nas_source": "192.168.2.63:/volume1/Backup_LLM",
|
||||
"nas_fstype": "nfs",
|
||||
"local_dump_dir": "/mnt/nas_backup/shinkan/prod/db/latest",
|
||||
"archive_sets_dir": "/mnt/nas_backup/shinkan/prod/db"
|
||||
}
|
||||
},
|
||||
"targets": {
|
||||
"pi-prod": {
|
||||
"application": "shinkan",
|
||||
"environment": "prod",
|
||||
"host": "raspberrypi5",
|
||||
"role": "source_only",
|
||||
"allow_restore": false,
|
||||
"compose_dir": "/home/lars/docker/shinkan-prod",
|
||||
"compose_files": ["docker-compose.yml"],
|
||||
"env_file": "/home/lars/docker/shinkan-prod/.env",
|
||||
"media_dir": "/shinkan-media/prod",
|
||||
"db_container": "shinkan-db-prod",
|
||||
"db_name": "shinkan",
|
||||
"db_user": "shinkan",
|
||||
"health_url": "http://localhost:8003/api/health/ready"
|
||||
},
|
||||
"llm-prod": {
|
||||
"application": "shinkan",
|
||||
"environment": "prod",
|
||||
"host": "llm-node",
|
||||
"role": "production_target",
|
||||
"allow_restore": true,
|
||||
"require_confirmation": "shinkan@llm-node",
|
||||
"compose_dir": "/home/llmadmin/docker/shinkan-prod",
|
||||
"compose_files": ["docker-compose.yml", "docker-compose.llm-node.yml"],
|
||||
"env_file": "/etc/shinkan/prod.env",
|
||||
"media_dir": "/srv/shinkan/prod/media",
|
||||
"db_container": "shinkan-db-prod",
|
||||
"db_name": "shinkan",
|
||||
"db_user": "shinkan",
|
||||
"health_url": "http://localhost:8003/api/health/ready",
|
||||
"restore_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-restore",
|
||||
"media_diff_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-media-diff"
|
||||
},
|
||||
"llm-restoretest": {
|
||||
"application": "shinkan",
|
||||
"environment": "restoretest",
|
||||
"host": "llm-node",
|
||||
"role": "isolated_test_target",
|
||||
"allow_restore": true,
|
||||
"compose_dir": "/home/llmadmin/docker/shinkan-prod",
|
||||
"compose_files": [
|
||||
"docker-compose.yml",
|
||||
"docker-compose.llm-node.yml",
|
||||
"/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml"
|
||||
],
|
||||
"project_name": "shinkan-restoretest",
|
||||
"env_file": "/etc/shinkan/prod.env",
|
||||
"media_dir": "/srv/shinkan/prod/media",
|
||||
"media_readonly": true,
|
||||
"db_container": "shinkan-db-restoretest",
|
||||
"db_name": "shinkan_restoretest",
|
||||
"db_user": "shinkan",
|
||||
"health_url": "http://127.0.0.1:18003/api/health/ready",
|
||||
"frontend_url": "http://127.0.0.1:13003",
|
||||
"restore_test_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-restore-test",
|
||||
"media_diff_script": "/etc/backup-orchestrator/scripts/shinkan/shinkan-media-diff"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"hostname": "raspberrypi5",
|
||||
"architecture": "arm64",
|
||||
"os": "Debian GNU/Linux 12 (bookworm)",
|
||||
"role": "backup_primary_and_legacy_app_host",
|
||||
"nas_mount": "/backup",
|
||||
"nas_source": "192.168.2.63:/volume1/Backup_Paperless",
|
||||
"nas_fstype": "nfs4",
|
||||
"local_dump_dir": "/media/lars/paperless/db-backup",
|
||||
"archive_sets_dir": "/backup/data_archiv/last/paperless/db-backup",
|
||||
"media_mounts": {
|
||||
"shinkan-prod": "/shinkan-media/prod",
|
||||
"shinkan-dev": "/shinkan-media/dev"
|
||||
},
|
||||
"conflict_services": [
|
||||
"raspiBackup.service"
|
||||
],
|
||||
"conflict_timers": [
|
||||
"paperless-data-backup.timer",
|
||||
"borgmatic.timer"
|
||||
],
|
||||
"cron_users": [
|
||||
"root",
|
||||
"lars"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
phase,id,script,critical,timeout_seconds,enabled
|
||||
10,paperless-db,/etc/backup-orchestrator/scripts/db-paperless.sh,true,600,true
|
||||
10,tandoor-db,/etc/backup-orchestrator/scripts/db-tandoor.sh,true,600,true
|
||||
10,mitai-prod-db,/etc/backup-orchestrator/scripts/db-mitai-prod.sh,true,900,true
|
||||
10,mitai-dev-db,/etc/backup-orchestrator/scripts/db-mitai-dev.sh,false,900,true
|
||||
10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,600,true
|
||||
10,shinkan-dev-db,/etc/backup-orchestrator/scripts/db-shinkan-dev.sh,false,600,true
|
||||
20,nas-data,/etc/backup-orchestrator/scripts/nas-data-job.sh,true,10800,true
|
||||
30,system-image,/etc/backup-orchestrator/scripts/system-image-job.sh,true,14400,false
|
||||
10,kairo-prod-db,/etc/backup-orchestrator/scripts/db-kairo-prod.sh,true,900,true
|
||||
10,kairo-dev-db,/etc/backup-orchestrator/scripts/db-kairo-dev.sh,false,900,true
|
||||
10,tsumugu-prod-db,/etc/backup-orchestrator/scripts/db-tsumugu-prod.sh,true,900,true
|
||||
10,tsumugu-dev-db,/etc/backup-orchestrator/scripts/db-tsumugu-dev.sh,false,900,true
|
||||
10,kansho-prod-db,/etc/backup-orchestrator/scripts/db-kansho-prod.sh,true,900,true
|
||||
10,kansho-dev-db,/etc/backup-orchestrator/scripts/db-kansho-dev.sh,false,900,true
|
||||
10,buyingagent-prod-db,/etc/backup-orchestrator/scripts/db-buyingagent-prod.sh,true,900,true
|
||||
10,buyingagent-dev-db,/etc/backup-orchestrator/scripts/db-buyingagent-dev.sh,false,900,true
|
||||
10,netbox-db,/etc/backup-orchestrator/scripts/db-netbox.sh,true,900,true
|
||||
10,bookstack-db,/etc/backup-orchestrator/scripts/db-bookstack.sh,true,900,true
|
||||
10,karatewiki-dev-db,/etc/backup-orchestrator/scripts/db-karatewiki-dev.sh,false,900,true
|
||||
|
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"timeout_seconds": 3,
|
||||
"webhooks": [],
|
||||
"kuma": {
|
||||
"overall_push_url": "",
|
||||
"job_push_urls": {
|
||||
"nas-data": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"host": "raspberrypi5",
|
||||
"hosts": {
|
||||
"raspberrypi5": {
|
||||
"hostname": "raspberrypi5",
|
||||
"arch": "arm64",
|
||||
"nas_mount": "/backup",
|
||||
"nas_source": "192.168.2.63:/volume1/Backup_Paperless",
|
||||
"nas_fstype": "nfs4",
|
||||
"local_dump_dir": "/media/lars/paperless/db-backup",
|
||||
"archive_sets_dir": "/backup/data_archiv/last/paperless/db-backup"
|
||||
}
|
||||
},
|
||||
"targets": {
|
||||
"pi-prod": {
|
||||
"application": "shinkan",
|
||||
"environment": "prod",
|
||||
"host": "raspberrypi5",
|
||||
"role": "source_only",
|
||||
"allow_restore": false,
|
||||
"compose_dir": "/home/lars/docker/shinkan-prod",
|
||||
"compose_files": ["docker-compose.yml"],
|
||||
"env_file": "/home/lars/docker/shinkan-prod/.env",
|
||||
"media_dir": "/shinkan-media/prod",
|
||||
"db_container": "shinkan-db-prod",
|
||||
"db_name": "shinkan",
|
||||
"db_user": "shinkan",
|
||||
"health_url": "http://localhost:8003/api/health/ready"
|
||||
}
|
||||
}
|
||||
}
|
||||
+110
@@ -0,0 +1,110 @@
|
||||
#!/usr/bin/env bash
|
||||
# Host-aware installer for Backup-Orchestrator v0.4
|
||||
# Usage: sudo ./install.sh [--host <raspberrypi5|llm-node>] [--force-jobs] [--install-systemd]
|
||||
set -Eeuo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
HOST_NAME="$(hostname -s 2>/dev/null || hostname)"
|
||||
FORCE_JOBS=0
|
||||
INSTALL_SYSTEMD=0
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--host)
|
||||
HOST_NAME="$2"
|
||||
shift 2
|
||||
;;
|
||||
--force-jobs)
|
||||
FORCE_JOBS=1
|
||||
shift
|
||||
;;
|
||||
--install-systemd)
|
||||
INSTALL_SYSTEMD=1
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
echo "Usage: sudo $0 [--host <hostname>] [--force-jobs] [--install-systemd]"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown argument: $1" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
HOST_DIR="${REPO_ROOT}/hosts/${HOST_NAME}"
|
||||
if [[ ! -d "${HOST_DIR}" ]]; then
|
||||
echo "ERROR: Unknown host configuration '${HOST_NAME}'. Expected directory: ${HOST_DIR}" >&2
|
||||
echo "Available hosts:" >&2
|
||||
ls -1 "${REPO_ROOT}/hosts" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
echo "=== Installing Backup-Orchestrator v0.4 for host '${HOST_NAME}' ==="
|
||||
|
||||
# 1. Backup existing binary if present
|
||||
if [[ -f /usr/local/sbin/backup-orchestrator ]]; then
|
||||
cp -a /usr/local/sbin/backup-orchestrator "/usr/local/sbin/backup-orchestrator.bak.$(date +%Y%m%d%H%M%S)"
|
||||
fi
|
||||
install -m 0755 "${REPO_ROOT}/orchestrator.py" /usr/local/sbin/backup-orchestrator
|
||||
|
||||
# 2. Prepare /etc/backup-orchestrator directories
|
||||
install -d -m 0750 /etc/backup-orchestrator
|
||||
install -d -m 0750 /etc/backup-orchestrator/scripts
|
||||
install -d -m 0750 /etc/backup-orchestrator/scripts/shinkan
|
||||
install -d -m 0700 /etc/backup-orchestrator/credentials
|
||||
install -d -m 0750 /var/log/backup-orchestrator
|
||||
|
||||
# 3. Install host-specific metadata & target registry
|
||||
install -m 0640 "${HOST_DIR}/host.json" /etc/backup-orchestrator/host.json
|
||||
install -m 0640 "${HOST_DIR}/targets.json" /etc/backup-orchestrator/targets.json
|
||||
|
||||
if [[ ! -f /etc/backup-orchestrator/jobs.csv || "${FORCE_JOBS}" -eq 1 ]]; then
|
||||
install -m 0640 "${HOST_DIR}/jobs.csv" /etc/backup-orchestrator/jobs.csv
|
||||
echo "Installed /etc/backup-orchestrator/jobs.csv from hosts/${HOST_NAME}/jobs.csv"
|
||||
else
|
||||
echo "Keeping existing /etc/backup-orchestrator/jobs.csv (use --force-jobs to overwrite)"
|
||||
fi
|
||||
|
||||
if [[ ! -f /etc/backup-orchestrator/notifications.json ]]; then
|
||||
install -m 0600 "${HOST_DIR}/notifications.json.example" /etc/backup-orchestrator/notifications.json
|
||||
echo "Created initial /etc/backup-orchestrator/notifications.json (mode 0600)"
|
||||
fi
|
||||
|
||||
# 4. Install shared common scripts + host-specific scripts + Shinkan adapters
|
||||
for s in "${REPO_ROOT}/scripts/common/"*.sh; do
|
||||
[[ -f "$s" ]] && install -m 0750 "$s" "/etc/backup-orchestrator/scripts/$(basename "$s")"
|
||||
done
|
||||
|
||||
for s in "${HOST_DIR}/scripts/"*.sh; do
|
||||
[[ -f "$s" ]] && install -m 0750 "$s" "/etc/backup-orchestrator/scripts/$(basename "$s")"
|
||||
done
|
||||
|
||||
for s in "${REPO_ROOT}/scripts/shinkan/"*; do
|
||||
if [[ -f "$s" ]]; then
|
||||
case "$s" in
|
||||
*.yml|*.yaml|*.json)
|
||||
install -m 0644 "$s" "/etc/backup-orchestrator/scripts/shinkan/$(basename "$s")"
|
||||
;;
|
||||
*)
|
||||
install -m 0755 "$s" "/etc/backup-orchestrator/scripts/shinkan/$(basename "$s")"
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
done
|
||||
|
||||
# 5. Install daily wrapper & optional systemd units
|
||||
if [[ -f "${HOST_DIR}/systemd/backup-orchestrator-daily" ]]; then
|
||||
install -m 0755 "${HOST_DIR}/systemd/backup-orchestrator-daily" /usr/local/sbin/backup-orchestrator-daily
|
||||
fi
|
||||
|
||||
if [[ "${INSTALL_SYSTEMD}" -eq 1 ]]; then
|
||||
install -m 0644 "${HOST_DIR}/systemd/backup-orchestrator-daily.service" /etc/systemd/system/backup-orchestrator-daily.service
|
||||
install -m 0644 "${HOST_DIR}/systemd/backup-orchestrator-daily.timer" /etc/systemd/system/backup-orchestrator-daily.timer
|
||||
systemctl daemon-reload
|
||||
echo "Installed systemd service and timer for ${HOST_NAME}."
|
||||
fi
|
||||
|
||||
echo "=== Verification (--check) ==="
|
||||
/usr/local/sbin/backup-orchestrator --check
|
||||
+52
-12
@@ -66,8 +66,15 @@ def lock_exclusive(fp):
|
||||
fcntl.flock(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
|
||||
|
||||
def load_host_config(path=DEFAULT_HOST_CONFIG):
|
||||
def resolve_repo_host_dir(host_name=None):
|
||||
host = host_name or socket.gethostname().split('.')[0]
|
||||
candidate = Path(__file__).resolve().parent / 'hosts' / host
|
||||
return candidate if candidate.is_dir() else None
|
||||
|
||||
|
||||
def load_host_config(path=DEFAULT_HOST_CONFIG, host_name=None):
|
||||
default = {
|
||||
'hostname': host_name or socket.gethostname(),
|
||||
'nas_mount': NAS_MOUNT,
|
||||
'nas_source': NAS_SOURCE,
|
||||
'nas_fstype': 'nfs4',
|
||||
@@ -77,16 +84,37 @@ def load_host_config(path=DEFAULT_HOST_CONFIG):
|
||||
'conflict_timers': ['paperless-data-backup.timer', 'borgmatic.timer'],
|
||||
'cron_users': ['root', 'lars'],
|
||||
}
|
||||
if path and Path(path).is_file():
|
||||
with Path(path).open(encoding='utf-8') as fp:
|
||||
data = json.load(fp)
|
||||
if isinstance(data, dict):
|
||||
default.update(data)
|
||||
candidates = []
|
||||
if host_name:
|
||||
host_dir = resolve_repo_host_dir(host_name)
|
||||
if host_dir:
|
||||
candidates.append(host_dir / 'host.json')
|
||||
if path:
|
||||
candidates.append(Path(path))
|
||||
host_dir = resolve_repo_host_dir()
|
||||
if host_dir:
|
||||
candidates.append(host_dir / 'host.json')
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
with candidate.open(encoding='utf-8') as fp:
|
||||
data = json.load(fp)
|
||||
if isinstance(data, dict):
|
||||
default.update(data)
|
||||
break
|
||||
return default
|
||||
|
||||
|
||||
def load_targets_config(path=DEFAULT_TARGETS_CONFIG):
|
||||
candidates = [Path(path)] if path else []
|
||||
def load_targets_config(path=DEFAULT_TARGETS_CONFIG, host_name=None):
|
||||
candidates = []
|
||||
if host_name:
|
||||
host_dir = resolve_repo_host_dir(host_name)
|
||||
if host_dir:
|
||||
candidates.append(host_dir / 'targets.json')
|
||||
if path:
|
||||
candidates.append(Path(path))
|
||||
host_dir = resolve_repo_host_dir()
|
||||
if host_dir:
|
||||
candidates.append(host_dir / 'targets.json')
|
||||
candidates.append(Path(__file__).resolve().parent / 'targets.json')
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
@@ -1396,6 +1424,7 @@ def main():
|
||||
mode.add_argument('--restore-test', action='store_true', help='Execute isolated disposable restore test')
|
||||
|
||||
parser.add_argument('--json', action='store_true', help='Machine-readable JSON output')
|
||||
parser.add_argument('--host', type=str, default=None, help='Select server profile from hosts/<host> (e.g. raspberrypi5, llm-node)')
|
||||
parser.add_argument('--config', type=Path, default=DEFAULT_CONFIG)
|
||||
parser.add_argument('--notify-config', type=Path, default=DEFAULT_NOTIFY_CONFIG)
|
||||
parser.add_argument('--targets-config', type=Path, default=DEFAULT_TARGETS_CONFIG)
|
||||
@@ -1419,6 +1448,17 @@ def main():
|
||||
parser.add_argument('--keep', action='store_true', help='Keep isolated test environment running after --restore-test')
|
||||
|
||||
args = parser.parse_args()
|
||||
if args.host:
|
||||
host_dir = resolve_repo_host_dir(args.host)
|
||||
if not host_dir:
|
||||
parser.error(f'Unknown host profile: {args.host!r}')
|
||||
if args.config == DEFAULT_CONFIG:
|
||||
args.config = host_dir / 'jobs.csv'
|
||||
if args.notify_config == DEFAULT_NOTIFY_CONFIG and (host_dir / 'notifications.json.example').is_file():
|
||||
args.notify_config = host_dir / 'notifications.json.example'
|
||||
if args.targets_config == DEFAULT_TARGETS_CONFIG and (host_dir / 'targets.json').is_file():
|
||||
args.targets_config = host_dir / 'targets.json'
|
||||
|
||||
if not 1 <= args.max_parallel <= 16:
|
||||
parser.error('--max-parallel must be 1..16')
|
||||
json_capable = (
|
||||
@@ -1434,7 +1474,7 @@ def main():
|
||||
return show_status(args.log_root, as_json=args.json)
|
||||
|
||||
if args.list_recovery_points:
|
||||
host_cfg = load_host_config()
|
||||
host_cfg = load_host_config(host_name=args.host)
|
||||
search_dirs = args.search_dirs or [
|
||||
host_cfg['archive_sets_dir'],
|
||||
host_cfg['local_dump_dir'],
|
||||
@@ -1472,7 +1512,7 @@ def main():
|
||||
if args.recovery_plan:
|
||||
if not args.point or not args.target:
|
||||
parser.error('--point and --target are required with --recovery-plan')
|
||||
targets_cfg = load_targets_config(args.targets_config)
|
||||
targets_cfg = load_targets_config(args.targets_config, host_name=args.host)
|
||||
plan_doc = generate_recovery_plan(args.point, args.target, targets_cfg, sessions_mode=args.sessions)
|
||||
if args.json:
|
||||
print(json.dumps(plan_doc, ensure_ascii=False, indent=2))
|
||||
@@ -1492,11 +1532,11 @@ def main():
|
||||
return plan_doc['compatibility']['exit_code']
|
||||
|
||||
if args.restore:
|
||||
targets_cfg = load_targets_config(args.targets_config)
|
||||
targets_cfg = load_targets_config(args.targets_config, host_name=args.host)
|
||||
return execute_recovery_command('restore', args, targets_cfg)
|
||||
|
||||
if args.restore_test:
|
||||
targets_cfg = load_targets_config(args.targets_config)
|
||||
targets_cfg = load_targets_config(args.targets_config, host_name=args.host)
|
||||
return execute_recovery_command('restore-test', args, targets_cfg)
|
||||
|
||||
jobs = load_jobs(args.config)
|
||||
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -239,6 +239,35 @@ class RecoveryPointAndManifestTests(unittest.TestCase):
|
||||
self.assertTrue((out_dir / 'shinkan-prod.sql.sha256').is_file())
|
||||
self.assertTrue((out_dir / 'shinkan-prod.sql.media.jsonl').is_file())
|
||||
|
||||
def test_host_profiles_raspberrypi5_and_llm_node(self):
|
||||
repo_root = Path(__file__).resolve().parents[1]
|
||||
for host_name, expected_jobs in (('raspberrypi5', 19), ('llm-node', 1)):
|
||||
host_dir = app.resolve_repo_host_dir(host_name)
|
||||
self.assertIsNotNone(host_dir, f'Missing host dir for {host_name}')
|
||||
host_cfg = app.load_host_config(host_name=host_name)
|
||||
self.assertEqual(host_cfg['hostname'], host_name)
|
||||
jobs = app.load_jobs(host_dir / 'jobs.csv')
|
||||
self.assertEqual(len(jobs), expected_jobs)
|
||||
notify = app.load_notifications(
|
||||
host_dir / 'notifications.json.example',
|
||||
known_ids=[j['id'] for j in jobs],
|
||||
)
|
||||
self.assertIn('kuma', notify)
|
||||
targets = app.load_targets_config(host_name=host_name)
|
||||
self.assertEqual(targets['host'], host_name)
|
||||
self.assertIn('pi-prod', targets['targets'])
|
||||
# Verify every job script exists in either hosts/<host>/scripts/ or scripts/common/
|
||||
for job in jobs:
|
||||
script_name = Path(job['script']).name
|
||||
if job['id'] == 'system-image':
|
||||
continue
|
||||
exists_in_host = (host_dir / 'scripts' / script_name).is_file()
|
||||
exists_in_common = (repo_root / 'scripts' / 'common' / script_name).is_file()
|
||||
self.assertTrue(
|
||||
exists_in_host or exists_in_common,
|
||||
f"Missing script {script_name} for job {job['id']} on host {host_name}",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user