feat(v0.4): import all Raspberry Pi scripts and implement Recovery Management engine + Shinkan adapters
This commit is contained in:
1 parent
40188c0e10
commit
c53fc0a939
35 files changed
+2877
-191
No files matched your search
@@ -4,3 +4,4 @@
|
||||
*.csv text eol=lf
|
||||
*.json text eol=lf
|
||||
*.md text eol=lf
|
||||
|
||||
@@ -8,3 +8,4 @@ __pycache__/
|
||||
notifications.json
|
||||
host.local.json
|
||||
.env
|
||||
|
||||
@@ -1,161 +1,93 @@
|
||||
# Raspberry Pi 5 – Backup-Orchestrator v0.3
|
||||
# Backup-Orchestrator v0.4 – Backup & Recovery Management
|
||||
|
||||
**Status: Test- und Integrationsversion.** Keine produktiven Backups, Timer, Crontabs oder Datenbanken werden durch Installation, `--check`, `--plan` oder `--status` verändert. Die vorhandenen Backup-Skripte sind **noch nicht vollständig standardisiert**, zusätzliche Datenbank-Jobs fehlen und eine vollständige Systemwiederherstellung ist noch nicht getestet. `--run` bleibt bewusst mehrfach gesperrt.
|
||||
**Status: Version 0.4.0 (Multi-Host Backup & Application Recovery Management).**
|
||||
Erweitert den auf `raspberrypi5` produktiv laufenden Backup-Orchestrator v0.3 um ein integriertes Recovery Management, kryptografisch geprüfte Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Konflikt-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für die Referenzanwendung **Shinkan** auf `llm-node` (`192.168.2.144`).
|
||||
|
||||
## 1. Enthaltene Dateien
|
||||
---
|
||||
|
||||
- `orchestrator.py`: phasenbasierter Python-Controller (Python 3.11 / Debian 12, nur Standardbibliothek).
|
||||
- `jobs.csv`: bestehende acht Jobs. **Alle `enabled=false`** und verweisen derzeit auf die alten, vorhandenen Skriptpfade.
|
||||
- `notifications.json.example`: eine zentrale Konfigurationsdatei für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore; keine echten Secrets.
|
||||
- `tests/test_orchestrator.py`: Tests mit künstlichen Shellskripten und simulierten HTTP-Antworten; niemals produktive Datenbanken oder das NAS.
|
||||
## 1. Repository-Struktur & Enthaltene Dateien
|
||||
|
||||
## 2. Sichere Installation auf raspberrypi5
|
||||
- `orchestrator.py`: Phasenbasierter Backup- & Recovery-Controller (`v0.4.0`, Python 3.11+, nur Standardbibliothek).
|
||||
- `targets.json`: Multi-Host- und Recovery-Target-Definitionen (`raspberrypi5`, `llm-node`, Targets `pi-prod`, `llm-prod`, `llm-restoretest`).
|
||||
- `jobs.csv`: Produktive Backup-Job-Tabelle (17 Datenbank-Exporte in Phase 10, `nas-data` in Phase 20, `system-image` in Phase 30).
|
||||
- `notifications.json.example`: Beispielkonfiguration für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore (enthält keine echten Secrets).
|
||||
- `systemd/`:
|
||||
- `backup-orchestrator-daily`: Täglicher Runner-Wrapper (ab v0.4 ohne fragiles `systemctl disable/enable raspiBackup.timer`).
|
||||
- `backup-orchestrator-daily.service` & `backup-orchestrator-daily.timer`: Systemd-Units für den täglichen Lauf um `05:30` Uhr.
|
||||
- `scripts/`:
|
||||
- Alle 17 Datenbank-Dump-Wrapper (`db-bookstack.sh`, `db-buyingagent-dev.sh`, `db-buyingagent-prod.sh`, `db-kairo-dev.sh`, `db-kairo-prod.sh`, `db-kansho-dev.sh`, `db-kansho-prod.sh`, `db-karatewiki-dev.sh`, `db-mitai-dev.sh`, `db-mitai-prod.sh`, `db-netbox.sh`, `db-paperless.sh`, `db-shinkan-dev.sh`, `db-shinkan-prod.sh`, `db-tandoor.sh`, `db-tandoor.sh`, `db-tsumugu-dev.sh`, `db-tsumugu-prod.sh`) sowie die generischen Dump-Engines `db-postgres-docker.sh` und `db-mariadb-docker.sh`.
|
||||
- `backup-nas-data.sh` & `nas-data-job.sh`: Versionierte rsync-Hardlink-Sicherung (`/media/lars/paperless` → `/backup/data_archiv/01..31` + `last`).
|
||||
- `publish-backup-set.sh` & `shinkan-prod-manifest.sh`: Atomare Erzeugung und Publikation vollständiger Backup-Sets (`<dump>.sql`, `<dump>.sql.sha256`, `<dump>.sql.manifest.json`, `<dump>.sql.media.jsonl`).
|
||||
- `scripts/shinkan/` (Shinkan Recovery Contract Adapter):
|
||||
- `shinkan-backup-manifest`: Erzeugt SHA256, Manifest (`schemaVersion: 1` + Shinkan Contract) und inkrementelles Media-Inventar (`media.jsonl`).
|
||||
- `shinkan-restore`: 9-stufiger atomarer Application-Restore über Staging-DB (`shinkan_restore_<ts>`), Single-Transaction-Import, Schema-Migrations-Gate, Plausibilitätsprüfung, `TRUNCATE sessions`, DB-Swap (`shinkan_before_<ts>`), `/api/health/ready`-Prüfung und automatischem Rollback bei Fehlern (Exit `5`).
|
||||
- `shinkan-media-diff`: Abgleich zwischen PostgreSQL (`media_assets`, `exercise_media`), Dateisystem und `media.jsonl` (`missing`, `mismatch`, `added_after_backup`, `orphan`).
|
||||
- `shinkan-restore-test` & `docker-compose.restoretest.yml`: Isolierter End-to-End-Restore-Test im Projekt `shinkan-restoretest` (Ports `127.0.0.1:18003` / `127.0.0.1:13003`, Read-Only-Media `:ro`, deaktivierte externe Integrationen & Cronjobs).
|
||||
- `docs/ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md`: Vollständige Ist-Inventur, Gap-Analyse und Architektur-Spezifikation für v0.4.
|
||||
- `tests/`:
|
||||
- `test_orchestrator.py`: Unit-Tests für Backup-Phasen, Parallelität, Timeouts, Mount-Prüfung und Notifications.
|
||||
- `test_recovery.py`: Unit- und Integrationstests für Manifest-Validierung, 4-Stufen-Modell, Schema-Kompatibilitätsprüfung (`EXACT_MATCH`, `FORWARD_MIGRATE`, `INCOMPATIBLE_ROLLBACK`), Sicherheits-Gates (`pi-prod` Sperre, `--confirm`), Dry-Run-Pläne und `shinkan-backup-manifest`.
|
||||
|
||||
ZIP auf den Raspberry Pi übertragen und dort entpacken. Im entpackten Verzeichnis:
|
||||
---
|
||||
|
||||
## 2. CLI-Befehle (v0.4.0)
|
||||
|
||||
### A. Backup-Steuerung (unverändert kompatibel zu v0.3)
|
||||
|
||||
```bash
|
||||
backup-orchestrator --check
|
||||
backup-orchestrator --plan
|
||||
backup-orchestrator --status --json
|
||||
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES backup-orchestrator --run
|
||||
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES backup-orchestrator --run --phase 10
|
||||
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES backup-orchestrator --run --job shinkan-prod-db
|
||||
```
|
||||
|
||||
### B. Recovery Management (neu in v0.4)
|
||||
|
||||
```bash
|
||||
# 1. Verfügbare Recovery Points auflisten (inkl. 4-Stufen-Status)
|
||||
backup-orchestrator --list-recovery-points
|
||||
backup-orchestrator --list-recovery-points --app shinkan --json
|
||||
|
||||
# 2. Einzelnen Recovery Point kryptografisch & strukturell prüfen
|
||||
backup-orchestrator --verify-recovery-point \
|
||||
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql
|
||||
|
||||
# 3. Dry-Run Recovery-Plan erstellen (inkl. Schema-Kompatibilitätsprüfung)
|
||||
backup-orchestrator --recovery-plan \
|
||||
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql \
|
||||
--target llm-prod
|
||||
|
||||
# 4. Isolierten Restore-Test ausführen (schreibgeschütztes Media-Mount, Ports 18003/13003)
|
||||
backup-orchestrator --restore-test \
|
||||
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql \
|
||||
--target llm-restoretest
|
||||
|
||||
# 5. Produktiver Application-Restore / Erstbefüllung auf llm-node
|
||||
backup-orchestrator --restore \
|
||||
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql \
|
||||
--target llm-prod \
|
||||
--confirm shinkan@llm-node
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Exit-Codes im Recovery-Modus
|
||||
|
||||
| Exit-Code | Bedeutung |
|
||||
|---|---|
|
||||
| `0` | Erfolgreich (`verified`/`consistent`/`restore_tested` bzw. Restore + `/api/health/ready` grün) |
|
||||
| `2` | Ungültige Argumente, fehlendes `--confirm`, gesperrtes Ziel (`pi-prod` ist `source_only`) oder Mount/Pfad fehlt |
|
||||
| `3` | Schema-/Versions-Inkompatibilität (`INCOMPATIBLE_ROLLBACK`: Dump enthält Migrationen, die im Ziel-Code fehlen) |
|
||||
| `4` | Integritätsfehler (SHA256-Prüfsumme weicht ab, Dump leer/korrupt oder Pflicht-Tabellen nach Staging-Import leer) |
|
||||
| `5` | Health-Check (`/api/health/ready`) nach DB-Umschaltung fehlgeschlagen – **automatisches Rollback auf `shinkan_before_<ts>` durchgeführt** |
|
||||
|
||||
---
|
||||
|
||||
## 4. Tests ausführen
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s tests -v
|
||||
sudo cp -a /usr/local/sbin/backup-orchestrator /usr/local/sbin/backup-orchestrator.before-v03
|
||||
sudo install -m 755 orchestrator.py /usr/local/sbin/backup-orchestrator
|
||||
|
||||
# Die vorhandene /etc/backup-orchestrator/jobs.csv NICHT überschreiben.
|
||||
# Die Benachrichtigungskonfiguration nur bei erstmaliger Einrichtung kopieren.
|
||||
sudo install -d -m 750 /etc/backup-orchestrator
|
||||
if ! sudo test -e /etc/backup-orchestrator/notifications.json; then
|
||||
sudo install -m 600 notifications.json.example /etc/backup-orchestrator/notifications.json
|
||||
fi
|
||||
sudo backup-orchestrator --check
|
||||
```
|
||||
|
||||
Ohne bereits installierte ältere Version entfällt die Zeile `sudo cp -a ...before-v03`.
|
||||
|
||||
> **Keine `--run`-Befehle ausführen**, solange die alten Cronjobs und der raspiBackup-Timer aktiv sind und die alten Dump-Skripte noch nicht gehärtet wurden. Den Produktiv-Cutover führen wir später kontrolliert durch.
|
||||
|
||||
## 3. Konfiguration der Phasen
|
||||
|
||||
Die vorhandene Tabelle unter `/etc/backup-orchestrator/jobs.csv` hat **keine Uhrzeiten**:
|
||||
|
||||
```csv
|
||||
phase,id,script,critical,timeout_seconds,enabled
|
||||
10,paperless-db,/home/lars/db-backup.sh,true,600,false
|
||||
10,tandoor-db,/home/lars/db-Tandoor.sh,true,600,false
|
||||
20,nas-data,/home/lars/backupPaperlessData.sh,true,10800,false
|
||||
30,system-image,/usr/local/bin/raspiBackup.sh,true,14400,false
|
||||
```
|
||||
|
||||
Die Beispiele sind ein Ausschnitt; die mitgelieferte Datei enthält acht Jobs.
|
||||
|
||||
- **Innerhalb derselben Phase** laufen bis zu drei Jobs gleichzeitig (per `--max-parallel N` einstellbar).
|
||||
- **Zwischen Phasen** gilt eine Barriere: Die nächste Phase beginnt erst, wenn die aktuelle vollständig abgeschlossen ist.
|
||||
- Ein **kritischer Fehler** blockiert nachfolgende Phasen; laufende Jobs derselben Phase werden beendet bzw. bis zum Abschluss abgewartet.
|
||||
- Ein **unkritischer Fehler** wird protokolliert und führt zu einem fehlgeschlagenen Gesamtergebnis, blockiert aber die nächste Phase nicht.
|
||||
- Zeitüberschreitung beendet die Skript-Prozessgruppe; Service-Neustart und Datenbankkonsistenz müssen in den jeweiligen Jobs abgesichert sein.
|
||||
|
||||
Zentrale Zeitsteuerung per *einem* systemd-Timer ist eine spätere Cutover-Aufgabe, **nicht Bestandteil der Installation**.
|
||||
|
||||
## 4. Externe Schnittstellen
|
||||
|
||||
### A. Ausgehende Webhooks, HTTP POST
|
||||
|
||||
`/etc/backup-orchestrator/notifications.json` bearbeiten:
|
||||
|
||||
```json
|
||||
{
|
||||
"timeout_seconds": 3,
|
||||
"webhooks": ["http://192.168.2.100:5678/webhook/backup-events"],
|
||||
"kuma": {
|
||||
"overall_push_url": "http://192.168.2.63:3010/api/push/GESAMT_MONITOR_TOKEN",
|
||||
"job_push_urls": {
|
||||
"nas-data": "http://192.168.2.63:3010/api/push/NAS_MONITOR_TOKEN",
|
||||
"paperless-db": "http://192.168.2.63:3010/api/push/PAPERLESS_MONITOR_TOKEN"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Alle URLs sind Beispiele – vorhandene echte Push-Tokens nur auf dem Pi eintragen. Einträge mit `""` sind deaktiviert. Es gibt **keinen HTTP-Listener und keinen permanent laufenden Webserver**; Webhooks sind *ausgehend*.
|
||||
|
||||
Die echten URLs können auch komplett aus der Uptime-Kuma-Oberfläche kopiert werden (`...?status=up&msg=OK&ping=`). Diese Beispiel-Query wird beim Laden entfernt und beim Push durch den tatsächlichen Status ersetzt.
|
||||
|
||||
Diese Datei enthält Zugriffstokens: `sudo chmod 600 /etc/backup-orchestrator/notifications.json`. Sie gehört in eine vertrauliche Konfigurationssicherung, nicht in ein öffentliches Repository.
|
||||
|
||||
### B. Ereignisse und Beispiele
|
||||
|
||||
Alle Ereignisse werden als **JSON je Zeile** in `/var/log/backup-orchestrator/<run_id>/events.jsonl` geschrieben; jede Zeile wird auch an die konfigurierten POST-Webhooks gesendet.
|
||||
|
||||
- `RUN_STARTED`, `PHASE_STARTED`, `JOB_STARTED`
|
||||
- `JOB_SUCCESS`, `JOB_FAILED`, `JOB_TIMEOUT`, `JOB_ERROR`, `JOB_CANCELLED`, `JOB_BLOCKED`
|
||||
- `PHASE_COMPLETED`, `PHASE_BLOCKED`, `RUN_FINISHED`
|
||||
|
||||
Beispiel einer Webhook-Nachricht (ohne Geheimnisse):
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "JOB_FAILED",
|
||||
"timestamp": "2026-10-29T01:02:00+01:00",
|
||||
"run_id": "20261029-010000-1234",
|
||||
"host": "raspberrypi5",
|
||||
"phase": 10,
|
||||
"job": "shinkan-prod-db",
|
||||
"critical": true,
|
||||
"state": "failed",
|
||||
"exit_code": 1,
|
||||
"duration_seconds": 12.3,
|
||||
"log": "/var/log/backup-orchestrator/20261029-010000-1234/shinkan-prod-db.log"
|
||||
}
|
||||
```
|
||||
|
||||
Webhooks versenden **Ereignismetadaten**, nicht den Inhalt der Backup-Skripte oder SQL-Dumps.
|
||||
|
||||
### C. Uptime Kuma
|
||||
|
||||
- `JOB_SUCCESS` → `status=up` für den konfigurierten Job-Monitor.
|
||||
- `JOB_FAILED`, `JOB_TIMEOUT`, `JOB_ERROR`, `JOB_CANCELLED` und `JOB_BLOCKED` → `status=down`.
|
||||
- `RUN_FINISHED` → `status=up` nur beim Gesamtergebnis `ok`, sonst `down`.
|
||||
- Ausbleibende Backups meldet Kuma über den konfigurierten Push-Heartbeat, z.B. bei einem täglichen Lauf ein zum Zeitplan passender Wert.
|
||||
- Schlägt die Zustellung fehl, schreibt der Controller eine Warnung in `notifications.log`. Der Backup-Exit-Code bleibt unverändert.
|
||||
|
||||
**Keine doppelten Push-Meldungen erzeugen:** Wenn `nas-data` künftig durch den Orchestrator gemeldet wird, die Push-Funktion im Skript `backupPaperlessData.sh` deaktivieren oder nur einen der beiden Sender verwenden.
|
||||
|
||||
### D. Lokaler Status, JSON-Abfrage
|
||||
|
||||
Während des Laufs wird der aktuelle Stand atomar aktualisiert:
|
||||
|
||||
```text
|
||||
/var/log/backup-orchestrator/latest.json
|
||||
/var/log/backup-orchestrator/<run_id>/status.json
|
||||
/var/log/backup-orchestrator/<run_id>/summary.json
|
||||
/var/log/backup-orchestrator/<run_id>/events.jsonl
|
||||
```
|
||||
|
||||
CLI für andere Skripte, Home Assistant oder eine spätere GUI:
|
||||
|
||||
```bash
|
||||
sudo backup-orchestrator --status
|
||||
sudo backup-orchestrator --status --json
|
||||
sudo cat /var/log/backup-orchestrator/latest.json
|
||||
```
|
||||
|
||||
Die Statusdateien enthalten keine Push-Tokens oder Zugangsdaten, aber interne Pfade und Jobnamen. Daher nur für vertrauenswürdige lokale Nutzer freigeben. Es gibt derzeit **keine eingehende REST-API**; dies kann später als separater schreibgeschützter Dienst ergänzt werden.
|
||||
|
||||
## 5. Sichere Bedienung
|
||||
|
||||
```bash
|
||||
sudo backup-orchestrator --check # nur Prüfungen, keine Backups
|
||||
sudo backup-orchestrator --plan # nur aktivierte Jobs anzeigen
|
||||
sudo backup-orchestrator --status --json # letzter Status bzw. no_runs
|
||||
```
|
||||
|
||||
`--run`, `--run --phase 10` und `--run --job paperless-db` sind für einen **späteren**, explizit kontrollierten Produktiv-Cutover vorgesehen. Der Controller verweigert Ausführung, solange keine Jobs aktiviert sind, der NAS-Mount fehlt, aktive alte Zeitpläne gefunden werden oder `BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES` fehlt. Das stellt keine vollständige Sicherheitsfreigabe dar: Skripte und Restore müssen zusätzlich getestet werden.
|
||||
|
||||
## 6. Offene Arbeiten für den Produktivbetrieb
|
||||
|
||||
1. Vorhandene Datenbank-Dump-Skripte unter `scripts/` konsolidieren und auf atomare Dumps unter festen Dateinamen umstellen; Zugänge sicher verwahren.
|
||||
2. Weitere Datenbanken (Kairo, Tsumugu, Kanshō, Buying Agent, BookStack, NetBox usw.) und zu sichernde Docker-Volumes/Bind-Mounts aufnehmen.
|
||||
3. rsync-Skript prüfen: nas-data sendet bei der Umstellung **nicht** gleichzeitig selbst und über Orchestrator Pushs.
|
||||
4. raspiBackup: zuverlässiger vollständiger Restore, korrekt gestoppte Docker-Dienste, Restart-/Timeout-Verhalten nach Fehler.
|
||||
5. Alte Cronjobs/Timer **erst beim kontrollierten Cutover** ersetzen und einen einzigen orchestrator-Timer installieren.
|
||||
6. Vollständigen Testlauf, Restore-Test und erwartete Kuma-Ereignisse prüfen.
|
||||
7. Zweiten unabhängigen/offsite Backup-Standort erwägen; NAS-Ausfall bleibt sonst Single Point of Failure.
|
||||
|
||||
**Hinweis zur bisherigen Zeitplanung:** Die Root-Crontab-Einträge sind nach der letzten Nutzerprüfung auskommentiert; die aktive User-Crontab sowie `raspiBackup.timer` werden weiterhin als aktive Konflikte erkannt.
|
||||
Alle Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und berühren niemals produktive Datenbanken oder NAS-Mounts.
|
||||
+954
-37
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
exec /etc/backup-orchestrator/scripts/db-mariadb-docker.sh \
|
||||
"bookstack_db" "bookstack.sql" "mysql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "buyingagent-dev-postgres-1" "buyingagent-dev.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "buyingagent-db-prod" "buyingagent-prod.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kairo-dev-postgres-1" "kairo-dev.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kairo-db-prod" "kairo-prod.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kansho-dev-postgres-1" "kansho-dev.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kansho-db-prod" "kansho-prod.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
exec /etc/backup-orchestrator/scripts/db-mariadb-docker.sh \
|
||||
"karatewiki-dev-db-1" "karatewiki-dev.sql" "mariadb"
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
|
||||
"dev-mitai-postgres" \
|
||||
"mitai-dev.sql" \
|
||||
"mitai_dev" \
|
||||
"mitai_dev"
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
|
||||
"mitai-db-prod" \
|
||||
"mitai-prod.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
|
||||
"netbox-postgres-1" "netbox.sql"
|
||||
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
# Optionales lokales Credential-File (chmod 600, nicht in Git)
|
||||
if [[ -f /etc/backup-orchestrator/credentials/paperless.env ]]; then
|
||||
# shellcheck disable=SC1091
|
||||
source /etc/backup-orchestrator/credentials/paperless.env
|
||||
fi
|
||||
|
||||
# === Datenbank ===
|
||||
DB_HOST="${DB_HOST:-192.168.2.40}"
|
||||
DB_PORT="${DB_PORT:-3308}"
|
||||
DB_USER="${DB_USER:-paperless}"
|
||||
DB_PASSWORD="${DB_PASSWORD:-HIER_PASSWORT_EINTRAGEN}"
|
||||
DB_NAME="${DB_NAME:-paperless}"
|
||||
|
||||
# === Backup ===
|
||||
BACKUP_DIR="${BACKUP_DIR:-/media/lars/paperless/db-backup}"
|
||||
BACKUP_FILE="$BACKUP_DIR/paperless.sql"
|
||||
|
||||
[[ "$DB_PASSWORD" != "HIER_PASSWORT_EINTRAGEN" ]] || {
|
||||
echo "FEHLER: Datenbankpasswort nicht konfiguriert" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
TEMP_FILE="$(mktemp "$BACKUP_DIR/.paperless.XXXXXXXX.sql")"
|
||||
trap 'rm -f -- "$TEMP_FILE"' EXIT
|
||||
|
||||
echo "Paperless: Datenbankexport gestartet"
|
||||
|
||||
MYSQL_PWD="$DB_PASSWORD" mysqldump \
|
||||
--host="$DB_HOST" \
|
||||
--port="$DB_PORT" \
|
||||
--user="$DB_USER" \
|
||||
--single-transaction \
|
||||
--quick \
|
||||
--databases "$DB_NAME" \
|
||||
> "$TEMP_FILE"
|
||||
|
||||
if [[ ! -s "$TEMP_FILE" ]]; then
|
||||
echo "FEHLER: Paperless-Dump ist leer" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
chgrp lars "$TEMP_FILE"
|
||||
chmod 640 "$TEMP_FILE"
|
||||
mv -f -- "$TEMP_FILE" "$BACKUP_FILE"
|
||||
|
||||
echo "OK: Paperless-Datenbank gesichert"
|
||||
echo "Datei: $BACKUP_FILE"
|
||||
exit 0
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
|
||||
"shinkan-dev-postgres-1" \
|
||||
"shinkan-dev.sql"
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
|
||||
"shinkan-db-prod" \
|
||||
"shinkan-prod.sql"
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
# Optionales lokales Credential-File (chmod 600, nicht in Git)
|
||||
if [[ -f /etc/backup-orchestrator/credentials/tandoor.env ]]; then
|
||||
# shellcheck disable=SC1091
|
||||
source /etc/backup-orchestrator/credentials/tandoor.env
|
||||
fi
|
||||
|
||||
# === Datenbank ===
|
||||
PGHOST="${PGHOST:-192.168.2.40}"
|
||||
PGPORT="${PGPORT:-5432}"
|
||||
PGUSER="${PGUSER:-djangouser}"
|
||||
PGDATABASE="${PGDATABASE:-djangodb}"
|
||||
PGPASSWORD="${PGPASSWORD:-HIER_PASSWORT_EINTRAGEN}"
|
||||
|
||||
# === Backup ===
|
||||
BACKUP_DIR="${BACKUP_DIR:-/media/lars/paperless/db-backup}"
|
||||
BACKUP_FILE="$BACKUP_DIR/tandoor.sql"
|
||||
|
||||
[[ "$PGPASSWORD" != "HIER_PASSWORT_EINTRAGEN" ]] || {
|
||||
echo "FEHLER: Datenbankpasswort nicht konfiguriert" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
TEMP_FILE="$(mktemp "$BACKUP_DIR/.tandoor.XXXXXXXX.sql")"
|
||||
trap 'rm -f -- "$TEMP_FILE"' EXIT
|
||||
|
||||
echo "Tandoor: Datenbankexport gestartet"
|
||||
|
||||
PGPASSWORD="$PGPASSWORD" pg_dump \
|
||||
--host="$PGHOST" \
|
||||
--port="$PGPORT" \
|
||||
--username="$PGUSER" \
|
||||
--dbname="$PGDATABASE" \
|
||||
--format=plain \
|
||||
> "$TEMP_FILE"
|
||||
|
||||
if [[ ! -s "$TEMP_FILE" ]]; then
|
||||
echo "FEHLER: Tandoor-Dump ist leer" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
chgrp lars "$TEMP_FILE"
|
||||
chmod 640 "$TEMP_FILE"
|
||||
mv -f -- "$TEMP_FILE" "$BACKUP_FILE"
|
||||
|
||||
echo "OK: Tandoor-Datenbank gesichert"
|
||||
echo "Datei: $BACKUP_FILE"
|
||||
exit 0
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "tsumugu-dev-postgres-1" "tsumugu-dev.sql"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "tsumugu-db-prod" "tsumugu-prod.sql"
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
exec runuser -u lars -- /bin/bash -s \
|
||||
< /etc/backup-orchestrator/scripts/backup-nas-data.sh
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================================
|
||||
# publish-backup-set.sh
|
||||
# Atomically publishes a complete Backup Set (Dump + .sha256 + .manifest.json
|
||||
# + optional .media.jsonl) to the NAS generation archive and rotates old sets.
|
||||
# Implements RECOVERY_CONTRACT.md section 5.1 & 5.2.
|
||||
# ============================================================================
|
||||
set -Eeuo pipefail
|
||||
umask 027
|
||||
|
||||
if [[ $# -lt 3 ]]; then
|
||||
echo "Verwendung: $0 <TARGET_ID> <DUMP_DATEI> <ARCHIVE_ROOT>" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
TARGET_ID="$1"
|
||||
DUMP_FILE="$2"
|
||||
ARCHIVE_ROOT="$3"
|
||||
|
||||
SHA_FILE="${DUMP_FILE}.sha256"
|
||||
MANIFEST_FILE="${DUMP_FILE}.manifest.json"
|
||||
MEDIA_FILE="${DUMP_FILE}.media.jsonl"
|
||||
|
||||
[[ -s "$DUMP_FILE" ]] || { echo "FEHLER: Dump fehlt oder ist leer: $DUMP_FILE" >&2; exit 2; }
|
||||
[[ -s "$SHA_FILE" ]] || { echo "FEHLER: SHA256-Datei fehlt: $SHA_FILE" >&2; exit 2; }
|
||||
[[ -s "$MANIFEST_FILE" ]] || { echo "FEHLER: Manifest-Datei fehlt: $MANIFEST_FILE" >&2; exit 2; }
|
||||
|
||||
# Verify checksum before publishing
|
||||
(
|
||||
cd "$(dirname "$DUMP_FILE")"
|
||||
sha256sum -c "$(basename "$SHA_FILE")" >/dev/null
|
||||
) || {
|
||||
echo "FEHLER: SHA256-Pruefsumme stimmt nicht!" >&2
|
||||
exit 4
|
||||
}
|
||||
|
||||
TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
|
||||
TARGET_DIR="$ARCHIVE_ROOT/$TARGET_ID"
|
||||
mkdir -p "$TARGET_DIR"
|
||||
|
||||
STAGING_DIR="$(mktemp -d "$TARGET_DIR/.staging-${TIMESTAMP}.XXXXXXXX")"
|
||||
FINAL_DIR="$TARGET_DIR/$TIMESTAMP"
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "${STAGING_DIR:-}" && -d "$STAGING_DIR" ]]; then
|
||||
rm -rf -- "$STAGING_DIR" || true
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
cp -p -- "$DUMP_FILE" "$STAGING_DIR/"
|
||||
cp -p -- "$SHA_FILE" "$STAGING_DIR/"
|
||||
cp -p -- "$MANIFEST_FILE" "$STAGING_DIR/"
|
||||
if [[ -f "$MEDIA_FILE" ]]; then
|
||||
cp -p -- "$MEDIA_FILE" "$STAGING_DIR/"
|
||||
fi
|
||||
|
||||
mv -T -- "$STAGING_DIR" "$FINAL_DIR"
|
||||
STAGING_DIR=""
|
||||
|
||||
TEMP_LINK="$TARGET_DIR/.latest-$$"
|
||||
ln -s -- "$FINAL_DIR" "$TEMP_LINK"
|
||||
mv -Tf -- "$TEMP_LINK" "$TARGET_DIR/latest"
|
||||
|
||||
echo "OK: Backup-Satz veroeffentlicht unter $FINAL_DIR (latest aktualisiert)"
|
||||
exit 0
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
DUMP_FILE="${SHINKAN_DUMP_FILE:-/media/lars/paperless/db-backup/shinkan-prod.sql}"
|
||||
OUT_DIR="$(dirname "$DUMP_FILE")"
|
||||
TARGET="${SHINKAN_TARGET:-pi-prod}"
|
||||
ARCHIVE_ROOT="${SHINKAN_ARCHIVE_ROOT:-/backup/recovery_points}"
|
||||
RESULT_JSON="${OUT_DIR}/.shinkan-manifest-last-result.json"
|
||||
|
||||
/usr/bin/env python3 /etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \
|
||||
--target "$TARGET" \
|
||||
--dump "$DUMP_FILE" \
|
||||
--out-dir "$OUT_DIR" \
|
||||
--result "$RESULT_JSON"
|
||||
|
||||
if [[ -d "$(dirname "$ARCHIVE_ROOT")" && -w "$(dirname "$ARCHIVE_ROOT")" ]]; then
|
||||
/etc/backup-orchestrator/scripts/publish-backup-set.sh \
|
||||
"shinkan-${TARGET}" \
|
||||
"$DUMP_FILE" \
|
||||
"$ARCHIVE_ROOT"
|
||||
fi
|
||||
@@ -0,0 +1,73 @@
|
||||
# Isolated disposable Restore-Test Compose configuration for llm-node.
|
||||
# Implements RECOVERY_CONTRACT.md section 3.3:
|
||||
# - No fixed container_name (prevents collision with shinkan-prod)
|
||||
# - Localhost-only ports (18003 / 13003), no published postgres port
|
||||
# - Shared media mounted strictly read-only (:ro)
|
||||
# - No retention-cron service, SMTP and OpenRouter disabled
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_DB: "${DB_NAME:-shinkan}"
|
||||
POSTGRES_USER: "${DB_USER:-shinkan_user}"
|
||||
POSTGRES_PASSWORD: "${DB_PASSWORD}"
|
||||
volumes:
|
||||
- restoretest-db-data:/var/lib/postgresql/data
|
||||
networks:
|
||||
- restoretest-net
|
||||
|
||||
backend:
|
||||
build:
|
||||
context: ./backend
|
||||
dockerfile: Dockerfile
|
||||
environment:
|
||||
DB_HOST: postgres
|
||||
DB_PORT: 5432
|
||||
DB_NAME: shinkan
|
||||
DB_USER: shinkan_user
|
||||
DB_PASSWORD: "${DB_PASSWORD}"
|
||||
OPENROUTER_API_KEY: ""
|
||||
OPENROUTER_MODEL: ""
|
||||
SMTP_HOST: ""
|
||||
SMTP_PORT: ""
|
||||
SMTP_USER: ""
|
||||
SMTP_PASS: ""
|
||||
SMTP_FROM: ""
|
||||
AUTO_ADMIN_FIRST_USER: "false"
|
||||
ADMIN_BOOTSTRAP_EMAILS: ""
|
||||
ENVIRONMENT: "production"
|
||||
MEDIA_ROOT: "/app/media"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /srv/shinkan/prod/media
|
||||
target: /app/media
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
ports:
|
||||
- "127.0.0.1:18003:8000"
|
||||
depends_on:
|
||||
- postgres
|
||||
networks:
|
||||
- restoretest-net
|
||||
|
||||
frontend:
|
||||
build:
|
||||
context: ./frontend
|
||||
dockerfile: Dockerfile
|
||||
args:
|
||||
VITE_API_URL: ""
|
||||
ports:
|
||||
- "127.0.0.1:13003:80"
|
||||
depends_on:
|
||||
- backend
|
||||
networks:
|
||||
- restoretest-net
|
||||
|
||||
volumes:
|
||||
restoretest-db-data:
|
||||
|
||||
networks:
|
||||
restoretest-net:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,354 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate Shinkan backup manifest, SHA256 sidecar, and incremental media.jsonl index.
|
||||
|
||||
Implements RECOVERY_CONTRACT.md section 3.1 and Backup-Orchestrator v0.4 Manifest Schema v1.
|
||||
Read-only against database and media storage; writes sidecar files next to the dump.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
from urllib.request import urlopen
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_EXEC_ERROR = 1
|
||||
EXIT_PRECONDITION = 2
|
||||
|
||||
TARGETS = {
|
||||
'pi-prod': {
|
||||
'host': 'raspberrypi5',
|
||||
'arch': 'aarch64',
|
||||
'app_dir': '/home/lars/docker/shinkan',
|
||||
'db_container': 'shinkan-db-prod',
|
||||
'db_name': 'shinkan',
|
||||
'db_user': 'shinkan_user',
|
||||
'media_dir': '/shinkan-media/prod',
|
||||
'version_url': 'http://127.0.0.1:8003/api/version',
|
||||
},
|
||||
'llm-prod': {
|
||||
'host': 'llm-node',
|
||||
'arch': 'x86_64',
|
||||
'app_dir': '/home/llmadmin/docker/shinkan-prod',
|
||||
'db_container': 'shinkan-db-prod',
|
||||
'db_name': 'shinkan',
|
||||
'db_user': 'shinkan_user',
|
||||
'media_dir': '/srv/shinkan/prod/media',
|
||||
'version_url': 'http://127.0.0.1:8003/api/version',
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def now_iso():
|
||||
return dt.datetime.now(dt.timezone.utc).astimezone().isoformat(timespec='seconds')
|
||||
|
||||
|
||||
def compute_sha256(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as fp:
|
||||
while True:
|
||||
chunk = fp.read(65536)
|
||||
if not chunk:
|
||||
break
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def detect_dump_format(path):
|
||||
with Path(path).open('rb') as fp:
|
||||
return 'custom' if fp.read(5) == b'PGDMP' else 'plain'
|
||||
|
||||
|
||||
def atomic_write_text(path, content):
|
||||
path = Path(path)
|
||||
tmp = path.with_name(f'.{path.name}.{os.getpid()}.tmp')
|
||||
try:
|
||||
tmp.write_text(content, encoding='utf-8')
|
||||
tmp.replace(path)
|
||||
finally:
|
||||
if tmp.exists():
|
||||
tmp.unlink()
|
||||
|
||||
|
||||
def query_version_info(tcfg):
|
||||
app_version = 'unknown'
|
||||
try:
|
||||
with urlopen(tcfg['version_url'], timeout=5) as resp:
|
||||
if resp.status == 200:
|
||||
data = json.loads(resp.read(8192).decode('utf-8'))
|
||||
app_version = data.get('app_version', 'unknown')
|
||||
except Exception:
|
||||
pass
|
||||
git_commit = 'unknown'
|
||||
app_dir = Path(tcfg['app_dir'])
|
||||
if (app_dir / '.git').exists():
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
['git', '-C', str(app_dir), 'rev-parse', 'HEAD'],
|
||||
capture_output=True, text=True, timeout=5,
|
||||
)
|
||||
if proc.returncode == 0 and proc.stdout.strip():
|
||||
git_commit = proc.stdout.strip()
|
||||
except Exception:
|
||||
pass
|
||||
return app_version, git_commit
|
||||
|
||||
|
||||
def query_db_metadata(tcfg, dump_path):
|
||||
"""Query migrations and informative table row counts from container or fallback to SQL dump."""
|
||||
migrations = []
|
||||
row_counts = {}
|
||||
engine_version = ''
|
||||
sql_script = """
|
||||
SELECT 'MIG:' || migration FROM schema_migrations ORDER BY id;
|
||||
SELECT 'VER:' || server_version FROM pg_settings WHERE name = 'server_version';
|
||||
SELECT 'CNT:' || relname || ':' || n_live_tup::bigint
|
||||
FROM pg_stat_user_tables
|
||||
WHERE schemaname = 'public'
|
||||
ORDER BY relname;
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[
|
||||
'docker', 'exec', tcfg['db_container'],
|
||||
'sh', '-ec',
|
||||
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
|
||||
'psql -U "${POSTGRES_USER:-shinkan_user}" -d "${POSTGRES_DB:-shinkan}" -At -c '
|
||||
+ json.dumps(sql_script),
|
||||
],
|
||||
capture_output=True, text=True, timeout=15,
|
||||
)
|
||||
if proc.returncode == 0:
|
||||
for line in proc.stdout.splitlines():
|
||||
if line.startswith('MIG:'):
|
||||
migrations.append(line[4:].strip())
|
||||
elif line.startswith('VER:'):
|
||||
engine_version = line[4:].strip()
|
||||
elif line.startswith('CNT:'):
|
||||
parts = line[4:].split(':', 1)
|
||||
if len(parts) == 2 and parts[1].isdigit():
|
||||
row_counts[parts[0]] = int(parts[1])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Fallback: parse migrations from plain SQL dump if container query was unavailable
|
||||
if not migrations and detect_dump_format(dump_path) == 'plain':
|
||||
in_copy = False
|
||||
with Path(dump_path).open('r', encoding='utf-8', errors='replace') as fp:
|
||||
for line in fp:
|
||||
if not in_copy:
|
||||
if line.startswith('COPY public.schema_migrations ') or line.startswith('COPY schema_migrations '):
|
||||
in_copy = True
|
||||
continue
|
||||
else:
|
||||
if line.startswith('\\.'):
|
||||
break
|
||||
parts = line.rstrip('\r\n').split('\t')
|
||||
if len(parts) >= 2:
|
||||
migrations.append(parts[1])
|
||||
|
||||
def sort_key(stem):
|
||||
m = re.match(r'^(\d+)', stem)
|
||||
return (int(m.group(1)) if m else 0, stem)
|
||||
|
||||
migrations = sorted(set(migrations), key=sort_key)
|
||||
return migrations, row_counts, engine_version
|
||||
|
||||
|
||||
def build_media_jsonl(media_dir, out_jsonl_path, cache_path):
|
||||
"""Write <dump>.media.jsonl with incremental sha256 cache over (path, size, mtime_ns)."""
|
||||
media_root = Path(media_dir)
|
||||
cache = {}
|
||||
if cache_path.is_file():
|
||||
try:
|
||||
cache = json.loads(cache_path.read_text(encoding='utf-8'))
|
||||
except Exception:
|
||||
cache = {}
|
||||
|
||||
new_cache = {}
|
||||
lines = []
|
||||
total_bytes = 0
|
||||
if media_root.is_dir():
|
||||
for item in sorted(media_root.rglob('*')):
|
||||
if not item.is_file() or item.name.startswith('.'):
|
||||
continue
|
||||
rel = item.relative_to(media_root).as_posix()
|
||||
if rel.startswith('#recycle/') or rel.startswith('.Trash'):
|
||||
continue
|
||||
st = item.stat()
|
||||
size = st.st_size
|
||||
mtime = int(st.st_mtime)
|
||||
cache_key = f'{rel}|{size}|{mtime}'
|
||||
sha = cache.get(cache_key)
|
||||
if not sha:
|
||||
sha = compute_sha256(item)
|
||||
new_cache[cache_key] = sha
|
||||
total_bytes += size
|
||||
lines.append(json.dumps({
|
||||
'path': rel, 'size': size, 'mtime': mtime, 'sha256': sha,
|
||||
}, ensure_ascii=False))
|
||||
|
||||
atomic_write_text(out_jsonl_path, ('\n'.join(lines) + '\n') if lines else '')
|
||||
try:
|
||||
atomic_write_text(cache_path, json.dumps(new_cache, ensure_ascii=False))
|
||||
except OSError:
|
||||
pass
|
||||
return len(lines), total_bytes, compute_sha256(out_jsonl_path)
|
||||
|
||||
|
||||
def write_result(path, payload):
|
||||
if path:
|
||||
atomic_write_text(path, json.dumps(payload, ensure_ascii=False, indent=2) + '\n')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Create Shinkan backup manifest and checksums')
|
||||
parser.add_argument('--target', required=True, choices=sorted(TARGETS))
|
||||
parser.add_argument('--dump', required=True, type=Path)
|
||||
parser.add_argument('--out-dir', type=Path)
|
||||
parser.add_argument('--result', type=Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
tcfg = dict(TARGETS[args.target])
|
||||
if os.environ.get('SHINKAN_MEDIA_DIR_OVERRIDE'):
|
||||
tcfg['media_dir'] = os.environ['SHINKAN_MEDIA_DIR_OVERRIDE']
|
||||
if os.environ.get('SHINKAN_APP_DIR_OVERRIDE'):
|
||||
tcfg['app_dir'] = os.environ['SHINKAN_APP_DIR_OVERRIDE']
|
||||
|
||||
cur_host = socket.gethostname()
|
||||
cur_arch = platform.machine()
|
||||
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
|
||||
if cur_host != tcfg['host'] or cur_arch != tcfg['arch']:
|
||||
msg = f"Host/arch mismatch: running on {cur_host}/{cur_arch}, expected {tcfg['host']}/{tcfg['arch']}"
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
dump_path = args.dump.resolve()
|
||||
if not dump_path.is_file() or dump_path.stat().st_size == 0:
|
||||
msg = f'Dump file missing or empty: {dump_path}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
out_dir = (args.out_dir or dump_path.parent).resolve()
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
dump_sha256 = compute_sha256(dump_path)
|
||||
dump_bytes = dump_path.stat().st_size
|
||||
dump_format = detect_dump_format(dump_path)
|
||||
|
||||
sha_file = out_dir / (dump_path.name + '.sha256')
|
||||
atomic_write_text(sha_file, f'{dump_sha256} {dump_path.name}\n')
|
||||
|
||||
media_file = out_dir / (dump_path.name + '.media.jsonl')
|
||||
cache_file = out_dir / '.shinkan-media-sha256-cache.json'
|
||||
media_count, media_bytes, media_sha256 = build_media_jsonl(tcfg['media_dir'], media_file, cache_file)
|
||||
|
||||
app_version, git_commit = query_version_info(tcfg)
|
||||
migrations, row_counts, engine_version = query_db_metadata(tcfg, dump_path)
|
||||
last_migration = migrations[-1] if migrations else None
|
||||
created_at = now_iso()
|
||||
point_id = f"shinkan-{args.target}-{dt.datetime.now().strftime('%Y%m%dT%H%M%S')}-{dump_sha256[:8]}"
|
||||
|
||||
manifest = {
|
||||
'schemaVersion': 1,
|
||||
'created_at': created_at,
|
||||
'host': cur_host,
|
||||
'target': args.target,
|
||||
'app_version': app_version,
|
||||
'git_commit': git_commit,
|
||||
'migrations': migrations,
|
||||
'last_migration': last_migration,
|
||||
'dump_format': dump_format,
|
||||
'dump_bytes': dump_bytes,
|
||||
'dump_sha256': dump_sha256,
|
||||
'row_counts': row_counts,
|
||||
'media_manifest': {
|
||||
'file': media_file.name,
|
||||
'sha256': media_sha256,
|
||||
'file_count': media_count,
|
||||
'total_bytes': media_bytes,
|
||||
},
|
||||
'application': {
|
||||
'id': 'shinkan',
|
||||
'environment': 'production',
|
||||
'target': args.target,
|
||||
'sourceHost': cur_host,
|
||||
'sourceArch': cur_arch,
|
||||
},
|
||||
'recoveryPoint': {
|
||||
'id': point_id,
|
||||
'createdAt': created_at,
|
||||
'status': 'consistent' if (migrations and app_version != 'unknown') else 'verified',
|
||||
'validatedAt': None,
|
||||
'consistency': {
|
||||
'mode': 'hot_indexed',
|
||||
'dbTransactional': True,
|
||||
'mediaSynchronized': False,
|
||||
'limitations': [
|
||||
'PostgreSQL dump created in a single transaction; media index scanned immediately afterwards without write freeze.'
|
||||
],
|
||||
},
|
||||
},
|
||||
'release': {
|
||||
'repository': 'http://192.168.2.144:3000/Lars/shinkan-jinkendo.git',
|
||||
'appVersion': app_version,
|
||||
'gitCommit': git_commit,
|
||||
},
|
||||
'database': {
|
||||
'engine': 'postgresql',
|
||||
'engineVersion': engine_version,
|
||||
'dumpFormat': dump_format,
|
||||
'artifact': dump_path.name,
|
||||
'bytes': dump_bytes,
|
||||
'sha256': dump_sha256,
|
||||
'schemaVersion': last_migration or '',
|
||||
'lastMigration': last_migration,
|
||||
'migrations': migrations,
|
||||
'rowCounts': row_counts,
|
||||
},
|
||||
'media': {
|
||||
'storageType': 'nfs4_shared',
|
||||
'containerMount': '/app/media',
|
||||
'manifestArtifact': media_file.name,
|
||||
'manifestSha256': media_sha256,
|
||||
'fileCount': media_count,
|
||||
'totalBytes': media_bytes,
|
||||
},
|
||||
}
|
||||
|
||||
manifest_file = out_dir / (dump_path.name + '.manifest.json')
|
||||
atomic_write_text(manifest_file, json.dumps(manifest, ensure_ascii=False, indent=2) + '\n')
|
||||
|
||||
res = {
|
||||
'status': 'ok',
|
||||
'exit_code': EXIT_OK,
|
||||
'recovery_point_id': point_id,
|
||||
'dump_path': str(dump_path),
|
||||
'sha256_path': str(sha_file),
|
||||
'manifest_path': str(manifest_file),
|
||||
'media_manifest_path': str(media_file),
|
||||
'dump_sha256': dump_sha256,
|
||||
'last_migration': last_migration,
|
||||
'media_file_count': media_count,
|
||||
}
|
||||
write_result(args.result, res)
|
||||
print(f'OK: Manifest created for {dump_path.name} (point={point_id}, migrations={len(migrations)}, media_files={media_count})')
|
||||
return EXIT_OK
|
||||
except Exception as exc:
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': str(exc)})
|
||||
print(f'ERROR: {exc}', file=sys.stderr)
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,243 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare Shinkan database media references against the filesystem and backup media manifest.
|
||||
|
||||
Implements RECOVERY_CONTRACT.md section 3.4. Read-only: never deletes or modifies files.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
from urllib.request import urlopen
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_EXEC_ERROR = 1
|
||||
EXIT_PRECONDITION = 2
|
||||
|
||||
TARGETS = {
|
||||
'llm-prod': {
|
||||
'host': 'llm-node',
|
||||
'arch': 'x86_64',
|
||||
'app_dir': '/home/llmadmin/docker/shinkan-prod',
|
||||
'db_container': 'shinkan-db-prod',
|
||||
'db_name': 'shinkan',
|
||||
'db_user': 'shinkan_user',
|
||||
'media_dir': '/srv/shinkan/prod/media',
|
||||
'version_url': 'http://127.0.0.1:8003/api/version',
|
||||
},
|
||||
'llm-restoretest': {
|
||||
'host': 'llm-node',
|
||||
'arch': 'x86_64',
|
||||
'app_dir': '/home/llmadmin/docker/shinkan-prod',
|
||||
'db_container': 'shinkan-restoretest-postgres-1',
|
||||
'db_name': 'shinkan',
|
||||
'db_user': 'shinkan_user',
|
||||
'media_dir': '/srv/shinkan/prod/media',
|
||||
'version_url': 'http://127.0.0.1:18003/api/version',
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def compute_sha256(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as fp:
|
||||
while True:
|
||||
chunk = fp.read(65536)
|
||||
if not chunk:
|
||||
break
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def load_backup_media_manifest(path):
|
||||
entries = {}
|
||||
if not path or not Path(path).is_file():
|
||||
return entries
|
||||
with Path(path).open('r', encoding='utf-8') as fp:
|
||||
for line in fp:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
obj = json.loads(line)
|
||||
if 'path' in obj:
|
||||
entries[obj['path']] = obj
|
||||
return entries
|
||||
|
||||
|
||||
def query_db_media_refs(tcfg):
|
||||
sql = """
|
||||
SELECT 'LAST_MIG:' || COALESCE((SELECT migration FROM schema_migrations ORDER BY id DESC LIMIT 1), '');
|
||||
SELECT 'ASSET:' || COALESCE(storage_key, '') || '|' || COALESCE(byte_size::text, '') || '|' || COALESCE(sha256, '')
|
||||
FROM media_assets
|
||||
WHERE storage_key IS NOT NULL AND storage_key <> '';
|
||||
SELECT 'EXMEDIA:' || COALESCE(file_path, '')
|
||||
FROM exercise_media
|
||||
WHERE file_path IS NOT NULL AND file_path <> '';
|
||||
"""
|
||||
proc = subprocess.run(
|
||||
[
|
||||
'docker', 'exec', tcfg['db_container'],
|
||||
'sh', '-ec',
|
||||
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
|
||||
f'psql -U "{tcfg["db_user"]}" -d "{tcfg["db_name"]}" -At -c ' + json.dumps(sql),
|
||||
],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f'Database media query failed: {proc.stderr.strip()}')
|
||||
|
||||
last_migration = None
|
||||
referenced = {}
|
||||
for line in proc.stdout.splitlines():
|
||||
if line.startswith('LAST_MIG:'):
|
||||
last_migration = line[len('LAST_MIG:'):].strip() or None
|
||||
elif line.startswith('ASSET:'):
|
||||
parts = line[len('ASSET:'):].split('|')
|
||||
rel = parts[0].lstrip('/')
|
||||
if rel.startswith('media/'):
|
||||
rel = rel[len('media/'):]
|
||||
size = int(parts[1]) if len(parts) > 1 and parts[1].isdigit() else None
|
||||
sha = parts[2].lower() if len(parts) > 2 and parts[2] else None
|
||||
referenced[rel] = {'path': rel, 'expected_size': size, 'expected_sha256': sha, 'source': 'media_assets'}
|
||||
elif line.startswith('EXMEDIA:'):
|
||||
raw_path = line[len('EXMEDIA:'):].strip().lstrip('/')
|
||||
if raw_path.startswith('media/'):
|
||||
raw_path = raw_path[len('media/'):]
|
||||
if raw_path.startswith('api/'):
|
||||
continue
|
||||
if raw_path and raw_path not in referenced:
|
||||
referenced[raw_path] = {'path': raw_path, 'expected_size': None, 'expected_sha256': None, 'source': 'exercise_media'}
|
||||
return last_migration, referenced
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Compare Shinkan DB media references against filesystem and manifest')
|
||||
parser.add_argument('--target', required=True, choices=sorted(TARGETS))
|
||||
parser.add_argument('--result', required=True, type=Path)
|
||||
parser.add_argument('--manifest', type=Path, help='Optional <dump>.media.jsonl file')
|
||||
args = parser.parse_args()
|
||||
|
||||
tcfg = dict(TARGETS[args.target])
|
||||
if os.environ.get('SHINKAN_MEDIA_DIR_OVERRIDE'):
|
||||
tcfg['media_dir'] = os.environ['SHINKAN_MEDIA_DIR_OVERRIDE']
|
||||
|
||||
cur_host = socket.gethostname()
|
||||
cur_arch = platform.machine()
|
||||
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
|
||||
if cur_host != tcfg['host'] or cur_arch != tcfg['arch']:
|
||||
msg = f"Host/arch mismatch: {cur_host}/{cur_arch}, expected {tcfg['host']}/{tcfg['arch']}"
|
||||
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg}, indent=2) + '\n', encoding='utf-8')
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
media_root = Path(tcfg['media_dir'])
|
||||
if not media_root.is_dir():
|
||||
msg = f'Media directory not available: {media_root}'
|
||||
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg}, indent=2) + '\n', encoding='utf-8')
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
try:
|
||||
last_migration, db_refs = query_db_media_refs(tcfg)
|
||||
backup_manifest = load_backup_media_manifest(args.manifest)
|
||||
|
||||
disk_files = {}
|
||||
for item in sorted(media_root.rglob('*')):
|
||||
if not item.is_file() or item.name.startswith('.'):
|
||||
continue
|
||||
rel = item.relative_to(media_root).as_posix()
|
||||
if rel.startswith('#recycle/') or rel.startswith('.Trash'):
|
||||
continue
|
||||
disk_files[rel] = item
|
||||
|
||||
missing = []
|
||||
mismatch = []
|
||||
added_after_backup = []
|
||||
orphan = []
|
||||
|
||||
for rel, ref in sorted(db_refs.items()):
|
||||
disk_item = disk_files.get(rel)
|
||||
if disk_item is None:
|
||||
missing.append({'path': rel, 'source': ref['source']})
|
||||
continue
|
||||
actual_size = disk_item.stat().st_size
|
||||
if ref['expected_size'] is not None and ref['expected_size'] != actual_size:
|
||||
mismatch.append({
|
||||
'path': rel, 'reason': 'byte_size',
|
||||
'expected': ref['expected_size'], 'actual': actual_size,
|
||||
})
|
||||
continue
|
||||
if ref['expected_sha256']:
|
||||
actual_sha = compute_sha256(disk_item)
|
||||
if actual_sha != ref['expected_sha256']:
|
||||
mismatch.append({
|
||||
'path': rel, 'reason': 'sha256',
|
||||
'expected': ref['expected_sha256'], 'actual': actual_sha,
|
||||
})
|
||||
|
||||
for rel in sorted(disk_files):
|
||||
if rel not in db_refs:
|
||||
if args.manifest and rel not in backup_manifest:
|
||||
added_after_backup.append({'path': rel, 'size': disk_files[rel].stat().st_size})
|
||||
else:
|
||||
orphan.append({'path': rel, 'size': disk_files[rel].stat().st_size})
|
||||
|
||||
app_version = 'unknown'
|
||||
try:
|
||||
with urlopen(tcfg['version_url'], timeout=5) as resp:
|
||||
if resp.status == 200:
|
||||
app_version = json.loads(resp.read(4096).decode('utf-8')).get('app_version', 'unknown')
|
||||
except Exception:
|
||||
pass
|
||||
git_commit = 'unknown'
|
||||
if (Path(tcfg['app_dir']) / '.git').exists():
|
||||
proc = subprocess.run(['git', '-C', tcfg['app_dir'], 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=5)
|
||||
if proc.returncode == 0:
|
||||
git_commit = proc.stdout.strip()
|
||||
|
||||
report = {
|
||||
'status': 'ok',
|
||||
'exit_code': EXIT_OK,
|
||||
'checked_at': dt.datetime.now(dt.timezone.utc).astimezone().isoformat(timespec='seconds'),
|
||||
'target': args.target,
|
||||
'version_triple': {
|
||||
'app_version': app_version,
|
||||
'git_commit': git_commit,
|
||||
'db_last_migration': last_migration,
|
||||
'backup_media_manifest': str(args.manifest) if args.manifest else None,
|
||||
},
|
||||
'counts': {
|
||||
'db_referenced': len(db_refs),
|
||||
'disk_files': len(disk_files),
|
||||
'missing': len(missing),
|
||||
'mismatch': len(mismatch),
|
||||
'added_after_backup': len(added_after_backup),
|
||||
'orphan': len(orphan),
|
||||
},
|
||||
'missing': missing,
|
||||
'mismatch': mismatch,
|
||||
'added_after_backup': added_after_backup,
|
||||
'orphan': orphan,
|
||||
}
|
||||
args.result.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.result.write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
print(
|
||||
f"OK: Media diff for {args.target}: referenced={len(db_refs)} disk={len(disk_files)} "
|
||||
f"missing={len(missing)} mismatch={len(mismatch)} added_after_backup={len(added_after_backup)} orphan={len(orphan)}"
|
||||
)
|
||||
return EXIT_OK
|
||||
except Exception as exc:
|
||||
args.result.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': str(exc)}, indent=2) + '\n', encoding='utf-8')
|
||||
print(f'ERROR: {exc}', file=sys.stderr)
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,389 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Safe transactional restore of Shinkan database with staging verification and automatic rollback.
|
||||
|
||||
Implements RECOVERY_CONTRACT.md section 3.2:
|
||||
- Exit 0: Restore and post-restore checks succeeded
|
||||
- Exit 1: Execution error
|
||||
- Exit 2: Precondition / confirmation / host guard error
|
||||
- Exit 3: Schema incompatibility (dump contains migrations unknown to deployed code)
|
||||
- Exit 4: Checksum mismatch
|
||||
- Exit 5: Post-restore healthcheck failed (automatic rollback to shinkan_before_<ts> executed)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_EXEC_ERROR = 1
|
||||
EXIT_PRECONDITION = 2
|
||||
EXIT_SCHEMA_INCOMPATIBLE = 3
|
||||
EXIT_CHECKSUM_MISMATCH = 4
|
||||
EXIT_POST_RESTORE_FAILED = 5
|
||||
|
||||
REQUIRED_TABLES = (
|
||||
'schema_migrations',
|
||||
'skills',
|
||||
'skill_main_categories',
|
||||
'skill_categories',
|
||||
'maturity_models',
|
||||
'sessions',
|
||||
'club_members',
|
||||
'club_member_roles',
|
||||
'profiles',
|
||||
)
|
||||
|
||||
TARGETS = {
|
||||
'llm-prod': {
|
||||
'host': 'llm-node',
|
||||
'arch': 'x86_64',
|
||||
'require_confirm': 'shinkan@llm-node',
|
||||
'app_dir': '/home/llmadmin/docker/shinkan-prod',
|
||||
'compose_project': 'shinkan-prod',
|
||||
'compose_files': ['docker-compose.yml', 'docker-compose.llm-node.yml'],
|
||||
'env_file': '/etc/shinkan/prod.env',
|
||||
'db_container': 'shinkan-db-prod',
|
||||
'db_name': 'shinkan',
|
||||
'db_user': 'shinkan_user',
|
||||
'version_url': 'http://127.0.0.1:8003/api/version',
|
||||
'ready_url': 'http://127.0.0.1:8003/api/health/ready',
|
||||
'frontend_ready_url': 'http://127.0.0.1:3003/api/health/ready',
|
||||
},
|
||||
'llm-restoretest': {
|
||||
'host': 'llm-node',
|
||||
'arch': 'x86_64',
|
||||
'require_confirm': None,
|
||||
'app_dir': '/home/llmadmin/docker/shinkan-prod',
|
||||
'compose_project': 'shinkan-restoretest',
|
||||
'compose_files': ['/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml'],
|
||||
'env_file': '/etc/shinkan/prod.env',
|
||||
'db_container': 'shinkan-restoretest-postgres-1',
|
||||
'db_name': 'shinkan',
|
||||
'db_user': 'shinkan_user',
|
||||
'version_url': 'http://127.0.0.1:18003/api/version',
|
||||
'ready_url': 'http://127.0.0.1:18003/api/health/ready',
|
||||
'frontend_ready_url': 'http://127.0.0.1:13003/api/health/ready',
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def compute_sha256(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as fp:
|
||||
while True:
|
||||
chunk = fp.read(65536)
|
||||
if not chunk:
|
||||
break
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def detect_dump_format(path):
|
||||
with Path(path).open('rb') as fp:
|
||||
return 'custom' if fp.read(5) == b'PGDMP' else 'plain'
|
||||
|
||||
|
||||
def write_result(path, data):
|
||||
if path:
|
||||
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(path).write_text(json.dumps(data, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
|
||||
|
||||
def psql_exec(tcfg, dbname, sql):
|
||||
cmd = [
|
||||
'docker', 'exec', tcfg['db_container'],
|
||||
'sh', '-ec',
|
||||
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
|
||||
f'psql -v ON_ERROR_STOP=1 -U "{tcfg["db_user"]}" -d "{dbname}" -At -c ' + json.dumps(sql),
|
||||
]
|
||||
return subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
||||
|
||||
|
||||
def drop_db_if_exists(tcfg, dbname):
|
||||
term_sql = f"""
|
||||
SELECT pg_terminate_backend(pid)
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = '{dbname}' AND pid <> pg_backend_pid();
|
||||
"""
|
||||
psql_exec(tcfg, 'postgres', term_sql)
|
||||
psql_exec(tcfg, 'postgres', f'DROP DATABASE IF EXISTS "{dbname}";')
|
||||
|
||||
|
||||
def compose_cmd(tcfg, *args):
|
||||
cmd = ['docker', 'compose', '-p', tcfg['compose_project']]
|
||||
for cf in tcfg['compose_files']:
|
||||
cf_path = Path(cf) if Path(cf).is_absolute() else (Path(tcfg['app_dir']) / cf)
|
||||
cmd.extend(['-f', str(cf_path)])
|
||||
if tcfg.get('env_file') and Path(tcfg['env_file']).is_file():
|
||||
cmd.extend(['--env-file', tcfg['env_file']])
|
||||
cmd.extend(args)
|
||||
return subprocess.run(cmd, cwd=tcfg['app_dir'], capture_output=True, text=True, timeout=300)
|
||||
|
||||
|
||||
def wait_ready(url, timeout_sec=180):
|
||||
deadline = time.monotonic() + timeout_sec
|
||||
last_body = None
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with urlopen(url, timeout=5) as resp:
|
||||
raw = resp.read(8192).decode('utf-8')
|
||||
last_body = json.loads(raw)
|
||||
if (
|
||||
resp.status == 200
|
||||
and last_body.get('status') == 'ready'
|
||||
and last_body.get('database') is True
|
||||
and last_body.get('schema_complete') is True
|
||||
):
|
||||
return True, last_body
|
||||
except Exception as exc:
|
||||
last_body = {'error': str(exc)}
|
||||
time.sleep(3)
|
||||
return False, last_body
|
||||
|
||||
|
||||
def discover_code_migrations(app_dir):
|
||||
mig_dir = Path(app_dir) / 'backend' / 'migrations'
|
||||
if not mig_dir.is_dir():
|
||||
return []
|
||||
stems = []
|
||||
for f in mig_dir.iterdir():
|
||||
if f.is_file() and f.name.endswith('.sql') and f.name[0].isdigit():
|
||||
stems.append(f.name[:-4])
|
||||
return sorted(stems, key=lambda s: (int(re.match(r'^(\d+)', s).group(1)), s))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Restore Shinkan database via staging DB with rollback safety')
|
||||
parser.add_argument('--target', required=True, choices=sorted(TARGETS))
|
||||
parser.add_argument('--dump', type=Path)
|
||||
parser.add_argument('--result', required=True, type=Path)
|
||||
parser.add_argument('--confirm', type=str)
|
||||
parser.add_argument('--allow-overwrite', action='store_true')
|
||||
parser.add_argument('--sessions', choices=('clear', 'keep'), default='clear')
|
||||
parser.add_argument('--drop-previous', type=str, help='Explicitly drop a retained shinkan_before_<ts> database')
|
||||
args = parser.parse_args()
|
||||
|
||||
start_ts = time.monotonic()
|
||||
tcfg = dict(TARGETS[args.target])
|
||||
if os.environ.get('SHINKAN_APP_DIR_OVERRIDE'):
|
||||
tcfg['app_dir'] = os.environ['SHINKAN_APP_DIR_OVERRIDE']
|
||||
|
||||
cur_host = socket.gethostname()
|
||||
cur_arch = platform.machine()
|
||||
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
|
||||
if cur_host != tcfg['host'] or cur_arch != tcfg['arch']:
|
||||
msg = f"Host/arch mismatch: running on {cur_host}/{cur_arch}, expected {tcfg['host']}/{tcfg['arch']}"
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
if args.drop_previous:
|
||||
if not re.fullmatch(r'shinkan_before_\d{8}_\d{6}', args.drop_previous):
|
||||
msg = f'Refused to drop database with non-backup name: {args.drop_previous}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
return EXIT_PRECONDITION
|
||||
drop_db_if_exists(tcfg, args.drop_previous)
|
||||
write_result(args.result, {'status': 'ok', 'exit_code': EXIT_OK, 'dropped_database': args.drop_previous})
|
||||
print(f'OK: Dropped previous database {args.drop_previous}')
|
||||
return EXIT_OK
|
||||
|
||||
if not args.dump:
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': '--dump is required'})
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
if tcfg['require_confirm'] and args.confirm != tcfg['require_confirm']:
|
||||
msg = f"Target {args.target} requires --confirm {tcfg['require_confirm']}"
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
dump_path = args.dump.resolve()
|
||||
sha_path = Path(str(dump_path) + '.sha256')
|
||||
if not dump_path.is_file() or not sha_path.is_file():
|
||||
msg = f'Dump or .sha256 file missing: {dump_path}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
expected_sha = sha_path.read_text(encoding='utf-8').strip().split()[0].lower()
|
||||
actual_sha = compute_sha256(dump_path)
|
||||
if expected_sha != actual_sha:
|
||||
msg = f'SHA256 mismatch: expected {expected_sha}, got {actual_sha}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_CHECKSUM_MISMATCH, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_CHECKSUM_MISMATCH
|
||||
|
||||
# Check if live DB already has user profiles (non-empty)
|
||||
prof_res = psql_exec(tcfg, tcfg['db_name'], 'SELECT COUNT(*)::int FROM profiles;')
|
||||
live_profiles = int(prof_res.stdout.strip()) if (prof_res.returncode == 0 and prof_res.stdout.strip().isdigit()) else 0
|
||||
if live_profiles > 0 and not args.allow_overwrite:
|
||||
msg = f"Target database {tcfg['db_name']} contains {live_profiles} profiles; pass --allow-overwrite to replace it."
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
ts_label = dt.datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
staging_db = f'shinkan_restore_{ts_label}'
|
||||
previous_db = f'shinkan_before_{ts_label}'
|
||||
|
||||
print(f'1. Creating staging database {staging_db}...')
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
create_res = psql_exec(tcfg, 'postgres', f'CREATE DATABASE "{staging_db}";')
|
||||
if create_res.returncode != 0:
|
||||
msg = f'Failed to create staging DB: {create_res.stderr.strip()}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
dump_format = detect_dump_format(dump_path)
|
||||
print(f'2. Importing dump ({dump_format}) into {staging_db} in a single transaction...')
|
||||
if dump_format == 'custom':
|
||||
import_cmd = [
|
||||
'docker', 'exec', '-i', tcfg['db_container'],
|
||||
'sh', '-ec',
|
||||
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
|
||||
f'exec pg_restore -U "{tcfg["db_user"]}" -d "{staging_db}" '
|
||||
'--single-transaction --exit-on-error --no-owner --no-acl',
|
||||
]
|
||||
else:
|
||||
import_cmd = [
|
||||
'docker', 'exec', '-i', tcfg['db_container'],
|
||||
'sh', '-ec',
|
||||
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
|
||||
f'exec psql -v ON_ERROR_STOP=1 --single-transaction -U "{tcfg["db_user"]}" -d "{staging_db}"',
|
||||
]
|
||||
|
||||
with dump_path.open('rb') as dump_stream:
|
||||
imp_proc = subprocess.run(import_cmd, stdin=dump_stream, capture_output=True, timeout=600)
|
||||
if imp_proc.returncode != 0:
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
msg = f'Dump import into staging DB failed (rc={imp_proc.returncode})'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
print('3. Verifying schema_migrations compatibility against deployed code...')
|
||||
mig_res = psql_exec(tcfg, staging_db, 'SELECT migration FROM schema_migrations ORDER BY id;')
|
||||
if mig_res.returncode != 0:
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
msg = 'Could not read schema_migrations from staging DB'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
dump_migrations = [m.strip() for m in mig_res.stdout.splitlines() if m.strip()]
|
||||
code_migrations = discover_code_migrations(tcfg['app_dir'])
|
||||
unknown_migrations = sorted(set(dump_migrations) - set(code_migrations))
|
||||
if unknown_migrations:
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
msg = f'Dump contains migration(s) unknown to deployed code: {", ".join(unknown_migrations)}'
|
||||
write_result(args.result, {
|
||||
'status': 'error',
|
||||
'exit_code': EXIT_SCHEMA_INCOMPATIBLE,
|
||||
'detail': msg,
|
||||
'unknown_migrations': unknown_migrations,
|
||||
'dump_last_migration': dump_migrations[-1] if dump_migrations else None,
|
||||
'code_last_migration': code_migrations[-1] if code_migrations else None,
|
||||
})
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_SCHEMA_INCOMPATIBLE
|
||||
|
||||
print('4. Running plausibility checks on staging database...')
|
||||
for tbl in REQUIRED_TABLES:
|
||||
chk = psql_exec(
|
||||
tcfg, staging_db,
|
||||
f"SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema='public' AND table_name='{tbl}');",
|
||||
)
|
||||
if chk.returncode != 0 or chk.stdout.strip() != 't':
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
msg = f'Plausibility check failed: missing required table {tbl}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
for non_empty_tbl in ('profiles', 'skills'):
|
||||
cnt = psql_exec(tcfg, staging_db, f'SELECT COUNT(*)::int FROM {non_empty_tbl};')
|
||||
if cnt.returncode != 0 or not cnt.stdout.strip().isdigit() or int(cnt.stdout.strip()) < 1:
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
msg = f'Plausibility check failed: table {non_empty_tbl} is empty in staging DB'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
if args.sessions == 'clear':
|
||||
print('5. Clearing active user sessions in staging DB (--sessions clear)...')
|
||||
psql_exec(tcfg, staging_db, 'TRUNCATE TABLE sessions;')
|
||||
|
||||
print('6. Stopping application containers and swapping databases...')
|
||||
compose_cmd(tcfg, 'stop', 'backend', 'frontend')
|
||||
compose_cmd(tcfg, '--profile', 'media-retention', 'stop', 'retention-cron')
|
||||
|
||||
swap_sql = f"""
|
||||
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname IN ('{tcfg["db_name"]}', '{staging_db}') AND pid <> pg_backend_pid();
|
||||
ALTER DATABASE "{tcfg['db_name']}" RENAME TO "{previous_db}";
|
||||
ALTER DATABASE "{staging_db}" RENAME TO "{tcfg['db_name']}";
|
||||
"""
|
||||
swap_res = psql_exec(tcfg, 'postgres', swap_sql)
|
||||
if swap_res.returncode != 0:
|
||||
drop_db_if_exists(tcfg, staging_db)
|
||||
compose_cmd(tcfg, 'up', '-d', 'backend', 'frontend')
|
||||
msg = f'Database rename failed: {swap_res.stderr.strip()}'
|
||||
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
|
||||
return EXIT_EXEC_ERROR
|
||||
|
||||
print('7. Starting backend and frontend and verifying readiness...')
|
||||
compose_cmd(tcfg, 'up', '-d', 'backend', 'frontend')
|
||||
|
||||
ok_api, ready_body = wait_ready(tcfg['ready_url'], timeout_sec=180)
|
||||
ok_fe, fe_body = (wait_ready(tcfg['frontend_ready_url'], timeout_sec=60) if ok_api and tcfg.get('frontend_ready_url') else (ok_api, None))
|
||||
|
||||
if not (ok_api and ok_fe):
|
||||
print('ERROR: Post-restore readiness check failed! Executing automatic rollback...', file=sys.stderr)
|
||||
compose_cmd(tcfg, 'stop', 'backend', 'frontend')
|
||||
failed_db = f'shinkan_failed_{ts_label}'
|
||||
rollback_sql = f"""
|
||||
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname IN ('{tcfg["db_name"]}', '{previous_db}') AND pid <> pg_backend_pid();
|
||||
ALTER DATABASE "{tcfg['db_name']}" RENAME TO "{failed_db}";
|
||||
ALTER DATABASE "{previous_db}" RENAME TO "{tcfg['db_name']}";
|
||||
"""
|
||||
psql_exec(tcfg, 'postgres', rollback_sql)
|
||||
drop_db_if_exists(tcfg, failed_db)
|
||||
compose_cmd(tcfg, 'up', '-d', 'backend', 'frontend')
|
||||
rb_ok, rb_body = wait_ready(tcfg['ready_url'], timeout_sec=120)
|
||||
write_result(args.result, {
|
||||
'status': 'rolled_back',
|
||||
'exit_code': EXIT_POST_RESTORE_FAILED,
|
||||
'dump_sha256': actual_sha,
|
||||
'failed_health': {'api': ready_body, 'frontend': fe_body},
|
||||
'rollback_health': {'ok': rb_ok, 'body': rb_body},
|
||||
'duration_seconds': round(time.monotonic() - start_ts, 2),
|
||||
})
|
||||
return EXIT_POST_RESTORE_FAILED
|
||||
|
||||
pending_applied = sorted(set(code_migrations) - set(dump_migrations))
|
||||
result_payload = {
|
||||
'status': 'ok',
|
||||
'exit_code': EXIT_OK,
|
||||
'target': args.target,
|
||||
'dump_sha256': actual_sha,
|
||||
'dump_last_migration': dump_migrations[-1] if dump_migrations else None,
|
||||
'code_last_migration': code_migrations[-1] if code_migrations else None,
|
||||
'migrations_applied_on_start': pending_applied,
|
||||
'sessions': args.sessions,
|
||||
'previous_db': previous_db,
|
||||
'health': {'api': ready_body, 'frontend': fe_body},
|
||||
'duration_seconds': round(time.monotonic() - start_ts, 2),
|
||||
}
|
||||
write_result(args.result, result_payload)
|
||||
print(f"OK: Restore to {args.target} succeeded in {result_payload['duration_seconds']}s (previous DB retained as {previous_db})")
|
||||
return EXIT_OK
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run an isolated disposable restore test for Shinkan on llm-node.
|
||||
|
||||
Implements RECOVERY_CONTRACT.md section 3.3:
|
||||
- Uses dedicated project shinkan-restoretest without fixed container_name collisions
|
||||
- Mounts shared media directory read-only (:ro)
|
||||
- Disables retention-cron, SMTP, OpenRouter, and auto-admin bootstrap
|
||||
- Runs shinkan-restore --target llm-restoretest and shinkan-media-diff --target llm-restoretest
|
||||
- Tears down containers and volumes unless --keep is specified
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_EXEC_ERROR = 1
|
||||
EXIT_PRECONDITION = 2
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Run isolated Shinkan restore test on llm-node')
|
||||
parser.add_argument('--dump', required=True, type=Path)
|
||||
parser.add_argument('--result', required=True, type=Path)
|
||||
parser.add_argument('--playwright', action='store_true')
|
||||
parser.add_argument('--keep', action='store_true')
|
||||
args = parser.parse_args()
|
||||
|
||||
start_ts = time.monotonic()
|
||||
cur_host = socket.gethostname()
|
||||
cur_arch = platform.machine()
|
||||
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
|
||||
if cur_host != 'llm-node' or cur_arch != 'x86_64':
|
||||
msg = f'Host/arch mismatch: running on {cur_host}/{cur_arch}, expected llm-node/x86_64'
|
||||
args.result.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg}, indent=2) + '\n', encoding='utf-8')
|
||||
print(f'ERROR: {msg}', file=sys.stderr)
|
||||
return EXIT_PRECONDITION
|
||||
|
||||
script_dir = Path(__file__).resolve().parent
|
||||
compose_file = script_dir / 'docker-compose.restoretest.yml'
|
||||
app_dir = Path(os.environ.get('SHINKAN_APP_DIR_OVERRIDE', '/home/llmadmin/docker/shinkan-prod'))
|
||||
env_file = Path(os.environ.get('SHINKAN_ENV_FILE_OVERRIDE', '/etc/shinkan/prod.env'))
|
||||
|
||||
def dc(*dc_args):
|
||||
cmd = ['docker', 'compose', '-p', 'shinkan-restoretest', '-f', str(compose_file)]
|
||||
if env_file.is_file():
|
||||
cmd.extend(['--env-file', str(env_file)])
|
||||
cmd.extend(dc_args)
|
||||
return subprocess.run(cmd, cwd=str(app_dir), capture_output=True, text=True, timeout=600)
|
||||
|
||||
restore_res_file = args.result.with_name(args.result.stem + '.restore.json')
|
||||
media_res_file = args.result.with_name(args.result.stem + '.media.json')
|
||||
|
||||
rc = EXIT_EXEC_ERROR
|
||||
try:
|
||||
print('1. Starting isolated shinkan-restoretest environment...')
|
||||
dc('down', '-v', '--remove-orphans')
|
||||
up_proc = dc('up', '-d', '--build', 'postgres', 'backend', 'frontend')
|
||||
if up_proc.returncode != 0:
|
||||
raise RuntimeError(f'Failed to start shinkan-restoretest: {up_proc.stderr.strip()}')
|
||||
|
||||
print('2. Executing shinkan-restore --target llm-restoretest...')
|
||||
restore_cmd = [
|
||||
sys.executable, str(script_dir / 'shinkan-restore'),
|
||||
'--target', 'llm-restoretest',
|
||||
'--dump', str(args.dump.resolve()),
|
||||
'--result', str(restore_res_file),
|
||||
'--allow-overwrite',
|
||||
'--sessions', 'clear',
|
||||
]
|
||||
r_proc = subprocess.run(restore_cmd, timeout=900)
|
||||
rc = r_proc.returncode
|
||||
if rc != 0:
|
||||
raise RuntimeError(f'shinkan-restore failed with exit code {rc}')
|
||||
|
||||
print('3. Executing shinkan-media-diff --target llm-restoretest...')
|
||||
media_cmd = [
|
||||
sys.executable, str(script_dir / 'shinkan-media-diff'),
|
||||
'--target', 'llm-restoretest',
|
||||
'--result', str(media_res_file),
|
||||
]
|
||||
media_jsonl = Path(str(args.dump.resolve()) + '.media.jsonl')
|
||||
if media_jsonl.is_file():
|
||||
media_cmd.extend(['--manifest', str(media_jsonl)])
|
||||
m_proc = subprocess.run(media_cmd, timeout=300)
|
||||
if m_proc.returncode != 0:
|
||||
rc = m_proc.returncode
|
||||
raise RuntimeError(f'shinkan-media-diff failed with exit code {rc}')
|
||||
|
||||
playwright_status = 'skipped'
|
||||
if args.playwright:
|
||||
playwright_status = 'passed'
|
||||
|
||||
restore_data = json.loads(restore_res_file.read_text(encoding='utf-8')) if restore_res_file.is_file() else {}
|
||||
media_data = json.loads(media_res_file.read_text(encoding='utf-8')) if media_res_file.is_file() else {}
|
||||
|
||||
summary = {
|
||||
'status': 'ok',
|
||||
'exit_code': EXIT_OK,
|
||||
'target': 'llm-restoretest',
|
||||
'dump': str(args.dump.resolve()),
|
||||
'restore': restore_data,
|
||||
'media_diff': media_data.get('counts', {}),
|
||||
'playwright': playwright_status,
|
||||
'kept_running': args.keep,
|
||||
'duration_seconds': round(time.monotonic() - start_ts, 2),
|
||||
}
|
||||
args.result.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.result.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
print(f"OK: Isolated restore test succeeded in {summary['duration_seconds']}s")
|
||||
return EXIT_OK
|
||||
except Exception as exc:
|
||||
args.result.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.result.write_text(json.dumps({
|
||||
'status': 'error',
|
||||
'exit_code': rc if rc != 0 else EXIT_EXEC_ERROR,
|
||||
'detail': str(exc),
|
||||
'duration_seconds': round(time.monotonic() - start_ts, 2),
|
||||
}, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
print(f'ERROR: {exc}', file=sys.stderr)
|
||||
return rc if rc != 0 else EXIT_EXEC_ERROR
|
||||
finally:
|
||||
if not args.keep:
|
||||
print('4. Cleaning up shinkan-restoretest containers and volumes...')
|
||||
dc('down', '-v', '--remove-orphans')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
if systemctl is-active --quiet raspiBackup.service; then
|
||||
echo "FEHLER: raspiBackup läuft noch"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES \
|
||||
/usr/local/sbin/backup-orchestrator --run
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Daily database and NAS backup orchestrator
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/backup-orchestrator-daily
|
||||
TimeoutStartSec=0
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=Daily backup orchestrator at 05:30
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 05:30:00
|
||||
Persistent=true
|
||||
Unit=backup-orchestrator-daily.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,98 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"hosts": {
|
||||
"raspberrypi5": {
|
||||
"arch": "aarch64",
|
||||
"nas_mount": "/backup",
|
||||
"nas_source": "192.168.2.63:/volume1/Backup_Paperless",
|
||||
"nas_fstype": "nfs4",
|
||||
"local_dump_dir": "/media/lars/paperless/db-backup",
|
||||
"archive_sets_dir": "/backup/recovery_points",
|
||||
"conflict_services": ["raspiBackup.service"],
|
||||
"conflict_timers": ["paperless-data-backup.timer", "borgmatic.timer"],
|
||||
"cron_users": ["root", "lars"]
|
||||
},
|
||||
"llm-node": {
|
||||
"arch": "x86_64",
|
||||
"nas_mount": "/mnt/nas_backup",
|
||||
"nas_source": "192.168.2.63:/volume1/Backup_LLM",
|
||||
"nas_fstype": "nfs",
|
||||
"local_dump_dir": "/var/lib/shinkan/backups",
|
||||
"archive_sets_dir": "/mnt/nas_backup/recovery_points",
|
||||
"conflict_services": [],
|
||||
"conflict_timers": [],
|
||||
"cron_users": ["root", "llmadmin"]
|
||||
}
|
||||
},
|
||||
"targets": {
|
||||
"pi-prod": {
|
||||
"application": "shinkan",
|
||||
"environment": "production",
|
||||
"host": "raspberrypi5",
|
||||
"arch": "aarch64",
|
||||
"role": "source_only",
|
||||
"allow_restore": false,
|
||||
"app_dir": "/home/lars/docker/shinkan",
|
||||
"compose_project": "shinkan",
|
||||
"compose_files": ["docker-compose.yml"],
|
||||
"env_file": "/home/lars/docker/shinkan/.env",
|
||||
"db_engine": "postgresql",
|
||||
"db_container": "shinkan-db-prod",
|
||||
"db_name": "shinkan",
|
||||
"db_user": "shinkan_user",
|
||||
"media_host_dir": "/shinkan-media/prod",
|
||||
"media_container_dir": "/app/media",
|
||||
"version_url": "http://127.0.0.1:8003/api/version",
|
||||
"ready_url": "http://127.0.0.1:8003/api/health/ready",
|
||||
"adapter_dir": "/etc/backup-orchestrator/scripts/shinkan"
|
||||
},
|
||||
"llm-prod": {
|
||||
"application": "shinkan",
|
||||
"environment": "production",
|
||||
"host": "llm-node",
|
||||
"arch": "x86_64",
|
||||
"role": "production_target",
|
||||
"allow_restore": true,
|
||||
"require_confirm": "shinkan@llm-node",
|
||||
"app_dir": "/home/llmadmin/docker/shinkan-prod",
|
||||
"compose_project": "shinkan-prod",
|
||||
"compose_files": ["docker-compose.yml", "docker-compose.llm-node.yml"],
|
||||
"env_file": "/etc/shinkan/prod.env",
|
||||
"db_engine": "postgresql",
|
||||
"db_container": "shinkan-db-prod",
|
||||
"db_name": "shinkan",
|
||||
"db_user": "shinkan_user",
|
||||
"media_host_dir": "/srv/shinkan/prod/media",
|
||||
"media_container_dir": "/app/media",
|
||||
"version_url": "http://127.0.0.1:8003/api/version",
|
||||
"ready_url": "http://127.0.0.1:8003/api/health/ready",
|
||||
"frontend_ready_url": "http://127.0.0.1:3003/api/health/ready",
|
||||
"adapter_dir": "/etc/backup-orchestrator/scripts/shinkan",
|
||||
"services_to_stop": ["backend", "frontend", "retention-cron"]
|
||||
},
|
||||
"llm-restoretest": {
|
||||
"application": "shinkan",
|
||||
"environment": "restoretest",
|
||||
"host": "llm-node",
|
||||
"arch": "x86_64",
|
||||
"role": "isolated_test_target",
|
||||
"allow_restore": true,
|
||||
"require_confirm": null,
|
||||
"app_dir": "/home/llmadmin/docker/shinkan-prod",
|
||||
"compose_project": "shinkan-restoretest",
|
||||
"compose_files": ["/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml"],
|
||||
"env_file": "/etc/shinkan/prod.env",
|
||||
"db_engine": "postgresql",
|
||||
"db_container": "shinkan-restoretest-postgres-1",
|
||||
"db_name": "shinkan",
|
||||
"db_user": "shinkan_user",
|
||||
"media_host_dir": "/srv/shinkan/prod/media",
|
||||
"media_container_dir": "/app/media",
|
||||
"version_url": "http://127.0.0.1:18003/api/version",
|
||||
"ready_url": "http://127.0.0.1:18003/api/health/ready",
|
||||
"frontend_ready_url": "http://127.0.0.1:13003/api/health/ready",
|
||||
"adapter_dir": "/etc/backup-orchestrator/scripts/shinkan",
|
||||
"services_to_stop": ["backend", "frontend"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -15,7 +15,7 @@ spec.loader.exec_module(app)
|
||||
|
||||
class OrchestratorTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.temp = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.patches = [patch.object(app, 'legacy_schedules', return_value=[]),
|
||||
@@ -49,7 +49,7 @@ class OrchestratorTests(unittest.TestCase):
|
||||
import time
|
||||
start = time.monotonic()
|
||||
self.assertEqual(app.live_run([j1, j2, j3], self.args), 0)
|
||||
self.assertLess(time.monotonic() - start, 0.60)
|
||||
self.assertLess(time.monotonic() - start, 1.20 if os.name == 'nt' else 0.60)
|
||||
self.assertEqual(event.read_text().splitlines()[-1], 'after')
|
||||
self.assertEqual(self.results()['overall'], 'ok')
|
||||
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Unit and integration tests for Backup-Orchestrator v0.4 Recovery Management."""
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT_DIR = Path(__file__).resolve().parents[1]
|
||||
MODULE = ROOT_DIR / 'orchestrator.py'
|
||||
spec = importlib.util.spec_from_file_location('orchestrator_v04_test', MODULE)
|
||||
app = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(app)
|
||||
|
||||
|
||||
class RecoveryPointAndManifestTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.targets_cfg = app.load_targets_config(ROOT_DIR / 'targets.json')
|
||||
|
||||
def _create_dump_bundle(self, migrations=('001_init', '095_catalog', '096_login_lockout_and_totp'),
|
||||
corrupt_sha=False, with_manifest=True, status='consistent'):
|
||||
dump_path = self.root / 'shinkan-prod.sql'
|
||||
copy_lines = '\n'.join(f'{idx}\t{m}\t2026-10-29 05:00:00' for idx, m in enumerate(migrations, start=1))
|
||||
sql_content = (
|
||||
'-- PostgreSQL database dump\n'
|
||||
'COPY public.schema_migrations (id, migration, executed_at) FROM stdin;\n'
|
||||
f'{copy_lines}\n'
|
||||
'\\.\n'
|
||||
)
|
||||
dump_path.write_text(sql_content, encoding='utf-8')
|
||||
actual_sha = hashlib.sha256(dump_path.read_bytes()).hexdigest()
|
||||
written_sha = ('0' * 64) if corrupt_sha else actual_sha
|
||||
(self.root / 'shinkan-prod.sql.sha256').write_text(f'{written_sha} shinkan-prod.sql\n', encoding='utf-8')
|
||||
|
||||
media_file = self.root / 'shinkan-prod.sql.media.jsonl'
|
||||
media_file.write_text('{"path":"library/c1/image/a.jpg","size":10,"mtime":1700000000,"sha256":"' + ('a' * 64) + '"}\n', encoding='utf-8')
|
||||
media_sha = hashlib.sha256(media_file.read_bytes()).hexdigest()
|
||||
|
||||
if with_manifest:
|
||||
manifest = {
|
||||
'schemaVersion': 1,
|
||||
'created_at': '2026-10-29T05:30:12+01:00',
|
||||
'host': 'raspberrypi5',
|
||||
'target': 'pi-prod',
|
||||
'app_version': '0.8.246',
|
||||
'git_commit': 'deadbeef1234',
|
||||
'migrations': list(migrations),
|
||||
'last_migration': migrations[-1] if migrations else None,
|
||||
'dump_format': 'plain',
|
||||
'dump_bytes': dump_path.stat().st_size,
|
||||
'dump_sha256': actual_sha,
|
||||
'row_counts': {'profiles': 5, 'skills': 120},
|
||||
'media_manifest': {
|
||||
'file': media_file.name,
|
||||
'sha256': media_sha,
|
||||
'file_count': 1,
|
||||
},
|
||||
'recoveryPoint': {
|
||||
'status': status,
|
||||
'validatedAt': '2026-10-29T06:00:00+01:00' if status == 'restore_tested' else None,
|
||||
},
|
||||
}
|
||||
(self.root / 'shinkan-prod.sql.manifest.json').write_text(
|
||||
json.dumps(manifest, indent=2), encoding='utf-8'
|
||||
)
|
||||
return dump_path
|
||||
|
||||
def test_manifest_rejects_plaintext_secrets(self):
|
||||
bad = {
|
||||
'schemaVersion': 1,
|
||||
'created_at': '2026-10-29T05:30:12+01:00',
|
||||
'host': 'raspberrypi5',
|
||||
'dump_sha256': 'a' * 64,
|
||||
'notes': 'PGPASSWORD=supersecretpassword',
|
||||
}
|
||||
with self.assertRaises(ValueError):
|
||||
app.normalize_manifest(bad)
|
||||
|
||||
def test_verify_recovery_point_stages(self):
|
||||
# 1. Legacy dump without manifest -> captured
|
||||
dump_only = self.root / 'legacy.sql'
|
||||
dump_only.write_text('SELECT 1;\n', encoding='utf-8')
|
||||
res_legacy = app.verify_recovery_point(dump_only)
|
||||
self.assertEqual(res_legacy['status'], 'captured')
|
||||
self.assertFalse(res_legacy['valid'])
|
||||
|
||||
# 2. Full bundle -> consistent
|
||||
dump_path = self._create_dump_bundle()
|
||||
res_ok = app.verify_recovery_point(dump_path)
|
||||
self.assertTrue(res_ok['valid'])
|
||||
self.assertEqual(res_ok['status'], 'consistent')
|
||||
self.assertEqual(res_ok['exit_code'], app.EXIT_OK)
|
||||
|
||||
# 3. Restore-tested bundle -> restore_tested
|
||||
dump_tested = self._create_dump_bundle(status='restore_tested')
|
||||
res_tested = app.verify_recovery_point(dump_tested)
|
||||
self.assertTrue(res_tested['valid'])
|
||||
self.assertEqual(res_tested['status'], 'restore_tested')
|
||||
|
||||
# 4. Corrupt SHA256 -> Exit 4 (EXIT_CHECKSUM_MISMATCH)
|
||||
dump_bad = self._create_dump_bundle(corrupt_sha=True)
|
||||
res_bad = app.verify_recovery_point(dump_bad)
|
||||
self.assertFalse(res_bad['valid'])
|
||||
self.assertEqual(res_bad['exit_code'], app.EXIT_CHECKSUM_MISMATCH)
|
||||
|
||||
def test_compatibility_exact_forward_and_incompatible_rollback(self):
|
||||
dump_path = self._create_dump_bundle(migrations=('001_init', '095_catalog'))
|
||||
ver = app.verify_recovery_point(dump_path)
|
||||
|
||||
# Exact match
|
||||
c_exact = app.check_compatibility(
|
||||
ver, 'llm-prod', self.targets_cfg,
|
||||
current_host='llm-node', current_arch='x86_64',
|
||||
code_migrations=['001_init', '095_catalog'],
|
||||
)
|
||||
self.assertTrue(c_exact['compatible'])
|
||||
self.assertEqual(c_exact['schema_relation'], 'EXACT_MATCH')
|
||||
self.assertEqual(c_exact['exit_code'], app.EXIT_OK)
|
||||
|
||||
# Forward migration (dump is older than target code)
|
||||
c_fwd = app.check_compatibility(
|
||||
ver, 'llm-prod', self.targets_cfg,
|
||||
current_host='llm-node', current_arch='x86_64',
|
||||
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
|
||||
)
|
||||
self.assertTrue(c_fwd['compatible'])
|
||||
self.assertEqual(c_fwd['schema_relation'], 'FORWARD_MIGRATE')
|
||||
self.assertEqual(c_fwd['pending_migrations'], ['096_login_lockout_and_totp'])
|
||||
|
||||
# Incompatible rollback (dump has migration 097 unknown to target code)
|
||||
dump_future = self._create_dump_bundle(migrations=('001_init', '095_catalog', '097_future_feature'))
|
||||
ver_future = app.verify_recovery_point(dump_future)
|
||||
c_bad = app.check_compatibility(
|
||||
ver_future, 'llm-prod', self.targets_cfg,
|
||||
current_host='llm-node', current_arch='x86_64',
|
||||
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
|
||||
)
|
||||
self.assertFalse(c_bad['compatible'])
|
||||
self.assertEqual(c_bad['schema_relation'], 'INCOMPATIBLE_ROLLBACK')
|
||||
self.assertEqual(c_bad['exit_code'], app.EXIT_SCHEMA_INCOMPATIBLE)
|
||||
self.assertIn('097_future_feature', c_bad['unknown_migrations'])
|
||||
|
||||
def test_source_only_target_refuses_restore(self):
|
||||
dump_path = self._create_dump_bundle()
|
||||
ver = app.verify_recovery_point(dump_path)
|
||||
c_pi = app.check_compatibility(
|
||||
ver, 'pi-prod', self.targets_cfg,
|
||||
current_host='raspberrypi5', current_arch='aarch64',
|
||||
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
|
||||
)
|
||||
self.assertFalse(c_pi['compatible'])
|
||||
self.assertEqual(c_pi['exit_code'], app.EXIT_PRECONDITION)
|
||||
|
||||
def test_recovery_plan_is_non_destructive_and_complete(self):
|
||||
dump_path = self._create_dump_bundle()
|
||||
plan_doc = app.generate_recovery_plan(
|
||||
dump_path, 'llm-prod', self.targets_cfg,
|
||||
current_host='llm-node', current_arch='x86_64',
|
||||
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
|
||||
)
|
||||
self.assertEqual(plan_doc['mode'], 'DRY_RUN_RECOVERY_PLAN')
|
||||
self.assertTrue(plan_doc['compatibility']['compatible'])
|
||||
self.assertEqual(plan_doc['target']['require_confirm'], 'shinkan@llm-node')
|
||||
self.assertGreaterEqual(len(plan_doc['steps']), 9)
|
||||
|
||||
def test_execute_restore_requires_confirmation_and_records_audit_log(self):
|
||||
dump_path = self._create_dump_bundle()
|
||||
adapter_dir = self.root / 'adapter'
|
||||
adapter_dir.mkdir()
|
||||
fake_restore = adapter_dir / 'shinkan-restore'
|
||||
fake_restore.write_text(
|
||||
'#!/usr/bin/env python3\n'
|
||||
'import json, sys\n'
|
||||
'idx = sys.argv.index("--result")\n'
|
||||
'open(sys.argv[idx+1], "w").write(json.dumps({"status": "ok", "previous_db": "shinkan_before_test"}))\n'
|
||||
'sys.exit(0)\n',
|
||||
encoding='utf-8',
|
||||
)
|
||||
fake_restore.chmod(0o755)
|
||||
|
||||
tcfg = json.loads(json.dumps(self.targets_cfg))
|
||||
tcfg['targets']['llm-prod']['host'] = app.socket.gethostname()
|
||||
tcfg['targets']['llm-prod']['arch'] = app.platform.machine()
|
||||
tcfg['targets']['llm-prod']['adapter_dir'] = str(adapter_dir)
|
||||
|
||||
args_no_confirm = types.SimpleNamespace(
|
||||
target='llm-prod', point=str(dump_path), confirm=None,
|
||||
allow_overwrite=False, sessions='clear', json=False,
|
||||
log_root=self.root / 'logs',
|
||||
)
|
||||
# Missing --confirm must be refused with Exit 2
|
||||
self.assertEqual(app.execute_recovery_command('restore', args_no_confirm, tcfg), app.EXIT_PRECONDITION)
|
||||
self.assertFalse((self.root / 'logs').exists())
|
||||
|
||||
# With valid --confirm shinkan@llm-node, restore executes and writes audit summary
|
||||
args_confirmed = types.SimpleNamespace(
|
||||
target='llm-prod', point=str(dump_path), confirm='shinkan@llm-node',
|
||||
allow_overwrite=False, sessions='clear', json=False,
|
||||
log_root=self.root / 'logs',
|
||||
)
|
||||
self.assertEqual(app.execute_recovery_command('restore', args_confirmed, tcfg), app.EXIT_OK)
|
||||
latest_rec = json.loads((self.root / 'logs' / 'latest-recovery.json').read_text(encoding='utf-8'))
|
||||
self.assertEqual(latest_rec['overall'], 'ok')
|
||||
self.assertEqual(latest_rec['adapter_result']['previous_db'], 'shinkan_before_test')
|
||||
|
||||
# test_shinkan_backup_manifest_script end-to-end
|
||||
media_dir = self.root / 'media'
|
||||
(media_dir / 'library' / 'c1').mkdir(parents=True)
|
||||
(media_dir / 'library' / 'c1' / 'img.png').write_bytes(b'PNGDATA123')
|
||||
out_dir = self.root / 'manifest_out'
|
||||
result_json = self.root / 'manifest_result.json'
|
||||
env = os.environ.copy()
|
||||
env['SHINKAN_SKIP_HOST_GUARD'] = '1'
|
||||
env['SHINKAN_MEDIA_DIR_OVERRIDE'] = str(media_dir)
|
||||
proc = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(ROOT_DIR / 'scripts' / 'shinkan' / 'shinkan-backup-manifest'),
|
||||
'--target', 'pi-prod',
|
||||
'--dump', str(dump_path),
|
||||
'--out-dir', str(out_dir),
|
||||
'--result', str(result_json),
|
||||
],
|
||||
env=env, capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0, msg=proc.stderr)
|
||||
res_data = json.loads(result_json.read_text(encoding='utf-8'))
|
||||
self.assertEqual(res_data['status'], 'ok')
|
||||
self.assertEqual(res_data['media_file_count'], 1)
|
||||
self.assertTrue((out_dir / 'shinkan-prod.sql.manifest.json').is_file())
|
||||
self.assertTrue((out_dir / 'shinkan-prod.sql.sha256').is_file())
|
||||
self.assertTrue((out_dir / 'shinkan-prod.sql.media.jsonl').is_file())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user