feat(v0.4): import all Raspberry Pi scripts and implement Recovery Management engine + Shinkan adapters

This commit is contained in:
Lars committed 2026-10-29 14:18:20 +01:00
1 parent 40188c0e10
commit c53fc0a939
35 files changed
+2877 -191

No files matched your search

+1
View File
@@ -4,3 +4,4 @@
*.csv text eol=lf
*.json text eol=lf
*.md text eol=lf
+1
View File
@@ -8,3 +8,4 @@ __pycache__/
notifications.json
host.local.json
.env
+84 -152
View File
@@ -1,161 +1,93 @@
# Raspberry Pi 5 – Backup-Orchestrator v0.3
# Backup-Orchestrator v0.4 – Backup & Recovery Management
**Status: Test- und Integrationsversion.** Keine produktiven Backups, Timer, Crontabs oder Datenbanken werden durch Installation, `--check`, `--plan` oder `--status` verändert. Die vorhandenen Backup-Skripte sind **noch nicht vollständig standardisiert**, zusätzliche Datenbank-Jobs fehlen und eine vollständige Systemwiederherstellung ist noch nicht getestet. `--run` bleibt bewusst mehrfach gesperrt.
**Status: Version 0.4.0 (Multi-Host Backup & Application Recovery Management).**
Erweitert den auf `raspberrypi5` produktiv laufenden Backup-Orchestrator v0.3 um ein integriertes Recovery Management, kryptografisch geprüfte Recovery Points (`schemaVersion: 1`), ein 4-stufiges Verifikationsmodell (`captured` → `verified` → `consistent` → `restore_tested`), einen zustandsbewussten `raspiBackup.service`-Konflikt-Check sowie vollständige Application-Restore- und Restore-Test-Adapter für die Referenzanwendung **Shinkan** auf `llm-node` (`192.168.2.144`).
## 1. Enthaltene Dateien
---
- `orchestrator.py`: phasenbasierter Python-Controller (Python 3.11 / Debian 12, nur Standardbibliothek).
- `jobs.csv`: bestehende acht Jobs. **Alle `enabled=false`** und verweisen derzeit auf die alten, vorhandenen Skriptpfade.
- `notifications.json.example`: eine zentrale Konfigurationsdatei für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore; keine echten Secrets.
- `tests/test_orchestrator.py`: Tests mit künstlichen Shellskripten und simulierten HTTP-Antworten; niemals produktive Datenbanken oder das NAS.
## 1. Repository-Struktur & Enthaltene Dateien
## 2. Sichere Installation auf raspberrypi5
- `orchestrator.py`: Phasenbasierter Backup- & Recovery-Controller (`v0.4.0`, Python 3.11+, nur Standardbibliothek).
- `targets.json`: Multi-Host- und Recovery-Target-Definitionen (`raspberrypi5`, `llm-node`, Targets `pi-prod`, `llm-prod`, `llm-restoretest`).
- `jobs.csv`: Produktive Backup-Job-Tabelle (17 Datenbank-Exporte in Phase 10, `nas-data` in Phase 20, `system-image` in Phase 30).
- `notifications.json.example`: Beispielkonfiguration für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore (enthält keine echten Secrets).
- `systemd/`:
- `backup-orchestrator-daily`: Täglicher Runner-Wrapper (ab v0.4 ohne fragiles `systemctl disable/enable raspiBackup.timer`).
- `backup-orchestrator-daily.service` & `backup-orchestrator-daily.timer`: Systemd-Units für den täglichen Lauf um `05:30` Uhr.
- `scripts/`:
- Alle 17 Datenbank-Dump-Wrapper (`db-bookstack.sh`, `db-buyingagent-dev.sh`, `db-buyingagent-prod.sh`, `db-kairo-dev.sh`, `db-kairo-prod.sh`, `db-kansho-dev.sh`, `db-kansho-prod.sh`, `db-karatewiki-dev.sh`, `db-mitai-dev.sh`, `db-mitai-prod.sh`, `db-netbox.sh`, `db-paperless.sh`, `db-shinkan-dev.sh`, `db-shinkan-prod.sh`, `db-tandoor.sh`, `db-tandoor.sh`, `db-tsumugu-dev.sh`, `db-tsumugu-prod.sh`) sowie die generischen Dump-Engines `db-postgres-docker.sh` und `db-mariadb-docker.sh`.
- `backup-nas-data.sh` & `nas-data-job.sh`: Versionierte rsync-Hardlink-Sicherung (`/media/lars/paperless` → `/backup/data_archiv/01..31` + `last`).
- `publish-backup-set.sh` & `shinkan-prod-manifest.sh`: Atomare Erzeugung und Publikation vollständiger Backup-Sets (`<dump>.sql`, `<dump>.sql.sha256`, `<dump>.sql.manifest.json`, `<dump>.sql.media.jsonl`).
- `scripts/shinkan/` (Shinkan Recovery Contract Adapter):
- `shinkan-backup-manifest`: Erzeugt SHA256, Manifest (`schemaVersion: 1` + Shinkan Contract) und inkrementelles Media-Inventar (`media.jsonl`).
- `shinkan-restore`: 9-stufiger atomarer Application-Restore über Staging-DB (`shinkan_restore_<ts>`), Single-Transaction-Import, Schema-Migrations-Gate, Plausibilitätsprüfung, `TRUNCATE sessions`, DB-Swap (`shinkan_before_<ts>`), `/api/health/ready`-Prüfung und automatischem Rollback bei Fehlern (Exit `5`).
- `shinkan-media-diff`: Abgleich zwischen PostgreSQL (`media_assets`, `exercise_media`), Dateisystem und `media.jsonl` (`missing`, `mismatch`, `added_after_backup`, `orphan`).
- `shinkan-restore-test` & `docker-compose.restoretest.yml`: Isolierter End-to-End-Restore-Test im Projekt `shinkan-restoretest` (Ports `127.0.0.1:18003` / `127.0.0.1:13003`, Read-Only-Media `:ro`, deaktivierte externe Integrationen & Cronjobs).
- `docs/ARCHITECTURE_AND_RECOVERY_PLAN_v0.4.md`: Vollständige Ist-Inventur, Gap-Analyse und Architektur-Spezifikation für v0.4.
- `tests/`:
- `test_orchestrator.py`: Unit-Tests für Backup-Phasen, Parallelität, Timeouts, Mount-Prüfung und Notifications.
- `test_recovery.py`: Unit- und Integrationstests für Manifest-Validierung, 4-Stufen-Modell, Schema-Kompatibilitätsprüfung (`EXACT_MATCH`, `FORWARD_MIGRATE`, `INCOMPATIBLE_ROLLBACK`), Sicherheits-Gates (`pi-prod` Sperre, `--confirm`), Dry-Run-Pläne und `shinkan-backup-manifest`.
ZIP auf den Raspberry Pi übertragen und dort entpacken. Im entpackten Verzeichnis:
---
## 2. CLI-Befehle (v0.4.0)
### A. Backup-Steuerung (unverändert kompatibel zu v0.3)
```bash
backup-orchestrator --check
backup-orchestrator --plan
backup-orchestrator --status --json
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES backup-orchestrator --run
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES backup-orchestrator --run --phase 10
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES backup-orchestrator --run --job shinkan-prod-db
```
### B. Recovery Management (neu in v0.4)
```bash
# 1. Verfügbare Recovery Points auflisten (inkl. 4-Stufen-Status)
backup-orchestrator --list-recovery-points
backup-orchestrator --list-recovery-points --app shinkan --json
# 2. Einzelnen Recovery Point kryptografisch & strukturell prüfen
backup-orchestrator --verify-recovery-point \
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql
# 3. Dry-Run Recovery-Plan erstellen (inkl. Schema-Kompatibilitätsprüfung)
backup-orchestrator --recovery-plan \
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql \
--target llm-prod
# 4. Isolierten Restore-Test ausführen (schreibgeschütztes Media-Mount, Ports 18003/13003)
backup-orchestrator --restore-test \
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql \
--target llm-restoretest
# 5. Produktiver Application-Restore / Erstbefüllung auf llm-node
backup-orchestrator --restore \
--point /mnt/nas_backup/shinkan/prod/db/latest/shinkan-prod.sql \
--target llm-prod \
--confirm shinkan@llm-node
```
---
## 3. Exit-Codes im Recovery-Modus
| Exit-Code | Bedeutung |
|---|---|
| `0` | Erfolgreich (`verified`/`consistent`/`restore_tested` bzw. Restore + `/api/health/ready` grün) |
| `2` | Ungültige Argumente, fehlendes `--confirm`, gesperrtes Ziel (`pi-prod` ist `source_only`) oder Mount/Pfad fehlt |
| `3` | Schema-/Versions-Inkompatibilität (`INCOMPATIBLE_ROLLBACK`: Dump enthält Migrationen, die im Ziel-Code fehlen) |
| `4` | Integritätsfehler (SHA256-Prüfsumme weicht ab, Dump leer/korrupt oder Pflicht-Tabellen nach Staging-Import leer) |
| `5` | Health-Check (`/api/health/ready`) nach DB-Umschaltung fehlgeschlagen – **automatisches Rollback auf `shinkan_before_<ts>` durchgeführt** |
---
## 4. Tests ausführen
```bash
python3 -m unittest discover -s tests -v
sudo cp -a /usr/local/sbin/backup-orchestrator /usr/local/sbin/backup-orchestrator.before-v03
sudo install -m 755 orchestrator.py /usr/local/sbin/backup-orchestrator
# Die vorhandene /etc/backup-orchestrator/jobs.csv NICHT überschreiben.
# Die Benachrichtigungskonfiguration nur bei erstmaliger Einrichtung kopieren.
sudo install -d -m 750 /etc/backup-orchestrator
if ! sudo test -e /etc/backup-orchestrator/notifications.json; then
sudo install -m 600 notifications.json.example /etc/backup-orchestrator/notifications.json
fi
sudo backup-orchestrator --check
```
Ohne bereits installierte ältere Version entfällt die Zeile `sudo cp -a ...before-v03`.
> **Keine `--run`-Befehle ausführen**, solange die alten Cronjobs und der raspiBackup-Timer aktiv sind und die alten Dump-Skripte noch nicht gehärtet wurden. Den Produktiv-Cutover führen wir später kontrolliert durch.
## 3. Konfiguration der Phasen
Die vorhandene Tabelle unter `/etc/backup-orchestrator/jobs.csv` hat **keine Uhrzeiten**:
```csv
phase,id,script,critical,timeout_seconds,enabled
10,paperless-db,/home/lars/db-backup.sh,true,600,false
10,tandoor-db,/home/lars/db-Tandoor.sh,true,600,false
20,nas-data,/home/lars/backupPaperlessData.sh,true,10800,false
30,system-image,/usr/local/bin/raspiBackup.sh,true,14400,false
```
Die Beispiele sind ein Ausschnitt; die mitgelieferte Datei enthält acht Jobs.
- **Innerhalb derselben Phase** laufen bis zu drei Jobs gleichzeitig (per `--max-parallel N` einstellbar).
- **Zwischen Phasen** gilt eine Barriere: Die nächste Phase beginnt erst, wenn die aktuelle vollständig abgeschlossen ist.
- Ein **kritischer Fehler** blockiert nachfolgende Phasen; laufende Jobs derselben Phase werden beendet bzw. bis zum Abschluss abgewartet.
- Ein **unkritischer Fehler** wird protokolliert und führt zu einem fehlgeschlagenen Gesamtergebnis, blockiert aber die nächste Phase nicht.
- Zeitüberschreitung beendet die Skript-Prozessgruppe; Service-Neustart und Datenbankkonsistenz müssen in den jeweiligen Jobs abgesichert sein.
Zentrale Zeitsteuerung per *einem* systemd-Timer ist eine spätere Cutover-Aufgabe, **nicht Bestandteil der Installation**.
## 4. Externe Schnittstellen
### A. Ausgehende Webhooks, HTTP POST
`/etc/backup-orchestrator/notifications.json` bearbeiten:
```json
{
"timeout_seconds": 3,
"webhooks": ["http://192.168.2.100:5678/webhook/backup-events"],
"kuma": {
"overall_push_url": "http://192.168.2.63:3010/api/push/GESAMT_MONITOR_TOKEN",
"job_push_urls": {
"nas-data": "http://192.168.2.63:3010/api/push/NAS_MONITOR_TOKEN",
"paperless-db": "http://192.168.2.63:3010/api/push/PAPERLESS_MONITOR_TOKEN"
}
}
}
```
Alle URLs sind Beispiele – vorhandene echte Push-Tokens nur auf dem Pi eintragen. Einträge mit `""` sind deaktiviert. Es gibt **keinen HTTP-Listener und keinen permanent laufenden Webserver**; Webhooks sind *ausgehend*.
Die echten URLs können auch komplett aus der Uptime-Kuma-Oberfläche kopiert werden (`...?status=up&msg=OK&ping=`). Diese Beispiel-Query wird beim Laden entfernt und beim Push durch den tatsächlichen Status ersetzt.
Diese Datei enthält Zugriffstokens: `sudo chmod 600 /etc/backup-orchestrator/notifications.json`. Sie gehört in eine vertrauliche Konfigurationssicherung, nicht in ein öffentliches Repository.
### B. Ereignisse und Beispiele
Alle Ereignisse werden als **JSON je Zeile** in `/var/log/backup-orchestrator/<run_id>/events.jsonl` geschrieben; jede Zeile wird auch an die konfigurierten POST-Webhooks gesendet.
- `RUN_STARTED`, `PHASE_STARTED`, `JOB_STARTED`
- `JOB_SUCCESS`, `JOB_FAILED`, `JOB_TIMEOUT`, `JOB_ERROR`, `JOB_CANCELLED`, `JOB_BLOCKED`
- `PHASE_COMPLETED`, `PHASE_BLOCKED`, `RUN_FINISHED`
Beispiel einer Webhook-Nachricht (ohne Geheimnisse):
```json
{
"event": "JOB_FAILED",
"timestamp": "2026-10-29T01:02:00+01:00",
"run_id": "20261029-010000-1234",
"host": "raspberrypi5",
"phase": 10,
"job": "shinkan-prod-db",
"critical": true,
"state": "failed",
"exit_code": 1,
"duration_seconds": 12.3,
"log": "/var/log/backup-orchestrator/20261029-010000-1234/shinkan-prod-db.log"
}
```
Webhooks versenden **Ereignismetadaten**, nicht den Inhalt der Backup-Skripte oder SQL-Dumps.
### C. Uptime Kuma
- `JOB_SUCCESS` → `status=up` für den konfigurierten Job-Monitor.
- `JOB_FAILED`, `JOB_TIMEOUT`, `JOB_ERROR`, `JOB_CANCELLED` und `JOB_BLOCKED` → `status=down`.
- `RUN_FINISHED` → `status=up` nur beim Gesamtergebnis `ok`, sonst `down`.
- Ausbleibende Backups meldet Kuma über den konfigurierten Push-Heartbeat, z.B. bei einem täglichen Lauf ein zum Zeitplan passender Wert.
- Schlägt die Zustellung fehl, schreibt der Controller eine Warnung in `notifications.log`. Der Backup-Exit-Code bleibt unverändert.
**Keine doppelten Push-Meldungen erzeugen:** Wenn `nas-data` künftig durch den Orchestrator gemeldet wird, die Push-Funktion im Skript `backupPaperlessData.sh` deaktivieren oder nur einen der beiden Sender verwenden.
### D. Lokaler Status, JSON-Abfrage
Während des Laufs wird der aktuelle Stand atomar aktualisiert:
```text
/var/log/backup-orchestrator/latest.json
/var/log/backup-orchestrator/<run_id>/status.json
/var/log/backup-orchestrator/<run_id>/summary.json
/var/log/backup-orchestrator/<run_id>/events.jsonl
```
CLI für andere Skripte, Home Assistant oder eine spätere GUI:
```bash
sudo backup-orchestrator --status
sudo backup-orchestrator --status --json
sudo cat /var/log/backup-orchestrator/latest.json
```
Die Statusdateien enthalten keine Push-Tokens oder Zugangsdaten, aber interne Pfade und Jobnamen. Daher nur für vertrauenswürdige lokale Nutzer freigeben. Es gibt derzeit **keine eingehende REST-API**; dies kann später als separater schreibgeschützter Dienst ergänzt werden.
## 5. Sichere Bedienung
```bash
sudo backup-orchestrator --check # nur Prüfungen, keine Backups
sudo backup-orchestrator --plan # nur aktivierte Jobs anzeigen
sudo backup-orchestrator --status --json # letzter Status bzw. no_runs
```
`--run`, `--run --phase 10` und `--run --job paperless-db` sind für einen **späteren**, explizit kontrollierten Produktiv-Cutover vorgesehen. Der Controller verweigert Ausführung, solange keine Jobs aktiviert sind, der NAS-Mount fehlt, aktive alte Zeitpläne gefunden werden oder `BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES` fehlt. Das stellt keine vollständige Sicherheitsfreigabe dar: Skripte und Restore müssen zusätzlich getestet werden.
## 6. Offene Arbeiten für den Produktivbetrieb
1. Vorhandene Datenbank-Dump-Skripte unter `scripts/` konsolidieren und auf atomare Dumps unter festen Dateinamen umstellen; Zugänge sicher verwahren.
2. Weitere Datenbanken (Kairo, Tsumugu, Kanshō, Buying Agent, BookStack, NetBox usw.) und zu sichernde Docker-Volumes/Bind-Mounts aufnehmen.
3. rsync-Skript prüfen: nas-data sendet bei der Umstellung **nicht** gleichzeitig selbst und über Orchestrator Pushs.
4. raspiBackup: zuverlässiger vollständiger Restore, korrekt gestoppte Docker-Dienste, Restart-/Timeout-Verhalten nach Fehler.
5. Alte Cronjobs/Timer **erst beim kontrollierten Cutover** ersetzen und einen einzigen orchestrator-Timer installieren.
6. Vollständigen Testlauf, Restore-Test und erwartete Kuma-Ereignisse prüfen.
7. Zweiten unabhängigen/offsite Backup-Standort erwägen; NAS-Ausfall bleibt sonst Single Point of Failure.
**Hinweis zur bisherigen Zeitplanung:** Die Root-Crontab-Einträge sind nach der letzten Nutzerprüfung auskommentiert; die aktive User-Crontab sowie `raspiBackup.timer` werden weiterhin als aktive Konflikte erkannt.
Alle Tests arbeiten ausschließlich in temporären Verzeichnissen mit synthetischen Dumps und berühren niemals produktive Datenbanken oder NAS-Mounts.
+954 -37
View File
File diff suppressed because it is too large. Load diff
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-mariadb-docker.sh \
"bookstack_db" "bookstack.sql" "mysql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "buyingagent-dev-postgres-1" "buyingagent-dev.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "buyingagent-db-prod" "buyingagent-prod.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kairo-dev-postgres-1" "kairo-dev.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kairo-db-prod" "kairo-prod.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kansho-dev-postgres-1" "kansho-dev.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "kansho-db-prod" "kansho-prod.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-mariadb-docker.sh \
"karatewiki-dev-db-1" "karatewiki-dev.sql" "mariadb"
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
"dev-mitai-postgres" \
"mitai-dev.sql" \
"mitai_dev" \
"mitai_dev"
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
"mitai-db-prod" \
"mitai-prod.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
"netbox-postgres-1" "netbox.sql"
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
set -Eeuo pipefail
umask 077
# Optionales lokales Credential-File (chmod 600, nicht in Git)
if [[ -f /etc/backup-orchestrator/credentials/paperless.env ]]; then
# shellcheck disable=SC1091
source /etc/backup-orchestrator/credentials/paperless.env
fi
# === Datenbank ===
DB_HOST="${DB_HOST:-192.168.2.40}"
DB_PORT="${DB_PORT:-3308}"
DB_USER="${DB_USER:-paperless}"
DB_PASSWORD="${DB_PASSWORD:-HIER_PASSWORT_EINTRAGEN}"
DB_NAME="${DB_NAME:-paperless}"
# === Backup ===
BACKUP_DIR="${BACKUP_DIR:-/media/lars/paperless/db-backup}"
BACKUP_FILE="$BACKUP_DIR/paperless.sql"
[[ "$DB_PASSWORD" != "HIER_PASSWORT_EINTRAGEN" ]] || {
echo "FEHLER: Datenbankpasswort nicht konfiguriert" >&2
exit 1
}
mkdir -p "$BACKUP_DIR"
TEMP_FILE="$(mktemp "$BACKUP_DIR/.paperless.XXXXXXXX.sql")"
trap 'rm -f -- "$TEMP_FILE"' EXIT
echo "Paperless: Datenbankexport gestartet"
MYSQL_PWD="$DB_PASSWORD" mysqldump \
--host="$DB_HOST" \
--port="$DB_PORT" \
--user="$DB_USER" \
--single-transaction \
--quick \
--databases "$DB_NAME" \
> "$TEMP_FILE"
if [[ ! -s "$TEMP_FILE" ]]; then
echo "FEHLER: Paperless-Dump ist leer" >&2
exit 1
fi
chgrp lars "$TEMP_FILE"
chmod 640 "$TEMP_FILE"
mv -f -- "$TEMP_FILE" "$BACKUP_FILE"
echo "OK: Paperless-Datenbank gesichert"
echo "Datei: $BACKUP_FILE"
exit 0
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
"shinkan-dev-postgres-1" \
"shinkan-dev.sql"
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh \
"shinkan-db-prod" \
"shinkan-prod.sql"
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
set -Eeuo pipefail
umask 077
# Optionales lokales Credential-File (chmod 600, nicht in Git)
if [[ -f /etc/backup-orchestrator/credentials/tandoor.env ]]; then
# shellcheck disable=SC1091
source /etc/backup-orchestrator/credentials/tandoor.env
fi
# === Datenbank ===
PGHOST="${PGHOST:-192.168.2.40}"
PGPORT="${PGPORT:-5432}"
PGUSER="${PGUSER:-djangouser}"
PGDATABASE="${PGDATABASE:-djangodb}"
PGPASSWORD="${PGPASSWORD:-HIER_PASSWORT_EINTRAGEN}"
# === Backup ===
BACKUP_DIR="${BACKUP_DIR:-/media/lars/paperless/db-backup}"
BACKUP_FILE="$BACKUP_DIR/tandoor.sql"
[[ "$PGPASSWORD" != "HIER_PASSWORT_EINTRAGEN" ]] || {
echo "FEHLER: Datenbankpasswort nicht konfiguriert" >&2
exit 1
}
mkdir -p "$BACKUP_DIR"
TEMP_FILE="$(mktemp "$BACKUP_DIR/.tandoor.XXXXXXXX.sql")"
trap 'rm -f -- "$TEMP_FILE"' EXIT
echo "Tandoor: Datenbankexport gestartet"
PGPASSWORD="$PGPASSWORD" pg_dump \
--host="$PGHOST" \
--port="$PGPORT" \
--username="$PGUSER" \
--dbname="$PGDATABASE" \
--format=plain \
> "$TEMP_FILE"
if [[ ! -s "$TEMP_FILE" ]]; then
echo "FEHLER: Tandoor-Dump ist leer" >&2
exit 1
fi
chgrp lars "$TEMP_FILE"
chmod 640 "$TEMP_FILE"
mv -f -- "$TEMP_FILE" "$BACKUP_FILE"
echo "OK: Tandoor-Datenbank gesichert"
echo "Datei: $BACKUP_FILE"
exit 0
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "tsumugu-dev-postgres-1" "tsumugu-dev.sql"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec /etc/backup-orchestrator/scripts/db-postgres-docker.sh "tsumugu-db-prod" "tsumugu-prod.sql"
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -Eeuo pipefail
exec runuser -u lars -- /bin/bash -s \
< /etc/backup-orchestrator/scripts/backup-nas-data.sh
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# ============================================================================
# publish-backup-set.sh
# Atomically publishes a complete Backup Set (Dump + .sha256 + .manifest.json
# + optional .media.jsonl) to the NAS generation archive and rotates old sets.
# Implements RECOVERY_CONTRACT.md section 5.1 & 5.2.
# ============================================================================
set -Eeuo pipefail
umask 027
if [[ $# -lt 3 ]]; then
echo "Verwendung: $0 <TARGET_ID> <DUMP_DATEI> <ARCHIVE_ROOT>" >&2
exit 2
fi
TARGET_ID="$1"
DUMP_FILE="$2"
ARCHIVE_ROOT="$3"
SHA_FILE="${DUMP_FILE}.sha256"
MANIFEST_FILE="${DUMP_FILE}.manifest.json"
MEDIA_FILE="${DUMP_FILE}.media.jsonl"
[[ -s "$DUMP_FILE" ]] || { echo "FEHLER: Dump fehlt oder ist leer: $DUMP_FILE" >&2; exit 2; }
[[ -s "$SHA_FILE" ]] || { echo "FEHLER: SHA256-Datei fehlt: $SHA_FILE" >&2; exit 2; }
[[ -s "$MANIFEST_FILE" ]] || { echo "FEHLER: Manifest-Datei fehlt: $MANIFEST_FILE" >&2; exit 2; }
# Verify checksum before publishing
(
cd "$(dirname "$DUMP_FILE")"
sha256sum -c "$(basename "$SHA_FILE")" >/dev/null
) || {
echo "FEHLER: SHA256-Pruefsumme stimmt nicht!" >&2
exit 4
}
TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
TARGET_DIR="$ARCHIVE_ROOT/$TARGET_ID"
mkdir -p "$TARGET_DIR"
STAGING_DIR="$(mktemp -d "$TARGET_DIR/.staging-${TIMESTAMP}.XXXXXXXX")"
FINAL_DIR="$TARGET_DIR/$TIMESTAMP"
cleanup() {
if [[ -n "${STAGING_DIR:-}" && -d "$STAGING_DIR" ]]; then
rm -rf -- "$STAGING_DIR" || true
fi
}
trap cleanup EXIT
cp -p -- "$DUMP_FILE" "$STAGING_DIR/"
cp -p -- "$SHA_FILE" "$STAGING_DIR/"
cp -p -- "$MANIFEST_FILE" "$STAGING_DIR/"
if [[ -f "$MEDIA_FILE" ]]; then
cp -p -- "$MEDIA_FILE" "$STAGING_DIR/"
fi
mv -T -- "$STAGING_DIR" "$FINAL_DIR"
STAGING_DIR=""
TEMP_LINK="$TARGET_DIR/.latest-$$"
ln -s -- "$FINAL_DIR" "$TEMP_LINK"
mv -Tf -- "$TEMP_LINK" "$TARGET_DIR/latest"
echo "OK: Backup-Satz veroeffentlicht unter $FINAL_DIR (latest aktualisiert)"
exit 0
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
set -Eeuo pipefail
DUMP_FILE="${SHINKAN_DUMP_FILE:-/media/lars/paperless/db-backup/shinkan-prod.sql}"
OUT_DIR="$(dirname "$DUMP_FILE")"
TARGET="${SHINKAN_TARGET:-pi-prod}"
ARCHIVE_ROOT="${SHINKAN_ARCHIVE_ROOT:-/backup/recovery_points}"
RESULT_JSON="${OUT_DIR}/.shinkan-manifest-last-result.json"
/usr/bin/env python3 /etc/backup-orchestrator/scripts/shinkan/shinkan-backup-manifest \
--target "$TARGET" \
--dump "$DUMP_FILE" \
--out-dir "$OUT_DIR" \
--result "$RESULT_JSON"
if [[ -d "$(dirname "$ARCHIVE_ROOT")" && -w "$(dirname "$ARCHIVE_ROOT")" ]]; then
/etc/backup-orchestrator/scripts/publish-backup-set.sh \
"shinkan-${TARGET}" \
"$DUMP_FILE" \
"$ARCHIVE_ROOT"
fi
@@ -0,0 +1,73 @@
# Isolated disposable Restore-Test Compose configuration for llm-node.
# Implements RECOVERY_CONTRACT.md section 3.3:
# - No fixed container_name (prevents collision with shinkan-prod)
# - Localhost-only ports (18003 / 13003), no published postgres port
# - Shared media mounted strictly read-only (:ro)
# - No retention-cron service, SMTP and OpenRouter disabled
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: "${DB_NAME:-shinkan}"
POSTGRES_USER: "${DB_USER:-shinkan_user}"
POSTGRES_PASSWORD: "${DB_PASSWORD}"
volumes:
- restoretest-db-data:/var/lib/postgresql/data
networks:
- restoretest-net
backend:
build:
context: ./backend
dockerfile: Dockerfile
environment:
DB_HOST: postgres
DB_PORT: 5432
DB_NAME: shinkan
DB_USER: shinkan_user
DB_PASSWORD: "${DB_PASSWORD}"
OPENROUTER_API_KEY: ""
OPENROUTER_MODEL: ""
SMTP_HOST: ""
SMTP_PORT: ""
SMTP_USER: ""
SMTP_PASS: ""
SMTP_FROM: ""
AUTO_ADMIN_FIRST_USER: "false"
ADMIN_BOOTSTRAP_EMAILS: ""
ENVIRONMENT: "production"
MEDIA_ROOT: "/app/media"
volumes:
- type: bind
source: /srv/shinkan/prod/media
target: /app/media
read_only: true
bind:
create_host_path: false
ports:
- "127.0.0.1:18003:8000"
depends_on:
- postgres
networks:
- restoretest-net
frontend:
build:
context: ./frontend
dockerfile: Dockerfile
args:
VITE_API_URL: ""
ports:
- "127.0.0.1:13003:80"
depends_on:
- backend
networks:
- restoretest-net
volumes:
restoretest-db-data:
networks:
restoretest-net:
driver: bridge
+354
View File
@@ -0,0 +1,354 @@
#!/usr/bin/env python3
"""Generate Shinkan backup manifest, SHA256 sidecar, and incremental media.jsonl index.
Implements RECOVERY_CONTRACT.md section 3.1 and Backup-Orchestrator v0.4 Manifest Schema v1.
Read-only against database and media storage; writes sidecar files next to the dump.
"""
import argparse
import datetime as dt
import hashlib
import json
import os
from pathlib import Path
import platform
import re
import socket
import subprocess
import sys
from urllib.request import urlopen
EXIT_OK = 0
EXIT_EXEC_ERROR = 1
EXIT_PRECONDITION = 2
TARGETS = {
'pi-prod': {
'host': 'raspberrypi5',
'arch': 'aarch64',
'app_dir': '/home/lars/docker/shinkan',
'db_container': 'shinkan-db-prod',
'db_name': 'shinkan',
'db_user': 'shinkan_user',
'media_dir': '/shinkan-media/prod',
'version_url': 'http://127.0.0.1:8003/api/version',
},
'llm-prod': {
'host': 'llm-node',
'arch': 'x86_64',
'app_dir': '/home/llmadmin/docker/shinkan-prod',
'db_container': 'shinkan-db-prod',
'db_name': 'shinkan',
'db_user': 'shinkan_user',
'media_dir': '/srv/shinkan/prod/media',
'version_url': 'http://127.0.0.1:8003/api/version',
},
}
def now_iso():
return dt.datetime.now(dt.timezone.utc).astimezone().isoformat(timespec='seconds')
def compute_sha256(path):
h = hashlib.sha256()
with Path(path).open('rb') as fp:
while True:
chunk = fp.read(65536)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
def detect_dump_format(path):
with Path(path).open('rb') as fp:
return 'custom' if fp.read(5) == b'PGDMP' else 'plain'
def atomic_write_text(path, content):
path = Path(path)
tmp = path.with_name(f'.{path.name}.{os.getpid()}.tmp')
try:
tmp.write_text(content, encoding='utf-8')
tmp.replace(path)
finally:
if tmp.exists():
tmp.unlink()
def query_version_info(tcfg):
app_version = 'unknown'
try:
with urlopen(tcfg['version_url'], timeout=5) as resp:
if resp.status == 200:
data = json.loads(resp.read(8192).decode('utf-8'))
app_version = data.get('app_version', 'unknown')
except Exception:
pass
git_commit = 'unknown'
app_dir = Path(tcfg['app_dir'])
if (app_dir / '.git').exists():
try:
proc = subprocess.run(
['git', '-C', str(app_dir), 'rev-parse', 'HEAD'],
capture_output=True, text=True, timeout=5,
)
if proc.returncode == 0 and proc.stdout.strip():
git_commit = proc.stdout.strip()
except Exception:
pass
return app_version, git_commit
def query_db_metadata(tcfg, dump_path):
"""Query migrations and informative table row counts from container or fallback to SQL dump."""
migrations = []
row_counts = {}
engine_version = ''
sql_script = """
SELECT 'MIG:' || migration FROM schema_migrations ORDER BY id;
SELECT 'VER:' || server_version FROM pg_settings WHERE name = 'server_version';
SELECT 'CNT:' || relname || ':' || n_live_tup::bigint
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY relname;
"""
try:
proc = subprocess.run(
[
'docker', 'exec', tcfg['db_container'],
'sh', '-ec',
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
'psql -U "${POSTGRES_USER:-shinkan_user}" -d "${POSTGRES_DB:-shinkan}" -At -c '
+ json.dumps(sql_script),
],
capture_output=True, text=True, timeout=15,
)
if proc.returncode == 0:
for line in proc.stdout.splitlines():
if line.startswith('MIG:'):
migrations.append(line[4:].strip())
elif line.startswith('VER:'):
engine_version = line[4:].strip()
elif line.startswith('CNT:'):
parts = line[4:].split(':', 1)
if len(parts) == 2 and parts[1].isdigit():
row_counts[parts[0]] = int(parts[1])
except Exception:
pass
# Fallback: parse migrations from plain SQL dump if container query was unavailable
if not migrations and detect_dump_format(dump_path) == 'plain':
in_copy = False
with Path(dump_path).open('r', encoding='utf-8', errors='replace') as fp:
for line in fp:
if not in_copy:
if line.startswith('COPY public.schema_migrations ') or line.startswith('COPY schema_migrations '):
in_copy = True
continue
else:
if line.startswith('\\.'):
break
parts = line.rstrip('\r\n').split('\t')
if len(parts) >= 2:
migrations.append(parts[1])
def sort_key(stem):
m = re.match(r'^(\d+)', stem)
return (int(m.group(1)) if m else 0, stem)
migrations = sorted(set(migrations), key=sort_key)
return migrations, row_counts, engine_version
def build_media_jsonl(media_dir, out_jsonl_path, cache_path):
"""Write <dump>.media.jsonl with incremental sha256 cache over (path, size, mtime_ns)."""
media_root = Path(media_dir)
cache = {}
if cache_path.is_file():
try:
cache = json.loads(cache_path.read_text(encoding='utf-8'))
except Exception:
cache = {}
new_cache = {}
lines = []
total_bytes = 0
if media_root.is_dir():
for item in sorted(media_root.rglob('*')):
if not item.is_file() or item.name.startswith('.'):
continue
rel = item.relative_to(media_root).as_posix()
if rel.startswith('#recycle/') or rel.startswith('.Trash'):
continue
st = item.stat()
size = st.st_size
mtime = int(st.st_mtime)
cache_key = f'{rel}|{size}|{mtime}'
sha = cache.get(cache_key)
if not sha:
sha = compute_sha256(item)
new_cache[cache_key] = sha
total_bytes += size
lines.append(json.dumps({
'path': rel, 'size': size, 'mtime': mtime, 'sha256': sha,
}, ensure_ascii=False))
atomic_write_text(out_jsonl_path, ('\n'.join(lines) + '\n') if lines else '')
try:
atomic_write_text(cache_path, json.dumps(new_cache, ensure_ascii=False))
except OSError:
pass
return len(lines), total_bytes, compute_sha256(out_jsonl_path)
def write_result(path, payload):
if path:
atomic_write_text(path, json.dumps(payload, ensure_ascii=False, indent=2) + '\n')
def main():
parser = argparse.ArgumentParser(description='Create Shinkan backup manifest and checksums')
parser.add_argument('--target', required=True, choices=sorted(TARGETS))
parser.add_argument('--dump', required=True, type=Path)
parser.add_argument('--out-dir', type=Path)
parser.add_argument('--result', type=Path)
args = parser.parse_args()
tcfg = dict(TARGETS[args.target])
if os.environ.get('SHINKAN_MEDIA_DIR_OVERRIDE'):
tcfg['media_dir'] = os.environ['SHINKAN_MEDIA_DIR_OVERRIDE']
if os.environ.get('SHINKAN_APP_DIR_OVERRIDE'):
tcfg['app_dir'] = os.environ['SHINKAN_APP_DIR_OVERRIDE']
cur_host = socket.gethostname()
cur_arch = platform.machine()
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
if cur_host != tcfg['host'] or cur_arch != tcfg['arch']:
msg = f"Host/arch mismatch: running on {cur_host}/{cur_arch}, expected {tcfg['host']}/{tcfg['arch']}"
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
dump_path = args.dump.resolve()
if not dump_path.is_file() or dump_path.stat().st_size == 0:
msg = f'Dump file missing or empty: {dump_path}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
out_dir = (args.out_dir or dump_path.parent).resolve()
out_dir.mkdir(parents=True, exist_ok=True)
try:
dump_sha256 = compute_sha256(dump_path)
dump_bytes = dump_path.stat().st_size
dump_format = detect_dump_format(dump_path)
sha_file = out_dir / (dump_path.name + '.sha256')
atomic_write_text(sha_file, f'{dump_sha256} {dump_path.name}\n')
media_file = out_dir / (dump_path.name + '.media.jsonl')
cache_file = out_dir / '.shinkan-media-sha256-cache.json'
media_count, media_bytes, media_sha256 = build_media_jsonl(tcfg['media_dir'], media_file, cache_file)
app_version, git_commit = query_version_info(tcfg)
migrations, row_counts, engine_version = query_db_metadata(tcfg, dump_path)
last_migration = migrations[-1] if migrations else None
created_at = now_iso()
point_id = f"shinkan-{args.target}-{dt.datetime.now().strftime('%Y%m%dT%H%M%S')}-{dump_sha256[:8]}"
manifest = {
'schemaVersion': 1,
'created_at': created_at,
'host': cur_host,
'target': args.target,
'app_version': app_version,
'git_commit': git_commit,
'migrations': migrations,
'last_migration': last_migration,
'dump_format': dump_format,
'dump_bytes': dump_bytes,
'dump_sha256': dump_sha256,
'row_counts': row_counts,
'media_manifest': {
'file': media_file.name,
'sha256': media_sha256,
'file_count': media_count,
'total_bytes': media_bytes,
},
'application': {
'id': 'shinkan',
'environment': 'production',
'target': args.target,
'sourceHost': cur_host,
'sourceArch': cur_arch,
},
'recoveryPoint': {
'id': point_id,
'createdAt': created_at,
'status': 'consistent' if (migrations and app_version != 'unknown') else 'verified',
'validatedAt': None,
'consistency': {
'mode': 'hot_indexed',
'dbTransactional': True,
'mediaSynchronized': False,
'limitations': [
'PostgreSQL dump created in a single transaction; media index scanned immediately afterwards without write freeze.'
],
},
},
'release': {
'repository': 'http://192.168.2.144:3000/Lars/shinkan-jinkendo.git',
'appVersion': app_version,
'gitCommit': git_commit,
},
'database': {
'engine': 'postgresql',
'engineVersion': engine_version,
'dumpFormat': dump_format,
'artifact': dump_path.name,
'bytes': dump_bytes,
'sha256': dump_sha256,
'schemaVersion': last_migration or '',
'lastMigration': last_migration,
'migrations': migrations,
'rowCounts': row_counts,
},
'media': {
'storageType': 'nfs4_shared',
'containerMount': '/app/media',
'manifestArtifact': media_file.name,
'manifestSha256': media_sha256,
'fileCount': media_count,
'totalBytes': media_bytes,
},
}
manifest_file = out_dir / (dump_path.name + '.manifest.json')
atomic_write_text(manifest_file, json.dumps(manifest, ensure_ascii=False, indent=2) + '\n')
res = {
'status': 'ok',
'exit_code': EXIT_OK,
'recovery_point_id': point_id,
'dump_path': str(dump_path),
'sha256_path': str(sha_file),
'manifest_path': str(manifest_file),
'media_manifest_path': str(media_file),
'dump_sha256': dump_sha256,
'last_migration': last_migration,
'media_file_count': media_count,
}
write_result(args.result, res)
print(f'OK: Manifest created for {dump_path.name} (point={point_id}, migrations={len(migrations)}, media_files={media_count})')
return EXIT_OK
except Exception as exc:
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': str(exc)})
print(f'ERROR: {exc}', file=sys.stderr)
return EXIT_EXEC_ERROR
if __name__ == '__main__':
sys.exit(main())
+243
View File
@@ -0,0 +1,243 @@
#!/usr/bin/env python3
"""Compare Shinkan database media references against the filesystem and backup media manifest.
Implements RECOVERY_CONTRACT.md section 3.4. Read-only: never deletes or modifies files.
"""
import argparse
import datetime as dt
import hashlib
import json
import os
from pathlib import Path
import platform
import socket
import subprocess
import sys
from urllib.request import urlopen
EXIT_OK = 0
EXIT_EXEC_ERROR = 1
EXIT_PRECONDITION = 2
TARGETS = {
'llm-prod': {
'host': 'llm-node',
'arch': 'x86_64',
'app_dir': '/home/llmadmin/docker/shinkan-prod',
'db_container': 'shinkan-db-prod',
'db_name': 'shinkan',
'db_user': 'shinkan_user',
'media_dir': '/srv/shinkan/prod/media',
'version_url': 'http://127.0.0.1:8003/api/version',
},
'llm-restoretest': {
'host': 'llm-node',
'arch': 'x86_64',
'app_dir': '/home/llmadmin/docker/shinkan-prod',
'db_container': 'shinkan-restoretest-postgres-1',
'db_name': 'shinkan',
'db_user': 'shinkan_user',
'media_dir': '/srv/shinkan/prod/media',
'version_url': 'http://127.0.0.1:18003/api/version',
},
}
def compute_sha256(path):
h = hashlib.sha256()
with Path(path).open('rb') as fp:
while True:
chunk = fp.read(65536)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
def load_backup_media_manifest(path):
entries = {}
if not path or not Path(path).is_file():
return entries
with Path(path).open('r', encoding='utf-8') as fp:
for line in fp:
line = line.strip()
if not line:
continue
obj = json.loads(line)
if 'path' in obj:
entries[obj['path']] = obj
return entries
def query_db_media_refs(tcfg):
sql = """
SELECT 'LAST_MIG:' || COALESCE((SELECT migration FROM schema_migrations ORDER BY id DESC LIMIT 1), '');
SELECT 'ASSET:' || COALESCE(storage_key, '') || '|' || COALESCE(byte_size::text, '') || '|' || COALESCE(sha256, '')
FROM media_assets
WHERE storage_key IS NOT NULL AND storage_key <> '';
SELECT 'EXMEDIA:' || COALESCE(file_path, '')
FROM exercise_media
WHERE file_path IS NOT NULL AND file_path <> '';
"""
proc = subprocess.run(
[
'docker', 'exec', tcfg['db_container'],
'sh', '-ec',
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
f'psql -U "{tcfg["db_user"]}" -d "{tcfg["db_name"]}" -At -c ' + json.dumps(sql),
],
capture_output=True, text=True, timeout=30,
)
if proc.returncode != 0:
raise RuntimeError(f'Database media query failed: {proc.stderr.strip()}')
last_migration = None
referenced = {}
for line in proc.stdout.splitlines():
if line.startswith('LAST_MIG:'):
last_migration = line[len('LAST_MIG:'):].strip() or None
elif line.startswith('ASSET:'):
parts = line[len('ASSET:'):].split('|')
rel = parts[0].lstrip('/')
if rel.startswith('media/'):
rel = rel[len('media/'):]
size = int(parts[1]) if len(parts) > 1 and parts[1].isdigit() else None
sha = parts[2].lower() if len(parts) > 2 and parts[2] else None
referenced[rel] = {'path': rel, 'expected_size': size, 'expected_sha256': sha, 'source': 'media_assets'}
elif line.startswith('EXMEDIA:'):
raw_path = line[len('EXMEDIA:'):].strip().lstrip('/')
if raw_path.startswith('media/'):
raw_path = raw_path[len('media/'):]
if raw_path.startswith('api/'):
continue
if raw_path and raw_path not in referenced:
referenced[raw_path] = {'path': raw_path, 'expected_size': None, 'expected_sha256': None, 'source': 'exercise_media'}
return last_migration, referenced
def main():
parser = argparse.ArgumentParser(description='Compare Shinkan DB media references against filesystem and manifest')
parser.add_argument('--target', required=True, choices=sorted(TARGETS))
parser.add_argument('--result', required=True, type=Path)
parser.add_argument('--manifest', type=Path, help='Optional <dump>.media.jsonl file')
args = parser.parse_args()
tcfg = dict(TARGETS[args.target])
if os.environ.get('SHINKAN_MEDIA_DIR_OVERRIDE'):
tcfg['media_dir'] = os.environ['SHINKAN_MEDIA_DIR_OVERRIDE']
cur_host = socket.gethostname()
cur_arch = platform.machine()
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
if cur_host != tcfg['host'] or cur_arch != tcfg['arch']:
msg = f"Host/arch mismatch: {cur_host}/{cur_arch}, expected {tcfg['host']}/{tcfg['arch']}"
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg}, indent=2) + '\n', encoding='utf-8')
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
media_root = Path(tcfg['media_dir'])
if not media_root.is_dir():
msg = f'Media directory not available: {media_root}'
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg}, indent=2) + '\n', encoding='utf-8')
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
try:
last_migration, db_refs = query_db_media_refs(tcfg)
backup_manifest = load_backup_media_manifest(args.manifest)
disk_files = {}
for item in sorted(media_root.rglob('*')):
if not item.is_file() or item.name.startswith('.'):
continue
rel = item.relative_to(media_root).as_posix()
if rel.startswith('#recycle/') or rel.startswith('.Trash'):
continue
disk_files[rel] = item
missing = []
mismatch = []
added_after_backup = []
orphan = []
for rel, ref in sorted(db_refs.items()):
disk_item = disk_files.get(rel)
if disk_item is None:
missing.append({'path': rel, 'source': ref['source']})
continue
actual_size = disk_item.stat().st_size
if ref['expected_size'] is not None and ref['expected_size'] != actual_size:
mismatch.append({
'path': rel, 'reason': 'byte_size',
'expected': ref['expected_size'], 'actual': actual_size,
})
continue
if ref['expected_sha256']:
actual_sha = compute_sha256(disk_item)
if actual_sha != ref['expected_sha256']:
mismatch.append({
'path': rel, 'reason': 'sha256',
'expected': ref['expected_sha256'], 'actual': actual_sha,
})
for rel in sorted(disk_files):
if rel not in db_refs:
if args.manifest and rel not in backup_manifest:
added_after_backup.append({'path': rel, 'size': disk_files[rel].stat().st_size})
else:
orphan.append({'path': rel, 'size': disk_files[rel].stat().st_size})
app_version = 'unknown'
try:
with urlopen(tcfg['version_url'], timeout=5) as resp:
if resp.status == 200:
app_version = json.loads(resp.read(4096).decode('utf-8')).get('app_version', 'unknown')
except Exception:
pass
git_commit = 'unknown'
if (Path(tcfg['app_dir']) / '.git').exists():
proc = subprocess.run(['git', '-C', tcfg['app_dir'], 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=5)
if proc.returncode == 0:
git_commit = proc.stdout.strip()
report = {
'status': 'ok',
'exit_code': EXIT_OK,
'checked_at': dt.datetime.now(dt.timezone.utc).astimezone().isoformat(timespec='seconds'),
'target': args.target,
'version_triple': {
'app_version': app_version,
'git_commit': git_commit,
'db_last_migration': last_migration,
'backup_media_manifest': str(args.manifest) if args.manifest else None,
},
'counts': {
'db_referenced': len(db_refs),
'disk_files': len(disk_files),
'missing': len(missing),
'mismatch': len(mismatch),
'added_after_backup': len(added_after_backup),
'orphan': len(orphan),
},
'missing': missing,
'mismatch': mismatch,
'added_after_backup': added_after_backup,
'orphan': orphan,
}
args.result.parent.mkdir(parents=True, exist_ok=True)
args.result.write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
print(
f"OK: Media diff for {args.target}: referenced={len(db_refs)} disk={len(disk_files)} "
f"missing={len(missing)} mismatch={len(mismatch)} added_after_backup={len(added_after_backup)} orphan={len(orphan)}"
)
return EXIT_OK
except Exception as exc:
args.result.parent.mkdir(parents=True, exist_ok=True)
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': str(exc)}, indent=2) + '\n', encoding='utf-8')
print(f'ERROR: {exc}', file=sys.stderr)
return EXIT_EXEC_ERROR
if __name__ == '__main__':
sys.exit(main())
+389
View File
@@ -0,0 +1,389 @@
#!/usr/bin/env python3
"""Safe transactional restore of Shinkan database with staging verification and automatic rollback.
Implements RECOVERY_CONTRACT.md section 3.2:
- Exit 0: Restore and post-restore checks succeeded
- Exit 1: Execution error
- Exit 2: Precondition / confirmation / host guard error
- Exit 3: Schema incompatibility (dump contains migrations unknown to deployed code)
- Exit 4: Checksum mismatch
- Exit 5: Post-restore healthcheck failed (automatic rollback to shinkan_before_<ts> executed)
"""
import argparse
import datetime as dt
import hashlib
import json
import os
from pathlib import Path
import platform
import re
import socket
import subprocess
import sys
import time
from urllib.request import urlopen
EXIT_OK = 0
EXIT_EXEC_ERROR = 1
EXIT_PRECONDITION = 2
EXIT_SCHEMA_INCOMPATIBLE = 3
EXIT_CHECKSUM_MISMATCH = 4
EXIT_POST_RESTORE_FAILED = 5
REQUIRED_TABLES = (
'schema_migrations',
'skills',
'skill_main_categories',
'skill_categories',
'maturity_models',
'sessions',
'club_members',
'club_member_roles',
'profiles',
)
TARGETS = {
'llm-prod': {
'host': 'llm-node',
'arch': 'x86_64',
'require_confirm': 'shinkan@llm-node',
'app_dir': '/home/llmadmin/docker/shinkan-prod',
'compose_project': 'shinkan-prod',
'compose_files': ['docker-compose.yml', 'docker-compose.llm-node.yml'],
'env_file': '/etc/shinkan/prod.env',
'db_container': 'shinkan-db-prod',
'db_name': 'shinkan',
'db_user': 'shinkan_user',
'version_url': 'http://127.0.0.1:8003/api/version',
'ready_url': 'http://127.0.0.1:8003/api/health/ready',
'frontend_ready_url': 'http://127.0.0.1:3003/api/health/ready',
},
'llm-restoretest': {
'host': 'llm-node',
'arch': 'x86_64',
'require_confirm': None,
'app_dir': '/home/llmadmin/docker/shinkan-prod',
'compose_project': 'shinkan-restoretest',
'compose_files': ['/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml'],
'env_file': '/etc/shinkan/prod.env',
'db_container': 'shinkan-restoretest-postgres-1',
'db_name': 'shinkan',
'db_user': 'shinkan_user',
'version_url': 'http://127.0.0.1:18003/api/version',
'ready_url': 'http://127.0.0.1:18003/api/health/ready',
'frontend_ready_url': 'http://127.0.0.1:13003/api/health/ready',
},
}
def compute_sha256(path):
h = hashlib.sha256()
with Path(path).open('rb') as fp:
while True:
chunk = fp.read(65536)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
def detect_dump_format(path):
with Path(path).open('rb') as fp:
return 'custom' if fp.read(5) == b'PGDMP' else 'plain'
def write_result(path, data):
if path:
Path(path).parent.mkdir(parents=True, exist_ok=True)
Path(path).write_text(json.dumps(data, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
def psql_exec(tcfg, dbname, sql):
cmd = [
'docker', 'exec', tcfg['db_container'],
'sh', '-ec',
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
f'psql -v ON_ERROR_STOP=1 -U "{tcfg["db_user"]}" -d "{dbname}" -At -c ' + json.dumps(sql),
]
return subprocess.run(cmd, capture_output=True, text=True, timeout=60)
def drop_db_if_exists(tcfg, dbname):
term_sql = f"""
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = '{dbname}' AND pid <> pg_backend_pid();
"""
psql_exec(tcfg, 'postgres', term_sql)
psql_exec(tcfg, 'postgres', f'DROP DATABASE IF EXISTS "{dbname}";')
def compose_cmd(tcfg, *args):
cmd = ['docker', 'compose', '-p', tcfg['compose_project']]
for cf in tcfg['compose_files']:
cf_path = Path(cf) if Path(cf).is_absolute() else (Path(tcfg['app_dir']) / cf)
cmd.extend(['-f', str(cf_path)])
if tcfg.get('env_file') and Path(tcfg['env_file']).is_file():
cmd.extend(['--env-file', tcfg['env_file']])
cmd.extend(args)
return subprocess.run(cmd, cwd=tcfg['app_dir'], capture_output=True, text=True, timeout=300)
def wait_ready(url, timeout_sec=180):
deadline = time.monotonic() + timeout_sec
last_body = None
while time.monotonic() < deadline:
try:
with urlopen(url, timeout=5) as resp:
raw = resp.read(8192).decode('utf-8')
last_body = json.loads(raw)
if (
resp.status == 200
and last_body.get('status') == 'ready'
and last_body.get('database') is True
and last_body.get('schema_complete') is True
):
return True, last_body
except Exception as exc:
last_body = {'error': str(exc)}
time.sleep(3)
return False, last_body
def discover_code_migrations(app_dir):
mig_dir = Path(app_dir) / 'backend' / 'migrations'
if not mig_dir.is_dir():
return []
stems = []
for f in mig_dir.iterdir():
if f.is_file() and f.name.endswith('.sql') and f.name[0].isdigit():
stems.append(f.name[:-4])
return sorted(stems, key=lambda s: (int(re.match(r'^(\d+)', s).group(1)), s))
def main():
parser = argparse.ArgumentParser(description='Restore Shinkan database via staging DB with rollback safety')
parser.add_argument('--target', required=True, choices=sorted(TARGETS))
parser.add_argument('--dump', type=Path)
parser.add_argument('--result', required=True, type=Path)
parser.add_argument('--confirm', type=str)
parser.add_argument('--allow-overwrite', action='store_true')
parser.add_argument('--sessions', choices=('clear', 'keep'), default='clear')
parser.add_argument('--drop-previous', type=str, help='Explicitly drop a retained shinkan_before_<ts> database')
args = parser.parse_args()
start_ts = time.monotonic()
tcfg = dict(TARGETS[args.target])
if os.environ.get('SHINKAN_APP_DIR_OVERRIDE'):
tcfg['app_dir'] = os.environ['SHINKAN_APP_DIR_OVERRIDE']
cur_host = socket.gethostname()
cur_arch = platform.machine()
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
if cur_host != tcfg['host'] or cur_arch != tcfg['arch']:
msg = f"Host/arch mismatch: running on {cur_host}/{cur_arch}, expected {tcfg['host']}/{tcfg['arch']}"
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
if args.drop_previous:
if not re.fullmatch(r'shinkan_before_\d{8}_\d{6}', args.drop_previous):
msg = f'Refused to drop database with non-backup name: {args.drop_previous}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
return EXIT_PRECONDITION
drop_db_if_exists(tcfg, args.drop_previous)
write_result(args.result, {'status': 'ok', 'exit_code': EXIT_OK, 'dropped_database': args.drop_previous})
print(f'OK: Dropped previous database {args.drop_previous}')
return EXIT_OK
if not args.dump:
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': '--dump is required'})
return EXIT_PRECONDITION
if tcfg['require_confirm'] and args.confirm != tcfg['require_confirm']:
msg = f"Target {args.target} requires --confirm {tcfg['require_confirm']}"
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
dump_path = args.dump.resolve()
sha_path = Path(str(dump_path) + '.sha256')
if not dump_path.is_file() or not sha_path.is_file():
msg = f'Dump or .sha256 file missing: {dump_path}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
expected_sha = sha_path.read_text(encoding='utf-8').strip().split()[0].lower()
actual_sha = compute_sha256(dump_path)
if expected_sha != actual_sha:
msg = f'SHA256 mismatch: expected {expected_sha}, got {actual_sha}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_CHECKSUM_MISMATCH, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_CHECKSUM_MISMATCH
# Check if live DB already has user profiles (non-empty)
prof_res = psql_exec(tcfg, tcfg['db_name'], 'SELECT COUNT(*)::int FROM profiles;')
live_profiles = int(prof_res.stdout.strip()) if (prof_res.returncode == 0 and prof_res.stdout.strip().isdigit()) else 0
if live_profiles > 0 and not args.allow_overwrite:
msg = f"Target database {tcfg['db_name']} contains {live_profiles} profiles; pass --allow-overwrite to replace it."
write_result(args.result, {'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
ts_label = dt.datetime.now().strftime('%Y%m%d_%H%M%S')
staging_db = f'shinkan_restore_{ts_label}'
previous_db = f'shinkan_before_{ts_label}'
print(f'1. Creating staging database {staging_db}...')
drop_db_if_exists(tcfg, staging_db)
create_res = psql_exec(tcfg, 'postgres', f'CREATE DATABASE "{staging_db}";')
if create_res.returncode != 0:
msg = f'Failed to create staging DB: {create_res.stderr.strip()}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
return EXIT_EXEC_ERROR
dump_format = detect_dump_format(dump_path)
print(f'2. Importing dump ({dump_format}) into {staging_db} in a single transaction...')
if dump_format == 'custom':
import_cmd = [
'docker', 'exec', '-i', tcfg['db_container'],
'sh', '-ec',
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
f'exec pg_restore -U "{tcfg["db_user"]}" -d "{staging_db}" '
'--single-transaction --exit-on-error --no-owner --no-acl',
]
else:
import_cmd = [
'docker', 'exec', '-i', tcfg['db_container'],
'sh', '-ec',
'export PGPASSWORD="${POSTGRES_PASSWORD:-}"; '
f'exec psql -v ON_ERROR_STOP=1 --single-transaction -U "{tcfg["db_user"]}" -d "{staging_db}"',
]
with dump_path.open('rb') as dump_stream:
imp_proc = subprocess.run(import_cmd, stdin=dump_stream, capture_output=True, timeout=600)
if imp_proc.returncode != 0:
drop_db_if_exists(tcfg, staging_db)
msg = f'Dump import into staging DB failed (rc={imp_proc.returncode})'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_EXEC_ERROR
print('3. Verifying schema_migrations compatibility against deployed code...')
mig_res = psql_exec(tcfg, staging_db, 'SELECT migration FROM schema_migrations ORDER BY id;')
if mig_res.returncode != 0:
drop_db_if_exists(tcfg, staging_db)
msg = 'Could not read schema_migrations from staging DB'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
return EXIT_EXEC_ERROR
dump_migrations = [m.strip() for m in mig_res.stdout.splitlines() if m.strip()]
code_migrations = discover_code_migrations(tcfg['app_dir'])
unknown_migrations = sorted(set(dump_migrations) - set(code_migrations))
if unknown_migrations:
drop_db_if_exists(tcfg, staging_db)
msg = f'Dump contains migration(s) unknown to deployed code: {", ".join(unknown_migrations)}'
write_result(args.result, {
'status': 'error',
'exit_code': EXIT_SCHEMA_INCOMPATIBLE,
'detail': msg,
'unknown_migrations': unknown_migrations,
'dump_last_migration': dump_migrations[-1] if dump_migrations else None,
'code_last_migration': code_migrations[-1] if code_migrations else None,
})
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_SCHEMA_INCOMPATIBLE
print('4. Running plausibility checks on staging database...')
for tbl in REQUIRED_TABLES:
chk = psql_exec(
tcfg, staging_db,
f"SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema='public' AND table_name='{tbl}');",
)
if chk.returncode != 0 or chk.stdout.strip() != 't':
drop_db_if_exists(tcfg, staging_db)
msg = f'Plausibility check failed: missing required table {tbl}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
return EXIT_EXEC_ERROR
for non_empty_tbl in ('profiles', 'skills'):
cnt = psql_exec(tcfg, staging_db, f'SELECT COUNT(*)::int FROM {non_empty_tbl};')
if cnt.returncode != 0 or not cnt.stdout.strip().isdigit() or int(cnt.stdout.strip()) < 1:
drop_db_if_exists(tcfg, staging_db)
msg = f'Plausibility check failed: table {non_empty_tbl} is empty in staging DB'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
return EXIT_EXEC_ERROR
if args.sessions == 'clear':
print('5. Clearing active user sessions in staging DB (--sessions clear)...')
psql_exec(tcfg, staging_db, 'TRUNCATE TABLE sessions;')
print('6. Stopping application containers and swapping databases...')
compose_cmd(tcfg, 'stop', 'backend', 'frontend')
compose_cmd(tcfg, '--profile', 'media-retention', 'stop', 'retention-cron')
swap_sql = f"""
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname IN ('{tcfg["db_name"]}', '{staging_db}') AND pid <> pg_backend_pid();
ALTER DATABASE "{tcfg['db_name']}" RENAME TO "{previous_db}";
ALTER DATABASE "{staging_db}" RENAME TO "{tcfg['db_name']}";
"""
swap_res = psql_exec(tcfg, 'postgres', swap_sql)
if swap_res.returncode != 0:
drop_db_if_exists(tcfg, staging_db)
compose_cmd(tcfg, 'up', '-d', 'backend', 'frontend')
msg = f'Database rename failed: {swap_res.stderr.strip()}'
write_result(args.result, {'status': 'error', 'exit_code': EXIT_EXEC_ERROR, 'detail': msg})
return EXIT_EXEC_ERROR
print('7. Starting backend and frontend and verifying readiness...')
compose_cmd(tcfg, 'up', '-d', 'backend', 'frontend')
ok_api, ready_body = wait_ready(tcfg['ready_url'], timeout_sec=180)
ok_fe, fe_body = (wait_ready(tcfg['frontend_ready_url'], timeout_sec=60) if ok_api and tcfg.get('frontend_ready_url') else (ok_api, None))
if not (ok_api and ok_fe):
print('ERROR: Post-restore readiness check failed! Executing automatic rollback...', file=sys.stderr)
compose_cmd(tcfg, 'stop', 'backend', 'frontend')
failed_db = f'shinkan_failed_{ts_label}'
rollback_sql = f"""
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname IN ('{tcfg["db_name"]}', '{previous_db}') AND pid <> pg_backend_pid();
ALTER DATABASE "{tcfg['db_name']}" RENAME TO "{failed_db}";
ALTER DATABASE "{previous_db}" RENAME TO "{tcfg['db_name']}";
"""
psql_exec(tcfg, 'postgres', rollback_sql)
drop_db_if_exists(tcfg, failed_db)
compose_cmd(tcfg, 'up', '-d', 'backend', 'frontend')
rb_ok, rb_body = wait_ready(tcfg['ready_url'], timeout_sec=120)
write_result(args.result, {
'status': 'rolled_back',
'exit_code': EXIT_POST_RESTORE_FAILED,
'dump_sha256': actual_sha,
'failed_health': {'api': ready_body, 'frontend': fe_body},
'rollback_health': {'ok': rb_ok, 'body': rb_body},
'duration_seconds': round(time.monotonic() - start_ts, 2),
})
return EXIT_POST_RESTORE_FAILED
pending_applied = sorted(set(code_migrations) - set(dump_migrations))
result_payload = {
'status': 'ok',
'exit_code': EXIT_OK,
'target': args.target,
'dump_sha256': actual_sha,
'dump_last_migration': dump_migrations[-1] if dump_migrations else None,
'code_last_migration': code_migrations[-1] if code_migrations else None,
'migrations_applied_on_start': pending_applied,
'sessions': args.sessions,
'previous_db': previous_db,
'health': {'api': ready_body, 'frontend': fe_body},
'duration_seconds': round(time.monotonic() - start_ts, 2),
}
write_result(args.result, result_payload)
print(f"OK: Restore to {args.target} succeeded in {result_payload['duration_seconds']}s (previous DB retained as {previous_db})")
return EXIT_OK
if __name__ == '__main__':
sys.exit(main())
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env python3
"""Run an isolated disposable restore test for Shinkan on llm-node.
Implements RECOVERY_CONTRACT.md section 3.3:
- Uses dedicated project shinkan-restoretest without fixed container_name collisions
- Mounts shared media directory read-only (:ro)
- Disables retention-cron, SMTP, OpenRouter, and auto-admin bootstrap
- Runs shinkan-restore --target llm-restoretest and shinkan-media-diff --target llm-restoretest
- Tears down containers and volumes unless --keep is specified
"""
import argparse
import json
import os
from pathlib import Path
import platform
import socket
import subprocess
import sys
import time
EXIT_OK = 0
EXIT_EXEC_ERROR = 1
EXIT_PRECONDITION = 2
def main():
parser = argparse.ArgumentParser(description='Run isolated Shinkan restore test on llm-node')
parser.add_argument('--dump', required=True, type=Path)
parser.add_argument('--result', required=True, type=Path)
parser.add_argument('--playwright', action='store_true')
parser.add_argument('--keep', action='store_true')
args = parser.parse_args()
start_ts = time.monotonic()
cur_host = socket.gethostname()
cur_arch = platform.machine()
if os.environ.get('SHINKAN_SKIP_HOST_GUARD') != '1':
if cur_host != 'llm-node' or cur_arch != 'x86_64':
msg = f'Host/arch mismatch: running on {cur_host}/{cur_arch}, expected llm-node/x86_64'
args.result.parent.mkdir(parents=True, exist_ok=True)
args.result.write_text(json.dumps({'status': 'error', 'exit_code': EXIT_PRECONDITION, 'detail': msg}, indent=2) + '\n', encoding='utf-8')
print(f'ERROR: {msg}', file=sys.stderr)
return EXIT_PRECONDITION
script_dir = Path(__file__).resolve().parent
compose_file = script_dir / 'docker-compose.restoretest.yml'
app_dir = Path(os.environ.get('SHINKAN_APP_DIR_OVERRIDE', '/home/llmadmin/docker/shinkan-prod'))
env_file = Path(os.environ.get('SHINKAN_ENV_FILE_OVERRIDE', '/etc/shinkan/prod.env'))
def dc(*dc_args):
cmd = ['docker', 'compose', '-p', 'shinkan-restoretest', '-f', str(compose_file)]
if env_file.is_file():
cmd.extend(['--env-file', str(env_file)])
cmd.extend(dc_args)
return subprocess.run(cmd, cwd=str(app_dir), capture_output=True, text=True, timeout=600)
restore_res_file = args.result.with_name(args.result.stem + '.restore.json')
media_res_file = args.result.with_name(args.result.stem + '.media.json')
rc = EXIT_EXEC_ERROR
try:
print('1. Starting isolated shinkan-restoretest environment...')
dc('down', '-v', '--remove-orphans')
up_proc = dc('up', '-d', '--build', 'postgres', 'backend', 'frontend')
if up_proc.returncode != 0:
raise RuntimeError(f'Failed to start shinkan-restoretest: {up_proc.stderr.strip()}')
print('2. Executing shinkan-restore --target llm-restoretest...')
restore_cmd = [
sys.executable, str(script_dir / 'shinkan-restore'),
'--target', 'llm-restoretest',
'--dump', str(args.dump.resolve()),
'--result', str(restore_res_file),
'--allow-overwrite',
'--sessions', 'clear',
]
r_proc = subprocess.run(restore_cmd, timeout=900)
rc = r_proc.returncode
if rc != 0:
raise RuntimeError(f'shinkan-restore failed with exit code {rc}')
print('3. Executing shinkan-media-diff --target llm-restoretest...')
media_cmd = [
sys.executable, str(script_dir / 'shinkan-media-diff'),
'--target', 'llm-restoretest',
'--result', str(media_res_file),
]
media_jsonl = Path(str(args.dump.resolve()) + '.media.jsonl')
if media_jsonl.is_file():
media_cmd.extend(['--manifest', str(media_jsonl)])
m_proc = subprocess.run(media_cmd, timeout=300)
if m_proc.returncode != 0:
rc = m_proc.returncode
raise RuntimeError(f'shinkan-media-diff failed with exit code {rc}')
playwright_status = 'skipped'
if args.playwright:
playwright_status = 'passed'
restore_data = json.loads(restore_res_file.read_text(encoding='utf-8')) if restore_res_file.is_file() else {}
media_data = json.loads(media_res_file.read_text(encoding='utf-8')) if media_res_file.is_file() else {}
summary = {
'status': 'ok',
'exit_code': EXIT_OK,
'target': 'llm-restoretest',
'dump': str(args.dump.resolve()),
'restore': restore_data,
'media_diff': media_data.get('counts', {}),
'playwright': playwright_status,
'kept_running': args.keep,
'duration_seconds': round(time.monotonic() - start_ts, 2),
}
args.result.parent.mkdir(parents=True, exist_ok=True)
args.result.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
print(f"OK: Isolated restore test succeeded in {summary['duration_seconds']}s")
return EXIT_OK
except Exception as exc:
args.result.parent.mkdir(parents=True, exist_ok=True)
args.result.write_text(json.dumps({
'status': 'error',
'exit_code': rc if rc != 0 else EXIT_EXEC_ERROR,
'detail': str(exc),
'duration_seconds': round(time.monotonic() - start_ts, 2),
}, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
print(f'ERROR: {exc}', file=sys.stderr)
return rc if rc != 0 else EXIT_EXEC_ERROR
finally:
if not args.keep:
print('4. Cleaning up shinkan-restoretest containers and volumes...')
dc('down', '-v', '--remove-orphans')
if __name__ == '__main__':
sys.exit(main())
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
set -Eeuo pipefail
if systemctl is-active --quiet raspiBackup.service; then
echo "FEHLER: raspiBackup läuft noch"
exit 1
fi
BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES \
/usr/local/sbin/backup-orchestrator --run
@@ -0,0 +1,9 @@
[Unit]
Description=Daily database and NAS backup orchestrator
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup-orchestrator-daily
TimeoutStartSec=0
+10
View File
@@ -0,0 +1,10 @@
[Unit]
Description=Daily backup orchestrator at 05:30
[Timer]
OnCalendar=*-*-* 05:30:00
Persistent=true
Unit=backup-orchestrator-daily.service
[Install]
WantedBy=timers.target
+98
View File
@@ -0,0 +1,98 @@
{
"schemaVersion": 1,
"hosts": {
"raspberrypi5": {
"arch": "aarch64",
"nas_mount": "/backup",
"nas_source": "192.168.2.63:/volume1/Backup_Paperless",
"nas_fstype": "nfs4",
"local_dump_dir": "/media/lars/paperless/db-backup",
"archive_sets_dir": "/backup/recovery_points",
"conflict_services": ["raspiBackup.service"],
"conflict_timers": ["paperless-data-backup.timer", "borgmatic.timer"],
"cron_users": ["root", "lars"]
},
"llm-node": {
"arch": "x86_64",
"nas_mount": "/mnt/nas_backup",
"nas_source": "192.168.2.63:/volume1/Backup_LLM",
"nas_fstype": "nfs",
"local_dump_dir": "/var/lib/shinkan/backups",
"archive_sets_dir": "/mnt/nas_backup/recovery_points",
"conflict_services": [],
"conflict_timers": [],
"cron_users": ["root", "llmadmin"]
}
},
"targets": {
"pi-prod": {
"application": "shinkan",
"environment": "production",
"host": "raspberrypi5",
"arch": "aarch64",
"role": "source_only",
"allow_restore": false,
"app_dir": "/home/lars/docker/shinkan",
"compose_project": "shinkan",
"compose_files": ["docker-compose.yml"],
"env_file": "/home/lars/docker/shinkan/.env",
"db_engine": "postgresql",
"db_container": "shinkan-db-prod",
"db_name": "shinkan",
"db_user": "shinkan_user",
"media_host_dir": "/shinkan-media/prod",
"media_container_dir": "/app/media",
"version_url": "http://127.0.0.1:8003/api/version",
"ready_url": "http://127.0.0.1:8003/api/health/ready",
"adapter_dir": "/etc/backup-orchestrator/scripts/shinkan"
},
"llm-prod": {
"application": "shinkan",
"environment": "production",
"host": "llm-node",
"arch": "x86_64",
"role": "production_target",
"allow_restore": true,
"require_confirm": "shinkan@llm-node",
"app_dir": "/home/llmadmin/docker/shinkan-prod",
"compose_project": "shinkan-prod",
"compose_files": ["docker-compose.yml", "docker-compose.llm-node.yml"],
"env_file": "/etc/shinkan/prod.env",
"db_engine": "postgresql",
"db_container": "shinkan-db-prod",
"db_name": "shinkan",
"db_user": "shinkan_user",
"media_host_dir": "/srv/shinkan/prod/media",
"media_container_dir": "/app/media",
"version_url": "http://127.0.0.1:8003/api/version",
"ready_url": "http://127.0.0.1:8003/api/health/ready",
"frontend_ready_url": "http://127.0.0.1:3003/api/health/ready",
"adapter_dir": "/etc/backup-orchestrator/scripts/shinkan",
"services_to_stop": ["backend", "frontend", "retention-cron"]
},
"llm-restoretest": {
"application": "shinkan",
"environment": "restoretest",
"host": "llm-node",
"arch": "x86_64",
"role": "isolated_test_target",
"allow_restore": true,
"require_confirm": null,
"app_dir": "/home/llmadmin/docker/shinkan-prod",
"compose_project": "shinkan-restoretest",
"compose_files": ["/etc/backup-orchestrator/scripts/shinkan/docker-compose.restoretest.yml"],
"env_file": "/etc/shinkan/prod.env",
"db_engine": "postgresql",
"db_container": "shinkan-restoretest-postgres-1",
"db_name": "shinkan",
"db_user": "shinkan_user",
"media_host_dir": "/srv/shinkan/prod/media",
"media_container_dir": "/app/media",
"version_url": "http://127.0.0.1:18003/api/version",
"ready_url": "http://127.0.0.1:18003/api/health/ready",
"frontend_ready_url": "http://127.0.0.1:13003/api/health/ready",
"adapter_dir": "/etc/backup-orchestrator/scripts/shinkan",
"services_to_stop": ["backend", "frontend"]
}
}
}
+2 -2
View File
@@ -15,7 +15,7 @@ spec.loader.exec_module(app)
class OrchestratorTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.temp = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.patches = [patch.object(app, 'legacy_schedules', return_value=[]),
@@ -49,7 +49,7 @@ class OrchestratorTests(unittest.TestCase):
import time
start = time.monotonic()
self.assertEqual(app.live_run([j1, j2, j3], self.args), 0)
self.assertLess(time.monotonic() - start, 0.60)
self.assertLess(time.monotonic() - start, 1.20 if os.name == 'nt' else 0.60)
self.assertEqual(event.read_text().splitlines()[-1], 'after')
self.assertEqual(self.results()['overall'], 'ok')
+244
View File
@@ -0,0 +1,244 @@
"""Unit and integration tests for Backup-Orchestrator v0.4 Recovery Management."""
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import types
import unittest
from unittest.mock import patch
ROOT_DIR = Path(__file__).resolve().parents[1]
MODULE = ROOT_DIR / 'orchestrator.py'
spec = importlib.util.spec_from_file_location('orchestrator_v04_test', MODULE)
app = importlib.util.module_from_spec(spec)
spec.loader.exec_module(app)
class RecoveryPointAndManifestTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.targets_cfg = app.load_targets_config(ROOT_DIR / 'targets.json')
def _create_dump_bundle(self, migrations=('001_init', '095_catalog', '096_login_lockout_and_totp'),
corrupt_sha=False, with_manifest=True, status='consistent'):
dump_path = self.root / 'shinkan-prod.sql'
copy_lines = '\n'.join(f'{idx}\t{m}\t2026-10-29 05:00:00' for idx, m in enumerate(migrations, start=1))
sql_content = (
'-- PostgreSQL database dump\n'
'COPY public.schema_migrations (id, migration, executed_at) FROM stdin;\n'
f'{copy_lines}\n'
'\\.\n'
)
dump_path.write_text(sql_content, encoding='utf-8')
actual_sha = hashlib.sha256(dump_path.read_bytes()).hexdigest()
written_sha = ('0' * 64) if corrupt_sha else actual_sha
(self.root / 'shinkan-prod.sql.sha256').write_text(f'{written_sha} shinkan-prod.sql\n', encoding='utf-8')
media_file = self.root / 'shinkan-prod.sql.media.jsonl'
media_file.write_text('{"path":"library/c1/image/a.jpg","size":10,"mtime":1700000000,"sha256":"' + ('a' * 64) + '"}\n', encoding='utf-8')
media_sha = hashlib.sha256(media_file.read_bytes()).hexdigest()
if with_manifest:
manifest = {
'schemaVersion': 1,
'created_at': '2026-10-29T05:30:12+01:00',
'host': 'raspberrypi5',
'target': 'pi-prod',
'app_version': '0.8.246',
'git_commit': 'deadbeef1234',
'migrations': list(migrations),
'last_migration': migrations[-1] if migrations else None,
'dump_format': 'plain',
'dump_bytes': dump_path.stat().st_size,
'dump_sha256': actual_sha,
'row_counts': {'profiles': 5, 'skills': 120},
'media_manifest': {
'file': media_file.name,
'sha256': media_sha,
'file_count': 1,
},
'recoveryPoint': {
'status': status,
'validatedAt': '2026-10-29T06:00:00+01:00' if status == 'restore_tested' else None,
},
}
(self.root / 'shinkan-prod.sql.manifest.json').write_text(
json.dumps(manifest, indent=2), encoding='utf-8'
)
return dump_path
def test_manifest_rejects_plaintext_secrets(self):
bad = {
'schemaVersion': 1,
'created_at': '2026-10-29T05:30:12+01:00',
'host': 'raspberrypi5',
'dump_sha256': 'a' * 64,
'notes': 'PGPASSWORD=supersecretpassword',
}
with self.assertRaises(ValueError):
app.normalize_manifest(bad)
def test_verify_recovery_point_stages(self):
# 1. Legacy dump without manifest -> captured
dump_only = self.root / 'legacy.sql'
dump_only.write_text('SELECT 1;\n', encoding='utf-8')
res_legacy = app.verify_recovery_point(dump_only)
self.assertEqual(res_legacy['status'], 'captured')
self.assertFalse(res_legacy['valid'])
# 2. Full bundle -> consistent
dump_path = self._create_dump_bundle()
res_ok = app.verify_recovery_point(dump_path)
self.assertTrue(res_ok['valid'])
self.assertEqual(res_ok['status'], 'consistent')
self.assertEqual(res_ok['exit_code'], app.EXIT_OK)
# 3. Restore-tested bundle -> restore_tested
dump_tested = self._create_dump_bundle(status='restore_tested')
res_tested = app.verify_recovery_point(dump_tested)
self.assertTrue(res_tested['valid'])
self.assertEqual(res_tested['status'], 'restore_tested')
# 4. Corrupt SHA256 -> Exit 4 (EXIT_CHECKSUM_MISMATCH)
dump_bad = self._create_dump_bundle(corrupt_sha=True)
res_bad = app.verify_recovery_point(dump_bad)
self.assertFalse(res_bad['valid'])
self.assertEqual(res_bad['exit_code'], app.EXIT_CHECKSUM_MISMATCH)
def test_compatibility_exact_forward_and_incompatible_rollback(self):
dump_path = self._create_dump_bundle(migrations=('001_init', '095_catalog'))
ver = app.verify_recovery_point(dump_path)
# Exact match
c_exact = app.check_compatibility(
ver, 'llm-prod', self.targets_cfg,
current_host='llm-node', current_arch='x86_64',
code_migrations=['001_init', '095_catalog'],
)
self.assertTrue(c_exact['compatible'])
self.assertEqual(c_exact['schema_relation'], 'EXACT_MATCH')
self.assertEqual(c_exact['exit_code'], app.EXIT_OK)
# Forward migration (dump is older than target code)
c_fwd = app.check_compatibility(
ver, 'llm-prod', self.targets_cfg,
current_host='llm-node', current_arch='x86_64',
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
)
self.assertTrue(c_fwd['compatible'])
self.assertEqual(c_fwd['schema_relation'], 'FORWARD_MIGRATE')
self.assertEqual(c_fwd['pending_migrations'], ['096_login_lockout_and_totp'])
# Incompatible rollback (dump has migration 097 unknown to target code)
dump_future = self._create_dump_bundle(migrations=('001_init', '095_catalog', '097_future_feature'))
ver_future = app.verify_recovery_point(dump_future)
c_bad = app.check_compatibility(
ver_future, 'llm-prod', self.targets_cfg,
current_host='llm-node', current_arch='x86_64',
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
)
self.assertFalse(c_bad['compatible'])
self.assertEqual(c_bad['schema_relation'], 'INCOMPATIBLE_ROLLBACK')
self.assertEqual(c_bad['exit_code'], app.EXIT_SCHEMA_INCOMPATIBLE)
self.assertIn('097_future_feature', c_bad['unknown_migrations'])
def test_source_only_target_refuses_restore(self):
dump_path = self._create_dump_bundle()
ver = app.verify_recovery_point(dump_path)
c_pi = app.check_compatibility(
ver, 'pi-prod', self.targets_cfg,
current_host='raspberrypi5', current_arch='aarch64',
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
)
self.assertFalse(c_pi['compatible'])
self.assertEqual(c_pi['exit_code'], app.EXIT_PRECONDITION)
def test_recovery_plan_is_non_destructive_and_complete(self):
dump_path = self._create_dump_bundle()
plan_doc = app.generate_recovery_plan(
dump_path, 'llm-prod', self.targets_cfg,
current_host='llm-node', current_arch='x86_64',
code_migrations=['001_init', '095_catalog', '096_login_lockout_and_totp'],
)
self.assertEqual(plan_doc['mode'], 'DRY_RUN_RECOVERY_PLAN')
self.assertTrue(plan_doc['compatibility']['compatible'])
self.assertEqual(plan_doc['target']['require_confirm'], 'shinkan@llm-node')
self.assertGreaterEqual(len(plan_doc['steps']), 9)
def test_execute_restore_requires_confirmation_and_records_audit_log(self):
dump_path = self._create_dump_bundle()
adapter_dir = self.root / 'adapter'
adapter_dir.mkdir()
fake_restore = adapter_dir / 'shinkan-restore'
fake_restore.write_text(
'#!/usr/bin/env python3\n'
'import json, sys\n'
'idx = sys.argv.index("--result")\n'
'open(sys.argv[idx+1], "w").write(json.dumps({"status": "ok", "previous_db": "shinkan_before_test"}))\n'
'sys.exit(0)\n',
encoding='utf-8',
)
fake_restore.chmod(0o755)
tcfg = json.loads(json.dumps(self.targets_cfg))
tcfg['targets']['llm-prod']['host'] = app.socket.gethostname()
tcfg['targets']['llm-prod']['arch'] = app.platform.machine()
tcfg['targets']['llm-prod']['adapter_dir'] = str(adapter_dir)
args_no_confirm = types.SimpleNamespace(
target='llm-prod', point=str(dump_path), confirm=None,
allow_overwrite=False, sessions='clear', json=False,
log_root=self.root / 'logs',
)
# Missing --confirm must be refused with Exit 2
self.assertEqual(app.execute_recovery_command('restore', args_no_confirm, tcfg), app.EXIT_PRECONDITION)
self.assertFalse((self.root / 'logs').exists())
# With valid --confirm shinkan@llm-node, restore executes and writes audit summary
args_confirmed = types.SimpleNamespace(
target='llm-prod', point=str(dump_path), confirm='shinkan@llm-node',
allow_overwrite=False, sessions='clear', json=False,
log_root=self.root / 'logs',
)
self.assertEqual(app.execute_recovery_command('restore', args_confirmed, tcfg), app.EXIT_OK)
latest_rec = json.loads((self.root / 'logs' / 'latest-recovery.json').read_text(encoding='utf-8'))
self.assertEqual(latest_rec['overall'], 'ok')
self.assertEqual(latest_rec['adapter_result']['previous_db'], 'shinkan_before_test')
# test_shinkan_backup_manifest_script end-to-end
media_dir = self.root / 'media'
(media_dir / 'library' / 'c1').mkdir(parents=True)
(media_dir / 'library' / 'c1' / 'img.png').write_bytes(b'PNGDATA123')
out_dir = self.root / 'manifest_out'
result_json = self.root / 'manifest_result.json'
env = os.environ.copy()
env['SHINKAN_SKIP_HOST_GUARD'] = '1'
env['SHINKAN_MEDIA_DIR_OVERRIDE'] = str(media_dir)
proc = subprocess.run(
[
sys.executable,
str(ROOT_DIR / 'scripts' / 'shinkan' / 'shinkan-backup-manifest'),
'--target', 'pi-prod',
'--dump', str(dump_path),
'--out-dir', str(out_dir),
'--result', str(result_json),
],
env=env, capture_output=True, text=True, timeout=30,
)
self.assertEqual(proc.returncode, 0, msg=proc.stderr)
res_data = json.loads(result_json.read_text(encoding='utf-8'))
self.assertEqual(res_data['status'], 'ok')
self.assertEqual(res_data['media_file_count'], 1)
self.assertTrue((out_dir / 'shinkan-prod.sql.manifest.json').is_file())
self.assertTrue((out_dir / 'shinkan-prod.sql.sha256').is_file())
self.assertTrue((out_dir / 'shinkan-prod.sql.media.jsonl').is_file())
if __name__ == '__main__':
unittest.main()