chore: import v0.3.0 baseline and add v0.4 recovery architecture plan
This commit is contained in:
commit
40188c0e10
12 files changed
+1506
No files matched your search
@@ -0,0 +1,6 @@
|
||||
* text=auto eol=lf
|
||||
*.sh text eol=lf
|
||||
*.py text eol=lf
|
||||
*.csv text eol=lf
|
||||
*.json text eol=lf
|
||||
*.md text eol=lf
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
.pytest_cache/
|
||||
*.tmp
|
||||
*.bak
|
||||
*.log
|
||||
notifications.json
|
||||
host.local.json
|
||||
.env
|
||||
@@ -0,0 +1,161 @@
|
||||
# Raspberry Pi 5 – Backup-Orchestrator v0.3
|
||||
|
||||
**Status: Test- und Integrationsversion.** Keine produktiven Backups, Timer, Crontabs oder Datenbanken werden durch Installation, `--check`, `--plan` oder `--status` verändert. Die vorhandenen Backup-Skripte sind **noch nicht vollständig standardisiert**, zusätzliche Datenbank-Jobs fehlen und eine vollständige Systemwiederherstellung ist noch nicht getestet. `--run` bleibt bewusst mehrfach gesperrt.
|
||||
|
||||
## 1. Enthaltene Dateien
|
||||
|
||||
- `orchestrator.py`: phasenbasierter Python-Controller (Python 3.11 / Debian 12, nur Standardbibliothek).
|
||||
- `jobs.csv`: bestehende acht Jobs. **Alle `enabled=false`** und verweisen derzeit auf die alten, vorhandenen Skriptpfade.
|
||||
- `notifications.json.example`: eine zentrale Konfigurationsdatei für ausgehende HTTP-POST-Webhooks und Uptime-Kuma-Push-Monitore; keine echten Secrets.
|
||||
- `tests/test_orchestrator.py`: Tests mit künstlichen Shellskripten und simulierten HTTP-Antworten; niemals produktive Datenbanken oder das NAS.
|
||||
|
||||
## 2. Sichere Installation auf raspberrypi5
|
||||
|
||||
ZIP auf den Raspberry Pi übertragen und dort entpacken. Im entpackten Verzeichnis:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s tests -v
|
||||
sudo cp -a /usr/local/sbin/backup-orchestrator /usr/local/sbin/backup-orchestrator.before-v03
|
||||
sudo install -m 755 orchestrator.py /usr/local/sbin/backup-orchestrator
|
||||
|
||||
# Die vorhandene /etc/backup-orchestrator/jobs.csv NICHT überschreiben.
|
||||
# Die Benachrichtigungskonfiguration nur bei erstmaliger Einrichtung kopieren.
|
||||
sudo install -d -m 750 /etc/backup-orchestrator
|
||||
if ! sudo test -e /etc/backup-orchestrator/notifications.json; then
|
||||
sudo install -m 600 notifications.json.example /etc/backup-orchestrator/notifications.json
|
||||
fi
|
||||
sudo backup-orchestrator --check
|
||||
```
|
||||
|
||||
Ohne bereits installierte ältere Version entfällt die Zeile `sudo cp -a ...before-v03`.
|
||||
|
||||
> **Keine `--run`-Befehle ausführen**, solange die alten Cronjobs und der raspiBackup-Timer aktiv sind und die alten Dump-Skripte noch nicht gehärtet wurden. Den Produktiv-Cutover führen wir später kontrolliert durch.
|
||||
|
||||
## 3. Konfiguration der Phasen
|
||||
|
||||
Die vorhandene Tabelle unter `/etc/backup-orchestrator/jobs.csv` hat **keine Uhrzeiten**:
|
||||
|
||||
```csv
|
||||
phase,id,script,critical,timeout_seconds,enabled
|
||||
10,paperless-db,/home/lars/db-backup.sh,true,600,false
|
||||
10,tandoor-db,/home/lars/db-Tandoor.sh,true,600,false
|
||||
20,nas-data,/home/lars/backupPaperlessData.sh,true,10800,false
|
||||
30,system-image,/usr/local/bin/raspiBackup.sh,true,14400,false
|
||||
```
|
||||
|
||||
Die Beispiele sind ein Ausschnitt; die mitgelieferte Datei enthält acht Jobs.
|
||||
|
||||
- **Innerhalb derselben Phase** laufen bis zu drei Jobs gleichzeitig (per `--max-parallel N` einstellbar).
|
||||
- **Zwischen Phasen** gilt eine Barriere: Die nächste Phase beginnt erst, wenn die aktuelle vollständig abgeschlossen ist.
|
||||
- Ein **kritischer Fehler** blockiert nachfolgende Phasen; laufende Jobs derselben Phase werden beendet bzw. bis zum Abschluss abgewartet.
|
||||
- Ein **unkritischer Fehler** wird protokolliert und führt zu einem fehlgeschlagenen Gesamtergebnis, blockiert aber die nächste Phase nicht.
|
||||
- Zeitüberschreitung beendet die Skript-Prozessgruppe; Service-Neustart und Datenbankkonsistenz müssen in den jeweiligen Jobs abgesichert sein.
|
||||
|
||||
Zentrale Zeitsteuerung per *einem* systemd-Timer ist eine spätere Cutover-Aufgabe, **nicht Bestandteil der Installation**.
|
||||
|
||||
## 4. Externe Schnittstellen
|
||||
|
||||
### A. Ausgehende Webhooks, HTTP POST
|
||||
|
||||
`/etc/backup-orchestrator/notifications.json` bearbeiten:
|
||||
|
||||
```json
|
||||
{
|
||||
"timeout_seconds": 3,
|
||||
"webhooks": ["http://192.168.2.100:5678/webhook/backup-events"],
|
||||
"kuma": {
|
||||
"overall_push_url": "http://192.168.2.63:3010/api/push/GESAMT_MONITOR_TOKEN",
|
||||
"job_push_urls": {
|
||||
"nas-data": "http://192.168.2.63:3010/api/push/NAS_MONITOR_TOKEN",
|
||||
"paperless-db": "http://192.168.2.63:3010/api/push/PAPERLESS_MONITOR_TOKEN"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Alle URLs sind Beispiele – vorhandene echte Push-Tokens nur auf dem Pi eintragen. Einträge mit `""` sind deaktiviert. Es gibt **keinen HTTP-Listener und keinen permanent laufenden Webserver**; Webhooks sind *ausgehend*.
|
||||
|
||||
Die echten URLs können auch komplett aus der Uptime-Kuma-Oberfläche kopiert werden (`...?status=up&msg=OK&ping=`). Diese Beispiel-Query wird beim Laden entfernt und beim Push durch den tatsächlichen Status ersetzt.
|
||||
|
||||
Diese Datei enthält Zugriffstokens: `sudo chmod 600 /etc/backup-orchestrator/notifications.json`. Sie gehört in eine vertrauliche Konfigurationssicherung, nicht in ein öffentliches Repository.
|
||||
|
||||
### B. Ereignisse und Beispiele
|
||||
|
||||
Alle Ereignisse werden als **JSON je Zeile** in `/var/log/backup-orchestrator/<run_id>/events.jsonl` geschrieben; jede Zeile wird auch an die konfigurierten POST-Webhooks gesendet.
|
||||
|
||||
- `RUN_STARTED`, `PHASE_STARTED`, `JOB_STARTED`
|
||||
- `JOB_SUCCESS`, `JOB_FAILED`, `JOB_TIMEOUT`, `JOB_ERROR`, `JOB_CANCELLED`, `JOB_BLOCKED`
|
||||
- `PHASE_COMPLETED`, `PHASE_BLOCKED`, `RUN_FINISHED`
|
||||
|
||||
Beispiel einer Webhook-Nachricht (ohne Geheimnisse):
|
||||
|
||||
```json
|
||||
{
|
||||
"event": "JOB_FAILED",
|
||||
"timestamp": "2026-10-29T01:02:00+01:00",
|
||||
"run_id": "20261029-010000-1234",
|
||||
"host": "raspberrypi5",
|
||||
"phase": 10,
|
||||
"job": "shinkan-prod-db",
|
||||
"critical": true,
|
||||
"state": "failed",
|
||||
"exit_code": 1,
|
||||
"duration_seconds": 12.3,
|
||||
"log": "/var/log/backup-orchestrator/20261029-010000-1234/shinkan-prod-db.log"
|
||||
}
|
||||
```
|
||||
|
||||
Webhooks versenden **Ereignismetadaten**, nicht den Inhalt der Backup-Skripte oder SQL-Dumps.
|
||||
|
||||
### C. Uptime Kuma
|
||||
|
||||
- `JOB_SUCCESS` → `status=up` für den konfigurierten Job-Monitor.
|
||||
- `JOB_FAILED`, `JOB_TIMEOUT`, `JOB_ERROR`, `JOB_CANCELLED` und `JOB_BLOCKED` → `status=down`.
|
||||
- `RUN_FINISHED` → `status=up` nur beim Gesamtergebnis `ok`, sonst `down`.
|
||||
- Ausbleibende Backups meldet Kuma über den konfigurierten Push-Heartbeat, z.B. bei einem täglichen Lauf ein zum Zeitplan passender Wert.
|
||||
- Schlägt die Zustellung fehl, schreibt der Controller eine Warnung in `notifications.log`. Der Backup-Exit-Code bleibt unverändert.
|
||||
|
||||
**Keine doppelten Push-Meldungen erzeugen:** Wenn `nas-data` künftig durch den Orchestrator gemeldet wird, die Push-Funktion im Skript `backupPaperlessData.sh` deaktivieren oder nur einen der beiden Sender verwenden.
|
||||
|
||||
### D. Lokaler Status, JSON-Abfrage
|
||||
|
||||
Während des Laufs wird der aktuelle Stand atomar aktualisiert:
|
||||
|
||||
```text
|
||||
/var/log/backup-orchestrator/latest.json
|
||||
/var/log/backup-orchestrator/<run_id>/status.json
|
||||
/var/log/backup-orchestrator/<run_id>/summary.json
|
||||
/var/log/backup-orchestrator/<run_id>/events.jsonl
|
||||
```
|
||||
|
||||
CLI für andere Skripte, Home Assistant oder eine spätere GUI:
|
||||
|
||||
```bash
|
||||
sudo backup-orchestrator --status
|
||||
sudo backup-orchestrator --status --json
|
||||
sudo cat /var/log/backup-orchestrator/latest.json
|
||||
```
|
||||
|
||||
Die Statusdateien enthalten keine Push-Tokens oder Zugangsdaten, aber interne Pfade und Jobnamen. Daher nur für vertrauenswürdige lokale Nutzer freigeben. Es gibt derzeit **keine eingehende REST-API**; dies kann später als separater schreibgeschützter Dienst ergänzt werden.
|
||||
|
||||
## 5. Sichere Bedienung
|
||||
|
||||
```bash
|
||||
sudo backup-orchestrator --check # nur Prüfungen, keine Backups
|
||||
sudo backup-orchestrator --plan # nur aktivierte Jobs anzeigen
|
||||
sudo backup-orchestrator --status --json # letzter Status bzw. no_runs
|
||||
```
|
||||
|
||||
`--run`, `--run --phase 10` und `--run --job paperless-db` sind für einen **späteren**, explizit kontrollierten Produktiv-Cutover vorgesehen. Der Controller verweigert Ausführung, solange keine Jobs aktiviert sind, der NAS-Mount fehlt, aktive alte Zeitpläne gefunden werden oder `BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES` fehlt. Das stellt keine vollständige Sicherheitsfreigabe dar: Skripte und Restore müssen zusätzlich getestet werden.
|
||||
|
||||
## 6. Offene Arbeiten für den Produktivbetrieb
|
||||
|
||||
1. Vorhandene Datenbank-Dump-Skripte unter `scripts/` konsolidieren und auf atomare Dumps unter festen Dateinamen umstellen; Zugänge sicher verwahren.
|
||||
2. Weitere Datenbanken (Kairo, Tsumugu, Kanshō, Buying Agent, BookStack, NetBox usw.) und zu sichernde Docker-Volumes/Bind-Mounts aufnehmen.
|
||||
3. rsync-Skript prüfen: nas-data sendet bei der Umstellung **nicht** gleichzeitig selbst und über Orchestrator Pushs.
|
||||
4. raspiBackup: zuverlässiger vollständiger Restore, korrekt gestoppte Docker-Dienste, Restart-/Timeout-Verhalten nach Fehler.
|
||||
5. Alte Cronjobs/Timer **erst beim kontrollierten Cutover** ersetzen und einen einzigen orchestrator-Timer installieren.
|
||||
6. Vollständigen Testlauf, Restore-Test und erwartete Kuma-Ereignisse prüfen.
|
||||
7. Zweiten unabhängigen/offsite Backup-Standort erwägen; NAS-Ausfall bleibt sonst Single Point of Failure.
|
||||
|
||||
**Hinweis zur bisherigen Zeitplanung:** Die Root-Crontab-Einträge sind nach der letzten Nutzerprüfung auskommentiert; die aktive User-Crontab sowie `raspiBackup.timer` werden weiterhin als aktive Konflikte erkannt.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Backup-Orchestrator 0.4 – Recovery Management: Bestandsaufnahme, Zielarchitektur & Umsetzungsplan
|
||||
|
||||
Dieses Dokument bildet das Ergebnis von **Meilenstein 1 (Bestandsaufnahme und Architektur)** gemäß Abschnitt 16 & 18 des Entwicklungsauftrags. Alle Angaben zu Hosts, Pfaden, Skripten und Laufzuständen wurden am **2026-10-29** lesend auf `raspberrypi5` (`192.168.2.49`), `llm-node` (`192.168.2.144`) und im lokalen Shinkan-Repository (`C:/Dev/shinkan-jinkendo`) verifiziert. Der produktive Codebestand von Version `0.3.0` wurde zudem in das lokale Projektverzeichnis `C:/Dev/backup-orchestrator` übernommen.
|
||||
|
||||
---
|
||||
|
||||
## 1. Ist-Architektur und vorhandene Funktionen (Backup-Orchestrator 0.3.0)
|
||||
|
||||
### 1.1 Bestätigte Fakten (Inventar vom 2026-10-29)
|
||||
|
||||
| Bereich | Bestätigter Ist-Zustand | Quelle / Nachweis |
|
||||
|---|---|---|
|
||||
| **Orchestrator-Code** | `orchestrator.py` (`VERSION = '0.3.0'`, SHA256 `0d07845b…`), identisch unter `/usr/local/sbin/backup-orchestrator`, `/home/lars/pi-backup-orchestrator/orchestrator.py` und `/home/lars/pi-backup-orchestrator-v0.3/orchestrator.py` auf `raspberrypi5`. | `sha256sum` auf `raspberrypi5` |
|
||||
| **Vorhandene Tests** | `tests/test_orchestrator.py` (`OrchestratorTests` + `NotificationTests`, 10 Unit-Tests mit temporären Dummy-Skripten und gemocktem HTTP). | `/home/lars/pi-backup-orchestrator/tests/test_orchestrator.py` |
|
||||
| **Konfiguration (`raspberrypi5`)** | `jobs.csv` (`/etc/backup-orchestrator/jobs.csv`, 19 Jobs: 17 DB-Jobs in Phase 10 `enabled=true`, `nas-data` in Phase 20 `enabled=true`, `system-image` in Phase 30 `enabled=false`). | `/etc/backup-orchestrator/jobs.csv` |
|
||||
| **Zeitsteuerung & Wrapper** | `backup-orchestrator-daily.timer` startet täglich um `05:30:00` den Oneshot-Service `/usr/local/sbin/backup-orchestrator-daily`. Dieser prüft `raspiBackup.service`, deaktiviert temporär `raspiBackup.timer` (Workaround für `legacy_schedules()`), setzt `BACKUP_ORCHESTRATOR_CUTOVER_APPROVED=YES` und ruft `/usr/local/sbin/backup-orchestrator --run` auf. | `/usr/local/sbin/backup-orchestrator-daily` |
|
||||
| **Letzter Produktivlauf** | Am **2026-10-29 um 05:30:12** (`run_id`: `20261029-053012-4129246`) liefen alle 17 DB-Jobs (Phase 10, Dauer 0,32s–3,51s) sowie `nas-data` (Phase 20, Dauer 21,51s) fehlerfrei (`overall: "ok"`). | `/var/log/backup-orchestrator/latest.json` |
|
||||
| **Datenbank-Dumps (Phase 10)** | `db-postgres-docker.sh` (13 Docker-Postgres-DBs), `db-mariadb-docker.sh` (2 Docker-MariaDB/MySQL-DBs: `bookstack_db`, `karatewiki-dev-db-1`), sowie `db-paperless.sh` (mysqldump gegen `192.168.2.40:3308`) und `db-tandoor.sh` (`pg_dump` gegen `192.168.2.40:5432`). Alle schreiben atomar via `mktemp` + `mv -f` nach `/media/lars/paperless/db-backup/<name>.sql` (`0640 root:lars`). | `/etc/backup-orchestrator/scripts/` |
|
||||
| **NAS-Datensicherung (Phase 20)** | `backup-nas-data.sh` sichert `/media/lars/paperless` (lokale NVMe `/dev/nvme0n1p2`) per `rsync -a -v -R --delete --stats --link-dest=/backup/data_archiv/last` in Tagesordner `/backup/data_archiv/01..31` (aktuell `last -> /backup/data_archiv/29`). Enthält außerdem historische zeitgestempelte Dumps (`*_20261002..20261028*.sql`). | `/backup/data_archiv/` |
|
||||
| **Vollständige Systemsicherung** | `raspiBackup.service` läuft täglich um `02:00` nach `/backup/backup/raspberrypi5/` (jüngster Stand: `raspberrypi5@debian12-rsync-backup-20261029-020053`). Stoppt vorab `apache2`, `mariadb`, `gitea-runner`, `docker`, `cron`, `containerd`. | `/usr/local/etc/raspiBackup.conf` |
|
||||
| **Borg-Repositories** | Auf `raspberrypi5`: `/backup/borg` ist initialisiert (`Oct 28 17:48`), enthält aber **noch keine Archive** (`data/0` hat nur 547 Bytes Init-Metadaten). Auf `llm-node`: `/mnt/nas_backup` (`192.168.2.63:/volume1/Backup_LLM`) enthält aktive Borg-Repos (`borg_repo`, `backup-repo`) sowie Disk-Images. | `/backup/borg`, `/mnt/nas_backup` |
|
||||
| **Zielhost `llm-node`** | `llm-node` (`192.168.2.144`, `x86_64`) hat `/etc/shinkan/prod.env` (`0600 llmadmin:llmadmin`), `/home/llmadmin/docker/shinkan-prod` (noch leer, da `Deploy Production llm-node` auf `main` noch nicht ausgelöst wurde) und den NFSv4-Mount `/srv/shinkan/prod/media` (`192.168.2.63:/volume1/shinkan-media/prod`). `backup-orchestrator` ist auf `llm-node` noch nicht installiert. | SSH-Inspektion `llmadmin@192.168.2.144` |
|
||||
|
||||
---
|
||||
|
||||
## 2. Ergebnisse aus der Shinkan-Recovery-Analyse
|
||||
|
||||
Die Vorarbeiten im Shinkan-Repository (`.claude/docs/technical/PROD_DEPLOYMENTS.md` und `.claude/docs/technical/RECOVERY_CONTRACT.md`) definieren bereits einen präzisen Schnittstellenvertrag zwischen dem Backup-Orchestrator und Shinkan:
|
||||
|
||||
1. **Verantwortungstrennung (Orchestrator ↔ Anwendungs-Adapter):**
|
||||
- **Orchestrator:** Zeitsteuerung, Ausführung von Phasen/Jobs, Erzeugung der Dumps, Aufbewahrung/Generationen, NAS-Synchronisation, Ausführungshistorie, Benachrichtigungen (Kuma/Webhooks) sowie CLI-Steuerung für Recovery-Planung und -Auslösung.
|
||||
- **Anwendung (`scripts/ops/shinkan-*`):** Kennt die internen Details (Schema-Migrationen in `schema_migrations` gemäß `backend/run_migrations.py`, Pflicht-Tabellen & Readiness gemäß `health_ready()` in `backend/main.py`, Sessions-Bereinigung, Medien-Struktur unter `/app/media` mit `media_assets` und `exercise_media`).
|
||||
2. **Definierte Shinkan-Kommandos & Exit-Code-Kontrakt:**
|
||||
- `shinkan-backup-manifest --target pi-prod|llm-prod --dump <datei> --out-dir <verzeichnis> --result <json>`
|
||||
- `shinkan-restore --target llm-prod|llm-restoretest --dump <datei> --result <json> [--confirm <db>@<host>] [--allow-overwrite] [--sessions clear|keep]`
|
||||
- `shinkan-restore-test --dump <datei> --result <json> [--playwright] [--keep]`
|
||||
- `shinkan-media-diff --target llm-prod|llm-restoretest --result <json> [--manifest <dump>.media.jsonl]`
|
||||
- **Standardisierte Exit-Codes:** `0` = OK, `1` = Ausführungsfehler, `2` = Vorbedingung/Host/Bestätigung fehlt, `3` = Schema-/Versionsinkompatibilität (Dump enthält Migrationen, die der Code nicht kennt), `4` = Checksummenfehler, `5` = Post-Restore-Check fehlgeschlagen (automatischer Rollback auf `shinkan_before_<ts>` durchgeführt).
|
||||
3. **Wichtige Erkenntnis zum Medienbestand von Shinkan:**
|
||||
- Auf `raspberrypi5` liegt `SHINKAN_MEDIA_HOST=/shinkan-media/prod` auf dem NFSv4-Share `192.168.2.63:/volume1/shinkan-media/prod`. Auf `llm-node` ist derselbe NFSv4-Share unter `/srv/shinkan/prod/media` eingebunden (`docker-compose.llm-node.yml`).
|
||||
- **Kritische Feststellung:** `/shinkan-media/prod` liegt bereits direkt auf dem Synology NAS, wird aber **nicht** durch `backup-nas-data.sh` (welches nur `/media/lars/paperless` sichert) in `/backup/data_archiv/01..31` versioniert! Während des Parallelbetriebs zwischen `raspberrypi5` und `llm-node` teilen sich beide Umgebungen denselben physischen Medienbestand, weshalb `retention-cron` auf `llm-node` über das Compose-Profil `media-retention` deaktiviert bleibt.
|
||||
|
||||
---
|
||||
|
||||
## 3. Gap-Analyse: Backup-Orchestrator 0.3 vs. Recovery Management 0.4
|
||||
|
||||
| Anforderung (Auftrag 0.4) | Stand in Backup-Orchestrator 0.3.0 | Identifizierte Lücke (Gap) & Risiko |
|
||||
|---|---|---|
|
||||
| **Multi-Host-Fähigkeit (`raspberrypi5` & `llm-node`)** | `orchestrator.py` hat `NAS_MOUNT = '/backup'` und `NAS_SOURCE = '192.168.2.63:/volume1/Backup_Paperless'` hartcodiert; `legacy_schedules()` prüft hart den User `lars` und Pi-spezifische Timer. | Läuft auf `llm-node` (`/mnt/nas_backup`, User `llmadmin`) nicht ohne Konfigurationsabstraktion (`host.json` / Host-Profile). |
|
||||
| **Timer-Koordination (`raspiBackup`)** | `legacy_schedules()` blockiert jeden Lauf, sobald `raspiBackup.timer` `enabled` ist – selbst wenn `raspiBackup.service` gar nicht läuft. Deshalb deaktiviert `/usr/local/sbin/backup-orchestrator-daily` den Timer vor jedem Lauf per Workaround. | Muss durch eine zustandsbewusste Prüfung ersetzt werden: `raspiBackup.service` darf nicht `active`/`activating` sein; ein ruhender Timer um 05:30 Uhr ist kein Konflikt. |
|
||||
| **Recovery Points & Manifeste** | Erzeugt nur isolierte `.sql`-Dateien unter festem Namen (`shinkan-prod.sql`). Keine `.sha256`, kein Release-/Commit-/Schema-Manifest, kein Medien-Manifest. | Nach einem Ausfall ist im Dump nicht ohne Weiteres erkennbar, welcher `git_commit`, welche `app_version`, welche `last_migration` und welcher Medienstand dazugehören. |
|
||||
| **Generationen für DB-Sätze** | Lokal wird `<app>.sql` täglich überschrieben; in `/backup/data_archiv/01..31` liegt nur 1 Stand pro Kalendertag ohne atomares Backup-Satz-Bündel (Dump + `.sha256` + Manifest + Media-Index). | Wenn ein verkürzter/fehlerhafter Lauf oder ein 2-stündlicher Takt eingeführt wird, fehlt eine generationssichere Rotation vollständiger Backup-Sätze (`pi-prod` vs. `llm-prod` getrennt). |
|
||||
| **Versions- & Schemakompatibilität** | Keine Prüfung zwischen Datenbank-Migrationsstand (`schema_migrations`) und Ziel-Code (`backend/migrations/*.sql`). | Risiko von Rollback-Brüchen (DB hat neuere Migrationen als der Code) oder unbemerkten Schema-Drifts. |
|
||||
| **Recovery-Planung (`--recovery-plan`) & Ausführung (`--restore`)** | CLI unterstützt nur `--check`, `--plan`, `--status`, `--run`. Keine Recovery-Befehle, kein Dry-Run-Plan, kein Staging-DB-Swap, kein Audit-Log für Restores. | Vollständige Erweiterung um Recovery-Subkommandos bzw. Modi erforderlich. |
|
||||
| **Shinkan-Skripte (`scripts/ops/shinkan-*`)** | In `RECOVERY_CONTRACT.md` spezifiziert (`Status: Entwurf zur Abstimmung, nichts umgesetzt`), unter `C:/Dev/shinkan-jinkendo/scripts/ops/` noch nicht angelegt. | Müssen (nach expliziter Freigabe für das Shinkan-Repo) exakt nach Vertrag implementiert werden. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Zielarchitektur und Recovery-Point-Datenmodell
|
||||
|
||||
Siehe `recovery_management_plan_v0_4.md` für das vollständige JSON-Schema (`schemaVersion: 1`) sowie die 4 Validierungsstufen (`captured`, `verified`, `consistent`, `restore_tested`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Priorisierter Implementierungsplan
|
||||
|
||||
1. **Meilenstein 1:** Bestandsaufnahme & Architekturkonzept (abgeschlossen).
|
||||
2. **Meilenstein 2:** Recovery-Metadaten, Host-Abstraktion (`host.json`), zustandsbewusste `raspiBackup`-Koordination, Manifest-Schema v1, Backup-Satz-Rotation und `shinkan-backup-manifest`.
|
||||
3. **Meilenstein 3:** Application Restore MVP (`--list-recovery-points`, `--recovery-plan`, `--restore`, `--restore-test`), `shinkan-restore`, `shinkan-restore-test`, `shinkan-media-diff` sowie Durchführung der Shinkan-Erstbefüllung auf `llm-node`.
|
||||
4. **Meilenstein 4:** Generische PostgreSQL-/MariaDB-Restore-Adapter für die weiteren 16 DBs sowie Storage-/Borg-Adapter.
|
||||
5. **Meilenstein 5:** System-Recovery-Prozess (`raspiBackup` + Application Recovery).
|
||||
@@ -0,0 +1,20 @@
|
||||
phase,id,script,critical,timeout_seconds,enabled
|
||||
10,paperless-db,/etc/backup-orchestrator/scripts/db-paperless.sh,true,600,true
|
||||
10,tandoor-db,/etc/backup-orchestrator/scripts/db-tandoor.sh,true,600,true
|
||||
10,mitai-prod-db,/etc/backup-orchestrator/scripts/db-mitai-prod.sh,true,900,true
|
||||
10,mitai-dev-db,/etc/backup-orchestrator/scripts/db-mitai-dev.sh,false,900,true
|
||||
10,shinkan-prod-db,/etc/backup-orchestrator/scripts/db-shinkan-prod.sh,true,600,true
|
||||
10,shinkan-dev-db,/etc/backup-orchestrator/scripts/db-shinkan-dev.sh,false,600,true
|
||||
20,nas-data,/etc/backup-orchestrator/scripts/nas-data-job.sh,true,10800,true
|
||||
30,system-image,/etc/backup-orchestrator/scripts/system-image-job.sh,true,14400,false
|
||||
10,kairo-prod-db,/etc/backup-orchestrator/scripts/db-kairo-prod.sh,true,900,true
|
||||
10,kairo-dev-db,/etc/backup-orchestrator/scripts/db-kairo-dev.sh,false,900,true
|
||||
10,tsumugu-prod-db,/etc/backup-orchestrator/scripts/db-tsumugu-prod.sh,true,900,true
|
||||
10,tsumugu-dev-db,/etc/backup-orchestrator/scripts/db-tsumugu-dev.sh,false,900,true
|
||||
10,kansho-prod-db,/etc/backup-orchestrator/scripts/db-kansho-prod.sh,true,900,true
|
||||
10,kansho-dev-db,/etc/backup-orchestrator/scripts/db-kansho-dev.sh,false,900,true
|
||||
10,buyingagent-prod-db,/etc/backup-orchestrator/scripts/db-buyingagent-prod.sh,true,900,true
|
||||
10,buyingagent-dev-db,/etc/backup-orchestrator/scripts/db-buyingagent-dev.sh,false,900,true
|
||||
10,netbox-db,/etc/backup-orchestrator/scripts/db-netbox.sh,true,900,true
|
||||
10,bookstack-db,/etc/backup-orchestrator/scripts/db-bookstack.sh,true,900,true
|
||||
10,karatewiki-dev-db,/etc/backup-orchestrator/scripts/db-karatewiki-dev.sh,false,900,true
|
||||
|
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"timeout_seconds": 3,
|
||||
"webhooks": [],
|
||||
"kuma": {
|
||||
"overall_push_url": "",
|
||||
"job_push_urls": {
|
||||
"nas-data": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
+600
@@ -0,0 +1,600 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase-based backup orchestrator v0.3.
|
||||
|
||||
Safety defaults: read-only inspection unless --run, jobs enabled and explicit cutover
|
||||
approval. No scheduler, database or Docker changes are performed by this program.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from collections import defaultdict
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
import csv
|
||||
import datetime as dt
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from urllib.parse import urlencode, urlsplit, urlunsplit
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
VERSION = '0.3.0'
|
||||
DEFAULT_CONFIG = Path('/etc/backup-orchestrator/jobs.csv')
|
||||
DEFAULT_NOTIFY_CONFIG = Path('/etc/backup-orchestrator/notifications.json')
|
||||
DEFAULT_LOG_ROOT = Path('/var/log/backup-orchestrator')
|
||||
DEFAULT_LOCK_FILE = Path('/run/lock/backup-orchestrator.lock')
|
||||
FIELDS = {'phase', 'id', 'script', 'critical', 'timeout_seconds', 'enabled'}
|
||||
CUTOVER_ENV = 'BACKUP_ORCHESTRATOR_CUTOVER_APPROVED'
|
||||
NAS_MOUNT = '/backup'
|
||||
NAS_SOURCE = '192.168.2.63:/volume1/Backup_Paperless'
|
||||
CANCELLED = threading.Event()
|
||||
|
||||
|
||||
def now_iso():
|
||||
return dt.datetime.now(dt.timezone.utc).astimezone().isoformat(timespec='seconds')
|
||||
|
||||
|
||||
def load_jobs(path):
|
||||
with path.open(newline='', encoding='utf-8-sig') as fp:
|
||||
reader = csv.DictReader(fp)
|
||||
if reader.fieldnames is None or set(reader.fieldnames) != FIELDS or len(reader.fieldnames) != len(FIELDS):
|
||||
raise ValueError('CSV header must contain exactly: ' + ','.join(sorted(FIELDS)))
|
||||
rows = list(reader)
|
||||
if not rows:
|
||||
raise ValueError('CSV contains no jobs')
|
||||
ids = set()
|
||||
jobs = []
|
||||
for index, row in enumerate(rows, start=2):
|
||||
if None in row or any(v is None for v in row.values()):
|
||||
raise ValueError(f'CSV line {index}: wrong number of columns')
|
||||
row = {k: v.strip() for k, v in row.items()}
|
||||
jid = row['id']
|
||||
if not re.fullmatch(r'[a-z][a-z0-9-]*', jid) or jid in ids:
|
||||
raise ValueError(f'CSV line {index}: invalid or duplicate job ID: {jid!r}')
|
||||
ids.add(jid)
|
||||
if row['critical'] not in ('true', 'false') or row['enabled'] not in ('true', 'false'):
|
||||
raise ValueError(f'CSV line {index}: critical and enabled must be true or false')
|
||||
if not re.fullmatch(r'\d+', row['phase']) or int(row['phase']) < 1:
|
||||
raise ValueError(f'CSV line {index}: phase must be a positive integer')
|
||||
if not re.fullmatch(r'\d+', row['timeout_seconds']) or not 1 <= int(row['timeout_seconds']) <= 86400:
|
||||
raise ValueError(f'CSV line {index}: timeout_seconds must be 1..86400')
|
||||
if not Path(row['script']).is_absolute():
|
||||
raise ValueError(f'CSV line {index}: script must be an absolute path')
|
||||
jobs.append({
|
||||
'phase': int(row['phase']), 'id': jid, 'script': row['script'],
|
||||
'critical': row['critical'] == 'true',
|
||||
'timeout_seconds': int(row['timeout_seconds']),
|
||||
'enabled': row['enabled'] == 'true',
|
||||
})
|
||||
return sorted(jobs, key=lambda x: (x['phase'], x['id']))
|
||||
|
||||
|
||||
def push_base_url(value):
|
||||
"""Accept Kuma's full copied URL and discard its example ?status=... query."""
|
||||
parsed = urlsplit(value)
|
||||
if parsed.scheme not in ('http', 'https') or not parsed.netloc or not parsed.path.startswith('/api/push/'):
|
||||
raise ValueError('Kuma Push URL must be an http(s) URL with /api/push/<token>')
|
||||
if parsed.username is not None or parsed.password is not None:
|
||||
raise ValueError('HTTP basic-auth in Push URL is unsupported')
|
||||
return urlunsplit((parsed.scheme, parsed.netloc, parsed.path, '', ''))
|
||||
|
||||
|
||||
def verify_http_url(value):
|
||||
parsed = urlsplit(value)
|
||||
if parsed.scheme not in ('http', 'https') or not parsed.netloc:
|
||||
raise ValueError('Webhook URL must start with http:// or https://')
|
||||
if parsed.username is not None or parsed.password is not None:
|
||||
raise ValueError('Webhook URL must not contain user credentials')
|
||||
return value
|
||||
|
||||
|
||||
def load_notifications(path, known_ids=None):
|
||||
"""Only outgoing HTTP. URLs are never printed in status/logs."""
|
||||
default = {'timeout_seconds': 3, 'webhooks': [], 'kuma': {'overall_push_url': '', 'job_push_urls': {}}}
|
||||
if not path.is_file():
|
||||
return default
|
||||
with path.open(encoding='utf-8') as fp:
|
||||
obj = json.load(fp)
|
||||
if not isinstance(obj, dict) or set(obj) - {'timeout_seconds', 'webhooks', 'kuma'}:
|
||||
raise ValueError('Invalid notifications.json root keys')
|
||||
timeout = obj.get('timeout_seconds', 3)
|
||||
if type(timeout) not in (int, float) or not 0.5 <= timeout <= 15:
|
||||
raise ValueError('timeout_seconds must be 0.5..15')
|
||||
webhooks = obj.get('webhooks', [])
|
||||
kuma = obj.get('kuma', {})
|
||||
if not isinstance(webhooks, list) or any(not isinstance(x, str) for x in webhooks):
|
||||
raise ValueError('webhooks must be a list of URLs')
|
||||
if not isinstance(kuma, dict) or set(kuma) - {'overall_push_url', 'job_push_urls'}:
|
||||
raise ValueError('Invalid kuma settings')
|
||||
overall = kuma.get('overall_push_url', '')
|
||||
per_job = kuma.get('job_push_urls', {})
|
||||
if not isinstance(overall, str) or not isinstance(per_job, dict):
|
||||
raise ValueError('Kuma URLs must be strings; job_push_urls must be an object')
|
||||
if any(not isinstance(k, str) or not isinstance(v, str) for k, v in per_job.items()):
|
||||
raise ValueError('Kuma job IDs and URLs must be strings')
|
||||
if known_ids is not None and set(per_job) - set(known_ids):
|
||||
raise ValueError('Unknown Kuma job ID(s): ' + ', '.join(sorted(set(per_job) - set(known_ids))))
|
||||
return {
|
||||
'timeout_seconds': timeout,
|
||||
'webhooks': [verify_http_url(x) for x in webhooks],
|
||||
'kuma': {
|
||||
'overall_push_url': push_base_url(overall) if overall else '',
|
||||
'job_push_urls': {k: push_base_url(v) if v else '' for k, v in per_job.items()},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def notification_counts(config):
|
||||
kuma = config['kuma']
|
||||
return len(config['webhooks']), int(bool(kuma['overall_push_url'])), sum(bool(v) for v in kuma['job_push_urls'].values())
|
||||
|
||||
|
||||
def nas_mount_ready():
|
||||
try:
|
||||
result = subprocess.run(['findmnt', '--mountpoint', NAS_MOUNT, '--types', 'nfs4', '--source', NAS_SOURCE],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=8)
|
||||
return result.returncode == 0
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
|
||||
def legacy_schedules():
|
||||
"""Best-effort detection of active legacy schedules (never comments)."""
|
||||
found = []
|
||||
for user in ('root', 'lars'):
|
||||
try:
|
||||
p = subprocess.run(['crontab', '-u', user, '-l'], capture_output=True, text=True, timeout=8)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
for line in p.stdout.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith('#') or not stripped:
|
||||
continue
|
||||
if re.search(r'/home/lars/(?:db-[\w-]+\.sh|backupPaperlessData\.sh)', stripped):
|
||||
found.append(f'{user} crontab: {stripped}')
|
||||
for timer in ('raspiBackup.timer', 'paperless-data-backup.timer', 'borgmatic.timer'):
|
||||
try:
|
||||
enabled = subprocess.run(['systemctl', 'is-enabled', timer], capture_output=True, text=True, timeout=8).stdout.strip()
|
||||
active = subprocess.run(['systemctl', 'is-active', timer], capture_output=True, text=True, timeout=8).stdout.strip()
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
if enabled in ('enabled', 'enabled-runtime', 'linked', 'linked-runtime') or active == 'active':
|
||||
found.append(f'{timer}: enabled={enabled or "unknown"}, active={active or "unknown"}')
|
||||
return found
|
||||
|
||||
|
||||
def script_issues(job):
|
||||
p = Path(job['script'])
|
||||
if not p.is_file():
|
||||
return ['MISSING']
|
||||
issues = []
|
||||
if not os.access(p, os.X_OK):
|
||||
issues.append('NOT EXECUTABLE')
|
||||
try:
|
||||
with p.open('rb') as fp:
|
||||
if fp.read(2) != b'#!':
|
||||
issues.append('NO SHEBANG')
|
||||
except OSError:
|
||||
issues.append('NOT READABLE')
|
||||
return issues
|
||||
|
||||
|
||||
def check(jobs, max_parallel, notify):
|
||||
print(f'Backup-Orchestrator {VERSION}: INSPECTION ONLY (no backups, no schedule changes)')
|
||||
print(f'Parallel limit per phase: {max_parallel}')
|
||||
for job in jobs:
|
||||
issues = script_issues(job)
|
||||
state = 'enabled' if job['enabled'] else 'disabled'
|
||||
print(f"phase {job['phase']:>2} {job['id']:<24} {state:<8} {job['timeout_seconds']:>5}s {job['script']} [{', '.join(issues) if issues else 'OK'}]")
|
||||
conflicts = legacy_schedules()
|
||||
print('\nLegacy schedules:')
|
||||
for line in conflicts:
|
||||
print(' ACTIVE: ' + line)
|
||||
if not conflicts:
|
||||
print(' None detected (best effort)')
|
||||
print(f'\nNAS mount {NAS_MOUNT}: {"OK" if nas_mount_ready() else "NOT AVAILABLE"} (expected {NAS_SOURCE})')
|
||||
a, b, c = notification_counts(notify)
|
||||
print(f'Notifications configured: webhooks={a}, Kuma overall={b}, Kuma jobs={c} (tokens hidden)')
|
||||
print('Same phase: parallel; next phase starts after all members complete.')
|
||||
print('Critical job failure: later phases BLOCKED; in-flight phase members finish.')
|
||||
print('No jobs execute without --run, enabled=true and explicit cutover approval.')
|
||||
return 0
|
||||
|
||||
|
||||
def select_jobs(jobs, args):
|
||||
enabled = [j for j in jobs if j['enabled']]
|
||||
if args.job:
|
||||
matches = [j for j in enabled if j['id'] == args.job]
|
||||
if not matches:
|
||||
raise ValueError(f'Job {args.job!r} not found or disabled')
|
||||
return matches
|
||||
if args.phase is not None:
|
||||
matches = [j for j in enabled if j['phase'] == args.phase]
|
||||
if not matches:
|
||||
raise ValueError(f'No enabled jobs in phase {args.phase}')
|
||||
return matches
|
||||
return enabled
|
||||
|
||||
|
||||
def plan(jobs, max_parallel):
|
||||
groups = defaultdict(list)
|
||||
for job in jobs:
|
||||
groups[job['phase']].append(job)
|
||||
print(f'MODE: PLAN ONLY — no jobs executed. Max parallel jobs per phase: {max_parallel}')
|
||||
for phase, members in sorted(groups.items()):
|
||||
print(f'Phase {phase} ({len(members)} job(s)):')
|
||||
for job in members:
|
||||
print(f" - {job['id']} ({'CRITICAL' if job['critical'] else 'optional'}, timeout {job['timeout_seconds']}s)")
|
||||
print(' BARRIER: wait for all jobs before advancing')
|
||||
|
||||
|
||||
def result_for(job, state, rc=None, seconds=0.0, logfile=None, reason=None):
|
||||
result = dict(phase=job['phase'], job=job['id'], critical=job['critical'],
|
||||
state=state, exit_code=rc, duration_seconds=round(seconds, 2))
|
||||
if logfile is not None:
|
||||
result['log'] = str(logfile)
|
||||
if reason:
|
||||
result['reason'] = reason
|
||||
return result
|
||||
|
||||
|
||||
def atomic_json(path, data):
|
||||
tmp = path.with_name(f'.{path.name}.{os.getpid()}.{threading.get_ident()}.tmp')
|
||||
try:
|
||||
with tmp.open('w', encoding='utf-8') as fp:
|
||||
json.dump(data, fp, ensure_ascii=False, indent=2)
|
||||
fp.write('\n')
|
||||
tmp.replace(path)
|
||||
finally:
|
||||
if tmp.exists():
|
||||
tmp.unlink()
|
||||
|
||||
|
||||
class Notifier:
|
||||
"""Outbound webhooks + Kuma adapter; notification errors never alter backup exit codes."""
|
||||
def __init__(self, config, folder):
|
||||
self.config = config
|
||||
self.folder = folder
|
||||
self.pool = ThreadPoolExecutor(max_workers=3, thread_name_prefix='backup-notify')
|
||||
self.futures = []
|
||||
self.lock = threading.Lock()
|
||||
self.errors = 0
|
||||
self.logfile = folder / 'notifications.log'
|
||||
|
||||
def _send_json(self, url, event):
|
||||
payload = json.dumps(event, ensure_ascii=False).encode('utf-8')
|
||||
req = Request(url, data=payload, method='POST', headers={'Content-Type': 'application/json', 'User-Agent': 'pi-backup-orchestrator/0.3'})
|
||||
with urlopen(req, timeout=self.config['timeout_seconds']) as response:
|
||||
if not 200 <= response.status < 300:
|
||||
raise RuntimeError(f'HTTP {response.status}')
|
||||
|
||||
def _send_kuma(self, url, status, message):
|
||||
sep = '&' if '?' in url else '?'
|
||||
address = url + sep + urlencode({'status': status, 'msg': message[:180]})
|
||||
req = Request(address, headers={'User-Agent': 'pi-backup-orchestrator/0.3'})
|
||||
with urlopen(req, timeout=self.config['timeout_seconds']) as response:
|
||||
if not 200 <= response.status < 300:
|
||||
raise RuntimeError(f'HTTP {response.status}')
|
||||
# Kuma usually responds with JSON {ok:true}; a 200 alone is insufficient if it says failure.
|
||||
body = response.read(4096)
|
||||
if body:
|
||||
try:
|
||||
parsed = json.loads(body)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
parsed = None
|
||||
if isinstance(parsed, dict) and parsed.get('ok') is False:
|
||||
raise RuntimeError('Kuma rejected push')
|
||||
|
||||
def _submit(self, channel, fn, *args):
|
||||
def guarded():
|
||||
try:
|
||||
fn(*args)
|
||||
except Exception as exc:
|
||||
# Never log target URLs or tokens; even HTTP exceptions may include their URL.
|
||||
if hasattr(exc, 'code') and isinstance(exc.code, int):
|
||||
detail = f'HTTP {exc.code}'
|
||||
else:
|
||||
detail = type(exc).__name__
|
||||
with self.lock:
|
||||
self.errors += 1
|
||||
with self.logfile.open('a', encoding='utf-8') as out:
|
||||
out.write(f'{now_iso()} {channel}: delivery failed ({detail})\n')
|
||||
self.futures.append(self.pool.submit(guarded))
|
||||
|
||||
def notify(self, event):
|
||||
for url in self.config['webhooks']:
|
||||
self._submit('webhook', self._send_json, url, event)
|
||||
name = event['event']
|
||||
if name in ('JOB_SUCCESS', 'JOB_FAILED', 'JOB_TIMEOUT', 'JOB_CANCELLED', 'JOB_ERROR', 'JOB_BLOCKED'):
|
||||
url = self.config['kuma']['job_push_urls'].get(event.get('job'), '')
|
||||
if url:
|
||||
status = 'up' if name == 'JOB_SUCCESS' else 'down'
|
||||
msg = f"{event['job']}: {event.get('state', 'unknown')} (run {event['run_id']})"
|
||||
self._submit('kuma/job', self._send_kuma, url, status, msg)
|
||||
if name == 'RUN_FINISHED':
|
||||
url = self.config['kuma']['overall_push_url']
|
||||
if url:
|
||||
status = 'up' if event.get('overall') == 'ok' else 'down'
|
||||
self._submit('kuma/overall', self._send_kuma, url, status, f"Run {event['run_id']}: {event.get('overall')}")
|
||||
|
||||
def close(self):
|
||||
self.pool.shutdown(wait=True)
|
||||
return self.errors
|
||||
|
||||
|
||||
class EventHub:
|
||||
"""Write append-only events.jsonl plus atomic current snapshot for consumers."""
|
||||
def __init__(self, folder, log_root, run_id, jobs, notifier, started):
|
||||
self.folder = folder
|
||||
self.log_root = log_root
|
||||
self.run_id = run_id
|
||||
self.notifier = notifier
|
||||
self.lock = threading.RLock()
|
||||
self.snapshot = {
|
||||
'version': VERSION, 'run_id': run_id, 'host': socket.gethostname(),
|
||||
'started': started, 'updated': started, 'finished': None,
|
||||
'overall': 'running',
|
||||
'jobs': {j['id']: result_for(j, 'pending') for j in jobs},
|
||||
}
|
||||
atomic_json(folder / 'status.json', self.snapshot)
|
||||
atomic_json(log_root / 'latest.json', self.snapshot)
|
||||
|
||||
def emit(self, event_name, **fields):
|
||||
event = {
|
||||
'event': event_name, 'timestamp': now_iso(), 'run_id': self.run_id,
|
||||
'host': self.snapshot['host'], **fields,
|
||||
}
|
||||
with self.lock:
|
||||
if 'job' in event and event['job'] in self.snapshot['jobs'] and 'state' in event:
|
||||
jid = event['job']
|
||||
self.snapshot['jobs'][jid].update({k: v for k, v in fields.items() if k in (
|
||||
'state', 'phase', 'critical', 'exit_code', 'duration_seconds', 'log', 'reason')})
|
||||
if event_name == 'RUN_FINISHED':
|
||||
self.snapshot['overall'] = fields['overall']
|
||||
self.snapshot['finished'] = event['timestamp']
|
||||
self.snapshot['updated'] = event['timestamp']
|
||||
with (self.folder / 'events.jsonl').open('a', encoding='utf-8') as fp:
|
||||
fp.write(json.dumps(event, ensure_ascii=False) + '\n')
|
||||
atomic_json(self.folder / 'status.json', self.snapshot)
|
||||
atomic_json(self.log_root / 'latest.json', self.snapshot)
|
||||
self.notifier.notify(event)
|
||||
return event
|
||||
|
||||
|
||||
def terminate_group(proc):
|
||||
"""Terminate ordinary children of a job and wait for shutdown before advancing phase."""
|
||||
if proc.poll() is not None:
|
||||
return
|
||||
try:
|
||||
os.killpg(proc.pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
os.killpg(proc.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
proc.wait()
|
||||
|
||||
|
||||
def execute_one(job, folder, hub):
|
||||
start = time.monotonic()
|
||||
logfile = folder / (job['id'] + '.log')
|
||||
if CANCELLED.is_set():
|
||||
result = result_for(job, 'cancelled', reason='Orchestrator interrupted')
|
||||
hub.emit('JOB_CANCELLED', **result)
|
||||
return result
|
||||
hub.emit('JOB_STARTED', job=job['id'], phase=job['phase'], critical=job['critical'], state='running')
|
||||
with logfile.open('wb') as stream:
|
||||
stream.write((f"job={job['id']} started={now_iso()}\n").encode())
|
||||
stream.flush()
|
||||
try:
|
||||
proc = subprocess.Popen([job['script']], stdout=stream, stderr=subprocess.STDOUT,
|
||||
start_new_session=True, cwd=str(Path(job['script']).parent))
|
||||
except OSError as exc:
|
||||
stream.write((f'EXEC ERROR: {exc}\n').encode())
|
||||
result = result_for(job, 'error', 127, time.monotonic()-start, logfile, 'Could not start script')
|
||||
hub.emit('JOB_ERROR', **result)
|
||||
return result
|
||||
deadline = time.monotonic() + job['timeout_seconds']
|
||||
while True:
|
||||
if CANCELLED.is_set():
|
||||
terminate_group(proc)
|
||||
state, code = 'cancelled', 130
|
||||
break
|
||||
if time.monotonic() >= deadline:
|
||||
terminate_group(proc)
|
||||
state, code = 'timeout', 124
|
||||
break
|
||||
try:
|
||||
code = proc.wait(timeout=min(0.5, max(0.01, deadline-time.monotonic())))
|
||||
state = 'ok' if code == 0 else 'failed'
|
||||
break
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
stream.write((f'\njob={job["id"]} finished={now_iso()} status={state} exit_code={code}\n').encode())
|
||||
result = result_for(job, state, code, time.monotonic()-start, logfile)
|
||||
event = {'ok': 'JOB_SUCCESS', 'failed': 'JOB_FAILED', 'timeout': 'JOB_TIMEOUT', 'cancelled': 'JOB_CANCELLED'}[state]
|
||||
hub.emit(event, **result)
|
||||
return result
|
||||
|
||||
|
||||
def live_run(jobs, args):
|
||||
if not jobs:
|
||||
print('Nothing enabled; no backups executed.')
|
||||
return 0
|
||||
if os.environ.get(CUTOVER_ENV) != 'YES':
|
||||
print(f'REFUSED: Set {CUTOVER_ENV}=YES only after testing scripts and coordinating existing schedules.', file=sys.stderr)
|
||||
return 2
|
||||
if not nas_mount_ready():
|
||||
print(f'REFUSED: {NAS_MOUNT} is not mounted from {NAS_SOURCE}', file=sys.stderr)
|
||||
return 2
|
||||
conflicts = legacy_schedules()
|
||||
if conflicts:
|
||||
print('REFUSED: existing scheduled backups could overlap this run:', file=sys.stderr)
|
||||
for issue in conflicts:
|
||||
print(' ' + issue, file=sys.stderr)
|
||||
return 2
|
||||
unsafe = [(j['id'], script_issues(j)) for j in jobs if script_issues(j)]
|
||||
if unsafe:
|
||||
for jid, issues in unsafe:
|
||||
print(f'REFUSED: {jid}: {", ".join(issues)}', file=sys.stderr)
|
||||
return 2
|
||||
notifications = load_notifications(args.notify_config)
|
||||
args.log_root.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
args.lock_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
with args.lock_file.open('a+') as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
print('REFUSED: another orchestrator run holds the lock', file=sys.stderr)
|
||||
return 3
|
||||
run_id = dt.datetime.now().strftime('%Y%m%d-%H%M%S') + f'-{os.getpid()}'
|
||||
folder = args.log_root / run_id
|
||||
folder.mkdir(mode=0o700)
|
||||
old_int = signal.getsignal(signal.SIGINT)
|
||||
old_term = signal.getsignal(signal.SIGTERM)
|
||||
CANCELLED.clear()
|
||||
signal.signal(signal.SIGINT, lambda signum, frame: CANCELLED.set())
|
||||
signal.signal(signal.SIGTERM, lambda signum, frame: CANCELLED.set())
|
||||
results = []
|
||||
critical_failed = False
|
||||
groups = defaultdict(list)
|
||||
for job in jobs:
|
||||
groups[job['phase']].append(job)
|
||||
began = now_iso()
|
||||
notifier = Notifier(notifications, folder)
|
||||
hub = EventHub(folder, args.log_root, run_id, jobs, notifier, began)
|
||||
try:
|
||||
hub.emit('RUN_STARTED', max_parallel=args.max_parallel, job_count=len(jobs))
|
||||
for phase, members in sorted(groups.items()):
|
||||
if critical_failed or CANCELLED.is_set():
|
||||
reason = 'earlier critical failure' if critical_failed else 'orchestrator interrupted'
|
||||
hub.emit('PHASE_BLOCKED', phase=phase, reason=reason)
|
||||
for job in members:
|
||||
result = result_for(job, 'blocked', reason=reason)
|
||||
results.append(result)
|
||||
hub.emit('JOB_BLOCKED', **result)
|
||||
print(f'Phase {phase} BLOCKED: {reason}', flush=True)
|
||||
continue
|
||||
hub.emit('PHASE_STARTED', phase=phase, job_count=len(members))
|
||||
print(f'Phase {phase}: starting {len(members)} job(s), max parallel {args.max_parallel}', flush=True)
|
||||
phase_results = []
|
||||
with ThreadPoolExecutor(max_workers=args.max_parallel) as pool:
|
||||
future_to_job = {pool.submit(execute_one, job, folder, hub): job for job in members}
|
||||
for future in as_completed(future_to_job):
|
||||
job = future_to_job[future]
|
||||
try:
|
||||
item = future.result()
|
||||
except Exception as exc:
|
||||
item = result_for(job, 'error', 127, reason=f'Internal runner error: {type(exc).__name__}')
|
||||
hub.emit('JOB_ERROR', **item)
|
||||
phase_results.append(item)
|
||||
print(f' {item["job"]}: {item["state"]} ({item["exit_code"]})', flush=True)
|
||||
results.extend(sorted(phase_results, key=lambda r: r['job']))
|
||||
if any(r['critical'] and r['state'] != 'ok' for r in phase_results):
|
||||
critical_failed = True
|
||||
hub.emit('PHASE_COMPLETED', phase=phase, failures=sum(r['state'] != 'ok' for r in phase_results),
|
||||
critical_failed=critical_failed)
|
||||
if critical_failed:
|
||||
print(f'Phase {phase}: critical failure; subsequent phases will not start.', flush=True)
|
||||
else:
|
||||
print(f'Phase {phase}: barrier reached; next phase may start.', flush=True)
|
||||
finally:
|
||||
signal.signal(signal.SIGINT, old_int)
|
||||
signal.signal(signal.SIGTERM, old_term)
|
||||
overall = 'interrupted' if CANCELLED.is_set() else 'ok' if all(r['state'] == 'ok' for r in results) else 'failed'
|
||||
summary = {
|
||||
'version': VERSION, 'run_id': run_id, 'host': socket.gethostname(),
|
||||
'started': began, 'finished': now_iso(), 'max_parallel': args.max_parallel,
|
||||
'interrupted': CANCELLED.is_set(), 'results': results, 'overall': overall,
|
||||
}
|
||||
atomic_json(folder / 'summary.json', summary)
|
||||
hub.emit('RUN_FINISHED', overall=overall, success_count=sum(r['state'] == 'ok' for r in results),
|
||||
failure_count=sum(r['state'] != 'ok' for r in results))
|
||||
delivery_failures = notifier.close()
|
||||
if delivery_failures:
|
||||
print(f'WARNING: {delivery_failures} notification delivery failure(s); see {folder / "notifications.log"}', flush=True)
|
||||
print('Summary: ' + str(folder / 'summary.json'), flush=True)
|
||||
return 130 if CANCELLED.is_set() else (1 if any(r['state'] != 'ok' for r in results) else 0)
|
||||
|
||||
|
||||
def show_status(log_root, as_json=False):
|
||||
latest = log_root / 'latest.json'
|
||||
if latest.is_file():
|
||||
with latest.open(encoding='utf-8') as fp:
|
||||
data = json.load(fp)
|
||||
if as_json:
|
||||
print(json.dumps(data, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
print(f"Run {data['run_id']}: {data['overall']} ({data['started']} -> {data['finished'] or 'running'})")
|
||||
for item in sorted(data['jobs'].values(), key=lambda x: (x['phase'], x['job'])):
|
||||
print(f" Phase {item['phase']:<3} {item['job']:<24} {item['state']:<10} {item['duration_seconds']}s")
|
||||
return 0
|
||||
summaries = sorted(log_root.glob('*/summary.json')) if log_root.is_dir() else []
|
||||
if not summaries:
|
||||
if as_json:
|
||||
print(json.dumps({'overall': 'no_runs', 'jobs': {}}, indent=2))
|
||||
else:
|
||||
print('No runs recorded yet')
|
||||
return 0
|
||||
with summaries[-1].open(encoding='utf-8') as fp:
|
||||
data = json.load(fp)
|
||||
if as_json:
|
||||
print(json.dumps(data, ensure_ascii=False, indent=2))
|
||||
else:
|
||||
print(f"Last run: {data['started']} -> {data['finished']} ; overall={data['overall']}")
|
||||
for item in data['results']:
|
||||
print(f" Phase {item['phase']:<3} {item['job']:<24} {item['state']:<10} {item['duration_seconds']}s")
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Phase-based backup orchestrator; safe inspection by default')
|
||||
mode = parser.add_mutually_exclusive_group(required=True)
|
||||
mode.add_argument('--check', action='store_true', help='Validate config and inspect schedules (read only)')
|
||||
mode.add_argument('--plan', action='store_true', help='Show phase plan (read only)')
|
||||
mode.add_argument('--status', action='store_true', help='Latest job status (read only)')
|
||||
mode.add_argument('--run', action='store_true', help='Execute enabled jobs (safety-gated)')
|
||||
parser.add_argument('--json', action='store_true', help='Machine-readable result for --status')
|
||||
parser.add_argument('--config', type=Path, default=DEFAULT_CONFIG)
|
||||
parser.add_argument('--notify-config', type=Path, default=DEFAULT_NOTIFY_CONFIG)
|
||||
parser.add_argument('--log-root', type=Path, default=DEFAULT_LOG_ROOT)
|
||||
parser.add_argument('--lock-file', type=Path, default=DEFAULT_LOCK_FILE)
|
||||
parser.add_argument('--max-parallel', type=int, default=3, help='Maximum concurrent jobs in a phase (1..16)')
|
||||
selection = parser.add_mutually_exclusive_group()
|
||||
selection.add_argument('--phase', type=int, help='Select one phase (--plan or --run)')
|
||||
selection.add_argument('--job', type=str, help='Select one job (--plan or --run)')
|
||||
args = parser.parse_args()
|
||||
if not 1 <= args.max_parallel <= 16:
|
||||
parser.error('--max-parallel must be 1..16')
|
||||
if args.json and not args.status:
|
||||
parser.error('--json is only valid with --status')
|
||||
if (args.job or args.phase is not None) and not (args.run or args.plan):
|
||||
parser.error('--phase and --job are only valid with --plan or --run')
|
||||
if args.status:
|
||||
return show_status(args.log_root, as_json=args.json)
|
||||
jobs = load_jobs(args.config)
|
||||
notify = load_notifications(args.notify_config, known_ids=[j['id'] for j in jobs])
|
||||
if args.check:
|
||||
return check(jobs, args.max_parallel, notify)
|
||||
jobs = select_jobs(jobs, args)
|
||||
if args.plan:
|
||||
return plan(jobs, args.max_parallel) or 0
|
||||
return live_run(jobs, args)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
sys.exit(main())
|
||||
except (ValueError, OSError, json.JSONDecodeError) as exc:
|
||||
print(f'ERROR: {exc}', file=sys.stderr)
|
||||
sys.exit(2)
|
||||
@@ -0,0 +1,257 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================
|
||||
# backupPaperlessData.sh
|
||||
# Versionierte NAS-Sicherung mit rsync
|
||||
# ============================================================
|
||||
|
||||
set -Eeuo pipefail
|
||||
umask 027
|
||||
|
||||
# ============================================================
|
||||
# KONFIGURATION
|
||||
# ============================================================
|
||||
|
||||
SOURCE="/media/lars/paperless"
|
||||
|
||||
NAS_MOUNT="/backup"
|
||||
NAS_SOURCE="192.168.2.63:/volume1/Backup_Paperless"
|
||||
|
||||
TARGET="/backup/data_archiv"
|
||||
LOG="/home/lars/backupPaperlessData.sh.log"
|
||||
|
||||
# Monatliche Rotation: 01 bis 31
|
||||
TODAY="$(date +%d)"
|
||||
|
||||
DEST="$TARGET/$TODAY"
|
||||
LAST="$TARGET/last"
|
||||
|
||||
STAGING=""
|
||||
OLD_DEST=""
|
||||
TEMP_LINK=""
|
||||
|
||||
# ============================================================
|
||||
# LOGGING
|
||||
# ============================================================
|
||||
|
||||
exec > >(tee -a "$LOG") 2>&1
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# FEHLERBEHANDLUNG
|
||||
# ============================================================
|
||||
|
||||
on_exit() {
|
||||
local rc=$?
|
||||
|
||||
trap - EXIT
|
||||
|
||||
if (( rc != 0 )); then
|
||||
log "FEHLER: Backup abgebrochen (Exit-Code $rc)"
|
||||
|
||||
fi
|
||||
|
||||
# Unvollstaendige temporaere Sicherung entfernen.
|
||||
# Bestehende Tages-Snapshots bleiben erhalten.
|
||||
|
||||
if [[ -n "$STAGING" && -d "$STAGING" ]]; then
|
||||
rm -rf -- "$STAGING" || true
|
||||
fi
|
||||
|
||||
if [[ -n "$TEMP_LINK" && -L "$TEMP_LINK" ]]; then
|
||||
rm -f -- "$TEMP_LINK" || true
|
||||
fi
|
||||
}
|
||||
|
||||
trap on_exit EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
|
||||
fail() {
|
||||
log "FEHLER: $*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# START
|
||||
# ============================================================
|
||||
|
||||
log "========================================"
|
||||
log "NAS-Datenbackup gestartet"
|
||||
log "Quelle: $SOURCE"
|
||||
log "Ziel: $DEST"
|
||||
|
||||
|
||||
# ============================================================
|
||||
# PARALLELE BACKUPS VERHINDERN
|
||||
# ============================================================
|
||||
|
||||
exec 9< "$0"
|
||||
|
||||
if ! flock -n 9; then
|
||||
fail "Ein weiterer Backup-Prozess laeuft bereits"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# NAS-MOUNT PRUEFEN
|
||||
# ============================================================
|
||||
|
||||
if ! findmnt \
|
||||
--mountpoint "$NAS_MOUNT" \
|
||||
--types nfs4 \
|
||||
--source "$NAS_SOURCE" \
|
||||
>/dev/null; then
|
||||
|
||||
fail "Synology-NFS-Freigabe nicht eingebunden: $NAS_MOUNT"
|
||||
fi
|
||||
|
||||
log "NAS-Mount erfolgreich geprueft"
|
||||
|
||||
# ============================================================
|
||||
# QUELLE UND ZIEL PRUEFEN
|
||||
# ============================================================
|
||||
|
||||
[[ -d "$SOURCE" ]] ||
|
||||
fail "Quellverzeichnis fehlt: $SOURCE"
|
||||
|
||||
[[ -n "$(find "$SOURCE" -mindepth 1 -maxdepth 1 -print -quit)" ]] ||
|
||||
fail "Quellverzeichnis ist leer: $SOURCE"
|
||||
|
||||
[[ -d "$TARGET" ]] ||
|
||||
fail "Backup-Zielverzeichnis fehlt: $TARGET"
|
||||
|
||||
[[ -w "$TARGET" ]] ||
|
||||
fail "Backup-Zielverzeichnis nicht beschreibbar: $TARGET"
|
||||
|
||||
# ============================================================
|
||||
# HARDLINK-REFERENZ
|
||||
# ============================================================
|
||||
|
||||
RSYNC_OPTIONS=(
|
||||
-a
|
||||
-v
|
||||
-R
|
||||
--delete
|
||||
--stats
|
||||
)
|
||||
|
||||
if [[ -L "$LAST" && ! -d "$LAST" ]]; then
|
||||
fail "Der last-Verweis ist ungueltig"
|
||||
fi
|
||||
|
||||
if [[ -d "$LAST" ]]; then
|
||||
RSYNC_OPTIONS+=("--link-dest=$LAST")
|
||||
log "Hardlink-Referenz: $(readlink -f "$LAST")"
|
||||
else
|
||||
log "Keine vorherige Hardlink-Referenz gefunden"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# TEMPORAEREN SNAPSHOT ANLEGEN
|
||||
# ============================================================
|
||||
|
||||
STAGING="$(mktemp -d "$TARGET/.inprogress-${TODAY}.XXXXXXXX")"
|
||||
|
||||
log "Temporaerer Snapshot: $STAGING"
|
||||
|
||||
# ============================================================
|
||||
# RSYNC AUSFUEHREN
|
||||
# ============================================================
|
||||
|
||||
log "rsync gestartet"
|
||||
|
||||
if rsync "${RSYNC_OPTIONS[@]}" \
|
||||
"$SOURCE" "$STAGING"; then
|
||||
|
||||
log "rsync erfolgreich abgeschlossen"
|
||||
|
||||
else
|
||||
RC=$?
|
||||
fail "rsync fehlgeschlagen (Exit-Code $RC)"
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# SICHERUNGSERGEBNIS PRUEFEN
|
||||
# ============================================================
|
||||
|
||||
EXPECTED="$STAGING/media/lars/paperless"
|
||||
|
||||
[[ -d "$EXPECTED" ]] ||
|
||||
fail "Erwartete Sicherungsstruktur fehlt: $EXPECTED"
|
||||
|
||||
# ============================================================
|
||||
# SNAPSHOT AKTIVIEREN
|
||||
# ============================================================
|
||||
|
||||
# Einen vorhandenen Tages-Snapshot erst verschieben,
|
||||
# nachdem die neue Sicherung vollstaendig ist.
|
||||
|
||||
if [[ -e "$DEST" ]]; then
|
||||
|
||||
[[ -d "$DEST" && ! -L "$DEST" ]] ||
|
||||
fail "Tagesziel ist kein normales Verzeichnis: $DEST"
|
||||
|
||||
OLD_DEST="$(mktemp -d "$TARGET/.replaced-${TODAY}.XXXXXXXX")"
|
||||
rmdir -- "$OLD_DEST"
|
||||
|
||||
mv -T -- "$DEST" "$OLD_DEST" ||
|
||||
fail "Alter Tages-Snapshot konnte nicht verschoben werden"
|
||||
fi
|
||||
|
||||
# Neue Sicherung an die Stelle des Tages-Snapshots setzen.
|
||||
|
||||
if ! mv -T -- "$STAGING" "$DEST"; then
|
||||
|
||||
# Bei einem Fehler den alten Tagesstand zurueckholen.
|
||||
|
||||
if [[ -n "$OLD_DEST" && -d "$OLD_DEST" ]]; then
|
||||
mv -T -- "$OLD_DEST" "$DEST" || true
|
||||
fi
|
||||
|
||||
fail "Neuer Tages-Snapshot konnte nicht aktiviert werden"
|
||||
fi
|
||||
|
||||
STAGING=""
|
||||
|
||||
# ============================================================
|
||||
# LAST-VERWEIS AKTUALISIEREN
|
||||
# ============================================================
|
||||
|
||||
TEMP_LINK="$TARGET/.last-new-$$"
|
||||
|
||||
ln -s -- "$DEST" "$TEMP_LINK" ||
|
||||
fail "Neuer last-Verweis konnte nicht erstellt werden"
|
||||
|
||||
mv -Tf -- "$TEMP_LINK" "$LAST" ||
|
||||
fail "last-Verweis konnte nicht aktualisiert werden"
|
||||
|
||||
TEMP_LINK=""
|
||||
|
||||
log "last zeigt jetzt auf: $DEST"
|
||||
|
||||
# ============================================================
|
||||
# ALTEN TAGES-SNAPSHOT ENTFERNEN
|
||||
# ============================================================
|
||||
|
||||
# Nur die durch diesen Lauf ersetzte Tagesversion loeschen.
|
||||
# Andere Sicherungsstaende bleiben unveraendert.
|
||||
|
||||
if [[ -n "$OLD_DEST" && -d "$OLD_DEST" ]]; then
|
||||
|
||||
if rm -rf -- "$OLD_DEST"; then
|
||||
log "Ersetzter Tages-Snapshot entfernt"
|
||||
else
|
||||
log "WARNUNG: Alter Tages-Snapshot konnte nicht vollstaendig entfernt werden"
|
||||
fi
|
||||
|
||||
fi
|
||||
|
||||
|
||||
log "NAS-Datenbackup erfolgreich abgeschlossen"
|
||||
|
||||
|
||||
log "========================================"
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
# Verwendung:
|
||||
# db-mariadb-docker.sh CONTAINER DATEINAME mariadb|mysql
|
||||
|
||||
if [[ $# -ne 3 ]]; then
|
||||
echo "FEHLER: Drei Parameter erforderlich" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
CONTAINER="$1"
|
||||
FILENAME="$2"
|
||||
PROFILE="$3"
|
||||
|
||||
BACKUP_DIR="/media/lars/paperless/db-backup"
|
||||
|
||||
[[ "$FILENAME" =~ ^[a-zA-Z0-9_-]+\.sql$ ]] || exit 2
|
||||
|
||||
[[ -d "$BACKUP_DIR" && -w "$BACKUP_DIR" ]] || {
|
||||
echo "FEHLER: Backup-Verzeichnis nicht verfuegbar" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
TARGET="$BACKUP_DIR/$FILENAME"
|
||||
TEMP="$(mktemp "$BACKUP_DIR/.${FILENAME}.XXXXXXXX")"
|
||||
|
||||
trap 'rm -f -- "$TEMP"' EXIT
|
||||
|
||||
echo "Starte MariaDB-Export: $CONTAINER"
|
||||
|
||||
case "$PROFILE" in
|
||||
mariadb)
|
||||
docker exec "$CONTAINER" sh -ec '
|
||||
: "${MARIADB_USER:?Benutzer fehlt}"
|
||||
: "${MARIADB_PASSWORD:?Passwort fehlt}"
|
||||
: "${MARIADB_DATABASE:?Datenbank fehlt}"
|
||||
|
||||
export MYSQL_PWD="$MARIADB_PASSWORD"
|
||||
|
||||
exec mariadb-dump \
|
||||
--host=127.0.0.1 \
|
||||
--user="$MARIADB_USER" \
|
||||
--single-transaction \
|
||||
--quick \
|
||||
--routines \
|
||||
--events \
|
||||
--triggers \
|
||||
--databases "$MARIADB_DATABASE"
|
||||
' > "$TEMP"
|
||||
;;
|
||||
|
||||
mysql)
|
||||
docker exec "$CONTAINER" sh -ec '
|
||||
: "${MYSQL_USER:?Benutzer fehlt}"
|
||||
: "${MYSQL_PASSWORD:?Passwort fehlt}"
|
||||
: "${MYSQL_DATABASE:?Datenbank fehlt}"
|
||||
|
||||
export MYSQL_PWD="$MYSQL_PASSWORD"
|
||||
|
||||
exec mariadb-dump \
|
||||
--host=127.0.0.1 \
|
||||
--user="$MYSQL_USER" \
|
||||
--single-transaction \
|
||||
--quick \
|
||||
--routines \
|
||||
--events \
|
||||
--triggers \
|
||||
--databases "$MYSQL_DATABASE"
|
||||
' > "$TEMP"
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "FEHLER: Unbekanntes Profil" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
[[ -s "$TEMP" ]] || {
|
||||
echo "FEHLER: Dump ist leer" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
chgrp lars "$TEMP"
|
||||
chmod 640 "$TEMP"
|
||||
|
||||
mv -f -- "$TEMP" "$TARGET"
|
||||
|
||||
echo "OK: $TARGET"
|
||||
stat -c 'Groesse: %s Bytes' "$TARGET"
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
umask 077
|
||||
|
||||
# Verwendung:
|
||||
# db-postgres-docker.sh CONTAINER DATEINAME
|
||||
# db-postgres-docker.sh CONTAINER DATEINAME DB_USER DB_NAME
|
||||
|
||||
BACKUP_DIR="/media/lars/paperless/db-backup"
|
||||
|
||||
if [[ $# -ne 2 && $# -ne 4 ]]; then
|
||||
echo "FEHLER: Ungueltige Parameteranzahl" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
CONTAINER="$1"
|
||||
FILENAME="$2"
|
||||
|
||||
if [[ ! "$FILENAME" =~ ^[a-zA-Z0-9_-]+\.sql$ ]]; then
|
||||
echo "FEHLER: Ungueltiger Dateiname" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ ! -d "$BACKUP_DIR" || ! -w "$BACKUP_DIR" ]]; then
|
||||
echo "FEHLER: Backup-Verzeichnis nicht verfuegbar" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TARGET="$BACKUP_DIR/$FILENAME"
|
||||
TEMP_FILE="$(mktemp "$BACKUP_DIR/.${FILENAME}.XXXXXXXX")"
|
||||
|
||||
trap 'rm -f -- "$TEMP_FILE"' EXIT
|
||||
|
||||
echo "Starte PostgreSQL-Backup: $CONTAINER"
|
||||
|
||||
# Standard: Datenbankparameter aus Container-Umgebung.
|
||||
# Ausnahme: explizite Parameter, z.B. Mitai Dev.
|
||||
|
||||
if [[ $# -eq 2 ]]; then
|
||||
CMD=(
|
||||
docker exec "$CONTAINER"
|
||||
sh -ec '
|
||||
: "${POSTGRES_USER:?POSTGRES_USER fehlt}"
|
||||
: "${POSTGRES_DB:?POSTGRES_DB fehlt}"
|
||||
export PGPASSWORD="${POSTGRES_PASSWORD:-}"
|
||||
exec pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"
|
||||
'
|
||||
)
|
||||
else
|
||||
CMD=(
|
||||
docker exec "$CONTAINER"
|
||||
pg_dump -U "$3" "$4"
|
||||
)
|
||||
fi
|
||||
|
||||
if "${CMD[@]}" > "$TEMP_FILE"; then
|
||||
echo "Datenbankexport abgeschlossen"
|
||||
else
|
||||
RC=$?
|
||||
echo "FEHLER: pg_dump fehlgeschlagen (Code $RC)" >&2
|
||||
exit "$RC"
|
||||
fi
|
||||
|
||||
if [[ ! -s "$TEMP_FILE" ]]; then
|
||||
echo "FEHLER: Datenbank-Dump ist leer" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Auch der Benutzer lars muss die Dateien spaeter
|
||||
# mit dem NAS-Kopierskript lesen koennen.
|
||||
chgrp lars "$TEMP_FILE"
|
||||
chmod 640 "$TEMP_FILE"
|
||||
|
||||
# Atomarer Austausch auf demselben Dateisystem.
|
||||
mv -f -- "$TEMP_FILE" "$TARGET"
|
||||
|
||||
echo "OK: $TARGET"
|
||||
stat -c 'Groesse: %s Bytes' "$TARGET"
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
UNIT=raspiBackup.service
|
||||
|
||||
if systemctl is-active --quiet "$UNIT"; then
|
||||
echo "FEHLER: raspiBackup läuft bereits" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RC=0
|
||||
systemctl start --wait "$UNIT" || RC=$?
|
||||
|
||||
RESULT=$(systemctl show --value -p Result "$UNIT")
|
||||
STATUS=$(systemctl show --value -p ExecMainStatus "$UNIT")
|
||||
|
||||
if (( RC != 0 )) || [[ "$RESULT" != "success" || "$STATUS" != "0" ]]; then
|
||||
echo "FEHLER: raspiBackup: Result=$RESULT, Status=$STATUS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: raspiBackup erfolgreich abgeschlossen"
|
||||
@@ -0,0 +1,178 @@
|
||||
"""Tests use temporary dummy scripts only. They never touch Raspberry Pi backups."""
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
MODULE = Path(__file__).resolve().parents[1] / 'orchestrator.py'
|
||||
spec = importlib.util.spec_from_file_location('orchestrator_test', MODULE)
|
||||
app = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(app)
|
||||
|
||||
|
||||
class OrchestratorTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.patches = [patch.object(app, 'legacy_schedules', return_value=[]),
|
||||
patch.object(app, 'nas_mount_ready', return_value=True),
|
||||
patch.dict(os.environ, {'BACKUP_ORCHESTRATOR_CUTOVER_APPROVED':'YES'})]
|
||||
for p in self.patches:
|
||||
p.start()
|
||||
self.addCleanup(p.stop)
|
||||
self.args = types.SimpleNamespace(max_parallel=3, log_root=self.root/'logs', lock_file=self.root/'runner.lock', notify_config=self.root/'notifications.json')
|
||||
|
||||
def script(self, name, body):
|
||||
p = self.root / f'{name}.sh'
|
||||
p.write_text('#!/bin/sh\n' + body + '\n')
|
||||
p.chmod(0o700)
|
||||
return str(p)
|
||||
|
||||
def job(self, phase, name, body, critical=True, timeout=10):
|
||||
return {'phase': phase, 'id': name, 'script': self.script(name, body),
|
||||
'critical': critical, 'timeout_seconds': timeout, 'enabled': True}
|
||||
|
||||
def results(self):
|
||||
import json
|
||||
path = list((self.root / 'logs').glob('*/summary.json'))[0]
|
||||
return json.loads(path.read_text())
|
||||
|
||||
def test_parallel_phase_then_barrier(self):
|
||||
event = self.root / 'events.txt'
|
||||
j1 = self.job(10, 'a', f'sleep 0.3; echo a >> "{event}"')
|
||||
j2 = self.job(10, 'b', f'sleep 0.3; echo b >> "{event}"')
|
||||
j3 = self.job(20, 'after', f'count=$(wc -l < "{event}"); [ "$count" -eq 2 ] && echo after >> "{event}"')
|
||||
import time
|
||||
start = time.monotonic()
|
||||
self.assertEqual(app.live_run([j1, j2, j3], self.args), 0)
|
||||
self.assertLess(time.monotonic() - start, 0.60)
|
||||
self.assertEqual(event.read_text().splitlines()[-1], 'after')
|
||||
self.assertEqual(self.results()['overall'], 'ok')
|
||||
|
||||
def test_critical_failure_blocks_next_phase(self):
|
||||
event = self.root / 'events.txt'
|
||||
j1 = self.job(10, 'fail', 'exit 7', critical=True)
|
||||
j2 = self.job(10, 'optional', 'exit 0', critical=False)
|
||||
j3 = self.job(20, 'after', f'echo wrong > "{event}"')
|
||||
self.assertEqual(app.live_run([j1, j2, j3], self.args), 1)
|
||||
self.assertFalse(event.exists())
|
||||
by_name = {r['job']:r['state'] for r in self.results()['results']}
|
||||
self.assertEqual(by_name, {'fail':'failed', 'optional':'ok', 'after':'blocked'})
|
||||
|
||||
def test_noncritical_failure_allows_next_phase(self):
|
||||
event = self.root / 'events.txt'
|
||||
j1 = self.job(10, 'optional-fail', 'exit 8', critical=False)
|
||||
j2 = self.job(20, 'after', f'echo ran > "{event}"')
|
||||
self.assertEqual(app.live_run([j1, j2], self.args), 1)
|
||||
self.assertTrue(event.exists())
|
||||
self.assertEqual([r['state'] for r in self.results()['results']], ['failed', 'ok'])
|
||||
|
||||
def test_timeout(self):
|
||||
j = self.job(10, 'hang', 'sleep 8', timeout=1)
|
||||
self.assertEqual(app.live_run([j], self.args), 1)
|
||||
self.assertEqual(self.results()['results'][0]['state'], 'timeout')
|
||||
|
||||
def test_conflicts_block_runs(self):
|
||||
j = self.job(10, 'no-run', 'exit 0')
|
||||
with patch.object(app, 'legacy_schedules', return_value=['raspiBackup.timer: enabled']):
|
||||
self.assertEqual(app.live_run([j], self.args), 2)
|
||||
self.assertFalse((self.root/'logs').exists())
|
||||
|
||||
def test_config_validation(self):
|
||||
path = self.root/'jobs.csv'
|
||||
path.write_text('phase,id,script,critical,timeout_seconds,enabled\n10,one,/bin/true,true,30,false\n')
|
||||
self.assertEqual(len(app.load_jobs(path)), 1)
|
||||
path.write_text('phase,id,script,critical,timeout_seconds,enabled\n10,one,/bin/true,maybe,30,false\n')
|
||||
with self.assertRaises(ValueError):
|
||||
app.load_jobs(path)
|
||||
|
||||
|
||||
# New outbound interface tests are intentionally declared outside the legacy test class.
|
||||
class NotificationTests(OrchestratorTests):
|
||||
class FakeResponse:
|
||||
status = 200
|
||||
def __enter__(self):
|
||||
return self
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
def read(self, *_):
|
||||
return b'{"ok":true}'
|
||||
|
||||
def test_outbound_webhooks_kuma_and_status_json(self):
|
||||
import json
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
observed = []
|
||||
def fake_request(req, timeout):
|
||||
observed.append((req.get_method(), req.full_url, req.data))
|
||||
return self.FakeResponse()
|
||||
self.args.notify_config.write_text(json.dumps({
|
||||
'timeout_seconds': 2,
|
||||
'webhooks': ['http://127.0.0.1:99/events'],
|
||||
'kuma': {'overall_push_url': 'http://localhost:3010/api/push/SUPERSECRET?status=up&msg=OK&ping=',
|
||||
'job_push_urls': {'one': 'http://localhost:3010/api/push/ANOTHERSECRET'}}
|
||||
}))
|
||||
job = self.job(10, 'one', 'echo test')
|
||||
with patch.object(app, 'urlopen', side_effect=fake_request):
|
||||
self.assertEqual(app.live_run([job], self.args), 0)
|
||||
methods = [a[0] for a in observed]
|
||||
self.assertIn('POST', methods)
|
||||
sent_events = [json.loads(a[2])['event'] for a in observed if a[0] == 'POST']
|
||||
self.assertIn('JOB_STARTED', sent_events)
|
||||
self.assertIn('JOB_SUCCESS', sent_events)
|
||||
self.assertIn('RUN_FINISHED', sent_events)
|
||||
gets = [a for a in observed if a[0] == 'GET']
|
||||
self.assertEqual(len(gets), 2)
|
||||
for _, url, _ in gets:
|
||||
self.assertEqual(parse_qs(urlsplit(url).query)['status'], ['up'])
|
||||
latest = json.loads((self.root/'logs'/'latest.json').read_text())
|
||||
self.assertEqual(latest['overall'], 'ok')
|
||||
self.assertEqual(latest['jobs']['one']['state'], 'ok')
|
||||
self.assertNotIn('SUPERSECRET', json.dumps(latest))
|
||||
events = list((self.root/'logs').glob('*/events.jsonl'))[0].read_text()
|
||||
self.assertNotIn('ANOTHERSECRET', events)
|
||||
self.assertIn('RUN_FINISHED', events)
|
||||
|
||||
def test_job_failure_sends_kuma_down_and_later_phase_blocked(self):
|
||||
import json
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
observed = []
|
||||
def fake_request(req, timeout):
|
||||
observed.append(req.full_url)
|
||||
return self.FakeResponse()
|
||||
self.args.notify_config.write_text(json.dumps({
|
||||
'kuma': {'overall_push_url': 'http://localhost/api/push/overall',
|
||||
'job_push_urls': {'fail': 'http://localhost/api/push/fail',
|
||||
'after': 'http://localhost/api/push/after'}}
|
||||
}))
|
||||
jobs = [self.job(10, 'fail', 'exit 9'), self.job(20, 'after', 'exit 0')]
|
||||
with patch.object(app, 'urlopen', side_effect=fake_request):
|
||||
self.assertEqual(app.live_run(jobs, self.args), 1)
|
||||
self.assertEqual(len(observed), 3)
|
||||
for url in observed:
|
||||
self.assertEqual(parse_qs(urlsplit(url).query)['status'], ['down'])
|
||||
latest = json.loads((self.root/'logs'/'latest.json').read_text())
|
||||
self.assertEqual(latest['jobs']['fail']['state'], 'failed')
|
||||
self.assertEqual(latest['jobs']['after']['state'], 'blocked')
|
||||
self.assertEqual(latest['overall'], 'failed')
|
||||
|
||||
def test_notification_failure_keeps_backup_result_and_sanitizes_logs(self):
|
||||
job = self.job(10, 'one', 'exit 0')
|
||||
self.args.notify_config.write_text('{"kuma":{"overall_push_url":"http://localhost/api/push/SECRET"}}')
|
||||
with patch.object(app, 'urlopen', side_effect=OSError('http://localhost/api/push/SECRET')):
|
||||
self.assertEqual(app.live_run([job], self.args), 0)
|
||||
log = list((self.root/'logs').glob('*/notifications.log'))[0].read_text()
|
||||
self.assertIn('delivery failed', log)
|
||||
self.assertNotIn('SECRET', log)
|
||||
|
||||
def test_validation_rejects_wrong_kuma_url(self):
|
||||
import json
|
||||
self.args.notify_config.write_text(json.dumps({'kuma': {'overall_push_url': 'file:///secrets'}}))
|
||||
with self.assertRaises(ValueError):
|
||||
app.load_notifications(self.args.notify_config)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user