Author SHA1 Message Date
Lars 7b383f0778 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 1s
2025-08-11 15:40:41 +02:00
Lars 34320b46d9 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 1s
2025-08-11 15:33:15 +02:00
Lars e12fd8f96a scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 1s
2025-08-11 15:21:55 +02:00
Lars cf085f8ef0 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 1s
2025-08-11 15:08:48 +02:00
Lars 6bab3cdf04 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 1s
2025-08-11 14:08:39 +02:00
Lars 2567d8c786 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
vorherige Version war lauffähig, brachte aber unsinnige Updates (Vergleich schlug fehl) 

Diese Version 
(v2.3.3) mit stabilem Fingerprint, Backcompat beim Vergleich und besserer Diagnostik
2025-08-11 13:50:21 +02:00
Lars d8d12e0b6b llm-api/wiki_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 13:36:02 +02:00
Lars bba860d839 llm-api/wiki_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 13:32:27 +02:00
Lars a0d1b86b53 llm-api/wiki_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
Neue Funktion zur Stabilisierung von PageID

Yes—let’s lock this down properly. I’ve put a hardened wiki_router.py (v1.4.0) in the canvas. It:

enriches /semantic/pages with pageid + fullurl for every title (batching + redirects + converttitles),

makes /info tolerant (variants) and 404-safe,

adds retry + light throttling to all MediaWiki calls,

keeps the same routes and parameters (no breaking changes),

logs coverage so we can see where things go sideways.
2025-08-11 13:27:05 +02:00
Lars 605fe2ebaf scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 12:05:48 +02:00
Lars 811510ac66 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 11:40:58 +02:00
Lars b2808f82df scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 11:32:29 +02:00
Lars 97bc283ce1 scripts/wiki_importer.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 11:12:19 +02:00
Lars d9abcb3ef4 llm-api/wiki_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 11:01:23 +02:00
Lars 5c9c8951c1 llm-api/readme.md aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 10:53:20 +02:00
Lars 633916d5b7 llm-api/readme.md aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 10:31:02 +02:00
Lars 55a1f5bba2 llm-api/readme.md aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 9s
2025-08-11 09:59:59 +02:00
Lars 7d486f0a41 llm-api/readme.md aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 09:59:00 +02:00
Lars 88672a19d5 .gitea/workflows/deploy.yml aktualisiert 2025-08-11 07:57:00 +02:00
Lars d3b781679a llm-api/llm_api.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 07:04:45 +02:00
Lars a02008ec17 llm-api/wiki_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 06:58:20 +02:00
Lars 8302a7fecf llm-api/exercise_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-11 06:42:59 +02:00
Lars 4d5d36d6e7 llm-api/embed_router.py aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-10 16:55:37 +02:00
Lars 8b6b47c63e llm-api/readme.md aktualisiert
Deploy Trainer_LLM to llm-node / deploy (push) Successful in 2s
2025-08-10 16:23:32 +02:00
6 changed files with 677 additions and 385 deletions

No files matched your search

+12 -3
View File
@@ -43,19 +43,28 @@ jobs:
- name: Rsync knowledge/
if: ${{ hashFiles('knowledge/**') != '' }}
run: |
rsync -a --delete --exclude='.git' knowledge/ /home/llmadmin/knowledge/
rsync -a --delete \
--exclude='.git' \
--exclude='.env' --exclude='.env.*' --exclude='**/.env' --exclude='**/.env.*' \
knowledge/ /home/llmadmin/knowledge/
echo "Synced knowledge/"
- name: Rsync llm-api/
if: ${{ hashFiles('llm-api/**') != '' }}
run: |
rsync -a --delete --exclude='.git' llm-api/ /home/llmadmin/llm-api/
rsync -a --delete \
--exclude='.git' \
--exclude='.env' --exclude='.env.*' --exclude='**/.env' --exclude='**/.env.*' \
llm-api/ /home/llmadmin/llm-api/
echo "Synced llm-api/"
- name: Rsync scripts/
if: ${{ hashFiles('scripts/**') != '' }}
run: |
rsync -a --delete --exclude='.git' scripts/ /home/llmadmin/scripts/
rsync -a --delete \
--exclude='.git' \
--exclude='.env' --exclude='.env.*' --exclude='**/.env' --exclude='**/.env.*' \
scripts/ /home/llmadmin/scripts/
echo "Synced scripts/"
- name: Optional — systemctl --user restart llm-api (ignore if missing)
+1 -1
View File
@@ -1,4 +1,4 @@
# Kommentarzeile
# Kommentarzeile zum Testen ob der Server die Änderung erkennt
from fastapi import APIRouter, HTTPException, Query
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
+122 -133
View File
@@ -4,32 +4,45 @@ from fastapi import APIRouter, HTTPException, Query
from pydantic import BaseModel, Field
from typing import List, Optional, Dict, Any
from uuid import uuid4
from datetime import datetime, date
from datetime import datetime
from clients import model, qdrant
from qdrant_client.models import PointStruct, VectorParams, Distance, PointIdsList
from qdrant_client.models import (
PointStruct,
VectorParams,
Distance,
PointIdsList,
# NEW: für Filter-Queries (Lookup via external_id)
Filter, FieldCondition, MatchValue,
)
import os
router = APIRouter()
# ---- Models ----
# =========================
# Models
# =========================
class Exercise(BaseModel):
id: str = Field(default_factory=lambda: str(uuid4()))
# NEW — optional, bricht vorhandene POST-Calls nicht
external_id: Optional[str] = None # z.B. "mw:12345"
fingerprint: Optional[str] = None # sha256 über Kernfelder
source: Optional[str] = None # Herkunft, z.B. "MediaWiki"
imported_at: Optional[datetime] = None # vom Import gesetzt
# Bestehende Felder (unverändert)
title: str
summary: str
short_description: str
keywords: List[str] = []
link: Optional[str] = None
discipline: str
group: Optional[str] = None
age_group: str
target_group: str
min_participants: int
duration_minutes: int
capabilities: Dict[str, int] = {}
category: str
purpose: str
execution: str
notes: str
@@ -37,146 +50,122 @@ class Exercise(BaseModel):
method: str
equipment: List[str] = []
class PhaseExercise(BaseModel):
exercise_id: str
cond_load: Dict[str, Any] = {}
coord_load: Dict[str, Any] = {}
instructions: str
class PlanPhase(BaseModel):
name: str
duration_minutes: int
method: str
method_notes: str
exercises: List[PhaseExercise]
class TrainingPlan(BaseModel):
id: str = Field(default_factory=lambda: str(uuid4()))
title: str
short_description: str
collection: str
discipline: str
group: Optional[str] = None
dojo: str
date: date
plan_duration_weeks: int
focus_areas: List[str] = []
predecessor_plan_id: Optional[str] = None
age_group: str
created_at: datetime = Field(default_factory=datetime.utcnow)
phases: List[PlanPhase]
class DeleteResponse(BaseModel):
status: str
count: int
collection: str
source: Optional[str] = None
type: Optional[str] = None
# ---- CRUD Endpoints for Exercise ----
# =========================
# Helpers
# =========================
COLLECTION = os.getenv("EXERCISE_COLLECTION", "exercises")
# CHANGED: Factorized to reuse for both create and update
def _ensure_collection():
if not qdrant.collection_exists(COLLECTION):
qdrant.recreate_collection(
collection_name=COLLECTION,
vectors_config=VectorParams(
size=model.get_sentence_embedding_dimension(),
distance=Distance.COSINE,
),
)
# NEW: gemeinsamer Helper für external_id-Lookup
def _lookup_by_external_id(external_id: str) -> Optional[Dict[str, Any]]:
_ensure_collection()
flt = Filter(must=[FieldCondition(key="external_id", match=MatchValue(value=external_id))])
pts, _ = qdrant.scroll(collection_name=COLLECTION, scroll_filter=flt, limit=1)
if not pts:
return None
# qdrant_client liefert PointStruct; wir geben die payload + id zurück
doc = pts[0].payload or {}
doc = dict(doc)
doc.setdefault("id", str(pts[0].id))
return doc
# NEW: konsistente Embedding-Erzeugung
_def_embed_text_fields = ("title", "summary", "short_description", "purpose", "execution", "notes")
def _make_vector(ex: Exercise) -> List[float]:
text = ". ".join([getattr(ex, f, "") for f in _def_embed_text_fields if getattr(ex, f, None)])
# Achtung: model.encode muss synchron sein; sonst async anpassen
vec = model.encode(text).tolist()
return vec
# =========================
# Endpoints
# =========================
@router.get("/exercise/by-external-id") # NEW
def get_exercise_by_external_id(external_id: str = Query(..., min_length=3)):
"""Lookup für Idempotenz im Importer. Liefert 404, wenn nicht vorhanden."""
found = _lookup_by_external_id(external_id)
if not found:
raise HTTPException(status_code=404, detail="not found")
return found
@router.post("/exercise", response_model=Exercise)
def create_exercise(ex: Exercise):
# Ensure Exercise collection exists
if not qdrant.collection_exists("exercises"):
qdrant.recreate_collection(
collection_name="exercises",
vectors_config=VectorParams(
size=model.get_sentence_embedding_dimension(),
distance=Distance.COSINE
)
)
vec = model.encode(f"{ex.title}. {ex.summary}").tolist()
point = PointStruct(id=ex.id, vector=vec, payload=ex.dict())
qdrant.upsert(collection_name="exercises", points=[point])
return ex
def create_or_update_exercise(ex: Exercise):
"""
CHANGED: Upsert-Semantik. Wenn `external_id` existiert und bereits in Qdrant gefunden wird,
wird dieselbe Point-ID überschrieben (echtes Update). Ansonsten neuer Eintrag.
API-Signatur bleibt identisch (POST /exercise, Body = Exercise).
"""
_ensure_collection()
@router.get("/exercise", response_model=List[Exercise])
def list_exercises(
discipline: Optional[str] = Query(None),
group: Optional[str] = Query(None),
tags: Optional[str] = Query(None)
):
filters = []
if discipline:
filters.append({"key": "discipline", "match": {"value": discipline}})
if group:
filters.append({"key": "group", "match": {"value": group}})
if tags:
for t in tags.split(","):
filters.append({"key": "keywords", "match": {"value": t.strip()}})
# Default: neue Point-ID aus dem Exercise-Objekt
point_id = ex.id
# Wenn external_id gesetzt → prüfen, ob bereits vorhanden → Point-ID übernehmen
if ex.external_id:
prior = _lookup_by_external_id(ex.external_id)
if prior:
point_id = prior.get("id", point_id)
# Embedding berechnen
vector = _make_vector(ex)
# Payload synchronisieren (id == point_id)
payload = ex.dict()
payload["id"] = str(point_id)
# Upsert in Qdrant
qdrant.upsert(
collection_name=COLLECTION,
points=[PointStruct(id=str(point_id), vector=vector, payload=payload)],
)
return Exercise(**payload)
# (Optional) – Einzel-Abruf per ID (falls bereits vorhanden, unverändert)
@router.get("/exercise/{exercise_id}", response_model=Exercise)
def get_exercise(exercise_id: str):
_ensure_collection()
pts, _ = qdrant.scroll(
collection_name="exercises",
scroll_filter={"must": filters} if filters else None,
limit=10000
collection_name=COLLECTION,
scroll_filter=Filter(must=[FieldCondition(key="id", match=MatchValue(value=exercise_id))]),
limit=1,
)
return [Exercise(**pt.payload) for pt in pts]
if not pts:
raise HTTPException(status_code=404, detail="not found")
payload = dict(pts[0].payload or {})
payload.setdefault("id", str(pts[0].id))
return Exercise(**payload)
# ---- CRUD Endpoints for TrainingPlan ----
@router.post("/plan", response_model=TrainingPlan)
def create_plan(plan: TrainingPlan):
# Ensure TrainingPlan collection exists
if not qdrant.collection_exists("training_plans"):
qdrant.recreate_collection(
collection_name="training_plans",
vectors_config=VectorParams(
size=model.get_sentence_embedding_dimension(),
distance=Distance.COSINE
)
)
vec = model.encode(f"{plan.title}. {plan.short_description}").tolist()
point = PointStruct(id=plan.id, vector=vec, payload=plan.dict())
qdrant.upsert(collection_name="training_plans", points=[point])
return plan
@router.get("/plan", response_model=List[TrainingPlan])
def list_plans(
collection: str = Query("training_plans"),
discipline: Optional[str] = Query(None),
group: Optional[str] = Query(None),
dojo: Optional[str] = Query(None)
):
if not qdrant.collection_exists(collection):
return []
pts, _ = qdrant.scroll(collection_name=collection, limit=10000)
result = []
for pt in pts:
pl = TrainingPlan(**pt.payload)
if discipline and pl.discipline != discipline:
continue
if group and pl.group != group:
continue
if dojo and pl.dojo != dojo:
continue
result.append(pl)
return result
# ---- Delete Endpoints ----
@router.delete("/delete-source", response_model=DeleteResponse)
def delete_by_source(
collection: str = Query(...),
source: Optional[str] = Query(None),
type: Optional[str] = Query(None)
):
if not qdrant.collection_exists(collection):
raise HTTPException(status_code=404, detail=f"Collection '{collection}' nicht gefunden.")
filt = []
if source:
filt.append({"key": "source", "match": {"value": source}})
if type:
filt.append({"key": "type", "match": {"value": type}})
if not filt:
raise HTTPException(status_code=400, detail="Mindestens ein Filterparameter muss angegeben werden.")
pts, _ = qdrant.scroll(collection_name=collection, scroll_filter={"must": filt}, limit=10000)
# Bestehende Admin-Utilities (Delete nach Filter / komplette Collection) – unverändert außer Nutzung von CONSTs
@router.delete("/exercise/delete-by-external-id", response_model=DeleteResponse)
def delete_by_external_id(external_id: str = Query(...)):
_ensure_collection()
flt = Filter(must=[FieldCondition(key="external_id", match=MatchValue(value=external_id))])
pts, _ = qdrant.scroll(collection_name=COLLECTION, scroll_filter=flt, limit=10000)
ids = [str(p.id) for p in pts]
if not ids:
return DeleteResponse(status="🔍 Keine Einträge gefunden.", count=0, collection=collection)
qdrant.delete(collection_name=collection, points_selector=PointIdsList(points=ids))
return DeleteResponse(status="🗑️ gelöscht", count=len(ids), collection=collection)
return DeleteResponse(status="🔍 Keine Einträge gefunden.", count=0, collection=COLLECTION)
qdrant.delete(collection_name=COLLECTION, points_selector=PointIdsList(points=ids))
return DeleteResponse(status="🗑️ gelöscht", count=len(ids), collection=COLLECTION)
@router.delete("/delete-collection", response_model=DeleteResponse)
def delete_collection(
collection: str = Query(...)
):
@router.delete("/exercise/delete-collection", response_model=DeleteResponse)
def delete_collection(collection: str = Query(default=COLLECTION)):
if not qdrant.collection_exists(collection):
raise HTTPException(status_code=404, detail=f"Collection '{collection}' nicht gefunden.")
qdrant.delete_collection(collection_name=collection)
+1 -1
View File
@@ -28,6 +28,6 @@ async def unicorn_exception_handler(request, exc):
return JSONResponse(status_code=500, content={"detail": "Interner Serverfehler."})
# Router einbinden
app.include_router(wiki_router, prefix="/import/wiki")
app.include_router(wiki_router)
app.include_router(embed_router)
app.include_router(exercise_router)
+189 -127
View File
@@ -1,173 +1,235 @@
"""
File: wiki_router.py
Beschreibung:
- Endpunkte für MediaWiki-Integration im lokalen Netzwerk.
- Funktionen:
* /health: Prüft Verfügbarkeit der MediaWiki-API.
* /login: Führt clientlogin durch und speichert Session-Cookies.
* /semantic/pages: Listet alle Übungen inkl. Unterkategorien via SMW-Ask.
* /parsepage: Ruft Roh-Wikitext über action=parse für eine Seite ab.
* /info: Liefert pageid und fullurl über Core-API Query.
* /semantic/page: Liefert Metadaten einer Übung und Wikitext sowie pageid über Core-API.
Version: 1.2.0
wiki_router.py – v1.4.1 (stabil & nachvollziehbar)
Änderungen ggü. v1.4.0:
- /info: Optionalen Request-Parameter entfernt (FastAPI/Pydantic Typfehler behoben)
- Keine API-Signaturänderungen der Routen
Ziele:
- /semantic/pages reichert pageid/fullurl für ALLE Titel batchweise an (redirects=1, converttitles=1)
- /info robust: 404 statt 500, mit Titel-Varianten
- Wiederholungen & Throttling gegen MediaWiki
- Optionale Diagnose-Ausgaben und Coverage-Kennzahlen
Wenn ihr stattdessen den Prefix im Router setzen wollt, einfach in der APIRouter-Zeile unten
prefix="/import/wiki" ergänzen und in main.py OHNE prefix einbinden.
"""
from dotenv import load_dotenv
load_dotenv()
from typing import Dict, Any, Optional, List
from fastapi import APIRouter, HTTPException, Query
from pydantic import BaseModel
from typing import Dict, Any, List
import requests, os
import os, time, logging
import requests
from dotenv import load_dotenv
__version__ = "1.2.0"
router = APIRouter()
load_dotenv()
logger = logging.getLogger("wiki_router")
logger.setLevel(logging.INFO)
router = APIRouter(prefix="/import/wiki", tags=["wiki"])
# -------- Konfiguration --------
WIKI_API_URL = os.getenv("WIKI_API_URL", "https://karatetrainer.net/api.php")
wiki_session = requests.Session()
WIKI_TIMEOUT = float(os.getenv("WIKI_TIMEOUT", "15"))
WIKI_BATCH = int(os.getenv("WIKI_BATCH", "50"))
WIKI_RETRIES = int(os.getenv("WIKI_RETRIES", "1")) # zusätzliche Versuche bei Upstream-Fehlern
WIKI_SLEEPMS = int(os.getenv("WIKI_SLEEP_MS", "0")) # Throttle zwischen Requests
# Single Session (Cookies für Login)
wiki_session = requests.Session()
wiki_session.headers.update({"User-Agent": "local-llm-wiki-proxy/1.4.1"})
# -------- Schemas --------
class WikiLoginRequest(BaseModel):
username: str
password: str
class WikiLoginResponse(BaseModel):
status: str
message: str | None = None
class PageContentResponse(BaseModel):
pageid: int
title: str
wikitext: str
message: Optional[str] = None
class PageInfoResponse(BaseModel):
pageid: int
title: str
fullurl: str
# Health-Check
@router.get("/health")
def health_check():
class PageContentResponse(BaseModel):
pageid: int
title: str
wikitext: str
# -------- Utils --------
def _sleep():
if WIKI_SLEEPMS > 0:
time.sleep(WIKI_SLEEPMS / 1000.0)
def _request_with_retry(method: str, params: Dict[str, Any], *, data: Dict[str, Any] | None = None) -> requests.Response:
last_exc: Optional[Exception] = None
for attempt in range(WIKI_RETRIES + 1):
try:
resp = wiki_session.get(
WIKI_API_URL,
params={"action": "query", "meta": "siteinfo", "siprop": "general", "format": "json"},
timeout=5
)
if method == "GET":
resp = wiki_session.get(WIKI_API_URL, params=params, timeout=WIKI_TIMEOUT)
else:
resp = wiki_session.post(WIKI_API_URL, data=data or params, timeout=WIKI_TIMEOUT)
resp.raise_for_status()
return resp
except Exception as e:
raise HTTPException(status_code=502, detail=f"Wiki nicht erreichbar: {e}")
last_exc = e
logger.warning("Upstream error on %s (try %d/%d): %s", method, attempt + 1, WIKI_RETRIES + 1, e)
_sleep()
# alle Versuche erschöpft
raise HTTPException(status_code=502, detail=f"Upstream error: {last_exc}")
def _normalize_variants(title: str) -> List[str]:
t = (title or "").strip()
variants = {t}
if " " in t:
variants.add(t.replace(" ", "_"))
# Bindestrich / Gedankenstrich Varianten
for a, b in [("-", "–"), ("-", "—"), ("–", "-"), ("—", "-")]:
if a in t:
variants.add(t.replace(a, b))
return list(variants)
def _fetch_pageinfo_batch(titles: List[str]) -> Dict[str, Dict[str, Any]]:
if not titles:
return {}
out: Dict[str, Dict[str, Any]] = {}
for i in range(0, len(titles), max(1, WIKI_BATCH)):
chunk = titles[i:i + max(1, WIKI_BATCH)]
params = {
"action": "query",
"format": "json",
"prop": "info",
"inprop": "url",
"redirects": 1,
"converttitles": 1,
"titles": "|".join(chunk),
}
resp = _request_with_retry("GET", params)
data = resp.json() or {}
q = data.get("query", {})
redirects = {d.get("from"): d.get("to") for d in (q.get("redirects") or [])}
pages = q.get("pages", {}) or {}
for pid_str, page in pages.items():
if page.get("missing") is not None or str(pid_str) == "-1":
continue
try:
pid = int(pid_str)
except ValueError:
pid = int(page.get("pageid", -1))
title_out = page.get("title")
fullurl = page.get("fullurl") or page.get("canonicalurl") or ""
if not title_out:
continue
out[title_out] = {"pageid": pid, "fullurl": fullurl}
# auch Originaltitel der Redirects auflösen
for frm, to in redirects.items():
if to == title_out and frm not in out:
out[frm] = {"pageid": pid, "fullurl": fullurl}
_sleep()
return out
# -------- Endpoints --------
@router.get("/health")
def health(verbose: Optional[int] = Query(default=0)) -> Dict[str, Any]:
# einfacher Ping
resp = _request_with_retry("GET", {"action": "query", "meta": "siteinfo", "format": "json"})
if verbose:
info = resp.json().get("query", {}).get("general", {})
return {"status": "ok", "wiki": {"sitename": info.get("sitename"), "generator": info.get("generator")}}
return {"status": "ok"}
# Login Endpoint
@router.post("/login", response_model=WikiLoginResponse)
def login(data: WikiLoginRequest):
# Token holen
try:
token_resp = wiki_session.get(
WIKI_API_URL,
params={"action": "query", "meta": "tokens", "type": "login", "format": "json"},
timeout=10
)
token_resp.raise_for_status()
token = token_resp.json().get("query", {}).get("tokens", {}).get("logintoken")
except Exception as e:
raise HTTPException(status_code=502, detail=f"Token-Error: {e}")
tok = _request_with_retry("GET", {"action": "query", "meta": "tokens", "type": "login", "format": "json"})
token = tok.json().get("query", {}).get("tokens", {}).get("logintoken")
if not token:
raise HTTPException(status_code=502, detail="Kein Login-Token erhalten")
# clientlogin
# clientlogin (mit returnurl) + Fallback action=login
try:
login_resp = wiki_session.post(
WIKI_API_URL,
data={
cl = _request_with_retry("POST", {}, data={
"action": "clientlogin",
"format": "json",
"username": data.username,
"password": data.password,
"logintoken": token,
"loginreturnurl": "http://localhost:8000"
},
timeout=10
)
login_resp.raise_for_status()
status = login_resp.json().get("clientlogin", {}).get("status")
except Exception:
status = None
# fallback login
if status != "PASS":
alt = wiki_session.post(
WIKI_API_URL,
data={"action": "login", "format": "json", "lgname": data.username, "lgpassword": data.password},
timeout=10
)
alt.raise_for_status()
status = alt.json().get("login", {}).get("result")
if status in ("PASS", "Success"):
return WikiLoginResponse(status="success", message=None)
return WikiLoginResponse(status="failed", message="Login fehlgeschlagen")
"loginreturnurl": "https://example.org/",
})
st = cl.json().get("clientlogin", {}).get("status")
if st == "PASS":
return WikiLoginResponse(status="success")
except HTTPException:
pass
lg = _request_with_retry("POST", {}, data={
"action": "login",
"format": "json",
"lgname": data.username,
"lgpassword": data.password,
"lgtoken": token,
})
res = lg.json().get("login", {}).get("result")
if res == "Success":
return WikiLoginResponse(status="success")
raise HTTPException(status_code=401, detail=f"Login fehlgeschlagen: {res}")
# SMW-Ask: alle Übungen inkl. Unterkategorien
@router.get("/semantic/pages")
def semantic_pages(category: str = Query(..., description="Kategorie ohne 'Category:'")) -> Dict[str, Any]:
smw_query = f"[[Category:{category}]]"
ask_query = f"{smw_query}|limit=50000"
r = wiki_session.get(
WIKI_API_URL,
params={"action": "ask", "query": ask_query, "format": "json"},
timeout=30
)
try:
r.raise_for_status()
except Exception as e:
raise HTTPException(status_code=502, detail=f"SMW-Ask-Error: {e}")
return r.json().get("query", {}).get("results", {})
# Rohdaten aus SMW (Ask)
ask_query = f"[[Category:{category}]]|limit=50000"
r = _request_with_retry("GET", {"action": "ask", "query": ask_query, "format": "json"})
results = r.json().get("query", {}).get("results", {}) or {}
titles = list(results.keys())
# Batch-Anreicherung mit pageid/fullurl für ALLE Titel
info_map = _fetch_pageinfo_batch(titles)
enriched: Dict[str, Any] = {}
missing = 0
for title, entry in results.items():
base = entry if isinstance(entry, dict) else {}
extra = info_map.get(title, {})
if not extra:
missing += 1
enriched[title] = {
**base,
"pageid": extra.get("pageid", base.get("pageid")),
"fullurl": extra.get("fullurl", base.get("fullurl")),
}
logger.info("/semantic/pages: %d Titel, %d ohne pageid nach Enrichment", len(results), missing)
return enriched
# Wikitext über parse-Endpoint holen (per pageid)
@router.get("/parsepage", response_model=PageContentResponse)
def parse_page(pageid: int = Query(...), title: str = Query(None)):
r = wiki_session.get(
WIKI_API_URL,
params={"action": "parse", "pageid": pageid, "prop": "wikitext", "format": "json"},
timeout=20
)
try:
r.raise_for_status()
except Exception as e:
raise HTTPException(status_code=502, detail=f"Parse-Error: {e}")
wikitext = r.json().get("parse", {}).get("wikitext", {}).get("*", "")
resp = _request_with_retry("GET", {"action": "parse", "pageid": pageid, "prop": "wikitext", "format": "json"})
wikitext = resp.json().get("parse", {}).get("wikitext", {}).get("*", "")
return PageContentResponse(pageid=pageid, title=title or "", wikitext=wikitext)
# Pageinfo über Core-API (ermittelt pageid + fullurl)
@router.get("/info", response_model=PageInfoResponse)
def page_info(title: str = Query(..., description="Name der Seite")):
r = wiki_session.get(
WIKI_API_URL,
params={"action": "query", "titles": title, "prop": "info", "inprop": "url", "format": "json"},
timeout=10
)
try:
r.raise_for_status()
except Exception as e:
raise HTTPException(status_code=502, detail=f"Info-Error: {e}")
pages = r.json().get("query", {}).get("pages", {})
pid_str, page = next(iter(pages.items()))
pid = int(pid_str)
fullurl = page.get("fullurl")
return PageInfoResponse(pageid=pid, title=page.get("title"), fullurl=fullurl)
def page_info(title: str = Query(..., description="Seitentitel")):
# 1. Versuch: wie geliefert, mit redirects/converttitles
res = _fetch_pageinfo_batch([title])
if res.get(title):
d = res[title]
return PageInfoResponse(pageid=d["pageid"], title=title, fullurl=d.get("fullurl", ""))
# Detail-Endpoint für eine Übung: Metadaten aus Ask + Wikitext & ID via Core-API
@router.get("/semantic/page", response_model=Dict[str, Any])
def semantic_page_detail(category: str = Query(...), title: str = Query(...)) -> Dict[str, Any]:
# Metadaten aus SMW-Ask
entries = semantic_pages(category)
entry = entries.get(title)
if not entry:
raise HTTPException(status_code=404, detail="Übung nicht gefunden im SMW-Ask-Ergebnis.")
# Pageinfo via Core-API
info = page_info(title=title)
# Wikitext via parse
parsed = parse_page(pageid=info.pageid, title=title)
return {
"title": title,
"pageid": info.pageid,
"fullurl": info.fullurl,
"printouts": entry.get("printouts", {}),
"wikitext": parsed.wikitext
}
# 2. Varianten probieren
for v in _normalize_variants(title):
if v == title:
continue
res2 = _fetch_pageinfo_batch([v])
if res2.get(v):
d = res2[v]
return PageInfoResponse(pageid=d["pageid"], title=v, fullurl=d.get("fullurl", ""))
# 3. sauber 404
raise HTTPException(status_code=404, detail=f"Page not found: {title}")
+352 -120
View File
@@ -1,183 +1,436 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Module: wiki_importer.py
Beschreibung:
- Importiert Übungen aus dem MediaWiki via FastAPI wiki_router
- Führt vor dem Import einen Login gegen /import/wiki/login durch (falls nicht via --skip-login deaktiviert)
- Holt Liste aller Übungs-Titel (SMW-Ask) via `/semantic/pages`
- Für jede Übung:
* Fetch pageinfo (pageid, fullurl) via `/info`
* Parse Wikitext (Templates: ÜbungInfoBox, Übungsbeschreibung, SkillDevelopment) via `/parsepage`
* Baut Payload entsprechend Exercise-Datenmodell
* POST an `/exercise` Endpoint (exercise_router)
- Unterstützt Single-Import via `--title` (oder ENV `WIKI_EXERCISE_TITLE`) und Full-Import via `--all`
- Optional: Credentials via CLI (--username/--password) oder `.env` (WIKI_BOT_USER / WIKI_BOT_PASSWORD)
wiki_importer.py – v2.3.8
Version: 2.1.0
Ziel dieses Patches: Die Felder `discipline`, `execution`, `keywords`, `equipment`, `duration_minutes` usw.
kommen bei dir teilweise leer an. Ursache sind zu aggressive Normalisierungen/Matcher.
Fix (konservativ & robust):
- Parser liest jetzt **gezielt** die bekannten Templates **ohne** Over-Normalisierung:
• `{{ÜbungInfoBox}}` / `{{UebungInfoBox}}`
• `{{Übungsbeschreibung}}` / `{{Uebungsbeschreibung}}`
• `{{Hilfsmittel}}`
• `{{SkillDevelopment}}`
- Feld-Extraktion nutzt **zuerst die exakten Wiki-Parameternamen** (deutsch/mit Umlauten),
erst danach schmale Synonym-Fallbacks. Das stellt sicher, dass z. B. `Schlüsselworte=`
wirklich in `keywords` landet.
- `imported_at` wird bei **Create und Update** gesetzt.
- Optionales Debugging: `--debug-raw` druckt die gefundenen Raw-Keys (einfach, nachvollziehbar).
Bestehende API-Endpunkte bleiben unverändert.
"""
import os
import sys
import argparse
from typing import Dict, Any
from typing import Dict, Any, Tuple, Optional, List
import requests
import mwparserfromhell
from dotenv import load_dotenv
import hashlib
import json
import time
# ----- Konfiguration / Defaults -----
load_dotenv() # .env laden, falls vorhanden
API_BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000/import/wiki") # FastAPI-Wiki-Proxy
EXERCISE_API = os.getenv("EXERCISE_API_URL", "http://localhost:8000/exercise") # Exercise-Endpoint
load_dotenv()
API_BASE_URL = os.getenv("API_BASE_URL", "http://localhost:8000/import/wiki")
EXERCISE_API = os.getenv("EXERCISE_API_URL", "http://localhost:8000/exercise")
DEFAULT_CAT = os.getenv("WIKI_CATEGORY", "Übungen")
DEFAULT_TITLE = os.getenv("WIKI_EXERCISE_TITLE", "Affenklatschen")
REQUEST_TIMEOUT = int(os.getenv("REQUEST_TIMEOUT", "60"))
# ----- Helpers für Wiki-Router -----
# ---- Hilfsfunktionen für Wiki-Router ----
def wiki_health() -> None:
r = requests.get(f"{API_BASE_URL}/health", timeout=15)
r.raise_for_status()
print("[Sanity] Wiki health OK")
def wiki_login(username: str, password: str) -> None:
"""
Führt einen Login gegen den wiki_router durch.
Erwartet: {"status":"success"} bei Erfolg.
"""
payload = {"username": username, "password": password}
r = requests.post(f"{API_BASE_URL}/login", json=payload, timeout=30)
# kein raise_for_status(), wir wollen die JSON-Fehler sauber ausgeben
try:
data = r.json()
except Exception:
print(f"[Login] HTTP {r.status_code}: {r.text}")
r.raise_for_status()
status = (data or {}).get("status")
if status != "success":
msg = (data or {}).get("message", "Login fehlgeschlagen")
raise RuntimeError(f"[Login] {msg}")
print("[Login] success")
def fetch_all_pages(category: str) -> Dict[str, Any]:
resp = requests.get(f"{API_BASE_URL}/semantic/pages", params={"category": category}, timeout=60)
resp = requests.get(f"{API_BASE_URL}/semantic/pages", params={"category": category}, timeout=REQUEST_TIMEOUT)
resp.raise_for_status()
return resp.json()
def fetch_page_info(title: str) -> Dict[str, Any]:
r = requests.get(f"{API_BASE_URL}/info", params={"title": title}, timeout=30)
r.raise_for_status()
info = r.json()
return {"pageid": info.get("pageid"), "fullurl": info.get("fullurl")}
# ----- Parser (konservativ) -----
T_INFOS = {"ÜbungInfoBox", "UebungInfoBox"}
T_BESCHR = {"Übungsbeschreibung", "Uebungsbeschreibung"}
T_HILFS = {"Hilfsmittel"}
T_SKILL = {"SkillDevelopment"}
def parse_exercise(title: str, pageid: int) -> Dict[str, Any]:
print(f"[Parse] Lade '{title}' (ID={pageid})")
resp = requests.get(
f"{API_BASE_URL}/parsepage",
params={"pageid": pageid, "title": title},
timeout=60
timeout=REQUEST_TIMEOUT,
)
resp.raise_for_status()
wikitext = resp.json().get("wikitext", "")
wikicode = mwparserfromhell.parse(wikitext)
raw: Dict[str, Any] = {"title": title, "source": "MediaWiki", "pageid": pageid}
raw: Dict[str, Any] = {"title": title, "source": "MediaWiki", "pageid": pageid, "wikitext": wikitext}
for tpl in wikicode.filter_templates():
name = str(tpl.name).strip()
if name == "ÜbungInfoBox":
if name in T_INFOS or name in T_BESCHR or name in T_HILFS:
for p in tpl.params:
raw[str(p.name).strip()] = str(p.value).strip()
elif name == "Übungsbeschreibung":
for p in tpl.params:
raw[str(p.name).strip()] = str(p.value).strip()
elif name == "SkillDevelopment":
key = str(p.name).strip()
val = str(p.value).strip()
raw[key] = val
elif name in T_SKILL:
raw.setdefault("capabilities", [])
def _getp(t, k):
try:
cap = str(tpl.get("PrimaryCapability").value).strip()
return str(t.get(k).value).strip()
except Exception:
cap = ""
return ""
cap = _getp(tpl, "PrimaryCapability")
lvl = _getp(tpl, "CapabilityLevel")
try:
lvl = int(str(tpl.get("CapabilityLevel").value).strip())
lvl_i = int(lvl)
except Exception:
lvl = 0
lvl_i = 0
if cap:
raw["capabilities"].append({"capability": cap, "level": lvl})
raw["wikitext"] = wikitext
raw["capabilities"].append({"capability": cap, "level": lvl_i})
return raw
def build_payload(raw: Dict[str, Any], fullurl: str, category: str) -> Dict[str, Any]:
# Exercise.capabilities erwartet Dict[str,int]
caps_list = raw.get("capabilities", [])
capabilities = {}
for c in caps_list:
cap = c.get("capability")
lvl = c.get("level")
# ----- Fingerprint (stabil, wie zuvor) -----
def _normalize(v: Any) -> str:
if v is None:
return ""
if isinstance(v, (list, tuple)):
return ",".join(_normalize(x) for x in v)
if isinstance(v, dict):
return json.dumps(v, sort_keys=True, ensure_ascii=False)
return str(v).strip()
def _norm_text(s: str) -> str:
if s is None:
return ""
s = str(s).replace("\u00a0", " ")
s = " ".join(s.split())
return s.strip()
def _canon_title(t: str) -> str:
t = (t or "").strip().replace("_", " ")
return t.replace("–", "-").replace("—", "-")
def compute_fingerprint(payload: Dict[str, Any]) -> str:
kws = payload.get("keywords") or []
kws = [k.replace("\u2013", "-").replace("\u2014", "-") for k in kws]
kws = sorted({(k or "").strip() for k in kws if (k or "").strip()}, key=str.casefold)
dur = payload.get("duration_minutes") or 0
try:
dur = int(round(float(dur)))
except Exception:
dur = 0
fields = [
_canon_title(payload.get("title", "")),
_norm_text(payload.get("summary", "")),
_norm_text(payload.get("execution", "")),
_norm_text(payload.get("notes", "")),
dur,
payload.get("capabilities", {}),
kws,
]
base = "|".join(_normalize(f) for f in fields)
return hashlib.sha256(base.encode("utf-8")).hexdigest()
# ----- Payload (exakte DE-Keys zuerst, dann schmale Fallbacks) -----
EXACT_KEYS = {
"summary": ["Summary", "Kurzbeschreibung"],
"execution": ["Durchführung", "Durchfuehrung", "Ablauf"],
"duration": ["Dauer", "Zeit"],
"keywords": ["Schlüsselworte", "Schlüsselwörter", "Schluesselworte", "Schluesselwoerter", "Keywords", "Tags"],
"equipment_prim": ["Hilfsmittel"],
"equipment_alt": ["Geräte", "Geraete", "Gerät", "Geraet", "Material"],
"discipline": ["Übungstyp", "Uebungstyp", "Disziplin"],
"group": ["Gruppengröße", "Gruppengroesse", "Group"],
"age_group": ["Altersgruppe"],
"target_group": ["Zielgruppe"],
"purpose": ["Ziel", "Zweck"],
"notes": ["Hinweise", "Notes"],
"preparation": ["Vorbereitung", "RefMethode"],
"method": ["Methode", "Method"],
}
def _first_any(raw: Dict[str, Any], keys: List[str]) -> Optional[str]:
for k in keys:
v = raw.get(k)
if isinstance(v, str) and v.strip():
return v.strip()
return None
def build_payload(raw: Dict[str, Any], fullurl: str, category: str, *, mutate: bool = False) -> Dict[str, Any]:
# Capabilities -> Dict[str,int]
capabilities: Dict[str, int] = {}
for c in raw.get("capabilities", []) or []:
cap = c.get("capability"); lvl = c.get("level")
if isinstance(cap, str) and cap:
try:
capabilities[cap] = int(lvl)
except Exception:
pass
# Defaults/Fallbacks
duration = 0.0
try:
duration = float(raw.get("Dauer", 0) or 0)
except Exception:
duration = 0.0
# Exakte Schlüssel zuerst
summary = _first_any(raw, EXACT_KEYS["summary"]) or ""
execution = _first_any(raw, EXACT_KEYS["execution"]) or ""
duration = _first_any(raw, EXACT_KEYS["duration"]) or "0"
kw_raw = _first_any(raw, EXACT_KEYS["keywords"]) or ""
if kw_raw:
parts = [p.strip() for p in kw_raw.replace("\n", ",").split(",")]
keywords = [p for p in parts if p]
else:
keywords = []
kw_raw = raw.get("Schlüsselworte", "")
if isinstance(kw_raw, str):
keywords = [k.strip() for k in kw_raw.split(",") if k.strip()]
eq_raw = _first_any(raw, EXACT_KEYS["equipment_prim"]) or _first_any(raw, EXACT_KEYS["equipment_alt"]) or ""
if eq_raw:
equipment = [e.strip() for e in eq_raw.replace("\n", ",").split(",") if e.strip()]
else:
equipment = []
eq_raw = raw.get("equipment", [])
if isinstance(eq_raw, str):
equipment = [e.strip() for e in eq_raw.split(",") if e.strip()]
elif isinstance(eq_raw, list):
equipment = [str(e).strip() for e in eq_raw if str(e).strip()]
notes = _first_any(raw, EXACT_KEYS["notes"]) or ""
discipline = _first_any(raw, EXACT_KEYS["discipline"]) or ""
group = _first_any(raw, EXACT_KEYS["group"]) or None
age_group = _first_any(raw, EXACT_KEYS["age_group"]) or ""
target_group = _first_any(raw, EXACT_KEYS["target_group"]) or ""
purpose = _first_any(raw, EXACT_KEYS["purpose"]) or ""
preparation = _first_any(raw, EXACT_KEYS["preparation"]) or ""
method = _first_any(raw, EXACT_KEYS["method"]) or ""
try:
duration_f = float(duration or 0)
except Exception:
duration_f = 0.0
payload: Dict[str, Any] = {
"title": raw.get("title") or "",
"summary": raw.get("Summary", "") or "",
"short_description": raw.get("Summary", "") or "",
"summary": summary,
"short_description": summary,
"keywords": keywords,
"link": fullurl or "",
"discipline": raw.get("Übungstyp", "") or "",
"group": raw.get("Gruppengröße", "") or None,
"age_group": raw.get("Altersgruppe", "") or "",
"target_group": raw.get("Zielgruppe", "") or "",
"discipline": discipline,
"group": group,
"age_group": age_group,
"target_group": target_group,
"min_participants": 1,
"duration_minutes": int(round(duration)), # Exercise erwartet int
"duration_minutes": int(round(duration_f)),
"capabilities": capabilities,
"category": category or "",
"purpose": raw.get("Ziel", "") or "",
"execution": raw.get("Durchführung", "") or "",
"notes": raw.get("Hinweise", "") or "",
"preparation": raw.get("RefMethode", "") or "",
"method": raw.get("method", "") or "", # falls im Wikitext vorhanden
"purpose": purpose,
"execution": execution,
"notes": (notes + (" [auto-update]" if mutate else "")).strip(),
"preparation": preparation,
"method": method,
"equipment": equipment,
"fullurl": fullurl or "", # optionales Feld
# Idempotenz (optional nutzbar in exercise_router):
"external_id": f"wiki:{raw.get('pageid')}",
"source": "MediaWiki"
"fullurl": fullurl or "",
"external_id": f"mw:{raw.get('pageid')}",
"source": "MediaWiki",
}
payload["fingerprint"] = compute_fingerprint(payload)
return payload
def ingest_exercise(payload: Dict[str, Any]) -> None:
title = payload.get("title", "<ohne Titel>")
resp = requests.post(EXERCISE_API, json=payload, timeout=60)
if resp.status_code == 422:
print(f"[Ingest] '{title}' -> FAILED 422:\n{resp.text}")
# ----- Lookup/Upsert -----
def lookup_by_external_id(external_id: str) -> Tuple[Optional[Dict[str, Any]], Optional[int]]:
url = f"{EXERCISE_API}/by-external-id"
try:
resp.raise_for_status()
except Exception:
pass
return
resp.raise_for_status()
print(f"[Ingest] '{title}' -> OK")
r = requests.get(url, params={"external_id": external_id}, timeout=REQUEST_TIMEOUT)
if r.status_code == 404:
return None, 404
r.raise_for_status()
return r.json(), r.status_code
except requests.HTTPError as e:
return {"error": str(e), "status_code": getattr(e.response, "status_code", None)}, getattr(e.response, "status_code", None)
except Exception as e:
return {"error": str(e)}, None
def _payload_subset_for_fp(p: Dict[str, Any]) -> Dict[str, Any]:
return {
"title": p.get("title"),
"summary": p.get("summary"),
"execution": p.get("execution"),
"notes": p.get("notes"),
"duration_minutes": p.get("duration_minutes"),
"capabilities": p.get("capabilities") or {},
"keywords": p.get("keywords") or [],
}
def _print_diff(before: Dict[str, Any], after: Dict[str, Any]) -> None:
keys = ["title","summary","execution","notes","duration_minutes","capabilities","keywords"]
b = {k: before.get(k) for k in keys}
a = {k: after.get(k) for k in keys}
def _kws(x):
return sorted({(k or "").strip() for k in (x or [])}, key=str.casefold)
b_norm = {
"title": _canon_title(b.get("title")),
"summary": _norm_text(b.get("summary")),
"execution": _norm_text(b.get("execution")),
"notes": _norm_text(b.get("notes")),
"duration_minutes": b.get("duration_minutes"),
"capabilities": b.get("capabilities"),
"keywords": _kws(b.get("keywords")),
}
a_norm = {
"title": _canon_title(a.get("title")),
"summary": _norm_text(a.get("summary")),
"execution": _norm_text(a.get("execution")),
"notes": _norm_text(a.get("notes")),
"duration_minutes": a.get("duration_minutes"),
"capabilities": a.get("capabilities"),
"keywords": _kws(a.get("keywords")),
}
diff = {k: (b_norm[k], a_norm[k]) for k in keys if b_norm.get(k) != a_norm.get(k)}
if diff:
print("[Diff] changes:", json.dumps(diff, ensure_ascii=False))
else:
print("[Diff] (none in hash fields)")
def _now_iso() -> str:
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
def upsert_exercise(payload: Dict[str, Any], *, dry_run: bool = False) -> str:
title = payload.get("title", "<ohne Titel>")
ext_id = payload.get("external_id")
fp_new = payload.get("fingerprint")
found, status = lookup_by_external_id(ext_id)
action = "create"; reason = "not found (lookup 404)"; found_payload = {}
if not (status == 404 or found is None):
if isinstance(found, dict):
found_payload = found.get("payload", found)
fp_old_stored = found.get("fingerprint") or found_payload.get("fingerprint")
fp_old_recalc = compute_fingerprint(_payload_subset_for_fp(found_payload))
if fp_new == fp_old_stored or fp_new == fp_old_recalc:
action, reason = "skip", "fingerprint unchanged"
else:
action, reason = "update", "fingerprint changed"
else:
action, reason = "create", "unexpected lookup type"
if dry_run:
print(f"[DryRun] {action.upper():6} '{title}' ({ext_id}) – {reason}")
if action == "update":
_print_diff(found_payload, payload)
return action
if action == "create":
body = dict(payload); body["imported_at"] = _now_iso()
resp = requests.post(EXERCISE_API, json=body, timeout=REQUEST_TIMEOUT)
if resp.status_code == 422:
print(f"[Create] '{title}' -> FAILED 422:\n{resp.text}")
try: resp.raise_for_status()
except Exception: pass
else:
resp.raise_for_status(); print(f"[Create] '{title}' – {reason} -> OK")
elif action == "update":
body = dict(payload); body["imported_at"] = _now_iso()
resp = requests.post(EXERCISE_API, json=body, timeout=REQUEST_TIMEOUT)
if resp.status_code == 422:
print(f"[Update] '{title}' -> FAILED 422:\n{resp.text}")
try: resp.raise_for_status()
except Exception: pass
else:
resp.raise_for_status(); print(f"[Update] '{title}' – {reason} -> OK"); _print_diff(found_payload, payload)
else:
print(f"[Skip] '{title}' – {reason}")
return action
# ----- Orchestrierung -----
def process_one(title: str, category: str, *, mutate: bool = False, dry_run: bool = False, debug_raw: bool = False) -> str:
info = fetch_page_info(title)
pid = info.get("pageid"); fullurl = info.get("fullurl") or ""
if not pid:
print(f"[Error] pageid für '{title}' nicht gefunden.", file=sys.stderr); return "failed"
raw = parse_exercise(title, pid)
if debug_raw:
print("[Debug] Raw-Keys:", sorted([k for k in raw.keys() if k not in {"wikitext"}]))
payload = build_payload(raw, fullurl, category, mutate=mutate)
return upsert_exercise(payload, dry_run=dry_run)
def process_all(category: str, *, dry_run: bool = False, debug_raw: bool = False) -> Dict[str, int]:
stats = {"created": 0, "updated": 0, "skipped": 0, "failed": 0}
print(f"[Main] Lade Liste der Übungen aus Kategorie '{category}'…")
pages = fetch_all_pages(category)
print(f"[Main] {len(pages)} Seiten gefunden.")
for idx, (title, entry) in enumerate(pages.items(), 1):
try:
getter = getattr(entry, "get", None)
pid = getter("pageid") if callable(getter) else None
fullurl = getter("fullurl") if callable(getter) else None
if not pid:
info = fetch_page_info(title); pid = info.get("pageid"); fullurl = fullurl or info.get("fullurl")
if not pid:
print(f"[Skip] '{title}' hat keine pageid"); stats["failed"] += 1; continue
raw = parse_exercise(title, pid)
if debug_raw and idx <= 5:
print(f"[Debug] #{idx} '{title}' Raw-Keys:", sorted([k for k in raw.keys() if k not in {"wikitext"}]))
payload = build_payload(raw, fullurl or "", category)
act = upsert_exercise(payload, dry_run=dry_run)
stats["created" if act=="create" else "updated" if act=="update" else "skipped"] += 1
except requests.HTTPError as e:
code = getattr(e, "response", None).status_code if getattr(e, "response", None) else None
if code == 404:
print(f"[Skip] '{title}': page not found (404)"); stats["failed"] += 1
else:
print(f"[Error] '{title}': {e}"); stats["failed"] += 1
except Exception as e:
print(f"[Error] '{title}': {e}"); stats["failed"] += 1
return stats
def run_smoke_test(title: str, category: str, *, debug_raw: bool = False) -> None:
print("\n[SmokeTest] Lauf 1/3: CREATE (Erstimport)"); act1 = process_one(title, category, mutate=False, debug_raw=debug_raw); print("[SmokeTest] Aktion:", act1)
print("\n[SmokeTest] Lauf 2/3: SKIP (Wiederholung, unverändert)"); act2 = process_one(title, category, mutate=False, debug_raw=debug_raw); print("[SmokeTest] Aktion:", act2)
print("\n[SmokeTest] Lauf 3/3: UPDATE (simulierte Wiki-Änderung an 'notes')"); act3 = process_one(title, category, mutate=True, debug_raw=debug_raw); print("[SmokeTest] Aktion:", act3)
print("\n[SmokeTest] Zusammenfassung:"); print(json.dumps({"run1": act1, "run2": act2, "run3": act3}, ensure_ascii=False, indent=2))
# ----- Main -----
def main() -> None:
parser = argparse.ArgumentParser(description="Import exercises from Wiki to Qdrant (via FastAPI wiki_router)")
parser.add_argument("--all", action="store_true", help="Alle Übungen importieren (SMW-Ask)")
@@ -186,13 +439,13 @@ def main() -> None:
parser.add_argument("--username", type=str, default=os.getenv("WIKI_BOT_USER"), help="Wiki-Login Benutzer (überschreibt .env)")
parser.add_argument("--password", type=str, default=os.getenv("WIKI_BOT_PASSWORD"), help="Wiki-Login Passwort (überschreibt .env)")
parser.add_argument("--skip-login", action="store_true", help="Login-Schritt überspringen (falls Session schon aktiv)")
parser.add_argument("--dry-run", action="store_true", help="Kein Schreiben; nur Entscheidungen (create/update/skip) + Gründe loggen")
parser.add_argument("--smoke-test", action="store_true", help="3 Durchläufe (create→skip→update) für --title")
parser.add_argument("--debug-raw", action="store_true", help="Zeigt die aus dem Wiki gelesenen Roh-Keys je Seite")
args = parser.parse_args()
# Sanity
wiki_health()
# Login (sofern nicht explizit übersprungen)
if not args.skip_login:
if not args.username or not args.password:
print("[Login] Fehler: fehlende Credentials. Setze .env (WIKI_BOT_USER/WIKI_BOT_PASSWORD) oder CLI --username/--password.", file=sys.stderr)
@@ -200,39 +453,18 @@ def main() -> None:
try:
wiki_login(args.username, args.password)
except Exception as e:
print(str(e), file=sys.stderr)
sys.exit(1)
print(str(e), file=sys.stderr); sys.exit(1)
if args.smoke_test:
run_smoke_test(args.title, args.category, debug_raw=args.debug_raw); return
# Einzel- oder Vollimport
if args.all:
print(f"[Main] Lade Liste der Übungen aus Kategorie '{args.category}'…")
pages = fetch_all_pages(args.category)
print(f"[Main] {len(pages)} Seiten gefunden.")
for title, entry in pages.items():
pid = entry.get("pageid")
fullurl = entry.get("fullurl")
if not pid:
# Core-Info nachschlagen
info = fetch_page_info(title)
pid = info.get("pageid")
fullurl = fullurl or info.get("fullurl")
if not pid:
print(f"[Skip] '{title}' hat keine pageid")
continue
raw = parse_exercise(title, pid)
payload = build_payload(raw, fullurl or "", args.category)
ingest_exercise(payload)
stats = process_all(args.category, dry_run=args.dry_run, debug_raw=args.debug_raw)
print("\n[Stats] created={created} updated={updated} skipped={skipped} failed={failed}".format(**stats))
else:
print(f"[Main] Import single exercise: {args.title}")
info = fetch_page_info(args.title)
pid = info.get("pageid")
fullurl = info.get("fullurl") or ""
if not pid:
print(f"[Error] pageid für '{args.title}' nicht gefunden.", file=sys.stderr)
sys.exit(1)
raw = parse_exercise(args.title, pid)
payload = build_payload(raw, fullurl, args.category)
ingest_exercise(payload)
result = process_one(args.title, args.category, mutate=False, dry_run=args.dry_run, debug_raw=args.debug_raw)
print(f"[Result] {result}")
if __name__ == "__main__":
main()