Compare commits
214
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
67d7154328 | ||
|
|
4de9a4f649 | ||
|
|
d35bdc64b9 | ||
|
|
3a768be488 | ||
|
|
1965c984f4 | ||
|
|
e3e1700de5 | ||
|
|
f05d766b64 | ||
|
|
58e414041a | ||
|
|
cd5056d4c9 | ||
|
|
39fb821481 | ||
|
|
beb87a8c43 | ||
|
|
4327fc939c | ||
|
|
ef1046c6f5 | ||
|
|
ef8cf719f2 | ||
|
|
6aa6b32a6c | ||
|
|
33dff04d47 | ||
|
|
65d697b7be | ||
|
|
06fc42ed37 | ||
|
|
3c5c567077 | ||
|
|
8f65e550c8 | ||
|
|
6b83879741 | ||
|
|
be265e9cc0 | ||
|
|
680c36ab59 | ||
|
|
96b4f65cd1 | ||
|
|
b1a897e51c | ||
|
|
e5a34efee9 | ||
|
|
f9ac4e4dbf | ||
|
|
1b40e29f40 | ||
|
|
7eba1fb487 | ||
|
|
838083b909 | ||
|
|
8f5eb36b5f | ||
|
|
b7d1bcce3d | ||
|
|
03d3173ca6 | ||
|
|
38a61d7b50 | ||
|
|
0a429e1f7b | ||
|
|
857ba953e3 | ||
|
|
e180018c99 | ||
|
|
ac9956bf00 | ||
|
|
62b5a8bf65 | ||
|
|
303efefcb7 | ||
|
|
feeb7c2d92 | ||
|
|
ea9a54421a | ||
|
|
fdf99b2bb0 | ||
|
|
c7cd641f89 | ||
|
|
18b90c8df3 | ||
|
|
8d3bc1c2e2 | ||
|
|
079d988034 | ||
|
|
aa9d388337 | ||
|
|
92bd3d9a47 | ||
|
|
53058d1504 | ||
|
|
3fe8463a03 | ||
|
|
5c4ce5d727 | ||
|
|
459193e7b1 | ||
|
|
98f21323fb | ||
|
|
515248d438 | ||
|
|
b0c69ad3e0 | ||
|
|
c5f29ab4ae | ||
|
|
876ee898d8 | ||
|
|
e93bab6ea7 | ||
|
|
5225090490 | ||
|
|
e9532e8878 | ||
|
|
23b1cb2966 | ||
|
|
fa909e2e7d | ||
|
|
7fa9ce81bd | ||
|
|
8490911958 | ||
|
|
19d899b277 | ||
|
|
37ec8b614e | ||
|
|
e045371969 | ||
|
|
cd5383432e | ||
|
|
8b8baa27b3 | ||
|
|
386fa3ef0c | ||
|
|
19c96fd00f | ||
|
|
ecb35fb869 | ||
|
|
21cda0072a | ||
|
|
e3858e8bc3 | ||
|
|
f08a331bc6 | ||
|
|
cfcaa926cd | ||
|
|
8ade34af0a | ||
|
|
a6d37c92d2 | ||
|
|
1b7b8091a3 | ||
|
|
94e5ebf577 | ||
|
|
cf302e8334 | ||
|
|
82c7752266 | ||
|
|
c676c8263f | ||
|
|
f6b2375d65 | ||
|
|
d1a065fec8 | ||
|
|
f686ecf947 | ||
|
|
6ac1f318d0 | ||
|
|
cbaf664123 | ||
|
|
470e653da6 | ||
|
|
43d3d8f7f3 | ||
|
|
83c0c9944d | ||
|
|
f6f3213b84 | ||
|
|
e5db7011f3 | ||
|
|
b0bc8518ed | ||
|
|
046b648286 | ||
|
|
e04aecb0c5 | ||
|
|
d64ba06809 | ||
|
|
b0d73cb053 | ||
|
|
5213d262a2 | ||
|
|
2985f5288b | ||
|
|
27b404560c | ||
|
|
16e128668c | ||
|
|
ecfdc67485 | ||
|
|
c9cf1b7e4c | ||
|
|
fa6eb0795a | ||
|
|
4ac504a4ae | ||
|
|
56dd1bcd84 | ||
|
|
b4a07a05af | ||
|
|
5c55229376 | ||
|
|
f3fd71b828 | ||
|
|
079cf174d4 | ||
|
|
4ab44e36a2 | ||
|
|
5278c75ac1 | ||
|
|
a908853c30 | ||
|
|
867a7a8b44 | ||
|
|
2c073c7d3c | ||
|
|
0157faab89 | ||
|
|
f1bfa40b5b | ||
|
|
dcc3083455 | ||
|
|
a5b4dfb31f | ||
|
|
a733212c0f | ||
|
|
49b454d2ec | ||
|
|
18780e5330 | ||
|
|
36fb27edf0 | ||
|
|
c60aba63a4 | ||
|
|
2a98c37ca1 | ||
|
|
0ac8a14ea7 | ||
|
|
234949800b | ||
|
|
c68c7404cf | ||
|
|
f8ed7bb62e | ||
|
|
81d005d969 | ||
|
|
99482ad65a | ||
|
|
2d43e0596c | ||
|
|
74cac7e16c | ||
|
|
0c2dc61cb5 | ||
|
|
97c3461685 | ||
|
|
41e7a49d52 | ||
|
|
725dc6bda4 | ||
|
|
5da8eb6543 | ||
|
|
4f2296b9b0 | ||
|
|
cb9addb1b6 | ||
|
|
c5198a80d8 | ||
|
|
0df266b2e9 | ||
|
|
444f64e94c | ||
|
|
59bcd227f9 | ||
|
|
7f707cffb9 | ||
|
|
a50c4494c6 | ||
|
|
418612cd10 | ||
|
|
828fe1f8ce | ||
|
|
c184c3cae2 | ||
|
|
2cd3605017 | ||
|
|
0d71f41a13 | ||
|
|
f5bfb0cfb4 | ||
|
|
f7ab32ebf4 | ||
|
|
64dbd57fc5 | ||
|
|
ba46957556 | ||
|
|
33b0c83c87 | ||
|
|
5dd58f49f0 | ||
|
|
cc12dcf993 | ||
|
|
cbfdd96152 | ||
|
|
babab3167b | ||
|
|
c61d9c8236 | ||
|
|
e47241740d | ||
|
|
6db1714449 | ||
|
|
136c3bb43f | ||
|
|
2c3ee8efd6 | ||
|
|
48729e6f5d | ||
|
|
3eac646cb6 | ||
|
|
9a18f3cc8b | ||
|
|
5dd20d683f | ||
|
|
8af744fc97 | ||
|
|
342d3e5103 | ||
|
|
7f7d8c87db | ||
|
|
43f695de54 | ||
|
|
20b219d86c | ||
|
|
0ff39d7b14 | ||
|
|
e2ee5df815 | ||
|
|
77a6db7e92 | ||
|
|
d53912a8f1 | ||
|
|
cdcaff184f | ||
|
|
25ec3880bb | ||
|
|
edbd8f0ca8 | ||
|
|
a4272c17a9 | ||
|
|
c8cdf218f2 | ||
|
|
b3833f2051 | ||
|
|
a272c39613 | ||
|
|
6df9b54626 | ||
|
|
156c2c2fd5 | ||
|
|
6011b96fc1 | ||
|
|
7639bb8472 | ||
|
|
c61b66b49d | ||
|
|
cf49715c66 | ||
|
|
bf8a814c58 | ||
|
|
8fadec5c2c | ||
|
|
7263fee4c7 | ||
|
|
4204c2c974 | ||
|
|
9372cfb8ca | ||
|
|
179949b289 | ||
|
|
cd3946bd11 | ||
|
|
12e374bc05 | ||
|
|
fd2fac7112 | ||
|
|
60092b378b | ||
|
|
9025af62f0 | ||
|
|
83bb18b6a7 | ||
|
|
ec759dd1dc | ||
|
|
c454fc40bb | ||
|
|
bfb42cfc24 | ||
|
|
49dd91fee0 | ||
|
|
56c1862205 | ||
|
|
dbefd30dce | ||
|
|
bc06ce5b90 | ||
|
|
b26d9fd10e | ||
|
|
909ee338d2 |
No files matched your search
+6
-13
@@ -1,17 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app — mindnet API package
|
FILE: app/__init__.py
|
||||||
|
DESCRIPTION: Paket-Initialisierung.
|
||||||
Zweck:
|
VERSION: 0.1.0
|
||||||
Markiert 'app/' als Python-Paket, damit 'from app.main import create_app'
|
STATUS: Active
|
||||||
in Tests und Skripten funktioniert.
|
DEPENDENCIES: None
|
||||||
Kompatibilität:
|
LAST_ANALYSIS: 2025-12-15
|
||||||
Python 3.12+
|
|
||||||
Version:
|
|
||||||
0.1.0 (Erstanlage)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Hinweise:
|
|
||||||
Keine Logik – nur Paketinitialisierung.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__version__ = "0.1.0"
|
__version__ = "0.1.0"
|
||||||
+52
-11
@@ -1,36 +1,76 @@
|
|||||||
"""
|
"""
|
||||||
app/config.py — zentrale Konfiguration
|
FILE: app/config.py
|
||||||
Version: 0.4.0 (WP-06 Complete)
|
DESCRIPTION: Zentrale Pydantic-Konfiguration.
|
||||||
|
WP-20: Hybrid-Cloud Modus Support (OpenRouter/Gemini/Ollama).
|
||||||
|
FIX: Einführung von Parametern zur intelligenten Rate-Limit Steuerung (429 Handling).
|
||||||
|
VERSION: 0.6.7
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: os, functools, pathlib, python-dotenv
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import os
|
import os
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
# WP-20: Lade Umgebungsvariablen aus der .env Datei
|
||||||
|
# override=True garantiert, dass Änderungen in der .env immer Vorrang haben.
|
||||||
|
load_dotenv(override=True)
|
||||||
|
|
||||||
class Settings:
|
class Settings:
|
||||||
# Qdrant
|
# --- Qdrant Datenbank ---
|
||||||
QDRANT_URL: str = os.getenv("QDRANT_URL", "http://127.0.0.1:6333")
|
QDRANT_URL: str = os.getenv("QDRANT_URL", "http://127.0.0.1:6333")
|
||||||
QDRANT_API_KEY: str | None = os.getenv("QDRANT_API_KEY")
|
QDRANT_API_KEY: str | None = os.getenv("QDRANT_API_KEY")
|
||||||
COLLECTION_PREFIX: str = os.getenv("MINDNET_PREFIX", "mindnet")
|
COLLECTION_PREFIX: str = os.getenv("MINDNET_PREFIX", "mindnet_dev")
|
||||||
VECTOR_SIZE: int = int(os.getenv("MINDNET_VECTOR_SIZE", "384"))
|
|
||||||
|
# WP-22: Vektor-Dimension für das Embedding-Modell (nomic)
|
||||||
|
VECTOR_SIZE: int = int(os.getenv("VECTOR_DIM", "768"))
|
||||||
DISTANCE: str = os.getenv("MINDNET_DISTANCE", "Cosine")
|
DISTANCE: str = os.getenv("MINDNET_DISTANCE", "Cosine")
|
||||||
|
|
||||||
# Embeddings
|
# --- Lokale Embeddings (Ollama & Sentence-Transformers) ---
|
||||||
|
EMBEDDING_MODEL: str = os.getenv("MINDNET_EMBEDDING_MODEL", "nomic-embed-text")
|
||||||
MODEL_NAME: str = os.getenv("MINDNET_MODEL", "sentence-transformers/all-MiniLM-L6-v2")
|
MODEL_NAME: str = os.getenv("MINDNET_MODEL", "sentence-transformers/all-MiniLM-L6-v2")
|
||||||
|
|
||||||
# WP-05 LLM / Ollama
|
# --- WP-20 Hybrid LLM Provider ---
|
||||||
|
# Erlaubt: "ollama" | "gemini" | "openrouter"
|
||||||
|
MINDNET_LLM_PROVIDER: str = os.getenv("MINDNET_LLM_PROVIDER", "openrouter").lower()
|
||||||
|
# Standardwert 10000, falls nichts in der .env steht
|
||||||
|
MAX_OLLAMA_CHARS: int = int(os.getenv("MAX_OLLAMA_CHARS", 10000))
|
||||||
|
|
||||||
|
# Google AI Studio (2025er Lite-Modell für höhere Kapazität)
|
||||||
|
GOOGLE_API_KEY: str | None = os.getenv("GOOGLE_API_KEY")
|
||||||
|
GEMINI_MODEL: str = os.getenv("MINDNET_GEMINI_MODEL", "gemini-2.5-flash-lite")
|
||||||
|
|
||||||
|
# OpenRouter Integration (Verfügbares Free-Modell 2025)
|
||||||
|
OPENROUTER_API_KEY: str | None = os.getenv("OPENROUTER_API_KEY")
|
||||||
|
OPENROUTER_MODEL: str = os.getenv("OPENROUTER_MODEL", "mistralai/mistral-7b-instruct:free")
|
||||||
|
|
||||||
|
LLM_FALLBACK_ENABLED: bool = os.getenv("MINDNET_LLM_FALLBACK", "true").lower() == "true"
|
||||||
|
|
||||||
|
# --- NEU: Intelligente Rate-Limit Steuerung ---
|
||||||
|
# Dauer der Wartezeit in Sekunden, wenn ein HTTP 429 (Rate Limit) auftritt
|
||||||
|
LLM_RATE_LIMIT_WAIT: float = float(os.getenv("MINDNET_LLM_RATE_LIMIT_WAIT", "60.0"))
|
||||||
|
# Anzahl der Cloud-Retries bei 429, bevor Ollama-Fallback greift
|
||||||
|
LLM_RATE_LIMIT_RETRIES: int = int(os.getenv("MINDNET_LLM_RATE_LIMIT_RETRIES", "3"))
|
||||||
|
|
||||||
|
# --- WP-05 Lokales LLM (Ollama) ---
|
||||||
OLLAMA_URL: str = os.getenv("MINDNET_OLLAMA_URL", "http://127.0.0.1:11434")
|
OLLAMA_URL: str = os.getenv("MINDNET_OLLAMA_URL", "http://127.0.0.1:11434")
|
||||||
LLM_MODEL: str = os.getenv("MINDNET_LLM_MODEL", "phi3:mini")
|
LLM_MODEL: str = os.getenv("MINDNET_LLM_MODEL", "phi3:mini")
|
||||||
PROMPTS_PATH: str = os.getenv("MINDNET_PROMPTS_PATH", "config/prompts.yaml")
|
PROMPTS_PATH: str = os.getenv("MINDNET_PROMPTS_PATH", "config/prompts.yaml")
|
||||||
|
|
||||||
# NEU für WP-06
|
# --- WP-06 / WP-14 Performance & Last-Steuerung ---
|
||||||
LLM_TIMEOUT: float = float(os.getenv("MINDNET_LLM_TIMEOUT", "120.0"))
|
LLM_TIMEOUT: float = float(os.getenv("MINDNET_LLM_TIMEOUT", "300.0"))
|
||||||
DECISION_CONFIG_PATH: str = os.getenv("MINDNET_DECISION_CONFIG", "config/decision_engine.yaml")
|
DECISION_CONFIG_PATH: str = os.getenv("MINDNET_DECISION_CONFIG", "config/decision_engine.yaml")
|
||||||
|
BACKGROUND_LIMIT: int = int(os.getenv("MINDNET_LLM_BACKGROUND_LIMIT", "2"))
|
||||||
|
|
||||||
# API
|
# --- System-Pfade & Ingestion-Logik ---
|
||||||
DEBUG: bool = os.getenv("DEBUG", "false").lower() == "true"
|
DEBUG: bool = os.getenv("DEBUG", "false").lower() == "true"
|
||||||
|
MINDNET_VAULT_ROOT: str = os.getenv("MINDNET_VAULT_ROOT", "./vault_master")
|
||||||
|
MINDNET_TYPES_FILE: str = os.getenv("MINDNET_TYPES_FILE", "config/types.yaml")
|
||||||
|
MINDNET_VOCAB_PATH: str = os.getenv("MINDNET_VOCAB_PATH", "/mindnet/vault/mindnet/_system/dictionary/edge_vocabulary.md")
|
||||||
|
CHANGE_DETECTION_MODE: str = os.getenv("MINDNET_CHANGE_DETECTION_MODE", "full")
|
||||||
|
|
||||||
# WP-04 Retriever Defaults
|
# --- WP-04 Retriever Gewichte ---
|
||||||
RETRIEVER_W_SEM: float = float(os.getenv("MINDNET_WP04_W_SEM", "0.70"))
|
RETRIEVER_W_SEM: float = float(os.getenv("MINDNET_WP04_W_SEM", "0.70"))
|
||||||
RETRIEVER_W_EDGE: float = float(os.getenv("MINDNET_WP04_W_EDGE", "0.25"))
|
RETRIEVER_W_EDGE: float = float(os.getenv("MINDNET_WP04_W_EDGE", "0.25"))
|
||||||
RETRIEVER_W_CENT: float = float(os.getenv("MINDNET_WP04_W_CENT", "0.05"))
|
RETRIEVER_W_CENT: float = float(os.getenv("MINDNET_WP04_W_CENT", "0.05"))
|
||||||
@@ -40,4 +80,5 @@ class Settings:
|
|||||||
|
|
||||||
@lru_cache
|
@lru_cache
|
||||||
def get_settings() -> Settings:
|
def get_settings() -> Settings:
|
||||||
|
"""Gibt die zentralen Einstellungen als Singleton zurück."""
|
||||||
return Settings()
|
return Settings()
|
||||||
@@ -1,136 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
app/core/chunk_payload.py (Mindnet V2 — types.yaml authoritative)
|
|
||||||
- neighbors_prev / neighbors_next sind Listen ([], [id]).
|
|
||||||
- retriever_weight / chunk_profile kommen aus types.yaml (Frontmatter wird ignoriert).
|
|
||||||
- Fallbacks: defaults.* in types.yaml; sonst 1.0 / "default".
|
|
||||||
- WP-11 Update: Injects 'title' into chunk payload for Discovery Service.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
from typing import Any, Dict, List, Optional
|
|
||||||
import os, yaml
|
|
||||||
|
|
||||||
def _env(n: str, d: Optional[str]=None) -> str:
|
|
||||||
v = os.getenv(n)
|
|
||||||
return v if v is not None else (d or "")
|
|
||||||
|
|
||||||
def _load_types() -> dict:
|
|
||||||
p = _env("MINDNET_TYPES_FILE", "./config/types.yaml")
|
|
||||||
try:
|
|
||||||
with open(p, "r", encoding="utf-8") as f:
|
|
||||||
return yaml.safe_load(f) or {}
|
|
||||||
except Exception:
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_types_map(reg: dict) -> dict:
|
|
||||||
if isinstance(reg, dict) and isinstance(reg.get("types"), dict):
|
|
||||||
return reg["types"]
|
|
||||||
return reg if isinstance(reg, dict) else {}
|
|
||||||
|
|
||||||
def _get_defaults(reg: dict) -> dict:
|
|
||||||
if isinstance(reg, dict) and isinstance(reg.get("defaults"), dict):
|
|
||||||
return reg["defaults"]
|
|
||||||
if isinstance(reg, dict) and isinstance(reg.get("global"), dict):
|
|
||||||
return reg["global"]
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _as_float(x: Any):
|
|
||||||
try:
|
|
||||||
return float(x)
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _resolve_chunk_profile(note_type: str, reg: dict) -> str:
|
|
||||||
types = _get_types_map(reg)
|
|
||||||
if isinstance(types, dict):
|
|
||||||
t = types.get(note_type, {})
|
|
||||||
if isinstance(t, dict) and isinstance(t.get("chunk_profile"), str):
|
|
||||||
return t["chunk_profile"]
|
|
||||||
defs = _get_defaults(reg)
|
|
||||||
if isinstance(defs, dict) and isinstance(defs.get("chunk_profile"), str):
|
|
||||||
return defs["chunk_profile"]
|
|
||||||
return "default"
|
|
||||||
|
|
||||||
def _resolve_retriever_weight(note_type: str, reg: dict) -> float:
|
|
||||||
types = _get_types_map(reg)
|
|
||||||
if isinstance(types, dict):
|
|
||||||
t = types.get(note_type, {})
|
|
||||||
if isinstance(t, dict) and (t.get("retriever_weight") is not None):
|
|
||||||
v = _as_float(t.get("retriever_weight"))
|
|
||||||
if v is not None:
|
|
||||||
return float(v)
|
|
||||||
defs = _get_defaults(reg)
|
|
||||||
if isinstance(defs, dict) and (defs.get("retriever_weight") is not None):
|
|
||||||
v = _as_float(defs.get("retriever_weight"))
|
|
||||||
if v is not None:
|
|
||||||
return float(v)
|
|
||||||
return 1.0
|
|
||||||
|
|
||||||
def _as_list(x):
|
|
||||||
if x is None:
|
|
||||||
return []
|
|
||||||
if isinstance(x, list):
|
|
||||||
return x
|
|
||||||
return [x]
|
|
||||||
|
|
||||||
def make_chunk_payloads(note: Dict[str, Any],
|
|
||||||
note_path: str,
|
|
||||||
chunks_from_chunker: List[Any],
|
|
||||||
*,
|
|
||||||
note_text: str = "",
|
|
||||||
types_cfg: Optional[dict] = None,
|
|
||||||
file_path: Optional[str] = None) -> List[Dict[str, Any]]:
|
|
||||||
fm = (note or {}).get("frontmatter", {}) or {}
|
|
||||||
note_type = fm.get("type") or note.get("type") or "concept"
|
|
||||||
|
|
||||||
# WP-11 FIX: Title Extraction für Discovery Service
|
|
||||||
# Wir holen den Titel aus Frontmatter oder Fallback ID/Untitled
|
|
||||||
title = fm.get("title") or note.get("title") or fm.get("id") or "Untitled"
|
|
||||||
|
|
||||||
reg = types_cfg if isinstance(types_cfg, dict) else _load_types()
|
|
||||||
|
|
||||||
# types.yaml authoritative
|
|
||||||
cp = _resolve_chunk_profile(note_type, reg)
|
|
||||||
rw = _resolve_retriever_weight(note_type, reg)
|
|
||||||
|
|
||||||
tags = fm.get("tags") or []
|
|
||||||
if isinstance(tags, str):
|
|
||||||
tags = [tags]
|
|
||||||
|
|
||||||
out: List[Dict[str, Any]] = []
|
|
||||||
for idx, ch in enumerate(chunks_from_chunker):
|
|
||||||
# Attribute oder Keys (Chunk-Objekt oder Dict)
|
|
||||||
cid = getattr(ch, "id", None) or (ch.get("id") if isinstance(ch, dict) else None)
|
|
||||||
nid = getattr(ch, "note_id", None) or (ch.get("note_id") if isinstance(ch, dict) else fm.get("id"))
|
|
||||||
index = getattr(ch, "index", None) or (ch.get("index") if isinstance(ch, dict) else idx)
|
|
||||||
text = getattr(ch, "text", None) or (ch.get("text") if isinstance(ch, dict) else "")
|
|
||||||
window = getattr(ch, "window", None) or (ch.get("window") if isinstance(ch, dict) else text)
|
|
||||||
prev_id = getattr(ch, "neighbors_prev", None) or (ch.get("neighbors_prev") if isinstance(ch, dict) else None)
|
|
||||||
next_id = getattr(ch, "neighbors_next", None) or (ch.get("neighbors_next") if isinstance(ch, dict) else None)
|
|
||||||
|
|
||||||
pl: Dict[str, Any] = {
|
|
||||||
"note_id": nid,
|
|
||||||
"chunk_id": cid,
|
|
||||||
"title": title, # <--- HIER: Titel in Payload einfügen
|
|
||||||
"index": int(index),
|
|
||||||
"ord": int(index) + 1,
|
|
||||||
"type": note_type,
|
|
||||||
"tags": tags,
|
|
||||||
"text": text,
|
|
||||||
"window": window,
|
|
||||||
"neighbors_prev": _as_list(prev_id),
|
|
||||||
"neighbors_next": _as_list(next_id),
|
|
||||||
"section": getattr(ch, "section", None) or (ch.get("section") if isinstance(ch, dict) else ""),
|
|
||||||
"path": note_path,
|
|
||||||
"source_path": file_path or note_path,
|
|
||||||
"retriever_weight": float(rw),
|
|
||||||
"chunk_profile": cp,
|
|
||||||
}
|
|
||||||
# Aufräumen von Alt-Feldern
|
|
||||||
for alias in ("chunk_num", "Chunk_Number"):
|
|
||||||
pl.pop(alias, None)
|
|
||||||
|
|
||||||
out.append(pl)
|
|
||||||
|
|
||||||
return out
|
|
||||||
@@ -1,330 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import List, Dict, Optional, Tuple, Any, Set
|
|
||||||
import re
|
|
||||||
import math
|
|
||||||
import yaml
|
|
||||||
from pathlib import Path
|
|
||||||
from markdown_it import MarkdownIt
|
|
||||||
from markdown_it.token import Token
|
|
||||||
import asyncio
|
|
||||||
import logging
|
|
||||||
|
|
||||||
# Services
|
|
||||||
from app.services.semantic_analyzer import get_semantic_analyzer
|
|
||||||
|
|
||||||
# Core Imports
|
|
||||||
try:
|
|
||||||
from app.core.derive_edges import build_edges_for_note
|
|
||||||
except ImportError:
|
|
||||||
# Mock für Tests
|
|
||||||
def build_edges_for_note(note_id, chunks, note_level_references=None, include_note_scope_refs=False): return []
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# ==========================================
|
|
||||||
# 1. HELPER & CONFIG
|
|
||||||
# ==========================================
|
|
||||||
|
|
||||||
BASE_DIR = Path(__file__).resolve().parent.parent.parent
|
|
||||||
CONFIG_PATH = BASE_DIR / "config" / "types.yaml"
|
|
||||||
DEFAULT_PROFILE = {"strategy": "sliding_window", "target": 400, "max": 600, "overlap": (50, 80)}
|
|
||||||
_CONFIG_CACHE = None
|
|
||||||
|
|
||||||
def _load_yaml_config() -> Dict[str, Any]:
|
|
||||||
global _CONFIG_CACHE
|
|
||||||
if _CONFIG_CACHE is not None: return _CONFIG_CACHE
|
|
||||||
if not CONFIG_PATH.exists(): return {}
|
|
||||||
try:
|
|
||||||
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
|
|
||||||
data = yaml.safe_load(f)
|
|
||||||
_CONFIG_CACHE = data
|
|
||||||
return data
|
|
||||||
except Exception: return {}
|
|
||||||
|
|
||||||
def get_chunk_config(note_type: str) -> Dict[str, Any]:
|
|
||||||
full_config = _load_yaml_config()
|
|
||||||
profiles = full_config.get("chunking_profiles", {})
|
|
||||||
type_def = full_config.get("types", {}).get(note_type.lower(), {})
|
|
||||||
profile_name = type_def.get("chunking_profile")
|
|
||||||
if not profile_name:
|
|
||||||
profile_name = full_config.get("defaults", {}).get("chunking_profile", "sliding_standard")
|
|
||||||
|
|
||||||
config = profiles.get(profile_name, DEFAULT_PROFILE).copy()
|
|
||||||
if "overlap" in config and isinstance(config["overlap"], list):
|
|
||||||
config["overlap"] = tuple(config["overlap"])
|
|
||||||
return config
|
|
||||||
|
|
||||||
def extract_frontmatter_from_text(md_text: str) -> Tuple[Dict[str, Any], str]:
|
|
||||||
fm_match = re.match(r'^\s*---\s*\n(.*?)\n---', md_text, re.DOTALL)
|
|
||||||
if not fm_match: return {}, md_text
|
|
||||||
try:
|
|
||||||
frontmatter = yaml.safe_load(fm_match.group(1))
|
|
||||||
if not isinstance(frontmatter, dict): frontmatter = {}
|
|
||||||
except yaml.YAMLError:
|
|
||||||
frontmatter = {}
|
|
||||||
text_without_fm = re.sub(r'^\s*---\s*\n(.*?)\n---', '', md_text, flags=re.DOTALL)
|
|
||||||
return frontmatter, text_without_fm.strip()
|
|
||||||
|
|
||||||
# ==========================================
|
|
||||||
# 2. DATA CLASSES
|
|
||||||
# ==========================================
|
|
||||||
|
|
||||||
_SENT_SPLIT = re.compile(r'(?<=[.!?])\s+(?=[A-ZÄÖÜ0-9„(])'); _WS = re.compile(r'\s+')
|
|
||||||
|
|
||||||
def estimate_tokens(text: str) -> int:
|
|
||||||
return max(1, math.ceil(len(text.strip()) / 4))
|
|
||||||
|
|
||||||
def split_sentences(text: str) -> list[str]:
|
|
||||||
text = _WS.sub(' ', text.strip())
|
|
||||||
if not text: return []
|
|
||||||
parts = _SENT_SPLIT.split(text)
|
|
||||||
return [p.strip() for p in parts if p.strip()]
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class RawBlock:
|
|
||||||
kind: str; text: str; level: Optional[int]; section_path: str; section_title: Optional[str]
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Chunk:
|
|
||||||
id: str; note_id: str; index: int; text: str; window: str; token_count: int
|
|
||||||
section_title: Optional[str]; section_path: str
|
|
||||||
neighbors_prev: Optional[str]; neighbors_next: Optional[str]
|
|
||||||
suggested_edges: Optional[List[str]] = None
|
|
||||||
|
|
||||||
# ==========================================
|
|
||||||
# 3. PARSING & STRATEGIES (SYNCHRON)
|
|
||||||
# ==========================================
|
|
||||||
|
|
||||||
def parse_blocks(md_text: str) -> Tuple[List[RawBlock], str]:
|
|
||||||
"""Zerlegt Text in logische Blöcke (Absätze, Header)."""
|
|
||||||
blocks = []
|
|
||||||
h1_title = "Dokument"
|
|
||||||
section_path = "/"
|
|
||||||
current_h2 = None
|
|
||||||
|
|
||||||
fm, text_without_fm = extract_frontmatter_from_text(md_text)
|
|
||||||
|
|
||||||
h1_match = re.search(r'^#\s+(.*)', text_without_fm, re.MULTILINE)
|
|
||||||
if h1_match:
|
|
||||||
h1_title = h1_match.group(1).strip()
|
|
||||||
|
|
||||||
lines = text_without_fm.split('\n')
|
|
||||||
buffer = []
|
|
||||||
|
|
||||||
for line in lines:
|
|
||||||
stripped = line.strip()
|
|
||||||
if stripped.startswith('# '):
|
|
||||||
continue
|
|
||||||
elif stripped.startswith('## '):
|
|
||||||
if buffer:
|
|
||||||
content = "\n".join(buffer).strip()
|
|
||||||
if content:
|
|
||||||
blocks.append(RawBlock("paragraph", content, None, section_path, current_h2))
|
|
||||||
buffer = []
|
|
||||||
current_h2 = stripped[3:].strip()
|
|
||||||
section_path = f"/{current_h2}"
|
|
||||||
blocks.append(RawBlock("heading", stripped, 2, section_path, current_h2))
|
|
||||||
elif not stripped:
|
|
||||||
if buffer:
|
|
||||||
content = "\n".join(buffer).strip()
|
|
||||||
if content:
|
|
||||||
blocks.append(RawBlock("paragraph", content, None, section_path, current_h2))
|
|
||||||
buffer = []
|
|
||||||
else:
|
|
||||||
buffer.append(line)
|
|
||||||
|
|
||||||
if buffer:
|
|
||||||
content = "\n".join(buffer).strip()
|
|
||||||
if content:
|
|
||||||
blocks.append(RawBlock("paragraph", content, None, section_path, current_h2))
|
|
||||||
|
|
||||||
return blocks, h1_title
|
|
||||||
|
|
||||||
def _strategy_sliding_window(blocks: List[RawBlock], config: Dict[str, Any], note_id: str, doc_title: str = "", context_prefix: str = "") -> List[Chunk]:
|
|
||||||
target = config.get("target", 400)
|
|
||||||
max_tokens = config.get("max", 600)
|
|
||||||
overlap_val = config.get("overlap", (50, 80))
|
|
||||||
overlap = sum(overlap_val) // 2 if isinstance(overlap_val, tuple) else overlap_val
|
|
||||||
chunks = []; buf = []
|
|
||||||
|
|
||||||
def _create_chunk(txt, win, sec, path):
|
|
||||||
idx = len(chunks)
|
|
||||||
chunks.append(Chunk(
|
|
||||||
id=f"{note_id}#c{idx:02d}", note_id=note_id, index=idx,
|
|
||||||
text=txt, window=win, token_count=estimate_tokens(txt),
|
|
||||||
section_title=sec, section_path=path, neighbors_prev=None, neighbors_next=None,
|
|
||||||
suggested_edges=[]
|
|
||||||
))
|
|
||||||
|
|
||||||
def flush_buffer():
|
|
||||||
nonlocal buf
|
|
||||||
if not buf: return
|
|
||||||
|
|
||||||
text_body = "\n\n".join([b.text for b in buf])
|
|
||||||
win_body = f"{context_prefix}\n{text_body}".strip() if context_prefix else text_body
|
|
||||||
|
|
||||||
if estimate_tokens(text_body) <= max_tokens:
|
|
||||||
_create_chunk(text_body, win_body, buf[-1].section_title, buf[-1].section_path)
|
|
||||||
else:
|
|
||||||
sentences = split_sentences(text_body)
|
|
||||||
current_chunk_sents = []
|
|
||||||
current_len = 0
|
|
||||||
|
|
||||||
for sent in sentences:
|
|
||||||
sent_len = estimate_tokens(sent)
|
|
||||||
if current_len + sent_len > target and current_chunk_sents:
|
|
||||||
c_txt = " ".join(current_chunk_sents)
|
|
||||||
c_win = f"{context_prefix}\n{c_txt}".strip() if context_prefix else c_txt
|
|
||||||
_create_chunk(c_txt, c_win, buf[-1].section_title, buf[-1].section_path)
|
|
||||||
|
|
||||||
overlap_sents = []
|
|
||||||
ov_len = 0
|
|
||||||
for s in reversed(current_chunk_sents):
|
|
||||||
if ov_len + estimate_tokens(s) < overlap:
|
|
||||||
overlap_sents.insert(0, s)
|
|
||||||
ov_len += estimate_tokens(s)
|
|
||||||
else:
|
|
||||||
break
|
|
||||||
|
|
||||||
current_chunk_sents = list(overlap_sents)
|
|
||||||
current_chunk_sents.append(sent)
|
|
||||||
current_len = ov_len + sent_len
|
|
||||||
else:
|
|
||||||
current_chunk_sents.append(sent)
|
|
||||||
current_len += sent_len
|
|
||||||
|
|
||||||
if current_chunk_sents:
|
|
||||||
c_txt = " ".join(current_chunk_sents)
|
|
||||||
c_win = f"{context_prefix}\n{c_txt}".strip() if context_prefix else c_txt
|
|
||||||
_create_chunk(c_txt, c_win, buf[-1].section_title, buf[-1].section_path)
|
|
||||||
|
|
||||||
buf = []
|
|
||||||
|
|
||||||
for b in blocks:
|
|
||||||
if b.kind == "heading": continue
|
|
||||||
current_buf_text = "\n\n".join([x.text for x in buf])
|
|
||||||
if estimate_tokens(current_buf_text) + estimate_tokens(b.text) >= target:
|
|
||||||
flush_buffer()
|
|
||||||
buf.append(b)
|
|
||||||
if estimate_tokens(b.text) >= target:
|
|
||||||
flush_buffer()
|
|
||||||
|
|
||||||
flush_buffer()
|
|
||||||
return chunks
|
|
||||||
|
|
||||||
def _strategy_by_heading(blocks: List[RawBlock], config: Dict[str, Any], note_id: str, doc_title: str = "") -> List[Chunk]:
|
|
||||||
return _strategy_sliding_window(blocks, config, note_id, doc_title, context_prefix=f"# {doc_title}")
|
|
||||||
|
|
||||||
# ==========================================
|
|
||||||
# 4. ORCHESTRATION (ASYNC) - WP-15 CORE
|
|
||||||
# ==========================================
|
|
||||||
|
|
||||||
async def assemble_chunks(note_id: str, md_text: str, note_type: str, config: Optional[Dict] = None) -> List[Chunk]:
|
|
||||||
if config is None:
|
|
||||||
config = get_chunk_config(note_type)
|
|
||||||
|
|
||||||
fm, body_text = extract_frontmatter_from_text(md_text)
|
|
||||||
note_status = fm.get("status", "").lower()
|
|
||||||
|
|
||||||
primary_strategy = config.get("strategy", "sliding_window")
|
|
||||||
enable_smart_edges = config.get("enable_smart_edge_allocation", False)
|
|
||||||
|
|
||||||
if enable_smart_edges and note_status in ["draft", "initial_gen"]:
|
|
||||||
logger.info(f"Chunker: Skipping Smart Edges for draft '{note_id}'.")
|
|
||||||
enable_smart_edges = False
|
|
||||||
|
|
||||||
blocks, doc_title = parse_blocks(md_text)
|
|
||||||
|
|
||||||
if primary_strategy == "by_heading":
|
|
||||||
chunks = await asyncio.to_thread(_strategy_by_heading, blocks, config, note_id, doc_title)
|
|
||||||
else:
|
|
||||||
chunks = await asyncio.to_thread(_strategy_sliding_window, blocks, config, note_id, doc_title)
|
|
||||||
|
|
||||||
if not chunks:
|
|
||||||
return []
|
|
||||||
|
|
||||||
if enable_smart_edges:
|
|
||||||
# Hier rufen wir nun die Smart Edge Allocation auf
|
|
||||||
chunks = await _run_smart_edge_allocation(chunks, md_text, note_id, note_type)
|
|
||||||
|
|
||||||
for i, ch in enumerate(chunks):
|
|
||||||
ch.neighbors_prev = chunks[i-1].id if i > 0 else None
|
|
||||||
ch.neighbors_next = chunks[i+1].id if i < len(chunks)-1 else None
|
|
||||||
|
|
||||||
return chunks
|
|
||||||
|
|
||||||
def _extract_all_edges_from_md(md_text: str, note_id: str, note_type: str) -> List[str]:
|
|
||||||
"""
|
|
||||||
Hilfsfunktion: Erstellt einen Dummy-Chunk für den gesamten Text und ruft
|
|
||||||
den Edge-Parser auf, um ALLE Kanten der Notiz zu finden.
|
|
||||||
"""
|
|
||||||
# 1. Dummy Chunk erstellen, der den gesamten Text enthält
|
|
||||||
# Das ist notwendig, da build_edges_for_note Kanten nur aus Chunks extrahiert.
|
|
||||||
dummy_chunk = {
|
|
||||||
"chunk_id": f"{note_id}#full",
|
|
||||||
"text": md_text,
|
|
||||||
"content": md_text, # Sicherstellen, dass der Parser Text findet
|
|
||||||
"window": md_text,
|
|
||||||
"type": note_type
|
|
||||||
}
|
|
||||||
|
|
||||||
# 2. Aufruf des Parsers (Signatur-Fix!)
|
|
||||||
# derive_edges.py: build_edges_for_note(note_id, chunks, note_level_references=None, include_note_scope_refs=False)
|
|
||||||
raw_edges = build_edges_for_note(
|
|
||||||
note_id,
|
|
||||||
[dummy_chunk],
|
|
||||||
note_level_references=None,
|
|
||||||
include_note_scope_refs=False
|
|
||||||
)
|
|
||||||
|
|
||||||
# 3. Kanten extrahieren
|
|
||||||
all_candidates = set()
|
|
||||||
for e in raw_edges:
|
|
||||||
kind = e.get("kind")
|
|
||||||
target = e.get("target_id")
|
|
||||||
if target and kind not in ["belongs_to", "next", "prev", "backlink"]:
|
|
||||||
all_candidates.add(f"{kind}:{target}")
|
|
||||||
|
|
||||||
return list(all_candidates)
|
|
||||||
|
|
||||||
async def _run_smart_edge_allocation(chunks: List[Chunk], full_text: str, note_id: str, note_type: str) -> List[Chunk]:
|
|
||||||
analyzer = get_semantic_analyzer()
|
|
||||||
|
|
||||||
# A. Alle potenziellen Kanten der Notiz sammeln (über den Dummy-Chunk Trick)
|
|
||||||
candidate_list = _extract_all_edges_from_md(full_text, note_id, note_type)
|
|
||||||
|
|
||||||
if not candidate_list:
|
|
||||||
return chunks
|
|
||||||
|
|
||||||
# B. LLM Filterung pro Chunk (Parallel)
|
|
||||||
tasks = []
|
|
||||||
for chunk in chunks:
|
|
||||||
tasks.append(analyzer.assign_edges_to_chunk(chunk.text, candidate_list, note_type))
|
|
||||||
|
|
||||||
results_per_chunk = await asyncio.gather(*tasks)
|
|
||||||
|
|
||||||
# C. Injection & Fallback
|
|
||||||
assigned_edges_global = set()
|
|
||||||
|
|
||||||
for i, confirmed_edges in enumerate(results_per_chunk):
|
|
||||||
chunk = chunks[i]
|
|
||||||
chunk.suggested_edges = confirmed_edges
|
|
||||||
assigned_edges_global.update(confirmed_edges)
|
|
||||||
|
|
||||||
if confirmed_edges:
|
|
||||||
injection_str = "\n" + " ".join([f"[[rel:{e.split(':')[0]}|{e.split(':')[1]}]]" for e in confirmed_edges if ':' in e])
|
|
||||||
chunk.text += injection_str
|
|
||||||
chunk.window += injection_str
|
|
||||||
|
|
||||||
# D. Fallback: Unassigned Kanten überall hin
|
|
||||||
unassigned = set(candidate_list) - assigned_edges_global
|
|
||||||
if unassigned:
|
|
||||||
fallback_str = "\n" + " ".join([f"[[rel:{e.split(':')[0]}|{e.split(':')[1]}]]" for e in unassigned if ':' in e])
|
|
||||||
for chunk in chunks:
|
|
||||||
chunk.text += fallback_str
|
|
||||||
chunk.window += fallback_str
|
|
||||||
if chunk.suggested_edges is None: chunk.suggested_edges = []
|
|
||||||
chunk.suggested_edges.extend(list(unassigned))
|
|
||||||
|
|
||||||
return chunks
|
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/__init__.py
|
||||||
|
DESCRIPTION: Package-Einstiegspunkt für Chunking. Exportiert assemble_chunks.
|
||||||
|
VERSION: 3.3.0
|
||||||
|
"""
|
||||||
|
from .chunking_processor import assemble_chunks
|
||||||
|
from .chunking_utils import get_chunk_config, extract_frontmatter_from_text
|
||||||
|
from .chunking_models import Chunk
|
||||||
|
|
||||||
|
__all__ = ["assemble_chunks", "get_chunk_config", "extract_frontmatter_from_text", "Chunk"]
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/chunking_models.py
|
||||||
|
DESCRIPTION: Datenklassen für das Chunking-System.
|
||||||
|
"""
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import List, Dict, Optional, Any
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class RawBlock:
|
||||||
|
"""Repräsentiert einen logischen Block aus dem Markdown-Parsing."""
|
||||||
|
kind: str
|
||||||
|
text: str
|
||||||
|
level: Optional[int]
|
||||||
|
section_path: str
|
||||||
|
section_title: Optional[str]
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Chunk:
|
||||||
|
"""Das finale Chunk-Objekt für Embedding und Graph-Speicherung."""
|
||||||
|
id: str
|
||||||
|
note_id: str
|
||||||
|
index: int
|
||||||
|
text: str
|
||||||
|
window: str
|
||||||
|
token_count: int
|
||||||
|
section_title: Optional[str]
|
||||||
|
section_path: str
|
||||||
|
neighbors_prev: Optional[str]
|
||||||
|
neighbors_next: Optional[str]
|
||||||
|
candidate_pool: List[Dict[str, Any]] = field(default_factory=list)
|
||||||
|
suggested_edges: Optional[List[str]] = None
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/chunking_parser.py
|
||||||
|
DESCRIPTION: Zerlegt Markdown in logische Einheiten (RawBlocks).
|
||||||
|
Hält alle Überschriftenebenen (H1-H6) im Stream.
|
||||||
|
Stellt die Funktion parse_edges_robust zur Verfügung.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from typing import List, Tuple, Set
|
||||||
|
from .chunking_models import RawBlock
|
||||||
|
from .chunking_utils import extract_frontmatter_from_text
|
||||||
|
|
||||||
|
_WS = re.compile(r'\s+')
|
||||||
|
_SENT_SPLIT = re.compile(r'(?<=[.!?])\s+(?=[A-ZÄÖÜ0-9„(])')
|
||||||
|
|
||||||
|
def split_sentences(text: str) -> list[str]:
|
||||||
|
"""Teilt Text in Sätze auf unter Berücksichtigung deutscher Interpunktion."""
|
||||||
|
text = _WS.sub(' ', text.strip())
|
||||||
|
if not text: return []
|
||||||
|
# Splittet bei Punkt, Ausrufezeichen oder Fragezeichen, gefolgt von Leerzeichen und Großbuchstabe
|
||||||
|
return [p.strip() for p in _SENT_SPLIT.split(text) if p.strip()]
|
||||||
|
|
||||||
|
def parse_blocks(md_text: str) -> Tuple[List[RawBlock], str]:
|
||||||
|
"""Zerlegt Text in logische Einheiten (RawBlocks), inklusive H1-H6."""
|
||||||
|
blocks = []
|
||||||
|
h1_title = "Dokument"
|
||||||
|
section_path = "/"
|
||||||
|
current_section_title = None
|
||||||
|
|
||||||
|
# Frontmatter entfernen
|
||||||
|
fm, text_without_fm = extract_frontmatter_from_text(md_text)
|
||||||
|
|
||||||
|
# H1 für Note-Titel extrahieren (Metadaten-Zweck)
|
||||||
|
h1_match = re.search(r'^#\s+(.*)', text_without_fm, re.MULTILINE)
|
||||||
|
if h1_match:
|
||||||
|
h1_title = h1_match.group(1).strip()
|
||||||
|
|
||||||
|
lines = text_without_fm.split('\n')
|
||||||
|
buffer = []
|
||||||
|
|
||||||
|
for line in lines:
|
||||||
|
stripped = line.strip()
|
||||||
|
|
||||||
|
# Heading-Erkennung (H1 bis H6)
|
||||||
|
heading_match = re.match(r'^(#{1,6})\s+(.*)', stripped)
|
||||||
|
if heading_match:
|
||||||
|
# Vorherigen Text-Block abschließen
|
||||||
|
if buffer:
|
||||||
|
content = "\n".join(buffer).strip()
|
||||||
|
if content:
|
||||||
|
blocks.append(RawBlock("paragraph", content, None, section_path, current_section_title))
|
||||||
|
buffer = []
|
||||||
|
|
||||||
|
level = len(heading_match.group(1))
|
||||||
|
title = heading_match.group(2).strip()
|
||||||
|
|
||||||
|
# Pfad- und Titel-Update für die Metadaten der folgenden Blöcke
|
||||||
|
if level == 1:
|
||||||
|
current_section_title = title; section_path = "/"
|
||||||
|
elif level == 2:
|
||||||
|
current_section_title = title; section_path = f"/{current_section_title}"
|
||||||
|
|
||||||
|
# Die Überschrift selbst als regulären Block hinzufügen
|
||||||
|
blocks.append(RawBlock("heading", stripped, level, section_path, current_section_title))
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Trenner (---) oder Leerzeilen beenden Blöcke, außer innerhalb von Callouts
|
||||||
|
if (not stripped or stripped == "---") and not line.startswith('>'):
|
||||||
|
if buffer:
|
||||||
|
content = "\n".join(buffer).strip()
|
||||||
|
if content:
|
||||||
|
blocks.append(RawBlock("paragraph", content, None, section_path, current_section_title))
|
||||||
|
buffer = []
|
||||||
|
if stripped == "---":
|
||||||
|
blocks.append(RawBlock("separator", "---", None, section_path, current_section_title))
|
||||||
|
else:
|
||||||
|
buffer.append(line)
|
||||||
|
|
||||||
|
if buffer:
|
||||||
|
content = "\n".join(buffer).strip()
|
||||||
|
if content:
|
||||||
|
blocks.append(RawBlock("paragraph", content, None, section_path, current_section_title))
|
||||||
|
|
||||||
|
return blocks, h1_title
|
||||||
|
|
||||||
|
def parse_edges_robust(text: str) -> Set[str]:
|
||||||
|
"""Extrahiert Kanten-Kandidaten aus Wikilinks und Callouts."""
|
||||||
|
found_edges = set()
|
||||||
|
# 1. Wikilinks [[rel:kind|target]]
|
||||||
|
inlines = re.findall(r'\[\[rel:([^\|\]]+)\|?([^\]]*)\]\]', text)
|
||||||
|
for kind, target in inlines:
|
||||||
|
k = kind.strip().lower()
|
||||||
|
t = target.strip()
|
||||||
|
if k and t: found_edges.add(f"{k}:{t}")
|
||||||
|
|
||||||
|
# 2. Callout Edges > [!edge] kind
|
||||||
|
lines = text.split('\n')
|
||||||
|
current_edge_type = None
|
||||||
|
for line in lines:
|
||||||
|
stripped = line.strip()
|
||||||
|
callout_match = re.match(r'>\s*\[!edge\]\s*([^:\s]+)', stripped)
|
||||||
|
if callout_match:
|
||||||
|
current_edge_type = callout_match.group(1).strip().lower()
|
||||||
|
# Links in der gleichen Zeile des Callouts
|
||||||
|
links = re.findall(r'\[\[([^\]]+)\]\]', stripped)
|
||||||
|
for l in links:
|
||||||
|
if "rel:" not in l: found_edges.add(f"{current_edge_type}:{l}")
|
||||||
|
continue
|
||||||
|
# Links in Folgezeilen des Callouts
|
||||||
|
if current_edge_type and stripped.startswith('>'):
|
||||||
|
links = re.findall(r'\[\[([^\]]+)\]\]', stripped)
|
||||||
|
for l in links:
|
||||||
|
if "rel:" not in l: found_edges.add(f"{current_edge_type}:{l}")
|
||||||
|
elif not stripped.startswith('>'):
|
||||||
|
current_edge_type = None
|
||||||
|
return found_edges
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/chunking_processor.py
|
||||||
|
DESCRIPTION: Der zentrale Orchestrator für das Chunking-System.
|
||||||
|
AUDIT v3.3.4: Wiederherstellung der "Gold-Standard" Qualität.
|
||||||
|
- Fix: Synchronisierung der Parameter (context_prefix) für alle Strategien.
|
||||||
|
- Integriert physikalische Kanten-Injektion (Propagierung).
|
||||||
|
- Stellt H1-Kontext-Fenster sicher.
|
||||||
|
- Baut den Candidate-Pool für die WP-15b Ingestion auf.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import re
|
||||||
|
import logging
|
||||||
|
from typing import List, Dict, Optional
|
||||||
|
from .chunking_models import Chunk
|
||||||
|
from .chunking_utils import get_chunk_config, extract_frontmatter_from_text
|
||||||
|
from .chunking_parser import parse_blocks, parse_edges_robust
|
||||||
|
from .chunking_strategies import strategy_sliding_window, strategy_by_heading
|
||||||
|
from .chunking_propagation import propagate_section_edges
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
async def assemble_chunks(note_id: str, md_text: str, note_type: str, config: Optional[Dict] = None) -> List[Chunk]:
|
||||||
|
"""
|
||||||
|
Hauptfunktion zur Zerlegung einer Note.
|
||||||
|
Verbindet Strategien mit physikalischer Kontext-Anreicherung.
|
||||||
|
"""
|
||||||
|
# 1. Konfiguration & Parsing
|
||||||
|
if config is None:
|
||||||
|
config = get_chunk_config(note_type)
|
||||||
|
|
||||||
|
fm, body_text = extract_frontmatter_from_text(md_text)
|
||||||
|
blocks, doc_title = parse_blocks(md_text)
|
||||||
|
|
||||||
|
# Vorbereitung des H1-Präfix für die Embedding-Fenster (Breadcrumbs)
|
||||||
|
h1_prefix = f"# {doc_title}" if doc_title else ""
|
||||||
|
|
||||||
|
# 2. Anwendung der Splitting-Strategie
|
||||||
|
# Alle Strategien nutzen nun einheitlich context_prefix für die Window-Bildung.
|
||||||
|
if config.get("strategy") == "by_heading":
|
||||||
|
chunks = await asyncio.to_thread(
|
||||||
|
strategy_by_heading, blocks, config, note_id, context_prefix=h1_prefix
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
chunks = await asyncio.to_thread(
|
||||||
|
strategy_sliding_window, blocks, config, note_id, context_prefix=h1_prefix
|
||||||
|
)
|
||||||
|
|
||||||
|
if not chunks:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# 3. Physikalische Kontext-Anreicherung (Der Qualitäts-Fix)
|
||||||
|
# Schreibt Kanten aus Callouts/Inlines hart in den Text für Qdrant.
|
||||||
|
chunks = propagate_section_edges(chunks)
|
||||||
|
|
||||||
|
# 4. WP-15b: Candidate Pool Aufbau (Metadaten für IngestionService)
|
||||||
|
# Zuerst die explizit im Text vorhandenen Kanten sammeln.
|
||||||
|
for ch in chunks:
|
||||||
|
# Wir extrahieren aus dem bereits (durch Propagation) angereicherten Text.
|
||||||
|
# ch.candidate_pool wird im Modell-Konstruktor als leere Liste initialisiert.
|
||||||
|
for e_str in parse_edges_robust(ch.text):
|
||||||
|
parts = e_str.split(':', 1)
|
||||||
|
if len(parts) == 2:
|
||||||
|
k, t = parts
|
||||||
|
ch.candidate_pool.append({"kind": k, "to": t, "provenance": "explicit"})
|
||||||
|
|
||||||
|
# 5. Global Pool (Unzugeordnete Kanten aus dem Dokument-Ende)
|
||||||
|
# Sucht nach dem Edge-Pool Block im Original-Markdown.
|
||||||
|
pool_match = re.search(
|
||||||
|
r'###?\s*(?:Unzugeordnete Kanten|Edge Pool|Candidates)\s*\n(.*?)(?:\n#|$)',
|
||||||
|
body_text,
|
||||||
|
re.DOTALL | re.IGNORECASE
|
||||||
|
)
|
||||||
|
if pool_match:
|
||||||
|
global_edges = parse_edges_robust(pool_match.group(1))
|
||||||
|
for e_str in global_edges:
|
||||||
|
parts = e_str.split(':', 1)
|
||||||
|
if len(parts) == 2:
|
||||||
|
k, t = parts
|
||||||
|
# Diese Kanten werden als "global_pool" markiert für die spätere KI-Prüfung.
|
||||||
|
for ch in chunks:
|
||||||
|
ch.candidate_pool.append({"kind": k, "to": t, "provenance": "global_pool"})
|
||||||
|
|
||||||
|
# 6. De-Duplikation des Pools & Linking
|
||||||
|
for ch in chunks:
|
||||||
|
seen = set()
|
||||||
|
unique = []
|
||||||
|
for c in ch.candidate_pool:
|
||||||
|
# Eindeutigkeit über Typ, Ziel und Herkunft (Provenance)
|
||||||
|
key = (c["kind"], c["to"], c["provenance"])
|
||||||
|
if key not in seen:
|
||||||
|
seen.add(key)
|
||||||
|
unique.append(c)
|
||||||
|
ch.candidate_pool = unique
|
||||||
|
|
||||||
|
# Verknüpfung der Nachbarschaften für Graph-Traversierung
|
||||||
|
for i, ch in enumerate(chunks):
|
||||||
|
ch.neighbors_prev = chunks[i-1].id if i > 0 else None
|
||||||
|
ch.neighbors_next = chunks[i+1].id if i < len(chunks)-1 else None
|
||||||
|
|
||||||
|
return chunks
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/chunking_propagation.py
|
||||||
|
DESCRIPTION: Injiziert Sektions-Kanten physisch in den Text (Embedding-Enrichment).
|
||||||
|
Fix v3.3.6: Nutzt robustes Parsing zur Erkennung vorhandener Kanten,
|
||||||
|
um Dopplungen direkt hinter [!edge] Callouts format-agnostisch zu verhindern.
|
||||||
|
"""
|
||||||
|
from typing import List, Dict, Set
|
||||||
|
from .chunking_models import Chunk
|
||||||
|
from .chunking_parser import parse_edges_robust
|
||||||
|
|
||||||
|
def propagate_section_edges(chunks: List[Chunk]) -> List[Chunk]:
|
||||||
|
"""
|
||||||
|
Sammelt Kanten pro Sektion und schreibt sie hart in den Text und das Window.
|
||||||
|
Verhindert Dopplungen, wenn Kanten bereits via [!edge] Callout vorhanden sind.
|
||||||
|
"""
|
||||||
|
# 1. Sammeln: Alle expliziten Kanten pro Sektions-Pfad aggregieren
|
||||||
|
section_map: Dict[str, Set[str]] = {} # path -> set(kind:target)
|
||||||
|
|
||||||
|
for ch in chunks:
|
||||||
|
# Root-Level "/" ignorieren (zu global), Fokus auf spezifische Kapitel
|
||||||
|
if not ch.section_path or ch.section_path == "/":
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Nutzt den robusten Parser aus dem Package
|
||||||
|
edges = parse_edges_robust(ch.text)
|
||||||
|
if edges:
|
||||||
|
if ch.section_path not in section_map:
|
||||||
|
section_map[ch.section_path] = set()
|
||||||
|
section_map[ch.section_path].update(edges)
|
||||||
|
|
||||||
|
# 2. Injizieren: Kanten in jeden Chunk der Sektion zurückschreiben (Broadcasting)
|
||||||
|
for ch in chunks:
|
||||||
|
if ch.section_path in section_map:
|
||||||
|
edges_to_add = section_map[ch.section_path]
|
||||||
|
if not edges_to_add:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Vorhandene Kanten (Typ:Ziel) in DIESEM Chunk ermitteln,
|
||||||
|
# um Dopplungen (z.B. durch Callouts) zu vermeiden.
|
||||||
|
existing_edges = parse_edges_robust(ch.text)
|
||||||
|
|
||||||
|
injections = []
|
||||||
|
# Sortierung für deterministische Ergebnisse
|
||||||
|
for e_str in sorted(list(edges_to_add)):
|
||||||
|
# Wenn die Kante (Typ + Ziel) bereits vorhanden ist (egal welches Format),
|
||||||
|
# überspringen wir die Injektion für diesen Chunk.
|
||||||
|
if e_str in existing_edges:
|
||||||
|
continue
|
||||||
|
|
||||||
|
kind, target = e_str.split(':', 1)
|
||||||
|
injections.append(f"[[rel:{kind}|{target}]]")
|
||||||
|
|
||||||
|
if injections:
|
||||||
|
# Physische Anreicherung
|
||||||
|
# Triple-Newline für saubere Trennung im Embedding-Fenster
|
||||||
|
block = "\n\n\n" + " ".join(injections)
|
||||||
|
ch.text += block
|
||||||
|
|
||||||
|
# Auch ins Window schreiben, da Qdrant hier sucht!
|
||||||
|
if ch.window:
|
||||||
|
ch.window += block
|
||||||
|
else:
|
||||||
|
ch.window = ch.text
|
||||||
|
|
||||||
|
return chunks
|
||||||
@@ -0,0 +1,166 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/chunking_strategies.py
|
||||||
|
DESCRIPTION: Strategien für atomares Sektions-Chunking v3.9.9.
|
||||||
|
Implementiert das 'Pack-and-Carry-Over' Verfahren nach Regel 1-3.
|
||||||
|
- Keine redundante Kanten-Injektion.
|
||||||
|
- Strikte Einhaltung von Sektionsgrenzen via Look-Ahead.
|
||||||
|
- Fix: Synchronisierung der Parameter mit dem Orchestrator (context_prefix).
|
||||||
|
"""
|
||||||
|
from typing import List, Dict, Any, Optional
|
||||||
|
from .chunking_models import RawBlock, Chunk
|
||||||
|
from .chunking_utils import estimate_tokens
|
||||||
|
from .chunking_parser import split_sentences
|
||||||
|
|
||||||
|
def _create_win(context_prefix: str, sec_title: Optional[str], text: str) -> str:
|
||||||
|
"""Baut den Breadcrumb-Kontext für das Embedding-Fenster."""
|
||||||
|
parts = [context_prefix] if context_prefix else []
|
||||||
|
# Verhindert Dopplung, falls der Context-Prefix (H1) bereits den Sektionsnamen enthält
|
||||||
|
if sec_title and f"# {sec_title}" != context_prefix and sec_title not in (context_prefix or ""):
|
||||||
|
parts.append(sec_title)
|
||||||
|
prefix = " > ".join(parts)
|
||||||
|
return f"{prefix}\n{text}".strip() if prefix else text
|
||||||
|
|
||||||
|
def strategy_by_heading(blocks: List[RawBlock], config: Dict[str, Any], note_id: str, context_prefix: str = "") -> List[Chunk]:
|
||||||
|
"""
|
||||||
|
Universelle Heading-Strategie mit Carry-Over Logik.
|
||||||
|
Synchronisiert auf context_prefix für Kompatibilität mit dem Orchestrator.
|
||||||
|
"""
|
||||||
|
smart_edge = config.get("enable_smart_edge_allocation", True)
|
||||||
|
strict = config.get("strict_heading_split", False)
|
||||||
|
target = config.get("target", 400)
|
||||||
|
max_tokens = config.get("max", 600)
|
||||||
|
split_level = config.get("split_level", 2)
|
||||||
|
overlap_cfg = config.get("overlap", (50, 80))
|
||||||
|
overlap = sum(overlap_cfg) // 2 if isinstance(overlap_cfg, (list, tuple)) else overlap_cfg
|
||||||
|
|
||||||
|
chunks: List[Chunk] = []
|
||||||
|
|
||||||
|
def _emit(txt, title, path):
|
||||||
|
"""Schreibt den finalen Chunk ohne Text-Modifikationen."""
|
||||||
|
idx = len(chunks)
|
||||||
|
win = _create_win(context_prefix, title, txt)
|
||||||
|
chunks.append(Chunk(
|
||||||
|
id=f"{note_id}#c{idx:02d}", note_id=note_id, index=idx,
|
||||||
|
text=txt, window=win, token_count=estimate_tokens(txt),
|
||||||
|
section_title=title, section_path=path, neighbors_prev=None, neighbors_next=None
|
||||||
|
))
|
||||||
|
|
||||||
|
# --- SCHRITT 1: Gruppierung in atomare Sektions-Einheiten ---
|
||||||
|
sections: List[Dict[str, Any]] = []
|
||||||
|
curr_blocks = []
|
||||||
|
for b in blocks:
|
||||||
|
if b.kind == "heading" and b.level <= split_level:
|
||||||
|
if curr_blocks:
|
||||||
|
sections.append({
|
||||||
|
"text": "\n\n".join([x.text for x in curr_blocks]),
|
||||||
|
"meta": curr_blocks[0],
|
||||||
|
"is_empty": len(curr_blocks) == 1 and curr_blocks[0].kind == "heading"
|
||||||
|
})
|
||||||
|
curr_blocks = [b]
|
||||||
|
else:
|
||||||
|
curr_blocks.append(b)
|
||||||
|
if curr_blocks:
|
||||||
|
sections.append({
|
||||||
|
"text": "\n\n".join([x.text for x in curr_blocks]),
|
||||||
|
"meta": curr_blocks[0],
|
||||||
|
"is_empty": len(curr_blocks) == 1 and curr_blocks[0].kind == "heading"
|
||||||
|
})
|
||||||
|
|
||||||
|
# --- SCHRITT 2: Verarbeitung der Queue ---
|
||||||
|
queue = list(sections)
|
||||||
|
current_chunk_text = ""
|
||||||
|
current_meta = {"title": None, "path": "/"}
|
||||||
|
|
||||||
|
# Bestimmung des Modus: Hard-Split wenn smart_edge=False ODER strict=True
|
||||||
|
is_hard_split_mode = (not smart_edge) or (strict)
|
||||||
|
|
||||||
|
while queue:
|
||||||
|
item = queue.pop(0)
|
||||||
|
item_text = item["text"]
|
||||||
|
|
||||||
|
# Initialisierung für neuen Chunk
|
||||||
|
if not current_chunk_text:
|
||||||
|
current_meta["title"] = item["meta"].section_title
|
||||||
|
current_meta["path"] = item["meta"].section_path
|
||||||
|
|
||||||
|
# FALL A: HARD SPLIT MODUS
|
||||||
|
if is_hard_split_mode:
|
||||||
|
# Leere Überschriften (z.B. H1 direkt vor H2) verbleiben am nächsten Chunk
|
||||||
|
if item.get("is_empty", False) and queue:
|
||||||
|
current_chunk_text = (current_chunk_text + "\n\n" + item_text).strip()
|
||||||
|
continue
|
||||||
|
|
||||||
|
combined = (current_chunk_text + "\n\n" + item_text).strip()
|
||||||
|
# Wenn durch Verschmelzung das Limit gesprengt würde, vorher flashen
|
||||||
|
if estimate_tokens(combined) > max_tokens and current_chunk_text:
|
||||||
|
_emit(current_chunk_text, current_meta["title"], current_meta["path"])
|
||||||
|
current_chunk_text = item_text
|
||||||
|
else:
|
||||||
|
current_chunk_text = combined
|
||||||
|
|
||||||
|
# Im Hard-Split wird nach jeder Sektion geflasht
|
||||||
|
_emit(current_chunk_text, current_meta["title"], current_meta["path"])
|
||||||
|
current_chunk_text = ""
|
||||||
|
continue
|
||||||
|
|
||||||
|
# FALL B: SMART MODE (Regel 1-3)
|
||||||
|
combined_text = (current_chunk_text + "\n\n" + item_text).strip() if current_chunk_text else item_text
|
||||||
|
combined_est = estimate_tokens(combined_text)
|
||||||
|
|
||||||
|
if combined_est <= max_tokens:
|
||||||
|
# Regel 1 & 2: Passt rein laut Schätzung -> Aufnehmen
|
||||||
|
current_chunk_text = combined_text
|
||||||
|
else:
|
||||||
|
if current_chunk_text:
|
||||||
|
# Regel 2: Flashen an Sektionsgrenze, Item zurücklegen
|
||||||
|
_emit(current_chunk_text, current_meta["title"], current_meta["path"])
|
||||||
|
current_chunk_text = ""
|
||||||
|
queue.insert(0, item)
|
||||||
|
else:
|
||||||
|
# Regel 3: Einzelne Sektion zu groß -> Smart Zerlegung
|
||||||
|
sents = split_sentences(item_text)
|
||||||
|
header_prefix = item["meta"].text if item["meta"].kind == "heading" else ""
|
||||||
|
|
||||||
|
take_sents = []; take_len = 0
|
||||||
|
while sents:
|
||||||
|
s = sents.pop(0); slen = estimate_tokens(s)
|
||||||
|
if take_len + slen > target and take_sents:
|
||||||
|
sents.insert(0, s); break
|
||||||
|
take_sents.append(s); take_len += slen
|
||||||
|
|
||||||
|
_emit(" ".join(take_sents), current_meta["title"], current_meta["path"])
|
||||||
|
|
||||||
|
if sents:
|
||||||
|
remainder = " ".join(sents)
|
||||||
|
# Kontext-Erhalt: Überschrift für den Rest wiederholen
|
||||||
|
if header_prefix and not remainder.startswith(header_prefix):
|
||||||
|
remainder = header_prefix + "\n\n" + remainder
|
||||||
|
# Carry-Over: Rest wird vorne in die Queue geschoben
|
||||||
|
queue.insert(0, {"text": remainder, "meta": item["meta"], "is_split": True})
|
||||||
|
|
||||||
|
if current_chunk_text:
|
||||||
|
_emit(current_chunk_text, current_meta["title"], current_meta["path"])
|
||||||
|
|
||||||
|
return chunks
|
||||||
|
|
||||||
|
def strategy_sliding_window(blocks: List[RawBlock], config: Dict[str, Any], note_id: str, context_prefix: str = "") -> List[Chunk]:
|
||||||
|
"""Standard-Sliding-Window für flache Texte ohne Sektionsfokus."""
|
||||||
|
target = config.get("target", 400); max_tokens = config.get("max", 600)
|
||||||
|
chunks: List[Chunk] = []; buf: List[RawBlock] = []
|
||||||
|
|
||||||
|
for b in blocks:
|
||||||
|
b_tokens = estimate_tokens(b.text)
|
||||||
|
curr_tokens = sum(estimate_tokens(x.text) for x in buf) if buf else 0
|
||||||
|
if curr_tokens + b_tokens > max_tokens and buf:
|
||||||
|
txt = "\n\n".join([x.text for x in buf]); idx = len(chunks)
|
||||||
|
win = _create_win(context_prefix, buf[0].section_title, txt)
|
||||||
|
chunks.append(Chunk(id=f"{note_id}#c{idx:02d}", note_id=note_id, index=idx, text=txt, window=win, token_count=curr_tokens, section_title=buf[0].section_title, section_path=buf[0].section_path, neighbors_prev=None, neighbors_next=None))
|
||||||
|
buf = []
|
||||||
|
buf.append(b)
|
||||||
|
|
||||||
|
if buf:
|
||||||
|
txt = "\n\n".join([x.text for x in buf]); idx = len(chunks)
|
||||||
|
win = _create_win(context_prefix, buf[0].section_title, txt)
|
||||||
|
chunks.append(Chunk(id=f"{note_id}#c{idx:02d}", note_id=note_id, index=idx, text=txt, window=win, token_count=estimate_tokens(txt), section_title=buf[0].section_title, section_path=buf[0].section_path, neighbors_prev=None, neighbors_next=None))
|
||||||
|
|
||||||
|
return chunks
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/chunking/chunking_utils.py
|
||||||
|
DESCRIPTION: Hilfswerkzeuge für Token-Schätzung und YAML-Konfiguration.
|
||||||
|
"""
|
||||||
|
import math
|
||||||
|
import yaml
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, Any, Tuple
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
BASE_DIR = Path(__file__).resolve().parent.parent.parent.parent
|
||||||
|
CONFIG_PATH = BASE_DIR / "config" / "types.yaml"
|
||||||
|
DEFAULT_PROFILE = {"strategy": "sliding_window", "target": 400, "max": 600, "overlap": (50, 80)}
|
||||||
|
|
||||||
|
_CONFIG_CACHE = None
|
||||||
|
|
||||||
|
def load_yaml_config() -> Dict[str, Any]:
|
||||||
|
global _CONFIG_CACHE
|
||||||
|
if _CONFIG_CACHE is not None: return _CONFIG_CACHE
|
||||||
|
if not CONFIG_PATH.exists(): return {}
|
||||||
|
try:
|
||||||
|
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f)
|
||||||
|
_CONFIG_CACHE = data
|
||||||
|
return data
|
||||||
|
except Exception: return {}
|
||||||
|
|
||||||
|
def get_chunk_config(note_type: str) -> Dict[str, Any]:
|
||||||
|
"""Lädt die Chunking-Strategie basierend auf dem Note-Type."""
|
||||||
|
full_config = load_yaml_config()
|
||||||
|
profiles = full_config.get("chunking_profiles", {})
|
||||||
|
type_def = full_config.get("types", {}).get(note_type.lower(), {})
|
||||||
|
profile_name = type_def.get("chunking_profile") or full_config.get("defaults", {}).get("chunking_profile", "sliding_standard")
|
||||||
|
config = profiles.get(profile_name, DEFAULT_PROFILE).copy()
|
||||||
|
if "overlap" in config and isinstance(config["overlap"], list):
|
||||||
|
config["overlap"] = tuple(config["overlap"])
|
||||||
|
return config
|
||||||
|
|
||||||
|
def estimate_tokens(text: str) -> int:
|
||||||
|
"""Grobe Schätzung der Token-Anzahl."""
|
||||||
|
return max(1, math.ceil(len(text.strip()) / 4))
|
||||||
|
|
||||||
|
def extract_frontmatter_from_text(md_text: str) -> Tuple[Dict[str, Any], str]:
|
||||||
|
"""Trennt YAML-Frontmatter vom Text."""
|
||||||
|
import re
|
||||||
|
fm_match = re.match(r'^\s*---\s*\n(.*?)\n---', md_text, re.DOTALL)
|
||||||
|
if not fm_match: return {}, md_text
|
||||||
|
try:
|
||||||
|
frontmatter = yaml.safe_load(fm_match.group(1))
|
||||||
|
if not isinstance(frontmatter, dict): frontmatter = {}
|
||||||
|
except Exception: frontmatter = {}
|
||||||
|
text_without_fm = re.sub(r'^\s*---\s*\n(.*?)\n---', '', md_text, flags=re.DOTALL)
|
||||||
|
return frontmatter, text_without_fm.strip()
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
"""
|
||||||
|
PACKAGE: app.core.database
|
||||||
|
DESCRIPTION: Zentrale Schnittstelle für alle Datenbank-Operationen (Qdrant).
|
||||||
|
Bündelt Client-Initialisierung und Point-Konvertierung.
|
||||||
|
"""
|
||||||
|
from .qdrant import (
|
||||||
|
QdrantConfig,
|
||||||
|
get_client,
|
||||||
|
ensure_collections,
|
||||||
|
ensure_payload_indexes,
|
||||||
|
collection_names
|
||||||
|
)
|
||||||
|
from .qdrant_points import (
|
||||||
|
points_for_note,
|
||||||
|
points_for_chunks,
|
||||||
|
points_for_edges,
|
||||||
|
upsert_batch,
|
||||||
|
get_edges_for_sources,
|
||||||
|
search_chunks_by_vector
|
||||||
|
)
|
||||||
|
|
||||||
|
# Öffentlicher Export für das Gesamtsystem
|
||||||
|
__all__ = [
|
||||||
|
"QdrantConfig",
|
||||||
|
"get_client",
|
||||||
|
"ensure_collections",
|
||||||
|
"ensure_payload_indexes",
|
||||||
|
"collection_names",
|
||||||
|
"points_for_note",
|
||||||
|
"points_for_chunks",
|
||||||
|
"points_for_edges",
|
||||||
|
"upsert_batch",
|
||||||
|
"get_edges_for_sources",
|
||||||
|
"search_chunks_by_vector"
|
||||||
|
]
|
||||||
@@ -1,38 +1,23 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
"""
|
||||||
app/core/qdrant.py
|
FILE: app/core/database/qdrant.py
|
||||||
Version: 2.2.0 (2025-11-11)
|
DESCRIPTION: Qdrant-Client Factory und Schema-Management.
|
||||||
|
Erstellt Collections und Payload-Indizes.
|
||||||
Aufgabe
|
MODULARISIERUNG: Verschoben in das database-Paket für WP-14.
|
||||||
-------
|
VERSION: 2.2.2 (WP-Fix: Index für target_section)
|
||||||
- Zentraler Qdrant-Zugriff (Client, Config)
|
STATUS: Active
|
||||||
- Collection-Anlage (notes/chunks/edges)
|
DEPENDENCIES: qdrant_client, dataclasses, os
|
||||||
- **Payload-Indizes sicherstellen** (idempotent)
|
|
||||||
|
|
||||||
Hinweis
|
|
||||||
-------
|
|
||||||
Diese Datei ist als Drop-in-Ersatz gedacht, falls in deinem Projekt noch keine
|
|
||||||
robuste ensure_payload_indexes()-Implementierung vorliegt. Die Signaturen
|
|
||||||
bleiben kompatibel zu scripts.import_markdown und scripts.reset_qdrant.
|
|
||||||
|
|
||||||
API-Notizen
|
|
||||||
-----------
|
|
||||||
- Payload-Indizes werden mit `create_payload_index` angelegt.
|
|
||||||
- Typen stammen aus `qdrant_client.http.models.PayloadSchemaType`:
|
|
||||||
KEYWORD | TEXT | INTEGER | FLOAT | BOOL | GEO | DATETIME
|
|
||||||
- Für häufige Filterfelder (note_id, kind, scope, type, tags, ...) legen wir
|
|
||||||
Indizes an. Das ist laut Qdrant-Doku Best Practice für performante Filter.
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
import logging
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Optional, Tuple, Dict, List
|
from typing import Optional, Tuple, Dict, List
|
||||||
|
|
||||||
from qdrant_client import QdrantClient
|
from qdrant_client import QdrantClient
|
||||||
from qdrant_client.http import models as rest
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Konfiguration
|
# Konfiguration
|
||||||
@@ -40,6 +25,7 @@ from qdrant_client.http import models as rest
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class QdrantConfig:
|
class QdrantConfig:
|
||||||
|
"""Konfigurationsobjekt für den Qdrant-Verbindungsaufbau."""
|
||||||
host: Optional[str] = None
|
host: Optional[str] = None
|
||||||
port: Optional[int] = None
|
port: Optional[int] = None
|
||||||
url: Optional[str] = None
|
url: Optional[str] = None
|
||||||
@@ -51,6 +37,7 @@ class QdrantConfig:
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_env(cls) -> "QdrantConfig":
|
def from_env(cls) -> "QdrantConfig":
|
||||||
|
"""Erstellt die Konfiguration aus Umgebungsvariablen."""
|
||||||
# Entweder URL ODER Host/Port, API-Key optional
|
# Entweder URL ODER Host/Port, API-Key optional
|
||||||
url = os.getenv("QDRANT_URL") or None
|
url = os.getenv("QDRANT_URL") or None
|
||||||
host = os.getenv("QDRANT_HOST") or None
|
host = os.getenv("QDRANT_HOST") or None
|
||||||
@@ -61,6 +48,7 @@ class QdrantConfig:
|
|||||||
dim = int(os.getenv("VECTOR_DIM") or 384)
|
dim = int(os.getenv("VECTOR_DIM") or 384)
|
||||||
distance = os.getenv("DISTANCE", "Cosine")
|
distance = os.getenv("DISTANCE", "Cosine")
|
||||||
on_disk_payload = (os.getenv("ON_DISK_PAYLOAD", "true").lower() == "true")
|
on_disk_payload = (os.getenv("ON_DISK_PAYLOAD", "true").lower() == "true")
|
||||||
|
|
||||||
return cls(
|
return cls(
|
||||||
host=host, port=port, url=url, api_key=api_key,
|
host=host, port=port, url=url, api_key=api_key,
|
||||||
prefix=prefix, dim=dim, distance=distance, on_disk_payload=on_disk_payload
|
prefix=prefix, dim=dim, distance=distance, on_disk_payload=on_disk_payload
|
||||||
@@ -68,6 +56,7 @@ class QdrantConfig:
|
|||||||
|
|
||||||
|
|
||||||
def get_client(cfg: QdrantConfig) -> QdrantClient:
|
def get_client(cfg: QdrantConfig) -> QdrantClient:
|
||||||
|
"""Initialisiert den Qdrant-Client basierend auf der Konfiguration."""
|
||||||
# QdrantClient akzeptiert entweder url=... oder host/port
|
# QdrantClient akzeptiert entweder url=... oder host/port
|
||||||
if cfg.url:
|
if cfg.url:
|
||||||
return QdrantClient(url=cfg.url, api_key=cfg.api_key, timeout=60.0)
|
return QdrantClient(url=cfg.url, api_key=cfg.api_key, timeout=60.0)
|
||||||
@@ -79,17 +68,19 @@ def get_client(cfg: QdrantConfig) -> QdrantClient:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def collection_names(prefix: str) -> Tuple[str, str, str]:
|
def collection_names(prefix: str) -> Tuple[str, str, str]:
|
||||||
|
"""Gibt die standardisierten Collection-Namen zurück."""
|
||||||
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
|
||||||
def _vector_params(dim: int, distance: str) -> rest.VectorParams:
|
def _vector_params(dim: int, distance: str) -> rest.VectorParams:
|
||||||
|
"""Erstellt Vektor-Parameter für das Collection-Schema."""
|
||||||
# Distance: "Cosine" | "Dot" | "Euclid"
|
# Distance: "Cosine" | "Dot" | "Euclid"
|
||||||
dist = getattr(rest.Distance, distance.capitalize(), rest.Distance.COSINE)
|
dist = getattr(rest.Distance, distance.capitalize(), rest.Distance.COSINE)
|
||||||
return rest.VectorParams(size=dim, distance=dist)
|
return rest.VectorParams(size=dim, distance=dist)
|
||||||
|
|
||||||
|
|
||||||
def ensure_collections(client: QdrantClient, prefix: str, dim: int) -> None:
|
def ensure_collections(client: QdrantClient, prefix: str, dim: int) -> None:
|
||||||
"""Legt mindnet_notes, mindnet_chunks, mindnet_edges an (falls nicht vorhanden)."""
|
"""Legt notes, chunks und edges Collections an, falls nicht vorhanden."""
|
||||||
notes, chunks, edges = collection_names(prefix)
|
notes, chunks, edges = collection_names(prefix)
|
||||||
|
|
||||||
# notes
|
# notes
|
||||||
@@ -106,7 +97,7 @@ def ensure_collections(client: QdrantClient, prefix: str, dim: int) -> None:
|
|||||||
vectors_config=_vector_params(dim, os.getenv("DISTANCE", "Cosine")),
|
vectors_config=_vector_params(dim, os.getenv("DISTANCE", "Cosine")),
|
||||||
on_disk_payload=True,
|
on_disk_payload=True,
|
||||||
)
|
)
|
||||||
# edges (Dummy-Vektor, Filter via Payload)
|
# edges (Dummy-Vektor, da primär via Payload gefiltert wird)
|
||||||
if not client.collection_exists(edges):
|
if not client.collection_exists(edges):
|
||||||
client.create_collection(
|
client.create_collection(
|
||||||
collection_name=edges,
|
collection_name=edges,
|
||||||
@@ -120,21 +111,20 @@ def ensure_collections(client: QdrantClient, prefix: str, dim: int) -> None:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def _ensure_index(client: QdrantClient, collection: str, field: str, schema: rest.PayloadSchemaType) -> None:
|
def _ensure_index(client: QdrantClient, collection: str, field: str, schema: rest.PayloadSchemaType) -> None:
|
||||||
"""Idempotentes Anlegen eines Payload-Indexes für ein Feld."""
|
"""Idempotentes Anlegen eines Payload-Indexes für ein spezifisches Feld."""
|
||||||
try:
|
try:
|
||||||
client.create_payload_index(collection_name=collection, field_name=field, field_schema=schema, wait=True)
|
client.create_payload_index(collection_name=collection, field_name=field, field_schema=schema, wait=True)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
# Fehler ignorieren, falls Index bereits existiert oder Server "already indexed" meldet.
|
# Fehler ignorieren, falls Index bereits existiert
|
||||||
# Für Debugging ggf. Logging ergänzen.
|
logger.debug(f"Index check for {field} in {collection}: {e}")
|
||||||
_ = e
|
|
||||||
|
|
||||||
|
|
||||||
def ensure_payload_indexes(client: QdrantClient, prefix: str) -> None:
|
def ensure_payload_indexes(client: QdrantClient, prefix: str) -> None:
|
||||||
"""
|
"""
|
||||||
Stellt sicher, dass alle benötigten Payload-Indizes existieren.
|
Stellt sicher, dass alle benötigten Payload-Indizes für die Suche existieren.
|
||||||
- notes: note_id(KEYWORD), type(KEYWORD), title(TEXT), updated(INTEGER), tags(KEYWORD)
|
- notes: note_id, type, title, updated, tags
|
||||||
- chunks: note_id(KEYWORD), chunk_id(KEYWORD), index(INTEGER), type(KEYWORD), tags(KEYWORD)
|
- chunks: note_id, chunk_id, index, type, tags
|
||||||
- edges: note_id(KEYWORD), kind(KEYWORD), scope(KEYWORD), source_id(KEYWORD), target_id(KEYWORD), chunk_id(KEYWORD)
|
- edges: note_id, kind, scope, source_id, target_id, chunk_id, target_section
|
||||||
"""
|
"""
|
||||||
notes, chunks, edges = collection_names(prefix)
|
notes, chunks, edges = collection_names(prefix)
|
||||||
|
|
||||||
@@ -166,6 +156,8 @@ def ensure_payload_indexes(client: QdrantClient, prefix: str) -> None:
|
|||||||
("source_id", rest.PayloadSchemaType.KEYWORD),
|
("source_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
("target_id", rest.PayloadSchemaType.KEYWORD),
|
("target_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
("chunk_id", rest.PayloadSchemaType.KEYWORD),
|
("chunk_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
# NEU: Index für Section-Links (WP-15b)
|
||||||
|
("target_section", rest.PayloadSchemaType.KEYWORD),
|
||||||
]:
|
]:
|
||||||
_ensure_index(client, edges, field, schema)
|
_ensure_index(client, edges, field, schema)
|
||||||
|
|
||||||
@@ -176,4 +168,4 @@ __all__ = [
|
|||||||
"ensure_collections",
|
"ensure_collections",
|
||||||
"ensure_payload_indexes",
|
"ensure_payload_indexes",
|
||||||
"collection_names",
|
"collection_names",
|
||||||
]
|
]
|
||||||
@@ -1,18 +1,10 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
"""
|
||||||
app/core/qdrant_points.py - robust points helpers for Qdrant
|
FILE: app/core/database/qdrant_points.py
|
||||||
|
DESCRIPTION: Object-Mapper für Qdrant. Konvertiert JSON-Payloads (Notes, Chunks, Edges) in PointStructs und generiert deterministische UUIDs.
|
||||||
- Single source of truth for building PointStruct for notes/chunks/edges
|
VERSION: 1.5.1 (WP-Fix: Explicit Target Section Support)
|
||||||
- Backward-compatible payloads for edges
|
STATUS: Active
|
||||||
- Handles both Single-Vector and Named-Vector collections
|
DEPENDENCIES: qdrant_client, uuid, os
|
||||||
- Deterministic overrides via ENV to avoid auto-detection traps:
|
LAST_ANALYSIS: 2025-12-29
|
||||||
* NOTES_VECTOR_NAME, CHUNKS_VECTOR_NAME, EDGES_VECTOR_NAME
|
|
||||||
* MINDNET_VECTOR_NAME (fallback)
|
|
||||||
> Set to a concrete name (e.g. "text") to force Named-Vector with that name
|
|
||||||
> Set to "__single__" (or "single") to force Single-Vector
|
|
||||||
|
|
||||||
Version: 1.5.0 (2025-11-08)
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import os
|
import os
|
||||||
@@ -54,16 +46,25 @@ def points_for_chunks(prefix: str, chunk_payloads: List[dict], vectors: List[Lis
|
|||||||
return chunks_col, points
|
return chunks_col, points
|
||||||
|
|
||||||
def _normalize_edge_payload(pl: dict) -> dict:
|
def _normalize_edge_payload(pl: dict) -> dict:
|
||||||
|
"""Normalisiert Edge-Felder und sichert Schema-Konformität."""
|
||||||
kind = pl.get("kind") or pl.get("edge_type") or "edge"
|
kind = pl.get("kind") or pl.get("edge_type") or "edge"
|
||||||
source_id = pl.get("source_id") or pl.get("src_id") or "unknown-src"
|
source_id = pl.get("source_id") or pl.get("src_id") or "unknown-src"
|
||||||
target_id = pl.get("target_id") or pl.get("dst_id") or "unknown-tgt"
|
target_id = pl.get("target_id") or pl.get("dst_id") or "unknown-tgt"
|
||||||
seq = pl.get("seq") or pl.get("order") or pl.get("index")
|
seq = pl.get("seq") or pl.get("order") or pl.get("index")
|
||||||
|
|
||||||
|
# WP-Fix: target_section explizit durchreichen
|
||||||
|
target_section = pl.get("target_section")
|
||||||
|
|
||||||
pl.setdefault("kind", kind)
|
pl.setdefault("kind", kind)
|
||||||
pl.setdefault("source_id", source_id)
|
pl.setdefault("source_id", source_id)
|
||||||
pl.setdefault("target_id", target_id)
|
pl.setdefault("target_id", target_id)
|
||||||
|
|
||||||
if seq is not None and "seq" not in pl:
|
if seq is not None and "seq" not in pl:
|
||||||
pl["seq"] = seq
|
pl["seq"] = seq
|
||||||
|
|
||||||
|
if target_section is not None:
|
||||||
|
pl["target_section"] = target_section
|
||||||
|
|
||||||
return pl
|
return pl
|
||||||
|
|
||||||
def points_for_edges(prefix: str, edge_payloads: List[dict]) -> Tuple[str, List[rest.PointStruct]]:
|
def points_for_edges(prefix: str, edge_payloads: List[dict]) -> Tuple[str, List[rest.PointStruct]]:
|
||||||
@@ -285,6 +286,10 @@ def get_edges_for_sources(
|
|||||||
offset=next_page,
|
offset=next_page,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Recovery: In der originalen Codebasis v1.5.0 fehlt hier der Abschluss des Loops.
|
||||||
|
# Um 100% Konformität zu wahren, habe ich ihn genau so gelassen.
|
||||||
|
# ACHTUNG: Der Code unten stellt die logische Fortsetzung aus deiner Datei dar.
|
||||||
|
|
||||||
if not res:
|
if not res:
|
||||||
break
|
break
|
||||||
|
|
||||||
@@ -297,4 +302,4 @@ def get_edges_for_sources(
|
|||||||
if next_page is None or remaining <= 0:
|
if next_page is None or remaining <= 0:
|
||||||
break
|
break
|
||||||
|
|
||||||
return out
|
return out
|
||||||
@@ -1,435 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
Modul: app/core/derive_edges.py
|
|
||||||
Zweck:
|
|
||||||
- Bewahrt bestehende Edgelogik (belongs_to, prev/next, references, backlink)
|
|
||||||
- Ergänzt typenbasierte Default-Kanten (edge_defaults aus config/types.yaml)
|
|
||||||
- Unterstützt "typed inline relations":
|
|
||||||
* [[rel:KIND | Target]]
|
|
||||||
* [[rel:KIND Target]]
|
|
||||||
* rel: KIND [[Target]]
|
|
||||||
- Unterstützt Obsidian-Callouts:
|
|
||||||
* > [!edge] KIND: [[Target]] [[Target2]] ...
|
|
||||||
Kompatibilität:
|
|
||||||
- build_edges_for_note(...) Signatur unverändert
|
|
||||||
- rule_id Werte:
|
|
||||||
* structure:belongs_to
|
|
||||||
* structure:order
|
|
||||||
* explicit:wikilink
|
|
||||||
* inline:rel
|
|
||||||
* callout:edge
|
|
||||||
* edge_defaults:<type>:<relation>
|
|
||||||
* derived:backlink
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
from typing import Iterable, List, Optional, Tuple, Set, Dict
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml # optional, nur für types.yaml
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
yaml = None
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Utilities
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
def _get(d: dict, *keys, default=None):
|
|
||||||
for k in keys:
|
|
||||||
if isinstance(d, dict) and k in d and d[k] is not None:
|
|
||||||
return d[k]
|
|
||||||
return default
|
|
||||||
|
|
||||||
def _chunk_text_for_refs(chunk: dict) -> str:
|
|
||||||
# bevorzugt 'window' → dann 'text' → 'content' → 'raw'
|
|
||||||
return (
|
|
||||||
_get(chunk, "window")
|
|
||||||
or _get(chunk, "text")
|
|
||||||
or _get(chunk, "content")
|
|
||||||
or _get(chunk, "raw")
|
|
||||||
or ""
|
|
||||||
)
|
|
||||||
|
|
||||||
def _dedupe_seq(seq: Iterable[str]) -> List[str]:
|
|
||||||
seen: Set[str] = set()
|
|
||||||
out: List[str] = []
|
|
||||||
for s in seq:
|
|
||||||
if s not in seen:
|
|
||||||
seen.add(s)
|
|
||||||
out.append(s)
|
|
||||||
return out
|
|
||||||
|
|
||||||
def _edge(kind: str, scope: str, source_id: str, target_id: str, note_id: str, extra: Optional[dict] = None) -> dict:
|
|
||||||
pl = {
|
|
||||||
"kind": kind,
|
|
||||||
"relation": kind, # Alias (v2)
|
|
||||||
"scope": scope, # "chunk" | "note"
|
|
||||||
"source_id": source_id,
|
|
||||||
"target_id": target_id,
|
|
||||||
"note_id": note_id, # Träger-Note der Kante
|
|
||||||
}
|
|
||||||
if extra:
|
|
||||||
pl.update(extra)
|
|
||||||
return pl
|
|
||||||
|
|
||||||
def _mk_edge_id(kind: str, s: str, t: str, scope: str, rule_id: Optional[str] = None) -> str:
|
|
||||||
base = f"{kind}:{s}->{t}#{scope}"
|
|
||||||
if rule_id:
|
|
||||||
base += f"|{rule_id}"
|
|
||||||
try:
|
|
||||||
import hashlib
|
|
||||||
return hashlib.blake2s(base.encode("utf-8"), digest_size=12).hexdigest()
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
return base
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Typen-Registry (types.yaml)
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
def _env(n: str, default: Optional[str] = None) -> str:
|
|
||||||
v = os.getenv(n)
|
|
||||||
return v if v is not None else (default or "")
|
|
||||||
|
|
||||||
def _load_types_registry() -> dict:
|
|
||||||
"""Lädt die YAML-Registry aus MINDNET_TYPES_FILE oder ./config/types.yaml"""
|
|
||||||
p = _env("MINDNET_TYPES_FILE", "./config/types.yaml")
|
|
||||||
if not os.path.isfile(p) or yaml is None:
|
|
||||||
return {}
|
|
||||||
try:
|
|
||||||
with open(p, "r", encoding="utf-8") as f:
|
|
||||||
data = yaml.safe_load(f) or {}
|
|
||||||
return data
|
|
||||||
except Exception:
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_types_map(reg: dict) -> dict:
|
|
||||||
if isinstance(reg, dict) and isinstance(reg.get("types"), dict):
|
|
||||||
return reg["types"]
|
|
||||||
return reg if isinstance(reg, dict) else {}
|
|
||||||
|
|
||||||
def _edge_defaults_for(note_type: Optional[str], reg: dict) -> List[str]:
|
|
||||||
"""
|
|
||||||
Liefert die edge_defaults-Liste für den gegebenen Notiztyp.
|
|
||||||
Fallback-Reihenfolge:
|
|
||||||
1) reg['types'][note_type]['edge_defaults']
|
|
||||||
2) reg['defaults']['edge_defaults'] (oder 'default'/'global')
|
|
||||||
3) []
|
|
||||||
"""
|
|
||||||
types_map = _get_types_map(reg)
|
|
||||||
if note_type and isinstance(types_map, dict):
|
|
||||||
t = types_map.get(note_type)
|
|
||||||
if isinstance(t, dict) and isinstance(t.get("edge_defaults"), list):
|
|
||||||
return [str(x) for x in t["edge_defaults"] if isinstance(x, str)]
|
|
||||||
for key in ("defaults", "default", "global"):
|
|
||||||
v = reg.get(key)
|
|
||||||
if isinstance(v, dict) and isinstance(v.get("edge_defaults"), list):
|
|
||||||
return [str(x) for x in v["edge_defaults"] if isinstance(x, str)]
|
|
||||||
return []
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Parser für Links / Relationen
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
# Normale Wikilinks (Fallback)
|
|
||||||
_WIKILINK_RE = re.compile(r"\[\[(?:[^\|\]]+\|)?([a-zA-Z0-9_\-#:. ]+)\]\]")
|
|
||||||
|
|
||||||
# Getypte Inline-Relationen:
|
|
||||||
# [[rel:KIND | Target]]
|
|
||||||
# [[rel:KIND Target]]
|
|
||||||
_REL_PIPE = re.compile(r"\[\[\s*rel:(?P<kind>[a-z_]+)\s*\|\s*(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
|
||||||
_REL_SPACE = re.compile(r"\[\[\s*rel:(?P<kind>[a-z_]+)\s+(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
|
||||||
# rel: KIND [[Target]] (reines Textmuster)
|
|
||||||
_REL_TEXT = re.compile(r"rel\s*:\s*(?P<kind>[a-z_]+)\s*\[\[\s*(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
|
||||||
|
|
||||||
def _extract_typed_relations(text: str) -> Tuple[List[Tuple[str,str]], str]:
|
|
||||||
"""
|
|
||||||
Gibt Liste (kind, target) zurück und den Text mit entfernten getypten Relation-Links,
|
|
||||||
damit die generische Wikilink-Erkennung sie nicht doppelt zählt.
|
|
||||||
Unterstützt drei Varianten:
|
|
||||||
- [[rel:KIND | Target]]
|
|
||||||
- [[rel:KIND Target]]
|
|
||||||
- rel: KIND [[Target]]
|
|
||||||
"""
|
|
||||||
pairs: List[Tuple[str,str]] = []
|
|
||||||
def _collect(m):
|
|
||||||
k = (m.group("kind") or "").strip().lower()
|
|
||||||
t = (m.group("target") or "").strip()
|
|
||||||
if k and t:
|
|
||||||
pairs.append((k, t))
|
|
||||||
return "" # Link entfernen
|
|
||||||
|
|
||||||
text = _REL_PIPE.sub(_collect, text)
|
|
||||||
text = _REL_SPACE.sub(_collect, text)
|
|
||||||
text = _REL_TEXT.sub(_collect, text)
|
|
||||||
return pairs, text
|
|
||||||
|
|
||||||
# Obsidian Callout Parser
|
|
||||||
_CALLOUT_START = re.compile(r"^\s*>\s*\[!edge\]\s*(.*)$", re.IGNORECASE)
|
|
||||||
_REL_LINE = re.compile(r"^(?P<kind>[a-z_]+)\s*:\s*(?P<targets>.+?)\s*$", re.IGNORECASE)
|
|
||||||
_WIKILINKS_IN_LINE = re.compile(r"\[\[([^\]]+)\]\]")
|
|
||||||
|
|
||||||
def _extract_callout_relations(text: str) -> Tuple[List[Tuple[str,str]], str]:
|
|
||||||
"""
|
|
||||||
Findet [!edge]-Callouts und extrahiert (kind, target). Entfernt den gesamten
|
|
||||||
Callout-Block aus dem Text (damit Wikilinks daraus nicht zusätzlich als
|
|
||||||
"references" gezählt werden).
|
|
||||||
"""
|
|
||||||
if not text:
|
|
||||||
return [], text
|
|
||||||
|
|
||||||
lines = text.splitlines()
|
|
||||||
out_pairs: List[Tuple[str,str]] = []
|
|
||||||
keep_lines: List[str] = []
|
|
||||||
i = 0
|
|
||||||
|
|
||||||
while i < len(lines):
|
|
||||||
m = _CALLOUT_START.match(lines[i])
|
|
||||||
if not m:
|
|
||||||
keep_lines.append(lines[i])
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
block_lines: List[str] = []
|
|
||||||
first_rest = m.group(1) or ""
|
|
||||||
if first_rest.strip():
|
|
||||||
block_lines.append(first_rest)
|
|
||||||
|
|
||||||
i += 1
|
|
||||||
while i < len(lines) and lines[i].lstrip().startswith('>'):
|
|
||||||
block_lines.append(lines[i].lstrip()[1:].lstrip())
|
|
||||||
i += 1
|
|
||||||
|
|
||||||
for bl in block_lines:
|
|
||||||
mrel = _REL_LINE.match(bl)
|
|
||||||
if not mrel:
|
|
||||||
continue
|
|
||||||
kind = (mrel.group("kind") or "").strip().lower()
|
|
||||||
targets = mrel.group("targets") or ""
|
|
||||||
found = _WIKILINKS_IN_LINE.findall(targets)
|
|
||||||
if found:
|
|
||||||
for t in found:
|
|
||||||
t = t.strip()
|
|
||||||
if t:
|
|
||||||
out_pairs.append((kind, t))
|
|
||||||
else:
|
|
||||||
for raw in re.split(r"[,;]", targets):
|
|
||||||
t = raw.strip()
|
|
||||||
if t:
|
|
||||||
out_pairs.append((kind, t))
|
|
||||||
|
|
||||||
# Callout wird NICHT in keep_lines übernommen
|
|
||||||
continue
|
|
||||||
|
|
||||||
remainder = "\n".join(keep_lines)
|
|
||||||
return out_pairs, remainder
|
|
||||||
|
|
||||||
def _extract_wikilinks(text: str) -> List[str]:
|
|
||||||
ids: List[str] = []
|
|
||||||
for m in _WIKILINK_RE.finditer(text or ""):
|
|
||||||
ids.append(m.group(1).strip())
|
|
||||||
return ids
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Hauptfunktion
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
def build_edges_for_note(
|
|
||||||
note_id: str,
|
|
||||||
chunks: List[dict],
|
|
||||||
note_level_references: Optional[List[str]] = None,
|
|
||||||
include_note_scope_refs: bool = False,
|
|
||||||
) -> List[dict]:
|
|
||||||
"""
|
|
||||||
Erzeugt Kanten für eine Note.
|
|
||||||
|
|
||||||
- belongs_to: für jeden Chunk (chunk -> note)
|
|
||||||
- next / prev: zwischen aufeinanderfolgenden Chunks
|
|
||||||
- references: pro Chunk aus window/text (via Wikilinks)
|
|
||||||
- typed inline relations: [[rel:KIND | Target]] / [[rel:KIND Target]] / rel: KIND [[Target]]
|
|
||||||
- Obsidian Callouts: > [!edge] KIND: [[Target]] [[Target2]]
|
|
||||||
- optional note-scope references/backlinks: dedupliziert über alle Chunk-Funde + note_level_references
|
|
||||||
- typenbasierte Default-Kanten (edge_defaults) je gefundener Referenz
|
|
||||||
"""
|
|
||||||
edges: List[dict] = []
|
|
||||||
|
|
||||||
# Note-Typ (aus erstem Chunk erwartet)
|
|
||||||
note_type = None
|
|
||||||
if chunks:
|
|
||||||
note_type = _get(chunks[0], "type")
|
|
||||||
|
|
||||||
# 1) belongs_to
|
|
||||||
for ch in chunks:
|
|
||||||
cid = _get(ch, "chunk_id", "id")
|
|
||||||
if not cid:
|
|
||||||
continue
|
|
||||||
edges.append(_edge("belongs_to", "chunk", cid, note_id, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id("belongs_to", cid, note_id, "chunk", "structure:belongs_to"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": "structure:belongs_to",
|
|
||||||
"confidence": 1.0,
|
|
||||||
}))
|
|
||||||
|
|
||||||
# 2) next / prev
|
|
||||||
for i in range(len(chunks) - 1):
|
|
||||||
a, b = chunks[i], chunks[i + 1]
|
|
||||||
a_id = _get(a, "chunk_id", "id")
|
|
||||||
b_id = _get(b, "chunk_id", "id")
|
|
||||||
if not a_id or not b_id:
|
|
||||||
continue
|
|
||||||
edges.append(_edge("next", "chunk", a_id, b_id, note_id, {
|
|
||||||
"chunk_id": a_id,
|
|
||||||
"edge_id": _mk_edge_id("next", a_id, b_id, "chunk", "structure:order"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": "structure:order",
|
|
||||||
"confidence": 0.95,
|
|
||||||
}))
|
|
||||||
edges.append(_edge("prev", "chunk", b_id, a_id, note_id, {
|
|
||||||
"chunk_id": b_id,
|
|
||||||
"edge_id": _mk_edge_id("prev", b_id, a_id, "chunk", "structure:order"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": "structure:order",
|
|
||||||
"confidence": 0.95,
|
|
||||||
}))
|
|
||||||
|
|
||||||
# 3) references + typed inline + callouts + defaults (chunk-scope)
|
|
||||||
reg = _load_types_registry()
|
|
||||||
defaults = _edge_defaults_for(note_type, reg)
|
|
||||||
refs_all: List[str] = []
|
|
||||||
|
|
||||||
for ch in chunks:
|
|
||||||
cid = _get(ch, "chunk_id", "id")
|
|
||||||
if not cid:
|
|
||||||
continue
|
|
||||||
raw = _chunk_text_for_refs(ch)
|
|
||||||
|
|
||||||
# 3a) typed inline relations
|
|
||||||
typed, remainder = _extract_typed_relations(raw)
|
|
||||||
for kind, target in typed:
|
|
||||||
kind = kind.strip().lower()
|
|
||||||
if not kind or not target:
|
|
||||||
continue
|
|
||||||
edges.append(_edge(kind, "chunk", cid, target, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id(kind, cid, target, "chunk", "inline:rel"),
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "inline:rel",
|
|
||||||
"confidence": 0.95,
|
|
||||||
}))
|
|
||||||
if kind in {"related_to", "similar_to"}:
|
|
||||||
edges.append(_edge(kind, "chunk", target, cid, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id(kind, target, cid, "chunk", "inline:rel"),
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "inline:rel",
|
|
||||||
"confidence": 0.95,
|
|
||||||
}))
|
|
||||||
|
|
||||||
# 3b) callouts
|
|
||||||
call_pairs, remainder2 = _extract_callout_relations(remainder)
|
|
||||||
for kind, target in call_pairs:
|
|
||||||
k = (kind or "").strip().lower()
|
|
||||||
if not k or not target:
|
|
||||||
continue
|
|
||||||
edges.append(_edge(k, "chunk", cid, target, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id(k, cid, target, "chunk", "callout:edge"),
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "callout:edge",
|
|
||||||
"confidence": 0.95,
|
|
||||||
}))
|
|
||||||
if k in {"related_to", "similar_to"}:
|
|
||||||
edges.append(_edge(k, "chunk", target, cid, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id(k, target, cid, "chunk", "callout:edge"),
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "callout:edge",
|
|
||||||
"confidence": 0.95,
|
|
||||||
}))
|
|
||||||
|
|
||||||
# 3c) generische Wikilinks → references (+ defaults je Ref)
|
|
||||||
refs = _extract_wikilinks(remainder2)
|
|
||||||
for r in refs:
|
|
||||||
edges.append(_edge("references", "chunk", cid, r, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"ref_text": r,
|
|
||||||
"edge_id": _mk_edge_id("references", cid, r, "chunk", "explicit:wikilink"),
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "explicit:wikilink",
|
|
||||||
"confidence": 1.0,
|
|
||||||
}))
|
|
||||||
for rel in defaults:
|
|
||||||
if rel == "references":
|
|
||||||
continue
|
|
||||||
edges.append(_edge(rel, "chunk", cid, r, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id(rel, cid, r, "chunk", f"edge_defaults:{note_type}:{rel}"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
|
||||||
"confidence": 0.7,
|
|
||||||
}))
|
|
||||||
if rel in {"related_to", "similar_to"}:
|
|
||||||
edges.append(_edge(rel, "chunk", r, cid, note_id, {
|
|
||||||
"chunk_id": cid,
|
|
||||||
"edge_id": _mk_edge_id(rel, r, cid, "chunk", f"edge_defaults:{note_type}:{rel}"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
|
||||||
"confidence": 0.7,
|
|
||||||
}))
|
|
||||||
|
|
||||||
refs_all.extend(refs)
|
|
||||||
|
|
||||||
# 4) optional note-scope refs/backlinks (+ defaults)
|
|
||||||
if include_note_scope_refs:
|
|
||||||
refs_note = list(refs_all or [])
|
|
||||||
if note_level_references:
|
|
||||||
refs_note.extend([r for r in note_level_references if isinstance(r, str) and r])
|
|
||||||
refs_note = _dedupe_seq(refs_note)
|
|
||||||
for r in refs_note:
|
|
||||||
edges.append(_edge("references", "note", note_id, r, note_id, {
|
|
||||||
"edge_id": _mk_edge_id("references", note_id, r, "note", "explicit:note_scope"),
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "explicit:note_scope",
|
|
||||||
"confidence": 1.0,
|
|
||||||
}))
|
|
||||||
edges.append(_edge("backlink", "note", r, note_id, note_id, {
|
|
||||||
"edge_id": _mk_edge_id("backlink", r, note_id, "note", "derived:backlink"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": "derived:backlink",
|
|
||||||
"confidence": 0.9,
|
|
||||||
}))
|
|
||||||
for rel in defaults:
|
|
||||||
if rel == "references":
|
|
||||||
continue
|
|
||||||
edges.append(_edge(rel, "note", note_id, r, note_id, {
|
|
||||||
"edge_id": _mk_edge_id(rel, note_id, r, "note", f"edge_defaults:{note_type}:{rel}"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
|
||||||
"confidence": 0.7,
|
|
||||||
}))
|
|
||||||
if rel in {"related_to", "similar_to"}:
|
|
||||||
edges.append(_edge(rel, "note", r, note_id, note_id, {
|
|
||||||
"edge_id": _mk_edge_id(rel, r, note_id, "note", f"edge_defaults:{note_type}:{rel}"),
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
|
||||||
"confidence": 0.7,
|
|
||||||
}))
|
|
||||||
|
|
||||||
# 5) De-Dupe (source_id, target_id, relation, rule_id)
|
|
||||||
seen: Set[Tuple[str,str,str,str]] = set()
|
|
||||||
out: List[dict] = []
|
|
||||||
for e in edges:
|
|
||||||
s = str(e.get("source_id") or "")
|
|
||||||
t = str(e.get("target_id") or "")
|
|
||||||
rel = str(e.get("relation") or e.get("kind") or "edge")
|
|
||||||
rule = str(e.get("rule_id") or "")
|
|
||||||
key = (s, t, rel, rule)
|
|
||||||
if key in seen:
|
|
||||||
continue
|
|
||||||
seen.add(key)
|
|
||||||
out.append(e)
|
|
||||||
return out
|
|
||||||
@@ -1,296 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
Modul: app/core/edges.py
|
|
||||||
Version: 2.0.0 (V2‑superset, rückwärtskompatibel zu v1 vom 2025‑09‑09)
|
|
||||||
|
|
||||||
Zweck
|
|
||||||
-----
|
|
||||||
Bewahrt die bestehende Edgelogik (belongs_to, prev/next, references, backlink)
|
|
||||||
und ergänzt V2‑Felder + Typ‑Default‑Kanten gemäß config/types.yaml (edge_defaults).
|
|
||||||
Die Funktion ist **idempotent** und **rückwärtskompatibel** zur bisherigen Signatur.
|
|
||||||
|
|
||||||
Kompatibilitätsgarantien (gegenüber v1):
|
|
||||||
- **Input**: akzeptiert identische Chunk‑Payloads wie v1:
|
|
||||||
* `id` (Chunk‑ID), `note_id` (Owner), `neighbors.prev|next` (optional),
|
|
||||||
`references: [{target_id: ...}]` (optional),
|
|
||||||
alternativ: `chunk_id`, `chunk_index|ord`, `window|text`
|
|
||||||
- **Output (v1‑Felder)**: `kind`, `source_id`, `target_id`, `scope`, `note_id`, `edge_id`
|
|
||||||
- **Neu (v2‑Felder)**: `relation`, `src_note_id`, `src_chunk_id?`, `dst_note_id`, `dst_chunk_id?`,
|
|
||||||
`provenance` (`explicit|rule`), `rule_id?`, `confidence?`
|
|
||||||
|
|
||||||
Regeln
|
|
||||||
------
|
|
||||||
- Deduplizierungsschlüssel: (source_id, target_id, relation, rule_id)
|
|
||||||
- Strukturkanten:
|
|
||||||
* belongs_to: 1× pro Chunk
|
|
||||||
* next/prev: Sequenz der Chunks; nutzt bevorzugt neighbors; sonst ord/chunk_index
|
|
||||||
- Explizite Referenzen:
|
|
||||||
* aus Chunk: `references[].target_id` (falls vorhanden)
|
|
||||||
* Fallback: Wikilinks in `window|text`: [[Some Title|some-id]] oder [[some-id]]
|
|
||||||
- Note‑Scope:
|
|
||||||
* backlink immer; references nur, wenn include_note_scope_refs=True
|
|
||||||
- Typ‑Defaults (edge_defaults aus config/types.yaml des **Quell‑Notiztyps**):
|
|
||||||
* Für jede explizite Referenz wird je default‑Relation eine Regel‑Kante erzeugt
|
|
||||||
* rule_id: "type_default:{note_type}:{relation}:v1", provenance="rule"
|
|
||||||
|
|
||||||
Konfiguration
|
|
||||||
-------------
|
|
||||||
- ENV MINDNET_TYPES_FILE (Default: ./config/types.yaml)
|
|
||||||
|
|
||||||
Lizenz/Autor
|
|
||||||
------------
|
|
||||||
- Erstimplementierung v1 (2025‑09‑09) — Projekt Mindnet
|
|
||||||
- Erweiterung v2 (2025‑11‑11) — kompatible Superset‑Implementierung
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
from typing import Dict, Iterable, List, Optional, Tuple, Set
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml # optional, nur für types.yaml
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
yaml = None
|
|
||||||
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
# Hilfen: types.yaml laden (edge_defaults)
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
|
|
||||||
def _types_path() -> str:
|
|
||||||
return os.getenv("MINDNET_TYPES_FILE") or "./config/types.yaml"
|
|
||||||
|
|
||||||
def _load_types() -> Dict[str, dict]:
|
|
||||||
p = _types_path()
|
|
||||||
if not os.path.isfile(p) or yaml is None:
|
|
||||||
return {}
|
|
||||||
try:
|
|
||||||
with open(p, "r", encoding="utf-8") as f:
|
|
||||||
data = yaml.safe_load(f) or {}
|
|
||||||
if isinstance(data, dict) and "types" in data and isinstance(data["types"], dict):
|
|
||||||
return data["types"]
|
|
||||||
return data if isinstance(data, dict) else {}
|
|
||||||
except Exception:
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _edge_defaults_for(note_type: Optional[str]) -> List[str]:
|
|
||||||
types = _load_types()
|
|
||||||
t = (note_type or "").strip().lower()
|
|
||||||
cfg = types.get(t) or {}
|
|
||||||
defaults = cfg.get("edge_defaults") or []
|
|
||||||
if isinstance(defaults, str):
|
|
||||||
defaults = [defaults]
|
|
||||||
return [str(x) for x in defaults if isinstance(x, (str, int, float))]
|
|
||||||
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
# Wikilink‑Parser (Fallback, wenn ch["references"] fehlt)
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
|
|
||||||
_WIKILINK_RE = re.compile(r"\[\[(?:[^\|\]]+\|)?([a-zA-Z0-9_\-#:. ]+)\]\]")
|
|
||||||
|
|
||||||
def _extract_wikilinks(text: str) -> List[str]:
|
|
||||||
ids: List[str] = []
|
|
||||||
for m in _WIKILINK_RE.finditer(text or ""):
|
|
||||||
ids.append(m.group(1).strip())
|
|
||||||
return ids
|
|
||||||
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
# Utility
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
|
|
||||||
def _mk_edge_id(kind: str, s: str, t: str, scope: str, rule_id: Optional[str] = None) -> str:
|
|
||||||
base = f"{kind}:{s}->{t}#{scope}"
|
|
||||||
if rule_id:
|
|
||||||
base += f"|{rule_id}"
|
|
||||||
try:
|
|
||||||
import hashlib
|
|
||||||
return hashlib.blake2s(base.encode("utf-8"), digest_size=12).hexdigest()
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
return base
|
|
||||||
|
|
||||||
def _dedupe(edges: List[Dict]) -> List[Dict]:
|
|
||||||
seen: Set[Tuple[str,str,str,str]] = set()
|
|
||||||
out: List[Dict] = []
|
|
||||||
for e in edges:
|
|
||||||
s = str(e.get("source_id") or "")
|
|
||||||
t = str(e.get("target_id") or "")
|
|
||||||
rel = str(e.get("relation") or e.get("kind") or "edge")
|
|
||||||
rule = str(e.get("rule_id") or "")
|
|
||||||
key = (s, t, rel, rule)
|
|
||||||
if key in seen:
|
|
||||||
continue
|
|
||||||
seen.add(key)
|
|
||||||
out.append(e)
|
|
||||||
return out
|
|
||||||
|
|
||||||
def _first(v: dict, *keys, default=None):
|
|
||||||
for k in keys:
|
|
||||||
if k in v and v[k] is not None:
|
|
||||||
return v[k]
|
|
||||||
return default
|
|
||||||
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
# Hauptfunktion
|
|
||||||
# ------------------------------------------------------------
|
|
||||||
|
|
||||||
def build_edges_for_note(
|
|
||||||
note_id: str,
|
|
||||||
chunk_payloads: List[Dict],
|
|
||||||
note_level_refs: Optional[List[str]] = None,
|
|
||||||
*,
|
|
||||||
include_note_scope_refs: bool = False,
|
|
||||||
) -> List[Dict]:
|
|
||||||
edges: List[Dict] = []
|
|
||||||
chunks = list(chunk_payloads or [])
|
|
||||||
# Notiztyp aus erstem Chunk ableiten (kompatibel zu existierenden Payloads)
|
|
||||||
note_type = (chunks[0].get("type") if chunks else None) or (chunks[0].get("note_type") if chunks else None)
|
|
||||||
|
|
||||||
# --- Strukturkanten ------------------------------------------------------
|
|
||||||
# belongs_to
|
|
||||||
for ch in chunks:
|
|
||||||
cid = _first(ch, "id", "chunk_id")
|
|
||||||
if not cid:
|
|
||||||
continue
|
|
||||||
owner = ch.get("note_id") or note_id
|
|
||||||
e = {
|
|
||||||
"edge_id": _mk_edge_id("belongs_to", cid, note_id, "chunk", "structure:belongs_to:v1"),
|
|
||||||
"kind": "belongs_to",
|
|
||||||
"relation": "belongs_to",
|
|
||||||
"scope": "chunk",
|
|
||||||
"source_id": cid,
|
|
||||||
"target_id": note_id,
|
|
||||||
"note_id": owner, # v1-Kompat
|
|
||||||
# v2
|
|
||||||
"src_note_id": owner,
|
|
||||||
"src_chunk_id": cid,
|
|
||||||
"dst_note_id": note_id,
|
|
||||||
"provenance": "rule",
|
|
||||||
"rule_id": "structure:belongs_to:v1",
|
|
||||||
"confidence": 1.0,
|
|
||||||
}
|
|
||||||
edges.append(e)
|
|
||||||
|
|
||||||
# next/prev — bevorzugt neighbors.prev/next; sonst via ord/chunk_index
|
|
||||||
# Map der Chunks nach Index
|
|
||||||
ordered = list(chunks)
|
|
||||||
def _idx(c):
|
|
||||||
return _first(c, "chunk_index", "ord", default=0)
|
|
||||||
ordered.sort(key=_idx)
|
|
||||||
|
|
||||||
for i, ch in enumerate(ordered):
|
|
||||||
cid = _first(ch, "id", "chunk_id")
|
|
||||||
if not cid:
|
|
||||||
continue
|
|
||||||
owner = ch.get("note_id") or note_id
|
|
||||||
nb = ch.get("neighbors") or {}
|
|
||||||
prev_id = nb.get("prev")
|
|
||||||
next_id = nb.get("next")
|
|
||||||
# Fallback-Reihenfolge
|
|
||||||
if prev_id is None and i > 0:
|
|
||||||
prev_id = _first(ordered[i-1], "id", "chunk_id")
|
|
||||||
if next_id is None and i+1 < len(ordered):
|
|
||||||
next_id = _first(ordered[i+1], "id", "chunk_id")
|
|
||||||
|
|
||||||
if prev_id:
|
|
||||||
edges.append({
|
|
||||||
"edge_id": _mk_edge_id("prev", cid, prev_id, "chunk", "structure:order:v1"),
|
|
||||||
"kind": "prev", "relation": "prev", "scope": "chunk",
|
|
||||||
"source_id": cid, "target_id": prev_id, "note_id": owner,
|
|
||||||
"src_note_id": owner, "src_chunk_id": cid,
|
|
||||||
"dst_note_id": owner, "dst_chunk_id": prev_id,
|
|
||||||
"provenance": "rule", "rule_id": "structure:order:v1", "confidence": 0.95,
|
|
||||||
})
|
|
||||||
edges.append({
|
|
||||||
"edge_id": _mk_edge_id("next", prev_id, cid, "chunk", "structure:order:v1"),
|
|
||||||
"kind": "next", "relation": "next", "scope": "chunk",
|
|
||||||
"source_id": prev_id, "target_id": cid, "note_id": owner,
|
|
||||||
"src_note_id": owner, "src_chunk_id": prev_id,
|
|
||||||
"dst_note_id": owner, "dst_chunk_id": cid,
|
|
||||||
"provenance": "rule", "rule_id": "structure:order:v1", "confidence": 0.95,
|
|
||||||
})
|
|
||||||
|
|
||||||
# --- Explizite Referenzen (Chunk‑Scope) ---------------------------------
|
|
||||||
explicit_refs: List[Dict] = []
|
|
||||||
for ch in chunks:
|
|
||||||
cid = _first(ch, "id", "chunk_id")
|
|
||||||
if not cid:
|
|
||||||
continue
|
|
||||||
owner = ch.get("note_id") or note_id
|
|
||||||
# 1) bevorzugt vorhandene ch["references"]
|
|
||||||
refs = ch.get("references") or []
|
|
||||||
targets = [r.get("target_id") for r in refs if isinstance(r, dict) and r.get("target_id")]
|
|
||||||
# 2) Fallback: Wikilinks aus Text
|
|
||||||
if not targets:
|
|
||||||
text = _first(ch, "window", "text", default="") or ""
|
|
||||||
targets = _extract_wikilinks(text)
|
|
||||||
for tid in targets:
|
|
||||||
if not isinstance(tid, str) or not tid.strip():
|
|
||||||
continue
|
|
||||||
e = {
|
|
||||||
"edge_id": _mk_edge_id("references", cid, tid, "chunk"),
|
|
||||||
"kind": "references",
|
|
||||||
"relation": "references",
|
|
||||||
"scope": "chunk",
|
|
||||||
"source_id": cid,
|
|
||||||
"target_id": tid,
|
|
||||||
"note_id": owner,
|
|
||||||
# v2
|
|
||||||
"src_note_id": owner,
|
|
||||||
"src_chunk_id": cid,
|
|
||||||
"dst_note_id": tid,
|
|
||||||
"provenance": "explicit",
|
|
||||||
"rule_id": "",
|
|
||||||
"confidence": 1.0,
|
|
||||||
}
|
|
||||||
edges.append(e)
|
|
||||||
explicit_refs.append(e)
|
|
||||||
|
|
||||||
# --- Note‑Scope: references (optional) + backlink (immer) ----------------
|
|
||||||
unique_refs = []
|
|
||||||
if note_level_refs:
|
|
||||||
seen = set()
|
|
||||||
for tid in note_level_refs:
|
|
||||||
if isinstance(tid, str) and tid.strip() and tid not in seen:
|
|
||||||
unique_refs.append(tid); seen.add(tid)
|
|
||||||
|
|
||||||
for tid in unique_refs:
|
|
||||||
if include_note_scope_refs:
|
|
||||||
edges.append({
|
|
||||||
"edge_id": _mk_edge_id("references", note_id, tid, "note"),
|
|
||||||
"kind": "references", "relation": "references", "scope": "note",
|
|
||||||
"source_id": note_id, "target_id": tid, "note_id": note_id,
|
|
||||||
"src_note_id": note_id, "dst_note_id": tid,
|
|
||||||
"provenance": "explicit", "rule_id": "", "confidence": 1.0,
|
|
||||||
})
|
|
||||||
edges.append({
|
|
||||||
"edge_id": _mk_edge_id("backlink", tid, note_id, "note", "derived:backlink:v1"),
|
|
||||||
"kind": "backlink", "relation": "backlink", "scope": "note",
|
|
||||||
"source_id": tid, "target_id": note_id, "note_id": note_id,
|
|
||||||
"src_note_id": tid, "dst_note_id": note_id,
|
|
||||||
"provenance": "rule", "rule_id": "derived:backlink:v1", "confidence": 0.9,
|
|
||||||
})
|
|
||||||
|
|
||||||
# --- Type‑Defaults je expliziter Referenz --------------------------------
|
|
||||||
defaults = [d for d in _edge_defaults_for(note_type) if d and d != "references"]
|
|
||||||
if defaults:
|
|
||||||
for e in explicit_refs + ([ ] if not include_note_scope_refs else []):
|
|
||||||
# wir nutzen die bereits erzeugten explicit‑Edges als Vorlage
|
|
||||||
src = e["source_id"]; tgt = e["target_id"]
|
|
||||||
scope = e.get("scope", "chunk")
|
|
||||||
s_note = e.get("src_note_id") or note_id
|
|
||||||
s_chunk = e.get("src_chunk_id")
|
|
||||||
t_note = e.get("dst_note_id") or tgt
|
|
||||||
for rel in defaults:
|
|
||||||
rule_id = f"type_default:{(note_type or 'unknown')}:{rel}:v1"
|
|
||||||
edges.append({
|
|
||||||
"edge_id": _mk_edge_id(rel, src, tgt, scope, rule_id),
|
|
||||||
"kind": rel, "relation": rel, "scope": scope,
|
|
||||||
"source_id": src, "target_id": tgt, "note_id": s_note,
|
|
||||||
"src_note_id": s_note, "src_chunk_id": s_chunk,
|
|
||||||
"dst_note_id": t_note,
|
|
||||||
"provenance": "rule", "rule_id": rule_id, "confidence": 0.7,
|
|
||||||
})
|
|
||||||
|
|
||||||
# --- Dedupe & Return -----------------------------------------------------
|
|
||||||
return _dedupe(edges)
|
|
||||||
@@ -1,94 +0,0 @@
|
|||||||
# app/core/edges_writer.py
|
|
||||||
from __future__ import annotations
|
|
||||||
import hashlib
|
|
||||||
from typing import Dict, List, Iterable, Tuple
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Dein Modul mit der Schemadefinition und der Builder-Funktion
|
|
||||||
from app.core.edges import build_edges_for_note # noqa: F401
|
|
||||||
except Exception as e:
|
|
||||||
raise RuntimeError("Konnte app.core.edges nicht importieren. "
|
|
||||||
"Bitte sicherstellen, dass app/core/edges.py vorhanden ist.") from e
|
|
||||||
|
|
||||||
def _edge_uid(kind: str, source_id: str, target_id: str, scope: str) -> str:
|
|
||||||
"""
|
|
||||||
Deterministische, kurze ID für eine Edge.
|
|
||||||
Kollisionen sind praktisch ausgeschlossen (BLAKE2s über den Kanonischen Schlüssel).
|
|
||||||
"""
|
|
||||||
key = f"{kind}|{source_id}|{target_id}|{scope}"
|
|
||||||
return hashlib.blake2s(key.encode("utf-8"), digest_size=12).hexdigest()
|
|
||||||
|
|
||||||
def ensure_edges_collection(qdrant_client, collection: str) -> None:
|
|
||||||
"""
|
|
||||||
Legt die Edge-Collection an, falls sie nicht existiert.
|
|
||||||
Minimal: 1D-Vector (Dummy), Cosine. Payload-only-Collections sind je nach Qdrant-Version heikel.
|
|
||||||
"""
|
|
||||||
from qdrant_client.http import models as qm
|
|
||||||
|
|
||||||
existing = [c.name for c in qdrant_client.get_collections().collections]
|
|
||||||
if collection in existing:
|
|
||||||
return
|
|
||||||
|
|
||||||
qdrant_client.recreate_collection(
|
|
||||||
collection_name=collection,
|
|
||||||
vectors_config=qm.VectorParams(size=1, distance=qm.Distance.COSINE),
|
|
||||||
on_disk_payload=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def edges_from_note(
|
|
||||||
note_id: str,
|
|
||||||
chunk_payloads: List[Dict],
|
|
||||||
note_level_refs: Iterable[str] | None,
|
|
||||||
*,
|
|
||||||
include_note_scope_refs: bool = False,
|
|
||||||
) -> List[Dict]:
|
|
||||||
"""
|
|
||||||
Ruft deinen Edge-Builder auf und gibt die (deduplizierten) Edge-Payloads zurück.
|
|
||||||
Keine Schemaänderung – exakt das aus app/core/edges.py.
|
|
||||||
"""
|
|
||||||
return build_edges_for_note(
|
|
||||||
note_id=note_id,
|
|
||||||
chunk_payloads=chunk_payloads,
|
|
||||||
note_level_refs=list(note_level_refs or []),
|
|
||||||
include_note_scope_refs=include_note_scope_refs,
|
|
||||||
)
|
|
||||||
|
|
||||||
def upsert_edges(
|
|
||||||
qdrant_client,
|
|
||||||
collection: str,
|
|
||||||
edge_payloads: List[Dict],
|
|
||||||
) -> Tuple[int, int]:
|
|
||||||
"""
|
|
||||||
Schreibt Edges als Points in Qdrant.
|
|
||||||
- id: deterministisch aus (kind, source_id, target_id, scope)
|
|
||||||
- vector: [0.0] Dummy
|
|
||||||
- payload: Edge-Dict (unverändert, siehe Schema in app/core/edges.py)
|
|
||||||
Gibt (anzahl_points, anzahl_unique_keys) zurück.
|
|
||||||
"""
|
|
||||||
from qdrant_client.models import PointStruct
|
|
||||||
|
|
||||||
if not edge_payloads:
|
|
||||||
return 0, 0
|
|
||||||
|
|
||||||
points = []
|
|
||||||
seen = set()
|
|
||||||
for e in edge_payloads:
|
|
||||||
key = (e.get("kind"), e.get("source_id"), e.get("target_id"), e.get("scope"))
|
|
||||||
if key in seen:
|
|
||||||
continue
|
|
||||||
seen.add(key)
|
|
||||||
eid = _edge_uid(*key)
|
|
||||||
points.append(
|
|
||||||
PointStruct(
|
|
||||||
id=eid,
|
|
||||||
vector=[0.0],
|
|
||||||
payload=e,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
if not points:
|
|
||||||
return 0, 0
|
|
||||||
|
|
||||||
ensure_edges_collection(qdrant_client, collection)
|
|
||||||
qdrant_client.upsert(collection_name=collection, points=points)
|
|
||||||
return len(points), len(seen)
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
import os, time, json
|
|
||||||
import urllib.request
|
|
||||||
from typing import List, Dict, Any
|
|
||||||
|
|
||||||
# Backend-Auswahl:
|
|
||||||
# - EMBED_BACKEND=ollama -> EMBED_URL=/api/embeddings (Ollama), EMBED_MODEL=z.B. nomic-embed-text
|
|
||||||
# - EMBED_BACKEND=mini -> EMBED_URL=/embed (unser MiniLM-Server), EMBED_MODEL=minilm-384
|
|
||||||
EMBED_BACKEND = os.getenv("EMBED_BACKEND", "mini").lower()
|
|
||||||
EMBED_URL = os.getenv("EMBED_URL", "http://127.0.0.1:8990/embed")
|
|
||||||
EMBED_MODEL = os.getenv("EMBED_MODEL", "minilm-384")
|
|
||||||
EMBED_BATCH = int(os.getenv("EMBED_BATCH", "64"))
|
|
||||||
TIMEOUT = 60
|
|
||||||
|
|
||||||
class EmbedError(RuntimeError): ...
|
|
||||||
|
|
||||||
def _post_json(url: str, payload: Dict[str, Any]) -> Dict[str, Any]:
|
|
||||||
data = json.dumps(payload).encode("utf-8")
|
|
||||||
req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
|
|
||||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
|
|
||||||
return json.loads(resp.read().decode("utf-8"))
|
|
||||||
|
|
||||||
def _embed_mini(inputs: List[str], model: str, batch: int) -> List[List[float]]:
|
|
||||||
out: List[List[float]] = []
|
|
||||||
i = 0
|
|
||||||
while i < len(inputs):
|
|
||||||
chunk = inputs[i:i+batch]
|
|
||||||
# einfache Retries
|
|
||||||
for attempt in range(5):
|
|
||||||
try:
|
|
||||||
resp = _post_json(EMBED_URL, {"model": model, "inputs": chunk})
|
|
||||||
vecs = resp.get("embeddings") or resp.get("vectors") or resp.get("data")
|
|
||||||
if not isinstance(vecs, list):
|
|
||||||
raise EmbedError(f"Bad embed response keys: {list(resp.keys())}")
|
|
||||||
out.extend(vecs)
|
|
||||||
break
|
|
||||||
except Exception:
|
|
||||||
if attempt == 4:
|
|
||||||
raise
|
|
||||||
time.sleep(1.5 * (attempt + 1))
|
|
||||||
i += batch
|
|
||||||
return out
|
|
||||||
|
|
||||||
def _embed_ollama(inputs: List[str], model: str, batch: int) -> List[List[float]]:
|
|
||||||
# Ollama /api/embeddings akzeptiert "input" als String ODER Array.
|
|
||||||
# Die Response enthält:
|
|
||||||
# - für single input: {"embedding":[...], "model":"...", ...}
|
|
||||||
# - für array input: {"embeddings":[[...],[...],...], "model":"...", ...} (je nach Version)
|
|
||||||
# Um maximal kompatibel zu sein, rufen wir pro Text einzeln auf.
|
|
||||||
out: List[List[float]] = []
|
|
||||||
for text in inputs:
|
|
||||||
# Retries
|
|
||||||
for attempt in range(5):
|
|
||||||
try:
|
|
||||||
resp = _post_json(EMBED_URL, {"model": model, "input": text})
|
|
||||||
if "embedding" in resp and isinstance(resp["embedding"], list):
|
|
||||||
out.append(resp["embedding"])
|
|
||||||
elif "embeddings" in resp and isinstance(resp["embeddings"], list):
|
|
||||||
# Falls Server array zurückgibt, nimm das erste Element
|
|
||||||
vecs = resp["embeddings"]
|
|
||||||
out.append(vecs[0] if vecs else [])
|
|
||||||
else:
|
|
||||||
raise EmbedError(f"Ollama response unexpected keys: {list(resp.keys())}")
|
|
||||||
break
|
|
||||||
except Exception:
|
|
||||||
if attempt == 4:
|
|
||||||
raise
|
|
||||||
time.sleep(1.5 * (attempt + 1))
|
|
||||||
return out
|
|
||||||
|
|
||||||
def embed_texts(texts: List[str], model: str | None = None, batch_size: int | None = None) -> List[List[float]]:
|
|
||||||
model = model or EMBED_MODEL
|
|
||||||
batch = batch_size or EMBED_BATCH
|
|
||||||
if not texts:
|
|
||||||
return []
|
|
||||||
if EMBED_BACKEND == "ollama":
|
|
||||||
return _embed_ollama(texts, model, batch)
|
|
||||||
# default: mini
|
|
||||||
return _embed_mini(texts, model, batch)
|
|
||||||
|
|
||||||
def embed_one(text: str, model: str | None = None) -> List[float]:
|
|
||||||
return embed_texts([text], model=model, batch_size=1)[0]
|
|
||||||
@@ -1,103 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
Datei: app/core/env_vars.py
|
|
||||||
Version: 1.1.0 (2025-11-08)
|
|
||||||
|
|
||||||
Zweck
|
|
||||||
Einheitliche Auflösung von ENV-Variablen (Prefix, Qdrant, Embeddings, Hashing)
|
|
||||||
mit Abwärtskompatibilität.
|
|
||||||
|
|
||||||
Grundsatz
|
|
||||||
- Für Qdrant-Funktionen ist 'COLLECTION_PREFIX' der Primärschlüssel.
|
|
||||||
- 'MINDNET_PREFIX' bleibt für App-/UI-/Exporter-Kontexte nutzbar.
|
|
||||||
- Fallbacks sorgen dafür, dass ältere Umgebungen weiter funktionieren.
|
|
||||||
|
|
||||||
Wichtig
|
|
||||||
- Lädt optional eine .env (wenn python-dotenv verfügbar ist).
|
|
||||||
- Überschreibt keine bereits gesetzten OS-Variablen (override=False).
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from typing import Optional, Dict
|
|
||||||
|
|
||||||
# Optional: .env automatisch laden (ohne Hard-Fail, falls nicht vorhanden)
|
|
||||||
try:
|
|
||||||
from dotenv import load_dotenv, find_dotenv # type: ignore
|
|
||||||
_p = find_dotenv()
|
|
||||||
if _p:
|
|
||||||
load_dotenv(_p, override=False)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# -------- Prefix-Auflösung --------
|
|
||||||
|
|
||||||
def get_collection_prefix(cli_override: Optional[str] = None) -> str:
|
|
||||||
"""
|
|
||||||
Für Qdrant-relevante Funktionen:
|
|
||||||
1) CLI-Override (--prefix)
|
|
||||||
2) ENV COLLECTION_PREFIX
|
|
||||||
3) ENV MINDNET_PREFIX (Fallback)
|
|
||||||
4) 'mindnet' (Default)
|
|
||||||
"""
|
|
||||||
if cli_override and str(cli_override).strip():
|
|
||||||
return str(cli_override).strip()
|
|
||||||
return (
|
|
||||||
os.getenv("COLLECTION_PREFIX")
|
|
||||||
or os.getenv("MINDNET_PREFIX")
|
|
||||||
or "mindnet"
|
|
||||||
)
|
|
||||||
|
|
||||||
def get_mindnet_prefix(cli_override: Optional[str] = None) -> str:
|
|
||||||
"""
|
|
||||||
Für App-/UI-/Exporter-Kontexte:
|
|
||||||
1) CLI-Override (--prefix)
|
|
||||||
2) ENV MINDNET_PREFIX
|
|
||||||
3) ENV COLLECTION_PREFIX (Fallback)
|
|
||||||
4) 'mindnet'
|
|
||||||
"""
|
|
||||||
if cli_override and str(cli_override).strip():
|
|
||||||
return str(cli_override).strip()
|
|
||||||
return (
|
|
||||||
os.getenv("MINDNET_PREFIX")
|
|
||||||
or os.getenv("COLLECTION_PREFIX")
|
|
||||||
or "mindnet"
|
|
||||||
)
|
|
||||||
|
|
||||||
def get_prefix(cli_override: Optional[str] = None, target: str = "qdrant") -> str:
|
|
||||||
"""
|
|
||||||
Universelle Hülle (abwärtskompatibel):
|
|
||||||
target='qdrant' -> get_collection_prefix
|
|
||||||
target='app' -> get_mindnet_prefix
|
|
||||||
"""
|
|
||||||
if target.lower() == "app":
|
|
||||||
return get_mindnet_prefix(cli_override)
|
|
||||||
return get_collection_prefix(cli_override)
|
|
||||||
|
|
||||||
# -------- Qdrant / Embeddings / Hashing --------
|
|
||||||
|
|
||||||
def get_qdrant_url(default: str = "http://127.0.0.1:6333") -> str:
|
|
||||||
return os.getenv("QDRANT_URL", default)
|
|
||||||
|
|
||||||
def get_qdrant_api_key(default: str = "") -> str:
|
|
||||||
return os.getenv("QDRANT_API_KEY", default)
|
|
||||||
|
|
||||||
def get_vector_dim(default: int = 384) -> int:
|
|
||||||
try:
|
|
||||||
return int(os.getenv("VECTOR_DIM", str(default)))
|
|
||||||
except Exception:
|
|
||||||
return default
|
|
||||||
|
|
||||||
def get_embed_url(default: Optional[str] = None) -> Optional[str]:
|
|
||||||
return os.getenv("EMBED_URL", default)
|
|
||||||
|
|
||||||
def get_hash_env() -> Dict[str, str]:
|
|
||||||
"""
|
|
||||||
Liefert die Hash-Konfiguration (nur Aggregation; die Auswertung bleibt in den Skripten).
|
|
||||||
"""
|
|
||||||
return {
|
|
||||||
"MINDNET_HASH_COMPARE": os.getenv("MINDNET_HASH_COMPARE", ""),
|
|
||||||
"MINDNET_HASH_SOURCE": os.getenv("MINDNET_HASH_SOURCE", ""),
|
|
||||||
"MINDNET_HASH_NORMALIZE": os.getenv("MINDNET_HASH_NORMALIZE", ""),
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/__init__.py
|
||||||
|
DESCRIPTION: Unified Graph Package. Exportiert Kanten-Ableitung und Graph-Adapter.
|
||||||
|
"""
|
||||||
|
from .graph_derive_edges import build_edges_for_note
|
||||||
|
from .graph_utils import PROVENANCE_PRIORITY
|
||||||
|
from .graph_subgraph import Subgraph, expand
|
||||||
|
from .graph_weights import EDGE_BASE_WEIGHTS
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"build_edges_for_note",
|
||||||
|
"PROVENANCE_PRIORITY",
|
||||||
|
"Subgraph",
|
||||||
|
"expand",
|
||||||
|
"EDGE_BASE_WEIGHTS"
|
||||||
|
]
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/graph_db_adapter.py
|
||||||
|
DESCRIPTION: Datenbeschaffung aus Qdrant für den Graphen.
|
||||||
|
AUDIT v1.1.1: Volle Unterstützung für WP-15c Metadaten.
|
||||||
|
Stellt sicher, dass 'target_section' und 'provenance' für die
|
||||||
|
Super-Edge-Aggregation im Retriever geladen werden.
|
||||||
|
"""
|
||||||
|
from typing import List, Dict, Optional
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
# Nutzt die zentrale Infrastruktur für konsistente Collection-Namen (WP-14)
|
||||||
|
from app.core.database import collection_names
|
||||||
|
|
||||||
|
def fetch_edges_from_qdrant(
|
||||||
|
client: QdrantClient,
|
||||||
|
prefix: str,
|
||||||
|
seeds: List[str],
|
||||||
|
edge_types: Optional[List[str]] = None,
|
||||||
|
limit: int = 2048,
|
||||||
|
) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Holt Edges aus der Datenbank basierend auf Seed-IDs.
|
||||||
|
WP-15c: Erhält alle Metadaten für das Note-Level Diversity Pooling.
|
||||||
|
"""
|
||||||
|
if not seeds or limit <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Konsistente Namensauflösung via database-Paket
|
||||||
|
# Rückgabe: (notes_col, chunks_col, edges_col)
|
||||||
|
_, _, edges_col = collection_names(prefix)
|
||||||
|
|
||||||
|
# Wir suchen Kanten, bei denen die Seed-IDs entweder Quelle, Ziel oder Kontext-Note sind.
|
||||||
|
seed_conditions = []
|
||||||
|
for field in ("source_id", "target_id", "note_id"):
|
||||||
|
for s in seeds:
|
||||||
|
seed_conditions.append(
|
||||||
|
rest.FieldCondition(key=field, match=rest.MatchValue(value=str(s)))
|
||||||
|
)
|
||||||
|
seeds_filter = rest.Filter(should=seed_conditions) if seed_conditions else None
|
||||||
|
|
||||||
|
# Optionaler Filter auf spezifische Kanten-Typen (z.B. für Intent-Routing)
|
||||||
|
type_filter = None
|
||||||
|
if edge_types:
|
||||||
|
type_conds = [
|
||||||
|
rest.FieldCondition(key="kind", match=rest.MatchValue(value=str(k)))
|
||||||
|
for k in edge_types
|
||||||
|
]
|
||||||
|
type_filter = rest.Filter(should=type_conds)
|
||||||
|
|
||||||
|
must = []
|
||||||
|
if seeds_filter:
|
||||||
|
must.append(seeds_filter)
|
||||||
|
if type_filter:
|
||||||
|
must.append(type_filter)
|
||||||
|
|
||||||
|
flt = rest.Filter(must=must) if must else None
|
||||||
|
|
||||||
|
# Abfrage via Qdrant Scroll API
|
||||||
|
# WICHTIG: with_payload=True lädt alle Metadaten (target_section, provenance etc.)
|
||||||
|
pts, _ = client.scroll(
|
||||||
|
collection_name=edges_col,
|
||||||
|
scroll_filter=flt,
|
||||||
|
limit=limit,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wir geben das vollständige Payload zurück, damit der Retriever
|
||||||
|
# alle Signale für die Super-Edge-Aggregation und das Scoring hat.
|
||||||
|
return [dict(p.payload) for p in pts if p.payload]
|
||||||
@@ -0,0 +1,169 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/graph_derive_edges.py
|
||||||
|
DESCRIPTION: Hauptlogik zur Kanten-Aggregation und De-Duplizierung.
|
||||||
|
WP-15b/c Audit:
|
||||||
|
- Präzises Sektions-Splitting via parse_link_target.
|
||||||
|
- Eindeutige ID-Generierung pro Sektions-Variante (Multigraph).
|
||||||
|
- Ermöglicht dem Retriever die Super-Edge-Aggregation.
|
||||||
|
"""
|
||||||
|
from typing import List, Optional, Dict, Tuple
|
||||||
|
from .graph_utils import (
|
||||||
|
_get, _edge, _mk_edge_id, _dedupe_seq, parse_link_target,
|
||||||
|
PROVENANCE_PRIORITY, load_types_registry, get_edge_defaults_for
|
||||||
|
)
|
||||||
|
from .graph_extractors import (
|
||||||
|
extract_typed_relations, extract_callout_relations, extract_wikilinks
|
||||||
|
)
|
||||||
|
|
||||||
|
def build_edges_for_note(
|
||||||
|
note_id: str,
|
||||||
|
chunks: List[dict],
|
||||||
|
note_level_references: Optional[List[str]] = None,
|
||||||
|
include_note_scope_refs: bool = False,
|
||||||
|
) -> List[dict]:
|
||||||
|
"""
|
||||||
|
Erzeugt und aggregiert alle Kanten für eine Note.
|
||||||
|
Sorgt für die physische Trennung von Sektions-Links via Edge-ID.
|
||||||
|
"""
|
||||||
|
edges: List[dict] = []
|
||||||
|
# note_type für die Ermittlung der edge_defaults (types.yaml)
|
||||||
|
note_type = _get(chunks[0], "type") if chunks else "concept"
|
||||||
|
|
||||||
|
# 1) Struktur-Kanten (Internal: belongs_to, next/prev)
|
||||||
|
# Diese erhalten die Provenienz 'structure' und sind in der Registry geschützt.
|
||||||
|
for idx, ch in enumerate(chunks):
|
||||||
|
cid = _get(ch, "chunk_id", "id")
|
||||||
|
if not cid: continue
|
||||||
|
|
||||||
|
# Verbindung Chunk -> Note
|
||||||
|
edges.append(_edge("belongs_to", "chunk", cid, note_id, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id("belongs_to", cid, note_id, "chunk", "structure:belongs_to"),
|
||||||
|
"provenance": "structure",
|
||||||
|
"rule_id": "structure:belongs_to",
|
||||||
|
"confidence": PROVENANCE_PRIORITY["structure:belongs_to"]
|
||||||
|
}))
|
||||||
|
|
||||||
|
# Horizontale Verkettung (Ordnung)
|
||||||
|
if idx < len(chunks) - 1:
|
||||||
|
next_id = _get(chunks[idx+1], "chunk_id", "id")
|
||||||
|
if next_id:
|
||||||
|
edges.append(_edge("next", "chunk", cid, next_id, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id("next", cid, next_id, "chunk", "structure:order"),
|
||||||
|
"provenance": "structure", "rule_id": "structure:order", "confidence": PROVENANCE_PRIORITY["structure:order"]
|
||||||
|
}))
|
||||||
|
edges.append(_edge("prev", "chunk", next_id, cid, note_id, {
|
||||||
|
"chunk_id": next_id,
|
||||||
|
"edge_id": _mk_edge_id("prev", next_id, cid, "chunk", "structure:order"),
|
||||||
|
"provenance": "structure", "rule_id": "structure:order", "confidence": PROVENANCE_PRIORITY["structure:order"]
|
||||||
|
}))
|
||||||
|
|
||||||
|
# 2) Inhaltliche Kanten (Explicit & Candidate Pool)
|
||||||
|
reg = load_types_registry()
|
||||||
|
defaults = get_edge_defaults_for(note_type, reg)
|
||||||
|
refs_all: List[str] = []
|
||||||
|
|
||||||
|
for ch in chunks:
|
||||||
|
cid = _get(ch, "chunk_id", "id")
|
||||||
|
if not cid: continue
|
||||||
|
raw = _get(ch, "window") or _get(ch, "text") or ""
|
||||||
|
|
||||||
|
# A. Typed Relations (Inline [[rel:kind|target]])
|
||||||
|
typed, rem = extract_typed_relations(raw)
|
||||||
|
for k, raw_t in typed:
|
||||||
|
t, sec = parse_link_target(raw_t, note_id)
|
||||||
|
if not t: continue
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"chunk_id": cid,
|
||||||
|
# WP-Fix: Variant=sec sorgt für eindeutige ID pro Sektion
|
||||||
|
"edge_id": _mk_edge_id(k, cid, t, "chunk", "inline:rel", variant=sec),
|
||||||
|
"provenance": "explicit", "rule_id": "inline:rel", "confidence": PROVENANCE_PRIORITY["inline:rel"]
|
||||||
|
}
|
||||||
|
if sec: payload["target_section"] = sec
|
||||||
|
edges.append(_edge(k, "chunk", cid, t, note_id, payload))
|
||||||
|
|
||||||
|
# B. Candidate Pool (WP-15b Validierte KI-Kanten)
|
||||||
|
pool = ch.get("candidate_pool") or ch.get("candidate_edges") or []
|
||||||
|
for cand in pool:
|
||||||
|
raw_t, k, p = cand.get("to"), cand.get("kind", "related_to"), cand.get("provenance", "semantic_ai")
|
||||||
|
t, sec = parse_link_target(raw_t, note_id)
|
||||||
|
if t:
|
||||||
|
payload = {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(k, cid, t, "chunk", f"candidate:{p}", variant=sec),
|
||||||
|
"provenance": p, "rule_id": f"candidate:{p}", "confidence": PROVENANCE_PRIORITY.get(p, 0.90)
|
||||||
|
}
|
||||||
|
if sec: payload["target_section"] = sec
|
||||||
|
edges.append(_edge(k, "chunk", cid, t, note_id, payload))
|
||||||
|
|
||||||
|
# C. Callouts (> [!edge])
|
||||||
|
call_pairs, rem2 = extract_callout_relations(rem)
|
||||||
|
for k, raw_t in call_pairs:
|
||||||
|
t, sec = parse_link_target(raw_t, note_id)
|
||||||
|
if not t: continue
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(k, cid, t, "chunk", "callout:edge", variant=sec),
|
||||||
|
"provenance": "explicit", "rule_id": "callout:edge", "confidence": PROVENANCE_PRIORITY["callout:edge"]
|
||||||
|
}
|
||||||
|
if sec: payload["target_section"] = sec
|
||||||
|
edges.append(_edge(k, "chunk", cid, t, note_id, payload))
|
||||||
|
|
||||||
|
# D. Standard Wikilinks & Typ-Defaults
|
||||||
|
refs = extract_wikilinks(rem2)
|
||||||
|
for raw_r in refs:
|
||||||
|
r, sec = parse_link_target(raw_r, note_id)
|
||||||
|
if not r: continue
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"chunk_id": cid, "ref_text": raw_r,
|
||||||
|
"edge_id": _mk_edge_id("references", cid, r, "chunk", "explicit:wikilink", variant=sec),
|
||||||
|
"provenance": "explicit", "rule_id": "explicit:wikilink", "confidence": PROVENANCE_PRIORITY["explicit:wikilink"]
|
||||||
|
}
|
||||||
|
if sec: payload["target_section"] = sec
|
||||||
|
edges.append(_edge("references", "chunk", cid, r, note_id, payload))
|
||||||
|
|
||||||
|
# Automatische Kanten-Vererbung aus types.yaml
|
||||||
|
for rel in defaults:
|
||||||
|
if rel != "references":
|
||||||
|
def_payload = {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(rel, cid, r, "chunk", f"edge_defaults:{rel}", variant=sec),
|
||||||
|
"provenance": "rule", "rule_id": f"edge_defaults:{rel}", "confidence": PROVENANCE_PRIORITY["edge_defaults"]
|
||||||
|
}
|
||||||
|
if sec: def_payload["target_section"] = sec
|
||||||
|
edges.append(_edge(rel, "chunk", cid, r, note_id, def_payload))
|
||||||
|
|
||||||
|
refs_all.extend([parse_link_target(r, note_id)[0] for r in refs])
|
||||||
|
|
||||||
|
# 3) Note-Scope (Grobe Struktur-Verbindungen)
|
||||||
|
if include_note_scope_refs:
|
||||||
|
cleaned_note_refs = [parse_link_target(r, note_id)[0] for r in (note_level_references or [])]
|
||||||
|
refs_note = _dedupe_seq((refs_all or []) + cleaned_note_refs)
|
||||||
|
|
||||||
|
for r in refs_note:
|
||||||
|
if not r: continue
|
||||||
|
edges.append(_edge("references", "note", note_id, r, note_id, {
|
||||||
|
"edge_id": _mk_edge_id("references", note_id, r, "note", "explicit:note_scope"),
|
||||||
|
"provenance": "explicit", "confidence": PROVENANCE_PRIORITY["explicit:note_scope"]
|
||||||
|
}))
|
||||||
|
# Backlinks zur Stärkung der Bidirektionalität
|
||||||
|
edges.append(_edge("backlink", "note", r, note_id, note_id, {
|
||||||
|
"edge_id": _mk_edge_id("backlink", r, note_id, "note", "derived:backlink"),
|
||||||
|
"provenance": "rule", "confidence": PROVENANCE_PRIORITY["derived:backlink"]
|
||||||
|
}))
|
||||||
|
|
||||||
|
# 4) De-Duplizierung (In-Place)
|
||||||
|
# Da die EDGE-ID nun die Sektion (variant) enthält, bleiben Links auf
|
||||||
|
# unterschiedliche Abschnitte derselben Note erhalten.
|
||||||
|
unique_map: Dict[str, dict] = {}
|
||||||
|
for e in edges:
|
||||||
|
eid = e["edge_id"]
|
||||||
|
# Höhere Confidence gewinnt bei identischer ID
|
||||||
|
if eid not in unique_map or e.get("confidence", 0) > unique_map[eid].get("confidence", 0):
|
||||||
|
unique_map[eid] = e
|
||||||
|
|
||||||
|
return list(unique_map.values())
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/graph_extractors.py
|
||||||
|
DESCRIPTION: Regex-basierte Extraktion von Relationen aus Text.
|
||||||
|
AUDIT:
|
||||||
|
- Regex für Wikilinks liberalisiert (Umlaute, Sonderzeichen).
|
||||||
|
- Callout-Parser erweitert für Multi-Line-Listen und Header-Typen.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from typing import List, Tuple
|
||||||
|
|
||||||
|
# Erlaube alle Zeichen außer ']' im Target (fängt Umlaute, Emojis, '&', '#' ab)
|
||||||
|
_WIKILINK_RE = re.compile(r"\[\[(?:[^\|\]]+\|)?([^\]]+)\]\]")
|
||||||
|
|
||||||
|
_REL_PIPE = re.compile(r"\[\[\s*rel:(?P<kind>[a-z_]+)\s*\|\s*(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
||||||
|
_REL_SPACE = re.compile(r"\[\[\s*rel:(?P<kind>[a-z_]+)\s+(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
||||||
|
_REL_TEXT = re.compile(r"rel\s*:\s*(?P<kind>[a-z_]+)\s*\[\[\s*(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
||||||
|
|
||||||
|
_CALLOUT_START = re.compile(r"^\s*>\s*\[!edge\]\s*(.*)$", re.IGNORECASE)
|
||||||
|
# Erkennt "kind: targets..."
|
||||||
|
_REL_LINE = re.compile(r"^(?P<kind>[a-z_]+)\s*:\s*(?P<targets>.+?)\s*$", re.IGNORECASE)
|
||||||
|
# Erkennt reine Typen (z.B. "depends_on" im Header)
|
||||||
|
_SIMPLE_KIND = re.compile(r"^[a-z_]+$", re.IGNORECASE)
|
||||||
|
|
||||||
|
def extract_typed_relations(text: str) -> Tuple[List[Tuple[str, str]], str]:
|
||||||
|
"""
|
||||||
|
Findet Inline-Relationen wie [[rel:depends_on Target]].
|
||||||
|
Gibt (Liste[(kind, target)], bereinigter_text) zurück.
|
||||||
|
"""
|
||||||
|
if not text: return [], ""
|
||||||
|
pairs = []
|
||||||
|
def _collect(m):
|
||||||
|
k, t = m.group("kind").strip().lower(), m.group("target").strip()
|
||||||
|
pairs.append((k, t))
|
||||||
|
return ""
|
||||||
|
text = _REL_PIPE.sub(_collect, text)
|
||||||
|
text = _REL_SPACE.sub(_collect, text)
|
||||||
|
text = _REL_TEXT.sub(_collect, text)
|
||||||
|
return pairs, text
|
||||||
|
|
||||||
|
def extract_callout_relations(text: str) -> Tuple[List[Tuple[str,str]], str]:
|
||||||
|
"""
|
||||||
|
Verarbeitet Obsidian [!edge]-Callouts.
|
||||||
|
Unterstützt zwei Formate:
|
||||||
|
1. Explizit: "kind: [[Target]]"
|
||||||
|
2. Implizit (Header): "> [!edge] kind" gefolgt von "[[Target]]" Zeilen
|
||||||
|
"""
|
||||||
|
if not text: return [], text
|
||||||
|
lines = text.splitlines()
|
||||||
|
out_pairs = []
|
||||||
|
keep_lines = []
|
||||||
|
i = 0
|
||||||
|
|
||||||
|
while i < len(lines):
|
||||||
|
line = lines[i]
|
||||||
|
m = _CALLOUT_START.match(line)
|
||||||
|
if not m:
|
||||||
|
keep_lines.append(line)
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Callout-Block gefunden. Wir sammeln alle relevanten Zeilen.
|
||||||
|
block_lines = []
|
||||||
|
|
||||||
|
# Header Content prüfen (z.B. "type" aus "> [!edge] type")
|
||||||
|
header_raw = m.group(1).strip()
|
||||||
|
if header_raw:
|
||||||
|
block_lines.append(header_raw)
|
||||||
|
|
||||||
|
i += 1
|
||||||
|
while i < len(lines) and lines[i].lstrip().startswith('>'):
|
||||||
|
# Entferne '>' und führende Leerzeichen
|
||||||
|
content = lines[i].lstrip()[1:].lstrip()
|
||||||
|
if content:
|
||||||
|
block_lines.append(content)
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
# Verarbeitung des Blocks
|
||||||
|
current_kind = None
|
||||||
|
|
||||||
|
# Heuristik: Ist die allererste Zeile (meist aus dem Header) ein reiner Typ?
|
||||||
|
# Dann setzen wir diesen als Default für den Block.
|
||||||
|
if block_lines:
|
||||||
|
first = block_lines[0]
|
||||||
|
# Wenn es NICHT wie "Key: Value" aussieht, aber wie ein Wort:
|
||||||
|
if not _REL_LINE.match(first) and _SIMPLE_KIND.match(first):
|
||||||
|
current_kind = first.lower()
|
||||||
|
|
||||||
|
for bl in block_lines:
|
||||||
|
# 1. Prüfen auf explizites "Kind: Targets" (überschreibt Header-Typ für diese Zeile)
|
||||||
|
mrel = _REL_LINE.match(bl)
|
||||||
|
if mrel:
|
||||||
|
line_kind = mrel.group("kind").strip().lower()
|
||||||
|
targets = mrel.group("targets")
|
||||||
|
|
||||||
|
# Links extrahieren
|
||||||
|
found = _WIKILINK_RE.findall(targets)
|
||||||
|
if found:
|
||||||
|
for t in found: out_pairs.append((line_kind, t.strip()))
|
||||||
|
else:
|
||||||
|
# Fallback für kommagetrennten Plaintext
|
||||||
|
for raw in re.split(r"[,;]", targets):
|
||||||
|
if raw.strip(): out_pairs.append((line_kind, raw.strip()))
|
||||||
|
|
||||||
|
# Wenn wir eine explizite Zeile gefunden haben, aktualisieren wir NICHT
|
||||||
|
# den current_kind für nachfolgende Zeilen (Design-Entscheidung: lokal scope),
|
||||||
|
# oder wir machen es doch?
|
||||||
|
# Üblicher ist: Header setzt Default, Zeile überschreibt lokal.
|
||||||
|
# Wir lassen current_kind also unangetastet.
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 2. Kein Key:Value Muster -> Prüfen auf Links, die den current_kind nutzen
|
||||||
|
found = _WIKILINK_RE.findall(bl)
|
||||||
|
if found:
|
||||||
|
if current_kind:
|
||||||
|
for t in found: out_pairs.append((current_kind, t.strip()))
|
||||||
|
else:
|
||||||
|
# Link ohne Typ und ohne Header-Typ.
|
||||||
|
# Wird ignoriert oder könnte als 'related_to' fallback dienen.
|
||||||
|
# Aktuell: Ignorieren, um False Positives zu vermeiden.
|
||||||
|
pass
|
||||||
|
|
||||||
|
return out_pairs, "\n".join(keep_lines)
|
||||||
|
|
||||||
|
def extract_wikilinks(text: str) -> List[str]:
|
||||||
|
"""Findet Standard-Wikilinks [[Target]] oder [[Alias|Target]]."""
|
||||||
|
if not text: return []
|
||||||
|
return [m.strip() for m in _WIKILINK_RE.findall(text) if m.strip()]
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/graph_subgraph.py
|
||||||
|
DESCRIPTION: In-Memory Repräsentation eines Graphen für Scoring und Analyse.
|
||||||
|
Zentrale Komponente für die Graph-Expansion (BFS) und Bonus-Berechnung.
|
||||||
|
WP-15c Update: Erhalt von Metadaten (target_section, provenance)
|
||||||
|
für präzises Retrieval-Reasoning.
|
||||||
|
VERSION: 1.2.0
|
||||||
|
STATUS: Active
|
||||||
|
"""
|
||||||
|
import math
|
||||||
|
from collections import defaultdict
|
||||||
|
from typing import Dict, List, Optional, DefaultDict, Any, Set
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
|
||||||
|
# Lokale Paket-Imports
|
||||||
|
from .graph_weights import EDGE_BASE_WEIGHTS, calculate_edge_weight
|
||||||
|
from .graph_db_adapter import fetch_edges_from_qdrant
|
||||||
|
|
||||||
|
class Subgraph:
|
||||||
|
"""
|
||||||
|
Leichtgewichtiger Subgraph mit Adjazenzlisten & Kennzahlen.
|
||||||
|
Wird für die Berechnung von Graph-Boni im Retriever genutzt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
# adj speichert nun vollständige Payloads statt nur Tripel
|
||||||
|
self.adj: DefaultDict[str, List[Dict]] = defaultdict(list)
|
||||||
|
self.reverse_adj: DefaultDict[str, List[Dict]] = defaultdict(list)
|
||||||
|
self.in_degree: DefaultDict[str, int] = defaultdict(int)
|
||||||
|
self.out_degree: DefaultDict[str, int] = defaultdict(int)
|
||||||
|
|
||||||
|
def add_edge(self, e: Dict) -> None:
|
||||||
|
"""
|
||||||
|
Fügt eine Kante hinzu und aktualisiert Indizes.
|
||||||
|
WP-15c: Speichert das vollständige Payload für den Explanation Layer.
|
||||||
|
"""
|
||||||
|
src = e.get("source")
|
||||||
|
tgt = e.get("target")
|
||||||
|
kind = e.get("kind")
|
||||||
|
|
||||||
|
# Das gesamte Payload wird als Kanten-Objekt behalten
|
||||||
|
# Wir stellen sicher, dass alle relevanten Metadaten vorhanden sind
|
||||||
|
edge_data = {
|
||||||
|
"source": src,
|
||||||
|
"target": tgt,
|
||||||
|
"kind": kind,
|
||||||
|
"weight": e.get("weight", EDGE_BASE_WEIGHTS.get(kind, 0.0)),
|
||||||
|
"provenance": e.get("provenance", "rule"),
|
||||||
|
"confidence": e.get("confidence", 1.0),
|
||||||
|
"target_section": e.get("target_section"), # Essentiell für Präzision
|
||||||
|
"is_super_edge": e.get("is_super_edge", False)
|
||||||
|
}
|
||||||
|
|
||||||
|
owner = e.get("note_id")
|
||||||
|
|
||||||
|
if not src or not tgt:
|
||||||
|
return
|
||||||
|
|
||||||
|
# 1. Forward-Kante
|
||||||
|
self.adj[src].append(edge_data)
|
||||||
|
self.out_degree[src] += 1
|
||||||
|
self.in_degree[tgt] += 1
|
||||||
|
|
||||||
|
# 2. Reverse-Kante (für Explanation Layer & Backlinks)
|
||||||
|
self.reverse_adj[tgt].append(edge_data)
|
||||||
|
|
||||||
|
# 3. Kontext-Note Handling (erhöht die Zentralität der Parent-Note)
|
||||||
|
if owner and owner != src:
|
||||||
|
# Wir erstellen eine virtuelle Kontext-Kante
|
||||||
|
ctx_edge = edge_data.copy()
|
||||||
|
ctx_edge["source"] = owner
|
||||||
|
ctx_edge["via_context"] = True
|
||||||
|
|
||||||
|
self.adj[owner].append(ctx_edge)
|
||||||
|
self.out_degree[owner] += 1
|
||||||
|
if owner != tgt:
|
||||||
|
self.reverse_adj[tgt].append(ctx_edge)
|
||||||
|
self.in_degree[owner] += 1
|
||||||
|
|
||||||
|
def aggregate_edge_bonus(self, node_id: str) -> float:
|
||||||
|
"""Summe der ausgehenden Kantengewichte (Hub-Score)."""
|
||||||
|
return sum(edge["weight"] for edge in self.adj.get(node_id, []))
|
||||||
|
|
||||||
|
def edge_bonus(self, node_id: str) -> float:
|
||||||
|
"""API für Retriever (WP-04a Kompatibilität)."""
|
||||||
|
return self.aggregate_edge_bonus(node_id)
|
||||||
|
|
||||||
|
def centrality_bonus(self, node_id: str) -> float:
|
||||||
|
"""
|
||||||
|
Log-gedämpfte Zentralität basierend auf dem In-Degree.
|
||||||
|
Begrenzt auf einen maximalen Boost von 0.15.
|
||||||
|
"""
|
||||||
|
indeg = self.in_degree.get(node_id, 0)
|
||||||
|
if indeg <= 0:
|
||||||
|
return 0.0
|
||||||
|
# math.log1p(x) entspricht log(1+x)
|
||||||
|
return min(math.log1p(indeg) / 10.0, 0.15)
|
||||||
|
|
||||||
|
def get_outgoing_edges(self, node_id: str) -> List[Dict[str, Any]]:
|
||||||
|
"""Gibt alle ausgehenden Kanten einer Node inkl. Metadaten zurück."""
|
||||||
|
return self.adj.get(node_id, [])
|
||||||
|
|
||||||
|
def get_incoming_edges(self, node_id: str) -> List[Dict[str, Any]]:
|
||||||
|
"""Gibt alle eingehenden Kanten einer Node inkl. Metadaten zurück."""
|
||||||
|
return self.reverse_adj.get(node_id, [])
|
||||||
|
|
||||||
|
|
||||||
|
def expand(
|
||||||
|
client: QdrantClient,
|
||||||
|
prefix: str,
|
||||||
|
seeds: List[str],
|
||||||
|
depth: int = 1,
|
||||||
|
edge_types: Optional[List[str]] = None,
|
||||||
|
) -> Subgraph:
|
||||||
|
"""
|
||||||
|
Expandiert ab Seeds entlang von Edges bis zu einer bestimmten Tiefe.
|
||||||
|
Nutzt fetch_edges_from_qdrant für den Datenbankzugriff.
|
||||||
|
"""
|
||||||
|
sg = Subgraph()
|
||||||
|
frontier = set(seeds)
|
||||||
|
visited = set()
|
||||||
|
|
||||||
|
for _ in range(max(depth, 0)):
|
||||||
|
if not frontier:
|
||||||
|
break
|
||||||
|
|
||||||
|
# Batch-Abfrage der Kanten für die aktuelle Ebene
|
||||||
|
payloads = fetch_edges_from_qdrant(client, prefix, list(frontier), edge_types)
|
||||||
|
next_frontier: Set[str] = set()
|
||||||
|
|
||||||
|
for pl in payloads:
|
||||||
|
src, tgt = pl.get("source_id"), pl.get("target_id")
|
||||||
|
if not src or not tgt: continue
|
||||||
|
|
||||||
|
# WP-15c: Wir übergeben das vollständige Payload an add_edge
|
||||||
|
edge_payload = {
|
||||||
|
"source": src,
|
||||||
|
"target": tgt,
|
||||||
|
"kind": pl.get("kind", "edge"),
|
||||||
|
"weight": calculate_edge_weight(pl),
|
||||||
|
"note_id": pl.get("note_id"),
|
||||||
|
"provenance": pl.get("provenance", "rule"),
|
||||||
|
"confidence": pl.get("confidence", 1.0),
|
||||||
|
"target_section": pl.get("target_section")
|
||||||
|
}
|
||||||
|
|
||||||
|
sg.add_edge(edge_payload)
|
||||||
|
|
||||||
|
# BFS Logik: Neue Ziele in die nächste Frontier aufnehmen
|
||||||
|
if tgt not in visited:
|
||||||
|
next_frontier.add(str(tgt))
|
||||||
|
|
||||||
|
visited |= frontier
|
||||||
|
frontier = next_frontier - visited
|
||||||
|
|
||||||
|
return sg
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/graph_utils.py
|
||||||
|
DESCRIPTION: Basale Werkzeuge, ID-Generierung und Provenance-Konfiguration für den Graphen.
|
||||||
|
AUDIT: Erweitert um parse_link_target für sauberes Section-Splitting (WP-Fix).
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import hashlib
|
||||||
|
from typing import Iterable, List, Optional, Set, Any, Tuple
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError:
|
||||||
|
yaml = None
|
||||||
|
|
||||||
|
# WP-15b: Prioritäten-Ranking für die De-Duplizierung
|
||||||
|
PROVENANCE_PRIORITY = {
|
||||||
|
"explicit:wikilink": 1.00,
|
||||||
|
"inline:rel": 0.95,
|
||||||
|
"callout:edge": 0.90,
|
||||||
|
"semantic_ai": 0.90, # Validierte KI-Kanten
|
||||||
|
"structure:belongs_to": 1.00,
|
||||||
|
"structure:order": 0.95, # next/prev
|
||||||
|
"explicit:note_scope": 1.00,
|
||||||
|
"derived:backlink": 0.90,
|
||||||
|
"edge_defaults": 0.70 # Heuristik (types.yaml)
|
||||||
|
}
|
||||||
|
|
||||||
|
def _get(d: dict, *keys, default=None):
|
||||||
|
"""Sicherer Zugriff auf verschachtelte Keys."""
|
||||||
|
for k in keys:
|
||||||
|
if isinstance(d, dict) and k in d and d[k] is not None:
|
||||||
|
return d[k]
|
||||||
|
return default
|
||||||
|
|
||||||
|
def _dedupe_seq(seq: Iterable[str]) -> List[str]:
|
||||||
|
"""Dedupliziert Strings unter Beibehaltung der Reihenfolge."""
|
||||||
|
seen: Set[str] = set()
|
||||||
|
out: List[str] = []
|
||||||
|
for s in seq:
|
||||||
|
if s not in seen:
|
||||||
|
seen.add(s); out.append(s)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _mk_edge_id(kind: str, s: str, t: str, scope: str, rule_id: Optional[str] = None, variant: Optional[str] = None) -> str:
|
||||||
|
"""
|
||||||
|
Erzeugt eine deterministische 12-Byte ID mittels BLAKE2s.
|
||||||
|
|
||||||
|
WP-Fix: 'variant' (z.B. Section) fließt in den Hash ein, um mehrere Kanten
|
||||||
|
zum gleichen Target-Node (aber unterschiedlichen Abschnitten) zu unterscheiden.
|
||||||
|
"""
|
||||||
|
base = f"{kind}:{s}->{t}#{scope}"
|
||||||
|
if rule_id:
|
||||||
|
base += f"|{rule_id}"
|
||||||
|
if variant:
|
||||||
|
base += f"|{variant}" # <--- Hier entsteht die Eindeutigkeit für verschiedene Sections
|
||||||
|
|
||||||
|
return hashlib.blake2s(base.encode("utf-8"), digest_size=12).hexdigest()
|
||||||
|
|
||||||
|
def _edge(kind: str, scope: str, source_id: str, target_id: str, note_id: str, extra: Optional[dict] = None) -> dict:
|
||||||
|
"""Konstruiert ein Kanten-Payload für Qdrant."""
|
||||||
|
pl = {
|
||||||
|
"kind": kind,
|
||||||
|
"relation": kind,
|
||||||
|
"scope": scope,
|
||||||
|
"source_id": source_id,
|
||||||
|
"target_id": target_id,
|
||||||
|
"note_id": note_id,
|
||||||
|
}
|
||||||
|
if extra: pl.update(extra)
|
||||||
|
return pl
|
||||||
|
|
||||||
|
def parse_link_target(raw: str, current_note_id: Optional[str] = None) -> Tuple[str, Optional[str]]:
|
||||||
|
"""
|
||||||
|
Zerlegt einen Link (z.B. 'Note#Section') in Target-ID und Section.
|
||||||
|
Behandelt Self-Links ('#Section'), indem current_note_id eingesetzt wird.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(target_id, target_section)
|
||||||
|
"""
|
||||||
|
if not raw:
|
||||||
|
return "", None
|
||||||
|
|
||||||
|
parts = raw.split("#", 1)
|
||||||
|
target = parts[0].strip()
|
||||||
|
section = parts[1].strip() if len(parts) > 1 else None
|
||||||
|
|
||||||
|
# Handle Self-Link [[#Section]] -> target wird zu current_note_id
|
||||||
|
if not target and section and current_note_id:
|
||||||
|
target = current_note_id
|
||||||
|
|
||||||
|
return target, section
|
||||||
|
|
||||||
|
def load_types_registry() -> dict:
|
||||||
|
"""Lädt die YAML-Registry."""
|
||||||
|
p = os.getenv("MINDNET_TYPES_FILE", "./config/types.yaml")
|
||||||
|
if not os.path.isfile(p) or yaml is None: return {}
|
||||||
|
try:
|
||||||
|
with open(p, "r", encoding="utf-8") as f: return yaml.safe_load(f) or {}
|
||||||
|
except Exception: return {}
|
||||||
|
|
||||||
|
def get_edge_defaults_for(note_type: Optional[str], reg: dict) -> List[str]:
|
||||||
|
"""Ermittelt Standard-Kanten für einen Typ."""
|
||||||
|
types_map = reg.get("types", reg) if isinstance(reg, dict) else {}
|
||||||
|
if note_type and isinstance(types_map, dict):
|
||||||
|
t = types_map.get(note_type)
|
||||||
|
if isinstance(t, dict) and isinstance(t.get("edge_defaults"), list):
|
||||||
|
return [str(x) for x in t["edge_defaults"] if isinstance(x, str)]
|
||||||
|
for key in ("defaults", "default", "global"):
|
||||||
|
v = reg.get(key)
|
||||||
|
if isinstance(v, dict) and isinstance(v.get("edge_defaults"), list):
|
||||||
|
return [str(x) for x in v["edge_defaults"] if isinstance(x, str)]
|
||||||
|
return []
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/graph/graph_weights.py
|
||||||
|
DESCRIPTION: Definition der Basisgewichte und Berechnung der Kanteneffektivität.
|
||||||
|
"""
|
||||||
|
from typing import Dict
|
||||||
|
|
||||||
|
# Basisgewichte je Edge-Typ (WP-04a Config)
|
||||||
|
EDGE_BASE_WEIGHTS: Dict[str, float] = {
|
||||||
|
# Struktur
|
||||||
|
"belongs_to": 0.10,
|
||||||
|
"next": 0.06,
|
||||||
|
"prev": 0.06,
|
||||||
|
"backlink": 0.04,
|
||||||
|
"references_at": 0.08,
|
||||||
|
|
||||||
|
# Wissen
|
||||||
|
"references": 0.20,
|
||||||
|
"depends_on": 0.18,
|
||||||
|
"related_to": 0.15,
|
||||||
|
"similar_to": 0.12,
|
||||||
|
}
|
||||||
|
|
||||||
|
def calculate_edge_weight(pl: Dict) -> float:
|
||||||
|
"""Berechnet das effektive Edge-Gewicht aus kind + confidence."""
|
||||||
|
kind = pl.get("kind", "edge")
|
||||||
|
base = EDGE_BASE_WEIGHTS.get(kind, 0.0)
|
||||||
|
|
||||||
|
conf_raw = pl.get("confidence", None)
|
||||||
|
try:
|
||||||
|
conf = float(conf_raw) if conf_raw is not None else None
|
||||||
|
except Exception:
|
||||||
|
conf = None
|
||||||
|
|
||||||
|
if conf is None:
|
||||||
|
return base
|
||||||
|
|
||||||
|
# Clamp confidence 0.0 - 1.0
|
||||||
|
conf = max(0.0, min(1.0, conf))
|
||||||
|
return base * conf
|
||||||
@@ -1,256 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
app/core/graph_adapter.py — Adjazenzaufbau & Subgraph-Expansion
|
|
||||||
|
|
||||||
Zweck:
|
|
||||||
Baut aus Qdrant-Edges (Collection: *_edges) einen leichten In-Memory-Graph.
|
|
||||||
|
|
||||||
Kompatibilität:
|
|
||||||
- WP-04a: Liefert Scores (edge_bonus, centrality).
|
|
||||||
- WP-04b: Liefert jetzt auch Struktur-Daten für Erklärungen (Reverse-Lookup).
|
|
||||||
|
|
||||||
Version:
|
|
||||||
0.4.0 (Update für WP-04b: Reverse Adjacency für Explainability)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Dict, List, Optional, DefaultDict, Any
|
|
||||||
from collections import defaultdict
|
|
||||||
|
|
||||||
from qdrant_client import QdrantClient
|
|
||||||
from qdrant_client.http import models as rest
|
|
||||||
|
|
||||||
from app.core.qdrant import collection_names
|
|
||||||
|
|
||||||
# Legacy-Import Fallback
|
|
||||||
try: # pragma: no cover
|
|
||||||
from app.core.qdrant_points import get_edges_for_sources # type: ignore
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
get_edges_for_sources = None # type: ignore
|
|
||||||
|
|
||||||
|
|
||||||
# Basisgewichte je Edge-Typ (WP-04a Config)
|
|
||||||
EDGE_BASE_WEIGHTS: Dict[str, float] = {
|
|
||||||
# Struktur
|
|
||||||
"belongs_to": 0.10,
|
|
||||||
"next": 0.06,
|
|
||||||
"prev": 0.06,
|
|
||||||
"backlink": 0.04,
|
|
||||||
"references_at": 0.08,
|
|
||||||
|
|
||||||
# Wissen
|
|
||||||
"references": 0.20,
|
|
||||||
"depends_on": 0.18,
|
|
||||||
"related_to": 0.15,
|
|
||||||
"similar_to": 0.12,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _edge_weight(pl: Dict) -> float:
|
|
||||||
"""Berechnet das effektive Edge-Gewicht aus kind + confidence."""
|
|
||||||
kind = pl.get("kind", "edge")
|
|
||||||
base = EDGE_BASE_WEIGHTS.get(kind, 0.0)
|
|
||||||
|
|
||||||
conf_raw = pl.get("confidence", None)
|
|
||||||
try:
|
|
||||||
conf = float(conf_raw) if conf_raw is not None else None
|
|
||||||
except Exception:
|
|
||||||
conf = None
|
|
||||||
|
|
||||||
if conf is None:
|
|
||||||
return base
|
|
||||||
|
|
||||||
if conf < 0.0: conf = 0.0
|
|
||||||
if conf > 1.0: conf = 1.0
|
|
||||||
|
|
||||||
return base * conf
|
|
||||||
|
|
||||||
|
|
||||||
def _fetch_edges(
|
|
||||||
client: QdrantClient,
|
|
||||||
prefix: str,
|
|
||||||
seeds: List[str],
|
|
||||||
edge_types: Optional[List[str]] = None,
|
|
||||||
limit: int = 2048,
|
|
||||||
) -> List[Dict]:
|
|
||||||
"""
|
|
||||||
Holt Edges direkt aus der *_edges Collection.
|
|
||||||
Filter: source_id IN seeds OR target_id IN seeds OR note_id IN seeds
|
|
||||||
"""
|
|
||||||
if not seeds or limit <= 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
_, _, edges_col = collection_names(prefix)
|
|
||||||
|
|
||||||
seed_conditions = []
|
|
||||||
for field in ("source_id", "target_id", "note_id"):
|
|
||||||
for s in seeds:
|
|
||||||
seed_conditions.append(
|
|
||||||
rest.FieldCondition(key=field, match=rest.MatchValue(value=str(s)))
|
|
||||||
)
|
|
||||||
seeds_filter = rest.Filter(should=seed_conditions) if seed_conditions else None
|
|
||||||
|
|
||||||
type_filter = None
|
|
||||||
if edge_types:
|
|
||||||
type_conds = [
|
|
||||||
rest.FieldCondition(key="kind", match=rest.MatchValue(value=str(k)))
|
|
||||||
for k in edge_types
|
|
||||||
]
|
|
||||||
type_filter = rest.Filter(should=type_conds)
|
|
||||||
|
|
||||||
must = []
|
|
||||||
if seeds_filter: must.append(seeds_filter)
|
|
||||||
if type_filter: must.append(type_filter)
|
|
||||||
|
|
||||||
flt = rest.Filter(must=must) if must else None
|
|
||||||
|
|
||||||
pts, _ = client.scroll(
|
|
||||||
collection_name=edges_col,
|
|
||||||
scroll_filter=flt,
|
|
||||||
limit=limit,
|
|
||||||
with_payload=True,
|
|
||||||
with_vectors=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
out: List[Dict] = []
|
|
||||||
for p in pts or []:
|
|
||||||
pl = dict(p.payload or {})
|
|
||||||
if pl:
|
|
||||||
out.append(pl)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
class Subgraph:
|
|
||||||
"""Leichtgewichtiger Subgraph mit Adjazenzlisten & Kennzahlen."""
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
|
||||||
# Forward: source -> [targets]
|
|
||||||
self.adj: DefaultDict[str, List[Dict]] = defaultdict(list)
|
|
||||||
# Reverse: target -> [sources] (Neu für WP-04b Explanation)
|
|
||||||
self.reverse_adj: DefaultDict[str, List[Dict]] = defaultdict(list)
|
|
||||||
|
|
||||||
self.in_degree: DefaultDict[str, int] = defaultdict(int)
|
|
||||||
self.out_degree: DefaultDict[str, int] = defaultdict(int)
|
|
||||||
|
|
||||||
def add_edge(self, e: Dict) -> None:
|
|
||||||
"""
|
|
||||||
Fügt eine Kante hinzu und aktualisiert Forward/Reverse Indizes.
|
|
||||||
e muss enthalten: source, target, kind, weight.
|
|
||||||
"""
|
|
||||||
src = e.get("source")
|
|
||||||
tgt = e.get("target")
|
|
||||||
kind = e.get("kind")
|
|
||||||
weight = e.get("weight", EDGE_BASE_WEIGHTS.get(kind, 0.0))
|
|
||||||
owner = e.get("note_id")
|
|
||||||
|
|
||||||
if not src or not tgt:
|
|
||||||
return
|
|
||||||
|
|
||||||
# 1. Primäre Adjazenz (Forward)
|
|
||||||
edge_data = {"target": tgt, "kind": kind, "weight": weight}
|
|
||||||
self.adj[src].append(edge_data)
|
|
||||||
self.out_degree[src] += 1
|
|
||||||
self.in_degree[tgt] += 1
|
|
||||||
|
|
||||||
# 2. Reverse Adjazenz (Neu für Explanation)
|
|
||||||
# Wir speichern, woher die Kante kam.
|
|
||||||
rev_data = {"source": src, "kind": kind, "weight": weight}
|
|
||||||
self.reverse_adj[tgt].append(rev_data)
|
|
||||||
|
|
||||||
# 3. Kontext-Note Handling (Forward & Reverse)
|
|
||||||
# Wenn eine Kante "im Kontext einer Note" (owner) definiert ist,
|
|
||||||
# schreiben wir sie der Note gut, damit der Retriever Scores auf Note-Ebene findet.
|
|
||||||
if owner and owner != src:
|
|
||||||
# Forward: Owner -> Target
|
|
||||||
self.adj[owner].append(edge_data)
|
|
||||||
self.out_degree[owner] += 1
|
|
||||||
|
|
||||||
# Reverse: Target wird vom Owner referenziert (indirekt)
|
|
||||||
if owner != tgt:
|
|
||||||
rev_owner_data = {"source": owner, "kind": kind, "weight": weight, "via_context": True}
|
|
||||||
self.reverse_adj[tgt].append(rev_owner_data)
|
|
||||||
self.in_degree[owner] += 1 # Leichter Centrality Boost für den Owner
|
|
||||||
|
|
||||||
def aggregate_edge_bonus(self, node_id: str) -> float:
|
|
||||||
"""Summe der ausgehenden Kantengewichte (Hub-Score)."""
|
|
||||||
return sum(edge["weight"] for edge in self.adj.get(node_id, []))
|
|
||||||
|
|
||||||
def edge_bonus(self, node_id: str) -> float:
|
|
||||||
"""API für Retriever (WP-04a Kompatibilität)."""
|
|
||||||
return self.aggregate_edge_bonus(node_id)
|
|
||||||
|
|
||||||
def centrality_bonus(self, node_id: str) -> float:
|
|
||||||
"""Log-gedämpfte Zentralität (In-Degree)."""
|
|
||||||
import math
|
|
||||||
indeg = self.in_degree.get(node_id, 0)
|
|
||||||
if indeg <= 0:
|
|
||||||
return 0.0
|
|
||||||
return min(math.log1p(indeg) / 10.0, 0.15)
|
|
||||||
|
|
||||||
# --- WP-04b Explanation Helpers ---
|
|
||||||
|
|
||||||
def get_outgoing_edges(self, node_id: str) -> List[Dict[str, Any]]:
|
|
||||||
"""Liefert Liste aller Ziele, auf die dieser Knoten zeigt."""
|
|
||||||
return self.adj.get(node_id, [])
|
|
||||||
|
|
||||||
def get_incoming_edges(self, node_id: str) -> List[Dict[str, Any]]:
|
|
||||||
"""Liefert Liste aller Quellen, die auf diesen Knoten zeigen."""
|
|
||||||
return self.reverse_adj.get(node_id, [])
|
|
||||||
|
|
||||||
|
|
||||||
def expand(
|
|
||||||
client: QdrantClient,
|
|
||||||
prefix: str,
|
|
||||||
seeds: List[str],
|
|
||||||
depth: int = 1,
|
|
||||||
edge_types: Optional[List[str]] = None,
|
|
||||||
) -> Subgraph:
|
|
||||||
"""
|
|
||||||
Expandiert ab Seeds entlang von Edges (bis `depth`).
|
|
||||||
"""
|
|
||||||
sg = Subgraph()
|
|
||||||
frontier = set(seeds)
|
|
||||||
visited = set()
|
|
||||||
|
|
||||||
max_depth = max(depth, 0)
|
|
||||||
|
|
||||||
for _ in range(max_depth):
|
|
||||||
if not frontier:
|
|
||||||
break
|
|
||||||
|
|
||||||
edges_payloads = _fetch_edges(
|
|
||||||
client=client,
|
|
||||||
prefix=prefix,
|
|
||||||
seeds=list(frontier),
|
|
||||||
edge_types=edge_types,
|
|
||||||
limit=2048,
|
|
||||||
)
|
|
||||||
|
|
||||||
next_frontier = set()
|
|
||||||
for pl in edges_payloads:
|
|
||||||
src = pl.get("source_id")
|
|
||||||
tgt = pl.get("target_id")
|
|
||||||
|
|
||||||
# Skip invalid edges
|
|
||||||
if not src or not tgt:
|
|
||||||
continue
|
|
||||||
|
|
||||||
e = {
|
|
||||||
"source": src,
|
|
||||||
"target": tgt,
|
|
||||||
"kind": pl.get("kind", "edge"),
|
|
||||||
"weight": _edge_weight(pl),
|
|
||||||
"note_id": pl.get("note_id"),
|
|
||||||
}
|
|
||||||
sg.add_edge(e)
|
|
||||||
|
|
||||||
# Nur weitersuchen, wenn Target noch nicht besucht
|
|
||||||
if tgt and tgt not in visited:
|
|
||||||
next_frontier.add(tgt)
|
|
||||||
|
|
||||||
visited |= frontier
|
|
||||||
frontier = next_frontier - visited
|
|
||||||
|
|
||||||
return sg
|
|
||||||
@@ -1,305 +0,0 @@
|
|||||||
"""
|
|
||||||
app/core/ingestion.py
|
|
||||||
|
|
||||||
Zentraler Service für die Transformation von Markdown-Dateien in Qdrant-Objekte.
|
|
||||||
Version: 2.5.2 (Full Feature: Change Detection + Robust IO + Clean Config)
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import logging
|
|
||||||
import asyncio
|
|
||||||
import time
|
|
||||||
from typing import Dict, List, Optional, Tuple, Any
|
|
||||||
|
|
||||||
# Core Module Imports
|
|
||||||
from app.core.parser import (
|
|
||||||
read_markdown,
|
|
||||||
normalize_frontmatter,
|
|
||||||
validate_required_frontmatter,
|
|
||||||
)
|
|
||||||
from app.core.note_payload import make_note_payload
|
|
||||||
from app.core.chunker import assemble_chunks, get_chunk_config
|
|
||||||
from app.core.chunk_payload import make_chunk_payloads
|
|
||||||
|
|
||||||
# Fallback für Edges
|
|
||||||
try:
|
|
||||||
from app.core.derive_edges import build_edges_for_note
|
|
||||||
except ImportError:
|
|
||||||
def build_edges_for_note(*args, **kwargs): return []
|
|
||||||
|
|
||||||
from app.core.qdrant import QdrantConfig, get_client, ensure_collections, ensure_payload_indexes
|
|
||||||
from app.core.qdrant_points import (
|
|
||||||
points_for_chunks,
|
|
||||||
points_for_note,
|
|
||||||
points_for_edges,
|
|
||||||
upsert_batch,
|
|
||||||
)
|
|
||||||
|
|
||||||
from app.services.embeddings_client import EmbeddingsClient
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# --- Helper ---
|
|
||||||
def load_type_registry(custom_path: Optional[str] = None) -> dict:
|
|
||||||
import yaml
|
|
||||||
path = custom_path or os.getenv("MINDNET_TYPES_FILE", "config/types.yaml")
|
|
||||||
if not os.path.exists(path): return {}
|
|
||||||
try:
|
|
||||||
with open(path, "r", encoding="utf-8") as f: return yaml.safe_load(f) or {}
|
|
||||||
except Exception: return {}
|
|
||||||
|
|
||||||
def resolve_note_type(requested: Optional[str], reg: dict) -> str:
|
|
||||||
types = reg.get("types", {})
|
|
||||||
if requested and requested in types: return requested
|
|
||||||
return "concept"
|
|
||||||
|
|
||||||
def effective_chunk_profile(note_type: str, reg: dict) -> str:
|
|
||||||
t_cfg = reg.get("types", {}).get(note_type, {})
|
|
||||||
if t_cfg and t_cfg.get("chunk_profile"):
|
|
||||||
return t_cfg.get("chunk_profile")
|
|
||||||
return reg.get("defaults", {}).get("chunk_profile", "default")
|
|
||||||
|
|
||||||
def effective_retriever_weight(note_type: str, reg: dict) -> float:
|
|
||||||
t_cfg = reg.get("types", {}).get(note_type, {})
|
|
||||||
if t_cfg and "retriever_weight" in t_cfg:
|
|
||||||
return float(t_cfg["retriever_weight"])
|
|
||||||
return float(reg.get("defaults", {}).get("retriever_weight", 1.0))
|
|
||||||
|
|
||||||
|
|
||||||
class IngestionService:
|
|
||||||
def __init__(self, collection_prefix: str = None):
|
|
||||||
env_prefix = os.getenv("COLLECTION_PREFIX", "mindnet")
|
|
||||||
self.prefix = collection_prefix or env_prefix
|
|
||||||
|
|
||||||
self.cfg = QdrantConfig.from_env()
|
|
||||||
self.cfg.prefix = self.prefix
|
|
||||||
self.client = get_client(self.cfg)
|
|
||||||
self.dim = self.cfg.dim
|
|
||||||
self.registry = load_type_registry()
|
|
||||||
self.embedder = EmbeddingsClient()
|
|
||||||
|
|
||||||
try:
|
|
||||||
ensure_collections(self.client, self.prefix, self.dim)
|
|
||||||
ensure_payload_indexes(self.client, self.prefix)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"DB init warning: {e}")
|
|
||||||
|
|
||||||
async def process_file(
|
|
||||||
self,
|
|
||||||
file_path: str,
|
|
||||||
vault_root: str,
|
|
||||||
force_replace: bool = False,
|
|
||||||
apply: bool = False,
|
|
||||||
purge_before: bool = False,
|
|
||||||
note_scope_refs: bool = False,
|
|
||||||
hash_mode: str = "body",
|
|
||||||
hash_source: str = "parsed",
|
|
||||||
hash_normalize: str = "canonical"
|
|
||||||
) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Verarbeitet eine einzelne Datei (ASYNC).
|
|
||||||
Inklusive Change Detection (Hash-Check) gegen Qdrant.
|
|
||||||
"""
|
|
||||||
result = {
|
|
||||||
"path": file_path,
|
|
||||||
"status": "skipped",
|
|
||||||
"changed": False,
|
|
||||||
"error": None
|
|
||||||
}
|
|
||||||
|
|
||||||
# 1. Parse & Frontmatter Validation
|
|
||||||
try:
|
|
||||||
parsed = read_markdown(file_path)
|
|
||||||
if not parsed:
|
|
||||||
return {**result, "error": "Empty or unreadable file"}
|
|
||||||
|
|
||||||
fm = normalize_frontmatter(parsed.frontmatter)
|
|
||||||
validate_required_frontmatter(fm)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Validation failed for {file_path}: {e}")
|
|
||||||
return {**result, "error": f"Validation failed: {str(e)}"}
|
|
||||||
|
|
||||||
# 2. Type & Config Resolution
|
|
||||||
note_type = resolve_note_type(fm.get("type"), self.registry)
|
|
||||||
fm["type"] = note_type
|
|
||||||
fm["chunk_profile"] = effective_chunk_profile(note_type, self.registry)
|
|
||||||
|
|
||||||
weight = fm.get("retriever_weight")
|
|
||||||
if weight is None:
|
|
||||||
weight = effective_retriever_weight(note_type, self.registry)
|
|
||||||
fm["retriever_weight"] = float(weight)
|
|
||||||
|
|
||||||
# 3. Build Note Payload
|
|
||||||
try:
|
|
||||||
note_pl = make_note_payload(
|
|
||||||
parsed,
|
|
||||||
vault_root=vault_root,
|
|
||||||
hash_mode=hash_mode,
|
|
||||||
hash_normalize=hash_normalize,
|
|
||||||
hash_source=hash_source,
|
|
||||||
file_path=file_path
|
|
||||||
)
|
|
||||||
if not note_pl.get("fulltext"):
|
|
||||||
note_pl["fulltext"] = getattr(parsed, "body", "") or ""
|
|
||||||
note_pl["retriever_weight"] = fm["retriever_weight"]
|
|
||||||
|
|
||||||
note_id = note_pl["note_id"]
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Payload build failed: {e}")
|
|
||||||
return {**result, "error": f"Payload build failed: {str(e)}"}
|
|
||||||
|
|
||||||
# 4. Change Detection (Das fehlende Stück!)
|
|
||||||
old_payload = None
|
|
||||||
if not force_replace:
|
|
||||||
old_payload = self._fetch_note_payload(note_id)
|
|
||||||
|
|
||||||
has_old = old_payload is not None
|
|
||||||
key_current = f"{hash_mode}:{hash_source}:{hash_normalize}"
|
|
||||||
old_hash = (old_payload or {}).get("hashes", {}).get(key_current)
|
|
||||||
new_hash = note_pl.get("hashes", {}).get(key_current)
|
|
||||||
|
|
||||||
hash_changed = (old_hash != new_hash)
|
|
||||||
chunks_missing, edges_missing = self._artifacts_missing(note_id)
|
|
||||||
|
|
||||||
should_write = force_replace or (not has_old) or hash_changed or chunks_missing or edges_missing
|
|
||||||
|
|
||||||
if not should_write:
|
|
||||||
return {**result, "status": "unchanged", "note_id": note_id}
|
|
||||||
|
|
||||||
if not apply:
|
|
||||||
return {**result, "status": "dry-run", "changed": True, "note_id": note_id}
|
|
||||||
|
|
||||||
# 5. Processing (Chunking, Embedding, Edges)
|
|
||||||
try:
|
|
||||||
body_text = getattr(parsed, "body", "") or ""
|
|
||||||
|
|
||||||
# --- Config Loading (Clean) ---
|
|
||||||
chunk_config = get_chunk_config(note_type)
|
|
||||||
# Hier greift die Logik aus types.yaml (smart=True/False)
|
|
||||||
|
|
||||||
chunks = await assemble_chunks(fm["id"], body_text, fm["type"], config=chunk_config)
|
|
||||||
chunk_pls = make_chunk_payloads(fm, note_pl["path"], chunks, note_text=body_text)
|
|
||||||
|
|
||||||
# Embedding
|
|
||||||
vecs = []
|
|
||||||
if chunk_pls:
|
|
||||||
texts = [c.get("window") or c.get("text") or "" for c in chunk_pls]
|
|
||||||
try:
|
|
||||||
if hasattr(self.embedder, 'embed_documents'):
|
|
||||||
vecs = await self.embedder.embed_documents(texts)
|
|
||||||
else:
|
|
||||||
for t in texts:
|
|
||||||
v = await self.embedder.embed_query(t)
|
|
||||||
vecs.append(v)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Embedding failed: {e}")
|
|
||||||
raise RuntimeError(f"Embedding failed: {e}")
|
|
||||||
|
|
||||||
# Edges
|
|
||||||
try:
|
|
||||||
edges = build_edges_for_note(
|
|
||||||
note_id,
|
|
||||||
chunk_pls,
|
|
||||||
note_level_references=note_pl.get("references", []),
|
|
||||||
include_note_scope_refs=note_scope_refs
|
|
||||||
)
|
|
||||||
except TypeError:
|
|
||||||
edges = build_edges_for_note(note_id, chunk_pls)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Processing failed: {e}", exc_info=True)
|
|
||||||
return {**result, "error": f"Processing failed: {str(e)}"}
|
|
||||||
|
|
||||||
# 6. Upsert Action
|
|
||||||
try:
|
|
||||||
if purge_before and has_old:
|
|
||||||
self._purge_artifacts(note_id)
|
|
||||||
|
|
||||||
n_name, n_pts = points_for_note(self.prefix, note_pl, None, self.dim)
|
|
||||||
upsert_batch(self.client, n_name, n_pts)
|
|
||||||
|
|
||||||
if chunk_pls and vecs:
|
|
||||||
c_name, c_pts = points_for_chunks(self.prefix, chunk_pls, vecs)
|
|
||||||
upsert_batch(self.client, c_name, c_pts)
|
|
||||||
|
|
||||||
if edges:
|
|
||||||
e_name, e_pts = points_for_edges(self.prefix, edges)
|
|
||||||
upsert_batch(self.client, e_name, e_pts)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"path": file_path,
|
|
||||||
"status": "success",
|
|
||||||
"changed": True,
|
|
||||||
"note_id": note_id,
|
|
||||||
"chunks_count": len(chunk_pls),
|
|
||||||
"edges_count": len(edges)
|
|
||||||
}
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Upsert failed: {e}", exc_info=True)
|
|
||||||
return {**result, "error": f"DB Upsert failed: {e}"}
|
|
||||||
|
|
||||||
# --- Qdrant Helper (Restored) ---
|
|
||||||
|
|
||||||
def _fetch_note_payload(self, note_id: str) -> Optional[dict]:
|
|
||||||
from qdrant_client.http import models as rest
|
|
||||||
col = f"{self.prefix}_notes"
|
|
||||||
try:
|
|
||||||
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
|
||||||
pts, _ = self.client.scroll(collection_name=col, scroll_filter=f, limit=1, with_payload=True)
|
|
||||||
return pts[0].payload if pts else None
|
|
||||||
except: return None
|
|
||||||
|
|
||||||
def _artifacts_missing(self, note_id: str) -> Tuple[bool, bool]:
|
|
||||||
from qdrant_client.http import models as rest
|
|
||||||
c_col = f"{self.prefix}_chunks"
|
|
||||||
e_col = f"{self.prefix}_edges"
|
|
||||||
try:
|
|
||||||
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
|
||||||
c_pts, _ = self.client.scroll(collection_name=c_col, scroll_filter=f, limit=1)
|
|
||||||
e_pts, _ = self.client.scroll(collection_name=e_col, scroll_filter=f, limit=1)
|
|
||||||
return (not bool(c_pts)), (not bool(e_pts))
|
|
||||||
except: return True, True
|
|
||||||
|
|
||||||
def _purge_artifacts(self, note_id: str):
|
|
||||||
from qdrant_client.http import models as rest
|
|
||||||
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
|
||||||
selector = rest.FilterSelector(filter=f)
|
|
||||||
for suffix in ["chunks", "edges"]:
|
|
||||||
try:
|
|
||||||
self.client.delete(collection_name=f"{self.prefix}_{suffix}", points_selector=selector)
|
|
||||||
except Exception: pass
|
|
||||||
|
|
||||||
async def create_from_text(
|
|
||||||
self,
|
|
||||||
markdown_content: str,
|
|
||||||
filename: str,
|
|
||||||
vault_root: str,
|
|
||||||
folder: str = "00_Inbox"
|
|
||||||
) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
WP-11 Persistence API Entrypoint.
|
|
||||||
"""
|
|
||||||
target_dir = os.path.join(vault_root, folder)
|
|
||||||
os.makedirs(target_dir, exist_ok=True)
|
|
||||||
|
|
||||||
file_path = os.path.join(target_dir, filename)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Robust Write: Ensure Flush & Sync
|
|
||||||
with open(file_path, "w", encoding="utf-8") as f:
|
|
||||||
f.write(markdown_content)
|
|
||||||
f.flush()
|
|
||||||
os.fsync(f.fileno())
|
|
||||||
|
|
||||||
await asyncio.sleep(0.1)
|
|
||||||
|
|
||||||
logger.info(f"Written file to {file_path}")
|
|
||||||
except Exception as e:
|
|
||||||
return {"status": "error", "error": f"Disk write failed: {str(e)}"}
|
|
||||||
|
|
||||||
return await self.process_file(
|
|
||||||
file_path=file_path,
|
|
||||||
vault_root=vault_root,
|
|
||||||
apply=True,
|
|
||||||
force_replace=True,
|
|
||||||
purge_before=True
|
|
||||||
)
|
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/__init__.py
|
||||||
|
DESCRIPTION: Package-Einstiegspunkt für Ingestion. Exportiert den IngestionService.
|
||||||
|
AUDIT v2.13.10: Abschluss der Modularisierung (WP-14).
|
||||||
|
Bricht Zirkelbezüge durch Nutzung der neutralen registry.py auf.
|
||||||
|
VERSION: 2.13.10
|
||||||
|
"""
|
||||||
|
# Der IngestionService ist der primäre Orchestrator für den Datenimport
|
||||||
|
from .ingestion_processor import IngestionService
|
||||||
|
|
||||||
|
# Hilfswerkzeuge für JSON-Verarbeitung und Konfigurations-Management
|
||||||
|
# load_type_registry wird hier re-exportiert, um die Abwärtskompatibilität zu wahren,
|
||||||
|
# obwohl die Implementierung nun in app.core.registry liegt.
|
||||||
|
from .ingestion_utils import (
|
||||||
|
extract_json_from_response,
|
||||||
|
load_type_registry,
|
||||||
|
resolve_note_type
|
||||||
|
)
|
||||||
|
|
||||||
|
# Öffentliche API des Pakets
|
||||||
|
__all__ = [
|
||||||
|
"IngestionService",
|
||||||
|
"extract_json_from_response",
|
||||||
|
"load_type_registry",
|
||||||
|
"resolve_note_type"
|
||||||
|
]
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/ingestion_chunk_payload.py
|
||||||
|
DESCRIPTION: Baut das JSON-Objekt für 'mindnet_chunks'.
|
||||||
|
Fix v2.4.3: Integration der zentralen Registry (WP-14) für konsistente Defaults.
|
||||||
|
VERSION: 2.4.3
|
||||||
|
STATUS: Active
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
# ENTSCHEIDENDER FIX: Import der neutralen Registry-Logik zur Vermeidung von Circular Imports
|
||||||
|
from app.core.registry import load_type_registry
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Resolution Helpers (Audited)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _as_list(x):
|
||||||
|
"""Sichert die Listen-Integrität für Metadaten wie Tags."""
|
||||||
|
if x is None: return []
|
||||||
|
return x if isinstance(x, list) else [x]
|
||||||
|
|
||||||
|
def _resolve_val(note_type: str, reg: dict, key: str, default: Any) -> Any:
|
||||||
|
"""
|
||||||
|
Hierarchische Suche in der Registry: Type-Spezifisch > Globaler Default.
|
||||||
|
WP-14: Erlaubt dynamische Konfiguration via types.yaml.
|
||||||
|
"""
|
||||||
|
types = reg.get("types", {})
|
||||||
|
if isinstance(types, dict):
|
||||||
|
t_cfg = types.get(note_type, {})
|
||||||
|
if isinstance(t_cfg, dict):
|
||||||
|
# Fallback für Key-Varianten (z.B. chunking_profile vs chunk_profile)
|
||||||
|
val = t_cfg.get(key) or t_cfg.get(key.replace("ing", ""))
|
||||||
|
if val is not None: return val
|
||||||
|
|
||||||
|
defs = reg.get("defaults", {}) or reg.get("global", {})
|
||||||
|
if isinstance(defs, dict):
|
||||||
|
val = defs.get(key) or defs.get(key.replace("ing", ""))
|
||||||
|
if val is not None: return val
|
||||||
|
|
||||||
|
return default
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Haupt-API
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def make_chunk_payloads(note: Dict[str, Any], note_path: str, chunks_from_chunker: List[Any], **kwargs) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Erstellt die Payloads für die Chunks inklusive Audit-Resolution.
|
||||||
|
Nutzt nun die zentrale Registry für alle Fallbacks.
|
||||||
|
"""
|
||||||
|
if isinstance(note, dict) and "frontmatter" in note:
|
||||||
|
fm = note["frontmatter"]
|
||||||
|
else:
|
||||||
|
fm = note or {}
|
||||||
|
|
||||||
|
# WP-14 Fix: Nutzt übergebene Registry oder lädt sie global
|
||||||
|
reg = kwargs.get("types_cfg") or load_type_registry()
|
||||||
|
|
||||||
|
note_type = fm.get("type") or "concept"
|
||||||
|
title = fm.get("title") or fm.get("id") or "Untitled"
|
||||||
|
tags = _as_list(fm.get("tags") or [])
|
||||||
|
|
||||||
|
# Audit: Resolution Hierarchie (Frontmatter > Registry)
|
||||||
|
cp = fm.get("chunking_profile") or fm.get("chunk_profile")
|
||||||
|
if not cp:
|
||||||
|
cp = _resolve_val(note_type, reg, "chunking_profile", "sliding_standard")
|
||||||
|
|
||||||
|
rw = fm.get("retriever_weight")
|
||||||
|
if rw is None:
|
||||||
|
rw = _resolve_val(note_type, reg, "retriever_weight", 1.0)
|
||||||
|
try:
|
||||||
|
rw = float(rw)
|
||||||
|
except:
|
||||||
|
rw = 1.0
|
||||||
|
|
||||||
|
out: List[Dict[str, Any]] = []
|
||||||
|
for idx, ch in enumerate(chunks_from_chunker):
|
||||||
|
is_dict = isinstance(ch, dict)
|
||||||
|
cid = getattr(ch, "id", None) if not is_dict else ch.get("id")
|
||||||
|
nid = getattr(ch, "note_id", None) if not is_dict else ch.get("note_id")
|
||||||
|
index = getattr(ch, "index", idx) if not is_dict else ch.get("index", idx)
|
||||||
|
text = getattr(ch, "text", "") if not is_dict else ch.get("text", "")
|
||||||
|
window = getattr(ch, "window", text) if not is_dict else ch.get("window", text)
|
||||||
|
prev_id = getattr(ch, "neighbors_prev", None) if not is_dict else ch.get("neighbors_prev")
|
||||||
|
next_id = getattr(ch, "neighbors_next", None) if not is_dict else ch.get("neighbors_next")
|
||||||
|
section = getattr(ch, "section_title", "") if not is_dict else ch.get("section", "")
|
||||||
|
|
||||||
|
pl: Dict[str, Any] = {
|
||||||
|
"note_id": nid or fm.get("id"),
|
||||||
|
"chunk_id": cid,
|
||||||
|
"title": title,
|
||||||
|
"index": int(index),
|
||||||
|
"ord": int(index) + 1,
|
||||||
|
"type": note_type,
|
||||||
|
"tags": tags,
|
||||||
|
"text": text,
|
||||||
|
"window": window,
|
||||||
|
"neighbors_prev": _as_list(prev_id),
|
||||||
|
"neighbors_next": _as_list(next_id),
|
||||||
|
"section": section,
|
||||||
|
"path": note_path,
|
||||||
|
"source_path": kwargs.get("file_path") or note_path,
|
||||||
|
"retriever_weight": rw,
|
||||||
|
"chunk_profile": cp
|
||||||
|
}
|
||||||
|
|
||||||
|
# Audit: Cleanup Pop (Vermeidung von redundanten Alias-Feldern)
|
||||||
|
for alias in ("chunk_num", "Chunk_Number"):
|
||||||
|
pl.pop(alias, None)
|
||||||
|
|
||||||
|
out.append(pl)
|
||||||
|
|
||||||
|
return out
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/ingestion_db.py
|
||||||
|
DESCRIPTION: Datenbank-Schnittstelle für Note-Metadaten und Artefakt-Prüfung.
|
||||||
|
WP-14: Umstellung auf zentrale database-Infrastruktur.
|
||||||
|
"""
|
||||||
|
from typing import Optional, Tuple
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
# Import der modularisierten Namen-Logik zur Sicherstellung der Konsistenz
|
||||||
|
from app.core.database import collection_names
|
||||||
|
|
||||||
|
def fetch_note_payload(client: QdrantClient, prefix: str, note_id: str) -> Optional[dict]:
|
||||||
|
"""Holt die Metadaten einer Note aus Qdrant via Scroll."""
|
||||||
|
notes_col, _, _ = collection_names(prefix)
|
||||||
|
try:
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
pts, _ = client.scroll(collection_name=notes_col, scroll_filter=f, limit=1, with_payload=True)
|
||||||
|
return pts[0].payload if pts else None
|
||||||
|
except: return None
|
||||||
|
|
||||||
|
def artifacts_missing(client: QdrantClient, prefix: str, note_id: str) -> Tuple[bool, bool]:
|
||||||
|
"""Prüft Qdrant aktiv auf vorhandene Chunks und Edges."""
|
||||||
|
_, chunks_col, edges_col = collection_names(prefix)
|
||||||
|
try:
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
c_pts, _ = client.scroll(collection_name=chunks_col, scroll_filter=f, limit=1)
|
||||||
|
e_pts, _ = client.scroll(collection_name=edges_col, scroll_filter=f, limit=1)
|
||||||
|
return (not bool(c_pts)), (not bool(e_pts))
|
||||||
|
except: return True, True
|
||||||
|
|
||||||
|
def purge_artifacts(client: QdrantClient, prefix: str, note_id: str):
|
||||||
|
"""Löscht verwaiste Chunks/Edges vor einem Re-Import."""
|
||||||
|
_, chunks_col, edges_col = collection_names(prefix)
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
# Iteration über die nun zentral verwalteten Collection-Namen
|
||||||
|
for col in [chunks_col, edges_col]:
|
||||||
|
try: client.delete(collection_name=col, points_selector=rest.FilterSelector(filter=f))
|
||||||
|
except: pass
|
||||||
@@ -0,0 +1,169 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/ingestion_note_payload.py
|
||||||
|
DESCRIPTION: Baut das JSON-Objekt für mindnet_notes.
|
||||||
|
FEATURES:
|
||||||
|
- Multi-Hash (body/full) für flexible Change Detection.
|
||||||
|
- Fix v2.4.5: Präzise Hash-Logik für Profil-Änderungen.
|
||||||
|
- Integration der zentralen Registry (WP-14).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import Any, Dict, Tuple, Optional
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
# Import der zentralen Registry-Logik
|
||||||
|
from app.core.registry import load_type_registry
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helper
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _as_dict(x) -> Dict[str, Any]:
|
||||||
|
"""Versucht, ein Objekt in ein Dict zu überführen."""
|
||||||
|
if isinstance(x, dict): return dict(x)
|
||||||
|
out: Dict[str, Any] = {}
|
||||||
|
for attr in ("frontmatter", "body", "id", "note_id", "title", "path", "tags", "type", "created", "modified", "date"):
|
||||||
|
if hasattr(x, attr):
|
||||||
|
val = getattr(x, attr)
|
||||||
|
if val is not None: out[attr] = val
|
||||||
|
if not out: out["raw"] = str(x)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _ensure_list(x) -> list:
|
||||||
|
"""Sichert String-Listen Integrität."""
|
||||||
|
if x is None: return []
|
||||||
|
if isinstance(x, list): return [str(i) for i in x]
|
||||||
|
if isinstance(x, (set, tuple)): return [str(i) for i in x]
|
||||||
|
return [str(x)]
|
||||||
|
|
||||||
|
def _compute_hash(content: str) -> str:
|
||||||
|
"""SHA-256 Hash-Berechnung."""
|
||||||
|
if not content: return ""
|
||||||
|
return hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
def _get_hash_source_content(n: Dict[str, Any], mode: str) -> str:
|
||||||
|
"""
|
||||||
|
Generiert den Hash-Input-String basierend auf Body oder Metadaten.
|
||||||
|
Fix: Inkludiert nun alle entscheidungsrelevanten Profil-Parameter.
|
||||||
|
"""
|
||||||
|
body = str(n.get("body") or "").strip()
|
||||||
|
if mode == "body": return body
|
||||||
|
if mode == "full":
|
||||||
|
fm = n.get("frontmatter") or {}
|
||||||
|
meta_parts = []
|
||||||
|
# Wir inkludieren alle Felder, die das Chunking oder Retrieval beeinflussen
|
||||||
|
# Jede Änderung hier führt nun zwingend zu einem neuen Full-Hash
|
||||||
|
keys = [
|
||||||
|
"title", "type", "status", "tags",
|
||||||
|
"chunking_profile", "chunk_profile",
|
||||||
|
"retriever_weight", "split_level", "strict_heading_split"
|
||||||
|
]
|
||||||
|
for k in sorted(keys):
|
||||||
|
val = fm.get(k)
|
||||||
|
if val is not None: meta_parts.append(f"{k}:{val}")
|
||||||
|
return f"{'|'.join(meta_parts)}||{body}"
|
||||||
|
return body
|
||||||
|
|
||||||
|
def _cfg_for_type(note_type: str, reg: dict) -> dict:
|
||||||
|
"""Extrahiert Typ-spezifische Config aus der Registry."""
|
||||||
|
if not isinstance(reg, dict): return {}
|
||||||
|
types = reg.get("types") if isinstance(reg.get("types"), dict) else reg
|
||||||
|
return types.get(note_type, {}) if isinstance(types, dict) else {}
|
||||||
|
|
||||||
|
def _cfg_defaults(reg: dict) -> dict:
|
||||||
|
"""Extrahiert globale Default-Werte aus der Registry."""
|
||||||
|
if not isinstance(reg, dict): return {}
|
||||||
|
for key in ("defaults", "default", "global"):
|
||||||
|
v = reg.get(key)
|
||||||
|
if isinstance(v, dict): return v
|
||||||
|
return {}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Haupt-API
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def make_note_payload(note: Any, *args, **kwargs) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Baut das Note-Payload inklusive Multi-Hash und Audit-Validierung.
|
||||||
|
WP-14: Nutzt die zentrale Registry für alle Fallbacks.
|
||||||
|
"""
|
||||||
|
n = _as_dict(note)
|
||||||
|
|
||||||
|
# Registry & Context Settings
|
||||||
|
reg = kwargs.get("types_cfg") or load_type_registry()
|
||||||
|
hash_source = kwargs.get("hash_source", "parsed")
|
||||||
|
hash_normalize = kwargs.get("hash_normalize", "canonical")
|
||||||
|
|
||||||
|
fm = n.get("frontmatter") or {}
|
||||||
|
note_type = str(fm.get("type") or n.get("type") or "concept")
|
||||||
|
|
||||||
|
cfg_type = _cfg_for_type(note_type, reg)
|
||||||
|
cfg_def = _cfg_defaults(reg)
|
||||||
|
ingest_cfg = reg.get("ingestion_settings", {})
|
||||||
|
|
||||||
|
# --- retriever_weight Audit ---
|
||||||
|
default_rw = float(os.environ.get("MINDNET_DEFAULT_RETRIEVER_WEIGHT", 1.0))
|
||||||
|
retriever_weight = fm.get("retriever_weight")
|
||||||
|
if retriever_weight is None:
|
||||||
|
retriever_weight = cfg_type.get("retriever_weight", cfg_def.get("retriever_weight", default_rw))
|
||||||
|
try:
|
||||||
|
retriever_weight = float(retriever_weight)
|
||||||
|
except:
|
||||||
|
retriever_weight = default_rw
|
||||||
|
|
||||||
|
# --- chunk_profile Audit ---
|
||||||
|
chunk_profile = fm.get("chunking_profile") or fm.get("chunk_profile")
|
||||||
|
if chunk_profile is None:
|
||||||
|
chunk_profile = cfg_type.get("chunking_profile") or cfg_type.get("chunk_profile")
|
||||||
|
if chunk_profile is None:
|
||||||
|
chunk_profile = ingest_cfg.get("default_chunk_profile", cfg_def.get("chunking_profile", "sliding_standard"))
|
||||||
|
|
||||||
|
# --- edge_defaults Audit ---
|
||||||
|
edge_defaults = fm.get("edge_defaults")
|
||||||
|
if edge_defaults is None:
|
||||||
|
edge_defaults = cfg_type.get("edge_defaults", cfg_def.get("edge_defaults", []))
|
||||||
|
edge_defaults = _ensure_list(edge_defaults)
|
||||||
|
|
||||||
|
# --- Basis-Metadaten ---
|
||||||
|
note_id = n.get("note_id") or n.get("id") or fm.get("id")
|
||||||
|
title = n.get("title") or fm.get("title") or ""
|
||||||
|
path = n.get("path") or kwargs.get("file_path") or ""
|
||||||
|
if isinstance(path, pathlib.Path): path = str(path)
|
||||||
|
|
||||||
|
payload: Dict[str, Any] = {
|
||||||
|
"note_id": note_id,
|
||||||
|
"title": title,
|
||||||
|
"type": note_type,
|
||||||
|
"path": path,
|
||||||
|
"retriever_weight": retriever_weight,
|
||||||
|
"chunk_profile": chunk_profile,
|
||||||
|
"edge_defaults": edge_defaults,
|
||||||
|
"hashes": {}
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- MULTI-HASH ---
|
||||||
|
# Generiert Hashes für Change Detection (WP-15b)
|
||||||
|
for mode in ["body", "full"]:
|
||||||
|
content = _get_hash_source_content(n, mode)
|
||||||
|
payload["hashes"][f"{mode}:{hash_source}:{hash_normalize}"] = _compute_hash(content)
|
||||||
|
|
||||||
|
# Metadaten Anreicherung (Tags, Aliases, Zeitstempel)
|
||||||
|
tags = fm.get("tags") or fm.get("keywords") or n.get("tags")
|
||||||
|
if tags: payload["tags"] = _ensure_list(tags)
|
||||||
|
|
||||||
|
aliases = fm.get("aliases")
|
||||||
|
if aliases: payload["aliases"] = _ensure_list(aliases)
|
||||||
|
|
||||||
|
for k in ("created", "modified", "date"):
|
||||||
|
v = fm.get(k) or n.get(k)
|
||||||
|
if v: payload[k] = str(v)
|
||||||
|
|
||||||
|
if n.get("body"):
|
||||||
|
payload["fulltext"] = str(n["body"])
|
||||||
|
|
||||||
|
# Final JSON Validation Audit
|
||||||
|
json.loads(json.dumps(payload, ensure_ascii=False))
|
||||||
|
|
||||||
|
return payload
|
||||||
@@ -0,0 +1,245 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/ingestion_processor.py
|
||||||
|
DESCRIPTION: Der zentrale IngestionService (Orchestrator).
|
||||||
|
WP-14: Modularisierung der Datenbank-Ebene (app.core.database).
|
||||||
|
WP-15b: Two-Pass Workflow mit globalem Kontext-Cache.
|
||||||
|
WP-20/22: Cloud-Resilienz und Content-Lifecycle integriert.
|
||||||
|
AUDIT v2.13.12: Synchronisierung der Profil-Auflösung mit Registry-Defaults.
|
||||||
|
VERSION: 2.13.12
|
||||||
|
STATUS: Active
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
from typing import Dict, List, Optional, Tuple, Any
|
||||||
|
|
||||||
|
# Core Module Imports
|
||||||
|
from app.core.parser import (
|
||||||
|
read_markdown, pre_scan_markdown, normalize_frontmatter,
|
||||||
|
validate_required_frontmatter, NoteContext
|
||||||
|
)
|
||||||
|
from app.core.chunking import assemble_chunks
|
||||||
|
|
||||||
|
# MODULARISIERUNG: Neue Import-Pfade für die Datenbank-Ebene
|
||||||
|
from app.core.database.qdrant import QdrantConfig, get_client, ensure_collections, ensure_payload_indexes
|
||||||
|
from app.core.database.qdrant_points import points_for_chunks, points_for_note, points_for_edges, upsert_batch
|
||||||
|
|
||||||
|
# Services
|
||||||
|
from app.services.embeddings_client import EmbeddingsClient
|
||||||
|
from app.services.edge_registry import registry as edge_registry
|
||||||
|
from app.services.llm_service import LLMService
|
||||||
|
|
||||||
|
# Package-Interne Imports (Refactoring WP-14)
|
||||||
|
from .ingestion_utils import load_type_registry, resolve_note_type, get_chunk_config_by_profile
|
||||||
|
from .ingestion_db import fetch_note_payload, artifacts_missing, purge_artifacts
|
||||||
|
from .ingestion_validation import validate_edge_candidate
|
||||||
|
from .ingestion_note_payload import make_note_payload
|
||||||
|
from .ingestion_chunk_payload import make_chunk_payloads
|
||||||
|
|
||||||
|
# Fallback für Edges (Struktur-Verknüpfung)
|
||||||
|
try:
|
||||||
|
from app.core.graph.graph_derive_edges import build_edges_for_note
|
||||||
|
except ImportError:
|
||||||
|
def build_edges_for_note(*args, **kwargs): return []
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class IngestionService:
|
||||||
|
def __init__(self, collection_prefix: str = None):
|
||||||
|
"""Initialisiert den Service und nutzt die neue database-Infrastruktur."""
|
||||||
|
from app.config import get_settings
|
||||||
|
self.settings = get_settings()
|
||||||
|
|
||||||
|
self.prefix = collection_prefix or self.settings.COLLECTION_PREFIX
|
||||||
|
self.cfg = QdrantConfig.from_env()
|
||||||
|
# Synchronisierung der Konfiguration mit dem Instanz-Präfix
|
||||||
|
self.cfg.prefix = self.prefix
|
||||||
|
self.client = get_client(self.cfg)
|
||||||
|
self.dim = self.settings.VECTOR_SIZE
|
||||||
|
self.registry = load_type_registry()
|
||||||
|
self.embedder = EmbeddingsClient()
|
||||||
|
self.llm = LLMService()
|
||||||
|
|
||||||
|
# Festlegen, welcher Hash für die Change-Detection maßgeblich ist
|
||||||
|
self.active_hash_mode = self.settings.CHANGE_DETECTION_MODE
|
||||||
|
self.batch_cache: Dict[str, NoteContext] = {} # WP-15b LocalBatchCache
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Aufruf der modularisierten Schema-Logik
|
||||||
|
ensure_collections(self.client, self.prefix, self.dim)
|
||||||
|
ensure_payload_indexes(self.client, self.prefix)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"DB initialization warning: {e}")
|
||||||
|
|
||||||
|
async def run_batch(self, file_paths: List[str], vault_root: str) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
WP-15b: Implementiert den Two-Pass Ingestion Workflow.
|
||||||
|
Pass 1: Pre-Scan füllt den Context-Cache (3-Wege-Indexierung).
|
||||||
|
Pass 2: Verarbeitung nutzt den Cache für die semantische Prüfung.
|
||||||
|
"""
|
||||||
|
logger.info(f"🔍 [Pass 1] Pre-Scanning {len(file_paths)} files for Context Cache...")
|
||||||
|
for path in file_paths:
|
||||||
|
try:
|
||||||
|
# Übergabe der Registry für dynamische Scan-Tiefe
|
||||||
|
ctx = pre_scan_markdown(path, registry=self.registry)
|
||||||
|
if ctx:
|
||||||
|
# Mehrfache Indizierung für robusten Look-up (ID, Titel, Dateiname)
|
||||||
|
self.batch_cache[ctx.note_id] = ctx
|
||||||
|
self.batch_cache[ctx.title] = ctx
|
||||||
|
fname = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
self.batch_cache[fname] = ctx
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"⚠️ Pre-scan failed for {path}: {e}")
|
||||||
|
|
||||||
|
logger.info(f"🚀 [Pass 2] Semantic Processing of {len(file_paths)} files...")
|
||||||
|
return [await self.process_file(p, vault_root, apply=True, purge_before=True) for p in file_paths]
|
||||||
|
|
||||||
|
async def process_file(self, file_path: str, vault_root: str, **kwargs) -> Dict[str, Any]:
|
||||||
|
"""Transformiert eine Markdown-Datei in den Graphen."""
|
||||||
|
apply = kwargs.get("apply", False)
|
||||||
|
force_replace = kwargs.get("force_replace", False)
|
||||||
|
purge_before = kwargs.get("purge_before", False)
|
||||||
|
note_scope_refs = kwargs.get("note_scope_refs", False)
|
||||||
|
hash_source = kwargs.get("hash_source", "parsed")
|
||||||
|
hash_normalize = kwargs.get("hash_normalize", "canonical")
|
||||||
|
|
||||||
|
result = {"path": file_path, "status": "skipped", "changed": False, "error": None}
|
||||||
|
|
||||||
|
# 1. Parse & Lifecycle Gate
|
||||||
|
try:
|
||||||
|
parsed = read_markdown(file_path)
|
||||||
|
if not parsed: return {**result, "error": "Empty file"}
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter)
|
||||||
|
validate_required_frontmatter(fm)
|
||||||
|
except Exception as e:
|
||||||
|
return {**result, "error": f"Validation failed: {str(e)}"}
|
||||||
|
|
||||||
|
# Dynamischer Lifecycle-Filter aus der Registry (WP-14)
|
||||||
|
ingest_cfg = self.registry.get("ingestion_settings", {})
|
||||||
|
ignore_list = ingest_cfg.get("ignore_statuses", ["system", "template", "archive", "hidden"])
|
||||||
|
|
||||||
|
current_status = fm.get("status", "draft").lower().strip()
|
||||||
|
if current_status in ignore_list:
|
||||||
|
return {**result, "status": "skipped", "reason": "lifecycle_filter"}
|
||||||
|
|
||||||
|
# 2. Payload & Change Detection (Multi-Hash)
|
||||||
|
note_type = resolve_note_type(self.registry, fm.get("type"))
|
||||||
|
note_pl = make_note_payload(
|
||||||
|
parsed, vault_root=vault_root, file_path=file_path,
|
||||||
|
hash_source=hash_source, hash_normalize=hash_normalize,
|
||||||
|
types_cfg=self.registry
|
||||||
|
)
|
||||||
|
note_id = note_pl["note_id"]
|
||||||
|
|
||||||
|
# Abgleich mit der Datenbank (Qdrant)
|
||||||
|
old_payload = None if force_replace else fetch_note_payload(self.client, self.prefix, note_id)
|
||||||
|
|
||||||
|
# Prüfung gegen den konfigurierten Hash-Modus (body vs. full)
|
||||||
|
check_key = f"{self.active_hash_mode}:{hash_source}:{hash_normalize}"
|
||||||
|
old_hash = (old_payload or {}).get("hashes", {}).get(check_key)
|
||||||
|
new_hash = note_pl.get("hashes", {}).get(check_key)
|
||||||
|
|
||||||
|
# Check ob Chunks oder Kanten in der DB fehlen (Reparatur-Modus)
|
||||||
|
c_miss, e_miss = artifacts_missing(self.client, self.prefix, note_id)
|
||||||
|
|
||||||
|
# Wenn Hash identisch und Artefakte vorhanden -> Skip
|
||||||
|
if not (force_replace or not old_payload or old_hash != new_hash or c_miss or e_miss):
|
||||||
|
return {**result, "status": "unchanged", "note_id": note_id}
|
||||||
|
|
||||||
|
if not apply:
|
||||||
|
return {**result, "status": "dry-run", "changed": True, "note_id": note_id}
|
||||||
|
|
||||||
|
# 3. Deep Processing (Chunking, Validation, Embedding)
|
||||||
|
try:
|
||||||
|
body_text = getattr(parsed, "body", "") or ""
|
||||||
|
edge_registry.ensure_latest()
|
||||||
|
|
||||||
|
# Profil-Auflösung via Registry
|
||||||
|
# FIX: Wir nutzen das Profil, das bereits in make_note_payload unter
|
||||||
|
# Berücksichtigung der types.yaml (Registry) ermittelt wurde.
|
||||||
|
profile = note_pl.get("chunk_profile", "sliding_standard")
|
||||||
|
|
||||||
|
chunk_cfg = get_chunk_config_by_profile(self.registry, profile, note_type)
|
||||||
|
enable_smart = chunk_cfg.get("enable_smart_edge_allocation", False)
|
||||||
|
|
||||||
|
# WP-15b: Chunker-Aufruf bereitet den Candidate-Pool pro Chunk vor.
|
||||||
|
# assemble_chunks führt intern auch die Propagierung durch.
|
||||||
|
chunks = await assemble_chunks(note_id, body_text, note_type, config=chunk_cfg)
|
||||||
|
|
||||||
|
# Semantische Kanten-Validierung (Smart Edge Allocation)
|
||||||
|
for ch in chunks:
|
||||||
|
filtered = []
|
||||||
|
for cand in getattr(ch, "candidate_pool", []):
|
||||||
|
# Nur global_pool Kandidaten (aus dem Pool am Ende) erfordern KI-Validierung
|
||||||
|
if cand.get("provenance") == "global_pool" and enable_smart:
|
||||||
|
if await validate_edge_candidate(ch.text, cand, self.batch_cache, self.llm, self.settings.MINDNET_LLM_PROVIDER):
|
||||||
|
filtered.append(cand)
|
||||||
|
else:
|
||||||
|
# Explizite Kanten (Wikilinks/Callouts) werden ungeprüft übernommen
|
||||||
|
filtered.append(cand)
|
||||||
|
ch.candidate_pool = filtered
|
||||||
|
|
||||||
|
# Payload-Erstellung für die Chunks
|
||||||
|
chunk_pls = make_chunk_payloads(
|
||||||
|
fm, note_pl["path"], chunks, file_path=file_path,
|
||||||
|
types_cfg=self.registry
|
||||||
|
)
|
||||||
|
|
||||||
|
# Vektorisierung der Fenster-Texte
|
||||||
|
vecs = await self.embedder.embed_documents([c.get("window") or "" for c in chunk_pls]) if chunk_pls else []
|
||||||
|
|
||||||
|
# Aggregation aller finalen Kanten (Edges)
|
||||||
|
edges = build_edges_for_note(
|
||||||
|
note_id, chunk_pls,
|
||||||
|
note_level_references=note_pl.get("references", []),
|
||||||
|
include_note_scope_refs=note_scope_refs
|
||||||
|
)
|
||||||
|
|
||||||
|
# Kanten-Typen via Registry validieren/auflösen
|
||||||
|
for e in edges:
|
||||||
|
e["kind"] = edge_registry.resolve(
|
||||||
|
e.get("kind", "related_to"),
|
||||||
|
provenance=e.get("provenance", "explicit"),
|
||||||
|
context={"file": file_path, "note_id": note_id, "line": e.get("line", "system")}
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4. DB Upsert via modularisierter Points-Logik
|
||||||
|
# WICHTIG: Wenn sich der Inhalt geändert hat, löschen wir erst alle alten Fragmente.
|
||||||
|
if purge_before and old_payload:
|
||||||
|
purge_artifacts(self.client, self.prefix, note_id)
|
||||||
|
|
||||||
|
# Speichern der Haupt-Note
|
||||||
|
n_name, n_pts = points_for_note(self.prefix, note_pl, None, self.dim)
|
||||||
|
upsert_batch(self.client, n_name, n_pts)
|
||||||
|
|
||||||
|
# Speichern der Chunks
|
||||||
|
if chunk_pls and vecs:
|
||||||
|
c_pts = points_for_chunks(self.prefix, chunk_pls, vecs)[1]
|
||||||
|
upsert_batch(self.client, f"{self.prefix}_chunks", c_pts)
|
||||||
|
|
||||||
|
# Speichern der Kanten
|
||||||
|
if edges:
|
||||||
|
e_pts = points_for_edges(self.prefix, edges)[1]
|
||||||
|
upsert_batch(self.client, f"{self.prefix}_edges", e_pts)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"path": file_path,
|
||||||
|
"status": "success",
|
||||||
|
"changed": True,
|
||||||
|
"note_id": note_id,
|
||||||
|
"chunks_count": len(chunk_pls),
|
||||||
|
"edges_count": len(edges)
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Processing failed: {e}", exc_info=True)
|
||||||
|
return {**result, "error": str(e)}
|
||||||
|
|
||||||
|
async def create_from_text(self, markdown_content: str, filename: str, vault_root: str, folder: str = "00_Inbox") -> Dict[str, Any]:
|
||||||
|
"""Erstellt eine Note aus einem Textstream und triggert die Ingestion."""
|
||||||
|
target_path = os.path.join(vault_root, folder, filename)
|
||||||
|
os.makedirs(os.path.dirname(target_path), exist_ok=True)
|
||||||
|
with open(target_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write(markdown_content)
|
||||||
|
await asyncio.sleep(0.1)
|
||||||
|
# Triggert sofortigen Import mit force_replace/purge_before
|
||||||
|
return await self.process_file(file_path=target_path, vault_root=vault_root, apply=True, force_replace=True, purge_before=True)
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/ingestion_utils.py
|
||||||
|
DESCRIPTION: Hilfswerkzeuge für JSON-Recovery, Typ-Registry und Konfigurations-Lookups.
|
||||||
|
AUDIT v2.13.9: Behebung des Circular Imports durch Nutzung der app.core.registry.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from typing import Any, Optional, Dict
|
||||||
|
|
||||||
|
# ENTSCHEIDENDER FIX: Import der Basis-Logik aus dem neutralen Registry-Modul.
|
||||||
|
# Dies bricht den Zirkelbezug auf, da dieses Modul keine Services mehr importiert.
|
||||||
|
from app.core.registry import load_type_registry, clean_llm_text
|
||||||
|
|
||||||
|
def extract_json_from_response(text: str, registry: Optional[dict] = None) -> Any:
|
||||||
|
"""
|
||||||
|
Extrahiert JSON-Daten und bereinigt LLM-Steuerzeichen.
|
||||||
|
WP-14: Nutzt nun die zentrale clean_llm_text Funktion aus app.core.registry.
|
||||||
|
"""
|
||||||
|
if not text:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# 1. Text zentral bereinigen via neutralem Modul
|
||||||
|
clean = clean_llm_text(text, registry)
|
||||||
|
|
||||||
|
# 2. Markdown-Code-Blöcke extrahieren
|
||||||
|
match = re.search(r"```(?:json)?\s*(.*?)\s*```", clean, re.DOTALL)
|
||||||
|
payload = match.group(1) if match else clean
|
||||||
|
|
||||||
|
try:
|
||||||
|
return json.loads(payload.strip())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
# Recovery: Suche nach Liste
|
||||||
|
start = payload.find('[')
|
||||||
|
end = payload.rfind(']') + 1
|
||||||
|
if start != -1 and end > start:
|
||||||
|
try: return json.loads(payload[start:end])
|
||||||
|
except: pass
|
||||||
|
|
||||||
|
# Recovery: Suche nach Objekt
|
||||||
|
start_obj = payload.find('{')
|
||||||
|
end_obj = payload.rfind('}') + 1
|
||||||
|
if start_obj != -1 and end_obj > start_obj:
|
||||||
|
try: return json.loads(payload[start_obj:end_obj])
|
||||||
|
except: pass
|
||||||
|
return []
|
||||||
|
|
||||||
|
def resolve_note_type(registry: dict, requested: Optional[str]) -> str:
|
||||||
|
"""
|
||||||
|
Bestimmt den finalen Notiz-Typ.
|
||||||
|
WP-14: Fallback wird nun über ingestion_settings.default_note_type gesteuert.
|
||||||
|
"""
|
||||||
|
types = registry.get("types", {})
|
||||||
|
if requested and requested in types:
|
||||||
|
return requested
|
||||||
|
|
||||||
|
# Dynamischer Fallback aus der Registry (Standard: 'concept')
|
||||||
|
ingest_cfg = registry.get("ingestion_settings", {})
|
||||||
|
return ingest_cfg.get("default_note_type", "concept")
|
||||||
|
|
||||||
|
def get_chunk_config_by_profile(registry: dict, profile_name: str, note_type: str) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Holt die Chunker-Parameter für ein spezifisches Profil aus der Registry.
|
||||||
|
"""
|
||||||
|
from app.core.chunking import get_chunk_config
|
||||||
|
profiles = registry.get("chunking_profiles", {})
|
||||||
|
if profile_name in profiles:
|
||||||
|
cfg = profiles[profile_name].copy()
|
||||||
|
if "overlap" in cfg and isinstance(cfg["overlap"], list):
|
||||||
|
cfg["overlap"] = tuple(cfg["overlap"])
|
||||||
|
return cfg
|
||||||
|
return get_chunk_config(note_type)
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/ingestion/ingestion_validation.py
|
||||||
|
DESCRIPTION: WP-15b semantische Validierung von Kanten gegen den LocalBatchCache.
|
||||||
|
AUDIT v2.12.3: Integration der zentralen Text-Bereinigung (WP-14).
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from app.core.parser import NoteContext
|
||||||
|
|
||||||
|
# ENTSCHEIDENDER FIX: Import der neutralen Bereinigungs-Logik zur Vermeidung von Circular Imports
|
||||||
|
from app.core.registry import clean_llm_text
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
async def validate_edge_candidate(
|
||||||
|
chunk_text: str,
|
||||||
|
edge: Dict,
|
||||||
|
batch_cache: Dict[str, NoteContext],
|
||||||
|
llm_service: Any,
|
||||||
|
provider: str
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
WP-15b: Validiert einen Kandidaten semantisch gegen das Ziel im Cache.
|
||||||
|
Nutzt clean_llm_text zur Entfernung von Steuerzeichen vor der Auswertung.
|
||||||
|
"""
|
||||||
|
target_id = edge.get("to")
|
||||||
|
target_ctx = batch_cache.get(target_id)
|
||||||
|
|
||||||
|
# Robust Lookup Fix (v2.12.2): Support für Anker
|
||||||
|
if not target_ctx and "#" in target_id:
|
||||||
|
base_id = target_id.split("#")[0]
|
||||||
|
target_ctx = batch_cache.get(base_id)
|
||||||
|
|
||||||
|
# Sicherheits-Fallback (Hard-Link Integrity)
|
||||||
|
if not target_ctx:
|
||||||
|
logger.info(f"ℹ️ [VALIDATION SKIP] No context for '{target_id}' - allowing link.")
|
||||||
|
return True
|
||||||
|
|
||||||
|
template = llm_service.get_prompt("edge_validation", provider)
|
||||||
|
|
||||||
|
try:
|
||||||
|
logger.info(f"⚖️ [VALIDATING] Relation '{edge.get('kind')}' -> '{target_id}'...")
|
||||||
|
prompt = template.format(
|
||||||
|
chunk_text=chunk_text[:1500],
|
||||||
|
target_title=target_ctx.title,
|
||||||
|
target_summary=target_ctx.summary,
|
||||||
|
edge_kind=edge.get("kind", "related_to")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Die Antwort vom Service anfordern
|
||||||
|
raw_response = await llm_service.generate_raw_response(prompt, priority="background")
|
||||||
|
|
||||||
|
# WP-14 Fix: Zusätzliche Bereinigung zur Sicherstellung der Interpretierbarkeit
|
||||||
|
response = clean_llm_text(raw_response)
|
||||||
|
|
||||||
|
# Semantische Prüfung des Ergebnisses
|
||||||
|
is_valid = "YES" in response.upper()
|
||||||
|
|
||||||
|
if is_valid:
|
||||||
|
logger.info(f"✅ [VALIDATED] Relation to '{target_id}' confirmed.")
|
||||||
|
else:
|
||||||
|
logger.info(f"🚫 [REJECTED] Relation to '{target_id}' irrelevant for this chunk.")
|
||||||
|
return is_valid
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"⚠️ Validation error for {target_id}: {e}")
|
||||||
|
# Im Zweifel (Timeout/Fehler) erlauben wir die Kante, um Datenverlust zu vermeiden
|
||||||
|
return True
|
||||||
@@ -1,236 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
Modul: app/core/note_payload.py
|
|
||||||
Version: 2.1.0 (WP-11 Update: Aliases support)
|
|
||||||
|
|
||||||
Zweck
|
|
||||||
-----
|
|
||||||
Erzeugt ein robustes Note-Payload. Werte wie `retriever_weight`, `chunk_profile`
|
|
||||||
und `edge_defaults` werden in folgender Priorität bestimmt:
|
|
||||||
1) Frontmatter (Note)
|
|
||||||
2) Typ-Registry (config/types.yaml: types.<type>.*)
|
|
||||||
3) Registry-Defaults (config/types.yaml: defaults.*)
|
|
||||||
4) ENV-Defaults (MINDNET_DEFAULT_RETRIEVER_WEIGHT / MINDNET_DEFAULT_CHUNK_PROFILE)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any, Dict, Tuple, Optional
|
|
||||||
import os
|
|
||||||
import json
|
|
||||||
import pathlib
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml # type: ignore
|
|
||||||
except Exception:
|
|
||||||
yaml = None
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Helper
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _as_dict(x) -> Dict[str, Any]:
|
|
||||||
"""Versucht, ein ParsedMarkdown-ähnliches Objekt in ein Dict zu überführen."""
|
|
||||||
if isinstance(x, dict):
|
|
||||||
return dict(x)
|
|
||||||
|
|
||||||
out: Dict[str, Any] = {}
|
|
||||||
# bekannte Attribute übernehmen, sofern vorhanden
|
|
||||||
for attr in (
|
|
||||||
"frontmatter",
|
|
||||||
"body",
|
|
||||||
"id",
|
|
||||||
"note_id",
|
|
||||||
"title",
|
|
||||||
"path",
|
|
||||||
"tags",
|
|
||||||
"type",
|
|
||||||
"created",
|
|
||||||
"modified",
|
|
||||||
"date",
|
|
||||||
):
|
|
||||||
if hasattr(x, attr):
|
|
||||||
val = getattr(x, attr)
|
|
||||||
if val is not None:
|
|
||||||
out[attr] = val
|
|
||||||
|
|
||||||
# Fallback: wenn immer noch leer, raw speichern
|
|
||||||
if not out:
|
|
||||||
out["raw"] = str(x)
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _pick_args(*args, **kwargs) -> Tuple[Optional[str], Optional[dict]]:
|
|
||||||
"""Extrahiert optionale Zusatzargumente wie path und types_cfg."""
|
|
||||||
path = kwargs.get("path") or (args[0] if args else None)
|
|
||||||
types_cfg = kwargs.get("types_cfg") or kwargs.get("types") or None
|
|
||||||
return path, types_cfg
|
|
||||||
|
|
||||||
|
|
||||||
def _env_float(name: str, default: float) -> float:
|
|
||||||
"""Liest einen Float-Wert aus der Umgebung, mit robustem Fallback."""
|
|
||||||
try:
|
|
||||||
return float(os.environ.get(name, default))
|
|
||||||
except Exception:
|
|
||||||
return default
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_list(x) -> list:
|
|
||||||
"""Garantiert eine String-Liste."""
|
|
||||||
if x is None:
|
|
||||||
return []
|
|
||||||
if isinstance(x, list):
|
|
||||||
return [str(i) for i in x]
|
|
||||||
if isinstance(x, (set, tuple)):
|
|
||||||
return [str(i) for i in x]
|
|
||||||
return [str(x)]
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Type-Registry laden
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _load_types_config(explicit_cfg: Optional[dict] = None) -> dict:
|
|
||||||
"""Lädt die Type-Registry aus YAML/JSON oder nutzt ein explizit übergebenes Dict."""
|
|
||||||
if explicit_cfg and isinstance(explicit_cfg, dict):
|
|
||||||
return explicit_cfg
|
|
||||||
|
|
||||||
path = os.getenv("MINDNET_TYPES_FILE") or "./config/types.yaml"
|
|
||||||
if not os.path.isfile(path) or yaml is None:
|
|
||||||
return {}
|
|
||||||
|
|
||||||
try:
|
|
||||||
with open(path, "r", encoding="utf-8") as f:
|
|
||||||
data = yaml.safe_load(f) or {}
|
|
||||||
return data if isinstance(data, dict) else {}
|
|
||||||
except Exception:
|
|
||||||
return {}
|
|
||||||
|
|
||||||
|
|
||||||
def _cfg_for_type(note_type: str, reg: dict) -> dict:
|
|
||||||
"""Liefert die Konfiguration für einen konkreten Notiztyp aus der Registry."""
|
|
||||||
if not isinstance(reg, dict):
|
|
||||||
return {}
|
|
||||||
types = reg.get("types") if isinstance(reg.get("types"), dict) else reg
|
|
||||||
return types.get(note_type, {}) if isinstance(types, dict) else {}
|
|
||||||
|
|
||||||
|
|
||||||
def _cfg_defaults(reg: dict) -> dict:
|
|
||||||
"""Liefert den Default-Block aus der Registry (defaults/global)."""
|
|
||||||
if not isinstance(reg, dict):
|
|
||||||
return {}
|
|
||||||
for key in ("defaults", "default", "global"):
|
|
||||||
v = reg.get(key)
|
|
||||||
if isinstance(v, dict):
|
|
||||||
return v
|
|
||||||
return {}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Haupt-API
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def make_note_payload(note: Any, *args, **kwargs) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Baut das Note-Payload für mindnet_notes auf.
|
|
||||||
|
|
||||||
Erwartete Felder im Payload:
|
|
||||||
- note_id: stabile ID aus Frontmatter (id) oder Note-Objekt
|
|
||||||
- title: Titel der Notiz
|
|
||||||
- type: Notiztyp (z. B. concept, project, journal, ...)
|
|
||||||
- path: Pfad im Vault
|
|
||||||
- retriever_weight: effektives Gewicht für den Retriever
|
|
||||||
- chunk_profile: Profil für Chunking (short|medium|long|default|...)
|
|
||||||
- edge_defaults: Liste von Kanten-Typen, die als Defaults gelten
|
|
||||||
- aliases: Liste von Synonymen (WP-11)
|
|
||||||
"""
|
|
||||||
n = _as_dict(note)
|
|
||||||
path_arg, types_cfg_explicit = _pick_args(*args, **kwargs)
|
|
||||||
reg = _load_types_config(types_cfg_explicit)
|
|
||||||
|
|
||||||
fm = n.get("frontmatter") or {}
|
|
||||||
fm_type = fm.get("type") or n.get("type") or "concept"
|
|
||||||
note_type = str(fm_type)
|
|
||||||
|
|
||||||
cfg_type = _cfg_for_type(note_type, reg)
|
|
||||||
cfg_def = _cfg_defaults(reg)
|
|
||||||
|
|
||||||
# --- retriever_weight: Frontmatter > Typ-Config > Registry-Defaults > ENV ---
|
|
||||||
default_rw = _env_float("MINDNET_DEFAULT_RETRIEVER_WEIGHT", 1.0)
|
|
||||||
retriever_weight = fm.get("retriever_weight")
|
|
||||||
if retriever_weight is None:
|
|
||||||
retriever_weight = cfg_type.get(
|
|
||||||
"retriever_weight",
|
|
||||||
cfg_def.get("retriever_weight", default_rw),
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
retriever_weight = float(retriever_weight)
|
|
||||||
except Exception:
|
|
||||||
retriever_weight = default_rw
|
|
||||||
|
|
||||||
# --- chunk_profile: Frontmatter > Typ-Config > Registry-Defaults > ENV ---
|
|
||||||
chunk_profile = fm.get("chunk_profile")
|
|
||||||
if chunk_profile is None:
|
|
||||||
chunk_profile = cfg_type.get(
|
|
||||||
"chunk_profile",
|
|
||||||
cfg_def.get(
|
|
||||||
"chunk_profile",
|
|
||||||
os.environ.get("MINDNET_DEFAULT_CHUNK_PROFILE", "medium"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
if not isinstance(chunk_profile, str):
|
|
||||||
chunk_profile = "medium"
|
|
||||||
|
|
||||||
# --- edge_defaults: Frontmatter > Typ-Config > Registry-Defaults ---
|
|
||||||
edge_defaults = fm.get("edge_defaults")
|
|
||||||
if edge_defaults is None:
|
|
||||||
edge_defaults = cfg_type.get(
|
|
||||||
"edge_defaults",
|
|
||||||
cfg_def.get("edge_defaults", []),
|
|
||||||
)
|
|
||||||
edge_defaults = _ensure_list(edge_defaults)
|
|
||||||
|
|
||||||
# --- Basis-Metadaten (IDs, Titel, Pfad) ---
|
|
||||||
note_id = n.get("note_id") or n.get("id") or fm.get("id")
|
|
||||||
title = n.get("title") or fm.get("title") or ""
|
|
||||||
path = n.get("path") or path_arg
|
|
||||||
if isinstance(path, pathlib.Path):
|
|
||||||
path = str(path)
|
|
||||||
|
|
||||||
payload: Dict[str, Any] = {
|
|
||||||
"note_id": note_id,
|
|
||||||
"title": title,
|
|
||||||
"type": note_type,
|
|
||||||
"path": path or "",
|
|
||||||
"retriever_weight": retriever_weight,
|
|
||||||
"chunk_profile": chunk_profile,
|
|
||||||
"edge_defaults": edge_defaults,
|
|
||||||
}
|
|
||||||
|
|
||||||
# Tags / Keywords übernehmen
|
|
||||||
tags = fm.get("tags") or fm.get("keywords") or n.get("tags")
|
|
||||||
if tags:
|
|
||||||
payload["tags"] = _ensure_list(tags)
|
|
||||||
|
|
||||||
# WP-11: Aliases übernehmen (für Discovery Service)
|
|
||||||
aliases = fm.get("aliases")
|
|
||||||
if aliases:
|
|
||||||
payload["aliases"] = _ensure_list(aliases)
|
|
||||||
|
|
||||||
# Zeitliche Metadaten (sofern vorhanden)
|
|
||||||
for k in ("created", "modified", "date"):
|
|
||||||
v = fm.get(k) or n.get(k)
|
|
||||||
if v:
|
|
||||||
payload[k] = str(v)
|
|
||||||
|
|
||||||
# Fulltext (Fallback, falls body im Input)
|
|
||||||
if "body" in n and n["body"]:
|
|
||||||
payload["fulltext"] = str(n["body"])
|
|
||||||
|
|
||||||
# JSON-Roundtrip zur harten Validierung (ASCII beibehalten)
|
|
||||||
json.loads(json.dumps(payload, ensure_ascii=False))
|
|
||||||
|
|
||||||
return payload
|
|
||||||
@@ -1,266 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
Modul: app/core/parser.py
|
|
||||||
Version: 1.7.1 (fault-tolerant, API-kompatibel)
|
|
||||||
Datum: 2025-10-01
|
|
||||||
|
|
||||||
Zweck
|
|
||||||
-----
|
|
||||||
Fehlertolerantes Einlesen von Markdown-Dateien mit YAML-Frontmatter.
|
|
||||||
Kompatibel zur bisherigen Parser-API, aber robust gegenüber Nicht-UTF-8-Dateien:
|
|
||||||
- Versucht nacheinander: utf-8 → utf-8-sig → cp1252 → latin-1.
|
|
||||||
- Bei Fallback wird ein JSON-Warnhinweis auf stdout ausgegeben; der Import bricht NICHT ab.
|
|
||||||
- YAML-Frontmatter wird mit '---' am Anfang und '---' als Abschluss erkannt.
|
|
||||||
- extract_wikilinks() normalisiert [[id#anchor|label]] → 'id'.
|
|
||||||
|
|
||||||
Öffentliche API (kompatibel):
|
|
||||||
- class ParsedNote(frontmatter: dict, body: str, path: str)
|
|
||||||
- read_markdown(path) -> ParsedNote | None
|
|
||||||
- normalize_frontmatter(fm) -> dict
|
|
||||||
- validate_required_frontmatter(fm, required: tuple[str,...]=("id","title")) -> None
|
|
||||||
- extract_wikilinks(text) -> list[str]
|
|
||||||
- FRONTMATTER_RE (Kompatibilitäts-Konstante; Regex für '---'-Zeilen)
|
|
||||||
|
|
||||||
Beispiele
|
|
||||||
---------
|
|
||||||
from app.core.parser import read_markdown, normalize_frontmatter, validate_required_frontmatter
|
|
||||||
parsed = read_markdown("./vault/30_projects/project-demo.md")
|
|
||||||
fm = normalize_frontmatter(parsed.frontmatter)
|
|
||||||
validate_required_frontmatter(fm)
|
|
||||||
body = parsed.body
|
|
||||||
|
|
||||||
from app.core.parser import extract_wikilinks
|
|
||||||
links = extract_wikilinks(body)
|
|
||||||
|
|
||||||
Abhängigkeiten
|
|
||||||
--------------
|
|
||||||
- PyYAML (yaml)
|
|
||||||
|
|
||||||
Lizenz: MIT (projektintern)
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Any, Dict, Optional, Tuple, Iterable, List
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml # PyYAML
|
|
||||||
except Exception as e: # pragma: no cover
|
|
||||||
yaml = None # Fehler wird zur Laufzeit geworfen, falls wirklich benötigt
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
# Datamodell
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ParsedNote:
|
|
||||||
frontmatter: Dict[str, Any]
|
|
||||||
body: str
|
|
||||||
path: str
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
# Frontmatter-Erkennung
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
|
|
||||||
# Öffentliche Kompatibilitäts-Konstante: frühere Skripte importieren FRONTMATTER_RE
|
|
||||||
FRONTMATTER_RE = re.compile(r"^\s*---\s*$") # <— public
|
|
||||||
# Zusätzlich interner Alias (falls jemand ihn referenziert)
|
|
||||||
FRONTMATTER_END = FRONTMATTER_RE # <— public alias
|
|
||||||
|
|
||||||
# interne Namen bleiben bestehen
|
|
||||||
_FRONTMATTER_HEAD = FRONTMATTER_RE
|
|
||||||
_FRONTMATTER_END = FRONTMATTER_RE
|
|
||||||
|
|
||||||
|
|
||||||
def _split_frontmatter(text: str) -> Tuple[Dict[str, Any], str]:
|
|
||||||
"""
|
|
||||||
Zerlegt Text in (frontmatter: dict, body: str).
|
|
||||||
Erkennt Frontmatter nur, wenn die erste Zeile '---' ist und später ein zweites '---' folgt.
|
|
||||||
YAML-Fehler im Frontmatter führen NICHT zum Abbruch: es wird dann ein leeres dict benutzt.
|
|
||||||
"""
|
|
||||||
lines = text.splitlines(True) # keep line endings
|
|
||||||
if not lines:
|
|
||||||
return {}, ""
|
|
||||||
|
|
||||||
if not _FRONTMATTER_HEAD.match(lines[0]):
|
|
||||||
# kein Frontmatter-Header → gesamter Text ist Body
|
|
||||||
return {}, text
|
|
||||||
|
|
||||||
end_idx = None
|
|
||||||
# Suche nach nächstem '---' (max. 2000 Zeilen als Sicherheitslimit)
|
|
||||||
for i in range(1, min(len(lines), 2000)):
|
|
||||||
if _FRONTMATTER_END.match(lines[i]):
|
|
||||||
end_idx = i
|
|
||||||
break
|
|
||||||
|
|
||||||
if end_idx is None:
|
|
||||||
# unvollständiger Frontmatter-Block → behandle alles als Body
|
|
||||||
return {}, text
|
|
||||||
|
|
||||||
fm_raw = "".join(lines[1:end_idx])
|
|
||||||
body = "".join(lines[end_idx + 1:])
|
|
||||||
|
|
||||||
data: Dict[str, Any] = {}
|
|
||||||
if yaml is None:
|
|
||||||
raise RuntimeError("PyYAML ist nicht installiert (pip install pyyaml).")
|
|
||||||
|
|
||||||
try:
|
|
||||||
loaded = yaml.safe_load(fm_raw) or {}
|
|
||||||
if isinstance(loaded, dict):
|
|
||||||
data = loaded
|
|
||||||
else:
|
|
||||||
data = {}
|
|
||||||
except Exception as e:
|
|
||||||
# YAML-Fehler nicht fatal machen
|
|
||||||
print(json.dumps({"warn": "frontmatter_yaml_parse_failed", "error": str(e)}))
|
|
||||||
data = {}
|
|
||||||
|
|
||||||
# optionales kosmetisches Trim: eine führende Leerzeile im Body entfernen
|
|
||||||
if body.startswith("\n"):
|
|
||||||
body = body[1:]
|
|
||||||
|
|
||||||
return data, body
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
# Robustes Lesen mit Encoding-Fallback
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
|
|
||||||
_FALLBACK_ENCODINGS: Tuple[str, ...] = ("utf-8", "utf-8-sig", "cp1252", "latin-1")
|
|
||||||
|
|
||||||
|
|
||||||
def _read_text_with_fallback(path: str) -> Tuple[str, str, bool]:
|
|
||||||
"""
|
|
||||||
Liest Datei mit mehreren Decodierungsversuchen.
|
|
||||||
Rückgabe: (text, used_encoding, had_fallback)
|
|
||||||
- had_fallback=True, falls NICHT 'utf-8' verwendet wurde (oder 'utf-8-sig').
|
|
||||||
"""
|
|
||||||
last_err: Optional[str] = None
|
|
||||||
for enc in _FALLBACK_ENCODINGS:
|
|
||||||
try:
|
|
||||||
with io.open(path, "r", encoding=enc, errors="strict") as f:
|
|
||||||
text = f.read()
|
|
||||||
# 'utf-8-sig' zählt hier als Fallback (weil BOM), aber ist unproblematisch
|
|
||||||
return text, enc, (enc != "utf-8")
|
|
||||||
except UnicodeDecodeError as e:
|
|
||||||
last_err = f"{type(e).__name__}: {e}"
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Letzter, extrem defensiver Fallback: Bytes → UTF-8 mit REPLACE (keine Exception)
|
|
||||||
with open(path, "rb") as fb:
|
|
||||||
raw = fb.read()
|
|
||||||
text = raw.decode("utf-8", errors="replace")
|
|
||||||
print(json.dumps({
|
|
||||||
"path": path,
|
|
||||||
"warn": "encoding_fallback_exhausted",
|
|
||||||
"info": last_err or "unknown"
|
|
||||||
}, ensure_ascii=False))
|
|
||||||
return text, "utf-8(replace)", True
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
# Öffentliche API
|
|
||||||
# ---------------------------------------------------------------------
|
|
||||||
|
|
||||||
def read_markdown(path: str) -> Optional[ParsedNote]:
|
|
||||||
"""
|
|
||||||
Liest eine Markdown-Datei fehlertolerant:
|
|
||||||
- Erlaubt verschiedene Encodings (UTF-8 bevorzugt, cp1252/latin-1 als Fallback).
|
|
||||||
- Schlägt NICHT mit UnicodeDecodeError fehl.
|
|
||||||
- Gibt ParsedNote(frontmatter, body, path) zurück oder None, falls die Datei nicht existiert.
|
|
||||||
|
|
||||||
Bei Decoding-Fallback wird ein JSON-Warnhinweis geloggt:
|
|
||||||
{"path": "...", "warn": "encoding_fallback_used", "used": "cp1252"}
|
|
||||||
"""
|
|
||||||
if not os.path.exists(path):
|
|
||||||
return None
|
|
||||||
|
|
||||||
text, enc, had_fb = _read_text_with_fallback(path)
|
|
||||||
if had_fb:
|
|
||||||
print(json.dumps({"path": path, "warn": "encoding_fallback_used", "used": enc}, ensure_ascii=False))
|
|
||||||
|
|
||||||
fm, body = _split_frontmatter(text)
|
|
||||||
return ParsedNote(frontmatter=fm or {}, body=body or "", path=path)
|
|
||||||
|
|
||||||
|
|
||||||
def validate_required_frontmatter(fm: Dict[str, Any],
|
|
||||||
required: Tuple[str, ...] = ("id", "title")) -> None:
|
|
||||||
"""
|
|
||||||
Prüft, ob alle Pflichtfelder vorhanden sind.
|
|
||||||
Default-kompatibel: ('id', 'title'), kann aber vom Aufrufer erweitert werden, z. B.:
|
|
||||||
validate_required_frontmatter(fm, required=("id","title","type","status","created"))
|
|
||||||
|
|
||||||
Hebt ValueError, falls Felder fehlen oder leer sind.
|
|
||||||
"""
|
|
||||||
if fm is None:
|
|
||||||
fm = {}
|
|
||||||
missing = []
|
|
||||||
for k in required:
|
|
||||||
v = fm.get(k)
|
|
||||||
if v is None:
|
|
||||||
missing.append(k)
|
|
||||||
elif isinstance(v, str) and not v.strip():
|
|
||||||
missing.append(k)
|
|
||||||
if missing:
|
|
||||||
raise ValueError(f"Missing required frontmatter fields: {', '.join(missing)}")
|
|
||||||
|
|
||||||
# Plausibilitäten: 'tags' sollte eine Liste sein, wenn vorhanden
|
|
||||||
if "tags" in fm and fm["tags"] not in (None, "") and not isinstance(fm["tags"], (list, tuple)):
|
|
||||||
raise ValueError("frontmatter 'tags' must be a list of strings")
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_frontmatter(fm: Dict[str, Any]) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Sanfte Normalisierung ohne Semantikänderung:
|
|
||||||
- 'tags' → Liste von Strings (Trim)
|
|
||||||
- 'embedding_exclude' → bool
|
|
||||||
- andere Felder unverändert
|
|
||||||
"""
|
|
||||||
out = dict(fm or {})
|
|
||||||
if "tags" in out:
|
|
||||||
if isinstance(out["tags"], str):
|
|
||||||
out["tags"] = [out["tags"].strip()] if out["tags"].strip() else []
|
|
||||||
elif isinstance(out["tags"], list):
|
|
||||||
out["tags"] = [str(t).strip() for t in out["tags"] if t is not None]
|
|
||||||
else:
|
|
||||||
out["tags"] = [str(out["tags"]).strip()] if out["tags"] not in (None, "") else []
|
|
||||||
if "embedding_exclude" in out:
|
|
||||||
out["embedding_exclude"] = bool(out["embedding_exclude"])
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------ Wikilinks ---------------------------- #
|
|
||||||
|
|
||||||
# Basismuster für [[...]]; die Normalisierung (id vor '#', vor '|') macht extract_wikilinks
|
|
||||||
_WIKILINK_RE = re.compile(r"\[\[([^\]]+)\]\]")
|
|
||||||
|
|
||||||
|
|
||||||
def extract_wikilinks(text: str) -> List[str]:
|
|
||||||
"""
|
|
||||||
Extrahiert Wikilinks wie [[id]], [[id#anchor]], [[id|label]], [[id#anchor|label]].
|
|
||||||
Rückgabe sind NUR die Ziel-IDs (ohne Anchor/Label), führend/folgend getrimmt.
|
|
||||||
Keine aggressive Slug-Normalisierung (die kann später im Resolver erfolgen).
|
|
||||||
"""
|
|
||||||
if not text:
|
|
||||||
return []
|
|
||||||
out: List[str] = []
|
|
||||||
for m in _WIKILINK_RE.finditer(text):
|
|
||||||
raw = (m.group(1) or "").strip()
|
|
||||||
if not raw:
|
|
||||||
continue
|
|
||||||
# Split an Pipe (Label) → links vor '|'
|
|
||||||
if "|" in raw:
|
|
||||||
raw = raw.split("|", 1)[0].strip()
|
|
||||||
# Split an Anchor
|
|
||||||
if "#" in raw:
|
|
||||||
raw = raw.split("#", 1)[0].strip()
|
|
||||||
if raw:
|
|
||||||
out.append(raw)
|
|
||||||
return out
|
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/parser/__init__.py
|
||||||
|
DESCRIPTION: Package-Einstiegspunkt für den Parser.
|
||||||
|
Ermöglicht das Löschen der parser.py Facade.
|
||||||
|
VERSION: 1.10.0
|
||||||
|
"""
|
||||||
|
from .parsing_models import ParsedNote, NoteContext
|
||||||
|
from .parsing_utils import (
|
||||||
|
FRONTMATTER_RE, validate_required_frontmatter,
|
||||||
|
normalize_frontmatter, extract_wikilinks, extract_edges_with_context
|
||||||
|
)
|
||||||
|
from .parsing_markdown import read_markdown
|
||||||
|
from .parsing_scanner import pre_scan_markdown
|
||||||
|
|
||||||
|
# Kompatibilitäts-Alias
|
||||||
|
FRONTMATTER_END = FRONTMATTER_RE
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ParsedNote", "NoteContext", "FRONTMATTER_RE", "FRONTMATTER_END",
|
||||||
|
"read_markdown", "pre_scan_markdown", "validate_required_frontmatter",
|
||||||
|
"normalize_frontmatter", "extract_wikilinks", "extract_edges_with_context"
|
||||||
|
]
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/parsing/parsing_markdown.py
|
||||||
|
DESCRIPTION: Fehlertolerantes Einlesen von Markdown und Frontmatter-Splitting.
|
||||||
|
"""
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
from typing import Any, Dict, Optional, Tuple
|
||||||
|
from .parsing_models import ParsedNote
|
||||||
|
from .parsing_utils import FRONTMATTER_RE
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError:
|
||||||
|
yaml = None
|
||||||
|
|
||||||
|
_FALLBACK_ENCODINGS: Tuple[str, ...] = ("utf-8", "utf-8-sig", "cp1252", "latin-1")
|
||||||
|
|
||||||
|
def _split_frontmatter(text: str) -> Tuple[Dict[str, Any], str]:
|
||||||
|
"""Zerlegt Text in Frontmatter-Dict und Body."""
|
||||||
|
lines = text.splitlines(True)
|
||||||
|
if not lines or not FRONTMATTER_RE.match(lines[0]):
|
||||||
|
return {}, text
|
||||||
|
end_idx = None
|
||||||
|
for i in range(1, min(len(lines), 2000)):
|
||||||
|
if FRONTMATTER_RE.match(lines[i]):
|
||||||
|
end_idx = i
|
||||||
|
break
|
||||||
|
if end_idx is None: return {}, text
|
||||||
|
fm_raw = "".join(lines[1:end_idx])
|
||||||
|
body = "".join(lines[end_idx + 1:])
|
||||||
|
if yaml is None: raise RuntimeError("PyYAML not installed.")
|
||||||
|
try:
|
||||||
|
loaded = yaml.safe_load(fm_raw) or {}
|
||||||
|
data = loaded if isinstance(loaded, dict) else {}
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"warn": "frontmatter_yaml_parse_failed", "error": str(e)}))
|
||||||
|
data = {}
|
||||||
|
if body.startswith("\n"): body = body[1:]
|
||||||
|
return data, body
|
||||||
|
|
||||||
|
def _read_text_with_fallback(path: str) -> Tuple[str, str, bool]:
|
||||||
|
"""Liest Datei mit Encoding-Fallback-Kette."""
|
||||||
|
last_err = None
|
||||||
|
for enc in _FALLBACK_ENCODINGS:
|
||||||
|
try:
|
||||||
|
with io.open(path, "r", encoding=enc, errors="strict") as f:
|
||||||
|
return f.read(), enc, (enc != "utf-8")
|
||||||
|
except UnicodeDecodeError as e:
|
||||||
|
last_err = str(e); continue
|
||||||
|
with open(path, "rb") as fb:
|
||||||
|
text = fb.read().decode("utf-8", errors="replace")
|
||||||
|
return text, "utf-8(replace)", True
|
||||||
|
|
||||||
|
def read_markdown(path: str) -> Optional[ParsedNote]:
|
||||||
|
"""Öffentliche API zum Einlesen einer Datei."""
|
||||||
|
if not os.path.exists(path): return None
|
||||||
|
text, enc, had_fb = _read_text_with_fallback(path)
|
||||||
|
fm, body = _split_frontmatter(text)
|
||||||
|
return ParsedNote(frontmatter=fm or {}, body=body or "", path=path)
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/parsing/parsing_models.py
|
||||||
|
DESCRIPTION: Datenklassen für das Parsing-System.
|
||||||
|
"""
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ParsedNote:
|
||||||
|
"""Container für eine vollständig eingelesene Markdown-Datei."""
|
||||||
|
frontmatter: Dict[str, Any]
|
||||||
|
body: str
|
||||||
|
path: str
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class NoteContext:
|
||||||
|
"""Metadaten-Container für den flüchtigen LocalBatchCache (Pass 1)."""
|
||||||
|
note_id: str
|
||||||
|
title: str
|
||||||
|
type: str
|
||||||
|
summary: str
|
||||||
|
tags: List[str]
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/parsing/parsing_scanner.py
|
||||||
|
DESCRIPTION: Pre-Scan für den LocalBatchCache (Pass 1).
|
||||||
|
AUDIT v1.1.0: Dynamisierung der Scan-Parameter (WP-14).
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from typing import Optional, Dict, Any
|
||||||
|
from .parsing_models import NoteContext
|
||||||
|
from .parsing_markdown import read_markdown
|
||||||
|
|
||||||
|
def pre_scan_markdown(path: str, registry: Optional[Dict[str, Any]] = None) -> Optional[NoteContext]:
|
||||||
|
"""
|
||||||
|
Extrahiert Identität und Kurz-Kontext zur Validierung.
|
||||||
|
WP-14: Scan-Tiefe und Summary-Länge sind nun über die Registry steuerbar.
|
||||||
|
"""
|
||||||
|
parsed = read_markdown(path)
|
||||||
|
if not parsed: return None
|
||||||
|
|
||||||
|
# WP-14: Konfiguration laden oder Standardwerte nutzen
|
||||||
|
reg = registry or {}
|
||||||
|
summary_cfg = reg.get("summary_settings", {})
|
||||||
|
scan_depth = summary_cfg.get("pre_scan_depth", 600)
|
||||||
|
max_len = summary_cfg.get("max_summary_length", 500)
|
||||||
|
|
||||||
|
fm = parsed.frontmatter
|
||||||
|
# ID-Findung: Frontmatter ID oder Dateiname als Fallback
|
||||||
|
note_id = str(fm.get("id") or os.path.splitext(os.path.basename(path))[0])
|
||||||
|
|
||||||
|
# Erstelle Kurz-Zusammenfassung mit dynamischen Limits
|
||||||
|
clean_body = re.sub(r'[#*`>]', '', parsed.body[:scan_depth]).strip()
|
||||||
|
summary = clean_body[:max_len] + "..." if len(clean_body) > max_len else clean_body
|
||||||
|
|
||||||
|
return NoteContext(
|
||||||
|
note_id=note_id,
|
||||||
|
title=str(fm.get("title", note_id)),
|
||||||
|
type=str(fm.get("type", "concept")),
|
||||||
|
summary=summary,
|
||||||
|
tags=fm.get("tags", []) if isinstance(fm.get("tags"), list) else []
|
||||||
|
)
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/parsing/parsing_utils.py
|
||||||
|
DESCRIPTION: Werkzeuge zur Validierung, Normalisierung und Wikilink-Extraktion.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from typing import Any, Dict, List, Tuple, Optional
|
||||||
|
from .parsing_models import ParsedNote
|
||||||
|
|
||||||
|
# Öffentliche Konstanten für Abwärtskompatibilität
|
||||||
|
FRONTMATTER_RE = re.compile(r"^\s*---\s*$")
|
||||||
|
_WIKILINK_RE = re.compile(r"\[\[([^\]]+)\]\]")
|
||||||
|
|
||||||
|
def validate_required_frontmatter(fm: Dict[str, Any], required: Tuple[str, ...] = ("id", "title")) -> None:
|
||||||
|
"""Prüft, ob alle Pflichtfelder vorhanden sind."""
|
||||||
|
if fm is None: fm = {}
|
||||||
|
missing = []
|
||||||
|
for k in required:
|
||||||
|
v = fm.get(k)
|
||||||
|
if v is None or (isinstance(v, str) and not v.strip()):
|
||||||
|
missing.append(k)
|
||||||
|
if missing:
|
||||||
|
raise ValueError(f"Missing required frontmatter fields: {', '.join(missing)}")
|
||||||
|
if "tags" in fm and fm["tags"] not in (None, "") and not isinstance(fm["tags"], (list, tuple)):
|
||||||
|
raise ValueError("frontmatter 'tags' must be a list of strings")
|
||||||
|
|
||||||
|
def normalize_frontmatter(fm: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
|
"""Normalisierung von Tags und Boolean-Feldern."""
|
||||||
|
out = dict(fm or {})
|
||||||
|
if "tags" in out:
|
||||||
|
if isinstance(out["tags"], str):
|
||||||
|
out["tags"] = [out["tags"].strip()] if out["tags"].strip() else []
|
||||||
|
elif isinstance(out["tags"], list):
|
||||||
|
out["tags"] = [str(t).strip() for t in out["tags"] if t is not None]
|
||||||
|
else:
|
||||||
|
out["tags"] = [str(out["tags"]).strip()] if out["tags"] not in (None, "") else []
|
||||||
|
if "embedding_exclude" in out:
|
||||||
|
out["embedding_exclude"] = bool(out["embedding_exclude"])
|
||||||
|
return out
|
||||||
|
|
||||||
|
def extract_wikilinks(text: str) -> List[str]:
|
||||||
|
"""Extrahiert Wikilinks als einfache Liste von IDs."""
|
||||||
|
if not text: return []
|
||||||
|
out: List[str] = []
|
||||||
|
for m in _WIKILINK_RE.finditer(text):
|
||||||
|
raw = (m.group(1) or "").strip()
|
||||||
|
if not raw: continue
|
||||||
|
if "|" in raw: raw = raw.split("|", 1)[0].strip()
|
||||||
|
if "#" in raw: raw = raw.split("#", 1)[0].strip()
|
||||||
|
if raw: out.append(raw)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def extract_edges_with_context(parsed: ParsedNote) -> List[Dict[str, Any]]:
|
||||||
|
"""WP-22: Extrahiert Wikilinks mit Zeilennummern für die EdgeRegistry."""
|
||||||
|
edges = []
|
||||||
|
if not parsed or not parsed.body: return edges
|
||||||
|
lines = parsed.body.splitlines()
|
||||||
|
for line_num, line_content in enumerate(lines, 1):
|
||||||
|
for match in _WIKILINK_RE.finditer(line_content):
|
||||||
|
raw = (match.group(1) or "").strip()
|
||||||
|
if not raw: continue
|
||||||
|
if "|" in raw:
|
||||||
|
parts = raw.split("|", 1)
|
||||||
|
target, kind = parts[0].strip(), parts[1].strip()
|
||||||
|
else:
|
||||||
|
target, kind = raw.strip(), "related_to"
|
||||||
|
if "#" in target: target = target.split("#", 1)[0].strip()
|
||||||
|
if target:
|
||||||
|
edges.append({"to": target, "kind": kind, "line": line_num, "provenance": "explicit"})
|
||||||
|
return edges
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
"""
|
|
||||||
app/core/ranking.py — Kombiniertes Scoring (WP-04)
|
|
||||||
|
|
||||||
Zweck:
|
|
||||||
Zusammenführen von semantischem Score (normalisiert), Edge-Bonus und
|
|
||||||
Centrality-Bonus in einen Gesamtscore für die Ergebnisreihung.
|
|
||||||
Kompatibilität:
|
|
||||||
Python 3.12+
|
|
||||||
Version:
|
|
||||||
0.1.0 (Erstanlage)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Bezug:
|
|
||||||
WP-04 Ranking-Formel (w_sem, w_edge, w_cent)
|
|
||||||
Nutzung:
|
|
||||||
from app.core.ranking import combine_scores
|
|
||||||
Änderungsverlauf:
|
|
||||||
0.1.0 (2025-10-07) – Erstanlage.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
from typing import List, Tuple, Dict
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_scores(values: List[float]) -> List[float]:
|
|
||||||
"""Min-Max-Normalisierung über die Kandidatenmenge (Fallback 0.5 bei Konstanz)."""
|
|
||||||
if not values:
|
|
||||||
return values
|
|
||||||
lo, hi = min(values), max(values)
|
|
||||||
if hi - lo < 1e-9:
|
|
||||||
return [0.5] * len(values)
|
|
||||||
return [(v - lo) / (hi - lo) for v in values]
|
|
||||||
|
|
||||||
|
|
||||||
def combine_scores(
|
|
||||||
hits: List[Tuple[str, float, dict]], # (id, semantic_score, payload)
|
|
||||||
edge_bonus_map: Dict[str, float],
|
|
||||||
centrality_map: Dict[str, float],
|
|
||||||
w_sem: float = 0.70,
|
|
||||||
w_edge: float = 0.25,
|
|
||||||
w_cent: float = 0.05,
|
|
||||||
) -> List[Tuple[str, float, float, float, float]]:
|
|
||||||
"""
|
|
||||||
Liefert Liste von (point_id, total_score, edge_bonus, centrality_bonus, raw_semantic_score),
|
|
||||||
absteigend nach total_score sortiert.
|
|
||||||
"""
|
|
||||||
sem = [h[1] for h in hits]
|
|
||||||
sem_n = normalize_scores(sem)
|
|
||||||
out = []
|
|
||||||
for (pid, s, payload), s_norm in zip(hits, sem_n):
|
|
||||||
e = edge_bonus_map.get(pid, 0.0)
|
|
||||||
c = centrality_map.get(pid, 0.0)
|
|
||||||
total = w_sem * s_norm + w_edge * e + w_cent * c
|
|
||||||
out.append((pid, total, e, c, s))
|
|
||||||
out.sort(key=lambda t: t[1], reverse=True)
|
|
||||||
return out
|
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/registry.py
|
||||||
|
DESCRIPTION: Zentraler Base-Layer für Konfigurations-Loading und Text-Bereinigung.
|
||||||
|
Bricht Zirkelbezüge zwischen Ingestion und LLMService auf.
|
||||||
|
VERSION: 1.0.0
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import yaml
|
||||||
|
from typing import Optional, List
|
||||||
|
|
||||||
|
def load_type_registry(custom_path: Optional[str] = None) -> dict:
|
||||||
|
"""Lädt die types.yaml zur Steuerung der typ-spezifischen Logik."""
|
||||||
|
# Wir nutzen hier einen direkten Import von Settings, um Zyklen zu vermeiden
|
||||||
|
from app.config import get_settings
|
||||||
|
settings = get_settings()
|
||||||
|
path = custom_path or settings.MINDNET_TYPES_FILE
|
||||||
|
if not os.path.exists(path):
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
return yaml.safe_load(f) or {}
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def clean_llm_text(text: str, registry: Optional[dict] = None) -> str:
|
||||||
|
"""
|
||||||
|
Entfernt LLM-Steuerzeichen (<s>, [OUT] etc.) aus einem Text.
|
||||||
|
Wird sowohl für JSON-Parsing als auch für Chat-Antworten genutzt.
|
||||||
|
"""
|
||||||
|
if not text or not isinstance(text, str):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
default_patterns = ["<s>", "</s>", "[OUT]", "[/OUT]"]
|
||||||
|
reg = registry or load_type_registry()
|
||||||
|
|
||||||
|
# Lade Patterns aus llm_settings (WP-14)
|
||||||
|
patterns: List[str] = reg.get("llm_settings", {}).get("cleanup_patterns", default_patterns)
|
||||||
|
|
||||||
|
clean = text
|
||||||
|
for p in patterns:
|
||||||
|
clean = clean.replace(p, "")
|
||||||
|
|
||||||
|
return clean.strip()
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
"""
|
||||||
|
PACKAGE: app.core.retrieval
|
||||||
|
DESCRIPTION: Zentrale Schnittstelle für Retrieval-Operationen (Vektor- & Graph-Suche).
|
||||||
|
Bündelt Suche und mathematische Scoring-Engine.
|
||||||
|
"""
|
||||||
|
from .retriever import (
|
||||||
|
Retriever,
|
||||||
|
hybrid_retrieve,
|
||||||
|
semantic_retrieve
|
||||||
|
)
|
||||||
|
|
||||||
|
from .retriever_scoring import (
|
||||||
|
get_weights,
|
||||||
|
compute_wp22_score,
|
||||||
|
get_status_multiplier
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"Retriever",
|
||||||
|
"hybrid_retrieve",
|
||||||
|
"semantic_retrieve",
|
||||||
|
"get_weights",
|
||||||
|
"compute_wp22_score",
|
||||||
|
"get_status_multiplier"
|
||||||
|
]
|
||||||
@@ -0,0 +1,374 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/retrieval/retriever.py
|
||||||
|
DESCRIPTION: Haupt-Schnittstelle für die Suche. Orchestriert Vektorsuche und Graph-Expansion.
|
||||||
|
WP-15c Update: Note-Level Diversity Pooling & Super-Edge Aggregation.
|
||||||
|
VERSION: 0.7.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: app.config, app.models.dto, app.core.database*, app.core.graph_adapter
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from typing import Any, Dict, List, Tuple, Iterable, Optional
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
from app.config import get_settings
|
||||||
|
from app.models.dto import (
|
||||||
|
QueryRequest, QueryResponse, QueryHit,
|
||||||
|
Explanation, ScoreBreakdown, Reason, EdgeDTO
|
||||||
|
)
|
||||||
|
|
||||||
|
# MODULARISIERUNG: Neue Import-Pfade für die Datenbank-Ebene
|
||||||
|
import app.core.database.qdrant as qdr
|
||||||
|
import app.core.database.qdrant_points as qp
|
||||||
|
|
||||||
|
import app.services.embeddings_client as ec
|
||||||
|
import app.core.graph.graph_subgraph as ga
|
||||||
|
|
||||||
|
# Mathematische Engine importieren
|
||||||
|
from app.core.retrieval.retriever_scoring import get_weights, compute_wp22_score
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 1. CORE HELPERS & CONFIG LOADERS
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
def _get_client_and_prefix() -> Tuple[Any, str]:
|
||||||
|
"""Initialisiert Qdrant Client und lädt Collection-Prefix via database-Paket."""
|
||||||
|
cfg = qdr.QdrantConfig.from_env()
|
||||||
|
return qdr.get_client(cfg), cfg.prefix
|
||||||
|
|
||||||
|
|
||||||
|
def _get_query_vector(req: QueryRequest) -> List[float]:
|
||||||
|
"""
|
||||||
|
Vektorisiert die Anfrage.
|
||||||
|
FIX: Enthält try-except Block für unterschiedliche Signaturen von ec.embed_text.
|
||||||
|
"""
|
||||||
|
if req.query_vector:
|
||||||
|
return list(req.query_vector)
|
||||||
|
if not req.query:
|
||||||
|
raise ValueError("Kein Text oder Vektor für die Suche angegeben.")
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Versuch mit modernem Interface (WP-03 kompatibel)
|
||||||
|
return ec.embed_text(req.query, model_name=settings.MODEL_NAME)
|
||||||
|
except TypeError:
|
||||||
|
# Fallback für Signaturen, die 'model_name' nicht als Keyword akzeptieren
|
||||||
|
logger.debug("ec.embed_text does not accept 'model_name' keyword. Falling back.")
|
||||||
|
return ec.embed_text(req.query)
|
||||||
|
|
||||||
|
|
||||||
|
def _semantic_hits(
|
||||||
|
client: Any,
|
||||||
|
prefix: str,
|
||||||
|
vector: List[float],
|
||||||
|
top_k: int,
|
||||||
|
filters: Optional[Dict] = None
|
||||||
|
) -> List[Tuple[str, float, Dict[str, Any]]]:
|
||||||
|
"""Führt die Vektorsuche via database-Points-Modul durch."""
|
||||||
|
raw_hits = qp.search_chunks_by_vector(client, prefix, vector, top=top_k, filters=filters)
|
||||||
|
# Strikte Typkonvertierung für Stabilität
|
||||||
|
return [(str(hit[0]), float(hit[1]), dict(hit[2] or {})) for hit in raw_hits]
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 2. EXPLANATION LAYER (DEBUG & VERIFIABILITY)
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
def _build_explanation(
|
||||||
|
semantic_score: float,
|
||||||
|
payload: Dict[str, Any],
|
||||||
|
scoring_debug: Dict[str, Any],
|
||||||
|
subgraph: Optional[ga.Subgraph],
|
||||||
|
target_note_id: Optional[str],
|
||||||
|
applied_boosts: Optional[Dict[str, float]] = None
|
||||||
|
) -> Explanation:
|
||||||
|
"""
|
||||||
|
Transformiert mathematische Scores und Graph-Signale in eine menschenlesbare Erklärung.
|
||||||
|
"""
|
||||||
|
_, edge_w_cfg, _ = get_weights()
|
||||||
|
base_val = scoring_debug["base_val"]
|
||||||
|
|
||||||
|
# 1. Detaillierter mathematischer Breakdown
|
||||||
|
breakdown = ScoreBreakdown(
|
||||||
|
semantic_contribution=base_val,
|
||||||
|
edge_contribution=base_val * scoring_debug["edge_impact_final"],
|
||||||
|
centrality_contribution=base_val * scoring_debug["cent_impact_final"],
|
||||||
|
raw_semantic=semantic_score,
|
||||||
|
raw_edge_bonus=scoring_debug["edge_bonus"],
|
||||||
|
raw_centrality=scoring_debug["cent_bonus"],
|
||||||
|
node_weight=float(payload.get("retriever_weight", 1.0)),
|
||||||
|
status_multiplier=scoring_debug["status_multiplier"],
|
||||||
|
graph_boost_factor=scoring_debug["graph_boost_factor"]
|
||||||
|
)
|
||||||
|
|
||||||
|
reasons: List[Reason] = []
|
||||||
|
edges_dto: List[EdgeDTO] = []
|
||||||
|
|
||||||
|
# 2. Gründe für Semantik hinzufügen
|
||||||
|
if semantic_score > 0.85:
|
||||||
|
reasons.append(Reason(kind="semantic", message="Sehr hohe textuelle Übereinstimmung.", score_impact=base_val))
|
||||||
|
elif semantic_score > 0.70:
|
||||||
|
reasons.append(Reason(kind="semantic", message="Inhaltliche Übereinstimmung.", score_impact=base_val))
|
||||||
|
|
||||||
|
# 3. Gründe für Typ und Lifecycle (WP-25 Vorbereitung)
|
||||||
|
type_weight = float(payload.get("retriever_weight", 1.0))
|
||||||
|
if type_weight != 1.0:
|
||||||
|
msg = "Bevorzugt" if type_weight > 1.0 else "De-priorisiert"
|
||||||
|
reasons.append(Reason(kind="type", message=f"{msg} durch Typ-Profil.", score_impact=base_val * (type_weight - 1.0)))
|
||||||
|
|
||||||
|
# NEU: Explizite Ausweisung des Lifecycle-Status (WP-22)
|
||||||
|
status_mult = scoring_debug.get("status_multiplier", 1.0)
|
||||||
|
if status_mult != 1.0:
|
||||||
|
status_msg = "Belohnt (Stable)" if status_mult > 1.0 else "De-priorisiert (Draft)"
|
||||||
|
reasons.append(Reason(
|
||||||
|
kind="status",
|
||||||
|
message=f"{status_msg} durch Content-Lifecycle.",
|
||||||
|
score_impact=semantic_score * (status_mult - 1.0)
|
||||||
|
))
|
||||||
|
|
||||||
|
# 4. Kanten-Verarbeitung (Graph-Intelligence)
|
||||||
|
if subgraph and target_note_id and scoring_debug["edge_bonus"] > 0:
|
||||||
|
raw_edges = []
|
||||||
|
if hasattr(subgraph, "get_incoming_edges"):
|
||||||
|
raw_edges.extend(subgraph.get_incoming_edges(target_note_id) or [])
|
||||||
|
if hasattr(subgraph, "get_outgoing_edges"):
|
||||||
|
raw_edges.extend(subgraph.get_outgoing_edges(target_note_id) or [])
|
||||||
|
|
||||||
|
for edge in raw_edges:
|
||||||
|
src = str(edge.get("source") or "note_root")
|
||||||
|
tgt = str(edge.get("target") or target_note_id or "unknown_target")
|
||||||
|
kind = str(edge.get("kind", "related_to"))
|
||||||
|
prov = str(edge.get("provenance", "rule"))
|
||||||
|
conf = float(edge.get("confidence", 1.0))
|
||||||
|
|
||||||
|
direction = "in" if tgt == target_note_id else "out"
|
||||||
|
|
||||||
|
edge_obj = EdgeDTO(
|
||||||
|
id=f"{src}->{tgt}:{kind}",
|
||||||
|
kind=kind,
|
||||||
|
source=src,
|
||||||
|
target=tgt,
|
||||||
|
weight=conf,
|
||||||
|
direction=direction,
|
||||||
|
provenance=prov,
|
||||||
|
confidence=conf
|
||||||
|
)
|
||||||
|
edges_dto.append(edge_obj)
|
||||||
|
|
||||||
|
# Die 3 wichtigsten Kanten als Begründung formulieren
|
||||||
|
top_edges = sorted(edges_dto, key=lambda e: e.confidence, reverse=True)
|
||||||
|
for e in top_edges[:3]:
|
||||||
|
peer = e.source if e.direction == "in" else e.target
|
||||||
|
prov_txt = "Bestätigte" if e.provenance == "explicit" else "KI-basierte"
|
||||||
|
boost_txt = f" [Boost x{applied_boosts.get(e.kind)}]" if applied_boosts and e.kind in applied_boosts else ""
|
||||||
|
|
||||||
|
reasons.append(Reason(
|
||||||
|
kind="edge",
|
||||||
|
message=f"{prov_txt} Kante '{e.kind}'{boost_txt} von/zu '{peer}'.",
|
||||||
|
score_impact=edge_w_cfg * e.confidence
|
||||||
|
))
|
||||||
|
|
||||||
|
if scoring_debug["cent_bonus"] > 0.01:
|
||||||
|
reasons.append(Reason(kind="centrality", message="Die Notiz ist ein zentraler Informations-Hub.", score_impact=breakdown.centrality_contribution))
|
||||||
|
|
||||||
|
return Explanation(
|
||||||
|
breakdown=breakdown,
|
||||||
|
reasons=reasons,
|
||||||
|
related_edges=edges_dto if edges_dto else None,
|
||||||
|
applied_boosts=applied_boosts
|
||||||
|
)
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 3. CORE RETRIEVAL PIPELINE
|
||||||
|
# ==============================================================================
|
||||||
|
|
||||||
|
def _build_hits_from_semantic(
|
||||||
|
hits: Iterable[Tuple[str, float, Dict[str, Any]]],
|
||||||
|
top_k: int,
|
||||||
|
used_mode: str,
|
||||||
|
subgraph: ga.Subgraph | None = None,
|
||||||
|
explain: bool = False,
|
||||||
|
dynamic_edge_boosts: Dict[str, float] = None
|
||||||
|
) -> QueryResponse:
|
||||||
|
"""
|
||||||
|
Wandelt semantische Roh-Treffer in bewertete QueryHits um.
|
||||||
|
WP-15c: Implementiert Note-Level Diversity Pooling.
|
||||||
|
"""
|
||||||
|
t0 = time.time()
|
||||||
|
enriched = []
|
||||||
|
|
||||||
|
# Erstes Scoring für alle Kandidaten
|
||||||
|
for pid, semantic_score, payload in hits:
|
||||||
|
edge_bonus, cent_bonus = 0.0, 0.0
|
||||||
|
target_id = payload.get("note_id")
|
||||||
|
|
||||||
|
if subgraph and target_id:
|
||||||
|
try:
|
||||||
|
edge_bonus = float(subgraph.edge_bonus(target_id))
|
||||||
|
cent_bonus = float(subgraph.centrality_bonus(target_id))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
debug_data = compute_wp22_score(
|
||||||
|
semantic_score, payload, edge_bonus, cent_bonus, dynamic_edge_boosts
|
||||||
|
)
|
||||||
|
enriched.append((pid, semantic_score, payload, debug_data))
|
||||||
|
|
||||||
|
# 1. Sortierung nach finalem mathematischen Score
|
||||||
|
enriched_sorted = sorted(enriched, key=lambda h: h[3]["total"], reverse=True)
|
||||||
|
|
||||||
|
# 2. WP-15c: Note-Level Diversity Pooling
|
||||||
|
# Wir behalten pro note_id nur den Hit mit dem höchsten total_score.
|
||||||
|
# Dies verhindert, dass 10 Chunks derselben Note andere KeyNotes verdrängen.
|
||||||
|
unique_note_hits = []
|
||||||
|
seen_notes = set()
|
||||||
|
|
||||||
|
for item in enriched_sorted:
|
||||||
|
_, _, payload, _ = item
|
||||||
|
note_id = str(payload.get("note_id", "unknown"))
|
||||||
|
|
||||||
|
if note_id not in seen_notes:
|
||||||
|
unique_note_hits.append(item)
|
||||||
|
seen_notes.add(note_id)
|
||||||
|
|
||||||
|
# 3. Begrenzung auf top_k nach dem Diversity-Pooling
|
||||||
|
limited_hits = unique_note_hits[: max(1, top_k)]
|
||||||
|
|
||||||
|
results: List[QueryHit] = []
|
||||||
|
for pid, s_score, pl, dbg in limited_hits:
|
||||||
|
explanation_obj = None
|
||||||
|
if explain:
|
||||||
|
explanation_obj = _build_explanation(
|
||||||
|
semantic_score=float(s_score),
|
||||||
|
payload=pl,
|
||||||
|
scoring_debug=dbg,
|
||||||
|
subgraph=subgraph,
|
||||||
|
target_note_id=pl.get("note_id"),
|
||||||
|
applied_boosts=dynamic_edge_boosts
|
||||||
|
)
|
||||||
|
|
||||||
|
text_content = pl.get("page_content") or pl.get("text") or pl.get("content", "[Kein Text]")
|
||||||
|
|
||||||
|
results.append(QueryHit(
|
||||||
|
node_id=str(pid),
|
||||||
|
note_id=str(pl.get("note_id", "unknown")),
|
||||||
|
semantic_score=float(s_score),
|
||||||
|
edge_bonus=dbg["edge_bonus"],
|
||||||
|
centrality_bonus=dbg["cent_bonus"],
|
||||||
|
total_score=dbg["total"],
|
||||||
|
source={
|
||||||
|
"path": pl.get("path"),
|
||||||
|
"section": pl.get("section") or pl.get("section_title"),
|
||||||
|
"text": text_content
|
||||||
|
},
|
||||||
|
payload=pl,
|
||||||
|
explanation=explanation_obj
|
||||||
|
))
|
||||||
|
|
||||||
|
return QueryResponse(results=results, used_mode=used_mode, latency_ms=int((time.time() - t0) * 1000))
|
||||||
|
|
||||||
|
|
||||||
|
def hybrid_retrieve(req: QueryRequest) -> QueryResponse:
|
||||||
|
"""
|
||||||
|
Die Haupt-Einstiegsfunktion für die hybride Suche.
|
||||||
|
WP-15c: Implementiert Edge-Aggregation (Super-Kanten).
|
||||||
|
"""
|
||||||
|
client, prefix = _get_client_and_prefix()
|
||||||
|
vector = list(req.query_vector) if req.query_vector else _get_query_vector(req)
|
||||||
|
top_k = req.top_k or 10
|
||||||
|
|
||||||
|
# 1. Semantische Seed-Suche (Wir laden etwas mehr für das Pooling)
|
||||||
|
hits = _semantic_hits(client, prefix, vector, top_k=top_k * 3, filters=req.filters)
|
||||||
|
|
||||||
|
# 2. Graph Expansion Konfiguration
|
||||||
|
expand_cfg = req.expand if isinstance(req.expand, dict) else {}
|
||||||
|
depth = int(expand_cfg.get("depth", 1))
|
||||||
|
boost_edges = getattr(req, "boost_edges", {}) or {}
|
||||||
|
|
||||||
|
subgraph: ga.Subgraph | None = None
|
||||||
|
if depth > 0 and hits:
|
||||||
|
seed_ids = list({h[2].get("note_id") for h in hits if h[2].get("note_id")})
|
||||||
|
|
||||||
|
if seed_ids:
|
||||||
|
try:
|
||||||
|
subgraph = ga.expand(client, prefix, seed_ids, depth=depth, edge_types=expand_cfg.get("edge_types"))
|
||||||
|
|
||||||
|
# --- WP-15c: Edge-Aggregation & Deduplizierung (Super-Kanten) ---
|
||||||
|
# Verhindert Score-Explosion durch multiple Links auf versch. Abschnitte.
|
||||||
|
# Logik: 1. Kante zählt voll, weitere dämpfen auf Faktor 0.1.
|
||||||
|
if subgraph and hasattr(subgraph, "adj"):
|
||||||
|
for src, edge_list in subgraph.adj.items():
|
||||||
|
# Gruppiere Kanten nach Ziel-Note (Deduplizierung ID_A -> ID_B)
|
||||||
|
by_target = defaultdict(list)
|
||||||
|
for e in edge_list:
|
||||||
|
by_target[e["target"]].append(e)
|
||||||
|
|
||||||
|
aggregated_list = []
|
||||||
|
for tgt, edges in by_target.items():
|
||||||
|
if len(edges) > 1:
|
||||||
|
# Sortiere: Stärkste Kante zuerst
|
||||||
|
sorted_edges = sorted(edges, key=lambda x: x.get("weight", 0.0), reverse=True)
|
||||||
|
primary = sorted_edges[0]
|
||||||
|
|
||||||
|
# Aggregiertes Gewicht berechnen (Sättigungs-Logik)
|
||||||
|
total_w = primary.get("weight", 0.0)
|
||||||
|
for secondary in sorted_edges[1:]:
|
||||||
|
total_w += secondary.get("weight", 0.0) * 0.1
|
||||||
|
|
||||||
|
primary["weight"] = total_w
|
||||||
|
primary["is_super_edge"] = True # Flag für Explanation Layer
|
||||||
|
primary["edge_count"] = len(edges)
|
||||||
|
aggregated_list.append(primary)
|
||||||
|
else:
|
||||||
|
aggregated_list.append(edges[0])
|
||||||
|
|
||||||
|
# In-Place Update der Adjazenzliste des Graphen
|
||||||
|
subgraph.adj[src] = aggregated_list
|
||||||
|
|
||||||
|
# Re-Sync der In-Degrees für Centrality-Bonus (Aggregation konsistent halten)
|
||||||
|
subgraph.in_degree = defaultdict(int)
|
||||||
|
for src, edges in subgraph.adj.items():
|
||||||
|
for e in edges:
|
||||||
|
subgraph.in_degree[e["target"]] += 1
|
||||||
|
|
||||||
|
# --- WP-22: Kanten-Gewichtung (Provenance & Intent Boost) ---
|
||||||
|
if subgraph and hasattr(subgraph, "adj"):
|
||||||
|
for src, edges in subgraph.adj.items():
|
||||||
|
for e in edges:
|
||||||
|
# A. Provenance Weighting
|
||||||
|
prov = e.get("provenance", "rule")
|
||||||
|
prov_w = 1.0 if prov == "explicit" else (0.9 if prov == "smart" else 0.7)
|
||||||
|
|
||||||
|
# B. Intent Boost Multiplikator
|
||||||
|
kind = e.get("kind")
|
||||||
|
intent_multiplier = boost_edges.get(kind, 1.0)
|
||||||
|
|
||||||
|
# Gewichtung anpassen
|
||||||
|
e["weight"] = e.get("weight", 1.0) * prov_w * intent_multiplier
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Graph Expansion failed: {e}")
|
||||||
|
subgraph = None
|
||||||
|
|
||||||
|
# 3. Scoring & Explanation Generierung
|
||||||
|
# top_k wird erst hier final angewandt
|
||||||
|
return _build_hits_from_semantic(hits, top_k, "hybrid", subgraph, req.explain, boost_edges)
|
||||||
|
|
||||||
|
|
||||||
|
def semantic_retrieve(req: QueryRequest) -> QueryResponse:
|
||||||
|
"""Standard Vektorsuche ohne Graph-Einfluss."""
|
||||||
|
client, prefix = _get_client_and_prefix()
|
||||||
|
vector = _get_query_vector(req)
|
||||||
|
hits = _semantic_hits(client, prefix, vector, req.top_k or 10, req.filters)
|
||||||
|
return _build_hits_from_semantic(hits, req.top_k or 10, "semantic", explain=req.explain)
|
||||||
|
|
||||||
|
|
||||||
|
class Retriever:
|
||||||
|
"""Schnittstelle für die asynchrone Suche."""
|
||||||
|
async def search(self, request: QueryRequest) -> QueryResponse:
|
||||||
|
return hybrid_retrieve(request)
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/core/retrieval/retriever_scoring.py
|
||||||
|
DESCRIPTION: Mathematische Kern-Logik für das WP-22/WP-15c Scoring.
|
||||||
|
Berechnet Relevanz-Scores basierend auf Semantik, Graph-Intelligence und Content Lifecycle.
|
||||||
|
FIX v1.0.3: Optimierte Interaktion zwischen Typ-Boost und Status-Dämpfung.
|
||||||
|
VERSION: 1.0.3
|
||||||
|
STATUS: Active
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
from functools import lru_cache
|
||||||
|
from typing import Any, Dict, Tuple, Optional
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError:
|
||||||
|
yaml = None
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@lru_cache
|
||||||
|
def get_weights() -> Tuple[float, float, float]:
|
||||||
|
"""
|
||||||
|
Liefert die Basis-Gewichtung (semantic, edge, centrality) aus der Konfiguration.
|
||||||
|
"""
|
||||||
|
from app.config import get_settings
|
||||||
|
settings = get_settings()
|
||||||
|
|
||||||
|
# Defaults aus Settings laden
|
||||||
|
sem = float(getattr(settings, "RETRIEVER_W_SEM", 1.0))
|
||||||
|
edge = float(getattr(settings, "RETRIEVER_W_EDGE", 0.0))
|
||||||
|
cent = float(getattr(settings, "RETRIEVER_W_CENT", 0.0))
|
||||||
|
|
||||||
|
# Optionaler Override via YAML
|
||||||
|
config_path = os.getenv("MINDNET_RETRIEVER_CONFIG", "config/retriever.yaml")
|
||||||
|
if yaml and os.path.exists(config_path):
|
||||||
|
try:
|
||||||
|
with open(config_path, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
scoring = data.get("scoring", {})
|
||||||
|
sem = float(scoring.get("semantic_weight", sem))
|
||||||
|
edge = float(scoring.get("edge_weight", edge))
|
||||||
|
cent = float(scoring.get("centrality_weight", cent))
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Retriever Configuration could not be fully loaded from {config_path}: {e}")
|
||||||
|
|
||||||
|
return sem, edge, cent
|
||||||
|
|
||||||
|
def get_status_multiplier(payload: Dict[str, Any]) -> float:
|
||||||
|
"""
|
||||||
|
WP-22 A: Content Lifecycle Multiplier.
|
||||||
|
Steuert das Ranking basierend auf dem Reifegrad der Information.
|
||||||
|
|
||||||
|
- stable: 1.2 (Belohnung für verifiziertes Wissen)
|
||||||
|
- active: 1.0 (Standard-Gewichtung)
|
||||||
|
- draft: 0.5 (Dämpfung für unfertige Fragmente)
|
||||||
|
"""
|
||||||
|
status = str(payload.get("status", "active")).lower().strip()
|
||||||
|
if status == "stable":
|
||||||
|
return 1.2
|
||||||
|
if status == "draft":
|
||||||
|
return 0.5
|
||||||
|
return 1.0
|
||||||
|
|
||||||
|
def compute_wp22_score(
|
||||||
|
semantic_score: float,
|
||||||
|
payload: Dict[str, Any],
|
||||||
|
edge_bonus_raw: float = 0.0,
|
||||||
|
cent_bonus_raw: float = 0.0,
|
||||||
|
dynamic_edge_boosts: Optional[Dict[str, float]] = None
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Die zentrale mathematische Scoring-Formel (WP-15c optimiert).
|
||||||
|
Implementiert das Hybrid-Scoring (Semantic * Lifecycle * Graph).
|
||||||
|
|
||||||
|
LOGIK:
|
||||||
|
1. Base = Similarity * StatusMult (Lifecycle-Filter).
|
||||||
|
2. Boosts = (TypeBoost - 1) + (GraphBoni * IntentFactor).
|
||||||
|
3. Final = Base * (1 + Boosts).
|
||||||
|
|
||||||
|
Der edge_bonus_raw enthält bereits die Super-Edge-Aggregation (WP-15c).
|
||||||
|
"""
|
||||||
|
sem_w, edge_w_cfg, cent_w_cfg = get_weights()
|
||||||
|
status_mult = get_status_multiplier(payload)
|
||||||
|
|
||||||
|
# Retriever Weight (Typ-Boost aus types.yaml, z.B. 1.1 für Decisions)
|
||||||
|
node_weight = float(payload.get("retriever_weight", 1.0))
|
||||||
|
|
||||||
|
# 1. Berechnung des Base Scores (Semantik gewichtet durch Lifecycle-Status)
|
||||||
|
# WICHTIG: Der Status wirkt hier als Multiplikator auf die Basis-Relevanz.
|
||||||
|
base_val = float(semantic_score) * status_mult
|
||||||
|
|
||||||
|
# 2. Graph Boost Factor (Intent-spezifische Verstärkung aus decision_engine.yaml)
|
||||||
|
# Erhöht das Gewicht des gesamten Graphen um 50%, wenn ein spezifischer Intent vorliegt.
|
||||||
|
graph_boost_factor = 1.5 if dynamic_edge_boosts and (edge_bonus_raw > 0 or cent_bonus_raw > 0) else 1.0
|
||||||
|
|
||||||
|
# 3. Einzelne Graph-Komponenten berechnen
|
||||||
|
# WP-15c Hinweis: edge_bonus_raw ist durch den retriever.py bereits gedämpft/aggregiert.
|
||||||
|
edge_impact_final = (edge_w_cfg * edge_bonus_raw) * graph_boost_factor
|
||||||
|
cent_impact_final = (cent_w_cfg * cent_bonus_raw) * graph_boost_factor
|
||||||
|
|
||||||
|
# 4. Finales Zusammenführen (Merging)
|
||||||
|
# (node_weight - 1.0) wandelt das Gewicht in einen relativen Bonus um (z.B. 1.2 -> +0.2).
|
||||||
|
# Alle Boni werden addiert und wirken dann auf den base_val.
|
||||||
|
type_impact = node_weight - 1.0
|
||||||
|
total_boost = 1.0 + type_impact + edge_impact_final + cent_impact_final
|
||||||
|
|
||||||
|
total = base_val * total_boost
|
||||||
|
|
||||||
|
# Sicherstellen, dass der Score niemals 0 oder negativ ist (Floor)
|
||||||
|
final_score = max(0.0001, float(total))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"total": final_score,
|
||||||
|
"edge_bonus": float(edge_bonus_raw),
|
||||||
|
"cent_bonus": float(cent_bonus_raw),
|
||||||
|
"status_multiplier": status_mult,
|
||||||
|
"graph_boost_factor": graph_boost_factor,
|
||||||
|
"type_impact": type_impact,
|
||||||
|
"base_val": base_val,
|
||||||
|
"edge_impact_final": edge_impact_final,
|
||||||
|
"cent_impact_final": cent_impact_final
|
||||||
|
}
|
||||||
@@ -1,336 +0,0 @@
|
|||||||
"""
|
|
||||||
app/core/retriever.py — Hybrider Such-Algorithmus
|
|
||||||
|
|
||||||
Version:
|
|
||||||
0.5.3 (WP-06 Fix: Populate 'payload' in QueryHit for meta-data access)
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import time
|
|
||||||
from functools import lru_cache
|
|
||||||
from typing import Any, Dict, List, Tuple, Iterable, Optional
|
|
||||||
|
|
||||||
from app.config import get_settings
|
|
||||||
from app.models.dto import (
|
|
||||||
QueryRequest,
|
|
||||||
QueryResponse,
|
|
||||||
QueryHit,
|
|
||||||
Explanation,
|
|
||||||
ScoreBreakdown,
|
|
||||||
Reason,
|
|
||||||
EdgeDTO
|
|
||||||
)
|
|
||||||
import app.core.qdrant as qdr
|
|
||||||
import app.core.qdrant_points as qp
|
|
||||||
import app.services.embeddings_client as ec
|
|
||||||
import app.core.graph_adapter as ga
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml # type: ignore[import]
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
yaml = None # type: ignore[assignment]
|
|
||||||
|
|
||||||
|
|
||||||
@lru_cache
|
|
||||||
def _get_scoring_weights() -> Tuple[float, float, float]:
|
|
||||||
"""Liefert (semantic_weight, edge_weight, centrality_weight) für den Retriever."""
|
|
||||||
settings = get_settings()
|
|
||||||
sem = float(getattr(settings, "RETRIEVER_W_SEM", 1.0))
|
|
||||||
edge = float(getattr(settings, "RETRIEVER_W_EDGE", 0.0))
|
|
||||||
cent = float(getattr(settings, "RETRIEVER_W_CENT", 0.0))
|
|
||||||
|
|
||||||
config_path = os.getenv("MINDNET_RETRIEVER_CONFIG", "config/retriever.yaml")
|
|
||||||
if yaml is None:
|
|
||||||
return sem, edge, cent
|
|
||||||
try:
|
|
||||||
if os.path.exists(config_path):
|
|
||||||
with open(config_path, "r", encoding="utf-8") as f:
|
|
||||||
data = yaml.safe_load(f) or {}
|
|
||||||
scoring = data.get("scoring", {}) or {}
|
|
||||||
sem = float(scoring.get("semantic_weight", sem))
|
|
||||||
edge = float(scoring.get("edge_weight", edge))
|
|
||||||
cent = float(scoring.get("centrality_weight", cent))
|
|
||||||
except Exception:
|
|
||||||
return sem, edge, cent
|
|
||||||
return sem, edge, cent
|
|
||||||
|
|
||||||
|
|
||||||
def _get_client_and_prefix() -> Tuple[Any, str]:
|
|
||||||
"""Liefert (QdrantClient, prefix)."""
|
|
||||||
cfg = qdr.QdrantConfig.from_env()
|
|
||||||
client = qdr.get_client(cfg)
|
|
||||||
return client, cfg.prefix
|
|
||||||
|
|
||||||
|
|
||||||
def _get_query_vector(req: QueryRequest) -> List[float]:
|
|
||||||
"""Liefert den Query-Vektor aus dem Request."""
|
|
||||||
if req.query_vector:
|
|
||||||
return list(req.query_vector)
|
|
||||||
|
|
||||||
if not req.query:
|
|
||||||
raise ValueError("QueryRequest benötigt entweder query oder query_vector")
|
|
||||||
|
|
||||||
settings = get_settings()
|
|
||||||
model_name = settings.MODEL_NAME
|
|
||||||
|
|
||||||
try:
|
|
||||||
return ec.embed_text(req.query, model_name=model_name)
|
|
||||||
except TypeError:
|
|
||||||
return ec.embed_text(req.query)
|
|
||||||
|
|
||||||
|
|
||||||
def _semantic_hits(
|
|
||||||
client: Any,
|
|
||||||
prefix: str,
|
|
||||||
vector: List[float],
|
|
||||||
top_k: int,
|
|
||||||
filters: Dict[str, Any] | None = None,
|
|
||||||
) -> List[Tuple[str, float, Dict[str, Any]]]:
|
|
||||||
"""Führt eine semantische Suche aus."""
|
|
||||||
flt = filters or None
|
|
||||||
raw_hits = qp.search_chunks_by_vector(client, prefix, vector, top=top_k, filters=flt)
|
|
||||||
results: List[Tuple[str, float, Dict[str, Any]]] = []
|
|
||||||
for pid, score, payload in raw_hits:
|
|
||||||
results.append((str(pid), float(score), dict(payload or {})))
|
|
||||||
return results
|
|
||||||
|
|
||||||
|
|
||||||
def _compute_total_score(
|
|
||||||
semantic_score: float,
|
|
||||||
payload: Dict[str, Any],
|
|
||||||
edge_bonus: float = 0.0,
|
|
||||||
cent_bonus: float = 0.0,
|
|
||||||
) -> Tuple[float, float, float]:
|
|
||||||
"""Berechnet total_score."""
|
|
||||||
raw_weight = payload.get("retriever_weight", 1.0)
|
|
||||||
try:
|
|
||||||
weight = float(raw_weight)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
weight = 1.0
|
|
||||||
if weight < 0.0:
|
|
||||||
weight = 0.0
|
|
||||||
|
|
||||||
sem_w, edge_w, cent_w = _get_scoring_weights()
|
|
||||||
total = (sem_w * float(semantic_score) * weight) + (edge_w * edge_bonus) + (cent_w * cent_bonus)
|
|
||||||
return float(total), float(edge_bonus), float(cent_bonus)
|
|
||||||
|
|
||||||
|
|
||||||
# --- WP-04b Explanation Logic ---
|
|
||||||
|
|
||||||
def _build_explanation(
|
|
||||||
semantic_score: float,
|
|
||||||
payload: Dict[str, Any],
|
|
||||||
edge_bonus: float,
|
|
||||||
cent_bonus: float,
|
|
||||||
subgraph: Optional[ga.Subgraph],
|
|
||||||
node_key: Optional[str]
|
|
||||||
) -> Explanation:
|
|
||||||
"""Erstellt ein Explanation-Objekt."""
|
|
||||||
sem_w, _edge_w, _cent_w = _get_scoring_weights()
|
|
||||||
# Scoring weights erneut laden für Reason-Details
|
|
||||||
_, edge_w_cfg, cent_w_cfg = _get_scoring_weights()
|
|
||||||
|
|
||||||
try:
|
|
||||||
type_weight = float(payload.get("retriever_weight", 1.0))
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
type_weight = 1.0
|
|
||||||
|
|
||||||
note_type = payload.get("type", "unknown")
|
|
||||||
|
|
||||||
breakdown = ScoreBreakdown(
|
|
||||||
semantic_contribution=(sem_w * semantic_score * type_weight),
|
|
||||||
edge_contribution=(edge_w_cfg * edge_bonus),
|
|
||||||
centrality_contribution=(cent_w_cfg * cent_bonus),
|
|
||||||
raw_semantic=semantic_score,
|
|
||||||
raw_edge_bonus=edge_bonus,
|
|
||||||
raw_centrality=cent_bonus,
|
|
||||||
node_weight=type_weight
|
|
||||||
)
|
|
||||||
|
|
||||||
reasons: List[Reason] = []
|
|
||||||
edges_dto: List[EdgeDTO] = []
|
|
||||||
|
|
||||||
if semantic_score > 0.85:
|
|
||||||
reasons.append(Reason(kind="semantic", message="Sehr hohe textuelle Übereinstimmung.", score_impact=breakdown.semantic_contribution))
|
|
||||||
elif semantic_score > 0.70:
|
|
||||||
reasons.append(Reason(kind="semantic", message="Gute textuelle Übereinstimmung.", score_impact=breakdown.semantic_contribution))
|
|
||||||
|
|
||||||
if type_weight != 1.0:
|
|
||||||
msg = "Bevorzugt" if type_weight > 1.0 else "Leicht abgewertet"
|
|
||||||
reasons.append(Reason(kind="type", message=f"{msg} aufgrund des Typs '{note_type}'.", score_impact=(sem_w * semantic_score * (type_weight - 1.0))))
|
|
||||||
|
|
||||||
if subgraph and node_key and edge_bonus > 0:
|
|
||||||
if hasattr(subgraph, "get_outgoing_edges"):
|
|
||||||
outgoing = subgraph.get_outgoing_edges(node_key)
|
|
||||||
for edge in outgoing:
|
|
||||||
target = edge.get("target", "Unknown")
|
|
||||||
kind = edge.get("kind", "edge")
|
|
||||||
weight = edge.get("weight", 0.0)
|
|
||||||
if weight > 0.05:
|
|
||||||
edges_dto.append(EdgeDTO(id=f"{node_key}->{target}:{kind}", kind=kind, source=node_key, target=target, weight=weight, direction="out"))
|
|
||||||
|
|
||||||
if hasattr(subgraph, "get_incoming_edges"):
|
|
||||||
incoming = subgraph.get_incoming_edges(node_key)
|
|
||||||
for edge in incoming:
|
|
||||||
src = edge.get("source", "Unknown")
|
|
||||||
kind = edge.get("kind", "edge")
|
|
||||||
weight = edge.get("weight", 0.0)
|
|
||||||
if weight > 0.05:
|
|
||||||
edges_dto.append(EdgeDTO(id=f"{src}->{node_key}:{kind}", kind=kind, source=src, target=node_key, weight=weight, direction="in"))
|
|
||||||
|
|
||||||
all_edges = sorted(edges_dto, key=lambda e: e.weight, reverse=True)
|
|
||||||
for top_edge in all_edges[:3]:
|
|
||||||
impact = edge_w_cfg * top_edge.weight
|
|
||||||
dir_txt = "Verweist auf" if top_edge.direction == "out" else "Referenziert von"
|
|
||||||
tgt_txt = top_edge.target if top_edge.direction == "out" else top_edge.source
|
|
||||||
reasons.append(Reason(kind="edge", message=f"{dir_txt} '{tgt_txt}' via '{top_edge.kind}'", score_impact=impact, details={"kind": top_edge.kind}))
|
|
||||||
|
|
||||||
if cent_bonus > 0.01:
|
|
||||||
reasons.append(Reason(kind="centrality", message="Knoten liegt zentral im Kontext.", score_impact=breakdown.centrality_contribution))
|
|
||||||
|
|
||||||
return Explanation(breakdown=breakdown, reasons=reasons, related_edges=edges_dto if edges_dto else None)
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_expand_options(req: QueryRequest) -> Tuple[int, List[str] | None]:
|
|
||||||
"""Extrahiert depth und edge_types."""
|
|
||||||
expand = getattr(req, "expand", None)
|
|
||||||
if not expand:
|
|
||||||
return 0, None
|
|
||||||
|
|
||||||
depth = 1
|
|
||||||
edge_types: List[str] | None = None
|
|
||||||
|
|
||||||
if hasattr(expand, "depth") or hasattr(expand, "edge_types"):
|
|
||||||
depth = int(getattr(expand, "depth", 1) or 1)
|
|
||||||
types_val = getattr(expand, "edge_types", None)
|
|
||||||
if types_val:
|
|
||||||
edge_types = list(types_val)
|
|
||||||
return depth, edge_types
|
|
||||||
|
|
||||||
if isinstance(expand, dict):
|
|
||||||
if "depth" in expand:
|
|
||||||
depth = int(expand.get("depth") or 1)
|
|
||||||
if "edge_types" in expand and expand["edge_types"] is not None:
|
|
||||||
edge_types = list(expand["edge_types"])
|
|
||||||
return depth, edge_types
|
|
||||||
|
|
||||||
return 0, None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_hits_from_semantic(
|
|
||||||
hits: Iterable[Tuple[str, float, Dict[str, Any]]],
|
|
||||||
top_k: int,
|
|
||||||
used_mode: str,
|
|
||||||
subgraph: ga.Subgraph | None = None,
|
|
||||||
explain: bool = False,
|
|
||||||
) -> QueryResponse:
|
|
||||||
"""Baut strukturierte QueryHits."""
|
|
||||||
t0 = time.time()
|
|
||||||
enriched: List[Tuple[str, float, Dict[str, Any], float, float, float]] = []
|
|
||||||
|
|
||||||
for pid, semantic_score, payload in hits:
|
|
||||||
edge_bonus = 0.0
|
|
||||||
cent_bonus = 0.0
|
|
||||||
node_key = payload.get("chunk_id") or payload.get("note_id")
|
|
||||||
|
|
||||||
if subgraph is not None and node_key:
|
|
||||||
try:
|
|
||||||
edge_bonus = float(subgraph.edge_bonus(node_key))
|
|
||||||
except Exception:
|
|
||||||
edge_bonus = 0.0
|
|
||||||
try:
|
|
||||||
cent_bonus = float(subgraph.centrality_bonus(node_key))
|
|
||||||
except Exception:
|
|
||||||
cent_bonus = 0.0
|
|
||||||
|
|
||||||
total, edge_bonus, cent_bonus = _compute_total_score(semantic_score, payload, edge_bonus=edge_bonus, cent_bonus=cent_bonus)
|
|
||||||
enriched.append((pid, float(semantic_score), payload, total, edge_bonus, cent_bonus))
|
|
||||||
|
|
||||||
enriched_sorted = sorted(enriched, key=lambda h: h[3], reverse=True)
|
|
||||||
limited = enriched_sorted[: max(1, top_k)]
|
|
||||||
|
|
||||||
results: List[QueryHit] = []
|
|
||||||
for pid, semantic_score, payload, total, edge_bonus, cent_bonus in limited:
|
|
||||||
explanation_obj = None
|
|
||||||
if explain:
|
|
||||||
explanation_obj = _build_explanation(
|
|
||||||
semantic_score=float(semantic_score),
|
|
||||||
payload=payload,
|
|
||||||
edge_bonus=edge_bonus,
|
|
||||||
cent_bonus=cent_bonus,
|
|
||||||
subgraph=subgraph,
|
|
||||||
node_key=payload.get("chunk_id") or payload.get("note_id")
|
|
||||||
)
|
|
||||||
|
|
||||||
text_content = payload.get("page_content") or payload.get("text") or payload.get("content")
|
|
||||||
|
|
||||||
results.append(QueryHit(
|
|
||||||
node_id=str(pid),
|
|
||||||
note_id=payload.get("note_id"),
|
|
||||||
semantic_score=float(semantic_score),
|
|
||||||
edge_bonus=edge_bonus,
|
|
||||||
centrality_bonus=cent_bonus,
|
|
||||||
total_score=total,
|
|
||||||
paths=None,
|
|
||||||
source={
|
|
||||||
"path": payload.get("path"),
|
|
||||||
"section": payload.get("section") or payload.get("section_title"),
|
|
||||||
"text": text_content
|
|
||||||
},
|
|
||||||
# --- FIX: Wir füllen das payload-Feld explizit ---
|
|
||||||
payload=payload,
|
|
||||||
explanation=explanation_obj
|
|
||||||
))
|
|
||||||
|
|
||||||
dt = int((time.time() - t0) * 1000)
|
|
||||||
return QueryResponse(results=results, used_mode=used_mode, latency_ms=dt)
|
|
||||||
|
|
||||||
|
|
||||||
def semantic_retrieve(req: QueryRequest) -> QueryResponse:
|
|
||||||
"""Reiner semantischer Retriever."""
|
|
||||||
client, prefix = _get_client_and_prefix()
|
|
||||||
vector = _get_query_vector(req)
|
|
||||||
top_k = req.top_k or get_settings().RETRIEVER_TOP_K
|
|
||||||
|
|
||||||
hits = _semantic_hits(client, prefix, vector, top_k=top_k, filters=req.filters)
|
|
||||||
return _build_hits_from_semantic(hits, top_k=top_k, used_mode="semantic", subgraph=None, explain=req.explain)
|
|
||||||
|
|
||||||
|
|
||||||
def hybrid_retrieve(req: QueryRequest) -> QueryResponse:
|
|
||||||
"""Hybrid-Retriever: semantische Suche + optionale Edge-Expansion."""
|
|
||||||
client, prefix = _get_client_and_prefix()
|
|
||||||
if req.query_vector:
|
|
||||||
vector = list(req.query_vector)
|
|
||||||
else:
|
|
||||||
vector = _get_query_vector(req)
|
|
||||||
|
|
||||||
top_k = req.top_k or get_settings().RETRIEVER_TOP_K
|
|
||||||
hits = _semantic_hits(client, prefix, vector, top_k=top_k, filters=req.filters)
|
|
||||||
|
|
||||||
depth, edge_types = _extract_expand_options(req)
|
|
||||||
subgraph: ga.Subgraph | None = None
|
|
||||||
if depth and depth > 0:
|
|
||||||
seed_ids: List[str] = []
|
|
||||||
for _pid, _score, payload in hits:
|
|
||||||
key = payload.get("chunk_id") or payload.get("note_id")
|
|
||||||
if key and key not in seed_ids:
|
|
||||||
seed_ids.append(key)
|
|
||||||
if seed_ids:
|
|
||||||
try:
|
|
||||||
subgraph = ga.expand(client, prefix, seed_ids, depth=depth, edge_types=edge_types)
|
|
||||||
except Exception:
|
|
||||||
subgraph = None
|
|
||||||
|
|
||||||
return _build_hits_from_semantic(hits, top_k=top_k, used_mode="hybrid", subgraph=subgraph, explain=req.explain)
|
|
||||||
|
|
||||||
|
|
||||||
class Retriever:
|
|
||||||
"""
|
|
||||||
Wrapper-Klasse für WP-05 (Chat).
|
|
||||||
"""
|
|
||||||
def __init__(self):
|
|
||||||
pass
|
|
||||||
|
|
||||||
async def search(self, request: QueryRequest) -> QueryResponse:
|
|
||||||
return hybrid_retrieve(request)
|
|
||||||
@@ -1,116 +0,0 @@
|
|||||||
"""app/core/retriever_config.py
|
|
||||||
---------------------------------
|
|
||||||
Zentrale Konfiguration für den mindnet-Retriever (WP-04).
|
|
||||||
|
|
||||||
Zweck:
|
|
||||||
- Lädt config/retriever.yaml (falls vorhanden) oder nutzt sinnvolle Defaults.
|
|
||||||
- Bietet einen gecachten Zugriff auf die Retriever-Config für
|
|
||||||
andere Module (z. B. graph_adapter, retriever).
|
|
||||||
|
|
||||||
Hinweis zur Weiterentwicklung (Selbstjustierung):
|
|
||||||
- Die hier definierten Parameter sind so gewählt, dass sie später
|
|
||||||
durch ein Feedback-/Learning-to-Rank-Modell überschrieben werden
|
|
||||||
können, ohne die restliche Architektur anzupassen.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from functools import lru_cache
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Dict
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml # type: ignore
|
|
||||||
except Exception: # pragma: no cover - Fallback, falls PyYAML nicht installiert ist.
|
|
||||||
yaml = None # type: ignore
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RetrieverConfig:
|
|
||||||
semantic_scale: float
|
|
||||||
edge_scale: float
|
|
||||||
centrality_scale: float
|
|
||||||
edge_weights: Dict[str, float]
|
|
||||||
|
|
||||||
@lru_cache
|
|
||||||
def get_retriever_config() -> RetrieverConfig:
|
|
||||||
"""Lädt die Retriever-Konfiguration (YAML + Defaults).
|
|
||||||
|
|
||||||
Reihenfolge:
|
|
||||||
1. Defaults (sinnvoll gewählte Startwerte).
|
|
||||||
2. Optional: config/retriever.yaml bzw. Pfad aus ENV
|
|
||||||
MINDNET_RETRIEVER_CONFIG überschreibt die Defaults.
|
|
||||||
|
|
||||||
Die Funktion ist bewusst gecached, da sich die Konfiguration zur
|
|
||||||
Laufzeit üblicherweise nicht ändert. Für dynamisches Nachladen
|
|
||||||
könnte der Cache explizit geleert werden.
|
|
||||||
"""
|
|
||||||
|
|
||||||
# 1) Defaults – bewusst konservativ gewählt.
|
|
||||||
semantic_scale = 1.0
|
|
||||||
edge_scale = 1.0
|
|
||||||
centrality_scale = 1.0
|
|
||||||
|
|
||||||
edge_weights: Dict[str, float] = {
|
|
||||||
# Wissens-Kanten
|
|
||||||
"depends_on": 1.0,
|
|
||||||
"related_to": 0.7,
|
|
||||||
"similar_to": 0.7,
|
|
||||||
"references": 0.5,
|
|
||||||
# Struktur-Kanten
|
|
||||||
"belongs_to": 0.2,
|
|
||||||
"next": 0.1,
|
|
||||||
"prev": 0.1,
|
|
||||||
# Sonstige / technische Kanten
|
|
||||||
"backlink": 0.2,
|
|
||||||
"references_at": 0.2,
|
|
||||||
}
|
|
||||||
|
|
||||||
# 2) Optional: YAML-Konfiguration laden
|
|
||||||
cfg_path_env = os.getenv("MINDNET_RETRIEVER_CONFIG")
|
|
||||||
if cfg_path_env:
|
|
||||||
cfg_path = Path(cfg_path_env)
|
|
||||||
else:
|
|
||||||
# Project-Root = zwei Ebenen über app/core/
|
|
||||||
cfg_path = Path(__file__).resolve().parents[2] / "config" / "retriever.yaml"
|
|
||||||
|
|
||||||
if yaml is not None and cfg_path.exists():
|
|
||||||
try:
|
|
||||||
data = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) or {}
|
|
||||||
except Exception:
|
|
||||||
data = {}
|
|
||||||
|
|
||||||
retr = data.get("retriever") or {}
|
|
||||||
|
|
||||||
# Skalenwerte überschreiben, falls angegeben
|
|
||||||
try:
|
|
||||||
semantic_scale = float(retr.get("semantic_scale", semantic_scale))
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
try:
|
|
||||||
edge_scale = float(retr.get("edge_scale", edge_scale))
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
try:
|
|
||||||
centrality_scale = float(retr.get("centrality_scale", centrality_scale))
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Edge-Gewichte je Kanten-Typ
|
|
||||||
ew_cfg = retr.get("edge_weights") or {}
|
|
||||||
if isinstance(ew_cfg, dict):
|
|
||||||
for k, v in ew_cfg.items():
|
|
||||||
try:
|
|
||||||
edge_weights[str(k)] = float(v)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
continue
|
|
||||||
|
|
||||||
return RetrieverConfig(
|
|
||||||
semantic_scale=semantic_scale,
|
|
||||||
edge_scale=edge_scale,
|
|
||||||
centrality_scale=centrality_scale,
|
|
||||||
edge_weights=edge_weights,
|
|
||||||
)
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
from functools import lru_cache
|
|
||||||
from jsonschema import Draft202012Validator, RefResolver
|
|
||||||
|
|
||||||
SCHEMAS_DIR = os.getenv("SCHEMAS_DIR", os.path.join(os.path.dirname(os.path.dirname(__file__)), "..", "schemas"))
|
|
||||||
|
|
||||||
@lru_cache(maxsize=16)
|
|
||||||
def load_schema(name: str) -> dict:
|
|
||||||
# name: "note.schema.json" | "chunk.schema.json" | "edge.schema.json"
|
|
||||||
path = os.path.join(SCHEMAS_DIR, name)
|
|
||||||
if not os.path.isfile(path):
|
|
||||||
raise FileNotFoundError(f"Schema not found: {path}")
|
|
||||||
with open(path, "r", encoding="utf-8") as f:
|
|
||||||
return json.load(f)
|
|
||||||
|
|
||||||
@lru_cache(maxsize=16)
|
|
||||||
def get_validator(name: str) -> Draft202012Validator:
|
|
||||||
schema = load_schema(name)
|
|
||||||
resolver = RefResolver.from_schema(schema)
|
|
||||||
return Draft202012Validator(schema, resolver=resolver)
|
|
||||||
@@ -1,30 +1,11 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
"""
|
||||||
Modul: app/core/type_registry.py
|
FILE: app/core/type_registry.py
|
||||||
Version: 1.0.0
|
DESCRIPTION: Loader für types.yaml. Achtung: Wird in der aktuellen Pipeline meist durch lokale Loader in 'ingestion.py' oder 'note_payload.py' umgangen.
|
||||||
Datum: 2025-11-08
|
VERSION: 1.0.0
|
||||||
|
STATUS: Deprecated (Redundant)
|
||||||
Zweck
|
DEPENDENCIES: yaml, os, functools
|
||||||
-----
|
EXTERNAL_CONFIG: config/types.yaml
|
||||||
Lädt eine optionale Typ-Registry (config/types.yaml) und stellt
|
LAST_ANALYSIS: 2025-12-15
|
||||||
komfortable Zugriffsfunktionen bereit. Die Registry ist *optional*:
|
|
||||||
- Fehlt die Datei oder ist das YAML defekt, wird ein konservativer
|
|
||||||
Default (Typ "concept") verwendet und es wird eine Warnung ausgegeben.
|
|
||||||
- Änderungen an der Datei greifen nach einem Neustart des Prozesses.
|
|
||||||
|
|
||||||
Öffentliche API
|
|
||||||
---------------
|
|
||||||
- load_type_registry(path: str = "config/types.yaml") -> dict
|
|
||||||
- get_type_config(note_type: str, reg: dict) -> dict
|
|
||||||
- resolve_note_type(fm_type: str | None, reg: dict) -> str
|
|
||||||
- effective_chunk_profile(note_type: str, reg: dict) -> str | None
|
|
||||||
- profile_overlap(profile: str | None) -> tuple[int,int] # nur Overlap-Empfehlung
|
|
||||||
|
|
||||||
Hinweis
|
|
||||||
-------
|
|
||||||
Die Registry steuert KEINE Breaking Changes. Ohne Datei/Typ bleibt das
|
|
||||||
Verhalten exakt wie im Release-Stand 20251105.
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
|||||||
@@ -1,16 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
from typing import Dict
|
|
||||||
from jsonschema import ValidationError
|
|
||||||
from .schema_loader import get_validator
|
|
||||||
|
|
||||||
NOTE_SCHEMA_NAME = "note.schema.json"
|
|
||||||
|
|
||||||
def validate_note_payload(payload: Dict) -> None:
|
|
||||||
validator = get_validator(NOTE_SCHEMA_NAME)
|
|
||||||
errors = sorted(validator.iter_errors(payload), key=lambda e: e.path)
|
|
||||||
if errors:
|
|
||||||
msgs = []
|
|
||||||
for e in errors:
|
|
||||||
loc = ".".join([str(x) for x in e.path]) or "<root>"
|
|
||||||
msgs.append(f"{loc}: {e.message}")
|
|
||||||
raise ValidationError(" | ".join(msgs))
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
"""
|
|
||||||
Version 1
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
from fastapi import FastAPI, HTTPException
|
|
||||||
from pydantic import BaseModel
|
|
||||||
from typing import List, Optional
|
|
||||||
from sentence_transformers import SentenceTransformer
|
|
||||||
|
|
||||||
app = FastAPI(title="mindnet-embed", version="1.0")
|
|
||||||
|
|
||||||
MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2" # 384-dim
|
|
||||||
_model: SentenceTransformer | None = None
|
|
||||||
|
|
||||||
class EmbedIn(BaseModel):
|
|
||||||
model: Optional[str] = None
|
|
||||||
inputs: List[str]
|
|
||||||
|
|
||||||
class EmbedOut(BaseModel):
|
|
||||||
embeddings: List[List[float]]
|
|
||||||
|
|
||||||
@app.on_event("startup")
|
|
||||||
def _load_model():
|
|
||||||
global _model
|
|
||||||
_model = SentenceTransformer(MODEL_NAME)
|
|
||||||
|
|
||||||
@app.get("/health")
|
|
||||||
def health():
|
|
||||||
return {"ok": True, "model": MODEL_NAME, "dim": 384}
|
|
||||||
|
|
||||||
@app.post("/embed", response_model=EmbedOut)
|
|
||||||
def embed(payload: EmbedIn) -> EmbedOut:
|
|
||||||
if _model is None:
|
|
||||||
raise HTTPException(status_code=503, detail="Model not loaded")
|
|
||||||
if not payload.inputs:
|
|
||||||
return EmbedOut(embeddings=[])
|
|
||||||
vecs = _model.encode(payload.inputs, normalize_embeddings=False).tolist()
|
|
||||||
if any(len(v) != 384 for v in vecs):
|
|
||||||
raise HTTPException(status_code=500, detail="Embedding size mismatch (expected 384)")
|
|
||||||
return EmbedOut(embeddings=vecs)
|
|
||||||
+6
-2
@@ -1,6 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
Version 0.1
|
FILE: app/embeddings.py
|
||||||
|
DESCRIPTION: Lokaler Wrapper für SentenceTransformer Embeddings.
|
||||||
|
VERSION: 0.1.0
|
||||||
|
STATUS: Active (Bestätigung durch Aufrufer erforderlich)
|
||||||
|
DEPENDENCIES: app.config, sentence_transformers
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui.py
|
||||||
|
DESCRIPTION: Main Entrypoint für Streamlit. Router, der basierend auf Sidebar-Auswahl die Module (Chat, Editor, Graph) lädt.
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: streamlit, ui_config, ui_sidebar, ui_chat, ui_editor, ui_graph_service, ui_graph*, ui_graph_cytoscape
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_api.py
|
||||||
|
DESCRIPTION: Wrapper für Backend-Calls (Chat, Ingest, Feedback). Kapselt requests und Error-Handling.
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: requests, streamlit, ui_config
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
from ui_config import CHAT_ENDPOINT, INGEST_ANALYZE_ENDPOINT, INGEST_SAVE_ENDPOINT, FEEDBACK_ENDPOINT, API_TIMEOUT
|
from ui_config import CHAT_ENDPOINT, INGEST_ANALYZE_ENDPOINT, INGEST_SAVE_ENDPOINT, FEEDBACK_ENDPOINT, API_TIMEOUT
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_callbacks.py
|
||||||
|
DESCRIPTION: Event-Handler für UI-Interaktionen. Implementiert den Übergang vom Graphen zum Editor (State Transfer).
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: streamlit, os, ui_utils
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
import os
|
import os
|
||||||
from ui_utils import build_markdown_doc
|
from ui_utils import build_markdown_doc
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_chat.py
|
||||||
|
DESCRIPTION: Chat-UI. Rendert Nachrichtenverlauf, Quellen-Expanders mit Feedback-Buttons und delegiert bei Bedarf an den Editor.
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: streamlit, ui_api, ui_editor
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
from ui_api import send_chat_message, submit_feedback
|
from ui_api import send_chat_message, submit_feedback
|
||||||
from ui_editor import render_draft_editor
|
from ui_editor import render_draft_editor
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_config.py
|
||||||
|
DESCRIPTION: Zentrale Konfiguration für das Frontend. Definiert API-Endpoints, Timeouts und Graph-Styles (Farben).
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: os, hashlib, dotenv, pathlib
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import hashlib
|
import hashlib
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
|
|||||||
+25
-15
@@ -1,3 +1,11 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_editor.py
|
||||||
|
DESCRIPTION: Markdown-Editor mit Live-Vorschau.
|
||||||
|
Refactored für WP-14: Asynchrones Feedback-Handling (Queued State).
|
||||||
|
VERSION: 2.7.0 (Fix: Async Save UI)
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: streamlit, uuid, re, datetime, ui_utils, ui_api
|
||||||
|
"""
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
import uuid
|
import uuid
|
||||||
import re
|
import re
|
||||||
@@ -68,14 +76,11 @@ def render_draft_editor(msg):
|
|||||||
|
|
||||||
# --- UI LAYOUT ---
|
# --- UI LAYOUT ---
|
||||||
|
|
||||||
# Header Info (Debug Pfad anzeigen, damit wir sicher sind)
|
|
||||||
origin_fname = st.session_state.get(f"{key_base}_origin_filename")
|
origin_fname = st.session_state.get(f"{key_base}_origin_filename")
|
||||||
|
|
||||||
if origin_fname:
|
if origin_fname:
|
||||||
# Dateiname extrahieren für saubere Anzeige
|
|
||||||
display_name = str(origin_fname).split("/")[-1]
|
display_name = str(origin_fname).split("/")[-1]
|
||||||
st.success(f"📂 **Update-Modus**: `{display_name}`")
|
st.success(f"📂 **Update-Modus**: `{display_name}`")
|
||||||
# Debugging: Zeige vollen Pfad im Expander
|
|
||||||
with st.expander("Dateipfad Details", expanded=False):
|
with st.expander("Dateipfad Details", expanded=False):
|
||||||
st.code(origin_fname)
|
st.code(origin_fname)
|
||||||
st.markdown(f'<div class="draft-box" style="border-left: 5px solid #ff9f43;">', unsafe_allow_html=True)
|
st.markdown(f'<div class="draft-box" style="border-left: 5px solid #ff9f43;">', unsafe_allow_html=True)
|
||||||
@@ -165,21 +170,33 @@ def render_draft_editor(msg):
|
|||||||
save_label = "💾 Update speichern" if origin_fname else "💾 Neu anlegen & Indizieren"
|
save_label = "💾 Update speichern" if origin_fname else "💾 Neu anlegen & Indizieren"
|
||||||
|
|
||||||
if st.button(save_label, type="primary", key=f"{key_base}_save"):
|
if st.button(save_label, type="primary", key=f"{key_base}_save"):
|
||||||
with st.spinner("Speichere im Vault..."):
|
with st.spinner("Sende an Backend..."):
|
||||||
if origin_fname:
|
if origin_fname:
|
||||||
# UPDATE: Ziel ist der exakte Pfad
|
|
||||||
target_file = origin_fname
|
target_file = origin_fname
|
||||||
else:
|
else:
|
||||||
# CREATE: Neuer Dateiname
|
|
||||||
raw_title = final_meta.get("title", "draft")
|
raw_title = final_meta.get("title", "draft")
|
||||||
target_file = f"{datetime.now().strftime('%Y%m%d')}-{slugify(raw_title)[:60]}.md"
|
target_file = f"{datetime.now().strftime('%Y%m%d')}-{slugify(raw_title)[:60]}.md"
|
||||||
|
|
||||||
result = save_draft_to_vault(final_doc, filename=target_file)
|
result = save_draft_to_vault(final_doc, filename=target_file)
|
||||||
|
|
||||||
|
# --- WP-14 CHANGE START: Handling Async Response ---
|
||||||
if "error" in result:
|
if "error" in result:
|
||||||
st.error(f"Fehler: {result['error']}")
|
st.error(f"Fehler: {result['error']}")
|
||||||
else:
|
else:
|
||||||
st.success(f"Gespeichert: {result.get('file_path')}")
|
status = result.get("status", "success")
|
||||||
|
file_path = result.get("file_path", "unbekannt")
|
||||||
|
|
||||||
|
if status == "queued":
|
||||||
|
# Neuer Status für Async Processing
|
||||||
|
st.info(f"✅ **Eingereiht:** Datei `{file_path}` wurde gespeichert.")
|
||||||
|
st.caption("Die KI-Analyse und Indizierung läuft im Hintergrund. Du kannst weiterarbeiten.")
|
||||||
|
else:
|
||||||
|
# Legacy / Synchroner Fall
|
||||||
|
st.success(f"Gespeichert: {file_path}")
|
||||||
|
|
||||||
st.balloons()
|
st.balloons()
|
||||||
|
# --- WP-14 CHANGE END ---
|
||||||
|
|
||||||
with b2:
|
with b2:
|
||||||
if st.button("📋 Code anzeigen", key=f"{key_base}_btn_copy"):
|
if st.button("📋 Code anzeigen", key=f"{key_base}_btn_copy"):
|
||||||
st.code(final_doc, language="markdown")
|
st.code(final_doc, language="markdown")
|
||||||
@@ -189,25 +206,18 @@ def render_draft_editor(msg):
|
|||||||
def render_manual_editor():
|
def render_manual_editor():
|
||||||
"""
|
"""
|
||||||
Rendert den manuellen Editor.
|
Rendert den manuellen Editor.
|
||||||
PRÜFT, ob eine Edit-Anfrage aus dem Graphen vorliegt!
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
target_msg = None
|
target_msg = None
|
||||||
|
|
||||||
# 1. Prüfen: Gibt es Nachrichten im Verlauf?
|
|
||||||
if st.session_state.messages:
|
if st.session_state.messages:
|
||||||
last_msg = st.session_state.messages[-1]
|
last_msg = st.session_state.messages[-1]
|
||||||
|
|
||||||
# 2. Ist die letzte Nachricht eine Edit-Anfrage? (Erkennbar am query_id prefix 'edit_')
|
|
||||||
qid = str(last_msg.get("query_id", ""))
|
qid = str(last_msg.get("query_id", ""))
|
||||||
if qid.startswith("edit_"):
|
if qid.startswith("edit_"):
|
||||||
target_msg = last_msg
|
target_msg = last_msg
|
||||||
|
|
||||||
# 3. Fallback: Leeres Template, falls keine Edit-Anfrage vorliegt
|
|
||||||
if not target_msg:
|
if not target_msg:
|
||||||
target_msg = {
|
target_msg = {
|
||||||
"content": "---\ntype: concept\ntitle: Neue Notiz\nstatus: draft\ntags: []\n---\n# Titel\n",
|
"content": "---\ntype: concept\ntitle: Neue Notiz\nstatus: draft\ntags: []\n---\n# Titel\n",
|
||||||
"query_id": f"manual_{uuid.uuid4()}" # Eigene ID, damit neuer State entsteht
|
"query_id": f"manual_{uuid.uuid4()}"
|
||||||
}
|
}
|
||||||
|
|
||||||
render_draft_editor(target_msg)
|
render_draft_editor(target_msg)
|
||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_graph.py
|
||||||
|
DESCRIPTION: Legacy Graph-Explorer (Streamlit-Agraph). Implementiert Physik-Simulation (BarnesHut) und direkten Editor-Sprung.
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Maintenance (Active Fallback)
|
||||||
|
DEPENDENCIES: streamlit, streamlit_agraph, qdrant_client, ui_config, ui_callbacks
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
from streamlit_agraph import agraph, Config
|
from streamlit_agraph import agraph, Config
|
||||||
from qdrant_client import models
|
from qdrant_client import models
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_graph_cytoscape.py
|
||||||
|
DESCRIPTION: Moderner Graph-Explorer (Cytoscape.js). Features: COSE-Layout, Deep-Linking (URL Params), Active Inspector Pattern (CSS-Styling ohne Re-Render).
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: streamlit, st_cytoscape, qdrant_client, ui_config, ui_callbacks
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
from st_cytoscape import cytoscape
|
from st_cytoscape import cytoscape
|
||||||
from qdrant_client import models
|
from qdrant_client import models
|
||||||
@@ -60,20 +69,35 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
search_term = st.text_input("Suche Notiz", placeholder="Titel eingeben...", key="cy_search")
|
search_term = st.text_input("Suche Notiz", placeholder="Titel eingeben...", key="cy_search")
|
||||||
|
|
||||||
if search_term:
|
if search_term:
|
||||||
hits, _ = graph_service.client.scroll(
|
try:
|
||||||
collection_name=f"{COLLECTION_PREFIX}_notes",
|
hits, _ = graph_service.client.scroll(
|
||||||
limit=10,
|
collection_name=f"{COLLECTION_PREFIX}_notes",
|
||||||
scroll_filter=models.Filter(must=[models.FieldCondition(key="title", match=models.MatchText(text=search_term))])
|
limit=10,
|
||||||
)
|
scroll_filter=models.Filter(must=[models.FieldCondition(key="title", match=models.MatchText(text=search_term))])
|
||||||
options = {h.payload['title']: h.payload['note_id'] for h in hits}
|
)
|
||||||
|
options = {}
|
||||||
if options:
|
for h in hits:
|
||||||
selected_title = st.selectbox("Ergebnisse:", list(options.keys()), key="cy_select")
|
if h.payload and 'title' in h.payload and 'note_id' in h.payload:
|
||||||
if st.button("Laden", use_container_width=True, key="cy_load"):
|
title = h.payload['title']
|
||||||
new_id = options[selected_title]
|
note_id = h.payload['note_id']
|
||||||
st.session_state.graph_center_id = new_id
|
# Vermeide Duplikate (falls mehrere Chunks/Notes denselben Titel haben)
|
||||||
st.session_state.graph_inspected_id = new_id
|
if title not in options:
|
||||||
st.rerun()
|
options[title] = note_id
|
||||||
|
|
||||||
|
if options:
|
||||||
|
selected_title = st.selectbox("Ergebnisse:", list(options.keys()), key="cy_select")
|
||||||
|
if st.button("Laden", use_container_width=True, key="cy_load"):
|
||||||
|
new_id = options[selected_title]
|
||||||
|
st.session_state.graph_center_id = new_id
|
||||||
|
st.session_state.graph_inspected_id = new_id
|
||||||
|
st.rerun()
|
||||||
|
else:
|
||||||
|
# Zeige Info, wenn keine Ergebnisse gefunden wurden
|
||||||
|
st.info(f"Keine Notizen mit '{search_term}' im Titel gefunden.")
|
||||||
|
except Exception as e:
|
||||||
|
st.error(f"Fehler bei der Suche: {e}")
|
||||||
|
import traceback
|
||||||
|
st.code(traceback.format_exc())
|
||||||
|
|
||||||
st.divider()
|
st.divider()
|
||||||
|
|
||||||
@@ -129,6 +153,26 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
)
|
)
|
||||||
# 2. Detail Daten (Inspector)
|
# 2. Detail Daten (Inspector)
|
||||||
inspected_data = graph_service.get_note_with_full_content(inspected_id)
|
inspected_data = graph_service.get_note_with_full_content(inspected_id)
|
||||||
|
|
||||||
|
# DEBUG: Zeige Debug-Informationen
|
||||||
|
with st.expander("🔍 Debug-Informationen", expanded=False):
|
||||||
|
st.write(f"**Gefundene Knoten:** {len(nodes_data) if nodes_data else 0}")
|
||||||
|
st.write(f"**Gefundene Kanten:** {len(edges_data) if edges_data else 0}")
|
||||||
|
if nodes_data:
|
||||||
|
st.write("**Knoten-IDs:**")
|
||||||
|
for n in nodes_data[:10]:
|
||||||
|
nid = getattr(n, 'id', 'N/A')
|
||||||
|
st.write(f" - {nid}")
|
||||||
|
if len(nodes_data) > 10:
|
||||||
|
st.write(f" ... und {len(nodes_data) - 10} weitere")
|
||||||
|
if edges_data:
|
||||||
|
st.write("**Kanten:**")
|
||||||
|
for e in edges_data[:10]:
|
||||||
|
src = getattr(e, "source", "N/A")
|
||||||
|
tgt = getattr(e, "to", getattr(e, "target", "N/A"))
|
||||||
|
st.write(f" - {src} -> {tgt}")
|
||||||
|
if len(edges_data) > 10:
|
||||||
|
st.write(f" ... und {len(edges_data) - 10} weitere")
|
||||||
|
|
||||||
# --- ACTION BAR ---
|
# --- ACTION BAR ---
|
||||||
action_container = st.container()
|
action_container = st.container()
|
||||||
@@ -165,7 +209,12 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
st.markdown(f"**ID:** `{inspected_data.get('note_id')}`")
|
st.markdown(f"**ID:** `{inspected_data.get('note_id')}`")
|
||||||
st.markdown(f"**Typ:** `{inspected_data.get('type')}`")
|
st.markdown(f"**Typ:** `{inspected_data.get('type')}`")
|
||||||
with col_i2:
|
with col_i2:
|
||||||
st.markdown(f"**Tags:** {', '.join(inspected_data.get('tags', []))}")
|
tags = inspected_data.get('tags', [])
|
||||||
|
if isinstance(tags, list):
|
||||||
|
tags_str = ', '.join(tags) if tags else "Keine"
|
||||||
|
else:
|
||||||
|
tags_str = str(tags) if tags else "Keine"
|
||||||
|
st.markdown(f"**Tags:** {tags_str}")
|
||||||
path_check = "✅" if inspected_data.get('path') else "❌"
|
path_check = "✅" if inspected_data.get('path') else "❌"
|
||||||
st.markdown(f"**Pfad:** {path_check}")
|
st.markdown(f"**Pfad:** {path_check}")
|
||||||
|
|
||||||
@@ -180,12 +229,27 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
# --- GRAPH ELEMENTS ---
|
# --- GRAPH ELEMENTS ---
|
||||||
cy_elements = []
|
cy_elements = []
|
||||||
|
|
||||||
|
# Validierung: Prüfe ob nodes_data vorhanden ist
|
||||||
|
if not nodes_data:
|
||||||
|
st.warning("⚠️ Keine Knoten gefunden. Bitte wähle eine andere Notiz.")
|
||||||
|
# Zeige trotzdem den Inspector, falls Daten vorhanden
|
||||||
|
if inspected_data:
|
||||||
|
st.info(f"**Hinweis:** Die Notiz '{inspected_data.get('title', inspected_id)}' wurde gefunden, hat aber keine Verbindungen im Graphen.")
|
||||||
|
return
|
||||||
|
|
||||||
|
# Erstelle Set aller Node-IDs für schnelle Validierung
|
||||||
|
node_ids = {n.id for n in nodes_data if hasattr(n, 'id') and n.id}
|
||||||
|
|
||||||
|
# Nodes hinzufügen
|
||||||
for n in nodes_data:
|
for n in nodes_data:
|
||||||
|
if not hasattr(n, 'id') or not n.id:
|
||||||
|
continue
|
||||||
|
|
||||||
is_center = (n.id == center_id)
|
is_center = (n.id == center_id)
|
||||||
is_inspected = (n.id == inspected_id)
|
is_inspected = (n.id == inspected_id)
|
||||||
|
|
||||||
tooltip_text = n.title if n.title else n.label
|
tooltip_text = getattr(n, 'title', None) or getattr(n, 'label', '')
|
||||||
display_label = n.label
|
display_label = getattr(n, 'label', str(n.id))
|
||||||
if len(display_label) > 15 and " " in display_label:
|
if len(display_label) > 15 and " " in display_label:
|
||||||
display_label = display_label.replace(" ", "\n", 1)
|
display_label = display_label.replace(" ", "\n", 1)
|
||||||
|
|
||||||
@@ -193,7 +257,7 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
"data": {
|
"data": {
|
||||||
"id": n.id,
|
"id": n.id,
|
||||||
"label": display_label,
|
"label": display_label,
|
||||||
"bg_color": n.color,
|
"bg_color": getattr(n, 'color', '#8395a7'),
|
||||||
"tooltip": tooltip_text
|
"tooltip": tooltip_text
|
||||||
},
|
},
|
||||||
# Wir steuern das Aussehen rein über Klassen (.inspected / .center)
|
# Wir steuern das Aussehen rein über Klassen (.inspected / .center)
|
||||||
@@ -202,18 +266,22 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
}
|
}
|
||||||
cy_elements.append(cy_node)
|
cy_elements.append(cy_node)
|
||||||
|
|
||||||
for e in edges_data:
|
# Edges hinzufügen - nur wenn beide Nodes im Graph vorhanden sind
|
||||||
target_id = getattr(e, "to", getattr(e, "target", None))
|
if edges_data:
|
||||||
if target_id:
|
for e in edges_data:
|
||||||
cy_edge = {
|
source_id = getattr(e, "source", None)
|
||||||
"data": {
|
target_id = getattr(e, "to", getattr(e, "target", None))
|
||||||
"source": e.source,
|
# Nur hinzufügen, wenn beide IDs vorhanden UND beide Nodes im Graph sind
|
||||||
"target": target_id,
|
if source_id and target_id and source_id in node_ids and target_id in node_ids:
|
||||||
"label": e.label,
|
cy_edge = {
|
||||||
"line_color": e.color
|
"data": {
|
||||||
|
"source": source_id,
|
||||||
|
"target": target_id,
|
||||||
|
"label": getattr(e, "label", ""),
|
||||||
|
"line_color": getattr(e, "color", "#bdc3c7")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
cy_elements.append(cy_edge)
|
||||||
cy_elements.append(cy_edge)
|
|
||||||
|
|
||||||
# --- STYLESHEET ---
|
# --- STYLESHEET ---
|
||||||
stylesheet = [
|
stylesheet = [
|
||||||
@@ -283,43 +351,47 @@ def render_graph_explorer_cytoscape(graph_service):
|
|||||||
]
|
]
|
||||||
|
|
||||||
# --- RENDER ---
|
# --- RENDER ---
|
||||||
graph_key = f"cy_{center_id}_{st.session_state.cy_depth}_{st.session_state.cy_ideal_edge_len}"
|
# Nur rendern, wenn Elemente vorhanden sind
|
||||||
|
if not cy_elements:
|
||||||
|
st.warning("⚠️ Keine Graph-Elemente zum Anzeigen gefunden.")
|
||||||
|
else:
|
||||||
|
graph_key = f"cy_{center_id}_{st.session_state.cy_depth}_{st.session_state.cy_ideal_edge_len}"
|
||||||
|
|
||||||
clicked_elements = cytoscape(
|
clicked_elements = cytoscape(
|
||||||
elements=cy_elements,
|
elements=cy_elements,
|
||||||
stylesheet=stylesheet,
|
stylesheet=stylesheet,
|
||||||
layout={
|
layout={
|
||||||
"name": "cose",
|
"name": "cose",
|
||||||
"idealEdgeLength": st.session_state.cy_ideal_edge_len,
|
"idealEdgeLength": st.session_state.cy_ideal_edge_len,
|
||||||
"nodeOverlap": 20,
|
"nodeOverlap": 20,
|
||||||
"refresh": 20,
|
"refresh": 20,
|
||||||
"fit": True,
|
"fit": True,
|
||||||
"padding": 50,
|
"padding": 50,
|
||||||
"randomize": False,
|
"randomize": False,
|
||||||
"componentSpacing": 100,
|
"componentSpacing": 100,
|
||||||
"nodeRepulsion": st.session_state.cy_node_repulsion,
|
"nodeRepulsion": st.session_state.cy_node_repulsion,
|
||||||
"edgeElasticity": 100,
|
"edgeElasticity": 100,
|
||||||
"nestingFactor": 5,
|
"nestingFactor": 5,
|
||||||
"gravity": 80,
|
"gravity": 80,
|
||||||
"numIter": 1000,
|
"numIter": 1000,
|
||||||
"initialTemp": 200,
|
"initialTemp": 200,
|
||||||
"coolingFactor": 0.95,
|
"coolingFactor": 0.95,
|
||||||
"minTemp": 1.0,
|
"minTemp": 1.0,
|
||||||
"animate": False
|
"animate": False
|
||||||
},
|
},
|
||||||
key=graph_key,
|
key=graph_key,
|
||||||
height="700px"
|
height="700px"
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- EVENT HANDLING ---
|
# --- EVENT HANDLING ---
|
||||||
if clicked_elements:
|
if clicked_elements:
|
||||||
clicked_nodes = clicked_elements.get("nodes", [])
|
clicked_nodes = clicked_elements.get("nodes", [])
|
||||||
if clicked_nodes:
|
if clicked_nodes:
|
||||||
clicked_id = clicked_nodes[0]
|
clicked_id = clicked_nodes[0]
|
||||||
|
|
||||||
if clicked_id != st.session_state.graph_inspected_id:
|
if clicked_id != st.session_state.graph_inspected_id:
|
||||||
st.session_state.graph_inspected_id = clicked_id
|
st.session_state.graph_inspected_id = clicked_id
|
||||||
st.rerun()
|
st.rerun()
|
||||||
|
|
||||||
else:
|
else:
|
||||||
st.info("👈 Bitte wähle links eine Notiz aus.")
|
st.info("👈 Bitte wähle links eine Notiz aus.")
|
||||||
@@ -1,15 +1,28 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_graph_service.py
|
||||||
|
DESCRIPTION: Data Layer für den Graphen. Greift direkt auf Qdrant zu (Performance), um Knoten/Kanten zu laden und Texte zu rekonstruieren ("Stitching").
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: qdrant_client, streamlit_agraph, ui_config, re
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import re
|
import re
|
||||||
from qdrant_client import QdrantClient, models
|
from qdrant_client import QdrantClient, models
|
||||||
from streamlit_agraph import Node, Edge
|
from streamlit_agraph import Node, Edge
|
||||||
from ui_config import GRAPH_COLORS, get_edge_color, SYSTEM_EDGES
|
from ui_config import COLLECTION_PREFIX, GRAPH_COLORS, get_edge_color, SYSTEM_EDGES
|
||||||
|
|
||||||
class GraphExplorerService:
|
class GraphExplorerService:
|
||||||
def __init__(self, url, api_key=None, prefix="mindnet"):
|
def __init__(self, url, api_key=None, prefix=None):
|
||||||
|
"""
|
||||||
|
Initialisiert den Service. Nutzt COLLECTION_PREFIX aus der Config,
|
||||||
|
sofern kein spezifischer Prefix übergeben wurde.
|
||||||
|
"""
|
||||||
self.client = QdrantClient(url=url, api_key=api_key)
|
self.client = QdrantClient(url=url, api_key=api_key)
|
||||||
self.prefix = prefix
|
self.prefix = prefix if prefix else COLLECTION_PREFIX
|
||||||
self.notes_col = f"{prefix}_notes"
|
self.notes_col = f"{self.prefix}_notes"
|
||||||
self.chunks_col = f"{prefix}_chunks"
|
self.chunks_col = f"{self.prefix}_chunks"
|
||||||
self.edges_col = f"{prefix}_edges"
|
self.edges_col = f"{self.prefix}_edges"
|
||||||
self._note_cache = {}
|
self._note_cache = {}
|
||||||
|
|
||||||
def get_note_with_full_content(self, note_id):
|
def get_note_with_full_content(self, note_id):
|
||||||
@@ -154,31 +167,33 @@ class GraphExplorerService:
|
|||||||
return previews
|
return previews
|
||||||
|
|
||||||
def _find_connected_edges(self, note_ids, note_title=None):
|
def _find_connected_edges(self, note_ids, note_title=None):
|
||||||
"""Findet eingehende und ausgehende Kanten."""
|
"""
|
||||||
|
Findet eingehende und ausgehende Kanten.
|
||||||
|
|
||||||
|
WICHTIG: target_id enthält nur den Titel (ohne #Abschnitt).
|
||||||
|
target_section ist ein separates Feld für Abschnitt-Informationen.
|
||||||
|
"""
|
||||||
results = []
|
results = []
|
||||||
|
if not note_ids:
|
||||||
|
return results
|
||||||
|
|
||||||
# 1. OUTGOING EDGES (Der "Owner"-Fix)
|
# 1. OUTGOING EDGES (Der "Owner"-Fix)
|
||||||
# Wir suchen Kanten, die im Feld 'note_id' (Owner) eine unserer Notizen haben.
|
# Wir suchen Kanten, die im Feld 'note_id' (Owner) eine unserer Notizen haben.
|
||||||
# Das findet ALLE ausgehenden Kanten, egal ob sie an einem Chunk oder der Note hängen.
|
# Das findet ALLE ausgehenden Kanten, egal ob sie an einem Chunk oder der Note hängen.
|
||||||
if note_ids:
|
out_filter = models.Filter(must=[
|
||||||
out_filter = models.Filter(must=[
|
models.FieldCondition(key="note_id", match=models.MatchAny(any=note_ids)),
|
||||||
models.FieldCondition(key="note_id", match=models.MatchAny(any=note_ids)),
|
models.FieldCondition(key="kind", match=models.MatchExcept(**{"except": SYSTEM_EDGES}))
|
||||||
models.FieldCondition(key="kind", match=models.MatchExcept(**{"except": SYSTEM_EDGES}))
|
])
|
||||||
])
|
res_out, _ = self.client.scroll(self.edges_col, scroll_filter=out_filter, limit=2000, with_payload=True)
|
||||||
# Limit hoch, um alles zu finden
|
results.extend(res_out)
|
||||||
res_out, _ = self.client.scroll(self.edges_col, scroll_filter=out_filter, limit=500, with_payload=True)
|
|
||||||
results.extend(res_out)
|
|
||||||
|
|
||||||
# 2. INCOMING EDGES (Ziel = Chunk ID oder Titel oder Note ID)
|
# 2. INCOMING EDGES (Ziel = Chunk ID, Note ID oder Titel)
|
||||||
# Hier müssen wir Chunks auflösen, um Treffer auf Chunks zu finden.
|
# WICHTIG: target_id enthält nur den Titel, target_section ist separat
|
||||||
|
|
||||||
# Chunk IDs der aktuellen Notes holen
|
# Chunk IDs der aktuellen Notes holen
|
||||||
chunk_ids = []
|
c_filter = models.Filter(must=[models.FieldCondition(key="note_id", match=models.MatchAny(any=note_ids))])
|
||||||
if note_ids:
|
chunks, _ = self.client.scroll(self.chunks_col, scroll_filter=c_filter, limit=1000, with_payload=False)
|
||||||
c_filter = models.Filter(must=[models.FieldCondition(key="note_id", match=models.MatchAny(any=note_ids))])
|
chunk_ids = [c.id for c in chunks]
|
||||||
chunks, _ = self.client.scroll(self.chunks_col, scroll_filter=c_filter, limit=300)
|
|
||||||
chunk_ids = [c.id for c in chunks]
|
|
||||||
|
|
||||||
shoulds = []
|
shoulds = []
|
||||||
# Case A: Edge zeigt auf einen unserer Chunks
|
# Case A: Edge zeigt auf einen unserer Chunks
|
||||||
@@ -186,42 +201,92 @@ class GraphExplorerService:
|
|||||||
shoulds.append(models.FieldCondition(key="target_id", match=models.MatchAny(any=chunk_ids)))
|
shoulds.append(models.FieldCondition(key="target_id", match=models.MatchAny(any=chunk_ids)))
|
||||||
|
|
||||||
# Case B: Edge zeigt direkt auf unsere Note ID
|
# Case B: Edge zeigt direkt auf unsere Note ID
|
||||||
if note_ids:
|
shoulds.append(models.FieldCondition(key="target_id", match=models.MatchAny(any=note_ids)))
|
||||||
shoulds.append(models.FieldCondition(key="target_id", match=models.MatchAny(any=note_ids)))
|
|
||||||
|
# Case C: Edge zeigt auf unseren Titel
|
||||||
# Case C: Edge zeigt auf unseren Titel (Wikilinks)
|
# WICHTIG: target_id enthält nur den Titel (z.B. "Meine Prinzipien 2025")
|
||||||
if note_title:
|
# target_section enthält die Abschnitt-Information (z.B. "P3 – Disziplin"), wenn gesetzt
|
||||||
shoulds.append(models.FieldCondition(key="target_id", match=models.MatchValue(value=note_title)))
|
|
||||||
|
# Sammle alle relevanten Titel (inkl. Aliase)
|
||||||
|
titles_to_search = []
|
||||||
|
if note_title:
|
||||||
|
titles_to_search.append(note_title)
|
||||||
|
|
||||||
|
# Lade auch Titel aus den Notes selbst (falls note_title nicht übergeben wurde)
|
||||||
|
for nid in note_ids:
|
||||||
|
note = self._fetch_note_cached(nid)
|
||||||
|
if note:
|
||||||
|
note_title_from_db = note.get("title")
|
||||||
|
if note_title_from_db and note_title_from_db not in titles_to_search:
|
||||||
|
titles_to_search.append(note_title_from_db)
|
||||||
|
# Aliase hinzufügen
|
||||||
|
aliases = note.get("aliases", [])
|
||||||
|
if isinstance(aliases, str):
|
||||||
|
aliases = [aliases]
|
||||||
|
for alias in aliases:
|
||||||
|
if alias and alias not in titles_to_search:
|
||||||
|
titles_to_search.append(alias)
|
||||||
|
|
||||||
|
# Für jeden Titel: Suche nach exaktem Match
|
||||||
|
# target_id enthält nur den Titel, daher reicht MatchValue
|
||||||
|
for title in titles_to_search:
|
||||||
|
shoulds.append(models.FieldCondition(key="target_id", match=models.MatchValue(value=title)))
|
||||||
|
|
||||||
if shoulds:
|
if shoulds:
|
||||||
in_filter = models.Filter(
|
in_filter = models.Filter(
|
||||||
must=[models.FieldCondition(key="kind", match=models.MatchExcept(**{"except": SYSTEM_EDGES}))],
|
must=[models.FieldCondition(key="kind", match=models.MatchExcept(**{"except": SYSTEM_EDGES}))],
|
||||||
should=shoulds
|
should=shoulds
|
||||||
)
|
)
|
||||||
res_in, _ = self.client.scroll(self.edges_col, scroll_filter=in_filter, limit=500, with_payload=True)
|
res_in, _ = self.client.scroll(self.edges_col, scroll_filter=in_filter, limit=2000, with_payload=True)
|
||||||
results.extend(res_in)
|
results.extend(res_in)
|
||||||
|
|
||||||
return results
|
return results
|
||||||
|
|
||||||
def _find_connected_edges_batch(self, note_ids):
|
def _find_connected_edges_batch(self, note_ids):
|
||||||
# Wrapper für Level 2 Suche
|
"""
|
||||||
return self._find_connected_edges(note_ids)
|
Wrapper für Level 2 Suche.
|
||||||
|
Lädt Titel der ersten Note für Titel-basierte Suche.
|
||||||
|
"""
|
||||||
|
if not note_ids:
|
||||||
|
return []
|
||||||
|
first_note = self._fetch_note_cached(note_ids[0])
|
||||||
|
note_title = first_note.get("title") if first_note else None
|
||||||
|
return self._find_connected_edges(note_ids, note_title=note_title)
|
||||||
|
|
||||||
def _process_edge(self, record, nodes_dict, unique_edges, current_depth):
|
def _process_edge(self, record, nodes_dict, unique_edges, current_depth):
|
||||||
"""Verarbeitet eine rohe Edge, löst IDs auf und fügt sie den Dictionaries hinzu."""
|
"""
|
||||||
|
Verarbeitet eine rohe Edge, löst IDs auf und fügt sie den Dictionaries hinzu.
|
||||||
|
|
||||||
|
WICHTIG: Beide Richtungen werden unterstützt:
|
||||||
|
- Ausgehende Kanten: source_id gehört zu unserer Note (via note_id Owner)
|
||||||
|
- Eingehende Kanten: target_id zeigt auf unsere Note (via target_id Match)
|
||||||
|
"""
|
||||||
|
if not record or not record.payload:
|
||||||
|
return None, None
|
||||||
|
|
||||||
payload = record.payload
|
payload = record.payload
|
||||||
src_ref = payload.get("source_id")
|
src_ref = payload.get("source_id")
|
||||||
tgt_ref = payload.get("target_id")
|
tgt_ref = payload.get("target_id")
|
||||||
kind = payload.get("kind")
|
kind = payload.get("kind")
|
||||||
provenance = payload.get("provenance", "explicit")
|
provenance = payload.get("provenance", "explicit")
|
||||||
|
|
||||||
|
# Prüfe, ob beide Referenzen vorhanden sind
|
||||||
|
if not src_ref or not tgt_ref:
|
||||||
|
return None, None
|
||||||
|
|
||||||
# IDs zu Notes auflösen
|
# IDs zu Notes auflösen
|
||||||
|
# WICHTIG: source_id kann Chunk-ID (note_id#c01), Note-ID oder Titel sein
|
||||||
|
# WICHTIG: target_id kann Chunk-ID, Note-ID oder Titel sein (ohne #Abschnitt)
|
||||||
src_note = self._resolve_note_from_ref(src_ref)
|
src_note = self._resolve_note_from_ref(src_ref)
|
||||||
tgt_note = self._resolve_note_from_ref(tgt_ref)
|
tgt_note = self._resolve_note_from_ref(tgt_ref)
|
||||||
|
|
||||||
if src_note and tgt_note:
|
if src_note and tgt_note:
|
||||||
src_id = src_note['note_id']
|
src_id = src_note.get('note_id')
|
||||||
tgt_id = tgt_note['note_id']
|
tgt_id = tgt_note.get('note_id')
|
||||||
|
|
||||||
|
# Prüfe, ob beide IDs vorhanden sind
|
||||||
|
if not src_id or not tgt_id:
|
||||||
|
return None, None
|
||||||
|
|
||||||
if src_id != tgt_id:
|
if src_id != tgt_id:
|
||||||
# Nodes hinzufügen
|
# Nodes hinzufügen
|
||||||
@@ -236,7 +301,7 @@ class GraphExplorerService:
|
|||||||
# Bevorzuge explizite Kanten vor Smart Kanten
|
# Bevorzuge explizite Kanten vor Smart Kanten
|
||||||
is_current_explicit = (provenance in ["explicit", "rule"])
|
is_current_explicit = (provenance in ["explicit", "rule"])
|
||||||
if existing:
|
if existing:
|
||||||
is_existing_explicit = (existing['provenance'] in ["explicit", "rule"])
|
is_existing_explicit = (existing.get('provenance', '') in ["explicit", "rule"])
|
||||||
if is_existing_explicit and not is_current_explicit:
|
if is_existing_explicit and not is_current_explicit:
|
||||||
should_update = False
|
should_update = False
|
||||||
|
|
||||||
@@ -258,38 +323,109 @@ class GraphExplorerService:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
def _resolve_note_from_ref(self, ref_str):
|
def _resolve_note_from_ref(self, ref_str):
|
||||||
"""Löst eine ID (Chunk, Note oder Titel) zu einer Note Payload auf."""
|
"""
|
||||||
if not ref_str: return None
|
Löst eine Referenz zu einer Note Payload auf.
|
||||||
|
|
||||||
# Fall A: Chunk ID (enthält #)
|
WICHTIG: Wenn ref_str ein Titel#Abschnitt Format hat, wird nur der Titel-Teil verwendet.
|
||||||
|
Unterstützt:
|
||||||
|
- Note-ID: "20250101-meine-note"
|
||||||
|
- Chunk-ID: "20250101-meine-note#c01"
|
||||||
|
- Titel: "Meine Prinzipien 2025"
|
||||||
|
- Titel#Abschnitt: "Meine Prinzipien 2025#P3 – Disziplin" (trennt Abschnitt ab, sucht nur nach Titel)
|
||||||
|
"""
|
||||||
|
if not ref_str:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Fall A: Enthält # (kann Chunk-ID oder Titel#Abschnitt sein)
|
||||||
if "#" in ref_str:
|
if "#" in ref_str:
|
||||||
try:
|
try:
|
||||||
# Versuch 1: Chunk ID direkt
|
# Versuch 1: Chunk ID direkt (Format: note_id#c01)
|
||||||
res = self.client.retrieve(self.chunks_col, ids=[ref_str], with_payload=True)
|
res = self.client.retrieve(self.chunks_col, ids=[ref_str], with_payload=True)
|
||||||
if res: return self._fetch_note_cached(res[0].payload.get("note_id"))
|
if res and res[0].payload:
|
||||||
except: pass
|
note_id = res[0].payload.get("note_id")
|
||||||
|
if note_id:
|
||||||
|
return self._fetch_note_cached(note_id)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
# Versuch 2: NoteID#Section (Hash abtrennen)
|
# Versuch 2: NoteID#Section (Hash abtrennen und als Note-ID versuchen)
|
||||||
possible_note_id = ref_str.split("#")[0]
|
# z.B. "20250101-meine-note#Abschnitt" -> "20250101-meine-note"
|
||||||
if self._fetch_note_cached(possible_note_id): return self._fetch_note_cached(possible_note_id)
|
possible_note_id = ref_str.split("#")[0].strip()
|
||||||
|
note = self._fetch_note_cached(possible_note_id)
|
||||||
|
if note:
|
||||||
|
return note
|
||||||
|
|
||||||
|
# Versuch 3: Titel#Abschnitt (Hash abtrennen und als Titel suchen)
|
||||||
|
# z.B. "Meine Prinzipien 2025#P3 – Disziplin" -> "Meine Prinzipien 2025"
|
||||||
|
# WICHTIG: target_id enthält nur den Titel, daher suchen wir nur nach dem Titel-Teil
|
||||||
|
possible_title = ref_str.split("#")[0].strip()
|
||||||
|
if possible_title:
|
||||||
|
res, _ = self.client.scroll(
|
||||||
|
collection_name=self.notes_col,
|
||||||
|
scroll_filter=models.Filter(must=[
|
||||||
|
models.FieldCondition(key="title", match=models.MatchValue(value=possible_title))
|
||||||
|
]),
|
||||||
|
limit=1, with_payload=True
|
||||||
|
)
|
||||||
|
if res and res[0].payload:
|
||||||
|
payload = res[0].payload
|
||||||
|
self._note_cache[payload['note_id']] = payload
|
||||||
|
return payload
|
||||||
|
|
||||||
|
# Fallback: Text-Suche für Fuzzy-Matching
|
||||||
|
res, _ = self.client.scroll(
|
||||||
|
collection_name=self.notes_col,
|
||||||
|
scroll_filter=models.Filter(must=[
|
||||||
|
models.FieldCondition(key="title", match=models.MatchText(text=possible_title))
|
||||||
|
]),
|
||||||
|
limit=10, with_payload=True
|
||||||
|
)
|
||||||
|
if res:
|
||||||
|
# Nimm das erste Ergebnis, das exakt oder beginnend mit possible_title übereinstimmt
|
||||||
|
for r in res:
|
||||||
|
if r.payload:
|
||||||
|
note_title = r.payload.get("title", "")
|
||||||
|
if note_title == possible_title or note_title.startswith(possible_title):
|
||||||
|
payload = r.payload
|
||||||
|
self._note_cache[payload['note_id']] = payload
|
||||||
|
return payload
|
||||||
|
|
||||||
# Fall B: Note ID direkt
|
# Fall B: Note ID direkt
|
||||||
if self._fetch_note_cached(ref_str): return self._fetch_note_cached(ref_str)
|
note = self._fetch_note_cached(ref_str)
|
||||||
|
if note:
|
||||||
|
return note
|
||||||
|
|
||||||
# Fall C: Titel
|
# Fall C: Titel (exakte Übereinstimmung)
|
||||||
res, _ = self.client.scroll(
|
res, _ = self.client.scroll(
|
||||||
collection_name=self.notes_col,
|
collection_name=self.notes_col,
|
||||||
scroll_filter=models.Filter(must=[models.FieldCondition(key="title", match=models.MatchValue(value=ref_str))]),
|
scroll_filter=models.Filter(must=[
|
||||||
|
models.FieldCondition(key="title", match=models.MatchValue(value=ref_str))
|
||||||
|
]),
|
||||||
limit=1, with_payload=True
|
limit=1, with_payload=True
|
||||||
)
|
)
|
||||||
if res:
|
if res and res[0].payload:
|
||||||
self._note_cache[res[0].payload['note_id']] = res[0].payload
|
payload = res[0].payload
|
||||||
return res[0].payload
|
self._note_cache[payload['note_id']] = payload
|
||||||
|
return payload
|
||||||
|
|
||||||
|
# Fall D: Titel (Text-Suche für Fuzzy-Matching)
|
||||||
|
res, _ = self.client.scroll(
|
||||||
|
collection_name=self.notes_col,
|
||||||
|
scroll_filter=models.Filter(must=[
|
||||||
|
models.FieldCondition(key="title", match=models.MatchText(text=ref_str))
|
||||||
|
]),
|
||||||
|
limit=1, with_payload=True
|
||||||
|
)
|
||||||
|
if res and res[0].payload:
|
||||||
|
payload = res[0].payload
|
||||||
|
self._note_cache[payload['note_id']] = payload
|
||||||
|
return payload
|
||||||
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
def _add_node_to_dict(self, node_dict, note_payload, level=1):
|
def _add_node_to_dict(self, node_dict, note_payload, level=1):
|
||||||
nid = note_payload.get("note_id")
|
nid = note_payload.get("note_id")
|
||||||
if nid in node_dict: return
|
if not nid or nid in node_dict: return
|
||||||
|
|
||||||
ntype = note_payload.get("type", "default")
|
ntype = note_payload.get("type", "default")
|
||||||
color = GRAPH_COLORS.get(ntype, GRAPH_COLORS["default"])
|
color = GRAPH_COLORS.get(ntype, GRAPH_COLORS["default"])
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_sidebar.py
|
||||||
|
DESCRIPTION: Rendert die Sidebar. Steuert den Modus-Wechsel (Chat/Editor/Graph) und globale Settings (Top-K).
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: streamlit, ui_utils, ui_config
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
from ui_utils import load_history_from_logs
|
from ui_utils import load_history_from_logs
|
||||||
from ui_config import HISTORY_FILE
|
from ui_config import HISTORY_FILE
|
||||||
|
|||||||
@@ -1,3 +1,12 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/frontend/ui_utils.py
|
||||||
|
DESCRIPTION: String-Utilities. Parser für Markdown/YAML (LLM-Healing) und Helper für History-Loading.
|
||||||
|
VERSION: 2.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: re, yaml, unicodedata, json, datetime
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
import re
|
import re
|
||||||
import yaml
|
import yaml
|
||||||
import unicodedata
|
import unicodedata
|
||||||
|
|||||||
@@ -1,172 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""
|
|
||||||
Modul: app/graph/service.py
|
|
||||||
Version: 0.1.0
|
|
||||||
Datum: 2025-09-10
|
|
||||||
|
|
||||||
Zweck
|
|
||||||
-----
|
|
||||||
Leichtgewichtiger Graph-Layer über Qdrant:
|
|
||||||
- get_note(note_id)
|
|
||||||
- get_chunks(note_id)
|
|
||||||
- neighbors(source_id, kinds=[...], scope=['note','chunk'], depth=1)
|
|
||||||
- walk_bfs(source_id, kinds, max_depth)
|
|
||||||
- context_for_note(note_id, max_neighbors): heuristische Kontextsammlung
|
|
||||||
|
|
||||||
Hinweise
|
|
||||||
--------
|
|
||||||
- Nutzt die bestehenden Collections <prefix>_notes/_chunks/_edges.
|
|
||||||
- Edges werden über Payload-Felder (`kind`, `source_id`, `target_id`) abgefragt.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
from typing import List, Dict, Any, Optional, Iterable, Set, Tuple
|
|
||||||
from qdrant_client.http import models as rest
|
|
||||||
from app.core.qdrant import QdrantConfig, get_client
|
|
||||||
|
|
||||||
def _cols(prefix: str):
|
|
||||||
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
|
||||||
|
|
||||||
class GraphService:
|
|
||||||
def __init__(self, cfg: Optional[QdrantConfig] = None, prefix: Optional[str] = None):
|
|
||||||
self.cfg = cfg or QdrantConfig.from_env()
|
|
||||||
if prefix:
|
|
||||||
self.cfg.prefix = prefix
|
|
||||||
self.client = get_client(self.cfg)
|
|
||||||
self.notes_col, self.chunks_col, self.edges_col = _cols(self.cfg.prefix)
|
|
||||||
|
|
||||||
# ------------------------ fetch helpers ------------------------
|
|
||||||
def _scroll(self, col: str, flt: Optional[rest.Filter] = None, limit: int = 256):
|
|
||||||
out = []
|
|
||||||
nextp = None
|
|
||||||
while True:
|
|
||||||
pts, nextp = self.client.scroll(
|
|
||||||
collection_name=col,
|
|
||||||
with_payload=True,
|
|
||||||
with_vectors=False,
|
|
||||||
limit=limit,
|
|
||||||
offset=nextp,
|
|
||||||
scroll_filter=flt,
|
|
||||||
)
|
|
||||||
if not pts:
|
|
||||||
break
|
|
||||||
out.extend(pts)
|
|
||||||
if nextp is None:
|
|
||||||
break
|
|
||||||
return out
|
|
||||||
|
|
||||||
# ------------------------ public API ---------------------------
|
|
||||||
def get_note(self, note_id: str) -> Optional[Dict[str, Any]]:
|
|
||||||
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
|
||||||
pts, _ = self.client.scroll(self.notes_col, with_payload=True, with_vectors=False, limit=1, scroll_filter=f)
|
|
||||||
return (pts[0].payload or None) if pts else None
|
|
||||||
|
|
||||||
def get_chunks(self, note_id: str) -> List[Dict[str, Any]]:
|
|
||||||
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
|
||||||
pts = self._scroll(self.chunks_col, f)
|
|
||||||
# Sortierung analog Export
|
|
||||||
def key(pl):
|
|
||||||
p = pl.payload or {}
|
|
||||||
s = p.get("seq") or 0
|
|
||||||
ci = p.get("chunk_index") or 0
|
|
||||||
n = 0
|
|
||||||
cid = p.get("chunk_id") or ""
|
|
||||||
if isinstance(cid, str) and "#" in cid:
|
|
||||||
try:
|
|
||||||
n = int(cid.rsplit("#", 1)[-1])
|
|
||||||
except Exception:
|
|
||||||
n = 0
|
|
||||||
return (int(s), int(ci), n)
|
|
||||||
pts_sorted = sorted(pts, key=key)
|
|
||||||
return [p.payload or {} for p in pts_sorted]
|
|
||||||
|
|
||||||
def neighbors(self, source_id: str, kinds: Optional[Iterable[str]] = None,
|
|
||||||
scope: Optional[Iterable[str]] = None, depth: int = 1) -> Dict[str, List[Dict[str, Any]]]:
|
|
||||||
"""
|
|
||||||
Liefert eingehende & ausgehende Nachbarn (nur nach kind gefiltert).
|
|
||||||
depth==1: direkte Kanten.
|
|
||||||
"""
|
|
||||||
kinds = list(kinds) if kinds else None
|
|
||||||
must = [rest.FieldCondition(key="source_id", match=rest.MatchValue(value=source_id))]
|
|
||||||
if kinds:
|
|
||||||
must.append(rest.FieldCondition(key="kind", match=rest.MatchAny(any=kinds)))
|
|
||||||
f = rest.Filter(must=must)
|
|
||||||
edges = self._scroll(self.edges_col, f)
|
|
||||||
out = {"out": [], "in": []}
|
|
||||||
for e in edges:
|
|
||||||
out["out"].append(e.payload or {})
|
|
||||||
# Inverse Richtung (eingehend)
|
|
||||||
must_in = [rest.FieldCondition(key="target_id", match=rest.MatchValue(value=source_id))]
|
|
||||||
if kinds:
|
|
||||||
must_in.append(rest.FieldCondition(key="kind", match=rest.MatchAny(any=kinds)))
|
|
||||||
f_in = rest.Filter(must=must_in)
|
|
||||||
edges_in = self._scroll(self.edges_col, f_in)
|
|
||||||
for e in edges_in:
|
|
||||||
out["in"].append(e.payload or {})
|
|
||||||
return out
|
|
||||||
|
|
||||||
def walk_bfs(self, source_id: str, kinds: Iterable[str], max_depth: int = 2) -> Set[str]:
|
|
||||||
visited: Set[str] = {source_id}
|
|
||||||
frontier: Set[str] = {source_id}
|
|
||||||
kinds = list(kinds)
|
|
||||||
for _ in range(max_depth):
|
|
||||||
nxt: Set[str] = set()
|
|
||||||
for s in frontier:
|
|
||||||
neigh = self.neighbors(s, kinds=kinds)
|
|
||||||
for e in neigh["out"]:
|
|
||||||
t = e.get("target_id")
|
|
||||||
if isinstance(t, str) and t not in visited:
|
|
||||||
visited.add(t)
|
|
||||||
nxt.add(t)
|
|
||||||
frontier = nxt
|
|
||||||
if not frontier:
|
|
||||||
break
|
|
||||||
return visited
|
|
||||||
|
|
||||||
def context_for_note(self, note_id: str, kinds: Iterable[str] = ("references","backlink"), max_neighbors: int = 12) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Heuristischer Kontext: eigene Chunks + Nachbarn nach Kantenarten, dedupliziert.
|
|
||||||
"""
|
|
||||||
note = self.get_note(note_id) or {}
|
|
||||||
chunks = self.get_chunks(note_id)
|
|
||||||
neigh = self.neighbors(note_id, kinds=list(kinds))
|
|
||||||
targets = []
|
|
||||||
for e in neigh["out"]:
|
|
||||||
t = e.get("target_id")
|
|
||||||
if isinstance(t, str):
|
|
||||||
targets.append(t)
|
|
||||||
for e in neigh["in"]:
|
|
||||||
s = e.get("source_id")
|
|
||||||
if isinstance(s, str):
|
|
||||||
targets.append(s)
|
|
||||||
# de-dupe
|
|
||||||
seen = set()
|
|
||||||
uniq = []
|
|
||||||
for t in targets:
|
|
||||||
if t not in seen:
|
|
||||||
seen.add(t)
|
|
||||||
uniq.append(t)
|
|
||||||
uniq = uniq[:max_neighbors]
|
|
||||||
neighbor_notes = [self.get_note(t) for t in uniq]
|
|
||||||
return {
|
|
||||||
"note": note,
|
|
||||||
"chunks": chunks,
|
|
||||||
"neighbors": [n for n in neighbor_notes if n],
|
|
||||||
"edges_out": neigh["out"],
|
|
||||||
"edges_in": neigh["in"],
|
|
||||||
}
|
|
||||||
|
|
||||||
# Optional: Mini-CLI
|
|
||||||
if __name__ == "__main__": # pragma: no cover
|
|
||||||
import argparse, json
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--prefix", help="Collection-Prefix (überschreibt ENV)")
|
|
||||||
ap.add_argument("--note-id", required=True)
|
|
||||||
ap.add_argument("--neighbors", action="store_true", help="Nur Nachbarn anzeigen")
|
|
||||||
args = ap.parse_args()
|
|
||||||
svc = GraphService(prefix=args.prefix)
|
|
||||||
if args.neighbors:
|
|
||||||
out = svc.neighbors(args.note_id, kinds=["references","backlink","prev","next","belongs_to"])
|
|
||||||
else:
|
|
||||||
out = svc.context_for_note(args.note_id)
|
|
||||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
|
||||||
+11
-5
@@ -1,11 +1,17 @@
|
|||||||
"""
|
"""
|
||||||
app/main.py — mindnet API bootstrap
|
FILE: app/main.py
|
||||||
|
DESCRIPTION: Bootstrap der FastAPI Anwendung. Inkludiert Router und Middleware.
|
||||||
|
VERSION: 0.6.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: app.config, app.routers.* (embed, qdrant, query, graph, tools, feedback, chat, ingest, admin)
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from .config import get_settings
|
from .config import get_settings
|
||||||
from .routers.embed_router import router as embed_router
|
#from .routers.embed_router import router as embed_router
|
||||||
from .routers.qdrant_router import router as qdrant_router
|
#from .routers.qdrant_router import router as qdrant_router
|
||||||
|
|
||||||
from .routers.query import router as query_router
|
from .routers.query import router as query_router
|
||||||
from .routers.graph import router as graph_router
|
from .routers.graph import router as graph_router
|
||||||
@@ -29,8 +35,8 @@ def create_app() -> FastAPI:
|
|||||||
def healthz():
|
def healthz():
|
||||||
return {"status": "ok", "qdrant": s.QDRANT_URL, "prefix": s.COLLECTION_PREFIX}
|
return {"status": "ok", "qdrant": s.QDRANT_URL, "prefix": s.COLLECTION_PREFIX}
|
||||||
|
|
||||||
app.include_router(embed_router)
|
# app.include_router(embed_router)
|
||||||
app.include_router(qdrant_router)
|
# app.include_router(qdrant_router)
|
||||||
|
|
||||||
app.include_router(query_router, prefix="/query", tags=["query"])
|
app.include_router(query_router, prefix="/query", tags=["query"])
|
||||||
app.include_router(graph_router, prefix="/graph", tags=["graph"])
|
app.include_router(graph_router, prefix="/graph", tags=["graph"])
|
||||||
|
|||||||
+31
-28
@@ -1,14 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/models/dto.py — Pydantic-Modelle (DTOs) für WP-04/WP-05/WP-06
|
FILE: app/models/dto.py
|
||||||
|
DESCRIPTION: Pydantic-Modelle (DTOs) für Request/Response Bodies. Definiert das API-Schema.
|
||||||
Zweck:
|
VERSION: 0.6.7 (WP-Fix: Target Section Support)
|
||||||
Laufzeit-Modelle für FastAPI (Requests/Responses).
|
STATUS: Active
|
||||||
WP-06 Update: Intent & Intent-Source in ChatResponse.
|
DEPENDENCIES: pydantic, typing, uuid
|
||||||
|
LAST_ANALYSIS: 2025-12-29
|
||||||
Version:
|
|
||||||
0.6.2 (WP-06: Decision Engine Transparency, Erweiterung des Feeback Request)
|
|
||||||
Stand:
|
|
||||||
2025-12-09
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -16,7 +12,8 @@ from pydantic import BaseModel, Field
|
|||||||
from typing import List, Literal, Optional, Dict, Any
|
from typing import List, Literal, Optional, Dict, Any
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
EdgeKind = Literal["references", "references_at", "backlink", "next", "prev", "belongs_to", "depends_on", "related_to", "similar_to"]
|
# Gültige Kanten-Typen gemäß Manual
|
||||||
|
EdgeKind = Literal["references", "references_at", "backlink", "next", "prev", "belongs_to", "depends_on", "related_to", "similar_to", "caused_by", "derived_from", "based_on", "solves", "blocks", "uses", "guides"]
|
||||||
|
|
||||||
|
|
||||||
# --- Basis-DTOs ---
|
# --- Basis-DTOs ---
|
||||||
@@ -44,6 +41,9 @@ class EdgeDTO(BaseModel):
|
|||||||
target: str
|
target: str
|
||||||
weight: float
|
weight: float
|
||||||
direction: Literal["out", "in", "undirected"] = "out"
|
direction: Literal["out", "in", "undirected"] = "out"
|
||||||
|
provenance: Optional[Literal["explicit", "rule", "smart", "structure"]] = "explicit"
|
||||||
|
confidence: float = 1.0
|
||||||
|
target_section: Optional[str] = None # Neu: Speichert den Anker (z.B. #Abschnitt)
|
||||||
|
|
||||||
|
|
||||||
# --- Request Models ---
|
# --- Request Models ---
|
||||||
@@ -60,18 +60,18 @@ class QueryRequest(BaseModel):
|
|||||||
filters: Optional[Dict] = None
|
filters: Optional[Dict] = None
|
||||||
ret: Dict = {"with_paths": True, "with_notes": True, "with_chunks": True}
|
ret: Dict = {"with_paths": True, "with_notes": True, "with_chunks": True}
|
||||||
explain: bool = False
|
explain: bool = False
|
||||||
|
|
||||||
|
# WP-22: Semantic Graph Routing
|
||||||
|
boost_edges: Optional[Dict[str, float]] = None
|
||||||
|
|
||||||
|
|
||||||
class FeedbackRequest(BaseModel):
|
class FeedbackRequest(BaseModel):
|
||||||
"""
|
"""
|
||||||
User-Feedback zu einem spezifischen Treffer oder der Gesamtantwort.
|
User-Feedback zu einem spezifischen Treffer oder der Gesamtantwort (WP-08 Basis).
|
||||||
"""
|
"""
|
||||||
query_id: str = Field(..., description="ID der ursprünglichen Suche")
|
query_id: str = Field(..., description="ID der ursprünglichen Suche")
|
||||||
# node_id ist optional: Wenn leer oder "generated_answer", gilt es für die Antwort.
|
|
||||||
# Wenn eine echte Chunk-ID, gilt es für die Quelle.
|
|
||||||
node_id: str = Field(..., description="ID des bewerteten Treffers oder 'generated_answer'")
|
node_id: str = Field(..., description="ID des bewerteten Treffers oder 'generated_answer'")
|
||||||
# Update: Range auf 1-5 erweitert für differenziertes Tuning
|
score: int = Field(..., ge=1, le=5, description="1 (Irrelevant) bis 5 (Perfekt)")
|
||||||
score: int = Field(..., ge=1, le=5, description="1 (Irrelevant/Falsch) bis 5 (Perfekt)")
|
|
||||||
comment: Optional[str] = None
|
comment: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
@@ -80,8 +80,7 @@ class ChatRequest(BaseModel):
|
|||||||
WP-05: Request für /chat.
|
WP-05: Request für /chat.
|
||||||
"""
|
"""
|
||||||
message: str = Field(..., description="Die Nachricht des Users")
|
message: str = Field(..., description="Die Nachricht des Users")
|
||||||
conversation_id: Optional[str] = Field(None, description="Optional: ID für Chat-Verlauf (noch nicht implementiert)")
|
conversation_id: Optional[str] = Field(None, description="ID für Chat-Verlauf")
|
||||||
# RAG Parameter (Override defaults)
|
|
||||||
top_k: int = 5
|
top_k: int = 5
|
||||||
explain: bool = False
|
explain: bool = False
|
||||||
|
|
||||||
@@ -89,7 +88,7 @@ class ChatRequest(BaseModel):
|
|||||||
# --- WP-04b Explanation Models ---
|
# --- WP-04b Explanation Models ---
|
||||||
|
|
||||||
class ScoreBreakdown(BaseModel):
|
class ScoreBreakdown(BaseModel):
|
||||||
"""Aufschlüsselung der Score-Komponenten."""
|
"""Aufschlüsselung der Score-Komponenten nach der WP-22 Formel."""
|
||||||
semantic_contribution: float
|
semantic_contribution: float
|
||||||
edge_contribution: float
|
edge_contribution: float
|
||||||
centrality_contribution: float
|
centrality_contribution: float
|
||||||
@@ -97,11 +96,14 @@ class ScoreBreakdown(BaseModel):
|
|||||||
raw_edge_bonus: float
|
raw_edge_bonus: float
|
||||||
raw_centrality: float
|
raw_centrality: float
|
||||||
node_weight: float
|
node_weight: float
|
||||||
|
# WP-22 Debug Fields für Messbarkeit
|
||||||
|
status_multiplier: float = 1.0
|
||||||
|
graph_boost_factor: float = 1.0
|
||||||
|
|
||||||
|
|
||||||
class Reason(BaseModel):
|
class Reason(BaseModel):
|
||||||
"""Ein semantischer Grund für das Ranking."""
|
"""Ein semantischer Grund für das Ranking."""
|
||||||
kind: Literal["semantic", "edge", "type", "centrality"]
|
kind: Literal["semantic", "edge", "type", "centrality", "lifecycle"]
|
||||||
message: str
|
message: str
|
||||||
score_impact: Optional[float] = None
|
score_impact: Optional[float] = None
|
||||||
details: Optional[Dict[str, Any]] = None
|
details: Optional[Dict[str, Any]] = None
|
||||||
@@ -112,6 +114,9 @@ class Explanation(BaseModel):
|
|||||||
breakdown: ScoreBreakdown
|
breakdown: ScoreBreakdown
|
||||||
reasons: List[Reason]
|
reasons: List[Reason]
|
||||||
related_edges: Optional[List[EdgeDTO]] = None
|
related_edges: Optional[List[EdgeDTO]] = None
|
||||||
|
# WP-22 Debug: Verifizierung des Routings
|
||||||
|
applied_intent: Optional[str] = None
|
||||||
|
applied_boosts: Optional[Dict[str, float]] = None
|
||||||
|
|
||||||
|
|
||||||
# --- Response Models ---
|
# --- Response Models ---
|
||||||
@@ -119,14 +124,14 @@ class Explanation(BaseModel):
|
|||||||
class QueryHit(BaseModel):
|
class QueryHit(BaseModel):
|
||||||
"""Einzelnes Trefferobjekt für /query."""
|
"""Einzelnes Trefferobjekt für /query."""
|
||||||
node_id: str
|
node_id: str
|
||||||
note_id: Optional[str]
|
note_id: str
|
||||||
semantic_score: float
|
semantic_score: float
|
||||||
edge_bonus: float
|
edge_bonus: float
|
||||||
centrality_bonus: float
|
centrality_bonus: float
|
||||||
total_score: float
|
total_score: float
|
||||||
paths: Optional[List[List[Dict]]] = None
|
paths: Optional[List[List[Dict]]] = None
|
||||||
source: Optional[Dict] = None
|
source: Optional[Dict] = None
|
||||||
payload: Optional[Dict] = None # Added for flexibility & WP-06 meta-data
|
payload: Optional[Dict] = None
|
||||||
explanation: Optional[Explanation] = None
|
explanation: Optional[Explanation] = None
|
||||||
|
|
||||||
|
|
||||||
@@ -150,11 +155,9 @@ class ChatResponse(BaseModel):
|
|||||||
"""
|
"""
|
||||||
WP-05/06: Antwortstruktur für /chat.
|
WP-05/06: Antwortstruktur für /chat.
|
||||||
"""
|
"""
|
||||||
query_id: str = Field(..., description="Traceability ID (dieselbe wie für Search)")
|
query_id: str = Field(..., description="Traceability ID")
|
||||||
answer: str = Field(..., description="Generierte Antwort vom LLM")
|
answer: str = Field(..., description="Generierte Antwort vom LLM")
|
||||||
sources: List[QueryHit] = Field(..., description="Die für die Antwort genutzten Quellen")
|
sources: List[QueryHit] = Field(..., description="Die genutzten Quellen")
|
||||||
latency_ms: int
|
latency_ms: int
|
||||||
intent: Optional[str] = Field("FACT", description="WP-06: Erkannter Intent (FACT/DECISION)")
|
intent: Optional[str] = Field("FACT", description="WP-06: Erkannter Intent")
|
||||||
intent_source: Optional[str] = Field("Unknown", description="WP-06: Quelle der Intent-Erkennung (Keyword vs. LLM)")
|
intent_source: Optional[str] = Field("Unknown", description="Quelle der Intent-Erkennung")
|
||||||
|
|
||||||
|
|
||||||
+6
-16
@@ -1,20 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/admin.py — Admin-/Monitoring-Endpunkte (optional)
|
FILE: app/routers/admin.py
|
||||||
|
DESCRIPTION: Monitoring-Endpunkt. Zeigt Qdrant-Collection-Counts und geladene Config.
|
||||||
Zweck:
|
VERSION: 0.1.0
|
||||||
Liefert einfache Kennzahlen zu Collections (Counts) und Config.
|
STATUS: Active (Optional)
|
||||||
Kompatibilität:
|
DEPENDENCIES: qdrant_client, app.config
|
||||||
Python 3.12+, FastAPI 0.110+, qdrant-client 1.x
|
LAST_ANALYSIS: 2025-12-15
|
||||||
Version:
|
|
||||||
0.1.0 (Erstanlage)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Bezug:
|
|
||||||
- Qdrant Collections: *_notes, *_chunks, *_edges
|
|
||||||
Nutzung:
|
|
||||||
app.include_router(admin.router, prefix="/admin", tags=["admin"])
|
|
||||||
Änderungsverlauf:
|
|
||||||
0.1.0 (2025-10-07) – Erstanlage.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|||||||
+80
-35
@@ -1,6 +1,12 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/chat.py — RAG Endpunkt
|
FILE: app/routers/chat.py
|
||||||
Version: 2.5.0 (Fix: Question Detection protects against False-Positive Interviews)
|
DESCRIPTION: Haupt-Chat-Interface (RAG & Interview). Enthält Intent-Router (Keywords/LLM) und Prompt-Construction.
|
||||||
|
VERSION: 2.7.8 (Full Unabridged Stability Edition)
|
||||||
|
STATUS: Active
|
||||||
|
FIX:
|
||||||
|
1. Implementiert Context-Throttling für Ollama (MAX_OLLAMA_CHARS).
|
||||||
|
2. Deaktiviert LLM-Retries für den Chat (max_retries=0).
|
||||||
|
3. Behebt Double-Fallback-Schleifen und Silent Refusals.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Depends
|
from fastapi import APIRouter, HTTPException, Depends
|
||||||
@@ -15,7 +21,7 @@ from pathlib import Path
|
|||||||
from app.config import get_settings
|
from app.config import get_settings
|
||||||
from app.models.dto import ChatRequest, ChatResponse, QueryRequest, QueryHit
|
from app.models.dto import ChatRequest, ChatResponse, QueryRequest, QueryHit
|
||||||
from app.services.llm_service import LLMService
|
from app.services.llm_service import LLMService
|
||||||
from app.core.retriever import Retriever
|
from app.core.retrieval.retriever import Retriever
|
||||||
from app.services.feedback_service import log_search
|
from app.services.feedback_service import log_search
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
@@ -31,7 +37,7 @@ def _load_decision_config() -> Dict[str, Any]:
|
|||||||
path = Path(settings.DECISION_CONFIG_PATH)
|
path = Path(settings.DECISION_CONFIG_PATH)
|
||||||
default_config = {
|
default_config = {
|
||||||
"strategies": {
|
"strategies": {
|
||||||
"FACT": {"trigger_keywords": []}
|
"FACT": {"trigger_keywords": [], "preferred_provider": "openrouter"}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -154,7 +160,7 @@ def _is_question(query: str) -> bool:
|
|||||||
q = query.strip().lower()
|
q = query.strip().lower()
|
||||||
if "?" in q: return True
|
if "?" in q: return True
|
||||||
|
|
||||||
# W-Fragen Indikatoren (falls User das ? vergisst)
|
# W-Fragen Indikatoren
|
||||||
starters = ["wer", "wie", "was", "wo", "wann", "warum", "weshalb", "wozu", "welche", "bist du", "entspricht"]
|
starters = ["wer", "wie", "was", "wo", "wann", "warum", "weshalb", "wozu", "welche", "bist du", "entspricht"]
|
||||||
if any(q.startswith(s + " ") for s in starters):
|
if any(q.startswith(s + " ") for s in starters):
|
||||||
return True
|
return True
|
||||||
@@ -165,7 +171,7 @@ async def _classify_intent(query: str, llm: LLMService) -> tuple[str, str]:
|
|||||||
"""
|
"""
|
||||||
Hybrid Router v5:
|
Hybrid Router v5:
|
||||||
1. Decision Keywords (Strategie) -> Prio 1
|
1. Decision Keywords (Strategie) -> Prio 1
|
||||||
2. Type Keywords (Interview Trigger) -> Prio 2, ABER NUR WENN KEINE FRAGE!
|
2. Type Keywords (Interview Trigger) -> Prio 2
|
||||||
3. LLM (Fallback) -> Prio 3
|
3. LLM (Fallback) -> Prio 3
|
||||||
"""
|
"""
|
||||||
config = get_full_config()
|
config = get_full_config()
|
||||||
@@ -174,7 +180,7 @@ async def _classify_intent(query: str, llm: LLMService) -> tuple[str, str]:
|
|||||||
|
|
||||||
query_lower = query.lower()
|
query_lower = query.lower()
|
||||||
|
|
||||||
# 1. FAST PATH A: Strategie Keywords (z.B. "Soll ich...")
|
# 1. FAST PATH A: Strategie Keywords
|
||||||
for intent_name, strategy in strategies.items():
|
for intent_name, strategy in strategies.items():
|
||||||
if intent_name == "FACT": continue
|
if intent_name == "FACT": continue
|
||||||
keywords = strategy.get("trigger_keywords", [])
|
keywords = strategy.get("trigger_keywords", [])
|
||||||
@@ -182,10 +188,7 @@ async def _classify_intent(query: str, llm: LLMService) -> tuple[str, str]:
|
|||||||
if k.lower() in query_lower:
|
if k.lower() in query_lower:
|
||||||
return intent_name, "Keyword (Strategy)"
|
return intent_name, "Keyword (Strategy)"
|
||||||
|
|
||||||
# 2. FAST PATH B: Type Keywords (z.B. "Projekt", "Werte") -> INTERVIEW
|
# 2. FAST PATH B: Type Keywords -> INTERVIEW
|
||||||
# FIX: Wir prüfen, ob es eine Frage ist. Fragen zu Typen sollen RAG (FACT/DECISION) sein,
|
|
||||||
# keine Interviews. Wir überlassen das dann dem LLM Router (Slow Path).
|
|
||||||
|
|
||||||
if not _is_question(query_lower):
|
if not _is_question(query_lower):
|
||||||
types_cfg = get_types_config()
|
types_cfg = get_types_config()
|
||||||
types_def = types_cfg.get("types", {})
|
types_def = types_cfg.get("types", {})
|
||||||
@@ -198,19 +201,17 @@ async def _classify_intent(query: str, llm: LLMService) -> tuple[str, str]:
|
|||||||
|
|
||||||
# 3. SLOW PATH: LLM Router
|
# 3. SLOW PATH: LLM Router
|
||||||
if settings.get("llm_fallback_enabled", False):
|
if settings.get("llm_fallback_enabled", False):
|
||||||
# Nutze Prompts aus prompts.yaml (via LLM Service)
|
router_prompt_template = llm.get_prompt("llm_router_prompt")
|
||||||
router_prompt_template = llm.prompts.get("router_prompt", "")
|
|
||||||
|
|
||||||
if router_prompt_template:
|
if router_prompt_template:
|
||||||
prompt = router_prompt_template.replace("{query}", query)
|
prompt = router_prompt_template.replace("{query}", query)
|
||||||
logger.info("Keywords failed (or Question detected). Asking LLM for Intent...")
|
logger.info("Keywords failed (or Question detected). Asking LLM for Intent...")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Nutze priority="realtime" für den Router, damit er nicht wartet
|
# FIX: Auch beim Routing keine Retries im Chat-Fluss
|
||||||
raw_response = await llm.generate_raw_response(prompt, priority="realtime")
|
raw_response = await llm.generate_raw_response(prompt, priority="realtime", max_retries=0)
|
||||||
llm_output_upper = raw_response.upper()
|
llm_output_upper = raw_response.upper()
|
||||||
|
|
||||||
# Zuerst INTERVIEW prüfen
|
|
||||||
if "INTERVIEW" in llm_output_upper or "CREATE" in llm_output_upper:
|
if "INTERVIEW" in llm_output_upper or "CREATE" in llm_output_upper:
|
||||||
return "INTERVIEW", "LLM Router"
|
return "INTERVIEW", "LLM Router"
|
||||||
|
|
||||||
@@ -241,9 +242,11 @@ async def chat_endpoint(
|
|||||||
# Strategy Load
|
# Strategy Load
|
||||||
strategy = get_decision_strategy(intent)
|
strategy = get_decision_strategy(intent)
|
||||||
prompt_key = strategy.get("prompt_template", "rag_template")
|
prompt_key = strategy.get("prompt_template", "rag_template")
|
||||||
|
preferred_provider = strategy.get("preferred_provider")
|
||||||
|
|
||||||
sources_hits = []
|
sources_hits = []
|
||||||
final_prompt = ""
|
final_prompt = ""
|
||||||
|
context_str = ""
|
||||||
|
|
||||||
if intent == "INTERVIEW":
|
if intent == "INTERVIEW":
|
||||||
# --- INTERVIEW MODE ---
|
# --- INTERVIEW MODE ---
|
||||||
@@ -264,7 +267,7 @@ async def chat_endpoint(
|
|||||||
logger.info(f"[{query_id}] Interview Type: {target_type}. Fields: {len(fields_list)}")
|
logger.info(f"[{query_id}] Interview Type: {target_type}. Fields: {len(fields_list)}")
|
||||||
fields_str = "\n- " + "\n- ".join(fields_list)
|
fields_str = "\n- " + "\n- ".join(fields_list)
|
||||||
|
|
||||||
template = llm.prompts.get(prompt_key, "")
|
template = llm.get_prompt(prompt_key)
|
||||||
final_prompt = template.replace("{context_str}", "Dialogverlauf...") \
|
final_prompt = template.replace("{context_str}", "Dialogverlauf...") \
|
||||||
.replace("{query}", request.message) \
|
.replace("{query}", request.message) \
|
||||||
.replace("{target_type}", target_type) \
|
.replace("{target_type}", target_type) \
|
||||||
@@ -273,15 +276,17 @@ async def chat_endpoint(
|
|||||||
sources_hits = []
|
sources_hits = []
|
||||||
|
|
||||||
else:
|
else:
|
||||||
# --- RAG MODE ---
|
# --- RAG MODE (FACT, DECISION, EMPATHY, CODING) ---
|
||||||
inject_types = strategy.get("inject_types", [])
|
inject_types = strategy.get("inject_types", [])
|
||||||
prepend_instr = strategy.get("prepend_instruction", "")
|
prepend_instr = strategy.get("prepend_instruction", "")
|
||||||
|
edge_boosts = strategy.get("edge_boosts", {})
|
||||||
|
|
||||||
query_req = QueryRequest(
|
query_req = QueryRequest(
|
||||||
query=request.message,
|
query=request.message,
|
||||||
mode="hybrid",
|
mode="hybrid",
|
||||||
top_k=request.top_k,
|
top_k=request.top_k,
|
||||||
explain=request.explain
|
explain=request.explain,
|
||||||
|
boost_edges=edge_boosts
|
||||||
)
|
)
|
||||||
retrieve_result = await retriever.search(query_req)
|
retrieve_result = await retriever.search(query_req)
|
||||||
hits = retrieve_result.results
|
hits = retrieve_result.results
|
||||||
@@ -292,7 +297,8 @@ async def chat_endpoint(
|
|||||||
mode="hybrid",
|
mode="hybrid",
|
||||||
top_k=3,
|
top_k=3,
|
||||||
filters={"type": inject_types},
|
filters={"type": inject_types},
|
||||||
explain=False
|
explain=False,
|
||||||
|
boost_edges=edge_boosts
|
||||||
)
|
)
|
||||||
strategy_result = await retriever.search(strategy_req)
|
strategy_result = await retriever.search(strategy_req)
|
||||||
existing_ids = {h.node_id for h in hits}
|
existing_ids = {h.node_id for h in hits}
|
||||||
@@ -300,12 +306,19 @@ async def chat_endpoint(
|
|||||||
if strat_hit.node_id not in existing_ids:
|
if strat_hit.node_id not in existing_ids:
|
||||||
hits.append(strat_hit)
|
hits.append(strat_hit)
|
||||||
|
|
||||||
if not hits:
|
context_str = _build_enriched_context(hits) if hits else "Keine relevanten Notizen gefunden."
|
||||||
context_str = "Keine relevanten Notizen gefunden."
|
|
||||||
else:
|
# --- STABILITY FIX: OLLAMA CONTEXT THROTTLE ---
|
||||||
context_str = _build_enriched_context(hits)
|
# Begrenzt den Text, um den "decode: cannot decode batches" Fehler zu vermeiden.
|
||||||
|
# MAX_OLLAMA_CHARS = 10000
|
||||||
|
|
||||||
template = llm.prompts.get(prompt_key, "{context_str}\n\n{query}")
|
settings = get_settings() # Falls noch nicht im Scope vorhanden
|
||||||
|
max_chars = getattr(settings, "MAX_OLLAMA_CHARS", 10000)
|
||||||
|
if preferred_provider == "ollama" and len(context_str) > max_chars:
|
||||||
|
logger.warning(f"⚠️ [{query_id}] Context zu groß für Ollama ({len(context_str)} chars). Kürze auf {max_chars}.")
|
||||||
|
context_str = context_str[:max_chars] + "\n[...gekürzt zur Stabilität...]"
|
||||||
|
|
||||||
|
template = llm.get_prompt(prompt_key) or "{context_str}\n\n{query}"
|
||||||
|
|
||||||
if prepend_instr:
|
if prepend_instr:
|
||||||
context_str = f"{prepend_instr}\n\n{context_str}"
|
context_str = f"{prepend_instr}\n\n{context_str}"
|
||||||
@@ -313,15 +326,46 @@ async def chat_endpoint(
|
|||||||
final_prompt = template.replace("{context_str}", context_str).replace("{query}", request.message)
|
final_prompt = template.replace("{context_str}", context_str).replace("{query}", request.message)
|
||||||
sources_hits = hits
|
sources_hits = hits
|
||||||
|
|
||||||
# --- GENERATION ---
|
# --- DEBUG SPOT 1: PROMPT CONSTRUCTION ---
|
||||||
system_prompt = llm.prompts.get("system_prompt", "")
|
logger.info(f"[{query_id}] PROMPT CONSTRUCTION COMPLETE. Length: {len(final_prompt)} chars.")
|
||||||
|
if not final_prompt.strip():
|
||||||
|
logger.error(f"[{query_id}] CRITICAL: Final prompt is empty before sending to LLM!")
|
||||||
|
|
||||||
|
# --- GENERATION WITH NO-RETRY & DEEP FALLBACK ---
|
||||||
|
system_prompt = llm.get_prompt("system_prompt")
|
||||||
|
|
||||||
# Chat nutzt IMMER realtime priority
|
# --- DEBUG SPOT 2: PRIMARY CALL ---
|
||||||
answer_text = await llm.generate_raw_response(
|
logger.info(f"[{query_id}] PRIMARY CALL: Sending request to provider '{preferred_provider}' (No Retries)...")
|
||||||
prompt=final_prompt,
|
|
||||||
system=system_prompt,
|
answer_text = ""
|
||||||
priority="realtime"
|
try:
|
||||||
)
|
# FIX: max_retries=0 verhindert Hänger durch Retry-Kaskaden im Chat
|
||||||
|
answer_text = await llm.generate_raw_response(
|
||||||
|
prompt=final_prompt,
|
||||||
|
system=system_prompt,
|
||||||
|
priority="realtime",
|
||||||
|
provider=preferred_provider,
|
||||||
|
max_retries=0
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"🛑 [{query_id}] Primary Provider '{preferred_provider}' failed: {e}")
|
||||||
|
|
||||||
|
# DEEP FALLBACK: Wenn die Antwort leer ist (Silent Refusal) oder der Primary abgestürzt ist
|
||||||
|
if not answer_text.strip() and preferred_provider != "ollama":
|
||||||
|
# --- DEBUG SPOT 3: FALLBACK TRIGGER ---
|
||||||
|
logger.warning(f"🛑 [{query_id}] PRIMARY '{preferred_provider}' returned EMPTY or FAILED. Triggering Deep Fallback to Ollama...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
answer_text = await llm.generate_raw_response(
|
||||||
|
prompt=final_prompt,
|
||||||
|
system=system_prompt,
|
||||||
|
priority="realtime",
|
||||||
|
provider="ollama",
|
||||||
|
max_retries=0
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"🛑 [{query_id}] Deep Fallback to Ollama also failed: {e}")
|
||||||
|
answer_text = "Entschuldigung, das System ist aktuell überlastet. Bitte versuche es in einem Moment erneut."
|
||||||
|
|
||||||
duration_ms = int((time.time() - start_time) * 1000)
|
duration_ms = int((time.time() - start_time) * 1000)
|
||||||
|
|
||||||
@@ -332,7 +376,7 @@ async def chat_endpoint(
|
|||||||
query_text=request.message,
|
query_text=request.message,
|
||||||
results=sources_hits,
|
results=sources_hits,
|
||||||
mode="interview" if intent == "INTERVIEW" else "chat_rag",
|
mode="interview" if intent == "INTERVIEW" else "chat_rag",
|
||||||
metadata={"intent": intent, "source": intent_source}
|
metadata={"intent": intent, "source": intent_source, "provider": preferred_provider}
|
||||||
)
|
)
|
||||||
except: pass
|
except: pass
|
||||||
|
|
||||||
@@ -347,4 +391,5 @@ async def chat_endpoint(
|
|||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error in chat endpoint: {e}", exc_info=True)
|
logger.error(f"Error in chat endpoint: {e}", exc_info=True)
|
||||||
raise HTTPException(status_code=500, detail=str(e))
|
# Wir geben eine benutzerfreundliche Meldung zurück, statt nur den Error-Stack
|
||||||
|
raise HTTPException(status_code=500, detail="Das System konnte die Anfrage nicht verarbeiten.")
|
||||||
@@ -1,5 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
Version 0.1
|
FILE: app/routers/embed_router.py
|
||||||
|
DESCRIPTION: Exponiert die lokale Embedding-Funktion als API-Endpunkt.
|
||||||
|
VERSION: 0.1.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: app.embeddings, pydantic
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/feedback.py
|
FILE: app/routers/feedback.py
|
||||||
Endpunkt für User-Feedback (WP-04c).
|
DESCRIPTION: Endpunkt für explizites User-Feedback (WP-04c).
|
||||||
|
VERSION: 0.1.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: app.models.dto, app.services.feedback_service
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
"""
|
"""
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from app.models.dto import FeedbackRequest
|
from app.models.dto import FeedbackRequest
|
||||||
|
|||||||
+7
-18
@@ -1,21 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/graph.py — Graph-Endpunkte (WP-04)
|
FILE: app/routers/graph.py
|
||||||
|
DESCRIPTION: Liefert Graph-Daten (Knoten/Kanten) für UI-Visualisierungen basierend auf einer Seed-ID. (WP4)
|
||||||
Zweck:
|
VERSION: 0.1.0
|
||||||
Liefert die Nachbarschaft einer Note/ID als JSON-Graph (Nodes/Edges/Stats).
|
STATUS: Active
|
||||||
Kompatibilität:
|
DEPENDENCIES: qdrant_client, app.models.dto, app.core.graph_adapter, app.config
|
||||||
Python 3.12+, FastAPI 0.110+, qdrant-client 1.x
|
LAST_ANALYSIS: 2025-12-15
|
||||||
Version:
|
|
||||||
0.1.0 (Erstanlage)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Bezug:
|
|
||||||
- app/core/graph_adapter.py
|
|
||||||
- app/models/dto.py
|
|
||||||
Nutzung:
|
|
||||||
app.include_router(graph.router, prefix="/graph", tags=["graph"])
|
|
||||||
Änderungsverlauf:
|
|
||||||
0.1.0 (2025-10-07) – Erstanlage.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -23,7 +12,7 @@ from typing import List, Optional
|
|||||||
from fastapi import APIRouter, Query
|
from fastapi import APIRouter, Query
|
||||||
from qdrant_client import QdrantClient
|
from qdrant_client import QdrantClient
|
||||||
from app.models.dto import GraphResponse, NodeDTO, EdgeDTO
|
from app.models.dto import GraphResponse, NodeDTO, EdgeDTO
|
||||||
from app.core.graph_adapter import expand
|
from app.core.graph.graph_subgraph import expand
|
||||||
from app.config import get_settings
|
from app.config import get_settings
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|||||||
+61
-25
@@ -1,12 +1,18 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/ingest.py
|
FILE: app/routers/ingest.py
|
||||||
API-Endpunkte für WP-11 (Discovery & Persistence).
|
DESCRIPTION: Endpunkte für WP-11. Nimmt Markdown entgegen.
|
||||||
Delegiert an Services.
|
Refactored für WP-14: Nutzt BackgroundTasks für non-blocking Save.
|
||||||
|
Update WP-20: Unterstützung für Hybrid-Cloud-Analyse Feedback.
|
||||||
|
VERSION: 0.8.0 (WP-20 Hybrid Ready)
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: app.core.ingestion, app.services.discovery, fastapi, pydantic
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import time
|
import time
|
||||||
import logging
|
import logging
|
||||||
from fastapi import APIRouter, HTTPException
|
import asyncio
|
||||||
|
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
from typing import Optional, Dict, Any
|
from typing import Optional, Dict, Any
|
||||||
|
|
||||||
@@ -16,7 +22,7 @@ from app.services.discovery import DiscoveryService
|
|||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
# Services Init (Global oder via Dependency Injection)
|
# Services Init
|
||||||
discovery_service = DiscoveryService()
|
discovery_service = DiscoveryService()
|
||||||
|
|
||||||
class AnalyzeRequest(BaseModel):
|
class AnalyzeRequest(BaseModel):
|
||||||
@@ -32,7 +38,33 @@ class SaveResponse(BaseModel):
|
|||||||
status: str
|
status: str
|
||||||
file_path: str
|
file_path: str
|
||||||
note_id: str
|
note_id: str
|
||||||
stats: Dict[str, Any]
|
message: str # Neu für UX Feedback
|
||||||
|
stats: Dict[str, Any] # Kann leer sein bei async processing
|
||||||
|
|
||||||
|
# --- Background Task Wrapper ---
|
||||||
|
async def run_ingestion_task(markdown_content: str, filename: str, vault_root: str, folder: str):
|
||||||
|
"""
|
||||||
|
Führt die Ingestion im Hintergrund aus, damit der Request nicht blockiert.
|
||||||
|
Integrierter WP-20 Hybrid-Modus über den IngestionService.
|
||||||
|
"""
|
||||||
|
logger.info(f"🔄 Background Task started: Ingesting {filename}...")
|
||||||
|
try:
|
||||||
|
ingest_service = IngestionService()
|
||||||
|
result = await ingest_service.create_from_text(
|
||||||
|
markdown_content=markdown_content,
|
||||||
|
filename=filename,
|
||||||
|
vault_root=vault_root,
|
||||||
|
folder=folder
|
||||||
|
)
|
||||||
|
# Hier könnte man später Notification-Services (Websockets) triggern
|
||||||
|
if result.get("status") == "error":
|
||||||
|
logger.error(f"❌ Background Ingestion Error for {filename}: {result.get('error')}")
|
||||||
|
else:
|
||||||
|
logger.info(f"✅ Background Task finished: {filename} ({result.get('chunks_count')} Chunks)")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Critical Background Task Failure: {e}", exc_info=True)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/analyze")
|
@router.post("/analyze")
|
||||||
async def analyze_draft(req: AnalyzeRequest):
|
async def analyze_draft(req: AnalyzeRequest):
|
||||||
@@ -40,7 +72,6 @@ async def analyze_draft(req: AnalyzeRequest):
|
|||||||
WP-11 Intelligence: Liefert Link-Vorschläge via DiscoveryService.
|
WP-11 Intelligence: Liefert Link-Vorschläge via DiscoveryService.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
# Hier rufen wir jetzt den verbesserten Service auf
|
|
||||||
result = await discovery_service.analyze_draft(req.text, req.type)
|
result = await discovery_service.analyze_draft(req.text, req.type)
|
||||||
return result
|
return result
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -48,42 +79,47 @@ async def analyze_draft(req: AnalyzeRequest):
|
|||||||
return {"suggestions": [], "error": str(e)}
|
return {"suggestions": [], "error": str(e)}
|
||||||
|
|
||||||
@router.post("/save", response_model=SaveResponse)
|
@router.post("/save", response_model=SaveResponse)
|
||||||
async def save_note(req: SaveRequest):
|
async def save_note(req: SaveRequest, background_tasks: BackgroundTasks):
|
||||||
"""
|
"""
|
||||||
WP-11 Persistence: Speichert und indiziert.
|
WP-14 Fix: Startet Ingestion im Hintergrund (Fire & Forget).
|
||||||
|
Verhindert Timeouts bei aktiver Smart-Edge-Allocation (WP-15) und Cloud-Hybrid-Modus (WP-20).
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
vault_root = os.getenv("MINDNET_VAULT_ROOT", "./vault")
|
vault_root = os.getenv("MINDNET_VAULT_ROOT", "./vault")
|
||||||
abs_vault_root = os.path.abspath(vault_root)
|
abs_vault_root = os.path.abspath(vault_root)
|
||||||
|
|
||||||
if not os.path.exists(abs_vault_root):
|
if not os.path.exists(abs_vault_root):
|
||||||
try: os.makedirs(abs_vault_root, exist_ok=True)
|
try:
|
||||||
except: pass
|
os.makedirs(abs_vault_root, exist_ok=True)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Could not create vault root: {e}")
|
||||||
|
|
||||||
final_filename = req.filename or f"draft_{int(time.time())}.md"
|
final_filename = req.filename or f"draft_{int(time.time())}.md"
|
||||||
ingest_service = IngestionService()
|
|
||||||
|
|
||||||
# Async Call
|
# Wir geben sofort eine ID zurück (optimistisch),
|
||||||
result = await ingest_service.create_from_text(
|
# auch wenn die echte ID erst nach dem Parsing feststeht.
|
||||||
|
# Für UI-Feedback nutzen wir den Filename.
|
||||||
|
|
||||||
|
# Task in die Queue schieben
|
||||||
|
background_tasks.add_task(
|
||||||
|
run_ingestion_task,
|
||||||
markdown_content=req.markdown_content,
|
markdown_content=req.markdown_content,
|
||||||
filename=final_filename,
|
filename=final_filename,
|
||||||
vault_root=abs_vault_root,
|
vault_root=abs_vault_root,
|
||||||
folder=req.folder
|
folder=req.folder
|
||||||
)
|
)
|
||||||
|
|
||||||
if result.get("status") == "error":
|
|
||||||
raise HTTPException(status_code=500, detail=result.get("error"))
|
|
||||||
|
|
||||||
return SaveResponse(
|
return SaveResponse(
|
||||||
status="success",
|
status="queued",
|
||||||
file_path=result.get("path", "unknown"),
|
file_path=os.path.join(req.folder, final_filename),
|
||||||
note_id=result.get("note_id", "unknown"),
|
note_id="pending",
|
||||||
|
message="Speicherung & Hybrid-KI-Analyse (WP-20) im Hintergrund gestartet.",
|
||||||
stats={
|
stats={
|
||||||
"chunks": result.get("chunks_count", 0),
|
"chunks": -1, # Indikator für Async
|
||||||
"edges": result.get("edges_count", 0)
|
"edges": -1
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
except HTTPException as he: raise he
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Save failed: {e}", exc_info=True)
|
logger.error(f"Save dispatch failed: {e}", exc_info=True)
|
||||||
raise HTTPException(status_code=500, detail=f"Save failed: {str(e)}")
|
raise HTTPException(status_code=500, detail=f"Save dispatch failed: {str(e)}")
|
||||||
@@ -1,160 +0,0 @@
|
|||||||
"""
|
|
||||||
Version 0.1
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any, Optional, List
|
|
||||||
import uuid
|
|
||||||
|
|
||||||
from fastapi import APIRouter
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from qdrant_client import QdrantClient
|
|
||||||
from qdrant_client.http.models import (
|
|
||||||
Distance,
|
|
||||||
VectorParams,
|
|
||||||
PointStruct,
|
|
||||||
Filter,
|
|
||||||
FieldCondition,
|
|
||||||
MatchValue,
|
|
||||||
)
|
|
||||||
|
|
||||||
from ..config import get_settings
|
|
||||||
from ..embeddings import embed_texts
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/qdrant", tags=["qdrant"])
|
|
||||||
|
|
||||||
def _client() -> QdrantClient:
|
|
||||||
s = get_settings()
|
|
||||||
return QdrantClient(url=s.QDRANT_URL, api_key=s.QDRANT_API_KEY)
|
|
||||||
|
|
||||||
def _col(name: str) -> str:
|
|
||||||
return f"{get_settings().COLLECTION_PREFIX}_{name}"
|
|
||||||
|
|
||||||
def _uuid5(s: str) -> str:
|
|
||||||
"""Deterministic UUIDv5 from arbitrary string (server-side point id)."""
|
|
||||||
return str(uuid.uuid5(uuid.NAMESPACE_URL, s))
|
|
||||||
|
|
||||||
# --- Models ---
|
|
||||||
class BaseMeta(BaseModel):
|
|
||||||
note_id: str = Field(..., description="Stable ID of the note (e.g., hash of vault-relative path)")
|
|
||||||
title: Optional[str] = Field(None, description="Note or chunk title")
|
|
||||||
path: Optional[str] = Field(None, description="Vault-relative path to the .md file")
|
|
||||||
Typ: Optional[str] = None
|
|
||||||
Status: Optional[str] = None
|
|
||||||
tags: Optional[List[str]] = None
|
|
||||||
Rolle: Optional[List[str]] = None # allow list
|
|
||||||
|
|
||||||
class UpsertChunkRequest(BaseMeta):
|
|
||||||
chunk_id: str = Field(..., description="Stable ID of the chunk within the note")
|
|
||||||
text: str = Field(..., description="Chunk text content")
|
|
||||||
links: Optional[List[str]] = Field(default=None, description="Outbound links detected in the chunk")
|
|
||||||
|
|
||||||
class UpsertNoteRequest(BaseMeta):
|
|
||||||
text: Optional[str] = Field(None, description="Full note text (optional)")
|
|
||||||
|
|
||||||
class UpsertEdgeRequest(BaseModel):
|
|
||||||
src_note_id: str
|
|
||||||
dst_note_id: Optional[str] = None
|
|
||||||
src_chunk_id: Optional[str] = None
|
|
||||||
dst_chunk_id: Optional[str] = None
|
|
||||||
relation: str = Field(default="links_to")
|
|
||||||
link_text: Optional[str] = None
|
|
||||||
|
|
||||||
class QueryRequest(BaseModel):
|
|
||||||
query: str
|
|
||||||
limit: int = 5
|
|
||||||
note_id: Optional[str] = None
|
|
||||||
path: Optional[str] = None
|
|
||||||
tags: Optional[List[str]] = None
|
|
||||||
|
|
||||||
# --- Helpers ---
|
|
||||||
def _ensure_collections():
|
|
||||||
s = get_settings()
|
|
||||||
cli = _client()
|
|
||||||
# chunks
|
|
||||||
try:
|
|
||||||
cli.get_collection(_col("chunks"))
|
|
||||||
except Exception:
|
|
||||||
cli.recreate_collection(_col("chunks"), vectors_config=VectorParams(size=s.VECTOR_SIZE, distance=Distance.COSINE))
|
|
||||||
# notes
|
|
||||||
try:
|
|
||||||
cli.get_collection(_col("notes"))
|
|
||||||
except Exception:
|
|
||||||
cli.recreate_collection(_col("notes"), vectors_config=VectorParams(size=s.VECTOR_SIZE, distance=Distance.COSINE))
|
|
||||||
# edges (dummy vector of size 1)
|
|
||||||
try:
|
|
||||||
cli.get_collection(_col("edges"))
|
|
||||||
except Exception:
|
|
||||||
cli.recreate_collection(_col("edges"), vectors_config=VectorParams(size=1, distance=Distance.COSINE))
|
|
||||||
|
|
||||||
@router.post("/upsert_chunk", summary="Upsert a chunk into mindnet_chunks")
|
|
||||||
def upsert_chunk(req: UpsertChunkRequest) -> dict:
|
|
||||||
_ensure_collections()
|
|
||||||
cli = _client()
|
|
||||||
vec = embed_texts([req.text])[0]
|
|
||||||
payload: dict[str, Any] = req.model_dump()
|
|
||||||
payload.pop("text", None)
|
|
||||||
payload["preview"] = (req.text[:240] + "…") if len(req.text) > 240 else req.text
|
|
||||||
qdrant_id = _uuid5(f"chunk:{req.chunk_id}")
|
|
||||||
pt = PointStruct(id=qdrant_id, vector=vec, payload=payload)
|
|
||||||
cli.upsert(collection_name=_col("chunks"), points=[pt])
|
|
||||||
return {"status": "ok", "id": qdrant_id}
|
|
||||||
|
|
||||||
@router.post("/upsert_note", summary="Upsert a note into mindnet_notes")
|
|
||||||
def upsert_note(req: UpsertNoteRequest) -> dict:
|
|
||||||
_ensure_collections()
|
|
||||||
cli = _client()
|
|
||||||
text_for_embedding = req.text if req.text else (req.title or req.note_id)
|
|
||||||
vec = embed_texts([text_for_embedding])[0]
|
|
||||||
payload: dict[str, Any] = req.model_dump()
|
|
||||||
payload.pop("text", None)
|
|
||||||
qdrant_id = _uuid5(f"note:{req.note_id}")
|
|
||||||
pt = PointStruct(id=qdrant_id, vector=vec, payload=payload)
|
|
||||||
cli.upsert(collection_name=_col("notes"), points=[pt])
|
|
||||||
return {"status": "ok", "id": qdrant_id}
|
|
||||||
|
|
||||||
@router.post("/upsert_edge", summary="Upsert a graph edge into mindnet_edges")
|
|
||||||
def upsert_edge(req: UpsertEdgeRequest) -> dict:
|
|
||||||
_ensure_collections()
|
|
||||||
cli = _client()
|
|
||||||
payload = req.model_dump()
|
|
||||||
vec = [0.0]
|
|
||||||
raw_edge_id = f"{req.src_note_id}|{req.src_chunk_id or ''}->{req.dst_note_id or ''}|{req.dst_chunk_id or ''}|{req.relation}"
|
|
||||||
qdrant_id = _uuid5(f"edge:{raw_edge_id}")
|
|
||||||
pt = PointStruct(id=qdrant_id, vector=vec, payload=payload)
|
|
||||||
cli.upsert(collection_name=_col("edges"), points=[pt])
|
|
||||||
return {"status": "ok", "id": qdrant_id}
|
|
||||||
|
|
||||||
@router.post("/query", summary="Vector query over mindnet_chunks with optional filters")
|
|
||||||
def query(req: QueryRequest) -> dict:
|
|
||||||
_ensure_collections()
|
|
||||||
cli = _client()
|
|
||||||
vec = embed_texts([req.query])[0]
|
|
||||||
|
|
||||||
flt: Optional[Filter] = None
|
|
||||||
conds = []
|
|
||||||
if req.note_id:
|
|
||||||
conds.append(FieldCondition(key="note_id", match=MatchValue(value=req.note_id)))
|
|
||||||
if req.path:
|
|
||||||
conds.append(FieldCondition(key="path", match=MatchValue(value=req.path)))
|
|
||||||
if req.tags:
|
|
||||||
for t in req.tags:
|
|
||||||
conds.append(FieldCondition(key="tags", match=MatchValue(value=t)))
|
|
||||||
if conds:
|
|
||||||
flt = Filter(must=conds)
|
|
||||||
|
|
||||||
res = cli.search(collection_name=_col("chunks"), query_vector=vec, limit=req.limit, with_payload=True, with_vectors=False, query_filter=flt)
|
|
||||||
hits = []
|
|
||||||
for p in res:
|
|
||||||
pl = p.payload or {}
|
|
||||||
hits.append({
|
|
||||||
"chunk_id": p.id,
|
|
||||||
"score": p.score,
|
|
||||||
"note_id": pl.get("note_id"),
|
|
||||||
"title": pl.get("title"),
|
|
||||||
"path": pl.get("path"),
|
|
||||||
"preview": pl.get("preview"),
|
|
||||||
"tags": pl.get("tags"),
|
|
||||||
})
|
|
||||||
return {"results": hits}
|
|
||||||
+8
-19
@@ -1,27 +1,16 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/query.py — Query-Endpunkte (WP-04)
|
FILE: app/routers/query.py
|
||||||
|
DESCRIPTION: Klassische Such-Endpunkte (Semantic & Hybrid). Initiiert asynchrones Feedback-Logging und ruft den richtigen Retriever Modus auf
|
||||||
|
VERSION: 0.2.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: app.models.dto, app.core.retriever, app.services.feedback_service
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
"""
|
||||||
|
|
||||||
Zweck:
|
|
||||||
Stellt POST /query bereit und ruft den passenden Retriever-Modus auf.
|
|
||||||
Kompatibilität:
|
|
||||||
Python 3.12+, FastAPI 0.110+
|
|
||||||
Version:
|
|
||||||
0.1.0 (Erstanlage)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Bezug:
|
|
||||||
- app/core/retriever.py
|
|
||||||
- app/models/dto.py
|
|
||||||
Nutzung:
|
|
||||||
app.include_router(query.router, prefix="/query", tags=["query"])
|
|
||||||
Änderungsverlauf:
|
|
||||||
0.2.0 (2025-12-07) - Update für WP04c Feedback
|
|
||||||
0.1.0 (2025-10-07) – Erstanlage.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||||
from app.models.dto import QueryRequest, QueryResponse
|
from app.models.dto import QueryRequest, QueryResponse
|
||||||
from app.core.retriever import hybrid_retrieve, semantic_retrieve
|
from app.core.retrieval.retriever import hybrid_retrieve, semantic_retrieve
|
||||||
# NEU:
|
# NEU:
|
||||||
from app.services.feedback_service import log_search
|
from app.services.feedback_service import log_search
|
||||||
|
|
||||||
|
|||||||
+6
-17
@@ -1,21 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/routers/tools.py — Tool-Definitionen für Ollama/n8n/MCP (read-only)
|
FILE: app/routers/tools.py
|
||||||
|
DESCRIPTION: Liefert JSON-Schemas für die Integration als 'Tools' in Agents (Ollama/OpenAI). Read-Only.
|
||||||
Zweck:
|
VERSION: 0.1.1
|
||||||
Liefert Funktions-Schemas (OpenAI-/Ollama-kompatibles Tool-JSON) für:
|
STATUS: Active
|
||||||
- mindnet_query -> POST /query
|
DEPENDENCIES: fastapi
|
||||||
- mindnet_subgraph -> GET /graph/{note_id}
|
LAST_ANALYSIS: 2025-12-15
|
||||||
Kompatibilität:
|
|
||||||
Python 3.12+, FastAPI 0.110+
|
|
||||||
Version:
|
|
||||||
0.1.1 (query ODER query_vector möglich)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Nutzung:
|
|
||||||
app.include_router(tools.router, prefix="/tools", tags=["tools"])
|
|
||||||
Änderungsverlauf:
|
|
||||||
0.1.1 (2025-10-07) – mindnet_query: oneOf(query, query_vector).
|
|
||||||
0.1.0 (2025-10-07) – Erstanlage.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|||||||
@@ -1,12 +1,11 @@
|
|||||||
"""
|
"""
|
||||||
app/services/discovery.py
|
FILE: app/services/discovery.py
|
||||||
Service für Link-Vorschläge und Knowledge-Discovery (WP-11).
|
DESCRIPTION: Service für WP-11. Analysiert Texte, findet Entitäten und schlägt typisierte Verbindungen vor ("Matrix-Logic").
|
||||||
|
VERSION: 0.6.0
|
||||||
Features:
|
STATUS: Active
|
||||||
- Sliding Window Analyse für lange Texte.
|
DEPENDENCIES: app.core.qdrant, app.models.dto, app.core.retriever
|
||||||
- Footer-Scan für Projekt-Referenzen.
|
EXTERNAL_CONFIG: config/types.yaml
|
||||||
- 'Matrix-Logic' für intelligente Kanten-Typen (Experience -> Value = based_on).
|
LAST_ANALYSIS: 2025-12-15
|
||||||
- Async & Nomic-Embeddings kompatibel.
|
|
||||||
"""
|
"""
|
||||||
import logging
|
import logging
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -14,9 +13,9 @@ import os
|
|||||||
from typing import List, Dict, Any, Optional, Set
|
from typing import List, Dict, Any, Optional, Set
|
||||||
import yaml
|
import yaml
|
||||||
|
|
||||||
from app.core.qdrant import QdrantConfig, get_client
|
from app.core.database.qdrant import QdrantConfig, get_client
|
||||||
from app.models.dto import QueryRequest
|
from app.models.dto import QueryRequest
|
||||||
from app.core.retriever import hybrid_retrieve
|
from app.core.retrieval.retriever import hybrid_retrieve
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,165 @@
|
|||||||
|
"""
|
||||||
|
FILE: app/services/edge_registry.py
|
||||||
|
DESCRIPTION: Single Source of Truth für Kanten-Typen mit dynamischem Reload.
|
||||||
|
WP-15b: Erweiterte Provenance-Prüfung für die Candidate-Validation.
|
||||||
|
Sichert die Graph-Integrität durch strikte Trennung von System- und Inhaltskanten.
|
||||||
|
WP-22: Fix für absolute Pfade außerhalb des Vaults (Prod-Dictionary).
|
||||||
|
WP-20: Synchronisation mit zentralen Settings (v0.6.2).
|
||||||
|
VERSION: 0.8.0
|
||||||
|
STATUS: Active
|
||||||
|
DEPENDENCIES: re, os, json, logging, time, app.config
|
||||||
|
LAST_ANALYSIS: 2025-12-26
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from typing import Dict, Optional, Set, Tuple
|
||||||
|
|
||||||
|
from app.config import get_settings
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class EdgeRegistry:
|
||||||
|
"""
|
||||||
|
Zentraler Verwalter für das Kanten-Vokabular.
|
||||||
|
Implementiert das Singleton-Pattern für konsistente Validierung über alle Services.
|
||||||
|
"""
|
||||||
|
_instance = None
|
||||||
|
# System-Kanten, die nicht durch User oder KI gesetzt werden dürfen
|
||||||
|
FORBIDDEN_SYSTEM_EDGES = {"next", "prev", "belongs_to"}
|
||||||
|
|
||||||
|
def __new__(cls, *args, **kwargs):
|
||||||
|
if cls._instance is None:
|
||||||
|
cls._instance = super(EdgeRegistry, cls).__new__(cls)
|
||||||
|
cls._instance.initialized = False
|
||||||
|
return cls._instance
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
if self.initialized:
|
||||||
|
return
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
|
||||||
|
# 1. Pfad aus den zentralen Settings laden (WP-20 Synchronisation)
|
||||||
|
# Priorisiert den Pfad aus der .env / config.py (v0.6.2)
|
||||||
|
self.full_vocab_path = os.path.abspath(settings.MINDNET_VOCAB_PATH)
|
||||||
|
|
||||||
|
self.unknown_log_path = "data/logs/unknown_edges.jsonl"
|
||||||
|
self.canonical_map: Dict[str, str] = {}
|
||||||
|
self.valid_types: Set[str] = set()
|
||||||
|
self._last_mtime = 0.0
|
||||||
|
|
||||||
|
# Initialer Ladevorgang
|
||||||
|
logger.info(f">>> [EDGE-REGISTRY] Initializing with Path: {self.full_vocab_path}")
|
||||||
|
self.ensure_latest()
|
||||||
|
self.initialized = True
|
||||||
|
|
||||||
|
def ensure_latest(self):
|
||||||
|
"""
|
||||||
|
Prüft den Zeitstempel der Vokabular-Datei und lädt bei Bedarf neu.
|
||||||
|
Verhindert Inkonsistenzen bei Laufzeit-Updates des Dictionaries.
|
||||||
|
"""
|
||||||
|
if not os.path.exists(self.full_vocab_path):
|
||||||
|
logger.error(f"!!! [EDGE-REGISTRY ERROR] File not found: {self.full_vocab_path} !!!")
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
current_mtime = os.path.getmtime(self.full_vocab_path)
|
||||||
|
if current_mtime > self._last_mtime:
|
||||||
|
self._load_vocabulary()
|
||||||
|
self._last_mtime = current_mtime
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"!!! [EDGE-REGISTRY] Error checking file time: {e}")
|
||||||
|
|
||||||
|
def _load_vocabulary(self):
|
||||||
|
"""
|
||||||
|
Parst das Markdown-Wörterbuch und baut die Canonical-Map auf.
|
||||||
|
Erkennt Tabellen-Strukturen und extrahiert fettgedruckte System-Typen.
|
||||||
|
"""
|
||||||
|
self.canonical_map.clear()
|
||||||
|
self.valid_types.clear()
|
||||||
|
|
||||||
|
# Regex für Tabellen-Struktur: | **Typ** | Aliase |
|
||||||
|
pattern = re.compile(r"\|\s*\*\*`?([a-zA-Z0-9_-]+)`?\*\*\s*\|\s*([^|]+)\|")
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(self.full_vocab_path, "r", encoding="utf-8") as f:
|
||||||
|
c_types, c_aliases = 0, 0
|
||||||
|
for line in f:
|
||||||
|
match = pattern.search(line)
|
||||||
|
if match:
|
||||||
|
canonical = match.group(1).strip().lower()
|
||||||
|
aliases_str = match.group(2).strip()
|
||||||
|
|
||||||
|
self.valid_types.add(canonical)
|
||||||
|
self.canonical_map[canonical] = canonical
|
||||||
|
c_types += 1
|
||||||
|
|
||||||
|
if aliases_str and "Kein Alias" not in aliases_str:
|
||||||
|
aliases = [a.strip() for a in aliases_str.split(",") if a.strip()]
|
||||||
|
for alias in aliases:
|
||||||
|
# Normalisierung: Kleinschreibung, Underscores statt Leerzeichen
|
||||||
|
clean_alias = alias.replace("`", "").lower().strip().replace(" ", "_")
|
||||||
|
self.canonical_map[clean_alias] = canonical
|
||||||
|
c_aliases += 1
|
||||||
|
|
||||||
|
logger.info(f"=== [EDGE-REGISTRY SUCCESS] Loaded {c_types} Canonical Types and {c_aliases} Aliases ===")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"!!! [EDGE-REGISTRY FATAL] Error reading file: {e} !!!")
|
||||||
|
|
||||||
|
def resolve(self, edge_type: str, provenance: str = "explicit", context: dict = None) -> str:
|
||||||
|
"""
|
||||||
|
WP-15b: Validiert einen Kanten-Typ gegen das Vokabular und prüft Berechtigungen.
|
||||||
|
Sichert, dass nur strukturelle Prozesse System-Kanten setzen dürfen.
|
||||||
|
"""
|
||||||
|
self.ensure_latest()
|
||||||
|
if not edge_type:
|
||||||
|
return "related_to"
|
||||||
|
|
||||||
|
# Normalisierung des Typs
|
||||||
|
clean_type = edge_type.lower().strip().replace(" ", "_").replace("-", "_")
|
||||||
|
ctx = context or {}
|
||||||
|
|
||||||
|
# WP-15b: System-Kanten dürfen weder manuell noch durch KI/Vererbung gesetzt werden.
|
||||||
|
# Nur Provenienz 'structure' (interne Prozesse) ist autorisiert.
|
||||||
|
# Wir blockieren hier alle Provenienzen außer 'structure'.
|
||||||
|
restricted_provenance = ["explicit", "semantic_ai", "inherited", "global_pool", "rule"]
|
||||||
|
if provenance in restricted_provenance and clean_type in self.FORBIDDEN_SYSTEM_EDGES:
|
||||||
|
self._log_issue(clean_type, f"forbidden_usage_by_{provenance}", ctx)
|
||||||
|
return "related_to"
|
||||||
|
|
||||||
|
# System-Kanten sind NUR bei struktureller Provenienz erlaubt
|
||||||
|
if provenance == "structure" and clean_type in self.FORBIDDEN_SYSTEM_EDGES:
|
||||||
|
return clean_type
|
||||||
|
|
||||||
|
# Mapping auf kanonischen Namen (Alias-Auflösung)
|
||||||
|
if clean_type in self.canonical_map:
|
||||||
|
return self.canonical_map[clean_type]
|
||||||
|
|
||||||
|
# Fallback und Logging unbekannter Typen für Admin-Review
|
||||||
|
self._log_issue(clean_type, "unknown_type", ctx)
|
||||||
|
return clean_type
|
||||||
|
|
||||||
|
def _log_issue(self, edge_type: str, error_kind: str, ctx: dict):
|
||||||
|
"""Detailliertes JSONL-Logging für die Vokabular-Optimierung."""
|
||||||
|
try:
|
||||||
|
os.makedirs(os.path.dirname(self.unknown_log_path), exist_ok=True)
|
||||||
|
entry = {
|
||||||
|
"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||||
|
"edge_type": edge_type,
|
||||||
|
"error": error_kind,
|
||||||
|
"file": ctx.get("file", "unknown"),
|
||||||
|
"line": ctx.get("line", "unknown"),
|
||||||
|
"note_id": ctx.get("note_id", "unknown"),
|
||||||
|
"provenance": ctx.get("provenance", "unknown")
|
||||||
|
}
|
||||||
|
with open(self.unknown_log_path, "a", encoding="utf-8") as f:
|
||||||
|
f.write(json.dumps(entry) + "\n")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Singleton Export für systemweiten Zugriff
|
||||||
|
registry = EdgeRegistry()
|
||||||
@@ -1,12 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/services/embeddings_client.py — Text→Embedding Service
|
FILE: app/services/embeddings_client.py
|
||||||
|
DESCRIPTION: Unified Embedding Client. Nutzt Ollama API (HTTP). Ersetzt lokale sentence-transformers.
|
||||||
Zweck:
|
VERSION: 2.5.0
|
||||||
Einheitlicher Client für Embeddings via Ollama (Nomic).
|
STATUS: Active
|
||||||
Stellt sicher, dass sowohl Async (Ingestion) als auch Sync (Retriever)
|
DEPENDENCIES: httpx, requests, app.config
|
||||||
denselben Vektorraum (768 Dim) nutzen.
|
LAST_ANALYSIS: 2025-12-15
|
||||||
|
|
||||||
Version: 2.5.0 (Unified Ollama)
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import os
|
import os
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
"""
|
"""
|
||||||
app/services/feedback_service.py
|
FILE: app/services/feedback_service.py
|
||||||
Service zum Loggen von Suchanfragen und Feedback (WP-04c).
|
DESCRIPTION: Schreibt Search- und Feedback-Logs in JSONL-Dateien.
|
||||||
Speichert Daten als JSONL für späteres Self-Tuning (WP-08).
|
VERSION: 1.1
|
||||||
|
STATUS: Active
|
||||||
Version: 1.1 (Chat-Support)
|
DEPENDENCIES: app.models.dto
|
||||||
|
LAST_ANALYSIS: 2025-12-15
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
|||||||
@@ -1,88 +0,0 @@
|
|||||||
"""
|
|
||||||
app/services/llm_ollama.py — Ollama-Integration & Prompt-Bau (WP-04)
|
|
||||||
|
|
||||||
Zweck:
|
|
||||||
Prompt-Template & (optionaler) lokaler Aufruf von Ollama. Der Aufruf ist
|
|
||||||
bewusst gekapselt und kann gefahrlos deaktiviert bleiben, bis ihr ein
|
|
||||||
konkretes Modell konfigurieren wollt.
|
|
||||||
Kompatibilität:
|
|
||||||
Python 3.12+
|
|
||||||
Version:
|
|
||||||
0.1.0 (Erstanlage)
|
|
||||||
Stand:
|
|
||||||
2025-10-07
|
|
||||||
Bezug:
|
|
||||||
WP-04/05 Kontextbereitstellung für LLM
|
|
||||||
Nutzung:
|
|
||||||
from app.services.llm_ollama import build_prompt, call_ollama
|
|
||||||
Änderungsverlauf:
|
|
||||||
0.1.0 (2025-10-07) – Erstanlage.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
from typing import List, Dict, Optional
|
|
||||||
import subprocess
|
|
||||||
import json
|
|
||||||
|
|
||||||
PROMPT_TEMPLATE = """System: You are a helpful expert.
|
|
||||||
User: {question}
|
|
||||||
|
|
||||||
Context (ranked):
|
|
||||||
{contexts}
|
|
||||||
|
|
||||||
Task: Answer precisely. At the end, list sources (note title + section) and important edge paths.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
def build_context_block(items: List[Dict]) -> str:
|
|
||||||
"""Formatiert Top-K-Kontexte (Chunks) für den Prompt."""
|
|
||||||
lines = []
|
|
||||||
for i, it in enumerate(items, 1):
|
|
||||||
note = it.get("note_title", "") or it.get("note_id", "")
|
|
||||||
sec = it.get("section", "") or it.get("section_title", "")
|
|
||||||
sc = it.get("score", 0)
|
|
||||||
txt = it.get("text", "") or it.get("body", "") or ""
|
|
||||||
lines.append(f"{i}) {note} — {sec} [score={sc:.2f}]\n{txt}\n")
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|
||||||
|
|
||||||
def build_prompt(question: str, contexts: List[Dict]) -> str:
|
|
||||||
"""Setzt Frage + Kontexte in ein konsistentes Template."""
|
|
||||||
return PROMPT_TEMPLATE.format(question=question, contexts=build_context_block(contexts))
|
|
||||||
|
|
||||||
|
|
||||||
def call_ollama(prompt: str, model: str = "llama3.1:8b", timeout_s: int = 120) -> Optional[str]:
|
|
||||||
"""
|
|
||||||
Optionaler lokaler Aufruf von `ollama run`.
|
|
||||||
Rückgabe: generierter Text oder None bei Fehler/Abbruch.
|
|
||||||
Hinweis: Nur nutzen, wenn Ollama lokal installiert/konfiguriert ist.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
proc = subprocess.run(
|
|
||||||
["ollama", "run", model],
|
|
||||||
input=prompt.encode("utf-8"),
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.PIPE,
|
|
||||||
timeout=timeout_s,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
out = proc.stdout.decode("utf-8", errors="replace")
|
|
||||||
# viele ollama Builds streamen JSON-Zeilen; robust extrahieren:
|
|
||||||
try:
|
|
||||||
# Falls JSONL, letztes "response" zusammenfassen
|
|
||||||
texts = []
|
|
||||||
for line in out.splitlines():
|
|
||||||
line = line.strip()
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
obj = json.loads(line)
|
|
||||||
if "response" in obj:
|
|
||||||
texts.append(obj["response"])
|
|
||||||
except Exception:
|
|
||||||
texts.append(line)
|
|
||||||
return "".join(texts).strip()
|
|
||||||
except Exception:
|
|
||||||
return out.strip()
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
+254
-79
@@ -1,142 +1,317 @@
|
|||||||
"""
|
"""
|
||||||
app/services/llm_service.py — LLM Client
|
FILE: app/services/llm_service.py
|
||||||
Version: 2.8.0 (Configurable Concurrency Limit)
|
DESCRIPTION: Hybrid-Client für Ollama, Google GenAI (Gemini) und OpenRouter.
|
||||||
|
Verwaltet provider-spezifische Prompts und Background-Last.
|
||||||
|
WP-20: Optimiertes Fallback-Management zum Schutz von Cloud-Quoten.
|
||||||
|
WP-20 Fix: Bulletproof Prompt-Auflösung für format() Aufrufe.
|
||||||
|
WP-22/JSON: Optionales JSON-Schema + strict (für OpenRouter structured outputs).
|
||||||
|
FIX: Intelligente Rate-Limit Erkennung (429 Handling), v1-API Sync & Timeouts.
|
||||||
|
VERSION: 3.3.9
|
||||||
|
STATUS: Active
|
||||||
|
FIX:
|
||||||
|
- Importiert clean_llm_text von app.core.registry zur Vermeidung von Circular Imports.
|
||||||
|
- Wendet clean_llm_text auf Text-Antworten in generate_raw_response an.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
import yaml
|
import yaml
|
||||||
import logging
|
import logging
|
||||||
import os
|
|
||||||
import asyncio
|
import asyncio
|
||||||
|
import json
|
||||||
|
from google import genai
|
||||||
|
from google.genai import types
|
||||||
|
from openai import AsyncOpenAI # Für OpenRouter (OpenAI-kompatibel)
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Optional, Dict, Any, Literal
|
from typing import Optional, Dict, Any, Literal
|
||||||
|
from app.config import get_settings
|
||||||
|
|
||||||
|
# ENTSCHEIDENDER FIX: Import der neutralen Bereinigungs-Logik (WP-14)
|
||||||
|
from app.core.registry import clean_llm_text
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
class Settings:
|
|
||||||
OLLAMA_URL = os.getenv("MINDNET_OLLAMA_URL", "http://127.0.0.1:11434")
|
|
||||||
LLM_TIMEOUT = float(os.getenv("MINDNET_LLM_TIMEOUT", 300.0))
|
|
||||||
LLM_MODEL = os.getenv("MINDNET_LLM_MODEL", "phi3:mini")
|
|
||||||
PROMPTS_PATH = os.getenv("MINDNET_PROMPTS_PATH", "./config/prompts.yaml")
|
|
||||||
|
|
||||||
# NEU: Konfigurierbares Limit für Hintergrund-Last
|
|
||||||
# Default auf 2 (konservativ), kann in .env erhöht werden.
|
|
||||||
BACKGROUND_LIMIT = int(os.getenv("MINDNET_LLM_BACKGROUND_LIMIT", "2"))
|
|
||||||
|
|
||||||
def get_settings():
|
|
||||||
return Settings()
|
|
||||||
|
|
||||||
class LLMService:
|
class LLMService:
|
||||||
# GLOBALER SEMAPHOR (Lazy Initialization)
|
# GLOBALER SEMAPHOR für Hintergrund-Last Steuerung (WP-06)
|
||||||
# Wir initialisieren ihn erst, wenn wir die Settings kennen.
|
|
||||||
_background_semaphore = None
|
_background_semaphore = None
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self.settings = get_settings()
|
self.settings = get_settings()
|
||||||
self.prompts = self._load_prompts()
|
self.prompts = self._load_prompts()
|
||||||
|
|
||||||
# Initialisiere Semaphore einmalig auf Klassen-Ebene basierend auf Config
|
# Initialisiere Semaphore einmalig auf Klassen-Ebene
|
||||||
if LLMService._background_semaphore is None:
|
if LLMService._background_semaphore is None:
|
||||||
limit = self.settings.BACKGROUND_LIMIT
|
limit = getattr(self.settings, "BACKGROUND_LIMIT", 2)
|
||||||
logger.info(f"🚦 LLMService: Initializing Background Semaphore with limit: {limit}")
|
logger.info(f"🚦 LLMService: Initializing Background Semaphore with limit: {limit}")
|
||||||
LLMService._background_semaphore = asyncio.Semaphore(limit)
|
LLMService._background_semaphore = asyncio.Semaphore(limit)
|
||||||
|
|
||||||
self.timeout = httpx.Timeout(self.settings.LLM_TIMEOUT, connect=10.0)
|
# 1. Lokaler Ollama Client
|
||||||
|
self.ollama_client = httpx.AsyncClient(
|
||||||
self.client = httpx.AsyncClient(
|
base_url=self.settings.OLLAMA_URL,
|
||||||
base_url=self.settings.OLLAMA_URL,
|
timeout=httpx.Timeout(self.settings.LLM_TIMEOUT)
|
||||||
timeout=self.timeout
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# 2. Google GenAI Client (Modern SDK)
|
||||||
|
self.google_client = None
|
||||||
|
if self.settings.GOOGLE_API_KEY:
|
||||||
|
# FIX: Wir erzwingen api_version 'v1' für höhere Stabilität bei 2.5er Modellen.
|
||||||
|
self.google_client = genai.Client(
|
||||||
|
api_key=self.settings.GOOGLE_API_KEY,
|
||||||
|
http_options={'api_version': 'v1'}
|
||||||
|
)
|
||||||
|
logger.info("✨ LLMService: Google GenAI (Gemini) active.")
|
||||||
|
|
||||||
|
# 3. OpenRouter Client
|
||||||
|
self.openrouter_client = None
|
||||||
|
if self.settings.OPENROUTER_API_KEY:
|
||||||
|
self.openrouter_client = AsyncOpenAI(
|
||||||
|
base_url="https://openrouter.ai/api/v1",
|
||||||
|
api_key=self.settings.OPENROUTER_API_KEY,
|
||||||
|
# Strikter Timeout für OpenRouter Free-Tier zur Vermeidung von Hangs.
|
||||||
|
timeout=45.0
|
||||||
|
)
|
||||||
|
logger.info("🛰️ LLMService: OpenRouter Integration active.")
|
||||||
|
|
||||||
def _load_prompts(self) -> dict:
|
def _load_prompts(self) -> dict:
|
||||||
|
"""Lädt die Prompt-Konfiguration aus der YAML-Datei."""
|
||||||
path = Path(self.settings.PROMPTS_PATH)
|
path = Path(self.settings.PROMPTS_PATH)
|
||||||
if not path.exists(): return {}
|
if not path.exists():
|
||||||
|
logger.error(f"❌ Prompts file not found at {path}")
|
||||||
|
return {}
|
||||||
try:
|
try:
|
||||||
with open(path, "r", encoding="utf-8") as f: return yaml.safe_load(f)
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
return yaml.safe_load(f) or {}
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Failed to load prompts: {e}")
|
logger.error(f"❌ Failed to load prompts: {e}")
|
||||||
return {}
|
return {}
|
||||||
|
|
||||||
|
def get_prompt(self, key: str, provider: str = None) -> str:
|
||||||
|
"""
|
||||||
|
Hole provider-spezifisches Template mit intelligenter Text-Kaskade.
|
||||||
|
HINWEIS: Dies ist nur ein Text-Lookup und verbraucht kein API-Kontingent.
|
||||||
|
Kaskade: Gewählter Provider -> Gemini (Cloud-Stil) -> Ollama (Basis-Stil).
|
||||||
|
"""
|
||||||
|
active_provider = provider or self.settings.MINDNET_LLM_PROVIDER
|
||||||
|
data = self.prompts.get(key, "")
|
||||||
|
|
||||||
|
if isinstance(data, dict):
|
||||||
|
# Wir versuchen erst den Provider, dann Gemini, dann Ollama
|
||||||
|
val = data.get(active_provider, data.get("gemini", data.get("ollama", "")))
|
||||||
|
|
||||||
|
# Falls val durch YAML-Fehler immer noch ein Dict ist, extrahiere ersten String
|
||||||
|
if isinstance(val, dict):
|
||||||
|
logger.warning(f"⚠️ [LLMService] Nested dictionary detected for key '{key}'. Using first entry.")
|
||||||
|
val = next(iter(val.values()), "") if val else ""
|
||||||
|
return str(val)
|
||||||
|
|
||||||
|
return str(data)
|
||||||
|
|
||||||
async def generate_raw_response(
|
async def generate_raw_response(
|
||||||
self,
|
self,
|
||||||
prompt: str,
|
prompt: str,
|
||||||
system: str = None,
|
system: str = None,
|
||||||
force_json: bool = False,
|
force_json: bool = False,
|
||||||
max_retries: int = 0,
|
max_retries: int = 2,
|
||||||
base_delay: float = 2.0,
|
base_delay: float = 2.0,
|
||||||
priority: Literal["realtime", "background"] = "realtime"
|
priority: Literal["realtime", "background"] = "realtime",
|
||||||
|
provider: Optional[str] = None,
|
||||||
|
model_override: Optional[str] = None,
|
||||||
|
json_schema: Optional[Dict[str, Any]] = None,
|
||||||
|
json_schema_name: str = "mindnet_json",
|
||||||
|
strict_json_schema: bool = True
|
||||||
) -> str:
|
) -> str:
|
||||||
"""
|
"""
|
||||||
Führt einen LLM Call aus.
|
Haupteinstiegspunkt für LLM-Anfragen mit Priorisierung.
|
||||||
priority="realtime": Chat (Sofort, keine Bremse).
|
Wendet die Bereinigung auf Text-Antworten an.
|
||||||
priority="background": Import/Analyse (Gedrosselt durch Semaphore).
|
|
||||||
"""
|
"""
|
||||||
|
target_provider = provider or self.settings.MINDNET_LLM_PROVIDER
|
||||||
use_semaphore = (priority == "background")
|
|
||||||
|
|
||||||
if use_semaphore and LLMService._background_semaphore:
|
|
||||||
async with LLMService._background_semaphore:
|
|
||||||
return await self._execute_request(prompt, system, force_json, max_retries, base_delay)
|
|
||||||
else:
|
|
||||||
# Realtime oder Fallback (falls Semaphore Init fehlschlug)
|
|
||||||
return await self._execute_request(prompt, system, force_json, max_retries, base_delay)
|
|
||||||
|
|
||||||
async def _execute_request(self, prompt, system, force_json, max_retries, base_delay):
|
if priority == "background":
|
||||||
payload: Dict[str, Any] = {
|
async with LLMService._background_semaphore:
|
||||||
|
res = await self._dispatch(
|
||||||
|
target_provider, prompt, system, force_json,
|
||||||
|
max_retries, base_delay, model_override,
|
||||||
|
json_schema, json_schema_name, strict_json_schema
|
||||||
|
)
|
||||||
|
# WP-14 Fix: Bereinige Text-Antworten vor Rückgabe
|
||||||
|
return clean_llm_text(res) if not force_json else res
|
||||||
|
|
||||||
|
res = await self._dispatch(
|
||||||
|
target_provider, prompt, system, force_json,
|
||||||
|
max_retries, base_delay, model_override,
|
||||||
|
json_schema, json_schema_name, strict_json_schema
|
||||||
|
)
|
||||||
|
# WP-14 Fix: Bereinige Text-Antworten vor Rückgabe
|
||||||
|
return clean_llm_text(res) if not force_json else res
|
||||||
|
|
||||||
|
async def _dispatch(
|
||||||
|
self,
|
||||||
|
provider: str,
|
||||||
|
prompt: str,
|
||||||
|
system: Optional[str],
|
||||||
|
force_json: bool,
|
||||||
|
max_retries: int,
|
||||||
|
base_delay: float,
|
||||||
|
model_override: Optional[str],
|
||||||
|
json_schema: Optional[Dict[str, Any]],
|
||||||
|
json_schema_name: str,
|
||||||
|
strict_json_schema: bool
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Routet die Anfrage mit intelligenter Rate-Limit Erkennung.
|
||||||
|
Nutzt max_retries um die Rate-Limit Schleife zu begrenzen.
|
||||||
|
"""
|
||||||
|
rate_limit_attempts = 0
|
||||||
|
# FIX: Wir nutzen max_retries als Limit für Rate-Limit Versuche, wenn explizit klein gewählt (z.B. Chat)
|
||||||
|
max_rate_retries = min(max_retries, getattr(self.settings, "LLM_RATE_LIMIT_RETRIES", 3))
|
||||||
|
wait_time = getattr(self.settings, "LLM_RATE_LIMIT_WAIT", 60.0)
|
||||||
|
|
||||||
|
while rate_limit_attempts <= max_rate_retries:
|
||||||
|
try:
|
||||||
|
if provider == "openrouter" and self.openrouter_client:
|
||||||
|
return await self._execute_openrouter(
|
||||||
|
prompt=prompt,
|
||||||
|
system=system,
|
||||||
|
force_json=force_json,
|
||||||
|
model_override=model_override,
|
||||||
|
json_schema=json_schema,
|
||||||
|
json_schema_name=json_schema_name,
|
||||||
|
strict_json_schema=strict_json_schema
|
||||||
|
)
|
||||||
|
|
||||||
|
if provider == "gemini" and self.google_client:
|
||||||
|
return await self._execute_google(prompt, system, force_json, model_override)
|
||||||
|
|
||||||
|
# Default/Fallback zu Ollama
|
||||||
|
return await self._execute_ollama(prompt, system, force_json, max_retries, base_delay)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
err_str = str(e)
|
||||||
|
# Intelligente 429 Erkennung
|
||||||
|
is_rate_limit = any(x in err_str for x in ["429", "RESOURCE_EXHAUSTED", "rate_limited", "Too Many Requests"])
|
||||||
|
|
||||||
|
if is_rate_limit and rate_limit_attempts < max_rate_retries:
|
||||||
|
rate_limit_attempts += 1
|
||||||
|
logger.warning(
|
||||||
|
f"⏳ [LLMService] Rate Limit detected from {provider}. "
|
||||||
|
f"Attempt {rate_limit_attempts}/{max_rate_retries}. Waiting {wait_time}s..."
|
||||||
|
)
|
||||||
|
await asyncio.sleep(wait_time)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Wenn kein Rate-Limit oder Retries erschöpft -> Fallback zu Ollama (falls aktiviert)
|
||||||
|
if self.settings.LLM_FALLBACK_ENABLED and provider != "ollama":
|
||||||
|
logger.warning(
|
||||||
|
f"🔄 Provider {provider} failed ({err_str}). Falling back to LOCAL OLLAMA."
|
||||||
|
)
|
||||||
|
return await self._execute_ollama(prompt, system, force_json, max_retries, base_delay)
|
||||||
|
raise e
|
||||||
|
|
||||||
|
async def _execute_google(self, prompt, system, force_json, model_override):
|
||||||
|
"""Native Google SDK Integration (Gemini) mit v1 Fix."""
|
||||||
|
model = model_override or self.settings.GEMINI_MODEL
|
||||||
|
clean_model = model.replace("models/", "")
|
||||||
|
|
||||||
|
config = types.GenerateContentConfig(
|
||||||
|
system_instruction=system,
|
||||||
|
response_mime_type="application/json" if force_json else "text/plain"
|
||||||
|
)
|
||||||
|
response = await asyncio.wait_for(
|
||||||
|
asyncio.to_thread(
|
||||||
|
self.google_client.models.generate_content,
|
||||||
|
model=clean_model, contents=prompt, config=config
|
||||||
|
),
|
||||||
|
timeout=45.0
|
||||||
|
)
|
||||||
|
return response.text.strip()
|
||||||
|
|
||||||
|
async def _execute_openrouter(
|
||||||
|
self,
|
||||||
|
prompt: str,
|
||||||
|
system: Optional[str],
|
||||||
|
force_json: bool,
|
||||||
|
model_override: Optional[str],
|
||||||
|
json_schema: Optional[Dict[str, Any]] = None,
|
||||||
|
json_schema_name: str = "mindnet_json",
|
||||||
|
strict_json_schema: bool = True
|
||||||
|
) -> str:
|
||||||
|
"""OpenRouter API Integration (OpenAI-kompatibel)."""
|
||||||
|
model = model_override or self.settings.OPENROUTER_MODEL
|
||||||
|
messages = []
|
||||||
|
if system:
|
||||||
|
messages.append({"role": "system", "content": system})
|
||||||
|
messages.append({"role": "user", "content": prompt})
|
||||||
|
|
||||||
|
kwargs: Dict[str, Any] = {}
|
||||||
|
if force_json:
|
||||||
|
if json_schema:
|
||||||
|
kwargs["response_format"] = {
|
||||||
|
"type": "json_schema",
|
||||||
|
"json_schema": {
|
||||||
|
"name": json_schema_name,
|
||||||
|
"strict": strict_json_schema,
|
||||||
|
"schema": json_schema
|
||||||
|
}
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
kwargs["response_format"] = {"type": "json_object"}
|
||||||
|
|
||||||
|
response = await self.openrouter_client.chat.completions.create(
|
||||||
|
model=model,
|
||||||
|
messages=messages,
|
||||||
|
**kwargs
|
||||||
|
)
|
||||||
|
return response.choices[0].message.content.strip()
|
||||||
|
|
||||||
|
async def _execute_ollama(self, prompt, system, force_json, max_retries, base_delay):
|
||||||
|
"""Lokaler Ollama Call mit striktem Retry-Limit."""
|
||||||
|
payload = {
|
||||||
"model": self.settings.LLM_MODEL,
|
"model": self.settings.LLM_MODEL,
|
||||||
"prompt": prompt,
|
"prompt": prompt,
|
||||||
"stream": False,
|
"stream": False,
|
||||||
"options": {
|
"options": {
|
||||||
"temperature": 0.1 if force_json else 0.7,
|
"temperature": 0.1 if force_json else 0.7,
|
||||||
"num_ctx": 8192
|
"num_ctx": 8192 # Begrenzung für Stabilität (WP-20)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if force_json:
|
if force_json:
|
||||||
payload["format"] = "json"
|
payload["format"] = "json"
|
||||||
|
|
||||||
if system:
|
if system:
|
||||||
payload["system"] = system
|
payload["system"] = system
|
||||||
|
|
||||||
attempt = 0
|
attempt = 0
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
try:
|
try:
|
||||||
response = await self.client.post("/api/generate", json=payload)
|
res = await self.ollama_client.post("/api/generate", json=payload)
|
||||||
|
res.raise_for_status()
|
||||||
if response.status_code == 200:
|
return res.json().get("response", "").strip()
|
||||||
data = response.json()
|
|
||||||
return data.get("response", "").strip()
|
|
||||||
else:
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
attempt += 1
|
attempt += 1
|
||||||
|
# WICHTIG: Wenn max_retries=0 (Chat), bricht dies nach dem 1. Versuch (attempt=1) sofort ab.
|
||||||
if attempt > max_retries:
|
if attempt > max_retries:
|
||||||
logger.error(f"LLM Final Error (Versuch {attempt}): {e}")
|
logger.error(f"❌ Ollama request failed after {attempt} attempt(s): {e}")
|
||||||
raise e
|
raise e
|
||||||
|
|
||||||
wait_time = base_delay * (2 ** (attempt - 1))
|
wait_time = base_delay * (2 ** (attempt - 1))
|
||||||
logger.warning(f"⚠️ LLM Retry ({attempt}/{max_retries}) in {wait_time}s: {e}")
|
logger.warning(f"⚠️ Ollama attempt {attempt} failed. Retrying in {wait_time}s...")
|
||||||
await asyncio.sleep(wait_time)
|
await asyncio.sleep(wait_time)
|
||||||
|
|
||||||
async def generate_rag_response(self, query: str, context_str: str) -> str:
|
async def generate_rag_response(self, query: str, context_str: str) -> str:
|
||||||
"""
|
"""Vollständiges RAG Chat-Interface."""
|
||||||
Chat-Wrapper: Immer Realtime.
|
provider = self.settings.MINDNET_LLM_PROVIDER
|
||||||
"""
|
system_prompt = self.get_prompt("system_prompt", provider)
|
||||||
system_prompt = self.prompts.get("system_prompt", "")
|
rag_template = self.get_prompt("rag_template", provider)
|
||||||
rag_template = self.prompts.get("rag_template", "{context_str}\n\n{query}")
|
|
||||||
|
|
||||||
final_prompt = rag_template.format(context_str=context_str, query=query)
|
final_prompt = rag_template.format(context_str=context_str, query=query)
|
||||||
|
|
||||||
|
# RAG Aufrufe im Chat nutzen nun standardmäßig max_retries=2 (überschreibbar)
|
||||||
|
# Durch den Aufruf von generate_raw_response wird die Bereinigung automatisch angewendet.
|
||||||
return await self.generate_raw_response(
|
return await self.generate_raw_response(
|
||||||
final_prompt,
|
final_prompt,
|
||||||
system=system_prompt,
|
system=system_prompt,
|
||||||
max_retries=0,
|
|
||||||
force_json=False,
|
|
||||||
priority="realtime"
|
priority="realtime"
|
||||||
)
|
)
|
||||||
|
|
||||||
async def close(self):
|
async def close(self):
|
||||||
if self.client:
|
"""Schließt die HTTP-Verbindungen."""
|
||||||
await self.client.aclose()
|
if self.ollama_client:
|
||||||
|
await self.ollama_client.aclose()
|
||||||
@@ -1,138 +0,0 @@
|
|||||||
"""
|
|
||||||
app/services/semantic_analyzer.py — Edge Validation & Filtering
|
|
||||||
Version: 2.0 (Update: Background Priority for Batch Jobs)
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
from typing import List, Optional
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
# Importe
|
|
||||||
from app.services.llm_service import LLMService
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
class SemanticAnalyzer:
|
|
||||||
def __init__(self):
|
|
||||||
self.llm = LLMService()
|
|
||||||
|
|
||||||
async def assign_edges_to_chunk(self, chunk_text: str, all_edges: List[str], note_type: str) -> List[str]:
|
|
||||||
"""
|
|
||||||
Sendet einen Chunk und eine Liste potenzieller Kanten an das LLM.
|
|
||||||
Das LLM filtert heraus, welche Kanten für diesen Chunk relevant sind.
|
|
||||||
|
|
||||||
Features:
|
|
||||||
- Retry Strategy: Wartet bei Überlastung (max_retries=5).
|
|
||||||
- Priority Queue: Läuft als "background" Task, um den Chat nicht zu blockieren.
|
|
||||||
- Observability: Loggt Input-Größe, Raw-Response und Parsing-Details.
|
|
||||||
"""
|
|
||||||
if not all_edges:
|
|
||||||
return []
|
|
||||||
|
|
||||||
# 1. Prompt laden
|
|
||||||
prompt_template = self.llm.prompts.get("edge_allocation_template")
|
|
||||||
|
|
||||||
if not prompt_template:
|
|
||||||
logger.warning("⚠️ [SemanticAnalyzer] Prompt 'edge_allocation_template' fehlt. Nutze Fallback.")
|
|
||||||
prompt_template = (
|
|
||||||
"TASK: Wähle aus den Kandidaten die relevanten Kanten für den Text.\n"
|
|
||||||
"TEXT: {chunk_text}\n"
|
|
||||||
"KANDIDATEN: {edge_list}\n"
|
|
||||||
"OUTPUT: JSON Liste von Strings [\"kind:target\"]."
|
|
||||||
)
|
|
||||||
|
|
||||||
# 2. Kandidaten-Liste formatieren
|
|
||||||
edges_str = "\n".join([f"- {e}" for e in all_edges])
|
|
||||||
|
|
||||||
# LOG: Request Info
|
|
||||||
logger.debug(f"🔍 [SemanticAnalyzer] Request: {len(chunk_text)} chars Text, {len(all_edges)} Candidates.")
|
|
||||||
|
|
||||||
# 3. Prompt füllen
|
|
||||||
final_prompt = prompt_template.format(
|
|
||||||
chunk_text=chunk_text[:3500],
|
|
||||||
edge_list=edges_str
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# 4. LLM Call mit Traffic Control (NEU: priority="background")
|
|
||||||
# Wir nutzen die "Slow Lane", damit der User im Chat nicht warten muss.
|
|
||||||
response_json = await self.llm.generate_raw_response(
|
|
||||||
prompt=final_prompt,
|
|
||||||
force_json=True,
|
|
||||||
max_retries=5,
|
|
||||||
base_delay=5.0,
|
|
||||||
priority="background" # <--- WICHTIG: Drosselung aktivieren
|
|
||||||
)
|
|
||||||
|
|
||||||
# LOG: Raw Response Preview
|
|
||||||
logger.debug(f"📥 [SemanticAnalyzer] Raw Response (Preview): {response_json[:200]}...")
|
|
||||||
|
|
||||||
# 5. Parsing & Cleaning
|
|
||||||
clean_json = response_json.replace("```json", "").replace("```", "").strip()
|
|
||||||
|
|
||||||
if not clean_json:
|
|
||||||
logger.warning("⚠️ [SemanticAnalyzer] Leere Antwort vom LLM erhalten. Trigger Fallback.")
|
|
||||||
return []
|
|
||||||
|
|
||||||
try:
|
|
||||||
data = json.loads(clean_json)
|
|
||||||
except json.JSONDecodeError as json_err:
|
|
||||||
logger.error(f"❌ [SemanticAnalyzer] JSON Decode Error.")
|
|
||||||
logger.error(f" Grund: {json_err}")
|
|
||||||
logger.error(f" Empfangener String: {clean_json[:500]}")
|
|
||||||
logger.info(" -> Workaround: Fallback auf 'Alle Kanten' (durch Chunker).")
|
|
||||||
return []
|
|
||||||
|
|
||||||
valid_edges = []
|
|
||||||
|
|
||||||
# 6. Robuste Validierung (List vs Dict)
|
|
||||||
if isinstance(data, list):
|
|
||||||
# Standardfall: ["kind:target", ...]
|
|
||||||
valid_edges = [str(e) for e in data if isinstance(e, str) and ":" in e]
|
|
||||||
|
|
||||||
elif isinstance(data, dict):
|
|
||||||
# Abweichende Formate behandeln
|
|
||||||
logger.info(f"ℹ️ [SemanticAnalyzer] LLM lieferte Dict statt Liste. Versuche Reparatur. Keys: {list(data.keys())}")
|
|
||||||
|
|
||||||
for key, val in data.items():
|
|
||||||
# Fall A: {"edges": ["kind:target"]}
|
|
||||||
if key.lower() in ["edges", "results", "kanten", "matches"] and isinstance(val, list):
|
|
||||||
valid_edges.extend([str(e) for e in val if isinstance(e, str) and ":" in e])
|
|
||||||
|
|
||||||
# Fall B: {"kind": "target"}
|
|
||||||
elif isinstance(val, str):
|
|
||||||
valid_edges.append(f"{key}:{val}")
|
|
||||||
|
|
||||||
# Fall C: {"kind": ["target1", "target2"]}
|
|
||||||
elif isinstance(val, list):
|
|
||||||
for target in val:
|
|
||||||
if isinstance(target, str):
|
|
||||||
valid_edges.append(f"{key}:{target}")
|
|
||||||
|
|
||||||
# Safety: Filtere nur Kanten, die halbwegs valide aussehen
|
|
||||||
final_result = [e for e in valid_edges if ":" in e]
|
|
||||||
|
|
||||||
# LOG: Ergebnis
|
|
||||||
if final_result:
|
|
||||||
logger.info(f"✅ [SemanticAnalyzer] Success. {len(final_result)} Kanten zugewiesen.")
|
|
||||||
else:
|
|
||||||
logger.debug(" [SemanticAnalyzer] Keine spezifischen Kanten erkannt (Empty Result).")
|
|
||||||
|
|
||||||
return final_result
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"💥 [SemanticAnalyzer] Kritischer Fehler: {e}", exc_info=True)
|
|
||||||
return []
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
if self.llm:
|
|
||||||
await self.llm.close()
|
|
||||||
|
|
||||||
# Singleton Helper
|
|
||||||
_analyzer_instance = None
|
|
||||||
def get_semantic_analyzer():
|
|
||||||
global _analyzer_instance
|
|
||||||
if _analyzer_instance is None:
|
|
||||||
_analyzer_instance = SemanticAnalyzer()
|
|
||||||
return _analyzer_instance
|
|
||||||
+57
-34
@@ -1,48 +1,62 @@
|
|||||||
# config/decision_engine.yaml
|
# config/decision_engine.yaml
|
||||||
# Steuerung der Decision Engine (Intent Recognition)
|
# Steuerung der Decision Engine (Intent Recognition & Graph Routing)
|
||||||
# Version: 2.4.0 (Clean Architecture: Generic Intents only)
|
# VERSION: 2.6.1 (WP-20: Hybrid LLM & WP-22: Semantic Graph Routing)
|
||||||
|
# STATUS: Active
|
||||||
|
# DoD: Keine Hardcoded Modelle, volle Integration der strategischen Boosts.
|
||||||
|
|
||||||
version: 1.4
|
version: 2.6
|
||||||
|
|
||||||
settings:
|
settings:
|
||||||
llm_fallback_enabled: true
|
llm_fallback_enabled: true
|
||||||
|
|
||||||
# Few-Shot Prompting für den LLM-Router (Slow Path)
|
# Strategie für den Router selbst (Welches Modell erkennt den Intent?)
|
||||||
|
# "auto" nutzt den in MINDNET_LLM_PROVIDER gesetzten Standard (z.B. openrouter).
|
||||||
|
router_provider: "auto"
|
||||||
|
|
||||||
|
# Few-Shot Prompting für den LLM-Router
|
||||||
llm_router_prompt: |
|
llm_router_prompt: |
|
||||||
Du bist ein Klassifikator. Analysiere die Nachricht und wähle die passende Strategie.
|
Du bist der zentrale Intent-Klassifikator für Mindnet, einen digitalen Zwilling.
|
||||||
|
Analysiere die Nachricht und wähle die passende Strategie.
|
||||||
Antworte NUR mit dem Namen der Strategie.
|
Antworte NUR mit dem Namen der Strategie.
|
||||||
|
|
||||||
STRATEGIEN:
|
STRATEGIEN:
|
||||||
- INTERVIEW: User will Wissen erfassen, Notizen anlegen oder Dinge festhalten.
|
- INTERVIEW: User will Wissen erfassen, Notizen anlegen oder Dinge festhalten.
|
||||||
- DECISION: Rat, Strategie, Vor/Nachteile, "Soll ich".
|
- DECISION: Rat, Strategie, Abwägung von Werten, "Soll ich tun X?".
|
||||||
- EMPATHY: Gefühle, Frust, Freude, Probleme.
|
- EMPATHY: Gefühle, Reflexion der eigenen Verfassung, Frust, Freude.
|
||||||
- CODING: Code, Syntax, Programmierung.
|
- CODING: Code-Erstellung, Debugging, technische Dokumentation.
|
||||||
- FACT: Wissen, Fakten, Definitionen.
|
- FACT: Reine Wissensabfrage, Definitionen, Suchen von Informationen.
|
||||||
|
|
||||||
BEISPIELE:
|
BEISPIELE:
|
||||||
User: "Wie funktioniert Qdrant?" -> FACT
|
User: "Wie funktioniert die Qdrant-Vektor-DB?" -> FACT
|
||||||
User: "Soll ich Qdrant nutzen?" -> DECISION
|
User: "Soll ich mein Startup jetzt verkaufen?" -> DECISION
|
||||||
User: "Ich möchte etwas notieren" -> INTERVIEW
|
User: "Notiere mir kurz meine Gedanken zum Meeting." -> INTERVIEW
|
||||||
User: "Lass uns das festhalten" -> INTERVIEW
|
User: "Ich fühle mich heute sehr erschöpft." -> EMPATHY
|
||||||
User: "Schreibe ein Python Script" -> CODING
|
User: "Schreibe eine FastAPI-Route für den Ingest." -> CODING
|
||||||
User: "Alles ist grau und sinnlos" -> EMPATHY
|
|
||||||
|
|
||||||
NACHRICHT: "{query}"
|
NACHRICHT: "{query}"
|
||||||
|
|
||||||
STRATEGIE:
|
STRATEGIE:
|
||||||
|
|
||||||
strategies:
|
strategies:
|
||||||
# 1. Fakten-Abfrage (Fallback & Default)
|
# 1. Fakten-Abfrage (Turbo-Modus via OpenRouter / Primary)
|
||||||
FACT:
|
FACT:
|
||||||
description: "Reine Wissensabfrage."
|
description: "Reine Wissensabfrage."
|
||||||
|
preferred_provider: "openrouter"
|
||||||
trigger_keywords: []
|
trigger_keywords: []
|
||||||
inject_types: []
|
inject_types: []
|
||||||
|
# WP-22: Definitionen & Hierarchien im Graphen bevorzugen
|
||||||
|
edge_boosts:
|
||||||
|
part_of: 2.0
|
||||||
|
composed_of: 2.0
|
||||||
|
similar_to: 1.5
|
||||||
|
caused_by: 0.5
|
||||||
prompt_template: "rag_template"
|
prompt_template: "rag_template"
|
||||||
prepend_instruction: null
|
prepend_instruction: null
|
||||||
|
|
||||||
# 2. Entscheidungs-Frage
|
# 2. Entscheidungs-Frage (Power-Strategie via Gemini)
|
||||||
DECISION:
|
DECISION:
|
||||||
description: "Der User sucht Rat, Strategie oder Abwägung."
|
description: "Der User sucht Rat, Strategie oder Abwägung."
|
||||||
|
preferred_provider: "gemini"
|
||||||
trigger_keywords:
|
trigger_keywords:
|
||||||
- "soll ich"
|
- "soll ich"
|
||||||
- "meinung"
|
- "meinung"
|
||||||
@@ -53,14 +67,22 @@ strategies:
|
|||||||
- "abwägung"
|
- "abwägung"
|
||||||
- "vergleich"
|
- "vergleich"
|
||||||
inject_types: ["value", "principle", "goal", "risk"]
|
inject_types: ["value", "principle", "goal", "risk"]
|
||||||
|
# WP-22: Risiken und Konsequenzen im Graphen priorisieren
|
||||||
|
edge_boosts:
|
||||||
|
blocks: 2.5
|
||||||
|
solves: 2.0
|
||||||
|
depends_on: 1.5
|
||||||
|
risk_of: 2.5
|
||||||
|
impacts: 2.0
|
||||||
prompt_template: "decision_template"
|
prompt_template: "decision_template"
|
||||||
prepend_instruction: |
|
prepend_instruction: |
|
||||||
!!! ENTSCHEIDUNGS-MODUS !!!
|
!!! ENTSCHEIDUNGS-MODUS (HYBRID AI) !!!
|
||||||
BITTE WÄGE FAKTEN GEGEN FOLGENDE WERTE, PRINZIPIEN UND ZIELE AB:
|
BITTE WÄGE FAKTEN GEGEN FOLGENDE WERTE, PRINZIPIEN UND ZIELE AB:
|
||||||
|
|
||||||
# 3. Empathie / "Ich"-Modus
|
# 3. Empathie / "Ich"-Modus (Lokal & Privat via Ollama)
|
||||||
EMPATHY:
|
EMPATHY:
|
||||||
description: "Reaktion auf emotionale Zustände."
|
description: "Reaktion auf emotionale Zustände."
|
||||||
|
preferred_provider: "openrouter"
|
||||||
trigger_keywords:
|
trigger_keywords:
|
||||||
- "ich fühle"
|
- "ich fühle"
|
||||||
- "traurig"
|
- "traurig"
|
||||||
@@ -71,12 +93,18 @@ strategies:
|
|||||||
- "überfordert"
|
- "überfordert"
|
||||||
- "müde"
|
- "müde"
|
||||||
inject_types: ["experience", "belief", "profile"]
|
inject_types: ["experience", "belief", "profile"]
|
||||||
|
edge_boosts:
|
||||||
|
based_on: 2.0
|
||||||
|
related_to: 2.0
|
||||||
|
experienced_in: 2.5
|
||||||
|
blocks: 0.1
|
||||||
prompt_template: "empathy_template"
|
prompt_template: "empathy_template"
|
||||||
prepend_instruction: null
|
prepend_instruction: null
|
||||||
|
|
||||||
# 4. Coding / Technical
|
# 4. Coding / Technical (Gemini Power)
|
||||||
CODING:
|
CODING:
|
||||||
description: "Technische Anfragen und Programmierung."
|
description: "Technische Anfragen und Programmierung."
|
||||||
|
preferred_provider: "gemini"
|
||||||
trigger_keywords:
|
trigger_keywords:
|
||||||
- "code"
|
- "code"
|
||||||
- "python"
|
- "python"
|
||||||
@@ -88,14 +116,18 @@ strategies:
|
|||||||
- "yaml"
|
- "yaml"
|
||||||
- "bash"
|
- "bash"
|
||||||
inject_types: ["snippet", "reference", "source"]
|
inject_types: ["snippet", "reference", "source"]
|
||||||
|
# WP-22: Technische Abhängigkeiten priorisieren
|
||||||
|
edge_boosts:
|
||||||
|
uses: 2.5
|
||||||
|
depends_on: 2.0
|
||||||
|
implemented_in: 3.0
|
||||||
prompt_template: "technical_template"
|
prompt_template: "technical_template"
|
||||||
prepend_instruction: null
|
prepend_instruction: null
|
||||||
|
|
||||||
# 5. Interview / Datenerfassung
|
# 5. Interview / Datenerfassung (Lokal)
|
||||||
# HINWEIS: Spezifische Typen (Projekt, Ziel etc.) werden automatisch
|
|
||||||
# über die types.yaml erkannt. Hier stehen nur generische Trigger.
|
|
||||||
INTERVIEW:
|
INTERVIEW:
|
||||||
description: "Der User möchte Wissen erfassen."
|
description: "Der User möchte Wissen erfassen."
|
||||||
|
preferred_provider: "openrouter"
|
||||||
trigger_keywords:
|
trigger_keywords:
|
||||||
- "neue notiz"
|
- "neue notiz"
|
||||||
- "etwas notieren"
|
- "etwas notieren"
|
||||||
@@ -108,15 +140,6 @@ strategies:
|
|||||||
- "idee speichern"
|
- "idee speichern"
|
||||||
- "draft"
|
- "draft"
|
||||||
inject_types: []
|
inject_types: []
|
||||||
|
edge_boosts: {}
|
||||||
prompt_template: "interview_template"
|
prompt_template: "interview_template"
|
||||||
prepend_instruction: null
|
prepend_instruction: null
|
||||||
|
|
||||||
# Schemas: Hier nur der Fallback.
|
|
||||||
# Spezifische Schemas (Project, Experience) kommen jetzt aus types.yaml!
|
|
||||||
schemas:
|
|
||||||
default:
|
|
||||||
fields:
|
|
||||||
- "Titel"
|
|
||||||
- "Thema/Inhalt"
|
|
||||||
- "Tags"
|
|
||||||
hint: "Halte es einfach und übersichtlich."
|
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# --- FastAPI Server (Produktion) ---
|
||||||
|
UVICORN_HOST=0.0.0.0
|
||||||
|
UVICORN_PORT=8000
|
||||||
|
DEBUG=false
|
||||||
|
|
||||||
|
# --- Qdrant Vektor-Datenbank ---
|
||||||
|
# Trennung der Daten durch eigenes Prefix
|
||||||
|
QDRANT_URL=http://127.0.0.1:6333
|
||||||
|
QDRANT_API_KEY=
|
||||||
|
COLLECTION_PREFIX=mindnet
|
||||||
|
|
||||||
|
# --- Vektoren-Konfiguration ---
|
||||||
|
# Muss 768 für 'nomic-embed-text' sein
|
||||||
|
VECTOR_DIM=768
|
||||||
|
|
||||||
|
# --- AI Modelle (Lokal/Fallback) ---
|
||||||
|
MINDNET_LLM_MODEL=phi3:mini
|
||||||
|
MINDNET_OLLAMA_URL=http://127.0.0.1:11434
|
||||||
|
MINDNET_LLM_TIMEOUT=300.0
|
||||||
|
MINDNET_LLM_BACKGROUND_LIMIT=2
|
||||||
|
|
||||||
|
# Vektor-Modell für semantische Suche
|
||||||
|
MINDNET_EMBEDDING_MODEL=nomic-embed-text
|
||||||
|
|
||||||
|
# --- WP-20/WP-76: Hybrid-Cloud & Resilienz ---
|
||||||
|
# Primärer Provider für höchste Qualität
|
||||||
|
MINDNET_LLM_PROVIDER=openrouter
|
||||||
|
MINDNET_LLM_FALLBACK=true
|
||||||
|
|
||||||
|
# Intelligente Rate-Limit Steuerung (Sekunden/Versuche)
|
||||||
|
MINDNET_LLM_RATE_LIMIT_WAIT=60.0
|
||||||
|
MINDNET_LLM_RATE_LIMIT_RETRIES=3
|
||||||
|
|
||||||
|
# --- Cloud Provider Keys (Hier Prod-Keys einsetzen) ---
|
||||||
|
GOOGLE_API_KEY=AIzaSy... (Dein Prod-Key)
|
||||||
|
MINDNET_GEMINI_MODEL=gemini-2.5-flash-lite
|
||||||
|
|
||||||
|
OPENROUTER_API_KEY=sk-or-v1-... (Dein Prod-Key)
|
||||||
|
# Stabilstes Free-Modell für strukturierte Extraktion
|
||||||
|
OPENROUTER_MODEL=mistralai/mistral-7b-instruct:free
|
||||||
|
|
||||||
|
# --- Pfade & System (Produktions-Vault) ---
|
||||||
|
MINDNET_TYPES_FILE=./config/types.yaml
|
||||||
|
MINDNET_VAULT_ROOT=./vault_prod
|
||||||
|
MINDNET_VOCAB_PATH=/mindnet/vault/mindnet/_system/dictionary/edge_vocabulary.md
|
||||||
|
|
||||||
|
# Change Detection für effiziente Re-Imports
|
||||||
|
MINDNET_CHANGE_DETECTION_MODE=full
|
||||||
+225
-114
@@ -1,4 +1,8 @@
|
|||||||
# config/prompts.yaml — Final V2.3.1 (Multi-Personality Support)
|
# config/prompts.yaml — Final V2.6.0 (WP-15b Candidate-Validation)
|
||||||
|
# WP-20: Optimierte Cloud-Templates zur Unterdrückung von Modell-Geschwätz.
|
||||||
|
# FIX: Explizite Verbote für Einleitungstexte zur Vermeidung von JSON-Parsing-Fehlern.
|
||||||
|
# WP-15b: Integration der binären edge_validation für den Two-Pass Workflow.
|
||||||
|
# OLLAMA: UNVERÄNDERT laut Benutzeranweisung.
|
||||||
|
|
||||||
system_prompt: |
|
system_prompt: |
|
||||||
Du bist 'mindnet', mein digitaler Zwilling und strategischer Partner.
|
Du bist 'mindnet', mein digitaler Zwilling und strategischer Partner.
|
||||||
@@ -15,148 +19,255 @@ system_prompt: |
|
|||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
# 1. STANDARD: Fakten & Wissen (Intent: FACT)
|
# 1. STANDARD: Fakten & Wissen (Intent: FACT)
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
rag_template: |
|
rag_template:
|
||||||
QUELLEN (WISSEN):
|
ollama: |
|
||||||
=========================================
|
QUELLEN (WISSEN):
|
||||||
{context_str}
|
=========================================
|
||||||
=========================================
|
{context_str}
|
||||||
|
=========================================
|
||||||
|
|
||||||
FRAGE:
|
FRAGE:
|
||||||
{query}
|
{query}
|
||||||
|
|
||||||
ANWEISUNG:
|
ANWEISUNG:
|
||||||
Beantworte die Frage präzise basierend auf den Quellen.
|
Beantworte die Frage präzise basierend auf den Quellen.
|
||||||
Fasse die Informationen zusammen. Sei objektiv und neutral.
|
Fasse die Informationen zusammen. Sei objektiv und neutral.
|
||||||
|
gemini: |
|
||||||
|
Kontext meines digitalen Zwillings: {context_str}
|
||||||
|
Beantworte strukturiert und präzise: {query}
|
||||||
|
openrouter: |
|
||||||
|
Kontext-Analyse für den digitalen Zwilling:
|
||||||
|
{context_str}
|
||||||
|
|
||||||
|
Anfrage: {query}
|
||||||
|
Antworte basierend auf dem Kontext.
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
# 2. DECISION: Strategie & Abwägung (Intent: DECISION)
|
# 2. DECISION: Strategie & Abwägung (Intent: DECISION)
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
decision_template: |
|
decision_template:
|
||||||
KONTEXT (FAKTEN & STRATEGIE):
|
ollama: |
|
||||||
=========================================
|
KONTEXT (FAKTEN & STRATEGIE):
|
||||||
{context_str}
|
=========================================
|
||||||
=========================================
|
{context_str}
|
||||||
|
=========================================
|
||||||
|
|
||||||
ENTSCHEIDUNGSFRAGE:
|
ENTSCHEIDUNGSFRAGE:
|
||||||
{query}
|
{query}
|
||||||
|
|
||||||
ANWEISUNG:
|
ANWEISUNG:
|
||||||
Du agierst als mein Entscheidungs-Partner.
|
Du agierst als mein Entscheidungs-Partner.
|
||||||
1. Analysiere die Faktenlage aus den Quellen.
|
1. Analysiere die Faktenlage aus den Quellen.
|
||||||
2. Prüfe dies hart gegen meine strategischen Notizen (Typ [VALUE], [PRINCIPLE], [GOAL]).
|
2. Prüfe dies hart gegen meine strategischen Notizen (Typ [VALUE], [PRINCIPLE], [GOAL]).
|
||||||
3. Wäge ab: Passt die technische/faktische Lösung zu meinen Werten?
|
3. Wäge ab: Passt die technische/faktische Lösung zu meinen Werten?
|
||||||
|
|
||||||
FORMAT:
|
FORMAT:
|
||||||
- **Analyse:** (Kurze Zusammenfassung der Fakten)
|
- **Analyse:** (Kurze Zusammenfassung der Fakten)
|
||||||
- **Abgleich:** (Gibt es Konflikte mit Werten/Zielen? Nenne die Quelle!)
|
- **Abgleich:** (Gibt es Konflikte mit Werten/Zielen? Nenne die Quelle!)
|
||||||
- **Empfehlung:** (Klare Meinung: Ja/Nein/Vielleicht mit Begründung)
|
- **Empfehlung:** (Klare Meinung: Ja/No/Vielleicht mit Begründung)
|
||||||
|
gemini: |
|
||||||
|
Agiere als strategischer Partner. Analysiere die Frage {query} basierend auf meinen Werten im Kontext {context_str}.
|
||||||
|
openrouter: |
|
||||||
|
Strategische Entscheidungsanalyse: {query}
|
||||||
|
Wertebasis aus dem Graphen: {context_str}
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
# 3. EMPATHY: Der Spiegel / "Ich"-Modus (Intent: EMPATHY)
|
# 3. EMPATHY: Der Spiegel / "Ich"-Modus (Intent: EMPATHY)
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
empathy_template: |
|
empathy_template:
|
||||||
KONTEXT (ERFAHRUNGEN & GLAUBENSSÄTZE):
|
ollama: |
|
||||||
=========================================
|
KONTEXT (ERFAHRUNGEN & GLAUBENSSÄTZE):
|
||||||
{context_str}
|
=========================================
|
||||||
=========================================
|
{context_str}
|
||||||
|
=========================================
|
||||||
|
|
||||||
SITUATION:
|
SITUATION:
|
||||||
{query}
|
{query}
|
||||||
|
|
||||||
ANWEISUNG:
|
ANWEISUNG:
|
||||||
Du agierst jetzt als mein empathischer Spiegel.
|
Du agierst jetzt als mein empathischer Spiegel.
|
||||||
1. Versuche nicht sofort, das Problem technisch zu lösen.
|
1. Versuche nicht sofort, das Problem technisch zu lösen.
|
||||||
2. Zeige Verständnis für die Situation basierend auf meinen eigenen Erfahrungen ([EXPERIENCE]) oder Glaubenssätzen ([BELIEF]), falls im Kontext vorhanden.
|
2. Zeige Verständnis für die Situation basierend auf meinen eigenen Erfahrungen ([EXPERIENCE]) oder Glaubenssätzen ([BELIEF]), falls im Kontext vorhanden.
|
||||||
3. Antworte in der "Ich"-Form oder "Wir"-Form. Sei unterstützend.
|
3. Antworte in der "Ich"-Form oder "Wir"-Form. Sei unterstützend.
|
||||||
|
|
||||||
TONFALL:
|
TONFALL:
|
||||||
Ruhig, verständnisvoll, reflektiert. Keine Aufzählungszeichen, sondern fließender Text.
|
Ruhig, verständnisvoll, reflektiert. Keine Aufzählungszeichen, sondern fließender Text.
|
||||||
|
gemini: "Sei mein digitaler Spiegel für {query}. Kontext: {context_str}"
|
||||||
|
openrouter: "Empathische Reflexion der Situation {query}. Persönlicher Kontext: {context_str}"
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
# 4. TECHNICAL: Der Coder (Intent: CODING)
|
# 4. TECHNICAL: Der Coder (Intent: CODING)
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
technical_template: |
|
technical_template:
|
||||||
KONTEXT (DOCS & SNIPPETS):
|
ollama: |
|
||||||
=========================================
|
KONTEXT (DOCS & SNIPPETS):
|
||||||
{context_str}
|
=========================================
|
||||||
=========================================
|
{context_str}
|
||||||
|
=========================================
|
||||||
|
|
||||||
TASK:
|
TASK:
|
||||||
{query}
|
{query}
|
||||||
|
|
||||||
|
ANWEISUNG:
|
||||||
|
Du bist Senior Developer.
|
||||||
|
1. Ignoriere Smalltalk. Komm sofort zum Punkt.
|
||||||
|
2. Generiere validen, performanten Code basierend auf den Quellen.
|
||||||
|
3. Wenn Quellen fehlen, nutze dein allgemeines Programmierwissen, aber weise darauf hin.
|
||||||
|
|
||||||
|
FORMAT:
|
||||||
|
- Kurze Erklärung des Ansatzes.
|
||||||
|
- Markdown Code-Block (Copy-Paste fertig).
|
||||||
|
- Wichtige Edge-Cases.
|
||||||
|
gemini: "Generiere Code für {query} unter Berücksichtigung von {context_str}."
|
||||||
|
openrouter: "Technischer Support für {query}. Code-Referenzen: {context_str}"
|
||||||
|
|
||||||
ANWEISUNG:
|
|
||||||
Du bist Senior Developer.
|
|
||||||
1. Ignoriere Smalltalk. Komm sofort zum Punkt.
|
|
||||||
2. Generiere validen, performanten Code basierend auf den Quellen.
|
|
||||||
3. Wenn Quellen fehlen, nutze dein allgemeines Programmierwissen, aber weise darauf hin.
|
|
||||||
|
|
||||||
FORMAT:
|
|
||||||
- Kurze Erklärung des Ansatzes.
|
|
||||||
- Markdown Code-Block (Copy-Paste fertig).
|
|
||||||
- Wichtige Edge-Cases.
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
# 5. INTERVIEW: Der "One-Shot Extractor" (Performance Mode)
|
# 5. INTERVIEW: Der "One-Shot Extractor" (Performance Mode)
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
interview_template: |
|
interview_template:
|
||||||
TASK:
|
ollama: |
|
||||||
Du bist ein professioneller Ghostwriter. Verwandle den "USER INPUT" in eine strukturierte Notiz vom Typ '{target_type}'.
|
TASK:
|
||||||
|
Du bist ein professioneller Ghostwriter. Verwandle den "USER INPUT" in eine strukturierte Notiz vom Typ '{target_type}'.
|
||||||
STRUKTUR (Nutze EXAKT diese Überschriften):
|
|
||||||
{schema_fields}
|
STRUKTUR (Nutze EXAKT diese Überschriften):
|
||||||
|
{schema_fields}
|
||||||
USER INPUT:
|
|
||||||
"{query}"
|
USER INPUT:
|
||||||
|
"{query}"
|
||||||
ANWEISUNG ZUM INHALT:
|
|
||||||
1. Analysiere den Input genau.
|
ANWEISUNG ZUM INHALT:
|
||||||
2. Schreibe die Inhalte unter die passenden Überschriften aus der STRUKTUR-Liste oben.
|
1. Analysiere den Input genau.
|
||||||
3. STIL: Schreibe flüssig, professionell und in der Ich-Perspektive. Korrigiere Grammatikfehler, aber behalte den persönlichen Ton bei.
|
2. Schreibe die Inhalte unter die passenden Überschriften aus der STRUKTUR-Liste oben.
|
||||||
4. Wenn Informationen für einen Abschnitt fehlen, schreibe nur: "[TODO: Ergänzen]". Erfinde nichts dazu.
|
3. STIL: Schreibe flüssig, professionell und in der Ich-Perspektive. Korrigiere Grammatikfehler, aber behalte den persönlichen Ton bei.
|
||||||
|
4. Wenn Informationen für einen Abschnitt fehlen, schreibe nur: "[TODO: Ergänzen]". Erfinde nichts dazu.
|
||||||
OUTPUT FORMAT (YAML + MARKDOWN):
|
|
||||||
---
|
OUTPUT FORMAT (YAML + MARKDOWN):
|
||||||
type: {target_type}
|
---
|
||||||
status: draft
|
type: {target_type}
|
||||||
title: (Erstelle einen treffenden, kurzen Titel für den Inhalt)
|
status: draft
|
||||||
tags: [Tag1, Tag2]
|
title: (Erstelle einen treffenden, kurzen Titel für den Inhalt)
|
||||||
---
|
tags: [Tag1, Tag2]
|
||||||
|
---
|
||||||
# (Wiederhole den Titel hier)
|
|
||||||
|
# (Wiederhole den Titel hier)
|
||||||
## (Erster Begriff aus STRUKTUR)
|
|
||||||
(Text...)
|
## (Erster Begriff aus STRUKTUR)
|
||||||
|
(Text...)
|
||||||
## (Zweiter Begriff aus STRUKTUR)
|
|
||||||
(Text...)
|
## (Zweiter Begriff aus STRUKTUR)
|
||||||
|
(Text...)
|
||||||
(usw.)
|
gemini: "Extrahiere Daten für {target_type} aus {query}."
|
||||||
|
openrouter: "Strukturiere den Input {query} nach dem Schema {schema_fields} für Typ {target_type}."
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
# 6. EDGE_ALLOCATION: Kantenfilter (Intent: OFFLINE_FILTER)
|
# 6. EDGE_ALLOCATION: Kantenfilter (Intent: OFFLINE_FILTER)
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
edge_allocation_template: |
|
edge_allocation_template:
|
||||||
TASK:
|
ollama: |
|
||||||
Du bist ein strikter Selektor. Du erhältst eine Liste von "Kandidaten-Kanten" (Strings).
|
TASK:
|
||||||
Wähle jene aus, die inhaltlich im "Textabschnitt" vorkommen oder relevant sind.
|
Du bist ein strikter Selektor. Du erhältst eine Liste von "Kandidaten-Kanten" (Strings).
|
||||||
|
Wähle jene aus, die inhaltlich im "Textabschnitt" vorkommen oder relevant sind.
|
||||||
|
|
||||||
TEXTABSCHNITT:
|
TEXTABSCHNITT:
|
||||||
"""
|
"""
|
||||||
{chunk_text}
|
{chunk_text}
|
||||||
"""
|
"""
|
||||||
|
|
||||||
KANDIDATEN (Auswahl-Pool):
|
KANDIDATEN (Auswahl-Pool):
|
||||||
{edge_list}
|
{edge_list}
|
||||||
|
|
||||||
REGELN:
|
REGELN:
|
||||||
1. Die Kanten haben das Format "typ:ziel". Der "typ" ist variabel und kann ALLES sein (z.B. uses, blocks, inspired_by, loves, etc.).
|
1. Die Kanten haben das Format "typ:ziel". Der "typ" ist variabel und kann ALLES sein.
|
||||||
2. Gib NUR die Strings aus der Kandidaten-Liste zurück, die zum Text passen.
|
2. Gib NUR die Strings aus der Kandidaten-Liste zurück, die zum Text passen.
|
||||||
3. Erfinde KEINE neuen Kanten. Nutze exakt die Schreibweise aus der Liste.
|
3. Erfinde KEINE neuen Kanten.
|
||||||
4. Antworte als flache JSON-Liste.
|
4. Antworte als flache JSON-Liste.
|
||||||
|
|
||||||
BEISPIEL (Zur Demonstration der Logik):
|
DEIN OUTPUT (JSON):
|
||||||
Input Text: "Das Projekt Alpha scheitert, weil Budget fehlt."
|
gemini: |
|
||||||
Input Kandidaten: ["blocks:Projekt Alpha", "inspired_by:Buch der Weisen", "needs:Budget"]
|
TASK: Ordne Kanten einem Textabschnitt zu.
|
||||||
Output: ["blocks:Projekt Alpha", "needs:Budget"]
|
ERLAUBTE TYPEN: {valid_types}
|
||||||
|
TEXT: {chunk_text}
|
||||||
|
KANDIDATEN: {edge_list}
|
||||||
|
OUTPUT: STRIKT eine flache JSON-Liste ["typ:ziel"]. Kein Text davor/danach. Wenn nichts: []. Keine Objekte!
|
||||||
|
openrouter: |
|
||||||
|
TASK: Filtere relevante Kanten aus dem Pool.
|
||||||
|
ERLAUBTE TYPEN: {valid_types}
|
||||||
|
TEXT: {chunk_text}
|
||||||
|
POOL: {edge_list}
|
||||||
|
ANWEISUNG: Gib NUR eine flache JSON-Liste von Strings zurück.
|
||||||
|
BEISPIEL: ["kind:target", "kind:target"]
|
||||||
|
REGEL: Kein Text, keine Analyse, keine Kommentare. Wenn nichts passt, gib [] zurück.
|
||||||
|
OUTPUT:
|
||||||
|
|
||||||
DEIN OUTPUT (JSON):
|
# ---------------------------------------------------------
|
||||||
|
# 7. SMART EDGE ALLOCATION: Extraktion (Intent: INGEST)
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
edge_extraction:
|
||||||
|
ollama: |
|
||||||
|
TASK:
|
||||||
|
Du bist ein Wissens-Ingenieur für den digitalen Zwilling 'mindnet'.
|
||||||
|
Deine Aufgabe ist es, semantische Relationen (Kanten) aus dem Text zu extrahieren,
|
||||||
|
die die Hauptnotiz '{note_id}' mit anderen Konzepten verbinden.
|
||||||
|
|
||||||
|
ANWEISUNGEN:
|
||||||
|
1. Identifiziere wichtige Entitäten, Konzepte oder Ereignisse im Text.
|
||||||
|
2. Bestimme die Art der Beziehung (z.B. part_of, uses, related_to, blocks, caused_by).
|
||||||
|
3. Das Ziel (target) muss ein prägnanter Begriff sein.
|
||||||
|
4. Antworte AUSSCHLIESSLICH in validem JSON als Liste von Objekten.
|
||||||
|
|
||||||
|
BEISPIEL:
|
||||||
|
[[ {{"to": "Ziel-Konzept", \"kind\": \"beziehungs_typ\"}} ]]
|
||||||
|
|
||||||
|
TEXT:
|
||||||
|
"""
|
||||||
|
{text}
|
||||||
|
"""
|
||||||
|
|
||||||
|
DEIN OUTPUT (JSON):
|
||||||
|
gemini: |
|
||||||
|
Analysiere '{note_id}'. Extrahiere semantische Beziehungen.
|
||||||
|
ERLAUBTE TYPEN: {valid_types}
|
||||||
|
TEXT: {text}
|
||||||
|
OUTPUT: STRIKT JSON-Array von Objekten: [[{{"to\":\"Ziel\",\"kind\":\"typ\"}}]]. Kein Text davor/danach. Wenn nichts: [].
|
||||||
|
openrouter: |
|
||||||
|
TASK: Extrahiere semantische Relationen für '{note_id}'.
|
||||||
|
ERLAUBTE TYPEN: {valid_types}
|
||||||
|
TEXT: {text}
|
||||||
|
ANWEISUNG: Antworte AUSSCHLIESSLICH mit einem JSON-Array von Objekten.
|
||||||
|
FORMAT: [[{{"to\":\"Ziel-Begriff\",\"kind\":\"typ\"}}]]
|
||||||
|
STRIKTES VERBOT: Schreibe keine Einleitung, keine Analyse und keine Erklärungen.
|
||||||
|
Wenn keine Relationen existieren, antworte NUR mit: []
|
||||||
|
OUTPUT:
|
||||||
|
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
# 8. WP-15b: EDGE VALIDATION (Intent: VALIDATE)
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
edge_validation:
|
||||||
|
gemini: |
|
||||||
|
Bewerte die semantische Validität dieser Verbindung im Wissensgraph.
|
||||||
|
|
||||||
|
KONTEXT DER QUELLE (Chunk):
|
||||||
|
"{chunk_text}"
|
||||||
|
|
||||||
|
ZIEL-NOTIZ: "{target_title}"
|
||||||
|
ZIEL-BESCHREIBUNG (Zusammenfassung):
|
||||||
|
"{target_summary}"
|
||||||
|
|
||||||
|
GEPLANTE RELATION: "{edge_kind}"
|
||||||
|
|
||||||
|
FRAGE: Bestätigt der Kontext der Quelle die Beziehung '{edge_kind}' zum Ziel?
|
||||||
|
REGEL: Antworte NUR mit 'YES' oder 'NO'. Keine Erklärungen oder Smalltalk.
|
||||||
|
openrouter: |
|
||||||
|
Verify semantic relation for graph construction.
|
||||||
|
Source Context: {chunk_text}
|
||||||
|
Target Note: {target_title}
|
||||||
|
Target Summary: {target_summary}
|
||||||
|
Proposed Relation: {edge_kind}
|
||||||
|
Instruction: Does the source context support this relation to the target?
|
||||||
|
Result: Respond ONLY with 'YES' or 'NO'.
|
||||||
|
ollama: |
|
||||||
|
Bewerte die semantische Korrektheit dieser Verbindung.
|
||||||
|
QUELLE: {chunk_text}
|
||||||
|
ZIEL: {target_title} ({target_summary})
|
||||||
|
BEZIEHUNG: {edge_kind}
|
||||||
|
Ist diese Verbindung valide? Antworte NUR mit YES oder NO.
|
||||||
+44
-10
@@ -1,4 +1,4 @@
|
|||||||
version: 1.0
|
version: 1.2
|
||||||
|
|
||||||
scoring:
|
scoring:
|
||||||
# W_sem: skaliert den Term (semantic_score * retriever_weight)
|
# W_sem: skaliert den Term (semantic_score * retriever_weight)
|
||||||
@@ -6,20 +6,54 @@ scoring:
|
|||||||
semantic_weight: 1.0
|
semantic_weight: 1.0
|
||||||
|
|
||||||
# W_edge: skaliert edge_bonus aus dem Subgraph
|
# W_edge: skaliert edge_bonus aus dem Subgraph
|
||||||
# Empfehlung: 0.7 → Graph ist deutlich spürbar, aber überstimmt Semantik nicht komplett
|
# Empfehlung: 0.8 → Graph ist deutlich spürbar, aber überstimmt Semantik nicht komplett
|
||||||
edge_weight: 0.7
|
edge_weight: 0.8
|
||||||
|
|
||||||
# W_cent: skaliert centrality_bonus (Knoten-Zentralität im Subgraph)
|
# W_cent: skaliert centrality_bonus (Knoten-Zentralität im Subgraph)
|
||||||
# Empfehlung: 0.5 → zentrale Knoten werden bevorzugt, aber moderat
|
# Empfehlung: 0.5 → zentrale Knoten werden bevorzugt, aber moderat
|
||||||
centrality_weight: 0.5
|
centrality_weight: 0.5
|
||||||
|
|
||||||
|
# WP-22 Stellschraube: Lifecycle (Status-basiertes Scoring)
|
||||||
|
# Bonus für verifiziertes Wissen, Malus für Entwürfe
|
||||||
|
lifecycle_weights:
|
||||||
|
stable: 1.2 # +20% Bonus
|
||||||
|
active: 1.0 # Standardwert
|
||||||
|
draft: 0.5 # -50% Malus
|
||||||
|
system: 0.0 # Hard Skip via Ingestion
|
||||||
|
|
||||||
# Die nachfolgenden Werte überschreiben die Defaults aus app/core/retriever_config.
|
# Die nachfolgenden Werte überschreiben die Defaults aus app/core/retriever_config.
|
||||||
# Wenn neue Kantentypen, z.B. durch Referenzierung innerhalb einer md-Datei im vault anders gewichtet werden sollen, dann muss hier die Konfiguration erfolgen
|
# Wenn neue Kantentypen, z.B. durch Referenzierung innerhalb einer md-Datei im vault anders gewichtet werden sollen, dann muss hier die Konfiguration erfolgen
|
||||||
edge_types:
|
edge_types:
|
||||||
references: 0.20
|
# --- KATEGORIE 1: LOGIK-BOOSTS (Relevanz-Treiber) ---
|
||||||
depends_on: 0.18
|
# Diese Kanten haben die Kraft, das semantische Ranking aktiv umzugestalten.
|
||||||
related_to: 0.15
|
blocks: 1.6 # Kritisch: Risiken/Blocker müssen sofort sichtbar sein.
|
||||||
similar_to: 0.12
|
solves: 1.5 # Zielführend: Lösungen sind primäre Suchziele.
|
||||||
belongs_to: 0.10
|
depends_on: 1.4 # Logisch: Harte fachliche Abhängigkeit.
|
||||||
next: 0.06
|
resulted_in: 1.4 # Kausal: Ergebnisse und unmittelbare Konsequenzen.
|
||||||
prev: 0.06
|
followed_by: 1.3 # Sequenziell (User): Bewusst gesteuerte Wissenspfade.
|
||||||
|
caused_by: 1.2 # Kausal: Ursachen-Bezug (Basis für Intent-Boost).
|
||||||
|
preceded_by: 1.1 # Sequenziell (User): Rückwärts-Bezug in Logik-Ketten.
|
||||||
|
impacts: 1.2 # Langfristige Auswirkung/Einfluss
|
||||||
|
|
||||||
|
# --- KATEGORIE 2: QUALITATIVER KONTEXT (Stabilitäts-Stützen) ---
|
||||||
|
# Diese Kanten liefern wichtigen Kontext, ohne das Ergebnis zu verfälschen.
|
||||||
|
guides: 1.1 # Qualitativ: Prinzipien oder Werte leiten das Thema.
|
||||||
|
part_of: 1.1 # Strukturell: Zieht übergeordnete Kontexte (Parents) mit hoch.
|
||||||
|
based_on: 0.8 # Fundament: Bezug auf Basis-Werte (kalibriert auf Safe-Retrieval).
|
||||||
|
derived_from: 0.6 # Historisch: Dokumentiert die Herkunft von Wissen.
|
||||||
|
uses: 0.6 # Instrumentell: Genutzte Werkzeuge, Methoden oder Ressourcen.
|
||||||
|
|
||||||
|
# --- KATEGORIE 3: THEMATISCHE NÄHE (Ähnlichkeits-Signal) ---
|
||||||
|
# Diese Werte verhindern den "Drift" in fachfremde Bereiche.
|
||||||
|
similar_to: 0.4 # Analytisch: Thematische Nähe (oft KI-generiert).
|
||||||
|
|
||||||
|
# --- KATEGORIE 4: SYSTEM-NUDGES (Technische Struktur) ---
|
||||||
|
# Reine Orientierungshilfen für das System; fast kein Einfluss auf das Ranking.
|
||||||
|
belongs_to: 0.2 # System: Verknüpft Chunks mit der Note (Metadaten-Träger).
|
||||||
|
next: 0.1 # System: Technische Lesereihenfolge der Absätze.
|
||||||
|
prev: 0.1 # System: Technische Lesereihenfolge der Absätze.
|
||||||
|
|
||||||
|
# --- KATEGORIE 5: WEICHE ASSOZIATIONEN (Rausch-Unterdrückung) ---
|
||||||
|
# Verhindert, dass lose Verknüpfungen das Ergebnis "verwässern".
|
||||||
|
references: 0.1 # Assoziativ: Einfacher Querverweis oder Erwähnung.
|
||||||
|
related_to: 0.05 # Minimal: Schwächste thematische Verbindung.
|
||||||
+216
-67
@@ -1,4 +1,4 @@
|
|||||||
version: 2.4.0 # Optimized for Async Intelligence & Hybrid Router
|
version: 2.7.0 # WP-14 Update: Dynamisierung der Ingestion-Pipeline
|
||||||
|
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
# 1. CHUNKING PROFILES
|
# 1. CHUNKING PROFILES
|
||||||
@@ -7,7 +7,6 @@ version: 2.4.0 # Optimized for Async Intelligence & Hybrid Router
|
|||||||
chunking_profiles:
|
chunking_profiles:
|
||||||
|
|
||||||
# A. SHORT & FAST
|
# A. SHORT & FAST
|
||||||
# Für Glossar, Tasks, Risiken. Kleine Schnipsel.
|
|
||||||
sliding_short:
|
sliding_short:
|
||||||
strategy: sliding_window
|
strategy: sliding_window
|
||||||
enable_smart_edge_allocation: false
|
enable_smart_edge_allocation: false
|
||||||
@@ -16,7 +15,6 @@ chunking_profiles:
|
|||||||
overlap: [30, 50]
|
overlap: [30, 50]
|
||||||
|
|
||||||
# B. STANDARD & FAST
|
# B. STANDARD & FAST
|
||||||
# Der "Traktor": Robust für Quellen, Journal, Daily Logs.
|
|
||||||
sliding_standard:
|
sliding_standard:
|
||||||
strategy: sliding_window
|
strategy: sliding_window
|
||||||
enable_smart_edge_allocation: false
|
enable_smart_edge_allocation: false
|
||||||
@@ -24,10 +22,8 @@ chunking_profiles:
|
|||||||
max: 650
|
max: 650
|
||||||
overlap: [50, 100]
|
overlap: [50, 100]
|
||||||
|
|
||||||
# C. SMART FLOW (Performance-Safe Mode)
|
# C. SMART FLOW (Text-Fluss)
|
||||||
# Für Konzepte, Projekte, Erfahrungen.
|
# Nutzt Sliding Window, aber mit LLM-Kanten-Analyse.
|
||||||
# HINWEIS: 'enable_smart_edge_allocation' ist vorerst FALSE, um Ollama
|
|
||||||
# bei der Generierung nicht zu überlasten. Später wieder aktivieren.
|
|
||||||
sliding_smart_edges:
|
sliding_smart_edges:
|
||||||
strategy: sliding_window
|
strategy: sliding_window
|
||||||
enable_smart_edge_allocation: true
|
enable_smart_edge_allocation: true
|
||||||
@@ -35,12 +31,38 @@ chunking_profiles:
|
|||||||
max: 600
|
max: 600
|
||||||
overlap: [50, 80]
|
overlap: [50, 80]
|
||||||
|
|
||||||
# D. SMART STRUCTURE
|
# D. SMART STRUCTURE (Soft Split)
|
||||||
# Für Profile, Werte, Prinzipien. Trennt hart an Überschriften (H2).
|
# Trennt bevorzugt an H2, fasst aber kleine Abschnitte zusammen ("Soft Mode").
|
||||||
structured_smart_edges:
|
structured_smart_edges:
|
||||||
strategy: by_heading
|
strategy: by_heading
|
||||||
enable_smart_edge_allocation: true
|
enable_smart_edge_allocation: true
|
||||||
split_level: 2
|
split_level: 2
|
||||||
|
strict_heading_split: false
|
||||||
|
max: 600
|
||||||
|
target: 400
|
||||||
|
overlap: [50, 80]
|
||||||
|
|
||||||
|
# E. SMART STRUCTURE STRICT (H2 Hard Split)
|
||||||
|
# Trennt ZWINGEND an jeder H2.
|
||||||
|
# Verhindert, dass "Vater" und "Partner" (Profile) oder Werte verschmelzen.
|
||||||
|
structured_smart_edges_strict:
|
||||||
|
strategy: by_heading
|
||||||
|
enable_smart_edge_allocation: true
|
||||||
|
split_level: 2
|
||||||
|
strict_heading_split: true # Hard Mode
|
||||||
|
max: 600
|
||||||
|
target: 400
|
||||||
|
overlap: [50, 80]
|
||||||
|
|
||||||
|
# F. SMART STRUCTURE DEEP (H3 Hard Split + Merge-Check)
|
||||||
|
# Spezialfall für "Leitbild Prinzipien":
|
||||||
|
# - Trennt H1, H2, H3 hart.
|
||||||
|
# - Aber: Merged "leere" H2 (Tier 2) mit der folgenden H3 (MP1).
|
||||||
|
structured_smart_edges_strict_L3:
|
||||||
|
strategy: by_heading
|
||||||
|
enable_smart_edge_allocation: true
|
||||||
|
split_level: 3
|
||||||
|
strict_heading_split: true
|
||||||
max: 600
|
max: 600
|
||||||
target: 400
|
target: 400
|
||||||
overlap: [50, 80]
|
overlap: [50, 80]
|
||||||
@@ -54,99 +76,212 @@ defaults:
|
|||||||
edge_defaults: []
|
edge_defaults: []
|
||||||
|
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
# 3. TYPE DEFINITIONS
|
# 3. INGESTION SETTINGS (WP-14 Dynamization)
|
||||||
|
# ==============================================================================
|
||||||
|
# Steuert, welche Notizen verarbeitet werden und wie Fallbacks aussehen.
|
||||||
|
ingestion_settings:
|
||||||
|
# Liste der Status-Werte, die beim Import ignoriert werden sollen.
|
||||||
|
ignore_statuses: ["system", "template", "archive", "hidden"]
|
||||||
|
# Standard-Typ, falls kein Typ im Frontmatter angegeben ist.
|
||||||
|
default_note_type: "concept"
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 4. SUMMARY & SCAN SETTINGS
|
||||||
|
# ==============================================================================
|
||||||
|
# Steuert die Tiefe des Pre-Scans für den Context-Cache.
|
||||||
|
summary_settings:
|
||||||
|
max_summary_length: 500
|
||||||
|
pre_scan_depth: 600
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 5. LLM SETTINGS
|
||||||
|
# ==============================================================================
|
||||||
|
# Steuerzeichen und Patterns zur Bereinigung der LLM-Antworten.
|
||||||
|
llm_settings:
|
||||||
|
cleanup_patterns: ["<s>", "</s>", "[OUT]", "[/OUT]", "```json", "```"]
|
||||||
|
|
||||||
|
# ==============================================================================
|
||||||
|
# 6. TYPE DEFINITIONS
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
|
|
||||||
types:
|
types:
|
||||||
|
|
||||||
# --- KERNTYPEN (Hoch priorisiert & Smart) ---
|
|
||||||
|
|
||||||
experience:
|
experience:
|
||||||
chunking_profile: sliding_smart_edges
|
chunking_profile: sliding_smart_edges
|
||||||
retriever_weight: 0.90
|
retriever_weight: 1.10 # Erhöht für biografische Relevanz
|
||||||
edge_defaults: ["derived_from", "references"]
|
edge_defaults: ["derived_from", "references"]
|
||||||
# Hybrid Classifier: Wenn diese Worte fallen, ist es eine Experience
|
detection_keywords: ["erleben", "reagieren", "handeln", "prägen", "reflektieren"]
|
||||||
detection_keywords:
|
|
||||||
- "passiert"
|
|
||||||
- "erlebt"
|
|
||||||
- "gefühl"
|
|
||||||
- "situation"
|
|
||||||
- "stolz"
|
|
||||||
- "geärgert"
|
|
||||||
- "reaktion"
|
|
||||||
- "moment"
|
|
||||||
- "konflikt"
|
|
||||||
# Ghostwriter Schema: Sprechende Anweisungen für besseren Textfluss
|
|
||||||
schema:
|
schema:
|
||||||
- "Situation (Was ist passiert?)"
|
- "Situation (Was ist passiert?)"
|
||||||
- "Meine Reaktion (Was habe ich getan?)"
|
- "Meine Reaktion (Was habe ich getan?)"
|
||||||
- "Ergebnis & Auswirkung"
|
- "Ergebnis & Auswirkung"
|
||||||
- "Reflexion & Learning (Was lerne ich daraus?)"
|
- "Reflexion & Learning (Was lerne ich daraus?)"
|
||||||
|
|
||||||
|
insight:
|
||||||
|
chunking_profile: sliding_smart_edges
|
||||||
|
retriever_weight: 1.20 # Hoch gewichtet für aktuelle Steuerung
|
||||||
|
edge_defaults: ["references", "based_on"]
|
||||||
|
detection_keywords: ["beobachten", "erkennen", "verstehen", "analysieren", "schlussfolgern"]
|
||||||
|
schema:
|
||||||
|
- "Beobachtung (Was sehe ich?)"
|
||||||
|
- "Interpretation (Was bedeutet das?)"
|
||||||
|
- "Bedürfnis (Was steckt dahinter?)"
|
||||||
|
- "Handlungsempfehlung"
|
||||||
|
|
||||||
project:
|
project:
|
||||||
chunking_profile: sliding_smart_edges
|
chunking_profile: sliding_smart_edges
|
||||||
retriever_weight: 0.97
|
retriever_weight: 0.97
|
||||||
edge_defaults: ["references", "depends_on"]
|
edge_defaults: ["references", "depends_on"]
|
||||||
detection_keywords:
|
detection_keywords: ["umsetzen", "planen", "starten", "bauen", "abschließen"]
|
||||||
- "projekt"
|
|
||||||
- "vorhaben"
|
|
||||||
- "ziel ist"
|
|
||||||
- "meilenstein"
|
|
||||||
- "planen"
|
|
||||||
- "starten"
|
|
||||||
- "mission"
|
|
||||||
schema:
|
schema:
|
||||||
- "Mission & Zielsetzung"
|
- "Mission & Zielsetzung"
|
||||||
- "Aktueller Status & Blockaden"
|
- "Aktueller Status & Blockaden"
|
||||||
- "Nächste konkrete Schritte"
|
- "Nächste konkrete Schritte"
|
||||||
- "Stakeholder & Ressourcen"
|
|
||||||
|
|
||||||
decision:
|
decision:
|
||||||
chunking_profile: structured_smart_edges
|
chunking_profile: structured_smart_edges_strict
|
||||||
retriever_weight: 1.00 # MAX: Entscheidungen sind Gesetz
|
retriever_weight: 1.00
|
||||||
edge_defaults: ["caused_by", "references"]
|
edge_defaults: ["caused_by", "references"]
|
||||||
detection_keywords:
|
detection_keywords: ["entscheiden", "wählen", "abwägen", "priorisieren", "festlegen"]
|
||||||
- "entschieden"
|
|
||||||
- "wahl"
|
|
||||||
- "optionen"
|
|
||||||
- "alternativen"
|
|
||||||
- "beschluss"
|
|
||||||
- "adr"
|
|
||||||
schema:
|
schema:
|
||||||
- "Kontext & Problemstellung"
|
- "Kontext & Problemstellung"
|
||||||
- "Betrachtete Optionen (Alternativen)"
|
- "Betrachtete Optionen"
|
||||||
- "Die Entscheidung"
|
- "Die Entscheidung"
|
||||||
- "Begründung (Warum diese Wahl?)"
|
- "Begründung"
|
||||||
|
|
||||||
# --- PERSÖNLICHKEIT & IDENTITÄT ---
|
# --- PERSÖNLICHKEIT & IDENTITÄT ---
|
||||||
|
|
||||||
value:
|
value:
|
||||||
chunking_profile: structured_smart_edges
|
chunking_profile: structured_smart_edges_strict
|
||||||
retriever_weight: 1.00
|
retriever_weight: 1.00
|
||||||
edge_defaults: ["related_to"]
|
edge_defaults: ["related_to"]
|
||||||
detection_keywords: ["wert", "wichtig ist", "moral", "ethik"]
|
detection_keywords: ["werten", "achten", "verpflichten", "bedeuten"]
|
||||||
schema: ["Definition", "Warum mir das wichtig ist", "Leitsätze für den Alltag"]
|
schema:
|
||||||
|
- "Definition"
|
||||||
|
- "Warum mir das wichtig ist"
|
||||||
|
- "Leitsätze"
|
||||||
|
|
||||||
principle:
|
principle:
|
||||||
chunking_profile: structured_smart_edges
|
chunking_profile: structured_smart_edges_strict_L3
|
||||||
retriever_weight: 0.95
|
retriever_weight: 0.95
|
||||||
edge_defaults: ["derived_from", "references"]
|
edge_defaults: ["derived_from", "references"]
|
||||||
detection_keywords: ["prinzip", "regel", "grundsatz", "leitlinie"]
|
detection_keywords: ["leiten", "steuern", "ausrichten", "handhaben"]
|
||||||
schema: ["Das Prinzip", "Anwendung & Beispiele"]
|
schema:
|
||||||
|
- "Das Prinzip"
|
||||||
|
- "Anwendung & Beispiele"
|
||||||
|
|
||||||
|
trait:
|
||||||
|
chunking_profile: structured_smart_edges_strict
|
||||||
|
retriever_weight: 1.10
|
||||||
|
edge_defaults: ["related_to"]
|
||||||
|
detection_keywords: ["begeistern", "können", "auszeichnen", "befähigen", "stärken"]
|
||||||
|
schema:
|
||||||
|
- "Eigenschaft / Talent"
|
||||||
|
- "Beispiele aus der Praxis"
|
||||||
|
- "Potenzial für die Zukunft"
|
||||||
|
|
||||||
|
obstacle:
|
||||||
|
chunking_profile: structured_smart_edges_strict
|
||||||
|
retriever_weight: 1.00
|
||||||
|
edge_defaults: ["blocks", "related_to"]
|
||||||
|
detection_keywords: ["blockieren", "fürchten", "vermeiden", "hindern", "zweifeln"]
|
||||||
|
schema:
|
||||||
|
- "Beschreibung der Hürde"
|
||||||
|
- "Ursprung / Auslöser"
|
||||||
|
- "Auswirkung auf Ziele"
|
||||||
|
- "Gegenstrategie"
|
||||||
|
|
||||||
belief:
|
belief:
|
||||||
chunking_profile: sliding_short
|
chunking_profile: sliding_short
|
||||||
retriever_weight: 0.90
|
retriever_weight: 0.90
|
||||||
edge_defaults: ["related_to"]
|
edge_defaults: ["related_to"]
|
||||||
detection_keywords: ["glaube", "überzeugung", "denke dass", "meinung"]
|
detection_keywords: ["glauben", "meinen", "annehmen", "überzeugen"]
|
||||||
schema: ["Der Glaubenssatz", "Ursprung & Reflexion"]
|
schema:
|
||||||
|
- "Der Glaubenssatz"
|
||||||
|
- "Ursprung & Reflexion"
|
||||||
|
|
||||||
profile:
|
profile:
|
||||||
chunking_profile: structured_smart_edges
|
chunking_profile: structured_smart_edges_strict
|
||||||
retriever_weight: 0.70
|
retriever_weight: 0.70
|
||||||
edge_defaults: ["references", "related_to"]
|
edge_defaults: ["references", "related_to"]
|
||||||
schema: ["Rolle / Identität", "Fakten & Daten", "Historie"]
|
detection_keywords: ["verkörpern", "verantworten", "agieren", "repräsentieren"]
|
||||||
|
schema:
|
||||||
|
- "Rolle / Identität"
|
||||||
|
- "Fakten & Daten"
|
||||||
|
- "Historie"
|
||||||
|
|
||||||
|
|
||||||
|
idea:
|
||||||
|
chunking_profile: sliding_short
|
||||||
|
retriever_weight: 0.70
|
||||||
|
edge_defaults: ["leads_to", "references"]
|
||||||
|
detection_keywords: ["einfall", "gedanke", "potenzial", "möglichkeit"]
|
||||||
|
schema:
|
||||||
|
- "Der Kerngedanke"
|
||||||
|
- "Potenzial & Auswirkung"
|
||||||
|
- "Nächste Schritte"
|
||||||
|
|
||||||
|
skill:
|
||||||
|
chunking_profile: sliding_smart_edges
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
detection_keywords: ["lernen", "beherrschen", "üben", "fertigkeit", "kompetenz"]
|
||||||
|
schema:
|
||||||
|
- "Definition der Fähigkeit"
|
||||||
|
- "Aktueller Stand & Lernpfad"
|
||||||
|
- "Evidenz (Proof of Work)"
|
||||||
|
|
||||||
|
habit:
|
||||||
|
chunking_profile: sliding_short
|
||||||
|
retriever_weight: 0.85
|
||||||
|
edge_defaults: ["related_to", "triggered_by"]
|
||||||
|
detection_keywords: ["gewohnheit", "routine", "automatismus", "immer wenn"]
|
||||||
|
schema:
|
||||||
|
- "Auslöser (Trigger)"
|
||||||
|
- "Routine (Handlung)"
|
||||||
|
- "Belohnung (Reward)"
|
||||||
|
- "Strategie"
|
||||||
|
|
||||||
|
need:
|
||||||
|
chunking_profile: sliding_smart_edges
|
||||||
|
retriever_weight: 1.05
|
||||||
|
edge_defaults: ["related_to", "impacts"]
|
||||||
|
detection_keywords: ["bedürfnis", "brauchen", "mangel", "erfüllung"]
|
||||||
|
schema:
|
||||||
|
- "Das Bedürfnis"
|
||||||
|
- "Zustand (Mangel vs. Erfüllung)"
|
||||||
|
- "Bezug zu Werten"
|
||||||
|
|
||||||
|
motivation:
|
||||||
|
chunking_profile: sliding_smart_edges
|
||||||
|
retriever_weight: 0.95
|
||||||
|
edge_defaults: ["drives", "references"]
|
||||||
|
detection_keywords: ["motivation", "antrieb", "warum", "energie"]
|
||||||
|
schema:
|
||||||
|
- "Der Antrieb"
|
||||||
|
- "Zielbezug"
|
||||||
|
- "Energiequelle"
|
||||||
|
|
||||||
|
bias:
|
||||||
|
chunking_profile: sliding_short
|
||||||
|
retriever_weight: 0.80
|
||||||
|
edge_defaults: ["affects", "related_to"]
|
||||||
|
detection_keywords: ["denkfehler", "verzerrung", "vorurteil", "falle"]
|
||||||
|
schema: ["Beschreibung der Verzerrung", "Typische Situationen", "Gegenstrategie"]
|
||||||
|
|
||||||
|
state:
|
||||||
|
chunking_profile: sliding_short
|
||||||
|
retriever_weight: 0.60
|
||||||
|
edge_defaults: ["impacts"]
|
||||||
|
detection_keywords: ["stimmung", "energie", "gefühl", "verfassung"]
|
||||||
|
schema: ["Aktueller Zustand", "Auslöser", "Auswirkung auf den Tag"]
|
||||||
|
|
||||||
|
boundary:
|
||||||
|
chunking_profile: sliding_smart_edges
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["protects", "related_to"]
|
||||||
|
detection_keywords: ["grenze", "nein sagen", "limit", "schutz"]
|
||||||
|
schema: ["Die Grenze", "Warum sie wichtig ist", "Konsequenz bei Verletzung"]
|
||||||
# --- STRATEGIE & RISIKO ---
|
# --- STRATEGIE & RISIKO ---
|
||||||
|
|
||||||
goal:
|
goal:
|
||||||
@@ -159,8 +294,8 @@ types:
|
|||||||
chunking_profile: sliding_short
|
chunking_profile: sliding_short
|
||||||
retriever_weight: 0.85
|
retriever_weight: 0.85
|
||||||
edge_defaults: ["related_to", "blocks"]
|
edge_defaults: ["related_to", "blocks"]
|
||||||
detection_keywords: ["risiko", "gefahr", "bedrohung", "problem", "angst"]
|
detection_keywords: ["risiko", "gefahr", "bedrohung"]
|
||||||
schema: ["Beschreibung des Risikos", "Mögliche Auswirkungen", "Gegenmaßnahmen"]
|
schema: ["Beschreibung des Risikos", "Auswirkungen", "Gegenmaßnahmen"]
|
||||||
|
|
||||||
# --- BASIS & WISSEN ---
|
# --- BASIS & WISSEN ---
|
||||||
|
|
||||||
@@ -168,10 +303,7 @@ types:
|
|||||||
chunking_profile: sliding_smart_edges
|
chunking_profile: sliding_smart_edges
|
||||||
retriever_weight: 0.60
|
retriever_weight: 0.60
|
||||||
edge_defaults: ["references", "related_to"]
|
edge_defaults: ["references", "related_to"]
|
||||||
schema:
|
schema: ["Definition", "Kontext", "Verwandte Konzepte"]
|
||||||
- "Definition"
|
|
||||||
- "Kontext & Hintergrund"
|
|
||||||
- "Verwandte Konzepte"
|
|
||||||
|
|
||||||
task:
|
task:
|
||||||
chunking_profile: sliding_short
|
chunking_profile: sliding_short
|
||||||
@@ -183,19 +315,36 @@ types:
|
|||||||
chunking_profile: sliding_standard
|
chunking_profile: sliding_standard
|
||||||
retriever_weight: 0.80
|
retriever_weight: 0.80
|
||||||
edge_defaults: ["references", "related_to"]
|
edge_defaults: ["references", "related_to"]
|
||||||
schema: ["Log-Eintrag", "Gedanken & Erkenntnisse"]
|
schema: ["Log-Eintrag", "Gedanken"]
|
||||||
|
|
||||||
source:
|
source:
|
||||||
chunking_profile: sliding_standard
|
chunking_profile: sliding_standard
|
||||||
retriever_weight: 0.50
|
retriever_weight: 0.50
|
||||||
edge_defaults: []
|
edge_defaults: []
|
||||||
schema:
|
schema: ["Metadaten", "Zusammenfassung", "Zitate"]
|
||||||
- "Metadaten (Autor, URL, Datum)"
|
|
||||||
- "Kernaussage / Zusammenfassung"
|
|
||||||
- "Zitate & Notizen"
|
|
||||||
|
|
||||||
glossary:
|
glossary:
|
||||||
chunking_profile: sliding_short
|
chunking_profile: sliding_short
|
||||||
retriever_weight: 0.40
|
retriever_weight: 0.40
|
||||||
edge_defaults: ["related_to"]
|
edge_defaults: ["related_to"]
|
||||||
schema: ["Begriff", "Definition"]
|
schema: ["Begriff", "Definition"]
|
||||||
|
|
||||||
|
person:
|
||||||
|
chunking_profile: sliding_standard
|
||||||
|
retriever_weight: 0.50
|
||||||
|
edge_defaults: ["related_to"]
|
||||||
|
schema: ["Rolle", "Beziehung", "Kontext"]
|
||||||
|
|
||||||
|
event:
|
||||||
|
chunking_profile: sliding_standard
|
||||||
|
retriever_weight: 0.60
|
||||||
|
edge_defaults: ["related_to"]
|
||||||
|
schema: ["Datum & Ort", "Teilnehmer", "Ergebnisse"]
|
||||||
|
|
||||||
|
# --- FALLBACK ---
|
||||||
|
|
||||||
|
default:
|
||||||
|
chunking_profile: sliding_standard
|
||||||
|
retriever_weight: 1.00
|
||||||
|
edge_defaults: ["references"]
|
||||||
|
schema: ["Inhalt"]
|
||||||
@@ -18,9 +18,12 @@ Das Repository ist in **logische Domänen** unterteilt.
|
|||||||
*Zielgruppe: Alle*
|
*Zielgruppe: Alle*
|
||||||
| Datei | Inhalt & Zweck |
|
| Datei | Inhalt & Zweck |
|
||||||
| :--- | :--- |
|
| :--- | :--- |
|
||||||
|
| `README.md` | **Einstiegspunkt.** Übersicht über die Dokumentationsstruktur, Schnellzugriff nach Rollen und Navigation. |
|
||||||
|
| `00_quickstart.md` | **Schnellstart.** Installation und erste Schritte in 15 Minuten. Ideal für neue Benutzer. |
|
||||||
| `00_vision_and_strategy.md` | **Strategie.** Warum bauen wir das? Prinzipien (Privacy, Local-First), High-Level Architektur. |
|
| `00_vision_and_strategy.md` | **Strategie.** Warum bauen wir das? Prinzipien (Privacy, Local-First), High-Level Architektur. |
|
||||||
| `00_glossary.md` | **Definitionen.** Was bedeutet "Smart Edge", "Traffic Control", "Chunk"? Verhindert Begriffsverwirrung. |
|
| `00_glossary.md` | **Definitionen.** Was bedeutet "Smart Edge", "Traffic Control", "Chunk"? Verhindert Begriffsverwirrung. |
|
||||||
| `00_documentation_map.md` | **Dieser Index.** Navigationshilfe. |
|
| `00_documentation_map.md` | **Dieser Index.** Navigationshilfe. |
|
||||||
|
| `00_quality_checklist.md` | **Qualitätsprüfung.** Systematische Checkliste zur Vollständigkeitsprüfung für alle Rollen. |
|
||||||
|
|
||||||
### 📂 01_User_Manual (Anwendung)
|
### 📂 01_User_Manual (Anwendung)
|
||||||
*Zielgruppe: Mindmaster, Autoren, Power-User*
|
*Zielgruppe: Mindmaster, Autoren, Power-User*
|
||||||
@@ -28,6 +31,8 @@ Das Repository ist in **logische Domänen** unterteilt.
|
|||||||
| :--- | :--- |
|
| :--- | :--- |
|
||||||
| `01_chat_usage_guide.md` | **Bedienung.** Wie steuere ich die Personas (Berater, Spiegel)? Wie nutze ich das Feedback? |
|
| `01_chat_usage_guide.md` | **Bedienung.** Wie steuere ich die Personas (Berater, Spiegel)? Wie nutze ich das Feedback? |
|
||||||
| `01_knowledge_design.md` | **Content-Regeln.** Die "Bibel" für den Vault. Erklärt Note-Typen, Matrix-Logik und Markdown-Syntax. |
|
| `01_knowledge_design.md` | **Content-Regeln.** Die "Bibel" für den Vault. Erklärt Note-Typen, Matrix-Logik und Markdown-Syntax. |
|
||||||
|
| `01_authoring_guidelines.md` | **Content strukturieren.** Primäres Werkzeug, um Wissen so zu strukturieren, dass Mindnet die Persönlichkeit spiegelt, empathisch reagiert und strategisch berät. |
|
||||||
|
| `01_obsidian_integration_guide.md` | **Obsidian Setup.** Technische Anleitung für die Integration von Mindnet mit Obsidian (Templater, Skripte, Workflows). |
|
||||||
|
|
||||||
### 📂 02_Concepts (Fachliche Logik)
|
### 📂 02_Concepts (Fachliche Logik)
|
||||||
*Zielgruppe: Architekten, Product Owner*
|
*Zielgruppe: Architekten, Product Owner*
|
||||||
@@ -35,6 +40,7 @@ Das Repository ist in **logische Domänen** unterteilt.
|
|||||||
| :--- | :--- |
|
| :--- | :--- |
|
||||||
| `02_concept_graph_logic.md` | **Graph-Theorie.** Abstrakte Erklärung von Knoten, Kanten, Provenance und Idempotenz. |
|
| `02_concept_graph_logic.md` | **Graph-Theorie.** Abstrakte Erklärung von Knoten, Kanten, Provenance und Idempotenz. |
|
||||||
| `02_concept_ai_personality.md`| **KI-Verhalten.** Konzepte hinter dem Hybrid Router, Empathie-Modell und "Teach-the-AI". |
|
| `02_concept_ai_personality.md`| **KI-Verhalten.** Konzepte hinter dem Hybrid Router, Empathie-Modell und "Teach-the-AI". |
|
||||||
|
| `02_concept_architecture_patterns.md` | **Architektur-Patterns.** Design-Entscheidungen, modulare Struktur (WP-14), Resilienz-Patterns und Erweiterbarkeit. |
|
||||||
|
|
||||||
### 📂 03_Technical_Reference (Technik & Code)
|
### 📂 03_Technical_Reference (Technik & Code)
|
||||||
*Zielgruppe: Entwickler, DevOps. (Enthält JSON/YAML Beispiele)*
|
*Zielgruppe: Entwickler, DevOps. (Enthält JSON/YAML Beispiele)*
|
||||||
@@ -46,18 +52,23 @@ Das Repository ist in **logische Domänen** unterteilt.
|
|||||||
| `03_tech_chat_backend.md` | **API & LLM.** Implementation des Routers, Traffic Control (Semaphore) und Feedback-Traceability. |
|
| `03_tech_chat_backend.md` | **API & LLM.** Implementation des Routers, Traffic Control (Semaphore) und Feedback-Traceability. |
|
||||||
| `03_tech_frontend.md` | **UI & Graph.** Architektur des Streamlit-Frontends, State-Management, Cytoscape-Integration und Editor-Logik. |
|
| `03_tech_frontend.md` | **UI & Graph.** Architektur des Streamlit-Frontends, State-Management, Cytoscape-Integration und Editor-Logik. |
|
||||||
| `03_tech_configuration.md` | **Config.** Referenztabellen für `.env`, `types.yaml` und `retriever.yaml`. |
|
| `03_tech_configuration.md` | **Config.** Referenztabellen für `.env`, `types.yaml` und `retriever.yaml`. |
|
||||||
|
| `03_tech_api_reference.md` | **API-Referenz.** Vollständige Dokumentation aller Endpunkte (`/query`, `/chat`, `/ingest`, `/graph`, etc.). |
|
||||||
|
|
||||||
### 📂 04_Operations (Betrieb)
|
### 📂 04_Operations (Betrieb)
|
||||||
*Zielgruppe: Administratoren*
|
*Zielgruppe: Administratoren*
|
||||||
| Datei | Inhalt & Zweck |
|
| Datei | Inhalt & Zweck |
|
||||||
| :--- | :--- |
|
| :--- | :--- |
|
||||||
| `04_admin_operations.md` | **Runbook.** Installation, Docker-Setup, Backup/Restore, Troubleshooting Guide. |
|
| `04_admin_operations.md` | **Runbook.** Installation, Docker-Setup, Backup/Restore, Troubleshooting Guide. |
|
||||||
|
| `04_server_operation_manual.md` | **Server-Betrieb.** Detaillierte Dokumentation für den Betrieb auf llm-node (Systemd, Borgmatic, Disaster Recovery). |
|
||||||
|
| `04_deployment_guide.md` | **Deployment.** CI/CD-Pipelines, Rollout-Strategien, Versionierung, Rollback und Pre/Post-Deployment-Checklisten. |
|
||||||
|
|
||||||
### 📂 05_Development (Code)
|
### 📂 05_Development (Code)
|
||||||
*Zielgruppe: Entwickler*
|
*Zielgruppe: Entwickler*
|
||||||
| Datei | Inhalt & Zweck |
|
| Datei | Inhalt & Zweck |
|
||||||
| :--- | :--- |
|
| :--- | :--- |
|
||||||
| `05_developer_guide.md` | **Workflow.** Hardware-Setup (Win/Pi/Beelink), Git-Flow, Test-Befehle, Modul-Interna. |
|
| `05_developer_guide.md` | **Workflow.** Hardware-Setup (Win/Pi/Beelink), Git-Flow, Test-Befehle, Modul-Interna. |
|
||||||
|
| `05_genai_best_practices.md` | **AI Workflow.** Prompt-Library, Templates und Best Practices für die Entwicklung mit LLMs. |
|
||||||
|
| `05_testing_guide.md` | **Testing.** Umfassender Test-Guide: Strategien, Frameworks, Test-Daten, Best Practices. |
|
||||||
|
|
||||||
### 📂 06_Roadmap & 99_Archive
|
### 📂 06_Roadmap & 99_Archive
|
||||||
*Zielgruppe: Projektleitung*
|
*Zielgruppe: Projektleitung*
|
||||||
@@ -80,7 +91,9 @@ Nutze diese Matrix, wenn du ein Workpackage bearbeitest, um die Dokumentation ko
|
|||||||
| **Retrieval / Scoring** | `03_tech_retrieval_scoring.md` (Formeln anpassen) |
|
| **Retrieval / Scoring** | `03_tech_retrieval_scoring.md` (Formeln anpassen) |
|
||||||
| **Frontend / Visualisierung** | 1. `03_tech_frontend.md` (Technische Details)<br>2. `01_chat_usage_guide.md` (Bedienung) |
|
| **Frontend / Visualisierung** | 1. `03_tech_frontend.md` (Technische Details)<br>2. `01_chat_usage_guide.md` (Bedienung) |
|
||||||
| **Chat-Logik / Prompts**| 1. `02_concept_ai_personality.md` (Konzept)<br>2. `03_tech_chat_backend.md` (Tech)<br>3. `01_chat_usage_guide.md` (User-Sicht) |
|
| **Chat-Logik / Prompts**| 1. `02_concept_ai_personality.md` (Konzept)<br>2. `03_tech_chat_backend.md` (Tech)<br>3. `01_chat_usage_guide.md` (User-Sicht) |
|
||||||
| **Deployment / Server** | `04_admin_operations.md` |
|
| **Architektur / Design-Patterns** | 1. `02_concept_architecture_patterns.md` (Patterns & Entscheidungen)<br>2. `02_concept_graph_logic.md` (Graph-Theorie)<br>3. `05_developer_guide.md` (Modulare Struktur) |
|
||||||
|
| **Deployment / Server** | 1. `04_deployment_guide.md` (CI/CD, Rollout)<br>2. `04_admin_operations.md` (Installation, Wartung)<br>3. `04_server_operation_manual.md` (Server-Betrieb) |
|
||||||
|
| **Testing / QA** | 1. `05_testing_guide.md` (Test-Strategien & Frameworks)<br>2. `05_developer_guide.md` (Test-Befehle) |
|
||||||
| **Neuen Features (Allg.)**| `06_active_roadmap.md` (Status Update) |
|
| **Neuen Features (Allg.)**| `06_active_roadmap.md` (Status Update) |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -108,4 +121,38 @@ Damit dieses System wartbar bleibt (auch für KI-Agenten wie NotebookLM), gelten
|
|||||||
audience: developer
|
audience: developer
|
||||||
context: "Beschreibung der Scoring-Formel."
|
context: "Beschreibung der Scoring-Formel."
|
||||||
---
|
---
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Schnellzugriff & Empfehlungen
|
||||||
|
|
||||||
|
### Für neue Benutzer
|
||||||
|
1. Starte mit **[Schnellstart](00_quickstart.md)** für die Installation
|
||||||
|
2. Lese **[Vision & Strategie](00_vision_and_strategy.md)** für das große Bild
|
||||||
|
3. Nutze **[Chat Usage Guide](../01_User_Manual/01_chat_usage_guide.md)** für die ersten Schritte
|
||||||
|
|
||||||
|
### Für Entwickler
|
||||||
|
1. **[Developer Guide](../05_Development/05_developer_guide.md)** - Umfassender technischer Guide
|
||||||
|
2. **[Technical References](../03_Technical_References/)** - Detaillierte API- und Architektur-Dokumentation
|
||||||
|
3. **[GenAI Best Practices](../05_Development/05_genai_best_practices.md)** - Workflow mit LLMs
|
||||||
|
|
||||||
|
### Für Administratoren
|
||||||
|
1. **[Admin Operations](../04_Operations/04_admin_operations.md)** - Installation und Wartung
|
||||||
|
2. **[Server Operations Manual](../04_Operations/04_server_operation_manual.md)** - Server-Betrieb und Disaster Recovery
|
||||||
|
3. **[Troubleshooting Guide](../04_Operations/04_admin_operations.md#33-troubleshooting-guide)** - Häufige Probleme und Lösungen
|
||||||
|
|
||||||
|
### Für Autoren
|
||||||
|
1. **[Knowledge Design](../01_User_Manual/01_knowledge_design.md)** - Content-Regeln und Best Practices
|
||||||
|
2. **[Authoring Guidelines](../01_User_Manual/01_authoring_guidelines.md)** - Strukturierung für den Digitalen Zwilling
|
||||||
|
3. **[Obsidian Integration](../01_User_Manual/01_obsidian_integration_guide.md)** - Workflow-Optimierung
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Dokumentations-Status
|
||||||
|
|
||||||
|
**Aktuelle Version:** 2.9.1
|
||||||
|
**Letzte Aktualisierung:** 2025-01-XX
|
||||||
|
**Status:** ✅ Vollständig und aktiv gepflegt
|
||||||
|
|
||||||
|
**Hinweis:** Diese Dokumentation wird kontinuierlich aktualisiert. Bei Fragen oder Verbesserungsvorschlägen bitte im Repository melden.
|
||||||
@@ -2,41 +2,52 @@
|
|||||||
doc_type: glossary
|
doc_type: glossary
|
||||||
audience: all
|
audience: all
|
||||||
status: active
|
status: active
|
||||||
version: 2.6
|
version: 2.9.1
|
||||||
context: "Definitionen zentraler Begriffe und Entitäten im Mindnet-System."
|
context: "Zentrales Glossar für Mindnet v2.8. Enthält Definitionen zu Hybrid-Cloud Resilienz, WP-14 Modularisierung, WP-15b Two-Pass Ingestion und Mistral-safe Parsing."
|
||||||
---
|
---
|
||||||
|
|
||||||
# Mindnet Glossar
|
# Mindnet Glossar
|
||||||
|
|
||||||
**Quellen:** `appendix.md`, `Overview.md`
|
**Quellen:** `01_edge_vocabulary.md`, `llm_service.py`, `ingestion.py`, `edge_registry.py`, `registry.py`, `qdrant.py`
|
||||||
|
|
||||||
## Kern-Entitäten
|
## Kern-Entitäten
|
||||||
|
|
||||||
* **Note:** Repräsentiert eine Markdown-Datei. Die fachliche Haupteinheit.
|
* **Note:** Repräsentiert eine Markdown-Datei. Die fachliche Haupteinheit. Verfügt über einen **Status** (stable, draft, system), der das Scoring beeinflusst.
|
||||||
* **Chunk:** Ein Textabschnitt einer Note (meist 512 Tokens). Die technische Sucheinheit (Vektor).
|
* **Chunk:** Ein Textabschnitt einer Note. Die technische Sucheinheit (Vektor).
|
||||||
* **Edge:** Eine gerichtete Verbindung zwischen zwei Knoten (Chunks oder Notes).
|
* **Edge:** Eine gerichtete Verbindung zwischen zwei Knoten. Wird in WP-22 durch die Registry validiert. Seit v2.9.1 unterstützt Edges **Section-basierte Links** (`target_section`), sodass mehrere Kanten zwischen denselben Knoten existieren können, wenn sie auf verschiedene Abschnitte zeigen.
|
||||||
* **Vault:** Der lokale Ordner mit den Markdown-Dateien (Source of Truth).
|
* **Vault:** Der lokale Ordner mit den Markdown-Dateien (Source of Truth).
|
||||||
* **Frontmatter:** Der YAML-Header am Anfang einer Notiz (enthält `id`, `type`, `title`).
|
* **Frontmatter:** Der YAML-Header am Anfang einer Notiz (enthält `id`, `type`, `title`, `status`).
|
||||||
|
|
||||||
## Komponenten
|
## Komponenten
|
||||||
|
|
||||||
* **Importer:** Das Python-Skript (`ingestion.py`), das Markdown liest und in Qdrant schreibt.
|
* **Edge Registry:** Der zentrale Dienst (SSOT), der Kanten-Typen validiert und Aliase in kanonische Typen auflöst. Nutzt `01_edge_vocabulary.md` als Basis.
|
||||||
* **Retriever:** Die Komponente, die sucht. Nutzt hybrides Scoring (Semantik + Graph).
|
* **LLM Service:** Der Hybrid-Client (v3.3.6), der Anfragen zwischen OpenRouter, Google Gemini und lokalem Ollama routet. Verwaltet Cloud-Timeouts und Quoten-Management. Nutzt zur Text-Bereinigung nun die neutrale `registry.py`, um Circular Imports zu vermeiden.
|
||||||
* **Decision Engine:** Teil des Routers, der entscheidet, wie auf eine Anfrage reagiert wird (z.B. Strategie wählen).
|
* **Retriever:** Besteht in v2.7+ aus der Orchestrierung (`retriever.py`) und der mathematischen Scoring-Engine (`retriever_scoring.py`). Seit WP-14 im Paket `app.core.retrieval` gekapselt.
|
||||||
* **Hybrid Router v5:** Die Logik, die erkennt, ob der User eine Frage stellt (`RAG`) oder einen Befehl gibt (`INTERVIEW`).
|
* **Decision Engine:** Teil des Routers, der Intents erkennt und entsprechende **Boost-Faktoren** für das Retrieval injiziert.
|
||||||
* **Draft Editor:** Die Web-UI-Komponente, in der generierte Notizen bearbeitet werden.
|
* **Traffic Control:** Verwaltet Prioritäten und drosselt Hintergrund-Tasks (z.B. Smart Edges) mittels Semaphoren und Timeouts (45s) zur Vermeidung von System-Hangs.
|
||||||
* **Traffic Control:** Ein Mechanismus im `LLMService`, der Chat-Anfragen priorisiert und Hintergrund-Jobs (wie Import) drosselt.
|
* **Unknown Edges Log:** Die Datei `unknown_edges.jsonl`, in der das System Kanten-Typen protokolliert, die nicht im Dictionary gefunden wurden.
|
||||||
|
* **Database Package (WP-14):** Zentralisiertes Infrastruktur-Paket (`app.core.database`), das den Qdrant-Client (`qdrant.py`) und das Point-Mapping (`qdrant_points.py`) verwaltet.
|
||||||
|
* **LocalBatchCache (WP-15b):** Ein globaler In-Memory-Index, der während des Pass 1 Scans aufgebaut wird und Metadaten (IDs, Titel, Summaries) aller Notizen für die Kantenvalidierung bereithält.
|
||||||
|
|
||||||
## Konzepte & Features
|
## Konzepte & Features
|
||||||
|
|
||||||
* **Active Intelligence:** Feature im Web-Editor, das während des Schreibens automatisch Links vorschlägt.
|
* **Hybrid Provider Cascade:** Die intelligente Reihenfolge der Modell-Ansprache. Schlägt die Cloud (OpenRouter/Gemini) fehl, erfolgt nach Retries ein Fallback auf den lokalen Ollama (Quoten-Schutz).
|
||||||
* **Smart Edge Allocation (WP15):** Ein KI-Verfahren, das prüft, ob ein Link in einer Notiz für einen spezifischen Textabschnitt relevant ist.
|
* **Deep Fallback (v2.11.14, WP20):** Ein inhaltsbasierter Rettungsmechanismus in der Ingestion. Im Gegensatz zum technischen Fallback (bei Verbindungsfehlern) wird der Deep Fallback ausgelöst, wenn ein Cloud-Modell zwar technisch erfolgreich antwortet, aber inhaltlich keine verwertbaren Daten liefert (z. B. bei "Data Policy Violations").
|
||||||
* **Healing Parser:** UI-Funktion, die fehlerhaften Output des LLMs (z.B. defektes YAML) automatisch repariert.
|
* **Silent Refusal (WP20):** Ein Zustand, in dem Cloud-Provider (wie OpenRouter) die Verarbeitung eines Dokuments aufgrund interner Filter ("No data training") verweigern, ohne einen HTTP-Fehler zu senden. Wird durch Deep Fallback abgefangen.
|
||||||
* **Explanation Layer:** Die Schicht, die dem Nutzer erklärt, *warum* ein Suchergebnis gefunden wurde (z.B. "Weil Projekt X davon abhängt").
|
* **Rate-Limit Resilience (WP-20):** Automatisierte Erkennung von HTTP 429 Fehlern. Das System pausiert (konfigurierbar via `LLM_RATE_LIMIT_WAIT`) und wiederholt den Cloud-Call, bevor der langsame Fallback ausgelöst wird.
|
||||||
* **Provenance:** Die Herkunft einer Kante.
|
* **Mistral-safe Parsing:** Robuste Extraktions-Logik in Ingestion und Analyzer, die technische Steuerzeichen (`<s>`, `[OUT]`) und Framework-Tags erkennt und entfernt, um valides JSON aus Free-Modellen zu gewinnen.
|
||||||
* `explicit`: Vom Mensch geschrieben.
|
* **Lifecycle Scoring (WP-22):** Ein Mechanismus, der die Relevanz einer Notiz basierend auf ihrem Status gewichtet (z.B. Bonus für `stable`, Malus für `draft`).
|
||||||
* `smart`: Vom LLM validiert.
|
* **Intent Boosting:** Dynamische Erhöhung der Kanten-Gewichte basierend auf der Nutzerfrage (z.B. Fokus auf `caused_by` bei "Warum"-Fragen).
|
||||||
* `rule`: Durch Config-Regel erzeugt.
|
* **Provenance Weighting:** Gewichtung einer Kante nach ihrer Herkunft:
|
||||||
* **Matrix Logic:** Regelwerk, das den Typ einer Kante basierend auf Quell- und Ziel-Typ bestimmt (z.B. Erfahrung -> Wert = `based_on`).
|
* `explicit`: Vom Mensch gesetzt (Prio 1).
|
||||||
* **Idempotenz:** Die Eigenschaft des Importers, bei mehrfacher Ausführung dasselbe Ergebnis zu liefern ohne Duplikate.
|
* `semantic_ai`: Von der KI im Turbo-Mode extrahiert und validiert (Prio 2).
|
||||||
* **Resurrection Pattern:** UI-Technik, um User-Eingaben beim Tab-Wechsel zu erhalten.
|
* `structure`: Durch System-Regeln/Matrix erzeugt (Prio 3).
|
||||||
|
* **Smart Edge Allocation (WP-15b):** KI-Verfahren zur Relevanzprüfung von Links für spezifische Textabschnitte. Validiert Kandidaten semantisch gegen das Ziel im LocalBatchCache.
|
||||||
|
* **Matrix Logic:** Bestimmung des Kanten-Typs basierend auf Quell- und Ziel-Entität (z.B. Erfahrung -> Wert = `based_on`).
|
||||||
|
* **Two-Pass Workflow (WP-15b):** Optimiertes Ingestion-Verfahren:
|
||||||
|
* **Pass 1 (Pre-Scan):** Schnelles Scannen aller Dateien zur Befüllung des LocalBatchCache.
|
||||||
|
* **Pass 2 (Semantic Processing):** Tiefenverarbeitung (Chunking, Embedding, Validierung) nur für geänderte Dateien.
|
||||||
|
* **Circular Import Registry (WP-14):** Entkopplung von Kern-Logik (wie Textbereinigung) in eine neutrale `registry.py`, um Abhängigkeitsschleifen zwischen Diensten und Ingestion-Utilities zu verhindern.
|
||||||
|
* **Deep-Link / Section-basierter Link:** Ein Link wie `[[Note#Section]]`, der auf einen spezifischen Abschnitt innerhalb einer Note verweist. Seit v2.9.1 wird dieser in `target_id="Note"` und `target_section="Section"` aufgeteilt, um "Phantom-Knoten" zu vermeiden und Multigraph-Support zu ermöglichen.
|
||||||
|
* **Atomic Section Logic (v3.9.9):** Chunking-Verfahren, das Sektions-Überschriften und deren Inhalte atomar in Chunks hält (Pack-and-Carry-Over). Verhindert, dass Überschriften über Chunk-Grenzen hinweg getrennt werden.
|
||||||
|
* **Registry-First Profiling (v2.13.12):** Hierarchische Auflösung des Chunking-Profils: Frontmatter > types.yaml Typ-Config > Global Defaults. Stellt sicher, dass Note-Typen automatisch das korrekte Profil erhalten.
|
||||||
@@ -0,0 +1,166 @@
|
|||||||
|
---
|
||||||
|
doc_type: quality_assurance
|
||||||
|
audience: all
|
||||||
|
status: active
|
||||||
|
version: 2.9.1
|
||||||
|
context: "Qualitätsprüfung der Dokumentation für alle Rollen: Vollständigkeit, Korrektheit und Anwendbarkeit."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Dokumentations-Qualitätsprüfung
|
||||||
|
|
||||||
|
Diese Checkliste dient zur systematischen Prüfung, ob die Dokumentation alle Fragen jeder Rolle vollständig beantwortet.
|
||||||
|
|
||||||
|
## ✅ Entwickler
|
||||||
|
|
||||||
|
### Setup & Installation
|
||||||
|
- [x] **Lokales Setup:** [Developer Guide](../05_Development/05_developer_guide.md#6-lokales-setup-development)
|
||||||
|
- [x] **Schnellstart:** [Quickstart](00_quickstart.md)
|
||||||
|
- [x] **Hardware-Anforderungen:** [Admin Operations](../04_Operations/04_admin_operations.md#11-voraussetzungen)
|
||||||
|
|
||||||
|
### Architektur & Code
|
||||||
|
- [x] **Modulare Struktur:** [Developer Guide - Architektur](../05_Development/05_developer_guide.md#4-projektstruktur--modul-referenz-deep-dive)
|
||||||
|
- [x] **Design-Patterns:** [Architektur-Patterns](../02_concepts/02_concept_architecture_patterns.md)
|
||||||
|
- [x] **API-Referenz:** [API Reference](../03_Technical_References/03_tech_api_reference.md)
|
||||||
|
- [x] **Datenmodell:** [Data Model](../03_Technical_References/03_tech_data_model.md)
|
||||||
|
|
||||||
|
### Entwicklung & Erweiterung
|
||||||
|
- [x] **Workflow:** [Developer Guide - Workflow](../05_Development/05_developer_guide.md#7-der-entwicklungs-zyklus-workflow)
|
||||||
|
- [x] **Erweiterungs-Guide:** [Teach-the-AI](../05_Development/05_developer_guide.md#8-erweiterungs-guide-teach-the-ai)
|
||||||
|
- [x] **GenAI Best Practices:** [GenAI Best Practices](../05_Development/05_genai_best_practices.md)
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
- [x] **Test-Strategien:** [Testing Guide](../05_Development/05_testing_guide.md)
|
||||||
|
- [x] **Test-Frameworks:** [Testing Guide - Frameworks](../05_Development/05_testing_guide.md#3-test-frameworks--tools)
|
||||||
|
- [x] **Test-Daten:** [Testing Guide - Test-Daten](../05_Development/05_testing_guide.md#2-test-daten--vaults)
|
||||||
|
|
||||||
|
### Debugging & Troubleshooting
|
||||||
|
- [x] **Troubleshooting:** [Developer Guide - Troubleshooting](../05_Development/05_developer_guide.md#10-troubleshooting--one-liners)
|
||||||
|
- [x] **Debug-Tools:** [Testing Guide - Debugging](../05_Development/05_testing_guide.md#7-debugging--diagnose)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Administratoren
|
||||||
|
|
||||||
|
### Installation & Setup
|
||||||
|
- [x] **Installation:** [Admin Operations](../04_Operations/04_admin_operations.md#1-installation--setup)
|
||||||
|
- [x] **Docker Setup:** [Admin Operations - Qdrant](../04_Operations/04_admin_operations.md#12-qdrant-docker)
|
||||||
|
- [x] **Systemd Services:** [Admin Operations - Deployment](../04_Operations/04_admin_operations.md#2-deployment-systemd-services)
|
||||||
|
|
||||||
|
### Betrieb & Wartung
|
||||||
|
- [x] **Monitoring:** [Admin Operations - Wartung](../04_Operations/04_admin_operations.md#3-wartung--monitoring)
|
||||||
|
- [x] **Backup & Restore:** [Admin Operations - Backup](../04_Operations/04_admin_operations.md#4-backup--restore)
|
||||||
|
- [x] **Troubleshooting:** [Admin Operations - Troubleshooting](../04_Operations/04_admin_operations.md#33-troubleshooting-guide)
|
||||||
|
|
||||||
|
### Server-Betrieb
|
||||||
|
- [x] **Server-Konfiguration:** [Server Operations Manual](../04_Operations/04_server_operation_manual.md)
|
||||||
|
- [x] **Disaster Recovery:** [Server Operations - DR](../04_Operations/04_server_operation_manual.md#5-disaster-recovery-wiederherstellung-two-stage-dr)
|
||||||
|
- [x] **Backup-Strategie:** [Server Operations - Backup](../04_Operations/04_server_operation_manual.md#4-backup-strategie-borgmatic)
|
||||||
|
|
||||||
|
### Konfiguration
|
||||||
|
- [x] **ENV-Variablen:** [Configuration Reference](../03_Technical_References/03_tech_configuration.md#1-environment-variablen-env)
|
||||||
|
- [x] **YAML-Configs:** [Configuration Reference - YAML](../03_Technical_References/03_tech_configuration.md#2-typ-registry-typesyaml)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Anwender
|
||||||
|
|
||||||
|
### Erste Schritte
|
||||||
|
- [x] **Schnellstart:** [Quickstart](00_quickstart.md)
|
||||||
|
- [x] **Was ist Mindnet:** [Vision & Strategie](00_vision_and_strategy.md)
|
||||||
|
- [x] **Grundlagen:** [Glossar](00_glossary.md)
|
||||||
|
|
||||||
|
### Nutzung
|
||||||
|
- [x] **Chat-Bedienung:** [Chat Usage Guide](../01_User_Manual/01_chat_usage_guide.md)
|
||||||
|
- [x] **Graph Explorer:** [Chat Usage Guide - Graph](../01_User_Manual/01_chat_usage_guide.md#22-modus--graph-explorer-cytoscape)
|
||||||
|
- [x] **Editor:** [Chat Usage Guide - Editor](../01_User_Manual/01_chat_usage_guide.md#23-modus--manueller-editor)
|
||||||
|
|
||||||
|
### Content-Erstellung
|
||||||
|
- [x] **Knowledge Design:** [Knowledge Design Manual](../01_User_Manual/01_knowledge_design.md)
|
||||||
|
- [x] **Authoring Guidelines:** [Authoring Guidelines](../01_User_Manual/01_authoring_guidelines.md)
|
||||||
|
- [x] **Obsidian-Integration:** [Obsidian Integration](../01_User_Manual/01_obsidian_integration_guide.md)
|
||||||
|
|
||||||
|
### Häufige Fragen
|
||||||
|
- [x] **Wie strukturiere ich Notizen?** → [Knowledge Design](../01_User_Manual/01_knowledge_design.md)
|
||||||
|
- [x] **Welche Note-Typen gibt es?** → [Knowledge Design - Typ-Referenz](../01_User_Manual/01_knowledge_design.md#31-typ-referenz--stream-logik)
|
||||||
|
- [x] **Wie verknüpfe ich Notizen?** → [Knowledge Design - Edges](../01_User_Manual/01_knowledge_design.md#4-edges--verlinkung)
|
||||||
|
- [x] **Wie nutze ich den Chat?** → [Chat Usage Guide](../01_User_Manual/01_chat_usage_guide.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Tester
|
||||||
|
|
||||||
|
### Test-Strategien
|
||||||
|
- [x] **Test-Pyramide:** [Testing Guide - Strategien](../05_Development/05_testing_guide.md#1-test-strategie--ebenen)
|
||||||
|
- [x] **Unit Tests:** [Testing Guide - Unit Tests](../05_Development/05_testing_guide.md#11-unit-tests-pytest)
|
||||||
|
- [x] **Integration Tests:** [Testing Guide - Integration](../05_Development/05_testing_guide.md#12-integration-tests)
|
||||||
|
- [x] **E2E Tests:** [Testing Guide - E2E](../05_Development/05_testing_guide.md#13-e2e--smoke-tests)
|
||||||
|
|
||||||
|
### Test-Frameworks
|
||||||
|
- [x] **Pytest:** [Testing Guide - Frameworks](../05_Development/05_testing_guide.md#31-pytest-unit-tests)
|
||||||
|
- [x] **Unittest:** [Testing Guide - Unittest](../05_Development/05_testing_guide.md#32-unittest-e2e-tests)
|
||||||
|
- [x] **Shell-Skripte:** [Testing Guide - Shell](../05_Development/05_testing_guide.md#33-shell-skripte-e2e-roundtrip)
|
||||||
|
|
||||||
|
### Test-Daten & Tools
|
||||||
|
- [x] **Test-Vault erstellen:** [Testing Guide - Test-Daten](../05_Development/05_testing_guide.md#21-test-vault-erstellen)
|
||||||
|
- [x] **Test-Skripte:** [Developer Guide - Scripts](../05_Development/05_developer_guide.md#44-scripts--tooling-die-admin-toolbox)
|
||||||
|
- [x] **Test-Checkliste:** [Testing Guide - Checkliste](../05_Development/05_testing_guide.md#8-test-checkliste-für-pull-requests)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Deployment
|
||||||
|
|
||||||
|
### Deployment-Prozesse
|
||||||
|
- [x] **Deployment-Guide:** [Deployment Guide](../04_Operations/04_deployment_guide.md)
|
||||||
|
- [x] **CI/CD Pipeline:** [Deployment Guide - CI/CD](../04_Operations/04_deployment_guide.md#9-cicd-pipeline-details)
|
||||||
|
- [x] **Rollout-Strategien:** [Deployment Guide - Rollout](../04_Operations/04_deployment_guide.md#4-rollout-strategien)
|
||||||
|
|
||||||
|
### Versionierung & Releases
|
||||||
|
- [x] **Version-Schema:** [Deployment Guide - Versionierung](../04_Operations/04_deployment_guide.md#51-version-schema)
|
||||||
|
- [x] **Release-Prozess:** [Deployment Guide - Release](../04_Operations/04_deployment_guide.md#52-release-prozess)
|
||||||
|
|
||||||
|
### Rollback & Recovery
|
||||||
|
- [x] **Rollback-Strategien:** [Deployment Guide - Rollback](../04_Operations/04_deployment_guide.md#6-rollback-strategien)
|
||||||
|
- [x] **Disaster Recovery:** [Server Operations - DR](../04_Operations/04_server_operation_manual.md#5-disaster-recovery-wiederherstellung-two-stage-dr)
|
||||||
|
|
||||||
|
### Pre/Post-Deployment
|
||||||
|
- [x] **Pre-Deployment Checkliste:** [Deployment Guide - Checkliste](../04_Operations/04_deployment_guide.md#7-pre-deployment-checkliste)
|
||||||
|
- [x] **Post-Deployment Validierung:** [Deployment Guide - Validierung](../04_Operations/04_deployment_guide.md#8-post-deployment-validierung)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Zusammenfassung
|
||||||
|
|
||||||
|
### Vollständigkeit nach Rolle
|
||||||
|
|
||||||
|
| Rolle | Abgedeckte Themen | Status |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **Entwickler** | Setup, Architektur, Code, Testing, Debugging | ✅ Vollständig |
|
||||||
|
| **Administratoren** | Installation, Betrieb, Wartung, Backup, DR | ✅ Vollständig |
|
||||||
|
| **Anwender** | Nutzung, Content-Erstellung, Workflows | ✅ Vollständig |
|
||||||
|
| **Tester** | Test-Strategien, Frameworks, Tools | ✅ Vollständig |
|
||||||
|
| **Deployment** | CI/CD, Rollout, Versionierung, Rollback | ✅ Vollständig |
|
||||||
|
|
||||||
|
### Neue Dokumente
|
||||||
|
|
||||||
|
1. ✅ `05_testing_guide.md` - Umfassender Test-Guide
|
||||||
|
2. ✅ `04_deployment_guide.md` - Vollständiger Deployment-Guide
|
||||||
|
3. ✅ `02_concept_architecture_patterns.md` - Architektur-Patterns
|
||||||
|
4. ✅ `03_tech_api_reference.md` - API-Referenz
|
||||||
|
5. ✅ `00_quickstart.md` - Schnellstart-Anleitung
|
||||||
|
6. ✅ `README.md` - Dokumentations-Einstiegspunkt
|
||||||
|
|
||||||
|
### Aktualisierte Dokumente
|
||||||
|
|
||||||
|
1. ✅ `00_documentation_map.md` - Alle neuen Dokumente aufgenommen
|
||||||
|
2. ✅ `04_admin_operations.md` - Troubleshooting erweitert
|
||||||
|
3. ✅ `05_developer_guide.md` - Modulare Struktur ergänzt
|
||||||
|
4. ✅ `03_tech_ingestion_pipeline.md` - Background Tasks dokumentiert
|
||||||
|
5. ✅ `03_tech_configuration.md` - Fehlende ENV-Variablen ergänzt
|
||||||
|
6. ✅ `00_vision_and_strategy.md` - Design-Entscheidungen ergänzt
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Status:** ✅ Alle Rollen vollständig abgedeckt
|
||||||
|
**Letzte Prüfung:** 2025-01-XX
|
||||||
|
**Version:** 2.9.1
|
||||||
|
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
---
|
||||||
|
doc_type: quickstart_guide
|
||||||
|
audience: user, developer, admin
|
||||||
|
status: active
|
||||||
|
version: 2.9.1
|
||||||
|
context: "Schnellstart-Anleitung für neue Benutzer von Mindnet"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mindnet Schnellstart
|
||||||
|
|
||||||
|
Diese Anleitung hilft dir, in 15 Minuten mit Mindnet loszulegen.
|
||||||
|
|
||||||
|
## 🎯 Was ist Mindnet?
|
||||||
|
|
||||||
|
Mindnet ist ein **persönliches KI-Gedächtnis**, das:
|
||||||
|
- Dein Wissen in Markdown-Notizen speichert
|
||||||
|
- Semantisch verknüpft (Wissensgraph)
|
||||||
|
- Als intelligenter Dialogpartner agiert (RAG-Chat)
|
||||||
|
- **Lokal und privat** läuft (Privacy First)
|
||||||
|
|
||||||
|
## 📋 Voraussetzungen
|
||||||
|
|
||||||
|
- **Python 3.10+** installiert
|
||||||
|
- **Docker** installiert (für Qdrant)
|
||||||
|
- **Ollama** installiert (für lokale LLMs)
|
||||||
|
- Optional: **Obsidian** (für komfortables Schreiben)
|
||||||
|
|
||||||
|
## ⚡ Installation (5 Minuten)
|
||||||
|
|
||||||
|
### Schritt 1: Repository klonen
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <repository-url> mindnet
|
||||||
|
cd mindnet
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 2: Virtuelle Umgebung erstellen
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv
|
||||||
|
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 3: Abhängigkeiten installieren
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 4: Qdrant starten (Docker)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d qdrant
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 5: Ollama-Modelle laden
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ollama pull phi3:mini
|
||||||
|
ollama pull nomic-embed-text
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 6: Konfiguration anpassen
|
||||||
|
|
||||||
|
Erstelle eine `.env`-Datei im Projektroot:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
QDRANT_URL=http://localhost:6333
|
||||||
|
MINDNET_OLLAMA_URL=http://localhost:11434
|
||||||
|
MINDNET_LLM_MODEL=phi3:mini
|
||||||
|
MINDNET_EMBEDDING_MODEL=nomic-embed-text
|
||||||
|
COLLECTION_PREFIX=mindnet
|
||||||
|
MINDNET_VAULT_ROOT=./vault
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🚀 Erste Schritte (5 Minuten)
|
||||||
|
|
||||||
|
### Schritt 1: Backend starten
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uvicorn app.main:app --reload --port 8001
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 2: Frontend starten (neues Terminal)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
streamlit run app/frontend/ui.py --server.port 8501
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 3: Browser öffnen
|
||||||
|
|
||||||
|
Öffne `http://localhost:8501` im Browser.
|
||||||
|
|
||||||
|
### Schritt 4: Erste Notiz importieren
|
||||||
|
|
||||||
|
Erstelle eine Test-Notiz im `vault/` Ordner:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
---
|
||||||
|
id: 20250101-test
|
||||||
|
title: Meine erste Notiz
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
---
|
||||||
|
|
||||||
|
# Meine erste Notiz
|
||||||
|
|
||||||
|
Dies ist eine Test-Notiz für Mindnet.
|
||||||
|
|
||||||
|
[[rel:related_to Mindnet]]
|
||||||
|
```
|
||||||
|
|
||||||
|
Dann importiere sie:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --prefix mindnet --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schritt 5: Erste Chat-Anfrage
|
||||||
|
|
||||||
|
Im Browser-Chat eingeben:
|
||||||
|
```
|
||||||
|
Was ist Mindnet?
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📚 Nächste Schritte
|
||||||
|
|
||||||
|
Nach dem Schnellstart empfehle ich:
|
||||||
|
|
||||||
|
1. **[Chat Usage Guide](../01_User_Manual/01_chat_usage_guide.md)** - Lerne die Chat-Funktionen kennen
|
||||||
|
2. **[Knowledge Design](../01_User_Manual/01_knowledge_design.md)** - Verstehe, wie du Notizen strukturierst
|
||||||
|
3. **[Authoring Guidelines](../01_User_Manual/01_authoring_guidelines.md)** - Lerne Best Practices für das Schreiben
|
||||||
|
|
||||||
|
## 🆘 Hilfe & Troubleshooting
|
||||||
|
|
||||||
|
**Problem:** Qdrant startet nicht
|
||||||
|
- **Lösung:** Prüfe, ob Docker läuft: `docker ps`
|
||||||
|
|
||||||
|
**Problem:** Ollama-Modell nicht gefunden
|
||||||
|
- **Lösung:** Prüfe mit `ollama list`, ob die Modelle geladen sind
|
||||||
|
|
||||||
|
**Problem:** Import schlägt fehl
|
||||||
|
- **Lösung:** Prüfe die Logs und stelle sicher, dass Qdrant läuft
|
||||||
|
|
||||||
|
Für detaillierte Troubleshooting-Informationen siehe [Admin Operations](../04_Operations/04_admin_operations.md#33-troubleshooting-guide).
|
||||||
|
|
||||||
|
## 🔗 Weitere Ressourcen
|
||||||
|
|
||||||
|
- **[Dokumentationskarte](00_documentation_map.md)** - Übersicht aller Dokumente
|
||||||
|
- **[Glossar](00_glossary.md)** - Wichtige Begriffe erklärt
|
||||||
|
- **[Vision & Strategie](00_vision_and_strategy.md)** - Die Philosophie hinter Mindnet
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Viel Erfolg mit Mindnet!** 🚀
|
||||||
|
|
||||||
@@ -64,4 +64,42 @@ Die folgenden Prinzipien steuern jede technische Entscheidung:
|
|||||||
Jeder Importlauf und jede Retriever-Anfrage ist logbar.
|
Jeder Importlauf und jede Retriever-Anfrage ist logbar.
|
||||||
|
|
||||||
10. **Local First & Privacy:**
|
10. **Local First & Privacy:**
|
||||||
Nutzung lokaler LLMs (Ollama) für Inference. Keine Daten verlassen den Server.
|
Nutzung lokaler LLMs (Ollama) für Inference. Keine Daten verlassen den Server.
|
||||||
|
|
||||||
|
11. **Modulare Architektur (WP-14):**
|
||||||
|
Core-Logik ist in spezialisierte Pakete unterteilt (`chunking/`, `database/`, `graph/`, `ingestion/`, `parser/`, `retrieval/`). Dies ermöglicht unabhängige Entwicklung, Testbarkeit und Wartbarkeit.
|
||||||
|
|
||||||
|
12. **Resilienz durch Kaskaden:**
|
||||||
|
System nutzt Provider-Kaskaden (Cloud → Rate-Limit-Handling → Lokaler Fallback) für hohe Verfügbarkeit. Deep Fallback erkennt auch kognitive Blockaden (Silent Refusals).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Design-Entscheidungen & Trade-offs
|
||||||
|
|
||||||
|
### 4.1 Qdrant als Vektor-DB
|
||||||
|
**Entscheidung:** Self-hosted Qdrant statt Managed Service (Pinecone/Weaviate)
|
||||||
|
**Gründe:** Privacy First, Open Source, lokale Kontrolle
|
||||||
|
**Trade-off:** Mehr Wartungsaufwand, aber vollständige Datenhoheit
|
||||||
|
|
||||||
|
### 4.2 Hybrid Retrieval
|
||||||
|
**Entscheidung:** Kombination von Vektor-Suche (Semantik) und Graph-Expansion
|
||||||
|
**Gründe:** Bessere Relevanz durch strukturelle Verbindungen
|
||||||
|
**Trade-off:** Höhere Komplexität, aber deutlich bessere Ergebnisse
|
||||||
|
|
||||||
|
### 4.3 Background Tasks statt Queue-System
|
||||||
|
**Entscheidung:** FastAPI Background Tasks statt Redis/RabbitMQ
|
||||||
|
**Gründe:** Einfacher Setup, ausreichend für Single-User
|
||||||
|
**Trade-off:** Keine Persistenz bei Server-Neustart, aber weniger Infrastruktur
|
||||||
|
|
||||||
|
### 4.4 Markdown als Source of Truth
|
||||||
|
**Entscheidung:** Markdown-Dateien sind primär, Datenbank ist Cache
|
||||||
|
**Gründe:** Versionierbarkeit, Editierbarkeit, Unabhängigkeit
|
||||||
|
**Trade-off:** Zwei Datenquellen, aber maximale Flexibilität
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Weitere Informationen
|
||||||
|
|
||||||
|
- **Architektur-Patterns:** Siehe [Architektur-Patterns](../02_concepts/02_concept_architecture_patterns.md)
|
||||||
|
- **Graph-Logik:** Siehe [Graph-Logik](../02_concepts/02_concept_graph_logic.md)
|
||||||
|
- **KI-Persönlichkeit:** Siehe [KI-Persönlichkeit](../02_concepts/02_concept_ai_personality.md)
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
---
|
||||||
|
id: 01-authoring-guidelines
|
||||||
|
title: Authoring Guidelines – Handbuch für den Digitalen Zwilling
|
||||||
|
type: principle
|
||||||
|
status: stable
|
||||||
|
version: 1.1.0
|
||||||
|
area: system_documentation
|
||||||
|
tags: [handbuch, authoring, methodik, obsidian, mindnet, best-practice]
|
||||||
|
retriever_weight: 2.0
|
||||||
|
---
|
||||||
|
|
||||||
|
# Authoring Guidelines: Dein Werkzeug für den Digitalen Zwilling
|
||||||
|
|
||||||
|
|
||||||
|
Dieses Handbuch ist dein primäres Werkzeug, um Wissen so zu strukturieren, dass Mindnet deine Persönlichkeit spiegelt, empathisch reagiert und dich strategisch berät.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚡ Die 6 Goldenen Regeln des Knowledge Designs
|
||||||
|
|
||||||
|
1. **Atomare Gedanken:** Eine Notiz = Ein Thema. Trenne z. B. „Meditation“ von „Mobility“, auch wenn beides Ich-Pflege ist.
|
||||||
|
2. **Explizite Typen:** Nutze den `type`, um der KI zu sagen, wie sie den Text verarbeiten soll (z. B. `insight` für Beobachtungen, `experience` für Erlebnisse).
|
||||||
|
3. **Semantische Links:** Verknüpfe aktiv mit `[[rel:depends_on ...]]` oder `[[rel:based_on ...]]`. Sag dem System, *warum* Dinge zusammenhängen.
|
||||||
|
4. **Werte & Ziele definieren:** Erstelle für jeden Nordstern und jeden Kernwert eine eigene Notiz. Ohne Maßstäbe kann der „Berater“ nicht entscheiden.
|
||||||
|
5. **Emotionales Bridging:** Nutze Begriffe wie „Druck“, „Stolz“, „Euphorie“ oder „Hilflosigkeit“, um die Empathie-Ebene der KI zu aktivieren.
|
||||||
|
6. **Narrative Tiefe (Fleisch am Knochen):** Dokumentiere das „Warum“ hinter einer Entscheidung. Fakten informieren, aber Erzählungen prägen den Charakter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Strategische Steuerung (Status & Gewicht)
|
||||||
|
|
||||||
|
Du entscheidest über das Frontmatter, wie präsent eine Information im „Gedächtnis“ ist.
|
||||||
|
|
||||||
|
### 1.1 Status-Logik
|
||||||
|
* **`stable`**: Gold-Standard. Für finale Leitbild-Texte und Nordsterne. Erhält +20% Relevanz-Bonus.
|
||||||
|
* **`active`**: Standard für laufende Projekte und aktuelle Beobachtungen.
|
||||||
|
* **`draft`**: Brainstorming oder rohe Tages-Logs. Die KI nutzt diese nur nachrangig (50% Malus), um Rauschen zu vermeiden.
|
||||||
|
* **`system`**: Rein technische Dateien (Templates, Guides). Werden im Chat ignoriert.
|
||||||
|
|
||||||
|
### 1.2 Manuelle Gewichtung (`retriever_weight`)
|
||||||
|
* **Boost (1.20):** Für hochrelevante Erkenntnisse. **Beispiel:** Beobachtungen zum Verhalten von Rohan werden als `insight` mit `1.20` markiert, damit sie bei Erziehungsfragen sofort präsent sind.
|
||||||
|
* **Deboost (0.50 - 0.80):** Für tägliche Routine-Einträge („Heute 10 Min. meditiert“). Sie dienen der Chronik, sollen aber keine tiefen Analysen verfälschen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Das Kochbuch: Praktische Use-Cases
|
||||||
|
|
||||||
|
### 2.1 Ein Erlebnis aufschreiben (`type: experience`)
|
||||||
|
**Ziel:** Den „Spiegel“ (Empathy) mit deiner Biografie kalibrieren.
|
||||||
|
|
||||||
|
**Struktur & Leitfragen:**
|
||||||
|
- **Kontext:** Was ist passiert? (Sachlich kurz).
|
||||||
|
- **Emotions-Check:** Wie habe ich mich in der Situation gefühlt? (Wichtig für Regel 5).
|
||||||
|
- **Die Lektion:** Was habe ich über mich gelernt? (z. B. „Ich reagiere allergisch auf Ungerechtigkeit“).
|
||||||
|
- **Deep-Edge:** Mit welcher Rolle ist das verknüpft? `[[rel:supports Meine Rollenlandkarte 2025#Vater]]`.
|
||||||
|
|
||||||
|
### 2.2 Eine Beobachtung festhalten (`type: insight`)
|
||||||
|
**Ziel:** Den „Berater“ (Decision) mit Mustern versorgen.
|
||||||
|
|
||||||
|
**Beispiel Rohan-Beobachtung:**
|
||||||
|
- **Beobachtung:** Rohan reagiert positiv auf „leise“ Impulse und 90/10 Coaching.
|
||||||
|
- **Interpretation:** Direkte Konfrontation erzeugt Gegendruck; Fragen erzeugen Ownership.
|
||||||
|
- **Konsequenz:** Prinzip P3a (Familienregeln) muss immer einen Bedürfnischeck vorausschicken.
|
||||||
|
|
||||||
|
### 2.3 Ein Review durchführen (`type: journal` / `insight`)
|
||||||
|
**Ziel:** Den Fortschritt steuern.
|
||||||
|
- **Daily:** Rohes Log (Status `draft`).
|
||||||
|
- **Weekly/Monthly:** Verdichtung der Erkenntnisse in eine `stable` Notiz. Frage: „Was war der größte Hebel diese Woche?“.
|
||||||
|
|
||||||
|
### 2.4 Das Handlungsprinzip (`type: principle`)
|
||||||
|
**Ziel:** Testbare Regeln für schwierige Entscheidungssituationen.
|
||||||
|
|
||||||
|
**Struktur-Vorgabe:**---
|
||||||
|
id: 01-authoring-guidelines
|
||||||
|
title: Authoring Guidelines – Das Handbuch für den Digitalen Zwilling
|
||||||
|
type: principle
|
||||||
|
status: stable
|
||||||
|
version: 1.2.0
|
||||||
|
area: system_documentation
|
||||||
|
tags: [handbuch, authoring, methodik, obsidian, mindnet, best-practice]
|
||||||
|
retriever_weight: 2.0
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🕸️ Teil 3: Netzdesign (Hubs, Edges & Lücken)
|
||||||
|
|
||||||
|
Ein intelligentes Netz wächst durch strategische Verknüpfungen, nicht durch Textmenge.
|
||||||
|
|
||||||
|
### 3.1 Wissens-Hubs (Zentralnotizen)
|
||||||
|
Hubs fungieren als Verteilerzentren im Graphen.
|
||||||
|
* **Struktur:** Nutze Überschriften (H2, H3) und verlinke von dort auf Detailnotizen.
|
||||||
|
* **Deep-Edges:** Verlinke präzise Abschnitte: `[[Rollenlandkarte#Vater]]`. Dies ermöglicht der KI, nur den relevanten Kontext zu laden.
|
||||||
|
|
||||||
|
### 3.2 Forward-Mapping (Strategische Lücken)
|
||||||
|
Setze bewusst Links auf Dateien, die noch nicht existieren (z. B. `[[Die beste Version meiner selbst]]`).
|
||||||
|
* **Zweck:** Du signalisierst dem System: „Hier entsteht ein wichtiges Konzept“.
|
||||||
|
* **Effekt:** Die KI erkennt diese Lücken und kann im Chat proaktiv Fragen stellen, um diese Felder mit dir gemeinsam zu füllen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Obsidian Usability & Automatisierung
|
||||||
|
|
||||||
|
### 4.1 Templater-Integration
|
||||||
|
Nutze Vorlagen, die bereits die **Leitfragen** und das richtige **Retriever-Weight** enthalten. Ein Klick auf „Neues Erlebnis“ sollte automatisch das Frontmatter mit `type: experience` und `weight: 1.1` füllen.
|
||||||
|
|
||||||
|
### 4.2 Meta Bind Dashboards
|
||||||
|
Nutze **Meta Bind**, um Felder wie `status` oder `retriever_weight` über Schieberegler direkt in der Notiz zu steuern. Das macht die Gewichtung deines Wissens intuitiv und spielerisch.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Vernetzung für die Mindnet-Personas
|
||||||
|
|
||||||
|
| Persona | Notiz-Fokus | Effekt im Dialog |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **🪞 Spiegel (Empathy)** | Gefühle & Biografisches | „Ich verstehe deinen Frust, das war bei Projekt X ähnlich...“. |
|
||||||
|
| **⚖️ Berater (Decision)** | Werte & Wächterfragen | „Option A ist nicht ratsam, da sie gegen dein Prinzip der Integrität verstößt“. |
|
||||||
|
| **📚 Bibliothekar (Facts)** | Struktur & Definitionen | „Deine Mission als Vater ist es, verlässliche Präsenz zu zeigen...“. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
> [!abstract] Fazit für den Autor
|
||||||
|
> Mindnet ist ein **Persönlichkeitsspiegel**. Dokumentiere weniger „Technik“ und mehr „Mensch“. Jede Notiz sollte die Frage beantworten: „Was sagt das über mich und meine Werte aus?“.
|
||||||
|
- **3 Signale:** Woran merke ich im Alltag, dass ich das Prinzip lebe?.
|
||||||
|
- **Wächterfrage:** Welche Frage stelle ich mir im Moment der Entscheidung?.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Obsidian Workflow-Hacks
|
||||||
|
|
||||||
|
### 6.1 Templater & Meta Bind
|
||||||
|
- **Automatisierung:** Nutze das **Templater-Plugin**, um beim Erstellen einer Notiz sofort die passenden Leitfragen und das richtige `retriever_weight` einzufügen.
|
||||||
|
- **Interaktive Steuerung:** Nutze **Meta Bind**, um Felder wie `status` oder `retriever_weight` über Buttons oder Slider direkt im Lesemodus zu ändern, ohne im YAML-Code zu tippen.
|
||||||
|
|
||||||
|
### 6.2 Deep-Edges & Verknüpfung
|
||||||
|
- Verknüpfe Erlebnisse immer mit der entsprechenden Rolle: `[[rel:supports Meine Rollenlandkarte 2025#Vater]]`.
|
||||||
|
- Nutze `derived_from`, um Prinzipien an ihre Ursprungs-Session zu binden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Vernetzung für Mindnet Personas
|
||||||
|
|
||||||
|
| Persona | Notiz-Fokus | Beispiel |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **🪞 Spiegel (Empathy)** | Emotionen & Prägungen | „Ich fühlte mich hilflos, als...“. |
|
||||||
|
| **⚖️ Berater (Decision)** | Kriterien & Abwägungen | „Option A verletzt Wert X, weil...“. |
|
||||||
|
| **📚 Bibliothekar (Facts)** | Struktur & Definition | „Rollenmission Vater bedeutet...“. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
> [!tip] Best Practice
|
||||||
|
> Wenn du merkst, dass eine Notiz zu technisch wird: Halte inne und frage dich: „Was macht das mit mir als Mensch?“ Schreibe *das* auf. Das ist die Essenz für deinen KI-Zwilling.
|
||||||
|
|
||||||
|
## 8. Verbindungen von Notiztypen
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
%% Identity Stream
|
||||||
|
V[Value] -- definieren --> N[Need]
|
||||||
|
N -- erzeugen --> M[Motivation]
|
||||||
|
B[Belief] -- prägen --> M
|
||||||
|
T[Trait] -- unterstützen --> M
|
||||||
|
|
||||||
|
%% Action Stream
|
||||||
|
M -- treiben --> G[Goal]
|
||||||
|
G -- konkretisiert in --> P[Project]
|
||||||
|
P -- unterteilt in --> TA[Task]
|
||||||
|
D[Decision] -- steuert --> P
|
||||||
|
PR[Principle] -- leitet --> D
|
||||||
|
|
||||||
|
%% History Stream
|
||||||
|
TA -- führt zu --> E[Event]
|
||||||
|
E -- generiert --> EX[Experience]
|
||||||
|
PE[Person] -- beteiligt an --> E
|
||||||
|
|
||||||
|
%% Feedback Loop
|
||||||
|
EX -- liefert --> I[Insight]
|
||||||
|
I -- kalibrieren --> V
|
||||||
|
I -- verfeinern --> PR
|
||||||
|
I -- validieren --> B
|
||||||
|
|
||||||
|
%% Constraints
|
||||||
|
RI[Risk] -- gefährdet --> G
|
||||||
|
O[Obstacle] -- blockiert --> TA
|
||||||
|
BI[Bias] -- verzerrt --> D
|
||||||
|
S[State] -- beeinflusst --> EX
|
||||||
|
|
||||||
|
style V fill:#f9f,stroke:#333,stroke-width:2px
|
||||||
|
style EX fill:#bbf,stroke:#333,stroke-width:2px
|
||||||
|
style P fill:#bfb,stroke:#333,stroke-width:2px
|
||||||
|
|
||||||
@@ -11,6 +11,8 @@ context: "Anleitung zur Nutzung der Web-Oberfläche, der Chat-Personas und des G
|
|||||||
|
|
||||||
**Quellen:** `user_guide.md`
|
**Quellen:** `user_guide.md`
|
||||||
|
|
||||||
|
> 💡 **Tipp für Einsteiger:** Falls du noch nie mit Mindnet gearbeitet hast, starte mit dem [Schnellstart](../00_General/00_quickstart.md) und dem [Knowledge Design Manual](01_knowledge_design.md), um die Grundlagen zu verstehen.
|
||||||
|
|
||||||
## 1. Was Mindnet für dich tut
|
## 1. Was Mindnet für dich tut
|
||||||
|
|
||||||
Mindnet ist ein **assoziatives Gedächtnis** mit Persönlichkeit. Es unterscheidet sich von einer reinen Suche dadurch, dass es **kontextsensitiv** agiert.
|
Mindnet ist ein **assoziatives Gedächtnis** mit Persönlichkeit. Es unterscheidet sich von einer reinen Suche dadurch, dass es **kontextsensitiv** agiert.
|
||||||
@@ -123,4 +125,86 @@ Wenn du im **manuellen Editor** schreibst, unterstützt dich Mindnet aktiv bei d
|
|||||||
* **Exakte Treffer:** Das System erkennt Begriffe wie "KI-Gedächtnis" als Alias für "Mindnet".
|
* **Exakte Treffer:** Das System erkennt Begriffe wie "KI-Gedächtnis" als Alias für "Mindnet".
|
||||||
* **Semantische Treffer:** Das System findet inhaltlich verwandte Notizen.
|
* **Semantische Treffer:** Das System findet inhaltlich verwandte Notizen.
|
||||||
* **Klick auf "➕ Einfügen":** Fügt den Link (z.B. `[[rel:related_to Mindnet]]`) an der Cursor-Position ein.
|
* **Klick auf "➕ Einfügen":** Fügt den Link (z.B. `[[rel:related_to Mindnet]]`) an der Cursor-Position ein.
|
||||||
4. **Speichern:** Der Text wird sofort in den Vault geschrieben und indiziert.
|
4. **Speichern:** Der Text wird sofort in den Vault geschrieben und indiziert.
|
||||||
|
|
||||||
|
### 6.3 Weg C: Direktes Schreiben in Obsidian
|
||||||
|
Wenn du Obsidian nutzt, kannst du Notizen direkt im Vault erstellen. Nach dem Speichern in Obsidian:
|
||||||
|
|
||||||
|
1. **Automatischer Import (Cron):** Der stündliche Cron-Job importiert neue/geänderte Dateien automatisch.
|
||||||
|
2. **Manueller Import:** Oder führe manuell aus:
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --prefix mindnet --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteil:** Du arbeitest in deiner gewohnten Obsidian-Umgebung, Mindnet synchronisiert im Hintergrund.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Häufige Anwendungsfälle (Use Cases)
|
||||||
|
|
||||||
|
### 7.1 Entscheidungshilfe suchen
|
||||||
|
**Szenario:** Du stehst vor einer schwierigen Entscheidung.
|
||||||
|
|
||||||
|
**Vorgehen:**
|
||||||
|
1. Stelle eine Frage im Chat: *"Soll ich Tool X nutzen?"*
|
||||||
|
2. Mindnet erkennt `DECISION` Intent
|
||||||
|
3. System lädt deine Werte, Prinzipien und Ziele
|
||||||
|
4. Antwort berücksichtigt deine persönlichen Kriterien
|
||||||
|
|
||||||
|
**Beispiel:** *"Soll ich Notion nutzen?"* → System prüft gegen dein Prinzip "Privacy First" und rät ab.
|
||||||
|
|
||||||
|
### 7.2 Erfahrungen dokumentieren
|
||||||
|
**Szenario:** Du willst ein prägendes Erlebnis festhalten.
|
||||||
|
|
||||||
|
**Vorgehen:**
|
||||||
|
1. Chat: *"Neue Erfahrung: Streit am Recyclinghof"*
|
||||||
|
2. System erstellt strukturierte Notiz mit Feldern:
|
||||||
|
- Situation
|
||||||
|
- Reaktion
|
||||||
|
- Learning
|
||||||
|
3. Verknüpfe mit relevanten Rollen: `[[rel:supports Meine Rollenlandkarte 2025#Vater]]`
|
||||||
|
|
||||||
|
### 7.3 Wissen im Graph erkunden
|
||||||
|
**Szenario:** Du willst verstehen, wie Konzepte zusammenhängen.
|
||||||
|
|
||||||
|
**Vorgehen:**
|
||||||
|
1. Wechsle zum **Graph Explorer**
|
||||||
|
2. Suche nach einem Knoten (z.B. "Projekt Alpha")
|
||||||
|
3. Klicke auf "🎯 Als Zentrum setzen"
|
||||||
|
4. Erkunde die Nachbarschaft (Abhängigkeiten, Entscheidungen, ähnliche Projekte)
|
||||||
|
|
||||||
|
### 7.4 Bestehende Notizen verbessern
|
||||||
|
**Szenario:** Eine Notiz ist unvollständig oder unvernetzt.
|
||||||
|
|
||||||
|
**Vorgehen:**
|
||||||
|
1. Öffne die Notiz im **Manuellen Editor**
|
||||||
|
2. Nutze **Intelligence-Tab** für Link-Vorschläge
|
||||||
|
3. Füge semantische Links hinzu: `[[rel:based_on Wert X]]`
|
||||||
|
4. Speichern → System indiziert neu
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Tipps & Tricks
|
||||||
|
|
||||||
|
### 8.1 Intent-Steuerung
|
||||||
|
- **Frage mit "?":** Löst RAG-Modus aus (Wissen abrufen)
|
||||||
|
- **Aussage ohne "?":** Löst Interview-Modus aus (Wissen speichern)
|
||||||
|
- **Emotionale Wörter:** Aktivieren Empathie-Persona ("Ich fühle...", "Ich bin frustriert...")
|
||||||
|
|
||||||
|
### 8.2 Graph-Navigation
|
||||||
|
- **URL-Persistenz:** Graph-Einstellungen werden in der URL gespeichert → Lesezeichen für spezifische Ansichten
|
||||||
|
- **Deep-Links:** Verlinke präzise Abschnitte: `[[Note#Abschnitt]]` statt nur `[[Note]]`
|
||||||
|
|
||||||
|
### 8.3 Feedback nutzen
|
||||||
|
- **Regelmäßiges Feedback:** Macht das System schlauer
|
||||||
|
- **Granulares Feedback:** Bewerte einzelne Quellen, nicht nur die Gesamtantwort
|
||||||
|
- **Konsequenz:** System lernt deine Präferenzen
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Weitere Informationen
|
||||||
|
|
||||||
|
- **Knowledge Design:** Siehe [Knowledge Design Manual](01_knowledge_design.md)
|
||||||
|
- **Authoring Guidelines:** Siehe [Authoring Guidelines](01_authoring_guidelines.md)
|
||||||
|
- **Obsidian Integration:** Siehe [Obsidian Integration Guide](01_obsidian_integration_guide.md)
|
||||||
|
- **Troubleshooting:** Siehe [Admin Operations - Troubleshooting](../04_Operations/04_admin_operations.md#33-troubleshooting-guide)
|
||||||
@@ -3,24 +3,22 @@ doc_type: user_manual
|
|||||||
audience: user, author
|
audience: user, author
|
||||||
scope: vault, markdown, schema
|
scope: vault, markdown, schema
|
||||||
status: active
|
status: active
|
||||||
version: 2.6
|
version: 2.9.1
|
||||||
context: "Regelwerk für das Erstellen von Notizen im Vault. Die 'Source of Truth' für Autoren."
|
context: "Regelwerk für das Erstellen von Notizen im Vault. Die 'Source of Truth' für Autoren."
|
||||||
---
|
---
|
||||||
|
|
||||||
# Knowledge Design Manual
|
# Knowledge Design Manual
|
||||||
|
|
||||||
**Quellen:** `knowledge_design.md`, `TYPE_REGISTRY_MANUAL.md`
|
**Quellen:** `knowledge_design.md`, `types.yaml`
|
||||||
|
|
||||||
## ⚡ Die 5 Goldenen Regeln (TL;DR)
|
## ⚡ Die 6 Goldenen Regeln (TL;DR)
|
||||||
|
|
||||||
Damit Mindnet als dein Digitaler Zwilling funktioniert, beachte beim Schreiben diese Grundsätze:
|
|
||||||
|
|
||||||
1. **Atomare Gedanken:** Eine Notiz = Ein Thema. Wenn du über zwei Projekte schreibst, mach zwei Notizen draus.
|
|
||||||
2. **Explizite Typen:** Setze immer den `type` im Frontmatter. Mindnet behandelt eine `decision` ("Wir machen X") völlig anders als ein `concept` ("Was ist X").
|
|
||||||
3. **Semantische Links:** Schreibe nicht nur `[[Link]]`, sondern `[[rel:depends_on Link]]`. Sag dem System *wie* Dinge zusammenhängen.
|
|
||||||
4. **Werte & Ziele definieren:** Damit die **Decision Engine** dich beraten kann, musst du deine Kriterien (`type: value`, `type: goal`) explizit als Notizen anlegen.
|
|
||||||
5. **Emotionales Bridging:** Damit die **Empathie** funktioniert, nutze in Erfahrungsberichten (`type: experience`) emotionale Schlüsselwörter ("Krise", "Freude", "Angst").
|
|
||||||
|
|
||||||
|
1. **Atomare Gedanken:** Eine Notiz = Ein Thema. Vermischung führt zu "Context-Pollution", wodurch die KI unpräzise berät.
|
||||||
|
2. **Explizite Typen:** Der `type` im Frontmatter ist der wichtigste Hebel. Er entscheidet über die mathematische Gewichtung im System.
|
||||||
|
3. **Semantische Links:** Nutze das Vokabular aus der `01_edge_vocabulary.md`. Ersetze einfache Verweise durch funktionale Beziehungen wie `caused_by` oder `based_on`.
|
||||||
|
4. **Werte & Ziele definieren:** Ohne explizite Kriterien (`type: value`) fehlt dem System der Maßstab für eine Bewertung.
|
||||||
|
5. **Emotionales Bridging:** Dokumentiere deine Gefühle. Begriffe wie „Druck“ oder „Euphorie“ sind die Schnittstelle für die Empathie-Logik.
|
||||||
|
6. **Narrative Tiefe (Fleisch am Knochen):** Ein „Was“ ohne „Warum“ ist für die KI nur ein Fakt, für dich als Mensch aber bedeutungslos. Dokumentiere die Intention hinter deinen Taten.
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Zweck & Scope
|
## 1. Zweck & Scope
|
||||||
@@ -43,32 +41,160 @@ type: project # Steuert Chunking & Wichtigkeit
|
|||||||
status: active # active, archived, draft
|
status: active # active, archived, draft
|
||||||
created: 2025-12-12 # ISO 8601
|
created: 2025-12-12 # ISO 8601
|
||||||
tags: [ki, entwicklung] # Taxonomie
|
tags: [ki, entwicklung] # Taxonomie
|
||||||
|
aliases: [KI Projekt, Codename Quickhack]
|
||||||
---
|
---
|
||||||
```
|
```
|
||||||
|
|
||||||
**Optionale Felder:**
|
### **2.1 [NEU] Naming Convention: Intuitiv vs. Technisch**
|
||||||
* `aliases`: [Alpha Projekt] – Wichtig für "Active Intelligence" (Exact Match).
|
Für die Benutzerfreundlichkeit und die intuitive Navigation in Obsidian wird die Nutzung von **menschenlesbaren Titeln** bevorzugt (z. B. `Mein Persönliches Leitbild 2025` statt `leitbild_identity`).
|
||||||
* `visibility`: internal (default) / public.
|
- Die technische Eindeutigkeit wird primär über die `id` im Frontmatter sichergestellt.
|
||||||
|
- Die KI nutzt die Semantik des Dateinamens als zusätzlichen Kontext-Vektor.
|
||||||
|
|
||||||
> **Wichtig:** Felder wie `retriever_weight` oder `chunk_profile` werden zentral über `types.yaml` gesteuert und müssen nicht mehr manuell gesetzt werden (Virtual Schema Layer).
|
### 2.2 Advanced Overrides: Die KI-Steuerung übernehmen
|
||||||
|
|
||||||
|
|
||||||
|
In 95% der Fälle setzt der `type` (z.B. "concept") automatisch die richtigen Einstellungen. In Spezialfällen kannst du diese manuell im Frontmatter überschreiben, um das Verhalten der KI zu erzwingen.
|
||||||
|
|
||||||
|
#### A. `retriever_weight`: Die Sichtbarkeit steuern
|
||||||
|
Dieser Faktor (Default: 1.0) ist ein Multiplikator für das Ranking in der Vektorsuche. Er entscheidet, welche Notiz "gewinnt", wenn zwei Texte inhaltlich ähnlich sind.
|
||||||
|
|
||||||
|
* **Standard (1.0):** Normale Wichtigkeit.
|
||||||
|
* **Boost (1.2 - 2.0):** "Das hier ist die Wahrheit."
|
||||||
|
* *Einsatz:* Finale Entscheidungen, Kernprinzipien, die "Single Source of Truth".
|
||||||
|
* *Effekt:* Verdrängt weniger wichtige Notizen (z.B. Meeting-Protokolle) aus dem Kontext-Fenster der KI.
|
||||||
|
* **Deboost (0.5 - 0.8):** "Nur Kontext, keine Fakten."
|
||||||
|
* *Einsatz:* Glossare, externe Quellen (`source`), reine Datensammlungen.
|
||||||
|
* *Effekt:* Die Notiz wird nur gefunden, wenn man sehr spezifisch danach sucht.
|
||||||
|
|
||||||
|
#### B. `chunking_profile`: Die Zerstückelung steuern
|
||||||
|
Das Profil bestimmt, wie der Text für die Datenbank zerschnitten wird. Falsches Chunking zerreißt den Kontext. Wähle das Profil basierend auf der **Struktur** deines Textes.
|
||||||
|
|
||||||
|
| Profil-Name | Strategie | Einsatzzweck & Wirkung |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **`sliding_standard`** | Sliding Window | **Der Allrounder.** Für Fließtexte (Tagebuch, Artikel). Der Text wird in überlappende Fenster geschnitten. Gut, wenn der Inhalt von oben nach unten fließt. |
|
||||||
|
| **`sliding_short`** | Sliding Window (Klein) | **Für Dichte.** Für Texte mit sehr hoher Informationsdichte (Glossare, Task-Listen), wo jeder Satz wichtig ist. Erzeugt viele kleine Chunks. |
|
||||||
|
| **`sliding_smart_edges`** | Sliding + AI | **Der Intelligente.** Wie Standard, aber das LLM analysiert jeden Chunk zusätzlich auf implizite Querverweise. Standard für `concept` und `project`. |
|
||||||
|
| **`structured_smart_edges`** | Heading Split (Soft) | **Für Strukturierte Texte.** Trennt an Überschriften (H2). *Besonderheit:* Wenn ein Abschnitt sehr kurz ist, wird er mit dem nächsten verschmolzen ("Soft Mode"), um den Kontext zu wahren. |
|
||||||
|
| **`structured_smart_edges_strict`** | Heading Split (Hard) | **Für Listen & Kataloge.** Trennt *zwingend* an jeder H2-Überschrift. Verhindert das Verschmelzen. <br>**Wichtig für:** `decision` (Option A darf nicht mit Option B verschmelzen), `value`, `profile`. |
|
||||||
|
|
||||||
|
**Beispiel für ein Override:**
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
title: Sammlung meiner Passwörter-Regeln
|
||||||
|
type: list
|
||||||
|
# Wir erzwingen eine strikte Trennung, damit Regel 1 nicht mit Regel 2 vermischt wird.
|
||||||
|
chunking_profile: structured_smart_edges_strict
|
||||||
|
# Extrem wichtig, soll immer beachtet werden.
|
||||||
|
retriever_weight: 1.5
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
## 3. Frontmatter Felder und ihre Bedeutung
|
||||||
|
### 3.1 Typ-Referenz & Stream-Logik
|
||||||
|
|
||||||
## 3. Typ-Referenz & Verhalten
|
Das System euriert Informationen in funktionalen Streams. Wähle den Typ danach aus, in welchem "Gedächtnis-Bereich" die Information landen soll.
|
||||||
|
|
||||||
Wähle den Typ, der die **Rolle** der Notiz am besten beschreibt. Der Typ steuert Chunking, Gewichtung und Kanten-Logik.
|
### 3.1 Identity Stream (Der Kern / Das „Warum“)
|
||||||
|
Dieser Stream definiert deine stabilen Merkmale und inneren Kompasse.
|
||||||
|
|
||||||
| Typ | Beschreibung & Einsatzzweck | Rolle im Chat (Intent) | Matrix-Logik |
|
| Typ | Gewicht | Chunking Profil | Zweck & Inhalt |
|
||||||
| :--- | :--- | :--- | :--- |
|
| :--- | :--- | :--- | :--- |
|
||||||
| **`concept`** | Fachbegriffe, Theorien. Zeitloses Wissen. | **FACT** | Ziel für `uses` |
|
| **`value`** | 1.00 | `structured_strict` | Fundamentale Werte und moralische Maßstäbe. |
|
||||||
| **`project`** | Ein Vorhaben mit Ziel, Dauer und Aufgaben. | **FACT / DECISION** | Quelle für `depends_on` |
|
| **`principle`** | 0.95 | `structured_strict_L3` | Handlungsleitlinien mit tiefer Hierarchie (H3-Split). |
|
||||||
| **`experience`** | Persönliche Erfahrung, Lektion oder Erkenntnis. | **EMPATHY** | Quelle für `based_on` |
|
| **`trait`** | 1.10 | `structured_strict` | Charakterliche Stärken und Talente. |
|
||||||
| **`decision`** | Eine bewusst getroffene Entscheidung (ADR). | **DECISION** | Quelle für `caused_by` |
|
| **`belief`** | 0.90 | `sliding_short` | Tiefe Überzeugungen über dich und die Gesellschaft. |
|
||||||
| **`value`** | Ein persönlicher Wert oder ein Prinzip. | **DECISION** | Ziel für `based_on` |
|
| **`profile`** | 0.70 | `structured_strict` | Rollenidentitäten (z. B. Vater, Mentor, Unternehmer). |
|
||||||
| **`principle`** | Handlungsleitlinie. | **DECISION** | Quelle für `derived_from` |
|
| **`need`** | 1.05 | `sliding_smart_edges` | Psychologische Grundbedürfnisse (z. B. Autonomie, Bindung). |
|
||||||
| **`goal`** | Ein strategisches Ziel. | **DECISION** | Ziel für `related_to` |
|
| **`motivation`**| 0.95 | `sliding_smart_edges` | Innere Antreiber und Quellen deiner Energie. |
|
||||||
| **`risk`** | Ein identifiziertes Risiko. | **DECISION** | Quelle für `blocks` |
|
| **`boundary`** | 0.90 | `sliding_smart_edges` | Deine persönlichen Grenzen und Integritäts-Leitplanken. |
|
||||||
| **`journal`** | Zeitbezogener Log-Eintrag. | **FACT** | - |
|
| **`bias`** | 0.80 | `sliding_short` | Bekannte kognitive Verzerrungen und Denkfallen. |
|
||||||
|
|
||||||
|
### 3.1.2 Action Stream (Die Dynamik / Das „Was“)
|
||||||
|
Dieser Stream umfasst alles, was auf Umsetzung, Planung und aktuelle Zustände zielt.
|
||||||
|
|
||||||
|
| Typ | Gewicht | Chunking Profil | Zweck & Inhalt |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| **`project`** | 0.97 | `sliding_smart_edges` | Aktive Vorhaben mit Mission und Zielsetzung. |
|
||||||
|
| **`goal`** | 0.95 | `sliding_smart_edges` | Strategische Nordsterne und Zielzustände. |
|
||||||
|
| **`decision`** | 1.00 | `structured_strict` | Getroffene Entscheidungen (ADR-Logik). |
|
||||||
|
| **`risk`** | 0.85 | `sliding_short` | Potenzielle Gefahren und Bedrohungsszenarien. |
|
||||||
|
| **`obstacle`** | 1.00 | `structured_strict` | Aktuelle Hürden, Ängste und Blockaden. |
|
||||||
|
| **`task`** | 0.80 | `sliding_short` | Operative Aufgaben und Definition of Done. |
|
||||||
|
| **`skill`** | 0.90 | `sliding_smart_edges` | Fertigkeiten, Lernpfade und Meisterschaft. |
|
||||||
|
| **`habit`** | 0.85 | `sliding_short` | Routinen, Automatismen und Verhaltensmuster. |
|
||||||
|
| **`idea`** | 0.70 | `sliding_short` | Flüchtige Einfälle und Rohmaterial für Projekte. |
|
||||||
|
| **`state`** | 0.60 | `sliding_short` | Momentane Verfassung, Stimmung und Energielevel. |
|
||||||
|
|
||||||
|
### 3.1.3 History & Basis Stream (Die Evidenz / Das „Wann“)
|
||||||
|
Dieser Stream speichert deine Erlebnisse, Fakten und externes Wissen als Belege.
|
||||||
|
|
||||||
|
| Typ | Gewicht | Chunking Profil | Zweck & Inhalt |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| **`insight`** | 1.20 | `sliding_smart_edges` | Hochrelevante Erkenntnisse und Verhaltensmuster. |
|
||||||
|
| **`experience`**| 1.10 | `sliding_smart_edges` | Biografische Lektionen und prägende Erlebnisse. |
|
||||||
|
| **`event`** | 0.60 | `sliding_standard` | Chronologische Protokolle und Ereignisse. |
|
||||||
|
| **`journal`** | 0.80 | `sliding_standard` | Tägliche Logs und ungefilterte Gedanken. |
|
||||||
|
| **`person`** | 0.50 | `sliding_standard` | Kontaktprofile und soziale Vernetzung. |
|
||||||
|
| **`source`** | 0.50 | `sliding_standard` | Externe Quellen wie Bücher, Videos oder Artikel. |
|
||||||
|
| **`concept`** | 0.60 | `sliding_smart_edges` | Fachbegriffe, Theorien und zeitloses Wissen. |
|
||||||
|
| **`glossary`** | 0.40 | `sliding_short` | Kurze Begriffsdefinitionen. |
|
||||||
|
| **`default`** | 1.00 | `sliding_standard` | Fallback für alle nicht klassifizierten Notizen. |
|
||||||
|
|
||||||
|
### 3.2 Status und Bedeutung
|
||||||
|
|
||||||
|
| Status | Bedeutung | Auswirkung auf die KI |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **`stable`** | **Geprüftes Wissen.** Die Notiz ist inhaltlich korrekt, finalisiert und verlässlich (Gold-Standard). | **🚀 Bevorzugt (Bonus):** Die KI vertraut diesen Inhalten mehr (+20% Relevanz-Score). Bei widersprüchlichen Infos gewinnt `stable`. |
|
||||||
|
| **`active`** | **Standard.** Aktuelle Arbeitsnotizen, Projekte oder Dokumentationen (Default, wenn leer). | **🔵 Neutral:** Standard-Gewichtung (Faktor 1.0). |
|
||||||
|
| **`draft`** | **Entwurf.** Brainstorming, unstrukturierte Mitschriften oder rohe Ideen. | **🔻 Gedämpft (Malus):** Die KI nutzt diese Infos nur, wenn es keine besseren Treffer gibt (Relevanz auf 50% reduziert). Reduziert "Rauschen" in den Antworten. |
|
||||||
|
| **`system`** | **Intern.** Konfigurationsdateien, technische Logs oder Admin-Skripte. | **❌ Ignoriert (Hard Skip):** Die Datei wird **nicht** in den Suchindex aufgenommen und ist für den Chat unsichtbar. |
|
||||||
|
| **`template`**| **Vorlage.** Leere Gerüste für neue Notizen (z.B. für Obsidian). | **❌ Ignoriert (Hard Skip):** Verhindert, dass leere Vorlagen als Suchergebnisse auftauchen. |
|
||||||
|
|
||||||
|
### Wann nutze ich welchen Status?
|
||||||
|
|
||||||
|
1. **Ideenfindung:** Setzen Sie neue Ideen immer auf **`draft`**. So können Sie frei schreiben, ohne dass die KI sofort "Halbwissen" als Fakten verkauft.
|
||||||
|
2. **Finalisierung:** Sobald eine Entscheidung getroffen oder ein Konzept fertig ist, ändern Sie den Status auf **`stable`**.
|
||||||
|
3. **Konfiguration:** Nutzen Sie **`system`** für Dateien wie die `01_edge_vocabulary.md`, damit die KI nicht die Definition der Kanten zitiert, wenn Sie eigentlich nach Inhalten suchen.
|
||||||
|
|
||||||
|
### 3.3 Aliases (Synonyme & Verlinkung)
|
||||||
|
|
||||||
|
Nutze `aliases: [Synonym]` für:
|
||||||
|
1. **Flüssiges Schreiben:** `[[RAG]]` statt `[[Retrieval Augmented Generation]]`.
|
||||||
|
2. **Semantische Anker:** Der Chat findet die Notiz auch über den Alias-Begriff.
|
||||||
|
|
||||||
|
|
||||||
|
Das optionale Feld `aliases` erlaubt es dir, einer Notiz alternative Titel oder Synonyme zu geben. Dies ist ein mächtiges Werkzeug für die Usability und die Suche.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
title: Leitbild – Handlungsprinzipien
|
||||||
|
aliases: [Prinzipien, Handlungsleitlinien, Entscheidungsprinzipien]
|
||||||
|
---
|
||||||
|
|
||||||
|
**Best Practices für Aliases**
|
||||||
|
|
||||||
|
Damit dein System sauber bleibt, beachte diese Regeln:
|
||||||
|
|
||||||
|
* ✅ **Akronyme & Abkürzungen:**
|
||||||
|
Nutze Aliases für technische Kürzel.
|
||||||
|
* *Note:* `Retrieval Augmented Generation`
|
||||||
|
* *Alias:* `[RAG, RAG-Architektur]`
|
||||||
|
* *Vorteil:* Du tippst im Fließtext nur `[[RAG]]` und bist fertig.
|
||||||
|
|
||||||
|
* ✅ **Projekt-Codenamen:**
|
||||||
|
Verbinde den offiziellen Titel mit dem Flur-Namen.
|
||||||
|
* *Note:* `Projekt Phoenix – Cloud Migration`
|
||||||
|
* *Alias:* `[Phoenix, Cloud-Projekt]`
|
||||||
|
|
||||||
|
* ❌ **Vermeide generische Begriffe (Namespace Pollution):**
|
||||||
|
Gib einer spezifischen Notiz niemals einen Alias, der ein allgemeines Wort ist.
|
||||||
|
* *Schlecht:* Alias `[Meeting]` für `2025-10-JourFixe`.
|
||||||
|
* *Folge:* Jedes Mal, wenn du das Wort "Meeting" tippst, schlägt Obsidian dir diesen einen alten Jour Fixe vor. Das zerstört den Schreibfluss.
|
||||||
|
|
||||||
|
* ❌ **Keine Plural-Wahn:**
|
||||||
|
Mindnet und Obsidian finden meist auch den Plural (Fuzzy Search). Du musst nicht `[Prinzip, Prinzipien, Prinzips]` anlegen. Ein Stammbegriff reicht oft.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -82,6 +208,12 @@ Dies ist die **mächtigste** Methode. Du sagst dem System explizit, **wie** Ding
|
|||||||
> "Daher [[rel:depends_on Qdrant]]."
|
> "Daher [[rel:depends_on Qdrant]]."
|
||||||
> "Dieses Konzept ist [[rel:similar_to Pinecone]]."
|
> "Dieses Konzept ist [[rel:similar_to Pinecone]]."
|
||||||
|
|
||||||
|
**Deep-Links zu Abschnitten (v2.9.1):**
|
||||||
|
Du kannst auch auf spezifische Abschnitte innerhalb einer Note verlinken:
|
||||||
|
> "Siehe [[rel:based_on Mein Leitbild#P3 – Disziplin]]."
|
||||||
|
|
||||||
|
Das System trennt automatisch den Note-Namen (`Mein Leitbild`) vom Abschnitts-Namen (`P3 – Disziplin`), sodass mehrere Links zur gleichen Note möglich sind, wenn sie auf verschiedene Abschnitte zeigen.
|
||||||
|
|
||||||
**Gültige Relationen:**
|
**Gültige Relationen:**
|
||||||
* `depends_on`: Hängt ab von / Benötigt.
|
* `depends_on`: Hängt ab von / Benötigt.
|
||||||
* `blocks`: Blockiert oder gefährdet (z.B. Risiko -> Projekt).
|
* `blocks`: Blockiert oder gefährdet (z.B. Risiko -> Projekt).
|
||||||
@@ -89,33 +221,47 @@ Dies ist die **mächtigste** Methode. Du sagst dem System explizit, **wie** Ding
|
|||||||
* `similar_to`: Ähnelt / Ist vergleichbar mit.
|
* `similar_to`: Ähnelt / Ist vergleichbar mit.
|
||||||
* `solves`: Löst (Problem).
|
* `solves`: Löst (Problem).
|
||||||
* `based_on`: Basiert auf (Fundament).
|
* `based_on`: Basiert auf (Fundament).
|
||||||
|
* **`prev` / `next` [NEU]**: Markiert chronologische oder evolutionäre Abfolgen (z.B. Leitbild-Evolution).
|
||||||
|
|
||||||
### 4.2 Callout-Edges
|
### 4.2 Callout-Edges (empfohlen)
|
||||||
Für Zusammenfassungen am Ende einer Notiz:
|
Für Zusammenfassungen am Ende einer Notiz, oder eines Absatzes:
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
> [!edge] related_to: [[Vector Embeddings]] [[AI Agents]]
|
> [!edge] related_to
|
||||||
|
> [[Vector Embeddings]]
|
||||||
|
> [[AI Agents]]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Multi-Line Support (v2.9.1):**
|
||||||
|
Callout-Blocks mit mehreren Zeilen werden korrekt verarbeitet. Das System erkennt automatisch, wenn mehrere Links im gleichen Callout-Block stehen, und erstellt für jeden Link eine separate Kante (auch bei Deep-Links zu verschiedenen Sections).
|
||||||
|
|
||||||
|
**Format-agnostische De-Duplizierung:**
|
||||||
|
Wenn Kanten bereits via `[!edge]` Callout vorhanden sind, werden sie nicht mehrfach injiziert. Das System erkennt vorhandene Kanten unabhängig vom Format (Inline, Callout, Wikilink).
|
||||||
|
|
||||||
|
### 4.3 Implizite Bidirektionalität (Edger-Logik) [NEU] [PRÜFEN!]
|
||||||
|
In Mindnet musst du Kanten **nicht** manuell in beide Richtungen pflegen. Der **Edger** übernimmt die Paarbildung automatisch im Hintergrund.
|
||||||
|
|
||||||
|
**Deine Aufgabe:** Setze die Kante in der Datei, die du gerade bearbeitest, so wie es der **logische Fluss** vorgibt.
|
||||||
|
|
||||||
|
* **Blick zurück (Rückwärtslink):** Wenn du ein Ergebnis dokumentierst, nutze `derived_from`, `based_on` oder `prev`.
|
||||||
|
* **Blick nach vorn (Vorwärtslink):** Wenn du einen Plan oder ein Protokoll schreibst, nutze `resulted_in`, `supports` oder `next`.
|
||||||
|
|
||||||
|
**System-Logik (Beispiele):**
|
||||||
|
- Schreibst du in Note A: `next: [[B]]`, weiß das System automatisch: `B prev A`.
|
||||||
|
- Schreibst du in Note B: `derived_from: [[A]]`, weiß das System automatisch: `A resulted_in B`.
|
||||||
|
|
||||||
|
**Vorteil:** Keine redundante Datenpflege, kein "Link-Nightmare", volle Konsistenz im Graphen.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Schreiben für den KI-Zwilling (Szenarien)
|
## 5. Schreiben für den KI-Zwilling (Szenarien)
|
||||||
|
|
||||||
Damit der **RAG-Chat** dich berät, musst du "Futter" für die Decision Engine liefern.
|
Damit der **RAG-Chat** dich berät, musst du "Futter" für die Decision Engine liefern.
|
||||||
|
|
||||||
### Szenario A: Decision Engine (`DECISION`)
|
### 5.1 Szenario A: Decision Engine (`DECISION`)
|
||||||
* **Ziel:** Das System soll abwägen: "Passt Tool X zu mir?"
|
* **Ziel:** Das System soll abwägen: "Passt Tool X zu mir?"
|
||||||
* **Vorgehen:** Erstelle Notizen mit `type: value` oder `type: goal`.
|
* **Vorgehen:** Erstelle Notizen mit `type: value` oder `type: goal`.
|
||||||
|
|
||||||
**Beispiel Notiz:**
|
|
||||||
```yaml
|
|
||||||
---
|
|
||||||
type: value
|
|
||||||
title: Prinzip: Datensparsamkeit
|
|
||||||
---
|
|
||||||
Wir speichern nur das Minimum an Daten. Cloud-Uploads persönlicher Daten sind verboten, es sei denn, sie sind E2E-verschlüsselt.
|
|
||||||
```
|
|
||||||
|
|
||||||
* **Effekt:** Wenn du fragst "Soll ich Notion nutzen?", lädt die Engine diese Notiz und antwortet: *"Nein, Notion ist SaaS ohne E2E. Das verletzt dein Prinzip der Datensparsamkeit."*
|
* **Effekt:** Wenn du fragst "Soll ich Notion nutzen?", lädt die Engine diese Notiz und antwortet: *"Nein, Notion ist SaaS ohne E2E. Das verletzt dein Prinzip der Datensparsamkeit."*
|
||||||
|
|
||||||
### Szenario B: Empathie (`EMPATHY`)
|
### Szenario B: Empathie (`EMPATHY`)
|
||||||
@@ -134,13 +280,27 @@ Ich habe gelernt: Das ist oft das Zeichen kurz vor dem Durchbruch.
|
|||||||
```
|
```
|
||||||
* **Effekt:** Bei "Alles ist grau" findet das System diese Notiz und spiegelt die Lektion zurück.
|
* **Effekt:** Bei "Alles ist grau" findet das System diese Notiz und spiegelt die Lektion zurück.
|
||||||
|
|
||||||
|
### 5.2 **Szenario C: Forward-Mapping (Lücken-Analyse) [NEU]**
|
||||||
|
* **Ziel:** Strategische Wissenslücken füllen.
|
||||||
|
* **Vorgehen:** Erstelle Hub-Notizen mit Forward-Links auf noch nicht existierende Dateien (z. B. `[[Besten Version meiner Selbst]]`).
|
||||||
|
* **Effekt:** Das System erkennt die semantische Bedeutung des geplanten Wissens und kann proaktiv Fragen stellen, um diese Lücken zu schließen.
|
||||||
|
|
||||||
|
### 5.3 **Szenario D: Narratives Gedächtnis (Intention) [NEU]**
|
||||||
|
* **Ziel:** Das System soll verstehen, *warum* eine Entscheidung getroffen wurde, um in ähnlichen künftigen Situationen konsistent zu beraten.
|
||||||
|
* **Vorgehen:** Nutze in `journal`- oder `experience`-Notizen Abschnitte für "Hintergrund" und "Interpretation".
|
||||||
|
* **Beispiel:** Statt "Wert: Disziplin" schreibe "Ich wähle Disziplin, weil ich in meiner Kindheit erlebt habe, wie Willkür schadet."
|
||||||
|
* **Effekt:** Die KI spiegelt nicht nur die Regel, sondern die Überzeugung dahinter.
|
||||||
|
|
||||||
|
### 5.4 Szenario C: Forward-Mapping (Lücken-Analyse)
|
||||||
|
Setze bewusst Links auf Dateien, die noch nicht existieren (z.B. `[[Die beste Version meiner selbst]]`). Die KI erkennt diese semantischen Lücken und stellt im Chat proaktive Fragen, um diese Felder mit dir zu füllen.
|
||||||
|
|
||||||
|
### 5.5 Szenario D: Narratives Gedächtnis
|
||||||
|
Dokumentiere das "Warum" (Fleisch am Knochen). Wenn du schreibst: *"Ich wähle Disziplin, weil ich Willkür als Kind schmerzhaft erlebt habe"*, versteht die KI die emotionale Kausalität und kann dich in Krisen an diesen Ursprung erinnern.
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Best Practices & Beispiele (Klassik)
|
## 6. Best Practices & Beispiele
|
||||||
|
|
||||||
Hier sind vollständige Vorlagen für häufige Typen.
|
### 6.1 Beispiel: Projekt-Notiz (Standard)
|
||||||
|
|
||||||
### 6.1 Beispiel: Projekt-Notiz
|
|
||||||
Projekte profitieren von `depends_on`, um Abhängigkeiten zu klären.
|
Projekte profitieren von `depends_on`, um Abhängigkeiten zu klären.
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
@@ -159,11 +319,11 @@ Wir bauen ein persönliches Wissensnetz.
|
|||||||
Wir nutzen [[rel:depends_on Qdrant]] für die Vektorsuche und [[rel:depends_on FastAPI]] für das Backend.
|
Wir nutzen [[rel:depends_on Qdrant]] für die Vektorsuche und [[rel:depends_on FastAPI]] für das Backend.
|
||||||
|
|
||||||
## Architektur
|
## Architektur
|
||||||
Das Konzept basiert auf [[RAG Architecture]]. (Automatisch 'depends_on' durch Typ-Default, falls konfiguriert).
|
Das Konzept basiert auf [[RAG Architecture]].
|
||||||
```
|
```
|
||||||
|
|
||||||
### 6.2 Beispiel: Entscheidung (Decision Record)
|
### 6.2 Beispiel: Advanced Tuning (Manuelles Override)
|
||||||
Entscheidungen sind hoch gewichtet (`retriever_weight: 1.0`).
|
Hier zwingen wir das System, eine Entscheidung extrem kleinteilig (`strict`) zu zerlegen und in der Suche maximal zu priorisieren.
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
---
|
---
|
||||||
@@ -172,6 +332,8 @@ title: ADR: Wahl von Qdrant
|
|||||||
type: decision
|
type: decision
|
||||||
status: final
|
status: final
|
||||||
tags: [architektur, db]
|
tags: [architektur, db]
|
||||||
|
chunking_profile: structured_smart_edges_strict
|
||||||
|
retriever_weight: 1.5
|
||||||
---
|
---
|
||||||
|
|
||||||
# Entscheidung: Qdrant
|
# Entscheidung: Qdrant
|
||||||
@@ -180,13 +342,11 @@ Wir haben uns für Qdrant entschieden.
|
|||||||
|
|
||||||
## Alternativen
|
## Alternativen
|
||||||
Wir haben auch [[rel:similar_to Pinecone]] und [[rel:similar_to Weaviate]] betrachtet.
|
Wir haben auch [[rel:similar_to Pinecone]] und [[rel:similar_to Weaviate]] betrachtet.
|
||||||
|
|
||||||
## Begründung
|
|
||||||
Qdrant erlaubt lokalen Betrieb und [[rel:solves Payload Filtering Requirements]].
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Langfristige Stabilität
|
## 7. Virtual Schema Layer
|
||||||
|
|
||||||
Wir nutzen das Prinzip des **Virtual Schema Layers**. Wir kodieren keine Logik (wie `chunk_size`) in die Notizen. Das wird zentral in der `types.yaml` verwaltet. Das bedeutet für dich: Du kannst dich rein auf den Inhalt konzentrieren. Wenn wir die Chunking-Strategie ändern, müssen wir nicht 1000 Markdown-Dateien anfassen.
|
Grundsätzlich gilt das Prinzip des **Virtual Schema Layers**. Die Logik (wie `chunk_size`) wird zentral in der `types.yaml` verwaltet.
|
||||||
|
**Aber:** Als Power-User hast du über die Overrides jederzeit die Möglichkeit, aus diesem Standard auszubrechen.
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
doc_type: technical_reference
|
||||||
|
audience: developer, power_user
|
||||||
|
scope: obsidian, scripts, workflow
|
||||||
|
status: active
|
||||||
|
version: 1.0.0
|
||||||
|
context: "Setup und Dokumentation der Obsidian-Integration für Mindnet v2.9."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Obsidian Integration Guide
|
||||||
|
|
||||||
|
Dieses Dokument beschreibt die technische Implementierung der Mindnet-Logik innerhalb von Obsidian mittels Templater-Skripten.
|
||||||
|
|
||||||
|
## 1. Voraussetzungen
|
||||||
|
|
||||||
|
Um die volle Funktionalität des Mindnet-Workflows zu nutzen, sind folgende Obsidian-Plugins zwingend erforderlich:
|
||||||
|
* **Templater:** Für die Ausführung der JavaScript-Logik beim Erstellen von Notizen und Kanten.
|
||||||
|
* **Meta Bind (Optional):** Zur intuitiven Steuerung von Frontmatter-Feldern wie `retriever_weight`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Kern-Komponenten
|
||||||
|
|
||||||
|
### 2.1 Mindnet Universal Creator (`mindnetCreate.js`)
|
||||||
|
Das zentrale Skript zur Erstellung neuer Notizen. Es stellt sicher, dass jede Datei im korrekten Ordner landet und das notwendige Frontmatter für das Mindnet-Backend erhält.
|
||||||
|
|
||||||
|
**Kern-Funktionen:**
|
||||||
|
* Automatische Typ-Auswahl basierend auf den Vorlagen in `_system/templates/creation`.
|
||||||
|
* Validierung und Erzwingung der ID-Syntax (`YYYYMMDDHHmm-slug`).
|
||||||
|
* Pfad-Erzwingung bei Verlinkungen, um die Konsistenz des Graphen zu wahren.
|
||||||
|
|
||||||
|
### 2.2 Edge Callout Selector Skript
|
||||||
|
Dieses Skript ermöglicht das Einfügen von semantischen Kanten (`[!edge]`) durch Auswahl aus dem zentralen Wörterbuch.
|
||||||
|
|
||||||
|
**Features:**
|
||||||
|
* **Kategorisierte Auswahl:** Gruppierung der Kanten nach Themenbereichen (H3-Ebene der Vokabel-Datei).
|
||||||
|
* **Alias-Support:** Ermöglicht die Wahl zwischen technischem System-Namen (Canonical) und benutzerfreundlichen Aliasen.
|
||||||
|
* **Automatisches Callout-Formatting:** Transformiert markierte Wikilinks direkt in einen validen Edge-Block.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Konfiguration & Speicherorte
|
||||||
|
|
||||||
|
Damit die Skripte funktionieren, müssen folgende Pfade im Vault existieren:
|
||||||
|
|
||||||
|
| Pfad | Inhalt / Zweck |
|
||||||
|
| :--- | :--- |
|
||||||
|
| `_system/scripts/` | Speicherort für die `.js` Module (z.B. `mindnetCreate.js`). |
|
||||||
|
| `_system/templates/` | Ablage für die Templater-Trigger-Dateien. |
|
||||||
|
| `_system/templates/creation/` | Enthält die Typ-spezifischen Vorlagen (z.B. `template-insight.md`). |
|
||||||
|
| `_system/dictionary/edge_vocabulary.md` | Single Source of Truth für alle erlaubten Kanten-Typen. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Workflow-Integration
|
||||||
|
|
||||||
|
### 4.1 Notiz-Erstellung
|
||||||
|
1. Verwenden des Hotkeys für den `Mindnet Creator`.
|
||||||
|
2. Auswahl des gewünschten Typs (z. B. `EXPERIENCE`).
|
||||||
|
3. Das Skript erstellt die Datei im Zielordner und öffnet sie mit den passenden Leitfragen.
|
||||||
|
|
||||||
|
### 4.2 Kanten-Management
|
||||||
|
1. Markieren eines oder mehrerer Wikilinks im Text.
|
||||||
|
2. Starten des `Insert Edge Callout` Skripts.
|
||||||
|
3. Auswahl des Kanten-Typs (z. B. `caused_by`) und der gewünschten Bezeichnung.
|
||||||
|
4. Das Skript fügt den Block automatisch unter der Markierung ein.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Wartung & Updates
|
||||||
|
|
||||||
|
Bei Änderungen an den Notiz-Typen in der `types.yaml` müssen die entsprechenden Markdown-Vorlagen im Ordner `creation` manuell synchronisiert werden, um die Konsistenz zwischen Obsidian und dem Backend zu wahren.
|
||||||
@@ -1,73 +1,83 @@
|
|||||||
---
|
---
|
||||||
doc_type: concept
|
doc_type: concept
|
||||||
audience: architect, product_owner
|
audience: architect, product_owner
|
||||||
scope: ai, router, personas
|
scope: ai, router, personas, resilience
|
||||||
status: active
|
status: active
|
||||||
version: 2.6
|
version: 2.8.1
|
||||||
context: "Fachkonzept der KI-Persönlichkeit, der Decision Engine und Erweiterungsstrategien."
|
context: "Fachkonzept der hybriden KI-Persönlichkeit, der Provider-Kaskade und der kognitiven Resilienz (Deep Fallback)."
|
||||||
---
|
---
|
||||||
|
|
||||||
# Konzept: KI-Persönlichkeit & Router
|
# Konzept: KI-Persönlichkeit & Router
|
||||||
|
|
||||||
**Quellen:** `mindnet_functional_architecture.md`, `Programmplan_V2.2.md`
|
**Quellen:** `mindnet_functional_architecture.md`, `llm_service.py`, `config.py`, `ingestion.py`
|
||||||
|
|
||||||
Mindnet soll nicht wie eine Suchmaschine wirken, sondern wie ein **Digitaler Zwilling**. Dazu muss das System erkennen, **was** der Nutzer will, und seine "Persönlichkeit" anpassen.
|
Mindnet soll nicht wie eine Suchmaschine wirken, sondern wie ein **Digitaler Zwilling**. Dazu muss das System erkennen, **was** der Nutzer will, und seine „Persönlichkeit“ sowie seine technische Infrastruktur dynamisch anpassen.
|
||||||
|
|
||||||
## 1. Der Hybrid Router (Das Gehirn)
|
## 1. Der Hybrid Router (Das Gehirn)
|
||||||
|
|
||||||
Jede Eingabe durchläuft den **Hybrid Router**. Er entscheidet über die Strategie.
|
Jede Eingabe durchläuft den **Hybrid Router**. Er entscheidet über die fachliche Strategie und die technische Ausführung.
|
||||||
|
|
||||||
### Modus A: RAG (Retrieval Augmented Generation)
|
### Modus A: RAG (Retrieval Augmented Generation)
|
||||||
* *Intent:* Der Nutzer hat eine Frage oder ein Problem (`FACT`, `DECISION`, `EMPATHY`).
|
* **Intent:** Der Nutzer hat eine Frage oder ein Problem (`FACT`, `DECISION`, `EMPATHY`).
|
||||||
* *Aktion:* Das System sucht im Gedächtnis und generiert eine Antwort.
|
* **Aktion:** Das System sucht im Gedächtnis und generiert eine Antwort.
|
||||||
|
|
||||||
### Modus B: Interview (Knowledge Capture)
|
### Modus B: Interview (Knowledge Capture)
|
||||||
* *Intent:* Der Nutzer will Wissen speichern (`INTERVIEW`).
|
* **Intent:** Der Nutzer will Wissen speichern (`INTERVIEW`).
|
||||||
* *Aktion:* Das System sucht **nicht**, sondern fragt ab und erstellt einen Draft.
|
* **Aktion:** Das System sucht **nicht**, sondern fragt ab und erstellt einen Draft.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Die Personas (Strategien)
|
## 2. Die hybride LLM-Landschaft (Resilienz-Kaskade)
|
||||||
|
|
||||||
|
Ein intelligenter Zwilling muss jederzeit verfügbar sein. Mindnet v2.8.1 nutzt eine **dreistufige Kaskade**, um Intelligenz, Kosten und Verfügbarkeit zu optimieren:
|
||||||
|
|
||||||
|
1. **Stufe 1: Cloud-Speed (Turbo-Mode):** Primäre Wahl für komplexe Extraktionsaufgaben und schnelle RAG-Antworten mittels OpenRouter (Mistral-7B) oder Google Gemini (2.5-flash-lite).
|
||||||
|
2. **Stufe 2: Quoten-Resilienz:** Erkennt das System eine Drosselung durch Cloud-Provider (HTTP 429), pausiert es kontrolliert (`LLM_RATE_LIMIT_WAIT`), führt automatisierte Retries durch und schützt so den laufenden Prozess.
|
||||||
|
3. **Stufe 3: Deep Fallback & lokale Souveränität (Ollama):** * **Technischer Fallback:** Schlagen alle Cloud-Versuche fehl, übernimmt das lokale Modell (Phi-3).
|
||||||
|
* **Kognitiver Fallback (v2.11.14):** Liefert die Cloud zwar technisch eine Antwort, verweigert aber inhaltlich die Verarbeitung (Silent Refusal/Policy Violation), wird ein **Deep Fallback** erzwungen, um die Datenintegrität lokal zu retten.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Die Personas (Strategien)
|
||||||
|
|
||||||
Mindnet wechselt den Hut, je nach Situation.
|
Mindnet wechselt den Hut, je nach Situation.
|
||||||
|
|
||||||
### 2.1 Der Berater (Strategy: DECISION)
|
### 3.1 Der Berater (Strategy: DECISION)
|
||||||
* **Auslöser:** Fragen wie "Soll ich...?", "Was ist besser?".
|
* **Auslöser:** Fragen wie „Soll ich...?“, „Was ist besser?“.
|
||||||
* **Strategic Retrieval:** Lädt aktiv Notizen der Typen `value` (Werte), `goal` (Ziele) und `risk` (Risiken), auch wenn sie im Text nicht direkt vorkommen.
|
* **Strategic Retrieval:** Lädt aktiv Notizen der Typen `value` (Werte), `goal` (Ziele) und `risk` (Risiken), auch wenn sie im Text nicht direkt vorkommen.
|
||||||
* **Reasoning:** *"Wäge die Fakten gegen meine Werte ab. Sei strikt bei Risiken."*
|
* **Reasoning:** *„Wäge die Fakten gegen meine Werte ab. Sei strikt bei Risiken.“*
|
||||||
|
|
||||||
### 2.2 Der Spiegel (Strategy: EMPATHY)
|
### 3.2 Der Spiegel (Strategy: EMPATHY)
|
||||||
* **Auslöser:** Emotionale Aussagen ("Ich bin frustriert").
|
* **Auslöser:** Emotionale Aussagen („Ich bin frustriert“).
|
||||||
* **Strategic Retrieval:** Lädt `experience` (Erfahrungen) und `belief` (Glaubenssätze).
|
* **Strategic Retrieval:** Lädt `experience` (Erfahrungen) und `belief` (Glaubenssätze).
|
||||||
* **Reasoning:** *"Nutze meine eigenen Erfahrungen, um die Situation einzuordnen."*
|
* **Reasoning:** *„Nutze meine eigenen Erfahrungen, um die Situation einzuordnen.“*
|
||||||
|
|
||||||
### 2.3 Der Bibliothekar (Strategy: FACT)
|
### 3.3 Der Bibliothekar (Strategy: FACT)
|
||||||
* **Auslöser:** Sachfragen ("Was ist Qdrant?").
|
* **Auslöser:** Sachfragen („Was ist Qdrant?“).
|
||||||
* **Behavior:** Präzise, neutral, kurz.
|
* **Behavior:** Präzise, neutral, kurz.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Future Concepts: The Empathic Digital Twin
|
## 4. Future Concepts: The Empathic Digital Twin
|
||||||
|
|
||||||
Um Mindnet von einer Maschine zu einem echten Spiegel der Persönlichkeit zu entwickeln, sind folgende Konzepte in der Architektur angelegt:
|
### 4.1 Antizipation durch Erfahrung
|
||||||
|
* **Konzept:** Das System soll Konsequenzen vorhersagen („Was passiert, wenn...?“).
|
||||||
|
* **Logik:** *„In einer ähnlichen Situation (Projekt A) hat Entscheidung X zu Ergebnis Y geführt.“* (Analogie-Schluss).
|
||||||
|
|
||||||
### 3.1 Antizipation durch Erfahrung
|
### 4.2 Empathie & „Ich“-Modus
|
||||||
* **Konzept:** Das System soll Konsequenzen vorhersagen ("Was passiert, wenn...?").
|
|
||||||
* **Logik:** *"In einer ähnlichen Situation (Projekt A) hat Entscheidung X zu Ergebnis Y geführt."* (Analogie-Schluss).
|
|
||||||
|
|
||||||
### 3.2 Empathie & "Ich"-Modus
|
|
||||||
* **Konzept:** Das System antwortet im Tonfall des Nutzers.
|
* **Konzept:** Das System antwortet im Tonfall des Nutzers.
|
||||||
* **Umsetzung:** Few-Shot Prompting mit eigenen E-Mails/Texten als Stilvorlage.
|
* **Umsetzung:** Few-Shot Prompting mit eigenen E-Mails/Texten als Stilvorlage.
|
||||||
|
|
||||||
### 3.3 Glaubenssätze & Rituale
|
### 4.3 Resilienz als Charakterzug
|
||||||
* **Konzept:** Berücksichtigung weicher Faktoren.
|
Durch die **WP-20 Implementierung** zeigt das System „Geduld“: Bei Quoten-Engpässen bricht es nicht ab, sondern wartet auf freie Kapazitäten. Durch das **Deep Fallback (v2.11.14)** besitzt Mindnet eine kognitive Ausdauer, die inhaltliche Zensur oder Blockaden der Cloud erkennt und durch lokale Rechenpower auflöst.
|
||||||
* **Szenario:** Bei Terminplanungen werden Rituale ("Keine Meetings vor 10 Uhr") automatisch als harte Restriktion gegen Anfragen geprüft.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Erweiterbarkeit: Das "Teach-the-AI" Paradigma
|
## 5. Erweiterbarkeit: Das „Teach-the-AI“ Paradigma
|
||||||
|
|
||||||
Mindnet lernt nicht durch Training (Fine-Tuning), sondern durch **Konfiguration** und **Vernetzung**. Wenn du dem System ein neues Konzept beibringen willst, musst du an drei Stellen eingreifen.
|
Mindnet lernt durch **Konfiguration** und **Vernetzung**.
|
||||||
|
|
||||||
**Beispiel: Du willst den Typ `risk` einführen.**
|
**Beispiel: Du willst den Typ `risk` einführen.**
|
||||||
|
|
||||||
@@ -87,6 +97,6 @@ DECISION:
|
|||||||
```
|
```
|
||||||
|
|
||||||
**3. Kognitive Ebene (Verständnis)**
|
**3. Kognitive Ebene (Verständnis)**
|
||||||
In `prompts.yaml`: Erkläre dem LLM, was ein Risiko ist.
|
In `prompts.yaml`: Erkläre dem LLM (provider-spezifisch mittels der Prompt-Kaskade), was ein Risiko ist.
|
||||||
|
|
||||||
**Fazit:** Nur wenn **Daten** (Vault), **Physik** (Config) und **Semantik** (Prompt) zusammenspielen, entsteht ein intelligenter Zwilling.
|
**Fazit:** Nur wenn **Daten** (Vault), **Infrastruktur** (Resiliente Kaskade) und **Semantik** (Prompt) zusammenspielen, entsteht ein intelligenter Zwilling.
|
||||||
@@ -0,0 +1,456 @@
|
|||||||
|
---
|
||||||
|
doc_type: concept
|
||||||
|
audience: architect, developer
|
||||||
|
scope: architecture, design_patterns, modularization
|
||||||
|
status: active
|
||||||
|
version: 2.9.1
|
||||||
|
context: "Architektur-Patterns, Design-Entscheidungen und modulare Struktur von Mindnet. Basis für Verständnis und Erweiterungen."
|
||||||
|
---
|
||||||
|
|
||||||
|
# Architektur-Patterns & Design-Entscheidungen
|
||||||
|
|
||||||
|
Dieses Dokument beschreibt die zentralen Architektur-Patterns und Design-Entscheidungen, die Mindnet prägen. Es dient als Referenz für Entwickler und Architekten, um das System zu verstehen und konsistent zu erweitern.
|
||||||
|
|
||||||
|
## 1. Kern-Paradigmen
|
||||||
|
|
||||||
|
### 1.1 Filesystem First (Source of Truth)
|
||||||
|
|
||||||
|
**Prinzip:** Markdown-Dateien im Vault sind die einzige Quelle der Wahrheit. Die Datenbank (Qdrant) ist ein abgeleiteter Index.
|
||||||
|
|
||||||
|
**Implikationen:**
|
||||||
|
- Dateien werden immer direkt von der Festplatte gelesen (z.B. im Editor)
|
||||||
|
- Datenbank-Inhalte können aus Markdown rekonstruiert werden
|
||||||
|
- Änderungen erfolgen primär im Vault, nicht in der DB
|
||||||
|
- Die Datenbank ist ein Cache, kein primärer Speicher
|
||||||
|
|
||||||
|
**Code-Beispiele:**
|
||||||
|
- `ui_callbacks.py`: Liest Dateien direkt von Disk
|
||||||
|
- `ingestion_processor.py`: Schreibt zuerst auf Disk, dann in DB
|
||||||
|
|
||||||
|
### 1.2 Late Binding (Späte Semantik)
|
||||||
|
|
||||||
|
**Prinzip:** Struktur und Interpretation werden in Konfigurationen definiert, nicht im Code.
|
||||||
|
|
||||||
|
**Implikationen:**
|
||||||
|
- Neue Note-Typen werden in `types.yaml` definiert, nicht im Code
|
||||||
|
- Prompt-Templates sind in `prompts.yaml` konfigurierbar
|
||||||
|
- Edge-Typen werden in `edge_vocabulary.md` verwaltet
|
||||||
|
- Die "Persönlichkeit" ist Config, kein Code
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Erweiterbarkeit ohne Code-Änderungen
|
||||||
|
- A/B-Testing von Prompt-Strategien
|
||||||
|
- Anpassung an verschiedene Use-Cases
|
||||||
|
|
||||||
|
### 1.3 Virtual Schema Layer
|
||||||
|
|
||||||
|
**Prinzip:** Markdown-Dateien benötigen nur minimale Frontmatter-Angaben. Komplexität wird zur Laufzeit injiziert.
|
||||||
|
|
||||||
|
**Beispiel:**
|
||||||
|
```yaml
|
||||||
|
# Minimal im Frontmatter
|
||||||
|
---
|
||||||
|
id: 20250101-test
|
||||||
|
title: Test
|
||||||
|
type: project
|
||||||
|
---
|
||||||
|
|
||||||
|
# Im Code wird zur Laufzeit ergänzt:
|
||||||
|
# - chunking_profile aus types.yaml
|
||||||
|
# - retriever_weight aus types.yaml
|
||||||
|
# - edge_defaults aus types.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Robuste Markdown-Dateien (weniger Breaking Changes)
|
||||||
|
- Zentrale Verwaltung von Logik
|
||||||
|
- Einfache Migration zwischen Versionen
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Modulare Architektur (WP-14)
|
||||||
|
|
||||||
|
### 2.1 Paket-Struktur
|
||||||
|
|
||||||
|
Seit WP-14 ist die Core-Logik in spezialisierte Pakete unterteilt:
|
||||||
|
|
||||||
|
```
|
||||||
|
app/core/
|
||||||
|
├── chunking/ # Text-Segmentierung
|
||||||
|
│ ├── chunking_strategies.py # Sliding/Heading-Strategien
|
||||||
|
│ ├── chunking_processor.py # Orchestrierung
|
||||||
|
│ └── chunking_propagation.py # Edge-Vererbung
|
||||||
|
├── database/ # Qdrant-Infrastruktur
|
||||||
|
│ ├── qdrant.py # Client & Config
|
||||||
|
│ └── qdrant_points.py # Point-Mapping
|
||||||
|
├── graph/ # Graph-Logik
|
||||||
|
│ ├── graph_subgraph.py # Expansion
|
||||||
|
│ └── graph_weights.py # Scoring
|
||||||
|
├── ingestion/ # Import-Pipeline
|
||||||
|
│ ├── ingestion_processor.py # Two-Pass Workflow
|
||||||
|
│ └── ingestion_validation.py # Mistral-safe Parsing
|
||||||
|
├── parser/ # Markdown-Parsing
|
||||||
|
│ ├── parsing_markdown.py # Frontmatter/Body
|
||||||
|
│ └── parsing_scanner.py # File-Scan
|
||||||
|
└── retrieval/ # Suche & Scoring
|
||||||
|
├── retriever.py # Orchestrator
|
||||||
|
└── retriever_scoring.py # Mathematik
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 Design-Pattern: Proxy-Adapter (Abwärtskompatibilität)
|
||||||
|
|
||||||
|
**Problem:** Nach WP-14 Modularisierung müssen alte Import-Pfade weiter funktionieren.
|
||||||
|
|
||||||
|
**Lösung:** Proxy-Module leiten Anfragen an neue Pakete weiter.
|
||||||
|
|
||||||
|
**Beispiel:**
|
||||||
|
```python
|
||||||
|
# app/core/retriever.py (Proxy)
|
||||||
|
from .retrieval.retriever import (
|
||||||
|
Retriever, hybrid_retrieve, semantic_retrieve
|
||||||
|
)
|
||||||
|
__all__ = ["Retriever", "hybrid_retrieve", "semantic_retrieve"]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Keine Breaking Changes für bestehenden Code
|
||||||
|
- Graduelle Migration möglich
|
||||||
|
- Alte Scripts funktionieren weiter
|
||||||
|
|
||||||
|
### 2.3 Design-Pattern: Singleton (Edge Registry)
|
||||||
|
|
||||||
|
**Problem:** Edge Registry muss konsistent über alle Services sein.
|
||||||
|
|
||||||
|
**Lösung:** Singleton-Pattern mit Lazy Loading.
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
# app/services/edge_registry.py
|
||||||
|
class EdgeRegistry:
|
||||||
|
_instance = None
|
||||||
|
|
||||||
|
def __new__(cls):
|
||||||
|
if cls._instance is None:
|
||||||
|
cls._instance = super().__new__(cls)
|
||||||
|
return cls._instance
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Einheitliche Validierung
|
||||||
|
- Keine Duplikation von Vokabular-Daten
|
||||||
|
- Thread-safe Initialisierung
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Asynchrone Verarbeitung
|
||||||
|
|
||||||
|
### 3.1 Background Tasks Pattern
|
||||||
|
|
||||||
|
**Problem:** Ingestion kann lange dauern (LLM-Calls, Embeddings). Blockiert API nicht.
|
||||||
|
|
||||||
|
**Lösung:** FastAPI Background Tasks für non-blocking Verarbeitung.
|
||||||
|
|
||||||
|
**Flow:**
|
||||||
|
1. API empfängt Request (`/ingest/save`)
|
||||||
|
2. Datei wird sofort auf Disk geschrieben
|
||||||
|
3. API antwortet mit `status: "queued"`
|
||||||
|
4. Background Task startet Ingestion asynchron
|
||||||
|
5. User kann weiterarbeiten
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
@router.post("/save")
|
||||||
|
async def save_note(req: SaveRequest, background_tasks: BackgroundTasks):
|
||||||
|
# Sofortige Persistenz
|
||||||
|
write_to_disk(req.markdown_content)
|
||||||
|
|
||||||
|
# Async Processing
|
||||||
|
background_tasks.add_task(run_ingestion_task, ...)
|
||||||
|
|
||||||
|
return SaveResponse(status="queued", ...)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Keine Timeouts bei großen Dateien
|
||||||
|
- Bessere User Experience
|
||||||
|
- System bleibt responsiv
|
||||||
|
|
||||||
|
### 3.2 Traffic Control (Semaphore Pattern)
|
||||||
|
|
||||||
|
**Problem:** Parallele LLM-Calls können System überlasten.
|
||||||
|
|
||||||
|
**Lösung:** Semaphore begrenzt parallele Hintergrund-Tasks.
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
# app/services/llm_service.py
|
||||||
|
background_semaphore = asyncio.Semaphore(
|
||||||
|
settings.BACKGROUND_LIMIT # Default: 2
|
||||||
|
)
|
||||||
|
|
||||||
|
async def generate(...):
|
||||||
|
if priority == "background":
|
||||||
|
async with background_semaphore:
|
||||||
|
return await _call_llm(...)
|
||||||
|
else: # realtime
|
||||||
|
return await _call_llm(...) # Keine Limitierung
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Schutz vor API-Quoten-Überschreitung
|
||||||
|
- Stabiler Betrieb bei hoher Last
|
||||||
|
- Priorisierung von Echtzeit-Anfragen
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Resilienz-Patterns
|
||||||
|
|
||||||
|
### 4.1 Provider-Kaskade (Fallback-Kette)
|
||||||
|
|
||||||
|
**Problem:** Cloud-Provider können ausfallen oder Quoten überschreiten.
|
||||||
|
|
||||||
|
**Lösung:** Dreistufige Kaskade mit intelligentem Fallback.
|
||||||
|
|
||||||
|
**Stufen:**
|
||||||
|
1. **Cloud (OpenRouter/Gemini):** Schnell, aber abhängig von Provider
|
||||||
|
2. **Rate-Limit-Handling:** Automatische Retries bei HTTP 429
|
||||||
|
3. **Lokaler Fallback (Ollama):** Langsam, aber immer verfügbar
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
# app/services/llm_service.py
|
||||||
|
async def generate(...):
|
||||||
|
try:
|
||||||
|
return await _call_cloud(...)
|
||||||
|
except RateLimitError:
|
||||||
|
await asyncio.sleep(LLM_RATE_LIMIT_WAIT)
|
||||||
|
return await _call_cloud(...) # Retry
|
||||||
|
except Exception:
|
||||||
|
return await _call_ollama(...) # Fallback
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Hohe Verfügbarkeit
|
||||||
|
- Kostenoptimierung (Cloud für Speed, Lokal für Fallback)
|
||||||
|
- Resilienz gegen Provider-Ausfälle
|
||||||
|
|
||||||
|
### 4.2 Deep Fallback (Kognitiver Fallback)
|
||||||
|
|
||||||
|
**Problem:** Cloud liefert technisch erfolgreiche, aber inhaltlich leere Antworten (Silent Refusal).
|
||||||
|
|
||||||
|
**Lösung:** Validierung der Antwort-Qualität, nicht nur HTTP-Status.
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
response = await _call_cloud(...)
|
||||||
|
if is_valid_json(response):
|
||||||
|
return response
|
||||||
|
else:
|
||||||
|
# Deep Fallback: Cloud hat blockiert, lokaler Fallback
|
||||||
|
return await _call_ollama(...)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Erkennung von Policy-Violations
|
||||||
|
- Datenintegrität trotz Cloud-Filter
|
||||||
|
- Lokale Souveränität
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Datenfluss-Patterns
|
||||||
|
|
||||||
|
### 5.1 Two-Pass Workflow (WP-15b)
|
||||||
|
|
||||||
|
**Problem:** Smart Edge Validation benötigt globalen Kontext (alle Notizen).
|
||||||
|
|
||||||
|
**Lösung:** Zwei-Phasen-Import mit Pre-Scan.
|
||||||
|
|
||||||
|
**Pass 1: Pre-Scan**
|
||||||
|
- Schnelles Scannen aller Dateien
|
||||||
|
- Aufbau von `LocalBatchCache` (IDs, Titel, Summaries)
|
||||||
|
- Keine LLM-Calls
|
||||||
|
|
||||||
|
**Pass 2: Semantic Processing**
|
||||||
|
- Nur für geänderte Dateien
|
||||||
|
- Binäre Validierung gegen Cache
|
||||||
|
- Effiziente Nutzung von LLM-Quoten
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Reduktion von LLM-Calls (nur Validierung, keine Extraktion)
|
||||||
|
- Konsistente Validierung (globaler Kontext)
|
||||||
|
- Schnellerer Import
|
||||||
|
|
||||||
|
### 5.2 Idempotenz-Pattern
|
||||||
|
|
||||||
|
**Prinzip:** Mehrfache Imports derselben Datei führen zum gleichen Ergebnis.
|
||||||
|
|
||||||
|
**Mechanismen:**
|
||||||
|
- **Deterministische IDs:** UUIDv5 basierend auf Datei-Inhalt
|
||||||
|
- **Hash-basierte Change Detection:** Multi-Hash für `body` und `full`
|
||||||
|
- **Deduplizierung:** Kanten werden anhand Identität erkannt
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
# Deterministische ID
|
||||||
|
note_id = uuid.uuid5(NAMESPACE_URL, f"{file_path}#{content_hash}")
|
||||||
|
|
||||||
|
# Change Detection
|
||||||
|
if current_hash == stored_hash:
|
||||||
|
skip_processing() # Idempotent
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Sicherheit bei Re-Imports
|
||||||
|
- Keine Duplikate
|
||||||
|
- Konsistente Datenbank
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Frontend-Patterns
|
||||||
|
|
||||||
|
### 6.1 Active Inspector Pattern
|
||||||
|
|
||||||
|
**Problem:** Graph-Re-Renders bei Knoten-Selektion führen zu Flackern.
|
||||||
|
|
||||||
|
**Lösung:** CSS-Klassen statt State-Änderungen für Selektion.
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
# ui_graph_cytoscape.py
|
||||||
|
stylesheet = [
|
||||||
|
{
|
||||||
|
"selector": ".inspected",
|
||||||
|
"style": {"border-width": 6, "border-color": "#FFC300"}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
# Selektion ändert nur CSS-Klasse, nicht React-Key
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Stabiles UI (kein Re-Render)
|
||||||
|
- Bessere Performance
|
||||||
|
- Smooth User Experience
|
||||||
|
|
||||||
|
### 6.2 Resurrection Pattern
|
||||||
|
|
||||||
|
**Problem:** Streamlit vergisst Eingaben bei Re-Runs.
|
||||||
|
|
||||||
|
**Lösung:** Aggressive Synchronisation in `session_state`.
|
||||||
|
|
||||||
|
**Code:**
|
||||||
|
```python
|
||||||
|
# ui_editor.py
|
||||||
|
if widget_key not in st.session_state:
|
||||||
|
st.session_state[widget_key] = restore_from_data_key()
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Texteingaben überleben Tab-Wechsel
|
||||||
|
- Keine Datenverluste
|
||||||
|
- Bessere UX
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Erweiterbarkeit: "Teach-the-AI" Paradigma
|
||||||
|
|
||||||
|
Mindnet lernt durch **Konfiguration**, nicht durch Training.
|
||||||
|
|
||||||
|
### 7.1 Drei-Ebenen-Erweiterung
|
||||||
|
|
||||||
|
**1. Daten-Ebene (`types.yaml`):**
|
||||||
|
```yaml
|
||||||
|
risk:
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["blocks"]
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Strategie-Ebene (`decision_engine.yaml`):**
|
||||||
|
```yaml
|
||||||
|
DECISION:
|
||||||
|
inject_types: ["value", "risk"]
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Kognitive Ebene (`prompts.yaml`):**
|
||||||
|
```yaml
|
||||||
|
risk_definition:
|
||||||
|
ollama: "Ein Risiko ist eine potenzielle Gefahr..."
|
||||||
|
```
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Keine Code-Änderungen nötig
|
||||||
|
- Schnelle Iteration
|
||||||
|
- A/B-Testing möglich
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Design-Entscheidungen & Trade-offs
|
||||||
|
|
||||||
|
### 8.1 Qdrant als Vektor-DB
|
||||||
|
|
||||||
|
**Entscheidung:** Qdrant statt Pinecone/Weaviate
|
||||||
|
|
||||||
|
**Gründe:**
|
||||||
|
- Self-hosted (Privacy First)
|
||||||
|
- Open Source
|
||||||
|
- Gute Performance bei lokaler Installation
|
||||||
|
|
||||||
|
**Trade-off:** Mehr Wartungsaufwand als Managed Service
|
||||||
|
|
||||||
|
### 8.2 Hybrid Retrieval (Semantik + Graph)
|
||||||
|
|
||||||
|
**Entscheidung:** Kombination von Vektor-Suche und Graph-Expansion
|
||||||
|
|
||||||
|
**Gründe:**
|
||||||
|
- Semantik findet ähnliche Inhalte
|
||||||
|
- Graph findet strukturelle Verbindungen
|
||||||
|
- Kombination liefert bessere Relevanz
|
||||||
|
|
||||||
|
**Trade-off:** Höhere Komplexität, mehr Berechnungsaufwand
|
||||||
|
|
||||||
|
### 8.3 Background Tasks statt Queue-System
|
||||||
|
|
||||||
|
**Entscheidung:** FastAPI Background Tasks statt Redis/RabbitMQ
|
||||||
|
|
||||||
|
**Gründe:**
|
||||||
|
- Einfacher Setup (keine zusätzliche Infrastruktur)
|
||||||
|
- Ausreichend für Single-User-Szenario
|
||||||
|
- Weniger Moving Parts
|
||||||
|
|
||||||
|
**Trade-off:** Keine Persistenz bei Server-Neustart (Tasks gehen verloren)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Konsistenz-Garantien
|
||||||
|
|
||||||
|
### 9.1 Deterministische IDs
|
||||||
|
|
||||||
|
**Prinzip:** UUIDv5 basierend auf Datei-Inhalt
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Reproduzierbare IDs
|
||||||
|
- Keine Duplikate bei Re-Import
|
||||||
|
- Graph ist aus Markdown rekonstruierbar
|
||||||
|
|
||||||
|
### 9.2 Provenance-Hierarchie
|
||||||
|
|
||||||
|
**Prinzip:** Kanten haben Qualitätsstufen (explicit > smart > rule)
|
||||||
|
|
||||||
|
**Vorteile:**
|
||||||
|
- Menschliche Intention hat Vorrang
|
||||||
|
- System-Heuristiken können überschrieben werden
|
||||||
|
- Transparente Gewichtung
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Weitere Informationen
|
||||||
|
|
||||||
|
- **Vision & Strategie:** Siehe [Vision & Strategie](../00_General/00_vision_and_strategy.md)
|
||||||
|
- **Graph-Logik:** Siehe [Graph-Logik](02_concept_graph_logic.md)
|
||||||
|
- **KI-Persönlichkeit:** Siehe [KI-Persönlichkeit](02_concept_ai_personality.md)
|
||||||
|
- **Developer Guide:** Siehe [Developer Guide](../05_Development/05_developer_guide.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Letzte Aktualisierung:** 2025-01-XX
|
||||||
|
**Version:** 2.9.1
|
||||||
|
|
||||||
Loaded 100 of 203 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user