Compare commits
441
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b29ce4a0a6 | ||
|
|
311e598b69 | ||
|
|
413dca770c | ||
|
|
9d9239b11e | ||
|
|
a571c41c61 | ||
|
|
ba1fab9dcf | ||
|
|
5dc9c13a71 | ||
|
|
70821bae5a | ||
|
|
486abf82dd | ||
|
|
19afe3af33 | ||
|
|
113042b7f4 | ||
|
|
f8eebd91cb | ||
|
|
00297fc354 | ||
|
|
5baac082f9 | ||
|
|
611b0aa54e | ||
|
|
110f0dbcf5 | ||
|
|
1688cf0a1e | ||
|
|
13b7c8858a | ||
|
|
b8b6f243d1 | ||
|
|
eaa6e805ce | ||
|
|
05dbb21c58 | ||
|
|
f1f4358dee | ||
|
|
3374c41f07 | ||
|
|
95affe87e5 | ||
|
|
5e86398fcd | ||
|
|
844f0cbe42 | ||
|
|
4d5d7847e5 | ||
|
|
06795feed6 | ||
|
|
a4548a7ee1 | ||
|
|
42216865e2 | ||
|
|
e48bdb2401 | ||
|
|
c67eb4b2f3 | ||
|
|
fafa74b0b8 | ||
|
|
049a39aba3 | ||
|
|
cd26430a61 | ||
|
|
aab010ff17 | ||
|
|
9a7bc32051 | ||
|
|
bfdd8ac6be | ||
|
|
018669fb87 | ||
|
|
fb2de78d62 | ||
|
|
7e8d6c1954 | ||
|
|
d21230c6c3 | ||
|
|
4e98d5bfa1 | ||
|
|
8998269377 | ||
|
|
6e9a227ad2 | ||
|
|
4e7506da5a | ||
|
|
06ad93c77c | ||
|
|
0bb7b34259 | ||
|
|
859c17b49d | ||
|
|
46b26c9624 | ||
|
|
fd215c18e4 | ||
|
|
efa3afc7dc | ||
|
|
e03dd66051 | ||
|
|
3476fe5fae | ||
|
|
3e08c8347e | ||
|
|
c691123d2d | ||
|
|
a4b81535e8 | ||
|
|
a898cf4b65 | ||
|
|
10009a0a43 | ||
|
|
4dfe47ff99 | ||
|
|
bd997e61d6 | ||
|
|
23d0670126 | ||
|
|
12c600edbe | ||
|
|
e4115e705e | ||
|
|
0cbc4a2d72 | ||
|
|
cea6d35729 | ||
|
|
4c56918d8a | ||
|
|
f17263417e | ||
|
|
6f0b463489 | ||
|
|
28c01caa1a | ||
|
|
95b59e9b0a | ||
|
|
bc967f1f6e | ||
|
|
4228b8a74f | ||
|
|
300086fc83 | ||
|
|
688eb262a9 | ||
|
|
bfb6f5dede | ||
|
|
88c0fbe56b | ||
|
|
3d74eff224 | ||
|
|
22d08afe2d | ||
|
|
a97f757e34 | ||
|
|
bbc8f13944 | ||
|
|
f18a40d76c | ||
|
|
c7644a36aa | ||
|
|
f394f31c12 | ||
|
|
0513673c2a | ||
|
|
a73542a391 | ||
|
|
c465797654 | ||
|
|
7ce3f982f2 | ||
|
|
6311456917 | ||
|
|
3c06559c5d | ||
|
|
5c9f2e8c83 | ||
|
|
62902faaca | ||
|
|
d22fd77b22 | ||
|
|
2f86f493d6 | ||
|
|
eafa0fad85 | ||
|
|
2e7c497b69 | ||
|
|
79dcfdcf9a | ||
|
|
487e28a904 | ||
|
|
c492f97a67 | ||
|
|
ec2ff74514 | ||
|
|
c501f8d6e6 | ||
|
|
b4287cbfda | ||
|
|
26724a1db1 | ||
|
|
d6368ee709 | ||
|
|
ea211a5c0b | ||
|
|
9e8b433c95 | ||
|
|
2a1c62aeed | ||
|
|
948d6f4b47 | ||
|
|
725271a7da | ||
|
|
4ea62e6886 | ||
|
|
c01cf3b078 | ||
|
|
c67777a6b9 | ||
|
|
e5e5e7560c | ||
|
|
6e1cd5fbbc | ||
|
|
8bd8f7c0bb | ||
|
|
1c42cd1f78 | ||
|
|
be090a43f2 | ||
|
|
2c81fa0cb5 | ||
|
|
c3815afcd4 | ||
|
|
8800160564 | ||
|
|
bdbc4a1bf7 | ||
|
|
597090bc45 | ||
|
|
af36c410b4 | ||
|
|
4dcd606c10 | ||
|
|
bbd5a7fa48 | ||
|
|
6dc37ccb66 | ||
|
|
b84906283e | ||
|
|
af0dcdf76a | ||
|
|
f1e1cde597 | ||
|
|
a7c5630e5b | ||
|
|
82151d6bc3 | ||
|
|
c93e7ad598 | ||
|
|
290b271cf6 | ||
|
|
a686bdbeaf | ||
|
|
2b84c62875 | ||
|
|
2de786fc64 | ||
|
|
2ddf034983 | ||
|
|
b2043f4f84 | ||
|
|
52eae52061 | ||
|
|
e451ea64ae | ||
|
|
eb6e4028ff | ||
|
|
ea45493941 | ||
|
|
da70f0e00c | ||
|
|
8bf3707511 | ||
|
|
804eeef858 | ||
|
|
36658f12fe | ||
|
|
f0e675ffd8 | ||
|
|
90911c1074 | ||
|
|
1729e0b40d | ||
|
|
70b3169e7f | ||
|
|
315a1c0945 | ||
|
|
b9316c693d | ||
|
|
820e52d869 | ||
|
|
444e8e4206 | ||
|
|
e6f2c8b4e8 | ||
|
|
f6b7652219 | ||
|
|
3cd00ab5cf | ||
|
|
0feae50d90 | ||
|
|
d79c55fc64 | ||
|
|
98a6f5c480 | ||
|
|
0fa4aa7280 | ||
|
|
3b192e2eed | ||
|
|
2794d26181 | ||
|
|
7b56f696d6 | ||
|
|
ead1b8c1bc | ||
|
|
b8126e00c4 | ||
|
|
62f8f4b313 | ||
|
|
21924541dc | ||
|
|
197242ca42 | ||
|
|
7922734dce | ||
|
|
afd7675425 | ||
|
|
8b3b343645 | ||
|
|
6713f342c3 | ||
|
|
79bb8a143c | ||
|
|
d03a181536 | ||
|
|
8e5304c156 | ||
|
|
c487db6534 | ||
|
|
0ca54f429e | ||
|
|
6c62f64784 | ||
|
|
3282f85007 | ||
|
|
b186569750 | ||
|
|
7cde2462ec | ||
|
|
c0b96f7c05 | ||
|
|
3a88caac69 | ||
|
|
31457e9240 | ||
|
|
307335a733 | ||
|
|
4eb5e34ea7 | ||
|
|
f4be219790 | ||
|
|
53591b6f27 | ||
|
|
a14d0bb7cb | ||
|
|
c2802e7cb3 | ||
|
|
c7f2815f1d | ||
|
|
afc90fe992 | ||
|
|
2f9ce824a0 | ||
|
|
a39b2a6950 | ||
|
|
70fa95966c | ||
|
|
10479014d5 | ||
|
|
148e22aa7d | ||
|
|
a9fe6196f7 | ||
|
|
ae20665a66 | ||
|
|
ccbf078463 | ||
|
|
cd1e46784f | ||
|
|
b49d97b8d7 | ||
|
|
3d44de9d87 | ||
|
|
987b3c1770 | ||
|
|
25bc6544c4 | ||
|
|
b241ae753b | ||
|
|
2041771b14 | ||
|
|
b5958a9f63 | ||
|
|
c05dbd4b3b | ||
|
|
44e468fc21 | ||
|
|
5f358d329b | ||
|
|
59fe8154f2 | ||
|
|
d5b2e9bb8f | ||
|
|
3a3907e1b6 | ||
|
|
f66cdc70b2 | ||
|
|
e299a497a7 | ||
|
|
eefb29d2ad | ||
|
|
668c485037 | ||
|
|
c83d5bb970 | ||
|
|
8638b3c95b | ||
|
|
c9b55c16f7 | ||
|
|
a3b9d53d06 | ||
|
|
215915cee2 | ||
|
|
df9271ed81 | ||
|
|
97e73a0fa0 | ||
|
|
e2ebe98e7f | ||
|
|
345e30de71 | ||
|
|
0084160e12 | ||
|
|
7280c111f0 | ||
|
|
763de11d2b | ||
|
|
9c0de3eeb2 | ||
|
|
9c81e9fbf1 | ||
|
|
40603f1652 | ||
|
|
4a34da31ad | ||
|
|
6e394faab1 | ||
|
|
25d0895cac | ||
|
|
f5f93cfb12 | ||
|
|
cf1ebe9d6f | ||
|
|
dd0d4ff8b1 | ||
|
|
98103f3270 | ||
|
|
a740ebb42e | ||
|
|
7c79f820cb | ||
|
|
f3b2326304 | ||
|
|
4eded247c5 | ||
|
|
3ffb857181 | ||
|
|
ec92906399 | ||
|
|
87671640ea | ||
|
|
c725e984bb | ||
|
|
8f05b57744 | ||
|
|
120b0147f2 | ||
|
|
ae695b4b15 | ||
|
|
89fee685c3 | ||
|
|
23eb9a21c8 | ||
|
|
a009efe91c | ||
|
|
dbd3111adf | ||
|
|
2b35e0c07a | ||
|
|
ba3354df03 | ||
|
|
7c8c19f416 | ||
|
|
35f83241cb | ||
|
|
5297f9b47e | ||
|
|
cbc50ec4eb | ||
|
|
198cc291e2 | ||
|
|
cb8c3a6ea6 | ||
|
|
66f597bbba | ||
|
|
fe237c8b4b | ||
|
|
ec7b08a625 | ||
|
|
9dd728579d | ||
|
|
0a6ca00f64 | ||
|
|
6e856ce05f | ||
|
|
1b8b78cfe9 | ||
|
|
49508d9f87 | ||
|
|
63a660223e | ||
|
|
2cbc2e2f43 | ||
|
|
eb2ad44755 | ||
|
|
c3979142a6 | ||
|
|
1a30de6bfc | ||
|
|
8c0df47966 | ||
|
|
4c177378ae | ||
|
|
a2db175819 | ||
|
|
2d0f050524 | ||
|
|
2d2b78edb3 | ||
|
|
627fa80f3e | ||
|
|
96cbbb093e | ||
|
|
6551308f14 | ||
|
|
9ccc2badd4 | ||
|
|
07a83bdeb7 | ||
|
|
224dfb6588 | ||
|
|
d8d6c68ac2 | ||
|
|
3e369e5464 | ||
|
|
abaaba97f9 | ||
|
|
bd49467d26 | ||
|
|
b44524e1a9 | ||
|
|
0319667f8e | ||
|
|
4c3ad1f83c | ||
|
|
a7a845187a | ||
|
|
e14a9a11d2 | ||
|
|
d7ce757d65 | ||
|
|
6f5447c316 | ||
|
|
727b67811c | ||
|
|
a85fb68015 | ||
|
|
2dfe7328cc | ||
|
|
cfc0a0ca8f | ||
|
|
2b1c7fe20d | ||
|
|
d5e1d9c09b | ||
|
|
554540451e | ||
|
|
c10134c22b | ||
|
|
9175b3a045 | ||
|
|
0ecae44888 | ||
|
|
a9c2a10fb5 | ||
|
|
e11094e5fa | ||
|
|
38913dc3e1 | ||
|
|
dd2b7932a2 | ||
|
|
2051e187ba | ||
|
|
ca817a8cc4 | ||
|
|
2d815f5902 | ||
|
|
0abdc647c9 | ||
|
|
fded2db73e | ||
|
|
2274c9faf5 | ||
|
|
280b236490 | ||
|
|
ef2c144cdd | ||
|
|
634cf0ff1b | ||
|
|
2dc3d5c169 | ||
|
|
fcad1fdfaa | ||
|
|
5c6cdfb5e1 | ||
|
|
1d6ba761a8 | ||
|
|
893ba6c8f4 | ||
|
|
a835eff14b | ||
|
|
029b4d8a6e | ||
|
|
0053b45691 | ||
|
|
f46812d1d2 | ||
|
|
8495df961b | ||
|
|
abc8be1077 | ||
|
|
84baf27f7e | ||
|
|
1a08d904f9 | ||
|
|
31ca9a95d7 | ||
|
|
ade91b26c3 | ||
|
|
d797957abc | ||
|
|
104ddddb96 | ||
|
|
c8c4c7fa00 | ||
|
|
3111a26229 | ||
|
|
26ef257e06 | ||
|
|
6864ea7b53 | ||
|
|
c2ecc15d89 | ||
|
|
9bbcc0952b | ||
|
|
179ec4d38b | ||
|
|
02a61cdf0d | ||
|
|
6a84b4e5aa | ||
|
|
497a9b9d58 | ||
|
|
c5972be618 | ||
|
|
43e77c4c6b | ||
|
|
d66985e597 | ||
|
|
ff4bed99c9 | ||
|
|
2b4c1b83bd | ||
|
|
f85aa8f604 | ||
|
|
e5ca3c4177 | ||
|
|
0f2f8208e3 | ||
|
|
ba17516814 | ||
|
|
c0979b69e2 | ||
|
|
4b5c33bb90 | ||
|
|
1ea4d53061 | ||
|
|
17b548e9b0 | ||
|
|
5755eb1f08 | ||
|
|
e2f15324d7 | ||
|
|
b0b0802a56 | ||
|
|
e14cc8b758 | ||
|
|
55c6a5058c | ||
|
|
52d5d01337 | ||
|
|
93ceff7f17 | ||
|
|
0a891cfd95 | ||
|
|
66adc6a354 | ||
|
|
e99f1c2ba0 | ||
|
|
f52a9face8 | ||
|
|
4fcf22131c | ||
|
|
d27324f8dd | ||
|
|
dd86a15555 | ||
|
|
28d173b8fd | ||
|
|
b4b7ea76ab | ||
|
|
6ea452cc3f | ||
|
|
6e0e965883 | ||
|
|
6b4b3ce2ba | ||
|
|
18cdbab885 | ||
|
|
ebca9b4caf | ||
|
|
6945ef8ddc | ||
|
|
f20bf03ee1 | ||
|
|
7cda15553d | ||
|
|
3c67fd5f9b | ||
|
|
0150931df4 | ||
|
|
5ff1b06009 | ||
|
|
5279bcae18 | ||
|
|
daa58e6b27 | ||
|
|
589d021744 | ||
|
|
e58b9c8ada | ||
|
|
507c28e006 | ||
|
|
2d74fe4057 | ||
|
|
e8312268d2 | ||
|
|
f4cd1e51c0 | ||
|
|
60e707e7df | ||
|
|
0d6eb4dac4 | ||
|
|
80d8f986bd | ||
|
|
9ee3bb9007 | ||
|
|
8a07010704 | ||
|
|
314fc8ac94 | ||
|
|
b46e5461a7 | ||
|
|
ada7c1797d | ||
|
|
54197995f5 | ||
|
|
13e510dd9c | ||
|
|
5c937fbffe | ||
|
|
58bce0ea19 | ||
|
|
3e888cc669 | ||
|
|
78446c74d3 | ||
|
|
51f08f193c | ||
|
|
4872374a6e | ||
|
|
81c1400ef4 | ||
|
|
298c1fb180 | ||
|
|
f8108bd724 | ||
|
|
65c40b287f | ||
|
|
13e50d4695 | ||
|
|
791cda23af | ||
|
|
466ea64e63 | ||
|
|
f5e6fcc097 | ||
|
|
9ef2e8d397 | ||
|
|
4637091125 | ||
|
|
43a8c80508 | ||
|
|
9362064ad1 | ||
|
|
23952aa3c5 | ||
|
|
79e19c85eb | ||
|
|
62d8299480 | ||
|
|
8f342ea2d3 | ||
|
|
5d5de06290 | ||
|
|
d7e5f398f0 | ||
|
|
5bf5316af5 | ||
|
|
d9fd6d8a8f | ||
|
|
e672eade8b | ||
|
|
ec346905ed | ||
|
|
c9cbbbfcb9 | ||
|
|
93739935b7 | ||
|
|
6a293e842e | ||
|
|
f4abb1d873 | ||
|
|
8f48701ea0 | ||
|
|
6f32c60c47 |
No files matched your search
+31
-12
@@ -1,22 +1,36 @@
|
|||||||
# .gitea/workflows/deploy.yml
|
# .gitea/workflows/deploy.yml
|
||||||
name: Deploy mindnet to llm-node
|
name: Deploy mindnet to llm-node
|
||||||
on: { push: { branches: [ "main" ] } }
|
|
||||||
# on: {}
|
|
||||||
|
|
||||||
concurrency: { group: deploy-mindnet, cancel-in-progress: false }
|
on:
|
||||||
|
push:
|
||||||
|
branches: ["main"]
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: deploy-mindnet
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
deploy:
|
deploy:
|
||||||
runs-on: linux_host
|
runs-on: linux_host
|
||||||
env:
|
env:
|
||||||
DEPLOY_DIRS: "app scripts schemas docker requirements.txt README.md"
|
TARGET_BASE: /home/llmadmin/mindnet
|
||||||
TARGET_BASE: "/home/llmadmin/mindnet"
|
DEPLOY_DIRS: "app scripts schemas docker tests config requirements.txt README.md"
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: https://github.com/actions/checkout@v4
|
uses: actions/checkout@v4
|
||||||
- name: Ensure target base
|
|
||||||
|
- name: Ensure target base exists
|
||||||
run: install -d "$TARGET_BASE"
|
run: install -d "$TARGET_BASE"
|
||||||
- name: Deploy whitelisted
|
|
||||||
|
- name: Stop API (graceful)
|
||||||
|
continue-on-error: true
|
||||||
|
run: |
|
||||||
|
if systemctl --user list-unit-files | grep -q '^mindnet-api.service'; then
|
||||||
|
systemctl --user stop mindnet-api.service || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
- name: Deploy whitelisted (rsync)
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
IFS=' ' read -r -a DIRS <<< "$DEPLOY_DIRS"
|
IFS=' ' read -r -a DIRS <<< "$DEPLOY_DIRS"
|
||||||
@@ -24,22 +38,27 @@ jobs:
|
|||||||
if [ -e "$d" ]; then
|
if [ -e "$d" ]; then
|
||||||
if [ -d "$d" ]; then
|
if [ -d "$d" ]; then
|
||||||
install -d "$TARGET_BASE/$d"
|
install -d "$TARGET_BASE/$d"
|
||||||
rsync -a --delete --exclude='.git' \
|
rsync -a --delete \
|
||||||
--exclude='.env' --exclude='.env.*' --exclude='**/.env*' \
|
--exclude='.git' \
|
||||||
|
--exclude='.env*' \
|
||||||
|
--exclude='.venv' \
|
||||||
|
--exclude='hf_cache' --exclude='hf_cache/**' \
|
||||||
"$d"/ "$TARGET_BASE/$d"/
|
"$d"/ "$TARGET_BASE/$d"/
|
||||||
else
|
else
|
||||||
rsync -a "$d" "$TARGET_BASE/$d"
|
rsync -a "$d" "$TARGET_BASE/$d"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
- name: Python venv (idempotent)
|
|
||||||
|
- name: Python venv & requirements
|
||||||
run: |
|
run: |
|
||||||
cd "$TARGET_BASE"
|
cd "$TARGET_BASE"
|
||||||
[ -d .venv ] || python3 -m venv .venv
|
[ -d .venv ] || python3 -m venv .venv
|
||||||
source .venv/bin/activate
|
source .venv/bin/activate
|
||||||
pip install --upgrade pip
|
pip install --upgrade pip
|
||||||
[ -f requirements.txt ] && pip install -r requirements.txt || true
|
[ -f requirements.txt ] && pip install -r requirements.txt || true
|
||||||
- name: Optional — restart mindnet-api
|
|
||||||
|
- name: Start/Restart mindnet-api
|
||||||
continue-on-error: true
|
continue-on-error: true
|
||||||
run: |
|
run: |
|
||||||
if systemctl --user list-unit-files | grep -q '^mindnet-api.service'; then
|
if systemctl --user list-unit-files | grep -q '^mindnet-api.service'; then
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
"""
|
||||||
|
app — mindnet API package
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Markiert 'app/' als Python-Paket, damit 'from app.main import create_app'
|
||||||
|
in Tests und Skripten funktioniert.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Hinweise:
|
||||||
|
Keine Logik – nur Paketinitialisierung.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
+16
-1
@@ -1,6 +1,14 @@
|
|||||||
"""
|
"""
|
||||||
Version 0.1
|
app/config.py — zentrale Konfiguration (ENV → Settings)
|
||||||
|
|
||||||
|
Version:
|
||||||
|
0.2.0 (WP-04: Retriever-Gewichte & Defaults ergänzt; keine Verhaltensänderung für bestehende Nutzung)
|
||||||
|
Stand:
|
||||||
|
2025-10-06
|
||||||
|
Hinweis:
|
||||||
|
Bestehende Attribute bleiben erhalten; neue WP-04 Felder sind optional.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import os
|
import os
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
@@ -16,6 +24,13 @@ class Settings:
|
|||||||
MODEL_NAME: str = os.getenv("MINDNET_MODEL", "sentence-transformers/all-MiniLM-L6-v2")
|
MODEL_NAME: str = os.getenv("MINDNET_MODEL", "sentence-transformers/all-MiniLM-L6-v2")
|
||||||
# API
|
# API
|
||||||
DEBUG: bool = os.getenv("DEBUG", "false").lower() == "true"
|
DEBUG: bool = os.getenv("DEBUG", "false").lower() == "true"
|
||||||
|
# WP-04 Retriever Defaults (optional; können per ENV überschrieben werden)
|
||||||
|
RETRIEVER_W_SEM: float = float(os.getenv("MINDNET_WP04_W_SEM", "0.70"))
|
||||||
|
RETRIEVER_W_EDGE: float = float(os.getenv("MINDNET_WP04_W_EDGE", "0.25"))
|
||||||
|
RETRIEVER_W_CENT: float = float(os.getenv("MINDNET_WP04_W_CENT", "0.05"))
|
||||||
|
RETRIEVER_TOP_K: int = int(os.getenv("MINDNET_WP04_TOP_K", "10"))
|
||||||
|
RETRIEVER_EXPAND_DEPTH: int = int(os.getenv("MINDNET_WP04_EXPAND_DEPTH", "1"))
|
||||||
|
RETRIEVER_EDGE_TYPES: str = os.getenv("MINDNET_WP04_EDGE_TYPES", "references,belongs_to,prev,next")
|
||||||
|
|
||||||
@lru_cache
|
@lru_cache
|
||||||
def get_settings() -> Settings:
|
def get_settings() -> Settings:
|
||||||
|
|||||||
+113
-143
@@ -1,159 +1,129 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Modul: app/core/chunk_payload.py
|
app/core/chunk_payload.py (Mindnet V2 — types.yaml authoritative)
|
||||||
Version: 1.1.3
|
- neighbors_prev / neighbors_next sind Listen ([], [id]).
|
||||||
Datum: 2025-09-09
|
- retriever_weight / chunk_profile kommen aus types.yaml (Frontmatter wird ignoriert).
|
||||||
|
- Fallbacks: defaults.* in types.yaml; sonst 1.0 / "default".
|
||||||
Kurzbeschreibung
|
|
||||||
----------------
|
|
||||||
Erzeugt Qdrant-Payloads für Text-Chunks einer Note. Jeder Chunk enthält
|
|
||||||
den tatsächlichen Text unter dem Schlüssel ``text``. Dadurch kann der
|
|
||||||
Exporter den vollständigen Body verlässlich aus Chunks rekonstruieren,
|
|
||||||
falls ``notes.payload.fulltext`` fehlt.
|
|
||||||
|
|
||||||
Wesentliche Features
|
|
||||||
--------------------
|
|
||||||
- Stabile, idempotente Payload-Struktur für Chunks
|
|
||||||
- Persistenter Chunk-Text (``text``)
|
|
||||||
- Extraktion von Wikilinks pro Chunk (``wikilinks`` & ``references``)
|
|
||||||
- Pfadübernahme (relativ zum Vault, wird vom Aufrufer geliefert)
|
|
||||||
- Bereinigung leerer Felder (keine ``None``/leere Collections im Payload)
|
|
||||||
|
|
||||||
Abhängigkeiten
|
|
||||||
--------------
|
|
||||||
- ``app.core.chunker.Chunk`` und ``assemble_chunks`` (nur für CLI-Demo)
|
|
||||||
- ``app.core.parser.extract_wikilinks`` und ``read_markdown`` (nur CLI-Demo)
|
|
||||||
|
|
||||||
Beispiele (CLI – Sichtprüfung)
|
|
||||||
------------------------------
|
|
||||||
python3 -m app.core.chunk_payload --from-file ./vault/demo.md --print
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from typing import Dict, List, Any
|
from typing import Any, Dict, List, Optional
|
||||||
import argparse
|
import os, yaml
|
||||||
import json
|
|
||||||
import os
|
|
||||||
|
|
||||||
# WICHTIG: Keine direkten Selbst- oder Kreuz-Imports, die Zyklen erzeugen!
|
def _env(n: str, d: Optional[str]=None) -> str:
|
||||||
# Für die Kernfunktion reicht extract_wikilinks als Typ-unabhängige Hilfsfunktion.
|
v = os.getenv(n)
|
||||||
try:
|
return v if v is not None else (d or "")
|
||||||
from app.core.parser import extract_wikilinks # nur Funktion, erzeugt keine Zyklen
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
from .parser import extract_wikilinks # type: ignore
|
|
||||||
|
|
||||||
# Typ-Hinweis für Chunk (zur Laufzeit nicht notwendig)
|
def _load_types() -> dict:
|
||||||
try:
|
p = _env("MINDNET_TYPES_FILE", "./config/types.yaml")
|
||||||
from app.core.chunker import Chunk # nur Typ-Annotation / Nutzung im CLI
|
|
||||||
except Exception: # pragma: no cover
|
|
||||||
class Chunk: # type: ignore
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Utils
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _drop_empty(d: Dict[str, Any]) -> Dict[str, Any]:
|
|
||||||
"""Entfernt leere/None-Felder aus einem Dict (für saubere Payloads)."""
|
|
||||||
return {k: v for k, v in d.items() if v not in (None, [], {}, "")}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Kernfunktion
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def make_chunk_payloads(note_meta: Dict[str, Any], path: str, chunks: List[Chunk]) -> List[Dict[str, Any]]:
|
|
||||||
"""
|
|
||||||
Baut Payloads für alle ``chunks`` der gegebenen Note.
|
|
||||||
|
|
||||||
Parameters
|
|
||||||
----------
|
|
||||||
note_meta : Dict[str, Any]
|
|
||||||
Minimale Metadaten der Note (mind. ``id``, ``title``; optional ``type``,
|
|
||||||
``area``, ``project``, ``tags``, ``lang``).
|
|
||||||
path : str
|
|
||||||
Relativer Pfad der Note innerhalb des Vaults (z. B. "area/topic/file.md").
|
|
||||||
chunks : List[Chunk]
|
|
||||||
Liste vorsegmentierter Chunks (vgl. app.core.chunker.Chunk).
|
|
||||||
|
|
||||||
Returns
|
|
||||||
-------
|
|
||||||
List[Dict[str, Any]]
|
|
||||||
Payload-Objekte, bereit für Qdrant-Upserts.
|
|
||||||
"""
|
|
||||||
res: List[Dict[str, Any]] = []
|
|
||||||
for ch in chunks:
|
|
||||||
# 'text' kann bei leerem Abschnitt None/'' sein → robust behandeln
|
|
||||||
text_val = getattr(ch, "text", "") or ""
|
|
||||||
wikilinks = extract_wikilinks(text_val)
|
|
||||||
payload = {
|
|
||||||
"id": getattr(ch, "id", None),
|
|
||||||
"note_id": note_meta.get("id"),
|
|
||||||
"note_title": note_meta.get("title"),
|
|
||||||
"chunk_index": getattr(ch, "index", None),
|
|
||||||
"char_start": getattr(ch, "char_start", None),
|
|
||||||
"char_end": getattr(ch, "char_end", None),
|
|
||||||
"token_count": getattr(ch, "token_count", None),
|
|
||||||
"section_title": getattr(ch, "section_title", None),
|
|
||||||
"section_path": getattr(ch, "section_path", None),
|
|
||||||
"lang": note_meta.get("lang"),
|
|
||||||
"wikilinks": wikilinks,
|
|
||||||
"external_links": [],
|
|
||||||
"references": [{"target_id": w, "kind": "wikilink"} for w in wikilinks],
|
|
||||||
"neighbors": {
|
|
||||||
"prev": getattr(ch, "neighbors_prev", None),
|
|
||||||
"next": getattr(ch, "neighbors_next", None),
|
|
||||||
},
|
|
||||||
"path": path, # vom Aufrufer relativ geliefert
|
|
||||||
"text": text_val, # WICHTIG für Export/Rekonstruktion
|
|
||||||
}
|
|
||||||
res.append(_drop_empty(payload))
|
|
||||||
return res
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# CLI – nur zur Sichtprüfung / Debug
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _cli() -> None:
|
|
||||||
# CLI lädt hier *lazy* die schweren Abhängigkeiten,
|
|
||||||
# damit die Modul-Initialisierung zirkularfrei bleibt.
|
|
||||||
try:
|
try:
|
||||||
from app.core.parser import read_markdown
|
with open(p, "r", encoding="utf-8") as f:
|
||||||
from app.core.chunker import assemble_chunks
|
return yaml.safe_load(f) or {}
|
||||||
except Exception: # pragma: no cover
|
except Exception:
|
||||||
from .parser import read_markdown # type: ignore
|
return {}
|
||||||
from .chunker import assemble_chunks # type: ignore
|
|
||||||
|
|
||||||
ap = argparse.ArgumentParser(description="Vorschau: Chunk-Payloads erzeugen und anzeigen")
|
def _get_types_map(reg: dict) -> dict:
|
||||||
ap.add_argument("--from-file", dest="src", required=True, help="Pfad zu einer Markdown-Datei")
|
if isinstance(reg, dict) and isinstance(reg.get("types"), dict):
|
||||||
ap.add_argument("--print", dest="do_print", action="store_true", help="Payload auf stdout ausgeben")
|
return reg["types"]
|
||||||
args = ap.parse_args()
|
return reg if isinstance(reg, dict) else {}
|
||||||
|
|
||||||
note = read_markdown(args.src)
|
def _get_defaults(reg: dict) -> dict:
|
||||||
note_meta = {
|
if isinstance(reg, dict) and isinstance(reg.get("defaults"), dict):
|
||||||
"id": note.frontmatter.get("id") or note.frontmatter.get("note_id"),
|
return reg["defaults"]
|
||||||
"title": note.frontmatter.get("title"),
|
if isinstance(reg, dict) and isinstance(reg.get("global"), dict):
|
||||||
"type": note.frontmatter.get("type"),
|
return reg["global"]
|
||||||
"area": note.frontmatter.get("area"),
|
return {}
|
||||||
"project": note.frontmatter.get("project"),
|
|
||||||
"tags": note.frontmatter.get("tags", []),
|
|
||||||
"lang": note.frontmatter.get("lang"),
|
|
||||||
}
|
|
||||||
|
|
||||||
chunks = assemble_chunks(note.frontmatter.get("id"), note.body or "", note.frontmatter.get("type", "concept"))
|
def _as_float(x: Any):
|
||||||
|
try:
|
||||||
|
return float(x)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
# Vault-Root heuristisch relativieren (nur für Demo)
|
def _resolve_chunk_profile(note_type: str, reg: dict) -> str:
|
||||||
vault_root = os.path.dirname(os.path.dirname(args.src)) # heuristisch
|
types = _get_types_map(reg)
|
||||||
rel_path = os.path.relpath(args.src, vault_root).replace("\\", "/").lstrip("/")
|
if isinstance(types, dict):
|
||||||
|
t = types.get(note_type, {})
|
||||||
|
if isinstance(t, dict) and isinstance(t.get("chunk_profile"), str):
|
||||||
|
return t["chunk_profile"]
|
||||||
|
defs = _get_defaults(reg)
|
||||||
|
if isinstance(defs, dict) and isinstance(defs.get("chunk_profile"), str):
|
||||||
|
return defs["chunk_profile"]
|
||||||
|
return "default"
|
||||||
|
|
||||||
payloads = make_chunk_payloads(note_meta, rel_path, chunks)
|
def _resolve_retriever_weight(note_type: str, reg: dict) -> float:
|
||||||
|
types = _get_types_map(reg)
|
||||||
|
if isinstance(types, dict):
|
||||||
|
t = types.get(note_type, {})
|
||||||
|
if isinstance(t, dict) and (t.get("retriever_weight") is not None):
|
||||||
|
v = _as_float(t.get("retriever_weight"))
|
||||||
|
if v is not None:
|
||||||
|
return float(v)
|
||||||
|
defs = _get_defaults(reg)
|
||||||
|
if isinstance(defs, dict) and (defs.get("retriever_weight") is not None):
|
||||||
|
v = _as_float(defs.get("retriever_weight"))
|
||||||
|
if v is not None:
|
||||||
|
return float(v)
|
||||||
|
return 1.0
|
||||||
|
|
||||||
if args.do_print:
|
def _as_list(x):
|
||||||
print(json.dumps(payloads, ensure_ascii=False, indent=2))
|
if x is None:
|
||||||
|
return []
|
||||||
|
if isinstance(x, list):
|
||||||
|
return x
|
||||||
|
return [x]
|
||||||
|
|
||||||
|
def make_chunk_payloads(note: Dict[str, Any],
|
||||||
|
note_path: str,
|
||||||
|
chunks_from_chunker: List[Any],
|
||||||
|
*,
|
||||||
|
note_text: str = "",
|
||||||
|
types_cfg: Optional[dict] = None,
|
||||||
|
file_path: Optional[str] = None) -> List[Dict[str, Any]]:
|
||||||
|
fm = (note or {}).get("frontmatter", {}) or {}
|
||||||
|
note_type = fm.get("type") or note.get("type") or "concept"
|
||||||
|
reg = types_cfg if isinstance(types_cfg, dict) else _load_types()
|
||||||
|
|
||||||
if __name__ == "__main__": # pragma: no cover
|
# types.yaml authoritative
|
||||||
_cli()
|
cp = _resolve_chunk_profile(note_type, reg)
|
||||||
|
rw = _resolve_retriever_weight(note_type, reg)
|
||||||
|
|
||||||
|
tags = fm.get("tags") or []
|
||||||
|
if isinstance(tags, str):
|
||||||
|
tags = [tags]
|
||||||
|
|
||||||
|
out: List[Dict[str, Any]] = []
|
||||||
|
for idx, ch in enumerate(chunks_from_chunker):
|
||||||
|
# Attribute oder Keys (Chunk-Objekt oder Dict)
|
||||||
|
cid = getattr(ch, "id", None) or (ch.get("id") if isinstance(ch, dict) else None)
|
||||||
|
nid = getattr(ch, "note_id", None) or (ch.get("note_id") if isinstance(ch, dict) else fm.get("id"))
|
||||||
|
index = getattr(ch, "index", None) or (ch.get("index") if isinstance(ch, dict) else idx)
|
||||||
|
text = getattr(ch, "text", None) or (ch.get("text") if isinstance(ch, dict) else "")
|
||||||
|
window = getattr(ch, "window", None) or (ch.get("window") if isinstance(ch, dict) else text)
|
||||||
|
prev_id = getattr(ch, "neighbors_prev", None) or (ch.get("neighbors_prev") if isinstance(ch, dict) else None)
|
||||||
|
next_id = getattr(ch, "neighbors_next", None) or (ch.get("neighbors_next") if isinstance(ch, dict) else None)
|
||||||
|
|
||||||
|
pl: Dict[str, Any] = {
|
||||||
|
"note_id": nid,
|
||||||
|
"chunk_id": cid,
|
||||||
|
"index": int(index),
|
||||||
|
"ord": int(index) + 1,
|
||||||
|
"type": note_type,
|
||||||
|
"tags": tags,
|
||||||
|
"text": text,
|
||||||
|
"window": window,
|
||||||
|
"neighbors_prev": _as_list(prev_id),
|
||||||
|
"neighbors_next": _as_list(next_id),
|
||||||
|
"section": getattr(ch, "section", None) or (ch.get("section") if isinstance(ch, dict) else ""),
|
||||||
|
"path": note_path,
|
||||||
|
"source_path": file_path or note_path,
|
||||||
|
"retriever_weight": float(rw),
|
||||||
|
"chunk_profile": cp,
|
||||||
|
}
|
||||||
|
# Aufräumen von Alt-Feldern
|
||||||
|
for alias in ("chunk_num", "Chunk_Number"):
|
||||||
|
pl.pop(alias, None)
|
||||||
|
|
||||||
|
out.append(pl)
|
||||||
|
|
||||||
|
return out
|
||||||
+418
-109
@@ -1,126 +1,435 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Name: app/core/derive_edges.py
|
Modul: app/core/derive_edges.py
|
||||||
Version: v1.1.0 (2025-09-05)
|
Zweck:
|
||||||
|
- Bewahrt bestehende Edgelogik (belongs_to, prev/next, references, backlink)
|
||||||
Kurzbeschreibung
|
- Ergänzt typenbasierte Default-Kanten (edge_defaults aus config/types.yaml)
|
||||||
Leitet Edges aus Wikilinks ([[…]]) ab und löst Zielnoten robust auf.
|
- Unterstützt "typed inline relations":
|
||||||
Erzeugt:
|
* [[rel:KIND | Target]]
|
||||||
- "references" (Note -> Note) mit seq="body", pro Match eine eigene Occurrence 'occ'
|
* [[rel:KIND Target]]
|
||||||
- "backlink" (inverse zu "references", gleiche seq/occ)
|
* rel: KIND [[Target]]
|
||||||
- "references_at" (Chunk -> Note) mit seq=<chunk_index> und eigener 'occ' je Match
|
- Unterstützt Obsidian-Callouts:
|
||||||
|
* > [!edge] KIND: [[Target]] [[Target2]] ...
|
||||||
Aufruf
|
Kompatibilität:
|
||||||
from app.core.derive_edges import build_note_index, derive_wikilink_edges
|
- build_edges_for_note(...) Signatur unverändert
|
||||||
|
- rule_id Werte:
|
||||||
Parameter / Felder
|
* structure:belongs_to
|
||||||
- note_payload: {"note_id","title","path","fulltext": <body> , …}
|
* structure:order
|
||||||
- chunks_payloads: [{"chunk_id","text",…}, …]
|
* explicit:wikilink
|
||||||
- note_index: build_note_index([...]) -> (by_id, by_slug, by_file_slug)
|
* inline:rel
|
||||||
|
* callout:edge
|
||||||
Kompatibilität
|
* edge_defaults:<type>:<relation>
|
||||||
- Rückwärtskompatible Payload-Felder, nur erweitert um 'seq' und 'occ'.
|
* derived:backlink
|
||||||
|
|
||||||
Changelog
|
|
||||||
v1.1.0: Occurrence-Zählung ('occ') je Match; 'seq="body"' für references.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
import re
|
import re
|
||||||
import unicodedata
|
from typing import Iterable, List, Optional, Tuple, Set, Dict
|
||||||
from typing import Dict, List, Tuple
|
|
||||||
|
|
||||||
# [[Ziel]], [[Ziel|Alias]], [[Ziel#Heading]], [[Ziel#Heading|Alias]]
|
try:
|
||||||
WIKILINK_RE = re.compile(r"\[\[([^\]|#]+)(?:#([^\]|]+))?(?:\|([^\]]+))?\]\]")
|
import yaml # optional, nur für types.yaml
|
||||||
|
except Exception: # pragma: no cover
|
||||||
|
yaml = None
|
||||||
|
|
||||||
def _slug(s: str) -> str:
|
# --------------------------------------------------------------------------- #
|
||||||
s = s.strip()
|
# Utilities
|
||||||
if s.endswith(".md"):
|
# --------------------------------------------------------------------------- #
|
||||||
s = s[:-3]
|
|
||||||
s = unicodedata.normalize("NFKD", s)
|
|
||||||
s = "".join(ch for ch in s if not unicodedata.combining(ch))
|
|
||||||
s = s.replace("\\", "/")
|
|
||||||
s = s.split("/")[-1]
|
|
||||||
s = s.lower().replace(" ", "-")
|
|
||||||
s = re.sub(r"[^a-z0-9\-]+", "", s)
|
|
||||||
s = re.sub(r"-{2,}", "-", s).strip("-")
|
|
||||||
return s
|
|
||||||
|
|
||||||
def build_note_index(notes_payloads: List[dict]) -> Tuple[Dict[str, dict], Dict[str, dict], Dict[str, dict]]:
|
def _get(d: dict, *keys, default=None):
|
||||||
by_id: Dict[str, dict] = {}
|
for k in keys:
|
||||||
by_slug: Dict[str, dict] = {}
|
if isinstance(d, dict) and k in d and d[k] is not None:
|
||||||
by_file_slug: Dict[str, dict] = {}
|
return d[k]
|
||||||
for n in notes_payloads:
|
return default
|
||||||
nid = n.get("note_id") or n.get("id")
|
|
||||||
if not nid:
|
def _chunk_text_for_refs(chunk: dict) -> str:
|
||||||
|
# bevorzugt 'window' → dann 'text' → 'content' → 'raw'
|
||||||
|
return (
|
||||||
|
_get(chunk, "window")
|
||||||
|
or _get(chunk, "text")
|
||||||
|
or _get(chunk, "content")
|
||||||
|
or _get(chunk, "raw")
|
||||||
|
or ""
|
||||||
|
)
|
||||||
|
|
||||||
|
def _dedupe_seq(seq: Iterable[str]) -> List[str]:
|
||||||
|
seen: Set[str] = set()
|
||||||
|
out: List[str] = []
|
||||||
|
for s in seq:
|
||||||
|
if s not in seen:
|
||||||
|
seen.add(s)
|
||||||
|
out.append(s)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _edge(kind: str, scope: str, source_id: str, target_id: str, note_id: str, extra: Optional[dict] = None) -> dict:
|
||||||
|
pl = {
|
||||||
|
"kind": kind,
|
||||||
|
"relation": kind, # Alias (v2)
|
||||||
|
"scope": scope, # "chunk" | "note"
|
||||||
|
"source_id": source_id,
|
||||||
|
"target_id": target_id,
|
||||||
|
"note_id": note_id, # Träger-Note der Kante
|
||||||
|
}
|
||||||
|
if extra:
|
||||||
|
pl.update(extra)
|
||||||
|
return pl
|
||||||
|
|
||||||
|
def _mk_edge_id(kind: str, s: str, t: str, scope: str, rule_id: Optional[str] = None) -> str:
|
||||||
|
base = f"{kind}:{s}->{t}#{scope}"
|
||||||
|
if rule_id:
|
||||||
|
base += f"|{rule_id}"
|
||||||
|
try:
|
||||||
|
import hashlib
|
||||||
|
return hashlib.blake2s(base.encode("utf-8"), digest_size=12).hexdigest()
|
||||||
|
except Exception: # pragma: no cover
|
||||||
|
return base
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Typen-Registry (types.yaml)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def _env(n: str, default: Optional[str] = None) -> str:
|
||||||
|
v = os.getenv(n)
|
||||||
|
return v if v is not None else (default or "")
|
||||||
|
|
||||||
|
def _load_types_registry() -> dict:
|
||||||
|
"""Lädt die YAML-Registry aus MINDNET_TYPES_FILE oder ./config/types.yaml"""
|
||||||
|
p = _env("MINDNET_TYPES_FILE", "./config/types.yaml")
|
||||||
|
if not os.path.isfile(p) or yaml is None:
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
with open(p, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
return data
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def _get_types_map(reg: dict) -> dict:
|
||||||
|
if isinstance(reg, dict) and isinstance(reg.get("types"), dict):
|
||||||
|
return reg["types"]
|
||||||
|
return reg if isinstance(reg, dict) else {}
|
||||||
|
|
||||||
|
def _edge_defaults_for(note_type: Optional[str], reg: dict) -> List[str]:
|
||||||
|
"""
|
||||||
|
Liefert die edge_defaults-Liste für den gegebenen Notiztyp.
|
||||||
|
Fallback-Reihenfolge:
|
||||||
|
1) reg['types'][note_type]['edge_defaults']
|
||||||
|
2) reg['defaults']['edge_defaults'] (oder 'default'/'global')
|
||||||
|
3) []
|
||||||
|
"""
|
||||||
|
types_map = _get_types_map(reg)
|
||||||
|
if note_type and isinstance(types_map, dict):
|
||||||
|
t = types_map.get(note_type)
|
||||||
|
if isinstance(t, dict) and isinstance(t.get("edge_defaults"), list):
|
||||||
|
return [str(x) for x in t["edge_defaults"] if isinstance(x, str)]
|
||||||
|
for key in ("defaults", "default", "global"):
|
||||||
|
v = reg.get(key)
|
||||||
|
if isinstance(v, dict) and isinstance(v.get("edge_defaults"), list):
|
||||||
|
return [str(x) for x in v["edge_defaults"] if isinstance(x, str)]
|
||||||
|
return []
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Parser für Links / Relationen
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
# Normale Wikilinks (Fallback)
|
||||||
|
_WIKILINK_RE = re.compile(r"\[\[(?:[^\|\]]+\|)?([a-zA-Z0-9_\-#:. ]+)\]\]")
|
||||||
|
|
||||||
|
# Getypte Inline-Relationen:
|
||||||
|
# [[rel:KIND | Target]]
|
||||||
|
# [[rel:KIND Target]]
|
||||||
|
_REL_PIPE = re.compile(r"\[\[\s*rel:(?P<kind>[a-z_]+)\s*\|\s*(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
||||||
|
_REL_SPACE = re.compile(r"\[\[\s*rel:(?P<kind>[a-z_]+)\s+(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
||||||
|
# rel: KIND [[Target]] (reines Textmuster)
|
||||||
|
_REL_TEXT = re.compile(r"rel\s*:\s*(?P<kind>[a-z_]+)\s*\[\[\s*(?P<target>[^\]]+?)\s*\]\]", re.IGNORECASE)
|
||||||
|
|
||||||
|
def _extract_typed_relations(text: str) -> Tuple[List[Tuple[str,str]], str]:
|
||||||
|
"""
|
||||||
|
Gibt Liste (kind, target) zurück und den Text mit entfernten getypten Relation-Links,
|
||||||
|
damit die generische Wikilink-Erkennung sie nicht doppelt zählt.
|
||||||
|
Unterstützt drei Varianten:
|
||||||
|
- [[rel:KIND | Target]]
|
||||||
|
- [[rel:KIND Target]]
|
||||||
|
- rel: KIND [[Target]]
|
||||||
|
"""
|
||||||
|
pairs: List[Tuple[str,str]] = []
|
||||||
|
def _collect(m):
|
||||||
|
k = (m.group("kind") or "").strip().lower()
|
||||||
|
t = (m.group("target") or "").strip()
|
||||||
|
if k and t:
|
||||||
|
pairs.append((k, t))
|
||||||
|
return "" # Link entfernen
|
||||||
|
|
||||||
|
text = _REL_PIPE.sub(_collect, text)
|
||||||
|
text = _REL_SPACE.sub(_collect, text)
|
||||||
|
text = _REL_TEXT.sub(_collect, text)
|
||||||
|
return pairs, text
|
||||||
|
|
||||||
|
# Obsidian Callout Parser
|
||||||
|
_CALLOUT_START = re.compile(r"^\s*>\s*\[!edge\]\s*(.*)$", re.IGNORECASE)
|
||||||
|
_REL_LINE = re.compile(r"^(?P<kind>[a-z_]+)\s*:\s*(?P<targets>.+?)\s*$", re.IGNORECASE)
|
||||||
|
_WIKILINKS_IN_LINE = re.compile(r"\[\[([^\]]+)\]\]")
|
||||||
|
|
||||||
|
def _extract_callout_relations(text: str) -> Tuple[List[Tuple[str,str]], str]:
|
||||||
|
"""
|
||||||
|
Findet [!edge]-Callouts und extrahiert (kind, target). Entfernt den gesamten
|
||||||
|
Callout-Block aus dem Text (damit Wikilinks daraus nicht zusätzlich als
|
||||||
|
"references" gezählt werden).
|
||||||
|
"""
|
||||||
|
if not text:
|
||||||
|
return [], text
|
||||||
|
|
||||||
|
lines = text.splitlines()
|
||||||
|
out_pairs: List[Tuple[str,str]] = []
|
||||||
|
keep_lines: List[str] = []
|
||||||
|
i = 0
|
||||||
|
|
||||||
|
while i < len(lines):
|
||||||
|
m = _CALLOUT_START.match(lines[i])
|
||||||
|
if not m:
|
||||||
|
keep_lines.append(lines[i])
|
||||||
|
i += 1
|
||||||
continue
|
continue
|
||||||
by_id[nid] = n
|
|
||||||
title = n.get("title", "")
|
|
||||||
path = n.get("path", "")
|
|
||||||
file_slug = _slug(path.split("/")[-1]) if path else ""
|
|
||||||
if title:
|
|
||||||
by_slug[_slug(title)] = n
|
|
||||||
if file_slug:
|
|
||||||
by_file_slug[file_slug] = n
|
|
||||||
return by_id, by_slug, by_file_slug
|
|
||||||
|
|
||||||
def resolve_target(note_like: str, idx: Tuple[Dict[str,dict],Dict[str,dict],Dict[str,dict]]):
|
block_lines: List[str] = []
|
||||||
by_id, by_slug, by_file_slug = idx
|
first_rest = m.group(1) or ""
|
||||||
key = note_like.strip()
|
if first_rest.strip():
|
||||||
if key in by_id:
|
block_lines.append(first_rest)
|
||||||
return by_id[key]["note_id"], "by_id"
|
|
||||||
s = _slug(key)
|
|
||||||
if s in by_slug:
|
|
||||||
return by_slug[s]["note_id"], "by_slug"
|
|
||||||
if s in by_file_slug:
|
|
||||||
return by_file_slug[s]["note_id"], "by_file_slug"
|
|
||||||
return None, "unresolved"
|
|
||||||
|
|
||||||
def derive_wikilink_edges(note_payload: dict, chunks_payloads: List[dict], note_index) -> List[dict]:
|
i += 1
|
||||||
|
while i < len(lines) and lines[i].lstrip().startswith('>'):
|
||||||
|
block_lines.append(lines[i].lstrip()[1:].lstrip())
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
for bl in block_lines:
|
||||||
|
mrel = _REL_LINE.match(bl)
|
||||||
|
if not mrel:
|
||||||
|
continue
|
||||||
|
kind = (mrel.group("kind") or "").strip().lower()
|
||||||
|
targets = mrel.group("targets") or ""
|
||||||
|
found = _WIKILINKS_IN_LINE.findall(targets)
|
||||||
|
if found:
|
||||||
|
for t in found:
|
||||||
|
t = t.strip()
|
||||||
|
if t:
|
||||||
|
out_pairs.append((kind, t))
|
||||||
|
else:
|
||||||
|
for raw in re.split(r"[,;]", targets):
|
||||||
|
t = raw.strip()
|
||||||
|
if t:
|
||||||
|
out_pairs.append((kind, t))
|
||||||
|
|
||||||
|
# Callout wird NICHT in keep_lines übernommen
|
||||||
|
continue
|
||||||
|
|
||||||
|
remainder = "\n".join(keep_lines)
|
||||||
|
return out_pairs, remainder
|
||||||
|
|
||||||
|
def _extract_wikilinks(text: str) -> List[str]:
|
||||||
|
ids: List[str] = []
|
||||||
|
for m in _WIKILINK_RE.finditer(text or ""):
|
||||||
|
ids.append(m.group(1).strip())
|
||||||
|
return ids
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Hauptfunktion
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def build_edges_for_note(
|
||||||
|
note_id: str,
|
||||||
|
chunks: List[dict],
|
||||||
|
note_level_references: Optional[List[str]] = None,
|
||||||
|
include_note_scope_refs: bool = False,
|
||||||
|
) -> List[dict]:
|
||||||
|
"""
|
||||||
|
Erzeugt Kanten für eine Note.
|
||||||
|
|
||||||
|
- belongs_to: für jeden Chunk (chunk -> note)
|
||||||
|
- next / prev: zwischen aufeinanderfolgenden Chunks
|
||||||
|
- references: pro Chunk aus window/text (via Wikilinks)
|
||||||
|
- typed inline relations: [[rel:KIND | Target]] / [[rel:KIND Target]] / rel: KIND [[Target]]
|
||||||
|
- Obsidian Callouts: > [!edge] KIND: [[Target]] [[Target2]]
|
||||||
|
- optional note-scope references/backlinks: dedupliziert über alle Chunk-Funde + note_level_references
|
||||||
|
- typenbasierte Default-Kanten (edge_defaults) je gefundener Referenz
|
||||||
|
"""
|
||||||
edges: List[dict] = []
|
edges: List[dict] = []
|
||||||
source_note_id = note_payload["note_id"]
|
|
||||||
|
|
||||||
def _make_edge(kind: str, src: str, tgt: str, seq=None, occ=None, extra: dict|None=None):
|
# Note-Typ (aus erstem Chunk erwartet)
|
||||||
e = {"edge_id": None, "kind": kind, "source_id": src, "target_id": tgt}
|
note_type = None
|
||||||
if seq is not None:
|
if chunks:
|
||||||
e["seq"] = seq
|
note_type = _get(chunks[0], "type")
|
||||||
if occ is not None:
|
|
||||||
e["occ"] = occ
|
|
||||||
if extra:
|
|
||||||
e.update(extra)
|
|
||||||
return e
|
|
||||||
|
|
||||||
# Volltext (Note-Ebene)
|
# 1) belongs_to
|
||||||
fulltext = note_payload.get("fulltext") or note_payload.get("body") or ""
|
for ch in chunks:
|
||||||
if fulltext:
|
cid = _get(ch, "chunk_id", "id")
|
||||||
for k, m in enumerate(WIKILINK_RE.finditer(fulltext), start=1):
|
if not cid:
|
||||||
raw_target, heading, alias = m.groups()
|
|
||||||
target_id, how = resolve_target(raw_target, note_index)
|
|
||||||
extra = {"raw": raw_target, "alias": alias, "heading": heading, "resolution": how}
|
|
||||||
if target_id:
|
|
||||||
edges.append(_make_edge("references", source_note_id, target_id, seq="body", occ=k, extra=extra))
|
|
||||||
edges.append(_make_edge("backlink", target_id, source_note_id, seq="body", occ=k, extra=extra))
|
|
||||||
else:
|
|
||||||
extra["status"] = "unresolved"
|
|
||||||
extra["target_label"] = raw_target
|
|
||||||
edges.append(_make_edge("references", source_note_id, raw_target, seq="body", occ=k, extra=extra))
|
|
||||||
|
|
||||||
# Chunks (Chunk-Ebene)
|
|
||||||
for i, ch in enumerate(chunks_payloads, start=1):
|
|
||||||
txt = ch.get("text") or ch.get("content") or ""
|
|
||||||
if not txt:
|
|
||||||
continue
|
continue
|
||||||
occ = 0
|
edges.append(_edge("belongs_to", "chunk", cid, note_id, note_id, {
|
||||||
for m in WIKILINK_RE.finditer(txt):
|
"chunk_id": cid,
|
||||||
occ += 1
|
"edge_id": _mk_edge_id("belongs_to", cid, note_id, "chunk", "structure:belongs_to"),
|
||||||
raw_target, heading, alias = m.groups()
|
"provenance": "rule",
|
||||||
target_id, how = resolve_target(raw_target, note_index)
|
"rule_id": "structure:belongs_to",
|
||||||
extra = {"raw": raw_target, "alias": alias, "heading": heading, "resolution": how}
|
"confidence": 1.0,
|
||||||
if target_id:
|
}))
|
||||||
edges.append(_make_edge("references_at", ch["chunk_id"], target_id, seq=i, occ=occ, extra=extra))
|
|
||||||
else:
|
# 2) next / prev
|
||||||
extra["status"] = "unresolved"
|
for i in range(len(chunks) - 1):
|
||||||
extra["target_label"] = raw_target
|
a, b = chunks[i], chunks[i + 1]
|
||||||
edges.append(_make_edge("references_at", ch["chunk_id"], raw_target, seq=i, occ=occ, extra=extra))
|
a_id = _get(a, "chunk_id", "id")
|
||||||
return edges
|
b_id = _get(b, "chunk_id", "id")
|
||||||
|
if not a_id or not b_id:
|
||||||
|
continue
|
||||||
|
edges.append(_edge("next", "chunk", a_id, b_id, note_id, {
|
||||||
|
"chunk_id": a_id,
|
||||||
|
"edge_id": _mk_edge_id("next", a_id, b_id, "chunk", "structure:order"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": "structure:order",
|
||||||
|
"confidence": 0.95,
|
||||||
|
}))
|
||||||
|
edges.append(_edge("prev", "chunk", b_id, a_id, note_id, {
|
||||||
|
"chunk_id": b_id,
|
||||||
|
"edge_id": _mk_edge_id("prev", b_id, a_id, "chunk", "structure:order"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": "structure:order",
|
||||||
|
"confidence": 0.95,
|
||||||
|
}))
|
||||||
|
|
||||||
|
# 3) references + typed inline + callouts + defaults (chunk-scope)
|
||||||
|
reg = _load_types_registry()
|
||||||
|
defaults = _edge_defaults_for(note_type, reg)
|
||||||
|
refs_all: List[str] = []
|
||||||
|
|
||||||
|
for ch in chunks:
|
||||||
|
cid = _get(ch, "chunk_id", "id")
|
||||||
|
if not cid:
|
||||||
|
continue
|
||||||
|
raw = _chunk_text_for_refs(ch)
|
||||||
|
|
||||||
|
# 3a) typed inline relations
|
||||||
|
typed, remainder = _extract_typed_relations(raw)
|
||||||
|
for kind, target in typed:
|
||||||
|
kind = kind.strip().lower()
|
||||||
|
if not kind or not target:
|
||||||
|
continue
|
||||||
|
edges.append(_edge(kind, "chunk", cid, target, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(kind, cid, target, "chunk", "inline:rel"),
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "inline:rel",
|
||||||
|
"confidence": 0.95,
|
||||||
|
}))
|
||||||
|
if kind in {"related_to", "similar_to"}:
|
||||||
|
edges.append(_edge(kind, "chunk", target, cid, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(kind, target, cid, "chunk", "inline:rel"),
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "inline:rel",
|
||||||
|
"confidence": 0.95,
|
||||||
|
}))
|
||||||
|
|
||||||
|
# 3b) callouts
|
||||||
|
call_pairs, remainder2 = _extract_callout_relations(remainder)
|
||||||
|
for kind, target in call_pairs:
|
||||||
|
k = (kind or "").strip().lower()
|
||||||
|
if not k or not target:
|
||||||
|
continue
|
||||||
|
edges.append(_edge(k, "chunk", cid, target, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(k, cid, target, "chunk", "callout:edge"),
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "callout:edge",
|
||||||
|
"confidence": 0.95,
|
||||||
|
}))
|
||||||
|
if k in {"related_to", "similar_to"}:
|
||||||
|
edges.append(_edge(k, "chunk", target, cid, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(k, target, cid, "chunk", "callout:edge"),
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "callout:edge",
|
||||||
|
"confidence": 0.95,
|
||||||
|
}))
|
||||||
|
|
||||||
|
# 3c) generische Wikilinks → references (+ defaults je Ref)
|
||||||
|
refs = _extract_wikilinks(remainder2)
|
||||||
|
for r in refs:
|
||||||
|
edges.append(_edge("references", "chunk", cid, r, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"ref_text": r,
|
||||||
|
"edge_id": _mk_edge_id("references", cid, r, "chunk", "explicit:wikilink"),
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "explicit:wikilink",
|
||||||
|
"confidence": 1.0,
|
||||||
|
}))
|
||||||
|
for rel in defaults:
|
||||||
|
if rel == "references":
|
||||||
|
continue
|
||||||
|
edges.append(_edge(rel, "chunk", cid, r, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(rel, cid, r, "chunk", f"edge_defaults:{note_type}:{rel}"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
||||||
|
"confidence": 0.7,
|
||||||
|
}))
|
||||||
|
if rel in {"related_to", "similar_to"}:
|
||||||
|
edges.append(_edge(rel, "chunk", r, cid, note_id, {
|
||||||
|
"chunk_id": cid,
|
||||||
|
"edge_id": _mk_edge_id(rel, r, cid, "chunk", f"edge_defaults:{note_type}:{rel}"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
||||||
|
"confidence": 0.7,
|
||||||
|
}))
|
||||||
|
|
||||||
|
refs_all.extend(refs)
|
||||||
|
|
||||||
|
# 4) optional note-scope refs/backlinks (+ defaults)
|
||||||
|
if include_note_scope_refs:
|
||||||
|
refs_note = list(refs_all or [])
|
||||||
|
if note_level_references:
|
||||||
|
refs_note.extend([r for r in note_level_references if isinstance(r, str) and r])
|
||||||
|
refs_note = _dedupe_seq(refs_note)
|
||||||
|
for r in refs_note:
|
||||||
|
edges.append(_edge("references", "note", note_id, r, note_id, {
|
||||||
|
"edge_id": _mk_edge_id("references", note_id, r, "note", "explicit:note_scope"),
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "explicit:note_scope",
|
||||||
|
"confidence": 1.0,
|
||||||
|
}))
|
||||||
|
edges.append(_edge("backlink", "note", r, note_id, note_id, {
|
||||||
|
"edge_id": _mk_edge_id("backlink", r, note_id, "note", "derived:backlink"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": "derived:backlink",
|
||||||
|
"confidence": 0.9,
|
||||||
|
}))
|
||||||
|
for rel in defaults:
|
||||||
|
if rel == "references":
|
||||||
|
continue
|
||||||
|
edges.append(_edge(rel, "note", note_id, r, note_id, {
|
||||||
|
"edge_id": _mk_edge_id(rel, note_id, r, "note", f"edge_defaults:{note_type}:{rel}"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
||||||
|
"confidence": 0.7,
|
||||||
|
}))
|
||||||
|
if rel in {"related_to", "similar_to"}:
|
||||||
|
edges.append(_edge(rel, "note", r, note_id, note_id, {
|
||||||
|
"edge_id": _mk_edge_id(rel, r, note_id, "note", f"edge_defaults:{note_type}:{rel}"),
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": f"edge_defaults:{note_type}:{rel}",
|
||||||
|
"confidence": 0.7,
|
||||||
|
}))
|
||||||
|
|
||||||
|
# 5) De-Dupe (source_id, target_id, relation, rule_id)
|
||||||
|
seen: Set[Tuple[str,str,str,str]] = set()
|
||||||
|
out: List[dict] = []
|
||||||
|
for e in edges:
|
||||||
|
s = str(e.get("source_id") or "")
|
||||||
|
t = str(e.get("target_id") or "")
|
||||||
|
rel = str(e.get("relation") or e.get("kind") or "edge")
|
||||||
|
rule = str(e.get("rule_id") or "")
|
||||||
|
key = (s, t, rel, rule)
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
out.append(e)
|
||||||
|
return out
|
||||||
+288
-37
@@ -1,45 +1,296 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Modul: app/core/edges.py
|
||||||
|
Version: 2.0.0 (V2‑superset, rückwärtskompatibel zu v1 vom 2025‑09‑09)
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
-----
|
||||||
|
Bewahrt die bestehende Edgelogik (belongs_to, prev/next, references, backlink)
|
||||||
|
und ergänzt V2‑Felder + Typ‑Default‑Kanten gemäß config/types.yaml (edge_defaults).
|
||||||
|
Die Funktion ist **idempotent** und **rückwärtskompatibel** zur bisherigen Signatur.
|
||||||
|
|
||||||
|
Kompatibilitätsgarantien (gegenüber v1):
|
||||||
|
- **Input**: akzeptiert identische Chunk‑Payloads wie v1:
|
||||||
|
* `id` (Chunk‑ID), `note_id` (Owner), `neighbors.prev|next` (optional),
|
||||||
|
`references: [{target_id: ...}]` (optional),
|
||||||
|
alternativ: `chunk_id`, `chunk_index|ord`, `window|text`
|
||||||
|
- **Output (v1‑Felder)**: `kind`, `source_id`, `target_id`, `scope`, `note_id`, `edge_id`
|
||||||
|
- **Neu (v2‑Felder)**: `relation`, `src_note_id`, `src_chunk_id?`, `dst_note_id`, `dst_chunk_id?`,
|
||||||
|
`provenance` (`explicit|rule`), `rule_id?`, `confidence?`
|
||||||
|
|
||||||
|
Regeln
|
||||||
|
------
|
||||||
|
- Deduplizierungsschlüssel: (source_id, target_id, relation, rule_id)
|
||||||
|
- Strukturkanten:
|
||||||
|
* belongs_to: 1× pro Chunk
|
||||||
|
* next/prev: Sequenz der Chunks; nutzt bevorzugt neighbors; sonst ord/chunk_index
|
||||||
|
- Explizite Referenzen:
|
||||||
|
* aus Chunk: `references[].target_id` (falls vorhanden)
|
||||||
|
* Fallback: Wikilinks in `window|text`: [[Some Title|some-id]] oder [[some-id]]
|
||||||
|
- Note‑Scope:
|
||||||
|
* backlink immer; references nur, wenn include_note_scope_refs=True
|
||||||
|
- Typ‑Defaults (edge_defaults aus config/types.yaml des **Quell‑Notiztyps**):
|
||||||
|
* Für jede explizite Referenz wird je default‑Relation eine Regel‑Kante erzeugt
|
||||||
|
* rule_id: "type_default:{note_type}:{relation}:v1", provenance="rule"
|
||||||
|
|
||||||
|
Konfiguration
|
||||||
|
-------------
|
||||||
|
- ENV MINDNET_TYPES_FILE (Default: ./config/types.yaml)
|
||||||
|
|
||||||
|
Lizenz/Autor
|
||||||
|
------------
|
||||||
|
- Erstimplementierung v1 (2025‑09‑09) — Projekt Mindnet
|
||||||
|
- Erweiterung v2 (2025‑11‑11) — kompatible Superset‑Implementierung
|
||||||
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from typing import List, Dict
|
|
||||||
|
|
||||||
def deriv_edges_for_note(note_meta: Dict, chunk_payloads: List[Dict]) -> List[Dict]:
|
import os
|
||||||
edges: List[Dict] = []
|
import re
|
||||||
|
from typing import Dict, Iterable, List, Optional, Tuple, Set
|
||||||
|
|
||||||
# Chunk → Note (belongs_to) + prev/next
|
try:
|
||||||
for idx, ch in enumerate(chunk_payloads):
|
import yaml # optional, nur für types.yaml
|
||||||
edges.append({
|
except Exception: # pragma: no cover
|
||||||
"src_id": ch["id"], "dst_id": note_meta["id"],
|
yaml = None
|
||||||
"edge_type": "belongs_to", "scope": "chunk"
|
|
||||||
})
|
|
||||||
prev_id = ch.get("neighbors",{}).get("prev")
|
|
||||||
next_id = ch.get("neighbors",{}).get("next")
|
|
||||||
if prev_id:
|
|
||||||
edges.append({"src_id": ch["id"], "dst_id": prev_id, "edge_type": "next", "scope": "chunk"})
|
|
||||||
edges.append({"src_id": prev_id, "dst_id": ch["id"], "edge_type": "prev", "scope": "chunk"})
|
|
||||||
if next_id:
|
|
||||||
edges.append({"src_id": ch["id"], "dst_id": next_id, "edge_type": "next", "scope": "chunk"})
|
|
||||||
edges.append({"src_id": next_id, "dst_id": ch["id"], "edge_type": "prev", "scope": "chunk"})
|
|
||||||
|
|
||||||
# references aus wikilinks (Chunk-Scope)
|
# ------------------------------------------------------------
|
||||||
for ref in ch.get("references", []):
|
# Hilfen: types.yaml laden (edge_defaults)
|
||||||
tid = ref.get("target_id")
|
# ------------------------------------------------------------
|
||||||
if not tid: continue
|
|
||||||
edges.append({"src_id": ch["id"], "dst_id": tid, "edge_type": "references", "scope": "chunk"})
|
|
||||||
edges.append({"src_id": tid, "dst_id": ch["id"], "edge_type": "backlink", "scope": "chunk"})
|
|
||||||
|
|
||||||
# depends_on / assigned_to (Note-Scope, aus Frontmatter)
|
def _types_path() -> str:
|
||||||
for dep in note_meta.get("depends_on", []) or []:
|
return os.getenv("MINDNET_TYPES_FILE") or "./config/types.yaml"
|
||||||
edges.append({"src_id": note_meta["id"], "dst_id": dep, "edge_type": "depends_on", "scope": "note"})
|
|
||||||
for ass in note_meta.get("assigned_to", []) or []:
|
|
||||||
edges.append({"src_id": note_meta["id"], "dst_id": ass, "edge_type": "assigned_to", "scope": "note"})
|
|
||||||
|
|
||||||
# Note-Level references (optional: falls du im Note-Payload `references` sammelst)
|
def _load_types() -> Dict[str, dict]:
|
||||||
for tid in note_meta.get("references", []) or []:
|
p = _types_path()
|
||||||
edges.append({"src_id": note_meta["id"], "dst_id": tid, "edge_type": "references", "scope": "note"})
|
if not os.path.isfile(p) or yaml is None:
|
||||||
edges.append({"src_id": tid, "dst_id": note_meta["id"], "edge_type": "backlink", "scope": "note"})
|
return {}
|
||||||
|
try:
|
||||||
|
with open(p, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
if isinstance(data, dict) and "types" in data and isinstance(data["types"], dict):
|
||||||
|
return data["types"]
|
||||||
|
return data if isinstance(data, dict) else {}
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
# Dedupe
|
def _edge_defaults_for(note_type: Optional[str]) -> List[str]:
|
||||||
uniq = {}
|
types = _load_types()
|
||||||
|
t = (note_type or "").strip().lower()
|
||||||
|
cfg = types.get(t) or {}
|
||||||
|
defaults = cfg.get("edge_defaults") or []
|
||||||
|
if isinstance(defaults, str):
|
||||||
|
defaults = [defaults]
|
||||||
|
return [str(x) for x in defaults if isinstance(x, (str, int, float))]
|
||||||
|
|
||||||
|
# ------------------------------------------------------------
|
||||||
|
# Wikilink‑Parser (Fallback, wenn ch["references"] fehlt)
|
||||||
|
# ------------------------------------------------------------
|
||||||
|
|
||||||
|
_WIKILINK_RE = re.compile(r"\[\[(?:[^\|\]]+\|)?([a-zA-Z0-9_\-#:. ]+)\]\]")
|
||||||
|
|
||||||
|
def _extract_wikilinks(text: str) -> List[str]:
|
||||||
|
ids: List[str] = []
|
||||||
|
for m in _WIKILINK_RE.finditer(text or ""):
|
||||||
|
ids.append(m.group(1).strip())
|
||||||
|
return ids
|
||||||
|
|
||||||
|
# ------------------------------------------------------------
|
||||||
|
# Utility
|
||||||
|
# ------------------------------------------------------------
|
||||||
|
|
||||||
|
def _mk_edge_id(kind: str, s: str, t: str, scope: str, rule_id: Optional[str] = None) -> str:
|
||||||
|
base = f"{kind}:{s}->{t}#{scope}"
|
||||||
|
if rule_id:
|
||||||
|
base += f"|{rule_id}"
|
||||||
|
try:
|
||||||
|
import hashlib
|
||||||
|
return hashlib.blake2s(base.encode("utf-8"), digest_size=12).hexdigest()
|
||||||
|
except Exception: # pragma: no cover
|
||||||
|
return base
|
||||||
|
|
||||||
|
def _dedupe(edges: List[Dict]) -> List[Dict]:
|
||||||
|
seen: Set[Tuple[str,str,str,str]] = set()
|
||||||
|
out: List[Dict] = []
|
||||||
for e in edges:
|
for e in edges:
|
||||||
k = (e["src_id"], e["edge_type"], e["dst_id"], e.get("scope","note"))
|
s = str(e.get("source_id") or "")
|
||||||
uniq[k] = e
|
t = str(e.get("target_id") or "")
|
||||||
return list(uniq.values())
|
rel = str(e.get("relation") or e.get("kind") or "edge")
|
||||||
|
rule = str(e.get("rule_id") or "")
|
||||||
|
key = (s, t, rel, rule)
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
out.append(e)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _first(v: dict, *keys, default=None):
|
||||||
|
for k in keys:
|
||||||
|
if k in v and v[k] is not None:
|
||||||
|
return v[k]
|
||||||
|
return default
|
||||||
|
|
||||||
|
# ------------------------------------------------------------
|
||||||
|
# Hauptfunktion
|
||||||
|
# ------------------------------------------------------------
|
||||||
|
|
||||||
|
def build_edges_for_note(
|
||||||
|
note_id: str,
|
||||||
|
chunk_payloads: List[Dict],
|
||||||
|
note_level_refs: Optional[List[str]] = None,
|
||||||
|
*,
|
||||||
|
include_note_scope_refs: bool = False,
|
||||||
|
) -> List[Dict]:
|
||||||
|
edges: List[Dict] = []
|
||||||
|
chunks = list(chunk_payloads or [])
|
||||||
|
# Notiztyp aus erstem Chunk ableiten (kompatibel zu existierenden Payloads)
|
||||||
|
note_type = (chunks[0].get("type") if chunks else None) or (chunks[0].get("note_type") if chunks else None)
|
||||||
|
|
||||||
|
# --- Strukturkanten ------------------------------------------------------
|
||||||
|
# belongs_to
|
||||||
|
for ch in chunks:
|
||||||
|
cid = _first(ch, "id", "chunk_id")
|
||||||
|
if not cid:
|
||||||
|
continue
|
||||||
|
owner = ch.get("note_id") or note_id
|
||||||
|
e = {
|
||||||
|
"edge_id": _mk_edge_id("belongs_to", cid, note_id, "chunk", "structure:belongs_to:v1"),
|
||||||
|
"kind": "belongs_to",
|
||||||
|
"relation": "belongs_to",
|
||||||
|
"scope": "chunk",
|
||||||
|
"source_id": cid,
|
||||||
|
"target_id": note_id,
|
||||||
|
"note_id": owner, # v1-Kompat
|
||||||
|
# v2
|
||||||
|
"src_note_id": owner,
|
||||||
|
"src_chunk_id": cid,
|
||||||
|
"dst_note_id": note_id,
|
||||||
|
"provenance": "rule",
|
||||||
|
"rule_id": "structure:belongs_to:v1",
|
||||||
|
"confidence": 1.0,
|
||||||
|
}
|
||||||
|
edges.append(e)
|
||||||
|
|
||||||
|
# next/prev — bevorzugt neighbors.prev/next; sonst via ord/chunk_index
|
||||||
|
# Map der Chunks nach Index
|
||||||
|
ordered = list(chunks)
|
||||||
|
def _idx(c):
|
||||||
|
return _first(c, "chunk_index", "ord", default=0)
|
||||||
|
ordered.sort(key=_idx)
|
||||||
|
|
||||||
|
for i, ch in enumerate(ordered):
|
||||||
|
cid = _first(ch, "id", "chunk_id")
|
||||||
|
if not cid:
|
||||||
|
continue
|
||||||
|
owner = ch.get("note_id") or note_id
|
||||||
|
nb = ch.get("neighbors") or {}
|
||||||
|
prev_id = nb.get("prev")
|
||||||
|
next_id = nb.get("next")
|
||||||
|
# Fallback-Reihenfolge
|
||||||
|
if prev_id is None and i > 0:
|
||||||
|
prev_id = _first(ordered[i-1], "id", "chunk_id")
|
||||||
|
if next_id is None and i+1 < len(ordered):
|
||||||
|
next_id = _first(ordered[i+1], "id", "chunk_id")
|
||||||
|
|
||||||
|
if prev_id:
|
||||||
|
edges.append({
|
||||||
|
"edge_id": _mk_edge_id("prev", cid, prev_id, "chunk", "structure:order:v1"),
|
||||||
|
"kind": "prev", "relation": "prev", "scope": "chunk",
|
||||||
|
"source_id": cid, "target_id": prev_id, "note_id": owner,
|
||||||
|
"src_note_id": owner, "src_chunk_id": cid,
|
||||||
|
"dst_note_id": owner, "dst_chunk_id": prev_id,
|
||||||
|
"provenance": "rule", "rule_id": "structure:order:v1", "confidence": 0.95,
|
||||||
|
})
|
||||||
|
edges.append({
|
||||||
|
"edge_id": _mk_edge_id("next", prev_id, cid, "chunk", "structure:order:v1"),
|
||||||
|
"kind": "next", "relation": "next", "scope": "chunk",
|
||||||
|
"source_id": prev_id, "target_id": cid, "note_id": owner,
|
||||||
|
"src_note_id": owner, "src_chunk_id": prev_id,
|
||||||
|
"dst_note_id": owner, "dst_chunk_id": cid,
|
||||||
|
"provenance": "rule", "rule_id": "structure:order:v1", "confidence": 0.95,
|
||||||
|
})
|
||||||
|
|
||||||
|
# --- Explizite Referenzen (Chunk‑Scope) ---------------------------------
|
||||||
|
explicit_refs: List[Dict] = []
|
||||||
|
for ch in chunks:
|
||||||
|
cid = _first(ch, "id", "chunk_id")
|
||||||
|
if not cid:
|
||||||
|
continue
|
||||||
|
owner = ch.get("note_id") or note_id
|
||||||
|
# 1) bevorzugt vorhandene ch["references"]
|
||||||
|
refs = ch.get("references") or []
|
||||||
|
targets = [r.get("target_id") for r in refs if isinstance(r, dict) and r.get("target_id")]
|
||||||
|
# 2) Fallback: Wikilinks aus Text
|
||||||
|
if not targets:
|
||||||
|
text = _first(ch, "window", "text", default="") or ""
|
||||||
|
targets = _extract_wikilinks(text)
|
||||||
|
for tid in targets:
|
||||||
|
if not isinstance(tid, str) or not tid.strip():
|
||||||
|
continue
|
||||||
|
e = {
|
||||||
|
"edge_id": _mk_edge_id("references", cid, tid, "chunk"),
|
||||||
|
"kind": "references",
|
||||||
|
"relation": "references",
|
||||||
|
"scope": "chunk",
|
||||||
|
"source_id": cid,
|
||||||
|
"target_id": tid,
|
||||||
|
"note_id": owner,
|
||||||
|
# v2
|
||||||
|
"src_note_id": owner,
|
||||||
|
"src_chunk_id": cid,
|
||||||
|
"dst_note_id": tid,
|
||||||
|
"provenance": "explicit",
|
||||||
|
"rule_id": "",
|
||||||
|
"confidence": 1.0,
|
||||||
|
}
|
||||||
|
edges.append(e)
|
||||||
|
explicit_refs.append(e)
|
||||||
|
|
||||||
|
# --- Note‑Scope: references (optional) + backlink (immer) ----------------
|
||||||
|
unique_refs = []
|
||||||
|
if note_level_refs:
|
||||||
|
seen = set()
|
||||||
|
for tid in note_level_refs:
|
||||||
|
if isinstance(tid, str) and tid.strip() and tid not in seen:
|
||||||
|
unique_refs.append(tid); seen.add(tid)
|
||||||
|
|
||||||
|
for tid in unique_refs:
|
||||||
|
if include_note_scope_refs:
|
||||||
|
edges.append({
|
||||||
|
"edge_id": _mk_edge_id("references", note_id, tid, "note"),
|
||||||
|
"kind": "references", "relation": "references", "scope": "note",
|
||||||
|
"source_id": note_id, "target_id": tid, "note_id": note_id,
|
||||||
|
"src_note_id": note_id, "dst_note_id": tid,
|
||||||
|
"provenance": "explicit", "rule_id": "", "confidence": 1.0,
|
||||||
|
})
|
||||||
|
edges.append({
|
||||||
|
"edge_id": _mk_edge_id("backlink", tid, note_id, "note", "derived:backlink:v1"),
|
||||||
|
"kind": "backlink", "relation": "backlink", "scope": "note",
|
||||||
|
"source_id": tid, "target_id": note_id, "note_id": note_id,
|
||||||
|
"src_note_id": tid, "dst_note_id": note_id,
|
||||||
|
"provenance": "rule", "rule_id": "derived:backlink:v1", "confidence": 0.9,
|
||||||
|
})
|
||||||
|
|
||||||
|
# --- Type‑Defaults je expliziter Referenz --------------------------------
|
||||||
|
defaults = [d for d in _edge_defaults_for(note_type) if d and d != "references"]
|
||||||
|
if defaults:
|
||||||
|
for e in explicit_refs + ([ ] if not include_note_scope_refs else []):
|
||||||
|
# wir nutzen die bereits erzeugten explicit‑Edges als Vorlage
|
||||||
|
src = e["source_id"]; tgt = e["target_id"]
|
||||||
|
scope = e.get("scope", "chunk")
|
||||||
|
s_note = e.get("src_note_id") or note_id
|
||||||
|
s_chunk = e.get("src_chunk_id")
|
||||||
|
t_note = e.get("dst_note_id") or tgt
|
||||||
|
for rel in defaults:
|
||||||
|
rule_id = f"type_default:{(note_type or 'unknown')}:{rel}:v1"
|
||||||
|
edges.append({
|
||||||
|
"edge_id": _mk_edge_id(rel, src, tgt, scope, rule_id),
|
||||||
|
"kind": rel, "relation": rel, "scope": scope,
|
||||||
|
"source_id": src, "target_id": tgt, "note_id": s_note,
|
||||||
|
"src_note_id": s_note, "src_chunk_id": s_chunk,
|
||||||
|
"dst_note_id": t_note,
|
||||||
|
"provenance": "rule", "rule_id": rule_id, "confidence": 0.7,
|
||||||
|
})
|
||||||
|
|
||||||
|
# --- Dedupe & Return -----------------------------------------------------
|
||||||
|
return _dedupe(edges)
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
# app/core/edges_writer.py
|
||||||
|
from __future__ import annotations
|
||||||
|
import hashlib
|
||||||
|
from typing import Dict, List, Iterable, Tuple
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Dein Modul mit der Schemadefinition und der Builder-Funktion
|
||||||
|
from app.core.edges import build_edges_for_note # noqa: F401
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError("Konnte app.core.edges nicht importieren. "
|
||||||
|
"Bitte sicherstellen, dass app/core/edges.py vorhanden ist.") from e
|
||||||
|
|
||||||
|
def _edge_uid(kind: str, source_id: str, target_id: str, scope: str) -> str:
|
||||||
|
"""
|
||||||
|
Deterministische, kurze ID für eine Edge.
|
||||||
|
Kollisionen sind praktisch ausgeschlossen (BLAKE2s über den Kanonischen Schlüssel).
|
||||||
|
"""
|
||||||
|
key = f"{kind}|{source_id}|{target_id}|{scope}"
|
||||||
|
return hashlib.blake2s(key.encode("utf-8"), digest_size=12).hexdigest()
|
||||||
|
|
||||||
|
def ensure_edges_collection(qdrant_client, collection: str) -> None:
|
||||||
|
"""
|
||||||
|
Legt die Edge-Collection an, falls sie nicht existiert.
|
||||||
|
Minimal: 1D-Vector (Dummy), Cosine. Payload-only-Collections sind je nach Qdrant-Version heikel.
|
||||||
|
"""
|
||||||
|
from qdrant_client.http import models as qm
|
||||||
|
|
||||||
|
existing = [c.name for c in qdrant_client.get_collections().collections]
|
||||||
|
if collection in existing:
|
||||||
|
return
|
||||||
|
|
||||||
|
qdrant_client.recreate_collection(
|
||||||
|
collection_name=collection,
|
||||||
|
vectors_config=qm.VectorParams(size=1, distance=qm.Distance.COSINE),
|
||||||
|
on_disk_payload=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def edges_from_note(
|
||||||
|
note_id: str,
|
||||||
|
chunk_payloads: List[Dict],
|
||||||
|
note_level_refs: Iterable[str] | None,
|
||||||
|
*,
|
||||||
|
include_note_scope_refs: bool = False,
|
||||||
|
) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Ruft deinen Edge-Builder auf und gibt die (deduplizierten) Edge-Payloads zurück.
|
||||||
|
Keine Schemaänderung – exakt das aus app/core/edges.py.
|
||||||
|
"""
|
||||||
|
return build_edges_for_note(
|
||||||
|
note_id=note_id,
|
||||||
|
chunk_payloads=chunk_payloads,
|
||||||
|
note_level_refs=list(note_level_refs or []),
|
||||||
|
include_note_scope_refs=include_note_scope_refs,
|
||||||
|
)
|
||||||
|
|
||||||
|
def upsert_edges(
|
||||||
|
qdrant_client,
|
||||||
|
collection: str,
|
||||||
|
edge_payloads: List[Dict],
|
||||||
|
) -> Tuple[int, int]:
|
||||||
|
"""
|
||||||
|
Schreibt Edges als Points in Qdrant.
|
||||||
|
- id: deterministisch aus (kind, source_id, target_id, scope)
|
||||||
|
- vector: [0.0] Dummy
|
||||||
|
- payload: Edge-Dict (unverändert, siehe Schema in app/core/edges.py)
|
||||||
|
Gibt (anzahl_points, anzahl_unique_keys) zurück.
|
||||||
|
"""
|
||||||
|
from qdrant_client.models import PointStruct
|
||||||
|
|
||||||
|
if not edge_payloads:
|
||||||
|
return 0, 0
|
||||||
|
|
||||||
|
points = []
|
||||||
|
seen = set()
|
||||||
|
for e in edge_payloads:
|
||||||
|
key = (e.get("kind"), e.get("source_id"), e.get("target_id"), e.get("scope"))
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
eid = _edge_uid(*key)
|
||||||
|
points.append(
|
||||||
|
PointStruct(
|
||||||
|
id=eid,
|
||||||
|
vector=[0.0],
|
||||||
|
payload=e,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not points:
|
||||||
|
return 0, 0
|
||||||
|
|
||||||
|
ensure_edges_collection(qdrant_client, collection)
|
||||||
|
qdrant_client.upsert(collection_name=collection, points=points)
|
||||||
|
return len(points), len(seen)
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Datei: app/core/env_vars.py
|
||||||
|
Version: 1.1.0 (2025-11-08)
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
Einheitliche Auflösung von ENV-Variablen (Prefix, Qdrant, Embeddings, Hashing)
|
||||||
|
mit Abwärtskompatibilität.
|
||||||
|
|
||||||
|
Grundsatz
|
||||||
|
- Für Qdrant-Funktionen ist 'COLLECTION_PREFIX' der Primärschlüssel.
|
||||||
|
- 'MINDNET_PREFIX' bleibt für App-/UI-/Exporter-Kontexte nutzbar.
|
||||||
|
- Fallbacks sorgen dafür, dass ältere Umgebungen weiter funktionieren.
|
||||||
|
|
||||||
|
Wichtig
|
||||||
|
- Lädt optional eine .env (wenn python-dotenv verfügbar ist).
|
||||||
|
- Überschreibt keine bereits gesetzten OS-Variablen (override=False).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from typing import Optional, Dict
|
||||||
|
|
||||||
|
# Optional: .env automatisch laden (ohne Hard-Fail, falls nicht vorhanden)
|
||||||
|
try:
|
||||||
|
from dotenv import load_dotenv, find_dotenv # type: ignore
|
||||||
|
_p = find_dotenv()
|
||||||
|
if _p:
|
||||||
|
load_dotenv(_p, override=False)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# -------- Prefix-Auflösung --------
|
||||||
|
|
||||||
|
def get_collection_prefix(cli_override: Optional[str] = None) -> str:
|
||||||
|
"""
|
||||||
|
Für Qdrant-relevante Funktionen:
|
||||||
|
1) CLI-Override (--prefix)
|
||||||
|
2) ENV COLLECTION_PREFIX
|
||||||
|
3) ENV MINDNET_PREFIX (Fallback)
|
||||||
|
4) 'mindnet' (Default)
|
||||||
|
"""
|
||||||
|
if cli_override and str(cli_override).strip():
|
||||||
|
return str(cli_override).strip()
|
||||||
|
return (
|
||||||
|
os.getenv("COLLECTION_PREFIX")
|
||||||
|
or os.getenv("MINDNET_PREFIX")
|
||||||
|
or "mindnet"
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_mindnet_prefix(cli_override: Optional[str] = None) -> str:
|
||||||
|
"""
|
||||||
|
Für App-/UI-/Exporter-Kontexte:
|
||||||
|
1) CLI-Override (--prefix)
|
||||||
|
2) ENV MINDNET_PREFIX
|
||||||
|
3) ENV COLLECTION_PREFIX (Fallback)
|
||||||
|
4) 'mindnet'
|
||||||
|
"""
|
||||||
|
if cli_override and str(cli_override).strip():
|
||||||
|
return str(cli_override).strip()
|
||||||
|
return (
|
||||||
|
os.getenv("MINDNET_PREFIX")
|
||||||
|
or os.getenv("COLLECTION_PREFIX")
|
||||||
|
or "mindnet"
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_prefix(cli_override: Optional[str] = None, target: str = "qdrant") -> str:
|
||||||
|
"""
|
||||||
|
Universelle Hülle (abwärtskompatibel):
|
||||||
|
target='qdrant' -> get_collection_prefix
|
||||||
|
target='app' -> get_mindnet_prefix
|
||||||
|
"""
|
||||||
|
if target.lower() == "app":
|
||||||
|
return get_mindnet_prefix(cli_override)
|
||||||
|
return get_collection_prefix(cli_override)
|
||||||
|
|
||||||
|
# -------- Qdrant / Embeddings / Hashing --------
|
||||||
|
|
||||||
|
def get_qdrant_url(default: str = "http://127.0.0.1:6333") -> str:
|
||||||
|
return os.getenv("QDRANT_URL", default)
|
||||||
|
|
||||||
|
def get_qdrant_api_key(default: str = "") -> str:
|
||||||
|
return os.getenv("QDRANT_API_KEY", default)
|
||||||
|
|
||||||
|
def get_vector_dim(default: int = 384) -> int:
|
||||||
|
try:
|
||||||
|
return int(os.getenv("VECTOR_DIM", str(default)))
|
||||||
|
except Exception:
|
||||||
|
return default
|
||||||
|
|
||||||
|
def get_embed_url(default: Optional[str] = None) -> Optional[str]:
|
||||||
|
return os.getenv("EMBED_URL", default)
|
||||||
|
|
||||||
|
def get_hash_env() -> Dict[str, str]:
|
||||||
|
"""
|
||||||
|
Liefert die Hash-Konfiguration (nur Aggregation; die Auswertung bleibt in den Skripten).
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"MINDNET_HASH_COMPARE": os.getenv("MINDNET_HASH_COMPARE", ""),
|
||||||
|
"MINDNET_HASH_SOURCE": os.getenv("MINDNET_HASH_SOURCE", ""),
|
||||||
|
"MINDNET_HASH_NORMALIZE": os.getenv("MINDNET_HASH_NORMALIZE", ""),
|
||||||
|
}
|
||||||
@@ -0,0 +1,305 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
app/core/graph_adapter.py — Adjazenzaufbau & Subgraph-Expansion (WP-04)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Baut aus Qdrant-Edges (Collection: *_edges) einen leichten In-Memory-Graph
|
||||||
|
und liefert Edge-basierte Kennzahlen (In-Degree, Out-Degree, edge_bonus).
|
||||||
|
|
||||||
|
Kompatibilität:
|
||||||
|
- Python 3.12+, qdrant-client 1.x
|
||||||
|
- Wird von app/core/retriever.py im Hybrid-Modus genutzt.
|
||||||
|
- Signaturen bleiben kompatibel zu den bestehenden Tests
|
||||||
|
(tests/test_retriever_edges.py patcht expand()).
|
||||||
|
|
||||||
|
Version:
|
||||||
|
0.3.1 (2025-12-04 – note_id-bewusste Expansion:
|
||||||
|
* Seeds = Note-IDs
|
||||||
|
* Subgraph pflegt zusätzlich Adjazenz auf note_id-Basis,
|
||||||
|
damit der Retriever mit note_id arbeiten kann)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Dict, List, Optional, DefaultDict
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
from app.core.qdrant import collection_names
|
||||||
|
|
||||||
|
# Legacy-Import (wird aktuell nicht mehr verwendet, bleibt aber erhalten,
|
||||||
|
# damit bestehende Importe/Mocks nicht brechen).
|
||||||
|
try: # pragma: no cover
|
||||||
|
from app.core.qdrant_points import get_edges_for_sources # type: ignore
|
||||||
|
except Exception: # pragma: no cover
|
||||||
|
get_edges_for_sources = None # type: ignore
|
||||||
|
|
||||||
|
|
||||||
|
# Basisgewichte je Edge-Typ.
|
||||||
|
# Diese Werte werden mit der in der Edge-Payload hinterlegten "confidence"
|
||||||
|
# multipliziert, falls vorhanden.
|
||||||
|
#
|
||||||
|
# Hinweis:
|
||||||
|
# - Strukturkanten (belongs_to, next/prev) sind schwächer gewichtet.
|
||||||
|
# - Wissenskanten (depends_on, related_to, similar_to, references) erhalten
|
||||||
|
# höhere Basisgewichte, damit sie sich im Retriever bemerkbar machen.
|
||||||
|
EDGE_BASE_WEIGHTS: Dict[str, float] = {
|
||||||
|
# Struktur / Navigationskanten
|
||||||
|
"belongs_to": 0.10,
|
||||||
|
"next": 0.06,
|
||||||
|
"prev": 0.06,
|
||||||
|
"backlink": 0.04,
|
||||||
|
"references_at": 0.08,
|
||||||
|
|
||||||
|
# Wissenskanten
|
||||||
|
"references": 0.20,
|
||||||
|
"depends_on": 0.18,
|
||||||
|
"related_to": 0.15,
|
||||||
|
"similar_to": 0.12,
|
||||||
|
# weitere Typen erhalten per Default 0.0 und wirken nur über centrality
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _edge_weight(pl: Dict) -> float:
|
||||||
|
"""Berechnet das effektive Edge-Gewicht aus kind + confidence."""
|
||||||
|
kind = pl.get("kind", "edge")
|
||||||
|
base = EDGE_BASE_WEIGHTS.get(kind, 0.0)
|
||||||
|
|
||||||
|
conf_raw = pl.get("confidence", None)
|
||||||
|
try:
|
||||||
|
conf = float(conf_raw) if conf_raw is not None else None
|
||||||
|
except Exception:
|
||||||
|
conf = None
|
||||||
|
|
||||||
|
if conf is None:
|
||||||
|
return base
|
||||||
|
|
||||||
|
# Confidence vorsichtig in [0.0, 1.0] clampen
|
||||||
|
if conf < 0.0:
|
||||||
|
conf = 0.0
|
||||||
|
if conf > 1.0:
|
||||||
|
conf = 1.0
|
||||||
|
|
||||||
|
return base * conf
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_edges(
|
||||||
|
client: QdrantClient,
|
||||||
|
prefix: str,
|
||||||
|
seeds: List[str],
|
||||||
|
edge_types: Optional[List[str]] = None,
|
||||||
|
limit: int = 2048,
|
||||||
|
) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Holt Edges direkt aus der *_edges Collection.
|
||||||
|
|
||||||
|
Filter:
|
||||||
|
- source_id IN seeds ODER target_id IN seeds ODER note_id IN seeds
|
||||||
|
- optional: kind IN edge_types
|
||||||
|
"""
|
||||||
|
if not seeds or limit <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
_, _, edges_col = collection_names(prefix)
|
||||||
|
|
||||||
|
# OR über source_id / target_id / note_id für alle Seeds
|
||||||
|
seed_conditions = []
|
||||||
|
for field in ("source_id", "target_id", "note_id"):
|
||||||
|
for s in seeds:
|
||||||
|
seed_conditions.append(
|
||||||
|
rest.FieldCondition(
|
||||||
|
key=field,
|
||||||
|
match=rest.MatchValue(value=str(s)),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
seeds_filter = rest.Filter(should=seed_conditions) if seed_conditions else None
|
||||||
|
|
||||||
|
# Optional: Filter auf bestimmte Edge-Typen (kind)
|
||||||
|
type_filter = None
|
||||||
|
if edge_types:
|
||||||
|
type_conds = [
|
||||||
|
rest.FieldCondition(
|
||||||
|
key="kind",
|
||||||
|
match=rest.MatchValue(value=str(k)),
|
||||||
|
)
|
||||||
|
for k in edge_types
|
||||||
|
]
|
||||||
|
type_filter = rest.Filter(should=type_conds)
|
||||||
|
|
||||||
|
flt = None
|
||||||
|
must = []
|
||||||
|
if seeds_filter:
|
||||||
|
must.append(seeds_filter)
|
||||||
|
if type_filter:
|
||||||
|
must.append(type_filter)
|
||||||
|
if must:
|
||||||
|
flt = rest.Filter(must=must)
|
||||||
|
|
||||||
|
pts, _ = client.scroll(
|
||||||
|
collection_name=edges_col,
|
||||||
|
scroll_filter=flt,
|
||||||
|
limit=limit,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
out: List[Dict] = []
|
||||||
|
for p in pts or []:
|
||||||
|
pl = dict(p.payload or {})
|
||||||
|
if pl:
|
||||||
|
out.append(pl)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class Subgraph:
|
||||||
|
"""Leichtgewichtiger Subgraph mit Adjazenzlisten & einfachen Kennzahlen."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.adj: DefaultDict[str, List[Dict]] = defaultdict(list)
|
||||||
|
self.in_degree: DefaultDict[str, int] = defaultdict(int)
|
||||||
|
self.out_degree: DefaultDict[str, int] = defaultdict(int)
|
||||||
|
|
||||||
|
def add_edge(self, e: Dict) -> None:
|
||||||
|
"""
|
||||||
|
Fügt eine Kante zum Subgraph hinzu.
|
||||||
|
|
||||||
|
e enthält:
|
||||||
|
- source: Knoten-ID (z. B. source_id)
|
||||||
|
- target: Knoten-ID (z. B. target_id)
|
||||||
|
- kind: Kantentyp
|
||||||
|
- weight: bereits berechnetes Gewicht
|
||||||
|
- note_id: (optional) Kontext-Note dieser Kante
|
||||||
|
|
||||||
|
Wichtig für den Retriever:
|
||||||
|
- Wir pflegen Adjazenz primär unter `source`.
|
||||||
|
- Zusätzlich pflegen wir, falls vorhanden, eine Adjazenz
|
||||||
|
unter `note_id`, damit der Retriever mit note_id arbeiten kann.
|
||||||
|
"""
|
||||||
|
src = e.get("source")
|
||||||
|
tgt = e.get("target")
|
||||||
|
kind = e.get("kind")
|
||||||
|
weight = e.get("weight", EDGE_BASE_WEIGHTS.get(kind, 0.0))
|
||||||
|
owner = e.get("note_id")
|
||||||
|
|
||||||
|
if not src or not tgt:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Primäre Adjazenz (z. B. source_id → target_id)
|
||||||
|
self.adj[src].append(
|
||||||
|
{
|
||||||
|
"target": tgt,
|
||||||
|
"kind": kind,
|
||||||
|
"weight": weight,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
self.out_degree[src] += 1
|
||||||
|
self.in_degree[tgt] += 1
|
||||||
|
|
||||||
|
# Sekundäre Adjazenz auf Note-Ebene:
|
||||||
|
# Falls eine Kontext-Note existiert, bekommt diese ebenfalls
|
||||||
|
# "Ausgangskanten" angerechnet. So kann der Retriever über note_id
|
||||||
|
# edge_bonus(...) und centrality_bonus(...) abfragen.
|
||||||
|
if owner:
|
||||||
|
# same edge unter owner registrieren (falls nicht identisch mit src),
|
||||||
|
# damit edge_bonus(owner) > 0 werden kann.
|
||||||
|
if owner != src:
|
||||||
|
self.adj[owner].append(
|
||||||
|
{
|
||||||
|
"target": tgt,
|
||||||
|
"kind": kind,
|
||||||
|
"weight": weight,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
self.out_degree[owner] += 1
|
||||||
|
|
||||||
|
# Zentralität der Kontext-Note leicht erhöhen
|
||||||
|
if owner != tgt:
|
||||||
|
self.in_degree[owner] += 1
|
||||||
|
|
||||||
|
def aggregate_edge_bonus(self, node_id: str) -> float:
|
||||||
|
"""
|
||||||
|
Summe der ausgehenden Kantengewichte für einen Knoten.
|
||||||
|
"""
|
||||||
|
return sum(edge["weight"] for edge in self.adj.get(node_id, []))
|
||||||
|
|
||||||
|
def edge_bonus(self, node_id: str) -> float:
|
||||||
|
"""
|
||||||
|
Kompatibilitäts-Methode für den Retriever.
|
||||||
|
|
||||||
|
Der Retriever ruft subgraph.edge_bonus(node_id) auf. Intern verwenden
|
||||||
|
wir aggregate_edge_bonus(...), um bestehende Tests nicht zu brechen.
|
||||||
|
"""
|
||||||
|
return self.aggregate_edge_bonus(node_id)
|
||||||
|
|
||||||
|
def centrality_bonus(self, node_id: str) -> float:
|
||||||
|
"""
|
||||||
|
Einfache log-gedämpfte Zentralität auf Basis des In-Degree.
|
||||||
|
Obergrenze: 0.15
|
||||||
|
"""
|
||||||
|
import math
|
||||||
|
|
||||||
|
indeg = self.in_degree.get(node_id, 0)
|
||||||
|
if indeg <= 0:
|
||||||
|
return 0.0
|
||||||
|
return min(math.log1p(indeg) / 10.0, 0.15)
|
||||||
|
|
||||||
|
|
||||||
|
def expand(
|
||||||
|
client: QdrantClient,
|
||||||
|
prefix: str,
|
||||||
|
seeds: List[str],
|
||||||
|
depth: int = 1,
|
||||||
|
edge_types: Optional[List[str]] = None,
|
||||||
|
) -> Subgraph:
|
||||||
|
"""
|
||||||
|
Expandiert ab Seeds entlang von Edges (bis `depth`), optional gefiltert
|
||||||
|
nach Edge-Typen.
|
||||||
|
|
||||||
|
Seeds sind stabile payload-IDs (in unserem Fall vor allem note_id).
|
||||||
|
Es werden Edges berücksichtigt, bei denen source_id ODER target_id ODER
|
||||||
|
note_id einem der Seeds entspricht.
|
||||||
|
"""
|
||||||
|
sg = Subgraph()
|
||||||
|
frontier = set(seeds)
|
||||||
|
visited = set()
|
||||||
|
|
||||||
|
max_depth = max(depth, 0)
|
||||||
|
|
||||||
|
for _ in range(max_depth):
|
||||||
|
if not frontier:
|
||||||
|
break
|
||||||
|
|
||||||
|
edges_payloads = _fetch_edges(
|
||||||
|
client=client,
|
||||||
|
prefix=prefix,
|
||||||
|
seeds=list(frontier),
|
||||||
|
edge_types=edge_types,
|
||||||
|
limit=2048,
|
||||||
|
)
|
||||||
|
|
||||||
|
next_frontier = set()
|
||||||
|
for pl in edges_payloads:
|
||||||
|
src = pl.get("source_id")
|
||||||
|
tgt = pl.get("target_id")
|
||||||
|
kind = pl.get("kind", "edge")
|
||||||
|
owner = pl.get("note_id")
|
||||||
|
|
||||||
|
e = {
|
||||||
|
"source": src,
|
||||||
|
"target": tgt,
|
||||||
|
"kind": kind,
|
||||||
|
"weight": _edge_weight(pl),
|
||||||
|
"note_id": owner,
|
||||||
|
}
|
||||||
|
sg.add_edge(e)
|
||||||
|
|
||||||
|
if tgt:
|
||||||
|
next_frontier.add(tgt)
|
||||||
|
|
||||||
|
visited |= frontier
|
||||||
|
frontier = next_frontier - visited
|
||||||
|
|
||||||
|
return sg
|
||||||
+194
-166
@@ -2,197 +2,225 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Modul: app/core/note_payload.py
|
Modul: app/core/note_payload.py
|
||||||
Version: 1.4.0
|
Version: 2.0.0
|
||||||
Datum: 2025-09-09
|
|
||||||
|
|
||||||
Kurzbeschreibung
|
Zweck
|
||||||
----------------
|
-----
|
||||||
Erzeugt den Qdrant-Payload für **Notes**. Neben stabiler Hash-Bildung zur
|
Erzeugt ein robustes Note-Payload. Werte wie `retriever_weight`, `chunk_profile`
|
||||||
Idempotenz wird der vollständige Body unter ``fulltext`` persistiert und
|
und `edge_defaults` werden in folgender Priorität bestimmt:
|
||||||
der Dateipfad relativ zum Vault gespeichert. Das erlaubt eine verlustfreie
|
1) Frontmatter (Note)
|
||||||
Rekonstruktion im Export (erst ``fulltext``, sonst Chunks).
|
2) Typ-Registry (config/types.yaml: types.<type>.*)
|
||||||
|
3) Registry-Defaults (config/types.yaml: defaults.*)
|
||||||
Hash-Steuerung
|
4) ENV-Defaults (MINDNET_DEFAULT_RETRIEVER_WEIGHT / MINDNET_DEFAULT_CHUNK_PROFILE)
|
||||||
--------------
|
|
||||||
- Modus: ``body`` (Default) | ``frontmatter`` | ``body+frontmatter``
|
|
||||||
* per Funktionsparameter ``hash_mode`` (prio 1)
|
|
||||||
* oder ENV ``MINDNET_HASH_MODE`` (prio 2)
|
|
||||||
- Normalisierung: ``canonical`` (Default) | ``none``
|
|
||||||
* per Funktionsparameter ``hash_normalize`` (prio 1)
|
|
||||||
* oder ENV ``MINDNET_HASH_NORMALIZE`` (prio 2)
|
|
||||||
|
|
||||||
Beispiele (CLI – Sichtprüfung)
|
|
||||||
------------------------------
|
|
||||||
python3 -m app.core.note_payload --from-file ./vault/demo.md --vault-root ./vault --print
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import argparse
|
|
||||||
import hashlib
|
from typing import Any, Dict, Tuple, Optional
|
||||||
import json
|
|
||||||
import os
|
import os
|
||||||
from typing import Any, Dict, Optional
|
import json
|
||||||
|
import pathlib
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from app.core.parser import read_markdown, extract_wikilinks
|
import yaml # type: ignore
|
||||||
except Exception: # pragma: no cover
|
except Exception:
|
||||||
from .parser import read_markdown, extract_wikilinks # type: ignore
|
yaml = None
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Hashing
|
# Helper
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def _canon_frontmatter(fm: Dict[str, Any]) -> str:
|
def _as_dict(x) -> Dict[str, Any]:
|
||||||
"""Kanonische, stabile JSON-Serialisierung der Frontmatter für Hashbildung."""
|
"""Versucht, ein ParsedMarkdown-ähnliches Objekt in ein Dict zu überführen."""
|
||||||
return json.dumps(fm or {}, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
|
if isinstance(x, dict):
|
||||||
|
return dict(x)
|
||||||
|
|
||||||
def _normalize_body(body: str, mode: str) -> str:
|
out: Dict[str, Any] = {}
|
||||||
"""Normalisiert den Body für reproduzierbare Hashes (oder nicht)."""
|
# bekannte Attribute übernehmen, sofern vorhanden
|
||||||
if mode == "none":
|
for attr in (
|
||||||
return body if body is not None else ""
|
"frontmatter",
|
||||||
# canonical: \r\n→\n, trailing spaces, mehrfach-blankzeilen trimmen
|
"body",
|
||||||
text = (body or "").replace("\r\n", "\n").replace("\r", "\n")
|
"id",
|
||||||
# trailing whitespace am Zeilenende entfernen
|
"note_id",
|
||||||
text = "\n".join(line.rstrip() for line in text.split("\n"))
|
"title",
|
||||||
# keine aggressivere Logik (keine inhaltliche Änderung)
|
"path",
|
||||||
return text
|
"tags",
|
||||||
|
"type",
|
||||||
|
"created",
|
||||||
|
"modified",
|
||||||
|
"date",
|
||||||
|
):
|
||||||
|
if hasattr(x, attr):
|
||||||
|
val = getattr(x, attr)
|
||||||
|
if val is not None:
|
||||||
|
out[attr] = val
|
||||||
|
|
||||||
def compute_hash(
|
# Fallback: wenn immer noch leer, raw speichern
|
||||||
*,
|
if not out:
|
||||||
body: Optional[str],
|
out["raw"] = str(x)
|
||||||
frontmatter: Optional[Dict[str, Any]],
|
|
||||||
mode: Optional[str] = None,
|
|
||||||
normalize: Optional[str] = None,
|
|
||||||
) -> str:
|
|
||||||
"""
|
|
||||||
Berechnet einen Hex-Hash gemäß ``mode`` und ``normalize``.
|
|
||||||
|
|
||||||
mode:
|
return out
|
||||||
- "body" (Default)
|
|
||||||
- "frontmatter"
|
|
||||||
- "body+frontmatter"
|
|
||||||
normalize:
|
|
||||||
- "canonical" (Default)
|
|
||||||
- "none"
|
|
||||||
"""
|
|
||||||
mode = (mode or os.environ.get("MINDNET_HASH_MODE", "body")).strip().lower()
|
|
||||||
normalize = (normalize or os.environ.get("MINDNET_HASH_NORMALIZE", "canonical")).strip().lower()
|
|
||||||
|
|
||||||
body = _normalize_body(body or "", normalize)
|
|
||||||
fm_s = _canon_frontmatter(frontmatter or {})
|
|
||||||
|
|
||||||
h = hashlib.sha256()
|
|
||||||
if mode == "frontmatter":
|
|
||||||
h.update(fm_s.encode("utf-8"))
|
|
||||||
elif mode == "body+frontmatter":
|
|
||||||
h.update(body.encode("utf-8"))
|
|
||||||
h.update(b"\n--FM--\n")
|
|
||||||
h.update(fm_s.encode("utf-8"))
|
|
||||||
else: # body
|
|
||||||
h.update(body.encode("utf-8"))
|
|
||||||
return h.hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
def _pick_args(*args, **kwargs) -> Tuple[Optional[str], Optional[dict]]:
|
||||||
# Kernfunktion
|
"""Extrahiert optionale Zusatzargumente wie path und types_cfg."""
|
||||||
# ---------------------------------------------------------------------------
|
path = kwargs.get("path") or (args[0] if args else None)
|
||||||
|
types_cfg = kwargs.get("types_cfg") or kwargs.get("types") or None
|
||||||
|
return path, types_cfg
|
||||||
|
|
||||||
def make_note_payload(
|
|
||||||
parsed: Any,
|
|
||||||
vault_root: Optional[str] = None,
|
|
||||||
*,
|
|
||||||
hash_mode: Optional[str] = None,
|
|
||||||
hash_normalize: Optional[str] = None,
|
|
||||||
) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Erzeugt den Payload für eine geparste Note.
|
|
||||||
|
|
||||||
Parameters
|
def _env_float(name: str, default: float) -> float:
|
||||||
----------
|
"""Liest einen Float-Wert aus der Umgebung, mit robustem Fallback."""
|
||||||
parsed : Any
|
|
||||||
Objekt mit Attributen/Keys ``frontmatter``, ``body``, ``path``.
|
|
||||||
vault_root : Optional[str]
|
|
||||||
Vault-Wurzel (für Pfad-Relativierung). Wenn ``None``, wird ``path`` unverändert übernommen.
|
|
||||||
hash_mode : Optional[str]
|
|
||||||
"body" | "frontmatter" | "body+frontmatter" (überschreibt ENV).
|
|
||||||
hash_normalize : Optional[str]
|
|
||||||
"canonical" | "none" (überschreibt ENV).
|
|
||||||
|
|
||||||
Returns
|
|
||||||
-------
|
|
||||||
Dict[str, Any]
|
|
||||||
Qdrant-Payload für die Notes-Collection.
|
|
||||||
"""
|
|
||||||
# "Duck typing": dict oder Objekt akzeptieren
|
|
||||||
fm = (
|
|
||||||
getattr(parsed, "frontmatter", None)
|
|
||||||
or getattr(parsed, "fm", None)
|
|
||||||
or getattr(parsed, "front_matter", None)
|
|
||||||
or (parsed.get("frontmatter") if isinstance(parsed, dict) else {})
|
|
||||||
) or {}
|
|
||||||
body = getattr(parsed, "body", None) or (parsed.get("body") if isinstance(parsed, dict) else "") or ""
|
|
||||||
path = getattr(parsed, "path", None) or (parsed.get("path") if isinstance(parsed, dict) else "") or ""
|
|
||||||
|
|
||||||
# Hash gem. Modus bilden
|
|
||||||
hash_fulltext = compute_hash(body=body, frontmatter=fm, mode=hash_mode, normalize=hash_normalize)
|
|
||||||
|
|
||||||
# Pfad relativieren
|
|
||||||
rel_path = path
|
|
||||||
try:
|
try:
|
||||||
if vault_root:
|
return float(os.environ.get(name, default))
|
||||||
rel = os.path.relpath(path, vault_root)
|
|
||||||
rel = rel.replace("\\", "/").lstrip("/") # normalisieren
|
|
||||||
rel_path = rel
|
|
||||||
except Exception:
|
except Exception:
|
||||||
pass # Pfad nicht kritisch für Hash/ID
|
return default
|
||||||
|
|
||||||
# Optionale Note-Level-Wikilinks (Fallback, wenn Chunks nicht geliefert werden)
|
|
||||||
note_level_refs = list(dict.fromkeys(extract_wikilinks(body))) if body else []
|
def _ensure_list(x) -> list:
|
||||||
|
"""Garantiert eine String-Liste."""
|
||||||
|
if x is None:
|
||||||
|
return []
|
||||||
|
if isinstance(x, list):
|
||||||
|
return [str(i) for i in x]
|
||||||
|
if isinstance(x, (set, tuple)):
|
||||||
|
return [str(i) for i in x]
|
||||||
|
return [str(x)]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Type-Registry laden
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _load_types_config(explicit_cfg: Optional[dict] = None) -> dict:
|
||||||
|
"""Lädt die Type-Registry aus YAML/JSON oder nutzt ein explizit übergebenes Dict."""
|
||||||
|
if explicit_cfg and isinstance(explicit_cfg, dict):
|
||||||
|
return explicit_cfg
|
||||||
|
|
||||||
|
path = os.getenv("MINDNET_TYPES_FILE") or "./config/types.yaml"
|
||||||
|
if not os.path.isfile(path) or yaml is None:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
return data if isinstance(data, dict) else {}
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg_for_type(note_type: str, reg: dict) -> dict:
|
||||||
|
"""Liefert die Konfiguration für einen konkreten Notiztyp aus der Registry."""
|
||||||
|
if not isinstance(reg, dict):
|
||||||
|
return {}
|
||||||
|
types = reg.get("types") if isinstance(reg.get("types"), dict) else reg
|
||||||
|
return types.get(note_type, {}) if isinstance(types, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg_defaults(reg: dict) -> dict:
|
||||||
|
"""Liefert den Default-Block aus der Registry (defaults/global)."""
|
||||||
|
if not isinstance(reg, dict):
|
||||||
|
return {}
|
||||||
|
for key in ("defaults", "default", "global"):
|
||||||
|
v = reg.get(key)
|
||||||
|
if isinstance(v, dict):
|
||||||
|
return v
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Haupt-API
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def make_note_payload(note: Any, *args, **kwargs) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Baut das Note-Payload für mindnet_notes auf.
|
||||||
|
|
||||||
|
Erwartete Felder im Payload:
|
||||||
|
- note_id: stabile ID aus Frontmatter (id) oder Note-Objekt
|
||||||
|
- title: Titel der Notiz
|
||||||
|
- type: Notiztyp (z. B. concept, project, journal, ...)
|
||||||
|
- path: Pfad im Vault
|
||||||
|
- retriever_weight: effektives Gewicht für den Retriever
|
||||||
|
- chunk_profile: Profil für Chunking (short|medium|long|default|...)
|
||||||
|
- edge_defaults: Liste von Kanten-Typen, die als Defaults gelten
|
||||||
|
"""
|
||||||
|
n = _as_dict(note)
|
||||||
|
path_arg, types_cfg_explicit = _pick_args(*args, **kwargs)
|
||||||
|
reg = _load_types_config(types_cfg_explicit)
|
||||||
|
|
||||||
|
fm = n.get("frontmatter") or {}
|
||||||
|
fm_type = fm.get("type") or n.get("type") or "concept"
|
||||||
|
note_type = str(fm_type)
|
||||||
|
|
||||||
|
cfg_type = _cfg_for_type(note_type, reg)
|
||||||
|
cfg_def = _cfg_defaults(reg)
|
||||||
|
|
||||||
|
# --- retriever_weight: Frontmatter > Typ-Config > Registry-Defaults > ENV ---
|
||||||
|
default_rw = _env_float("MINDNET_DEFAULT_RETRIEVER_WEIGHT", 1.0)
|
||||||
|
retriever_weight = fm.get("retriever_weight")
|
||||||
|
if retriever_weight is None:
|
||||||
|
retriever_weight = cfg_type.get(
|
||||||
|
"retriever_weight",
|
||||||
|
cfg_def.get("retriever_weight", default_rw),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
retriever_weight = float(retriever_weight)
|
||||||
|
except Exception:
|
||||||
|
retriever_weight = default_rw
|
||||||
|
|
||||||
|
# --- chunk_profile: Frontmatter > Typ-Config > Registry-Defaults > ENV ---
|
||||||
|
chunk_profile = fm.get("chunk_profile")
|
||||||
|
if chunk_profile is None:
|
||||||
|
chunk_profile = cfg_type.get(
|
||||||
|
"chunk_profile",
|
||||||
|
cfg_def.get(
|
||||||
|
"chunk_profile",
|
||||||
|
os.environ.get("MINDNET_DEFAULT_CHUNK_PROFILE", "medium"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if not isinstance(chunk_profile, str):
|
||||||
|
chunk_profile = "medium"
|
||||||
|
|
||||||
|
# --- edge_defaults: Frontmatter > Typ-Config > Registry-Defaults ---
|
||||||
|
edge_defaults = fm.get("edge_defaults")
|
||||||
|
if edge_defaults is None:
|
||||||
|
edge_defaults = cfg_type.get(
|
||||||
|
"edge_defaults",
|
||||||
|
cfg_def.get("edge_defaults", []),
|
||||||
|
)
|
||||||
|
edge_defaults = _ensure_list(edge_defaults)
|
||||||
|
|
||||||
|
# --- Basis-Metadaten (IDs, Titel, Pfad) ---
|
||||||
|
note_id = n.get("note_id") or n.get("id") or fm.get("id")
|
||||||
|
title = n.get("title") or fm.get("title") or ""
|
||||||
|
path = n.get("path") or path_arg
|
||||||
|
if isinstance(path, pathlib.Path):
|
||||||
|
path = str(path)
|
||||||
|
|
||||||
payload: Dict[str, Any] = {
|
payload: Dict[str, Any] = {
|
||||||
"note_id": fm.get("id") or fm.get("note_id"),
|
"note_id": note_id,
|
||||||
"title": fm.get("title"),
|
"title": title,
|
||||||
"type": fm.get("type"),
|
"type": note_type,
|
||||||
"status": fm.get("status"),
|
"path": path or "",
|
||||||
"created": fm.get("created"),
|
"retriever_weight": retriever_weight,
|
||||||
"updated": fm.get("updated"),
|
"chunk_profile": chunk_profile,
|
||||||
"path": rel_path or fm.get("path"),
|
"edge_defaults": edge_defaults,
|
||||||
"tags": fm.get("tags"),
|
|
||||||
"hash_fulltext": hash_fulltext,
|
|
||||||
# Volltext persistieren (verlustfreie Rekonstruktion)
|
|
||||||
"fulltext": body,
|
|
||||||
# Fallback-Refs auf Note-Ebene
|
|
||||||
"references": note_level_refs,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for k in ("area", "project", "source", "lang", "slug"):
|
# Tags / Keywords übernehmen
|
||||||
if k in fm:
|
tags = fm.get("tags") or fm.get("keywords") or n.get("tags")
|
||||||
payload[k] = fm[k]
|
if tags:
|
||||||
|
payload["tags"] = _ensure_list(tags)
|
||||||
|
|
||||||
|
# Zeitliche Metadaten (sofern vorhanden)
|
||||||
|
for k in ("created", "modified", "date"):
|
||||||
|
v = fm.get(k) or n.get(k)
|
||||||
|
if v:
|
||||||
|
payload[k] = str(v)
|
||||||
|
|
||||||
|
# JSON-Roundtrip zur harten Validierung (ASCII beibehalten)
|
||||||
|
json.loads(json.dumps(payload, ensure_ascii=False))
|
||||||
|
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# CLI – Sichtprüfung
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _cli() -> None:
|
|
||||||
ap = argparse.ArgumentParser(description="Note-Payload aus Markdown erzeugen und anzeigen")
|
|
||||||
ap.add_argument("--from-file", dest="src", required=True, help="Pfad zur Markdown-Datei")
|
|
||||||
ap.add_argument("--vault-root", dest="vault_root", default=None, help="Vault-Wurzel zur Pfad-Relativierung")
|
|
||||||
ap.add_argument("--print", dest="do_print", action="store_true", help="Payload auf stdout ausgeben")
|
|
||||||
ap.add_argument("--hash-mode", choices=["body", "frontmatter", "body+frontmatter"], default=None)
|
|
||||||
ap.add_argument("--hash-normalize", choices=["canonical", "none"], default=None)
|
|
||||||
args = ap.parse_args()
|
|
||||||
|
|
||||||
parsed = read_markdown(args.src)
|
|
||||||
payload = make_note_payload(parsed, vault_root=args.vault_root,
|
|
||||||
hash_mode=args.hash_mode, hash_normalize=args.hash_normalize)
|
|
||||||
|
|
||||||
if args.do_print:
|
|
||||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__": # pragma: no cover
|
|
||||||
_cli()
|
|
||||||
+244
-45
@@ -1,67 +1,266 @@
|
|||||||
from __future__ import annotations
|
#!/usr/bin/env python3
|
||||||
from dataclasses import dataclass
|
# -*- coding: utf-8 -*-
|
||||||
import re
|
"""
|
||||||
import os
|
Modul: app/core/parser.py
|
||||||
import unicodedata
|
Version: 1.7.1 (fault-tolerant, API-kompatibel)
|
||||||
import yaml
|
Datum: 2025-10-01
|
||||||
from typing import Tuple, Dict
|
|
||||||
|
|
||||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n?", re.DOTALL) # YAML-Frontmatter am Anfang
|
Zweck
|
||||||
|
-----
|
||||||
|
Fehlertolerantes Einlesen von Markdown-Dateien mit YAML-Frontmatter.
|
||||||
|
Kompatibel zur bisherigen Parser-API, aber robust gegenüber Nicht-UTF-8-Dateien:
|
||||||
|
- Versucht nacheinander: utf-8 → utf-8-sig → cp1252 → latin-1.
|
||||||
|
- Bei Fallback wird ein JSON-Warnhinweis auf stdout ausgegeben; der Import bricht NICHT ab.
|
||||||
|
- YAML-Frontmatter wird mit '---' am Anfang und '---' als Abschluss erkannt.
|
||||||
|
- extract_wikilinks() normalisiert [[id#anchor|label]] → 'id'.
|
||||||
|
|
||||||
|
Öffentliche API (kompatibel):
|
||||||
|
- class ParsedNote(frontmatter: dict, body: str, path: str)
|
||||||
|
- read_markdown(path) -> ParsedNote | None
|
||||||
|
- normalize_frontmatter(fm) -> dict
|
||||||
|
- validate_required_frontmatter(fm, required: tuple[str,...]=("id","title")) -> None
|
||||||
|
- extract_wikilinks(text) -> list[str]
|
||||||
|
- FRONTMATTER_RE (Kompatibilitäts-Konstante; Regex für '---'-Zeilen)
|
||||||
|
|
||||||
|
Beispiele
|
||||||
|
---------
|
||||||
|
from app.core.parser import read_markdown, normalize_frontmatter, validate_required_frontmatter
|
||||||
|
parsed = read_markdown("./vault/30_projects/project-demo.md")
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter)
|
||||||
|
validate_required_frontmatter(fm)
|
||||||
|
body = parsed.body
|
||||||
|
|
||||||
|
from app.core.parser import extract_wikilinks
|
||||||
|
links = extract_wikilinks(body)
|
||||||
|
|
||||||
|
Abhängigkeiten
|
||||||
|
--------------
|
||||||
|
- PyYAML (yaml)
|
||||||
|
|
||||||
|
Lizenz: MIT (projektintern)
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any, Dict, Optional, Tuple, Iterable, List
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml # PyYAML
|
||||||
|
except Exception as e: # pragma: no cover
|
||||||
|
yaml = None # Fehler wird zur Laufzeit geworfen, falls wirklich benötigt
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Datamodell
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class ParsedNote:
|
class ParsedNote:
|
||||||
frontmatter: Dict
|
frontmatter: Dict[str, Any]
|
||||||
body: str
|
body: str
|
||||||
path: str
|
path: str
|
||||||
|
|
||||||
def _strip_bom(text: str) -> str:
|
|
||||||
return text.lstrip("\ufeff")
|
|
||||||
|
|
||||||
def _normalize_text(t: str) -> str:
|
# ---------------------------------------------------------------------
|
||||||
# Unicode-NFKC + vereinheitlichte Zeilenenden + Trim
|
# Frontmatter-Erkennung
|
||||||
t = unicodedata.normalize("NFKC", t)
|
# ---------------------------------------------------------------------
|
||||||
t = t.replace("\r\n", "\n").replace("\r", "\n")
|
|
||||||
return t
|
|
||||||
|
|
||||||
def read_markdown(path: str) -> ParsedNote:
|
# Öffentliche Kompatibilitäts-Konstante: frühere Skripte importieren FRONTMATTER_RE
|
||||||
with open(path, "r", encoding="utf-8") as f:
|
FRONTMATTER_RE = re.compile(r"^\s*---\s*$") # <— public
|
||||||
raw = _strip_bom(f.read())
|
# Zusätzlich interner Alias (falls jemand ihn referenziert)
|
||||||
raw = _normalize_text(raw)
|
FRONTMATTER_END = FRONTMATTER_RE # <— public alias
|
||||||
|
|
||||||
m = FRONTMATTER_RE.match(raw)
|
# interne Namen bleiben bestehen
|
||||||
front, body = {}, raw
|
_FRONTMATTER_HEAD = FRONTMATTER_RE
|
||||||
if m:
|
_FRONTMATTER_END = FRONTMATTER_RE
|
||||||
yaml_block = m.group(1)
|
|
||||||
body = raw[m.end():]
|
|
||||||
|
def _split_frontmatter(text: str) -> Tuple[Dict[str, Any], str]:
|
||||||
|
"""
|
||||||
|
Zerlegt Text in (frontmatter: dict, body: str).
|
||||||
|
Erkennt Frontmatter nur, wenn die erste Zeile '---' ist und später ein zweites '---' folgt.
|
||||||
|
YAML-Fehler im Frontmatter führen NICHT zum Abbruch: es wird dann ein leeres dict benutzt.
|
||||||
|
"""
|
||||||
|
lines = text.splitlines(True) # keep line endings
|
||||||
|
if not lines:
|
||||||
|
return {}, ""
|
||||||
|
|
||||||
|
if not _FRONTMATTER_HEAD.match(lines[0]):
|
||||||
|
# kein Frontmatter-Header → gesamter Text ist Body
|
||||||
|
return {}, text
|
||||||
|
|
||||||
|
end_idx = None
|
||||||
|
# Suche nach nächstem '---' (max. 2000 Zeilen als Sicherheitslimit)
|
||||||
|
for i in range(1, min(len(lines), 2000)):
|
||||||
|
if _FRONTMATTER_END.match(lines[i]):
|
||||||
|
end_idx = i
|
||||||
|
break
|
||||||
|
|
||||||
|
if end_idx is None:
|
||||||
|
# unvollständiger Frontmatter-Block → behandle alles als Body
|
||||||
|
return {}, text
|
||||||
|
|
||||||
|
fm_raw = "".join(lines[1:end_idx])
|
||||||
|
body = "".join(lines[end_idx + 1:])
|
||||||
|
|
||||||
|
data: Dict[str, Any] = {}
|
||||||
|
if yaml is None:
|
||||||
|
raise RuntimeError("PyYAML ist nicht installiert (pip install pyyaml).")
|
||||||
|
|
||||||
|
try:
|
||||||
|
loaded = yaml.safe_load(fm_raw) or {}
|
||||||
|
if isinstance(loaded, dict):
|
||||||
|
data = loaded
|
||||||
|
else:
|
||||||
|
data = {}
|
||||||
|
except Exception as e:
|
||||||
|
# YAML-Fehler nicht fatal machen
|
||||||
|
print(json.dumps({"warn": "frontmatter_yaml_parse_failed", "error": str(e)}))
|
||||||
|
data = {}
|
||||||
|
|
||||||
|
# optionales kosmetisches Trim: eine führende Leerzeile im Body entfernen
|
||||||
|
if body.startswith("\n"):
|
||||||
|
body = body[1:]
|
||||||
|
|
||||||
|
return data, body
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Robustes Lesen mit Encoding-Fallback
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
_FALLBACK_ENCODINGS: Tuple[str, ...] = ("utf-8", "utf-8-sig", "cp1252", "latin-1")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_text_with_fallback(path: str) -> Tuple[str, str, bool]:
|
||||||
|
"""
|
||||||
|
Liest Datei mit mehreren Decodierungsversuchen.
|
||||||
|
Rückgabe: (text, used_encoding, had_fallback)
|
||||||
|
- had_fallback=True, falls NICHT 'utf-8' verwendet wurde (oder 'utf-8-sig').
|
||||||
|
"""
|
||||||
|
last_err: Optional[str] = None
|
||||||
|
for enc in _FALLBACK_ENCODINGS:
|
||||||
try:
|
try:
|
||||||
front = yaml.safe_load(yaml_block) or {}
|
with io.open(path, "r", encoding=enc, errors="strict") as f:
|
||||||
if not isinstance(front, dict):
|
text = f.read()
|
||||||
raise ValueError("Frontmatter must be a mapping")
|
# 'utf-8-sig' zählt hier als Fallback (weil BOM), aber ist unproblematisch
|
||||||
except yaml.YAMLError as e:
|
return text, enc, (enc != "utf-8")
|
||||||
raise ValueError(f"Invalid YAML frontmatter in {path}: {e}") from e
|
except UnicodeDecodeError as e:
|
||||||
|
last_err = f"{type(e).__name__}: {e}"
|
||||||
|
continue
|
||||||
|
|
||||||
return ParsedNote(frontmatter=front, body=body, path=path)
|
# Letzter, extrem defensiver Fallback: Bytes → UTF-8 mit REPLACE (keine Exception)
|
||||||
|
with open(path, "rb") as fb:
|
||||||
|
raw = fb.read()
|
||||||
|
text = raw.decode("utf-8", errors="replace")
|
||||||
|
print(json.dumps({
|
||||||
|
"path": path,
|
||||||
|
"warn": "encoding_fallback_exhausted",
|
||||||
|
"info": last_err or "unknown"
|
||||||
|
}, ensure_ascii=False))
|
||||||
|
return text, "utf-8(replace)", True
|
||||||
|
|
||||||
RE_WIKILINK = re.compile(r"\[\[([^\]\|#]+)(?:#[^\]]+)?(?:\|[^\]]+)?\]\]") # [[id]] | [[id#anchor]] | [[id|label]]
|
|
||||||
|
|
||||||
def extract_wikilinks(text: str) -> list[str]:
|
# ---------------------------------------------------------------------
|
||||||
return list({m.group(1).strip() for m in RE_WIKILINK.finditer(text)})
|
# Öffentliche API
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
def validate_required_frontmatter(fm: dict, required=("title","id","type","status","created")):
|
def read_markdown(path: str) -> Optional[ParsedNote]:
|
||||||
missing = [k for k in required if k not in fm or fm[k] in (None, "")]
|
"""
|
||||||
|
Liest eine Markdown-Datei fehlertolerant:
|
||||||
|
- Erlaubt verschiedene Encodings (UTF-8 bevorzugt, cp1252/latin-1 als Fallback).
|
||||||
|
- Schlägt NICHT mit UnicodeDecodeError fehl.
|
||||||
|
- Gibt ParsedNote(frontmatter, body, path) zurück oder None, falls die Datei nicht existiert.
|
||||||
|
|
||||||
|
Bei Decoding-Fallback wird ein JSON-Warnhinweis geloggt:
|
||||||
|
{"path": "...", "warn": "encoding_fallback_used", "used": "cp1252"}
|
||||||
|
"""
|
||||||
|
if not os.path.exists(path):
|
||||||
|
return None
|
||||||
|
|
||||||
|
text, enc, had_fb = _read_text_with_fallback(path)
|
||||||
|
if had_fb:
|
||||||
|
print(json.dumps({"path": path, "warn": "encoding_fallback_used", "used": enc}, ensure_ascii=False))
|
||||||
|
|
||||||
|
fm, body = _split_frontmatter(text)
|
||||||
|
return ParsedNote(frontmatter=fm or {}, body=body or "", path=path)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_required_frontmatter(fm: Dict[str, Any],
|
||||||
|
required: Tuple[str, ...] = ("id", "title")) -> None:
|
||||||
|
"""
|
||||||
|
Prüft, ob alle Pflichtfelder vorhanden sind.
|
||||||
|
Default-kompatibel: ('id', 'title'), kann aber vom Aufrufer erweitert werden, z. B.:
|
||||||
|
validate_required_frontmatter(fm, required=("id","title","type","status","created"))
|
||||||
|
|
||||||
|
Hebt ValueError, falls Felder fehlen oder leer sind.
|
||||||
|
"""
|
||||||
|
if fm is None:
|
||||||
|
fm = {}
|
||||||
|
missing = []
|
||||||
|
for k in required:
|
||||||
|
v = fm.get(k)
|
||||||
|
if v is None:
|
||||||
|
missing.append(k)
|
||||||
|
elif isinstance(v, str) and not v.strip():
|
||||||
|
missing.append(k)
|
||||||
if missing:
|
if missing:
|
||||||
raise ValueError(f"Missing required frontmatter fields: {', '.join(missing)}")
|
raise ValueError(f"Missing required frontmatter fields: {', '.join(missing)}")
|
||||||
|
|
||||||
# einfache Plausibilitäten
|
# Plausibilitäten: 'tags' sollte eine Liste sein, wenn vorhanden
|
||||||
if not isinstance(fm.get("tags", []), (list, tuple)):
|
if "tags" in fm and fm["tags"] not in (None, "") and not isinstance(fm["tags"], (list, tuple)):
|
||||||
if "tags" in fm and fm["tags"] not in (None, ""):
|
raise ValueError("frontmatter 'tags' must be a list of strings")
|
||||||
raise ValueError("tags must be a list of strings")
|
|
||||||
|
|
||||||
def normalize_frontmatter(fm: dict) -> dict:
|
|
||||||
# kleinere Normalisierungen ohne die Semantik zu verändern
|
def normalize_frontmatter(fm: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
out = dict(fm)
|
"""
|
||||||
if "tags" in out and isinstance(out["tags"], list):
|
Sanfte Normalisierung ohne Semantikänderung:
|
||||||
out["tags"] = [str(t).strip() for t in out["tags"]]
|
- 'tags' → Liste von Strings (Trim)
|
||||||
|
- 'embedding_exclude' → bool
|
||||||
|
- andere Felder unverändert
|
||||||
|
"""
|
||||||
|
out = dict(fm or {})
|
||||||
|
if "tags" in out:
|
||||||
|
if isinstance(out["tags"], str):
|
||||||
|
out["tags"] = [out["tags"].strip()] if out["tags"].strip() else []
|
||||||
|
elif isinstance(out["tags"], list):
|
||||||
|
out["tags"] = [str(t).strip() for t in out["tags"] if t is not None]
|
||||||
|
else:
|
||||||
|
out["tags"] = [str(out["tags"]).strip()] if out["tags"] not in (None, "") else []
|
||||||
if "embedding_exclude" in out:
|
if "embedding_exclude" in out:
|
||||||
out["embedding_exclude"] = bool(out["embedding_exclude"])
|
out["embedding_exclude"] = bool(out["embedding_exclude"])
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------ Wikilinks ---------------------------- #
|
||||||
|
|
||||||
|
# Basismuster für [[...]]; die Normalisierung (id vor '#', vor '|') macht extract_wikilinks
|
||||||
|
_WIKILINK_RE = re.compile(r"\[\[([^\]]+)\]\]")
|
||||||
|
|
||||||
|
|
||||||
|
def extract_wikilinks(text: str) -> List[str]:
|
||||||
|
"""
|
||||||
|
Extrahiert Wikilinks wie [[id]], [[id#anchor]], [[id|label]], [[id#anchor|label]].
|
||||||
|
Rückgabe sind NUR die Ziel-IDs (ohne Anchor/Label), führend/folgend getrimmt.
|
||||||
|
Keine aggressive Slug-Normalisierung (die kann später im Resolver erfolgen).
|
||||||
|
"""
|
||||||
|
if not text:
|
||||||
|
return []
|
||||||
|
out: List[str] = []
|
||||||
|
for m in _WIKILINK_RE.finditer(text):
|
||||||
|
raw = (m.group(1) or "").strip()
|
||||||
|
if not raw:
|
||||||
|
continue
|
||||||
|
# Split an Pipe (Label) → links vor '|'
|
||||||
|
if "|" in raw:
|
||||||
|
raw = raw.split("|", 1)[0].strip()
|
||||||
|
# Split an Anchor
|
||||||
|
if "#" in raw:
|
||||||
|
raw = raw.split("#", 1)[0].strip()
|
||||||
|
if raw:
|
||||||
|
out.append(raw)
|
||||||
|
return out
|
||||||
+147
-120
@@ -1,152 +1,179 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Name: app/core/qdrant.py
|
app/core/qdrant.py
|
||||||
Version: v1.3.1 (2025-09-05)
|
Version: 2.2.0 (2025-11-11)
|
||||||
|
|
||||||
Kurzbeschreibung:
|
Aufgabe
|
||||||
Qdrant-Client & Collection-Setup für mindnet.
|
-------
|
||||||
- Stellt sicher, dass {prefix}_notes / {prefix}_chunks / {prefix}_edges existieren.
|
- Zentraler Qdrant-Zugriff (Client, Config)
|
||||||
- Edges-Collection nutzt 1D Dummy-Vektor (Workaround für Python-Client).
|
- Collection-Anlage (notes/chunks/edges)
|
||||||
- **Nicht-destruktiv per Default**: ensure_collections(..., destructive=False).
|
- **Payload-Indizes sicherstellen** (idempotent)
|
||||||
- **Abwärtskompatibel**: collection_names(prefix) wieder verfügbar.
|
|
||||||
|
|
||||||
Aufruf/Verwendung:
|
Hinweis
|
||||||
from app.core.qdrant import QdrantConfig, get_client, ensure_collections, collection_names
|
-------
|
||||||
|
Diese Datei ist als Drop-in-Ersatz gedacht, falls in deinem Projekt noch keine
|
||||||
|
robuste ensure_payload_indexes()-Implementierung vorliegt. Die Signaturen
|
||||||
|
bleiben kompatibel zu scripts.import_markdown und scripts.reset_qdrant.
|
||||||
|
|
||||||
Umgebungsvariablen (optional):
|
API-Notizen
|
||||||
QDRANT_URL | QDRANT_HOST/QDRANT_PORT, QDRANT_API_KEY,
|
-----------
|
||||||
COLLECTION_PREFIX (Default "mindnet"), VECTOR_DIM (Default 384)
|
- Payload-Indizes werden mit `create_payload_index` angelegt.
|
||||||
|
- Typen stammen aus `qdrant_client.http.models.PayloadSchemaType`:
|
||||||
Änderungen:
|
KEYWORD | TEXT | INTEGER | FLOAT | BOOL | GEO | DATETIME
|
||||||
v1.3.1: Helper collection_names(prefix) wiederhergestellt (für reset_qdrant usw.).
|
- Für häufige Filterfelder (note_id, kind, scope, type, tags, ...) legen wir
|
||||||
v1.3.0: ensure_collections(..., destructive=False) – keine stillen Drops im Dry-Run.
|
Indizes an. Das ist laut Qdrant-Doku Best Practice für performante Filter.
|
||||||
Edges-Collection nur bei explicit destructive=True neu anlegen.
|
|
||||||
≤v1.2.x: Konnte {prefix}_edges bei fehlender VectorConfig automatisch neu erstellen (riskant).
|
|
||||||
|
|
||||||
Bezug/Quelle:
|
|
||||||
Alte Core-Variante enthielt collection_names(prefix); diverse Scripts nutzen das weiterhin. :contentReference[oaicite:1]{index=1}
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
import os
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Optional, Tuple
|
from typing import Optional, Tuple, Dict, List
|
||||||
|
|
||||||
from qdrant_client import QdrantClient
|
from qdrant_client import QdrantClient
|
||||||
from qdrant_client.http import models as rest
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
|
||||||
# -------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Konfiguration
|
# Konfiguration
|
||||||
# -------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class QdrantConfig:
|
class QdrantConfig:
|
||||||
url: str
|
host: Optional[str] = None
|
||||||
api_key: Optional[str]
|
port: Optional[int] = None
|
||||||
prefix: str
|
url: Optional[str] = None
|
||||||
dim: int
|
api_key: Optional[str] = None
|
||||||
|
prefix: str = "mindnet"
|
||||||
|
dim: int = 384
|
||||||
|
distance: str = "Cosine" # Cosine | Dot | Euclid
|
||||||
|
on_disk_payload: bool = True
|
||||||
|
|
||||||
@staticmethod
|
@classmethod
|
||||||
def from_env() -> "QdrantConfig":
|
def from_env(cls) -> "QdrantConfig":
|
||||||
url = os.getenv("QDRANT_URL")
|
# Entweder URL ODER Host/Port, API-Key optional
|
||||||
if not url:
|
url = os.getenv("QDRANT_URL") or None
|
||||||
host = os.getenv("QDRANT_HOST", "127.0.0.1")
|
host = os.getenv("QDRANT_HOST") or None
|
||||||
port = int(os.getenv("QDRANT_PORT", "6333"))
|
port = os.getenv("QDRANT_PORT")
|
||||||
url = f"http://{host}:{port}"
|
port = int(port) if port else None
|
||||||
api_key = os.getenv("QDRANT_API_KEY") or None
|
api_key = os.getenv("QDRANT_API_KEY") or None
|
||||||
prefix = os.getenv("COLLECTION_PREFIX", "mindnet")
|
prefix = os.getenv("COLLECTION_PREFIX") or "mindnet"
|
||||||
dim = int(os.getenv("VECTOR_DIM", "384"))
|
dim = int(os.getenv("VECTOR_DIM") or 384)
|
||||||
return QdrantConfig(url=url, api_key=api_key, prefix=prefix, dim=dim)
|
distance = os.getenv("DISTANCE", "Cosine")
|
||||||
|
on_disk_payload = (os.getenv("ON_DISK_PAYLOAD", "true").lower() == "true")
|
||||||
|
return cls(
|
||||||
|
host=host, port=port, url=url, api_key=api_key,
|
||||||
|
prefix=prefix, dim=dim, distance=distance, on_disk_payload=on_disk_payload
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# -------------------------------
|
|
||||||
# Client
|
|
||||||
# -------------------------------
|
|
||||||
|
|
||||||
def get_client(cfg: QdrantConfig) -> QdrantClient:
|
def get_client(cfg: QdrantConfig) -> QdrantClient:
|
||||||
return QdrantClient(url=cfg.url, api_key=cfg.api_key)
|
# QdrantClient akzeptiert entweder url=... oder host/port
|
||||||
|
if cfg.url:
|
||||||
|
return QdrantClient(url=cfg.url, api_key=cfg.api_key, timeout=60.0)
|
||||||
|
return QdrantClient(host=cfg.host or "127.0.0.1", port=cfg.port or 6333, api_key=cfg.api_key, timeout=60.0)
|
||||||
|
|
||||||
|
|
||||||
# -------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Collection-Erzeuger (Hilfsfunktionen)
|
# Collections
|
||||||
# -------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def _create_notes(client: QdrantClient, name: str, dim: int) -> None:
|
|
||||||
if not client.collection_exists(name):
|
|
||||||
client.create_collection(
|
|
||||||
collection_name=name,
|
|
||||||
vectors_config=rest.VectorParams(size=dim, distance=rest.Distance.COSINE),
|
|
||||||
)
|
|
||||||
|
|
||||||
def _create_chunks(client: QdrantClient, name: str, dim: int) -> None:
|
|
||||||
if not client.collection_exists(name):
|
|
||||||
client.create_collection(
|
|
||||||
collection_name=name,
|
|
||||||
vectors_config=rest.VectorParams(size=dim, distance=rest.Distance.COSINE),
|
|
||||||
)
|
|
||||||
|
|
||||||
def _create_edges(client: QdrantClient, name: str) -> None:
|
|
||||||
if not client.collection_exists(name):
|
|
||||||
client.create_collection(
|
|
||||||
collection_name=name,
|
|
||||||
vectors_config=rest.VectorParams(size=1, distance=rest.Distance.DOT), # 1D-Dummy
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# -------------------------------
|
|
||||||
# Public API
|
|
||||||
# -------------------------------
|
|
||||||
|
|
||||||
def ensure_collections(client: QdrantClient, prefix: str, dim: int, destructive: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Stellt sicher, dass die drei Collections existieren.
|
|
||||||
- Default **nicht destruktiv**: vorhandene Collections bleiben unangetastet.
|
|
||||||
- Nur wenn 'destructive=True', wird eine ungeeignete Edges-Collection gelöscht und neu angelegt.
|
|
||||||
"""
|
|
||||||
notes = f"{prefix}_notes"
|
|
||||||
chunks = f"{prefix}_chunks"
|
|
||||||
edges = f"{prefix}_edges"
|
|
||||||
|
|
||||||
_create_notes(client, notes, dim)
|
|
||||||
_create_chunks(client, chunks, dim)
|
|
||||||
|
|
||||||
if client.collection_exists(edges):
|
|
||||||
# Prüfen, ob die Edges-Collection bereits eine Vektorkonfig hat
|
|
||||||
try:
|
|
||||||
info = client.get_collection(edges)
|
|
||||||
vectors_cfg = getattr(getattr(info.result, "config", None), "params", None)
|
|
||||||
has_vectors = getattr(vectors_cfg, "vectors", None) is not None
|
|
||||||
except Exception:
|
|
||||||
# konservativ: nichts anfassen, um Datenverlust zu vermeiden
|
|
||||||
has_vectors = True
|
|
||||||
|
|
||||||
if not has_vectors:
|
|
||||||
if destructive:
|
|
||||||
client.delete_collection(edges)
|
|
||||||
_create_edges(client, edges)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"[ensure_collections] WARN: '{edges}' ohne VectorConfig gefunden; "
|
|
||||||
f"keine destruktive Änderung (destructive=False).",
|
|
||||||
flush=True,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
_create_edges(client, edges)
|
|
||||||
|
|
||||||
|
|
||||||
def collection_names(prefix: str) -> Tuple[str, str, str]:
|
def collection_names(prefix: str) -> Tuple[str, str, str]:
|
||||||
"""
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
Abwärtskompatibler Helper für Scripts:
|
|
||||||
returns (f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges")
|
|
||||||
"""
|
|
||||||
return (f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges")
|
|
||||||
|
|
||||||
|
|
||||||
def wipe_collections(client: QdrantClient, prefix: str) -> None:
|
def _vector_params(dim: int, distance: str) -> rest.VectorParams:
|
||||||
|
# Distance: "Cosine" | "Dot" | "Euclid"
|
||||||
|
dist = getattr(rest.Distance, distance.capitalize(), rest.Distance.COSINE)
|
||||||
|
return rest.VectorParams(size=dim, distance=dist)
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_collections(client: QdrantClient, prefix: str, dim: int) -> None:
|
||||||
|
"""Legt mindnet_notes, mindnet_chunks, mindnet_edges an (falls nicht vorhanden)."""
|
||||||
|
notes, chunks, edges = collection_names(prefix)
|
||||||
|
|
||||||
|
# notes
|
||||||
|
if not client.collection_exists(notes):
|
||||||
|
client.create_collection(
|
||||||
|
collection_name=notes,
|
||||||
|
vectors_config=_vector_params(dim, os.getenv("DISTANCE", "Cosine")),
|
||||||
|
on_disk_payload=True,
|
||||||
|
)
|
||||||
|
# chunks
|
||||||
|
if not client.collection_exists(chunks):
|
||||||
|
client.create_collection(
|
||||||
|
collection_name=chunks,
|
||||||
|
vectors_config=_vector_params(dim, os.getenv("DISTANCE", "Cosine")),
|
||||||
|
on_disk_payload=True,
|
||||||
|
)
|
||||||
|
# edges (Dummy-Vektor, Filter via Payload)
|
||||||
|
if not client.collection_exists(edges):
|
||||||
|
client.create_collection(
|
||||||
|
collection_name=edges,
|
||||||
|
vectors_config=_vector_params(1, "Dot"),
|
||||||
|
on_disk_payload=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Payload-Indizes
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _ensure_index(client: QdrantClient, collection: str, field: str, schema: rest.PayloadSchemaType) -> None:
|
||||||
|
"""Idempotentes Anlegen eines Payload-Indexes für ein Feld."""
|
||||||
|
try:
|
||||||
|
client.create_payload_index(collection_name=collection, field_name=field, field_schema=schema, wait=True)
|
||||||
|
except Exception as e:
|
||||||
|
# Fehler ignorieren, falls Index bereits existiert oder Server "already indexed" meldet.
|
||||||
|
# Für Debugging ggf. Logging ergänzen.
|
||||||
|
_ = e
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_payload_indexes(client: QdrantClient, prefix: str) -> None:
|
||||||
"""
|
"""
|
||||||
Löscht alle drei Collections – nur verwenden, wenn bewusst ein Clean-Rebuild gewünscht ist.
|
Stellt sicher, dass alle benötigten Payload-Indizes existieren.
|
||||||
|
- notes: note_id(KEYWORD), type(KEYWORD), title(TEXT), updated(INTEGER), tags(KEYWORD)
|
||||||
|
- chunks: note_id(KEYWORD), chunk_id(KEYWORD), index(INTEGER), type(KEYWORD), tags(KEYWORD)
|
||||||
|
- edges: note_id(KEYWORD), kind(KEYWORD), scope(KEYWORD), source_id(KEYWORD), target_id(KEYWORD), chunk_id(KEYWORD)
|
||||||
"""
|
"""
|
||||||
for name in collection_names(prefix):
|
notes, chunks, edges = collection_names(prefix)
|
||||||
if client.collection_exists(name):
|
|
||||||
client.delete_collection(name)
|
# NOTES
|
||||||
|
for field, schema in [
|
||||||
|
("note_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("type", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("title", rest.PayloadSchemaType.TEXT),
|
||||||
|
("updated", rest.PayloadSchemaType.INTEGER),
|
||||||
|
("tags", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
]:
|
||||||
|
_ensure_index(client, notes, field, schema)
|
||||||
|
|
||||||
|
# CHUNKS
|
||||||
|
for field, schema in [
|
||||||
|
("note_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("chunk_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("index", rest.PayloadSchemaType.INTEGER),
|
||||||
|
("type", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("tags", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
]:
|
||||||
|
_ensure_index(client, chunks, field, schema)
|
||||||
|
|
||||||
|
# EDGES
|
||||||
|
for field, schema in [
|
||||||
|
("note_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("kind", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("scope", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("source_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("target_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
("chunk_id", rest.PayloadSchemaType.KEYWORD),
|
||||||
|
]:
|
||||||
|
_ensure_index(client, edges, field, schema)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"QdrantConfig",
|
||||||
|
"get_client",
|
||||||
|
"ensure_collections",
|
||||||
|
"ensure_payload_indexes",
|
||||||
|
"collection_names",
|
||||||
|
]
|
||||||
+233
-69
@@ -1,55 +1,38 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
app/core/qdrant_points.py
|
app/core/qdrant_points.py - robust points helpers for Qdrant
|
||||||
|
|
||||||
Zweck
|
- Single source of truth for building PointStruct for notes/chunks/edges
|
||||||
- Gemeinsame Helfer zum Erzeugen von Qdrant-Points für Notes, Chunks und Edges.
|
- Backward-compatible payloads for edges
|
||||||
- Abwärtskompatibel zu altem Edge-Payload-Schema aus edges.py:
|
- Handles both Single-Vector and Named-Vector collections
|
||||||
- alt: {'edge_type','src_id','dst_id', ...}
|
- Deterministic overrides via ENV to avoid auto-detection traps:
|
||||||
- neu: {'kind','source_id','target_id', ...}
|
* NOTES_VECTOR_NAME, CHUNKS_VECTOR_NAME, EDGES_VECTOR_NAME
|
||||||
|
* MINDNET_VECTOR_NAME (fallback)
|
||||||
|
> Set to a concrete name (e.g. "text") to force Named-Vector with that name
|
||||||
|
> Set to "__single__" (or "single") to force Single-Vector
|
||||||
|
|
||||||
Version
|
Version: 1.5.0 (2025-11-08)
|
||||||
- 1.3 (2025-09-08)
|
|
||||||
|
|
||||||
Änderungen (ggü. 1.2)
|
|
||||||
- points_for_edges() akzeptiert jetzt beide Edge-Schemata.
|
|
||||||
- Normalisiert alte Felder auf 'kind' / 'source_id' / 'target_id' und schreibt eine
|
|
||||||
stabile 'edge_id' zurück in die Payload.
|
|
||||||
- Verhindert, dass mehrere Edges dieselbe Point-ID erhalten (Root Cause deiner 1-Edge-Sammlung).
|
|
||||||
|
|
||||||
Aufruf / Verwendung
|
|
||||||
- Wird von Import-/Backfill-Skripten via:
|
|
||||||
from app.core.qdrant_points import points_for_note, points_for_chunks, points_for_edges, upsert_batch
|
|
||||||
eingebunden. Keine CLI.
|
|
||||||
|
|
||||||
Hinweise
|
|
||||||
- Edges bekommen absichtlich einen 1D-Dummy-Vektor [0.0], damit Qdrant das Objekt akzeptiert.
|
|
||||||
- Die Point-IDs werden deterministisch aus stabilen Strings (UUIDv5) abgeleitet.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
import os
|
||||||
import uuid
|
import uuid
|
||||||
from typing import List, Tuple
|
from typing import List, Tuple, Iterable, Optional, Dict, Any
|
||||||
from qdrant_client.http import models as rest
|
|
||||||
|
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
|
||||||
|
# --------------------- ID helpers ---------------------
|
||||||
|
|
||||||
|
def _to_uuid(stable_key: str) -> str:
|
||||||
|
return str(uuid.uuid5(uuid.NAMESPACE_URL, stable_key))
|
||||||
|
|
||||||
def _names(prefix: str) -> Tuple[str, str, str]:
|
def _names(prefix: str) -> Tuple[str, str, str]:
|
||||||
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
# --------------------- Points builders ---------------------
|
||||||
|
|
||||||
def _to_uuid(stable_key: str) -> str:
|
def points_for_note(prefix: str, note_payload: dict, note_vec: List[float] | None, dim: int) -> Tuple[str, List[rest.PointStruct]]:
|
||||||
"""Stabile UUIDv5 aus einem String-Key (deterministisch)."""
|
|
||||||
return str(uuid.uuid5(uuid.NAMESPACE_URL, stable_key))
|
|
||||||
|
|
||||||
|
|
||||||
def points_for_note(
|
|
||||||
prefix: str,
|
|
||||||
note_payload: dict,
|
|
||||||
note_vec: List[float] | None,
|
|
||||||
dim: int,
|
|
||||||
) -> Tuple[str, List[rest.PointStruct]]:
|
|
||||||
"""Notes-Collection: falls kein Note-Embedding -> Nullvektor der Länge dim."""
|
|
||||||
notes_col, _, _ = _names(prefix)
|
notes_col, _, _ = _names(prefix)
|
||||||
vector = note_vec if note_vec is not None else [0.0] * int(dim)
|
vector = note_vec if note_vec is not None else [0.0] * int(dim)
|
||||||
raw_note_id = note_payload.get("note_id") or note_payload.get("id") or "missing-note-id"
|
raw_note_id = note_payload.get("note_id") or note_payload.get("id") or "missing-note-id"
|
||||||
@@ -57,18 +40,7 @@ def points_for_note(
|
|||||||
pt = rest.PointStruct(id=point_id, vector=vector, payload=note_payload)
|
pt = rest.PointStruct(id=point_id, vector=vector, payload=note_payload)
|
||||||
return notes_col, [pt]
|
return notes_col, [pt]
|
||||||
|
|
||||||
|
def points_for_chunks(prefix: str, chunk_payloads: List[dict], vectors: List[List[float]]) -> Tuple[str, List[rest.PointStruct]]:
|
||||||
def points_for_chunks(
|
|
||||||
prefix: str,
|
|
||||||
chunk_payloads: List[dict],
|
|
||||||
vectors: List[List[float]],
|
|
||||||
) -> Tuple[str, List[rest.PointStruct]]:
|
|
||||||
"""
|
|
||||||
Chunks-Collection: erwartet pro Chunk einen Vektor.
|
|
||||||
Robustheit:
|
|
||||||
- Fehlt 'chunk_id', nutze 'id', sonst baue '${note_id}#${i}' (1-basiert).
|
|
||||||
- Schreibe die abgeleitete ID zurück in die Payload (pl['chunk_id']).
|
|
||||||
"""
|
|
||||||
_, chunks_col, _ = _names(prefix)
|
_, chunks_col, _ = _names(prefix)
|
||||||
points: List[rest.PointStruct] = []
|
points: List[rest.PointStruct] = []
|
||||||
for i, (pl, vec) in enumerate(zip(chunk_payloads, vectors), start=1):
|
for i, (pl, vec) in enumerate(zip(chunk_payloads, vectors), start=1):
|
||||||
@@ -81,22 +53,12 @@ def points_for_chunks(
|
|||||||
points.append(rest.PointStruct(id=point_id, vector=vec, payload=pl))
|
points.append(rest.PointStruct(id=point_id, vector=vec, payload=pl))
|
||||||
return chunks_col, points
|
return chunks_col, points
|
||||||
|
|
||||||
|
|
||||||
def _normalize_edge_payload(pl: dict) -> dict:
|
def _normalize_edge_payload(pl: dict) -> dict:
|
||||||
"""
|
|
||||||
Sorgt für kompatible Feldnamen.
|
|
||||||
akzeptiert:
|
|
||||||
- neu: kind, source_id, target_id, seq?
|
|
||||||
- alt: edge_type, src_id, dst_id, order?/index?
|
|
||||||
schreibt zurück: kind, source_id, target_id, seq?
|
|
||||||
"""
|
|
||||||
# bereits neu?
|
|
||||||
kind = pl.get("kind") or pl.get("edge_type") or "edge"
|
kind = pl.get("kind") or pl.get("edge_type") or "edge"
|
||||||
source_id = pl.get("source_id") or pl.get("src_id") or "unknown-src"
|
source_id = pl.get("source_id") or pl.get("src_id") or "unknown-src"
|
||||||
target_id = pl.get("target_id") or pl.get("dst_id") or "unknown-tgt"
|
target_id = pl.get("target_id") or pl.get("dst_id") or "unknown-tgt"
|
||||||
seq = pl.get("seq") or pl.get("order") or pl.get("index")
|
seq = pl.get("seq") or pl.get("order") or pl.get("index")
|
||||||
|
|
||||||
# in Payload zurückschreiben (ohne alte Felder zu entfernen → maximal kompatibel)
|
|
||||||
pl.setdefault("kind", kind)
|
pl.setdefault("kind", kind)
|
||||||
pl.setdefault("source_id", source_id)
|
pl.setdefault("source_id", source_id)
|
||||||
pl.setdefault("target_id", target_id)
|
pl.setdefault("target_id", target_id)
|
||||||
@@ -104,18 +66,11 @@ def _normalize_edge_payload(pl: dict) -> dict:
|
|||||||
pl["seq"] = seq
|
pl["seq"] = seq
|
||||||
return pl
|
return pl
|
||||||
|
|
||||||
|
|
||||||
def points_for_edges(prefix: str, edge_payloads: List[dict]) -> Tuple[str, List[rest.PointStruct]]:
|
def points_for_edges(prefix: str, edge_payloads: List[dict]) -> Tuple[str, List[rest.PointStruct]]:
|
||||||
"""
|
|
||||||
Edges-Collection mit 1D-Dummy-Vektor.
|
|
||||||
- Akzeptiert sowohl neues als auch altes Edge-Schema (siehe _normalize_edge_payload).
|
|
||||||
- Fehlt 'edge_id', wird sie stabil aus (kind, source_id, target_id, seq) konstruiert.
|
|
||||||
"""
|
|
||||||
_, _, edges_col = _names(prefix)
|
_, _, edges_col = _names(prefix)
|
||||||
points: List[rest.PointStruct] = []
|
points: List[rest.PointStruct] = []
|
||||||
for raw in edge_payloads:
|
for raw in edge_payloads:
|
||||||
pl = _normalize_edge_payload(raw)
|
pl = _normalize_edge_payload(raw)
|
||||||
|
|
||||||
edge_id = pl.get("edge_id")
|
edge_id = pl.get("edge_id")
|
||||||
if not edge_id:
|
if not edge_id:
|
||||||
kind = pl.get("kind", "edge")
|
kind = pl.get("kind", "edge")
|
||||||
@@ -124,13 +79,222 @@ def points_for_edges(prefix: str, edge_payloads: List[dict]) -> Tuple[str, List[
|
|||||||
seq = pl.get("seq") or ""
|
seq = pl.get("seq") or ""
|
||||||
edge_id = f"{kind}:{s}->{t}#{seq}"
|
edge_id = f"{kind}:{s}->{t}#{seq}"
|
||||||
pl["edge_id"] = edge_id
|
pl["edge_id"] = edge_id
|
||||||
|
|
||||||
point_id = _to_uuid(edge_id)
|
point_id = _to_uuid(edge_id)
|
||||||
points.append(rest.PointStruct(id=point_id, vector=[0.0], payload=pl))
|
points.append(rest.PointStruct(id=point_id, vector=[0.0], payload=pl))
|
||||||
return edges_col, points
|
return edges_col, points
|
||||||
|
|
||||||
|
# --------------------- Vector schema & overrides ---------------------
|
||||||
|
|
||||||
def upsert_batch(client, collection: str, points: List[rest.PointStruct]) -> None:
|
def _preferred_name(candidates: List[str]) -> str:
|
||||||
|
for k in ("text", "default", "embedding", "content"):
|
||||||
|
if k in candidates:
|
||||||
|
return k
|
||||||
|
return sorted(candidates)[0]
|
||||||
|
|
||||||
|
def _env_override_for_collection(collection: str) -> Optional[str]:
|
||||||
|
"""
|
||||||
|
Returns:
|
||||||
|
- "__single__" to force single-vector
|
||||||
|
- concrete name (str) to force named-vector with that name
|
||||||
|
- None to auto-detect
|
||||||
|
"""
|
||||||
|
base = os.getenv("MINDNET_VECTOR_NAME")
|
||||||
|
if collection.endswith("_notes"):
|
||||||
|
base = os.getenv("NOTES_VECTOR_NAME", base)
|
||||||
|
elif collection.endswith("_chunks"):
|
||||||
|
base = os.getenv("CHUNKS_VECTOR_NAME", base)
|
||||||
|
elif collection.endswith("_edges"):
|
||||||
|
base = os.getenv("EDGES_VECTOR_NAME", base)
|
||||||
|
|
||||||
|
if not base:
|
||||||
|
return None
|
||||||
|
val = base.strip()
|
||||||
|
if val.lower() in ("__single__", "single"):
|
||||||
|
return "__single__"
|
||||||
|
return val # concrete name
|
||||||
|
|
||||||
|
def _get_vector_schema(client: QdrantClient, collection_name: str) -> dict:
|
||||||
|
"""
|
||||||
|
Return {"kind": "single", "size": int} or {"kind": "named", "names": [...], "primary": str}.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
info = client.get_collection(collection_name=collection_name)
|
||||||
|
vecs = getattr(info, "vectors", None)
|
||||||
|
# Single-vector config
|
||||||
|
if hasattr(vecs, "size") and isinstance(vecs.size, int):
|
||||||
|
return {"kind": "single", "size": vecs.size}
|
||||||
|
# Named-vectors config (dict-like in .config)
|
||||||
|
cfg = getattr(vecs, "config", None)
|
||||||
|
if isinstance(cfg, dict) and cfg:
|
||||||
|
names = list(cfg.keys())
|
||||||
|
if names:
|
||||||
|
return {"kind": "named", "names": names, "primary": _preferred_name(names)}
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return {"kind": "single", "size": None}
|
||||||
|
|
||||||
|
def _as_named(points: List[rest.PointStruct], name: str) -> List[rest.PointStruct]:
|
||||||
|
out: List[rest.PointStruct] = []
|
||||||
|
for pt in points:
|
||||||
|
vec = getattr(pt, "vector", None)
|
||||||
|
if isinstance(vec, dict):
|
||||||
|
if name in vec:
|
||||||
|
out.append(pt)
|
||||||
|
else:
|
||||||
|
# take any existing entry; if empty dict fallback to [0.0]
|
||||||
|
fallback_vec = None
|
||||||
|
try:
|
||||||
|
fallback_vec = list(next(iter(vec.values())))
|
||||||
|
except Exception:
|
||||||
|
fallback_vec = [0.0]
|
||||||
|
out.append(rest.PointStruct(id=pt.id, vector={name: fallback_vec}, payload=pt.payload))
|
||||||
|
elif vec is not None:
|
||||||
|
out.append(rest.PointStruct(id=pt.id, vector={name: vec}, payload=pt.payload))
|
||||||
|
else:
|
||||||
|
out.append(pt)
|
||||||
|
return out
|
||||||
|
|
||||||
|
# --------------------- Qdrant ops ---------------------
|
||||||
|
|
||||||
|
def upsert_batch(client: QdrantClient, collection: str, points: List[rest.PointStruct]) -> None:
|
||||||
if not points:
|
if not points:
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# 1) ENV overrides come first
|
||||||
|
override = _env_override_for_collection(collection)
|
||||||
|
if override == "__single__":
|
||||||
|
client.upsert(collection_name=collection, points=points, wait=True)
|
||||||
|
return
|
||||||
|
elif isinstance(override, str):
|
||||||
|
client.upsert(collection_name=collection, points=_as_named(points, override), wait=True)
|
||||||
|
return
|
||||||
|
|
||||||
|
# 2) Auto-detect schema
|
||||||
|
schema = _get_vector_schema(client, collection)
|
||||||
|
if schema.get("kind") == "named":
|
||||||
|
name = schema.get("primary") or _preferred_name(schema.get("names") or [])
|
||||||
|
client.upsert(collection_name=collection, points=_as_named(points, name), wait=True)
|
||||||
|
return
|
||||||
|
|
||||||
|
# 3) Fallback single-vector
|
||||||
client.upsert(collection_name=collection, points=points, wait=True)
|
client.upsert(collection_name=collection, points=points, wait=True)
|
||||||
|
|
||||||
|
# --- Optional search helpers ---
|
||||||
|
|
||||||
|
def _filter_any(field: str, values: Iterable[str]) -> rest.Filter:
|
||||||
|
return rest.Filter(should=[rest.FieldCondition(key=field, match=rest.MatchValue(value=v)) for v in values])
|
||||||
|
|
||||||
|
def _merge_filters(*filters: Optional[rest.Filter]) -> Optional[rest.Filter]:
|
||||||
|
fs = [f for f in filters if f is not None]
|
||||||
|
if not fs:
|
||||||
|
return None
|
||||||
|
if len(fs) == 1:
|
||||||
|
return fs[0]
|
||||||
|
must = []
|
||||||
|
for f in fs:
|
||||||
|
if getattr(f, "must", None):
|
||||||
|
must.extend(f.must)
|
||||||
|
if getattr(f, "should", None):
|
||||||
|
must.append(rest.Filter(should=f.should))
|
||||||
|
return rest.Filter(must=must)
|
||||||
|
|
||||||
|
def _filter_from_dict(filters: Optional[Dict[str, Any]]) -> Optional[rest.Filter]:
|
||||||
|
if not filters:
|
||||||
|
return None
|
||||||
|
parts = []
|
||||||
|
for k, v in filters.items():
|
||||||
|
if isinstance(v, (list, tuple, set)):
|
||||||
|
parts.append(_filter_any(k, [str(x) for x in v]))
|
||||||
|
else:
|
||||||
|
parts.append(rest.Filter(must=[rest.FieldCondition(key=k, match=rest.MatchValue(value=v))]))
|
||||||
|
return _merge_filters(*parts)
|
||||||
|
|
||||||
|
def search_chunks_by_vector(client: QdrantClient, prefix: str, vector: List[float], top: int = 10, filters: Optional[Dict[str, Any]] = None) -> List[Tuple[str, float, dict]]:
|
||||||
|
_, chunks_col, _ = _names(prefix)
|
||||||
|
flt = _filter_from_dict(filters)
|
||||||
|
res = client.search(collection_name=chunks_col, query_vector=vector, limit=top, with_payload=True, with_vectors=False, query_filter=flt)
|
||||||
|
out: List[Tuple[str, float, dict]] = []
|
||||||
|
for r in res:
|
||||||
|
out.append((str(r.id), float(r.score), dict(r.payload or {})))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# --- Edge retrieval helper ---
|
||||||
|
|
||||||
|
def get_edges_for_sources(
|
||||||
|
client: QdrantClient,
|
||||||
|
prefix: str,
|
||||||
|
source_ids: Iterable[str],
|
||||||
|
edge_types: Optional[Iterable[str]] = None,
|
||||||
|
limit: int = 2048,
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Retrieve edge payloads from the <prefix>_edges collection.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
client: QdrantClient instance.
|
||||||
|
prefix: Mindnet collection prefix (e.g. "mindnet").
|
||||||
|
source_ids: Iterable of source_id values (typically chunk_ids or note_ids).
|
||||||
|
edge_types: Optional iterable of edge kinds (e.g. ["references", "depends_on"]). If None,
|
||||||
|
all kinds are returned.
|
||||||
|
limit: Maximum number of edge payloads to return.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A list of edge payload dicts, e.g.:
|
||||||
|
{
|
||||||
|
"note_id": "...",
|
||||||
|
"chunk_id": "...",
|
||||||
|
"kind": "references" | "depends_on" | ...,
|
||||||
|
"scope": "chunk",
|
||||||
|
"source_id": "...",
|
||||||
|
"target_id": "...",
|
||||||
|
"rule_id": "...",
|
||||||
|
"confidence": 0.7,
|
||||||
|
...
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
source_ids = list(source_ids)
|
||||||
|
if not source_ids or limit <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Resolve collection name
|
||||||
|
_, _, edges_col = _names(prefix)
|
||||||
|
|
||||||
|
# Build filter: source_id IN source_ids
|
||||||
|
src_filter = _filter_any("source_id", [str(s) for s in source_ids])
|
||||||
|
|
||||||
|
# Optional: kind IN edge_types
|
||||||
|
kind_filter = None
|
||||||
|
if edge_types:
|
||||||
|
kind_filter = _filter_any("kind", [str(k) for k in edge_types])
|
||||||
|
|
||||||
|
flt = _merge_filters(src_filter, kind_filter)
|
||||||
|
|
||||||
|
out: List[Dict[str, Any]] = []
|
||||||
|
next_page = None
|
||||||
|
remaining = int(limit)
|
||||||
|
|
||||||
|
# Use paginated scroll API; we don't need vectors, only payloads.
|
||||||
|
while remaining > 0:
|
||||||
|
batch_limit = min(256, remaining)
|
||||||
|
res, next_page = client.scroll(
|
||||||
|
collection_name=edges_col,
|
||||||
|
scroll_filter=flt,
|
||||||
|
limit=batch_limit,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
offset=next_page,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not res:
|
||||||
|
break
|
||||||
|
|
||||||
|
for r in res:
|
||||||
|
out.append(dict(r.payload or {}))
|
||||||
|
remaining -= 1
|
||||||
|
if remaining <= 0:
|
||||||
|
break
|
||||||
|
|
||||||
|
if next_page is None or remaining <= 0:
|
||||||
|
break
|
||||||
|
|
||||||
|
return out
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""
|
||||||
|
app/core/ranking.py — Kombiniertes Scoring (WP-04)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Zusammenführen von semantischem Score (normalisiert), Edge-Bonus und
|
||||||
|
Centrality-Bonus in einen Gesamtscore für die Ergebnisreihung.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
WP-04 Ranking-Formel (w_sem, w_edge, w_cent)
|
||||||
|
Nutzung:
|
||||||
|
from app.core.ranking import combine_scores
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import List, Tuple, Dict
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_scores(values: List[float]) -> List[float]:
|
||||||
|
"""Min-Max-Normalisierung über die Kandidatenmenge (Fallback 0.5 bei Konstanz)."""
|
||||||
|
if not values:
|
||||||
|
return values
|
||||||
|
lo, hi = min(values), max(values)
|
||||||
|
if hi - lo < 1e-9:
|
||||||
|
return [0.5] * len(values)
|
||||||
|
return [(v - lo) / (hi - lo) for v in values]
|
||||||
|
|
||||||
|
|
||||||
|
def combine_scores(
|
||||||
|
hits: List[Tuple[str, float, dict]], # (id, semantic_score, payload)
|
||||||
|
edge_bonus_map: Dict[str, float],
|
||||||
|
centrality_map: Dict[str, float],
|
||||||
|
w_sem: float = 0.70,
|
||||||
|
w_edge: float = 0.25,
|
||||||
|
w_cent: float = 0.05,
|
||||||
|
) -> List[Tuple[str, float, float, float, float]]:
|
||||||
|
"""
|
||||||
|
Liefert Liste von (point_id, total_score, edge_bonus, centrality_bonus, raw_semantic_score),
|
||||||
|
absteigend nach total_score sortiert.
|
||||||
|
"""
|
||||||
|
sem = [h[1] for h in hits]
|
||||||
|
sem_n = normalize_scores(sem)
|
||||||
|
out = []
|
||||||
|
for (pid, s, payload), s_norm in zip(hits, sem_n):
|
||||||
|
e = edge_bonus_map.get(pid, 0.0)
|
||||||
|
c = centrality_map.get(pid, 0.0)
|
||||||
|
total = w_sem * s_norm + w_edge * e + w_cent * c
|
||||||
|
out.append((pid, total, e, c, s))
|
||||||
|
out.sort(key=lambda t: t[1], reverse=True)
|
||||||
|
return out
|
||||||
@@ -0,0 +1,300 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from functools import lru_cache
|
||||||
|
from typing import Any, Dict, List, Tuple, Iterable
|
||||||
|
|
||||||
|
from app.config import get_settings
|
||||||
|
from app.models.dto import QueryRequest, QueryResponse, QueryHit
|
||||||
|
import app.core.qdrant as qdr
|
||||||
|
import app.core.qdrant_points as qp
|
||||||
|
import app.services.embeddings_client as ec
|
||||||
|
import app.core.graph_adapter as ga
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml # type: ignore[import]
|
||||||
|
except Exception: # pragma: no cover - Fallback, falls PyYAML nicht installiert ist
|
||||||
|
yaml = None # type: ignore[assignment]
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache
|
||||||
|
def _get_scoring_weights() -> Tuple[float, float, float]:
|
||||||
|
"""Liefert (semantic_weight, edge_weight, centrality_weight) für den Retriever.
|
||||||
|
|
||||||
|
Priorität:
|
||||||
|
1. Werte aus config/retriever.yaml (falls vorhanden und gültig),
|
||||||
|
Abschnitt:
|
||||||
|
scoring:
|
||||||
|
semantic_weight: 1.0
|
||||||
|
edge_weight: 0.5
|
||||||
|
centrality_weight: 0.5
|
||||||
|
2. Fallback auf Settings.RETRIEVER_W_* (ENV-basiert).
|
||||||
|
"""
|
||||||
|
settings = get_settings()
|
||||||
|
sem = float(getattr(settings, "RETRIEVER_W_SEM", 1.0))
|
||||||
|
edge = float(getattr(settings, "RETRIEVER_W_EDGE", 0.0))
|
||||||
|
cent = float(getattr(settings, "RETRIEVER_W_CENT", 0.0))
|
||||||
|
|
||||||
|
# YAML-Override, falls konfiguriert
|
||||||
|
config_path = os.getenv("MINDNET_RETRIEVER_CONFIG", "config/retriever.yaml")
|
||||||
|
if yaml is None:
|
||||||
|
return sem, edge, cent
|
||||||
|
try:
|
||||||
|
if os.path.exists(config_path):
|
||||||
|
with open(config_path, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
scoring = data.get("scoring", {}) or {}
|
||||||
|
sem = float(scoring.get("semantic_weight", sem))
|
||||||
|
edge = float(scoring.get("edge_weight", edge))
|
||||||
|
cent = float(scoring.get("centrality_weight", cent))
|
||||||
|
except Exception:
|
||||||
|
# Bei Fehlern in der YAML-Konfiguration defensiv auf Defaults zurückfallen
|
||||||
|
return sem, edge, cent
|
||||||
|
return sem, edge, cent
|
||||||
|
|
||||||
|
|
||||||
|
def _get_client_and_prefix() -> Tuple[Any, str]:
|
||||||
|
"""Liefert (QdrantClient, prefix) basierend auf QdrantConfig.from_env()."""
|
||||||
|
cfg = qdr.QdrantConfig.from_env()
|
||||||
|
client = qdr.get_client(cfg)
|
||||||
|
return client, cfg.prefix
|
||||||
|
|
||||||
|
|
||||||
|
def _get_query_vector(req: QueryRequest) -> List[float]:
|
||||||
|
"""
|
||||||
|
Liefert den Query-Vektor aus dem Request.
|
||||||
|
|
||||||
|
- Falls req.query_vector gesetzt ist, wird dieser unverändert genutzt.
|
||||||
|
- Andernfalls wird req.query über den Embedding-Service in einen Vektor
|
||||||
|
transformiert.
|
||||||
|
"""
|
||||||
|
if req.query_vector:
|
||||||
|
return list(req.query_vector)
|
||||||
|
|
||||||
|
if not req.query:
|
||||||
|
raise ValueError("QueryRequest benötigt entweder query oder query_vector")
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
model_name = settings.MODEL_NAME
|
||||||
|
|
||||||
|
# Kompatibel mit Fakes in Unit-Tests (ohne model_name-Parameter)
|
||||||
|
try:
|
||||||
|
return ec.embed_text(req.query, model_name=model_name) # type: ignore[call-arg]
|
||||||
|
except TypeError:
|
||||||
|
# Fallback: einfache Signatur embed_text(text)
|
||||||
|
return ec.embed_text(req.query) # type: ignore[call-arg]
|
||||||
|
|
||||||
|
|
||||||
|
def _semantic_hits(
|
||||||
|
client: Any,
|
||||||
|
prefix: str,
|
||||||
|
vector: List[float],
|
||||||
|
top_k: int,
|
||||||
|
filters: Dict[str, Any] | None = None,
|
||||||
|
) -> List[Tuple[str, float, Dict[str, Any]]]:
|
||||||
|
"""Führt eine semantische Suche über mindnet_chunks aus und liefert Roh-Treffer.
|
||||||
|
|
||||||
|
Rückgabeformat: Liste von (point_id, score, payload)
|
||||||
|
|
||||||
|
Erwartetes Format von qp.search_chunks_by_vector:
|
||||||
|
List[Tuple[str, float, dict]]
|
||||||
|
"""
|
||||||
|
flt = filters or None
|
||||||
|
raw_hits = qp.search_chunks_by_vector(client, prefix, vector, top=top_k, filters=flt)
|
||||||
|
results: List[Tuple[str, float, Dict[str, Any]]] = []
|
||||||
|
for pid, score, payload in raw_hits:
|
||||||
|
results.append((str(pid), float(score), dict(payload or {})))
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def _get_semantic_score_and_payload(
|
||||||
|
hit: Tuple[str, float, Dict[str, Any]]
|
||||||
|
) -> Tuple[float, Dict[str, Any]]:
|
||||||
|
"""Extrahiert semantic_score und Payload aus einem Raw-Hit."""
|
||||||
|
_, score, payload = hit
|
||||||
|
return float(score), payload or {}
|
||||||
|
|
||||||
|
|
||||||
|
def _compute_total_score(
|
||||||
|
semantic_score: float,
|
||||||
|
payload: Dict[str, Any],
|
||||||
|
edge_bonus: float = 0.0,
|
||||||
|
cent_bonus: float = 0.0,
|
||||||
|
) -> Tuple[float, float, float]:
|
||||||
|
"""Berechnet total_score aus semantic_score, retriever_weight und Graph-Boni.
|
||||||
|
|
||||||
|
Formel (WP-04, konfigurierbar über config/retriever.yaml bzw. ENV):
|
||||||
|
total_score = W_sem * semantic_score * max(retriever_weight, 0.0)
|
||||||
|
+ W_edge * edge_bonus
|
||||||
|
+ W_cent * cent_bonus
|
||||||
|
|
||||||
|
- retriever_weight stammt aus dem Chunk-Payload (types.yaml).
|
||||||
|
- W_sem / W_edge / W_cent kommen aus _get_scoring_weights().
|
||||||
|
"""
|
||||||
|
raw_weight = payload.get("retriever_weight", 1.0)
|
||||||
|
try:
|
||||||
|
weight = float(raw_weight)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
weight = 1.0
|
||||||
|
if weight < 0.0:
|
||||||
|
weight = 0.0
|
||||||
|
|
||||||
|
sem_w, edge_w, cent_w = _get_scoring_weights()
|
||||||
|
total = (sem_w * float(semantic_score) * weight) + (edge_w * edge_bonus) + (cent_w * cent_bonus)
|
||||||
|
return float(total), float(edge_bonus), float(cent_bonus)
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_expand_options(req: QueryRequest) -> Tuple[int, List[str] | None]:
|
||||||
|
"""Extrahiert depth und edge_types aus req.expand, falls vorhanden.
|
||||||
|
|
||||||
|
- Falls expand nicht gesetzt ist: depth=0, edge_types=None (keine Expansion).
|
||||||
|
- Unterstützt sowohl Pydantic-Modelle als auch plain dicts.
|
||||||
|
"""
|
||||||
|
expand = getattr(req, "expand", None)
|
||||||
|
if not expand:
|
||||||
|
return 0, None
|
||||||
|
|
||||||
|
depth = 1
|
||||||
|
edge_types: List[str] | None = None
|
||||||
|
|
||||||
|
# Pydantic-Modell oder Objekt mit Attributen
|
||||||
|
if hasattr(expand, "depth") or hasattr(expand, "edge_types"):
|
||||||
|
depth = int(getattr(expand, "depth", 1) or 1)
|
||||||
|
types_val = getattr(expand, "edge_types", None)
|
||||||
|
if types_val:
|
||||||
|
edge_types = list(types_val)
|
||||||
|
return depth, edge_types
|
||||||
|
|
||||||
|
# Plain dict
|
||||||
|
if isinstance(expand, dict):
|
||||||
|
if "depth" in expand:
|
||||||
|
depth = int(expand.get("depth") or 1)
|
||||||
|
if "edge_types" in expand and expand["edge_types"] is not None:
|
||||||
|
edge_types = list(expand["edge_types"])
|
||||||
|
return depth, edge_types
|
||||||
|
|
||||||
|
return 0, None
|
||||||
|
|
||||||
|
|
||||||
|
def _build_hits_from_semantic(
|
||||||
|
hits: Iterable[Tuple[str, float, Dict[str, Any]]],
|
||||||
|
top_k: int,
|
||||||
|
used_mode: str,
|
||||||
|
subgraph: ga.Subgraph | None = None,
|
||||||
|
) -> QueryResponse:
|
||||||
|
"""Baut aus Raw-Hits und optionalem Subgraph strukturierte QueryHits.
|
||||||
|
|
||||||
|
- Aggregation auf Note-Ebene (note_id),
|
||||||
|
- Berechnung von total_score unter Nutzung von retriever_weight + Graph-Boni,
|
||||||
|
- Rückgabe als QueryResponse mit Latenz.
|
||||||
|
"""
|
||||||
|
t0 = time.time()
|
||||||
|
|
||||||
|
enriched: List[Tuple[str, float, Dict[str, Any], float, float, float]] = []
|
||||||
|
|
||||||
|
for pid, semantic_score, payload in hits:
|
||||||
|
# Graph-Scores, falls Subgraph und stabiler Key vorhanden
|
||||||
|
edge_bonus = 0.0
|
||||||
|
cent_bonus = 0.0
|
||||||
|
if subgraph is not None:
|
||||||
|
node_key = payload.get("chunk_id") or payload.get("note_id")
|
||||||
|
if node_key:
|
||||||
|
try:
|
||||||
|
edge_bonus = float(subgraph.edge_bonus(node_key))
|
||||||
|
except Exception:
|
||||||
|
edge_bonus = 0.0
|
||||||
|
try:
|
||||||
|
cent_bonus = float(subgraph.centrality_bonus(node_key))
|
||||||
|
except Exception:
|
||||||
|
cent_bonus = 0.0
|
||||||
|
|
||||||
|
total, edge_bonus, cent_bonus = _compute_total_score(
|
||||||
|
semantic_score,
|
||||||
|
payload,
|
||||||
|
edge_bonus=edge_bonus,
|
||||||
|
cent_bonus=cent_bonus,
|
||||||
|
)
|
||||||
|
enriched.append((pid, float(semantic_score), payload, total, edge_bonus, cent_bonus))
|
||||||
|
|
||||||
|
# Sortierung nach total_score absteigend
|
||||||
|
enriched_sorted = sorted(enriched, key=lambda h: h[3], reverse=True)
|
||||||
|
limited = enriched_sorted[: max(1, top_k)]
|
||||||
|
|
||||||
|
results: List[QueryHit] = []
|
||||||
|
for pid, semantic_score, payload, total, edge_bonus, cent_bonus in limited:
|
||||||
|
note_id = payload.get("note_id")
|
||||||
|
path = payload.get("path")
|
||||||
|
# mindnet_chunks: section (aktuell genutzt); ältere Stände nutzen ggf. section_title
|
||||||
|
section = payload.get("section") or payload.get("section_title")
|
||||||
|
|
||||||
|
results.append(
|
||||||
|
QueryHit(
|
||||||
|
node_id=str(pid),
|
||||||
|
note_id=note_id,
|
||||||
|
semantic_score=float(semantic_score),
|
||||||
|
edge_bonus=edge_bonus,
|
||||||
|
centrality_bonus=cent_bonus,
|
||||||
|
total_score=total,
|
||||||
|
paths=None,
|
||||||
|
source={
|
||||||
|
"path": path,
|
||||||
|
"section": section,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
dt = int((time.time() - t0) * 1000)
|
||||||
|
return QueryResponse(results=results, used_mode=used_mode, latency_ms=dt)
|
||||||
|
|
||||||
|
|
||||||
|
def semantic_retrieve(req: QueryRequest) -> QueryResponse:
|
||||||
|
"""Reiner semantischer Retriever (ohne Edge-Expansion)."""
|
||||||
|
client, prefix = _get_client_and_prefix()
|
||||||
|
vector = _get_query_vector(req)
|
||||||
|
top_k = req.top_k or get_settings().RETRIEVER_TOP_K
|
||||||
|
|
||||||
|
hits = _semantic_hits(client, prefix, vector, top_k=top_k, filters=req.filters)
|
||||||
|
return _build_hits_from_semantic(hits, top_k=top_k, used_mode="semantic", subgraph=None)
|
||||||
|
|
||||||
|
|
||||||
|
def hybrid_retrieve(req: QueryRequest) -> QueryResponse:
|
||||||
|
"""Hybrid-Retriever: semantische Suche + optionale Edge-Expansion.
|
||||||
|
|
||||||
|
Aktueller Stand (Step 3):
|
||||||
|
- Semantische Suche über mindnet_chunks (wie semantic_retrieve),
|
||||||
|
- Bei expand.depth > 0 wird ein Subgraph aus mindnet_edges aufgebaut,
|
||||||
|
und edge_bonus / centrality_bonus in das Scoring einbezogen.
|
||||||
|
"""
|
||||||
|
client, prefix = _get_client_and_prefix()
|
||||||
|
|
||||||
|
# Query-Vektor:
|
||||||
|
# - Falls explizit gesetzt, unverändert nutzen.
|
||||||
|
# - Andernfalls über embed_text erzeugen.
|
||||||
|
if req.query_vector:
|
||||||
|
vector = list(req.query_vector)
|
||||||
|
else:
|
||||||
|
vector = _get_query_vector(req)
|
||||||
|
|
||||||
|
top_k = req.top_k or get_settings().RETRIEVER_TOP_K
|
||||||
|
|
||||||
|
hits = _semantic_hits(client, prefix, vector, top_k=top_k, filters=req.filters)
|
||||||
|
|
||||||
|
depth, edge_types = _extract_expand_options(req)
|
||||||
|
subgraph: ga.Subgraph | None = None
|
||||||
|
if depth and depth > 0:
|
||||||
|
# Seeds: stabile IDs aus dem Payload (chunk_id bevorzugt, sonst note_id)
|
||||||
|
seed_ids: List[str] = []
|
||||||
|
for _pid, _score, payload in hits:
|
||||||
|
key = payload.get("chunk_id") or payload.get("note_id")
|
||||||
|
if key and key not in seed_ids:
|
||||||
|
seed_ids.append(key)
|
||||||
|
|
||||||
|
if seed_ids:
|
||||||
|
try:
|
||||||
|
subgraph = ga.expand(client, prefix, seed_ids, depth=depth, edge_types=edge_types)
|
||||||
|
except Exception:
|
||||||
|
# Edge-Expansion ist optional: bei Fehlern weiter ohne Graph-Boni
|
||||||
|
subgraph = None
|
||||||
|
|
||||||
|
return _build_hits_from_semantic(hits, top_k=top_k, used_mode="hybrid", subgraph=subgraph)
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
"""app/core/retriever_config.py
|
||||||
|
---------------------------------
|
||||||
|
Zentrale Konfiguration für den mindnet-Retriever (WP-04).
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
- Lädt config/retriever.yaml (falls vorhanden) oder nutzt sinnvolle Defaults.
|
||||||
|
- Bietet einen gecachten Zugriff auf die Retriever-Config für
|
||||||
|
andere Module (z. B. graph_adapter, retriever).
|
||||||
|
|
||||||
|
Hinweis zur Weiterentwicklung (Selbstjustierung):
|
||||||
|
- Die hier definierten Parameter sind so gewählt, dass sie später
|
||||||
|
durch ein Feedback-/Learning-to-Rank-Modell überschrieben werden
|
||||||
|
können, ohne die restliche Architektur anzupassen.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from functools import lru_cache
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml # type: ignore
|
||||||
|
except Exception: # pragma: no cover - Fallback, falls PyYAML nicht installiert ist.
|
||||||
|
yaml = None # type: ignore
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RetrieverConfig:
|
||||||
|
semantic_scale: float
|
||||||
|
edge_scale: float
|
||||||
|
centrality_scale: float
|
||||||
|
edge_weights: Dict[str, float]
|
||||||
|
|
||||||
|
@lru_cache
|
||||||
|
def get_retriever_config() -> RetrieverConfig:
|
||||||
|
"""Lädt die Retriever-Konfiguration (YAML + Defaults).
|
||||||
|
|
||||||
|
Reihenfolge:
|
||||||
|
1. Defaults (sinnvoll gewählte Startwerte).
|
||||||
|
2. Optional: config/retriever.yaml bzw. Pfad aus ENV
|
||||||
|
MINDNET_RETRIEVER_CONFIG überschreibt die Defaults.
|
||||||
|
|
||||||
|
Die Funktion ist bewusst gecached, da sich die Konfiguration zur
|
||||||
|
Laufzeit üblicherweise nicht ändert. Für dynamisches Nachladen
|
||||||
|
könnte der Cache explizit geleert werden.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# 1) Defaults – bewusst konservativ gewählt.
|
||||||
|
semantic_scale = 1.0
|
||||||
|
edge_scale = 1.0
|
||||||
|
centrality_scale = 1.0
|
||||||
|
|
||||||
|
edge_weights: Dict[str, float] = {
|
||||||
|
# Wissens-Kanten
|
||||||
|
"depends_on": 1.0,
|
||||||
|
"related_to": 0.7,
|
||||||
|
"similar_to": 0.7,
|
||||||
|
"references": 0.5,
|
||||||
|
# Struktur-Kanten
|
||||||
|
"belongs_to": 0.2,
|
||||||
|
"next": 0.1,
|
||||||
|
"prev": 0.1,
|
||||||
|
# Sonstige / technische Kanten
|
||||||
|
"backlink": 0.2,
|
||||||
|
"references_at": 0.2,
|
||||||
|
}
|
||||||
|
|
||||||
|
# 2) Optional: YAML-Konfiguration laden
|
||||||
|
cfg_path_env = os.getenv("MINDNET_RETRIEVER_CONFIG")
|
||||||
|
if cfg_path_env:
|
||||||
|
cfg_path = Path(cfg_path_env)
|
||||||
|
else:
|
||||||
|
# Project-Root = zwei Ebenen über app/core/
|
||||||
|
cfg_path = Path(__file__).resolve().parents[2] / "config" / "retriever.yaml"
|
||||||
|
|
||||||
|
if yaml is not None and cfg_path.exists():
|
||||||
|
try:
|
||||||
|
data = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) or {}
|
||||||
|
except Exception:
|
||||||
|
data = {}
|
||||||
|
|
||||||
|
retr = data.get("retriever") or {}
|
||||||
|
|
||||||
|
# Skalenwerte überschreiben, falls angegeben
|
||||||
|
try:
|
||||||
|
semantic_scale = float(retr.get("semantic_scale", semantic_scale))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
edge_scale = float(retr.get("edge_scale", edge_scale))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
centrality_scale = float(retr.get("centrality_scale", centrality_scale))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Edge-Gewichte je Kanten-Typ
|
||||||
|
ew_cfg = retr.get("edge_weights") or {}
|
||||||
|
if isinstance(ew_cfg, dict):
|
||||||
|
for k, v in ew_cfg.items():
|
||||||
|
try:
|
||||||
|
edge_weights[str(k)] = float(v)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
|
||||||
|
return RetrieverConfig(
|
||||||
|
semantic_scale=semantic_scale,
|
||||||
|
edge_scale=edge_scale,
|
||||||
|
centrality_scale=centrality_scale,
|
||||||
|
edge_weights=edge_weights,
|
||||||
|
)
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Modul: app/core/type_registry.py
|
||||||
|
Version: 1.0.0
|
||||||
|
Datum: 2025-11-08
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
-----
|
||||||
|
Lädt eine optionale Typ-Registry (config/types.yaml) und stellt
|
||||||
|
komfortable Zugriffsfunktionen bereit. Die Registry ist *optional*:
|
||||||
|
- Fehlt die Datei oder ist das YAML defekt, wird ein konservativer
|
||||||
|
Default (Typ "concept") verwendet und es wird eine Warnung ausgegeben.
|
||||||
|
- Änderungen an der Datei greifen nach einem Neustart des Prozesses.
|
||||||
|
|
||||||
|
Öffentliche API
|
||||||
|
---------------
|
||||||
|
- load_type_registry(path: str = "config/types.yaml") -> dict
|
||||||
|
- get_type_config(note_type: str, reg: dict) -> dict
|
||||||
|
- resolve_note_type(fm_type: str | None, reg: dict) -> str
|
||||||
|
- effective_chunk_profile(note_type: str, reg: dict) -> str | None
|
||||||
|
- profile_overlap(profile: str | None) -> tuple[int,int] # nur Overlap-Empfehlung
|
||||||
|
|
||||||
|
Hinweis
|
||||||
|
-------
|
||||||
|
Die Registry steuert KEINE Breaking Changes. Ohne Datei/Typ bleibt das
|
||||||
|
Verhalten exakt wie im Release-Stand 20251105.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from functools import lru_cache
|
||||||
|
from typing import Dict, Any, Optional, Tuple
|
||||||
|
import os
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml # PyYAML
|
||||||
|
except Exception:
|
||||||
|
yaml = None # wird erst benötigt, wenn eine Datei gelesen werden soll
|
||||||
|
|
||||||
|
# Konservativer Default – bewusst minimal
|
||||||
|
_DEFAULT_REGISTRY: Dict[str, Any] = {
|
||||||
|
"version": "1.0",
|
||||||
|
"types": {
|
||||||
|
"concept": {
|
||||||
|
"chunk_profile": "medium",
|
||||||
|
"edge_defaults": ["references", "related_to"],
|
||||||
|
"retriever_weight": 1.0,
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"_using_defaults": True,
|
||||||
|
"_warning": "types.yaml missing or invalid – using built-in defaults (type=concept).",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Chunk-Profile → Overlap-Empfehlungen (nur für synthetische Fensterbildung)
|
||||||
|
# Die absoluten Chunk-Längen bleiben Aufgabe des Chunkers (assemble_chunks).
|
||||||
|
_PROFILE_TO_OVERLAP: Dict[str, Tuple[int, int]] = {
|
||||||
|
"short": (20, 30),
|
||||||
|
"medium": (40, 60),
|
||||||
|
"long": (60, 80),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def load_type_registry(path: str = "config/types.yaml") -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Lädt die Registry aus 'path'. Bei Fehlern wird ein konserviver Default geliefert.
|
||||||
|
Die Rückgabe ist *prozessweit* gecached.
|
||||||
|
"""
|
||||||
|
if not path:
|
||||||
|
return dict(_DEFAULT_REGISTRY)
|
||||||
|
|
||||||
|
if not os.path.isfile(path):
|
||||||
|
return dict(_DEFAULT_REGISTRY)
|
||||||
|
|
||||||
|
if yaml is None:
|
||||||
|
# PyYAML fehlt → auf Default zurückfallen
|
||||||
|
return dict(_DEFAULT_REGISTRY)
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
# Minimal validieren
|
||||||
|
if not isinstance(data, dict) or "types" not in data or not isinstance(data["types"], dict):
|
||||||
|
return dict(_DEFAULT_REGISTRY)
|
||||||
|
data.setdefault("version", "1.0")
|
||||||
|
data.setdefault("_using_defaults", False)
|
||||||
|
return data
|
||||||
|
except Exception:
|
||||||
|
return dict(_DEFAULT_REGISTRY)
|
||||||
|
|
||||||
|
|
||||||
|
def get_type_config(note_type: Optional[str], reg: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
|
t = (note_type or "concept").strip().lower()
|
||||||
|
types = (reg or {}).get("types", {}) if isinstance(reg, dict) else {}
|
||||||
|
return types.get(t) or types.get("concept") or _DEFAULT_REGISTRY["types"]["concept"]
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_note_type(fm_type: Optional[str], reg: Dict[str, Any]) -> str:
|
||||||
|
"""Liefert einen gültigen Typ (unbekannt → 'concept')."""
|
||||||
|
t = (fm_type or "concept").strip().lower()
|
||||||
|
types = (reg or {}).get("types", {}) if isinstance(reg, dict) else {}
|
||||||
|
return t if t in types else "concept"
|
||||||
|
|
||||||
|
|
||||||
|
def effective_chunk_profile(note_type: Optional[str], reg: Dict[str, Any]) -> Optional[str]:
|
||||||
|
cfg = get_type_config(note_type, reg)
|
||||||
|
prof = cfg.get("chunk_profile")
|
||||||
|
if isinstance(prof, str) and prof.strip():
|
||||||
|
return prof.strip().lower()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def profile_overlap(profile: Optional[str]) -> Tuple[int, int]:
|
||||||
|
"""Gibt eine Overlap-Empfehlung (low, high) für das Profil zurück."""
|
||||||
|
if not profile:
|
||||||
|
return _PROFILE_TO_OVERLAP["medium"]
|
||||||
|
return _PROFILE_TO_OVERLAP.get(profile.strip().lower(), _PROFILE_TO_OVERLAP["medium"])
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Modul: app/graph/service.py
|
||||||
|
Version: 0.1.0
|
||||||
|
Datum: 2025-09-10
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
-----
|
||||||
|
Leichtgewichtiger Graph-Layer über Qdrant:
|
||||||
|
- get_note(note_id)
|
||||||
|
- get_chunks(note_id)
|
||||||
|
- neighbors(source_id, kinds=[...], scope=['note','chunk'], depth=1)
|
||||||
|
- walk_bfs(source_id, kinds, max_depth)
|
||||||
|
- context_for_note(note_id, max_neighbors): heuristische Kontextsammlung
|
||||||
|
|
||||||
|
Hinweise
|
||||||
|
--------
|
||||||
|
- Nutzt die bestehenden Collections <prefix>_notes/_chunks/_edges.
|
||||||
|
- Edges werden über Payload-Felder (`kind`, `source_id`, `target_id`) abgefragt.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import List, Dict, Any, Optional, Iterable, Set, Tuple
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
def _cols(prefix: str):
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
class GraphService:
|
||||||
|
def __init__(self, cfg: Optional[QdrantConfig] = None, prefix: Optional[str] = None):
|
||||||
|
self.cfg = cfg or QdrantConfig.from_env()
|
||||||
|
if prefix:
|
||||||
|
self.cfg.prefix = prefix
|
||||||
|
self.client = get_client(self.cfg)
|
||||||
|
self.notes_col, self.chunks_col, self.edges_col = _cols(self.cfg.prefix)
|
||||||
|
|
||||||
|
# ------------------------ fetch helpers ------------------------
|
||||||
|
def _scroll(self, col: str, flt: Optional[rest.Filter] = None, limit: int = 256):
|
||||||
|
out = []
|
||||||
|
nextp = None
|
||||||
|
while True:
|
||||||
|
pts, nextp = self.client.scroll(
|
||||||
|
collection_name=col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=limit,
|
||||||
|
offset=nextp,
|
||||||
|
scroll_filter=flt,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
out.extend(pts)
|
||||||
|
if nextp is None:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
# ------------------------ public API ---------------------------
|
||||||
|
def get_note(self, note_id: str) -> Optional[Dict[str, Any]]:
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
pts, _ = self.client.scroll(self.notes_col, with_payload=True, with_vectors=False, limit=1, scroll_filter=f)
|
||||||
|
return (pts[0].payload or None) if pts else None
|
||||||
|
|
||||||
|
def get_chunks(self, note_id: str) -> List[Dict[str, Any]]:
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
pts = self._scroll(self.chunks_col, f)
|
||||||
|
# Sortierung analog Export
|
||||||
|
def key(pl):
|
||||||
|
p = pl.payload or {}
|
||||||
|
s = p.get("seq") or 0
|
||||||
|
ci = p.get("chunk_index") or 0
|
||||||
|
n = 0
|
||||||
|
cid = p.get("chunk_id") or ""
|
||||||
|
if isinstance(cid, str) and "#" in cid:
|
||||||
|
try:
|
||||||
|
n = int(cid.rsplit("#", 1)[-1])
|
||||||
|
except Exception:
|
||||||
|
n = 0
|
||||||
|
return (int(s), int(ci), n)
|
||||||
|
pts_sorted = sorted(pts, key=key)
|
||||||
|
return [p.payload or {} for p in pts_sorted]
|
||||||
|
|
||||||
|
def neighbors(self, source_id: str, kinds: Optional[Iterable[str]] = None,
|
||||||
|
scope: Optional[Iterable[str]] = None, depth: int = 1) -> Dict[str, List[Dict[str, Any]]]:
|
||||||
|
"""
|
||||||
|
Liefert eingehende & ausgehende Nachbarn (nur nach kind gefiltert).
|
||||||
|
depth==1: direkte Kanten.
|
||||||
|
"""
|
||||||
|
kinds = list(kinds) if kinds else None
|
||||||
|
must = [rest.FieldCondition(key="source_id", match=rest.MatchValue(value=source_id))]
|
||||||
|
if kinds:
|
||||||
|
must.append(rest.FieldCondition(key="kind", match=rest.MatchAny(any=kinds)))
|
||||||
|
f = rest.Filter(must=must)
|
||||||
|
edges = self._scroll(self.edges_col, f)
|
||||||
|
out = {"out": [], "in": []}
|
||||||
|
for e in edges:
|
||||||
|
out["out"].append(e.payload or {})
|
||||||
|
# Inverse Richtung (eingehend)
|
||||||
|
must_in = [rest.FieldCondition(key="target_id", match=rest.MatchValue(value=source_id))]
|
||||||
|
if kinds:
|
||||||
|
must_in.append(rest.FieldCondition(key="kind", match=rest.MatchAny(any=kinds)))
|
||||||
|
f_in = rest.Filter(must=must_in)
|
||||||
|
edges_in = self._scroll(self.edges_col, f_in)
|
||||||
|
for e in edges_in:
|
||||||
|
out["in"].append(e.payload or {})
|
||||||
|
return out
|
||||||
|
|
||||||
|
def walk_bfs(self, source_id: str, kinds: Iterable[str], max_depth: int = 2) -> Set[str]:
|
||||||
|
visited: Set[str] = {source_id}
|
||||||
|
frontier: Set[str] = {source_id}
|
||||||
|
kinds = list(kinds)
|
||||||
|
for _ in range(max_depth):
|
||||||
|
nxt: Set[str] = set()
|
||||||
|
for s in frontier:
|
||||||
|
neigh = self.neighbors(s, kinds=kinds)
|
||||||
|
for e in neigh["out"]:
|
||||||
|
t = e.get("target_id")
|
||||||
|
if isinstance(t, str) and t not in visited:
|
||||||
|
visited.add(t)
|
||||||
|
nxt.add(t)
|
||||||
|
frontier = nxt
|
||||||
|
if not frontier:
|
||||||
|
break
|
||||||
|
return visited
|
||||||
|
|
||||||
|
def context_for_note(self, note_id: str, kinds: Iterable[str] = ("references","backlink"), max_neighbors: int = 12) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Heuristischer Kontext: eigene Chunks + Nachbarn nach Kantenarten, dedupliziert.
|
||||||
|
"""
|
||||||
|
note = self.get_note(note_id) or {}
|
||||||
|
chunks = self.get_chunks(note_id)
|
||||||
|
neigh = self.neighbors(note_id, kinds=list(kinds))
|
||||||
|
targets = []
|
||||||
|
for e in neigh["out"]:
|
||||||
|
t = e.get("target_id")
|
||||||
|
if isinstance(t, str):
|
||||||
|
targets.append(t)
|
||||||
|
for e in neigh["in"]:
|
||||||
|
s = e.get("source_id")
|
||||||
|
if isinstance(s, str):
|
||||||
|
targets.append(s)
|
||||||
|
# de-dupe
|
||||||
|
seen = set()
|
||||||
|
uniq = []
|
||||||
|
for t in targets:
|
||||||
|
if t not in seen:
|
||||||
|
seen.add(t)
|
||||||
|
uniq.append(t)
|
||||||
|
uniq = uniq[:max_neighbors]
|
||||||
|
neighbor_notes = [self.get_note(t) for t in uniq]
|
||||||
|
return {
|
||||||
|
"note": note,
|
||||||
|
"chunks": chunks,
|
||||||
|
"neighbors": [n for n in neighbor_notes if n],
|
||||||
|
"edges_out": neigh["out"],
|
||||||
|
"edges_in": neigh["in"],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Optional: Mini-CLI
|
||||||
|
if __name__ == "__main__": # pragma: no cover
|
||||||
|
import argparse, json
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", help="Collection-Prefix (überschreibt ENV)")
|
||||||
|
ap.add_argument("--note-id", required=True)
|
||||||
|
ap.add_argument("--neighbors", action="store_true", help="Nur Nachbarn anzeigen")
|
||||||
|
args = ap.parse_args()
|
||||||
|
svc = GraphService(prefix=args.prefix)
|
||||||
|
if args.neighbors:
|
||||||
|
out = svc.neighbors(args.note_id, kinds=["references","backlink","prev","next","belongs_to"])
|
||||||
|
else:
|
||||||
|
out = svc.context_for_note(args.note_id)
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
+21
-3
@@ -1,13 +1,23 @@
|
|||||||
"""
|
"""
|
||||||
Version 0.3
|
app/main.py — mindnet API bootstrap (WP-04 Hooks)
|
||||||
|
Version: 0.4.2 • Stand: 2025-10-07
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from .config import get_settings
|
from .config import get_settings
|
||||||
from .routers.embed_router import router as embed_router
|
from .routers.embed_router import router as embed_router
|
||||||
from .routers.qdrant_router import router as qdrant_router
|
from .routers.qdrant_router import router as qdrant_router
|
||||||
|
|
||||||
|
# WP-04 Router:
|
||||||
|
from .routers.query import router as query_router
|
||||||
|
from .routers.graph import router as graph_router
|
||||||
|
from .routers.tools import router as tools_router
|
||||||
|
# Optional:
|
||||||
|
try:
|
||||||
|
from .routers.admin import router as admin_router
|
||||||
|
except Exception:
|
||||||
|
admin_router = None
|
||||||
|
|
||||||
def create_app() -> FastAPI:
|
def create_app() -> FastAPI:
|
||||||
app = FastAPI(title="mindnet API", version="0.1.0")
|
app = FastAPI(title="mindnet API", version="0.1.0")
|
||||||
@@ -17,9 +27,17 @@ def create_app() -> FastAPI:
|
|||||||
def healthz():
|
def healthz():
|
||||||
return {"status": "ok", "qdrant": s.QDRANT_URL, "prefix": s.COLLECTION_PREFIX}
|
return {"status": "ok", "qdrant": s.QDRANT_URL, "prefix": s.COLLECTION_PREFIX}
|
||||||
|
|
||||||
|
# Bestehende Router (unverändert)
|
||||||
app.include_router(embed_router)
|
app.include_router(embed_router)
|
||||||
app.include_router(qdrant_router)
|
app.include_router(qdrant_router)
|
||||||
|
|
||||||
|
# WP-04 Endpunkte
|
||||||
|
app.include_router(query_router, prefix="/query", tags=["query"])
|
||||||
|
app.include_router(graph_router, prefix="/graph", tags=["graph"])
|
||||||
|
app.include_router(tools_router, prefix="/tools", tags=["tools"])
|
||||||
|
if admin_router:
|
||||||
|
app.include_router(admin_router, prefix="/admin", tags=["admin"])
|
||||||
|
|
||||||
return app
|
return app
|
||||||
|
|
||||||
|
|
||||||
app = create_app()
|
app = create_app()
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
"""
|
||||||
|
app/models/__init__.py — Package-Init für mindnet DTOs (WP-04)
|
||||||
|
Version: 0.1.0 • Stand: 2025-10-07
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .dto import (
|
||||||
|
EdgeKind,
|
||||||
|
NodeDTO,
|
||||||
|
EdgeDTO,
|
||||||
|
QueryRequest,
|
||||||
|
QueryHit,
|
||||||
|
QueryResponse,
|
||||||
|
GraphResponse,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"EdgeKind",
|
||||||
|
"NodeDTO",
|
||||||
|
"EdgeDTO",
|
||||||
|
"QueryRequest",
|
||||||
|
"QueryHit",
|
||||||
|
"QueryResponse",
|
||||||
|
"GraphResponse",
|
||||||
|
]
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
"""
|
||||||
|
app/models/dto.py — Pydantic-Modelle (DTOs) für WP-04 Endpunkte
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Laufzeit-Modelle für FastAPI (Requests/Responses), getrennt von JSON-Schemas.
|
||||||
|
Deckt die Graph-/Retriever-Endpunkte ab, ohne bestehende Funktionen zu ändern.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, Pydantic 2.x, FastAPI 0.110+
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage für WP-04)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
- schemas/*.json (Speicherschema für Notes/Chunks/Edges)
|
||||||
|
- WP-04 API-Design (Query- und Graph-Endpunkte)
|
||||||
|
Nutzung:
|
||||||
|
from app.models.dto import QueryRequest, QueryResponse, GraphResponse
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from typing import List, Literal, Optional, Dict
|
||||||
|
|
||||||
|
EdgeKind = Literal["references", "references_at", "backlink", "next", "prev", "belongs_to"]
|
||||||
|
|
||||||
|
|
||||||
|
class NodeDTO(BaseModel):
|
||||||
|
"""Darstellung eines Knotens (Note oder Chunk) im API-Graph."""
|
||||||
|
id: str
|
||||||
|
type: Literal["note", "chunk"]
|
||||||
|
title: Optional[str] = None
|
||||||
|
note_id: Optional[str] = None
|
||||||
|
tags: Optional[List[str]] = None
|
||||||
|
in_degree: Optional[int] = None
|
||||||
|
out_degree: Optional[int] = None
|
||||||
|
score: Optional[float] = None
|
||||||
|
section_title: Optional[str] = None
|
||||||
|
section_path: Optional[str] = None
|
||||||
|
path: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
class EdgeDTO(BaseModel):
|
||||||
|
"""Darstellung einer Kante im API-Graph."""
|
||||||
|
id: str
|
||||||
|
kind: EdgeKind
|
||||||
|
source: str
|
||||||
|
target: str
|
||||||
|
weight: float
|
||||||
|
direction: Literal["out", "in", "undirected"] = "out"
|
||||||
|
|
||||||
|
|
||||||
|
class QueryRequest(BaseModel):
|
||||||
|
"""
|
||||||
|
Request für /query:
|
||||||
|
- mode: 'semantic' | 'edge' | 'hybrid'
|
||||||
|
- query: (optional) Freitext; Embedding wird später angebunden
|
||||||
|
- query_vector: (optional) direkter Vektor (384d) für Quick-Tests ohne Embedding
|
||||||
|
"""
|
||||||
|
mode: Literal["semantic", "edge", "hybrid"] = "hybrid"
|
||||||
|
query: Optional[str] = None
|
||||||
|
query_vector: Optional[List[float]] = None
|
||||||
|
top_k: int = 10
|
||||||
|
expand: Dict = {"depth": 1, "edge_types": ["references", "belongs_to", "prev", "next"]}
|
||||||
|
filters: Optional[Dict] = None
|
||||||
|
ret: Dict = {"with_paths": True, "with_notes": True, "with_chunks": True}
|
||||||
|
|
||||||
|
|
||||||
|
class QueryHit(BaseModel):
|
||||||
|
"""Einzelnes Trefferobjekt für /query."""
|
||||||
|
node_id: str
|
||||||
|
note_id: Optional[str]
|
||||||
|
semantic_score: float
|
||||||
|
edge_bonus: float
|
||||||
|
centrality_bonus: float
|
||||||
|
total_score: float
|
||||||
|
paths: Optional[List[List[Dict]]] = None
|
||||||
|
source: Optional[Dict] = None
|
||||||
|
|
||||||
|
|
||||||
|
class QueryResponse(BaseModel):
|
||||||
|
"""Antwortstruktur für /query (Liste von Treffern + Telemetrie)."""
|
||||||
|
results: List[QueryHit]
|
||||||
|
used_mode: str
|
||||||
|
latency_ms: int
|
||||||
|
|
||||||
|
|
||||||
|
class GraphResponse(BaseModel):
|
||||||
|
"""Antwortstruktur für /graph/{note_id} (Nachbarschaft)."""
|
||||||
|
center_note_id: str
|
||||||
|
nodes: List[NodeDTO]
|
||||||
|
edges: List[EdgeDTO]
|
||||||
|
stats: Dict[str, int]
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
"""
|
||||||
|
app/models/__init__.py — Package-Init für mindnet DTOs
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Stellt zentrale API-/DTO-Modelle als Import-Fassade bereit.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, Pydantic 2.x
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage für WP-04)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
Handbuch / Knowledge Design (WP-03/04)
|
||||||
|
Nutzung:
|
||||||
|
from app.models import QueryRequest, QueryResponse, GraphResponse
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .dto import (
|
||||||
|
EdgeKind,
|
||||||
|
NodeDTO,
|
||||||
|
EdgeDTO,
|
||||||
|
QueryRequest,
|
||||||
|
QueryHit,
|
||||||
|
QueryResponse,
|
||||||
|
GraphResponse,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"EdgeKind",
|
||||||
|
"NodeDTO",
|
||||||
|
"EdgeDTO",
|
||||||
|
"QueryRequest",
|
||||||
|
"QueryHit",
|
||||||
|
"QueryResponse",
|
||||||
|
"GraphResponse",
|
||||||
|
]
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""
|
||||||
|
app/routers/admin.py — Admin-/Monitoring-Endpunkte (optional)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Liefert einfache Kennzahlen zu Collections (Counts) und Config.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, FastAPI 0.110+, qdrant-client 1.x
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
- Qdrant Collections: *_notes, *_chunks, *_edges
|
||||||
|
Nutzung:
|
||||||
|
app.include_router(admin.router, prefix="/admin", tags=["admin"])
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from fastapi import APIRouter
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from app.config import get_settings
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
@router.get("/stats")
|
||||||
|
def stats():
|
||||||
|
s = get_settings()
|
||||||
|
client = QdrantClient(url=s.QDRANT_URL, api_key=s.QDRANT_API_KEY)
|
||||||
|
notes = f"{s.COLLECTION_PREFIX}_notes"
|
||||||
|
chunks = f"{s.COLLECTION_PREFIX}_chunks"
|
||||||
|
edges = f"{s.COLLECTION_PREFIX}_edges"
|
||||||
|
|
||||||
|
def _count(col: str) -> int:
|
||||||
|
try:
|
||||||
|
return client.count(collection_name=col, exact=True).count
|
||||||
|
except Exception:
|
||||||
|
return -1
|
||||||
|
|
||||||
|
return {
|
||||||
|
"collections": {
|
||||||
|
"notes": {"name": notes, "count": _count(notes)},
|
||||||
|
"chunks": {"name": chunks, "count": _count(chunks)},
|
||||||
|
"edges": {"name": edges, "count": _count(edges)},
|
||||||
|
},
|
||||||
|
"config": {
|
||||||
|
"qdrant": s.QDRANT_URL,
|
||||||
|
"prefix": s.COLLECTION_PREFIX,
|
||||||
|
"vector_size": s.VECTOR_SIZE,
|
||||||
|
"distance": s.DISTANCE,
|
||||||
|
"retriever": {
|
||||||
|
"w_sem": s.RETRIEVER_W_SEM,
|
||||||
|
"w_edge": s.RETRIEVER_W_EDGE,
|
||||||
|
"w_cent": s.RETRIEVER_W_CENT,
|
||||||
|
"top_k": s.RETRIEVER_TOP_K,
|
||||||
|
"expand_depth": s.RETRIEVER_EXPAND_DEPTH,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
"""
|
||||||
|
app/routers/graph.py — Graph-Endpunkte (WP-04)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Liefert die Nachbarschaft einer Note/ID als JSON-Graph (Nodes/Edges/Stats).
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, FastAPI 0.110+, qdrant-client 1.x
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
- app/core/graph_adapter.py
|
||||||
|
- app/models/dto.py
|
||||||
|
Nutzung:
|
||||||
|
app.include_router(graph.router, prefix="/graph", tags=["graph"])
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import List, Optional
|
||||||
|
from fastapi import APIRouter, Query
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from app.models.dto import GraphResponse, NodeDTO, EdgeDTO
|
||||||
|
from app.core.graph_adapter import expand
|
||||||
|
from app.config import get_settings
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
@router.get("/{note_id}", response_model=GraphResponse)
|
||||||
|
def get_graph(note_id: str, depth: int = 1, edge_types: Optional[List[str]] = Query(None)) -> GraphResponse:
|
||||||
|
s = get_settings()
|
||||||
|
client = QdrantClient(url=s.QDRANT_URL, api_key=s.QDRANT_API_KEY)
|
||||||
|
|
||||||
|
sg = expand(client, s.COLLECTION_PREFIX, [note_id], depth=depth, edge_types=edge_types)
|
||||||
|
|
||||||
|
# Seed-Node (Payload optional später ergänzen über get_note_payload)
|
||||||
|
nodes = [NodeDTO(id=note_id, type="note",
|
||||||
|
title=None, tags=[],
|
||||||
|
in_degree=sg.in_degree.get(note_id, 0),
|
||||||
|
out_degree=sg.out_degree.get(note_id, 0))]
|
||||||
|
|
||||||
|
# direkte Out-Kanten des Seeds
|
||||||
|
edges = []
|
||||||
|
for e in sg.adj.get(note_id, []):
|
||||||
|
edges.append(EdgeDTO(
|
||||||
|
id=f"{note_id}->{e['target']}:{e['kind']}",
|
||||||
|
kind=e["kind"], source=note_id, target=e["target"],
|
||||||
|
weight=float(e["weight"]), direction="out"
|
||||||
|
))
|
||||||
|
|
||||||
|
return GraphResponse(center_note_id=note_id, nodes=nodes, edges=edges,
|
||||||
|
stats={"node_count": len(nodes), "edge_count": len(edges)})
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
"""
|
||||||
|
app/routers/query.py — Query-Endpunkte (WP-04)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Stellt POST /query bereit und ruft den passenden Retriever-Modus auf.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, FastAPI 0.110+
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
- app/core/retriever.py
|
||||||
|
- app/models/dto.py
|
||||||
|
Nutzung:
|
||||||
|
app.include_router(query.router, prefix="/query", tags=["query"])
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from fastapi import APIRouter, HTTPException
|
||||||
|
from app.models.dto import QueryRequest, QueryResponse
|
||||||
|
from app.core.retriever import hybrid_retrieve, semantic_retrieve
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
@router.post("", response_model=QueryResponse)
|
||||||
|
def post_query(req: QueryRequest) -> QueryResponse:
|
||||||
|
try:
|
||||||
|
if req.mode == "semantic":
|
||||||
|
return semantic_retrieve(req)
|
||||||
|
# default: hybrid
|
||||||
|
return hybrid_retrieve(req)
|
||||||
|
except ValueError as e:
|
||||||
|
raise HTTPException(status_code=400, detail=str(e))
|
||||||
|
except Exception as e:
|
||||||
|
raise HTTPException(status_code=500, detail=f"query failed: {e}")
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
"""
|
||||||
|
app/routers/tools.py — Tool-Definitionen für Ollama/n8n/MCP (read-only)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Liefert Funktions-Schemas (OpenAI-/Ollama-kompatibles Tool-JSON) für:
|
||||||
|
- mindnet_query -> POST /query
|
||||||
|
- mindnet_subgraph -> GET /graph/{note_id}
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, FastAPI 0.110+
|
||||||
|
Version:
|
||||||
|
0.1.1 (query ODER query_vector möglich)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Nutzung:
|
||||||
|
app.include_router(tools.router, prefix="/tools", tags=["tools"])
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.1 (2025-10-07) – mindnet_query: oneOf(query, query_vector).
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from fastapi import APIRouter
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
TOOLS = {
|
||||||
|
"tools": [
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "mindnet_query",
|
||||||
|
"description": "Hybrid-Retrieval über mindnet (Semantik + Edges).",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"oneOf": [
|
||||||
|
{"required": ["query"]},
|
||||||
|
{"required": ["query_vector"]}
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"query": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Freitext-Query; wird serverseitig in 384-d Embedding konvertiert."
|
||||||
|
},
|
||||||
|
"query_vector": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"type": "number"},
|
||||||
|
"description": "Direkter 384-d Query-Vektor (optional)."
|
||||||
|
},
|
||||||
|
"top_k": {"type":"integer","default":10,"minimum":1,"maximum":50},
|
||||||
|
"expand_depth": {"type":"integer","default":1,"minimum":0,"maximum":3},
|
||||||
|
"edge_types": {
|
||||||
|
"type":"array","items":{"type":"string"},
|
||||||
|
"default": ["references","belongs_to","prev","next"]
|
||||||
|
},
|
||||||
|
"filters": {"type":"object","description":"payload-Filter (tags etc.)"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "mindnet_subgraph",
|
||||||
|
"description": "Gibt die Nachbarschaft (Edges) einer Note/Seed-ID zurück.",
|
||||||
|
"parameters": {
|
||||||
|
"type":"object",
|
||||||
|
"properties": {
|
||||||
|
"note_id":{"type":"string"},
|
||||||
|
"depth":{"type":"integer","default":1,"minimum":0,"maximum":3},
|
||||||
|
"edge_types":{
|
||||||
|
"type":"array","items":{"type":"string"},
|
||||||
|
"default":["references","belongs_to","prev","next","backlink"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required":["note_id"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
@router.get("/ollama")
|
||||||
|
def get_ollama_tools():
|
||||||
|
return TOOLS
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
"""
|
||||||
|
app/services/embeddings_client.py — Text→Embedding (WP-04)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Liefert 384-d Embeddings für Textqueries (lazy load, einmal pro Prozess).
|
||||||
|
Standard: Sentence-Transformers (MODEL_NAME aus app.config.Settings).
|
||||||
|
Hinweis: Kein Netz-Zugriff; nutzt lokal installierte Modelle.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, sentence-transformers 5.x
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
- app/core/retriever.py (nutzt embed_text_if_needed)
|
||||||
|
- app/config.py (MODEL_NAME, VECTOR_SIZE)
|
||||||
|
Nutzung:
|
||||||
|
from app.services.embeddings_client import embed_text
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import List
|
||||||
|
from functools import lru_cache
|
||||||
|
from app.config import get_settings
|
||||||
|
|
||||||
|
# Lazy import, damit Testläufe ohne Modell-Laden schnell sind
|
||||||
|
def _load_model():
|
||||||
|
from sentence_transformers import SentenceTransformer # import hier, nicht top-level
|
||||||
|
s = get_settings()
|
||||||
|
return SentenceTransformer(s.MODEL_NAME, device="cpu")
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def _cached_model():
|
||||||
|
return _load_model()
|
||||||
|
|
||||||
|
def embed_text(text: str) -> List[float]:
|
||||||
|
"""
|
||||||
|
Erzeugt einen 384-d Vektor (oder laut Settings.VECTOR_SIZE) für den gegebenen Text.
|
||||||
|
"""
|
||||||
|
if not text or not text.strip():
|
||||||
|
raise ValueError("embed_text: leerer Text")
|
||||||
|
model = _cached_model()
|
||||||
|
vec = model.encode([text], normalize_embeddings=True)[0]
|
||||||
|
return vec.astype(float).tolist()
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""
|
||||||
|
app/services/llm_ollama.py — Ollama-Integration & Prompt-Bau (WP-04)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Prompt-Template & (optionaler) lokaler Aufruf von Ollama. Der Aufruf ist
|
||||||
|
bewusst gekapselt und kann gefahrlos deaktiviert bleiben, bis ihr ein
|
||||||
|
konkretes Modell konfigurieren wollt.
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
WP-04/05 Kontextbereitstellung für LLM
|
||||||
|
Nutzung:
|
||||||
|
from app.services.llm_ollama import build_prompt, call_ollama
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from typing import List, Dict, Optional
|
||||||
|
import subprocess
|
||||||
|
import json
|
||||||
|
|
||||||
|
PROMPT_TEMPLATE = """System: You are a helpful expert.
|
||||||
|
User: {question}
|
||||||
|
|
||||||
|
Context (ranked):
|
||||||
|
{contexts}
|
||||||
|
|
||||||
|
Task: Answer precisely. At the end, list sources (note title + section) and important edge paths.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def build_context_block(items: List[Dict]) -> str:
|
||||||
|
"""Formatiert Top-K-Kontexte (Chunks) für den Prompt."""
|
||||||
|
lines = []
|
||||||
|
for i, it in enumerate(items, 1):
|
||||||
|
note = it.get("note_title", "") or it.get("note_id", "")
|
||||||
|
sec = it.get("section", "") or it.get("section_title", "")
|
||||||
|
sc = it.get("score", 0)
|
||||||
|
txt = it.get("text", "") or it.get("body", "") or ""
|
||||||
|
lines.append(f"{i}) {note} — {sec} [score={sc:.2f}]\n{txt}\n")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def build_prompt(question: str, contexts: List[Dict]) -> str:
|
||||||
|
"""Setzt Frage + Kontexte in ein konsistentes Template."""
|
||||||
|
return PROMPT_TEMPLATE.format(question=question, contexts=build_context_block(contexts))
|
||||||
|
|
||||||
|
|
||||||
|
def call_ollama(prompt: str, model: str = "llama3.1:8b", timeout_s: int = 120) -> Optional[str]:
|
||||||
|
"""
|
||||||
|
Optionaler lokaler Aufruf von `ollama run`.
|
||||||
|
Rückgabe: generierter Text oder None bei Fehler/Abbruch.
|
||||||
|
Hinweis: Nur nutzen, wenn Ollama lokal installiert/konfiguriert ist.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(
|
||||||
|
["ollama", "run", model],
|
||||||
|
input=prompt.encode("utf-8"),
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
timeout=timeout_s,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
out = proc.stdout.decode("utf-8", errors="replace")
|
||||||
|
# viele ollama Builds streamen JSON-Zeilen; robust extrahieren:
|
||||||
|
try:
|
||||||
|
# Falls JSONL, letztes "response" zusammenfassen
|
||||||
|
texts = []
|
||||||
|
for line in out.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
obj = json.loads(line)
|
||||||
|
if "response" in obj:
|
||||||
|
texts.append(obj["response"])
|
||||||
|
except Exception:
|
||||||
|
texts.append(line)
|
||||||
|
return "".join(texts).strip()
|
||||||
|
except Exception:
|
||||||
|
return out.strip()
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
version: 1.0
|
||||||
|
|
||||||
|
scoring:
|
||||||
|
# W_sem: skaliert den Term (semantic_score * retriever_weight)
|
||||||
|
# Empfehlung Startwert: 1.0 → Semantik bleibt Hauptsignal
|
||||||
|
semantic_weight: 1.0
|
||||||
|
|
||||||
|
# W_edge: skaliert edge_bonus aus dem Subgraph
|
||||||
|
# Empfehlung: 0.7 → Graph ist deutlich spürbar, aber überstimmt Semantik nicht komplett
|
||||||
|
edge_weight: 0.7
|
||||||
|
|
||||||
|
# W_cent: skaliert centrality_bonus (Knoten-Zentralität im Subgraph)
|
||||||
|
# Empfehlung: 0.5 → zentrale Knoten werden bevorzugt, aber moderat
|
||||||
|
centrality_weight: 0.5
|
||||||
|
|
||||||
|
# Optional, für später (aktuell noch nicht ausgewertet, aber als Blaupause):
|
||||||
|
edge_types:
|
||||||
|
references: 0.20
|
||||||
|
depends_on: 0.18
|
||||||
|
related_to: 0.15
|
||||||
|
similar_to: 0.12
|
||||||
|
belongs_to: 0.10
|
||||||
|
next: 0.06
|
||||||
|
prev: 0.06
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
version: 1.0
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
retriever_weight: 1.0
|
||||||
|
chunk_profile: default
|
||||||
|
edge_defaults: [] # Leerliste → keine impliziten Kanten, wenn nicht gesetzt
|
||||||
|
|
||||||
|
types:
|
||||||
|
# Wissensbausteine
|
||||||
|
concept:
|
||||||
|
chunk_profile: medium
|
||||||
|
retriever_weight: 0.60
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
|
||||||
|
source:
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.50
|
||||||
|
edge_defaults: [] # Quellen sind i. d. R. Ziel von Kanten, erzeugen selbst keine Default-Kanten
|
||||||
|
|
||||||
|
glossary:
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.40
|
||||||
|
edge_defaults: ["related_to"]
|
||||||
|
|
||||||
|
# Person & Persönlichkeit
|
||||||
|
profile: # z. B. "Persönliches Leitbild (Kurzprofil)"
|
||||||
|
chunk_profile: long
|
||||||
|
retriever_weight: 0.66
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
|
||||||
|
value: # Werte (z. B. "Mut", "Fairness")
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["related_to"]
|
||||||
|
|
||||||
|
principle: # Prinzipien/Leitsätze
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["related_to", "references"]
|
||||||
|
|
||||||
|
experience: # prägende Erlebnisse/Erfahrungen
|
||||||
|
chunk_profile: medium
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["related_to", "references"]
|
||||||
|
|
||||||
|
# Ziele, Entscheidungen, Meilensteine
|
||||||
|
goal:
|
||||||
|
chunk_profile: medium
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["depends_on", "related_to", "references"]
|
||||||
|
|
||||||
|
decision:
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.90
|
||||||
|
edge_defaults: ["depends_on", "references", "related_to"] # „Worauf stützte sich die Entscheidung?“
|
||||||
|
|
||||||
|
milestone:
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.70
|
||||||
|
edge_defaults: ["related_to", "references"]
|
||||||
|
|
||||||
|
# Operativ: Projekte, Aufgaben, Journal
|
||||||
|
project:
|
||||||
|
chunk_profile: long
|
||||||
|
retriever_weight: 0.97
|
||||||
|
edge_defaults: ["references", "depends_on"] # Projekt stützt sich auf Konzepte/Quellen und hängt von Bausteinen ab
|
||||||
|
|
||||||
|
task:
|
||||||
|
chunk_profile: short
|
||||||
|
retriever_weight: 0.80
|
||||||
|
edge_defaults: ["depends_on", "references"]
|
||||||
|
|
||||||
|
journal:
|
||||||
|
chunk_profile: medium
|
||||||
|
retriever_weight: 0.80
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Changelog — WP-03 Type-Registry (2025-11-08)
|
||||||
|
|
||||||
|
## Added
|
||||||
|
- `app/core/type_registry.py` — Loader & Helper (cached), robust Fallbacks.
|
||||||
|
- `config/types.yaml` — Beispiel-Registry (version 1.0).
|
||||||
|
- `scripts/import_markdown.py` v3.8.0 — Registry-Integration:
|
||||||
|
- typvalidierung (unbekannt → `concept`, Warnung)
|
||||||
|
- `chunk_profile` → an `make_chunk_payloads(...)`
|
||||||
|
- `retriever_weight` → persistiert im Note-Payload
|
||||||
|
- `edge_defaults["references"]` schaltet Note-Scope-Refs additiv ein
|
||||||
|
- `app/core/chunk_payload.py` v2.3.0 — optionales `chunk_profile` für
|
||||||
|
synthetische Overlaps (keine Schemaänderung).
|
||||||
|
- `app/core/derive_edges.py` v1.5.0 — Type-Registry aktiviert note-scope
|
||||||
|
References/Backlinks additiv.
|
||||||
|
|
||||||
|
## Changed
|
||||||
|
- Keine Breaking Changes. Alle CLI-Flags, ENV-Variablen und Collection-
|
||||||
|
Prefix-Logik bleiben unverändert.
|
||||||
|
|
||||||
|
## Removed
|
||||||
|
- N/A
|
||||||
+222
-63
@@ -1,19 +1,42 @@
|
|||||||
# 📘 Mindnet Skript-Handbuch
|
# 📘 Mindnet Skript-Handbuch (aktualisiert)
|
||||||
|
|
||||||
Dieses Handbuch dokumentiert die implementierten Skripte für das Projekt **mindnet Wissensnetzwerk**.
|
Dieses Handbuch dokumentiert die Skripte und Tests für das Projekt **mindnet Wissensnetzwerk**.
|
||||||
Es beschreibt Zweck, Parameter und typische Aufrufe. Alle Beispiele sind geprüft und lauffähig.
|
Es beschreibt Zweck, Parameter, typische Aufrufe sowie aktuelle Verhaltensweisen (Hash-Modi, Parser-Robustheit, `window`/`text`-Trennung in Chunks, Export-Fixes).
|
||||||
|
Alle Beispiele sind so formatiert, dass sie direkt kopierbar sind. **Code/CLI-Beispiele sind eingerückt**, damit die Markdown-Darstellung nicht abbricht.
|
||||||
Alle Skripte liegen in `~/mindnet/scripts/`.
|
|
||||||
Ausführung erfolgt **im aktivierten venv**:
|
|
||||||
|
|
||||||
cd ~/mindnet
|
|
||||||
source .venv/bin/activate
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Vault anlegen (für Tests)
|
## 0) Umgebung & Konfiguration
|
||||||
|
|
||||||
### `make_test_vault.py`
|
**Wichtige ENV-Variablen**
|
||||||
|
|
||||||
|
- `QDRANT_URL` – z. B. `http://localhost:6333`
|
||||||
|
- `QDRANT_API_KEY` – API-Key (falls aktiviert)
|
||||||
|
- `COLLECTION_PREFIX` – Prefix der drei Collections, Standard: `mindnet` → `mindnet_notes`, `mindnet_chunks`, `mindnet_edges`
|
||||||
|
- `VECTOR_DIM` – Embedding-Dimension (Chunks), Standard: `384`
|
||||||
|
|
||||||
|
**Hash-Vergleich & Änderungserkennung**
|
||||||
|
|
||||||
|
- `MINDNET_HASH_COMPARE` – steuert, **welcher Anteil** für „changed“ verglichen wird
|
||||||
|
Werte: `Body` (Standard), `Frontmatter`, `Full`
|
||||||
|
- `MINDNET_HASH_SOURCE` – Quelle: `parsed` (Standard), `raw`, `file`
|
||||||
|
- `MINDNET_HASH_NORMALIZE` – Normalisierung: `canonical` (Standard) oder `none`
|
||||||
|
|
||||||
|
**Beispiel (nur Body berücksichtigen, kanonisch, geparst):**
|
||||||
|
|
||||||
|
export MINDNET_HASH_COMPARE=Body
|
||||||
|
export MINDNET_HASH_SOURCE=parsed
|
||||||
|
export MINDNET_HASH_NORMALIZE=canonical
|
||||||
|
source .venv/bin/activate
|
||||||
|
|
||||||
|
**Baseline-Signaturen (mehrere Modi):**
|
||||||
|
Der Importer kann **Signaturen aller Modi** (Body/Frontmatter/Full) vorhalten, damit spätere Moduswechsel deterministisch sind (keine „Mass-Changes“). Siehe `--baseline-modes` unter **Importer**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1) Vault anlegen (für Tests)
|
||||||
|
|
||||||
|
### `scripts/make_test_vault.py`
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Erstellt einen kleinen, nachvollziehbaren Test-Vault mit Markdown-Dateien.
|
Erstellt einen kleinen, nachvollziehbaren Test-Vault mit Markdown-Dateien.
|
||||||
@@ -25,14 +48,14 @@ Szenarien: offene Links, nachträgliche Anlage fehlender Notes, geänderte Chunk
|
|||||||
- `--with-missing` : Optional – auch die `missing-note` gleich anlegen
|
- `--with-missing` : Optional – auch die `missing-note` gleich anlegen
|
||||||
|
|
||||||
**Hinweise für Tests**
|
**Hinweise für Tests**
|
||||||
1) Erster Import (dry-run oder --apply): Es gibt einen „leeren“ Link [[missing-note]].
|
1. Erster Import (dry-run oder `--apply`): Es gibt einen „leeren“ Link `[[missing-note]]`.
|
||||||
2) Lege danach eine Datei für „missing-note“ an (mit gleicher ID im YAML) und importiere erneut:
|
2. Lege danach eine Datei für „missing-note“ an (mit gleicher `id` im YAML) und importiere erneut:
|
||||||
-> Erwartung: Edges für ehemals leeren Link werden korrekt nachgezogen (references + backlink).
|
→ Erwartung: Edges für ehemals leeren Link werden korrekt nachgezogen (`references` + `backlink`).
|
||||||
3) Ändere den Body von exp-one.md so, dass andere Chunk-Grenzen entstehen und importiere erneut:
|
3. Ändere den Body von `exp-one.md` so, dass andere Chunk-Grenzen entstehen und importiere erneut:
|
||||||
-> Erwartung: Chunks/Edges/Note werden für betroffene Noten aktualisiert.
|
→ Erwartung: Chunks/Edges/Note werden für betroffene Noten aktualisiert.
|
||||||
|
|
||||||
Kompatibel mit:
|
**Kompatibilität:**
|
||||||
- note.schema.json: 'created'/'updated' müssen strings sein. (Wichtig!)
|
- `note.schema.json`: `created`/`updated` sind Strings.
|
||||||
|
|
||||||
**Beispiele:**
|
**Beispiele:**
|
||||||
|
|
||||||
@@ -41,41 +64,60 @@ Kompatibel mit:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Import in Qdrant
|
## 2) Import in Qdrant
|
||||||
|
|
||||||
### `import_markdown.py`
|
### `scripts/import_markdown.py`
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Importiert Notes/Chunks/Edges aus einem Vault nach Qdrant.
|
Importiert Notes/Chunks/Edges aus einem Vault nach Qdrant.
|
||||||
- prüft Frontmatter
|
- validiert Frontmatter
|
||||||
- chunked Inhalte + Embeddings
|
- chunked Inhalte + Embeddings
|
||||||
- erzeugt Edges (`references`, `references_at`, `backlink`)
|
- erzeugt Edges (`references`, `references_at`, `backlink`, `next`, `prev`, `belongs_to`)
|
||||||
- erkennt geänderte Dateien automatisch
|
- erkennt geänderte Dateien **modusgesteuert** (Body/Frontmatter/Full)
|
||||||
- optional: Purge der alten Chunks/Edges pro Note
|
- optional: Purge der alten Chunks/Edges pro Note
|
||||||
|
|
||||||
**Parameter:**
|
**Parameter:**
|
||||||
- `--vault PATH` : (Pflicht) Vault-Verzeichnis
|
- `--vault PATH` : (Pflicht) Vault-Verzeichnis
|
||||||
- `--apply` : Änderungen wirklich durchführen (sonst Dry-Run)
|
- `--apply` : Änderungen wirklich durchführen (ohne: Dry-Run)
|
||||||
- `--purge-before-upsert` : Vor Upsert alte Chunks/Edges **nur der geänderten Notes** löschen
|
- `--purge-before-upsert` : Vor Upsert alte Chunks/Edges **nur der geänderten Notes** löschen
|
||||||
- `--note-id ID` : Nur eine bestimmte Note importieren
|
- `--prefix STR` : überschreibt `COLLECTION_PREFIX`
|
||||||
|
- `--note-id ID` : nur eine bestimmte Note importieren
|
||||||
|
- `--note-scope-refs` : zusätzlich **note-weite** `references` erzeugen (nicht nur chunk-weise)
|
||||||
|
- `--baseline-modes` : berechne & speichere Hash-Signaturen für **alle** Modi (`Body`, `Frontmatter`, `Full`) als Baseline
|
||||||
|
|
||||||
|
**Erwartetes Verhalten:**
|
||||||
|
- Unveränderte Notes → `"changed": false`, Entscheidung: `apply-skip-unchanged` (bei `--apply`)
|
||||||
|
- Geänderte Notes → `"changed": true` (abhängig von `MINDNET_HASH_COMPARE`)
|
||||||
|
- Idempotenz: mehrfacher Lauf ohne Änderungen erzeugt **keine** neuen Punkte
|
||||||
|
- Baseline: Bei `--baseline-modes` erscheinen Felder wie `hash_signature`, `hash_fulltext`, ggf. `hash_frontmatter` und `needs_baseline_for_mode: false` bei Folgeläufen.
|
||||||
|
|
||||||
**Beispiele:**
|
**Beispiele:**
|
||||||
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault
|
# Dry-Run (zeigt Änderungen, schreibt NICHT)
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply
|
python3 -m scripts.import_markdown --vault ./vault --prefix mindnet
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply --purge-before-upsert
|
|
||||||
|
# Apply + gezielte Vorreinigung pro geänderter Note
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --prefix mindnet --apply --purge-before-upsert
|
||||||
|
|
||||||
|
# Baseline-Signaturen aller Modi vorhalten
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --apply --baseline-modes
|
||||||
|
|
||||||
|
**Hash-Modi in der Praxis:**
|
||||||
|
- `MINDNET_HASH_COMPARE=Body` → Änderungen im Body werden erkannt; reine Frontmatter-Änderungen bleiben **ohne Effekt**.
|
||||||
|
- `MINDNET_HASH_COMPARE=Frontmatter` → nur Frontmatter maßgeblich.
|
||||||
|
- `MINDNET_HASH_COMPARE=Full` → beide Anteile zählen.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Konsistenzprüfungen
|
## 3) Konsistenzprüfungen
|
||||||
|
|
||||||
### `audit_vault_vs_qdrant.py`
|
### `scripts/audit_vault_vs_qdrant.py`
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Vergleicht Vault-Inhalte mit Qdrant.
|
Vergleicht Vault-Inhalte mit Qdrant.
|
||||||
- Zählt Notes/Chunks/Edges
|
- Zählt Notes/Chunks/Edges
|
||||||
- Verifiziert Anzahl Wikilinks vs. Edges
|
- Verifiziert Anzahl Wikilinks vs. Edges
|
||||||
- Meldet Mismatches pro Note
|
- Meldet Deltas (z. B. `references` im Vault vs. in Qdrant)
|
||||||
|
|
||||||
**Parameter:**
|
**Parameter:**
|
||||||
- `--vault PATH` : Vault-Verzeichnis
|
- `--vault PATH` : Vault-Verzeichnis
|
||||||
@@ -85,15 +127,13 @@ Vergleicht Vault-Inhalte mit Qdrant.
|
|||||||
|
|
||||||
python3 -m scripts.audit_vault_vs_qdrant --vault ./vault --prefix mindnet
|
python3 -m scripts.audit_vault_vs_qdrant --vault ./vault --prefix mindnet
|
||||||
|
|
||||||
---
|
### `scripts/validate_edges.py`
|
||||||
|
|
||||||
### `validate_edges.py`
|
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Prüft die Edges-Collection in Qdrant.
|
Prüft die Edges-Collection in Qdrant.
|
||||||
- Zählt Edges pro Typ
|
- Zählt Edges pro Typ (`references`, `references_at`, `backlink`, `next`, `prev`, `belongs_to`)
|
||||||
- Kontrolliert Invarianten (z. B. jede Reference hat Backlink)
|
- Kontrolliert Invarianten (z. B. jede `references` hat einen `backlink`)
|
||||||
- Listet unresolved Edges auf
|
- Listet `unresolved` Edges auf
|
||||||
|
|
||||||
**Parameter:**
|
**Parameter:**
|
||||||
- `--prefix STR` : Prefix für Qdrant-Collections
|
- `--prefix STR` : Prefix für Qdrant-Collections
|
||||||
@@ -105,35 +145,69 @@ Prüft die Edges-Collection in Qdrant.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Qdrant zurücksetzen
|
## 4) Export Qdrant → Markdown (Vault)
|
||||||
|
|
||||||
### `reset_qdrant.py`
|
### `scripts/export_markdown.py`
|
||||||
|
|
||||||
|
**Zweck:**
|
||||||
|
Exportiert Notes + rekonstruierten Body in einen Zielordner.
|
||||||
|
- Pfade werden **relativ** aus `note.payload.path` abgeleitet (führende `/` entfernt, Backslashes → Slashes; Unicode-NFC empfohlen)
|
||||||
|
- **Body-Rekonstruktion** priorisiert:
|
||||||
|
1. `fulltext` (falls in Note-Payload vorhanden)
|
||||||
|
2. sonst **Chunks**: Sortierung `seq` → `chunk_index` → Nummer in `chunk_id`; **nur `text`** (Overlap bereits entfernt)
|
||||||
|
- `--prefix` (CLI) wird unterstützt (oder ENV `COLLECTION_PREFIX`)
|
||||||
|
|
||||||
|
**Parameter:**
|
||||||
|
- `--out PATH` : Ziel-Root (muss existieren)
|
||||||
|
- `--overwrite` : vorhandene Dateien überschreiben
|
||||||
|
- `--note-id ID` : nur eine bestimmte Note exportieren
|
||||||
|
- `--prefix STR` : überschreibt `COLLECTION_PREFIX`
|
||||||
|
- `--include-edges {none,footer,yaml}` *(optional, falls aktiviert)*: Edges als YAML-Liste im Frontmatter oder im Footer beilegen
|
||||||
|
|
||||||
|
**Beispiele:**
|
||||||
|
|
||||||
|
export COLLECTION_PREFIX="mindnet"
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault
|
||||||
|
|
||||||
|
# Nur eine bestimmte Note:
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault --note-id 20250821-architektur-ki-trainerassistent-761cfe
|
||||||
|
|
||||||
|
# Existierende Dateien überschreiben:
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault --overwrite
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5) Qdrant zurücksetzen
|
||||||
|
|
||||||
|
### `scripts/reset_qdrant.py`
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Setzt die Qdrant-Collections zurück.
|
Setzt die Qdrant-Collections zurück.
|
||||||
- „wipe“: löscht Collections komplett (inkl. Schema)
|
- `wipe`: löscht Collections komplett (inkl. Schema)
|
||||||
- „truncate“: löscht nur Inhalte, behält Collections
|
- `truncate`: löscht nur Inhalte, behält Collections
|
||||||
|
- `truncate-edges`: löscht nur Punkte in `*_edges`
|
||||||
|
|
||||||
**Parameter:**
|
**Parameter:**
|
||||||
- `--mode {wipe,truncate}` : Aktion
|
- `--mode {wipe,truncate,truncate-edges}`
|
||||||
- `--prefix STR` : Prefix der Collections (Default: `mindnet`)
|
- `--prefix STR` : Prefix der Collections (Default: `mindnet`)
|
||||||
- `--yes` : Automatische Bestätigung
|
- `--yes` : Automatische Bestätigung
|
||||||
- `--dry-run` : Nur anzeigen, nicht ausführen
|
- `--dry-run` : Nur anzeigen, nicht ausführen
|
||||||
|
|
||||||
**Beispiele:**
|
**Beispiele:**
|
||||||
|
|
||||||
|
python3 -m scripts.reset_qdrant --mode truncate-edges --prefix mindnet
|
||||||
python3 -m scripts.reset_qdrant --mode truncate --prefix mindnet --yes
|
python3 -m scripts.reset_qdrant --mode truncate --prefix mindnet --yes
|
||||||
python3 -m scripts.reset_qdrant --mode wipe --prefix mindnet --yes
|
python3 -m scripts.reset_qdrant --mode wipe --prefix mindnet --yes
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Diagnose
|
## 6) Diagnose & Tools
|
||||||
|
|
||||||
### `diagnose_changed.py`
|
### `scripts/diagnose_changed.py`
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Prüft, ob die Datei-Hashes (`sha256`) mit den gespeicherten `hash_fulltext` in Qdrant übereinstimmen.
|
Prüft, ob Datei-Hashes (`sha256`) mit gespeicherten `hash_*` in Qdrant übereinstimmen.
|
||||||
Hilft, wenn der Importer zu viele Dateien als „changed“ erkennt.
|
Hilft, wenn der Importer zu viele Dateien als „changed“ erkennt oder Moduswechsel geplant sind.
|
||||||
|
|
||||||
**Parameter:**
|
**Parameter:**
|
||||||
- `--vault PATH` : Vault-Verzeichnis
|
- `--vault PATH` : Vault-Verzeichnis
|
||||||
@@ -145,14 +219,11 @@ Hilft, wenn der Importer zu viele Dateien als „changed“ erkennt.
|
|||||||
|
|
||||||
python3 -m scripts.diagnose_changed --vault ./vault --prefix mindnet --all
|
python3 -m scripts.diagnose_changed --vault ./vault --prefix mindnet --all
|
||||||
|
|
||||||
---
|
### `scripts/resolve_unresolved_references.py`
|
||||||
|
|
||||||
## 6. Unresolved Links nachträglich auflösen
|
|
||||||
|
|
||||||
### `resolve_unresolved_references.py`
|
|
||||||
|
|
||||||
**Zweck:**
|
**Zweck:**
|
||||||
Prüft offene Wikilinks (`status=unresolved`) und fügt Edges nach, wenn inzwischen passende Notizen existieren.
|
Prüft offene Wikilinks (`status=unresolved`) und fügt Edges nach, wenn inzwischen passende Notizen existieren.
|
||||||
|
Ersetzt `unresolved` durch gültige `references`/`backlink`.
|
||||||
|
|
||||||
**Parameter:**
|
**Parameter:**
|
||||||
- `--prefix STR` : Prefix für Qdrant-Collections
|
- `--prefix STR` : Prefix für Qdrant-Collections
|
||||||
@@ -165,26 +236,114 @@ Prüft offene Wikilinks (`status=unresolved`) und fügt Edges nach, wenn inzwisc
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# ✅ Typische Workflows
|
## 7) Parser-Verhalten (robust & abwärtskompatibel)
|
||||||
|
|
||||||
### A) Vault neu importieren
|
- Der Parser liest Markdown **fehlertolerant** (UTF-8 mit Fallbacks), extrahiert YAML-Frontmatter und Body.
|
||||||
|
- Keine harten Abhängigkeiten von alten Regex-Symbolen (`FRONTMATTER_RE` entfällt zugunsten interner Marker).
|
||||||
|
- `read_markdown(path)` gibt `(frontmatter_dict, body_str)` zurück.
|
||||||
|
- **Sonderfälle aus Fremd-Tools** (z. B. Silverbullet-Dateien wie `CONFIG.md`, `index.md`) werden **nicht importiert**, sofern sie nicht zum Vault gehören.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8) Chunk-Payload (Fenster vs. Kern)
|
||||||
|
|
||||||
|
- **`window`** = kontextualisierter Fenster-Text (mit linkem Overlap)
|
||||||
|
- **`text`** = **Kerntext** ohne das vom vorherigen Chunk übernommene Overlap
|
||||||
|
- Felder: `start`, `end`, `overlap_left`, `overlap_right`, `chunk_index`, `seq`, `path`, `section_title`, `section_path`, `token_count` (falls vorhanden)
|
||||||
|
- **Fallback (synthetisch):** Falls der Chunker **keinen** Overlap-Fenstertext liefert, erzeugt die Payload-Schicht Fenster mit typgerechtem Overlap aus `chunk_config.get_sizes(type)['overlap']` (unterer Wert).
|
||||||
|
- **Edges** werden aus `window` abgeleitet (um Links an Chunk-Grenzen nicht zu verlieren); der **Export** rekonstruiert den Body aus `text` (Overlap-frei).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9) Edge-Modell & IDs
|
||||||
|
|
||||||
|
- Collections:
|
||||||
|
`*_notes` (optional Vektor / Nullvektor), `*_chunks` (384-d Embeddings), `*_edges` (1-d Dummy-Vektor)
|
||||||
|
- **Deterministische IDs** (UUIDv5 / stabile Keys):
|
||||||
|
- `note_id` stabil aus Frontmatter/Slug
|
||||||
|
- `chunk_id` i. d. R. `note_id#<n>`
|
||||||
|
- `edge_id` stabil aus `(kind, source_id, target_id, seq)`
|
||||||
|
- Kantenarten: `references`, `references_at`, `backlink`, `next`, `prev`, `belongs_to`
|
||||||
|
- `unresolved` wird gekennzeichnet und kann per Resolver später aufgelöst werden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10) Tests & Qualitätssicherung
|
||||||
|
|
||||||
|
**Tests unter `tests/`**
|
||||||
|
|
||||||
|
- `check_chunks_window_vs_text.py` – vergleicht je Chunk `window` vs. `text`, schätzt Overlap
|
||||||
|
- `compare_vaults.py` – rekursiver Vault-Vergleich (Body/Frontmatter/All, Key: auto/id/title/filename), Unicode-NFC-robust
|
||||||
|
- `verify_chunks_integrity.py` – prüft, ob Rekonstruktion aus Chunks (`text`) zu `fulltext` bzw. zum Vault-Body passt
|
||||||
|
- `list_md_seen_by_importer.py` – zeigt, welche `.md` der Importer sieht (inkl. Gründe/Filter)
|
||||||
|
- `inspect_one_note.py` – Parser-Diagnose für eine konkrete Datei
|
||||||
|
- `run_e2e_roundtrip.sh` – End-to-End: Reset → Import → Audit → (optional) Hash-Reporter → Export → Vergleich
|
||||||
|
|
||||||
|
**Typische Aufrufe:**
|
||||||
|
|
||||||
|
# Overlap-Qualität (Fenster vs. Kern)
|
||||||
|
python3 tests/check_chunks_window_vs_text.py --prefix "$COLLECTION_PREFIX"
|
||||||
|
|
||||||
|
# Roundtrip-Vergleich (rekursiv)
|
||||||
|
python3 tests/compare_vaults.py --src ./vault --dst ./_exportVault --focus body
|
||||||
|
|
||||||
|
# Chunks-Rekonstruktion (gegen Vault-Body)
|
||||||
|
python3 tests/verify_chunks_integrity.py --prefix "$COLLECTION_PREFIX" --vault ./vault
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11) Typische Workflows
|
||||||
|
|
||||||
|
**A) Clean-Start mit Test-Vault**
|
||||||
|
|
||||||
|
python3 -m scripts.make_test_vault --out ./test_vault --force
|
||||||
python3 -m scripts.reset_qdrant --mode truncate --prefix mindnet --yes
|
python3 -m scripts.reset_qdrant --mode truncate --prefix mindnet --yes
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply
|
python3 -m scripts.import_markdown --vault ./test_vault --prefix mindnet
|
||||||
python3 -m scripts.audit_vault_vs_qdrant --vault ./vault --prefix mindnet
|
python3 -m scripts.import_markdown --vault ./test_vault --prefix mindnet --apply --purge-before-upsert
|
||||||
|
python3 -m scripts.audit_vault_vs_qdrant --vault ./test_vault --prefix mindnet
|
||||||
|
python3 -m scripts.validate_edges --prefix mindnet --details
|
||||||
|
export COLLECTION_PREFIX="mindnet"
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault
|
||||||
|
|
||||||
### B) Änderungen einspielen (mit Purge für geänderte Notes)
|
**B) Änderungs-Propagation & Idempotenz**
|
||||||
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply --purge-before-upsert
|
python3 -m scripts.import_markdown --vault ./test_vault --prefix mindnet
|
||||||
python3 -m scripts.audit_vault_vs_qdrant --vault ./vault --prefix mindnet
|
python3 -m scripts.import_markdown --vault ./test_vault --prefix mindnet --apply --purge-before-upsert
|
||||||
|
python3 -m scripts.audit_vault_vs_qdrant --vault ./test_vault --prefix mindnet
|
||||||
|
python3 -m scripts.validate_edges --prefix mindnet --details
|
||||||
|
python3 -m scripts.import_markdown --vault ./test_vault --prefix mindnet
|
||||||
|
|
||||||
### C) Unresolved Links nachziehen
|
**C) Unresolved Links lösen**
|
||||||
|
|
||||||
|
python3 -m scripts.import_markdown --vault ./test_vault --apply --prefix mindnet
|
||||||
python3 -m scripts.resolve_unresolved_references --prefix mindnet --apply
|
python3 -m scripts.resolve_unresolved_references --prefix mindnet --apply
|
||||||
python3 -m scripts.validate_edges --prefix mindnet --details
|
python3 -m scripts.validate_edges --prefix mindnet --details
|
||||||
|
|
||||||
### D) Test-Vault Szenario durchspielen
|
**D) Export & Roundtrip (Produkt-Vault)**
|
||||||
|
|
||||||
python3 -m scripts.make_test_vault --out ./test_vault --force
|
export COLLECTION_PREFIX="mindnet"
|
||||||
python3 -m scripts.import_markdown --vault ./test_vault --apply
|
python3 -m scripts.import_markdown --vault ./vault --apply --purge-before-upsert --prefix "$COLLECTION_PREFIX"
|
||||||
python3 -m scripts.audit_vault_vs_qdrant --vault ./test_vault --prefix mindnet
|
python3 -m scripts.export_markdown --out ./_exportVault --prefix "$COLLECTION_PREFIX"
|
||||||
|
python3 tests/compare_vaults.py --src ./vault --dst ./_exportVault --focus body
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12) Troubleshooting (kurz)
|
||||||
|
|
||||||
|
- **Importer markiert alle als changed (Moduswechsel):**
|
||||||
|
→ `--baseline-modes` einmalig laufen lassen; sicherstellen: `MINDNET_HASH_COMPARE` passt zum Ziel.
|
||||||
|
- **Edges verschwinden / nur 1 Edge sichtbar:**
|
||||||
|
→ `app/core/qdrant_points.py` muss Edge-Payload normalisieren (Schema Alt/Neu) und `edge_id` eindeutig bilden.
|
||||||
|
- **Export ohne Body / falsches Zielverzeichnis:**
|
||||||
|
→ Exporter nutzt relativierte Pfade und rekonstruiert Body aus `fulltext` oder Chunks (`text`). ENV/CLI `--prefix` beachten.
|
||||||
|
- **Parser-Fehler (UTF-8):**
|
||||||
|
→ Neuer Parser arbeitet mit Fallbacks; mit `tests/inspect_one_note.py` prüfen, was ankommt.
|
||||||
|
- **„Fehlende“ Dateien im Vergleich:**
|
||||||
|
→ Prüfe, ob Dateien **wirklich zum Vault gehören** (z. B. Silverbullet-Artefakte `CONFIG.md`, `index.md` ausblenden).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13) Hinweise zu Silverbullet-Dateien
|
||||||
|
|
||||||
|
- Dateien wie `CONFIG.md` oder `index.md`, die **nicht** zum inhaltlichen Vault gehören, werden **nicht importiert** und erscheinen daher nicht im Export-Roundtrip.
|
||||||
|
- Diese Dateien bitte vom produktiven Vault trennen oder im Vergleich (`tests/compare_vaults.py
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
# Kurzhandbuch — Type-Registry (WP-03)
|
||||||
|
|
||||||
|
## Zweck
|
||||||
|
Die Type-Registry macht den `type:`-Wert im Frontmatter maschinenlesbar
|
||||||
|
und steuert:
|
||||||
|
- Chunk-Profile (indirekt über Overlap bei fensterlosen Chunks),
|
||||||
|
- typabhängige Default-Kanten (hier: Aktivierung von Note-Scope-References),
|
||||||
|
- optionale Retriever-Gewichtungen (`retriever_weight`) für WP‑04.
|
||||||
|
|
||||||
|
## Aktivierung
|
||||||
|
1. Legen Sie `config/types.yaml` an (siehe Beispiel in diesem Paket).
|
||||||
|
2. Starten Sie Import/Server neu. Die Registry wird beim Prozessstart
|
||||||
|
geladen und gecached.
|
||||||
|
|
||||||
|
Fehlt die Datei oder ist sie defekt, bleibt das System voll funktionsfähig
|
||||||
|
(der Typ `concept` wird genutzt; Warnhinweis im Log).
|
||||||
|
|
||||||
|
## Schema (Beispiel)
|
||||||
|
```yaml
|
||||||
|
version: 1.0
|
||||||
|
types:
|
||||||
|
concept:
|
||||||
|
chunk_profile: long
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
retriever_weight: 1.0
|
||||||
|
task:
|
||||||
|
chunk_profile: short
|
||||||
|
edge_defaults: ["depends_on", "belongs_to"]
|
||||||
|
retriever_weight: 0.8
|
||||||
|
experience:
|
||||||
|
chunk_profile: medium
|
||||||
|
edge_defaults: ["derived_from", "inspired_by"]
|
||||||
|
retriever_weight: 0.9
|
||||||
|
project:
|
||||||
|
chunk_profile: medium
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
retriever_weight: 0.95
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verhalten
|
||||||
|
- **Chunking**: Die eigentliche Chunk-Länge bleibt weiterhin typabhängig über
|
||||||
|
`chunk_config.get_sizes(type)` gesteuert. Falls der Chunker keine Fenster
|
||||||
|
mit Overlap liefert, werden in `chunk_payload` synthetische Fenster erzeugt.
|
||||||
|
Hier greift das `chunk_profile` mit folgenden Overlap-Empfehlungen:
|
||||||
|
- short → (20,30), medium → (40,60), long → (60,80).
|
||||||
|
- **Edges**: Enthält `edge_defaults` den Eintrag `references`, werden Note-
|
||||||
|
Scope-References/Backlinks additiv aktiviert (kein Abbruch, keine Duplikate).
|
||||||
|
Weitere Kantenarten bleiben unverändert.
|
||||||
|
- **Retriever**: `retriever_weight` wird in der Note gespeichert und kann von
|
||||||
|
WP‑04 im Ranking/Scoring berücksichtigt werden.
|
||||||
|
|
||||||
|
## Tests / Validierung
|
||||||
|
- Dry-Run:
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown --vault ./test_vault
|
||||||
|
```
|
||||||
|
- Vollimport + Purge:
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown --vault ./test_vault --apply --purge-before-upsert --prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
- Chunks-Qualität (Window vs Text):
|
||||||
|
```bash
|
||||||
|
python3 tests/check_chunks_window_vs_text.py --prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
- Roundtrip:
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault --prefix "$COLLECTION_PREFIX"
|
||||||
|
python3 tests/compare_vaults.py --src ./test_vault --dst ./_exportVault --focus body
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fallbacks & Robustheit
|
||||||
|
- Registry nicht vorhanden/ungültig → `concept`, Warnung, keine Abbrüche.
|
||||||
|
- Alle bestehenden ENV/Flags bleiben identisch.
|
||||||
|
- Keine Änderungen an Punkt-IDs, Feldern oder Vektoren.
|
||||||
+106
-116
@@ -13,74 +13,67 @@ Entwickle eine robuste, parser-freundliche **Chunking-Strategie** für Markdown-
|
|||||||
---
|
---
|
||||||
|
|
||||||
## 2) Annahmen aus `knowledge_design.md`
|
## 2) Annahmen aus `knowledge_design.md`
|
||||||
- **Stabile IDs** & Dateinamen-Schema (z. B. `YYYYMMDD-HHMM-type-slug` bzw. `type-slug`).
|
- **IDs**: deterministische, stabile `id` pro Note (Slug / semantischer Identifier), ggf. `aliases`.
|
||||||
- **Pflichtfelder** in YAML: `title`, `id`, `type`, `status`, `created`, `tags` (empfohlene Felder ggf. mehr).
|
- **Dateinamen**: entweder `YYYY-MM-DD_title.md` oder `slug.md`; Pfade sind **relativ** zum Vault.
|
||||||
- **Abschnitts-Konventionen** je `type` (z. B. `## Zusammenfassung`, `## Kontext` …) → ideale **Chunk-Grenzen**.
|
- **Frontmatter-Pflichtfelder**: `id`, `title`, `type`, `created`, `updated` (Details in `knowledge_design.md`).
|
||||||
- **Edge-Typen**: `belongs_to`, `references`, `backlink`, `depends_on`, `assigned_to`, `discussed_in`, `authored_by`, `related`. Für Chunking primär `belongs_to` & `references` (+ optionale `prev`/`next`).
|
- **Backlink-/Link-Regeln**: Wikilinks `[[id]]` im Body; am Ende optional redaktionelle Sektion „Mögliche Verbindungen“ (erzeugt **keine** Kanten).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3) Chunking-Prinzipien
|
## 3) Chunking-Prinzipien
|
||||||
1. **Frontmatter extrahieren** → als strukturiertes `metadata` (nicht in `text` des Chunks).
|
- **Semantische Schnitte**: Chunks enden an Absätzen, Überschriften, Listenpunkten, Codeblöcken, Zitaten.
|
||||||
2. **Markdown AST** (oder strukturierter Parser) nutzen:
|
- **Frontmatter**: YAML **nicht** embedden; stattdessen komplett im Note-Payload speichern.
|
||||||
- Primäre Grenzen: **Überschriftenebene H2/H3** (H1 = Dokumenttitel).
|
- **Verlustfreiheit**:
|
||||||
- Sekundär: **Absätze**, **nummerierte/unnummerierte Listen** als atomare Einheiten, **Codeblöcke** als zusammenhängender Block (nicht splitten).
|
- Note-Payload: `fulltext` (gesamter Body).
|
||||||
3. **Längensteuerung**: Semantik vor harter Länge. Wenn Abschnitt > Zielgröße, **weich segmentieren** entlang von Absätzen/Listenpunkten.
|
- Chunk-Payload: `text` (Originalchunk).
|
||||||
4. **Referenz-Extraktion**: `[[Wikilinks]]` + Markdown-Links → `references`. Backlinks werden downstream generiert.
|
Damit kann der Export den Body **verlustfrei** rekonstruieren (erst `fulltext`, sonst Chunks).
|
||||||
5. **Kontext-Overlap**: Chunks erhalten **Überlappung** über Satz-/Absatzgrenzen, um Kohärenz beim Retrieval zu erhöhen.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4) Optimale Chunkgrößen & Overlaps
|
## 4) Optimale Chunkgrößen & Overlaps
|
||||||
> Ziel: gutes Recall@k, wenig Fragmentierung, stabile Antwortqualität.
|
- **Zielgröße**: 250–400 Tokens (Richtwert, abhängig vom `type`).
|
||||||
|
- **Maxgröße**: 600–800 Tokens (nur in Ausnahmen).
|
||||||
|
- **Mingröße**: 80–120 Tokens (sonst mit Nachbar verschmelzen).
|
||||||
|
- **Overlap**: 40–60 Tokens, um Kontextverlust zu vermeiden.
|
||||||
|
- **Kontextabhängig**:
|
||||||
|
- `concept`, `thought`, `experience`: 250–350 Tokens
|
||||||
|
- `task`, `project`: eher 200–300 Tokens (mehr Bullet-Struktur)
|
||||||
|
- `journal`: 300–400 Tokens (tagesabschnittsweise)
|
||||||
|
|
||||||
| Notiz-Typ (`type`) | Inhalt | Zielgröße (Tokens) | Max. Größe (Tokens) | Overlap (Tokens) | Hinweise |
|
---
|
||||||
|---|---|---:|---:|---:|---|
|
|
||||||
| `thought` | kurze Thesen/Ideen | 150–250 | 300 | 30–40 | meist 1–2 Chunks |
|
|
||||||
| `experience` | Kontext/Beobachtung/Interpretation | 250–350 | 450 | 40–60 | Trenne entlang der Template-Sektionen |
|
|
||||||
| `journal` | Tagebuch, episodisch | 200–300 | 400 | 30–50 | Tagesabschnitte bündeln |
|
|
||||||
| `task` | knapp, Checklisten | 120–200 | 250 | 20–30 | Listen nicht splitten |
|
|
||||||
| `project` | Scope/WPs/Risiken/Status | 300–450 | 600 | 50–70 | Pro Hauptsektion ein Chunk |
|
|
||||||
| `concept` | definierte Begriffe, Erklärungen | 250–400 | 550 | 40–60 | Definition separat halten |
|
|
||||||
| `source` | Metadaten + Auszüge/Notizen | 200–350 | 500 | 30–50 | Zitate als eigene Chunks (Urheberrecht) |
|
|
||||||
|
|
||||||
## 4.1 Overlap-Regeln
|
## 4.1 Overlap-Regeln
|
||||||
- Overlap **nie mitten im Satz** beenden → mindestens **ein kompletter Satz** wandert in den nächsten Chunk.
|
- Overlap betrifft **nur den Body**; die YAML-Frontmatter bleibt **außerhalb**.
|
||||||
- An **Absatzgrenzen** ausrichten, wenn möglich.
|
- Keine Overlaps, die Codeblöcke oder Tabellen „aufreißen“. Falls nötig: Chunk-Grenze an Blockgrenze ziehen.
|
||||||
- Bei **Listen** ggf. den vorherigen Punkt als Overlap mitführen (falls < 50 Tokens).
|
- Bei reinen Listen: zusammenhängende Bullets gruppieren, ggf. kleinere Overlaps.
|
||||||
- Codeblöcke und Tabellen **nicht splitten**; wenn sie größer als `Max-Tokens` sind, als **eigener Chunk** belassen (Ausnahmefall zulassen).
|
|
||||||
**Daumenregel (zeichenzentriert):** ~4 Zeichen ≈ 1 Token → 800–1600 Zeichen Ziel; Overlap ~120–240 Zeichen (≈ 30–60 Tokens).
|
|
||||||
**Mindestanforderung:** Keine Trennung innerhalb von Sätzen. Sicherstellung, dass immer an Satzenden gechunct wird
|
|
||||||
**Default für mindnet:** **~300 Tokens** + **~50 Tokens Overlap**; per `type` feinjustieren (siehe Tabelle).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5) Semantische Regeln je Strukturelement
|
## 5) Semantische Regeln je Strukturelement
|
||||||
- **Überschriften (H2/H3)**: new chunk start; speichere `section_path` (z. B. `/Kontext/Beobachtung`).
|
- **Überschriften**: vorzugsweise an H2/H3 trennen; H1 meist Dokumenttitel → Teil der Frontmatter.
|
||||||
- **Absätze**: Primäre Untereinheiten; wenn Zusammenfassung + Beispiel direkt folgen, zusammen lassen.
|
- **Absätze**: primäre Trennkandidaten.
|
||||||
- **Listen**: Jeder **Listeneintrag** bleibt zusammen mit seinem einleitenden Satz (falls vorhanden). Große Listen → in **logische Blöcke** (5–8 Items) splitten.
|
- **Listen**: zusammenhängende Gruppe als ein Chunk, wenn sie einen semantischen Block bilden.
|
||||||
- **Codeblöcke**: Nicht splitten; ggf. separater Chunk, **aber** im Retrieval **downweighten** (optional).
|
- **Codeblöcke**: niemals splitten; Code bleibt in einem Chunk.
|
||||||
- **Zitate/Blockquotes**: zusammenhalten; Quelle im `references_meta` anreichern (falls Link vorhanden).
|
- **Zitate**: ab 2+ Zeilen als eigener Chunk.
|
||||||
- **Tabellen**: als ein Chunk; zusätzlich **Plain-Text-Extrakt** (Header + 1–2 Zeilen) in `summary` für Retriever, falls Modelle Tabellen schlechter verarbeiten.
|
- **Bilder/Anhänge**: Referenzen im Text belassen; Pfade relativ halten.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6) Normalisierung & Preprocessing
|
## 6) Normalisierung & Preprocessing
|
||||||
- **Nicht embedden:** YAML-Frontmatter (aber spezifische Felder als `metadata.*` mitschreiben: `type`, `tags`, `area`, `project`, `priority`, `people`, `aliases`).
|
- **Zeilenenden**: `\r\n` → `\n`, trailing spaces trimmen (nur für Hashing, nicht für Anzeige).
|
||||||
- **Bereinigen:** führende/trailing Spaces, Mehrfach-Leerzeilen → 1; Unicode-NFKC; Zeilenenden normalisieren.
|
- **Whitespace**: mehrfach-Blankzeilen reduzieren (optional, Hash-Modus beachten).
|
||||||
- **Bewahren:** Markdown-Semantik (Überschriften-Hashes, Listenpräfixe, Codefences) **im Text beibehalten**, damit RAG-Antworten sauber zitieren können.
|
- **Links**: `[[Titel|id]]` zu `(display="Titel", target_id="id")` normalisieren.
|
||||||
- **Sprachhinweis:** `lang` heuristisch bestimmen (DE/EN/…); in `metadata.lang` ablegen → später für sprachspezifische Embeddings nützlich.
|
- **Pfade**: `path` im Payload **zwingend relativ** (Backslashes → Slashes).
|
||||||
- **Stop-Abschnitte:** „Mögliche Verbindungen“ am Ende **separat** chunken (nur Links), um Link-Graph sauber zu extrahieren.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7) Edge-Modell (Graph)
|
## 7) Edge-Modell (Graph)
|
||||||
- **Chunk → Note:** `belongs_to(note_id)` (obligatorisch).
|
- **Chunk → Note:** `belongs_to(note_id)` (obligatorisch; Edge-Payload enthält zusätzlich `note_id` als Owner-Note).
|
||||||
- **Chunk → Chunk (gleiche Note):** `prev`, `next` (lineare Leserichtung).
|
- **Chunk → Chunk (gleiche Note):** `prev`, `next` (symmetrisch für lineare Navigation; beide Richtungen werden erzeugt).
|
||||||
- **Note/Chunk → Note:** `references(target_id)` aus `[[Wikilinks]]` & Markdown-Links; optional `related` (schwach).
|
- **Chunk → Note:** `references(target_id)` aus `[[Wikilinks]]` oder Markdown-Links im Chunk-Text (Standard; *Chunk-Scope*).
|
||||||
- **Backlink:** systemisch generiert: inverse Kante `backlink`.
|
- **Note → Note:** `backlink(target_id)` automatisch generiert, dedupliziert pro Note (*Note-Scope*).
|
||||||
- **Sektionen:** `has_section(note_id, section_path)` (optional, für Navigations-UI).
|
- **Optional:** `references:note` (Note-Scope-References) per Flag/ENV aktivierbar; **Default = aus**, um Doppelzählungen zu vermeiden.
|
||||||
- **Task-Spezialfälle:** `depends_on`, `assigned_to(person-id)` bleiben **Note-Level** (nicht Chunk-Level).
|
- **Unresolved:** Wenn `target_id` nicht existiert, wird eine Kante mit `status: "unresolved"` angelegt (später durch Resolver/Agent heilbar).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -100,26 +93,49 @@ Entwickle eine robuste, parser-freundliche **Chunking-Strategie** für Markdown-
|
|||||||
"project": "project-mindnet",
|
"project": "project-mindnet",
|
||||||
"tags": ["area/mindnet","type/thought","topic/yaml"],
|
"tags": ["area/mindnet","type/thought","topic/yaml"],
|
||||||
"chunk_index": 2,
|
"chunk_index": 2,
|
||||||
"char_start": 1240,
|
"char_start": 1024,
|
||||||
"char_end": 2012,
|
"char_end": 1789,
|
||||||
"token_count": 305,
|
|
||||||
"section_title": "Begründung / Details",
|
"section_title": "Begründung / Details",
|
||||||
"section_path": "/Begründung / Details",
|
"section_path": "Begründung / Details",
|
||||||
"lang": "de",
|
"neighbors": {"prev": "…#c01", "next": "…#c03"},
|
||||||
"wikilinks": ["concept-vektorsuche-qdrant","source-obsidian-properties"],
|
"wikilinks": ["concept-yaml-styleguide"],
|
||||||
"external_links": [],
|
"references": [{"kind":"wikilink", "target_id":"concept-yaml-styleguide"}],
|
||||||
"references": [
|
"text": "… ursprünglicher Chunk-Text …"
|
||||||
{"target_id": "concept-vektorsuche-qdrant", "kind": "wikilink"}
|
|
||||||
],
|
|
||||||
"neighbors": {"prev": "#c01", "next": "#c03"},
|
|
||||||
"created_at": "2025-09-02T18:35:00+02:00"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
**Collection-Einstellungen (Empfehlung)**
|
**Point (Note)**
|
||||||
- `distance`: `Cosine` (gemischte Textlängen)
|
|
||||||
- `vectors`: 384–1024 Dim (abh. Embedding-Modell)
|
{
|
||||||
- `hnsw_ef_construct`: 128–256; `m`: 16–32 (Workload-abhängig)
|
"id": "20250902-1830-thought-yaml-recall",
|
||||||
|
"vector": [/* optional Nullvektor oder Zentroid */],
|
||||||
|
"payload": {
|
||||||
|
"note_id": "20250902-1830-thought-yaml-recall",
|
||||||
|
"title": "Gedanke: Einheitliche YAML-Standards steigern Recall",
|
||||||
|
"type": "thought",
|
||||||
|
"status": "draft",
|
||||||
|
"created": "2025-09-02",
|
||||||
|
"updated": "2025-09-02",
|
||||||
|
"path": "10_thoughts/20250902-1830-thought-yaml-recall.md",
|
||||||
|
"tags": ["area/mindnet","type/thought","topic/yaml"],
|
||||||
|
"hash_fulltext": "sha256:…",
|
||||||
|
"fulltext": "… gesamter Body für verlustfreie Rekonstruktion …",
|
||||||
|
"references": ["concept-yaml-styleguide"] // Fallback für Note-Level-Backlinks
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
**Edges (Payload; neues Schema)**
|
||||||
|
|
||||||
|
{
|
||||||
|
"kind": "references" | "backlink" | "belongs_to" | "prev" | "next",
|
||||||
|
"source_id": "<chunk_id|note_id>",
|
||||||
|
"target_id": "<note_id|chunk_id>",
|
||||||
|
"scope": "chunk" | "note",
|
||||||
|
"note_id": "<owner-note-id>", // neu: für schnelles Filtern/Purge
|
||||||
|
"status": "ok" | "unresolved" // optional
|
||||||
|
}
|
||||||
|
|
||||||
|
**Performance-Hinweis (Qdrant):** Für schnelle Filter/Deletes Payload-Indizes anlegen: **edges** → `kind`, `scope`, `source_id`, `target_id`, `note_id`; **chunks** → `note_id`, `chunk_index`; **notes** → `note_id`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -136,79 +152,53 @@ Entwickle eine robuste, parser-freundliche **Chunking-Strategie** für Markdown-
|
|||||||
blocks = group_blocks(sec, keep_code=True, keep_lists=True)
|
blocks = group_blocks(sec, keep_code=True, keep_lists=True)
|
||||||
for group in soft_wrap(blocks, target_tokens=target_size(meta["type"]),
|
for group in soft_wrap(blocks, target_tokens=target_size(meta["type"]),
|
||||||
max_tokens=max_size(meta["type"])):
|
max_tokens=max_size(meta["type"])):
|
||||||
raw_chunks.append(render_markdown(group))
|
raw_chunks.append(render_md(group))
|
||||||
|
|
||||||
chunks = apply_overlap(raw_chunks, overlap_tokens=overlap_size(meta["type"]))
|
chunks = apply_overlap(raw_chunks, overlap_tokens=overlap(meta["type"]))
|
||||||
chunks = trim_whitespace(chunks)
|
chunks = normalize_chunks(chunks) # trim, fix code fences, etc.
|
||||||
|
|
||||||
# link extraction
|
# IDs & Nachbarn
|
||||||
for i, ch in enumerate(chunks):
|
for i, ch in enumerate(chunks, start=1):
|
||||||
ch.payload.update({
|
ch.id = f"{meta['id']}#c{i:02d}"
|
||||||
"wikilinks": extract_wikilinks(ch.text),
|
ch.index = i
|
||||||
"external_links": extract_md_links(ch.text),
|
ch.neighbors_prev = chunks[i-2].id if i > 1 else None
|
||||||
"references": [{"target_id": t, "kind": "wikilink"} for t in extract_wikilinks(ch.text)],
|
ch.neighbors_next = chunks[i].id if i < len(chunks) else None
|
||||||
"neighbors": {"prev": f"#c{i-1}" if i>0 else None,
|
|
||||||
"next": f"#c{i+1}" if i<len(chunks)-1 else None}
|
|
||||||
})
|
|
||||||
|
|
||||||
# attach metadata
|
# Wikilinks je Chunk
|
||||||
for idx, ch in enumerate(chunks):
|
for ch in chunks:
|
||||||
ch.id = f'{meta["id"]}#c{idx:02d}'
|
ch.wikilinks = extract_wikilinks(ch.text)
|
||||||
ch.payload |= minimal_payload_from_meta(meta, file_path, idx, ch)
|
|
||||||
|
|
||||||
return chunks
|
return chunks
|
||||||
|
|
||||||
**Hinweise zur Implementierung**
|
|
||||||
- **Tokenizer**: gleiche Tokenizer-Funktion für **Längensteuerung** und **Embedding** nutzen (Leak vermeiden).
|
|
||||||
- **Overlap** implementieren auf **Satz-/Absatzebene**; notfalls anhand Token-Fenster (kein Wort-Cut).
|
|
||||||
- **Heuristik** für `target_size`/`overlap_size` per `type` (siehe Tabelle).
|
|
||||||
- **Später**: Re-Chunking ermöglichen (idempotente Re-Runs), Chunk-IDs stabil halten (Index + Hash über `section_path + first_64_tokens`).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10) Einbettung (Embeddings)
|
## 10) Einbettung (Embeddings)
|
||||||
- **Standard:** nur `chunk.text` embedden (ohne YAML), **aber**:
|
- Modell: MiniLM/all-MiniLM-L6-v2 (384d) oder E5 (besseres Retrieval, ggf. multilingual).
|
||||||
- `aliases`, `title` können als **synthetischer Kontext** *vorangestellt* werden (konkateniert, durch Trennzeichen), falls Recall für Synonyme wichtig ist.
|
- **Note-Embeddings** optional (Zentroid der Chunks).
|
||||||
- **Mehrsprachigkeit:** pro Chunk die Sprache detektieren; ggf. **mehrsprachige Modelle**/Pipelines verwenden.
|
- **Hashing für Change-Detection** (Importer):
|
||||||
- **Code-Gewichtung:** optional zweite Vektor-Spur für Code (separate Collection) falls viel Technik.
|
- `MINDNET_HASH_MODE = body|frontmatter|body+frontmatter` (Default: `body`)
|
||||||
|
- `MINDNET_HASH_NORMALIZE = canonical|none` (Default: `canonical`)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 11) Qualitätssicherung
|
## 11) Qualitätssicherung
|
||||||
- **Unit-Tests**: Gold-Notes mit erwarteten Section-Splits und Link-Graph.
|
- **Zähler**: `notes/chunks/edges` je Lauf prüfen.
|
||||||
- **Metrics**: Retrieval-Eval (nDCG@k, Recall@k) auf internen QA-Fragen.
|
- **Edge-Invarianten**:
|
||||||
- **Linting**: Pre-commit prüft Pflichtfelder & Link-Integrität (IDs, Dateinamen, Wikilinks).
|
- `belongs_to == #Chunks`
|
||||||
|
- `next == prev == Σ(max(chunks_in_note-1, 0))`
|
||||||
|
- `references (chunk) == Σ(Chunk-Wikilinks)`
|
||||||
|
- `backlink == Σ(eindeutige Wikilinks je Note)`
|
||||||
|
- **Unresolved**: sollten durch Resolver-Jobs über Zeit abnehmen.
|
||||||
|
- **Roundtrip-Test**: Import → Export → Diff; exakt rekonstruierbare Bodies.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 12) Deliverables
|
## 12) Deliverables
|
||||||
1. **Python-Modul** `mindnet_chunker/` (Parser, Chunker, Link-Extractor, Qdrant-Client).
|
- Code: Parser, Chunker, Importer, Exporter, Edges.
|
||||||
2. **CLI** `mindnet-chunk` (stdin→stdout, oder Datei→NDJSON).
|
- Schemas: `note.schema.json`, Payload-Layouts.
|
||||||
3. **Docs**: README mit Schema & Beispielen; Tests + Fixtures.
|
- Tests: Mini-Vault (`scripts/make_test_vault.py`), Audit/Validate.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 13) Prompt für neues Chatfenster (kopieren & einfügen)
|
## 13) Prompt für neues Chatfenster (kopieren & einfügen)
|
||||||
|
*(Unverändert)*
|
||||||
**Projekt:** mindnet Wissensnetzwerk
|
|
||||||
**Rolle:** Du bist mein Assistent für NLP-Strategien.
|
|
||||||
**Ziel:** Hilf mir, eine robuste Chunking-Strategie für Markdown-Notizen zu entwickeln, die später in Qdrant gespeichert werden.
|
|
||||||
|
|
||||||
**Anforderungen:**
|
|
||||||
- Nutze das YAML-Frontmatter-Schema und die Strukturregeln aus `knowledge_design.md`. (Pflichtfelder, Abschnitts-Konventionen, Edge-Typen)
|
|
||||||
- Definiere optimale Chunkgrößen (Tokens) nach Notiz-Typ, inkl. **Overlap-Empfehlungen**.
|
|
||||||
- Beschreibe, wie **logische Abschnitte** (Überschriften/Absätze/Listen/Code) als Grenzen dienen.
|
|
||||||
- Erkläre das **Edge-Modell** zwischen **Chunks** und **Notizen** (`belongs_to`, `references`, `backlink`, `prev/next`).
|
|
||||||
- Ergebnis: ein **konkretes Konzept + Python-Outline**, das ich direkt implementieren kann (inkl. Datenmodell für Qdrant).
|
|
||||||
|
|
||||||
**Kontext zum System:**
|
|
||||||
- IDs/Dateinamen/Tags gemäß `knowledge_design.md`.
|
|
||||||
- Zielgröße pro Chunk ~**300 Tokens**, **Overlap ~50 Tokens** als Default; per `type` feinjustiert (Tabelle).
|
|
||||||
- YAML wird **nicht** in die Embeddings gemischt; ausgewählte Felder als `payload`.
|
|
||||||
|
|
||||||
Bitte liefere:
|
|
||||||
1) Zusammenfassung der Strategie,
|
|
||||||
2) Tabelle mit Größen/Overlaps,
|
|
||||||
3) Python-Pseudocode für Chunker,
|
|
||||||
4) Qdrant-Payload-Beispiel,
|
|
||||||
5) Test-Plan kurz.
|
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# mindnet Retriever (WP-04 / Step 4a) – Kurz-Dokumentation
|
||||||
|
|
||||||
|
## 1. Überblick
|
||||||
|
|
||||||
|
Der Retriever besteht aus:
|
||||||
|
|
||||||
|
- **Semantic Retriever** (`semantic_retrieve`)
|
||||||
|
- **Hybrid Retriever** (`hybrid_retrieve`)
|
||||||
|
- Qdrant-Backend (`*_chunks`, `*_edges`)
|
||||||
|
- optionaler Edge-Expansion über `graph_adapter.expand`
|
||||||
|
|
||||||
|
Die API nutzt das bestehende `/query`-Endpoint (FastAPI), das `QueryRequest` und `QueryResponse` aus `app.models.dto` verwendet.
|
||||||
|
|
||||||
|
## 2. Scoring-Formel
|
||||||
|
|
||||||
|
Pro Treffer werden folgende Komponenten geführt:
|
||||||
|
|
||||||
|
- `semantic_score` – direkt aus Qdrant (Cosine/Euclidean, je nach Collection)
|
||||||
|
- `retriever_weight` – aus Chunk-Payload (`types.yaml` / Frontmatter)
|
||||||
|
- `edge_bonus` – Bonus aus dem Subgraph (z. B. Zahl der eingehenden/ausgehenden Edges, Beziehungstypen)
|
||||||
|
- `centrality_bonus` – Bonus aus einer einfachen Zentralitäts-Heuristik im Subgraph
|
||||||
|
|
||||||
|
Aktuelle Formel (Step 4a):
|
||||||
|
|
||||||
|
```text
|
||||||
|
total_score = semantic_score * max(retriever_weight, 0.0)
|
||||||
|
+ edge_bonus
|
||||||
|
+ centrality_bonus
|
||||||
|
```
|
||||||
|
|
||||||
|
Die Sortierung der Ergebnisse erfolgt absteigend nach `total_score`.
|
||||||
|
|
||||||
|
## 3. Modi
|
||||||
|
|
||||||
|
### 3.1 Semantic Mode
|
||||||
|
|
||||||
|
- Funktion: `semantic_retrieve`
|
||||||
|
- Request-Feld `mode = "semantic"`
|
||||||
|
- Keine Edge-Expansion, kein Graph-Call.
|
||||||
|
- Kandidaten nur aus Vektorsuche in `<prefix>_chunks`.
|
||||||
|
|
||||||
|
### 3.2 Hybrid Mode
|
||||||
|
|
||||||
|
- Funktion: `hybrid_retrieve`
|
||||||
|
- Request-Feld `mode = "hybrid"`
|
||||||
|
- Schritte:
|
||||||
|
1. Semantische Chunk-Suche wie im Semantic Mode
|
||||||
|
2. Falls `expand.depth > 0` gesetzt ist:
|
||||||
|
- Seeds = `chunk_id` (Fallback: `note_id`) der Treffer
|
||||||
|
- Aufruf `graph_adapter.expand(client, prefix, seed_ids, depth, edge_types)`
|
||||||
|
- Ergebnis-Subgraph liefert `edge_bonus` und `centrality_bonus` pro Node
|
||||||
|
3. Scores werden nach obiger Formel kombiniert.
|
||||||
|
|
||||||
|
## 4. Request-Struktur (JSON)
|
||||||
|
|
||||||
|
Beispiel-Hybrid-Request:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mode": "hybrid",
|
||||||
|
"query": "embeddings und qdrant",
|
||||||
|
"top_k": 10,
|
||||||
|
"expand": {
|
||||||
|
"depth": 1,
|
||||||
|
"edge_types": ["references", "depends_on"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Minimaler Semantic-Request:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mode": "semantic",
|
||||||
|
"query": "karate trainingsplan",
|
||||||
|
"top_k": 5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Smoke-Test
|
||||||
|
|
||||||
|
Für einen schnellen End-to-End-Test kann folgendes Skript verwendet werden:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/test_retriever_smoke.py --query "karate" --mode hybrid --expand-depth 1 --top-k 5
|
||||||
|
```
|
||||||
|
|
||||||
|
Das Skript ruft `/query` auf, wertet Statuscode und JSON-Antwort aus und zeigt die Scores der Top-K-Treffer an.
|
||||||
|
|
||||||
|
## 6. Tests
|
||||||
|
|
||||||
|
Die wichtigsten Tests für den Retriever sind:
|
||||||
|
|
||||||
|
- `tests/test_retriever_basic.py` – Basisfunktion (Semantic + Hybrid)
|
||||||
|
- `tests/test_retriever_weight.py` – Einfluss von `retriever_weight` auf das Ranking
|
||||||
|
- `tests/test_retriever_edges.py` – Einfluss von Edge-/Centrality-Boni im Hybrid-Modus
|
||||||
+159
-379
@@ -1,436 +1,216 @@
|
|||||||
---
|
|
||||||
title: "mindnet – Knowledge Design"
|
|
||||||
id: "design-knowledge-architecture"
|
|
||||||
type: "guide"
|
|
||||||
status: "active"
|
|
||||||
version: "1.2.0"
|
|
||||||
created: 2025-09-02
|
|
||||||
updated: 2025-09-02
|
|
||||||
tags: ["type/guide","area/mindnet","topic/obsidian","topic/knowledge-architecture"]
|
|
||||||
owner: "Ich"
|
|
||||||
---
|
|
||||||
|
|
||||||
# mindnet – Knowledge Design
|
# mindnet – Knowledge Design
|
||||||
|
**Version:** 1.5.0 (aktualisiert 2025-11-09)
|
||||||
|
|
||||||
## 1. Gesamtziel, Anspruch & Nutzen
|
> Dieses Dokument beschreibt das Ziel, die Architektur, die Datenmodelle sowie die Regeln und Prozesse für den Aufbau von **mindnet** – einem persönlichen, lokal betriebenen Wissensnetz, das auf Markdown-Quellen, Graph-Beziehungen und Vektor-Suche basiert. Es soll die Persönlichkeit, Werte und Erfahrungen des Besitzers widerspiegeln und künftig Antworten generieren, die diese Perspektive authentisch berücksichtigen.
|
||||||
|
|
||||||
**Ziel:**
|
|
||||||
*mindnet* ist ein persönliches Wissensnetzwerk, das **alle** relevanten Inhalte – Gedanken, Erfahrungen, Aufgaben, Projekte, Quellen, Personen, Meetings, Tagebuch/Journal, Wendepunkte und Leitbilder – **einheitlich** in Markdown + YAML ablegt. Es bildet einen **Spiegel der Persönlichkeit** (Werte, Überzeugungen, Erlebnisse) und ist zugleich so strukturiert, dass Tools (Dataview, Embeddings, Vektorsuche/Qdrant, Agenten) es **maschinenlesbar** nutzen können.
|
|
||||||
|
|
||||||
**Anspruch an die Struktur:**
|
|
||||||
- **Stabil**: robuste IDs, deterministische Dateinamen, konsistente Feldnamen
|
|
||||||
- **Atomar**: ein Kerninhalt pro Datei (keine Sammelcontainer)
|
|
||||||
- **Erweiterbar**: neue Typen (z. B. `milestone`, `manifesto`) jederzeit möglich
|
|
||||||
- **Portabel**: reines Markdown + YAML → funktioniert in Obsidian, Git und Parsern
|
|
||||||
- **Vernetzbar**: systematische Links/Backlinks erzeugen ein sinnvoll navigierbares Netz
|
|
||||||
- **Parser-freundlich**: wiederkehrende Abschnittsüberschriften; Felder/Enums klar definiert
|
|
||||||
|
|
||||||
**Späterer Nutzen / Einsatzzwecke:**
|
|
||||||
- **Schnelles Wiederfinden**: Dataview-Abfragen, Filter, Dashboards
|
|
||||||
- **Selbstreflexion & Entwicklung**: persönliche Werte, Wendepunkte, Kinderentwicklung, Lernpfade
|
|
||||||
- **Automatisierung**: Vektor-Index (Qdrant), Embeddings, LLM-Q&A, Agenten-Workflows
|
|
||||||
- **Langfristiges Gedächtnis**: eigene Lebens-/Projektchronik reproduzierbar pflegen
|
|
||||||
- **Aufgaben-/Projektsteuerung**: im selben System, vernetzt mit Kontextwissen
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. YAML-Frontmatter-Schema
|
## 1. Ziel von mindnet
|
||||||
|
|
||||||
> **Pflichtfelder:** `title`, `id`, `type`, `status`, `created`, `tags`
|
**1.1 Zweck**
|
||||||
> **Empfohlen:** `updated`, `area`, `project`, `priority`, `aliases` (u. a.)
|
- mindnet dient als persönliches Wissensnetz, das Erfahrungen, Werte, Prinzipien, Ziele und prägenden Ereignisse langfristig strukturiert.
|
||||||
|
- Es ermöglicht spätere Einsichten in Entscheidungen und Entwicklungen und kann in Zukunft für die Familie (z. B. Kinder) eine nachvollziehbare Quelle sein.
|
||||||
|
|
||||||
### 2.1 Feldübersicht
|
**1.2 Leitidee**
|
||||||
|
- Statt einer reinen Dokumentablage wird Wissen durch **Knoten (Notes/Chunks)** und **Kanten (Edges)** vernetzt.
|
||||||
|
- Antworten sollen **persönlich** kontextualisiert werden (Perspektive, Konditionierung, Persönlichkeit).
|
||||||
|
|
||||||
| Feld | Typ | Bedeutung | Beispiel(e) |
|
**1.3 Langfristigkeit**
|
||||||
|---------------------|-------------------------------|---------------------------------------------------------------------------|------------------------------------------------------------------------------|
|
- Stabiler, zukunftsfähiger Entwurf; spätere Massenmigrationen sollen vermieden werden.
|
||||||
| `title` | string | Menschlich lesbarer Titel | `"Vektorsuche in Qdrant"` |
|
- Architektur unterstützt schrittweise Erweiterung (neue Quellen, neue Agenten, neue Regeln).
|
||||||
| `id` | string (slug/identifier) | **Stabile** technische ID (nie umbenennen) | `"20250902-1710-thought-obsidian-struktur"`, `"concept-persoenliches-leitbild"` |
|
|
||||||
| `type` | enum | Notiz-Typ | `thought`, `experience`, `task`, `project`, `source`, `concept`, `person`, `meeting`, `journal`, `milestone` |
|
|
||||||
| `status` | enum | Lebenszyklus | `idea`, `draft`, `active`, `paused`, `blocked`, `done`, `archived` |
|
|
||||||
| `created` | date/datetime | Erstellzeitpunkt | `2025-09-02` oder `2025-09-02T17:10:00+02:00` |
|
|
||||||
| `updated` | date/datetime | Letzte inhaltliche Änderung | `2025-09-02T18:05:00+02:00` |
|
|
||||||
| `tags` | string[] | Schlagwörter gemäß Taxonomie | `["area/mindnet","topic/obsidian","type/thought"]` |
|
|
||||||
| `area` | enum/string | Lebens-/Arbeitsbereich | `mindnet`, `family`, `karate`, `travel`, `health`, `home-automation`, … |
|
|
||||||
| `role` | string | Rolle/Hut (Filter) | `"father"`, `"researcher"`, `"project-owner"` |
|
|
||||||
| `project` | string | Zugehöriges Projekt (ID oder sprechend) | `"project-mindnet"` |
|
|
||||||
| `priority` | int (1–5) | 1 = hoch, 5 = niedrig | `2` |
|
|
||||||
| `effort_min` | int | Aufwand in Minuten (für `task`) | `90` |
|
|
||||||
| `due` | date | Fälligkeit (Task/Projekt) | `2025-09-10` |
|
|
||||||
| `start` | date | geplanter Start (Task/Projekt) | `2025-09-05` |
|
|
||||||
| `people` | string[] (Wikilinks erlaubt) | Beteiligte Personen | `["Ich","[[person-kathie]]"]` |
|
|
||||||
| `location` | string | Ort/Platz | `"Home Office"`, `"München"` |
|
|
||||||
| `source_url` | url | Kanonische URL (für `source`) | `"https://qdrant.tech/docs/"` |
|
|
||||||
| `authors` | string[] | Autoren (für `source`) | `["Jane Doe"]` |
|
|
||||||
| `published` | date | Veröffentlichungsdatum (für `source`) | `2024-11-20` |
|
|
||||||
| `accessed` | date | Zugriffsdatum (für `source`) | `2025-09-02` |
|
|
||||||
| `aliases` | string[] | Alternative Namen/Titel | `["Manifest","Leitbild"]` |
|
|
||||||
| `related_ids` | string[] | ID-basierte Verknüpfungen | `["concept-persoenliches-leitbild"]` |
|
|
||||||
| `embedding_exclude` | boolean | Von Embeddings/Index ausschließen | `false` |
|
|
||||||
|
|
||||||
### 2.2 Konventionen
|
|
||||||
- **IDs:** entweder Zeitstempel-basiert (`YYYYMMDD-HHMM-type-slug`) **oder** stabile, sprechende (`concept-*`, `project-*`, `person-*`).
|
|
||||||
- **Datums-/Zeitformat:** ISO 8601, Zeiten möglichst mit TZ (z. B. `+02:00`).
|
|
||||||
- **Felder:** klein geschrieben; zusammengesetzte Felder in `snake_case`.
|
|
||||||
|
|
||||||
### 2.3 Minimalbeispiel Frontmatter
|
|
||||||
---
|
|
||||||
title: "Kurzer, klarer Gedanke"
|
|
||||||
id: "20250902-1715-thought-klarer-gedanke"
|
|
||||||
type: "thought"
|
|
||||||
status: "idea"
|
|
||||||
created: 2025-09-02T17:15:00+02:00
|
|
||||||
updated: 2025-09-02T17:15:00+02:00
|
|
||||||
tags: ["area/mindnet","type/thought","topic/obsidian"]
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2.4 Enum-Werte
|
|
||||||
|
|
||||||
Zur Sicherstellung der Konsistenz werden für bestimmte Felder feste Wertemengen (Enums) verwendet:
|
|
||||||
|
|
||||||
- **type**
|
|
||||||
- `idea` – spontane Gedanken, Inspirationen
|
|
||||||
- `experience` – persönliche Erfahrungen, Reflexionen
|
|
||||||
- `task` – konkrete Aufgaben
|
|
||||||
- `project` – Projekte, bestehend aus mehreren Tasks
|
|
||||||
- `concept` – abstrakte Konzepte, Modelle, Methoden
|
|
||||||
- `milestone` – wichtige Ereignisse oder Wendepunkte
|
|
||||||
- `person` – Personen (z. B. Kontakte, Vorbilder)
|
|
||||||
- `meeting` – Besprechungen, Treffen, Gespräche
|
|
||||||
- `resource` – externe Quellen (Artikel, Bücher, Links)
|
|
||||||
- `journal` – Tagebuchnotizen
|
|
||||||
|
|
||||||
- **status**
|
|
||||||
- `open`
|
|
||||||
- `in-progress`
|
|
||||||
- `done`
|
|
||||||
- `blocked`
|
|
||||||
- `canceled`
|
|
||||||
|
|
||||||
- **priority**
|
|
||||||
- `low`
|
|
||||||
- `medium`
|
|
||||||
- `high`
|
|
||||||
|
|
||||||
- **area** (Beispiele – erweiterbar)
|
|
||||||
- `personal`
|
|
||||||
- `work`
|
|
||||||
- `family`
|
|
||||||
- `karate`
|
|
||||||
- `health`
|
|
||||||
- `finance`
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Dateinamen-Konventionen
|
## 2. Zielarchitektur (High Level)
|
||||||
|
|
||||||
> Keine Sonderzeichen wie `: / \` im Dateinamen.
|
- **Datenquellen:** Markdown-Vault (Obsidian-kompatibel), perspektivisch weitere Quellen (MediaWiki, PDFs, Web-Exporte, E-Mail-Auszüge).
|
||||||
|
- **Import-Pipeline:** Parser → Frontmatter/Body → Chunking → Embedding → Qdrant (Notes/Chunks/Edges).
|
||||||
- **Zeitbasierte Notizen (Gedanken, Journal, tagesgebundene Einträge):**
|
- **Speicher:** Qdrant (Vektoren + Payload + Textindex), Dateien bleiben im Filesystem (Quelle ist Referenz).
|
||||||
`YYYYMMDD-HHMM-<type>-<slug>.md`
|
- **Graph-Schicht:** Edges explizit modelliert (Typ, Richtung, Evidenz, Confidence); optionale abgeleitete Edges per Batch-Job.
|
||||||
Beispiel: `20250902-1710-thought-obsidian-struktur.md`
|
- **Retriever/Composer:** Hybrid-Retrieval (Vektor + Filter + optional Volltext), Komposition zu Antwortkandidaten.
|
||||||
|
- **Erklärbarkeit:** Begründungspfade (Top-Chunks, Graph-Kette, Provenienz).
|
||||||
- **Zeitunabhängige Wissensknoten (Konzepte, Personen, Projekte, Leitbild):**
|
- **Schnittstellen:** REST/CLI, n8n-Glue, lokale LLM-Anbindung (Ollama).
|
||||||
`<type>-<slug>.md`
|
|
||||||
Beispiele: `concept-vektorsuche-qdrant.md`, `project-mindnet.md`, `person-kathie.md`, `concept-persoenliches-leitbild.md`
|
|
||||||
|
|
||||||
- **Ordnerstruktur (Empfehlung):**
|
|
||||||
00_inbox/
|
|
||||||
10_thoughts/
|
|
||||||
20_experiences/
|
|
||||||
30_projects/
|
|
||||||
31_tasks/
|
|
||||||
40_concepts/
|
|
||||||
50_sources/
|
|
||||||
60_people/
|
|
||||||
70_meetings/
|
|
||||||
80_journal/
|
|
||||||
90_templates/
|
|
||||||
_attachments/
|
|
||||||
_meta/
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Tagging-Regeln & Taxonomie
|
## 3. Collections in Qdrant
|
||||||
|
|
||||||
- **Hierarchische Tags:**
|
- `mindnet_notes` – Note-Metadaten (id, title, type, path, timestamps, tags, …).
|
||||||
- `area/*` – Lebens-/Arbeitsbereiche: `area/mindnet`, `area/family`, `area/karate`, `area/health`, …
|
- `mindnet_chunks` – inhaltstragende Segmente inkl. Vektoren und Textindex.
|
||||||
- `type/*` – optionaler Spiegel von `type`: `type/thought`, `type/experience`, …
|
- `mindnet_edges` – Beziehungen (src/dst, relation, direction, evidence, confidence, …).
|
||||||
- `topic/*` – thematische Schwerpunkte: `topic/obsidian`, `topic/yaml`, `topic/qdrant`, `topic/werte`, …
|
- (optional) `mindnet_people` – strukturierte Personenobjekte (Eigen-/Fremdbezug).
|
||||||
- optional: `place/*`, `person/*`, `level/*` (z. B. `person/rouven`, `person/rohan`)
|
|
||||||
- **Regeln:**
|
|
||||||
- Jede Notiz hat **mindestens ein** `area/*`.
|
|
||||||
- `topic/*` sparsam, aber konsistent; ein Mapping in `_meta/tags.md` pflegen.
|
|
||||||
- Möglichst **keine** losen Ein-Wort-Tags ohne Präfix (verhindert Wildwuchs).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Backlink- & Link-Regeln
|
## 4. Identitäten & IDs
|
||||||
|
|
||||||
- **Wikilinks** verwenden: `[[stabile-id-oder-dateiname]]`
|
- **Deterministische IDs** für Notes (z. B. Namespace-basierte UUIDv5) und stabile, ableitbare IDs für Chunks/Edges.
|
||||||
- Im **Fließtext** bei erster Nennung verlinken; am **Ende** die Sektion **„Mögliche Verbindungen“** pflegen:
|
- ID-Stabilität ist Voraussetzung für Idempotenz (Re-Imports ohne Duplikate).
|
||||||
|
|
||||||
## Mögliche Verbindungen
|
|
||||||
- [[concept-persoenliches-leitbild]]
|
|
||||||
- [[project-mindnet]]
|
|
||||||
|
|
||||||
- **Linkrichtung:** „aufwärts“ (Detail → Konzept/Leitbild) und „seitwärts“ (verwandte Knoten).
|
|
||||||
- Optional zusätzlich **ID-Relationen** im YAML (`related_ids`) für Parser.
|
|
||||||
|
|
||||||
## 5.1 Edge-Typen
|
|
||||||
|
|
||||||
Beziehungen zwischen Notizen, Chunks und Objekten werden als **Edges** gespeichert. Jeder Edge erhält einen `type` und optionale `meta`-Informationen (z. B. Gewichtung, Richtung).
|
|
||||||
|
|
||||||
- **belongs_to**
|
|
||||||
Chunk → Note (ein Chunk gehört zu genau einer Note)
|
|
||||||
|
|
||||||
- **references**
|
|
||||||
Note → Note (eine Notiz verweist auf eine andere)
|
|
||||||
|
|
||||||
- **backlink**
|
|
||||||
Automatisch generiert: Gegenkante von `references`
|
|
||||||
|
|
||||||
- **depends_on**
|
|
||||||
Task → Task oder Task → Project (Aufgabe hängt von anderer ab)
|
|
||||||
|
|
||||||
- **assigned_to**
|
|
||||||
Task → Person (eine Aufgabe ist einer Person zugeordnet)
|
|
||||||
|
|
||||||
- **discussed_in**
|
|
||||||
Note/Concept → Meeting (ein Thema wurde in einem Meeting behandelt)
|
|
||||||
|
|
||||||
- **authored_by**
|
|
||||||
Note → Person (z. B. externer Autor oder eigene Urheberschaft)
|
|
||||||
|
|
||||||
- **related**
|
|
||||||
Generische Relation für schwache Assoziationen
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Abschnitts-Konventionen im Body
|
## 5. Frontmatter & Typen
|
||||||
|
|
||||||
Einheitliche Überschriften erleichtern spätere Parser und Abfragen.
|
- Pflichtfelder: `id`, `title`, `type`, `created`, `updated`, `tags`, `source_path`.
|
||||||
|
- Note-Typen (Auszug): `experience`, `principle`, `value`, `goal`, `role`, `persona`, `plan`, `project`, `journal`, `reference`, `translation`.
|
||||||
- **Gedanke (`thought`)**
|
- **types.yaml** definiert Regeln/Defaults je Typ und für Relationstypen (siehe Abschnitt 10).
|
||||||
`## Zusammenfassung` · `## Begründung / Details` · `## Belege / Quellen` · `## Nächste Schritte` · `## Mögliche Verbindungen`
|
|
||||||
- **Erfahrung (`experience`)**
|
|
||||||
`## Kontext` · `## Beobachtung` · `## Interpretation` · `## Implikationen` · `## Mögliche Verbindungen`
|
|
||||||
- **Aufgabe (`task`)**
|
|
||||||
`## Checkliste` · `## Notizen` · `## Mögliche Verbindungen`
|
|
||||||
- **Projekt (`project`)**
|
|
||||||
`## Scope` · `## Arbeitspakete` · `## Risiken & Gegenmaßnahmen` · `## Status / Fortschritt` · `## Mögliche Verbindungen`
|
|
||||||
- **Journal (`journal`)**
|
|
||||||
`## Tageszusammenfassung` · `## Highlights` · `## Gelernt / Notizen` · `## Mögliche Verbindungen`
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Abbildung persönlicher Inhalte auf Typen
|
## 6. Chunking-Strategie
|
||||||
|
|
||||||
- **Persönliches Leitbild** → `type: "concept"` (oder optional neuer Typ `manifesto`)
|
- Satz- bis Absatz-Chunks, moderate Überlappung; Meta-Felder verweisen auf Note/Quelle.
|
||||||
Datei: `concept-persoenliches-leitbild.md` (zeitunabhängig)
|
- Named Vectors (z. B. `content`, optional `title`, `tags`) ermöglichen hybride Scores.
|
||||||
Verlinkt zu prägenden Erfahrungen und Wendepunkten.
|
|
||||||
- **Erfahrungen, die dich geprägt haben** → `type: "experience"`
|
|
||||||
Je **eine** prägende Erfahrung pro Datei; klarer Kontext + Interpretation.
|
|
||||||
- **Wendepunkte im Leben** → `type: "milestone"`
|
|
||||||
Je Wendepunkt ein Knoten; verlinkt zu den auslösenden Erfahrungen.
|
|
||||||
- **Aktuelle Erfahrungen** → `type: "experience"`, `status: "active"`
|
|
||||||
Bei Bedarf später in „prägende“ Erfahrungen überführen.
|
|
||||||
- **Entwicklungsschritte der Kinder** → `type: "journal"` (chronologisch) **oder** `experience` (wenn es prägend ist); Tags `person/<name>`.
|
|
||||||
- **Tagebucheinträge** → `type: "journal"`; ein Eintrag pro Tag; tagesbasierte Dateinamen.
|
|
||||||
|
|
||||||
Damit entsteht ein **netzartiges Persönlichkeitsprofil** mit klaren Pfaden zwischen Werten (Leitbild), Ereignissen (Erfahrungen), Wendepunkten und fortlaufenden Beobachtungen (Journal).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Beispiel-Notizen
|
## 7. Embeddings & Vektoren
|
||||||
|
|
||||||
### 8.1 Gedanke (`thought`)
|
- Einheitliche Metrik (cosine), reproduzierbares Embedding-Setup.
|
||||||
---
|
- Konsistenz über Zeit durch Versionsfeld (`embedding_model`, `embedding_version`).
|
||||||
title: "Gedanke: Einheitliche YAML-Standards steigern Recall"
|
|
||||||
id: "20250902-1830-thought-yaml-recall"
|
|
||||||
type: "thought"
|
|
||||||
status: "idea"
|
|
||||||
created: 2025-09-02T18:30:00+02:00
|
|
||||||
updated: 2025-09-02T18:30:00+02:00
|
|
||||||
tags: ["area/mindnet","type/thought","topic/yaml","topic/recall"]
|
|
||||||
area: "mindnet"
|
|
||||||
project: "project-mindnet"
|
|
||||||
aliases: []
|
|
||||||
---
|
|
||||||
|
|
||||||
## Zusammenfassung
|
|
||||||
Konsistente YAML-Felder verbessern Filter, Dataview-Queries und Vektor-Retrieval.
|
|
||||||
|
|
||||||
## Begründung / Details
|
|
||||||
- Parser benötigen stabile Feldnamen (z. B. `type`, `status`).
|
|
||||||
- Einheitliche Werte erleichtern Dashboards und Automationen.
|
|
||||||
|
|
||||||
## Belege / Quellen
|
|
||||||
- [[source-obsidian-properties]]
|
|
||||||
- [[concept-vektorsuche-qdrant]]
|
|
||||||
|
|
||||||
## Nächste Schritte
|
|
||||||
- [ ] `tags`-Mapping in `_meta/tags.md` erstellen
|
|
||||||
- [ ] Beispiel-Queries für Dataview hinterlegen
|
|
||||||
|
|
||||||
## Mögliche Verbindungen
|
|
||||||
- [[project-mindnet]]
|
|
||||||
- [[20250902-1730-task-yaml-standard-migration]]
|
|
||||||
|
|
||||||
### 8.2 Erfahrung (`experience`)
|
|
||||||
---
|
|
||||||
title: "Erfahrung: Journal → Erfahrung extrahieren erhöht Klarheit"
|
|
||||||
id: "20250902-1840-experience-journal-extraktion"
|
|
||||||
type: "experience"
|
|
||||||
status: "active"
|
|
||||||
created: 2025-09-02T18:40:00+02:00
|
|
||||||
updated: 2025-09-02T18:40:00+02:00
|
|
||||||
tags: ["area/personal","type/experience","topic/journal","topic/reflexion"]
|
|
||||||
area: "personal"
|
|
||||||
project: "project-mindnet"
|
|
||||||
---
|
|
||||||
|
|
||||||
## Kontext
|
|
||||||
Wiederkehrende Journaleinträge enthalten Rohmaterial für prägende Einsichten.
|
|
||||||
|
|
||||||
## Beobachtung
|
|
||||||
Wird Material regelmäßig zu `experience`-Knoten verdichtet, sind spätere Bezüge einfacher.
|
|
||||||
|
|
||||||
## Interpretation
|
|
||||||
Abstraktion + Verlinkung (z. B. auf [[concept-persoenliches-leitbild]]) erhöhen Nutzen und Wiederfindbarkeit.
|
|
||||||
|
|
||||||
## Implikationen
|
|
||||||
Pro Woche 15–30 Minuten „Verdichtungs-Review“ einplanen.
|
|
||||||
|
|
||||||
## Mögliche Verbindungen
|
|
||||||
- [[journal-20250902]]
|
|
||||||
- [[concept-persoenliches-leitbild]]
|
|
||||||
|
|
||||||
### 8.3 Aufgabe (`task`)
|
|
||||||
---
|
|
||||||
title: "Task: YAML-Standard in Altbeständen ergänzen"
|
|
||||||
id: "20250902-1850-task-yaml-migration"
|
|
||||||
type: "task"
|
|
||||||
status: "active"
|
|
||||||
created: 2025-09-02
|
|
||||||
updated: 2025-09-02
|
|
||||||
due: 2025-09-10
|
|
||||||
start: 2025-09-03
|
|
||||||
priority: 2
|
|
||||||
effort_min: 120
|
|
||||||
area: "mindnet"
|
|
||||||
project: "project-mindnet"
|
|
||||||
tags: ["area/mindnet","type/task","topic/migration"]
|
|
||||||
depends_on: []
|
|
||||||
---
|
|
||||||
|
|
||||||
## Checkliste
|
|
||||||
- [ ] Triage: Notiz-Typ bestimmen
|
|
||||||
- [ ] Minimal-YAML ergänzen (Pflichtfelder)
|
|
||||||
- [ ] Dateinamen vereinheitlichen
|
|
||||||
- [ ] „Mögliche Verbindungen“ pflegen
|
|
||||||
|
|
||||||
## Notizen
|
|
||||||
Beginne mit zuletzt genutzten Notizen (Pareto 20/80).
|
|
||||||
|
|
||||||
## Mögliche Verbindungen
|
|
||||||
- [[project-mindnet]]
|
|
||||||
- [[20250902-1830-thought-yaml-recall]]
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 9. Schrittweise Migration bestehender freier Notizen
|
## 8. Volltext & Filter
|
||||||
|
|
||||||
**Phase A – Triage (schnell)**
|
- Textindex auf `mindnet_chunks.body` für exakte Term/Operator-Queries.
|
||||||
1. Sichtung & **Typ** bestimmen (`type`).
|
- Payload-Filter zur schnellen Einschränkung (tags, types, time-buckets, visibility u. a.).
|
||||||
2. **Minimal-YAML** einfügen (`title`, `id`, `type`, `status=draft`, `created≈Dateidatum`, `tags`).
|
|
||||||
3. **Dateiname** grob anpassen (Schema + Slug).
|
|
||||||
|
|
||||||
**Phase B – Strukturieren (fokussiert)**
|
|
||||||
1. **Abschnitte** nach Typ-Template ergänzen.
|
|
||||||
2. **Zerlegen**: mehrere Gedanken/Ereignisse → je **eigene Datei**, ursprüngliche Datei wird zum **Index** mit Links.
|
|
||||||
3. **Verlinken**: relevante `[[Knoten]]` + **„Mögliche Verbindungen“** pflegen.
|
|
||||||
|
|
||||||
**Phase C – Verfeinern (bei Bedarf)**
|
|
||||||
1. **Status & Felder** pflegen (`status`, `priority`, `due`, `effort_min`).
|
|
||||||
2. **Tag-Harmonisierung**: Mapping-Tabelle pflegen (`_meta/tags.md`).
|
|
||||||
3. **Aliases** ergänzen (Such-/Linkvarianten).
|
|
||||||
|
|
||||||
**Priorisierung:** Starte mit **aktuellen & häufig genutzten** Notizen (Pareto-Prinzip), Altbestände nachziehen, wenn sie wieder gebraucht werden.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10. Dataview-Beispiele (direkt nutzbar)
|
## 9. Edges – Beziehungen
|
||||||
|
|
||||||
**Offene Aufgaben dieser Woche (nach Priorität):**
|
- **Explizit**: Jede tatsächliche Beziehung wird als Kante gespeichert (kein Workaround).
|
||||||
|
- Felder u. a.: `src_id`, `dst_id`, `relation`, `direction`, `confidence (0–1)`, `evidence` (Chunk-IDs), `created_at`, `updated_at`.
|
||||||
TABLE due, priority, effort_min
|
- **edge_defaults** liefern **Interpretations-Hints** (Gewichte, Symmetrie, Transitivität) und parametrisieren abgeleitete Edges; sie **ersetzen** keine Originalkanten.
|
||||||
FROM "31_tasks"
|
|
||||||
WHERE type = "task" AND status != "done"
|
|
||||||
AND due >= date(today) - dur(1 week)
|
|
||||||
AND due <= date(today) + dur(1 week)
|
|
||||||
SORT priority ASC, due ASC
|
|
||||||
|
|
||||||
**Aktive Projekte mit Meilensteinen:**
|
|
||||||
|
|
||||||
LIST
|
|
||||||
FROM "30_projects"
|
|
||||||
WHERE type = "project" AND status = "active"
|
|
||||||
SORT priority ASC, file.ctime DESC
|
|
||||||
|
|
||||||
**Gedanken im Bereich mindnet, jüngste zuerst:**
|
|
||||||
|
|
||||||
TABLE created, status
|
|
||||||
FROM "10_thoughts"
|
|
||||||
WHERE type = "thought" AND area = "mindnet"
|
|
||||||
SORT created DESC
|
|
||||||
LIMIT 50
|
|
||||||
|
|
||||||
**Quellen ohne `source_url` (Pflegebedarf):**
|
|
||||||
|
|
||||||
TABLE title, created
|
|
||||||
FROM "50_sources"
|
|
||||||
WHERE type = "source" AND (source_url = "" OR !source_url)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 11. Validierung & Pflege
|
## 10. Regeln & Policies (types.yaml)
|
||||||
|
|
||||||
- **Wöchentlicher Review-Slot (15–30 Min.):**
|
- Per Typ/Relation:
|
||||||
- Neue Notizen triagieren (`type`, `status`, `area`)
|
- Default-Eigenschaften (Gewicht, Directedness, Symmetrie, Transitivität).
|
||||||
- 5 Alt-Notizen migrieren (YAML + Dateiname + Verlinkung)
|
- Scoring-Hooks für Graph-Kontext im Retriever.
|
||||||
- **Dataview-Checks:** fehlende `source_url`, offene Tasks, aktive Projekte, veraltete `status`.
|
- (später) Privacy-Hooks je Relation (Mindest-Visibility).
|
||||||
- **Optional (technisch):**
|
|
||||||
- **Pre-commit Hook**: prüft Pflichtfelder (`title`, `id`, `type`, `status`, `created`, `tags`)
|
|
||||||
- **ID-Generator**: `YYYYMMDD-HHMM-type-slug` bei Neuanlage
|
|
||||||
- **Link-Linter**: prüft tote `[[Links]]`
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 12. Versions- & Änderungsrichtlinien
|
## 11. Retrieval & Ranking (Komposition)
|
||||||
|
|
||||||
- Dieses Dokument pflegt `version` & `updated`.
|
- Hybrid: Vektor-Score + Filter + optional BM25/Volltext.
|
||||||
- **Breaking Changes** (z. B. Feldumbenennungen) in `_meta/changelog.md` dokumentieren.
|
- Zeitgewicht (Decays) und Graph-Kontext werden komponiert (Formel siehe 16.3).
|
||||||
- **Migrationen** (Tag-Refactorings, Typänderungen) als Skript oder Batch-Checkliste planen.
|
- Konfiguration über `.env`/`config.yaml` (gewichts- und halbwertszeit-basiert).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 13. Typen-Kurzreferenz
|
## 12. Erklärbarkeit
|
||||||
|
|
||||||
- `thought` – Gedanke/These
|
- „Warum-Pfad“: Top-Chunks mit Score-Zerlegung (Vektor/Time/Graph), Graph-Kette, Provenienz.
|
||||||
- `experience` – Beobachtung/Erfahrung mit Interpretation & Implikationen
|
- Mehrschichtige Narrative (Kindheit/Erlebnisse, Werte/Prinzipien, Ziele/Leitbild, Gegenwart).
|
||||||
- `task` – umsetzbare Arbeitseinheit (mit `due`, `effort_min`, `priority`)
|
|
||||||
- `project` – Container für Ziele/Tasks/Meilensteine
|
|
||||||
- `source` – externe Quelle (Artikel, Buch, Video)
|
|
||||||
- `concept` – definierter Begriff / Prinzip / Leitbild
|
|
||||||
- `person` – Bezugsperson/Kontakt
|
|
||||||
- `meeting` – Termin mit Agenda, Notizen, erzeugten Aufgaben
|
|
||||||
- `journal` – Tagebuch-/Reiseeintrag (tagesbasiert)
|
|
||||||
- `milestone` – Wendepunkt / Meilenstein im Lebens-/Projektverlauf
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 14 Erweiterungen
|
## 13. Betrieb & Sicherheit
|
||||||
|
|
||||||
Zukünftige Typen möglich (z. B. decision, recommendation).
|
- Lokal auf Single-Node starten (Backups via Snapshots).
|
||||||
|
- Zugriff nur intern/Reverse-Proxy; Festplattenverschlüsselung OS-seitig.
|
||||||
|
- Migrationspfad zu Qdrant-Distributed (Shards, Replikation) einplanen.
|
||||||
|
|
||||||
Personalisierung durch zusätzliche Properties (values, goals)
|
---
|
||||||
|
|
||||||
|
## 14. Qualität & Monitoring
|
||||||
|
|
||||||
|
- Offline-Eval-Sets, Telemetrie (lokal), Drift-Checks, manuelle Reviews.
|
||||||
|
- Kennzahlen: Hit-Rate, Overlap, Quellen-Alter, Erklärungspfad-Nutzung.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. Offene Punkte
|
||||||
|
|
||||||
|
- Ausdetaillierung `types.yaml` (Relationen, Hooks).
|
||||||
|
- UI-Oberfläche für Erklärpfade und Provenienz.
|
||||||
|
- Batch-Jobs für abgeleitete Edges.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 16. Ergänzungen 2025-11-09 (v1.5.0)
|
||||||
|
|
||||||
|
**Kontext:** Basierend auf den neuen Rahmenbedingungen (nur Deutsch; perspektivische externe Abfragen; einheitliche Vertraulichkeit mit Option auf feinere Stufen; Recency-Priorisierung als Default; Regeln später konfigurierbar; zunächst lokale Speicherung mit optionaler Migration; heutige Hardware, später skalierbar; hohe Erklärbarkeit mit mehreren Erklärungsebenen) werden folgende Entscheidungen und Erweiterungen getroffen. Alle bisherigen Inhalte bleiben gültig und werden **nicht** verändert.
|
||||||
|
|
||||||
|
### 16.1 Sprach- und Inhaltsrichtlinie (nur Deutsch)
|
||||||
|
- `lang: "de"` als verpflichtendes Feld in allen Note-, Chunk- und Edge-Payloads.
|
||||||
|
- Für spätere Mehrsprachigkeit **nicht** die Collection splitten, sondern ein Feld `lang` + optional `lang_original` behalten; Translation-Artefakte als eigener `type: translation` (verweisend auf `source_id`).
|
||||||
|
|
||||||
|
### 16.2 Vertraulichkeit & Sichtbarkeit
|
||||||
|
- Heute: einheitliche Stufe `visibility: "internal"`.
|
||||||
|
- Vorausschauend: reservierte Felder in allen Payloads
|
||||||
|
- `visibility`: `"public" | "internal" | "restricted"`
|
||||||
|
- `sensitivity`: Integer 0–3 (0 = unkritisch, 3 = hochsensibel)
|
||||||
|
- `acl`: Liste erlaubter Rollen/Personen-IDs (App-Layer erzwingt Filter vor Query).
|
||||||
|
- Access-Control erfolgt **im Applikations-Layer** mittels Qdrant-Payload-Filtern (`must: [ { key: "visibility", match: { any: [...] }}, ... ]`); Qdrant bleibt „dumm“ in Bezug auf ACL.
|
||||||
|
|
||||||
|
### 16.3 Recency-Modell & Ranking (konfigurierbar)
|
||||||
|
- Default: **Zeitnähe** priorisieren (sanfte Abwertung älterer Inhalte).
|
||||||
|
- Im Retrieval kombinieren wir:
|
||||||
|
1) Vektor-Score (cosine) aus Qdrant
|
||||||
|
2) Zeitgewicht `time_decay = exp( - ln(2) * age_days / half_life_days )`
|
||||||
|
3) Graph-Kontext aus Edges (z. B. Degree, Path-Hops zum Anker „Ich/Persona“)
|
||||||
|
- Konfigurierbare Gewichte (per `.env` oder `config.yaml`):
|
||||||
|
```text
|
||||||
|
FINAL_SCORE = w_vec * norm(vec_score)
|
||||||
|
+ w_time * time_decay
|
||||||
|
+ w_graph * graph_context_score
|
||||||
|
# Vorschlag: w_vec=0.70, w_time=0.20, w_graph=0.10; half_life_days=365
|
||||||
|
```
|
||||||
|
- Für Fragen, die „zeitlos“ sind, kann `half_life_days = inf` gesetzt werden (kein Decay).
|
||||||
|
|
||||||
|
### 16.4 Erklärbarkeit („Warum diese Antwort?“)
|
||||||
|
- Jeder Antwort wird ein **Erklärpfad** mitgegeben (optional im UI):
|
||||||
|
- Top-Chunks (mit Score-Zerlegung w_vec/w_time/w_graph)
|
||||||
|
- **Kausalpfad** im Graphen: `Note → Edge(relation, confidence) → Note …`
|
||||||
|
- **Provenienz** (Quelle, Erstellungszeit, letzte Änderung).
|
||||||
|
- Ein **Multi-Layer-Explanations**-Modul ordnet Beiträge zu: Kindheit/Erlebnisse, Werte & Prinzipien, aktuelle Ziele/Leitbild, Gegenwartskontext.
|
||||||
|
|
||||||
|
### 16.5 Qdrant-Schemata (Ergänzungen, rückwärtskompatibel)
|
||||||
|
- **Collections** bleiben: `mindnet_notes`, `mindnet_chunks`, `mindnet_edges` (+ `mindnet_people` falls noch nicht vorhanden).
|
||||||
|
- Gemeinsame Zusatzfelder (alle Payloads):
|
||||||
|
- `lang: "de"`
|
||||||
|
- `visibility`/`sensitivity`/`acl` (s. 16.2)
|
||||||
|
- `created_at` (ISO-8601), `updated_at` (ISO-8601), `age_days` (App-seitig berechnet)
|
||||||
|
- **Chunks**:
|
||||||
|
- Named-Vectors (z. B. `"content"`, optional `"title"`, `"tags"` für Hybrid-Scores).
|
||||||
|
- `text_index` auf `body` für BM25-ähnliche Volltextsuche (Qdrant-Textindex).
|
||||||
|
- `recency_group`: Jahr-Monat `YYYY-MM` (ermöglicht Pre-Filter für „letzte N Monate“).
|
||||||
|
- **Edges** (explizit, keine Workarounds):
|
||||||
|
- Felder: `src_id`, `dst_id`, `relation`, `direction`, `confidence` (0–1), `evidence` (Chunk-IDs), `created_at`, `updated_at`.
|
||||||
|
- **edge_defaults** bleiben als **Interpretations-Hints**: z. B. `default_weight`, `symmetry`, `transitivity`. Sie ersetzen **nicht** echte Kanten, sondern parametrisieren abgeleitete/zusätzliche Kantenbildung.
|
||||||
|
- Abgeleitete Edges: optionaler Batch-Job erzeugt *ergänzende* Kanten entlang `types.yaml`-Regeln (z. B. Backlinks, „inspired_by“ aus Zitaten), niemals statt der Originalkanten.
|
||||||
|
|
||||||
|
### 16.6 Regeln & Typen (konfigurierbar)
|
||||||
|
- `types.yaml` erweitert um:
|
||||||
|
- Relationstypen mit Default-Eigenschaften (`weight`, `directed`, `symmetric`, `transitive`).
|
||||||
|
- **Scoring-Hooks**: pro Relation optionaler Einfluss auf `graph_context_score`.
|
||||||
|
- **Privacy-Hooks**: pro Relation Minimal-Visibility (z. B. Beziehungen zu Personen sind mindestens `internal`).
|
||||||
|
|
||||||
|
### 16.7 Speicher- & Betriebsstrategie (heute lokal, später skalierbar)
|
||||||
|
- **Heute**: Single-Node Qdrant mit HNSW-Index, Text-Index und Payload-Filtern; regelmäßiges Backup (Snapshots).
|
||||||
|
- **Morgen**: Migrationspfad in **Qdrant-Distributed** (Shards, ggf. Replikation) bei Bedarf; Parameter-Gleichheit sichern (named-vectors, Metrik, HNSW-Param).
|
||||||
|
- **Sicherheit**: Zugriff nur über internes Netz/Reverse-Proxy, Auth vor Qdrant; Festplatten-Verschlüsselung auf OS-Ebene (z. B. LUKS).
|
||||||
|
|
||||||
|
### 16.8 Persona-Layer („meine Essenz“)
|
||||||
|
- Eigenständiger **Persona-Knoten** (z. B. Note `person/lars`), verbunden mit:
|
||||||
|
- Werten/Prinzipien, Leitbild, prägenden Erlebnissen, Zielen, Rollen.
|
||||||
|
- Jede Antwort kann gegen diesen Knoten **kontextualisiert** werden (Graph-Pfadnähe).
|
||||||
|
- Feld `persona_weight` in Chunks/Notes zur Priorisierung persönlicher Relevanz.
|
||||||
|
|
||||||
|
### 16.9 Evaluierung & Qualitätssicherung
|
||||||
|
- **Offline-Sets** aus echten Fragen (inkl. gewünschter Perspektiven) mit erwarteten Top-Quellen.
|
||||||
|
- **Telemetrie** (lokal): Query-Arten, Treffer-Overlaps, Zeitgewicht-Effekte, „Warum-Pfad“ genutzt/ignoriert.
|
||||||
|
- **Drift-Checks**: Anteil sehr alter vs. neuer Quellen in Antworten; manuelle Reviews.
|
||||||
|
|
||||||
|
### 16.10 Offene Punkte / Nächste Schritte
|
||||||
|
1) `types.yaml` um Relationseigenschaften erweitern (s. 16.6).
|
||||||
|
2) Retriever-Komposition inkl. Zeitgewicht (s. 16.3) implementieren.
|
||||||
|
3) Explanations-Pfad inkl. Provenienzoberfläche (s. 16.4) im UI.
|
||||||
|
4) Feld-Erweiterungen in Import-Pipelines (`lang`, `visibility`, Zeitfelder).
|
||||||
|
5) Optional: Batch-Job für abgeleitete Edges nach Regeln (ergänzend, nicht ersetzend).
|
||||||
@@ -0,0 +1,291 @@
|
|||||||
|
# Mindnet – Fachliche Architektur (Stand: 2025-11-17)
|
||||||
|
|
||||||
|
> Dieses Dokument beschreibt **was** Mindnet fachlich tut und **warum** – mit Fokus auf die Erzeugung und Nutzung von **Edges** (Kanten) zwischen Notizen/Chunks. Die technische Umsetzung (Module, Flags, Schemas) wird nur dort angerissen, wo es zum Verständnis nötig ist und im technischen Dokument detailliert.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0) Zielbild & Grundprinzip
|
||||||
|
|
||||||
|
Mindnet wandelt Obsidian-Markdown-Notizen in einen **Graph aus Punkten und Kanten** um.
|
||||||
|
Die drei zentralen Artefakt-Sammlungen lauten:
|
||||||
|
|
||||||
|
- `mindnet_notes` – genau **eine** Note pro Markdown-Datei (Metadaten & Hashes)
|
||||||
|
- `mindnet_chunks` – semantische Teilstücke einer Note (Fenster/„Chunks“)
|
||||||
|
- `mindnet_edges` – gerichtete Beziehungen zwischen Knoten (Chunks/Notes)
|
||||||
|
|
||||||
|
Die Import-Pipeline erzeugt diese Artefakte **deterministisch** und **idempotent** (erneute Läufe überschreiben konsistent statt zu duplizieren). Die Import-Schritte sind: *parse → chunk → edge → upsert*. :contentReference[oaicite:0]{index=0}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1) Notizen & Chunks (fachliche Perspektive)
|
||||||
|
|
||||||
|
### 1.1 Notiz (Note)
|
||||||
|
- Repräsentiert eine fachliche Einheit (z. B. *„Vector DB Basics“*, *„KI-Projekt“*).
|
||||||
|
- Trägt Eigenschaften (Titel, Typ, Zeitstempel, optionale Policies).
|
||||||
|
- **Typ** (*type*) steuert u. a. Chunk-Profil und Default-Relationen (siehe §4).
|
||||||
|
- **retriever_weight** und **chunk_profile** werden systematisch an Note **und** Chunks gespiegelt, damit der Retriever beides nutzen kann. :contentReference[oaicite:1]{index=1}
|
||||||
|
|
||||||
|
### 1.2 Chunk
|
||||||
|
- Ausschnitt/Textfenster aus der Note, als eigenständiger Such-Anker.
|
||||||
|
- Jeder Chunk gehört **genau einer** Note.
|
||||||
|
- Chunks bilden eine Sequenz (1…N) – das ermöglicht *next/prev*.
|
||||||
|
|
||||||
|
> **Wichtig:** Chunking-Profile (short/medium/long) kommen aus `types.yaml` (per Note-Typ), können aber lokal überschrieben werden. Die effektiven Werte werden bei der Payload-Erzeugung bestimmt. :contentReference[oaicite:2]{index=2}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2) Edges – fachliche Typen & Bedeutung
|
||||||
|
|
||||||
|
Edges kodieren Beziehungen. Sie sind **gerichtet** und werden in `mindnet_edges` gespeichert.
|
||||||
|
|
||||||
|
### 2.1 Struktur-Kanten
|
||||||
|
- **belongs_to**: *Chunk → Note* (Zuordnung)
|
||||||
|
- **next / prev**: Kettenbeziehungen zwischen aufeinanderfolgenden Chunks derselben Note
|
||||||
|
→ ermöglichen „lineares Weiterlesen“ oder Kontextpfade in Antworten.
|
||||||
|
Diese Kanten entstehen immer, unabhängig von Inhalten. :contentReference[oaicite:3]{index=3}
|
||||||
|
|
||||||
|
### 2.2 Inhalts-Kanten (explizit)
|
||||||
|
- **references (chunk-scope)**: aus dem Textfenster des Chunks extrahierte Links.
|
||||||
|
Standardquelle: **Wikilinks** `[[…]]` – einschließlich Varianten `[[id#anchor]]`, `[[id|label]]`.
|
||||||
|
→ Mindnet reduziert robust auf die Ziel-ID (Titel/ID-Normalisierung). :contentReference[oaicite:4]{index=4}
|
||||||
|
- **Explizite Inline-Relationen**: im Fließtext notierte Relationen, z. B.:
|
||||||
|
rel: similar_to [[Vector DB Basics]]
|
||||||
|
→ werden als Relation **similar_to** mit hoher Confidence materialisiert; *rule_id* `inline:rel`.
|
||||||
|
(Diese Form ist besonders kompakt in Obsidian editierbar.)
|
||||||
|
- **Callout-Edges** (optional):
|
||||||
|
> [!edge] related_to: [[Vector DB Basics]] [[Embeddings 101]]
|
||||||
|
→ erzeugt mehrere **related_to**-Kanten aus einer Zeile; *rule_id* `callout:edge`.
|
||||||
|
(Für kuratierte, sichtbare Linklisten am Notizende.)
|
||||||
|
|
||||||
|
> Alle expliziten Quellen (Wikilink, Inline-Rel, Callout) sind **gleichwertige Primärbelege** und werden als *provenance=explicit* geführt. Wo zutreffend, tragen Edges *rule_id* wie `explicit:wikilink`, `inline:rel`, `callout:edge`. :contentReference[oaicite:5]{index=5}
|
||||||
|
|
||||||
|
### 2.3 Typ-basierte Default-Kanten (regelbasiert)
|
||||||
|
Jede Note hat einen *Typ* (z. B. `concept`, `project`, `profile`). In `config/types.yaml` definieren wir pro Typ **edge_defaults**, z. B.:
|
||||||
|
|
||||||
|
- `concept`: `["references","related_to"]`
|
||||||
|
- `project`: `["references","depends_on"]`
|
||||||
|
|
||||||
|
Regel: **Für jede gefundene explizite Referenz** (s. o.) werden **zusätzliche** Edges nach diesen Defaults erzeugt. Beispiel: Ein *project* mit `edge_defaults=["depends_on"]` erzeugt zu *jedem* explizit referenzierten Ziel **zusätzlich** eine `depends_on`-Kante.
|
||||||
|
Diese Kanten tragen *provenance=rule* und eine **rule_id** der Form
|
||||||
|
`edge_defaults:{note_type}:{relation}` (z. B. `edge_defaults:project:depends_on`). :contentReference[oaicite:6]{index=6}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3) Edge-Payload – Felder & Semantik
|
||||||
|
|
||||||
|
Jede Kante hat mindestens:
|
||||||
|
|
||||||
|
- `kind` – Beziehungsart *(belongs_to, next, prev, references, related_to, depends_on, similar_to, …)*
|
||||||
|
- `scope` – `"chunk"` (Standard) oder `"note"` (optional für Backlinks/Note-Scope)
|
||||||
|
- `source_id`, `target_id` – Quell-/Ziel-Knoten (Chunk- oder Note-ID)
|
||||||
|
- `note_id` – **Owner-Note** (die Note, aus der die Kante stammt; dient der schnellen Löschung/Filterung)
|
||||||
|
|
||||||
|
Erweiterte/abgeleitete Felder (V2-Superset):
|
||||||
|
|
||||||
|
- `provenance` – `"explicit"` (Wikilink/Inline/Callout) oder `"rule"` (Typ-Defaults)
|
||||||
|
- `rule_id` – maschinenlesbare Regelquelle, z. B.
|
||||||
|
- `explicit:wikilink`, `inline:rel`, `callout:edge`
|
||||||
|
- `edge_defaults:{note_type}:{relation}`
|
||||||
|
- `confidence` – 0.0–1.0; Heuristik, z. B. `1.0` für Wikilinks/Backlinks, `0.95` für `inline:rel`, `0.7` für Typ-Defaults.
|
||||||
|
|
||||||
|
**Dedup-Schlüssel** (fachlich): Gleichlautende `(source_id, target_id, kind, scope, rule_id)` werden nicht mehrfach geschrieben. :contentReference[oaicite:7]{index=7}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4) Typ-Registry (`config/types.yaml`)
|
||||||
|
|
||||||
|
### 4.1 Zweck
|
||||||
|
- Steuert **Chunking-Profile** (`short|medium|long`) **pro Typ**
|
||||||
|
- Liefert **retriever_weight** (Note-/Chunk-Gewichtung im Ranking) **pro Typ**
|
||||||
|
- Definiert **edge_defaults** je Typ (s. o.)
|
||||||
|
|
||||||
|
Der Importer lädt die Registry aus `MINDNET_TYPES_FILE` oder nutzt Fallbacks. **Frontmatter-Overrides** (z. B. `retriever_weight`) werden respektiert; am Ende schreibt der Importer die **effektiven** Werte zuverlässig in Note- und Chunk-Payload. :contentReference[oaicite:8]{index=8}
|
||||||
|
|
||||||
|
### 4.2 Beispiel (auslieferungsnah)
|
||||||
|
version: 1.0
|
||||||
|
types:
|
||||||
|
concept:
|
||||||
|
chunk_profile: medium
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
retriever_weight: 0.35
|
||||||
|
project:
|
||||||
|
chunk_profile: long
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
retriever_weight: 0.97
|
||||||
|
profile:
|
||||||
|
chunk_profile: long
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
retriever_weight: 0.66
|
||||||
|
|
||||||
|
**Auflösung im Importer**
|
||||||
|
- `effective_chunk_profile(note_type, registry)` → `"short|medium|long"|None`
|
||||||
|
- `effective_retriever_weight(note_type, registry)` → `float|None`
|
||||||
|
- Ergebnis wird in `note_payload` **und** `chunk_payloads` gespiegelt. :contentReference[oaicite:9]{index=9}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5) Wie entstehen Edges aus Markdown?
|
||||||
|
|
||||||
|
### 5.1 Quelle „Wikilinks“ (Primärfall)
|
||||||
|
- Parser extrahiert alle `[[…]]` (Label/Anker werden auf die Ziel-ID reduziert).
|
||||||
|
- Pro Fund entsteht **eine** `references`-Kante (Chunk-Scope).
|
||||||
|
- Zusätzlich: Typ-Defaults (z. B. `depends_on`, `related_to`) pro Ziel. :contentReference[oaicite:10]{index=10}
|
||||||
|
|
||||||
|
### 5.2 Quelle „Inline-Relation“ (Optional, kompakt)
|
||||||
|
- Im Fließtext markierte Relation, z. B.:
|
||||||
|
rel: similar_to [[Vector DB Basics]]
|
||||||
|
- Erzeugt `similar_to`-Kanten; *rule_id* `inline:rel`; `confidence≈0.95`.
|
||||||
|
- Geeignet für sparsame, gut lesbare Referenzierung. (Erfassung in der Chunk-Window-Analyse.) :contentReference[oaicite:11]{index=11}
|
||||||
|
|
||||||
|
### 5.3 Quelle „Callout-Edges“ (Optional, kuratiert)
|
||||||
|
- Klar sichtbare, redaktionell gepflegte Liste am Ende einer Note:
|
||||||
|
> [!edge] related_to: [[Vector DB Basics]] [[Embeddings 101]]
|
||||||
|
- Erzeugt je Ziel eine `related_to`-Kante; *rule_id* `callout:edge`.
|
||||||
|
- Eignet sich für „Leselisten“ und „Siehe auch“. (Zählung als *callout_total* in Reports, wo vorhanden.) :contentReference[oaicite:12]{index=12}
|
||||||
|
|
||||||
|
### 5.4 Struktur-Kanten
|
||||||
|
- `belongs_to` pro Chunk
|
||||||
|
- `next/prev` aus Nachbarbezug (aus Chunk-Sequenz/Neighbors) :contentReference[oaicite:13]{index=13}
|
||||||
|
|
||||||
|
> **Hinweis zu Note-Scope:** Optional können zusätzlich Note-weite *references/backlinks* aus der Gesamtheit aller Chunk-Funde abgeleitet werden (Konfig-Flag/ENV). Das ist für globale Graph-Analysen nützlich, aber im Retrieval seltener erforderlich. :contentReference[oaicite:14]{index=14}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6) Confidence & Provenance – wozu?
|
||||||
|
|
||||||
|
Der Retriever kann Edges gewichten:
|
||||||
|
- **provenance**: *explicit* > *rule*
|
||||||
|
- **confidence**: numerische Feinsteuerung (z. B. Inline-Relation etwas höher, Defaults etwas niedriger)
|
||||||
|
- **retriever_weight (Note/Chunk)**: skaliert die Relevanz des gesamten Knotens
|
||||||
|
|
||||||
|
Eine typische Gewichtung (konfigurierbar) ist:
|
||||||
|
explicit: 1.0, rule: 0.8, default_resolved: 0.6, default: 0.2
|
||||||
|
Damit bevorzugt der Graph **kuratiertes Wissen** (explizit notierte Links) vor „erweiterten“ Default-Ableitungen. :contentReference[oaicite:15]{index=15}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7) Semantik ausgewählter `kind`-Werte
|
||||||
|
|
||||||
|
- `references` – „Nutzt/erwähnt“; neutral, aber stützend für Kontext.
|
||||||
|
- `related_to` – Ähnlichkeit/Verwandtschaft (symmetrisch interpretierbar).
|
||||||
|
- `similar_to` – noch engere Ähnlichkeit; oft aus Inline-Rel (bewusst gesetzt).
|
||||||
|
- `depends_on` – fachliche Abhängigkeit (z. B. „Projekt X hängt von Y ab“).
|
||||||
|
- `belongs_to`, `next`, `prev` – Struktur.
|
||||||
|
|
||||||
|
> Symmetrische Relationen (z. B. `related_to`, `similar_to`) können **explizit** nur einseitig notiert sein, aber im Retriever beidseitig interpretiert werden (z. B. Query-Zeit-Expansion), um Redundanz im Vault zu vermeiden. :contentReference[oaicite:16]{index=16}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8) Frontmatter-Eigenschaften – Rolle & Empfehlung
|
||||||
|
|
||||||
|
Frontmatter-Eigenschaften (Properties) bleiben **minimiert**:
|
||||||
|
- **type** – steuert Typ-Defaults via Registry (Pflicht für differenziertes Verhalten).
|
||||||
|
- **retriever_weight** *(optional)* – kann typbasierte Defaults überschreiben.
|
||||||
|
- **chunk_profile** *(optional)* – kann typbasierte Defaults überschreiben.
|
||||||
|
- **links** *(optional, Bestandskompatibilität)* – wenn vorhanden (z. B. `depends_on: …`), können sie als explizite Kanten materialisiert werden; die Pflege in freiem Text (Inline/Callout) ist jedoch **benutzerfreundlicher** und weniger fehleranfällig.
|
||||||
|
|
||||||
|
Empfehlung: **Links primär über Text (Wikilink/Inline/Callout) pflegen**, nicht als Properties. Die Registry regelt Standards; Properties dienen nur gezielten Ausnahmen. :contentReference[oaicite:17]{index=17}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9) Lösch-/Update-Garantien (Idempotenz)
|
||||||
|
|
||||||
|
- Jede Note hat einen stabilen **note_id** (Frontmatter/Hash).
|
||||||
|
- Vor einem Upsert können *alte Chunks/Edges einer Note* gefiltert gelöscht werden (`note_id`-Filter) – das hält Collections sauber bei Re-Imports.
|
||||||
|
- Es gibt Abkürzungen für „fehlertolerante Rebuilds“: *Existenz-Checks* statt Full-Scan. :contentReference[oaicite:18]{index=18}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10) Beispiel – Von Markdown zu Kanten
|
||||||
|
|
||||||
|
**Markdown (Auszug)**
|
||||||
|
# Relations Showcase
|
||||||
|
…
|
||||||
|
rel: similar_to [[Vector DB Basics]]
|
||||||
|
…
|
||||||
|
[[Embeddings 101]]
|
||||||
|
|
||||||
|
> [!edge] related_to: [[Vector DB Basics]] [[Embeddings 101]]
|
||||||
|
|
||||||
|
**Ergebnis (fachlich)**
|
||||||
|
- `references(Chunk→Vector DB Basics)` + `references(Chunk→Embeddings 101)`
|
||||||
|
- `similar_to(Chunk→Vector DB Basics)` mit `rule_id=inline:rel`, `confidence≈0.95`
|
||||||
|
- `related_to(Chunk→Vector DB Basics)` und `related_to(Chunk→Embeddings 101)`
|
||||||
|
via Callout; `rule_id=callout:edge`
|
||||||
|
- Zusätzliche **Typ-Defaults** je Quell-Typ, z. B. `edge_defaults:concept:related_to` → weitere `related_to`-Kanten
|
||||||
|
|
||||||
|
*Strukturkanten* (`belongs_to`, `next/prev`) fallen wie üblich an. (Die tatsächlichen Zähllogiken erscheinen im Import-Report/Smoke-Tests.) :contentReference[oaicite:19]{index=19}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11) Abfrage-Implikationen (fachlich)
|
||||||
|
|
||||||
|
Retriever und Query-Layer berücksichtigen:
|
||||||
|
1) **Knoten-Gewicht** (`retriever_weight`) aus Typ/Property (Note & Chunk).
|
||||||
|
2) **Edge-Provenance/Confidence** – explizit > regelbasiert; höhere Confidence bevorzugt.
|
||||||
|
3) **Edge-Topologie** – z. B. 1-Hop-Nachbarschaft, Sequenzpfade (`next/prev`), Backlinks (optional).
|
||||||
|
4) **Policy-Filter** – Privacy, Tags, Created usw. via Payload-Indizes.
|
||||||
|
|
||||||
|
Die Abfrage kann so z. B. Ergebnisse bevorzugen, die **explizit** in einem *Projekt* als `depends_on` genannt sind – und nur nachrangig solche, die „typbedingt“ (`edge_defaults`) ähnlich/abhängig sind. :contentReference[oaicite:20]{index=20}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12) Qualitäts- und Testkriterien (fachlich)
|
||||||
|
|
||||||
|
- **Keine Duplikate** gleicher `(src, relation, dst)` in der Collectionsicht (Dedup).
|
||||||
|
- **Explizite Links** werden vollständig materialisiert (*explicit_total*).
|
||||||
|
- **Typ-Defaults** ergänzen, aber überdecken nicht – Counts bleiben plausibel (*defaults_total*).
|
||||||
|
- **Inline/Callout** werden separat quantifiziert (*inline_total*, *callout_total*), damit Redaktionen ihre Pflegewirkung beobachten können. :contentReference[oaicite:21]{index=21}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13) Praxisempfehlungen für den Vault
|
||||||
|
|
||||||
|
1) **Wikilinks** überall, wo eine inhaltliche Nennung stattfindet.
|
||||||
|
2) Für **gezielte Relationen** (Ähnlichkeit/Abhängigkeit) kurze **Inline-Rel**-Zeilen; z. B.:
|
||||||
|
rel: similar_to [[X]]
|
||||||
|
rel: related_to [[Y]] [[Z]]
|
||||||
|
3) Am Notizende **Callout-Edges** für kuratierte Überblicksblöcke:
|
||||||
|
> [!edge] related_to: [[X]] [[Y]]
|
||||||
|
4) **Properties** schlank halten: mindestens `type`, optional `retriever_weight`/`chunk_profile`.
|
||||||
|
5) **types.yaml** pflegen (zentrale Governance): *Welche Typen gibt es? Welche Default-Relationen sind fachlich sinnvoll?*
|
||||||
|
6) **Regelmäßige Smoke-Tests/Reports** prüfen die Kanten-Counts und Provenance-Verteilung. :contentReference[oaicite:22]{index=22}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14) Referenzen (Projektdateien & Leitlinien)
|
||||||
|
|
||||||
|
- Import-Pipeline & Registry-Auflösung (Importer, Flags, Idempotenz): `scripts/import_markdown.py`. :contentReference[oaicite:23]{index=23}
|
||||||
|
- Robuste Kantenbildung (Wikilinks, Struktur, optionale Note-Scope): `app/core/derive_edges.py` (v1.4+). :contentReference[oaicite:24]{index=24}
|
||||||
|
- V2-Edge-Superset (Provenance, rule_id, Confidence, Typ-Defaults): `app/core/edges.py` (v2.0+). :contentReference[oaicite:25]{index=25}
|
||||||
|
- Typ-Registry laden/auflösen (Fallbacks, Effektivwerte): Import-Helper / *Type-Registry*-Hilfsfunktionen. :contentReference[oaicite:26]{index=26}
|
||||||
|
- Test-/Playbook-Leitplanken (Policies, KPIs, Roadmap): *Implementation Playbook*. :contentReference[oaicite:27]{index=27}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15) Anhang – Mini-Beispiel einer Projekt-Note (verkürzt)
|
||||||
|
|
||||||
|
---
|
||||||
|
id: 20251110-ki-trainerassistent-projekt-56d2aa
|
||||||
|
title: KI Trainerassistent (Projekt)
|
||||||
|
type: project
|
||||||
|
retriever_weight: 1.0 # optional Override
|
||||||
|
chunk_profile: default # optional Override
|
||||||
|
---
|
||||||
|
|
||||||
|
# KI Trainerassistent (Projekt)
|
||||||
|
|
||||||
|
Wir nutzen [[Qdrant Vektordatenbank]] und [[Ollama LLM]] …
|
||||||
|
|
||||||
|
rel: depends_on [[Qdrant Vektordatenbank]]
|
||||||
|
rel: related_to [[Ollama LLM]]
|
||||||
|
|
||||||
|
> [!edge] depends_on: [[Qdrant Vektordatenbank]]
|
||||||
|
> [!edge] related_to: [[Ollama LLM]]
|
||||||
|
|
||||||
|
*Erwartung*:
|
||||||
|
- `references` (Wikilinks) auf **beide** Ziele
|
||||||
|
- `depends_on` & `related_to` via **Inline** und **Callout** (provenance=`explicit`, rule_id `inline:rel`/`callout:edge`)
|
||||||
|
- Zusätzlich **Typ-Defaults** aus `types.yaml` für `project` (z. B. weitere `depends_on`) – provenance=`rule`, `rule_id=edge_defaults:project:depends_on`. :contentReference[oaicite:28]{index=28}
|
||||||
|
|
||||||
|
---
|
||||||
@@ -0,0 +1,699 @@
|
|||||||
|
# Mindnet – Technische Architektur (V2, Stand: 2025-11-xx)
|
||||||
|
|
||||||
|
> **Ziel dieses Dokuments:**
|
||||||
|
> Vollständige, konsolidierte Beschreibung der aktuellen technischen Architektur von **Mindnet V2** – insbesondere der Verarbeitung von Markdown-Notizen, des Chunkings, der Kantenableitung (Edges) inkl. `rule_id` / `confidence` und der Speicherung in Qdrant.
|
||||||
|
> Das Dokument bildet den **aktuellen Implementierungsstand** ab, nicht mehr gültige Annahmen sind bereinigt, neue Konzepte (Typ-Defaults, Inline-Relationen, Edges-Schema) sind integriert.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Systemüberblick
|
||||||
|
|
||||||
|
### 1.1 Zielbild
|
||||||
|
|
||||||
|
Mindnet ist ein **persönliches Wissensnetz**. Technisch bedeutet das:
|
||||||
|
|
||||||
|
- Markdown-Notizen in einem Vault (Obsidian-kompatibel) werden
|
||||||
|
- geparst (Frontmatter + Body),
|
||||||
|
- in **Notes**, **Chunks** und **Edges** überführt,
|
||||||
|
- in **Qdrant** als drei Collections abgelegt:
|
||||||
|
- `<prefix>_notes`
|
||||||
|
- `<prefix>_chunks`
|
||||||
|
- `<prefix>_edges`.
|
||||||
|
|
||||||
|
- Das System ist so konzipiert, dass es:
|
||||||
|
- **deterministisch** arbeitet (IDs, Chunks, Kanten),
|
||||||
|
- über `types.yaml` **konfigurierbar** ist,
|
||||||
|
- später von einem LLM-basieren Agenten als **Graph-Retriever** genutzt werden kann.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 1.2 Verzeichnisstruktur (Server)
|
||||||
|
|
||||||
|
/home/llmadmin/mindnet/:
|
||||||
|
|
||||||
|
- **/app**
|
||||||
|
- `main.py` – FastAPI-App / Einstieg
|
||||||
|
- **/app/core**
|
||||||
|
- `qdrant.py` – QdrantConfig, Client-Factory, Collection-Management
|
||||||
|
- `qdrant_points.py` – Upserts / Deletes / Queries auf Punktebene
|
||||||
|
- `note_payload.py` – Bau von Note-Payloads
|
||||||
|
- `chunk_payload.py` – Bau von Chunk-Payloads
|
||||||
|
- `chunker.py` – Chunking-Logik
|
||||||
|
- `edges.py` – Datenstruktur & Hilfsfunktionen für Edges
|
||||||
|
- `derive_edges.py` – Ableitung von Edges aus Text und Typen
|
||||||
|
- **/app/models**
|
||||||
|
- `dto.py` – DTOs für API / Services
|
||||||
|
- **/app/routers**
|
||||||
|
- `qdrant_router.py` – API-Layer Richtung Qdrant
|
||||||
|
- **/app/services**
|
||||||
|
- `llm_ollama.py` – Anbindung an Ollama-LLM
|
||||||
|
- `embeddings_client.py` – Embedding-Service (z. B. Ollama / lokaler Dienst)
|
||||||
|
- **/config**
|
||||||
|
- `types.yaml` – Typdefinitionen inkl. Chunk-Profilen und Edge-Defaults
|
||||||
|
- **/scripts**
|
||||||
|
- `import_markdown.py` – Hauptimporter für Vault
|
||||||
|
- `reset_qdrant.py` – Collections löschen/neu anlegen (inkl. Indizes)
|
||||||
|
- `setup_mindnet_collections.py` – Collection-Setup (historisch / Hilfsskript)
|
||||||
|
- diverse Diagnose- und Testskripte, z. B.
|
||||||
|
- `payload_dryrun.py`
|
||||||
|
- `edges_full_check.py`
|
||||||
|
- **/tests**
|
||||||
|
- `test_edges_smoke.py` – einfache Edge-Smoke-Tests
|
||||||
|
- `test_edges_all.py` – umfassendere Checks (Counts, Plausibilität)
|
||||||
|
- `ensure_indexes_and_show.py`, `assert_payload_schema.py` – Schema/Index-Checks
|
||||||
|
- **/vault**
|
||||||
|
- produktives Vault (Obsidian)
|
||||||
|
- **/mindnet_v2_test_vault**
|
||||||
|
- Test-Vault für V2-Funktionen (Chunking, Edges, Typ-Defaults)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Datenmodell & Collections
|
||||||
|
|
||||||
|
### 2.1 Note, Chunk, Edge – Begriffe
|
||||||
|
|
||||||
|
- **Note**
|
||||||
|
Entspricht einer Markdown-Datei mit YAML-Frontmatter (`id`, `title`, `type`, …).
|
||||||
|
In Qdrant als ein Punkt in `<prefix>_notes`.
|
||||||
|
|
||||||
|
- **Chunk**
|
||||||
|
Ein Abschnitt einer Note (z. B. Absatz, Überschrifts-Block, Sliding Window).
|
||||||
|
In Qdrant als Punkt in `<prefix>_chunks`.
|
||||||
|
|
||||||
|
- **Edge**
|
||||||
|
Eine gerichtete Kante, die eine Beziehung zwischen:
|
||||||
|
- zwei Chunks,
|
||||||
|
- Chunk ↔ Note,
|
||||||
|
- oder (konzeptionell) Note ↔ Note beschreibt.
|
||||||
|
Gespeichert in `<prefix>_edges`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2.2 Qdrant Collections (Stand V2)
|
||||||
|
|
||||||
|
**Namenskonvention:**
|
||||||
|
Prefix aus ENV (`COLLECTION_PREFIX`, z. B. `mindnet`):
|
||||||
|
|
||||||
|
- `mindnet_notes`
|
||||||
|
- `mindnet_chunks`
|
||||||
|
- `mindnet_edges`
|
||||||
|
|
||||||
|
Die Collections werden über:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.reset_qdrant --mode wipe --prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
|
||||||
|
**vollständig neu angelegt**, inkl. Payload-Indizes (Details in 6.1).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2.3 Notes – Payload & Felder
|
||||||
|
|
||||||
|
Minimale, indexierte Felder (Auszug, Schema laut `diag_payload_indexes`):
|
||||||
|
|
||||||
|
- `note_id` (keyword) – stabile Note-ID (Frontmatter `id`)
|
||||||
|
- `title` (text) – Notiz-Titel
|
||||||
|
- `type` (keyword) – logischer Typ (z. B. `concept`, `project`, `profile`, `journal`, `source`)
|
||||||
|
- `tags` (keyword[]) – Tags aus Frontmatter
|
||||||
|
- `updated` (integer) – Zeitstempel (z. B. `YYYYMMDDHHMMSS` oder epoch; Implementierungsdetail)
|
||||||
|
|
||||||
|
Weitere Felder (nicht alle indexiert, aber im Payload vorhanden):
|
||||||
|
|
||||||
|
- `retriever_weight` (float) – wie stark diese Note/Chunks im Retrieval gewichtet werden
|
||||||
|
- `chunk_profile` (string) – Name des verwendeten Chunk-Profils
|
||||||
|
- `edge_defaults` (string[]) – Liste von Default-Relationen pro Typ (z. B. `["references","related_to"]`)
|
||||||
|
- `confidentiality`, `visibility`, weitere Metadaten aus Frontmatter
|
||||||
|
|
||||||
|
Diese Felder werden in `app/core/note_payload.py` aus Frontmatter + `types.yaml` konstruiert.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2.4 Chunks – Payload & Felder
|
||||||
|
|
||||||
|
Indexierte Kernfelder:
|
||||||
|
|
||||||
|
- `note_id` (keyword) – Referenz auf die übergeordnete Note
|
||||||
|
- `chunk_id` (keyword) – deterministische Chunk-ID:
|
||||||
|
`"{note_id}#c{index:02d}"` (z. B. `20251110-ollama-llm-9f0a12#c02`)
|
||||||
|
- `index` (integer) – 0-basiert; Position in der Note
|
||||||
|
- `type` (keyword) – Note-Typ (vom Note geerbt)
|
||||||
|
- `tags` (keyword[]) – Tags (vom Note geerbt)
|
||||||
|
|
||||||
|
Weitere wichtige Payload-Felder:
|
||||||
|
|
||||||
|
- `ord` (integer) – 1-basiertes Pendant zu `index` (für menschliche Lesbarkeit)
|
||||||
|
- `text` (string) – eigentlicher Chunk-Text (ggf. gereinigt/normalisiert)
|
||||||
|
- `window` (string) – Kontext-Fenster (z. B. Chunk ± Nachbarn)
|
||||||
|
- `retriever_weight` (float) – gültiger Wert aus `types.yaml` (oder Default)
|
||||||
|
- `chunk_profile` (string) – verwendetes Profil, z. B. `short`, `medium`, `long`, `by_heading`
|
||||||
|
- `neighbors_prev` (string[]) – Liste von `chunk_id`s direkter Vorgänger (i. d. R. 0 oder 1 ID)
|
||||||
|
- `neighbors_next` (string[]) – Liste von `chunk_id`s direkter Nachfolger
|
||||||
|
|
||||||
|
Die Konstruktion erfolgt in `app/core/chunk_payload.py` auf Basis der Chunks aus `chunker.py` und der aufgelösten Typ-Defaults.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2.5 Edges – Payload & Felder
|
||||||
|
|
||||||
|
Indexierte Felder:
|
||||||
|
|
||||||
|
- `note_id` (keyword) – Note, zu der der **Quell-Chunk** gehört
|
||||||
|
- `source_id` (keyword) – i. d. R. `chunk_id` des Quell-Chunks (z. B. `2025…#c00`)
|
||||||
|
- `target_id` (keyword) – Zielobjekt (z. B.:
|
||||||
|
- `chunk_id` eines anderen Chunks
|
||||||
|
- Note-Titel (`"Embeddings 101"`) bei **konzeptionellen Referenzen**
|
||||||
|
- `kind` (keyword) – Kantenart:
|
||||||
|
- Struktur: `belongs_to`, `next`, `prev`
|
||||||
|
- Explizit: `references`
|
||||||
|
- Inline: z. B. `similar_to`, `depends_on`, `related_to`
|
||||||
|
- Typ-Defaults: ebenfalls z. B. `depends_on`, `related_to` (aus `edge_defaults`)
|
||||||
|
- `scope` (keyword) – aktuell immer `"chunk"` (Kante wird auf Chunk-Ebene geführt)
|
||||||
|
- `chunk_id` (keyword) – `chunk_id` des Quell-Chunks
|
||||||
|
|
||||||
|
Weitere Payload-Felder:
|
||||||
|
|
||||||
|
- `rule_id` (string) – Herkunftsregel:
|
||||||
|
- Struktur:
|
||||||
|
- `structure:belongs_to`
|
||||||
|
- `structure:next`
|
||||||
|
- `structure:prev`
|
||||||
|
- Explizite Wiki-Links:
|
||||||
|
- `explicit:wikilink`
|
||||||
|
- Inline-Relationen:
|
||||||
|
- `inline:rel`
|
||||||
|
- Typ-Defaults aus `types.yaml`:
|
||||||
|
- `edge_defaults:<type>:<relation>`
|
||||||
|
z. B. `edge_defaults:concept:related_to`
|
||||||
|
- `confidence` (float) – Stärke / Vertrauensgrad:
|
||||||
|
- Struktur-Kanten: `1.0`
|
||||||
|
- Explizite Wiki-Links: `1.0`
|
||||||
|
- Inline-Relationen (`[[rel:… …]]`): z. B. `0.95`
|
||||||
|
- Typ-Defaults: z. B. `0.7`
|
||||||
|
|
||||||
|
Edges werden in `app/core/derive_edges.py` erzeugt und in `mindnet_edges` gespeichert.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Typ-Konfiguration (`config/types.yaml`)
|
||||||
|
|
||||||
|
### 3.1 Struktur des Files
|
||||||
|
|
||||||
|
`types.yaml` (vereinfachtes Beispiel):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
version: 1.0
|
||||||
|
default:
|
||||||
|
retriever_weight: 1.0
|
||||||
|
chunk_profile: default
|
||||||
|
|
||||||
|
types:
|
||||||
|
concept:
|
||||||
|
chunk_profile: medium
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
retriever_weight: 0.35
|
||||||
|
|
||||||
|
task:
|
||||||
|
chunk_profile: short
|
||||||
|
edge_defaults: ["depends_on", "belongs_to"]
|
||||||
|
retriever_weight: 0.8
|
||||||
|
|
||||||
|
experience:
|
||||||
|
chunk_profile: medium
|
||||||
|
edge_defaults: ["derived_from", "inspired_by"]
|
||||||
|
retriever_weight: 0.9
|
||||||
|
|
||||||
|
project:
|
||||||
|
chunk_profile: long
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
retriever_weight: 0.97
|
||||||
|
|
||||||
|
profile:
|
||||||
|
chunk_profile: long
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
retriever_weight: 0.66
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Typauflösung – Reihenfolge der Prioritäten
|
||||||
|
|
||||||
|
Für eine Note mit `type` und optionalen Frontmatter Feldern `retriever_weight`, `chunk_profile` gilt aktuell:
|
||||||
|
|
||||||
|
1. **Typ-spezifische Defaults** aus `types.yaml`:
|
||||||
|
- `retriever_weight` ← `types.<type>.retriever_weight` (falls vorhanden)
|
||||||
|
- `chunk_profile` ← `types.<type>.chunk_profile` (falls vorhanden)
|
||||||
|
2. **Globaler Default** `default` in `types.yaml`, falls Typ-Eintrag fehlt.
|
||||||
|
3. **Frontmatter `retriever_weight` / `chunk_profile`** werden aktuell *nicht* mehr als Konfigurationsebene verwendet, sondern eher als historische Relikte. (Dies kann zukünftig wieder aktiviert werden, ist aber im aktuellen Stand bewusst abgeschaltet zugunsten klarer, zentraler Konfiguration über `types.yaml`.)
|
||||||
|
|
||||||
|
Das Ergebnis dieser Auflösung wird im Payload als:
|
||||||
|
|
||||||
|
- `note_payload["retriever_weight"]`
|
||||||
|
- `note_payload["chunk_profile"]`
|
||||||
|
|
||||||
|
gesetzt und **1:1 an alle Chunks vererbt**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Import-Pipeline (Markdown → Qdrant)
|
||||||
|
|
||||||
|
### 4.1 Runbook
|
||||||
|
|
||||||
|
Standard-Import (mit Purge-Before-Upsert):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export COLLECTION_PREFIX=mindnet
|
||||||
|
export MINDNET_TYPES_FILE="$(pwd)/config/types.yaml"
|
||||||
|
|
||||||
|
python3 -m scripts.import_markdown \
|
||||||
|
--vault ./mindnet_v2_test_vault \
|
||||||
|
--apply \
|
||||||
|
--purge-before-upsert \
|
||||||
|
--prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
|
||||||
|
Nur Sync-Deletes (entfernte Dateien in Qdrant aufräumen):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown \
|
||||||
|
--vault ./mindnet_v2_test_vault \
|
||||||
|
--sync-deletes \
|
||||||
|
--apply \
|
||||||
|
--prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4.2 Verarbeitungsschritte
|
||||||
|
|
||||||
|
**1. Discovery & Parsing**
|
||||||
|
|
||||||
|
- `import_markdown.py` traversiert das Vault‐Verzeichnis.
|
||||||
|
- Für jede `.md`:
|
||||||
|
- YAML-Frontmatter extrahieren (id, title, type, created, updated, tags, …).
|
||||||
|
- Body (Markdown) als String.
|
||||||
|
|
||||||
|
**2. Typauflösung & Payload-Bau (Note)**
|
||||||
|
|
||||||
|
- Laden & Auflösen von `types.yaml` (über ENV `MINDNET_TYPES_FILE` oder Default `./config/types.yaml`).
|
||||||
|
- Bestimmung des effektiven Typs (`type` aus Frontmatter; ggf. Fallback).
|
||||||
|
- Anwendung der Typdefaults:
|
||||||
|
- `retriever_weight` und `chunk_profile` werden aus `types.yaml` gelesen.
|
||||||
|
- Konstruktion des Note-Payloads via `make_note_payload` (in `note_payload.py`):
|
||||||
|
- Felder wie `note_id`, `title`, `type`, `tags`, `updated`, `retriever_weight`, `chunk_profile`, `edge_defaults`, …
|
||||||
|
|
||||||
|
**3. Chunking**
|
||||||
|
|
||||||
|
- Übergabe des Note-Bodies + `chunk_profile` an `chunker.py`.
|
||||||
|
- `chunker.py` nutzt `chunk_config.py`, das definierte Profile bereitstellt, z. B.:
|
||||||
|
|
||||||
|
```python
|
||||||
|
CHUNK_PROFILES = {
|
||||||
|
"short": {"max_tokens": 128, "overlap": 32},
|
||||||
|
"medium": {"max_tokens": 256, "overlap": 64},
|
||||||
|
"long": {"max_tokens": 512, "overlap": 128},
|
||||||
|
"by_heading": {"strategy": "by_heading", "max_chars": 2000},
|
||||||
|
"default": {"max_tokens": 256, "overlap": 64},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- Ergebnis: Liste geordneter Chunks mit:
|
||||||
|
- `chunk_id` (`note_id#cXX`),
|
||||||
|
- `index` (0..N-1),
|
||||||
|
- `ord` (1..N),
|
||||||
|
- `text`,
|
||||||
|
- `window` (inkl. Nachbartexte),
|
||||||
|
- `neighbors_prev` / `neighbors_next`.
|
||||||
|
|
||||||
|
**4. Chunk-Payloads**
|
||||||
|
|
||||||
|
- In `chunk_payload.py` werden für jeden Chunk die Payloads gebaut:
|
||||||
|
- Vererbung von `note_id`, `type`, `tags`, `retriever_weight`, `chunk_profile`.
|
||||||
|
- Setzen von `index`, `ord`, `neighbors_prev`, `neighbors_next`.
|
||||||
|
- Persistieren von `text` und `window`.
|
||||||
|
|
||||||
|
**5. Kantenableitung (Edges)**
|
||||||
|
|
||||||
|
- `derive_edges.py` wird mit Note + Chunks + Typinformationen aufgerufen.
|
||||||
|
- Es entstehen mehrere Gruppen von Kanten:
|
||||||
|
1. **Struktur-Kanten**:
|
||||||
|
- `belongs_to`: Chunk → Note
|
||||||
|
- `next`: Chunk[i] → Chunk[i+1]
|
||||||
|
- `prev`: Chunk[i] → Chunk[i-1]
|
||||||
|
2. **Explizite Kanten aus Text**:
|
||||||
|
- Wiki-Links: `[[Target Title]]` → `kind="references"`, `rule_id="explicit:wikilink"`, `confidence=1.0`.
|
||||||
|
- Inline-Relationen: `[[rel:depends_on Embeddings 101]]`:
|
||||||
|
- `kind="depends_on"`
|
||||||
|
- `rule_id="inline:rel"`
|
||||||
|
- `confidence≈0.95`
|
||||||
|
- `target_id="Embeddings 101"` (Titel, nicht ID – Auflösung erfolgt später im Agenten).
|
||||||
|
3. **Typ-basierte Defaults (`edge_defaults`)**:
|
||||||
|
- Aus `types.yaml` je `type`: Liste von Relations-Kinds, z. B.:
|
||||||
|
- `concept.edge_defaults = ["references","related_to"]`
|
||||||
|
- `project.edge_defaults = ["references","depends_on"]`
|
||||||
|
- Für jede explizite Referenz Quelle→Ziel (z. B. `references`, `similar_to`) werden **zusätzliche Kanten** erzeugt, z. B.:
|
||||||
|
- `edge_defaults:project:depends_on`
|
||||||
|
- `edge_defaults:concept:related_to`
|
||||||
|
- `confidence≈0.7`.
|
||||||
|
4. **Deduplication**:
|
||||||
|
- Alle Kanten werden in einem Set anhand eines Keys wie
|
||||||
|
- `(kind, scope, source_id, target_id, rule_id)`
|
||||||
|
dedupliziert (idempotent).
|
||||||
|
|
||||||
|
**6. Upsert nach Qdrant**
|
||||||
|
|
||||||
|
- Punkte (Notes, Chunks, Edges) werden via `qdrant_points.py` in die entsprechenden Collections upserted.
|
||||||
|
- Option `--purge-before-upsert` sorgt dafür, dass alte Punkte mit denselben IDs entfernt werden, bevor neue geschrieben werden (stabiler Reimport).
|
||||||
|
- `--sync-deletes` entfernt Punkte, deren zugehörige Dateien im Vault nicht mehr existieren.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Edges im Detail – Semantik & Regeln
|
||||||
|
|
||||||
|
### 5.1 Struktur-Kanten
|
||||||
|
|
||||||
|
Erzeugt für **jede Note** mit >0 Chunks:
|
||||||
|
|
||||||
|
- **belongs_to**
|
||||||
|
- pro Chunk genau eine Kante:
|
||||||
|
- `source_id = chunk_id`
|
||||||
|
- `target_id = note_id` *oder* (konzeptionell) ein Note-Identifikator
|
||||||
|
- `kind = "belongs_to"`
|
||||||
|
- `scope = "chunk"`
|
||||||
|
- `rule_id = "structure:belongs_to"`
|
||||||
|
- `confidence = 1.0`
|
||||||
|
- **next / prev**
|
||||||
|
- für Index i (`0..N-2`):
|
||||||
|
- `next`: `chunk_i → chunk_{i+1}`
|
||||||
|
- `prev`: `chunk_{i+1} → chunk_i`
|
||||||
|
- `kind = "next"` bzw. `"prev"`
|
||||||
|
- `scope = "chunk"`
|
||||||
|
- `rule_id = "structure:next"` / `"structure:prev"`
|
||||||
|
- `confidence = 1.0`
|
||||||
|
|
||||||
|
Tests (`tests.test_edges_smoke` / `tests.test_edges_all`) stellen sicher:
|
||||||
|
|
||||||
|
- `belongs_to` == `#chunks`
|
||||||
|
- `next` == `prev` == `#chunks - 1`
|
||||||
|
- keine Duplikate der Strukturkanten.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5.2 Explizite Wiki-Links (`[[Target Title]]`)
|
||||||
|
|
||||||
|
- Parser in `derive_edges.py` sucht im Chunk-Text nach Mustern:
|
||||||
|
- `[[Some Page]]` (kein `rel:` Präfix)
|
||||||
|
- Erzeugt Kanten:
|
||||||
|
- `kind = "references"`
|
||||||
|
- `scope = "chunk"`
|
||||||
|
- `source_id = chunk_id`
|
||||||
|
- `target_id = "<Titel aus dem Link>"` (z. B. `"Vector DB Basics"`)
|
||||||
|
- `rule_id = "explicit:wikilink"`
|
||||||
|
- `confidence = 1.0`
|
||||||
|
|
||||||
|
Diese Kanten bilden das **explizite Referenznetz** des Nutzers ab (Obsidian-typische Wikilinks).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5.3 Inline-Relationen (`[[rel:… …]]`)
|
||||||
|
|
||||||
|
Aktuell **implementiert und funktional** ist folgende Form:
|
||||||
|
|
||||||
|
- **Format (aktuell unterstützt):**
|
||||||
|
|
||||||
|
```md
|
||||||
|
[[rel:depends_on Embeddings 101]]
|
||||||
|
[[rel:similar_to Vector DB Basics]]
|
||||||
|
[[rel:related_to Qdrant Vektordatenbank]]
|
||||||
|
```
|
||||||
|
|
||||||
|
- Parsing-Strategie:
|
||||||
|
- Link-Inhalt wird gesplittet:
|
||||||
|
- Erster Token: `rel:depends_on`
|
||||||
|
- Rest: Zielbezeichnung (`Embeddings 101`)
|
||||||
|
- `kind` wird aus dem Teil `depends_on` extrahiert (z. B. `similar_to`, `related_to`).
|
||||||
|
- `target_id` wird aus dem Rest (meist ein Titel) abgeleitet.
|
||||||
|
|
||||||
|
- Ergebnis-Kante:
|
||||||
|
- `kind = "<Relation aus rel:…>"`
|
||||||
|
- `scope = "chunk"`
|
||||||
|
- `source_id = chunk_id`
|
||||||
|
- `target_id = "<Ziel-Titel>"`
|
||||||
|
- `rule_id = "inline:rel"`
|
||||||
|
- `confidence ≈ 0.95`
|
||||||
|
|
||||||
|
- **Nicht unterstützt (Stand jetzt):**
|
||||||
|
|
||||||
|
```md
|
||||||
|
rel: depends_on [[Embeddings 101]] [[Vector DB Basics]]
|
||||||
|
```
|
||||||
|
|
||||||
|
Diese Syntax wird derzeit **nicht** erkannt.
|
||||||
|
Multi-Target-Inline-Relationen (`rel: similar_to [[A]] [[B]]`) sind ein **offener Task**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5.4 Typ-basierte Default-Edges (`edge_defaults`)
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
|
||||||
|
- Für bestimmte Typen (vor allem `concept`, `project`, `profile`) sollen zusätzliche Kanten automatisch ergänzt werden, um das **Semantiknetz zu verdichten**, ohne dass der Nutzer jede Relation explizit pflegen muss.
|
||||||
|
|
||||||
|
Konfiguration:
|
||||||
|
|
||||||
|
- In `types.yaml` definiert, z. B.:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
types:
|
||||||
|
concept:
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
project:
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Aktuelle Implementierung:
|
||||||
|
|
||||||
|
- Für jede **explizite** Referenz (Wiki-Link oder Inline-Relation) von Chunk → Ziel:
|
||||||
|
- `kind="references"` (Wiki) oder die Inline-Relation (`depends_on`, `similar_to`, …)
|
||||||
|
- Auf Basis des Note-Typs (`type`) wird pro Default-Relation eine zusätzliche Kante erzeugt:
|
||||||
|
|
||||||
|
Beispiel:
|
||||||
|
|
||||||
|
- Note-Typ: `project`
|
||||||
|
- `edge_defaults: ["references","depends_on"]`
|
||||||
|
- Explizit: `references` von `Edge Index` → `Embeddings 101`
|
||||||
|
- Abgeleitete Kanten:
|
||||||
|
- `kind="references"`, `rule_id="edge_defaults:project:references"`, `confidence=0.7`
|
||||||
|
- `kind="depends_on"`, `rule_id="edge_defaults:project:depends_on"`, `confidence=0.7`
|
||||||
|
|
||||||
|
- Symmetrische Kanten:
|
||||||
|
- Für einige Relations-Typen (z. B. `related_to`) wird zusätzlich eine **spiegelbildliche** Kante Ziel → Quelle erzeugt.
|
||||||
|
- Dies erklärt in Tests z. B. Paare wie:
|
||||||
|
- `related_to (Source=Ollama LLM, Target=Qdrant Vektordatenbank)`
|
||||||
|
- `related_to (Source=Qdrant Vektordatenbank, Target=Ollama LLM)`
|
||||||
|
|
||||||
|
- Deduplication:
|
||||||
|
- Es wird verhindert, dass mehrfach identische Kanten geschrieben werden.
|
||||||
|
- Schlüssel: `(kind, scope, source_id, target_id, rule_id)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5.5 `rule_id` & `confidence` – Bedeutung
|
||||||
|
|
||||||
|
**`rule_id`** kodiert **Herkunft** und **Regel**, z. B.:
|
||||||
|
|
||||||
|
- `structure:belongs_to`, `structure:next`, `structure:prev`
|
||||||
|
- `explicit:wikilink`
|
||||||
|
- `inline:rel`
|
||||||
|
- `edge_defaults:<type>:<relation>`
|
||||||
|
|
||||||
|
**`confidence`** dient als späterer Gewichtungsfaktor im Retrieval / Ranking:
|
||||||
|
|
||||||
|
- `1.0` → harte Struktur- oder explizite Kante (vom Nutzer direkt gesetzt)
|
||||||
|
- `0.95` → Inline-Relation (stark, aber nicht so „hart“ wie die reine Referenz)
|
||||||
|
- `0.7` → Typ-Default-Edges (abgeleitete Hypothesen aus Typwissen)
|
||||||
|
|
||||||
|
Ein künftiger LLM-Agent kann daraus z. B.:
|
||||||
|
|
||||||
|
- Kanten mit hoher `confidence` priorisieren,
|
||||||
|
- Default-Edges als schwächere Hinweise betrachten,
|
||||||
|
- Inline-Relationen gezielt als semantische Aussagen über die Beziehung interpretieren.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Qdrant-Schema & Indizes
|
||||||
|
|
||||||
|
### 6.1 Erstellung & Indizes
|
||||||
|
|
||||||
|
Die Erstellung erfolgt über:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.reset_qdrant --mode wipe --prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
|
||||||
|
Intern:
|
||||||
|
|
||||||
|
- `qdrant.py` / `setup_mindnet_collections.py`:
|
||||||
|
- Collections anlegen, ggf. mit Vektor-Konfiguration (Single-Vector; Named-Vectors sind vorbereitet aber optional).
|
||||||
|
- `ensure_payload_indexes()`:
|
||||||
|
- Legt für jede Collection die relevanten Payload-Indizes an.
|
||||||
|
|
||||||
|
Verifizierbar durch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m tests.ensure_indexes_and_show
|
||||||
|
```
|
||||||
|
|
||||||
|
Beispielauszug (bereits real so gesehen):
|
||||||
|
|
||||||
|
- **mindnet_notes**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"tags": { "data_type": "keyword", "points": 8 },
|
||||||
|
"note_id":{ "data_type": "keyword", "points": 8 },
|
||||||
|
"title": { "data_type": "text", "points": 8 },
|
||||||
|
"updated":{ "data_type": "integer", "points": 0 },
|
||||||
|
"type": { "data_type": "keyword", "points": 8 }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- **mindnet_chunks**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"note_id": { "data_type": "keyword", "points": 44 },
|
||||||
|
"chunk_id":{ "data_type": "keyword", "points": 44 },
|
||||||
|
"index": { "data_type": "integer", "points": 44 },
|
||||||
|
"type": { "data_type": "keyword", "points": 44 },
|
||||||
|
"tags": { "data_type": "keyword", "points": 44 }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- **mindnet_edges**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"scope": { "data_type": "keyword", "points": 135 },
|
||||||
|
"kind": { "data_type": "keyword", "points": 135 },
|
||||||
|
"chunk_id":{ "data_type": "keyword", "points": 135 },
|
||||||
|
"target_id":{ "data_type": "keyword", "points": 135 },
|
||||||
|
"note_id": { "data_type": "keyword", "points": 135 },
|
||||||
|
"source_id":{ "data_type": "keyword", "points": 135 }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 Schema-Validierung
|
||||||
|
|
||||||
|
`tests/assert_payload_schema.py` prüft:
|
||||||
|
|
||||||
|
- ob die Minimalfelder (`note_id`, `type`, `title`, `updated`, `tags` etc.) vorhanden sind,
|
||||||
|
- berichtet fehlende Felder.
|
||||||
|
|
||||||
|
Nach Korrektur der Indexanlage sollte:
|
||||||
|
|
||||||
|
- `ok = true` sein, bzw. zumindest alle Kernfelder unter `present` geführt werden.
|
||||||
|
(Im aktuellen Stand werden sie als Indizes geführt; Qdrant speichert zusätzliche Felder ohne explizite Schema-Zeile, was technisch unproblematisch ist.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Nutzung im Retrieval (Ausblick)
|
||||||
|
|
||||||
|
### 7.1 Typbewusstes Chunk-Retrieval
|
||||||
|
|
||||||
|
Ein typischer Query-Pfad eines LLM-Agents:
|
||||||
|
|
||||||
|
1. Query → Embedding
|
||||||
|
2. Suche in `mindnet_chunks` mit Filter, z. B.:
|
||||||
|
|
||||||
|
```python
|
||||||
|
f = rest.Filter(
|
||||||
|
must=[
|
||||||
|
rest.FieldCondition(key="type", match=rest.MatchValue(value="concept")),
|
||||||
|
rest.FieldCondition(key="tags", match=rest.MatchAny(any=["ki","ollama"]))
|
||||||
|
]
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Treffer werden nach:
|
||||||
|
- Vektor-Score,
|
||||||
|
- `retriever_weight`,
|
||||||
|
- ggf. Edges (z. B. Projekte mit `depends_on` → Konzepte) gewichtet.
|
||||||
|
|
||||||
|
### 7.2 Graph-orientierte Navigation
|
||||||
|
|
||||||
|
Basierend auf `mindnet_edges` kann der Agent:
|
||||||
|
|
||||||
|
- ausgehend von einem relevanten Chunk:
|
||||||
|
- Struktur-Kanten (`next`, `prev`) zur Kontextvergrößerung nutzen,
|
||||||
|
- `references` / `depends_on` / `related_to` interpretieren,
|
||||||
|
- `edge_defaults` als semantische Ausdehnung (z. B. von `project` zu seinen `concept`-Bausteinen) nutzen.
|
||||||
|
|
||||||
|
Die Bedeutung der Kanten ist dabei durch `rule_id` und `confidence` klar nachvollziehbar.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Offene Punkte / Known Limitations
|
||||||
|
|
||||||
|
1. **Multi-Target Inline-Relationen**
|
||||||
|
Syntax wie:
|
||||||
|
|
||||||
|
```md
|
||||||
|
[[rel:similar_to Vector DB Basics]] [[rel:similar_to Embeddings 101]]
|
||||||
|
```
|
||||||
|
funktioniert (zwei separate Inline-Links).
|
||||||
|
**Nicht** unterstützt (Stand jetzt):
|
||||||
|
|
||||||
|
```md
|
||||||
|
[[rel:similar_to Vector DB Basics]] [[Embeddings 101]]
|
||||||
|
rel: similar_to [[Vector DB Basics]] [[Embeddings 101]]
|
||||||
|
```
|
||||||
|
|
||||||
|
→ Offener Task: Parser erweitern, um mehrere Ziele in einer Inline-Relation zu unterstützen.
|
||||||
|
|
||||||
|
2. **Callout-Edges (`[!edge]`)**
|
||||||
|
- Konzeptionell vorgesehen, z. B.:
|
||||||
|
|
||||||
|
```md
|
||||||
|
> [!edge] related_to: [[Vector DB Basics]]
|
||||||
|
```
|
||||||
|
|
||||||
|
- Im aktuellen Implementierungsstand werden Callouts entweder:
|
||||||
|
- gar nicht, oder
|
||||||
|
- nur teilweise verarbeitet (in den aktuellen Test-Runs war `callout_total = 0`).
|
||||||
|
→ Offener Task: Callout-Parser robust implementieren und mit Tests absichern.
|
||||||
|
|
||||||
|
3. **Frontmatter-Overrides für `retriever_weight` / `chunk_profile`**
|
||||||
|
- Aktuell **nicht** wirksam.
|
||||||
|
- Typdefaults aus `types.yaml` sind der **Single Source of Truth**.
|
||||||
|
- Später können kontrollierte Overrides (z. B. für Sonderfälle) eingeführt werden.
|
||||||
|
|
||||||
|
4. **Vektor-Konfiguration für Edges**
|
||||||
|
- `mindnet_edges` wird derzeit ohne Vektoren betrieben (`vectors = null`).
|
||||||
|
- Später könnte z. B. eine semantische Repräsentation von Kanten eingeführt werden (z. B. Embedding von `(source_chunk_text + relation + target_title)`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Zusammenfassung
|
||||||
|
|
||||||
|
- Die Mindnet-V2-Architektur implementiert einen **dreistufigen Speicher**:
|
||||||
|
- Notes (`mindnet_notes`),
|
||||||
|
- Chunks (`mindnet_chunks`),
|
||||||
|
- Edges (`mindnet_edges`).
|
||||||
|
|
||||||
|
- **`types.yaml`** steuert:
|
||||||
|
- `retriever_weight` und `chunk_profile` je Typ (inkl. Default),
|
||||||
|
- `edge_defaults` je Typ (automatische Relations-Ableitung).
|
||||||
|
|
||||||
|
- Das **Chunking** ist profilbasiert (short/medium/long/by_heading) und deterministisch.
|
||||||
|
|
||||||
|
- Die **Kantenableitung** in `derive_edges.py` erzeugt:
|
||||||
|
- strukturelle Kanten (belongs_to / next / prev),
|
||||||
|
- explizite Kanten aus Wiki-Links (`[[Target]]`),
|
||||||
|
- semantische Inline-Relationen (`[[rel:depends_on Target]]`),
|
||||||
|
- typbasierte Default-Kanten (z. B. `edge_defaults:project:depends_on`).
|
||||||
|
|
||||||
|
- `rule_id` und `confidence` machen die Herkunft jeder Kante transparent und ermöglichen später eine **gewichtete Auswertung** durch LLM-basierte Agenten.
|
||||||
|
|
||||||
|
- Qdrant-Collections und Payload-Indizes sind konsistent eingerichtet und durch Tests verifiziert.
|
||||||
|
|
||||||
|
Dieses Dokument bildet damit das aktuelle technische Gesamtbild von Mindnet V2 ab, inkl. aller relevanten Mechanismen rund um Typen, Chunking, Edges und Qdrant-Schema.
|
||||||
|
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
# mindnet v2 – Implementierungs-Playbook & Handover (Side‑by‑Side zu v1)
|
||||||
|
*Stand:* 2025-11-10 08:03
|
||||||
|
*Autor:* ChatGPT (Projekt‑Handover)
|
||||||
|
*Zweck:* Vollständige, ausführbare Anleitung zur Weiterentwicklung von **mindnet** auf eine versionierte, testbare v2‑Architektur (neben v1), inklusive Prompt für den Folgethread, Akzeptanzkriterien, Dateiliste, Test‑ und Rollback‑Pläne.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0) Kontext & Zielbild (kurz)
|
||||||
|
- **Mission:** Persönliches Wissensnetz, das langfristig die eigene Persönlichkeit, Erfahrungen und Entscheidungslogik abbildet; später nachvollziehbare Erklärungen für Familie/Nachwelt.
|
||||||
|
- **Heutiger Stand (v1):** `mindnet_notes` in Qdrant vorhanden; **Chunks** und **Edges** bisher nur teilweise/nicht konsistent; Edge‑Defaults („1 Dot“) vorhanden, aber Ziele teils nicht materialisiert. Importskripte: `import_markdown.py`, `edges.py` (historisch).
|
||||||
|
- **Ziel (v2):**
|
||||||
|
- **Collections:** `mindnet_notes_v2`, `mindnet_chunks_v2`, `mindnet_edges_v2` (saubere Payload‑Schemata + Indizes).
|
||||||
|
- **Chunker v2:** Block/Heading‑aware, Ziel 900–1200 Zeichen, Überlappung 120–180, deterministische `chunk_id`s.
|
||||||
|
- **Edge‑Builder v2:** Reihenfolge: **explicit → rule → default_resolved → default**; saubere `provenance`.
|
||||||
|
- **Importer‑Pipeline v2:** Parse → Chunk → Edge → Embeddings → Batch‑Upsert + Payload‑Indexpflege + Snapshots.
|
||||||
|
- **Policies:** Privacy/Recency/Type‑Gewichte als **Konfiguration** (YAML + JSON‑Schema), nicht hart im Code.
|
||||||
|
- **Observability:** FastAPI + OpenTelemetry (Tracing/Metrics); Import‑Reports (CSV/JSON).
|
||||||
|
- **Side‑by‑Side:** v2 **parallel** zu v1; kein Big‑Bang.
|
||||||
|
|
||||||
|
> **Warum so?** Qdrant‑Payload‑Indizes/Filter → schnelle, erklärbare Selektionen; Snapshots → betriebssicheres Rollback; OTel → Nachvollziehbarkeit; YAML/JSON‑Schema → Validierbarkeit & Stabilität.
|
||||||
|
|
||||||
|
### 0.1 Aktueller Implementierungsstand (2025-11-18)
|
||||||
|
|
||||||
|
- Es existiert eine produktive mindnet-Instanz mit den Collections:
|
||||||
|
- `${prefix}_notes`
|
||||||
|
- `${prefix}_chunks`
|
||||||
|
- `${prefix}_edges`
|
||||||
|
- Import-Pipeline:
|
||||||
|
- `scripts/import_markdown.py` erzeugt Notes, Chunks und Edges vollständig.
|
||||||
|
- `types.yaml` definiert per Note-Typ:
|
||||||
|
- `retriever_weight`
|
||||||
|
- `chunk_profile`
|
||||||
|
- `edge_defaults` (z. B. `depends_on`, `related_to`).
|
||||||
|
- Edges:
|
||||||
|
- Strukturkanten: `belongs_to`, `next`, `prev`
|
||||||
|
- Explizite Kanten aus Wikilinks: `kind="references"`, `rule_id="explicit:wikilink"`
|
||||||
|
- Typbasierte Default-Kanten aus `edge_defaults`: `rule_id="edge_defaults:<type>:<relation>"`
|
||||||
|
- Inline-Kanten im Format `[[rel:<relation> <Zielnote>]]` mit `rule_id="inline:rel"`
|
||||||
|
- alle Edges tragen eine `confidence` im Bereich `[0,1]` als Hinweis für spätere Ranking-Logiken.
|
||||||
|
- Tests:
|
||||||
|
- `tests/test_edges_smoke.py`, `tests/test_edges_all.py` und `scripts/edges_full_check.py` validieren Counts, Konsistenz (`belongs_to == chunks`, `next/prev`-Ketten, keine Duplikate) und Provenienz-Zählungen (`explicit`, `defaults`, `inline`, `structure`).
|
||||||
|
- Offene Punkte:
|
||||||
|
- Mehrere Ziele in einer Inline-Relation (`rel: similar_to [[A]] [[B]]`) sind noch nicht implementiert.
|
||||||
|
- Retrieval-Service nutzt die neuen Edges und Weights bisher nur teilweise; ein edge-bewusster Retriever ist als nächster Schritt geplant.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1) Datenmodelle (Schemata, v2)
|
||||||
|
Die tatsächlichen JSON‑Schemata (2020‑12) werden im Repo abgelegt und in CI validiert.
|
||||||
|
|
||||||
|
### 1.1 note.schema.json
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://example.org/mindnet/note.schema.json",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["id", "title", "type", "privacy", "created", "hash_body"],
|
||||||
|
"properties": {
|
||||||
|
"id": { "type": "string", "pattern": "^[a-zA-Z0-9_-]+$" },
|
||||||
|
"title": { "type": "string", "minLength": 1 },
|
||||||
|
"type": { "type": "string" },
|
||||||
|
"privacy": { "type": "string", "enum": ["public","internal","private"] },
|
||||||
|
"created": { "type": "string", "format": "date-time" },
|
||||||
|
"modified": { "type": "string", "format": "date-time" },
|
||||||
|
"tags": { "type": "array", "items": { "type": "string" } },
|
||||||
|
"lang": { "type": "string", "default": "de" },
|
||||||
|
"source_path": { "type": "string" },
|
||||||
|
"source_collection": { "type": "string" },
|
||||||
|
"hash_body": { "type": "string" },
|
||||||
|
"hash_frontmatter": { "type": "string" },
|
||||||
|
"token_count": { "type": "integer", "minimum": 0 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.2 chunk.schema.json
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://example.org/mindnet/chunk.schema.json",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["chunk_id","note_id","text","ord"],
|
||||||
|
"properties": {
|
||||||
|
"chunk_id": { "type": "string", "pattern": "^[a-zA-Z0-9_-]+#c\d{4}$" },
|
||||||
|
"note_id": { "type": "string" },
|
||||||
|
"text": { "type": "string" },
|
||||||
|
"ord": { "type": "integer", "minimum": 0 },
|
||||||
|
"span_char_start": { "type": "integer", "minimum": 0 },
|
||||||
|
"span_char_end": { "type": "integer", "minimum": 0 },
|
||||||
|
"heading_path": { "type": "array", "items": { "type": "string" } },
|
||||||
|
"section_title": { "type": "string" },
|
||||||
|
"tokens_start": { "type": "integer", "minimum": 0 },
|
||||||
|
"tokens_end": { "type": "integer", "minimum": 0 },
|
||||||
|
"embeddings_version": { "type": "string" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.3 edge.schema.json
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://example.org/mindnet/edge.schema.json",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["edge_id","src_note_id","relation","provenance"],
|
||||||
|
"properties": {
|
||||||
|
"edge_id": { "type": "string" },
|
||||||
|
"src_note_id": { "type": "string" },
|
||||||
|
"src_chunk_id": { "type": "string" },
|
||||||
|
"dst_note_id": { "type": "string" },
|
||||||
|
"dst_chunk_id": { "type": "string" },
|
||||||
|
"relation": { "type": "string" },
|
||||||
|
"evidence_spans":{ "type": "array", "items": {"type":"object","properties":{"chunk_id":{"type":"string"},"span":[{"type":"integer"},{"type":"integer"}]]} },
|
||||||
|
"provenance": { "type": "string", "enum": ["explicit","rule","default_resolved","default"] },
|
||||||
|
"rule_id": { "type": "string" },
|
||||||
|
"confidence": { "type": "number", "minimum": 0, "maximum": 1 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.4 default_edge.schema.json
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://example.org/mindnet/default_edge.schema.json",
|
||||||
|
"type": "object",
|
||||||
|
"required": ["src_note_id","relation","target_kind"],
|
||||||
|
"properties": {
|
||||||
|
"src_note_id": { "type": "string" },
|
||||||
|
"relation": { "type": "string" },
|
||||||
|
"target_kind": { "type": "string" },
|
||||||
|
"when": { "type": "string" },
|
||||||
|
"strength_hint":{ "type": "number", "minimum": 0, "maximum": 1 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2) Qdrant: Collections & Indizes (v2)
|
||||||
|
Anlegen **neben** v1:
|
||||||
|
- `mindnet_notes_v2` (Payload‑Index: `type, privacy, created, modified, tags, source_path`)
|
||||||
|
- `mindnet_chunks_v2` (Payload‑Index: `note_id, ord, heading_path`)
|
||||||
|
- `mindnet_edges_v2` (Payload‑Index: `src_note_id, dst_note_id, relation, provenance`)
|
||||||
|
|
||||||
|
> Snapshots für Backups/Restore einplanen (Runbook siehe unten).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3) Import‑Pipeline v2 – Module & Flags
|
||||||
|
- **Chunker v2** (`chunking/block_chunker.py`): `target_len=1000`, `overlap=150`, `respect_headings=True`, `min_chunk_len=600`, `max_chunk_len=1400`, `chunk_id = f\"{note_id}#c{ordinal:04d}\"`
|
||||||
|
- **Edge‑Builder v2** (`graph/edge_builder_v2.py`): Reihenfolge **explicit → rule → default_resolved → default**, `provenance` korrekt setzen.
|
||||||
|
- **Importer** (`scripts/import_markdown.py` erweitern oder `scripts/import_markdown_v2.py`):
|
||||||
|
Flags: `--schema v2`, `--chunker v2`, `--edges v2`, `--dry-run`, `--apply`, `--prefix "$COLLECTION_PREFIX"`.
|
||||||
|
|
||||||
|
**Beispiel‑Run (dry‑run):**
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown_v2 --vault ./vault --schema v2 --chunker v2 --edges v2 --dry-run --prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Apply:**
|
||||||
|
```bash
|
||||||
|
python3 -m scripts.import_markdown_v2 --vault ./vault --schema v2 --chunker v2 --edges v2 --apply --prefix "$COLLECTION_PREFIX"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4) Policies (konfigurierbar, kein Hard‑Code)
|
||||||
|
- `policies/retrieval.schema.json` (JSON‑Schema) und `policies/retrieval.yaml`, z. B.:
|
||||||
|
```yaml
|
||||||
|
privacy_order: [private, internal, public]
|
||||||
|
recency_boost_half_life_days: 90
|
||||||
|
type_priority:
|
||||||
|
person: 1.2
|
||||||
|
event: 1.1
|
||||||
|
concept: 1.0
|
||||||
|
max_chunks_per_note: 8
|
||||||
|
edge_provenance_weights:
|
||||||
|
explicit: 1.0
|
||||||
|
rule: 0.8
|
||||||
|
default_resolved: 0.6
|
||||||
|
default: 0.2
|
||||||
|
rule_sets:
|
||||||
|
- id: "rules:types-v1"
|
||||||
|
enabled: true
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5) Observability & Reports
|
||||||
|
- **FastAPI + OpenTelemetry**: Traces um Phasen *parse/chunk/edge/upsert*; Metriken (Counter/Histogramme) pro Phase.
|
||||||
|
- **Import‑Report** je Lauf (CSV/JSON): counts für Notes/Chunks/Edges, Fehler, Dauer, Embedding‑Version.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6) Teststrategie
|
||||||
|
- **Gold‑Notizen** (3–5 repräsentative .md) mit erwarteten *Chunk‑Counts* und *Edge‑Sets*.
|
||||||
|
- **Regression:** fixer Soll‑Count (z. B. 171 ± 15 %) über Gesamt‑Vault; keine Duplikate `(src,relation,dst)`.
|
||||||
|
- **Dry‑Run vs Apply**: identische Counts, nur ohne Upserts.
|
||||||
|
- **Filter‑Smoke‑Tests:** Qdrant‑Filter nach `privacy`, `type`, `tags`, `created` funktionieren performant.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7) Akzeptanzkriterien (KPI)
|
||||||
|
1. **Chunk‑Qualität:** ≥ 90 % der Chunks enden an semantischen Grenzen; Ziel‑Länge 900–1200 Zeichen; deterministische IDs.
|
||||||
|
2. **Edge‑Kohärenz:** Keine Duplikate gleicher `(src,relation,dst)`; ≥ 95 % der expliziten Links materialisiert; `default` nur, wenn kein Ziel existiert.
|
||||||
|
3. **Filterbarkeit:** Queries nach `privacy/type/tags/created` performant (Payload‑Index vorhanden).
|
||||||
|
4. **Betriebsfestigkeit:** Telemetrie aktiv; fehlertoleranter Import; Qdrant‑Snapshot nach erfolgreichem Lauf.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8) Roadmap in kleinen, testbaren Schritten
|
||||||
|
|
||||||
|
### Step 0 – Safeguards
|
||||||
|
- Qdrant‑Snapshot v1 erstellen + Restore‑Probe.
|
||||||
|
- ENV: `MINDNET_SCHEMA_VERSION=2` (Importer wird v2 schreiben, v1 bleibt unberührt).
|
||||||
|
|
||||||
|
### Step 1 – Schemata (nur Dateien)
|
||||||
|
- Ablage: `/schemas/note.schema.json`, `/schemas/chunk.schema.json`, `/schemas/edge.schema.json`, `/schemas/default_edge.schema.json`
|
||||||
|
- CI‑Job: `make schema-validate` (jsonschema).
|
||||||
|
- **Abnahme:** Validator grün auf 3 Gold‑Notizen.
|
||||||
|
|
||||||
|
### Step 2 – Qdrant v2‑Collections (Optional)
|
||||||
|
- Anlegen der 3 v2‑Collections + Payload‑Indizes.
|
||||||
|
- **Abnahme:** Filter‑Query liefert erwartbare Ergebnisse.
|
||||||
|
|
||||||
|
### Step 2a – Edge-Pipeline v1 stabilisieren (ERLEDIGT)
|
||||||
|
- Ziel: Vollständige Generierung von Struktur-, expliziten, typbasierten und Inline-Kanten in den bestehenden Collections `${prefix}_notes`, `${prefix}_chunks`, `${prefix}_edges`.
|
||||||
|
- Status: Implementiert und durch `tests/test_edges_smoke.py`, `tests/test_edges_all.py` und `scripts/edges_full_check.py` verifiziert.
|
||||||
|
|
||||||
|
|
||||||
|
### Step 3 – Chunker v2
|
||||||
|
- Implementierung & Flag `--chunker v2`.
|
||||||
|
- **Abnahme:** Chunk‑Counts ~ alt (≈171 ± 15 %), semantische Schnitte.
|
||||||
|
|
||||||
|
### Step 4 – Edge‑Builder v2
|
||||||
|
- Reihenfolge & `provenance` strikt umsetzen, `default_resolved` integrieren.
|
||||||
|
- **Abnahme:** erwartete Relationensätze auf Gold‑Notizen, keine Duplikate.
|
||||||
|
|
||||||
|
### Step 4a – Edge-bewusster Retriever
|
||||||
|
- Implementiere einen Retrieval-Service, der:
|
||||||
|
- Chunks über Vektor-Suche aus `${prefix}_chunks` holt,
|
||||||
|
- dazu passende Nachbar-Chunks (`next/prev`) und typrelevante Notizen (`edge_defaults`, Inline-Relationen, Wikilinks) über `${prefix}_edges` ergänzt,
|
||||||
|
- `retriever_weight` und `confidence` beim Scoring berücksichtigt.
|
||||||
|
- Liefere einen API-Endpoint `/mindnet/query` mit einem einfachen JSON-Response (Chunks + Begründung).
|
||||||
|
- Akzeptanz:
|
||||||
|
- mindestens ein Testfall mit dem Relations-Showcase-Vault,
|
||||||
|
- sichtbarer Einfluss von `edge_defaults` und Inline-Relationen auf die Ergebnisliste.
|
||||||
|
|
||||||
|
|
||||||
|
### Step 5 – Importer‑Pipeline v2
|
||||||
|
- `--schema v2` Side‑by‑Side; Batch‑Upserts; vollständige Payloads; Recency‑Boost Konfig **nur** in Policy.
|
||||||
|
- **Abnahme:** Dry‑Run/Apply‑Parität, Reports, Telemetrie‑Ereignisse sichtbar.
|
||||||
|
|
||||||
|
### Step 6 – Observability
|
||||||
|
- OTel‑Instrumentierung + `/import/status` + Reports.
|
||||||
|
- **Abnahme:** Traces & Metriken vorhanden.
|
||||||
|
|
||||||
|
### Step 7 – Snapshots & Runbook
|
||||||
|
- Snapshot nach Erfolg; Restore‑Dokument.
|
||||||
|
- **Abnahme:** Restore‑Probe erfolgreich.
|
||||||
|
|
||||||
|
### Step 8 – Policies
|
||||||
|
- `policies/retrieval.yaml` + JSON‑Schema; A/B‑Test (Policy ändern → Retrieval ändert sich ohne Reimport).
|
||||||
|
- **Abnahme:** Sichtbarer Effekt laut Testfall.
|
||||||
|
|
||||||
|
### Step 9 – Umschalten
|
||||||
|
- Abnahme‑Doku, KPI‑Delta dokumentiert; Alias/Flag‑Switch auf v2.
|
||||||
|
- **Abnahme:** Funktionale Gleichwertigkeit + Qualitätsgewinn.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9) Dateien (neu/ändern)
|
||||||
|
**Neu:**
|
||||||
|
- `schemas/note.schema.json`
|
||||||
|
- `schemas/chunk.schema.json`
|
||||||
|
- `schemas/edge.schema.json`
|
||||||
|
- `schemas/default_edge.schema.json`
|
||||||
|
- `policies/retrieval.schema.json`
|
||||||
|
- `policies/retrieval.yaml`
|
||||||
|
- `chunking/block_chunker.py`
|
||||||
|
- `graph/edge_builder_v2.py`
|
||||||
|
- `docs/OPERATIONS.md` (Snapshots/Restore/Runbook)
|
||||||
|
- `tests/gold_notes/manifest.yaml` (+ 3–5 Notizen Kopien)
|
||||||
|
|
||||||
|
**Änderungen:**
|
||||||
|
- `scripts/import_markdown.py` **oder** `scripts/import_markdown_v2.py` neu (Flags `--schema/--chunker/--edges`).
|
||||||
|
- `edges.py` (falls weiterverwendet): auf **v2‑Edge‑Builder** migrieren oder auslaufen lassen.
|
||||||
|
- `types.yaml`: Regel‑Definitionen klarisieren (IDs, Bedingungen), **nicht** hart in Code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10) Beispiel‑Kommandos
|
||||||
|
```bash
|
||||||
|
# Step 2: Collections prüfen
|
||||||
|
curl -s http://127.0.0.1:6333/collections | jq
|
||||||
|
|
||||||
|
# Step 3: Chunker v2 (dry-run)
|
||||||
|
python3 -m scripts.import_markdown_v2 --vault ./vault --schema v2 --chunker v2 --edges v2 --dry-run --prefix "$COLLECTION_PREFIX"
|
||||||
|
|
||||||
|
# Step 5: Apply + Report
|
||||||
|
python3 -m scripts.import_markdown_v2 --vault ./vault --schema v2 --chunker v2 --edges v2 --apply --prefix "$COLLECTION_PREFIX" --report ./reports/import_$(date +%F_%H%M).json
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11) Rollback
|
||||||
|
- v2‑Collections droppen (wenn Side‑by‑Side).
|
||||||
|
- Aus Snapshot wiederherstellen (v1).
|
||||||
|
- Flags zurück auf v1.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12) Referenzen (offizielle Quellen)
|
||||||
|
- Qdrant – Collections, Payload‑Index, Filter: <https://qdrant.tech/documentation/>
|
||||||
|
- Qdrant – Snapshots/Backup: <https://qdrant.tech/documentation/guides/backup/>
|
||||||
|
- OpenTelemetry (Python/FastAPI): <https://opentelemetry.io/docs/instrumentation/python/>
|
||||||
|
- JSON‑Schema 2020‑12: <https://json-schema.org/>
|
||||||
|
- YAML 1.2: <https://yaml.org/spec/1.2.2/>
|
||||||
|
- Obsidian Frontmatter: <https://help.obsidian.md/Editing+and+formatting/Properties>
|
||||||
|
- Ollama CLI/API (lokale Modelle): <https://github.com/ollama/ollama>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13) **PROMPT für den neuen Chat (bitte exakt so einfügen)**
|
||||||
|
```
|
||||||
|
Rolle: Du bist mein Senior‑Entwickler & Architekt für mindnet. Arbeite strikt in kleinen, testbaren Schritten (Side‑by‑Side v2 neben v1). Liefere komplette Dateien als Downloads. Frage immer zuerst nach den **aktuellsten** Projektdateien, wenn du sie brauchst (z. B. import_markdown.py, edges.py, types.yaml), ändere keine Systemfunktion „workaround‑artig“ ohne die Gesamtwirkung zu prüfen.
|
||||||
|
|
||||||
|
Kontext (Kurzfassung):
|
||||||
|
- v1 hat nur mindnet_notes zuverlässig. Chunks/Edges sind inkonsistent/teilweise leer. Ziel: v2 mit drei Collections (notes/chunks/edges), neuem Chunker v2, Edge‑Builder v2 (explicit → rule → default_resolved → default), Policies (YAML), Observability (OTel), Snapshots. Kein Big‑Bang; v2 parallel zu v1.
|
||||||
|
- Siehe angehängte Datei **mindnet_v2_implementation_playbook.md** (Pfad/Download im Chat). Alles daraus ist verbindlich.
|
||||||
|
|
||||||
|
Deine ersten Aufgaben (Step‑by‑Step):
|
||||||
|
1) **Schemata anlegen (Step 1)**
|
||||||
|
- Erstelle die Dateien:
|
||||||
|
- schemas/note.schema.json
|
||||||
|
- schemas/chunk.schema.json
|
||||||
|
- schemas/edge.schema.json
|
||||||
|
- schemas/default_edge.schema.json
|
||||||
|
- Nutze die in der Playbook‑Datei vorgegebenen Strukturen als Basis.
|
||||||
|
- Liefere zusätzlich ein einfaches `Makefile`‑Ziel `schema-validate` (jsonschema via Python), plus `requirements.txt`.
|
||||||
|
- Output: komplette Dateien als Downloads + kurze Testanleitung.
|
||||||
|
|
||||||
|
2) **Qdrant v2‑Collections (Step 2)**
|
||||||
|
- Erzeuge die Collections + Payload‑Indizes.
|
||||||
|
- Liefere ein kleines Python‑Skript `tools/qdrant_bootstrap_v2.py`, das diese Anlage idempotent durchführt (inkl. Prüf‑Output).
|
||||||
|
|
||||||
|
3) **Chunker v2 (Step 3)**
|
||||||
|
- Implementiere `chunking/block_chunker.py` mit den im Playbook genannten Parametern.
|
||||||
|
- Teste gegen 3 Gold‑Notizen (liefere mini Test‑Harness in `tests/gold_notes/…`).
|
||||||
|
|
||||||
|
Arbeitsweise:
|
||||||
|
- Jede Aufgabe einzeln, mit klaren Akzeptanzkriterien (aus Playbook) und Download‑Artefakten.
|
||||||
|
- Keine parallelen Großumbauten.
|
||||||
|
- Immer deterministische IDs, idempotente Upserts.
|
||||||
|
|
||||||
|
Dateien, die ich dir anfänglich bereitstelle:
|
||||||
|
- **mindnet_v2_implementation_playbook.md** (diese Datei)
|
||||||
|
- **import_markdown.py** (aktuelle Version aus dem Projekt)
|
||||||
|
- **edges.py** (aktuelle Version aus dem Projekt)
|
||||||
|
- **types.yaml** (aktuelle Version)
|
||||||
|
|
||||||
|
Sage mir jeweils, welche Datei du als Nächstes brauchst. Beginne jetzt mit Aufgabe 1 (Schemata).
|
||||||
|
```
|
||||||
@@ -0,0 +1,260 @@
|
|||||||
|
# WP-04 / Step 4a – Retriever Scoring & Konfiguration
|
||||||
|
|
||||||
|
Dieses Dokument beschreibt den aktuellen Stand (2025-11-30) der Scoring-Logik des mindnet-Retrievers in WP-04 / Step 4a.
|
||||||
|
Es dient als Referenz für:
|
||||||
|
|
||||||
|
- Nachvollziehbarkeit des Rankings
|
||||||
|
- Konfiguration und Feintuning (ohne Codeänderungen)
|
||||||
|
- spätere Automatisierung / „Selbstjustierung“ des Retrievers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Überblick
|
||||||
|
|
||||||
|
Der Retriever kombiniert drei Signalquellen zu einem einheitlichen Score:
|
||||||
|
|
||||||
|
1. **Semantik** – Vektorsuche über `mindnet_chunks`
|
||||||
|
2. **Typ-Gewichtung** – `retriever_weight` pro Note/Chunk (aus `types.yaml`)
|
||||||
|
3. **Graph-Boni** – `edge_bonus` und `centrality_bonus` aus dem Subgraph (Edges)
|
||||||
|
|
||||||
|
Die Berechnung erfolgt in `app/core/retriever.py` und ist über `config/retriever.yaml` konfigurierbar.
|
||||||
|
Die API stellt die Funktionalität über den `/query`-Endpoint (FastAPI) bereit.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Datenbasis (Kurzüberblick)
|
||||||
|
|
||||||
|
### 2.1 Notes (`<prefix>_notes`)
|
||||||
|
|
||||||
|
Relevante Felder:
|
||||||
|
|
||||||
|
- note_id: keyword
|
||||||
|
- title: text
|
||||||
|
- type: keyword
|
||||||
|
- retriever_weight: float
|
||||||
|
- chunk_profile: keyword
|
||||||
|
- edge_defaults: keyword[]
|
||||||
|
- tags: keyword[]
|
||||||
|
- path: text
|
||||||
|
- fulltext: text
|
||||||
|
|
||||||
|
### 2.2 Chunks (`<prefix>_chunks`)
|
||||||
|
|
||||||
|
Relevante Felder:
|
||||||
|
|
||||||
|
- note_id: keyword
|
||||||
|
- chunk_id: keyword
|
||||||
|
- text: text
|
||||||
|
- window: text
|
||||||
|
- retriever_weight: float
|
||||||
|
- type: keyword
|
||||||
|
- path: text
|
||||||
|
- section: text
|
||||||
|
- neighbors_prev / neighbors_next
|
||||||
|
- chunk_profile: keyword
|
||||||
|
|
||||||
|
### 2.3 Edges (`<prefix>_edges`)
|
||||||
|
|
||||||
|
Relevante Felder:
|
||||||
|
|
||||||
|
- kind: keyword
|
||||||
|
- source_id: keyword
|
||||||
|
- target_id: keyword
|
||||||
|
- note_id: keyword
|
||||||
|
- confidence: float
|
||||||
|
- rule_id, provenance, edge_id, scope, relation, ref_text
|
||||||
|
|
||||||
|
Edges stammen stets aus der Import-/Edge-Pipeline (WP-03) und bilden die graphbasierte Grundlage des Retrievers.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Scoring-Formel (Retriever)
|
||||||
|
|
||||||
|
### 3.1 Eingangsgrößen
|
||||||
|
|
||||||
|
Für jeden Treffer gelten:
|
||||||
|
|
||||||
|
semantic_score – aus der Vektorsuche (Qdrant)
|
||||||
|
retriever_weight – typabhängiges Gewicht (aus types.yaml)
|
||||||
|
edge_bonus – aggregierter Graph-Bonus
|
||||||
|
centrality_bonus – Zentralitätsbonus aus dem Subgraph
|
||||||
|
|
||||||
|
### 3.2 Globale Gewichte (aus retriever.yaml)
|
||||||
|
|
||||||
|
Datei: `config/retriever.yaml`
|
||||||
|
Beispiel:
|
||||||
|
|
||||||
|
version: 1.0
|
||||||
|
|
||||||
|
scoring:
|
||||||
|
semantic_weight: 1.0 # W_sem
|
||||||
|
edge_weight: 0.7 # W_edge
|
||||||
|
centrality_weight: 0.5 # W_cent
|
||||||
|
|
||||||
|
Falls diese Datei fehlt, greifen ENV-basierte Defaults (`RETRIEVER_W_*`).
|
||||||
|
|
||||||
|
### 3.3 Formel
|
||||||
|
|
||||||
|
total_score =
|
||||||
|
W_sem * semantic_score * max(retriever_weight, 0.0)
|
||||||
|
+ W_edge * edge_bonus
|
||||||
|
+ W_cent * centrality_bonus
|
||||||
|
|
||||||
|
Bemerkungen:
|
||||||
|
|
||||||
|
- Negative `retriever_weight` werden abgefangen.
|
||||||
|
- Die rohen `edge_bonus` und `centrality_bonus` werden im API-/Smoke-Output angezeigt.
|
||||||
|
- Die Gewichte aus `retriever.yaml` beeinflussen **nur den total_score**, nicht die Anzeige.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Konfigurationsdateien
|
||||||
|
|
||||||
|
### 4.1 Typen-Registry (`types.yaml`)
|
||||||
|
|
||||||
|
Beispielauszug:
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
retriever_weight: 1.0
|
||||||
|
chunk_profile: default
|
||||||
|
|
||||||
|
types:
|
||||||
|
types:
|
||||||
|
concept:
|
||||||
|
retriever_weight: 0.60
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
|
||||||
|
project:
|
||||||
|
retriever_weight: 0.97
|
||||||
|
edge_defaults: ["references", "depends_on"]
|
||||||
|
|
||||||
|
journal:
|
||||||
|
retriever_weight: 0.80
|
||||||
|
edge_defaults: ["references", "related_to"]
|
||||||
|
|
||||||
|
source:
|
||||||
|
retriever_weight: 0.50
|
||||||
|
edge_defaults: []
|
||||||
|
|
||||||
|
Diese Werte bestimmen, wie wichtig eine Note/Chunk **prinzipiell** für den Retriever ist.
|
||||||
|
|
||||||
|
### 4.2 Retriever-Konfiguration (`retriever.yaml`)
|
||||||
|
|
||||||
|
Beispiel:
|
||||||
|
|
||||||
|
version: 1.0
|
||||||
|
|
||||||
|
scoring:
|
||||||
|
semantic_weight: 1.0
|
||||||
|
edge_weight: 0.7
|
||||||
|
centrality_weight: 0.5
|
||||||
|
|
||||||
|
# Blueprint für spätere Edge-Feinsteuerung:
|
||||||
|
edge_types:
|
||||||
|
references: 0.20
|
||||||
|
depends_on: 0.18
|
||||||
|
related_to: 0.15
|
||||||
|
similar_to: 0.12
|
||||||
|
belongs_to: 0.10
|
||||||
|
next: 0.06
|
||||||
|
prev: 0.06
|
||||||
|
|
||||||
|
Die Datei befindet sich standardmäßig unter `config/retriever.yaml`.
|
||||||
|
Override per ENV:
|
||||||
|
|
||||||
|
MINDNET_RETRIEVER_CONFIG=/pfad/zur/datei.yaml
|
||||||
|
|
||||||
|
### 4.3 Aktive Gewichte auslesen
|
||||||
|
|
||||||
|
python - << 'PY'
|
||||||
|
from app.core.retriever import _get_scoring_weights
|
||||||
|
print(_get_scoring_weights())
|
||||||
|
PY
|
||||||
|
|
||||||
|
Beispielausgabe:
|
||||||
|
|
||||||
|
(1.0, 0.7, 0.5)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Betriebsmodi
|
||||||
|
|
||||||
|
### 5.1 Semantikmodus (semantic)
|
||||||
|
|
||||||
|
Ablauf:
|
||||||
|
|
||||||
|
- Query-Vektor über embed_text()
|
||||||
|
- Qdrant-Suche über mindnet_chunks
|
||||||
|
- kein Subgraph → edge_bonus = centrality = 0
|
||||||
|
- total_score = W_sem * semantic_score * retriever_weight
|
||||||
|
|
||||||
|
### 5.2 Hybridmodus (hybrid)
|
||||||
|
|
||||||
|
Ablauf:
|
||||||
|
|
||||||
|
- Semantik wie oben
|
||||||
|
- falls expand.depth > 0:
|
||||||
|
- Subgraph-Berechnung über ga.expand()
|
||||||
|
- Ermittlung von edge_bonus und centrality_bonus
|
||||||
|
- total_score = kombinierte Formel aus Semantik + Graph
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Tests
|
||||||
|
|
||||||
|
### Ausführung (Unit & Integration)
|
||||||
|
|
||||||
|
pytest tests/test_retriever_basic.py \
|
||||||
|
tests/test_retriever_weight.py \
|
||||||
|
tests/test_retriever_edges.py
|
||||||
|
|
||||||
|
pytest tests/test_query_unit.py \
|
||||||
|
tests/test_query_text_embed_unit.py
|
||||||
|
|
||||||
|
Tests decken ab:
|
||||||
|
|
||||||
|
- Semantiktreffer
|
||||||
|
- Typ-Gewichte
|
||||||
|
- Edge-Boni
|
||||||
|
- Hybridmodus / FastAPI-Endpunkt
|
||||||
|
|
||||||
|
### Smoke-Test
|
||||||
|
|
||||||
|
python tests/test_retriever_smoke.py \
|
||||||
|
--url "http://127.0.0.1:8001/query" \
|
||||||
|
--query "embeddings" \
|
||||||
|
--mode hybrid \
|
||||||
|
--expand-depth 1 \
|
||||||
|
--top-k 5
|
||||||
|
|
||||||
|
Beispielausgabe:
|
||||||
|
|
||||||
|
[1] note_id=...retriever-design
|
||||||
|
total=1.5054 (semantic=0.4038 edge=1.5910 centrality=0.0000)
|
||||||
|
|
||||||
|
Interpretation:
|
||||||
|
|
||||||
|
- `semantic`, `edge`, `centrality` = Rohwerte
|
||||||
|
- `total` = skaliert mit `retriever.yaml`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Betriebs-Hinweise
|
||||||
|
|
||||||
|
### 7.1 Änderung der Gewichte
|
||||||
|
|
||||||
|
- retriever.yaml anpassen
|
||||||
|
- FastAPI/uvicorn neu starten
|
||||||
|
- Smoke-Test prüfen
|
||||||
|
|
||||||
|
### 7.2 Zielbild WP-04
|
||||||
|
|
||||||
|
Der jetzige Stand ermöglicht:
|
||||||
|
|
||||||
|
- transparente, konfigurierbare Score-Berechnung
|
||||||
|
- Kombination aus Semantik, Typwissen und Graphstruktur
|
||||||
|
- spätere agentenbasierte Selbstjustierung
|
||||||
|
- alternative Profile (z. B. „mehr Semantik“, „mehr Graph“)
|
||||||
|
|
||||||
|
Dieses Dokument dient als stabile Referenz für Entwickler, Tests und spätere Automatisierungsschritte.
|
||||||
|
|
||||||
+2
-1
@@ -1,9 +1,10 @@
|
|||||||
|
|
||||||
fastapi
|
fastapi
|
||||||
uvicorn[standard]
|
uvicorn[standard]
|
||||||
qdrant-client
|
qdrant-client
|
||||||
|
pydantic>=2
|
||||||
sentence-transformers
|
sentence-transformers
|
||||||
torch
|
torch
|
||||||
python-frontmatter
|
python-frontmatter
|
||||||
tqdm
|
tqdm
|
||||||
requests
|
requests
|
||||||
|
numpy
|
||||||
+79
-27
@@ -2,37 +2,89 @@
|
|||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
"title": "mindnet_chunk",
|
"title": "mindnet_chunk",
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"required": ["id", "note_id", "chunk_index", "type", "path", "token_count", "lang"],
|
"description": "Chunk-Payload (Qdrant). Kompatibel mit Alt-Feldern und neuen Feldern für Export/Roundtrip.",
|
||||||
|
"required": ["id", "note_id", "chunk_index", "path"],
|
||||||
"properties": {
|
"properties": {
|
||||||
"id": {"type": "string"},
|
"id": { "type": "string" },
|
||||||
"note_id": {"type": "string"},
|
"scope": { "type": "string", "enum": ["chunk"] },
|
||||||
"note_title": {"type": "string"},
|
"note_id": { "type": "string" },
|
||||||
"chunk_index": {"type": "integer"},
|
"note_title": { "type": "string" },
|
||||||
"char_start": {"type": "integer"},
|
"note_type": { "type": "string" },
|
||||||
"char_end": {"type": "integer"},
|
"note_status": { "type": "string" },
|
||||||
"token_count": {"type": "integer"},
|
"type": { "type": "string", "description": "Legacy: früherer Chunk-Typ; kann dem Note-Typ entsprechen" },
|
||||||
"type": {"type": "string"},
|
"area": { "type": "string" },
|
||||||
"area": {"type": "string"},
|
"project": { "type": "string" },
|
||||||
"project": {"type": "string"},
|
"tags": { "type": "array", "items": { "type": "string" } },
|
||||||
"tags": {"type": "array", "items": {"type": "string"}},
|
|
||||||
"section_title": {"type": "string"},
|
"note_path": { "type": "string" },
|
||||||
"section_path": {"type": "string"},
|
"path": { "type": "string" },
|
||||||
"lang": {"type": "string"},
|
|
||||||
"wikilinks": {"type": "array", "items": {"type": "string"}},
|
"chunk_index": { "type": "integer" },
|
||||||
"external_links": {"type": "array", "items": {"type": "string"}},
|
"section_title":{ "type": ["string","null"] },
|
||||||
|
"section_path": { "type": ["string","null"] },
|
||||||
|
|
||||||
|
"char_start": { "type": ["integer","null"] },
|
||||||
|
"char_end": { "type": ["integer","null"] },
|
||||||
|
"char_len": { "type": "integer" },
|
||||||
|
|
||||||
|
"token_count": { "type": "integer", "description": "Legacy: frühere Token-Zahl" },
|
||||||
|
"token_est": { "type": "integer", "description": "Neue grobe Token-Schätzung (≈ len(text)/4)" },
|
||||||
|
|
||||||
|
"neighbors": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"prev": { "type": ["string","null"] },
|
||||||
|
"next": { "type": ["string","null"] }
|
||||||
|
},
|
||||||
|
"additionalProperties": false
|
||||||
|
},
|
||||||
|
|
||||||
|
"text": { "type": "string" },
|
||||||
|
"text_sha256": { "type": "string", "pattern": "^sha256:[0-9a-fA-F]{64}$" },
|
||||||
|
"lang": { "type": "string" },
|
||||||
|
|
||||||
|
"wikilinks": { "type": "array", "items": { "type": "string" } },
|
||||||
|
"external_links": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"anyOf": [
|
||||||
|
{ "type": "string" },
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"href": { "type": "string" },
|
||||||
|
"label": { "type": ["string","null"] }
|
||||||
|
},
|
||||||
|
"required": ["href"],
|
||||||
|
"additionalProperties": false
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
"references": {
|
"references": {
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {"type": "object", "properties": {
|
"items": {
|
||||||
"target_id": {"type": "string"},
|
"type": "object",
|
||||||
"kind": {"type": "string"}
|
"properties": {
|
||||||
}, "required": ["target_id", "kind"]}
|
"target_id": { "type": "string" },
|
||||||
|
"kind": { "type": "string" }
|
||||||
|
},
|
||||||
|
"required": ["target_id","kind"],
|
||||||
|
"additionalProperties": true
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"neighbors": {"type": "object", "properties": {
|
|
||||||
"prev": {"type": ["string","null"]},
|
"embed_model": { "type": "string" },
|
||||||
"next": {"type": ["string","null"]}
|
"embed_dim": { "type": "integer" },
|
||||||
}},
|
"embed_version": { "type": "integer" },
|
||||||
"path": {"type": "string"},
|
|
||||||
"created_at": {"type": "string"}
|
"created_at": { "type": "string" }
|
||||||
},
|
},
|
||||||
|
|
||||||
|
"allOf": [
|
||||||
|
{ "anyOf": [ { "required": ["token_count"] }, { "required": ["token_est"] } ] },
|
||||||
|
{ "anyOf": [ { "required": ["type"] }, { "required": ["note_type"] } ] }
|
||||||
|
],
|
||||||
|
|
||||||
"additionalProperties": true
|
"additionalProperties": true
|
||||||
}
|
}
|
||||||
@@ -2,15 +2,30 @@
|
|||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
"title": "mindnet_edge",
|
"title": "mindnet_edge",
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"required": ["src_id", "dst_id", "edge_type", "scope"],
|
"description": "Edge-Payload (Qdrant). Unterstützt Legacy (edge_type/src_id/dst_id) und neues Schema (kind/source_id/target_id/note_id/status).",
|
||||||
|
|
||||||
"properties": {
|
"properties": {
|
||||||
"src_id": {"type": "string"},
|
"scope": { "type": "string", "enum": ["note","chunk"] },
|
||||||
"dst_id": {"type": "string"},
|
|
||||||
"edge_type": {"type": "string"},
|
"edge_type": { "type": "string", "description": "Legacy: z. B. references/backlink/belongs_to/prev/next" },
|
||||||
"scope": {"type": "string", "enum": ["note","chunk"]},
|
"src_id": { "type": "string", "description": "Legacy: source_id" },
|
||||||
"weight": {"type": "number"},
|
"dst_id": { "type": "string", "description": "Legacy: target_id" },
|
||||||
"meta": {"type": "object"},
|
|
||||||
"created_at": {"type": "string"}
|
"kind": { "type": "string", "description": "Neu: z. B. references/backlink/belongs_to/prev/next" },
|
||||||
|
"source_id": { "type": "string" },
|
||||||
|
"target_id": { "type": "string" },
|
||||||
|
"note_id": { "type": "string", "description": "Owner-Note für diesen Edge (Filter/Purge)" },
|
||||||
|
"status": { "type": "string", "description": "optional, z. B. 'unresolved'" },
|
||||||
|
|
||||||
|
"weight": { "type": "number" },
|
||||||
|
"meta": { "type": "object" },
|
||||||
|
"created_at":{ "type": "string" }
|
||||||
},
|
},
|
||||||
|
|
||||||
|
"anyOf": [
|
||||||
|
{ "required": ["src_id", "dst_id", "edge_type", "scope"] },
|
||||||
|
{ "required": ["source_id", "target_id", "kind", "scope"] }
|
||||||
|
],
|
||||||
|
|
||||||
"additionalProperties": true
|
"additionalProperties": true
|
||||||
}
|
}
|
||||||
+38
-40
@@ -1,47 +1,45 @@
|
|||||||
{
|
{
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||||
"title": "mindnet_note",
|
"title": "mindnet note payload",
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"required": ["note_id", "title", "type", "status", "created", "path"],
|
|
||||||
"properties": {
|
"properties": {
|
||||||
"note_id": { "type": "string" },
|
"note_id": { "type": "string" },
|
||||||
"title": { "type": "string" },
|
"title": { "type": ["string","null"] },
|
||||||
"type": { "type": "string" },
|
"type": { "type": ["string","null"] },
|
||||||
"status": { "type": "string" },
|
"status": { "type": ["string","null"] },
|
||||||
"created": { "type": "string" },
|
"created": { "type": ["string","null"] },
|
||||||
"updated": { "type": "string" },
|
"updated": { "type": ["string","null"] },
|
||||||
"tags": {
|
"path": { "type": ["string","null"] },
|
||||||
"type": "array",
|
"tags": { "type": ["array","null"], "items": { "type": "string" } },
|
||||||
"items": { "type": "string" }
|
"area": { "type": ["string","null"] },
|
||||||
},
|
"project": { "type": ["string","null"] },
|
||||||
"area": { "type": "string" },
|
"source": { "type": ["string","null"] },
|
||||||
"project": { "type": "string" },
|
"lang": { "type": ["string","null"] },
|
||||||
"priority": { "type": "number" },
|
"slug": { "type": ["string","null"] },
|
||||||
"effort_min": { "type": "number" },
|
"aliases": { "type": ["array","null"], "items": { "type": "string" } },
|
||||||
"due": { "type": "string" },
|
|
||||||
"people": {
|
"fulltext": { "type": ["string","null"] },
|
||||||
"type": "array",
|
"references": { "type": ["array","null"], "items": { "type": "string" } },
|
||||||
"items": { "type": "string" }
|
|
||||||
},
|
"hash_fulltext": { "type": ["string","null"], "pattern": "^[a-f0-9]{64}$" },
|
||||||
"aliases": {
|
"hash_signature": { "type": ["string","null"] },
|
||||||
"type": "array",
|
|
||||||
"items": { "type": "string" }
|
"hash_body": { "type": ["string","null"], "pattern": "^[a-f0-9]{64}$" },
|
||||||
},
|
"hash_frontmatter": { "type": ["string","null"], "pattern": "^[a-f0-9]{64}$" },
|
||||||
"references": {
|
"hash_full": { "type": ["string","null"], "pattern": "^[a-f0-9]{64}$" },
|
||||||
"type": "array",
|
|
||||||
"items": { "type": "string" }
|
"hashes": {
|
||||||
},
|
"type": ["object","null"],
|
||||||
"depends_on": {
|
"description": "Mapping: <mode>:<source>:<normalize> -> sha256 hex",
|
||||||
"type": "array",
|
"patternProperties": {
|
||||||
"items": { "type": "string" }
|
"^(body|frontmatter|full):(parsed|raw):(canonical|none)$": {
|
||||||
},
|
"type": "string",
|
||||||
"assigned_to": {
|
"pattern": "^[a-f0-9]{64}$"
|
||||||
"type": "array",
|
}
|
||||||
"items": { "type": "string" }
|
},
|
||||||
},
|
"additionalProperties": false
|
||||||
"lang": { "type": "string" },
|
}
|
||||||
"path": { "type": "string" },
|
|
||||||
"hash_fulltext": { "type": "string" }
|
|
||||||
},
|
},
|
||||||
|
"required": ["note_id"],
|
||||||
"additionalProperties": true
|
"additionalProperties": true
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/assert_payload_schema.py (compat v1.1)
|
||||||
|
|
||||||
|
Prüft, ob die erwarteten Payload-Indizes (payload_schema) auf den drei Collections vorhanden sind.
|
||||||
|
Kompatibel mit älteren qdrant-client Versionen (ohne with_payload_schema im Wrapper).
|
||||||
|
|
||||||
|
Collections & Pflichtfelder:
|
||||||
|
- mindnet_notes : note_id, type, title, updated, tags
|
||||||
|
- mindnet_chunks : note_id, chunk_id, index, type, tags
|
||||||
|
- mindnet_edges : note_id, kind, scope, source_id, target_id, chunk_id
|
||||||
|
|
||||||
|
Nutzung:
|
||||||
|
python3 -m tests.assert_payload_schema
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client, collection_names
|
||||||
|
|
||||||
|
REQUIRED = {
|
||||||
|
"notes": ["note_id", "type", "title", "updated", "tags"],
|
||||||
|
"chunks": ["note_id", "chunk_id", "index", "type", "tags"],
|
||||||
|
"edges": ["note_id", "kind", "scope", "source_id", "target_id", "chunk_id"],
|
||||||
|
}
|
||||||
|
|
||||||
|
def _safe_model_dump(obj: Any) -> Dict[str, Any]:
|
||||||
|
if hasattr(obj, "model_dump"):
|
||||||
|
return obj.model_dump()
|
||||||
|
if hasattr(obj, "dict"):
|
||||||
|
return obj.dict()
|
||||||
|
return obj if isinstance(obj, dict) else {}
|
||||||
|
|
||||||
|
def _cfg_base_url(cfg) -> str:
|
||||||
|
if getattr(cfg, "url", None):
|
||||||
|
return cfg.url.rstrip('/')
|
||||||
|
host = getattr(cfg, "host", None) or os.getenv("QDRANT_HOST") or "127.0.0.1"
|
||||||
|
port = getattr(cfg, "port", None) or os.getenv("QDRANT_PORT") or "6333"
|
||||||
|
return f"http://{host}:{port}"
|
||||||
|
|
||||||
|
def _http_get(url: str, api_key: str | None) -> Dict[str, Any]:
|
||||||
|
req = urllib.request.Request(url, method="GET")
|
||||||
|
if api_key:
|
||||||
|
req.add_header("api-key", api_key)
|
||||||
|
req.add_header("Authorization", f"Bearer {api_key}")
|
||||||
|
req.add_header("Accept", "application/json")
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
data = resp.read()
|
||||||
|
return json.loads(data.decode("utf-8"))
|
||||||
|
|
||||||
|
def get_collection_payload_schema(client, cfg, name: str) -> Dict[str, Any] | None:
|
||||||
|
# 1) OpenAPI client mit Flag
|
||||||
|
try:
|
||||||
|
oc = getattr(client, "openapi_client", None)
|
||||||
|
if oc is not None and hasattr(oc, "collections_api"):
|
||||||
|
api = oc.collections_api
|
||||||
|
info = api.get_collection(collection_name=name, with_payload_schema=True)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
return (d.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# 2) Wrapper (manche Versionen liefern Schema auch ohne Flag)
|
||||||
|
try:
|
||||||
|
info = client.get_collection(collection_name=name)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
ps = (d.get("result") or {}).get("payload_schema")
|
||||||
|
if ps is not None:
|
||||||
|
return ps
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# 3) Direkter HTTP-Call
|
||||||
|
try:
|
||||||
|
base = _cfg_base_url(cfg)
|
||||||
|
url = f"{base}/collections/{name}?with_payload_schema=true"
|
||||||
|
raw = _http_get(url, getattr(cfg, "api_key", None))
|
||||||
|
return (raw.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def ensure_fields(schema: Dict[str, dict] | None, required: List[str]) -> Dict[str, bool]:
|
||||||
|
ks = set((schema or {}).keys())
|
||||||
|
return {k: (k in ks) for k in required}
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
notes, chunks, edges = collection_names(cfg.prefix)
|
||||||
|
names = {"notes": notes, "chunks": chunks, "edges": edges}
|
||||||
|
|
||||||
|
report = {}
|
||||||
|
ok_all = True
|
||||||
|
for kind, col in names.items():
|
||||||
|
sch = get_collection_payload_schema(client, cfg, col)
|
||||||
|
checks = ensure_fields(sch, REQUIRED[kind])
|
||||||
|
ok = all(checks.values())
|
||||||
|
ok_all = ok_all and ok
|
||||||
|
report[kind] = {
|
||||||
|
"collection": col,
|
||||||
|
"ok": ok,
|
||||||
|
"missing": [k for k,v in checks.items() if not v],
|
||||||
|
"present": [k for k,v in checks.items() if v],
|
||||||
|
}
|
||||||
|
|
||||||
|
print(json.dumps({"prefix": cfg.prefix, "ok": ok_all, "report": report}, ensure_ascii=False, indent=2))
|
||||||
|
if not ok_all:
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/debug_note_payload.py
|
||||||
|
Zeigt, welche app.core.note_payload geladen wird, ruft make_note_payload auf
|
||||||
|
und druckt Typ/Preview der Rückgabe.
|
||||||
|
|
||||||
|
Aufruf:
|
||||||
|
python3 -m scripts.debug_note_payload --file ./test_vault/40_concepts/concept-alpha.md --vault-root ./test_vault
|
||||||
|
"""
|
||||||
|
import argparse, json, os, pprint
|
||||||
|
from app.core import note_payload as np
|
||||||
|
from app.core.parser import read_markdown
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--file", required=True)
|
||||||
|
ap.add_argument("--vault-root", default=None)
|
||||||
|
ap.add_argument("--hash-mode", default=None)
|
||||||
|
ap.add_argument("--hash-normalize", default=None)
|
||||||
|
ap.add_argument("--hash-source", default=None)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
print("module_path:", getattr(np, "__file__", "<unknown>"))
|
||||||
|
parsed = read_markdown(args.file)
|
||||||
|
res = np.make_note_payload(parsed, vault_root=args.vault_root,
|
||||||
|
hash_mode=args.hash_mode,
|
||||||
|
hash_normalize=args.hash_normalize,
|
||||||
|
hash_source=args.hash_source,
|
||||||
|
file_path=args.file)
|
||||||
|
print("return_type:", type(res).__name__)
|
||||||
|
if isinstance(res, dict):
|
||||||
|
print("keys:", list(res.keys()))
|
||||||
|
print(json.dumps(res, ensure_ascii=False, indent=2))
|
||||||
|
else:
|
||||||
|
preview = repr(res)
|
||||||
|
if len(preview) > 600: preview = preview[:600] + "…"
|
||||||
|
print("preview:", preview)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/debug_qdrant_state.py
|
||||||
|
Zeigt Prefix, Collections und einfache Counts (notes/chunks/edges) + sample note_ids.
|
||||||
|
|
||||||
|
Aufruf:
|
||||||
|
export COLLECTION_PREFIX="mindnet"
|
||||||
|
python3 -m scripts.debug_qdrant_state
|
||||||
|
# oder:
|
||||||
|
python3 -m scripts.debug_qdrant_state --prefix mindnet
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, os, json
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
def count_points(client, collection: str) -> int:
|
||||||
|
try:
|
||||||
|
res = client.count(collection, exact=True)
|
||||||
|
return int(getattr(res, "count", 0))
|
||||||
|
except Exception:
|
||||||
|
pts, _ = client.scroll(collection, limit=1)
|
||||||
|
return 1 if pts else 0
|
||||||
|
|
||||||
|
def sample_ids(client, collection: str, id_key: str, limit: int = 5):
|
||||||
|
pts, _ = client.scroll(collection, with_payload=True, with_vectors=False, limit=limit)
|
||||||
|
out = []
|
||||||
|
for p in pts or []:
|
||||||
|
pl = p.payload or {}
|
||||||
|
if id_key in pl:
|
||||||
|
out.append(pl[id_key])
|
||||||
|
return out
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", default=os.environ.get("COLLECTION_PREFIX", "mindnet"))
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
cfg.prefix = args.prefix
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
notes = f"{cfg.prefix}_notes"
|
||||||
|
chunks = f"{cfg.prefix}_chunks"
|
||||||
|
edges = f"{cfg.prefix}_edges"
|
||||||
|
|
||||||
|
print(json.dumps({
|
||||||
|
"prefix": cfg.prefix,
|
||||||
|
"collections": {"notes": notes, "chunks": chunks, "edges": edges},
|
||||||
|
"counts": {
|
||||||
|
"notes": count_points(client, notes),
|
||||||
|
"chunks": count_points(client, chunks),
|
||||||
|
"edges": count_points(client, edges),
|
||||||
|
},
|
||||||
|
"samples": {
|
||||||
|
"notes.note_id": sample_ids(client, notes, "note_id"),
|
||||||
|
"chunks.note_id": sample_ids(client, chunks, "note_id"),
|
||||||
|
"edges.note_id": sample_ids(client, edges, "note_id"),
|
||||||
|
}
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/diag_payload_indexes.py (compat v1.2)
|
||||||
|
|
||||||
|
Zeigt payload_schema je Collection; kompatibel mit älteren qdrant-client Versionen.
|
||||||
|
Strategie (in dieser Reihenfolge):
|
||||||
|
1) openapi_client.collections_api.get_collection(..., with_payload_schema=True)
|
||||||
|
2) client.get_collection(...)
|
||||||
|
3) HTTP GET /collections/{name}?with_payload_schema=true
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import urllib.request
|
||||||
|
import urllib.error
|
||||||
|
from typing import Any, Dict
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client, collection_names
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_model_dump(obj: Any) -> Dict[str, Any]:
|
||||||
|
if hasattr(obj, "model_dump"):
|
||||||
|
return obj.model_dump()
|
||||||
|
if hasattr(obj, "dict"):
|
||||||
|
return obj.dict()
|
||||||
|
return obj if isinstance(obj, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg_base_url(cfg) -> str:
|
||||||
|
if getattr(cfg, "url", None):
|
||||||
|
return cfg.url.rstrip("/")
|
||||||
|
host = getattr(cfg, "host", None) or os.getenv("QDRANT_HOST") or "127.0.0.1"
|
||||||
|
port = getattr(cfg, "port", None) or os.getenv("QDRANT_PORT") or "6333"
|
||||||
|
return f"http://{host}:{port}"
|
||||||
|
|
||||||
|
|
||||||
|
def _http_get(url: str, api_key: str | None) -> Dict[str, Any]:
|
||||||
|
req = urllib.request.Request(url, method="GET")
|
||||||
|
if api_key:
|
||||||
|
req.add_header("api-key", api_key)
|
||||||
|
req.add_header("Authorization", f"Bearer {api_key}")
|
||||||
|
req.add_header("Accept", "application/json")
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
data = resp.read()
|
||||||
|
return json.loads(data.decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def get_collection_payload_schema(client, cfg, name: str) -> Dict[str, Any] | None:
|
||||||
|
# Strategy 1: OpenAPI client with flag
|
||||||
|
try:
|
||||||
|
oc = getattr(client, "openapi_client", None)
|
||||||
|
if oc is not None and hasattr(oc, "collections_api"):
|
||||||
|
api = oc.collections_api
|
||||||
|
info = api.get_collection(collection_name=name, with_payload_schema=True)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
return (d.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# Strategy 2: wrapper (manche Versionen liefern Schema auch ohne Flag)
|
||||||
|
try:
|
||||||
|
info = client.get_collection(collection_name=name) # kein with_payload_schema kwarg!
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
ps = (d.get("result") or {}).get("payload_schema")
|
||||||
|
if ps is not None:
|
||||||
|
return ps
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# Strategy 3: direkter HTTP Call
|
||||||
|
try:
|
||||||
|
base = _cfg_base_url(cfg)
|
||||||
|
url = f"{base}/collections/{name}?with_payload_schema=true"
|
||||||
|
raw = _http_get(url, getattr(cfg, "api_key", None))
|
||||||
|
return (raw.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
notes, chunks, edges = collection_names(cfg.prefix)
|
||||||
|
cols = [notes, chunks, edges]
|
||||||
|
out = []
|
||||||
|
for name in cols:
|
||||||
|
ps = get_collection_payload_schema(client, cfg, name)
|
||||||
|
# Vektorinfos (optional)
|
||||||
|
try:
|
||||||
|
info = client.get_collection(collection_name=name)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
vectors = (d.get("result") or {}).get("vectors")
|
||||||
|
if isinstance(vectors, dict) and "config" in vectors:
|
||||||
|
vectors = vectors["config"]
|
||||||
|
except Exception:
|
||||||
|
vectors = None
|
||||||
|
out.append({"name": name, "vectors": vectors, "payload_schema": ps})
|
||||||
|
print(json.dumps({"prefix": cfg.prefix, "collections": out}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# scripts/edges_dryrun.py
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Dry-Run: Erzeuge Edges aus einem Vault **ohne** Qdrant-Upsert.
|
||||||
|
- Liest Markdown mit YAML-Frontmatter
|
||||||
|
- Chunking: einfacher Absatz-Chunker (Index + text)
|
||||||
|
- Kanten: nutzt app.core.edges.build_edges_for_note()
|
||||||
|
- Ausgabe: JSON pro Note mit Edge-Counts und 3 Beispiel-Payloads
|
||||||
|
|
||||||
|
Aufruf:
|
||||||
|
python3 -m scripts.edges_dryrun --vault ./vault
|
||||||
|
|
||||||
|
Optional:
|
||||||
|
--include-note-scope-refs # auch Frontmatter-Links (links[].target_id) als Note-Scope-Referenzen
|
||||||
|
|
||||||
|
Voraussetzungen:
|
||||||
|
- app/core/parser.py (read_markdown, normalize_frontmatter)
|
||||||
|
- app/core/edges.py (dieses Modul forwardet zur v2-Implementierung)
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
|
||||||
|
from app.core.parser import read_markdown, normalize_frontmatter
|
||||||
|
from app.core.edges import build_edges_for_note
|
||||||
|
|
||||||
|
def _iter_markdown(vault: str):
|
||||||
|
for p in Path(vault).rglob("*.md"):
|
||||||
|
if p.name.startswith("."): # ignore hidden
|
||||||
|
continue
|
||||||
|
yield p
|
||||||
|
|
||||||
|
def _simple_chunker(body: str, note_id: str, note_type: str) -> List[Dict]:
|
||||||
|
# Absatzbasiert, minimal ausreichend für Edges (window/text, chunk_id, ord, note_id, type)
|
||||||
|
paras = [s.strip() for s in re.split(r"\n{2,}", body or "") if s.strip()]
|
||||||
|
chunks = []
|
||||||
|
for i, t in enumerate(paras):
|
||||||
|
chunks.append({
|
||||||
|
"chunk_id": f"{note_id}#c{i:04d}",
|
||||||
|
"ord": i,
|
||||||
|
"text": t,
|
||||||
|
"note_id": note_id,
|
||||||
|
"type": note_type,
|
||||||
|
})
|
||||||
|
return chunks
|
||||||
|
|
||||||
|
def _fm_note_refs(fm: Dict) -> List[str]:
|
||||||
|
out = []
|
||||||
|
links = fm.get("links") or []
|
||||||
|
if isinstance(links, list):
|
||||||
|
for e in links:
|
||||||
|
if isinstance(e, dict):
|
||||||
|
tid = e.get("target_id")
|
||||||
|
if isinstance(tid, str) and tid.strip():
|
||||||
|
out.append(tid.strip())
|
||||||
|
return out
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--vault", required=True)
|
||||||
|
ap.add_argument("--include-note-scope-refs", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
vault = args.vault
|
||||||
|
include_note_scope = args.include_note_scope_refs
|
||||||
|
|
||||||
|
report = []
|
||||||
|
for p in _iter_markdown(vault):
|
||||||
|
parsed = read_markdown(str(p))
|
||||||
|
if not parsed:
|
||||||
|
continue
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter or {})
|
||||||
|
note_id = fm.get("id") or p.stem
|
||||||
|
note_type = (fm.get("type") or "concept").lower()
|
||||||
|
chunks = _simple_chunker(parsed.body, note_id, note_type)
|
||||||
|
note_refs = _fm_note_refs(fm)
|
||||||
|
|
||||||
|
edges = build_edges_for_note(
|
||||||
|
note_id=note_id,
|
||||||
|
chunk_payloads=chunks,
|
||||||
|
note_level_refs=note_refs,
|
||||||
|
include_note_scope_refs=include_note_scope,
|
||||||
|
)
|
||||||
|
kinds = {}
|
||||||
|
for e in edges:
|
||||||
|
key = (e.get("relation") or e.get("kind") or "edge")
|
||||||
|
kinds[key] = kinds.get(key, 0) + 1
|
||||||
|
report.append({
|
||||||
|
"path": str(p),
|
||||||
|
"note_id": note_id,
|
||||||
|
"type": note_type,
|
||||||
|
"chunks": len(chunks),
|
||||||
|
"edges_total": len(edges),
|
||||||
|
"edges_by_kind": kinds,
|
||||||
|
"samples": edges[:3],
|
||||||
|
})
|
||||||
|
|
||||||
|
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/edges_full_check.py
|
||||||
|
Zählt und validiert Kanten in Qdrant. Erkennt folgende Rule-Gruppen:
|
||||||
|
- explicit_total: rule_id startswith "explicit:" (z.B. explicit:wikilink, explicit:note_scope)
|
||||||
|
- callout_total: rule_id == "callout:edge"
|
||||||
|
- inline_total: rule_id startswith "inline:" (z.B. inline:rel)
|
||||||
|
- defaults_total: rule_id startswith "edge_defaults:"
|
||||||
|
- structure: rule_id in {"structure:belongs_to","structure:order"}
|
||||||
|
|
||||||
|
Gibt zusätzlich:
|
||||||
|
- edges_by_kind (aggregiert)
|
||||||
|
- notes/chunks/edges Anzahlen
|
||||||
|
- multi_callout_detected: True, falls ein Chunk mehrere Callout-Ziele der gleichen Relation enthält
|
||||||
|
- per_note_checks: belongs_to == chunks, next == prev == (chunks-1)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
from typing import Dict, Any, List, Tuple
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
|
||||||
|
def _count_collection_points(client, name: str) -> int:
|
||||||
|
try:
|
||||||
|
res = client.count(collection_name=name, exact=True)
|
||||||
|
return res.count or 0
|
||||||
|
except Exception:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _scroll_all(client, collection: str) -> List[Any]:
|
||||||
|
pts_all = []
|
||||||
|
offset = None
|
||||||
|
while True:
|
||||||
|
pts, offset = client.scroll(
|
||||||
|
collection_name=collection,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=2048,
|
||||||
|
offset=offset,
|
||||||
|
)
|
||||||
|
pts_all.extend(pts or [])
|
||||||
|
if offset is None:
|
||||||
|
break
|
||||||
|
return pts_all
|
||||||
|
|
||||||
|
|
||||||
|
def _rule_group(rule_id: str) -> str:
|
||||||
|
if not rule_id:
|
||||||
|
return "unknown"
|
||||||
|
if rule_id == "callout:edge":
|
||||||
|
return "callout"
|
||||||
|
if rule_id.startswith("inline:"): # <—— wichtig für "inline:rel"
|
||||||
|
return "inline"
|
||||||
|
if rule_id.startswith("edge_defaults:"):
|
||||||
|
return "defaults"
|
||||||
|
if rule_id.startswith("explicit:"):
|
||||||
|
return "explicit"
|
||||||
|
if rule_id in ("structure:belongs_to", "structure:order"):
|
||||||
|
return "structure"
|
||||||
|
return "other"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
col_notes = f"{cfg.prefix}_notes"
|
||||||
|
col_chunks = f"{cfg.prefix}_chunks"
|
||||||
|
col_edges = f"{cfg.prefix}_edges"
|
||||||
|
|
||||||
|
# High-level counts
|
||||||
|
notes_n = _count_collection_points(client, col_notes)
|
||||||
|
chunks_n = _count_collection_points(client, col_chunks)
|
||||||
|
edges_pts = _scroll_all(client, col_edges)
|
||||||
|
edges_n = len(edges_pts)
|
||||||
|
|
||||||
|
# By kind / by rule group
|
||||||
|
by_kind = Counter()
|
||||||
|
group_counts = Counter()
|
||||||
|
callout_buckets: Dict[Tuple[str, str], int] = defaultdict(int) # (chunk_id, kind) -> n targets
|
||||||
|
per_note = defaultdict(lambda: {"chunks": 0, "belongs_to": 0, "next": 0, "prev": 0})
|
||||||
|
|
||||||
|
# Für per_note checks: chunks pro note_id aus mindnet_chunks laden
|
||||||
|
chunks_pts = _scroll_all(client, col_chunks)
|
||||||
|
chunks_by_note = Counter([p.payload.get("note_id") for p in chunks_pts if p.payload])
|
||||||
|
|
||||||
|
for p in edges_pts:
|
||||||
|
pl = p.payload or {}
|
||||||
|
kind = str(pl.get("kind") or pl.get("relation") or "edge")
|
||||||
|
rule_id = str(pl.get("rule_id") or "")
|
||||||
|
note_id = str(pl.get("note_id") or "")
|
||||||
|
chunk_id = str(pl.get("chunk_id") or "")
|
||||||
|
by_kind[kind] += 1
|
||||||
|
|
||||||
|
group = _rule_group(rule_id)
|
||||||
|
group_counts[group] += 1
|
||||||
|
|
||||||
|
# Multi-Callout-Erkennung: mehrere callout-Edges gleicher Relation aus demselben Chunk
|
||||||
|
if group == "callout" and chunk_id and kind:
|
||||||
|
callout_buckets[(chunk_id, kind)] += 1
|
||||||
|
|
||||||
|
# Per-note Strukturchecks
|
||||||
|
if note_id:
|
||||||
|
if kind == "belongs_to":
|
||||||
|
per_note[note_id]["belongs_to"] += 1
|
||||||
|
elif kind == "next":
|
||||||
|
per_note[note_id]["next"] += 1
|
||||||
|
elif kind == "prev":
|
||||||
|
per_note[note_id]["prev"] += 1
|
||||||
|
|
||||||
|
# set chunks count for per_note
|
||||||
|
for n_id, c in chunks_by_note.items():
|
||||||
|
per_note[n_id]["chunks"] = c
|
||||||
|
|
||||||
|
# final checks per note
|
||||||
|
per_note_checks = {}
|
||||||
|
for n_id, stats in per_note.items():
|
||||||
|
c = stats.get("chunks", 0)
|
||||||
|
bt = stats.get("belongs_to", 0)
|
||||||
|
nx = stats.get("next", 0)
|
||||||
|
pv = stats.get("prev", 0)
|
||||||
|
per_note_checks[n_id] = {
|
||||||
|
"chunks": c,
|
||||||
|
"belongs_to": bt,
|
||||||
|
"next": nx,
|
||||||
|
"prev": pv,
|
||||||
|
"checks": {
|
||||||
|
"belongs_to_equals_chunks": (bt == c),
|
||||||
|
"next_prev_match": (nx == pv == max(c - 1, 0)),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
multi_callout_detected = any(v > 1 for v in callout_buckets.values())
|
||||||
|
|
||||||
|
out = {
|
||||||
|
"prefix": cfg.prefix,
|
||||||
|
"counts": {
|
||||||
|
"notes": notes_n,
|
||||||
|
"chunks": chunks_n,
|
||||||
|
"edges": edges_n,
|
||||||
|
"edges_by_kind": dict(by_kind),
|
||||||
|
"explicit_total": group_counts.get("explicit", 0),
|
||||||
|
"defaults_total": group_counts.get("defaults", 0),
|
||||||
|
"callout_total": group_counts.get("callout", 0),
|
||||||
|
"inline_total": group_counts.get("inline", 0),
|
||||||
|
"structure_total": group_counts.get("structure", 0),
|
||||||
|
},
|
||||||
|
"per_note_checks": per_note_checks,
|
||||||
|
"multi_callout_detected": bool(multi_callout_detected),
|
||||||
|
"has_duplicates": False, # dedupe passiert beim Upsert
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+165
-188
@@ -1,249 +1,226 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Script: export_markdown.py — Qdrant → Markdown (Obsidian-kompatibel)
|
Script: scripts/export_markdown.py — Qdrant → Markdown (Vault)
|
||||||
Version: 1.4.0
|
Version: 1.4.1
|
||||||
Datum: 2025-09-09
|
Datum: 2025-09-10
|
||||||
|
|
||||||
Kurzbeschreibung
|
Funktion
|
||||||
----------------
|
--------
|
||||||
Exportiert Markdown-Notizen aus Qdrant in einen Zielordner (Vault).
|
Exportiert Notes (Frontmatter + Body) aus Qdrant in einen Zielordner. Der Body wird
|
||||||
Rekonstruiert YAML-Frontmatter und Body.
|
bevorzugt aus dem Feld `fulltext` rekonstruiert; falls leer/nicht vorhanden, aus Chunks
|
||||||
|
(Sortierung: seq → chunk_index → Nummer in chunk_id). Pfade werden **relativ** geschrieben.
|
||||||
|
|
||||||
Body-Rekonstruktions-Priorität (abwärtskompatibel):
|
Optionen
|
||||||
1) notes.payload.fulltext (verlustfrei, wenn beim Import gespeichert)
|
--------
|
||||||
2) ansonsten aus allen zugehörigen Chunks: payload.text → payload.content → payload.raw
|
--out PATH Zielordner (erforderlich)
|
||||||
in stabiler, sequentieller Reihenfolge (seq/chunk_index/ID-Nummer)
|
--prefix TEXT Collection-Prefix (CLI überschreibt ENV COLLECTION_PREFIX)
|
||||||
|
--note-id ID Nur eine Note exportieren
|
||||||
|
--overwrite Existierende Dateien überschreiben
|
||||||
|
--include-edges MODE none|yaml|footer (Default: none)
|
||||||
|
--flatten-paths Alle Dateien flach schreiben; Originalpfad in FM: orig_path
|
||||||
|
|
||||||
Wichtige Fixes
|
ENV
|
||||||
--------------
|
---
|
||||||
- **Pfad-Normalisierung**: erzwingt relative Pfade (führe führende '/' ab, backslashes → slashes),
|
COLLECTION_PREFIX, QDRANT_URL | QDRANT_HOST/QDRANT_PORT | QDRANT_API_KEY
|
||||||
damit ``--out`` nicht ignoriert wird.
|
|
||||||
- **`--prefix` (optional)**: Überschreibt COLLECTION_PREFIX; ENV bleibt Default (rückwärtskompatibel).
|
|
||||||
|
|
||||||
ENV / Qdrant
|
|
||||||
------------
|
|
||||||
- QDRANT_URL (oder QDRANT_HOST/QDRANT_PORT)
|
|
||||||
- QDRANT_API_KEY (optional)
|
|
||||||
- COLLECTION_PREFIX (Default: mindnet)
|
|
||||||
|
|
||||||
Aufruf
|
|
||||||
------
|
|
||||||
python3 -m scripts.export_markdown --out ./_exportVault
|
|
||||||
python3 -m scripts.export_markdown --out ./_exportVault --note-id 20250821-foo
|
|
||||||
python3 -m scripts.export_markdown --out ./_exportVault --overwrite
|
|
||||||
python3 -m scripts.export_markdown --out ./_exportVault --prefix mindnet_dev # optional
|
|
||||||
|
|
||||||
Beispiele
|
Beispiele
|
||||||
---------
|
---------
|
||||||
COLLECTION_PREFIX=mindnet QDRANT_URL=http://127.0.0.1:6333 \\
|
export COLLECTION_PREFIX="mindnet"
|
||||||
python3 -m scripts.export_markdown --out ./_exportVault --overwrite
|
python3 -m scripts.export_markdown --out ./_exportVault
|
||||||
"""
|
|
||||||
|
|
||||||
|
# Nur eine Note, Edges als YAML-Feld 'references'
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault --note-id concept-alpha --include-edges yaml
|
||||||
|
|
||||||
|
# Flach schreiben mit Überschreiben
|
||||||
|
python3 -m scripts.export_markdown --out ./_exportVault --flatten-paths --overwrite
|
||||||
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
|
||||||
import os
|
import os
|
||||||
import re
|
import json
|
||||||
from typing import Dict, Iterable, List, Optional, Tuple
|
from typing import Dict, List, Optional, Tuple, Any
|
||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
from qdrant_client.http import models as rest
|
from qdrant_client.http import models as rest
|
||||||
from qdrant_client import QdrantClient
|
|
||||||
|
|
||||||
from app.core.qdrant import QdrantConfig, get_client, ensure_collections
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
from app.core.qdrant import ensure_collections # safety
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
# Utilities
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _names(prefix: str) -> Tuple[str, str, str]:
|
|
||||||
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def _norm_rel_path(path: str) -> str:
|
||||||
|
p = (path or "").replace("\\", "/").lstrip("/")
|
||||||
|
return p if p else ""
|
||||||
|
|
||||||
def _ensure_dir(path: str) -> None:
|
def _ensure_dir(path: str) -> None:
|
||||||
d = os.path.dirname(path)
|
d = os.path.dirname(path)
|
||||||
if d and not os.path.isdir(d):
|
if d and not os.path.exists(d):
|
||||||
os.makedirs(d, exist_ok=True)
|
os.makedirs(d, exist_ok=True)
|
||||||
|
|
||||||
|
def _yaml_dump(data: Dict[str, Any]) -> str:
|
||||||
|
return yaml.safe_dump(data, allow_unicode=True, sort_keys=False).strip()
|
||||||
|
|
||||||
def _normalize_rel_path(p: str) -> str:
|
def _frontmatter_block(fm: Dict[str, Any]) -> str:
|
||||||
"""Pfad relativ halten & normalisieren (slashes, führende / entfernen)."""
|
y = _yaml_dump(fm)
|
||||||
p = (p or "").replace("\\", "/")
|
return f"---\n{y}\n---\n"
|
||||||
return p.lstrip("/")
|
|
||||||
|
|
||||||
|
def _scroll_all(client, collection: str, flt: Optional[rest.Filter] = None) -> List[Any]:
|
||||||
def _to_md(frontmatter: dict, body: str) -> str:
|
|
||||||
fm = yaml.safe_dump(frontmatter, sort_keys=False, allow_unicode=True).strip()
|
|
||||||
return f"---\n{fm}\n---\n{(body or '').rstrip()}\n"
|
|
||||||
|
|
||||||
|
|
||||||
def _scroll_all(
|
|
||||||
client: QdrantClient,
|
|
||||||
col: str,
|
|
||||||
flt: Optional[rest.Filter] = None,
|
|
||||||
with_payload: bool = True,
|
|
||||||
with_vectors: bool = False,
|
|
||||||
limit: int = 256,
|
|
||||||
):
|
|
||||||
"""Scrollt durch alle Punkte einer Collection und liefert eine Liste mit Points."""
|
|
||||||
out = []
|
out = []
|
||||||
next_page = None
|
nextp = None
|
||||||
while True:
|
while True:
|
||||||
pts, next_page = client.scroll(
|
pts, nextp = client.scroll(
|
||||||
collection_name=col,
|
collection_name=collection,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=256,
|
||||||
scroll_filter=flt,
|
scroll_filter=flt,
|
||||||
with_payload=with_payload,
|
offset=nextp,
|
||||||
with_vectors=with_vectors,
|
|
||||||
limit=limit,
|
|
||||||
offset=next_page,
|
|
||||||
)
|
)
|
||||||
if not pts:
|
if not pts:
|
||||||
break
|
break
|
||||||
out.extend(pts)
|
out.extend(pts)
|
||||||
if not next_page:
|
if nextp is None:
|
||||||
break
|
break
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
def _reconstruct_body_from_chunks(chunks: List[Any]) -> str:
|
||||||
|
def seq_key(pl: Dict[str, Any]) -> Tuple[int, int, int]:
|
||||||
|
s = pl.get("seq")
|
||||||
|
ci = pl.get("chunk_index")
|
||||||
|
cid = pl.get("chunk_id") or ""
|
||||||
|
n = 0
|
||||||
|
if isinstance(cid, str) and "#" in cid:
|
||||||
|
try:
|
||||||
|
n = int(cid.rsplit("#", 1)[-1])
|
||||||
|
except Exception:
|
||||||
|
n = 0
|
||||||
|
return (int(s) if isinstance(s, int) else 0,
|
||||||
|
int(ci) if isinstance(ci, int) else 0,
|
||||||
|
n)
|
||||||
|
chunks_sorted = sorted(chunks, key=lambda p: seq_key(p.payload or {}))
|
||||||
|
texts: List[str] = []
|
||||||
|
for p in chunks_sorted:
|
||||||
|
pl = p.payload or {}
|
||||||
|
t = pl.get("text") or pl.get("content") or pl.get("raw") or ""
|
||||||
|
if isinstance(t, str) and t:
|
||||||
|
texts.append(t)
|
||||||
|
return "\n".join(texts).strip()
|
||||||
|
|
||||||
def _load_chunks_for_note(client: QdrantClient, chunks_col: str, note_id: str) -> List[dict]:
|
def _collect_forward_refs_from_edges(edges: List[Any]) -> List[str]:
|
||||||
flt = rest.Filter(must=[rest.FieldCondition(
|
refs = []
|
||||||
key="note_id",
|
for p in edges:
|
||||||
match=rest.MatchValue(value=note_id),
|
pl = p.payload or {}
|
||||||
)])
|
if pl.get("kind") == "references" and isinstance(pl.get("target_id"), str):
|
||||||
pts = _scroll_all(client, chunks_col, flt, with_payload=True, with_vectors=False)
|
refs.append(pl["target_id"])
|
||||||
# Sortierung: bevorzugt seq → chunk_index → Nummer in id
|
# de-dupe, preserve order
|
||||||
def _seq(pl: dict) -> Tuple[int, int, int]:
|
seen = set()
|
||||||
s1 = pl.get("seq", pl.get("chunk_index", -1))
|
out = []
|
||||||
s2 = pl.get("chunk_index", -1)
|
for r in refs:
|
||||||
# Nummer-Anteil aus "noteid#<n>"
|
if r not in seen:
|
||||||
s3 = 0
|
seen.add(r)
|
||||||
try:
|
out.append(r)
|
||||||
m = re.search(r"#(\\d+)$", pl.get("id") or "")
|
return out
|
||||||
if m:
|
|
||||||
s3 = int(m.group(1))
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return (int(s1) if isinstance(s1, int) else -1, int(s2) if isinstance(s2, int) else -1, s3)
|
|
||||||
|
|
||||||
pts_sorted = sorted(pts, key=lambda p: _seq(p.payload or {}))
|
# ---------------------------------------------------------------------
|
||||||
return [p.payload or {} for p in pts_sorted]
|
|
||||||
|
|
||||||
|
|
||||||
def _reconstruct_body(note_pl: dict, chunk_payloads: List[dict]) -> str:
|
|
||||||
# 1) Volltext vorhanden?
|
|
||||||
fulltext = note_pl.get("fulltext")
|
|
||||||
if isinstance(fulltext, str) and fulltext.strip():
|
|
||||||
return fulltext
|
|
||||||
|
|
||||||
# 2) Aus Chunks zusammensetzen: text → content → raw
|
|
||||||
parts: List[str] = []
|
|
||||||
for ch in chunk_payloads:
|
|
||||||
text = ch.get("text") or ch.get("content") or ch.get("raw")
|
|
||||||
if isinstance(text, str) and text.strip():
|
|
||||||
parts.append(text.rstrip())
|
|
||||||
return ("\n\n".join(parts)).rstrip() + ("\n" if parts else "")
|
|
||||||
|
|
||||||
|
|
||||||
def _export_one_note(
|
|
||||||
client: QdrantClient,
|
|
||||||
prefix: str,
|
|
||||||
note_pl: dict,
|
|
||||||
out_root: str,
|
|
||||||
overwrite: bool,
|
|
||||||
) -> dict:
|
|
||||||
notes_col, chunks_col, _ = _names(prefix)
|
|
||||||
note_id = note_pl.get("note_id") or note_pl.get("id")
|
|
||||||
|
|
||||||
# Pfad robust bestimmen und relativ halten
|
|
||||||
path = note_pl.get("path") or f"{note_id}.md"
|
|
||||||
path = _normalize_rel_path(path)
|
|
||||||
out_path = os.path.join(out_root, path).replace("\\", "/")
|
|
||||||
|
|
||||||
# Frontmatter aus Payload zurückführen (nur bekannte Felder)
|
|
||||||
fm: Dict[str, object] = {}
|
|
||||||
for k in [
|
|
||||||
"title", "id", "type", "status", "created", "updated", "tags",
|
|
||||||
"priority", "effort_min", "due", "people", "aliases",
|
|
||||||
"depends_on", "assigned_to", "lang",
|
|
||||||
]:
|
|
||||||
v = note_pl.get(k) if k in note_pl else note_pl.get(f"note_{k}")
|
|
||||||
if v not in (None, [], ""):
|
|
||||||
fm[k] = v
|
|
||||||
|
|
||||||
# Mindestfelder
|
|
||||||
if "id" not in fm and note_id:
|
|
||||||
fm["id"] = note_id
|
|
||||||
if "title" not in fm and note_pl.get("title"):
|
|
||||||
fm["title"] = note_pl["title"]
|
|
||||||
|
|
||||||
# Body beschaffen
|
|
||||||
chunks = _load_chunks_for_note(client, chunks_col, note_id)
|
|
||||||
body = _reconstruct_body(note_pl, chunks)
|
|
||||||
|
|
||||||
# Schreiben?
|
|
||||||
if os.path.exists(out_path) and not overwrite:
|
|
||||||
return {"note_id": note_id, "path": path, "status": "skip_exists"}
|
|
||||||
|
|
||||||
_ensure_dir(out_path)
|
|
||||||
with open(out_path, "w", encoding="utf-8") as f:
|
|
||||||
f.write(_to_md(fm, body))
|
|
||||||
|
|
||||||
return {"note_id": note_id, "path": path, "status": "written"}
|
|
||||||
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Main
|
# Main
|
||||||
# -----------------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
ap = argparse.ArgumentParser()
|
ap = argparse.ArgumentParser()
|
||||||
ap.add_argument("--out", required=True, help="Zielordner für den Export-Vault")
|
ap.add_argument("--out", required=True, help="Zielordner für exportierte Markdown-Dateien")
|
||||||
ap.add_argument("--note-id", help="Nur eine Note exportieren (Note-ID)")
|
ap.add_argument("--prefix", help="Collection-Prefix (überschreibt ENV COLLECTION_PREFIX)")
|
||||||
ap.add_argument("--overwrite", action="store_true", help="Bestehende Dateien überschreiben")
|
ap.add_argument("--note-id", help="Nur eine bestimmte Note-ID exportieren")
|
||||||
ap.add_argument("--prefix", help="(Optional) überschreibt COLLECTION_PREFIX aus ENV")
|
ap.add_argument("--overwrite", action="store_true", help="Existierende Dateien überschreiben")
|
||||||
|
ap.add_argument("--include-edges", choices=["none", "yaml", "footer"], default="none",
|
||||||
|
help="Forward-Links mit exportieren (aus Edges oder Note-Payload)")
|
||||||
|
ap.add_argument("--flatten-paths", action="store_true", help="Alle Dateien flach schreiben (orig_path in Frontmatter)")
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
# Qdrant-Konfiguration
|
|
||||||
cfg = QdrantConfig.from_env()
|
cfg = QdrantConfig.from_env()
|
||||||
if args.prefix:
|
if args.prefix:
|
||||||
cfg.prefix = args.prefix # abwärtskompatibel: ENV bleibt Default
|
cfg.prefix = args.prefix.strip()
|
||||||
|
|
||||||
client = get_client(cfg)
|
client = get_client(cfg)
|
||||||
ensure_collections(client, cfg.prefix, cfg.dim)
|
ensure_collections(client, cfg.prefix, cfg.dim)
|
||||||
|
|
||||||
notes_col, _, _ = _names(cfg.prefix)
|
out_root = os.path.abspath(args.out)
|
||||||
|
os.makedirs(out_root, exist_ok=True)
|
||||||
|
|
||||||
# Notes holen (optional gefiltert)
|
notes_col, chunks_col, edges_col = collections(cfg.prefix)
|
||||||
|
|
||||||
|
# Filter nach note-id (optional)
|
||||||
flt = None
|
flt = None
|
||||||
if args.note_id:
|
if args.note_id:
|
||||||
flt = rest.Filter(must=[rest.FieldCondition(
|
flt = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=args.note_id))])
|
||||||
key="note_id",
|
|
||||||
match=rest.MatchValue(value=args.note_id),
|
|
||||||
)])
|
|
||||||
|
|
||||||
note_pts = _scroll_all(client, notes_col, flt, with_payload=True, with_vectors=False)
|
notes = _scroll_all(client, notes_col, flt)
|
||||||
if not note_pts:
|
total = 0
|
||||||
print(json.dumps({"exported": 0, "out": args.out, "message": "Keine Notes gefunden."}, ensure_ascii=False))
|
for n in notes:
|
||||||
return
|
pl = n.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
rel_path = _norm_rel_path(pl.get("path") or "")
|
||||||
|
if args.flatten_paths or not rel_path:
|
||||||
|
fname = f"{(nid or 'note')}.md"
|
||||||
|
out_path = os.path.join(out_root, fname)
|
||||||
|
else:
|
||||||
|
out_path = os.path.join(out_root, rel_path)
|
||||||
|
out_path = out_path.replace("\\", "/")
|
||||||
|
_ensure_dir(out_path)
|
||||||
|
|
||||||
results = []
|
# Frontmatter aufbauen (nur sinnvolle Felder)
|
||||||
for p in note_pts:
|
fm_fields = ["id","title","type","status","created","updated","tags","area","project","source","lang","slug","aliases"]
|
||||||
pl = p.payload or {}
|
fm: Dict[str, Any] = {}
|
||||||
try:
|
fm["id"] = nid
|
||||||
res = _export_one_note(client, cfg.prefix, pl, args.out, args.overwrite)
|
for k in fm_fields:
|
||||||
except Exception as e:
|
if k == "id":
|
||||||
res = {"note_id": pl.get("note_id") or pl.get("id"), "error": str(e)}
|
continue
|
||||||
results.append(res)
|
if k in pl and pl[k] is not None:
|
||||||
|
fm[k] = pl[k]
|
||||||
|
if args.flatten_paths and rel_path:
|
||||||
|
fm["orig_path"] = rel_path
|
||||||
|
|
||||||
print(json.dumps({
|
# Body ermitteln (fulltext oder Chunks)
|
||||||
"exported": len([r for r in results if r.get("status") == "written"]),
|
body = (pl.get("fulltext") or "").strip()
|
||||||
"skipped": len([r for r in results if r.get("status") == "skip_exists"]),
|
if not body:
|
||||||
"out": args.out,
|
flt_chunks = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=nid))])
|
||||||
"details": results,
|
chunks = _scroll_all(client, chunks_col, flt_chunks)
|
||||||
}, ensure_ascii=False))
|
body = _reconstruct_body_from_chunks(chunks)
|
||||||
|
|
||||||
|
# Edges (optional)
|
||||||
|
refs: List[str] = []
|
||||||
|
if args.include_edges != "none":
|
||||||
|
# aus Note-Payload, falls vorhanden
|
||||||
|
if isinstance(pl.get("references"), list) and pl["references"]:
|
||||||
|
refs = [r for r in pl["references"] if isinstance(r, str)]
|
||||||
|
else:
|
||||||
|
flt_edges = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=nid))])
|
||||||
|
edges = _scroll_all(client, edges_col, flt_edges)
|
||||||
|
refs = _collect_forward_refs_from_edges(edges)
|
||||||
|
if args.include_edges == "yaml" and refs:
|
||||||
|
fm["references"] = refs
|
||||||
|
|
||||||
|
# Datei schreiben
|
||||||
|
if (not args.overwrite) and os.path.exists(out_path):
|
||||||
|
print(json.dumps({"note_id": nid, "path": out_path, "decision": "skip-exists"}))
|
||||||
|
continue
|
||||||
|
|
||||||
|
content = _frontmatter_block(fm) + (body + "\n" if body else "")
|
||||||
|
if args.include_edges == "footer" and refs:
|
||||||
|
content += "\n---\nLinks:\n" + "\n".join(f"- [[{r}]]" for r in refs) + "\n"
|
||||||
|
|
||||||
|
with open(out_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write(content)
|
||||||
|
|
||||||
|
print(json.dumps({"note_id": nid, "path": out_path, "decision": "write"}))
|
||||||
|
total += 1
|
||||||
|
|
||||||
|
print(f"Done. Exported notes: {total}")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -0,0 +1,340 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
scripts/health_check_mindnet.py
|
||||||
|
|
||||||
|
Health-Check für den mindnet-Retriever-/Query-Endpoint.
|
||||||
|
|
||||||
|
Funktion:
|
||||||
|
- Führt POST-Requests auf /query in verschiedenen Modi aus (standard: semantic + hybrid).
|
||||||
|
- Prüft Status-Code, JSON-Struktur und Anzahl der Treffer.
|
||||||
|
- Kennzeichnet Probleme als:
|
||||||
|
- status="ok"
|
||||||
|
- status="warning" (z.B. Timeout)
|
||||||
|
- status="error" (harte Fehler wie HTTP-Fehler, JSON-Fehler etc.)
|
||||||
|
|
||||||
|
Exit-Code:
|
||||||
|
- Default (tolerant):
|
||||||
|
- overall_status = "ok" (inkl. warnings) → Exit-Code 0
|
||||||
|
- overall_status = "error" → Exit-Code 1
|
||||||
|
- Mit --strict:
|
||||||
|
- warnings werden wie errors behandelt → Exit-Code 1
|
||||||
|
|
||||||
|
Beispiele:
|
||||||
|
|
||||||
|
python scripts/health_check_mindnet.py \
|
||||||
|
--url http://127.0.0.1:8001/query \
|
||||||
|
--query "embeddings" \
|
||||||
|
--top-k 3
|
||||||
|
|
||||||
|
python scripts/health_check_mindnet.py \
|
||||||
|
--url http://127.0.0.1:8001/query \
|
||||||
|
--query "embeddings" \
|
||||||
|
--top-k 3 \
|
||||||
|
--timeout 15 \
|
||||||
|
--modes hybrid
|
||||||
|
|
||||||
|
# Strenger Modus (warnings → Exit-Code 1)
|
||||||
|
python scripts/health_check_mindnet.py --strict
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
|
|
||||||
|
def _post_query(
|
||||||
|
url: str,
|
||||||
|
mode: str,
|
||||||
|
query: str,
|
||||||
|
top_k: int,
|
||||||
|
timeout: float,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Führt einen POST auf den /query-Endpoint aus und gibt das dekodierte JSON zurück.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
urllib.error.URLError, urllib.error.HTTPError, RuntimeError
|
||||||
|
"""
|
||||||
|
payload = {
|
||||||
|
"mode": mode,
|
||||||
|
"query": query,
|
||||||
|
"top_k": top_k,
|
||||||
|
"expand": {
|
||||||
|
"depth": 1 if mode == "hybrid" else 0,
|
||||||
|
"edge_types": ["references", "belongs_to", "prev", "next"],
|
||||||
|
},
|
||||||
|
"ret": {
|
||||||
|
"with_paths": True,
|
||||||
|
"with_notes": True,
|
||||||
|
"with_chunks": True,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
data = json.dumps(payload).encode("utf-8")
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url=url,
|
||||||
|
data=data,
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
|
||||||
|
start = time.monotonic()
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
status_code = resp.getcode()
|
||||||
|
body = resp.read().decode("utf-8")
|
||||||
|
end = time.monotonic()
|
||||||
|
|
||||||
|
latency_ms = int((end - start) * 1000)
|
||||||
|
|
||||||
|
if status_code != 200:
|
||||||
|
raise RuntimeError(f"HTTP {status_code}: {body}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
doc = json.loads(body)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise RuntimeError(f"Antwort ist kein gültiges JSON: {exc}") from exc
|
||||||
|
|
||||||
|
if not isinstance(doc, dict):
|
||||||
|
raise RuntimeError("Antwort ist kein JSON-Objekt")
|
||||||
|
|
||||||
|
doc["_latency_ms"] = latency_ms
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_response(
|
||||||
|
mode: str,
|
||||||
|
doc: Dict[str, Any],
|
||||||
|
min_results: int,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Prüft die wichtigsten Invarianten des Query-Responses.
|
||||||
|
|
||||||
|
Rückgabe-Format:
|
||||||
|
|
||||||
|
{
|
||||||
|
"status": "ok" | "warning" | "error",
|
||||||
|
"message": str,
|
||||||
|
"latency_ms": int,
|
||||||
|
"used_mode": str,
|
||||||
|
"result_count": int
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
latency_ms = int(doc.get("_latency_ms", -1))
|
||||||
|
used_mode = str(doc.get("used_mode", "unknown"))
|
||||||
|
results = doc.get("results", [])
|
||||||
|
|
||||||
|
if not isinstance(results, list):
|
||||||
|
return {
|
||||||
|
"status": "error",
|
||||||
|
"message": "results ist kein Array",
|
||||||
|
"latency_ms": latency_ms,
|
||||||
|
"used_mode": used_mode,
|
||||||
|
"result_count": -1,
|
||||||
|
}
|
||||||
|
|
||||||
|
result_count = len(results)
|
||||||
|
|
||||||
|
if result_count < min_results:
|
||||||
|
# Zu wenige Ergebnisse sind aus Sicht eines Health-Checks eher ein Error,
|
||||||
|
# da die Retrieval-Qualität nicht gewährleistet ist.
|
||||||
|
return {
|
||||||
|
"status": "error",
|
||||||
|
"message": f"zu wenige Ergebnisse: {result_count} < {min_results}",
|
||||||
|
"latency_ms": latency_ms,
|
||||||
|
"used_mode": used_mode,
|
||||||
|
"result_count": result_count,
|
||||||
|
}
|
||||||
|
|
||||||
|
sample_msg = "OK"
|
||||||
|
if result_count > 0:
|
||||||
|
first = results[0]
|
||||||
|
if not isinstance(first, dict):
|
||||||
|
sample_msg = "first result ist kein Objekt"
|
||||||
|
else:
|
||||||
|
# einfache Plausibilitätschecks (nur Zugriff, kein strikter Typcheck)
|
||||||
|
_ = first.get("note_id")
|
||||||
|
_ = first.get("chunk_id")
|
||||||
|
_ = first.get("total_score")
|
||||||
|
|
||||||
|
status = "ok" if sample_msg == "OK" else "warning"
|
||||||
|
return {
|
||||||
|
"status": status,
|
||||||
|
"message": sample_msg,
|
||||||
|
"latency_ms": latency_ms,
|
||||||
|
"used_mode": used_mode,
|
||||||
|
"result_count": result_count,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def run_health_check(
|
||||||
|
url: str,
|
||||||
|
query: str,
|
||||||
|
top_k: int,
|
||||||
|
timeout: float,
|
||||||
|
modes: List[str],
|
||||||
|
min_results: int,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Führt den Health-Check für die angegebenen Modi aus.
|
||||||
|
"""
|
||||||
|
report: Dict[str, Any] = {
|
||||||
|
"url": url,
|
||||||
|
"query": query,
|
||||||
|
"top_k": top_k,
|
||||||
|
"timeout_s": timeout,
|
||||||
|
"checks": [],
|
||||||
|
"overall_status": "ok", # wird unten ggf. auf warning oder error gesetzt
|
||||||
|
}
|
||||||
|
|
||||||
|
overall_status = "ok"
|
||||||
|
|
||||||
|
for mode in modes:
|
||||||
|
entry: Dict[str, Any] = {
|
||||||
|
"mode": mode,
|
||||||
|
"status": "error",
|
||||||
|
"message": "",
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
doc = _post_query(url=url, mode=mode, query=query, top_k=top_k, timeout=timeout)
|
||||||
|
validation = _validate_response(mode=mode, doc=doc, min_results=min_results)
|
||||||
|
entry.update(validation)
|
||||||
|
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
entry["status"] = "error"
|
||||||
|
entry["message"] = f"HTTP-Fehler: {exc}"
|
||||||
|
entry["latency_ms"] = -1
|
||||||
|
entry["used_mode"] = mode
|
||||||
|
entry["result_count"] = -1
|
||||||
|
|
||||||
|
except urllib.error.URLError as exc:
|
||||||
|
# URLError kann u.a. socket.timeout enthalten.
|
||||||
|
lat = -1
|
||||||
|
entry["latency_ms"] = lat
|
||||||
|
entry["used_mode"] = mode
|
||||||
|
entry["result_count"] = -1
|
||||||
|
|
||||||
|
if isinstance(exc.reason, socket.timeout):
|
||||||
|
entry["status"] = "warning"
|
||||||
|
entry["message"] = f"Timeout (URLError/socket.timeout): {exc}"
|
||||||
|
else:
|
||||||
|
entry["status"] = "error"
|
||||||
|
entry["message"] = f"HTTP/Netzwerkfehler: {exc}"
|
||||||
|
|
||||||
|
except socket.timeout as exc:
|
||||||
|
# Direkter Timeout (falls nicht in URLError verpackt)
|
||||||
|
entry["status"] = "warning"
|
||||||
|
entry["message"] = f"Timeout (socket.timeout): {exc}"
|
||||||
|
entry["latency_ms"] = -1
|
||||||
|
entry["used_mode"] = mode
|
||||||
|
entry["result_count"] = -1
|
||||||
|
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
entry["status"] = "error"
|
||||||
|
entry["message"] = f"unerwarteter Fehler: {exc}"
|
||||||
|
entry["latency_ms"] = -1
|
||||||
|
entry["used_mode"] = mode
|
||||||
|
entry["result_count"] = -1
|
||||||
|
|
||||||
|
# overall_status bestimmen:
|
||||||
|
# - error > warning > ok
|
||||||
|
if entry["status"] == "error":
|
||||||
|
overall_status = "error"
|
||||||
|
elif entry["status"] == "warning" and overall_status == "ok":
|
||||||
|
overall_status = "warning"
|
||||||
|
|
||||||
|
report["checks"].append(entry)
|
||||||
|
|
||||||
|
report["overall_status"] = overall_status
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: List[str]) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Health-Check für mindnet /query-Retriever",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--url",
|
||||||
|
default="http://127.0.0.1:8001/query",
|
||||||
|
help="Vollständige URL des /query-Endpunkts (Default: %(default)s)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--query",
|
||||||
|
default="embeddings",
|
||||||
|
help="Test-Query für den Health-Check (Default: %(default)s)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--top-k",
|
||||||
|
type=int,
|
||||||
|
default=3,
|
||||||
|
help="Top-K für die Testabfrage (Default: %(default)s)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--timeout",
|
||||||
|
type=float,
|
||||||
|
default=5.0,
|
||||||
|
help="Timeout in Sekunden pro Request (Default: %(default)s)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--modes",
|
||||||
|
nargs="+",
|
||||||
|
default=["semantic", "hybrid"],
|
||||||
|
help="Zu prüfende Modi (Default: semantic hybrid)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--min-results",
|
||||||
|
type=int,
|
||||||
|
default=1,
|
||||||
|
help="Minimale Anzahl erwarteter Ergebnisse (Default: %(default)s)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--strict",
|
||||||
|
action="store_true",
|
||||||
|
help="Warnings als Fehler behandeln (overall_status=warning führt zu Exit-Code 1)",
|
||||||
|
)
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: List[str]) -> int:
|
||||||
|
args = parse_args(argv)
|
||||||
|
|
||||||
|
report = run_health_check(
|
||||||
|
url=args.url,
|
||||||
|
query=args.query,
|
||||||
|
top_k=args.top_k,
|
||||||
|
timeout=args.timeout,
|
||||||
|
modes=args.modes,
|
||||||
|
min_results=args.min_results,
|
||||||
|
)
|
||||||
|
|
||||||
|
overall = report.get("overall_status", "error")
|
||||||
|
|
||||||
|
# Menschlich lesbare Zusammenfassung:
|
||||||
|
print(f"mindnet health: {overall}")
|
||||||
|
for c in report["checks"]:
|
||||||
|
mode = c.get("mode")
|
||||||
|
status = c.get("status")
|
||||||
|
latency = c.get("latency_ms")
|
||||||
|
result_count = c.get("result_count")
|
||||||
|
msg = c.get("message", "")
|
||||||
|
print(f" - {mode}: {status} (latency={latency} ms, results={result_count}) {msg}")
|
||||||
|
|
||||||
|
# JSON-Ausgabe (z.B. für n8n):
|
||||||
|
print(json.dumps(report, ensure_ascii=False))
|
||||||
|
|
||||||
|
if args.strict:
|
||||||
|
# strict: warning wird wie error behandelt
|
||||||
|
return 0 if overall == "ok" else 1
|
||||||
|
else:
|
||||||
|
# tolerant: nur echte errors führen zu Exit-Code 1
|
||||||
|
return 0 if overall in ("ok", "warning") else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main(sys.argv[1:]))
|
||||||
+468
-145
@@ -1,62 +1,79 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Script: import_markdown.py — Markdown → Qdrant (Notes, Chunks, Edges)
|
scripts/import_markdown.py
|
||||||
Version: 3.2.0
|
|
||||||
Datum: 2025-09-09
|
|
||||||
|
|
||||||
Kurzbeschreibung
|
Zweck
|
||||||
----------------
|
-----
|
||||||
Liest Markdown-Dateien aus einem Vault ein und schreibt Notes, Chunks und Edges
|
- Liest Markdown-Notizen aus einem Vault ein
|
||||||
idempotent nach Qdrant. Change-Detection basiert standardmäßig auf dem **Body-Hash**.
|
- Erzeugt Note-Payload, Chunk-Payloads (+ optionale Embeddings) und Edges
|
||||||
Neu: Hash-Modus und Normalisierung sind auch per **CLI** steuerbar.
|
- Schreibt alles idempotent in Qdrant (Notes, Chunks, Edges)
|
||||||
|
- Integriert eine optionale Type-Registry (types.yaml), um z. B. chunk_profile
|
||||||
|
und retriever_weight pro Notiz-Typ zu steuern.
|
||||||
|
|
||||||
ENV / Qdrant
|
Wesentliche Fixes ggü. vorherigen fehlerhaften Ständen
|
||||||
|
------------------------------------------------------
|
||||||
|
- `embed_texts` wird optional importiert und defensiv geprüft (kein NameError mehr)
|
||||||
|
- `effective_chunk_profile` / `effective_retriever_weight` und Registry-Helfer
|
||||||
|
sind VOR `main()` definiert (kein NameError mehr)
|
||||||
|
- `retriever_weight` wird in Note- und Chunk-Payload zuverlässig gesetzt
|
||||||
|
- Robuste Kantenbildung; Fehler bei Edges blockieren Notes/Chunks nicht
|
||||||
|
- Korrekte Verwendung von `scroll_filter` beim Qdrant-Client
|
||||||
|
- `--purge-before-upsert` entfernt alte Chunks/Edges einer Note vor dem Upsert
|
||||||
|
|
||||||
|
Qdrant / ENV
|
||||||
------------
|
------------
|
||||||
- QDRANT_URL (oder QDRANT_HOST/QDRANT_PORT)
|
- QDRANT_URL | QDRANT_HOST/QDRANT_PORT | QDRANT_API_KEY
|
||||||
- QDRANT_API_KEY (optional)
|
- COLLECTION_PREFIX (Default: mindnet), via --prefix überschreibbar
|
||||||
- COLLECTION_PREFIX (Default: mindnet)
|
|
||||||
- VECTOR_DIM (Default: 384)
|
- VECTOR_DIM (Default: 384)
|
||||||
- MINDNET_HASH_MODE: "body" (Default) | "frontmatter" | "body+frontmatter"
|
- MINDNET_NOTE_SCOPE_REFS: true|false (Default: false)
|
||||||
- MINDNET_HASH_NORMALIZE: "canonical" (Default) | "none"
|
- MINDNET_TYPES_FILE: Pfad zu types.yaml (optional; Default: ./types.yaml)
|
||||||
|
|
||||||
CLI (übersteuert ENV)
|
Beispiele
|
||||||
---------------------
|
---------
|
||||||
--hash-mode body|frontmatter|body+frontmatter
|
# Standard (Body, parsed, canonical)
|
||||||
--hash-normalize canonical|none
|
|
||||||
|
|
||||||
Aufruf
|
|
||||||
------
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault
|
python3 -m scripts.import_markdown --vault ./vault
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply --purge-before-upsert
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault --note-id 20250821-foo --apply
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault --apply --embed-note
|
|
||||||
# Feingranulare Erkennung (jede Kleinigkeit im Body zählt):
|
|
||||||
python3 -m scripts.import_markdown --vault ./vault --hash-normalize none
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
# Erstimport nach truncate (Create-Fall)
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --apply --purge-before-upsert
|
||||||
|
|
||||||
|
# Nur eine Datei (Diagnose)
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --only-path ./vault/30_projects/project-demo.md --apply
|
||||||
|
|
||||||
|
# Sync-Deletes (Dry-Run → Apply)
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --sync-deletes
|
||||||
|
python3 -m scripts.import_markdown --vault ./vault --sync-deletes --apply
|
||||||
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from typing import Dict, List, Optional, Tuple
|
from typing import Dict, List, Optional, Tuple, Any, Set
|
||||||
|
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
from qdrant_client.http import models as rest
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
# --- Projekt-Imports ---
|
||||||
from app.core.parser import (
|
from app.core.parser import (
|
||||||
read_markdown,
|
read_markdown,
|
||||||
normalize_frontmatter,
|
normalize_frontmatter,
|
||||||
validate_required_frontmatter,
|
validate_required_frontmatter,
|
||||||
extract_wikilinks,
|
|
||||||
)
|
)
|
||||||
from app.core.note_payload import make_note_payload
|
from app.core.note_payload import make_note_payload
|
||||||
from app.core.chunker import assemble_chunks
|
from app.core.chunker import assemble_chunks
|
||||||
from app.core.chunk_payload import make_chunk_payloads
|
from app.core.chunk_payload import make_chunk_payloads
|
||||||
from app.core.qdrant import QdrantConfig, get_client, ensure_collections
|
try:
|
||||||
|
from app.core.derive_edges import build_edges_for_note
|
||||||
|
except Exception: # pragma: no cover
|
||||||
|
from app.core.edges import build_edges_for_note # type: ignore
|
||||||
|
from app.core.qdrant import (
|
||||||
|
QdrantConfig,
|
||||||
|
get_client,
|
||||||
|
ensure_collections,
|
||||||
|
ensure_payload_indexes,
|
||||||
|
)
|
||||||
from app.core.qdrant_points import (
|
from app.core.qdrant_points import (
|
||||||
points_for_chunks,
|
points_for_chunks,
|
||||||
points_for_note,
|
points_for_note,
|
||||||
@@ -64,16 +81,111 @@ from app.core.qdrant_points import (
|
|||||||
upsert_batch,
|
upsert_batch,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# embeddings sind optional (z. B. im reinen Payload-Backfill)
|
||||||
try:
|
try:
|
||||||
from app.core.embed import embed_texts, embed_one # optional
|
from app.core.embed import embed_texts # optional
|
||||||
except Exception:
|
except Exception: # pragma: no cover
|
||||||
embed_texts = None
|
embed_texts = None
|
||||||
embed_one = None
|
|
||||||
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
# Hilfsfunktionen
|
# Type-Registry (types.yaml) – Helper (robust, optional)
|
||||||
# -----------------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _env(name: str, default: Optional[str] = None) -> Optional[str]:
|
||||||
|
v = os.getenv(name)
|
||||||
|
return v if v is not None else default
|
||||||
|
|
||||||
|
def _load_json_or_yaml(path: str) -> dict:
|
||||||
|
import io
|
||||||
|
data: dict = {}
|
||||||
|
if not path or not os.path.exists(path):
|
||||||
|
return data
|
||||||
|
try:
|
||||||
|
import yaml # type: ignore
|
||||||
|
with io.open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
return {}
|
||||||
|
return data
|
||||||
|
except Exception:
|
||||||
|
# YAML evtl. nicht installiert – versuche JSON
|
||||||
|
try:
|
||||||
|
with io.open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
return {}
|
||||||
|
return data
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def load_type_registry() -> dict:
|
||||||
|
# Reihenfolge: ENV > ./types.yaml (im aktuellen Arbeitsverzeichnis)
|
||||||
|
p = _env("MINDNET_TYPES_FILE", None)
|
||||||
|
if p and os.path.exists(p):
|
||||||
|
return _load_json_or_yaml(p)
|
||||||
|
fallback = os.path.abspath("./config/types.yaml") if os.path.exists("./config/types.yaml") else os.path.abspath("./types.yaml")
|
||||||
|
return _load_json_or_yaml(fallback)
|
||||||
|
|
||||||
|
def get_type_config(note_type: Optional[str], reg: dict) -> dict:
|
||||||
|
if not reg or not isinstance(reg, dict):
|
||||||
|
return {}
|
||||||
|
types = reg.get("types", {}) if isinstance(reg.get("types"), dict) else {}
|
||||||
|
if note_type and isinstance(note_type, str) and note_type in types:
|
||||||
|
return types[note_type] or {}
|
||||||
|
# Fallback: concept
|
||||||
|
return types.get("concept", {}) or {}
|
||||||
|
|
||||||
|
def resolve_note_type(requested: Optional[str], reg: dict) -> str:
|
||||||
|
if requested and isinstance(requested, str):
|
||||||
|
return requested
|
||||||
|
# Fallback wenn nichts gesetzt ist
|
||||||
|
types = reg.get("types", {}) if isinstance(reg.get("types"), dict) else {}
|
||||||
|
return "concept" if "concept" in types else (requested or "concept")
|
||||||
|
|
||||||
|
def effective_chunk_profile(note_type: str, reg: dict) -> Optional[str]:
|
||||||
|
"""Resolve chunk_profile for type or from defaults/global.
|
||||||
|
Accepts symbolic profiles: short|medium|long|default.
|
||||||
|
"""
|
||||||
|
cfg = get_type_config(note_type, reg)
|
||||||
|
prof = (cfg.get("chunk_profile") if isinstance(cfg, dict) else None)
|
||||||
|
if isinstance(prof, str) and prof:
|
||||||
|
return prof
|
||||||
|
# defaults fallbacks
|
||||||
|
for key in ("defaults", "default", "global"):
|
||||||
|
dcfg = reg.get(key) if isinstance(reg, dict) else None
|
||||||
|
if isinstance(dcfg, dict):
|
||||||
|
dprof = dcfg.get("chunk_profile")
|
||||||
|
if isinstance(dprof, str) and dprof:
|
||||||
|
return dprof
|
||||||
|
return "default"
|
||||||
|
|
||||||
|
def effective_retriever_weight(note_type: str, reg: dict) -> Optional[float]:
|
||||||
|
"""Resolve retriever_weight for type or defaults; returns float.
|
||||||
|
"""
|
||||||
|
cfg = get_type_config(note_type, reg)
|
||||||
|
w = (cfg.get("retriever_weight") if isinstance(cfg, dict) else None)
|
||||||
|
try:
|
||||||
|
if w is not None:
|
||||||
|
return float(w)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# defaults fallbacks
|
||||||
|
for key in ("defaults", "default", "global"):
|
||||||
|
dcfg = reg.get(key) if isinstance(reg, dict) else None
|
||||||
|
if isinstance(dcfg, dict):
|
||||||
|
dw = dcfg.get("retriever_weight")
|
||||||
|
try:
|
||||||
|
if dw is not None:
|
||||||
|
return float(dw)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return 1.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
# Sonstige Helper
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
def iter_md(root: str) -> List[str]:
|
def iter_md(root: str) -> List[str]:
|
||||||
out: List[str] = []
|
out: List[str] = []
|
||||||
@@ -91,177 +203,388 @@ def iter_md(root: str) -> List[str]:
|
|||||||
def collections(prefix: str) -> Tuple[str, str, str]:
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
def fetch_existing_note_hash(client, prefix: str, note_id: str) -> Optional[str]:
|
def fetch_existing_note_payload(client, prefix: str, note_id: str) -> Optional[Dict]:
|
||||||
notes_col, _, _ = collections(prefix)
|
notes_col, _, _ = collections(prefix)
|
||||||
f = rest.Filter(must=[rest.FieldCondition(
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
key="note_id",
|
|
||||||
match=rest.MatchValue(value=note_id),
|
|
||||||
)])
|
|
||||||
points, _ = client.scroll(
|
points, _ = client.scroll(
|
||||||
collection_name=notes_col,
|
collection_name=notes_col,
|
||||||
scroll_filter=f,
|
scroll_filter=f, # wichtig: scroll_filter (nicht: filter)
|
||||||
with_payload=True,
|
with_payload=True,
|
||||||
with_vectors=False,
|
with_vectors=False,
|
||||||
limit=1,
|
limit=1,
|
||||||
)
|
)
|
||||||
if not points:
|
if not points:
|
||||||
return None
|
return None
|
||||||
pl = points[0].payload or {}
|
return points[0].payload or {}
|
||||||
return pl.get("hash_fulltext")
|
|
||||||
|
def list_qdrant_note_ids(client, prefix: str) -> Set[str]:
|
||||||
|
notes_col, _, _ = collections(prefix)
|
||||||
|
out: Set[str] = set()
|
||||||
|
next_page = None
|
||||||
|
while True:
|
||||||
|
pts, next_page = client.scroll(
|
||||||
|
collection_name=notes_col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=256,
|
||||||
|
offset=next_page,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
for p in pts:
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
if isinstance(nid, str):
|
||||||
|
out.add(nid)
|
||||||
|
if next_page is None:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
def purge_note_artifacts(client, prefix: str, note_id: str) -> None:
|
def purge_note_artifacts(client, prefix: str, note_id: str) -> None:
|
||||||
_, chunks_col, edges_col = collections(prefix)
|
_, chunks_col, edges_col = collections(prefix)
|
||||||
f_chunks = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
filt = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
client.delete(collection_name=chunks_col, points_selector=f_chunks, wait=True)
|
for col in (chunks_col, edges_col):
|
||||||
should = [
|
try:
|
||||||
rest.FieldCondition(key="source_id", match=rest.MatchText(text=f"{note_id}#")),
|
client.delete(
|
||||||
rest.FieldCondition(key="target_id", match=rest.MatchText(text=f"{note_id}#")),
|
collection_name=col,
|
||||||
rest.FieldCondition(key="source_id", match=rest.MatchValue(value=note_id)),
|
points_selector=rest.FilterSelector(filter=filt),
|
||||||
rest.FieldCondition(key="target_id", match=rest.MatchValue(value=note_id)),
|
wait=True
|
||||||
]
|
)
|
||||||
f_edges = rest.Filter(should=should)
|
except Exception as e:
|
||||||
client.delete(collection_name=edges_col, points_selector=f_edges, wait=True)
|
print(json.dumps({"note_id": note_id, "warn": f"delete in {col} via filter failed: {e}"}))
|
||||||
|
|
||||||
def _normalize_rel_path(abs_path: str, vault_root: str) -> str:
|
def delete_note_everywhere(client, prefix: str, note_id: str) -> None:
|
||||||
try:
|
notes_col, chunks_col, edges_col = collections(prefix)
|
||||||
rel = os.path.relpath(abs_path, vault_root)
|
filt = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
except Exception:
|
for col in (edges_col, chunks_col, notes_col):
|
||||||
rel = abs_path
|
try:
|
||||||
return rel.replace("\\", "/").lstrip("/")
|
client.delete(
|
||||||
|
collection_name=col,
|
||||||
|
points_selector=rest.FilterSelector(filter=filt),
|
||||||
|
wait=True
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"note_id": note_id, "warn": f"delete in {col} failed: {e}"}))
|
||||||
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# --- Neu: Existenz-Checks für Artefakte (fehlertoleranter Rebuild) ---
|
||||||
|
|
||||||
|
def _has_any_point(client, collection: str, note_id: str) -> bool:
|
||||||
|
"""Prüft, ob es mind. einen Punkt mit note_id in der Collection gibt."""
|
||||||
|
filt = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
pts, _ = client.scroll(
|
||||||
|
collection_name=collection,
|
||||||
|
scroll_filter=filt,
|
||||||
|
with_payload=False,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=1,
|
||||||
|
)
|
||||||
|
return bool(pts)
|
||||||
|
|
||||||
|
def artifacts_missing(client, prefix: str, note_id: str) -> Tuple[bool, bool]:
|
||||||
|
"""Gibt (chunks_missing, edges_missing) zurück."""
|
||||||
|
_, chunks_col, edges_col = collections(prefix)
|
||||||
|
chunks_missing = not _has_any_point(client, chunks_col, note_id)
|
||||||
|
edges_missing = not _has_any_point(client, edges_col, note_id)
|
||||||
|
return chunks_missing, edges_missing
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------
|
||||||
# Main
|
# Main
|
||||||
# -----------------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _resolve_mode(m: Optional[str]) -> str:
|
||||||
|
m = (m or "body").strip().lower()
|
||||||
|
return m if m in {"body", "frontmatter", "full"} else "body"
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
load_dotenv()
|
load_dotenv()
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--vault", required=True, help="Pfad zum Obsidian-Vault (Root-Ordner)")
|
ap = argparse.ArgumentParser(
|
||||||
ap.add_argument("--apply", action="store_true", help="Schreibt in Qdrant; ohne Flag nur Dry-Run")
|
prog="scripts.import_markdown",
|
||||||
ap.add_argument("--purge-before-upsert", action="store_true",
|
description="Importiert Markdown-Notizen in Qdrant (Notes/Chunks/Edges)."
|
||||||
help="Vor Upsert Chunks & Edges der GEÄNDERTEN Note löschen")
|
)
|
||||||
ap.add_argument("--note-id", help="Nur eine bestimmte Note-ID verarbeiten")
|
ap.add_argument("--vault", required=True, help="Pfad zum Vault (Ordner mit .md-Dateien)")
|
||||||
ap.add_argument("--embed-note", action="store_true", help="Optional: Note-Volltext einbetten")
|
ap.add_argument("--only-path", help="Nur diese Datei verarbeiten (absolut oder relativ)")
|
||||||
ap.add_argument("--force-replace", action="store_true",
|
ap.add_argument("--apply", action="store_true", help="Schreibt nach Qdrant (sonst Dry-Run)")
|
||||||
help="Änderungserkennung ignorieren und immer upserten (+ optional Purge)")
|
ap.add_argument("--purge-before-upsert", action="store_true", help="Alte Chunks/Edges der Note vorher löschen")
|
||||||
# NEU: Hash-Steuerung per CLI
|
ap.add_argument("--force-replace", action="store_true", help="Note/Chunks/Edges unabhängig von Hash neu schreiben")
|
||||||
ap.add_argument("--hash-mode", choices=["body", "frontmatter", "body+frontmatter"], default=None)
|
ap.add_argument("--note-id", help="Nur Notes mit dieser ID verarbeiten (Filter)")
|
||||||
ap.add_argument("--hash-normalize", choices=["canonical", "none"], default=None)
|
ap.add_argument("--note-scope-refs", action="store_true", help="Note-scope References/Backlinks erzeugen")
|
||||||
|
ap.add_argument("--hash-mode", help="body|frontmatter|full (Default body)")
|
||||||
|
ap.add_argument("--hash-source", help="parsed|raw (Default parsed)")
|
||||||
|
ap.add_argument("--hash-normalize", help="canonical|none (Default canonical)")
|
||||||
|
ap.add_argument("--compare-text", action="store_true", help="Parsed fulltext zusätzlich direkt vergleichen")
|
||||||
|
ap.add_argument("--baseline-modes", action="store_true", help="Fehlende Hash-Varianten still nachtragen (Notes)")
|
||||||
|
ap.add_argument("--sync-deletes", action="store_true", help="Qdrant->Vault Lösch-Sync (Dry-Run; mit --apply ausführen)")
|
||||||
|
ap.add_argument("--prefix", help="Collection-Prefix (überschreibt ENV COLLECTION_PREFIX)")
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
mode = _resolve_mode(args.hash_mode) # body|frontmatter|full
|
||||||
|
src = _env("MINDNET_HASH_SOURCE", args.hash_source or "parsed") # parsed|raw
|
||||||
|
norm = _env("MINDNET_HASH_NORMALIZE", args.hash_normalize or "canonical") # canonical|none
|
||||||
|
note_scope_refs_env = (_env("MINDNET_NOTE_SCOPE_REFS", "false") == "true")
|
||||||
|
note_scope_refs = args.note_scope_refs or note_scope_refs_env
|
||||||
|
compare_text = args.compare_text or (_env("MINDNET_COMPARE_TEXT", "false") == "true")
|
||||||
|
|
||||||
|
# Qdrant
|
||||||
cfg = QdrantConfig.from_env()
|
cfg = QdrantConfig.from_env()
|
||||||
|
if args.prefix:
|
||||||
|
cfg.prefix = args.prefix.strip()
|
||||||
client = get_client(cfg)
|
client = get_client(cfg)
|
||||||
ensure_collections(client, cfg.prefix, cfg.dim)
|
ensure_collections(client, cfg.prefix, cfg.dim)
|
||||||
notes_col, chunks_col, edges_col = collections(cfg.prefix)
|
ensure_payload_indexes(client, cfg.prefix)
|
||||||
|
|
||||||
|
# Type-Registry laden (optional)
|
||||||
|
reg = load_type_registry()
|
||||||
|
|
||||||
root = os.path.abspath(args.vault)
|
root = os.path.abspath(args.vault)
|
||||||
files = iter_md(root)
|
|
||||||
|
# Dateiliste
|
||||||
|
if args.only_path:
|
||||||
|
only = os.path.abspath(args.only_path)
|
||||||
|
files = [only]
|
||||||
|
else:
|
||||||
|
files = iter_md(root)
|
||||||
if not files:
|
if not files:
|
||||||
print("Keine Markdown-Dateien gefunden.", file=sys.stderr)
|
print("Keine Markdown-Dateien gefunden.", file=sys.stderr)
|
||||||
sys.exit(2)
|
sys.exit(2)
|
||||||
|
|
||||||
|
# Optional: Sync-Deletes vorab
|
||||||
|
if args.sync_deletes:
|
||||||
|
vault_note_ids: Set[str] = set()
|
||||||
|
for path in files:
|
||||||
|
try:
|
||||||
|
parsed = read_markdown(path)
|
||||||
|
if not parsed:
|
||||||
|
continue
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter)
|
||||||
|
nid = fm.get("id")
|
||||||
|
if isinstance(nid, str):
|
||||||
|
vault_note_ids.add(nid)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
qdrant_note_ids = list_qdrant_note_ids(client, cfg.prefix)
|
||||||
|
to_delete = sorted(qdrant_note_ids - vault_note_ids)
|
||||||
|
print(json.dumps({
|
||||||
|
"action": "sync-deletes",
|
||||||
|
"prefix": cfg.prefix,
|
||||||
|
"qdrant_total": len(qdrant_note_ids),
|
||||||
|
"vault_total": len(vault_note_ids),
|
||||||
|
"to_delete_count": len(to_delete),
|
||||||
|
"to_delete": to_delete[:50] + (["…"] if len(to_delete) > 50 else [])
|
||||||
|
}, ensure_ascii=False))
|
||||||
|
if args.apply and to_delete:
|
||||||
|
for nid in to_delete:
|
||||||
|
print(json.dumps({"action": "delete", "note_id": nid, "decision": "apply"}))
|
||||||
|
delete_note_everywhere(client, cfg.prefix, nid)
|
||||||
|
|
||||||
|
key_current = f"{mode}:{src}:{norm}"
|
||||||
|
|
||||||
processed = 0
|
processed = 0
|
||||||
for path in files:
|
for path in files:
|
||||||
parsed = read_markdown(path)
|
|
||||||
fm = normalize_frontmatter(parsed.frontmatter)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
validate_required_frontmatter(fm)
|
parsed = read_markdown(path)
|
||||||
|
if not parsed:
|
||||||
|
continue
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(json.dumps({"path": path, "error": f"Frontmatter invalid: {e}"}))
|
print(json.dumps({"path": path, "error": f"read_markdown failed: {type(e).__name__}: {e}"}))
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if args.note_id and fm.get("id") != args.note_id:
|
# --- Frontmatter prüfen ---
|
||||||
|
try:
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter)
|
||||||
|
validate_required_frontmatter(fm)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": path, "error": f"Frontmatter invalid: {type(e).__name__}: {e}"}))
|
||||||
|
continue
|
||||||
|
|
||||||
|
if args.note_id and not args.only_path and fm.get("id") != args.note_id:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
processed += 1
|
processed += 1
|
||||||
|
|
||||||
# Note-Payload (mit expliziten Hash-Parametern)
|
# --- Type-Registry anwenden (chunk_profile / retriever_weight) ---
|
||||||
note_pl = make_note_payload(parsed, vault_root=root,
|
try:
|
||||||
hash_mode=args.hash_mode, hash_normalize=args.hash_normalize)
|
note_type = resolve_note_type(fm.get("type"), reg)
|
||||||
if "fulltext" not in (note_pl or {}):
|
except Exception:
|
||||||
note_pl["fulltext"] = parsed.body or ""
|
note_type = (fm.get("type") or "concept")
|
||||||
if note_pl.get("path"):
|
fm["type"] = note_type or fm.get("type") or "concept"
|
||||||
note_pl["path"] = _normalize_rel_path(
|
|
||||||
os.path.join(root, note_pl["path"]) if not os.path.isabs(note_pl["path"]) else note_pl["path"], root
|
prof = effective_chunk_profile(note_type, reg)
|
||||||
|
if prof:
|
||||||
|
fm["chunk_profile"] = prof
|
||||||
|
|
||||||
|
weight = effective_retriever_weight(note_type, reg)
|
||||||
|
if weight is not None:
|
||||||
|
try:
|
||||||
|
fm["retriever_weight"] = float(weight)
|
||||||
|
except Exception:
|
||||||
|
pass # falls FM string-inkonsistent ist
|
||||||
|
|
||||||
|
# --- Payload aufbauen (inkl. Hashes) ---
|
||||||
|
try:
|
||||||
|
note_pl = make_note_payload(
|
||||||
|
parsed,
|
||||||
|
vault_root=root,
|
||||||
|
hash_mode=mode,
|
||||||
|
hash_normalize=norm,
|
||||||
|
hash_source=src,
|
||||||
|
file_path=path,
|
||||||
)
|
)
|
||||||
else:
|
except Exception as e:
|
||||||
note_pl["path"] = _normalize_rel_path(parsed.path, root)
|
print(json.dumps({"path": path, "error": f"make_note_payload failed: {type(e).__name__}: {e}"}))
|
||||||
|
continue
|
||||||
|
|
||||||
note_id = note_pl["note_id"]
|
if not note_pl.get("fulltext"):
|
||||||
|
note_pl["fulltext"] = getattr(parsed, "body", "") or ""
|
||||||
|
|
||||||
# Change-Detection
|
# retriever_weight sicher in Note-Payload spiegeln (für spätere Filter)
|
||||||
new_hash = note_pl.get("hash_fulltext")
|
if "retriever_weight" not in note_pl and fm.get("retriever_weight") is not None:
|
||||||
old_hash = None if args.force_replace else fetch_existing_note_hash(client, cfg.prefix, note_id)
|
try:
|
||||||
changed = args.force_replace or (old_hash != new_hash)
|
note_pl["retriever_weight"] = float(fm.get("retriever_weight"))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
# Chunks + Embeddings (Nullvektor-Fallback)
|
note_id = note_pl.get("note_id") or fm.get("id")
|
||||||
chunks = assemble_chunks(fm["id"], parsed.body, fm.get("type", "concept"))
|
if not note_id:
|
||||||
chunk_pls = make_chunk_payloads(fm, note_pl["path"], chunks)
|
print(json.dumps({"path": path, "error": "Missing note_id after payload build"}))
|
||||||
if embed_texts:
|
continue
|
||||||
vecs = embed_texts([getattr(c, "text", "") for c in chunks])
|
|
||||||
else:
|
|
||||||
vecs = [[0.0] * cfg.dim for _ in chunks]
|
|
||||||
|
|
||||||
# Edges (leichtgewichtig direkt hier ableiten)
|
# --- bestehenden Payload laden (zum Diff) ---
|
||||||
# belongs_to / prev/next aus Chunk-Nachbarschaften + Wikilinks als references
|
old_payload = None if args.force_replace else fetch_existing_note_payload(client, cfg.prefix, note_id)
|
||||||
edges: List[Dict] = []
|
has_old = old_payload is not None
|
||||||
for ch in chunk_pls:
|
|
||||||
cid = ch["id"]
|
|
||||||
edges.append({"src_id": cid, "dst_id": note_id, "edge_type": "belongs_to", "scope": "chunk"})
|
|
||||||
nb = ch.get("neighbors") or {}
|
|
||||||
if nb.get("prev"):
|
|
||||||
edges.append({"src_id": nb["prev"], "dst_id": cid, "edge_type": "next", "scope": "chunk"})
|
|
||||||
edges.append({"src_id": cid, "dst_id": nb["prev"], "edge_type": "prev", "scope": "chunk"})
|
|
||||||
if nb.get("next"):
|
|
||||||
edges.append({"src_id": cid, "dst_id": nb["next"], "edge_type": "next", "scope": "chunk"})
|
|
||||||
edges.append({"src_id": nb["next"], "dst_id": cid, "edge_type": "prev", "scope": "chunk"})
|
|
||||||
for ref in (ch.get("references") or []):
|
|
||||||
tid = ref.get("target_id")
|
|
||||||
if tid:
|
|
||||||
edges.append({"src_id": cid, "dst_id": tid, "edge_type": "references", "scope": "chunk"})
|
|
||||||
for tid in (note_pl.get("references") or []):
|
|
||||||
edges.append({"src_id": note_id, "dst_id": tid, "edge_type": "references", "scope": "note"})
|
|
||||||
edges.append({"src_id": tid, "dst_id": note_id, "edge_type": "backlink", "scope": "note"})
|
|
||||||
# Dedupe
|
|
||||||
_uniq = {}
|
|
||||||
for e in edges:
|
|
||||||
_uniq[(e["src_id"], e["dst_id"], e["edge_type"], e.get("scope", ""))] = e
|
|
||||||
edges = list(_uniq.values())
|
|
||||||
|
|
||||||
# Zusammenfassung
|
old_hashes = (old_payload or {}).get("hashes") or {}
|
||||||
|
old_hash_exact = old_hashes.get(key_current)
|
||||||
|
new_hash_exact = (note_pl.get("hashes") or {}).get(key_current)
|
||||||
|
needs_baseline = (old_hash_exact is None)
|
||||||
|
|
||||||
|
hash_changed = (old_hash_exact is not None and new_hash_exact is not None and old_hash_exact != new_hash_exact)
|
||||||
|
|
||||||
|
text_changed = False
|
||||||
|
if compare_text:
|
||||||
|
old_text = (old_payload or {}).get("fulltext") or ""
|
||||||
|
new_text = note_pl.get("fulltext") or ""
|
||||||
|
text_changed = (old_text != new_text)
|
||||||
|
|
||||||
|
changed = args.force_replace or (not has_old) or hash_changed or text_changed
|
||||||
|
do_baseline_only = (args.baseline_modes and has_old and needs_baseline and not changed)
|
||||||
|
|
||||||
|
# --- Chunks + Embeddings vorbereiten ---
|
||||||
|
try:
|
||||||
|
body_text = getattr(parsed, "body", "") or ""
|
||||||
|
chunks = assemble_chunks(fm["id"], body_text, fm.get("type", "concept"))
|
||||||
|
chunk_pls: List[Dict[str, Any]] = make_chunk_payloads(fm, note_pl["path"], chunks, note_text=body_text)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": path, "note_id": note_id, "error": f"chunk build failed: {type(e).__name__}: {e}"}))
|
||||||
|
continue
|
||||||
|
|
||||||
|
# retriever_weight auf Chunk-Payload spiegeln
|
||||||
|
if fm.get("retriever_weight") is not None:
|
||||||
|
try:
|
||||||
|
rw = float(fm.get("retriever_weight"))
|
||||||
|
for pl in chunk_pls:
|
||||||
|
# Feld nur setzen, wenn noch nicht vorhanden
|
||||||
|
if "retriever_weight" not in pl:
|
||||||
|
pl["retriever_weight"] = rw
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Embeddings (fallback: Nullvektoren)
|
||||||
|
vecs: List[List[float]] = [[0.0] * int(cfg.dim) for _ in chunk_pls]
|
||||||
|
if embed_texts and chunk_pls:
|
||||||
|
try:
|
||||||
|
texts_for_embed = [(pl.get("window") or pl.get("text") or "") for pl in chunk_pls]
|
||||||
|
vecs = embed_texts(texts_for_embed)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": path, "note_id": note_id, "warn": f"embed_texts failed, using zeros: {e}"}))
|
||||||
|
|
||||||
|
# --- Fehlende Artefakte in Qdrant ermitteln ---
|
||||||
|
chunks_missing, edges_missing = artifacts_missing(client, cfg.prefix, note_id)
|
||||||
|
|
||||||
|
# --- Edges (robust) ---
|
||||||
|
edges: List[Dict[str, Any]] = []
|
||||||
|
edges_failed = False
|
||||||
|
should_build_edges = (changed and (not do_baseline_only)) or edges_missing
|
||||||
|
if should_build_edges:
|
||||||
|
try:
|
||||||
|
note_refs = note_pl.get("references") or []
|
||||||
|
edges = build_edges_for_note(
|
||||||
|
note_id,
|
||||||
|
chunk_pls,
|
||||||
|
note_level_references=note_refs,
|
||||||
|
include_note_scope_refs=note_scope_refs,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
edges_failed = True
|
||||||
|
edges = []
|
||||||
|
print(json.dumps({"path": path, "note_id": note_id, "warn": f"build_edges_for_note failed, skipping edges: {type(e).__name__}: {e}"}))
|
||||||
|
|
||||||
|
# --- Summary (stdout) ---
|
||||||
summary = {
|
summary = {
|
||||||
"note_id": note_id,
|
"note_id": note_id,
|
||||||
"title": fm.get("title"),
|
"title": fm.get("title"),
|
||||||
"chunks": len(chunk_pls),
|
"chunks": len(chunk_pls),
|
||||||
"edges": len(edges),
|
"edges": len(edges),
|
||||||
|
"edges_failed": edges_failed,
|
||||||
"changed": changed,
|
"changed": changed,
|
||||||
"decision": ("apply" if args.apply and changed else
|
"chunks_missing": chunks_missing,
|
||||||
"apply-skip-unchanged" if args.apply and not changed else
|
"edges_missing": edges_missing,
|
||||||
|
"needs_baseline_for_mode": needs_baseline,
|
||||||
|
"decision": ("baseline-only" if args.apply and do_baseline_only else
|
||||||
|
"apply" if args.apply and (changed or chunks_missing or edges_missing) else
|
||||||
|
"apply-skip-unchanged" if args.apply and not (changed or chunks_missing or edges_missing) else
|
||||||
"dry-run"),
|
"dry-run"),
|
||||||
"path": note_pl["path"],
|
"path": note_pl["path"],
|
||||||
"hash_mode": args.hash_mode or os.environ.get("MINDNET_HASH_MODE", "body"),
|
"hash_mode": mode,
|
||||||
"hash_normalize": args.hash_normalize or os.environ.get("MINDNET_HASH_NORMALIZE", "canonical"),
|
"hash_normalize": norm,
|
||||||
"hash_old": old_hash,
|
"hash_source": src,
|
||||||
"hash_new": new_hash,
|
"prefix": cfg.prefix,
|
||||||
}
|
}
|
||||||
print(json.dumps(summary, ensure_ascii=False))
|
print(json.dumps(summary, ensure_ascii=False))
|
||||||
|
|
||||||
|
# --- Writes ---
|
||||||
if not args.apply:
|
if not args.apply:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if changed and args.purge_before_upsert:
|
if do_baseline_only:
|
||||||
purge_note_artifacts(client, cfg.prefix, note_id)
|
merged_hashes = {}
|
||||||
|
merged_hashes.update(old_hashes)
|
||||||
|
merged_hashes.update(note_pl.get("hashes") or {})
|
||||||
|
if old_payload:
|
||||||
|
note_pl["hash_fulltext"] = old_payload.get("hash_fulltext", note_pl.get("hash_fulltext"))
|
||||||
|
note_pl["hash_signature"] = old_payload.get("hash_signature", note_pl.get("hash_signature"))
|
||||||
|
note_pl["hashes"] = merged_hashes
|
||||||
|
notes_name, note_pts = points_for_note(cfg.prefix, note_pl, None, cfg.dim)
|
||||||
|
upsert_batch(client, notes_name, note_pts)
|
||||||
|
continue
|
||||||
|
|
||||||
notes_name, note_pts = points_for_note(cfg.prefix, note_pl, None, cfg.dim)
|
# Wenn nichts geändert und keine Artefakte fehlen → nichts zu tun
|
||||||
upsert_batch(client, notes_name, note_pts)
|
if not changed and not (chunks_missing or edges_missing):
|
||||||
|
continue
|
||||||
|
|
||||||
chunks_name, chunk_pts = points_for_chunks(cfg.prefix, chunk_pls, vecs)
|
# Purge nur bei echten Änderungen (unverändert + fehlende Artefakte ≠ Purge)
|
||||||
upsert_batch(client, chunks_name, chunk_pts)
|
if args.purge_before_upsert and has_old and changed:
|
||||||
|
try:
|
||||||
|
purge_note_artifacts(client, cfg.prefix, note_id)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": path, "note_id": note_id, "warn": f"purge failed: {e}"}))
|
||||||
|
|
||||||
edges_name, edge_pts = points_for_edges(cfg.prefix, edges)
|
# Note nur bei Änderungen neu schreiben
|
||||||
upsert_batch(client, edges_name, edge_pts)
|
if changed:
|
||||||
|
notes_name, note_pts = points_for_note(cfg.prefix, note_pl, None, cfg.dim)
|
||||||
|
upsert_batch(client, notes_name, note_pts)
|
||||||
|
|
||||||
|
# Chunks schreiben, wenn geändert ODER vorher fehlend
|
||||||
|
if chunk_pls and (changed or chunks_missing):
|
||||||
|
chunks_name, chunk_pts = points_for_chunks(cfg.prefix, chunk_pls, vecs)
|
||||||
|
upsert_batch(client, chunks_name, chunk_pts)
|
||||||
|
|
||||||
|
# Edges schreiben, wenn vorhanden und (geändert ODER vorher fehlend)
|
||||||
|
if edges and (changed or edges_missing):
|
||||||
|
edges_name, edge_pts = points_for_edges(cfg.prefix, edges)
|
||||||
|
upsert_batch(client, edges_name, edge_pts)
|
||||||
|
|
||||||
print(f"Done. Processed notes: {processed}")
|
print(f"Done. Processed notes: {processed}")
|
||||||
|
|
||||||
|
|||||||
+18
-14
@@ -3,31 +3,32 @@
|
|||||||
"""
|
"""
|
||||||
make_test_vault.py — erzeugt einen minimalen, nachvollziehbaren Test-Vault
|
make_test_vault.py — erzeugt einen minimalen, nachvollziehbaren Test-Vault
|
||||||
|
|
||||||
Version: 1.1 (2025-09-05)
|
Version: 1.2 (2025-09-09)
|
||||||
Änderungen ggü. 1.0:
|
Änderungen ggü. 1.1:
|
||||||
- created/updated werden als QUOTED ISO-Strings geschrieben, damit der YAML-Parser
|
- Fügt **externe Links** (http/mailto) in `exp-two.md` hinzu, damit der Chunk-Payload-Exporter
|
||||||
sie nicht in datetime-Objekte konvertiert (Schema verlangt strings).
|
`external_links` realistisch testen kann.
|
||||||
- Kommentare/Output gestrafft.
|
- Kleine Textanpassungen zur stabilen Chunk-Bildung (Absatzstruktur).
|
||||||
|
|
||||||
Zweck
|
Zweck
|
||||||
- Kleiner Obsidian-ähnlicher Vault zum Durchspielen des Importers (Chunks/Edges).
|
- Kleiner Obsidian-ähnlicher Vault zum Durchspielen des Importers (Chunks/Edges).
|
||||||
- Szenarien: leere Links, spätere Anlage fehlender Noten, Chunk-Neuaufteilung.
|
- Szenarien: leere Links, spätere Anlage fehlender Noten, Chunk-Neuaufteilung, externe Links.
|
||||||
|
|
||||||
Struktur (Default: ./test_vault)
|
Struktur (Default: ./test_vault)
|
||||||
- 40_concepts/concept-alpha.md
|
- 40_concepts/concept-alpha.md
|
||||||
- 20_experiences/exp-one.md (verlinkt auf [[concept-alpha]] und [[missing-note]])
|
- 20_experiences/exp-one.md (verlinkt auf [[concept-alpha]] und [[missing-note]])
|
||||||
- 20_experiences/exp-two.md (verlinkt auf [[concept-alpha]])
|
- 20_experiences/exp-two.md (verlinkt auf [[concept-alpha]] + externe Links)
|
||||||
- 30_projects/project-demo.md (verlinkt auf [[concept-alpha]] und [[exp-one]])
|
- 30_projects/project-demo.md (verlinkt auf [[concept-alpha]] und [[exp-one]])
|
||||||
|
|
||||||
Voraussetzungen
|
Voraussetzungen
|
||||||
- Keine. Rein Dateigenerierung.
|
- Keine. Rein Dateigenerierung.
|
||||||
|
|
||||||
Aufruf
|
Aufruf
|
||||||
python3 -m scripts.make_test_vault [--out ./test_vault] [--force]
|
python3 -m scripts.make_test_vault [--out ./test_vault] [--force] [--with-missing]
|
||||||
|
|
||||||
Parameter
|
Parameter
|
||||||
--out Zielverzeichnis (Standard: ./test_vault)
|
--out Zielverzeichnis (Standard: ./test_vault)
|
||||||
--force Bestehenden Ordner löschen und neu anlegen.
|
--force Bestehenden Ordner löschen und neu anlegen.
|
||||||
|
--with-missing Auch die 'missing-note' direkt anlegen (Standard: erst im 2. Lauf)
|
||||||
|
|
||||||
Hinweise für Tests
|
Hinweise für Tests
|
||||||
1) Erster Import (dry-run oder --apply): Es gibt einen „leeren“ Link [[missing-note]].
|
1) Erster Import (dry-run oder --apply): Es gibt einen „leeren“ Link [[missing-note]].
|
||||||
@@ -35,6 +36,7 @@ Hinweise für Tests
|
|||||||
-> Erwartung: Edges für ehemals leeren Link werden korrekt nachgezogen (references + backlink).
|
-> Erwartung: Edges für ehemals leeren Link werden korrekt nachgezogen (references + backlink).
|
||||||
3) Ändere den Body von exp-one.md so, dass andere Chunk-Grenzen entstehen und importiere erneut:
|
3) Ändere den Body von exp-one.md so, dass andere Chunk-Grenzen entstehen und importiere erneut:
|
||||||
-> Erwartung: Chunks/Edges/Note werden für betroffene Noten aktualisiert.
|
-> Erwartung: Chunks/Edges/Note werden für betroffene Noten aktualisiert.
|
||||||
|
4) Prüfe externe Links (verify_chunk_texts.py betrifft nur Text; externe Links siehst du in chunk_payloads).
|
||||||
|
|
||||||
Kompatibel mit:
|
Kompatibel mit:
|
||||||
- note.schema.json: 'created'/'updated' müssen strings sein. (Wichtig!)
|
- note.schema.json: 'created'/'updated' müssen strings sein. (Wichtig!)
|
||||||
@@ -52,7 +54,6 @@ def iso_now() -> str:
|
|||||||
|
|
||||||
def q(s: str) -> str:
|
def q(s: str) -> str:
|
||||||
# Sicheres Quoten für YAML-Frontmatter
|
# Sicheres Quoten für YAML-Frontmatter
|
||||||
# (einfaches doppelt-Quoten reicht hier, da wir keine eingebetteten """ verwenden)
|
|
||||||
return f"\"{s}\""
|
return f"\"{s}\""
|
||||||
|
|
||||||
TEMPLATE_CONCEPT = """---
|
TEMPLATE_CONCEPT = """---
|
||||||
@@ -66,7 +67,7 @@ tags: ["area/mindnet","type/concept","topic/test"]
|
|||||||
---
|
---
|
||||||
|
|
||||||
## Beschreibung
|
## Beschreibung
|
||||||
Dies ist ein einfaches Konzept für Testzwecke.
|
Dies ist ein einfaches Konzept für Testzwecke. Es wird von anderen Notizen referenziert.
|
||||||
|
|
||||||
## Mögliche Verbindungen
|
## Mögliche Verbindungen
|
||||||
- [[exp-one]]
|
- [[exp-one]]
|
||||||
@@ -109,10 +110,12 @@ tags: ["area/mindnet","type/experience","topic/test"]
|
|||||||
---
|
---
|
||||||
|
|
||||||
## Kontext
|
## Kontext
|
||||||
Zweite Experience, die auf dasselbe Konzept verweist.
|
Zweite Experience, die auf dasselbe Konzept verweist, ergänzt um **externe Links**.
|
||||||
|
|
||||||
## Beobachtung
|
## Beobachtung
|
||||||
Verweis auf [[concept-alpha]].
|
Verweis auf [[concept-alpha]]. Zusätzliche Ressourcen:
|
||||||
|
- Link: [Beispielseite](https://example.com "Referenz")
|
||||||
|
- Kontakt: [E-Mail](mailto:info@example.com)
|
||||||
|
|
||||||
## Mögliche Verbindungen
|
## Mögliche Verbindungen
|
||||||
- [[concept-alpha]]
|
- [[concept-alpha]]
|
||||||
@@ -218,6 +221,7 @@ def main():
|
|||||||
print(" 1) Zuerst ohne missing-note anlegen (Standard), importieren, Edges prüfen.")
|
print(" 1) Zuerst ohne missing-note anlegen (Standard), importieren, Edges prüfen.")
|
||||||
print(" 2) Dann erneut ausführen mit --force --with-missing (missing-note wird erzeugt),")
|
print(" 2) Dann erneut ausführen mit --force --with-missing (missing-note wird erzeugt),")
|
||||||
print(" und Import wiederholen → Edges/Backlinks sollten nachgezogen werden.")
|
print(" und Import wiederholen → Edges/Backlinks sollten nachgezogen werden.")
|
||||||
|
print(" 3) In exp-two.md sind **externe Links** enthalten → prüfe `external_links` in Chunk-Payloads.")
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/ollama_tool_runner.py — Minimaler Tool-Caller für Ollama + mindnet
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
Führt den Tool-Call-Loop für mindnet_query / mindnet_subgraph aus.
|
||||||
|
Nutzt /tools/ollama (Schema) und deine FastAPI (/query, /graph).
|
||||||
|
ENV:
|
||||||
|
OLLAMA=http://127.0.0.1:11434
|
||||||
|
API_BASE=http://127.0.0.1:8000
|
||||||
|
MODEL=llama3.1
|
||||||
|
Nutzung:
|
||||||
|
python3 scripts/ollama_tool_runner.py "Frage an den Agenten"
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import os, sys, json, requests
|
||||||
|
|
||||||
|
OLLAMA = os.getenv("OLLAMA", "http://127.0.0.1:11434")
|
||||||
|
API_BASE = os.getenv("API_BASE", "http://127.0.0.1:8000")
|
||||||
|
MODEL = os.getenv("MODEL", "llama3.1")
|
||||||
|
|
||||||
|
def load_tools():
|
||||||
|
r = requests.get(f"{API_BASE}/tools/ollama", timeout=30)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.json()["tools"]
|
||||||
|
|
||||||
|
def call_model(messages, tools):
|
||||||
|
payload = {
|
||||||
|
"model": MODEL,
|
||||||
|
"messages": messages,
|
||||||
|
"tools": tools,
|
||||||
|
"stream": False,
|
||||||
|
"tool_choice": "auto",
|
||||||
|
}
|
||||||
|
r = requests.post(f"{OLLAMA}/api/chat", json=payload, timeout=180)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
def execute_tool(name: str, arguments: dict):
|
||||||
|
if name == "mindnet_query":
|
||||||
|
# mappe expand_depth/edge_types auf /query-Body
|
||||||
|
body = {
|
||||||
|
"mode": "hybrid",
|
||||||
|
"top_k": int(arguments.get("top_k", 10)),
|
||||||
|
"expand": {
|
||||||
|
"depth": int(arguments.get("expand_depth", 1)),
|
||||||
|
"edge_types": arguments.get("edge_types") or ["references","belongs_to","prev","next"],
|
||||||
|
},
|
||||||
|
"filters": arguments.get("filters"),
|
||||||
|
}
|
||||||
|
if "query_vector" in arguments:
|
||||||
|
body["query_vector"] = arguments["query_vector"]
|
||||||
|
if "query" in arguments:
|
||||||
|
body["query"] = arguments["query"]
|
||||||
|
r = requests.post(f"{API_BASE}/query", json=body, timeout=120)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
if name == "mindnet_subgraph":
|
||||||
|
params = {
|
||||||
|
"depth": int(arguments.get("depth", 1)),
|
||||||
|
}
|
||||||
|
edge_types = arguments.get("edge_types")
|
||||||
|
if edge_types:
|
||||||
|
# mehrfacher Query-Param 'edge_types' wird von requests automatisch gebaut
|
||||||
|
params["edge_types"] = edge_types
|
||||||
|
note_id = arguments["note_id"]
|
||||||
|
r = requests.get(f"{API_BASE}/graph/{note_id}", params=params, timeout=60)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
raise ValueError(f"Unbekanntes Tool: {name}")
|
||||||
|
|
||||||
|
def main():
|
||||||
|
if len(sys.argv) < 2:
|
||||||
|
print("Usage: python3 scripts/ollama_tool_runner.py \"Deine Frage\"")
|
||||||
|
return 2
|
||||||
|
|
||||||
|
user_prompt = sys.argv[1]
|
||||||
|
tools = load_tools()
|
||||||
|
messages = [{"role": "user", "content": user_prompt}]
|
||||||
|
|
||||||
|
while True:
|
||||||
|
resp = call_model(messages, tools)
|
||||||
|
msg = resp.get("message") or {}
|
||||||
|
tool_calls = msg.get("tool_calls") or []
|
||||||
|
|
||||||
|
if not tool_calls:
|
||||||
|
# finale Antwort
|
||||||
|
print("\n=== Antwort ===\n")
|
||||||
|
print(msg.get("content", "").strip())
|
||||||
|
return 0
|
||||||
|
|
||||||
|
# Führe Toolcalls nacheinander aus
|
||||||
|
for tc in tool_calls:
|
||||||
|
name = tc["function"]["name"]
|
||||||
|
args = tc["function"].get("arguments") or {}
|
||||||
|
if isinstance(args, str):
|
||||||
|
try: args = json.loads(args)
|
||||||
|
except Exception: args = {"raw": args}
|
||||||
|
|
||||||
|
print(f"[tool] {name}({args})")
|
||||||
|
result = execute_tool(name, args)
|
||||||
|
|
||||||
|
# Übergib Toolresultat ans Modell zurück
|
||||||
|
messages.append({"role": "tool", "content": json.dumps(result), "name": name})
|
||||||
|
|
||||||
|
# füttere auch die originale Antwort (falls enthalten) in den Verlauf
|
||||||
|
if msg.get("content"):
|
||||||
|
messages.append({"role": "assistant", "content": msg["content"]})
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/payload_dryrun.py (zeigt, was VOR dem Upsert tatsächlich in den Payloads steht)
|
||||||
|
- KEIN Überschreiben der Note-Payload mehr
|
||||||
|
- types.yaml ist maßgeblich (gemäß app/core/note_payload.py & chunk_payload.py)
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, os, json
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
from app.core.parser import read_markdown, normalize_frontmatter, validate_required_frontmatter
|
||||||
|
from app.core.note_payload import make_note_payload
|
||||||
|
from app.core.chunker import assemble_chunks
|
||||||
|
from app.core.chunk_payload import make_chunk_payloads
|
||||||
|
try:
|
||||||
|
from app.core.derive_edges import build_edges_for_note
|
||||||
|
except Exception:
|
||||||
|
from app.core.edges import build_edges_for_note # type: ignore
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--vault", required=True)
|
||||||
|
ap.add_argument("--note-id")
|
||||||
|
ap.add_argument("--with-edges", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
root = os.path.abspath(args.vault)
|
||||||
|
|
||||||
|
files: List[str] = []
|
||||||
|
for dp, _, fns in os.walk(root):
|
||||||
|
for fn in fns:
|
||||||
|
if fn.lower().endswith(".md"):
|
||||||
|
files.append(os.path.join(dp, fn))
|
||||||
|
files.sort()
|
||||||
|
|
||||||
|
for path in files:
|
||||||
|
parsed = read_markdown(path)
|
||||||
|
if not parsed:
|
||||||
|
continue
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter)
|
||||||
|
try:
|
||||||
|
validate_required_frontmatter(fm)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": path, "error": f"invalid frontmatter: {e}"}))
|
||||||
|
continue
|
||||||
|
|
||||||
|
if args.note_id and fm.get("id") != args.note_id:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Note-Payload exakt so, wie der Importer ihn baut (types.yaml maßgeblich)
|
||||||
|
note_pl = make_note_payload(parsed,
|
||||||
|
vault_root=root,
|
||||||
|
hash_mode="body",
|
||||||
|
hash_normalize="canonical",
|
||||||
|
hash_source="parsed",
|
||||||
|
file_path=path)
|
||||||
|
|
||||||
|
body_text = getattr(parsed, "body", "") or ""
|
||||||
|
chunks = assemble_chunks(fm["id"], body_text, fm.get("type","concept"))
|
||||||
|
|
||||||
|
chunk_note = {
|
||||||
|
"frontmatter": fm,
|
||||||
|
"id": fm.get("id"),
|
||||||
|
"type": fm.get("type"),
|
||||||
|
"title": fm.get("title"),
|
||||||
|
"path": note_pl.get("path") or path,
|
||||||
|
"note_id": note_pl.get("note_id"),
|
||||||
|
"tags": fm.get("tags"),
|
||||||
|
}
|
||||||
|
# make_chunk_payloads bestimmt Werte ebenfalls aus types.yaml (Frontmatter wird ignoriert)
|
||||||
|
chunk_pls = make_chunk_payloads(
|
||||||
|
chunk_note,
|
||||||
|
note_pl["path"],
|
||||||
|
chunks,
|
||||||
|
note_text=body_text,
|
||||||
|
types_cfg=None, # Loader aus Datei; kein Override von außen
|
||||||
|
file_path=path,
|
||||||
|
)
|
||||||
|
|
||||||
|
out = {
|
||||||
|
"note_id": note_pl.get("note_id") or fm.get("id"),
|
||||||
|
"title": fm.get("title"),
|
||||||
|
"type": fm.get("type"),
|
||||||
|
"note_payload": {
|
||||||
|
"retriever_weight": note_pl.get("retriever_weight"),
|
||||||
|
"chunk_profile": note_pl.get("chunk_profile")
|
||||||
|
},
|
||||||
|
"chunks_summary": {
|
||||||
|
"count": len(chunk_pls),
|
||||||
|
"first": [
|
||||||
|
{k: chunk_pls[i].get(k) for k in ("chunk_id","index","ord","retriever_weight","chunk_profile","neighbors_prev","neighbors_next")}
|
||||||
|
for i in range(min(3, len(chunk_pls)))
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"path": note_pl.get("path")
|
||||||
|
}
|
||||||
|
|
||||||
|
if args.with_edges:
|
||||||
|
edges = build_edges_for_note(
|
||||||
|
note_id=note_pl.get("note_id") or fm.get("id"),
|
||||||
|
chunk_payloads=chunk_pls,
|
||||||
|
note_level_refs=note_pl.get("references") or [],
|
||||||
|
include_note_scope_refs=False,
|
||||||
|
)
|
||||||
|
kinds = {}
|
||||||
|
for e in edges:
|
||||||
|
k = (e.get("relation") or e.get("kind") or "edge")
|
||||||
|
kinds[k] = kinds.get(k, 0) + 1
|
||||||
|
out["edges_summary"] = {"total": len(edges), "by_kind": kinds}
|
||||||
|
|
||||||
|
print(json.dumps(out, ensure_ascii=False))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Script: scripts/report_hashes.py — Übersicht & Lücken bei Mehrfach-Hashes
|
||||||
|
Version: 1.0.0
|
||||||
|
Datum: 2025-09-10
|
||||||
|
|
||||||
|
Funktion
|
||||||
|
--------
|
||||||
|
Listet je Note die vorhandenen Einträge im Feld `hashes` (Signaturen: <mode>:<source>:<normalize>)
|
||||||
|
und meldet fehlende Soll-Keys. Eignet sich als CI-Check.
|
||||||
|
|
||||||
|
Optionen
|
||||||
|
--------
|
||||||
|
--prefix TEXT Collection-Prefix (CLI überschreibt ENV)
|
||||||
|
--require K [K ...] Zusätzliche Soll-Keys (Default: body|frontmatter|full:parsed:canonical)
|
||||||
|
--fail-on-missing Exitcode 2, wenn fehlende Keys gefunden werden
|
||||||
|
|
||||||
|
Beispiele
|
||||||
|
---------
|
||||||
|
python3 -m scripts.report_hashes --prefix mindnet
|
||||||
|
python3 -m scripts.report_hashes --require frontmatter:raw:none --fail-on-missing
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from typing import List, Dict, Any
|
||||||
|
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
def collections(prefix: str):
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def _scroll_all(client, collection: str):
|
||||||
|
out = []
|
||||||
|
nextp = None
|
||||||
|
while True:
|
||||||
|
pts, nextp = client.scroll(collection_name=collection, with_payload=True, with_vectors=False, limit=256, offset=nextp)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
out.extend(pts)
|
||||||
|
if nextp is None:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", help="Collection-Prefix (überschreibt ENV COLLECTION_PREFIX)")
|
||||||
|
ap.add_argument("--require", nargs="+", help="Zusätzliche Soll-Keys <mode>:<source>:<normalize>")
|
||||||
|
ap.add_argument("--fail-on-missing", action="store_true", help="Exitcode 2 bei fehlenden Keys")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
if args.prefix:
|
||||||
|
cfg.prefix = args.prefix.strip()
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
notes_col, _, _ = collections(cfg.prefix)
|
||||||
|
pts = _scroll_all(client, notes_col)
|
||||||
|
|
||||||
|
required = set(args.require or [])
|
||||||
|
required |= {
|
||||||
|
"body:parsed:canonical",
|
||||||
|
"frontmatter:parsed:canonical",
|
||||||
|
"full:parsed:canonical",
|
||||||
|
}
|
||||||
|
|
||||||
|
missing_total = 0
|
||||||
|
for p in pts:
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
hashes = pl.get("hashes") or {}
|
||||||
|
present = set(hashes.keys())
|
||||||
|
missing = sorted(list(required - present))
|
||||||
|
obj = {
|
||||||
|
"note_id": nid,
|
||||||
|
"present_count": len(present),
|
||||||
|
"missing": missing,
|
||||||
|
}
|
||||||
|
print(json.dumps(obj, ensure_ascii=False))
|
||||||
|
missing_total += len(missing)
|
||||||
|
|
||||||
|
if args.fail_on_missing and missing_total > 0:
|
||||||
|
raise SystemExit(2)
|
||||||
|
|
||||||
|
print(json.dumps({"summary_missing_total": missing_total}))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
+33
-10
@@ -2,28 +2,33 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Name: scripts/reset_qdrant.py
|
Name: scripts/reset_qdrant.py
|
||||||
Version: v1.1.1 (2025-09-05)
|
Version: v1.2.0 (2025-11-11)
|
||||||
Kurzbeschreibung:
|
Kurzbeschreibung:
|
||||||
Sicheres Zurücksetzen der Qdrant-Collections für EIN Projektpräfix. Das Skript
|
Sicheres Zurücksetzen der Qdrant-Collections für EIN Projektpräfix. Das Skript
|
||||||
ermittelt zunächst die tatsächlich betroffenen Collections und zeigt eine
|
ermittelt zunächst die tatsächlich betroffenen Collections und zeigt eine
|
||||||
Übersicht an. Anschließend wird der User um Bestätigung gebeten (interaktive
|
Übersicht an. Anschließend wird der User um Bestätigung gebeten (interaktive
|
||||||
Abfrage), bevor eine der beiden Aktionen ausgeführt wird:
|
Abfrage), bevor eine der Aktionen ausgeführt wird:
|
||||||
|
|
||||||
- wipe: Löscht die Collections komplett und legt sie gemäß Projekt-Defaults neu an
|
- wipe: Löscht die Collections komplett und legt sie gemäß Projekt-Defaults neu an.
|
||||||
- truncate: Löscht nur alle Points (Inhalte), behält die Collection-Settings
|
**Neu**: Danach werden *immer* die Payload-Indizes & Schema-Ergänzungen
|
||||||
|
über ensure_payload_indexes() idempotent eingerichtet (falls nicht mit --no-indexes deaktiviert).
|
||||||
|
- truncate: Löscht nur alle Points (Inhalte), behält die Collection-Settings.
|
||||||
|
**Neu**: Optional (Default) werden im Anschluss die Payload-Indizes geprüft/ergänzt.
|
||||||
|
|
||||||
Aufruf (aus Projekt-Root, im venv):
|
Aufruf (aus Projekt-Root, im venv):
|
||||||
python3 -m scripts.reset_qdrant --mode wipe [--prefix PREFIX] [--yes] [--dry-run]
|
python3 -m scripts.reset_qdrant --mode wipe [--prefix PREFIX] [--yes] [--dry-run]
|
||||||
python3 -m scripts.reset_qdrant --mode truncate [--prefix PREFIX] [--yes] [--dry-run]
|
python3 -m scripts.reset_qdrant --mode truncate [--prefix PREFIX] [--yes] [--dry-run]
|
||||||
|
|
||||||
Parameter:
|
Parameter:
|
||||||
--mode wipe | truncate (Pflicht)
|
--mode wipe | truncate (Pflicht)
|
||||||
--prefix Collection-Prefix (Default: env COLLECTION_PREFIX oder 'mindnet')
|
--prefix Collection-Prefix (Default: env COLLECTION_PREFIX oder 'mindnet')
|
||||||
--yes Keine Rückfrage, direkt ausführen (CI/CD geeignet)
|
--yes Keine Rückfrage, direkt ausführen (CI/CD geeignet)
|
||||||
--dry-run Nur anzeigen, was passieren würde; nichts ändern
|
--dry-run Nur anzeigen, was passieren würde; nichts ändern
|
||||||
|
--no-indexes Überspringt den Schritt ensure_payload_indexes() (Standard: Indizes/Schema werden geprüft/ergänzt)
|
||||||
|
|
||||||
Umgebungsvariablen (optional):
|
Umgebungsvariablen (optional):
|
||||||
QDRANT_URL, QDRANT_API_KEY, COLLECTION_PREFIX, VECTOR_DIM (Default 384)
|
QDRANT_URL, QDRANT_API_KEY, COLLECTION_PREFIX, VECTOR_DIM (Default 384)
|
||||||
|
(Zusätzliche ENV für Named Vectors etc. werden innerhalb ensure_collections/ensure_payload_indexes berücksichtigt.)
|
||||||
|
|
||||||
Sicherheitsmerkmale:
|
Sicherheitsmerkmale:
|
||||||
- Betrachtet ausschließlich Collections des angegebenen Präfixes.
|
- Betrachtet ausschließlich Collections des angegebenen Präfixes.
|
||||||
@@ -34,6 +39,8 @@ Exitcodes:
|
|||||||
0 = OK, 1 = abgebrochen/keine Aktion, 2 = Verbindungs-/Konfigurationsfehler
|
0 = OK, 1 = abgebrochen/keine Aktion, 2 = Verbindungs-/Konfigurationsfehler
|
||||||
|
|
||||||
Changelog:
|
Changelog:
|
||||||
|
v1.2.0: ensure_payload_indexes() nach wipe/truncate standardmäßig ausführen (idempotent); --no-indexes Flag ergänzt.
|
||||||
|
v1.1.1: Stabilisierung & Preview (2025-09-05).
|
||||||
v1.1.0: Interaktive Bestätigung, --yes/--dry-run hinzugefügt, Preview der betroffenen Collections.
|
v1.1.0: Interaktive Bestätigung, --yes/--dry-run hinzugefügt, Preview der betroffenen Collections.
|
||||||
v1.0.0: Wipe/Truncate ohne Bestätigungsabfrage.
|
v1.0.0: Wipe/Truncate ohne Bestätigungsabfrage.
|
||||||
"""
|
"""
|
||||||
@@ -46,7 +53,13 @@ from typing import List
|
|||||||
from qdrant_client import QdrantClient
|
from qdrant_client import QdrantClient
|
||||||
from qdrant_client.http import models as rest
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
from app.core.qdrant import QdrantConfig, get_client, ensure_collections, collection_names
|
from app.core.qdrant import (
|
||||||
|
QdrantConfig,
|
||||||
|
get_client,
|
||||||
|
ensure_collections,
|
||||||
|
ensure_payload_indexes,
|
||||||
|
collection_names,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def resolve_existing_collections(client: QdrantClient, prefix: str) -> List[str]:
|
def resolve_existing_collections(client: QdrantClient, prefix: str) -> List[str]:
|
||||||
@@ -111,12 +124,13 @@ def wipe_collections(client: QdrantClient, all_col_names: List[str], existing: L
|
|||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
ap = argparse.ArgumentParser(description="Wipe oder truncate mindnet-Collections in Qdrant (mit Bestätigung).")
|
ap = argparse.ArgumentParser(description="Wipe oder truncate mindnet-Collections in Qdrant (mit Bestätigung & Index-Setup).")
|
||||||
ap.add_argument("--mode", choices=["wipe", "truncate"], required=True,
|
ap.add_argument("--mode", choices=["wipe", "truncate"], required=True,
|
||||||
help="wipe = Collections löschen & neu anlegen; truncate = nur Inhalte löschen")
|
help="wipe = Collections löschen & neu anlegen; truncate = nur Inhalte löschen")
|
||||||
ap.add_argument("--prefix", help="Collection-Prefix (Default: env COLLECTION_PREFIX oder 'mindnet')")
|
ap.add_argument("--prefix", help="Collection-Prefix (Default: env COLLECTION_PREFIX oder 'mindnet')")
|
||||||
ap.add_argument("--yes", action="store_true", help="Ohne Rückfrage ausführen (CI/CD)")
|
ap.add_argument("--yes", action="store_true", help="Ohne Rückfrage ausführen (CI/CD)")
|
||||||
ap.add_argument("--dry-run", action="store_true", help="Nur anzeigen, was passieren würde; nichts ändern")
|
ap.add_argument("--dry-run", action="store_true", help="Nur anzeigen, was passieren würde; nichts ändern")
|
||||||
|
ap.add_argument("--no-indexes", action="store_true", help="Überspringt ensure_payload_indexes()")
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
# Qdrant-Konfiguration
|
# Qdrant-Konfiguration
|
||||||
@@ -155,13 +169,22 @@ def main():
|
|||||||
if args.mode == "wipe":
|
if args.mode == "wipe":
|
||||||
wipe_collections(client, all_col_names, existing)
|
wipe_collections(client, all_col_names, existing)
|
||||||
ensure_collections(client, cfg.prefix, cfg.dim)
|
ensure_collections(client, cfg.prefix, cfg.dim)
|
||||||
|
if not args.no_indexes:
|
||||||
|
ensure_payload_indexes(client, cfg.prefix)
|
||||||
print(f"Wiped & recreated (Prefix={cfg.prefix}): {all_col_names}")
|
print(f"Wiped & recreated (Prefix={cfg.prefix}): {all_col_names}")
|
||||||
|
if not args.no_indexes:
|
||||||
|
print("Payload-Indizes & Schema-Ergänzungen geprüft/ergänzt (ensure_payload_indexes).")
|
||||||
else:
|
else:
|
||||||
if not existing:
|
if not existing:
|
||||||
print("Keine existierenden Collections zum Truncaten gefunden. Beende ohne Aktion.")
|
print("Keine existierenden Collections zum Truncaten gefunden. Beende ohne Aktion.")
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
delete_all_points(client, existing)
|
delete_all_points(client, existing)
|
||||||
|
if not args.no_indexes:
|
||||||
|
# Auch nach truncate sicherstellen, dass neue/fehlende Indizes ergänzt werden
|
||||||
|
ensure_payload_indexes(client, cfg.prefix)
|
||||||
print(f"Truncated (deleted all points in): {existing}")
|
print(f"Truncated (deleted all points in): {existing}")
|
||||||
|
if not args.no_indexes:
|
||||||
|
print("Payload-Indizes & Schema-Ergänzungen geprüft/ergänzt (ensure_payload_indexes).")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -0,0 +1,185 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Script: scripts/verify_chunk_texts.py
|
||||||
|
Version: 1.0.0
|
||||||
|
Datum: 2025-09-09
|
||||||
|
|
||||||
|
Kurzbeschreibung
|
||||||
|
---------------
|
||||||
|
Verifiziert, dass in Qdrant für jede Note die zugehörigen Chunks ein Textfeld
|
||||||
|
enthalten und der Body (notes.payload.fulltext) aus den Chunk-Texten sinnvoll
|
||||||
|
rekonstruiert werden kann.
|
||||||
|
|
||||||
|
Prüfungen pro Note:
|
||||||
|
- Alle Chunks vorhanden (>=1).
|
||||||
|
- Jedes Chunk-Payload hat einen nutzbaren Textschlüssel: "text" (bevorzugt), sonst "content", sonst "raw".
|
||||||
|
- Reihenfolge der Chunks wird stabil bestimmt (payload.chunk_index -> Nummer aus chunk_id).
|
||||||
|
- Coverage: Summe der im Fulltext gefundenen Chunk-Textsegmente / len(Fulltext) (Toleranz für Overlaps).
|
||||||
|
-> OK wenn coverage >= 0.90 (konfigurierbar via --min-coverage)
|
||||||
|
|
||||||
|
Ausgabe:
|
||||||
|
- JSON mit Gesamtsummen und Details je Note.
|
||||||
|
|
||||||
|
ENV:
|
||||||
|
- QDRANT_URL | QDRANT_HOST/QDRANT_PORT | QDRANT_API_KEY
|
||||||
|
- COLLECTION_PREFIX (Fallback, wenn --prefix fehlt)
|
||||||
|
|
||||||
|
Beispiele:
|
||||||
|
# Alle Notes prüfen (Prefix aus ENV)
|
||||||
|
python3 -m scripts.verify_chunk_texts
|
||||||
|
|
||||||
|
# Nur eine Note prüfen
|
||||||
|
python3 -m scripts.verify_chunk_texts --note-id concept-alpha
|
||||||
|
|
||||||
|
# Prefix explizit setzen und strengere Coverage verlangen
|
||||||
|
python3 -m scripts.verify_chunk_texts --prefix mindnet --min-coverage 0.95
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, json, os, re, sys
|
||||||
|
from typing import Dict, List, Tuple, Optional
|
||||||
|
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
def _names(prefix: str) -> Tuple[str,str,str]:
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def _client() -> QdrantClient:
|
||||||
|
url = os.getenv("QDRANT_URL")
|
||||||
|
if not url:
|
||||||
|
host = os.getenv("QDRANT_HOST", "127.0.0.1")
|
||||||
|
port = int(os.getenv("QDRANT_PORT", "6333"))
|
||||||
|
url = f"http://{host}:{port}"
|
||||||
|
api_key = os.getenv("QDRANT_API_KEY") or None
|
||||||
|
return QdrantClient(url=url, api_key=api_key)
|
||||||
|
|
||||||
|
def _chunk_sort_key(p: Dict, pid: str) -> Tuple[int,int,str]:
|
||||||
|
# Primär: payload.chunk_index, sekundär: Nummer am Ende der ID (#cNN oder #NN), sonst 0
|
||||||
|
ci = p.get("chunk_index")
|
||||||
|
n = 0
|
||||||
|
m = re.search(r'#c?(\d+)$', pid or "")
|
||||||
|
if m:
|
||||||
|
try:
|
||||||
|
n = int(m.group(1))
|
||||||
|
except Exception:
|
||||||
|
n = 0
|
||||||
|
return (ci if isinstance(ci, int) else 1_000_000 + n, n, pid)
|
||||||
|
|
||||||
|
def _choose_text(payload: Dict) -> Optional[str]:
|
||||||
|
for k in ("text", "content", "raw"):
|
||||||
|
v = payload.get(k)
|
||||||
|
if isinstance(v, str) and v.strip():
|
||||||
|
return v
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _coverage(fulltext: str, pieces: List[str]) -> float:
|
||||||
|
"""Berechnet die Abdeckungsquote der Stücke im Fulltext (sequenzielles Matching)."""
|
||||||
|
if not fulltext:
|
||||||
|
return 0.0 if pieces else 1.0
|
||||||
|
cursor = 0
|
||||||
|
covered = 0
|
||||||
|
ft = fulltext
|
||||||
|
for piece in pieces:
|
||||||
|
if not piece:
|
||||||
|
continue
|
||||||
|
# Tolerant gegen Whitespace-Unterschiede: normalisieren nur \r\n→\n
|
||||||
|
p = piece.replace("\r\n", "\n").replace("\r", "\n")
|
||||||
|
idx = ft.find(p, cursor)
|
||||||
|
if idx == -1:
|
||||||
|
# Versuche ein paar Heuristiken: trimmen
|
||||||
|
p2 = p.strip()
|
||||||
|
if p2 and len(p2) > 8:
|
||||||
|
idx = ft.find(p2, cursor)
|
||||||
|
if idx != -1:
|
||||||
|
covered += len(p)
|
||||||
|
cursor = idx + len(p)
|
||||||
|
# sonst: nicht abgedeckt
|
||||||
|
return covered / max(1, len(ft))
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", help="Collection-Prefix (Default: ENV COLLECTION_PREFIX oder 'mindnet')")
|
||||||
|
ap.add_argument("--note-id", help="Nur eine bestimmte Note prüfen")
|
||||||
|
ap.add_argument("--min-coverage", type=float, default=0.90, help="Mindestabdeckung durch Chunks (Default: 0.90)")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
prefix = args.prefix or os.getenv("COLLECTION_PREFIX", "mindnet")
|
||||||
|
notes_col, chunks_col, _ = _names(prefix)
|
||||||
|
cli = _client()
|
||||||
|
|
||||||
|
# Notes abrufen (optional filter by note_id)
|
||||||
|
notes_filter = None
|
||||||
|
if args.note_id:
|
||||||
|
notes_filter = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=args.note_id))])
|
||||||
|
|
||||||
|
notes: List[Dict] = []
|
||||||
|
off = None
|
||||||
|
while True:
|
||||||
|
pts, off = cli.scroll(collection_name=notes_col, scroll_filter=notes_filter,
|
||||||
|
with_payload=True, with_vectors=False, limit=256, offset=off)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
for p in pts:
|
||||||
|
notes.append({"id": p.id, "payload": p.payload or {}})
|
||||||
|
if off is None:
|
||||||
|
break
|
||||||
|
|
||||||
|
results = []
|
||||||
|
total_missing_text = 0
|
||||||
|
total_notes_ok = 0
|
||||||
|
for n in notes:
|
||||||
|
pl = n["payload"]
|
||||||
|
nid = pl.get("note_id") or pl.get("id") or n.get("id")
|
||||||
|
fulltext = pl.get("fulltext") or ""
|
||||||
|
# Chunks der Note holen
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=nid))])
|
||||||
|
chunks = []
|
||||||
|
off = None
|
||||||
|
while True:
|
||||||
|
pts, off = cli.scroll(collection_name=chunks_col, scroll_filter=f,
|
||||||
|
with_payload=True, with_vectors=False, limit=256, offset=off)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
for p in pts:
|
||||||
|
chunks.append({"id": p.id, "payload": p.payload or {}})
|
||||||
|
if off is None:
|
||||||
|
break
|
||||||
|
# sortieren
|
||||||
|
chunks.sort(key=lambda c: _chunk_sort_key(c["payload"], c["id"]))
|
||||||
|
texts = []
|
||||||
|
missing_text = 0
|
||||||
|
for c in chunks:
|
||||||
|
t = _choose_text(c["payload"])
|
||||||
|
if t is None:
|
||||||
|
missing_text += 1
|
||||||
|
texts.append("")
|
||||||
|
else:
|
||||||
|
texts.append(t)
|
||||||
|
cov = _coverage(fulltext, texts)
|
||||||
|
ok = (missing_text == 0) and (cov >= args.min_coverage or not fulltext)
|
||||||
|
if ok:
|
||||||
|
total_notes_ok += 1
|
||||||
|
total_missing_text += missing_text
|
||||||
|
results.append({
|
||||||
|
"note_id": nid,
|
||||||
|
"title": pl.get("title"),
|
||||||
|
"chunks": len(chunks),
|
||||||
|
"missing_chunk_texts": missing_text,
|
||||||
|
"coverage": round(cov, 4),
|
||||||
|
"has_fulltext": bool(fulltext),
|
||||||
|
"ok": ok
|
||||||
|
})
|
||||||
|
|
||||||
|
out = {
|
||||||
|
"collections": {"notes": notes_col, "chunks": chunks_col},
|
||||||
|
"notes_checked": len(notes),
|
||||||
|
"notes_ok": total_notes_ok,
|
||||||
|
"total_missing_chunk_texts": total_missing_text,
|
||||||
|
"min_coverage": args.min_coverage,
|
||||||
|
"details": results
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/wp04_smoketest.py — E2E-Schnelltest der WP-04 Endpunkte
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
- Holt exemplarisch einen Vektor aus {prefix}_chunks (falls verfügbar),
|
||||||
|
- ruft POST /query mit diesem Vektor auf,
|
||||||
|
- ruft GET /graph/<note_id> für die Top-Note auf,
|
||||||
|
- druckt Kurzresultate auf STDOUT.
|
||||||
|
|
||||||
|
Kompatibilität:
|
||||||
|
Python 3.12+, requests, qdrant-client
|
||||||
|
Version:
|
||||||
|
0.1.0 (Erstanlage)
|
||||||
|
Stand:
|
||||||
|
2025-10-07
|
||||||
|
Bezug:
|
||||||
|
WP-04 /query & /graph
|
||||||
|
Nutzung:
|
||||||
|
export QDRANT_URL="http://localhost:6333"
|
||||||
|
export MINDNET_PREFIX="mindnet"
|
||||||
|
python3 scripts/wp04_smoketest.py --api http://localhost:8000
|
||||||
|
Änderungsverlauf:
|
||||||
|
0.1.0 (2025-10-07) – Erstanlage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import argparse
|
||||||
|
import requests
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
|
||||||
|
|
||||||
|
def get_any_chunk_vector(client: QdrantClient, prefix: str):
|
||||||
|
"""Holt eine beliebige Chunk-Vector/Note-ID-Kombi aus Qdrant (falls vorhanden)."""
|
||||||
|
col = f"{prefix}_chunks"
|
||||||
|
pts, _ = client.scroll(collection_name=col, limit=1, with_vectors=True, with_payload=True)
|
||||||
|
if not pts:
|
||||||
|
return None, None, None
|
||||||
|
p = pts[0]
|
||||||
|
vec = p.vector
|
||||||
|
payload = p.payload or {}
|
||||||
|
node_id = str(p.id)
|
||||||
|
note_id = payload.get("note_id")
|
||||||
|
return vec, node_id, note_id
|
||||||
|
|
||||||
|
|
||||||
|
def run(api_base: str, qdrant_url: str, prefix: str) -> int:
|
||||||
|
print(f"[i] API: {api_base}")
|
||||||
|
print(f"[i] Qdrant: {qdrant_url}")
|
||||||
|
client = QdrantClient(url=qdrant_url)
|
||||||
|
|
||||||
|
vec, node_id, note_id = get_any_chunk_vector(client, prefix)
|
||||||
|
if vec is None:
|
||||||
|
print("[!] Keine Chunks mit Vektoren gefunden. Bitte Import (WP-03) prüfen.")
|
||||||
|
return 2
|
||||||
|
|
||||||
|
print(f"[i] Beispiel-Chunk-ID: {node_id}, Note-ID: {note_id}")
|
||||||
|
print("[i] Rufe /query (hybrid) mit Beispielvektor auf ...")
|
||||||
|
|
||||||
|
q_payload = {
|
||||||
|
"mode": "hybrid",
|
||||||
|
"query_vector": vec, # 384d
|
||||||
|
"top_k": 5,
|
||||||
|
"expand": {"depth": 1, "edge_types": ["references", "belongs_to", "prev", "next"]},
|
||||||
|
}
|
||||||
|
r = requests.post(f"{api_base.rstrip('/')}/query", json=q_payload, timeout=60)
|
||||||
|
print(f"[i] /query Status: {r.status_code}")
|
||||||
|
if r.status_code != 200:
|
||||||
|
print(r.text)
|
||||||
|
return 3
|
||||||
|
|
||||||
|
data = r.json()
|
||||||
|
hits = data.get("results", [])
|
||||||
|
print(f"[i] Treffer: {len(hits)}")
|
||||||
|
for i, h in enumerate(hits, 1):
|
||||||
|
print(f" {i:02d}. node_id={h['node_id']} note_id={h.get('note_id')} total={h['total_score']:.3f}")
|
||||||
|
|
||||||
|
# Graph-Test (wenn Note-ID vorhanden)
|
||||||
|
gid = note_id or (hits[0].get("note_id") if hits else None)
|
||||||
|
if gid:
|
||||||
|
print(f"[i] Rufe /graph/{gid} ...")
|
||||||
|
r2 = requests.get(f"{api_base.rstrip('/')}/graph/{gid}?depth=1", timeout=60)
|
||||||
|
print(f"[i] /graph Status: {r2.status_code}")
|
||||||
|
if r2.status_code == 200:
|
||||||
|
g = r2.json()
|
||||||
|
print(f" Nodes: {len(g.get('nodes', []))}, Edges: {len(g.get('edges', []))}")
|
||||||
|
else:
|
||||||
|
print(r2.text)
|
||||||
|
|
||||||
|
print("[✔] Smoke-Test abgeschlossen.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--api", default="http://localhost:8000", help="Basis-URL der FastAPI.")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
qdrant_url = os.getenv("QDRANT_URL", "http://127.0.0.1:6333")
|
||||||
|
prefix = os.getenv("MINDNET_PREFIX", "mindnet")
|
||||||
|
sys.exit(run(args.api, qdrant_url, prefix))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/assert_payload_schema.py (compat v1.1)
|
||||||
|
|
||||||
|
Prüft, ob die erwarteten Payload-Indizes (payload_schema) auf den drei Collections vorhanden sind.
|
||||||
|
Kompatibel mit älteren qdrant-client Versionen (ohne with_payload_schema im Wrapper).
|
||||||
|
|
||||||
|
Collections & Pflichtfelder:
|
||||||
|
- mindnet_notes : note_id, type, title, updated, tags
|
||||||
|
- mindnet_chunks : note_id, chunk_id, index, type, tags
|
||||||
|
- mindnet_edges : note_id, kind, scope, source_id, target_id, chunk_id
|
||||||
|
|
||||||
|
Nutzung:
|
||||||
|
python3 -m tests.assert_payload_schema
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client, collection_names
|
||||||
|
|
||||||
|
REQUIRED = {
|
||||||
|
"notes": ["note_id", "type", "title", "updated", "tags"],
|
||||||
|
"chunks": ["note_id", "chunk_id", "index", "type", "tags"],
|
||||||
|
"edges": ["note_id", "kind", "scope", "source_id", "target_id", "chunk_id"],
|
||||||
|
}
|
||||||
|
|
||||||
|
def _safe_model_dump(obj: Any) -> Dict[str, Any]:
|
||||||
|
if hasattr(obj, "model_dump"):
|
||||||
|
return obj.model_dump()
|
||||||
|
if hasattr(obj, "dict"):
|
||||||
|
return obj.dict()
|
||||||
|
return obj if isinstance(obj, dict) else {}
|
||||||
|
|
||||||
|
def _cfg_base_url(cfg) -> str:
|
||||||
|
if getattr(cfg, "url", None):
|
||||||
|
return cfg.url.rstrip('/')
|
||||||
|
host = getattr(cfg, "host", None) or os.getenv("QDRANT_HOST") or "127.0.0.1"
|
||||||
|
port = getattr(cfg, "port", None) or os.getenv("QDRANT_PORT") or "6333"
|
||||||
|
return f"http://{host}:{port}"
|
||||||
|
|
||||||
|
def _http_get(url: str, api_key: str | None) -> Dict[str, Any]:
|
||||||
|
req = urllib.request.Request(url, method="GET")
|
||||||
|
if api_key:
|
||||||
|
req.add_header("api-key", api_key)
|
||||||
|
req.add_header("Authorization", f"Bearer {api_key}")
|
||||||
|
req.add_header("Accept", "application/json")
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
data = resp.read()
|
||||||
|
return json.loads(data.decode("utf-8"))
|
||||||
|
|
||||||
|
def get_collection_payload_schema(client, cfg, name: str) -> Dict[str, Any] | None:
|
||||||
|
# 1) OpenAPI client mit Flag
|
||||||
|
try:
|
||||||
|
oc = getattr(client, "openapi_client", None)
|
||||||
|
if oc is not None and hasattr(oc, "collections_api"):
|
||||||
|
api = oc.collections_api
|
||||||
|
info = api.get_collection(collection_name=name, with_payload_schema=True)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
return (d.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# 2) Wrapper (manche Versionen liefern Schema auch ohne Flag)
|
||||||
|
try:
|
||||||
|
info = client.get_collection(collection_name=name)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
ps = (d.get("result") or {}).get("payload_schema")
|
||||||
|
if ps is not None:
|
||||||
|
return ps
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# 3) Direkter HTTP-Call
|
||||||
|
try:
|
||||||
|
base = _cfg_base_url(cfg)
|
||||||
|
url = f"{base}/collections/{name}?with_payload_schema=true"
|
||||||
|
raw = _http_get(url, getattr(cfg, "api_key", None))
|
||||||
|
return (raw.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def ensure_fields(schema: Dict[str, dict] | None, required: List[str]) -> Dict[str, bool]:
|
||||||
|
ks = set((schema or {}).keys())
|
||||||
|
return {k: (k in ks) for k in required}
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
notes, chunks, edges = collection_names(cfg.prefix)
|
||||||
|
names = {"notes": notes, "chunks": chunks, "edges": edges}
|
||||||
|
|
||||||
|
report = {}
|
||||||
|
ok_all = True
|
||||||
|
for kind, col in names.items():
|
||||||
|
sch = get_collection_payload_schema(client, cfg, col)
|
||||||
|
checks = ensure_fields(sch, REQUIRED[kind])
|
||||||
|
ok = all(checks.values())
|
||||||
|
ok_all = ok_all and ok
|
||||||
|
report[kind] = {
|
||||||
|
"collection": col,
|
||||||
|
"ok": ok,
|
||||||
|
"missing": [k for k,v in checks.items() if not v],
|
||||||
|
"present": [k for k,v in checks.items() if v],
|
||||||
|
}
|
||||||
|
|
||||||
|
print(json.dumps({"prefix": cfg.prefix, "ok": ok_all, "report": report}, ensure_ascii=False, indent=2))
|
||||||
|
if not ok_all:
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/audit_edges_now.py — Zählt Edges nach kind & scope und zeigt Chunk- und Note-Anzahl.
|
||||||
|
"""
|
||||||
|
import os, sys, json
|
||||||
|
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
def scroll_all(c, col, flt=None):
|
||||||
|
out, nextp = [], None
|
||||||
|
while True:
|
||||||
|
pts, nextp = c.scroll(collection_name=col, with_payload=True, with_vectors=False, limit=256, scroll_filter=flt, offset=nextp)
|
||||||
|
if not pts: break
|
||||||
|
out.extend(pts)
|
||||||
|
if nextp is None: break
|
||||||
|
return out
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
c = get_client(cfg)
|
||||||
|
notes = f"{cfg.prefix}_notes"
|
||||||
|
chunks = f"{cfg.prefix}_chunks"
|
||||||
|
edges = f"{cfg.prefix}_edges"
|
||||||
|
|
||||||
|
es = scroll_all(c, edges)
|
||||||
|
counts = {}
|
||||||
|
by_scope = {}
|
||||||
|
for p in es:
|
||||||
|
pl = p.payload or {}
|
||||||
|
k = pl.get("kind") or "?"
|
||||||
|
s = pl.get("scope") or "?"
|
||||||
|
counts[k] = counts.get(k, 0) + 1
|
||||||
|
key = f"{k}:{s}"
|
||||||
|
by_scope[key] = by_scope.get(key, 0) + 1
|
||||||
|
|
||||||
|
print(json.dumps({
|
||||||
|
"collections": {"notes": notes, "chunks": chunks, "edges": edges},
|
||||||
|
"total_edges": len(es),
|
||||||
|
"counts": counts,
|
||||||
|
"counts_by_scope": by_scope
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/check_chunks_window_vs_text.py
|
||||||
|
Prüft pro Note und global:
|
||||||
|
- ob window == text (sollte bei Overlap normalerweise NEIN sein)
|
||||||
|
- ungefähre linke Overlap-Länge (Suffix von prev.text vs. Prefix von cur.window)
|
||||||
|
- Größen & einfache Statistiken
|
||||||
|
|
||||||
|
Nutzung:
|
||||||
|
python3 tests/check_chunks_window_vs_text.py --prefix mindnet [--vault ./test_vault]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import os, sys, json, argparse
|
||||||
|
from typing import List, Dict, Any, Tuple, Optional
|
||||||
|
|
||||||
|
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def scroll_chunks_by_note(client, chunks_col: str, note_id: str) -> List[Dict[str, Any]]:
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
out = []
|
||||||
|
next_page = None
|
||||||
|
while True:
|
||||||
|
pts, next_page = client.scroll(
|
||||||
|
collection_name=chunks_col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=256,
|
||||||
|
offset=next_page,
|
||||||
|
scroll_filter=f,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
out.extend([p.payload or {} for p in pts])
|
||||||
|
if next_page is None:
|
||||||
|
break
|
||||||
|
# sortieren: seq -> chunk_index -> chunk_id
|
||||||
|
def keyer(pl):
|
||||||
|
return (pl.get("seq", 1<<30), pl.get("chunk_index", 1<<30), str(pl.get("chunk_id", "")))
|
||||||
|
return sorted(out, key=keyer)
|
||||||
|
|
||||||
|
def list_note_ids(client, notes_col: str) -> List[str]:
|
||||||
|
out = []
|
||||||
|
next_page = None
|
||||||
|
while True:
|
||||||
|
pts, next_page = client.scroll(
|
||||||
|
collection_name=notes_col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=256,
|
||||||
|
offset=next_page,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
for p in pts:
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
if isinstance(nid, str):
|
||||||
|
out.append(nid)
|
||||||
|
if next_page is None:
|
||||||
|
break
|
||||||
|
return sorted(set(out))
|
||||||
|
|
||||||
|
def common_overlap_len(a_suffix: str, b_prefix: str, max_probe: int = 256) -> int:
|
||||||
|
"""
|
||||||
|
Länge des längsten Suffix von a_suffix, das Prefix von b_prefix ist (bruteforce, begrenzt).
|
||||||
|
"""
|
||||||
|
n = min(len(a_suffix), len(b_prefix), max_probe)
|
||||||
|
if n <= 0:
|
||||||
|
return 0
|
||||||
|
a = a_suffix[-n:]
|
||||||
|
b = b_prefix[:n]
|
||||||
|
# steigere den Match von 0..n
|
||||||
|
for k in range(n, 0, -1):
|
||||||
|
if a[-k:] == b[:k]:
|
||||||
|
return k
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", required=True)
|
||||||
|
ap.add_argument("--vault", help="optional, nur für Vergleichsausgaben")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
cfg.prefix = args.prefix
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
notes_col, chunks_col, _ = collections(cfg.prefix)
|
||||||
|
note_ids = list_note_ids(client, notes_col)
|
||||||
|
if not note_ids:
|
||||||
|
print(json.dumps({"error": "no notes found in qdrant for this prefix"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
global_total = 0
|
||||||
|
global_ident = 0
|
||||||
|
global_left_ov_sum = 0
|
||||||
|
detail = []
|
||||||
|
|
||||||
|
for nid in note_ids:
|
||||||
|
chunks = scroll_chunks_by_note(client, chunks_col, nid)
|
||||||
|
if not chunks:
|
||||||
|
detail.append({"note_id": nid, "chunks": 0})
|
||||||
|
continue
|
||||||
|
ident = 0
|
||||||
|
left_ov_sum = 0
|
||||||
|
for i, ch in enumerate(chunks):
|
||||||
|
txt = (ch.get("text") or "").replace("\r\n", "\n")
|
||||||
|
win = (ch.get("window") or "").replace("\r\n", "\n")
|
||||||
|
if txt == win:
|
||||||
|
ident += 1
|
||||||
|
if i > 0:
|
||||||
|
prev_txt = (chunks[i-1].get("text") or "").replace("\r\n", "\n")
|
||||||
|
left_ov = common_overlap_len(prev_txt, win, max_probe=1024)
|
||||||
|
left_ov_sum += left_ov
|
||||||
|
global_total += len(chunks)
|
||||||
|
global_ident += ident
|
||||||
|
global_left_ov_sum += left_ov_sum
|
||||||
|
detail.append({
|
||||||
|
"note_id": nid,
|
||||||
|
"chunks": len(chunks),
|
||||||
|
"identical_text_window": ident,
|
||||||
|
"identical_share": round(ident / max(1, len(chunks)), 3),
|
||||||
|
"avg_left_overlap_est": round(left_ov_sum / max(1, len(chunks)-1), 2) if len(chunks) > 1 else 0.0
|
||||||
|
})
|
||||||
|
|
||||||
|
out = {
|
||||||
|
"notes": len(note_ids),
|
||||||
|
"chunks_total": global_total,
|
||||||
|
"identical_total": global_ident,
|
||||||
|
"identical_share_global": round(global_ident / max(1, global_total), 3),
|
||||||
|
"avg_left_overlap_est_global": round(global_left_ov_sum / max(1, global_total - len(note_ids)), 2) if global_total > len(note_ids) else 0.0,
|
||||||
|
"details": detail[:100] # capped
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/check_chunks_window_vs_text.py
|
||||||
|
Prüft pro Note und global:
|
||||||
|
- ob window == text (sollte bei Overlap normalerweise NEIN sein)
|
||||||
|
- ungefähre linke Overlap-Länge (Suffix von prev.text vs. Prefix von cur.window)
|
||||||
|
- Größen & einfache Statistiken
|
||||||
|
|
||||||
|
Nutzung:
|
||||||
|
python3 tests/check_chunks_window_vs_text.py --prefix mindnet [--vault ./test_vault]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import os, sys, json, argparse
|
||||||
|
from typing import List, Dict, Any, Tuple, Optional
|
||||||
|
|
||||||
|
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def scroll_chunks_by_note(client, chunks_col: str, note_id: str) -> List[Dict[str, Any]]:
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))])
|
||||||
|
out = []
|
||||||
|
next_page = None
|
||||||
|
while True:
|
||||||
|
pts, next_page = client.scroll(
|
||||||
|
collection_name=chunks_col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=256,
|
||||||
|
offset=next_page,
|
||||||
|
scroll_filter=f,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
out.extend([p.payload or {} for p in pts])
|
||||||
|
if next_page is None:
|
||||||
|
break
|
||||||
|
# sortieren: seq -> chunk_index -> chunk_id
|
||||||
|
def keyer(pl):
|
||||||
|
return (pl.get("seq", 1<<30), pl.get("chunk_index", 1<<30), str(pl.get("chunk_id", "")))
|
||||||
|
return sorted(out, key=keyer)
|
||||||
|
|
||||||
|
def list_note_ids(client, notes_col: str) -> List[str]:
|
||||||
|
out = []
|
||||||
|
next_page = None
|
||||||
|
while True:
|
||||||
|
pts, next_page = client.scroll(
|
||||||
|
collection_name=notes_col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=256,
|
||||||
|
offset=next_page,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
for p in pts:
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
if isinstance(nid, str):
|
||||||
|
out.append(nid)
|
||||||
|
if next_page is None:
|
||||||
|
break
|
||||||
|
return sorted(set(out))
|
||||||
|
|
||||||
|
def common_overlap_len(a_suffix: str, b_prefix: str, max_probe: int = 256) -> int:
|
||||||
|
"""
|
||||||
|
Länge des längsten Suffix von a_suffix, das Prefix von b_prefix ist (bruteforce, begrenzt).
|
||||||
|
"""
|
||||||
|
n = min(len(a_suffix), len(b_prefix), max_probe)
|
||||||
|
if n <= 0:
|
||||||
|
return 0
|
||||||
|
a = a_suffix[-n:]
|
||||||
|
b = b_prefix[:n]
|
||||||
|
# steigere den Match von 0..n
|
||||||
|
for k in range(n, 0, -1):
|
||||||
|
if a[-k:] == b[:k]:
|
||||||
|
return k
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", required=True)
|
||||||
|
ap.add_argument("--vault", help="optional, nur für Vergleichsausgaben")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
cfg.prefix = args.prefix
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
notes_col, chunks_col, _ = collections(cfg.prefix)
|
||||||
|
note_ids = list_note_ids(client, notes_col)
|
||||||
|
if not note_ids:
|
||||||
|
print(json.dumps({"error": "no notes found in qdrant for this prefix"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
global_total = 0
|
||||||
|
global_ident = 0
|
||||||
|
global_left_ov_sum = 0
|
||||||
|
detail = []
|
||||||
|
|
||||||
|
for nid in note_ids:
|
||||||
|
chunks = scroll_chunks_by_note(client, chunks_col, nid)
|
||||||
|
if not chunks:
|
||||||
|
detail.append({"note_id": nid, "chunks": 0})
|
||||||
|
continue
|
||||||
|
ident = 0
|
||||||
|
left_ov_sum = 0
|
||||||
|
for i, ch in enumerate(chunks):
|
||||||
|
txt = (ch.get("text") or "").replace("\r\n", "\n")
|
||||||
|
win = (ch.get("window") or "").replace("\r\n", "\n")
|
||||||
|
if txt == win:
|
||||||
|
ident += 1
|
||||||
|
if i > 0:
|
||||||
|
prev_txt = (chunks[i-1].get("text") or "").replace("\r\n", "\n")
|
||||||
|
left_ov = common_overlap_len(prev_txt, win, max_probe=1024)
|
||||||
|
left_ov_sum += left_ov
|
||||||
|
global_total += len(chunks)
|
||||||
|
global_ident += ident
|
||||||
|
global_left_ov_sum += left_ov_sum
|
||||||
|
detail.append({
|
||||||
|
"note_id": nid,
|
||||||
|
"chunks": len(chunks),
|
||||||
|
"identical_text_window": ident,
|
||||||
|
"identical_share": round(ident / max(1, len(chunks)), 3),
|
||||||
|
"avg_left_overlap_est": round(left_ov_sum / max(1, len(chunks)-1), 2) if len(chunks) > 1 else 0.0
|
||||||
|
})
|
||||||
|
|
||||||
|
out = {
|
||||||
|
"notes": len(note_ids),
|
||||||
|
"chunks_total": global_total,
|
||||||
|
"identical_total": global_ident,
|
||||||
|
"identical_share_global": round(global_ident / max(1, global_total), 3),
|
||||||
|
"avg_left_overlap_est_global": round(global_left_ov_sum / max(1, global_total - len(note_ids)), 2) if global_total > len(note_ids) else 0.0,
|
||||||
|
"details": detail[:100] # capped
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,190 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
check_types_registry_edges.py
|
||||||
|
--------------------------------------------------
|
||||||
|
Prüft, ob die in config/types.yaml hinterlegten Default-Kanten
|
||||||
|
pro Note-Typ tatsächlich in der Qdrant *edges*-Collection auftauchen.
|
||||||
|
"""
|
||||||
|
import os, sys, json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List, Optional, Tuple, Set
|
||||||
|
from dataclasses import dataclass, asdict
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml # PyYAML
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"error": f"PyYAML not installed: {e}"})); sys.exit(2)
|
||||||
|
|
||||||
|
try:
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.http import models as rest # noqa: F401
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"error": f"qdrant-client not installed: {e}"})); sys.exit(2)
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Cfg:
|
||||||
|
url: str
|
||||||
|
api_key: Optional[str]
|
||||||
|
prefix: str
|
||||||
|
notes: str
|
||||||
|
edges: str
|
||||||
|
types_file: Path
|
||||||
|
|
||||||
|
def _env_prefix() -> Optional[str]:
|
||||||
|
for k in ("COLLECTION_PREFIX", "MINDNET_PREFIX"):
|
||||||
|
v = os.environ.get(k, "").strip()
|
||||||
|
if v:
|
||||||
|
return v
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _load_types_yaml(path: Path) -> Dict:
|
||||||
|
if not path.exists():
|
||||||
|
print(json.dumps({"warn": f"types.yaml fehlt: {path}. Fallback: keine Vorgaben."}))
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
with path.open("r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f) or {}
|
||||||
|
return data if isinstance(data, dict) else {}
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"warn": f"types.yaml defekt ({path}): {e}. Fallback: keine Vorgaben."}))
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def _cfg_from_env(argv_prefix: Optional[str]) -> Cfg:
|
||||||
|
url = os.environ.get("QDRANT_URL", "http://127.0.0.1:6333").strip()
|
||||||
|
api_key = os.environ.get("QDRANT_API_KEY", "").strip() or None
|
||||||
|
prefix = (argv_prefix or _env_prefix() or "mindnet").strip()
|
||||||
|
notes = f"{prefix}_notes"
|
||||||
|
edges = f"{prefix}_edges"
|
||||||
|
types_path = Path(os.environ.get("TYPES_FILE", "config/types.yaml")).resolve()
|
||||||
|
return Cfg(url=url, api_key=api_key, prefix=prefix, notes=notes, edges=edges, types_file=types_path)
|
||||||
|
|
||||||
|
def _mk_client(cfg: Cfg) -> QdrantClient:
|
||||||
|
return QdrantClient(url=cfg.url, api_key=cfg.api_key, timeout=30.0)
|
||||||
|
|
||||||
|
def _scroll_all_notes(client: QdrantClient, notes_col: str) -> Dict[str, Dict]:
|
||||||
|
out = {}
|
||||||
|
offset = None
|
||||||
|
while True:
|
||||||
|
points, offset = client.scroll(
|
||||||
|
collection_name=notes_col,
|
||||||
|
scroll_filter=None, limit=256, offset=offset,
|
||||||
|
with_payload=True, with_vectors=False,
|
||||||
|
)
|
||||||
|
if not points:
|
||||||
|
break
|
||||||
|
for p in points:
|
||||||
|
payload = p.payload or {}
|
||||||
|
nid = payload.get("note_id") or payload.get("id") or payload.get("uid") or payload.get("slug")
|
||||||
|
if not nid:
|
||||||
|
nid = str(p.id)
|
||||||
|
out[str(nid)] = payload
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _scroll_all_edges(client: QdrantClient, edges_col: str) -> List[Tuple[str, dict]]:
|
||||||
|
out = []
|
||||||
|
offset = None
|
||||||
|
while True:
|
||||||
|
points, offset = client.scroll(
|
||||||
|
collection_name=edges_col,
|
||||||
|
scroll_filter=None, limit=256, offset=offset,
|
||||||
|
with_payload=True, with_vectors=False,
|
||||||
|
)
|
||||||
|
if not points:
|
||||||
|
break
|
||||||
|
for p in points:
|
||||||
|
out.append((str(p.id), p.payload or {}))
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _guess_src_note_id(ed_pl: dict) -> Optional[str]:
|
||||||
|
for k in ("src_note_id", "note_id", "src_id", "src"):
|
||||||
|
nid = ed_pl.get(k)
|
||||||
|
if nid:
|
||||||
|
return str(nid)
|
||||||
|
for k in ("src_ref", "from"):
|
||||||
|
nid = ed_pl.get(k)
|
||||||
|
if isinstance(nid, dict):
|
||||||
|
for kk in ("note_id", "id"):
|
||||||
|
if nid.get(kk):
|
||||||
|
return str(nid[kk])
|
||||||
|
elif isinstance(nid, str):
|
||||||
|
return nid
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _edge_type(ed_pl: dict) -> Optional[str]:
|
||||||
|
for k in ("edge_type", "type", "rel", "relation"):
|
||||||
|
if ed_pl.get(k):
|
||||||
|
return str(ed_pl[k])
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _cfg_to_jsonable(cfg: Cfg) -> dict:
|
||||||
|
d = asdict(cfg)
|
||||||
|
d["types_file"] = str(d.get("types_file"))
|
||||||
|
return d
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
import argparse
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", help="Collection-Prefix (mindnet, etc.)")
|
||||||
|
args = ap.parse_args(argv)
|
||||||
|
|
||||||
|
cfg = _cfg_from_env(args.prefix)
|
||||||
|
print(json.dumps({"cfg": _cfg_to_jsonable(cfg)}, ensure_ascii=False))
|
||||||
|
|
||||||
|
tr = _load_types_yaml(cfg.types_file)
|
||||||
|
types_def = (tr.get("types") if isinstance(tr, dict) else {}) or {}
|
||||||
|
print(json.dumps({"types_defined": list(types_def.keys())}, ensure_ascii=False))
|
||||||
|
|
||||||
|
client = _mk_client(cfg)
|
||||||
|
|
||||||
|
notes = _scroll_all_notes(client, cfg.notes)
|
||||||
|
print(json.dumps({"notes_count": len(notes)}, ensure_ascii=False))
|
||||||
|
|
||||||
|
edges = _scroll_all_edges(client, cfg.edges)
|
||||||
|
print(json.dumps({"edges_count": len(edges)}, ensure_ascii=False))
|
||||||
|
|
||||||
|
note_type = {}
|
||||||
|
for nid, pl in notes.items():
|
||||||
|
t = pl.get("type") or "concept"
|
||||||
|
note_type[str(nid)] = str(t)
|
||||||
|
|
||||||
|
seen = {}
|
||||||
|
counts = {}
|
||||||
|
|
||||||
|
for edge_pid, ed_pl in edges:
|
||||||
|
et = _edge_type(ed_pl)
|
||||||
|
if not et:
|
||||||
|
continue
|
||||||
|
src_nid = _guess_src_note_id(ed_pl)
|
||||||
|
if not src_nid:
|
||||||
|
continue
|
||||||
|
t = note_type.get(src_nid)
|
||||||
|
if not t:
|
||||||
|
continue
|
||||||
|
seen.setdefault(t, set()).add(et)
|
||||||
|
counts.setdefault(t, {}).setdefault(et, 0)
|
||||||
|
counts[t][et] += 1
|
||||||
|
|
||||||
|
expected = {}
|
||||||
|
for tname, tdef in types_def.items():
|
||||||
|
eddefs = (tdef or {}).get("edge_defaults") or []
|
||||||
|
expected[tname] = set([str(x) for x in eddefs if x])
|
||||||
|
|
||||||
|
for tname, exp in expected.items():
|
||||||
|
obs = seen.get(tname, set())
|
||||||
|
missing = sorted(list(exp - obs))
|
||||||
|
extra = sorted(list(obs - exp))
|
||||||
|
print(json.dumps({
|
||||||
|
"type": tname,
|
||||||
|
"expected_defaults": sorted(list(exp)),
|
||||||
|
"observed": sorted(list(obs)),
|
||||||
|
"missing": missing,
|
||||||
|
"extra": extra,
|
||||||
|
"counts": counts.get(tname, {}),
|
||||||
|
}, ensure_ascii=False))
|
||||||
|
|
||||||
|
if not expected:
|
||||||
|
print(json.dumps({"warn": "Keine Typ-Defaults in types.yaml gefunden (edge_defaults leer?)."}))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
import os, json, argparse, random
|
||||||
|
from typing import Any, Dict, List, Tuple, Optional
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
|
||||||
|
def _get_prefix(explicit: Optional[str]) -> str:
|
||||||
|
return (explicit or os.environ.get("COLLECTION_PREFIX") or os.environ.get("MINDNET_PREFIX") or "mindnet").strip()
|
||||||
|
|
||||||
|
def _names(prefix: str) -> Dict[str,str]:
|
||||||
|
return {
|
||||||
|
"notes": f"{prefix}_notes",
|
||||||
|
"chunks": f"{prefix}_chunks",
|
||||||
|
"edges": f"{prefix}_edges",
|
||||||
|
}
|
||||||
|
|
||||||
|
def _scroll(client: QdrantClient, collection: str, *, limit=5, with_payload=True):
|
||||||
|
# Support both modern and older SDKs (filter vs scroll_filter)
|
||||||
|
try:
|
||||||
|
return client.scroll(collection_name=collection, limit=limit, with_payload=with_payload)
|
||||||
|
except TypeError as e:
|
||||||
|
# older signatures might accept scroll_filter kw, but if we passed none it's fine
|
||||||
|
return client.scroll(collection_name=collection, limit=limit, with_payload=with_payload)
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--host", default=os.environ.get("QDRANT_HOST", "127.0.0.1"))
|
||||||
|
ap.add_argument("--port", default=int(os.environ.get("QDRANT_PORT", "6333")))
|
||||||
|
ap.add_argument("--prefix", default=None)
|
||||||
|
ap.add_argument("--limit", type=int, default=5)
|
||||||
|
ap.add_argument("--note-id", default=None)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
prefix = _get_prefix(args.prefix)
|
||||||
|
names = _names(prefix)
|
||||||
|
client = QdrantClient(host=args.host, port=args.port, prefer_grpc=False)
|
||||||
|
|
||||||
|
# Notes sample
|
||||||
|
try:
|
||||||
|
pts, _ = _scroll(client, names["notes"], limit=args.limit, with_payload=True)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"error": f"notes scroll failed: {type(e).__name__}: {e}", "collection": names["notes"]}))
|
||||||
|
return
|
||||||
|
|
||||||
|
sample = []
|
||||||
|
for p in pts:
|
||||||
|
pl = (p.payload or {})
|
||||||
|
sample.append({
|
||||||
|
"point_id": str(p.id),
|
||||||
|
"note_id": pl.get("note_id"),
|
||||||
|
"id": pl.get("id"),
|
||||||
|
"title": pl.get("title"),
|
||||||
|
"type": pl.get("type"),
|
||||||
|
"retriever_weight": pl.get("retriever_weight"),
|
||||||
|
"chunk_profile": pl.get("chunk_profile"),
|
||||||
|
})
|
||||||
|
print(json.dumps({"notes_sample": sample}, ensure_ascii=False))
|
||||||
|
|
||||||
|
# Pick a note to fetch chunks
|
||||||
|
target_note_id = args.note_id or None
|
||||||
|
if not target_note_id:
|
||||||
|
for s in sample:
|
||||||
|
if s.get("note_id"):
|
||||||
|
target_note_id = s["note_id"]
|
||||||
|
break
|
||||||
|
|
||||||
|
if not target_note_id:
|
||||||
|
print(json.dumps({"warn": "no note_id available to fetch chunks"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
# Chunks for target note
|
||||||
|
try:
|
||||||
|
# pull a few chunks and filter client-side by note_id
|
||||||
|
pts, _ = _scroll(client, names["chunks"], limit=50, with_payload=True)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"error": f"chunks scroll failed: {type(e).__name__}: {e}", "collection": names["chunks"]}))
|
||||||
|
return
|
||||||
|
|
||||||
|
chunks = [{
|
||||||
|
"id": str(p.id),
|
||||||
|
"chunk_id": (p.payload or {}).get("chunk_id"),
|
||||||
|
"note_id": (p.payload or {}).get("note_id"),
|
||||||
|
"type": (p.payload or {}).get("type"),
|
||||||
|
"retriever_weight": (p.payload or {}).get("retriever_weight"),
|
||||||
|
"window_len": len((p.payload or {}).get("window") or ""),
|
||||||
|
"text_len": len((p.payload or {}).get("text") or ""),
|
||||||
|
} for p in pts if (p.payload or {}).get("note_id") == target_note_id][:5]
|
||||||
|
|
||||||
|
print(json.dumps({"target_note_id": target_note_id, "chunk_samples": chunks}, ensure_ascii=False))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/compare_vaults.py
|
||||||
|
|
||||||
|
Vergleicht zwei Markdown-Vaults rekursiv:
|
||||||
|
- Schlüsselauswahl: id | title | filename | auto (default: auto = id>title>filename)
|
||||||
|
- Fokus: body | frontmatter | all (default: body)
|
||||||
|
- Unicode: NFC-Normalisierung für IDs/Titel/Dateinamen und Text
|
||||||
|
|
||||||
|
Beispiele:
|
||||||
|
python3 tests/compare_vaults.py --src ./vault --dst ./_exportVault --focus body
|
||||||
|
python3 tests/compare_vaults.py --src ./vault --dst ./_exportVault --key id --focus all
|
||||||
|
|
||||||
|
Ausgabe:
|
||||||
|
- Einträge mit 'missing' (nur in src oder nur in dst)
|
||||||
|
- Einträge mit 'diff' (vorhanden in beiden, aber Abweichungen gem. Fokus)
|
||||||
|
- Summary-Objekt am Ende
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import argparse
|
||||||
|
import unicodedata
|
||||||
|
from typing import Dict, Tuple, Optional, Any
|
||||||
|
|
||||||
|
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
# Wir nutzen denselben Parser wie im Projekt (fehlertolerant)
|
||||||
|
from app.core.parser import read_markdown # erwartet: (frontmatter, body)
|
||||||
|
|
||||||
|
MD_EXTS = {".md", ".markdown"}
|
||||||
|
|
||||||
|
def nfc(s: Optional[str]) -> str:
|
||||||
|
return unicodedata.normalize("NFC", s or "")
|
||||||
|
|
||||||
|
def norm_text(txt: str) -> str:
|
||||||
|
# normalize line endings + strip trailing spaces per line + NFC
|
||||||
|
t = txt.replace("\r\n", "\n").replace("\r", "\n")
|
||||||
|
t = "\n".join(line.rstrip() for line in t.split("\n")).strip()
|
||||||
|
return nfc(t)
|
||||||
|
|
||||||
|
def iter_md_files(root: str):
|
||||||
|
for base, _dirs, files in os.walk(root):
|
||||||
|
for fn in files:
|
||||||
|
ext = os.path.splitext(fn)[1].lower()
|
||||||
|
if ext in MD_EXTS:
|
||||||
|
yield os.path.join(base, fn)
|
||||||
|
|
||||||
|
def key_from(front: Dict[str, Any], path: str, pref: str) -> str:
|
||||||
|
"""
|
||||||
|
Liefert den Vergleichsschlüssel.
|
||||||
|
pref in {"auto","id","title","filename"}
|
||||||
|
"""
|
||||||
|
if pref == "id":
|
||||||
|
return nfc(str(front.get("id", "")).strip())
|
||||||
|
if pref == "title":
|
||||||
|
return nfc(str(front.get("title", "")).strip())
|
||||||
|
if pref == "filename":
|
||||||
|
name = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
return nfc(name.strip())
|
||||||
|
|
||||||
|
# auto: id > title > filename
|
||||||
|
if "id" in front and str(front["id"]).strip():
|
||||||
|
return nfc(str(front["id"]).strip())
|
||||||
|
if "title" in front and str(front["title"]).strip():
|
||||||
|
return nfc(str(front["title"]).strip())
|
||||||
|
name = os.path.splitext(os.path.basename(path))[0]
|
||||||
|
return nfc(name.strip())
|
||||||
|
|
||||||
|
def load_index(root: str, key_pref: str) -> Dict[str, Tuple[str, Dict[str, Any], str]]:
|
||||||
|
"""
|
||||||
|
Baut ein Index-Dict: key -> (path, frontmatter, body)
|
||||||
|
"""
|
||||||
|
idx: Dict[str, Tuple[str, Dict[str, Any], str]] = {}
|
||||||
|
for p in iter_md_files(root):
|
||||||
|
try:
|
||||||
|
fm, body = read_markdown(p)
|
||||||
|
k = key_from(fm or {}, p, key_pref)
|
||||||
|
if not k:
|
||||||
|
# Fallback: filename
|
||||||
|
k = key_from({}, p, "filename")
|
||||||
|
idx[k] = (p, fm or {}, body or "")
|
||||||
|
except Exception as e:
|
||||||
|
# Datei nicht parsebar -> trotzdem indexieren nach Dateiname (leer FM/Body)
|
||||||
|
k = key_from({}, p, key_pref if key_pref != "auto" else "filename")
|
||||||
|
idx[k] = (p, {}, "")
|
||||||
|
return idx
|
||||||
|
|
||||||
|
def compare_entries(src: Tuple[str, Dict[str, Any], str],
|
||||||
|
dst: Tuple[str, Dict[str, Any], str],
|
||||||
|
focus: str) -> Dict[str, Any]:
|
||||||
|
sp, sfm, sbody = src
|
||||||
|
dp, dfm, dbody = dst
|
||||||
|
|
||||||
|
res = {"status": "ok"} # wird 'diff' falls Unterschiede
|
||||||
|
diffs = {}
|
||||||
|
|
||||||
|
if focus in ("frontmatter", "all"):
|
||||||
|
# Vergleiche FM minimal: id und title
|
||||||
|
s_id = nfc(str(sfm.get("id", "")).strip())
|
||||||
|
d_id = nfc(str(dfm.get("id", "")).strip())
|
||||||
|
s_title = nfc(str(sfm.get("title", "")).strip())
|
||||||
|
d_title = nfc(str(dfm.get("title", "")).strip())
|
||||||
|
if s_id != d_id:
|
||||||
|
diffs["frontmatter.id"] = {"src": s_id, "dst": d_id}
|
||||||
|
if s_title != d_title:
|
||||||
|
diffs["frontmatter.title"] = {"src": s_title, "dst": d_title}
|
||||||
|
|
||||||
|
if focus in ("body", "all"):
|
||||||
|
if norm_text(sbody) != norm_text(dbody):
|
||||||
|
diffs["body"] = {"src_len": len(sbody or ""), "dst_len": len(dbody or "")}
|
||||||
|
|
||||||
|
if diffs:
|
||||||
|
res["status"] = "diff"
|
||||||
|
res["diffs"] = diffs
|
||||||
|
res["src_path"] = sp
|
||||||
|
res["dst_path"] = dp
|
||||||
|
return res
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--src", required=True, help="Quell-Vault-Ordner")
|
||||||
|
ap.add_argument("--dst", required=True, help="Export-Vault-Ordner")
|
||||||
|
ap.add_argument("--focus", choices=["body", "frontmatter", "all"], default="body")
|
||||||
|
ap.add_argument("--key", choices=["auto", "id", "title", "filename"], default="auto",
|
||||||
|
help="Vergleichsschlüssel (default: auto=id>title>filename)")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
src_idx = load_index(args.src, args.key)
|
||||||
|
dst_idx = load_index(args.dst, args.key)
|
||||||
|
|
||||||
|
src_keys = set(src_idx.keys())
|
||||||
|
dst_keys = set(dst_idx.keys())
|
||||||
|
|
||||||
|
# Fehlende
|
||||||
|
only_src = sorted(src_keys - dst_keys)
|
||||||
|
only_dst = sorted(dst_keys - src_keys)
|
||||||
|
|
||||||
|
count = 0
|
||||||
|
for k in only_src:
|
||||||
|
print(json.dumps({"note_id": k, "status": "missing", "src": True, "dst": False}, ensure_ascii=False))
|
||||||
|
count += 1
|
||||||
|
for k in only_dst:
|
||||||
|
print(json.dumps({"note_id": k, "status": "missing", "src": False, "dst": True}, ensure_ascii=False))
|
||||||
|
count += 1
|
||||||
|
|
||||||
|
# Vergleiche, wo vorhanden
|
||||||
|
diff_count = 0
|
||||||
|
common = sorted(src_keys & dst_keys)
|
||||||
|
for k in common:
|
||||||
|
res = compare_entries(src_idx[k], dst_idx[k], args.focus)
|
||||||
|
if res["status"] == "diff":
|
||||||
|
res["note_id"] = k
|
||||||
|
print(json.dumps(res, ensure_ascii=False))
|
||||||
|
diff_count += 1
|
||||||
|
|
||||||
|
summary = {
|
||||||
|
"summary": "OK" if (count == 0 and diff_count == 0) else "DIFFS",
|
||||||
|
"missing_count": count,
|
||||||
|
"diff_count": diff_count,
|
||||||
|
"total_src": len(src_keys),
|
||||||
|
"total_dst": len(dst_keys),
|
||||||
|
"focus": args.focus,
|
||||||
|
"key": args.key,
|
||||||
|
}
|
||||||
|
print(json.dumps(summary, ensure_ascii=False))
|
||||||
|
# Exit-Code für CI
|
||||||
|
if summary["summary"] != "OK":
|
||||||
|
# non-zero exit optional; hier 0 lassen, damit CLI-Ausgaben sichtbar bleiben
|
||||||
|
pass
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
# Sorgt dafür, dass 'app' als Paket importierbar ist,
|
||||||
|
# wenn pytest aus dem Repo-Root oder aus 'tests/' gestartet wird.
|
||||||
|
import os, sys
|
||||||
|
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, ROOT)
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Kleine Diagnosehilfe für mindnet_edges.
|
||||||
|
|
||||||
|
Nutzung (aus Projektwurzel):
|
||||||
|
|
||||||
|
(.venv) python tests/diag_edges_for_note.py \
|
||||||
|
--note-id 20251130-relshow-embeddings-101
|
||||||
|
|
||||||
|
Optional:
|
||||||
|
|
||||||
|
--prefix mindnet
|
||||||
|
|
||||||
|
Ziel:
|
||||||
|
- Zeigt alle Edges aus der Collection <prefix>_edges,
|
||||||
|
die sich auf die angegebene note_id beziehen:
|
||||||
|
* note_id == NOTE
|
||||||
|
* oder source_id == NOTE
|
||||||
|
* oder target_id == NOTE
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from typing import List
|
||||||
|
|
||||||
|
# Projekt-Root (Ordner mit "app/") zum sys.path hinzufügen,
|
||||||
|
# damit "import app.core.qdrant" funktioniert, auch wenn das Script
|
||||||
|
# direkt aus tests/ gestartet wird.
|
||||||
|
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
PROJECT_ROOT = os.path.dirname(THIS_DIR)
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
from qdrant_client.http import models as rest # type: ignore
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client, collection_names
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_edges_for_note(note_id: str, prefix: str | None, limit: int = 512) -> List[dict]:
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
if prefix:
|
||||||
|
cfg.prefix = prefix
|
||||||
|
|
||||||
|
client = get_client(cfg)
|
||||||
|
_, _, edges_col = collection_names(cfg.prefix)
|
||||||
|
|
||||||
|
conditions = []
|
||||||
|
for field in ("note_id", "source_id", "target_id"):
|
||||||
|
conditions.append(
|
||||||
|
rest.FieldCondition(
|
||||||
|
key=field,
|
||||||
|
match=rest.MatchValue(value=str(note_id)),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
flt = rest.Filter(
|
||||||
|
should=conditions,
|
||||||
|
)
|
||||||
|
|
||||||
|
points, _ = client.scroll(
|
||||||
|
collection_name=edges_col,
|
||||||
|
scroll_filter=flt,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
out: List[dict] = []
|
||||||
|
for p in points or []:
|
||||||
|
pl = dict(p.payload or {})
|
||||||
|
if pl:
|
||||||
|
out.append(pl)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--note-id", required=True, help="Note-ID, z. B. 20251130-relshow-embeddings-101")
|
||||||
|
parser.add_argument("--prefix", default=None, help="Qdrant-Prefix (Default: aus ENV/QdrantConfig)")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
note_id = args.note_id
|
||||||
|
prefix = args.prefix
|
||||||
|
|
||||||
|
edges = fetch_edges_for_note(note_id, prefix)
|
||||||
|
print(f"Prefix: {prefix or '(aus ENV)'}")
|
||||||
|
print(f"Note-ID: {note_id}")
|
||||||
|
print(f"Gefundene Edges: {len(edges)}")
|
||||||
|
print("-" * 60)
|
||||||
|
|
||||||
|
for i, e in enumerate(edges, start=1):
|
||||||
|
print(f"[{i}] kind={e.get('kind')} confidence={e.get('confidence')}")
|
||||||
|
print(f" note_id = {e.get('note_id')}")
|
||||||
|
print(f" source_id= {e.get('source_id')}")
|
||||||
|
print(f" target_id= {e.get('target_id')}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/diag_payload_indexes.py (v1.1)
|
||||||
|
|
||||||
|
Zeigt payload_schema (Indizes) je Collection.
|
||||||
|
WICHTIG: Einige Qdrant-Versionen liefern payload_schema nur, wenn
|
||||||
|
`with_payload_schema=true` gesetzt wird. Daher setzen wir das Flag explizit.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, json
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client, collection_names
|
||||||
|
|
||||||
|
def compact_schema(ps: dict | None) -> dict:
|
||||||
|
if not isinstance(ps, dict):
|
||||||
|
return {}
|
||||||
|
out = {}
|
||||||
|
for k, v in ps.items():
|
||||||
|
if isinstance(v, dict) and "type" in v:
|
||||||
|
out[k] = v["type"]
|
||||||
|
else:
|
||||||
|
out[k] = v
|
||||||
|
return out
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--raw", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
notes, chunks, edges = collection_names(cfg.prefix)
|
||||||
|
|
||||||
|
cols = [notes, chunks, edges]
|
||||||
|
result = []
|
||||||
|
for name in cols:
|
||||||
|
info = client.get_collection(collection_name=name, with_payload_schema=True)
|
||||||
|
d = info.model_dump() if hasattr(info, "model_dump") else (info.dict() if hasattr(info, "dict") else info)
|
||||||
|
payload_schema = (d.get("result") or {}).get("payload_schema")
|
||||||
|
vectors = (d.get("result") or {}).get("vectors")
|
||||||
|
if isinstance(vectors, dict) and "config" in vectors:
|
||||||
|
vectors = vectors.get("config")
|
||||||
|
if args.raw:
|
||||||
|
result.append({"collection": name, "raw": d})
|
||||||
|
else:
|
||||||
|
result.append({
|
||||||
|
"name": name,
|
||||||
|
"vectors": vectors,
|
||||||
|
"payload_schema": compact_schema(payload_schema),
|
||||||
|
"segments_count": (d.get("result") or {}).get("segments_count"),
|
||||||
|
})
|
||||||
|
print(json.dumps({"prefix": cfg.prefix, "collections": result}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Diagnose für den Graph-Adapter (Subgraph) in mindnet.
|
||||||
|
|
||||||
|
Nutzung (aus Projektwurzel):
|
||||||
|
|
||||||
|
(.venv) python tests/diag_subgraph_for_note.py \
|
||||||
|
--note-id 20251130-relshow-embeddings-101 \
|
||||||
|
--depth 1
|
||||||
|
|
||||||
|
Optional:
|
||||||
|
|
||||||
|
--prefix mindnet
|
||||||
|
--edge-type references --edge-type related_to ...
|
||||||
|
|
||||||
|
Ziel:
|
||||||
|
- Ruft app.core.graph_adapter.expand(...) für eine gegebene note_id auf.
|
||||||
|
- Gibt edge_bonus und centrality_bonus für diese note_id aus.
|
||||||
|
- Zeigt zusätzlich einige Nachbarn und deren Boni.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from typing import List, Optional, Tuple
|
||||||
|
|
||||||
|
# sys.path anpassen, damit "import app...." aus tests/ funktioniert
|
||||||
|
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
PROJECT_ROOT = os.path.dirname(THIS_DIR)
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client # type: ignore
|
||||||
|
import app.core.graph_adapter as ga # type: ignore
|
||||||
|
|
||||||
|
|
||||||
|
def build_subgraph_for_note(
|
||||||
|
note_id: str,
|
||||||
|
prefix: Optional[str],
|
||||||
|
depth: int,
|
||||||
|
edge_types: Optional[List[str]] = None,
|
||||||
|
) -> Tuple[ga.Subgraph, str]:
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
if prefix:
|
||||||
|
cfg.prefix = prefix
|
||||||
|
|
||||||
|
client = get_client(cfg)
|
||||||
|
sg = ga.expand(client, cfg.prefix, [note_id], depth=depth, edge_types=edge_types)
|
||||||
|
return sg, cfg.prefix
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument(
|
||||||
|
"--note-id",
|
||||||
|
required=True,
|
||||||
|
help="Note-ID, z. B. 20251130-relshow-embeddings-101",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--depth",
|
||||||
|
type=int,
|
||||||
|
default=1,
|
||||||
|
help="Expansionstiefe (Default: 1)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--prefix",
|
||||||
|
default=None,
|
||||||
|
help="Qdrant-Prefix (Default: aus ENV/QdrantConfig)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--edge-type",
|
||||||
|
action="append",
|
||||||
|
default=None,
|
||||||
|
help="Edge-Typ-Filter (kann mehrfach angegeben werden), z. B. --edge-type references",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
note_id = args.note_id
|
||||||
|
depth = max(0, int(args.depth))
|
||||||
|
prefix = args.prefix
|
||||||
|
edge_types = args.edge_type
|
||||||
|
|
||||||
|
sg, eff_prefix = build_subgraph_for_note(note_id, prefix, depth, edge_types=edge_types)
|
||||||
|
|
||||||
|
print(f"Prefix: {eff_prefix}")
|
||||||
|
print(f"Note-ID (Seed): {note_id}")
|
||||||
|
print(f"Depth: {depth}")
|
||||||
|
print(f"Edge-Types-Filter: {edge_types or '(keiner)'}")
|
||||||
|
print("-" * 60)
|
||||||
|
|
||||||
|
# Boni für die Seed-Note
|
||||||
|
eb = sg.edge_bonus(note_id)
|
||||||
|
cb = sg.centrality_bonus(note_id)
|
||||||
|
print(f"edge_bonus({note_id}) = {eb:.4f}")
|
||||||
|
print(f"centrality_bonus({note_id}) = {cb:.4f}")
|
||||||
|
print("-" * 60)
|
||||||
|
|
||||||
|
# Ein paar weitere Knoten anzeigen (falls vorhanden)
|
||||||
|
print("Beispiele weiterer Knoten mit Boni (max. 10):")
|
||||||
|
printed = 0
|
||||||
|
for node, edges in sg.adj.items():
|
||||||
|
if node == note_id:
|
||||||
|
continue
|
||||||
|
if printed >= 10:
|
||||||
|
break
|
||||||
|
eb2 = sg.edge_bonus(node)
|
||||||
|
cb2 = sg.centrality_bonus(node)
|
||||||
|
print(f"- node={node}")
|
||||||
|
print(f" edge_bonus={eb2:.4f} centrality={cb2:.4f}")
|
||||||
|
printed += 1
|
||||||
|
|
||||||
|
if printed == 0:
|
||||||
|
print("(keine weiteren Knoten im Subgraph registriert)")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
dump_edges_payload_sample.py
|
||||||
|
-----------------------------------
|
||||||
|
Zeigt einige Beispiel-Payloads aus der {prefix}_edges Collection,
|
||||||
|
um Feldnamen (edge_type, src_note_id, src_id, etc.) zu verifizieren.
|
||||||
|
|
||||||
|
Nutzung:
|
||||||
|
python3 tests/dump_edges_payload_sample.py --prefix mindnet --n 10
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
try:
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"error": f"qdrant-client not installed: {e}"}))
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
import argparse
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", default=os.environ.get("COLLECTION_PREFIX") or os.environ.get("MINDNET_PREFIX") or "mindnet")
|
||||||
|
ap.add_argument("--n", type=int, default=10)
|
||||||
|
args = ap.parse_args(argv)
|
||||||
|
|
||||||
|
url = os.environ.get("QDRANT_URL", "http://127.0.0.1:6333").strip()
|
||||||
|
api_key = os.environ.get("QDRANT_API_KEY") or None
|
||||||
|
edges = f"{args.prefix}_edges"
|
||||||
|
|
||||||
|
client = QdrantClient(url=url, api_key=api_key, timeout=30)
|
||||||
|
|
||||||
|
left = args.n
|
||||||
|
offset = None
|
||||||
|
while left > 0:
|
||||||
|
limit = min(left, 256)
|
||||||
|
pts, offset = client.scroll(
|
||||||
|
collection_name=edges,
|
||||||
|
scroll_filter=None,
|
||||||
|
limit=limit,
|
||||||
|
offset=offset,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
for p in pts:
|
||||||
|
print(json.dumps({"point_id": str(p.id), "payload_keys": list((p.payload or {}).keys()), "payload": p.payload}, ensure_ascii=False))
|
||||||
|
left -= len(pts)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/ensure_indexes_and_show.py (compat v1.0.1)
|
||||||
|
|
||||||
|
Erzwingt ensure_payload_indexes(...) und zeigt danach payload_schema je Collection.
|
||||||
|
Kompatibel mit qdrant-client Versionen ohne 'with_payload_schema' im Wrapper.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import urllib.request
|
||||||
|
import urllib.error
|
||||||
|
from typing import Any, Dict
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client, ensure_payload_indexes, collection_names
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_model_dump(obj: Any) -> Dict[str, Any]:
|
||||||
|
if hasattr(obj, "model_dump"):
|
||||||
|
return obj.model_dump()
|
||||||
|
if hasattr(obj, "dict"):
|
||||||
|
return obj.dict()
|
||||||
|
return obj if isinstance(obj, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg_base_url(cfg) -> str:
|
||||||
|
if getattr(cfg, "url", None):
|
||||||
|
return cfg.url.rstrip("/")
|
||||||
|
host = getattr(cfg, "host", None) or os.getenv("QDRANT_HOST") or "127.0.0.1"
|
||||||
|
port = getattr(cfg, "port", None) or os.getenv("QDRANT_PORT") or "6333"
|
||||||
|
return f"http://{host}:{port}"
|
||||||
|
|
||||||
|
|
||||||
|
def _http_get(url: str, api_key: str | None) -> Dict[str, Any]:
|
||||||
|
req = urllib.request.Request(url, method="GET")
|
||||||
|
if api_key:
|
||||||
|
req.add_header("api-key", api_key)
|
||||||
|
req.add_header("Authorization", f"Bearer {api_key}")
|
||||||
|
req.add_header("Accept", "application/json")
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
data = resp.read()
|
||||||
|
return json.loads(data.decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def get_collection_payload_schema(client, cfg, name: str) -> Dict[str, Any] | None:
|
||||||
|
# Strategy 1: OpenAPI client with flag
|
||||||
|
try:
|
||||||
|
oc = getattr(client, "openapi_client", None)
|
||||||
|
if oc is not None and hasattr(oc, "collections_api"):
|
||||||
|
api = oc.collections_api
|
||||||
|
info = api.get_collection(collection_name=name, with_payload_schema=True)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
return (d.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# Strategy 2: wrapper
|
||||||
|
try:
|
||||||
|
info = client.get_collection(collection_name=name)
|
||||||
|
d = _safe_model_dump(info)
|
||||||
|
ps = (d.get("result") or {}).get("payload_schema")
|
||||||
|
if ps is not None:
|
||||||
|
return ps
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# Strategy 3: direct HTTP
|
||||||
|
try:
|
||||||
|
base = _cfg_base_url(cfg)
|
||||||
|
url = f"{base}/collections/{name}?with_payload_schema=true"
|
||||||
|
raw = _http_get(url, getattr(cfg, "api_key", None))
|
||||||
|
return (raw.get("result") or {}).get("payload_schema")
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
ensure_payload_indexes(client, cfg.prefix)
|
||||||
|
notes, chunks, edges = collection_names(cfg.prefix)
|
||||||
|
res = {}
|
||||||
|
for name in (notes, chunks, edges):
|
||||||
|
res[name] = get_collection_payload_schema(client, cfg, name)
|
||||||
|
print(json.dumps(res, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/inspect_one_note.py
|
||||||
|
Diagnose einer einzelnen Markdown-Datei (kein Upsert).
|
||||||
|
Usage:
|
||||||
|
python3 tests/inspect_one_note.py --path ./test_vault/30_projects/project-demo.md
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import os, sys, json, argparse
|
||||||
|
|
||||||
|
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
from app.core.parser import read_markdown, normalize_frontmatter, validate_required_frontmatter
|
||||||
|
from app.core.chunker import assemble_chunks
|
||||||
|
from app.core.chunk_payload import make_chunk_payloads
|
||||||
|
try:
|
||||||
|
from app.core.derive_edges import build_edges_for_note
|
||||||
|
except Exception:
|
||||||
|
from app.core.edges import build_edges_for_note # type: ignore
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--path", required=True)
|
||||||
|
args = ap.parse_args()
|
||||||
|
p = os.path.abspath(args.path)
|
||||||
|
|
||||||
|
try:
|
||||||
|
parsed = read_markdown(p)
|
||||||
|
if parsed is None:
|
||||||
|
print(json.dumps({"path": p, "error": "read_markdown returned None"}))
|
||||||
|
return
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": p, "error": f"read_markdown failed: {type(e).__name__}: {e}"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
fm = normalize_frontmatter(parsed.frontmatter)
|
||||||
|
validate_required_frontmatter(fm)
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"path": p, "error": f"frontmatter invalid: {type(e).__name__}: {e}"}))
|
||||||
|
return
|
||||||
|
|
||||||
|
body = getattr(parsed, "body", "") or ""
|
||||||
|
chunks = assemble_chunks(fm["id"], body, fm.get("type", "concept"))
|
||||||
|
rel_for_payload = os.path.relpath(p, os.path.join(p, "..", ".."))
|
||||||
|
chunk_pls = make_chunk_payloads(fm, rel_for_payload, chunks, note_text=body)
|
||||||
|
|
||||||
|
edge_error = None
|
||||||
|
edges_count = 0
|
||||||
|
try:
|
||||||
|
edges = build_edges_for_note(fm["id"], chunk_pls, include_note_scope_refs=True)
|
||||||
|
edges_count = len(edges)
|
||||||
|
except Exception as e:
|
||||||
|
edge_error = f"{type(e).__name__}: {e}"
|
||||||
|
|
||||||
|
print(json.dumps({
|
||||||
|
"path": p,
|
||||||
|
"id": fm.get("id"),
|
||||||
|
"title": fm.get("title"),
|
||||||
|
"body_len": len(body),
|
||||||
|
"chunks": len(chunk_pls),
|
||||||
|
"edges": edges_count,
|
||||||
|
"edge_error": edge_error
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/list_md_seen_by_importer.py
|
||||||
|
Zeigt, welche Markdown-Dateien der Importer sehen würde (gleiche Filter & Sortierung).
|
||||||
|
Usage:
|
||||||
|
python3 tests/list_md_seen_by_importer.py --vault ./test_vault
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, os, sys, json
|
||||||
|
|
||||||
|
def iter_md(root: str):
|
||||||
|
out = []
|
||||||
|
for dirpath, _, filenames in os.walk(root):
|
||||||
|
for fn in filenames:
|
||||||
|
p = os.path.join(dirpath, fn)
|
||||||
|
pn = p.replace("\\", "/")
|
||||||
|
item = {
|
||||||
|
"path": os.path.abspath(p),
|
||||||
|
"rel": os.path.relpath(p, root).replace("\\", "/"),
|
||||||
|
"included": False,
|
||||||
|
"reason": "",
|
||||||
|
}
|
||||||
|
if not fn.lower().endswith(".md"):
|
||||||
|
item["reason"] = "skip_ext"
|
||||||
|
elif "/.obsidian/" in pn or "/_backup_frontmatter/" in pn or "/_imported/" in pn:
|
||||||
|
item["reason"] = "skip_special_dir"
|
||||||
|
else:
|
||||||
|
item["included"] = True
|
||||||
|
item["reason"] = "ok"
|
||||||
|
out.append(item)
|
||||||
|
# auch die ausgeschlossenen zeigen:
|
||||||
|
if not item["included"]:
|
||||||
|
print(json.dumps(item, ensure_ascii=False))
|
||||||
|
out_sorted = sorted(out, key=lambda x: x["rel"])
|
||||||
|
for i, it in enumerate(out_sorted):
|
||||||
|
it["order"] = i
|
||||||
|
print(json.dumps(it, ensure_ascii=False))
|
||||||
|
print(json.dumps({"summary":"done","included":len(out_sorted)}, ensure_ascii=False))
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--vault", required=True)
|
||||||
|
args = ap.parse_args()
|
||||||
|
root = os.path.abspath(args.vault)
|
||||||
|
iter_md(root)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# tests/run_e2e_roundtrip.sh
|
||||||
|
# End-to-End Testpipeline für mindnet WP-03
|
||||||
|
#
|
||||||
|
# Schritte:
|
||||||
|
# 1) Qdrant truncate (Prefix!)
|
||||||
|
# 2) Import (Apply + Purge) => Create-Fall
|
||||||
|
# 3) Import (Apply - ohne Änderungen) => Idempotenz
|
||||||
|
# 4) Hash-Reporter (Fehlende Hash-Varianten)
|
||||||
|
# 5) Export nach _exportVault
|
||||||
|
# 6) Vergleich Vault vs. Export (Body-Fokus)
|
||||||
|
# 7) Sync-Deletes: Eine Datei temporär „entfernen“, Dry-Run + Apply, dann zurückrollen
|
||||||
|
#
|
||||||
|
# Aufrufe:
|
||||||
|
# ./tests/run_e2e_roundtrip.sh [--vault ./test_vault] [--prefix mindnet]
|
||||||
|
#
|
||||||
|
# ENV:
|
||||||
|
# COLLECTION_PREFIX (Default mindnet)
|
||||||
|
|
||||||
|
VAULT="./test_vault"
|
||||||
|
PREFIX="${COLLECTION_PREFIX:-mindnet}"
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--vault) VAULT="$2"; shift 2;;
|
||||||
|
--prefix) PREFIX="$2"; shift 2;;
|
||||||
|
*) echo "Unbekannte Option: $1" >&2; exit 2;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "== E2E Roundtrip =="
|
||||||
|
echo "Vault: $VAULT"
|
||||||
|
echo "Prefix: $PREFIX"
|
||||||
|
echo
|
||||||
|
|
||||||
|
export COLLECTION_PREFIX="$PREFIX"
|
||||||
|
|
||||||
|
echo "-- 1) Qdrant truncate"
|
||||||
|
python3 -m scripts.reset_qdrant --mode truncate --prefix "$PREFIX"
|
||||||
|
python3 -m scripts.debug_qdrant_state --prefix "$PREFIX"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "-- 2) Import (Create-Fall)"
|
||||||
|
python3 -m scripts.import_markdown --vault "$VAULT" --apply --purge-before-upsert --prefix "$PREFIX"
|
||||||
|
python3 -m scripts.debug_qdrant_state --prefix "$PREFIX"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "-- 3) Import (Idempotenz: keine Änderungen erwartet)"
|
||||||
|
python3 -m scripts.import_markdown --vault "$VAULT" --apply --prefix "$PREFIX"
|
||||||
|
python3 -m scripts.debug_qdrant_state --prefix "$PREFIX"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "-- 4) Hash-Reporter"
|
||||||
|
python3 -m scripts.report_hashes --prefix "$PREFIX" || true
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "-- 5) Export nach _exportVault"
|
||||||
|
OUT_DIR="_exportVault"
|
||||||
|
rm -rf "$OUT_DIR"
|
||||||
|
python3 -m scripts.export_markdown --out "$OUT_DIR" --prefix "$PREFIX"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "-- 6) Vergleich Vault vs. Export (Body-Fokus)"
|
||||||
|
python3 tests/compare_vaults.py --src "$VAULT" --dst "$OUT_DIR" --focus body
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "-- 7) Sync-Deletes (simuliert)"
|
||||||
|
# wähle eine Datei und verschiebe sie temporär weg
|
||||||
|
TMP_DIR="$(mktemp -d)"
|
||||||
|
SAMPLE_MD="$(find "$VAULT" -type f -name '*.md' | head -n 1 || true)"
|
||||||
|
if [[ -n "${SAMPLE_MD:-}" ]]; then
|
||||||
|
mv "$SAMPLE_MD" "$TMP_DIR/"
|
||||||
|
echo "Temporär entfernt: $(basename "$SAMPLE_MD")"
|
||||||
|
|
||||||
|
echo " 7a) Dry-Run"
|
||||||
|
python3 -m scripts.import_markdown --vault "$VAULT" --sync-deletes --prefix "$PREFIX"
|
||||||
|
echo " 7b) Apply"
|
||||||
|
python3 -m scripts.import_markdown --vault "$VAULT" --sync-deletes --apply --prefix "$PREFIX"
|
||||||
|
|
||||||
|
# zurückrollen
|
||||||
|
mv "$TMP_DIR/$(basename "$SAMPLE_MD")" "$SAMPLE_MD"
|
||||||
|
rm -rf "$TMP_DIR"
|
||||||
|
|
||||||
|
echo " 7c) Re-Import (wieder anlegen)"
|
||||||
|
python3 -m scripts.import_markdown --vault "$VAULT" --apply --prefix "$PREFIX"
|
||||||
|
python3 -m scripts.debug_qdrant_state --prefix "$PREFIX"
|
||||||
|
else
|
||||||
|
echo "Hinweis: Keine MD-Datei im Vault gefunden – Sync-Deletes übersprungen."
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "== E2E Roundtrip abgeschlossen =="
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/test_retriever_smoke.py - einfacher Smoke-Test für den mindnet-Retriever
|
||||||
|
|
||||||
|
Zweck:
|
||||||
|
- Startet eine Query gegen die laufende FastAPI-Instanz
|
||||||
|
- Prüft, ob der /query-Endpunkt antwortet und sinnvolle Treffer liefert
|
||||||
|
- Gibt Top-K-Ergebnisse inkl. Scores auf stdout aus
|
||||||
|
|
||||||
|
Verwendung:
|
||||||
|
1) Stelle sicher, dass der mindnet-Service läuft, z. B.:
|
||||||
|
uvicorn app.main:create_app --factory --reload
|
||||||
|
|
||||||
|
2) Führe dieses Skript aus:
|
||||||
|
python scripts/test_retriever_smoke.py --query "karate" --mode hybrid
|
||||||
|
|
||||||
|
3) Optional:
|
||||||
|
python scripts/test_retriever_smoke.py --query "embeddings" --mode semantic --top-k 5
|
||||||
|
|
||||||
|
Hinweis:
|
||||||
|
- Standard-URL ist http://127.0.0.1:8000
|
||||||
|
- Endpoint ist /query (bestehender Router), der QueryRequest erwartet
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from typing import Any, Dict
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
def _build_payload(args: argparse.Namespace) -> Dict[str, Any]:
|
||||||
|
payload: Dict[str, Any] = {
|
||||||
|
"mode": args.mode,
|
||||||
|
"query": args.query,
|
||||||
|
"top_k": args.top_k,
|
||||||
|
}
|
||||||
|
if args.expand_depth is not None and args.expand_depth > 0:
|
||||||
|
expand: Dict[str, Any] = {"depth": args.expand_depth}
|
||||||
|
if args.expand_edge_types:
|
||||||
|
expand["edge_types"] = args.expand_edge_types
|
||||||
|
payload["expand"] = expand
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Smoke-Test für mindnet-Retriever (/query)")
|
||||||
|
parser.add_argument("--url", default="http://127.0.0.1:8000/query", help="Basis-URL des Query-Endpunkts")
|
||||||
|
parser.add_argument("--query", required=True, help="Text-Query für den Retriever")
|
||||||
|
parser.add_argument("--mode", choices=["semantic", "hybrid"], default="hybrid", help="Retrieval-Modus")
|
||||||
|
parser.add_argument("--top-k", type=int, default=5, help="Anzahl der gewünschten Treffer")
|
||||||
|
parser.add_argument("--expand-depth", type=int, default=0, help="Edge-Expansionstiefe (nur hybrid)")
|
||||||
|
parser.add_argument(
|
||||||
|
"--expand-edge-types",
|
||||||
|
nargs="*",
|
||||||
|
default=None,
|
||||||
|
help="Liste von Edge-Typen für die Expansion (z. B. references depends_on)",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
payload = _build_payload(args)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = requests.post(args.url, json=payload, timeout=10)
|
||||||
|
except Exception as exc: # pragma: no cover - Netzwerkfehler
|
||||||
|
print(f"[ERROR] HTTP-Request fehlgeschlagen: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if resp.status_code != 200:
|
||||||
|
print(f"[ERROR] Unerwarteter Statuscode: {resp.status_code}", file=sys.stderr)
|
||||||
|
print(resp.text, file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = resp.json()
|
||||||
|
except Exception as exc: # pragma: no cover - JSON-Fehler
|
||||||
|
print(f"[ERROR] Antwort ist kein gültiges JSON: {exc}", file=sys.stderr)
|
||||||
|
print(resp.text, file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print("=== mindnet Retriever Smoke-Test ===")
|
||||||
|
print(f"URL: {args.url}")
|
||||||
|
print(f"Mode: {args.mode}")
|
||||||
|
print(f"Query: {args.query!r}")
|
||||||
|
print(f"Top-K: {args.top_k}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
results = data.get("results") or []
|
||||||
|
used_mode = data.get("used_mode")
|
||||||
|
latency_ms = data.get("latency_ms")
|
||||||
|
|
||||||
|
print(f"used_mode = {used_mode}, latency = {latency_ms} ms")
|
||||||
|
print(f"Treffer insgesamt: {len(results)}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
for i, hit in enumerate(results, start=1):
|
||||||
|
note_id = hit.get("note_id")
|
||||||
|
node_id = hit.get("node_id")
|
||||||
|
total = hit.get("total_score")
|
||||||
|
sem = hit.get("semantic_score")
|
||||||
|
edge_bonus = hit.get("edge_bonus")
|
||||||
|
cent_bonus = hit.get("centrality_bonus")
|
||||||
|
src = hit.get("source") or {}
|
||||||
|
path = src.get("path")
|
||||||
|
section = src.get("section")
|
||||||
|
|
||||||
|
print(f"[{i}] note_id={note_id} node_id={node_id}")
|
||||||
|
print(f" total={total:.4f} (semantic={sem:.4f} edge={edge_bonus:.4f} centrality={cent_bonus:.4f})")
|
||||||
|
if path or section:
|
||||||
|
print(f" source: {path or '-'} :: {section or '-'}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__": # pragma: no cover
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/show_edges_for_note.py
|
||||||
|
|
||||||
|
Zeigt Kanten einer Note (gefiltert nach kind/scope) in einer kompakten Form.
|
||||||
|
Aufrufbeispiele:
|
||||||
|
python3 -m scripts.show_edges_for_note --note-id 20251110-ollama-llm-9f0a12 --kinds references,next,prev --limit 10
|
||||||
|
python3 -m scripts.show_edges_for_note --title "Qdrant Vektordatenbank" --scope note
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, json, os, sys
|
||||||
|
from typing import Dict, Any, List, Tuple
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def find_note_by_title(client, prefix: str, title: str) -> str | None:
|
||||||
|
notes_col, _, _ = collections(prefix)
|
||||||
|
f = rest.Filter(must=[rest.FieldCondition(key="title", match=rest.MatchText(text=title))])
|
||||||
|
pts, _ = client.scroll(collection_name=notes_col, scroll_filter=f, with_payload=True, with_vectors=False, limit=1)
|
||||||
|
if not pts:
|
||||||
|
return None
|
||||||
|
return pts[0].payload.get("note_id")
|
||||||
|
|
||||||
|
def fetch_edges_for_note(client, prefix: str, note_id: str, kinds: List[str] | None, scope: str | None, limit: int) -> List[Dict[str, Any]]:
|
||||||
|
_, _, edges_col = collections(prefix)
|
||||||
|
must = [rest.FieldCondition(key="note_id", match=rest.MatchValue(value=note_id))]
|
||||||
|
if scope:
|
||||||
|
must.append(rest.FieldCondition(key="scope", match=rest.MatchValue(value=scope)))
|
||||||
|
if kinds:
|
||||||
|
must.append(rest.FieldCondition(key="kind", match=rest.MatchAny(any=kinds)))
|
||||||
|
f = rest.Filter(must=must)
|
||||||
|
pts, _ = client.scroll(collection_name=edges_col, scroll_filter=f, with_payload=True, with_vectors=False, limit=limit)
|
||||||
|
return [p.payload for p in pts]
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--note-id")
|
||||||
|
ap.add_argument("--title")
|
||||||
|
ap.add_argument("--kinds", help="CSV: references,next,prev,belongs_to,backlink")
|
||||||
|
ap.add_argument("--scope", choices=["note","chunk"])
|
||||||
|
ap.add_argument("--limit", type=int, default=25)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
nid = args.note_id
|
||||||
|
if not nid and args.title:
|
||||||
|
nid = find_note_by_title(client, cfg.prefix, args.title)
|
||||||
|
if not nid:
|
||||||
|
print(json.dumps({"error": f"note with title '{args.title}' not found"}))
|
||||||
|
sys.exit(2)
|
||||||
|
if not nid:
|
||||||
|
print(json.dumps({"error": "please provide --note-id or --title"}))
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
kinds = None
|
||||||
|
if args.kinds:
|
||||||
|
kinds = [s.strip() for s in args.kinds.split(",") if s.strip()]
|
||||||
|
|
||||||
|
edges = fetch_edges_for_note(client, cfg.prefix, nid, kinds, args.scope, args.limit)
|
||||||
|
print(json.dumps({"note_id": nid, "count": len(edges), "edges": edges}, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/test_edges_all.py
|
||||||
|
Ein knapper Integrationscheck:
|
||||||
|
- Es existieren Notes/Chunks/Edges
|
||||||
|
- Inline-Edges (rule_id startswith "inline:") werden erkannt
|
||||||
|
- Callout-Edges (rule_id == "callout:edge") werden erkannt
|
||||||
|
- Defaults (rule_id startswith "edge_defaults:") werden erkannt
|
||||||
|
- Strukturkanten stimmen (belongs_to == chunks; next == prev == chunks-1)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
from typing import Dict, Any, List, Tuple
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
|
||||||
|
def _scroll_all(client, collection: str):
|
||||||
|
pts_all = []
|
||||||
|
offset = None
|
||||||
|
while True:
|
||||||
|
pts, offset = client.scroll(
|
||||||
|
collection_name=collection,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=2048,
|
||||||
|
offset=offset,
|
||||||
|
)
|
||||||
|
pts_all.extend(pts or [])
|
||||||
|
if offset is None:
|
||||||
|
break
|
||||||
|
return pts_all
|
||||||
|
|
||||||
|
|
||||||
|
def _rule_group(rule_id: str) -> str:
|
||||||
|
if not rule_id:
|
||||||
|
return "unknown"
|
||||||
|
if rule_id == "callout:edge":
|
||||||
|
return "callout"
|
||||||
|
if rule_id.startswith("inline:"): # <—— wichtig für inline:rel
|
||||||
|
return "inline"
|
||||||
|
if rule_id.startswith("edge_defaults:"):
|
||||||
|
return "defaults"
|
||||||
|
if rule_id.startswith("explicit:"):
|
||||||
|
return "explicit"
|
||||||
|
if rule_id in ("structure:belongs_to", "structure:order"):
|
||||||
|
return "structure"
|
||||||
|
return "other"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
col_notes = f"{cfg.prefix}_notes"
|
||||||
|
col_chunks = f"{cfg.prefix}_chunks"
|
||||||
|
col_edges = f"{cfg.prefix}_edges"
|
||||||
|
|
||||||
|
notes_n = client.count(collection_name=col_notes, exact=True).count
|
||||||
|
chunks_pts = _scroll_all(client, col_chunks)
|
||||||
|
edges_pts = _scroll_all(client, col_edges)
|
||||||
|
|
||||||
|
ok = True
|
||||||
|
|
||||||
|
# Basisbedingungen
|
||||||
|
if notes_n == 0 or len(chunks_pts) == 0 or len(edges_pts) == 0:
|
||||||
|
ok = False
|
||||||
|
|
||||||
|
# Gruppen zählen
|
||||||
|
g = Counter(_rule_group((p.payload or {}).get("rule_id", "")) for p in edges_pts)
|
||||||
|
structure = g.get("structure", 0)
|
||||||
|
explicit = g.get("explicit", 0)
|
||||||
|
inline = g.get("inline", 0)
|
||||||
|
callout = g.get("callout", 0)
|
||||||
|
defaults = g.get("defaults", 0)
|
||||||
|
|
||||||
|
if structure == 0:
|
||||||
|
ok = False
|
||||||
|
# mindestens eine der expliziten Varianten vorhanden
|
||||||
|
if (explicit + inline + callout) == 0:
|
||||||
|
ok = False
|
||||||
|
# defaults dürfen 0 sein, wenn types.yaml keine edge_defaults liefert – daher nur Info
|
||||||
|
|
||||||
|
# per-note checks
|
||||||
|
chunks_by_note = Counter([p.payload.get("note_id") for p in chunks_pts if p.payload])
|
||||||
|
belongs = Counter(
|
||||||
|
(p.payload or {}).get("note_id")
|
||||||
|
for p in edges_pts
|
||||||
|
if (p.payload or {}).get("kind") == "belongs_to"
|
||||||
|
)
|
||||||
|
nxt = Counter(
|
||||||
|
(p.payload or {}).get("note_id")
|
||||||
|
for p in edges_pts
|
||||||
|
if (p.payload or {}).get("kind") == "next"
|
||||||
|
)
|
||||||
|
prv = Counter(
|
||||||
|
(p.payload or {}).get("note_id")
|
||||||
|
for p in edges_pts
|
||||||
|
if (p.payload or {}).get("kind") == "prev"
|
||||||
|
)
|
||||||
|
|
||||||
|
for n_id, c in chunks_by_note.items():
|
||||||
|
if belongs.get(n_id, 0) != c:
|
||||||
|
ok = False
|
||||||
|
if (nxt.get(n_id, 0) != max(c - 1, 0)) or (prv.get(n_id, 0) != max(c - 1, 0)):
|
||||||
|
ok = False
|
||||||
|
|
||||||
|
print(json.dumps({"ok": ok, "notes_checked": len(chunks_by_note)}, ensure_ascii=False))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
tests/test_edges_defaults_smoke.py
|
||||||
|
Aggregiert Edges je Relation (kind) in Qdrant zur schnellen Sichtprüfung.
|
||||||
|
Zeigt, ob typenbasierte edge_defaults-Relationen (z. B. depends_on/related_to) auftauchen.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
from collections import Counter
|
||||||
|
import json
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
cl = get_client(cfg)
|
||||||
|
col = f"{cfg.prefix}_edges"
|
||||||
|
pts, _ = cl.scroll(collection_name=col, with_payload=True, with_vectors=False, limit=10000)
|
||||||
|
by_rel = Counter()
|
||||||
|
for p in pts:
|
||||||
|
rel = p.payload.get("kind") or p.payload.get("relation") or "edge"
|
||||||
|
by_rel[rel] += 1
|
||||||
|
print(json.dumps({"prefix": cfg.prefix, "relations": dict(by_rel)}, ensure_ascii=False))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
|
||||||
|
def _rel(pl: dict) -> str:
|
||||||
|
return pl.get("relation") or pl.get("kind") or "edge"
|
||||||
|
|
||||||
|
|
||||||
|
def _scroll(client, col):
|
||||||
|
pts = []
|
||||||
|
next_page = None
|
||||||
|
while True:
|
||||||
|
res, next_page = client.scroll(
|
||||||
|
collection_name=col,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=1024,
|
||||||
|
offset=next_page,
|
||||||
|
)
|
||||||
|
pts.extend(res)
|
||||||
|
if next_page is None:
|
||||||
|
break
|
||||||
|
return pts
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
prefix = os.environ.get("COLLECTION_PREFIX", cfg.prefix)
|
||||||
|
|
||||||
|
cols = {
|
||||||
|
"notes": f"{prefix}_notes",
|
||||||
|
"chunks": f"{prefix}_chunks",
|
||||||
|
"edges": f"{prefix}_edges",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Index: notes -> title/type
|
||||||
|
notes_meta = {}
|
||||||
|
for p in _scroll(client, cols["notes"]):
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
if nid:
|
||||||
|
notes_meta[nid] = {
|
||||||
|
"title": pl.get("title", ""),
|
||||||
|
"type": pl.get("type", ""),
|
||||||
|
}
|
||||||
|
|
||||||
|
# chunks je note
|
||||||
|
chunks_by_note = defaultdict(int)
|
||||||
|
for p in _scroll(client, cols["chunks"]):
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
if nid:
|
||||||
|
chunks_by_note[nid] += 1
|
||||||
|
|
||||||
|
# edges je note
|
||||||
|
edges_by_note = defaultdict(list)
|
||||||
|
edges_all = _scroll(client, cols["edges"])
|
||||||
|
for p in edges_all:
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
if nid:
|
||||||
|
edges_by_note[nid].append(pl)
|
||||||
|
|
||||||
|
# pro note ausgeben
|
||||||
|
summary_edges = Counter()
|
||||||
|
total_chunks = 0
|
||||||
|
for nid in sorted(notes_meta.keys()):
|
||||||
|
meta = notes_meta[nid]
|
||||||
|
chunks = chunks_by_note.get(nid, 0)
|
||||||
|
total_chunks += chunks
|
||||||
|
|
||||||
|
kinds = Counter(_rel(pl) for pl in edges_by_note[nid])
|
||||||
|
summary_edges.update(kinds)
|
||||||
|
|
||||||
|
row = {
|
||||||
|
"note_id": nid,
|
||||||
|
"title": meta["title"],
|
||||||
|
"type": meta["type"],
|
||||||
|
"chunks": chunks,
|
||||||
|
"edges_by_kind": dict(kinds),
|
||||||
|
"checks": {
|
||||||
|
"belongs_to_equals_chunks": (kinds.get("belongs_to", 0) == chunks),
|
||||||
|
"next_prev_match": (kinds.get("next", 0) == kinds.get("prev", 0) == max(0, chunks - 1)),
|
||||||
|
"no_duplicate_edges": _no_dupes(edges_by_note[nid]),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
print(json.dumps(row, ensure_ascii=False))
|
||||||
|
|
||||||
|
# Gesamtsummary
|
||||||
|
total_notes = len(notes_meta)
|
||||||
|
out = {
|
||||||
|
"prefix": prefix,
|
||||||
|
"summary": {
|
||||||
|
"notes": total_notes,
|
||||||
|
"chunks": total_chunks,
|
||||||
|
"belongs_to": summary_edges.get("belongs_to", 0),
|
||||||
|
"next": summary_edges.get("next", 0),
|
||||||
|
"prev": summary_edges.get("prev", 0),
|
||||||
|
"refs_chunk": summary_edges.get("references", 0),
|
||||||
|
"refs_note": summary_edges.get("references_note", 0),
|
||||||
|
"backlink": summary_edges.get("backlink", 0),
|
||||||
|
"dup_edges": 0, # per-Note geprüft
|
||||||
|
},
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False))
|
||||||
|
|
||||||
|
|
||||||
|
def _no_dupes(pls):
|
||||||
|
seen = set()
|
||||||
|
for pl in pls:
|
||||||
|
key = (
|
||||||
|
str(pl.get("source_id") or ""),
|
||||||
|
str(pl.get("target_id") or ""),
|
||||||
|
str(pl.get("relation") or pl.get("kind") or ""),
|
||||||
|
str(pl.get("rule_id") or ""),
|
||||||
|
)
|
||||||
|
if key in seen:
|
||||||
|
return False
|
||||||
|
seen.add(key)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
scripts/test_edges_smoke_fast.py — Zählung über /count (sehr schnell)
|
||||||
|
|
||||||
|
Verwendet Qdrant 'count' API je Note/Kind (anstatt scroll), dadurch sehr schnelle Ausführung.
|
||||||
|
Optionen:
|
||||||
|
--max-notes N : prüft nur die ersten N Notizen
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import argparse, json
|
||||||
|
from typing import Dict, Any, List, Tuple
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
KINDS = ["belongs_to", "next", "prev", "references", "backlink"]
|
||||||
|
|
||||||
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def list_note_ids(client, notes_col: str, max_notes: int | None) -> List[Dict[str, Any]]:
|
||||||
|
pts, _ = client.scroll(collection_name=notes_col, with_payload=True, with_vectors=False, limit=max_notes or 1024)
|
||||||
|
out = []
|
||||||
|
for p in pts or []:
|
||||||
|
pl = p.payload or {}
|
||||||
|
nid = pl.get("note_id") or pl.get("id")
|
||||||
|
if nid:
|
||||||
|
out.append({"note_id": nid, "title": pl.get("title"), "type": pl.get("type")})
|
||||||
|
return out
|
||||||
|
|
||||||
|
def count_points(client, col: str, filt: rest.Filter) -> int:
|
||||||
|
res = client.count(collection_name=col, count_filter=filt, exact=True)
|
||||||
|
return int(getattr(res, "count", 0))
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--max-notes", type=int)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
client = get_client(cfg)
|
||||||
|
notes_col, chunks_col, edges_col = collections(cfg.prefix)
|
||||||
|
notes = list_note_ids(client, notes_col, args.max_notes)
|
||||||
|
|
||||||
|
summary = {"notes": 0, "chunks": 0, "edges": 0}
|
||||||
|
for n in notes:
|
||||||
|
nid = n["note_id"]
|
||||||
|
summary["notes"] += 1
|
||||||
|
|
||||||
|
filt_note = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=nid))])
|
||||||
|
chunk_cnt = count_points(client, chunks_col, filt_note)
|
||||||
|
summary["chunks"] += chunk_cnt
|
||||||
|
|
||||||
|
# counts per kind (edges)
|
||||||
|
by_kind: Dict[str, int] = {}
|
||||||
|
for k in KINDS:
|
||||||
|
f = rest.Filter(must=[
|
||||||
|
rest.FieldCondition(key="note_id", match=rest.MatchValue(value=nid)),
|
||||||
|
rest.FieldCondition(key="kind", match=rest.MatchValue(value=k)),
|
||||||
|
])
|
||||||
|
c = count_points(client, edges_col, f)
|
||||||
|
if c:
|
||||||
|
by_kind[k] = c
|
||||||
|
|
||||||
|
summary["edges"] += sum(by_kind.values())
|
||||||
|
line = {"note_id": nid, "title": n.get("title"), "type": n.get("type"), "chunks": chunk_cnt, "edges_by_kind": by_kind}
|
||||||
|
print(json.dumps(line, ensure_ascii=False))
|
||||||
|
|
||||||
|
print(json.dumps({"prefix": cfg.prefix, "summary": summary}, ensure_ascii=False))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from app.main import create_app
|
||||||
|
import app.core.graph_adapter as ga
|
||||||
|
|
||||||
|
def _fake_get_edges_for_sources(client, prefix, source_ids, edge_types=None, limit=2048):
|
||||||
|
note_id = source_ids[0]
|
||||||
|
return [
|
||||||
|
{"source_id":note_id,"target_id":f"{note_id}#c1","kind":"references","weight":0.2},
|
||||||
|
{"source_id":note_id,"target_id":f"{note_id}#c2","kind":"belongs_to","weight":0.1},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_graph_neighbors(monkeypatch):
|
||||||
|
monkeypatch.setattr(ga, "get_edges_for_sources", _fake_get_edges_for_sources, raising=False)
|
||||||
|
app = create_app()
|
||||||
|
with TestClient(app) as c:
|
||||||
|
r = c.get("/graph/TEST-NOTE?depth=1")
|
||||||
|
assert r.status_code == 200
|
||||||
|
g = r.json()
|
||||||
|
assert g["center_note_id"] == "TEST-NOTE"
|
||||||
|
assert g["stats"]["edge_count"] == 2
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from app.main import create_app
|
||||||
|
|
||||||
|
def test_healthz_ok():
|
||||||
|
app = create_app()
|
||||||
|
with TestClient(app) as c:
|
||||||
|
r = c.get("/healthz")
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["status"] == "ok"
|
||||||
|
assert "qdrant" in body and "prefix" in body
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# tests/test_importer_detect_changes.sh
|
||||||
|
# Prüft Änderungserkennung:
|
||||||
|
# - Body-Change -> wird erkannt bei hash-mode=body
|
||||||
|
# - Frontmatter-Change -> wird NICHT erkannt bei hash-mode=body
|
||||||
|
# - Frontmatter-Change -> wird erkannt bei hash-mode=frontmatter (bzw. MINDNET_HASH_COMPARE=Frontmatter)
|
||||||
|
#
|
||||||
|
# Aufruf:
|
||||||
|
# ./tests/test_importer_detect_changes.sh [--vault ./test_vault] [--prefix mindnet] [--note-id concept-alpha]
|
||||||
|
|
||||||
|
VAULT="./test_vault"
|
||||||
|
PREFIX="${COLLECTION_PREFIX:-mindnet}"
|
||||||
|
NOTE_ID="concept-alpha"
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--vault) VAULT="$2"; shift 2;;
|
||||||
|
--prefix) PREFIX="$2"; shift 2;;
|
||||||
|
--note-id) NOTE_ID="$2"; shift 2;;
|
||||||
|
*) echo "Unbekannte Option: $1" >&2; exit 2;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
export COLLECTION_PREFIX="$PREFIX"
|
||||||
|
|
||||||
|
# Hilfsfunktion: lese Pfad zur Note aus Frontmatter id:
|
||||||
|
PYFIND=$(cat <<'PY'
|
||||||
|
import sys, os, glob, yaml
|
||||||
|
vault = sys.argv[1]
|
||||||
|
nid = sys.argv[2]
|
||||||
|
for p in glob.glob(os.path.join(vault, "**", "*.md"), recursive=True):
|
||||||
|
with open(p, "r", encoding="utf-8") as f:
|
||||||
|
s = f.read()
|
||||||
|
if s.startswith("---"):
|
||||||
|
try:
|
||||||
|
fm_txt, body = s.split("\n---\n", 1)
|
||||||
|
fm = yaml.safe_load(fm_txt.strip("- \n")) or {}
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
if fm.get("id") == nid:
|
||||||
|
print(p)
|
||||||
|
break
|
||||||
|
PY
|
||||||
|
)
|
||||||
|
NOTE_PATH=$(python3 -c "$PYFIND" "$VAULT" "$NOTE_ID")
|
||||||
|
if [[ -z "${NOTE_PATH:-}" ]]; then
|
||||||
|
echo "Note mit id=$NOTE_ID nicht gefunden im Vault=$VAULT" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
echo "Note: $NOTE_PATH"
|
||||||
|
|
||||||
|
# Backup
|
||||||
|
ORIG="$(mktemp)"
|
||||||
|
cp "$NOTE_PATH" "$ORIG"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
cp "$ORIG" "$NOTE_PATH" || true
|
||||||
|
rm -f "$ORIG"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
# Ausgangszustand importieren (Baseline)
|
||||||
|
python3 -m scripts.import_markdown --vault "$VAULT" --apply --prefix "$PREFIX" >/dev/null
|
||||||
|
|
||||||
|
echo "-- Test A: Body-Änderung soll erkannt werden (hash-mode=body)"
|
||||||
|
# Body um eine Zeile erweitern
|
||||||
|
python3 - "$NOTE_PATH" <<'PY'
|
||||||
|
import sys
|
||||||
|
p=sys.argv[1]
|
||||||
|
with open(p,"r+",encoding="utf-8") as f:
|
||||||
|
s=f.read()
|
||||||
|
if s.startswith("---"):
|
||||||
|
fm, body = s.split("\n---\n",1)
|
||||||
|
body = body + "\n\nBodyTest: Änderung A"
|
||||||
|
f.seek(0); f.truncate(); f.write(fm+"\n---\n"+body)
|
||||||
|
PY
|
||||||
|
OUT=$(python3 -m scripts.import_markdown --vault "$VAULT" --prefix "$PREFIX" | grep "\"note_id\": \"$NOTE_ID\"" || true)
|
||||||
|
echo "$OUT"
|
||||||
|
echo "$OUT" | grep -q '"changed": true' || { echo "ERWARTET: changed=true (Body) — FEHLER"; exit 4; }
|
||||||
|
|
||||||
|
echo "-- Test B: Frontmatter-Änderung soll NICHT erkannt werden bei mode=body"
|
||||||
|
# Frontmatter-Feld hinzufügen
|
||||||
|
python3 - "$NOTE_PATH" <<'PY'
|
||||||
|
import sys, yaml, io
|
||||||
|
p=sys.argv[1]
|
||||||
|
with open(p,"r+",encoding="utf-8") as f:
|
||||||
|
s=f.read()
|
||||||
|
if s.startswith("---"):
|
||||||
|
fm_txt, body = s.split("\n---\n",1)
|
||||||
|
fm = yaml.safe_load(fm_txt.strip("- \n")) or {}
|
||||||
|
fm["test_flag"] = True
|
||||||
|
buf = io.StringIO()
|
||||||
|
yaml.safe_dump(fm, buf, allow_unicode=True, sort_keys=False)
|
||||||
|
f.seek(0); f.truncate(); f.write("---\n"+buf.getvalue()+"---\n"+body)
|
||||||
|
PY
|
||||||
|
OUT=$(python3 -m scripts.import_markdown --vault "$VAULT" --prefix "$PREFIX" | grep "\"note_id\": \"$NOTE_ID\"" || true)
|
||||||
|
echo "$OUT"
|
||||||
|
echo "$OUT" | grep -q '"changed": false' || { echo "ERWARTET: changed=false (FM bei mode=body) — FEHLER"; exit 5; }
|
||||||
|
|
||||||
|
echo "-- Test C: Frontmatter-Änderung soll erkannt werden bei mode=frontmatter"
|
||||||
|
OUT=$(MINDNET_HASH_COMPARE=Frontmatter python3 -m scripts.import_markdown --vault "$VAULT" --prefix "$PREFIX" | grep "\"note_id\": \"$NOTE_ID\"" || true)
|
||||||
|
echo "$OUT"
|
||||||
|
echo "$OUT" | grep -q '"changed": true' || { echo "ERWARTET: changed=true (FM bei mode=frontmatter) — FEHLER"; exit 6; }
|
||||||
|
|
||||||
|
echo "✔ Änderungserkennung: OK"
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from app.main import create_app
|
||||||
|
import app.services.embeddings_client as ec
|
||||||
|
import app.core.qdrant_points as qp
|
||||||
|
import app.core.graph_adapter as ga
|
||||||
|
|
||||||
|
def _fake_embed_text(text: str):
|
||||||
|
# Liefert stabilen 384-d Vektor ohne echtes Modell
|
||||||
|
return [0.01] * 384
|
||||||
|
|
||||||
|
def _fake_search_chunks_by_vector(client, prefix, vector, top=10, filters=None):
|
||||||
|
# einfache Hitliste
|
||||||
|
return [
|
||||||
|
("chunk:1", 0.9, {"note_id":"note:1","path":"a.md","section_title":"S1"}),
|
||||||
|
("chunk:2", 0.7, {"note_id":"note:2","path":"b.md","section_title":"S2"}),
|
||||||
|
("chunk:3", 0.5, {"note_id":"note:3","path":"c.md","section_title":"S3"}),
|
||||||
|
]
|
||||||
|
|
||||||
|
def _fake_get_edges_for_sources(client, prefix, source_ids, edge_types=None, limit=2048):
|
||||||
|
out=[]
|
||||||
|
for sid in source_ids:
|
||||||
|
out.append({"source_id":sid,"target_id":f"{sid}-x","kind":"references","weight":0.2})
|
||||||
|
return out
|
||||||
|
|
||||||
|
def test_query_with_text(monkeypatch):
|
||||||
|
# Patch Embedding + Qdrant-Aufrufe
|
||||||
|
monkeypatch.setattr(ec, "embed_text", _fake_embed_text)
|
||||||
|
monkeypatch.setattr(qp, "search_chunks_by_vector", _fake_search_chunks_by_vector)
|
||||||
|
monkeypatch.setattr(ga, "get_edges_for_sources", _fake_get_edges_for_sources, raising=False)
|
||||||
|
|
||||||
|
app = create_app()
|
||||||
|
with TestClient(app) as c:
|
||||||
|
payload = {"mode":"hybrid","query":"karate trainingsplan", "top_k":2,
|
||||||
|
"expand":{"depth":1,"edge_types":["references"]}}
|
||||||
|
r = c.post("/query", json=payload)
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
assert body["used_mode"] == "hybrid"
|
||||||
|
assert len(body["results"]) == 2
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from app.main import create_app
|
||||||
|
import app.core.qdrant_points as qp
|
||||||
|
import app.core.graph_adapter as ga
|
||||||
|
|
||||||
|
def _fake_search_chunks_by_vector(client, prefix, vector, top=10, filters=None):
|
||||||
|
return [
|
||||||
|
("chunk:1", 0.8, {"note_id":"note:1","path":"a.md","section_title":"S1"}),
|
||||||
|
("chunk:2", 0.6, {"note_id":"note:2","path":"b.md","section_title":"S2"}),
|
||||||
|
("chunk:3", 0.4, {"note_id":"note:3","path":"c.md","section_title":"S3"}),
|
||||||
|
]
|
||||||
|
|
||||||
|
def _fake_get_edges_for_sources(client, prefix, source_ids, edge_types=None, limit=2048):
|
||||||
|
# Erzeuge ein paar Edges von den Seeds weg
|
||||||
|
out=[]
|
||||||
|
for sid in source_ids:
|
||||||
|
out.append({"source_id":sid,"target_id":f"{sid}->n1","kind":"references","weight":0.2})
|
||||||
|
out.append({"source_id":sid,"target_id":f"{sid}->n2","kind":"belongs_to","weight":0.1})
|
||||||
|
return out
|
||||||
|
|
||||||
|
def test_query_hybrid_with_monkeypatch(monkeypatch):
|
||||||
|
monkeypatch.setattr(qp, "search_chunks_by_vector", _fake_search_chunks_by_vector)
|
||||||
|
monkeypatch.setattr(qp, "get_edges_for_sources", _fake_get_edges_for_sources) # falls indirekt genutzt
|
||||||
|
# graph_adapter.expand ruft get_edges_for_sources via qp; wir patchen direkt dort:
|
||||||
|
monkeypatch.setattr(ga, "get_edges_for_sources", _fake_get_edges_for_sources, raising=False)
|
||||||
|
|
||||||
|
app = create_app()
|
||||||
|
with TestClient(app) as c:
|
||||||
|
payload = {"mode":"hybrid","query_vector":[0.0]*384,"top_k":2,
|
||||||
|
"expand":{"depth":1,"edge_types":["references","belongs_to"]}}
|
||||||
|
r = c.post("/query", json=payload)
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
data = r.json()
|
||||||
|
assert data["used_mode"] == "hybrid"
|
||||||
|
assert len(data["results"]) == 2
|
||||||
|
# Scores absteigend
|
||||||
|
tots = [h["total_score"] for h in data["results"]]
|
||||||
|
assert tots[0] >= tots[1]
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
from app.core import retriever as r
|
||||||
|
from app.models.dto import QueryRequest
|
||||||
|
import app.core.qdrant as qdr
|
||||||
|
import app.core.qdrant_points as qp
|
||||||
|
import app.services.embeddings_client as ec
|
||||||
|
|
||||||
|
|
||||||
|
class _DummyClient:
|
||||||
|
"""Minimaler Platzhalter für QdrantClient in Unit-Tests."""
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_get_client(cfg):
|
||||||
|
return _DummyClient()
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_embed_text(text: str):
|
||||||
|
# Liefert stabilen Vektor; Größe ist für diesen Test egal,
|
||||||
|
# da search_chunks_by_vector ebenfalls gefaked wird.
|
||||||
|
return [0.0] * 16
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_search_chunks_by_vector(client, prefix, vector, top=10, filters=None):
|
||||||
|
# einfache Hitliste in absteigender Score-Reihenfolge
|
||||||
|
return [
|
||||||
|
("chunk:1", 0.9, {"note_id": "note:1", "path": "a.md", "section_title": "S1"}),
|
||||||
|
("chunk:2", 0.7, {"note_id": "note:2", "path": "b.md", "section_title": "S2"}),
|
||||||
|
("chunk:3", 0.4, {"note_id": "note:3", "path": "c.md", "section_title": "S3"}),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_semantic_retrieve_basic(monkeypatch):
|
||||||
|
# Qdrant- und Embedding-Aufrufe faken
|
||||||
|
monkeypatch.setattr(qdr, "get_client", _fake_get_client)
|
||||||
|
monkeypatch.setattr(ec, "embed_text", _fake_embed_text)
|
||||||
|
monkeypatch.setattr(qp, "search_chunks_by_vector", _fake_search_chunks_by_vector)
|
||||||
|
|
||||||
|
req = QueryRequest(mode="semantic", query="karate trainingsplan", top_k=2)
|
||||||
|
resp = r.semantic_retrieve(req)
|
||||||
|
|
||||||
|
assert resp.used_mode == "semantic"
|
||||||
|
assert len(resp.results) == 2
|
||||||
|
|
||||||
|
# Scores absteigend sortiert
|
||||||
|
scores = [h.total_score for h in resp.results]
|
||||||
|
assert scores[0] >= scores[1]
|
||||||
|
# Korrekte Zuordnung von note_id und Quelle
|
||||||
|
assert resp.results[0].note_id == "note:1"
|
||||||
|
assert resp.results[0].source["path"] == "a.md"
|
||||||
|
|
||||||
|
|
||||||
|
def test_hybrid_retrieve_basic(monkeypatch):
|
||||||
|
# Qdrant- und Embedding-Aufrufe faken
|
||||||
|
monkeypatch.setattr(qdr, "get_client", _fake_get_client)
|
||||||
|
monkeypatch.setattr(qp, "search_chunks_by_vector", _fake_search_chunks_by_vector)
|
||||||
|
|
||||||
|
# Im Hybrid-Modus arbeiten wir im Step-1-Stand nur mit query_vector
|
||||||
|
req = QueryRequest(mode="hybrid", query_vector=[0.0] * 16, top_k=2)
|
||||||
|
resp = r.hybrid_retrieve(req)
|
||||||
|
|
||||||
|
assert resp.used_mode == "hybrid"
|
||||||
|
assert len(resp.results) == 2
|
||||||
|
scores = [h.total_score for h in resp.results]
|
||||||
|
assert scores[0] >= scores[1]
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
from app.core import retriever as r
|
||||||
|
from app.models.dto import QueryRequest
|
||||||
|
import app.core.qdrant as qdr
|
||||||
|
import app.core.qdrant_points as qp
|
||||||
|
import app.core.graph_adapter as ga
|
||||||
|
import app.services.embeddings_client as ec
|
||||||
|
|
||||||
|
|
||||||
|
class _DummyClient:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_get_client(cfg):
|
||||||
|
return _DummyClient()
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_embed_text(text: str):
|
||||||
|
# Dimension ist egal, da wir search_chunks_by_vector faken
|
||||||
|
return [0.0] * 384
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_search_chunks_by_vector(client, prefix, vector, top=10, filters=None):
|
||||||
|
# Drei Treffer mit identischem semantic_score und retriever_weight,
|
||||||
|
# aber unterschiedlichen Graph-Boni
|
||||||
|
return [
|
||||||
|
("chunk:1", 0.8, {"note_id": "note:1", "chunk_id": "c1", "path": "a.md", "section_title": "S1"}),
|
||||||
|
("chunk:2", 0.8, {"note_id": "note:2", "chunk_id": "c2", "path": "b.md", "section_title": "S2"}),
|
||||||
|
("chunk:3", 0.8, {"note_id": "note:3", "chunk_id": "c3", "path": "c.md", "section_title": "S3"}),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class _DummySubgraph:
|
||||||
|
def __init__(self, edge_scores, cent_scores):
|
||||||
|
self._edge = edge_scores
|
||||||
|
self._cent = cent_scores
|
||||||
|
|
||||||
|
def edge_bonus(self, node_id: str) -> float:
|
||||||
|
return self._edge.get(node_id, 0.0)
|
||||||
|
|
||||||
|
def centrality_bonus(self, node_id: str) -> float:
|
||||||
|
return self._cent.get(node_id, 0.0)
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_expand(client, prefix, seeds, depth=1, edge_types=None):
|
||||||
|
# Wir gewichten c2 > c1 > c3
|
||||||
|
edge_scores = {"c1": 0.1, "c2": 0.5, "c3": 0.0}
|
||||||
|
cent_scores = {"c1": 0.05, "c2": 0.2, "c3": 0.0}
|
||||||
|
return _DummySubgraph(edge_scores, cent_scores)
|
||||||
|
|
||||||
|
|
||||||
|
def test_hybrid_retrieve_uses_edge_scores(monkeypatch):
|
||||||
|
# Qdrant-Client, Embeddings, Chunk-Suche und Graph-Expansion faken
|
||||||
|
monkeypatch.setattr(qdr, "get_client", _fake_get_client)
|
||||||
|
monkeypatch.setattr(ec, "embed_text", _fake_embed_text)
|
||||||
|
monkeypatch.setattr(qp, "search_chunks_by_vector", _fake_search_chunks_by_vector)
|
||||||
|
monkeypatch.setattr(ga, "expand", _fake_expand)
|
||||||
|
|
||||||
|
# expand.depth > 0 triggern, edge_types sind hier egal
|
||||||
|
req = QueryRequest(mode="hybrid", query="karate", top_k=3, expand={"depth": 1, "edge_types": ["references"]})
|
||||||
|
resp = r.hybrid_retrieve(req)
|
||||||
|
|
||||||
|
assert len(resp.results) == 3
|
||||||
|
|
||||||
|
# Da alle semantic_scores und retriever_weight gleich sind,
|
||||||
|
# bestimmt die Summe aus edge_bonus + centrality_bonus das Ranking.
|
||||||
|
first = resp.results[0]
|
||||||
|
second = resp.results[1]
|
||||||
|
third = resp.results[2]
|
||||||
|
|
||||||
|
assert first.note_id == "note:2" # c2 hat höchste Graph-Boni
|
||||||
|
assert second.note_id == "note:1"
|
||||||
|
assert third.note_id == "note:3"
|
||||||
|
|
||||||
|
assert first.total_score > second.total_score > third.total_score
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
#
|
||||||
|
# test_retriever_smoke.py – End-to-End Smoke-Test für den mindnet Retriever
|
||||||
|
#
|
||||||
|
# Dieser Smoke-Test ist für manuelle End-to-End-Tests gedacht.
|
||||||
|
# Er wird NICHT automatisch von pytest ausgeführt.
|
||||||
|
#
|
||||||
|
# Empfohlener Speicherort: tests/test_retriever_smoke.py
|
||||||
|
#
|
||||||
|
# Aufrufbeispiele:
|
||||||
|
#
|
||||||
|
# python tests/test_retriever_smoke.py --query "embeddings" --mode semantic
|
||||||
|
#
|
||||||
|
# python tests/test_retriever_smoke.py \
|
||||||
|
# --query "embeddings" \
|
||||||
|
# --mode hybrid \
|
||||||
|
# --expand-depth 1 \
|
||||||
|
# --expand-edge-types references depends_on
|
||||||
|
#
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from typing import Any, Dict
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
def _build_payload(args: argparse.Namespace) -> Dict[str, Any]:
|
||||||
|
payload: Dict[str, Any] = {
|
||||||
|
"mode": args.mode,
|
||||||
|
"query": args.query,
|
||||||
|
"top_k": args.top_k,
|
||||||
|
}
|
||||||
|
if args.expand_depth is not None and args.expand_depth > 0:
|
||||||
|
expand: Dict[str, Any] = {"depth": args.expand_depth}
|
||||||
|
if args.expand_edge_types:
|
||||||
|
expand["edge_types"] = args.expand_edge_types
|
||||||
|
payload["expand"] = expand
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Smoke-Test für mindnet-Retriever (/query)")
|
||||||
|
parser.add_argument("--url", default="http://127.0.0.1:8000/query", help="URL des Query-Endpunkts")
|
||||||
|
parser.add_argument("--query", required=True, help="Text-Query")
|
||||||
|
parser.add_argument("--mode", choices=["semantic", "hybrid"], default="hybrid")
|
||||||
|
parser.add_argument("--top-k", type=int, default=5)
|
||||||
|
parser.add_argument("--expand-depth", type=int, default=0)
|
||||||
|
parser.add_argument(
|
||||||
|
"--expand-edge-types",
|
||||||
|
nargs="*",
|
||||||
|
default=None,
|
||||||
|
help="Edge-Typen (z. B. references depends_on)",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
payload = _build_payload(args)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = requests.post(args.url, json=payload, timeout=10)
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"[ERROR] HTTP-Request fehlgeschlagen: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if resp.status_code != 200:
|
||||||
|
print(f"[ERROR] Statuscode: {resp.status_code}", file=sys.stderr)
|
||||||
|
print(resp.text, file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = resp.json()
|
||||||
|
except Exception:
|
||||||
|
print("[ERROR] Ungültiges JSON:", resp.text, file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print("=== mindnet Retriever Smoke-Test ===")
|
||||||
|
print(f"URL: {args.url}")
|
||||||
|
print(f"Mode: {args.mode}")
|
||||||
|
print(f"Query: {args.query!r}")
|
||||||
|
print(f"Top-K: {args.top_k}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
results = data.get("results") or []
|
||||||
|
print(f"used_mode: {data.get('used_mode')}")
|
||||||
|
print(f"latency: {data.get('latency_ms')} ms")
|
||||||
|
print(f"Treffer: {len(results)}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
for i, hit in enumerate(results, start=1):
|
||||||
|
nid = hit.get("note_id")
|
||||||
|
cid = hit.get("node_id")
|
||||||
|
tot = hit.get("total_score")
|
||||||
|
sem = hit.get("semantic_score")
|
||||||
|
edge = hit.get("edge_bonus")
|
||||||
|
cen = hit.get("centrality_bonus")
|
||||||
|
src = hit.get("source") or {}
|
||||||
|
path = src.get("path")
|
||||||
|
sec = src.get("section")
|
||||||
|
|
||||||
|
print(f"[{i}] note_id={nid} node_id={cid}")
|
||||||
|
try:
|
||||||
|
print(
|
||||||
|
f" total={float(tot):.4f} "
|
||||||
|
f"(semantic={float(sem):.4f} edge={float(edge):.4f} centrality={float(cen):.4f})"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
print(f" total={tot}, semantic={sem}, edge={edge}, centrality={cen}")
|
||||||
|
|
||||||
|
if path or sec:
|
||||||
|
print(f" source: {path or '-'} :: {sec or '-'}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
from app.core import retriever as r
|
||||||
|
from app.models.dto import QueryRequest
|
||||||
|
import app.core.qdrant as qdr
|
||||||
|
import app.core.qdrant_points as qp
|
||||||
|
import app.services.embeddings_client as ec
|
||||||
|
|
||||||
|
|
||||||
|
class _DummyClient:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_get_client(cfg):
|
||||||
|
return _DummyClient()
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_embed_text(text: str):
|
||||||
|
# Dimension egal, da wir search_chunks_by_vector faken
|
||||||
|
return [0.0] * 384
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_search_chunks_by_vector_weights(client, prefix, vector, top=10, filters=None):
|
||||||
|
# Zwei Treffer mit unterschiedlichem retriever_weight
|
||||||
|
return [
|
||||||
|
("chunk:low", 0.9, {"note_id": "note:low", "path": "low.md", "section_title": "S1", "retriever_weight": 0.2}),
|
||||||
|
("chunk:high", 0.8, {"note_id": "note:high", "path": "high.md", "section_title": "S2", "retriever_weight": 1.5}),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_retriever_weight_influences_ranking(monkeypatch):
|
||||||
|
# Konfiguration: Qdrant & Embedding faken
|
||||||
|
monkeypatch.setattr(qdr, "get_client", _fake_get_client)
|
||||||
|
monkeypatch.setattr(ec, "embed_text", _fake_embed_text)
|
||||||
|
monkeypatch.setattr(qp, "search_chunks_by_vector", _fake_search_chunks_by_vector_weights)
|
||||||
|
|
||||||
|
req = QueryRequest(mode="semantic", query="karate", top_k=2)
|
||||||
|
resp = r.semantic_retrieve(req)
|
||||||
|
|
||||||
|
assert len(resp.results) == 2
|
||||||
|
|
||||||
|
first = resp.results[0]
|
||||||
|
second = resp.results[1]
|
||||||
|
|
||||||
|
# Semantic Scores: 0.9 (low), 0.8 (high)
|
||||||
|
# Total Scores: 0.9 * 0.2 = 0.18 vs. 0.8 * 1.5 = 1.2 -> high sollte oben liegen
|
||||||
|
assert first.note_id == "note:high"
|
||||||
|
assert second.note_id == "note:low"
|
||||||
|
|
||||||
|
assert first.total_score > second.total_score
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from app.main import create_app
|
||||||
|
|
||||||
|
def test_tools_ollama_schema():
|
||||||
|
app = create_app()
|
||||||
|
with TestClient(app) as c:
|
||||||
|
r = c.get("/tools/ollama")
|
||||||
|
assert r.status_code == 200
|
||||||
|
data = r.json()
|
||||||
|
assert "tools" in data and isinstance(data["tools"], list)
|
||||||
|
names = [t["function"]["name"] for t in data["tools"] if t.get("function")]
|
||||||
|
assert "mindnet_query" in names
|
||||||
|
assert "mindnet_subgraph" in names
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import os, tempfile, textwrap
|
||||||
|
from app.core.type_registry import load_type_registry, resolve_note_type, get_type_config, effective_chunk_profile, profile_overlap
|
||||||
|
|
||||||
|
def test_defaults_when_missing():
|
||||||
|
# ensure default kicks in if file does not exist
|
||||||
|
reg = load_type_registry(os.path.join("config", "does_not_exist.yaml"))
|
||||||
|
assert reg.get("_using_defaults", False) is True
|
||||||
|
t = resolve_note_type(None, reg)
|
||||||
|
assert t == "concept"
|
||||||
|
cfg = get_type_config(t, reg)
|
||||||
|
assert cfg.get("chunk_profile") in ("medium", "long", "short")
|
||||||
|
|
||||||
|
def test_custom_file():
|
||||||
|
with tempfile.TemporaryDirectory() as td:
|
||||||
|
p = os.path.join(td, "types.yaml")
|
||||||
|
with open(p, "w", encoding="utf-8") as f:
|
||||||
|
f.write(textwrap.dedent('''
|
||||||
|
version: 1.0
|
||||||
|
types:
|
||||||
|
concept:
|
||||||
|
chunk_profile: medium
|
||||||
|
edge_defaults: ["references"]
|
||||||
|
retriever_weight: 1.0
|
||||||
|
task:
|
||||||
|
chunk_profile: short
|
||||||
|
edge_defaults: ["depends_on"]
|
||||||
|
retriever_weight: 0.8
|
||||||
|
'''))
|
||||||
|
reg = load_type_registry(p)
|
||||||
|
assert resolve_note_type("task", reg) == "task"
|
||||||
|
assert effective_chunk_profile("task", reg) == "short"
|
||||||
|
assert profile_overlap("short")[0] < profile_overlap("long")[0]
|
||||||
@@ -0,0 +1,193 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
Script: tests/verify_chunks_integrity.py
|
||||||
|
Version: 1.0.1
|
||||||
|
Datum: 2025-09-10
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
-----
|
||||||
|
Verifiziert die Text-Integrität der gespeicherten Chunks:
|
||||||
|
1) Rekonstruiert den Body aus den Chunks (Sortierung: seq → chunk_index → # in chunk_id).
|
||||||
|
2) Vergleicht mit dem in Qdrant gespeicherten Note-`fulltext` (falls vorhanden).
|
||||||
|
3) Optional: Vergleicht zusätzlich mit dem Body der zugehörigen Markdown-Datei im Vault.
|
||||||
|
|
||||||
|
Aufrufe
|
||||||
|
-------
|
||||||
|
# Nur gegen Qdrant (fulltext vs. Chunks)
|
||||||
|
python3 tests/verify_chunks_integrity.py --prefix mindnet
|
||||||
|
|
||||||
|
# Zusätzlich gegen den Vault abgleichen (Body der .md-Datei)
|
||||||
|
python3 tests/verify_chunks_integrity.py --prefix mindnet --vault ./test_vault
|
||||||
|
|
||||||
|
# Streng + CI-geeignet (Fehlercode bei Abweichungen):
|
||||||
|
python3 tests/verify_chunks_integrity.py --prefix mindnet --vault ./test_vault --strict --fail-on-mismatch
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from typing import Any, Dict, List, Optional, Tuple
|
||||||
|
|
||||||
|
# --- FIX: Projekt-Root in sys.path aufnehmen, damit 'app.*' importierbar ist ---
|
||||||
|
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||||
|
if PROJECT_ROOT not in sys.path:
|
||||||
|
sys.path.insert(0, PROJECT_ROOT)
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from qdrant_client.http import models as rest
|
||||||
|
|
||||||
|
from app.core.qdrant import QdrantConfig, get_client
|
||||||
|
|
||||||
|
# --------------------------- Helpers ---------------------------
|
||||||
|
|
||||||
|
def collections(prefix: str) -> Tuple[str, str, str]:
|
||||||
|
return f"{prefix}_notes", f"{prefix}_chunks", f"{prefix}_edges"
|
||||||
|
|
||||||
|
def scroll_all(client, collection: str, flt: Optional[rest.Filter] = None, limit: int = 256) -> List[Any]:
|
||||||
|
out: List[Any] = []
|
||||||
|
nextp = None
|
||||||
|
while True:
|
||||||
|
pts, nextp = client.scroll(
|
||||||
|
collection_name=collection,
|
||||||
|
with_payload=True,
|
||||||
|
with_vectors=False,
|
||||||
|
limit=limit,
|
||||||
|
offset=nextp,
|
||||||
|
scroll_filter=flt,
|
||||||
|
)
|
||||||
|
if not pts:
|
||||||
|
break
|
||||||
|
out.extend(pts)
|
||||||
|
if nextp is None:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
def sort_key_for_chunk_payload(pl: Dict[str, Any]) -> Tuple[int, int, int]:
|
||||||
|
s = pl.get("seq") or 0
|
||||||
|
ci = pl.get("chunk_index") or 0
|
||||||
|
n = 0
|
||||||
|
cid = pl.get("chunk_id") or ""
|
||||||
|
if isinstance(cid, str) and "#" in cid:
|
||||||
|
try:
|
||||||
|
n = int(cid.rsplit("#", 1)[-1])
|
||||||
|
except Exception:
|
||||||
|
n = 0
|
||||||
|
return (int(s), int(ci), int(n))
|
||||||
|
|
||||||
|
def reconstruct_from_chunks(chunks_points: List[Any]) -> Tuple[str, int, int]:
|
||||||
|
"""Gibt (text, total_chunks, chunks_with_text) zurück."""
|
||||||
|
chunks_sorted = sorted(chunks_points, key=lambda p: sort_key_for_chunk_payload(p.payload or {}))
|
||||||
|
texts: List[str] = []
|
||||||
|
have = 0
|
||||||
|
for p in chunks_sorted:
|
||||||
|
pl = p.payload or {}
|
||||||
|
t = pl.get("text") or pl.get("content") or pl.get("raw") or ""
|
||||||
|
if isinstance(t, str) and t:
|
||||||
|
have += 1
|
||||||
|
texts.append(t)
|
||||||
|
return ("\n".join(texts).strip(), len(chunks_sorted), have)
|
||||||
|
|
||||||
|
def normalize(s: str, strict: bool = False) -> str:
|
||||||
|
if s is None:
|
||||||
|
return ""
|
||||||
|
s = s.replace("\r\n", "\n").replace("\r", "\n")
|
||||||
|
if strict:
|
||||||
|
return s
|
||||||
|
lines = [ln.rstrip() for ln in s.strip().split("\n")]
|
||||||
|
return "\n".join(lines).strip()
|
||||||
|
|
||||||
|
def read_vault_body(vault_root: str, rel_path: str) -> Optional[str]:
|
||||||
|
if not rel_path:
|
||||||
|
return None
|
||||||
|
p = os.path.join(vault_root, rel_path.replace("\\", "/").lstrip("/"))
|
||||||
|
if not os.path.exists(p):
|
||||||
|
return None
|
||||||
|
with open(p, "r", encoding="utf-8") as f:
|
||||||
|
s = f.read()
|
||||||
|
if s.startswith("---"):
|
||||||
|
try:
|
||||||
|
fm_txt, body = s.split("\n---\n", 1)
|
||||||
|
except Exception:
|
||||||
|
return s
|
||||||
|
else:
|
||||||
|
body = s
|
||||||
|
return body
|
||||||
|
|
||||||
|
# --------------------------- Main ------------------------------
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--prefix", help="Collection-Prefix (überschreibt ENV COLLECTION_PREFIX)")
|
||||||
|
ap.add_argument("--vault", help="Optional: Vault-Wurzelordner für Abgleich gegen .md")
|
||||||
|
ap.add_argument("--strict", action="store_true", help="Strikter Vergleich (kein Trimmen/Normalisieren)")
|
||||||
|
ap.add_argument("--fail-on-mismatch", action="store_true", help="Exit 1, wenn ein Mismatch gefunden wurde")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
cfg = QdrantConfig.from_env()
|
||||||
|
if args.prefix:
|
||||||
|
cfg.prefix = args.prefix.strip()
|
||||||
|
client = get_client(cfg)
|
||||||
|
|
||||||
|
notes_col, chunks_col, _ = collections(cfg.prefix)
|
||||||
|
notes = scroll_all(client, notes_col)
|
||||||
|
|
||||||
|
mismatches = 0
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
for n in notes:
|
||||||
|
total += 1
|
||||||
|
pl = n.payload or {}
|
||||||
|
nid = pl.get("note_id")
|
||||||
|
rel_path = (pl.get("path") or "").replace("\\", "/").lstrip("/")
|
||||||
|
fulltext = (pl.get("fulltext") or "").strip()
|
||||||
|
|
||||||
|
flt = rest.Filter(must=[rest.FieldCondition(key="note_id", match=rest.MatchValue(value=nid))])
|
||||||
|
chunks = scroll_all(client, chunks_col, flt)
|
||||||
|
recon, cnt, have = reconstruct_from_chunks(chunks)
|
||||||
|
|
||||||
|
norm_recon = normalize(recon, strict=args.strict)
|
||||||
|
norm_full = normalize(fulltext, strict=args.strict)
|
||||||
|
|
||||||
|
match_full = (bool(norm_full) and (norm_recon == norm_full))
|
||||||
|
match_vault = None
|
||||||
|
|
||||||
|
issues: List[str] = []
|
||||||
|
if cnt == 0:
|
||||||
|
issues.append("no_chunks")
|
||||||
|
if have < cnt:
|
||||||
|
issues.append(f"missing_text_in_chunks:{cnt-have}/{cnt}")
|
||||||
|
if norm_full == "":
|
||||||
|
issues.append("note_fulltext_empty")
|
||||||
|
if not match_full:
|
||||||
|
issues.append("reconstructed_vs_fulltext_mismatch")
|
||||||
|
|
||||||
|
if args.vault:
|
||||||
|
body = read_vault_body(args.vault, rel_path) or ""
|
||||||
|
norm_body = normalize(body, strict=args.strict)
|
||||||
|
match_vault = (norm_body == norm_recon)
|
||||||
|
if not match_vault:
|
||||||
|
issues.append("reconstructed_vs_vault_body_mismatch")
|
||||||
|
|
||||||
|
obj = {
|
||||||
|
"note_id": nid,
|
||||||
|
"path": rel_path,
|
||||||
|
"chunks_count": cnt,
|
||||||
|
"chunks_with_text": have,
|
||||||
|
"match_fulltext": match_full,
|
||||||
|
"match_vault": match_vault,
|
||||||
|
"issues": issues,
|
||||||
|
}
|
||||||
|
print(json.dumps(obj, ensure_ascii=False))
|
||||||
|
if ("reconstructed_vs_fulltext_mismatch" in issues) or ("reconstructed_vs_vault_body_mismatch" in issues):
|
||||||
|
mismatches += 1
|
||||||
|
|
||||||
|
summary = {"summary": "OK" if mismatches == 0 else "DIFFS", "mismatch_count": mismatches, "notes": total}
|
||||||
|
print(json.dumps(summary, ensure_ascii=False))
|
||||||
|
if args.fail_on_mismatch and mismatches:
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
title: Freund – November-Fahrplan (30 Tage)
|
||||||
|
id: freund-kanal-30tage-nov2025
|
||||||
|
type: project
|
||||||
|
status: active
|
||||||
|
created: 2025-10-31
|
||||||
|
updated: 2025-10-31
|
||||||
|
area: personal_development
|
||||||
|
project: leitbild_refresh_2025
|
||||||
|
tags: [friend/new-circle, planning]
|
||||||
|
lang: de
|
||||||
|
source: coach/leitbild
|
||||||
|
depends_on:
|
||||||
|
- freund-kanal-gewaltschutz-nov2025
|
||||||
|
- freund-kanal-tanzen-nov2025
|
||||||
|
---
|
||||||
|
|
||||||
|
# Kalender & Cadence
|
||||||
|
- **KW45 (Nov W1):** Gewaltschutz-Termin & Tanz-Slot festlegen; Einladungen verschicken.
|
||||||
|
- **KW46 (W2):** Infoabend Gewaltschutz #1 · Practice Night Tanz #1.
|
||||||
|
- **KW47 (W3):** Practice Night #2 · 4–6 **1:1-Gespräche**.
|
||||||
|
- **KW48 (W4):** Follow-ups & Retro; Dezember-Slots bestätigen.
|
||||||
|
|
||||||
|
# Minimal-Tracker (CSV)
|
||||||
|
date,event,type,people,new_contacts,1on1_scheduled,notes
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
---
|
||||||
|
title: Freund – Kanal 1: Gewaltschutz **Mini-Kurs** (Nov 2025)
|
||||||
|
id: freund-kanal-gewaltschutz-minikurs-nov2025
|
||||||
|
type: project
|
||||||
|
status: active
|
||||||
|
created: 2025-10-31
|
||||||
|
updated: 2025-10-31
|
||||||
|
area: personal_development
|
||||||
|
project: leitbild_refresh_2025
|
||||||
|
tags: [friend/new-circle, gewaltschutz, community, workshop, mini-course]
|
||||||
|
lang: de
|
||||||
|
source: coach/leitbild
|
||||||
|
depends_on:
|
||||||
|
- leitbild-ziele-portfolio
|
||||||
|
- leitbild-rollen
|
||||||
|
- freund-kanal-30tage-nov2025
|
||||||
|
references:
|
||||||
|
- "/mnt/data/Lars- Mein Leitbild.pdf"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Ziel & Zielgruppe
|
||||||
|
- **Ziel:** Über einen **nüchternen**, wertschätzenden **Mini-Kurs** (3×90′) neue Kontakte **auf Augenhöhe** gewinnen – für gute Gespräche, gemeinsame Werte und Anschluss an deinen neuen Freundeskreis (ohne Alkoholmilieu).
|
||||||
|
- **Zielgruppe (primär):** Erwachsene (Eltern, Lehrkräfte, Interessierte, **18+**).
|
||||||
|
*Optional (separat ankündigen):* Eltern+Teen (16–17) – sensibel, ohne Rollenspiele mit Übergriffigkeitsinhalten.
|
||||||
|
|
||||||
|
# Format (3×90′)
|
||||||
|
- **S1 – Bewusst & klar (Awareness & Haltung)**
|
||||||
|
*Outcome:* Teilnehmende können Situationen besser **lesen**, **Grenzen** spüren/zeigen und **klar ansprechen**.
|
||||||
|
**Ablauf:**
|
||||||
|
1) Warm-in & Werte (5′) – „Prävention ohne Angst“
|
||||||
|
2) Wahrnehmung & Distanz (20′) – Raum, Blick, Stimme
|
||||||
|
3) Grenzen setzen (20′) – „Stopp“, Körpersprache, Ich-Botschaft
|
||||||
|
4) Mini-Drills (30′) – Alltagsmikros: ÖPNV, Flur, Parkplatz (ohne Körperkontakt)
|
||||||
|
5) Reflexion & Transfer (15′) – persönlicher Boundary-Satz, Heimübung
|
||||||
|
|
||||||
|
- **S2 – Deeskalation & Ausstiege (Verbal & taktisch)**
|
||||||
|
*Outcome:* Teilnehmende **reduzieren Eskalation**, nutzen **sichere Ausstiege** und holen Hilfe.
|
||||||
|
**Ablauf:**
|
||||||
|
1) Kurzreview S1 (10′)
|
||||||
|
2) Deeskalationsleitfaden (25′) – Ton, Tempo, Wörter
|
||||||
|
3) Ausstiegsmuster (30′) – Positionswechsel, Tür-/Tresen-Taktik, „Buddy-Call“
|
||||||
|
4) Bystander-Optionen (15′) – sicher intervenieren
|
||||||
|
5) Plan A/B/C (10′) – persönlicher Ausstiegsplan
|
||||||
|
|
||||||
|
- **S3 – Alltag festigen (Routine & Netz)**
|
||||||
|
*Outcome:* Teilnehmende **verankern Routinen**, kennen **Fallbacks** und haben **nächste Schritte** (auch sozial).
|
||||||
|
**Ablauf:**
|
||||||
|
1) Warm-in & Qs (10′)
|
||||||
|
2) Alltagsszenarien (30′) – Feier/ÖPNV/Arbeit *ohne* Alkoholfokus
|
||||||
|
3) Ritual & Check (25′) – „Wenn-dann“-Pläne, Buddy-System, Notfallkontakte
|
||||||
|
4) Mikro-Netz (15′) – Einladungen zu **Walk & Talk**, **Buch/Essay-Abend**, **Tanzen**
|
||||||
|
5) Abschluss & Feedback (10′)
|
||||||
|
|
||||||
|
# Einladung (Kurztext, Copy-Paste)
|
||||||
|
**Gewaltschutz – Mini-Kurs (3×90′, alkoholfrei)**
|
||||||
|
Prävention ohne Angst: klare Grenzen, Deeskalation, sichere Ausstiege – alltagstauglich.
|
||||||
|
**Wann:** [Mi, 19:00–20:30 an: 12.11., 19.11., 26.11.] **/ Ort:** [Dojo/Vereinsraum, Adresse]
|
||||||
|
**Plätze:** max. 14 · **Kosten:** kostenlos
|
||||||
|
**Getränke:** Tee/Wasser (alkoholfrei)
|
||||||
|
**Anmeldung:** [E-Mail/Link] – kurze Nachricht reicht.
|
||||||
|
Ich freue mich auf gute Gespräche & ein starkes Netz. – Lars
|
||||||
|
|
||||||
|
# Orga & Material
|
||||||
|
- Raum: Stühle im Halbkreis, ggf. Matten (nicht nötig), gute Akustik; Schild „alkoholfrei“.
|
||||||
|
- Ausstattung: Flipchart/Marker, Timer, Wasser/Tee, **Feedback-Mini-Karte (3 Fragen)**, Teilnehmerliste.
|
||||||
|
- Sicherheit: **keine** aggressiven Rollenspiele; **kein** körperlicher Zwang; Inhalte trauma-sensibel; Notfallnummern sichtbar.
|
||||||
|
|
||||||
|
# Messung (verknüpft mit Nordstern „Freund – Neuaufbau“)
|
||||||
|
- **Leading:** ≥ **12 Anmeldungen**, **≥10** Teilnehmende gesamt, **≥6** 1:1-Folgetermine in 14 Tagen, **≥1** Mikro-Event-Einladung angenommen.
|
||||||
|
- **Lagging:** **≥2** neue starke Bindungen bis Jan 2026.
|
||||||
|
|
||||||
|
# Follow-up (≤48 h nach jedem Termin)
|
||||||
|
- Dankesmail + 1-Pager (3 Kernhebel)
|
||||||
|
- Einladungen: **Walk & Talk**, **Buch/Essay-Abend**, **Tanzen – Practice Night**
|
||||||
|
- 1:1-Einladungen an 2–4 passende Personen (Coffee/Walk)
|
||||||
|
|
||||||
|
# Checkliste (je Termin)
|
||||||
|
- [ ] Raum vorbereitet (Schild „alkoholfrei“)
|
||||||
|
- [ ] Teilnehmerliste / Einverständnis Foto **nein** (Standard: keine Fotos)
|
||||||
|
- [ ] Inhalte: Folien/Flipchart-Skizzen vorbereitet
|
||||||
|
- [ ] Wasser/Tee bereitgestellt
|
||||||
|
- [ ] Feedback-Karten & Stifte
|
||||||
|
- [ ] Timer, Notfallkontakte sichtbar
|
||||||
|
|
||||||
|
# ICS-Skelett (3 Events – Platzhalter anpassen)
|
||||||
|
BEGIN:VCALENDAR
|
||||||
|
VERSION:2.0
|
||||||
|
PRODID:-//Lars//Gewaltschutz-Minikurs//DE
|
||||||
|
BEGIN:VEVENT
|
||||||
|
UID:mini1-20251112T190000Z@lars
|
||||||
|
DTSTART:20251112T190000
|
||||||
|
DTEND:20251112T203000
|
||||||
|
SUMMARY:Gewaltschutz Mini-Kurs – Teil 1 (Prävention ohne Angst)
|
||||||
|
LOCATION:[Ort/Adresse]
|
||||||
|
DESCRIPTION:Awareness, Grenzen, Mini-Drills. Alkoholfrei.
|
||||||
|
END:VEVENT
|
||||||
|
BEGIN:VEVENT
|
||||||
|
UID:mini2-20251119T190000Z@lars
|
||||||
|
DTSTART:20251119T190000
|
||||||
|
DTEND:20251119T203000
|
||||||
|
SUMMARY:Gewaltschutz Mini-Kurs – Teil 2 (Deeskalation & Ausstiege)
|
||||||
|
LOCATION:[Ort/Adresse]
|
||||||
|
DESCRIPTION:Verbal deeskalieren, Ausstiege, Bystander-Optionen.
|
||||||
|
END:VEVENT
|
||||||
|
BEGIN:VEVENT
|
||||||
|
UID:mini3-20251126T190000Z@lars
|
||||||
|
DTSTART:20251126T190000
|
||||||
|
DTEND:20251126T203000
|
||||||
|
SUMMARY:Gewaltschutz Mini-Kurs – Teil 3 (Alltag festigen)
|
||||||
|
LOCATION:[Ort/Adresse]
|
||||||
|
DESCRIPTION:Wenn-dann-Pläne, Buddy-System, nächste Schritte.
|
||||||
|
END:VEVENT
|
||||||
|
END:VCALENDAR
|
||||||
|
|
||||||
|
# Risiken & Guardrails
|
||||||
|
- **Kein** „Heldentum“/Konfrontationsglorifizierung; Grenzen/Deeskalation vor „Technik“.
|
||||||
|
- **Kein** Körperkontakt ohne explizite Zustimmung; **Abbruch-Signal** definieren.
|
||||||
|
- **Keine** Minderjährigen ohne Begleitung (bei Eltern+Teen: separate Einladung, andere Beispiele).
|
||||||
|
- **Familienzeit schützen:** pünktlicher Start/Schluss; Nachklang maximal 20′.
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
---
|
||||||
|
title: Freund – Kanal 1: Gewaltschutz-Workshop (Nov 2025)
|
||||||
|
id: freund-kanal-gewaltschutz-nov2025
|
||||||
|
type: project
|
||||||
|
status: active
|
||||||
|
created: 2025-10-31
|
||||||
|
updated: 2025-10-31
|
||||||
|
area: personal_development
|
||||||
|
project: leitbild_refresh_2025
|
||||||
|
tags: [friend/new-circle, gewaltschutz, community, workshop]
|
||||||
|
lang: de
|
||||||
|
source: coach/leitbild
|
||||||
|
depends_on:
|
||||||
|
- leitbild-ziele-portfolio
|
||||||
|
- leitbild-rollen
|
||||||
|
---
|
||||||
|
|
||||||
|
# Ziel
|
||||||
|
Über einen **niedrigschwelligen Gewaltschutz-Impuls** neue Kontakte auf Augenhöhe knüpfen (eltern-/gemeinwohlorientiert, nüchternes Umfeld), die zu **guten Gesprächen** und **weiteren Treffen** führen.
|
||||||
|
|
||||||
|
# Format (wähle 1)
|
||||||
|
- **A) Infoabend (75–90′, kostenlos)** – kurzer, starker Impuls, Q&A, Tee/Wasser.
|
||||||
|
- **B) Mini-Kurs (3×90′, 3 Wochen)** – kleine, konstante Gruppe (max. 14), mehr Bindung.
|
||||||
|
|
||||||
|
# Agenda (Beispiel Infoabend 90′)
|
||||||
|
1) Ankommen & Werte (5′) – „Warum Prävention ohne Angst“
|
||||||
|
2) Aufmerksamkeit & Grenzen (15′) – Wahrnehmung, Stimme, Haltung
|
||||||
|
3) Einfache Hebel (25′) – Distanz, klare Ansagen, Ausstiege
|
||||||
|
4) Alltagsszenarien (20′) – Schule/ÖPNV/Feier (alkoholfrei)
|
||||||
|
5) Q&A + Einladung zu weiterem Treffen (20′)
|
||||||
|
|
||||||
|
# Einladungstext (Kurzfassung, Copy-Paste)
|
||||||
|
Liebe Nachbar:innen und Eltern,
|
||||||
|
ich biete am **[Datum, Uhrzeit]** einen **kostenlosen Gewaltschutz-Infoabend (90′)** im **[Ort]** an. Es geht um **klare, alltagstaugliche Strategien** für Kinder/Jugendliche & Eltern – **ohne Angstmache, ohne Heldentum**.
|
||||||
|
👉 Inhalte: Wahrnehmung schärfen, Grenzen setzen, einfache Ausstiege.
|
||||||
|
Getränke: **Tee/Wasser**, **kein Alkohol**.
|
||||||
|
**Anmeldung:** [Kontakt/Link] – max. **20 Plätze**. Ich freue mich auf gute Gespräche!
|
||||||
|
– Lars
|
||||||
|
|
||||||
|
# Orga & Material
|
||||||
|
- Raum (Matten optional), Stühle im Halbkreis, Flipchart/Marker, Timer, Haftungshinweis.
|
||||||
|
- Teilnehmerliste, Namensschilder, Wasser/Tee, Feedback-Mini-Karte (3 Fragen).
|
||||||
|
|
||||||
|
# Messung (für „Freund“-Nordstern)
|
||||||
|
- **Leading:** ≥ **12 Anmeldungen**, **10+ Teilnehmende**, **≥4** Folge-1:1s in 14 Tagen.
|
||||||
|
- **Lagging:** **≥2** neue „starke Bindungen“ bis Jan 2026.
|
||||||
|
|
||||||
|
# Nachbereitung (innerh. 48 h)
|
||||||
|
- Dankesmail + 1-Pager (3 Kernhebel) + Einladung zu **Walk & Talk / Buch-Abend**.
|
||||||
|
- 1:1-Einladungen an 4–6 passende Kontakte (Coffee/Walk).
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
---
|
||||||
|
title: Freund – Kanal 2: Tanzen (Nov 2025)
|
||||||
|
id: freund-kanal-tanzen-nov2025
|
||||||
|
type: project
|
||||||
|
status: active
|
||||||
|
created: 2025-10-31
|
||||||
|
updated: 2025-10-31
|
||||||
|
area: personal_development
|
||||||
|
project: leitbild_refresh_2025
|
||||||
|
tags: [friend/new-circle, tanzen, sport, community]
|
||||||
|
lang: de
|
||||||
|
source: coach/leitbild
|
||||||
|
depends_on:
|
||||||
|
- leitbild-ziele-portfolio
|
||||||
|
- leitbild-rollen
|
||||||
|
---
|
||||||
|
|
||||||
|
# Ziel
|
||||||
|
Über **körperliche Leichtigkeit + gute Gespräche** neue Kontakte auf Augenhöhe finden; Alkoholfreiheit, Respektkultur, klare Zeiten.
|
||||||
|
|
||||||
|
# Routen (wähle 1)
|
||||||
|
- **A) Bestehenden Kurs/Community beitreten** (z. B. **Swing/Lindy Hop** oder **Tango Argentino** – kommunikationsstark, community-orientiert, häufig alkoholfrei).
|
||||||
|
- **B) Eigene „Practice Night“ (60–90′, 1×/W)** – **Grundschritte + Balance + Musikgefühl**, offenes Ende mit Tee/Wasser.
|
||||||
|
- **C) „Dance & Mobility“ im Dojo (Monat)** – leichter Mix aus Rhythmus, Koordination, Dehnung.
|
||||||
|
|
||||||
|
# Kickoff-Plan (30 Tage)
|
||||||
|
- **W1:** Stil & Slot wählen; Raum sichern; 3-Zeiler Einladung schreiben.
|
||||||
|
- **W2:** Startabend #1 (60–90′): Warm-up, 2–3 Basics, Partnerwechsel, 15′ Gesprächsrunde.
|
||||||
|
- **W3:** Startabend #2; Interessentenliste pflegen; 1:1-Coffees anstoßen.
|
||||||
|
- **W4:** Retrospektive; feste Cadence (z. B. Mi 19:00) festziehen.
|
||||||
|
|
||||||
|
# Einladungstext (eigene Practice Night)
|
||||||
|
Lust auf **Tanzen ohne Druck**?
|
||||||
|
Am **[Datum, Uhrzeit]** startet eine **kleine Practice Night (60–90′)** in **[Ort]** – **Basics, Rhythmus, Balance**, viel Lachen, **ohne Alkohol**.
|
||||||
|
Alle Levels willkommen, Solo oder mit Partner:in.
|
||||||
|
Bringt saubere Schuhe/Strümpfe mit. Danach kurzer Tee-Talk.
|
||||||
|
**Anmeldung:** [Kontakt/Link] – **max. 16 Plätze**.
|
||||||
|
– Lars
|
||||||
|
|
||||||
|
# Messung
|
||||||
|
- **Leading:** **2 Startabende**, **≥12 Teilnehmende kumuliert**, **≥6** 1:1-Gespräche in 14 Tagen.
|
||||||
|
- **Lagging:** **≥2** neue, regelmäßige Mittänzer:innen im Kernkreis bis Jan 2026.
|
||||||
|
|
||||||
|
# Guardrails
|
||||||
|
- **Alkoholfrei**, respektvolle Ansprache, pünktlicher Start/Schluss (Familienzeit schützen).
|
||||||
|
- Keine „Lehrerrolle“ übernehmen – **Facilitator-Modus**: Raum bieten, nicht dominieren.
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
title: Leitbild – Start-Experiment (14 Tage · R1–R7)
|
||||||
|
id: leitbild-start-experiment-2025-11-01
|
||||||
|
type: project
|
||||||
|
status: active
|
||||||
|
created: 2025-10-31
|
||||||
|
updated: 2025-10-31
|
||||||
|
area: personal_development
|
||||||
|
project: leitbild_refresh_2025
|
||||||
|
tags: [experiment, rituale, NACh, tracker, review]
|
||||||
|
lang: de
|
||||||
|
source: coach/leitbild
|
||||||
|
version: 1.0.0
|
||||||
|
depends_on:
|
||||||
|
- leitbild-rituale-system
|
||||||
|
- leitbild-review
|
||||||
|
- leitbild-ziele-portfolio
|
||||||
|
references:
|
||||||
|
- "/mnt/data/Lars- Mein Leitbild.pdf"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Zweck
|
||||||
|
14-Tage-Testlauf, um die täglichen (R1–R3, R6), wöchentlichen (R5, R7) und 14-tägigen (R4) Rituale **unter realen Bedingungen** zu prüfen, Adhärenz zu messen und minimal zu tunen.
|
||||||
|
|
||||||
|
# Hypothesen
|
||||||
|
- **H1 (Ich-Pflege):** R1/R2/R3 erhöhen Ruhe/Klarheit spürbar (Ruhe-Ø **+2** Punkte; Spannungsgefühl **−2**; Stimmung-Ø **≥7/10**).
|
||||||
|
- **H2 (Vater):** R6 (1:1 Rohan) steigert Bindungsgefühl (Ø **+2** Punkte) und reduziert Konfliktlatenz.
|
||||||
|
- **H3 (Partner):** R5 (Tanzen) erhöht Nähe/Verbundenheit (Ø **≥8/10**) und senkt Reibungen bei Entscheidungen (über R4).
|
||||||
|
- **H4 (System):** Adhärenz **≥80 %** über alle täglich geplanten Rituale.
|
||||||
|
|
||||||
|
# Erfolgskriterien (DoD)
|
||||||
|
- **Adhärenz:** R1, R2, R3 jeweils **≥6/7** pro Woche; R6 **≥5/7** pro Woche.
|
||||||
|
- **Events:** R5 **≥1×/Woche**, R7 **≥1×/Woche**, R4 **falls fällig: 1×** in 14 Tagen.
|
||||||
|
- **Wirkung:** Ruhe-Ø **+2**, Spannungsgefühl **−2**, Stimmung-Ø **≥7/10**; Bindung Rohan **+2**.
|
||||||
|
|
||||||
|
# Protokoll
|
||||||
|
- **Start:** 2025-11-01 (Sa) · **Ende:** 2025-11-14 (Fr) · **Mid-Tune:** Tag 7 (2025-11-07).
|
||||||
|
- **Täglicher Ablauf (2–3′):** Tracker ausfüllen (siehe Tabellen unten).
|
||||||
|
- **Weekly Mini-Review (So/Mo):** Scoreboard füllen → 1 Decision Note.
|
||||||
|
- **Regel bei Ausfall:** **Mini-Reset ≤24 h** (kein „alles-egal“).
|
||||||
|
|
||||||
|
# NACh-Reminder (Kurzfassung)
|
||||||
|
- **R1** Meditation 20′ – *Trigger:* erster ruhiger 30′-Block – *Check:* Ruhe 1–10
|
||||||
|
- **R2** Mobility 10′ – *Trigger:* erste 10′ des Lunch – *Check:* Spannung Δ
|
||||||
|
- **R3** Journal – *Trigger:* bevor das Licht ausgeht – *Check:* Stimmung 1–10
|
||||||
|
- **R6** 1:1 Rohan 5′ – *Trigger:* nach Abendessen – *Check:* Bindung 1–10
|
||||||
|
- **R5** Tanzen So 17:00 (Backup Fr 20:00) – *Check:* Nähe 1–10
|
||||||
|
- **R7** 1:1 Rouven Sa 11:00 (Backup Mi 19:30) – *Check:* Nähe 1–10
|
||||||
|
- **R4** Familienrat So 14:00 (alle 2 Wochen; Backup Mo 19:00)
|
||||||
|
|
||||||
|
# Tages-Tracker (14 Tage)
|
||||||
|
date,weekday,R1_y_n,R1_calm_1_10,R2_y_n,R2_tension_delta,R3_y_n,mood_1_10,R6_rohan_y_n,bond_rohan_1_10,R5_tanzen_y_n,closeness_partner_1_10,R7_rouven_y_n,closeness_rouven_1_10,R4_familienrat_y_n,notes
|
||||||
|
2025-11-01,Sa,,,,,,,,,,,,,,"Starttag"
|
||||||
|
2025-11-02,So,,,,,,,,,,,,,,"R5 Tanzen 17:00; ggf. Weekly Mini-Review"
|
||||||
|
2025-11-03,Mo,,,,,,,,,,,,,,
|
||||||
|
2025-11-04,Di,,,,,,,,,,,,,,
|
||||||
|
2025-11-05,Mi,,,,,,,,,,,,,,"Backup-Slot R7 19:30 (falls Sa ausfiel)"
|
||||||
|
2025-11-06,Do,,,,,,,,,,,,,,
|
||||||
|
2025-11-07,Fr,,,,,,,,,,,,,,"Tag 7 – Mid-Tune"
|
||||||
|
2025-11-08,Sa,,,,,,,,,,,,,,"R7 1:1 Rouven 11:00"
|
||||||
|
2025-11-09,So,,,,,,,,,,,,,,"R5 Tanzen 17:00; R4 Familienrat 14:00 (falls fällig)"
|
||||||
|
2025-11-10,Mo,,,,,,,,,,,,,,"Weekly Mini-Review (Backup)"
|
||||||
|
2025-11-11,Di,,,,,,,,,,,,,,
|
||||||
|
2025-11-12,Mi,,,,,,,,,,,,,,"Backup-Slot R7 19:30 (falls Sa ausfiel)"
|
||||||
|
2025-11-13,Do,,,,,,,,,,,,,,
|
||||||
|
2025-11-14,Fr,,,,,,,,,,,,,,"Abschluss – 30′ Abschlussreview"
|
||||||
|
|
||||||
|
# Weekly Mini-Scoreboard (2 Wochen)
|
||||||
|
week_ending,R1_days,R2_days,R3_days,R6_days,R5_done_y_n,R7_done_y_n,bond_rohan_avg,closeness_partner_avg,closeness_rouven,notes
|
||||||
|
2025-11-02,,,,,, , , , "Woche 1 (kurz)"
|
||||||
|
2025-11-09,,,,,, , , , "Woche 2 (voll)"
|
||||||
|
|
||||||
|
# Mid-Tune (Tag 7 · 10 Minuten)
|
||||||
|
- **Fragen:**
|
||||||
|
1) *Was verhindert Adhärenz?* (Energie, Timing, Trigger, Umgebung)
|
||||||
|
2) *Welche 1 Mini-Änderung* steigert die Chance auf „y“ morgen?
|
||||||
|
- **Aktionen (Beispiele):** Meditationskissen sichtbar; Alarm „Lunch-10“; Journalkarte ans Bett; 20:30 Wecker für R6.
|
||||||
|
- **Festhalten:** eine **Decision Note** mit genau **einer** Systemänderung.
|
||||||
|
|
||||||
|
# Abschluss-Review (30 Minuten · 2025-11-14)
|
||||||
|
- **Fakten:** Adhärenz pro Ritual; Ø Ruhe/Spannung/Stimmung; Bindung Rohan.
|
||||||
|
- **Wertung:** Welche Rituale tragen die größte Wirkung? Welche NACh-Teile (Trigger/Aktion/Check) werden fixiert?
|
||||||
|
- **Entscheidung:**
|
||||||
|
- **Behalten:** R1–R3–R6 (und ggf. R5/R7/R4) mit kleinen Anpassungen
|
||||||
|
- **Start/Stop:** 1 neue Mikro-Gewohnheit **oder** 1 weglassen
|
||||||
|
- **Review-System:** `leitbild-review` KPIs ggf. anpassen
|
||||||
|
|
||||||
|
# Verknüpfungen
|
||||||
|
- **Basis & Details:** `leitbild-rituale-system`
|
||||||
|
- **Reviews & KPIs:** `leitbild-review`
|
||||||
|
- **Ziele/Nordsterne:** `leitbild-ziele-portfolio`
|
||||||
@@ -0,0 +1,159 @@
|
|||||||
|
---
|
||||||
|
title: Leitbild – Leitfaden (Betrieb & Rhythmus)
|
||||||
|
id: leitbild-ops-guide
|
||||||
|
type: concept
|
||||||
|
status: active
|
||||||
|
created: 2025-10-31
|
||||||
|
updated: 2025-10-31
|
||||||
|
area: personal_development
|
||||||
|
project: leitbild_refresh_2025
|
||||||
|
tags: [leitbild, operating-guide, review, habits, nordsterne, mindnet]
|
||||||
|
lang: de
|
||||||
|
source: coach/leitbild
|
||||||
|
depends_on:
|
||||||
|
- leitbild-text
|
||||||
|
- leitbild-rituale-system
|
||||||
|
- leitbild-review
|
||||||
|
- leitbild-ziele-portfolio
|
||||||
|
- leitbild-rollen
|
||||||
|
- leitbild-prinzipien
|
||||||
|
references:
|
||||||
|
- "/mnt/data/Lars- Mein Leitbild.pdf"
|
||||||
|
version: 1.0.0
|
||||||
|
---
|
||||||
|
|
||||||
|
# TL;DR – 80/20
|
||||||
|
- **Täglich 10–15′:** R1/R2/R3 (+ R6) ausführen & im Tracker abhaken; abends **3-Zeilen-Journal** notieren.
|
||||||
|
- **Wöchentlich 20–30′:** `leitbild-review` (Weekly) → Scoreboard füllen → **1 Decision Note** schreiben.
|
||||||
|
- **Monatlich 45–60′:** Roll-up + Top-3 Outcomes festlegen.
|
||||||
|
- **Quartalsweise 90′:** Nordsterne & Meilensteine prüfen, Kurs justieren.
|
||||||
|
|
||||||
|
Wenn gar nichts geht: **Mini-Reset ≤24 h** (kein „alles-egal“).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 1) Deine Karte – „Welche Datei wofür?“
|
||||||
|
- **`leitbild-text` (Kompass):** Kurz lesen, wenn du Richtung brauchst. *Wofür stehe ich? Was ist jetzt wichtig?*
|
||||||
|
- **`leitbild-rituale-system` (System):** NACh-Formeln, Trigger, Fallbacks, Tracker. *Ausführen & abhaken.*
|
||||||
|
- **`leitbild-review` (Steuerung):** Weekly/Monthly/Quarterly-Vorlagen, Scoreboards, Decision/Deviation Notes.
|
||||||
|
- **`leitbild-ziele-portfolio` (Ziele):** Nordsterne → Jahresziele → Milestones. *Nur bei echter Planarbeit öffnen.*
|
||||||
|
- **`leitbild-rollen` (Kontext):** Mission + Qualitätskriterien je Rolle. *Wenn Entscheidungen Rollen betreffen.*
|
||||||
|
- **`leitbild-prinzipien` / `leitbild-werte` (Why & How):** Wenn du Dilemmata klärst oder Leitplanken schärfst.
|
||||||
|
- **`leitbild-start-experiment-2025-11-01` (Pilot):** 14-Tage-Test – Logs & Mid-Tune.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 2) Dein Tag in 3 Mikro-Ankern (≤15 Minuten Netto)
|
||||||
|
**Morgens:**
|
||||||
|
- **R1 Meditation 20′** (oder Fallback ≤5′).
|
||||||
|
- *Gedanke (10 Sek.):* „Welche **eine Sache** zahlt heute **sichtbar** auf welchen **Nordstern** ein?“
|
||||||
|
|
||||||
|
**Mittags:**
|
||||||
|
- **R2 Mobility 10′** (erste 10′ des Lunch; Fallback ≤3′).
|
||||||
|
|
||||||
|
**Abends:**
|
||||||
|
- **R6 1:1 Rohan 5′** (nach dem Essen; Fallback: 3 Sätze).
|
||||||
|
- **R3 Journal** – 3 Zeilen: *1 gut · 1 gelernt · 1 nächster Schritt.*
|
||||||
|
- *Abhaken im Tracker* (in `leitbild-rituale-system`).
|
||||||
|
|
||||||
|
> **Hinweis:** R5 (Tanzen So 17:00) & R7 (Rouven Sa 11:00) sind **Termin-Rituale**. R4 (Familienrat So 14:00) alle 2 Wochen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 3) Weekly Review – SOP (20–30′, So Abend oder Mo früh)
|
||||||
|
1. **Scoreboard füllen** (aus `leitbild-review`, Tabelle „Weekly“). *Nur Fakten, keine Bewertung.*
|
||||||
|
2. **Kurz-Reflexion** (5 Fragen aus `leitbild-review` · 10′):
|
||||||
|
- Was war **wirksam** für einen Nordstern?
|
||||||
|
- Was **hinderte** Adhärenz (Energie/Timing/Scope)?
|
||||||
|
- Wo brauchte es **Repair** (ehrlich klären/entschuldigen)?
|
||||||
|
- 2–3 Momente von **Ruhe/Klarheit** – was wiederholen?
|
||||||
|
- **Anerkennung** (Partner/Kinder/Team).
|
||||||
|
3. **Fokus & Entscheidung** (5–10′):
|
||||||
|
- **Top-3 Fokus** kommender Woche (je 1 Zeile, Nordstern zuordnen).
|
||||||
|
- **Genau 1 Decision Note** anlegen (Titel, Entscheidung, Warum, Erfolgsmessung).
|
||||||
|
4. **Kalender checken**: R5/R7/R4 Slots bestätigt? Wenn nicht: **verschieben, nicht streichen**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 4) Monthly Review – SOP (45–60′)
|
||||||
|
- **Monats-Scoreboard** (Roll-up) befüllen.
|
||||||
|
- **Rückblick**: Beste Entscheidung des Monats? Wo trugen Rituale auch an Low-Energy-Tagen? Welche Regel/Standard braucht Feinschliff?
|
||||||
|
- **Plan & Risiken**: Top-3 Outcomes (mit Messgröße), Risiken & Gegenmaßnahmen.
|
||||||
|
- **Decision Notes**: 1–2 Richtungsentscheide.
|
||||||
|
- **Ziel-/Milestone-Check** in `leitbild-ziele-portfolio` (nur falls nötig).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 5) Quarterly Review – SOP (90′)
|
||||||
|
- **Nordsterne-Check**: noch gültig? Welche 1–2 Metriken je Nordstern waren am prädiktivsten?
|
||||||
|
- **Strategie-Retro**: Keep/Start/Stop, System-Retro (Regeln/SLAs, Nein-Liste), Ritual-Tuning.
|
||||||
|
- **Kurskorrektur**: Meilensteine datieren/verschieben, 1–2 größere **Decision Notes**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 6) Decision & Deviation – „Wie entscheide ich sauber?“
|
||||||
|
- **Decision Note**: Eine Seite, ein Vorgang. *Kontext → Optionen → Entscheidung (Warum) → Erfolgsmetrik → Review-Datum.*
|
||||||
|
- **Deviation („Schuldschein“)**: Wenn du bewusst von Regel/Prinzip abweichst. *Abweichung → Begründung → Impact → Repair → Next-Check.*
|
||||||
|
- Beide Vorlagen stehen in `leitbild-review` (eingrückte Templates), damit du sie kopieren kannst.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 7) Start-Experiment (14 Tage) nutzen
|
||||||
|
- **Täglich 2–3′**: Zeile im Tages-Tracker füllen (R1/R2/R3/R6 …).
|
||||||
|
- **Tag 7**: **Mid-Tune** – 1 kleine Systemänderung (z. B. Reminder, Cue, Fallback). **Als Decision Note festhalten.**
|
||||||
|
- **Tag 14**: **Abschlussreview** – Was behalten/ändern/streichen? *Version bump* in `leitbild-rituale-system`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 8) Ordnung halten – Dateien, Edges, Versionen
|
||||||
|
- **Edges**: In jeder Note `depends_on`/`references` prüfen (v. a. `leitbild-text` ↔ Werte/Prinzipien/Rollen/Rituale/Review).
|
||||||
|
- **Versionen**: Bei jeder inhaltlichen Anpassung **version** + **Changelog** aktualisieren.
|
||||||
|
- **Archiv/Git**: Commit + Tag, z. B. `leitbild-text-1.0.0`, `review-0.1.0`, `rituale-0.5.2`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 9) Optional: Kalender-Snippets (iCal, anpassbar)
|
||||||
|
BEGIN:VEVENT
|
||||||
|
SUMMARY:Weekly Review (Leitbild)
|
||||||
|
RRULE:FREQ=WEEKLY;BYDAY=SU;BYHOUR=19;BYMINUTE=30;BYSECOND=0
|
||||||
|
END:VEVENT
|
||||||
|
|
||||||
|
BEGIN:VEVENT
|
||||||
|
SUMMARY:Monthly Review (Leitbild)
|
||||||
|
RRULE:FREQ=MONTHLY;BYMONTHDAY=28;BYHOUR=18;BYMINUTE=0;BYSECOND=0
|
||||||
|
END:VEVENT
|
||||||
|
|
||||||
|
BEGIN:VEVENT
|
||||||
|
SUMMARY:Quarterly Review (Leitbild)
|
||||||
|
RRULE:FREQ=MONTHLY;INTERVAL=3;BYMONTHDAY=2;BYHOUR=09;BYMINUTE=00;BYSECOND=0
|
||||||
|
END:VEVENT
|
||||||
|
|
||||||
|
> Passe Tag/Uhrzeit an deine Slots an (oder lösche diesen Abschnitt, wenn du Kalender separat pflegst).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 10) Wenn Zeit & Energie knapp sind
|
||||||
|
- **Regel:** **„Weniger, dafür besser“** – Fallbacks **ziehen**, nicht auslassen.
|
||||||
|
- **Heute nur 1 Sache:** *Welche Mini-Aktion bringt den größten Schritt für **einen** Nordstern?*
|
||||||
|
- **Repair first:** Wenn etwas verletzt, **klären** (ehrlich, respektvoll).
|
||||||
|
- **Mini-Reset ≤24 h**: Ausfall bleibt **Ereignis**, wird **nicht** zur neuen Regel.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 11) Qualitätssicherung (Definition of Done)
|
||||||
|
- Weekly Review in Kalender absolviert; **1 Decision Note pro Woche** geschrieben.
|
||||||
|
- R1/R2/R3 **≥6/7**, R6 **≥5/7**, R5 **≥3/4**, R7 **≥1/1**, R4 **≥2/Monat**.
|
||||||
|
- Alle Anpassungen mit **Version + Changelog** dokumentiert.
|
||||||
|
- **Edges konsistent** (Backlinks zwischen Kompass, System, Review, Zielen, Rollen, Prinzipien).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 12) Quick-Open (Was öffne ich, wenn …?)
|
||||||
|
- **… ich mich verliere?** → `leitbild-text` (Kompass, 2 Minuten lesen).
|
||||||
|
- **… mein Tag klemmt?** → `leitbild-rituale-system` (NACh + Fallbacks, 1 Aktion jetzt).
|
||||||
|
- **… ich eine Entscheidung schiebe?** → `leitbild-review` → **Decision Note** anlegen.
|
||||||
|
- **… Haushalt/Regeln knirschen?** → R4 Familienrat (Agenda) + Deviation/Decision Note.
|
||||||
|
- **… ich Fokus brauche?** → Weekly Review „Top-3 Fokus“ erneut anschauen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
Loaded 100 of 145 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user