feat: Refactor activity data handling and improve CSV import logic
All checks were successful
Deploy Development / deploy (push) Successful in 53s
Build Test / pytest-backend (push) Successful in 4s
Build Test / lint-backend (push) Successful in 0s
Build Test / build-frontend (push) Successful in 16s

- Updated `ACTIVITY_PRODUCTION_ARCHITECTURE_AND_PHASES.md` to clarify the derivation of `ACTIVITY_MODULE_REGISTRY_FIELD_KEYS` from `csv_parser.module_registry`.
- Enhanced `activity_data_canon.py` to eliminate hardcoded key lists, ensuring all registry fields are derived dynamically.
- Refactored the `_import_activity` function to remove redundant parameters and streamline the import process.
- Improved the `insert_activity_csv_minimal` function to handle metrics exclusively through `update_activity_columns`, preventing hardcoded values.
- Updated frontend components to manage editable activity log fields more effectively, ensuring proper handling of metrics during CSV imports.
- Added unit tests to validate the new logic and ensure consistency in activity session metrics handling.
This commit is contained in:
Lars 2026-04-16 10:35:08 +02:00
parent cd29c7d433
commit 5cda485458
9 changed files with 191 additions and 135 deletions

View File

@ -41,7 +41,7 @@ KI / UI / Export
### 2.2 `activity_log` (Spine + heiße Skalare)
**Maschinenlesbarer Kanon:** `backend/data_layer/activity_data_canon.py` (`ACTIVITY_MODULE_REGISTRY_FIELD_KEYS`, `ACTIVITY_EAV_PRIMARY_PARAMETER_KEYS`, Legacy-Lesefallback für EAV-primäre Parameter).
**Maschinenlesbarer Kanon:** `backend/data_layer/activity_data_canon.py` `ACTIVITY_MODULE_REGISTRY_FIELD_KEYS` wird aus `csv_parser.module_registry` (`activity.fields`) abgeleitet; zusätzlich `ACTIVITY_EAV_PRIMARY_PARAMETER_KEYS` und Legacy-Lesefallback für EAV-primäre Parameter.
**Immer (fachlich minimal + listenfähig):** `id`, `profile_id`, Kalender-/Zeitfenster (`date`, `started_at`/`ended_at`, ggf. `start_time`/`end_time` bis Konsolidierung), `duration_min`, `training_type_id` (+ ggf. denormalisierte Kategorie), Legacy `activity_type`, `notes`, `source`, `created`.
@ -101,7 +101,7 @@ Phasen sind **sequentiell** wo „Abhängigkeit“ steht; Teile können parallel
**Inhalt:** Schriftliche **Kanon-Tabelle**: pro Messgröße genau eine Quelle (`activity_log` | `eav_scalar` | `eav_composite` | `session_quality`). Liste der Keys, für die **Sync/Spiegelung** endet.
**Definition of Done:** Review im Team; Referenz in diesem Dokument oder Verweis auf Gitea-Kommentar; keine Code-Änderung zwingend.
**Definition of Done:** Review im Team; Referenz in diesem Dokument oder Verweis auf Gitea-Kommentar. **Code (2026-04-16):** Spine-Keys des CSV-Moduls `activity` sind nur noch die Registry-Keys (`get_activity_module_registry_field_keys`); CSV-Minimal-Insert + `update_activity_columns` + DB-Read im Import-Eval-Hook — keine duplizierte hr_avg-Verdrahtung im Executor.
**Erster konkreter Schritt:** Kanon-Tabelle als Checkliste (Spreadsheet oder Gitea-Issue) **eine Zeile pro Semantik**.

View File

@ -6,7 +6,7 @@
**Zielarchitektur, Phasenplan (Produktionsreife):** [`ACTIVITY_PRODUCTION_ARCHITECTURE_AND_PHASES.md`](./ACTIVITY_PRODUCTION_ARCHITECTURE_AND_PHASES.md) Kanon `activity_log`/EAV, Composites, Import, Layer 1/2, Reihenfolge AF.
**Kanon (Code):** `backend/data_layer/activity_data_canon.py` (Repo-Root) — CSV-Modul `activity` vs. EAV-primär; Migration **057**.
**Kanon (Code):** `backend/data_layer/activity_data_canon.py` — Spine-Keys **nur** aus `csv_parser.module_registry` (`get_activity_module_registry_field_keys()` → `ACTIVITY_MODULE_REGISTRY_FIELD_KEYS`); kein paralleles Hardcoding. EAV-primär + Migration **057** unverändert.
---
@ -97,7 +97,7 @@ Router: `backend/routers/admin_training_parameters.py`, `backend/routers/admin_a
Siehe **Phasen AF** in [`ACTIVITY_PRODUCTION_ARCHITECTURE_AND_PHASES.md`](./ACTIVITY_PRODUCTION_ARCHITECTURE_AND_PHASES.md). Kurz:
- [ ] **Phase A:** Kanon-Tabelle (eine Quelle pro Semantik).
- [x] **Phase A (Code-Kanon):** Spine-Felder = Registry `activity.fields`; `insert_activity_csv_minimal` nur Kopf, Metriken via `update_activity_columns` / `activity_csv_registry_updates_from_mapped`; Import-Eval liest Session aus DB. *(Spreadsheet „eine Semantik pro Zeile“ weiterhin fachlich empfohlen.)*
- [ ] **Phase B:** Lesepfad Layer 1 härten (Consumer-Audit).
- [ ] **Phase C:** Schreibpfad: Doppelhaltung / Sync stufenweise abschalten.
- [ ] **Phase D:** Composite-MVP (ein Archetyp E2E).

View File

@ -899,12 +899,6 @@ def _import_activity(
start_time=workout_start_t,
end_time=end_str or None,
activity_type=wtype,
duration_min=registry_updates.get("duration_min"),
kcal_active=registry_updates.get("kcal_active"),
kcal_resting=registry_updates.get("kcal_resting"),
hr_avg=registry_updates.get("hr_avg"),
hr_max=registry_updates.get("hr_max"),
distance_km=registry_updates.get("distance_km"),
training_type_id=training_type_id,
training_category=training_category,
training_subcategory=training_subcategory,
@ -921,14 +915,7 @@ def _import_activity(
cur,
profile_id,
str(aid),
workout_date=iso,
training_type_id=training_type_id,
duration_min=registry_updates.get("duration_min"),
hr_avg=registry_updates.get("hr_avg"),
hr_max=registry_updates.get("hr_max"),
distance_km=registry_updates.get("distance_km"),
kcal_active=registry_updates.get("kcal_active"),
kcal_resting=registry_updates.get("kcal_resting"),
)
upsert_session_metrics_from_csv_mapped(
cur,

View File

@ -5,30 +5,31 @@ Single Source für: welche Felder das CSV-/Registry-Modul „activity“ direkt
und welche training_parameters primär über EAV laufen (mit optionalem Lesefallback auf Legacy-Spalten).
Normative Doku: .claude/docs/technical/ACTIVITY_PRODUCTION_ARCHITECTURE_AND_PHASES.md
Phase A: Keine zweite hartcodierte Key-Liste Registry-Felder kommen ausschließlich aus
``csv_parser.module_registry.MODULE_DEFINITIONS["activity"].fields``.
"""
from __future__ import annotations
from typing import Dict, Final
from csv_parser.module_registry import get_module_definition
def get_activity_module_registry_field_keys() -> frozenset[str]:
"""Keys des Universal-CSV-Moduls ``activity`` (= Spine-Spalten-Namen in activity_log)."""
mod = get_module_definition("activity")
if not mod:
return frozenset()
return frozenset((mod.get("fields") or {}).keys())
# ── activity_log: Modul „activity“ (Universal-CSV-Kern) ───────────────────────
# Nur diese Keys erscheinen in csv_parser.module_registry MODULE_DEFINITIONS["activity"].fields.
# Alles Weitere: training_parameters + EAV (Import über upsert_session_metrics_from_csv_mapped).
ACTIVITY_MODULE_REGISTRY_FIELD_KEYS: Final[frozenset[str]] = frozenset(
{
"date",
"start_time",
"end_time",
"activity_type",
"duration_min",
"kcal_active",
"kcal_resting",
"distance_km",
"hr_avg",
"hr_max",
"rpe",
"notes",
}
)
# Ableitung aus module_registry — bei neuen Registry-Feldern hier kein manuelles Update nötig.
ACTIVITY_MODULE_REGISTRY_FIELD_KEYS: Final[frozenset[str]] = get_activity_module_registry_field_keys()
# Teil-UPDATEs (Import): alle Registry-Kernfelder außer ``date`` (Identität/Duplikat-Key).
ACTIVITY_LOG_PATCHABLE_COLUMNS: Final[frozenset[str]] = ACTIVITY_MODULE_REGISTRY_FIELD_KEYS - {"date"}
# Parameter-Keys (training_parameters.key), die primär in EAV geführt werden; source_field nach Migration 057 NULL.
# Lesefallback: activity_log-Spalte unter ACTIVITY_LOG_LEGACY_COLUMN_FOR_EAV_PRIMARY_PARAM, falls EAV leer.
@ -58,21 +59,3 @@ ACTIVITY_LOG_LEGACY_COLUMN_FOR_EAV_PRIMARY_PARAM: Final[Dict[str, str]] = {
"avg_hr_percent": "avg_hr_percent",
"kcal_per_km": "kcal_per_km",
}
# Spalten, die mit training_parameters.source_field (nach Migration 057) noch activity_log abbilden.
# Erweiterte Metriken sind EAV-primär — nicht hier auflisten.
ACTIVITY_LOG_PATCHABLE_COLUMNS: Final[frozenset[str]] = frozenset(
{
"start_time",
"end_time",
"activity_type",
"duration_min",
"kcal_active",
"kcal_resting",
"hr_avg",
"hr_max",
"distance_km",
"rpe",
"notes",
}
)

View File

@ -15,6 +15,7 @@ from typing import Any, Dict, List, Mapping, Optional
from models import ActivityEntry
from csv_parser.module_registry import get_module_definition
from data_layer.activity_data_canon import get_activity_module_registry_field_keys
logger = logging.getLogger(__name__)
@ -50,10 +51,8 @@ _ACTIVITY_CSV_REGISTRY_EXCLUDE = frozenset({"date", "start_time", "end_time", "a
def activity_registry_field_keys() -> frozenset[str]:
mod = get_module_definition("activity")
if not mod:
return frozenset()
return frozenset((mod.get("fields") or {}).keys())
"""Gleiche Menge wie ``ACTIVITY_MODULE_REGISTRY_FIELD_KEYS`` (Phase A: eine Quelle)."""
return get_activity_module_registry_field_keys()
def activity_csv_registry_updates_from_mapped(mapped: Mapping[str, Any]) -> Dict[str, Any]:
@ -217,18 +216,17 @@ def insert_activity_csv_minimal(
start_time: Any,
end_time: Any,
activity_type: str,
duration_min: Any,
kcal_active: Any,
kcal_resting: Any,
hr_avg: Any,
hr_max: Any,
distance_km: Any,
training_type_id: Any,
training_category: Any,
training_subcategory: Any,
source: str,
) -> None:
"""INSERT minimale activity_log-Zeile (Universal-CSV)."""
"""
INSERT Kopfzeile für Universal-CSV / Legacy-Import.
Metriken aus ``activity_csv_registry_updates_from_mapped`` (oder manuelles Dict)
ausschließlich via ``update_activity_columns``; keine fest verdrahteten hr_avg-Parameter.
"""
cur.execute(
"""
INSERT INTO activity_log (
@ -236,7 +234,7 @@ def insert_activity_csv_minimal(
kcal_active, kcal_resting, hr_avg, hr_max, distance_km,
source, training_type_id, training_category, training_subcategory, created
)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,CURRENT_TIMESTAMP)
VALUES (%s,%s,%s,%s,%s,%s,NULL,NULL,NULL,NULL,NULL,NULL,%s,%s,%s,%s,CURRENT_TIMESTAMP)
""",
(
eid,
@ -245,12 +243,6 @@ def insert_activity_csv_minimal(
start_time,
end_time,
activity_type,
duration_min,
kcal_active,
kcal_resting,
hr_avg,
hr_max,
distance_km,
source,
training_type_id,
training_category,
@ -288,37 +280,32 @@ def run_activity_post_write_hooks_import(
profile_id: str,
eid: str,
*,
workout_date: str,
training_type_id: Optional[int],
duration_min: Any,
hr_avg: Any,
hr_max: Any,
distance_km: Any,
kcal_active: Any,
kcal_resting: Any,
training_type_id: Optional[int] = None,
) -> None:
"""Auto-Eval nach Import. Kein Spalte→EAV-Sync (siehe run_activity_post_write_hooks)."""
if _EVALUATION_AVAILABLE and training_type_id and _evaluate_and_save_activity:
try:
activity_dict = {
"id": eid,
"profile_id": profile_id,
"date": workout_date,
"training_type_id": training_type_id,
"duration_min": duration_min,
"hr_avg": hr_avg,
"hr_max": hr_max,
"distance_km": distance_km,
"kcal_active": kcal_active,
"kcal_resting": kcal_resting,
"rpe": None,
"pace_min_per_km": None,
"cadence": None,
"elevation_gain": None,
}
_evaluate_and_save_activity(cur, eid, activity_dict, training_type_id, profile_id)
except Exception as eval_err:
logger.warning("[activity import] Auto-Eval fehlgeschlagen: %s", eval_err)
"""Auto-Eval nach Import — liest die Session aus der DB (gleiche Felder wie REST-Hook)."""
if not _EVALUATION_AVAILABLE or not _evaluate_and_save_activity:
return
cur.execute(
"""
SELECT id, profile_id, date, training_type_id, duration_min,
hr_avg, hr_max, distance_km, kcal_active, kcal_resting,
rpe, pace_min_per_km, cadence, elevation_gain
FROM activity_log
WHERE id = %s AND profile_id = %s
""",
(eid, profile_id),
)
row = cur.fetchone()
if not row:
return
activity_dict = dict(row)
tid = training_type_id if training_type_id is not None else activity_dict.get("training_type_id")
if not tid:
return
try:
_evaluate_and_save_activity(cur, eid, activity_dict, int(tid), profile_id)
except Exception as eval_err:
logger.warning("[activity import] Auto-Eval fehlgeschlagen: %s", eval_err)
def merge_activity_csv_module_fields(

View File

@ -9,8 +9,10 @@ import logging
from decimal import Decimal
from typing import Any, Dict, List, Mapping, Optional, Sequence
from csv_parser.module_registry import get_module_definition
from data_layer.activity_data_canon import ACTIVITY_LOG_LEGACY_COLUMN_FOR_EAV_PRIMARY_PARAM
from data_layer.activity_data_canon import (
ACTIVITY_LOG_LEGACY_COLUMN_FOR_EAV_PRIMARY_PARAM,
ACTIVITY_MODULE_REGISTRY_FIELD_KEYS,
)
logger = logging.getLogger(__name__)
@ -29,6 +31,16 @@ ACTIVITY_LOG_PATCH_FORBIDDEN = frozenset(
)
def _parameter_value_stored_in_eav_only(spec: Mapping[str, Any], parameter_key: str) -> bool:
"""False = kanonisch activity_log (Modul-Registry oder training_parameters.source_field)."""
if parameter_key in ACTIVITY_MODULE_REGISTRY_FIELD_KEYS:
return False
sf = spec.get("source_field")
if sf is not None and str(sf).strip():
return False
return True
class ActivitySessionMetricsError(Exception):
"""Raised by Layer 1; routers map to HTTP (404/400)."""
@ -299,8 +311,6 @@ def upsert_session_metrics_from_csv_mapped(
row = cur.fetchone()
if not row or str(row["profile_id"]) != str(profile_id):
return
mod = get_module_definition("activity") or {}
activity_registry_keys = frozenset((mod.get("fields") or {}).keys())
schema = resolve_activity_attribute_schema(cur, training_category, training_type_id)
for spec in schema:
pkey = spec["key"]
@ -309,10 +319,7 @@ def upsert_session_metrics_from_csv_mapped(
raw = mapped[pkey]
if raw is None or raw == "":
continue
if pkey in activity_registry_keys:
continue
sf_raw = spec.get("source_field")
if sf_raw is not None and str(sf_raw).strip():
if not _parameter_value_stored_in_eav_only(spec, pkey):
continue
tid = spec["training_parameter_id"]
dt = spec["data_type"]
@ -568,6 +575,8 @@ def replace_activity_session_metrics(
if not s["required"]:
continue
itk = s["key"]
if not _parameter_value_stored_in_eav_only(s, itk):
continue
hit = payload_by_key.get(itk)
if hit is None or hit.get("value") is None:
raise ActivitySessionMetricsError(400, f"Pflichtfeld fehlt: {itk}")
@ -580,9 +589,11 @@ def replace_activity_session_metrics(
for item in metrics:
k = str(item["parameter_key"]).strip()
spec = by_key[k]
if not _parameter_value_stored_in_eav_only(spec, k):
continue
val = item.get("value")
if val is None:
if spec["required"]:
if spec["required"] and _parameter_value_stored_in_eav_only(spec, k):
raise ActivitySessionMetricsError(400, f"Pflichtfeld fehlt: {k}")
continue
rules = _validation_rules_dict(spec["validation_rules"])

View File

@ -641,14 +641,7 @@ async def import_activity_csv(file: UploadFile=File(...), x_profile_id: Optional
cur,
pid,
str(existing_id),
workout_date=workout_date,
training_type_id=training_type_id,
duration_min=duration_min,
hr_avg=hr_av,
hr_max=hr_mx,
distance_km=dist_km,
kcal_active=kcal_a,
kcal_resting=kcal_r,
)
else:
new_id = new_activity_id()
@ -660,30 +653,31 @@ async def import_activity_csv(file: UploadFile=File(...), x_profile_id: Optional
start_time=workout_start_t,
end_time=row.get("End", "") or None,
activity_type=wtype,
duration_min=duration_min,
kcal_active=kcal_a,
kcal_resting=kcal_r,
hr_avg=hr_av,
hr_max=hr_mx,
distance_km=dist_km,
training_type_id=training_type_id,
training_category=training_category,
training_subcategory=training_subcategory,
source="apple_health",
)
apple_metrics = {
k: v
for k, v in {
"duration_min": duration_min,
"kcal_active": kcal_a,
"kcal_resting": kcal_r,
"hr_avg": hr_av,
"hr_max": hr_mx,
"distance_km": dist_km,
}.items()
if v is not None
}
if apple_metrics:
update_activity_columns(cur, pid, new_id, apple_metrics)
inserted += 1
run_activity_post_write_hooks_import(
cur,
pid,
new_id,
workout_date=workout_date,
training_type_id=training_type_id,
duration_min=duration_min,
hr_avg=hr_av,
hr_max=hr_mx,
distance_km=dist_km,
kcal_active=kcal_a,
kcal_resting=kcal_r,
)
except Exception as e:
logger.warning(f"Import row failed: {e}")

View File

@ -11,6 +11,7 @@ from data_layer.activity_session_metrics import (
enrich_sessions_with_metrics,
merge_column_backed_and_eav_metrics,
merge_parameter_schema_rows,
replace_activity_session_metrics,
resolve_activity_attribute_schema,
upsert_session_metrics_from_csv_mapped,
_row_value_tuple,
@ -358,6 +359,61 @@ def test_upsert_csv_skips_parameter_with_source_field(mock_schema):
assert cur.asm_inserts == 0
@patch("data_layer.activity_session_metrics.fetch_activity_session_metrics")
@patch("data_layer.activity_session_metrics.resolve_activity_attribute_schema")
def test_replace_metrics_skips_column_backed_kcal(mock_schema, mock_fetch):
"""PUT /metrics: keine EAV-Zeile für kcal_active (liegt in activity_log)."""
pid = str(uuid.uuid4())
eid = str(uuid.uuid4())
mock_schema.return_value = [
{
"training_parameter_id": 1,
"key": "kcal_active",
"data_type": "float",
"validation_rules": {},
"source_field": "kcal_active",
"required": False,
},
{
"training_parameter_id": 2,
"key": "custom_reps",
"data_type": "integer",
"validation_rules": {"min": 0},
"source_field": None,
"required": False,
},
]
mock_fetch.return_value = []
class Cur:
def __init__(self):
self.asm_inserts = 0
def execute(self, sql, params=None):
if "INSERT INTO activity_session_metrics" in sql:
self.asm_inserts += 1
def fetchone(self):
return {
"profile_id": pid,
"training_category": "strength",
"training_type_id": 1,
}
cur = Cur()
replace_activity_session_metrics(
cur,
pid,
eid,
[
{"parameter_key": "kcal_active", "value": 450.0},
{"parameter_key": "custom_reps", "value": 12},
],
)
assert cur.asm_inserts == 1
mock_fetch.assert_called_once_with(cur, eid)
def test_merge_eav_primary_falls_back_to_legacy_hr_min_column():
"""Kanon: min_hr ohne source_field / ohne EAV — Lesefallback Spalte hr_min."""
schema = [
@ -374,3 +430,18 @@ def test_merge_eav_primary_falls_back_to_legacy_hr_min_column():
assert len(out) == 1
assert out[0]["key"] == "min_hr"
assert out[0]["value"] == 88
def test_activity_module_registry_field_keys_match_csv_module_definition():
"""Phase A: Kanon-Spine = module_registry „activity“.fields (keine zweite Liste)."""
from csv_parser.module_registry import get_module_definition
from data_layer.activity_data_canon import (
ACTIVITY_MODULE_REGISTRY_FIELD_KEYS,
get_activity_module_registry_field_keys,
)
mod = get_module_definition("activity")
assert mod is not None
expected = frozenset((mod.get("fields") or {}).keys())
assert get_activity_module_registry_field_keys() == expected
assert ACTIVITY_MODULE_REGISTRY_FIELD_KEYS == expected

View File

@ -96,6 +96,16 @@ const ACTIVITY_LOG_PAYLOAD_KEYS = new Set([
'training_subcategory',
])
/** activity_log-Spalten, die im EntryForm editiert werden (nicht aus metricDraft überschreiben). */
const ACTIVITY_ENTRY_FORM_COLUMNS = new Set([
'duration_min',
'kcal_active',
'hr_avg',
'hr_max',
'rpe',
'notes',
])
function empty() {
return {
date: dayjs().format('YYYY-MM-DD'),
@ -113,6 +123,9 @@ function empty() {
function buildMetricsPayload(schema, draft) {
const out = []
for (const s of schema) {
if (s.source_field && ACTIVITY_LOG_PAYLOAD_KEYS.has(String(s.source_field))) {
continue
}
const raw = draft[s.key]
if (s.data_type === 'boolean') {
if (raw === '' || raw === null || raw === undefined) {
@ -148,14 +161,23 @@ function SessionMetricsFields({ schema, values, setValues, metrics }) {
const schemaList = Array.isArray(schema) ? schema : []
const metricRows = Array.isArray(metrics) ? metrics : []
const schemaKeys = new Set(schemaList.map((s) => s.key))
const orphanMetrics = metricRows.filter((row) => row && row.key && !schemaKeys.has(row.key))
const isHeadColumnMetric = (s) =>
s && s.source_field && ACTIVITY_LOG_PAYLOAD_KEYS.has(String(s.source_field))
const schemaForProfileOnly = schemaList.filter((s) => !isHeadColumnMetric(s))
const orphanMetrics = metricRows.filter(
(row) =>
row &&
row.key &&
!schemaKeys.has(row.key) &&
!ACTIVITY_LOG_PAYLOAD_KEYS.has(row.key)
)
if (schemaList.length === 0 && orphanMetrics.length === 0) return null
if (schemaForProfileOnly.length === 0 && orphanMetrics.length === 0) return null
const set = (k, v) => setValues((prev) => ({ ...prev, [k]: v }))
return (
<div style={{ marginTop: 12, paddingTop: 12, borderTop: '1px solid var(--border)' }}>
<div style={{ fontSize: 13, fontWeight: 600, marginBottom: 8 }}>Weitere Kennwerte (Profil)</div>
{schemaList.map((s) => (
{schemaForProfileOnly.map((s) => (
<div key={s.key} className="form-row">
<label className="form-label">
{s.name_de}
@ -588,8 +610,9 @@ export default function ActivityPage() {
for (const s of sessionDetail?.schema || []) {
const col = s.source_field
if (!col || !ACTIVITY_LOG_PAYLOAD_KEYS.has(col)) continue
if (!(s.key in metricDraft)) continue
const raw = metricDraft[s.key]
const useFormColumn = ACTIVITY_ENTRY_FORM_COLUMNS.has(col)
if (!useFormColumn && !(s.key in metricDraft)) continue
const raw = useFormColumn ? editing[col] : metricDraft[s.key]
const rawStr = raw === null || raw === undefined ? '' : String(raw).trim()
if (rawStr === '') {
payload[col] = null