- Incremented DB_SCHEMA_VERSION to "20260409b" to reflect changes related to the vitals_baseline.source CSV migration.
- Updated comments to clarify the migration context for better maintainability.
- Updated the aggregate_mapped_rows function to support multiple row policies, allowing for flexible handling of duplicate keys during CSV imports.
- Introduced deduplication of identical rows before aggregation, improving data integrity.
- Enhanced validation for multi_row_policy and dedupe_identical_rows in import_row_processing specifications.
- Updated the AdminCsvTemplateEditorPage to include options for multi-row policies and deduplication settings, improving user experience in template management.
- Added comprehensive tests to validate new aggregation behaviors and ensure correct error handling for multiple rows.
- Bumped version numbers for csv_import to 0.3.1 and admin_csv_templates to 0.2.0, reflecting recent enhancements.
- Added support for import_row_processing_default in the CSV modules endpoint, improving data handling capabilities.
- Introduced new row aggregation operations in the AdminCsvTemplateEditorPage, allowing for more flexible data processing options.
- Implemented parsing and validation for custom row processing configurations, enhancing user experience in template management.
- Updated the CSV import logic to support new row processing specifications for weight and vitals baseline, allowing for better data aggregation and validation.
- Implemented handling for multiple rows on the same day, enabling aggregation of values such as averages for vitals and last values for weight.
- Enhanced test coverage for the new import functionalities, ensuring correct behavior during data processing and validation.
- Refactored the module registry to include default import row processing options for better flexibility in handling CSV data.
- Updated the nutrition import logic to utilize a new row processing specification, improving data aggregation and validation.
- Refactored the template rendering process in the workflow executor to use Jinja2's Environment with ChainableUndefined for better handling of missing attributes.
- Added backward-compatible shortcuts for accessing decision signals in node contexts, enhancing flexibility in template usage.
- Introduced import row processing options in CSV templates, allowing for more customizable data handling during imports.
- Updated the test execution command in the CI workflow to run all tests excluding slow ones, improving efficiency.
- Enhanced the AdminCsvTemplateEditorPage to support custom equivalence for unit conversions, allowing for more flexible data handling.
- Added markers in pytest configuration for categorizing tests, facilitating better test management.
- Adjusted the extract_value_raw function to return failure for unavailable values in strict mode.
- Expanded the circumference detection logic in infer_unit_strict to include additional terms for better accuracy in unit inference.
- Updated the source_unit_choices_for_field function to include a custom option for user-defined conversion factors, improving flexibility in unit conversions.
- Modified the AdminCsvTemplateEditorPage to support custom conversion factors, allowing users to input specific scaling factors for their data.
- Added tests to ensure the custom option is correctly included in the source unit choices and functions as expected in the template editor.
- Changed source unit representation from "kJ" to "kj" for consistency across CSV templates and migrations.
- Updated CI workflow to enhance testing conditions, ensuring tests run in the correct environment based on deployment context.
- Improved job steps for backend testing and syntax checking by utilizing deployed application directories, streamlining the CI process.
- Added workflow_run triggers for "Deploy Development" and "Deploy Production" to ensure tests run only after successful deployments.
- Updated Python version in CI from 3.12 to 3.11 for better compatibility with the Debian 12 ARM64 runner.
- Enhanced job conditions to skip tests on failed workflow runs.
- Improved frontend build process by updating Node.js setup and ensuring correct directory navigation.
- Refined CSV parsing logic to handle custom and unknown source units, enhancing conversion flexibility.
- Added new tests for custom source unit handling in CSV conversions, ensuring accurate processing.
- Updated the CSV import architecture to clarify the distinction between import and data layer responsibilities, as outlined in the new section of ARCHITECTURE.md.
- Enhanced the build_row_after_mapping function to include module-specific context for improved data processing.
- Introduced source unit options in the admin CSV template editor to facilitate user-defined conversions, improving flexibility in handling various data formats.
- Added new tests to validate the handling of source units and ensure accurate conversions during CSV imports.
- Updated module definitions to include unit specifications for nutritional and activity data fields, enhancing data integrity.
- Integrated date parsing improvements using dateutil for better handling of various date formats in sleep data.
- Added total sleep hours to the nights dictionary for comprehensive sleep analysis.
- Updated the import logic to handle cases where sleep duration is zero, providing appropriate warnings.
- Enhanced the CSV import interface to detect Apple sleep CSV format and provide user feedback on template selection.
- Improved the admin CSV template editor to accommodate new sleep import requirements and clarify usage instructions.
- Added support for new CSV import modules: sleep and vitals_baseline, expanding the import capabilities.
- Implemented backend logic for handling CSV imports related to sleep and vitals baseline, including error handling and data processing.
- Updated frontend components to include new modules in the CSV import interface, improving user experience.
- Introduced unit tests for the new import functionalities to ensure reliability and correctness.
- Enhanced existing CSV analysis features to accommodate the new modules, ensuring consistent behavior across the application.
- Added new functions for calculating header signature recall and ranking metrics, improving the analysis of CSV templates.
- Updated existing CSV analysis endpoints to utilize the new ranking metrics, enhancing the accuracy of template matching.
- Refactored related code to replace Jaccard score calculations with the new metrics, providing a more comprehensive evaluation of CSV structure.
- Improved documentation for new functions to clarify their purpose and usage in the context of CSV template analysis.
- Added a new endpoint for analyzing uploaded CSV files, providing suggestions for field mappings and type conversions.
- Implemented validation for required field targets to ensure all mandatory fields are mapped correctly.
- Enhanced the admin CSV templates interface with new routes and navigation options in the frontend.
- Updated API utility functions to support the new CSV analysis functionality.
- Improved error handling for CSV uploads, including file size and row count checks.
- Added a new function to strip header keys of unwanted characters, improving CSV import consistency.
- Updated CSV row iteration to utilize the new header normalization function, ensuring cleaner data processing.
- Enhanced date parsing capabilities to support flexible formats, accommodating various date representations in CSV files.
- Introduced additional tests to validate the new header normalization and date parsing functionalities.
- Bumped version of csv_import to 0.3.0, reflecting new analysis capabilities.
- Modified analyze_csv endpoint to allow optional module filtering, improving flexibility in template selection.
- Enhanced the import process to support both system and user-defined templates, ensuring backward compatibility.
- Updated frontend to streamline mapping choices and improve user experience during CSV analysis and import.
- Added detailed error handling and user feedback for import operations.
- Updated version for csv_import to 0.2.0, reflecting new features.
- Implemented a new POST endpoint for universal CSV import, supporting nutrition, weight, and blood pressure modules.
- Added CSV parsing function to yield rows as dictionaries for easier data handling.
- Enhanced error handling and logging for import operations.
- Introduced tests for the new CSV parsing functionality to ensure reliability.
- Changed feature ID from 'csv_import' to 'data_import' in the features table.
- Updated foreign key references in tier_limits, user_feature_restrictions, user_feature_usage, and widget_feature_requirements.
- Removed the old 'csv_import' feature entry after ensuring all references are updated.
- Simplified the migration process by using an INSERT with ON CONFLICT for the new feature entry.
- Added permissions for editing and deleting CSV field mappings.
- Created type converter for CSV cells to handle various data types.
- Implemented database migrations for CSV field mappings and import logs.
- Seeded initial system templates for nutrition and activity data imports.
- Developed admin endpoints for managing system CSV templates.
- Introduced user endpoints for CSV import analysis and mapping retrieval.
- Added tests for core CSV parser functionalities, including delimiter detection and value conversion.
ROOT ARCHITECTURAL CHANGE:
Multiple questions with same type are now supported!
Problem:
- question_augmenter used q.type as LLM key
- If two questions had type="unsicherheit":
- LLM saw duplicate keys: "- unsicherheit: [ja/nein]"
- Could only answer one
- Signals were ambiguous
Solution:
- Use question.id as LLM key (unique by design)
- Keep type for normalization logic
- Map id → type internally
Backend question_augmenter.py:
- format_question_list() now uses q.id as key
- Format: "- **q21**: [ja/nein] # Question text"
- Question text as comment for LLM context
Backend workflow_executor.py:
- Removed type→id mapping (no longer needed)
- decision_signals now keyed by id (from LLM)
- Build id→type catalog for normalization
- NormalizedSignal.question_type stores id (not type!)
- End Node template: signal_{id} directly available
Flow:
1. Questions sent to LLM: "- q21: [ja/nein] # Ist Protein unsicher?"
2. LLM answers: "- q21: nein"
3. Normalization: id→type lookup for spectrum/rules
4. Template: {{ node_4.signal_q21 }} = "nein"
Example (TWO unsicherheit questions):
Questions:
- q21: type=unsicherheit, question="Ist Protein unsicher?"
- q22: type=unsicherheit, question="Ist Energie unsicher?"
LLM Prompt:
```
## Entscheidungsfragen
- **q21**: [ja/nein] # Ist Protein unsicher?
- **q22**: [ja/nein] # Ist Energie unsicher?
```
LLM Response:
```
- q21: nein
- q22: ja
```
Template:
```
{{ node_4.signal_q21 }} → "nein"
{{ node_4.signal_q22 }} → "ja"
```
BREAKING CHANGE:
- Old workflows with decision_signals keyed by type will break
- Need to re-execute workflows after update
Issue: Cannot have multiple questions with same type
Version: 0.9p (workflow module)
Part 3: End Node Template Engine - ARCHITECTURAL FIX
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root Cause:
- Previous index-based mapping assumed signals come in same order as questions
- But LLM response order can differ from question configuration order
- Led to signal values being assigned to wrong question IDs
Old Logic (BUGGY):
1. Build question_type → [list of IDs]
2. Track index per type
3. Get Nth ID from list
→ Assumes LLM answers in question definition order ❌
New Logic (CORRECT):
1. Build question_type → question_id (direct mapping)
2. For each signal: lookup type → get ID
→ Order-independent ✅
Backend workflow_executor.py:
- Removed index tracking (type_counts)
- Direct lookup: question_type_to_id[signal.question_type]
- Added ERROR log if duplicate question types found
- Added INFO log for each mapped signal (debugging)
Important:
- Each question MUST have a UNIQUE type
- If two questions share same type: ERROR logged
- System designed for unique types (LLM can't answer duplicates)
Example Debug Output:
```
Mapped signal: protein_ausreichend → signal_q21 = 'nein'
Mapped signal: kohlenhydrate_strategie → signal_q1775... = 'von Proteinen'
```
Issue: Signal values assigned to wrong question IDs
Version: 0.9p (workflow module)
Part 3: End Node Template Engine - Signal Mapping Fix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three major improvements for workflow templates:
1. **Normalized Signal Placeholders:**
- Signals now available as {{ node_4.signal_kalorienbilanz }}
- Uses normalized_value (not raw decision_signals)
- Enables structured decision-based outputs
2. **Question Text Placeholders:**
- Question texts available as {{ node_4.question_kalorienbilanz }}
- Extracted from workflow graph (question_augmentations)
- Allows displaying questions alongside answers
3. **Clean End Node Output:**
- End Node output no longer duplicated with "## node_4" headers
- aggregate_results() detects End Nodes via graph.nodes
- Only shows final template-rendered output
- Backward compatible: Falls back to combined_analysis if no End Node
Backend workflow_executor.py:
- execute_end_node(): Added normalized signals to template context
- execute_end_node(): Added question texts to template context
- execute_workflow(): Added graph to context for End Node access
- aggregate_results(): Signature change to accept graph
- aggregate_results(): Detects End Nodes and uses only their output
Frontend WorkflowResultViewer.jsx:
- Now uses aggregated.analysis_core (primary output)
- Removed fallback to combined_analysis (was showing duplicates)
Example Template:
```jinja2
**Frage:** {{ node_4.question_kalorienbilanz }}
**Antwort:** {{ node_4.signal_kalorienbilanz }}
---
{{ node_4.analysis_core }}
```
Issue: Signal placeholders empty, question texts unavailable, duplicate output
Version: 0.9p (workflow module)
Part 3: End Node Template Engine - Complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root Cause:
- Frontend serialized as "questions"
- Backend expected "question_augmentations"
- Analysis Nodes WITH questions configured sent empty array to backend
- Questions were never added to LLM prompt
Frontend workflowSerializer.js:
- Serialization: questions → question_augmentations (Backend field name)
- Deserialization: question_augmentations → questions (Frontend data object)
- Backward compatible: Falls back to "questions" for old workflows
Backend workflow_executor.py:
- Removed incorrect load_prompt_questions() function (was a misunderstanding)
- Back to original logic: Only use node.question_augmentations
- Simplified normalization logging
Impact:
- Analysis Node questions are now correctly sent to backend
- Questions augment the base prompt as intended
- LLM receives structured questions
- Decision signals are generated and accessible as placeholders
Example:
- Node configures question with id="q21"
- Signal becomes accessible as {{ node_2.signal_q21 }}
- Can be used in Logic Nodes and End Node templates
Issue: Workflow questions not sent to LLM (field name mismatch)
Version: 0.9p (workflow module)
Part 3: End Node Template Engine - Critical Fix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend workflow_executor.py:
- New function: load_prompt_questions() loads questions from base prompt
- execute_node() now implements Hybrid Model correctly:
* IF node has question_augmentations → use those (override)
* ELSE load questions from referenced base prompt (fallback)
- Normalization now uses `questions` variable (not node.question_augmentations)
- This fixes base prompts having questions that were ignored in workflows
Root Cause:
- Phase 1 Hybrid Model was incomplete
- Node-specific questions worked, but base prompt questions were ignored
- augment_prompt_with_questions() was only called when node.question_augmentations existed
Impact:
- Analysis Nodes WITHOUT custom questions now use base prompt questions
- LLM receives proper question augmentation
- Decision signals are generated and normalized correctly
Issue: Workflow questions not sent to LLM
Version: 0.9p (workflow module)
Part 3: End Node Template Engine - Critical Fix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend workflow_executor.py:
- load_prompt_template() now uses modern resolve_placeholders() from prompt_executor
- Calls get_placeholder_example_values() to populate ALL registered placeholders
- Passes catalog for |d modifier support
- Fixes issue where basis prompts had empty/null placeholder values in workflows
Backend placeholder_resolver.py:
- get_placeholder_catalog() now includes ALL placeholders from PLACEHOLDER_MAP
- Uncategorized placeholders added to "Sonstiges" category
- Fixes discrepancy: 111 total placeholders but only ~30 shown in picker
Root Cause:
- Workflow used old resolve_placeholders() (only PLACEHOLDER_MAP, no variables)
- Isolated execution used modern resolve_placeholders() (full variables dict)
- Catalog excluded non-registry placeholders from PLACEHOLDER_MAP
Impact:
- All placeholders now resolve correctly in workflow execution
- PlaceholderPicker shows all 111+ placeholders (not just registry ones)
Version: 0.9p (workflow module)
Part 3: End Node Template Engine - Bug Fixes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend:
- Moved /placeholders endpoint BEFORE /{prompt_id} catch-all
- Prevents "placeholders" being parsed as UUID parameter
- Fixes 500 Internal Server Error preventing placeholder loading
Frontend:
- PlaceholderPicker can now load ~120+ system placeholders
Root Cause:
- FastAPI matches routes in order
- Generic /{prompt_id} was catching /placeholders first
- psycopg2 error: invalid input syntax for type uuid: "placeholders"
Version: 0.9p (workflow module)
Part 3: End Node Template Engine
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Added new API endpoints for listing and updating widget-feature assignments, allowing for custom feature requirements.
- Introduced a new admin page for managing widget-feature assignments, enhancing the admin interface.
- Updated navigation to include a link to the new widget-feature assignments page.
- Refactored widget access logic to support AND-based feature requirements for widgets.
- Bumped app_dashboard version to 1.11.0 to reflect these changes and improvements.
- Added new API endpoints for managing the product dashboard standard, including retrieval, update, and deletion functionalities.
- Enhanced the DashboardConfigurePage to support admin mode for configuring the product dashboard standard.
- Updated the admin navigation to include a link for the product dashboard standard configuration.
- Refactored the dashboard layout logic to utilize the new product standard management features.
- Bumped app_dashboard version to 1.10.0 to reflect these enhancements and changes.
- Updated dashboard layout schema to introduce separate default layouts for product and lab dashboards.
- Added new functions for managing product and lab default layouts, improving user customization options.
- Updated app_dashboard version to 1.9.0 to reflect the introduction of product vs lab layout defaults and new API fields for dashboard configuration.
- Enhanced tests to validate new layout functionalities and ensure proper widget visibility based on user settings.
- Added new "Dashboard-Lab-Widgets" entry to the documentation for better guidance on widget configuration.
- Updated the app_dashboard version to 1.8.0 to reflect the introduction of widget catalog features and layout entitlements.
- Enhanced widget catalog entries to include optional feature requirements for better visibility and access control.
- Improved the DashboardLabPage to manage widget visibility based on feature entitlements, ensuring a more tailored user experience.
- Introduced "nutrition_detail_charts", "recovery_charts_panel", and "progress_photos" widgets to the dashboard.
- Updated widget configuration validation to support new widgets, including chart days for nutrition and recovery charts.
- Enhanced the widget catalog and dashboard layout to include the new features.
- Bumped app_dashboard version to 1.7.0 to reflect these additions and improvements.
- Added support for the "quick_capture" widget, allowing users to configure visibility for weight and baseline vitals (resting HR, HRV, VO₂max).
- Implemented validation logic to ensure correct configuration input and prevent errors.
- Updated the widget catalog and dashboard layout to reflect the new quick capture features.
- Removed the "training_type_distribution" widget from the catalog as part of the refactor.
- Bumped app_dashboard version to 1.6.2 to incorporate these enhancements.
- Updated the dashboard layout schema to include new widgets: DashboardGreeting, QuickWeightToday, BodyStatStrip, StatusPills, ProfileGoalsProgress, TrendKcalWeight, NutritionActivitySummary, RecoverySleepRest, and TrainingTypeDistribution.
- Improved widget configuration validation to support new features, including chart days for trend and distribution widgets.
- Refactored the default lab layout to align with the updated widget catalog and ensure proper default activation.
- Bumped app_dashboard version to 1.6.0 to reflect the addition of new widgets and configuration enhancements.
- Enhanced the KPI board widget to support tile configuration, allowing users to select and order tiles.
- Updated validation logic to ensure proper handling of tile IDs and configuration fields.
- Removed legacy chart_days support, transitioning to a fixed analysis window for KPI metrics.
- Improved the DashboardLabPage to integrate the new KpiBoardConfigEditor for managing tile selections.
- Bumped app_dashboard version to 1.5.0 to reflect these significant changes.
- Added support for the "kpi_board" widget in the dashboard configuration, allowing for chart_days validation.
- Updated the widget catalog description to reflect the new configuration options for KPI tiles.
- Enhanced the DashboardLabPage to manage chart_days input for the KPI board, improving user experience.
- Introduced normalization functions for KPI kcal window days to maintain consistent behavior.
- Bumped app_dashboard version to 1.4.0 to reflect these enhancements.
- Added functions to handle Apple Health sleep data in both segment and summary formats.
- Implemented robust error handling for date parsing and data conversion.
- Updated documentation to reflect new CSV format support and data aggregation logic.
- Bumped version in version.py to reflect the changes in the activity module.
- Added support for the "activity_overview" widget in the dashboard configuration, allowing for chart_days validation.
- Refactored validation logic to streamline error handling for both "body_overview" and "activity_overview" widgets.
- Updated the widget catalog description to reflect the new configuration options.
- Enhanced the DashboardLabPage to manage chart_days input for both widgets, improving user experience.
- Bumped app_dashboard version to 1.3.0 to reflect these enhancements.
- Added validation for widget configuration in the DashboardWidgetEntry model to ensure proper data structure.
- Updated the DashboardLayoutPayload to include widget configuration in the serialized output.
- Improved the PilotBodySection and DashboardLabPage components to support dynamic chart days configuration for the body overview widget.
- Refactored layout editor functions to normalize widget configurations for better handling.
- Bumped app_dashboard version to 1.2.0 to reflect the new features and improvements.
- Integrated a new API endpoint for fetching the widget catalog in the Dashboard-Lab.
- Updated the dashboard layout schema to utilize the widget catalog for dynamic widget management.
- Refactored DashboardLabPage and PilotVizPage to leverage the new widget rendering system.
- Removed deprecated widget metadata from the frontend, streamlining the widget management process.
- Bumped app_dashboard version to 1.1.0 to reflect the new features and improvements.
- Added new routes and API endpoints for the Dashboard-Lab layout in the app.
- Updated main.py to include the app_dashboard router for backend integration.
- Enhanced App.jsx to include a route for the DashboardLabPage.
- Modified SettingsPage to add a link to the new Dashboard-Lab layout, improving user access to dashboard features.
- Updated version.py to reflect the new app_dashboard module version.
- Bumped version of reference_values module to 1.3.0.
- Added new imports and functionality for reference values in the backend, enhancing data retrieval.
- Introduced a new PilotVizPage in the frontend for visualizing data, linked from the SettingsPage for easy access.
- Updated routing in App.jsx to include the new pilot visualization route.
- Introduced a new API endpoint for fetching a summary of profile reference values, providing the latest and previous entries for each reference type.
- Updated ProfileReferenceValuesPage to display summary tiles with trend indicators for better user insights.
- Enhanced CSS for responsive layout of reference value tiles, improving the overall user experience on different screen sizes.
- Implemented trend calculation logic to visually represent changes between the latest and previous reference values.
- Added a new API endpoint for reordering reference value types based on user-defined order.
- Updated the AdminReferenceValueTypesPage to allow users to reorder types using up/down buttons.
- Introduced a consistent confidence level sorting mechanism across the application.
- Refactored related components to remove unused sort order fields and improve user experience.
- Updated the backend to include new fields for validation rules and metadata in reference value types.
- Enhanced the AdminReferenceValueTypesPage to support new validation rules for different data types.
- Improved the ProfileReferenceValuesPage to handle validation and metadata for profile reference values.
- Added API endpoint for fetching reference value metadata enums to support frontend validation.
- Refactored frontend forms to incorporate new fields and validation logic for a better user experience.
