diff --git a/reflexio/server/README.md b/reflexio/server/README.md index b4b8d957..875deac2 100644 --- a/reflexio/server/README.md +++ b/reflexio/server/README.md @@ -462,11 +462,12 @@ Key files: - `shadow_comparison/judge.py`: Per-turn regular-vs-shadow judge - `shadow_comparison/dispatcher.py` and `shadow_comparison/worker.py`: Publish-time dispatch and bounded background execution for shadow verdict writes - `shadow_comparison/outcome.py`: Verdict outcome model helpers -- `evaluation_overview/service.py`: Aggregates evaluation-page metrics +- `evaluation_overview/service.py`: Bulk-loads evaluation-page metrics, first-request sources, citations, Braintrust scores, and optional shadow verdicts; source-set cohorts are computed server-side so the dashboard does not re-aggregate. - `evaluation_overview/components/hero_state.py`, `evaluation_overview/components/distribution.py`, `evaluation_overview/components/rule_attribution.py`, `evaluation_overview/components/shadow_aggregation.py`: Focused aggregation helpers - `evaluation_overview/eval_sampler.py`: Evaluation sampling helpers that remain root-level +- `models/api_schema/eval_overview_schema.py`: `GetEvaluationOverviewRequest/Response`, including optional `source_sets` input and `source_set_comparison` output keyed by `(user_id, session_id)`. -**Pattern**: Session-level agent success evaluation remains in `agent_success_evaluation/`; dashboard-facing rollups and per-turn shadow verdict analysis live in these companion directories. +**Pattern**: Session-level agent success evaluation remains in `agent_success_evaluation/`; dashboard-facing rollups, first-source cohort comparisons, and per-turn shadow verdict analysis live in these companion directories. ### Playbook Optimizer and Braintrust diff --git a/reflexio/server/services/evaluation_overview/README.md b/reflexio/server/services/evaluation_overview/README.md index 264da11e..1e3eebc7 100644 --- a/reflexio/server/services/evaluation_overview/README.md +++ b/reflexio/server/services/evaluation_overview/README.md @@ -2,14 +2,18 @@ Read-side aggregation module for `POST /api/get_evaluation_overview`. -- `service.py` is the request-path entry point. It loads evaluation, citation, Braintrust, and optional shadow verdict data, then composes `GetEvaluationOverviewResponse`. +- `service.py` is the request-path entry point. It bulk-loads evaluation results (without embeddings), first-request sources, citations, Braintrust scores, and optional shadow verdicts, then composes `GetEvaluationOverviewResponse` for the dashboard. - `components/` contains pure read-side aggregation helpers used by the service and focused tests. - `eval_sampler.py` stays at the package root because regenerate jobs also use it to sample evaluation sessions. +- `reflexio/models/api_schema/eval_overview_schema.py` defines the public request/response contract, including optional `source_sets` cohorts and `source_set_comparison` output. The overview reports task success across every evaluated session and a separate behavior-success metric that excludes `failure_type=system_error` rows from both numerator and denominator. A window with no behavior-evaluable rows returns a null behavior rate plus eligible/excluded counts for honest UI -rendering. +rendering. Source-set comparison groups sessions by the first request source, +returns collision-safe `(user_id, session_id)` identities, and rejects duplicate +labels or overlapping source values so the frontend can compare cohorts without +client-side re-aggregation. This module mutates no core state. Keep response-shape changes in API schema tests and service integration tests.