Adjust correctly the level of metrics calculated during cross-validation

[andyreus17]

Summary

The base branch already carved a partition out of cross-validation, but it only fed explainability: it produced no metric at all. At the same time the folds still stored their results under the test name, even though a fold score is an estimate obtained by resampling.

That combination had a concrete consequence. The model comparison table showed a column labelled "Test" that actually held averaged validation scores. Since that column is what the user reads to pick a model, the value stops being an unbiased estimate of generalization the moment it is used to select. So, now cross validation has validation metrics (obtained from the multiple folds evaluation) and train metrics, obtained from the independent set, thereby aligning itself with the standard of literature (validation metrics for model selection and test metrics for model assessment).

In short, the following changes were made:

• Folds now report their metrics as validation, which is what they are.
• The reserved rows are now scored once by the final model and reported as test, the only estimate that no fold and no hyperparameter search ever influenced.
• With the test name free, the EVALUATION_PARTITION indirection that existed to avoid the name collision is removed.
• The reserved proportion is renamed from holdout to test_size, matching what it does and no longer clashing with HoldoutSplitter / HoldoutEvaluationStrategy.

Behaviour with test_size = 0 is unchanged from the base branch (apply cross validation over all data).

EXTRA (fixed existing bug): Disabling hyperparameter optimization on a run left its old optimizer, optimizer parameters, goal metric and nested-CV config still attached and showing in the "Model" sidebar, since nothing cleared that state when the optimize flags were unchecked. The edit dialog now clears all four when no parameter is marked for optimization.

---

Type of Change

Check all that apply like this [x]:

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Backend

• DashAI/back/evaluation/cv.py: Folds write SplitEnum.VALIDATION instead of TEST at FOLD level; the HPO objective in evaluate() and its TRIAL metrics move to validation, as does the outer loop of _nested_cv() at OUTER_FOLD level. A new STEP 5 scores the reserved rows once with the final model and stores the result as TEST at LAST level, without a standard deviation.

• DashAI/back/splitters/fold_splitter.py:

  • The fold loop produces validation instead of testindexes and asks for a "validation" partition when building the DatasetDicts. The full_dataset entry keeps {train, test} with the reserved rows. explainable_partitions() returns {"train", "test"} instead of {"train", "holdout"}, and EVALUATION_PARTITION is removed.
  • Some names were changed for the sake of consistency: holdout -> test_size, HOLDOUT_STRATEGY -> TEST_SPLIT_STRATEGY, _carve_holdout() -> _carve_test_split()
  • Use of select() method to filter dataset rows is changed for split_dataset_cv() due to a slow performance of the first method. That way, split() method of fold based splitters went from ~35s to ~0.06s (using the student dataset).

• DashAI/back/splitters/base_splitter.py: The EVALUATION_PARTITION attribute is removed. The gate in explainable_splits() checks the test partition directly.

• DashAI/back/splitters/k_fold.py, stratified_k_fold.py, group_k_fold.py, stratified_group_k_fold.py, repeated_k_fold.py, repeated_stratified_k_fold.py, leave_one_out.py:

  • Schema field holdout -> test_size.
  • The alias changes from "Held out for explanations" to "Test set", and the description is rewritten in all five languages: "it states that no fold and no hyperparameter search sees those rows, that they serve both as the test metric and as explainable data, that setting it to 0 leaves the run with neither, and that fold metrics are validation estimates which may carry an optimistic bias if used as the final evaluation".

• DashAI/back/splitters/splits_payload.py: explainable_indexes() resolves the evaluation slot with partitions.get("test").

• DashAI/back/dataloaders/classes/dashai_dataset.py: split_dataset_cv() accepts second_split_name (default "test"), so folds return {train, validation} while the trailing element keeps {train, test}.

• DashAI/back/api/api_v1/endpoints/runs.py:

  • get_metrics_for_run() builds its response by iterating SplitEnum instead of listing keys by hand.
  • attach_metrics_to_run() replaces the two duplicated assignment blocks in get_runs and get_run_by_id.
  • fold-metrics goes from Literal["train","test"] defaulting to "test" to Literal["train","validation"] defaulting to "validation", since no test metric exists at fold level anymore.
  • reset_run() also clears the _std fields.

• DashAI/back/api/api_v1/endpoints/explainers.py: Docstring of the explainable-splits endpoint: a cross-validation run no longer exposes its reserved rows under the name holdout.

• DashAI/back/job/explainer_job.py: valid_splits becomes ["train", "val", "all", "test"].

• DashAI\back\api\api_v1\endpoints\runs.py: update_run keys the nested update off model_fields_set instead of the value, so an explicit null clears the nested CV config instead of being indistinguishable from an omitted field. The modification guard moved from any([...]) to model_fields_set as well, since truthiness read a cleared optimizer ("" / {}) as "nothing changed".

---

Frontend

• DashAI/front/src/utils/splitsPayload.js: hasPartition() asserted that cross-validation never builds a validation partition. Now In CV, train and validation now always receive rows, and test does only when test_size > 0.

• DashAI/front/src/components/models/AddModelDialog.jsx and RunEditForm.jsx: maxInnerFolds discounts the reserved proportion since nested CV inner folds are built over the pool, not over the whole dataset.

• DashAI/front/src/components/models/runResults/ResultsTabsHeader.jsx: Reads test_size instead of holdout when deciding whether the run has explainable data.

• DashAI/front/src/components/models/ModelComparisonTable.jsx: The +/- no longer depends on the evaluation strategy but on the value being present. The validation column shows it and the test column does not, with no conditional branch.

• DashAI/front/src/pages/results/components/ResultsTabMetricsToggle.jsx and ResultsTabMetrics.jsx: The toggle becomes data-driven through hasTrainData / hasValidationData / hasTestData instead of hiding validation whenever the strategy is CV. ResultsTabMetrics derives them from the run's metrics.

• DashAI/front/src/components/models/LiveMetricsChart.jsx: The validation tab was conditioned on the session not being CV. It is now always shown.

• DashAI/front/src/components/models/FoldMetricsChart.jsx: The button TEST becomes VALIDATION.

• DashAI/front/src/components/models/OuterFoldMetricsTable.jsx: Reads validation_metrics / validation_metrics_std, which is where nested CV now stores its outer fold results.

• DashAI/front/src/components/models/StatisticalTestsModal.jsx: Default split validation, reads run.validation_metrics, and the selector is reduced to train/validation: a paired test needs one sample per fold, and the reserved rows yield a single value.

• DashAI/front/src/utils/i18n/locales/{en,es,pt,de,zh}/common.json: The holdout key is removed because it was not being used.

• DashAI\front\src\hooks\useRunEditForm.js:

  • doSave() now derives the payload from hasOptimizableParams, sending empty values for the optimizer, its parameters and the goal metric, and null for the nested config, when no parameter is marked for optimization.
  • isDirty compared the inner-CV config against the full stored splitter config, two different shapes that never matched, leaving Save permanently enabled. It now compares against the normalized shape, and only while nested CV is on.

---

Tests

• tests/back/splitters/test_holdout_carve.py -> test_carve_test_split.py
  Renamed along with its functions and variables. Assertions on the folds move to validation_indexes and to the validation key of the DatasetDict, and the configuration uses test_size.

• tests/back/splitters/test_explainable_partitions.py
ProbeSplitter and the two tests that used it are removed: they existed to prove a splitter could freely name its unseen partition. Now CV asserts the reserved rows are offered as test; the refusal test is unified because both families now fail with the same message.

• tests/back/api/test_explainer_jobs.py
  The endpoint assertion goes from {"name": "holdout"} to {"name": "test"}, and the file's helpers drop the previous vocabulary.