[BUG] fit(train_metrics=/val_metrics=) is wired wrong: arguments are swapped and built-in metrics crash

[ChrisW09]

Describe the bug

Two independent defects in the same two call sites (lightning_module.py:420 and :476) make the
documented fit(..., val_metrics={...}) feature unusable and silently wrong:

1. Arguments are swapped. The loop calls metric_fn(preds, labels), but DeepTabMetric.__call__
   is defined and documented as (y_true, y_pred). Verified with a probe metric on an LSS model: the
   y_true parameter received shape (8, 2) — the two Normal distribution parameters — and y_pred
   received (8, 1), the actual targets. Exactly backwards. For asymmetric metrics the logged number
   is simply wrong: R2Score gives 0.7080 correct vs 0.5229 swapped; GammaDeviance -0.1631 vs
   0.3013.
2. Built-in metrics crash. Every metric in deeptab.metrics is numpy-based (np.asarray(...)),
   but the call sites pass live torch tensors. Verified: RuntimeError: Can't call numpy() on Tensor that requires grad for train_metrics even on CPU, and TypeError: can't convert mps:0 device type tensor to numpy for val_metrics on the default accelerator.

So the usage shown in deeptab/metrics/__init__.py, deeptab/metrics/base.py and
docs/api/metrics/index.rst — model.fit(..., val_metrics={"mae": MeanAbsoluteError()}) — cannot run
at all, and a user-supplied metric that does survive is computed backwards.

SklearnBaseLSS.evaluate() calls the same metric objects the correct way round, so val_crps logged
during fit and crps from evaluate() disagree for the same model and data.

---

train_metrics / val_metrics are called with (y_pred, y_true) — every logged metric is computed backwards

Where: deeptab/training/lightning_module.py (420, 476)

DeepTabMetric.call has the signature (y_true, y_pred) — documented in deeptab/metrics/base.py, in the regression.py module docstring ("metric(y_true, y_pred) -> float"), and honoured by evaluate()/score() — but TaskModel.training_step and validation_step call metric_fn(preds_transformed, labels), i.e. with the arguments reversed, so every train/val_ metric logged during fit() is an asymmetric metric evaluated with predictions in the y_true slot.

Observed: logged=-1019.367737 correct=-0.080763
logged=-1030.184082 correct=-0.080489
A companion spy that just records shapes/values confirms arg 1 is the prediction tensor and arg 2 is the label tensor. The two logged val_r2 values differ from the true R2 by four orders of magnitude and would drive early stopping / HPO / ReduceLROnPlateau on a monitored custom metric completely astray.

Expected: metric_fn(labels, preds_transformed) — the same (y_true, y_pred) order that evaluate() (regressor_base.py:303) and _score() (_mixins/predict.py:124) already use, so that val_r2 equals -0.0808 here.

Repro

# cd into your own scratch dir first (Lightning writes into cwd)
import warnings; warnings.simplefilter("ignore")
import numpy as np, pandas as pd, torch
from deeptab.models import MLPRegressor
from deeptab.configs import MLPConfig, TrainerConfig
from deeptab.metrics import R2Score, DeepTabMetric

rows = []
class Spy(DeepTabMetric):
    name = "spy"; higher_is_better = False
    def __call__(self, a, b):
        a = np.asarray(a.detach()) if torch.is_tensor(a) else np.asarray(a)
        b = np.asarray(b.detach()) if torch.is_tensor(b) else np.asarray(b)
        rows.append((R2Score()(a, b), R2Score()(b, a)))   # (as DeepTab calls it, correct order)
        return 0.0

rng = np.random.default_rng(0)
X = pd.DataFrame(rng.normal(size=(60, 4)), columns=list("abcd"))
y = X["a"].to_numpy() * 2 + rng.normal(0, .1, 60)
m = MLPRegressor(model_config=MLPConfig(layer_sizes=[8]),
                 trainer_config=TrainerConfig(max_epochs=1))
m.fit(X, y, batch_size=16, val_size=0.25, accelerator="cpu",
      val_metrics={"spy": Spy()}, enable_progress_bar=False,
      enable_model_summary=False, logger=False)
for logged, correct in rows:
    print("logged=%.6f  correct=%.6f" % (logged, correct))

---

Any built-in DeepTabMetric passed to fit(val_metrics=/train_metrics=) crashes: metrics get raw torch tensors, not numpy

Where: deeptab/training/lightning_module.py (420, 476)

The metric objects in deeptab.metrics are numpy-based (they all start with np.asarray(...)), but training_step/validation_step hand them the live torch tensors. On the default accelerator (mps here, cuda elsewhere) validation metrics raise TypeError, and train_metrics raise RuntimeError even on CPU because preds still requires grad — so the documented model.fit(..., val_metrics={"mae": MeanAbsoluteError()}) usage from deeptab/metrics/init.py and deeptab/metrics/base.py cannot run at all.

Observed: (a) TypeError can't convert mps:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
(b) RuntimeError Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
(val_metrics on an explicit accelerator="cpu" run is the only combination that survives, because validation runs under no_grad.)

Expected: Both fits complete and log val_mae / train_mae. The call sites should pass preds.detach().cpu() and labels.detach().cpu() (or the metrics should coerce tensors themselves) before invoking the numpy-based metric.

Repro

import warnings; warnings.simplefilter("ignore")
import numpy as np, pandas as pd
from deeptab.models import MLPRegressor
from deeptab.configs import MLPConfig, TrainerConfig
from deeptab.metrics import MeanAbsoluteError
rng = np.random.default_rng(0)
X = pd.DataFrame(rng.normal(size=(60, 4)), columns=list("abcd"))
y = X["a"].to_numpy() * 2 + rng.normal(0, .1, 60)
mk = lambda: MLPRegressor(model_config=MLPConfig(layer_sizes=[8]),
                          trainer_config=TrainerConfig(max_epochs=1))
common = dict(batch_size=16, val_size=0.25, enable_progress_bar=False,
              enable_model_summary=False, logger=False)

# (a) default accelerator (mps/cuda) + val_metrics
try:
    mk().fit(X, y, val_metrics={"mae": MeanAbsoluteError()}, **common)
    print("(a) OK")
except Exception as e:
    print("(a)", type(e).__name__, e)

# (b) explicit CPU + train_metrics
try:
    mk().fit(X, y, accelerator="cpu", train_metrics={"mae": MeanAbsoluteError()}, **common)
    print("(b) OK")
except Exception as e:
    print("(b)", type(e).__name__, e)

---

Expected behavior
metric_fn(labels, preds) matching the documented (y_true, y_pred) contract, with
.detach().cpu() applied before handing tensors to the numpy-based metrics.

Screenshots
n/a

Desktop (please complete the following information):

• OS: macOS (Darwin 25.5.0, arm64)
• Python version: 3.11.15
• deeptab Version: 2.0.0 (main @ 4e6a359)

Additional context
torch 2.9.1, lightning 2.6.5, scikit-learn 1.9.0, numpy 2.4.6. Found in a second-pass review of v2.0.0
(seven independent lenses, each finding adversarially re-verified by a second reviewer, then re-run by
hand). Distinct from the already-filed #409-#426.