Multi-measure Perf support, BiB-only direction, and multiple-comparison correction

CLAUDE.md

@@ -90,11 +90,14 @@ statistic → derived comparisons/plots.
   without recomputing bootstrap samples).
 
 - **`interface.py` — `Perf` and `Difference`**: the main user-facing entry point (re-exported at
-  package root). `Perf(y_true, *y_pred, name=..., score_func=..., error_func=..., **kwargs)` wraps
-  one or more systems' predictions against shared ground truth. Exactly one of `score_func` /
-  `error_func` must be set (asserted via XOR) — `score_func` implies bigger-is-better (`BiB=True`),
-  `error_func` implies smaller-is-better (`BiB=False`). Internally holds a `StatisticSamples` keyed
-  by system name; new predictions can be added later via `perf(y_pred, name=...)` (`__call__`).
+  package root). `Perf(y_true, *y_pred, name=..., func=..., BiB=True, **kwargs)` wraps one or more
+  systems' predictions against shared ground truth. `func` is a single callable or a list of
+  callables (a multi-measure `Perf`); direction (bigger-is-better vs smaller-is-better) comes from
+  each callable's own `.BiB` attribute when it has one (e.g. set by a `metrics.py` wrapper's
+  `.measure` factory), falling back to the constructor's `BiB` default otherwise — `BiB` is the
+  single source of truth for direction, there is no separate `score_func`/`error_func` split.
+  Internally holds a `StatisticSamples` keyed by system name; new predictions can be added later via
+  `perf(y_pred, name=...)` (`__call__`).
   `Perf.difference(wrt=...)` produces a `Difference` instance (comparing every system against the
   best, or an explicit reference) whose `p_value()` is computed directly from the bootstrap
   distribution of paired differences — no parametric test assumptions. `Perf.plot()` /
@@ -104,10 +107,11 @@ statistic → derived comparisons/plots.
 - **`metrics.py`**: thin wrappers around `sklearn.metrics` functions (`accuracy_score`,
   `balanced_accuracy_score`, `top_k_accuracy_score`, `f1_score`, etc.). Each wrapper closes over the
   sklearn metric (plus its metric-specific kwargs like `average`, `normalize`) and constructs a
-  `Perf` with that as `score_func`/`error_func`. The `@metrics_docs` decorator (from `utils.py`)
+  `Perf` with that as `func`, tagging the inner closure's `.BiB` (`True` for a score, `False` for an
+  error) so direction travels with the callable. The `@metrics_docs` decorator (from `utils.py`)
   injects the shared `Perf`-style docstring (params like `num_samples`, `n_jobs`, `use_tqdm`) into
   each wrapper automatically — when adding a new metric wrapper, follow this same
-  `@metrics_docs(hy_name=..., attr_name=...)` + inner-function-closure pattern rather than duplicating
+  `@metrics_docs(hy_name=..., bib=...)` + inner-function-closure pattern rather than duplicating
   docstrings.
 
 - **`measurements.py`**: stateless helpers — `CI` (percentile bootstrap confidence interval), `SE`
@@ -131,9 +135,10 @@ statistic → derived comparisons/plots.
 - Bootstrap resampling must stay *paired* across systems being compared — `StatisticSamples.samples`
   caches resample indices by population size `N` precisely so every system's bootstrap replicate `i`
   uses the same resampled indices. Don't introduce per-system independent resampling.
-- `BiB` (Bigger is Better) must be threaded consistently: `score_func` → `BiB=True`, `error_func` →
-  `BiB=False`. Sorting, `best`, and p-value sign logic throughout `interface.py`/`performance.py`
-  depend on this flag rather than re-deriving it from the function.
+- `BiB` (Bigger is Better) must be threaded consistently: a callable's own `.BiB` attribute wins when
+  present, otherwise `Perf`'s `BiB` constructor argument is the default. Sorting, `best`, and p-value
+  sign logic throughout `interface.py`/`performance.py` read this flag (`self._bib`/
+  `statistic_samples.BiB`) rather than re-deriving it from which argument a function was passed as.
 - `sklearn.base.clone` / `__sklearn_clone__` is used to duplicate `Perf`/`StatisticSamples` instances
   while reusing already-computed bootstrap samples (e.g. `Perf.difference()`, `performance.difference`).
   Don't replace these with plain re-instantiation, as that silently redraws new bootstrap samples and

CompStats/__init__.py

@@ -17,4 +17,4 @@
 from CompStats.performance import performance, difference, all_differences, plot_performance, plot_difference
 from CompStats.performance import performance_multiple_metrics, difference_multiple, plot_performance_multiple, plot_difference_multiple
 from CompStats.performance import all_differences_multiple, plot_performance2, plot_difference2, plot_scatter_matrix
-from CompStats.interface import Perf
+from CompStats.interface import Perf

CompStats/bootstrap.py

@@ -18,7 +18,7 @@
 
 
 class StatisticSamples:
-    """Apply the statistic to `num_samples` samples taken with replacement 
+    """Apply the statistic to `num_samples` samples taken with replacement
     from the population (arguments).
 
     :param statistic: Statistic.
@@ -46,10 +46,10 @@ class StatisticSamples:
     """
 
     def __init__(self,
-                 statistic: Callable[[np.ndarray], float]=np.mean,
-                 num_samples: int=500,
-                 n_jobs: int=1,
-                 BiB: bool=True):
+                 statistic: Callable[[np.ndarray], float] = np.mean,
+                 num_samples: int = 500,
+                 n_jobs: int = 1,
+                 BiB: bool = True):
         self.statistic = statistic
         self.num_samples = num_samples
         self.n_jobs = n_jobs
@@ -62,7 +62,7 @@ def __init__(self,
     def info(self):
         """Information about the samples"""
         return self._info
-    
+
     @info.setter
     def info(self, value):
         self._info = value
@@ -85,7 +85,7 @@ def __sklearn_clone__(self):
     def calls(self):
         """Dictionary containing the output of the calls when a name is given"""
         return self._calls
-    
+
     @calls.setter
     def calls(self, value):
         self._calls = value
@@ -129,7 +129,7 @@ def statistic_samples(self, value):
 
     def samples(self, N):
         """Samples.
-        
+
         :param N: Population size.
         :type N: int
         """
@@ -144,7 +144,7 @@ def inner(N):
                 return inner(N)
         except AttributeError:
             return inner(N)
-        
+
     def keys(self):
         """calls keys"""
         return self.calls.keys()
@@ -153,7 +153,7 @@ def __getitem__(self, key):
         return self.calls[key]
 
     def __call__(self, *args: np.ndarray, name=None) -> np.ndarray:
-        """Population where the bootstrap process will be performed. 
+        """Population where the bootstrap process will be performed.
 
         :param *args: Population
         :type *args: np.ndarray
@@ -178,16 +178,15 @@ def melt(self, var_name='Algorithm', value_name='Score'):
                                              value_name=value_name)
 
 
-
 # class CI(StatisticSamples):
 #     """Compute the Confidence Interval of a statistic using bootstrap.
-    
-#     :param alpha: :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`. 
+
+#     :param alpha: :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`.
 #     :type alpha: float
 
 #     >>> from IngeoML import CI
 #     >>> from sklearn.metrics import accuracy_score
-#     >>> import numpy as np    
+#     >>> import numpy as np
 #     >>> labels = np.r_[[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]]
 #     >>> pred   = np.r_[[0, 0, 1, 0, 0, 1, 1, 1, 0, 1]]
 #     >>> acc = CI(statistic=accuracy_score)
@@ -204,24 +203,24 @@ def melt(self, var_name='Algorithm', value_name='Score'):
 #         """The interval is computed for :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`.
 #         """
 #         return self._alpha
-    
+
 #     @alpha.setter
 #     def alpha(self, value):
 #         self._alpha = value / 2
 
 #     def __call__(self, *args: np.ndarray) -> np.ndarray:
 #         B =  super().__call__(*args)
-#         alpha  = self.alpha  
-#         return (np.percentile(B, alpha * 100, axis=0), 
+#         alpha  = self.alpha
+#         return (np.percentile(B, alpha * 100, axis=0),
 #                 np.percentile(B, (1 - alpha) * 100, axis=0))
-    
+
 
 # class SE(StatisticSamples):
 #     """Compute the Standard Error of a statistic using bootstrap.
 
 #     >>> from IngeoML import SE
 #     >>> from sklearn.metrics import accuracy_score
-#     >>> import numpy as np    
+#     >>> import numpy as np
 #     >>> labels = np.r_[[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]]
 #     >>> pred   = np.r_[[0, 0, 1, 0, 0, 1, 1, 1, 0, 1]]
 #     >>> se = SE(statistic=accuracy_score)
@@ -235,8 +234,8 @@ def melt(self, var_name='Algorithm', value_name='Score'):
 
 
 # class Difference(CI):
-#     def __init__(self, y: np.ndarray, 
-#                  algorithms: dict={}, 
+#     def __init__(self, y: np.ndarray,
+#                  algorithms: dict={},
 #                  performance: Callable[[np.ndarray, np.ndarray], float]=lambda y, hy: f1_score(y, hy, average='macro'),
 #                  **kwargs) -> None:
 #         super(Difference, self).__init__(populations=algorithms, statistic=performance)
@@ -249,7 +248,7 @@ def melt(self, var_name='Algorithm', value_name='Score'):
 #     @property
 #     def y(self):
 #         return self._y
-    
+
 #     @y.setter
 #     def y(self, value):
 #         self._y = value
@@ -278,7 +277,7 @@ def melt(self, var_name='Algorithm', value_name='Score'):
 #         delta = perf(y, algs[self.best]) - perf(y, algs[key])
 #         self._delta[key] = delta
 #         return delta
-    
+
 #     def samples(self, key):
 #         if key in self.statistic_samples:
 #             return self.statistic_samples[key]
@@ -287,12 +286,12 @@ def melt(self, var_name='Algorithm', value_name='Score'):
 #         output = np.array([self.statistic(y[s], data[s])
 #                            for s in self.bootstrap])
 #         self.statistic_samples[key] = output
-#         return output    
-    
+#         return output
+
 #     @property
 #     def best_performance(self):
 #         return self.samples(self.best)
-        
+
 #     def distribution(self, key):
 #         best = self.best
 #         assert key != best
@@ -322,11 +321,10 @@ def melt(self, var_name='Algorithm', value_name='Score'):
 #         else:
 #             self._pvalue_l[key] = c
 #         return c
-    
+
 #     def sort(self, side='right'):
 #         best = self.best
 #         algs = [(k, self.pvalue(k, side=side))
 #                 for k in self.populations if k != best]
 #         algs.sort(key=lambda x: x[1], reverse=True)
 #         return [k for k, _ in algs]
-