diff --git a/CLAUDE.md b/CLAUDE.md index d4dcbca..eb5b063 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,11 +90,14 @@ statistic → derived comparisons/plots. without recomputing bootstrap samples). - **`interface.py` — `Perf` and `Difference`**: the main user-facing entry point (re-exported at - package root). `Perf(y_true, *y_pred, name=..., score_func=..., error_func=..., **kwargs)` wraps - one or more systems' predictions against shared ground truth. Exactly one of `score_func` / - `error_func` must be set (asserted via XOR) — `score_func` implies bigger-is-better (`BiB=True`), - `error_func` implies smaller-is-better (`BiB=False`). Internally holds a `StatisticSamples` keyed - by system name; new predictions can be added later via `perf(y_pred, name=...)` (`__call__`). + package root). `Perf(y_true, *y_pred, name=..., func=..., BiB=True, **kwargs)` wraps one or more + systems' predictions against shared ground truth. `func` is a single callable or a list of + callables (a multi-measure `Perf`); direction (bigger-is-better vs smaller-is-better) comes from + each callable's own `.BiB` attribute when it has one (e.g. set by a `metrics.py` wrapper's + `.measure` factory), falling back to the constructor's `BiB` default otherwise — `BiB` is the + single source of truth for direction, there is no separate `score_func`/`error_func` split. + Internally holds a `StatisticSamples` keyed by system name; new predictions can be added later via + `perf(y_pred, name=...)` (`__call__`). `Perf.difference(wrt=...)` produces a `Difference` instance (comparing every system against the best, or an explicit reference) whose `p_value()` is computed directly from the bootstrap distribution of paired differences — no parametric test assumptions. `Perf.plot()` / @@ -104,10 +107,11 @@ statistic → derived comparisons/plots. - **`metrics.py`**: thin wrappers around `sklearn.metrics` functions (`accuracy_score`, `balanced_accuracy_score`, `top_k_accuracy_score`, `f1_score`, etc.). Each wrapper closes over the sklearn metric (plus its metric-specific kwargs like `average`, `normalize`) and constructs a - `Perf` with that as `score_func`/`error_func`. The `@metrics_docs` decorator (from `utils.py`) + `Perf` with that as `func`, tagging the inner closure's `.BiB` (`True` for a score, `False` for an + error) so direction travels with the callable. The `@metrics_docs` decorator (from `utils.py`) injects the shared `Perf`-style docstring (params like `num_samples`, `n_jobs`, `use_tqdm`) into each wrapper automatically — when adding a new metric wrapper, follow this same - `@metrics_docs(hy_name=..., attr_name=...)` + inner-function-closure pattern rather than duplicating + `@metrics_docs(hy_name=..., bib=...)` + inner-function-closure pattern rather than duplicating docstrings. - **`measurements.py`**: stateless helpers — `CI` (percentile bootstrap confidence interval), `SE` @@ -131,9 +135,10 @@ statistic → derived comparisons/plots. - Bootstrap resampling must stay *paired* across systems being compared — `StatisticSamples.samples` caches resample indices by population size `N` precisely so every system's bootstrap replicate `i` uses the same resampled indices. Don't introduce per-system independent resampling. -- `BiB` (Bigger is Better) must be threaded consistently: `score_func` → `BiB=True`, `error_func` → - `BiB=False`. Sorting, `best`, and p-value sign logic throughout `interface.py`/`performance.py` - depend on this flag rather than re-deriving it from the function. +- `BiB` (Bigger is Better) must be threaded consistently: a callable's own `.BiB` attribute wins when + present, otherwise `Perf`'s `BiB` constructor argument is the default. Sorting, `best`, and p-value + sign logic throughout `interface.py`/`performance.py` read this flag (`self._bib`/ + `statistic_samples.BiB`) rather than re-deriving it from which argument a function was passed as. - `sklearn.base.clone` / `__sklearn_clone__` is used to duplicate `Perf`/`StatisticSamples` instances while reusing already-computed bootstrap samples (e.g. `Perf.difference()`, `performance.difference`). Don't replace these with plain re-instantiation, as that silently redraws new bootstrap samples and diff --git a/CompStats/__init__.py b/CompStats/__init__.py index 976141f..6454a22 100644 --- a/CompStats/__init__.py +++ b/CompStats/__init__.py @@ -17,4 +17,4 @@ from CompStats.performance import performance, difference, all_differences, plot_performance, plot_difference from CompStats.performance import performance_multiple_metrics, difference_multiple, plot_performance_multiple, plot_difference_multiple from CompStats.performance import all_differences_multiple, plot_performance2, plot_difference2, plot_scatter_matrix -from CompStats.interface import Perf \ No newline at end of file +from CompStats.interface import Perf diff --git a/CompStats/bootstrap.py b/CompStats/bootstrap.py index beb2ec3..5b9ff5f 100644 --- a/CompStats/bootstrap.py +++ b/CompStats/bootstrap.py @@ -18,7 +18,7 @@ class StatisticSamples: - """Apply the statistic to `num_samples` samples taken with replacement + """Apply the statistic to `num_samples` samples taken with replacement from the population (arguments). :param statistic: Statistic. @@ -46,10 +46,10 @@ class StatisticSamples: """ def __init__(self, - statistic: Callable[[np.ndarray], float]=np.mean, - num_samples: int=500, - n_jobs: int=1, - BiB: bool=True): + statistic: Callable[[np.ndarray], float] = np.mean, + num_samples: int = 500, + n_jobs: int = 1, + BiB: bool = True): self.statistic = statistic self.num_samples = num_samples self.n_jobs = n_jobs @@ -62,7 +62,7 @@ def __init__(self, def info(self): """Information about the samples""" return self._info - + @info.setter def info(self, value): self._info = value @@ -85,7 +85,7 @@ def __sklearn_clone__(self): def calls(self): """Dictionary containing the output of the calls when a name is given""" return self._calls - + @calls.setter def calls(self, value): self._calls = value @@ -129,7 +129,7 @@ def statistic_samples(self, value): def samples(self, N): """Samples. - + :param N: Population size. :type N: int """ @@ -144,7 +144,7 @@ def inner(N): return inner(N) except AttributeError: return inner(N) - + def keys(self): """calls keys""" return self.calls.keys() @@ -153,7 +153,7 @@ def __getitem__(self, key): return self.calls[key] def __call__(self, *args: np.ndarray, name=None) -> np.ndarray: - """Population where the bootstrap process will be performed. + """Population where the bootstrap process will be performed. :param *args: Population :type *args: np.ndarray @@ -178,16 +178,15 @@ def melt(self, var_name='Algorithm', value_name='Score'): value_name=value_name) - # class CI(StatisticSamples): # """Compute the Confidence Interval of a statistic using bootstrap. - -# :param alpha: :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`. + +# :param alpha: :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`. # :type alpha: float # >>> from IngeoML import CI # >>> from sklearn.metrics import accuracy_score -# >>> import numpy as np +# >>> import numpy as np # >>> labels = np.r_[[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]] # >>> pred = np.r_[[0, 0, 1, 0, 0, 1, 1, 1, 0, 1]] # >>> acc = CI(statistic=accuracy_score) @@ -204,24 +203,24 @@ def melt(self, var_name='Algorithm', value_name='Score'): # """The interval is computed for :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`. # """ # return self._alpha - + # @alpha.setter # def alpha(self, value): # self._alpha = value / 2 # def __call__(self, *args: np.ndarray) -> np.ndarray: # B = super().__call__(*args) -# alpha = self.alpha -# return (np.percentile(B, alpha * 100, axis=0), +# alpha = self.alpha +# return (np.percentile(B, alpha * 100, axis=0), # np.percentile(B, (1 - alpha) * 100, axis=0)) - + # class SE(StatisticSamples): # """Compute the Standard Error of a statistic using bootstrap. # >>> from IngeoML import SE # >>> from sklearn.metrics import accuracy_score -# >>> import numpy as np +# >>> import numpy as np # >>> labels = np.r_[[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]] # >>> pred = np.r_[[0, 0, 1, 0, 0, 1, 1, 1, 0, 1]] # >>> se = SE(statistic=accuracy_score) @@ -235,8 +234,8 @@ def melt(self, var_name='Algorithm', value_name='Score'): # class Difference(CI): -# def __init__(self, y: np.ndarray, -# algorithms: dict={}, +# def __init__(self, y: np.ndarray, +# algorithms: dict={}, # performance: Callable[[np.ndarray, np.ndarray], float]=lambda y, hy: f1_score(y, hy, average='macro'), # **kwargs) -> None: # super(Difference, self).__init__(populations=algorithms, statistic=performance) @@ -249,7 +248,7 @@ def melt(self, var_name='Algorithm', value_name='Score'): # @property # def y(self): # return self._y - + # @y.setter # def y(self, value): # self._y = value @@ -278,7 +277,7 @@ def melt(self, var_name='Algorithm', value_name='Score'): # delta = perf(y, algs[self.best]) - perf(y, algs[key]) # self._delta[key] = delta # return delta - + # def samples(self, key): # if key in self.statistic_samples: # return self.statistic_samples[key] @@ -287,12 +286,12 @@ def melt(self, var_name='Algorithm', value_name='Score'): # output = np.array([self.statistic(y[s], data[s]) # for s in self.bootstrap]) # self.statistic_samples[key] = output -# return output - +# return output + # @property # def best_performance(self): # return self.samples(self.best) - + # def distribution(self, key): # best = self.best # assert key != best @@ -322,11 +321,10 @@ def melt(self, var_name='Algorithm', value_name='Score'): # else: # self._pvalue_l[key] = c # return c - + # def sort(self, side='right'): # best = self.best # algs = [(k, self.pvalue(k, side=side)) # for k in self.populations if k != best] # algs.sort(key=lambda x: x[1], reverse=True) # return [k for k, _ in algs] - \ No newline at end of file diff --git a/CompStats/interface.py b/CompStats/interface.py index 6e28e63..fba6691 100644 --- a/CompStats/interface.py +++ b/CompStats/interface.py @@ -14,6 +14,7 @@ from dataclasses import dataclass from sklearn.metrics import balanced_accuracy_score from sklearn.base import clone +from statsmodels.stats.multitest import multipletests import pandas as pd import numpy as np from CompStats.bootstrap import StatisticSamples @@ -28,10 +29,10 @@ class Perf(object): :param y_true: True measurement or could be a pandas.DataFrame where column label 'y' corresponds to the true measurement. :type y_true: numpy.ndarray or pandas.DataFrame - :param score_func: Function (or list of functions) to measure the performance, it is assumed that the best algorithm has the highest value. :py:attr:`score_func` and :py:attr:`error_func` can be given simultaneously to combine score-type and error-type measures into a single, multi-measure :py:class:`Perf.` - :type score_func: Function, or list of functions, where the first argument is :math:`y` and the second is :math:`\\hat{y}.` - :param error_func: Function (or list of functions) to measure the performance where the best algorithm has the lowest value. - :type error_func: Function, or list of functions, where the first argument is :math:`y` and the second is :math:`\\hat{y}.` + :param func: Function (or list of functions) to measure the performance. Whether the best algorithm has the highest or the lowest value is given by :py:attr:`BiB` -- either the constructor's default, or, when a callable already carries its own :py:attr:`BiB` attribute (e.g. built by a :py:mod:`CompStats.metrics` wrapper's ``.measure`` factory), that tag takes precedence. A list of functions combines them into a single, multi-measure :py:class:`Perf.` + :type func: Function, or list of functions, where the first argument is :math:`y` and the second is :math:`\\hat{y}.` + :param BiB: Bigger is Better; the default direction used for any measure in :py:attr:`func` that doesn't already carry its own :py:attr:`BiB` attribute. A single bool applies to every measure; a list applies element-wise, one entry per measure in :py:attr:`func`. + :type BiB: bool or list of bool :param measure_names: Display name for each measure, only relevant when more than one measure is given; defaults to each function's ``__name__``. :type measure_names: list :param y_pred: Predictions, the algorithms will be identified with alg-k where k=1 is the first argument included in :py:attr:`args.` @@ -58,15 +59,16 @@ class Perf(object): >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) + >>> perf = Perf(y_val, hy, name='LinearSVC') >>> ens = RandomForestClassifier().fit(X_train, y_train) - >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) + >>> perf(ens.predict(X_val), name='forest') >>> perf Statistic with its standard error (se) statistic (se) - 0.9792 (0.0221) <= alg-1 + 0.9792 (0.0221) <= LinearSVC 0.9744 (0.0246) <= forest - + If an algorithm's prediction is missing, this can be included by calling the instance, as can be seen in the following instruction. Note that the algorithm's name can also be given with the keyword :py:attr:`name.` >>> lr = LogisticRegression().fit(X_train, y_train) @@ -77,11 +79,12 @@ class Perf(object): 1.0000 (0.0000) <= Log. Reg. 0.9792 (0.0221) <= alg-1 0.9744 (0.0246) <= forest - + The performance function used to compare the algorithms can be changed, and the same bootstrap samples would be used if the instance were cloned. Consequently, the values are computed using the same samples, as can be seen in the following example. >>> perf_error = clone(perf) - >>> perf_error.error_func = lambda y, hy: (y != hy).mean() + >>> perf_error.func = lambda y, hy: (y != hy).mean() + >>> perf_error.BiB = False >>> perf_error Statistic with its standard error (se) @@ -90,38 +93,62 @@ class Perf(object): 0.0222 (0.0237) <= alg-1 0.0222 (0.0215) <= forest + When several algorithms are compared, :py:meth:`difference`'s p-values can be + adjusted for multiple comparisons by passing a + :py:func:`statsmodels.stats.multitest.multipletests` method name (e.g. + ``'bonferroni'``, ``'holm'``, ``'fdr_bh'``) as :py:attr:`correction` -- + both to :py:meth:`Difference.p_value` directly and, so that plots reflect + the same adjusted significance, to :py:meth:`plot`/:py:meth:`dataframe`. + + >>> diff = perf.difference() + >>> diff.p_value() + {'alg-1': np.float64(0.3), 'forest': np.float64(0.2)} + >>> diff.p_value(correction='bonferroni') + {'alg-1': np.float64(0.6), 'forest': np.float64(0.4)} + >>> perf.plot(correction='bonferroni') + Two or more measures can be combined into a single :py:class:`Perf` instance (e.g. macro-F1 together with macro-recall) by passing a list of functions - to :py:attr:`score_func`/:py:attr:`error_func` -- see :py:mod:`CompStats.metrics`'s + to :py:attr:`func` -- see :py:mod:`CompStats.metrics`'s ``.measure`` factories (e.g. :py:func:`~CompStats.metrics.f1_score.measure`). Every measure is evaluated on the same bootstrap resamples, so comparisons across algorithms remain paired for each measure. >>> from CompStats.metrics import f1_score, recall_score >>> perf = Perf(y_val, hy, forest=ens.predict(X_val), - ... score_func=[f1_score.measure(average='macro'), - ... recall_score.measure(average='macro')]) + ... func=[f1_score.measure(average='macro'), + ... recall_score.measure(average='macro')]) + + With multiple measures, :py:attr:`correction` is applied independently + per measure (i.e. per column), so one metric's correction never mixes + with another's. + + >>> diff = perf.difference() + >>> diff.p_value() + {'alg-1': array([1. , 0.3]), 'forest': array([0.2, 1. ])} + >>> diff.p_value(correction='bonferroni') + {'alg-1': array([1. , 0.6]), 'forest': array([0.4, 1. ])} """ + def __init__(self, y_true, *y_pred, - name:str=None, - score_func=balanced_accuracy_score, - error_func=None, - measure_names:list=None, - num_samples: int=500, - n_jobs: int=-1, + name: str = None, + func=balanced_accuracy_score, + BiB: bool = True, + measure_names: list = None, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): - assert (len(self._as_list(score_func)) - + len(self._as_list(error_func))) >= 1 - self._score_func = score_func - self._error_func = error_func + assert len(self._as_list(func)) >= 1 + self._func = func + self._BiB = BiB self.measure_names = measure_names algs = {} if name is not None: if isinstance(name, str): name = [name] else: - name = [f'alg-{k+1}' for k, _ in enumerate(y_pred)] + name = [f'alg-{k + 1}' for k, _ in enumerate(y_pred)] for key, v in zip(name, y_pred): algs[key] = np.asanyarray(v) algs.update(**kwargs) @@ -135,7 +162,7 @@ def __init__(self, y_true, *y_pred, @staticmethod def _as_list(value): - """Normalize a score_func/error_func argument into a list of callables""" + """Normalize a func argument into a list of callables""" if value is None: return [] if isinstance(value, (list, tuple)): @@ -148,22 +175,29 @@ def _measures(self): Each callable's own :py:attr:`BiB` attribute (set by, e.g., a :py:meth:`metrics.py ` wrapper's ``.measure`` factory) - takes precedence over the default direction implied by which - argument (:py:attr:`score_func` or :py:attr:`error_func`) it came from. + takes precedence over :py:attr:`BiB`, the constructor's default direction. """ - def tagged(funcs, default_bib): - return [(f, bool(getattr(f, 'BiB', default_bib))) - for f in self._as_list(funcs)] - return tagged(self.score_func, True) + tagged(self.error_func, False) + funcs = self._as_list(self.func) + default = self.BiB + if isinstance(default, (list, tuple, np.ndarray)): + defaults = list(default) + else: + defaults = [default] * len(funcs) + return [(f, bool(getattr(f, 'BiB', d))) + for f, d in zip(funcs, defaults)] + + @property + def _bib(self): + """Scalar or per-measure array combining every measure's tagged BiB""" + measures = self._measures + if len(measures) == 1: + return measures[0][1] + return np.array([b for _, b in measures]) def _init(self): """Compute the bootstrap statistic""" - measures = self._measures - if len(measures) == 1: - bib = measures[0][1] - else: - bib = np.array([b for _, b in measures]) + bib = self._bib if hasattr(self, '_statistic_samples'): _ = self.statistic_samples _.BiB = bib @@ -179,8 +213,8 @@ def get_params(self): """Parameters""" return dict(y_true=self.y_true, - score_func=self.score_func, - error_func=self.error_func, + func=self.func, + BiB=self.BiB, measure_names=self._measure_names, num_samples=self.num_samples, n_jobs=self.n_jobs) @@ -196,23 +230,17 @@ def __sklearn_clone__(self): def __repr__(self): """Prediction statistics with standard error in parenthesis""" - if self.error_func is None: - arg = 'score_func' - elif self.score_func is None: - arg = 'error_func' - else: - arg = 'score_func/error_func' func_name = self.statistic_func.__name__ statistic = self.statistic if isinstance(statistic, dict): - return f"<{self.__class__.__name__}({arg}={func_name})>\n{self}" + return f"<{self.__class__.__name__}(func={func_name})>\n{self}" elif isinstance(statistic, float): - return f"<{self.__class__.__name__}({arg}={func_name}, statistic={statistic:0.4f}, se={self.se:0.4f})>" + return f"<{self.__class__.__name__}(func={func_name}, statistic={statistic:0.4f}, se={self.se:0.4f})>" desc = [f'{k:0.4f}' for k in statistic] desc = ', '.join(desc) desc_se = [f'{k:0.4f}' for k in self.se] desc_se = ', '.join(desc_se) - return f"<{self.__class__.__name__}({arg}={func_name}, statistic=[{desc}], se=[{desc_se}])>" + return f"<{self.__class__.__name__}(func={func_name}, statistic=[{desc}], se=[{desc_se}])>" def __str__(self): """Prediction statistics with standard error in parenthesis""" @@ -249,7 +277,7 @@ def __call__(self, y_pred, name=None): del calls[name] return self - def difference(self, wrt: str=None): + def difference(self, wrt: str = None): """Compute the difference w.r.t any algorithm by default is the best >>> from sklearn.svm import LinearSVC @@ -268,7 +296,7 @@ def difference(self, wrt: str=None): >>> perf.difference() difference p-values w.r.t alg-1 - forest 0.06 + forest 0.06 """ if wrt is None: wrt = self.best @@ -350,11 +378,12 @@ def statistic(self): >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> perf.statistic - {'alg-1': 1.0, 'forest': 0.9500891265597148} + {'alg-1': 1.0, 'forest': 0.9500891265597148} """ if hasattr(self, '_statistic') and self._statistic is not None: return self._statistic - BiB = True if self.score_func is not None else False + bib = self._bib + BiB = bool(np.all(bib)) if isinstance(bib, np.ndarray) else bib data = sorted([(k, self.statistic_func(self.y_true, v)) for k, v in self.predictions.items()], key=lambda x: self.sorting_func(x[1]), @@ -373,7 +402,7 @@ def statistic(self, value): @property def se(self): """Standard Error - + >>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris @@ -398,7 +427,7 @@ def se(self): @property def ci(self): """Confidence interval - + >>> from sklearn.svm import LinearSVC >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split @@ -419,20 +448,21 @@ def ci(self): return list(output.values())[0] return output - def plot(self, value_name:str=None, - var_name:str='Performance', - alg_legend:str='Algorithm', - perf_names:list=None, - CI:float=0.05, - kind:str='point', linestyle:str='none', - col_wrap:int=3, capsize:float=0.2, - comparison:bool=True, - right:bool=True, - comp_legend:str='Comparison', - winner_legend:str='Best', - tie_legend:str='Equivalent', - loser_legend:str='Different', - palette:object=None, + def plot(self, value_name: str = None, + var_name: str = 'Performance', + alg_legend: str = 'Algorithm', + perf_names: list = None, + CI: float = 0.05, + kind: str = 'point', linestyle: str = 'none', + col_wrap: int = 3, capsize: float = 0.2, + comparison: bool = True, + right: bool = True, + correction: str = None, + comp_legend: str = 'Comparison', + winner_legend: str = 'Best', + tie_legend: str = 'Equivalent', + loser_legend: str = 'Different', + palette: object = None, **kwargs): """plot with seaborn @@ -447,16 +477,17 @@ def plot(self, value_name:str=None, >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) - >>> perf = Perf(y_val, hy, score_func=None, - error_func=lambda y, hy: (y != hy).mean(), + >>> perf = Perf(y_val, hy, + func=lambda y, hy: (y != hy).mean(), BiB=False, forest=ens.predict(X_val)) >>> perf.plot() """ import seaborn as sns if value_name is None: - if len(self._measures) > 1: + measures = self._measures + if len(measures) > 1: value_name = 'Value' - elif self.score_func is not None: + elif measures[0][1]: value_name = 'Score' else: value_name = 'Error' @@ -469,7 +500,8 @@ def plot(self, value_name:str=None, df = self.dataframe(value_name=value_name, var_name=var_name, alg_legend=alg_legend, perf_names=perf_names, comparison=comparison, alpha=CI, right=right, - comp_legend=comp_legend, + correction=correction, + comp_legend=comp_legend, winner_legend=winner_legend, tie_legend=tie_legend, loser_legend=loser_legend) @@ -481,9 +513,9 @@ def plot(self, value_name:str=None, kwargs.update(dict(hue=comp_legend)) if palette is None: pal = sns.color_palette("Paired") - palette = {winner_legend:pal[1], - tie_legend:pal[3], - loser_legend: pal[5]} + palette = {winner_legend: pal[1], + tie_legend: pal[3], + loser_legend: pal[5]} f_grid = sns.catplot(df, x=value_name, errorbar=ci, y=alg_legend, col=var_name, kind=kind, linestyle=linestyle, @@ -492,19 +524,20 @@ def plot(self, value_name:str=None, **kwargs) return f_grid - def dataframe(self, comparison:bool=False, - right:bool=True, - alpha:float=0.05, - value_name:str='Score', - var_name:str='Performance', - alg_legend:str='Algorithm', - comp_legend:str='Comparison', - winner_legend:str='Best', - tie_legend:str='Equivalent', - loser_legend:str='Different', - perf_names:str=None): + def dataframe(self, comparison: bool = False, + right: bool = True, + alpha: float = 0.05, + correction: str = None, + value_name: str = 'Score', + var_name: str = 'Performance', + alg_legend: str = 'Algorithm', + comp_legend: str = 'Comparison', + winner_legend: str = 'Best', + tie_legend: str = 'Equivalent', + loser_legend: str = 'Different', + perf_names: str = None): """Dataframe - + >>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris @@ -535,13 +568,13 @@ def dataframe(self, comparison:bool=False, diff = self.difference() best = self.best if isinstance(best, str): - for name, p in diff.p_value(right=right).items(): + for name, p in diff.p_value(right=right, correction=correction).items(): if p >= alpha: continue df.loc[df[alg_legend] == name, comp_legend] = loser_legend df.loc[df[alg_legend] == best, comp_legend] = winner_legend else: - p_values = diff.p_value(right=right) + p_values = diff.p_value(right=right, correction=correction) systems = list(p_values.keys()) p_values = np.array([p_values[k] for k in systems]) for name, p_value, winner in zip(perf_names, @@ -571,11 +604,10 @@ def n_jobs(self, value): def statistic_func(self): """Statistic function - A single :py:attr:`score_func`/:py:attr:`error_func` callable is - returned as-is; when more than one measure is given (either as a - list, or by mixing :py:attr:`score_func` and :py:attr:`error_func`), - a composite callable is returned that concatenates every measure's - output into a single vector, evaluated on the same bootstrap samples. + A single :py:attr:`func` callable is returned as-is; when more than + one measure is given (a list passed to :py:attr:`func`), a composite + callable is returned that concatenates every measure's output into a + single vector, evaluated on the same bootstrap samples. """ measures = self._measures if len(measures) == 1: @@ -662,38 +694,33 @@ def y_true(self, value): self._y_true = np.asanyarray(value) @property - def score_func(self): - """Score function""" - return self._score_func - - @score_func.setter - def score_func(self, value): - self._score_func = value - if value is not None: - self.error_func = None - if hasattr(self, '_statistic_samples'): - self._statistic_samples.statistic = value - self._statistic_samples.BiB = True + def func(self): + """Function (or list of functions) used to measure the performance""" + return self._func + + @func.setter + def func(self, value): + self._func = value + if hasattr(self, '_statistic_samples'): + self._statistic_samples.statistic = self.statistic_func + self._statistic_samples.BiB = self._bib @property - def error_func(self): - """Error function""" - return self._error_func + def BiB(self): + """Bigger is Better; default direction for measures without their own :py:attr:`BiB` tag""" + return self._BiB - @error_func.setter - def error_func(self, value): - self._error_func = value - if value is not None: - self.score_func = None - if hasattr(self, '_statistic_samples'): - self._statistic_samples.statistic = value - self._statistic_samples.BiB = False + @BiB.setter + def BiB(self, value): + self._BiB = value + if hasattr(self, '_statistic_samples'): + self._statistic_samples.BiB = self._bib @dataclass class Difference: """Difference - + >>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris @@ -714,18 +741,18 @@ class Difference: 0.0780 <= forest """ - statistic_samples:StatisticSamples=None - statistic:dict=None - best:str=None + statistic_samples: StatisticSamples = None + statistic: dict = None + best: str = None @property def sorting_func(self): """Rank systems when multiple performances are used""" return self._sorting_func - + @sorting_func.setter def sorting_func(self, value): - self._sorting_func = value + self._sorting_func = value def __repr__(self): """p-value""" @@ -758,19 +785,22 @@ def _delta_best(self): return self.statistic[self.best] keys = np.unique(self.best) statistic = np.array([self.statistic[k] - for k in keys]) + for k in keys]) m = {v: k for k, v in enumerate(keys)} best = np.array([m[x] for x in self.best]) return statistic[best, np.arange(best.shape[0])] - def p_value(self, right:bool=True): + def p_value(self, right: bool = True, correction: str = None): """Compute p_value of the differences :param right: Estimate the p-value using :math:`\\text{sample} \\geq 2\\delta` - :type right: bool - + :type right: bool + :param correction: Method to adjust for multiple comparisons, passed to :py:func:`statsmodels.stats.multitest.multipletests` (e.g. ``'bonferroni'``, ``'holm'``, ``'fdr_bh'``); ``None`` (default) leaves the p-values uncorrected. With a single measure, the family of comparisons is every other system against :py:attr:`best`; with multiple measures, each measure is corrected as its own family (i.e. per column) so metrics do not contaminate each other's correction. + :type correction: str + >>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier + >>> from sklearn.naive_bayes import GaussianNB >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from sklearn.base import clone @@ -780,11 +810,16 @@ def p_value(self, right:bool=True): >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) + >>> perf = Perf(y_val, hy, name='LinearSVC') >>> ens = RandomForestClassifier().fit(X_train, y_train) - >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) + >>> perf(ens.predict(X_val), name='forest') + >>> nb = GaussianNB().fit(X_train, y_train) + >>> perf(nb.predict(X_val), name='bayes') >>> diff = perf.difference() >>> diff.p_value() - {'forest': np.float64(0.3)} + {'forest': np.float64(0.3), 'bayes': np.float64(0.2)} + >>> diff.p_value(correction='bonferroni') + {'forest': np.float64(0.6), 'bayes': np.float64(0.4)} """ values = [] BiB = self.statistic_samples.BiB @@ -803,15 +838,26 @@ def p_value(self, right:bool=True): else: values.append((k, (v <= 0).mean(axis=0))) values.sort(key=lambda x: self.sorting_func(x[1])) - return dict(values) - - def dataframe(self, value_name:str='Score', - var_name:str='Best', - alg_legend:str='Algorithm', - sig_legend:str='Significant', - perf_names:str=None, - right:bool=True, - alpha:float=0.05): + if correction is None: + return dict(values) + keys = [k for k, _ in values] + raw = np.array([v for _, v in values]) + if raw.ndim == 1: + corrected = multipletests(raw, method=correction)[1] + else: + corrected = np.column_stack( + [multipletests(raw[:, col], method=correction)[1] + for col in range(raw.shape[1])]) + return dict(zip(keys, corrected)) + + def dataframe(self, value_name: str = 'Score', + var_name: str = 'Best', + alg_legend: str = 'Algorithm', + sig_legend: str = 'Significant', + perf_names: str = None, + right: bool = True, + alpha: float = 0.05, + correction: str = None): """Dataframe""" if perf_names is None and isinstance(self.best, np.ndarray): perf_names = [f'{alg}({k})' @@ -822,12 +868,12 @@ def dataframe(self, value_name:str='Score', perf_names=perf_names) df[sig_legend] = False if isinstance(self.best, str): - for name, p in self.p_value(right=right).items(): + for name, p in self.p_value(right=right, correction=correction).items(): if p >= alpha: continue df.loc[df[alg_legend] == name, sig_legend] = True else: - p_values = self.p_value(right=right) + p_values = self.p_value(right=right, correction=correction) systems = list(p_values.keys()) p_values = np.array([p_values[k] for k in systems]) for name, p_value in zip(perf_names, p_values.T): @@ -839,16 +885,17 @@ def dataframe(self, value_name:str='Score', df.loc[_, sig_legend] = True return df - def plot(self, value_name:str='Difference', - var_name:str='Best', - alg_legend:str='Algorithm', - sig_legend:str='Significant', - perf_names:list=None, - alpha:float=0.05, - right:bool=True, - kind:str='point', linestyle:str='none', - col_wrap:int=3, capsize:float=0.2, - set_refline:bool=True, + def plot(self, value_name: str = 'Difference', + var_name: str = 'Best', + alg_legend: str = 'Algorithm', + sig_legend: str = 'Significant', + perf_names: list = None, + alpha: float = 0.05, + right: bool = True, + correction: str = None, + kind: str = 'point', linestyle: str = 'none', + col_wrap: int = 3, capsize: float = 0.2, + set_refline: bool = True, **kwargs): """Plot @@ -874,12 +921,13 @@ def plot(self, value_name:str='Difference', alg_legend=alg_legend, sig_legend=sig_legend, perf_names=perf_names, - alpha=alpha, right=right) - title = var_name + alpha=alpha, right=right, + correction=correction) + title = var_name if var_name not in df.columns: var_name = None col_wrap = None - ci = lambda x: measurements.CI(x, alpha=2*alpha) + ci = lambda x: measurements.CI(x, alpha=2 * alpha) f_grid = sns.catplot(df, x=value_name, errorbar=ci, y=alg_legend, col=var_name, kind=kind, linestyle=linestyle, diff --git a/CompStats/measurements.py b/CompStats/measurements.py index 7cbf06d..db52e4b 100644 --- a/CompStats/measurements.py +++ b/CompStats/measurements.py @@ -20,12 +20,12 @@ def CI(samples: np.ndarray, alpha=0.05): """Compute the Confidence Interval of a statistic using bootstrap. :param samples: Bootstrap samples :type samples: np.ndarray - :param alpha: :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`. + :param alpha: :math:`[\\frac{\\alpha}{2}, 1 - \\frac{\\alpha}{2}]`. :type alpha: float >>> from CompStats import StatisticSamples, CI >>> from sklearn.metrics import accuracy_score - >>> import numpy as np + >>> import numpy as np >>> labels = np.r_[[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]] >>> pred = np.r_[[0, 0, 1, 0, 0, 1, 1, 1, 0, 1]] >>> bootstrap = StatisticSamples(statistic=accuracy_score) @@ -42,10 +42,10 @@ def CI(samples: np.ndarray, alpha=0.05): def SE(samples: np.ndarray): """Compute the Standard Error of a statistic using bootstrap. - + >>> from CompStats import StatisticSamples, SE >>> from sklearn.metrics import accuracy_score - >>> import numpy as np + >>> import numpy as np >>> labels = np.r_[[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]] >>> pred = np.r_[[0, 0, 1, 0, 0, 1, 1, 1, 0, 1]] >>> bootstrap = StatisticSamples(statistic=accuracy_score) @@ -56,7 +56,7 @@ def SE(samples: np.ndarray): return {k: SE(v) for k, v in samples.calls.items()} return np.std(samples, axis=0) - + def difference_p_value(samples: np.ndarray, BiB: bool = True): """Compute the difference p-value""" if isinstance(samples, StatisticSamples): @@ -68,4 +68,4 @@ def difference_p_value(samples: np.ndarray, BiB: bool = True): if BiB: return np.mean(samples > 2 * np.mean(samples, axis=0), axis=0) else: - return np.mean(samples < 2 * np.mean(samples, axis=0), axis=0) \ No newline at end of file + return np.mean(samples < 2 * np.mean(samples, axis=0), axis=0) diff --git a/CompStats/metrics.py b/CompStats/metrics.py index f220c9c..26f0033 100644 --- a/CompStats/metrics.py +++ b/CompStats/metrics.py @@ -35,18 +35,18 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def accuracy_score(y_true, *y_pred, normalize=True, sample_weight=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """accuracy_score""" return Perf(y_true, *y_pred, - score_func=_accuracy_score_measure(normalize=normalize, - sample_weight=sample_weight), + func=_accuracy_score_measure(normalize=normalize, + sample_weight=sample_weight), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -67,18 +67,18 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def balanced_accuracy_score(y_true, *y_pred, sample_weight=None, adjusted=False, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """balanced_accuracy_score""" return Perf(y_true, *y_pred, - score_func=_balanced_accuracy_score_measure(sample_weight=sample_weight, - adjusted=adjusted), + func=_balanced_accuracy_score_measure(sample_weight=sample_weight, + adjusted=adjusted), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -99,20 +99,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_score', attr_name='score_func') +@metrics_docs(hy_name='y_score', bib=True) def top_k_accuracy_score(y_true, *y_score, k=2, normalize=True, sample_weight=None, labels=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """top_k_accuracy_score""" return Perf(y_true, *y_score, - score_func=_top_k_accuracy_score_measure(k=k, normalize=normalize, - sample_weight=sample_weight, - labels=labels), + func=_top_k_accuracy_score_measure(k=k, normalize=normalize, + sample_weight=sample_weight, + labels=labels), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -133,19 +133,19 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_score', attr_name='score_func') +@metrics_docs(hy_name='y_score', bib=True) def average_precision_score(y_true, *y_score, average='macro', sample_weight=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """average_precision_score""" return Perf(y_true, *y_score, - score_func=_average_precision_score_measure(average=average, - sample_weight=sample_weight), + func=_average_precision_score_measure(average=average, + sample_weight=sample_weight), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -166,20 +166,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_proba', attr_name='error_func') +@metrics_docs(hy_name='y_proba', bib=False) def brier_score_loss(y_true, *y_proba, sample_weight=None, pos_label=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs ): """brier_score_loss""" - return Perf(y_true, *y_proba, score_func=None, - error_func=_brier_score_loss_measure(sample_weight=sample_weight, - pos_label=pos_label), + return Perf(y_true, *y_proba, + func=_brier_score_loss_measure(sample_weight=sample_weight, + pos_label=pos_label), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -203,19 +203,19 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def f1_score(y_true, *y_pred, labels=None, pos_label=1, average='binary', sample_weight=None, - zero_division='warn', num_samples: int=500, - n_jobs: int=-1, use_tqdm=True, + zero_division='warn', num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """f1_score""" return Perf(y_true, *y_pred, - score_func=_f1_score_measure(labels=labels, pos_label=pos_label, - average=average, - sample_weight=sample_weight, - zero_division=zero_division), + func=_f1_score_measure(labels=labels, pos_label=pos_label, + average=average, + sample_weight=sample_weight, + zero_division=zero_division), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -236,21 +236,21 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def log_loss(y_true, *y_pred, normalize=True, sample_weight=None, labels=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """log_loss""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_log_loss_measure(normalize=normalize, - sample_weight=sample_weight, - labels=labels), + return Perf(y_true, *y_pred, + func=_log_loss_measure(normalize=normalize, + sample_weight=sample_weight, + labels=labels), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -275,7 +275,7 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def precision_score(y_true, *y_pred, labels=None, @@ -283,17 +283,17 @@ def precision_score(y_true, average='binary', sample_weight=None, zero_division='warn', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """precision_score""" return Perf(y_true, *y_pred, - score_func=_precision_score_measure(labels=labels, pos_label=pos_label, - average=average, - sample_weight=sample_weight, - zero_division=zero_division), + func=_precision_score_measure(labels=labels, pos_label=pos_label, + average=average, + sample_weight=sample_weight, + zero_division=zero_division), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -318,7 +318,7 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def recall_score(y_true, *y_pred, labels=None, @@ -326,17 +326,17 @@ def recall_score(y_true, average='binary', sample_weight=None, zero_division='warn', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """recall_score""" return Perf(y_true, *y_pred, - score_func=_recall_score_measure(labels=labels, pos_label=pos_label, - average=average, - sample_weight=sample_weight, - zero_division=zero_division), + func=_recall_score_measure(labels=labels, pos_label=pos_label, + average=average, + sample_weight=sample_weight, + zero_division=zero_division), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -361,7 +361,7 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def jaccard_score(y_true, *y_pred, labels=None, @@ -369,17 +369,17 @@ def jaccard_score(y_true, average='binary', sample_weight=None, zero_division='warn', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """jaccard_score""" return Perf(y_true, *y_pred, - score_func=_jaccard_score_measure(labels=labels, pos_label=pos_label, - average=average, - sample_weight=sample_weight, - zero_division=zero_division), + func=_jaccard_score_measure(labels=labels, pos_label=pos_label, + average=average, + sample_weight=sample_weight, + zero_division=zero_division), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -404,7 +404,7 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_score', attr_name='score_func') +@metrics_docs(hy_name='y_score', bib=True) def roc_auc_score(y_true, *y_score, average='macro', @@ -412,18 +412,18 @@ def roc_auc_score(y_true, max_fpr=None, multi_class='raise', labels=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """roc_auc_score""" return Perf(y_true, *y_score, - score_func=_roc_auc_score_measure(average=average, - sample_weight=sample_weight, - max_fpr=max_fpr, - multi_class=multi_class, - labels=labels), + func=_roc_auc_score_measure(average=average, + sample_weight=sample_weight, + max_fpr=max_fpr, + multi_class=multi_class, + labels=labels), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -438,26 +438,25 @@ def _d2_log_loss_score_measure(sample_weight=None, labels=None): @wraps(metrics.d2_log_loss_score) def inner(y, hy): return metrics.d2_log_loss_score(y, hy, - sample_weight=sample_weight, - labels=labels) + sample_weight=sample_weight, + labels=labels) inner.BiB = True return inner -@metrics_docs(hy_name='y_proba', attr_name='score_func') +@metrics_docs(hy_name='y_proba', bib=True) def d2_log_loss_score(y_true, *y_proba, sample_weight=None, labels=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """d2_log_loss_score""" return Perf(y_true, *y_proba, - score_func=_d2_log_loss_score_measure(sample_weight=sample_weight, - labels=labels), - error_func=None, + func=_d2_log_loss_score_measure(sample_weight=sample_weight, + labels=labels), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -476,9 +475,9 @@ def _macro_f1_measure(labels=None, sample_weight=None, zero_division='warn'): def macro_f1(y_true, *y_pred, labels=None, sample_weight=None, zero_division='warn', - num_samples: int=500, n_jobs: int=-1, use_tqdm=True, + num_samples: int = 500, n_jobs: int = -1, use_tqdm=True, **kwargs): - """:py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.f1_score` (as :py:attr:`score_func`) with the parameteres needed to compute the macro score. The parameters not described can be found in :py:func:`~sklearn.metrics.f1_score` + """:py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.f1_score` (as :py:attr:`func`) with the parameteres needed to compute the macro score. The parameters not described can be found in :py:func:`~sklearn.metrics.f1_score` :param y_true: True measurement or could be a pandas.DataFrame where column label 'y' corresponds to the true measurement. :type y_true: numpy.ndarray or pandas.DataFrame @@ -512,9 +511,9 @@ def _macro_recall_measure(labels=None, sample_weight=None, zero_division='warn') def macro_recall(y_true, *y_pred, labels=None, sample_weight=None, zero_division='warn', - num_samples: int=500, n_jobs: int=-1, use_tqdm=True, + num_samples: int = 500, n_jobs: int = -1, use_tqdm=True, **kwargs): - """:py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.recall_score` (as :py:attr:`score_func`) with the parameteres needed to compute the macro score. The parameters not described can be found in :py:func:`~sklearn.metrics.recall_score` + """:py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.recall_score` (as :py:attr:`func`) with the parameteres needed to compute the macro score. The parameters not described can be found in :py:func:`~sklearn.metrics.recall_score` :param y_true: True measurement or could be a pandas.DataFrame where column label 'y' corresponds to the true measurement. :type y_true: numpy.ndarray or pandas.DataFrame @@ -548,9 +547,9 @@ def _macro_precision_measure(labels=None, sample_weight=None, zero_division='war def macro_precision(y_true, *y_pred, labels=None, sample_weight=None, zero_division='warn', - num_samples: int=500, n_jobs: int=-1, use_tqdm=True, + num_samples: int = 500, n_jobs: int = -1, use_tqdm=True, **kwargs): - """:py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.precision_score` (as :py:attr:`score_func`) with the parameteres needed to compute the macro score. The parameters not described can be found in :py:func:`~sklearn.metrics.precision_score` + """:py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.precision_score` (as :py:attr:`func`) with the parameteres needed to compute the macro score. The parameters not described can be found in :py:func:`~sklearn.metrics.precision_score` :param y_true: True measurement or could be a pandas.DataFrame where column label 'y' corresponds to the true measurement. :type y_true: numpy.ndarray or pandas.DataFrame @@ -593,22 +592,22 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def explained_variance_score(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', force_finite=True, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """explained_variance_score""" return Perf(y_true, *y_pred, - score_func=_explained_variance_score_measure(sample_weight=sample_weight, - multioutput=multioutput, - force_finite=force_finite), + func=_explained_variance_score_measure(sample_weight=sample_weight, + multioutput=multioutput, + force_finite=force_finite), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -627,16 +626,16 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def max_error(y_true, *y_pred, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """max_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_max_error_measure(), + return Perf(y_true, *y_pred, + func=_max_error_measure(), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -657,20 +656,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def mean_absolute_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """mean_absolute_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_mean_absolute_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_mean_absolute_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -691,20 +690,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def mean_squared_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """mean_squared_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_mean_squared_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_mean_squared_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -725,20 +724,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def root_mean_squared_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """root_mean_squared_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_root_mean_squared_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_root_mean_squared_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -759,20 +758,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def mean_squared_log_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """mean_squared_log_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_mean_squared_log_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_mean_squared_log_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -793,20 +792,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def root_mean_squared_log_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """root_mean_squared_log_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_root_mean_squared_log_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_root_mean_squared_log_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -827,20 +826,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def median_absolute_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """median_absolute_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_median_absolute_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_median_absolute_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -862,23 +861,22 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='score_func') +@metrics_docs(hy_name='y_pred', bib=True) def r2_score(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', force_finite=True, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """r2_score""" return Perf(y_true, *y_pred, - score_func=_r2_score_measure(sample_weight=sample_weight, - multioutput=multioutput, - force_finite=force_finite), - error_func=None, + func=_r2_score_measure(sample_weight=sample_weight, + multioutput=multioutput, + force_finite=force_finite), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -898,18 +896,18 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def mean_poisson_deviance(y_true, *y_pred, sample_weight=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """mean_poisson_deviance""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_mean_poisson_deviance_measure(sample_weight=sample_weight), + return Perf(y_true, *y_pred, + func=_mean_poisson_deviance_measure(sample_weight=sample_weight), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -929,18 +927,18 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def mean_gamma_deviance(y_true, *y_pred, sample_weight=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """mean_gamma_deviance""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_mean_gamma_deviance_measure(sample_weight=sample_weight), + return Perf(y_true, *y_pred, + func=_mean_gamma_deviance_measure(sample_weight=sample_weight), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -961,20 +959,20 @@ def inner(y, hy): return inner -@metrics_docs(hy_name='y_pred', attr_name='error_func') +@metrics_docs(hy_name='y_pred', bib=False) def mean_absolute_percentage_error(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """mean_absolute_percentage_error""" - return Perf(y_true, *y_pred, score_func=None, - error_func=_mean_absolute_percentage_error_measure(sample_weight=sample_weight, - multioutput=multioutput), + return Perf(y_true, *y_pred, + func=_mean_absolute_percentage_error_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -999,16 +997,15 @@ def d2_absolute_error_score(y_true, *y_pred, sample_weight=None, multioutput='uniform_average', - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): """d2_absolute_error_score""" return Perf(y_true, *y_pred, - score_func=_d2_absolute_error_score_measure(sample_weight=sample_weight, - multioutput=multioutput), - error_func=None, + func=_d2_absolute_error_score_measure(sample_weight=sample_weight, + multioutput=multioutput), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) @@ -1031,11 +1028,11 @@ def inner(y, hy): def pearsonr(y_true, *y_pred, alternative='two-sided', method=None, - num_samples: int=500, - n_jobs: int=-1, + num_samples: int = 500, + n_jobs: int = -1, use_tqdm=True, **kwargs): - """:py:class:`~CompStats.interface.Perf` with :py:func:`~scipy.stats.pearsonr` as :py:attr:`score_func.` + """:py:class:`~CompStats.interface.Perf` with :py:func:`~scipy.stats.pearsonr` as :py:attr:`func.` :param y_true: True measurement or could be a pandas.DataFrame where column label 'y' corresponds to the true measurement. :type y_true: numpy.ndarray or pandas.DataFrame @@ -1052,8 +1049,7 @@ def pearsonr(y_true, *y_pred, """ return Perf(y_true, *y_pred, - score_func=_pearsonr_measure(alternative=alternative, method=method), - error_func=None, + func=_pearsonr_measure(alternative=alternative, method=method), num_samples=num_samples, n_jobs=n_jobs, use_tqdm=use_tqdm, **kwargs) diff --git a/CompStats/performance.py b/CompStats/performance.py index cb25084..10ae471 100644 --- a/CompStats/performance.py +++ b/CompStats/performance.py @@ -26,12 +26,12 @@ def performance(data: pd.DataFrame, - gold: str='y', - score: Callable[[np.ndarray, np.ndarray], float]=accuracy_score, - num_samples: int=500, - n_jobs: int=-1, - BiB: bool=True, - statistic_samples: StatisticSamples=None) -> StatisticSamples: + gold: str = 'y', + score: Callable[[np.ndarray, np.ndarray], float] = accuracy_score, + num_samples: int = 500, + n_jobs: int = -1, + BiB: bool = True, + statistic_samples: StatisticSamples = None) -> StatisticSamples: """Calculate bootstrap samples of a performance score for a given dataset. Parameters: @@ -63,11 +63,11 @@ def performance(data: pd.DataFrame, if column == gold: continue statistic_samples(y, data[column], name=column) - + return statistic_samples -def difference(statistic_samples: StatisticSamples): #, best_index: int=-1): +def difference(statistic_samples: StatisticSamples): # , best_index: int=-1): """ Computes the difference in performance between the best performing algorithm and others using bootstrap samples. @@ -144,24 +144,24 @@ def all_differences(statistic_samples: StatisticSamples): # Calculamos el rendimiento medio y ordenamos los algoritmos basándonos en este perf = [(k, v, np.mean(v)) for k, v in items] perf.sort(key=lambda x: x[2], reverse=statistic_samples.BiB) # Orden por rendimiento medio - + diffs = {} # Diccionario para guardar las diferencias - + # Iteramos sobre todos los pares posibles de algoritmos ordenados for i in range(len(perf)): for j in range(i + 1, len(perf)): name_i, perf_i, _ = perf[i] name_j, perf_j, _ = perf[j] - + # Diferencia de i a j diff_key_i_to_j = f"{name_i} - {name_j}" diffs[diff_key_i_to_j] = np.array(perf_i) - np.array(perf_j) output = clone(statistic_samples) output.calls = diffs return output - -def plot_performance(statistic_samples: StatisticSamples, CI: float=0.05, + +def plot_performance(statistic_samples: StatisticSamples, CI: float = 0.05, var_name='Algorithm', value_name='Score', capsize=0.2, linestyle='none', kind='point', sharex=False, **kwargs): @@ -193,7 +193,7 @@ def plot_performance(statistic_samples: StatisticSamples, CI: float=0.05, 2. Converts the data into a long format DataFrame. 3. Computes the confidence intervals if CI is provided as a float. 4. Plots the performance data with confidence intervals using seaborn's catplot. - + >>> from CompStats import performance, plot_performance >>> from CompStats.tests.test_performance import DATA >>> from sklearn.metrics import f1_score @@ -208,7 +208,7 @@ def plot_performance(statistic_samples: StatisticSamples, CI: float=0.05, lista_ordenada = sorted(statistic_samples.calls.items(), key=lambda x: np.mean(x[1]), reverse=statistic_samples.BiB) diccionario_ordenado = {nombre: muestras for nombre, muestras in lista_ordenada} df2 = pd.DataFrame(diccionario_ordenado).melt(var_name=var_name, - value_name=value_name) + value_name=value_name) else: df2 = statistic_samples if isinstance(CI, float): @@ -219,7 +219,7 @@ def plot_performance(statistic_samples: StatisticSamples, CI: float=0.05, return f_grid -def plot_difference(statistic_samples: StatisticSamples, CI: float=0.05, +def plot_difference(statistic_samples: StatisticSamples, CI: float = 0.05, var_name='Comparison', value_name='Difference', set_refline=True, set_title=True, hue='Significant', palette=None, @@ -246,7 +246,7 @@ def plot_difference(statistic_samples: StatisticSamples, CI: float=0.05, 2. Adds a 'Significant' column to indicate whether the confidence interval includes zero. 3. Plots the differences with confidence intervals using the plot_performance function. 4. Optionally sets a reference line at x=0 and a title indicating the best performing algorithm. - + >>> from CompStats import performance, difference, plot_difference >>> from CompStats.tests.test_performance import DATA >>> from sklearn.metrics import f1_score @@ -261,7 +261,7 @@ def plot_difference(statistic_samples: StatisticSamples, CI: float=0.05, lista_ordenada = sorted(statistic_samples.calls.items(), key=lambda x: np.mean(x[1]), reverse=statistic_samples.BiB) diccionario_ordenado = {nombre: muestras for nombre, muestras in lista_ordenada} df2 = pd.DataFrame(diccionario_ordenado).melt(var_name=var_name, - value_name=value_name) + value_name=value_name) if hue is not None: df2[hue] = True at_least_one = False @@ -273,7 +273,7 @@ def plot_difference(statistic_samples: StatisticSamples, CI: float=0.05, if at_least_one and palette is None: palette = ['r', 'b'] else: - palette = ['b'] + palette = ['b'] f_grid = plot_performance(df2, CI=CI, var_name=var_name, value_name=value_name, hue=hue, palette=palette, @@ -285,7 +285,8 @@ def plot_difference(statistic_samples: StatisticSamples, CI: float=0.05, f_grid.facet_axis(0, 0).set_title(f'Best: {best}') return f_grid -def performance_multiple_metrics(data: pd.DataFrame, gold: str, + +def performance_multiple_metrics(data: pd.DataFrame, gold: str, scores: List[dict], num_samples: int = 500, n_jobs: int = -1): """ @@ -334,7 +335,7 @@ def performance_multiple_metrics(data: pd.DataFrame, gold: str, >>> results = performance_multiple_metrics(df, gold='target', scores=scores, num_samples=1000) """ results, performance_dict, perfo, dist, ccv, cppi, compg, cBiB = {}, {}, {}, {}, {}, {}, {}, {} - n,m = data.shape + n, m = data.shape # definimos las funciones para las metricas cv = lambda x: np.std(x, ddof=1) / np.mean(x) * 100 dista = lambda x: np.abs(np.max(x) - np.median(x)) @@ -355,26 +356,27 @@ def performance_multiple_metrics(data: pd.DataFrame, gold: str, if column == gold: continue results[metric_name][column] = statistic_samples(data[gold], data[column]) - perfo[metric_name][column] = statistic(data[gold], data[column]) + perfo[metric_name][column] = statistic(data[gold], data[column]) ccv[metric_name] = cv(np.array(list(perfo[metric_name].values()))) dist[metric_name] = dista(np.array(list(perfo[metric_name].values()))) cppi[metric_name] = ppi(np.array(list(perfo[metric_name].values()))) cBiB[metric_name] = score_BiB - compg = {'n' : n, - 'm' : m-1, - 'cv' : ccv, - 'dist' : dist, - 'PPI' : cppi} - performance_dict = {'samples' : results, - 'performance' : perfo, - 'compg' : compg, + compg = {'n': n, + 'm': m - 1, + 'cv': ccv, + 'dist': dist, + 'PPI': cppi} + performance_dict = {'samples': results, + 'performance': perfo, + 'compg': compg, 'BiB': cBiB} - return performance_dict + return performance_dict -def plot_performance2(results: dict, CI: float=0.05, - var_name='Algorithm', value_name='Score', - capsize=0.2, linestyle='none', kind='point', - sharex=False, **kwargs): + +def plot_performance2(results: dict, CI: float = 0.05, + var_name='Algorithm', value_name='Score', + capsize=0.2, linestyle='none', kind='point', + sharex=False, **kwargs): """ Plot the performance with confidence intervals. This function is used by plot_difference_multiple @@ -397,12 +399,12 @@ def plot_performance2(results: dict, CI: float=0.05, 2. Converts the sorted data into a long format DataFrame. 3. Computes the confidence intervals if CI is provided as a float. 4. Uses seaborn's catplot to create and display the performance plot with confidence intervals. - """ + """ if isinstance(results, dict): lista_ordenada = sorted(results.items(), key=lambda x: np.mean(x[1]), reverse=True) diccionario_ordenado = {nombre: muestras for nombre, muestras in lista_ordenada} df2 = pd.DataFrame(diccionario_ordenado).melt(var_name=var_name, - value_name=value_name) + value_name=value_name) if isinstance(CI, float): ci = lambda x: measurements.CI(x, alpha=CI) @@ -412,9 +414,7 @@ def plot_performance2(results: dict, CI: float=0.05, return f_grid - - -def difference_multiple(results_dict, CI: float=0.05,): +def difference_multiple(results_dict, CI: float = 0.05,): """ Calculate performance differences for multiple metrics, excluding the comparison of the best with itself. Additionally, identify the best performing algorithm for each metric. @@ -462,33 +462,32 @@ def difference_multiple(results_dict, CI: float=0.05,): else: best_alg = min(scores_arrays, key=lambda alg: np.mean(scores_arrays[alg])) best_scores = scores_arrays[best_alg] - + # Calculate differences to the best performing algorithm, excluding the best from comparing with itself differences = {alg: best_scores - scores for alg, scores in scores_arrays.items() if alg != best_alg} # Calculate Confidence interval for differences to the bet performing algorithm. CI_differences = {alg: measurements.CI(np.array(scores), alpha=CI) for alg, scores in differences.items()} - p_value_differences = {alg: measurements.difference_p_value(np.array(scores), BiB= results_dict['BiB'][metric]) for alg, scores in differences.items()} - + p_value_differences = {alg: measurements.difference_p_value(np.array(scores), BiB=results_dict['BiB'][metric]) for alg, scores in differences.items()} # Store the differences and the best algorithm under the current metric - winner[metric] = {'best': best_alg, 'diff': differences,'CI':CI_differences, - 'p_value': p_value_differences, - 'none': sum(valor > alpha for valor in p_value_differences.values()), - 'bonferroni': sum(multipletests(list(p_value_differences.values()), method='bonferroni')[1] > alpha), - 'holm': sum(multipletests(list(p_value_differences.values()), method='holm')[1] > alpha), - 'HB': sum(multipletests(list(p_value_differences.values()), method='fdr_bh')[1] > alpha) } + winner[metric] = {'best': best_alg, 'diff': differences, 'CI': CI_differences, + 'p_value': p_value_differences, + 'none': sum(valor > alpha for valor in p_value_differences.values()), + 'bonferroni': sum(multipletests(list(p_value_differences.values()), method='bonferroni')[1] > alpha), + 'holm': sum(multipletests(list(p_value_differences.values()), method='holm')[1] > alpha), + 'HB': sum(multipletests(list(p_value_differences.values()), method='fdr_bh')[1] > alpha)} differences_dict['winner'] = winner return differences_dict def plot_difference2(diff_dictionary: dict, CI: float = 0.05, - var_name='Comparison', value_name='Difference', - set_refline=True, set_title=True, - hue='Significant', palette=None, BiB: bool=True, - **kwargs): + var_name='Comparison', value_name='Difference', + set_refline=True, set_title=True, + hue='Significant', palette=None, BiB: bool = True, + **kwargs): """Plot the difference in performance with its confidence intervals - + >>> from CompStats import performance, difference, plot_difference >>> from CompStats.tests.test_performance import DATA >>> from sklearn.metrics import f1_score @@ -503,7 +502,7 @@ def plot_difference2(diff_dictionary: dict, CI: float = 0.05, lista_ordenada = sorted(diff_dictionary['diff'].items(), key=lambda x: np.mean(x[1]), reverse=BiB) diccionario_ordenado = {nombre: muestras for nombre, muestras in lista_ordenada} df2 = pd.DataFrame(diccionario_ordenado).melt(var_name=var_name, - value_name=value_name) + value_name=value_name) if hue is not None: df2[hue] = True at_least_one = False @@ -518,7 +517,7 @@ def plot_difference2(diff_dictionary: dict, CI: float = 0.05, palette = ['b'] f_grid = plot_performance(df2, CI=CI, var_name=var_name, value_name=value_name, hue=hue, - palette=palette, + palette=palette, **kwargs) if set_refline: f_grid.refline(x=0) @@ -527,13 +526,14 @@ def plot_difference2(diff_dictionary: dict, CI: float = 0.05, f_grid.facet_axis(0, 0).set_title(f'Best: {best}') return f_grid -def plot_performance_multiple(results_dict: dict, CI: float = 0.05, capsize: float = 0.2, + +def plot_performance_multiple(results_dict: dict, CI: float = 0.05, capsize: float = 0.2, linestyle: str = 'none', kind: str = 'point', **kwargs): """ Create multiple performance plots, one for each performance metric in the results dictionary. Parameters: - results_dict (dict): A dictionary where keys are metric names and values are dictionaries + results_dict (dict): A dictionary where keys are metric names and values are dictionaries with algorithm names as keys and lists of performance scores as values. CI (float, optional): Confidence interval level for error bars. Defaults to 0.05. capsize (float, optional): Cap size for error bars. Defaults to 0.2. @@ -554,17 +554,17 @@ def plot_performance_multiple(results_dict: dict, CI: float = 0.05, capsize: flo >>> from CompStats import plot_performance_multiple >>> results = { >>> 'accuracy': { - >>> 'alg1': [0.1, 0.2, 0.15], + >>> 'alg1': [0.1, 0.2, 0.15], >>> 'alg2': [0.05, 0.1, 0.07] >>> }, >>> 'f1_score': { - >>> 'alg1': [0.3, 0.25, 0.2], + >>> 'alg1': [0.3, 0.25, 0.2], >>> 'alg2': [0.2, 0.15, 0.1] >>> } >>> } >>> plot_performance_multiple(results, CI=0.05) """ - + for metric_name, metric_results in results_dict['samples'].items(): BiB = results_dict['BiB'].get(metric_name, True) # Convert results to long format DataFrame @@ -572,19 +572,19 @@ def plot_performance_multiple(results_dict: dict, CI: float = 0.05, capsize: flo lista_ordenada = sorted(metric_results.items(), key=lambda x: np.mean(x[1]), reverse=BiB) diccionario_ordenado = {nombre: muestras for nombre, muestras in lista_ordenada} df2 = pd.DataFrame(diccionario_ordenado).melt(var_name='Algorithm', - value_name='Score') - + value_name='Score') + # Define the confidence interval function if isinstance(CI, float): ci = lambda x: measurements.CI(x, alpha=CI) - + # Create the plot - g = sns.catplot(df2, x='Score', y='Algorithm', capsize=capsize, linestyle=linestyle, + g = sns.catplot(df2, x='Score', y='Algorithm', capsize=capsize, linestyle=linestyle, kind=kind, errorbar=ci, **kwargs) - + # Set the title of the plot g.figure.suptitle(metric_name) - + # Display the plot plt.show() @@ -592,30 +592,28 @@ def plot_performance_multiple(results_dict: dict, CI: float = 0.05, capsize: flo def plot_difference_multiple(results_dict, CI=0.05, capsize=0.2, linestyle='none', kind='point', **kwargs): """ Create multiple performance plots, one for each performance metric in the results dictionary. - + :param results_dict: A dictionary where keys are metric names and values are dictionaries with algorithm names as keys and lists of scores as values. :param CI: Confidence interval level for error bars. :param capsize: Cap size for error bars. :param linestyle: Line style for the plot. :param kind: Type of the plot, e.g., 'point', 'bar'. :param kwargs: Additional keyword arguments for seaborn.catplot. - """ + """ for metric_name, metric_results in results_dict['winner'].items(): BiB = results_dict['BiB'].get(metric_name, True) - # Usa catplot para crear y mostrar el gráfico + # Usa catplot para crear y mostrar el gráfico g = plot_difference2(metric_results, BiB=BiB, CI=CI) - g.figure.suptitle(metric_name) + g.figure.suptitle(metric_name) # plt.show() - - -### este por el momento no. +# este por el momento no. def plot_scatter_matrix(perf): """ Generate a scatter plot matrix comparing the performance of the same algorithm across different metrics contained in the 'perf' dictionary. - + :param perf: A dictionary where keys are metric names and values are dictionaries with algorithm names as keys and lists of performance scores as values. """ @@ -624,21 +622,20 @@ def plot_scatter_matrix(perf): {"Metric": metric, "Algorithm": alg, "Score": score, "Indice": i} for metric, alg_scores in perf['samples'].items() for alg, scores in alg_scores.items() - for i, (score) in enumerate(scores) - ]) - df_wide = df_long.pivot(index=['Algorithm','Indice'],columns='Metric',values='Score') + for i, (score) in enumerate(scores) + ]) + df_wide = df_long.pivot(index=['Algorithm', 'Indice'], columns='Metric', values='Score') df_wide = df_wide.reset_index(level=[0]) - sns.pairplot(df_wide, diag_kind='kde',hue="Algorithm", corner=True) + sns.pairplot(df_wide, diag_kind='kde', hue="Algorithm", corner=True) plt.suptitle('Scatter Plot Matrix of Algorithms Performance Across Different Metrics', y=1.02) plt.show() - -def all_differences_multiple(results_dict, alpha: float=0.05): +def all_differences_multiple(results_dict, alpha: float = 0.05): """ Calculate performance differences for unique pairs of algorithms for multiple metrics. Also, calculates the confidence interval for the differences. - + :param results_dict: A dictionary where keys are metric names and values are dictionaries. Each sub-dictionary has algorithm names as keys and lists of performance scores as values. :return: A dictionary where each metric name maps to another dictionary. @@ -649,33 +646,31 @@ def all_differences_multiple(results_dict, alpha: float=0.05): all = {} for metric, results in results_dict['samples'].items(): # Convert scores to arrays for vectorized operations - scores_arrays = {alg: np.array(scores) for alg, scores in results.items()} + scores_arrays = {alg: np.array(scores) for alg, scores in results.items()} scores_arrays = dict(sorted(scores_arrays.items(), key=lambda item: np.mean(item[1]), reverse=results_dict['BiB'][metric])) - differences = {} p_value_differences = {} - + algorithms = list(scores_arrays.keys()) # Calculate differences for unique pairs of algorithms for i, alg_a in enumerate(algorithms): - for alg_b in algorithms[i+1:]: # Start from the next algorithm to avoid duplicate comparisons + for alg_b in algorithms[i + 1:]: # Start from the next algorithm to avoid duplicate comparisons # Calculate the difference between alg_a and alg_b diff = scores_arrays[alg_a] - scores_arrays[alg_b] differences[f"{alg_a} vs {alg_b}"] = diff - + # Placeholder for confidence interval calculation # Replace the string with an actual call to your CI calculation function p_value_differences[f"{alg_a} vs {alg_b}"] = measurements.difference_p_value(diff, BiB=results_dict['BiB'][metric]) # For example: # CI_differences[f"{alg_a} vs {alg_b}"] = measurements.CI(diff, alpha=CI) - + # Store the differences under the current metric - all[metric] = {'diff': differences, 'p_value': p_value_differences, - 'none': sum(valor > alpha for valor in p_value_differences.values()), - 'bonferroni': sum(multipletests(list(p_value_differences.values()), method='bonferroni')[1] > alpha), - 'holm': sum(multipletests(list(p_value_differences.values()), method='holm')[1] > alpha), - 'HB': sum(multipletests(list(p_value_differences.values()), method='fdr_bh')[1] > alpha) } + all[metric] = {'diff': differences, 'p_value': p_value_differences, + 'none': sum(valor > alpha for valor in p_value_differences.values()), + 'bonferroni': sum(multipletests(list(p_value_differences.values()), method='bonferroni')[1] > alpha), + 'holm': sum(multipletests(list(p_value_differences.values()), method='holm')[1] > alpha), + 'HB': sum(multipletests(list(p_value_differences.values()), method='fdr_bh')[1] > alpha)} differences_dict['all'] = all return differences_dict - diff --git a/CompStats/tests/__init__.py b/CompStats/tests/__init__.py index 2f1fa6e..68bdaf6 100644 --- a/CompStats/tests/__init__.py +++ b/CompStats/tests/__init__.py @@ -10,4 +10,4 @@ # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and -# limitations under the License. \ No newline at end of file +# limitations under the License. diff --git a/CompStats/tests/test_bootstrap.py b/CompStats/tests/test_bootstrap.py index 752ed97..0bc759f 100644 --- a/CompStats/tests/test_bootstrap.py +++ b/CompStats/tests/test_bootstrap.py @@ -18,13 +18,13 @@ def problem_algorithms(): """Problem and three predictions""" labels = [0, 0, 0, 0, 0, - 1, 1, 1, 1, 1] + 1, 1, 1, 1, 1] a = [0, 0, 0, 0, 0, - 1, 1, 1, 1, 0] + 1, 1, 1, 1, 0] b = [0, 0, 1, 0, 0, - 1, 1, 1, 1, 0] + 1, 1, 1, 1, 0] c = [0, 0, 0, 1, 0, - 1, 1, 0, 1, 0] + 1, 1, 0, 1, 0] return (np.array(labels), dict(a=np.array(a), b=np.array(b), @@ -65,7 +65,6 @@ def test_StatisticSamples_melt(): assert isinstance(df, pd.DataFrame) - # def test_CI(): # """Test CI""" # statistic = CI() @@ -120,4 +119,4 @@ def test_StatisticSamples_melt(): # labels, algs = problem_algorithms() # diff = Difference(labels, algs) # for x, r in zip(diff.sort(), ['b', 'c']): -# assert x == r \ No newline at end of file +# assert x == r diff --git a/CompStats/tests/test_interface.py b/CompStats/tests/test_interface.py index 5c20bf0..00d9e8f 100644 --- a/CompStats/tests/test_interface.py +++ b/CompStats/tests/test_interface.py @@ -219,7 +219,7 @@ def test_difference_str__(): num_samples=50, average=average) diff = perf.difference() print(diff) - + def test_Perf(): """Test perf""" @@ -274,7 +274,8 @@ def test_Perf_clone(): perf = Perf(y_val, forest=ens.predict(X_val), num_samples=50) samples = perf.statistic_samples._samples perf2 = clone(perf) - perf2.error_func = lambda y, hy: (y != hy).mean() + perf2.func = lambda y, hy: (y != hy).mean() + perf2.BiB = False assert 'forest' in perf2.statistic_samples.calls assert np.all(samples == perf2.statistic_samples._samples) @@ -330,9 +331,9 @@ def test_Perf_multi_measure_score_only(): ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) nb = GaussianNB().fit(X_train, y_train) perf = Perf(y_val, ens.predict(X_val), nb=nb.predict(X_val), - score_func=[f1_score.measure(average='macro'), - recall_score.measure(average='macro')], - num_samples=20) + func=[f1_score.measure(average='macro'), + recall_score.measure(average='macro')], + num_samples=20) assert perf.measure_names == ['f1_score', 'recall_score'] assert isinstance(perf.statistic['alg-1'], np.ndarray) assert perf.statistic['alg-1'].shape == (2,) @@ -361,8 +362,8 @@ def error_stat(y, hy): hyA = np.full(10, 5.0) hyB = np.full(10, 2.0) perf = Perf(y_true, A=hyA, B=hyB, - score_func=score_stat, error_func=error_stat, - num_samples=5) + func=[score_stat, error_stat], BiB=[True, False], + num_samples=5) assert perf.measure_names == ['score_stat', 'error_stat'] assert np.all(perf.statistic_samples.BiB == np.array([True, False])) # A has the higher value (wins the score-type column), @@ -374,9 +375,9 @@ def error_stat(y, hy): assert np.allclose(p_values['B'], [0.0, 1.0]) -def test_Perf_measure_tag_overrides_list_position(): +def test_Perf_measure_tag_overrides_default_bib(): """A callable's own .BiB (set by a .measure() factory) wins over the - default direction implied by score_func/error_func placement""" + constructor's BiB default when the two disagree""" from CompStats.interface import Perf from CompStats.metrics import f1_score @@ -384,11 +385,52 @@ def test_Perf_measure_tag_overrides_list_position(): hy = np.array([0, 0, 0, 0, 1, 1, 1, 1, 0, 1]) tagged = f1_score.measure(average='macro') assert tagged.BiB is True - perf = Perf(y_true, alg=hy, score_func=None, - error_func=tagged, num_samples=5) + perf = Perf(y_true, alg=hy, func=tagged, BiB=False, num_samples=5) assert bool(perf.statistic_samples.BiB) is True +def test_Perf_statistic_sort_order_follows_tagged_bib(): + """Perf.statistic's sort order must follow the measure's tagged .BiB, + not the constructor's BiB default (issue #36 regression: this used to + be derived from ``score_func is not None`` and ignored the tag)""" + from CompStats.interface import Perf + from CompStats.metrics import mean_absolute_error + + y_true = np.zeros(10) + perf = Perf(y_true, low=np.zeros(10), high=np.ones(10), + func=mean_absolute_error.measure(), num_samples=5) + # mean_absolute_error.measure() is tagged BiB=False (error-type) even + # though the constructor's own BiB default is True; the smaller-error + # 'low' prediction must rank first + assert list(perf.statistic.keys()) == ['low', 'high'] + + +def test_Perf_repr_label_uses_func(): + """Perf.__repr__ labels the measure as 'func=', regardless of its + tagged direction (issue #36: score_func/error_func no longer exist)""" + from CompStats.interface import Perf + from CompStats.metrics import mean_absolute_error + + y_true = np.zeros(10) + perf = Perf(y_true, low=np.zeros(10), high=np.ones(10), + func=mean_absolute_error.measure(), num_samples=5) + assert 'func=mean_absolute_error' in repr(perf) + + +def test_Perf_plot_value_name_follows_tagged_bib(): + """Perf.plot's default value_name label ('Score' vs 'Error') must follow + the measure's tagged .BiB, not the constructor's BiB default (issue #36 + regression: this used to be derived from ``score_func is not None``)""" + from CompStats.interface import Perf + from CompStats.metrics import mean_absolute_error + + y_true = np.zeros(10) + perf = Perf(y_true, low=np.zeros(10), high=np.ones(10), + func=mean_absolute_error.measure(), num_samples=5) + f_grid = perf.plot() + assert 'Error' in f_grid.data.columns + + def test_Perf_multi_measure_clone(): """Test that cloning a multi-measure Perf preserves measures and samples""" from sklearn.base import clone @@ -401,9 +443,9 @@ def test_Perf_multi_measure_clone(): ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) nb = GaussianNB().fit(X_train, y_train) perf = Perf(y_val, forest=ens.predict(X_val), nb=nb.predict(X_val), - score_func=[f1_score.measure(average='macro'), - recall_score.measure(average='macro')], - num_samples=20) + func=[f1_score.measure(average='macro'), + recall_score.measure(average='macro')], + num_samples=20) samples = perf.statistic_samples._samples perf2 = clone(perf) assert perf2.measure_names == ['f1_score', 'recall_score'] @@ -428,4 +470,100 @@ def test_Perf_call(): print(_) perf(hy, name='alg-2') assert 'alg-2' not in perf._statistic_samples.calls - assert 'alg-1' in perf._statistic_samples.calls \ No newline at end of file + assert 'alg-1' in perf._statistic_samples.calls + + +def test_Difference_p_value_correction_single_measure(): + """Test Difference.p_value multiple-comparison correction (single measure)""" + from statsmodels.stats.multitest import multipletests + from CompStats.metrics import f1_score + + X, y = load_digits(return_X_y=True) + _ = train_test_split(X, y, test_size=0.3, random_state=0) + X_train, X_val, y_train, y_val = _ + ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) + nb = GaussianNB().fit(X_train, y_train) + svm = LinearSVC().fit(X_train, y_train) + score = f1_score(y_val, ens.predict(X_val), average='macro', + num_samples=50) + score(nb.predict(X_val)) + score(svm.predict(X_val)) + diff = score.difference() + raw = diff.p_value() + keys = list(raw.keys()) + expected = multipletests(list(raw.values()), method='bonferroni')[1] + corrected = diff.p_value(correction='bonferroni') + assert list(corrected.keys()) == keys + assert np.allclose(list(corrected.values()), expected) + + +def test_Difference_p_value_correction_multi_measure(): + """Test Difference.p_value multiple-comparison correction is applied per measure""" + from statsmodels.stats.multitest import multipletests + from CompStats.interface import Perf + from CompStats.metrics import f1_score, recall_score + + X, y = load_digits(return_X_y=True) + _ = train_test_split(X, y, test_size=0.3, random_state=0) + X_train, X_val, y_train, y_val = _ + ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) + nb = GaussianNB().fit(X_train, y_train) + svm = LinearSVC().fit(X_train, y_train) + perf = Perf(y_val, ens.predict(X_val), nb=nb.predict(X_val), + svm=svm.predict(X_val), + func=[f1_score.measure(average='macro'), + recall_score.measure(average='macro')], + num_samples=20) + diff = perf.difference() + raw = diff.p_value() + corrected = diff.p_value(correction='bonferroni') + keys = list(raw.keys()) + for col in range(2): + expected = multipletests([raw[k][col] for k in keys], + method='bonferroni')[1] + actual = [corrected[k][col] for k in keys] + assert np.allclose(actual, expected) + for k in keys: + assert np.all(corrected[k] >= raw[k] - 1e-12) + + +def test_Difference_dataframe_correction_changes_significant_flag(): + """Test that correcting p-values only makes Significant more conservative""" + from CompStats.metrics import f1_score + + X, y = load_digits(return_X_y=True) + _ = train_test_split(X, y, test_size=0.3, random_state=0) + X_train, X_val, y_train, y_val = _ + ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) + nb = GaussianNB().fit(X_train, y_train) + svm = LinearSVC().fit(X_train, y_train) + score = f1_score(y_val, ens.predict(X_val), average='macro', + num_samples=50) + score(nb.predict(X_val)) + score(svm.predict(X_val)) + diff = score.difference() + n_significant = diff.dataframe()['Significant'].sum() + n_significant_corrected = diff.dataframe( + correction='bonferroni')['Significant'].sum() + assert n_significant_corrected <= n_significant + + +def test_Perf_dataframe_correction_changes_comparison_legend(): + """Test that Perf.dataframe's Comparison legend reflects corrected p-values""" + from CompStats.metrics import f1_score + + X, y = load_digits(return_X_y=True) + _ = train_test_split(X, y, test_size=0.3, random_state=0) + X_train, X_val, y_train, y_val = _ + ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) + nb = GaussianNB().fit(X_train, y_train) + svm = LinearSVC().fit(X_train, y_train) + score = f1_score(y_val, ens.predict(X_val), average='macro', + num_samples=50) + score(nb.predict(X_val)) + score(svm.predict(X_val)) + uncorrected = score.dataframe(comparison=True) + corrected = score.dataframe(comparison=True, correction='bonferroni') + n_different = (uncorrected['Comparison'] == 'Different').sum() + n_different_corrected = (corrected['Comparison'] == 'Different').sum() + assert n_different_corrected <= n_different diff --git a/CompStats/tests/test_measurements.py b/CompStats/tests/test_measurements.py index bc8c9b7..fcfa3eb 100644 --- a/CompStats/tests/test_measurements.py +++ b/CompStats/tests/test_measurements.py @@ -21,6 +21,7 @@ DATA = os.path.join(os.path.dirname(__file__), 'data.csv') + def test_CI(): """Test confidence interval""" diff --git a/CompStats/tests/test_metrics.py b/CompStats/tests/test_metrics.py index 2a3b5be..5863ae2 100644 --- a/CompStats/tests/test_metrics.py +++ b/CompStats/tests/test_metrics.py @@ -39,6 +39,44 @@ def test_f1_score(): assert str(perf) is not None +def test_f1_score_difference_p_value_correction(): + """Test Difference.p_value(correction=...) through the f1_score wrapper""" + import numpy as np + from statsmodels.stats.multitest import multipletests + from CompStats.metrics import f1_score + + X, y = load_iris(return_X_y=True) + _ = train_test_split(X, y, test_size=0.3, random_state=0) + X_train, X_val, y_train, y_val = _ + ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) + nb = GaussianNB().fit(X_train, y_train) + perf = f1_score(y_val, forest=ens.predict(X_val), + nb=nb.predict(X_val), average='macro', + num_samples=50) + diff = perf.difference() + raw = diff.p_value() + expected = multipletests(list(raw.values()), method='bonferroni')[1] + corrected = diff.p_value(correction='bonferroni') + assert list(corrected.keys()) == list(raw.keys()) + assert np.allclose(list(corrected.values()), expected) + + +def test_f1_score_plot_correction(): + """Test Perf.plot(correction=...) runs through the f1_score wrapper""" + from CompStats.metrics import f1_score + + X, y = load_iris(return_X_y=True) + _ = train_test_split(X, y, test_size=0.3, random_state=0) + X_train, X_val, y_train, y_val = _ + ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) + nb = GaussianNB().fit(X_train, y_train) + perf = f1_score(y_val, forest=ens.predict(X_val), + nb=nb.predict(X_val), average='macro', + num_samples=50) + f_grid = perf.plot(correction='bonferroni') + assert f_grid is not None + + def test_macro_f1_score(): """Test f1_score""" from CompStats.metrics import macro_f1 @@ -51,7 +89,7 @@ def test_macro_f1_score(): perf = macro_f1(y_val, forest=hy, num_samples=50) assert isinstance(perf.statistic, float) _ = metrics.f1_score(y_val, hy, average='macro') - assert _ == perf.statistic + assert _ == perf.statistic def test_accuracy_score(): @@ -110,8 +148,8 @@ def test_average_precision_score(): ens = RandomForestClassifier().fit(X_train, y_train) hy = ens.predict_proba(X_val) perf = average_precision_score(y_val, - forest=hy, - num_samples=50) + forest=hy, + num_samples=50) _ = metrics.average_precision_score(y_val, hy) assert _ == perf.statistic @@ -195,8 +233,8 @@ def test_recall_score(): ens = RandomForestClassifier().fit(X_train, y_train) hy = ens.predict(X_val) perf = recall_score(y_val, - forest=hy, - num_samples=50, average='macro') + forest=hy, + num_samples=50, average='macro') _ = metrics.recall_score(y_val, hy, average='macro') assert _ == perf.statistic @@ -264,8 +302,8 @@ def test_d2_log_loss_score(): ens = RandomForestClassifier().fit(X_train, y_train) hy = ens.predict_proba(X_val) perf = d2_log_loss_score(y_val, - forest=hy, - num_samples=50) + forest=hy, + num_samples=50) _ = metrics.d2_log_loss_score(y_val, hy) assert _ == perf.statistic @@ -328,8 +366,8 @@ def test_mean_squared_error(): ens = RandomForestRegressor().fit(X_train, y_train) hy = ens.predict(X_val) perf = mean_squared_error(y_val, - forest=hy, - num_samples=50) + forest=hy, + num_samples=50) _ = metrics.mean_squared_error(y_val, hy) assert _ == perf.statistic @@ -360,8 +398,8 @@ def test_mean_squared_log_error(): ens = RandomForestRegressor().fit(X_train, y_train) hy = ens.predict(X_val) perf = mean_squared_log_error(y_val, - forest=hy, - num_samples=50) + forest=hy, + num_samples=50) _ = metrics.mean_squared_log_error(y_val, hy) assert _ == perf.statistic @@ -392,8 +430,8 @@ def test_median_absolute_error(): ens = RandomForestRegressor().fit(X_train, y_train) hy = ens.predict(X_val) perf = median_absolute_error(y_val, - forest=hy, - num_samples=50) + forest=hy, + num_samples=50) _ = metrics.median_absolute_error(y_val, hy) assert _ == perf.statistic @@ -441,10 +479,10 @@ def test_mean_gamma_deviance(): ens = RandomForestRegressor().fit(X_train, y_train) hy = ens.predict(X_val) perf = mean_gamma_deviance(y_val, - forest=hy, - num_samples=50) + forest=hy, + num_samples=50) _ = metrics.mean_gamma_deviance(y_val, hy) - assert _ == perf.statistic + assert _ == perf.statistic def test_mean_absolute_percentage_error(): @@ -498,7 +536,7 @@ def test_pearsonr(): def test_measure_factories_tag_bib(): """Every wrapper's .measure() factory returns a callable tagged with the - same direction (BiB) implied by its wrapper's score_func/error_func""" + same direction (BiB) the wrapper itself passes as func""" from CompStats import metrics as compstats_metrics score_type = ['accuracy_score', 'balanced_accuracy_score', @@ -535,15 +573,15 @@ def test_measure_compose_multi_metric_perf(): ens = RandomForestClassifier(random_state=0).fit(X_train, y_train) nb = GaussianNB().fit(X_train, y_train) perf = Perf(y_val, ens.predict(X_val), nb=nb.predict(X_val), - score_func=[f1_score.measure(average='macro'), - recall_score.measure(average='macro')], - num_samples=20) + func=[f1_score.measure(average='macro'), + recall_score.measure(average='macro')], + num_samples=20) assert perf.measure_names == ['f1_score', 'recall_score'] assert perf.statistic['alg-1'].shape == (2,) # error-type measure composed together with a score-type one perf2 = Perf(y_val, ens.predict(X_val), nb=nb.predict(X_val), - score_func=f1_score.measure(average='macro'), - error_func=mean_absolute_error.measure(), - num_samples=20) + func=[f1_score.measure(average='macro'), + mean_absolute_error.measure()], + num_samples=20) assert list(perf2.statistic_samples.BiB) == [True, False] diff --git a/CompStats/tests/test_performance.py b/CompStats/tests/test_performance.py index 2eb88d8..f2d7db2 100644 --- a/CompStats/tests/test_performance.py +++ b/CompStats/tests/test_performance.py @@ -22,7 +22,6 @@ from CompStats import plot_difference2, plot_difference_multiple - DATA = os.path.join(os.path.dirname(__file__), 'data.csv') @@ -33,7 +32,7 @@ def test_performance(): assert 'BoW' in perf.calls assert 'y' not in perf.calls assert perf.n_jobs == -1 - + def test_plot_performance(): """Test plot_performance""" @@ -71,7 +70,7 @@ def test_performance_multiple_metrics(): {"func": f1_score, "args": {"average": "macro"}, 'BiB': True}, {"func": precision_score, "args": {"average": "macro"}, 'BiB': True}, {"func": mean_absolute_error, 'BiB': False} - ] + ] perf = performance_multiple_metrics(df, "y", metrics) plot_performance_multiple(perf) assert 'accuracy_score' in perf['samples'] @@ -87,7 +86,7 @@ def test_difference_multiple(): {"func": f1_score, "args": {"average": "macro"}, 'BiB': True}, {"func": precision_score, "args": {"average": "macro"}, 'BiB': True}, {"func": mean_absolute_error, 'BiB': False} - ] + ] perf = performance_multiple_metrics(df, "y", metrics) diff = difference_multiple(perf) plot_difference_multiple(diff) @@ -104,7 +103,7 @@ def test_difference_summary(): {"func": f1_score, "args": {"average": "macro"}, 'BiB': True}, {"func": precision_score, "args": {"average": "macro"}, 'BiB': True}, {"func": mean_absolute_error, 'BiB': False} - ] + ] perf = performance_multiple_metrics(df, "y", metrics) diff = difference_multiple(perf) all_dif = all_differences_multiple(diff) diff --git a/CompStats/utils.py b/CompStats/utils.py index 735c981..e5d9b56 100644 --- a/CompStats/utils.py +++ b/CompStats/utils.py @@ -19,7 +19,7 @@ USE_TQDM = False -def progress_bar(arg, use_tqdm: bool=True, **kwargs): +def progress_bar(arg, use_tqdm: bool = True, **kwargs): """Wrap `arg` in a :py:class:`tqdm.tqdm` progress bar. Returns `arg` unchanged when tqdm is not installed or :py:attr:`use_tqdm` is @@ -37,7 +37,7 @@ def progress_bar(arg, use_tqdm: bool=True, **kwargs): return tqdm(arg, **kwargs) -def metrics_docs(hy_name='y_pred', attr_name='score_func'): +def metrics_docs(hy_name='y_pred', bib: bool = True): """Decorator that injects the shared :py:class:`~CompStats.interface.Perf` docstring into a :py:mod:`CompStats.metrics` wrapper (e.g. :py:func:`~CompStats.metrics.f1_score`). @@ -46,15 +46,18 @@ def metrics_docs(hy_name='y_pred', attr_name='score_func'): docstring (e.g. ``y_pred`` or ``y_score``, matching the wrapped :py:mod:`sklearn.metrics` function's own parameter name). :type hy_name: str - :param attr_name: Which :py:class:`~CompStats.interface.Perf` argument the - wrapped function's measure is passed as, ``score_func`` or ``error_func``. - :type attr_name: str + :param bib: Whether the wrapped function's measure is score-type (bigger + is better) or error-type (smaller is better); used only to describe + the measure's direction in the generated docstring, matching the + ``.BiB`` tag set on the wrapper's ``.measure`` factory. + :type bib: bool """ def perf_docs(func): """Decorator to Perf to write :py:class:`~sklearn.metrics` documentation""" - func.__doc__ = f""":py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.{func.__name__}` as :py:attr:`{attr_name}.` The parameters not described can be found in :py:func:`~sklearn.metrics.{func.__name__}`. + direction = 'score-type (bigger is better)' if bib else 'error-type (smaller is better)' + func.__doc__ = f""":py:class:`~CompStats.interface.Perf` with :py:func:`~sklearn.metrics.{func.__name__}` as a {direction} :py:attr:`func.` The parameters not described can be found in :py:func:`~sklearn.metrics.{func.__name__}`. :param y_true: True measurement or could be a pandas.DataFrame where column label 'y' corresponds to the true measurement. :type y_true: numpy.ndarray or pandas.DataFrame @@ -69,7 +72,7 @@ def perf_docs(func): :param use_tqdm: Whether to use tqdm.tqdm to visualize the progress, default=True :type use_tqdm: bool - :py:func:`~CompStats.metrics.{func.__name__}.measure` builds the tagged callable used internally as :py:attr:`{attr_name}`; call it directly (e.g. ``{func.__name__}.measure(...)``) to combine this metric with others into a single, multi-measure :py:class:`~CompStats.interface.Perf` -- see :py:class:`~CompStats.interface.Perf`'s class docstring for a worked example. + :py:func:`~CompStats.metrics.{func.__name__}.measure` builds the tagged callable used internally as :py:attr:`func`; call it directly (e.g. ``{func.__name__}.measure(...)``) to combine this metric with others into a single, multi-measure :py:class:`~CompStats.interface.Perf` -- see :py:class:`~CompStats.interface.Perf`'s class docstring for a worked example. """ + func.__doc__ @@ -81,10 +84,10 @@ def inner(*args, **kwargs): return perf_docs -def dataframe(instance, value_name:str='Score', - var_name:str='Performance', - alg_legend:str='Algorithm', - perf_names:list=None): +def dataframe(instance, value_name: str = 'Score', + var_name: str = 'Performance', + alg_legend: str = 'Algorithm', + perf_names: list = None): """Melt a :py:class:`~CompStats.interface.Perf` or :py:class:`~CompStats.interface.Difference` instance's bootstrap samples into a long-format :py:class:`pandas.DataFrame`, ready for seaborn's ``catplot`` @@ -111,7 +114,7 @@ def dataframe(instance, value_name:str='Score', if not isinstance(statistic, dict): iter = instance.statistic_samples.keys() else: - iter = statistic + iter = statistic if isinstance(instance.best, str): calls = instance.statistic_samples.calls df = pd.DataFrame({k: calls[k] @@ -126,4 +129,4 @@ def dataframe(instance, value_name:str='Score', var_name=var_name) _df[alg_legend] = key df = pd.concat((df, _df)) - return df \ No newline at end of file + return df diff --git a/README.rst b/README.rst index 82da3ba..58231b9 100644 --- a/README.rst +++ b/README.rst @@ -49,7 +49,7 @@ Once the predictions are available, it is time to measure the algorithm's perfor >>> score = f1_score(y_val, hy, average='macro') >>> score - + The previous code shows the macro-f1 score and its standard error. The actual performance value is stored in the attributes `statistic` function, and `se` @@ -60,7 +60,7 @@ Continuing with the example, let us assume that one wants to test another classi >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> score(ens.predict(X_val), name='Random Forest') - + Statistic with its standard error (se) statistic (se) 0.9720 (0.0076) <= Random Forest @@ -72,7 +72,7 @@ Let us incorporate another predictions, now with Naive Bayes classifier, and His >>> score(nb.predict(X_val), name='Naive Bayes') >>> hist = HistGradientBoostingClassifier().fit(X_train, y_train) >>> score(hist.predict(X_val), name='Hist. Grad. Boost. Tree') - + Statistic with its standard error (se) statistic (se) 0.9759 (0.0068) <= Hist. Grad. Boost. Tree diff --git a/docs/CompStats_metrics.ipynb b/docs/CompStats_metrics.ipynb index 28c8838..ef3556f 100644 --- a/docs/CompStats_metrics.ipynb +++ b/docs/CompStats_metrics.ipynb @@ -157,7 +157,7 @@ "output_type": "execute_result", "data": { "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -235,7 +235,7 @@ "output_type": "execute_result", "data": { "text/plain": [ - "\n", + "\n", "Statistic with its standard error (se)\n", "statistic (se)\n", "0.9720 (0.0076) <= Random Forest\n", @@ -284,7 +284,7 @@ "output_type": "execute_result", "data": { "text/plain": [ - "\n", + "\n", "Statistic with its standard error (se)\n", "statistic (se)\n", "0.9759 (0.0068) <= Hist. Grad. Boost. Tree\n", diff --git a/docs/source/conf.py b/docs/source/conf.py index 786eb38..89222ca 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -45,10 +45,10 @@ 'sphinx.ext.coverage', 'sphinx.ext.mathjax', 'sphinx.ext.viewcode', - 'sphinx.ext.intersphinx' + 'sphinx.ext.intersphinx' ] -# intersphinx_mapping = {} +#  intersphinx_mapping = {} intersphinx_mapping = {'sklearn': ('https://scikit-learn.org/stable/', None)} # Add any paths that contain templates here, relative to this directory. @@ -105,7 +105,7 @@ # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -# html_static_path = ['_static'] +#  html_static_path = ['_static'] # Custom sidebar templates, must be a dictionary that maps document names # to template names. diff --git a/docs/source/metrics_api.rst b/docs/source/metrics_api.rst index 0f0e8d5..39da543 100644 --- a/docs/source/metrics_api.rst +++ b/docs/source/metrics_api.rst @@ -49,7 +49,7 @@ Once the predictions are available, it is time to measure the algorithm's perfor >>> score = f1_score(y_val, hy, average='macro') >>> score - + The previous code shows the macro-f1 score and, in parenthesis, its standard error. The actual performance value is stored in the attributes :py:func:`~CompStats.interface.Perf.statistic` and :py:func:`~CompStats.interface.Perf.se` @@ -60,7 +60,7 @@ Continuing with the example, let us assume that one wants to test another classi >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> score(ens.predict(X_val), name='Random Forest') - + Statistic with its standard error (se) statistic (se) 0.9720 (0.0076) <= Random Forest @@ -72,7 +72,7 @@ Let us incorporate another predictions, now with Naive Bayes classifier, and His >>> score(nb.predict(X_val), name='Naive Bayes') >>> hist = HistGradientBoostingClassifier().fit(X_train, y_train) >>> score(hist.predict(X_val), name='Hist. Grad. Boost. Tree') - + Statistic with its standard error (se) statistic (se) 0.9759 (0.0068) <= Hist. Grad. Boost. Tree @@ -105,18 +105,18 @@ The class :py:class:`~CompStats.Difference` has the :py:class:`~CompStats.Differ Multi-measure Perf -------------------- -A single competition can also be evaluated with more than one measure at once (e.g., macro-F1 together with macro-recall) by passing a list of functions to :py:attr:`score_func`/:py:attr:`error_func`. :py:attr:`score_func` and :py:attr:`error_func` can even be combined to mix score-type and error-type measures, with different Bigger-is-Better (BiB) directions, into a single :py:class:`~CompStats.interface.Perf` instance. Every measure is evaluated on the same bootstrap resamples, so comparisons across algorithms remain paired for each measure. +A single competition can also be evaluated with more than one measure at once (e.g., macro-F1 together with macro-recall) by passing a list of functions to :py:attr:`func`. Score-type and error-type measures, with different Bigger-is-Better (BiB) directions, can even be combined this way into a single :py:class:`~CompStats.interface.Perf` instance, as long as each callable is tagged with its own :py:attr:`BiB` (or a matching list is passed to :py:attr:`BiB`). Every measure is evaluated on the same bootstrap resamples, so comparisons across algorithms remain paired for each measure. -Every :py:mod:`CompStats.metrics` wrapper exposes a ``.measure`` factory (e.g. :py:func:`~CompStats.metrics.f1_score.measure`) that builds the tagged callable used internally as :py:attr:`score_func`/:py:attr:`error_func`; call it directly to compose several measures, as shown next. +Every :py:mod:`CompStats.metrics` wrapper exposes a ``.measure`` factory (e.g. :py:func:`~CompStats.metrics.f1_score.measure`) that builds the tagged callable used internally as :py:attr:`func`; call it directly to compose several measures, as shown next. >>> from CompStats.interface import Perf >>> from CompStats.metrics import f1_score, recall_score >>> mperf = Perf(y_val, hy, forest=ens.predict(X_val), -... score_func=[f1_score.measure(average='macro'), -... recall_score.measure(average='macro')], +... func=[f1_score.measure(average='macro'), +... recall_score.measure(average='macro')], ... measure_names=['macro-F1', 'macro-Recall']) >>> mperf - + Statistic with its standard error (se) statistic (se) 0.9783 (0.0061), 0.9786 (0.0060) <= forest @@ -145,8 +145,8 @@ The convenience wrappers :py:func:`~CompStats.metrics.macro_f1`, :py:func:`~Comp >>> from CompStats.metrics import macro_f1, macro_recall >>> Perf(y_val, hy, forest=ens.predict(X_val), -... score_func=[macro_f1.measure(), macro_recall.measure()]) - +... func=[macro_f1.measure(), macro_recall.measure()]) + Statistic with its standard error (se) statistic (se) 0.9783 (0.0061), 0.9786 (0.0060) <= forest diff --git a/setup.py b/setup.py index fc1f76c..6068493 100644 --- a/setup.py +++ b/setup.py @@ -1,3 +1,3 @@ from setuptools import setup -setup() \ No newline at end of file +setup()