Skip to content

fix(preprocessor): handle_missing="error" lets NaN into the output #16

Description

@ChrisW09

Summary

Preprocessor(handle_missing="error") is documented to make missing values raise. For all
numerical methods except PLE it does not: the imputer is dropped, NaN flows straight through
the transformer, and the returned array silently contains NaN.

The Preprocessor docstring (pretab/preprocessor.py:118-123) says:

"error" drops that imputer so missing values are not silently filled and reach the
transformers, which then raise on NaN.

Only ple actually raises.

Reproduction

import warnings
import numpy as np, pandas as pd
from pretab import Preprocessor

rng = np.random.default_rng(0)
df = pd.DataFrame({"a": rng.normal(size=200)})
df.loc[:9, "a"] = np.nan
y = rng.normal(size=200)

for method in ("minmax", "rbf", "ple"):
    try:
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            pre = Preprocessor(numerical_method=method, handle_missing="error").fit(df, y)
            out = pre.transform(df, return_array=True)
        print(f"{method:7s}: no error; output all finite = {np.isfinite(out.astype(float)).all()}")
    except Exception as e:
        print(f"{method:7s}: raised {type(e).__name__} - {e}")
minmax : no error; output all finite = False
rbf    : no error; output all finite = False
ple    : raised ValueError - Input contains NaN.

For rbf the failure is worse than pass-through: the unsupervised placement path computes
centers with np.percentile over a column containing NaN
(pretab/transformers/feature_maps/_base.py:118-127), so every center becomes NaN and
the entire feature block is NaN, not just the ten affected rows.

Expected

With handle_missing="error", a NaN anywhere in the numerical input raises a clear
pretab error at fit/transform, for every method.

Actual

Only ple raises. Every other method returns NaN-contaminated output with no error and no
warning.

Root cause

handle_missing only controls whether the SimpleImputer step is added
(pretab/preprocessor.py:381 passes add_imputer=self.handle_missing != "error"), and is
forwarded as a constructor argument to exactly one transformer — ple — via
NUMERICAL_METHODS in pretab/pipeline/registry.py:63.

Everything else relies on the transformer noticing NaN itself, but the shared validator
pretab.core.validation.validate_2d_allow_nan is called with the transformer's own
_allow_nan class attribute, which is True for the PreTab transformers
(pretab/core/base.py:30), and the plain scikit-learn scalers (MinMaxScaler,
StandardScaler, RobustScaler, QuantileTransformer) deliberately ignore NaN by design.

Suggested fix

Make the policy enforced in one place rather than per-transformer. The simplest version is
an explicit validation step at the head of the numerical pipeline when
handle_missing == "error", in get_numerical_transformer_steps
(pretab/pipeline/numerical.py:121-123) — the same slot the imputer occupies today:

if add_imputer:
    steps.append(("imputer", SimpleImputer(strategy=imputer_strategy, **imputer_kwargs)))
else:
    steps.append(("nan_check", RaiseOnNaN()))   # small transformer: check_array(..., ensure_all_finite=True)

That gives one consistent PretabDataError for every method and keeps the documented
contract literally true.

If the intent is instead that "error" merely means "do not impute", then the docstring at
pretab/preprocessor.py:118-123 should be corrected and the NaN pass-through documented,
because a silently NaN-filled feature matrix is a difficult failure to trace back to this
setting.

Either way, worth a test matrix over numerical_method asserting the chosen behaviour.

Impact

Preprocessor(handle_missing="error") with any numerical_method other than ple
i.e. the default minmax scaling path and all feature maps and splines.

Environment

  • pretab 0.1.0 (main @ 51c3043)
  • Python 3.11.15, numpy 2.4.6, pandas 2.3.3, scikit-learn 1.9.0, scipy 1.17.1
  • macOS (darwin 25.5.0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions