Summary
Preprocessor(handle_missing="error") is documented to make missing values raise. For all
numerical methods except PLE it does not: the imputer is dropped, NaN flows straight through
the transformer, and the returned array silently contains NaN.
The Preprocessor docstring (pretab/preprocessor.py:118-123) says:
"error" drops that imputer so missing values are not silently filled and reach the
transformers, which then raise on NaN.
Only ple actually raises.
Reproduction
import warnings
import numpy as np, pandas as pd
from pretab import Preprocessor
rng = np.random.default_rng(0)
df = pd.DataFrame({"a": rng.normal(size=200)})
df.loc[:9, "a"] = np.nan
y = rng.normal(size=200)
for method in ("minmax", "rbf", "ple"):
try:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
pre = Preprocessor(numerical_method=method, handle_missing="error").fit(df, y)
out = pre.transform(df, return_array=True)
print(f"{method:7s}: no error; output all finite = {np.isfinite(out.astype(float)).all()}")
except Exception as e:
print(f"{method:7s}: raised {type(e).__name__} - {e}")
minmax : no error; output all finite = False
rbf : no error; output all finite = False
ple : raised ValueError - Input contains NaN.
For rbf the failure is worse than pass-through: the unsupervised placement path computes
centers with np.percentile over a column containing NaN
(pretab/transformers/feature_maps/_base.py:118-127), so every center becomes NaN and
the entire feature block is NaN, not just the ten affected rows.
Expected
With handle_missing="error", a NaN anywhere in the numerical input raises a clear
pretab error at fit/transform, for every method.
Actual
Only ple raises. Every other method returns NaN-contaminated output with no error and no
warning.
Root cause
handle_missing only controls whether the SimpleImputer step is added
(pretab/preprocessor.py:381 passes add_imputer=self.handle_missing != "error"), and is
forwarded as a constructor argument to exactly one transformer — ple — via
NUMERICAL_METHODS in pretab/pipeline/registry.py:63.
Everything else relies on the transformer noticing NaN itself, but the shared validator
pretab.core.validation.validate_2d_allow_nan is called with the transformer's own
_allow_nan class attribute, which is True for the PreTab transformers
(pretab/core/base.py:30), and the plain scikit-learn scalers (MinMaxScaler,
StandardScaler, RobustScaler, QuantileTransformer) deliberately ignore NaN by design.
Suggested fix
Make the policy enforced in one place rather than per-transformer. The simplest version is
an explicit validation step at the head of the numerical pipeline when
handle_missing == "error", in get_numerical_transformer_steps
(pretab/pipeline/numerical.py:121-123) — the same slot the imputer occupies today:
if add_imputer:
steps.append(("imputer", SimpleImputer(strategy=imputer_strategy, **imputer_kwargs)))
else:
steps.append(("nan_check", RaiseOnNaN())) # small transformer: check_array(..., ensure_all_finite=True)
That gives one consistent PretabDataError for every method and keeps the documented
contract literally true.
If the intent is instead that "error" merely means "do not impute", then the docstring at
pretab/preprocessor.py:118-123 should be corrected and the NaN pass-through documented,
because a silently NaN-filled feature matrix is a difficult failure to trace back to this
setting.
Either way, worth a test matrix over numerical_method asserting the chosen behaviour.
Impact
Preprocessor(handle_missing="error") with any numerical_method other than ple —
i.e. the default minmax scaling path and all feature maps and splines.
Environment
- pretab 0.1.0 (
main @ 51c3043)
- Python 3.11.15, numpy 2.4.6, pandas 2.3.3, scikit-learn 1.9.0, scipy 1.17.1
- macOS (darwin 25.5.0)
Summary
Preprocessor(handle_missing="error")is documented to make missing values raise. For allnumerical methods except PLE it does not: the imputer is dropped, NaN flows straight through
the transformer, and the returned array silently contains NaN.
The
Preprocessordocstring (pretab/preprocessor.py:118-123) says:Only
pleactually raises.Reproduction
For
rbfthe failure is worse than pass-through: the unsupervised placement path computescenters with
np.percentileover a column containing NaN(
pretab/transformers/feature_maps/_base.py:118-127), so every center becomes NaN andthe entire feature block is NaN, not just the ten affected rows.
Expected
With
handle_missing="error", a NaN anywhere in the numerical input raises a clearpretaberror atfit/transform, for every method.Actual
Only
pleraises. Every other method returns NaN-contaminated output with no error and nowarning.
Root cause
handle_missingonly controls whether theSimpleImputerstep is added(
pretab/preprocessor.py:381passesadd_imputer=self.handle_missing != "error"), and isforwarded as a constructor argument to exactly one transformer —
ple— viaNUMERICAL_METHODSinpretab/pipeline/registry.py:63.Everything else relies on the transformer noticing NaN itself, but the shared validator
pretab.core.validation.validate_2d_allow_nanis called with the transformer's own_allow_nanclass attribute, which isTruefor the PreTab transformers(
pretab/core/base.py:30), and the plain scikit-learn scalers (MinMaxScaler,StandardScaler,RobustScaler,QuantileTransformer) deliberately ignore NaN by design.Suggested fix
Make the policy enforced in one place rather than per-transformer. The simplest version is
an explicit validation step at the head of the numerical pipeline when
handle_missing == "error", inget_numerical_transformer_steps(
pretab/pipeline/numerical.py:121-123) — the same slot the imputer occupies today:That gives one consistent
PretabDataErrorfor every method and keeps the documentedcontract literally true.
If the intent is instead that
"error"merely means "do not impute", then the docstring atpretab/preprocessor.py:118-123should be corrected and the NaN pass-through documented,because a silently NaN-filled feature matrix is a difficult failure to trace back to this
setting.
Either way, worth a test matrix over
numerical_methodasserting the chosen behaviour.Impact
Preprocessor(handle_missing="error")with anynumerical_methodother thanple—i.e. the default
minmaxscaling path and all feature maps and splines.Environment
main@ 51c3043)