Skip to content

fix(onehot): categorical_method onehot_from_ordinal fails on strings #17

Description

@ChrisW09

Summary

Preprocessor(categorical_method="onehot_from_ordinal") crashes on any string categorical
column with a bare ValueError from the numpy internals. The Preprocessor docstring
describes this method as "integer codes then one-hot", but the pipeline never inserts the
integer-coding step — it appends OneHotFromOrdinalTransformer alone, which requires input
that is already ordinal-encoded.

Reproduction

import pandas as pd
from pretab import Preprocessor

df = pd.DataFrame({"c": ["a", "b", "c"] * 30})
Preprocessor(categorical_method="onehot_from_ordinal").fit(df, None)
ValueError: invalid literal for int() with base 10: 'c'

The traceback surfaces from np.max(X, axis=0).astype(int) in
OneHotFromOrdinalTransformer.fit; nothing in the message points at the chosen
categorical_method.

Expected

One of:

  • the method encodes strings as integer codes and then one-hots them, matching the
    Preprocessor docstring; or
  • it raises a pretab error explaining that the column must already be ordinal-encoded and
    suggesting categorical_method="one-hot", and the docstring is corrected.

Actual

An unhandled ValueError from numpy with no indication of the cause or the fix.

Root cause

Docs disagree with each other, and the implementation follows the narrower reading:

  • pretab/preprocessor.py:56"onehot_from_ordinal (integer codes then one-hot)"
  • docs/user_guide/preprocessing.md:80"One-hot from pre-encoded ordinals" (correct)
  • README.md:107"One-hot on pre-encoded categoricals" (correct)

pretab/pipeline/categorical.py:51-52 builds the pipeline as:

elif method == "onehot_from_ordinal":
    steps.append(("onehot_from_ordinal", OneHotFromOrdinalTransformer()))

No ContinuousOrdinalTransformer ahead of it, so OneHotFromOrdinalTransformer.fit
(pretab/transformers/onehot/onehot.py:49-51) runs np.max(X, axis=0).astype(int) on
string data and dies.

Note the class itself is fine and correctly documented — the defect is the pipeline wiring
plus the Preprocessor docstring.

Suggested fix

Preferred, because it makes the documented behaviour true and the method actually useful
from the Preprocessor (where columns arrive raw):

elif method == "onehot_from_ordinal":
    steps.append(("continuous_ordinal", ContinuousOrdinalTransformer()))
    steps.append(("onehot_from_ordinal", OneHotFromOrdinalTransformer()))

ContinuousOrdinalTransformer numbers categories from 1 and reserves 0 for unseen values,
so max_bins_ becomes n_categories + 1 and unseen categories at transform time land in
the reserved 0 column — reasonable, but worth an explicit test.

Alternative, if the method is meant to stay strictly "already ordinal": leave the pipeline
as is, add a numeric-input guard in OneHotFromOrdinalTransformer.fit that raises
PretabDataError with a pointer to "one-hot", and fix pretab/preprocessor.py:56 to
match the user guide and README.

Either way pretab/preprocessor.py:56 needs updating so the three descriptions agree.

Impact

Preprocessor(categorical_method="onehot_from_ordinal") on any string column — which is the
common case, since _detect_column_types routes non-numeric columns to the categorical
side. Fails hard, so no silent data corruption.

Environment

  • pretab 0.1.0 (main @ 51c3043)
  • Python 3.11.15, numpy 2.4.6, pandas 2.3.3, scikit-learn 1.9.0, scipy 1.17.1
  • macOS (darwin 25.5.0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions