Skip to content

fix(preprocessor): raise a clear error on duplicate column names #37

Description

@ChrisW09

Summary

Fitting on a DataFrame with duplicate column labels fails with an opaque
AttributeError: 'DataFrame' object has no attribute 'dtype' from deep inside
_detect_column_types, with nothing pointing at the real cause.

Reproduction

import numpy as np, pandas as pd
from pretab import Preprocessor

rng = np.random.default_rng(0)
dup = pd.DataFrame(np.column_stack([rng.normal(size=50)] * 2), columns=["a", "a"])
Preprocessor(numerical_method="minmax").fit(dup, rng.normal(size=50))
AttributeError: 'DataFrame' object has no attribute 'dtype'
  ... pretab/preprocessor.py, line 307, in _detect_column_types

Expected

A PretabDataError naming the duplicated labels and explaining that column names must be
unique — the ColumnTransformer this builds keys its transformers by column name, so
duplicates cannot be routed unambiguously even if detection were fixed.

Actual

AttributeError from numpy/pandas internals.

Root cause

pretab/preprocessor.py:288-312. The loop does X[col], which returns a DataFrame
rather than a Series when col is duplicated, so X[col].dtype does not exist. (nunique()
on the preceding line happens to work on a frame, which is why the failure surfaces on the
dtype access rather than earlier.)

Suggested fix

Check up front in fit / _detect_column_types, after the dict/ndarray coercion:

duplicated = X.columns[X.columns.duplicated()].unique().tolist()
if duplicated:
    raise PretabDataError(
        f"Duplicate column names are not supported: {duplicated}.\n"
        "Fix: rename the columns so every name is unique."
    )

Cheap to check and turns a confusing internal failure into an actionable one.

Environment

  • pretab 0.1.0 (main @ 51c3043)
  • Python 3.11.15, numpy 2.4.6, pandas 2.3.3, scikit-learn 1.9.0, scipy 1.17.1
  • macOS (darwin 25.5.0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions