Skip to content

Add CodSpeed performance benchmarks and CI workflow - #984

Merged
solegalli merged 3 commits into
mainfrom
codspeed-wizard-1785415799557
Jul 31, 2026
Merged

Add CodSpeed performance benchmarks and CI workflow#984
solegalli merged 3 commits into
mainfrom
codspeed-wizard-1785415799557

Conversation

@codspeed-hq

@codspeed-hq codspeed-hq Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What this adds

Continuous performance measurement for Feature-engine with CodSpeed, so performance regressions are caught on pull requests instead of after a release.

Benchmark suite (benchmarks/)

109 benchmarks written with pytest-codspeed, one module per transformer family. fit and transform are benchmarked separately because their performance profiles are very different (learning parameters vs. applying them):

File Covers
test_imputation.py Mean/median, arbitrary, end tail, categorical and random sample imputers, missing indicator, drop missing data
test_encoding.py Count, ordinal, mean (with and without smoothing), WoE, one-hot, rare label, decision tree and string similarity encoders
test_discretisation.py Equal frequency, equal width, geometric width, arbitrary and decision tree discretisers
test_outliers.py Winsoriser (gaussian, iqr, quantiles, mad), arbitrary capper, outlier trimmer
test_transformation.py Log, power, reciprocal, Box-Cox, Yeo-Johnson, arcsin, arcsinh, mean normalisation
test_creation.py Math, relative, cyclical and decision tree features
test_datetime.py Datetime feature extraction (including parsing dates stored as strings), subtraction, ordinal
test_timeseries.py Lag, window and expanding window features
test_selection.py Constant, duplicate, correlated, high PSI, information value, target encoding and single feature performance selectors
test_variable_handling.py The find_*/check_* helpers, which run inside every transformer's fit
test_pipeline.py Two end-to-end pipelines plus MatchVariables and MatchCategories

Details worth noting:

  • Data is synthetic and built in session-scoped conftest.py fixtures (numerical, strictly positive, bounded, categorical with rare labels, and datetime variables, with a variant containing missing values). Data generation therefore never lands inside a measured section.
  • Three dataframe sizes are used so every benchmark stays in the millisecond range under CPU simulation: 10k rows for the vectorised transformers, 1k rows for the ones that train models under the hood, 500 rows for the row-wise ones (MathFeatures).
  • Only non-deprecated public classes are used, so the suite produces no FutureWarning and will not break at the 2.1.0 removals.

CI workflow (.github/workflows/codspeed.yml)

Runs on pushes to main, on pull requests, and on workflow_dispatch (needed by CodSpeed to backfill initial data). It uses CodSpeedHQ/action@v5 in simulation mode on ubuntu-latest, with OIDC authentication (id-token: write), so no token secret is required.

Simulation mode was chosen because the benchmarks are CPU-bound: it gives deterministic, hardware-agnostic measurements with under 1% variance and produces flame graphs for each benchmark.

Other changes

  • pyproject.toml: added testpaths = ["tests"] to the pytest configuration. Without it, a bare pytest (as used by the codecov tox environment) would also collect benchmarks/, which needs pytest-codspeed installed. Running pytest benchmarks/ explicitly still works.
  • README.md: added the CodSpeed badge to the Testing row.

Verification

The suite was run locally in the same mode as CI:

codspeed run --mode simulation -- pytest benchmarks/ --codspeed
109 passed in 816.36s

All 109 benchmarks pass, and a plain pytest benchmarks/ --codspeed walltime run completes in about 7 minutes. Expect the CI job to take roughly 15 minutes, most of it CPU simulation overhead. flake8, black and isort were run on the new files.

Next steps

  • Merge this PR so the main branch gets a performance baseline. Until then, PR reports have nothing to compare against.
  • Once a baseline exists, CodSpeed comments on each PR with the benchmarks that moved, and flame graphs are available per benchmark to see where the time goes.
  • Extend the suite as needed: add a benchmark next to the relevant module whenever a transformer's performance matters, following the guidance in benchmarks/README.md.

@codspeed-hq

codspeed-hq Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

@codspeed-hq
codspeed-hq Bot marked this pull request as ready for review July 30, 2026 14:29
@codspeed-hq
codspeed-hq Bot requested a review from solegalli July 30, 2026 14:31
@solegalli

solegalli commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

closes #964 #592 #622

Add a concurrency group so pushing new commits to an open PR cancels
the previous, still-running benchmark job instead of letting both run
to completion (~15 min each under CPU simulation). Push events to main
are excluded from cancellation, since each one records a new CodSpeed
baseline and losing one would create a gap in the history.
Only run the CodSpeed job on pull requests once a maintainer adds the
runcodespeed label, after reviewing the PR. It keeps re-running on
further pushes to that PR for as long as the label stays attached;
removing the label goes back to no auto-runs. Doc-only changes
(docs/**, *.md, *.rst, the JOSS paper) are excluded from the trigger
entirely, since they can't affect performance.

Pushes to main and workflow_dispatch runs are unaffected: both still
run unconditionally, since every main push records a CodSpeed
baseline.
@solegalli
solegalli merged commit 1629bff into main Jul 31, 2026
14 checks passed
@solegalli
solegalli deleted the codspeed-wizard-1785415799557 branch July 31, 2026 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants