Skip to content

perf: optimization of stats code for rank_genes_groups#4204

Open
zboldyga wants to merge 10 commits into
scverse:mainfrom
zboldyga:zb/rank-genes-groups-stats
Open

perf: optimization of stats code for rank_genes_groups#4204
zboldyga wants to merge 10 commits into
scverse:mainfrom
zboldyga:zb/rank-genes-groups-stats

Conversation

@zboldyga

@zboldyga zboldyga commented Jul 8, 2026

Copy link
Copy Markdown
Contributor
  • Closes #
  • Tests included or not required because:

@ilan-gold replayed changes of #4118 onto main

@zboldyga

zboldyga commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@ilan-gold regarding these points, perhaps I open separate PR(s):

  • addressing the scanpy 2.0 change in how cells with group NaN/unassigned are handled (uniform treatment across aggregate, rank_genes_groups, anywhere else relevant?, + docs update for this).

  • adding benchmark code for rank_genes_groups paths that are fast-enough to run quickly (e.g. < 1 min).

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.16981% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.95%. Comparing base (c52da39) to head (3062104).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/scanpy/tools/_rank_genes_groups.py 97.16% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4204      +/-   ##
==========================================
+ Coverage   79.93%   79.95%   +0.01%     
==========================================
  Files         121      121              
  Lines       12958    12985      +27     
==========================================
+ Hits        10358    10382      +24     
- Misses       2600     2603       +3     
Flag Coverage Δ
hatch-test.low-vers 79.05% <97.16%> (-0.02%) ⬇️
hatch-test.pre 79.91% <97.16%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/scanpy/tools/_rank_genes_groups.py 93.81% <97.16%> (-0.36%) ⬇️

@zboldyga zboldyga marked this pull request as ready for review July 9, 2026 01:23
@flying-sheep flying-sheep changed the title performance optimization of stats code for rank_genes_groups perf: optimization of stats code for rank_genes_groups Jul 13, 2026
@ilan-gold ilan-gold added this to the 1.12.3 milestone Jul 14, 2026
@scverse-benchmark

Copy link
Copy Markdown

Benchmark changes

Change Before [c52da39] After [3062104] Ratio Benchmark (Parameter)
- 7.70±0s 4.75±0s 0.62 preprocessing_counts.RankGenesGroups.time_agg('ovo', True)
- 10.5±0.02s 7.26±0s 0.69 preprocessing_counts.RankGenesGroups.time_agg('ovr', False)
- 14.4±0.08s 5.88±0s 0.41 preprocessing_counts.RankGenesGroups.time_agg('ovr', True)

Comparison: https://github.com/scverse/scanpy/compare/c52da398956c8de9a5bc1945a7966ad86b8431be..30621045a30c46a6e2227ea1ac2a60604086b3ce
Last changed: Tue, 14 Jul 2026 16:12:27 +0000

More details: https://github.com/scverse/scanpy/pull/4204/checks?check_run_id=87113112149

return expm1_func(x)


@numba.njit # noqa: TID251 (inner kernel called from _vars_rest's nopython loop)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't fully understand this - could you explain more? Why do we need to use numba.njit here?

n_r, m2_r = stats_after_g[0], stats_after_g[2]
denom = n_r - 1.0 if n_r >= 2.0 else 1.0
v = m2_r / denom
vars_rest[g, j] = v if v > 0.0 else 0.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
vars_rest[g, j] = v if v > 0.0 else 0.0
vars_rest[g, j] = v

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not?

else:
# g == 0 has no groups before it, so its rest is just the groups after
n_r, m2_r = stats_after_g[0], stats_after_g[2]
denom = n_r - 1.0 if n_r >= 2.0 else 1.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
denom = n_r - 1.0 if n_r >= 2.0 else 1.0
denom = n_r - 1.0

why not?

self.stats[group_name, "names"] = self.var_names[global_indices]

self.stats[group_name, "scores"] = scores[global_indices]
def _build_stats_dataframe(

@ilan-gold ilan-gold Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this function just copy-pasted?

@ilan-gold

Copy link
Copy Markdown
Contributor

P.S Any thought why the csr-ovo case didn't improve?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants