feat(dataflow): skip_stats_by_scope — per-scope stat skiplist; xorq skips histogram on filt+clean

[paddymul]

Summary

Adds a generic DataFlow.skip_stats_by_scope class attribute that lets a backend declare "don't run stat X when computing scope Y's SD". The motivating case (and only override shipping here): XorqDataflow skips histogram on the filt and clean scopes — on xorq the per-column histogram queries dominate state_change latency on remote tables, and only the bare (raw) histogram is rendered in merged_sd. Running the histogram on filt + clean too was wasted engine work that didn't show in the UI. Pandas / polars dataflows keep the default empty dict, so their filt/clean histograms continue to compute and render.

class XorqDataflow(CustomizableDataflow):
    skip_stats_by_scope = {'filt': {'histogram'}, 'clean': {'histogram'}}

Adapted from the working draft on smorg/post-785-playground (4949e56c), stripped of references to the closed #787 spike (recompute_summary_sd, _defer_summary_sd) and the not-yet-merged #809 (cost_classes).

Plumbing

skip_stat_names threads from the dataflow into the underlying pipelines:

• _summary_sd (filt scope) applies the filt skip when quick_command_args is truthy — can't use operations because the cascade hasn't updated it yet when this fires.
• _populate_sd_cache (raw + clean scopes) uses _effective_skip(scope, chains), which returns None when the scope is degenerate (no filter → filt == clean; no cleaning → clean == raw). Keeps the no-filter / no-cleaning case collapsing raw + clean + filt under one cache key — the lazy-postprocessor invariant on XorqBuckarooInfiniteWidget (TestLazyPostprocessor) still holds.
• _scope_cache_key incorporates the effective skip, so raw + filt cache entries don't collide when their skips differ.
• StatPipeline.process_column skips at iteration (stat-func name match) and strips at output (covers v1 ColAnalysis wrappers where one StatFunc provides many keys).
• XorqStatPipeline.process_table skips in both batch-aggregate and per-column phases — actual engine queries are elided, not just the output.
• DfStats (v1) accepts the kwarg for API parity and strips at output (v1 AnalysisPipeline has no per-stat filtering).

Tests

Failing-then-fix split per repo TDD policy. CI saw 5 failing tests on the first push (commit d79ad48e) and green on the fix (38d76795):

• test_skip_stats_by_scope_excludes_named_stat_from_filt — skip mean on filt; filtered_mean absent, bare mean present.
• test_skip_stats_by_scope_excludes_from_raw_and_clean — skip on raw/clean; bare mean absent, filtered_mean present.
• test_skip_stats_by_scope_default_empty_runs_all_stats — no override → unchanged.
• test_xorq_dataflow_skips_histogram_on_filt_and_clean — pins XorqDataflow's default.
• test_polars_dataflow_keeps_histogram — Polars keeps the default empty dict.

Full tests/unit/ suite green locally (1049 passed, 6 skipped) — including TestLazyPostprocessor which caught an early version of the cache-key change that re-ran big_a 3× because raw/clean/filt no longer shared a key.

Test plan

• Failing test commit pushed first; CI runs the new tests red
• Fix commit makes them pass
• Full unit suite green locally (1049 passed)
• Verify in JupyterLab against the boston restaurant data that filtered/cleaned histogram rows are gone and bare histogram still renders
• Measure state_change latency on boston-restaurant before/after — expect the ~6.5s histogram-query slice to drop to ~0.5s (per the 4949e56c commit message claim of ~13× speedup)

🤖 Generated with Claude Code

[github-actions]

📦 TestPyPI package published

pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.14.3.dev26252900166

or with uv:

uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.14.3.dev26252900166

MCP server for Claude Code

claude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.14.3.dev26252900166" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table

📖 Docs preview

🎨 Storybook preview

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38d7679568

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Determine filt-skip from realized filter ops

Use of bool(self.quick_command_args) here can misclassify "no filter" states as filtered. generate_quick_ops explicitly treats payloads like {'search': ['']} as no-op, so quick_command_args may be non-empty while no quick-command op exists; in that case _summary_sd still applies the filt skip, and because raw/clean/filt chains collapse to one key, _populate_sd_cache can store the skipped summary_sd under the shared key and skip recomputing raw, dropping bare stats (e.g. raw histogram) after clearing search.

Useful? React with 👍 / 👎.