Skip to content

Add CLI filtering flags to aggregation-helper - #703

Open
SandeepTuniki wants to merge 1 commit into
masterfrom
aggregation-filtering-flags
Open

Add CLI filtering flags to aggregation-helper#703
SandeepTuniki wants to merge 1 commit into
masterfrom
aggregation-filtering-flags

Conversation

@SandeepTuniki

Copy link
Copy Markdown
Contributor

This PR adds the following filtering flags to aggregation-helper:

  • select
  • reject
  • include_disabled

The select/reject filters are used to filter calculation steps by calculation enum type or import name using explicit prefixes (type: or import:).

The --include_disabled flag (defaults to false), when set to true, allows calculation steps marked disabled: true to be executed if they match active selection filters. This is added as a quick override during runtime without having to update config files.

Usage Examples

1. Run only Place Aggregations for an import (even if disabled: true in YAML)

python main.py --import_list '["CensusACS5YearSurvey"]' \
  --select "type:PLACE_AGGREGATION" \
  --include_disabled

2. Run Place & StatVar Aggregations, but skip a specific import

python main.py --import_list '["CensusACS5YearSurvey"]' \
  --select "type:PLACE_AGGREGATION,type:STAT_VAR_AGGREGATION" \
  --reject "import:CensusACS5YearSurvey_AggCountry" \
  --include_disabled

3. Run all enabled aggregations except Linked Edges and Provenance Summary

python main.py --import_list '["CensusACS5YearSurvey"]' \
  --reject "type:LINKED_EDGES,type:PROVENANCE_SUMMARY"

4. Test filter combinations safely using --dry_run

Logs the selected and rejected calculation stages without submitting BigQuery jobs:

python main.py --import_list '["CensusACS5YearSurvey"]' \
  --select "type:PLACE_AGGREGATION" \
  --include_disabled \
  --dry_run

…ggregation-helper

Implement dynamic selection and rejection of aggregation calculation steps by calculation type (type:) or import name (import:), alongside --include_disabled to allow running disabled YAML steps without modifying configs/*.yaml.
@codacy-production

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 1 medium · 5 minor

Alerts:
⚠ 6 issues (≤ 0 issues of at least minor severity)

Results:
6 new issues

Category Results
CodeStyle 5 minor
Complexity 1 medium

View in Codacy

🟢 Metrics 41 complexity

Metric Results
Complexity 41

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces command-line filtering flags (--select, --reject, and --include_disabled) to dynamically filter calculation steps by type or import name in the aggregation orchestrator, avoiding manual edits to YAML configuration files. The feedback identifies an issue where global calculations (which are import-independent) would be incorrectly filtered out when using import-based selection filters, and suggests bypassing import-based filters for global calculations.

Comment on lines +635 to 636
if self.config.select_types and calc_type not in self.config.select_types:
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Global calculations (such as EMBEDDING_GENERATION) are import-independent and do not have associated imports in their configuration. Currently, if --select is used with an import filter (e.g., --select import:CensusACS5YearSurvey), global calculations will be incorrectly filtered out because they do not match the selected import.

To prevent this, we should bypass import-based selection and rejection filters for global calculations by returning True early if the calculation type is in GLOBAL_CALCULATION_TYPES (after checking type-based filters).

Suggested change
if self.config.select_types and calc_type not in self.config.select_types:
return False
if self.config.select_types and calc_type not in self.config.select_types:
return False
if calc_type in GLOBAL_CALCULATION_TYPES:
return True

@SandeepTuniki
SandeepTuniki requested a review from vish-cs July 31, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant