Add CLI filtering flags to aggregation-helper - #703
Conversation
…ggregation-helper Implement dynamic selection and rejection of aggregation calculation steps by calculation type (type:) or import name (import:), alongside --include_disabled to allow running disabled YAML steps without modifying configs/*.yaml.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| CodeStyle | 5 minor |
| Complexity | 1 medium |
🟢 Metrics 41 complexity
Metric Results Complexity 41
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Code Review
This pull request introduces command-line filtering flags (--select, --reject, and --include_disabled) to dynamically filter calculation steps by type or import name in the aggregation orchestrator, avoiding manual edits to YAML configuration files. The feedback identifies an issue where global calculations (which are import-independent) would be incorrectly filtered out when using import-based selection filters, and suggests bypassing import-based filters for global calculations.
| if self.config.select_types and calc_type not in self.config.select_types: | ||
| return False |
There was a problem hiding this comment.
Global calculations (such as EMBEDDING_GENERATION) are import-independent and do not have associated imports in their configuration. Currently, if --select is used with an import filter (e.g., --select import:CensusACS5YearSurvey), global calculations will be incorrectly filtered out because they do not match the selected import.
To prevent this, we should bypass import-based selection and rejection filters for global calculations by returning True early if the calculation type is in GLOBAL_CALCULATION_TYPES (after checking type-based filters).
| if self.config.select_types and calc_type not in self.config.select_types: | |
| return False | |
| if self.config.select_types and calc_type not in self.config.select_types: | |
| return False | |
| if calc_type in GLOBAL_CALCULATION_TYPES: | |
| return True |
This PR adds the following filtering flags to aggregation-helper:
selectrejectinclude_disabledThe
select/rejectfilters are used to filter calculation steps by calculation enum type or import name using explicit prefixes (type:orimport:).The
--include_disabledflag (defaults tofalse), when set to true, allows calculation steps markeddisabled: trueto be executed if they match active selection filters. This is added as a quick override during runtime without having to update config files.Usage Examples
1. Run only Place Aggregations for an import (even if
disabled: truein YAML)2. Run Place & StatVar Aggregations, but skip a specific import
3. Run all enabled aggregations except Linked Edges and Provenance Summary
4. Test filter combinations safely using
--dry_runLogs the selected and rejected calculation stages without submitting BigQuery jobs: