Fix treatment of early retirement in myopic mode#340
Conversation
|
Warning Review limit reached
More reviews will be available in 1 minute and 21 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (19)
WalkthroughThe PR adds myopic retired-existing-capacity support across model, loading, capacity, and test code, and updates two pre-commit hook revisions. ChangesMyopic capacity adjustments
Pre-commit revisions
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
temoa/data_io/component_manifest.py (1)
448-448: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDead
validation_mapafter removing the validator.With
validator_nameremoved,_filter_datareturns early (no validator), sovalidation_map=(0, 2, 3)is never used. Consider dropping it to avoid implying validation that no longer happens.♻️ Suggested cleanup
LoadItem( component=model.lifetime_survival_curve, table='lifetime_survival_curve', columns=['region', 'period', 'tech', 'vintage', 'fraction'], - validation_map=(0, 2, 3), is_period_filtered=False, is_table_required=False, ),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@temoa/data_io/component_manifest.py` at line 448, The validation_map argument is now dead because _filter_data no longer uses it once validator_name was removed, so the ComponentManifest-related call still implies validation that never happens. Update the relevant _filter_data invocation to drop validation_map=(0, 2, 3), and keep the remaining arguments consistent with the no-validator flow in component_manifest.py.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@temoa/components/time.py`:
- Around line 202-203: The branch in the time-period logic is unnecessarily
using an elif after a preceding return, which triggers RET505. Update the
conditional in the relevant function in time.py so the check after the earlier
return is expressed as a plain if/early-return style, keeping the same behavior
while removing the redundant elif.
In `@temoa/data_io/hybrid_loader.py`:
- Around line 639-642: The docstring on the method currently describes behavior
copied from `_load_existing_capacity` that this code does not perform. Update
the documentation for the affected method in `hybrid_loader.py` so it accurately
reflects this method’s actual purpose and remove references to handling myopic
vs. standard runs or populating `tech_exist`; use the surrounding method name
and its current implementation to rewrite the wording consistently.
---
Outside diff comments:
In `@temoa/data_io/component_manifest.py`:
- Line 448: The validation_map argument is now dead because _filter_data no
longer uses it once validator_name was removed, so the ComponentManifest-related
call still implies validation that never happens. Update the relevant
_filter_data invocation to drop validation_map=(0, 2, 3), and keep the remaining
arguments consistent with the no-validator flow in component_manifest.py.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 10b82a90-3eef-45e4-8c3a-95a14a1189f5
📒 Files selected for processing (9)
.pre-commit-config.yamltemoa/components/capacity.pytemoa/components/limits.pytemoa/components/technology.pytemoa/components/time.pytemoa/components/utils.pytemoa/core/model.pytemoa/data_io/component_manifest.pytemoa/data_io/hybrid_loader.py
idelder
changed the title
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
tests/conftest.py (1)
205-214: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winIsolate the myopic database per parametrized run.
config_myopic_capacities.tomlhard-codes the same SQLite file for input/output,pytest_configure()only rebuilds it once, and this fixture now reuses that file across all myopic parametrizations. That breaks test isolation:_load_retired_existing_capacity()reloadsoutput_retired_capacityfrom the DB, andtest_myopic_stress_tests()sumsoutput_objectivewithout filtering by scenario, so later cases can inherit rows from earlier ones.Proposed fix
config = TemoaConfig.build_config( config_file=config_file, output_path=tmp_path, silent=True, ) + if config.scenario_mode.name == 'MYOPIC': + import shutil + + seed_db = Path(config.input_database) + isolated_db = tmp_path / seed_db.name + shutil.copy2(seed_db, isolated_db) + config.input_database = isolated_db + config.output_database = isolated_db + # Allow parametrized tests to override myopic settings per case. myopic_overrides = request.param.get('myopic') if myopic_overrides is not None: config.myopic_inputs = {**(config.myopic_inputs or {}), **myopic_overrides}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/conftest.py` around lines 205 - 214, The myopic test fixture is reusing the same SQLite-backed database across parametrized runs, so per-case data leaks between scenarios. Update the myopic setup in `pytest_configure()`/`config_myopic_capacities.toml` handling and the `config = TemoaConfig.build_config(...)` fixture path so each parametrization gets its own isolated DB file or a fresh rebuild before use. Make sure `_load_retired_existing_capacity()` reads only the current case’s data and that `test_myopic_stress_tests()` no longer sees rows from prior runs by keeping the DB scoped to the active parametrized `request.param`.temoa/components/technology.py (1)
445-453: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winKeep the hard failure for non-myopic missing existing-capacity processes.
This now downgrades every positive-capacity entry with remaining life but no
process_periodsentry to a warning. The new myopic-retirement path is the legitimate exception, but in standard runs this still means the model is silently dropping capacity that should be active. Please limit the warning to the adjusted-capacity/myopic case and preserve the old exception everywhere else.Suggested direction
- if (r, t, v) not in model.process_periods and v + life > model.time_optimize.first(): + if (r, t, v) not in model.process_periods and v + life > model.time_optimize.first(): msg = ( f'Existing capacity {r, t, v} with lifetime {life} and capacity {cap} ' 'should extend into future periods but is not an active process. ' 'May be missing from Efficiency table or too small to carry forward ' 'if running in myopic mode.' ) - logger.warning(msg) + if value(model.myopic_discounting_year) > 0: + logger.warning(msg) + else: + raise ValueError(msg)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@temoa/components/technology.py` around lines 445 - 453, The existing-capacity handling currently turns every positive-capacity entry with remaining life but no model.process_periods match into a warning, which removes the hard failure in standard runs. Update the logic around lifetime_process, process_periods, and time_optimize so only the myopic adjusted-capacity retirement path is downgraded to logger.warning, and keep the original exception/abort behavior for all other non-myopic missing active-process cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@temoa/data_io/hybrid_loader.py`:
- Around line 672-732: The custom lifetime loaders dropped the region-aware
sparse-index filters, so restore the old viability checks in
_load_lifetime_tech, _load_lifetime_process, and _load_lifetime_survival_curve.
In _load_lifetime_tech, filter rows by the full (region, tech) key set from
lifetime_tech_rt rather than only tech names; in the other two methods, apply
the matching viable_rtv-based restriction so only indexed lifetime rows are
loaded. Use the existing loader methods and model params in TemoaModel to keep
the sparse-index contract intact.
---
Outside diff comments:
In `@temoa/components/technology.py`:
- Around line 445-453: The existing-capacity handling currently turns every
positive-capacity entry with remaining life but no model.process_periods match
into a warning, which removes the hard failure in standard runs. Update the
logic around lifetime_process, process_periods, and time_optimize so only the
myopic adjusted-capacity retirement path is downgraded to logger.warning, and
keep the original exception/abort behavior for all other non-myopic missing
active-process cases.
In `@tests/conftest.py`:
- Around line 205-214: The myopic test fixture is reusing the same SQLite-backed
database across parametrized runs, so per-case data leaks between scenarios.
Update the myopic setup in `pytest_configure()`/`config_myopic_capacities.toml`
handling and the `config = TemoaConfig.build_config(...)` fixture path so each
parametrization gets its own isolated DB file or a fresh rebuild before use.
Make sure `_load_retired_existing_capacity()` reads only the current case’s data
and that `test_myopic_stress_tests()` no longer sees rows from prior runs by
keeping the DB scoped to the active parametrized `request.param`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 811e6f94-eae9-4c5f-bed5-4f0fc8be7969
📒 Files selected for processing (15)
temoa/components/technology.pytemoa/components/time.pytemoa/core/model.pytemoa/data_io/component_manifest.pytemoa/data_io/hybrid_loader.pytemoa/data_io/loader_manifest.pytests/conftest.pytests/legacy_test_values.pytests/test_full_runs.pytests/testing_configs/config_myopic_capacities.tomltests/testing_data/mediumville.sqltests/testing_data/mediumville_sets.jsontests/testing_data/myopic_capacities.sqltests/testing_data/test_system_sets.jsontests/testing_data/utopia_sets.json
💤 Files with no reviewable changes (2)
- temoa/components/time.py
- tests/testing_data/mediumville.sql
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
temoa/data_io/hybrid_loader.py (1)
649-677: 🩺 Stability & Availability | 🟠 MajorGuard
output_retired_capacityaccess for older schemas / first myopic step.The
output_retired_capacitytable is defined intemoa_schema_v4.sqlbut is absent in earlier schema versions. This loader queries the table directly without a schema guard. On a database initialized from an older schema, or in a run where the output table has not yet been created,cur.execute(...)will raise anOperationalError, causing the process to abort.Use the existing
table_existshelper to safely skip loading if the table is missing.🛡️ Suggested fix
if not mi: # for now, we only use this in myopic mode return + if not self.table_exists('output_retired_capacity'): + return + rows_to_load = []🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@temoa/data_io/hybrid_loader.py` around lines 649 - 677, The _load_retired_existing_capacity method queries output_retired_capacity directly, so add a table-existence guard before the SELECT using the existing table_exists helper. If the table is missing (older schema or first myopic step), return early and skip loading; otherwise keep the existing query and _load_component_data call unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@temoa/components/technology.py`:
- Around line 449-459: Move the surviving_cap calculation inside the conditional
in the technology processing logic so it is only evaluated when the warning is
actually emitted. In the block around the existing p/model.time_optimize.first()
and logger.warning(msg) flow, keep the process_periods and lifetime check first,
then compute model.lifetime_survival_curve[r, p, t, v] and surviving_cap only
within that branch to avoid unnecessary indexed access and KeyError on the
normal path.
In `@temoa/data_io/hybrid_loader.py`:
- Around line 644-647: Remove the unused self.viable_existing_techs assignment
in hybrid_loader.py and keep the existing downstream filters based on
self.viable_existing_rt and self.viable_existing_rtv unchanged. The change
should be applied in the logic that collects existing capacity data indices, and
you should ensure no other code in HybridLoader or related helpers still depends
on viable_existing_techs before deleting it.
---
Outside diff comments:
In `@temoa/data_io/hybrid_loader.py`:
- Around line 649-677: The _load_retired_existing_capacity method queries
output_retired_capacity directly, so add a table-existence guard before the
SELECT using the existing table_exists helper. If the table is missing (older
schema or first myopic step), return early and skip loading; otherwise keep the
existing query and _load_component_data call unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7e7c3941-18ea-43f0-8df9-e6ffb1d84274
📒 Files selected for processing (2)
temoa/components/technology.pytemoa/data_io/hybrid_loader.py
idelder
force-pushed
the
fix/myopic_retirement
branch
from
21a5be4 to
c7ff488
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
temoa/data_io/hybrid_loader.py (1)
648-676: 🩺 Stability & Availability | 🟡 MinorGuard the direct query with
table_existsto honoris_table_required=False.The manifest (temoa/data_io/component_manifest.py) explicitly marks
output_retired_capacityasis_table_required=False. While the standard_fetch_datapath checkstable_existsand skips optional tables gracefully, the custom loader_load_retired_existing_capacity(temoa/data_io/hybrid_loader.py:648-676) executes a direct SQL query againstoutput_retired_capacitywithout this check.If the database schema lacks this table (e.g., older schemas or partial outputs), this unguarded query raises an
OperationalError, aborting the myopic run. Other custom loaders in the same file (lines 476, 532, 631) correctly wrap queries withself.table_exists().Suggested fix:
Wrap theSELECTstatement in_load_retired_existing_capacitywithif self.table_exists('output_retired_capacity'):to prevent crashes when the optional table is missing.Diff
prev_period = prev_period_res[0] if prev_period_res else -1 - rows_to_load = cur.execute( - 'SELECT region, period, tech, vintage, cap_early FROM output_retired_capacity WHERE ' - 'period <= ? AND scenario = ? AND cap_early > 0 ', - (prev_period, self.config.scenario), - ).fetchall() + if self.table_exists('output_retired_capacity'): + rows_to_load = cur.execute( + 'SELECT region, period, tech, vintage, cap_early FROM output_retired_capacity ' + 'WHERE period <= ? AND scenario = ? AND cap_early > 0 ', + (prev_period, self.config.scenario), + ).fetchall()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@temoa/data_io/hybrid_loader.py` around lines 648 - 676, The custom loader `_load_retired_existing_capacity` is querying `output_retired_capacity` directly even though the manifest marks it optional, so add a `self.table_exists('output_retired_capacity')` guard before the SQL in `hybrid_loader.py`. If the table is missing, skip the load and return cleanly, matching the pattern used by the other optional-table loaders in this class and preserving `is_table_required=False` behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@temoa/data_io/hybrid_loader.py`:
- Around line 648-676: The custom loader `_load_retired_existing_capacity` is
querying `output_retired_capacity` directly even though the manifest marks it
optional, so add a `self.table_exists('output_retired_capacity')` guard before
the SQL in `hybrid_loader.py`. If the table is missing, skip the load and return
cleanly, matching the pattern used by the other optional-table loaders in this
class and preserving `is_table_required=False` behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: bbf7276b-fc9c-4614-9267-8875a8b320a3
📒 Files selected for processing (2)
temoa/components/technology.pytemoa/data_io/hybrid_loader.py
updates: - [github.com/astral-sh/uv-pre-commit: 0.11.19 → 0.11.21](astral-sh/uv-pre-commit@0.11.19...0.11.21) - [github.com/astral-sh/ruff-pre-commit: v0.15.16 → v0.15.17](astral-sh/ruff-pre-commit@v0.15.16...v0.15.17)
updates: - [github.com/astral-sh/uv-pre-commit: 0.11.21 → 0.11.23](astral-sh/uv-pre-commit@0.11.21...0.11.23) - [github.com/astral-sh/ruff-pre-commit: v0.15.17 → v0.15.18](astral-sh/ruff-pre-commit@v0.15.17...v0.15.18)
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
…counting purposes Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
…ly retirement, and growth rate constraints all together Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
…capacities Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
Signed-off-by: Davey Elder <iandavidelder@gmail.com>
idelder
force-pushed
the
fix/myopic_retirement
branch
from
0bf774b to
b48e1d8
Compare
1 participant
This PR fixes myopic inter-period consistency and adds stronger regression coverage.
Summary by CodeRabbit
Summary
New Features
Bug Fixes
Tests
Chores