Edisgo run edisgo - #629
Open
MoritzSchloesser wants to merge 75 commits into
Open
Conversation
Adds optional overlying_grid_data kwarg to run_edisgo() that is stashed on RunContext for downstream tasks instead of being passed as a keyword to every task (which broke task signatures). task_import_overlying_grid_data now: - accepts the standard (edisgo, ctx, *, ...) signature - reads overlying_grid_data from ctx, falls back to overlying_grid.path in the runner config - loads dispatchable + renewables_potential CSVs and applies them via set_time_series_active_power_predefined - shifts the year and reindexes overlying-grid attributes to the active edisgo timeindex so CSV-based input lines up with OEDB time series task_set_timeindex now reduces existing time-series data to the new index (via reduce_timeseries_data_to_given_timeindex) instead of silently leaving stale data behind. Renames the uc4_example_MS preset to uc4_example.
…ndex
powermodels_io.from_powermodels now slices the destination time index
explicitly when writing OPF flex results back to the EDisGo
timeseries (gen_nd, heatpumps, electromobility, dsm, storage). Using
`loc[:, names]` overwrites every row of the underlying DataFrame
even when the OPF only covered a subset of timesteps; restricting
to `timeseries.timeindex` keeps untouched rows intact.
task_optimize:
- fixes typo `flexbile` → `flexible` in the dsm shortcut
- corrects the dsm condition (was checking `is not None`, should
populate when `None`)
- materializes empty flex lists once instead of repeating `or []`
at every call site
task_import_overlying_grid_data reindexes the three SOC attributes
(storage_units_soc, thermal_storage_units_{central,decentral}_soc) to
timeindex + 1 extra step, because PowerModels expects the
end-of-period SOC value; non-SOC overlying-grid attributes still
reindex to the plain timeindex.
uc4_example preset reworked: switches to opf_version 3 (HV
constraints from overlying grid), drops base_reinforce + check_integrity
from the pipeline, adds an explicit `overlying_grid.path` config slot,
splits import_electromobility kwargs onto separate lines, and enables
archive + save_opf_results on the final save step.
Adds optional overlying_grid_data kwarg to run_edisgo() that is stashed on RunContext for downstream tasks instead of being passed as a keyword to every task (which broke task signatures). task_import_overlying_grid_data now: - accepts the standard (edisgo, ctx, *, ...) signature - reads overlying_grid_data from ctx, falls back to overlying_grid.path in the runner config - loads dispatchable + renewables_potential CSVs and applies them via set_time_series_active_power_predefined - shifts the year and reindexes overlying-grid attributes to the active edisgo timeindex so CSV-based input lines up with OEDB time series task_set_timeindex now reduces existing time-series data to the new index (via reduce_timeseries_data_to_given_timeindex) instead of silently leaving stale data behind. Renames the uc4_example_MS preset to uc4_example.
…ndex
powermodels_io.from_powermodels now slices the destination time index
explicitly when writing OPF flex results back to the EDisGo
timeseries (gen_nd, heatpumps, electromobility, dsm, storage). Using
`loc[:, names]` overwrites every row of the underlying DataFrame
even when the OPF only covered a subset of timesteps; restricting
to `timeseries.timeindex` keeps untouched rows intact.
task_optimize:
- fixes typo `flexbile` → `flexible` in the dsm shortcut
- corrects the dsm condition (was checking `is not None`, should
populate when `None`)
- materializes empty flex lists once instead of repeating `or []`
at every call site
task_import_overlying_grid_data reindexes the three SOC attributes
(storage_units_soc, thermal_storage_units_{central,decentral}_soc) to
timeindex + 1 extra step, because PowerModels expects the
end-of-period SOC value; non-SOC overlying-grid attributes still
reindex to the plain timeindex.
uc4_example preset reworked: switches to opf_version 3 (HV
constraints from overlying grid), drops base_reinforce + check_integrity
from the pipeline, adds an explicit `overlying_grid.path` config slot,
splits import_electromobility kwargs onto separate lines, and enables
archive + save_opf_results on the final save step.
…rlyingGrid object
A cached SSH-tunneled egon-data engine is reused across many grids in an eGo
run. During a grid's multi-minute OPF the pooled connection sits idle and the
server/SSH tunnel closes it, so a later grid got a dead connection
("server closed the connection unexpectedly").
- pool_pre_ping=True: validate (and transparently replace) a connection
before use.
- pool_recycle=3600: proactively drop connections older than an hour.
- set_keepalive=30.0 on the SSH tunnel: keep the transport alive during long
idle periods so the tunnel is not torn down.
Large SOC OPF instances often end with barrier status SUBOPTIMAL: a
feasible primal point exists but the convergence tolerances were not
met. The previous check treated every status != OPTIMAL as infeasible
and called compute_conflict!, which raises Gurobi error 10015 ("Cannot
compute IIS on a feasible model"), killing the Julia process and
failing the grid run.
Now the solution is used whenever the solver proves optimality or
reports a feasible primal point; the IIS conflict is only computed
when there is genuinely no primal solution. The SOC tightness check
remains restricted to proven optima.
Add a `select_timesteps` pipeline task that reduces the analysed time index to a selected subset, in two modes: - manual: an explicit set/range of timesteps (positioned before the imports so egon_data downloads are restricted to the selected steps); - auto: the most critical time intervals, via either a power flow (`get_most_critical_time_intervals(by="power_flow")`) or the residual load (`by="residual_load"`, no power flow — intervals centered on the highest/lowest residual-load steps, snapped to `time_step_day_start`). Auto selection may yield two disconnected intervals. `pm_optimize` now detects a non-contiguous time index and runs a separate, independent OPF per contiguous interval (storage/heat state does not carry across the gap), merging the per-interval results and reporting per-interval solve status. `reinforce` handles the reduced index unchanged. Architecture: pipeline tasks stay thin (mode selection + data transfer); the computation lives in edisgo core: - selection logic in tools.temporal_complexity_reduction (`get_most_critical_time_steps`/`_intervals` gain a `by` mode; public `select_two_intervals`/`intervals_overlap`); - the multi-interval OPF split/merge in opf.powermodels_opf.pm_optimize. Also: - EV flexibility bands built in a dedicated `build_flexibility_bands` task and aligned (year + frequency) to the analysis index in `reduce_timeseries_data_to_given_timeindex`; - `OPFResults.interval_results` for the per-interval solve report; - config surface: a top-level `timeseries_selection` block + `uc5_select_timesteps` preset + `run_example_05.py`; - tests in tests/run, tests/tools, tests/opf. Co-authored-by: Moritz Schloesser <moritz.schloesser@rl-institut.de>
Adds spatial_reduce/spatial_restore tasks bracketing optimize, so the OPF
can run on a spatially-reduced grid while reinforcement still runs on the
full topology. New core function apply_reduced_results_to_full_grid maps
optimized flexible-component dispatch (charging points, heat pumps, DSM
loads, storage) back onto the full grid, by name or disaggregated onto
old_name members per time step (weighted by each member's pre-OPF
flexibility envelope) when aggregation_mode=True. Reactive power is
recomputed on restore, mirroring how pm_optimize itself handles it.
optimize now also records its flexible-component name lists on ctx.flags
and declares provides={"optimized_dispatch"}, so the validator's existing
requires/provides system enforces spatial_restore running after both
spatial_reduce and optimize.
get_flexibility_bands only resamples bands to the active time-series FREQUENCY, not the active date range - so after a manual/short select_timesteps window, flexibility_bands kept spanning the full underlying SimBEV data range with no later step ever trimming it down. Any consumer indexing flexibility_bands by edisgo.timeseries.timeindex (e.g. the OPF's charging-point constraint builder) could then hit missing time steps. build_flexibility_bands now trims electromobility data to the active time index right after building the bands.
Standalone preset wiring spatial_reduce/spatial_restore around optimize, disabled by default via spatial_reduction.enabled. run_example_06.py runs it against a real ding0 grid.
CONTEXT.md gains a glossary section for spatial reduction terms (busmap, disaggregation rule, config surface, ordering). docs_notes/ carries the full design-session record plus two follow-up issue write-ups: the flexibility-bands time-index gap (fixed) and the aggregation_mode=True + charging-points gap (not yet fixed, tracked for later).
The preset's header comment and self-contained-run example still said "UC5"/uc5_spatial_reduction from when the file was copied from uc5_select_timesteps.yaml. Also adds a full issue description covering what's done and what's still open (aggregation_mode=True + charging points) for the spatial-reduction pipeline integration.
* Change database from OEP to local
* feat: spatial complexity reduction for the run pipeline
Adds spatial_reduce/spatial_restore tasks bracketing optimize, so the OPF
can run on a spatially-reduced grid while reinforcement still runs on the
full topology. New core function apply_reduced_results_to_full_grid maps
optimized flexible-component dispatch (charging points, heat pumps, DSM
loads, storage) back onto the full grid, by name or disaggregated onto
old_name members per time step (weighted by each member's pre-OPF
flexibility envelope) when aggregation_mode=True. Reactive power is
recomputed on restore, mirroring how pm_optimize itself handles it.
optimize now also records its flexible-component name lists on ctx.flags
and declares provides={"optimized_dispatch"}, so the validator's existing
requires/provides system enforces spatial_restore running after both
spatial_reduce and optimize.
* fix: trim electromobility flexibility bands to the active time index
get_flexibility_bands only resamples bands to the active time-series
FREQUENCY, not the active date range - so after a manual/short
select_timesteps window, flexibility_bands kept spanning the full
underlying SimBEV data range with no later step ever trimming it down.
Any consumer indexing flexibility_bands by edisgo.timeseries.timeindex
(e.g. the OPF's charging-point constraint builder) could then hit missing
time steps. build_flexibility_bands now trims electromobility data to the
active time index right after building the bands.
* Add uc6_spatial_reduction preset and example runner
Standalone preset wiring spatial_reduce/spatial_restore around optimize,
disabled by default via spatial_reduction.enabled. run_example_06.py runs
it against a real ding0 grid.
* docs: spatial reduction glossary and grilling-session notes
CONTEXT.md gains a glossary section for spatial reduction terms (busmap,
disaggregation rule, config surface, ordering). docs_notes/ carries the
full design-session record plus two follow-up issue write-ups: the
flexibility-bands time-index gap (fixed) and the aggregation_mode=True +
charging-points gap (not yet fixed, tracked for later).
* Change overlying_grid path
* Fix stale UC5 naming in uc6_spatial_reduction.yaml, add issue writeup
The preset's header comment and self-contained-run example still said
"UC5"/uc5_spatial_reduction from when the file was copied from
uc5_select_timesteps.yaml. Also adds a full issue description covering
what's done and what's still open (aggregation_mode=True + charging
points) for the spatial-reduction pipeline integration.
* style: apply pre-commit hooks to the spatial-reduction commits
Hooks weren't run before the original commits landed. Adds the missing
license header to edisgo/run/context.py, applies ruff-format's line
collapsing across a few files, and manually wraps two docstring
:func: cross-references that were still over the line-length limit
after formatting.
---------
CONTEXT.md and various docs_notes/ writeups were meant to stay local but got included in the spatial complexity reduction merge. Untrack them (kept on disk) and gitignore them going forward.
…into edisgo_run_edisgo # Conflicts: # edisgo/edisgo.py # edisgo/run/tasks/spatial.py # run_example_06.py # tests/tools/test_spatial_complexity_reduction.py
Notebooks, a scratch runner script, and results/ outputs that live locally but aren't meant to be tracked on this branch.
MoritzSchloesser
force-pushed
the
edisgo_run_edisgo
branch
from
July 16, 2026 14:54
f11ccd6 to
beedffd
Compare
apply_charging_strategy (all three modes: dumb/reduced/residual) wrote a full SimBEV-simulation-length series into loads_active_power/ loads_reactive_power unconditionally, via add_component_time_series's raw concat - a shorter or gapped edisgo.timeseries.timeindex (e.g. after select_timesteps) never trimmed the excess, leaving stale rows in the private _loads_active_power that any direct reader (not just the timeindex-scoped public getter) would see. Trim to edisgo.timeseries.timeindex when SimBEV's stepsize already matches eDisGo's own frequency, the common case. When it doesn't, this function internally resamples edisgo_obj.timeseries to reconcile frequencies and back - that round-trip currently fabricates a contiguous index, silently reopening any gap select_timesteps (auto mode) left in place, so the trim is skipped there rather than risk operating on the wrong window. Documented as a known limitation pending a fix to TimeSeries.resample() itself (tracked separately, not part of this change). Part of the #703 checklist (docs_notes/issue_temporal_reduction_flexibility_bands.md).
…meindex (#711) operating_strategy computed loads_active_power from all rows of heat_demand_df/cop_df, with no scoping to edisgo.timeseries.timeindex. import_heat_pumps trims both to the timeindex active at import time, but nothing re-trims them if the timeindex changes afterward (e.g. a later select_timesteps step) - any such staleness would silently propagate into loads_active_power via add_component_time_series's raw concat. Scope both operands to edisgo.timeseries.timeindex via .loc[] before dividing. Raises KeyError if heat_demand_df/cop_df are missing data for a time step in the active timeindex, rather than silently writing rows outside it - this is data staleness the caller should fix (re-import or re-set the heat pump time series), not something to paper over. Part of the #703 checklist (docs_notes/issue_temporal_reduction_flexibility_bands.md).
…ps (#712) set_time_series_manual only warned when no time index was set, but never validated that a user-supplied DataFrame actually covers a pre-existing edisgo.timeseries.timeindex - a DataFrame missing required timestamps was silently accepted and concatenated via add_component_time_series's raw concat, leaving a partially-populated (NaN-gapped) series with no indication anything was wrong. Add a coverage check: raise ValueError naming the missing timestamps when a non-empty DataFrame doesn't cover the active timeindex. Two exemptions, both required for existing behavior: - Skip entirely when the timeindex is empty - this is the documented "set data first, timeindex later" workflow, and there is nothing meaningful to validate coverage against. - Skip DataFrames with zero columns - nothing is written, so there is nothing to check. Confirmed necessary: eGo's edisgo_integration.py has a real call site passing a zero-column DataFrame as a placeholder no-op. Part of the #703 checklist (docs_notes/issue_temporal_reduction_flexibility_bands.md).
…ex (#713) The self-provided-DataFrame branches of TimeSeries's four predefined_* methods (predefined_fluctuating_generators_by_technology, predefined_dispatchable_generators_by_technology, predefined_conventional_loads_by_sector, predefined_charging_points_by_use_case) had no coverage check against edisgo.timeseries.timeindex - same gap as set_time_series_manual (#712), just at different call sites. Their 'oedb'/'demandlib' string-option siblings were already correctly scoped via _timeindex_helper_func. Reuse the check_timeindex_coverage helper from #712, relocated from edisgo.py to edisgo/tools/tools.py (both edisgo.py and edisgo/network/timeseries.py need it; a direct cross-import the other way would be circular). Apply it to all four self-provided-DataFrame branches, right after their existing type/emptiness validation. Part of the #703 checklist (docs_notes/issue_temporal_reduction_flexibility_bands.md).
get_flexibility_bands built EV charging flexibility bands spanning SimBEV's own native calendar and simulated date range, independent of edisgo.timeseries.timeindex. SimBEV's calendar is commonly a fixed reference year (2011 in the test fixtures); a scenario's own timeindex is commonly a different year (e.g. 2035) and/or a much shorter window - indexing the returned bands by such a timeindex raised KeyError, the literal crash originally reported in #703. A workaround already existed at one call site (task_build_flexibility_bands, which called reduce_timeseries_data_to_given_timeindex right after get_flexibility_bands), but this was a local patch, not a fix to the function's own contract - any other caller was still exposed. get_flexibility_bands now year-aligns (reusing align_series_to_timeindex) and trims its returned/stored bands to edisgo_obj.timeseries.timeindex whenever it's non-empty, regardless of the resample parameter, since this is a correctness fix rather than an optional resampling convenience. Band construction itself is untouched (still spans SimBEV's full native range) - a charging event straddling a later window's boundary must still count toward the band inside that window. Empty timeindex remains a no-op. Remove the now-redundant explicit reduce_timeseries_data_to_given_timeindex call in task_build_flexibility_bands, since the fix makes it a no-op there.
DSM had no resample() method, unlike every sibling container (TimeSeries, HeatPump, Electromobility, OverlyingGrid). EDisGo.resample_timeseries() - the public API for changing an entire EDisGo object's time resolution - never touched dsm, silently leaving p_min/p_max/e_min/e_max at their original frequency while everything else changed, with no warning. Add DSM.resample(method, freq), mirroring HeatPump.resample_timeseries's signature/pattern and iterating the existing DSM._attributes list. Wire it into EDisGo.resample_timeseries and update its docstring's affected-attributes list, which previously omitted DSM entirely. Also remove a manual DSM-resample workaround in tests/network/test_overlying_grid.py's setup_flexibility_data() that predated this fix (it hand-rolled exactly what DSM.resample() now does correctly). With the real fix in place, that workaround double-resampled already-correct data and broke test_distribute_overlying_grid_timeseries with a shape mismatch - removing it resolves that. Part of the #703 checklist (docs_notes/issue_temporal_reduction_flexibility_bands.md).
reduce_timeseries_data_to_given_timeindex (edisgo/tools/tools.py) is a well-implemented, correct function that trims TimeSeries, Electromobility.flexibility_bands (with year-align), HeatPump, DSM, and OverlyingGrid attributes consistently to a target timeindex - but it was only ever importable from edisgo.tools.tools, never exposed on the EDisGo class. Its only prior callers were internal run-pipeline tasks; anyone using EDisGo directly had no discoverable way to find or use it. Add EDisGo.reduce_timeseries_data_to_given_timeindex(...), a thin wrapper delegating to the existing free function with the same signature and defaults, mirroring how EDisGo.resample_timeseries already delegates to underlying implementations. No changes to the tested logic in tools.py; the three existing internal call sites can keep importing the free function directly. Part of the #703 checklist (docs_notes/issue_temporal_reduction_flexibility_bands.md). This completes the #703 checklist (7/7).
#719) _build_electromobility and _build_component_timeseries's electromobility branch read edisgo_obj.electromobility.flexibility_bands positionally (.iloc[0] for static bounds, .values.tolist() for the full per-timestep column) instead of aligning to edisgo_obj.timeseries.timeindex first. This was only safe because the standard run pipeline always finalizes the timeindex before building bands and running optimize - an implicit ordering convention, not something either function or the validator enforced. Breaking that ordering would silently feed wrong/misaligned or length-mismatched OPF input to PowerModels/Julia rather than raising a clear error. Both functions now explicitly select flexibility_bands[key].loc[timeindex] (or .loc[timeindex, flexible_cps]) before any positional access, mirroring the alignment pattern already established in Electromobility.get_flexibility_bands (#703). A genuine mismatch now raises KeyError immediately instead of silently producing wrong data. No separate length assertion against pm["time_series"]["num_steps"] was needed: to_powermodels always builds psa_net via edisgo_object.to_pypsa() with no explicit timesteps override, so psa_net.snapshots is always exactly edisgo_obj.timeseries.timeindex here - the .loc[timeindex] alignment already guarantees the length matches. Fixes #718.
The shared tools.resample() function called pandas' own .resample() directly on each attribute's data, which always produces a contiguous bucket sequence spanning the data's own min-to-max timestamp - any gap in a legitimately gapped timeindex (e.g. as produced by select_timesteps in auto mode, which deliberately keeps two disjoint intervals separate) got silently filled with resample artifacts (forward-filled/averaged copies of adjacent real values) rather than preserved. TimeSeries.resample compounded this: after calling the shared function, it also rebuilt self._timeindex via one pd.date_range(first, last, freq) span, discarding the gap from the index itself, not just the data. Add split_into_contiguous_runs, splitting a DataFrame into its maximal contiguous runs (a run boundary is any gap larger than the original frequency). The shared resample() function now resamples each run independently and concatenates the results, so a gap is never bridged. TimeSeries.resample's own _timeindex rebuild now reconstructs the new index per contiguous run and unions them back together, fixing the compounding issue on top. Affects every caller of the shared function: TimeSeries.resample, HeatPump.resample_timeseries, DSM.resample, OverlyingGrid.resample. Electromobility.resample is unaffected - it already warns on non-continuous input rather than silently fabricating data. Found during the #703 PR series (time-series writers not uniformly respecting a pre-existing timeindex), tracked separately since it's a distinct pre-existing bug in shared infrastructure, not part of #703's own scope.
…residual charging strategy (#726) Fixes #724. residual previously tiled (cyclically repeated) residual_load to cover charging events beyond the active eDisGo timeindex, ranking timesteps against a fabricated, non-periodic-in-reality signal. Now scopes to the active timeindex: fully out-of-window events are dropped, boundary-straddling events are prorated by in-window parking-time fraction, and both the residual_dumb sub-bucket and the argpartition ranking correctly exclude out-of-window steps even when a single event's window spans a gap in a non-contiguous active timeindex.
…fore building (#727) Part of #703 follow-up (ADR 0002). dumb/reduced previously wrote each charging event's deterministic interval unconditionally into a full-SimBEV-length series, relying solely on a later crop step to trim to the active timeindex. Now clips the placement slice itself to whatever overlaps the active timeindex, at unchanged power - correctly handling a gap inside a single event's interval by writing each in-window sub-slice independently. get_flexibility_bands had the same underlying issue: its output values were already correct after the existing final .loc[timeindex] clip, but construction itself built over EVERY charging process and SimBEV's entire simulated range regardless of how much shorter the active timeindex was - real "build full, then crop" waste, just hidden behind correct output. Now filters to events overlapping the active timeindex first and sizes the construction array to only what those events plus the active window require, while avoiding a pre-existing edge case (an event's true end landing exactly on the array's last row gets silently excluded) that the tighter sizing would otherwise trigger far more often. See docs/adr/0002-charging-time-series-reduced-to-active-timeindex-before-building.md.
reinforce_lines_voltage_issues() determines the node at two thirds of the feeder length (node_2_3) and normally disconnects the line there and reconnects it to the station. If node_2_3 turns out to be the feeder representative - the bus directly connected to the station - no line can be disconnected and the measure falls back to reinforcing lines instead. In that fallback only the single line between the station and the representative was reinforced, not the remaining lines on the path to the critical node. Since the voltage deviation at the critical node is generally dominated by the segments further away from the station, the measure frequently did not change the reported voltage issue at all. Grid reinforcement then repeated the same ineffective measure in every iteration and aborted with MaximumIterationError, typically reporting a residual deviation in the per mille range. LVGrid_5 of the test grid shows this: for a voltage issue at Bus_BranchTee_LVGrid_5_2, node_2_3 is moved back to the representative Bus_BranchTee_LVGrid_5_1. Only Line_50000003 was reinforced, which is 0.56 m long, while Line_50000002 - 30 m of NAYY 4x1x35 causing nearly all of the voltage drop - was left untouched. All lines on the path from the station to the critical node are now reinforced. Feeders consisting of a single line are unaffected. Short feeders are affected most, which is why the error shows up predominantly in grids that went through spatial complexity reduction, where node_2_3 frequently coincides with the representative. Test expectations for the number of equipment changes and grid expansion costs are updated accordingly, as more lines are reinforced per measure. Applied directly on edisgo_run_edisgo instead of merging dev. Same change as 64f3ed6 on fix/reinforce-lines-voltage-issues-all-segments, without the whatsnew entry to keep a later merge with dev conflict-free.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please include a summary of the change and which issue is fixed.
Fixes # (issue)
Type of change
Please delete options that are not relevant.
Checklist:
pre-commithooks