benchmarks: reusable registry, new model types, new phases, CI smoke

[FBumann]

Overhaul of the internal benchmark suite

Brings benchmarks/ from a hand-wired pytest collection into a
typer-driven suite with cross-version sweeps, per-PR CodSpeed CI, and
plotly views.

Core pieces

• Reusable model registry. from benchmarks import REGISTRY exposes
  10 self-registering specs (LP / QP / MILP / SOS / piecewise / sparse
  networks / PyPSA carbon) with build(size) + feature & phase tags.
  Adding a model is one file.
• Phase coverage. Five phases per spec: build, matrices,
  lp_write, netcdf round-trip, solver_handoff (highs/gurobi/
  mosek/xpress). Plus PyPSA end-to-end.
• One CLI. python -m benchmarks --help covers run / smoke / sweep
  / compare / plot / notebook / memory{save,sweep,compare}.
• Cross-version sweeps. sweep 0.6.7 0.7.0 builds one uv venv per
  version; same flow for memory via memray.Tracker (model
  construction excluded from the peak).
• Plotly views. plot <snapshots> picks scaling (1), scatter
  (2, default), or sweep heatmap (3+). --facets {phase, model}
  splits across subplots. Auto-detects timing vs memory JSON.
• Per-PR CodSpeed + Dependabot loop. CodSpeed CI gives cachegrind-
  attributed regression detection on every PR. [benchmarks] pins
  perf-relevant deps individually (numpy/scipy/xarray/pandas/polars/
  dask/highspy/netcdf4); tooling deps grouped into one monthly PR.
  Each perf-relevant bump → one PR → attributed CodSpeed delta.
• Notebook walkthrough at benchmarks/notebooks/registry_usage.ipynb,
  executed in CI so examples can't rot.

Sample output: 0.6.7 → 0.7.0

Default scatter view — each test at (baseline_time, ratio), top-right
corner = real regressions (slow tests that got slower):

Same comparison, memory dimension (peak RSS via memray.Tracker):

Per-phase delta breakdown (compare --facets phase):

Going forward

• Enable CodSpeed: install the GitHub app on the org, add the repo,
  drop CODSPEED_TOKEN into Actions secrets. Until then the CodSpeed
  job fails but smoke + notebook execution still run.
• Add a model: drop a file under benchmarks/models/, register it.
  Phase coverage and CLI surface follow automatically.
• Add a phase: one test_<phase>.py parametrized off the registry.
• Bump pins: [benchmarks] is the perf-attribution surface;
  [project.dependencies] stays loose so downstream consumers see no
  change.

[FBumann]

It went so well that we can consider wiring up https://codspeed.io/. I would like to test it.
@FabianHofmann you would need to accept their app to be installed on the repo. But maybe a quick chat would be good

[FBumann]

Dogfooding: does this suite actually help during dev?

Short version: yes. I pointed it at a real, currently-unreleased linopy change — the CSR / freeze_constraints work ("30–120× faster matrix generation for direct solver APIs", "~10× faster to_highspy") — and it both refused a false positive and isolated the real effect.

What happened

1. No false positive. A cross-version sweep 0.6.7 0.7.0 --phase solver_handoff came back flat (~1.0×). Correct — the CSR work isn't in 0.7.0 (it's unreleased), so the suite reported no change instead of inventing one.
2. Why I used bench, not sweep (corrected — an earlier version of this comment blamed a highspy version mismatch; that was wrong). sweep pins highspy==1.13.1 into every per-version venv, so the solver is held constant across arms — there's no highspy confound within a sweep. The real blocker is that the CSR win is gated on the per-model flag freeze_constraints=True, and no registry model sets it (to_highspy doesn't auto-freeze — proven below: mutable 18 ms vs frozen 1 ms on the same linopy). So a stock sweep of 0.7.0 → master would run the mutable path in both arms and show ~nothing. Demonstrating it via sweep would mean adding a frozen-constraints model variant (plus a git/file spec for the unreleased "after" arm — sweep accepts those, and now labels the snapshot by the spec's ref/sha so e.g. git+…@<sha> writes a clean linopy-<sha>.json). bench was simply the quickest way to toggle freeze in-process: same linopy, same highspy, only the constraint representation differs.
3. End-to-end killed the caching confound. Building the model once and timing only the handoff hides the freeze cost (it's paid eagerly at add_constraints). So I measured build-only and the fused build+handoff too — and the parts add up (mutable 24+18 ≈ 44, frozen 35+1 ≈ 37), a nice consistency check on bench itself.

Results

Representative numbers (Apple M-series, highspy 1.12, the sparse many-term balance model at n_buses=300). Absolute ms are machine-dependent; the ratios and their direction reproduce.

measurement	mutable	frozen (CSR)
to_highspy matrix construction	18 ms	1 ms	~16–21× faster
model build (fresh)	24 ms	35 ms	~1.7× slower (freeze paid upfront)
build + handoff, end-to-end	44 ms	37 ms	~1.2× faster, grows per extra handoff

The magnitude tracks terms-per-row: dense few-term carbon-management ≈ 2.9×, basic ≈ 1.7×, this 300-term model ≈ 20× — exactly matching "for constraints with many terms". Net: freezing trades ~11 ms upfront build for ~17 ms saved per handoff → a single build+solve already wins slightly, and anything that touches the matrix again (LP write, re-export, iterative re-solve) compounds the win. "Frozen is slower" only holds if you build and never hand off — not a real workload.

Reproduce

bench_csr.py — self-contained, seeded, no pypsa

import numpy as np
import pandas as pd
import xarray as xr

import linopy
import linopy.io as lio
from benchmarks import bench


def build_sparse(n_buses: int, *, freeze: bool) -> linopy.Model:
    rng = np.random.default_rng(42)
    n_lines, n_time = n_buses, min(n_buses, 24)
    buses = pd.RangeIndex(n_buses, name="bus")
    lines = pd.RangeIndex(n_lines, name="line")
    time = pd.RangeIndex(n_time, name="time")
    bus_from = np.arange(n_lines)
    bus_to = (bus_from + 1) % n_buses

    m = linopy.Model(freeze_constraints=freeze)
    gen = m.add_variables(lower=0, coords=[buses, time], name="gen")
    flow = m.add_variables(lower=-100, upper=100, coords=[lines, time], name="flow")
    incidence = np.zeros((n_buses, n_lines))
    incidence[bus_to, np.arange(n_lines)] = 1
    incidence[bus_from, np.arange(n_lines)] = -1
    incidence_da = xr.DataArray(incidence, coords=[buses, lines])
    demand = xr.DataArray(rng.uniform(10, 100, size=(n_buses, n_time)), coords=[buses, time])
    m.add_constraints(gen + (flow * incidence_da).sum("line") == demand, name="balance")
    m.add_objective(gen.sum())
    return m


def report(label, rs):
    by = {r.label: r.stats["min"] for r in rs.results}
    print(f"{label}: mutable={by['mutable']*1e3:.0f}ms frozen={by['frozen']*1e3:.0f}ms "
          f"-> frozen {by['mutable']/by['frozen']:.1f}x")


N = 300
mut, frz = build_sparse(N, freeze=False), build_sparse(N, freeze=True)

report("handoff   ", bench.compare({
    "mutable": lambda: lio.to_highspy(mut, set_names=False),
    "frozen":  lambda: lio.to_highspy(frz, set_names=False)}, rounds=20))
report("build     ", bench.compare({
    "mutable": lambda: build_sparse(N, freeze=False),
    "frozen":  lambda: build_sparse(N, freeze=True)}, rounds=20))
report("end-to-end", bench.compare({
    "mutable": lambda: lio.to_highspy(build_sparse(N, freeze=False), set_names=False),
    "frozen":  lambda: lio.to_highspy(build_sparse(N, freeze=True), set_names=False)}, rounds=20))

PYTHONPATH=. python bench_csr.py   # needs a linopy with Model(freeze_constraints=...)

So the suite catches real wins, refuses false ones, and attributes them to the right regime (terms-per-row). The reason this particular one needed bench rather than sweep is that the win is gated on a per-model flag (freeze_constraints) no registry model exercises — bench toggled it in-process on one linopy + one solver. Once CSR ships in a release (and/or a frozen model variant is added to the registry), a plain cross-version sweep would capture it directly.

---

This investigation and write-up (including the measurements above) were produced with Claude Code.