Skip to content

refactor(int32): per-model label dtype instead of sticky process-global#828

Open
FBumann wants to merge 5 commits into
perf/int32-auto-widenfrom
perf/int32-model-label-dtype
Open

refactor(int32): per-model label dtype instead of sticky process-global#828
FBumann wants to merge 5 commits into
perf/int32-auto-widenfrom
perf/int32-model-label-dtype

Conversation

@FBumann

@FBumann FBumann commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

This implements int dtype control per Model instead of through a global config option (global kept).
This resolves all potential issues with the global for little complexity.
I think this is the right approach.

Note

The following content was generated by AI.

Stacked on #822. Keeps its auto-widening behavior but moves the label dtype from the sticky process-global option onto the Model itself.

What changed

  • The label dtype becomes a Model() argument: Model(label_dtype=np.int64), default np.int32, exposed as Model.label_dtype. The linopy.options["label_dtype"] setting is removed entirely — embedders forward constructor kwargs instead (PyPSA: n.optimize(model_kwargs={"label_dtype": np.int64})), so config.py returns to its pre-perf: default integer arrays to int32 for ~25% memory reduction #566 state.
  • A model widens only itself to int64 when its labels would exceed the int32 maximum; the UserWarning recommends passing label_dtype=np.int64 upfront.
  • All cast sites read the dtype from their object's model reference; common.save_join takes it as an explicit fill_dtype parameter instead of reading the global.
  • read_netcdf restores the widened dtype per model (from the counters, as before). Pickle round-trips preserve it automatically since it is an instance attribute.
  • OptionSettings.widen_label_dtype() and its sticky-reset() semantics are removed, along with the test-suite fixture that guarded against global-state leakage.

Why per-model

#822 chose the process-global because "the four float→int cast sites … have no reliable model reference (expressions can be detached)". That premise doesn't hold in the current codebase: BaseExpression.__init__ raises unless model is a Model instance, and every label_dtype cast site is a method on an object with a validated model reference (BaseExpression, Variable, Variables, Constraints, CSRConstraint). The only model-less site, save_join, is called with integer_dtype=True from exactly three container properties that all have self.model in scope.

With a reliable model reference, a per-model monotonic widen gives the same no-narrowing guarantee at the same zero hot-path cost (an attribute read), and removes the global design's edge cases:

  • No revert footgun. Under the global design, following the documented revert (options["label_dtype"] = np.int32) while a widened model is alive lets the cast sites silently narrow >2^31 labels — int64 → int32 .astype() wraps to negative values without any error. Per-model, there is nothing to revert.
  • Widening travels with the model. A widened model shipped to another process via pickle/cloudpickle (dask, multiprocessing) keeps its dtype; the global flag does not cross process boundaries.
  • No cross-model contamination. perf(int32): auto-widen labels to int64 on overflow instead of raising #822 documents that one huge model permanently costs every later small model in the process its int32 memory/speed win. Per-model, small models are unaffected.
  • One knob in one place: the constructor argument. Nothing needs test isolation.
Cast-site → model-reference verification
Site Enclosing scope Model access
expressions.py vars cast in init BaseExpression.__init__ model param, enforced isinstance(model, Model) (check moved above the cast)
expressions.py sanitize BaseExpression self.model
expressions.py simplify (2×) LinearExpression self.model
variables.py ffill/bfill/sanitize Variable self.model (init enforces isinstance(model, Model))
variables.py flat remap Variables self.model
constraints.py _to_dataset (2×) CSRConstraint self._model
constraints.py flat remap Constraints self.model
common.py save_join free function dtype passed by callers (Variables.labels, Constraints.labels, Constraints.vars)

Tests

test/test_dtypes.py: the widen tests assert other models stay int32; new test_label_dtype_init_arg (+ invalid-dtype rejection), test_widen_applies_to_expressions, test_auto_widen_survives_pickle; the netcdf test asserts the loaded model is widened. The restore_label_dtype fixture and the option tests are gone. Full suite passes.

🤖 Generated with Claude Code

FBumann and others added 5 commits July 14, 2026 15:41
…ss-global

Store the label dtype on the Model (snapshot of options['label_dtype'] at
creation, exposed as Model.label_dtype) and widen only that model to int64
on overflow. Every cast site reads the dtype from its object's model
reference; save_join takes it as an explicit fill_dtype parameter.

This removes the sticky global state: the option and other models are
unaffected by one model's widening, reset() keeps its usual semantics,
and the widened dtype travels with the model through netCDF and pickle
round-trips instead of relying on process-global stickiness.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Model(label_dtype=np.int64) is more direct than mutating the global
option before construction; None falls back to options['label_dtype']
so embedders that construct the Model internally keep a process-wide
default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract Model._allocate_labels (shared by variable and constraint label
assignment) and config.validate_label_dtype (shared by the option setter
and Model.__init__).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… argument

The option predated the constructor argument; with per-model storage it
only duplicated the knob. Embedders forward constructor kwargs (PyPSA:
n.optimize(model_kwargs={'label_dtype': np.int64})), so no process-wide
setting is needed. config.py returns to its pre-int32 state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant