perf(int32): auto-widen labels to int64 on overflow instead of raising

[FabianHofmann]

I argue that auto-adjustment of dtypes is the way to go and not over-complex in the implementation. this allows user to go wild with their problem sizes.

Note

The following content was generated by AI.

Stacked on #566 (perf/int32). Answers the "auto-adjust the dtype once the boundary is hit" question from #566's review, replacing the hard ValueError overflow guard with automatic widening.

What changed

When add_variables / add_constraints would push the label counter past the int32 maximum (~2.1 billion), instead of raising, options["label_dtype"] is widened to int64:

• Monotonic — only ever int32 → int64, never back.
• Sticky — survives options.reset() (a new OptionSettings.widen_label_dtype() moves the default too), so once any model in the process mints a >2^31 label, int64 becomes the floor everywhere for the rest of the process. Revert explicitly with options["label_dtype"] = np.int32.

Why sticky-global rather than per-model

The four float→int cast sites in expressions.py / common.save_join run during expression building and have no reliable model reference (expressions can be detached). A per-model dtype would leave those sites able to silently narrow an already-int64 label back to int32; guarding them with a per-array .max() would tax the exact build path #566 speeds up. Reading the process-global keeps them correct at zero hot-path cost, and the monotonic+sticky invariant guarantees no array is ever asked to narrow int64 values.

Previously-allocated int32 batches stay valid (they only ever held small values) and upcast to int64 on concat — verified that a widened label 2147483648 survives a variables.flat / constraints.flat round-trip without wrapping.

Trade-off

A process that builds one >2.1B-label model then many small ones keeps int64 labels throughout (losing the int32 memory/speed win for the small ones). Documented in the release note and user-overridable.

Tests

test/test_dtypes.py: the two test_overflow_guard_* tests (which asserted ValueError) are replaced with test_auto_widen_variables / test_auto_widen_constraints (assert labels become int64 and the option flips) and test_auto_widen_is_sticky (widen survives reset). A restore_label_dtype fixture resets the sticky global so the widen tests don't leak into the rest of the suite.

[FBumann]

@FabianHofmann I would say auto-widening should stay a convenience. Users should get a Warning.
Its recommended to set the dtype explicitly if large models are expected.
This will probably be faster, need less memory and cant lead to upcasting issues.

[FabianHofmann]

@FBumann good point, doing this now

[codspeed-hq]

Merging this PR will not alter performance

✅ 173 untouched benchmarks
⏩ 173 skipped benchmarks¹

---

_{Comparing perf/int32-auto-widen (e215297) with master (4ca8d5a)}

Footnotes

1. 173 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[FBumann]

We need to rerun the codpseed after master finished with the original in32 PR

[FabianHofmann]

We need to rerun the codpseed after master finished with the original in32 PR

do you know how to retrigger it?

[FBumann]

@FabianHofmann pushing an empty commit

[FBumann]

@FabianHofmann I think if we want to have an auto-widening that doesnt risk silent corruption. Therefore we need to store the labels dtype on the model, read from global options on creation, and optionally overwritable.

Without it we will have issues with:

• uncatchable errors when a user reverts the global option while an int64 Model lives
• pickling and reloading a Model with int64

Also, Expressions actually always have a related Model (cant be detachted!)

[FBumann]

I think we should go for a per Model dtype instead. See #828