Add Xpress direct and persistent solver interfaces#3987
Open
XPRSc4v4 wants to merge 6 commits into
Open
Conversation
* TXP-8770: add new Xpress direct solver (LP/MIP core) for new Pyomo solver interface
* TXP-8770: optimize and clean up Xpress direct solver post-review
* TXP-8770: add Xpress-specific test suite covering controls, labels and error paths
* TXP-8770: add warmstart, solver stats and parity/beyond-Gurobi improvements
* TXP-8770: tighten test assertions and add solver marker
* TXP-8771: add XpressPersistent solver and refactor connector shared code
New xpress_base.py (replaces xpress_direct_base.py):
- EntityMaps dataclass: groups id(VarData)->xp.var, ConstraintData->list[xp.constraint],
SOSConstraintData->xp.sos stable handle maps; avoids integer index rebuilds after deletions
- XpressSolutionLoaderBase: unified loader with _fetch_var_values helper, get_duals
walking handle spans with max-abs for 2-row range constraints
- build_loadmip_arrays, extract_sos_arrays, get_rhs_and_sense: shared compile utilities
- build_xpress_problem: full compile pipeline; deactivates SOSConstraint components before
LinearStandardFormCompiler (which rejects them) and reactivates in a finally block
- XpressSolverMixin: solve scaffold and _create_xpress_model dispatch extracted from
XpressDirect, now shared with XpressPersistent
New xpress_persistent.py:
- XpressPersistent: reuses xp.problem() across solves via ModelChangeDetector observer
- Incremental updates: _update_variables (add/remove/modify, fix/unfix reprocesses rows),
_update_constraints (remove+re-add on expr change), _update_sos_constraints,
_update_objectives (added/removed/sense), _update_parameters
- _initializing flag suppresses callbacks during ModelChangeDetector initial snapshot
- Fixed variable handling: compiler folds fixed vars into constants; extra zero-coefficient
columns registered so unfix triggers correct row reprocessing
- Bulk API: chgBounds with interleaved L/U layout via np.repeat/np.tile, chgColType with
np.int8 array, addRows/addSets with pre-allocated np.int64 start arrays
- Public persistent API delegates to change detector to keep observer callbacks in sync
xpress_direct.py: reduced to ~30 lines delegating entirely to build_xpress_problem
xpress_direct_base.py: deleted, superseded by xpress_base.py
test_xpress_direct.py:
- TestXpressPersistentObjective: objective removal between solves (Reason.removed path)
- TestXpressPersistentSOS: SOS1 via set_instance + incremental remove via del
- load_vars/get_vars/get_reduced_costs subset tests for direct solver loader
test_solvers.py: XpressPersistent added to all_solvers and mip_solvers
* TXP-8771: Phase A -- eager invalidation, symbolic_solver_labels propagation, write(), multi-active-obj hardening
* TXP-8771: add XpressPersistentConfig with auto_updates and wire through ModelChangeDetector
* TXP-8771: mutable-param helpers -- remove loadMIP from set_instance, incremental build, chgMCoef batching, minimal helper state
* TXP-8771: type annotations, update_parameters signature fix, and minor cleanups across connector
* TXP-8771: handle accessors, release/reset, fix public API bugs, fill test coverage to 95%
* TXP-8771: fix real LSP errors; suppress false positives from Pyomo untyped API
* TXP-8771: fix TC.solverFailure bug (use TC.error), proper cast() for value() return type, str() for Path arg
* TXP-8771: add get_xpress_problem() for full Xpress API access
* TXP-8772: QP/QCP support and connector refactor
Quadratic support:
- _quad_arrays() encodes both Xpress scaling conventions (diag_scale=1 for
addQMatrix constraints, diag_scale=2 for chgMQObj 0.5*x'Hx objective)
- _add_constr_impl calls addQMatrix for QCP constraints after addRows
- _set_objective_impl calls chgMQObj for QP objectives
- _MutableQuadConstraint: pre-computed handles + 0.5 off-diagonal scaling;
rebuild() does delQMatrix+addQMatrix on param change
- _MutableObjective extended with quadratic state; diagonal coefs pre-scaled
by 2 at construction so update() calls value() unconditionally
Compilation: LinearStandardFormCompiler replaced by per-constraint
generate_standard_repn(quadratic=True, compute_values=False). Fixed variables
become explicit lb=ub columns. EntityMaps.cons holds a single xp.constraint
handle per row; range constraints use Xpress 'R'-type rows via get_rhs_and_sense().
Shared _impl statics on XpressSolverMixin: _add_vars_impl, _add_constr_impl,
_add_sos_impl, _set_objective_impl, _compile_constraint_repns,
_compile_objective_repn (unified NL guard), _xp_obj_sense.
Mutable helpers consume repn directly with no intermediate lists; _xp_con
passed via constructor. _update_parameters merged to single pass over
affected_cons. _populate_results uses lazy-built status lookup dicts.
_update_objectives uses get_objective() and skips the active-obj scan when
no objective was added.
Tests: 18 new tests across QP/QCP, edge cases (empty model, constant objective,
range constraints, mutable coef path, fix/unfix rebuild, SOS-only block).
Coverage: xpress_direct 100%, xpress_persistent 99%, xpress_base 95%.
* TXP-8774: NLP support, unified walker, type annotations, and correctness fixes
Core: replace generate_standard_repn-based LP/QP-only path with
XpressExpressionWalker (StreamBasedExpressionVisitor) that handles linear,
quadratic, and nonlinear expressions uniformly. Variables now registered
lazily via _register_variable (addVariable with lb/ub/type in one call),
eliminating the pre-scan and duplicate-column bug in SOS handling.
Pyomo-to-Xpress maps (_OBJ_SENSE_MAP, _SOL_STATUS_MAP, _STOP_TYPE_MAP,
_XP_FUNCTION_MAP, _VAR_XP_TYPE_MAP) consolidated into _init_xpress() called
once at module load. NL functions and their decompositions (sinh/cosh/...)
merged into a single _XP_FUNCTION_MAP; AbsExpression unified with other
unary functions. Exit handlers refactored with _make_binary_const,
_make_binary_general, _make_variadic_general, _flag_mutable, _nl_unary to
eliminate repetition. Type annotations added throughout (ExpressionBase,
UnaryFunctionExpression, _NodeResult alias, visitor: XpressExpressionWalker).
Bug fixes: mutable param sign error in NegationExpression contexts (fixed
via NegationExpression in _DEPTH_NODES); mutable body constants not tracked
for RHS updates (fixed is_mutable detection and _has_mutable_nonlinear flag);
_update_parameters single-pass ordering could apply chgMCoef to a stale
integer row index after delConstraint shifted rows (fixed with two-pass: NL
rebuilds first, linear collects after).
Tests: 208 pass. New tests cover NL/quadratic/linear param interactions in
all row orderings (including add/remove/re-add cycles), SOS duplicate-column
and KeyError bugs, and walker unit coverage for NLP expressions.
* TXP-8774: mutable tracking refactor -- _coef_acc/chgRHS/body-const, walker perf, bug fixes, new tests
* TXP-8775: register Xpress in NLP/QCP test suite, add has_instance(), add warmstart config flag
* TXP-8775: fix mutable quadratic objective with mixed linear coefs; register Xpress in QCP/NLP test suites
* TXP-8775: add xpress_persistent to nlp_solvers -- NLP Persistent not available in Gurobi
* TXP-8775: coverage 93->97%; new tests for edge cases, has_instance, NLP Persistent registration
* TXP-8774: track_mutable=False for XpressDirect -- skip mutable tracking overhead
* TXP-8774/8771: quadratic coef tracking, affine expansion, optimizations, coverage, and docs
* TXP-8774: walker simplification + ExternalFunction support; mutable tracking moved to persistent
xpress_base.py -- removed ~800 lines:
- _NodeResult (ExprType compound tuple), WalkerResult dataclass, _coef_acc/_coef_stack
mechanism, track_mutable flag, walk_linear, finalizeResult all deleted
- all ExprType-keyed exit handlers (_exit_negation/sum/product_*/pow_*/division_*/
unary_const, _define_exit_handlers) deleted
- _expand_affine_product, _before_pow, _before_product_bilinear, _extract_affine_form,
_flag_mutable, _track_mutable_lin_coef, _track_mutable_const and helpers deleted
- ExitNodeDispatcher, ExprType, initialize_exit_node_dispatcher dropped from imports
- walker is now a pure xp expression builder; exit dispatch replaced by flat
_EXIT_HANDLERS (_ExitHandlerMap dict with MRO fallback for unknown subclasses)
- _before_leaf unifies _before_param/_before_npv/_before_native_*; domain errors
propagate as ValueError -- no silent nan from try-except in _before_npv
- _exit_external_function: ExternalFunctionExpression -> xp.user; supports _fcn/
_grad/_fgh callbacks; unique __name__ per PythonCallbackFunction avoids name clash
- SolStatus.FEASIBLE added to _SOL_STATUS_MAP for SLP local-optimum (COMPLETED+FEASIBLE)
xpress_persistent.py: mutable tracking moved here, based on generate_standard_repn;
_UpdateBatch gains nl_old/new_cons for batched NL del+add; _MutableConstraint NL path
xpress_direct.py: API alignment (use_names, plain result object, no track_mutable)
tests: 5 new ExternalFunction tests; walker and direct tests updated for new interfaces
* TXP-8775: solution pool, test coverage overhaul, hyperbolic decomposition, tolerance tightening to places=6
* TXP-8775: test suite overhaul for the Xpress connector
- Add _solve_and_check helper (TypedDict expected, mandatory obj+vars for
optimal, nvariables() completeness check) and split shared utilities into
_xpress_test_utils.py; split test_xpress_direct.py into separate
direct/persistent files
- Tighten all assertAlmostEqual tolerances to places=6 (matching Xpress
internal precision); redesign degenerate tests with unique optimal solutions
- Parametrize 8 structurally identical trig/hyperbolic walker tests into one
@parameterized.expand method; add _trivial_model, _solve_lp_no_load, and
_solve_check_mutate_check helpers to reduce per-test boilerplate
- Reduce .solve( calls from 163 to 34 (79%) by replacing solve+assert
sequences with _solve_and_check throughout direct and persistent suites
* TXP-8775: add Sphinx documentation for XpressDirect and XpressPersistent connectors
* TXP-8775: walker NLP formula assertions, IIS support, pool test split, bug comment fixes
- Add NLP formula checks to _walk_and_assert (column indices, scalar CON tokens,
operator IFUN/OP tokens via nlpGetFormula); replace product-of-NL*linear tests
with product-of-NL*NL to avoid the SLP coefficient-formula storage path where
nlpGetFormula returns EOF by design
- Add write_iis() and get_iis() to XpressPersistent; write_iis writes the IIS
in LP format, get_iis returns conflicting Pyomo ConstraintData and VarData objects
mapped via _maps; 3 tests added to TestXpressPersistentIIS
- Split TestXpressSolutionPool into test_xpress_pool.py (shared direct/persistent
feature, no longer tied to the direct-only test file)
- Fix stale comment on test_sos1_vars_not_in_objective (bug fixed by guard in
_create_xpress_model; comment was describing the old broken behavior)
* TXP-8775: _assemble_xp_expr refactor, pool simplification, stable_xp mutation fix
- Extract _assemble_xp_expr helper shared by _make_xp_con, _MutableObjective.update,
and _build_xp_from_repn; uses xp.Sum([...]) to avoid aliasing stable_xp
- Fix stable_xp mutation bug: xp.expression += float mutates in place for multi-term
expressions; _MutableObjective now builds expression without aliasing stable_xp
- pool_solutions: drop negative-N rolling-window variant; N>=0 always keeps last N
solutions (NonNegativeInt domain); serializepreintsol=1 for determinism
- Update config descriptions and test expectations to match new semantics
- Fix test_time_limit_zero to assert timelimit instead of removed worklimit branch
- Update hyperbolic scalar expectations after /2 -> *0.5 substitution
* TXP-8775: update pool_solutions docs to reflect rolling-window-only API
* TXP-8775: add symbolic_labels support for SOS set names via addNames
Co-authored-by: Claude <noreply@anthropic.com>
- Keep Xpress solver registrations alongside new SCIP registrations in plugins.py - Use places=3 for NLP log test assertions (matches upstream) - Fix stale doc string: "Direct interface to Xpress" (not scipy-based) Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes # .
Summary/Motivation:
Adds two new solver interfaces for FICO Xpress under
pyomo/contrib/solver/solvers/xpress/:XpressDirect-- builds a freshxp.problem()per solveXpressPersistent-- reusesxp.problem()across solves, with efficient incremental updates viaModelChangeDetectorBoth interfaces support LP, MIP, QP, QCP, and NLP problems.
Changes proposed in this PR:
xpress/solver package with shared base, direct, and persistent implementationsxpress_directandxpress_persistentinplugins.pytest_solvers.py) for LP/MIP/QP/QCP/NLP problem classesAI-Use Disclosure
or
AI tools contributed to the development of this PR
Review process (select ONE):
Notes for reviewers (optional):
Legal Acknowledgement
By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution: