PyROS Add caching for computed uncertain parameter bounds

[jas-yao]

Fixes

_fbbt_parameter_bounds in uncertainty_sets.py

Summary/Motivation:

PyROS solves optimization bounding problems for every uncertain parameter multiple times throughout its routine using _compute_exact_parameter_bounds. There are up to 4 instances of PyROS accessing this method during its runtime.

1. Validation in is_bounded. This occurs when no parameter bounds are provided and FBBT fails to find bounds. Only the bounds that FBBT has not found are evaluated.
2. Validation in is_nonempty. This occurs for intersection, polyhedral, and custom uncertainty sets, where a feasibility problem is constructed.
3. Preprocessing in get_effective_uncertain_dimensions if parameter bounds are not provided or the provided ones are not exact.
4. Separation problem construction in add_uncertainty_set_constraints if no parameter bounds are provided.

The time taken to repeatedly solve these bounding problems may add up and be significant for larger uncertainty sets.

This PR adds a method for caching the solutions of these bounding problems so that subsequent processes do not need to solve the bounding problems again.

This PR also fixes a small bug in _fbbt_parameter_bounds where the returned bounds are not a float (e.g., a binary variable m.Var = pyo.Var(within=pyo.Binary, bounds=(0,1)) or a binary variable in a model that FBBT has been used on has lower bound max(0,0) and upper bound min(1,1)).

Changes proposed in this PR:

• Add a _solve_bounds_optimization method that uses functools.cache to cache solutions for bounding problems solved for any uncertain parameter that is used by _compute_exact_parameter_bounds
• Add a line for clearing the _solve_bounds_optimization cache during validation, which is run at the start of every PyROS solve.
• Add tests for checking caching behavior of _solve_bounds_optimization
• Fix the issue in _fbbt_parameter_bounds

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[jas-yao]

@shermanjasonaf @jsiirola

[jsiirola]

This looks pretty good. Some questions and a couple minor suggestions.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.12%. Comparing base (dccdbdd) to head (4ebca37).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3877   +/-   ##
=======================================
  Coverage   90.11%   90.12%           
=======================================
  Files         905      905           
  Lines      107502   107522   +20     
=======================================
+ Hits        96878    96899   +21     
+ Misses      10624    10623    -1     

Flag	Coverage Δ
builders	29.13% <18.75%> (+<0.01%)	⬆️
default	86.46% <100.00%> (?)
expensive	35.52% <18.75%> (?)
linux	87.61% <100.00%> (-2.01%)	⬇️
linux_other	87.61% <100.00%> (+<0.01%)	⬆️
oldsolvers	28.07% <18.75%> (+<0.01%)	⬆️
osx	82.99% <100.00%> (+<0.01%)	⬆️
win	85.53% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shermanjasonaf]

Good PR. I have a few questions about edge cases and testing.

[jsiirola]

@jas-yao: we are adopting a new review process where we convert PRs that are "waiting on the author" back to "draft" (to signal the PR state to both the author and the dev team). Once you have had a chance to address the comments, please mark it as "ready for review" so the developers can get it back into the review queue.

[shermanjasonaf]

@jas-yao @jsiirola As PR #3927 has been merged ahead of this PR, we will need to add bounds caching for the new CartesianProductSet class here. Also, the PyROS version number and changelog were not updated ahead of merging #3927. I have opened #3951 to address that:

1. If it makes sense to have separate version numbers for Add CartesianProductSet to PyROS #3927 and this PR, then let's wait until Update PyROS version number and changelog after #3927 #3951 is merged before updating the version number and changelog in the present PR.
2. Otherwise, we can close Update PyROS version number and changelog after #3927 #3951 and perform a single version number + changelog bump here.

[blnicho]

@shermanjasonaf I vote for option 2 and doing a single update to the PyROS version and changelog

[shermanjasonaf]

After the more recent changes, this PR looks good overall. I have comments on functionality for IntersectionSet and CartesianProductSet cases. I also have comments on testing and several suggestions on documentation.

@jas-yao To address the second question of your earlier comment: I do not think that we should make UncertaintySet.parameter_bounds return the results of UncertaintySet._compute_exact_parameter_bounds() by default, for a few reasons:

• _compute_exact_parameter_bounds() formally requires the user to provide a solver, whereas parameter_bounds does not.
• parameter_bounds is a property (basically an attribute), so we may want the calculations performed within to be somewhat quick. In contrast, _compute_exact_parameter_bounds() is generally slow when the bounds are not cached, due to the time required to construct and repeatedly solve the bounding model.
• parameter_bounds is public (and therefore accessible). If the default behavior of parameter_bounds is changed in the suggested way, then accessing parameter_bounds results by default in population of the cache, a protected attribute that must be clear(ed) ahead of calling PyROS.solve().

[shermanjasonaf]

Isn't it possible that a_set._cache gets populated here, particularly for cases where calling a_set.validate() entails calling a_set._compute_exact_parameter_bounds()? Do we need to be careful to clear a_set._cache here?

[jsiirola]

@shermanjasonaf @jas-yao: I pushed the updated cache context manager. Are we good to "final review" and merge? (@shermanjasonaf: are you satisfied with how caching interacts with the CartesianProductSet, or do you want to wait for additional testing?)

[shermanjasonaf]

@jsiirola @jas-yao I think we are close to merging. The new cache context manager works nicely. I am satisfied with how caching interacts with the CartesianProductSet; my comments include suggestions for additional testing. We may also want a similar custom implementation for IntersectionSet._cache_manager(), as bounds caching of the operand UncertaintySet attributes may also occur in some (seemingly rare) cases.

[shermanjasonaf]

This works nicely. Should IntersectionSet._cache_manager() be implemented similarly, to ensure that the bounds caches of an IntersectionSet's operand UncertaintySet objects are also cleared?

[jas-yao]

Yes, that makes sense to me. I have added this.

[shermanjasonaf]

I would add here a test based on a problem with a CartesianProductSet, to formally check that CartesianProductSet._cache_manager() works according to expectations. Something like:

diff --git a/pyomo/contrib/pyros/tests/test_grcs.py b/pyomo/contrib/pyros/tests/test_grcs.py
index 410593c82..8f577d879 100644
--- a/pyomo/contrib/pyros/tests/test_grcs.py
+++ b/pyomo/contrib/pyros/tests/test_grcs.py
@@ -75,6 +75,7 @@ from pyomo.contrib.pyros.uncertainty_sets import (
     FactorModelSet,
     Geometry,
     IntersectionSet,
+    PolyhedralSet,
     UncertaintyQuantification,
     UncertaintySet,
 )
@@ -5468,6 +5469,71 @@ class TestPyROSCacheUncertaintySetBounds(unittest.TestCase):
         self.assertAlmostEqual(interval._cache[0, minimize], 25, places=2)
         self.assertAlmostEqual(interval._cache[0, maximize], 200, places=2)
 
+    def test_solve_cartesian_product_set_bounds_cache(self):
+        """
+        Test management of uncertainty set bounds caches
+        is carried out as expected in the context of a PyROS solve.
+        """
+        # deterministic model
+        m = ConcreteModel()
+        m.q = Param(range(4), initialize=0, mutable=True)
+        m.x = Var(initialize=0, bounds=(0, 100))
+        m.obj = Objective(expr=m.x, sense=minimize)
+        m.con = Constraint(expr=m.x >= sum(m.q.values()))
+
+        # uncertainty set(s)
+        poly_set = PolyhedralSet(
+            # this is just the cube [-1, 1]^3
+            lhs_coefficients_mat=np.vstack([np.eye(3), -np.eye(3)]),
+            rhs_vec=[1] * 6,
+        )
+        # inclusion of PolyhedralSet means bounds caching takes place
+        cpset = CartesianProductSet([BoxSet([[0, 1]]), poly_set])
+        res1 = SolverFactory("pyros").solve(
+            model=m,
+            first_stage_variables=m.x,
+            second_stage_variables=[],
+            uncertain_params=m.q,
+            uncertainty_set=cpset,
+            local_solver=SolverFactory("ipopt"),
+            global_solver=SolverFactory("ipopt"),
+            objective_focus="worst_case",
+            solve_master_globally=True,
+        )
+
+        # check caches cleared
+        self.assertEqual(cpset._cache, {})
+        self.assertEqual(cpset._all_sets[0]._cache, {})
+        self.assertEqual(cpset._all_sets[1]._cache, {})
+        # check results: worst case objective is just sum of the
+        #                uncertainty set upper bounds
+        self.assertEqual(res1.iterations, 2)
+        self.assertAlmostEqual(res1.final_objective_value, 4.0, places=6)
+        self.assertAlmostEqual(m.x.value, 4.0, places=6)
+
+        # expand the polyhedralset to the cube [-2, 2]^3
+        poly_set.rhs_vec = [2] * 6
+        # solve again. since caches cleared, PyROS should work normally
+        res2 = SolverFactory("pyros").solve(
+            model=m,
+            first_stage_variables=m.x,
+            second_stage_variables=[],
+            uncertain_params=m.q,
+            uncertainty_set=cpset,
+            local_solver=SolverFactory("ipopt"),
+            global_solver=SolverFactory("ipopt"),
+            objective_focus="worst_case",
+            solve_master_globally=True,
+        )
+        # check caches cleared
+        self.assertEqual(cpset._cache, {})
+        self.assertEqual(cpset._all_sets[0]._cache, {})
+        self.assertEqual(cpset._all_sets[1]._cache, {})
+        # results have changed, since the polyhedral set was expanded
+        self.assertEqual(res2.iterations, 2)
+        self.assertAlmostEqual(res2.final_objective_value, 7.0, places=6)
+        self.assertAlmostEqual(m.x.value, 7.0, places=6)
+
 
 if __name__ == "__main__":
     unittest.main()

[jas-yao]

I have added this. I renamed test_solve_cartesian_product_set_bounds_cache to test_solve_cartesian_product_and_intersection_set_bounds_cache and added a couple additional checks for IntersectionSet.

[jas-yao]

@jsiirola @shermanjasonaf , I've taken a look at the caching behavior with the latest updates and everything works as expected. I added the suggestions made by @shermanjasonaf for IntersectionSet._cache_manager() and tests.

Looks good to me for merging, pending any final feedback on my latest changes.