Make it possible to run Pytest in parallel#1291
Open
mhucka wants to merge 19 commits intoquantumlib:mainfrom
Open
Make it possible to run Pytest in parallel#1291mhucka wants to merge 19 commits intoquantumlib:mainfrom
mhucka wants to merge 19 commits intoquantumlib:mainfrom
Conversation
Setting the seed in conftest.py like this makes it possible to run pytest in parallel with the `-n` option from pytest-xdist.
Some of the test code did not work with parallel pytest execution because it attempted to write to and delete the same files in the shared `src/openfermion/testing/data` directory. Specifically, `SaveLoadOperatorTest` and `MolecularDataTest` used fixed filenames:(like `test_file.data` and `dummy_molecule.hdf5`) which led to collisions and errors during parallel execution. The changes implemented here are: * Refactor `SaveLoadOperatorTest` in `operator_utils_test.py` to use a unique temporary directory for each test instance through the use of `tempfile.mkdtemp()`. * Refactor `MolecularDataTest` in `molecular_data_test.py` to use a temporary directory for tests that perform disk writes (`test_dummy_save` and `test_abstract_molecule`).
Although `check/pytest` accepts command line arguments, `pytest-and-incremental-coverage` does not, so we can't instruct contributors to add `-n auto` to it to run it in parallel. Since it's something that is less commonly run by contributors, I took the more expedient route of simply adding the parallelism flag directly to the script.
Now that parallel Pytest execution works for OpenFermion, we can tell users about it.
Turns out the use of the `pytest-randomly` plugin and using `np.random.default_rng` seems to make it possible to run pytest in parallel. Evidently, that seeds every test individually based on a master seed, and this solves the xdist collection issue automatically without creating inter-test dependencies.
Regenerate dependency files after adding pytest-randomly.
Belongs in a separate PR.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the project's contribution guidelines and test infrastructure to support parallel execution using pytest-xdist and mypy parallel flags. It also introduces pytest-randomly for better test isolation and updates numerous dependencies across the environment files. Test files molecular_data_test.py and operator_utils_test.py were refactored to use temporary directories for file-based tests, improving test cleanliness and isolation. In conftest.py, global random seeds were added to ensure deterministic test behavior. Feedback was provided regarding redundant calls to np.random.default_rng(0) in conftest.py, as these calls do not affect the global random state and their return values are unused.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes it possible to run parallel Pytest execution via the
pytest-xdistplugin.Set a fixed random seed for testing, in
src/openfermion/conftest.py. This is necessary forpytest-xdistworker processes to be able to perform identical test collections. Otherwise, different workers might generate different values and you end up with "Inconsistent collection" errors during execution.Change some tests that were writing to fixed-path files and folders. Those needed to be changed to use temporary directories so that parallel Pytest processes don't collide trying to write to the same things.
Add the pytest plugin
pytest-randomly, which executes tests in random order, as a way to help make sure that the changes above don't have some hidden assumption about order of execution.Update the requirements files by running the
re-pip-compile-in-docker.shscript (necessary because of the addition of thepytest-randomlypackage).Update
CONTRIBUTING.mdto mention the use of options for parallel testing.