You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While reviewing PR #71, I observed several unexpected test errors in the submitted results. Some of these errors appear to be caused by running pytest with multiple processes, while others appear to be incorrectly classified as failures, and may indicate issues in how test outcomes are handled or executed.
Expected Behavior
Tests that are not applicable under certain conditions (e.g., some operators have not yet been implemented for some platforms) should be marked as skipped, rather than reported as errors.
Running tests with different levels of parallelism (e.g., multiple workers) should not introduce unexpected errors.
Some level of non-determinism may occasionally introduce transient errors, but such errors should not be persistent.
Actual Behavior
Prior to chore: add pull request template #71, there are ~58 errors. In this PR, the submitted logs show ~5 errors. The reason for this reduction is unclear, but it is suspected to be due to environment changes (e.g., different PyTorch versions)
Using pytest -n 1 instead of multiple workers can reduce the number of errors reported;
The persistent errors should technically be classified as "skipped".
Logs / Screenshots
For instance, the final submitted nvidia.log that only uses a single worker (i.e., pytest -n 1):
=================================== FAILURES ===================================
______________________________ tests/test_cast.py ______________________________
[gw0] linux -- Python 3.10.16 /home/huangjiacheng/.venv/bin/python
worker 'gw0' crashed while running 'tests/test_cast.py::test_cast[cuda-input_dtype0-out_dtype0-0.001-0.001-shape0-None-None]'
_____________________________ tests/test_linear.py _____________________________
[gw1] linux -- Python 3.10.16 /home/huangjiacheng/.venv/bin/python
worker 'gw1' crashed while running 'tests/test_linear.py::test_linear[cuda-dtype0-0.01-0.05-False-False-False-a_shape0-b_shape0-out_shape0]'
______________________________ tests/test_cast.py ______________________________
[gw2] linux -- Python 3.10.16 /home/huangjiacheng/.venv/bin/python
worker 'gw2' crashed while running 'tests/test_cast.py::test_cast[cuda-input_dtype4-out_dtype4-0.01-0.005-shape4-None-None]'
_____________________________ tests/test_linear.py _____________________________
[gw3] linux -- Python 3.10.16 /home/huangjiacheng/.venv/bin/python
worker 'gw3' crashed while running 'tests/test_linear.py::test_linear[cuda-dtype0-0.01-0.05-False-False-False-a_shape1-b_shape1-out_shape1]'
______________________________ tests/test_cast.py ______________________________
[gw4] linux -- Python 3.10.16 /home/huangjiacheng/.venv/bin/python
worker 'gw4' crashed while running 'tests/test_cast.py::test_cast[cuda-input_dtype4-out_dtype4-0.01-0.005-shape3-None-None]'
================== xdist: maximum crashed workers reached: 4 ===================
=========================== short test summary info ============================
FAILED tests/test_cast.py::test_cast[cuda-input_dtype0-out_dtype0-0.001-0.001-shape0-None-None]
FAILED tests/test_linear.py::test_linear[cuda-dtype0-0.01-0.05-False-False-False-a_shape0-b_shape0-out_shape0]
FAILED tests/test_cast.py::test_cast[cuda-input_dtype4-out_dtype4-0.01-0.005-shape4-None-None]
FAILED tests/test_linear.py::test_linear[cuda-dtype0-0.01-0.05-False-False-False-a_shape1-b_shape1-out_shape1]
FAILED tests/test_cast.py::test_cast[cuda-input_dtype4-out_dtype4-0.01-0.005-shape3-None-None]
=========== 5 failed, 3108 passed, 1000 skipped in 191.93s (0:03:11) ===========
Description
While reviewing PR #71, I observed several unexpected test errors in the submitted results. Some of these errors appear to be caused by running
pytestwith multiple processes, while others appear to be incorrectly classified as failures, and may indicate issues in how test outcomes are handled or executed.Expected Behavior
Actual Behavior
pytest -n 1instead of multiple workers can reduce the number of errors reported;Logs / Screenshots
For instance, the final submitted
nvidia.logthat only uses a single worker (i.e.,pytest -n 1):But when using more than 1 worker: