Add validation + compatibility tests for DeepLabCut-live#56
Open
deruyter92 wants to merge 20 commits into
Open
Add validation + compatibility tests for DeepLabCut-live#56deruyter92 wants to merge 20 commits into
deruyter92 wants to merge 20 commits into
Conversation
deruyter92
changed the title
deruyter92
changed the title
C-Achard
approved these changes
Feb 27, 2026
C-Achard
reviewed
Feb 27, 2026
Contributor
There was a problem hiding this comment.
@deruyter92 If you'd like I can merge #55 right now, so then we can see how the tests go in CI ?
Contributor
Author
|
Yes great thanks. Sorry I see that I didn't install the pre-commit |
Contributor
|
Sorry, in the above I meant #55. It is now merged, happy to take care of conflicts/CI if you're busy ! |
20 tasks
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>
C-Achard
force-pushed
the
jaap/add_compatibility_tests
branch
from
54ee55e to
693c931
Compare
Remove the earlier pytest commands block and update the tox 'commands' to run pytest with the marker excluding both 'hardware' and 'dlclive_compat'. Adjust coverage invocation to use the installed package path (--cov={envsitepackagesdir}/dlclivegui) and emit per-env XML coverage files (.coverage.{envname}.xml). This aligns tox behavior with the GitHub Actions job and removes the prior posargs-based command duplication.
C-Achard
reviewed
Feb 27, 2026
Update the GitHub Actions testing matrix to run on Python 3.12 instead of 3.11. This moves CI to test against the newer Python runtime while keeping existing matrix include entries unchanged.
Contributor
Author
|
I see that some tests are failing still. I am happy to pick this up later. Importantly this shouldn't hold you back from releasing the bèta-test version. These validation measures can always be added later. |
Add a PoseBackends Enum and switch PoseSource.backend from a string to that enum for stronger typing and clarity. Update default PosePacket/PoseSource instances and the validate_pose_array signature to use PoseBackends.DLC_LIVE. Import Enum/auto and remove an unused sentinel variable. These are small refactors to improve type-safety and consistency around backend identification.
Relax and correct DLCLive compatibility checks: only consider keyword/positional params when inspecting __init__, drop the assertion that __init__ must accept **kwargs, and only require 'frame' for init_inference/get_pose (frame_time is passed as a kwarg to processors). Also update FakeDLCLive.get_pose to return an array of shape (2, 3) to match the expected pose output shape.
Set the test step shell to `bash -eo pipefail` and redirect `tox` stderr into `tee` (changed `tox -q` to `tox -q 2>&1 | tee tox-output.log`). This ensures pipeline failures are detected (pipefail) and that tox's stderr is recorded in `tox-output.log` for improved debugging in the CI workflow.
Replace ad-hoc DLCLive installs in the GitHub Actions job with tox-based testenvs. The workflow fixes Python to 3.12, installs required Qt/OpenGL runtime libs on Ubuntu, installs tox and tox-gh-actions, and runs tox -e matrix.tox_env. tox.ini was extended with dlclive-pypi and dlclive-github testenvs (pypi pinned and GitHub main respectively) to run the compatibility pytest, and the new envs were added to env_list to allow local and CI execution.
Contributor
|
@deruyter92 A few additions :
|
Add tests/compat/conftest.py to inject a stub torch module into sys.modules when torch is not installed. This prevents ImportError during DLCLive compatibility tests so the API can be validated without requiring torch to be installed. This is a pragmatic workaround and includes a note to remove or replace it once imports are properly guarded in the package.
Contributor
|
I also had to monkeypatch torch in the compat tests due to unguarded imports in DLCLive, see DeepLabCut/DeepLabCut-live#168 |
Replace QTimer.singleShot with a cancellable QTimer instance (self._camera_validation_timer) that is single-shot, connected to _validate_configured_cameras, and started with a 100ms delay. The closeEvent now stops the timer if it's still active to prevent validation from firing while the window is closing, avoiding modal QMessageBox races/crashes during tests/CI teardown.
C-Achard
added
DLClive
C-Achard
approved these changes
Mar 2, 2026
Contributor
There was a problem hiding this comment.
Approving again, will wait for your feedback next week @deruyter92 before merging as this is agreed not to be beta-critical; thanks again for all the valuable additions !
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds explicit pose-output contract validation for DeepLabCut-live results and introduces a CI-compatible test suite to verify DeepLabCut-live API/package compatibility across supported versions.
Changes:
- Added
PosePacket+validate_pose_array()to validate DLCLive pose outputs and emit a structured packet alongside pose data. - Introduced a
dlclive_compatpytest marker and new compatibility tests, plus tox environments to run them against PyPI and GitHub-main DLCLive. - Improved CI robustness and GUI teardown reliability (camera validation timer cancellation on close).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
dlclivegui/services/dlc_processor.py |
Adds pose schema/packet and pose array validation integrated into the processing pipeline. |
tests/services/test_pose_contract.py |
Adds unit coverage for pose-array validation (shape/dtype error cases). |
tests/conftest.py |
Updates FakeDLCLive pose shape to match the new (.., 3) pose contract. |
tests/compat/test_dlclive_package_compat.py |
Adds DLCLive import/signature checks and an opt-in smoke test. |
tests/compat/conftest.py |
Adds a torch stub to let compatibility tests run without torch. |
pyproject.toml |
Adds dlclive_compat marker definition. |
tox.ini |
Adds dlclive compatibility tox envs and excludes compat tests from default coverage run. |
dlclivegui/gui/main_window.py |
Replaces a one-shot timer with a cancellable timer and stops it in closeEvent. |
.github/workflows/testing-ci.yml |
Adds DLCLive compatibility job and improves test step robustness/logging. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dlc_processor: broaden validate_pose_array signature to accept source_backend as PoseBackends|str and add a check_finite option (default True) that raises on non-finite values to catch invalid pose outputs early. tests: make torch stubbing more robust by using importlib.util.find_spec in tests/compat/conftest.py; update tests/compat/test_dlclive_package_compat.py to refine _get_signature_params typing, import Parameter, and handle constructors that accept **kwargs more gracefully (provide clearer diagnostic when expected GUI kwargs may be passed through). tox.ini: add a [testenv:dlclive] passenv block for DLCLIVE test env vars and propagate that passenv to dlclive-pypi and dlclive-github envs so DLCLive compatibility tests pick up required environment variables.
Add helper functions to inspect method signatures (_signature_accepts_positional_after_self and _signature_accepts_keyword) and replace previous name-based parameter checks for DLCLive.init_inference and DLCLive.get_pose. Tests now verify that these methods accept a frame-like positional argument after self and that get_pose accepts frame_time as a keyword (or **kwargs). Also update an example in the docstring to match the new signature helper usage and improve assertion messages.
Update dlclive-github testenv to run compatibility tests against GitHub main without installing the local package. Adds a comment explaining the rationale, sets package = skip, adds pytest to deps, and inserts a small diagnostic python command to print the imported dlclive file and deeplabcut-live version before running the compatibility test.
Add --confcutdir=tests/compat to the pytest command in tox.ini for the dlclive compatibility test. This forces pytest to stop config discovery at the tests/compat directory so the compat tests run with their intended configuration and aren't affected by higher-level pytest configs.
Contributor
|
@deruyter92 Can we merge ? |
C-Achard
approved these changes
May 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
Since DeepLabCut-live-GUI heavily depends on DeepLabCut-live, but currenly it is implicitly assumed that that package provides the right output. It would be good to add some basic explicit validation of the returned output of DeepLabCut-live to check compatibility of the returned data. Also, including functional compatibility checks in CI for different DeepLabcut-live versions, allows to systematically tets the package compatibility before release.
Changes:
PosePacketa lightweight schema validation of the output from DeepLabCut-liveThis pull request introduces significant improvements to DLCLive compatibility testing and pose validation in both the codebase and CI infrastructure. The main changes include adding a dedicated compatibility test suite for DLCLive versions, implementing robust pose array validation, and updating the workflow and test environments to support these enhancements.
DLCLive compatibility testing:
tests/compat/test_dlclive_package_compat.pythat verifies DLCLive package importability, constructor signatures, required methods, and performs a minimal inference smoke test. This ensures that the GUI remains compatible with supported DLCLive versions.torchintests/compat/conftest.pyto allow compatibility tests to run even iftorchis not installed, improving test robustness.Pose validation and contract enforcement:
validate_pose_arrayfunction indlclivegui/services/dlc_processor.pyto enforce shape and dtype constraints on pose outputs, preventing downstream errors from invalid data.tests/services/test_pose_contract.py, covering both valid and invalid cases.Continuous integration and test environment updates:
dlclive-compatjob to.github/workflows/testing-ci.ymlto run compatibility tests against multiple DLCLive versions (PyPI and GitHub main), and updated tox configuration to include corresponding environments. [1] [2] [3]pyproject.tomlto includedlclive_compatand excluded these tests from the default coverage run. [1] [2]GUI camera validation timer improvements:
dlclivegui/gui/main_window.pyto use a cancellable QTimer attribute and ensure it is stopped incloseEvent, reducing test/CI errors related to GUI teardown timing. [1] [2]These changes improve reliability, maintainability, and future-proofing of the DLCLive integration and pose processing pipeline.