perf(tests): reduce runtime and GPU memory

[yuecideng]

Description

This PR reduces unit-test wall time and GPU memory pressure by separating pure-Python, real-simulation, distributed-GPU, and remaining GPU workloads according to their resource needs.

It also removes redundant simulation cleanup waits, defers task discovery and renderer initialization until required, keeps mock-only tests out of the simulation lane, marks expensive coverage explicitly, and prevents cuRobo's process-wide TF32 setting from contaminating later solver tests. No dependencies are changed.

Measured impact

GPU memory was sampled every 250 ms from each pytest process tree with nvidia-smi.

Workload	Result	Wall time	Peak / GPU	Peak aggregate
Baseline non-doc, GPU-skipped serial suite	1122 passed, 106 skipped	28m 10.5s	not sampled	not sampled
Optimized pure + real-simulation lanes	1122 passed, 8 skipped	18m 52s	9,171 MiB	9,171 MiB
Isolated distributed GPU lane	1 passed	31.64s	5,543 MiB	10,748 MiB
Remaining serial GPU lane	89 passed, 12 skipped	5m 54.45s	8,853 MiB	8,853 MiB
Full staged run, including docs/import check	1224 passed, 20 skipped	about 25m 30s	9,171 MiB	10,748 MiB

For comparable non-GPU coverage, wall time drops by about 9m 18.5s (33.0%). Isolating distributed training lowers the observed aggregate GPU peak from 12,709 MiB to 10,748 MiB (15.4%), while the pure-Python xdist lane consumes no GPU memory.

Type of change

• Enhancement (non-breaking change which improves an existing functionality)

Screenshots

Not applicable.

Validation

• black . — 600 files unchanged
• Full staged suite: docs, four-worker GPU-hidden pure tests, serial real-simulation tests, isolated distributed GPU test, and remaining serial GPU tests
• cuRobo followed by SRS CUDA regression: 8 passed
• SRS CPU/CUDA regression: 10 passed
• Online data and SimManager regression: 60 passed
• Latest-main overlap regression (SimManager, EmbodiedEnv, material initialization): 59 passed
• Articulation CPU regression: 21 passed
• Workflow YAML parse and git diff --check passed (actionlint was not installed in the local image)

Checklist

• I have run the black . command to format the code base.
• Documentation changes are not required for this test/CI-only behavior change.
• I have added tests that prove the cleanup fix is effective and run proportional regression coverage.
• Dependencies are unchanged.

[greptile-apps]

Greptile Summary

The PR reorganizes the test suite into resource-aware CI lanes and defers simulation/task initialization to reduce runtime and GPU pressure.

• Adds dynamic simulation classification, xdist grouping, explicit resource markers, and lazy task discovery.
• Avoids unnecessary simulation cleanup waits and isolates distributed GPU coverage.
• Restores process-wide TF32 settings after cuRobo tests.
• Makes small tensor-conversion and SRS solver compatibility improvements.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code failure identified.

The resource-aware workflow still provides lanes for the current test categories, and the simulation cleanup, lazy initialization, TF32 restoration, and solver changes preserve their relevant runtime contracts.

Important Files Changed

Filename	Overview
.github/workflows/main.yml	Splits tests into documentation, parallel pure-Python, serial simulation, isolated distributed-GPU, and remaining GPU lanes.
tests/conftest.py	Defers task and renderer initialization, dynamically classifies simulation tests, and groups tests for resource-aware xdist execution.
embodichain/lab/sim/sim_manager.py	Makes cleanup queue draining race-safe and avoids waiting when no destruction task was pending.
embodichain/lab/sim/objects/articulation.py	Detaches drive tensors before NumPy conversion and emits scalars for single-value rows.
embodichain/lab/sim/solvers/srs_solver.py	Corrects module exports and supports detached Torch tensors in the CPU IK pose conversion path.
pyproject.toml	Excludes slow tests by default and registers the new resource and task-discovery markers.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Collected pytest items] --> B{Documentation tests?}
  B -->|Yes| C[Documentation lane]
  B -->|No| D{requires_sim marker or detection}
  D -->|Yes| E{GPU marked?}
  D -->|No| F{GPU marked?}
  E -->|No| G[Serial real-simulation lane]
  E -->|Yes| H{Distributed RL test?}
  H -->|Yes| I[Isolated distributed GPU lane]
  H -->|No| J[Remaining serial GPU lane]
  F -->|No| K[GPU-hidden xdist pure-Python lane]
  F -->|Yes| J

Loading

_{Reviews (1): Last reviewed commit: "perf(tests): reduce runtime and GPU memo..." | Re-trigger Greptile}

[Copilot]

Pull request overview

This PR restructures test execution to reduce overall wall time and GPU memory pressure by marking and routing tests based on simulation/GPU/task-discovery needs, and by tightening a few hot-path teardown/initialization behaviors.

Changes:

• Adds/adjusts pytest markers (requires_sim, no_sim, requires_tasks, gpu, slow, xdist_group) and updates collection logic to better separate pure-Python vs real-simulation vs GPU workloads.
• Improves simulation cleanup behavior by making SimulationManager.flush_cleanup_queue() return immediately when there’s nothing pending, avoiding unnecessary global waits.
• Updates CI workflow to run tests in staged lanes (docs, pure-Python xdist, real-sim, isolated distributed GPU, remaining GPU).

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/toolkits/test_batch_convex_collision.py	Marks CUDA-dependent collision test as gpu and reorders imports.
tests/sim/test_sim_profiler.py	Marks profiler unit tests as no_sim to keep them out of the simulation lane.
tests/sim/test_sim_manager.py	Adds regression coverage for cleanup-queue behavior and marks mock-only tests as no_sim.
tests/sim/planners/test_toppra_batched.py	Reduces default env/batch sizes and marks heavier multiprocessing coverage as slow.
tests/sim/planners/test_curobo_planner.py	Adds a module fixture to restore Torch TF32/matmul precision after cuRobo tests.
tests/sim/objects/test_rigid_constraint.py	Marks mock-only constraint config tests as no_sim.
tests/learning/test_shared_rollout.py	Splits mock-only vs real-sim + tasks tests using no_sim / requires_* markers.
tests/learning/test_rl.py	Marks RL training tests as requires_sim, requires_tasks, and slow.
tests/learning/test_rl_distributed.py	Marks distributed torchrun test as gpu and slow.
tests/learning/test_newton_planar_reach.py	Marks expensive APG training test as slow.
tests/gym/envs/test_replay.py	Marks replay env tests as requires_sim and slow.
tests/gym/envs/test_profiler_integration.py	Marks integration test as requires_sim.
tests/gym/envs/test_embodied_env.py	Marks env tests as requires_sim.
tests/gym/envs/test_base_env.py	Marks env tests as requires_sim.
tests/data_pipeline/test_online_data.py	Avoids starting the sim subprocess for sampling-only tests to reduce teardown waits.
tests/conftest.py	Defers task discovery to requires_tasks tests; adds real-sim classification and xdist_group routing; defers DexSim init.
pyproject.toml	Sets default -m "not slow" and registers new markers, plus warning filtering.
embodichain/lab/sim/solvers/srs_solver.py	Fixes __all__, adds future annotations, and supports Torch tensors for target poses.
embodichain/lab/sim/sim_manager.py	Makes cleanup flush skip global waits when nothing was drained.
embodichain/lab/sim/objects/articulation.py	Normalizes drive args to Python scalars when appropriate to reduce tensor->numpy friction.
.github/workflows/main.yml	Splits CI tests into docs/pure/real-sim/distributed-GPU/remaining-GPU stages.

Suppressed comments (2)

.github/workflows/main.yml:168

• Same mark-expression issue as above: -m "requires_sim and not gpu" overrides the default -m "not slow", so slow real-simulation tests will run in this job.

          echo "Real-simulation tests (one process, GPU tests excluded)"
          export HF_ENDPOINT=https://hf-mirror.com
          pytest tests --ignore=tests/docs -m "requires_sim and not gpu"

.github/workflows/main.yml:181

• This GPU lane also overrides the default -m "not slow" by passing -m gpu, so any gpu and slow tests will be included here (even though slow tests are intended to be excluded by default). If the isolated distributed job is the intended home for slow GPU coverage, the remaining GPU suite should explicitly exclude slow.

      - name: Run remaining GPU tests serially
        run: |
          echo "Dedicated GPU test suite"
          export HF_ENDPOINT=https://hf-mirror.com
          pytest tests --ignore=tests/docs \
            --ignore=tests/learning/test_rl_distributed.py --run-gpu -m gpu

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.