Skip to content

Optimize Newton replicated camera startup - #6928

Draft
kellyguo11 wants to merge 2 commits into
isaac-sim:developfrom
kellyguo11:kellyg/fix-reorient-replication
Draft

Optimize Newton replicated camera startup#6928
kellyguo11 wants to merge 2 commits into
isaac-sim:developfrom
kellyguo11:kellyg/fix-reorient-replication

Conversation

@kellyguo11

@kellyguo11 kellyguo11 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

Optimize the full replicated-scene startup path used by the 4K Shadow Hand
camera task while preserving heterogeneous-environment support.

The change is intentionally kept in one draft PR so the cloning, camera,
Newton import, mesh/collision, articulation, sensor, and semantic-mapping
optimizations can be tested together.

What changed

  • Construct Shadow Hand reorientation assets and cameras inside
    ReplicateSession, so backend replication sees the complete asset queue.
  • Avoid automatic concrete asset clones for pure headless Newton + Newton Warp
    runs; retain concrete USD for PhysX, RTX, GUI/offscreen output, visualizers,
    and explicit USD clone consumers.
  • Use compact internal USD references when concrete destinations are required.
  • Resolve camera prototypes through the clone plan, cache shared camera
    intrinsics, and skip Newton-irrelevant physics validation in FrameView.
  • Import clone-external Newton USD roots directly instead of traversing the
    replicated environment tree from the pseudo-root.
  • Scope heightfield and collision-approximation scans to imported roots/shapes,
    and register only the selected solver's builder attributes.
  • Cache repeated articulation target-mode resolution and reuse joint-wrench and
    site-frame view work.
  • Resolve segmentation labels through the clone plan, reading semantic labels
    once per prototype while preserving per-environment instance identities and
    heterogeneous source selection.

Dependencies: none.

Performance

Measured against upstream/develop at e3c7dd87b6 on an RTX PRO 6000
Blackwell with warm asset and kernel caches.

Shadow Hand camera, 4096 environments

uv run --frozen isaaclab benchmark startup \
  --task Isaac-Reorient-Cube-Shadow-Camera-Benchmark-Direct \
  --num_envs 4096 \
  physics=newton_mjwarp renderer=newton_renderer \
  --benchmark_formatter summary
Metric Before After Change
Scene creation 19.34 s 3.47 s -15.87 s (-82.1%)
Builder finalization 2.52 s 2.46 s -0.06 s (-2.4%)
Semantic mapping 5.14 s 0.62 s -4.53 s (-88.0%)
Solver initialization 1.70 s 1.69 s effectively unchanged
CUDA graph capture 1.51 s 1.52 s effectively unchanged
Simulation start 14.82 s 8.46 s -6.36 s (-42.9%)
Profiled environment creation 35.83 s 13.57 s -22.26 s (-62.1%)
First step 0.22 s 0.23 s effectively unchanged
Profiled launch through first step 38.52 s 16.15 s -22.37 s (-58.1%)

A second post-fix run measured 15.91 s through the first step, showing the
expected small run-to-run variance. This does not yet meet the 10 s
end-to-end goal. The largest remaining fixed costs include builder finalization
(~2.46 s), solver initialization plus graph capture (~3.22 s), and the remaining
model/callback/reset work around them.

Kuka-Allegro lift camera, 4096 heterogeneous environments

uv run --frozen isaaclab benchmark startup \
  --task Isaac-Lift-KukaAllegro-Camera \
  --num_envs 4096 \
  physics=newton_mjwarp renderer=newton_renderer \
  presets=single_camera,rgb64 \
  --benchmark_formatter summary
Metric Before After Change
Scene creation 13.13 s 12.42 s -0.71 s (-5.4%)
Simulation start 8.73 s 7.35 s -1.38 s (-15.9%)
Profiled environment creation 25.10 s 22.67 s -2.43 s (-9.7%)
First step 0.53 s 0.52 s -0.01 s (-1.9%)
Profiled launch through first step 29.39 s 26.93 s -2.46 s (-8.4%)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Performance improvement

Validation

  • 33 focused tests passed, covering replication policy, compact USD references,
    articulation caching, scoped global/collision imports, clone-plan semantic
    mapping, and Kuka heterogeneous rendering.
  • Verified the new regression tests fail against unmodified upstream/develop.
  • Kuka heterogeneous Newton-renderer golden tests passed for both RGB and
    semantic segmentation.
  • Isaac-Lift-KukaAllegro-Camera: RSL-RL, 16 environments, Newton MJWarp +
    Newton renderer, two learning iterations completed.
  • Isaac-Reorient-Cube-Shadow-Camera-Direct: RSL-RL, 16 environments, Newton
    MJWarp + Newton renderer, two learning iterations completed.
  • Both 4096-environment benchmarks above completed through the first step.
  • uv run --frozen python tools/changelog/cli.py check develop: passed.
  • uv run --frozen isaaclab -f: passed.
  • Local Isaac Sim PhysX + RTX benchmarking is unavailable in this kitless
    worktree because the full Isaac Sim/Kit runtime is not installed. This remains
    an explicit CI/reviewer validation request.

Screenshots

Not applicable.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks
  • Documentation changes are not required for this internal startup path
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package
  • I have added my name to CONTRIBUTORS.md or my name already exists there

Construct reorientation assets inside ReplicateSession so backend
replication observes the complete asset queue without authoring every
environment first. Preserve the existing PhysX collision filtering
behavior after replication.
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Aug 6, 2026
Avoid materializing unused USD clones for pure headless Newton runs and
resolve cameras from clone prototypes. Scope global USD import and collision
processing to relevant source roots, cache repeated articulation and sensor
work, and map segmentation labels through the clone plan.

Keep concrete USD replication for PhysX, RTX, visualizers, and heterogeneous
prototype mappings that require it.
@kellyguo11 kellyguo11 changed the title Fix reorient task replication setup Optimize Newton replicated camera startup Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant