Skip to content

Add Newton physics backend support - #276

Draft
yuecideng wants to merge 142 commits into
mainfrom
feature/newton-physics-backend
Draft

Add Newton physics backend support#276
yuecideng wants to merge 142 commits into
mainfrom
feature/newton-physics-backend

Conversation

@yuecideng

@yuecideng yuecideng commented May 21, 2026

Copy link
Copy Markdown
Contributor

Description

This PR makes Newton a first-class physics backend in EmbodiChain while preserving the existing default DexSim path. It also standardizes scene construction around DexSim Spawn and adds an end-to-end differentiable execution path for analytic policy gradient workloads.

The motivation is to keep backend-specific lifecycle, state access, spawning, and differentiation behind explicit contracts instead of scattering Newton conditionals through simulation objects and environments.

Issue reference: N/A.

What changed

Physics backend and lifecycle

  • Adds typed DefaultPhysicsCfg and NewtonPhysicsCfg configuration, including Newton solver, collision-pipeline, device, substep, CUDA graph, and gradient settings.
  • Introduces the PhysicsBackend abstraction with concrete default and Newton implementations selected from physics_cfg.
  • Moves Newton ownership to the world/Spawn lifecycle and exposes SimulationManager.prepare(), backend capabilities, scene context, and differentiable-runtime delegators.
  • Handles multi-environment spawning, rebuild/rebinding, multi-instance isolation, and deterministic cleanup through explicit runtime contracts.

Spawn and object model

  • Adds embodichain.lab.sim.spawn to translate EmbodiChain configs and USD sources into DexSim Spawn descriptors.
  • Adds backend-neutral rigid-body and articulation views for default, Newton, and Spawn-backed access.
  • Supports Newton rigid objects, rigid-object groups, articulations, and robots, including batched pose/state access, physical-property mutation, joint control, collision filtering, and multi-env cloning.
  • Adds explicit asset-physics preserve / overlay behavior and typed backend-specific rigid, collision, material, articulation-root, and joint-drive properties.
  • Introduces a unified deformable-object API with volume/soft and surface/cloth specializations while retaining compatibility aliases.

Differentiable execution

  • Adds a Warp tape to PyTorch autograd bridge in embodichain.lab.sim.diff.
  • Adds DifferentiableEmbodiedEnv with explicit dynamics and kinematics execution modes, manager-owned Newton trajectories, deferred terminal resets, and safe forward/backward cleanup.
  • Adds the FrankaReachApg-v0 reference task to exercise differentiable stepping and analytic policy gradient behavior end to end.

Compatibility, examples, and documentation

  • Preserves legacy simulation configuration accessors and the default-backend behavior covered by the compatibility tests.
  • Updates environments, sensors, planners, tutorials, benchmarks, and examples to use the explicit prepare/Spawn lifecycle.
  • Updates API reference pages, simulation guides, design documents, and agent context for the backend, Spawn, object, and differentiable-environment contracts.
  • Adds focused coverage for configuration parsing, backend parity, Spawn descriptors, rigid/articulation behavior, lifecycle, multi-env execution, and gradients.

Backend capability summary

Capability Default Newton
Rigid objects Yes Yes
Articulations / robots Yes Yes
Rigid-object groups Yes Yes
Volume deformables / soft bodies Yes No
Surface deformables / cloth Yes No
Disable manual update Yes No
Differentiable autograd bridge No Yes, with gradient mode

Known limitations

  • Newton volume and surface deformables remain unsupported and fail fast through backend capability checks.
  • Newton cannot disable manual update.
  • Differentiable dynamics require NewtonPhysicsCfg(requires_grad=True) with the semi_implicit solver.
  • The Franka APG example intentionally uses the explicit kinematics route because the semi-implicit target-position path does not currently propagate gradients to body poses.
  • Newton callers should use SimulationManager.spawn_result and Spawn batch APIs; the removed independently owned NewtonManager / compatibility PhysicsScene ownership path is not exposed.

Dependencies

No new dependency is added by this branch. The implementation requires the Spawn/Newton APIs provided by the existing dexsim_engine==0.4.3 dependency.

Suggested review order

  1. embodichain/lab/sim/cfg.py, physics/, and sim_manager.py
  2. embodichain/lab/sim/spawn/ and objects/backends/
  3. Rigid object, articulation, robot, and deformable-object integrations
  4. embodichain/lab/sim/diff/, DifferentiableEmbodiedEnv, and the Franka APG task
  5. Tests, examples, and documentation

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (non-breaking change which improves existing functionality)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Screenshots

Not applicable.

Validation

Focused post-main merge validation:

  • Formatting checks passed with Black 26.3.1.
  • Public API documentation gate passed: 1,798 documented exports, 0 missing.
  • Preview asset, gravity settler, tutorial utilities, and articulation checks: 57 passed, 1 skipped.
  • Semantic-skill tutorial checks: 12 passed.
  • Rigid-object CPU and Newton checks: 6 passed.
  • Agent-context registry checks: 8 passed.
  • API-documentation checker tests: 8 passed.
  • Sphinx dummy build completed successfully; optional-import and existing documentation warnings remain.
  • GitHub CI is still running for the latest head.

Checklist

  • I have run the Black formatting checks.
  • I have made corresponding changes to the documentation.
  • Public API changes are reflected in the API docs.
  • I have added tests for the affected backend, Spawn, object, and differentiable paths.
  • Dependencies have been reviewed; no new dependency was added.

yuecideng and others added 30 commits April 12, 2026 14:58
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Jietao Chen <chenjietao@dexforce.com>
Co-authored-by: Yueci Deng <dengyueci@qq.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: yuanhaonan <yuanhaonan@dexforce.top>
…entation (#247)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: yuecideng <dengyueci@qq.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ration (#239)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Chen Jian <mtfl1996@outlook.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: WaferLi <63717327+WaferLi@users.noreply.github.com>
Co-authored-by: liwenfeng <liwenfeng@dexforce.top>
Co-authored-by: chenjian <chenjian@dexforce.com>
Co-authored-by: daojun <lookangela@qq.com>
Co-authored-by: Chen Jian <mtfl1996@outlook.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
yuecideng and others added 19 commits August 5, 2026 13:52
…s-backend

# Conflicts:
#	docs/source/api_reference/embodichain/embodichain.lab.sim.cfg.rst
#	docs/source/features/interaction/preview_asset.md
#	docs/source/guides/configuration.md
#	docs/source/overview/sim/sim_manager.md
#	embodichain/lab/gym/envs/base_env.py
#	embodichain/lab/gym/utils/gym_utils.py
#	embodichain/lab/scripts/preview_asset.py
#	embodichain/lab/sim/cfg.py
#	embodichain/lab/sim/objects/articulation.py
#	embodichain/lab/sim/objects/cloth_object.py
#	embodichain/lab/sim/objects/rigid_object.py
#	embodichain/lab/sim/sim_manager.py
#	embodichain/learning/rl/train.py
#	embodichain_tasks/embodichain_tasks/special/franka_reach_apg.py
#	examples/sim/demo/pick_up_cloth.py
#	examples/sim/gizmo/gizmo_camera.py
#	examples/sim/gizmo/gizmo_object.py
#	examples/sim/gizmo/gizmo_robot.py
#	examples/sim/gizmo/gizmo_scene.py
#	examples/sim/gizmo/gizmo_w1.py
#	examples/sim/planners/neural_planner.py
#	examples/sim/sensors/batch_camera.py
#	examples/sim/sensors/create_contact_sensor.py
#	examples/sim/solvers/differential_solver.py
#	examples/sim/solvers/opw_solver.py
#	examples/sim/solvers/pink_solver.py
#	examples/sim/solvers/pinocchio_solver.py
#	examples/sim/solvers/pytorch_solver.py
#	examples/sim/solvers/srs_solver.py
#	examples/sim/utility/workspace_analyzer/analyze_cartesian_workspace.py
#	examples/sim/utility/workspace_analyzer/analyze_joint_workspace.py
#	examples/sim/utility/workspace_analyzer/analyze_plane_workspace.py
#	scripts/tutorials/gym/modular_env.py
#	scripts/tutorials/gym/random_reach.py
#	scripts/tutorials/sim/create_cloth.py
#	scripts/tutorials/sim/create_rigid_object_group.py
#	scripts/tutorials/sim/create_scene.py
#	scripts/tutorials/sim/create_softbody.py
#	scripts/tutorials/sim/gizmo_robot.py
#	scripts/tutorials/sim/motion_generator.py
#	scripts/tutorials/sim/srs_solver.py
#	tests/gym/utils/test_gym_utils.py
#	tests/sim/objects/test_rigid_object.py
#	tests/sim/workspace/test_sim_utils.py
# Conflicts:
#	scripts/benchmark/__main__.py
#	scripts/benchmark/planners/neural_planner/run_benchmark.py
#	tests/sim/planners/test_toppra_batched.py
Co-authored-by: xiemenghong <xiemenghong@dexforce.top>
Co-authored-by: yuecideng <dengyueci@qq.com>
…s-backend

# Conflicts:
#	agent_context/MAP.yaml
#	embodichain/gen_sim/scene_engine/pipeline/utils/assets_gravity_settler.py
#	embodichain/lab/scripts/preview_asset.py
#	embodichain/lab/sim/objects/__init__.py
#	embodichain/lab/sim/objects/rigid_object.py
#	scripts/tutorials/atomic_action/hand_over.py
#	scripts/tutorials/atomic_action/tutorial_utils.py
#	tests/sim/objects/test_articulation.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physics Things related to physics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants