Integrate dexsim gizmo controllers

agent_context/MAP.yaml

@@ -21,6 +21,10 @@ topics:
       - simulation
       - SimulationManager
       - SimulationManagerCfg
+      - EntityGizmoManipulator
+      - enable_entity_gizmo
+      - disable_entity_gizmo
+      - robot_ik_gizmo
       - DexSim
       - world
       - arena
@@ -349,11 +353,18 @@ topics:
       - deformable
       - soft body
       - cloth
+      - gizmo
+      - IKGizmoController
+      - create_robot_ik_gizmo_controller
+      - GizmoCfg
+      - ScenePicker
+      - PickCommand
     paths:
       - topics/sim-visualization/sim-visualization.md
     source_of_truth:
       - embodichain/lab/sim/sim_manager.py
       - embodichain/lab/visualization/
+      - embodichain/lab/sim/objects/gizmo.py
       - embodichain/lab/gym/utils/gym_utils.py
       - embodichain/lab/gym/envs/base_env.py
       - embodichain/lab/scripts/preview_asset.py

agent_context/topics/ik-solvers/ik-solvers.md

@@ -34,6 +34,16 @@ path for performance.
 
 ## Solver Hierarchy
 
+Gizmo solver selection belongs to `embodichain/lab/sim/objects/gizmo.py`, not
+to a new solver subclass. Native DexSim `IKGizmoController` remains the
+controller in both modes: `GizmoCfg.ik_solver="dexsim"` selects Newton IK;
+`"embodichain"` adapts `robot.get_solver(control_part)` for the native window
+and Viser. It preserves configured solver convergence/limits, maps joint names
+between solver and robot order, and holds current qpos on failure/non-finite
+results. Gizmo root/end links must match the configured solver; TCP overrides
+are converted without mutating that solver. See the robot gizmo example's
+`--ik-solver pink` option for an executable Pink configuration.
+
 ```
 SolverCfg  (@configclass, abstract)
   ├── SRSSolverCfg

agent_context/topics/sim-visualization/sim-visualization.md

@@ -252,6 +252,32 @@ Soft bodies and cloth require GPU physics. Their live vertices are sampled at
 
 ## Browser Controls and Overlays
 
+Gizmo implementation lives in `embodichain/lab/sim/objects/gizmo.py`. Native
+windows delegate object picking/manipulation to DexSim's entity gizmo and robot
+targets to its `IKGizmoController`. Entity interaction defaults on at the first
+native window open; `SimulationManagerCfg.enable_entity_gizmo=False` or
+`sim.disable_entity_gizmo()` opts out and reopening preserves the choice.
+Pure headless and Viser runs do not automatically create native controls.
+`SimulationManagerCfg.robot_ik_gizmo` automatically registers robot parts with
+solver chain/TCP metadata: native IK activates on I by default, or on the first
+update with an open window when `GizmoCfg(ik_start_enabled=True)`. The robot
+tutorial uses this startup option. Viser builds IK on the first drag. The manager updates both and preserves native controls across window
+reopen. Explicit configuration and disable calls take precedence; caller-owned
+native factory controllers are not duplicated. Automatic Viser registration
+requires `allow_commands`, independently of native interaction preferences.
+Viser transports poses to the
+simulation thread. Both robot paths use Newton IK by default or, with
+`GizmoCfg(ik_solver="embodichain")`, reuse the configured control-part solver
+(including Pink). Chain roots follow the live robot link, including upstream
+joint motion; TCP overrides adapt targets without changing the shared solver.
+
+Viser click picking runs on the visualization worker via the existing GUI
+event queue. A manifest invalidates cached pick poses until its matching frame
+arrives; stale clicks are dropped. The manager validates `PickCommand` run and
+revision before attaching a gizmo and tracks picker ownership independently
+from explicitly created gizmos. Clearing selection releases only the picker
+gizmo. Native entity selection remains entirely DexSim-owned.
+
 The browser GUI has:
 
 - **Environments** — visibility per exported environment;

agent_context/topics/simulation-system/simulation-system.md

@@ -71,6 +71,15 @@ then advances the world for the requested number of physics steps. Each
 environment control step normally calls it with
 `sim_steps_per_control`.
 
+`scripts/tutorials/sim/gizmo_robot.py` supports only manual physics. It initializes
+GPU physics after robot creation when needed, sets both current and target
+joint positions, and advances once before opening the window. It explicitly
+sets `GizmoCfg(ik_start_enabled=True)` so the native controller activates on the
+first update after opening the window. Its loop only
+calls `sim.update(step=1)`; the manager owns native IK updates and Viser
+commands/capture. The loop is paced by `physics_dt` and has no automatic
+physics polling path.
+
 `ArticulationCfg.enable_gravity` defaults to `True`. During articulation
 construction, `Articulation` applies this explicit runtime flag to every native
 entity before the first physics update, including when
@@ -118,6 +127,45 @@ do not belong to the simulation object; use
 
 ## Configuration Flow
 
+### Gizmo ownership
+
+Native entity manipulation belongs to DexSim 0.5.0. The first successful native
+window open enables its world-owned `EntityGizmoManipulator` by default, after
+the scene and default plane are ready. The manager registers the default plane
+as a static external target. `SimulationManagerCfg.enable_entity_gizmo=False`
+opts out; Gym deployments accept the same top-level JSON/YAML field through
+`gym.utils.gym_utils.config_to_cfg()`.
+
+`sim.enable_entity_gizmo(config)` explicitly enables/configures the controller;
+`sim.disable_entity_gizmo()` also cancels pending automatic enablement before
+the first window. These explicit calls take precedence over the startup default.
+Query through `sim.get_world().get_entity_gizmo()`. DexSim owns window
+detach/reopen and controller state; reopening never reapplies the default or
+overwrites an explicit native disable. Pure headless and Viser runs do not
+automatically create a native entity controller.
+
+`SimulationManagerCfg.robot_ik_gizmo` defaults to `GizmoCfg()`. During normal
+updates the manager registers robot control parts with complete solver chain/TCP
+metadata in single-environment interactive runs. Pure headless, read-only Viser, and
+multi-environment runs do not register automatic controls. The first native I
+press creates DexSim's `IKGizmoController` by default;
+`GizmoCfg(ik_start_enabled=True)` opts into activation on the first update with
+an open window. The startup attempt is consumed once, including on failure;
+later key presses can retry. Viser constructs IK on its first drag.
+Registration never writes drive targets. `Gizmo` owns managed native input and
+target-node cleanup, detaches input on window close, and reattaches the same
+controller on reopen. Robot removal releases all its managed controls.
+
+Set `robot_ik_gizmo=None` to opt out or supply `GizmoCfg` overrides; Gym
+JSON/YAML accepts the same mapping/null. `enable_gizmo()` can override one part,
+and `disable_gizmo()` prevents automatic recreation (all parts when omitted).
+The explicit `create_robot_ik_gizmo_controller()` factory still returns
+caller-owned controllers; a weak registry prevents automatic duplicates.
+Both robot paths
+default to native Newton IK; `GizmoCfg(ik_solver="embodichain")` adapts the
+control part's existing solver, such as PinkSolver. Both support one environment
+and write only selected non-mimic joint drive targets through `Robot`.
+
 `SimulationManagerCfg` owns window size, headless mode, rendering, GPU/CPU
 selection, arena count and spacing, physics timestep, physics and GPU-memory
 settings, recording, profiling, and browser visualization.

docs/source/api_reference/embodichain/embodichain.lab.sim.objects.rst

@@ -42,6 +42,7 @@ bodies.
     RobotWorkspaceCfg
     Gizmo
     GizmoCfg
+    create_robot_ik_gizmo_controller
     RigidConstraint
 
 .. currentmodule:: embodichain.lab.sim.objects
@@ -191,6 +192,27 @@ Gizmo
     :show-inheritance:
     :exclude-members: __init__, copy, replace, to_dict, validate
 
+.. autofunction:: create_robot_ik_gizmo_controller
+
+SimulationManager automatically discovers robot control parts with configured
+IK chain metadata. Native controllers activate on the first I press by default, while
+Viser constructs its solver on the first drag. ``sim.update()`` owns updates
+and cleanup; ordinary applications do not need the explicit factory.
+Use ``SimulationManagerCfg(robot_ik_gizmo=None)`` to disable automatic setup.
+Set ``GizmoCfg(ik_start_enabled=True)`` to activate native IK on the first update
+with an open window; subsequent visibility toggles and reopening are preserved.
+
+The native controller defaults to DexSim Newton IK. With a ``PinkSolverCfg``
+(or another EmbodiChain solver) configured for the robot's control part, pass
+``GizmoCfg(ik_solver="embodichain")`` to select that solver for either a native
+controller or a Viser gizmo. Its iteration limits and convergence settings
+remain owned by the configured solver; ``ik_iterations`` applies to Newton IK.
+Only the selected control part's drive targets are written, and failed
+EmbodiChain IK solutions preserve the current joint positions.
+
+The runnable example ``examples/sim/gizmo/gizmo_robot.py`` exposes
+``--ik-solver dexsim|pytorch|pink`` for both the native window and ``--viser``.
+
 Rigid Constraint
 ----------------
 

docs/source/api_reference/embodichain/embodichain.lab.sim.sim_manager.rst

@@ -13,6 +13,22 @@ configuration. Downstream components (environments, planners, IK solvers, the
 visualization runtime) look up the active manager through its class-level
 instance registry instead of passing it around explicitly.
 
+Native entity interaction defaults on when the first native window opens.
+Set ``SimulationManagerCfg(enable_entity_gizmo=False)`` to opt out, or call
+``sim.disable_entity_gizmo()`` at runtime. Explicit enable/disable calls and
+custom DexSim controller settings survive window close/reopen. Pure headless
+and Viser runs do not automatically create native gizmos.
+
+``SimulationManagerCfg.robot_ik_gizmo`` defaults to ``GizmoCfg()`` and registers
+robot control parts with configured IK-chain/TCP metadata during normal updates.
+Native IK activates on the first **I** press by default; Viser constructs its solver on
+the first drag and requires ``visualization.allow_commands``. Registration does
+not write drive targets. Set this field to ``None`` to opt out or select
+``GizmoCfg(ik_solver="embodichain")`` to reuse configured solvers. Explicit
+``enable_gizmo()`` settings override automatic defaults, and ``disable_gizmo()``
+prevents automatic recreation. ``GizmoCfg(ik_start_enabled=True)`` activates
+native IK on the first update with an open window, as used by the robot tutorial.
+
 .. rubric:: Classes
 
 .. autosummary::

docs/source/api_reference/embodichain/embodichain.lab.sim.utility.rst

@@ -17,7 +17,6 @@ action/solver adaptation.
    action_utils
    atom_action_utils
    cfg_utils
-   gizmo_utils
    import_utils
    io_utils
    keyboard_utils
@@ -46,12 +45,6 @@ Configuration Utilities
 .. automodule:: embodichain.lab.sim.utility.cfg_utils
    :members:
 
-Gizmo Utilities
-~~~~~~~~~~~~~~~
-
-.. automodule:: embodichain.lab.sim.utility.gizmo_utils
-   :members:
-
 Import Utilities
 ~~~~~~~~~~~~~~~~
 

docs/source/api_reference/embodichain/embodichain.lab.visualization.rst

@@ -103,6 +103,14 @@ Scene Export
    :members:
    :undoc-members:
 
+``PickCommand`` identifies a node in a specific simulation run and scene
+revision. An empty selection releases only the gizmo created by click picking;
+explicitly configured gizmos retain their ownership.
+
+.. autoclass:: PickCommand
+   :members:
+   :undoc-members:
+
 .. autoclass:: JointControlSpec
    :members:
    :undoc-members:

docs/source/api_reference/public_api.rst

@@ -865,10 +865,15 @@ embodichain.lab.sim.objects.gizmo
 
 .. currentmodule:: embodichain.lab.sim.objects.gizmo
 
+Native robot targets use DexSim's controller with Newton IK by default.
+Set ``GizmoCfg.ik_solver="embodichain"`` to reuse the robot control part's
+configured solver, including PinkSolver; Viser uses the same solver adapter.
+
 .. autosummary::
 
    Gizmo
    GizmoCfg
+   create_robot_ik_gizmo_controller
 
 embodichain.lab.sim.objects.rigid_object
 ----------------------------------------
@@ -1499,6 +1504,22 @@ embodichain.lab.visualization.cli
    add_viser_args_to_parser
    visualization_cfg_from_args
 
+embodichain.lab.visualization.picker
+------------------------------------
+
+.. currentmodule:: embodichain.lab.visualization.picker
+
+Browser picking caches triangle geometry and returns the closest node hit by
+a world-space ray. The Viser worker pairs this geometry with poses from the
+same scene revision before producing a pick command.
+
+.. autosummary::
+
+   ScenePicker
+
+.. autoclass:: ScenePicker
+   :members:
+
 embodichain.lab.visualization.protocol
 --------------------------------------
 
@@ -1520,6 +1541,7 @@ embodichain.lab.visualization.protocol
    JointControlSpec
    JointControlState
    MeshGeometry
+   PickCommand
    PointCloudOverlay
    SceneFrame
    SceneManifest