feat(task-engine): add orchestration and end-to-end integration

[skywhite1024]

Stack

• Layer: 6/6
• Base: ljd/gen-sim-refactor-06-execution-agent
• Depends on: feat(action-engine): add execution runtime and ActionAgent #537
• Followed by: none

Description

This PR adds Task Engine orchestration across semantic interpretation, Scene Engine materialization/editing, scene adaptation and feasibility, Action Engine bundle generation, execution, retries, run directories, artifacts, state-machine events, and the unified task-engine CLI.

It is the end-to-end integration layer for the rebuilt six-PR stack.

Type of change

• Bug fix
• Enhancement
• New feature
• Breaking change
• Documentation update

Validation

• black --check --diff --color ./ (969 files unchanged)
• python docs/scripts/check_api_docs.py (2337/2337 exports documented)
• pytest tests/docs/test_check_api_docs.py -q --confcutdir=tests/docs (8 passed)
• pytest tests/gen_sim/action_engine tests/gen_sim/task_engine tests/gen_sim/scene_engine/test_pipeline_api.py tests/gen_sim/test_video_archive.py -q (741 passed)

Sphinx dummy build was not run because Sphinx is not installed in the available project environments.

Checklist

• Code is formatted with Black 26.3.1
• Public API documentation is aligned
• Focused cross-layer integration tests pass
• Package data includes Action Engine and Task Engine YAML defaults

[greptile-apps]

Greptile Summary

The PR adds a unified Task Engine orchestration pipeline spanning scene preparation, candidate planning, bundle publication, subprocess execution, and CLI integration.

• Adds versioned orchestration, scene adaptation, feasibility, workflow-state, and artifact contracts.
• Adds isolated run directories, configuration defaults, prepared-bundle execution, and end-to-end tests.
• Extends grasp generation and pickup behavior with diagnostics and fallback handling.

Confidence Score: 4/5

The prepared-bundle CLI should not be merged until its success policy is recalculated for the environment count actually launched.

A caller can override the replica count used by execution while acceptance retains a threshold derived from the configured count, causing deterministic false rejection or false acceptance under the all policy.

Files Needing Attention: embodichain/gen_sim/task_engine/cli.py and embodichain/gen_sim/task_engine/config.py

Important Files Changed

Filename	Overview
embodichain/gen_sim/task_engine/cli.py	Adds the unified command interface and prepared-bundle acceptance logic; environment-count overrides are evaluated against a stale configured threshold.
embodichain/gen_sim/task_engine/workflow.py	Implements the end-to-end workflow, parallel interpretation and scene work, bounded retries, publication, and optional execution.
embodichain/gen_sim/task_engine/orchestration/coordinator.py	Coordinates candidate grounding, feasibility checks, planning fallback, preflight, and transactional bundle publication.
embodichain/gen_sim/task_engine/orchestration/scene_source.py	Adds source resolution, content fingerprinting, dependency hashing, and mutation detection for externally owned scene projects.
embodichain/gen_sim/task_engine/_bundle_runner.py	Adds the private subprocess boundary, bundle validation, integrity verification, and Action Engine preflight.
embodichain/gen_sim/task_engine/config.py	Defines strict workflow, planning, and vectorized execution policies loaded from packaged or caller-supplied YAML.
embodichain/lab/sim/atomic_actions/primitives/pick_up.py	Extends pickup planning with grasp diagnostics and support-plane fallback behavior.

Sequence Diagram

sequenceDiagram
    participant CLI
    participant Workflow as TaskEngineWorkflow
    participant Scene as SceneBackend
    participant Coordinator
    participant Executor as SubprocessActionExecutor
    participant Runner as Bundle Runner
    CLI->>Workflow: prepare / run-all request
    Workflow->>Scene: materialize and inspect scene
    Scene-->>Workflow: scene revision and inspection
    Workflow->>Coordinator: prepare candidates and bundle
    Coordinator-->>Workflow: published executable bundle
    alt run-all
        Workflow->>Executor: execute bundle with num_envs
        Executor->>Runner: launch Action Engine subprocess
        Runner-->>Executor: execution report
        Executor-->>Workflow: environment outcomes
    end
    Workflow-->>CLI: manifest and final status

Loading

Prompt To Fix All With AI

### Issue 1
embodichain/gen_sim/task_engine/cli.py:181
**Success threshold uses stale count**

When `--num-envs` differs from the configured environment count, execution uses the override but `required_successes` remains based on the configured count, causing `success_policy=all` to reject every smaller run or accept a larger run without all launched environments succeeding.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "feat(task-engine): add orchestration and..." | Re-trigger Greptile}

[greptile-apps]

Success threshold uses stale count

When --num-envs differs from the configured environment count, execution uses the override but required_successes remains based on the configured count, causing success_policy=all to reject every smaller run or accept a larger run without all launched environments succeeding.

Prompt To Fix With AI

This is a comment left during a code review.
Path: embodichain/gen_sim/task_engine/cli.py
Line: 181

Comment:
**Success threshold uses stale count**

When `--num-envs` differs from the configured environment count, execution uses the override but `required_successes` remains based on the configured count, causing `success_policy=all` to reject every smaller run or accept a larger run without all launched environments succeeding.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.