fix(agent): append system_prompt to the Claude Code preset instead of replacing

docs/AB_EXPERIMENTS.md

@@ -133,8 +133,8 @@ From `ExperimentVariant` (`coder_eval/models/experiment.py`):
 
 The `agent` dict is the lever for most A/B tests. Anything on `AgentConfig` is
 fair game: `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`,
-`plugins`, `system_prompt` / `system_prompt_file`, `setting_sources`,
-`claude_settings`, `sdk_options`.
+`plugins`, `system_prompt` / `system_prompt_file`, `system_prompt_mode`
+(append-vs-replace arms), `setting_sources`, `claude_settings`, `sdk_options`.
 
 > **Path-resolution gotcha.** Relative file paths in variant config resolve
 > against _different_ base directories depending on the field:

docs/REPORT_SCHEMA.md

@@ -137,6 +137,14 @@ The authoritative per-replicate record.
 `ClaudeAgentOptions` dump), `sandbox_path`, `task_config`
 (`{resolved, source_yaml, source_file, lineage}` — `lineage` maps each field to
 `{value, source, source_detail}` so you can trace which config layer set it).
+`environment_info.system_prompt_semantics` (`"append"` / `"replace"`) records the
+system-prompt regime the agent ran with; runs predating the marker used
+replace-on-set / empty-on-unset semantics and are not score-comparable, so
+consumers should segment on it (absent key ⇒ pre-append regime).
+`sdk_options.system_prompt` is a `SystemPromptPreset` dict
+(`{type: "preset", preset: "claude_code", exclude_dynamic_sections: true, append?: str}`)
+on append-mode Claude Code runs and a plain string only in replace mode — it is
+no longer `str | null`, so consumers must not string-handle it unconditionally.
 
 **Telemetry/totals:** `total_token_usage` ([TokenUsage](#tokenusage)),
 `command_stats` (`CommandStatistics`), `total_assistant_turns`, `expected_commands` /

docs/agents/ANTIGRAVITY.md

@@ -109,6 +109,14 @@ Antigravity exposes a `thinking_level` field (`minimal` / `low` / `medium` /
 Antigravity-specific — Claude Code and Codex don't take this field. Thinking tokens
 are billed as **output** tokens (see [Telemetry](#telemetry)).
 
+### `system_prompt`
+
+`agent.system_prompt` is passed to the SDK as `system_instructions`, whose string
+shorthand maps to `TemplatedSystemInstructions` — a named section **appended** to
+the harness's default system instructions, never a replacement. This matches the
+append-only semantics of the shared config field across agents (Claude Code appends
+via the `claude_code` preset; Codex via `developer_instructions`).
+
 ### Skills (SKILL.md)
 
 Antigravity supports [Agent Skills](https://agentskills.io/specification)

docs/agents/CLAUDE_CODE.md

@@ -99,7 +99,8 @@ agent:
 | `allowed_tools` | `list[str] \| null` | Tool allowlist. Unset ⇒ all tools allowed. |
 | `disallowed_tools` | `list[str] \| null` | Tool denylist. (`ToolSearch` is always appended for Bedrock parity.) |
 | `plugins` | `list[{type: local, path}]` | Local plugin/skill directories; `$VAR` in `path` is expanded and resolved to an absolute path. |
-| `system_prompt` | `str \| null` | **Replaces** the default system prompt (there is no *append* seam). Mutually exclusive with `system_prompt_file`. |
+| `system_prompt` | `str \| null` | **Appended** to the default Claude Code system prompt (via the SDK's `claude_code` preset) — the default's behavioral guidance is kept unless `system_prompt_mode: replace` opts out. Mutually exclusive with `system_prompt_file`. |
+| `system_prompt_mode` | `"append"` (default) / `"replace"` | `replace` sends `system_prompt` as the **entire** system prompt (no preset) and requires `system_prompt` / `system_prompt_file` to be set (validated at load). Used by judge sub-agents and the user simulator, which must not carry the coding-agent persona; rarely needed in tasks. |
 | `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. |
 | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
 | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
@@ -111,6 +112,17 @@ agent:
 > `setting_sources`, `include_partial_messages`, …) are rejected there — set those
 > through their typed fields or `-D run_limits.*`. MCP servers are not a YAML field.
 
+> **System-prompt reproducibility.** In `append` mode the preset's *dynamic
+> sections* (working directory, git status, auto-memory) are excluded so the system
+> prompt stays identical across runs — the per-run sandbox tempdir path would
+> otherwise be baked into it, breaking prompt caching and run comparability. The
+> SDK re-injects the stripped content into the first user message, so the agent
+> loses nothing. Note the default-prompt baseline tracks the installed Claude Code
+> CLI version; `environment_info.claude_code_cli` in `run.json` records which
+> version a run used, and `environment_info.system_prompt_semantics`
+> (`append` / `replace`) records the prompt regime — runs predating that marker
+> used replace-on-set / empty-on-unset semantics and are not score-comparable.
+
 ### Setting fields from the CLI
 
 Any of these merge-resolve through `-D` / `--set` (see

docs/agents/CODEX.md

@@ -213,6 +213,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 | **SDK Type** | Subprocess (CLI via JSON generator) | Sync client (app-server subprocess) |
 | **Command Tracking** | Full telemetry (tool name, params, duration) | Streamed telemetry: shell → `Bash`, apply_patch → `Write` |
 | **Model Selection** | Direct via `--model` or config | `agent.model` pinned into `thread_start` |
+| **System prompt** | `system_prompt` appended to the default prompt (SDK `claude_code` preset) | `system_prompt` passed as `developer_instructions` on top of the Codex base prompt |
 | **Session Resume** | `--resume {session_id}` | Via thread ID |
 | **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
 | **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |

src/coder_eval/agents/antigravity_agent.py

@@ -348,7 +348,10 @@ async def start(
                 # Autonomous execution: approve every tool call (incl. run_command),
                 # which the default LocalAgentConfig policy would otherwise deny.
                 policies=[policy.allow_all()],
-                system_instructions=self.config.system_prompt or None,
+                # Forward verbatim (None stays None): an explicit "" is a
+                # configured-but-empty prompt and must be forwarded, matching
+                # the `is not None` semantics in the Claude Code / Codex agents.
+                system_instructions=self.config.system_prompt,
                 # Skill discovery: hand the harness the search-path roots that parent
                 # the UiPath skill dirs. Unlike Codex (which symlinks into
                 # .agents/skills/), Antigravity takes skill search paths natively.
@@ -561,6 +564,9 @@ def get_environment_info(self) -> dict[str, Any]:
         return {
             "antigravity_model": self._effective_model(),
             "antigravity_thinking_level": self.config.thinking_level,
+            # Antigravity has always appended (TemplatedSystemInstructions);
+            # emitted for cross-agent uniformity of the marker.
+            "system_prompt_semantics": "append",
         }
 
     def _conversation_or_none(self) -> Any:

src/coder_eval/agents/claude_code_agent.py

@@ -10,7 +10,7 @@
 from contextlib import suppress
 from datetime import datetime, timedelta
 from pathlib import Path
-from typing import Any, ClassVar
+from typing import Any, ClassVar, Literal
 
 from claude_agent_sdk import (
     ClaudeAgentOptions,
@@ -28,6 +28,10 @@
 # its kill target and timeouts will no longer be enforced at the agent layer.
 from claude_agent_sdk._internal.transport.subprocess_cli import SubprocessCLITransport
 
+# SystemPromptPreset is not re-exported from the SDK root, so claude_agent_sdk.types
+# is the only import route (same treatment as evaluation/verdict_tool.py).
+from claude_agent_sdk.types import SystemPromptPreset
+
 from coder_eval.agent import Agent, AgentState
 from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event
 from coder_eval.agents.registry import AgentRegistry
@@ -1173,6 +1177,26 @@ def _build_claude_query(
         if "ToolSearch" not in disallowed_tools:
             disallowed_tools.append("ToolSearch")
 
+        # The SDK maps system_prompt=None to `--system-prompt ""` (an explicit
+        # EMPTY custom prompt) and a plain string to a full replacement — either
+        # way Claude Code's default behavioral guidance (parallel tool-call
+        # batching, conciseness) is lost. So ALWAYS send the claude_code preset:
+        # without `append` the CLI runs its default prompt; with it the configured
+        # prompt is appended. exclude_dynamic_sections keeps the prompt static
+        # across runs (the per-run tempdir path would otherwise be baked into the
+        # system prompt, breaking prompt caching and run comparability); the SDK
+        # re-injects the stripped sections into the first user message.
+        # system_prompt_mode="replace" (judge sub-agents) opts out of the preset:
+        # the configured prompt IS the entire system prompt.
+        system_prompt: str | SystemPromptPreset
+        if self._effective_prompt_mode() == "replace":
+            assert self.config.system_prompt is not None  # guaranteed by _effective_prompt_mode
+            system_prompt = self.config.system_prompt
+        else:
+            system_prompt = SystemPromptPreset(type="preset", preset="claude_code", exclude_dynamic_sections=True)
+            if self.config.system_prompt is not None:
+                system_prompt["append"] = self.config.system_prompt
+
         # as_posix(), not str(): bash on Windows strips backslashes from unquoted
         # paths, so a redirect like `> D:\foo\bar` ends up writing to "Dfoobar".
         options = ClaudeAgentOptions(
@@ -1192,7 +1216,7 @@ def _build_claude_query(
             # summing per-message values undercounts by 10x+. Without this flag
             # StreamEvents are suppressed by the SDK.
             include_partial_messages=True,
-            system_prompt=self.config.system_prompt,
+            system_prompt=system_prompt,
             setting_sources=self.config.setting_sources if self.config.setting_sources is not None else ["project"],
             resume=self._session_id,
             settings=json.dumps(self.config.claude_settings)
@@ -1216,6 +1240,30 @@ def _build_claude_query(
 
         return options, transport, effective_model
 
+    def _effective_prompt_mode(self) -> Literal["append", "replace"]:
+        """The system-prompt regime that actually goes on the wire.
+
+        Single source of truth for both the options builder and the
+        ``system_prompt_semantics`` run-record marker, so the persisted regime
+        can never disagree with what was sent. ``replace`` requires a configured
+        prompt (the config validator rejects the pair at load, but a mutated or
+        hand-built config falls back to the preset here — fail open to append).
+        """
+        if self.config.system_prompt_mode == "replace" and self.config.system_prompt is not None:
+            return "replace"
+        return "append"
+
+    def get_environment_info(self) -> dict[str, Any]:
+        """Record which system-prompt regime built this run's prompts.
+
+        ``append`` = the claude_code preset (dynamic sections excluded) with the
+        configured system_prompt, if any, appended; ``replace`` = the configured
+        prompt is the ENTIRE system prompt (judge sub-agents). Runs from before
+        this marker existed used replace-on-set / empty-on-unset semantics —
+        trend dashboards must not pool scores across that boundary.
+        """
+        return {"system_prompt_semantics": self._effective_prompt_mode()}
+
     async def stop(self) -> None:
         """Stop the agent and clean up resources."""
         self.client = None

src/coder_eval/agents/codex_agent.py

@@ -937,17 +937,25 @@ def _close_client(self) -> None:
     def get_environment_info(self) -> dict[str, Any]:
         """Record the resolved Codex routing so runs are auditable/comparable.
 
-        Only emits when a custom endpoint is configured (CODEX_BASE_URL). On a
-        custom endpoint the model is an operator-chosen alias (a deployment name
-        on Azure), so two operators' ``gpt-5-codex`` deployments are otherwise
-        indistinguishable in run artifacts. The host (not the full URL) is
-        recorded to avoid leaking any embedded credentials; the API key is never
-        recorded.
+        Always emits ``system_prompt_semantics``. The routing keys
+        (``codex_base_url_host`` / ``codex_wire_api`` / ``codex_api_version`` /
+        ``codex_model_is_deployment``) are added only when a custom endpoint is
+        configured (CODEX_BASE_URL): on a custom endpoint the model is an
+        operator-chosen alias (a deployment name on Azure), so two operators'
+        ``gpt-5-codex`` deployments are otherwise indistinguishable in run
+        artifacts. The host (not the full URL) is recorded to avoid leaking any
+        embedded credentials; the API key is never recorded.
         """
+        # system_prompt_semantics: Codex appends system_prompt as
+        # developer_instructions on top of its base prompt. Runs from before this
+        # marker existed silently DROPPED the field — dashboards must not pool
+        # system_prompt-setting tasks across that boundary.
+        info: dict[str, Any] = {"system_prompt_semantics": "append"}
         base_url = self._resolve_base_url()
         if not base_url:
-            return {}
+            return info
         return {
+            **info,
             "codex_base_url_host": urlparse(base_url).hostname or "",
             "codex_wire_api": _CODEX_WIRE_API,
             "codex_api_version": self._resolve_api_version() or "",
@@ -1276,6 +1284,14 @@ def _build_thread_options(self) -> dict[str, Any]:
             options["model"] = effective_model
             self._log.debug(f"Codex model pinned to {effective_model}")
 
+        # system_prompt maps to developer_instructions: injected ON TOP of Codex's
+        # base prompt, matching the append-only contract of the shared config field
+        # (Claude Code appends via the claude_code preset; Antigravity via
+        # TemplatedSystemInstructions). base_instructions (full replacement of the
+        # base prompt) is deliberately not exposed.
+        if self.config.system_prompt is not None:
+            options["developer_instructions"] = self.config.system_prompt
+
         permission_mode = self.config.permission_mode.value
         approval_mode_str = _CODEX_APPROVAL_MODE
 

src/coder_eval/criteria/agent_judge.py

@@ -63,6 +63,11 @@
 logger = logging.getLogger(__name__)
 
 
+# This is the judge's ENTIRE identity: _build_agent_config forces
+# system_prompt_mode="replace" so the Claude Code coding-agent preset never
+# reaches the scoring instrument — the judge must not carry an engineering
+# persona (terse, proactively edits files) ahead of its grading role, and its
+# verdicts must not shift when the preset does.
 _SYSTEM_PROMPT = """\
 You are a strict code reviewer evaluating a project generated by a coding agent.
 
@@ -263,6 +268,10 @@ def _build_agent_config(
         user_overrides["sdk_options"] = {**defaults.sdk_options, **user_overrides["sdk_options"]}
     config = defaults.model_copy(update=user_overrides, deep=True)
     config.system_prompt = system_prompt
+    # Force replace regardless of user YAML: the judge prompt is its entire
+    # identity — the coding-agent preset must never prefix the scoring
+    # instrument (see the note on _SYSTEM_PROMPT).
+    config.system_prompt_mode = "replace"
     # SECURITY: force setting_sources=[] regardless of user YAML so the SDK
     # does NOT load .claude/settings.json or .mcp.json from the judge's cwd.
     # Those files can install pre-LLM lifecycle hooks (SessionStart /

src/coder_eval/evaluation/sub_agent.py

@@ -87,6 +87,17 @@ def __init__(
                 "SubAgentRunner requires agent_config.setting_sources=[] so the SDK does not "
                 + "load .claude/settings.json or .mcp.json from the sub-agent's working directory."
             )
+        # A sub-agent's system_prompt is its entire identity (judge instructions,
+        # simulator persona) — the claude_code coding-agent preset must never
+        # prefix it. Same fail-loud contract as setting_sources above: callers
+        # own their config, so a misconfigured one raises instead of being
+        # silently mutated.
+        if agent_config.system_prompt is not None and agent_config.system_prompt_mode != "replace":
+            raise ValueError(
+                "SubAgentRunner requires agent_config.system_prompt_mode='replace' when a "
+                + "system_prompt is set: the sub-agent prompt is its entire identity and must "
+                + "not be appended to the claude_code coding-agent preset."
+            )
         assert sandbox.sandbox_dir is not None, "sandbox not initialized"
         self._sandbox = sandbox
         self._agent_config = agent_config

src/coder_eval/models/agent_config.py

@@ -151,7 +151,9 @@ class BaseAgentConfig(BaseModel):
     system_prompt: str | None = Field(
         default=None,
         description=(
-            "Custom system prompt. Replaces the default system prompt. "
+            "Custom system prompt. Built-in agents layer it on top of their default system prompt "
+            "rather than replacing it; Claude Code can opt out via system_prompt_mode: replace. "
+            "Each agent's doc page (docs/agents/) states the exact mechanism. "
             "Supports inline text or multi-line YAML strings. "
             "Mutually exclusive with system_prompt_file."
         ),
@@ -197,6 +199,15 @@ class ClaudeCodeAgentConfig(BaseAgentConfig):
 
     type: Literal[AgentKind.CLAUDE_CODE]  # type: ignore[assignment]
 
+    system_prompt_mode: Literal["append", "replace"] = Field(
+        default="append",
+        description=(
+            "How system_prompt combines with the Claude Code default prompt: 'append' layers it "
+            "after the SDK 'claude_code' preset, keeping the default's behavioral guidance; "
+            "'replace' sends it as the ENTIRE system prompt. Judge sub-agents force 'replace' so "
+            "the scoring instrument never carries the coding-agent persona."
+        ),
+    )
     claude_settings: str | dict[str, Any] | None = MergeField(
         strategy="deep",
         default=None,
@@ -244,6 +255,23 @@ def _validate_sdk_options_keys(cls, v: dict[str, Any]) -> dict[str, Any]:
                 )
         return v
 
+    @model_validator(mode="after")
+    def check_replace_mode_has_prompt(self) -> Self:
+        """Reject ``system_prompt_mode: replace`` with no prompt to replace with.
+
+        Without a configured prompt the options builder would fall back to the
+        claude_code preset (the append regime) while run.json's
+        ``system_prompt_semantics`` marker could label the run 'replace' —
+        silently mis-bucketing trend dashboards. ``system_prompt_file`` counts:
+        the task loader inlines it into ``system_prompt`` at resolution time.
+        """
+        if self.system_prompt_mode == "replace" and self.system_prompt is None and self.system_prompt_file is None:
+            raise ValueError(
+                "system_prompt_mode='replace' requires system_prompt (or system_prompt_file) to be set — "
+                + "there is no prompt to replace the Claude Code default with"
+            )
+        return self
+
 
 class CodexAgentConfig(BaseAgentConfig):
     """Codex agent configuration."""