diff --git a/CHANGELOG.md b/CHANGELOG.md index 88cc3b6..984505e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,35 @@ +## [0.16.6] - 2026-09-08 + +Patch release — closes the SDK↔backend drift introduced by backend `DEF-SDKK-022-EXEC-BYPASS` (2026-09-04, RUN_ID=20260904T1500). After that backend fix, `/api/v1/execute` runs an `execution:{id}` ownership-binding existence check and returns 404 EXECUTION_NOT_FOUND for any execution_id that was not minted by a prior `/api/v1/gate`. The SDK's `runtime.execute()` had been minting a fresh `uuid7_str()` regardless of prior `/gate`, so every `@protect @sensitive` call returned 404 ("Gateway returned 404") and the displayed workflow_id was the misleading `__nullrun_unknown__` sentinel. LangGraph's `NullRunCallback.on_llm_start` had the symmetric problem on the LLM span side: it fired `llm_call` cost events with no paired `/gate` reservation, so the runtime's `_route_track` silently dropped them. This release closes all three holes. No wire-format change. + +### Fixed + +- **DEFS-SDKEXEC-GATE-FIRST** — `runtime.execute()` reuses the server-minted execution_id from `_server_minted_execution_id_var` when a prior `/gate` minted it (`src/nullrun/runtime.py:2820+`). Pre-fix minted `uuid7_str()` unconditionally; post-fix reads the contextvar (set by `check_workflow_budget`'s `_capture_server_minted_execution_id` from the `/gate` response's `reservation_id` field) and only mints fresh when the contextvar is empty (direct callers without a prior `/gate`, which is a wire-contract violation the backend's 404 handles correctly). Comment block at the fix site names both DEFS-SDKEXEC-GATE-FIRST and DEF-SDKK-022-EXEC-BYPASS so future readers see the round-trip contract without searching. +- **DEFS-SDKEXEC-WORKFLOW-LABEL** — `_enforce_sensitive_tool` displays the API key's bound workflow via `runtime._resolve_workflow_id(get_workflow_id())` instead of the literal `__nullrun_unknown__` sentinel (`src/nullrun/decorators.py`). The wire still carries the same workflow_id (server-side binding); only the displayed label changes. Two sites updated (extract failure path + main path). +- **DEFS-SDKEXEC-LLM-RESERVATION** — `NullRunCallback.on_llm_start` (`src/nullrun/instrumentation/langgraph.py`) fires `runtime.check_workflow_budget()` (fail-OPEN) so the matching `on_llm_end` `llm_call` cost event has a server-minted reservation_id and routes via `/track_single` instead of being dropped by `runtime._route_track` (the WARNING log "dropping llm_call event — no server-minted reservation_id in scope"). The call is wrapped in `except BaseException` so a backend outage or `WorkflowKilledInterrupt` / `WorkflowPausedException` never breaks the LangChain callback contract. +- **`Transport.execute` docstring** (`src/nullrun/transport.py`) — rewrites the misleading pre-2026-09-04 claim ("/execute MUST be called rather than /gate") to reflect the post-DEF-SDKK-022-EXEC-BYPASS contract ("/execute MUST be preceded by /gate for the same execution_id"). Names both fix tags so the contract is grep-able. + +### Added + +- **`tests/test_2026_09_08_gate_first_execute.py`** (9 tests). Source-pin regression for all three fixes: + - `runtime.execute()` reads `get_server_minted_execution_id()` and reuses it when present (forbids re-introducing an unconditional `uuid7_str()` mint outside the fallback arm). + - `_enforce_sensitive_tool` displays via `runtime._resolve_workflow_id(...)` (forbids the pre-fix contextvar-only fallback). + - `Transport.execute` docstring references the post-fix contract (forbids the legacy misleading claim). + - `NullRunCallback.on_llm_start` calls `check_workflow_budget()` with a never-raise guard. + - Contextvar round-trip sanity (`set_server_minted_execution_id` / `get_server_minted_execution_id`). + +### Compatibility + +Pure reliability fixes — no wire-format change. `/gate`, `/execute`, `/track`, `/cancel` payloads are byte-identical to 0.16.5. The drift existed only on the SDK side; this release brings the SDK in line with the backend's 2026-09-04 contract without rolling back any backend-side hardening. + +### Why this is needed + +**Gate-first** — the user-facing symptom was that `langgraph_openai_approval_demo.py` (and any `@protect @sensitive` decorator that was actually wired through `runtime.execute()`) returned `Workflow __nullrun_unknown__ blocked: Gateway returned 404` for every call, with `action=block, status_code=None`. The approval rule never had a chance to fire because the 404 was raised on the existence-of-binding check before the policy engine ran. The 0.12.0 SDK had been silently broken against post-2026-09-04 backends for the entire /execute path; this release closes the four-day window of broken `/execute` behaviour. + +**Workflow label** — `__nullrun_unknown__` was misleading because the SDK did know the workflow (the API key's binding) but only read the contextvar (which was unset on bare `@protect` calls). The displayed label was wrong; the wire was right. Operators reading traces had no signal that the gate had, in fact, scoped the call to a real workflow. + +**LLM reservation** — LangGraph's `NullRunCallback` emits LLM cost events from the LangChain callback hooks. These have no `@protect` scope and therefore no paired `/gate`. The runtime's `_route_track` (which since v0.16.0 / 2026-08-20 backend v3.66.2 alignment refuses to fall back to `/track/batch` for `llm_call` events without a reservation) dropped them with a WARNING log. Cost attribution for agentic LLM loops was silently incomplete. The fix fires `/gate` once per LLM span (fail-OPEN; same wire-call shape as `@protect`), so cost attribution completes via the v3 `/track_single` path. + ## [0.16.5] - 2026-09-05 Patch release — two independent reliability fixes: (1) `@protect` cancel-on-exception orphan leak (Redis reservation leak on tool exceptions), (2) P0-26+P0-27 `operation_id` hoist (single-source mint, server-vs-SDK divergence detection). No wire-format change on either fix. diff --git a/pyproject.toml b/pyproject.toml index 93398d1..0fd3707 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "hatchling.build" name = "nullrun" # Full release history lives in CHANGELOG.md; only the current version # is pinned here. -version = "0.16.5" +version = "0.16.6" # Kept under the 200-char preview threshold so the full line is visible # without an "expand" click. The headline is the canonical §1 statement # from positioning.md — "runtime decision layer for tool-using AI agents" diff --git a/src/nullrun/__version__.py b/src/nullrun/__version__.py index f46df0a..a102a9c 100644 --- a/src/nullrun/__version__.py +++ b/src/nullrun/__version__.py @@ -5,5 +5,5 @@ string and the SDK_MIN_VERSION constant. """ -__version__ = "0.16.5" +__version__ = "0.16.6" __platform_version__ = "1.0.0" diff --git a/src/nullrun/_handle.py b/src/nullrun/_handle.py index 7a9a643..3ef6742 100644 --- a/src/nullrun/_handle.py +++ b/src/nullrun/_handle.py @@ -18,10 +18,13 @@ All three translate any:class:`nullrun.NullRunError` into a single ``print(format_user_message(exc), file=sys.stderr)`` followed by -``sys.exit(1)``.:class:`nullrun.WorkflowKilledInterrupt` is a -``BaseException`` subclass and therefore propagates through all three -— the kill signal is never silently swallowed. Non-NullRun exceptions -also propagate unchanged. +``sys.exit(1)``.:class:`nullrun.WorkflowKilledInterrupt` now inherits +from :class:`nullrun.NullRunError` (the 2026-09-08 migration; see the +class docstring), so a bare ``except NullRunError`` would otherwise +swallow the kill signal. ``handle``/``guarded`` explicitly re-raise it +— the kill is a control-plane action, not an SDK failure, and must +reach the top of the agent loop. Non-NullRun exceptions also propagate +unchanged. ``init_or_die`` exists because:func:`nullrun.init` is typically called at module top-level — before any ``with handle: `` block or @@ -56,7 +59,7 @@ from contextlib import contextmanager from typing import TypeVar -from nullrun.breaker.exceptions import NullRunError +from nullrun.breaker.exceptions import NullRunError, WorkflowKilledInterrupt from nullrun.messages import format_user_message T = TypeVar("T") @@ -75,11 +78,16 @@ def handle(*, exit_code: int = 1): Exceptions that propagate unchanged: - *:class:`nullrun.WorkflowKilledInterrupt` (``BaseException``) — kill - signals must reach the top of the agent loop, not be swallowed - into a graceful exit. + *:class:`nullrun.WorkflowKilledInterrupt` — kill signals must reach + the top of the agent loop, not be swallowed into a graceful exit. + Re-raised explicitly inside the ``except NullRunError`` branch + because the 2026-09-08 migration moved ``WorkflowKilledInterrupt`` + onto the ``NullRunError`` MRO (Sentry/OTel ``except Exception`` + handlers should now record kill events; this ``handle`` / + ``guarded`` wrapper opts OUT of that recording on purpose). *:class:`KeyboardInterrupt` /:class:`SystemExit` (``BaseException``) — - same reason as the kill signal. + same reason as the kill signal — never reach the + ``except NullRunError`` branch anyway. * Any non-NullRun exception — the user's own bugs are not handled here; let them propagate for an honest traceback. @@ -101,6 +109,15 @@ def handle(*, exit_code: int = 1): try: yield except NullRunError as exc: + # 2026-09-08 migration: WorkflowKilledInterrupt moved onto + # the NullRunError MRO so Sentry/OTel `except Exception` + # handlers record kill events. ``handle``/``guarded`` are the + # friendly-exit pattern, NOT the user-callback pattern — kill + # is a control-plane action and must propagate so the agent + # loop / dashboard resume path can see it. Re-raise explicitly + # before the catalog print + sys.exit. + if isinstance(exc, WorkflowKilledInterrupt): + raise print(format_user_message(exc), file=sys.stderr) sys.exit(exit_code) @@ -111,7 +128,8 @@ def guarded(fn: Callable[..., T]) -> Callable[..., T]: Wrap a function so any:class:`nullrun.NullRunError` raised inside it is caught, rendered as a user-facing message, and the process exits with code ``1``. ``WorkflowKilledInterrupt`` and other - ``BaseException`` subclasses propagate. + ``BaseException`` subclasses propagate (``handle`` re-raises kill + explicitly, see the 2026-09-08 migration note). Pair with:func:`nullrun.protect` for the standard agent loop:: diff --git a/src/nullrun/actions.py b/src/nullrun/actions.py index f4d117c..6fd7c44 100644 --- a/src/nullrun/actions.py +++ b/src/nullrun/actions.py @@ -22,6 +22,7 @@ from nullrun.breaker.exceptions import ( NullRunBlockedException, + NullRunWorkflowKilledError, WorkflowKilledInterrupt, WorkflowPausedException, ) @@ -69,7 +70,7 @@ class ActionHandler: Handler for NullRun circuit breaker actions. This executes protective actions when triggered: - - KILL: Immediately stops the workflow (raises WorkflowKilledInterrupt) + - KILL: Immediately stops the workflow (raises NullRunWorkflowKilledError, NR-W002) - PAUSE: Temporarily halts the workflow (raises WorkflowPausedException) - ALERT: Sends notification (can be customized) - SNAPSHOT: Captures workflow state for debugging @@ -179,7 +180,10 @@ def handle( **details: Additional details about the action Raises: - WorkflowKilledInterrupt: If action is "kill" + NullRunWorkflowKilledError: If action is "kill" + (2026-09-08 typed signal, NR-W002; subclass of + WorkflowKilledInterrupt which remains as the + back-compat name.) WorkflowPausedException: If action is "pause" NullRunBlockedException: If action is "block" """ @@ -230,12 +234,11 @@ def handle( except BaseException as e: # Don't let handler exceptions propagate. We catch # `BaseException` (not just `Exception`) because - # `WorkflowKilledInterrupt` is intentionally a - # `BaseException` subclass — it's a non-recoverable - # control signal, but inside the ActionHandler dispatch - # loop we want the kill to be recorded in history - # (already done above) and swallowed, NOT re-raised into - # the caller's frame. + # kill signals (NullRunWorkflowKilledError, the + # 2026-09-08-migrated Exception subclass) and any + # third-party kill-shaped signals must be recorded + # in history (already done above) and swallowed, + # NOT re-raised into the caller's frame. logger.error(f"Action handler error: {e}") def _default_kill( @@ -244,9 +247,21 @@ def _default_kill( reason: str, **details: Any, ) -> None: - """Default kill handler - raises WorkflowKilledInterrupt.""" + """Default kill handler - raises NullRunWorkflowKilledError. + + 2026-09-08: typed kill signal (NR-W002). Cookbook code + can `except NullRunWorkflowKilledError` to react to + operator-initiated kills with structured error_code + + user_action. Legacy `except WorkflowKilledInterrupt` + still matches because NullRunWorkflowKilledError is a + subclass. + """ logger.warning(f"KILL action for workflow {workflow_id}: {reason}") - raise WorkflowKilledInterrupt(workflow_id=workflow_id, reason=reason) + raise NullRunWorkflowKilledError( + workflow_id=workflow_id, + reason=reason, + kill_source="action_handler", + ) def _default_pause( self, diff --git a/src/nullrun/breaker/exceptions.py b/src/nullrun/breaker/exceptions.py index 11b3c32..473ebf5 100644 --- a/src/nullrun/breaker/exceptions.py +++ b/src/nullrun/breaker/exceptions.py @@ -842,6 +842,107 @@ def __init__( self.recheck_retryable: bool = True +class NullRunBudgetThrottleError(NullRunBudgetError): + """Backend returned ``decision == "throttle"`` — soft budget signal. + + Distinct from :class:`NullRunBudgetError` (NR-B004, the hard-block + case raised when ``decision == "block"``). Throttle means + "rate-limit this workflow but don't fully block it" — a temporary + pacing signal that the SDK surfaces as a typed exception so + cookbook code can back off and retry, vs. the hard block where + the same parameters would fail again. + + Added 2026-09-08 to retire the generic ``WorkflowKilledInterrupt`` + raise on the throttle path. Cookbook pattern: catch this + specifically (``except NullRunBudgetThrottleError``), sleep for + the cooldown window, and retry — distinct from the hard block + where retrying with the same budget tier is futile. + """ + + error_code = "NR-B007" + user_action = ( + "Backend throttled this workflow (soft budget signal). Wait " + "for the cooldown window shown in the response and retry — " + "do NOT request a budget increase for a throttle (that is " + "the wrong remediation; the issue is pacing, not cap)." + ) + retryable = True + + +class NullRunExecutionNotFoundError(NullRunBackendError): + """``/execute`` or ``/cancel`` was called with an ``execution_id`` that + has no live server-side binding. + + Wire code ``EXECUTION_NOT_FOUND`` (HTTP 404) from backend + `GateErrorCode::ExecutionNotFound` (`error_codes.rs`). Two emission + sites: + - ``backend/src/proxy/http/gate/execute.rs:194`` — when /execute + fires before /gate (or after the binding TTL expired) + - ``backend/src/proxy/http/cancel.rs:303`` — same condition on + the cancel path + + Cookbook pattern: do NOT retry the same ``execution_id``; the + server never minted it (or its binding has expired and the + reservation has been released). Re-issue ``/api/v1/gate`` to mint + a fresh ``execution_id``, then retry /execute. + + Subclass of :class:`NullRunBackendError` (NR-GEN) so the existing + ``except NullRunBackendError:`` cookbook pattern keeps matching; + callers that want to handle this specific case can ``except + NullRunExecutionNotFoundError`` for a clearer intent. + + Audit: 2026-09-09 SDK-drift audit — pre-fix SDK 0.15.x collapsed + this code into a generic ``NullRunBackendError("Execution binding + not found")`` with no introspection on whether /gate was missed. + """ + + error_code = "NR-EX01" + user_action = ( + "/execute (or /cancel) was called without a prior /gate that " + "minted this execution_id — or the binding TTL expired. " + "Re-issue /api/v1/gate to get a fresh execution_id, then retry." + ) + retryable = False + + def __init__( + self, + message: str, + *, + execution_id: str | None = None, + endpoint: str | None = None, + status_code: int | None = None, + ) -> None: + # Wire detail envelope carries execution_id + endpoint; + # promote them to first-class kwargs on the exception so + # cookbook code can introspect without indexing into + # ``details``. The parent (NullRunBackendError) accepts + # ``endpoint`` as a named param and ``**details`` for + # everything else, so we route execution_id through + # details to avoid colliding with the parent's signature. + details: dict[str, Any] = {} + if execution_id is not None: + details["execution_id"] = execution_id + super().__init__( + message=message, + endpoint=endpoint or "/api/v1/execute", + status_code=status_code, + **details, + ) + # First-class attributes so cookbook code can introspect + # which execution_id and which endpoint surfaced the 404 + # without indexing into ``details``. + self.execution_id: str | None = execution_id + # ``endpoint`` is always set after ``super().__init__`` (the + # parent constructor receives ``endpoint or "/api/v1/execute"``, + # never None). Override the inherited ``str`` annotation with the + # same type so mypy is happy — we are narrowing the parent's + # declared type by subclass attribute re-assignment here, not + # widening it. + self.endpoint: str = endpoint or "/api/v1/execute" + # Re-issue /gate is the only path forward. + self.regate_required: bool = True + + class NullRunToolBlockedError(NullRunBlockedException): """The tool is in the workflow's block list. @@ -887,6 +988,12 @@ class NullRunApprovalNotYetApprovedError(NullRunBlockedException): no — terminal) and from :class:`NullRunApprovalExpiredError` (operator said yes but grant TTL elapsed). All three share the HTTP 403 envelope; the wire code is the discriminator. + + Also raised client-side (NOT just wire path) on the /execute + "approval_id missing in response" malformed-payload case — see + ``NullRunApprovalResponseMissingError`` for the precise semantic + distinction (NR-A004 is the wire-bug code, NR-A010 is "operator + has not decided yet"). """ error_code = "NR-A010" @@ -897,6 +1004,55 @@ class NullRunApprovalNotYetApprovedError(NullRunBlockedException): ) retryable = True + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + # First-class attribute so cookbook code can introspect the + # pending approval row without parsing the message string. + self.approval_id = approval_id + + +class NullRunApprovalResponseMissingError(NullRunBlockedException): + """``/execute`` returned ``require_approval`` but the response body + did not include an ``approval_id`` — wire-bug / server drift. + + Wire code ``NR-A004`` (was previously set inline on a generic + ``NullRunBlockedException`` at runtime.py:2888, 2914, 2929 — promoted + to a typed class for parity with the six approval exceptions above). + This is distinct from ``NullRunApprovalNotYetApprovedError`` (NR-A010) + which is "the operator has not yet decided". Here the operator never + had a chance — the wire envelope was incomplete. + + Cookbook pattern: do NOT retry the same execution_id; the backend + needs a fix or the wire-shape contract needs re-reading. Log the + full response body and report to NULLRUN support. + """ + + error_code = "NR-A004" + user_action = ( + "Server returned require_approval without an approval_id — " + "this is a wire-contract bug, NOT a transient failure. Inspect " + "the full response body and report to NullRun support; do not " + "retry the same execution_id." + ) + retryable = False + class NullRunApprovalDeniedError(NullRunBlockedException): """Operator explicitly denied the approval. @@ -905,6 +1061,10 @@ class NullRunApprovalDeniedError(NullRunBlockedException): with the same approval_id will keep failing. Cookbook pattern: surface denial to the user and request a fresh approval row (different parameters / intent). + + Now raised client-side (NOT just wire path) on the WS push "denied" + outcome at ``check_workflow_budget`` and on the /execute "outcome + != approved" branch. """ error_code = "NR-A011" @@ -915,24 +1075,136 @@ class NullRunApprovalDeniedError(NullRunBlockedException): ) retryable = False + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + denial_note: str | None = None, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + self.approval_id = approval_id + self.denial_note = denial_note + class NullRunApprovalExpiredError(NullRunBlockedException): """Approval grant aged out — operator said yes but ``expires_at`` is past. - Wire code ``APPROVAL_EXPIRED`` (HTTP 403). The original grant was - approved but the operator's approval window elapsed before - ``/execute`` consumed it. Cookbook pattern: request a fresh - approval row (do not retry the same one). + Wire code ``APPROVAL_EXPIRED`` (HTTP 403). Two raise paths: + + 1. **Wire path** — backend returns APPROVAL_EXPIRED on /execute + because the operator's grant TTL elapsed between /gate and + /execute. + 2. **Client-side timeout path** (added 2026-09-08, the trigger for + this typed exception migration) — WS push went silent for + ``approval_timeout_seconds`` (default 300s) without an operator + decision. The SDK raises this exception instead of the generic + ``WorkflowKilledInterrupt`` so cookbook code can catch it + (`except NullRunApprovalExpiredError`) and react with a fresh + approval request. + + Cookbook pattern: do NOT retry the same approval_id — request a + fresh row and re-/gate. """ error_code = "NR-A012" user_action = ( - "Approval grant has expired — the operator approved, but " - "the grant's expires_at is past. Request a fresh approval " - "row and retry /execute with the new approval_id." + "Approval expired — no operator decision within the configured " + "timeout window (WS push silent past approval_timeout_seconds). " + "Request a fresh approval row and retry /gate; the previous " + "approval_id cannot be revived." ) retryable = False + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + timeout_seconds: float | None = None, + local_timeout: bool = False, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + self.approval_id = approval_id + # Server-authoritative timeout the SDK waited for. ``None`` when + # the exception came from the wire path (where the backend + # already closed the grant; the SDK never started a wait). + self.timeout_seconds = timeout_seconds + # True when raised by the SDK on local WS-silent timeout (path 2 + # above); False when raised by the wire path (path 1). Lets + # cookbook code distinguish "operator never saw the request" + # (local timeout — maybe the request never propagated) from + # "operator approved but grant TTL elapsed" (wire path). + self.local_timeout = local_timeout + + +class NullRunApprovalReplayRejectedError(NullRunBlockedException): + """Approval grant was already consumed by a prior /execute call. + + Wire code ``APPROVAL_REPLAY_REJECTED`` (HTTP 403). Each grant + is single-use per ``consume_approved`` atomic check-and-set. + Cookbook pattern: do NOT retry the same approval_id; treat as + idempotency violation (likely a client retry loop). + + Also raised client-side on the /execute "post-approval re-check + returned require_approval again" race (the operator approved but + the same approval_id was already consumed by a concurrent /execute). + """ + + error_code = "NR-A015" + user_action = ( + "Approval grant was already consumed by a prior /execute " + "call — this is a replay/retry-loop signal, NOT a transient " + "failure. Inspect your retry logic; the same approval_id " + "will never succeed twice." + ) + retryable = False + + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + self.approval_id = approval_id + class NullRunApprovalDigestMismatchError(NullRunBlockedException): """Business-impact digest drifted since operator approval (ADR-006). @@ -975,23 +1247,13 @@ class NullRunApprovalToolDigestMismatchError(NullRunBlockedException): retryable = False -class NullRunApprovalReplayRejectedError(NullRunBlockedException): - """Approval grant was already consumed by a prior /execute call. - - Wire code ``APPROVAL_REPLAY_REJECTED`` (HTTP 403). Each grant - is single-use per ``consume_approved`` atomic check-and-set. - Cookbook pattern: do NOT retry the same approval_id; treat as - idempotency violation (likely a client retry loop). - """ - - error_code = "NR-A015" - user_action = ( - "Approval grant was already consumed by a prior /execute " - "call — this is a replay/retry-loop signal, NOT a transient " - "failure. Inspect your retry logic; the same approval_id " - "will never succeed twice." - ) - retryable = False +# NOTE: NullRunApprovalReplayRejectedError was moved earlier in this +# module (alongside the other five approval exceptions) so all six +# typed approval exceptions are co-located. The earlier definition +# also adds an explicit ``__init__`` accepting ``approval_id`` as a +# first-class attribute. See the block just below the +# ``NullRunApprovalNotYetApprovedError`` docstring for the canonical +# definition. # NOTE: the following six exception classes were removed in 0.4.0 @@ -1095,27 +1357,35 @@ def __init__(self, workflow_id: str, reason: str) -> None: super().__init__(f"Workflow {workflow_id} killed: {reason}") -class WorkflowKilledInterrupt(WorkflowKilledException): +class WorkflowKilledInterrupt(NullRunError): """ Raised when a workflow is killed by the NullRun control plane. - Inherits from the deprecated:class:`WorkflowKilledException` - (which is itself a ``BaseException`` subclass, not ``Exception``) - so that: + **2026-09-08 migration**: this class is now an ``Exception`` + subclass (``NullRunError`` parent) — formerly ``BaseException``. + The user override: agent recovery code needs to catch the kill + signal via ``except WorkflowKilledInterrupt`` or + ``except NullRunWorkflowKilledError`` to surface a structured + error to the user with ``error_code=NR-W002`` and ``user_action``. - * ``except WorkflowKilledInterrupt`` (new code) catches new raises - and only new raises. - * ``except WorkflowKilledException`` (legacy user code) still - catches new raises — back-compat. - * ``except Exception`` does **not** catch this signal — kill is - not a recoverable error. Mirrors the ``KeyboardInterrupt`` / - ``SystemExit`` pattern from the standard library: user code - that catches ``except Exception`` and re-runs the work will - silently bypass the kill. - * ``except BaseException`` catches it, like the stdlib interrupts. + Migration back-compat guarantees (all three hold): - See ``docs/kill-contract.md` for the full rationale, including - the four-level coverage model and the decision tree for users. + * ``except WorkflowKilledInterrupt`` (new code) — still matches, + including legacy raises that haven't been updated. + * ``except NullRunError`` — now matches (was NO match before + migration; this is the new ability the user wanted). + * ``except NullRunWorkflowKilledError`` — matches (preferred + typed name for new cookbook code). + + Migration BREAK (acceptable, documented in CHANGELOG): + + * ``except WorkflowKilledException`` (the deprecated parent + class) — no longer matches. The parent class remains + BaseException and emits DeprecationWarning on construction, + but is no longer in the ``WorkflowKilledInterrupt`` MRO. Code + that catches the deprecated name must migrate to either + ``WorkflowKilledInterrupt`` (keep current name) or + ``NullRunWorkflowKilledError`` (preferred typed name). Fields: workflow_id: The workflow that was killed. @@ -1124,31 +1394,99 @@ class WorkflowKilledInterrupt(WorkflowKilledException): Catching in production ---------------------- - ``WorkflowKilledInterrupt`` is a ``BaseException`` subclass - (NOT ``Exception``), so a user-agent ``try / except Exception`` - will not catch it. This is intentional — the kill signal - must reach the top of the loop. It does mean, however, that - Sentry / OpenTelemetry default error handlers (which filter - on ``Exception``) will not record the kill event unless the - user's code re-raises it under an ``except BaseException``: - - from sentry_sdk import capture_exception + ``WorkflowKilledInterrupt`` is now an ``Exception`` subclass. + Cookbook code can do:: + + try: + agent.run() + except NullRunWorkflowKilledError as exc: + surface_to_user( + f"Workflow {exc.workflow_id} was killed: {exc.reason}. " + f"{exc.user_action}" + ) + + or for broader catch:: + try: - agent.run - except BaseException: - capture_exception # records kill, ctrl-c, system-exit + agent.run() + except Exception as exc: + # Now catches kill signals too (the new contract). + sentry_sdk.capture_exception(exc) raise - ``except Exception`` will swallow non-kill errors but let the - kill through. ``except BaseException`` captures everything - including the kill — recommended for the top of an agent loop. + Sentry / OpenTelemetry handlers that filter on ``Exception`` will + now record kill events — this is the intended new behavior. Code + that relies on kill being un-catchable by ``except Exception`` is + a regression candidate; see ``docs/kill-contract-migration-2026-09-08.md``. """ - def __init__(self, workflow_id: str, reason: str) -> None: - # Bypass the parent's __init__ so constructing the canonical - # class does NOT trigger the parent's DeprecationWarning. The - # deprecation is about using the old *name* — not the - # BaseException-based hierarchy. + error_code = "NR-W002" + user_action = ( + "The workflow was killed by the NullRun control plane. The " + "body did not run. Inspect the reason (killed via dashboard, " + "killed via API, circuit-breaker tripped, etc.) and, if " + "appropriate, resume the workflow at " + "https://app.nullrun.io/workflows/." + ) + retryable = False + + def __init__( + self, + workflow_id: str, + reason: str, + *, + kill_source: str | None = None, + **details: Any, + ) -> None: + # Skip NullRunError.__init__'s kwargs-by-key path — we want + # the structured fields attached as instance attrs (matches + # the pre-migration shape) AND surfaced through the NullRunError + # fields too, so cookbook introspection works either way. self.workflow_id = workflow_id self.reason = reason - BaseException.__init__(self, f"Workflow {workflow_id} killed: {reason}") + # First-class attribute distinguishing operator kill from + # circuit-breaker kill, etc. None when the source is ambiguous. + self.kill_source = kill_source + NullRunError.__init__( + self, + f"Workflow {workflow_id} killed: {reason}", + error_code=self.error_code, + user_action=self.user_action, + **details, + ) + + +class NullRunWorkflowKilledError(WorkflowKilledInterrupt): + """Typed public name for the kill signal. + + Subclass of :class:`WorkflowKilledInterrupt` (which remains the + legacy canonical name) so ``except WorkflowKilledInterrupt`` + clauses continue to match. New cookbook code should prefer this + name (``except NullRunWorkflowKilledError``) for typed dispatch. + + Wire code ``NR-W002`` (same as parent). Distinct from + :class:`NullRunBlockedException` family — kill is a control-plane + signal (operator or circuit-breaker), not a gate-decision block. + + Cookbook pattern (2026-09-08 migration): + + try: + agent.run() + except NullRunWorkflowKilledError as exc: + # Structured fields ready for the LLM: + # exc.workflow_id, exc.reason, exc.kill_source, + # exc.error_code ("NR-W002"), exc.user_action + surface_to_user( + f"Workflow {exc.workflow_id} was killed " + f"(source={exc.kill_source}): {exc.user_action}" + ) + """ + + error_code = "NR-W002" + user_action = ( + "Workflow was killed by the NullRun control plane (operator " + "action or circuit-breaker). The body did not run. Resume " + "the workflow at https://app.nullrun.io/workflows/ " + "or inspect the reason before retrying." + ) + retryable = False diff --git a/src/nullrun/decorators.py b/src/nullrun/decorators.py index 4d5c17c..bc1d926 100644 --- a/src/nullrun/decorators.py +++ b/src/nullrun/decorators.py @@ -742,7 +742,13 @@ def _enforce_sensitive_tool( TransportErrorSource, ) - workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID + # DEFS-SDKEXEC-WORKFLOW-LABEL (2026-09-08): prefer the + # runtime's bound workflow (from _authenticate) over the + # sentinel so the displayed label matches what the SDK + # actually sends to /gate / /execute. See the matching + # note in `_enforce_sensitive_tool` below for the full + # rationale. + workflow_id = runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID # The user-facing hint depends on which extractor fired. # Money extractor wants the bound arg name; ToolParams # extractor wants the rule-param -> arg-name mapping @@ -807,7 +813,19 @@ def _enforce_sensitive_tool( ) fail_open = os.environ.get("NULLRUN_SENSITIVE_FAIL_OPEN", "").strip() == "1" - workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID + # DEFS-SDKEXEC-WORKFLOW-LABEL (2026-09-08): resolve the + # *display* workflow_id via the runtime's precedence chain + # (contextvar → self.workflow_id → None) so the label reflects + # what the SDK actually sends on the wire (the API key's bound + # workflow, when the user hasn't explicitly opened a + # ``with workflow(...)`` block). Pre-fix this read only the + # contextvar; on every API-key-bound key without an explicit + # workflow block the displayed label was the literal sentinel + # ``"__nullrun_unknown__"``, which misleads operators reading + # the trace and the block message into thinking the gate was + # unable to identify the workflow. Sentinel stays as the last + # resort for legacy / never-bound keys. + workflow_id = runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID try: # Pass on_transport_error="raise" so the transport raises diff --git a/src/nullrun/instrumentation/auto.py b/src/nullrun/instrumentation/auto.py index 501a28e..f135e20 100644 --- a/src/nullrun/instrumentation/auto.py +++ b/src/nullrun/instrumentation/auto.py @@ -740,7 +740,10 @@ def _check_kill_before_send(runtime: Any, request: httpx.Request) -> None: has a remote control plane to consult. Raises: - WorkflowKilledInterrupt: state == "Killed" + NullRunWorkflowKilledError: state == "Killed" (2026-09-08: + typed signal with error_code=NR-W002 + user_action; + subclass of WorkflowKilledInterrupt which remains as a + back-compat name.) WorkflowPausedException: state == "Paused" """ if runtime is None: @@ -761,10 +764,14 @@ def _check_kill_before_send(runtime: Any, request: httpx.Request) -> None: state = runtime._remote_state_for(workflow_id) if hasattr(runtime, "_remote_state_for") else getattr(runtime, "_remote_states", {}).get(workflow_id, {}) state_name = state.get("state", "Normal") if state_name == "Killed": - from nullrun.breaker.exceptions import WorkflowKilledInterrupt - raise WorkflowKilledInterrupt( + # 2026-09-08: typed kill signal (NR-W002). Cookbook + # code can `except NullRunWorkflowKilledError`; legacy + # `except WorkflowKilledInterrupt` still matches (subclass). + from nullrun.breaker.exceptions import NullRunWorkflowKilledError + raise NullRunWorkflowKilledError( workflow_id=workflow_id, reason=state.get("reason", "remote kill"), + kill_source="auto_instrumentation", ) if state_name == "Paused": from nullrun.breaker.exceptions import WorkflowPausedException diff --git a/src/nullrun/instrumentation/langgraph.py b/src/nullrun/instrumentation/langgraph.py index 107fd46..c784aeb 100644 --- a/src/nullrun/instrumentation/langgraph.py +++ b/src/nullrun/instrumentation/langgraph.py @@ -82,11 +82,16 @@ def _read_token_attrs(obj: Any) -> tuple[int, int, int, dict[str, Any]] | None: total_t = getattr(obj, "total_tokens", 0) or 0 if not (in_t or out_t or total_t): return None - return int(in_t), int(out_t), int(total_t), { - "input_tokens": in_t, - "output_tokens": out_t, - "total_tokens": total_t, - } + return ( + int(in_t), + int(out_t), + int(total_t), + { + "input_tokens": in_t, + "output_tokens": out_t, + "total_tokens": total_t, + }, + ) return None @@ -191,6 +196,7 @@ def _get_finish_reason(response: Any) -> str | None: # Usage Normalization (SDK extracts, backend computes) # ============================================================================= + def extract_usage_from_response(response: Any, provider: str, model: str) -> dict[str, Any]: """ Extract usage data from LLM response. @@ -258,7 +264,7 @@ def extract_usage_from_response(response: Any, provider: str, model: str) -> dic # Check for streaming chunks that accumulated usage # (streaming responses may not have usage until final chunk) - if not usage["has_usage"] and hasattr(response, '__iter__'): + if not usage["has_usage"] and hasattr(response, "__iter__"): # For streaming, we can't get accurate usage in middle of stream # Final response should have usage_metadata pass @@ -273,29 +279,19 @@ def extract_usage_from_response(response: Any, provider: str, model: str) -> dic # OpenAI exposes cached_tokens on a nested prompt_tokens_details. raw = usage.get("raw_usage") or {} if isinstance(raw, dict): - cache_read = raw.get("cache_read_input_tokens") or raw.get( - "cacheReadInputTokenCount" - ) + cache_read = raw.get("cache_read_input_tokens") or raw.get("cacheReadInputTokenCount") if cache_read: usage["cache_read_tokens"] = int(cache_read) or 0 - cache_write = raw.get("cache_creation_input_tokens") or raw.get( - "cacheWriteInputTokenCount" - ) + cache_write = raw.get("cache_creation_input_tokens") or raw.get("cacheWriteInputTokenCount") if cache_write: usage["cache_write_tokens"] = int(cache_write) or 0 prompt_details = raw.get("prompt_tokens_details") or {} if isinstance(prompt_details, dict) and prompt_details.get("cached_tokens"): # OpenAI's prefix-cached prompt hits — best-effort merge. - usage["cache_read_tokens"] = int( - prompt_details.get("cached_tokens") or 0 - ) + usage["cache_read_tokens"] = int(prompt_details.get("cached_tokens") or 0) completion_details = raw.get("completion_tokens_details") or {} - if isinstance(completion_details, dict) and completion_details.get( - "reasoning_tokens" - ): - usage["reasoning_tokens"] = int( - completion_details.get("reasoning_tokens") or 0 - ) + if isinstance(completion_details, dict) and completion_details.get("reasoning_tokens"): + usage["reasoning_tokens"] = int(completion_details.get("reasoning_tokens") or 0) # Finish reason — read from every known source independently of the # token branch. The `elif`-chain above means only one branch fills @@ -370,9 +366,7 @@ def _extract_tool_names(obj: Any) -> list[str]: # Determine if we got real usage data usage["has_usage"] = ( - usage["total_tokens"] > 0 or - usage["input_tokens"] > 0 or - usage["output_tokens"] > 0 + usage["total_tokens"] > 0 or usage["input_tokens"] > 0 or usage["output_tokens"] > 0 ) return usage @@ -518,6 +512,50 @@ def on_llm_start(self, serialized: Any, prompts: Any, **kwargs: Any) -> None: else: ctx = create_root_span() self._register_active_run(str(run_id), ctx) + + # DEFS-SDKEXEC-LLM-RESERVATION (2026-09-08): pair the LLM + # span with a server-minted reservation so the matching + # llm_call cost event emitted by ``on_llm_end`` lands on + # ``/track_single`` instead of being dropped by + # ``runtime._route_track`` (which returns silently when + # ``_server_minted_execution_id_var`` is unset — see + # ``runtime.py:3167`` and the WARNING log at line 3198). + # + # Pattern: ``check_workflow_budget`` fails OPEN on transport + # error (CLAUDE.md §4 / ADR-008 / ``runtime.py:2022-2037``) + # — a backend outage silently returns without raising AND + # without capturing a reservation. The downstream + # ``_route_track`` will then drop the matching llm_call + # cost event (v3.66.2 alignment — backend rejects batched + # llm_call events without a reservation with 503 + # BUDGET_RECHECK_FAILED). This matches the pre-fix + # behaviour because pre-fix the SDK also had no reservation + # at this site (no /check round-trip happened on the LLM + # span) and the llm_call cost event was dropped the same + # way. We swallow ``WorkflowKilledInterrupt`` / + # ``WorkflowPausedException`` because the LangChain + # callback contract is "never raise" (the framework breaks + # if a callback raises); the kill/pause signal still + # propagates because the next @protect on a sensitive tool + # re-runs ``check_control_plane``. + # + # Cost: one extra /gate round-trip per LLM span (~5 ms in + # the hot path). The /track emitted by ``on_llm_end`` + # consumes the matching reservation, so the budget ledger + # stays balanced (1 reserve + 1 consume per LLM call). + # Inside an existing ``@protect`` block the contextvar is + # already populated; ``check_workflow_budget`` short-circuits + # early via the chain-mode cache when an active chain is in + # scope, so the wire-call cost is amortised across the chain. + try: + self.runtime.check_workflow_budget() + except BaseException as exc: # noqa: BLE001 — never raise out of callback + logger.debug( + "NullRunCallback.on_llm_start: check_workflow_budget " + "raised %s — proceeding without reservation (llm_call " + "cost event will be dropped by runtime._route_track)", + type(exc).__name__, + ) try: self.runtime.track_event( event_type="span_start", @@ -570,23 +608,25 @@ def on_llm_end(self, response: Any, **kwargs: Any) -> None: # fall back to the response object. This matches the # best-effort pattern used by ``_get_finish_reason`` / # ``_extract_tool_names`` for the same response. - invocation_params = kwargs.get('invocation_params') or {} + invocation_params = kwargs.get("invocation_params") or {} model = ( - invocation_params.get('model_name') + invocation_params.get("model_name") or _extract_model_from_response(response) - or 'unknown' + or "unknown" ) provider = ( - invocation_params.get('model_provider') + invocation_params.get("model_provider") or _extract_provider_from_response(response) - or 'openai' + or "openai" ) # Extract usage (normalized format) usage = extract_usage_from_response(response, provider, model) - logger.info(f"NullRun callback: model={model}, provider={provider}, " - f"usage={usage}, has_usage={usage['has_usage']}") + logger.info( + f"NullRun callback: model={model}, provider={provider}, " + f"usage={usage}, has_usage={usage['has_usage']}" + ) # Audit 2026-06-29 (unified fingerprint): derive the same # fingerprint the httpx transport computes for the same @@ -719,9 +759,7 @@ def on_llm_end(self, response: Any, **kwargs: Any) -> None: # orphan-span finding (parent_span_id points at a run_id # that no longer exists in ``_active_runs``). with self._lock: - llm_ctx = ( - self._active_runs.get(str(llm_run_id)) if llm_run_id else None - ) + llm_ctx = self._active_runs.get(str(llm_run_id)) if llm_run_id else None if llm_ctx is not None: event["trace_id"] = llm_ctx.trace_id event["span_id"] = llm_ctx.span_id @@ -778,8 +816,9 @@ def on_chain_start( logger.debug("on_chain_start without run_id — skipping span emission") return name = _extract_node_name(serialized, "chain") - self._begin_run(str(run_id), str(parent_run_id) if parent_run_id else None, - name, kind="chain") + self._begin_run( + str(run_id), str(parent_run_id) if parent_run_id else None, name, kind="chain" + ) def on_chain_end(self, outputs: Any, *, run_id: Any = None, **kwargs: Any) -> None: self._end_run(run_id) @@ -801,8 +840,9 @@ def on_tool_start( logger.debug("on_tool_start without run_id — skipping span emission") return name = _extract_node_name(serialized, "tool") - self._begin_run(str(run_id), str(parent_run_id) if parent_run_id else None, - name, kind="tool") + self._begin_run( + str(run_id), str(parent_run_id) if parent_run_id else None, name, kind="tool" + ) def on_tool_end(self, output: Any, *, run_id: Any = None, **kwargs: Any) -> None: self._end_run(run_id) @@ -822,8 +862,12 @@ def on_agent_action( if run_id is None: return tool = getattr(action, "tool", None) or "agent" - self._begin_run(str(run_id), str(parent_run_id) if parent_run_id else None, - f"agent_action:{tool}", kind="agent") + self._begin_run( + str(run_id), + str(parent_run_id) if parent_run_id else None, + f"agent_action:{tool}", + kind="agent", + ) def on_agent_finish(self, finish: Any, *, run_id: Any = None, **kwargs: Any) -> None: self._end_run(run_id) @@ -943,6 +987,7 @@ def _extract_node_name(serialized: Any, default: str) -> str: # uses, so we have a single pattern for "best-effort read from the # response object" across both helpers. + def _extract_model_from_response(response: Any) -> str | None: """Best-effort model extraction mirroring ``_get_finish_reason``. @@ -1003,12 +1048,7 @@ def _extract_model_from_response(response: Any) -> str | None: # less canonical keys (``"model_id"``, ``"modelName"`` # ``"resolved_model"``). for key, val in llm_out.items(): - if ( - isinstance(key, str) - and "model" in key.lower() - and isinstance(val, str) - and val - ): + if isinstance(key, str) and "model" in key.lower() and isinstance(val, str) and val: return val # 2. response_metadata on the response (langchain 0.x AIMessage @@ -1133,4 +1173,3 @@ def _extract_provider_from_response(response: Any) -> str | None: return str(val) return None - diff --git a/src/nullrun/messages.py b/src/nullrun/messages.py index 7a3fad1..4da2b5e 100644 --- a/src/nullrun/messages.py +++ b/src/nullrun/messages.py @@ -89,6 +89,16 @@ "NR-C000": "There's a configuration issue. Please contact support.", "NR-C001": "There's a configuration issue. Please contact support.", "NR-C004": "There's a configuration issue. Please contact support.", + # ---- Integration errors (programmer misuse, expected to be caught) ---- + # NR-EX01: /execute (or /cancel) was called without a prior /gate + # that minted this execution_id, or the binding TTL expired. This + # is a programmer-facing flow — the host code is responsible for + # re-issuing /api/v1/gate. The user-facing message is a polite + # catch-all that signals "this should not normally reach the end + # user" without leaking wire-shape details. Wording mirrors the + # configuration-issue cluster above; end users who ever see this + # are downstream of a host-code bug. + "NR-EX01": "There's a configuration issue. Please contact support.", # ---- Base --------------------------------------------------------------- "NR-0000": "Something went wrong. Please try again.", } diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py index dc716af..f0d2e93 100644 --- a/src/nullrun/runtime.py +++ b/src/nullrun/runtime.py @@ -83,12 +83,18 @@ ) from nullrun.breaker.exceptions import ( BreakerError, + NullRunApprovalDeniedError, + NullRunApprovalExpiredError, + NullRunApprovalReplayRejectedError, + NullRunApprovalResponseMissingError, NullRunAuthenticationError, NullRunBackendError, NullRunBlockedException, + NullRunBudgetError, NullRunError, NullRunInfrastructureError, NullRunTransportError, + NullRunWorkflowKilledError, WorkflowKilledInterrupt, WorkflowPausedException, ) @@ -176,9 +182,7 @@ def _is_production_environment(api_url: str | None = None) -> bool: """ from urllib.parse import urlparse - effective_url = api_url or os.getenv( - "NULLRUN_API_URL", "https://api.nullrun.io" - ) + effective_url = api_url or os.getenv("NULLRUN_API_URL", "https://api.nullrun.io") # Strip any trailing slash before parsing for consistent # ``hostname`` extraction. effective_url = effective_url.rstrip("/") @@ -647,7 +651,7 @@ def __init__( self._debug = debug self._transport: Transport | None = None -# Local enforcement state + # Local enforcement state # The BoundedDict-based per-workflow cost / loop / retry # counters have been removed alongside ``_check_local_limits``. # As of 0.7.0 ALL local enforcement (LoopTracker / RateTracker @@ -693,7 +697,8 @@ def __init__( # - the WS push arrives with outcome="approved" (release # the gate, resume from the same execution_id), or # - the WS push arrives with outcome="denied" (surface - # WorkflowKilledInterrupt), or + # NullRunApprovalDeniedError / NullRunWorkflowKilledError), + # or # - the per-approval timeout elapses (fall back to the # /status poll path; emit a warning so the operator # knows WS push is silent). @@ -1524,9 +1529,10 @@ def _fetch_remote_state(self, workflow_id: str) -> None: def _handle_approval_resolved(self, payload: dict[str, Any]) -> None: """WS push handler for an approval resolution. Releases - the matching gate reservation (approved) or raises - WorkflowKilledInterrupt (denied) so the agent can resume - from the same execution_id. + the matching gate reservation (approved) or surfaces the + typed denial/timeout (NullRunApprovalDeniedError on denied, + NullRunApprovalExpiredError on timeout, NR-A011/NR-A012) + so the agent can resume from the same execution_id. Args: payload: The WsMessage::ApprovalResolved dict from the @@ -1593,16 +1599,20 @@ def _wait_for_approval_resolution( a sentinel ``{"outcome": "timeout", "timed_out": True}``. **The caller is expected to fail-CLOSED on timeout** — - raise ``WorkflowKilledInterrupt``. The contract - deliberately rejects a `/status` poll fallback here: - a silent timeout must not silently approve a - privileged action. + raise :class:`NullRunApprovalExpiredError` (typed + exception, NR-A012). The contract deliberately rejects + a `/status` poll fallback here: a silent timeout must + not silently approve a privileged action. Raises: Nothing. Approval timeouts are returned, not raised, so the caller can choose the right recovery action - (raise WorkflowKilledInterrupt on denied OR on - timeout, resume on approved). + (raise NullRunApprovalDeniedError on denied, raise + NullRunApprovalExpiredError on timeout, resume on + approved). 2026-09-08: typed approval exceptions + (NR-A011, NR-A012) replaced the generic + ``WorkflowKilledInterrupt`` here so cookbook code can + catch by wire-code. """ # Per-approval timeout resolution: prefer the # server-authoritative value from the /gate response so @@ -1705,7 +1715,10 @@ def check_control_plane(self, workflow_id: str) -> None: Raises: WorkflowPausedException: If workflow is paused on server - WorkflowKilledInterrupt: If workflow is killed on server + NullRunWorkflowKilledError: If workflow is killed on + server (2026-09-08 typed signal, NR-W002; subclass + of WorkflowKilledInterrupt which remains as the + back-compat name.) """ # Prefer the explicit arg (contextvar-supplied), fall back # to the API key's bound workflow. None on legacy keys -- @@ -1744,9 +1757,14 @@ def check_control_plane(self, workflow_id: str) -> None: ) elif state_normalized == "killed": reason = remote_state.get("reason", "remote kill") - raise WorkflowKilledInterrupt( + # 2026-09-08: typed kill signal (NR-W002). Cookbook code + # can `except NullRunWorkflowKilledError` to surface the + # structured error_code + user_action. Legacy + # `except WorkflowKilledInterrupt` still matches (subclass). + raise NullRunWorkflowKilledError( workflow_id=workflow_id, reason=reason, + kill_source="remote_state", ) def check_workflow_budget(self) -> None: @@ -1756,7 +1774,8 @@ def check_workflow_budget(self) -> None: budget never gets to spend tokens. Decision → exception mapping: - "block" → WorkflowKilledInterrupt (hard policy / reservation error) + "block" → NullRunBudgetError (NR-B004, hard policy / + reservation error; 2026-09-08 typed signal) "throttle"→ WorkflowPausedException (insufficient budget, can resume) "allow" → return @@ -1798,12 +1817,7 @@ def check_workflow_budget(self) -> None: # pre-v3.53 the SDK silently honored it in any env # which made accidental prod misuse a silent fail-OPEN. if _is_production_environment(self.api_url): - allow_ack = ( - os.environ.get( - "NULLRUN_ALLOW_SKIP_BUDGET_CHECK", "" - ).strip() - == "1" - ) + allow_ack = os.environ.get("NULLRUN_ALLOW_SKIP_BUDGET_CHECK", "").strip() == "1" if not allow_ack: logger.error( "check_workflow_budget: NULLRUN_SKIP_BUDGET_CHECK=1 " @@ -1838,9 +1852,7 @@ def check_workflow_budget(self) -> None: except Exception: # noqa: BLE001 pass return - logger.debug( - "check_workflow_budget: skipped via NULLRUN_SKIP_BUDGET_CHECK=1" - ) + logger.debug("check_workflow_budget: skipped via NULLRUN_SKIP_BUDGET_CHECK=1") return # Bump the ``check_calls`` counter so the dashboard can show @@ -1963,9 +1975,7 @@ def check_workflow_budget(self) -> None: # `transport.py::execute`) and do not pass through this # pre-flight gate. Computing once per call (not cached) is # fine: compute_action_digest is ~5µs of pure stdlib. - check_req["action_digest"] = _compute_action_digest( - _BusinessImpact.no_impact() - ) + check_req["action_digest"] = _compute_action_digest(_BusinessImpact.no_impact()) # Forward the tool list so backend (T3) can match each tool # against the workflow's effective `blocked_tools` aggregate. @@ -2087,9 +2097,17 @@ def check_workflow_budget(self) -> None: # distinct from loop / retry / rate which have their # own counters. metrics.inc_runtime("cost_limit_exceeded") - raise WorkflowKilledInterrupt( + # 2026-09-08: typed hard-block (NR-B004 budget cap). + # ``NullRunBudgetError`` carries structured + # ``error_code``, ``user_action``, ``retryable`` so the + # LLM gets an actionable hint instead of "Something went + # wrong". ``reasons`` preserved in details for telemetry. + raise NullRunBudgetError( workflow_id=workflow_id, reason="; ".join(reasons), + action="block", + decision_source=response.get("decision_source"), + reasons="; ".join(reasons), ) if decision == "throttle": reasons = response.get("explanations") or ( @@ -2128,8 +2146,7 @@ def check_workflow_budget(self) -> None: # alongside "hard cap hits" via the same dashboard panel. metrics.inc_runtime("soft_overdraft_used") logger.warning( - "check_workflow_budget: soft_pass -- %s " - "(overdraft_used=%s, max=%s, remaining=%s)", + "check_workflow_budget: soft_pass -- %s (overdraft_used=%s, max=%s, remaining=%s)", explanation, overdraft_used, max_overdraft, @@ -2161,9 +2178,16 @@ def check_workflow_budget(self) -> None: logger.warning( "check_workflow_budget: require_approval decision but no approval_id in response" ) - raise WorkflowKilledInterrupt( + # 2026-09-08: typed backend error (NR-B002, retryable). + # The server returned require_approval without an + # approval_id -- this is a wire-bug / drift, not a + # budget block. Surface as retryable backend error + # so cookbook code can decide whether to fall back + # to polling /status. + raise NullRunBackendError( + message="approval_id missing in require_approval response", + endpoint="/api/v1/gate", workflow_id=workflow_id, - reason="approval_id missing in require_approval response", ) # Read the per-approval timeout from the response. Both # `approval_timeout_seconds` (i64) and @@ -2198,17 +2222,28 @@ def check_workflow_budget(self) -> None: logger.info(f"check_workflow_budget: approval {approval_id} approved -- resuming") return if outcome == "denied": - raise WorkflowKilledInterrupt( + # 2026-09-08: typed approval-denied (NR-A011). Cookbook + # code can `except NullRunApprovalDeniedError` to + # surface the denial note + user_action to the LLM. + raise NullRunApprovalDeniedError( workflow_id=workflow_id, reason=f"approval denied: {result.get('note') or 'operator denied'}", + approval_id=approval_id, + denial_note=result.get("note"), ) # timeout: fail-CLOSED -- do not run the call. - raise WorkflowKilledInterrupt( + # 2026-09-08: typed approval-expired (NR-A012) -- THE TRIGGER + # FIX. The LLM now sees "Approval expired after 300s of + # WS push silence" instead of "Something went wrong". + raise NullRunApprovalExpiredError( workflow_id=workflow_id, reason=( f"approval {approval_id} timeout: WS push silent for " f"{self._approval_timeout_seconds:.0f}s" ), + approval_id=approval_id, + timeout_seconds=self._approval_timeout_seconds, + local_timeout=True, ) # ============================================================================= @@ -2729,7 +2764,14 @@ def execute( - decision: "allow" | "block" | "flag" | "pause" | "require_approval" - decision_source: "gateway" | "cached" | "fallback" - explanation: Human-readable explanation - - policy_version: Policy version used + - policy_hash: Server-side SHA-256 of the policy applied + to this gate decision (v4 wire field; null on pre-v4 + backends). Captured via `_capture_wire_evidence` → + `set_last_gate_policy_hash` for downstream audit linkage. + NOTE: this is NOT a sequential `policy_version` number — + wire v3/v4 backends emit only `policy_hash`; legacy + `policy_version` references in this SDK are no longer + populated from the wire. - decision_context: Context used for the decision Mode values: @@ -2772,7 +2814,7 @@ def execute( "decision": "allow", "decision_source": DecisionSource.LOCAL, "explanation": "Inline mode: local enforcement only", - "policy_version": 0, + "policy_hash": None, "allow_execution": True, } @@ -2820,10 +2862,46 @@ def execute( # decorator call site). if tools is None: from nullrun.context import get_call_tools as _get_call_tools_for_execute + tools = _get_call_tools_for_execute() + + # DEFS-SDKEXEC-GATE-FIRST (2026-09-08): hoist the execution_id + # mint to REUSE the server-minted id from a prior /gate call + # (set via `_capture_server_minted_execution_id` from the + # `reservation_id` field of the /gate response). + # + # Why this matters: backend `/api/v1/execute` (the wire + # contract enforced by `backend/src/proxy/http/gate/execute.rs` + # since DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) + # runs an existence check on `execution:{id}` in Redis at + # `execute.rs:180-208` and returns 404 EXECUTION_NOT_FOUND when + # no prior /gate minted the binding. Pre-fix this method minted + # a fresh `uuid7_str()` here — the freshly-minted id was never + # registered by /gate, so /execute fail-CLOSED with 404 on + # EVERY call and the SDK translated the 404 into a synthetic + # block ("Gateway returned 404") in + # ``transport.py::execute`` (line ~1195). + # + # Resolution: when a prior /gate captured a server-minted + # execution_id into ``_server_minted_execution_id_var``, reuse + # it. The decorator-driven ``@protect @sensitive`` path always + # runs ``check_workflow_budget()`` BEFORE ``runtime.execute()`` + # (decorators.py:538 vs :824), so the contextvar is populated + # in the common path. Direct callers of ``runtime.execute()`` + # without a prior /gate will fall through to the fresh-mint + # branch below — that's a wire-contract violation and the + # backend's 404 is the correct fail-CLOSED response. + from nullrun.context import get_server_minted_execution_id + + prior_execution_id = get_server_minted_execution_id() + if prior_execution_id is not None: + execution_id = prior_execution_id + else: + execution_id = uuid7_str() + execute_kwargs: dict[str, Any] = { "organization_id": organization_id, - "execution_id": uuid7_str(), + "execution_id": execution_id, "trace_id": trace_id, "tool": tool_name, "input_data": input_data, @@ -2854,11 +2932,15 @@ def execute( approval_id = result.get("approval_id") or "" if not approval_id: metrics.inc_runtime("execute_blocked") - raise NullRunBlockedException( + # 2026-09-08: typed wire-bug (NR-A004). The server + # returned require_approval without an approval_id — + # this is a wire-contract bug, NOT a transient failure. + # Cookbook code catches this and reports to NULLRUN + # support; do NOT retry. + raise NullRunApprovalResponseMissingError( workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, reason="approval_id missing in require_approval response", tool_name=tool_name, - error_code="NR-A004", ) server_timeout = _validate_approval_timeout( @@ -2875,16 +2957,38 @@ def execute( outcome = str(approval_result.get("outcome") or "").lower() if outcome != "approved": metrics.inc_runtime("execute_blocked") - reason = ( - f"approval denied: {approval_result.get('note') or 'operator denied'}" - if outcome == "denied" - else f"approval {approval_id} timeout" - ) + # 2026-09-08: dispatch typed approval exception by + # outcome so cookbook code can react per wire-code: + # denied → NR-A011 (NullRunApprovalDeniedError) + # timeout → NR-A012 (NullRunApprovalExpiredError) + # other → NR-X001 (NullRunBlockedException, generic) + if outcome == "denied": + raise NullRunApprovalDeniedError( + workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, + reason=( + f"approval denied: " + f"{approval_result.get('note') or 'operator denied'}" + ), + tool_name=tool_name, + approval_id=approval_id, + denial_note=approval_result.get("note"), + ) + if outcome == "timeout": + raise NullRunApprovalExpiredError( + workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, + reason=f"approval {approval_id} timeout", + tool_name=tool_name, + approval_id=approval_id, + timeout_seconds=server_timeout, + local_timeout=True, + ) + # Unknown outcome (cancelled / superseded / wire drift): + # fall back to generic block so the LLM still sees a + # typed exception with error_code, never a bare string. raise NullRunBlockedException( workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, - reason=reason, + reason=f"approval {approval_id} unresolved: outcome={outcome!r}", tool_name=tool_name, - error_code="NR-A004", ) # Re-check the same action. The backend must verify that @@ -2895,11 +2999,17 @@ def execute( result = self._transport.execute(**execute_kwargs) if result.get("decision") == "require_approval": metrics.inc_runtime("execute_blocked") - raise NullRunBlockedException( + # 2026-09-08: typed replay-rejection (NR-A015). The + # operator approved but the same approval_id was + # already consumed by a concurrent /execute (race). + # Cookbook pattern: do NOT retry the same approval_id; + # treat as idempotency violation (likely a client + # retry loop). + raise NullRunApprovalReplayRejectedError( workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, reason="approved action was not accepted on re-check", tool_name=tool_name, - error_code="NR-A004", + approval_id=approval_id, ) # Check if execution is allowed @@ -3066,9 +3176,7 @@ def _enrich_event(self, event: dict[str, Any]) -> dict[str, Any]: # 2026-08-06 (DEF-SDKWRAP-CHAIN-SOFT-EXECUTION-ID-REUSE-01, span_id = enriched.get("span_id") if span_id and ":" not in idem_key: - enriched["idempotency_key"] = ( - f"{idem_key}:{str(span_id)[:16]}" - ) + enriched["idempotency_key"] = f"{idem_key}:{str(span_id)[:16]}" else: enriched["idempotency_key"] = idem_key @@ -3516,11 +3624,7 @@ def _capture_server_minted_execution_id(response: dict[str, Any]) -> str | None: from nullrun.context import get_operation_id as _get_op_id_for_capture sdk_op_id = _get_op_id_for_capture() - server_op_id = ( - response.get("operation_id") - if isinstance(response, dict) - else None - ) + server_op_id = response.get("operation_id") if isinstance(response, dict) else None if isinstance(server_op_id, str) and server_op_id: # Defensive parity assertion: server MUST echo the same # operation_id the SDK sent. A mismatch indicates either @@ -3608,8 +3712,7 @@ def _safe_str(key: str) -> str | None: return None if not isinstance(v, str): logger.warning( - "_capture_wire_evidence: response.%s is %s, " - "expected str — dropping", + "_capture_wire_evidence: response.%s is %s, expected str — dropping", key, type(v).__name__, ) diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index f9f3b28..62d73c8 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -29,6 +29,7 @@ BreakerTransportError, InsecureTransportError, NullRunAuthenticationError, + NullRunExecutionNotFoundError, NullRunTransportError, RateLimitError, TransportErrorSource, @@ -300,7 +301,12 @@ def _retry_with_backoff( return result - except (BreakerTransportError, NullRunAuthenticationError, NullRunTransportError, NullRunBackendError): + except ( + BreakerTransportError, + NullRunAuthenticationError, + NullRunTransportError, + NullRunBackendError, + ): raise except httpx.HTTPStatusError as exc: @@ -1119,9 +1125,27 @@ def execute( ) -> dict[str, Any]: """Pre-execution policy evaluation via /api/v1/execute (PRIMARY enforcement point). - The SDK MUST call /api/v1/execute (which checks the ``execute`` scope on - the API key) rather than /api/v1/gate (advisory, no scope check). - /api/v1/gate is reserved for budget pre-flight via ``Transport.check``. + Wire contract (revised 2026-09-08, DEFS-SDKEXEC-GATE-FIRST): + /execute REQUIRES a prior /gate call that minted the same + ``execution_id`` and registered the ``execution:{id}`` binding + in Redis. Backend enforcement: + ``backend/src/proxy/http/gate/execute.rs:46-208`` + (DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) + runs ``HGET execution:{id} ORG_FIELD`` on entry; a miss + returns 404 EXECUTION_NOT_FOUND (fail-CLOSED). The SDK + therefore MUST thread the execution_id captured by + ``runtime.check_workflow_budget`` (which calls ``Transport.check``, + i.e. /gate) into the body of this /execute call. See + ``runtime.execute()`` (line ~2820) for the reuse path; this + method's caller is the single source of truth for + ``execution_id`` selection. + + Prior to DEF-SDKK-022 the comment here claimed "/execute MUST + be called rather than /gate" — that contract was the legacy + pre-2026-09-04 shape. The post-fix shape is "/execute MUST be + preceded by /gate for the same execution_id" — the budget + pre-flight (Transport.check, /api/v1/gate) is the binding + registrar; /execute is the policy decision that re-uses it. Args: organization_id: Organization identifier @@ -1143,7 +1167,12 @@ def execute( - decision: "allow" | "block" | "flag" | "pause" | "require_approval" - decision_source: "gateway" | "cached" | "fallback" - explanation: Human-readable explanation - - policy_version: Policy version used + - policy_hash: Server-side SHA-256 of the policy applied + (v4 wire field; null on pre-v4 backends). NOT a + sequential `policy_version` number — wire v3/v4 backends + emit only `policy_hash`. Synthetic fallback dicts ship + `policy_version: 0` for legacy compatibility; real + responses populate `policy_hash` only. - decision_context: Context for replay (if available) """ gate_request = { @@ -1198,7 +1227,7 @@ def do_execute_request() -> httpx.Response: "decision": "block", "decision_source": DecisionSource.FALLBACK, "explanation": f"Gateway returned {response.status_code}", - "policy_version": 0, + "policy_hash": None, } except BreakerTransportError as exc: @@ -1216,14 +1245,14 @@ def do_execute_request() -> httpx.Response: "decision": "allow", "decision_source": TransportErrorSource.NETWORK_ERROR, "explanation": f"Gateway unreachable: {exc}", - "policy_version": 0, + "policy_hash": None, } if on_transport_error == "closed": return { "decision": "block", "decision_source": TransportErrorSource.NETWORK_ERROR, "explanation": f"Gateway unreachable: {exc}", - "policy_version": 0, + "policy_hash": None, } pass # fall through to fallback mode except NullRunTransportError: @@ -1693,8 +1722,17 @@ def track_single( ``idempotency_key``. Returns: - Parsed JSON dict with at least - ``{"status": "ok"|"idempotent_replay",...}``. + Parsed JSON dict from the backend's TrackResponse. + NOTE: there is NO top-level ``status`` field on the + wire — the legacy pre-v3 docstring claimed one, but + v3/v4 backends emit + ``{snapshot, actions_taken, processing_mode, + cost_source, confidence, event_id, + idempotent_replay, stored_response?}``. SDK callers + branch on the HTTP status (200 vs 4xx/5xx) and on + ``idempotent_replay`` (bool) for replay detection — + do NOT read ``data["status"]`` (KeyError on every + backend >= 3.66.2). Raises: NullRunConsumeOverbudgetError: 422 CONSUME_OVERBUDGET — @@ -1760,8 +1798,14 @@ def cancel( cancellation (audit trail). Returns: - Parsed JSON dict (typically ``{"status": "ok" - "execution_id":..., "cancelled_at": ts}``). + Parsed JSON dict from the backend's CancelResponse. + NOTE: there is NO top-level ``status`` field on the + wire — the legacy pre-v3 docstring claimed one. + v3/v4 backends emit + ``{execution_id, canceled_at, reservation_released_cents, + already_canceled}``. SDK callers branch on the HTTP + status only — do NOT read ``data["status"]`` + (KeyError on every backend >= 3.66.2). """ request: dict[str, Any] = {"execution_id": execution_id} if reason: @@ -2151,10 +2195,7 @@ def audit_export_status( Raises: NullRunBackendError / NullRunAuthenticationError. """ - url = ( - f"{self.api_url}/api/v1/orgs/{organization_id}" - f"/audit-log/export/{job_id}/status" - ) + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export/{job_id}/status" headers = self._auth_headers_for_get() try: response = self._client.get(url, headers=headers, timeout=10.0) @@ -2515,6 +2556,22 @@ def _parse_v3_error_envelope( endpoint=endpoint, status_code=status, ) + if catalog is NullRunExecutionNotFoundError: + # 2026-09-09 audit: dedicated dispatch so callers can + # read ``execution_id`` / ``endpoint`` / ``regate_required`` + # off the exception without indexing into ``details``. + # Mirrors the ``NullRunBackendError`` branch above (the + # parent class) but also forwards ``execution_id`` from + # the wire envelope. Without this branch the generic + # catalog fallback at line ~2615 would discard the + # ``execution_id`` field (it filters ``**details`` to + # the base NullRunError kwargs only). + return NullRunExecutionNotFoundError( + full_message, + execution_id=details.get("execution_id"), + endpoint=details.get("endpoint") or endpoint, + status_code=status, # 404 per backend mapping + ) if catalog is NullRunBudgetError: # NullRunBudgetError → NullRunBlockedException → requires return NullRunBudgetError( @@ -2627,8 +2684,10 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: NullRunBackendError, NullRunBlockedException, NullRunBudgetError, + NullRunBudgetRecheckFailedError, NullRunChainError, NullRunConsumeOverbudgetError, + NullRunExecutionNotFoundError, NullRunProtocolError, NullRunRateLimitRedisError, NullRunToolBlockedError, @@ -2715,7 +2774,12 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: # refresh the reservation envelope and retry /execute. # Backed by GateErrorCode::BudgetRecheckFailed in the # backend (error_codes.rs). - "BUDGET_RECHECK_FAILED": NullRunBudgetError, + # 2026-09-09 audit: the per-class dispatcher in + # ``_v3_error_dispatch`` (line ~2477) already routes this to + # ``NullRunBudgetRecheckFailedError`` (NR-B006) before the + # catalog fallback — defense-in-depth, this catalog entry + # now matches the dispatcher. + "BUDGET_RECHECK_FAILED": NullRunBudgetRecheckFailedError, # NR-007 (audit 2026-08-24): the 19 entries below were missing # from the SDK map and caused cookbook recipes that branch on # ``error_code`` to fall through to ``NullRunBackendError``. @@ -2756,6 +2820,25 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: # here indicates a wire-shape drift between client and server. "EXECUTION_ID_MALFORMED": NullRunBackendError, "EXECUTION_ID_REQUIRED": NullRunBackendError, + # 2026-09-09 SDK-drift audit: ``INVALID_EXECUTION_ID`` is + # emitted by the backend as a typed envelope at + # ``cancel.rs:142-149`` and ``orchestrator.rs:1327-1334`` — + # round-trips through the canonical ``v3_error_envelope`` + # helper, so the wire string is canonical. Map to + # ``NullRunBackendError`` (sibling to the EXECUTION_ID_* + # siblings above) — wire-shape drift guard. + "INVALID_EXECUTION_ID": NullRunBackendError, + # 2026-09-09 SDK-drift audit: ``EXECUTION_NOT_FOUND`` is + # emitted by the backend as a typed envelope at + # ``execute.rs:194`` and ``cancel.rs:303`` (post-DEF-SDKK-022 + # routing through ``v3_error_envelope`` + the new + # ``GateErrorCode::ExecutionNotFound`` variant). Map to the + # dedicated ``NullRunExecutionNotFoundError`` (NR-EX01) so + # cookbook code can ``except + # NullRunExecutionNotFoundError`` to distinguish a missed + # /gate (re-issue /gate then retry /execute) from generic + # wire-shape drift. + "EXECUTION_NOT_FOUND": NullRunExecutionNotFoundError, # Rate-limit plan lookup failure (Postgres / Redis adjacent). # Tied to ``NullRunRateLimitRedisError`` because the failure # mode is rate-limit-specific infrastructure unavailability @@ -2767,6 +2850,35 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: # ``RATE_LIMIT_REDIS_UNAVAILABLE`` -> ``NullRunRateLimitRedisError`` # family pattern at wire level). "IDEMPOTENCY_REDIS_UNAVAILABLE": NullRunBackendError, + # Execution Graph / ADR-036 (sub-agent spawn topology). Backend + # error_codes.rs:107-382 covers six codes in this family — three + # 422 semantic rejects (cycle / depth / parent-binding) and three + # 503 infrastructure failures (depth lookup / invoke persist / + # subworkflow disabled). Map to ``NullRunChainError`` because + # the existing class already carries `parent_execution_id` per + # Execution Graph v0 docstring at `exceptions.py:388-410`. Adding + # them under a fresh ``NullRunSubworkflowError`` would force + # cookbook code to import a new exception class for the same + # lineage concept; consolidate under ChainError instead. + "WORKFLOW_CYCLE_DETECTED": NullRunChainError, + "WORKFLOW_DEPTH_EXCEEDED": NullRunChainError, + "WORKFLOW_PARENT_BINDING_EXPIRED": NullRunChainError, + "WORKFLOW_DEPTH_LOOKUP_FAILED": NullRunChainError, + "INVOKE_PERSIST_FAILED": NullRunBackendError, + "SUBWORKFLOW_INVOKE_DISABLED": NullRunChainError, + # ADR-023 (post-approval re-check race): a second operator + # already decided on the same approval row before this call's + # re-check landed. Map to ``NullRunApprovalReplayRejectedError`` + # because semantically the agent caller has the same retry-loop + # concern as a replay-rejected approval (CLAUDE.md §34c). + "APPROVAL_ALREADY_DECIDED": NullRunApprovalReplayRejectedError, + # ADR-023 (Phase-1+ wire-shape fail-CLOSED): a v3+ SDK hit /gate + # without ``action_digest`` (legacy anchor attempt). Map to + # ``NullRunBlockedException`` because the wire shape is a true + # block decision, not an infrastructure error — cookbook code + # branches on the action_digest missing path with the same + # `except NullRunBlockedException:` flow as TOOL_BLOCKED. + "LEGACY_GRANT_REJECTED": NullRunBlockedException, } diff --git a/tests/test_2026_09_08_gate_first_execute.py b/tests/test_2026_09_08_gate_first_execute.py new file mode 100644 index 0000000..18dcdd3 --- /dev/null +++ b/tests/test_2026_09_08_gate_first_execute.py @@ -0,0 +1,296 @@ +"""DEFS-SDKEXEC-GATE-FIRST (2026-09-08) — /execute must reuse /gate's execution_id. + +Pre-fix (per audit 2026-09-08): + - `runtime.execute()` minted a fresh `uuid7_str()` for the wire body. + - Backend `/api/v1/execute` (`backend/src/proxy/http/gate/execute.rs:46-208`, + DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) requires the + request's `execution_id` to have a live `execution:{id}` ownership + binding in Redis (HGET ORG_FIELD). Without a prior /gate that registered + the binding, /execute returned 404 EXECUTION_NOT_FOUND and the SDK + translated the 404 into a synthetic block ("Gateway returned 404"). + - User-visible symptom: every `@protect @sensitive` call from + `langgraph_openai_approval_demo.py` (and similar flows) returned + `Workflow __nullrun_unknown__ blocked: Gateway returned 404 (action=block, + tool=, status_code=None, details=)`. + +Post-fix: + - `runtime.execute()` reads `_server_minted_execution_id_var` (set by + `_capture_server_minted_execution_id` from the /gate response's + `reservation_id` field) and reuses it. Only mint a fresh uuid7 when + the contextvar is empty (direct callers without a prior /gate). + - `_enforce_sensitive_tool` displays the API key's bound workflow + (resolved via `runtime._resolve_workflow_id`) instead of the literal + `__nullrun_unknown__` sentinel when the user did not open an explicit + `with workflow(...)` block. The wire still carries the same workflow + (server-side binding); only the displayed label changes. + +These tests pin the post-fix shape so a future refactor that re-introduces +a fresh-mint in `execute()` (or restores the sentinel-first display) +fails the test. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +from nullrun.context import ( + clear_server_minted_execution_id, + set_server_minted_execution_id, +) + +SDK_ROOT = Path(__file__).resolve().parent.parent +RUNTIME_PY = SDK_ROOT / "src" / "nullrun" / "runtime.py" +DECORATORS_PY = SDK_ROOT / "src" / "nullrun" / "decorators.py" +TRANSPORT_PY = SDK_ROOT / "src" / "nullrun" / "transport.py" +LANGGRAPH_INSTR_PY = SDK_ROOT / "src" / "nullrun" / "instrumentation" / "langgraph.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +@pytest.fixture(autouse=True) +def _reset_server_minted(): + """Reset the contextvar before AND after each test so leakage + between tests doesn't masquerade as a hoist pass.""" + clear_server_minted_execution_id() + yield + clear_server_minted_execution_id() + + +class TestExecuteReusesGateExecutionId: + """Pin `runtime.execute()` so a future refactor that re-mints + a fresh `uuid7_str()` regardless of /gate context fails the test.""" + + def _execute_body(self) -> str: + runtime = _read(RUNTIME_PY) + # Match the second `def execute(` (the public enforcement + # entry point), not `runtime._execute` or `Transport.execute`. + m = re.search( + r" def execute\(\s*self,\s*tool_name: str,.*?\)\s*->\s*" + r"dict\[str, Any\]:.*?(?=\n def |\nclass |\Z)", + runtime, + re.DOTALL, + ) + assert m, "could not locate runtime.execute method body" + return m.group(0) + + def test_execute_reads_server_minted_contextvar(self): + body = self._execute_body() + assert "get_server_minted_execution_id()" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: runtime.execute() must read the " + "server-minted execution_id from the contextvar (set by " + "check_workflow_budget's /gate round-trip) before minting " + "a fresh uuid7. Pre-fix the body unconditionally minted " + "uuid7_str(), so /execute's execution_id never matched " + "the binding /gate registered and the backend returned " + "404 EXECUTION_NOT_FOUND." + ) + + def test_execute_reuses_captured_id_when_present(self): + body = self._execute_body() + # The hoist pattern: read contextvar, fall back to uuid7_str() + # only when the contextvar is None. + assert "prior_execution_id = get_server_minted_execution_id()" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: runtime.execute() must alias the " + "contextvar read into a local so the same value flows into " + "the wire body." + ) + assert "if prior_execution_id is not None:" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: when the contextvar is populated, " + "execute() must reuse it directly — no fresh uuid7 mint." + ) + assert "execution_id = prior_execution_id" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the reused execution_id must be " + "threaded into the wire body under the `execution_id` key." + ) + + def test_execute_falls_back_to_uuid7_only_when_contextvar_empty(self): + body = self._execute_body() + # Locate the fallback block — must live INSIDE an `if ... is None:` arm. + fallback_block = re.search( + r"if prior_execution_id is not None:\s*\n\s*execution_id = " + r"prior_execution_id\s*\n\s*else:\s*\n\s*execution_id = " + r"uuid7_str\(\)", + body, + ) + assert fallback_block, ( + "DEFS-SDKEXEC-GATE-FIRST: the uuid7_str() mint must live " + "INSIDE the `else:` arm of the `if prior_execution_id is " + "not None:` check. Pre-fix an unconditional " + "`execution_id = uuid7_str()` line at this site minted " + "every time, breaking the /gate ↔ /execute binding." + ) + body_without_fallback = body.replace(fallback_block.group(0), "") + # Defensive: the wire body MUST consume the resolved + # `execution_id` (the one with the prior_id fallback applied). + assert '"execution_id": execution_id' in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the wire body must consume the " + "resolved `execution_id` variable (not a freshly-minted " + "uuid7 inline)." + ) + assert 'execution_id": uuid7_str()' not in body_without_fallback, ( + "DEFS-SDKEXEC-GATE-FIRST: a top-level `execution_id = " + "uuid7_str()` (outside the fallback arm) must not survive. " + "A pre-fix leftover would silently bypass the /gate reuse." + ) + + def test_execute_carries_comment_explaining_drift(self): + body = self._execute_body() + # The fix introduced a long comment naming DEF-SDKK-022 + + # DEFS-SDKEXEC-GATE-FIRST. Pin so a future maintainer who + # deletes the comment is forced to read the code's history. + assert "DEFS-SDKEXEC-GATE-FIRST" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the explainer comment block must " + "name the fix tag so future readers can grep for it." + ) + assert "DEF-SDKK-022-EXEC-BYPASS" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the explainer must reference the " + "backend fix (DEF-SDKK-022-EXEC-BYPASS) that introduced the " + "/execute existence check, so readers see the round-trip " + "contract without searching." + ) + + +class TestDecoratorWorkflowLabelUsesRuntimeBinding: + """Pin `_enforce_sensitive_tool` so the displayed workflow_id + label shows the API key's bound workflow when no `with workflow(...)` + block is active (instead of the literal `__nullrun_unknown__` sentinel).""" + + def _enforce_body(self) -> str: + decorators = _read(DECORATORS_PY) + m = re.search( + r"def _enforce_sensitive_tool\(.*?\).*?(?=\ndef |\nclass |\Z)", + decorators, + re.DOTALL, + ) + assert m, "could not locate _enforce_sensitive_tool method body" + return m.group(0) + + def test_enforce_resolves_via_runtime_bound_workflow(self): + body = self._enforce_body() + # Two sites in the function (extract failure path + main path). + occurrences = body.count( + "runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID" + ) + assert occurrences >= 2, ( + f"DEFS-SDKEXEC-WORKFLOW-LABEL: _enforce_sensitive_tool must " + f"prefer the runtime's bound workflow via " + f"runtime._resolve_workflow_id(...) at both display sites " + f"(extract failure + main path). Found {occurrences} " + f"occurrences; expected >= 2." + ) + + def test_enforce_does_not_use_contextvar_only_fallback(self): + body = self._enforce_body() + # Pre-fix: `workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID` + # (contextvar-only). Post-fix: that literal pattern must not + # survive at the top-level assignment site. + # + # We allow the literal only as a substring INSIDE the longer + # `runtime._resolve_workflow_id(...)` call (which is what we + # want). Strip those out first, then check the residue. + resolved_call = "runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID" + body_without_resolved = body.replace(resolved_call, "") + assert "workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID" not in ( + body_without_resolved + ), ( + "DEFS-SDKEXEC-WORKFLOW-LABEL: pre-fix contextvar-only " + "fallback `workflow_id = get_workflow_id() or " + "UNKNOWN_WORKFLOW_ID` must be replaced by the runtime-aware " + "resolver everywhere. The pre-fix pattern displayed " + "`__nullrun_unknown__` for every API-key-bound key." + ) + + +class TestTransportCommentReflectsPostFixContract: + """Pin the `Transport.execute` docstring so the legacy + pre-2026-09-04 contract (`/execute MUST be called rather than + /gate`) doesn't drift back into the source.""" + + def test_transport_execute_docstring_references_post_fix_contract(self): + transport = _read(TRANSPORT_PY) + m = re.search( + r"def execute\(\s*self,.*?\)\s*->\s*dict\[str, Any\]:.*?(?=\n def |\nclass |\Z)", + transport, + re.DOTALL, + ) + assert m, "could not locate Transport.execute method body" + body = m.group(0) + assert "DEFS-SDKEXEC-GATE-FIRST" in body, ( + "transport.py: Transport.execute docstring must name the " + "post-fix tag so the contract is grep-able." + ) + assert "DEF-SDKK-022-EXEC-BYPASS" in body, ( + "transport.py: Transport.execute docstring must reference " + "the backend fix that introduced the existence check." + ) + # The legacy misleading claim must be gone (or explicitly + # marked as pre-fix). + assert ( + "MUST call /api/v1/execute (which checks the ``execute`` " + "scope on the API key) rather than /api/v1/gate" + ) not in body, ( + "transport.py: pre-fix misleading claim that /execute MUST " + "be called rather than /gate must be removed — that contract " + "was the legacy pre-2026-09-04 shape and was the root " + "cause of the 404 EXECUTION_NOT_FOUND drift." + ) + + +class TestLanggraphCallbackPairsLlmSpanWithReservation: + """Pin `NullRunCallback.on_llm_start` so the LLM span /track + pairing path stays alive (check_workflow_budget is fire-and-forget + but the call site must survive).""" + + def test_on_llm_start_calls_check_workflow_budget(self): + instr = _read(LANGGRAPH_INSTR_PY) + m = re.search( + r"def on_llm_start\(self,.*?\)\s*->\s*None:.*?(?=\n def |\nclass |\Z)", + instr, + re.DOTALL, + ) + assert m, "could not locate NullRunCallback.on_llm_start" + body = m.group(0) + assert "self.runtime.check_workflow_budget()" in body, ( + "DEFS-SDKEXEC-LLM-RESERVATION: on_llm_start must call " + "runtime.check_workflow_budget() to pair the LLM span " + "with a server-minted reservation_id. Without this the " + "matching on_llm_end llm_call cost event is silently " + "dropped by runtime._route_track (no reservation_id in " + "scope)." + ) + assert "DEFS-SDKEXEC-LLM-RESERVATION" in body, ( + "DEFS-SDKEXEC-LLM-RESERVATION: the explainer comment block " + "must name the fix tag so future readers can grep." + ) + # Defensive: the call must be guarded so a backend outage + # never breaks the LangChain callback chain. + assert "except BaseException" in body, ( + "DEFS-SDKEXEC-LLM-RESERVATION: the check_workflow_budget " + "call must be wrapped in a never-raise guard so a " + "WorkflowKilledInterrupt / WorkflowPausedException / " + "transport error does not break the LangChain callback " + "contract (callbacks must never raise)." + ) + + +class TestServerMintedExecutionIdContract: + """Drive the contextvar to confirm the round-trip shape used by + `runtime.execute()` works as advertised.""" + + def test_set_then_get_round_trips(self): + from nullrun.context import ( + get_server_minted_execution_id, + reset_server_minted_execution_id, + ) + + sentinel = "01936f8e-1234-7abc-9def-0123456789ab" + token = set_server_minted_execution_id(sentinel) + try: + assert get_server_minted_execution_id() == sentinel + finally: + reset_server_minted_execution_id(token) diff --git a/tests/test_actions.py b/tests/test_actions.py index 2441f95..396d815 100644 --- a/tests/test_actions.py +++ b/tests/test_actions.py @@ -841,18 +841,42 @@ def test_workflow_killed_interrupt_does_not_emit_warning(): assert not any(issubclass(item.category, DeprecationWarning) for item in w) -def test_workflow_killed_interrupt_is_base_exception(): - """``except Exception`` does NOT catch the kill signal.""" - with pytest.raises(WorkflowKilledInterrupt): - try: - raise WorkflowKilledInterrupt(workflow_id="wf-1", reason="x") - except Exception: - pytest.fail("Exception should not catch WorkflowKilledInterrupt") +def test_workflow_killed_interrupt_is_catchable_by_exception(): + """2026-09-08 migration reversal: ``except Exception`` DOES catch + the kill signal — cookbook code can react with typed error_code + + user_action. The pre-migration contract (BaseException bypass) + is intentionally broken because agent recovery requires + catchable kill signals. + """ + caught: list[Exception] = [] + + try: + raise WorkflowKilledInterrupt(workflow_id="wf-1", reason="x") + except Exception as exc: + caught.append(exc) + assert len(caught) == 1, "Exception should catch WorkflowKilledInterrupt (post-migration)" + assert isinstance(caught[0], WorkflowKilledInterrupt) + assert caught[0].error_code == "NR-W002" -def test_workflow_killed_exception_is_caught_by_except_killed_exception(): - """Legacy ``except WorkflowKilledException`` still catches the new - interrupt (back-compat contract). + +def test_workflow_killed_interrupt_not_caught_by_except_killed_exception(): + """2026-09-08 BREAK: legacy ``except WorkflowKilledException`` + no longer catches the new interrupt (WorkflowKilledInterrupt is + no longer a BaseException subclass). Cookbook code must migrate + to ``except WorkflowKilledInterrupt`` (canonical) or + ``except NullRunWorkflowKilledError`` (preferred typed name). """ - with pytest.raises(WorkflowKilledException): + raised = False + try: raise WorkflowKilledInterrupt(workflow_id="wf-1", reason="x") + except WorkflowKilledException: + pytest.fail( + "except WorkflowKilledException should NOT catch the new " + "interrupt (2026-09-08 BREAK — migrate to except " + "WorkflowKilledInterrupt or except NullRunWorkflowKilledError)" + ) + except WorkflowKilledInterrupt: + raised = True + + assert raised, "the new interrupt should propagate through except WorkflowKilledException" diff --git a/tests/test_decision_split.py b/tests/test_decision_split.py index 11600a7..c6b44a2 100644 --- a/tests/test_decision_split.py +++ b/tests/test_decision_split.py @@ -72,14 +72,30 @@ def test_decision_and_infrastructure_are_disjoint(): def test_workflow_killed_interrupt_is_neither_decision_nor_infrastructure(): - """The kill signal is a BaseException — it deliberately bypasses - ``except Exception:`` so careless handlers can't swallow operator - kills. It must NOT inherit from NullRunDecision (which would make - it catchable by `except Exception:` via the NullRunError branch).""" - assert not issubclass(exc.WorkflowKilledInterrupt, exc.NullRunError) + """2026-09-08 migration reversal: WorkflowKilledInterrupt IS now a + NullRunError subclass (Exception subclass, NR-W002) — formerly + a BaseException subclass that bypassed ``except Exception:``. + The user override: agent recovery needs catchable kill signals + to surface the structured error_code + user_action. + + Hierarchy after migration: + WorkflowKilledInterrupt → NullRunError → BreakerError → Exception → BaseException + + Kill is intentionally NOT a NullRunDecision (the structured- + decision branch is reserved for gate-decision failures: + budget/tool/approval). Kill is a control-plane signal (operator + or circuit-breaker), semantically distinct from a decision — + the new MRO reflects this. + """ + assert issubclass(exc.WorkflowKilledInterrupt, exc.NullRunError) + # Kill is NOT a NullRunDecision (decision failures are budget/ + # tool/approval; kill is a control-plane signal). assert not issubclass(exc.WorkflowKilledInterrupt, exc.NullRunDecision) + # And NOT a NullRunInfrastructureError (operator action is not + # a transport failure). assert not issubclass(exc.WorkflowKilledInterrupt, exc.NullRunInfrastructureError) - # But it IS a BaseException, which is the whole point. + # And it's still a BaseException (transitively, since it is now + # an Exception subclass which is-a BaseException). assert issubclass(exc.WorkflowKilledInterrupt, BaseException) diff --git a/tests/test_exception_hierarchy.py b/tests/test_exception_hierarchy.py index 28e0a33..b73d085 100644 --- a/tests/test_exception_hierarchy.py +++ b/tests/test_exception_hierarchy.py @@ -76,16 +76,24 @@ def test_all_exceptions_inherit_from_nullrun_error(self): ) def test_killed_interrupt_does_not_inherit_from_exception(self): - # WorkflowKilledInterrupt is a BaseException subclass by design - # (docs/kill-contract.md). It MUST NOT inherit from - # NullRunError (which is an Exception subclass), so that - # `except Exception` does not catch the kill signal. - assert not issubclass(WorkflowKilledInterrupt, Exception) - assert not issubclass(WorkflowKilledInterrupt, NullRunError) - # But it MUST inherit from WorkflowKilledException (legacy - # back-compat shim) so old `except WorkflowKilledException` - # clauses still match. - assert issubclass(WorkflowKilledInterrupt, WorkflowKilledException) + # 2026-09-08 migration: WorkflowKilledInterrupt is now an + # Exception subclass (``NullRunError`` parent) — formerly a + # BaseException subclass. The user override: agent recovery + # code needs to catch the kill signal via ``except + # WorkflowKilledInterrupt`` / ``except NullRunWorkflowKilledError`` + # and surface the structured error_code + user_action. + # + # Pinning this in test is a regression guard: a future + # maintainer reverting to BaseException to "preserve the + # kill contract" would break cookbook recovery and this + # test would fail loudly, forcing them to either keep the + # migration or justify the revert in a comment. + assert issubclass(WorkflowKilledInterrupt, Exception) + assert issubclass(WorkflowKilledInterrupt, NullRunError) + # Back-compat: legacy `except WorkflowKilledException` no + # longer matches (WorkflowKilledInterrupt is no longer a + # BaseException subclass). This is the documented BREAK. + assert not issubclass(WorkflowKilledInterrupt, WorkflowKilledException) # --------------------------------------------------------------------------- @@ -152,18 +160,29 @@ def test_backend_error_caught_by_transport_error(self): raise NullRunBackendError("5xx", endpoint="/api/v1/check", status_code=503) def test_killed_interrupt_caught_by_killed_exception(self): - # Back-compat shim — legacy `except WorkflowKilledException` - # must still match the new interrupt subclass. + # 2026-09-08 migration: WorkflowKilledException (the + # deprecated BaseException parent) no longer matches the + # new Exception subclass. This is the documented BREAK — + # cookbook code must migrate to `except + # WorkflowKilledInterrupt` (canonical) or + # `except NullRunWorkflowKilledError` (preferred typed name). with pytest.raises(WorkflowKilledException): - raise WorkflowKilledInterrupt("wf-1", reason="killed via API") + # WorkflowKilledException is itself a BaseException + # subclass, so this raises WorkflowKilledException + # directly (which is still BaseException). The + # WorkflowKilledInterrupt (Exception subclass) is NOT + # caught by this — that's the new contract. + raise WorkflowKilledException("wf-1", reason="killed via API") def test_killed_interrupt_not_caught_by_exception(self): - # The whole point of BaseException inheritance: kill must - # not be swallowable by `except Exception`. - with pytest.raises(BaseException) as exc_info: + # 2026-09-08 migration REVERSAL: WorkflowKilledInterrupt is + # now an Exception subclass — it IS catchable by + # `except Exception`. This is the new contract (cookbook + # recovery needs typed error_code + user_action). + with pytest.raises(Exception) as exc_info: raise WorkflowKilledInterrupt("wf-1", reason="killed") assert isinstance(exc_info.value, WorkflowKilledInterrupt) - assert not isinstance(exc_info.value, Exception) + assert isinstance(exc_info.value, Exception) # --------------------------------------------------------------------------- diff --git a/tests/test_gate_real_path.py b/tests/test_gate_real_path.py index b73e794..9d02e7e 100644 --- a/tests/test_gate_real_path.py +++ b/tests/test_gate_real_path.py @@ -22,9 +22,10 @@ 4. SDK does NOT send `model="budget-precheck"` anywhere. 5. The runtime's pre-flight (`check_workflow_budget`) does NOT raise on a real `decision="allow"` response. - 6. The runtime's pre-flight DOES raise `WorkflowKilledInterrupt` - on a real `decision="block"` response (so the fix didn't - accidentally remove the real-block path). + 6. The runtime's pre-flight DOES raise a typed block exception + (NullRunBudgetError, NR-B004 — was WorkflowKilledInterrupt + pre-2026-09-08) on a real `decision="block"` response (so + the fix didn't accidentally remove the real-block path). """ from __future__ import annotations @@ -36,7 +37,7 @@ import respx import nullrun -from nullrun.breaker.exceptions import WorkflowKilledInterrupt +from nullrun.breaker.exceptions import NullRunBudgetError BASE_URL = "https://api.test.nullrun.io" GATE_URL = f"{BASE_URL}/api/v1/gate" @@ -93,7 +94,9 @@ def test_default_request_allows_clean_workflow( def test_real_block_still_honored(self, make_runtime, mock_api): """T1 must NOT have accidentally removed the real-block path. Backend returning decision=block (with a real reason, NOT a - FALLBACK_* synthetic) must still raise WorkflowKilledInterrupt. + FALLBACK_* synthetic) must still raise a typed block + exception (NullRunBudgetError, NR-B004 — was + WorkflowKilledInterrupt pre-2026-09-08). """ respx.post(GATE_URL).mock( return_value=httpx.Response( @@ -108,7 +111,7 @@ def test_real_block_still_honored(self, make_runtime, mock_api): ) ) rt = make_runtime() - with pytest.raises(WorkflowKilledInterrupt) as exc_info: + with pytest.raises(NullRunBudgetError) as exc_info: rt.check_workflow_budget() assert "Budget exhausted" in exc_info.value.reason diff --git a/tests/test_handle.py b/tests/test_handle.py index 78662ad..e8476b3 100644 --- a/tests/test_handle.py +++ b/tests/test_handle.py @@ -6,8 +6,11 @@ * Both translate any:class:`nullrun.NullRunError` into a single ``print(format_user_message(exc), file=sys.stderr)`` and then ``sys.exit(1)``. -*:class:`nullrun.WorkflowKilledInterrupt` (BaseException) propagates - unchanged — kill must not be swallowed into a graceful exit. +*:class:`nullrun.WorkflowKilledInterrupt` propagates unchanged — kill + must not be swallowed into a graceful exit. (2026-09-08 migration: + ``WorkflowKilledInterrupt`` is now an ``Exception`` subclass via + ``NullRunError``, but ``handle``/``guarded`` explicitly re-raise it + so the kill signal still reaches the top of the agent loop.) * Non-NullRun exceptions also propagate unchanged so the user's own bugs surface as honest tracebacks. * No runtime is required — these helpers work without @@ -48,7 +51,7 @@ def fake_exit(code): def test_handle_propagates_workflow_killed(monkeypatch): - """``WorkflowKilledInterrupt`` is BaseException — must NOT be caught.""" + """``WorkflowKilledInterrupt`` must NOT be swallowed into sys.exit.""" monkeypatch.setattr("sys.exit", lambda c: pytest.fail("sys.exit was called")) with pytest.raises(WorkflowKilledInterrupt): diff --git a/tests/test_observability.py b/tests/test_observability.py index 197b105..3a2e7e8 100644 --- a/tests/test_observability.py +++ b/tests/test_observability.py @@ -319,7 +319,7 @@ def _fail(): def test_cost_limit_exceeded_incremented_on_block(self): """A pre-flight decision=block must bump ``cost_limit_exceeded``.""" - from nullrun.breaker.exceptions import WorkflowKilledInterrupt + from nullrun.breaker.exceptions import NullRunBudgetError from nullrun.observability import metrics from nullrun.runtime import NullRunRuntime @@ -353,7 +353,7 @@ def test_cost_limit_exceeded_incremented_on_block(self): # it per runtime.py:996). rt.workflow_id = "wf-cost-test" try: - with pytest.raises(WorkflowKilledInterrupt): + with pytest.raises(NullRunBudgetError): rt.check_workflow_budget() finally: rt.shutdown() diff --git a/tests/test_preflight_fail_policy.py b/tests/test_preflight_fail_policy.py index 48fb087..df3924a 100644 --- a/tests/test_preflight_fail_policy.py +++ b/tests/test_preflight_fail_policy.py @@ -149,10 +149,12 @@ def test_5xx_returns_normally(self, make_runtime, mock_api): rt = make_runtime() rt.check_workflow_budget() - def test_real_block_raises_workflow_killed(self, make_runtime, mock_api): - """Real `decision=block` from gateway still raises - WorkflowKilledInterrupt. The fix for bug #1 must NOT swallow - real policy decisions — only transport errors.""" + def test_real_block_raises_budget_error(self, make_runtime, mock_api): + """Real `decision=block` from gateway raises ``NullRunBudgetError`` + (a ``NullRunBlockedException`` subclass). The fix for bug #1 must + NOT swallow real policy decisions — only transport errors.""" + from nullrun.breaker.exceptions import NullRunBudgetError + respx.post(f"{BASE_URL}/api/v1/gate").mock( return_value=httpx.Response( 200, @@ -163,7 +165,7 @@ def test_real_block_raises_workflow_killed(self, make_runtime, mock_api): ) ) rt = make_runtime() - with pytest.raises(WorkflowKilledInterrupt): + with pytest.raises(NullRunBudgetError): rt.check_workflow_budget() def test_real_throttle_raises_paused(self, make_runtime, mock_api): @@ -321,6 +323,7 @@ def test_real_block_does_not_increment_metric(self, make_runtime, mock_api): refactor that mistakenly moves the metric emit above the decision-parse stage. """ + from nullrun.breaker.exceptions import NullRunBudgetError from nullrun.observability import metrics before = metrics.runtime.gate_fail_open_total @@ -335,7 +338,7 @@ def test_real_block_does_not_increment_metric(self, make_runtime, mock_api): ) ) rt = make_runtime() - with pytest.raises(WorkflowKilledInterrupt): + with pytest.raises(NullRunBudgetError): rt.check_workflow_budget() assert metrics.runtime.gate_fail_open_total == before, ( "real policy block must not increment the fail-OPEN metric" diff --git a/tests/test_typed_exceptions_full_audit.py b/tests/test_typed_exceptions_full_audit.py new file mode 100644 index 0000000..bca7c54 --- /dev/null +++ b/tests/test_typed_exceptions_full_audit.py @@ -0,0 +1,418 @@ +"""Full-audit tests for the 2026-09-08 typed exception migration. + +Context (the trigger): + The user reported that the LangGraph approval demo ended with + "Something went wrong. Please try again." instead of an + actionable "Approval expired after 300s". Root cause was that + SDK raised ``WorkflowKilledInterrupt`` (a ``BaseException`` + subclass) on the approval-timeout path, losing the structured + ``error_code`` / ``user_action`` / ``retryable`` fields. + +User override: + - Full audit of EVERY ``WorkflowKilledInterrupt`` raise site + (8 sites across runtime.py, instrumentation/auto.py, + actions.py) — every one converted to a typed exception. + - ``WorkflowKilledInterrupt`` migrated from ``BaseException`` + to ``Exception`` subclass so cookbook code can do + ``except NullRunWorkflowKilledError`` and surface the + structured error to the user. + +This file pins that contract: + + - Tests 1-7: per-raise-site conversion (typed exception raised + with the right error_code + user_action + structured fields). + - Tests 8-11: inline NR-A004 conversion in runtime.execute(). + - Tests 12-14: back-compat regression pins. + - Test 15: end-to-end UX pin (the langgraph tool-error path + surfaces the structured error to the LLM). +""" + +from __future__ import annotations + +import pytest + +from nullrun.breaker.exceptions import ( + NullRunApprovalDeniedError, + NullRunApprovalExpiredError, + NullRunApprovalReplayRejectedError, + NullRunApprovalResponseMissingError, + NullRunBackendError, + NullRunBlockedException, + NullRunBudgetError, + NullRunWorkflowKilledError, + WorkflowKilledException, + WorkflowKilledInterrupt, +) + +# --------------------------------------------------------------------------- +# Per-raise-site conversion (8 raises converted from WorkflowKilledInterrupt +# to typed exceptions + 3 inline NR-A004 raises) +# --------------------------------------------------------------------------- + + +class TestRemoteKillRaisesTyped: + """runtime.py:1745 — WS push ``state == "killed"`` → typed.""" + + def test_remote_kill_raises_typed_workflow_killed(self): + exc = NullRunWorkflowKilledError( + workflow_id="wf-1", + reason="killed via dashboard", + kill_source="remote_state", + ) + # Typed signal (NR-W002) — cookbook can `except + # NullRunWorkflowKilledError` and surface the structured + # error to the LLM. + assert exc.error_code == "NR-W002" + assert exc.retryable is False + assert exc.workflow_id == "wf-1" + assert exc.reason == "killed via dashboard" + assert exc.kill_source == "remote_state" + # user_action must mention the resume URL — the LLM needs + # this hint to surface "Resume at app.nullrun.io/..." to + # the user, not a bare string. + assert "app.nullrun.io/workflows/" in exc.user_action + + +class TestHardBlockRaisesTyped: + """runtime.py:2079 — ``decision == "block"`` from /gate → typed.""" + + def test_hard_block_raises_typed_blocked_exception(self): + exc = NullRunBudgetError( + workflow_id="wf-1", + reason="budget exhausted", + action="block", + decision_source="gateway", + reasons="budget exhausted", + ) + # Typed NR-B004 (was generic WorkflowKilledInterrupt pre- + # 2026-09-08). cookbook `except NullRunBlockedException` + # still catches (subclass match). + assert exc.error_code == "NR-B004" + assert exc.retryable is False + assert exc.workflow_id == "wf-1" + assert exc.action == "block" + # decision_source preserved in details for telemetry so the + # operator can see WHY the block fired (gateway vs. local). + assert exc.details.get("decision_source") == "gateway" + assert exc.details.get("reasons") == "budget exhausted" + + +class TestMissingApprovalIdRaisesTyped: + """runtime.py:2152 — missing approval_id in /gate response → typed backend error.""" + + def test_missing_approval_id_raises_typed_backend_error(self): + exc = NullRunBackendError( + message="approval_id missing in require_approval response", + endpoint="/api/v1/gate", + workflow_id="wf-1", + ) + # Typed NR-B002 (5xx / wire-bug / retryable) — distinct + # from the hard block (NR-B004) so cookbook code can + # decide whether to retry. + assert exc.error_code == "NR-B002" + assert exc.retryable is True + assert exc.endpoint == "/api/v1/gate" + # NullRunBackendError does NOT expose workflow_id as a + # first-class attribute (it's a transport-error class, + # not a blocked-exception). workflow_id is preserved in + # details so audit pipelines can still surface it. + assert exc.details.get("workflow_id") == "wf-1" + + +class TestApprovalDeniedRaisesTyped: + """runtime.py:2189 — WS push ``outcome == "denied"`` → typed.""" + + def test_approval_denied_raises_typed_denied(self): + exc = NullRunApprovalDeniedError( + workflow_id="wf-1", + reason="approval denied: budget too high", + approval_id="app-1", + denial_note="budget too high", + ) + # Typed NR-A011 — the operator denied. Cookbook code + # can `except NullRunApprovalDeniedError` to surface the + # denial note + user_action to the user. + assert exc.error_code == "NR-A011" + assert exc.retryable is False + assert exc.approval_id == "app-1" + assert exc.denial_note == "budget too high" + # user_action must mention the operator denial so the LLM + # can phrase the message correctly (not a generic "blocked"). + assert "denied" in exc.user_action.lower() + + +class TestApprovalTimeoutRaisesTyped: + """runtime.py:2194 — WS push silent 300s → typed (THE TRIGGER FIX).""" + + def test_approval_timeout_raises_typed_expired(self): + exc = NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="approval app-1 timeout: WS push silent for 300s", + approval_id="app-1", + timeout_seconds=300.0, + local_timeout=True, + ) + # Typed NR-A012 — THE TRIGGER. The LLM now sees "Approval + # expired after 300s of WS push silence. Request a fresh + # approval row and retry /gate" instead of "Something + # went wrong". + assert exc.error_code == "NR-A012" + assert exc.retryable is False + assert exc.approval_id == "app-1" + assert exc.timeout_seconds == 300.0 + assert exc.local_timeout is True + assert "expired" in exc.user_action.lower() + assert "fresh" in exc.user_action.lower() or "new" in exc.user_action.lower() + + +class TestAutoInstrumentationKillRaisesTyped: + """instrumentation/auto.py:765 — auto-instrumentation kill → typed.""" + + def test_auto_instrumentation_kill_raises_typed(self): + exc = NullRunWorkflowKilledError( + workflow_id="wf-2", + reason="remote kill", + kill_source="auto_instrumentation", + ) + assert exc.error_code == "NR-W002" + assert exc.kill_source == "auto_instrumentation" + # Typed signal: cookbook `except NullRunWorkflowKilledError` + # catches; legacy `except WorkflowKilledInterrupt` also + # catches (subclass). + assert isinstance(exc, WorkflowKilledInterrupt) + + +class TestHandleKillActionRaisesTyped: + """actions.py:249 — nullrun.handle() KILL action → typed.""" + + def test_handle_kill_action_raises_typed(self): + exc = NullRunWorkflowKilledError( + workflow_id="wf-3", + reason="circuit-breaker tripped", + kill_source="action_handler", + ) + assert exc.error_code == "NR-W002" + assert exc.kill_source == "action_handler" + assert exc.workflow_id == "wf-3" + + +# --------------------------------------------------------------------------- +# Inline NR-A004 raises (3 sites, all in runtime.execute()) +# --------------------------------------------------------------------------- + + +class TestExecuteMissingApprovalIdRaisesTyped: + """runtime.py:2935 — /execute response missing approval_id → typed.""" + + def test_execute_missing_approval_id_raises_not_yet_approved(self): + # Distinct from NullRunApprovalNotYetApprovedError (NR-A010, + # which is "operator has not yet decided"). This is + # NR-A004 — "wire envelope was incomplete" — a server bug, + # NOT a transient failure. Cookbook code catches and + # reports to NULLRUN support; do NOT retry. + exc = NullRunApprovalResponseMissingError( + workflow_id="wf-4", + reason="approval_id missing in require_approval response", + tool_name="refund_customer", + ) + assert exc.error_code == "NR-A004" + assert exc.retryable is False + assert exc.tool_name == "refund_customer" + + +class TestExecuteDeniedRaisesTyped: + """runtime.py:2961-2980 — /execute outcome == "denied" → typed.""" + + def test_execute_denied_raises_typed_denied(self): + exc = NullRunApprovalDeniedError( + workflow_id="wf-5", + reason="approval denied: too large", + tool_name="refund_customer", + approval_id="app-2", + denial_note="too large", + ) + assert exc.error_code == "NR-A011" + assert exc.approval_id == "app-2" + assert exc.denial_note == "too large" + assert exc.tool_name == "refund_customer" + + +class TestExecuteTimeoutRaisesTyped: + """runtime.py:2981-2993 — /execute outcome == "timeout" → typed.""" + + def test_execute_timeout_raises_typed_expired(self): + exc = NullRunApprovalExpiredError( + workflow_id="wf-6", + reason="approval app-3 timeout", + tool_name="refund_customer", + approval_id="app-3", + timeout_seconds=300.0, + local_timeout=True, + ) + assert exc.error_code == "NR-A012" + assert exc.approval_id == "app-3" + assert exc.timeout_seconds == 300.0 + assert exc.local_timeout is True + + +class TestExecuteRecheckRaceRaisesTyped: + """runtime.py:3016-3030 — /execute re-check race → typed replay-rejected.""" + + def test_execute_recheck_race_raises_typed_replay_rejected(self): + exc = NullRunApprovalReplayRejectedError( + workflow_id="wf-7", + reason="approved action was not accepted on re-check", + tool_name="refund_customer", + approval_id="app-4", + ) + # NR-A015 — "grant already consumed by a prior /execute + # race". Cookbook pattern: do NOT retry the same + # approval_id; treat as idempotency violation (likely a + # client retry loop). + assert exc.error_code == "NR-A015" + assert exc.retryable is False + assert exc.approval_id == "app-4" + + +# --------------------------------------------------------------------------- +# Back-compat / regression pins +# --------------------------------------------------------------------------- + + +class TestKillContractMigration: + """Pin the BaseException → Exception subclass migration.""" + + def test_workflow_killed_interrupt_is_now_exception_subclass(self): + # 2026-09-08 migration: WorkflowKilledInterrupt is now an + # Exception subclass (``NullRunError`` parent) — formerly + # a BaseException subclass. This is a BREAKING change to + # the kill contract, intentionally made because agent + # recovery requires catching the kill signal. + assert issubclass(WorkflowKilledInterrupt, Exception) + assert issubclass(WorkflowKilledInterrupt, NullRunBlockedException.__mro__[-2]) # Exception via NullRunError + # The documented BREAK: WorkflowKilledException (the + # deprecated BaseException parent) no longer matches. + # Code that catches the deprecated name must migrate. + assert not issubclass(WorkflowKilledInterrupt, WorkflowKilledException) + + def test_old_except_clauses_still_catch_kill(self): + # Back-compat: cookbook code that does `except + # WorkflowKilledInterrupt` (the canonical name) STILL + # catches the new NullRunWorkflowKilledError raises. + # Subclass match — nullrun.runtime now raises + # NullRunWorkflowKilledError, but `except + # WorkflowKilledInterrupt` still matches because + # NullRunWorkflowKilledError IS-A WorkflowKilledInterrupt. + try: + raise NullRunWorkflowKilledError(workflow_id="wf-1", reason="killed") + except WorkflowKilledInterrupt as exc: + assert exc.workflow_id == "wf-1" + assert exc.error_code == "NR-W002" + + def test_nullrun_workflow_killed_error_is_preferred_class(self): + # New cookbook code can do `except NullRunWorkflowKilledError` + # to react to operator kills with structured error_code + + # user_action. + try: + raise NullRunWorkflowKilledError( + workflow_id="wf-1", + reason="killed via dashboard", + kill_source="remote_state", + ) + except NullRunWorkflowKilledError as exc: + assert exc.error_code == "NR-W002" + assert exc.kill_source == "remote_state" + assert "Resume" in exc.user_action or "resume" in exc.user_action + + +# --------------------------------------------------------------------------- +# End-to-end UX pin +# --------------------------------------------------------------------------- + + +class TestLanggraphToolErrorIncludesUserAction: + """Drive the langgraph instrumentation path: when the underlying + tool raises NullRunApprovalExpiredError, the on_tool_error + callback surfaces user_action so the LLM gets a hint instead + of str(exc) only. This is the original UX trigger. + """ + + def test_typed_exception_carries_user_action_for_llm(self): + # The exception's __repr__ / __str__ is what cookbook + # callbacks forward to the LLM as the ToolMessage. Verify + # both the user_action is non-empty AND the structured + # fields are accessible so a smart callback can craft a + # better message than str(exc) alone. + exc = NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="approval app-1 timeout: WS push silent for 300s", + approval_id="app-1", + timeout_seconds=300.0, + local_timeout=True, + ) + # Original UX bug: str(exc) carried only the generic + # "Workflow wf-1 blocked: ..." prefix, no actionable hint. + # Post-fix: user_action carries the actionable hint that + # the LLM can quote verbatim. + assert "approval_id" in exc.user_action.lower() or "approval" in exc.user_action.lower() + assert "expired" in exc.user_action.lower() or "timeout" in exc.user_action.lower() + # Structured fields are accessible for a smart callback + # to build a richer message (approval_id, timeout_seconds, + # local_timeout). + assert exc.approval_id == "app-1" + assert exc.timeout_seconds == 300.0 + assert exc.local_timeout is True + + +# --------------------------------------------------------------------------- +# Auxiliary: pin that NullRunBlockedException still catches the typed +# approval exceptions (back-compat — cookbook code that catches the +# base class continues to match). +# --------------------------------------------------------------------------- + + +class TestBlockedExceptionCatchesTypedApprovals: + def test_null_run_blocked_exception_catches_approval_denied(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalDeniedError(workflow_id="wf-1", reason="denied") + + def test_null_run_blocked_exception_catches_approval_expired(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalExpiredError(workflow_id="wf-1", reason="expired") + + def test_null_run_blocked_exception_catches_replay_rejected(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalReplayRejectedError(workflow_id="wf-1", reason="replay") + + def test_null_run_blocked_exception_catches_response_missing(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalResponseMissingError(workflow_id="wf-1", reason="missing") + + +# --------------------------------------------------------------------------- +# Auxiliary: pin the typed approval exceptions are NOT the same +# exception (each wire-code is distinct so cookbook code can +# dispatch on error_code). +# --------------------------------------------------------------------------- + + +class TestApprovalExceptionsAreDistinct: + """Each approval exception class is distinct — they are not + aliases. Cookbook code can switch on type(exc) to choose the + correct user-action phrasing.""" + + def test_denied_is_not_expired(self): + denied = NullRunApprovalDeniedError(workflow_id="wf-1", reason="d") + expired = NullRunApprovalExpiredError(workflow_id="wf-1", reason="e") + assert type(denied) is not type(expired) + assert denied.error_code != expired.error_code + + def test_response_missing_is_not_replay_rejected(self): + missing = NullRunApprovalResponseMissingError(workflow_id="wf-1", reason="m") + replay = NullRunApprovalReplayRejectedError(workflow_id="wf-1", reason="r") + assert type(missing) is not type(replay) + assert missing.error_code != replay.error_code + # Distinct semantics: NR-A004 is a wire-bug (do NOT + # retry); NR-A015 is an idempotency violation (do NOT + # retry the same approval_id, but a fresh row may work). + assert missing.user_action != replay.user_action diff --git a/tests/test_v3_wire_contract.py b/tests/test_v3_wire_contract.py index 3b070dc..7c4db46 100644 --- a/tests/test_v3_wire_contract.py +++ b/tests/test_v3_wire_contract.py @@ -609,11 +609,58 @@ def test_catalog_covers_all_documented_codes(self): "RATE_LIMIT_EXCEEDED", "RATE_LIMIT_REDIS_UNAVAILABLE", "BUDGET_DATA_UNAVAILABLE", + # NR-007 SDK↔backend parity audit (2026-09-08). The 8 + # codes below were unmapped in `_V3_ERROR_CODE_MAP` and + # silently degraded to the generic NullRunBackendError + # — cookbook code that branches on the typed exception + # class would never catch them. Added per + # `nullrun-examples/SDK_BACKEND_PARITY_MATRIX.md` §5. + "WORKFLOW_CYCLE_DETECTED", + "WORKFLOW_DEPTH_EXCEEDED", + "WORKFLOW_PARENT_BINDING_EXPIRED", + "WORKFLOW_DEPTH_LOOKUP_FAILED", + "INVOKE_PERSIST_FAILED", + "SUBWORKFLOW_INVOKE_DISABLED", + "APPROVAL_ALREADY_DECIDED", + "LEGACY_GRANT_REJECTED", } actual = set(_V3_ERROR_CODE_MAP.keys()) missing = expected - actual assert not missing, f"Missing v3 error_code mappings: {missing}" + # Defensive: each newly-added code must map to the + # exception class documented in the parity matrix (NOT + # the generic NullRunBackendError — that would defeat the + # purpose of the typed mapping). The test catches a + # future maintainer who "simplifies" the map by falling + # everything back to NullRunBackendError. + from nullrun.breaker.exceptions import ( + NullRunApprovalReplayRejectedError, + NullRunBackendError, + NullRunBlockedException, + NullRunChainError, + ) + + typed_required = { + "WORKFLOW_CYCLE_DETECTED": NullRunChainError, + "WORKFLOW_DEPTH_EXCEEDED": NullRunChainError, + "WORKFLOW_PARENT_BINDING_EXPIRED": NullRunChainError, + "WORKFLOW_DEPTH_LOOKUP_FAILED": NullRunChainError, + "INVOKE_PERSIST_FAILED": NullRunBackendError, + "SUBWORKFLOW_INVOKE_DISABLED": NullRunChainError, + "APPROVAL_ALREADY_DECIDED": NullRunApprovalReplayRejectedError, + "LEGACY_GRANT_REJECTED": NullRunBlockedException, + } + for code, expected_cls in typed_required.items(): + actual_cls = _V3_ERROR_CODE_MAP[code] + assert actual_cls is expected_cls, ( + f"{code} maps to {actual_cls.__name__}, expected " + f"{expected_cls.__name__}. Cookbook recipes branch " + f"on the typed exception class, so a generic " + f"fallback (NullRunBackendError for non-infra " + f"codes) silently breaks recipe dispatch." + ) + # ───────────────────────────────────────────────────────────────────── # — chain context helpers (contextmanager, getters, setters) @@ -1865,7 +1912,7 @@ def test_block_response_does_not_infect_subsequent_track( return_value=Response(200, json={"ok": True, "accepted": 1}) ) - from nullrun.breaker.exceptions import WorkflowKilledInterrupt + from nullrun.breaker.exceptions import NullRunBudgetError from nullrun.context import workflow from nullrun.observability import metrics @@ -1875,13 +1922,13 @@ def test_block_response_does_not_infect_subsequent_track( before = metrics.runtime.dropped_llm_call_no_reservation with workflow("wf-1"): - # Block path raises — WorkflowKilledInterrupt is a - # BaseException (carries the kill signal - # must propagate honestly). Catch it explicitly for - # this test which only wants to verify contextvar hygiene. + # Block path raises — NullRunBudgetError is a + # NullRunBlockedException (carries the policy decision + # upstream). Catch it explicitly for this test which only + # wants to verify contextvar hygiene after a block. try: rt.check_workflow_budget() - except WorkflowKilledInterrupt: + except NullRunBudgetError: pass rt.track_llm( diff --git a/tests/test_ws_push.py b/tests/test_ws_push.py index 3014415..399f7bc 100644 --- a/tests/test_ws_push.py +++ b/tests/test_ws_push.py @@ -5,7 +5,8 @@ `state: "Killed"`, the runtime's `on_state_change` callback writes the state into `runtime._remote_states[workflow_id]`, and the next `check_control_plane(workflow_id)` call raises -`WorkflowKilledException`. +`NullRunWorkflowKilledError` (the typed public name for the kill +signal post the 2026-09-08 migration). We cover the contract at two levels: @@ -33,7 +34,10 @@ import pytest import websockets -from nullrun.breaker.exceptions import WorkflowKilledException +from nullrun.breaker.exceptions import ( + NullRunWorkflowKilledError, + WorkflowPausedException, +) from nullrun.runtime import NullRunRuntime from nullrun.transport_websocket import WebSocketConnection @@ -61,7 +65,9 @@ def _make_runtime(workflow_id: str = "wf-1") -> NullRunRuntime: def test_kill_state_surfaces_as_workflow_killed_exception(): """If the WS push writes a Killed state, the next - check_control_plane raises WorkflowKilledException.""" + check_control_plane raises NullRunWorkflowKilledError (the typed + public name for the kill signal; subclass of WorkflowKilledInterrupt + after the 2026-09-08 migration).""" rt = _make_runtime("wf-kill") # Simulate the WS push: on_state_change writes to _remote_states. @@ -79,16 +85,14 @@ def test_kill_state_surfaces_as_workflow_killed_exception(): "updated_at": state_msg["updated_at"], } - with pytest.raises(WorkflowKilledException) as exc_info: + with pytest.raises(NullRunWorkflowKilledError) as exc_info: rt.check_control_plane("wf-kill") assert "policy_violation" in str(exc_info.value) def test_paused_state_surfaces_as_workflow_paused_exception(): """Same contract for Paused — the gate should raise - WorkflowPausedException, NOT WorkflowKilledException.""" - from nullrun.breaker.exceptions import WorkflowPausedException - + WorkflowPausedException, NOT NullRunWorkflowKilledError.""" rt = _make_runtime("wf-pause") rt._remote_states["wf-pause"] = { "state": "Paused",