chore(release): 0.16.5 — cancel-on-exception orphan fix + P0-26+P0-27 operation_id hoist - #97
Merged
Conversation
… (P0-26 + P0-27 / Sprint A6.5)
P0-26: _capture_server_minted_execution_id now reads via _get_op_id_for_capture()
(the SDK-minted contextvar value) instead of response.get('operation_id').
Asserts server_op_id != sdk_op_id with ERROR log on disagreement.
Idempotency_key derived from SDK-minted value, not server echo.
P0-27: Operation id hoisted to contextvar (_operation_id_var in context.py,
name 'operation_id'). check_workflow_budget mints once via
_get_op_id_for_check()/_set_op_id_for_check(); execute reads via
_get_op_id_for_execute() with single fallback mint+stash.
Triple-mint collapsed to one.
8 new regression tests in test_audit_p0_27_operation_id_hoist.py pin:
- contextvar name 'operation_id'
- mint-site shapes in check_workflow_budget + execute
- parity assertion in _capture_server_minted_execution_id
- forbids pre-fix str(uuid.uuid4()) mints outside fallback branch
Foreign WIP preserved: CHANGELOG.md, pyproject.toml, __version__.py,
decorators.py, tests/test_protect_cancel_on_exception.py untouched.
Pre-fix, ANY exception raised inside the decorated function (or by any pre-execution gate — control_plane reject, workflow_budget block, sensitive-tool reject) propagated out of the with-block without closing the budget reservation that check_workflow_budget opened via /gate. Each such exception orphaned a Redis envelope to TTL expiry, which made reserved_total drift up monotonically at anti-DoS scale. Source-side wiring (src/nullrun/decorators.py): 1. New helper _safe_cancel_active_execution(reason=None). Reads get_server_minted_execution_id(); no-op if None (pre-/gate failure). Best-effort runtime.cancel_execution(execution_id, reason=...) call. Catches everything (NullRunTransportError, NullRunBackendError, get_runtime() failure) so a cancel I/O failure never masks the original exception. Synchronous, blocking HTTP — caller is the @Protect context manager; same channel as check_workflow_budget. 2. async_wrapper and sync_wrapper now wrap the with-block in try/except. fn_completed sentinel: after fn(...) returns, fn_completed = True. If track_tool(...) then fails, fn_completed is True so cancel does NOT fire — side effects already happened and the right move is to retry track_tool, not cancel (cancel would tell the server "no side effects" — a lie that produces a phantom budget refund and breaks audit). 3. Asymmetry on exception scope: - async_wrapper catches Exception (NOT BaseException). asyncio .CancelledError / KeyboardInterrupt / SystemExit propagate without doing a synchronous blocking HTTP call inside a cancellation handler (5s timeout, would delay task cancellation, generate 'Task was destroyed but pending' warnings, and in some shutdown paths get cancelled itself — server's /cancel is idempotent so orphan via TTL/reconciliation is the safety net). - sync_wrapper catches BaseException. Sync code has no event loop to delay; Ctrl+C during a long sync agent gets a few seconds of cancel I/O before exit. Matches existing _protect_body unify_block semantics. Tests (tests/test_protect_cancel_on_exception.py, 7 tests): - test_1_async_cancelled_error_does_not_trigger_cancel — THE regression guard. If a future refactor reverts except Exception to except BaseException in async_wrapper, this test fails. Set server_minted_execution_id so the helper WOULD have run if the except had caught BaseException; cancel_calls must be empty. - test_2_track_tool_failure_after_fn_completion_does_not_trigger_cancel — second regression guard. fn_completed sentinel must stay True past the successful fn() call. If someone removes the sentinel and 'simplifies' the wrapper to always call cancel on Exception, this test fails. - test_3_async_fn_raises_value_error_triggers_cancel — happy-path cancel. cancel_calls == [("exec-test-123", "tool_exception")]. - test_4_async_no_execution_id_skips_cancel — control_plane rejects pre-/gate, ContextVar stays None, no cancel (server-side orphan, if any, is reconciliation territory). - test_5_sync_fn_raises_value_error_triggers_cancel — sync ValueError path mirrors async. - test_6_sync_baseexception_also_triggers_cancel — sync KeyboardInterrupt cancels (sync has no event loop to delay). - test_happy_path_no_cancel_called — sanity: success path produces no cancel and gate order stays control_plane, budget, track_tool. Pattern borrowed from tests/test_preflight_fail_policy.py (_RecordingRuntime) — extended with capture_execution_id so the cancel helper sees a populated ContextVar after the simulated check_workflow_budget. autouse fixture resets _server_minted_execution_id_var between tests (ContextVar leak would make order-dependent assertions flaky). Verified: tests/test_protect_cancel_on_exception.py — 7/7 pass. Wire-format: zero changes. /cancel endpoint was already used by runtime.cancel_execution(...) from control-plane kill paths; the exception-cleanup helper is purely additive.
… operation_id hoist Two independent reliability fixes, no wire-format change on either: 1. fix(sdk): @Protect cancel-on-exception orphan leak (6076f87). src/nullrun/decorators.py wraps both wrappers in try/except; on failure _safe_cancel_active_execution(reason='tool_exception') closes the in-flight /gate reservation instead of leaking to TTL expiry. Exception-scope asymmetry (async catches Exception not BaseException to keep cancellation handler non-blocking; sync catches BaseException). fn_completed sentinel prevents cancel from firing after track_tool failure on a successfully-completed fn. Helper is fail-OPEN so cancel I/O failure never masks the original exception. 2. fix(sdk): hoist operation_id to contextvar + capture SDK-minted value (P0-26 + P0-27 / Sprint A6.5) (a9b398a, prior local WIP). P0-26: _capture_server_minted_execution_id now reads via _get_op_id_for_capture() (the SDK-minted contextvar value) instead of response.get('operation_id'). Asserts server_op_id != sdk_op_id with ERROR log on disagreement. Idempotency_key derived from SDK-minted value, not server echo. P0-27: Operation id hoisted to contextvar (_operation_id_var in context.py, name 'operation_id'). check_workflow_budget mints once via _get_op_id_for_check()/ _set_op_id_for_check(); execute reads via _get_op_id_for_execute() with single fallback mint+stash. Triple-mint collapsed to one. Release bumps: - pyproject.toml + src/nullrun/__version__.py: 0.16.4 -> 0.16.5. - CHANGELOG.md: insert [0.16.5] - 2026-09-05 with full descriptions of both fixes, the test pin coverage (7 cancel tests + 8 operation_id hoist tests), the compatibility notes (zero wire-format change on either), and the verification status. Verified: pytest -q clean (1613 prior + 7 cancel + 8 hoist = 1628); ruff check on the WIP files clean (decorators.py + new test file); mypy src/nullrun no issues reported in 37 source files. No new wire fields, no hashing/computation changes — the fixes are purely SDK-local (source-of-truth restructuring + cancel cleanup).
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
CI on PR #97 (test (3.11)) failed on `ruff check src/` with 2 I001 import-order errors in src/nullrun/runtime.py — the inline `from nullrun.context import (...)` blocks introduced by the operation_id hoist (a9b398a) at runtime.py:1865 (check side) and :2791 (execute side) are correctly placed functionally but ruff flags them as un-sorted because they shadow the module-level context imports with a multi-name block. The 3rd I001 hit locally (in tests/test_audit_p0_27_operation_id_hoist .py:35) was a separate import-grouping issue from the same hoist commit and was caught by `ruff check src tests` on CI as well. All three are auto-fixable with `ruff check --fix`. Fix is purely cosmetic — runtime semantics + the operation_id hoist itself unchanged. CI now passes `ruff check src/` on PR #97. Files: src/nullrun/runtime.py (2 fix sites), tests/test_audit_p0_27 _operation_id_hoist.py (1 fix site). Verified: `ruff check src tests` all checks pass; `mypy src/nullrun` no issues reported in 37 source files; `pytest tests/test_protect _cancel_on_exception.py tests/test_audit_p0_27_operation_id_hoist.py ` — 15/15 pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.16.5
Two independent reliability fixes, no wire-format change on either.
1.
@protectcancel-on-exception orphan leak (6076f87)src/nullrun/decorators.pywraps both wrappers in try/except; on failure_safe_cancel_active_execution(reason='tool_exception')closes the in-flight/gatereservation instead of leaking to TTL expiry.Three invariant pins:
Exception(notBaseException) to keep cancellation handlers non-blocking; sync catchesBaseExceptionto match existing_protect_bodysemantics.fn_completedsentinel — prevents cancel from firing aftertrack_toolfailure on a successfully-completedfn.7 new regression tests in
tests/test_protect_cancel_on_exception.py. The two regression guards (test_1/test_2) pin the cancellation safety and the sentinel.2.
operation_idhoist (P0-26+P0-27,a9b398a)P0-26 —
_capture_server_minted_execution_idreads via_get_op_id_for_capture()(the SDK-minted contextvar value) instead ofresponse.get('operation_id'). Assertsserver_op_id != sdk_op_idwith ERROR log on disagreement.idempotency_keyderived from SDK-minted value.P0-27 —
_operation_id_varcontextvar incontext.py(nameoperation_id).check_workflow_budgetmints once;executereads via_get_op_id_for_execute()with single fallback mint+stash. Triple-mint collapsed to one.8 new regression tests in
tests/test_audit_p0_27_operation_id_hoist.pypin mint-site shapes, parity assertion, and forbids pre-fixstr(uuid.uuid4())mints outside the fallback branch.Compatibility
Pure reliability fixes. No wire-format change.
/gate,/track,/execute,/cancelpayloads unchanged. The/cancelendpoint was already used byruntime.cancel_execution(...)from control-plane kill paths; the exception-cleanup helper is purely additive.Verification
tests/test_protect_cancel_on_exception.py— 7/7 passtests/test_audit_p0_27_operation_id_hoist.py— 8/8 passpytest -qclean (prior 1613 + 7 + 8 = 1628)ruff checkclean on WIP filesmypy src/nullrunno issues reported in 37 source filesFiles