Skip to content

fix(benchmarks): accept the multi-scenario verify.py invocation its docs advertise - #237

Merged
lesnik512 merged 2 commits into
mainfrom
fix/235-verify-multi-scenario
Sep 20, 2026
Merged

lesnik512 merged 2 commits into
mainfrom
fix/235-verify-multi-scenario

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #235.

verify.py's docstring advertised three scenarios, its parser took one, and because the parser is
built with description=__doc__ the --help output printed the invocation the file rejected.

Implements option 1 from the issue: nargs="+", with each scenario re-invoking sys.executable,
copying the pattern run.py:bench() already uses.

Why not a loop

verify.py calls sentry_sdk.init() and applies sentry_scenarios.PATCHES in-process, and
inprocess.py states the constraint:

One process per scenario is not tidiness: sentry_sdk.init() monkeypatches Starlette and the
stdlib logging module, set_tracer_provider is set-once, and the Prometheus registry is global.
Nothing here can be undone between scenarios.

A loop in one process would run scenario 2 through scenario 1's monkeypatches and report results
that look fine and are wrong, which is worse than the loud error it replaced. The one surviving
comment in main() points at inprocess.py so the next reader does not collapse it back.

A single scenario still runs in-process, unchanged: the parent has done nothing at that point.

Validation moved up front

Not asked for by #235, and worth flagging as added scope. Option 1 makes a late typo destructive:
verify.py errors_only nope would otherwise run the first scenario, then fail with a raw
CalledProcessError traceback. Every scenario is now resolved before anything spawns.

This also upgrades the single-scenario path. On main, verify.py nope died with a bare KeyError.

invocation before after
verify.py a b error: unrecognized arguments: b one JSON line per scenario
verify.py errors_only nope n/a error: unknown scenario: nope, exit 2, nothing run
verify.py nope KeyError traceback error: unknown scenario: nope, exit 2
verify.py off error: scenario 'off' captures nothing unchanged

Verification

No test covers benchmarks/, so the seam is a runnable acceptance check rather than a pytest test:
N scenarios in one invocation must equal N separate invocations.

My first version of that check compared raw output and failed. The difference was
errors_only_no_txn's trace_id, which is freshly random per process precisely because that
scenario does not continue the incoming trace. The check was wrong, not the code. It now compares
semantics and additionally asserts that a continued trace keeps its incoming id while a fresh one
differs across processes, which turns the nondeterminism into evidence that the isolation is real.

All four scenarios pass, including errors_only_skip_txn, which applies a monkeypatch and would
leak under a loop. Review confirmed independently that running it first does not leak into a
following scenario either.

benchmarks/README.md §8's verify.py line now uses the four-scenario form, runs verbatim, and
reproduces §5's trade-off table exactly:

errors_only               continues trace, breadcrumbs
errors_only_no_txn        no trace continuation, breadcrumbs
errors_only_logging_lean  continues trace, no breadcrumbs
errors_only_skip_txn      continues trace, breadcrumbs

ruff format, ruff check, ty, 307 tests and mkdocs build --strict are clean.

Review changes

Both axes flagged that describe_one's None guard was unreachable once main() validated. It is
removed properly rather than deleted: a resolve() helper returns the narrowed builder, so each
error message exists once and ty keeps its narrowing (it treats parser.error as NoReturn,
confirmed before committing to the shape). Comments went from five lines to one. HERE dropped
typing.Final to match run.py:21 and run_http.py:24, the positional became scenarios to match
run.py:68, and describe_one became run_scenario since describe() already sits beside it.

Two suggestions declined

  • RawDescriptionHelpFormatter. Argparse does collapse the docstring, so the example reads badly
    in --help. But all six scripts in benchmarks/ use plain ArgumentParser(description=__doc__),
    so this would break a consistent convention for a cosmetic gain on a pre-existing wart.
  • Softening check=True on the subprocess call. After pre-flight validation a child crash is
    genuinely exceptional, and failing fast beats continuing to the next scenario.

Known asymmetry

run.py spawns a separate worker module (inprocess.py); verify.py spawns itself, which is what
forces the len(scenarios) > 1 fork. Extracting a worker module for a 140-line script would cost
more than it returns, but it is the reason this file has two execution modes where run.py has one.

Benchmarks only. No production code, no runtime behaviour.

@lesnik512
lesnik512 merged commit dcbfe77 into main Sep 20, 2026
13 checks passed
@lesnik512
lesnik512 deleted the fix/235-verify-multi-scenario branch September 20, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

benchmarks/verify.py advertises a multi-scenario CLI its argparse rejects

1 participant