Skip to content

test(profiler): Remove wall-clock race in continuous profiler auto-start tests - #7483

Draft
LouisDeconinck wants to merge 1 commit into
getsentry:masterfrom
LouisDeconinck:fix/continuous-profiler-test-flake
Draft

test(profiler): Remove wall-clock race in continuous profiler auto-start tests#7483
LouisDeconinck wants to merge 1 commit into
getsentry:masterfrom
LouisDeconinck:fix/continuous-profiler-test-flake

Conversation

@LouisDeconinck

Copy link
Copy Markdown

Description

Refs #6957.

Source of the flake

In test_continuous_profiler_auto_start_and_stop_sampled{,_span_streaming}, the block between the first and second transaction/segment relied on time.sleep(0.03) followed by assert get_profiler_id() is not None and, later, len(profiler_ids) == 1 / max_chunks=1. Those assertions only hold if the test thread wakes before the sampler's soft shutdown commits.

Soft shutdown needs at least two sampler cycles after the last profile stops — the first observes and removes the inactive profile, the next sees empty queues and returns True, and running is set to False at the end of that cycle's interval. At the mocked 21 Hz frequency that is >= ~47.6 ms after profile.stop(), so the 30 ms sleep had only ~17 ms of margin: if the test thread oversleeps or is preempted past the window (routine on a loaded CI runner), the profiler legitimately stops, and either profiler should be running fails or transaction 2 spins up a new profiler session, breaking the single-profiler-id/single-chunk assertions. This is the same class of race as the originally reported assert get_profiler_id() is None failure (fixed in #7364), just in the other direction. Reproduced deterministically by lengthening the gap sleep past the shutdown window.

Fix

Added suspend_profiler_sampling(), a small helper that wraps scheduler.sampler with a gate (threading.Event). While suspended, the sampler thread parks inside self.sampler() and cannot observe stopped profiles or complete soft shutdown, so get_profiler_id() keeps reporting the running profiler regardless of test-thread scheduling. Transaction/segment 2's profile is already queued by auto_start() before the gate is released inside its body, so the same buffer/profiler session is reused deterministically — no wall-clock assumptions remain in the critical section. The original sampler is restored in a finally, so the parked thread always resumes and self-terminates normally even on failure.

This does not mask genuine profiler failures: an early/unexpected shutdown still fails profiler should be running, and a profiler that never stops still fails wait_for_profiler_to_stop()'s bounded poll. That poll's ceiling was raised to 5 s — it waits for an event that must occur once the queues are empty, so the bound only guards against scheduler starvation, not timing.

Test coverage / verification

  • TESTPATH=tests/profiler/test_continuous_profiler.py uv run tox -e py3.12-gevent — 92 passed (covers thread and gevent params)
  • pytest tests/profiler/ — 105 passed
  • Formerly flaky tests repeated 45x, including 15x under full CPU saturation (nproc busy loops) — all green
  • Before the fix, the same test deterministically fails when the inter-transaction gap exceeds the shutdown window; with the gate, injected 0.5 s+0.3 s delays inside the suspended window still pass
  • uv run ruff check / ruff format — clean; uv run --group typing mypy sentry_sdk — no issues

Reminders

  • Please add tests to validate your changes, and lint your code using uv run ruff.
  • Add GH Issue ID & Linear ID (if applicable)
  • PR title should use conventional commit style (feat:, fix:, ref:, meta:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant