Skip to content

Fix flaky expression language probe test setup - #7570

Open
P403n1x87 wants to merge 1 commit into
mainfrom
dd/fix-flaky-expression-language-probe-setup-20260824
Open

Fix flaky expression language probe test setup#7570
P403n1x87 wants to merge 1 commit into
mainfrom
dd/fix-flaky-expression-language-probe-setup-20260824

Conversation

@P403n1x87

Copy link
Copy Markdown
Contributor

Motivation

test_expression_language_comparison_operators is flaky (~0.2%, 1/483 runs in the last 30 days). The test creates 96 probes (48 expressions × 2 probe types for Python). Under CI load, the default 30s timeout for wait_for_all_probes can expire before all probes reach INSTALLED. Because the return value was ignored, the test proceeded to send a weblog request against uninstalled probes that never emit, producing a confusing "probes are not emitting: RECEIVED" assertion failure instead of a clear setup error.

Changes

  • tests/debugger/test_debugger_expression_language.py:
    • _setup: check the wait_for_all_probes(statuses=["INSTALLED"], timeout=60) return value; on failure, append to self.setup_failures and return early so the test does not request against uninstalled probes. Also bumped wait_for_all_snapshots() to timeout=60.
    • _assert: added self.assert_setup_ok() as the first assertion after self.collect() so setup failures surface as a clear error.

This aligns _setup/_assert with the established robust pattern already used by sibling debugger tests (test_debugger_capture_expressions.py, test_debugger_probe_snapshot.py).

Testing

  • Ran ruff check on the modified file — all checks pass.
  • Change is behavior-preserving on the happy path; it only adds a longer install timeout and converts a silent flaky failure into an explicit, deterministic setup-failure assertion.

PR by Bits - View session in Datadog

Comment @DataDog to request changes

Co-authored-by: P403n1x87 <20231758+P403n1x87@users.noreply.github.com>
@datadog-official

Copy link
Copy Markdown

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

@datadog-prod-us1-5

Copy link
Copy Markdown

I can only run on private repositories.

@P403n1x87
P403n1x87 marked this pull request as ready for review August 24, 2026 15:20
@P403n1x87
P403n1x87 requested review from a team as code owners August 24, 2026 15:20
@github-actions

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/debugger/test_debugger_expression_language.py                     @DataDog/debugger @DataDog/system-tests-core

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e53bbfd3b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +21 to +25
if not self.wait_for_all_probes(statuses=["INSTALLED"], timeout=60):
self.setup_failures.append("Probes did not reach INSTALLED status")
# Stop the test if the probes did not reach INSTALLED status since the probe won't exist
# to send a snapshot.
return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Initialize Ruby remote config before waiting for probes

For the enabled Ruby rails72, rails80, and uds-rails variants, this new early return runs before any request can start the Ruby Remote Configuration client. BaseDebuggerTest.initialize_weblog_remote_config() documents that Ruby starts it only from request middleware, but this class never calls that helper; therefore the INSTALLED wait times out, records a setup failure, and every expression-language test fails instead of exercising its probes. Initialize remote config before sending and waiting for probes (which also resets setup_failures per setup).

Useful? React with 👍 / 👎.

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

One probe installation timeout changes a shared failure list. This makes later expression-language tests fail even when their setup succeeds.

Open Bits AI session

🤖 Datadog Autotest · Commit 4e53bbf · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

self.send_rc_probes()
self.wait_for_all_probes(statuses=["INSTALLED"])
if not self.wait_for_all_probes(statuses=["INSTALLED"], timeout=60):
self.setup_failures.append("Probes did not reach INSTALLED status")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Keep the setup failure on one test instance

One rare probe timeout can cause many later expression-language tests to fail and can hide their real results.

Assertion details
  • Input: One expression-language setup times out before all probes reach INSTALLED, and later tests in the same process continue.
  • Expected: The timeout failure must apply only to the test instance whose probes do not install.
  • Actual: The append changes the class-level list. Each later test instance reads the same failure. Its assert_setup_ok() call then fails even when its setup succeeds.
Suggested change
self.setup_failures.append("Probes did not reach INSTALLED status")
self.setup_failures = ["Probes did not reach INSTALLED status"]

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants