Skip to content

Flaky: Test Drivers (selenium_headless, chunky_png) — one input row renders differently ~1 run in 5 #285

Description

@pftg

Split out of #284 so it does not sit in the release path.

BrowserScreenshotTest#test_compare_crops_only_when_other_part_is_not_working fails
intermittently on one matrix cell, Test Drivers (selenium_headless, chunky_png):

the change spans 4378 of 16800 px (26.06% of the 200x84 image)
  changed region: [0,19,199,41] (left,top,right,bottom edges)
  max color distance: 50.9
  judged against: tolerance 0.13, color_distance_limit 40

What is established

  • Base rate: 4 success / 1 failure over the last 12 Test runs on that cell. Every
    other Drivers cell is 100%.
  • Not environment drift. The failing job (97473161378) and a passing job at the same
    commit (97450518382) ran on the identical runner image 20260816.277.1, same
    runner version, same Ruby.
  • Not antialiasing. The changed region is 200x22 = 4400 px against 4378 px changed —
    the entire first input row and nothing else — at max colour distance 50.9. That is a
    materially different render of #first-field, not sub-pixel drift.
  • The fixture (test/fixtures/app/index-without-img.html) loads no remote resources;
    the font stack is local with fallbacks, so it is not the usual web-font race.

What is not established

The actual cause. A plausible shape is leftover focus/selection state on #first-field
from the test that does fill_in "First Field:", but that is unverified — and it
cannot be verified from CI artifacts until #284 is fixed, because the failure's own
evidence is overwritten before upload.

Blocked on #284

Fix the shared screenshot name first, then let this flake reproduce once with real
artifacts. Debugging it before that means guessing.

Also worth deciding

fail-fast turns this single flaky cell into three red cells (two siblings report
cancelled), so a run reports far more damage than occurred. Same mechanism hid the JRuby
lane for 15 runs (#283).

Scheduling

Not a 2.0.0 blocker: it predates the release, and chunky_png is deprecated in 2.0 and
removed in 2.1 — so this cell disappears on its own in 2.1. Filed against 2.1.0 so the
deletion is a deliberate outcome rather than an accidental one, and so nobody mistakes the
removal for a fix.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions