test: Upgrade Playwright to 1.63 and fix the Chromium 153 fallout - #24273
Open
logaretm wants to merge 7 commits into
Open
test: Upgrade Playwright to 1.63 and fix the Chromium 153 fallout#24273logaretm wants to merge 7 commits into
logaretm wants to merge 7 commits into
Conversation
Contributor
size-limit report 📦
|
logaretm
force-pushed
the
awad/soft-nav-web-vitals-e2e
branch
from
September 9, 2026 22:59
6b626d2 to
6bf5302
Compare
logaretm
force-pushed
the
awad/soft-nav-web-vitals-e2e
branch
from
September 9, 2026 23:18
6bf5302 to
edec2c5
Compare
logaretm
added this pull request to stack #24274
September 9, 2026 23:18
logaretm
force-pushed
the
awad/soft-nav-web-vitals-e2e
branch
from
September 10, 2026 03:32
edec2c5 to
55c4475
Compare
Playwright 1.56 bundles Chromium 141, which has no Soft Navigations API, so the soft navigation code paths in the web vitals integration never run under test. 1.63 bundles Chromium 153, which supports the API unflagged. `page.route` resolves with a `Disposable` as of 1.63, so returning it from a `Promise<void>` helper no longer type-checks.
…e hiding the page
web-vitals defers processing an interaction's event entries into
`requestIdleCallback(..., { timeout: 1000 })`. Chromium 141 ran that well before the tests
hid the page; Chromium 153 only runs it once the timeout elapses, so hiding first forced a
report while the metric was still unset and no vital was emitted at all.
`hidePage` now waits for an idle callback of its own first. Idle callbacks run in scheduling
order, so one queued at hide time running means web-vitals' earlier callback has too.
CLS lands on the pageload span when it ends on the idle timeout, so nothing has to force the page away to finalize it. The reload raced the envelope instead: on Chromium 153 the page reloads before the in-flight send leaves, so no transaction was ever received and the test timed out. It also looped, since the reloaded page re-ran the same subject.
web-vitals finalizes LCP on the first trusted input or visibility change and checks `isTrusted`, so the synthetic `visibilitychange` this test dispatched never finalized it. It passed anyway while `reportAllChanges` was on, since every LCP entry was reported as it arrived. Soft navigation reporting turns that off, so on a browser that supports the Soft Navigations API no LCP span was emitted at all. Clicking finalizes it the way a real user does.
`page.route` resolves with a `Disposable` as of Playwright 1.63, so returning it from a
`Promise<void>` helper no longer type-checks.
The INP tests in react-17, react-17-static and the react-router 7/8 SPA apps hid the page
500ms after the interaction, which is the same race already fixed in react-router-6:
web-vitals defers entry processing to `requestIdleCallback(..., { timeout: 1000 })` and
Chromium 153 only runs it on the timeout, so the forced report found an unset metric.
solid-tanstack-router's LCP test dispatched a synthetic `visibilitychange`, which web-vitals
ignores for finalization because it checks `isTrusted`. It passed only while
`reportAllChanges` was on. It now finalizes with a real click, as in the browser integration
suite.
Playwright 1.63 ships a WebKit that reports `deliveryType`, so the attribute is no longer Chromium-only and the expectation moves out of the non-WebKit branch. That WebKit also resolves the intercepted routes without measurable delay, so the TTFB range check keeps its upper bound (which is what verifies we report seconds, not milliseconds) but only holds the other engines to a non-zero value.
Two assumptions in the bfcache app stopped holding on newer Chromium, and both traced back to `unload`. An unload listener still makes a page ineligible, but Chrome no longer hands out the `unload-listener` reason for it, only the privacy-masked one. The test waited for both, so it hung on a reason that never arrives. It now asserts the masked reason, which is what the browser actually reports and which older versions reported too. The child-frame test used an unload listener inside the iframe purely as a blocker, so it lost its reason as well. It blocks with a held-up IndexedDB version upgrade instead, mirroring the top-level botcher, which keeps the child-frame classification covered.
logaretm
force-pushed
the
awad/soft-nav-web-vitals-e2e
branch
from
September 10, 2026 18:19
b41854e to
d40223d
Compare
logaretm
changed the title
logaretm
marked this pull request as ready for review
September 10, 2026 19:04
logaretm
requested review from
Lms24 and
msonnb
and removed request for
a team
September 10, 2026 19:04
Lms24
approved these changes
Sep 11, 2026
| // `requestIdleCallback(..., { timeout: 1000 })`. Chromium only runs that on the timeout here, so | ||
| // hiding the page any earlier forces a report before the interaction has been processed and no | ||
| // INP is emitted at all. | ||
| await page.waitForTimeout(1500); |
Member
There was a problem hiding this comment.
l: Probably fine for now but if this flakes I think we need some other mechanism than the timeout (totally fine to leave as-is, since this was already an issue before).
Another question: Below we still trigger a visibilityChange event. Given we removed some of these calls in other tests, do we still want to keep this one? (same pattern in the other react and react-router e2e tests). I guess maybe we still need it for INP?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades Playwright to 1.63 and fixes the existing tests that the newer browser broke. This unblocks being able to test soft navigation web vitals.
Some assertions needed updates due to browser behavior changes that wasn't accounted for:
unloadlistener now has amaskedreason instead which is surprising.