Skip to content

Infrastructure: Add Storybook end-to-end test pipelines #4256

Open
Open
Infrastructure: Add Storybook end-to-end test pipelines#4256

Description

@JasonVMo
Jason Morse (JasonVMo)
opened on Aug 21, 2026

Summary

Turn the on-device Storybook app into a deterministic, portable test surface.
Define a story-test authoring convention, run the generated suite locally on
each desktop endpoint, and phase CI from non-interactive checks to required
macOS, Windows Fabric, and Win32 gates with truthful results and published
artifacts.

The app already exposes a story index, control channel, stable selectors, a
Windows smoke harness, and a portable driver implementation on the test-driver
branch. This task connects those pieces into the pipeline.

Goal

Give apps/storybook end-to-end test pipelines: a documented authoring convention
for story tests, deterministic locally reproducible suites per desktop endpoint, and
a phased set of CI jobs that start non-interactive and non-required and end as
required interactive desktop gates with published artifacts and an explicit flake
policy.

Stage

Stage 2 (beta) for phases 0-2; Stage 3 (production) for phases 3-4.

The Storybook app already has every ingredient except the pipeline: a story index, a
control channel, stable testID selectors, a Windows automation harness, and (on the
test-driver branch) a portable driver and a story-test manifest format. Observed:
no CI workflow in this repository mentions apps/storybook at all. This task builds
the pipeline around those ingredients and defines what "green" is allowed to mean at
each phase.

Why it matters

Observed. The agentic component library has 23 *.stories.tsx files under
packages/agentic/components/src plus Callout.stories.tsx. The only automated
on-device assertions live in a single Windows-only harness that is not run by
CI: three smoke stories
(apps/storybook/scripts/smoke-stories.json)
plus eleven focus tests added by
4777e54f
(PR #4208) as
regression coverage for a React Native Windows Fabric focus crash.

Inferred. That commit is the sharpest available argument for this task: a crash
that terminated the application on click was caught by an on-device harness that no
pipeline runs, so nothing would have prevented it from reaching main in the first
place, and nothing prevents its recurrence today.

Observed. The existing repository-wide E2E suite (apps/E2E)
targets FluentTester, not Storybook, and requires per-platform spec files, so it does
not extend to the agentic library without duplicating tests per platform.

Inferred. Beta readiness claims for the agentic components rest on Storybook
being the demonstration surface. If Storybook is not itself tested in CI, a
regression in a component, in the theme header, in the story index, or in the native
host reaches consumers undetected.

Observed current state

What exists

Capability Evidence State
Story index and control REST/WebSocket/MCP server apps/storybook/storybook-server.cjs, GET /index.json, POST /select-story-sync/<storyId>, POST /send-event on 127.0.0.1:7007 Working, used by agents
Control CLI apps/storybook/scripts/storybook-control.cjs - list, select, args, smoke Working
Windows on-device assertions apps/storybook/windows-tests/storybook-smoke.test.cjs via jest.windows.config.cjs, testEnvironment: '@react-native-windows/automation', WinAppDriver 1.2.1 3 smoke stories plus 11 focus tests (1 focus-movement, 10 focus-crash regression) added by 4777e54f; JSON artifacts under artifacts/windows; run manually only
Windows agent session orchestration windows:agent, windows:agent:start, windows:agent:stop, artifacts/windows/agent-session.json Working, records exact PIDs
Bundle-only validation (no native toolchain) bundle:macos -> dist/index.macos.jsbundle, bundle:windows -> dist/index.windows.bundle Working
Declared Windows CI build chain windows:ci: yarn bundle:windows && yarn windows:generate && yarn windows:build Observed: declared in package.json but referenced by no workflow
macOS on-device assertions - Observed: none; apps/storybook/scripts/ contains only Windows scripts and the Storybook control CLIs
Portable story tests test-driver branch: parameters.desktopTest, desktop-driver stories generate, wdio.conf.ts, desktop:test:* Not on main; see test-driver.md
Visual regression - Observed: none anywhere in the repository; screenshots exist only as failure evidence in apps/E2E/errorShots

CI shape available today

Observed from .github/workflows/pr.yml:

  • JS PR on ubuntu-latest runs yarn lage buildci.
  • macOS PR on macos-26 does yarn build, yarn bundle:macos, pod install,
    xcodebuild, then yarn e2eprep:macos and yarn e2etest:macos for
    apps/fluent-tester, and uploads apps/E2E/reports and apps/E2E/errorShots.
  • Windows PR and Win32 PR on windows-latest install WinAppDriver 1.1 with
    msiexec, build/bundle, and run the corresponding E2E suites, uploading the same
    artifact paths.
  • Toolchain setup uses
    microsoft/react-native-test-app/.github/actions/setup-toolchain pinned by SHA,
    with node-version: 22.

Inferred. The interactive-desktop question is therefore already answered in
practice for Windows: GitHub-hosted windows-latest runners currently execute
WinAppDriver-based UI automation for apps/E2E in this repository. Observed
caveat: the desktop driver documentation states Windows clicks use synthetic input
and fail on a locked workstation, and that a locked session still returns source and
attributes - so a Storybook job must detect and report that condition rather than
assume the runner is interactive.

Deterministic versus visual

Observed constraints that force this distinction:

  • "WinAppDriver 1.2.1 can attach to this WinAppSDK window and inspect its UI
    Automation tree, but its screenshot endpoint does not reliably capture React
    Native Windows Composition content"
    (apps/storybook/README.md).
  • apps/storybook/AGENTS.md: "WinAppDriver screenshots are not a reliable capture
    path for WinAppSDK Composition content... use the agent host's desktop screenshot
    tool when visual evidence is required."
  • The test-driver branch excludes "visual-diff approval infrastructure" from the
    package's scope and leaves a Windows Graphics Capture fallback as an open
    decision.
  • No baseline image store, approval workflow, or diffing tool exists in the
    repository.

Inferred. Screenshot comparison cannot be a pass/fail gate on Windows today.
Visual output must be treated as evidence attached to a run, not as an assertion.

Scope

Deterministic gates (pass/fail)

These are the only checks allowed to fail a build:

  1. Story index integrity - the generated story index is non-empty and matches
    the expected set of story IDs across endpoints.
  2. Manifest validity and digest - desktop-driver stories generate succeeds,
    and the digest recorded by each endpoint's run matches.
  3. Bundle gates - bundle:macos, bundle:windows (and, when it exists,
    bundle:win32) succeed with duplicate-module enforcement intact.
  4. Fake-backend contract run - the full generated suite runs against the fake
    backend with no GUI.
  5. Native build gates - windows:ci (and the macOS macos:build) complete.
  6. Real-endpoint story tests - every story with parameters.desktopTest passes
    its inline plan or linked spec on the endpoint under test: element existence,
    enabled/selected state, text and value, focus, and interaction outcomes read from
    the accessibility tree.
  7. Readiness and lifecycle truthfulness - the run reports a terminal state, and
    an application crash or driver-host failure cannot report success.

Optional visual evidence (never a gate)

  • Per-test screenshots and source.xml captured on failure into the run artifact
    directory.
  • Optional full-story screenshot sweeps captured by a host-side desktop capture tool
    where available.
  • Uploaded as CI artifacts for human review, explicitly labeled non-authoritative,
    and subject to the redaction and review rules below.

Phased coverage

Phase 0 - authoring and non-interactive gate (local + ubuntu-latest).
Document the story-test authoring convention next to the existing story authoring
instructions. Add story tests for a first cohort of components beyond Button. Add
a CI step that runs desktop:generate, validates the manifest, runs
desktop:test:fake, and runs bundle:macos / bundle:windows. Inferred: this
belongs in or beside the existing JS PR job because it needs no native toolchain
and no GUI.

Phase 1 - documented local real-platform runs. Make the full local loop
reproducible and documented for macOS and Windows Fabric: start the channel server,
build/launch the app, generate, run, read artifacts, stop the session. Reuse the
existing windows:agent session model for process ownership. Expand story-test
coverage to the primitives and the components used by the current smoke set.

Phase 2 - first interactive CI job, non-required. Add a Windows Fabric job on
windows-latest modeled on the existing Windows PR job: toolchain setup,
yarn build, windows:ci, WinAppDriver install, register/launch, channel server,
run the desktop suite, always-upload artifacts, and a bounded timeout. Keep it
non-required while flake rate is measured.

Phase 3 - required gates per endpoint. Promote the Windows Fabric job to
required once its flake rate meets the policy threshold; add the macOS job on
macos-26; add the Win32 job once
win32-storybook.md delivers the endpoint. All three run the
same manifest and must report the same digest.

Phase 4 - production operations. Flake policy and quarantine mechanism, artifact
retention and redaction rules, runner-capacity ownership, and a decision on the
visual-evidence pipeline and on retiring the legacy Windows Jest smoke harness.

Cross-cutting concerns to resolve in this task

  • Native CI requirements. macOS needs Xcode, CocoaPods, and a WebDriverAgentMac
    build cache; Windows needs Visual Studio 2022 with RNW prerequisites plus
    WinAppDriver. Observed: the repository already installs the Windows
    prerequisites for apps/E2E and already runs pod install + xcodebuild for
    macOS, so the incremental cost is the Storybook app's own generate/build steps.
  • Interactive desktop constraints. The job must fail with a distinguishable
    infrastructure error when the session is locked or non-interactive, rather than
    reporting a product failure.
  • Artifacts. Publish run.json, events.ndjson, junit.xml, ownership.json,
    driver-host.log, per-test result.json/source.xml/screenshot.png, and
    native build logs from artifacts/windows/build-logs. Observed: artifact
    directories are already git-ignored (apps/storybook/.gitignore ignores
    artifacts/ and dist/).
  • Flake policy. Define a retry budget per spec, a quarantine list with an expiry
    date and an owner, and a rule that quarantined coverage does not count toward exit
    criteria. Observed precedent: apps/E2E/wdio.conf.win32.js sets
    specFileRetries: 1 and bail: 1; the Storybook suite should choose its policy
    deliberately rather than inherit it.
  • Security and lifecycle. Keep the run service loopback-only; never expose the
    channel server or run service to a non-loopback interface in CI; use attach mode
    for interactive sessions and launch mode only where the job owns the app; stop
    exactly the recorded PIDs. Observed: artifacts "can contain private
    application content" and must be reviewed before sharing.

Out of scope

  • Building the driver itself - see test-driver.md and
    test-driver-release-readiness.md.
  • Creating the Win32 endpoint - see win32-storybook.md; this
    task only adds its CI job once the endpoint exists.
  • Migrating apps/E2E or the FluentTester suites.
  • Image-diff baselines, approval workflows, or a hosted visual-review service; see
    Suggested Additions in the workstream README.
  • Component-level unit and snapshot tests, which the Components workstream owns.

Deliverables

  1. Story-test authoring documentation in apps/storybook/README.md and
    packages/agentic/components story guidance, including the deterministic-versus-
    evidence rule.
  2. Story tests for a documented cohort of components and primitives beyond Button.
  3. A non-interactive CI step (generate + validate + fake run + bundles) on every PR.
  4. A Windows Fabric Storybook CI job, initially non-required, with artifact upload
    and a bounded timeout.
  5. A macOS Storybook CI job with the same shape.
  6. A written flake and quarantine policy, and an artifact retention/redaction policy.
  7. A decision record on retaining or retiring the
    @react-native-windows/automation smoke harness, including the port/command
    conflict with desktop:service and how its focus-crash regression coverage is
    preserved.

Acceptance criteria

  • apps/storybook documents how to add a story test, which assertions are
    deterministic gates, and that screenshots are evidence rather than assertions.
  • At least eight stories across at least five components/primitives declare
    parameters.desktopTest and pass locally on one real endpoint.
  • A PR-triggered CI step runs manifest generation, manifest validation,
    desktop:test:fake, bundle:macos, and bundle:windows, and fails the build
    on any of them.
  • A Windows Fabric CI job builds the Storybook app with windows:ci, launches
    it, runs the real desktop suite, and uploads run.json, junit.xml,
    events.ndjson, ownership.json, and failure screenshots with
    if: always().
  • The Windows job reports a distinguishable infrastructure error - not a test
    failure - when the desktop session is locked or non-interactive.
  • The job stops every process it started, verified from ownership.json or the
    recorded session manifest, and leaves no orphaned Metro, channel server,
    WinAppDriver, or app process.
  • A macOS CI job runs the same manifest and reports the same manifest digest as
    the Windows job.
  • Flake rate for each required job is measured over a documented window and meets
    the written threshold before promotion to required.
  • A quarantine mechanism exists, quarantined tests are listed with owner and
    expiry, and quarantined stories are excluded from coverage claims.
  • No CI job binds the Storybook channel server or the desktop run service to a
    non-loopback address.
  • yarn lage test-links passes for all modified documentation.
  • The focus-crash regression coverage added by
    4777e54f
    is either still executed by a CI job or ported to desktop story tests before
    the legacy Windows harness is retired.

Dependencies and ordering

  • Depends on: test-driver.md for phases 0-2;
    test-driver-release-readiness.md before any
    job becomes required; win32-storybook.md for the Win32 job
    in phase 3.
  • Cross-workstream: the Components workstream's "Component Tests" task should
    define which behaviors belong in Jest component tests versus desktop story tests,
    so the same assertion is not paid for twice.
  • Ordering (Inferred): non-interactive gate first (cheap, safe, always
    required), then one interactive endpoint proven end to end, then replication.
    Adding three interactive jobs simultaneously would make flake attribution
    impossible.

Risks and open decisions

Item Evidence Risk or decision
CI time and runner cost Observed: pr.yml already runs six platform jobs with 60-minute timeouts; the Storybook app adds another native build per platform. Decide whether Storybook jobs run on every PR, on a path filter, or on a schedule.
Interactive session assumptions Observed: RNW clicks use synthetic input and fail on a locked workstation; a locked session still returns source and attributes. Without explicit detection, a locked runner produces misleading failures.
Screenshot fidelity on Windows Observed: WinAppDriver cannot reliably capture WinAppSDK Composition content. Blocks any Windows visual gate; also weakens screenshot-based triage.
Two competing Windows harnesses Observed: windows:test (Jest + @react-native-windows/automation, WinAppDriver 1.2.1) and the desktop-driver path (WinAppDriver via appium-windows-driver) "must not be run at the same time". Observed: windows:test currently carries the only regression coverage for the focus crash fixed in 4777e54f. Decide retention or retirement before both are wired into CI; retiring it without porting that coverage would lose a proven regression guard.
WinAppDriver version skew Observed: CI installs 1.1; the Storybook harness requires 1.2.1. The Storybook job must install and pin its own version explicitly.
Release-build limitation on Windows Observed: the Storybook color-picker image "is intentionally not packaged because the Yarn pnpm asset path exceeds Windows' deployment path limit". Any offline/Release-based CI job must account for the missing asset.
Flake budget definition Observed: no existing flake policy in the repository; apps/E2E uses specFileRetries: 1 and bail: 1. Retries hide real intermittent product bugs if the budget is set without a quarantine and review loop.
Coverage source of truth Observed: smoke-stories.json and the desktop manifest are two independent story allowlists. Consolidate on the manifest, or document why both exist.

Evidence and references

Retrieved 2026-08-20.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions