Infrastructure: Deliver a reliable Windows Composition screenshot capture path

[JasonVMo]

Summary

Evaluate whether WinAppDriver's takeScreenshot endpoint captures React Native
Windows Composition content on an interactive, unlocked desktop, and if it does
not, implement a host-side desktop capture fallback so that every Storybook
desktop run can attach visual evidence to its artifact bundle. Screenshots must
be labeled as non-authoritative evidence and never used as a pass/fail gate.

The limitation is documented in two places today - the Storybook app README and
its AGENTS guidance - and the test-driver branch records the capture question as
an open decision that explicitly gates any screenshot-based Windows visual gate.

Goal

Produce a reliable per-story screenshot for the Windows Fabric Storybook endpoint
that can be attached to a run artifact, uploaded in CI, and reviewed by a human
without being treated as a deterministic assertion. Do this by first measuring
whether WinAppDriver screenshots capture Composition content on a real, unlocked
Windows session; if they do not, implement a host-side fallback invoked by the
desktop-driver artifact pipeline and document the result.

Stage

Stage 2 (beta) for the real-platform evaluation; Stage 3 (production) for the
fallback implementation, CI integration, and artifact-policy documentation.

Observed. The Storybook app README states: "WinAppDriver 1.2.1 can attach to
this WinAppSDK window and inspect its UI Automation tree, but its screenshot
endpoint does not reliably capture React Native Windows Composition content.
Agents that have a desktop screenshot tool should use it after selecting a story
with storybook:control; UI Automation remains the deterministic automated
validation gate."
(apps/storybook/README.md)

Observed. The Storybook AGENTS guidance echoes this directly: "WinAppDriver
screenshots are not a reliable capture path for WinAppSDK Composition content.
After selecting a story with storybook:control, use the agent host's desktop
screenshot tool when visual evidence is required."
(apps/storybook/AGENTS.md)

Observed. The test-driver branch DESIGN.md states: "WebDriver screenshots may
not reliably capture all WinAppSDK Composition content. Visual evidence requires
separate real-platform verification." Its open-decisions table records: "Is a
Windows Graphics Capture fallback required? Decide after inspecting unlocked
real-app screenshots containing Composition content."
(8f971021 DESIGN.md)

Observed. The branch's NEXT-STEPS.md Priority 0 list explicitly includes
"inspect an unlocked WinAppDriver screenshot for WinAppSDK Composition content"
as an outstanding verification item.
(8f971021 NEXT-STEPS.md)

Why it matters

Inferred. The Storybook E2E pipeline (see
storybook-e2e.md) explicitly classifies screenshots as
optional visual evidence, never as a gate. That classification is correct and
must be preserved. However, "optional evidence" still requires evidence that is
actually useful. A blank or black screenshot attached to a CI run is technically
captured but provides no diagnostic value for human review.

Observed. The existing E2E suite at apps/E2E
captures per-failure screenshots to errorShots/ in all three desktop WDIO
configs:

• Windows: apps/E2E/wdio.conf.windows.js saves to ./errorShots/<name>.png on
  test failure
  (apps/E2E/wdio.conf.windows.js)
• Win32: apps/E2E/wdio.conf.win32.js saves to ./errorShots/ScreenshotOfFailure.png
  (apps/E2E/wdio.conf.win32.js)
• macOS: apps/E2E/wdio.conf.macos.js saves to ./errorShots/<name>.png
  (apps/E2E/wdio.conf.macos.js)

Observed. The apps/E2E suite targets the FluentTester apps, not the Storybook
app, and the FluentTester uses WPF-based rendering under Win32, not WinAppSDK
Composition. Inferred: the screenshot behavior that works for FluentTester may
not transfer to the Composition-rendered Storybook app; the existing errorShots
precedent shows the intent and upload pattern but does not confirm whether
WinAppDriver screenshots capture Composition content.

Observed. The test-driver branch scopes out "visual-diff approval
infrastructure" explicitly. The present task is narrower: it is about whether a
single per-story screenshot captured at the end of a test is informative enough
for human triage - not about image diffing, baselines, or approval workflows.

Observed. CI already uploads apps/E2E/errorShots as artifacts in the Windows
and Win32 jobs via actions/upload-artifact
(.github/workflows/pr.yml).
The pattern is therefore established; the open question is whether the same
mechanism produces useful output for the Composition window.

Inferred. Without resolving this, "screenshots are attached as evidence" in the
Storybook E2E pipeline means attaching evidence that may be blank. Closing the
open decision removes a documented gap from the release gate.

Observed current state

The capture limitation

Observed. WinAppDriver 1.2.1 implements the WebDriver GET /session/{id}/screenshot
endpoint. On WPF-hosted applications (FluentTester / Win32), GDI-based screen
capture works as expected and the errorShots pattern produces usable images.

Observed. React Native Windows (Fabric/New Architecture) renders through
WinAppSDK Composition: the visual tree is hosted in a Microsoft.UI.Composition
layer that may not be captured by GDI-based screen grabs, depending on the
capture API and whether the window is in the foreground on an unlocked desktop.

Inferred. The current statement "does not reliably capture" is based on agent
observation during automated sessions, which may include locked or partially
interactive sessions. The open decision explicitly defers judgment until an
unlocked interactive run is observed. It is therefore possible that WinAppDriver
screenshots work on a real, fully unlocked, foreground desktop.

Fallback surface

Observed. The test-driver branch DESIGN.md documents captureArtifacts() in
the driver's ArtifactManifest interface and describes screenshot.png as one of
the per-test artifact files.
(8f971021 DESIGN.md)

Observed. The AGENTS guidance in the Storybook app already names the
mitigation: "use the agent host's desktop screenshot tool when visual evidence is
required." On Windows, the Windows.Graphics.Capture API (available from Windows 10
SDK 1803) captures DirectX/Composition surfaces and does not depend on GDI.
PowerShell 7 exposes this through WinRT interop or through available tooling.

Observed. The test-driver branch does not include a host-side capture
implementation and leaves the decision open.

Existing E2E screenshot artifact policy

Observed. The existing apps/E2E suites capture screenshots only on failure
(afterTest hook, when test.passed === false). This is a failure-evidence
pattern, not a full-sweep pattern.

Observed. apps/storybook/.gitignore ignores artifacts/ and dist/.
(apps/storybook/.gitignore)

Inferred. The Storybook E2E artifact contract (from
storybook-e2e.md) should follow the same rule: screenshots
captured into artifacts/windows/ under .gitignore, never committed.

Scope

In scope

1. Real-platform evaluation. On a real, fully unlocked, interactive Windows
   desktop session with WinAppDriver 1.2.1 running against the Storybook
   Composition window, capture GET /session/{id}/screenshot after selecting at
   least three different stories and record whether the returned PNG contains
   Composition-rendered content. This closes the open decision documented in the
   test-driver branch.

2. Fallback implementation (conditional on evaluation). If WinAppDriver
   screenshots do not produce useful images, implement a host-side fallback:

   • A lightweight Node.js or PowerShell script invoked by the desktop-driver
     captureArtifacts() hook that captures the foreground Composition window
     to a file using the Windows.Graphics.Capture API (or confirms that GDI
     BitBlt capture works correctly for this Composition window on the target
     OS version).
   • The fallback must: capture only the identified window (not the whole
     desktop), write the result to the configured artifact directory, and
     complete without error when the window is not foreground (producing a
     partial capture with a logged warning, not a thrown exception).
   • The implementation is conditioned on the Storybook driver being the calling
     context; it must not be invoked directly from CI workflow YAML.

3. Artifact labeling. All captured screenshots must be written into the run
   artifact bundle as non-authoritative evidence. The WDIO config and run output
   must include a "screenshotSource": "winappdriver" or "screenshotSource": "host-capture" field so reviewers know which mechanism produced each file.
   Neither mechanism is ever part of a deterministic pass/fail gate.

4. Locked-session detection. The CI job must detect and report a locked or
   non-interactive session before attempting capture, producing a distinguishable
   infrastructure error rather than silently attaching blank screenshots. This
   reinforces the locked-session detection requirement already stated in
   storybook-e2e.md.

5. Privacy and redaction. The capture tool must crop or replace any detected
   system tray, taskbar, notification area, or out-of-app content that appears
   within the captured rectangle if the captured region extends beyond the
   Storybook window. Captures containing only the app window are preferred and
   meet the requirement without additional processing.

6. Documentation. Update apps/storybook/README.md to replace the current
   open-ended "WinAppDriver screenshots do not reliably capture Composition
   content" note with a resolved statement: either "WinAppDriver screenshots work
   on an unlocked session and are the capture path" or "WinAppDriver screenshots
   do not capture Composition content; host-side capture is used instead, see
   [link to implementation]."

Out of scope

• Image diffing, baseline management, pixel-comparison gates, or any
  visual-regression service. Screenshots are evidence only; no diff gate is
  introduced by this task.
• macOS screenshot capture; Mac2 exposes a working takeScreenshot route through
  XCTest and this limitation is Windows-specific.
• Win32 / Paper screenshot capture; Win32 uses WPF rendering (not Composition) and
  existing WinAppDriver screenshots are expected to work there.
• Capturing any window other than the identified Storybook application window.
• Replacing or retiring the existing apps/E2E errorShots mechanism.
• Storing screenshots in source control or a shared image store.

Deliverables

1. A recorded evaluation of WinAppDriver 1.2.1 screenshot output against the
   Composition-rendered Storybook window on a real, unlocked, interactive
   desktop, with at least three story captures and a clear determination:
   captures are usable / captures are blank or missing content.
2. If the evaluation shows WinAppDriver screenshots are not usable: a host-side
   capture script (Node.js or PowerShell) integrated with captureArtifacts()
   in @fluentui-react-native/desktop-driver, with unit-testable logic for the
   window-targeting and artifact-write steps.
3. screenshotSource metadata in every per-test artifact entry so CI runs are
   self-documenting about capture provenance.
4. Locked-session detection that emits a named infrastructure error and a warning
   log line before any capture attempt.
5. Updated apps/storybook/README.md replacing the open-ended statement with the
   resolved capture approach.
6. Updated apps/storybook/AGENTS.md guidance reflecting the resolved approach
   so agents do not fall back to an external desktop tool unnecessarily if
   WinAppDriver works.

Acceptance criteria

• The evaluation records at least three WinAppDriver screenshots taken on an
  unlocked interactive Windows session, and the result commits a note stating
  whether Composition content is visible.
• If WinAppDriver screenshots are not usable, a host-side fallback script
  exists, is invoked by captureArtifacts(), and captures a file containing
  Composition-rendered content on an unlocked session.
• Every captured screenshot carries a screenshotSource field in its
  artifact entry JSON.
• A locked-session check runs before any capture attempt and emits an
  infrastructure error with an unambiguous message (not a test failure).
• No screenshot is ever a deterministic test gate; existing gate criteria in
  storybook-e2e.md are unmodified.
• Captured screenshots are written to artifacts/windows/ (already
  git-ignored) and are never committed.
• The capture tool writes only to the artifact directory and does not shell
  out to commands constructed from story IDs, session handles, or other
  test-runtime values.
• apps/storybook/README.md and apps/storybook/AGENTS.md reflect the
  resolved capture path rather than the open-ended limitation note.
• yarn lage test-links passes for all modified documentation.

Dependencies and ordering

• Depends on: test-driver.md - the captureArtifacts()
  interface and artifact directory contract must be in place before a fallback
  implementation can be integrated.
• Depends on: test-driver-release-readiness.md -
  the Priority 0 real-platform evaluation in that task already includes "inspect
  an unlocked WinAppDriver screenshot for WinAppSDK Composition content." The
  present task extends that single measurement into a systematic evaluation and
  may share the same hardware session.
• Precedes: storybook-e2e.md phase 3 artifact
  requirements - the production CI pipeline's visual-evidence section depends
  on a resolved capture path. Phases 0-2 can proceed with screenshots marked
  as potentially blank.
• Independent of: macOS, Win32, and bundle-size tasks; those have no
  dependency on Windows Composition capture.

Risks and open decisions

Item	Evidence	Risk or decision
WinAppDriver may work on a real unlocked session	Inferred from: current guidance was written during automated / partially interactive sessions; the open decision in NEXT-STEPS.md defers to unlocked real-app observation	If WinAppDriver works, the fallback implementation is unnecessary and this task closes with documentation only.
Windows.Graphics.Capture API availability	Inferred: available from Windows 10 SDK 1803; windows-latest GitHub runner is Windows Server 2022	Verify SDK version in CI environment; PowerShell 7 can access WinRT types via interop.
Capture extending beyond the app window	Inferred: CI runner desktops may show system content outside the app frame	Constrain capture rectangle to the identified window handle, not the full desktop.
Locked-session silent failures	Observed (8f971021 DESIGN.md): "A locked session can still return source, attributes, and screenshots, making the failure look like an application defect."	Blank or partial captures must never cause a test to report a false product failure.
Image diff gate scope creep	Observed: the storybook-e2e task and the test-driver branch both exclude image-diff gates explicitly	Any PR that introduces pixel-comparison assertions should be blocked as out of scope.

Evidence and references

Retrieved 2026-08-21.

• apps/storybook/README.md -
  "WinAppDriver 1.2.1 can attach to this WinAppSDK window and inspect its UI
  Automation tree, but its screenshot endpoint does not reliably capture React
  Native Windows Composition content."
• apps/storybook/AGENTS.md -
  "WinAppDriver screenshots are not a reliable capture path for WinAppSDK
  Composition content. After selecting a story with storybook:control, use the
  agent host's desktop screenshot tool when visual evidence is required."
• 8f971021 DESIGN.md -
  platform constraints section ("WebDriver screenshots may not reliably capture
  all WinAppSDK Composition content") and open-decisions table ("Is a Windows
  Graphics Capture fallback required?").
• 8f971021 NEXT-STEPS.md -
  Priority 0 item: "inspect an unlocked WinAppDriver screenshot for WinAppSDK
  Composition content."
• apps/E2E/wdio.conf.windows.js,
  apps/E2E/wdio.conf.win32.js,
  apps/E2E/wdio.conf.macos.js -
  per-failure errorShots screenshot pattern; establishes precedent for
  failure-evidence screenshots uploaded as CI artifacts.
• apps/storybook/.gitignore -
  artifacts/ is git-ignored; screenshot artifacts must live there.
• .github/workflows/pr.yml -
  actions/upload-artifact from apps/E2E/reports and apps/E2E/errorShots;
  establishes CI artifact upload pattern.
• storybook-e2e.md - "Optional visual evidence (never a gate)"
  section; screenshot-fidelity risk row; artifact list including screenshot.png.
• test-driver-release-readiness.md -
  Priority 0 verification list includes the WinAppDriver screenshot inspection item.

[JasonVMo]

Status update - 2026-09-03

PR #4306 completed the capture implementation. The shared native
Windows/Win32 provider now uses window-scoped Windows Graphics Capture with
D3D11 readback and WIC PNG encoding, supports element crops, records scale
factor, remains occlusion-independent, and reports minimized or unavailable
capture explicitly. The new driver no longer depends on WinAppDriver, so the
issue's conditional WinAppDriver fallback design is obsolete.

Remaining: qualify window and element capture against real Windows Fabric
and Win32 applications on unlocked interactive desktops at 100%, 150%, and
200% DPI; measure hosted-runner reliability; add explicit per-artifact capture
provenance; verify locked/non-interactive diagnostics before capture; and
confirm artifact confinement, privacy, and documentation against the observed
captures. Screenshots remain optional evidence and never a deterministic gate.