Infrastructure: Add accessibility assertions to Storybook story tests

[JasonVMo]

Summary

Extend the story-test schema and the desktop driver to support deterministic
real-platform assertions on accessibility role, accessible name, interactive
state, and Tab-order keyboard reachability. Define a portable role vocabulary
and a platform normalization table that maps it to Windows UI Automation
ControlType values and macOS XCUI element types. Add inline plan steps for
the initial component cohort, update the fake-backend scene format and the
run.json report to carry assertion results, and establish this as a distinct
gate that is explicitly separate from package-level Jest prop-propagation tests
and from manual screen-reader validation.

Goal

Give every story test the ability to declare that a designated element exposes
a specific role, an accessible name, an enabled or selected state, and
keyboard reachability via Tab-order traversal, and have those assertions fail
deterministically on real desktop platforms when the accessibility tree does not
match the declared contract.

Stage

Stages 2 and 3.

Depends on storybook-e2e.md Phase 1 (documented local
real-platform loop). Accessibility step actions must be expressible in inline
story plans before the Phase 2 interactive CI job lands, so that every story
test added in Phase 1 can carry role and name assertions from the start. The
report and fake-backend changes are pre-requisites for Phase 0 fake-run
coverage of the new steps.

Why it matters

Observed. The current Storybook Windows smoke harness
(apps/storybook/windows-tests/storybook-smoke.test.cjs)
asserts exactly three things about the accessibility tree: testID visibility
(via findElementByTestID and isDisplayed), HasKeyboardFocus after a
click-then-Tab sequence, and one hard-coded XPath selector
//Button[@Name="Open report"] that embeds both a role and a name as a locator
rather than as an explicit assertion. There are no assertions on ControlType
(role), Name (accessible name), IsEnabled, or Toggle.ToggleState for any
agentic component story.

Observed. The test-driver branch's inline story plan vocabulary
(packages/agentic/test-driver/src/types.ts)
defines StoryStepProperty as 'text' | 'value' | 'displayed' | 'enabled' | 'selected' and StoryPlanStep has no expectRole, expectName, or
keyboard-reachability action. The fake-backend element shape DesktopFakeElement
declares role?: string and name?: string fields but no inline plan step can
assert them.

Observed. The legacy apps/E2E suite does assert accessibility role and
name for the legacy V1 components. It reads ControlType and Name via
compareAttribute in BasePage
(apps/E2E/src/common/consts.ts,
apps/E2E/src/CheckboxV1/specs/CheckboxV1.spec.win.ts).
That suite targets apps/fluent-tester, not apps/storybook, and is not
available for agentic components.

Observed. Every agentic component sets accessibilityRole explicitly in
its use<Component> hook (for example, accessibilityRole: 'checkbox' in
packages/agentic/components/src/components/checkbox/useCheckbox.ts,
accessibilityRole: 'switch' in
packages/agentic/components/src/components/switch/useSwitch.ts).
Package-level Jest tests verify prop propagation in the React tree (for example,
expect(root.props.accessibilityRole).toBe('checkbox') in
packages/agentic/components/src/components/checkbox/checkbox.test.tsx)
but run against a mock host and do not prove that the prop reaches the platform
accessibility tree on a real device.

Inferred. A regression where the React Native Windows or macOS bridge
silently drops accessibilityRole or remaps it to the wrong UIA ControlType
would pass every current Jest test and every current story test. Real-platform
assertions on role and name close that gap.

Observed current state

Accessibility attribute models

Observed. On Windows, WinAppDriver exposes the Windows UI Automation tree.
The apps/E2E suite reads attributes by raw UIA property name string:

UIA property string	Mapped from React Native prop	Notes
ControlType	accessibilityRole	Value is ControlType.<Name> string
Name	accessibilityLabel	Falls back to visible text if unset
HasKeyboardFocus	Keyboard focus state	Value is 'True' or 'False'
IsEnabled	disabled prop (inverted)	Value is 'True' or 'False'
IsKeyboardFocusable	Tab-stop eligibility	Value is 'True' or 'False'
Toggle.ToggleState	Checked or toggled state	Value is 'On', 'Off', or 'Indeterminate'
ExpandCollapse.ExpandCollapseState	Expanded or collapsed	Value is 'Expanded' or 'Collapsed'

Observed. The existing smoke harness already proves that
app.findElementByTestID resolves via AutomationId in the UIA tree and
getAttribute('HasKeyboardFocus') reads a UIA property via WinAppDriver. The
same getAttribute call works for ControlType, Name, and IsEnabled, as
demonstrated by the apps/E2E V1 component suite.

Observed. On macOS, the mac2 Appium driver exposes the XCUI element tree.
A node's accessible role is reported as elementType (an integer enum from
XCUIElementType) and its accessible name is reported as label. The existing
apps/E2E macOS config already uses elementType in XPath selectors
(apps/E2E/wdio.conf.macos.js,
line 121: '//*[@title="Fluent Tester" and @elementType=4]'). Tab-order
keyboard reachability is not a static attribute; it requires navigating via
keyboard input and checking that the element receives focus.

Inferred. Because raw attribute names and value formats differ between
Windows and macOS, the inline story plan schema must use portable aliases that
the driver resolves to platform-specific attribute reads.

Portable role vocabulary and platform normalization

Inferred. A portable alias layer is required: story authors should not write
platform strings like 'ControlType.Button' or integer elementType values in
inline plans. The driver should resolve a portable alias to the
platform-specific attribute name and expected value. The following initial
mapping covers the agentic component cohort.

Portable alias	Windows ControlType value	macOS elementType (XCUIElementType)
button	ControlType.Button	41 (XCUIElementTypeButton)
checkbox	ControlType.CheckBox	12 (XCUIElementTypeCheckBox)
radiobutton	ControlType.RadioButton	33 (XCUIElementTypeRadioButton)
tabitem	ControlType.TabItem	39 (XCUIElementTypeTab)
menuitem	ControlType.MenuItem	25 (XCUIElementTypeMenuItem)
listitem	ControlType.ListItem	21 (XCUIElementTypeCell, inside a list)
image	ControlType.Image	22 (XCUIElementTypeImage)
separator	ControlType.Separator	38 (XCUIElementTypeSeparator)
progressbar	ControlType.ProgressBar	32 (XCUIElementTypeProgressIndicator)
text	ControlType.Text	48 (XCUIElementTypeStaticText)

Observed. React Native does not define an official Windows UIA mapping for
accessibilityRole: 'switch'. Inferred. The Windows ControlType for
switch must be verified on a real device before a switch portable alias is
added to the table. Confirming the runtime mapping is a required pre-condition
for adding Switch story assertions to the cohort.

Story plan steps available on the test-driver branch

Observed. The test-driver branch defines StoryPlanStep actions:
expectVisible, expectHidden, expectEnabled, expectDisabled, press,
clearValue, setValue, scrollIntoView, wait, screenshot, and a generic
expect covering properties text, value, displayed, enabled,
selected. Tab-key keyboard navigation is achievable today via a
keys(['\uE004']) call in a spec-file plan (kind: 'spec') but has no inline
step action.

Observed. The fake-backend scene format (DesktopFakeElement) already
carries role and name fields for element modeling, but no inline plan action
can assert them. This means a fake-run cannot exercise the new step actions
without adding fake-backend support in the same change.

Component accessibility coverage in on-device stories

Observed. The existing Windows focus tests prove that eleven agentic
components expose a testID-reachable element and receive keyboard focus after
a click in the components-*--default and components-*--selected stories:
accordion, button, card, checkbox, list-item, listbox-item,
menu-item, radio, switch, tab, tag.

Observed. Seven components have no on-device coverage at all in any current
harness: avatar, badge, divider, input, progress-bar, skeleton,
spinner.

Observed. Package-level Jest tests confirm accessibilityRole prop
propagation for checkbox ('checkbox'), switch ('switch'), radio
('radio'), tab ('tab'), accordion header ('button'), list-item
('button' default), card ('button' for interactive), input
('textbox'), progress-bar ('progressbar'), spinner ('progressbar'),
divider ('separator'), avatar ('image'), tag ('button'). These pass
against a macOS mock host and do not verify the live UIA or XCUI tree.

Scope

Schema changes (story plan and fake backend)

1. Add expectRole to StoryPlanStep:
   { action: 'expectRole'; target: StoryStepTarget; role: string }.
   The role value is a portable alias from the normalization table. The driver
   resolves it to the platform-specific attribute read and expected value before
   executing.

2. Add expectName to StoryPlanStep:
   { action: 'expectName'; target: StoryStepTarget; name: string }.
   On Windows the driver reads Name; on macOS it reads label.

3. Add expectKeyboardReachable to StoryPlanStep:
   { action: 'expectKeyboardReachable'; target: StoryStepTarget }.
   On Windows the driver reads IsKeyboardFocusable and asserts 'True'. On
   macOS the driver sends a Tab keypress from a known start element and asserts
   that the target receives focus. Timeout must be configurable.

4. Add expectNotKeyboardReachable as the negative counterpart. Required to
   assert that non-interactive elements such as decorative icons and static text
   are not Tab stops.

5. Extend DesktopFakeElement.role and DesktopFakeElement.name to be
   assertable by the new inline plan actions in fake-backend runs. The fake
   backend must return a failure result when a declared expectRole or
   expectName step does not match the scene element's fields. This makes
   fake-run coverage of the new actions non-trivial and tests that the schema
   round-trips correctly.

6. Add keyboard_reachable to DesktopFakeElement (boolean, default true
   for elements with a testID that are visible and enabled) so
   expectKeyboardReachable has a fake-backend path.

Platform normalization layer

1. Define the normalization table in a single file in the driver package (for
   example, src/accessibility/roles.ts). Export a function that maps
   (platform, portableAlias) to the platform-specific attribute key and
   expected string value.

2. The normalization file must be the only place in the driver that contains
   platform-specific role strings. No spec or plan author should write
   'ControlType.Button' directly in a story test.

3. Document the normalization table in the driver's README.md or USAGE.md
   so a story author can look up which portable alias to use without reading
   driver source.

4. Verify the Windows switch mapping on a real device before adding switch
   to the table. Record the verified ControlType string as an Observed
   fact in this file once confirmed.

Driver execution changes

1. The Windows backend must read ControlType, Name, and IsKeyboardFocusable
   via getAttribute on the element resolved by findElementByTestID. These
   three properties are already readable in the existing WinAppDriver and UIA
   path as demonstrated by apps/E2E/src/common/consts.ts.

2. The macOS backend must read elementType and label via getAttribute.
   Tab-key reachability must use a Tab keypress and poll for focus, with a
   bounded timeout and a clear error message on timeout.

3. The isFocused portable command already exists in PortableCommand on the
   test-driver branch. expectKeyboardReachable is implemented on top of it
   rather than as a new PortableCommand.

Run report changes

1. Each DesktopTestResult record must carry the observed role, name, and
   IsKeyboardFocusable values (or their macOS equivalents) for the primary
   element under test, regardless of whether an assertion was declared. This
   makes accessibility attribute readings available for post-run analysis without
   requiring story authors to add assertions first.

2. Add an accessibility summary section to run.json:
   roleAssertions.passed, roleAssertions.failed,
   nameAssertions.passed, nameAssertions.failed,
   reachabilityAssertions.passed, reachabilityAssertions.failed.

3. The protocolVersion field in DesktopRunReport must be incremented when
   the new fields are added.

Initial component cohort

The following stories and assertions form the initial cohort. All seven
interactive components already have testID-reachable elements and proven
keyboard-focus behavior in the existing Windows smoke harness.

Story ID	Primary testID	Portable role	Expected name source	Keyboard reachable
components-button--default	agentic-storybook-button-overview-primary	button	accessibilityLabel prop	Yes
components-checkbox--default	agentic-storybook-checkbox	checkbox	Visible label text fallback	Yes
components-radio--default	agentic-storybook-radio	radiobutton	Visible label text fallback	Yes
components-switch--default	agentic-storybook-switch	pending verification	Visible label text fallback	Yes
components-tag--default	agentic-storybook-tag	button	Visible text	Yes
components-tab--selected	agentic-storybook-tab-selected	tabitem	Visible text	Yes
components-accordion--default	accordion-header	button	Visible heading text	Yes

One non-interactive story should be added to prove negative reachability:

Story ID	Primary testID	Portable role	Keyboard reachable
components-divider--default	requires a testID added to the story	separator	No

Inferred. The cohort is intentionally narrow so the schema and normalization
table can be validated on real devices before coverage expands. The remaining
covered components (avatar, badge, card, input, list-item,
listbox-item, menu-item, progress-bar, skeleton, spinner) should
receive assertions in a follow-on pass once the platform mappings are confirmed.

Out of scope

• Package-level Jest prop-propagation tests. Tests such as
  expect(root.props.accessibilityRole).toBe('checkbox') in
  packages/agentic/components/src/components/checkbox/checkbox.test.tsx
  are the responsibility of the Components workstream
  (component-test-strategy.md).
  This task adds on-device story assertions that complement but do not replace
  the Jest pass.

• Manual screen-reader and AT validation. Asserting that NVDA reads the
  correct role announcement, that VoiceOver on macOS reads the accessible name
  through its verbal output channel, or that any assistive technology interacts
  correctly with a component is explicitly out of scope. Real-platform UIA and
  XCUI attribute assertions are a necessary precondition for AT validation but
  are not a substitute for it.

• Keyboard interaction contracts. Tab-order reachability (is the element a
  Tab stop?) is in scope. Correctness of keyboard interactions after focus
  (Space activates a button, arrow keys move between radio buttons) belongs to
  the behavioral story-test layer defined in
  storybook-e2e.md.

• Android and iOS. This task targets desktop platforms (Windows and macOS)
  only. Mobile accessibility validation is a separate concern not addressed by
  any current task in this workstream.

• ARIA or web accessibility. This repository contains no web layer; WCAG
  compliance and ARIA attribute assertions are not applicable here.

• Creating the Storybook E2E pipeline, the CI jobs, or the driver itself.
  Those are in storybook-e2e.md,
  test-driver.md, and
  test-driver-release-readiness.md.

Deliverables

1. expectRole, expectName, expectKeyboardReachable, and
   expectNotKeyboardReachable added to StoryPlanStep in the driver's
   types.ts.
2. A portable role normalization module in the driver package with the full
   initial table and a verified switch entry once confirmed on device.
3. Windows and macOS backend implementations reading ControlType and Name
   and IsKeyboardFocusable (Windows) and elementType and label and
   Tab-nav focus (macOS) in the driver's execution layer.
4. Fake-backend support: expectRole, expectName, and
   expectKeyboardReachable steps resolve against DesktopFakeElement fields
   and produce pass or fail results in fake-run output.
5. run.json accessibility summary section and per-result observed-attribute
   fields, with an incremented protocolVersion.
6. Inline parameters.desktopTest plans for the seven interactive cohort
   stories and one non-interactive divider story, each asserting role, name
   where deterministic, and keyboard reachability.
7. Documentation for the portable role vocabulary and normalization table added
   to the driver's README.md or USAGE.md.
8. Changesets for the driver package's public type surface and report schema.

Acceptance criteria

• expectRole, expectName, expectKeyboardReachable, and
  expectNotKeyboardReachable are valid StoryPlanStep action strings and
  are recognized by the manifest generator.
• A fake-run (desktop:test:fake) of a story with expectRole and
  expectName steps passes when the fake scene's role and name match
  and fails with a clear error when they do not.
• On a real Windows device, expectRole: 'button' on the default Button
  story asserts ControlType.Button from the live UIA tree and passes.
• On a real Windows device, expectRole: 'checkbox' on the default
  Checkbox story asserts ControlType.CheckBox and passes.
• On a real Windows device, expectName on a story element passes when
  the Name UIA attribute matches the declared value.
• On a real Windows device, expectKeyboardReachable passes for all seven
  interactive cohort stories and fails for the divider story.
• On macOS, expectRole: 'button' on the default Button story asserts
  elementType 41 from the live XCUI tree and passes.
• On macOS, expectName on a story element passes when label matches.
• run.json contains an accessibility summary section with counts for
  role, name, and reachability assertions.
• The portable role normalization module is the only location in the driver
  source that contains a ControlType. string or a numeric elementType
  literal.
• yarn lage test-links passes for all modified documentation.
• The switch portable alias is either verified and added to the table with
  an Observed citation, or its story test is filed as pending in the
  acceptance record.

Dependencies and ordering

• Depends on: test-driver.md - the driver package must
  exist on main before the new step actions and normalization module can be
  added to it.
• Depends on: storybook-e2e.md Phase 0 - the story
  manifest generator and fake-run infrastructure must exist before the new step
  actions can be exercised in a non-interactive gate.
• Blocks: Phase 2 of storybook-e2e.md for stories
  that declare accessibility assertions. The interactive CI job should run a
  manifest that already includes role and name steps so the gate proves the full
  assertion surface, not only visibility and focus.
• Cross-workstream: The Components workstream
  (component-test-strategy.md)
  owns the package-level Jest prop-propagation tests. When those tests assert
  accessibilityRole, they assert React tree props against a mock host. This
  task asserts the live platform accessibility tree. Both layers are required;
  neither replaces the other.
• Inferred ordering: Verify the Windows switch mapping first on a real
  device using the existing getAttribute('ControlType') path before committing
  the normalization table. An incorrect alias causes every Switch story test to
  fail spuriously.

Risks and open decisions

• Open decision. The final portable role vocabulary and normalization table
  require real Windows and macOS evidence. Platform-specific strings must not
  leak into story authoring while a guessed mapping is treated as stable.
• Open decision. Keyboard reachability may be asserted from
  IsKeyboardFocusable on Windows, but macOS requires bounded Tab traversal and
  focus polling. The two implementations need one documented semantic contract.
• Risk. Incrementing the report protocol for accessibility results can make
  an older Storybook app and a newer driver incompatible. Version mismatch must
  fail explicitly rather than dropping assertion data.
• Risk. Fake-backend coverage can prove schema and reporting behavior but
  cannot validate platform normalization. Promotion to a required gate depends
  on real endpoint evidence.

Evidence and references

Sources reflect the repository and linked branch as of 2026-08-21.

• apps/storybook/windows-tests/storybook-smoke.test.cjs -
  current Windows on-device assertion surface showing visibility, HasKeyboardFocus,
  and one role-and-name XPath locator.
• apps/E2E/src/common/consts.ts -
  UIA attribute name strings, role constants, and the Attribute enum used in
  the V1 E2E suite.
• apps/E2E/src/CheckboxV1/specs/CheckboxV1.spec.win.ts -
  concrete precedent for ControlType and Name assertions via WinAppDriver.
• apps/E2E/wdio.conf.macos.js -
  mac2 driver capabilities and elementType XPath usage in the macOS E2E config.
• origin/user/jasonvmo/test-driver at
  8f971021:
  packages/agentic/test-driver/src/types.ts -
  current StoryPlanStep, StoryStepProperty, DesktopFakeElement, and
  DesktopRunReport type contracts showing what role and name fields exist
  without corresponding inline plan actions.
• packages/agentic/components/src/components/checkbox/useCheckbox.ts,
  packages/agentic/components/src/components/switch/useSwitch.ts,
  packages/agentic/components/src/components/button/useButton.ts -
  accessibilityRole and accessibilityLabel assignments in agentic component hooks.
• packages/agentic/components/src/components/checkbox/checkbox.test.tsx -
  representative package-level Jest prop-propagation test that this task
  complements but does not replace.

[JasonVMo]

Status update - 2026-09-03

PR #4306 completed the native accessibility foundation: the macOS and shared
Windows/Win32 providers expose normalized accessibility snapshots and support
portable role, accessible-name, state, text, and value lookup/assertion paths.
stories-and-tests now executes component-authored plans through those native
providers. Button, Checkbox, and Input currently provide the first role,
state, focus, and value plans.

The merged V1 schema uses generic
{ expect: { state, target, value } } steps rather than the issue's earlier
proposed expectRole/expectName action spellings; acceptance should be judged
by the portable behavior, not those obsolete draft names.

Remaining: expand beyond the three-story cohort, add deterministic
accessible-name and positive/negative keyboard-reachability coverage, verify
the switch mapping, add accessibility-specific result summaries, and run the
full initial cohort on authoritative interactive Windows and managed macOS
environments. Hosted capability skips are coverage information, not completion.