GUI Agent for E2E Testing
AI-powered vision. Cross-platform. Batteries included.
Midscene combines a vision-driven GUI Agent with a testing kit for writing, verifying, and debugging UI tests across web, mobile, and desktop apps through the same Agent APIs. The Playwright example below shows a web test; configure a model and open your app in an existing Playwright page to get started:
import { PlaywrightAgent } from '@midscene/web/playwright';
const agent = new PlaywrightAgent(page);
// Let the Agent carry out a flow, then verify the result.
await agent.aiAct('Search for headphones, then filter the results to under $100');
await agent.aiWaitFor('The filtered search results are displayed');
await agent.aiAssert('Every product in the search results has a price below $100');Open the generated HTML report to inspect screenshots, actions, and assertion results. Follow the Playwright guide for model setup, a complete example, and test runner integration.
Midscene models both UI actions and assertions on how people use software: look at the screen, act on what you see, and check the visible result. Describe the task and expected outcome in natural language; Midscene uses screenshots to decide where to interact and whether the interface meets your expectations.
Like a person finding a control on screen, Midscene locates elements by their appearance and position, then clicks, types, or scrolls to carry out your instructions. You can target icon-only buttons, custom controls, <canvas>, and elements inside cross-origin iframes without writing selectors or adding semantic annotations.
The same Agent APIs work across Web, Android, iOS, HarmonyOS, and desktop apps. You can also connect a custom interface by providing screenshot and action capabilities.
Assertions follow the same visual approach: Midscene inspects the screen as a human tester would to judge whether the expected result is visible. Describe the expected appearance in natural language to check colors, selection highlights, layout, and visual feedback — including content drawn on <canvas> or displayed in native apps.
await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');| Benchmark | Pass@1 | Model used in the reported run |
|---|---|---|
| AndroidWorld | 93.1% | Gemini-3.5-Flash |
| MobileWorld | 78.6% | Gemini-3.6-Flash |
| AppControlBench | 96.7% | Doubao Seed 2.1 Turbo |
Each report includes the run configuration and task results; see AndroidWorld's report for its environment and validator adjustments.
Screenshot-based UI actions avoid sending large DOM trees to the model. In the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo completed the 60-task evaluation with $0.59 in total model cost, achieving 58 passes. The report provides per-task costs and comparisons across models.
Midscene supports multimodal models such as Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. Start with one model, or combine planning and vision models for your workload. For data extraction and page understanding, you can opt in to include DOM. See Model Strategy.
- Web Automation - Automatically register the GitHub form in a web browser and pass all field validations
- iOS Automation - Meituan coffee order
- iOS Automation - Auto-like the first @midscene_ai tweet
- Android Automation - DCar: Xiaomi SU7 specs
- Android Automation - Booking a hotel for Christmas
- robotic arm + vision + voice for in-vehicle testing
Batteries included: Midscene provides the test framework, observability, and integration APIs needed to turn GUI automation into a maintainable E2E test project.
Midscene Test (@midscene/test, Beta) separates declarative test intent from programmable engineering. Write UI flows and expectations in YAML, and wrap API calls, data preparation, and cleanup in reusable TypeScript Nodes. A refund test can prepare an order through an API, request a refund through the UI, and verify the result in one workflow.
The framework includes project scaffolding, platform presets, lifecycle hooks, retries, and execution-project isolation and concurrency. It also generates a Markdown reference from registered Nodes and their parameter schemas, so people and AI Agents can discover the same capabilities and co-maintain test cases. See Create and extend a project and Write and run tests.
Interactive HTML reports show screenshots, element locations, the AI decision process, and action and assertion results. Midscene Test records the inputs, outputs, duration, and status of each AI step and custom business operation. Reports and runtime logs give both developers and AI Agents the context to investigate failures. Use the Playground to try and refine instructions against your interface.
Use aiAct for autonomous flows, aiTap and aiInput for individual actions, aiAssert for assertions, and aiQuery for structured data extraction. Combine these Agent APIs with your existing code, fixtures, and assertions through Playwright, Puppeteer, or the JavaScript SDK. You can adopt Midscene's visual capabilities within your current testing framework. AI coding agents can also operate interfaces through Midscene Skills.
- Try Midscene in the Playground — experiment with natural-language actions, data extraction, and visual assertions before writing scripts. Start with the Chrome extension, or launch a Playground for mobile or desktop.
- Write tests with the SDK or YAML — start with Playwright, Puppeteer, or the Midscene Test.
- Let your AI agent operate the UI — install Midscene Skills.
- Test on another platform — follow the guides for Android, iOS, HarmonyOS, or desktop.
- Documentation: https://midscenejs.com
- Sample projects: midscene-example
- API reference: https://midscenejs.com/reference/#common
Community projects that extend Midscene.js capabilities:
- midscene-ios - iOS Mirror automation support for Midscene
- midscene-pc - PC operation device for Windows, macOS, and Linux
- midscene-pc-docker - Docker image with Midscene-PC server pre-installed
- Midscene-Python - Python SDK for Midscene automation
- midscene-java by @Master-Frank - Java SDK for Midscene automation
- midscene-java by @alstafeev - Java SDK for Midscene automation
We would like to thank the following projects:
- Rsbuild and Rslib for the build tools.
- UI-TARS for the open-source agent model UI-TARS.
- Qwen-VL for the open-source multimodal model Qwen-VL.
- scrcpy and yume-chan for browser-based Android device control.
- appium-adb for its JavaScript bridge to ADB.
- appium-webdriveragent for controlling XCTest from JavaScript.
- YADB for improving text input performance.
- libnut-core for cross-platform native keyboard and mouse control.
- Puppeteer for browser automation and control.
- Playwright for browser automation, control, and testing.
If you use Midscene.js in your research or project, please cite:
@software{Midscene.js,
author = {Xiao Zhou, Tao Yu, YiBing Lin},
title = {Midscene.js: GUI Agent for E2E Testing.},
year = {2025},
publisher = {GitHub},
url = {https://github.com/web-infra-dev/midscene}
}Midscene.js is MIT licensed.