web-infra-dev/midscene

See interfaces like a human. E2E testing in natural language.
A vision-based GUI Agent and an extensible testing framework for web, mobile, and desktop apps.

Website · English / 简体中文

Midscene.js helps developers bring AI to E2E testing. Describe UI steps and expected results in natural language, then use the visual GUI Agent to carry them out and verify what users see. Organize tests with Midscene Test, or integrate the Agent APIs into your existing testing framework.

After creating a Midscene Test project and configuring a model and the Web runtime, write a case like this. Replace the example URL with your app's address:

beforeEach:
  - gotoUrl: https://your-shop.example

cases:
  - name: Find headphones under $100
    steps:
      - aiAct: Search for headphones and filter the results to prices under $100
      - aiWaitFor: The filtered search results are visible
      - aiAssert: There are products in the search results, and each costs less than $100

You can also use the Agent directly in your existing tests. Configure a model and open your app in a Playwright page, then describe the same steps in code:

import { PlaywrightAgent } from '@midscene/web/playwright';

const agent = new PlaywrightAgent(page);

await agent.aiAct('Search for headphones and filter the results to prices under $100');
await agent.aiWaitFor('The filtered search results are visible');
await agent.aiAssert('There are products in the search results, and each costs less than $100');

Open the generated HTML report to inspect screenshots, actions, and assertion results. Follow the guides for Midscene Test or Playwright for setup and complete examples.

Midscene models UI actions and assertions on how people use software: look at the screen, act on what you see, and check the visible result.

Midscene locates elements by their appearance and position, then clicks, types, or scrolls to carry out your instructions. You can target icon-only buttons, custom controls, <canvas>, and elements inside cross-origin iframes without maintaining selectors or adding semantic annotations.

Describe the expected appearance in natural language to check colors, selection highlights, layout, and visual feedback — including content drawn on <canvas> or displayed in native apps.

await agent.aiAssert('The selected plan has a blue border and a checkmark');
await agent.aiAssert('The error message is visible below the email field');

Screenshot-based UI actions avoid sending large DOM trees to the model. In the reported AppControlBench run, Midscene with Doubao Seed 2.1 Turbo passed 58 of 60 tasks with $0.59 in total model cost. The report provides per-task costs and comparisons across models.

Midscene supports Qwen3.x, Doubao-Seed-2.1, DeepSeek V4 Flash, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. See Model Strategy to choose a model and Model Configuration to set it up.

Use the same Agent APIs to describe actions and assertions across platforms:

Midscene Test (@midscene/test, Beta) turns GUI automation into a maintainable E2E test project:

• Describe test cases in YAML. Combine UI actions, assertions, and business operations to express the flow and expected results.
• Reuse business capabilities through TypeScript Nodes. Wrap data preparation, API calls, and cleanup in nodes that different cases can share. A refund test can prepare an order through an API, request a refund through the UI, and verify the result in one workflow.
• Generate a Node Spec for developers and AI Agents. Export registered nodes and their parameter schemas as Markdown, so both can discover project capabilities and co-maintain test cases.

Start with Create and use test projects, then learn how to Develop custom Nodes.

Use aiAct for autonomous flows, aiTap and aiInput for individual actions, aiAssert for assertions, and aiQuery for structured data extraction. Combine these Agent APIs with your existing code, fixtures, and assertions through Playwright, Puppeteer, or the JavaScript SDK.

Interactive HTML reports show screenshots, element locations, the AI decision process, and action and assertion results. Midscene Test also records the inputs, outputs, duration, and status of custom business operations, giving developers and AI Agents context to investigate failures. Use the Playground to try and refine instructions against your interface.

Midscene Skills lets AI coding agents operate applications, verify interfaces, and inspect execution results.

Each report includes the run configuration and task results; see AndroidWorld's report for its environment and validator adjustments.

Explore cross-platform showcases, from web form validation to mobile workflows and community-built in-vehicle testing.

• Write YAML test cases — create a Midscene Test project.
• Add visual capabilities to existing tests — integrate with Playwright or Puppeteer.
• Try it before writing tests — use the Chrome extension, or launch a Playground for mobile or desktop.
• Let your AI agent operate the UI — install Midscene Skills.

• Documentation: https://midscenejs.com
• Sample projects: midscene-example
• API reference: https://midscenejs.com/reference/#common

• Discord
• Follow us on X
• Lark Group (飞书交流群)

Community projects that extend Midscene.js capabilities:

• midscene-ios - iOS Mirror automation support for Midscene
• midscene-pc - PC operation device for Windows, macOS, and Linux
• midscene-pc-docker - Docker image with Midscene-PC server pre-installed
• Midscene-Python - Python SDK for Midscene automation
• midscene-java by @Master-Frank - Java SDK for Midscene automation
• midscene-java by @alstafeev - Java SDK for Midscene automation

We would like to thank the following projects:

• Rsbuild and Rslib for the build tools.
• UI-TARS for the open-source agent model UI-TARS.
• Qwen-VL for the open-source multimodal model Qwen-VL.
• scrcpy and yume-chan for browser-based Android device control.
• appium-adb for its JavaScript bridge to ADB.
• appium-webdriveragent for controlling XCTest from JavaScript.
• YADB for improving text input performance.
• libnut-core for cross-platform native keyboard and mouse control.
• Puppeteer for browser automation and control.
• Playwright for browser automation, control, and testing.

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: GUI Agent for E2E Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

Midscene.js is MIT licensed.