#@midscene/test: Next-Gen AI End-to-End Testing Framework

#Testing Engineering Challenges in the AI Era

Even though GUI Agents can already drive E2E testing processes, there are still many practical demands before actual production deployment. Typical challenges include:

1. Need to Integrate Deterministic Scripts: In actual testing scenarios, there is a clear need to combine deterministic scripts (such as API calls, data preparation, and test case lifecycle control) to handle background or setup/teardown tasks, ensuring test engineering remains stable, fast, and cost-effective.
2. Preference for a Declarative Core Flow: A strong desire for the core testing flow to be in a YAML-like declarative language, rather than embedding a large amount of natural language inside .ts / .js code, which burdens the codebase and hurts the long-term maintenance experience.
3. Requirement for Extensibility to Co-maintain Scripts: The need for sufficient extensibility to let Agents and humans co-maintain test scripts, keeping clear boundaries and collaboration models between deterministic engineering and natural language intent to avoid tangled code and natural language.

To address these practical deployment challenges, Midscene has designed a new testing framework, @midscene/test.

#Design Philosophy: Balancing Declarative Semantics and Deterministic Engineering

To embrace the evolution of test engineering, @midscene/test returns to the first principles of end-to-end (E2E) testing and advocates a design philosophy of balancing "Declarative Semantics" and "Deterministic Engineering".

To address these challenges, Midscene's test framework focuses on providing the following capabilities:

#Out-of-the-Box Atomic Capabilities with High Extensibility

Midscene provides built-in atomic capabilities (including not only AI interactions like aiAct and aiAssert, but also environment initialization, device, and browser configurations) out of the box, enabling a quick start with zero configuration.

To handle complex business scenarios, Midscene supports custom atomic nodes (custom Nodes) defined in TypeScript. You can wrap APIs, data preparation, or custom toolchains into cohesive blocks and call them directly in upper-level YAML test cases. This keeps the authoring process simple while allowing the test engineering to scale for custom requirements.

#Writing Daily Test Cases via Declarative YAML Files

Focusing on "What to test", expressing business test intent. At this layer, test authors can describe UI actions (such as aiAct) and assertions (such as aiAssert) in natural language, while calling and composing custom business capabilities provided by "Deterministic Engineering" (such as order.prepare). Test authors can focus on business flows without needing to worry about underlying details, which enhances authoring efficiency and long-term maintainability.

#Auto-Generating Agent-Friendly Markdown Documentation

The framework provides the describe-nodes tool, which automatically compiles registered custom Nodes and their validation rules (Zod Schema) into standard Markdown documentation. This eliminates the cost of manually maintaining API manuals for your customized framework, allowing human authors and AI Agents to easily consult and consume these custom capabilities.

#Engineering Demands of E2E Projects

In actual enterprise-level adoption, E2E testing projects are by no means one-time, AI-driven instant verifications. Instead, they must run continuously as long-term assets, facing real and rigorous engineering demands such as stability, speed, and maintainability. To address this, Midscene provides supporting engineering capabilities to ensure the stability and efficiency of E2E projects in production:

• Unified Observability and Logging: Both AI visual steps and your custom Node executions are centrally recorded in the same execution lifecycle. The test report and runtime logs visualize the inputs, outputs, duration, execution status, and screenshots of every step. This simplifies troubleshooting and replay for human engineers and provides structured context for AI Agents to autonomously diagnose issues and optimize test cases.
• Standardized Lifecycle and Concurrency Control: Out-of-the-box lifecycle hooks (Before/After), concurrent environment isolation, and sandboxing ensure that your custom framework and test cases run stably and predictably in CI (Continuous Integration) pipelines.

#Example Scenario: Verify an Order Refund Flow

Suppose an e-commerce team needs to verify the refund flow for paid orders:

1. Deterministic Engineering (TypeScript Nodes): call an order API or script before each case to create test data, and clean up the test order when the case finishes.
2. Declarative Semantics (AI Natural Language): use YAML instructions to have the Agent submit a refund request in the UI and verify the result.

#Project File Structure

The recommended file structure for this example project is as follows:

ecommerce-tests/
├── cases/
│   └── refund.yaml            # Declarative cases: describe test cases and steps via YAML files
└── midscene.config.ts         # Deterministic engineering: register custom Nodes (e.g. order.prepare), define execution environments

• midscene.config.ts: Maintained by automation/test platform developers to build the underlying "Deterministic Engineering" (writing custom Zod Schemas, execution logic for custom Nodes, configuring browsers, etc.).
• cases/refund.yaml: Maintained by QA engineers (or AI Agents) to compose and express high-level business test intents via "Declarative Semantics" (combining natural language and custom Nodes).

#1. Auto-Generated Node Reference Example

In this scenario, the custom Node order.prepare defined by engineering builders automatically compiles into the following Markdown reference using the describe-nodes tool:

## order.prepare

- **Title**: Prepare order data
- **Description**: Pre-create an order with a specified status in the database.
- **Parameter schema (JSON Schema)**:

{
  "type": "object",
  "properties": {
    "status": {
      "type": "string",
      "description": "Order status, such as paid or refunded."
    }
  },
  "required": ["status"]
}

#2. Writing and Running Test Cases (YAML)

After reading the reference, test authors (or AI Agents) can directly call and compose the Node in the declarative YAML file:

# 1. Deterministic Engineering: prepare data and the environment
beforeEach:
  - order.prepare:
      status: paid
  - browser.openRefundPage: {}

# 2. Declarative Semantics: express business test intent
cases:
  - name: A paid order supports a full refund
    steps:
      - aiAct: Click Apply for refund, select Full refund, and submit
      - aiAssert:
          prompt: The page shows that the refund request was submitted and the refund amount equals the full order amount
          message: Failed to submit the full refund request

  - name: A paid order supports a partial refund
    steps:
      - aiAct: Click Apply for refund, enter a refund amount of 10, and submit
      - aiAssert:
          prompt: The page shows that the refund request was submitted and the refund amount is 10
          message: Failed to submit the partial refund request

# 3. Deterministic Engineering: clean up resources
afterEach:
  - order.cleanup: {}

Framework maintainers register custom Nodes such as order.prepare, browser.openRefundPage, and order.cleanup to manage data and pages. Midscene Nodes handle UI actions and assertions. Test intent remains separate from technical implementation, so the two can evolve independently.

#Next steps

• Read Extend and maintain Test Runner to learn how to register Nodes, manage runtime resources, and configure a Test Project.
• Read Write and run test cases to learn about Cases, Steps, lifecycles, and test results.