Midscene

Midscene is an open-source, vision-first UI automation SDK. Instead of reading the DOM or accessibility tree, it works from screenshots alone: you describe each step in natural language (aiAct, aiQuery, aiAssert), and the model finds and interacts with whatever a human could see — icon-only buttons, canvas elements, native apps, even cross-origin iframes. It covers web, Android, iOS, HarmonyOS, and desktop through one API, plugs into Playwright/Vitest suites, and exposes Skills and MCP for AI agents.

The honest picture: vision-based automation is slower and more token-hungry than selector-based tests, so large suites get expensive and flaky in different ways (model misreads rather than selector breaks). You can self-host the UI-TARS open model for privacy and cost control, but that needs GPU infra. The project is community-driven, so enterprise support is essentially DIY. For visually complex UIs where selectors fail, though, it's unmatched.

Install the JS SDK or use the Chrome extension to start without code. Bring your own LLM key (GPT-4o, Gemini, Qwen-VL) or self-host UI-TARS.