Testing desktop apps

tapstep drives desktop apps the same way it drives a browser: snapshot the screen, tap by selector, assert what appeared. Any Tauri/webview app that embeds the tapstep bridge is a target, and the flows look exactly like web flows — selectors resolve through the same DOM walk.

The bridge rides Tauri’s own eval rather than CDP, so one flow file runs on macOS, Windows and Linux without a per-platform driver.

The two-instance model

A window never drives itself. There is always a target — the app under test, started with the bridge on — and a control that drives it: the tapstep CLI, or your everyday tapstep desktop window.

# target: your app, with the bridge on
TAPSTEP_TEST_BRIDGE=9223 /path/to/your-app
# control: drive it
tapstep test flows/login.flow.yaml --driver app:9223

--driver app alone means port 9223. The bridge is inert without the environment variable — no port is opened and nothing is injected, so a shipped build is not drivable by accident.

There is no flow-header key for the app target: pick it with --driver app[:port] in the CLI, or the Desktop app row in the desktop’s device list.

Declare it in config.yaml

app:
  port: 9223                 # bridge port (default 9223)
  launch: ["./my-app"]       # how to start the target; omit to attach only

A running bridge is picked up for any project: the Run dialog says Desktop app detected on :9223 and the device list gains a Desktop app (:9223) row. app.launch only adds a Launch app button that starts the target for you; a target the desktop spawned gets a Stop button, one you attached to does not. app.port moves the probe off 9223.

Write the tests with AI

The chat has no device picker. Run a flow once against the Desktop app device (or ask the agent to use device app) — the agent drives the last run’s device and can switch with its select_device tool. It snapshots the target, writes the .flow.yaml, runs it against the target and heals selectors — all in the target window, never its own. The same target works for an external agent over MCP:

tapstep mcp --driver app:9223

Embed the bridge in your own app

The bridge is a small loopback eval-RPC that only turns on when TAPSTEP_TEST_BRIDGE=<port> is set. Protocol: newline-delimited JSON over TCP on 127.0.0.1:<port>, one object per line, id is an integer:

• {id, js} → evaluate js in the webview labelled main; reply {id, result} with the JSON-serialisable value (undefined becomes null) or {id, error} with the exception text.
• {id, screenshot: true} → reply {id, result} where result is a base64 PNG of the window; an empty string means capture is unavailable.

A minimal Tauri implementation, in two parts. The agent injected on every page load evals and posts the result back over IPC:

// injected once per page load (only when the env var is set)
window.__tapstepBridge = {
  run(id, js) {
    let result = null, error = null;
    try { result = (0, eval)(js); } catch (e) { error = String(e && e.stack || e); }
    if (result === undefined) result = null;
    window.__TAURI_INTERNALS__.invoke("bridge_result", { id, result, error });
  },
};

The listener hands each line to that agent and answers when bridge_result comes back:

// only when TAPSTEP_TEST_BRIDGE is set: 127.0.0.1:<port>, one JSON per line
async fn handle_conn(app: AppHandle, stream: TcpStream) {
    let (read, mut write) = stream.into_split();
    let mut lines = BufReader::new(read).lines();
    while let Ok(Some(line)) = lines.next_line().await {
        let Ok(req) = serde_json::from_str::<Value>(&line) else { continue };
        let Some(id) = req.get("id").and_then(Value::as_u64) else { continue };
        let resp = if req.get("screenshot").and_then(Value::as_bool) == Some(true) {
            json!({ "id": id, "result": capture_window_png_base64().unwrap_or_default() })
        } else if let Some(js) = req.get("js").and_then(Value::as_str) {
            let win = app.get_webview_window("main").expect("main window");
            let (tx, rx) = oneshot::channel();
            PENDING.lock().unwrap().insert(id, tx);       // completed by bridge_result
            win.eval(&format!("window.__tapstepBridge.run({id}, {})", json!(js))).ok();
            match tokio::time::timeout(Duration::from_secs(15), rx).await {
                Ok(Ok((result, None))) => json!({ "id": id, "result": result }),
                Ok(Ok((_, Some(error)))) => json!({ "id": id, "error": error }),
                _ => json!({ "id": id, "error": "webview eval timed out" }),
            }
        } else { continue };
        if write.write_all(format!("{resp}\n").as_bytes()).await.is_err() { break; }
    }
}

#[tauri::command]
fn bridge_result(id: u64, result: Value, error: Option<String>) {
    if let Some(tx) = PENDING.lock().unwrap().remove(&id) { let _ = tx.send((result, error)); }
}

That is the whole contract — the driver builds every operation as JS on its side, so the app never needs to know about selectors or flows.

tapstep tests itself with it

launch: [self] starts a copy of tapstep as the target: isolated data directory, inherited session, no first-run walls. That copy is how the desktop app’s own suite runs, which is also the honest answer to “do you use it yourselves”.

Limitations

The app driver has the core surface only — snapshot, tap, type, swipe, a few keys, screenshot — and none of the optional capabilities other drivers add:

• css: selectors, openLink, longPressOn, clearText, hideKeyboard, clearState, grantPermissions, setLocation and the other capability-backed commands fail with not supported.
• No network or console capture — no failed-request context in reports.
• No screencast: the live mirror is polled screenshots, and --video is a per-step frame montage rather than a real recording.
• pressKey covers Enter, Backspace, Tab, Escape; key combos are not supported.
• swipe/scroll are emulated with a wheel event plus scrollBy.
• Taps are synthetic DOM events on the element under the point — native menus, system dialogs and file pickers are out of reach.
• Only the webview labelled main is driven; each bridge request times out after 15 s.
• The driver reports platform: web, so platform: conditions take the web branch.

Notes

• Only where the app runs. The window must actually open — a desktop, or a CI runner with a display session. There is no headless webview.
• Screenshots come from a native window capture, feeding the live mirror and visual asserts. On macOS the first capture asks for Screen Recording permission; deny it and flows still run, just without frames.
• Device id is app for the default port, app:<port> otherwise — the same string the CLI takes.