AI agents

tapstep exposes its whole toolbox — see the screen, tap, run a test, heal a selector — to AI agents. There are three ways in, from zero-setup to bring-your-own-agent.

Built-in AI chat (desktop)

The ✨ icon in the desktop app opens a chat panel next to your tests. The agent sees the workspace and a device: it records steps via snapshot/act, writes .flow.yaml files, runs them and heals failing selectors. Every file edit shows up as a diff in the transcript, and a version checkpoint is saved before the first change.

The chat has no device picker: the agent starts on the last run’s device and switches with its select_device tool when you ask (or when a flow needs another platform).

The chat is one stream: the agent’s steps stand inline in the conversation where they happened, and a single Details toggle opens or closes every step’s input and result at once. The status line under the transcript names the phase the turn is in — Thinking, Working (with what it is working on) or Verifying — and every turn ends with a done, whatever it ended on.

What the panel gives you, briefly:

Fast / Thorough (⚡) — capped thinking for snappier steps, or full thinking between steps (Anthropic API and Claude Code; the other providers ignore it). Per machine.
Draft / Verified (✓) — Verified makes the turn run the test and heal it until it passes before it may finish (costs runs); Draft leaves that to the model.
Steering — type while a turn is running and the message lands in the turn in flight; when the provider can’t take it mid-turn, it is queued for the next one.
Attachments — images and text files, by button or drag-and-drop (up to 4 images, 8 files per message).
History / New chat — conversations are saved per project; New chat archives the current one, History brings any back.
Quick chips under the input — one-tap directives such as Re-runnable, Parameterize, Adapt to state, Run to green.
Usage counter — input / cached / output tokens for the session. Tokens only, no money.

Six providers, in the chat’s ⋯ menu → AI settings:

Anthropic API key — stored in the OS keychain. Endpoint and model are configurable (ai.endpoint, ai.model in config.yaml).
OpenAI-compatible — any chat/completions endpoint: a local Ollama or LM Studio, a self-hosted vLLM, or a hosted API. Point it at e.g. http://localhost:11434/v1, name a tool-capable model, and nothing leaves your machine. The API key is optional — local servers need none.
Claude Code — no API key: the chat drives a locally installed Claude Code signed in to your Claude subscription. Install it once and you’re done:
Terminal window
```
npm install -g @anthropic-ai/claude-code
claude   # sign in once
```
Codex — same idea for ChatGPT plans: npm install -g @openai/codex, then codex login.
Gemini CLI — npm install -g @google/gemini-cli. Google retired the free personal sign-in, so create an API key at aistudio.google.com/apikey (it has its own free quota) and paste it into the app — it lives in the OS keychain and is handed to the CLI per run.
OpenCode — the open-source terminal agent, which brings its own model provider, so the chat costs whatever that provider costs: npm install -g opencode-ai, then opencode auth login. The app registers the tapstep bridge in the project’s opencode.json.

The desktop keeps the chosen provider in the same ai: block of config.yaml (omitted for the Anthropic API default), next to endpoint, model and the privacy flag below. Keys never land in the file.

Privacy masking

Settings → AI → Privacy masking runs an on-device model that replaces names, emails, phones and secrets with stable pseudonyms before a chat request leaves the machine; the transcript shows the real values again locally. It is the machine default; a project overrides it with ai.privacy: true|false in config.yaml. With the CLI-agent providers only your own turns are masked — their tool traffic runs in the external CLI, out of reach.

The CLI has the same masking for its AI steps: tapstep privacy install downloads the model once (~945 MB), tapstep privacy status shows what is enabled, and TAPSTEP_PRIVACY=1 turns it on for a run.

Connect an external agent (MCP)

Settings → AI → MCP server starts a local MCP server bound to a device — localhost only, token-protected. The same row copies a ready command:

claude mcp add --transport http tapstep http://127.0.0.1:PORT/mcp \
  --header "Authorization: Bearer TOKEN"

Any MCP client works the same way (Cursor, Windsurf, …): point it at the URL with that header. The agent gets snapshot, try, act, resolve, diff, run_flow, validate_flow, suggest, patch_flow and heal against the selected device, plus the desktop’s own list_devices, select_device, list_runs, run_details and request_secret (asks you for a credential in a dialog and writes it to .env, so the value never enters the conversation).

The CLI is an MCP server too:

tapstep mcp [--driver web|android|ios|app[:port]] [--browser …]

It speaks MCP over stdio — this is what the .mcp.json below points at — and needs a signed-in machine like every other command. Without --driver it takes the connected device, else a browser. Tools: snapshot, try, act, resolve, diff, run_flow, validate_flow, take_screenshot, cheat_sheet, list_devices, suggest, patch_flow, heal, convert_curl, request_to_curl, import_playwright, export_playwright and export_extester. run_flow resolves the project’s defaultEnvironment, and the two export tools take the folder from the project’s exports: setting when the call names none.

snapshot answers in a compact notation — one line per element, with what is stable ($testId / #id), how many duplicates the page holds (x6), the field’s value and state, and which row or dialog it stands in — under a page header that says whether the screen has settled. It is documented in full at Snapshot notation, and the same legend ships in the tool’s own description and in cheat_sheet, so an agent never has to be told. snapshot { detail: "regions" } is the cheap overview of the same screen: one line per container instead of one per element.

How an agent drives a screen now

Reading the whole screen before every step is the expensive way to drive one. The cheap loop is two calls:

snapshot { detail: "regions" } on arrival — the overview. One line per container with what it holds (header: 2 buttons, 1 link "Log in", table "Orders": 20 rows, nav: 4 links), then the controls no container holds. About 25 lines whatever the screen; read it again after a page turn, not before every step.
try { step: … } for the step itself. step is one command in flow YAML, exactly what a .flow.yaml carries — tapOn, inputText, scroll — so the step that works here is the line to paste into the flow.

try resolves that step’s selector the way a run resolves it and acts only when it names exactly one element. Then it answers with the receipt (done tapOn e41 button "Create order" in:header), the events the two screens prove (went, dialog opened, toast, focus, value, error) and a net: line with the requests the action set off.

Nothing matched — none "Create Order!" — or several did — ambiguous "Save": 3 — and the screen is not touched: the answer lists up to five candidates with the anchors that tell them apart, and via:exact|norm|sub|score on each says how far the name had to stretch to reach it. Narrow the step (index:/of:, rightOf:, childOf:) and try again; read a full snapshot only after try has said none or ambiguous twice about the same screen.

expect is an optional assert* command run right after the step, for when the outcome matters; it comes back in the same answer as expect: ok or expect: failed — …, and a failed expectation is not a failed call — the action did happen. ref names an element from an earlier step instead of describing it again: try { ref: "e7", step: "tapOn: '-'" } acts on e7 whatever the step’s own selector says. request, sql and assertTrue steps go through try too — they need no screen, so none is read.

CLI workspaces

In a repo, scaffold agent config once:

tapstep init --agents

This writes .mcp.json (so Claude Code in that folder picks up the tapstep tools automatically — it runs tapstep mcp) and an AGENTS.md cheat sheet with the flow-file conventions.

There is also an agent skill — a deeper playbook (CLI + flow reference, working conventions) the agent loads on demand:

tapstep skill install

It installs for every detected agent — Claude Code (~/.claude/skills), Codex (~/.codex/skills), Gemini CLI (~/.gemini/skills), Cursor (~/.cursor/skills), OpenCode (~/.config/opencode/skills) — plus a shared ~/.agents/skills copy that Codex, Gemini CLI, Cursor and OpenCode also read. At least one of those home directories must exist, and you must be signed in. Idempotent. tapstep skill status says, per agent, whether the copy is installed and whether it matches this binary; tapstep skill refresh re-syncs the installed copies and runs by itself after tapstep update. The desktop offers the same install under Settings → AI → Agent skill.

The desktop app also ships an integrated terminal (the Terminal button in the status bar, or Ctrl+</kbd>) — handy for running claudeortapstep` right in the workspace.

AI steps in flows

Three commands ask a model about the current screen — snapshot text plus a screenshot when the driver has one:

- assertWithAI: "the cart shows exactly one item"        # bare form: optional
- assertWithAI: { assertion: "no error banner", optional: false }
- assertNoDefectsWithAI                                    # layout/visual defects
- extractTextWithAI: "the order number"                    # → ${aiText}
- extractTextWithAI: { query: "the total", into: total, pattern: "\\d+\\.\\d{2}" }

The bare forms are optional: true — a failed check (or an AI error) is a warning in the report, not a failed step; optional: false fails the flow. extractTextWithAI writes into into (default aiText); with pattern the value must match that regex (one retry, then the step fails). Details: assertWithAI and neighbours in the command reference.

The connection comes from config.yaml (via --config; a block naming both endpoint and model wins over the environment) or from the environment:

ai:
  endpoint: https://api.anthropic.com   # or any OpenAI-compatible base URL
  model: claude-opus-4-8
  key: sk-ant-…                         # optional; a sk-ant-… key means Anthropic

TAPSTEP_AI_ENDPOINT=http://localhost:11434/v1 TAPSTEP_AI_MODEL=qwen2.5 tapstep test flows/

A sk-ant-… key selects the Anthropic Messages API (with the env vars TAPSTEP_AI_ENDPOINT may then be omitted); anything else is treated as OpenAI-compatible chat/completions. tapstep test --analyze uses the same connection to triage failed flows after a run.

In the desktop app, AI steps use the chat’s connection — the Anthropic API key or the OpenAI-compatible endpoint of the project. The CLI-agent providers (Claude Code, Codex, Gemini CLI) have no HTTP endpoint to call, so with them assertWithAI & co. fail as unconfigured (AI commands need a model endpoint).