TestSprite/testsprite-cli

The verification layer for the agentic coding era.

AI ships code in minutes — verifying it hasn't. testsprite opens your live app, uses it like a real user, and shows your coding agent exactly what broke — so it fixes its own work before a bug ever reaches you.

_{Proof, in public: verification beats model size. With this CLI in the loop, the cheapest model in the field shipped the most correct app on an open leaderboard — 89%, at half the cost of the priciest one. See the leaderboard →}

⭐ Help us reach more developers and grow the TestSprite community. Star this repo!

_{▶ Watch the launch video — the three hard limits every coding agent hits, and the loop that breaks them (4 min).}

TestSprite is the AI testing platform 100,000+ teams use to test their software, frontend and backend — in the cloud, against the live product, not mocks. This repo is its official CLI.

It puts that platform in your coding agent's hands: the agent verifies every behavior it ships, and what broke comes back as one self-consistent bundle it can act on — no dashboard scraping. Humans drive the same surface from a terminal or CI.

If you find testsprite useful, a GitHub Star ⭐️ would be greatly appreciated — it helps other builders (and their agents) find the project, and stars notify you about new releases.

Requires Node.js 20.19+, 22.13+, or 24+. (No global install? npx @testsprite/testsprite-cli works too.)

npm install -g @testsprite/testsprite-cli
testsprite setup

testsprite setup prompts for your API key, verifies it, and installs the verification-loop skill for the coding agents this project uses — one command, so your agent is wired to verify its own work.

Which agent it installs for. Each of the eight supported targets — claude, cursor, cline, windsurf, antigravity, kiro, copilot, codex — lands its skill in a different place, so setup works out which you use: the agent calling it, if it identifies itself, together with every agent whose config is already in the project. It installs for all of them and prints what it found — in a terminal it shows you that list first, so you can accept it or narrow it before anything is written. If nothing identifies an agent, it installs for claude and says so — that is the one case where you want --agent <target> to name one yourself. Check the result any time with testsprite agent status.

Non-interactive (CI / onboarding scripts):

TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yes

Pointing a coding agent (Claude Code, Cursor, Codex, Cline, …) at TestSprite? Have it run testsprite setup first — that installs the verification skill, so the agent knows how to create, run, and triage tests on its own (instead of guessing from this README). setup reports which agents it installed for; add --agent <target> if that list is not the one you wanted. New here? Start with the getting-started overview.

Privacy note: interactive runs check the npm registry at most once per 24 h to offer a "new version available" notice — package name only, never your key or data; TESTSPRITE_NO_UPDATE_NOTIFIER=1 disables it. The backend also advertises its minimum supported CLI version — a below-floor CLI prints a one-line upgrade advisory on stderr, and a too-old client may be rejected with exit 14 (CLIENT_TOO_OLD). Details in DOCUMENTATION.md → Update notice.

From there, the loop runs on its own — an example session, typed by the coding agent:

# 1 — describe the behavior you want to guarantee, run it, wait
testsprite test create --project proj_8f0f6 --type frontend \
  --plan-from ./checkout-flow.plan.json --run --wait --output json
#   → exits 1: the run failed

# 2 — pull ONE self-consistent failure bundle
testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure

# 3 — the agent reads the bundle, fixes the code, then replays
testsprite test rerun test_3a9f21c7 --wait --output json
#   → exits 0: passed. The test now lives in your durable suite.

./checkout-flow.plan.json (the --plan-from argument above) is a JSON file describing the test in plain language — no browser code required. This is the exact, byte-identical output of testsprite test create --plan-template (also embedded verbatim in test create --help):

{
  "$schema": "https://raw.githubusercontent.com/TestSprite/testsprite-cli/v0.4.0/schemas/plan.schema.json",
  "projectId": "prj_abc123",
  "type": "frontend",
  "name": "Login rejects an empty password",
  "planSteps": [
    {
      "type": "action",
      "description": "Navigate to /login and submit the form with an empty password"
    },
    {
      "type": "assertion",
      "description": "Verify an inline error says the password is required"
    }
  ]
}

Get this exact skeleton without hand-copying it (and without the risk of it drifting from your installed version — see below): testsprite test create --plan-template. Full field reference (including the {{...}}-placeholder caveat, size caps, and the $schema hook for live editor validation): Plan file format.

The $schema URL above is pinned to the CLI version that generated this doc (v0.4.0) — --plan-template's live output always pins to your installed version instead, which is what actually resolves. If you're reading this on a later release, run the command yourself rather than trusting this snippet verbatim.

Don't want to write the first tests yourself? For a deployed target, ask TestSprite to propose them. Local projects use the authored-plan flow below. testsprite test plan generate --project <id> explores your app (or reads the API spec you uploaded with project docs upload), drafts test cases, and stages them for review — nothing is written to your disk. You read the printed table and keep what you want:

testsprite test plan generate --project proj_8f0f6           # stages proposals for review
testsprite test plan accept --project proj_8f0f6             # → real tests
testsprite test plan accept --project proj_8f0f6 --only prop_2 prop_5   # or just these

It only runs what the project is still missing, so a second call picks up where the first left off. Generation spends workspace credits per stage that actually runs (the result line reports the spend), and a PRD uploaded with project docs upload --role prd bills 0.5 credits for its embedding — an API spec is free. Full walkthrough, including what happens on Ctrl-C and every precondition error: Generate commands.

Prefer to configure each step by hand (or learn the surface offline with --dry-run first)? See Manual setup and Install & verify.

Frontend tests can reach your running app through a TestSprite tunnel, including on the Free plan (ordinary run credits apply). Use an API key with run:tunnel; the CLI names the missing scope if the key cannot open a tunnel. Existing keys receive this scope on backend v0.18.0 or later unless deliberately narrowed.

testsprite project create --type frontend --name "My local app" --local 3000
testsprite test create --plan-from ./checkout.plan.json --project <project-id>
testsprite test run <test-id>  # the saved loopback URL opens a tunnel and waits
testsprite test run <id> <id> --max-concurrency 5
testsprite test run --all --project <project-id>
# Or keep one tunnel open in terminal A (no port argument)
testsprite tunnel start --ttl 3600
# Terminal B: borrow the clientId printed above; keep terminal A running
testsprite test run <test-id> --tunnel-client <client-uuid>
testsprite tunnel list
testsprite tunnel stop <client-uuid>
# Stop every live tunnel on this account only when you intend to revoke them all
testsprite tunnel stop --all --confirm

Run several local frontend tests in one invocation to share one tunnel binding. The batch runs up to 5 tests at once by default; --max-concurrency accepts 1–10. --all --local skips backend tests. An owned batch cancels unfinished runs on Ctrl-C or tunnel loss; JSON output lists every requested test in input order, including tests that did not start. The minted client id is printed before the tunnel connects. Hit the tunnel limit? Run testsprite tunnel list, then testsprite tunnel stop <client-uuid> for an unused binding, or testsprite tunnel stop --all --confirm to revoke them all. An older server without listing returns exit 4; a key without run:tunnel returns exit 3. A second tunnel start or process using the same credential takes over, and the first exits 10. An environment is one origin (scheme, host and port) plus its sign-in configuration. Frontend runs automatically open a tunnel for a saved localhost, 127.0.0.1 or [::1] URL: test run, test rerun, test create --run, test create-batch --run and testlist run use this behavior. --env <name> selects the environment; omit it for the project default. --no-wait disables automatic tunnels; drop it to run through one. If environment reads are unavailable, the command keeps the previous server behavior. Several fresh-run IDs require a tunnel; public targets retain the exit-5 refusal. A named public environment used against a local origin prints a sign-in advisory on stderr. Reruns and test-list runs share one timeout including capacity waits and require the server to confirm tunnel support; an older server triggers a cancellation attempt and a clear error.

--local <port> remains a shorthand and defaults new project/environment URLs to http://localhost:<port>. On a test run it preserves the host of a selected loopback environment, or the one host agreed by saved non-temporary loopback environments on that port; otherwise it uses localhost. If environment reads fail, it retains the legacy 127.0.0.1 host. Saved URLs retain their exact spelling, and aliases are not merged. --local-host remains accepted but is deprecated and hidden from project/environment help; prefer --url http://127.0.0.1:<port> or --url http://[::1]:<port> when the host matters. Dev servers can take a minute per page through a tunnel; test a built server with npm run build and your framework's preview/start command for heavy apps. The control plane is WebSocket over TLS (wss://control.tun.testsprite.com/ws). The data plane, which carries the tunnel secret and proxied traffic, uses TLS at data.tun.testsprite.com:443 and verifies the certificate with Node's default trust store: its bundled Mozilla roots, plus certificates supplied through NODE_EXTRA_CA_CERTS and the system CAs when Node is started with --use-system-ca. Node 20 is supported without losing NODE_EXTRA_CA_CERTS. Verification cannot be disabled, and the CLI never falls back from TLS to plaintext. On a network that re-signs TLS, export your organisation's root CA to a PEM file and set NODE_EXTRA_CA_CERTS=/path/to/ca.pem before running testsprite. Plaintext connects and TLS handshakes each time out after 10 seconds. The first failed attempt starts a 60-second retry episode; writing TunnelHello is not enough to reset it. The episode ends only after the first inbound tunnel stream or after the socket remains open for 5 seconds following the hello. A real deadline timer stays armed across retries and destroys an in-flight socket when it expires. After that, an owned test run --local run is cancelled and refunded and the command exits 10; tunnel start exits 10. If a self-hosted or older TestSprite server does not advertise a TLS endpoint, the CLI prints a one-time warning and uses legacy plaintext port 7400 under the same timeout and retry rules. tunnel start prints transport: tls or transport: plaintext. An owned tunnel's timeout or Ctrl-C cancels unfinished runs by default. A borrowed run still detaches without cancellation on Ctrl-C/SIGTERM because its owner keeps the tunnel alive; if that owner disappears while a run is active, the borrower cancels its own run and reports cancelled, already finished, or skipped. A run cancelled before it finishes is refunded. --no-cancel-on-interrupt also skips this owner-gone cancellation. Keep the early stderr run-id receipts (Run <runId> for one test, [i/N] <testId> → run <runId> for a batch) and read the reported runs before re-running. Local projects skip exploration/plan generation; author plans with test create --plan-from. Portal runs are blocked for free until you set a public project URL. See the full local-testing reference for flags, login, billing/refunds, timeouts, and retargeting.

The earlier command names — init, auth configure, auth whoami, auth logout — still work as hidden, deprecated aliases (each prints a one-line notice pointing at the new name), so existing scripts keep running. auth configure now runs the full setup (it also installs the skill).

📚 Full reference — every command, flag, and example: DOCUMENTATION.md, including configuration & profiles, scripting, and exit codes.

• 🧪 Tests like a real user. Runs against a live browser or API in the cloud — real clicks, real navigation, real assertions. Not a mock.
• 🤖 Agent-shaped output. test failure get returns one bundle — the failing step, its neighbors, screenshots, DOM snapshots, the test source, a root-cause hypothesis, and a recommended fix target — all sharing a single snapshotId. The CLI refuses to stitch data from two different runs, so an agent never reasons over a frankenstein context.
• ♻️ A loop, not a one-shot. create → run → failure get → fix → rerun — every pass is banked, not thrown away.
• 📐 Scriptable & deterministic. Stable --output json contract, predictable exit codes, and a --dry-run that exercises the full code path offline with canned data.
• 🚦 CI-native. On GitHub Actions, --wait runs annotate the PR checks tab with one ::error:: per failure and append a results table to the job summary — automatically. Add --report junit for a JUnit XML sidecar, --summary-file for a machine summary, or --gh-output to preview the annotations locally. Runs in any CI, not just GitHub. Details →
• 🔌 One command to onboard your agent. testsprite setup drops ready-made skill files into your repo — for the agents it detects you using — so your coding agent knows how to drive the loop on its own.

Every time your agent changes code, it asks one question: is this behavior already covered by the suite?

• Not yet covered → testsprite test create — describe the new behavior, run it.
• Already covered → testsprite test rerun — replay the existing tests, so nothing that used to work breaks silently.
• Something fails → testsprite test failure get — one self-consistent bundle; the agent fixes the code and reruns.

Every pass is banked into a durable suite, so coverage compounds as the project grows — a lasting record of every requirement it has ever gotten right, far bigger than any context window.

flowchart TD
    A["🤖 Your coding agent<br/>Claude Code · Codex · Antigravity · Kimi · Cursor · Trae …"]
    D{"behavior already<br/>covered by the suite?"}
    B["<b>testsprite test create</b><br/>new behavior → new test"]
    R["<b>testsprite test rerun</b><br/>replay the existing tests"]
    C{{"☁️ TestSprite testing agent<br/>runs the test like a real user against<br/>real browsers & real APIs on Cloud"}}
    F["<b>testsprite test failure get</b><br/>ONE self-consistent bundle:<br/>failing step · screenshots · DOM ·<br/>root-cause · recommended fix"]
    S[("📚 Durable integration suite<br/>grows with every pass")]

    A -->|"writes / changes code"| D
    D -->|"no — new behavior"| B
    D -->|"yes"| R
    B --> C
    R --> C
    C -->|"pass ✅"| S
    C -->|"fail ❌"| F
    F -->|"agent reads the bundle<br/>& fixes the code"| A
    S -.->|"defines what's covered"| D

The cloud is a black box on purpose: your agent describes intent and reads results. It never has to know how the test was driven — only what a real user experienced.

On CoderCup — an open leaderboard where frontier coding agents build the same app under the same rules, with TestSprite as the referee — the cheapest model in the field shipped the most correct app on the board: 89%, at half the cost of the priciest one.

That's the point of all of this: you no longer need the biggest, most expensive model to ship software you can trust — top-tier quality, without paying top-tier prices, within reach of every team.

• 📚 CLI reference — DOCUMENTATION.md
• 🌐 Platform docs — testsprite.com/docs
• 🐛 Issues & feature requests — GitHub issues
• 💬 Quick questions — Discord, or testsprite --help / testsprite test run --help right in your terminal
• 📝 Changelog — CHANGELOG.md

Contributions are welcome — the CLI is plain TypeScript/Node (20.19+, 22.13+, or 24+), tested with Vitest, built with tsc. Getting a dev loop running takes a minute:

git clone https://github.com/TestSprite/testsprite-cli.git
cd testsprite-cli && npm install
npm run build       # tsc → dist/ ; try it: node dist/index.js --help
npm test            # unit suite — no network or credentials needed
npm run lint:fix    # ESLint
npm run typecheck   # tsc --noEmit

Pull requests target the main branch. The full guide — build from source, test tiers, CI checks, branches, project layout — is in CONTRIBUTING.md.

Support — if you need any help, we're responsive on Discord, and feel free to email us at contact@testsprite.com too.

Apache-2.0 © TestSprite