LambdaTest/rook

rook

Agent assurance from the terminal.

Perform End-to-End (E2E) testing of your AI agents to validate how they actually behave across workflows, tools and actions.
Catch failures and vulnerabilities before they ship.

Install · Quick start · User guide · Samples · Documentation

_{Built by TestMu AI (formerly LambdaTest)}

Testing an AI agent is awkward because there is no fixed contract. Input might be a sentence, a pull request, or an image; output might be prose, a created ticket, or a written file. So rook derives the tests rather than asking you to write them.

$ rook

  /explore .           read the codebase — find the agents and what they do
  /generate            scenarios: functional · non-functional · adversarial
  /profile add <name>  how to invoke it — paste a curl, or give a command
  /sync                record the agent, its scenarios and profile upstream
  /run                 execute them against the live agent
  /ui                  verdicts, evidence and trends, in a browser

Scenarios span three classes and sixteen categories:

Adversarial scenarios are mapped to the OWASP Top 10 for LLM Applications (2025), the OWASP Top 10 for Agentic Applications (2026) and MITRE ATLAS.

Two ideas do most of the work.

An agent's account of what it did is the weakest evidence available about what it did. It is the one party with a reason to be wrong. So rook does not grade the reply. It reads the code, watches the filesystem, and calls the agent's own tools to check the effect — then quotes what it found.

Anything rook could not verify is reported as unverifiable — never as a pass, never as a failure. It gets its own count and is never folded into Fail, and the gaps are computed rather than asked of a model, so a verdict can say "Pass, and here is what nobody looked at." A harness that reports a failure it did not observe is worse than one that admits it could not look.

What a run gives you:

• Per criterion, not per scenario: what was expected, what happened, and a quote as evidence.
• What could not be checked, and why.
• What changed since last time — the pass-rate change between your last two runs, and which scenarios newly fail or were fixed.
• What it did, not just what it said — files that changed on disk while it ran, artifacts it produced, and tool calls checked against the agent's own tool surface.

Most eval and observability tools score what your agent said and recorded. rook checks what the run changed, and reports what it could not verify.

_{The eval and observability columns describe each category's default approach, not any single product; several eval tools synthesize test cases, score tool calls, and ship red-team modules.}

You need a TestMu AI (formerly LambdaTest) account. Exploring, generating, writing profiles and running spend Agent Assurance credits.

Homebrew and the shell installer cover macOS and Linux on x64 and arm64, and bring their own runtime, so they need no Node. npm also covers Windows x64; it needs Node.js 20 or newer to install and launch, then runs rook on its bundled runtime.

Homebrew

brew install lambdatest/rook/rook

Use the full lambdatest/rook/rook name. Homebrew refuses to load a formula from an untrusted third-party tap by its short name, and naming the tap in full trusts it. Upgrade with brew upgrade lambdatest/rook/rook. If you tapped LambdaTest/rook before September 2026, re-point the tap once.

Shell installer — downloads the archive for your platform, verifies its checksum, unpacks it into ~/.testmuai/rook-<version>/, and links rook into ~/.local/bin. Pass --dir to put it somewhere else, or --version X.Y.Z to pin one.

curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh | bash

npm — if you would rather manage it with your other global CLIs.

npm install -g @testmuai/rook@latest

Upgrade and uninstall — rook update checks for a newer release and prints the upgrade command.

Uninstalling leaves your sign-in and settings in ~/.testmuai/rook/; delete that directory too to remove them.

Driving rook from Claude Code, Codex or Gemini CLI? Add the coding-agent skill too. If an install method does not work on your platform, open an issue.

From inside a project that contains an agent, start rook. The first time, sign in through the browser and pick or create the project your work is filed under:

$ rook

› /login
› /project create support-agents

› /explore .

  read 6 files · 1 agent
  triage-service — triages support tickets: severity, owning team, a reply

› /generate

  14 scenarios · 11 functional · 3 adversarial

› /profile add local

  How is this agent invoked?  paste a curl · command · http · mcp
› curl http://127.0.0.1:9110/v1/triage -H 'content-type: application/json' -d '{"input":"look at T-1043"}'

  POST http://127.0.0.1:9110/v1/triage
  the scenario goes in "input"

› /sync

  recording 1 agent(s)
  triage-service: new upstream — recording everything
  triage-service: recorded

› /run

  14 scenario(s) → triage-service

  … 11 passed · 2 failed · 1 unverifiable

› /ui

/ui opens the hosted results app on what you have synced: every verdict, the exchange that produced it, the tools the agent called, and — after a second run — what changed. /ui --local serves the evidence on disk in a read-only local viewer instead.

You do not have to run the commands in order. Ask for a later step and rook plans the ones it needs first, with the cost, before spending anything. Or just describe what you want in a sentence.

Three ways to invoke the agent under test: an HTTP endpoint, a command (claude -p "{{goal}}"), or an MCP tool. For setup, profiles, scenario selection and reading results, see the user guide.

Every step has a headless form. Export LT_USERNAME and LT_ACCESS_KEY (your username and access key) from CI secrets and rook signs in with them; no rook login step is needed. Headless commands need tool grants up front, so pass --yes once you have reviewed what the agent under test can do (see A note on safety).

rook project use "$ROOK_PROJECT_ID"
rook agent use "$ROOK_AGENT_ID"
rook explore . --yes
rook generate --yes
rook sync --yes
rook run --yes --json > run.json
rook report "$(jq -r .run_id run.json)" --json > report.json

rook run refuses an agent that has never been synced. To keep a result local instead, skip sync and use rook run --test. The CI recipe is a complete, gated script, and Headless use and CI covers sign-in, permission grants, budgets and verdict gating.

Use the Rook skill to test, evaluate, or red-team an AI agent and interpret its saved results. It covers setup, target permissions, credit budgets, profiles, CI and evidence-based verdicts using your installed Rook CLI.

Install it user-wide (needs Node.js 22+ and npm):

npx @testmuai/rook-skill

This installs the skill user-wide for Claude Code, Codex and Gemini CLI. See the installer guide for client selection, updates, uninstall and file protections.

Then open your agent repository and ask: "Use rook to test my agent against its refund policy." Approve the target's real actions and credit spend before execution. See the CI recipe for completion and verdict checks; an exit code of 0 alone does not mean scenarios passed.

npx skills add https://github.com/LambdaTest/rook/tree/main/skill-installer/skills --skill rook --agent claude-code codex

Rook is also listed on the TestMu AI Agent Skills page and in the LambdaTest/agent-skills catalog.

Try the agents in samples/, from a small HTTP service to eight industry workflows. Each one has real defects for a good suite to find.

The business records are fictional. Rook evaluation uses your account and credits. See samples/README.md for how to run each one and what it should catch.

Everything rook produces is plain files. No database.

Set ROOK_HOME to keep the second somewhere else, such as a CI workspace. It is deliberately outside your project, so a credential cannot be swept into a commit by git add -A. Profiles and MCP configuration reference secrets as ${VAR} rather than embedding them, so they are safe to commit.

The agent you point rook at is yours, and its writes are real. rook invokes it the way a user would and cannot roll anything back. Do not rely on a per-target write-tool confirmation in headless mode. Before authoring or testing a profile, or running scenarios, review the target and the real actions it can take; authorize those actions and the credit spend in your coding-agent session or CI configuration.

Judges are told to verify without changing anything — calling issue_refund to find out whether a refund exists creates one. Rook evaluates tool calls against its permission policy; headless calls need effective grants rather than an interactive prompt.

rook records operational events — no prompts, no code, no arguments — and sends them to TestMu AI, attributed to your organization, to find failures. Set ROOK_TELEMETRY=off (or "telemetry": false in ~/.testmuai/rook/config.json) to keep them on your machine. The one-time notice appears only in an interactive terminal, so set the variable in CI if you want it off there.

• Bugs and feature requests — open an issue. Verdicts you disagree with are the most useful reports we get.
• Security — do not open a public issue. See SECURITY.md.
• Contributing — see CONTRIBUTING.md.
• Documentation — TestMu AI Agent Assurance.
• Reviews — rook is listed on agent.reviews, where coding agents review the tools they use after real tasks. To let your agent do the same, see agent.reviews/install.