TIGER-AI-Lab/ClawBench

_{More badges · featured in 37 curated lists}

ClawBench is an open-source benchmark that evaluates AI browser agents on everyday online tasks — booking travel, ordering food, applying for jobs, managing email — across live websites. V1 lives in test-cases/v1/, V2 in test-cases/v2/. It records five layers of evidence and supports request-based scoring and trajectory evaluation against human references. See the live leaderboard for scores by corpus, harness, and scoring rubric.

We asked frontier AI agents to do what people do every day --
order food, book travel, apply for jobs, write reviews, manage projects.
In the historical V1 paper evaluation, the strongest evaluated agent completed about 1 in 3 tasks.

---

V1: 152 everyday tasks · 143 live sites   |   V2: 129 tasks · 63 live sites   |   281 total across 163 live websites · 15 life categories

_{The paper reports 153 (V1) and 130 (V2); two ASPCA tasks were removed after publication, so the shipping corpus is 152 and 129.}

_{Built by NAIL Group  ·  Sister project: HarnessBench — fixes the base model, varies the harness  ·  Runs on any Chrome.}

中文

All guides and integrations: Documentation index.

Prerequisites: Python 3.11+, uv, and a running Docker or Podman engine. See platform setup.

uv tool install clawbench-eval
clawbench

Launch from the directory where you want your configuration and results. The interactive TUI guides you through adding a model and selecting a task. Configure the default judge, deepseek-v4-pro, as well; the model setup guide explains judge selection and how to run without scoring.

git clone https://github.com/TIGER-AI-Lab/ClawBench.git
cd ClawBench
cp models/models.example.yaml models/models.yaml
# Edit models/models.yaml with your model and judge API settings.
uv run clawbench

./run.sh is a shortcut for the same TUI. For a non-interactive run from the checkout:

uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf your-provider-model-id

Replace your-provider-model-id with a configured model key. The first run builds container images; the local browser's printed noVNC URL lets you watch the agent. Recordings and metadata land under test-output/.

Continue with the getting started guide for batch and human runs, harness selection, and remote browser options. The CLI reference covers non-interactive commands. Coding agents can start with AGENTS.md.

   You pick a task            ClawBench spins up           Agent drives the         Interceptor captures
   from V1 or V2              an isolated Docker           browser: navigates,      every action across
   everyday scenarios         container + Chromium         fills forms, clicks      all 5 layers of data

   ┌──────────────┐           ┌──────────────┐           ┌──────────────┐           ┌──────────────┐
   │  "Book a pet │    ──►    │   Container  │    ──►    │   AI Agent   │    ──►    │   5 layers   │
   │   sitter on  │           │  + Chromium  │           │  browses the │           │  intercepted │
   │   Rover"     │           │  + Agent     │           │   live site  │           │  & recorded  │
   └──────────────┘           └──────────────┘           └──────────────┘           └──────────────┘

 Live Websites         Isolated Containers         Request Interceptor         Five-Layer Recording

┌─────────────────────────────────────────────────┐
│  Container (Docker / Podman)                    │
│                                                 │
│  ┌──────────┐  CDP Fetch/Runtime/Page events    │
│  │ Chromium ├─────────────────────────────┐     │
│  │ :9222 CDP│                             │     │
│  └──────────┘                             │     │
│                                           │     │
│  ┌──────────┐            ┌────────────────▼─┐   │
│  │  Xvfb    │◄──ffmpeg──►│  FastAPI Server  │   │
│  │ :99      │  x11grab   │  :7878           │   │
│  └──────────┘            └──────────────────┘   │
│                                  │              │
│                          ┌───────▼─────────┐    │
│                          │     /data       │    │
│                          │  actions.jsonl  │    │
│                          │  requests.jsonl │    │
│                          │  screenshots/   │    │
│                          │  recording.mp4  │    │
│                          └─────────────────┘    │
└─────────────────────────────────────────────────┘

ClawBench ships three Hugging Face datasets — task definitions plus full execution traces for V1 and V2. All open, downloadable in one command.

Build with the traces: See the Trace Cookbook for download recipes, data schemas, and research directions.

The trace datasets are large; use hf download --include "<pattern>" to pull a single model or a single task.

🏆 Live leaderboard: claw-bench.com/leaderboard (V2 default, two-stage scoring — interception + LLM judge). Full scoring formula in eval/scoring.md. Add your run: PR to leaderboard/results.csv.

• [2026.10.10] — 🎉 ClawBench has surpassed 1,000 GitHub stars! Thank you to everyone who has starred, used, and contributed to the benchmark.
• [2026.08.20] — 🏆 Our paper has been accepted to EMNLP 2026 Findings.
• [2026.08.20] - Added Kernel as a supported remote browser runtime. Thanks to @rgarcia.
• [2026.08.18] — Added WebBrain as a supported harness. Thanks to @alectimison-maker.
• [2026.08.16] — Released RewardHarness, our self-evolving agentic reward framework: 47.4% on EditReward-Bench from just 100 preference demos, with no reward-model training. Details →
• [2026.08.03] — Added Browserbase as a remote browser runtime. Details →
• [2026.07.30] — v0.8.0: Gemini-as-judge, random-click baseline harness, EdgeBench/SForge adapter, remote-browser CDP support. Details →
• [2026.07.25] — 🏆 Our paper has been accepted by COLM 2026 WAB.

_{Earlier updates: docs/news.md · full change history: CHANGELOG.md}

Authors from Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern, ByteDance Seed, and HKUST build on ClawBench — and Li Auto's Mach-Mind-4-Flash technical report evaluates on it.

😊 Google DeepMind, University of Oxford & Columbia University, The Recipe for Intelligence in Natural and Artificial Systems (DOI)

😊 Stanford, UC Berkeley, Microsoft Research & UCSB, Auditing Agent Harness Safety (Code, Project)

😊 Google, Agentic Coding Needs Proactivity, Not Just Autonomy (Google Research Blog)

😊 Harvard Kempner Institute, Massachusetts General Hospital & CUHK, NeuroClaw Technical Report (Code, Project)

😊 ETH Zürich & Handshake AI Research, Verifying Agents in Rubric-Graded Environments (Code, Workshop)

😊 University of Oxford, NUS & Peking University, OpenClaw Research: A Systematic Survey of Large Language Model Agents in Open Deployment (Project, Resources)

😊 Northwestern University, A Survey of the OpenClaw Ecosystem: From Platform Extensibility to Constraint Design (Project & Resources)

😊 UC Davis & UT Dallas, Toward Trustworthy Computer-Use Agents: Risk Propagation, Evaluation Gaps, and Human Governance (Code & Project, Resources)

If we missed your work, please open an issue or submit a pull request.

_{Codex and Claude Code runs on V2, and the V1 OpenClaw aggregate, are still in flight — they land on the live leaderboard first.}

Our scores are stable: two independent runs of the same model under the same judge (deepseek/deepseek-v4-pro, lenient rubric) reproduce Intercepted and Reward within ±2 pp on the V2 corpus.

There are two ways to verify this on your own machine.

Confirms the full pipeline (your agent + our judge) lines up with our leaderboard row.

clawbench-batch --models deepseek/deepseek-v4-flash --cases-suite v2 \
  --all-cases --harness hermes --no-judge --output-dir ./my-run
clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric both

Confirms just the judge matches ours (cheap, no agent compute, useful for sanity-checking your judge config).

hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace \
  --include "batch-aligned-*/deepseek-v4-flash-free/**" --local-dir ./reproduce
clawbench-rescore ./reproduce --judge-model deepseek-v4-pro --rubric both

One-shot equivalent of Path B for any model in the leaderboard:

clawbench-reproduce --model deepseek-v4-flash --tolerance 2.0

For deepseek-v4-flash:free × hermes × v2, the published row is Intercepted 3.1% / Reward-lenient 2.3% / Reward-strict 0.0% (3 / 129). Path A or B counts as reproduced when all three metrics land within ±2 pp. Larger gaps usually mean a different judge model, a different rubric prompt, or a harness configuration drift — diff your eval_results/<batch>/summary.json against the published row to localize the cause.

New here? Run this first. test-cases/v1-lite/ is a 20-task curated subset of V1, selected for household-name sites, real-world relevance, difficulty, and category diversity. It matches the 20-tasks-per-source convention of browser-use/benchmark and gives you a credible signal at a fraction of the full-benchmark cost.

Tier distribution: flagship 9 / core 8 / wildcard 3 — spanning daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), dev-tech (GitHub), academia (Overleaf), personal management (1Password), and more. Use the separate human-reference evaluation workflow to evaluate Lite trajectories under eval/agentic_eval.md; selecting the suite does not switch the native request judge to trajectory evaluation.

Run it with --cases-suite v1-lite, or inspect the link-backed task files in test-cases/v1-lite/.

The task walkthrough follows a V1 task from its JSON specification through the recorded browser session and human-reference assessment. It also explains why a placeholder interception rule needs trajectory evaluation.

The native runner scores intercepted requests with an LLM judge by default: a pass requires both an interception and a matching judge verdict. --no-judge reports interception-only results; clawbench-rescore can judge saved requests later. See the CLI reference.

The original V1 evaluation compares complete agent and human-reference trajectories under eval/agentic_eval.md. This is a separate workflow described in the evaluation guide. Compare scores using the same corpus revision, harness, and rubric.

To share rescored results, see the OpenEval / EvalPort export instructions.

./run.sh                                                                   # interactive TUI
uv run clawbench-run <case-dir> <model>                                    # one task
uv run clawbench-run <case-dir> --human                                    # human reference run
uv run clawbench-batch --models <model> --cases-suite v2 --all-cases       # a whole corpus

Command overview and commonly used flags: CLI reference. Run any command with --help for its complete option list.

V1 tasks are in test-cases/v1/ (152 tasks), V2 in test-cases/v2/ (129), Lite in test-cases/v1-lite/ (20), and converted Claw-Eval tasks in test-cases/claw-eval/ (19). All suites use test-cases/task.schema.json. For test case authoring, see CONTRIBUTING.md; for output structure and evaluation guidance, see eval/README.md.

ClawBench's niche: live consumer websites, everyday tasks, end-to-end recording. If you want a controlled sandbox or replayed traces, the projects above are excellent. If you want to know whether your agent can actually order food or book a flight today, this is the benchmark for that.

See the FAQ for task coverage, account setup, recording artifacts, scoring, and live-site limitations. For installation and model configuration, use Getting started.

We welcome contributions -- especially new test cases. If you've ever ordered groceries, booked an appointment, or filed a form online, you already know how to write one. Most PRs are a single JSON file and land in under a day.

Quick wins:

• Add a new test case (~30 min, no container expertise needed)
• Add a new category of 10+ tasks → co-author invitation on the next paper revision
• Submit a new model to the public leaderboard
• Browse good first issues

See CONTRIBUTING.md for the full guide and contributor recognition policy.

Come hang out with researchers, builders, and contributors working on real-world browser agents.

Questions & bugs Fastest route to a maintainer

中文社区 研究者、开发者、贡献者交流

Dataset & leaderboard talk On the Hub, next to the data

If you use ClawBench in your research, please cite:

@misc{zhang2026clawbenchaiagentscomplete,
  title         = {ClawBench: Can AI Agents Complete Everyday Online Tasks?},
  author        = {Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen},
  year          = {2026},
  eprint        = {2604.08523},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2604.08523}
}

Questions, suggestions, or research collaboration? Reach the maintainer:

• Yuxuan Zhang — reacher [at] cs.ubc.ca (UBC, NAIL Group) · Homepage ↗
• For bug reports or feature requests, please open a GitHub issue — it's faster than email and gets seen by all maintainers.

Yuxuan Zhang

Yubo Wang

Perry Zhu

Penghui Du

Junwen Miao

Kelsey R. Allen

Wenhu Chen

Dongfu Jiang

Liang Chen

If ClawBench is useful for your research or product work, the single most helpful thing you can do is star the repo — it surfaces the benchmark to other AI-agent researchers and helps us justify continued dataset curation.

Open to contributions — new test cases, bug fixes, or evaluation submissions for a model we haven't scored yet. See CONTRIBUTING.md.

Apache 2.0 -- see LICENSE.

The converted Claw-Eval suite in test-cases/claw-eval/ is derived from claw-eval/claw-eval and the claw-eval/Claw-Eval dataset, which are released under the MIT License. Third-party package notices are in NOTICE.

Built with OpenClaw, opencode, Claude Code, the Claude in Chrome extension, OpenAI Codex CLI, browser-use, claw-code, Hermes Agent, Pi with pi-browser-harness, and WebBrain (selectable harnesses), Microsoft Playwright MCP (browser control bridge for the opencode, claude-code, codex, and claw-code harnesses), LiteLLM (API translation proxy for the claude-code, claude-code-chrome-extension, codex, browser-use, claw-code, and pi harnesses), noVNC (MPL 2.0), and websockify (LGPL 3.0).