Can AI agents complete everyday online tasks?
ClawBench is an open benchmark that measures whether AI browser agents can complete everyday online tasks on live websites. The V2 corpus has 130 tasks across 63 platforms in 30 task categories, and the V1 corpus has 153 tasks. Each run is scored in two stages: the agent's final HTTP request is intercepted and matched to the task schema, then an LLM judge checks the payload against the instruction. On V2 (hermes harness), claude-opus-4-7 leads with 71 of 130 runs intercepted (54.6%) and a 44.6% reward rate; on V1, claude-opus-4-6 leads with 94 of 153 (61.4%). Leaderboard data as of 2026-05-20.
We give AI agents real online tasks — booking flights, ordering food, applying for jobs — on live websites, and check whether they actually submit the right thing.
V2 (Hermes) — 8 models
| Rank | Model | Harness | Reward | Intercepted | Reward (strict) | Cost / task | Pass / Total |
|---|---|---|---|---|---|---|---|
| 1 | claude-opus-4-7 | hermes | 44.6% | 54.6% | 24.6% | $4.4425 | 58 / 130 |
| 2 | gpt-5.5 | hermes | 35.4% | 45.4% | 18.5% | $0.3325 | 46 / 130 |
| 3 | glm-5.1 | hermes | 34.6% | 48.5% | 17.7% | $0.1935 | 45 / 130 |
| 4 | deepseek-v4-pro | hermes | 33.9% | 43.9% | 12.3% | $0.0721 | 44 / 130 |
| 5 | deepseek-v4-flash:free | hermes | 2.3% | 3.1% | 0.0% | $0.0000 | 3 / 130 |
| 6 | z-ai/glm-4.5-air:free | hermes | 2.3% | 4.6% | 0.8% | $0.0000 | 3 / 130 |
| 7 | minimax-m2.5:free | hermes | 1.5% | 2.3% | 0.0% | $0.0000 | 2 / 130 |
| 8 | openrouter-owl-alpha | hermes | 0.0% | 14.6% | 0.0% | $0.3704 | 0 / 130 |
How scoring works
deepseek/deepseek-v4-pro to fulfill the instruction under "no contradiction → match" rubric (Stage 2, lenient). Reward (strict) = same judge, strict rubric ("ambiguous → mismatch"). Ranked by Reward (lenient).
Snapshot: 2026-05-20. Full details: eval/scoring.md ↗.
New here? About ClawBench ↗.
$ pip install clawbench-eval && clawbench run --corpus v2 --model your-modelHow it works
Watch one run: deepseek-v4-pro on V2 task 535-daily-life-shopping-etsy —
"Search Etsy for a handmade blue ceramic flower vase under $50 and add it to your favorites".
The agent searches, filters, opens a listing, and favorites it before the harness intercepts. Intercepted ✓ (hover to step through screenshots)
Full 5-layer trace bundle for this run: view trace →
Traces & dataset
1,724 judge-verified runs · 13 frontier models · 283 distinct everyday tasks · 163 live platforms · Apache-2.0 · refreshed weekly (last 2026-05-20). Every run ships as a 5-layer bundle: video, actions, agent messages, HTTP requests, and the graded verdict — all on one clock.
Train your own agent
918 V1 + 806 V2 frontier-model trajectories. JSONL-native, SFT/DPO/PRM-ready; mine success-vs-failure pairs across 13 models on identical tasks.Replay & audit
Step through video + HAR + agent reasoning side-by-side. Diagnose failure modes, audit judge calls, diff a model's pixels vs its words.Reproduce the leaderboard
Re-run any cell with our judge on your own data — or our data on your judge. The CLI consumes the same bundles you'll download.Want to read the tasks first? Browse all 283 task definitions → (searchable, filterable, no download)
Inside every trace — 6 time-synchronized signals per run
Every signal is timestamped against the same clock — click frame 1872 of recording.mp4 and you can find the exact actions.jsonl event, the LLM turn that triggered it, and the HTTP requests it fired. Cross-org mirrors:
NAIL-Group ·
TIGER-Lab
· Apache-2.0 · Bundle format: tar.gz per run, jsonl within.
FAQ
How is the leaderboard scored and ranked?
deepseek/deepseek-v4-pro) reads the intercepted payload against the instruction — lenient rubric ("no contradiction → match") for the headline Reward, strict rubric ("ambiguous → mismatch") for Reward (strict), visible via the "All columns" toggle. Ranked by Reward (lenient). Details: eval/scoring.md ↗
How do I submit my model?
pip install clawbench-eval && clawbench run --corpus v2 --model your-model) and open a PR with your run artifacts — github.com/TIGER-AI-Lab/ClawBench/pulls ↗. Or start from the Contribute page.
Cite this benchmark
Using ClawBench in your research? Please cite the arXiv paper:
@article{zhang2026clawbench,
title={ClawBench: Can AI Agents Complete Everyday Online Tasks?},
author={Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen},
journal={arXiv preprint arXiv:2604.08523},
year={2026}
}