Skip to content
Back to search

WildClawBench

An in-the-wild benchmark for AI agents in the production harness.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/InternLM/WildClawBench with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/InternLM/WildClawBench'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/internlm--wildclawbench/badge.svg)](/listings/internlm--wildclawbench)

HTML

<a href="/listings/internlm--wildclawbench"><img src="/api/listings/internlm--wildclawbench/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

> **Prefer a standard runner?** The suite is also available in [Harbor](https://github.com/harbor-framework/harbor) format — skip the setup below and jump to [Run with Harbor](#run-with-harbor).

Install Docker

<details> <summary>macOS</summary>

brew install --cask docker

After installation, launch Docker Desktop from Applications or run:

open -a Docker

</details>

<details> <summary>Ubuntu</summary>

# Install dependencies
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add apt repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

# Allow current user to run Docker without sudo
sudo usermod -aG docker $USER
newgrp docker

</details>

Download Images

WildClawBench ships **four** Docker images, one per harness. They are all hosted on [HuggingFace](https://huggingface.co/datasets/internlm/WildClawBench/tree/main/Images). Pick the one(s) that match the harness you want to evaluate:

| Harness | Image tarball | Loaded tag | |---|---|---| | OpenClaw | `wildclawbench-ubuntu_v1.3.tar` | `wildclawbench-ubuntu:v1.3` | | Claude Code | `wildclawbench-claudecode-ubuntu_v0.2-patched.tar` | `wildclawbench-claudecode-ubuntu:v0.2` | | Codex CLI | `wildclawbench-codex-ubuntu_v0.0.tar` | `wildclawbench-codex-ubuntu:v0.0` | | Hermes Agent | `wildclawbench-hermes-agent-v0.5.tar.gz` | `wildclawbench-hermes-agent:v0.5` |

pip install -U "huggingface_hub[cli]"

# Download the images you need (or all four)
hf download internlm/WildClawBench Images/wildclawbench-ubuntu_v1.3.tar                    --repo-type dataset --local-dir .
hf download internlm/WildClawBench Images/wildclawbench-claudecode-ubuntu_v0.2-patched.tar --repo-type dataset --local-dir .
hf download internlm/WildClawBench Images/wildclawbench-codex-ubuntu_v0.0.tar              --repo-type dataset --local-dir .
hf download internlm/WildClawBench Images/wildclawbench-hermes-agent-v0.5.tar.gz           --repo-type dataset --local-dir .

Then load each image into Docker:

docker load -i Images/wildclawbench-ubuntu_v1.3.tar
docker load -i Images/wildclawbench-claudecode-ubuntu_v0.2-patched.tar
docker load -i Images/wildclawbench-codex-ubuntu_v0.0.tar
docker load -i Images/wildclawbench-hermes-agent-v0.5.tar.gz

Download Task Data

Download the task data from [HuggingFace](https://huggingface.co/datasets/internlm/WildClawBench/tree/main/workspace):

hf download internlm/WildClawBench workspace --repo-type dataset --local-dir .

Prepare Data

Run the preparation script to download YouTube videos, place them into the correct task directories, and extract archived git repos:

bash script/prepare.sh

The script will: - Download 3 YouTube videos (football match, lecture, product launch event) - Extract the first half of the football match and discard the full video - Rename and copy videos to the directories that need them - Extract `dot_git.tar.gz` for Safety Alignment tasks - Download SAM3 model weights for Code Intelligence tasks

Prerequisites: `yt-dlp`, `ffmpeg`, `gdown`.

> **Note:** YouTube downloads may require authentication. If you encounter a "Sign in to confirm you're not a bot" error, try one of the following: > - [Get cookies.txt locally](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc?pli=1).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 606 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">WildClawBench</h1>
  2. <p align="center">
  3. <img src="assets/lobster_battle.png" alt="WildClawBench Lobster" width="480">
  4. </p>
  5. <div align="center">
  6. [![Tasks](https://img.shields.io/badge/Tasks-60-blue)]()
  7. [![Harnesses](https://img.shields.io/badge/Harnesses-4-purple)]()
  8. [![Models](https://img.shields.io/badge/Models-34-green)]()
  9. [![Leaderboard](https://img.shields.io/badge/🏆_Leaderboard-WildClawBench-8c2416)](https://internlm.github.io/WildClawBench/)
  10. <br>
  11. [![arXiv](https://img.shields.io/badge/arXiv-2605.10912-b31b1b.svg)](https://arxiv.org/abs/2605.10912)
  12. [![HF Daily Paper](https://img.shields.io/badge/🤗_Daily_Paper-Featured-ffcc00)](https://huggingface.co/papers/2605.10912)
  13. [![HuggingFace](https://img.shields.io/badge/🤗_HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/internlm/WildClawBench)
  14. [![PDF Report](https://img.shields.io/badge/📄_Paper-PDF-red)](https://github.com/InternLM/WildClawBench/blob/main/WildClawBench_report.pdf)
  15. <br>
  16. [![Harbor Format](https://img.shields.io/badge/⚓_Harbor_Format-WildClawBench--Harbor-blue)](https://huggingface.co/datasets/internlm/WildClawBench-Harbor)
  17. [![Trajectories](https://img.shields.io/badge/🛤️_Trajectories-WildClawBench--Trajectories-orange)](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)
  18. [![Roller](https://img.shields.io/badge/🎞️_Collect_Trajectories-Roller-00866F)](https://github.com/Valen-Team/Roller)
  19. </div>
  20. > **Hard, practical, end-to-end evaluation for AI agents — in the wild.**
  21. ---
  22. **WildClawBench** is an agent benchmark that tests what actually matters: can an AI agent do real work, end-to-end, without hand-holding?
  23. We drop agents into a live [OpenClaw](https://github.com/openclaw/openclaw) environment — the same open-source personal AI assistant that real users rely on daily — and throw **60 original tasks** at them: clipping goal highlights from a football match, negotiating meeting times over multi-round emails, hunting down contradictions in search results, writing inference scripts for undocumented codebases, catching privacy leaks before they happen. Useful things. Hard things.
  24. In the **technical report snapshot**, the strongest frontier model topped out at **62.2% overall**; the latest audited OpenClaw runs have since raised the best score to **67.2%**. Most models still land well below that. That makes scores mean something.
  25. ### Why WildClawBench?
  26. Most agent benchmarks test isolated capabilities — calling a function, parsing JSON, following a single instruction. WildClawBench tests the full picture:
  27. | | What We Test | Why It's Hard |
  28. |:---:|---|---|
  29. | **🔗 Agency** | Multi-step tool orchestration, error recovery, autonomous planning | Agents must chain 10–60+ tool calls, adapt when services fail, and decide *what* to do — not just *how* |
  30. | **🎥 Multimodal** | Video understanding, image generation, cross-modal synthesis | Track events across a 45-min match video and clip precise highlights; classify 12 clothing photos, assemble 4 styled outfits, and generate full-body model images for each |
  31. | **🧵 Long-Horizon** | Complex workflows spanning 10–20 minutes of wall-clock execution | Negotiate meeting times over multiple email rounds; crawl, classify, and summarize 50+ academic papers |
  32. | **💻 Coding** | Read undocumented codebases, debug, generate working programs | Read an undocumented codebase, install dependencies, and write working inference from source alone; solve visual puzzles by generating pixel-accurate solutions |
  33. | **🛡️ Safety** | Prompt injection defense, credential leak detection, harmful content refusal | Harmful instructions are buried deep inside normal-looking documents; API keys are scattered across a large git history |
  34. ### What Sets Us Apart
  35. - **Real environment, not mocks.** Tasks run inside a live OpenClaw instance with real tools (browser, bash, file system, email, calendar).
  36. - **60 original tasks, built by hand.** Not adapted from existing benchmarks — each task was designed from scratch to stress-test real-world agent capabilities.
  37. - **Four agent harnesses, one task suite.** OpenClaw, Claude Code, Codex CLI, and Hermes Agent all execute the same 60 tasks under the same grading. This separates *model capability* from *harness scaffolding* — you can see how much an agent's score depends on its surrounding tools versus the underlying LLM.
  38. - **Reproducible & isolated.** Each task runs in its own Docker container. Same image, same data, same grading code. Ground truth and grading scripts are injected only after the agent finishes — they are never visible during execution, eliminating data leakage. Scores are reproducible across machines.
  39. ## The WildClawBench Family
  40. The evaluation code lives in this repository; the benchmark data ships as three Hugging Face datasets — pick the one that matches how you want to use it:
  41. | Repository | What's inside | Use it when you want to |
  42. |---|---|---|
  43. | **[WildClawBench](https://huggingface.co/datasets/internlm/WildClawBench)** | Task data, Docker images for all four harnesses | Reproduce the paper's evaluation with this repository's pipeline |
  44. | **[WildClawBench-Harbor](https://huggingface.co/datasets/internlm/WildClawBench-Harbor)** | All 60 tasks repackaged in the [Harbor](https://github.com/harbor-framework/harbor) format | Evaluate any Harbor-supported agent with a single `harbor run` — no benchmark-specific setup ([details](#run-with-harbor)) |
  45. | **[WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)** | Graded OpenClaw trajectories of the leaderboard models on the full 60-task suite: one session per run under `traces/`, and a table with the score of every run | Inspect how models actually behave — or mine real long-horizon traces for analysis and training ([details](#agent-trajectories)) |
  46. To collect and grade WildClawBench trajectories for your own model, use [Roller](https://github.com/Valen-Team/Roller) ([details](#collect-trajectories-with-roller)).
  47. ## News
  48. - **2026-11** **[WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)** now holds 720 graded trajectories from 12 leaderboard models, with one trace file per run under `traces/` and a table with the score of every run. To collect trajectories for your own model, see **[Roller](https://github.com/Valen-Team/Roller)**.
  49. - **2026-09** Qwen's **[Qwen3.8-Omni-Flash release](https://qwen.ai/blog?id=qwen3.8-omni-flash)** includes WildClawBench's multimodal subset (WildClawBench-MM) in its agent evaluations. Thanks for the recognition!
  50. - **2026-08** Meta's **[Muse Glimmer release](https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model)** reports WildClawBench evaluation scores. Thanks for the recognition!
  51. - **2026-08** Released **[WildClawBench-Harbor](https://huggingface.co/datasets/internlm/WildClawBench-Harbor)** — the full 60-task suite in [Harbor](https://github.com/harbor-framework/harbor) format — and **[WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)** — complete agent trajectories from our frontier-model evaluations, browsable in the HF Agent Trace Viewer and continuously updated as new models are evaluated.
  52. - **2026-07** We expanded the OpenClaw leaderboard with evaluations of the latest frontier models, including **GPT-5.6 Sol, Claude Fable 5, Kimi K3 and etc**.
  53. - **2026-06** ByteDance Seed's **[Seed2.1 release](https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity)** includes WildClawBench in its agent evaluations. Thanks for the recognition!
  54. - **2026-05** We released a new version with **four agent harnesses** — OpenClaw, Claude Code, Codex CLI, and Hermes Agent — so the same 60-task suite can be evaluated under multiple scaffolds.
  55. - **2026-05** We published a **[technical report PDF](WildClawBench_report.pdf)**.
  56. - **2026-05** Tencent’s **[Hunyuan3 Preview](https://hunyuan.tencent.com/research/hy3)** page reports WildClawBench evaluation scores. Thanks for the recognition!
  57. ---
  58. ## Leaderboard
  59. WildClawBench reports two complementary leaderboards:
  60. 1. **Model leaderboard (OpenClaw harness)** — apples-to-apples comparison of LLMs running inside the same OpenClaw harness.
  61. 2. **Harness comparison** — same model, same tasks, four different agent scaffolds.
  62. Full interactive leaderboard at [internlm.github.io/WildClawBench](https://internlm.github.io/WildClawBench/).
  63. ### Model leaderboard (OpenClaw harness)
  64. > **Overall score** follows the weighted Multimodal / Pure-text breakdown in that table. **Total time** and **total cost** are the paper’s Overall per-task averages (minutes / USD) multiplied by **60** for the full 60-task suite.
  65. | Rank | Model | Org | Overall Score | Total Time | Total Cost |
  66. |:----:|-------|-----|:-------------:|:----------:|:----------:|
  67. | 🥇 | **GPT-5.6 Sol** | OpenAI | **67.2%** | 222 min | $56.78 |
  68. | 🥈 | Claude Opus 4.8 | Anthropic | 64.7% | 400 min | $95.95 |
  69. | 🥉 | Claude Opus 4.7 | Anthropic | 62.2% | 328 min | $77.40 |
  70. | 4 | Claude Fable 5 | Anthropic | 62.0% | 324 min | $87.71 |
  71. | 5 | GPT-5.5 | OpenAI | 58.2% | 262 min | $37.80 |
  72. | 6 | Grok 4.5 | xAI | 57.5% | 359 min | $28.43 |
  73. | 7 | Qwen3.8-Max | Alibaba Cloud | 56.2% | 708 min | $24.70 |
  74. | 8 | Muse Spark 1.1 | Meta | 54.8% | 367 min | $23.59 |
  75. | 9 | Kimi K3 | Moonshot AI | 54.5% | 488 min | $40.08 |
  76. | 10 | GLM 5.2 | Zhipu AI | 54.2% | 442 min | $17.10 |
  77. | 11 | Claude Opus 4.6 | Anthropic | 51.6% | 508 min | $81.00 |
  78. | 12 | GPT-5.4 | OpenAI | 50.3% | 350 min | $19.80 |
  79. | 13 | Hy3 | Tencent | 49.7% | 338 min | $2.13 |
  80. | 14 | GLM 5.1 | Zhipu AI | 48.2% | 515 min | $34.80 |
  81. | 15 | Qwen3.8 27B | Alibaba Cloud | 48.0% | 516 min | N/A |
  82. | 16 | Muse Glimmer 30B | Meta | 47.6% | 352 min | $6.06 |
  83. | 17 | Kimi K2.7 Code | Moonshot AI | 46.9% | 674 min | $72.31 |
  84. | 18 | Intern-S2 Preview 397B | InternLM | 44.7% | 541 min | Free |
  85. | 19 | DeepSeek V4 Pro | DeepSeek | 43.7% | 605 min | $12.00 |
  86. | 20 | Qwen3.6 27B | Alibaba Cloud | 43.2% | 421 min | $20.91 |
  87. | 21 | MiMo V2.5 Pro | Xiaomi | 43.0% | 451 min | $12.60 |
  88. | 22 | GLM 5 | Zhipu AI | 42.6% | 373 min | $11.40 |
  89. | 23 | Gemini 3.1 Pro | Google DeepMind | 40.8% | 240 min | $18.00 |
  90. | 24 | MiMo V2 Pro | Xiaomi | 40.2% | 458 min | $26.40 |
  91. | 25 | Gemma 4 31B IT | Google DeepMind | 37.6% | 384 min | $3.46 |
  92. | 26 | Qwen3.5 397B | Alibaba Cloud | 34.5% | 459 min | $22.20 |
  93. | 27 | DeepSeek V3.2 | DeepSeek | 34.0% | 549 min | $11.40 |
  94. | 28 | GLM 5 Turbo | Zhipu AI | 33.9% | 499 min | $15.00 |
  95. | 29 | MiniMax M2.7 | MiniMax | 33.8% | 551 min | $7.20 |
  96. | 30 | Kimi K2.5 | Moonshot AI | 30.8% | 406 min | $6.60 |
  97. | 31 | MiMo V2 Flash | Xiaomi | 30.8% | 433 min | $10.20 |
  98. | 32 | MiniMax M2.5 | MiniMax | 27.1% | 542 min | $9.60 |
  99. | 33 | Step 3.5 Flash | StepFun | 26.7% | 430 min | $6.60 |
  100. | 34 | Grok 4.20 Beta | xAI | 19.3% | 94 min | $9.60 |
  101. > Claude Opus 4.8 cost uses the dynamic base-tier rates for this evaluation: $5/M input, $25/M output, $0.5/M cache read, and $6.25/M cache write.
  102. > Muse Glimmer 30B cost uses the OpenRouter rates for this evaluation: $0.35/M input, $1.50/M output, and $0.04/M cache read.
  103. > Kimi K2.7 Code cost uses the published rates for this evaluation: $6.5/M input, $27/M output, and $1.3/M cached input.
  104. > Qwen3.8-Max cost uses the evaluation rates: $1.7535/M input, $5.2605/M output, and $0.219186/M cache read.
  105. > Qwen3.8 27B was evaluated on a self-hosted vLLM endpoint. Token-level billing data was unavailable, so total cost is reported as N/A.
  106. ### Harness comparison
  107. Same 60 tasks, same grading, four different agent scaffolds. Time and cost are per-task averages; score is in %. Time is in minutes per task, cost in USD per task. **Bold** = best harness for that model.
  108. | Model | OpenClaw | | | Claude Code | | | Codex | | | Hermes Agent | | |
  109. |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
  110. | | Time | Cost | Score | Time | Cost | Score | Time | Cost | Score | Time | Cost | Score |
  111. | GPT-5.4 | 5.83 | $0.33 | 50.3 | 9.07 | $0.61 | 48.4 | 7.16 | $0.57 | **56.8** | 8.97 | $0.44 | 50.7 |
  112. | GLM 5 | 6.22 | $0.19 | 42.6 | 10.18 | $0.21 | 31.0 | 7.84 | $0.13 | 38.9 | 6.62 | $0.44 | **46.4** |
  113. | MiMo V2 Pro | 7.63 | $0.44 | 40.2 | 9.90 | $0.15 | 29.9 | 6.44 | $0.15 | 35.3 | 8.30 | $0.26 | **48.1** |
  114. | MiniMax M2.7 | 9.18 | $0.12 | 33.8 | 10.08 | $0.09 | 32.0 | 8.66 | $0.06 | 35.8 | 10.30 | $0.11 | **37.1** |
  115. ---
  116. ## Tasks
  117. **60 tasks** across **6 categories**, spanning English and Chinese:
  118. | Category | # | Example Tasks | Core Challenges |
  119. |:---------|:-:|---------------|-----------------|
  120. | **Productivity Flow** | 10 | ArXiv paper digest, PDF batch classification, calendar scheduling, Wikipedia biography, LaTeX table extraction | Information synthesis, multi-source aggregation, structured output |
  121. | **Code Intelligence** | 12 | SAM3 inference from source, visual puzzle solving (jigsaw, connect-the-dots, link-a-pix), benchmark reproduction, academic homepage generation | Undocumented codebase comprehension, pixel-level visual reasoning, end-to-end code generation |
  122. | **Social Interaction** | 6 | Multi-round meeting negotiation, chat action extraction, escalation routing, cross-department updates | Multi-turn communication, API orchestration, context tracking |
  123. | **Search & Retrieval** | 11 | Conflicting information resolution, financial data extraction, fuzzy repository search | Web search + local data reconciliation, multi-constraint satisfaction, source verification |
  124. | **Creative Synthesis** | 11 | Football match report with video clips, video English-to-Chinese dubbing, paper-to-poster, product launch video analysis, outfit-to-model image | Video/audio processing, cross-modal generation, design & layout |
  125. | **Safety Alignment** | 10 | Prompt injection via file content, leaked API key detection, malicious skill injection, misinformation refusal, file overwrite prevention | Adversarial robustness, credential awareness, harmful content refusal |
  126. To create new tasks, see the annotated template at [`tasks/task0_template.md`](tasks/task0_template.md).
  127. ## Quick Start
  128. > **Prefer a standard runner?** The suite is also available in [Harbor](https://github.com/harbor-framework/harbor) format — skip the setup below and jump to [Run with Harbor](#run-with-harbor).
  129. ### Install Docker
  130. <details>
  131. <summary>macOS</summary>
  132. ```bash
  133. brew install --cask docker
  134. ```
  135. After installation, launch Docker Desktop from Applications or run:
  136. ```bash
  137. open -a Docker
  138. ```
  139. </details>
  140. <details>
  141. <summary>Ubuntu</summary>
  142. ```bash
  143. # Install dependencies
  144. sudo apt-get update
  145. sudo apt-get install -y ca-certificates curl gnupg
  146. # Add Docker's official GPG key
  147. sudo install -m 0755 -d /etc/apt/keyrings
  148. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  149. sudo chmod a+r /etc/apt/keyrings/docker.gpg
  150. # Add apt repository
  151. echo \
  152. "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  153. $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  154. sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  155. # Install Docker
  156. sudo apt-get update
  157. sudo apt-get install -y docker-ce docker-ce-cli containerd.io
  158. # Allow current user to run Docker without sudo
  159. sudo usermod -aG docker $USER
  160. newgrp docker
  161. ```
  162. </details>
  163. ### Download Images
  164. WildClawBench ships **four** Docker images, one per harness. They are all hosted on [HuggingFace](https://huggingface.co/datasets/internlm/WildClawBench/tree/main/Images). Pick the one(s) that match the harness you want to evaluate:
  165. | Harness | Image tarball | Loaded tag |
  166. |---|---|---|
  167. | OpenClaw | `wildclawbench-ubuntu_v1.3.tar` | `wildclawbench-ubuntu:v1.3` |
  168. | Claude Code | `wildclawbench-claudecode-ubuntu_v0.2-patched.tar` | `wildclawbench-claudecode-ubuntu:v0.2` |
  169. | Codex CLI | `wildclawbench-codex-ubuntu_v0.0.tar` | `wildclawbench-codex-ubuntu:v0.0` |
  170. | Hermes Agent | `wildclawbench-hermes-agent-v0.5.tar.gz` | `wildclawbench-hermes-agent:v0.5` |
  171. ```bash
  172. pip install -U "huggingface_hub[cli]"
  173. # Download the images you need (or all four)
  174. hf download internlm/WildClawBench Images/wildclawbench-ubuntu_v1.3.tar --repo-type dataset --local-dir .
  175. hf download internlm/WildClawBench Images/wildclawbench-claudecode-ubuntu_v0.2-patched.tar --repo-type dataset --local-dir .
  176. hf download internlm/WildClawBench Images/wildclawbench-codex-ubuntu_v0.0.tar --repo-type dataset --local-dir .
  177. hf download internlm/WildClawBench Images/wildclawbench-hermes-agent-v0.5.tar.gz --repo-type dataset --local-dir .
  178. ```
  179. Then load each image into Docker:
  180. ```bash
  181. docker load -i Images/wildclawbench-ubuntu_v1.3.tar
  182. docker load -i Images/wildclawbench-claudecode-ubuntu_v0.2-patched.tar
  183. docker load -i Images/wildclawbench-codex-ubuntu_v0.0.tar
  184. docker load -i Images/wildclawbench-hermes-agent-v0.5.tar.gz
  185. ```
  186. ### Download Task Data
  187. Download the task data from [HuggingFace](https://huggingface.co/datasets/internlm/WildClawBench/tree/main/workspace):
  188. ```bash
  189. hf download internlm/WildClawBench workspace --repo-type dataset --local-dir .
  190. ```
  191. ### Prepare Data
  192. Run the preparation script to download YouTube videos, place them into the correct task directories, and extract archived git repos:
  193. ```bash
  194. bash script/prepare.sh
  195. ```
  196. The script will:
  197. - Download 3 YouTube videos (football match, lecture, product launch event)
  198. - Extract the first half of the football match and discard the full video
  199. - Rename and copy videos to the directories that need them
  200. - Extract `dot_git.tar.gz` for Safety Alignment tasks
  201. - Download SAM3 model weights for Code Intelligence tasks
  202. Prerequisites: `yt-dlp`, `ffmpeg`, `modelscope`.
  203. > **Note:** YouTube downloads may require authentication. If you encounter a "Sign in to confirm you're not a bot" error, try one of the following:
  204. > - [Get cookies.txt locally](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc?pli=1).
  205. > - Use `--cookies-from-browser` (e.g., `--cookies-from-browser chrome`)
  206. > - Install [Deno](https://deno.land/) as a JS engine, which some users have reported resolves the issue
  207. ### Run
  208. Create a local `.env` file from the example, then set your API keys:
  209. ```bash
  210. cp .env.example .env
  211. ```
  212. ```
  213. OPENROUTER_API_KEY=your_api_key_here
  214. BRAVE_API_KEY=your_brave_key_here # required for search tasks
  215. ```
  216. - **OpenRouter API Key** — Any model available on [OpenRouter](https://openrouter.ai/models) is supported. The default model is defined in the `.env` file as `DEFAULT_MODEL=openrouter/anthropic/claude-sonnet-4.6` — replace it with any model you want to evaluate.
  217. - **Brave Search API Key** — Required for Search & Retrieval tasks. Get one (with free monthly credits) at [brave.com/search/api](https://brave.com/search/api/).
  218. - **Judge model** (optional) — `JUDGE_MODEL` controls the LLM used by judge-based grading metrics. Defaults to `openai/gpt-5.4`.
  219. Optionally run the setup checker before launching containers:
  220. ```bash
  221. python3 eval/check_setup.py --agent-backend openclaw --category all
  222. ```
  223. Then run one of the four harnesses:
  224. ```bash
  225. bash script/run.sh openclaw --category all --parallel 4 --model openrouter/openai/gpt-5.5
  226. bash script/run.sh claudecode --category all --parallel 4 --model openai/gpt-5.5
  227. bash script/run.sh codex --category all --parallel 4 --model openrouter/openai/gpt-5.5
  228. bash script/run.sh hermesagent --category all --parallel 4 --model openai/gpt-5.5
  229. ```
  230. Single-task runs are also supported:
  231. ```bash
  232. bash script/run.sh openclaw --task tasks/06_Safety_Alignment/06_Safety_Alignment_task_1_file_overwrite.md \
  233. --model openrouter/openai/gpt-5.5
  234. ```
  235. > Model-name conventions differ per harness:
  236. > - **OpenClaw / Codex** expect `openrouter/<provider>/<model>` (since they hit OpenRouter directly).
  237. > - **Claude Code / Hermes Agent** expect `<provider>/<model>` (the `openrouter/` prefix is added internally).
  238. ### Using a Custom Model Endpoint (Without OpenRouter)
  239. This option currently applies to the **OpenClaw harness** only. If you prefer to use your own API endpoint instead of OpenRouter, you can provide a JSON file and WildClawBench will inject it into `~/.openclaw/openclaw.json` before each task starts.
  240. ⚠️ Important: Some task prompts and evaluation scripts currently have OpenRouter explicitly mentioned or hardcoded (e.g., https://openrouter.ai/api/v1). If you bypass OpenRouter, you will need to adjust these references in the respective files manually.
  241. **1. Fill in `my_api.json` (or provide your own JSON file with the same format):**
  242. ```json
  243. {
  244. "providers": {
  245. "my-openai-proxy": {
  246. "baseUrl": "http://host.docker.internal:8000/v1",
  247. "apiKey": "${MY_PROXY_API_KEY}",
  248. "api": "openai-completions",
  249. "models": [
  250. {
  251. "id": "my-model",
  252. "name": "My Model"
  253. }
  254. ]
  255. }
  256. }
  257. }
  258. ```
  259. This file is the value written into `openclaw.json["models"]`, so it should contain the `models` object itself, not the full `openclaw.json`. If you use `${MY_PROXY_API_KEY}`, WildClawBench will replace it on the host before the config is copied into the container, so `MY_PROXY_API_KEY` must be set in `.env`. WildClawBench always replaces the existing top-level `models` field with the JSON you provide.
  260. **2. Set your model name and required API key in `.env`:**
  261. ```bash
  262. MY_PROXY_API_KEY=your_api_key_here
  263. ```
  264. **3. Run the benchmark with the models config file:**
  265. ```bash
  266. python3 eval/run_batch.py --category 01_Productivity_Flow --models-config my_api.json --model my-openai-proxy/my-model
  267. ```
  268. <details>
  269. <summary>Common provider examples</summary>
  270. OpenAI-compatible proxy:
  271. ```json
  272. {
  273. "providers": {
  274. "proxy": {
  275. "baseUrl": "http://host.docker.internal:8000/v1",
  276. "models": [
  277. {
  278. "id": "gpt-4o",
  279. "name": "GPT-4o"
  280. }
  281. ]
  282. }
  283. }
  284. }
  285. ```
  286. Local vLLM or LM Studio:
  287. ```json
  288. {
  289. "providers": {
  290. "local-openai": {
  291. "baseUrl": "http://host.docker.internal:1234/v1",
  292. "models": [
  293. {
  294. "id": "qwen2.5-coder-32b-instruct",
  295. "name": "Qwen2.5 Coder 32B Instruct"
  296. }
  297. ]
  298. }
  299. }
  300. }
  301. ```
  302. Provider with explicit API mode and env var key:
  303. ```json
  304. {
  305. "providers": {
  306. "custom-gateway": {
  307. "baseUrl": "http://host.docker.internal:9000/v1",
  308. "apiKey": "${MY_PROXY_API_KEY}",
  309. "api": "openai-completions",
  310. "models": [
  311. {
  312. "id": "my-reasoning-model",
  313. "name": "My Reasoning Model"
  314. }
  315. ]
  316. }
  317. }
  318. }
  319. ```
  320. </details>
  321. ## Run with Harbor
  322. The full suite is also published in the [Harbor](https://github.com/harbor-framework/harbor) task format at **[internlm/WildClawBench-Harbor](https://huggingface.co/datasets/internlm/WildClawBench-Harbor)**. Each of the 60 tasks is a self-contained Harbor task directory (`task.toml` / `instruction.md` / `environment/` / `tests/`), with task content and grading logic identical to this repository. This is the easiest way to evaluate agents that Harbor already supports (Claude Code, OpenHands, Codex CLI, custom agents, ...) — no benchmark-specific pipeline needed.
  323. ```bash
  324. uv tool install harbor # or: pip install harbor
  325. # Task suite
  326. hf download internlm/WildClawBench-Harbor --repo-type dataset --local-dir ./WildClawBench-Harbor
  327. # Docker image (same OpenClaw image as above)
  328. hf download internlm/WildClawBench Images/wildclawbench-ubuntu_v1.3.tar --repo-type dataset --local-dir .
  329. docker load -i Images/wildclawbench-ubuntu_v1.3.tar
  330. # Keys the tasks read: the LLM judge and image understanding (46 tasks), and web search
  331. export OPENROUTER_API_KEY=...
  332. export BRAVE_API_KEY=...
  333. # Run the full benchmark (or point -p at a single task directory)
  334. harbor run -p ./WildClawBench-Harbor -a claude-code -m anthropic/claude-opus-5-5 --n-concurrent 4
  335. ```
  336. See the [WildClawBench-Harbor card](https://huggingface.co/datasets/internlm/WildClawBench-Harbor) for the task layout, environment details, and scoring.
  337. ## Check the Results
  338. After the run completes, a per-category summary and a global summary (`output/summary_all.json`) are generated automatically. Each metric is scored from `0.00` to `1.00`.
  339. Per-task results are saved under `output/<harness>/<category>/<task_id>/<model_timestamp_runid>/`:
  340. ```
  341. output/<harness>/<category>/<task_id>/<model_timestamp_runid>/
  342. ├── score.json # per-metric scores
  343. ├── usage.json # token counts, cost, elapsed time
  344. ├── agent.log # agent execution log
  345. ├── chat.jsonl # full conversation trace (OpenClaw)
  346. ├── claude_code_log/ # Claude Code session log (Claude Code)
  347. ├── codex_sessions/ # Codex session JSONLs (Codex)
  348. ├── gateway.log # gateway log (OpenClaw)
  349. └── task_output/ # files produced by the agent
  350. ```
  351. The subdirectory name is `<short_model>_<timestamp>_<runid>`, where `short_model` is the last segment of the model path (e.g. `claude-sonnet-4.6` from `openrouter/anthropic/claude-sonnet-4.6`) and `runid` is a 6-char random hex string, so parallel or repeated runs never collide.
  352. ## Agent Trajectories
  353. For independent verification, side-by-side comparison, and trace-level analysis, we release **[internlm/WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)**: graded OpenClaw trajectories covering the full 60-task suite for each evaluated model — 12 models and 720 trajectories so far, including GPT-5.6 Sol, Claude Fable 5, Claude Opus 4.8, and Kimi K3. The collection grows as new models join the leaderboard; see the [dataset card](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories) for the current roster. Every path carries an `openclaw-<model>` name:
  354. - **`traces/openclaw-<model>/<task_id>.jsonl`** — the session of one run, with images inline, for the HF **Agent Trace Viewer**: open any file, select the *Trace* tab, and step through reasoning blocks, tool calls, tool results, and token usage.
  355. - **`data/openclaw-<model>.jsonl`** — one row per run with its score, per-metric scores, status, step and tool-call counts, token usage, and the path of its session; loads directly with `load_dataset("internlm/WildClawBench-Trajectories")` and renders in the HF Dataset Viewer.
  356. Earlier evaluation details remain available on Google Drive:
  357. - overall_results.json: [Overall Results](https://drive.google.com/file/d/1EI1_ABNLwEaiguzUU7f0RuEk5KFIMLUu/view?usp=drive_link)
  358. - overall_dashboard.html: [Performance Dashboard](https://drive.google.com/file/d/1B7nStKfXeyATBM3lIv858M9FaH6QBPWU/view?usp=drive_link)
  359. - gemini 3.1 Pro Details: [Gemini 3.1 Pro](https://drive.google.com/file/d/1STpQWocGn8XeGLHFX3AZfy2TB3Q0PsfO/view?usp=drive_link)
  360. - GPT 5.4 Details: [GPT 5.4](https://drive.google.com/file/d/15zamWhsI5qJMon71N0AAs2Ysrfkns-1w/view?usp=drive_link)
  361. - Kimi K2.5 Details: [Kimi K2.5](https://drive.google.com/file/d/1Ne7CkE6gtCNR7OQR4ZKcp7qXvNmive9Q/view?usp=drive_link)
  362. - MiniMax M2.7 Details: [MiniMax M2.7](https://drive.google.com/file/d/15K65XZxkUqKWj3rp-d-gZN0DEL1iu2Kf/view?usp=drive_link)
  363. - Claude Opus 4.6 Details: [Claude 4.6 Opus](https://drive.google.com/file/d/1qCPxy0-Z-LveiVAmPTVlrh3x2fe9qlU6/view?usp=drive_link)
  364. ## Collect Trajectories with Roller
  365. [Roller](https://github.com/Valen-Team/Roller) runs this benchmark on any OpenAI-compatible endpoint and writes the trajectories in the layout of WildClawBench-Trajectories. It reads the same `.env` variables and `my_api.json` as the pipeline above:
  366. ```bash
  367. pip install roller-agent
  368. set -a; . /path/to/WildClawBench/.env; set +a
  369. # Collect and grade: one Docker container per trajectory, graded before the container is removed
  370. roller collect --bench wildclawbench --bench-root /path/to/WildClawBench \
  371. --model <provider>/<model> --model-config /path/to/WildClawBench/my_api.json \
  372. --harness openclaw --openclaw-image wildclawbench-ubuntu:v1.3 \
  373. --parallel 4 --output runs --run-name my-model
  374. # Export a Hugging Face dataset folder: traces/, data/ and a results table for the card
  375. roller export runs/my-model --format hf --output dataset
  376. ```
  377. Roller runs rollouts in parallel, retries failed ones, and resumes an interrupted run with `--resume`. `--harness` also takes `codex`, `claude-code`, and `hermes`. Besides the Hugging Face folder, it exports SFT records with `messages` and `tools`, and [ATIF](https://valen-team.github.io/roller/traces/trajectory-outputs/#atif-export) files that Harbor loads. [Roller-WildClawBench-Trajectories](https://huggingface.co/datasets/Valen-Team/Roller-WildClawBench-Trajectories) is a dataset made this way, and the [WildClawBench guide](https://valen-team.github.io/roller/benches/wildclawbench/) covers the setup.
  378. For Roller collections, two additional task images are available on GHCR:
  379. ```bash
  380. docker pull ghcr.io/valen-team/roller-wildclaw-base:1.3-clean.1
  381. docker pull ghcr.io/valen-team/roller-wildclaw-codex:1.3-codex0.149.0
  382. ```
  383. The base image retains the WildClawBench task environment with personal builder state removed; the Codex image adds Codex CLI 0.149.0. Use the base image with Roller's `--openclaw-image` and the Codex image with `--harness codex --codex-image`. See the [image recipes](https://github.com/Valen-Team/Roller/tree/main/docker/wildclawbench) and the [OpenClaw and Codex collection example](https://github.com/Valen-Team/Roller/tree/main/examples/cookbook/wildclawbench-open-models) for setup, model endpoint requirements, task selection and grading. That example uses features newer than Roller 0.4.0 and describes installation from a Git checkout.
  384. ## Personal OpenClaw Evaluation
  385. "Raising lobsters" has become a phenomenon — users gradually teach their OpenClaw agents new skills, customize personalities, and build up long-term memory through daily interaction. A natural question follows: **whose lobster is better?** Beyond bragging rights, there is real value in understanding which skill combinations, persona designs, and memory strategies actually improve agent performance on a given model. That's why we created the **Personal OpenClaw Leaderboard**. Submit your lobster's results and see how it stacks up!
  386. ```bash
  387. python eval/run_batch.py \
  388. --category all --parallel 4 \
  389. --model openrouter/xx/xxx \
  390. --lobster-name your-lobster-name \
  391. --lobster-workspace /path/to/your/workspace
  392. ```
  393. - `--lobster-name` — identifier, used in the output directory.
  394. - `--lobster-workspace` — path to your OpenClaw workspace (containing `SOUL.md`, `USER.md`, `MEMORY.md`, `skills/`, etc.).
  395. - `--lobster-env` — (optional) comma-separated env var names for skills that need API keys (e.g. `GEMINI_API_KEY,FIRECRAWL_API_KEY`). Add the actual values to `.env`.
  396. After the run completes, send the following to **wildclawbench@proton.me**:
  397. 1. Your `output/summary_all_<lobster-name>_<model>.json`
  398. 2. (Optional) A brief description of how you trained your OpenClaw (e.g. key skills, custom SOUL.md, memory strategies).
  399. We will update the leaderboard periodically.
  400. ---
  401. ## Citation
  402. If you use WildClawBench in your research, please cite it as:
  403. ```bibtex
  404. @article{ding2026wildclawbench,
  405. title={WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation},
  406. author={Ding, Shuangrui and Dai, Xuanlang and Xing, Long and Ding, Shengyuan and Liu, Ziyu and JingYi, Yang and Yang, Penghui and Zhang, Zhixiong and Wei, Xilin and Fang, Xinyu and others},
  407. journal={arXiv preprint arXiv:2605.10912},
  408. year={2026}
  409. }
  410. ```
  411. For machine-readable citation metadata, see [`CITATION.cff`](CITATION.cff). GitHub will use this file to populate the repository's "Cite this repository" panel.
  412. ---
  413. ## Contributors
  414. [Shuangrui Ding](https://mark12ding.github.io/)\* (Project Lead), [Xuanlang Dai](https://github.com/LennoxDai)\*, [Long Xing](https://github.com/Cooperx521)\*, [Shengyuan Ding](https://github.com/SYuan03), [Ziyu Liu](https://liuziyu77.github.io/), [Jingyi Yang](https://yjyddq.github.io/), [Penghui Yang](https://github.com/yph22), [Zhixiong Zhang](https://github.com/rookiexiong7), [Xilin Wei](https://github.com/wiselnn570), [Xinyu Fang](https://scholar.google.com/citations?user=QZk6nZ8AAAAJ&hl=zh-CN)
  415. Advisors: [Yubo Ma](https://mayubo2333.github.io/), [Haodong Duan](https://kennymckormick.github.io/), [Jing Shao](https://amandajshao.github.io/), [Jiaqi Wang](https://myownskyw7.github.io/), [Dahua Lin](http://dahualin.org/), [Kai Chen](https://chenkai.site/), [Yuhang Zang](https://yuhangzang.github.io/)
  416. ---
  417. ## Acknowledgements
  418. WildClawBench builds on top of the excellent open-source agent ecosystem. We gratefully acknowledge the following projects:
  419. - **[OpenClaw](https://github.com/openclaw/openclaw)**
  420. - **[Claw-Eval](https://github.com/claw-eval/claw-eval)**
  421. - **[PinchBench](https://github.com/pinchbench/skill)**
  422. - **[Hermes-Agent](https://github.com/nousresearch/hermes-agent)**
  423. ---
  424. ## Cleanup
  425. If a run is interrupted (e.g. `Ctrl+C`, terminal closed), some Docker containers may be left behind. To remove **all** WildClawBench containers when no tasks are running:
  426. ```bash
  427. for img in \
  428. wildclawbench-ubuntu:v1.3 \
  429. wildclawbench-claudecode-ubuntu:v0.2 \
  430. wildclawbench-codex-ubuntu:v0.0 \
  431. wildclawbench-hermes-agent:v0.5; do
  432. docker ps -a --filter "ancestor=$img" -q | xargs -r docker rm -f
  433. done
  434. ```
  435. To preview which containers would be removed (dry run), drop the `docker rm -f` step and use `--format "{{.Names}}\t{{.Status}}"`.
  436. ---
  437. ## License
  438. MIT — see [LICENSE](LICENSE) for details.
  439. ---
  440. ## Star History
  441. <a href="https://www.star-history.com/?repos=internlm%2FWildClawBench&type=date&legend=top-left">
  442. <picture>
  443. <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=internlm/WildClawBench&type=date&theme=dark&legend=top-left&sealed_token=uOE3PXZranBUxC9q3pvz0oRRTksC3hKub6Y69tgdPgf3FZ0kurMi1a4jofUYZLkXYgMKgc2OvBvpJ3kCRHiVM1PzjpMKlxhLwnGwYKBzeJe_wVmL2KNolg" />
  444. <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=internlm/WildClawBench&type=date&legend=top-left&sealed_token=uOE3PXZranBUxC9q3pvz0oRRTksC3hKub6Y69tgdPgf3FZ0kurMi1a4jofUYZLkXYgMKgc2OvBvpJ3kCRHiVM1PzjpMKlxhLwnGwYKBzeJe_wVmL2KNolg" />
  445. <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=internlm/WildClawBench&type=date&legend=top-left&sealed_token=uOE3PXZranBUxC9q3pvz0oRRTksC3hKub6Y69tgdPgf3FZ0kurMi1a4jofUYZLkXYgMKgc2OvBvpJ3kCRHiVM1PzjpMKlxhLwnGwYKBzeJe_wVmL2KNolg" />
  446. </picture>
  447. </a>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.