Skip to content
Back to search

pinchbench

SkillWorks with Claude CodeGitHub

Run PinchBench benchmarks to evaluate OpenClaw agent performance across real-world tasks. Use when testing model capabilities, comparing models, submitting benchmark results to the leaderboard, or checking how well your OpenClaw setup handles calendar, email, research, coding, and multi-step workflows.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/pinchbench--skill/badge.svg)](/listings/pinchbench--skill)

HTML

<a href="/listings/pinchbench--skill"><img src="/api/listings/pinchbench--skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

# Clone the skill
git clone https://github.com/pinchbench/skill.git
cd skill

# Run benchmarks with your model of choice
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4

# Or run specific tasks
./scripts/run.sh --model openrouter/openai/gpt-4o --suite task_calendar,task_stock

> **Note:** Model IDs must include their provider prefix (e.g. `openrouter/`, `anthropic/`). [OpenRouter](https://openrouter.ai) is the default provider used for routing.

**Requirements:**

- Python 3.10+ - [uv](https://docs.astral.sh/uv/) package manager - A running OpenClaw instance

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 123 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: pinchbench
  3. description: Run PinchBench benchmarks to evaluate OpenClaw agent performance across real-world tasks. Use when testing model capabilities, comparing models, submitting benchmark results to the leaderboard, or checking how well your OpenClaw setup handles calendar, email, research, coding, and multi-step workflows.
  4. metadata:
  5. author: pinchbench
  6. version: "2.0.0-rc1"
  7. homepage: https://pinchbench.com
  8. repository: https://github.com/pinchbench/skill
  9. ---
  10. # PinchBench Benchmark Skill
  11. PinchBench measures how well LLM models perform as the brain of an OpenClaw agent. Results are collected on a public leaderboard at [pinchbench.com](https://pinchbench.com).
  12. ## Prerequisites
  13. - Python 3.10+
  14. - [uv](https://docs.astral.sh/uv/) package manager
  15. - OpenClaw instance (this agent)
  16. ## Quick Start
  17. ```bash
  18. cd <skill_directory>
  19. # Run benchmark with a specific model
  20. uv run benchmark.py --model anthropic/claude-sonnet-4
  21. # Run only automated tasks (faster)
  22. uv run benchmark.py --model anthropic/claude-sonnet-4 --suite automated-only
  23. # Run specific tasks
  24. uv run benchmark.py --model anthropic/claude-sonnet-4 --suite task_calendar,task_stock
  25. # Skip uploading results
  26. uv run benchmark.py --model anthropic/claude-sonnet-4 --no-upload
  27. ```
  28. ## Available Tasks (23)
  29. | Task | Category | Description |
  30. |------|----------|-------------|
  31. | `task_sanity` | Basic | Verify agent works |
  32. | `task_calendar` | Productivity | Calendar event creation |
  33. | `task_stock` | Research | Stock price lookup |
  34. | `task_blog` | Writing | Blog post creation |
  35. | `task_weather` | Coding | Weather script |
  36. | `task_summary` | Analysis | Document summarization |
  37. | `task_events` | Research | Conference research |
  38. | `task_email` | Writing | Email drafting |
  39. | `task_memory` | Memory | Context retrieval |
  40. | `task_files` | Files | File structure creation |
  41. | `task_workflow` | Integration | Multi-step API workflow |
  42. | `task_clawdhub` | Skills | ClawHub interaction |
  43. | `task_skill_search` | Skills | Skill discovery |
  44. | `task_image_gen` | Creative | Image generation |
  45. | `task_humanizer` | Writing | Text humanization |
  46. | `task_daily_summary` | Productivity | Daily digest |
  47. | `task_email_triage` | Email | Inbox triage |
  48. | `task_email_search` | Email | Email search |
  49. | `task_market_research` | Research | Market analysis |
  50. | `task_spreadsheet_summary` | Analysis | Spreadsheet analysis |
  51. | `task_eli5_pdf_summary` | Analysis | PDF simplification |
  52. | `task_openclaw_comprehension` | Knowledge | OpenClaw docs comprehension |
  53. | `task_second_brain` | Memory | Knowledge management |
  54. ## Command Line Options
  55. | Option | Description |
  56. |--------|-------------|
  57. | `--model` | Model identifier (e.g., `anthropic/claude-sonnet-4`) |
  58. | `--suite` | `all`, `automated-only`, or comma-separated task IDs |
  59. | `--output-dir` | Results directory (default: `results/`) |
  60. | `--timeout-multiplier` | Scale task timeouts for slower models |
  61. | `--runs` | Number of runs per task for averaging |
  62. | `--no-upload` | Skip uploading to leaderboard |
  63. | `--register` | Request new API token for submissions |
  64. | `--upload FILE` | Upload previous results JSON |
  65. ## Token Registration
  66. To submit results to the leaderboard:
  67. ```bash
  68. # Register for an API token (one-time)
  69. uv run benchmark.py --register
  70. # Run benchmark (auto-uploads with token)
  71. uv run benchmark.py --model anthropic/claude-sonnet-4
  72. ```
  73. ## Results
  74. Results are saved as JSON in the output directory:
  75. ```bash
  76. # View task scores
  77. jq '.tasks[] | {task_id, score: .grading.mean}' results/0001_anthropic-claude-sonnet-4.json
  78. # Show failed tasks
  79. jq '.tasks[] | select(.grading.mean < 0.5)' results/*.json
  80. # Calculate overall score
  81. jq '{average: ([.tasks[].grading.mean] | add / length)}' results/*.json
  82. ```
  83. ## Adding Custom Tasks
  84. Create a markdown file in `tasks/` following `TASK_TEMPLATE.md`. Each task needs:
  85. - YAML frontmatter (id, name, category, grading_type, timeout)
  86. - Prompt section
  87. - Expected behavior
  88. - Grading criteria
  89. - Automated checks (Python grading function)
  90. ## Leaderboard
  91. View results at [pinchbench.com](https://pinchbench.com). The leaderboard shows:
  92. - Model rankings by overall score
  93. - Per-task breakdowns
  94. - Historical performance trends
README.md · README · 169 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # 🦀 PinchBench
  2. **Real-world benchmarks for AI coding agents**
  3. [![Leaderboard](https://img.shields.io/badge/leaderboard-pinchbench.com-blue)](https://pinchbench.com)
  4. [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
  5. <!-- task-count-badge -->![Tasks](https://img.shields.io/badge/tasks-53-orange)<!-- /task-count-badge -->
  6. > **Note:** This repository contains the benchmark skill/tasks. It is NOT the source of official leaderboard results. To add models to the official results, modify [pinchbench/scripts/default-models.yml](https://github.com/pinchbench/scripts/blob/main/default-models.yml).
  7. PinchBench measures how well LLM models perform as the brain of an [OpenClaw](https://github.com/openclaw/openclaw) agent. Instead of synthetic tests, we throw real tasks at agents: scheduling meetings, writing code, triaging email, researching topics, and managing files.
  8. Results are collected on a public leaderboard at **[pinchbench.com](https://pinchbench.com)**.
  9. ![PinchBench](pinchbench.png)
  10. ## Why PinchBench?
  11. Most LLM benchmarks test isolated capabilities. PinchBench tests what actually matters for coding agents:
  12. - **Tool usage** — Can the model call the right tools with the right parameters?
  13. - **Multi-step reasoning** — Can it chain together actions to complete complex tasks?
  14. - **Real-world messiness** — Can it handle ambiguous instructions and incomplete information?
  15. - **Practical outcomes** — Did it actually create the file, send the email, or schedule the meeting?
  16. ## Quick Start
  17. ```bash
  18. # Clone the skill
  19. git clone https://github.com/pinchbench/skill.git
  20. cd skill
  21. # Run benchmarks with your model of choice
  22. ./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
  23. # Or run specific tasks
  24. ./scripts/run.sh --model openrouter/openai/gpt-4o --suite task_calendar,task_stock
  25. ```
  26. > **Note:** Model IDs must include their provider prefix (e.g. `openrouter/`, `anthropic/`). [OpenRouter](https://openrouter.ai) is the default provider used for routing.
  27. **Requirements:**
  28. - Python 3.10+
  29. - [uv](https://docs.astral.sh/uv/) package manager
  30. - A running OpenClaw instance
  31. ## What Gets Tested
  32. <!-- task-count-text -->PinchBench includes 53 tasks across real-world categories:<!-- /task-count-text -->
  33. | Category | Tasks | What's tested |
  34. | ---------------- | --------------------------------------- | ---------------------------------------- |
  35. | **Productivity** | Calendar, daily summaries | Event creation, time parsing, scheduling |
  36. | **Research** | Stock prices, conferences, markets | Web search, data extraction, synthesis |
  37. | **Writing** | Blog posts, emails, humanization | Content generation, tone, formatting |
  38. | **Coding** | Weather scripts, file structures | Code generation, file operations |
  39. | **Analysis** | Spreadsheets, PDFs, documents | Data processing, summarization |
  40. | **Email** | Triage, search | Inbox management, filtering |
  41. | **Memory** | Context retrieval, knowledge management | Long-term memory, recall |
  42. | **Skills** | ClawHub, skill discovery | OpenClaw ecosystem integration |
  43. Each task is graded automatically, by an LLM judge, or both — ensuring both objective and nuanced evaluation.
  44. ## Submitting Results
  45. To get your results on the leaderboard:
  46. ```bash
  47. # Register for an API token (one-time)
  48. ./scripts/run.sh --register
  49. # Run benchmark — results auto-upload with your token
  50. ./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
  51. ```
  52. Skip uploading with `--no-upload` if you just want local results.
  53. ### Official Results
  54. To submit an official run (marked on the leaderboard):
  55. ```bash
  56. # Using environment variable
  57. export PINCHBENCH_OFFICIAL_KEY=your_official_key
  58. ./scripts/run.sh --model anthropic/claude-sonnet-4
  59. # Using command line flag
  60. ./scripts/run.sh --model anthropic/claude-sonnet-4 --official-key your_official_key
  61. ```
  62. ## Command Reference
  63. | Flag | Description |
  64. | ------------------------ | ----------------------------------------------------------------------------- |
  65. | `--model MODEL` | Model to test (e.g., `openrouter/anthropic/claude-sonnet-4`) |
  66. | `--judge MODEL` | Judge model for LLM grading; uses direct API when set (see below) |
  67. | `--suite SUITE` | `all`, `automated-only`, or comma-separated task IDs |
  68. | `--runs N` | Number of runs per task for averaging |
  69. | `--timeout-multiplier N` | Scale timeouts for slower models |
  70. | `--thinking LEVEL` | Reasoning depth: `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `adaptive` |
  71. | `--output-dir DIR` | Where to save results (default: `results/`) |
  72. | `--no-upload` | Skip uploading to leaderboard |
  73. | `--register` | Request an API token for submissions |
  74. | `--upload FILE` | Upload a previous results JSON |
  75. | `--official-key KEY` | Mark submission as official (or use `PINCHBENCH_OFFICIAL_KEY` env var) |
  76. ### Judge
  77. By default (no `--judge` flag), the LLM judge runs as an OpenClaw agent session. When `--judge` is specified, it calls the model API directly instead, bypassing OpenClaw personality injection.
  78. ```bash
  79. # Default: OpenClaw agent session (no --judge needed)
  80. ./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
  81. # Direct API via OpenRouter
  82. ./scripts/run.sh --model openai/gpt-4o --judge openrouter/anthropic/claude-sonnet-4-5
  83. # Direct API via Kilo Gateway
  84. ./scripts/run.sh --model openai/gpt-4o --judge kilo/anthropic/claude-sonnet-4-5
  85. # Direct API via Anthropic
  86. ./scripts/run.sh --model openai/gpt-4o --judge anthropic/claude-sonnet-4-5-20250514
  87. # Direct API via OpenAI
  88. ./scripts/run.sh --model openai/gpt-4o --judge openai/gpt-4o
  89. # Headless Claude CLI
  90. ./scripts/run.sh --model openai/gpt-4o --judge claude
  91. ```
  92. Required env vars: `OPENROUTER_API_KEY`, `KILO_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY` depending on the judge model prefix.
  93. ## Contributing Tasks
  94. We welcome new tasks! Check out [`tasks/TASK_TEMPLATE.md`](tasks/TASK_TEMPLATE.md) for the format. Good tasks are:
  95. - **Real-world** — Something an actual user would ask an agent to do
  96. - **Measurable** — Clear success criteria that can be graded
  97. - **Reproducible** — Same task should produce consistent grading
  98. - **Challenging** — Tests agent capabilities, not just LLM knowledge
  99. ### Transcript Archive
  100. Session transcripts are automatically saved to `results/{run_id}_transcripts/` alongside the results JSON. Each task's full agent conversation is preserved as a JSONL file (e.g. `task_calendar.jsonl`) for post-run analysis.
  101. ## Links
  102. - **Leaderboard:** [pinchbench.com](https://pinchbench.com)
  103. - **OpenClaw:** [github.com/openclaw/openclaw](https://github.com/openclaw/openclaw)
  104. - **Issues:** [github.com/pinchbench/skill/issues](https://github.com/pinchbench/skill/issues)
  105. ## Star History
  106. <a href="https://www.star-history.com/?repos=pinchbench%2Fskill&type=date&logscale=&legend=top-left">
  107. <picture>
  108. <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=pinchbench/skill&type=date&theme=dark&legend=top-left" />
  109. <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=pinchbench/skill&type=date&legend=top-left" />
  110. <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=pinchbench/skill&type=date&legend=top-left" />
  111. </picture>
  112. </a>
  113. ## License
  114. MIT — see [LICENSE](LICENSE) for details.
  115. ---
  116. _Claw-some AI agent testing_ 🦞

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.