Skip to content
Back to search

coder_eval

Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/UiPath/coder_eval with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/UiPath/coder_eval'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/uipath--coder-eval/badge.svg)](/listings/uipath--coder-eval)

HTML

<a href="/listings/uipath--coder-eval"><img src="/api/listings/uipath--coder-eval/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the runtime of at least one coding agent** — plus that agent's own model credentials. Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra, the other two are separate CLIs you install yourself:

| Agent | `agent.type` | Runtime | Guide | | --- | --- | --- | --- | | Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) | | OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) | | Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) | | OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) |

The examples below use the default `claude-code` agent. Developed on macOS; CI runs on Linux.

git clone https://github.com/UiPath/coder_eval.git
cd coder_eval

uv sync                      # install the framework (add --extra codex /
                             # --extra antigravity for those runtimes)
cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip that: an
                             # existing Claude Code login (`claude login`) is
                             # picked up automatically

uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
uv run coder-eval report runs/latest           # view the result

New here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**.

Two more extras you only need on purpose: `--extra dev` adds the contributor toolchain (pytest, ruff, pyright, pre-commit — see [CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath` SDK for local sandbox parity (public PyPI, no credentials). Without either, the framework still runs end-to-end.

**Just want the CLI, without cloning?** Install the published package — this is also what a CI job or another repo does:

uv tool install coder-eval    # puts the `coder-eval` CLI on your PATH,
                              # in its own isolated environment

uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
coder-eval --version                              # verify the install

To add it as a project dependency instead: `uv add coder-eval` or `pip install coder-eval`. In a real CI gate, pin to a specific released version so a harness upgrade can't silently move your results. (The example `tasks/` live in this repo — clone it or point the CLI at your own task files.) See [Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for the full setup.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 35 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
  3. "name": "coder-eval",
  4. "owner": {
  5. "name": "UiPath",
  6. "email": "coder-eval@uipath.com",
  7. "url": "https://github.com/UiPath/coder_eval"
  8. },
  9. "description": "Test whether your Claude Code skills actually trigger, and benchmark AI coding agents against your own tasks.",
  10. "plugins": [
  11. {
  12. "name": "coder-eval",
  13. "displayName": "Coder Eval",
  14. "source": "./plugins/coder-eval",
  15. "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
  16. "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
  17. "homepage": "https://coder-eval.com",
  18. "repository": "https://github.com/UiPath/coder_eval",
  19. "license": "Apache-2.0",
  20. "category": "testing",
  21. "keywords": [
  22. "claude-code-skills",
  23. "skill-testing",
  24. "skill-activation",
  25. "evaluation",
  26. "benchmark",
  27. "agent-testing",
  28. "github-actions",
  29. "sandbox",
  30. "llm-judge",
  31. "ci"
  32. ]
  33. }
  34. ]
  35. }
README.md · README · 386 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Coder Eval — evaluate and benchmark AI coding agents and their skills
  2. [![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
  3. [![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
  4. [![Claude Code plugin](https://img.shields.io/badge/claude__code__plugin-coder--eval-d97757.svg)](docs/PLUGIN.md)
  5. [![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
  6. [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
  7. [![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
  8. [![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
  9. <p align="center">
  10. <strong>Playwright for coding agents</strong> — one declarative test file, any agent
  11. runtime, a real sandbox, and a pass/fail gate in CI.
  12. </p>
  13. **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an
  14. open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
  15. agents and their skills** — built for benchmark authors, CLI builders, and skill
  16. builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
  17. agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**,
  18. **OpenCode**, or **Pi** — in a sandbox against declarative YAML tasks, then scores the files and
  19. commands it actually produced. Changing harness is one field (`agent.type`); the
  20. tasks, criteria, scoring, telemetry, and reports stay the same.
  21. Reach for it when you want to **benchmark agents on your own domain tasks**,
  22. **test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
  23. vs. Codex vs. Gemini vs. OpenCode vs. Pi** (or model vs. model, prompt vs. prompt), or
  24. **gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
  25. SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
  26. and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
  27. check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you*
  28. care about. See [How it compares](https://coder-eval.com/docs/comparison).
  29. 📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
  30. <p align="center">
  31. <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
  32. </p>
  33. - **Declarative YAML tasks** with pinned dependencies and clear success criteria
  34. - **Sandboxed execution** in isolated environments with resource limits
  35. - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
  36. - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
  37. - **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
  38. - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
  39. - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
  40. ## Watch the intro
  41. <div align="center">
  42. <video src="https://github.com/user-attachments/assets/6dc1c88e-e5b7-4fc5-ba9c-4b117efc8b54" controls>
  43. <a href="https://www.youtube.com/watch?v=Iyq-5m1CnuI">
  44. <img src="https://img.youtube.com/vi/Iyq-5m1CnuI/maxresdefault.jpg" alt="Video: Coder Eval — UiPath open-source framework to test AI coding agents" width="70%">
  45. </a>
  46. </video>
  47. </div>
  48. 🔊 **Turn the sound on** — GitHub's inline player always starts muted.
  49. ▶ Also on YouTube: **[Coder Eval: UiPath open-source framework to test AI Coding Agents](https://www.youtube.com/watch?v=Iyq-5m1CnuI)**
  50. — what the framework does, and how a run works end to end.
  51. ## What you can do with it
  52. - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
  53. - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
  54. - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
  55. - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
  56. - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
  57. - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
  58. > **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
  59. > skills are continuously re-evaluated against the latest model — a skill that quietly
  60. > stops triggering surfaces as a failing criterion before your users hit it. See
  61. > **[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md)**.
  62. ## Quick Start
  63. **Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the
  64. runtime of at least one coding agent** — plus that agent's own model credentials.
  65. Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra,
  66. the other two are separate CLIs you install yourself:
  67. | Agent | `agent.type` | Runtime | Guide |
  68. | --- | --- | --- | --- |
  69. | Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) |
  70. | OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) |
  71. | Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) |
  72. | OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) |
  73. The examples below use the default `claude-code` agent. Developed on macOS; CI runs on
  74. Linux.
  75. ```bash
  76. git clone https://github.com/UiPath/coder_eval.git
  77. cd coder_eval
  78. uv sync # install the framework (add --extra codex /
  79. # --extra antigravity for those runtimes)
  80. cp .env.example .env # then set ANTHROPIC_API_KEY — or skip that: an
  81. # existing Claude Code login (`claude login`) is
  82. # picked up automatically
  83. uv run coder-eval plan tasks/hello_date.yaml # validate (no tokens spent)
  84. uv run coder-eval run tasks/hello_date.yaml # run your first evaluation
  85. uv run coder-eval report runs/latest # view the result
  86. ```
  87. New here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**.
  88. Two more extras you only need on purpose: `--extra dev` adds the contributor
  89. toolchain (pytest, ruff, pyright, pre-commit — see
  90. [CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath`
  91. SDK for local sandbox parity (public PyPI, no credentials). Without either, the
  92. framework still runs end-to-end.
  93. **Just want the CLI, without cloning?** Install the published package — this is also
  94. what a CI job or another repo does:
  95. ```bash
  96. uv tool install coder-eval # puts the `coder-eval` CLI on your PATH,
  97. # in its own isolated environment
  98. uv tool install "coder-eval[codex,antigravity]" # same, with agent extras
  99. coder-eval --version # verify the install
  100. ```
  101. To add it as a project dependency instead: `uv add coder-eval` or
  102. `pip install coder-eval`. In a real CI gate, pin to a specific released version
  103. so a harness upgrade can't silently move your results. (The example `tasks/`
  104. live in this repo — clone it or point the CLI at your own task files.) See
  105. [Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for
  106. the full setup.
  107. ## Use inside Claude Code
  108. Coder Eval evaluates any of the supported agents, and it also ships an authoring
  109. front-end for one of them: this repo is a **Claude Code plugin marketplace**, so the
  110. whole loop — scaffold a suite, author a task, check whether a skill triggers, read the
  111. results — runs inside Claude Code. The suites you author this way run on every harness:
  112. ```
  113. /plugin marketplace add UiPath/coder_eval
  114. /plugin install coder-eval@coder-eval
  115. ```
  116. That adds six slash commands: `/coder-eval:init`, `/coder-eval:check-skill`,
  117. `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze` and
  118. `/coder-eval:ci`. They drive the `coder-eval` CLI, so install it too
  119. (`uv tool install coder-eval`). See [Claude Code Plugin](docs/PLUGIN.md).
  120. ## Use as a GitHub Action
  121. A composite action — on the Marketplace as
  122. [**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs
  123. `coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
  124. JUnit XML report, reports where its artifacts landed, and fails the step on any
  125. task failure:
  126. ```yaml
  127. - uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI…
  128. with: { node-version: '20' }
  129. - run: npm install -g @anthropic-ai/claude-code
  130. - uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
  131. id: eval
  132. with:
  133. args: |
  134. tests/tasks/**/*.yaml
  135. --model
  136. claude-sonnet-5
  137. env: |
  138. ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
  139. ```
  140. Eight inputs, and **none of them is a `coder-eval run` flag**. The CLI has 21;
  141. GitHub silently ignores an input the referenced tag does not define, so a
  142. forwarding input that is mistyped or newer than your pin yields a run that
  143. measured something else and still exits 0. A wrong CLI flag is a hard error. So
  144. flags and task globs all go through `args`, and an input exists only where the
  145. action does something with the value besides pass it along.
  146. | Input | Default | Purpose |
  147. | --- | --- | --- |
  148. | `args` | — | Task paths/globs and every flag for `coder-eval run`, one argument per line, verbatim |
  149. | `version` | pinned release | PyPI version, or `local` to install from the checkout |
  150. | `extras` | — | coder-eval extras, composed into the install requirement (`codex`, `antigravity,litellm`) |
  151. | `extra-packages` | — | Extra requirements installed into coder-eval's environment (`--with`), one per line |
  152. | `install-flags` | — | Flags for `uv tool install`, one per line (`--prerelease=allow`, `--extra-index-url …`) |
  153. | `env` | — | Credentials/backend passthrough: newline-separated `NAME=VALUE` pairs, exported for the run step only |
  154. | `working-directory` | `.` | Directory every step of the action runs in |
  155. | `run-dir` | `runs/ci` | Run directory; also where the reports are written |
  156. Outputs: `run-dir`, `junit-path` (`<run-dir>/junit.xml`) and `run-md-path`
  157. (`<run-dir>/run.md`). The action writes nothing to the job summary — a consumer
  158. that has to redact the report first cannot undo a write that already happened:
  159. ```yaml
  160. - if: always()
  161. run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
  162. - uses: mikepenz/action-junit-report@v5
  163. if: always()
  164. with:
  165. report_paths: ${{ steps.eval.outputs.junit-path }}
  166. ```
  167. **Credentials and backend config** are the sole responsibility of `env` — a
  168. passthrough exported for the run step only (never written to `$GITHUB_ENV`, so
  169. it can't leak into later steps). Set whatever the run needs, Anthropic or not:
  170. ```yaml
  171. - uses: UiPath/coder_eval@v0
  172. with:
  173. args: tests/tasks/**/*.yaml
  174. env: |
  175. API_BACKEND=bedrock
  176. AWS_BEARER_TOKEN_BEDROCK=${{ secrets.BEDROCK_TOKEN }}
  177. ```
  178. The step's exit code is coder-eval's own: non-zero on any failed task.
  179. > **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
  180. > it installs `coder-eval` but no coding-agent runtime, which is why the example
  181. > above starts with `actions/setup-node` +
  182. > `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent
  183. > needs the `claude` CLI on `PATH` before the action runs. Swap those steps for
  184. > your own agent's runtime as needed.
  185. > **Security.** Evaluated tasks execute agent-generated code. Do **not** run this
  186. > action under `pull_request_target` with secrets exposed to untrusted fork PRs —
  187. > use `pull_request` and gate on the same-repo condition, as this repo's own
  188. > dogfood job does.
  189. ## Telemetry
  190. > 📊 **Usage telemetry is on by default.** `coder-eval` sends **anonymous** usage
  191. > telemetry (command names, outcomes, counts, durations, an anonymous install id,
  192. > platform info) to help improve the tool. It **never** captures prompts, file
  193. > contents, or repo paths, and prints a one-time notice on first run. **To disable
  194. > it, set `TELEMETRY_ENABLED=false`** in your `.env` or environment. See
  195. > [Usage Telemetry](docs/USER_GUIDE.md#usage-telemetry) for details and how to route
  196. > it to your own resource.
  197. ## Documentation
  198. <!-- docs-index:start -->
  199. | Guide | What's in it |
  200. | --- | --- |
  201. | [Tutorials](docs/tutorials/README.md) | Step-by-step walkthroughs — start here |
  202. | [User Guide](docs/USER_GUIDE.md) | Full CLI, configuration, output, and environment-variable reference |
  203. | [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates |
  204. | [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
  205. | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
  206. | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
  207. | [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
  208. | [Pi](docs/agents/PI.md) | Running the Pi agent on open-weight models |
  209. | [Delegate](docs/agents/DELEGATE.md) | Running UiPath Autopilot's Delegate agent |
  210. | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
  211. | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
  212. | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
  213. | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
  214. | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
  215. | [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, run reports |
  216. | [Claude Code Plugin](docs/PLUGIN.md) | Install the Claude Code plugin — author, run, and analyze suites from inside the agent |
  217. | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
  218. | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
  219. | [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
  220. <!-- docs-index:end -->
  221. | Repo doc | What's in it |
  222. | --- | --- |
  223. | [CLAUDE.md](CLAUDE.md) | Architecture, key patterns, and extension points |
  224. | [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, quality bar, and how to contribute |
  225. ## How it compares
  226. - **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
  227. Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
  228. still wrap a fixed dataset via [Bring Your Own Dataset](docs/DATASETS.md)).
  229. - **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
  230. Coder Eval targets weighted, skill-aware suites gated in CI.
  231. - **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
  232. runs a full agent in a sandbox and scores the files and commands it produced.
  233. - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
  234. cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
  235. See the full [comparison — with sources](https://coder-eval.com/docs/comparison).
  236. ## Task Definition
  237. A task is a YAML file: a prompt, the agent config, a sandbox, and success criteria.
  238. ```yaml
  239. task_id: "hello_world"
  240. description: "Create a Python script that prints Hello, World!"
  241. initial_prompt: "Create hello.py that prints 'Hello, World!'"
  242. agent:
  243. type: "claude-code"
  244. permission_mode: "acceptEdits"
  245. allowed_tools: ["Read", "Write", "Bash"]
  246. sandbox:
  247. driver: "tempdir"
  248. python: {}
  249. success_criteria:
  250. - type: "file_exists"
  251. path: "hello.py"
  252. description: "hello.py must be created"
  253. - type: "run_command"
  254. command: "python hello.py"
  255. timeout: 10
  256. description: "Script must execute successfully"
  257. ```
  258. `agent.type` is the only harness-specific line: swap it for `codex`, `antigravity`, or
  259. `opencode` — or override it per run with `coder-eval run … -D agent.type=opencode` — and
  260. the same criteria score the same way. Tasks can omit the `agent` section entirely —
  261. defaults resolve from the experiment layer (`experiments/default.yaml`). For the full
  262. schema and every criterion type, see the
  263. [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md); for what each `run_limits` field
  264. means on each harness, see [Run-Limit Parity](docs/agents/HARNESS_PARITY.md).
  265. > **Tip:** With the [Claude Code plugin](docs/PLUGIN.md) installed, use
  266. > `/coder-eval:task` to scaffold a task from a natural-language description, and
  267. > `/coder-eval:analyze runs/latest` to get improvement suggestions from a completed run.
  268. ## Development
  269. ```bash
  270. make install # package + dev + [uipath] deps + pre-commit hooks
  271. make verify # format + lint + typecheck + test + coverage (CI equivalent)
  272. ```
  273. Run `make verify` before pushing — it mirrors CI (80% coverage threshold). See
  274. [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow, commit conventions, and
  275. extension points (new criteria, new agents).
  276. ## Known limits & non-goals
  277. - **Not a fixed benchmark or leaderboard** — Coder Eval scores *your* tasks and ships
  278. example tasks, not a canonical scored dataset.
  279. - **Tasks execute real code** — run untrusted tasks only under the container driver
  280. (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
  281. security boundary.
  282. - **Bring your own agent runtime and model credentials** — Coder Eval never supplies
  283. model access, and it only ships an agent runtime where an extra says so (`codex`,
  284. `antigravity`); Claude Code and OpenCode are separate CLIs. Supply the runtime (see
  285. [Quick Start](#quick-start)) and the keys it needs — Anthropic, Bedrock, OpenAI,
  286. Gemini, or an OpenRouter key for open-weight models via OpenCode.
  287. - **Python 3.13+ only.**
  288. ## Support & security
  289. - **Security vulnerabilities** — report privately via [SECURITY.md](SECURITY.md); never open a public issue.
  290. - **Bugs & questions** — open a [GitHub issue](https://github.com/UiPath/coder_eval/issues).
  291. - **Everything else** — reach the maintainers privately at **coder-eval@uipath.com**.
  292. ## Citation
  293. If you use Coder Eval in your research, please cite it:
  294. ```bibtex
  295. @software{coder_eval,
  296. author = {Religa, Tomasz and Shanbhogue, Akshaya Vishnu Kudlu and Li, Bai and Balsells-Rodas, Carles and Ma, Tao},
  297. title = {{Coder Eval}: Evaluate and Benchmark {AI} Coding Agents and Their Skills},
  298. year = {2026},
  299. organization = {UiPath},
  300. publisher = {GitHub},
  301. url = {https://github.com/UiPath/coder_eval},
  302. license = {Apache-2.0}
  303. }
  304. ```
  305. ## License
  306. © 2026 UiPath. Licensed under the Apache License, Version 2.0 — see
  307. [LICENSE](LICENSE) and [NOTICE](NOTICE).
  308. ## Acknowledgments
  309. Built with [Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/),
  310. and [Rich](https://rich.readthedocs.io/), on top of the harnesses it drives — the
  311. [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk), the
  312. [Codex SDK](https://github.com/openai/codex), [Google Antigravity](https://antigravity.google/),
  313. and [OpenCode](https://opencode.ai).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.