coder_eval
Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/UiPath/coder_eval with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/UiPath/coder_eval'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/uipath--coder-eval)HTML
<a href="/listings/uipath--coder-eval"><img src="/api/listings/uipath--coder-eval/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
**Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the runtime of at least one coding agent** — plus that agent's own model credentials. Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra, the other two are separate CLIs you install yourself:
| Agent | `agent.type` | Runtime | Guide | | --- | --- | --- | --- | | Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) | | OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) | | Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) | | OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) |
The examples below use the default `claude-code` agent. Developed on macOS; CI runs on Linux.
git clone https://github.com/UiPath/coder_eval.git
cd coder_eval
uv sync # install the framework (add --extra codex /
# --extra antigravity for those runtimes)
cp .env.example .env # then set ANTHROPIC_API_KEY — or skip that: an
# existing Claude Code login (`claude login`) is
# picked up automatically
uv run coder-eval plan tasks/hello_date.yaml # validate (no tokens spent)
uv run coder-eval run tasks/hello_date.yaml # run your first evaluation
uv run coder-eval report runs/latest # view the resultNew here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**.
Two more extras you only need on purpose: `--extra dev` adds the contributor toolchain (pytest, ruff, pyright, pre-commit — see [CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath` SDK for local sandbox parity (public PyPI, no credentials). Without either, the framework still runs end-to-end.
**Just want the CLI, without cloning?** Install the published package — this is also what a CI job or another repo does:
uv tool install coder-eval # puts the `coder-eval` CLI on your PATH,
# in its own isolated environment
uv tool install "coder-eval[codex,antigravity]" # same, with agent extras
coder-eval --version # verify the installTo add it as a project dependency instead: `uv add coder-eval` or `pip install coder-eval`. In a real CI gate, pin to a specific released version so a harness upgrade can't silently move your results. (The example `tasks/` live in this repo — clone it or point the CLI at your own task files.) See [Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for the full setup.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 35 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
- "name": "coder-eval",
- "owner": {
- "name": "UiPath",
- "email": "coder-eval@uipath.com",
- "url": "https://github.com/UiPath/coder_eval"
- },
- "description": "Test whether your Claude Code skills actually trigger, and benchmark AI coding agents against your own tasks.",
- "plugins": [
- {
- "name": "coder-eval",
- "displayName": "Coder Eval",
- "source": "./plugins/coder-eval",
- "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
- "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
- "homepage": "https://coder-eval.com",
- "repository": "https://github.com/UiPath/coder_eval",
- "license": "Apache-2.0",
- "category": "testing",
- "keywords": [
- "claude-code-skills",
- "skill-testing",
- "skill-activation",
- "evaluation",
- "benchmark",
- "agent-testing",
- "github-actions",
- "sandbox",
- "llm-judge",
- "ci"
- ]
- }
- ]
- }
README.md · README · 386 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # Coder Eval — evaluate and benchmark AI coding agents and their skills
- [](https://pypi.org/project/coder-eval/)
- [](https://github.com/marketplace/actions/coder_eval)
- [](docs/PLUGIN.md)
- [](https://coder-eval.com)
- [](LICENSE)
- [](https://www.python.org/downloads/)
- [](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)
- <p align="center">
- <strong>Playwright for coding agents</strong> — one declarative test file, any agent
- runtime, a real sandbox, and a pass/fail gate in CI.
- </p>
- **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an
- open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
- agents and their skills** — built for benchmark authors, CLI builders, and skill
- builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
- agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**,
- **OpenCode**, or **Pi** — in a sandbox against declarative YAML tasks, then scores the files and
- commands it actually produced. Changing harness is one field (`agent.type`); the
- tasks, criteria, scoring, telemetry, and reports stay the same.
- Reach for it when you want to **benchmark agents on your own domain tasks**,
- **test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
- vs. Codex vs. Gemini vs. OpenCode vs. Pi** (or model vs. model, prompt vs. prompt), or
- **gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
- SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
- and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
- check, an A/B experiment layer, and per-tool cost telemetry, over whatever work *you*
- care about. See [How it compares](https://coder-eval.com/docs/comparison).
- 📚 **Full docs:** **[coder-eval.com/docs](https://coder-eval.com/docs)**.
- <p align="center">
- <img src="docs/assets/hero.gif" alt="Coder Eval running the hello_date task: a sandboxed agent writes and runs a script from a YAML task, then the scored result is browsed in evalboard" width="100%">
- </p>
- - **Declarative YAML tasks** with pinned dependencies and clear success criteria
- - **Sandboxed execution** in isolated environments with resource limits
- - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
- - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
- - **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
- - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
- - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
- ## Watch the intro
- <div align="center">
- <video src="https://github.com/user-attachments/assets/6dc1c88e-e5b7-4fc5-ba9c-4b117efc8b54" controls>
- <a href="https://www.youtube.com/watch?v=Iyq-5m1CnuI">
- <img src="https://img.youtube.com/vi/Iyq-5m1CnuI/maxresdefault.jpg" alt="Video: Coder Eval — UiPath open-source framework to test AI coding agents" width="70%">
- </a>
- </video>
- </div>
- 🔊 **Turn the sound on** — GitHub's inline player always starts muted.
- ▶ Also on YouTube: **[Coder Eval: UiPath open-source framework to test AI Coding Agents](https://www.youtube.com/watch?v=Iyq-5m1CnuI)**
- — what the framework does, and how a run works end to end.
- ## What you can do with it
- - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
- - **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
- - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
- - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
- - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
- - **Bring your own dataset** — fan one task out over many rows for larger benchmark suites
- > **Keeping skills fresh?** Run Coder Eval as a scheduled GitHub Actions job so your
- > skills are continuously re-evaluated against the latest model — a skill that quietly
- > stops triggering surfaces as a failing criterion before your users hit it. See
- > **[Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md)**.
- ## Quick Start
- **Prerequisites:** Python 3.13+, [uv](https://docs.astral.sh/uv/) 0.8+, and **the
- runtime of at least one coding agent** — plus that agent's own model credentials.
- Pick the agent you want to evaluate; two of the four ship with a Coder Eval extra,
- the other two are separate CLIs you install yourself:
- | Agent | `agent.type` | Runtime | Guide |
- | --- | --- | --- | --- |
- | Claude Code (default) | `claude-code` | `brew install claude` — separate CLI | [Claude Code](docs/agents/CLAUDE_CODE.md) |
- | OpenAI Codex | `codex` | `uv sync --extra codex` — the extra ships the Codex SDK + CLI | [Codex](docs/agents/CODEX.md) |
- | Google Antigravity (Gemini) | `antigravity` | `uv sync --extra antigravity` — the extra ships the harness binary | [Antigravity](docs/agents/ANTIGRAVITY.md) |
- | OpenCode (open-weight models) | `opencode` | `npm install -g opencode-ai` — separate CLI | [OpenCode](docs/agents/OPENCODE.md) |
- The examples below use the default `claude-code` agent. Developed on macOS; CI runs on
- Linux.
- ```bash
- git clone https://github.com/UiPath/coder_eval.git
- cd coder_eval
- uv sync # install the framework (add --extra codex /
- # --extra antigravity for those runtimes)
- cp .env.example .env # then set ANTHROPIC_API_KEY — or skip that: an
- # existing Claude Code login (`claude login`) is
- # picked up automatically
- uv run coder-eval plan tasks/hello_date.yaml # validate (no tokens spent)
- uv run coder-eval run tasks/hello_date.yaml # run your first evaluation
- uv run coder-eval report runs/latest # view the result
- ```
- New here? Follow **[Tutorial 01 — Your First Evaluation](docs/tutorials/01-first-evaluation.md)**.
- Two more extras you only need on purpose: `--extra dev` adds the contributor
- toolchain (pytest, ruff, pyright, pre-commit — see
- [CONTRIBUTING.md](CONTRIBUTING.md)), and `--extra uipath` adds the in-host `uipath`
- SDK for local sandbox parity (public PyPI, no credentials). Without either, the
- framework still runs end-to-end.
- **Just want the CLI, without cloning?** Install the published package — this is also
- what a CI job or another repo does:
- ```bash
- uv tool install coder-eval # puts the `coder-eval` CLI on your PATH,
- # in its own isolated environment
- uv tool install "coder-eval[codex,antigravity]" # same, with agent extras
- coder-eval --version # verify the install
- ```
- To add it as a project dependency instead: `uv add coder-eval` or
- `pip install coder-eval`. In a real CI gate, pin to a specific released version
- so a harness upgrade can't silently move your results. (The example `tasks/`
- live in this repo — clone it or point the CLI at your own task files.) See
- [Tutorial 02 — Running Coder Eval in CI](docs/tutorials/02-ci-pipeline.md) for
- the full setup.
- ## Use inside Claude Code
- Coder Eval evaluates any of the supported agents, and it also ships an authoring
- front-end for one of them: this repo is a **Claude Code plugin marketplace**, so the
- whole loop — scaffold a suite, author a task, check whether a skill triggers, read the
- results — runs inside Claude Code. The suites you author this way run on every harness:
- ```
- /plugin marketplace add UiPath/coder_eval
- /plugin install coder-eval@coder-eval
- ```
- That adds six slash commands: `/coder-eval:init`, `/coder-eval:check-skill`,
- `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze` and
- `/coder-eval:ci`. They drive the `coder-eval` CLI, so install it too
- (`uv tool install coder-eval`). See [Claude Code Plugin](docs/PLUGIN.md).
- ## Use as a GitHub Action
- A composite action — on the Marketplace as
- [**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs
- `coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
- JUnit XML report, reports where its artifacts landed, and fails the step on any
- task failure:
- ```yaml
- - uses: actions/setup-node@v4 # the claude-code agent needs the Claude CLI…
- with: { node-version: '20' }
- - run: npm install -g @anthropic-ai/claude-code
- - uses: UiPath/coder_eval@v0 # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
- id: eval
- with:
- args: |
- tests/tasks/**/*.yaml
- --model
- claude-sonnet-5
- env: |
- ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
- ```
- Eight inputs, and **none of them is a `coder-eval run` flag**. The CLI has 21;
- GitHub silently ignores an input the referenced tag does not define, so a
- forwarding input that is mistyped or newer than your pin yields a run that
- measured something else and still exits 0. A wrong CLI flag is a hard error. So
- flags and task globs all go through `args`, and an input exists only where the
- action does something with the value besides pass it along.
- | Input | Default | Purpose |
- | --- | --- | --- |
- | `args` | — | Task paths/globs and every flag for `coder-eval run`, one argument per line, verbatim |
- | `version` | pinned release | PyPI version, or `local` to install from the checkout |
- | `extras` | — | coder-eval extras, composed into the install requirement (`codex`, `antigravity,litellm`) |
- | `extra-packages` | — | Extra requirements installed into coder-eval's environment (`--with`), one per line |
- | `install-flags` | — | Flags for `uv tool install`, one per line (`--prerelease=allow`, `--extra-index-url …`) |
- | `env` | — | Credentials/backend passthrough: newline-separated `NAME=VALUE` pairs, exported for the run step only |
- | `working-directory` | `.` | Directory every step of the action runs in |
- | `run-dir` | `runs/ci` | Run directory; also where the reports are written |
- Outputs: `run-dir`, `junit-path` (`<run-dir>/junit.xml`) and `run-md-path`
- (`<run-dir>/run.md`). The action writes nothing to the job summary — a consumer
- that has to redact the report first cannot undo a write that already happened:
- ```yaml
- - if: always()
- run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
- - uses: mikepenz/action-junit-report@v5
- if: always()
- with:
- report_paths: ${{ steps.eval.outputs.junit-path }}
- ```
- **Credentials and backend config** are the sole responsibility of `env` — a
- passthrough exported for the run step only (never written to `$GITHUB_ENV`, so
- it can't leak into later steps). Set whatever the run needs, Anthropic or not:
- ```yaml
- - uses: UiPath/coder_eval@v0
- with:
- args: tests/tasks/**/*.yaml
- env: |
- API_BACKEND=bedrock
- AWS_BEARER_TOKEN_BEDROCK=${{ secrets.BEDROCK_TOKEN }}
- ```
- The step's exit code is coder-eval's own: non-zero on any failed task.
- > **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
- > it installs `coder-eval` but no coding-agent runtime, which is why the example
- > above starts with `actions/setup-node` +
- > `npm install -g @anthropic-ai/claude-code`: the default `claude-code` agent
- > needs the `claude` CLI on `PATH` before the action runs. Swap those steps for
- > your own agent's runtime as needed.
- > **Security.** Evaluated tasks execute agent-generated code. Do **not** run this
- > action under `pull_request_target` with secrets exposed to untrusted fork PRs —
- > use `pull_request` and gate on the same-repo condition, as this repo's own
- > dogfood job does.
- ## Telemetry
- > 📊 **Usage telemetry is on by default.** `coder-eval` sends **anonymous** usage
- > telemetry (command names, outcomes, counts, durations, an anonymous install id,
- > platform info) to help improve the tool. It **never** captures prompts, file
- > contents, or repo paths, and prints a one-time notice on first run. **To disable
- > it, set `TELEMETRY_ENABLED=false`** in your `.env` or environment. See
- > [Usage Telemetry](docs/USER_GUIDE.md#usage-telemetry) for details and how to route
- > it to your own resource.
- ## Documentation
- <!-- docs-index:start -->
- | Guide | What's in it |
- | --- | --- |
- | [Tutorials](docs/tutorials/README.md) | Step-by-step walkthroughs — start here |
- | [User Guide](docs/USER_GUIDE.md) | Full CLI, configuration, output, and environment-variable reference |
- | [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) | The task-file schema — all criterion types, scoring, templates |
- | [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
- | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
- | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
- | [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
- | [Pi](docs/agents/PI.md) | Running the Pi agent on open-weight models |
- | [Delegate](docs/agents/DELEGATE.md) | Running UiPath Autopilot's Delegate agent |
- | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
- | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
- | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
- | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
- | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
- | [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, run reports |
- | [Claude Code Plugin](docs/PLUGIN.md) | Install the Claude Code plugin — author, run, and analyze suites from inside the agent |
- | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
- | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |
- | [How It Compares](docs/comparison.md) | vs. SWE-bench, SkillsBench, Harbor, OpenAI Evals, hand-rolled scripts |
- <!-- docs-index:end -->
- | Repo doc | What's in it |
- | --- | --- |
- | [CLAUDE.md](CLAUDE.md) | Architecture, key patterns, and extension points |
- | [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, quality bar, and how to contribute |
- ## How it compares
- - **vs. fixed benchmarks (SWE-bench, SkillsBench)** — they score a canonical dataset;
- Coder Eval scores *your* tasks with continuous 0.0–1.0 weighted criteria (and can
- still wrap a fixed dataset via [Bring Your Own Dataset](docs/DATASETS.md)).
- - **vs. large-scale / RL harnesses (Harbor)** — Harbor targets scale and RL rollouts;
- Coder Eval targets weighted, skill-aware suites gated in CI.
- - **vs. model-output eval tools (OpenAI Evals)** — they grade model text; Coder Eval
- runs a full agent in a sandbox and scores the files and commands it produced.
- - **vs. hand-rolled scripts** — reproducible sandboxes, weighted criteria,
- cost/token telemetry, A/B experiments, and CI-ready pass/fail gates out of the box.
- See the full [comparison — with sources](https://coder-eval.com/docs/comparison).
- ## Task Definition
- A task is a YAML file: a prompt, the agent config, a sandbox, and success criteria.
- ```yaml
- task_id: "hello_world"
- description: "Create a Python script that prints Hello, World!"
- initial_prompt: "Create hello.py that prints 'Hello, World!'"
- agent:
- type: "claude-code"
- permission_mode: "acceptEdits"
- allowed_tools: ["Read", "Write", "Bash"]
- sandbox:
- driver: "tempdir"
- python: {}
- success_criteria:
- - type: "file_exists"
- path: "hello.py"
- description: "hello.py must be created"
- - type: "run_command"
- command: "python hello.py"
- timeout: 10
- description: "Script must execute successfully"
- ```
- `agent.type` is the only harness-specific line: swap it for `codex`, `antigravity`, or
- `opencode` — or override it per run with `coder-eval run … -D agent.type=opencode` — and
- the same criteria score the same way. Tasks can omit the `agent` section entirely —
- defaults resolve from the experiment layer (`experiments/default.yaml`). For the full
- schema and every criterion type, see the
- [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md); for what each `run_limits` field
- means on each harness, see [Run-Limit Parity](docs/agents/HARNESS_PARITY.md).
- > **Tip:** With the [Claude Code plugin](docs/PLUGIN.md) installed, use
- > `/coder-eval:task` to scaffold a task from a natural-language description, and
- > `/coder-eval:analyze runs/latest` to get improvement suggestions from a completed run.
- ## Development
- ```bash
- make install # package + dev + [uipath] deps + pre-commit hooks
- make verify # format + lint + typecheck + test + coverage (CI equivalent)
- ```
- Run `make verify` before pushing — it mirrors CI (80% coverage threshold). See
- [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow, commit conventions, and
- extension points (new criteria, new agents).
- ## Known limits & non-goals
- - **Not a fixed benchmark or leaderboard** — Coder Eval scores *your* tasks and ships
- example tasks, not a canonical scored dataset.
- - **Tasks execute real code** — run untrusted tasks only under the container driver
- (see [Docker Isolation](docs/DOCKER_ISOLATION.md)); the `tempdir` driver is not a
- security boundary.
- - **Bring your own agent runtime and model credentials** — Coder Eval never supplies
- model access, and it only ships an agent runtime where an extra says so (`codex`,
- `antigravity`); Claude Code and OpenCode are separate CLIs. Supply the runtime (see
- [Quick Start](#quick-start)) and the keys it needs — Anthropic, Bedrock, OpenAI,
- Gemini, or an OpenRouter key for open-weight models via OpenCode.
- - **Python 3.13+ only.**
- ## Support & security
- - **Security vulnerabilities** — report privately via [SECURITY.md](SECURITY.md); never open a public issue.
- - **Bugs & questions** — open a [GitHub issue](https://github.com/UiPath/coder_eval/issues).
- - **Everything else** — reach the maintainers privately at **coder-eval@uipath.com**.
- ## Citation
- If you use Coder Eval in your research, please cite it:
- ```bibtex
- @software{coder_eval,
- author = {Religa, Tomasz and Shanbhogue, Akshaya Vishnu Kudlu and Li, Bai and Balsells-Rodas, Carles and Ma, Tao},
- title = {{Coder Eval}: Evaluate and Benchmark {AI} Coding Agents and Their Skills},
- year = {2026},
- organization = {UiPath},
- publisher = {GitHub},
- url = {https://github.com/UiPath/coder_eval},
- license = {Apache-2.0}
- }
- ```
- ## License
- © 2026 UiPath. Licensed under the Apache License, Version 2.0 — see
- [LICENSE](LICENSE) and [NOTICE](NOTICE).
- ## Acknowledgments
- Built with [Pydantic](https://pydantic.dev/), [Typer](https://typer.tiangolo.com/),
- and [Rich](https://rich.readthedocs.io/), on top of the harnesses it drives — the
- [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk), the
- [Codex SDK](https://github.com/openai/codex), [Google Antigravity](https://antigravity.google/),
- and [OpenCode](https://opencode.ai).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.