Skip to content
Back to search

verdictpack

Measure the measurer: six agent skills for validating graders, scoring agent runs, auditing eval coverage, designing near-miss tests, calibrating LLM judges, and layering an eval program.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ranadhir19--verdictpack/badge.svg)](/listings/ranadhir19--verdictpack)

HTML

<a href="/listings/ranadhir19--verdictpack"><img src="/api/listings/ranadhir19--verdictpack/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/ranadhir19/verdictpack.git
cp -r verdictpack/skills/* ~/.claude/skills/        # personal

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 113 lines
  1. ![verdictpack — measure the measurer](assets/banner.svg)
  2. # verdictpack
  3. **Measure the measurer.** Six agent skills that turn AI evaluation
  4. from a vibe into an instrument: validate your graders before you
  5. trust their scores, score how an agent *executed* a run (not just
  6. whether it finished), audit what your eval suite actually covers,
  7. design the near-miss tests that catch silent substitution, calibrate
  8. an LLM judge before its verdicts feed anything, and layer the whole
  9. programme by cost so cheap tests run constantly and expensive ones
  10. earn their cadence.
  11. Every score is a claim made by a measuring instrument. These skills
  12. are the procedures for measuring the instruments — each one
  13. self-contained, executable by an agent with no other context, with
  14. a worked miniature example and, where counting is involved, a
  15. stdlib-only script whose numbers you can check.
  16. ## Quickstart
  17. ```bash
  18. git clone https://github.com/ranadhir19/verdictpack.git
  19. cp -r verdictpack/skills/* ~/.claude/skills/ # personal
  20. ```
  21. ## The skills
  22. | Skill | What it does | Reach for it when |
  23. |---|---|---|
  24. | [validate-llm-grader](skills/validate-llm-grader/SKILL.md) | Measures a grader/checker/judge against gold-labelled data under an acceptance bound frozen *before* the run; returns accept / fall-back. | A benchmark, harness, or CI gate relies on a grader whose own accuracy has never been measured. |
  25. | [score-agent-run](skills/score-agent-run/SKILL.md) | Scores a recorded agent run against its task spec across eight dimensions — outcome, tool selection, argument validity, step economy, error recovery, policy (hard gate), cost/latency, groundedness. | Reviewing a trace, gating a change on run quality, or investigating a run that "succeeded" badly. |
  26. | [audit-eval-coverage](skills/audit-eval-coverage/SKILL.md) | Builds the domain catalogue from the domain (not the suite), maps suite items to it, reports coverage as topics covered of topics claimed, and probes for broken tests. | Pass rates are quoted as capability evidence, or a suite was written by the same process it tests. |
  27. | [design-near-miss-tests](skills/design-near-miss-tests/SKILL.md) | Generates near-miss tests from supported requests — qualifier mutations, near neighbours, set operations, boundaries, empty cases — each with a required behaviour: answer-modified, refuse, or clarify. | A test set holds only happy-path requests, or a system keeps answering an easier question than the one asked. |
  28. | [calibrate-llm-judge](skills/calibrate-llm-judge/SKILL.md) | Calibrates an LLM-as-judge: agreement, Cohen's kappa, prevalence reported beside kappa (with the paradox explained and reproduced), per-class agreement and precision, disagreement analysis. | A model's judgements feed a score, gate, leaderboard, or annotation pipeline. |
  29. | [layer-your-eval-program](skills/layer-your-eval-program/SKILL.md) | Layers an eval programme by cost — deterministic assertions on every change, human & model eval on a fixed cadence, live comparison only with real users — with synthetic input generation against a feature-by-scenario grid and the pass bar set in advance as a product decision. | Scattered tests exist but no programme: no cadence, no pre-set bar, or judges running before the cheap layer exists. |
  30. ## Install
  31. The pack uses the standard agent-skill format: each skill is a
  32. folder containing a `SKILL.md` (name + description frontmatter,
  33. then the procedure) plus any scripts and templates it needs.
  34. **Claude Code** — copy the skill folders into your skills
  35. directory:
  36. ```bash
  37. git clone https://github.com/ranadhir19/verdictpack.git
  38. cp -r verdictpack/skills/* ~/.claude/skills/ # personal
  39. # or, per project:
  40. cp -r verdictpack/skills/* <your-project>/.claude/skills/
  41. ```
  42. **Other agent hosts** — any host that loads skill folders in this
  43. format (a `SKILL.md` with `name`/`description` frontmatter) can use
  44. the pack the same way: place the folders from `skills/` wherever
  45. your host discovers skills. The procedures themselves are plain
  46. Markdown and the scripts are Python standard library only, so the
  47. pack also works read directly by a human or an agent with no skill
  48. loader at all.
  49. **Verify the pack** after cloning:
  50. ```bash
  51. python3 validate.py # must end with OVERALL PASS
  52. ```
  53. The harness parses every skill's frontmatter, checks that internal
  54. links resolve, and runs each bundled script's self-test.
  55. ## Where the methods come from
  56. The pack is the method layer of two open tools by the same author:
  57. - [evalcrucible](https://github.com/ranadhir19/evalcrucible) — the
  58. grader tool: validates graders against labelled data and audits
  59. static (prompt → gold) task suites. The validate-llm-grader
  60. acceptance bound and the audit probes come from its
  61. pre-registered experiment.
  62. - [runreckon](https://github.com/ranadhir19/runreckon) — the eval
  63. tool: scores a recorded run against its task spec across the
  64. eight dimensions used by score-agent-run, with a fault-injection
  65. self-test and measured judge calibration.
  66. The skills teach and operationalise the procedures; the tools run
  67. them at scale. The validation protocol behind audit-eval-coverage
  68. and design-near-miss-tests — domain-first acceptance questions,
  69. near-miss negatives, qualifier preservation, never trusting the
  70. system's own classification as the verification signal — was
  71. distilled from a real benchmark review failure and its
  72. investigation. The layer-your-eval-program skill is the frame the
  73. others sit inside: it organises the pack's instruments into a
  74. cost-layered programme with cadences and a pre-set bar.
  75. ## Honest scope
  76. - These are *procedures*, not oracles. Each one ends in a
  77. measurement with stated provenance and limits, or in a refusal to
  78. claim more than the inputs support (an n/a dimension, an UNTESTED
  79. error-recovery score, a kappa flagged uninterpretable).
  80. - The bundled scripts count labels and check structures. They do
  81. not call models, need no API keys, and make no network calls.
  82. - Worked examples are miniature and synthetic, built so every
  83. number in them can be re-derived — by running the bundled script
  84. where one exists, or by inspection of the stated counts where
  85. not.
  86. Small-n results are labelled as smoke tests, not measurements —
  87. the pack holds itself to the standards it teaches.
  88. ## License
  89. MIT — see [LICENSE](LICENSE). Copyright (c) 2026 Ranadhir Ghosh.
  90. Originality and IP record: [COMPLIANCE.md](COMPLIANCE.md).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.