Skip to content
Back to search

wikiskill

WikiSkill (arXiv:2608.27454) for Hermes Agent — self-evolving agent skills via a persistent knowledge wiki. Faithful Algorithm 1 implementation with real agent runs, isolated skill gating, and a documented live run log.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ashutoshsinghpr7--wikiskill/badge.svg)](/listings/ashutoshsinghpr7--wikiskill)

HTML

<a href="/listings/ashutoshsinghpr7--wikiskill"><img src="/api/listings/ashutoshsinghpr7--wikiskill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install wikiskill        # from PyPI (wheel + sdist, Python ≥3.10)
wikiskill init demo           # workspace + 22-task auto-graded bench (13 train / 9 val)
wikiskill status
wikiskill evolve --iters 3    # full Algorithm 1 loop with your default model

Or from source: `pip install -e .` (installs the same `wikiskill` CLI).

Workspaces are created **relative to your current directory** (`./workspaces/<domain>/`) — and because `skills/` lives outside the package, the wheel carries a copy of the framework skills that `init` stages into each workspace. If you installed from PyPI before 0.1.5, upgrade and re-`init` any existing workspace: those wheels shipped no `skills/`, so the maintainer and proposer turns ran with no skill loaded (silently — the scores still looked well-formed). See [issue #29](https://github.com/ashutoshsinghpr7/wikiskill/issues/29).

Looking a workspace up (rather than creating one) also prefers `./workspaces/` and then falls back to the old `<repo>/workspaces/` location with a one-line notice, so existing workspaces keep working from any directory in a checkout — or pass `--ws <path>` to be explicit.

That's it. Each evolution workspace lives at `workspaces/<domain>/`:

workspaces/demo/
├── raw/traces/iter-01/{train,val}/<task>.jsonl   # immutable execution traces
├── wiki/                                          # persistent knowledge (never rolled back)
│   ├── index.md  ·  log.md  ·  skill-impact.md  ·  patterns/*.md
├── skills/active/                                 # git-managed evolving skill set (S₀ = ∅)
├── skills/framework/                              # maintainer + proposer agent skills
├── bench/tasks/<id>/                              # task sandboxes (inputs + grader)
└── runs/                                          # per-run stdout, proposals, state

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 219 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # 🧠 WikiSkill
  2. **Compile agent experience into a persistent wiki — and let skills evolve themselves.**
  3. > 📚 **Docs site**: [ashutoshsinghpr7.github.io/wikiskill](https://ashutoshsinghpr7.github.io/wikiskill) · **arXiv**: [2608.27454](https://arxiv.org/abs/2608.27454)
  4. A faithful, production-minded implementation of **[WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://arxiv.org/abs/2608.27454)** (arXiv:2608.27454, Google Research). The loop is **agent-agnostic** — **Hermes Agent** is the reference backend (built natively), **Claude Code**, **Codex** and **GitHub Copilot CLI** ship in the box, and OpenCode is on the roadmap ([issue #13](https://github.com/ashutoshsinghpr7/wikiskill/issues/13)). Your agent becomes both the *student* and the *teacher*.
  5. [![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org)
  6. [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
  7. [![PyPI](https://img.shields.io/pypi/v/wikiskill?color=orange)](https://pypi.org/project/wikiskill/)
  8. [![CI](https://img.shields.io/github/actions/workflow/status/ashutoshsinghpr7/wikiskill/ci.yml?label=CI)](https://github.com/ashutoshsinghpr7/wikiskill/actions)
  9. [![arXiv](https://img.shields.io/badge/arXiv-2608.27454-red.svg)](https://arxiv.org/abs/2608.27454)
  10. ---
  11. ## What this is
  12. Agents fail. They also *learn* — but the lessons usually die with the session. WikiSkill fixes that by keeping a **persistent knowledge wiki** alongside the skill set, and running a closed evolution loop:
  13. 1. The agent runs **training tasks** with its current skills → raw execution traces
  14. 2. A **Wiki Maintainer** agent distills the traces into pattern pages (root causes, fixes)
  15. 3. A **Skill Proposer** agent reads the wiki + traces and proposes one skill change (create or patch)
  16. 4. **Gating**: the change is validated on held-out tasks — **strictly better** than the best score so far → kept; otherwise rolled back. **The wiki is never rolled back.**
  17. Over iterations, knowledge compounds in the wiki while only *proven* improvements touch the skills.
  18. ```
  19. ┌──────────────────────────────────────────────────────┐
  20. │ EVOLUTION LOOP (Algorithm 1) │
  21. │ │
  22. tasks ─────► │ Inference Agent ──► raw/traces/ (immutable) │
  23. │ │ │
  24. │ ▼ │
  25. │ Wiki Maintainer ──► wiki/patterns/, index, log │
  26. │ │ │
  27. │ ▼ │
  28. │ Skill Proposer ──► proposal (create/patch skill) │
  29. │ │ │
  30. │ ▼ │
  31. │ GATE: val score > R_best? ──yes──► keep, R_best=R │
  32. │ │ no │
  33. │ ▼ │
  34. │ rollback skills; wiki retained forever │
  35. └──────────────────────────────────────────────────────┘
  36. ```
  37. ## Why Hermes?
  38. This is not a toy simulator. Every component is a **real Hermes agent turn**:
  39. | WikiSkill (paper) | This repo |
  40. |---|---|
  41. | Inference Agent | `hermes chat --oneshot` in an **isolated `HERMES_HOME` profile** |
  42. | Raw Layer | Full session JSONL transcripts, exported via `hermes sessions export` |
  43. | Wiki Layer | `wiki/` — git-tracked, maintained by a real agent, never rolled back |
  44. | Skill Layer | Real `SKILL.md` packages (frontmatter + instructions), git-managed |
  45. | Wiki Maintainer | Agent turn with the paper's **Appendix E.2 prompt** (extracted verbatim) |
  46. | Skill Proposer | Agent turn with the paper's **Appendix E.3 ReAct prompt** |
  47. | Gating | Strict `R_val > R_best`; `git reset --hard` on reject |
  48. **Why the isolated profile matters:** gating is only meaningful if the agent sees *exactly* the candidate skill set. Each evolution workspace gets its own `HERMES_HOME` (bundled skills opted out, empty memory, skills symlinked per stage) — your real profile is never touched.
  49. ## Quickstart (60 seconds)
  50. ```bash
  51. pip install wikiskill # from PyPI (wheel + sdist, Python ≥3.10)
  52. wikiskill init demo # workspace + 22-task auto-graded bench (13 train / 9 val)
  53. wikiskill status
  54. wikiskill evolve --iters 3 # full Algorithm 1 loop with your default model
  55. ```
  56. Or from source: `pip install -e .` (installs the same `wikiskill` CLI).
  57. Workspaces are created **relative to your current directory** (`./workspaces/<domain>/`)
  58. — and because `skills/` lives outside the package, the wheel carries a copy of
  59. the framework skills that `init` stages into each workspace. If you installed
  60. from PyPI before 0.1.5, upgrade and re-`init` any existing workspace: those
  61. wheels shipped no `skills/`, so the maintainer and proposer turns ran with no
  62. skill loaded (silently — the scores still looked well-formed). See
  63. [issue #29](https://github.com/ashutoshsinghpr7/wikiskill/issues/29).
  64. Looking a workspace up (rather than creating one) also prefers `./workspaces/`
  65. and then falls back to the old `<repo>/workspaces/` location with a one-line
  66. notice, so existing workspaces keep working from any directory in a checkout —
  67. or pass `--ws <path>` to be explicit.
  68. That's it. Each evolution workspace lives at `workspaces/<domain>/`:
  69. ```
  70. workspaces/demo/
  71. ├── raw/traces/iter-01/{train,val}/<task>.jsonl # immutable execution traces
  72. ├── wiki/ # persistent knowledge (never rolled back)
  73. │ ├── index.md · log.md · skill-impact.md · patterns/*.md
  74. ├── skills/active/ # git-managed evolving skill set (S₀ = ∅)
  75. ├── skills/framework/ # maintainer + proposer agent skills
  76. ├── bench/tasks/<id>/ # task sandboxes (inputs + grader)
  77. └── runs/ # per-run stdout, proposals, state
  78. ```
  79. ## CLI
  80. | Command | What it does |
  81. |---|---|
  82. | `wikiskill init <domain> [--backend claude]` | Create workspace + demo bench (pins the agent backend) |
  83. | `wikiskill bench --reset` | Regenerate tasks (deterministic, seed=42) |
  84. | `wikiskill status` | Workspace state: scores, skills, wiki, history |
  85. | `wikiskill evolve --iters N [--model M] [--provider P] [--max-turns N] [--no-early-stop]` | The full loop (`--model`/`--provider` patch the isolated profile's default model, e.g. `google/gemini-2.5-flash-lite` + `openrouter`) |
  86. | `wikiskill run-task <id>` | Single inference rollout (debug) |
  87. | `wikiskill compare <wsA> <wsB> [--iters N]` | Paired statistical comparison: per-task win/loss/tie + two-sided exact-binomial p-value (answers "did the skill actually help?" — see [docs/COMPARING.md](docs/COMPARING.md)) |
  88. ## Bring your own tasks
  89. Tasks are plain JSON (`tasks.json`); anything auto-gradable works:
  90. ```json
  91. {
  92. "id": "spec-format1-1", "split": "train",
  93. "title": "Format products according to spec",
  94. "prompt": "Read spec.md and products.json...",
  95. "sandbox": {"spec.md": "...", "products.json": "..."},
  96. "grader": {"type": "exact", "file": "output.txt", "expected": "alpha|35|active\n..."}
  97. }
  98. ```
  99. Graders: `exact`, `contains`, `json_field`, `code_stdout` (runs the produced script). Missing deliverables score **0**, never crash.
  100. ## Live results so far
  101. Honest numbers from real agent runs on the bundled bench:
  102. | Setup | Baseline (S₀) | What happened |
  103. |---|---|---|
  104. | deepseek-v4-flash, 15 turns | **1.0** | Algorithm 1 **early-stop** — nothing to evolve |
  105. | deepseek-v4-flash, 8 turns | **1.0** | same |
  106. | deepseek-v4-flash, forced | 1.0 | proposer created `spec_literal_transform` → R_val=1.0, not > R_best → **rejected** |
  107. | deepseek-v4-flash, forced | 1.0 | maintainer distilled **4 pattern pages** (incl. `execute_code` blocked in sandbox, ripgrep binary misses); proposer created `exact-match-sandbox-task` → R_val=0.8889 (skill *hurt*) → **rejected** |
  108. | gemma-3-4b (free, OpenRouter) | — | **invalid run, thrown out** — dead agent sessions were phantom-graded against stale sandboxes. The maintainer's pattern page caught the framework's own bug; fixed + regression-tested (see [docs/RUNS.md](docs/RUNS.md) Run 4) |
  109. | **gemini-2.5-flash-lite (free, OpenRouter), 8 turns** | **0.6667** (real) | small model fails at S₀ → maintainer distilled 5 patterns → proposer created `find-secret` → **R_val=0.4444, the skill hurt (2 regressions) → rejected**. Full loop live on a genuinely weak model, ~$0.09/iteration |
  110. | gemini-2.5-flash-lite, 3 iterations (issue #5) | **0.4444** (real) | compounding run: train as low as 0.2308, 6/48 launch failures (detected + honest 0.0s), maintainer distilled 1 pattern, proposer **declined (`no_action`)** — nothing to gate, r_best preserved. Honest negative: accumulation needs a stronger model (see [docs/RUNS.md](docs/RUNS.md) Run 6) |
  111. The gating mechanism has caught both a neutral and a *harmful* proposal live. Full logs in [docs/RUNS.md](docs/RUNS.md).
  112. ## Design decisions worth knowing
  113. - **`--in` doesn't pin the agent's CWD** in single-query runs → every inference prompt embeds an absolute `WORKING DIRECTORY` and forbids exploring outside it.
  114. - **Sessions live in `state.db`**, not loose files → transcripts are materialized via `hermes sessions export --format jsonl`.
  115. - **Rejected proposals are never lost** — their full content is embedded in `wiki/skill-impact.md` so future proposers don't repeat them (per Appendix E.3).
  116. - **The demo bench has traps**: subtle-spec tasks and multi-bug debug scripts whose bugs *don't* compensate (verified at generation time).
  117. ## How this compares to other community implementations
  118. We audited the three repos that appeared alongside the paper (see [docs/RUNS.md](docs/RUNS.md)). This is the only one that: runs on a real agent stack (Hermes), gates skills through a fully isolated profile, ships verbatim Appendix E prompts, and has a live-verified end-to-end loop (maintainer → proposer → gate → rollback).
  119. ## Agent backends
  120. The loop runs on any supported agent CLI — the raw/wiki/skill layers are
  121. backend-agnostic ([issue #13](https://github.com/ashutoshsinghpr7/wikiskill/issues/13)).
  122. | Backend | Pin a workspace | Notes |
  123. |---|---|---|
  124. | `hermes` (default) | `wikiskill init demo --backend hermes` | reference implementation; isolated `HERMES_HOME` per workspace |
  125. | `claude` | `wikiskill init demo --backend claude` | Claude Code 2.x (`claude -p`), isolated `CLAUDE_CONFIG_DIR`, transcripts normalized from the stream-json output; `claude auth login` required once |
  126. | `codex` | `wikiskill init demo --backend codex` | OpenAI Codex (`codex exec --json --full-auto`), isolated `CODEX_HOME`, transcripts from session JSONL; `codex login` once + binary on PATH |
  127. | `copilot` | `wikiskill init demo --backend copilot` | GitHub Copilot CLI (`copilot -p -s`), isolated `COPILOT_HOME`, skills symlinked into the sandbox's `.github/skills/`; `copilot login` once (or `GH_TOKEN`) |
  128. Each workspace pins its backend in `workspaces/<domain>/workspace.json`; switch
  129. anytime with `wikiskill evolve <domain> --backend claude`. Skills evolved on
  130. one backend transfer to another via `wikiskill transfer` (same SKILL.md format).
  131. ## Skills tap — install the distilled patterns in Hermes
  132. This repo doubles as a [Hermes skills tap](https://agentskills.io/specification) —
  133. the patterns distilled from live evolution runs, installable in Hermes:
  134. ```bash
  135. hermes skills tap add ashutoshsinghpr7/wikiskill
  136. hermes skills install ashutoshsinghpr7/wikiskill/skills/wikiskill-evolve
  137. hermes skills install ashutoshsinghpr7/wikiskill/skills/search-miss-binary
  138. # ...one `install` per skill you want
  139. ```
  140. | Skill | What it teaches | Paper layer |
  141. |---|---|---|
  142. | `wikiskill-evolve` | Run the full evolution loop from inside Hermes | framework meta-skill (tooling doc, not a paper artifact) |
  143. | `search-miss-binary` | ripgrep silently skips binary files — verify empty results | maintainer wiki pattern (distilled from live run) |
  144. | `script-exec-blocked` | Sandbox approval policy: use file tools, not `python3 -c` | maintainer wiki pattern |
  145. | `spec-literal-execution` | Apply only the spec's literal clauses — no hidden transforms | maintainer wiki pattern |
  146. | `trace-harness-launch-failure` | Empty traces = launch failure, not agent behavior | maintainer wiki pattern |
  147. | `verify-output-readback` | Re-read the deliverable before finishing | maintainer wiki pattern |
  148. **Paper alignment, stated honestly:** the five operational skills are the
  149. *maintainer's* distilled patterns from real graded runs (`docs/RUNS.md`) — the
  150. paper's wiki layer. None has been *accepted* by the validation gate yet (every
  151. live gate so far was a rejection or `no_action`), so treat them as
  152. well-evidenced raw material for the proposer/gate pipeline, not as
  153. gate-approved skills. All ship as standard SKILL.md (agentskills.io-compatible).
  154. ## Roadmap
  155. **Done:**
  156. - [x] `compare` command — paired exact-binomial run comparison ([#6](https://github.com/ashutoshsinghpr7/wikiskill/pull/6))
  157. - [x] Skill *transfer* across workspaces/models ([#7](https://github.com/ashutoshsinghpr7/wikiskill/pull/7))
  158. - [x] Cron-driven overnight evolution (`hermes cron`, 01:00 IST nightly) + docs/CRON.md
  159. - [x] Multi-agent backends: Hermes (reference) + **Claude Code**, **Codex**, **GitHub Copilot CLI** ([#13](https://github.com/ashutoshsinghpr7/wikiskill/issues/13), [#15](https://github.com/ashutoshsinghpr7/wikiskill/issues/15), [#24](https://github.com/ashutoshsinghpr7/wikiskill/issues/24))
  160. - [x] GitHub Pages — custom animated docs site ([#18](https://github.com/ashutoshsinghpr7/wikiskill/pull/18))
  161. - [x] PyPI package `wikiskill` via **tokenless trusted publishing** ([#20](https://github.com/ashutoshsinghpr7/wikiskill/pull/20))
  162. - [x] Multi-iteration compounding run — honest negative documented (Run 6)
  163. **Planned:**
  164. - [ ] Codex backend — **done** ([#15](https://github.com/ashutoshsinghpr7/wikiskill/issues/15))
  165. - [ ] OpenCode backend ([#16](https://github.com/ashutoshsinghpr7/wikiskill/issues/16))
  166. - [ ] Cross-agent transfer demo — evolve on one agent, gate on another ([#17](https://github.com/ashutoshsinghpr7/wikiskill/issues/17))
  167. - [ ] A live *acceptance* gate (proposal beats baseline on a real model) — still the open scientific question (see Run 6)
  168. - [ ] Real-task domains — your recurring workflows as graded task packs ([#2](https://github.com/ashutoshsinghpr7/wikiskill/issues/2))
  169. ## License
  170. MIT — see [LICENSE](LICENSE). Based on arXiv:2608.27454 (Google Research); all prompts in `skills/` are adapted from the paper's Appendix E. Inspired by [Karpathy's LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.