Skip to content
Back to search

graph-of-skills

[EMNLP '26] Dependency-Aware Structural Retrieval for Massive Agent Skills

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/davidliuk--graph-of-skills/badge.svg)](/listings/davidliuk--graph-of-skills)

HTML

<a href="/listings/davidliuk--graph-of-skills"><img src="/api/listings/davidliuk--graph-of-skills/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requirements

- Python 3.10 -- 3.12 - [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip` - An embedding API key (OpenAI, Gemini, or any OpenAI-compatible provider)

Setup

git clone https://github.com/davidliuk/graph-of-skills.git
cd graph-of-skills
uv sync
cp .env.example .env   # then fill in your API keys

<details> <summary><strong>Provider: OpenAI (direct)</strong></summary>

OPENAI_API_KEY=sk-...
# Use the ``openai/...`` prefix so LiteLLM targets the OpenAI API (omit OPENAI_BASE_URL).
GOS_EMBEDDING_MODEL=openai/text-embedding-3-large
GOS_EMBEDDING_DIM=3072

OpenRouter exact-response caching is enabled by default for reproducible retries during indexing (`GOS_OPENROUTER_RESPONSE_CACHE=true`). Cached responses must be disabled or isolated when repeated graph builds are intended to measure LLM construction variance. </details>

<details> <summary><strong>Provider: OpenRouter</strong></summary>

OPENROUTER_API_KEY=<openrouter-key>
OPENAI_BASE_URL=https://openrouter.ai/api/v1
GOS_EMBEDDING_MODEL=openrouter/openai/text-embedding-3-large
GOS_EMBEDDING_DIM=3072

</details>

<details> <summary><strong>Provider: Azure AI (OpenAI-compatible)</strong></summary>

OPENAI_API_KEY=<azure-api-key>
OPENAI_BASE_URL=https://YOUR-RESOURCE.services.ai.azure.com/openai/v1
# Must match your **deployment name** in Azure (not necessarily ``text-embedding-3-large``).
GOS_EMBEDDING_MODEL=openai/<your-deployment-name>
GOS_EMBEDDING_DIM=<vector-dimension-for-that-model>

</details>

<details> <summary><strong>Provider: Google Gemini</strong></summary>

GEMINI_API_KEY=<your-key>
GOS_EMBEDDING_MODEL=gemini/gemini-embedding-001
GOS_EMBEDDING_DIM=3072

</details>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 547 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center">Graph of Skills (GoS)</h1>
  2. <p align="center">
  3. <strong>Dependency-Aware Structural Retrieval for Massive Agent Skills</strong>
  4. </p>
  5. <p align="center">
  6. <a href="https://davidliuk.github.io/"><strong>Dawei Liu</strong></a><sup>*</sup> &middot;
  7. <a href="https://zli12321.github.io/"><strong>Zongxia Li</strong></a><sup>*</sup> &middot;
  8. <a href="https://hongyang-du.github.io/"><strong>Hongyang Du</strong></a> &middot;
  9. <a href="https://wuxiyang1996.github.io/"><strong>Xiyang Wu</strong></a> &middot;
  10. <strong>Shihang Gui</strong></a> &middot;
  11. <strong>Yongbei Kuang</strong></a> &middot;
  12. <a href="https://lichao-sun.github.io/"><strong>Lichao Sun</strong></a>
  13. </p>
  14. <p align="center">
  15. <a href="https://2026.emnlp.org/"><img src="https://img.shields.io/badge/EMNLP%202026-Main%20Conference-8B0000" alt="EMNLP 2026"></a>
  16. <a href="https://arxiv.org/abs/2604.05333"><img src="https://img.shields.io/badge/arXiv-2604.05333-b31b1b?logo=arxiv" alt="Paper"></a>
  17. <a href="https://huggingface.co/papers/2604.05333"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Paper-yellow" alt="HF Paper"></a>
  18. <a href="https://huggingface.co/datasets/DLPenn/graph-of-skills-data"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Data-yellow" alt="Data"></a>
  19. <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue" alt="License"></a>
  20. <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.10--3.12-3776ab?logo=python&logoColor=white" alt="Python"></a>
  21. </p>
  22. ---
  23. ## 🔥 Updates
  24. - **[2026-08-20]** 🎉 **Graph of Skills is accepted to EMNLP 2026 Main Conference** (Budapest). The camera-ready adds a component ablation, a budget-matched retrieval study, library-size scaling to 2,000 skills, and graph-construction diagnostics — all summarized in [Results](#results).
  25. - **[2026-04-15]** Released a [Claude Code MCP plugin](skills/) for graph of skills retrieval — drop-in integration for Claude Code agents.
  26. - **[2026-04-07]** Paper released on [arXiv:2604.05333](https://arxiv.org/abs/2604.05333) and [HuggingFace Papers](https://huggingface.co/papers/2604.05333).
  27. - **[2026-04-06]** Code open-sourced on [GitHub](https://github.com/davidliuk/graph-of-skills).
  28. - **[2026-04-04]** Skill libraries, prebuilt workspaces, and benchmark data released on [HuggingFace](https://huggingface.co/datasets/DLPenn/graph-of-skills-data).
  29. ---
  30. ## Overview
  31. Graph of Skills builds a **skill graph** offline from a library of `SKILL.md` documents, then retrieves a small, ranked set of relevant skills at task time. Instead of flooding the agent context with an entire skill library, GoS surfaces only the skills most likely to help -- along with their prerequisites and related capabilities.
  32. <p align="center">
  33. <img src="assets/demo.png" alt="Comparison of Vanilla Skills, Vector Skills, and Graph of Skills" width="800">
  34. </p>
  35. ## How It Works
  36. <p align="center">
  37. <img src="assets/pipeline.png" alt="GoS pipeline: Offline Indexing → Graph Construction → Online Retrieval" width="800">
  38. </p>
  39. **Retrieval pipeline:**
  40. 1. **Seed** -- retrieve semantic candidates (embedding similarity) and lexical candidates (exact-match tokens)
  41. 2. **Merge** -- combine both candidate pools
  42. 3. **Rerank** -- rerank using the skill-graph structure (dependencies, co-occurrence)
  43. 4. **Return** -- emit a capped, agent-readable skill bundle
  44. ## Results
  45. GoS is evaluated on **SkillsBench** (87 dockerized coding tasks) and **ALFWorld** (140 episodes) across three model families. **R** = average reward (%), **T** = average total tokens (uncached input + cached input + output), **S** = agent-only runtime (s). ↑ higher is better, ↓ lower is better. All numbers are means over two runs per setting.
  46. ### Main results (end-to-end)
  47. | Model | Method | SB R↑ | SB T↓ | SB S↓ | AW R↑ | AW T↓ | AW S↓ |
  48. |-------|--------|------:|------:|------:|------:|------:|------:|
  49. | Claude Sonnet 4.5 | Vanilla Skills | 25.0 | 967,791 | 465.8 | 89.3 | 1,524,401 | 53.2 |
  50. | | Vector Skills | 19.3 | 894,640 | **357.3** | 93.6 | 28,407 | **37.8** |
  51. | | **GoS (ours)** | **31.0** | **860,315** | 364.9 | **97.9** | **27,215** | 49.2 |
  52. | MiniMax M2.7 | Vanilla Skills | 17.2 | 942,113 | 580.7 | 47.1 | 2,184,823 | 88.6 |
  53. | | Vector Skills | 10.4 | **852,881** | 552.9 | 50.7 | 66,109 | 73.4 |
  54. | | **GoS (ours)** | **18.7** | 867,452 | **502.5** | **54.3** | **65,227** | **68.8** |
  55. | GPT-5.2 Codex | Vanilla Skills | 27.4 | 3,187,749 | **686.8** | 89.3 | 1,435,614 | 83.3 |
  56. | | Vector Skills | 21.5 | **1,243,648** | 773.0 | 92.9 | **34,436** | **57.0** |
  57. | | **GoS (ours)** | **34.4** | 1,379,773 | 715.6 | **93.6** | 46,462 | 64.7 |
  58. GoS achieves the **highest reward on every model** on both benchmarks while cutting total tokens by up to **56×** (ALFWorld, Claude Sonnet 4.5) vs. Vanilla Skills.
  59. ### Scaling with library size
  60. Reward and cost as the repository grows (SkillsBench, GPT-5.2 Codex). **T** in millions of tokens.
  61. | Skills | Method | R↑ | T (M)↓ | S↓ |
  62. |-------:|--------|---:|-------:|---:|
  63. | 200 | Vanilla Skills | **32.5** | 1.85 | **701.6** |
  64. | | Vector Skills | 21.2 | **1.06** | 833.8 |
  65. | | **GoS (ours)** | 32.1 | 1.36 | 731.2 |
  66. | 500 | Vanilla Skills | 26.0 | 1.93 | **756.8** |
  67. | | Vector Skills | 20.7 | **1.10** | 849.5 |
  68. | | **GoS (ours)** | **31.4** | 1.16 | 890.3 |
  69. | 1,000 | Vanilla Skills | 27.4 | 3.19 | **686.8** |
  70. | | Vector Skills | 21.5 | **1.24** | 773.0 |
  71. | | **GoS (ours)** | **34.4** | 1.38 | 715.6 |
  72. | 2,000 | Vanilla Skills | 26.7 | 5.84 | **733.5** |
  73. | | Vector Skills | 23.8 | **1.11** | 799.8 |
  74. | | **GoS (ours)** | **31.3** | 1.14 | 788.0 |
  75. From 500 skills onward GoS leads both baselines by 5.4, 7.0, and 4.6 reward points. Flat exposure rises from 1.93M to 5.84M tokens (~3x) across the same range while both retrievers stay near-constant — the scaling burden is in the prompt, not in retrieval.
  76. ### Component ablation
  77. Full SkillsBench, GPT-5.2 Codex, 1,000-skill library.
  78. | Variant | R↑ | T (M)↓ | S↓ |
  79. |---------|---:|-------:|---:|
  80. | **Full GoS** (reverse-aware PPR) | **34.4** | 1.38 | **715.6** |
  81. | Forward-only PPR (γ_r = 0) | 25.3 | 1.20 | 742.7 |
  82. | w/o graph propagation | 29.3 | **0.89** | 766.2 |
  83. | w/o lexical + rerank | 26.7 | 1.01 | 747.7 |
  84. **Traversal direction matters more than the graph itself.** Replacing reverse traversal with forward propagation costs **9.1** reward points — a *larger* loss than removing graph propagation entirely (5.1). The gain comes from traversing dependencies backwards, not from graph diffusion as such.
  85. ### Isolating the graph operation
  86. The table above compares full agent runs, which entangle the graph with hydration, reranking, and the agent interface. This budget-matched offline study holds seeding, reranker, top-N selection, character budgets, and hydration fixed across all arms, so **only the graph operation changes** (200-skill library, 87 queries).
  87. | Retrieval operation | R_avail↑ | R_full↑ | C↑ | D↑ |
  88. |---------------------|---------:|--------:|---:|---:|
  89. | **Reverse-aware PPR (GoS)** | **0.575** | **0.502** | **0.421** | **0.654** |
  90. | Forward-only PPR (γ_r = 0) | 0.490 | 0.428 | 0.311 | 0.362 |
  91. | No-Graph (matched flat) | 0.542 | 0.473 | 0.368 | 0.481 |
  92. | One-Hop dependency | 0.542 | 0.473 | 0.368 | 0.481 |
  93. `R_avail` / `R_full` = available- and full-oracle recall, `C` = exact bundle completeness, `D` = dependency-pair co-recovery. The same ordering reproduces at retrieval level: forward propagation is *worse than not using the graph at all*, and dependency-pair co-recovery falls from **0.654 to 0.362**.
  94. ### Construction cost and robustness
  95. - **One-time offline cost:** $0.29 at 200 skills, $2.36 at 2,000. Online propagation adds ~1.5 ms at p50.
  96. - **Threshold is not delicate:** sweeping the acceptance threshold ζ from 0.4 to 0.8 moves the deterministic edge count from 172 to 46 while changing recall by less than 0.005.
  97. - **Documentation quality is the main limiting factor:** dropping half of all I/O fields costs 0.024 recall, whereas dropping half of the descriptions costs 0.146.
  98. ## Citation
  99. If you find this work useful, please cite the EMNLP 2026 version:
  100. ```bibtex
  101. @inproceedings{liu2026graphofskills,
  102. title = {Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills},
  103. author = {Liu, Dawei and Li, Zongxia and Du, Hongyang and Wu, Xiyang and
  104. Gui, Shihang and Kuang, Yongbei and Sun, Lichao},
  105. booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  106. year = {2026}
  107. }
  108. ```
  109. <details>
  110. <summary>arXiv preprint</summary>
  111. ```bibtex
  112. @misc{liu2026graphofskillsarxiv,
  113. title = {Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skills},
  114. author = {Dawei Liu and Zongxia Li and Hongyang Du and Xiyang Wu and
  115. Shihang Gui and Yongbei Kuang and Lichao Sun},
  116. year = {2026},
  117. eprint = {2604.05333},
  118. archivePrefix = {arXiv},
  119. primaryClass = {cs.AI},
  120. url = {https://arxiv.org/abs/2604.05333}
  121. }
  122. ```
  123. </details>
  124. ## Installation
  125. ### Requirements
  126. - Python 3.10 -- 3.12
  127. - [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
  128. - An embedding API key (OpenAI, Gemini, or any OpenAI-compatible provider)
  129. ### Setup
  130. ```bash
  131. git clone https://github.com/davidliuk/graph-of-skills.git
  132. cd graph-of-skills
  133. uv sync
  134. cp .env.example .env # then fill in your API keys
  135. ```
  136. <details>
  137. <summary><strong>Provider: OpenAI (direct)</strong></summary>
  138. ```bash
  139. OPENAI_API_KEY=sk-...
  140. # Use the ``openai/...`` prefix so LiteLLM targets the OpenAI API (omit OPENAI_BASE_URL).
  141. GOS_EMBEDDING_MODEL=openai/text-embedding-3-large
  142. GOS_EMBEDDING_DIM=3072
  143. ```
  144. OpenRouter exact-response caching is enabled by default for reproducible retries during
  145. indexing (`GOS_OPENROUTER_RESPONSE_CACHE=true`). Cached responses must be disabled or
  146. isolated when repeated graph builds are intended to measure LLM construction variance.
  147. </details>
  148. <details>
  149. <summary><strong>Provider: OpenRouter</strong></summary>
  150. ```bash
  151. OPENROUTER_API_KEY=<openrouter-key>
  152. OPENAI_BASE_URL=https://openrouter.ai/api/v1
  153. GOS_EMBEDDING_MODEL=openrouter/openai/text-embedding-3-large
  154. GOS_EMBEDDING_DIM=3072
  155. ```
  156. </details>
  157. <details>
  158. <summary><strong>Provider: Azure AI (OpenAI-compatible)</strong></summary>
  159. ```bash
  160. OPENAI_API_KEY=<azure-api-key>
  161. OPENAI_BASE_URL=https://YOUR-RESOURCE.services.ai.azure.com/openai/v1
  162. # Must match your **deployment name** in Azure (not necessarily ``text-embedding-3-large``).
  163. GOS_EMBEDDING_MODEL=openai/<your-deployment-name>
  164. GOS_EMBEDDING_DIM=<vector-dimension-for-that-model>
  165. ```
  166. </details>
  167. <details>
  168. <summary><strong>Provider: Google Gemini</strong></summary>
  169. ```bash
  170. GEMINI_API_KEY=<your-key>
  171. GOS_EMBEDDING_MODEL=gemini/gemini-embedding-001
  172. GOS_EMBEDDING_DIM=3072
  173. ```
  174. </details>
  175. ## Quick Start
  176. **Goal:** install the package, pull the published skill libraries, build (or download) a graph workspace, then run retrieval from the shell.
  177. **Read next:** [DATA.md](DATA.md) for every download flag and asset size; [`.env.example`](.env.example) for embedding providers. After GoS works locally, use [evaluation/README.md](evaluation/README.md) for benchmark runners and [evaluation/skillsbench/README.md](evaluation/skillsbench/README.md) for Harbor-based SkillsBench.
  178. ### Step 0: Install (once per machine)
  179. Complete [Installation](#installation) above: clone, `uv sync`, `cp .env.example .env`, and set **embedding** (and optional LLM) keys. Indexing and retrieval load `.env` from the repo root when you use `uv run gos …`.
  180. ### Step 1: Download skill libraries
  181. The collections **`skills_200`**, **`skills_500`**, **`skills_1000`**, **`skills_2000`** are directories of `SKILL.md` files on [HuggingFace](https://huggingface.co/datasets/DLPenn/graph-of-skills-data), **not** in git. They unpack to:
  182. - `data/skillsets/skills_200/` … `data/skillsets/skills_2000/`
  183. ```bash
  184. ./scripts/download_data.sh --skillsets
  185. ```
  186. This tries each archive, **skips** directories that already have files, and logs `[skip]` if an archive is not yet on the Hub. Gated datasets: `HF_TOKEN=hf_... ./scripts/download_data.sh --skillsets`. Full reference (tasks, workspaces, selective flags): **[DATA.md](DATA.md)**.
  187. **Tiny smoke test without HuggingFace:** index the built-in folder `skills/` (only a few skills) with any `--workspace` path you like.
  188. ### Step 2: Workspace directory layout (recommended for benchmarks)
  189. `--workspace` is where GoS stores the **indexed graph** (vectors + graph storage). Use the **same** path for `gos retrieve`, `gos status`, and `gos add`.
  190. For **ALFWorld** and **SkillsBench** defaults, keep this mapping (see [evaluation/README.md](evaluation/README.md) and `evaluation/skillsbench/graphskills_benchmark.py`):
  191. | Skill tree you index | Recommended `--workspace` |
  192. |----------------------|---------------------------|
  193. | `data/skillsets/skills_200` | `data/gos_workspace/skills_200_v1` |
  194. | `data/skillsets/skills_500` | `data/gos_workspace/skills_500_v1` |
  195. | `data/skillsets/skills_1000` | `data/gos_workspace/skills_1000_v1` |
  196. | `data/skillsets/skills_2000` | `data/gos_workspace/skills_2000_v1` |
  197. ### Step 3: Get a workspace (choose one path)
  198. **A. Build locally** (needs embedding API; duration grows with library size):
  199. ```bash
  200. mkdir -p data/gos_workspace
  201. uv run gos index data/skillsets/skills_200 \
  202. --workspace data/gos_workspace/skills_200_v1 --clear
  203. ```
  204. Use the matching pair for other sets (e.g. `skills_1000` → `data/gos_workspace/skills_1000_v1`). **Embedding model and dimension in `.env` must stay the same** for later retrieval (see [Configuration](#configuration)).
  205. **B. Download a prebuilt workspace** (no `gos index`; must match the embedding used to build that archive):
  206. ```bash
  207. ./scripts/download_data.sh --workspace
  208. ```
  209. > **Compatibility note:** the legacy `*_v1` archives were serialized by an
  210. > undirected cleanup pipeline and are retained only for reproducing earlier
  211. > results. The repaired typed-directed constructor rejects them because edge
  212. > direction cannot be recovered safely. Build a fresh workspace from the
  213. > downloaded skill library for current GoS retrieval and new experiments.
  214. See **[DATA.md](DATA.md)** for which `gos_workspace_skills_*_v1.tar.gz` files exist on the Hub and how they map to `data/gos_workspace/`.
  215. ### Step 4: Retrieve
  216. ```bash
  217. uv run gos retrieve "parse binary STL file, calculate volume and mass" \
  218. --workspace data/gos_workspace/skills_200_v1 --max-skills 5
  219. ```
  220. ### Step 5: Inspect or extend
  221. ```bash
  222. uv run gos status --workspace data/gos_workspace/skills_200_v1
  223. uv run gos add path/to/NEW_SKILL.md --workspace data/gos_workspace/skills_200_v1
  224. uv run gos relink --workspace data/gos_workspace/skills_200_v1 \
  225. --concurrency 8 --checkpoint-every 10 --resume
  226. ```
  227. `gos relink` writes three secret-free telemetry artifacts beside the graph:
  228. - `relink_progress.json`: atomic resume ledger and cumulative counters;
  229. - `relink_events.jsonl`: append-only attempt/checkpoint events with token,
  230. cost, timing, throughput, per-batch focus results, and failures;
  231. - `construction_report.json`: final graph, provenance, configuration, aggregate
  232. usage, stage timing, and throughput summary.
  233. The console checkpoint line also reports cumulative calls, input/output tokens,
  234. provider cost, failures, edges, and elapsed wall time. Logs never include API
  235. keys, request headers, prompts, raw skill documents, or model responses.
  236. ### Step 6: What to run next
  237. - **End-to-end sanity check** (retrieval + one Docker task): see the **Minimal Verification** collapsible in the [Evaluation](#evaluation) section.
  238. - **Paper benchmarks** (ALFWorld, SkillsBench): **[evaluation/README.md](evaluation/README.md)** (overview) and **[evaluation/skillsbench/README.md](evaluation/skillsbench/README.md)** (Harbor / task generation).
  239. ## Agent Integration
  240. ### Claude Code (MCP Plugin)
  241. GoS ships with a built-in [MCP](https://modelcontextprotocol.io/) server that gives Claude Code direct access to the skill graph. When you open this project, **Claude Code auto-discovers the server** via `.mcp.json` — no manual setup.
  242. **Quick start:**
  243. ```bash
  244. uv sync # install deps (once)
  245. cp .env.example .env # fill in API keys
  246. ./scripts/download_data.sh --workspace # download prebuilt workspaces
  247. ```
  248. Then open the project in Claude Code. The `graph-of-skills` MCP server is ready. Ask naturally:
  249. > "Find skills for processing 3D mesh files with GoS, then follow the skill instructions to complete the task."
  250. Claude Code will call the GoS tools, retrieve relevant skills, and follow their instructions autonomously.
  251. <details>
  252. <summary><strong>Manual registration (if auto-discovery is not available)</strong></summary>
  253. ```bash
  254. claude mcp add graph-of-skills -- uv run --directory /path/to/graph-of-skills gos-claude
  255. # Or with an explicit workspace
  256. claude mcp add graph-of-skills -- uv run --directory /path/to/graph-of-skills gos-claude \
  257. --workspace data/gos_workspace/skills_200_v1
  258. ```
  259. </details>
  260. **Available tools:**
  261. | Tool | Purpose |
  262. |------|---------|
  263. | `search_skills` | Quick ranked summary of relevant skills for a task |
  264. | `retrieve_skill_bundle` | Full agent-ready skill content (SKILL.md bodies, scripts, graph evidence) |
  265. | `hydrate_skills` | Load specific skills by exact name |
  266. | `list_skills` | Browse all indexed skills with descriptions |
  267. | `get_skill_detail` | Full metadata, I/O schema, and graph neighbors for one skill |
  268. | `get_skill_neighbors` | Dependency / workflow / semantic edges for a skill |
  269. | `get_status` | Workspace stats (skill count, edge count, retrieval config) |
  270. | `index_skills` | Build the skill graph from a directory of SKILL.md files |
  271. | `add_skill` | Incrementally add new skills to an existing graph |
  272. See **[CLAUDE.md](CLAUDE.md)** for the full integration guide, example workflows, and configuration.
  273. ### Docker / Container (SkillsBench)
  274. Inside a Docker container, an agent calls `graphskills-query` with a natural-language task description and receives a bounded skill bundle:
  275. ```bash
  276. graphskills-query "parse binary STL file and calculate mass"
  277. ```
  278. Each returned skill includes a `Source:` path the agent can open directly:
  279. ```
  280. Source: /opt/graphskills/skills/mesh-analysis/SKILL.md
  281. ```
  282. Set `GOS_SKILLS_DIR` to control path rewriting, so the same workspace can be indexed on a host and queried inside a container.
  283. ## CLI Reference
  284. | Command | Description |
  285. |---------|-------------|
  286. | `gos index <dir>` | Build a graph workspace from a skill directory |
  287. | `gos add <file>` | Add a single skill to an existing workspace |
  288. | `gos relink` | Build/resume edges with bounded concurrency and durable telemetry |
  289. | `gos retrieve <query>` | Retrieve a ranked skill bundle for a query |
  290. | `gos query <query>` | Compact retrieval output (for debugging) |
  291. | `gos status` | Show workspace statistics |
  292. | `gos experiment` | Run built-in experiment presets |
  293. | `gos experiment-presets` | List available experiment presets |
  294. | `graphskills-query` | Agent-facing retrieval (rewrites `Source:` paths for containers) |
  295. | `gos-server` | Start the MCP server for tool-based retrieval |
  296. | `gos-claude` | Start the MCP server for Claude Code (auto-discovered via `.mcp.json`) |
  297. ## Configuration
  298. All runtime settings are driven by environment variables. See [`.env.example`](.env.example) for the full template. Download paths and workspace layout on disk are documented in **[DATA.md](DATA.md)**.
  299. | Variable | Default | Description |
  300. |----------|---------|-------------|
  301. | `GOS_EMBEDDING_MODEL` | `openai/text-embedding-3-large` | Embedding model for indexing and retrieval (use `openai/<deployment>` on Azure) |
  302. | `GOS_EMBEDDING_DIM` | `3072` | Embedding dimension (must match the model output) |
  303. | `GOS_PREBUILT_WORKING_DIR` | -- | Path to a prebuilt workspace for retrieval |
  304. | `GOS_RETRIEVAL_TOP_N` | `8` | Maximum number of skills returned |
  305. | `GOS_SEED_TOP_K` | `5` | Initial seed count before graph expansion |
  306. | `GOS_MAX_CONTEXT_CHARS` | `12000` | Hard cap on total returned bundle size (chars) |
  307. | `GOS_SKILLS_DIR` | -- | Container-side skill root (for `Source:` path rewriting) |
  308. > **Note:** The embedding model at retrieval time **must** match the model used when the workspace was indexed.
  309. ## Evaluation
  310. **Docs:** Start with **[evaluation/README.md](evaluation/README.md)** (all tracks, modes, env vars). For SkillsBench + Harbor only, use **[evaluation/skillsbench/README.md](evaluation/skillsbench/README.md)**. Dataset files and scripts are described in **[DATA.md](DATA.md)**.
  311. We evaluate GoS on two benchmarks:
  312. | Benchmark | Type | Tasks |
  313. |-----------|------|-------|
  314. | **ALFWorld** | Interactive household tasks | 140 episodes |
  315. | **SkillsBench** | Dockerized coding tasks | 87 tasks |
  316. For **running these evaluations**, we recommend routing the agent's chat / completion API through [OpenRouter](https://openrouter.ai/): use an OpenAI-compatible `BASE_URL` (for example `https://openrouter.ai/api/v1`) and the API key your runner documents. The GoS project's own evaluation testing is done mainly this way. Embeddings for indexing and retrieval are separate; configure them in `.env` as in [`.env.example`](.env.example) (OpenRouter, direct OpenAI, Gemini, or Azure).
  317. Benchmark data is hosted externally and **not** included in this repository:
  318. ```bash
  319. ./scripts/download_data.sh # download all assets (~780 MB); options in DATA.md
  320. ```
  321. Selective downloads and workspace rebuild steps: **[DATA.md](DATA.md)**.
  322. <details>
  323. <summary><strong>Minimal Verification (end-to-end smoke test)</strong></summary>
  324. **Scope:** GoS retrieval against a real workspace, then **one** [SkillsBench](evaluation/skillsbench/README.md) task in Docker via [Harbor](https://github.com/harbor-ai/harbor). This is a smoke test, not a full benchmark sweep. For all tracks, see [evaluation/README.md](evaluation/README.md).
  325. **Prerequisites:**
  326. - `uv sync` and a filled `.env` (embedding provider for the workspace you use, plus **`GEMINI_API_KEY`** for the Harbor agent when using `gemini-cli`).
  327. - **Docker** running (Harbor drives the task container).
  328. - **Harbor** on your `PATH`, e.g. `uv tool install harbor` (see [evaluation/skillsbench/README.md](evaluation/skillsbench/README.md)).
  329. - **Skill library** `data/skillsets/skills_200/` (from `./scripts/download_data.sh --skillsets` or the full download script).
  330. - A **workspace** at `data/gos_workspace/skills_200_v1`: either build with `gos index` or download with `./scripts/download_data.sh --workspace` ([Quick Start, Step 3](#step-3-get-a-workspace-choose-one-path); full detail in [DATA.md](DATA.md)).
  331. The embedding model in `.env` must match how that workspace was built (same `GOS_EMBEDDING_MODEL` / `GOS_EMBEDDING_DIM` as at index time).
  332. **1. Retrieval smoke test**
  333. ```bash
  334. uv run gos retrieve "unit tests with pytest" \
  335. --workspace data/gos_workspace/skills_200_v1 --max-skills 3
  336. ```
  337. You should see **`SKILL_HIT`** and at least one skill block. If you get errors about embedding dimension or missing keys, fix `.env` before continuing.
  338. **2. Generate a single graph-skills task pack**
  339. From the **repository root**, materialize one task (`dialogue-parser` is a small, standard example; it must exist under `evaluation/skillsbench/tasks/`):
  340. ```bash
  341. uv run python evaluation/skillsbench/graphskills_benchmark.py \
  342. --skillset-name skills_200 \
  343. --task dialogue-parser \
  344. --skip-allskills --skip-vectorskills \
  345. --output-root evaluation/skillsbench/generated_verify
  346. ```
  347. This writes `evaluation/skillsbench/generated_verify/tasks_graph_skills/dialogue-parser/` with the graph-retrieval sidecar and mounts your workspace into the task image.
  348. **3. Run that task with Harbor**
  349. Still from the repo, load keys then run Harbor **from `evaluation/skillsbench/`** so paths resolve like the rest of the eval docs:
  350. ```bash
  351. cd evaluation/skillsbench
  352. set -a && source ../../.env && set +a
  353. harbor run --agent gemini-cli \
  354. --model gemini/gemini-3-flash-preview \
  355. --force-build \
  356. -p generated_verify/tasks_graph_skills/dialogue-parser \
  357. -o jobs/verify-sample
  358. ```
  359. Use a `--model` string your Harbor agent accepts (often the same family as in `.env`). First run may spend time on **image build**.
  360. **4. What "success" looks like**
  361. - Harbor finishes with **`Errors: 0`** in the summary table.
  362. - A **`result.json`** appears under `evaluation/skillsbench/jobs/verify-sample/<timestamp>/`.
  363. - **Reward** may be `0.0`, partial (e.g. `0.5`), or `1.0` depending on the task and agent; that is normal. The goal of this minimal path is to confirm **retrieval, task packaging, Docker, and the agent all run**, not to maximize score.
  364. For more agents, configs, and batch YAML, see [evaluation/skillsbench/README.md](evaluation/skillsbench/README.md).
  365. </details>
  366. ## Repository Layout
  367. ```
  368. graph-of-skills/
  369. ├── gos/ # Core GoS package
  370. │ ├── core/ # Engine, retrieval, parsing, schema
  371. │ ├── interfaces/ # CLI, MCP server, Claude Code plugin
  372. │ └── utils/ # Configuration (pydantic-settings)
  373. ├── data/ # Downloaded data (gitignored; see DATA.md)
  374. │ ├── skillsets/ # Skill libraries (skills_200, 500, 1000, 2000)
  375. │ └── gos_workspace/ # Indexed or prebuilt graph workspaces
  376. ├── evaluation/ # See evaluation/README.md
  377. │ ├── alfworld_run.py # ALFWorld benchmark runner
  378. │ ├── skill.py # SkillModule adapter for GoS
  379. │ ├── analysis/ # Offline analyses — evaluation/analysis/README.md
  380. │ └── skillsbench/ # SkillsBench — evaluation/skillsbench/README.md
  381. │ └── experiments/ # Ablation conditions, batch configs, dashboard
  382. ├── skills/ # Agent bootstrap skills for retrieval
  383. ├── scripts/ # Utility scripts (data download, etc.)
  384. ├── tests/ # Test suite
  385. ├── pyproject.toml # Package definition & CLI entry points
  386. ├── .env.example # Environment variable template
  387. ├── DATA.md # Data & downloads
  388. └── CLAUDE.md # Claude Code / MCP integration guide
  389. ```
  390. ## Documentation
  391. | Document | What it covers |
  392. |----------|----------------|
  393. | [CLAUDE.md](CLAUDE.md) | **Claude Code integration**: MCP plugin setup, available tools, example workflows, configuration |
  394. | [DATA.md](DATA.md) | Downloading skill sets, SkillsBench tasks, and prebuilt workspaces (`scripts/download_data.sh`); rebuilding a workspace from source; packaging uploads for HuggingFace |
  395. | [evaluation/README.md](evaluation/README.md) | **Evaluation overview**: ALFWorld, SkillsBench runners, retrieval modes (`gos` / `vector` / `all_full` / `none`), environment setup for benchmark tracks |
  396. | [evaluation/analysis/README.md](evaluation/analysis/README.md) | **Offline analysis tools**: graph diagnostics, build comparison, matched retrieval ablations, metadata stress, edge audit |
  397. | [evaluation/skillsbench/README.md](evaluation/skillsbench/README.md) | **SkillsBench detail**: Harbor, Docker, generating task variants (`graphskills_benchmark.py`), batch configs, agents |
  398. | [`.env.example`](.env.example) | All `GOS_*` and provider variables for indexing, retrieval, and CLI |
  399. | [CONTRIBUTING.md](CONTRIBUTING.md) | Dev setup, tests, project layout for contributors |
  400. ## License
  401. This project is licensed under the [MIT License](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.