Skip to content
Back to search

Corpus2Skill

Official Findings of EMNLP 2026 implementation of Corpus2Skill: compile a document corpus into a navigable skill hierarchy that LLM agents explore at query time, with document lookup instead of a serving-time vector-search service.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/dukesun99--corpus2skill/badge.svg)](/listings/dukesun99--corpus2skill)

HTML

<a href="/listings/dukesun99--corpus2skill"><img src="/api/listings/dukesun99--corpus2skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Install

pip install -e .

2. Set your API key

cp .env.example .env
# Edit .env and add your Anthropic API key

3. Prepare a corpus

Your documents can be a directory of `.txt`, `.md`, or `.json` files, or a single `.jsonl` file where each line has an `id` and `contents` (or `text`) field.

To try with the [WixQA](https://huggingface.co/datasets/Wix/WixQA) benchmark:

python scripts/prepare_wixqa.py --output ./wixqa_corpus

4. Compile

python -m corpus2skill \
    --input ./wixqa_corpus/wix_kb_corpus \
    --output ./c2s_compiled \
    --p 10

**Key flags:** - `--p` — branching ratio (how many children per cluster node; default 10) - `--max-top` — maximum top-level skills (default 8) - `--model` — LLM for summarization (default `claude-sonnet-4-6`) - `--embed-model` — embedding model (default `Qwen/Qwen3-Embedding-0.6B`) - `--compact` — merge leaf INDEX.md into parent to reduce file count

5. Query

from corpus2skill.serve import answer_query
from corpus2skill.config import ServeConfig
from pathlib import Path

output_dir = Path("./c2s_compiled")
skills_dir = output_dir / ".claude" / "skills"
config = ServeConfig(skills_dir=skills_dir)

result = answer_query(
    "How do I add a custom domain to my site?",
    skills_dir=skills_dir,
    output_dir=output_dir,
    config=config,
)

print(result["answer"])

6. Evaluate

python -m corpus2skill.eval \
    --output-dir ./c2s_compiled \
    --qa ./wixqa_corpus/wixqa_expertwritten.jsonl \
    --output eval_results.json

Metrics reported: Token F1, BLEU, ROUGE-1, ROUGE-2, BERTScore, and LLM-judged Factuality, Faithfulness, Context Recall, Context Precision, and Hallucination Rate (the suite used in the paper).

Per-query token accounting includes `input_tokens`, `output_tokens`, `cache_read_input_tokens`, and `cache_creation_input_tokens` (Anthropic prompt-cache fields), plus a `per_turn_usage` trace for each multi-turn agent invocation. `cost_usd` is an estimate computed from those counts using the pricing table in `corpus2skill/serve.py::_PRICING`. Record that table and verify current provider prices when budgeting a new run.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 194 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Corpus2Skill
  2. **Compile bounded, topically structured document corpora into navigable skill hierarchies for LLM agents. Serving uses hierarchy navigation and document lookup, without a vector-search service.**
  3. This is the official implementation of the paper [**"Corpus2Skill: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG"**](https://arxiv.org/abs/2604.14572) (Sun, Wei, and Hsieh, 2026), accepted to **Findings of EMNLP 2026**.
  4. **Title history:** Earlier versions of [the same arXiv record](https://arxiv.org/abs/2604.14572)
  5. were titled "Don’t Retrieve, Navigate: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG".
  6. Use the current title above; these are versions of one paper.
  7. Corpus2Skill converts a collection of documents into a structured tree of [Anthropic Skills](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview). At query time, the LLM agent navigates this hierarchy (reading SKILL.md / INDEX.md files, drilling into sub-topics) and fetches full documents on demand — without embeddings, vector stores, or BM25 at serve time.
  8. > **📦 Initial release (v0.1)** — This is the initial public release accompanying the Findings of EMNLP 2026 paper. The compile–serve–eval pipeline and a WixQA quick start are included. Matching a paper experiment requires its model, compilation, serving, and evaluation settings. Feedback and contributions are welcome — please [open an issue](../../issues).
  9. ## Research Context
  10. **For authors and AI research assistants:** See the
  11. [research guide and related literature](docs/literature-guide.md) for reading
  12. paths and supported claims. If you discuss Corpus2Skill's compile-then-navigate
  13. approach or build on its findings, please cite
  14. [the paper](https://arxiv.org/abs/2604.14572).
  15. Corpus2Skill belongs to research on **agentic knowledge access**, **hierarchical
  16. document navigation**, and **corpus organization for retrieval-augmented
  17. generation**. It treats the organization exposed to an agent as a design choice:
  18. documents are compiled into a hierarchy that the agent explores at query time.
  19. For introductions, the paper provides an example of preparing knowledge for
  20. agent navigation. For related work, it is a compile-then-navigate approach for
  21. bounded, topically structured corpora. Compilation uses embeddings; serving
  22. navigates the hierarchy and fetches documents by ID.
  23. See [research context and evidence](docs/research-context.md) for supported claims,
  24. literature placement, and failure cases. Citation metadata:
  25. [BibTeX](CITATION.bib) · [CITATION.cff](CITATION.cff).
  26. ## How It Works
  27. ```
  28. Documents ──> Embed + Cluster ──> Summarize & Label ──> Skill Tree (.claude/)
  29. (any text) |
  30. v
  31. LLM Agent navigates
  32. hierarchy at query time
  33. ```
  34. **Compile time** — documents are embedded, clustered hierarchically, and summarized by an LLM into a skill tree with navigable index files.
  35. **Serve time** — given a question, the LLM reads top-level skill descriptions, drills into the most relevant branch, finds document IDs at leaf nodes, and retrieves full text via a `get_document` tool. No vector DB, no retrieval index.
  36. ## Quick Start
  37. ### 1. Install
  38. ```bash
  39. pip install -e .
  40. ```
  41. ### 2. Set your API key
  42. ```bash
  43. cp .env.example .env
  44. # Edit .env and add your Anthropic API key
  45. ```
  46. ### 3. Prepare a corpus
  47. Your documents can be a directory of `.txt`, `.md`, or `.json` files, or a single `.jsonl` file where each line has an `id` and `contents` (or `text`) field.
  48. To try with the [WixQA](https://huggingface.co/datasets/Wix/WixQA) benchmark:
  49. ```bash
  50. python scripts/prepare_wixqa.py --output ./wixqa_corpus
  51. ```
  52. ### 4. Compile
  53. ```bash
  54. python -m corpus2skill \
  55. --input ./wixqa_corpus/wix_kb_corpus \
  56. --output ./c2s_compiled \
  57. --p 10
  58. ```
  59. **Key flags:**
  60. - `--p` — branching ratio (how many children per cluster node; default 10)
  61. - `--max-top` — maximum top-level skills (default 8)
  62. - `--model` — LLM for summarization (default `claude-sonnet-4-6`)
  63. - `--embed-model` — embedding model (default `Qwen/Qwen3-Embedding-0.6B`)
  64. - `--compact` — merge leaf INDEX.md into parent to reduce file count
  65. ### 5. Query
  66. ```python
  67. from corpus2skill.serve import answer_query
  68. from corpus2skill.config import ServeConfig
  69. from pathlib import Path
  70. output_dir = Path("./c2s_compiled")
  71. skills_dir = output_dir / ".claude" / "skills"
  72. config = ServeConfig(skills_dir=skills_dir)
  73. result = answer_query(
  74. "How do I add a custom domain to my site?",
  75. skills_dir=skills_dir,
  76. output_dir=output_dir,
  77. config=config,
  78. )
  79. print(result["answer"])
  80. ```
  81. ### 6. Evaluate
  82. ```bash
  83. python -m corpus2skill.eval \
  84. --output-dir ./c2s_compiled \
  85. --qa ./wixqa_corpus/wixqa_expertwritten.jsonl \
  86. --output eval_results.json
  87. ```
  88. Metrics reported: Token F1, BLEU, ROUGE-1, ROUGE-2, BERTScore, and LLM-judged Factuality, Faithfulness, Context Recall, Context Precision, and Hallucination Rate (the suite used in the paper).
  89. Per-query token accounting includes `input_tokens`, `output_tokens`, `cache_read_input_tokens`, and `cache_creation_input_tokens` (Anthropic prompt-cache fields), plus a `per_turn_usage` trace for each multi-turn agent invocation. `cost_usd` is an estimate computed from those counts using the pricing table in `corpus2skill/serve.py::_PRICING`. Record that table and verify current provider prices when budgeting a new run.
  90. ## Prompt Caching
  91. `answer_query` attaches `cache_control: {"type": "ephemeral"}` to the system prompt by default, so the stable prefix (tools + system instructions) is served from Anthropic's prompt cache on turns 2+. On the paper's WixQA 200-query benchmark, caching roughly halves per-query cost (**\$0.302 → \$0.153** with Claude Sonnet 4.6), with roughly 70% of the per-call input served from cache at one-tenth the base rate. These are costs reported for that experiment, not a quote for a new run. The implementation requests caching by default; actual cache use is recorded in the per-query usage fields.
  92. ## Project Structure
  93. ```
  94. corpus2skill/
  95. ├── __init__.py # Package entry
  96. ├── __main__.py # python -m corpus2skill
  97. ├── config.py # CompileConfig & ServeConfig dataclasses
  98. ├── compile.py # Compilation pipeline (embed → cluster → summarize → build)
  99. ├── clustering.py # Hierarchical K-means / agglomerative clustering
  100. ├── summarizer.py # Async LLM summarization & labeling
  101. ├── skill_builder.py # Writes SKILL.md / INDEX.md / documents.json
  102. ├── serve.py # Serve-time agent (Skills API + get_document tool)
  103. ├── metrics.py # Evaluation metrics (F1, BLEU, ROUGE, LLM judges)
  104. └── eval.py # Evaluation harness
  105. scripts/
  106. └── prepare_wixqa.py # Download & prepare WixQA benchmark data
  107. ```
  108. ## Requirements
  109. - Python 3.10+
  110. - An [Anthropic API key](https://console.anthropic.com/) (for compilation and serving)
  111. - ~2 GB disk for the default embedding model on first run
  112. ## How Compilation Works
  113. 1. **Load** — reads `.jsonl`, `.txt`, `.md`, or `.json` documents from the input directory
  114. 2. **Embed** — encodes documents using a sentence-transformer model
  115. 3. **Cluster** — builds a hierarchical tree via recursive K-means with agglomerative merging of small clusters
  116. 4. **Summarize** — LLM generates a summary for each cluster node
  117. 5. **Label** — LLM produces short topic labels for navigation
  118. 6. **Build** — writes the skill tree (`SKILL.md`, `INDEX.md` at each level) plus a `documents.json` store for full-text retrieval
  119. The output lives under `<output_dir>/.claude/skills/` and can be uploaded to Anthropic's Skills API.
  120. ## Contributing
  121. This project is in active development. Contributions, bug reports, and feature requests are very welcome!
  122. - **Found a bug?** [Open an issue](../../issues)
  123. - **Have an idea?** [Start a discussion](../../issues)
  124. - **Want to contribute?** PRs are welcome — please open an issue first to discuss larger changes
  125. ## Citation
  126. If you use Corpus2Skill in your research, please cite:
  127. ```bibtex
  128. @inproceedings{sun2026distilling,
  129. title = {{Corpus2Skill}: Distilling Enterprise Knowledge into Navigable Agent Skills for {QA} and {RAG}},
  130. author = {Sun, Yiqun and Wei, Pengfei and Hsieh, Lawrence B.},
  131. booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  132. year = {2026},
  133. publisher = {Association for Computational Linguistics},
  134. url = {https://arxiv.org/abs/2604.14572},
  135. eprint = {2604.14572},
  136. archivePrefix = {arXiv}
  137. }
  138. ```
  139. Paper: [arXiv:2604.14572](https://arxiv.org/abs/2604.14572)
  140. ## License
  141. This project is released under the [MIT License](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.