Skip to content
Back to search

SR-Agents

SRA-Bench and SR-Agents: a benchmark and toolkit for skill-retrieval-augmented LLM agents.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/oneal2000--sr-agents/badge.svg)](/listings/oneal2000--sr-agents)

HTML

<a href="/listings/oneal2000--sr-agents"><img src="/api/listings/oneal2000--sr-agents/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires Python 3.10 – 3.12.

pip install -e .       # or: uv sync

SRA-Bench ships with this repository under `data/bench/`. After cloning, unzip the skill corpus once:

unzip data/bench/corpus/corpus.json.zip -d data/bench/corpus/

The same data is also published on [HuggingFace](https://huggingface.co/datasets/WeihangSu/SRA-Bench) — use either source, the contents are identical:

huggingface-cli download WeihangSu/SRA-Bench --repo-type dataset \
    --local-dir data/bench

<details><summary>ToolQA external corpus (only needed to run ToolQA)</summary>

Download from the [ToolQA Google Drive link](https://drive.google.com/file/d/1zRbHzPW2x4dDcfmphBWlan8cxUCRNmqk/view?usp=drive_link), unzip, and place the result under `data/external/toolqa/`. </details>

Inference requires an OpenAI-compatible chat endpoint. Point `--api-base` at any compatible server (OpenAI, vLLM, SGLang, Ollama, …). `--model` is the served model identifier — a model ID like `gpt-4o-mini` for hosted APIs, or whatever string the local server is serving (often a path like `/models/Qwen3-32B`) for vLLM/SGLang. For endpoints that require auth, set `OPENAI_API_KEY`; local unauthenticated servers accept any value.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 466 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center">
  2. Skill Retrieval Augmentation for Agentic AI
  3. </h1>
  4. <h2 align="center">
  5. RAG retrieves knowledge. SRA retrieves capabilities.
  6. </h2>
  7. <p align="center">
  8. <a href="https://sr-agents.github.io">
  9. <img src="https://img.shields.io/badge/🌐%20Project%20Website-SR--Agents.github.io-blue?style=for-the-badge" alt="Project Website">
  10. </a>
  11. </p>
  12. <p align="center">
  13. <a href="https://arxiv.org/abs/2604.24594">
  14. <img src="https://img.shields.io/badge/arXiv-2604.24594-b31b1b.svg" alt="arXiv">
  15. </a>
  16. <a href="https://huggingface.co/datasets/WeihangSu/SRA-Bench">
  17. <img src="https://img.shields.io/badge/🤗%20HuggingFace-SRA--Bench-yellow" alt="HuggingFace">
  18. </a>
  19. <a href="https://github.com/oneal2000/SR-Agents/stargazers">
  20. <img src="https://img.shields.io/github/stars/oneal2000/SR-Agents?style=social" alt="GitHub stars">
  21. </a>
  22. <a href="https://github.com/oneal2000/SR-Agents/blob/main/LICENSE">
  23. <img src="https://img.shields.io/badge/License-MIT-green.svg" alt="License">
  24. </a>
  25. </p>
  26. <p align="center">
  27. <b>A community resource for studying and evaluating Skill Retrieval Augmentation (SRA).</b><br><br>
  28. This repository releases SRA-Bench and SR-Agents, providing data, baselines, and evaluation pipelines for research on retrieval-based skill augmentation in LLM agents.<br><br>
  29. ⭐ If you find this resource useful, we would be truly grateful if you could star this repo and cite our paper~
  30. </p>
  31. ---
  32. ## 🔥 Why Skill Retrieval Augmentation?
  33. Modern LLM agents are no longer just text generators. They increasingly rely on external **skills**: reusable capability packages that may include natural-language instructions, tool-use procedures, executable code, and auxiliary resources.
  34. However, the dominant way to use skills today is still simple but unscalable: put available skills into the prompt and ask the model to choose. As skill libraries grow, this approach quickly runs into context-window limits and selection failures.
  35. **Skill Retrieval Augmentation (SRA)** treats skills as a large external capability corpus. Instead of showing all skills in context, an agent must:
  36. 1. **Retrieve** relevant skills from a large skill library;
  37. 2. **Incorporate** the useful skills into its active problem-solving state;
  38. 3. **Apply** the skills correctly to improve downstream task performance.
  39. This makes SRA a capability-centric counterpart to RAG:
  40. RAG retrieves knowledge; **SRA retrieves executable capabilities**.
  41. ![SRA paradigm overview](assets/overall.png)
  42. ---
  43. ## 🚀 What is included?
  44. This repository releases the code and data for **Skill Retrieval Augmentation for Agentic AI**.
  45. ### SRA-Bench
  46. The first benchmark for decomposed evaluation of Skill Retrieval Augmentation:
  47. - **5,400** capability-intensive test instances;
  48. - **636** manually constructed gold skills;
  49. - **26,262** total skills in the retrieval corpus;
  50. - Gold skills mixed with **25,626** web-collected distractor skills;
  51. - Six task families: TheoremQA, LogicBench, ToolQA, MedCalc-Bench, CHAMP, and BigCodeBench;
  52. - Evaluation of the full SRA pipeline: skill retrieval, skill incorporation, and end-task execution.
  53. | Dataset | Capability Type | #Inst. | #Skills | Skill Mapping | Evaluation |
  54. |---|---|---:|---:|---|---|
  55. | TheoremQA | Theorem Application | 747 | 320 | Single | Rule-Based |
  56. | LogicBench | Logical Reasoning Patterns | 760 | 19 | Single | Rule-Based |
  57. | ToolQA | Tool-Use Workflows | 1,430 | 14 | Single | Rule-Based |
  58. | MedCalc-Bench | Medical Calculators | 1,100 | 55 | Single | Rule-Based |
  59. | CHAMP | Mathematical Concepts | 223 | 89 | Multi | Rule-Based |
  60. | BigCodeBench | Software Libraries | 1,140 | 139 | Multi | Execution |
  61. ### SR-Agents
  62. A research toolkit and baseline family for building and evaluating skill-retrieval-augmented agents:
  63. - Skill retrieval with BM25, TF-IDF, BGE, Contriever, Hybrid retrieval, and BM25 + LLM reranking;
  64. - Skill-use methods including LLM Direct, Oracle Skill, Full-Skill Injection, LLM Selection, and Progressive Disclosure;
  65. - End-to-end CLI pipeline: `retrieve → infer → evaluate`;
  66. - Support for OpenAI-compatible endpoints, including hosted APIs, vLLM, SGLang, Ollama, and local model servers.
  67. ---
  68. ## ⭐ Star and cite
  69. If you find **SRA-Bench** useful, or if any findings in our paper help your research, we would be truly grateful if you could:
  70. ⭐ star this repository to help more researchers discover this resource;
  71. 📌 cite our paper in your work.
  72. Your support really means a lot to us!
  73. ```bibtex
  74. @article{su2026skill,
  75. title={Skill Retrieval Augmentation for Agentic AI},
  76. author={Su, Weihang and Long, Jianming and Ai, Qingyao and Tang, Yichen and Wang, Changyue and Tu, Yiteng and Liu, Yiqun},
  77. journal={arXiv preprint arXiv:2604.24594},
  78. year={2026}
  79. }
  80. ```
  81. ## Install
  82. Requires Python 3.10 – 3.12.
  83. ```bash
  84. pip install -e . # or: uv sync
  85. ```
  86. SRA-Bench ships with this repository under `data/bench/`. After
  87. cloning, unzip the skill corpus once:
  88. ```bash
  89. unzip data/bench/corpus/corpus.json.zip -d data/bench/corpus/
  90. ```
  91. The same data is also published on
  92. [HuggingFace](https://huggingface.co/datasets/WeihangSu/SRA-Bench) —
  93. use either source, the contents are identical:
  94. ```bash
  95. huggingface-cli download WeihangSu/SRA-Bench --repo-type dataset \
  96. --local-dir data/bench
  97. ```
  98. <details><summary>ToolQA external corpus (only needed to run ToolQA)</summary>
  99. Download from the
  100. [ToolQA Google Drive link](https://drive.google.com/file/d/1zRbHzPW2x4dDcfmphBWlan8cxUCRNmqk/view?usp=drive_link),
  101. unzip, and place the result under `data/external/toolqa/`.
  102. </details>
  103. Inference requires an OpenAI-compatible chat endpoint. Point
  104. `--api-base` at any compatible server (OpenAI, vLLM, SGLang, Ollama, …).
  105. `--model` is the served model identifier — a model ID like
  106. `gpt-4o-mini` for hosted APIs, or whatever string the local server is
  107. serving (often a path like `/models/Qwen3-32B`) for vLLM/SGLang. For
  108. endpoints that require auth, set `OPENAI_API_KEY`; local unauthenticated
  109. servers accept any value.
  110. ## Quickstart
  111. An end-to-end run of the three stages on TheoremQA using Full-Skill
  112. Injection with the BM25 top-1 skill:
  113. ```bash
  114. # Pick one (examples; any OpenAI-compatible endpoint works).
  115. # Hosted: MODEL=gpt-4o-mini API_BASE=https://api.openai.com/v1
  116. # Local : MODEL=/models/Qwen3-32B API_BASE=http://localhost:8000/v1
  117. MODEL=<MODEL>
  118. API_BASE=<API_BASE>
  119. # 1. Retrieve — BM25 top-50 per query.
  120. sragents retrieve --retriever bm25 \
  121. --corpus data/bench/corpus/corpus.json \
  122. --instances data/bench/instances/theoremqa.json \
  123. --output results/retrieval/theoremqa-bm25.json
  124. # 2. Infer — prepend the top-1 BM25 skill and generate an answer.
  125. sragents infer \
  126. --instances data/bench/instances/theoremqa.json \
  127. --output results/inference/theoremqa-bm25_top1.jsonl \
  128. --model $MODEL --api-base $API_BASE \
  129. --provider topk \
  130. --provider-arg source=results/retrieval/theoremqa-bm25.json \
  131. --provider-arg k=1 \
  132. --engine direct --label bm25_top1
  133. # 3. Evaluate — extract + score against ground truth.
  134. sragents evaluate \
  135. --input results/inference/theoremqa-bm25_top1.jsonl \
  136. --instances data/bench/instances/theoremqa.json \
  137. --output results/eval/theoremqa-bm25_top1.json
  138. ```
  139. The evaluator prints overall accuracy and saves a JSON of the form
  140. ```json
  141. {
  142. "dataset": "theoremqa", "method": "bm25_top1", "model": "...",
  143. "metrics": {"accuracy": 0.XX, "correct": N, "total": 747},
  144. "details": [{"instance_id": "...", "extracted_answer": "...",
  145. "correct": true, "ground_truth": "...", ...}]
  146. }
  147. ```
  148. ## Pipeline
  149. The toolkit is organised as three CLI stages that pass JSON files
  150. between them:
  151. ```
  152. sragents retrieve ─▶ retrieval/*.json ranked skills + Recall@K, nDCG@K
  153. sragents infer ─▶ inference/*.jsonl raw model output per instance
  154. sragents evaluate ─▶ eval/*.json end-task accuracy
  155. ```
  156. Each stage's output is consumed by the next via an explicit path
  157. argument, so any stage can be swapped or rerun in isolation, and
  158. every stage supports per-instance resume.
  159. `infer` is parameterised by two orthogonal axes:
  160. * **SkillProvider** — which candidate skills the instance receives:
  161. none, oracle (gold), top-K retrieval, LLM-selected, or oracle plus
  162. hard-negative distractors. The pool may already be narrowed to a
  163. single skill, or left wide for the engine to narrow further.
  164. * **InferenceEngine** — how those candidates are turned into an
  165. answer. `direct` statically prepends everything it receives;
  166. agentic engines (`progressive_disclosure`, `react`,
  167. `react_progressive_disclosure`) interleave further skill selection
  168. with solving inside their reasoning loop.
  169. The five skill-use methods studied in our experiments are specific
  170. (Provider, Engine) combinations:
  171. | Method | Provider | Engine | Description |
  172. |---|---|---|---|
  173. | **LLM Direct** | `none` | `direct` | No external skill — parametric-only baseline |
  174. | **Oracle Skill** | `oracle` | `direct` | Annotated gold skill prepended (upper bound) |
  175. | **Full-Skill Injection** | `topk(k=1)` | `direct` | Full content of the BM25 rank-1 skill prepended to the prompt |
  176. | **LLM Selection** | `llm_select(pool=50)` | `direct` | Model picks one skill from BM25 top-50, then answers |
  177. | **Progressive Disclosure** | `topk(k=50)` | `progressive_disclosure` | Model sees a compact skill catalog and loads skills on demand |
  178. For ToolQA the `direct` engine is replaced by `react` and
  179. `progressive_disclosure` by `react_progressive_disclosure`; the
  180. experiment runner selects the right engine per dataset.
  181. ## CLI
  182. A single `sragents` command is installed (also invokable as
  183. `python -m sragents.cli.main`). Every subcommand has `--help`.
  184. ```bash
  185. sragents list retrievers # bm25 tfidf bge contriever
  186. sragents list providers # none oracle topk llm_select oracle_distractor
  187. sragents list engines # direct progressive_disclosure react react_progressive_disclosure
  188. sragents list datasets # theoremqa logicbench toolqa champ medcalcbench bigcodebench
  189. sragents list experiments # main, retrieval_comparison, topk_sweep, distractor, ...
  190. ```
  191. ### 1. Retrieve
  192. ```bash
  193. sragents retrieve \
  194. --retriever bm25 \
  195. --corpus data/bench/corpus/corpus.json \
  196. --instances data/bench/instances/theoremqa.json \
  197. --output results/retrieval/theoremqa-bm25.json \
  198. --top-k 50
  199. ```
  200. Dense retrievers default to the same checkpoints used in our
  201. experiments — `BAAI/bge-base-en-v1.5` for `bge` and
  202. `facebook/contriever-msmarco` for `contriever`. Override with
  203. `--retriever-arg model_path=<hf-name>` to swap in a different model.
  204. Retrievers can also be cascaded. `sragents rerank` is a second-stage
  205. retriever: it reads an existing retrieval file, uses an LLM to reorder
  206. the top-K candidates per query, and writes a new retrieval file in the
  207. same format. A typical cascade is BM25 → LLM rerank:
  208. ```bash
  209. sragents rerank \
  210. --input results/retrieval/theoremqa-bm25.json \
  211. --output results/retrieval/theoremqa-rerank_bm25.json \
  212. --instances data/bench/instances/theoremqa.json \
  213. --model <MODEL> --api-base <API_BASE> \
  214. --top-k 50
  215. ```
  216. `sragents hybrid` is the round-robin fuser used for our Hybrid
  217. (BM25 + BGE) retriever — see the reproduction recipe below.
  218. ### 2. Infer
  219. ```bash
  220. sragents infer \
  221. --instances data/bench/instances/theoremqa.json \
  222. --output results/inference/theoremqa-Qwen3-32B-bm25_top1.jsonl \
  223. --model <MODEL> --api-base <API_BASE> \
  224. --provider topk \
  225. --provider-arg source=results/retrieval/theoremqa-bm25.json \
  226. --provider-arg k=1 \
  227. --engine direct \
  228. --label bm25_top1
  229. ```
  230. Common recipes:
  231. | Method | CLI fragment |
  232. |---|---|
  233. | LLM Direct | `--provider none --engine direct` |
  234. | Oracle Skill | `--provider oracle --engine direct` |
  235. | Full-Skill Injection | `--provider topk --provider-arg source=… --provider-arg k=1 --engine direct` |
  236. | LLM Selection | `--provider llm_select --provider-arg source=… --provider-arg pool=50 --engine direct` |
  237. | Progressive Disclosure | `--provider topk --provider-arg source=… --provider-arg k=50 --engine progressive_disclosure` |
  238. | ToolQA | For any of the above, use `--engine react` in place of `direct`, or `--engine react_progressive_disclosure` in place of `progressive_disclosure` |
  239. ### 3. Evaluate
  240. ```bash
  241. sragents evaluate \
  242. --input results/inference/theoremqa-Qwen3-32B-bm25_top1.jsonl \
  243. --instances data/bench/instances/theoremqa.json \
  244. --output results/eval/theoremqa-Qwen3-32B-bm25_top1.json
  245. ```
  246. ## Reproducing our experiments
  247. The retrievers used across all experiments are BM25, TF-IDF, BGE,
  248. Contriever, a round-robin hybrid of BM25 and BGE, and an LLM rerank
  249. of BM25's top-50. Pre-compute the first-stage retrievers, then fuse
  250. BM25 + BGE into the hybrid file:
  251. ```bash
  252. DATASETS="theoremqa logicbench toolqa champ medcalcbench bigcodebench"
  253. # First-stage retrievers.
  254. for ds in $DATASETS; do
  255. for r in bm25 tfidf bge contriever; do
  256. sragents retrieve --retriever $r \
  257. --corpus data/bench/corpus/corpus.json \
  258. --instances data/bench/instances/$ds.json \
  259. --output results/retrieval/$ds-$r.json
  260. done
  261. done
  262. # Round-robin fusion of BM25 + BGE.
  263. for ds in $DATASETS; do
  264. sragents hybrid \
  265. --input results/retrieval/$ds-bm25.json \
  266. results/retrieval/$ds-bge.json \
  267. --output results/retrieval/$ds-hybrid_bm25_bge.json
  268. done
  269. ```
  270. The LLM rerank output is model-dependent (file name:
  271. `*-rerank_bm25-<model>.json`), so it has to be produced once per
  272. evaluated model:
  273. ```bash
  274. MODEL=<MODEL> # same identifier you pass to `sragents infer --model`
  275. API_BASE=<API_BASE>
  276. for ds in $DATASETS; do
  277. sragents rerank \
  278. --input results/retrieval/$ds-bm25.json \
  279. --output results/retrieval/$ds-rerank_bm25-$(basename $MODEL).json \
  280. --instances data/bench/instances/$ds.json \
  281. --model $MODEL --api-base $API_BASE --top-k 50
  282. done
  283. ```
  284. `sragents experiment --exp retrieval_comparison` triggers
  285. `sragents rerank` on demand if the rerank file is missing.
  286. Skill-retrieval metrics (Recall@K, nDCG@K) are computed by
  287. `sragents retrieve` and `sragents rerank` directly into their
  288. output JSONs, so no separate end-to-end run is needed to obtain
  289. them. The end-to-end experiment runners produce inference traces
  290. and end-task accuracy:
  291. ```bash
  292. # Main results: 5 skill-use methods × 6 datasets.
  293. sragents experiment --exp main \
  294. --model <MODEL> --api-base <API_BASE>
  295. # Retriever comparison: rank-1 end-to-end under BM25, TF-IDF, BGE,
  296. # Contriever, and BM25 + Rerank. The rank-1 Hybrid is omitted — its
  297. # round-robin fusion always returns BM25's top-1 at rank 1, so its
  298. # end-to-end numbers are identical to BM25.
  299. sragents experiment --exp retrieval_comparison \
  300. --model <MODEL> --api-base <API_BASE>
  301. # Noise robustness: gold skill plus N ∈ {0, 2, 4, 8} hard-negative
  302. # distractors, drawn by alternately picking non-gold candidates from
  303. # BM25 and BGE rank lists (both retrieval files must already exist).
  304. # Run under Full Skill Injection and Progressive Disclosure exposure.
  305. sragents experiment --exp distractor \
  306. --model <MODEL> --api-base <API_BASE>
  307. ```
  308. The skill-loading-rate analyses in our paper are derived from the
  309. inference traces left behind by `--exp main` rather than from a
  310. separate run.
  311. The catalog also includes `--exp topk_sweep`, an additional
  312. ablation that varies BM25 top-K (K ∈ {1, 2, 4, 8}) under both
  313. exposure modes. Use `--exp topk_sweep_injection` or
  314. `--exp topk_sweep_progressive_disclosure` to restrict to a single
  315. mode.
  316. Narrow the scope of any experiment with `--dataset theoremqa
  317. logicbench` or `--methods bm25_top1 progressive_disclosure` (use
  318. the technical method labels, not the human-readable display names).
  319. The runner invokes `sragents infer` and `sragents evaluate` for
  320. each (dataset, method) cell and skips cells whose output files
  321. already exist.
  322. ## Project layout
  323. ```
  324. src/sragents/
  325. ├── config.py # path constants, ALL_DATASETS
  326. ├── corpus.py # skill corpus loading
  327. ├── llm.py # OpenAI-compatible client
  328. ├── prompts.py # per-dataset prompt builders
  329. │
  330. ├── retrieve/ # Stage 1
  331. │ ├── base.py # Retriever protocol + registry
  332. │ ├── bm25.py · tfidf.py · dense.py · hybrid.py · llm_rerank.py
  333. │ └── metrics.py · schema.py
  334. │
  335. ├── infer/ # Stage 2
  336. │ ├── base.py # SkillProvider + InferenceEngine protocols
  337. │ ├── runner.py # per-instance parallel execution + resume
  338. │ ├── providers/ # none · oracle · topk · llm_select · oracle_distractor
  339. │ └── engines/ # direct · progressive_disclosure · react · react_progressive_disclosure
  340. │
  341. ├── evaluate/ # Stage 3
  342. │ ├── base.py # Evaluator protocol + registry
  343. │ ├── datasets/ # 6 per-dataset evaluators
  344. │ └── metrics.py
  345. │
  346. ├── toolqa/ # ToolQA tools (used by ReAct engines)
  347. ├── experiments/ # experiment catalog + runner
  348. └── cli/ # sragents <subcommand> entry points
  349. data/bench/
  350. ├── corpus/corpus.json # 26,262 skills (636 gold + 25,626 web)
  351. └── instances/*.json # 6 datasets
  352. ```
  353. ## License
  354. See [LICENSE](LICENSE).
  355. ## ⭐ Star and cite
  356. If you find **SRA-Bench** useful, or if any findings in our paper help your research, we would be truly grateful if you could:
  357. ⭐ star this repository to help more researchers discover this resource;
  358. 📌 cite our paper in your work.
  359. Your support really means a lot to us!
  360. ```bibtex
  361. @article{su2026skill,
  362. title={Skill Retrieval Augmentation for Agentic AI},
  363. author={Su, Weihang and Long, Jianming and Ai, Qingyao and Tang, Yichen and Wang, Changyue and Tu, Yiteng and Liu, Yiqun},
  364. journal={arXiv preprint arXiv:2604.24594},
  365. year={2026}
  366. }
  367. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.