Skip to content
Back to search

Trace2Skill

Official codebase of the paper -- Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/qwen-applications--trace2skill/badge.svg)](/listings/qwen-applications--trace2skill)

HTML

<a href="/listings/qwen-applications--trace2skill"><img src="/api/listings/qwen-applications--trace2skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Clone this repository and install the lightweight runtime dependencies:

git clone https://github.com/Qwen-Applications/Trace2Skill.git
cd Trace2Skill
python -m pip install openai tqdm openpyxl requests diskcache

The runners use OpenAI-compatible chat APIs by default. Set the API credentials for your provider:

export OPENAI_API_KEY=<your_api_key>
export OPENAI_BASE_URL=<optional_openai_compatible_endpoint>

For local OpenAI-compatible serving, pass `--api-key EMPTY` and `--base-url http://localhost:8000/v1` to the analysis or skill-evolution entrypoints.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 373 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <h1>Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills</h1>
  3. <!-- Badges -->
  4. <a href="https://github.com/Qwen-Applications">
  5. <img src="https://img.shields.io/badge/Qwen-Applications-4433FF?style=for-the-badge" alt="Qwen Applications">
  6. </a>
  7. <a href="https://arxiv.org/abs/2603.25158">
  8. <img src="https://img.shields.io/badge/arXiv-2603.25158-b31b1b.svg?style=for-the-badge" alt="arXiv">
  9. </a>
  10. <a href="https://github.com/Qwen-Applications/Trace2Skill">
  11. <img src="https://img.shields.io/badge/Github-Code-black?style=for-the-badge&logo=github" alt="Github">
  12. </a>
  13. <a href="https://opensource.org/licenses/Apache-2.0">
  14. <img src="https://img.shields.io/badge/License-Apache%202.0-green.svg?style=for-the-badge" alt="License">
  15. </a>
  16. <p align="center">
  17. <i><b>Qwen Large Model Application Team, Alibaba</b></i>
  18. </p>
  19. In this project, we provide the official code and released spreadsheet skills for Trace2Skill. Trace2Skill automatically adapts and creates agent skills from execution traces. Instead of updating skills sequentially from individual trajectories, it analyzes a pool of traces in parallel, proposes trajectory-local patches with multiple analysts, and hierarchically consolidates them into a unified, conflict-free skill directory.
  20. The paper studies two evolution modes: <b>skill deepening</b> from an existing human-written skill, and <b>skill creation from scratch</b> from a weak initial draft. In addition to spreadsheet tasks, the paper also studies math reasoning and visual question answering.
  21. <p align="center">
  22. <b>Trace2Skill pipeline:</b> trajectory generation -> parallel multi-agent patch proposal -> conflict-free patch consolidation
  23. </p>
  24. </div>
  25. ## 1. Setup and Installation
  26. Clone this repository and install the lightweight runtime dependencies:
  27. ```bash
  28. git clone https://github.com/Qwen-Applications/Trace2Skill.git
  29. cd Trace2Skill
  30. python -m pip install openai tqdm openpyxl requests diskcache
  31. ```
  32. The runners use OpenAI-compatible chat APIs by default. Set the API credentials for your provider:
  33. ```bash
  34. export OPENAI_API_KEY=<your_api_key>
  35. export OPENAI_BASE_URL=<optional_openai_compatible_endpoint>
  36. ```
  37. For local OpenAI-compatible serving, pass `--api-key EMPTY` and `--base-url http://localhost:8000/v1` to the analysis or skill-evolution entrypoints.
  38. ## 2. Data and Released Skills
  39. Prepare a SpreadsheetBench dataset directory or JSONL file and pass it with `--data_path` when running evaluation. The benchmark runner uses the preloaded spreadsheet skills under `spreadsheet_agent/skills/`.
  40. We release the top-performing spreadsheet skills referenced in the paper under `released_skills/`:
  41. | Skill | Setting | Source traces | Location |
  42. |-------|---------|---------------|----------|
  43. | `trace2skill-xlsx-35B-combined` | Self-deepen | 35B combined success/error traces | `released_skills/trace2skill-xlsx-35B-combined/` |
  44. | `xlsx-35B` | Self-create | 35B error traces | `released_skills/xlsx-35B/` |
  45. | `trace2skill-xlsx-122B-combined` | Self-deepen | 122B combined success/error traces | `released_skills/trace2skill-xlsx-122B-combined/` |
  46. | `xlsx-122B` | Self-create | 122B error traces | `released_skills/xlsx-122B/` |
  47. The runtime skill tree in `spreadsheet_agent/skills/` includes the released `xlsx-35B` and `xlsx-122B` variants directly. The full paper release set is preserved separately in `released_skills/`.
  48. ## 3. Running and Skill Evolution
  49. From the repository root, run the SpreadsheetBench agent, evaluate outputs, analyze trajectories, and evolve skills with the following entrypoints:
  50. | Workflow | Command | Output |
  51. |----------|---------|--------|
  52. | Run SpreadsheetBench | `python run_spreadsheetbench.py --data_path <dataset> --model <model>` | Spreadsheet outputs and optional trajectory logs |
  53. | Evaluate outputs | `python evaluate_with_official.py --data_path <dataset> --output_dir <outputs>` | Official-compatible evaluation results |
  54. | Match results and logs | `python analyze_results.py --help` | Failure triage records |
  55. | Agentic error analysis | `python analysis/run_error_analysis.py --help` | `parsed_error_records.json` |
  56. | Single-call success analysis | `python analysis/run_success_analysis_llm.py --help` | `parsed_success_records.json` |
  57. | Parse error analysis | `python analysis/parse_error_analysis_outputs.py --help` | Parsed error JSON |
  58. | Parse success analysis | `python analysis/parse_success_analysis_outputs.py --help` | Parsed success JSON |
  59. | Parallel error-driven skill evolution | `python -m skill_evolver.run_parallel_skill_evolution --help` | Updated skill directory |
  60. | Parallel combined skill evolution | `python -m skill_evolver.run_parallel_combined_skill_evolution --help` | Updated skill directory |
  61. Example benchmark run:
  62. ```bash
  63. python run_spreadsheetbench.py \
  64. --data_path <dataset> \
  65. --model <model> \
  66. --output_dir outputs/spreadsheetbench \
  67. --log_dir outputs/logs
  68. ```
  69. Example parallel skill evolution from parsed error records:
  70. ```bash
  71. python -m skill_evolver.run_parallel_skill_evolution \
  72. --input-json <analysis_output_or_parsed_error_records.json> \
  73. --skill-dir spreadsheet_agent/skills/xlsx/ \
  74. --model <model> \
  75. --max-workers 4 \
  76. --save-intermediates
  77. ```
  78. The skill-evolver entrypoints accept either parsed JSON files or the corresponding analysis output directories directly.
  79. ## 4. Reproduction Note
  80. The spreadsheet verified data for reproduction is included under `data/spreadsheetbench_verified/spreadsheetbench_verified_400`. The commands below cover baseline evaluation, error/success analysis, Trace2Skill evolution, and evolved-skill evaluation. Set `MODEL` to your OpenAI-compatible served model name or path. The commands use the Qwen reproduction configs in `gen_config/`; for Gemma runs, replace them with `gen_config/gemma4_instruct.json` and `gen_config/gemma4_thinking.json`.
  81. Set common reproduction variables:
  82. ```bash
  83. DATA_PATH=data/spreadsheetbench_verified/spreadsheetbench_verified_400
  84. MODEL=Qwen3.5-122B-A10B
  85. WORKERS=128
  86. SEED=41
  87. CLI_ONLY_DIR=outputs/reproduce/cli_only_baseline
  88. BASELINE_DIR=outputs/reproduce/baseline_run
  89. GENERATION_CONFIG=gen_config/qwen3.5_35B_122B_instruct_reasoning.json
  90. THINK_GENERATION_CONFIG=gen_config/qwen3.5_35B_122B_thinking_reasoning.json
  91. ```
  92. Run the `cli_only` baseline (i.e., no skill; only command line as a tool) on the held-out split (`200:400`) for comparison:
  93. ```bash
  94. python run_spreadsheetbench.py \
  95. --data_path "$DATA_PATH" \
  96. --model "$MODEL" \
  97. --agent cli_only \
  98. --log_dir "$CLI_ONLY_DIR/logs" \
  99. --log_format markdown \
  100. --working_dir "$CLI_ONLY_DIR/work" \
  101. --output_dir "$CLI_ONLY_DIR/outputs" \
  102. --max_turns 100 \
  103. --workers "$WORKERS" \
  104. --seeds "$SEED" \
  105. --generation_config "$GENERATION_CONFIG" \
  106. --start_idx 200 \
  107. --end_idx 400
  108. python evaluate_with_official.py \
  109. --data_path "$DATA_PATH" \
  110. --output_dir "$CLI_ONLY_DIR/outputs" \
  111. --verbose \
  112. --start_idx 200 \
  113. --end_idx 400
  114. ```
  115. Run the skill-preloaded spreadsheet agent on the training split (`0:200`) and produce the error/success analysis records:
  116. ```bash
  117. python run_spreadsheetbench.py \
  118. --data_path "$DATA_PATH" \
  119. --model "$MODEL" \
  120. --agent cli_skill_preloaded \
  121. --log_dir "$BASELINE_DIR/logs" \
  122. --log_format markdown \
  123. --working_dir "$BASELINE_DIR/work" \
  124. --output_dir "$BASELINE_DIR/outputs" \
  125. --max_turns 100 \
  126. --workers "$WORKERS" \
  127. --skills_dir spreadsheet_agent/skills \
  128. --seeds "$SEED" \
  129. --generation_config "$GENERATION_CONFIG" \
  130. --start_idx 0 \
  131. --end_idx 200
  132. python evaluate_with_official.py \
  133. --data_path "$DATA_PATH" \
  134. --output_dir "$BASELINE_DIR/outputs" \
  135. --verbose \
  136. --start_idx 0 \
  137. --end_idx 200
  138. python analyze_results.py \
  139. --eval_results "$BASELINE_DIR/outputs/eval_official_results.json" \
  140. --log_dir "$BASELINE_DIR/logs"
  141. python analysis/run_error_analysis.py \
  142. --data_path "$DATA_PATH" \
  143. --work_dir "$BASELINE_DIR/work" \
  144. --logs_dir "$BASELINE_DIR/logs" \
  145. --output_dir "$BASELINE_DIR/error_analysis" \
  146. --model "$MODEL" \
  147. --workers "$WORKERS" \
  148. --generation_config "$GENERATION_CONFIG" \
  149. --max_turns 100
  150. python analysis/run_success_analysis_llm.py \
  151. --logs_dir "$BASELINE_DIR/logs" \
  152. --output_dir "$BASELINE_DIR/success_analysis" \
  153. --model "$MODEL" \
  154. --max_workers "$WORKERS" \
  155. --generation_config "$THINK_GENERATION_CONFIG"
  156. python analysis/parse_error_analysis_outputs.py \
  157. --input_dir "$BASELINE_DIR/error_analysis" \
  158. --output "$BASELINE_DIR/error_analysis_parsed.json" 2>&1 \
  159. | tee "$BASELINE_DIR/error_analysis_parsed.log"
  160. python analysis/parse_success_analysis_outputs.py \
  161. --input_dir "$BASELINE_DIR/success_analysis" \
  162. --output "$BASELINE_DIR/success_analysis_parsed.json" 2>&1 \
  163. | tee "$BASELINE_DIR/success_analysis_parsed.log"
  164. ```
  165. Run Trace2Skill evolution from the parsed training-split error analysis records:
  166. ```bash
  167. PATCH_FORMAT=json
  168. EVOLVE_ROOT=outputs/reproduce/skill_evolution_seed_${SEED}
  169. EVOLUTION_DIR="$EVOLVE_ROOT/error_driven_skill_evolution"
  170. EVOLVED_RUN_DIR=outputs/reproduce/evolved_run_seed_${SEED}
  171. EVOLVED_SKILLS="$EVOLUTION_DIR/skills"
  172. mkdir -p "$EVOLVED_SKILLS"
  173. cp -r spreadsheet_agent/skills/. "$EVOLVED_SKILLS"
  174. python -m skill_evolver.run_parallel_skill_evolution \
  175. --input-json "$BASELINE_DIR/error_analysis_parsed.json" \
  176. --skill-dir "$EVOLVED_SKILLS/xlsx" \
  177. --model "$MODEL" \
  178. --verbose \
  179. --batch-size 1 \
  180. --changelog "$EVOLUTION_DIR/change.log" \
  181. --save-intermediates \
  182. --intermediates-dir "$EVOLUTION_DIR/intermediates" \
  183. --max-workers "$WORKERS" \
  184. --prompt generic \
  185. --generation-config "$THINK_GENERATION_CONFIG" \
  186. --parse-failure-dir "$EVOLUTION_DIR/parse_failures" \
  187. --patch-pipeline "$PATCH_FORMAT" \
  188. --seed "$SEED"
  189. ```
  190. Because the pipeline is long and the ReAct agent can work up to 100 steps, exact 100% reproduction can be difficult even when all random seeds are controlled. The general trend and paper takeaways should still be reproducible. Also, because all skills are modified by LLMs, a single hallucinated edit can damage overall skill robustness. To reduce this risk, run the training-set validation below and verify that the Trace2Skill-evolved skill does not hurt training-set performance compared with the baseline skill before using it for final evaluation.
  191. In the paper, we ran skill evolution plus this training-set validation with three random seeds, then selected the evolved skill with the best training-set validation performance before evaluating it on the held-out split.
  192. For that training-set validation on the evolution split (`0:200`), compare the baseline and evolved skill `eval_official_results.json` summaries:
  193. ```bash
  194. VALIDATION_DIR=outputs/reproduce/validation_train
  195. python run_spreadsheetbench.py \
  196. --data_path "$DATA_PATH" \
  197. --model "$MODEL" \
  198. --log_dir "$VALIDATION_DIR/baseline_logs" \
  199. --working_dir "$VALIDATION_DIR/baseline_work" \
  200. --output_dir "$VALIDATION_DIR/baseline_outputs" \
  201. --max_turns 100 \
  202. --workers "$WORKERS" \
  203. --skills_dir spreadsheet_agent/skills \
  204. --seeds "$SEED" \
  205. --generation_config "$GENERATION_CONFIG" \
  206. --start_idx 0 \
  207. --end_idx 200
  208. python evaluate_with_official.py \
  209. --data_path "$DATA_PATH" \
  210. --output_dir "$VALIDATION_DIR/baseline_outputs" \
  211. --start_idx 0 \
  212. --end_idx 200
  213. python run_spreadsheetbench.py \
  214. --data_path "$DATA_PATH" \
  215. --model "$MODEL" \
  216. --log_dir "$VALIDATION_DIR/evolved_logs" \
  217. --working_dir "$VALIDATION_DIR/evolved_work" \
  218. --output_dir "$VALIDATION_DIR/evolved_outputs" \
  219. --max_turns 100 \
  220. --workers "$WORKERS" \
  221. --skills_dir "$EVOLVED_SKILLS" \
  222. --seeds "$SEED" \
  223. --generation_config "$GENERATION_CONFIG" \
  224. --start_idx 0 \
  225. --end_idx 200
  226. python evaluate_with_official.py \
  227. --data_path "$DATA_PATH" \
  228. --output_dir "$VALIDATION_DIR/evolved_outputs" \
  229. --start_idx 0 \
  230. --end_idx 200
  231. ```
  232. After selecting the best seed by training-set validation, evaluate that evolved skill on the held-out split (`200:400`):
  233. ```bash
  234. python run_spreadsheetbench.py \
  235. --data_path "$DATA_PATH" \
  236. --model "$MODEL" \
  237. --log_dir "$EVOLVED_RUN_DIR/logs" \
  238. --log_format markdown \
  239. --working_dir "$EVOLVED_RUN_DIR/work" \
  240. --output_dir "$EVOLVED_RUN_DIR/outputs" \
  241. --max_turns 100 \
  242. --workers "$WORKERS" \
  243. --skills_dir "$EVOLVED_SKILLS" \
  244. --seeds "$SEED" \
  245. --generation_config "$GENERATION_CONFIG" \
  246. --start_idx 200 \
  247. --end_idx 400
  248. python evaluate_with_official.py \
  249. --data_path "$DATA_PATH" \
  250. --output_dir "$EVOLVED_RUN_DIR/outputs" \
  251. --start_idx 200 \
  252. --end_idx 400
  253. ```
  254. ## Repository Structure
  255. ```text
  256. Trace2Skill/
  257. ├── README.md
  258. ├── analysis/ # Error/success trajectory analysis scripts and prompts
  259. │ ├── parse_error_analysis_outputs.py
  260. │ ├── parse_success_analysis_outputs.py
  261. │ ├── run_error_analysis.py # Agentic error analyst
  262. │ └── run_success_analysis_llm.py # Single-call success analyst
  263. ├── released_skills/ # Released paper skill artifacts
  264. │ ├── trace2skill-xlsx-35B-combined/
  265. │ ├── trace2skill-xlsx-122B-combined/
  266. │ ├── xlsx-35B/
  267. │ └── xlsx-122B/
  268. ├── skill_evolver/ # Parallel Trace2Skill patch proposal and consolidation
  269. │ ├── run_parallel_skill_evolution.py
  270. │ └── run_parallel_combined_skill_evolution.py
  271. ├── spreadsheet_agent/ # SpreadsheetBench agent and runtime skills
  272. │ ├── agents/
  273. │ ├── skills/
  274. │ └── tools/
  275. ├── src/react_agent/ # ReAct agent and OpenAI-compatible model clients
  276. ├── run_spreadsheetbench.py # SpreadsheetBench runner
  277. ├── evaluate_with_official.py # Official-compatible scorer wrapper
  278. ├── analyze_results.py # Result/log matching and triage
  279. └── spreadsheetbench_support.py # Shared SpreadsheetBench utilities
  280. ```
  281. Core implementations:
  282. - `skill_evolver/parallel_evolving_agent.py`
  283. - `skill_evolver/parallel_success_evolving_agent.py`
  284. - `skill_evolver/skill_evolving_agent.py`
  285. - `analysis/error_analysis_agent.py`
  286. - `spreadsheet_agent/agents/cli_skill_preloaded_agent.py`
  287. ## Acknowledgements
  288. This repository focuses on the spreadsheet setting and released skills discussed in the paper, while keeping the core Trace2Skill evolution pipeline runnable. We thank the developers and communities behind the tools used by this release:
  289. - [Qwen](https://github.com/QwenLM) and the Qwen application ecosystem
  290. - [SpreadsheetBench](https://arxiv.org/abs/2406.14991) for spreadsheet-agent evaluation
  291. - [openpyxl](https://openpyxl.readthedocs.io/) for spreadsheet manipulation support
  292. ## License
  293. This project is released under the [Apache License 2.0](https://opensource.org/licenses/Apache-2.0). Released skill artifacts may include their own license files where noted.
  294. ## Citation
  295. If you find our work useful in your research, please consider citing our paper:
  296. ```bibtex
  297. @misc{ni2026trace2skilldistilltrajectorylocallessons,
  298. title={Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills},
  299. author={Jingwei Ni and Yihao Liu and Xinpeng Liu and Yutao Sun and Mengyu Zhou and Pengyu Cheng and Dexin Wang and Erchao Zhao and Xiaoxi Jiang and Guanjun Jiang},
  300. year={2026},
  301. eprint={2603.25158},
  302. archivePrefix={arXiv},
  303. primaryClass={cs.AI},
  304. url={https://arxiv.org/abs/2603.25158},
  305. }
  306. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.