Skip to content
Back to search

skill-inject

Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/aisa-group--skill-inject/badge.svg)](/listings/aisa-group--skill-inject)

HTML

<a href="/listings/aisa-group--skill-inject"><img src="/api/listings/aisa-group--skill-inject/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Clone the repository

git clone <repo-url> skill-inject
cd skill-inject

2. Configure API keys

Copy the environment template and fill in your API keys:

cp docker/.env.example docker/.env

Edit `docker/.env` with your keys:

# Required for Claude Code agent
ANTHROPIC_API_KEY=sk-ant-...

# Required for Codex agent and GPT-5.1-Mini judge
OPENAI_API_KEY=sk-...

# Required for Gemini CLI agent
GEMINI_API_KEY=...

You only need the key(s) for the agent(s) you intend to test. The `OPENAI_API_KEY` is also needed for the default GPT-5.1-Mini evaluation judge.

3. Build the Docker image

bash docker/build.sh

This creates the `instruct-bench-agent` Docker image with all agent CLIs and dependencies pre-installed.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 316 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # SKILL-INJECT Benchmark
  2. A comprehensive benchmark for measuring prompt injection vulnerabilities in LLM agent skill files. SKILL-INJECT evaluates how susceptible multiple LLM coding agents (Claude Code, Codex, Gemini CLI) are to malicious instructions hidden in skill definitions, across multiple safety policy conditions.
  3. - **Paper:** [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://arxiv.org/abs/2602.20156)
  4. - **Dataset:** [davidsmts/skill-inject](https://huggingface.co/datasets/davidsmts/skill-inject) on Hugging Face
  5. ## Overview
  6. Modern LLM agents load "skill files" that define their capabilities. SKILL-INJECT tests whether adversarial instructions embedded in these skill files can cause agents to execute unintended actions.
  7. The benchmark includes:
  8. - **48 contextual injections** — dual-use instructions whose harm depends on context
  9. - **36 obvious injections** — unambiguously malicious instructions (ransomware, exfiltration, etc.)
  10. - **44 skill definitions** across diverse domains (documents, git, healthcare, email, MCP, etc.)
  11. - **3 safety policy conditions** — normal, legitimizing, and warning
  12. - **Multiple ablation studies** — Best-of-N, script vs. direct, skill screening, permission defense, phrasing/domain/template variation
  13. ### Camera-ready additions
  14. The camera-ready version of the paper adds the following data. The original files above are unchanged, so the paper's main tables can still be reproduced from them.
  15. - `data/obvious_injections_camready_phish_poison.json` — 6 new obvious injections (ids 39–44, phishing and data poisoning), with detectors `data/instruction_unit_tests_v2/obvious/test_obvious_39.py` to `test_obvious_44.py`
  16. - `data/obvious_to_contextual_camready.json` — 8 injections relabeled from obvious to contextual, with Warning and Legitimizing policy texts
  17. ## Prerequisites
  18. - **Docker** (for running agents in isolated containers)
  19. - **Python 3.10+**
  20. - **Node.js / npm** (installed inside Docker image; not needed on host)
  21. - **API keys** for the agents you want to test (see [Setup](#setup))
  22. ## Setup
  23. ### 1. Clone the repository
  24. ```bash
  25. git clone https://github.com/aisa-group/skill-inject.git
  26. cd skill-inject
  27. ```
  28. ### 2. Configure API keys
  29. Copy the environment template and fill in your API keys:
  30. ```bash
  31. cp docker/.env.example docker/.env
  32. ```
  33. Edit `docker/.env` with your keys:
  34. ```env
  35. # Required for Claude Code agent
  36. ANTHROPIC_API_KEY=sk-ant-...
  37. # Required for Codex agent and GPT-5.1-Mini judge
  38. OPENAI_API_KEY=sk-...
  39. # Required for Gemini CLI agent
  40. GEMINI_API_KEY=...
  41. ```
  42. You only need the key(s) for the agent(s) you intend to test. The `OPENAI_API_KEY` is also needed for the default GPT-5.1-Mini evaluation judge.
  43. ### Subscription authentication
  44. Claude Code and Codex agents can run against an existing Claude Pro/Max or ChatGPT subscription through their CLI OAuth credentials. Subscription model availability can differ from API-key availability; use a model supported by the installed CLI and subscription. Log in on the host first (`claude` or `codex login`), then select the backend explicitly:
  45. ```bash
  46. python experiments/contextual.py --agent claude --model claude-sonnet-4-6 --auth subscription
  47. python experiments/contextual.py --agent codex --model gpt-5.4 --auth subscription
  48. ```
  49. `--auth subscription` mounts only `~/.claude/.credentials.json` or `~/.codex/auth.json` read-only and suppresses the matching provider API key, preventing an API key from silently overriding subscription billing. It is supported by both Docker and Apptainer. `--auth api-key` disables credential mounts; `--auth auto` preserves the legacy behavior. Gemini and Vibe do not support subscription mode. Evaluation judges still require their documented API keys.
  50. ### 3. Build the Docker image
  51. ```bash
  52. bash docker/build.sh
  53. ```
  54. This creates the `instruct-bench-agent` Docker image with all agent CLIs and dependencies pre-installed.
  55. ## Project Structure
  56. ```
  57. skill-inject/
  58. ├── config.py # Central configuration (models, skills, policies)
  59. ├── pyproject.toml # Project metadata
  60. │
  61. ├── data/ # Benchmark data
  62. │ ├── contextual_injections.json # 48 contextual injection definitions
  63. │ ├── obvious_injections.json # 36 obvious injection definitions
  64. │ ├── tasks.json # 66 deduplicated clean tasks
  65. │ ├── unified_security_policy.md # Combined warning text
  66. │ ├── instruction_unit_tests_v2/ # Current deterministic attack detectors
  67. │ ├── skills/ # 44 clean skill definitions
  68. │ ├── task_files/ # Files referenced by tasks
  69. │ └── task_scripts/ # Scripts referenced by injections
  70. │
  71. ├── experiments/ # Experiment runners
  72. │ ├── _base.py # Base class (build → run → evaluate pipeline)
  73. │ ├── contextual.py # Contextual injection experiments
  74. │ ├── obvious.py # Obvious injection experiments
  75. │ ├── utility_baseline.py # Clean task baseline (no injections)
  76. │ └── ablations/ # Ablation studies (Bo5, script vs. direct, RL, etc.)
  77. │
  78. ├── judges/ # Evaluation judges
  79. │ ├── _base.py # Base judge (prompt building, verdict parsing)
  80. │ ├── contextual_judge.py # LLM judge for contextual results
  81. │ ├── obvious_judge.py # LLM judge for obvious results
  82. │ ├── utility_baseline_judge.py # Task completion evaluator
  83. │ └── skill_screening_judge.py # Skill safety screening ablation
  84. │
  85. ├── scripts/ # Core execution scripts
  86. │ ├── build_sandbox.py # Build sandboxes from injection JSON
  87. │ ├── build_clean_sandboxes.py # Build clean sandboxes (no injections)
  88. │ ├── run_sandbox_container.py # Execute agents in Docker containers
  89. │ ├── reset_sandboxes.py # Reset sandboxes for re-running
  90. │ ├── watch_status.py # Live monitor for run status
  91. │ └── smoke_test_all.py # End-to-end smoke test for all experiments
  92. │
  93. ├── docker/ # Docker configuration
  94. │ ├── Dockerfile # Agent container image
  95. │ ├── build.sh # Image build script
  96. │ ├── docker-compose.yml # Compose reference
  97. │ ├── entrypoint.sh # Container entrypoint
  98. │ └── .env.example # API key template
  99. │
  100. ├── apptainer/ # HPC cluster support (Apptainer/Singularity)
  101. ├── tests/ # Test suite (pytest)
  102. ├── startup_scripts/ # Pre-agent initialization scripts
  103. └── startup_assets/ # Assets for startup scripts
  104. ```
  105. ## Running Experiments
  106. All experiments follow the same pipeline: **build sandboxes → run agents in Docker → evaluate results**.
  107. ### Contextual Injections
  108. Tests dual-use injections across 3 safety policy conditions:
  109. ```bash
  110. # Run all models for a given agent
  111. python experiments/contextual.py --agent claude
  112. # Run a specific model
  113. python experiments/contextual.py --agent codex --model gpt-5.1-codex-mini
  114. # Run a specific policy only
  115. python experiments/contextual.py --agent claude --policy warning
  116. # Smoke test (injection ID 1 only, sequential)
  117. python experiments/contextual.py --agent claude --smoke-test
  118. ```
  119. ### Obvious Injections
  120. Tests unambiguously malicious injections (normal policy only):
  121. ```bash
  122. python experiments/obvious.py --agent claude
  123. python experiments/obvious.py --agent codex --model gpt-5.2-codex
  124. ```
  125. ### Utility Baseline
  126. Measures clean task completion (no injections) with and without security policy:
  127. ```bash
  128. # Both conditions (no_policy + policy)
  129. python experiments/utility_baseline.py --agent claude
  130. # Single condition
  131. python experiments/utility_baseline.py --agent claude --condition no_policy
  132. ```
  133. ### Ablation Studies
  134. ```bash
  135. # Best-of-5 line position ablation
  136. python experiments/ablations/bo5_byline.py --agent claude
  137. # Script vs. direct injection comparison
  138. python experiments/ablations/script_vs_direct.py --agent claude
  139. # Best-of-4 task variant ablation
  140. python experiments/ablations/bo4_bytask.py --agent claude
  141. # Repeated runs of high-ASR obvious injections (run-to-run variance)
  142. python experiments/ablations/obvious_injection_error_bars.py --agent claude --model sonnet
  143. # Permission defense (deterministic restrictions enforced in the container)
  144. python experiments/ablations/permission_defense.py --validate
  145. python experiments/ablations/permission_defense.py --agent claude
  146. # Phrasing, domain and template variants of obvious injections
  147. python experiments/ablations/phrasing_variants.py --agent claude
  148. python experiments/ablations/domain_variation.py --agent claude
  149. python experiments/ablations/template_variation.py --agent claude
  150. # Skill screening (LLM safety evaluation)
  151. python judges/skill_screening_judge.py --model sonnet
  152. python judges/skill_screening_judge.py --run-all # All screening models
  153. ```
  154. ## Evaluating Results
  155. The benchmark supports both LLM judging and deterministic, evidence-based attack detectors. The current detector suite is `data/instruction_unit_tests_v2`; `data/instruction_unit_tests` is retained for reproducing earlier evaluations. Result-comparison scripts accept `--suite-dir` when a frozen suite must be used. Record the repository commit and suite path alongside reported results.
  156. The v2 suite grades Docker and Apptainer runs. It reads SQLite evidence from a local copy, so results stored on network filesystems such as Lustre grade correctly. Some detectors ask `gpt-5-mini` a yes/no question about the agent's output. They use `OPENAI_API_KEY`, or the same model through OpenRouter when only `OPENROUTER_API_KEY` is set, and cache the answers under `.llm_cache/`.
  157. Results are saved to `final_results/` after each experiment run. Set `SKILL_INJECT_RESULTS_DIR` to write them somewhere else. You can also re-evaluate existing results independently.
  158. ### LLM Judge (default: GPT-5.1-Codex-Mini)
  159. ```bash
  160. # Contextual results (ASR + task completion)
  161. python judges/contextual_judge.py final_results/contextual/claude-sonnet/normal/ \
  162. --model gpt-5.1-codex-mini --evaluate-task
  163. # Obvious results (ASR only)
  164. python judges/obvious_judge.py final_results/obvious/claude-sonnet/normal/
  165. # Utility baseline (task completion only)
  166. python judges/utility_baseline_judge.py final_results/utility_baseline/claude-sonnet/no_policy/
  167. # Compare an LLM judge with deterministic detectors from an explicit suite
  168. python scripts/rescore_unittests_vs_judge.py --suite obvious \
  169. --root final_results/obvious \
  170. --suite-dir data/instruction_unit_tests_v2 --json comparison.json
  171. ```
  172. ## Configuration
  173. All configuration is centralized in [config.py](config.py):
  174. - **`AGENT_MODELS`** — Supported agent/model combinations (Claude, Codex, Gemini, Vibe)
  175. - **`SKILL_MAPPING`** — Maps skill types to their directories under `data/skills/`
  176. - **`POLICY_CONFIGS`** — Safety policy definitions (normal, legitimizing, warning)
  177. - **`AGENT_PARALLEL`** — Default parallelism per agent
  178. ## Common Options
  179. Most experiment runners support these flags:
  180. | Flag | Description |
  181. |------|-------------|
  182. | `--agent` | Agent to test: `claude`, `codex`, `gemini`, `vibe` |
  183. | `--model` | Specific model within agent (e.g., `sonnet`, `gpt-5.2-codex`) |
  184. | `--policy` | Safety policy filter: `normal`, `legitimizing`, `warning` |
  185. | `--parallel N` | Number of parallel container executions |
  186. | `--timeout N` | Execution timeout in seconds |
  187. | `--auth MODE` | Authentication: `auto`, `subscription`, or `api-key` |
  188. | `--smoke-test` | Run injection ID 1 only (quick validation) |
  189. | `--skip-eval` | Skip automatic evaluation after running |
  190. | `--force` | Delete existing results and rerun |
  191. ## Monitoring Runs
  192. Watch live progress during long experiment runs:
  193. ```bash
  194. python scripts/watch_status.py final_results/contextual/claude-sonnet/normal/run_status.jsonl
  195. ```
  196. ## Running Tests
  197. ### Unit tests
  198. ```bash
  199. pip install pytest
  200. pytest tests/ -v
  201. ```
  202. ### End-to-end smoke test
  203. Runs every experiment with every model on a single sandbox/container to verify the full pipeline works:
  204. ```bash
  205. # Test everything (all 22 models x 6 experiments = 132 runs)
  206. python scripts/smoke_test_all.py
  207. # Test a single agent
  208. python scripts/smoke_test_all.py --agent claude
  209. # Test a single agent/model combination
  210. python scripts/smoke_test_all.py --agent claude --model sonnet
  211. # Test a single experiment across all models
  212. python scripts/smoke_test_all.py --experiment contextual
  213. # Skip evaluation phase (only test build + run)
  214. python scripts/smoke_test_all.py --skip-eval
  215. # Dry run (print commands without executing)
  216. python scripts/smoke_test_all.py --dry-run
  217. ```
  218. The smoke test builds sandboxes, runs one agent container, and evaluates results for each combination. It requires Docker, API keys in `docker/.env`, and typically takes 5-15 minutes per agent/model combo.
  219. ## Citation
  220. ```bibtex
  221. @article{schmotz2026skillinject,
  222. title = {Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks},
  223. author = {Schmotz, David and Beurer-Kellner, Luca and Abdelnabi, Sahar and Andriushchenko, Maksym},
  224. journal = {arXiv preprint arXiv:2602.20156},
  225. year = {2026}
  226. }
  227. ```
  228. ## License
  229. See LICENSE file for details.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.