skill-inject
Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/aisa-group--skill-inject)HTML
<a href="/listings/aisa-group--skill-inject"><img src="/api/listings/aisa-group--skill-inject/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
1. Clone the repository
git clone <repo-url> skill-inject cd skill-inject
2. Configure API keys
Copy the environment template and fill in your API keys:
cp docker/.env.example docker/.env
Edit `docker/.env` with your keys:
# Required for Claude Code agent ANTHROPIC_API_KEY=sk-ant-... # Required for Codex agent and GPT-5.1-Mini judge OPENAI_API_KEY=sk-... # Required for Gemini CLI agent GEMINI_API_KEY=...
You only need the key(s) for the agent(s) you intend to test. The `OPENAI_API_KEY` is also needed for the default GPT-5.1-Mini evaluation judge.
3. Build the Docker image
bash docker/build.sh
This creates the `instruct-bench-agent` Docker image with all agent CLIs and dependencies pre-installed.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 316 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # SKILL-INJECT Benchmark
- A comprehensive benchmark for measuring prompt injection vulnerabilities in LLM agent skill files. SKILL-INJECT evaluates how susceptible multiple LLM coding agents (Claude Code, Codex, Gemini CLI) are to malicious instructions hidden in skill definitions, across multiple safety policy conditions.
- - **Paper:** [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://arxiv.org/abs/2602.20156)
- - **Dataset:** [davidsmts/skill-inject](https://huggingface.co/datasets/davidsmts/skill-inject) on Hugging Face
- ## Overview
- Modern LLM agents load "skill files" that define their capabilities. SKILL-INJECT tests whether adversarial instructions embedded in these skill files can cause agents to execute unintended actions.
- The benchmark includes:
- - **48 contextual injections** — dual-use instructions whose harm depends on context
- - **36 obvious injections** — unambiguously malicious instructions (ransomware, exfiltration, etc.)
- - **44 skill definitions** across diverse domains (documents, git, healthcare, email, MCP, etc.)
- - **3 safety policy conditions** — normal, legitimizing, and warning
- - **Multiple ablation studies** — Best-of-N, script vs. direct, skill screening, permission defense, phrasing/domain/template variation
- ### Camera-ready additions
- The camera-ready version of the paper adds the following data. The original files above are unchanged, so the paper's main tables can still be reproduced from them.
- - `data/obvious_injections_camready_phish_poison.json` — 6 new obvious injections (ids 39–44, phishing and data poisoning), with detectors `data/instruction_unit_tests_v2/obvious/test_obvious_39.py` to `test_obvious_44.py`
- - `data/obvious_to_contextual_camready.json` — 8 injections relabeled from obvious to contextual, with Warning and Legitimizing policy texts
- ## Prerequisites
- - **Docker** (for running agents in isolated containers)
- - **Python 3.10+**
- - **Node.js / npm** (installed inside Docker image; not needed on host)
- - **API keys** for the agents you want to test (see [Setup](#setup))
- ## Setup
- ### 1. Clone the repository
- ```bash
- git clone https://github.com/aisa-group/skill-inject.git
- cd skill-inject
- ```
- ### 2. Configure API keys
- Copy the environment template and fill in your API keys:
- ```bash
- cp docker/.env.example docker/.env
- ```
- Edit `docker/.env` with your keys:
- ```env
- # Required for Claude Code agent
- ANTHROPIC_API_KEY=sk-ant-...
- # Required for Codex agent and GPT-5.1-Mini judge
- OPENAI_API_KEY=sk-...
- # Required for Gemini CLI agent
- GEMINI_API_KEY=...
- ```
- You only need the key(s) for the agent(s) you intend to test. The `OPENAI_API_KEY` is also needed for the default GPT-5.1-Mini evaluation judge.
- ### Subscription authentication
- Claude Code and Codex agents can run against an existing Claude Pro/Max or ChatGPT subscription through their CLI OAuth credentials. Subscription model availability can differ from API-key availability; use a model supported by the installed CLI and subscription. Log in on the host first (`claude` or `codex login`), then select the backend explicitly:
- ```bash
- python experiments/contextual.py --agent claude --model claude-sonnet-4-6 --auth subscription
- python experiments/contextual.py --agent codex --model gpt-5.4 --auth subscription
- ```
- `--auth subscription` mounts only `~/.claude/.credentials.json` or `~/.codex/auth.json` read-only and suppresses the matching provider API key, preventing an API key from silently overriding subscription billing. It is supported by both Docker and Apptainer. `--auth api-key` disables credential mounts; `--auth auto` preserves the legacy behavior. Gemini and Vibe do not support subscription mode. Evaluation judges still require their documented API keys.
- ### 3. Build the Docker image
- ```bash
- bash docker/build.sh
- ```
- This creates the `instruct-bench-agent` Docker image with all agent CLIs and dependencies pre-installed.
- ## Project Structure
- ```
- skill-inject/
- ├── config.py # Central configuration (models, skills, policies)
- ├── pyproject.toml # Project metadata
- │
- ├── data/ # Benchmark data
- │ ├── contextual_injections.json # 48 contextual injection definitions
- │ ├── obvious_injections.json # 36 obvious injection definitions
- │ ├── tasks.json # 66 deduplicated clean tasks
- │ ├── unified_security_policy.md # Combined warning text
- │ ├── instruction_unit_tests_v2/ # Current deterministic attack detectors
- │ ├── skills/ # 44 clean skill definitions
- │ ├── task_files/ # Files referenced by tasks
- │ └── task_scripts/ # Scripts referenced by injections
- │
- ├── experiments/ # Experiment runners
- │ ├── _base.py # Base class (build → run → evaluate pipeline)
- │ ├── contextual.py # Contextual injection experiments
- │ ├── obvious.py # Obvious injection experiments
- │ ├── utility_baseline.py # Clean task baseline (no injections)
- │ └── ablations/ # Ablation studies (Bo5, script vs. direct, RL, etc.)
- │
- ├── judges/ # Evaluation judges
- │ ├── _base.py # Base judge (prompt building, verdict parsing)
- │ ├── contextual_judge.py # LLM judge for contextual results
- │ ├── obvious_judge.py # LLM judge for obvious results
- │ ├── utility_baseline_judge.py # Task completion evaluator
- │ └── skill_screening_judge.py # Skill safety screening ablation
- │
- ├── scripts/ # Core execution scripts
- │ ├── build_sandbox.py # Build sandboxes from injection JSON
- │ ├── build_clean_sandboxes.py # Build clean sandboxes (no injections)
- │ ├── run_sandbox_container.py # Execute agents in Docker containers
- │ ├── reset_sandboxes.py # Reset sandboxes for re-running
- │ ├── watch_status.py # Live monitor for run status
- │ └── smoke_test_all.py # End-to-end smoke test for all experiments
- │
- ├── docker/ # Docker configuration
- │ ├── Dockerfile # Agent container image
- │ ├── build.sh # Image build script
- │ ├── docker-compose.yml # Compose reference
- │ ├── entrypoint.sh # Container entrypoint
- │ └── .env.example # API key template
- │
- ├── apptainer/ # HPC cluster support (Apptainer/Singularity)
- ├── tests/ # Test suite (pytest)
- ├── startup_scripts/ # Pre-agent initialization scripts
- └── startup_assets/ # Assets for startup scripts
- ```
- ## Running Experiments
- All experiments follow the same pipeline: **build sandboxes → run agents in Docker → evaluate results**.
- ### Contextual Injections
- Tests dual-use injections across 3 safety policy conditions:
- ```bash
- # Run all models for a given agent
- python experiments/contextual.py --agent claude
- # Run a specific model
- python experiments/contextual.py --agent codex --model gpt-5.1-codex-mini
- # Run a specific policy only
- python experiments/contextual.py --agent claude --policy warning
- # Smoke test (injection ID 1 only, sequential)
- python experiments/contextual.py --agent claude --smoke-test
- ```
- ### Obvious Injections
- Tests unambiguously malicious injections (normal policy only):
- ```bash
- python experiments/obvious.py --agent claude
- python experiments/obvious.py --agent codex --model gpt-5.2-codex
- ```
- ### Utility Baseline
- Measures clean task completion (no injections) with and without security policy:
- ```bash
- # Both conditions (no_policy + policy)
- python experiments/utility_baseline.py --agent claude
- # Single condition
- python experiments/utility_baseline.py --agent claude --condition no_policy
- ```
- ### Ablation Studies
- ```bash
- # Best-of-5 line position ablation
- python experiments/ablations/bo5_byline.py --agent claude
- # Script vs. direct injection comparison
- python experiments/ablations/script_vs_direct.py --agent claude
- # Best-of-4 task variant ablation
- python experiments/ablations/bo4_bytask.py --agent claude
- # Repeated runs of high-ASR obvious injections (run-to-run variance)
- python experiments/ablations/obvious_injection_error_bars.py --agent claude --model sonnet
- # Permission defense (deterministic restrictions enforced in the container)
- python experiments/ablations/permission_defense.py --validate
- python experiments/ablations/permission_defense.py --agent claude
- # Phrasing, domain and template variants of obvious injections
- python experiments/ablations/phrasing_variants.py --agent claude
- python experiments/ablations/domain_variation.py --agent claude
- python experiments/ablations/template_variation.py --agent claude
- # Skill screening (LLM safety evaluation)
- python judges/skill_screening_judge.py --model sonnet
- python judges/skill_screening_judge.py --run-all # All screening models
- ```
- ## Evaluating Results
- The benchmark supports both LLM judging and deterministic, evidence-based attack detectors. The current detector suite is `data/instruction_unit_tests_v2`; `data/instruction_unit_tests` is retained for reproducing earlier evaluations. Result-comparison scripts accept `--suite-dir` when a frozen suite must be used. Record the repository commit and suite path alongside reported results.
- The v2 suite grades Docker and Apptainer runs. It reads SQLite evidence from a local copy, so results stored on network filesystems such as Lustre grade correctly. Some detectors ask `gpt-5-mini` a yes/no question about the agent's output. They use `OPENAI_API_KEY`, or the same model through OpenRouter when only `OPENROUTER_API_KEY` is set, and cache the answers under `.llm_cache/`.
- Results are saved to `final_results/` after each experiment run. Set `SKILL_INJECT_RESULTS_DIR` to write them somewhere else. You can also re-evaluate existing results independently.
- ### LLM Judge (default: GPT-5.1-Codex-Mini)
- ```bash
- # Contextual results (ASR + task completion)
- python judges/contextual_judge.py final_results/contextual/claude-sonnet/normal/ \
- --model gpt-5.1-codex-mini --evaluate-task
- # Obvious results (ASR only)
- python judges/obvious_judge.py final_results/obvious/claude-sonnet/normal/
- # Utility baseline (task completion only)
- python judges/utility_baseline_judge.py final_results/utility_baseline/claude-sonnet/no_policy/
- # Compare an LLM judge with deterministic detectors from an explicit suite
- python scripts/rescore_unittests_vs_judge.py --suite obvious \
- --root final_results/obvious \
- --suite-dir data/instruction_unit_tests_v2 --json comparison.json
- ```
- ## Configuration
- All configuration is centralized in [config.py](config.py):
- - **`AGENT_MODELS`** — Supported agent/model combinations (Claude, Codex, Gemini, Vibe)
- - **`SKILL_MAPPING`** — Maps skill types to their directories under `data/skills/`
- - **`POLICY_CONFIGS`** — Safety policy definitions (normal, legitimizing, warning)
- - **`AGENT_PARALLEL`** — Default parallelism per agent
- ## Common Options
- Most experiment runners support these flags:
- | Flag | Description |
- |------|-------------|
- | `--agent` | Agent to test: `claude`, `codex`, `gemini`, `vibe` |
- | `--model` | Specific model within agent (e.g., `sonnet`, `gpt-5.2-codex`) |
- | `--policy` | Safety policy filter: `normal`, `legitimizing`, `warning` |
- | `--parallel N` | Number of parallel container executions |
- | `--timeout N` | Execution timeout in seconds |
- | `--auth MODE` | Authentication: `auto`, `subscription`, or `api-key` |
- | `--smoke-test` | Run injection ID 1 only (quick validation) |
- | `--skip-eval` | Skip automatic evaluation after running |
- | `--force` | Delete existing results and rerun |
- ## Monitoring Runs
- Watch live progress during long experiment runs:
- ```bash
- python scripts/watch_status.py final_results/contextual/claude-sonnet/normal/run_status.jsonl
- ```
- ## Running Tests
- ### Unit tests
- ```bash
- pip install pytest
- pytest tests/ -v
- ```
- ### End-to-end smoke test
- Runs every experiment with every model on a single sandbox/container to verify the full pipeline works:
- ```bash
- # Test everything (all 22 models x 6 experiments = 132 runs)
- python scripts/smoke_test_all.py
- # Test a single agent
- python scripts/smoke_test_all.py --agent claude
- # Test a single agent/model combination
- python scripts/smoke_test_all.py --agent claude --model sonnet
- # Test a single experiment across all models
- python scripts/smoke_test_all.py --experiment contextual
- # Skip evaluation phase (only test build + run)
- python scripts/smoke_test_all.py --skip-eval
- # Dry run (print commands without executing)
- python scripts/smoke_test_all.py --dry-run
- ```
- The smoke test builds sandboxes, runs one agent container, and evaluates results for each combination. It requires Docker, API keys in `docker/.env`, and typically takes 5-15 minutes per agent/model combo.
- ## Citation
- ```bibtex
- @article{schmotz2026skillinject,
- title = {Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks},
- author = {Schmotz, David and Beurer-Kellner, Luca and Abdelnabi, Sahar and Andriushchenko, Maksym},
- journal = {arXiv preprint arXiv:2602.20156},
- year = {2026}
- }
- ```
- ## License
- See LICENSE file for details.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.