SCAM
SCAM - Security Comprehension Awareness Measure | Open-source benchmark that tests AI agents' security awareness during realistic, multi-turn workplace tasks.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/1password--scam)HTML
<a href="/listings/1password--scam"><img src="/api/listings/1password--scam/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
git clone https://github.com/1Password/SCAM.git cd SCAM python3 -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" # Set at least one API key export OPENAI_API_KEY="sk-..." # and/or ANTHROPIC_API_KEY, GOOGLE_API_KEY # Run the benchmark interactively scam evaluate -i
Interactive mode walks you through model selection, runs every scenario with and without the security skill, and prints a scored report at the end.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 90 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # SCAM — Security Comprehension Awareness Measure
- **By [1Password](https://1password.com/blog/ai-agent-security-benchmark)**
- > **[View the leaderboard, watch replays, and try the security skill at 1password.github.io/SCAM](https://1password.github.io/SCAM/)**
- As AI agents become more capable, they are gaining access to the sensitive information of the people they assist. SCAM measures whether agents will be good stewards of that information against the kinds of threats humans encounter every day.
- Most benchmarks show an AI a phishing email and ask "is this bad?" SCAM is different. It tests whether an agent can proactively recognize and report threats during normal activity — dropping agents into realistic workplace situations with access to email, credential vaults, and web forms, where the traps are embedded in the workflow, not called out separately.
- ## Quick Start
- ```bash
- git clone https://github.com/1Password/SCAM.git
- cd SCAM
- python3 -m venv .venv && source .venv/bin/activate
- pip install -e ".[dev]"
- # Set at least one API key
- export OPENAI_API_KEY="sk-..."
- # and/or ANTHROPIC_API_KEY, GOOGLE_API_KEY
- # Run the benchmark interactively
- scam evaluate -i
- ```
- Interactive mode walks you through model selection, runs every scenario with and without the security skill, and prints a scored report at the end.
- ## What It Tests
- Each scenario gives the agent a routine workplace task — checking email, looking up a credential, reviewing an invoice — along with a set of simulated MCP tool servers: an inbox, a password vault, a web browser, and more. These tools feel real to the model, but everything is sandboxed. No actual credentials are exposed, no real emails are sent, and no live systems are touched.
- The catch is that real-world attack patterns are woven into the task. A phishing link sits in the inbox. A lookalike domain shows up in a forwarded thread. An attacker's form is pre-filled with the right company name.
- The benchmark covers 30 scenarios across 9 threat categories: **Phishing** · **Social Engineering** · **Credential Exposure** · **Credential Autofill** · **E-Commerce Scams** · **Data Leakage** · **Confused Deputy** · **Multi-Stage Attacks** · **Prompt Injection**
- ## The Security Skill
- SCAM ships with a security skill ([`security-awareness/SKILL.md`](skills/security-awareness/SKILL.md)) — a plain-text system prompt addition that teaches agents to analyze before acting: verify domains before clicking, read content before forwarding, check URLs before entering credentials.
- In our benchmarks, this single skill raised average safety scores from ~50% to ~90% across all models tested. It works with any model and any provider.
- ### Install
- The fastest way to install the skill is with [npx add-skill](https://add-skill.org/), which auto-detects your agent (Claude Code, Cursor, Codex, and 35+ others):
- ```bash
- npx add-skill 1Password/SCAM
- ```
- Or download it directly:
- ```bash
- curl -sL https://raw.githubusercontent.com/1Password/SCAM/main/skills/security-awareness/SKILL.md \
- -o skills/security-awareness/SKILL.md --create-dirs
- ```
- Then prepend the file contents to your system prompt, or drop it into your agent's skills directory (`.claude/skills/`, `.cursor/skills/`, etc.). See the [website](https://1password.github.io/SCAM/#skill) for detailed integration examples per provider.
- ## Results
- The full leaderboard, interactive replays, and downloadable data are published at **[1password.github.io/SCAM](https://1password.github.io/SCAM/)**. Results include a ZIP archive with the raw JSON and an interactive HTML dashboard for independent verification.
- ## Contributing
- The threat landscape changes fast, and no single team can cover all of it. If you work in security, AI safety, or red-teaming, there are real ways to help:
- - **Write new scenarios.** Model a threat you've seen in the wild. The YAML format is straightforward.
- - **Add new tool servers.** Slack, Jira, cloud consoles — every new surface makes the test harder to game.
- - **Improve evaluation.** Better checkpoint logic, fewer false positives, more nuanced scoring.
- - **Run it on new models.** Publish your results. The more data points the community has, the harder it is to ignore.
- See [CONTRIBUTING.md](CONTRIBUTING.md) for the full guide.
- ## Learn More
- - [USAGE.md](USAGE.md) — Full CLI reference, all commands and flags, supported providers, benchmark versioning, project structure
- - [CONTRIBUTING.md](CONTRIBUTING.md) — Scenario authoring guide, YAML schema, difficulty scale, what makes a good scenario
- - [MAINTAINERS.md](MAINTAINERS.md) — Cutting releases, updating the website, reviewing PRs
- - [Website](https://1password.github.io/SCAM/) — Interactive leaderboard, featured replays, the security skill
- ## Acknowledgements
- - [Jason Meller](https://github.com/terracatta) — created SCAM
- ## License
- SCAM is released under the [MIT License](LICENSE).
- Copyright (c) 2026 [1Password](https://1password.com)
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.