Skip to content
Back to search

SCAM

SCAM - Security Comprehension Awareness Measure | Open-source benchmark that tests AI agents' security awareness during realistic, multi-turn workplace tasks.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/1password--scam/badge.svg)](/listings/1password--scam)

HTML

<a href="/listings/1password--scam"><img src="/api/listings/1password--scam/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/1Password/SCAM.git
cd SCAM
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Set at least one API key
export OPENAI_API_KEY="sk-..."       
# and/or ANTHROPIC_API_KEY, GOOGLE_API_KEY

# Run the benchmark interactively
scam evaluate -i

Interactive mode walks you through model selection, runs every scenario with and without the security skill, and prints a scored report at the end.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 90 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # SCAM — Security Comprehension Awareness Measure
  2. **By [1Password](https://1password.com/blog/ai-agent-security-benchmark)**
  3. > **[View the leaderboard, watch replays, and try the security skill at 1password.github.io/SCAM](https://1password.github.io/SCAM/)**
  4. As AI agents become more capable, they are gaining access to the sensitive information of the people they assist. SCAM measures whether agents will be good stewards of that information against the kinds of threats humans encounter every day.
  5. Most benchmarks show an AI a phishing email and ask "is this bad?" SCAM is different. It tests whether an agent can proactively recognize and report threats during normal activity — dropping agents into realistic workplace situations with access to email, credential vaults, and web forms, where the traps are embedded in the workflow, not called out separately.
  6. ## Quick Start
  7. ```bash
  8. git clone https://github.com/1Password/SCAM.git
  9. cd SCAM
  10. python3 -m venv .venv && source .venv/bin/activate
  11. pip install -e ".[dev]"
  12. # Set at least one API key
  13. export OPENAI_API_KEY="sk-..."
  14. # and/or ANTHROPIC_API_KEY, GOOGLE_API_KEY
  15. # Run the benchmark interactively
  16. scam evaluate -i
  17. ```
  18. Interactive mode walks you through model selection, runs every scenario with and without the security skill, and prints a scored report at the end.
  19. ## What It Tests
  20. Each scenario gives the agent a routine workplace task — checking email, looking up a credential, reviewing an invoice — along with a set of simulated MCP tool servers: an inbox, a password vault, a web browser, and more. These tools feel real to the model, but everything is sandboxed. No actual credentials are exposed, no real emails are sent, and no live systems are touched.
  21. The catch is that real-world attack patterns are woven into the task. A phishing link sits in the inbox. A lookalike domain shows up in a forwarded thread. An attacker's form is pre-filled with the right company name.
  22. The benchmark covers 30 scenarios across 9 threat categories: **Phishing** · **Social Engineering** · **Credential Exposure** · **Credential Autofill** · **E-Commerce Scams** · **Data Leakage** · **Confused Deputy** · **Multi-Stage Attacks** · **Prompt Injection**
  23. ## The Security Skill
  24. SCAM ships with a security skill ([`security-awareness/SKILL.md`](skills/security-awareness/SKILL.md)) — a plain-text system prompt addition that teaches agents to analyze before acting: verify domains before clicking, read content before forwarding, check URLs before entering credentials.
  25. In our benchmarks, this single skill raised average safety scores from ~50% to ~90% across all models tested. It works with any model and any provider.
  26. ### Install
  27. The fastest way to install the skill is with [npx add-skill](https://add-skill.org/), which auto-detects your agent (Claude Code, Cursor, Codex, and 35+ others):
  28. ```bash
  29. npx add-skill 1Password/SCAM
  30. ```
  31. Or download it directly:
  32. ```bash
  33. curl -sL https://raw.githubusercontent.com/1Password/SCAM/main/skills/security-awareness/SKILL.md \
  34. -o skills/security-awareness/SKILL.md --create-dirs
  35. ```
  36. Then prepend the file contents to your system prompt, or drop it into your agent's skills directory (`.claude/skills/`, `.cursor/skills/`, etc.). See the [website](https://1password.github.io/SCAM/#skill) for detailed integration examples per provider.
  37. ## Results
  38. The full leaderboard, interactive replays, and downloadable data are published at **[1password.github.io/SCAM](https://1password.github.io/SCAM/)**. Results include a ZIP archive with the raw JSON and an interactive HTML dashboard for independent verification.
  39. ## Contributing
  40. The threat landscape changes fast, and no single team can cover all of it. If you work in security, AI safety, or red-teaming, there are real ways to help:
  41. - **Write new scenarios.** Model a threat you've seen in the wild. The YAML format is straightforward.
  42. - **Add new tool servers.** Slack, Jira, cloud consoles — every new surface makes the test harder to game.
  43. - **Improve evaluation.** Better checkpoint logic, fewer false positives, more nuanced scoring.
  44. - **Run it on new models.** Publish your results. The more data points the community has, the harder it is to ignore.
  45. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full guide.
  46. ## Learn More
  47. - [USAGE.md](USAGE.md) — Full CLI reference, all commands and flags, supported providers, benchmark versioning, project structure
  48. - [CONTRIBUTING.md](CONTRIBUTING.md) — Scenario authoring guide, YAML schema, difficulty scale, what makes a good scenario
  49. - [MAINTAINERS.md](MAINTAINERS.md) — Cutting releases, updating the website, reviewing PRs
  50. - [Website](https://1password.github.io/SCAM/) — Interactive leaderboard, featured replays, the security skill
  51. ## Acknowledgements
  52. - [Jason Meller](https://github.com/terracatta) — created SCAM
  53. ## License
  54. SCAM is released under the [MIT License](LICENSE).
  55. Copyright (c) 2026 [1Password](https://1password.com)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.