promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/promptfoo/promptfoo with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/promptfoo/promptfoo'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/promptfoo-promptfoo)HTML
<a href="/listings/promptfoo-promptfoo"><img src="/api/listings/promptfoo-promptfoo/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
npm install -g promptfoo promptfoo init --example getting-started
Also available via `brew install promptfoo` and `pip install promptfoo`. Or use `npx promptfoo@latest` to run any command without installing.
Most LLM providers require an API key. Set yours as an environment variable:
export OPENAI_API_KEY=sk-abc123
Run an eval and view results:
cd getting-started promptfoo eval promptfoo view
> **See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more!**
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 21 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "name": "promptfoo",
- "owner": {
- "name": "promptfoo"
- },
- "metadata": {
- "description": "Agent skills for connecting, evaluating, and red teaming LLM applications with promptfoo"
- },
- "plugins": [
- {
- "name": "promptfoo",
- "source": "./plugins/promptfoo",
- "description": "Connect providers and targets, write eval suites, and set up or run red team scans with promptfoo",
- "homepage": "https://www.promptfoo.dev/docs/integrations/agent-skill",
- "repository": "https://github.com/promptfoo/promptfoo",
- "license": "MIT",
- "keywords": ["eval", "redteam", "provider", "testing", "llm", "promptfoo"],
- "category": "productivity"
- }
- ]
- }
README.md · README · 98 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- # Promptfoo: LLM evals & red teaming
- <p align="center">
- <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/v/promptfoo" alt="npm"></a>
- <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/dm/promptfoo" alt="npm"></a>
- <a href="https://github.com/promptfoo/promptfoo/actions/workflows/main.yml"><img src="https://img.shields.io/github/actions/workflow/status/promptfoo/promptfoo/main.yml" alt="GitHub Workflow Status"></a>
- <a href="https://github.com/promptfoo/promptfoo/blob/main/LICENSE"><img src="https://img.shields.io/github/license/promptfoo/promptfoo" alt="MIT license"></a>
- <a href="https://discord.gg/gHPS9jjfbs"><img src="https://img.shields.io/discord/1146610656779440188?logo=discord&label=promptfoo" alt="Discord"></a>
- </p>
- <p align="center">
- <code>promptfoo</code> is a CLI and library for evaluating and red-teaming LLM-based apps. Stop using trial-and-error... start shipping secure, reliable agents
- </p>
- <p align="center">
- <a href="https://www.promptfoo.dev">Website</a> ·
- <a href="https://www.promptfoo.dev/docs/getting-started/">Getting Started</a> ·
- <a href="https://www.promptfoo.dev/docs/red-team/">Red Teaming</a> ·
- <a href="https://www.promptfoo.dev/docs/">Documentation</a> ·
- <a href="https://discord.gg/gHPS9jjfbs">Discord</a>
- </p>
- > Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the [company update](https://www.promptfoo.dev/blog/promptfoo-joining-openai/).
- ## Quick Start
- ```sh
- npm install -g promptfoo
- promptfoo init --example getting-started
- ```
- Also available via `brew install promptfoo` and `pip install promptfoo`. Or use `npx promptfoo@latest` to run any command without installing.
- Most LLM providers require an API key. Set yours as an environment variable:
- ```sh
- export OPENAI_API_KEY=sk-abc123
- ```
- Run an eval and view results:
- ```sh
- cd getting-started
- promptfoo eval
- promptfoo view
- ```
- > **See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more!**
- ## What can you do with Promptfoo?
- - **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)
- - **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning
- - **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))
- - **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)
- - **Review pull requests** for LLM-related security and compliance issues with [code scanning](https://www.promptfoo.dev/docs/code-scanning/)
- - **Share results** with your team
- Here's what it looks like in action:
- <img src="site/static/img/claude-vs-gpt-example@2x.png" alt="prompt evaluation matrix - web viewer" width="700">
- It works on the command line too:
- <img src="https://www.promptfoo.dev/img/docs/self-grading.gif" alt="promptfoo command line" width="700">
- It also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):
- <img src="https://www.promptfoo.dev/img/redteam-vulnerability-report-hero@2x.png" alt="gen ai red team" width="700">
- ## Why Promptfoo?
- - **Developer-first**: Fast, with features like live reload and caching
- - **Private**: LLM evals run 100% locally - your prompts never leave your machine
- - **Flexible**: Works with any LLM API or programming language
- - **Battle-tested**: Powers LLM apps serving 10M+ users in production
- - **Data-driven**: Make decisions based on metrics, not gut feel
- - **Open source**: MIT licensed, with an active community
- ## Learn More
- - [Getting Started](https://www.promptfoo.dev/docs/getting-started/)
- - [Full Documentation](https://www.promptfoo.dev/docs/intro/)
- - [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)
- - [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)
- - [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)
- - [Supported Models](https://www.promptfoo.dev/docs/providers/)
- - [Code Scanning Guide](https://www.promptfoo.dev/docs/code-scanning/)
- ## Contributing
- We welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.
- Join our [Discord community](https://discord.gg/gHPS9jjfbs) for help and discussion.
- <a href="https://github.com/promptfoo/promptfoo/graphs/contributors">
- <img src="https://contrib.rocks/image?repo=promptfoo/promptfoo" />
- </a>
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.