Skip to content
Back to search

promptfoo

MarketplaceWorks with Claude CodeGitHub

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/promptfoo/promptfoo with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/promptfoo/promptfoo'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/promptfoo-promptfoo/badge.svg)](/listings/promptfoo-promptfoo)

HTML

<a href="/listings/promptfoo-promptfoo"><img src="/api/listings/promptfoo-promptfoo/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

npm install -g promptfoo
promptfoo init --example getting-started

Also available via `brew install promptfoo` and `pip install promptfoo`. Or use `npx promptfoo@latest` to run any command without installing.

Most LLM providers require an API key. Set yours as an environment variable:

export OPENAI_API_KEY=sk-abc123

Run an eval and view results:

cd getting-started
promptfoo eval
promptfoo view

> **See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more!**

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 21 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "promptfoo",
  3. "owner": {
  4. "name": "promptfoo"
  5. },
  6. "metadata": {
  7. "description": "Agent skills for connecting, evaluating, and red teaming LLM applications with promptfoo"
  8. },
  9. "plugins": [
  10. {
  11. "name": "promptfoo",
  12. "source": "./plugins/promptfoo",
  13. "description": "Connect providers and targets, write eval suites, and set up or run red team scans with promptfoo",
  14. "homepage": "https://www.promptfoo.dev/docs/integrations/agent-skill",
  15. "repository": "https://github.com/promptfoo/promptfoo",
  16. "license": "MIT",
  17. "keywords": ["eval", "redteam", "provider", "testing", "llm", "promptfoo"],
  18. "category": "productivity"
  19. }
  20. ]
  21. }
README.md · README · 98 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # Promptfoo: LLM evals & red teaming
  2. <p align="center">
  3. <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/v/promptfoo" alt="npm"></a>
  4. <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/dm/promptfoo" alt="npm"></a>
  5. <a href="https://github.com/promptfoo/promptfoo/actions/workflows/main.yml"><img src="https://img.shields.io/github/actions/workflow/status/promptfoo/promptfoo/main.yml" alt="GitHub Workflow Status"></a>
  6. <a href="https://github.com/promptfoo/promptfoo/blob/main/LICENSE"><img src="https://img.shields.io/github/license/promptfoo/promptfoo" alt="MIT license"></a>
  7. <a href="https://discord.gg/gHPS9jjfbs"><img src="https://img.shields.io/discord/1146610656779440188?logo=discord&label=promptfoo" alt="Discord"></a>
  8. </p>
  9. <p align="center">
  10. <code>promptfoo</code> is a CLI and library for evaluating and red-teaming LLM-based apps. Stop using trial-and-error... start shipping secure, reliable agents
  11. </p>
  12. <p align="center">
  13. <a href="https://www.promptfoo.dev">Website</a> ·
  14. <a href="https://www.promptfoo.dev/docs/getting-started/">Getting Started</a> ·
  15. <a href="https://www.promptfoo.dev/docs/red-team/">Red Teaming</a> ·
  16. <a href="https://www.promptfoo.dev/docs/">Documentation</a> ·
  17. <a href="https://discord.gg/gHPS9jjfbs">Discord</a>
  18. </p>
  19. > Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the [company update](https://www.promptfoo.dev/blog/promptfoo-joining-openai/).
  20. ## Quick Start
  21. ```sh
  22. npm install -g promptfoo
  23. promptfoo init --example getting-started
  24. ```
  25. Also available via `brew install promptfoo` and `pip install promptfoo`. Or use `npx promptfoo@latest` to run any command without installing.
  26. Most LLM providers require an API key. Set yours as an environment variable:
  27. ```sh
  28. export OPENAI_API_KEY=sk-abc123
  29. ```
  30. Run an eval and view results:
  31. ```sh
  32. cd getting-started
  33. promptfoo eval
  34. promptfoo view
  35. ```
  36. > **See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more!**
  37. ## What can you do with Promptfoo?
  38. - **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)
  39. - **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning
  40. - **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))
  41. - **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)
  42. - **Review pull requests** for LLM-related security and compliance issues with [code scanning](https://www.promptfoo.dev/docs/code-scanning/)
  43. - **Share results** with your team
  44. Here's what it looks like in action:
  45. <img src="site/static/img/claude-vs-gpt-example@2x.png" alt="prompt evaluation matrix - web viewer" width="700">
  46. It works on the command line too:
  47. <img src="https://www.promptfoo.dev/img/docs/self-grading.gif" alt="promptfoo command line" width="700">
  48. It also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):
  49. <img src="https://www.promptfoo.dev/img/redteam-vulnerability-report-hero@2x.png" alt="gen ai red team" width="700">
  50. ## Why Promptfoo?
  51. - **Developer-first**: Fast, with features like live reload and caching
  52. - **Private**: LLM evals run 100% locally - your prompts never leave your machine
  53. - **Flexible**: Works with any LLM API or programming language
  54. - **Battle-tested**: Powers LLM apps serving 10M+ users in production
  55. - **Data-driven**: Make decisions based on metrics, not gut feel
  56. - **Open source**: MIT licensed, with an active community
  57. ## Learn More
  58. - [Getting Started](https://www.promptfoo.dev/docs/getting-started/)
  59. - [Full Documentation](https://www.promptfoo.dev/docs/intro/)
  60. - [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)
  61. - [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)
  62. - [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)
  63. - [Supported Models](https://www.promptfoo.dev/docs/providers/)
  64. - [Code Scanning Guide](https://www.promptfoo.dev/docs/code-scanning/)
  65. ## Contributing
  66. We welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started.
  67. Join our [Discord community](https://discord.gg/gHPS9jjfbs) for help and discussion.
  68. <a href="https://github.com/promptfoo/promptfoo/graphs/contributors">
  69. <img src="https://contrib.rocks/image?repo=promptfoo/promptfoo" />
  70. </a>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.