Skip to content
Back to search

deepeval-plugins

MarketplaceWorks with Claude CodeGitHub

DeepEval plugins for LLM evaluation, tracing, and testing in Claude Code.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/confident-ai/deepeval with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/confident-ai/deepeval'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/confident-ai-deepeval/badge.svg)](/listings/confident-ai-deepeval)

HTML

<a href="/listings/confident-ai-deepeval"><img src="/api/listings/confident-ai-deepeval/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your agent, RAG pipeline, or chatbot, and ask it to generate a dataset, write the eval suite, run `deepeval test run`, and iterate on the failing metrics.

[Start with the 5-minute vibe-coder guide](https://deepeval.com/docs/vibe-coder-quickstart?utm_source=GitHub).

<br />

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 24 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "deepeval-plugins",
  3. "owner": {
  4. "name": "Confident AI",
  5. "email": "open-source@confident-ai.com"
  6. },
  7. "description": "DeepEval plugins for LLM evaluation, tracing, and testing in Claude Code.",
  8. "plugins": [
  9. {
  10. "name": "deepeval",
  11. "source": "./",
  12. "description": "Skills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications.",
  13. "category": "developer-tools",
  14. "keywords": [
  15. "deepeval",
  16. "llm",
  17. "evaluation",
  18. "tracing",
  19. "datasets",
  20. "confident-ai"
  21. ]
  22. }
  23. ]
  24. }
README.md · README · 693 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <picture>
  3. <source media="(prefers-color-scheme: dark)" srcset="assets/hero/wordmark-dark.svg">
  4. <img alt="DeepEval." src="assets/hero/wordmark-light.svg" width="520">
  5. </picture>
  6. </p>
  7. <p align="center">
  8. <h1 align="center">The LLM Evaluation Framework</h1>
  9. </p>
  10. <p align="center">
  11. <a href="https://trendshift.io/repositories/5917" target="_blank"><img src="https://trendshift.io/api/badge/repositories/5917" alt="confident-ai%2Fdeepeval | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
  12. </p>
  13. <p align="center">
  14. <a href="https://discord.gg/3SEyvpgu2f">
  15. <img alt="discord-invite" src="https://dcbadge.limes.pink/api/server/3SEyvpgu2f?style=flat">
  16. </a>
  17. <a href="https://www.reddit.com/r/deepeval/">
  18. <img alt="reddit-community" src="https://img.shields.io/badge/Reddit-r%2Fdeepeval-FF4500?logo=reddit&logoColor=white">
  19. </a>
  20. </p>
  21. <h4 align="center">
  22. <p>
  23. <a href="https://deepeval.com/docs/getting-started?utm_source=GitHub">Documentation</a> |
  24. <a href="#-metrics-and-features">Metrics and Features</a> |
  25. <a href="#-quickstart">Getting Started</a> |
  26. <a href="#-integrations">Integrations</a> |
  27. <a href="https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=header_nav&ref_page=github/readme">Confident AI</a>
  28. <p>
  29. </h4>
  30. <p align="center">
  31. <a href="https://github.com/confident-ai/deepeval/releases">
  32. <img alt="GitHub release" src="https://img.shields.io/github/release/confident-ai/deepeval.svg?color=violet">
  33. </a>
  34. <a href="https://colab.research.google.com/drive/1PPxYEBa6eu__LquGoFFJZkhYgWVYE6kh?usp=sharing">
  35. <img alt="Try Quickstart in Colab" src="https://colab.research.google.com/assets/colab-badge.svg">
  36. </a>
  37. <a href="https://github.com/confident-ai/deepeval/blob/master/LICENSE.md">
  38. <img alt="License" src="https://img.shields.io/github/license/confident-ai/deepeval.svg?color=yellow">
  39. </a>
  40. <a href="https://x.com/deepeval">
  41. <img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/deepeval?style=social&logo=x">
  42. </a>
  43. </p>
  44. <p align="center">
  45. <!-- Keep these links. Translations will automatically update with the README. -->
  46. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=de">Deutsch</a> |
  47. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=es">Español</a> |
  48. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=fr">français</a> |
  49. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ja">日本語</a> |
  50. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ko">한국어</a> |
  51. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=pt">Português</a> |
  52. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=ru">Русский</a> |
  53. <a href="https://www.readme-i18n.com/confident-ai/deepeval?lang=zh">中文</a>
  54. </p>
  55. **DeepEval** is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which use LLM-as-a-judge and other NLP models that run **locally on your machine**.
  56. Whether you're building AI agents, voice agents, multi-turn chatbots, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
  57. - **LLM apps end-to-end** as black boxes
  58. - **Complete AI agent trajectories** across every decision and action
  59. - **Individual AI agent steps** such as LLM calls, tool use, retrieval, and sub-agent handoffs
  60. Use these evaluations to determine the optimal models, prompts, and architecture to improve your AI quality, prevent prompt drifting, or even transition from OpenAI to Claude with confidence.
  61. > Jev is now available in DeepEval, both in existing metrics and the latest [JevEval.](https://deepeval.com/docs/metrics-jev-eval)
  62. > [!IMPORTANT]
  63. > If you're already in production, [Confident AI](https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=signup_callout&ref_page=github/readme) surfaces failing traces before anyone has written an eval for them, clusters them into failure modes, and turns them into your next eval.
  64. >
  65. > ![Demo GIF](assets/demo.gif)
  66. <br />
  67. # 🔥 Metrics and Features
  68. - 📐 Large variety of ready-to-use LLM eval metrics (all with explanations) powered by **ANY** LLM of your choice, statistical methods, or NLP models that run **locally on your machine** covering all use cases:
  69. - **Custom, All-Purpose Metrics:**
  70. - [JevEval](https://deepeval.com/docs/metrics-jev-eval) — a custom metric scored by Jev, a System One model that answers your questions with calibrated probabilities instead of generated text
  71. - [G-Eval](https://deepeval.com/docs/metrics-llm-evals) — a research-backed LLM-as-a-judge metric for evaluating on any custom criteria with human-like accuracy
  72. - [DAG](https://deepeval.com/docs/metrics-dag) — DeepEval's graph-based deterministic LLM-as-a-judge metric builder
  73. - <details>
  74. <summary><b>AI Agent Metrics</b></summary>
  75. - [Task Completion](https://deepeval.com/docs/metrics-task-completion) — evaluate whether an AI agent accomplished its goal
  76. - [Tool Correctness](https://deepeval.com/docs/metrics-tool-correctness) — check if the right tools were called with the right arguments
  77. - [Goal Accuracy](https://deepeval.com/docs/metrics-goal-accuracy) — measure how accurately the agent achieved the intended goal
  78. - [Step Efficiency](https://deepeval.com/docs/metrics-step-efficiency) — evaluate whether the agent took unnecessary steps
  79. - [Plan Adherence](https://deepeval.com/docs/metrics-plan-adherence) — check if the agent followed the expected plan
  80. - [Plan Quality](https://deepeval.com/docs/metrics-plan-quality) — evaluate the quality of the agent's plan
  81. - [Tool Use](https://deepeval.com/docs/metrics-tool-use) — measure quality of tool usage
  82. - [Argument Correctness](https://deepeval.com/docs/metrics-argument-correctness) — validate tool call arguments
  83. </details>
  84. - <details>
  85. <summary><b>Voice Agent Metrics</b></summary>
  86. - [Agent Responsiveness](https://deepeval.com/docs/metrics-agent-responsiveness) — check whether the voice agent answered when it was spoken to
  87. - [Turn-Taking Naturalness](https://deepeval.com/docs/metrics-turn-taking-naturalness) — measure response latency and interruptions across the call
  88. - [Voice Naturalness](https://deepeval.com/docs/metrics-voice-naturalness) — evaluate how natural the voice agent sounds
  89. - [Transcription Accuracy](https://deepeval.com/docs/metrics-transcription-accuracy) — check whether the voice agent's speech-to-text heard the caller correctly
  90. - [Audio Glitch](https://deepeval.com/docs/metrics-audio-glitch) — detect audio artifacts in the voice agent's speech
  91. </details>
  92. - <details>
  93. <summary><b>Multi-Turn Metrics</b></summary>
  94. - [Knowledge Retention](https://deepeval.com/docs/metrics-knowledge-retention) — evaluate whether the chatbot retains factual information throughout a conversation
  95. - [Conversation Completeness](https://deepeval.com/docs/metrics-conversation-completeness) — measure whether the chatbot satisfies user needs throughout a conversation
  96. - [Turn Relevancy](https://deepeval.com/docs/metrics-turn-relevancy) — evaluate whether the chatbot generates consistently relevant responses throughout a conversation
  97. - [Turn Faithfulness](https://deepeval.com/docs/metrics-turn-faithfulness) — check if the chatbot's responses are factually grounded in retrieval context across turns
  98. - [Role Adherence](https://deepeval.com/docs/metrics-role-adherence) — evaluate whether the chatbot adheres to its assigned role throughout a conversation
  99. </details>
  100. - <details>
  101. <summary><b>RAG Metrics</b></summary>
  102. - [Answer Relevancy](https://deepeval.com/docs/metrics-answer-relevancy) — measure how relevant the output is to the input
  103. - [Faithfulness](https://deepeval.com/docs/metrics-faithfulness) — evaluate whether the RAG pipeline's output factually aligns with the retrieval context
  104. - [Contextual Recall](https://deepeval.com/docs/metrics-contextual-recall) — measure how well the RAG pipeline's retrieval context aligns with the expected output
  105. - [Contextual Precision](https://deepeval.com/docs/metrics-contextual-precision) — evaluate whether relevant nodes in the RAG pipeline's retrieval context are ranked higher
  106. - [Contextual Relevancy](https://deepeval.com/docs/metrics-contextual-relevancy) — measure the overall relevance of the RAG pipeline's retrieval context to the input
  107. - [RAGAS](https://deepeval.com/docs/metrics-ragas) — average of answer relevancy, faithfulness, contextual precision, and contextual recall
  108. </details>
  109. - <details>
  110. <summary><b>MCP Metrics</b></summary>
  111. - [MCP Task Completion](https://deepeval.com/docs/metrics-mcp-task-completion) — evaluate how effectively an MCP-based agent accomplishes a task
  112. - [MCP Use](https://deepeval.com/docs/metrics-mcp-use) — measure how effectively an agent uses its available MCP servers
  113. - [Multi-Turn MCP Use](https://deepeval.com/docs/metrics-multi-turn-mcp-use) — evaluate MCP server usage across conversation turns
  114. </details>
  115. - <details>
  116. <summary><b>Multimodal Metrics</b></summary>
  117. - [Text to Image](https://deepeval.com/docs/multimodal-metrics-text-to-image) — evaluate image generation quality based on semantic consistency and perceptual quality
  118. - [Image Editing](https://deepeval.com/docs/multimodal-metrics-image-editing) — evaluate image editing quality based on semantic consistency and perceptual quality
  119. - [Image Coherence](https://deepeval.com/docs/multimodal-metrics-image-coherence) — measure how well images align with their accompanying text
  120. - [Image Helpfulness](https://deepeval.com/docs/multimodal-metrics-image-helpfulness) — evaluate how effectively images contribute to user comprehension of the text
  121. - [Image Reference](https://deepeval.com/docs/multimodal-metrics-image-reference) — evaluate how accurately images are referred to or explained by accompanying text
  122. </details>
  123. - <details>
  124. <summary><b>Other Metrics</b></summary>
  125. - [Hallucination](https://deepeval.com/docs/metrics-hallucination) — check whether the LLM generates factually correct information against provided context
  126. - [Summarization](https://deepeval.com/docs/metrics-summarization) — evaluate whether summaries are factually correct and include necessary details
  127. - [Bias](https://deepeval.com/docs/metrics-bias) — detect gender, racial, or political bias in LLM outputs
  128. - [Toxicity](https://deepeval.com/docs/metrics-toxicity) — evaluate toxicity in LLM outputs
  129. - [JSON Correctness](https://deepeval.com/docs/metrics-json-correctness) — check whether the output matches an expected JSON schema
  130. - [Prompt Alignment](https://deepeval.com/docs/metrics-prompt-alignment) — measure whether the output aligns with instructions in the prompt template
  131. </details>
  132. - 🎯 Supports both end-to-end and component-level LLM evaluation.
  133. - 🧩 Build your own custom metrics that are automatically integrated with DeepEval's ecosystem.
  134. - 🔮 Generate single and multi-turn synthetic datasets, and simulate live voice calls against your voice agent.
  135. - 🔗 Integrates seamlessly with **ANY** CI/CD environment.
  136. - 🧬 Optimize prompts automatically based on evaluation results.
  137. - 🏆 Easily benchmark **ANY** LLM on popular LLM benchmarks in [under 10 lines of code.](https://deepeval.com/docs/benchmarks-introduction?utm_source=GitHub), including MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K.
  138. <br />
  139. # 🔌 Integrations
  140. DeepEval plugs into any LLM framework — OpenAI Agents, LangChain, CrewAI, and more — and into voice stacks like Vapi, ElevenLabs, LiveKit, and Pipecat. The same traces can go to **Confident AI** once you're live, where evals run on every production trace.
  141. ## Frameworks
  142. - [LangChain](https://www.deepeval.com/integrations/frameworks/langchain?utm_source=GitHub) — evaluate LangChain applications with a callback handler
  143. - [LangGraph](https://www.deepeval.com/integrations/frameworks/langgraph?utm_source=GitHub) — evaluate LangGraph agents with a callback handler
  144. - [Pydantic AI](https://www.deepeval.com/integrations/frameworks/pydanticai?utm_source=GitHub) — evaluate Pydantic AI agents with type-safe validation
  145. - [CrewAI](https://www.deepeval.com/integrations/frameworks/crewai?utm_source=GitHub) — evaluate CrewAI multi-agent systems
  146. - [Anthropic](https://www.deepeval.com/integrations/frameworks/anthropic?utm_source=GitHub) — evaluate and trace Claude applications via a client wrapper
  147. - [AWS AgentCore](https://www.deepeval.com/integrations/frameworks/agentcore?utm_source=GitHub) — evaluate agents deployed on Amazon AgentCore
  148. - [Google ADK](https://www.deepeval.com/integrations/frameworks/google-adk?utm_source=GitHub) — evaluate Google ADK agents and multi-agent workflows
  149. - [AI SDK](https://www.deepeval.com/integrations/frameworks/ai-sdk?utm_source=GitHub) — evaluate AI SDK generations and tool-loop trajectories
  150. - [Mastra](https://www.deepeval.com/integrations/frameworks/mastra?utm_source=GitHub) — evaluate Mastra agents and workflows with native tracing
  151. - [OpenAI](https://www.deepeval.com/integrations/frameworks/openai?utm_source=GitHub) — evaluate and trace OpenAI applications via a client wrapper
  152. - [OpenAI Agents](https://www.deepeval.com/integrations/frameworks/openai-agents?utm_source=GitHub) — evaluate OpenAI Agents end-to-end in under a minute
  153. - [LlamaIndex](https://www.deepeval.com/integrations/frameworks/llamaindex?utm_source=GitHub) — evaluate AI agents and RAG applications built with LlamaIndex
  154. ## Voice Agent Connectors
  155. - [Vapi](https://www.deepeval.com/integrations/voice-agent-connectors/vapi?utm_source=GitHub) — simulate calls to a Vapi assistant over its WebSocket transport
  156. - [ElevenLabs](https://www.deepeval.com/integrations/voice-agent-connectors/elevenlabs?utm_source=GitHub) — simulate calls to an ElevenLabs conversational agent
  157. - [LiveKit](https://www.deepeval.com/integrations/voice-agent-connectors/livekit?utm_source=GitHub) — join your agent's LiveKit room and trade audio over WebRTC
  158. - [Pipecat](https://www.deepeval.com/integrations/voice-agent-connectors/pipecat?utm_source=GitHub) — connect to your Pipecat pipeline over its WebSocket transport
  159. ## ☁️ Platform + Ecosystem
  160. [Confident AI](https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=platform_section&ref_page=github/readme) is AI monitoring for applications in production. DeepEval tests what you know to test before you ship; Confident AI finds what you didn't, and it's model- and framework-agnostic.
  161. - Failing traces surface before anyone has written an eval for them, clustered into failure modes, with user sentiment and behavior across traces.
  162. - Failing traces route to the right annotators and become goldens, so production issues turn into your next eval.
  163. - For enterprise teams, one system for AI quality: evals, monitoring, human annotation, and voice-native observability, with governance as release gates.
  164. - Prefer to stay in your editor? Inspect failing traces and pull the datasets built from them in Claude Code or Cursor through Confident AI's [MCP server](https://github.com/confident-ai/confident-mcp-server).
  165. <p align="center">
  166. <img src="assets/confident-mcp-architecture.png" alt="Confident AI MCP Architecture" width="500">
  167. </p>
  168. <br />
  169. # 🤖 Vibe-Coder QuickStart
  170. Want your coding agent to add evals and fix failures for you? Install the DeepEval skill, point it at your AI agent, voice agent, multi-turn chatbot, RAG pipeline, or any other LLM app, and ask it to generate a dataset, write the eval suite, run `deepeval test run`, and iterate on the failing metrics.
  171. [Start with the 5-minute vibe-coder guide](https://deepeval.com/docs/vibe-coder-quickstart?utm_source=GitHub).
  172. <br />
  173. # 🚀 Human QuickStart
  174. Let's pretend your LLM application is a customer support AI agent; here's how DeepEval can help test what you've built.
  175. ## Installation
  176. Deepeval works with **Python>=3.10+**.
  177. ```
  178. pip install -U deepeval
  179. ```
  180. ## Create an account (highly recommended)
  181. Using the `deepeval` platform will allow you to generate sharable testing reports on the cloud. It is free, takes no additional code to setup, and we highly recommend giving it a try.
  182. To login, run:
  183. ```
  184. deepeval login
  185. ```
  186. Follow the instructions in the CLI to create an account, copy your API key, and paste it into the CLI. All test cases will automatically be logged (find more information on data privacy [here](https://deepeval.com/docs/data-privacy?utm_source=GitHub)).
  187. ## Write your first test case
  188. Create a test file:
  189. ```bash
  190. touch test_ai_agent.py
  191. ```
  192. Open `test_ai_agent.py` and write your first test case to run an **end-to-end** evaluation using DeepEval, which treats your LLM app as a black-box:
  193. ```python
  194. import pytest
  195. from deepeval import assert_test
  196. from deepeval.metrics import GEval
  197. from deepeval.test_case import LLMTestCase, SingleTurnParams
  198. def test_case():
  199. correctness_metric = GEval(
  200. name="Correctness",
  201. criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
  202. evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
  203. threshold=0.5
  204. )
  205. test_case = LLMTestCase(
  206. input="What if these shoes don't fit?",
  207. # Replace this with the actual output from your LLM application
  208. actual_output="You have 30 days to get a full refund at no extra cost.",
  209. expected_output="We offer a 30-day full refund at no extra costs.",
  210. )
  211. assert_test(test_case, [correctness_metric])
  212. ```
  213. Set your `OPENAI_API_KEY` as an environment variable (you can also evaluate using your own custom model, for more details visit [this part of our docs](https://deepeval.com/docs/metrics-introduction#using-a-custom-llm?utm_source=GitHub)):
  214. ```
  215. export OPENAI_API_KEY="..."
  216. ```
  217. And finally, run `test_ai_agent.py` in the CLI:
  218. ```
  219. deepeval test run test_ai_agent.py
  220. ```
  221. **Congratulations! Your test case should have passed ✅** Let's break down what happened.
  222. - The variable `input` mimics a user input, and `actual_output` is a placeholder for what your application's supposed to output based on this input.
  223. - The variable `expected_output` represents the ideal answer for a given `input`, and [`GEval`](https://deepeval.com/docs/metrics-llm-evals) is a research-backed metric provided by `deepeval` for you to evaluate your LLM outputs on any custom criteria with human-like accuracy.
  224. - In this example, the metric `criteria` is correctness of the `actual_output` based on the provided `expected_output`.
  225. - All metric scores range from 0 - 1, which the `threshold=0.5` ultimately determines if your test has passed or not.
  226. [Read our documentation](https://deepeval.com/docs/getting-started?utm_source=GitHub) for more information!
  227. <br />
  228. ## Evals With Full Traceability
  229. Use `evals_iterator()` to run the same dataset through your app, whether you instrument it manually or through one of DeepEval's framework integrations. Because tracing captures the ordered sequence of model decisions, tool calls, and intermediate steps, you can run [trajectory-based evaluations](https://deepeval.com/docs/evaluation-trajectory-based-llm-evals?utm_source=GitHub) over the complete path your AI agent takes.
  230. Here's an example of manual instrumentation:
  231. ```python
  232. from deepeval.tracing import observe, update_current_span
  233. from deepeval.test_case import LLMTestCase
  234. from deepeval.metrics import TaskCompletionMetric
  235. @observe()
  236. def inner_component(input: str):
  237. output = "result"
  238. update_current_span(test_case=LLMTestCase(input=input, actual_output=output))
  239. return output
  240. @observe()
  241. def app(input: str):
  242. return inner_component(input)
  243. # This metric evaluates the complete trajectory captured for this run.
  244. for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
  245. app(golden.input)
  246. ```
  247. <details>
  248. <summary><b>OpenAI</b></summary>
  249. ```python
  250. from deepeval.openai import OpenAI
  251. from deepeval.tracing import trace
  252. from deepeval.metrics import TaskCompletionMetric
  253. client = OpenAI()
  254. # This metric evaluates the complete trajectory captured for this run.
  255. for golden in dataset.evals_iterator():
  256. with trace(metrics=[TaskCompletionMetric()]):
  257. client.chat.completions.create(
  258. model="gpt-4o",
  259. messages=[{"role": "user", "content": golden.input}],
  260. )
  261. ```
  262. </details>
  263. <details>
  264. <summary><b>OpenAI Agents</b></summary>
  265. ```python
  266. from agents import Runner
  267. from deepeval.metrics import TaskCompletionMetric
  268. # This metric evaluates the complete trajectory captured for this run.
  269. for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
  270. Runner.run_sync(agent, golden.input)
  271. ```
  272. </details>
  273. <details>
  274. <summary><b>AI SDK (TypeScript)</b></summary>
  275. ```typescript
  276. import { generateText } from "ai";
  277. import { configureAiSdkTracing } from "deepeval/integrations/ai-sdk";
  278. import { TaskCompletionMetric } from "deepeval/metrics";
  279. const tracer = configureAiSdkTracing({ name: "my-agent" });
  280. const ask = (input: string) =>
  281. generateText({
  282. model,
  283. prompt: input,
  284. experimental_telemetry: { isEnabled: true, tracer },
  285. });
  286. // This metric evaluates the complete trajectory captured for this run.
  287. for await (const golden of dataset.evalsIterator({
  288. metrics: [new TaskCompletionMetric()],
  289. })) {
  290. await ask(golden.input);
  291. }
  292. ```
  293. </details>
  294. <details>
  295. <summary><b>Mastra (TypeScript)</b></summary>
  296. ```typescript
  297. import { Mastra } from "@mastra/core/mastra";
  298. import { Observability } from "@mastra/observability";
  299. import { DeepEvalExporter } from "deepeval/integrations/mastra";
  300. import { TaskCompletionMetric } from "deepeval/metrics";
  301. const mastra = new Mastra({
  302. agents: { agent },
  303. observability: new Observability({
  304. configs: {
  305. deepeval: { exporters: [new DeepEvalExporter()] },
  306. },
  307. }),
  308. });
  309. // This metric evaluates the complete trajectory captured for this run.
  310. for await (const golden of dataset.evalsIterator({
  311. metrics: [new TaskCompletionMetric()],
  312. })) {
  313. await mastra.getAgent("agent").generate(golden.input);
  314. }
  315. ```
  316. </details>
  317. <details>
  318. <summary><b>Anthropic</b></summary>
  319. ```python
  320. from deepeval.anthropic import Anthropic
  321. from deepeval.tracing import trace
  322. from deepeval.metrics import TaskCompletionMetric
  323. client = Anthropic()
  324. # This metric evaluates the complete trajectory captured for this run.
  325. for golden in dataset.evals_iterator():
  326. with trace(metrics=[TaskCompletionMetric()]):
  327. client.messages.create(
  328. model="claude-sonnet-4-5",
  329. max_tokens=1024,
  330. messages=[{"role": "user", "content": golden.input}],
  331. )
  332. ```
  333. </details>
  334. <details>
  335. <summary><b>LangChain</b></summary>
  336. ```python
  337. from deepeval.integrations.langchain import CallbackHandler
  338. from deepeval.metrics import TaskCompletionMetric
  339. # This metric evaluates the complete trajectory captured for this run.
  340. for golden in dataset.evals_iterator():
  341. llm.invoke(
  342. golden.input,
  343. config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
  344. )
  345. ```
  346. </details>
  347. <details>
  348. <summary><b>LangGraph</b></summary>
  349. ```python
  350. from deepeval.integrations.langchain import CallbackHandler
  351. from deepeval.metrics import TaskCompletionMetric
  352. # This metric evaluates the complete trajectory captured for this run.
  353. for golden in dataset.evals_iterator():
  354. agent.invoke(
  355. {"messages": [{"role": "user", "content": golden.input}]},
  356. config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
  357. )
  358. ```
  359. </details>
  360. <details>
  361. <summary><b>Pydantic AI</b></summary>
  362. ```python
  363. from deepeval.metrics import TaskCompletionMetric
  364. # This metric evaluates the complete trajectory captured for this run.
  365. for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
  366. agent.run_sync(golden.input)
  367. ```
  368. </details>
  369. <details>
  370. <summary><b>CrewAI</b></summary>
  371. ```python
  372. from deepeval.integrations.crewai import instrument_crewai
  373. from deepeval.metrics import TaskCompletionMetric
  374. instrument_crewai()
  375. # This metric evaluates the complete trajectory captured for this run.
  376. for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
  377. crew.kickoff({"input": golden.input})
  378. ```
  379. </details>
  380. <details>
  381. <summary><b>AWS AgentCore</b></summary>
  382. ```python
  383. from deepeval.integrations.agentcore import instrument_agentcore
  384. from deepeval.metrics import TaskCompletionMetric
  385. instrument_agentcore()
  386. # This metric evaluates the complete trajectory captured for this run.
  387. for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
  388. invoke({"prompt": golden.input})
  389. ```
  390. </details>
  391. <details>
  392. <summary><b>LlamaIndex</b></summary>
  393. ```python
  394. import asyncio
  395. from deepeval.evaluate.configs import AsyncConfig
  396. from deepeval.metrics import TaskCompletionMetric
  397. # This metric evaluates the complete trajectory captured for this run.
  398. for golden in dataset.evals_iterator(
  399. async_config=AsyncConfig(run_async=True),
  400. metrics=[TaskCompletionMetric()],
  401. ):
  402. task = asyncio.create_task(agent.run(golden.input))
  403. dataset.evaluate(task)
  404. ```
  405. </details>
  406. <details>
  407. <summary><b>Google ADK</b></summary>
  408. ```python
  409. import asyncio
  410. from deepeval.evaluate.configs import AsyncConfig
  411. from deepeval.integrations.google_adk import instrument_google_adk
  412. from deepeval.metrics import TaskCompletionMetric
  413. instrument_google_adk()
  414. # This metric evaluates the complete trajectory captured for this run.
  415. for golden in dataset.evals_iterator(
  416. async_config=AsyncConfig(run_async=True),
  417. metrics=[TaskCompletionMetric()],
  418. ):
  419. task = asyncio.create_task(run_agent(golden.input))
  420. dataset.evaluate(task)
  421. ```
  422. </details>
  423. <details>
  424. <summary><b>Strands</b></summary>
  425. ```python
  426. from deepeval.integrations.strands import instrument_strands
  427. from deepeval.metrics import TaskCompletionMetric
  428. instrument_strands()
  429. # This metric evaluates the complete trajectory captured for this run.
  430. for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric()]):
  431. agent(golden.input)
  432. ```
  433. </details>
  434. Learn more about component-level evaluations [here.](https://www.deepeval.com/docs/evaluation-component-level-llm-evals)
  435. <br />
  436. ## Evaluate Without Pytest Integration
  437. Alternatively, you can evaluate without Pytest, which is more suited for a notebook environment.
  438. ```python
  439. from deepeval import evaluate
  440. from deepeval.metrics import AnswerRelevancyMetric
  441. from deepeval.test_case import LLMTestCase
  442. answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
  443. test_case = LLMTestCase(
  444. input="What if these shoes don't fit?",
  445. # Replace this with the actual output from your LLM application
  446. actual_output="We offer a 30-day full refund at no extra costs.",
  447. retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
  448. )
  449. evaluate([test_case], [answer_relevancy_metric])
  450. ```
  451. ## Using Standalone Metrics
  452. DeepEval is extremely modular, making it easy for anyone to use any of our metrics. Continuing from the previous example:
  453. ```python
  454. from deepeval.metrics import AnswerRelevancyMetric
  455. from deepeval.test_case import LLMTestCase
  456. answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
  457. test_case = LLMTestCase(
  458. input="What if these shoes don't fit?",
  459. # Replace this with the actual output from your LLM application
  460. actual_output="We offer a 30-day full refund at no extra costs.",
  461. retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
  462. )
  463. answer_relevancy_metric.measure(test_case)
  464. print(answer_relevancy_metric.score)
  465. # All metrics also offer an explanation
  466. print(answer_relevancy_metric.reason)
  467. ```
  468. Note that some metrics are built for specific use cases, such as AI agents, voice agents, multi-turn chatbots, RAG, and more. Make sure to use our docs to pick the right one for your use case.
  469. ## A Note on Env Variables (.env / .env.local)
  470. DeepEval auto-loads `.env.local` then `.env` from the current working directory **at import time**.
  471. **Precedence:** process env -> `.env.local` -> `.env`.
  472. Opt out with `DEEPEVAL_DISABLE_DOTENV=1`.
  473. ```bash
  474. cp .env.example .env.local
  475. # then edit .env.local (ignored by git)
  476. ```
  477. # DeepEval With Confident AI
  478. [Confident AI](https://www.confident-ai.com?utm_source=deepeval&utm_medium=github&utm_content=cli_login_section&ref_page=github/readme) monitors AI applications in production. Evals run on every production trace, failing traces are clustered into failure modes, and each one can be routed to the right annotators — engineers, QA, domain experts or PMs — and turned into a golden for your next eval. The traces your app already produces with DeepEval stream to it with no code changes.
  479. Log in from the CLI to connect:
  480. ```bash
  481. deepeval login
  482. ```
  483. Test runs also appear on the platform once you're logged in.
  484. ![Demo GIF](assets/demo.gif)
  485. Prefer to stay in your IDE? Inspect failing traces and pull the datasets built from them through [Confident AI's MCP server](https://github.com/confident-ai/confident-mcp-server) without leaving your editor.
  486. <p align="center">
  487. <img src="assets/confident-mcp-architecture.png" alt="Confident AI MCP Architecture" width="500">
  488. </p>
  489. Learn more in the [Confident AI documentation](https://www.confident-ai.com/docs?utm_source=deepeval&utm_medium=github&utm_content=cloud_docs&ref_page=github/readme).
  490. <br />
  491. # Contributing
  492. Please read [CONTRIBUTING.md](https://github.com/confident-ai/deepeval/blob/main/CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
  493. <br />
  494. # Roadmap
  495. Features:
  496. - [x] Integration with Confident AI
  497. - [x] Implement G-Eval
  498. - [x] Implement RAG metrics
  499. - [x] Implement Conversational metrics
  500. - [x] Evaluation Dataset Creation
  501. - [x] Red-Teaming
  502. - [ ] DAG custom metrics
  503. - [ ] Guardrails
  504. <br />
  505. # Authors
  506. Built by the founders of Confident AI. Contact jeffreyip@confident-ai.com for all enquiries.
  507. <br />
  508. # License
  509. DeepEval is licensed under Apache 2.0 - see the [LICENSE.md](https://github.com/confident-ai/deepeval/blob/main/LICENSE.md) file for details.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.