Skip to content
Back to search

mcp-evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mclenhard--mcp-evals/badge.svg)](/listings/mclenhard--mcp-evals)

HTML

<a href="/listings/mclenhard--mcp-evals"><img src="/api/listings/mclenhard--mcp-evals/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

As a Node.js Package

npm install mcp-evals

As a GitHub Action

Add the following to your workflow file:

name: Run MCP Evaluations
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Run MCP Evaluations
        uses: mclenhard/mcp-evals@v1.0.9
        with:
          evals_path: 'src/evals/evals.ts'    # Can also use .yaml files
          server_path: 'src/index.ts'
          openai_api_key: ${{ secrets.OPENAI_API_KEY }}
          model: 'gpt-4'  # Optional, defaults to gpt-4

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 225 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # MCP Evals
  2. A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, **with built-in observability support**. This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics.
  3. ## Installation
  4. ### As a Node.js Package
  5. ```bash
  6. npm install mcp-evals
  7. ```
  8. ### As a GitHub Action
  9. Add the following to your workflow file:
  10. ```yaml
  11. name: Run MCP Evaluations
  12. on:
  13. pull_request:
  14. types: [opened, synchronize, reopened]
  15. jobs:
  16. evaluate:
  17. runs-on: ubuntu-latest
  18. permissions:
  19. contents: read
  20. pull-requests: write
  21. steps:
  22. - uses: actions/checkout@v4
  23. - name: Setup Node.js
  24. uses: actions/setup-node@v4
  25. with:
  26. node-version: '20'
  27. - name: Install dependencies
  28. run: npm install
  29. - name: Run MCP Evaluations
  30. uses: mclenhard/mcp-evals@v1.0.9
  31. with:
  32. evals_path: 'src/evals/evals.ts' # Can also use .yaml files
  33. server_path: 'src/index.ts'
  34. openai_api_key: ${{ secrets.OPENAI_API_KEY }}
  35. model: 'gpt-4' # Optional, defaults to gpt-4
  36. ```
  37. ## Usage -- Evals
  38. ### 1. Create Your Evaluation File
  39. You can create evaluation configurations in either TypeScript or YAML format.
  40. #### Option A: TypeScript Configuration
  41. Create a file (e.g., `evals.ts`) that exports your evaluation configuration:
  42. ```typescript
  43. import { EvalConfig } from 'mcp-evals';
  44. import { openai } from "@ai-sdk/openai";
  45. import { grade, EvalFunction} from "mcp-evals";
  46. const weatherEval: EvalFunction = {
  47. name: 'Weather Tool Evaluation',
  48. description: 'Evaluates the accuracy and completeness of weather information retrieval',
  49. run: async () => {
  50. const result = await grade(openai("gpt-4"), "What is the weather in New York?");
  51. return JSON.parse(result);
  52. }
  53. };
  54. const config: EvalConfig = {
  55. model: openai("gpt-4"),
  56. evals: [weatherEval]
  57. };
  58. export default config;
  59. export const evals = [
  60. weatherEval,
  61. // add other evals here
  62. ];
  63. ```
  64. #### Option B: YAML Configuration
  65. For simpler configuration, you can use YAML format (e.g., `evals.yaml`):
  66. ```yaml
  67. # Model configuration
  68. model:
  69. provider: openai # 'openai' or 'anthropic'
  70. name: gpt-4o # Model name
  71. # api_key: sk-... # Optional, uses OPENAI_API_KEY env var by default
  72. # List of evaluations to run
  73. evals:
  74. - name: weather_query_basic
  75. description: Test basic weather information retrieval
  76. prompt: "What is the current weather in San Francisco?"
  77. expected_result: "Should return current weather data for San Francisco including temperature, conditions, etc."
  78. - name: weather_forecast
  79. description: Test weather forecast functionality
  80. prompt: "Can you give me the 3-day weather forecast for Seattle?"
  81. expected_result: "Should return a multi-day forecast for Seattle"
  82. - name: invalid_location
  83. description: Test handling of invalid location requests
  84. prompt: "What's the weather in Atlantis?"
  85. expected_result: "Should handle invalid location gracefully with appropriate error message"
  86. ```
  87. ### 2. Run the Evaluations
  88. #### As a Node.js Package
  89. You can run the evaluations using the CLI with either TypeScript or YAML files:
  90. ```bash
  91. # Using TypeScript configuration
  92. npx mcp-eval path/to/your/evals.ts path/to/your/server.ts
  93. # Using YAML configuration
  94. npx mcp-eval path/to/your/evals.yaml path/to/your/server.ts
  95. ```
  96. #### As a GitHub Action
  97. The action will automatically:
  98. 1. Run your evaluations
  99. 2. Post the results as a comment on the PR
  100. 3. Update the comment if the PR is updated
  101. ## Evaluation Results
  102. Each evaluation returns an object with the following structure:
  103. ```typescript
  104. interface EvalResult {
  105. accuracy: number; // Score from 1-5
  106. completeness: number; // Score from 1-5
  107. relevance: number; // Score from 1-5
  108. clarity: number; // Score from 1-5
  109. reasoning: number; // Score from 1-5
  110. overall_comments: string; // Summary of strengths and weaknesses
  111. }
  112. ```
  113. ## Configuration
  114. ### Environment Variables
  115. - `OPENAI_API_KEY`: Your OpenAI API key (required for OpenAI models)
  116. - `ANTHROPIC_API_KEY`: Your Anthropic API key (required for Anthropic models)
  117. > [!NOTE]
  118. > If you're using this GitHub Action with open source software, enable data sharing in the OpenAI billing dashboard to claim 2.5 million free GPT-4o mini tokens per day, making this Action effectively free to use.
  119. ### Evaluation Configuration
  120. #### TypeScript Configuration
  121. The `EvalConfig` interface requires:
  122. - `model`: The language model to use for evaluation (e.g., GPT-4)
  123. - `evals`: Array of evaluation functions to run
  124. Each evaluation function must implement:
  125. - `name`: Name of the evaluation
  126. - `description`: Description of what the evaluation tests
  127. - `run`: Async function that takes a model and returns an `EvalResult`
  128. #### YAML Configuration
  129. YAML configuration files support:
  130. **Model Configuration:**
  131. - `provider`: Either 'openai' or 'anthropic'
  132. - `name`: Model name (e.g., 'gpt-4o', 'claude-3-opus-20240229')
  133. - `api_key`: Optional API key (uses environment variables by default)
  134. **Evaluation Configuration:**
  135. - `name`: Name of the evaluation (required)
  136. - `description`: Description of what the evaluation tests (required)
  137. - `prompt`: The prompt to send to your MCP server (required)
  138. - `expected_result`: Optional description of expected behavior
  139. **Supported File Extensions:** `.yaml`, `.yml`
  140. ## Usage -- Monitoring
  141. > **Note:** The metrics functionality is still in alpha. Features and APIs may change, and breaking changes are possible.
  142. 1. Add the following to your application before you initilize the MCP server.
  143. ```typescript
  144. import { metrics } from 'mcp-evals';
  145. metrics.initialize(9090, { enableTracing: true, otelEndpoint: 'http://localhost:4318/v1/traces' });
  146. ```
  147. 2. Start the monitoring stack:
  148. ```bash
  149. docker-compose up -d
  150. ```
  151. 3. Run your MCP server and it will automatically connect to the monitoring stack.
  152. ### Accessing the Dashboards
  153. - **Prometheus**: http://localhost:9090
  154. - **Grafana**: http://localhost:3000 (username: admin, password: admin)
  155. - **Jaeger UI**: http://localhost:16686
  156. ## Metrics Available
  157. - **Tool Calls**: Number of tool calls by tool name
  158. - **Tool Errors**: Number of errors by tool name
  159. - **Tool Latency**: Distribution of latency times by tool name
  160. ## License
  161. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.