Skip to content
Back to search

skunit

skUnit is a testing tool for AI units, such as IChatClient, MCP Servers and agents.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mehrandvd--skunit/badge.svg)](/listings/mehrandvd--skunit)

HTML

<a href="/listings/mehrandvd--skunit"><img src="/api/listings/mehrandvd--skunit/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Imagine you have an `AIAgent` configured to answer questions about a bank account. Here's a test scenario in Markdown, `balance-test.md`:

# SCENARIO Balance Test

## [USER]
What is the account balance for John Doe?

## [ASSISTANT]

### ASSERT SemanticCondition
The answer mentions that the account balance is $1,234.56.

And the C# test:

[Fact]
public async Task SimpleTest()
{
    var markdown = File.ReadAllText("balance-test.md");
    var scenario = ChatScenario.Parse(markdown);

    await agent.RunAsync(scenario);
}

That's it. skUnit drives the conversation, calls your AI, and **semantically verifies** the response.

> **Tip — Parameterized scenarios:** Use `{placeholders}` in your Markdown and replace them with C# string manipulation to run the same scenario with different inputs. > > ````markdown > ## [USER] > What is the account balance for {CustomerName}? > > ## [ASSISTANT] > > ### ASSERT SemanticCondition > The answer mentions that the account balance is {CustomerBalance}. > ````

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 277 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # skUnit
  2. [![Build and Deploy](https://github.com/mehrandvd/skUnit/actions/workflows/build.yml/badge.svg)](https://github.com/mehrandvd/skUnit/actions/workflows/build.yml)
  3. [![NuGet version (skUnit)](https://img.shields.io/nuget/v/skUnit.svg?style=flat)](https://www.nuget.org/packages/skUnit/)
  4. [![NuGet downloads](https://img.shields.io/nuget/dt/skUnit.svg?style=flat)](https://www.nuget.org/packages/skUnit)
  5. [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/mehrandvd/skunit)
  6. **skUnit** is a semantic testing framework for .NET — write AI tests in **plain Markdown**, run them with **xUnit, NUnit, or MSTest**.
  7. Test anything that talks to AI:
  8. - **AI Agents** (Microsoft Agent Framework, A2A)
  9. - **IChatClient** (Microsoft Extensions AI)
  10. - **MCP servers** (Model Context Protocol)
  11. - **Custom AI integrations**
  12. ```bash
  13. dotnet add package skUnit
  14. ```
  15. ## Quick Start
  16. Imagine you have an `AIAgent` configured to answer questions about a bank account. Here's a test scenario in Markdown, `balance-test.md`:
  17. ````markdown
  18. # SCENARIO Balance Test
  19. ## [USER]
  20. What is the account balance for John Doe?
  21. ## [ASSISTANT]
  22. ### ASSERT SemanticCondition
  23. The answer mentions that the account balance is $1,234.56.
  24. ````
  25. And the C# test:
  26. ```csharp
  27. [Fact]
  28. public async Task SimpleTest()
  29. {
  30. var markdown = File.ReadAllText("balance-test.md");
  31. var scenario = ChatScenario.Parse(markdown);
  32. await agent.RunAsync(scenario);
  33. }
  34. ```
  35. That's it. skUnit drives the conversation, calls your AI, and **semantically verifies** the response.
  36. > **Tip — Parameterized scenarios:** Use `{placeholders}` in your Markdown and replace them with C# string manipulation to run the same scenario with different inputs.
  37. >
  38. > ````markdown
  39. > ## [USER]
  40. > What is the account balance for {CustomerName}?
  41. >
  42. > ## [ASSISTANT]
  43. >
  44. > ### ASSERT SemanticCondition
  45. > The answer mentions that the account balance is {CustomerBalance}.
  46. > ````
  47. ## Key Features
  48. ### 1. Function Call Assertion
  49. Verify that your agent calls the **right tool with the right arguments**:
  50. ````markdown
  51. # SCENARIO Balance Test
  52. ## [USER]
  53. What is the account balance for John Doe?
  54. ## [ASSISTANT]
  55. ### ASSERT SemanticCondition
  56. The answer mentions that the account balance is $1,234.56.
  57. ### ASSERT FunctionCall
  58. {
  59. "function_name": "get-account-balance"
  60. }
  61. ````
  62. You can also assert on the **arguments** passed to the function, using exact or semantic matching:
  63. ```json
  64. {
  65. "function_name": "get-account-balance",
  66. "arguments": {
  67. "accountHolder": ["Equals", "John Doe"]
  68. }
  69. }
  70. ```
  71. ```json
  72. {
  73. "function_name": "get-account-balance",
  74. "arguments": {
  75. "accountHolder": ["SemanticSimilar", "John Doe"]
  76. }
  77. }
  78. ```
  79. > See [ASSERT JsonCheck](docs/check-statements-spec.md#assert-jsoncheck) for full JSON schema validation.
  80. ---
  81. ### 2. Multi-Turn Conversations
  82. Test **realistic back-and-forth conversations** — each turn gets its own assertions:
  83. ````markdown
  84. # SCENARIO Multi-Turn Balance Test
  85. ## [USER]
  86. What is the account balance for John Doe?
  87. ## [ASSISTANT]
  88. The account balance for John Doe is $1,234.56.
  89. ### ASSERT SemanticCondition
  90. The answer mentions that the account balance is $1,234.56.
  91. ## [USER]
  92. What about his credit score?
  93. ## [ASSISTANT]
  94. His credit score is 750.
  95. ### ASSERT SemanticCondition
  96. It mentions that the credit score is 750.
  97. ````
  98. ![skUnit Chat Scenario Structure](https://github.com/mehrandvd/skunit/assets/5070766/156b0831-e4f3-4e4b-b1b0-e2ec868efb5f)
  99. Each `[ASSISTANT]` turn can carry **multiple ASSERT statements** to verify different aspects of the response independently.
  100. ---
  101. ### 3. MCP Server Testing
  102. Test [Model Context Protocol](https://modelcontextprotocol.io/) servers end-to-end — same Markdown format, no extra APIs to learn:
  103. ````markdown
  104. # SCENARIO MCP Time Server
  105. ## [USER]
  106. What time is it?
  107. ## [ASSISTANT]
  108. It's currently 2:30 PM PST
  109. ### ASSERT FunctionCall
  110. {
  111. "function_name": "current_time"
  112. }
  113. ### ASSERT SemanticCondition
  114. It mentions a specific time
  115. ````
  116. Wire up your MCP server in the test setup:
  117. ```csharp
  118. var mcp = await McpClientFactory.CreateAsync(clientTransport);
  119. var tools = await mcp.ListToolsAsync();
  120. var baseChatClient = // your IChatClient (Azure OpenAI, OpenAI, etc.)
  121. var agent = baseChatClient.AsAgent(
  122. instructions: "You are a helpful assistant that can answer questions and call tools.",
  123. tools: tools.ToArray());
  124. await agent.ExecuteScenarioAsync(scenario, assertionClient: baseChatClient);
  125. // assertionClient = the AI that evaluates responses semantically
  126. // agent = the system under test that actually calls tools
  127. ```
  128. > Need to test **multiple MCP servers** together? Merge their tool lists before creating the agent — see [MCP Testing Guide](docs/mcp-testing-guide.md).
  129. ---
  130. ### 4. Hallucination Mitigation with `ScenarioRunOptions`
  131. LLM outputs vary between runs. **Don't let a single flaky response break your build.**
  132. Run each scenario multiple times and require a minimum number of passes:
  133. ```csharp
  134. var options = new ScenarioRunOptions
  135. {
  136. TotalRuns = 3, // Run the scenario 3 times
  137. RequiredSuccessRuns = 2 // At least 2 must pass
  138. };
  139. await agent.ExecuteScenarioAsync(scenario, assertionClient: baseChatClient, options: options);
  140. ```
  141. **Recommended thresholds:**
  142. | Scenario type | TotalRuns | RequiredSuccessRuns |
  143. |---|---|---|
  144. | Deterministic / low-temp | 1 | 1 |
  145. | Function / tool invocation | 3 | 2 |
  146. | Creative generation | 5 | 3 |
  147. | Critical CI gating | 5 | 4 |
  148. When a scenario falls below the threshold, you get a clear failure message:
  149. ```
  150. Only 2 of 5 runs passed, which is below the required success runs of 4.
  151. ```
  152. > See [Scenario Run Options](docs/scenario-run-options.md) for the full guide.
  153. ---
  154. ## Setup
  155. ```csharp
  156. public class BankAccountTests
  157. {
  158. private readonly ChatScenarioRunner ScenarioRunner;
  159. private readonly IChatClient systemUnderTestClient;
  160. public BankAccountTests(ITestOutputHelper output) // xUnit
  161. {
  162. var assertionClient = new AzureOpenAIClient(endpoint, credential)
  163. .GetChatClient(deploymentName)
  164. .AsIChatClient();
  165. ScenarioRunner = new ChatScenarioRunner(assertionClient, output.WriteLine);
  166. }
  167. [Fact]
  168. public async Task TestBalance()
  169. {
  170. var agentUnderTest = /* the client/agent you are testing */
  171. var scenarios = ChatScenario.Parse(File.ReadAllText("balance-test.md"));
  172. await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient);
  173. }
  174. }
  175. ```
  176. Two clients, two roles:
  177. - **`assertionClient`** — evaluates semantic assertions (`SemanticCondition`, `SemanticSimilar`, etc.)
  178. - **`agentUnderTest`** — the agent/client whose behavior you are testing
  179. > **MSTest / NUnit:** Replace `ITestOutputHelper.WriteLine` with `TestContext.WriteLine`. Everything else stays the same. See [Test Framework Integration](docs/test-framework-integration.md).
  180. ---
  181. ## Documentation
  182. | Doc | Description |
  183. |---|---|
  184. | [Chat Scenario Spec](docs/chat-scenario-spec.md) | Complete guide to writing scenarios |
  185. | [ASSERT Statement Spec](docs/check-statements-spec.md) | All available assertion types |
  186. | [Test Framework Integration](docs/test-framework-integration.md) | xUnit, MSTest, NUnit setup |
  187. | [MCP Testing Guide](docs/mcp-testing-guide.md) | Testing MCP servers |
  188. | [Multi-Modal Support](docs/multi-modal-support.md) | Images and other media |
  189. | [Scenario Run Options](docs/scenario-run-options.md) | Multi-run hallucination mitigation |
  190. ## Examples
  191. See the `/demos` folder for complete, runnable projects:
  192. - **[Moody Chef AI](demos/Demo.MoodyChef)**: A step-by-step tutorial project that shows how to build and test an AI agent using skUnit.
  193. ## Requirements
  194. - **.NET 8.0** or higher
  195. - An **AI provider** (Azure OpenAI, OpenAI, Anthropic, ...) for semantic assertions
  196. - A **test framework** of your choice — xUnit, NUnit, or MSTest
  197. ## Contributing
  198. Contributions are welcome! Check [open issues](https://github.com/mehrandvd/skunit/issues) or submit a PR.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.