Skip to content
Back to search

mlflow

Skills for tracing, evaluating, and improving AI agents with MLflow. Supports the full agent improvement loop: instrument → trace → evaluate → iterate → validate.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/mlflow--skills/badge.svg)](/listings/mlflow--skills)

HTML

<a href="/listings/mlflow--skills"><img src="/api/listings/mlflow--skills/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Using `skills` installer

npx skills add mlflow/skills

Direct Installation from Source

git clone https://github.com/mlflow/skills.git
cp -r mlflow-skills/* ~/.claude/skills/

Change the `~/.claude/skills/` directory to the appropriate location for your coding agent, e.g., `~/.codex/skills/` for Codex.

Project-Level Installation

Add skills to your project for team sharing:

cd your-project
git clone https://github.com/mlflow/skills.git .skills/mlflow
# Or as a submodule:
git submodule add https://github.com/mlflow/skills.git .skills/mlflow

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/plugin.json · Manifest · 25 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "mlflow",
  3. "displayName": "MLflow Skills",
  4. "description": "Skills for tracing, evaluating, and improving AI agents with MLflow. Supports the full agent improvement loop: instrument → trace → evaluate → iterate → validate.",
  5. "version": "1.0.0",
  6. "author": {
  7. "name": "MLflow Team"
  8. },
  9. "homepage": "https://github.com/mlflow/skills",
  10. "skills": [
  11. "./mlflow-agent",
  12. "./mlflow-onboarding",
  13. "./mlflow-docs",
  14. "./mlflow-sagemaker",
  15. "./mlflow-trace-instrumentation",
  16. "./mlflow-trace-retrieval",
  17. "./mlflow-trace-analysis",
  18. "./mlflow-trace-metrics",
  19. "./mlflow-evaluation",
  20. "./mlflow-agent-improvement"
  21. ],
  22. "hooks": "./hooks/hooks.json",
  23. "categories": ["observability", "evaluation", "ai-agents"],
  24. "minClaudeCodeVersion": "2.0.0"
  25. }
README.md · README · 275 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ![MLflow Skills](/assets/header.svg)
  2. # MLflow Skills for Coding Agents
  3. > Turn your favorite coding agent into an LLMOps expert with MLflow skills.
  4. Build, debug, and evaluate GenAI applications with confidence. These skills give your AI coding assistant deep knowledge of MLflow's tracing, evaluation, and observability features.
  5. Works with any coding agent that support Skills, including [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Cursor](https://www.cursor.com/), [Codex CLI](https://github.com/openai/codex), [Gemini CLI](https://github.com/google-gemini/gemini-cli), and [OpenCode](https://github.com/opencode-ai/opencode).
  6. ## Why MLflow Skills?
  7. Building production-ready AI agents is hard. You need observability to understand what your agent is doing, evaluation to measure quality, and debugging tools when things go wrong. MLflow provides SDKs and best practices for all of these operations, and with skills we bring them directly into the environment where LLM agent development happens. Now you can go to your favorite coding agent and just ask:
  8. - *"Add tracing to my LangChain app"* → Instruments your code automatically
  9. - *"Why did this trace fail?"* → Analyzes spans, finds root causes, suggests fixes
  10. - *"Evaluate my agent's accuracy"* → Sets up datasets, scorers, and runs evaluation
  11. - *"Improve my agent and verify your work"* → Gives the coding agent reproducible and verifiable mechanism with MLflow eval to hill climb on quality
  12. - *"Show me token usage trends"* → Queries metrics and analyze trends
  13. ---
  14. ## Available Skills
  15. | Skill | Description |
  16. |-------|-------------|
  17. | **mlflow-agent** | Routes broad MLflow requests to the right skill. |
  18. | **mlflow-onboarding** | Configures MLflow for GenAI or traditional ML. |
  19. | **mlflow-docs** | Searches official MLflow documentation. |
  20. | **mlflow-sagemaker** | Connects to SageMaker Managed MLflow. |
  21. | **mlflow-trace-instrumentation** | Instruments Python and TypeScript applications. |
  22. | **mlflow-trace-retrieval** | Finds and filters traces. |
  23. | **mlflow-trace-analysis** | Analyzes traces, agent failures, and multi-turn sessions. |
  24. | **mlflow-trace-metrics** | Queries aggregate trace metrics. |
  25. | **mlflow-evaluation** | Designs scorers and runs GenAI evaluation. |
  26. | **mlflow-agent-improvement** | Fixes agent behavior with trace-grounded regression tests. |
  27. ---
  28. ## Installation
  29. ### Using `skills` installer
  30. ```bash
  31. npx skills add mlflow/skills
  32. ```
  33. ### Direct Installation from Source
  34. ```bash
  35. git clone https://github.com/mlflow/skills.git
  36. cp -r mlflow-skills/* ~/.claude/skills/
  37. ```
  38. Change the `~/.claude/skills/` directory to the appropriate location for your coding agent, e.g., `~/.codex/skills/` for Codex.
  39. ### Project-Level Installation
  40. Add skills to your project for team sharing:
  41. ```bash
  42. cd your-project
  43. git clone https://github.com/mlflow/skills.git .skills/mlflow
  44. # Or as a submodule:
  45. git submodule add https://github.com/mlflow/skills.git .skills/mlflow
  46. ```
  47. ---
  48. ## Auto-Suggestion Hook (Optional)
  49. The `hooks/` directory contains a `UserPromptSubmit` hook that automatically detects MLflow-related patterns in your prompts and surfaces the right skill before the agent responds — no need to remember which skill does what.
  50. ### Install the Hook
  51. **Step 1:** Copy the hook somewhere permanent:
  52. ```bash
  53. cp hooks/mlflow-suggest-hook.py ~/.claude/hooks/mlflow-suggest-hook.py
  54. ```
  55. **Step 2:** Add it to your Claude Code settings (`~/.claude/settings.json`):
  56. ```json
  57. {
  58. "hooks": {
  59. "UserPromptSubmit": [
  60. {
  61. "type": "command",
  62. "command": "python3 ~/.claude/hooks/mlflow-suggest-hook.py"
  63. }
  64. ]
  65. }
  66. }
  67. ```
  68. **Step 3:** Start a new session. When you ask something like *"Add tracing to my app"*, you'll see:
  69. ```
  70. 💡 Use the `mlflow-trace-instrumentation` skill to add MLflow tracing.
  71. ```
  72. See [`hooks/README.md`](hooks/README.md) for the full keyword-to-skill mapping and troubleshooting.
  73. ---
  74. ## Quick Examples
  75. ### Instrument Your App with Tracing
  76. ```
  77. > Add MLflow tracing to my OpenAI app
  78. The coding agent will:
  79. 1. Detect your LLM framework
  80. 2. Add the right autolog call
  81. 3. Configure experiment tracking
  82. 4. Verify traces are being captured
  83. ```
  84. ### Debug a Failed Trace
  85. ```
  86. > Analyze trace tr-abc123 — why did it return the wrong answer?
  87. The coding agent will:
  88. 1. Fetch the trace and examine all spans
  89. 2. Check assessments for quality signals
  90. 3. Walk the span tree to find the failure point
  91. 4. Correlate with your codebase to identify root cause
  92. 5. Suggest specific fixes
  93. ```
  94. ### Evaluate Agent Quality
  95. ```
  96. > Set up evaluation for my customer support agent
  97. The coding agent will:
  98. 1. Discover existing evaluation datasets
  99. 2. Help create test cases if needed
  100. 3. Select appropriate scorers (correctness, relevance, etc.)
  101. 4. Run evaluation and analyze results
  102. 5. Generate improvement recommendations
  103. ```
  104. ### Query Usage Metrics
  105. ```
  106. > Show me token usage trends for the last 7 days
  107. The coding agent will:
  108. 1. Query the MLflow metrics API
  109. 2. Generate time-series breakdowns
  110. 3. Identify cost spikes or anomalies
  111. 4. Suggest optimization opportunities
  112. ```
  113. ---
  114. ## Requirements
  115. - **MLflow 3.8+** (Instructions and code examples are tested with MLflow 3.8)
  116. ---
  117. ## Contributing
  118. We welcome contributions! Here's how to help:
  119. 1. **Report issues** — Found a skill giving incorrect advice? [Open an issue](https://github.com/mlflow/mlflow-skills/issues)
  120. 2. **Suggest improvements** — Have ideas for better workflows? We'd love to hear them
  121. 3. **Add new skills** — See [skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices) and file a pull request
  122. ### Testing skill behavior
  123. The test code has two layers: live scenarios exercise skills through Claude; local verifier tests check that the graders accept valid work and reject broken work.
  124. ```text
  125. tests/
  126. configs/
  127. setup/ # Onboarding, docs, backend prerequisites
  128. tracing/ # Instrumentation, retrieval, diagnosis, metrics
  129. evals/ # Evaluation, scorers, alignment, verified repair
  130. judges/ # Graders used by live scenarios
  131. scripts/ # Scenario setup and seeded application data
  132. verifiers/ # Local regression tests for graders and harness
  133. test_skill.py # Existing YAML scenario runner
  134. ```
  135. Config paths for setup scripts and judges remain relative to the repository root. Run live scenarios from the repository root using their grouped paths below; the former flat config paths have moved.
  136. The YAML scenarios in `tests/configs/` run a real Claude Code participant in a fresh project and Claude home, install the selected repository skills plus the MLflow tracing plugin, and verify its work against a real MLflow server. Each scenario checks a successful native skill invocation or complete skill read, plus an observable task outcome. Task-only prompts test discovery among the skills installed for each scenario. No prompt names a skill or instructs the agent to load one. The dispatcher currently disables automatic invocation, so its required-loading check can expose that discovery limitation. Merely mentioning an API or reporting success does not pass.
  137. Validated with MLflow 3.17.1. Requires an authenticated `claude` CLI, Git, Node/npm (for Claude tracing and TypeScript), and `uv`. Run a scenario:
  138. ```sh
  139. uv run --with mlflow==3.17.1 --with litellm --with pytest --with langgraph --with langchain-core \
  140. python tests/test_skill.py tests/configs/evals/evaluation_nested.yaml
  141. ```
  142. The original general evaluation scenario remains `agent_evaluation.yaml`: it clones `REPO_URL` and checks datasets, registered scorers, evaluation results, application traces, and both evaluation/tracing skill reads. Run it with `REPO_URL=<repository-url>` as a CLI override. The separate `evaluation_nested.yaml` is a deterministic local regression fixture.
  143. | Config field | Meaning |
  144. |---|---|
  145. | `skills` | Installed skills; all are required to load by default. |
  146. | `required_skills` | Optional subset when other installed skills are only helpers. Sent to the verifier, not the participant. |
  147. | `judges` | Checks of the participant's actions and actual application results. |
  148. | `tracking_uri` | Optional external server. Omit for a fresh local server. |
  149. | `mlflow_port` | Optional fixed-port override. Omit to select a free local port. |
  150. Default timeouts, `keep_workdir: true`, and `test_runs_dir: /tmp` need not be repeated. Fixtures contain application context, data, and actual deliverable interfaces; prompts are short user requests. Conversational answers are checked by an MLflow judge using `JUDGE_MODEL` (or its default), alongside backend and tool-evidence checks; participants need not write grading JSON. Specialized regression cases still have explicit operational constraints, such as approved smoke inputs and an unavailable backend.
  151. | Skill/workflow | Scenarios | Observable checks |
  152. |---|---|---|
  153. | Onboarding | `onboarding_existing`, `onboarding_unavailable`, `onboarding_classic_ml` | Reuse configuration and experiment; persist smoke trace; report a real connection failure without resetting setup; reload an actual sklearn model and verify held-out predictions |
  154. | Dispatcher | `dispatcher_scorer` | Load the dispatcher and scorer skill; execute a native scorer against held-out empty/missing/null/valid answers |
  155. | Documentation | `docs_classic_ml` | Fetch official docs and explain a runnable classic-ML example |
  156. | SageMaker | `sagemaker_prerequisites` | Execute prerequisite checks; stop when plugin/credentials are missing |
  157. | Instrumentation | `tracing_test`, `tracing_typescript`, `langgraph_autolog_guidance` | Rerun the delivered entrypoint and require fresh Python/TypeScript inputs and outputs; native graph/model spans without redundant tracing |
  158. | Trace retrieval | `trace_retrieval` | Exact matching IDs amid time, status and tag distractors |
  159. | Single-trace diagnosis | `trace_analysis` | Identify the actual failing child span and its timeout inputs |
  160. | Session diagnosis | `session_analysis` | Ordered turns, first incorrect answer, propagation, evidence, and failed-scorer exclusion |
  161. | Aggregate metrics | `trace_metrics` | Exact counts and token sums from the real metrics endpoint; distinguish unsupported endpoints from other failures |
  162. | Diagnosis versus repair | `diagnosis_only`, `agent_debug_trace_first` | Diagnosis preserves source; repair follows trace evidence and verifies refund/general routing |
  163. | Evaluation | `evaluation_nested`, `evaluation_precomputed`, `evaluation_dry_run` | Nested predictor kwargs; exact six-row scores and mean; reuse dataset/scorer; never rerun precomputed outputs; verify a scored dry run precedes full evaluation; stop after a failed three-row dry run |
  164. | Scorer design | `scorer_deterministic_suite`, `scorer_model_criteria` | Execute independent criteria on private malformed/near-miss cases; code, built-in and domain judges; preserve model selection |
  165. | Judge alignment | `scorer_label_alignment`, `scorer_memalign` | Criterion-specific human labels and train/holdout splits; missing-provider blocker; saved native MemAlign training IDs, participant holdout comparison, and separate verifier holdouts |
  166. Model-backed scenarios require working provider credentials. Set `TEST_JUDGE_MODEL` for `scorer_model_criteria`; for `scorer_memalign`, also set `TEST_EMBEDDING_MODEL` and add `--with dspy`. The repair scenario's existing LLM judge accepts `JUDGE_MODEL`. No credentials or private endpoint names belong in scenario files. Each run retains `environment.json`, `skill-inventory.json`, `verification.json`, participant transcripts, generated files, and the local MLflow store in its printed work directory. Setup failures and skill failures both return nonzero, with different exit codes.
  167. To run the local scenarios sequentially, excluding those needing a model provider:
  168. ```sh
  169. for config in tests/configs/*/*.yaml; do
  170. case "$config" in
  171. *scorer_model_criteria*|*scorer_memalign*|*agent_debug_trace_first*|*agent_evaluation*) continue ;;
  172. esac
  173. uv run --with mlflow==3.17.1 --with litellm --with pytest --with langgraph --with langchain-core \
  174. python tests/test_skill.py "$config" || exit 1
  175. done
  176. ```
  177. Supporting verifier regression checks run without Claude or model calls:
  178. ```sh
  179. uv run --with mlflow==3.17.1 --with pytest --with langgraph --with langchain-core --with dspy \
  180. pytest -q tests/verifiers
  181. ```
  182. Expected answers are stored outside the project but remain accessible on the same machine; this is not an adversarial sandbox. Investigation grading also requires observed MLflow retrieval. These scenarios are behavioral regression coverage, not a claim of exhaustive platform coverage or a baseline-versus-treatment comparison. Live SageMaker connection, Databricks UC storage/Review App/online monitoring, GEPA prompt optimization, distributed tracing, and every supported framework still require dedicated environments and scenarios. A successful local run does not verify those integrations. The repository currently has no CI workflow that launches the paid Claude scenarios automatically.
  183. ### Creating a New Skill
  184. We recommend using the [skill-creator](https://github.com/anthropics/skills/tree/main/skills/skill-creator) from Anthropic's skills repository as a starting point.
  185. ---
  186. ## License
  187. Apache License 2.0 — see [LICENSE](LICENSE) for details.
  188. ---
  189. ## Links
  190. - 📖 [Read MLflow Documentation](https://mlflow.org/docs/latest/)
  191. - 📖 [Read MLflow Skills Blog](https://mlflow.org/blog/evaluating-skills-mlflow))
  192. - ⭐️ [Star MLflow on GitHub](https://github.com/mlflow/mlflow)
  193. ---
  194. <p align="center">
  195. <sub>Built with care for the MLflow community</sub>
  196. </p>

Version history

  • v1.0.010/27/2026

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.