Skip to content
Back to search

Tracely-ai

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/Jwuthri/Tracely-ai with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/Jwuthri/Tracely-ai'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/jwuthri--tracely-ai/badge.svg)](/listings/jwuthri--tracely-ai)

HTML

<a href="/listings/jwuthri--tracely-ai"><img src="/api/listings/jwuthri--tracely-ai/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Prerequisites:** Docker + Docker Compose. (For local dev also [uv](https://docs.astral.sh/uv/) and Node 20+ / pnpm.)

Everything in Docker — the whole product, populated

git clone https://github.com/Jwuthri/Tracely-ai && cd Tracely
docker compose --profile demo up -d --build --wait
open http://localhost:3001

That brings up ClickHouse, Postgres, Redis and MinIO, runs every migration, seeds the default project and ingest key (`tracely_dev_key`), then populates **traces, clusters, cases and gates** — so the app opens with the screenshots above rather than an empty shell.

docker compose down            # stop  (add -v to wipe data)

Host ports default to web **:3001** and backend **:8000**; remap with `TRACELY_WEB_PORT` / `TRACELY_BACKEND_PORT`. `backend`/`worker`/`frontend` run off source volume-mounts, so most edits need only `docker compose restart <svc>` — **except** the Celery worker, which doesn't hot-reload.

Local dev (hot reload)

cp .env.example .env
make infra-up      # clickhouse, postgres, redis, minio
make install       # uv sync + pnpm install
make migrate       # ClickHouse DDL + Alembic (Postgres)
make seed          # default project + ingest key → tracely_dev_key

make backend       # FastAPI  :8000  (OpenAPI at /docs)   ┐
make workers       # Celery ingestion/eval worker          ├ three terminals
make frontend      # Next.js  :3001                        ┘

make demo          # populate the WHOLE product: traces + clusters + cases + gates
make test          # backend unit tests (no infra, ~6s)

Deploy your own

One click provisions the whole stack on [Railway](https://railway.com) — API, worker, UI, Postgres (pgvector), ClickHouse, Redis and MinIO, wired together with volumes and private networking. Migrations and seeding run on the first deploy; set `SESSION_SECRET` and `SECRETS_ENCRYPTION_KEY` (`openssl rand -hex 32` each) when prompted, then open the frontend's domain and create your workspace.

Prefer to wire it yourself, or deploying somewhere else? The manual walkthrough is [`deploy/railway/README.md`](deploy/railway/README.md) (every variable pre-written in [`.env.railway.example`](deploy/railway/.env.railway.example)), and the production-hardening runbook — auth guards, backups, worker pool, post-deploy verification — is [`guides/DEPLOY.md`](guides/DEPLOY.md).

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 556 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. <img src="frontend/app/icon.svg" width="76" alt="Tracely" />
  3. # Tracely
  4. ### Production failures become regression tests.
  5. **Trace-native CI/CD for AI agents.** Tracely grades every agent trace as it lands, clusters the
  6. failures into issues, freezes the bad runs into hermetic replayable cases, blocks the pull request
  7. that would ship them again — and tells you the moment any of it happens.
  8. <p>
  9. <code>production trace</code> → <code>failure detection</code> → <code>regression test</code> → <code>CI gate</code> → <code>alert</code>
  10. </p>
  11. [**Website**](https://tracely-ai.com) · [**Docs**](https://doc.tracely-ai.com) · [**Product guide**](https://doc.tracely-ai.com/product) · [**Agent skill**](#teach-your-coding-agent-tracely) · [**Guided tour**](guides/OVERVIEW.md) · [**2-min demo**](guides/DEMO.md) · [**Design dossier**](design/README.md)
  12. [![CI](https://github.com/Jwuthri/Tracely-ai/actions/workflows/ci.yml/badge.svg)](https://github.com/Jwuthri/Tracely-ai/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/tracely-ai?logo=pypi&logoColor=white)](https://pypi.org/project/tracely-ai/) [![Python](https://img.shields.io/badge/python-3.10%2B-blue?logo=python&logoColor=white)](https://pypi.org/project/tracely-ai/) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Stars](https://img.shields.io/github/stars/Jwuthri/Tracely-ai?style=flat&logo=github)](https://github.com/Jwuthri/Tracely-ai/stargazers) [![MCPVault: claimed](https://mcpvault.io/badge/tracely.svg?theme=dark)](https://mcpvault.io/servers/tracely/health?utm_source=external_badge&utm_medium=referral&utm_campaign=mcp_health_report)
  13. **Self-host the whole stack in one click** — API, worker, UI, Postgres, ClickHouse, Redis and MinIO:
  14. [![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/n5n_LE?referralCode=WCq5Cn&utm_medium=integration&utm_source=template&utm_campaign=generic)
  15. [![Deploy to Railyard](https://app.railyard.run/deploy/badge)](https://app.railyard.run/deploy?repo=https://github.com/Jwuthri/Tracely-ai)
  16. <br />
  17. <img src=".github/assets/dashboard.png" alt="Tracely dashboard — trace and failure counts, the biggest failure clusters, recent traces and regression cases" width="100%" />
  18. <sub><i>One workspace: what ran, what broke, what is already pinned as a test.</i></sub>
  19. </div>
  20. ---
  21. ## Why another agent-observability tool?
  22. Because observability stops at the dashboard. You can see that your agent broke — then what?
  23. Every eval tool asks you to **hand-author a dataset**: sit down, invent questions, write ideal
  24. answers, keep them current as the product changes. That dataset is a guess about what might break.
  25. Production already handed you the real thing: a trace of the exact run that failed, with the exact
  26. input, the exact tool calls, the exact model responses.
  27. > **The recorded run _is_ the test.** Tracely freezes that trace into a hermetic regression case and
  28. > replays it on every PR. Everything else — quality scores, failure clusters, suggested fixes, CI
  29. > verdicts, trends, alerts — is **derived from the trace**. There are no hand-authored datasets.
  30. | | Dataset-first tools | Tracely |
  31. |---|---|---|
  32. | Where tests come from | You write them | Promoted from real failing traces |
  33. | Fidelity to production | A guess | The exact failing run, byte for byte |
  34. | Cost to replay in CI | Live model calls | **$0** — recorded tool/LLM fixtures |
  35. | What happens on regression | A dashboard number moves | **The PR is blocked** |
  36. | How you find out | You go and look | **It comes to you** — Slack, email, your own webhook |
  37. ---
  38. ## The loop
  39. Five steps, five pages in the app.
  40. ### 1 · Observe — every run, hierarchically
  41. Traces arrive over plain OTLP. Agent semantics (`agent.id`, `conversation.id`, `turn`, `step`) are
  42. promoted to first-class indexed columns, so runs group into conversation threads instead of a flat
  43. span soup. The waterfall shows agent → thinking → skill → generation → hand-off, with the failing
  44. span in red and its I/O beside it.
  45. <img src=".github/assets/trace-timeline.png" alt="Trace timeline — agent, thinking, skill, generation and delegate spans, with a nested sub-agent and the span's input" width="100%" />
  46. Evaluators are **columns on the trace table**, not a separate tab — each one grades at conversation,
  47. run or span level and writes its verdict into the grid. Scores stream in live over SSE as judges
  48. finish, so you watch a run get graded in place. Tokens, cost, latency, metadata and the rolling
  49. per-turn summary are columns too.
  50. <img src=".github/assets/traces.png" alt="Trace explorer — conversation threads with agent, duration, rolling summary, metadata and usage columns" width="100%" />
  51. <details>
  52. <summary><b>Two extra ways to read one conversation</b> — a step-by-step replay, and a pixel-art office</summary>
  53. <br />
  54. **Replay** walks the conversation event by event on a timeline. **Fleet** renders the same
  55. conversation as a room: each agent is a character, hand-offs are walks across the floor, tools are
  56. objects on the wall — which turns out to be the fastest way to explain a multi-agent system to
  57. someone who doesn't read waterfalls.
  58. <img src=".github/assets/fleet.png" alt="Fleet view — a conversation rendered as a pixel-art office with one character per agent" width="100%" />
  59. </details>
  60. ### 2 · Detect & triage — failures group into issues
  61. Online evaluators grade every run as it lands (LLM-as-judge at conversation / run / span level, plus
  62. structural checks that need no model at all). Failures then cluster — structurally and semantically —
  63. so 31 broken runs become **one issue with a count**, not 31 rows to read. Each cluster can suggest
  64. the evaluator that would have caught it.
  65. <img src=".github/assets/clusters.png" alt="Failure clusters — auto-detected failures grouped into issues with occurrence counts and taxonomy" width="100%" />
  66. ### 3 · Test — freeze the failure
  67. One click promotes a failing trace into a hermetic case: recorded input, tool and LLM outputs bundled
  68. as fixtures, and a **fail-to-pass contract** attached — the case must fail on the old code and pass on
  69. the fix, or the promotion isn't trusted. Multi-turn behaviour gets **scenarios** instead: a scripted
  70. conversation, or an adversarial goal a red-team model improvises against.
  71. <img src=".github/assets/case.png" alt="Regression case — promoted, fail-to-pass validated, with assertions and the reference trajectory" width="100%" />
  72. ### 4 · Ship — block the PR
  73. The suite replays in CI against recorded fixtures: deterministic, offline, **no API keys and no model
  74. spend**. `tracely gate` exits non-zero, posts a commit status, and upserts a PR comment.
  75. <img src=".github/assets/gate-run.png" alt="A failing CI gate run — FAIL, per-case verdicts and the judge's reason" width="100%" />
  76. ### 5 · Tell me — alerts as a flow
  77. A rule has two halves: **when** (a gate fails, a live conversation breaks on a judge, a failure mode
  78. appears that nobody has seen before, or a rate crosses a line) and **what happens** — drawn on a
  79. canvas. Conditions that gate the rest of the flow, Slack, email, a webhook with your own method and
  80. headers, an **LLM step** whose answer the next step can use, and a Python expression for a number a
  81. template can't compute. Every field is a template over the failure's own variables, dragged in as
  82. chips. Or describe it to the in-app assistant and it draws the whole thing onto the canvas.
  83. <img src=".github/assets/alerts-flow.png" alt="The alert flow builder — When → condition → LLM prompt → Slack, with the inspector's input chips, step config and declared outputs" width="100%" />
  84. ### Plus — trends, cross-metric analysis, judge calibration
  85. Daily failure and gate pass-rates, latency percentiles, token spend, per-agent meta-analysis
  86. (Spearman correlations + z-score outliers, LLM-synthesized) — and a calibration screen where you
  87. label judge verdicts against human review and see each judge's agreement, so you catch an
  88. over-flagging judge *before* you let it gate a release.
  89. <img src=".github/assets/trends.png" alt="Trends — failure rate, gate pass-rate, latency percentiles, token and cost metrics over time" width="100%" />
  90. ---
  91. ## Everything in the box
  92. | | What it does | Where |
  93. |---|---|---|
  94. | **Ingest** | OTLP/HTTP from any language; blob-first durability (S3 before the queue); agent semantics as indexed columns; three message conventions normalized | [`backend/`](backend/README.md) |
  95. | **Evaluators** | Columns on the trace table. Structural checks (no model) + LLM judges at conversation / run / span level, multi-output (score, boolean, number, text, JSON schema), basic or `@VARIABLE` advanced prompts with live preview, batch or sequential, per-agent/env targeting, deterministic sampling, advisory verdicts | [Docs](https://doc.tracely-ai.com/product/traces) |
  96. | **Failure clusters** | Structural signature + semantic embedding clustering, taxonomy, suggested evaluators, promote-to-case | [Docs](https://doc.tracely-ai.com/product/clusters) |
  97. | **Regression cases** | Hermetic fixture bundles, fail-to-pass contracts, assertions, reference trajectory, replay history | [Docs](https://doc.tracely-ai.com/product/cases) |
  98. | **Scenarios** | Multi-turn conversations Tracely drives at your agent's own endpoint, or an adversarial goal a red-team model pursues (goal achieved = FAIL) | [Docs](https://doc.tracely-ai.com/product/scenarios) |
  99. | **CI gates** | `tracely simulate` / `replay` / `gate` + a GitHub Action: commit status, PR comment, non-zero exit | [Docs](https://doc.tracely-ai.com/product/gates) |
  100. | **Alerts** | Trigger + visual flow: conditions, Slack, email, webhooks with headers, an LLM step, Python expressions. Event triggers fire inline; thresholds poll a window. Test runs show what each step actually sent | [Docs](https://doc.tracely-ai.com/product/alerts) |
  101. | **Trends & analysis** | Daily traces/failures/gate pass-rate, latency + cost, per-agent cross-metric meta-analysis | [Docs](https://doc.tracely-ai.com/product/trends) |
  102. | **Judge calibration** | Label judge verdicts against human review; per-evaluator agreement, missed failures vs over-flagging | [Docs](https://doc.tracely-ai.com/product/calibration) |
  103. | **Conversation intelligence** | Rolling per-turn summary (backs the judge's `@HISTORY`), declared agent catalog, state deltas, Replay + Fleet views, shareable read-only links | [Docs](https://doc.tracely-ai.com/product/traces) |
  104. | **In-app assistant** | ⌘J — an agent over *your* workspace: reads traces, creates evaluators, scenarios, cases and alerts, as you. ⌘K jumps anywhere | [Docs](https://doc.tracely-ai.com/product/assistant) |
  105. | **MCP server** | The API doubles as an MCP endpoint (`/mcp`) so a coding agent drives the workspace with an ordinary ingest key | [Docs](https://doc.tracely-ai.com/mcp) |
  106. | **Auth & teams** | `dev` (open) · `local` (email/password + invites) · `clerk`. Organizations, workspaces, seats, API keys, per-workspace OpenRouter key | [Docs](https://doc.tracely-ai.com/product/settings) |
  107. | **Ops** | Beat self-check (queue depth, worker liveness, ingest freshness) + `/health/queue`, optional Sentry, optional trace quota + Stripe billing | [`guides/DEPLOY.md`](guides/DEPLOY.md) |
  108. Near-term plan: [design/part2-tracely/11-prd-next-steps.md](design/part2-tracely/11-prd-next-steps.md) ·
  109. Long-term roadmap: [10-mvp-and-roadmap.md](design/part2-tracely/10-mvp-and-roadmap.md)
  110. ---
  111. ## Quickstart
  112. **Prerequisites:** Docker + Docker Compose. (For local dev also [uv](https://docs.astral.sh/uv/) and Node 20+ / pnpm.)
  113. ### Everything in Docker — the whole product, populated
  114. ```bash
  115. git clone https://github.com/Jwuthri/Tracely-ai && cd Tracely
  116. docker compose --profile demo up -d --build --wait
  117. open http://localhost:3001
  118. ```
  119. That brings up ClickHouse, Postgres, Redis and MinIO, runs every migration, seeds the default project
  120. and ingest key (`tracely_dev_key`), then populates **traces, clusters, cases and gates** — so the app
  121. opens with the screenshots above rather than an empty shell.
  122. ```bash
  123. docker compose down # stop (add -v to wipe data)
  124. ```
  125. Host ports default to web **:3001** and backend **:8000**; remap with `TRACELY_WEB_PORT` / `TRACELY_BACKEND_PORT`.
  126. `backend`/`worker`/`frontend` run off source volume-mounts, so most edits need only `docker compose restart <svc>` —
  127. **except** the Celery worker, which doesn't hot-reload.
  128. ### Local dev (hot reload)
  129. ```bash
  130. cp .env.example .env
  131. make infra-up # clickhouse, postgres, redis, minio
  132. make install # uv sync + pnpm install
  133. make migrate # ClickHouse DDL + Alembic (Postgres)
  134. make seed # default project + ingest key → tracely_dev_key
  135. make backend # FastAPI :8000 (OpenAPI at /docs) ┐
  136. make workers # Celery ingestion/eval worker ├ three terminals
  137. make frontend # Next.js :3001 ┘
  138. make demo # populate the WHOLE product: traces + clusters + cases + gates
  139. make test # backend unit tests (no infra, ~6s)
  140. ```
  141. ### Deploy your own
  142. [![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/n5n_LE?referralCode=WCq5Cn&utm_medium=integration&utm_source=template&utm_campaign=generic)
  143. One click provisions the whole stack on [Railway](https://railway.com) — API, worker, UI, Postgres
  144. (pgvector), ClickHouse, Redis and MinIO, wired together with volumes and private networking.
  145. Migrations and seeding run on the first deploy; set `SESSION_SECRET` and `SECRETS_ENCRYPTION_KEY`
  146. (`openssl rand -hex 32` each) when prompted, then open the frontend's domain and create your
  147. workspace.
  148. **Self-hosting on [Coolify](https://coolify.io)?** One command sets up the same layout, one card
  149. per component with its own logs, metrics and CPU/RAM limits:
  150. `python3 deploy/coolify/setup.py --coolify-url … --app-domain … --api-domain …`. See
  151. [`deploy/coolify/README.md`](deploy/coolify/README.md), which also covers the single-bundle
  152. [`docker-compose.coolify.yml`](docker-compose.coolify.yml).
  153. Prefer to wire it yourself, or deploying somewhere else? The manual walkthrough is
  154. [`deploy/railway/README.md`](deploy/railway/README.md) (every variable pre-written in
  155. [`.env.railway.example`](deploy/railway/.env.railway.example)). On AWS EKS or any other Kubernetes cluster,
  156. [`deploy/eks/README.md`](deploy/eks/README.md) runs the whole stack in one namespace with no cloud IAM.
  157. The production-hardening runbook
  158. — auth guards, backups, worker pool, post-deploy verification — is
  159. [`guides/DEPLOY.md`](guides/DEPLOY.md).
  160. ---
  161. ## Send your first trace
  162. ```bash
  163. pip install "tracely-ai[openai]" # or [anthropic], [langchain], [all]
  164. ```
  165. Initialize once at startup, then wrap a run — your normal provider calls are captured automatically,
  166. with no span code:
  167. ```python
  168. import tracely_sdk as tracely
  169. tracely.init(
  170. endpoint="http://localhost:8000", # your Tracely API
  171. api_key="tracely_dev_key", # an ingest key
  172. service_name="support-agent",
  173. env="prod", # prod | staging | ci | dev — the gating axis
  174. instrument="auto", # auto-detect openai / anthropic / google / mistral / langchain
  175. )
  176. with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
  177. client.chat.completions.create(
  178. model="gpt-4o",
  179. messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
  180. )
  181. ```
  182. That produces a **GENERATION** span with model, messages, tokens, latency, tool calls and cost.
  183. Need spans for your own logic? `@observe` and the manual `agent` / `tool` / `llm` / `retriever` /
  184. `guardrail` context managers are all there.
  185. Any OTLP/HTTP exporter works too — point it at `POST {endpoint}/v1/traces` with
  186. `Authorization: Bearer tracely_dev_key`. Tracely reads standard `gen_ai.*` / OpenInference attributes
  187. plus first-class hints: `tracely.agent.id` (auto-registered), `tracely.agent.version`,
  188. `tracely.conversation.id` / `turn.*` / `step.*`, `tracely.observation.type`, and `tracely.env`
  189. (`prod|staging|ci|dev` — the gating axis).
  190. <details>
  191. <summary><b>Declare your agents and record your state</b> — two optional lines that make a conversation self-describing</summary>
  192. <br />
  193. **The agent catalog** tells Tracely which agents, tools, prompts and models the conversation *has*
  194. (not just which ones fired) — it fills the **Conversation Agents** panel and the Fleet view, and is
  195. readable from judge prompts as `@AGENTS`. **State deltas** record what each step wrote to your
  196. shared state, folded into the **Conversation State** drawer and the per-message **State Δ** column:
  197. ```python
  198. AGENTS = [{
  199. "name": "support",
  200. "description": "front-line agent; routes billing questions",
  201. "system_prompt": "You are the support agent for Acme…", # free-form keys kept verbatim
  202. "model": "gpt-5.2",
  203. "tools": {"lookup_order": {"name": "lookup_order", "description": "order by id",
  204. "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}}}},
  205. }]
  206. with tracely.trace(agent="support", conversation="conv-1", agents=AGENTS):
  207. ...
  208. tracely.set_state({"cart": cart, "last_action": "add_to_cart"}) # inside any span/@observe
  209. ```
  210. LangGraph users get state for free (node outputs are captured as deltas automatically), and
  211. non-Python services can push the catalog with `POST /api/sessions/{conversation_id}/config`.
  212. </details>
  213. Full instrumentation guide → **[doc.tracely-ai.com](https://doc.tracely-ai.com)** · [`sdk/README.md`](sdk/README.md)
  214. ---
  215. ## Gate your PRs
  216. A promoted production failure becomes a regression test that **blocks the PR** that reintroduces it.
  217. All you need is your ingest key (it identifies your workspace) and your Tracely API URL. Three ways to
  218. wire it, depending on how your CI can reach your agent.
  219. <details open>
  220. <summary><b>Option A — let Tracely call your agent</b> (no agent code in CI, any language)</summary>
  221. Register your agent's HTTP endpoint once, author [scenarios](https://doc.tracely-ai.com/product/scenarios) —
  222. multi-turn conversations, or an adversarial goal a red-team model improvises against — and Tracely
  223. drives them itself. **Nothing to install, import or shim**, so a TypeScript or Go service gates exactly
  224. like a Python one.
  225. ```yaml
  226. - uses: Jwuthri/Tracely-ai/.github/actions/tracely-gate@master
  227. with:
  228. api: https://tracely.your-co.dev
  229. key: ${{ secrets.TRACELY_KEY }}
  230. # agent: planner,support-agent ← a subset; omit it to gate EVERY agent with scenarios
  231. ```
  232. Scenarios belong to an agent, so leaving `agent` blank gates each one in its own run and fails the job
  233. if any of them fails — a new agent is covered the day someone writes its first scenario.
  234. </details>
  235. <details>
  236. <summary><b>Option B — gate the traces your CI already emits</b></summary>
  237. If your pipeline already runs your agent instrumented with `tracely.env=ci`, the gate matches those
  238. traces to your promoted cases (by input) and returns PASS/FAIL.
  239. ```yaml
  240. # .github/workflows/tracely.yml
  241. name: Tracely gate
  242. on: pull_request
  243. permissions:
  244. contents: read
  245. statuses: write # post the blocking commit status
  246. pull-requests: write # upsert the results comment
  247. jobs:
  248. gate:
  249. runs-on: ubuntu-latest
  250. steps:
  251. - uses: actions/checkout@v4
  252. # → your existing step(s) that run the agent and emit env=ci traces go here ←
  253. - uses: Jwuthri/Tracely-ai/.github/actions/tracely-gate@master
  254. with:
  255. mode: gate # grade the ci traces this workflow emitted
  256. agent: planner # which agent's promoted suite to run
  257. api: https://tracely.your-co.dev # your Tracely backend (TRACELY_API)
  258. key: ${{ secrets.TRACELY_KEY }} # your ingest key = your workspace
  259. ```
  260. </details>
  261. <details>
  262. <summary><b>Option C — replay recorded cases against your code</b> (hermetic, $0)</summary>
  263. Re-runs your agent on each promoted case's *recorded input*, serving the recorded tool/LLM outputs as
  264. fixtures — deterministic, offline, **no API keys, no cost** — then gates. Guarantees the exact failing
  265. inputs are tested.
  266. ```yaml
  267. - run: pip install tracely-ai
  268. - run: tracely replay planner --entrypoint my_pkg.agent:run # a Python agent
  269. # …or any language: tracely replay planner --cmd "node run.js" (reads $TRACELY_INPUT)
  270. env:
  271. TRACELY_API: https://tracely.your-co.dev
  272. TRACELY_KEY: ${{ secrets.TRACELY_KEY }}
  273. ```
  274. Hermetic replay requires your agent to route tool/model calls through the SDK's `call_tool` /
  275. `call_llm` seam (see [the SDK guide](sdk/README.md)); add `--live` to make real calls instead.
  276. </details>
  277. Both commands auto-detect the PR/commit from the Actions context; `web-url` / `TRACELY_WEB_URL` is
  278. optional and only builds the "view gate run" link in the PR comment.
  279. > ℹ️ The repo's own [`.github/workflows/tracely-gate.yml`](.github/workflows/tracely-gate.yml) is
  280. > Tracely **dogfooding itself** — it replays the bundled `weather_agent` example, which is why it uses
  281. > an in-repo `pip install ./sdk`. **Your** integration is one of the three options above, not that file.
  282. ---
  283. ## Get told, don't go looking
  284. Settings → **Alerts**. Pick what fires the rule, then draw what happens.
  285. | Trigger | Fires when |
  286. |---|---|
  287. | `CI gate failed` | a gate run finished `FAIL` — or `NO_COVERAGE`, the suite that *could not run*, which is the quietly-green failure nobody notices |
  288. | `Conversation failed` | a live turn failed a non-advisory evaluator. Filter by evaluator, or by the text of the judge's own reason |
  289. | `New failure mode` | a failure signature no trace has produced before — the alert you cannot get from logs |
  290. | `Evaluator FAIL rate` / `Average score` / `Overall failure rate` | a rate or average crosses a line over a sliding window |
  291. Steps: **condition** (a Jinja gate that stops the rest of the flow), **Slack**, **email**,
  292. **webhook** (any verb, your own headers — `Authorization: Bearer …` — and a templated JSON body),
  293. **LLM prompt** (with declared output fields the next step reads as `{{ steps[0].result.field }}`),
  294. **Python expression**. **Run test** executes the flow against a real recent failure and shows what
  295. each step actually sent, after templating.
  296. <img src=".github/assets/alerts.png" alt="The alerts list — a gallery of use cases, and rules showing their trigger and their flow as a strip of steps" width="100%" />
  297. The same rules are reachable from the [in-app assistant](https://doc.tracely-ai.com/product/assistant)
  298. (*"Slack me when the refund judge fails a conversation"*) and from the MCP server, so an agent can
  299. arm one for you. [Alerts guide →](https://doc.tracely-ai.com/product/alerts)
  300. ---
  301. ## Drive it from your editor
  302. Every backend serves an [MCP](https://modelcontextprotocol.io) endpoint at `/mcp`, so a coding agent
  303. reads your traces and writes your evaluators without any glue code:
  304. ```bash
  305. claude mcp add --transport http tracely http://localhost:8000/mcp \
  306. --header "Authorization: Bearer tracely_dev_key"
  307. ```
  308. Then: *"look at the last 20 traces, find what's failing, and add an evaluation column that catches
  309. it."* Tools over traces, failure clusters, evaluators and trends — scoped to the key's workspace,
  310. same as every other call. On hosted Tracely the endpoint is
  311. `https://api.tracely-ai.com/mcp`. [Docs](https://doc.tracely-ai.com/mcp)
  312. ---
  313. ## Teach your coding agent Tracely
  314. MCP gives your agent your *data*. The **Tracely skill** gives it the *know-how* — how to instrument,
  315. what to evaluate, and how to wire the gate — so "add Tracely to this agent" is one sentence instead
  316. of a docs tab.
  317. ```bash
  318. npx skills add https://github.com/Jwuthri/Tracely-ai --skill tracely
  319. ```
  320. Works with Claude Code, Cursor, Copilot, Antigravity and anything else the
  321. [`skills`](https://github.com/vercel-labs/skills) CLI supports — add `-g` for a global install,
  322. `--agent '*'` for every agent on the machine.
  323. <table>
  324. <tr><td width="50%">
  325. **Knows the whole surface**
  326. - **Automatic tracing** — `init(instrument="auto")`, provider + framework extras, `@observe`,
  327. drop-ins, LangGraph, LiteLLM, agent SDKs, redaction
  328. - **Manual spans** — every observation type, handoffs, RAG, state deltas, multimodal I/O
  329. - **Non-Python** — the OTLP conventions to emit from TypeScript, Go or Ruby
  330. - **Evaluators** — structural vs judge, levels, `@VARIABLE` templates, advisory verdicts, sampling
  331. - **CI gate** — scenarios, red-team runs, hermetic replay, the GitHub Action
  332. - **Troubleshooting** — symptom → cause → fix for the failures that look like success
  333. </td><td width="50%">
  334. **And the traps that silently produce a useless workspace**
  335. - a missing `conversation` id turns one thread into twelve orphan rows
  336. - a swallowed tool error is invisible to detection, clustering *and* the gate
  337. - no `flush()` and a script loses its last spans
  338. - a dropped `traceparent` makes the gate blind to what your agent *did*
  339. - an adversarial scenario is **inverted** — goal achieved means the attack won
  340. </td></tr>
  341. </table>
  342. Prefer to read it yourself? It's plain Markdown: [`skills/tracely/`](skills/tracely/SKILL.md).
  343. ---
  344. ## Architecture
  345. The write path deliberately mirrors Langfuse's proven design — reimplemented in Python, with agent
  346. semantics promoted to **first-class indexed columns** (Langfuse keeps them as read-time strings):
  347. ```
  348. SDK/OTLP → POST /v1/traces → S3 blob (durable FIRST) → Redis/Celery
  349. → worker: otel mapping → registry upsert → ClickHouse events
  350. → evaluate_run_task → scores + structural clustering → alert flows
  351. ```
  352. | Layer | Tech | Where |
  353. |---|---|---|
  354. | Backend (API + domain) | **FastAPI** + Pydantic v2 | [`backend/`](backend/README.md) |
  355. | Workers | **Celery + Redis** | [`workers/`](workers/README.md) |
  356. | Traces + scores (OLAP) | **ClickHouse** (`ReplacingMergeTree`) | `backend/tracely/infrastructure/clickhouse/ddl` |
  357. | Registry (OLTP) | **Postgres + pgvector** + SQLAlchemy 2.0 + Alembic | `backend/migrations` |
  358. | Queue / Blobs | **Redis** / **MinIO·S3** (blob-first, source of truth) | — |
  359. | Frontend | **Next.js 15** (App Router) + Tailwind + React Flow | [`frontend/`](frontend/README.md) |
  360. | SDK + CI gate CLI | **`tracely-ai`** (OTel wrapper + `tracely` CLI) | [`sdk/`](sdk/README.md) |
  361. | Tooling | **uv** workspace (Python) · **pnpm** (web) | — |
  362. One deliberate adaptation: ClickHouse server-side `async_insert` instead of an in-process write buffer
  363. (Celery tasks don't share memory). [Why](design/part2-tracely/01-steal-and-do-not-copy.md)
  364. ### Repo map — each folder has its own detailed README
  365. | Folder | What's inside |
  366. |---|---|
  367. | [`backend/`](backend/README.md) | The `tracely` package: FastAPI API + shared domain (OTLP mapping, ClickHouse/Postgres/S3, registry, evaluators, failure intelligence, regression, gate, alert flows, auth, Celery tasks). |
  368. | [`workers/`](workers/README.md) | The deployable Celery worker runtime. |
  369. | [`frontend/`](frontend/README.md) | The Next.js web app — trace explorer, clusters, cases, gates, trends, alerts builder, settings, auth. |
  370. | [`sdk/`](sdk/README.md) | The Python SDK (instrument agents over OTLP, hermetic record-replay) + the `tracely` CI gate CLI. |
  371. | [`docs/`](docs/README.md) | The published docs site (Nextra) — SDK reference **and** the per-screen product guide. `make docs` → :3002. |
  372. | [`skills/`](skills/tracely/SKILL.md) | The Tracely agent skill — `npx skills add https://github.com/Jwuthri/Tracely-ai --skill tracely`. |
  373. | [`scripts/`](scripts/README.md) | Dev/demo helpers (raw-OTLP sender, one-command `seed_demo.py`, gate shim). |
  374. | [`design/`](design/README.md) | The full design dossier — reverse-engineered Langfuse + every Tracely design decision. |
  375. ---
  376. ## Key environment variables
  377. | Variable | Default | Purpose |
  378. |---|---|---|
  379. | `AUTH_MODE` | `dev` | `dev` (open, no login) · `local` (email/password, self-host) · `clerk` (hosted SaaS). |
  380. | `SESSION_SECRET` | — | Required when `AUTH_MODE=local`: HS256 signing key for JWTs (≥32 chars). |
  381. | `SECRETS_ENCRYPTION_KEY` | — | Encrypts a workspace's own OpenRouter key at rest (≥32 chars). |
  382. | `CLERK_ISSUER` | — | Required when `AUTH_MODE=clerk`. |
  383. | `OPENROUTER_API_KEY` | — | Server-wide LLM access for judges, failure intelligence, meta-analysis. Workspaces can bring their own key instead. |
  384. | `RESEND_API_KEY` | — | Transactional email: invites, password resets, and the alert flow's email step. |
  385. | `ALLOW_PRIVATE_URLS` | unset | Whether outbound calls (agent endpoints, alert webhooks) may target private addresses. Unset = allowed everywhere except prod. |
  386. | `TRACELY_BACKEND_PORT` | `8000` | Backend host port (Docker compose override). |
  387. | `TRACELY_WEB_PORT` | `3001` | Frontend host port (Docker compose override). |
  388. With no LLM key at all the pipeline still runs — judges, failure intelligence and meta-analysis
  389. degrade rather than crash. Full list: [`backend/tracely/config.py`](backend/tracely/config.py).
  390. ---
  391. ## Contributing
  392. Issues and PRs welcome. Before pushing:
  393. ```bash
  394. uv run pytest -q backend/tests sdk/tests # what CI runs
  395. uv run ruff check . && uv run ruff format .
  396. cd frontend && pnpm test && pnpm build # vitest + tsc typecheck + lint
  397. ```
  398. ## License
  399. [MIT](LICENSE) © Julien Wuthrich
  400. <div align="center">
  401. **[tracely-ai.com](https://tracely-ai.com)** · [Docs](https://doc.tracely-ai.com) · [Product guide](https://doc.tracely-ai.com/product) · [Star on GitHub](https://github.com/Jwuthri/Tracely-ai)
  402. If Tracely is useful to you, a ⭐ helps other people find it.
  403. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.