Skip to content
Back to search

data-factory

MarketplaceWorks with Claude CodeGitHub

Open-source AI data team: AI agents for data engineering, analytics, data science & BI. One data person + 5 agents, every number verified. Claude Code plugin.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nimrodfisher--data-factory/badge.svg)](/listings/nimrodfisher--data-factory)

HTML

<a href="/listings/nimrodfisher--data-factory"><img src="/api/listings/nimrodfisher--data-factory/badge.svg" alt="findsafeskills safety rating" /></a>

Install

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 21 lines
  1. {
  2. "name": "data-factory",
  3. "owner": {
  4. "name": "Nimrod Fisher",
  5. "url": "https://github.com/nimrodfisher"
  6. },
  7. "metadata": {
  8. "description": "Run a full AI data team — data engineer, analyst, data scientist, BI engineer and QA inspector — as one person.",
  9. "version": "0.1.0"
  10. },
  11. "plugins": [
  12. {
  13. "name": "data-factory",
  14. "source": "./plugins/data-factory",
  15. "description": "The software-factory model for data: specs in, verified data products out. 5 specialist agents, holdout checks, human gates, any warehouse.",
  16. "version": "0.1.0",
  17. "category": "data",
  18. "keywords": ["data-engineering", "analytics", "data-science", "bi", "dbt", "agents", "software-factory"]
  19. }
  20. ]
  21. }
README.md · README · 242 lines
  1. <h1 align="center">Data Factory — the open-source AI data team</h1>
  2. <p align="center"><img src="assets/banner.svg" alt="Data Factory: open-source AI agents for data engineering, analytics, data science and BI — a Claude Code plugin" width="100%"></p>
  3. <p align="center">
  4. <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-3B82F6" alt="MIT"></a>
  5. <img src="https://img.shields.io/badge/Claude%20Code-plugin-7C3AED" alt="Claude Code plugin">
  6. <img src="https://img.shields.io/badge/warehouses-Snowflake%20·%20BigQuery%20·%20Databricks%20·%20Postgres%20·%20Redshift%20·%20DuckDB-0B0F1E" alt="warehouses">
  7. <a href="CONTRIBUTING.md"><img src="https://img.shields.io/badge/PRs-welcome-3B82F6" alt="PRs welcome"></a>
  8. <a href="https://nimrodfisher.github.io/data-factory/"><img src="https://img.shields.io/badge/website-data--factory-A78BFA" alt="Website"></a>
  9. </p>
  10. <h2 align="center">Your data team is one person.<br>Data Factory gives them a whole team.</h2>
  11. <p align="center">
  12. A <b>data engineer</b>, an <b>analyst</b>, a <b>data scientist</b> and a <b>BI engineer</b> do the work.<br>
  13. A <b>quality inspector</b> checks every number. <b>You just approve.</b>
  14. </p>
  15. <p align="center"><img src="assets/how-it-works.gif" alt="Animation: a one-person data team today vs. an AI data team with Data Factory — AI data engineer, AI data analyst, AI data scientist, AI BI engineer and a quality inspector agent" width="100%"></p>
  16. <p align="center">
  17. <a href="#start-here"><b>▶ Try it in 5 minutes</b></a> &nbsp;·&nbsp;
  18. <a href="examples/mrr-drop"><b>👀 See a real run</b></a> &nbsp;·&nbsp;
  19. <a href="docs/use-cases.md"><b>💡 What can I ask it?</b></a>
  20. </p>
  21. ## What is Data Factory?
  22. **Data Factory is an open-source AI data team.** It's a [Claude Code](https://code.claude.com) plugin with **AI agents for data engineering, analytics, data science and BI**: an AI data engineer, an AI data analyst, an AI data scientist and an AI BI engineer, plus a **quality inspector agent** that verifies every number before it reaches you.
  23. It applies the **software factory** model (also called an *agentic* or *dark* factory) to data work, so **one data person can run a full data team**: you write the intent and approve; the agents build pipelines, dbt models, analyses, forecasts and dashboards; the inspector checks them against hidden tests and your real finance numbers. It works with **dbt, SQLMesh, Snowflake, BigQuery, Databricks, Redshift, Postgres and DuckDB**, and with **Looker, Power BI, Tableau, Metabase, Superset and Lightdash**.
  24. ## How it works, in plain words
  25. 1. **You ask** in plain English → `/data-factory:run Why did MRR drop in August?`
  26. 2. **It writes a spec** that pins down every word ("revenue", "active", "last month") → **you approve it**
  27. 3. **Four AI specialists build** the tables, the analysis, the forecast and the dashboard
  28. 4. **An inspector checks every number** against hidden tests and your finance figures → mistakes go back for rework
  29. 5. **You get the answer + the proof** → **you decide** what ships. Nothing touches production without you.
  30. It's the **software factory** idea that engineering teams adopted in 2026 (agents build, an independent system verifies, humans set intent) — applied to data. [Why it works →](docs/concept.md)
  31. ### It catches what people miss
  32. > In the [reference run](examples/mrr-drop), every visible check passed on the first build. The inspector's **hidden checks** still caught **1,132 duplicated invoices** (revenue overstated by 46%) and **3 internal test accounts** that made the August drop look 76% bigger and blamed the wrong customers. One rework loop later, every number matched finance — to the dollar.
  33. <p align="center"><img src="assets/dashboard.png" alt="MRR dashboard built by the Data Factory AI agents on the demo dataset" width="90%"></p>
  34. ---
  35. ## Start here
  36. **Try it on demo data (5 minutes, no warehouse):**
  37. ```bash
  38. git clone https://github.com/nimrodfisher/data-factory && cd data-factory/demo
  39. pip install duckdb pandas numpy && python generate_data.py
  40. claude
  41. ```
  42. ```
  43. /plugin install data-factory --marketplace nimrodfisher/data-factory
  44. /data-factory:run MRR dropped in August. Why — and was the Starter price increase a mistake? Forecast to December and give me a dashboard to track it.
  45. ```
  46. **Use it on your own data (3 steps):**
  47. 1. Open Claude Code in your data repo (dbt or SQL) and install: `/plugin install data-factory --marketplace nimrodfisher/data-factory`
  48. 2. `/data-factory:init` — it detects your stack and asks you at most 3 questions.
  49. 3. Ask anything: `/data-factory:run <your request in plain words>`
  50. <sub>Older Claude Code: `/plugin marketplace add nimrodfisher/data-factory` then `/plugin install data-factory@data-factory`.</sub>
  51. Then fill two files and you're 80% there: `.data-factory/context/metrics.md` (your definitions) and `.data-factory/context/truths.md` (3 numbers you trust from finance or the board deck).
  52. ---
  53. ## What you can hand to it
  54. | You need… | Type | You do |
  55. |---|---|---|
  56. | "Why did this number change?" | `/data-factory:run Signups dropped 18% last week — real? why?` | approve spec, read answer |
  57. | A Monday KPI review, automatically | `/data-factory:run weekly-kpi-review` (or schedule it) | skim the PR |
  58. | A dashboard | `/data-factory:run dashboard-request` | approve brief, publish |
  59. | A new data source in the warehouse | `/data-factory:run new-source-onboarding` | approve each step, merge PR |
  60. | An A/B test readout | `/data-factory:run experiment-readout` | approve the spec |
  61. | A forecast for the board | `/data-factory:run monthly-forecast` | review scenarios |
  62. | A second opinion on existing work | `/data-factory:inspect models/marts/finance/` | read the verdict |
  63. | To scope a vague ask | `/data-factory:spec "a single view of the customer"` | send the questions back |
  64. Six of these are **ready-made specs** that `init` installs into `.data-factory/specs/` — edit once, re-run forever.
  65. → Details and examples: [**docs/use-cases.md**](docs/use-cases.md)
  66. ## Where your team talks to it
  67. | Who | Where | Setup |
  68. |---|---|---|
  69. | You (data lead) | Claude Code — terminal, IDE, web, phone | install the plugin |
  70. | Stakeholders | a **"📊 Data request" GitHub issue** → spec comment → you approve → PR | [2 files](integrations/) |
  71. | Stakeholders | **Slack**: tag `@Claude` in `#data-requests` | Claude Team/Enterprise |
  72. | Nobody | **scheduled** runs (weekly KPIs, monthly forecast) → PR | [1 file](integrations/github/scheduled-recipe.yml) |
  73. → Copy-paste setup: [**docs/interfaces.md**](docs/interfaces.md) · Running a 1–2 person team on it: [**docs/lean-team.md**](docs/lean-team.md)
  74. ---
  75. ## Under the hood
  76. ```mermaid
  77. flowchart LR
  78. R([Your request]) --> S[SPEC<br/>definitions +<br/>acceptance checks]
  79. S --> G1{{You approve}}
  80. G1 --> H[SEAL<br/>inspector writes<br/>hidden holdouts]
  81. H --> P[PLAN<br/>work orders]
  82. P --> DE[data-engineer]
  83. DE --> DA[data-analyst]
  84. DE --> DS[data-scientist]
  85. DA --> BI[bi-engineer]
  86. DS --> BI
  87. BI --> I[INSPECT<br/>checks · holdouts ·<br/>independent re-derivation]
  88. I -- REWORK --> P
  89. I -- PASS --> SH[SHIP<br/>answer + evidence]
  90. SH --> G2{{You approve}}
  91. G2 --> L[LEARN<br/>context + lessons]
  92. ```
  93. | Software factory idea | In Data Factory |
  94. |---|---|
  95. | Specs drive agents | `spec.md` — the decision, every metric word defined, testable acceptance checks |
  96. | Scenarios hidden from coding agents | **Sealed holdouts** — truth anchors, invariants and known traps the builders never see |
  97. | Specialised agents | Four builder stations with deep, on-demand knowledge |
  98. | Verification you can trust | An **independent inspector** that re-derives headline numbers from raw data |
  99. | Humans set intent & approve | Two gates: the spec and the ship. Nothing ever touches production on its own |
  100. | The trace, not the code | Every run leaves `spec → orders → handoffs → inspection → ship` |
  101. | Compounding | Definitions, source quirks and lessons are written back after every run |
  102. Read the full reasoning in [**docs/concept.md**](docs/concept.md).
  103. ## The team
  104. | Agent | Does | Knows (loaded on demand) |
  105. |---|---|---|
  106. | 🔵 **data-engineer** | sources → staging → marts, tests, pipelines | ingestion & CDC, incremental/idempotent loads, Kimball & SCD2, orchestration, SQL traps & cost |
  107. | 🟢 **data-analyst** | questions → reproducible findings memo | metric trees, "why did X change" playbook, cohorts/funnels, mix-shift & Simpson's, pyramid-style memos |
  108. | 🟣 **data-scientist** | forecasts, experiments, causal estimates, models | baselines first, A/B design & SRM & CUPED, forecasting & backtests, leakage hunting, DiD/synthetic control, model cards |
  109. | 🟠 **bi-engineer** | semantic layer, dashboards | dashboard design for decisions, chart selection, metric governance, LookML/DAX/Tableau/Metabase/…, zero-dependency HTML dashboards |
  110. | 🔴 **quality-inspector** | sealed holdouts, inspection, verdict | test pyramid, reconciliation, holdout design, adversarial review per station |
  111. You are the **Factory Manager's boss**: the main Claude Code session plays Factory Manager (spec, plan, dispatch, gates); you own intent, definitions, truths and approvals.
  112. ## Why you can trust the output
  113. 1. **Definitions before SQL.** Vague words ("active", "revenue", "last month") must be defined in the spec before any station starts.
  114. 2. **Truth anchors.** Put 3–5 known-good numbers (finance close, board deck, CRM count) in `context/truths.md`. Outputs must reconcile to them.
  115. 3. **Sealed holdouts.** The inspector writes them *before* the build; builders are told never to read them. Visible checks can be gamed; sealed ones measure.
  116. 4. **Independent re-derivation.** The inspector recomputes the 1–3 headline numbers from raw data with its own query.
  117. 5. **Safety interlock.** A hook blocks `DROP`, `TRUNCATE`, unbounded `DELETE`/`UPDATE`, writes to your prod schema and dbt/SQLMesh prod targets.
  118. 6. **Two human gates.** You approve the spec and anything that ships. Production changes arrive as a PR.
  119. ## Works with your stack
  120. | Layer | Supported |
  121. |---|---|
  122. | Warehouse | Snowflake · BigQuery · Databricks · Redshift · Postgres · DuckDB/MotherDuck (+ any SQL via CLI or MCP) |
  123. | Transform | dbt · SQLMesh · plain SQL |
  124. | Orchestration | Airflow · Dagster · Prefect · dbt Cloud · GitHub Actions |
  125. | BI | Looker · Power BI · Tableau · Metabase · Superset · Lightdash · Hex · Streamlit · self-contained HTML |
  126. | Run SQL via | any CLI (`snow sql`, `bq`, `psql`, `duckdb`, `dbt show`) or a database **MCP server** |
  127. Missing yours? It's one markdown file — see [CONTRIBUTING.md](CONTRIBUTING.md).
  128. ## Commands
  129. | Command | What it does |
  130. |---|---|
  131. | `/data-factory:init` | detect your stack, create `.data-factory/` (config, context, memory) |
  132. | `/data-factory:run <request>` | the full line: spec → seal → plan → build → inspect → ship → learn |
  133. | `/data-factory:spec <request>` | just write the spec (great for scoping requests from stakeholders) |
  134. | `/data-factory:inspect [path]` | run the inspector on a run, a model, a query or someone's analysis |
  135. | `/data-factory:status` | the factory floor: every run, its stage, what's waiting on you |
  136. | `/data-factory:learn <fact>` | teach a definition, source quirk, truth anchor or lesson |
  137. You can also call any station directly: *"use the data-scientist agent to design an A/B test for the new onboarding"*.
  138. ## Autonomy dial
  139. | `autonomy:` | You approve | Use when |
  140. |---|---|---|
  141. | `supervised` | spec, every handoff, ship | first runs, sensitive data |
  142. | `gated` (default) | spec, ship | day-to-day |
  143. | `lights-out` | nothing during the run — you read the trace | recurring requests with strong truth anchors |
  144. Where is your team? → [**The 6 levels of the Data Factory**](docs/levels.md).
  145. ## What's in the repo
  146. ```
  147. .claude-plugin/marketplace.json ← install with /plugin marketplace add
  148. plugins/data-factory/
  149. ├── agents/ 5 stations (markdown)
  150. ├── commands/ init · run · spec · inspect · status · learn
  151. ├── skills/ 7 knowledge packs, 3-layer: SKILL.md → router.md → references/*.md
  152. │ ├── factory-playbook/ (+ templates and 6 ready-made recipe specs)
  153. │ ├── data-engineering/ data-analysis/ data-science/ bi-engineering/ data-quality/
  154. │ └── platforms/ Snowflake, BigQuery, Databricks, Postgres/Redshift, DuckDB, dbt/SQLMesh, orchestrators, BI tools
  155. ├── hooks/ safety interlock
  156. └── scripts/ sql_guard.py
  157. integrations/ copy-paste: .claude/settings.json, GitHub issue form, request & scheduled workflows
  158. demo/ synthetic SaaS in DuckDB + pre-filled .data-factory/
  159. examples/mrr-drop/ a full reference run: trace, SQL, memo, model card, dashboard
  160. docs/ use-cases · interfaces · lean-team · concept · levels · customize
  161. ```
  162. Skills load in three layers (lean entry → router → one deep reference), so each agent pulls only the knowledge its task needs.
  163. ## FAQ
  164. **What is an AI data factory?** A way of running data work like a software factory: specialised AI agents build, an independent agent verifies, and humans only set intent and approve. Data Factory is an open-source implementation for data teams. See [docs/concept.md](docs/concept.md).
  165. **Is this related to Azure Data Factory?** No. Azure Data Factory is Microsoft's data-integration (ETL) service. This project is an open-source **AI agent team** for data work, and it runs on top of whatever warehouse and pipeline tools you already use.
  166. **Can AI agents replace a data team?** They can do most of the building: SQL, dbt models, analyses, forecasts and dashboards. What they can't own is intent, definitions and accountability. Data Factory keeps those with a human data lead and makes the agents prove their numbers.
  167. **Is it an AI data analyst?** It includes one, plus an engineer, a data scientist, a BI engineer and an inspector. A single AI analyst writes queries; a factory also checks whether the answer is right.
  168. **How small can the data team be?** One person. The factory covers the four builder crafts; you cover intent, definitions, truths and approvals — about 3 hours a week of "factory management" once it's set up ([lean-team guide](docs/lean-team.md)).
  169. **Is this "no human in the loop"?** No. It's "no human in the *grind*". You own intent, definitions, truths and approvals — the four things that actually determine whether data is right.
  170. **Does it work without Claude Code?** The agents and skills are plain markdown; see [AGENTS.md](AGENTS.md) for Codex, Cursor, Gemini CLI and others. The orchestration commands and the safety hook are Claude Code–specific.
  171. **Will it touch production?** No. Agents build in your dev environment; the hook blocks destructive SQL and prod targets; shipping means a PR you merge. Also give the agent a dev-only warehouse role — defence in depth.
  172. **How much does a run cost?** Depends on the model and the request. An M-size request (2–3 stations) is a normal Claude Code session; L-size runs dispatch more agents. Start with `/data-factory:spec` to scope.
  173. ## Contributing
  174. Platform adapters, new stations (analytics engineer, ML engineer, data steward), better holdout patterns, more demos — see [CONTRIBUTING.md](CONTRIBUTING.md). If this is useful, a ⭐ helps others find it.
  175. ## Credits
  176. Built by [Nimrod Fisher](https://github.com/nimrodfisher) ([AI Analytics Hub](https://ai-analytics-hub.com)). The software-factory idea comes from the teams who published theirs in 2026 — StrongDM, Factory.ai and others; see [docs/concept.md](docs/concept.md) for sources.
  177. MIT licensed.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.