Skip to content
Back to search

DataFlow-WebUI

An NL2Pipeline Harness for building AI-ready data workflows.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/opendcai--dataflow-webui/badge.svg)](/listings/opendcai--dataflow-webui)

HTML

<a href="/listings/opendcai--dataflow-webui"><img src="/api/listings/opendcai--dataflow-webui/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

The `webui` profile installs the complete browser-based stack. Run the commands below from the repository root and use the same activated environment for setup and startup. Python **3.10 is recommended and is the minimum supported version**.

Install prerequisites

You need Git, Python 3.10+, and Node.js 20+ (which includes npm) for the `webui` profile. The `harness` profile needs Python 3.10+ only. The `skills` profile needs Python 3.9+ only (no packages installed). The installer will offer to install `uv` (the recommended Python package manager) if missing; you can also install it manually or use `--pip` to fall back to pip.

| System | Python and Git | Node.js 20+ and npm | uv (optional — installer can add it) | |---|---|---|---| | macOS | `brew install python@3.10 git` | `brew install nvm` then `nvm install 20` | `brew install uv` | | Ubuntu/Debian | `sudo apt update && sudo apt install -y python3.10 python3.10-venv git` | [nvm](https://github.com/nvm-sh/nvm), then `nvm install 20` | [uv installer](https://docs.astral.sh/uv/getting-started/installation/) | | Windows | `winget install Python.Python.3.10 Git.Git` | `winget install OpenJS.NodeJS.LTS` | [PowerShell installer](https://docs.astral.sh/uv/getting-started/installation/) or let the installer add it |

On macOS/Linux, the uv installer is:

curl -LsSf https://astral.sh/uv/install.sh | sh

On Windows PowerShell:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Restart the terminal after installing tools, then verify: - `python --version` (or `python3 --version`) - `node --version` (must be 20 or newer, only for `webui`) - `npm --version` (only for `webui`) - `uv --version` (optional — the installer will offer to add it if missing)

Create and activate a Python environment

Choose **one** of the following; do not mix environments.

# venv (macOS/Linux; Windows PowerShell uses .venv\Scripts\Activate.ps1)
python3.10 -m venv .venv
source .venv/bin/activate

# Or conda (all systems)
conda create -n dataflow python=3.10 -y
conda activate dataflow

On Ubuntu/Debian, install `python3.10-venv` if `venv` reports that `ensurepip` is missing.

**On Windows**, the install script (`.sh` files) must be run through **Git Bash** (bundled with Git for Windows). Do not run `.sh` files in PowerShell or `cmd.exe` — they will not execute. After installing Git for Windows, open **Git Bash** from the Start menu or right-click in the repository folder and select "Git Bash Here", then run:

# In Git Bash (Windows) or any terminal (macOS/Linux)
./install.sh --profile webui

If you activated a conda environment in PowerShell before opening Git Bash, the environment carries over. Otherwise, ensure the Python you want is on PATH or set `DF_PYTHON=python` (or `DF_PYTHON=/c/path/to/python.exe`) before running `./install.sh`.

Install, start, and use

./install.sh --profile webui
./scripts/start.sh             # foreground; Ctrl+C stops it
# or: ./scripts/start.sh --daemon

**Windows users**: run these commands in **Git Bash**, not PowerShell or cmd.exe.

Open <http://localhost:8000/>. In the chat panel select an installed agent, describe the pipeline you want, and inspect the resulting DAG on the canvas. Check or stop a background server with `./scripts/start.sh --status` and `./scripts/start.sh --stop`.

The installer uses `uv` by default (faster, more reliable). If `uv` is not installed, the installer will offer to install it automatically. To use `pip` instead, pass `--pip`:

./install.sh --profile webui --pip

Configure an AI agent

The WebUI chat workflow requires at least one supported agent. Install and authenticate an agent, then explicitly configure its MCP connection:

# Claude Code
curl -fsSL https://claude.ai/code/install.sh | sh
export ANTHROPIC_API_KEY=sk-ant-...
./install.sh configure-agent --agent claude

# Codex (API key or `codex login` OAuth)

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 301 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # DataFlow-Harness
  2. [![](https://img.shields.io/github/repo-size/OpenDCAI/DataFlow-WebUI?color=green)](https://github.com/OpenDCAI/DataFlow-WebUI)
  3. [![Technical Report](https://img.shields.io/badge/Technical%20Report-arXiv%3A2607.16617-B31B1B?logo=arxiv&logoColor=white)](https://arxiv.org/pdf/2607.16617)
  4. 中文文档:**[README_zh.md](README_zh.md)**
  5. Build, run and manage [DataFlow](https://github.com/OpenDCAI/DataFlow) data pipelines with a coding agent — through a visual canvas, through MCP, or with skills alone.
  6. ## Names you will see
  7. | Name | What it refers to |
  8. |---|---|
  9. | **DataFlow** | The upstream data-processing framework ([OpenDCAI/DataFlow](https://github.com/OpenDCAI/DataFlow)). Installed as the `open-dataflow` package. Not this repo. |
  10. | **DataFlow-Harness** | The system this repo builds: skills + MCP + WebUI working together. The name of the product, and of the paper. |
  11. | **DataFlow-WebUI** | The repository name, and the visual-canvas layer specifically. Kept for URL stability. |
  12. | **`DataFlow-WebUI-<version>.zip`** | A release package with the frontend pre-built, for running without a clone. See [docs/RELEASE-PACKAGE.md](docs/RELEASE-PACKAGE.md). |
  13. This repo ships **three independent layers**. Install only the one you need.
  14. They are independent at install and runtime; internally, all agent variants are
  15. rendered from the same `skills/canonical/` source so the standalone and
  16. MCP-aware instructions cannot drift.
  17. ## Which layer do I want?
  18. | | `webui` | `harness` | `skills` |
  19. |---|---|---|---|
  20. | **You get** | Visual DAG canvas + backend + MCP | Backend + MCP, no browser UI | Agent skills only |
  21. | **You need** | Python 3.10+, Node 20+ | Python 3.10+ | Python 3.9+ (to render the skills) |
  22. | **Installs packages** | yes (uv + npm) | yes (uv) | **no** |
  23. | **Runs a server** | yes, port 8000 | yes, port 8000 | **no** |
  24. | **Agent writes pipelines** | ✅ | ✅ | ✅ |
  25. | **Agent sees live operator registry** | ✅ | ✅ | ✗ (uses bundled reference) |
  26. | **Pipelines appear on a canvas** | ✅ | ✗ | ✗ |
  27. | **Install time** | minutes | ~1 min | seconds |
  28. The `skills` profile installs no packages, but it does run a Python script to
  29. render the skill files, so Python 3.9+ must be present.
  30. **30-second decision:**
  31. - You want to *see and edit* pipelines as a graph → **`webui`**
  32. - You drive everything from Claude Code / Codex / Cursor and never open a browser → **`harness`**
  33. - You just want your agent to write correct DataFlow code, with no server → **`skills`**
  34. ```bash
  35. git clone https://github.com/OpenDCAI/DataFlow-WebUI.git
  36. cd DataFlow-WebUI
  37. ./install.sh --list # compare the three, in detail
  38. ./install.sh --profile skills # or harness, or webui
  39. ```
  40. Every profile supports `--check` (prerequisites only), `--dry-run` (show the plan, change nothing) and `--uninstall`.
  41. Full per-layer guides: **[webui](docs/profiles/webui.md)** · **[harness](docs/profiles/harness.md)** · **[skills](docs/profiles/skills.md)**
  42. ## For agents setting this up
  43. If an AI coding agent is configuring this repository, point it to
  44. **[docs/agents/SETUP.md](docs/agents/SETUP.md)** before running installation.
  45. That guide defines the authorization boundaries, exact commands, verification
  46. steps, and actions that require human approval.
  47. ## Installation and usage
  48. The `webui` profile installs the complete browser-based stack. Run the commands
  49. below from the repository root and use the same activated environment for setup
  50. and startup. Python **3.10 is recommended and is the minimum supported version**.
  51. ### Install prerequisites
  52. You need Git, Python 3.10+, and Node.js 20+ (which includes npm) for the `webui` profile. The `harness` profile needs Python 3.10+ only. The `skills` profile needs Python 3.9+ only (no packages installed). The installer will offer to install `uv` (the recommended Python package manager) if missing; you can also install it manually or use `--pip` to fall back to pip.
  53. | System | Python and Git | Node.js 20+ and npm | uv (optional — installer can add it) |
  54. |---|---|---|---|
  55. | macOS | `brew install python@3.10 git` | `brew install nvm` then `nvm install 20` | `brew install uv` |
  56. | Ubuntu/Debian | `sudo apt update && sudo apt install -y python3.10 python3.10-venv git` | [nvm](https://github.com/nvm-sh/nvm), then `nvm install 20` | [uv installer](https://docs.astral.sh/uv/getting-started/installation/) |
  57. | Windows | `winget install Python.Python.3.10 Git.Git` | `winget install OpenJS.NodeJS.LTS` | [PowerShell installer](https://docs.astral.sh/uv/getting-started/installation/) or let the installer add it |
  58. On macOS/Linux, the uv installer is:
  59. ```bash
  60. curl -LsSf https://astral.sh/uv/install.sh | sh
  61. ```
  62. On Windows PowerShell:
  63. ```powershell
  64. powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  65. ```
  66. Restart the terminal after installing tools, then verify:
  67. - `python --version` (or `python3 --version`)
  68. - `node --version` (must be 20 or newer, only for `webui`)
  69. - `npm --version` (only for `webui`)
  70. - `uv --version` (optional — the installer will offer to add it if missing)
  71. ### Create and activate a Python environment
  72. Choose **one** of the following; do not mix environments.
  73. ```bash
  74. # venv (macOS/Linux; Windows PowerShell uses .venv\Scripts\Activate.ps1)
  75. python3.10 -m venv .venv
  76. source .venv/bin/activate
  77. # Or conda (all systems)
  78. conda create -n dataflow python=3.10 -y
  79. conda activate dataflow
  80. ```
  81. On Ubuntu/Debian, install `python3.10-venv` if `venv` reports that `ensurepip`
  82. is missing.
  83. **On Windows**, the install script (`.sh` files) must be run through **Git Bash** (bundled with Git for Windows). Do not run `.sh` files in PowerShell or `cmd.exe` — they will not execute. After installing Git for Windows, open **Git Bash** from the Start menu or right-click in the repository folder and select "Git Bash Here", then run:
  84. ```bash
  85. # In Git Bash (Windows) or any terminal (macOS/Linux)
  86. ./install.sh --profile webui
  87. ```
  88. If you activated a conda environment in PowerShell before opening Git Bash, the environment carries over. Otherwise, ensure the Python you want is on PATH or set `DF_PYTHON=python` (or `DF_PYTHON=/c/path/to/python.exe`) before running `./install.sh`.
  89. ### Install, start, and use
  90. ```bash
  91. ./install.sh --profile webui
  92. ./scripts/start.sh # foreground; Ctrl+C stops it
  93. # or: ./scripts/start.sh --daemon
  94. ```
  95. **Windows users**: run these commands in **Git Bash**, not PowerShell or cmd.exe.
  96. Open <http://localhost:8000/>. In the chat panel select an installed agent,
  97. describe the pipeline you want, and inspect the resulting DAG on the canvas.
  98. Check or stop a background server with `./scripts/start.sh --status` and
  99. `./scripts/start.sh --stop`.
  100. The installer uses `uv` by default (faster, more reliable). If `uv` is not installed, the installer will offer to install it automatically. To use `pip` instead, pass `--pip`:
  101. ```bash
  102. ./install.sh --profile webui --pip
  103. ```
  104. ### Configure an AI agent
  105. The WebUI chat workflow requires at least one supported agent. Install and
  106. authenticate an agent, then explicitly configure its MCP connection:
  107. ```bash
  108. # Claude Code
  109. curl -fsSL https://claude.ai/code/install.sh | sh
  110. export ANTHROPIC_API_KEY=sk-ant-...
  111. ./install.sh configure-agent --agent claude
  112. # Codex (API key or `codex login` OAuth)
  113. npm install --global @openai/codex
  114. codex login # OAuth, or export OPENAI_API_KEY=sk-...
  115. ./install.sh configure-agent --agent codex
  116. # Cursor: install the IDE, open this repository, then:
  117. ./install.sh configure-agent --agent cursor
  118. ```
  119. Agent configuration is intentionally a separate step from installation. The
  120. installer does not write API keys. See [Agent setup](docs/agents/SETUP.md) for
  121. the exact authorization boundaries and verification steps.
  122. ## Installing never writes agent configuration
  123. Installing and configuring an agent are two separate commands, on purpose:
  124. ```bash
  125. ./install.sh --profile harness # installs; writes no agent config
  126. ./install.sh configure-agent --agent claude # project-scoped, shows a diff first
  127. ./install.sh configure-agent --agent codex --scope user # asks before writing ~/.codex/
  128. ```
  129. Installing **never writes MCP configuration**, in either scope. That is
  130. `configure-agent`'s job.
  131. ### Where the skills land, per agent
  132. `--scope` only affects Claude Code. The other two agents read from the repo, so
  133. their assets are always installed here:
  134. | Agent | Installed to | Available in | Verify |
  135. |---|---|---|---|
  136. | **Claude Code** | `./.claude/skills/` (default), or `~/.claude/skills/` with `--scope user` | this repo, or every project with `--scope user` | `/generating-dataflow-pipeline` appears in completion |
  137. | **Codex** | `./AGENTS.md` + `./.codex/skills/` in this repo | this repo only — Codex reads `AGENTS.md` from the directory it starts in | open `AGENTS.md`; it routes to `.codex/skills/` |
  138. | **Cursor** | `./.cursor/skills/` and `./.cursor/rules/` in this repo | this repo only, when opened as a project in Cursor | rules appear under Settings → Rules |
  139. There is no global install for Codex or Cursor — both are directory-scoped by
  140. design. `--scope user` does not change that.
  141. Each profile declares the paths it must not touch, and the installer fingerprints
  142. them before and after the install and fails if any changed. Anything it cannot
  143. prove it wrote is never overwritten without `--force`.
  144. Existing MCP servers in your config are merged, never overwritten. No API keys are read, written or logged.
  145. Upgrading from `scripts/setup_all.sh`? See [docs/migration/from-setup-scripts.md](docs/migration/from-setup-scripts.md).
  146. ## What DataFlow-Harness is
  147. DataFlow-Harness combines **skills** (procedural knowledge about operator selection, schema links and assembly order), **MCP** (a live connection to the operator registry and current pipeline state), and the **WebUI** (turning agent-built workflows into persistent, editable DAGs).
  148. <p align="center">
  149. <img src="https://github.com/user-attachments/assets/c54d92ea-a6d5-4c3c-8144-87c3cdfee276" alt="DataFlow-WebUI" width="90%">
  150. </p>
  151. *The dual-modality interface: conversational agent and visual DAG editor stay in sync.*
  152. <p align="center">
  153. <img width="1280" height="638" alt="DataFlow-Harness architecture" src="https://github.com/user-attachments/assets/d9c862ac-a1a2-42b1-9f07-440d20d59d8f" />
  154. </p>
  155. *A shared pipeline representation synchronized across the agent runtime and the WebUI. Skills guide construction; the validation engine checks DAG structure and schema compatibility.*
  156. ### Why it helps
  157. **Bridges the NL2Pipeline gap.** Natural-language intent becomes a persistent, platform-native pipeline artifact you can inspect, edit, run and reuse.
  158. **Procedural knowledge, not just tool lists.** Building a VQA dataset from textbooks needs PDF parsing, layout understanding, OCR, image-text alignment, QA extraction and quality filtering *in the right order*. That ordering knowledge is what the skills encode.
  159. **Reusable workflows.** Data preparation becomes reusable pipelines rather than one-off scripts.
  160. **Reported results.** The DataFlow-Harness paper reports a 93.3% end-to-end pass
  161. rate on a 12-task data-engineering benchmark, with 72.5% lower cost and 49.9%
  162. lower generation latency than vanilla Claude Code.
  163. Those figures come from the paper's own harness and are **not reproducible from
  164. this repository** — the benchmark tasks, run configuration and raw results are not
  165. included here. Treat them as published findings, not as a claim you can verify by
  166. installing this repo. Details and methodology: [DataFlow-Harness](https://huggingface.co/papers/2607.16617).
  167. ## Supported agents
  168. | Agent | Mode | MCP config | Auth |
  169. |---|---|---|---|
  170. | **Claude Code** | WebUI-dispatched, or in your terminal | `.mcp.json` (project) | `ANTHROPIC_API_KEY`, or `ANTHROPIC_BASE_URL` for a gateway |
  171. | **Codex** | WebUI-dispatched, or in your terminal | `~/.codex/config.toml` | `OPENAI_API_KEY` (+ optional `OPENAI_BASE_URL`), or `codex login` OAuth |
  172. | **Cursor** | IDE only — not dispatched by the WebUI | `.cursor/mcp.json` (project) | Cursor built-in |
  173. Cursor is used by opening this project in the IDE; its agent discovers the MCP server and pushes pipelines onto the canvas. It is not driven from the WebUI chat panel.
  174. ## Architecture and decisions
  175. - [Component boundaries](docs/architecture/overview.md) — how the three layers depend on each other
  176. - [ADR-001: single source of truth](docs/architecture/adr-001-source-of-truth.md) — why skills live in one repo
  177. - [ADR-002: one package manager](docs/architecture/adr-002-package-managers.md) — npm vs Yarn, resolved
  178. - [ADR-003: install/configure split](docs/architecture/adr-003-install-configure-split.md) — why installing writes no agent config
  179. - [Branch audit](docs/architecture/branch-audit.md) — status of every remote branch
  180. ## Contributing
  181. Skills are edited in `skills/canonical/` only. The files under `.claude/skills/`,
  182. `.cursor/skills/`, `.cursor/rules/<skill>.mdc`, `.codex/skills/` and `AGENTS.md`
  183. are generated build output and are **not tracked in git** — a fresh clone will not
  184. have them:
  185. ```bash
  186. make skills # generate agent assets from skills/canonical/
  187. make check # run every static check CI runs
  188. ```
  189. CI regenerates and diffs, so a hand-edited generated file fails the build.
  190. ## Citation
  191. If you use the DataFlow-Harness platform, its agent workflow, MCP integration,
  192. or editable pipeline interface, cite the DataFlow-Harness paper:
  193. ```bibtex
  194. @article{he2026dataflow,
  195. title={DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines},
  196. author={He, Runming and Wong, Zhen Hao and Liang, Hao and Meng, Zimo and Shen, Chengyu and Ma, Xiaochen and Zhang, Wentao},
  197. journal={arXiv preprint arXiv:2607.16617},
  198. year={2026}
  199. }
  200. ```
  201. If you use or cite the underlying DataFlow framework, cite the DataFlow paper:
  202. ```bibtex
  203. @article{liang2025dataflow,
  204. title={DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI},
  205. author={Liang, Hao and Ma, Xiaochen and Liu, Zhou and Wong, Zhen Hao and Zhao, Zhengyang and Meng, Zimo and He, Runming and Shen, Chengyu and Cai, Qifeng and Han, Zhaoyang and others},
  206. journal={arXiv preprint arXiv:2512.16676},
  207. year={2025}
  208. }
  209. ```
  210. ## Community
  211. <p align="center">
  212. <img src="https://github.com/user-attachments/assets/b958dc89-d76e-4e47-9277-22b3f6661944" alt="DataFlow community QR code" width="85%">
  213. </p>
  214. Licensed under Apache 2.0 — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.