DataFlow-WebUI
An NL2Pipeline Harness for building AI-ready data workflows.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/opendcai--dataflow-webui)HTML
<a href="/listings/opendcai--dataflow-webui"><img src="/api/listings/opendcai--dataflow-webui/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
These steps run a script from the network that findsafeskills does not scan.
The `webui` profile installs the complete browser-based stack. Run the commands below from the repository root and use the same activated environment for setup and startup. Python **3.10 is recommended and is the minimum supported version**.
Install prerequisites
You need Git, Python 3.10+, and Node.js 20+ (which includes npm) for the `webui` profile. The `harness` profile needs Python 3.10+ only. The `skills` profile needs Python 3.9+ only (no packages installed). The installer will offer to install `uv` (the recommended Python package manager) if missing; you can also install it manually or use `--pip` to fall back to pip.
| System | Python and Git | Node.js 20+ and npm | uv (optional — installer can add it) | |---|---|---|---| | macOS | `brew install python@3.10 git` | `brew install nvm` then `nvm install 20` | `brew install uv` | | Ubuntu/Debian | `sudo apt update && sudo apt install -y python3.10 python3.10-venv git` | [nvm](https://github.com/nvm-sh/nvm), then `nvm install 20` | [uv installer](https://docs.astral.sh/uv/getting-started/installation/) | | Windows | `winget install Python.Python.3.10 Git.Git` | `winget install OpenJS.NodeJS.LTS` | [PowerShell installer](https://docs.astral.sh/uv/getting-started/installation/) or let the installer add it |
On macOS/Linux, the uv installer is:
curl -LsSf https://astral.sh/uv/install.sh | sh
On Windows PowerShell:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Restart the terminal after installing tools, then verify: - `python --version` (or `python3 --version`) - `node --version` (must be 20 or newer, only for `webui`) - `npm --version` (only for `webui`) - `uv --version` (optional — the installer will offer to add it if missing)
Create and activate a Python environment
Choose **one** of the following; do not mix environments.
# venv (macOS/Linux; Windows PowerShell uses .venv\Scripts\Activate.ps1) python3.10 -m venv .venv source .venv/bin/activate # Or conda (all systems) conda create -n dataflow python=3.10 -y conda activate dataflow
On Ubuntu/Debian, install `python3.10-venv` if `venv` reports that `ensurepip` is missing.
**On Windows**, the install script (`.sh` files) must be run through **Git Bash** (bundled with Git for Windows). Do not run `.sh` files in PowerShell or `cmd.exe` — they will not execute. After installing Git for Windows, open **Git Bash** from the Start menu or right-click in the repository folder and select "Git Bash Here", then run:
# In Git Bash (Windows) or any terminal (macOS/Linux) ./install.sh --profile webui
If you activated a conda environment in PowerShell before opening Git Bash, the environment carries over. Otherwise, ensure the Python you want is on PATH or set `DF_PYTHON=python` (or `DF_PYTHON=/c/path/to/python.exe`) before running `./install.sh`.
Install, start, and use
./install.sh --profile webui ./scripts/start.sh # foreground; Ctrl+C stops it # or: ./scripts/start.sh --daemon
**Windows users**: run these commands in **Git Bash**, not PowerShell or cmd.exe.
Open <http://localhost:8000/>. In the chat panel select an installed agent, describe the pipeline you want, and inspect the resulting DAG on the canvas. Check or stop a background server with `./scripts/start.sh --status` and `./scripts/start.sh --stop`.
The installer uses `uv` by default (faster, more reliable). If `uv` is not installed, the installer will offer to install it automatically. To use `pip` instead, pass `--pip`:
./install.sh --profile webui --pip
Configure an AI agent
The WebUI chat workflow requires at least one supported agent. Install and authenticate an agent, then explicitly configure its MCP connection:
# Claude Code curl -fsSL https://claude.ai/code/install.sh | sh export ANTHROPIC_API_KEY=sk-ant-... ./install.sh configure-agent --agent claude # Codex (API key or `codex login` OAuth)
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
README.md · README · 301 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # DataFlow-Harness
- [](https://github.com/OpenDCAI/DataFlow-WebUI)
- [](https://arxiv.org/pdf/2607.16617)
- 中文文档:**[README_zh.md](README_zh.md)**
- Build, run and manage [DataFlow](https://github.com/OpenDCAI/DataFlow) data pipelines with a coding agent — through a visual canvas, through MCP, or with skills alone.
- ## Names you will see
- | Name | What it refers to |
- |---|---|
- | **DataFlow** | The upstream data-processing framework ([OpenDCAI/DataFlow](https://github.com/OpenDCAI/DataFlow)). Installed as the `open-dataflow` package. Not this repo. |
- | **DataFlow-Harness** | The system this repo builds: skills + MCP + WebUI working together. The name of the product, and of the paper. |
- | **DataFlow-WebUI** | The repository name, and the visual-canvas layer specifically. Kept for URL stability. |
- | **`DataFlow-WebUI-<version>.zip`** | A release package with the frontend pre-built, for running without a clone. See [docs/RELEASE-PACKAGE.md](docs/RELEASE-PACKAGE.md). |
- This repo ships **three independent layers**. Install only the one you need.
- They are independent at install and runtime; internally, all agent variants are
- rendered from the same `skills/canonical/` source so the standalone and
- MCP-aware instructions cannot drift.
- ## Which layer do I want?
- | | `webui` | `harness` | `skills` |
- |---|---|---|---|
- | **You get** | Visual DAG canvas + backend + MCP | Backend + MCP, no browser UI | Agent skills only |
- | **You need** | Python 3.10+, Node 20+ | Python 3.10+ | Python 3.9+ (to render the skills) |
- | **Installs packages** | yes (uv + npm) | yes (uv) | **no** |
- | **Runs a server** | yes, port 8000 | yes, port 8000 | **no** |
- | **Agent writes pipelines** | ✅ | ✅ | ✅ |
- | **Agent sees live operator registry** | ✅ | ✅ | ✗ (uses bundled reference) |
- | **Pipelines appear on a canvas** | ✅ | ✗ | ✗ |
- | **Install time** | minutes | ~1 min | seconds |
- The `skills` profile installs no packages, but it does run a Python script to
- render the skill files, so Python 3.9+ must be present.
- **30-second decision:**
- - You want to *see and edit* pipelines as a graph → **`webui`**
- - You drive everything from Claude Code / Codex / Cursor and never open a browser → **`harness`**
- - You just want your agent to write correct DataFlow code, with no server → **`skills`**
- ```bash
- git clone https://github.com/OpenDCAI/DataFlow-WebUI.git
- cd DataFlow-WebUI
- ./install.sh --list # compare the three, in detail
- ./install.sh --profile skills # or harness, or webui
- ```
- Every profile supports `--check` (prerequisites only), `--dry-run` (show the plan, change nothing) and `--uninstall`.
- Full per-layer guides: **[webui](docs/profiles/webui.md)** · **[harness](docs/profiles/harness.md)** · **[skills](docs/profiles/skills.md)**
- ## For agents setting this up
- If an AI coding agent is configuring this repository, point it to
- **[docs/agents/SETUP.md](docs/agents/SETUP.md)** before running installation.
- That guide defines the authorization boundaries, exact commands, verification
- steps, and actions that require human approval.
- ## Installation and usage
- The `webui` profile installs the complete browser-based stack. Run the commands
- below from the repository root and use the same activated environment for setup
- and startup. Python **3.10 is recommended and is the minimum supported version**.
- ### Install prerequisites
- You need Git, Python 3.10+, and Node.js 20+ (which includes npm) for the `webui` profile. The `harness` profile needs Python 3.10+ only. The `skills` profile needs Python 3.9+ only (no packages installed). The installer will offer to install `uv` (the recommended Python package manager) if missing; you can also install it manually or use `--pip` to fall back to pip.
- | System | Python and Git | Node.js 20+ and npm | uv (optional — installer can add it) |
- |---|---|---|---|
- | macOS | `brew install python@3.10 git` | `brew install nvm` then `nvm install 20` | `brew install uv` |
- | Ubuntu/Debian | `sudo apt update && sudo apt install -y python3.10 python3.10-venv git` | [nvm](https://github.com/nvm-sh/nvm), then `nvm install 20` | [uv installer](https://docs.astral.sh/uv/getting-started/installation/) |
- | Windows | `winget install Python.Python.3.10 Git.Git` | `winget install OpenJS.NodeJS.LTS` | [PowerShell installer](https://docs.astral.sh/uv/getting-started/installation/) or let the installer add it |
- On macOS/Linux, the uv installer is:
- ```bash
- curl -LsSf https://astral.sh/uv/install.sh | sh
- ```
- On Windows PowerShell:
- ```powershell
- powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
- ```
- Restart the terminal after installing tools, then verify:
- - `python --version` (or `python3 --version`)
- - `node --version` (must be 20 or newer, only for `webui`)
- - `npm --version` (only for `webui`)
- - `uv --version` (optional — the installer will offer to add it if missing)
- ### Create and activate a Python environment
- Choose **one** of the following; do not mix environments.
- ```bash
- # venv (macOS/Linux; Windows PowerShell uses .venv\Scripts\Activate.ps1)
- python3.10 -m venv .venv
- source .venv/bin/activate
- # Or conda (all systems)
- conda create -n dataflow python=3.10 -y
- conda activate dataflow
- ```
- On Ubuntu/Debian, install `python3.10-venv` if `venv` reports that `ensurepip`
- is missing.
- **On Windows**, the install script (`.sh` files) must be run through **Git Bash** (bundled with Git for Windows). Do not run `.sh` files in PowerShell or `cmd.exe` — they will not execute. After installing Git for Windows, open **Git Bash** from the Start menu or right-click in the repository folder and select "Git Bash Here", then run:
- ```bash
- # In Git Bash (Windows) or any terminal (macOS/Linux)
- ./install.sh --profile webui
- ```
- If you activated a conda environment in PowerShell before opening Git Bash, the environment carries over. Otherwise, ensure the Python you want is on PATH or set `DF_PYTHON=python` (or `DF_PYTHON=/c/path/to/python.exe`) before running `./install.sh`.
- ### Install, start, and use
- ```bash
- ./install.sh --profile webui
- ./scripts/start.sh # foreground; Ctrl+C stops it
- # or: ./scripts/start.sh --daemon
- ```
- **Windows users**: run these commands in **Git Bash**, not PowerShell or cmd.exe.
- Open <http://localhost:8000/>. In the chat panel select an installed agent,
- describe the pipeline you want, and inspect the resulting DAG on the canvas.
- Check or stop a background server with `./scripts/start.sh --status` and
- `./scripts/start.sh --stop`.
- The installer uses `uv` by default (faster, more reliable). If `uv` is not installed, the installer will offer to install it automatically. To use `pip` instead, pass `--pip`:
- ```bash
- ./install.sh --profile webui --pip
- ```
- ### Configure an AI agent
- The WebUI chat workflow requires at least one supported agent. Install and
- authenticate an agent, then explicitly configure its MCP connection:
- ```bash
- # Claude Code
- curl -fsSL https://claude.ai/code/install.sh | sh
- export ANTHROPIC_API_KEY=sk-ant-...
- ./install.sh configure-agent --agent claude
- # Codex (API key or `codex login` OAuth)
- npm install --global @openai/codex
- codex login # OAuth, or export OPENAI_API_KEY=sk-...
- ./install.sh configure-agent --agent codex
- # Cursor: install the IDE, open this repository, then:
- ./install.sh configure-agent --agent cursor
- ```
- Agent configuration is intentionally a separate step from installation. The
- installer does not write API keys. See [Agent setup](docs/agents/SETUP.md) for
- the exact authorization boundaries and verification steps.
- ## Installing never writes agent configuration
- Installing and configuring an agent are two separate commands, on purpose:
- ```bash
- ./install.sh --profile harness # installs; writes no agent config
- ./install.sh configure-agent --agent claude # project-scoped, shows a diff first
- ./install.sh configure-agent --agent codex --scope user # asks before writing ~/.codex/
- ```
- Installing **never writes MCP configuration**, in either scope. That is
- `configure-agent`'s job.
- ### Where the skills land, per agent
- `--scope` only affects Claude Code. The other two agents read from the repo, so
- their assets are always installed here:
- | Agent | Installed to | Available in | Verify |
- |---|---|---|---|
- | **Claude Code** | `./.claude/skills/` (default), or `~/.claude/skills/` with `--scope user` | this repo, or every project with `--scope user` | `/generating-dataflow-pipeline` appears in completion |
- | **Codex** | `./AGENTS.md` + `./.codex/skills/` in this repo | this repo only — Codex reads `AGENTS.md` from the directory it starts in | open `AGENTS.md`; it routes to `.codex/skills/` |
- | **Cursor** | `./.cursor/skills/` and `./.cursor/rules/` in this repo | this repo only, when opened as a project in Cursor | rules appear under Settings → Rules |
- There is no global install for Codex or Cursor — both are directory-scoped by
- design. `--scope user` does not change that.
- Each profile declares the paths it must not touch, and the installer fingerprints
- them before and after the install and fails if any changed. Anything it cannot
- prove it wrote is never overwritten without `--force`.
- Existing MCP servers in your config are merged, never overwritten. No API keys are read, written or logged.
- Upgrading from `scripts/setup_all.sh`? See [docs/migration/from-setup-scripts.md](docs/migration/from-setup-scripts.md).
- ## What DataFlow-Harness is
- DataFlow-Harness combines **skills** (procedural knowledge about operator selection, schema links and assembly order), **MCP** (a live connection to the operator registry and current pipeline state), and the **WebUI** (turning agent-built workflows into persistent, editable DAGs).
- <p align="center">
- <img src="https://github.com/user-attachments/assets/c54d92ea-a6d5-4c3c-8144-87c3cdfee276" alt="DataFlow-WebUI" width="90%">
- </p>
- *The dual-modality interface: conversational agent and visual DAG editor stay in sync.*
- <p align="center">
- <img width="1280" height="638" alt="DataFlow-Harness architecture" src="https://github.com/user-attachments/assets/d9c862ac-a1a2-42b1-9f07-440d20d59d8f" />
- </p>
- *A shared pipeline representation synchronized across the agent runtime and the WebUI. Skills guide construction; the validation engine checks DAG structure and schema compatibility.*
- ### Why it helps
- **Bridges the NL2Pipeline gap.** Natural-language intent becomes a persistent, platform-native pipeline artifact you can inspect, edit, run and reuse.
- **Procedural knowledge, not just tool lists.** Building a VQA dataset from textbooks needs PDF parsing, layout understanding, OCR, image-text alignment, QA extraction and quality filtering *in the right order*. That ordering knowledge is what the skills encode.
- **Reusable workflows.** Data preparation becomes reusable pipelines rather than one-off scripts.
- **Reported results.** The DataFlow-Harness paper reports a 93.3% end-to-end pass
- rate on a 12-task data-engineering benchmark, with 72.5% lower cost and 49.9%
- lower generation latency than vanilla Claude Code.
- Those figures come from the paper's own harness and are **not reproducible from
- this repository** — the benchmark tasks, run configuration and raw results are not
- included here. Treat them as published findings, not as a claim you can verify by
- installing this repo. Details and methodology: [DataFlow-Harness](https://huggingface.co/papers/2607.16617).
- ## Supported agents
- | Agent | Mode | MCP config | Auth |
- |---|---|---|---|
- | **Claude Code** | WebUI-dispatched, or in your terminal | `.mcp.json` (project) | `ANTHROPIC_API_KEY`, or `ANTHROPIC_BASE_URL` for a gateway |
- | **Codex** | WebUI-dispatched, or in your terminal | `~/.codex/config.toml` | `OPENAI_API_KEY` (+ optional `OPENAI_BASE_URL`), or `codex login` OAuth |
- | **Cursor** | IDE only — not dispatched by the WebUI | `.cursor/mcp.json` (project) | Cursor built-in |
- Cursor is used by opening this project in the IDE; its agent discovers the MCP server and pushes pipelines onto the canvas. It is not driven from the WebUI chat panel.
- ## Architecture and decisions
- - [Component boundaries](docs/architecture/overview.md) — how the three layers depend on each other
- - [ADR-001: single source of truth](docs/architecture/adr-001-source-of-truth.md) — why skills live in one repo
- - [ADR-002: one package manager](docs/architecture/adr-002-package-managers.md) — npm vs Yarn, resolved
- - [ADR-003: install/configure split](docs/architecture/adr-003-install-configure-split.md) — why installing writes no agent config
- - [Branch audit](docs/architecture/branch-audit.md) — status of every remote branch
- ## Contributing
- Skills are edited in `skills/canonical/` only. The files under `.claude/skills/`,
- `.cursor/skills/`, `.cursor/rules/<skill>.mdc`, `.codex/skills/` and `AGENTS.md`
- are generated build output and are **not tracked in git** — a fresh clone will not
- have them:
- ```bash
- make skills # generate agent assets from skills/canonical/
- make check # run every static check CI runs
- ```
- CI regenerates and diffs, so a hand-edited generated file fails the build.
- ## Citation
- If you use the DataFlow-Harness platform, its agent workflow, MCP integration,
- or editable pipeline interface, cite the DataFlow-Harness paper:
- ```bibtex
- @article{he2026dataflow,
- title={DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines},
- author={He, Runming and Wong, Zhen Hao and Liang, Hao and Meng, Zimo and Shen, Chengyu and Ma, Xiaochen and Zhang, Wentao},
- journal={arXiv preprint arXiv:2607.16617},
- year={2026}
- }
- ```
- If you use or cite the underlying DataFlow framework, cite the DataFlow paper:
- ```bibtex
- @article{liang2025dataflow,
- title={DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI},
- author={Liang, Hao and Ma, Xiaochen and Liu, Zhou and Wong, Zhen Hao and Zhao, Zhengyang and Meng, Zimo and He, Runming and Shen, Chengyu and Cai, Qifeng and Han, Zhaoyang and others},
- journal={arXiv preprint arXiv:2512.16676},
- year={2025}
- }
- ```
- ## Community
- <p align="center">
- <img src="https://github.com/user-attachments/assets/b958dc89-d76e-4e47-9277-22b3f6661944" alt="DataFlow community QR code" width="85%">
- </p>
- Licensed under Apache 2.0 — see [LICENSE](LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.