Skip to content
Back to search

data-master

SkillWorks with Claude CodeGitHub

"Analyze datasets with preserved raw inputs, explicit cleaning decisions, reproducible figures and PDF reports. Full statistics for one tabular file; descriptive time-series analysis; integrity profiling for text, image, audio, geographic and mixed inputs. Use for data cleaning, EDA, group comparisons, statistical reports, 跑数据 or 数据分析 with a traceable evidence trail."

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/maga2010--data-master/badge.svg)](/listings/maga2010--data-master)

HTML

<a href="/listings/maga2010--data-master"><img src="/api/listings/maga2010--data-master/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install the Python package from a clone:

git clone https://github.com/MAGA2010/data-master.git
cd data-master
python -m venv .venv
# Windows: .venv\Scripts\python.exe -m pip install .
# macOS/Linux: .venv/bin/python -m pip install .
# Windows: .venv\Scripts\data-master.exe doctor
# macOS/Linux: .venv/bin/data-master doctor
# Windows: .venv\Scripts\data-master.exe skill install
# macOS/Linux: .venv/bin/data-master skill install

The Python package path uses the activated project virtual environment. The `data-master setup --extra <package>` helper is provided by the npm launcher; when it is unavailable, an unknown-format branch stays profile-only instead of installing a parser into another interpreter.

Run directly from a checkout:

python scripts/run_pipeline.py \
  --in examples/iris/iris_dirty.csv \
  --out outputs/iris-demo \
  --title "Iris morphometrics" \
  --journal nature

Use a release tag or commit SHA in GitHub npm specifications when reproducible installation matters. The project is not currently published to the npm or PyPI registries.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 287 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: data-master
  3. description: "Analyze datasets with preserved raw inputs, explicit cleaning decisions, reproducible figures and PDF reports. Full statistics for one tabular file; descriptive time-series analysis; integrity profiling for text, image, audio, geographic and mixed inputs. Use for data cleaning, EDA, group comparisons, statistical reports, 跑数据 or 数据分析 with a traceable evidence trail."
  4. allowed-tools: Bash(python *), Bash(py *), PowerShell(python *), PowerShell(py *), Bash(data-master *), PowerShell(data-master *), WebSearch, WebFetch
  5. ---
  6. # data-master — universal data processing pipeline
  7. Turn **any data** into a **cleaned, audited, Nature-grade report + charts** with one command. The agent reads the data, decides modality, picks methods, runs the pipeline, and writes evidence-backed outputs. The user never has to choose a statistical test or a chart type.
  8. ## Contract
  9. Installed CLI: `data-master analyze <file> --out <dir>`; use
  10. `data-master doctor --json` to diagnose the environment and
  11. `data-master verify <dir> --json` to check saved artifact hashes.
  12. `data-master demo --out <new-dir>` runs a synthetic example, and
  13. `data-master skill install` installs this skill atomically from the npm or
  14. Python distribution. Existing
  15. `python scripts/run_pipeline.py` commands below remain supported; installing
  16. the CLI alone does not register the skill until `skill install` is run.
  17. - **Input**: `$ARGUMENTS` = a file, a folder, or a glob. Optional flags:
  18. - `--report nature` (default) | `standard` | `executive` — report template.
  19. - `--out <dir>` — overrides the output root.
  20. - `--answers <file>` — answers `answers.template.json` decisions by id; every id and action is validated, and changing the file invalidates the stale cleaning stages on resume. The file may also carry a `run` block declaring `unit_key` / `group` / `outcomes` / `design` / `journal` / `covariates` (validated against the profile; explicit CLI values always win). A decision may name a single `column` of a multi-column flag.
  21. - `--covariates <a,b>` — declare numeric covariates; the adjusted (ANCOVA) comparison becomes primary, unadjusted stays as cross-check, and the homogeneity-of-slopes assumption is tested and reported.
  22. - `--unit-key <column>`, `--group <column>`, and `--outcomes <a,b>` — all three must be explicitly declared before the analysis is labelled confirmatory. Automatic group/outcome selection is exploratory.
  23. - `--max-input-mb <n>` — override the 300 MB in-memory input gate (readers hold files whole, as text).
  24. - `--export-lockfile` — writes `lockfile.txt` with the exact dependency versions that ran.
  25. - **Output** (all under `<dataset_dir>\outputs\`):
  26. - `cleaned/` — cleaned data + `cleaning_plan.json` + `cleaning_report.md` + reproducible `clean_*.py`
  27. - `analysis/` — `profile.json`, `stats.json`, `validation_report.md`, `chart_decisions.json`, `sensitivity.json`
  28. - `charts/` — `*.png` figures
  29. - `report.md` — main report (the citable source text)
  30. - `report.html` — self-contained shareable export of report.md (figures embedded; verification hashes it)
  31. - `report.pdf` — **primary deliverable**; rendered by a three-tier cascade
  32. - `executive_summary.md` — 1-page TL;DR
  33. - `report_build.json` — provenance for every number + PDF backend trail + self-audit + pipeline commit
  34. - `audit/final_audit.md` — 8-gate + journal-checklist verdict
  35. - `charts_rendered.json` — per-figure ink ratio and blank-page check
  36. - `answers.template.json` — the never-automatic decisions this dataset raised, fillable and fed back with `--answers`
  37. - `lockfile.txt` — (with `--export-lockfile`) exact dependency versions that ran
  38. - **Invariant — additive only**: NEVER modify, move, or delete any raw input file. Re-runs may overwrite `outputs/`.
  39. - **Full-auto with gates**: never block on a method choice; always pick the most defensible default and document. ONLY ask the user when the input itself is ambiguous.
  40. - **Method provenance**: every AI judgment records `rule_source` + `alternative_considered` + `rejected_reason`. Every number cites the script that produced it.
  41. ## Data integrity contract
  42. These four rules are what make the output worth trusting. They are not
  43. precautions; each one exists because the alternative has already produced a
  44. plausible-looking report about data that was no longer there.
  45. **1. The reader never guesses a type.** CSV / TSV / Excel are read with every
  46. cell as a string (`dm_types.read_table`, `read_mode="text"`). `pd.read_csv`
  47. turns the postcode `02134` into `2134`, which then collides with the genuinely
  48. different postcode `2134` — and it does so *before any audit exists*, so the
  49. leading zero and the evidence that a decision was made are both unrecoverable.
  50. Types are instead decided by an op in the plan, where they are disclosed and
  51. reversible. JSON and Parquet are exempt: their schemas state the types.
  52. `--read-mode infer` is an explicit escape hatch.
  53. **2. A conversion runs only if every value parses.** No parse-rate threshold
  54. exists in this skill, by design. A threshold does not make the decision safer; it
  55. only chooses how much to destroy quietly. Four situations hold the conversion and
  56. ask instead — `leading_zero`, `currency_or_rate`, `numeric_or_date`,
  57. `mixed_with_text` — each with the evidence attached. See
  58. [references/cleaning-rule-catalog.md](references/cleaning-rule-catalog.md).
  59. **3. A value may only change in a column an operation named.** After cleaning,
  60. `execute_cleaning.py` compares the raw file against the output in text space and
  61. fails the run if any untouched column gained a value. The blanket hygiene ops are
  62. replayed exactly, so the check is not vacuous. This is what catches a reader
  63. reinterpreting the file behind the plan's back.
  64. **4. Types are decided once and re-applied, never re-guessed.** The run records
  65. a `dtype_manifest`; `dm_types.load_cleaned` applies it. An identifier cannot
  66. quietly become an integer in the analysis after the report said it was text. With
  67. no manifest, the read is labelled as a guess rather than presented as fact.
  68. Every cleaning run is also checked for **byte-for-byte reproducibility**: the
  69. generated standalone script re-reads the raw file and must reproduce the output
  70. hash exactly.
  71. ## Three-stage pipeline
  72. ```
  73. [Stage 1: Detect & Clean] [Stage 2: Analyze (Nature)] [Stage 3: Deliver]
  74. -------------------------- ---------------------------- ---------------------
  75. data-router (modality) data-explorer (EDA) visualizer (charts)
  76. | | |
  77. data-profiler (profile) data-validator (gates) report-generator (report)
  78. | |
  79. data-cleaner (clean) data-transformer (transform)
  80. |
  81. statistical-analyzer (stats)
  82. |
  83. deep-analyzer (model — optional)
  84. ```
  85. Every stage has a **gate**. A gate fails when evidence is missing, the data is too dirty, or a Nature requirement is unmet. A failed gate freezes the run to `audit/` with an actionable message; the user can fix the input and re-run.
  86. ### The 10 sub-skills
  87. The router in this file owns the run; each sub-skill owns one stage, one artifact set and
  88. one gate. Load only the stage you are executing, plus the reference it names.
  89. | # | Sub-skill | Stage | Owns | Gate |
  90. |---|---|---|---|---|
  91. | 1 | [data-router](skills/data-router/SKILL.md) | 1a | modality + `route.json` | every input has a modality |
  92. | 2 | [data-profiler](skills/data-profiler/SKILL.md) | 1b | `profile.json` | every column typed, every rejected group explained |
  93. | 3 | [data-cleaner](skills/data-cleaner/SKILL.md) | 1c | `cleaning_plan.json` + `cleaned/*` | reproducibility + untouched-column audit |
  94. | 4 | [data-explorer](skills/data-explorer/SKILL.md) | 2a | `descriptive.json` | mean *and* median, missingness screened |
  95. | 5 | [data-validator](skills/data-validator/SKILL.md) | 3c | `audit/final_audit.*` | 8 Nature gates + journal checklist |
  96. | 6 | [data-transformer](skills/data-transformer/SKILL.md) | 2b | `transform_ledger.json` | every derived column reversible and recorded |
  97. | 7 | [statistical-analyzer](skills/statistical-analyzer/SKILL.md) | 2c | `inferential.json` | effect size + CI, family corrected, no NaN p, primary at the unit level the structure supports |
  98. | 8 | [deep-analyzer](skills/deep-analyzer/SKILL.md) | 2d | `model.json` | every coefficient has a CI and a diagnostic |
  99. | 9 | [visualizer](skills/visualizer/SKILL.md) | 3a | `chart_decisions.json` + `charts/*.png` | one argument per figure, blank-page check |
  100. | 10 | [report-generator](skills/report-generator/SKILL.md) | 3b | `report.md` + `report.pdf` | every number traced, no forbidden phrasing |
  101. Stages 2a-2d may run **in parallel** across modalities; they all read the same cleaned
  102. artifact and write disjoint JSON. Stage 3 is sequential: figures before report, report
  103. before audit.
  104. ## Routing
  105. The table states what ACTUALLY runs today. Non-tabular modalities take the
  106. profile-only branch: routing, profiling, a profile-only PDF and an integrity
  107. audit — no inferential claim. Do not improvise the unimplemented stages; a
  108. reference file that does not exist is a stop, not a suggestion.
  109. | File pattern | Modality | Stage 1 rule | Stage 2 stack (as implemented) |
  110. |---|---|---|---|
  111. | `*.csv *.tsv *.xlsx *.json *.jsonl *.parquet *.sqlite *.sqlite3 *.db` | tabular | `references/cleaning-rule-catalog.md#tabular` | profiler → stats (unit-aware) → sensitivity → charts |
  112. | `*.txt` (free text, no delimiter) | text-corpus | `references/cleaning-rule-catalog.md#text` | profiler (duplicates / empty / encoding) — **profile-only** |
  113. | timestamped tabular / `*.csv` with date column | time-series | `references/cleaning-rule-catalog.md#timeseries` | profiler (gaps / duplicate stamps / regime change / ADF screen) → **audited descriptive statistics**: Mann-Kendall trend + Theil-Sen slope with CI, ACF/PACF with white-noise bounds, STL strengths, seeded CUSUM changepoint screen (`analysis/timeseries.json`). Descriptive scope: NO forecast, NO ARIMA, NO causal claim |
  114. | `*.jpg *.png *.webp` | image-set | `references/cleaning-rule-catalog.md#image` | profiler (corrupt / near-duplicate / blank) — **profile-only** |
  115. | `*.wav *.mp3 *.flac` | audio-set | `references/cleaning-rule-catalog.md#audio` | profiler (duration / clipping / silence) — **profile-only** |
  116. | `*.tif` with geo tag | geospatial | `references/cleaning-rule-catalog.md#geo` | profiler (CRS / bounds / null island / coincidence) — **profile-only** |
  117. | `*.pkl *.pickle *.joblib *.dill *.pck *.pckl *.pt *.pth` | unknown | `references/research-protocol.md#unsafe-python-object-serialization` | refuse deserialization; report why and stop |
  118. | anything else / no extension | research-track | `references/research-protocol.md` | ad-hoc via WebSearch / WebFetch |
  119. Full table in [references/modality-routing.md](references/modality-routing.md).
  120. ## Workflow (agent-side)
  121. The agent executing this skill should follow this loop. Each step has full instructions in the linked reference.
  122. 0. **Prefer the installed CLI** (`data-master analyze` / `data-master resume`). The
  123. npm launcher runs inside its managed Python environment. Use the source runner
  124. (`python scripts/run_pipeline.py`) only when working from a checkout with the
  125. required dependencies already available in that interpreter. The runner wraps every
  126. deterministic stage, records `state.json`, can select the highest-ranked
  127. grouping column for an exploratory screen, and resumes without repeating
  128. stages whose evidence still exists:
  129. ```powershell
  130. data-master analyze <file|folder|glob> --out outputs --title "<title>"
  131. data-master resume outputs/state.json
  132. ```
  133. Use the step-by-step commands below only when debugging or overriding a stage.
  134. Non-tabular and multi-file inputs take the documented profile-only path: they
  135. receive a PDF report and integrity audit, but no inferential claim.
  136. 1. **Resolve input** (`references/workflow-contract.md#step-1`). If $ARGUMENTS is empty, list data-looking files in cwd and ask the user which to process. This is the only allowed clarification.
  137. 2. **Route modality** (`references/modality-routing.md`). Read file extensions, sniff a small sample, pick one of the 7 modalities. Mixed folder → run one branch per modality and merge reports.
  138. 3. **Check dependencies** (`scripts/check_deps.py --modalities <list>`). Do not run
  139. raw `pip install` commands or modify the system Python. After user approval, use
  140. `data-master setup` for declared dependencies or
  141. `data-master setup --extra <package>` for a format parser so installation stays in
  142. the data-master managed environment. If that launcher is unavailable, report the
  143. missing dependency and stop the affected branch.
  144. 4. **Profile** (`scripts/profile_<modality>.py`). Produce `profile.json`. Always do this; it grounds every later decision.
  145. 5. **Plan cleaning** (`scripts/cleaning_plan.py`). Read `references/cleaning-rule-catalog.md` and `references/never-automatic-list.md`. Produce `cleaning_plan.json`. Any never-automatic item becomes a USER_INPUT_NEEDED flag — and `answers.template.json` is written beside the plan so the user can answer every flag by id and re-run with `--answers <file>`. Every answer is validated against the flag it names and recorded with who/when/why.
  146. 6. **Execute cleaning** (`scripts/execute_cleaning.py`). Run only the safe subset automatically. Write `clean_<slug>.py` so the user can re-run. Write `cleaning_report.md`.
  147. 7. **Validate** (`scripts/validate_clean.py`). Gate: row loss < 30% (else fail), no NaN inflation, target/label cardinality preserved, no column dropped without a plan entry, and every dtype matches the recorded manifest. Note what `dtype_stable` compares: the cleaned types against **what the run decided and recorded**, not against what pandas guessed about the raw file. Comparing against the guess condemns the run for refusing to repeat the guess.
  148. 8. **Re-derive, do not re-read.** Every analysis stage opens the cleaned data through `dm_types.load_cleaned`, which applies the manifest. Never call `read_csv` on a cleaned file directly — that is the same type-guessing mistake one stage earlier.
  149. 9. **Stats** (`scripts/stats_descriptive.py` + `scripts/stats_inferential.py` + `scripts/effect_size.py`). Always run descriptive; run inferential only when the data has a comparison structure. A comparison is confirmatory only when the user explicitly declares `--unit-key`, `--group`, and `--outcomes`; otherwise record and label it as exploratory, and list the missing declarations in the report and `answers.template.json`. When `--unit-key` shows rows repeating within units, the inferential stage classifies the structure (clustered / paired / complete blocks / incomplete repeated) and runs the primary analysis at the level the structure supports — per-unit means, paired tests, Friedman, or a unit random-intercept mixed model fitted by maximum likelihood with convergence checks; failed fits remain unresolved. Declared `--covariates` make the ANCOVA comparison primary with the slopes assumption tested. Cite every p-value with test name, n, df, CI, effect size.
  150. 9b. **Time-series inputs** take their own mode: after profiling, `scripts/timeseries_analysis.py` computes Mann-Kendall + Theil-Sen slope with CI, ACF/PACF with white-noise bounds, STL strengths, and a seeded CUSUM changepoint screen into `analysis/timeseries.json`, rendered into the report with an overview figure and a phrase-scanned audit. Descriptive scope only.
  151. 10. **Sensitivity** (`scripts/sensitivity_analysis.py`). Perturb the primary analysis — leave-one-group-out, block jackknife, pre-specified extreme-value fence, strict complete cases, imputation sensitivity — and write `analysis/sensitivity.json`. Gate 6 reads this artifact; the rank cross-check alone no longer satisfies it.
  152. 11. **Charts** (`scripts/chart_decision.py` + `scripts/render_chart.py`). 1 chart per argument task. 2D preferred; 3D only when it changes the ranking. Title = conclusion, caption = definition, body = fact → interpretation → implication. Scatter panels carry a bootstrap band on the smoother and a CI on r.
  153. 12. **Report** (`scripts/assemble_report.py`). Builds `report.md`, `executive_summary.md`,
  154. `report_build.json`, then renders `report.pdf` through reportlab -> fpdf2 ->
  155. headless browser -> BLOCKED. The backend that actually ran is recorded, never assumed:
  156. ```powershell
  157. python scripts/assemble_report.py --raw <raw.csv> --outdir outputs --title "<title>"
  158. ```
  159. 13. **Audit** (`scripts/final_audit.py`). Re-derives all 8 gates from the artifacts (it
  160. does not trust the prose), routes the run to its EQUATOR checklist and journal family,
  161. and writes `audit/final_audit.md`. Exit 1 means the report exists but is not
  162. submission-ready:
  163. ```powershell
  164. python scripts/final_audit.py --outdir outputs --journal nature
  165. python scripts/final_audit.py --outdir outputs --journal nejm --design randomized_trial
  166. ```
  167. 14. **State** (`scripts/state_keeper.py`). Inspect or validate `state.json`; resume
  168. with either `state_keeper.py --resume` or `run_pipeline.py --resume <state>`. Saved artifacts are checked by SHA-256; altered evidence, code, dependencies, or options invalidate dependent stages. Verify without rebuilding using `python scripts/verify_artifacts.py --outdir outputs --json`. Legacy state without hashes is rebuilt.
  169. `--export-lockfile` writes `lockfile.txt` pinning the exact versions that ran.
  170. Agent meta-rules:
  171. - If a step is missing a reference, stop and read the linked file — do NOT improvise on methods.
  172. - For unknown formats, follow `references/research-protocol.md`: inspect file details
  173. locally, search only public format names, and never send file-derived bytes, headers,
  174. metadata, paths, or values to WebSearch or WebFetch.
  175. - Never deserialize Pickle-based files (`.pkl`, `.pickle`, `.joblib`, `.dill`, `.pck`,
  176. `.pckl`, `.pt`, `.pth`) by default. Keep them on the unknown route and report that
  177. loading was refused because deserialization can execute code.
  178. - If a Python script fails with ImportError, fall back per workflow-contract step 0 (use py -3.12).
  179. - If the user explicitly asks for a non-default method (e.g. Welch t-test), record the override in `cleaning_plan.json` and proceed.
  180. ## Nature-grade standards (default)
  181. Drawn from `nature-statistics` and applied as a gate, not decoration:
  182. 1. **n and the question are explicit.** The user declares the independent unit (`--unit-key`), grouping column (`--group`), and outcomes (`--outcomes`) before a comparison is labelled confirmatory. Automatic selections stay exploratory. Independent experimental unit ≠ technical replicate ≠ repeated measure. When rows repeat within the declared unit, the primary analysis runs at the level the structure supports (unit means / paired / blocks), with the row-level results demoted to a labelled secondary.
  183. 2. **Missingness assumptions are explicit.** The mechanism stays undetermined without collection context; missing fractions never establish MCAR. Any imputation is labelled.
  184. 3. **Effect size > p-value.** Cohen d / Cliff delta / odds ratio + 95% CI always reported.
  185. 4. **Uncertainty is visible.** Every number has a CI / SE / IQR; every chart has error bars, a band, or a stated smoother interval.
  186. 5. **Multiple comparisons corrected.** Bonferroni / BH / Holm as appropriate.
  187. 6. **Sensitivity tested — for real.** `scripts/sensitivity_analysis.py` re-runs the primary question on perturbed data (leave-one-group-out, block jackknife, extreme-value fence, strict complete cases, imputation sensitivity) and writes `analysis/sensitivity.json`; Appendix A1 reports every conflict. The rank cross-check alone is assumption robustness, not sensitivity, and no longer passes this gate.
  188. 7. **No upgrade from association to mechanism.** Claims stay bounded to evidence.
  189. 8. **Unknowns are flagged AUTHOR_INPUT_NEEDED.** Never invented sample sizes, p-values, or methods.
  190. For `standard` or `executive` templates, gates 3-6 are relaxed but 1, 2, 7, 8 remain.
  191. ## Default-FAIL contract (from my-mathmodel-agent)
  192. Every gate starts `passes: false`. A gate flips to `true` only when its acceptance criterion has machine-checkable evidence: a saved JSON, a passing script exit, a rendered figure with its source script. A claim is not evidence.
  193. ## File map
  194. ```
  195. data-master/
  196. |-- SKILL.md # this file
  197. |-- docs/DESIGN.md # architectural design
  198. |-- references/
  199. | |-- modality-routing.md # full routing table + sample sniffers
  200. | |-- cleaning-rule-catalog.md # rule catalog per modality
  201. | |-- never-automatic-list.md # ops that always require human decision
  202. | |-- nature-statistics-checklist.md # 8 Nature-grade gates
  203. | |-- journal-reporting-standards.md # NEJM / Science / Lancet / Nature routing
  204. | |-- journal-reporting-standards.json# machine-readable routing table
  205. | |-- chart-decision-catalog.md # 25 chart types + 8 complexity principles
  206. | |-- report-template-catalog.md # 3 report templates
  207. | |-- workflow-contract.md # per-step agent contract
  208. | `-- research-protocol.md # how to handle unknown modalities
  209. |-- skills/ # 10 sub-skills, one stage each
  210. |-- scripts/ # deterministic Python tools
  211. | |-- assemble_report.py # markdown + PDF cascade + provenance
  212. | |-- sensitivity_analysis.py # LOGO / jackknife / fence / complete-case perturbations
  213. | |-- final_audit.py # 8 gates + journal checklist
  214. | `-- profile_tabular.py ... # profiler, cleaner, stats, charts
  215. |-- examples/iris/ # end-to-end iris example
  216. |-- docs/assets/ # README screenshots
  217. `-- .github/workflows/ci.yml # cross-platform regression CI
  218. ```
  219. ## Related files
  220. | Topic | File | Load when |
  221. |---|---|---|
  222. | Modality routing + sniffers | references/modality-routing.md | Step 2 — picking the modality |
  223. | Cleaning rule catalog | references/cleaning-rule-catalog.md | Step 5 — building cleaning plan |
  224. | Never-automatic operations | references/never-automatic-list.md | Step 5 — flagging ops for human |
  225. | Nature checklist | references/nature-statistics-checklist.md | Step 7 + 12 — validation + audit |
  226. | Journal standards (NEJM / Science / Lancet / Nature) | references/journal-reporting-standards.md | Step 12 — choosing the EQUATOR checklist |
  227. | Chart decision catalog | references/chart-decision-catalog.md | Step 10 — picking the figure |
  228. | Report templates | references/report-template-catalog.md | Step 11 — assembling the report |
  229. | Workflow contract | references/workflow-contract.md | always — the per-step rules |
  230. | Research protocol | references/research-protocol.md | Step 2 — unknown modality |
  231. | Design document | docs/DESIGN.md | when user wants the why |
  232. ## Source hierarchy
  233. 1. User-supplied data + their override request (highest priority).
  234. 2. Nature Portfolio reporting standards + CONSORT / STROBE / PRISMA / ARRIVE when relevant.
  235. 3. This skill's reference files (built from nature-statistics, himcm-chart-system, data-storytelling, my-mathmodel-agent, clean-data, magic-data).
  236. 4. Industry tools (ydata-profiling / great-expectations / deepchecks) as inspiration, not blind adoption.
  237. 5. Conservative statistical reporting practice.
  238. If the supplied material is insufficient for a defensible decision, leave the field as AUTHOR_INPUT_NEEDED rather than guessing.
  239. ## Red lines
  240. - Never modify, move, or delete the user's raw data files.
  241. - Never invent p-values, sample sizes, effect sizes, CIs, or test names.
  242. - Never call a comparison "significant" without n, test, df, p, effect size, and CI.
  243. - Never silently drop rows, impute, merge, deduplicate across datasets, or publish.
  244. - Never let Nature standards silently decay to "standard" without telling the user.
  245. - Never start writing the report before every gate has machine-checkable proof.
README.md · README · 238 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # data-master
  2. **Turn a messy dataset into an auditable PDF report from one command.**
  3. [![CI](https://github.com/MAGA2010/data-master/actions/workflows/ci.yml/badge.svg)](https://github.com/MAGA2010/data-master/actions/workflows/ci.yml)
  4. [![Release](https://img.shields.io/github/v/release/MAGA2010/data-master)](https://github.com/MAGA2010/data-master/releases)
  5. [![GitHub stars](https://img.shields.io/github/stars/MAGA2010/data-master?style=flat)](https://github.com/MAGA2010/data-master/stargazers)
  6. [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-3776AB)](https://www.python.org/)
  7. [![License: MIT](https://img.shields.io/badge/license-MIT-2EA44F)](LICENSE)
  8. [![Codex Skill](https://img.shields.io/badge/Codex-skill-111827)](SKILL.md)
  9. [中文](README.zh-CN.md) · [Examples](#reproducible-case-studies) · [Evaluation](EVALUATION.md) · [Roadmap](ROADMAP.md) · [Launch kit](docs/LAUNCH_KIT.md)
  10. ![data-master turns raw data into a verified report](docs/assets/demo.gif)
  11. `data-master` preserves the raw input, makes every cleaning decision explicit,
  12. runs statistics only at the level the data supports, tests sensitivity, and
  13. delivers Markdown, PDF and machine-readable evidence. It is a Codex skill and a
  14. standalone cross-platform CLI.
  15. ## Try it
  16. Requires Node.js 20+ and Python 3.11+.
  17. ```bash
  18. npm install -g github:MAGA2010/data-master#v0.3.0
  19. data-master setup
  20. data-master skill install
  21. data-master demo --out outputs/first-demo
  22. ```
  23. For an additional format parser, install it only into the managed environment:
  24. ```bash
  25. data-master setup --extra h5py
  26. ```
  27. Optional format parsers are never installed automatically. This option accepts a
  28. package name only; use a parser named in its official format documentation.
  29. Open `outputs/first-demo/report.pdf`, then inspect the verdict in
  30. `outputs/first-demo/audit/final_audit.md`.
  31. If the verified demo is useful, [star the repository](https://github.com/MAGA2010/data-master) so other researchers and analysts can find it.
  32. The npm package has no install hooks and no JavaScript dependencies. `setup` is
  33. the only command that downloads Python dependencies; it creates an isolated
  34. environment. `skill install` atomically installs the bundled skill and refuses
  35. to overwrite an existing copy.
  36. ## Why use it
  37. | A plausible analysis can fail when… | data-master records or prevents it by… |
  38. |---|---|
  39. | `02134` silently becomes `2134` | Reading delimited cells as text until an explicit conversion is approved |
  40. | Repeated rows inflate the sample size | Declaring the independent unit and moving the primary analysis to that level |
  41. | Cleaning alters an unrelated column | Comparing untouched columns with the raw input in text space |
  42. | A p-value appears without context | Saving the test, n, df, effect size, interval and correction family |
  43. | “Robust” means one convenient re-fit | Running saved leave-one-group-out, jackknife, fence and complete-case perturbations |
  44. | A polished report hides an unresolved decision | Producing the report and a separate audit that can still refuse submission |
  45. | A resumed run reuses changed evidence | Hashing stage artifacts, code, references, dependencies and run options |
  46. The audit enforces reporting discipline. It does not replace statistical
  47. judgment, collection context, peer review or a study protocol.
  48. ## What you receive
  49. ```text
  50. outputs/
  51. |-- report.pdf primary human-readable report
  52. |-- report.md citable report source
  53. |-- report.html self-contained shareable export of report.md
  54. |-- executive_summary.md
  55. |-- state.json stage ledger and provenance
  56. |-- answers.template.json decisions that require a human
  57. |-- analysis/
  58. | |-- profile.json
  59. | |-- descriptive.json
  60. | |-- inferential.json
  61. | `-- sensitivity.json
  62. |-- cleaned/
  63. | |-- cleaned.csv
  64. | |-- clean_<dataset>.py replayable cleaning program
  65. | `-- validation.json
  66. |-- charts/
  67. `-- audit/
  68. |-- final_audit.md
  69. `-- final_audit.json
  70. ```
  71. Verify a delivered run without modifying it:
  72. ```bash
  73. data-master verify outputs/first-demo --json
  74. ```
  75. ## Analyze your data
  76. ```bash
  77. data-master analyze measurements.csv \
  78. --out outputs/study \
  79. --group treatment \
  80. --unit-key participant_id \
  81. --outcomes response_time,accuracy
  82. ```
  83. Resume interrupted work:
  84. ```bash
  85. data-master resume outputs/study/state.json
  86. ```
  87. If a conversion or study decision is unsafe to infer, the run writes
  88. `answers.template.json`. Fill its stable decision IDs and resume with:
  89. ```bash
  90. data-master resume outputs/study/state.json --answers outputs/study/answers.json
  91. ```
  92. Comparisons are labelled **exploratory** unless the run explicitly declares
  93. `--unit-key`, `--group`, and `--outcomes`. The report and `answers.template.json`
  94. show which declarations are missing. Supply all three to treat the comparison as
  95. confirmatory.
  96. Use `data-master analyze --help` for all pipeline options, including declared
  97. covariates, journal/checklist routing, input-size limits, lockfile export and PDF
  98. backend selection.
  99. Unknown binary formats are profiled locally. File bytes, headers, metadata and records
  100. are never sent to web search. Pickle-based files (`.pkl`, `.pickle`, `.joblib`, `.dill`,
  101. `.pck`, `.pckl`, `.pt`, `.pth`) are refused by default because loading them can execute
  102. code.
  103. ## Reproducible case studies
  104. | Case | Failure being tested | Evidence |
  105. |---|---|---|
  106. | [Identifier traps](docs/case-studies/identifier-traps.md) | Leading zeros, dates, currency and mixed values are silently retyped | Plan, replay and untouched-column validation |
  107. | [Date or number?](docs/case-studies/identifier-traps.md#benchmark-date-or-number) | `20240115` is an integer and a date; the file does not say which | `coerce_numeric_or_date` hold with evidence and an answers-file resolution |
  108. | [Synthetic survey](docs/case-studies/survey-data.md) | Survey labels, -999 missing codes and zero-padded ids are silently retyped or imputed | Three evidenced holds; labels and codes survive cleaning |
  109. | [Repeated rows](docs/case-studies/unit-aware-comparison.md) | Technical or repeated observations are counted as independent samples | Unit-level primary analysis and labelled row-level secondary |
  110. | [Audit refusal](docs/case-studies/audit-refusal.md) | A complete-looking report is treated as submission-ready without a defined unit | Report delivered; audit gate remains failed |
  111. Every case links its raw synthetic input, command and inspectable artifacts.
  112. ## Supported inputs
  113. | Input | Current behavior |
  114. |---|---|
  115. | CSV, TSV, Excel, JSON, JSONL, Parquet, single-table SQLite (`.sqlite`, `.sqlite3`, `.db`) | Full profile, cleaning, unit-aware analysis, sensitivity, charts, report and audit |
  116. | Timestamped tabular data | Audited descriptive trend, slope interval, autocorrelation, decomposition and changepoint screen |
  117. | Text, image, audio or geospatial data | Explicitly labelled integrity profile and profile-only report |
  118. | Mixed folders | Per-modality profiles and a combined profile-only report |
  119. Full inferential analysis currently applies to one tabular file. Time-series
  120. mode is descriptive and does not forecast. Unsupported methods remain marked as
  121. unsupported instead of being replaced with a weaker claim.
  122. ## Other installation paths
  123. Install the Python package from a clone:
  124. ```bash
  125. git clone https://github.com/MAGA2010/data-master.git
  126. cd data-master
  127. python -m venv .venv
  128. # Windows: .venv\Scripts\python.exe -m pip install .
  129. # macOS/Linux: .venv/bin/python -m pip install .
  130. # Windows: .venv\Scripts\data-master.exe doctor
  131. # macOS/Linux: .venv/bin/data-master doctor
  132. # Windows: .venv\Scripts\data-master.exe skill install
  133. # macOS/Linux: .venv/bin/data-master skill install
  134. ```
  135. The Python package path uses the activated project virtual environment. The
  136. `data-master setup --extra <package>` helper is provided by the npm launcher; when it
  137. is unavailable, an unknown-format branch stays profile-only instead of installing a
  138. parser into another interpreter.
  139. Run directly from a checkout:
  140. ```bash
  141. python scripts/run_pipeline.py \
  142. --in examples/iris/iris_dirty.csv \
  143. --out outputs/iris-demo \
  144. --title "Iris morphometrics" \
  145. --journal nature
  146. ```
  147. Use a release tag or commit SHA in GitHub npm specifications when reproducible
  148. installation matters. The project is not currently published to the npm or
  149. PyPI registries.
  150. ## Evidence and methodology
  151. - [Evaluation dashboard](EVALUATION.md)
  152. - [Validation and release checks](docs/VALIDATION.md)
  153. - [Architecture and design decisions](docs/DESIGN.md)
  154. - [Workflow contract](references/workflow-contract.md)
  155. - [Cleaning rules](references/cleaning-rule-catalog.md)
  156. - [Statistical reporting gates](references/nature-statistics-checklist.md)
  157. - [Journal and EQUATOR routing](references/journal-reporting-standards.md)
  158. - [Current roadmap](ROADMAP.md)
  159. CI exercises Windows and Linux with Python 3.11, 3.12 and 3.13, and runs the
  160. distribution suite on Windows, Linux and macOS with Python 3.13. Numerical
  161. tests compare key methods with independent equations or reference
  162. implementations. Distribution tests build and install the actual wheel and npm
  163. tarball outside the checkout before running the demo and verifying its
  164. evidence. The skill has also been installed and driven end to end in ZCode
  165. 3.14.4 on Windows (see [Evaluation](EVALUATION.md)).
  166. ## Scope boundaries
  167. - Missingness mechanisms remain undetermined without collection context.
  168. - Covariates are declared by the user; the pipeline does not select them from outcomes.
  169. - Failed mixed models remain unresolved rather than falling back to an invalid independent-row test.
  170. - Forecasting, causal identification and arbitrary row deletion are outside the current contract.
  171. - Artifact verification detects changes against the saved ledger; it cannot establish authenticity if both artifacts and ledger are rewritten.
  172. See the [open roadmap](ROADMAP.md) for planned compatibility and evaluation work.
  173. ## Contributing
  174. External datasets, failure reports and small focused pull requests are welcome.
  175. Start with [CONTRIBUTING.md](CONTRIBUTING.md) and the
  176. [contribution-sized roadmap tasks](ROADMAP.md#contribution-sized-tasks).
  177. A statistical decision change needs a regression or numerical-reference test
  178. that would have failed before the change. Remove personal and confidential data
  179. before opening an issue.
  180. ## Citation and license
  181. Citation metadata is available in [CITATION.cff](CITATION.cff). Every run records
  182. the package version or repository commit that produced it.
  183. Released under the [MIT License](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.