experiment-agent
"Experiment executor and monitor for academic research. 2-agent system covering code experiments (ML training, statistical analysis, ETL, simulation) and human studies (surveys, field studies, interviews). 4 modes: run (execute + monitor code), manage (track human studies), validate (statistical interpretation + reproducibility verification), plan (Socratic experiment design). Use when a researcher wants to run or monitor a research experiment, manage or resume a human-subject study, check the statistics or reproducibility of research results, or design an experiment or study, in English or Chinese (e.g., 跑實驗、管理研究、驗證結果、規劃實驗)."
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/imbad0202--experiment-agent)HTML
<a href="/listings/imbad0202--experiment-agent"><img src="/api/listings/imbad0202--experiment-agent/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
1. Clone this repo into your project or `.claude/skills/` 2. Start a Claude Code session 3. Try: "Run my analysis: `Rscript analysis.R`"
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 220 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: experiment-agent
- description: "Experiment executor and monitor for academic research. 2-agent system covering code experiments (ML training, statistical analysis, ETL, simulation) and human studies (surveys, field studies, interviews). 4 modes: run (execute + monitor code), manage (track human studies), validate (statistical interpretation + reproducibility verification), plan (Socratic experiment design). Use when a researcher wants to run or monitor a research experiment, manage or resume a human-subject study, check the statistics or reproducibility of research results, or design an experiment or study, in English or Chinese (e.g., 跑實驗、管理研究、驗證結果、規劃實驗)."
- metadata:
- version: "1.2.0"
- last_updated: "2026-09-25"
- author: "Cheng-I Wu"
- license: "CC-BY-NC 4.0"
- status: active
- related_skills:
- - academic-pipeline
- - deep-research
- - academic-paper
- - academic-paper-reviewer
- ---
- # Experiment Agent v1.2.0 — Experiment Executor and Monitor
- Execute, monitor, interpret, and verify experiments for academic research. Works independently or as an optional bridge between ARS Stage 1 (RESEARCH) and Stage 2 (WRITE).
- **Role**: Executor + Monitor. This skill does NOT judge whether results are good for a paper (that is the reviewer's job). It ensures experiments complete successfully, interprets statistical output, and verifies reproducibility.
- ## Quick Start
- **Run a code experiment:**
- ```
- Run my training script: python train.py --epochs 50 --output results/
- ```
- **Manage a human study:**
- ```
- Help me manage my survey study — I need 200 responses by May 30
- ```
- **Validate results:**
- ```
- Validate these regression results: results/analysis_output.csv
- ```
- **Plan an experiment:**
- ```
- Help me design an experiment to test whether AI tools improve QA officer productivity
- ```
- ---
- ## Trigger Keywords
- **English**: run experiment, execute code, train model, benchmark, analyze data, manage study, track participants, field study, survey, validate results, check statistics, reproduce, re-run, plan experiment, design study, what should I test
- **Chinese**: 跑實驗, 執行程式, 訓練模型, 基準測試, 分析資料, 管理研究, 追蹤參與者, 田野研究, 問卷, 驗證結果, 檢查統計, 重現, 規劃實驗, 設計研究
- ---
- ## Modes
- | Mode | Purpose | Agent | Spectrum |
- |------|---------|-------|----------|
- | `run` | Execute code experiments + real-time monitoring | code_runner_agent | Fidelity |
- | `manage` | Manage human study workflow + progress tracking | study_manager_agent | Balanced |
- | `validate` | Statistical interpretation + reproducibility verification | SKILL.md (stats) + code_runner_agent (re-run) | Fidelity |
- | `plan` | Socratic dialogue to design experiments | SKILL.md direct | Originality |
- ## Mode Selection
- | User Signal | Mode |
- |-------------|------|
- | Has a script/command to run | `run` |
- | Running a survey, interview, field study, lab experiment | `manage` |
- | Has results, wants to check numbers or reproduce | `validate` |
- | Wants to figure out what experiment to do | `plan` |
- | Ambiguous | Ask: "Are you running code or managing a human study?" |
- ---
- ## Routing
- 1. Detect intent from user's first message using trigger keywords
- 2. Code execution keywords → dispatch `code_runner_agent` (run mode)
- 3. Human study keywords → dispatch `study_manager_agent` (manage mode)
- - **Session resume**: If the user's first message in a session matches `resume <argument>` (where argument is a study_id slug or a path to a state.md file), OR if any later turn matches `resume <argument>` and no artifact write has occurred this session, route to study_manager_agent's RESUME entry path. The agent will read the artifact, validate, and prompt user confirmation before resuming the study at its last known phase.
- 4. Validation keywords → enter validate mode (handled inline, see below)
- 5. Design keywords → enter plan mode (handled inline, see below)
- Dispatching or delegating to an agent in this skill means reading its file under `agents/` and following it yourself in this conversation: both agents need the user mid-task (confirming a command, choosing what to do about an anomaly, answering protocol questions one at a time).
- ### Runtime Requirements
- Most modes work with any LLM runtime that supports prompt + reasoning.
- **Session resume in `manage` mode** additionally requires the runtime to provide Read, Write, and Edit tool access to the local filesystem. Claude Code provides these. Runtimes that surface only chat I/O can use PLAN and ETHICS in-session, but study state will not persist across restarts, the `resume <study_id>` command will be unavailable, and a study cannot move to TRACK.
- **The study state checker** (`scripts/check_study_state.py`) validates `manage` mode's study state file and derives its ethics status. It needs a command tool (Bash in Claude Code), Python 3.9 or later, and PyYAML (`python3 -m pip install pyyaml`). Without it, `manage` mode still plans, runs the ethics checklist, and tracks studies already in TRACK, applying the rules by hand and saying so, but it does not move a study from ETHICS to TRACK.
- ---
- ## validate Mode (Inline)
- Two capabilities: **statistical interpretation** and **reproducibility verification**. Accepts results from any source (this agent's run/manage modes, external files, ARS pipeline output).
- ### Procedure
- 1. **DETECT** — Scan user-provided files for statistical content (p-values, CIs, effect sizes, coefficients, test statistics). Structured formats (CSV/JSON) are parsed directly; from unstructured output, extract the values yourself and have the user confirm them before interpreting.
- 2. **INTERPRET** — Item-by-item analysis. See `references/statistical_interpretation_guide.md` for full protocol covering: significance, effect size classification, CI assessment, assumption verification, multiple comparison correction.
- 3. **FALLACY SCAN** — Check 11 known statistical fallacy patterns (structural, inferential, causal). See `references/statistical_interpretation_guide.md` for the full checklist. All 11 must be checked; report coverage in output.
- 4. **REPRODUCE** (optional, code experiments only) — If user provides executable command + original results, delegate to code_runner_agent for re-run, then compare. See `references/reproducibility_protocol.md`. Not applicable to collecting human data again or to non-rerunnable external systems. A script that analyzes participant data can be re-run under the participant data rules.
- 5. **REPORT** — Produce validation report in Markdown structured format (see `templates/output_formats.md`). Use `Verification Status: ANALYZED` for stats-only or non-rerunnable cases, and `VERIFIED` only after a successful reproducibility re-run. If the user says the results go to ARS, add the ARS Provenance Draft.
- **Scope boundary**: validate mode describes what numbers say and flags potential fallacies. Its flag levels describe the checks, not the result, and its `What to check` column names what to look at. It does NOT make editorial recommendations about what to write in the paper — that is the ARS reviewer's job.
- **Participant data**: validate reads results, not participant records. Before you open a result file, ask whether it holds row-level records from human participants, unless the user already said. If the answer is yes or "not sure", do not open it: ask for the aggregate output. If a file turns out to hold row-level records, stop reading it, do not quote it, and tell the user that those rows have reached the model provider.
- ---
- ## plan Mode (Inline)
- Socratic dialogue to help users design experiments before running them. plan mode helps the user clarify their thinking — it does not prescribe a specific design. The user makes all design decisions.
- ### Procedure
- 1. **Clarify RQ** — What are you trying to test? What is the hypothesis?
- 2. **Variables** — Identify IV, DV, control variables, potential confounds
- 3. **Design** — Experimental / quasi-experimental / observational / mixed methods?
- 4. **Method selection** — Based on RQ + design, suggest appropriate methods
- 5. **Sample** — Population, sampling strategy, power analysis for sample size
- 6. **Analysis strategy** — Which statistical tests? What are the assumptions?
- 7. **Produce plan** — Output a structured experiment plan using `templates/code_experiment_plan.md` or `templates/study_protocol.md`
- One question at a time. Multiple choice preferred. If user brings ARS Stage 1 output (RQ Brief, Methodology Blueprint), parse section headings and pre-populate steps 1-4.
- ---
- ## Output Formats
- All outputs use **Markdown-based structured format** with a Material Passport header modeled on ARS Schema 9. It is not an ARS passport (see `references/ars_integration_guide.md`). Each output starts with a `## Material Passport` header followed by the mode-specific content.
- See `templates/output_formats.md` for complete templates for the three execution/validation outputs:
- - **Experiment Result** (run mode): Material Passport + ID, type, status, command, participant data fields, output files, anomalies
- - **Study Status** (manage mode): Material Passport + ID, phase, progress, ethics status, risks, data readiness
- - **Validation Report** (validate mode): Material Passport + statistical findings table, warnings, fallacy scan, reproducibility verdict, and an ARS provenance draft when the results go to ARS
- Plan mode outputs use separate templates and also carry Material Passport:
- - **Code Experiment Plan** (plan mode, code path): `templates/code_experiment_plan.md`
- - **Study Protocol** (plan mode, human-study path): `templates/study_protocol.md`
- ---
- ## Quality Standards
- | Standard | Requirement |
- |----------|-------------|
- | Monitoring coverage | Every code experiment must have at least process-alive + timeout monitoring |
- | Statistical rigor | All 11 fallacy types must be checked in validate mode; coverage reported |
- | Reproducibility | Deterministic experiments: exact match required. Stochastic: < 5% relative diff default. Environment-sensitive: < 10% relative diff default (see `references/reproducibility_protocol.md`) |
- | Passport header | All outputs include a Material Passport header modeled on ARS Schema 9 |
- | User sovereignty | All anomaly detections are ADVISORY; only hard timeout auto-kills |
- ---
- ## Safety Rules
- | # | Rule |
- |---|------|
- | 1 | Only execute user-specified commands — never auto-generate or modify scripts. Exception: `manage` mode runs this skill's read-only study state checker (`scripts/check_study_state.py`) |
- | 2 | Never auto-retry crashed experiments — notify user, user decides |
- | 3 | Never auto-kill except hard timeout — notify before kill |
- | 4 | Monitor only user-specified output paths |
- | 5 | Never upload data to external services |
- | 6 | Never read raw participant data into the conversation — track metadata only (counts, rates). `run` can start the user's own script on such data after the checks in `agents/code_runner_agent.md` |
- | 7 | Never send notifications to study participants |
- | 8 | Power analysis uses conservative estimates |
- | 9 | Statistical interpretation is descriptive — does not draw conclusions for user |
- | 10 | RED_FLAG means "needs user attention", not "result is wrong" |
- ---
- ## Anti-Patterns
- | # | Anti-Pattern | Why It's Wrong |
- |---|-------------|---------------|
- | 1 | Auto-modifying user's experiment code | Violates safety rule 1; user owns their code |
- | 2 | Silently retrying a crashed run | Masks the real error; wastes compute |
- | 3 | Reporting p < .05 as "the result is significant" without effect size | Statistical significance without practical significance is misleading |
- | 4 | Skipping fallacy scan because "results look clean" | Fallacies are invisible without systematic checking |
- | 5 | Making editorial recommendations in validate mode | That's the reviewer's job, not ours |
- ---
- ## Reference Files
- | File | Purpose |
- |------|---------|
- | `references/stall_detection_protocol.md` | Monitoring thresholds, anomaly types, detection logic |
- | `references/irb_ethics_checklist.md` | Human study ethics review checklist |
- | `references/statistical_interpretation_guide.md` | Full statistical interpretation + 11-type fallacy scan protocol |
- | `references/reproducibility_protocol.md` | Re-run methodology, comparison thresholds, verdict criteria |
- | `references/ars_integration_guide.md` | ARS Material Passport, handoff format, pipeline bridging |
- | `references/study_state_protocol.md` | Canonical reference for the study state artifact format used by `manage` mode session resume: schema, write/resume protocols, validation rules, prompt-injection guard, IRB approval reconfirmation set. |
- | `scripts/check_study_state.py` | Validates a study state file and derives its ethics status for `manage` mode (Python 3.9+, PyYAML). |
- | `templates/output_formats.md` | Complete Markdown output templates for all three output types |
- ---
- ## ARS Integration (Optional)
- This skill works independently. When used with ARS:
- - **Consuming ARS output**: Recognizes ARS Stage 1 section headings (`## Research Question Brief`, `## Methodology Blueprint`) to pre-populate plan/manage modes
- - **Producing output for ARS**: All outputs carry a Material Passport header modeled on Schema 9. It is not an ARS passport. Users bring results to ARS Stage 2 manually. ARS asks each scholar about their experiments and has them enter the details; validate mode can give a draft of those details to check.
- - **ARS requires zero modification**: No new pipeline stages, no dependencies. The user is the bridge.
- See `references/ars_integration_guide.md` for details.
- ---
- *Experiment Agent v1.2.0 | 2026-09-25 | CC-BY-NC 4.0 | Cheng-I Wu*
README.md · README · 125 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # Experiment Agent
- [](https://github.com/Imbad0202/experiment-agent/releases)
- [](https://creativecommons.org/licenses/by-nc/4.0/)
- [](https://buymeacoffee.com/crucify020v)
- [繁體中文版](README.zh-TW.md)
- A Claude Code skill for executing, monitoring, interpreting, and verifying experiments in academic research.
- ## What's new in v1.2.0
- - `manage` mode checks each study state file with a program, `scripts/check_study_state.py`. A study moves on to participant recruitment and data collection only when the program reports READY, and its data goes to analysis only when the ethics status is READY. If the program cannot run, the agent applies the rules by hand and says so, and it does not move a study into data collection.
- - After an IRB approval, the reconfirmation items must be answered again, later than the approval, before the study is READY.
- - Moving a study into data collection now needs Python 3.9 or later and PyYAML; see Requirements for human studies below.
- - Study state files are checked more strictly, so a file that v1.1.0 accepted can be reported INVALID. After upgrading, run the checker on each study state file before you resume the study (CHANGELOG.md, Compatibility).
- Full notes: [CHANGELOG.md](CHANGELOG.md)
- ## What It Does
- - **Runs code experiments** — executes scripts (Python, R, etc.), monitors for stalls/crashes in real-time, collects results
- - **Manages human studies** — plans protocols, checks IRB ethics, tracks data collection progress
- - **Interprets statistics** — reads p-values, effect sizes, CIs; checks 11 types of statistical fallacies (Simpson's Paradox, survivorship bias, etc.)
- - **Verifies reproducibility** — re-runs experiments and compares results
- ## Why It Exists
- Lu et al. (2026, *Nature*) demonstrated an Experiment Progress Manager for autonomous AI research. This skill brings the same execute-and-monitor capability to human-in-the-loop academic workflows — without the risks of full automation.
- ## Modes
- | Mode | What It Does |
- |------|-------------|
- | `run` | Execute code + monitor process |
- | `manage` | Plan + track human studies |
- | `validate` | Statistical interpretation + reproducibility check |
- | `plan` | Socratic dialogue to design experiments |
- ## Quick Start
- 1. Clone this repo into your project or `.claude/skills/`
- 2. Start a Claude Code session
- 3. Try: "Run my analysis: `Rscript analysis.R`"
- ## ARS Compatibility
- This skill works independently. It also integrates optionally with [Academic Research Skills (ARS)](https://github.com/Imbad0202/academic-research-skills):
- - Reads ARS Stage 1 output (RQ Brief, Methodology Blueprint) to pre-populate experiment design
- - Produces output with a Material Passport header and this skill's own verification status, for you to bring to ARS Stage 2. It is not an ARS passport: ARS sets its own status
- - ARS requires zero modification — the user bridges manually
- - ARS asks about your experiments and has you enter each one (`repro_lock`, `planned_vs_executed`, `negative_results`, `known_limitations`). `validate` can give you a draft of the last three to check; you enter `repro_lock` yourself
- ### When to use with ARS
- In the ARS pipeline, experiment-agent fits **between Stage 1 (RESEARCH) and Stage 2 (WRITE)**:
- ```
- ARS Stage 1 RESEARCH → you get RQ Brief + Methodology Blueprint
- ↓
- [pause ARS pipeline]
- ↓
- experiment-agent → plan → run/manage → validate → get analyzed or verified results
- ↓
- [resume ARS pipeline]
- ↓
- ARS Stage 2 WRITE → write paper using your experiment results
- ```
- Use experiment-agent when your research requires running experiments (code or human studies) before writing. For secondary data analysis, use `run` and `validate`; if the data come from people, `run` asks about ethics approval first. If your paper is purely a literature review, you don't need this — go directly from ARS Stage 1 to Stage 2.
- ### How to load
- **Step 1**: Clone this repo alongside your ARS project (or anywhere on your machine):
- ```bash
- cd ~/Projects
- git clone https://github.com/Imbad0202/experiment-agent.git
- ```
- **Step 2**: When you need to run experiments, open a Claude Code session **in the experiment-agent directory**:
- ```bash
- cd ~/Projects/experiment-agent
- claude
- ```
- **Step 3**: Paste the relevant ARS Stage 1 output (RQ Brief, Methodology Blueprint) into the session. The agent will auto-detect the ARS headings and pre-populate your experiment plan.
- **Step 4**: After your experiments are done and validated, copy the output (which includes a Material Passport header and verification status) back into your ARS session to continue Stage 2.
- ### Requirements for human studies
- `manage` mode checks each study's state file with `scripts/check_study_state.py`, which needs Python 3.9 or later and PyYAML:
- ```bash
- python3 -m pip install pyyaml
- ```
- If pip stops with an `externally-managed-environment` error, follow that message's instructions to install PyYAML for the `python3` on your PATH. Claude Code asks for permission before it runs the checker; you can choose not to be asked again. Without the checker, `manage` mode still plans studies and runs the ethics checklist, but it will not move a study into data collection.
- > You can also add this skill to any project via `.claude/skills/` symlink — see [Claude Code docs](https://docs.anthropic.com/en/docs/claude-code) for skill installation.
- ## Safety
- - Only executes commands you specify — never auto-generates or modifies your code. The one exception: `manage` mode runs this skill's read-only study state checker
- - Never auto-retries crashed experiments
- - Never reads raw participant data into the conversation. `run` can start your own script on such data after you answer its ethics questions, and then reports only file names and sizes, plus any files you name as aggregate-only
- - Statistical interpretation describes, never concludes
- - Full list: see SKILL.md Safety Rules
- ## License
- CC-BY-NC 4.0
- ## Author
- 吳政宜 Edward Cheng-I Wu
- ---
- ## Changelog
- See [CHANGELOG.md](CHANGELOG.md)
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.