Skip to content
Back to search

experiment-agent

"Experiment executor and monitor for academic research. 2-agent system covering code experiments (ML training, statistical analysis, ETL, simulation) and human studies (surveys, field studies, interviews). 4 modes: run (execute + monitor code), manage (track human studies), validate (statistical interpretation + reproducibility verification), plan (Socratic experiment design). Use when a researcher wants to run or monitor a research experiment, manage or resume a human-subject study, check the statistics or reproducibility of research results, or design an experiment or study, in English or Chinese (e.g., 跑實驗、管理研究、驗證結果、規劃實驗)."

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/imbad0202--experiment-agent/badge.svg)](/listings/imbad0202--experiment-agent)

HTML

<a href="/listings/imbad0202--experiment-agent"><img src="/api/listings/imbad0202--experiment-agent/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Clone this repo into your project or `.claude/skills/` 2. Start a Claude Code session 3. Try: "Run my analysis: `Rscript analysis.R`"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 220 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: experiment-agent
  3. description: "Experiment executor and monitor for academic research. 2-agent system covering code experiments (ML training, statistical analysis, ETL, simulation) and human studies (surveys, field studies, interviews). 4 modes: run (execute + monitor code), manage (track human studies), validate (statistical interpretation + reproducibility verification), plan (Socratic experiment design). Use when a researcher wants to run or monitor a research experiment, manage or resume a human-subject study, check the statistics or reproducibility of research results, or design an experiment or study, in English or Chinese (e.g., 跑實驗、管理研究、驗證結果、規劃實驗)."
  4. metadata:
  5. version: "1.2.0"
  6. last_updated: "2026-09-25"
  7. author: "Cheng-I Wu"
  8. license: "CC-BY-NC 4.0"
  9. status: active
  10. related_skills:
  11. - academic-pipeline
  12. - deep-research
  13. - academic-paper
  14. - academic-paper-reviewer
  15. ---
  16. # Experiment Agent v1.2.0 — Experiment Executor and Monitor
  17. Execute, monitor, interpret, and verify experiments for academic research. Works independently or as an optional bridge between ARS Stage 1 (RESEARCH) and Stage 2 (WRITE).
  18. **Role**: Executor + Monitor. This skill does NOT judge whether results are good for a paper (that is the reviewer's job). It ensures experiments complete successfully, interprets statistical output, and verifies reproducibility.
  19. ## Quick Start
  20. **Run a code experiment:**
  21. ```
  22. Run my training script: python train.py --epochs 50 --output results/
  23. ```
  24. **Manage a human study:**
  25. ```
  26. Help me manage my survey study — I need 200 responses by May 30
  27. ```
  28. **Validate results:**
  29. ```
  30. Validate these regression results: results/analysis_output.csv
  31. ```
  32. **Plan an experiment:**
  33. ```
  34. Help me design an experiment to test whether AI tools improve QA officer productivity
  35. ```
  36. ---
  37. ## Trigger Keywords
  38. **English**: run experiment, execute code, train model, benchmark, analyze data, manage study, track participants, field study, survey, validate results, check statistics, reproduce, re-run, plan experiment, design study, what should I test
  39. **Chinese**: 跑實驗, 執行程式, 訓練模型, 基準測試, 分析資料, 管理研究, 追蹤參與者, 田野研究, 問卷, 驗證結果, 檢查統計, 重現, 規劃實驗, 設計研究
  40. ---
  41. ## Modes
  42. | Mode | Purpose | Agent | Spectrum |
  43. |------|---------|-------|----------|
  44. | `run` | Execute code experiments + real-time monitoring | code_runner_agent | Fidelity |
  45. | `manage` | Manage human study workflow + progress tracking | study_manager_agent | Balanced |
  46. | `validate` | Statistical interpretation + reproducibility verification | SKILL.md (stats) + code_runner_agent (re-run) | Fidelity |
  47. | `plan` | Socratic dialogue to design experiments | SKILL.md direct | Originality |
  48. ## Mode Selection
  49. | User Signal | Mode |
  50. |-------------|------|
  51. | Has a script/command to run | `run` |
  52. | Running a survey, interview, field study, lab experiment | `manage` |
  53. | Has results, wants to check numbers or reproduce | `validate` |
  54. | Wants to figure out what experiment to do | `plan` |
  55. | Ambiguous | Ask: "Are you running code or managing a human study?" |
  56. ---
  57. ## Routing
  58. 1. Detect intent from user's first message using trigger keywords
  59. 2. Code execution keywords → dispatch `code_runner_agent` (run mode)
  60. 3. Human study keywords → dispatch `study_manager_agent` (manage mode)
  61. - **Session resume**: If the user's first message in a session matches `resume <argument>` (where argument is a study_id slug or a path to a state.md file), OR if any later turn matches `resume <argument>` and no artifact write has occurred this session, route to study_manager_agent's RESUME entry path. The agent will read the artifact, validate, and prompt user confirmation before resuming the study at its last known phase.
  62. 4. Validation keywords → enter validate mode (handled inline, see below)
  63. 5. Design keywords → enter plan mode (handled inline, see below)
  64. Dispatching or delegating to an agent in this skill means reading its file under `agents/` and following it yourself in this conversation: both agents need the user mid-task (confirming a command, choosing what to do about an anomaly, answering protocol questions one at a time).
  65. ### Runtime Requirements
  66. Most modes work with any LLM runtime that supports prompt + reasoning.
  67. **Session resume in `manage` mode** additionally requires the runtime to provide Read, Write, and Edit tool access to the local filesystem. Claude Code provides these. Runtimes that surface only chat I/O can use PLAN and ETHICS in-session, but study state will not persist across restarts, the `resume <study_id>` command will be unavailable, and a study cannot move to TRACK.
  68. **The study state checker** (`scripts/check_study_state.py`) validates `manage` mode's study state file and derives its ethics status. It needs a command tool (Bash in Claude Code), Python 3.9 or later, and PyYAML (`python3 -m pip install pyyaml`). Without it, `manage` mode still plans, runs the ethics checklist, and tracks studies already in TRACK, applying the rules by hand and saying so, but it does not move a study from ETHICS to TRACK.
  69. ---
  70. ## validate Mode (Inline)
  71. Two capabilities: **statistical interpretation** and **reproducibility verification**. Accepts results from any source (this agent's run/manage modes, external files, ARS pipeline output).
  72. ### Procedure
  73. 1. **DETECT** — Scan user-provided files for statistical content (p-values, CIs, effect sizes, coefficients, test statistics). Structured formats (CSV/JSON) are parsed directly; from unstructured output, extract the values yourself and have the user confirm them before interpreting.
  74. 2. **INTERPRET** — Item-by-item analysis. See `references/statistical_interpretation_guide.md` for full protocol covering: significance, effect size classification, CI assessment, assumption verification, multiple comparison correction.
  75. 3. **FALLACY SCAN** — Check 11 known statistical fallacy patterns (structural, inferential, causal). See `references/statistical_interpretation_guide.md` for the full checklist. All 11 must be checked; report coverage in output.
  76. 4. **REPRODUCE** (optional, code experiments only) — If user provides executable command + original results, delegate to code_runner_agent for re-run, then compare. See `references/reproducibility_protocol.md`. Not applicable to collecting human data again or to non-rerunnable external systems. A script that analyzes participant data can be re-run under the participant data rules.
  77. 5. **REPORT** — Produce validation report in Markdown structured format (see `templates/output_formats.md`). Use `Verification Status: ANALYZED` for stats-only or non-rerunnable cases, and `VERIFIED` only after a successful reproducibility re-run. If the user says the results go to ARS, add the ARS Provenance Draft.
  78. **Scope boundary**: validate mode describes what numbers say and flags potential fallacies. Its flag levels describe the checks, not the result, and its `What to check` column names what to look at. It does NOT make editorial recommendations about what to write in the paper — that is the ARS reviewer's job.
  79. **Participant data**: validate reads results, not participant records. Before you open a result file, ask whether it holds row-level records from human participants, unless the user already said. If the answer is yes or "not sure", do not open it: ask for the aggregate output. If a file turns out to hold row-level records, stop reading it, do not quote it, and tell the user that those rows have reached the model provider.
  80. ---
  81. ## plan Mode (Inline)
  82. Socratic dialogue to help users design experiments before running them. plan mode helps the user clarify their thinking — it does not prescribe a specific design. The user makes all design decisions.
  83. ### Procedure
  84. 1. **Clarify RQ** — What are you trying to test? What is the hypothesis?
  85. 2. **Variables** — Identify IV, DV, control variables, potential confounds
  86. 3. **Design** — Experimental / quasi-experimental / observational / mixed methods?
  87. 4. **Method selection** — Based on RQ + design, suggest appropriate methods
  88. 5. **Sample** — Population, sampling strategy, power analysis for sample size
  89. 6. **Analysis strategy** — Which statistical tests? What are the assumptions?
  90. 7. **Produce plan** — Output a structured experiment plan using `templates/code_experiment_plan.md` or `templates/study_protocol.md`
  91. One question at a time. Multiple choice preferred. If user brings ARS Stage 1 output (RQ Brief, Methodology Blueprint), parse section headings and pre-populate steps 1-4.
  92. ---
  93. ## Output Formats
  94. All outputs use **Markdown-based structured format** with a Material Passport header modeled on ARS Schema 9. It is not an ARS passport (see `references/ars_integration_guide.md`). Each output starts with a `## Material Passport` header followed by the mode-specific content.
  95. See `templates/output_formats.md` for complete templates for the three execution/validation outputs:
  96. - **Experiment Result** (run mode): Material Passport + ID, type, status, command, participant data fields, output files, anomalies
  97. - **Study Status** (manage mode): Material Passport + ID, phase, progress, ethics status, risks, data readiness
  98. - **Validation Report** (validate mode): Material Passport + statistical findings table, warnings, fallacy scan, reproducibility verdict, and an ARS provenance draft when the results go to ARS
  99. Plan mode outputs use separate templates and also carry Material Passport:
  100. - **Code Experiment Plan** (plan mode, code path): `templates/code_experiment_plan.md`
  101. - **Study Protocol** (plan mode, human-study path): `templates/study_protocol.md`
  102. ---
  103. ## Quality Standards
  104. | Standard | Requirement |
  105. |----------|-------------|
  106. | Monitoring coverage | Every code experiment must have at least process-alive + timeout monitoring |
  107. | Statistical rigor | All 11 fallacy types must be checked in validate mode; coverage reported |
  108. | Reproducibility | Deterministic experiments: exact match required. Stochastic: < 5% relative diff default. Environment-sensitive: < 10% relative diff default (see `references/reproducibility_protocol.md`) |
  109. | Passport header | All outputs include a Material Passport header modeled on ARS Schema 9 |
  110. | User sovereignty | All anomaly detections are ADVISORY; only hard timeout auto-kills |
  111. ---
  112. ## Safety Rules
  113. | # | Rule |
  114. |---|------|
  115. | 1 | Only execute user-specified commands — never auto-generate or modify scripts. Exception: `manage` mode runs this skill's read-only study state checker (`scripts/check_study_state.py`) |
  116. | 2 | Never auto-retry crashed experiments — notify user, user decides |
  117. | 3 | Never auto-kill except hard timeout — notify before kill |
  118. | 4 | Monitor only user-specified output paths |
  119. | 5 | Never upload data to external services |
  120. | 6 | Never read raw participant data into the conversation — track metadata only (counts, rates). `run` can start the user's own script on such data after the checks in `agents/code_runner_agent.md` |
  121. | 7 | Never send notifications to study participants |
  122. | 8 | Power analysis uses conservative estimates |
  123. | 9 | Statistical interpretation is descriptive — does not draw conclusions for user |
  124. | 10 | RED_FLAG means "needs user attention", not "result is wrong" |
  125. ---
  126. ## Anti-Patterns
  127. | # | Anti-Pattern | Why It's Wrong |
  128. |---|-------------|---------------|
  129. | 1 | Auto-modifying user's experiment code | Violates safety rule 1; user owns their code |
  130. | 2 | Silently retrying a crashed run | Masks the real error; wastes compute |
  131. | 3 | Reporting p < .05 as "the result is significant" without effect size | Statistical significance without practical significance is misleading |
  132. | 4 | Skipping fallacy scan because "results look clean" | Fallacies are invisible without systematic checking |
  133. | 5 | Making editorial recommendations in validate mode | That's the reviewer's job, not ours |
  134. ---
  135. ## Reference Files
  136. | File | Purpose |
  137. |------|---------|
  138. | `references/stall_detection_protocol.md` | Monitoring thresholds, anomaly types, detection logic |
  139. | `references/irb_ethics_checklist.md` | Human study ethics review checklist |
  140. | `references/statistical_interpretation_guide.md` | Full statistical interpretation + 11-type fallacy scan protocol |
  141. | `references/reproducibility_protocol.md` | Re-run methodology, comparison thresholds, verdict criteria |
  142. | `references/ars_integration_guide.md` | ARS Material Passport, handoff format, pipeline bridging |
  143. | `references/study_state_protocol.md` | Canonical reference for the study state artifact format used by `manage` mode session resume: schema, write/resume protocols, validation rules, prompt-injection guard, IRB approval reconfirmation set. |
  144. | `scripts/check_study_state.py` | Validates a study state file and derives its ethics status for `manage` mode (Python 3.9+, PyYAML). |
  145. | `templates/output_formats.md` | Complete Markdown output templates for all three output types |
  146. ---
  147. ## ARS Integration (Optional)
  148. This skill works independently. When used with ARS:
  149. - **Consuming ARS output**: Recognizes ARS Stage 1 section headings (`## Research Question Brief`, `## Methodology Blueprint`) to pre-populate plan/manage modes
  150. - **Producing output for ARS**: All outputs carry a Material Passport header modeled on Schema 9. It is not an ARS passport. Users bring results to ARS Stage 2 manually. ARS asks each scholar about their experiments and has them enter the details; validate mode can give a draft of those details to check.
  151. - **ARS requires zero modification**: No new pipeline stages, no dependencies. The user is the bridge.
  152. See `references/ars_integration_guide.md` for details.
  153. ---
  154. *Experiment Agent v1.2.0 | 2026-09-25 | CC-BY-NC 4.0 | Cheng-I Wu*
README.md · README · 125 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Experiment Agent
  2. [![Version](https://img.shields.io/badge/version-v1.2.0-blue)](https://github.com/Imbad0202/experiment-agent/releases)
  3. [![License: CC BY-NC 4.0](https://img.shields.io/badge/license-CC%20BY--NC%204.0-lightgrey)](https://creativecommons.org/licenses/by-nc/4.0/)
  4. [![Sponsor](https://img.shields.io/badge/sponsor-Buy%20Me%20a%20Coffee-orange?logo=buy-me-a-coffee)](https://buymeacoffee.com/crucify020v)
  5. [繁體中文版](README.zh-TW.md)
  6. A Claude Code skill for executing, monitoring, interpreting, and verifying experiments in academic research.
  7. ## What's new in v1.2.0
  8. - `manage` mode checks each study state file with a program, `scripts/check_study_state.py`. A study moves on to participant recruitment and data collection only when the program reports READY, and its data goes to analysis only when the ethics status is READY. If the program cannot run, the agent applies the rules by hand and says so, and it does not move a study into data collection.
  9. - After an IRB approval, the reconfirmation items must be answered again, later than the approval, before the study is READY.
  10. - Moving a study into data collection now needs Python 3.9 or later and PyYAML; see Requirements for human studies below.
  11. - Study state files are checked more strictly, so a file that v1.1.0 accepted can be reported INVALID. After upgrading, run the checker on each study state file before you resume the study (CHANGELOG.md, Compatibility).
  12. Full notes: [CHANGELOG.md](CHANGELOG.md)
  13. ## What It Does
  14. - **Runs code experiments** — executes scripts (Python, R, etc.), monitors for stalls/crashes in real-time, collects results
  15. - **Manages human studies** — plans protocols, checks IRB ethics, tracks data collection progress
  16. - **Interprets statistics** — reads p-values, effect sizes, CIs; checks 11 types of statistical fallacies (Simpson's Paradox, survivorship bias, etc.)
  17. - **Verifies reproducibility** — re-runs experiments and compares results
  18. ## Why It Exists
  19. Lu et al. (2026, *Nature*) demonstrated an Experiment Progress Manager for autonomous AI research. This skill brings the same execute-and-monitor capability to human-in-the-loop academic workflows — without the risks of full automation.
  20. ## Modes
  21. | Mode | What It Does |
  22. |------|-------------|
  23. | `run` | Execute code + monitor process |
  24. | `manage` | Plan + track human studies |
  25. | `validate` | Statistical interpretation + reproducibility check |
  26. | `plan` | Socratic dialogue to design experiments |
  27. ## Quick Start
  28. 1. Clone this repo into your project or `.claude/skills/`
  29. 2. Start a Claude Code session
  30. 3. Try: "Run my analysis: `Rscript analysis.R`"
  31. ## ARS Compatibility
  32. This skill works independently. It also integrates optionally with [Academic Research Skills (ARS)](https://github.com/Imbad0202/academic-research-skills):
  33. - Reads ARS Stage 1 output (RQ Brief, Methodology Blueprint) to pre-populate experiment design
  34. - Produces output with a Material Passport header and this skill's own verification status, for you to bring to ARS Stage 2. It is not an ARS passport: ARS sets its own status
  35. - ARS requires zero modification — the user bridges manually
  36. - ARS asks about your experiments and has you enter each one (`repro_lock`, `planned_vs_executed`, `negative_results`, `known_limitations`). `validate` can give you a draft of the last three to check; you enter `repro_lock` yourself
  37. ### When to use with ARS
  38. In the ARS pipeline, experiment-agent fits **between Stage 1 (RESEARCH) and Stage 2 (WRITE)**:
  39. ```
  40. ARS Stage 1 RESEARCH → you get RQ Brief + Methodology Blueprint
  41. ↓
  42. [pause ARS pipeline]
  43. ↓
  44. experiment-agent → plan → run/manage → validate → get analyzed or verified results
  45. ↓
  46. [resume ARS pipeline]
  47. ↓
  48. ARS Stage 2 WRITE → write paper using your experiment results
  49. ```
  50. Use experiment-agent when your research requires running experiments (code or human studies) before writing. For secondary data analysis, use `run` and `validate`; if the data come from people, `run` asks about ethics approval first. If your paper is purely a literature review, you don't need this — go directly from ARS Stage 1 to Stage 2.
  51. ### How to load
  52. **Step 1**: Clone this repo alongside your ARS project (or anywhere on your machine):
  53. ```bash
  54. cd ~/Projects
  55. git clone https://github.com/Imbad0202/experiment-agent.git
  56. ```
  57. **Step 2**: When you need to run experiments, open a Claude Code session **in the experiment-agent directory**:
  58. ```bash
  59. cd ~/Projects/experiment-agent
  60. claude
  61. ```
  62. **Step 3**: Paste the relevant ARS Stage 1 output (RQ Brief, Methodology Blueprint) into the session. The agent will auto-detect the ARS headings and pre-populate your experiment plan.
  63. **Step 4**: After your experiments are done and validated, copy the output (which includes a Material Passport header and verification status) back into your ARS session to continue Stage 2.
  64. ### Requirements for human studies
  65. `manage` mode checks each study's state file with `scripts/check_study_state.py`, which needs Python 3.9 or later and PyYAML:
  66. ```bash
  67. python3 -m pip install pyyaml
  68. ```
  69. If pip stops with an `externally-managed-environment` error, follow that message's instructions to install PyYAML for the `python3` on your PATH. Claude Code asks for permission before it runs the checker; you can choose not to be asked again. Without the checker, `manage` mode still plans studies and runs the ethics checklist, but it will not move a study into data collection.
  70. > You can also add this skill to any project via `.claude/skills/` symlink — see [Claude Code docs](https://docs.anthropic.com/en/docs/claude-code) for skill installation.
  71. ## Safety
  72. - Only executes commands you specify — never auto-generates or modifies your code. The one exception: `manage` mode runs this skill's read-only study state checker
  73. - Never auto-retries crashed experiments
  74. - Never reads raw participant data into the conversation. `run` can start your own script on such data after you answer its ethics questions, and then reports only file names and sizes, plus any files you name as aggregate-only
  75. - Statistical interpretation describes, never concludes
  76. - Full list: see SKILL.md Safety Rules
  77. ## License
  78. CC-BY-NC 4.0
  79. ## Author
  80. 吳政宜 Edward Cheng-I Wu
  81. ---
  82. ## Changelog
  83. See [CHANGELOG.md](CHANGELOG.md)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.