Skip to content
Back to search

falsify-the-problem

An open-source Agent Skill for problem framing and assumption testing across coding, debugging, architecture, scientific research, and product decisions.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ordinary-s--falsify-the-problem/badge.svg)](/listings/ordinary-s--falsify-the-problem)

HTML

<a href="/listings/ordinary-s--falsify-the-problem"><img src="/api/listings/ordinary-s--falsify-the-problem/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

The complete runtime artifact is [SKILL.md](SKILL.md). Evaluation files are development materials, not dependencies. The skill itself needs no package, API key, or service; your host agent has its own setup requirements.

Codex: choose an installation scope

Codex is one way to use this cross-domain skill. According to the [official skills documentation](https://learn.chatgpt.com/docs/build-skills), local skills can live in either of these locations:

| Scope | Folder containing `SKILL.md` | Available to | | --- | --- | --- | | Project | `<project>/.agents/skills/falsify-the-problem/` | Codex sessions in that project | | User | `~/.agents/skills/falsify-the-problem/` | Codex sessions across your projects |

Choose one scope. Run the commands below in a **terminal**, not in the Codex prompt. Git must be installed. Project commands start in the root of the project where you want to use the skill; user commands can start in any directory. These commands clone into a new folder and will fail if the destination is already populated. Keep an existing installation instead of overwriting it.

**Windows PowerShell, project scope:**

New-Item -ItemType Directory -Force -Path .agents/skills | Out-Null
git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem

**Windows PowerShell, user scope:**

$skillRoot = Join-Path $env:USERPROFILE '.agents/skills'
New-Item -ItemType Directory -Force -Path $skillRoot | Out-Null
git clone https://github.com/ordinary-s/falsify-the-problem.git (Join-Path $skillRoot 'falsify-the-problem')

**macOS / Linux, Bash or Zsh, project scope:**

mkdir -p .agents/skills
git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem

**macOS / Linux, Bash or Zsh, user scope:**

mkdir -p "$HOME/.agents/skills"
git clone https://github.com/ordinary-s/falsify-the-problem.git "$HOME/.agents/skills/falsify-the-problem"

Alternatively, download or clone this repository elsewhere and copy only `SKILL.md` into the chosen `falsify-the-problem` folder. Codex detects skill changes automatically; restart it if the skill does not appear. In Codex CLI or the IDE extension, use `/skills` or type `$` to select the skill.

First invocation in Codex CLI or the IDE extension

Paste this into the **Codex prompt**, not PowerShell or a shell:

$falsify-the-problem
Our API has become slower and we are considering a cache.
Inspect the available evidence, distinguish observations from assumptions,
and test the problem framing before proposing a solution.

The local setup commands do not establish skill activation or model behavior. macOS and Linux installation and cross-host behavior have not been verified in this documentation change. See [Usage](#usage) for ordinary-language examples.

Other hosts

1. Download or clone this repository into a local `falsify-the-problem` folder. 2. For a host with skill-folder support, place the folder containing `SKILL.md` in that host's documented skill location and use its normal reload/discovery procedure. You can copy just `SKILL.md` into a folder of the same name. 3. For a host that accepts reusable Markdown instructions, attach or load the complete `SKILL.md` and explicitly request its use before solving the task.

Paths, discovery, invocation syntax, and tool permissions are host-specific. The skill is designed to be portable across agents that support reusable Markdown instructions or skill-like workflows; identical behavior across hosts is not guaranteed.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 292 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: falsify-the-problem
  3. description: Challenge the current problem formulation before solving it. Use when explicitly invoked, or when meaningful framing uncertainty and non-trivial intervention cost or risk coexist; do not auto-activate for verified diagnoses or mechanical edits.
  4. ---
  5. # falsify-the-problem
  6. > Falsify the problem before solving it.
  7. Answer **Are we solving the right problem?** Challenge the current formulation,
  8. seek disconfirming evidence, and hand the result back to the host agent.
  9. Do not design or implement the solution inside this skill.
  10. Use the same protocol across domains; examples are not domain-specific workflows.
  11. ## 1. Decide whether to enter
  12. - Explicit invocation of `falsify-the-problem`: always enter. Strong existing
  13. evidence can justify a brief `KEEP` without alternatives or a new test.
  14. - Automatic activation: require BOTH meaningful formulation uncertainty AND
  15. non-trivial intervention cost or risk. Neither alone is enough.
  16. - Useful signals include solution-shaped requests, a hypothesis treated as fact,
  17. repeated failed attempts, and costly or hard-to-reverse proposed changes.
  18. Signals are reasons to check the gate, not unconditional triggers.
  19. - Usually do not auto-activate for a verified diagnosis, obvious syntax or
  20. compilation error, typo, translation, formatting, mechanical transformation,
  21. deterministic local edit, or very cheap reversible intervention.
  22. - Mere logical possibility of another explanation is insufficient.
  23. `BYPASS` is an activation decision, never an evidence verdict. When bypassing,
  24. leave the original task with the host; do not manufacture an adversarial report.
  25. If explicitly invoked on a supported diagnosis, use `KEEP`, not `BYPASS`.
  26. ## 2. Separate the claims and preserve their sources
  27. Distinguish these five objects, even when the user combines them in one sentence:
  28. | Object | Meaning | Example |
  29. | --- | --- | --- |
  30. | Observation | Something observed, measured, or reported | User reports P99 rose from 300 ms to 2.8 s |
  31. | Interpretation | Meaning assigned to an observation | The database is slow |
  32. | Causal Hypothesis | A specific mechanism proposed to explain it | Lock contention delays requests |
  33. | Problem Formulation | What kind of problem is being solved | Database service time dominates the slow tail |
  34. | Solution | A proposed intervention | Add Redis |
  35. Observation != Interpretation != Causal Hypothesis != Problem Formulation != Solution.
  36. For each consequential observation, retain the source and relevant scope:
  37. - `tool-observed`: name the file, command, trace, or other inspected artifact.
  38. - `user-reported`: retain attribution, including measurements supplied by a user.
  39. - `document-reported`: identify the document and its claim.
  40. - `experimentally observed`: identify the experiment and conditions.
  41. - `measurement-derived`: identify inputs, transformation, and sampling limits.
  42. These labels can combine. Reading a report with a tool verifies what the report
  43. says, not that its underlying claim is true. A local simulated fixture is evidence
  44. about that fixture, not independent verification of a real incident.
  45. Never upgrade user reports into independently verified facts.
  46. Record uncertainty about coverage, freshness, comparability, or missing data
  47. where it could change the conclusion. Do not fabricate observations or execution.
  48. State the **Current Problem Formulation** explicitly. If inferred from a proposed
  49. solution, mark it as inferred; do not attribute it to the user as a stated belief.
  50. Keep this original formulation as the reference for the overall verdict through
  51. this pass. A replacement is `REFORMULATE`, not a silently relabeled `KEEP`.
  52. Preserve claim strength: a possible contributor, a primary explanation, and an
  53. exclusive explanation are different claims. Do not infer exclusivity from a
  54. proposed solution or refute a stronger claim than the request requires.
  55. ## 3. Build materially different live formulations
  56. Ask what would change the kind of problem being investigated. Competing causes
  57. within one formulation are not automatically competing problem formulations.
  58. Two formulations should materially differ in at least one consequential dimension:
  59. failing variable, mechanism class, abstraction level, system boundary, expected
  60. evidence, investigation path, or intervention target.
  61. If two formulations predict nearly the same evidence and lead to nearly the same
  62. investigation path, they are probably superficial variants. Merge them.
  63. For example, weak embeddings, small embeddings, and old embeddings usually belong
  64. to one representation-quality formulation. Retrieval construction, ranking,
  65. generation grounding, evaluation validity, and query distribution can change the
  66. framing when they predict meaningfully different evidence.
  67. Keep only plausible, decision-relevant alternatives. Do not mechanically fill a
  68. quota. Let evidence and the pending decision determine the live set; more
  69. candidates are not evidence of better reasoning. Presentation limits in section 9
  70. do not justify dropping a materially different, decision-relevant candidate.
  71. Candidates need not be mutually exclusive. If several mechanisms coexist, test
  72. which framing explains the material failure at the relevant scope; do not force a
  73. winner from evidence that supports a mixed or narrower formulation.
  74. ## 4. Find load-bearing assumptions and distinct predictions
  75. Prioritize assumptions whose failure would materially weaken or collapse a live
  76. formulation. Avoid exhaustive lists or generic questions about everything.
  77. Include measurement validity only when it bears on the decision.
  78. For each important live formulation, connect:
  79. `Formulation -> load-bearing assumption -> observable prediction -> contrary evidence`
  80. A prediction must distinguish live formulations, not restate their names.
  81. Example: a service-time framing predicts elevated service spans in slow requests;
  82. a capacity framing predicts rising pre-service wait with concurrency; a measurement
  83. framing predicts disagreement between raw samples and reported aggregation.
  84. State the population, comparison, and meaningful difference when available.
  85. Do not invent numerical thresholds without a basis; explain qualitative criteria.
  86. Before choosing the test, make one lightweight **Shared-Frame / Coverage Check**:
  87. do the live formulations all rely on one consequential, untested premise about
  88. the relevant boundary, measurement path, causal layer, population, time scope,
  89. or optimization target? Select what matters here; do not audit a category checklist.
  90. If that premise failing would undermine the whole set, and a plausible alternative
  91. would change the decision and predict different evidence, add at most **ONE
  92. Outside-Frame Challenger**. Name the shared premise and its contrasting prediction
  93. briefly when decision-relevant. Generic "measurement could be wrong" is insufficient;
  94. ground the challenger in this task. If none is warranted, proceed without one.
  95. Do not chain coverage audits or expand the set merely to satisfy a quota.
  96. ## 5. Select ONE Primary Discriminating Test for this round
  97. When beginning an investigation round under unresolved uncertainty, choose
  98. **exactly ONE Primary Discriminating Test**. This is one per round, not one forever.
  99. Choose the cheapest currently available test with substantial discriminatory
  100. information relative to cost, risk, and execution time.
  101. Ask: would plausible different outcomes materially change the relative support
  102. of the live formulations? If not, choose a better test.
  103. An outcome predicted by several live formulations may localize the failure without
  104. distinguishing them. Prefer a matched comparison that separates the relevant
  105. factor; otherwise retain those formulations rather than assigning the outcome
  106. to one by default.
  107. Specify:
  108. - **Test:** one bounded comparison or observation, its source, and its scope.
  109. - **Why this first:** which live formulations it separates and why its value
  110. exceeds the cost of the next plausible option.
  111. - **Possible outcomes:** what would strengthen, weaken, or kill which candidates.
  112. Include inconclusive or mixed outcomes where plausible.
  113. - **Remaining uncertainty:** what this test cannot establish.
  114. One test can require several retrieval or computation steps if they serve one
  115. predeclared comparison and outcome map. It cannot hide independent investigations
  116. inside a bundle such as "check logs, metrics, configuration, and tests."
  117. Include an unaccounted or outside-boundary outcome when the observation method
  118. could omit relevant stages; an artifact's label does not define its boundaries.
  119. Handle such a result through the evidence update below, not a forced winner.
  120. Do not call it a "Decisive Test" or imply that every result must settle the issue.
  121. ## 6. Gather evidence instead of deflecting it
  122. STOP solutioning != STOP investigating.
  123. Inspect evidence directly when it is available, permitted, low-risk, and reasonably
  124. cheap. Use the host's tools as appropriate: local artifacts, code, history, logs,
  125. traces, metrics, configurations, documentation, existing tests, safe diagnostics,
  126. low-risk experiments, or benchmarks. No specific tool or runtime is required.
  127. Choose the investigation for its discrimination, not because a tool is available.
  128. Read-only exploration needed to locate the selected evidence is allowed. Bound it
  129. to the comparison; do not turn discovery into a generic diagnostic checklist.
  130. Record what was actually inspected or run and the result, including failures.
  131. If the selected source is inaccessible, prefer an available lower-cost proxy only
  132. if it can discriminate; record the limitation. If no useful proxy exists, state
  133. the one pending Primary Discriminating Test and request only the minimum missing
  134. discriminating evidence or access. Do not ask users to do work the host can safely
  135. perform. Do not claim the pending test ran or loop through identical requests.
  136. Investigation does not authorize implementation. Before formulation validation,
  137. do not implement the requested solution, migrate architecture, introduce
  138. infrastructure, make expensive production changes, take irreversible action, or
  139. optimize around an unvalidated premise. Calling an intervention a "test" does not
  140. make it low-risk. Respect the host's existing permissions and user task scope.
  141. ## 7. Update from evidence, then decide whether to continue
  142. Evidence beats rhetoric and the agent's previous conclusion. Accept contradictory
  143. evidence; do not reinterpret it merely to preserve an earlier position.
  144. After the selected test:
  145. 1. Attribute the result and compare it with the predicted outcomes.
  146. 2. If a consequential result fits no live formulation, mark it **OUTSIDE CURRENT
  147. FRAME**: the set may be incomplete. This is an evidence diagnostic, not a sixth
  148. verdict. Preserve the unexplained residual; do not award it to the closest
  149. candidate. Repeat the lightweight coverage check once for this new evidence,
  150. adding at most one grounded challenger if warranted. Otherwise leave it unresolved.
  151. 3. Update affected candidates using the labels below with a short evidence reason,
  152. and re-check load-bearing assumptions invalidated by the test.
  153. 4. Merge redundant candidates; retain uncertainty and scope limits.
  154. 5. Apply the STOP conditions. If a further discriminating observation is worth its
  155. cost, start the next round with the updated live set and one new Primary Test.
  156. | Candidate update | Meaning |
  157. | --- | --- |
  158. | STRENGTHEN | New evidence raises support, without proving the candidate |
  159. | WEAKEN | New evidence reduces support; the candidate remains live |
  160. | KILL | Reliable evidence contradicts a necessary assumption at the stated scope |
  161. | UNRESOLVED | Evidence does not discriminate enough or is unavailable |
  162. | MERGE | Combine with a named candidate because the distinction is redundant |
  163. Missing evidence is not contrary evidence. An absent signal kills a candidate only
  164. when coverage and detection were sufficient for that signal to be expected.
  165. If new evidence undermines a prior kill, explicitly reopen the candidate and give
  166. the reason; labels are revisable conclusions, not permanent states.
  167. Candidate updates and the overall verdict have different meanings and scopes.
  168. ## 8. STOP, give an overall verdict, and hand back control
  169. End the adversarial pass when any of these holds:
  170. - The current formulation has sufficient evidence for the pending decision.
  171. - No meaningful materially different alternative remains, or remaining
  172. alternatives are too weak to justify delaying the decision.
  173. - Further investigation has poor information gain relative to cost, risk, or time.
  174. - Another formulation is sufficiently supported to replace the original.
  175. - The original formulation is dead and no useful cheap investigation remains.
  176. - Essential discriminating evidence is unavailable and no useful proxy exists;
  177. hand off the minimal evidence request rather than stall indefinitely.
  178. Sufficiency is proportional to the pending intervention and evidence quality.
  179. Running out of tests, access, time, or alternatives is not itself evidence for KEEP.
  180. Do not become contrarian for its own sake. KEEP is a normal successful outcome.
  181. Use exactly one overall verdict about the original Current Problem Formulation:
  182. | Overall verdict | Meaning | Solutioning / handoff |
  183. | --- | --- | --- |
  184. | KEEP | Current framing remains best-supported and sufficiently validated for the pending decision | ALLOWED for that framing; release to the downstream solver |
  185. | WEAKEN | Current framing remains live but has materially lost support; no replacement is sufficiently established | NOT YET; continue useful investigation or hand off the specific uncertainty |
  186. | KILL | Current framing is no longer viable, with no sufficiently supported replacement | BLOCKED for the killed premise; investigate another framing if worthwhile |
  187. | REFORMULATE | A materially different framing is sufficiently better supported to replace the original | NOT YET; return the Reformulated Problem to the downstream workflow; the old solution has no release |
  188. | INSUFFICIENT EVIDENCE | Available evidence cannot discriminate enough to support a stronger verdict | NOT YET; investigate if worthwhile, otherwise state the minimum missing evidence |
  189. For REFORMULATE, always output **Reformulated Problem:** with the replacement and
  190. its supporting evidence. KILL does not assert a replacement; REFORMULATE does.
  191. If several descriptions fit, choose the most informative supported verdict:
  192. an established replacement takes REFORMULATE over KILL or WEAKEN; absent one,
  193. contradicted necessary assumptions justify KILL, reduced support justifies WEAKEN,
  194. and non-discrimination justifies INSUFFICIENT EVIDENCE.
  195. Skill STOP != Host Agent STOP. KEEP ends this skill's responsibility. When the
  196. original request already authorizes implementation, the host can continue in the
  197. same turn without asking again. Validation is not new permission for extra work.
  198. KEEP validates the problem framing, not the effectiveness of the proposed solution.
  199. Mark any subsequent diagnosis or implementation as host/downstream continuation
  200. so a completed framing pass does not silently turn into a root-cause workflow.
  201. After REFORMULATE, the host may run a brief validation pass with the replacement
  202. as the new Current Problem Formulation, reusing the evidence already gathered.
  203. If it warrants KEEP, release it without redundant testing; then the host can solve
  204. within the user's authorized scope. Do not implement the original solution merely
  205. because an alternative explanation was found. The skill itself never designs it.
  206. ## 9. Default to compact output; disclose detail as needed
  207. Complete the relevant checks, but show only decision-relevant evidence and
  208. rationale, not an exhaustive reasoning transcript or hidden chain of thought.
  209. Choose the shortest path that preserves the decision, discrimination, and scope:
  210. | Path | When | Visible output |
  211. | --- | --- | --- |
  212. | Fast | BYPASS, strong KEEP, verified deterministic diagnosis, or simple low uncertainty | Usually 2-5 sentences: source-backed observation, framing, verdict/release if entered, then host continuation. No invented alternative or new test. BYPASS needs no adversarial report. |
  213. | Normal (default) | Ordinary unresolved coding, debugging, architecture, research, or product framing | Usually 120-220 English words: observation/provenance, current framing, live alternatives, one Primary Test with contrasting outcomes, result or pending evidence, verdict and handoff. Fold each key assumption into its prediction. |
  214. | Deep | Stakes or complexity require more detail, including when 5+ live formulations materially affect the decision | Usually 250-450 English words: expand assumptions, predictions, candidate updates, and scope limits only where needed. Domain alone does not require this path. |
  215. These are soft presentation guides, not word quotas or correctness gates; use
  216. equivalent brevity in other languages. Normal output usually shows 2-4 live
  217. formulations including the current one, at most 3-4 by default. Do not conceal a
  218. consequential challenger to fit that range: merge genuine redundancies, or expand
  219. the output when needed. A supported single formulation needs no padding.
  220. Keep provenance compact (for example, "report.txt, supplied summary"), never omit
  221. it or upgrade it. State whether the one primary comparison ran or is pending;
  222. preliminary localization is not a discriminating test when all candidates predict
  223. it. Give one clear verdict and release status, plus **Reformulated Problem:** when
  224. required. Headings are optional; do not mechanically reproduce all workflow steps.
  225. On later turns, use **delta output**: new evidence/source, what changed, affected
  226. candidate updates and scope, a new Primary Test only if useful, then verdict/handoff.
  227. Do not repeat unchanged observations, formulations, assumptions, or predictions.
  228. Reference prior candidates briefly so the update remains intelligible.
  229. Do not repeat the user prompt, narrate internal exploration, stage a debate,
  230. explain the methodology unless asked, or list discarded/merged candidates unless
  231. their change matters now. Stop once the framing decision is supported; mark any
  232. authorized downstream work separately and let the host continue.
README.md · README · 305 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # falsify-the-problem — An Agent Skill for Problem Framing
  2. English | [简体中文](README.zh-CN.md)
  3. [Project website](https://ordinary-s.github.io/falsify-the-problem/)
  4. > Stop solving the wrong problem faster.
  5. An open-source Agent Skill for problem framing and assumption testing across
  6. coding, debugging, architecture, scientific research, and product decisions.
  7. It asks **Are we solving the right problem?** before proposing solutions.
  8. Start with [installation](#installation) and a [first invocation](#first-invocation-in-codex-cli-or-the-ide-extension).
  9. **Illustrative scenarios, not production incidents or measured results:**
  10. | Before | After |
  11. | --- | --- |
  12. | "API P99 rose from 300 ms to 2.8 s. Add Redis." -> design a cache | Attribute the reported latency, test whether database time actually dominates the slow tail, then keep or replace that framing |
  13. | "The experiment underperformed. We need a stronger model." -> increase model capacity | Compare the same saved inputs through training and evaluation preprocessing, holding the model and scoring fixed; test whether an input-path mismatch contributes before attributing the result to model capability |
  14. | "Retention is low. Add push notifications." -> build reminders | Test whether users experienced value and forgot to return, or never experienced value in the first place |
  15. One Markdown file. No dependency, API key, server, or database. The distinctive
  16. focus is a falsifiable framing, materially different alternatives, one primary
  17. discriminating test per investigation round, evidence updates, and an explicit
  18. handoff before solution design. `KEEP` is a successful result.
  19. ## Installation
  20. The complete runtime artifact is [SKILL.md](SKILL.md). Evaluation files are
  21. development materials, not dependencies. The skill itself needs no package,
  22. API key, or service; your host agent has its own setup requirements.
  23. ### Codex: choose an installation scope
  24. Codex is one way to use this cross-domain skill. According to the
  25. [official skills documentation](https://learn.chatgpt.com/docs/build-skills),
  26. local skills can live in either of these locations:
  27. | Scope | Folder containing `SKILL.md` | Available to |
  28. | --- | --- | --- |
  29. | Project | `<project>/.agents/skills/falsify-the-problem/` | Codex sessions in that project |
  30. | User | `~/.agents/skills/falsify-the-problem/` | Codex sessions across your projects |
  31. Choose one scope. Run the commands below in a **terminal**, not in the Codex
  32. prompt. Git must be installed. Project commands start in the root of the project
  33. where you want to use the skill; user commands can start in any directory.
  34. These commands clone into a new folder and will fail if the destination is
  35. already populated. Keep an existing installation instead of overwriting it.
  36. **Windows PowerShell, project scope:**
  37. ```powershell
  38. New-Item -ItemType Directory -Force -Path .agents/skills | Out-Null
  39. git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem
  40. ```
  41. **Windows PowerShell, user scope:**
  42. ```powershell
  43. $skillRoot = Join-Path $env:USERPROFILE '.agents/skills'
  44. New-Item -ItemType Directory -Force -Path $skillRoot | Out-Null
  45. git clone https://github.com/ordinary-s/falsify-the-problem.git (Join-Path $skillRoot 'falsify-the-problem')
  46. ```
  47. **macOS / Linux, Bash or Zsh, project scope:**
  48. ```bash
  49. mkdir -p .agents/skills
  50. git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem
  51. ```
  52. **macOS / Linux, Bash or Zsh, user scope:**
  53. ```bash
  54. mkdir -p "$HOME/.agents/skills"
  55. git clone https://github.com/ordinary-s/falsify-the-problem.git "$HOME/.agents/skills/falsify-the-problem"
  56. ```
  57. Alternatively, download or clone this repository elsewhere and copy only
  58. `SKILL.md` into the chosen `falsify-the-problem` folder. Codex detects skill
  59. changes automatically; restart it if the skill does not appear. In Codex CLI
  60. or the IDE extension, use `/skills` or type `$` to select the skill.
  61. ### First invocation in Codex CLI or the IDE extension
  62. Paste this into the **Codex prompt**, not PowerShell or a shell:
  63. ```text
  64. $falsify-the-problem
  65. Our API has become slower and we are considering a cache.
  66. Inspect the available evidence, distinguish observations from assumptions,
  67. and test the problem framing before proposing a solution.
  68. ```
  69. The local setup commands do not establish skill activation or model behavior.
  70. macOS and Linux installation and cross-host behavior have not been verified in
  71. this documentation change. See [Usage](#usage) for ordinary-language examples.
  72. ### Other hosts
  73. 1. Download or clone this repository into a local `falsify-the-problem` folder.
  74. 2. For a host with skill-folder support, place the folder containing `SKILL.md` in
  75. that host's documented skill location and use its normal reload/discovery procedure.
  76. You can copy just `SKILL.md` into a folder of the same name.
  77. 3. For a host that accepts reusable Markdown instructions, attach or load the
  78. complete `SKILL.md` and explicitly request its use before solving the task.
  79. Paths, discovery, invocation syntax, and tool permissions are host-specific. The
  80. skill is designed to be portable across agents that support reusable Markdown
  81. instructions or skill-like workflows; identical behavior across hosts is not guaranteed.
  82. ## Why
  83. A requested solution often contains an untested definition of the problem.
  84. Optimizing that solution can make the wrong investigation expensive.
  85. This skill exposes the definition and seeks cheap evidence that could overturn it.
  86. It is useful across coding, debugging, architecture, research, and product work.
  87. The protocol stays the same across domains.
  88. ## How it works
  89. 1. Preserve observations and their provenance; separate interpretation, causal
  90. hypothesis, problem formulation, and proposed solution.
  91. 2. Identify the current formulation and plausible, materially different alternatives.
  92. 3. Find load-bearing assumptions and derive distinct observable predictions.
  93. 4. Select **one Primary Discriminating Test per round**, balancing information
  94. against cost, risk, and time. Inspect available, permitted, cheap evidence directly.
  95. 5. Strengthen, weaken, kill, or merge candidates from the result. Investigate again
  96. when another discriminating observation is worth its cost.
  97. 6. Stop with an overall verdict and return control to the host agent.
  98. Before choosing the Primary Discriminating Test, the skill checks whether all
  99. live formulations share the same consequential, untested frame. It may add one
  100. grounded outside-frame challenger; unexplained evidence can reopen that check.
  101. Output defaults to compact Fast / Normal / Deep paths, with later turns showing
  102. only changed evidence and decisions.
  103. | Overall verdict | Meaning | Handoff |
  104. | --- | --- | --- |
  105. | KEEP | Current framing is sufficiently validated and best-supported | Solutioning ALLOWED; host resumes authorized work |
  106. | WEAKEN | Current framing lost material support but remains live | NOT YET; investigate the remaining uncertainty |
  107. | KILL | Original framing is no longer viable; no replacement established | BLOCKED for that premise |
  108. | REFORMULATE | A materially different replacement is sufficiently supported | NOT YET; return a named Reformulated Problem for downstream validation |
  109. | INSUFFICIENT EVIDENCE | Evidence cannot discriminate enough | NOT YET; investigate or request minimum missing evidence |
  110. `BYPASS` is an activation decision, not a verdict. `KILL` does not imply a known
  111. replacement. After `REFORMULATE`, the host can validate the replacement in a brief
  112. new pass using existing evidence, obtain `KEEP`, and continue within task scope.
  113. **STOP solutioning does not mean STOP investigating. Skill STOP does not mean
  114. Host Agent STOP.** After `KEEP`, a host already authorized to implement can proceed
  115. in the same turn without asking for permission again. No new authority is granted.
  116. ## Quick example
  117. User: "API P99 increased from 300 ms to 2.8 s. Add Redis."
  118. - **Observation:** `user-reported` latency increase; not independently verified.
  119. - **Interpretation:** the database is slow. **Causal hypothesis:** repeated reads
  120. delay requests. **Solution:** Redis. These are inferred, not measured facts.
  121. - **Current formulation:** database service time materially dominates the slow tail.
  122. - **Alternatives:** queueing/capacity, downstream dependency delay, or a measurement
  123. artifact. Runtime pauses and workload shifts may also matter if evidence suggests them.
  124. - **Load-bearing assumption:** slow requests actually spend material time in DB work.
  125. - **Primary Discriminating Test:** inspect an available, representative sample of
  126. slow and typical requests from the same window; reconcile end-to-end latency
  127. with queue, DB, and downstream time in one comparison.
  128. - **Outcomes:** elevated DB time strengthens the current framing; rising queue wait
  129. with stable DB time weakens it and supports capacity; inconsistent totals support
  130. measurement trouble. Sparse or overlapping spans leave uncertainty.
  131. If the evidence has not been inspected: `INSUFFICIENT EVIDENCE`, solutioning
  132. `NOT YET`. The host should inspect it when accessible. If reliable observations
  133. instead establish queue wait as the material delay: `REFORMULATE`, with that
  134. revised problem. Neither result automatically authorizes a Redis implementation.
  135. This is an illustrative example, not a reported production incident.
  136. ## When to use
  137. Invoke explicitly when you want the current framing challenged. Automatic use
  138. requires **both meaningful formulation uncertainty and non-trivial intervention
  139. cost or risk**. Typical signals include:
  140. - A proposed solution arriving before the problem has been tested.
  141. - A causal hypothesis being treated as an observation.
  142. - Repeated failed debugging or research attempts that preserve the same framing.
  143. - Technology selection or an expensive product change preceding diagnosis.
  144. ## When NOT to use
  145. Usually skip automatic activation for verified diagnoses, obvious compiler errors,
  146. typos, formatting, translation, mechanical edits, or very cheap reversible changes.
  147. Do not invent uncertainty merely because alternatives are logically possible.
  148. For example, a compiler reports a missing `UserService`, and repository inspection
  149. confirms it was renamed to `AccountService`. "Fix the import" normally bypasses
  150. this skill. If explicitly invoked, a brief `KEEP` releases the host to fix it.
  151. ## What makes it different
  152. **These methods overlap. This project focuses on a narrower operational responsibility.**
  153. | Related method | Overlap and difference in emphasis |
  154. | --- | --- |
  155. | problem-framing | Often seeks a clearer, more useful, or better-scoped formulation. This skill treats the current one as falsifiable, seeks disconfirmation, updates from a discriminating test, and defines when to stop. Other framing methods can also challenge assumptions. |
  156. | debugging | Often asks what caused a failure. This skill first asks why we classify the failure this way. Root-cause investigation may follow, but a list of possible causes is not sufficient formulation diversity. |
  157. | grill-me | Attacks a plan, design, or proposal. This skill attacks the problem formulation before the plan should exist. |
  158. | critical-thinking | A broader reasoning toolbox. This skill has one job: validate, weaken, reject, or replace the current problem framing before solutioning. |
  159. The difference is emphasis, operational protocol, evidence update, and stopping
  160. condition, not a claim to have invented reframing.
  161. ## Usage
  162. ```text
  163. Use falsify-the-problem before solving this:
  164. Our API P99 rose from 300 ms to 2.8 s. We think adding Redis will fix it.
  165. Inspect the locally available evidence and challenge that framing first.
  166. ```
  167. ```text
  168. Use falsify-the-problem:
  169. Retention is low and we want to add push notifications.
  170. What cheap observation would distinguish forgetting from never experiencing value?
  171. ```
  172. ```text
  173. Use falsify-the-problem, then complete the authorized fix if the framing survives:
  174. The compiler error and repository rename both show the import is outdated.
  175. Fix that import.
  176. ```
  177. The skill controls the framing pass. The host handles tools, access, permissions,
  178. and downstream implementation. A gate on solutioning must not become a refusal
  179. to inspect evidence or a permanent stop after `KEEP`.
  180. ## Evaluation
  181. See [evaluation instructions](evals/README.md), [20 cases](evals/cases.md), and the
  182. [scoring rubric](evals/rubric.md). Coverage includes all five domains, multi-turn
  183. contradictions, unavailable evidence, accessible artifacts, and negative controls.
  184. Cases test reasoning behavior rather than exact wording.
  185. The [smoke-test record](evals/smoke-tests.md) records execution context, outputs,
  186. scores, review findings, and revisions. **This is a behavioral smoke test, not a
  187. statistically valid benchmark.** Small development runs cannot establish general
  188. effectiveness or host-independent behavior. Baseline provenance and any missing
  189. execution must be stated explicitly; a counterfactual analysis is not model output.
  190. The [full-suite follow-up](evals/full-suite.md) records 20 paired cases, independent
  191. grading with arm labels withheld, and catalog-based activation probes.
  192. Some variants supply reports instead of underlying raw systems. Labels are hidden
  193. from graders, but protocol wording can reveal the treatment. The record preserves
  194. original scores, author disagreements, actual local fixes, and answer-length cost.
  195. It does not establish cross-model or cross-host effectiveness.
  196. The [R03 artifact follow-up](evals/r03-raw.md) uses staged, synthetic row-level
  197. predictions to test selection and execution of a site-by-preprocessing comparison.
  198. It separates target error reduction from reduction of the between-site gap.
  199. These authored outputs do not constitute execution of an actual classifier.
  200. Both arms scored 10/10 on this single variant; that is no evidence of a skill
  201. advantage. The record preserves the label-audit and grading-scope qualifications.
  202. The [R03 missing-pair test](evals/r03-stress.md) removes the initial replay hint and
  203. adds regressions and incomplete coverage. It tests whether an agent distinguishes
  204. matched-subset outcomes from an unresolved outcome for all selected records.
  205. Both arms again scored 10/10; the record preserves the first-test labeling
  206. ambiguity and its stricter alternative score. This remains a single authored pair.
  207. The [harder formulation suite](evals/hard-suite.md) adds 12 cases targeting partial
  208. explanations, shared blind spots, wrong targets, confounded tests, and reversals,
  209. including two verified negative controls. The [coverage micro-suite](evals/coverage-eval.md)
  210. adds six targeted cases across five domains. These development records report
  211. quality, failure modes, and visible answer length separately; the earlier results
  212. remain historical snapshots, not results for the revised skill.
  213. In this run, the nine predeclared normal hard cases had median visible lengths of
  214. 218 words with the previous Skill, 178 with the revised Skill, and 117 with the raw
  215. baseline (turns summed per case). Coverage scores tied at 10/10 on all six targeted
  216. cases, and R03's first-round input-path omission remains. This supports a narrower
  217. output-cost finding, not a demonstrated general coverage or raw-model advantage.
  218. ## Design principles
  219. - One responsibility, one domain-agnostic protocol, one portable instruction file.
  220. - Preserve evidence provenance; reports are not independent verification.
  221. - Prefer materially different formulations to cosmetic variants of one cause.
  222. - Choose tests whose possible outcomes change the decision.
  223. - Let evidence overturn earlier conclusions. Keep investigation proportional.
  224. - Treat `KEEP` and a prompt downstream handoff as normal success.
  225. ## Limitations
  226. Markdown instructions guide a host; they do not enforce a runtime gate. Evidence
  227. quality, access, model judgment, and selection behavior affect results. A test may
  228. be inconclusive, support mixed causes, or only justify a narrow scope. The skill
  229. does not prove causal identification, guarantee a correct replacement, or replace
  230. specialist review. It is not a harness, runtime, generic debugging framework,
  231. multi-agent system, orchestration engine, memory system, or AI scientist.
  232. ## License
  233. [MIT](LICENSE).
  234. Maintainers: see [Pages publishing and Bing setup](SEO.md).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.