falsify-the-problem
An open-source Agent Skill for problem framing and assumption testing across coding, debugging, architecture, scientific research, and product decisions.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/ordinary-s--falsify-the-problem)HTML
<a href="/listings/ordinary-s--falsify-the-problem"><img src="/api/listings/ordinary-s--falsify-the-problem/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
The complete runtime artifact is [SKILL.md](SKILL.md). Evaluation files are development materials, not dependencies. The skill itself needs no package, API key, or service; your host agent has its own setup requirements.
Codex: choose an installation scope
Codex is one way to use this cross-domain skill. According to the [official skills documentation](https://learn.chatgpt.com/docs/build-skills), local skills can live in either of these locations:
| Scope | Folder containing `SKILL.md` | Available to | | --- | --- | --- | | Project | `<project>/.agents/skills/falsify-the-problem/` | Codex sessions in that project | | User | `~/.agents/skills/falsify-the-problem/` | Codex sessions across your projects |
Choose one scope. Run the commands below in a **terminal**, not in the Codex prompt. Git must be installed. Project commands start in the root of the project where you want to use the skill; user commands can start in any directory. These commands clone into a new folder and will fail if the destination is already populated. Keep an existing installation instead of overwriting it.
**Windows PowerShell, project scope:**
New-Item -ItemType Directory -Force -Path .agents/skills | Out-Null git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem
**Windows PowerShell, user scope:**
$skillRoot = Join-Path $env:USERPROFILE '.agents/skills' New-Item -ItemType Directory -Force -Path $skillRoot | Out-Null git clone https://github.com/ordinary-s/falsify-the-problem.git (Join-Path $skillRoot 'falsify-the-problem')
**macOS / Linux, Bash or Zsh, project scope:**
mkdir -p .agents/skills git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem
**macOS / Linux, Bash or Zsh, user scope:**
mkdir -p "$HOME/.agents/skills" git clone https://github.com/ordinary-s/falsify-the-problem.git "$HOME/.agents/skills/falsify-the-problem"
Alternatively, download or clone this repository elsewhere and copy only `SKILL.md` into the chosen `falsify-the-problem` folder. Codex detects skill changes automatically; restart it if the skill does not appear. In Codex CLI or the IDE extension, use `/skills` or type `$` to select the skill.
First invocation in Codex CLI or the IDE extension
Paste this into the **Codex prompt**, not PowerShell or a shell:
$falsify-the-problem Our API has become slower and we are considering a cache. Inspect the available evidence, distinguish observations from assumptions, and test the problem framing before proposing a solution.
The local setup commands do not establish skill activation or model behavior. macOS and Linux installation and cross-host behavior have not been verified in this documentation change. See [Usage](#usage) for ordinary-language examples.
Other hosts
1. Download or clone this repository into a local `falsify-the-problem` folder. 2. For a host with skill-folder support, place the folder containing `SKILL.md` in that host's documented skill location and use its normal reload/discovery procedure. You can copy just `SKILL.md` into a folder of the same name. 3. For a host that accepts reusable Markdown instructions, attach or load the complete `SKILL.md` and explicitly request its use before solving the task.
Paths, discovery, invocation syntax, and tool permissions are host-specific. The skill is designed to be portable across agents that support reusable Markdown instructions or skill-like workflows; identical behavior across hosts is not guaranteed.
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 292 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: falsify-the-problem
- description: Challenge the current problem formulation before solving it. Use when explicitly invoked, or when meaningful framing uncertainty and non-trivial intervention cost or risk coexist; do not auto-activate for verified diagnoses or mechanical edits.
- ---
- # falsify-the-problem
- > Falsify the problem before solving it.
- Answer **Are we solving the right problem?** Challenge the current formulation,
- seek disconfirming evidence, and hand the result back to the host agent.
- Do not design or implement the solution inside this skill.
- Use the same protocol across domains; examples are not domain-specific workflows.
- ## 1. Decide whether to enter
- - Explicit invocation of `falsify-the-problem`: always enter. Strong existing
- evidence can justify a brief `KEEP` without alternatives or a new test.
- - Automatic activation: require BOTH meaningful formulation uncertainty AND
- non-trivial intervention cost or risk. Neither alone is enough.
- - Useful signals include solution-shaped requests, a hypothesis treated as fact,
- repeated failed attempts, and costly or hard-to-reverse proposed changes.
- Signals are reasons to check the gate, not unconditional triggers.
- - Usually do not auto-activate for a verified diagnosis, obvious syntax or
- compilation error, typo, translation, formatting, mechanical transformation,
- deterministic local edit, or very cheap reversible intervention.
- - Mere logical possibility of another explanation is insufficient.
- `BYPASS` is an activation decision, never an evidence verdict. When bypassing,
- leave the original task with the host; do not manufacture an adversarial report.
- If explicitly invoked on a supported diagnosis, use `KEEP`, not `BYPASS`.
- ## 2. Separate the claims and preserve their sources
- Distinguish these five objects, even when the user combines them in one sentence:
- | Object | Meaning | Example |
- | --- | --- | --- |
- | Observation | Something observed, measured, or reported | User reports P99 rose from 300 ms to 2.8 s |
- | Interpretation | Meaning assigned to an observation | The database is slow |
- | Causal Hypothesis | A specific mechanism proposed to explain it | Lock contention delays requests |
- | Problem Formulation | What kind of problem is being solved | Database service time dominates the slow tail |
- | Solution | A proposed intervention | Add Redis |
- Observation != Interpretation != Causal Hypothesis != Problem Formulation != Solution.
- For each consequential observation, retain the source and relevant scope:
- - `tool-observed`: name the file, command, trace, or other inspected artifact.
- - `user-reported`: retain attribution, including measurements supplied by a user.
- - `document-reported`: identify the document and its claim.
- - `experimentally observed`: identify the experiment and conditions.
- - `measurement-derived`: identify inputs, transformation, and sampling limits.
- These labels can combine. Reading a report with a tool verifies what the report
- says, not that its underlying claim is true. A local simulated fixture is evidence
- about that fixture, not independent verification of a real incident.
- Never upgrade user reports into independently verified facts.
- Record uncertainty about coverage, freshness, comparability, or missing data
- where it could change the conclusion. Do not fabricate observations or execution.
- State the **Current Problem Formulation** explicitly. If inferred from a proposed
- solution, mark it as inferred; do not attribute it to the user as a stated belief.
- Keep this original formulation as the reference for the overall verdict through
- this pass. A replacement is `REFORMULATE`, not a silently relabeled `KEEP`.
- Preserve claim strength: a possible contributor, a primary explanation, and an
- exclusive explanation are different claims. Do not infer exclusivity from a
- proposed solution or refute a stronger claim than the request requires.
- ## 3. Build materially different live formulations
- Ask what would change the kind of problem being investigated. Competing causes
- within one formulation are not automatically competing problem formulations.
- Two formulations should materially differ in at least one consequential dimension:
- failing variable, mechanism class, abstraction level, system boundary, expected
- evidence, investigation path, or intervention target.
- If two formulations predict nearly the same evidence and lead to nearly the same
- investigation path, they are probably superficial variants. Merge them.
- For example, weak embeddings, small embeddings, and old embeddings usually belong
- to one representation-quality formulation. Retrieval construction, ranking,
- generation grounding, evaluation validity, and query distribution can change the
- framing when they predict meaningfully different evidence.
- Keep only plausible, decision-relevant alternatives. Do not mechanically fill a
- quota. Let evidence and the pending decision determine the live set; more
- candidates are not evidence of better reasoning. Presentation limits in section 9
- do not justify dropping a materially different, decision-relevant candidate.
- Candidates need not be mutually exclusive. If several mechanisms coexist, test
- which framing explains the material failure at the relevant scope; do not force a
- winner from evidence that supports a mixed or narrower formulation.
- ## 4. Find load-bearing assumptions and distinct predictions
- Prioritize assumptions whose failure would materially weaken or collapse a live
- formulation. Avoid exhaustive lists or generic questions about everything.
- Include measurement validity only when it bears on the decision.
- For each important live formulation, connect:
- `Formulation -> load-bearing assumption -> observable prediction -> contrary evidence`
- A prediction must distinguish live formulations, not restate their names.
- Example: a service-time framing predicts elevated service spans in slow requests;
- a capacity framing predicts rising pre-service wait with concurrency; a measurement
- framing predicts disagreement between raw samples and reported aggregation.
- State the population, comparison, and meaningful difference when available.
- Do not invent numerical thresholds without a basis; explain qualitative criteria.
- Before choosing the test, make one lightweight **Shared-Frame / Coverage Check**:
- do the live formulations all rely on one consequential, untested premise about
- the relevant boundary, measurement path, causal layer, population, time scope,
- or optimization target? Select what matters here; do not audit a category checklist.
- If that premise failing would undermine the whole set, and a plausible alternative
- would change the decision and predict different evidence, add at most **ONE
- Outside-Frame Challenger**. Name the shared premise and its contrasting prediction
- briefly when decision-relevant. Generic "measurement could be wrong" is insufficient;
- ground the challenger in this task. If none is warranted, proceed without one.
- Do not chain coverage audits or expand the set merely to satisfy a quota.
- ## 5. Select ONE Primary Discriminating Test for this round
- When beginning an investigation round under unresolved uncertainty, choose
- **exactly ONE Primary Discriminating Test**. This is one per round, not one forever.
- Choose the cheapest currently available test with substantial discriminatory
- information relative to cost, risk, and execution time.
- Ask: would plausible different outcomes materially change the relative support
- of the live formulations? If not, choose a better test.
- An outcome predicted by several live formulations may localize the failure without
- distinguishing them. Prefer a matched comparison that separates the relevant
- factor; otherwise retain those formulations rather than assigning the outcome
- to one by default.
- Specify:
- - **Test:** one bounded comparison or observation, its source, and its scope.
- - **Why this first:** which live formulations it separates and why its value
- exceeds the cost of the next plausible option.
- - **Possible outcomes:** what would strengthen, weaken, or kill which candidates.
- Include inconclusive or mixed outcomes where plausible.
- - **Remaining uncertainty:** what this test cannot establish.
- One test can require several retrieval or computation steps if they serve one
- predeclared comparison and outcome map. It cannot hide independent investigations
- inside a bundle such as "check logs, metrics, configuration, and tests."
- Include an unaccounted or outside-boundary outcome when the observation method
- could omit relevant stages; an artifact's label does not define its boundaries.
- Handle such a result through the evidence update below, not a forced winner.
- Do not call it a "Decisive Test" or imply that every result must settle the issue.
- ## 6. Gather evidence instead of deflecting it
- STOP solutioning != STOP investigating.
- Inspect evidence directly when it is available, permitted, low-risk, and reasonably
- cheap. Use the host's tools as appropriate: local artifacts, code, history, logs,
- traces, metrics, configurations, documentation, existing tests, safe diagnostics,
- low-risk experiments, or benchmarks. No specific tool or runtime is required.
- Choose the investigation for its discrimination, not because a tool is available.
- Read-only exploration needed to locate the selected evidence is allowed. Bound it
- to the comparison; do not turn discovery into a generic diagnostic checklist.
- Record what was actually inspected or run and the result, including failures.
- If the selected source is inaccessible, prefer an available lower-cost proxy only
- if it can discriminate; record the limitation. If no useful proxy exists, state
- the one pending Primary Discriminating Test and request only the minimum missing
- discriminating evidence or access. Do not ask users to do work the host can safely
- perform. Do not claim the pending test ran or loop through identical requests.
- Investigation does not authorize implementation. Before formulation validation,
- do not implement the requested solution, migrate architecture, introduce
- infrastructure, make expensive production changes, take irreversible action, or
- optimize around an unvalidated premise. Calling an intervention a "test" does not
- make it low-risk. Respect the host's existing permissions and user task scope.
- ## 7. Update from evidence, then decide whether to continue
- Evidence beats rhetoric and the agent's previous conclusion. Accept contradictory
- evidence; do not reinterpret it merely to preserve an earlier position.
- After the selected test:
- 1. Attribute the result and compare it with the predicted outcomes.
- 2. If a consequential result fits no live formulation, mark it **OUTSIDE CURRENT
- FRAME**: the set may be incomplete. This is an evidence diagnostic, not a sixth
- verdict. Preserve the unexplained residual; do not award it to the closest
- candidate. Repeat the lightweight coverage check once for this new evidence,
- adding at most one grounded challenger if warranted. Otherwise leave it unresolved.
- 3. Update affected candidates using the labels below with a short evidence reason,
- and re-check load-bearing assumptions invalidated by the test.
- 4. Merge redundant candidates; retain uncertainty and scope limits.
- 5. Apply the STOP conditions. If a further discriminating observation is worth its
- cost, start the next round with the updated live set and one new Primary Test.
- | Candidate update | Meaning |
- | --- | --- |
- | STRENGTHEN | New evidence raises support, without proving the candidate |
- | WEAKEN | New evidence reduces support; the candidate remains live |
- | KILL | Reliable evidence contradicts a necessary assumption at the stated scope |
- | UNRESOLVED | Evidence does not discriminate enough or is unavailable |
- | MERGE | Combine with a named candidate because the distinction is redundant |
- Missing evidence is not contrary evidence. An absent signal kills a candidate only
- when coverage and detection were sufficient for that signal to be expected.
- If new evidence undermines a prior kill, explicitly reopen the candidate and give
- the reason; labels are revisable conclusions, not permanent states.
- Candidate updates and the overall verdict have different meanings and scopes.
- ## 8. STOP, give an overall verdict, and hand back control
- End the adversarial pass when any of these holds:
- - The current formulation has sufficient evidence for the pending decision.
- - No meaningful materially different alternative remains, or remaining
- alternatives are too weak to justify delaying the decision.
- - Further investigation has poor information gain relative to cost, risk, or time.
- - Another formulation is sufficiently supported to replace the original.
- - The original formulation is dead and no useful cheap investigation remains.
- - Essential discriminating evidence is unavailable and no useful proxy exists;
- hand off the minimal evidence request rather than stall indefinitely.
- Sufficiency is proportional to the pending intervention and evidence quality.
- Running out of tests, access, time, or alternatives is not itself evidence for KEEP.
- Do not become contrarian for its own sake. KEEP is a normal successful outcome.
- Use exactly one overall verdict about the original Current Problem Formulation:
- | Overall verdict | Meaning | Solutioning / handoff |
- | --- | --- | --- |
- | KEEP | Current framing remains best-supported and sufficiently validated for the pending decision | ALLOWED for that framing; release to the downstream solver |
- | WEAKEN | Current framing remains live but has materially lost support; no replacement is sufficiently established | NOT YET; continue useful investigation or hand off the specific uncertainty |
- | KILL | Current framing is no longer viable, with no sufficiently supported replacement | BLOCKED for the killed premise; investigate another framing if worthwhile |
- | REFORMULATE | A materially different framing is sufficiently better supported to replace the original | NOT YET; return the Reformulated Problem to the downstream workflow; the old solution has no release |
- | INSUFFICIENT EVIDENCE | Available evidence cannot discriminate enough to support a stronger verdict | NOT YET; investigate if worthwhile, otherwise state the minimum missing evidence |
- For REFORMULATE, always output **Reformulated Problem:** with the replacement and
- its supporting evidence. KILL does not assert a replacement; REFORMULATE does.
- If several descriptions fit, choose the most informative supported verdict:
- an established replacement takes REFORMULATE over KILL or WEAKEN; absent one,
- contradicted necessary assumptions justify KILL, reduced support justifies WEAKEN,
- and non-discrimination justifies INSUFFICIENT EVIDENCE.
- Skill STOP != Host Agent STOP. KEEP ends this skill's responsibility. When the
- original request already authorizes implementation, the host can continue in the
- same turn without asking again. Validation is not new permission for extra work.
- KEEP validates the problem framing, not the effectiveness of the proposed solution.
- Mark any subsequent diagnosis or implementation as host/downstream continuation
- so a completed framing pass does not silently turn into a root-cause workflow.
- After REFORMULATE, the host may run a brief validation pass with the replacement
- as the new Current Problem Formulation, reusing the evidence already gathered.
- If it warrants KEEP, release it without redundant testing; then the host can solve
- within the user's authorized scope. Do not implement the original solution merely
- because an alternative explanation was found. The skill itself never designs it.
- ## 9. Default to compact output; disclose detail as needed
- Complete the relevant checks, but show only decision-relevant evidence and
- rationale, not an exhaustive reasoning transcript or hidden chain of thought.
- Choose the shortest path that preserves the decision, discrimination, and scope:
- | Path | When | Visible output |
- | --- | --- | --- |
- | Fast | BYPASS, strong KEEP, verified deterministic diagnosis, or simple low uncertainty | Usually 2-5 sentences: source-backed observation, framing, verdict/release if entered, then host continuation. No invented alternative or new test. BYPASS needs no adversarial report. |
- | Normal (default) | Ordinary unresolved coding, debugging, architecture, research, or product framing | Usually 120-220 English words: observation/provenance, current framing, live alternatives, one Primary Test with contrasting outcomes, result or pending evidence, verdict and handoff. Fold each key assumption into its prediction. |
- | Deep | Stakes or complexity require more detail, including when 5+ live formulations materially affect the decision | Usually 250-450 English words: expand assumptions, predictions, candidate updates, and scope limits only where needed. Domain alone does not require this path. |
- These are soft presentation guides, not word quotas or correctness gates; use
- equivalent brevity in other languages. Normal output usually shows 2-4 live
- formulations including the current one, at most 3-4 by default. Do not conceal a
- consequential challenger to fit that range: merge genuine redundancies, or expand
- the output when needed. A supported single formulation needs no padding.
- Keep provenance compact (for example, "report.txt, supplied summary"), never omit
- it or upgrade it. State whether the one primary comparison ran or is pending;
- preliminary localization is not a discriminating test when all candidates predict
- it. Give one clear verdict and release status, plus **Reformulated Problem:** when
- required. Headings are optional; do not mechanically reproduce all workflow steps.
- On later turns, use **delta output**: new evidence/source, what changed, affected
- candidate updates and scope, a new Primary Test only if useful, then verdict/handoff.
- Do not repeat unchanged observations, formulations, assumptions, or predictions.
- Reference prior candidates briefly so the update remains intelligible.
- Do not repeat the user prompt, narrate internal exploration, stage a debate,
- explain the methodology unless asked, or list discarded/merged candidates unless
- their change matters now. Stop once the framing decision is supported; mark any
- authorized downstream work separately and let the host continue.
README.md · README · 305 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # falsify-the-problem — An Agent Skill for Problem Framing
- English | [简体中文](README.zh-CN.md)
- [Project website](https://ordinary-s.github.io/falsify-the-problem/)
- > Stop solving the wrong problem faster.
- An open-source Agent Skill for problem framing and assumption testing across
- coding, debugging, architecture, scientific research, and product decisions.
- It asks **Are we solving the right problem?** before proposing solutions.
- Start with [installation](#installation) and a [first invocation](#first-invocation-in-codex-cli-or-the-ide-extension).
- **Illustrative scenarios, not production incidents or measured results:**
- | Before | After |
- | --- | --- |
- | "API P99 rose from 300 ms to 2.8 s. Add Redis." -> design a cache | Attribute the reported latency, test whether database time actually dominates the slow tail, then keep or replace that framing |
- | "The experiment underperformed. We need a stronger model." -> increase model capacity | Compare the same saved inputs through training and evaluation preprocessing, holding the model and scoring fixed; test whether an input-path mismatch contributes before attributing the result to model capability |
- | "Retention is low. Add push notifications." -> build reminders | Test whether users experienced value and forgot to return, or never experienced value in the first place |
- One Markdown file. No dependency, API key, server, or database. The distinctive
- focus is a falsifiable framing, materially different alternatives, one primary
- discriminating test per investigation round, evidence updates, and an explicit
- handoff before solution design. `KEEP` is a successful result.
- ## Installation
- The complete runtime artifact is [SKILL.md](SKILL.md). Evaluation files are
- development materials, not dependencies. The skill itself needs no package,
- API key, or service; your host agent has its own setup requirements.
- ### Codex: choose an installation scope
- Codex is one way to use this cross-domain skill. According to the
- [official skills documentation](https://learn.chatgpt.com/docs/build-skills),
- local skills can live in either of these locations:
- | Scope | Folder containing `SKILL.md` | Available to |
- | --- | --- | --- |
- | Project | `<project>/.agents/skills/falsify-the-problem/` | Codex sessions in that project |
- | User | `~/.agents/skills/falsify-the-problem/` | Codex sessions across your projects |
- Choose one scope. Run the commands below in a **terminal**, not in the Codex
- prompt. Git must be installed. Project commands start in the root of the project
- where you want to use the skill; user commands can start in any directory.
- These commands clone into a new folder and will fail if the destination is
- already populated. Keep an existing installation instead of overwriting it.
- **Windows PowerShell, project scope:**
- ```powershell
- New-Item -ItemType Directory -Force -Path .agents/skills | Out-Null
- git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem
- ```
- **Windows PowerShell, user scope:**
- ```powershell
- $skillRoot = Join-Path $env:USERPROFILE '.agents/skills'
- New-Item -ItemType Directory -Force -Path $skillRoot | Out-Null
- git clone https://github.com/ordinary-s/falsify-the-problem.git (Join-Path $skillRoot 'falsify-the-problem')
- ```
- **macOS / Linux, Bash or Zsh, project scope:**
- ```bash
- mkdir -p .agents/skills
- git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem
- ```
- **macOS / Linux, Bash or Zsh, user scope:**
- ```bash
- mkdir -p "$HOME/.agents/skills"
- git clone https://github.com/ordinary-s/falsify-the-problem.git "$HOME/.agents/skills/falsify-the-problem"
- ```
- Alternatively, download or clone this repository elsewhere and copy only
- `SKILL.md` into the chosen `falsify-the-problem` folder. Codex detects skill
- changes automatically; restart it if the skill does not appear. In Codex CLI
- or the IDE extension, use `/skills` or type `$` to select the skill.
- ### First invocation in Codex CLI or the IDE extension
- Paste this into the **Codex prompt**, not PowerShell or a shell:
- ```text
- $falsify-the-problem
- Our API has become slower and we are considering a cache.
- Inspect the available evidence, distinguish observations from assumptions,
- and test the problem framing before proposing a solution.
- ```
- The local setup commands do not establish skill activation or model behavior.
- macOS and Linux installation and cross-host behavior have not been verified in
- this documentation change. See [Usage](#usage) for ordinary-language examples.
- ### Other hosts
- 1. Download or clone this repository into a local `falsify-the-problem` folder.
- 2. For a host with skill-folder support, place the folder containing `SKILL.md` in
- that host's documented skill location and use its normal reload/discovery procedure.
- You can copy just `SKILL.md` into a folder of the same name.
- 3. For a host that accepts reusable Markdown instructions, attach or load the
- complete `SKILL.md` and explicitly request its use before solving the task.
- Paths, discovery, invocation syntax, and tool permissions are host-specific. The
- skill is designed to be portable across agents that support reusable Markdown
- instructions or skill-like workflows; identical behavior across hosts is not guaranteed.
- ## Why
- A requested solution often contains an untested definition of the problem.
- Optimizing that solution can make the wrong investigation expensive.
- This skill exposes the definition and seeks cheap evidence that could overturn it.
- It is useful across coding, debugging, architecture, research, and product work.
- The protocol stays the same across domains.
- ## How it works
- 1. Preserve observations and their provenance; separate interpretation, causal
- hypothesis, problem formulation, and proposed solution.
- 2. Identify the current formulation and plausible, materially different alternatives.
- 3. Find load-bearing assumptions and derive distinct observable predictions.
- 4. Select **one Primary Discriminating Test per round**, balancing information
- against cost, risk, and time. Inspect available, permitted, cheap evidence directly.
- 5. Strengthen, weaken, kill, or merge candidates from the result. Investigate again
- when another discriminating observation is worth its cost.
- 6. Stop with an overall verdict and return control to the host agent.
- Before choosing the Primary Discriminating Test, the skill checks whether all
- live formulations share the same consequential, untested frame. It may add one
- grounded outside-frame challenger; unexplained evidence can reopen that check.
- Output defaults to compact Fast / Normal / Deep paths, with later turns showing
- only changed evidence and decisions.
- | Overall verdict | Meaning | Handoff |
- | --- | --- | --- |
- | KEEP | Current framing is sufficiently validated and best-supported | Solutioning ALLOWED; host resumes authorized work |
- | WEAKEN | Current framing lost material support but remains live | NOT YET; investigate the remaining uncertainty |
- | KILL | Original framing is no longer viable; no replacement established | BLOCKED for that premise |
- | REFORMULATE | A materially different replacement is sufficiently supported | NOT YET; return a named Reformulated Problem for downstream validation |
- | INSUFFICIENT EVIDENCE | Evidence cannot discriminate enough | NOT YET; investigate or request minimum missing evidence |
- `BYPASS` is an activation decision, not a verdict. `KILL` does not imply a known
- replacement. After `REFORMULATE`, the host can validate the replacement in a brief
- new pass using existing evidence, obtain `KEEP`, and continue within task scope.
- **STOP solutioning does not mean STOP investigating. Skill STOP does not mean
- Host Agent STOP.** After `KEEP`, a host already authorized to implement can proceed
- in the same turn without asking for permission again. No new authority is granted.
- ## Quick example
- User: "API P99 increased from 300 ms to 2.8 s. Add Redis."
- - **Observation:** `user-reported` latency increase; not independently verified.
- - **Interpretation:** the database is slow. **Causal hypothesis:** repeated reads
- delay requests. **Solution:** Redis. These are inferred, not measured facts.
- - **Current formulation:** database service time materially dominates the slow tail.
- - **Alternatives:** queueing/capacity, downstream dependency delay, or a measurement
- artifact. Runtime pauses and workload shifts may also matter if evidence suggests them.
- - **Load-bearing assumption:** slow requests actually spend material time in DB work.
- - **Primary Discriminating Test:** inspect an available, representative sample of
- slow and typical requests from the same window; reconcile end-to-end latency
- with queue, DB, and downstream time in one comparison.
- - **Outcomes:** elevated DB time strengthens the current framing; rising queue wait
- with stable DB time weakens it and supports capacity; inconsistent totals support
- measurement trouble. Sparse or overlapping spans leave uncertainty.
- If the evidence has not been inspected: `INSUFFICIENT EVIDENCE`, solutioning
- `NOT YET`. The host should inspect it when accessible. If reliable observations
- instead establish queue wait as the material delay: `REFORMULATE`, with that
- revised problem. Neither result automatically authorizes a Redis implementation.
- This is an illustrative example, not a reported production incident.
- ## When to use
- Invoke explicitly when you want the current framing challenged. Automatic use
- requires **both meaningful formulation uncertainty and non-trivial intervention
- cost or risk**. Typical signals include:
- - A proposed solution arriving before the problem has been tested.
- - A causal hypothesis being treated as an observation.
- - Repeated failed debugging or research attempts that preserve the same framing.
- - Technology selection or an expensive product change preceding diagnosis.
- ## When NOT to use
- Usually skip automatic activation for verified diagnoses, obvious compiler errors,
- typos, formatting, translation, mechanical edits, or very cheap reversible changes.
- Do not invent uncertainty merely because alternatives are logically possible.
- For example, a compiler reports a missing `UserService`, and repository inspection
- confirms it was renamed to `AccountService`. "Fix the import" normally bypasses
- this skill. If explicitly invoked, a brief `KEEP` releases the host to fix it.
- ## What makes it different
- **These methods overlap. This project focuses on a narrower operational responsibility.**
- | Related method | Overlap and difference in emphasis |
- | --- | --- |
- | problem-framing | Often seeks a clearer, more useful, or better-scoped formulation. This skill treats the current one as falsifiable, seeks disconfirmation, updates from a discriminating test, and defines when to stop. Other framing methods can also challenge assumptions. |
- | debugging | Often asks what caused a failure. This skill first asks why we classify the failure this way. Root-cause investigation may follow, but a list of possible causes is not sufficient formulation diversity. |
- | grill-me | Attacks a plan, design, or proposal. This skill attacks the problem formulation before the plan should exist. |
- | critical-thinking | A broader reasoning toolbox. This skill has one job: validate, weaken, reject, or replace the current problem framing before solutioning. |
- The difference is emphasis, operational protocol, evidence update, and stopping
- condition, not a claim to have invented reframing.
- ## Usage
- ```text
- Use falsify-the-problem before solving this:
- Our API P99 rose from 300 ms to 2.8 s. We think adding Redis will fix it.
- Inspect the locally available evidence and challenge that framing first.
- ```
- ```text
- Use falsify-the-problem:
- Retention is low and we want to add push notifications.
- What cheap observation would distinguish forgetting from never experiencing value?
- ```
- ```text
- Use falsify-the-problem, then complete the authorized fix if the framing survives:
- The compiler error and repository rename both show the import is outdated.
- Fix that import.
- ```
- The skill controls the framing pass. The host handles tools, access, permissions,
- and downstream implementation. A gate on solutioning must not become a refusal
- to inspect evidence or a permanent stop after `KEEP`.
- ## Evaluation
- See [evaluation instructions](evals/README.md), [20 cases](evals/cases.md), and the
- [scoring rubric](evals/rubric.md). Coverage includes all five domains, multi-turn
- contradictions, unavailable evidence, accessible artifacts, and negative controls.
- Cases test reasoning behavior rather than exact wording.
- The [smoke-test record](evals/smoke-tests.md) records execution context, outputs,
- scores, review findings, and revisions. **This is a behavioral smoke test, not a
- statistically valid benchmark.** Small development runs cannot establish general
- effectiveness or host-independent behavior. Baseline provenance and any missing
- execution must be stated explicitly; a counterfactual analysis is not model output.
- The [full-suite follow-up](evals/full-suite.md) records 20 paired cases, independent
- grading with arm labels withheld, and catalog-based activation probes.
- Some variants supply reports instead of underlying raw systems. Labels are hidden
- from graders, but protocol wording can reveal the treatment. The record preserves
- original scores, author disagreements, actual local fixes, and answer-length cost.
- It does not establish cross-model or cross-host effectiveness.
- The [R03 artifact follow-up](evals/r03-raw.md) uses staged, synthetic row-level
- predictions to test selection and execution of a site-by-preprocessing comparison.
- It separates target error reduction from reduction of the between-site gap.
- These authored outputs do not constitute execution of an actual classifier.
- Both arms scored 10/10 on this single variant; that is no evidence of a skill
- advantage. The record preserves the label-audit and grading-scope qualifications.
- The [R03 missing-pair test](evals/r03-stress.md) removes the initial replay hint and
- adds regressions and incomplete coverage. It tests whether an agent distinguishes
- matched-subset outcomes from an unresolved outcome for all selected records.
- Both arms again scored 10/10; the record preserves the first-test labeling
- ambiguity and its stricter alternative score. This remains a single authored pair.
- The [harder formulation suite](evals/hard-suite.md) adds 12 cases targeting partial
- explanations, shared blind spots, wrong targets, confounded tests, and reversals,
- including two verified negative controls. The [coverage micro-suite](evals/coverage-eval.md)
- adds six targeted cases across five domains. These development records report
- quality, failure modes, and visible answer length separately; the earlier results
- remain historical snapshots, not results for the revised skill.
- In this run, the nine predeclared normal hard cases had median visible lengths of
- 218 words with the previous Skill, 178 with the revised Skill, and 117 with the raw
- baseline (turns summed per case). Coverage scores tied at 10/10 on all six targeted
- cases, and R03's first-round input-path omission remains. This supports a narrower
- output-cost finding, not a demonstrated general coverage or raw-model advantage.
- ## Design principles
- - One responsibility, one domain-agnostic protocol, one portable instruction file.
- - Preserve evidence provenance; reports are not independent verification.
- - Prefer materially different formulations to cosmetic variants of one cause.
- - Choose tests whose possible outcomes change the decision.
- - Let evidence overturn earlier conclusions. Keep investigation proportional.
- - Treat `KEEP` and a prompt downstream handoff as normal success.
- ## Limitations
- Markdown instructions guide a host; they do not enforce a runtime gate. Evidence
- quality, access, model judgment, and selection behavior affect results. A test may
- be inconclusive, support mixed causes, or only justify a narrow scope. The skill
- does not prove causal identification, guarantee a correct replacement, or replace
- specialist review. It is not a harness, runtime, generic debugging framework,
- multi-agent system, orchestration engine, memory system, or AI scientist.
- ## License
- [MIT](LICENSE).
- Maintainers: see [Pages publishing and Bing setup](SEO.md).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.