concord-skill-lean-testing
Plan, write, review and consolidate tests so the test set is complete and minimal — every required behavior is covered, every test catches a failure the rest of the suite would miss, no test pins config values or copies the code, new tests are placed against the existing suite instead of piled on top of it, and no defensive guard adds branches that exist only to be tested. Use this skill whenever you write a test plan, add or change tests, add a regression test for a bug fix, find that a new test duplicates or contradicts an existing one, decide what to do with an existing test that a change turned red, consolidate tests after a change, review a test diff, decide whether something needs a test at all, or are about to add a guard, an assert or a validation check — even when the request only says "add tests" or "make it robust".
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/bqp007--concord-skill-lean-testing)HTML
<a href="/listings/bqp007--concord-skill-lean-testing"><img src="/api/listings/bqp007--concord-skill-lean-testing/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Clone the repository into your Claude Code skills folder:
git clone https://github.com/bqp007/concord-skill-lean-testing.git ~/.claude/skills/concord-skill-lean-testing
Update it with `git -C ~/.claude/skills/concord-skill-lean-testing pull`. If you keep the checkout somewhere else, link it instead:
ln -s /path/to/concord-skill-lean-testing ~/.claude/skills/concord-skill-lean-testing
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 108 lines
- ---
- name: concord-skill-lean-testing
- description: Plan, write, review and consolidate tests so the test set is complete and minimal — every required behavior is covered, every test catches a failure the rest of the suite would miss, no test pins config values or copies the code, new tests are placed against the existing suite instead of piled on top of it, and no defensive guard adds branches that exist only to be tested. Use this skill whenever you write a test plan, add or change tests, add a regression test for a bug fix, find that a new test duplicates or contradicts an existing one, decide what to do with an existing test that a change turned red, consolidate tests after a change, review a test diff, decide whether something needs a test at all, or are about to add a guard, an assert or a validation check — even when the request only says "add tests" or "make it robust".
- ---
- # Lean testing
- A test earns its place when it catches a failure that no other test catches and passes on every change the spec allows. Extra tests are not free: each one is code to maintain, and a test that breaks on legitimate changes slows every later change without protecting anything.
- Agents drift toward too many tests for three reasons. Tests written right after reading the implementation assert what the code does, not what the spec requires. More tests feel safer, and nothing says when to stop. Tests are rarely deleted.
- This skill states the principles and what a finished test set must show. It does not prescribe a technique or a count. How you find the behaviors, and how many cases they take, is your judgment.
- ## Principles
- ### The expected value must outrank the code
- A failing test signals a mismatch between the code and an expectation. The failure helps only when a source independent of the code can settle which side is wrong.
- Sources that outrank the code:
- - the spec, the acceptance criteria, or a documented requirement;
- - a contract that another module, repository or team consumes: an API, a schema, a wire or file format;
- - a reference implementation the code must match;
- - a mathematical law or invariant: round trip, idempotence, ordering, monotonicity;
- - a golden file that the spec owner reviewed and approved.
- Sources with the same authority as the code:
- - config values: thresholds, lists, prompts, model names, version pins, flags;
- - the code's own output, recorded without review (a snapshot);
- - constants copied from the code;
- - the algorithm re-implemented inside the test.
- When a same-authority test fails, nobody can tell which side is wrong, so the fix is to make the two agree again. Such a test catches no failure, and it fails on every legitimate change. An unreviewed snapshot is a change detector, not a test, until someone establishes the behavior it should show. A reviewed golden file or a fixture can also be stale, so when a test fails, go back to the source to decide whether the code or the test changes.
- ### The test set is a filter on changes
- A good test set rejects every change that breaks a promise. It accepts every change the spec allows: refactors, config retunes, performance work, added features, renames and log text. A test that rejects an allowed change is as much a defect as a test that accepts a broken one.
- ### Config is not behavior
- A value is config when the spec names the parameter but not its value, and the value changes for tuning rather than for correctness. A test that asserts a config value only repeats the config file: whoever retunes the value must also edit the test, so the test catches nothing. A value is a contract when the spec fixes it exactly, such as a legal retention period or a protocol constant, and then a test may assert it.
- Test the logic around a config value with values the test chooses. For "flag a score at or above the threshold", inject a threshold `T` and assert that a score just below `T` is not flagged, that `T` is flagged, and that a score above `T` is flagged. The test survives any retune. The case at `T` catches `>` written for `>=`, the case above `T` catches `==`, and the case below `T` catches a rule that flags everything. Two inputs with the same expected result both earn their place when they catch different failures.
- Config still needs checks, and none of them asserts a tuned value:
- - The shipped files obey rules that outrank their values: required keys, types, ranges, and references between files. Enforce those rules in the production loader, so that production fails fast too, and run that loader on the final shipped artifacts.
- - Which profile is loaded and which override wins is behavior. When it changes, test it through the normal entry point, with fixture values you can tell apart.
- - A result that only means something with production data, models or config belongs in an evaluation, with a metric and a threshold the owner sets. Release checks and evaluations may read shipped config when it is part of what they verify.
- ### Test through contracts
- Test behavior through an interface that real callers use and that a refactor inside the unit leaves alone. A test that reaches a private symbol, calls a helper extracted only for the test, or mocks one of your own modules is tied to how the code is built, not to what it promises, so it breaks on refactors that keep behavior. Which layer to test at is your call: pick the one where the promise is stated and a failure shows clearly.
- ### Guards belong at trust boundaries
- A routine does not re-check its own preconditions; its callers own them. Validate input once, where it enters the system, and trust the types and the callers inside that boundary. Every guard adds a branch, and every branch needs a test or a reason, so a guard for a state that cannot occur costs twice and protects nothing. Where you can, choose a type that cannot hold the bad state.
- State that something else can change between the check and the use is input too: a balance that two requests spend at once, a row, a file, a remote resource. Enforce that invariant where the state changes, in one atomic step. That check is a trust boundary, not a defensive copy.
- ### The suite outlives the task
- The tests you write join a suite that other tasks wrote and later tasks will change. Each test is a standing claim: this behavior, this expected result, from this source. A task changes that set of claims, so weigh each new case against the tests that are already there. A new case earns its place only by catching a failure that the existing tests miss. When it catches everything an existing test catches, at the point where that test has to catch it, the older test has nothing left to add. A nightly test does not replace a check that must fail before merge. Whether the case becomes a new test, a row in an existing one or a replacement is your call.
- A new expectation that contradicts an existing test is a conflict between expectations, and the governing source settles it. An existing test that turns red is evidence first, and the source decides whether the code or the test changes: the requirement may have changed, the old expectation may have been wrong or tied to an incidental detail, or the new code may be wrong. When two sources of equal rank disagree, ask the owner. Do not edit an expectation until it passes without saying what the source supports.
- An expectation whose behavior the change removed can keep passing, so nothing flags it. Remove it in the same change, and keep the checks in that test for behavior that survives.
- Maintain the tests of the behaviors the task touches and leave the assertions of the others alone; a suite-wide cleanup is a task of its own. Do not remove a test because another test runs the same lines: equal coverage does not mean that the same failures are caught.
- All of this works only when a test says what it guards. Name each test for its behavior, not for a ticket or a bug number, and make its source findable. A test of unknown origin is a change detector until someone establishes what it should show.
- ## What a finished test set shows
- **Complete**
- - Every distinct behavior that the change must have, including its failure modes, is covered by at least one assertion that fails when the behavior breaks. A case may cover several behaviors. Behaviors come from the requirement, not from the lines changed.
- - No path through the changed code is left unaccounted for. Each one is exercised, or you can say where it runs (another platform, a deployment variant), which invariant prevents it, or what prevents verification. A required behavior that could not be verified stays open: report the gap, and do not count it as covered. Delete code only on evidence that it is dead across the supported configurations. A coverage percentage is not a goal.
- - A changed path that introduces an observable behavior no requirement asks for is an undecided behavior. Ask the owner to decide it, or remove it; a test would make it permanent. A path that keeps behavior, such as an equivalent fast path, needs no decision.
- **Minimal**
- - Every case catches a failure that the rest of the suite would miss, and you can say which one. Merge or remove a case when the remaining suite preserves its coverage.
- - Existing tests come first. Keep, extend or correct the test that already covers a behavior, and add or extend a case only when it catches a failure that the suite misses. When a test covered the behavior and let a bug through, fix that test.
- - No test asserts a tuned config value, repeats the algorithm, or is tied to incidental detail such as private symbols, its own setup or wall-clock time.
- - No guard exists for a state that cannot occur.
- **Consistent with the suite**
- - Every expectation whose meaning changed, or that was removed, is explained: by the requirement that changed, by why the old expectation was unsupported, or by the remaining test that keeps its coverage.
- - No expectation remains for a behavior the change removed, and the checks for behavior that survives are kept.
- - Tests of behaviors the task did not touch still exercise and verify what they did before. An edit to shared setup is fine when it preserves that.
- When all three hold, stop writing tests. At the end of a task, go over the tests it touched once more against these points.
- ## Reviewing tests
- A finding on a test names the affected behavior and shows evidence, such as:
- - a wrong implementation that the test still accepts (the test cannot fail);
- - an allowed change that the test rejects, such as a retuned config value or a refactor that keeps behavior (the test pins something it should not);
- - a required case that is skipped, never collected, or nondeterministic.
- Without evidence, the finding is a question, not a defect.
- ## Limits
- - Completeness holds only relative to the behaviors you listed and the paths you examined. Whether the behaviors match the real requirement is for the spec and its owner to settle.
- - Coverage shows that a branch ran, not that a test checked its result. Mutation testing on the diff shows the second, but it costs much more; weigh that cost against what the project requires.
- - Equal coverage alone does not establish that a test is redundant. That takes an argument about the failures each test catches.
README.md · README · 166 lines
- # concord-skill-lean-testing
- <p>
- <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
- <img alt="Claude Code skill" src="https://img.shields.io/badge/Claude%20Code-skill-orange.svg">
- </p>
- **Tests that defend the code without freezing it.**
- AI coding agents write tests faster than anyone can read them, and they write too many. They pin config values, mirror the implementation they just wrote, reach into private helpers, and add a guard (and a test for it) for every state they can imagine. Each test passes on the day it is written. Then every retune, refactor or rename turns some of them red, and the fix is always to edit the test. The suite has stopped defending the code and started freezing it.
- This skill gives an agent a small set of principles for what a test may assert, and a description of a finished test set, complete and minimal, so that it knows when to stop. It also says how new tests join a suite that already exists: what to do with a duplicate, a weaker old test, a contradiction, or a test that went stale. It prescribes no technique and no test count; how to get there is left to the agent's judgment.
- ## What changes
- Before: the test repeats a value from the config file.
- ```python
- def test_alert_threshold():
- assert config.ALERT_THRESHOLD == 0.40
- ```
- This test catches nothing, because whoever retunes the threshold also edits the test. It fails on every retune.
- After: the test checks the rule, with a threshold that the test chooses.
- ```python
- @pytest.mark.parametrize("score, flagged", [(0.69, False), (0.70, True), (0.71, True)])
- def test_flags_scores_at_or_above_the_threshold(score, flagged):
- assert classify(score, threshold=0.70) is flagged
- ```
- This test survives any retune. The case at 0.70 fails if someone writes `>` where the rule says `>=`, the case at 0.71 fails for `==`, and the case at 0.69 fails for a rule that flags everything. Two inputs with the same expected result both earn their place, because they catch different failures.
- The same rule removes snapshot tests that nobody reviewed, tests of private helpers, mocks of the project's own modules, guards for states that cannot occur, and the reflex of adding a new regression test for every bug when an existing test should have caught it.
- ## How it works
- ```mermaid
- flowchart TD
- R[Requirement] --> B[Behaviors the change must have]
- D[Diff] --> BR[Paths through the changed code]
- B --> C{Complete?}
- BR --> C
- C -- a behavior or a path is unaccounted for --> T[Cover the behavior or account for the path]
- T --> C
- C -- yes --> M{Minimal?}
- M -- a case catches nothing new --> X[Merge or remove it]
- X --> M
- M -- yes --> K{Consistent with the suite?}
- K -- a changed or removed expectation is unexplained --> J[Explain it from its source or restore it]
- J --> K
- K -- yes --> S[Stop]
- ```
- Each new case is weighed against the tests that already exist:
- ```mermaid
- flowchart TD
- N[New case] --> Q{Compare with the existing tests of that behavior}
- Q -- catches a failure none of them catches --> A[Keep it as a new test or a new row]
- Q -- the existing suite catches every failure this case catches --> DN[Do not add it]
- Q -- catches everything an existing test catches where that test must catch it --> RP[The older test can go]
- Q -- contradicts an existing test --> CF{What does the source say?}
- CF -- the requirement changed --> RQ[Replace the old expectation and name the requirement]
- CF -- the old expectation was wrong or pinned a detail --> FX[Correct or remove it and say why]
- CF -- the new code is wrong --> CD[Change the code]
- CF -- sources of equal rank disagree --> OW[Ask the owner]
- ```
- ### The core rule
- A failing test signals a mismatch between the code and an expectation. The failure helps only when a source independent of the code can settle which side is wrong.
- | Outranks the code: a test may assert it | Same authority as the code: a test learns nothing from it |
- |---|---|
- | the spec or the acceptance criteria | config values: thresholds, lists, prompts, model names, version pins |
- | a contract that someone else consumes: an API, a schema, a file format | the code's own output, recorded without review |
- | a reference implementation | constants copied from the code |
- | a mathematical law: round trip, ordering, idempotence | the algorithm re-implemented inside the test |
- | a golden file that the spec owner approved | |
- When a same-authority test fails, nobody can tell which side is wrong, so the only fix is to make the two agree again. An unreviewed snapshot is a change detector, not a test. A reviewed golden file can also go stale, so a failure sends you back to the source to decide whether the code or the test changes.
- ### The principles
- 1. **The expected value must outrank the code.** See the table above.
- 2. **The test set is a filter on changes.** It rejects every change that breaks a promise and accepts every change the spec allows.
- 3. **Config is not behavior.** No test asserts a tuned value; unit tests inject their own. The production loader validates the shipped files, profile selection and override precedence are tested through the normal entry point, and quality on real data is an evaluation with an owner-set metric.
- 4. **Test through contracts.** Test behavior through an interface that real callers use and that a refactor leaves alone. A test tied to private symbols or to mocks of your own modules follows how the code is built, not what it promises. The layer is the agent's call.
- 5. **Guards belong at trust boundaries.** Validate where input enters the system, and enforce invariants on shared state in one atomic step where that state changes. A guard for a state that cannot occur adds a branch, and that branch then asks for a test.
- 6. **The suite outlives the task.** Each test is a standing claim, and a task changes the set of claims. A new case earns its place only by catching a failure that the existing tests miss. A contradiction is a conflict between expectations, and the governing source decides whether the code or the test changes. An expectation whose behavior is gone is removed in the same change, and checks for surviving behavior stay. Only the tests of the touched behaviors are maintained, and no test is removed on equal coverage alone.
- ### What a finished test set shows
- The test set is **complete** when every distinct behavior the change must have is covered by at least one assertion that fails when the behavior breaks, and no path through the changed code is left unaccounted for: it is exercised, or you can say where it runs, which invariant prevents it, or what prevents verification. A required behavior that could not be verified is reported as open, not counted as covered.
- The test set is **minimal** when every case catches a failure that the rest of the suite would miss, existing tests were reused before new ones were added, and no test asserts a tuned config value or is tied to incidental detail.
- The test set is **consistent with the suite** when every expectation whose meaning changed, or that was removed, is explained (by the requirement that changed, by why the old expectation was unsupported, or by the remaining test that keeps its coverage), no expectation remains for a removed behavior while checks for surviving behavior are kept, and tests of untouched behaviors still exercise and verify what they did before.
- A changed path that introduces an observable behavior no requirement asks for is an undecided behavior. The owner decides it, or the code goes. A test would only make it permanent.
- ## Install
- Clone the repository into your Claude Code skills folder:
- ```bash
- git clone https://github.com/bqp007/concord-skill-lean-testing.git ~/.claude/skills/concord-skill-lean-testing
- ```
- Update it with `git -C ~/.claude/skills/concord-skill-lean-testing pull`. If you keep the checkout somewhere else, link it instead:
- ```bash
- ln -s /path/to/concord-skill-lean-testing ~/.claude/skills/concord-skill-lean-testing
- ```
- ## Use
- Claude Code picks the skill up when a task plans, writes, reviews or consolidates tests, or is about to add a guard or a validation check. You can also ask for it by name.
- To make a planner load it at fixed points, add a line like this to your `CLAUDE.md`:
- ```text
- Test plan rows, failing tests and the consolidation pass: load the concord-skill-lean-testing skill first.
- ```
- The skill is a companion to [Concord](https://github.com/bqp007/concord-agent-harness), where a strong model writes the plan and the tests and cheap models write the code. It works the same way without Concord.
- ## Why these rules
- - **A test suite is a filter on changes.** It should reject every change that breaks a promise and accept every change the spec allows: refactors, retunes, performance work, new features, renames and log text. Most over-testing is a filter that rejects allowed changes.
- - **Information needs authority.** A test checked against config or against the code's own output can only report that something changed. It cannot report that something is wrong.
- - **Completeness and minimality come from two lists.** One list holds the behaviors that the requirement decides. The other holds the paths through the changed code. Each behavior is covered, each path is exercised or explained, and each case catches a failure the others miss. The size of the suite follows from the behaviors, not from a count fixed in advance.
- - **Principles, not procedure.** A capable model does not need a recipe, and a recipe constrains it. The skill says what a good test set is and leaves the method open.
- - **A red existing test is evidence, and an edited expectation is a claim.** Refactors, new features and bug fixes should leave sound expectations alone. So an edit to an expectation names the requirement that changed, or explains why the old expectation was unsupported. The alternative, editing tests until they pass, is how a suite stops defending anything.
- - **Deleting is maintenance; pruning on coverage is not.** In a study of six Java projects, Pinto, Sinha and Orso found that test repair is only one part of how suites evolve: many apparent additions and deletions were moves or renames, and obsolescence explained most of the failing tests that were deleted. Removing tests because coverage looks equal is different. Comparing coverage alone can flag tests as redundant when they are not (Koochakzadeh and Garousi), and coverage-based minimization has lost detection of seeded faults in some studies (Rothermel et al.).
- - **Guards cost twice.** Every guard adds a branch, and every branch asks for a test. Validating input once, at the boundary, keeps both the code and the tests small.
- - **Coverage percentages reward extra tests.** The stopping check asks that every path be accounted for, not that a number goes up.
- ## Limits
- - Completeness holds only relative to the behaviors you listed and the paths you examined. Whether the behaviors match the real requirement is for the spec and its owner to settle.
- - Coverage shows that a branch ran, not that a test checked its result. Mutation testing on the diff shows that, at a much higher cost.
- - Equal coverage alone does not establish that a test is redundant. That takes an argument about the failures each test catches.
- ## Sources
- - Vladimir Khorikov, *Unit Testing Principles, Practices, and Patterns* (Manning, 2020): the four pillars of a good test, the code-type quadrant, mocks only for unmanaged dependencies.
- - Kent Beck, "Test Desiderata" (2019): tests should be behavioral and structure-insensitive.
- - Google Testing Blog, "Change-Detector Tests Considered Harmful" (2015).
- - Thomas Ostrand and Marc Balcer, "The Category-Partition Method for Specifying and Generating Functional Tests," *Communications of the ACM*, 1988.
- - Matthias Grochtmann and Klaus Grimm, "Classification Trees for Partition Testing," *Software Testing, Verification and Reliability*, 1993.
- - RTCA DO-178C and Hayhurst, Veerhusen, Chilenski and Rierson, *A Practical Tutorial on Modified Condition/Decision Coverage*, NASA/TM-2001-210876 (2001): the formal version of "every condition can change the result", for projects that require it.
- - Mary Jean Harrold, Rajiv Gupta and Mary Lou Soffa, "A Methodology for Controlling the Size of a Test Suite," *ACM TOSEM*, 1993.
- - Leandro Sales Pinto, Saurabh Sinha and Alessandro Orso, "Understanding Myths and Realities of Test-Suite Evolution," FSE 2012.
- - Gregg Rothermel, Mary Jean Harrold, Jeffery Ostrin and Christie Hong, "An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites," ICSM 1998.
- - Shin Yoo and Mark Harman, "Regression Testing Minimization, Selection and Prioritization: A Survey," *Software Testing, Verification and Reliability*, 2012.
- - Negar Koochakzadeh and Vahid Garousi, "A Tester-Assisted Methodology for Test Redundancy Detection," *Advances in Software Engineering*, 2010.
- - Erik Kuefler, "Unit Testing," chapter 12 of Titus Winters, Tom Manshreck and Hyrum Wright (eds.), *Software Engineering at Google* (O'Reilly, 2020): tests that do not change when behavior does not.
- - Bertrand Meyer, *Object-Oriented Software Construction*, 2nd ed. (1997): Design by Contract and the non-redundancy principle.
- - Alexis King, "Parse, Don't Validate" (2019).
- ## License
- MIT. See [LICENSE](LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.