Skip to content
Back to search

test-feedback-loop

Fast independent checks for low-effort AI implementation and repeatable repair verification.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/onoderakosen89--test-feedback-loop/badge.svg)](/listings/onoderakosen89--test-feedback-loop)

HTML

<a href="/listings/onoderakosen89--test-feedback-loop"><img src="/api/listings/onoderakosen89--test-feedback-loop/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 194 lines
  1. # Test Feedback Loop
  2. **Independent checks for fast, token-efficient implementation with smaller models.**
  3. Once planning, design, and coordination have established the task, give the implementation agent a narrow, repeatable failure and the expected behavior. A short Check loop in Plan–Do–Check–Act gives a smaller model or a low-effort turn a concrete change to make. Independently rerun the failing case and a normal control to verify the result.
  4. [日本語](README.ja.md) · [简体中文](README.zh-CN.md) · [Agent Skill](skills/test-feedback-loop/SKILL.md)
  5. ## Try the complete repair loop
  6. Use Node.js 24 or later. Both demos run without a model account or API key.
  7. ```sh
  8. git clone https://github.com/onoderakosen89/test-feedback-loop.git
  9. cd test-feedback-loop
  10. npm run --silent demo:repair
  11. ```
  12. The demo reproduces a failure, hands the assertion evidence to a scripted repair, and independently reruns the regression and its normal control. To see what happens when two repairs fail, run:
  13. ```sh
  14. npm run --silent demo:repair -- --show-stop
  15. ```
  16. This exits with code 1 and returns a `consultation` object with both attempts, the final difference, the latest evidence, and the premise to check. An inconclusive run exits with code 2. The caller can display or route the consultation.
  17. Run `npm run --silent demo:repair -- --chatgpt` to have Codex apply the fix at fixed Low using your ChatGPT sign-in. The demo uses a disposable checkout-free fixture. A follow-up appends only the new check result to the same Codex session. Reported usage includes cached input when Codex supplies it. The [runRepairLoop API](src/repair-loop.mjs) defaults to two repair attempts. Callers can set `maxRepairs` to a positive integer.
  18. To try the diagnosis path without model calls, run `npm run --silent demo:repair -- --show-diagnosis`. Pass a `diagnose` callback to enable this path in your integration. After two failed repairs by default, it receives the latest check, attempts, and diff with `effort: "high"`. Run it with read-only access. Return `{ action, cause, evidence, next }` with nonempty strings. Use `repair` when the evidence supports a concrete change within the current requirements, `unknown` when the cause is unclear, or `human_decision` when approval is needed. Only `repair` permits one more repair at `effort: "low"`, followed by the same independent check. The caller sets the model effort and provides read-only isolation.
  19. ## The workflow
  20. 1. **Reproduce.** Run a small check against the behavior you expect, with a normal case to check for regressions.
  21. 2. **Make one bounded change.** Give Codex or Claude Code the failed check, what you expected, and what happened. Low effort is a starting point for a repair this well scoped.
  22. 3. **Verify.** Rerun the same check and the normal case. If the result stays inconclusive or the next repair would be guesswork, keep the evidence and stop to review the cause.
  23. In one trial of a historical message-delivery bug, GPT-6.1 Terra and GPT-6.2 Sol both repaired the targeted failure and preserved the normal case at fixed Low and High. Each model ran once per setting.
  24. | Model | Low | High |
  25. | --- | --- | --- |
  26. | GPT-6.1 Terra | 45 s, 1,677 output tokens, 10/15 broader checks | 67 s, 2,694 output tokens, 10/15 |
  27. | GPT-6.2 Sol | 94 s, 3,137 output tokens, 11/15 broader checks | 141 s, 5,347 output tokens, 10/15 |
  28. **In this one repair, fixed Low took about 33% less time and used 38–41% fewer output tokens than fixed High.** Both models met the targeted check at both settings. See the [full results and limits](docs/implementation-evaluation.md) for input and cached tokens and the checks that stayed unresolved.
  29. ## Check your own project
  30. Run the check command directly on your own tests. To hand failures to a repair agent automatically, connect your caller to [runRepairLoop](src/repair-loop.mjs). Point the check at a trusted Node.js test file:
  31. ```sh
  32. npm run check:node -- --cwd /path/to/project --check-id regression --id before test/regression.test.mjs
  33. ```
  34. The output leads with the check result, followed by the failing assertion and its location if available. After a repair, use the same files and check ID, plus `--id after --resolves before`.
  35. | Exit code | Meaning |
  36. | --- | --- |
  37. | 0 | The check passed |
  38. | 1 | An assertion failure was confirmed |
  39. | 2 | The check was inconclusive or could not run |
  40. An inconclusive check prints a reason when available. For machine-readable output, run `npm run --silent check:node -- --json` with your test arguments. Pass `observation` to `RunContext` and keep `evidence` for the repair agent. The check command makes no model calls.
  41. For a Python project, use an interpreter with pytest installed:
  42. ```sh
  43. npm run check:pytest -- --cwd /path/to/project --python /path/to/venv/bin/python --check-id regression --id before tests/test_regression.py
  44. ```
  45. An assertion failure in the test body produces evidence. Collection and setup failures produce an unknown result.
  46. To run the checker from other projects, run `npm link` in this checkout. Then run `tfl-check --help` from your project. For a Python test, use `tfl-check --runner pytest --python /path/to/venv/bin/python tests/test_regression.py`. The link also installs the optional proxy command `test-feedback-loop`.
  47. ## Share an independent check
  48. Have a useful pytest, Vitest, TypeScript, Cargo, schema, or other check? Open a [check proposal](https://github.com/onoderakosen89/test-feedback-loop/issues/new?template=check-adapter.yml) with a small failing example, a passing recheck, and a case where the runner could not determine the result. Proposals are welcome before implementation. If you have a working adapter, open a pull request with those examples as tests. [Contribution guide](CONTRIBUTING.md)
  49. Bundled checkers support Node.js tests and pytest.
  50. ## Optional effort switching
  51. The check and repair workflow works with a fixed effort. For experiments with automatic effort switching, the repository includes a local HTTP controller and examples that call the native Codex and Claude Code interfaces. [Wire specification (Japanese)](SPEC.md)
  52. ### Run the switching demo
  53. Use Node.js 24 or later. The demo starts a mock upstream and the controller on loopback and closes both when it finishes. It needs no dependencies, API keys, or model calls.
  54. ```sh
  55. npm run demo
  56. ```
  57. ```text
  58. Initial call: low
  59. Verified failure: high
  60. Matching check passed: low
  61. ```
  62. The demo runs a real Node.js assertion against a temporary implementation, changes the implementation, and runs the same assertion again. The mock upstream reports the effort it received after each check.
  63. ### Use Codex without the HTTP proxy
  64. With Codex CLI installed, run `npm run demo:codex`. This starts Codex App Server with a loopback mock and verifies that three turns in one thread send `low → high → low` upstream. It uses the same checker and policy as the HTTP demo.
  65. To try with your existing ChatGPT sign-in, run `npm run demo:codex -- --chatgpt`. This makes three short model calls against your Codex plan in a temporary read-only workspace. Add `--model MODEL_ID` if your plan offers a different model. The example checks the account mode before starting the three turns. It creates its own App Server session. [Codex App Server](https://developers.openai.com/codex/app-server)
  66. ### Use the official Claude Code CLI
  67. With Claude Code installed and signed in through its own sign-in flow, run `npm run demo:claude -- --use-subscription`. It makes three short calls in one Claude Code session, setting `--effort` and resuming that session for each follow-up. It uses the same independent check and policy. Claude Code handles all authentication and model calls. [Claude Code CLI reference](https://code.claude.com/docs/en/cli-reference) · [Anthropic's authentication guidance](https://code.claude.com/docs/en/legal-and-compliance#authentication-and-credential-use)
  68. ### Desktop sessions
  69. In an open desktop conversation, you can run the project check and manually choose the next effort with the app's built-in control. The automatic examples above own their respective sessions. [Codex desktop model control](https://learn.chatgpt.com/docs/models?surface=app) · [Claude Desktop effort menu](https://code.claude.com/docs/en/desktop)
  70. ### How the controller switches
  71. 1. The caller starts a run at the configured **base effort** (default: `low`).
  72. 2. After an independent check finds a failure attributable to reasoning or generated output, the caller attaches that result to the next request. The controller uses the configured **failure effort** (default: `high`) while a classified failure remains unresolved.
  73. 3. When the same check passes, the caller references the original failure. Effort returns to the base level after all outstanding failures have been resolved.
  74. Permission, approval, quota, and transport failures do not raise effort. The controller treats requests without valid context as passthrough. It leaves the contents of model messages, tools, and unrelated request fields intact.
  75. ```mermaid
  76. flowchart LR
  77. Base["Base effort"] --> Check["Run an independent check"]
  78. Check -->|Verified failure| Retry["Failure effort"]
  79. Retry --> Recheck["Run the same check"]
  80. Recheck -->|Pass| Base
  81. Check -->|Inconclusive| Keep["Keep current effort"]
  82. Recheck -->|Still unresolved| Retry
  83. ```
  84. ### Connect a caller
  85. ```sh
  86. npm start -- --upstream http://127.0.0.1:8317 --model YOUR_MODEL_ID --port 8318
  87. ```
  88. Use `http://127.0.0.1:8318/v1` as the caller's API base URL. The upstream must be an HTTP server listening on IPv4 loopback. The controller supports OpenAI-compatible `/v1/chat/completions` and `/v1/responses` requests. The upstream must support the requested effort values.
  89. Set the two levels at startup:
  90. | Sequence | Options |
  91. | --- | --- |
  92. | High → Low → High | `--base-effort high --failure-effort low` |
  93. | Low → XHigh → Low | `--base-effort low --failure-effort xhigh` |
  94. Accepted values are `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max` (case-insensitive). Choose levels your upstream model supports. The options change the values sent at each state while keeping the same switching conditions.
  95. Use `RunContext` from `src/adapter.mjs` to add the `X-Effort-Context` header:
  96. ```js
  97. import { RunContext } from './src/adapter.mjs';
  98. const run = new RunContext();
  99. const initialHeaders = run.next();
  100. const repairHeaders = run.next([
  101. { id: 'failure-1', check_id: 'regression', outcome: 'failure', category: 'reasoning' },
  102. ]);
  103. const recheckHeaders = run.next([
  104. { id: 'pass-1', check_id: 'regression', outcome: 'success', resolves: 'failure-1' },
  105. ]);
  106. ```
  107. Send these on successive logical calls, alongside your normal request headers. Link an observation to the call that follows the check. On an HTTP retry, reuse that call's exact headers and body. Use a separate run for unrelated work and keep calls within each run in order. Give capability probes and other auxiliary requests their own context or send them without the control header.
  108. ### Bundled Node.js check
  109. Use `runNodeTestCheck` after an edit to run an explicit set of trusted test files:
  110. ```js
  111. import { runNodeTestCheck } from './src/checkers/node-test.mjs';
  112. const check = { files: ['./test/regression.test.mjs'], checkId: 'regression' };
  113. const observation = await runNodeTestCheck({ ...check, id: 'check-1' });
  114. const nextHeaders = run.next([observation]);
  115. ```
  116. For a recheck, use the same files and check ID, a new observation ID, and `resolves` set to the original failure ID. The adapter classifies assertion failures from structured runner events. An incomplete run or an unclassified error does not change the effort state.
  117. ## License
  118. [MIT](LICENSE)
  119. ## Scope
  120. - Loopback deployment for trusted callers. The context header carries control data and is not an authentication mechanism.
  121. - In-memory decisions. HTTP retries still reach the upstream, and a restart starts a new run.
  122. - The caller runs the checks and owns task success, escalation, or approval. The controller changes reasoning effort and logs its decisions.
  123. - The HTTP demo and integration suite use mock models. See the [implementation evaluation](docs/implementation-evaluation.md) for a separate model trial.
  124. ```sh
  125. npm test
  126. npm run test:integration
  127. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.