Skip to content
Back to search

dsh-benign-exit

Deterministically annotate benign non-zero bash/pwsh exit codes in DeepSeek Harness (grep 1 = no match, git diff --exit-code = differences, test/which/jq -e) so models stop over-investigating normal results.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/sunruize93-cmyk--dsh-benign-exit/badge.svg)](/listings/sunruize93-cmyk--dsh-benign-exit)

HTML

<a href="/listings/sunruize93-cmyk--dsh-benign-exit"><img src="/api/listings/sunruize93-cmyk--dsh-benign-exit/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Requires DeepSeek Harness `0.1.0-rc.5+` and Node `^22.19.0 || >=24.0.0`.

# from npm (once published)
dsh plugin --profile web add dsh-benign-exit

# or from source
git clone https://github.com/sunruize93-cmyk/dsh-benign-exit
cd dsh-benign-exit
pnpm install
dsh plugin --profile web add .

The `dsh.bundle` declaration in this package's `package.json` makes `dsh plugin add` register it as an active profile layer automatically (verified against `0.1.0-rc.5`). After adding it, restart the profile.

To verify it is active, either:

dsh --profile web --dump-config | grep benign-exit     # should show the layer

or run a no-match `grep` in the agent and check the result shows `(exit 1 = no matching lines — the command's documented meaning)` above the exit marker.

Headless / one-shot tasks

Works for `--profile headless` too:

dsh plugin --profile headless add dsh-benign-exit
dsh --profile headless "search for getLegacyConfig and report whether it exists"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 157 lines
  1. # dsh-benign-exit
  2. **English** | [简体中文](README.zh.md)
  3. Deterministically annotate benign non-zero bash/pwsh exit codes in DeepSeek Harness so models can stop **over-investigating normal results**.
  4. `grep` exits `1` when nothing matches — that is normal, not a failure. But DeepSeek Harness's bash tool tells the model:
  5. > "Check the `[exit code: N]` marker on every bash result; **investigate failures before moving on**."
  6. A model that runs `grep` and finds nothing sees `[exit code: 1]`, may read the standing instruction as "this failed, investigate", and can re-check a perfectly normal result — burning requests and tokens.
  7. This plugin makes the exit code **self-describing** at the layer the model actually sees: when the leading command is a known tool and the exit code is a documented benign outcome, the model-facing result is annotated so the model sees *why* the exit is normal.
  8. > **Scope of this claim.** The plugin provably changes what the model sees for benign outcomes (verified against the real harness machinery below). A live before/after experiment found a **pooled −20% billed input tokens** on an engineered 4-task set (see below) — but the per-run typical effect is small (~+3%), and the win concentrates in shell-heavy re-verification tasks. Read the full honest breakdown in the experiment section.
  9. ## How it works
  10. Two layers:
  11. 1. **Result layer (deterministic).** A `tools/post-execute` hook rewrites the model-facing content of `bash` / `pwsh` results, but only when the structured result confirms a real non-zero exit (no signal kill, no timeout, no abort). For example, a real `grep` run that found nothing renders as:
  12. ```
  13. (no output)
  14. (exit 1 = no matching lines — the command's documented meaning)
  15. [exit code: 1]
  16. ```
  17. The model sees the exit marked as expected — a change to what the model *sees*, independent of model mood or prompt wording.
  18. 2. **Prompt layer (reinforcement).** A small system-prompt section teaches the benign exit-code vocabulary, so the standing "investigate failures" guidance is read correctly. (The harness's own instruction is not removed; this is a corrective note, not a replacement.)
  19. ## What counts as benign
  20. The built-in table only covers **documented** normal outcomes:
  21. | Command | Exit | Meaning |
  22. |---|---|---|
  23. | `grep`, `egrep`, `fgrep`, `rg`, `ack` | 1 | no matching lines |
  24. | `diff`, `cmp`, `comm` | 1 | inputs differ |
  25. | `test`, `[` | 1 | condition is false |
  26. | `which`, `type` | 1 | not found (informational) |
  27. | `jq -e` | 1 | filter evaluated to false or null |
  28. | `git grep` | 1 | no matching lines |
  29. | `git diff --exit-code` / `--quiet` | 1 | differences exist |
  30. | `git diff-index --exit-code` | 1 | differences exist |
  31. The annotation is inserted as a line **above** the real `[exit code: N]` marker — the marker text itself is never rewritten, because the harness's `parseExitStatus` (used by the UI terminal card) requires the marker to stay the literal final line.
  32. ## Coverage and limits (read this before judging)
  33. **Conservative by design.** The plugin only annotates a *single, simple, foreground* command whose leading program is a known tool and whose exit code is a documented benign outcome. Deliberately NOT covered:
  34. - **Compound commands** (`grep -q p f && echo yes`, `grep x | head`, `if grep ...; then`, `$(...)`, subshells): the reported exit code is ambiguous there, so they pass through untouched.
  35. - **Background jobs**: `job_output` renders a different format (`[status: completed, exit code: 1]`) via a different tool; not annotated.
  36. - **Terminal channels** (`terminal_send` / `terminal_read`): a different marker format (`exited code=N`); not annotated.
  37. - **Explicit paths** (`./script.sh`): never treated as a known tool.
  38. - **Wrappers** (`sudo`, `env`, `nice`, `nohup`, `timeout`, `sh -c`, `command`, leading env-assignments) and **any redirection** (`>`, `<`, `2>&1`, heredocs): rejected outright — a wrapper or redirect failure settles on exit 1 and must not be attributed to the tool.
  39. - **Real failures** (`grep` exit 2, any unknown command, any exit not in the table): untouched. The built-in table is authoritative — user extra rules can never mask a real failure for a built-in command.
  40. - **Semantic limit:** for `test`/`[`/`which`/`type`/`diff`, exit 1 is the tool's *documented* meaning, but it can still be **task-relevant** (e.g. `test -f /critical/file` missing is the answer). The annotation wording therefore says "report it, don't re-investigate" rather than "this is fine" — the model is told to report the outcome, just not to treat the command as broken. If you need `test`/`which` to never be annotated, set `extraRules` to shadow them is not supported for built-ins; instead disable `annotate` for those cases (or open an issue).
  41. - **Windows pwsh kills**: on Windows a killed pwsh process settles as bare exit 1 with no signal marker, so on Windows hosts the plugin does **not** annotate pwsh results at all (a real kill must not be masked). On macOS/Linux, pwsh kills report a signal and are skipped by the structured-exit gate. pwsh is hooked by design but has no e2e coverage in this repo (no PowerShell here).
  42. Also note: DeepSeek Harness ships a `grep` *tool* (`@deepseek-ai/dsh-tool-fs-search`, in the base bundle) that already steers file-content search away from shell `grep` and returns "0 matches" as a success. The remaining exposure this plugin addresses is therefore mostly **non-file-search greps** (`git grep`, greps over command output), other benign exits (`git diff --exit-code`, `test -f`, `which`, `diff`, `rg`, `jq -e`), and the model's *standing instruction* to investigate every non-zero exit.
  43. ## Install
  44. Requires DeepSeek Harness `0.1.0-rc.5+` and Node `^22.19.0 || >=24.0.0`.
  45. ```bash
  46. # from npm (once published)
  47. dsh plugin --profile web add dsh-benign-exit
  48. # or from source
  49. git clone https://github.com/sunruize93-cmyk/dsh-benign-exit
  50. cd dsh-benign-exit
  51. pnpm install
  52. dsh plugin --profile web add .
  53. ```
  54. The `dsh.bundle` declaration in this package's `package.json` makes `dsh plugin add` register it as an active profile layer automatically (verified against `0.1.0-rc.5`). After adding it, restart the profile.
  55. To verify it is active, either:
  56. ```bash
  57. dsh --profile web --dump-config | grep benign-exit # should show the layer
  58. ```
  59. or run a no-match `grep` in the agent and check the result shows
  60. `(exit 1 = no matching lines — the command's documented meaning)` above the exit marker.
  61. ### Headless / one-shot tasks
  62. Works for `--profile headless` too:
  63. ```bash
  64. dsh plugin --profile headless add dsh-benign-exit
  65. dsh --profile headless "search for getLegacyConfig and report whether it exists"
  66. ```
  67. ## Configuration
  68. | Key | Type | Default | Meaning |
  69. |---|---|---|---|
  70. | `annotate` | boolean | `true` | Rewrite benign exit markers in tool results |
  71. | `promptSection` | boolean | `true` | Add the benign-exit guidance system-prompt section |
  72. | `extraRules` | `{command, exitCodes, reason}[]` | `[]` | Annotate YOUR tools' benign exits (only for commands not in the built-in table) |
  73. Example:
  74. ```yaml
  75. # profile cordis.patch.yml
  76. - insert:
  77. - id: dsh-benign-exit
  78. name: 'dsh-benign-exit'
  79. config:
  80. annotate: true
  81. promptSection: true
  82. extraRules:
  83. - command: my-checker
  84. exitCodes: [3]
  85. reason: 'resource already provisioned'
  86. ```
  87. ## Verification
  88. > **What is proven / what is not.** The plugin deterministically changes the model-facing text, verified against the real harness machinery below. A live before/after experiment (see the section above) measured token/time deltas on an engineered 4-task set; read its honest caveats before quoting any number. What is *mechanically* proven:
  89. - **36 unit tests** of the parser/annotator (edge cases, compound-command rejection, masking-prevention, structured-exit gating, wrapper forms, last-marker anchoring).
  90. - **10 integration tests** inside the real `ToolRuntime` dispatch pipeline (`tools/post-execute` waterfall): benign markers get annotated; real errors (exit 2), unknown commands, non-bash tools, **exit-0 runs whose output merely contains `[exit code: 1]`**, **signal-killed runs**, and **timed-out runs** all pass through untouched; downstream `additionalContexts` survive; the system-prompt section registers.
  91. - **6 end-to-end tests with the REAL bash tool** (spawning real `grep`/`git` processes): a real no-match `grep` produces model-facing content `(exit 1 = no matching lines — the command's documented meaning)\n[exit code: 1]`; real `grep` error (exit 2) and matching `grep` (exit 0) pass through untouched; `git diff --exit-code` with differences is annotated; and the harness's own `parseExitStatus` still reads the annotated result as exit **1** (UI exit pill intact).
  92. Run them yourself:
  93. ```bash
  94. cd dsh-benign-exit && npm test # unit tests
  95. # integration + real-bash tests run inside a DeepSeek Harness source checkout:
  96. # npx vitest run <path-to>/verification/dsh-benign-exit.integration.spec.ts
  97. # npx vitest run <path-to>/verification/dsh-benign-exit.real-bash.spec.ts
  98. # (set DSH_BENIGN_EXIT_PATH if the plugin lives elsewhere)
  99. ```
  100. ### Real before/after experiment (2026-08-17)
  101. We ran a real before/after comparison on `deepseek-v4-flash` (24 headless runs: 4 tasks × 3 rounds × with/without the plugin). Full raw data + report: [`results/experiment-20260817/`](results/experiment-20260817/report.md).
  102. Pooled over the 24 runs: **≈13% fewer steps, ≈20% fewer billed input tokens, ≈22% lower estimated cost** — but this is a token-weighted total dominated by one task. The **per-run typical effect is small (~+3%)**; pooled **excluding the dominant task**: −2%. The one solid signal is a "repeatedly re-verify a huge search surface via shell" task (all 3 after-rounds beat all 3 before-rounds; −38% billed input). The task set was deliberately negative-result-shaped, so these numbers are an **upper bound** for representative bash work, not a lower bound. Answers were equivalent with/without the plugin on every task. The harness steers file-content search to its built-in `grep` tool (out of scope), so bash-averse workflows get ~zero benefit.
  103. Third-party field data (Bohu, 2026-08) measured dsh at 61 vs 32 requests, 10:38 vs 4:55 wall time, and **2.9M vs 548K input tokens**, and $0.17 vs $0.05 cost on the same task vs `pi`. That benchmark compares structurally different harnesses (its attribution to this one mechanism is the author's hypothesis, not an isolated experiment), and we have not reproduced those numbers. The over-investigation loop is the mechanism this plugin targets; it does not claim to close the whole gap.
  104. ## Compatibility
  105. - Tested against DeepSeek Harness `0.1.0-rc.5` (`tool-bash`/`dsh-tools`) as of 2026-08-17, cordis `4.0.1`, schemastery `3.18.1`. All verification in this README is pinned to this revision.
  106. - DeepSeek Harness is in developer preview ("THERE WILL BE COMPATIBILITY-BREAKING CHANGES"). Pin versions; this plugin locks to the injected `tools/post-execute` and `systemPrompt` seams, which are core and stable.
  107. ## License
  108. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.