Skip to content
Back to search

claw-compactor

SkillWorks with Claude CodeGitHub

Claw Compactor — 6-layer token compression skill for OpenClaw agents. Cuts workspace token spend by 50–97% using deterministic rule-engines plus Engram: a real-time, LLM-driven Observational Memory system. Run at session start for automatic savings reporting.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked yesterday
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/open-compress--claw-compactor/badge.svg)](/listings/open-compress--claw-compactor)

HTML

<a href="/listings/open-compress--claw-compactor"><img src="/api/listings/open-compress--claw-compactor/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install from PyPI

pip install claw-compactor

Or clone from source

git clone https://github.com/open-compress/claw-compactor.git
cd claw-compactor
pip install -e .

Run

# Benchmark your workspace (non-destructive)
claw-compactor benchmark /path/to/workspace

# Full compression pipeline
claw-compactor compress /path/to/workspace

**Requirements:** Python 3.9+. Optional: `pip install claw-compactor[accurate]` for exact token counts via tiktoken.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 420 lines
  1. ---
  2. name: claw-compactor
  3. version: "1.0.0"
  4. description: >
  5. Claw Compactor — 6-layer token compression skill for OpenClaw agents.
  6. Cuts workspace token spend by 50–97% using deterministic rule-engines plus
  7. Engram: a real-time, LLM-driven Observational Memory system.
  8. Run at session start for automatic savings reporting.
  9. triggers:
  10. - "compress memory"
  11. - "compress workspace"
  12. - "save tokens"
  13. - "token savings"
  14. - "compress context"
  15. - "run engram"
  16. - "engram observe"
  17. - "engram reflect"
  18. - "memory compression"
  19. - "benchmark compression"
  20. ---
  21. # Claw Compactor — OpenClaw Skill Reference
  22. ## Overview
  23. Claw Compactor reduces token usage across the full OpenClaw workspace using
  24. 6 compression layers:
  25. | Layer | Name | Cost | Notes |
  26. |-------|------|------|-------|
  27. | 1 | Rule Engine | Free | Dedup, strip filler, merge sections |
  28. | 2 | Dictionary Encoding | Free | Auto-codebook, `$XX` substitution |
  29. | 3 | Observation Compression | Free | Session JSONL → structured summaries |
  30. | 4 | RLE Patterns | Free | Path/IP/enum shorthand |
  31. | 5 | Compressed Context Protocol | Free | Format abbreviations |
  32. | **6** | **Engram** | LLM API | Real-time Observational Memory |
  33. **Skill location:** `skills/claw-compactor/`
  34. **Entry point:** `scripts/mem_compress.py`
  35. **Engram CLI:** `scripts/engram_cli.py`
  36. ---
  37. ## Auto Mode (Recommended — Run at Session Start)
  38. ```bash
  39. python3 skills/claw-compactor/scripts/mem_compress.py <workspace> auto
  40. ```
  41. Automatically compresses all workspace files, tracks token counts between
  42. runs, and reports savings. Run this at the start of every session.
  43. ---
  44. ## Core Commands
  45. ### Full Pipeline (All Layers)
  46. ```bash
  47. python3 scripts/mem_compress.py <workspace> full
  48. ```
  49. Runs all 5 deterministic layers in optimal order. Typical: 50%+ combined savings.
  50. ### Benchmark (Non-Destructive)
  51. ```bash
  52. python3 scripts/mem_compress.py <workspace> benchmark
  53. # JSON output:
  54. python3 scripts/mem_compress.py <workspace> benchmark --json
  55. ```
  56. Dry-run report showing potential savings without writing any files.
  57. ### Individual Layers
  58. ```bash
  59. # Layer 1: Rule-based compression
  60. python3 scripts/mem_compress.py <workspace> compress
  61. # Layer 2: Dictionary encoding
  62. python3 scripts/mem_compress.py <workspace> dict
  63. # Layer 3: Observation compression (session JSONL → summaries)
  64. python3 scripts/mem_compress.py <workspace> observe
  65. # Layer 4: RLE pattern encoding (runs inside `compress`)
  66. # Layer 5: Tokenizer optimization
  67. python3 scripts/mem_compress.py <workspace> optimize
  68. # Tiered summaries (L0/L1/L2)
  69. python3 scripts/mem_compress.py <workspace> tiers
  70. # Cross-file deduplication
  71. python3 scripts/mem_compress.py <workspace> dedup
  72. # Token count report
  73. python3 scripts/mem_compress.py <workspace> estimate
  74. # Workspace health check
  75. python3 scripts/mem_compress.py <workspace> audit
  76. ```
  77. ### Global Options
  78. ```
  79. --json Machine-readable JSON output
  80. --dry-run Preview without writing files
  81. --since DATE Filter sessions by date (YYYY-MM-DD)
  82. --auto-merge Auto-merge duplicates (dedup command)
  83. ```
  84. ---
  85. ## Engram — Layer 6: Real-Time Observational Memory
  86. Engram is the flagship layer. It operates as a live engine alongside conversations,
  87. automatically compressing messages into structured, priority-annotated knowledge.
  88. ### Prerequisites
  89. Configure via `engram.yaml` (recommended) or environment variables:
  90. ```yaml
  91. # engram.yaml — place in claw-compactor root
  92. llm:
  93. provider: openai-compatible
  94. base_url: http://localhost:8403
  95. model: claude-code/sonnet
  96. max_tokens: 4096
  97. threads:
  98. default:
  99. observer_threshold: 30000 # pending tokens before Observer fires
  100. reflector_threshold: 40000 # observation tokens before Reflector fires
  101. concurrency:
  102. max_workers: 4 # parallel thread workers
  103. ```
  104. ```bash
  105. # Alternative: environment variables
  106. export ANTHROPIC_API_KEY=sk-ant-... # Preferred
  107. # or
  108. export OPENAI_API_KEY=sk-... # OpenAI-compatible fallback
  109. export OPENAI_BASE_URL=https://... # Optional: custom endpoint (local LLM, etc.)
  110. ```
  111. ### Engram Auto-Mode (Recommended for Production)
  112. Auto-detects all active threads and processes them concurrently (4 workers):
  113. ```bash
  114. # Single run — auto-detects all threads
  115. python3 scripts/engram_auto.py --workspace ~/.openclaw/workspace
  116. # Via shell wrapper
  117. bash scripts/engram-auto.sh
  118. # Via CLI
  119. python3 scripts/engram_cli.py <workspace> auto --config engram.yaml
  120. python3 scripts/engram_cli.py <workspace> status --thread openclaw-main
  121. python3 scripts/engram_cli.py <workspace> observe --thread openclaw-main
  122. python3 scripts/engram_cli.py <workspace> reflect --thread openclaw-main
  123. ```
  124. **Retry:** LLM calls retry on 429/5xx with exponential backoff (2s→4s→8s, max 3 attempts).
  125. No retry on 400/401/403 (fail fast on config errors).
  126. ### Engram via Unified Entry Point
  127. ```bash
  128. # Check all thread statuses
  129. python3 scripts/mem_compress.py <workspace> engram status
  130. # Force Observer for a thread
  131. python3 scripts/mem_compress.py <workspace> engram observe --thread <thread-id>
  132. # Force Reflector for a thread
  133. python3 scripts/mem_compress.py <workspace> engram reflect --thread <thread-id>
  134. # Print injectable context
  135. python3 scripts/mem_compress.py <workspace> engram context --thread <thread-id>
  136. ```
  137. ### Engram via Dedicated CLI
  138. ```bash
  139. # Status: all threads
  140. python3 scripts/engram_cli.py <workspace> status
  141. # Status: single thread
  142. python3 scripts/engram_cli.py <workspace> status --thread <thread-id>
  143. # Force observe
  144. python3 scripts/engram_cli.py <workspace> observe --thread <thread-id>
  145. # Force reflect
  146. python3 scripts/engram_cli.py <workspace> reflect --thread <thread-id>
  147. # Import conversation from file (JSON array or JSONL)
  148. python3 scripts/engram_cli.py <workspace> ingest \
  149. --thread <thread-id> --input /path/to/conversation.jsonl
  150. # Get injectable context string (ready for system prompt)
  151. python3 scripts/engram_cli.py <workspace> context --thread <thread-id>
  152. # JSON output for any command
  153. python3 scripts/engram_cli.py <workspace> status --json
  154. python3 scripts/engram_cli.py <workspace> context --thread <id> --json
  155. ```
  156. ### Engram Daemon Mode (Real-Time Streaming)
  157. ```bash
  158. # Start daemon, pipe JSONL messages via stdin
  159. python3 scripts/engram_cli.py <workspace> daemon --thread <thread-id>
  160. # Pipe a message:
  161. echo '{"role":"user","content":"Hello!","timestamp":"12:00"}' | \
  162. python3 scripts/engram_cli.py <workspace> daemon --thread <thread-id>
  163. # Control commands (send as JSONL):
  164. echo '{"__cmd":"observe"}' # force observe now
  165. echo '{"__cmd":"reflect"}' # force reflect now
  166. echo '{"__cmd":"status"}' # print thread status JSON
  167. echo '{"__cmd":"quit"}' # exit daemon
  168. # Quiet mode (suppress startup messages on stderr)
  169. python3 scripts/engram_cli.py <workspace> daemon --thread <id> --quiet
  170. ```
  171. ### Engram Python API
  172. ```python
  173. from scripts.lib.engram import EngramEngine
  174. engine = EngramEngine(
  175. workspace_path="/path/to/workspace",
  176. observer_threshold=30_000, # tokens before auto-observe
  177. reflector_threshold=40_000, # tokens before auto-reflect
  178. anthropic_api_key="sk-ant-...", # or set ANTHROPIC_API_KEY env
  179. )
  180. # Add a message — auto-triggers observe/reflect when thresholds exceeded
  181. status = engine.add_message("thread-id", role="user", content="Hello!")
  182. # Returns: {"observed": bool, "reflected": bool, "pending_tokens": int, ...}
  183. # Manual trigger regardless of thresholds
  184. obs_text = engine.observe("thread-id") # returns None if no pending msgs
  185. ref_text = engine.reflect("thread-id") # returns None if no observations
  186. # Get full context dict
  187. ctx = engine.get_context("thread-id")
  188. # Returns: {"thread_id", "observations", "reflection", "recent_messages", "stats", "meta"}
  189. # Build injectable system context string
  190. ctx_str = engine.build_system_context("thread-id")
  191. # Ready to prepend to system prompt
  192. ```
  193. ### Engram Configuration Variables
  194. | Variable | Default | Description |
  195. |----------|---------|-------------|
  196. | `ANTHROPIC_API_KEY` | — | Anthropic API key (preferred) |
  197. | `OPENAI_API_KEY` | — | OpenAI-compatible API key |
  198. | `OPENAI_BASE_URL` | `https://api.openai.com` | Custom endpoint for local LLMs |
  199. | `OM_OBSERVER_THRESHOLD` | `30000` | Pending tokens before auto-observe |
  200. | `OM_REFLECTOR_THRESHOLD` | `40000` | Observation tokens before auto-reflect |
  201. | `OM_MODEL` | `claude-opus-4-5` | LLM model override |
  202. ### Threshold Tuning Quick Reference
  203. Each Observer call ≈ 2K output tokens (Sonnet). Daily volume at default 30K threshold:
  204. | Channel | Daily Tokens | @30K threshold | @10K threshold |
  205. |---------|-------------|----------------|----------------|
  206. | #aimm | ~149K | ~5×/day | ~15×/day |
  207. | openclaw-main | ~138K | ~4.5×/day | ~14×/day |
  208. | #open-compress | ~68K | ~2.3×/day | ~7×/day |
  209. | #general | ~62K | ~2×/day | ~6×/day |
  210. | subagent | ~43K | ~1.4×/day | ~4×/day |
  211. | cron | ~9K | ~0.3×/day | ~1×/day |
  212. | **Total** | **~470K/day** | **~16×/day (~32K output tokens)** | **~47×/day (~94K output tokens)** |
  213. Start at `observer_threshold: 30000`. Tune down for fresher context; tune up to reduce cost.
  214. ### Engram Benchmark Summary
  215. | Strategy | Token Savings | ROUGE-L | IR-F1 | Latency | LLM Calls |
  216. |----------|--------------|---------|-------|---------|-----------|
  217. | **Engram (L6)** | **87.5%** | 0.038 | 0.414 | ~35s | 2 |
  218. | RuleCompressor (L1–5) | 9.0% | 0.923 | 0.958 | ~6ms | 0 |
  219. | RandomDrop | 21.5% | 0.852 | 0.911 | ~0ms | 0 |
  220. - Engram low ROUGE-L = semantic restructuring, not verbatim copy — intent is preserved
  221. - Use RuleCompressor for instant prompt compression; Engram for long-term memory
  222. - Full results → `benchmark/RESULTS.md`
  223. ### Observation Format
  224. Engram produces structured, bilingual (EN/中文) priority-annotated logs:
  225. ```
  226. Date: 2026-03-05
  227. - 🔴 12:10 User building OpenCompress; deadline one week / 用户在构建 OpenCompress,deadline 一周内
  228. - 🔴 12:10 Using ModernBERT-large / 使用 ModernBERT-large
  229. - 🟡 12:12 Discussed annotation strategy / 讨论了标注策略
  230. - 🟡 12:30 Deployment pipeline discussion on M3 Ultra
  231. - 🟢 12:45 User prefers concise replies
  232. ```
  233. - 🔴 **Critical** — goals, deadlines, blockers, key decisions (never dropped)
  234. - 🟡 **Important** — technical details, ongoing work, preferences
  235. - 🟢 **Useful** — background, mentions, soft context
  236. ### Memory Storage Layout
  237. ```
  238. memory/engram/{thread_id}/
  239. ├── pending.jsonl # Unobserved message buffer (auto-cleared after observe)
  240. ├── observations.md # Observer output — append-only structured log
  241. ├── reflections.md # Reflector output — compressed long-term memory (overwrites)
  242. └── meta.json # Timestamps and token counts
  243. ```
  244. ---
  245. ## Integration with OpenClaw Memory System
  246. ### System Prompt Injection
  247. Inject Engram context at the start of each session:
  248. ```python
  249. from scripts.lib.engram import EngramEngine
  250. engine = EngramEngine(workspace_path)
  251. ctx_str = engine.build_system_context("my-session")
  252. if ctx_str:
  253. system_prompt = ctx_str + "\n\n" + base_system_prompt
  254. ```
  255. The `build_system_context()` output structure:
  256. ```
  257. ## Long-Term Memory (Reflections)
  258. <Reflector output — long-term compressed context>
  259. ## Recent Observations
  260. <Last 200 lines of Observer output>
  261. <!-- engram_tokens: 1234 -->
  262. ```
  263. ### Combining Engram with Deterministic Layers
  264. After an Engram session, run the deterministic pipeline on the output files:
  265. ```bash
  266. # Engram produces observations.md and reflections.md
  267. # Then apply deterministic compression to further reduce those:
  268. python3 scripts/mem_compress.py <workspace> full
  269. ```
  270. ### Recommended Workflow for Long-Running Agent Sessions
  271. 1. **Session start:** inject `build_system_context()` into system prompt
  272. 2. **Each message:** call `engine.add_message()` — auto-triggers observe/reflect
  273. 3. **Session end / weekly cron:** run `full` pipeline on workspace
  274. 4. **Multi-session continuity:** context persists in `memory/engram/{thread}/`
  275. ---
  276. ## OpenClaw Skill Installation
  277. To install as an OpenClaw skill, ensure the skill directory is available at:
  278. ```
  279. ~/.openclaw/workspace/skills/claw-compactor/
  280. ```
  281. or configure the path in your OpenClaw skill registry.
  282. SKILL.md is read by the OpenClaw agent dispatcher. The `description` and
  283. `triggers` fields above control when this skill is automatically activated.
  284. ---
  285. ## Heartbeat / Cron Automation
  286. ```markdown
  287. ## Memory Maintenance (weekly)
  288. - python3 skills/claw-compactor/scripts/mem_compress.py <workspace> benchmark
  289. - If savings > 5%: run full pipeline
  290. - If pending Engram messages: run engram observe --thread <id>
  291. ```
  292. Cron (Sunday 3am):
  293. ```bash
  294. 0 3 * * 0 cd /path/to/skills/claw-compactor && \
  295. python3 scripts/mem_compress.py /path/to/workspace full
  296. ```
  297. ---
  298. ## Output Artifacts Reference
  299. | Artifact | Location | Description |
  300. |----------|----------|-------------|
  301. | Dictionary codebook | `memory/.codebook.json` | Must travel with memory files |
  302. | Observed session log | `memory/.observed-sessions.json` | Tracks processed transcripts |
  303. | Layer 3 summaries | `memory/observations/` | Observation compression output |
  304. | Engram observations | `memory/engram/{thread}/observations.md` | Live Observer log |
  305. | Engram reflections | `memory/engram/{thread}/reflections.md` | Distilled long-term memory |
  306. | Level 0 summary | `memory/MEMORY-L0.md` | ~200 token ultra-compressed summary |
  307. | Level 1 summary | `memory/MEMORY-L1.md` | ~500 token compressed summary |
  308. ---
  309. ## Troubleshooting
  310. | Problem | Solution |
  311. |---------|----------|
  312. | `FileNotFoundError` on workspace | Point path to workspace root containing `memory/` |
  313. | Dictionary decompression fails | Check `memory/.codebook.json` is valid JSON |
  314. | Zero savings on `benchmark` | Workspace already optimized |
  315. | `observe` finds no transcripts | Check `sessions/` for `.jsonl` files |
  316. | Engram: "no API key configured" | Set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` |
  317. | Engram Observer returns `None` | No pending messages for that thread |
  318. | Token counts seem wrong | Install tiktoken: `pip3 install tiktoken` |
README.md · README · 422 lines
  1. <!--
  2. <script type="application/ld+json">
  3. {
  4. "@context": "https://schema.org",
  5. "@type": "SoftwareApplication",
  6. "name": "Claw Compactor",
  7. "description": "14-stage Fusion Pipeline for LLM token compression with reversible compression, AST-aware code analysis, and intelligent content routing",
  8. "applicationCategory": "DeveloperApplication",
  9. "operatingSystem": "Cross-platform",
  10. "softwareVersion": "7.0.0",
  11. "license": "https://opensource.org/licenses/MIT",
  12. "url": "https://github.com/open-compress/claw-compactor",
  13. "downloadUrl": "https://github.com/open-compress/claw-compactor",
  14. "author": {
  15. "@type": "Organization",
  16. "name": "OpenClaw",
  17. "url": "https://openclaw.ai"
  18. },
  19. "offers": {
  20. "@type": "Offer",
  21. "price": "0",
  22. "priceCurrency": "USD"
  23. },
  24. "keywords": "token compression, LLM, AI agent, fusion pipeline, reversible compression, AST code analysis, context window optimization"
  25. }
  26. </script>
  27. -->
  28. <div align="center">
  29. # Claw Compactor
  30. ### 14-Stage Fusion Pipeline for LLM Token Compression
  31. ![Claw Compactor Banner](assets/banner.png)
  32. [![CI](https://github.com/open-compress/claw-compactor/actions/workflows/ci.yml/badge.svg)](https://github.com/open-compress/claw-compactor/actions)
  33. [![codecov](https://codecov.io/gh/open-compress/claw-compactor/graph/badge.svg)](https://codecov.io/gh/open-compress/claw-compactor)
  34. [![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://python.org)
  35. [![License](https://img.shields.io/badge/license-MIT-purple)](LICENSE)
  36. [![PyPI](https://img.shields.io/pypi/v/claw-compactor?color=blue&label=PyPI)](https://pypi.org/project/claw-compactor/)
  37. [![Downloads](https://img.shields.io/pypi/dm/claw-compactor?color=green&label=downloads)](https://pypi.org/project/claw-compactor/)
  38. [![Stars](https://img.shields.io/github/stars/open-compress/claw-compactor?style=social)](https://github.com/open-compress/claw-compactor)
  39. **15–82% compression depending on content &middot; Zero LLM inference cost &middot; Reversible &middot; 1600+ tests**
  40. [Documentation](https://open-compress.github.io/claw-compactor) &middot; [Architecture](ARCHITECTURE.md) &middot; [Benchmarks](#benchmarks) &middot; [Quick Start](#quick-start) &middot; [API](#api)
  41. </div>
  42. ---
  43. ## What is Claw Compactor?
  44. Claw Compactor is an open-source **LLM token compression engine** built around a 14-stage **Fusion Pipeline**. Each stage is a specialized compressor — from AST-aware code analysis to JSON statistical sampling to simhash-based deduplication — chained through an immutable data flow architecture where each stage's output feeds the next.
  45. ### Demo
  46. ```
  47. $ claw-compactor benchmark ./my-workspace
  48. Claw Compactor v7.0 — Fusion Pipeline Benchmark
  49. ─────────────────────────────────────────────────
  50. Scanning workspace... 47 files, 234,891 tokens
  51. Stage Results:
  52. ┌──────────────────┬──────────┬───────────┬──────────┐
  53. │ Stage │ Applied │ Reduction │ Time │
  54. ├──────────────────┼──────────┼───────────┼──────────┤
  55. │ Cortex │ 47/47 │ — │ 12ms │
  56. │ Photon │ 3/47 │ 2.1% │ 4ms │
  57. │ RLE │ 41/47 │ 8.3% │ 6ms │
  58. │ SemanticDedup │ 47/47 │ 12.7% │ 18ms │
  59. │ Ionizer │ 8/47 │ 71.2% │ 9ms │
  60. │ Neurosyntax │ 23/47 │ 18.4% │ 31ms │
  61. │ TokenOpt │ 47/47 │ 4.1% │ 3ms │
  62. │ Abbrev │ 12/47 │ 6.8% │ 5ms │
  63. └──────────────────┴──────────┴───────────┴──────────┘
  64. Summary:
  65. Before: 234,891 tokens ($2.35 at GPT-4 rates)
  66. After: 108,250 tokens ($1.08)
  67. Saved: 126,641 tokens (53.9%) — $1.27/run
  68. Time: 88ms total
  69. Estimated monthly savings at 100 runs/day: $3,810
  70. ```
  71. ---
  72. ## How It Compares
  73. | Feature | Claw Compactor | LLMLingua-2 | SelectiveContext | gzip + base64 |
  74. |:--------|:-:|:-:|:-:|:-:|
  75. | Compression rate | 15–82% | 30–70% | 10–40% | 60–80% |
  76. | ROUGE-L @ 0.3 | **0.653** | 0.346 | ~0.4 | N/A |
  77. | ROUGE-L @ 0.5 | **0.723** | 0.570 | ~0.6 | N/A |
  78. | LLM inference cost | **$0** | ~$0.02/call | **$0** | **$0** |
  79. | Latency | **<50ms** | ~300ms | ~200ms | <10ms |
  80. | Reversible | **Yes** | No | No | Yes (manual) |
  81. | Content-aware routing | **14 stages** | 1 (perplexity) | 1 (self-info) | None |
  82. | AST-aware code handling | **Yes** (tree-sitter) | No | No | No |
  83. | JSON schema sampling | **Yes** | No | No | No |
  84. | Log/diff/search stages | **Yes** | No | No | No |
  85. | Required dependencies | **0** | torch, transformers | torch | zlib |
  86. | LLM-readable output | **Yes** | Partial | Partial | **No** |
  87. **Why Claw Compactor wins:** LLMLingua-2 drops tokens by perplexity score — effective for natural language, but destroys code identifiers, JSON keys, and log patterns. Claw Compactor uses content-type-aware stages that understand the structure of what they're compressing.
  88. ---
  89. ```
  90. Input
  91. |
  92. v
  93. ┌─────────────────────────────────────────────────────────────────────────┐
  94. │ FUSION PIPELINE │
  95. │ │
  96. │ QuantumLock ─> Cortex ─> Photon ─> RLE ─> SemanticDedup ─> Ionizer │
  97. │ | | | | | | │
  98. │ KV-cache auto-detect base64 path simhash JSON │
  99. │ alignment 16 languages strip shorten dedup sampling │
  100. │ │
  101. │ ─> LogCrunch ─> SearchCrunch ─> DiffCrunch ─> StructuralCollapse │
  102. │ | | | | │
  103. │ log folding result dedup context fold import merge │
  104. │ │
  105. │ ─> Neurosyntax ─> Nexus ─> TokenOpt ─> Abbrev ─────────> Output │
  106. │ | | | | │
  107. │ AST compress ML token format NL shorten │
  108. │ (tree-sitter) classify optimize (text only) │
  109. │ │
  110. │ [ RewindStore ] ── hash-addressed LRU for reversible retrieval │
  111. └─────────────────────────────────────────────────────────────────────────┘
  112. ```
  113. Key design principles:
  114. - **Immutable data flow** — `FusionContext` is a frozen dataclass. Every stage produces a new `FusionResult`; nothing is mutated in-place.
  115. - **Gate-before-compress** — Each stage has `should_apply()` that inspects context type, language, and role before doing any work. Stages that don't apply are skipped at zero cost.
  116. - **Content-aware routing** — Cortex auto-detects content type (code, JSON, logs, diffs, search results) and language (Python, Go, Rust, TypeScript, etc.), then downstream stages make type-aware compression decisions.
  117. - **Reversible compression** — Ionizer stores originals in a hash-addressed `RewindStore`. The LLM can call a tool to retrieve any compressed section by its marker ID.
  118. ---
  119. ## Benchmarks
  120. ### Real-World Compression (FusionEngine v7 vs Legacy Regex)
  121. | Content Type | Legacy | FusionEngine | Improvement |
  122. |:-------------|-------:|-------------:|:-----------:|
  123. | Python source | 7.3% | **25.0%** | 3.4x |
  124. | JSON (100 items) | 12.6% | **81.9%** | 6.5x |
  125. | Build logs | 5.5% | **24.1%** | 4.4x |
  126. | Agent conversation | 5.7% | **31.0%** | 5.4x |
  127. | Git diff | 6.2% | **15.0%** | 2.4x |
  128. | Search results | 5.3% | **40.7%** | 7.7x |
  129. | **Weighted average** | **9.2%** | **36.3%** | **3.9x** |
  130. ### SWE-bench Real Tasks
  131. Tested on real SWE-bench instances with actual repository code:
  132. | Instance | Size | Compression |
  133. |:---------|-----:|------------:|
  134. | django__django-11620 | 4.5K | **14.5%** |
  135. | sympy__sympy-14396 | 5.5K | **19.1%** |
  136. | scikit-learn-25747 | 11.8K | **15.9%** |
  137. | scikit-learn-13554 | 73K | **11.8%** |
  138. | scikit-learn-25308 | 81K | **14.4%** |
  139. ### vs LLMLingua-2 (ROUGE-L Fidelity)
  140. | Compression Rate | Claw Compactor | LLMLingua-2 | Delta |
  141. |:-----------------|---------------:|------------:|------:|
  142. | 0.3 (aggressive) | **0.653** | 0.346 | +88.2% |
  143. | 0.5 (balanced) | **0.723** | 0.570 | +26.8% |
  144. Claw Compactor preserves more semantic content at the same compression ratio, with zero LLM inference cost.
  145. ---
  146. ## Quick Start
  147. ### Install from PyPI
  148. ```bash
  149. pip install claw-compactor
  150. ```
  151. ### Or clone from source
  152. ```bash
  153. git clone https://github.com/open-compress/claw-compactor.git
  154. cd claw-compactor
  155. pip install -e .
  156. ```
  157. ### Run
  158. ```bash
  159. # Benchmark your workspace (non-destructive)
  160. claw-compactor benchmark /path/to/workspace
  161. # Full compression pipeline
  162. claw-compactor compress /path/to/workspace
  163. ```
  164. **Requirements:** Python 3.9+. Optional: `pip install claw-compactor[accurate]` for exact token counts via tiktoken.
  165. ---
  166. ## API
  167. ### FusionEngine — Single Text
  168. ```python
  169. from scripts.lib.fusion.engine import FusionEngine
  170. engine = FusionEngine()
  171. result = engine.compress(
  172. text="def hello():\n # greeting function\n print('hello')",
  173. content_type="code", # or let Cortex auto-detect
  174. language="python", # optional hint
  175. )
  176. print(result["compressed"]) # compressed output
  177. print(result["stats"]) # per-stage timing + token counts
  178. print(result["markers"]) # Rewind markers for reversibility
  179. ```
  180. ### FusionEngine — Chat Messages
  181. ```python
  182. messages = [
  183. {"role": "system", "content": "You are a coding assistant..."},
  184. {"role": "user", "content": "Fix the auth bug in login.py"},
  185. {"role": "assistant", "content": "I found the issue. Here's the fix:\n```python\n..."},
  186. {"role": "tool", "content": '{"results": [{"file": "login.py", ...}, ...]}'},
  187. ]
  188. result = engine.compress_messages(messages)
  189. # Cross-message dedup runs first, then per-message pipeline
  190. print(result["stats"]["reduction_pct"]) # aggregate compression %
  191. print(result["per_message"]) # per-message breakdown
  192. ```
  193. ### Rewind — Reversible Retrieval
  194. ```python
  195. engine = FusionEngine(enable_rewind=True)
  196. result = engine.compress(large_json, content_type="json")
  197. # LLM sees compressed output with markers like [rewind:abc123...]
  198. # When the LLM needs the original, it calls the Rewind tool:
  199. original = engine.rewind_store.retrieve("abc123def456...")
  200. ```
  201. ### Custom Stage
  202. ```python
  203. from scripts.lib.fusion.base import FusionStage, FusionContext, FusionResult
  204. class MyStage(FusionStage):
  205. name = "my_compressor"
  206. order = 22 # runs between StructuralCollapse (20) and Neurosyntax (25)
  207. def should_apply(self, ctx: FusionContext) -> bool:
  208. return ctx.content_type == "log"
  209. def apply(self, ctx: FusionContext) -> FusionResult:
  210. compressed = my_compression_logic(ctx.content)
  211. return FusionResult(
  212. content=compressed,
  213. original_tokens=estimate_tokens(ctx.content),
  214. compressed_tokens=estimate_tokens(compressed),
  215. )
  216. # Add to pipeline
  217. pipeline = engine.pipeline.add(MyStage())
  218. ```
  219. ---
  220. ## The 14 Stages
  221. | # | Stage | Order | Purpose | Applies To |
  222. |:-:|:------|:-----:|:--------|:-----------|
  223. | 1 | **QuantumLock** | 3 | Isolates dynamic content in system prompts to maximize KV-cache hit rate | system messages |
  224. | 2 | **Cortex** | 5 | Auto-detects content type and programming language (16 languages) | untyped content |
  225. | 3 | **Photon** | 8 | Detects and compresses base64-encoded images | all |
  226. | 4 | **RLE** | 10 | Path shorthand (`$WS`), IP prefix compression, enum compaction | all |
  227. | 5 | **SemanticDedup** | 12 | SimHash fingerprint deduplication across content blocks | all |
  228. | 6 | **Ionizer** | 15 | JSON array statistical sampling with schema discovery + error preservation | json |
  229. | 7 | **LogCrunch** | 16 | Folds repeated log lines with occurrence counts | log |
  230. | 8 | **SearchCrunch** | 17 | Deduplicates search/grep results | search |
  231. | 9 | **DiffCrunch** | 18 | Folds unchanged context lines in git diffs | diff |
  232. | 10 | **StructuralCollapse** | 20 | Merges import blocks, collapses repeated assertions/patterns | code |
  233. | 11 | **Neurosyntax** | 25 | AST-aware code compression via tree-sitter (safe regex fallback). Never shortens identifiers. | code |
  234. | 12 | **Nexus** | 35 | ML token-level compression (stopword removal fallback without model) | text |
  235. | 13 | **TokenOpt** | 40 | Tokenizer format optimization — strips bold/italic markers, normalizes whitespace | all |
  236. | 14 | **Abbrev** | 45 | Natural language abbreviation. Only fires on text — never touches code, JSON, or structured data. | text |
  237. Each stage is independent and stateless. Stages communicate only through the immutable `FusionContext` that flows forward through the pipeline.
  238. ---
  239. ## Workspace Commands
  240. ```bash
  241. python3 scripts/mem_compress.py <workspace> <command> [options]
  242. ```
  243. | Command | Description |
  244. |:--------|:-----------|
  245. | `full` | Run complete compression pipeline |
  246. | `benchmark` | Dry-run compression report |
  247. | `compress` | Rule-based compression only |
  248. | `dict` | Dictionary encoding with auto-learned codebook |
  249. | `observe` | Session transcript JSONL to structured observations |
  250. | `tiers` | Generate L0/L1/L2 tiered summaries |
  251. | `dedup` | Cross-file duplicate detection |
  252. | `estimate` | Token count report |
  253. | `audit` | Workspace health check |
  254. | `optimize` | Tokenizer-level format optimization |
  255. | `auto` | Watch mode — compress on file changes |
  256. Options: `--json`, `--dry-run`, `--since YYYY-MM-DD`, `--quiet`
  257. ---
  258. ## Architecture
  259. See [ARCHITECTURE.md](ARCHITECTURE.md) for the full technical deep-dive:
  260. - Immutable data flow design
  261. - Stage execution model and gating
  262. - Rewind reversible compression protocol
  263. - Cross-message semantic deduplication
  264. - How to extend the pipeline
  265. ```
  266. 12,000+ lines Python · 1,600+ tests · 14 fusion stages · 0 external ML dependencies
  267. ```
  268. ---
  269. ## Installation
  270. ```bash
  271. # Clone
  272. git clone https://github.com/open-compress/claw-compactor.git
  273. cd claw-compactor
  274. # Optional: exact token counting
  275. pip install tiktoken
  276. # Optional: AST-aware code compression (Neurosyntax)
  277. pip install tree-sitter-language-pack
  278. # Development
  279. pip install -e ".[dev,accurate]"
  280. ```
  281. **Zero required dependencies.** tiktoken and tree-sitter are optional enhancements — the pipeline runs with built-in heuristic fallbacks for both.
  282. ---
  283. ## Who Uses This
  284. | Project | How |
  285. |:--------|:----|
  286. | [OpenClaw](https://openclaw.ai) | Built-in skill for all OpenClaw AI agents — compresses workspace context before every LLM call |
  287. | [OpenCompress](https://opencompress.ai) | Production compression engine powering the OpenCompress API |
  288. Using Claw Compactor? [Open a PR](https://github.com/open-compress/claw-compactor/pulls) to add yourself here.
  289. ---
  290. ## Project Stats
  291. | Metric | Value |
  292. |:-------|:------|
  293. | Tests | 1,600+ passed |
  294. | Python source | 12,000+ lines |
  295. | Fusion stages | 14 |
  296. | Languages detected | 16 |
  297. | Required dependencies | 0 |
  298. | Compression (code) | 15–25% |
  299. | Compression (JSON peak) | 81.9% |
  300. | ROUGE-L @ 0.3 rate | 0.653 |
  301. | License | MIT |
  302. ---
  303. ## Contributing
  304. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on:
  305. - Setting up the development environment
  306. - Adding new Fusion stages
  307. - Running the test suite
  308. - Submitting PRs
  309. ---
  310. ## Related
  311. - [OpenClaw](https://openclaw.ai) — AI agent platform
  312. - [ClawhubAI](https://clawhub.com) — Agent skills marketplace
  313. - [OpenClaw Discord](https://discord.com/invite/clawd) — Community
  314. - [OpenClaw Docs](https://docs.openclaw.ai) — Documentation
  315. - [Full Documentation](https://open-compress.github.io/claw-compactor) — GitHub Pages docs
  316. ---
  317. `token-compression` `llm-tools` `fusion-pipeline` `reversible-compression` `ast-code-analysis` `context-compression` `ai-agent` `openclaw` `python` `developer-tools`
  318. ## License
  319. [MIT](LICENSE)
proxy/server.mjs · Bundled script · 2,100 lines · 1 finding marked
  1. /**
  2. * Claude Code Proxy v0.5.0
  3. *
  4. * Central proxy for multiple OpenClaw instances.
  5. * Wraps claude CLI into OpenAI-compatible API with:
  6. * - Fair queuing (round-robin between sources)
  7. * - Rate limiting (95% of Max plan limits)
  8. * - Priority by model (opus=high, sonnet=normal, haiku=low)
  9. * - Authentication via Bearer token
  10. * - Per-source metrics & monitoring
  11. * - Process registry with zombie detection & reaper
  12. * - Retry with exponential backoff + jitter
  13. * - Stream heartbeat & execution timeout
  14. * - Graceful shutdown
  15. *
  16. * All OpenClaw instances point their claude-code provider to this proxy.
  17. * Requests are queued fairly and processed through local claude CLI,
  18. * using the Max subscription (flat monthly fee, no per-token cost).
  19. */
  20. import { createServer, request as httpRequest } from "node:http";
  21. import { request as httpsRequest } from "node:https";
  22. import { spawn } from "node:child_process";
  23. import { randomUUID } from "node:crypto";
  24. import { readFile } from "node:fs/promises";
  25. import { fileURLToPath } from "node:url";
  26. import { dirname, join } from "node:path";
  27. import { createFairQueue } from "./fair-queue.mjs";
  28. import { createProcessRegistry } from "./process-registry.mjs";
  29. import { createRetryPolicy } from "./retry.mjs";
  30. import { createEventLog } from "./event-log.mjs";
  31. import { createTokenTracker } from "./token-tracker.mjs";
  32. import { createMetricsStore } from "./metrics-store.mjs";
  33. import { createRateLimiter } from "./rate-limiter.mjs";
  34. import { createRedisClient } from "./redis-client.mjs";
  35. import { createSessionAffinity } from "./session-affinity.mjs";
  36. const __dirname = dirname(fileURLToPath(import.meta.url));
  37. // ============================================================
  38. // Configuration
  39. // ============================================================
  40. const PORT = parseInt(process.env.CLAUDE_PROXY_PORT || "8403", 10);
  41. const CLAUDE_BIN = process.env.CLAUDE_BIN || "claude";
  42. const AUTH_TOKEN = process.env.PROXY_AUTH_TOKEN || "local-proxy";
  43. // ============================================================
  44. // CLI Router Objects: smart routing with failover
  45. // WORKERS env var: JSON array of {name, bin, token} objects
  46. // Routing strategy:
  47. // - Default: all traffic to PRIMARY_WORKER (env, default "2")
  48. // - On rate limit: switch entirely to the other CLI router
  49. // - Every HEALTH_CHECK_MS: probe if cooled-down router is back
  50. // - When recovered: resume load-balancing (round-robin)
  51. // ============================================================
  52. const PRIMARY_WORKER = process.env.PRIMARY_WORKER || "2";
  53. const HEALTH_CHECK_MS = parseInt(process.env.HEALTH_CHECK_MS || "600000", 10); // 10 min
  54. // CLI agent mode: workers are full autonomous agents (all tools enabled, --dangerously-skip-permissions)
  55. // When true, ALL requests route through CLI workers; API direct path is bypassed.
  56. const USE_CLI_AGENTS = process.env.USE_CLI_AGENTS === "true";
  57. // Default worker pool: Mac native + Ubuntu VM (OrbStack)
  58. const DEFAULT_WORKERS = [
  59. { name: "1", bin: "/opt/homebrew/bin/claude" },
  60. { name: "2", bin: "/opt/homebrew/bin/claude-ubuntu" },
  61. ];
  62. let _workerPool = [];
  63. try {
  64. if (process.env.WORKERS) {
  65. _workerPool = JSON.parse(process.env.WORKERS);
  66. }
  67. } catch (err) {
  68. console.warn(`[CLIRouter] Failed to parse WORKERS env: ${err.message} — using defaults`);
  69. }
  70. if (_workerPool.length === 0) {
  71. _workerPool = DEFAULT_WORKERS;
  72. }
  73. // Assign default names if missing
  74. _workerPool.forEach((w, i) => { if (!w.name) w.name = String(i + 1); });
  75. // Worker health state
  76. const _workerHealth = new Map(); // name -> { limited: boolean, limitedAt: number }
  77. for (const w of _workerPool) {
  78. _workerHealth.set(w.name, { limited: false, limitedAt: 0 });
  79. }
  80. // Round-robin index for load-balancing mode
  81. let _rrIndex = 0;
  82. console.log(`[CLIRouter] Pool: ${_workerPool.map((w) => `${w.name}=${w.bin}`).join(" | ")}`);
  83. console.log(`[CLIRouter] Primary: ${PRIMARY_WORKER} | Health check: ${HEALTH_CHECK_MS / 1000}s`);
  84. // ============================================================
  85. // Fallback API: last-resort model when all CLI routers fail
  86. // Forwards as an OpenAI-compatible /v1/chat/completions request
  87. // ============================================================
  88. const FALLBACK_API = (() => {
  89. if (process.env.FALLBACK_API_URL) {
  90. return {
  91. baseUrl: process.env.FALLBACK_API_URL,
  92. apiKey: process.env.FALLBACK_API_KEY || "none",
  93. model: process.env.FALLBACK_MODEL || "default",
  94. name: process.env.FALLBACK_NAME || "fallback",
  95. };
  96. }
  97. // Default: MiniMax local endpoint
  98. return {
  99. baseUrl: "http://172.28.216.81:8080/v1",
  100. apiKey: "none",
  101. model: "MiniMax-M2.5-Q8_0-00001-of-00006.gguf",
  102. name: "minimax-local",
  103. };
  104. })();
  105. console.log(`[Fallback] ${FALLBACK_API.name} → ${FALLBACK_API.baseUrl} model=${FALLBACK_API.model}`);
  106. // ============================================================
  107. // Anthropic Direct API — for tool-enabled requests (bypass CLI)
  108. // When request includes `tools`, call Anthropic API directly so the
  109. // gateway receives tool_calls in OpenAI format and executes them.
  110. // CLI path remains for text-only requests (flat fee via Max sub).
  111. // ============================================================
  112. const ANTHROPIC_API_BASE = "https://api.anthropic.com";
  113. const ANTHROPIC_API_VERSION = "2023-06-01";
  114. const ANTHROPIC_MODEL_IDS = {
  115. sonnet: process.env.ANTHROPIC_SONNET_MODEL || "claude-sonnet-4-6",
  116. opus: process.env.ANTHROPIC_OPUS_MODEL || "claude-opus-4-6",
  117. haiku: process.env.ANTHROPIC_HAIKU_MODEL || "claude-haiku-4-5-20251001",
  118. };
  119. // Token pool for API direct round-robin.
  120. // Each token represents an independent OAuth credential with its own rate limits.
  121. // OAuth requires `anthropic-beta: oauth-2025-04-20` header to work with the raw API.
  122. const TOKEN_POOL = (() => {
  123. const tokens = [];
  124. // Worker tokens from WORKERS env (reuse existing config)
  125. for (const w of _workerPool) {
  126. if (w.token) tokens.push({ name: w.name, token: w.token, type: "oauth_flat" });
  127. }
  128. // Fallback to process-level CLAUDE_CODE_OAUTH_TOKEN
  129. if (tokens.length === 0) {
  130. const oat = process.env.CLAUDE_CODE_OAUTH_TOKEN;
  131. if (oat) tokens.push({ name: "default", token: oat, type: "oauth_flat" });
  132. }
  133. // Last resort: API key (per-token billing)
  134. if (tokens.length === 0) {
  135. const key = process.env.ANTHROPIC_API_KEY;
  136. if (key) tokens.push({ name: "apikey", token: key, type: "api_key_billed" });
  137. }
  138. return tokens;
  139. })();
  140. let _tokenRrIndex = 0;
  141. function getNextToken() {
  142. const idx = _tokenRrIndex++ % TOKEN_POOL.length;
  143. return TOKEN_POOL[idx];
  144. }
  145. // Backward compat: ANTHROPIC_AUTH still used for logging/health checks
  146. const ANTHROPIC_AUTH = TOKEN_POOL.length > 0 ? TOKEN_POOL[0] : null;
  147. if (TOKEN_POOL.length > 0) {
  148. console.log(`[ApiDirect] Token pool: ${TOKEN_POOL.length} token(s) — [${TOKEN_POOL.map(t => `${t.name}:${t.type}`).join(", ")}] — ALL requests via API direct`);
  149. } else {
  150. console.log(`[ApiDirect] No tokens configured — falling back to CLI workers`);
  151. }
  152. // ============================================================
  153. // Worker traffic & error tracking — exposed via /metrics for dashboard
  154. // ============================================================
  155. const workerStats = {
  156. // Per-worker traffic: { "1": { requests: 0, errors: 0 }, "3": { ... } }
  157. traffic: Object.fromEntries(_workerPool.map(w => [w.name, { requests: 0, errors: 0, lastReqAt: null }])),
  158. // Error categories: { cli_crash: N, cli_killed: N, context_overflow: N, ... }
  159. errors: {
  160. cli_crash: 0, // code=1 — CLI error (nested session, auth, prompt too long, etc.)
  161. cli_killed: 0, // code=143 — SIGTERM (reaper killed, heartbeat timeout)
  162. context_overflow: 0, // fallback API "Context size exceeded"
  163. api_error: 0, // Anthropic API errors (401, 429, 500)
  164. stream_retry: 0, // quick-fail retries on alternate worker
  165. timeout: 0, // heartbeat or exec timeout
  166. queue_timeout: 0, // queue timeout (waited too long)
  167. other: 0,
  168. },
  169. // Recent error log (ring buffer, last 100)
  170. recentErrors: [],
  171. };
  172. function recordWorkerRequest(workerName) {
  173. const w = workerStats.traffic[workerName];
  174. if (w) { w.requests++; w.lastReqAt = Date.now(); }
  175. }
  176. function recordWorkerError(workerName, category, detail) {
  177. const w = workerStats.traffic[workerName];
  178. if (w) w.errors++;
  179. if (workerStats.errors[category] !== undefined) workerStats.errors[category]++;
  180. else workerStats.errors.other++;
  181. workerStats.recentErrors.push({ ts: Date.now(), worker: workerName, category, detail: (detail || "").slice(0, 200) });
  182. if (workerStats.recentErrors.length > 100) workerStats.recentErrors.shift();
  183. }
  184. // _loadBalanceMode starts true: round-robin across all healthy workers
  185. // Falls back to single-worker mode when one worker is rate-limited
  186. let _loadBalanceMode = true;
  187. // Active connection tracking — for least-connections routing
  188. const _activeConns = new Map(_workerPool.map(w => [w.name, 0]));
  189. function workerAcquire(name) { _activeConns.set(name, (_activeConns.get(name) || 0) + 1); }
  190. function workerRelease(name) { const v = _activeConns.get(name) || 0; _activeConns.set(name, Math.max(0, v - 1)); }
  191. function leastLoadedWorker(pool) {
  192. let best = pool[0];
  193. let bestConns = _activeConns.get(best.name) ?? Infinity;
  194. let bestTotal = workerStats.traffic[best.name]?.requests ?? 0;
  195. for (let i = 1; i < pool.length; i++) {
  196. const c = _activeConns.get(pool[i].name) ?? 0;
  197. const t = workerStats.traffic[pool[i].name]?.requests ?? 0;
  198. // Primary: least active connections; Secondary: least total requests (evens out over time)
  199. if (c < bestConns || (c === bestConns && t < bestTotal)) {
  200. best = pool[i]; bestConns = c; bestTotal = t;
  201. }
  202. }
  203. return best;
  204. }
  205. /**
  206. * Get the next worker, respecting session affinity when available.
  207. *
  208. * @param {string} [sessionKey] - Session key for affinity lookup
  209. * @returns {object} worker from _workerPool
  210. */
  211. function getNextWorker(sessionKey) {
  212. const isHealthy = (name) => !_workerHealth.get(name)?.limited;
  213. const healthy = _workerPool.filter((w) => isHealthy(w.name));
  214. if (healthy.length === 0) {
  215. // All workers limited — pick the one that was limited longest ago
  216. const sorted = [..._workerPool].sort(
  217. (a, b) => _workerHealth.get(a.name).limitedAt - _workerHealth.get(b.name).limitedAt,
  218. );
  219. console.log(`[CLIRouter] ALL LIMITED — trying oldest-limited: ${sorted[0].name}`);
  220. return sorted[0];
  221. }
  222. if (healthy.length === 1) {
  223. return healthy[0];
  224. }
  225. // Degraded mode: only use primary
  226. if (!_loadBalanceMode) {
  227. const primary = healthy.find((w) => w.name === PRIMARY_WORKER);
  228. return primary || healthy[0];
  229. }
  230. // --- Least-connections is primary strategy ---
  231. // Session affinity is only a tiebreaker when workers have equal load.
  232. const least = leastLoadedWorker(healthy);
  233. const leastConns = _activeConns.get(least.name) || 0;
  234. if (sessionKey) {
  235. const aff = sessionAffinity.lookup(sessionKey, isHealthy);
  236. if (aff?.hit) {
  237. const affinityWorker = _workerPool.find((w) => w.name === aff.workerName);
  238. if (affinityWorker) {
  239. const affConns = _activeConns.get(affinityWorker.name) || 0;
  240. // Use affinity only if it's strictly less loaded (not just equal)
  241. if (affConns < leastConns) return affinityWorker;
  242. }
  243. }
  244. }
  245. return least;
  246. }
  247. function markWorkerLimited(workerName) {
  248. const h = _workerHealth.get(workerName);
  249. if (h && !h.limited) {
  250. h.limited = true;
  251. h.limitedAt = Date.now();
  252. _loadBalanceMode = false; // back to single-worker mode
  253. const other = _workerPool.find((w) => w.name !== workerName);
  254. console.log(`[CLIRouter] ${workerName} RATE LIMITED — switching all traffic to ${other?.name || "?"}`);
  255. eventLog.push("worker_limited", { worker: workerName, switchedTo: other?.name });
  256. }
  257. }
  258. function markWorkerRecovered(workerName) {
  259. const h = _workerHealth.get(workerName);
  260. if (h && h.limited) {
  261. h.limited = false;
  262. h.limitedAt = 0;
  263. _loadBalanceMode = true; // both workers healthy → share the load
  264. console.log(`[CLIRouter] ${workerName} RECOVERED — entering load-balance mode (round-robin)`);
  265. eventLog.push("worker_recovered", { worker: workerName, loadBalance: true });
  266. }
  267. }
  268. function isRateLimitError(exitCode, stderr) {
  269. if (!stderr) return false;
  270. const lower = stderr.toLowerCase();
  271. return (
  272. lower.includes("rate limit") ||
  273. lower.includes("429") ||
  274. lower.includes("too many requests") ||
  275. lower.includes("overloaded") ||
  276. lower.includes("you've hit your limit")
  277. );
  278. }
  279. // Health check timer: every HEALTH_CHECK_MS, try to recover limited workers
  280. setInterval(() => {
  281. for (const w of _workerPool) {
  282. const h = _workerHealth.get(w.name);
  283. if (h.limited && Date.now() - h.limitedAt >= HEALTH_CHECK_MS) {
  284. console.log(`[CLIRouter] Health check: ${w.name} cooldown expired (${Math.round((Date.now() - h.limitedAt) / 1000)}s) — marking recovered`);
  285. markWorkerRecovered(w.name);
  286. }
  287. }
  288. }, Math.min(HEALTH_CHECK_MS, 60000)); // Check at least every 60s
  289. // Whitelist of env vars safe to pass to CLI workers.
  290. // Everything else is blocked to prevent parent-process leakage
  291. // (e.g. CLAUDECODE causing "nested session" crash).
  292. const WORKER_ENV_WHITELIST = new Set([
  293. // System essentials
  294. "PATH", "HOME", "USER", "LOGNAME", "SHELL", "LANG", "LC_ALL", "LC_CTYPE",
  295. "TMPDIR", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME",
  296. // Node.js
  297. "NODE_PATH", "NODE_OPTIONS", "NODE_EXTRA_CA_CERTS",
  298. // SSH (agent forwarding, keys)
  299. "SSH_AUTH_SOCK", "SSH_AGENT_PID",
  300. // Proxy/network
  301. "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "http_proxy", "https_proxy", "no_proxy",
  302. // Anthropic (will be overridden per-worker below)
  303. "ANTHROPIC_API_KEY",
  304. ]);
  305. function workerEnv(worker) {
  306. // Start from a clean env — only whitelisted vars from parent process
  307. const env = {};
  308. for (const key of WORKER_ENV_WHITELIST) {
  309. if (process.env[key] !== undefined) env[key] = process.env[key];
  310. }
  311. // Ensure /opt/homebrew/bin + wrapper scripts dir are in PATH
  312. const path = env.PATH || "/usr/bin:/bin";
  313. const homeDir = process.env.HOME || "/Users/duke_nukem_opcdbase";
  314. const extraPaths = [`${homeDir}/.openclaw/bin`, "/opt/homebrew/bin"];
  315. let finalPath = path;
  316. for (const p of extraPaths) {
  317. if (!finalPath.includes(p)) finalPath = `${p}:${finalPath}`;
  318. }
  319. env.PATH = finalPath;
  320. // Per-worker OAuth token (overrides any inherited value)
  321. if (worker.token) {
  322. env.CLAUDE_CODE_OAUTH_TOKEN = worker.token;
  323. }
  324. // Headless / non-interactive mode — prevent ALL macOS interactive prompts
  325. env.CI = "true"; // suppress macOS permission popups
  326. env.TERM_PROGRAM = "dumb"; // skip terminal-specific osascript detection
  327. env.TERM = "dumb"; // reinforce non-interactive terminal
  328. env.NO_COLOR = "1"; // no ANSI escape codes
  329. env.ELECTRON_NO_ATTACH_CONSOLE = "1"; // suppress Electron console
  330. env.ELECTRON_RUN_AS_NODE = "1"; // skip Electron UI/keychain integration
  331. env.CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC = "1"; // skip telemetry/updates
  332. env.CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY = "1"; // skip interactive surveys
  333. // Prevent macOS Keychain access prompts: if safeStorage is attempted,
  334. // the OS prompts because node isn't in the Keychain ACL.
  335. // Setting ELECTRON_RUN_AS_NODE bypasses Electron's safeStorage layer.
  336. return env;
  337. }
  338. const MAX_CONCURRENT = parseInt(process.env.MAX_CONCURRENT || "10", 10);
  339. const MAX_QUEUE_TOTAL = parseInt(process.env.MAX_QUEUE_TOTAL || "200", 10);
  340. const MAX_QUEUE_PER_SOURCE = parseInt(process.env.MAX_QUEUE_PER_SOURCE || "50", 10);
  341. const QUEUE_TIMEOUT_MS = parseInt(process.env.QUEUE_TIMEOUT_MS || "300000", 10);
  342. const MAX_RETRIES = parseInt(process.env.MAX_RETRIES || "3", 10);
  343. const RETRY_BASE_MS = parseInt(process.env.RETRY_BASE_MS || "2000", 10);
  344. const STREAM_TIMEOUT_MS = parseInt(process.env.STREAM_TIMEOUT_MS || "1800000", 10); // 30 min (was 10 min)
  345. const SYNC_TIMEOUT_MS = parseInt(process.env.SYNC_TIMEOUT_MS || "600000", 10); // 10 min (was 5 min)
  346. // Per-model heartbeat timeouts — autonomous agents may go silent during tool execution
  347. const HEARTBEAT_BY_MODEL = Object.freeze({
  348. opus: 1800000, // 30 min — long tool chains (SSH compile, multi-step ops)
  349. sonnet: 1200000, // 20 min
  350. haiku: 600000, // 10 min
  351. });
  352. const DEFAULT_HEARTBEAT_MS = 1200000; // 20 min fallback
  353. const MAX_PROCESS_AGE_MS = parseInt(process.env.MAX_PROCESS_AGE_MS || "1800000", 10); // 30 min (was 10 min)
  354. const MAX_IDLE_MS = parseInt(process.env.MAX_IDLE_MS || "600000", 10); // 10 min (was 2 min)
  355. const REAPER_INTERVAL_MS = parseInt(process.env.REAPER_INTERVAL_MS || "15000", 10);
  356. // ============================================================
  357. // Rate Limits — 95% of Claude Max plan limits (shared globally)
  358. // ============================================================
  359. const RATE_LIMITS = {
  360. sonnet: { requestsPerMin: 57, tokensPerMin: 190000 },
  361. opus: { requestsPerMin: 28, tokensPerMin: 57000 },
  362. haiku: { requestsPerMin: 95, tokensPerMin: 380000 },
  363. };
  364. // Model priority mapping
  365. const MODEL_PRIORITY = { opus: "high", sonnet: "normal", haiku: "low" };
  366. // Model name -> CLI flag
  367. const MODEL_MAP = {
  368. "claude-code": "sonnet",
  369. sonnet: "sonnet",
  370. "sonnet-4.6": "sonnet",
  371. "claude-sonnet-4-6": "sonnet",
  372. opus: "opus",
  373. "opus-4.6": "opus",
  374. "claude-opus-4-6": "opus",
  375. haiku: "haiku",
  376. "haiku-4.5": "haiku",
  377. "claude-haiku-4-5": "haiku",
  378. };
  379. function resolveModel(model) {
  380. const stripped = (model || "sonnet").replace("claude-code/", "");
  381. return MODEL_MAP[stripped] || "sonnet";
  382. }
  383. // ============================================================
  384. // Redis + Module Instances
  385. // ============================================================
  386. // Redis — connect first, then pass to all modules
  387. let redis = null;
  388. try {
  389. redis = await createRedisClient();
  390. console.log("[Redis] Connected and ready");
  391. } catch (err) {
  392. console.warn(`[Redis] Connection failed: ${err.message} — running in memory-only mode`);
  393. redis = null;
  394. }
  395. const queue = createFairQueue({
  396. maxConcurrent: MAX_CONCURRENT,
  397. maxPerSource: MAX_QUEUE_PER_SOURCE,
  398. maxTotal: MAX_QUEUE_TOTAL,
  399. queueTimeoutMs: QUEUE_TIMEOUT_MS,
  400. maxLeaseMs: STREAM_TIMEOUT_MS + 60_000, // stream timeout + 1 min grace
  401. });
  402. const rateLimiter = createRateLimiter({ limits: RATE_LIMITS, redis });
  403. const registry = createProcessRegistry({
  404. maxProcessAgeMs: MAX_PROCESS_AGE_MS,
  405. maxIdleMs: MAX_IDLE_MS,
  406. reaperIntervalMs: REAPER_INTERVAL_MS,
  407. redis,
  408. });
  409. const retryPolicy = createRetryPolicy({
  410. maxRetries: MAX_RETRIES,
  411. baseDelayMs: RETRY_BASE_MS,
  412. });
  413. const eventLog = createEventLog({ maxEvents: 500, redis });
  414. const tokenTracker = createTokenTracker({ redis });
  415. // Metrics store: persistent time-series data for dashboard charts
  416. const metricsStore = createMetricsStore({ redis });
  417. // Session affinity: sticky routing for conversation sessions
  418. const sessionAffinity = createSessionAffinity({ ttlMs: 5 * 60 * 1000 }); // 5 min — short TTL for better distribution
  419. // Wire reaper events into event log + SSE
  420. registry.onReap((zombie) => {
  421. const ageS = Math.round(zombie.age / 1000);
  422. const idleS = Math.round(zombie.idle / 1000);
  423. eventLog.push("reap", {
  424. pid: zombie.pid,
  425. reqId: zombie.requestId,
  426. model: zombie.model,
  427. mode: zombie.mode,
  428. source: zombie.source,
  429. ageSec: ageS,
  430. idleSec: idleS,
  431. });
  432. sseBroadcast("reap", {
  433. pid: zombie.pid,
  434. reqId: zombie.requestId,
  435. model: zombie.model,
  436. ageSec: ageS,
  437. idleSec: idleS,
  438. });
  439. });
  440. // ============================================================
  441. // SSE Broadcast — real-time stream to dashboard subscribers
  442. // ============================================================
  443. let sseClients = new Set();
  444. function sseBroadcast(event, data) {
  445. if (sseClients.size === 0) return;
  446. const payload = `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`;
  447. for (const client of sseClients) {
  448. try {
  449. client.write(payload);
  450. } catch {
  451. sseClients = new Set([...sseClients].filter((c) => c !== client));
  452. }
  453. }
  454. }
  455. /**
  456. * Gather current metrics snapshot for the metrics store sampler.
  457. */
  458. function gatherMetricsSnapshot() {
  459. const qs = queue.getStats();
  460. const rs = registry.getStats();
  461. const counts = eventLog.getCounts();
  462. return {
  463. tokens: tokenTracker.getTotals(),
  464. tokensByModel: tokenTracker.getByModel(),
  465. queue: { active: qs.active, totalQueued: qs.totalQueued, metrics: qs.metrics },
  466. processes: rs,
  467. liveTokens: rs.liveTokens,
  468. events: counts,
  469. sessionAffinity: sessionAffinity.getStats(),
  470. };
  471. }
  472. // ============================================================
  473. // Auth & Source identification
  474. // ============================================================
  475. function authenticate(req) {
  476. const authHeader = req.headers["authorization"] || "";
  477. const apiKey = req.headers["x-api-key"] || "";
  478. const bearer = authHeader.startsWith("Bearer ") ? authHeader.slice(7) : "";
  479. // Accept any of: Bearer token, x-api-key, or query param
  480. return bearer === AUTH_TOKEN || apiKey === AUTH_TOKEN || AUTH_TOKEN === "local-proxy";
  481. }
  482. function identifySource(req) {
  483. // Priority: explicit header > api key > remote IP
  484. return (
  485. req.headers["x-openclaw-source"] ||
  486. req.headers["x-source"] ||
  487. req.headers["x-api-key"] ||
  488. req.socket.remoteAddress ||
  489. "unknown"
  490. );
  491. }
  492. // ============================================================
  493. // Claude CLI execution
  494. // ============================================================
  495. // Max prompt characters (~50K chars ≈ ~12K tokens, leaves room for CLI agent's own tool calls).
  496. // Opus has 200K token context; we reserve most of it for the agent's multi-turn tool execution.
  497. const MAX_PROMPT_CHARS = parseInt(process.env.MAX_PROMPT_CHARS || "50000", 10);
  498. function extractPrompt(messages) {
  499. if (!Array.isArray(messages) || messages.length === 0) {
  500. return { prompt: "", systemPrompt: null };
  501. }
  502. let systemPrompt = null;
  503. const systemMsg = messages.find((m) => m.role === "system" || m.role === "developer");
  504. if (systemMsg) {
  505. systemPrompt = typeof systemMsg.content === "string"
  506. ? systemMsg.content
  507. : JSON.stringify(systemMsg.content);
  508. }
  509. // Collect all non-system messages
  510. const allParts = [];
  511. for (const msg of messages) {
  512. if (msg.role === "system" || msg.role === "developer") continue;
  513. const text = typeof msg.content === "string" ? msg.content : JSON.stringify(msg.content);
  514. if (msg.role === "user") allParts.push(text);
  515. else if (msg.role === "assistant") allParts.push(`[Previous assistant]: ${text}`);
  516. }
  517. // Truncate from the front (keep recent messages) to fit within MAX_PROMPT_CHARS.
  518. // Always keep the LAST message (the actual user request).
  519. let totalLen = 0;
  520. const kept = [];
  521. for (let i = allParts.length - 1; i >= 0; i--) {
  522. const part = allParts[i];
  523. if (totalLen + part.length > MAX_PROMPT_CHARS && kept.length > 0) {
  524. // Budget exceeded — prepend a truncation notice and stop
  525. kept.unshift("[... earlier conversation history truncated ...]");
  526. break;
  527. }
  528. totalLen += part.length;
  529. kept.unshift(part);
  530. }
  531. return { prompt: kept.join("\n\n"), systemPrompt };
  532. }
  533. function buildCliArgs(prompt, model, systemPrompt, stream) {
  534. // Pass prompt via stdin (not CLI arg) to avoid OrbStack/OS arg length limits.
  535. // The `-p` flag without a positional prompt tells Claude CLI to read from stdin.
  536. // --dangerously-skip-permissions: full autonomous agent, all tools enabled
  537. // System prompt is prepended to stdin (not --system-prompt CLI arg) to avoid arg length limits.
  538. const args = ["-p", "--model", model, "--dangerously-skip-permissions"];
  539. if (stream) {
  540. args.push("--output-format", "stream-json", "--verbose", "--include-partial-messages");
  541. } else {
  542. args.push("--output-format", "text");
    • Note: Quotes wording that overrides an agent's instructions or safety (in quotes, code, a table or an example): "output-format", "text"); } return args; } // build the full stdin payload: system prompt"
    • Line 607 on GitHub
  543. }
  544. return args;
  545. }
  546. // Build the full stdin payload: system prompt (if any) + user prompt
  547. function buildStdinPayload(prompt, systemPrompt) {
  548. if (systemPrompt) {
  549. return `[System Instructions]\n${systemPrompt}\n\n[User Request]\n${prompt}`;
  550. }
  551. return prompt;
  552. }
  553. function runCliOnce(prompt, model, systemPrompt, requestId = "", source = "", workerOverride = null, sessionKey = "") {
  554. return new Promise((resolve, reject) => {
  555. const args = buildCliArgs(prompt, model, systemPrompt, false);
  556. const worker = workerOverride || getNextWorker(sessionKey);
  557. if (sessionKey) sessionAffinity.assign(sessionKey, worker.name);
  558. console.log(`[${ts()}] CLIROUTER obj=${worker.name} bin=${worker.bin} reqId=${requestId} model=${model}`);
  559. recordWorkerRequest(worker.name);
  560. workerAcquire(worker.name);
  561. const proc = spawn(worker.bin, args, {
  562. env: workerEnv(worker),
  563. stdio: ["pipe", "pipe", "pipe"],
  564. });
  565. // Write full payload (system prompt + user prompt) to stdin
  566. if (proc.stdin) {
  567. proc.stdin.write(buildStdinPayload(prompt, systemPrompt));
  568. proc.stdin.end();
  569. }
  570. // Track in process registry
  571. if (proc.pid) {
  572. registry.register({
  573. pid: proc.pid,
  574. requestId,
  575. model,
  576. mode: "sync",
  577. source,
  578. worker: `${worker.name}:${worker.bin}`,
  579. promptPreview: typeof prompt === "string" ? prompt.slice(0, 80) : "[structured]",
  580. });
  581. }
  582. // Execution timeout — kill if running too long
  583. const execTimer = setTimeout(() => {
  584. eventLog.push("timeout", { kind: "sync", pid: proc.pid, reqId: requestId, model });
  585. recordWorkerError(worker.name, "timeout", `sync_timeout pid=${proc.pid}`);
  586. console.log(`[${ts()}] SYNC_TIMEOUT pid=${proc.pid} reqId=${requestId} model=${model}`);
  587. try { proc.kill("SIGTERM"); } catch { /* ignore */ }
  588. const err = new Error(`Execution timeout after ${SYNC_TIMEOUT_MS}ms`);
  589. err.exitCode = -1;
  590. err.workerName = worker.name;
  591. reject(err);
  592. }, SYNC_TIMEOUT_MS);
  593. let stdout = "";
  594. let stderr = "";
  595. proc.stdout.on("data", (d) => {
  596. stdout += d.toString();
  597. if (proc.pid) registry.touch(proc.pid);
  598. });
  599. proc.stderr.on("data", (d) => { stderr += d.toString(); });
  600. proc.on("close", (code) => {
  601. clearTimeout(execTimer);
  602. workerRelease(worker.name);
  603. if (proc.pid) registry.unregister(proc.pid);
  604. // Detect rate limit from stderr
  605. if (isRateLimitError(code, stderr)) {
  606. markWorkerLimited(worker.name);
  607. }
  608. if (code !== 0) {
  609. const err = new Error(`CLI exit ${code}: ${stderr}`);
  610. err.exitCode = code;
  611. err.workerName = worker.name;
  612. err.isRateLimit = isRateLimitError(code, stderr);
  613. reject(err);
  614. } else {
  615. resolve(stdout.trim());
  616. }
  617. });
  618. proc.on("error", (err) => {
  619. clearTimeout(execTimer);
  620. workerRelease(worker.name);
  621. if (proc.pid) registry.unregister(proc.pid);
  622. err.workerName = worker.name;
  623. reject(err);
  624. });
  625. });
  626. }
  627. /**
  628. * Run CLI with retry + exponential backoff + jitter.
  629. * Uses retry policy for consistent retry behavior.
  630. */
  631. async function runCli(prompt, model, systemPrompt, requestId = "", source = "", sessionKey = "") {
  632. return retryPolicy.withRetry(
  633. () => runCliOnce(prompt, model, systemPrompt, requestId, source, null, sessionKey),
  634. {
  635. onRetry: (attempt, error, delayMs) => {
  636. eventLog.push("retry", { reqId: requestId, attempt: attempt + 1, model, delay: delayMs, error: error.message });
  637. console.log(
  638. `[${ts()}] RETRY attempt=${attempt + 1}/${MAX_RETRIES} ` +
  639. `model=${model} delay=${delayMs}ms err=${error.message}`
  640. );
  641. },
  642. },
  643. );
  644. }
  645. function spawnCliStream(prompt, model, systemPrompt, worker) {
  646. const args = buildCliArgs(prompt, model, systemPrompt, true);
  647. const proc = spawn(worker.bin, args, {
  648. env: workerEnv(worker),
  649. stdio: ["pipe", "pipe", "pipe"],
  650. });
  651. // Write full payload (system prompt + user prompt) to stdin
  652. if (proc.stdin) {
  653. proc.stdin.write(buildStdinPayload(prompt, systemPrompt));
  654. proc.stdin.end();
  655. }
  656. proc._workerName = worker.name;
  657. proc._spawnedAt = Date.now();
  658. return proc;
  659. }
  660. function trackStreamProc(proc, requestId, model, source, worker) {
  661. if (proc.pid) {
  662. registry.register({
  663. pid: proc.pid,
  664. requestId,
  665. model,
  666. mode: "stream",
  667. source,
  668. worker: `${worker.name}:${worker.bin}`,
  669. promptPreview: "[stream]",
  670. liveInputTokens: 0,
  671. liveOutputTokens: 0,
  672. });
  673. proc.on("close", () => registry.unregister(proc.pid));
  674. proc.on("error", () => registry.unregister(proc.pid));
  675. }
  676. }
  677. // Pick a specific worker by name, or fallback to round-robin
  678. function getWorkerByName(name) {
  679. return _workerPool.find((w) => w.name === name) || null;
  680. }
  681. function getAlternateWorker(excludeName) {
  682. const healthy = _workerPool.filter(
  683. (w) => w.name !== excludeName && !_workerHealth.get(w.name)?.limited
  684. );
  685. return healthy.length > 0 ? healthy[0] : null;
  686. }
  687. // ============================================================
  688. // Fallback API: stream from an OpenAI-compatible HTTP endpoint
  689. // Used as last resort when all CLI routers fail
  690. // ============================================================
  691. function streamFromFallbackApi(messages, model, reqId, source, res) {
  692. const fb = FALLBACK_API;
  693. const url = new URL(`${fb.baseUrl}/chat/completions`);
  694. const isHttps = url.protocol === "https:";
  695. const doRequest = isHttps ? httpsRequest : httpRequest;
  696. const body = JSON.stringify({
  697. model: fb.model,
  698. messages,
  699. stream: true,
  700. });
  701. console.log(`[${ts()}] FALLBACK reqId=${reqId} api=${fb.name} model=${fb.model} src=${source}`);
  702. eventLog.push("fallback", { reqId, model, source, fallbackApi: fb.name, fallbackModel: fb.model });
  703. // Safe write helper — prevent writing to already-closed response
  704. const safeWrite = (data) => { if (!res.writableEnded) res.write(data); };
  705. const safeEnd = () => { if (!res.writableEnded) res.end(); };
  706. const apiReq = doRequest(
  707. url,
  708. {
  709. method: "POST",
  710. headers: {
  711. "content-type": "application/json",
  712. authorization: `Bearer ${fb.apiKey}`,
  713. "content-length": Buffer.byteLength(body),
  714. },
  715. },
  716. (apiRes) => {
  717. if (apiRes.statusCode !== 200) {
  718. let errBody = "";
  719. apiRes.on("data", (d) => { errBody += d.toString(); });
  720. apiRes.on("end", () => {
  721. console.log(`[${ts()}] FALLBACK_ERROR reqId=${reqId} status=${apiRes.statusCode} body=${errBody.slice(0, 200)}`);
  722. recordWorkerError("fallback", errBody.includes("Context size") ? "context_overflow" : "api_error", `HTTP ${apiRes.statusCode} ${errBody.slice(0, 100)}`);
  723. safeWrite(sseChunk(reqId, `[Fallback ${fb.name} error: HTTP ${apiRes.statusCode}]`));
  724. safeWrite(sseChunk(reqId, null, "stop"));
  725. safeWrite("data: [DONE]\n\n");
  726. safeEnd();
  727. });
  728. return;
  729. }
  730. // Pipe the SSE stream from the fallback API directly to the client
  731. let buf = "";
  732. let outputChars = 0;
  733. apiRes.on("data", (chunk) => {
  734. buf += chunk.toString();
  735. const lines = buf.split("\n");
  736. buf = lines.pop() || "";
  737. for (const line of lines) {
  738. const trimmed = line.trim();
  739. if (!trimmed || trimmed === "data: [DONE]") {
  740. if (trimmed === "data: [DONE]") {
  741. safeWrite("data: [DONE]\n\n");
  742. }
  743. continue;
  744. }
  745. if (trimmed.startsWith("data: ")) {
  746. try {
  747. const ev = JSON.parse(trimmed.slice(6));
  748. const delta = ev.choices?.[0]?.delta?.content;
  749. const finish = ev.choices?.[0]?.finish_reason;
  750. if (delta) {
  751. safeWrite(sseChunk(reqId, delta));
  752. outputChars += delta.length;
  753. sseBroadcast("chunk", { reqId, model: fb.model, source, text: delta, tokens: outputChars });
  754. }
  755. if (finish) {
  756. safeWrite(sseChunk(reqId, null, finish));
  757. }
  758. } catch { /* skip malformed */ }
  759. }
  760. }
  761. });
  762. apiRes.on("end", () => {
  763. tokenTracker.record(reqId, fb.model, 0, Math.ceil(outputChars / 4));
  764. eventLog.push("complete", { reqId, mode: "fallback", model: fb.model, source, exitCode: 0, outputChars });
  765. sseBroadcast("complete", { reqId, model: fb.model, source, exitCode: 0 });
  766. if (outputChars === 0) {
  767. safeWrite(sseChunk(reqId, `[Fallback ${fb.name}: empty response]`));
  768. }
  769. safeWrite(sseChunk(reqId, null, "stop"));
  770. safeWrite("data: [DONE]\n\n");
  771. safeEnd();
  772. });
  773. },
  774. );
  775. apiReq.on("error", (err) => {
  776. console.log(`[${ts()}] FALLBACK_NET_ERROR reqId=${reqId} err=${err.message}`);
  777. safeWrite(sseChunk(reqId, `[Fallback ${fb.name} unreachable: ${err.message}]`));
  778. safeWrite(sseChunk(reqId, null, "stop"));
  779. safeWrite("data: [DONE]\n\n");
  780. safeEnd();
  781. });
  782. apiReq.write(body);
  783. apiReq.end();
  784. }
  785. // ============================================================
  786. // OpenAI response formatting
  787. // ============================================================
  788. function sseChunk(id, content, finishReason = null) {
  789. return `data: ${JSON.stringify({
  790. id,
  791. object: "chat.completion.chunk",
  792. created: Math.floor(Date.now() / 1000),
  793. model: "claude-code",
  794. choices: [{ index: 0, delta: content ? { content } : {}, finish_reason: finishReason }],
  795. })}\n\n`;
  796. }
  797. function completionResponse(id, content, model) {
  798. return {
  799. id,
  800. object: "chat.completion",
  801. created: Math.floor(Date.now() / 1000),
  802. model,
  803. choices: [{ index: 0, message: { role: "assistant", content }, finish_reason: "stop" }],
  804. usage: { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 },
  805. };
  806. }
  807. // ============================================================
  808. // Anthropic Direct API: format conversion + tool_calls support
  809. // ============================================================
  810. /**
  811. * Convert OpenAI tool definitions to Anthropic format.
  812. */
  813. function convertToolsToAnthropic(openaiTools) {
  814. if (!openaiTools || !Array.isArray(openaiTools)) return [];
  815. return openaiTools
  816. .filter(t => t.type === "function" && t.function)
  817. .map(t => ({
  818. name: t.function.name,
  819. description: t.function.description || "",
  820. input_schema: t.function.parameters || { type: "object", properties: {} },
  821. }));
  822. }
  823. /**
  824. * Convert OpenAI messages to Anthropic Messages API format.
  825. * Returns { system, messages } — system extracted as separate param.
  826. */
  827. function convertMessagesToAnthropic(openaiMessages) {
  828. let system;
  829. const rawMsgs = [];
  830. for (const msg of openaiMessages) {
  831. if (msg.role === "system" || msg.role === "developer") {
  832. const text = typeof msg.content === "string" ? msg.content : JSON.stringify(msg.content);
  833. system = system ? `${system}\n\n${text}` : text;
  834. continue;
  835. }
  836. if (msg.role === "user") {
  837. const content = typeof msg.content === "string"
  838. ? [{ type: "text", text: msg.content }]
  839. : Array.isArray(msg.content) ? msg.content : [{ type: "text", text: String(msg.content) }];
  840. rawMsgs.push({ role: "user", content });
  841. continue;
  842. }
  843. if (msg.role === "assistant") {
  844. const content = [];
  845. if (msg.content) {
  846. const text = typeof msg.content === "string" ? msg.content : JSON.stringify(msg.content);
  847. if (text) content.push({ type: "text", text });
  848. }
  849. if (Array.isArray(msg.tool_calls)) {
  850. for (const tc of msg.tool_calls) {
  851. let input = {};
  852. try {
  853. input = typeof tc.function?.arguments === "string"
  854. ? JSON.parse(tc.function.arguments)
  855. : (tc.function?.arguments || {});
  856. } catch { /* use empty object */ }
  857. content.push({
  858. type: "tool_use",
  859. id: tc.id,
  860. name: tc.function?.name || "unknown",
  861. input,
  862. });
  863. }
  864. }
  865. if (content.length > 0) rawMsgs.push({ role: "assistant", content });
  866. continue;
  867. }
  868. if (msg.role === "tool") {
  869. rawMsgs.push({
  870. role: "user",
  871. content: [{
  872. type: "tool_result",
  873. tool_use_id: msg.tool_call_id,
  874. content: typeof msg.content === "string" ? msg.content : JSON.stringify(msg.content),
  875. }],
  876. });
  877. continue;
  878. }
  879. }
  880. // Merge consecutive same-role messages (Anthropic requires alternating roles)
  881. const messages = [];
  882. for (const msg of rawMsgs) {
  883. const prev = messages[messages.length - 1];
  884. if (prev && prev.role === msg.role) {
  885. const prevContent = Array.isArray(prev.content)
  886. ? prev.content : [{ type: "text", text: String(prev.content) }];
  887. const newContent = Array.isArray(msg.content)
  888. ? msg.content : [{ type: "text", text: String(msg.content) }];
  889. prev.content = [...prevContent, ...newContent];
  890. } else {
  891. messages.push({
  892. ...msg,
  893. content: Array.isArray(msg.content) ? [...msg.content] : msg.content,
  894. });
  895. }
  896. }
  897. return { system, messages };
  898. }
  899. /** SSE chunk: initial tool_call with id + name. */
  900. function sseToolCallStartChunk(id, index, callId, name) {
  901. return `data: ${JSON.stringify({
  902. id,
  903. object: "chat.completion.chunk",
  904. created: Math.floor(Date.now() / 1000),
  905. model: "claude-code",
  906. choices: [{
  907. index: 0,
  908. delta: {
  909. tool_calls: [{ index, id: callId, type: "function", function: { name, arguments: "" } }],
  910. },
  911. finish_reason: null,
  912. }],
  913. })}\n\n`;
  914. }
  915. /** SSE chunk: streaming tool_call argument delta. */
  916. function sseToolCallDeltaChunk(id, index, argsDelta) {
  917. return `data: ${JSON.stringify({
  918. id,
  919. object: "chat.completion.chunk",
  920. created: Math.floor(Date.now() / 1000),
  921. model: "claude-code",
  922. choices: [{
  923. index: 0,
  924. delta: {
  925. tool_calls: [{ index, function: { arguments: argsDelta } }],
  926. },
  927. finish_reason: null,
  928. }],
  929. })}\n\n`;
  930. }
  931. /** SSE chunk: finish_reason only (no content). */
  932. function sseFinishChunk(id, finishReason) {
  933. return `data: ${JSON.stringify({
  934. id,
  935. object: "chat.completion.chunk",
  936. created: Math.floor(Date.now() / 1000),
  937. model: "claude-code",
  938. choices: [{ index: 0, delta: {}, finish_reason: finishReason }],
  939. })}\n\n`;
  940. }
  941. /** Non-streaming response with optional tool_calls. */
  942. function completionResponseWithTools(id, content, toolCalls, model, usage) {
  943. const message = { role: "assistant", content: content || null };
  944. if (toolCalls && toolCalls.length > 0) {
  945. message.tool_calls = toolCalls.map(tc => ({
  946. id: tc.id,
  947. type: "function",
  948. function: { name: tc.name, arguments: tc.arguments },
  949. }));
  950. }
  951. return {
  952. id,
  953. object: "chat.completion",
  954. created: Math.floor(Date.now() / 1000),
  955. model,
  956. choices: [{
  957. index: 0,
  958. message,
  959. finish_reason: toolCalls && toolCalls.length > 0 ? "tool_calls" : "stop",
  960. }],
  961. usage: usage || { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 },
  962. };
  963. }
  964. /**
  965. * Stream response from Anthropic Messages API, converting to OpenAI SSE.
  966. * Handles text content + tool_use blocks.
  967. */
  968. function streamFromAnthropicDirect(body, model, reqId, source, res, release, tokenEntry) {
  969. const anthropicModel = ANTHROPIC_MODEL_IDS[model] || ANTHROPIC_MODEL_IDS.sonnet;
  970. const anthropicTools = body.tools ? convertToolsToAnthropic(body.tools) : [];
  971. const { system, messages } = convertMessagesToAnthropic(body.messages);
  972. const requestBody = {
  973. model: anthropicModel,
  974. max_tokens: body.max_tokens || 16384,
  975. stream: true,
  976. messages,
  977. };
  978. if (system) requestBody.system = system;
  979. if (anthropicTools.length > 0) requestBody.tools = anthropicTools;
  980. if (body.tool_choice) {
  981. if (body.tool_choice === "auto") requestBody.tool_choice = { type: "auto" };
  982. else if (body.tool_choice === "none") requestBody.tool_choice = { type: "none" };
  983. else if (body.tool_choice === "required") requestBody.tool_choice = { type: "any" };
  984. else if (body.tool_choice?.type === "function") {
  985. requestBody.tool_choice = { type: "tool", name: body.tool_choice.function.name };
  986. }
  987. }
  988. const bodyStr = JSON.stringify(requestBody);
  989. const authHeaderName = tokenEntry.type === "oauth_flat" ? "authorization" : "x-api-key";
  990. const authHeaderValue = tokenEntry.type === "oauth_flat" ? `Bearer ${tokenEntry.token}` : tokenEntry.token;
  991. console.log(
  992. `[${ts()}] ANTHROPIC_STREAM reqId=${reqId} model=${anthropicModel} ` +
  993. `tools=${anthropicTools.length} msgs=${messages.length} auth=${tokenEntry.type} token=${tokenEntry.name} src=${source}`
  994. );
  995. eventLog.push("anthropic_direct", {
  996. reqId, model: anthropicModel, tools: anthropicTools.length, source, auth: tokenEntry.type, token: tokenEntry.name,
  997. });
  998. const safeWrite = (data) => { if (!res.writableEnded) res.write(data); };
  999. const safeEnd = () => { if (!res.writableEnded) res.end(); };
  1000. let released = false;
  1001. const doRelease = () => { if (!released) { released = true; release(); } };
  1002. const url = new URL(`${ANTHROPIC_API_BASE}/v1/messages`);
  1003. const headers = {
  1004. "content-type": "application/json",
  1005. "anthropic-version": ANTHROPIC_API_VERSION,
  1006. ...(tokenEntry.type === "oauth_flat" ? { "anthropic-beta": "oauth-2025-04-20" } : {}),
  1007. "content-length": String(Buffer.byteLength(bodyStr)),
  1008. };
  1009. headers[authHeaderName] = authHeaderValue;
  1010. const apiReq = httpsRequest(url, { method: "POST", headers }, (apiRes) => {
  1011. if (apiRes.statusCode !== 200) {
  1012. let errBody = "";
  1013. apiRes.on("data", (d) => { errBody += d.toString(); });
  1014. apiRes.on("end", () => {
  1015. console.log(`[${ts()}] ANTHROPIC_ERROR reqId=${reqId} status=${apiRes.statusCode} body=${errBody.slice(0, 500)}`);
  1016. eventLog.push("error", { reqId, mode: "anthropic_direct", model, source, status: apiRes.statusCode });
  1017. safeWrite(sseChunk(reqId, `[Anthropic API error: HTTP ${apiRes.statusCode}]`));
  1018. safeWrite(sseFinishChunk(reqId, "stop"));
  1019. safeWrite("data: [DONE]\n\n");
  1020. safeEnd();
  1021. doRelease();
  1022. });
  1023. return;
  1024. }
  1025. let buf = "";
  1026. let toolCallIndex = -1;
  1027. const toolCalls = [];
  1028. let inputTokens = 0;
  1029. let outputTokens = 0;
  1030. apiRes.on("data", (chunk) => {
  1031. buf += chunk.toString();
  1032. const lines = buf.split("\n");
  1033. buf = lines.pop() || "";
  1034. for (const line of lines) {
  1035. const trimmed = line.trim();
  1036. if (!trimmed || !trimmed.startsWith("data: ")) continue;
  1037. let ev;
  1038. try { ev = JSON.parse(trimmed.slice(6)); } catch { continue; }
  1039. if (ev.type === "message_start") {
  1040. inputTokens = ev.message?.usage?.input_tokens || 0;
  1041. } else if (ev.type === "content_block_start") {
  1042. const block = ev.content_block;
  1043. if (block?.type === "tool_use") {
  1044. toolCallIndex++;
  1045. toolCalls.push({ index: toolCallIndex, id: block.id, name: block.name, arguments: "" });
  1046. safeWrite(sseToolCallStartChunk(reqId, toolCallIndex, block.id, block.name));
  1047. }
  1048. // text blocks and thinking blocks: no special start action needed
  1049. } else if (ev.type === "content_block_delta") {
  1050. if (ev.delta?.type === "text_delta" && ev.delta.text) {
  1051. safeWrite(sseChunk(reqId, ev.delta.text));
  1052. } else if (ev.delta?.type === "input_json_delta" && ev.delta.partial_json !== undefined) {
  1053. const tc = toolCalls[toolCalls.length - 1];
  1054. if (tc) {
  1055. tc.arguments += ev.delta.partial_json;
  1056. safeWrite(sseToolCallDeltaChunk(reqId, tc.index, ev.delta.partial_json));
  1057. }
  1058. }
  1059. // thinking_delta: skip silently
  1060. } else if (ev.type === "message_delta") {
  1061. outputTokens = ev.usage?.output_tokens || outputTokens;
  1062. const stop = ev.delta?.stop_reason;
  1063. if (stop) {
  1064. const finish = stop === "tool_use" ? "tool_calls"
  1065. : stop === "end_turn" ? "stop"
  1066. : stop === "max_tokens" ? "length"
  1067. : "stop";
  1068. safeWrite(sseFinishChunk(reqId, finish));
  1069. }
  1070. }
  1071. // message_stop, content_block_stop, ping: no action needed
  1072. }
  1073. });
  1074. apiRes.on("end", () => {
  1075. tokenTracker.record(reqId, model, inputTokens, outputTokens);
  1076. eventLog.push("complete", {
  1077. reqId, mode: "anthropic_direct", model, source,
  1078. inputTokens, outputTokens, toolCalls: toolCalls.length,
  1079. });
  1080. sseBroadcast("complete", { reqId, model, source, inputTokens, outputTokens });
  1081. safeWrite("data: [DONE]\n\n");
  1082. safeEnd();
  1083. doRelease();
  1084. });
  1085. apiRes.on("error", (err) => {
  1086. console.log(`[${ts()}] ANTHROPIC_STREAM_ERR reqId=${reqId} err=${err.message}`);
  1087. safeWrite(sseChunk(reqId, `[Anthropic stream error: ${err.message}]`));
  1088. safeWrite("data: [DONE]\n\n");
  1089. safeEnd();
  1090. doRelease();
  1091. });
  1092. });
  1093. // Client disconnect: abort Anthropic API request
  1094. res.on("close", () => {
  1095. if (!apiReq.destroyed) {
  1096. console.log(`[${ts()}] CLIENT_DISCONNECT reqId=${reqId} — aborting Anthropic API request`);
  1097. apiReq.destroy();
  1098. }
  1099. doRelease();
  1100. });
  1101. apiReq.on("error", (err) => {
  1102. console.log(`[${ts()}] ANTHROPIC_NET_ERR reqId=${reqId} err=${err.message}`);
  1103. eventLog.push("error", { reqId, mode: "anthropic_direct", model, source, error: err.message });
  1104. safeWrite(sseChunk(reqId, `[Anthropic API unreachable: ${err.message}]`));
  1105. safeWrite(sseFinishChunk(reqId, "stop"));
  1106. safeWrite("data: [DONE]\n\n");
  1107. safeEnd();
  1108. doRelease();
  1109. });
  1110. apiReq.write(bodyStr);
  1111. apiReq.end();
  1112. }
  1113. /**
  1114. * Call Anthropic Messages API synchronously (non-streaming).
  1115. * Returns { content, toolCalls, usage, stopReason }.
  1116. */
  1117. function callAnthropicDirect(body, model, reqId, source, tokenEntry) {
  1118. return new Promise((resolve, reject) => {
  1119. const anthropicModel = ANTHROPIC_MODEL_IDS[model] || ANTHROPIC_MODEL_IDS.sonnet;
  1120. const anthropicTools = body.tools ? convertToolsToAnthropic(body.tools) : [];
  1121. const { system, messages } = convertMessagesToAnthropic(body.messages);
  1122. const requestBody = {
  1123. model: anthropicModel,
  1124. max_tokens: body.max_tokens || 16384,
  1125. messages,
  1126. };
  1127. if (system) requestBody.system = system;
  1128. if (anthropicTools.length > 0) requestBody.tools = anthropicTools;
  1129. const bodyStr = JSON.stringify(requestBody);
  1130. const authHeaderName = tokenEntry.type === "oauth_flat" ? "authorization" : "x-api-key";
  1131. const authHeaderValue = tokenEntry.type === "oauth_flat" ? `Bearer ${tokenEntry.token}` : tokenEntry.token;
  1132. console.log(
  1133. `[${ts()}] ANTHROPIC_SYNC reqId=${reqId} model=${anthropicModel} ` +
  1134. `tools=${anthropicTools.length} auth=${tokenEntry.type} token=${tokenEntry.name} src=${source}`
  1135. );
  1136. const url = new URL(`${ANTHROPIC_API_BASE}/v1/messages`);
  1137. const headers = {
  1138. "content-type": "application/json",
  1139. "anthropic-version": ANTHROPIC_API_VERSION,
  1140. ...(tokenEntry.type === "oauth_flat" ? { "anthropic-beta": "oauth-2025-04-20" } : {}),
  1141. "content-length": String(Buffer.byteLength(bodyStr)),
  1142. };
  1143. headers[authHeaderName] = authHeaderValue;
  1144. const timer = setTimeout(() => {
  1145. apiReq.destroy();
  1146. reject(new Error(`Anthropic API timeout after ${SYNC_TIMEOUT_MS}ms`));
  1147. }, SYNC_TIMEOUT_MS);
  1148. const apiReq = httpsRequest(url, { method: "POST", headers }, (apiRes) => {
  1149. let resBody = "";
  1150. apiRes.on("data", (d) => { resBody += d.toString(); });
  1151. apiRes.on("end", () => {
  1152. clearTimeout(timer);
  1153. if (apiRes.statusCode !== 200) {
  1154. return reject(new Error(`Anthropic API HTTP ${apiRes.statusCode}: ${resBody.slice(0, 500)}`));
  1155. }
  1156. try {
  1157. const result = JSON.parse(resBody);
  1158. let textContent = "";
  1159. const toolCalls = [];
  1160. for (const block of (result.content || [])) {
  1161. if (block.type === "text") textContent += block.text;
  1162. else if (block.type === "tool_use") {
  1163. toolCalls.push({ id: block.id, name: block.name, arguments: JSON.stringify(block.input) });
  1164. }
  1165. }
  1166. resolve({
  1167. content: textContent || null,
  1168. toolCalls,
  1169. usage: {
  1170. prompt_tokens: result.usage?.input_tokens || 0,
  1171. completion_tokens: result.usage?.output_tokens || 0,
  1172. total_tokens: (result.usage?.input_tokens || 0) + (result.usage?.output_tokens || 0),
  1173. },
  1174. stopReason: result.stop_reason,
  1175. });
  1176. } catch (err) {
  1177. reject(new Error(`Failed to parse Anthropic response: ${err.message}`));
  1178. }
  1179. });
  1180. });
  1181. apiReq.on("error", (err) => {
  1182. clearTimeout(timer);
  1183. reject(err);
  1184. });
  1185. apiReq.write(bodyStr);
  1186. apiReq.end();
  1187. });
  1188. }
  1189. /**
  1190. * Handle ALL requests via direct Anthropic API with token round-robin.
  1191. * Supports both tool-enabled and text-only requests.
  1192. */
  1193. async function handleApiDirect(body, model, stream, source, req, res) {
  1194. const priority = MODEL_PRIORITY[model] || "normal";
  1195. const estTokens = Math.min(Math.ceil(JSON.stringify(body.messages).length / 4), 5000);
  1196. const reqId = `chatcmpl-${randomUUID().replace(/-/g, "").slice(0, 24)}`;
  1197. let release;
  1198. try {
  1199. release = await queue.acquire(source, priority);
  1200. } catch (err) {
  1201. return sendJson(res, 503, {
  1202. error: { message: `Queue full: ${err.message}`, type: "queue_full", retry_after_ms: 10000 },
  1203. }, { "retry-after": "10" });
  1204. }
  1205. let rateWaitTotal = 0;
  1206. while (true) {
  1207. const rateCheck = rateLimiter.check(model, estTokens);
  1208. if (rateCheck.ok) break;
  1209. if (rateWaitTotal >= 300000) {
  1210. release();
  1211. return sendJson(res, 503, {
  1212. error: { message: "Rate limit wait exceeded", type: "rate_limit_timeout" },
  1213. });
  1214. }
  1215. const sleepMs = Math.min(rateCheck.waitMs, 5000);
  1216. await new Promise(r => setTimeout(r, sleepMs));
  1217. rateWaitTotal += sleepMs;
  1218. }
  1219. rateLimiter.record(model, estTokens);
  1220. eventLog.push("request", {
  1221. reqId, mode: stream ? "stream_tools" : "sync_tools", model, source, priority,
  1222. toolCount: body.tools?.length || 0,
  1223. });
  1224. sseBroadcast("request", {
  1225. reqId, mode: stream ? "stream_tools" : "sync_tools", model, source, priority,
  1226. });
  1227. const tokenEntry = getNextToken();
  1228. console.log(
  1229. `[${ts()}] ${stream ? "STREAM" : "SYNC"}_API src=${source} model=${model} ` +
  1230. `tools=${body.tools?.length || 0} token=${tokenEntry.name} reqId=${reqId}`
  1231. );
  1232. if (stream) {
  1233. res.writeHead(200, {
  1234. "content-type": "text/event-stream",
  1235. "cache-control": "no-cache",
  1236. connection: "keep-alive",
  1237. "x-accel-buffering": "no",
  1238. });
  1239. res.flushHeaders();
  1240. if (res.socket) res.socket.setNoDelay(true);
  1241. streamFromAnthropicDirect(body, model, reqId, source, res, release, tokenEntry);
  1242. } else {
  1243. try {
  1244. const result = await callAnthropicDirect(body, model, reqId, source, tokenEntry);
  1245. release();
  1246. tokenTracker.record(reqId, model, result.usage.prompt_tokens, result.usage.completion_tokens);
  1247. eventLog.push("complete", {
  1248. reqId, mode: "anthropic_direct_sync", model, source, ...result.usage,
  1249. });
  1250. sendJson(res, 200, completionResponseWithTools(
  1251. reqId, result.content, result.toolCalls, model, result.usage,
  1252. ));
  1253. } catch (err) {
  1254. release();
  1255. console.error(`[${ts()}] TOOL_REQ_ERROR reqId=${reqId} src=${source} ${err.message}`);
  1256. eventLog.push("error", { reqId, mode: "anthropic_direct", model, source, error: err.message });
  1257. sendJson(res, 500, { error: { message: err.message, type: "anthropic_api_error" } });
  1258. }
  1259. }
  1260. }
  1261. // ============================================================
  1262. // Request handler: /v1/chat/completions
  1263. // ============================================================
  1264. async function handleCompletions(req, res) {
  1265. const source = identifySource(req);
  1266. const chunks = [];
  1267. for await (const chunk of req) chunks.push(chunk);
  1268. let body;
  1269. try {
  1270. body = JSON.parse(Buffer.concat(chunks).toString());
  1271. } catch {
  1272. return sendJson(res, 400, { error: { message: "Invalid JSON body" } });
  1273. }
  1274. const { messages, model: rawModel = "claude-code", stream = false } = body;
  1275. if (!messages || !Array.isArray(messages)) {
  1276. return sendJson(res, 400, { error: { message: "messages array required" } });
  1277. }
  1278. // API direct path: only when CLI agent mode is off and tokens are available
  1279. if (!USE_CLI_AGENTS && TOKEN_POOL.length > 0) {
  1280. return handleApiDirect(body, resolveModel(rawModel), stream, source, req, res);
  1281. }
  1282. const { prompt, systemPrompt } = extractPrompt(messages);
  1283. if (!prompt) {
  1284. return sendJson(res, 400, { error: { message: "No user message found" } });
  1285. }
  1286. // Session affinity: derive a key so the same conversation sticks to the same worker
  1287. const sessionKey = sessionAffinity.deriveKey({
  1288. source,
  1289. sessionId: req.headers["x-session-id"] || "",
  1290. systemPrompt: systemPrompt || "",
  1291. });
  1292. const model = resolveModel(rawModel);
  1293. const priority = MODEL_PRIORITY[model] || "normal";
  1294. // Estimated tokens for rate-limiter: use a small fixed cap.
  1295. // chars/4 wildly over-estimates (code/JSON has low token density).
  1296. // The real rate limit is Anthropic's 429 response; our limiter is
  1297. // just a courtesy throttle. Cap at 5000 so ~11 opus requests/min
  1298. // can coexist (57000/5000). If Anthropic 429s, the retry loop handles it.
  1299. const estTokens = Math.min(Math.ceil(prompt.length / 4), 5000);
  1300. const reqId = `chatcmpl-${randomUUID().replace(/-/g, "").slice(0, 24)}`;
  1301. // Acquire slot via fair queue (waits for turn, never rejects)
  1302. let release;
  1303. try {
  1304. release = await queue.acquire(source, priority);
  1305. } catch (err) {
  1306. // Only rejects if queue is truly full (100+ pending)
  1307. console.log(`[${ts()}] QUEUE_FULL src=${source} model=${model} ${err.message}`);
  1308. return sendJson(res, 503, {
  1309. error: { message: `Queue full, try again shortly: ${err.message}`, type: "queue_full", retry_after_ms: 10000 },
  1310. }, { "retry-after": "10" });
  1311. }
  1312. // Wait for rate limit window (sleep instead of rejecting)
  1313. let rateWaitTotal = 0;
  1314. const MAX_RATE_WAIT_MS = 300000;
  1315. while (true) {
  1316. const rateCheck = rateLimiter.check(model, estTokens);
  1317. if (rateCheck.ok) break;
  1318. if (rateWaitTotal >= MAX_RATE_WAIT_MS) {
  1319. release();
  1320. console.log(`[${ts()}] RATE_TIMEOUT src=${source} model=${model} waited ${rateWaitTotal}ms`);
  1321. return sendJson(res, 503, {
  1322. error: { message: `Rate limit wait exceeded ${MAX_RATE_WAIT_MS}ms`, type: "rate_limit_timeout" },
  1323. });
  1324. }
  1325. const sleepMs = Math.min(rateCheck.waitMs, 5000);
  1326. console.log(`[${ts()}] RATE_WAIT src=${source} model=${model} sleeping ${sleepMs}ms (${rateCheck.reason})`);
  1327. await new Promise((r) => setTimeout(r, sleepMs));
  1328. rateWaitTotal += sleepMs;
  1329. }
  1330. rateLimiter.record(model, estTokens);
  1331. eventLog.push("request", { reqId, mode: stream ? "stream" : "sync", model, source, priority });
  1332. sseBroadcast("request", { reqId, mode: stream ? "stream" : "sync", model, source, priority, promptPreview: prompt.slice(0, 80) });
  1333. console.log(`[${ts()}] ${stream ? "STREAM" : "SYNC"} src=${source} model=${model} prio=${priority} session=${sessionKey.slice(0, 30)} prompt=${prompt.slice(0, 60)}...`);
  1334. if (stream) {
  1335. res.writeHead(200, {
  1336. "content-type": "text/event-stream",
  1337. "cache-control": "no-cache",
  1338. connection: "keep-alive",
  1339. "x-accel-buffering": "no", // hint to reverse proxies: don't buffer
  1340. });
  1341. res.flushHeaders(); // force headers out immediately
  1342. if (res.socket) {
  1343. res.socket.setNoDelay(true); // disable Nagle — send chunks immediately
  1344. }
  1345. // Immediate keepalive: prevents Gateway from timing out while CLI spawns.
  1346. // Without this, there's a 4-10s gap between headers and first CLI output,
  1347. // causing ~49% of requests to be disconnected by Gateway.
  1348. res.write(":proxy-accepted\n\n");
  1349. // Stream with auto-retry: if a worker fails quickly (<5s, no content),
  1350. // automatically retry on a different worker before giving up.
  1351. // If ALL CLI routers fail, fall back to the API endpoint (e.g. MiniMax).
  1352. const QUICK_FAIL_MS = 5000;
  1353. const MAX_RETRIES = _workerPool.length; // try each router once
  1354. const inputEstimate = Math.ceil(prompt.length / 4);
  1355. const originalMessages = messages; // preserve for fallback API
  1356. let retryCount = 0;
  1357. const triedRouters = new Set();
  1358. let activeProc = null; // track current CLI process for client-disconnect cleanup
  1359. // If client disconnects, kill the CLI process to free resources
  1360. res.on("close", () => {
  1361. if (activeProc && !activeProc.killed) {
  1362. console.log(`[${ts()}] CLIENT_DISCONNECT reqId=${reqId} — killing CLI pid=${activeProc.pid}`);
  1363. try { activeProc.kill("SIGTERM"); } catch { /* ignore */ }
  1364. }
  1365. });
  1366. function pipeStream(workerOverride, isRetry) {
  1367. const worker = workerOverride || getNextWorker(sessionKey);
  1368. // Bind this session to the chosen worker
  1369. sessionAffinity.assign(sessionKey, worker.name);
  1370. triedRouters.add(worker.name);
  1371. console.log(`[${ts()}] CLIROUTER obj=${worker.name} bin=${worker.bin} reqId=${reqId} model=${model} src=${source}${isRetry ? ` RETRY#${retryCount}` : ""}`);
  1372. recordWorkerRequest(worker.name);
  1373. workerAcquire(worker.name);
  1374. const proc = spawnCliStream(prompt, model, systemPrompt, worker);
  1375. activeProc = proc; // update for client-disconnect handler
  1376. trackStreamProc(proc, reqId, model, source, worker);
  1377. let buffer = "";
  1378. let stderrBuf = "";
  1379. let sentContent = false;
  1380. let reqTokens = { input: 0, output: 0 };
  1381. let outputChars = 0;
  1382. const spawnedAt = Date.now();
  1383. proc.stderr.on("data", (d) => { stderrBuf += d.toString(); });
  1384. // First-byte warning: if CLI hasn't produced stdout within 8s, log a warning.
  1385. // This helps diagnose macOS auth dialogs, slow spawns, or keychain prompts.
  1386. const FIRST_BYTE_WARN_MS = 8_000;
  1387. const firstByteTimer = setTimeout(() => {
  1388. console.log(`[${ts()}] SLOW_SPAWN pid=${proc.pid} reqId=${reqId} model=${model} router=${worker.name} elapsed=${FIRST_BYTE_WARN_MS}ms — no stdout yet (possible macOS dialog or slow startup)`);
  1389. eventLog.push("timeout", { kind: "slow_spawn", pid: proc.pid, reqId, model, source, elapsed: FIRST_BYTE_WARN_MS });
  1390. }, FIRST_BYTE_WARN_MS);
  1391. const heartbeatMs = HEARTBEAT_BY_MODEL[model] || DEFAULT_HEARTBEAT_MS;
  1392. let heartbeatTimer = setTimeout(() => {
  1393. eventLog.push("timeout", { kind: "heartbeat", pid: proc.pid, reqId, model, source, heartbeatMs });
  1394. console.log(`[${ts()}] HEARTBEAT_TIMEOUT pid=${proc.pid} reqId=${reqId} model=${model} src=${source} limit=${heartbeatMs}ms`);
  1395. try { proc.kill("SIGTERM"); } catch { /* ignore */ }
  1396. }, heartbeatMs);
  1397. function resetHeartbeat() {
  1398. clearTimeout(heartbeatTimer);
  1399. heartbeatTimer = setTimeout(() => {
  1400. eventLog.push("timeout", { kind: "heartbeat", pid: proc.pid, reqId, model, source, heartbeatMs });
  1401. console.log(`[${ts()}] HEARTBEAT_TIMEOUT pid=${proc.pid} reqId=${reqId} model=${model} src=${source} limit=${heartbeatMs}ms`);
  1402. try { proc.kill("SIGTERM"); } catch { /* ignore */ }
  1403. }, heartbeatMs);
  1404. }
  1405. const execTimer = setTimeout(() => {
  1406. eventLog.push("timeout", { kind: "stream_exec", pid: proc.pid, reqId, model });
  1407. console.log(`[${ts()}] STREAM_TIMEOUT pid=${proc.pid} reqId=${reqId} model=${model} age=${STREAM_TIMEOUT_MS}ms`);
  1408. try { proc.kill("SIGTERM"); } catch { /* ignore */ }
  1409. }, STREAM_TIMEOUT_MS);
  1410. // SSE keepalive: send comment lines to prevent upstream (Gateway) HTTP timeout.
  1411. // SSE spec allows `:comment\n\n` — client parsers ignore it but the TCP stays alive.
  1412. // Phase 1: fast keepalive (5s) during CLI startup; Phase 2: slow (30s) after first content.
  1413. const FAST_KEEPALIVE_MS = 5_000;
  1414. const SLOW_KEEPALIVE_MS = 30_000;
  1415. let keepaliveMs = FAST_KEEPALIVE_MS;
  1416. let keepaliveInterval = setInterval(() => {
  1417. if (!res.writableEnded) {
  1418. try { res.write(":keepalive\n\n"); } catch { /* ignore write errors */ }
  1419. }
  1420. }, keepaliveMs);
  1421. function slowDownKeepalive() {
  1422. if (keepaliveMs === FAST_KEEPALIVE_MS) {
  1423. keepaliveMs = SLOW_KEEPALIVE_MS;
  1424. clearInterval(keepaliveInterval);
  1425. keepaliveInterval = setInterval(() => {
  1426. if (!res.writableEnded) {
  1427. try { res.write(":keepalive\n\n"); } catch { /* ignore */ }
  1428. }
  1429. }, SLOW_KEEPALIVE_MS);
  1430. }
  1431. }
  1432. proc.stdout.on("data", (data) => {
  1433. clearTimeout(firstByteTimer); // CLI is alive — cancel slow-spawn warning
  1434. resetHeartbeat();
  1435. slowDownKeepalive(); // CLI is producing output, switch to slow keepalive
  1436. buffer += data.toString();
  1437. const lines = buffer.split("\n");
  1438. buffer = lines.pop() || "";
  1439. for (const line of lines) {
  1440. if (!line.trim()) continue;
  1441. try {
  1442. const ev = JSON.parse(line);
  1443. const canWrite = !res.writableEnded;
  1444. // stream_event: incremental deltas from --include-partial-messages
  1445. if (ev.type === "stream_event" && ev.event?.type === "content_block_delta") {
  1446. const text = ev.event.delta?.text;
  1447. if (text) {
  1448. if (canWrite) res.write(sseChunk(reqId, text));
  1449. outputChars += text.length;
  1450. sentContent = true;
  1451. sseBroadcast("chunk", { reqId, model, source, text, tokens: outputChars });
  1452. }
  1453. } else if (ev.type === "stream_event" && ev.event?.type === "message_delta") {
  1454. const usage = ev.event.usage;
  1455. if (usage) {
  1456. // Total input = non-cached + cache-created + cache-read
  1457. const totalInput = (usage.input_tokens || 0)
  1458. + (usage.cache_creation_input_tokens || 0)
  1459. + (usage.cache_read_input_tokens || 0);
  1460. reqTokens = { input: totalInput, output: usage.output_tokens || 0 };
  1461. }
  1462. } else if (ev.type === "assistant" && ev.message?.content) {
  1463. if (!sentContent) {
  1464. for (const b of ev.message.content) {
  1465. if (b.type === "text" && b.text) {
  1466. if (canWrite) res.write(sseChunk(reqId, b.text));
  1467. outputChars += b.text.length;
  1468. sentContent = true;
  1469. sseBroadcast("chunk", { reqId, model, source, text: b.text, tokens: outputChars });
  1470. }
  1471. }
  1472. }
  1473. } else if (ev.type === "content_block_delta" && ev.delta?.text) {
  1474. if (canWrite) res.write(sseChunk(reqId, ev.delta.text));
  1475. outputChars += ev.delta.text.length;
  1476. sentContent = true;
  1477. sseBroadcast("chunk", { reqId, model, source, text: ev.delta.text, tokens: outputChars });
  1478. } else if (ev.type === "result" && ev.result && !sentContent) {
  1479. if (canWrite) res.write(sseChunk(reqId, ev.result));
  1480. sentContent = true;
  1481. }
  1482. // Capture token usage (include cached tokens in total input)
  1483. const usage = ev.usage || ev.message?.usage;
  1484. if (usage) {
  1485. const totalInput = (usage.input_tokens || usage.prompt_tokens || 0)
  1486. + (usage.cache_creation_input_tokens || 0)
  1487. + (usage.cache_read_input_tokens || 0);
  1488. reqTokens = {
  1489. input: totalInput,
  1490. output: usage.output_tokens || usage.completion_tokens || 0,
  1491. };
  1492. }
  1493. } catch { /* non-JSON line, skip */ }
  1494. }
  1495. if (proc.pid) {
  1496. const liveInput = reqTokens.input > 0 ? reqTokens.input : inputEstimate;
  1497. const liveOutput = reqTokens.output > 0 ? reqTokens.output : Math.ceil(outputChars / 4);
  1498. registry.touch(proc.pid, { liveInputTokens: liveInput, liveOutputTokens: liveOutput });
  1499. }
  1500. });
  1501. proc.on("close", (code) => {
  1502. clearTimeout(firstByteTimer);
  1503. clearTimeout(heartbeatTimer);
  1504. clearTimeout(execTimer);
  1505. clearInterval(keepaliveInterval);
  1506. workerRelease(worker.name);
  1507. // Quick-fail auto-retry: if worker failed fast with no content, try another
  1508. const elapsed = Date.now() - proc._spawnedAt;
  1509. if (code !== 0 && !sentContent && elapsed < QUICK_FAIL_MS && retryCount < MAX_RETRIES) {
  1510. // Find an untried router, or any alternate
  1511. const untried = _workerPool.find(
  1512. (w) => !triedRouters.has(w.name) && !_workerHealth.get(w.name)?.limited
  1513. );
  1514. const alt = untried || getAlternateWorker(worker.name);
  1515. if (alt) {
  1516. retryCount++;
  1517. console.log(`[${ts()}] STREAM_RETRY reqId=${reqId} failedRouter=${worker.name} code=${code} elapsed=${elapsed}ms -> retrying on ${alt.name} (attempt ${retryCount}/${MAX_RETRIES})`);
  1518. recordWorkerError(worker.name, "stream_retry", `code=${code} elapsed=${elapsed}ms`);
  1519. eventLog.push("retry", { reqId, model, source, failedWorker: worker.name, retryWorker: alt.name, code, elapsed, retryCount });
  1520. pipeStream(alt, true);
  1521. return; // don't finalize response — retry will handle it
  1522. }
  1523. }
  1524. release();
  1525. if (code !== 0) {
  1526. const diag = stderrBuf.trim() || buffer.trim().slice(0, 200) || "(no output)";
  1527. console.log(`[${ts()}] CLI_EXIT reqId=${reqId} code=${code} sent=${sentContent} router=${worker.name} stderr=${diag.slice(0, 300)}`);
  1528. const errCat = code === 143 ? "cli_killed" : "cli_crash";
  1529. recordWorkerError(worker.name, errCat, `code=${code} ${diag.slice(0, 100)}`);
  1530. }
  1531. if (proc._workerName && (isRateLimitError(code, stderrBuf) || (!sentContent && isRateLimitError(code, buffer)))) {
  1532. markWorkerLimited(proc._workerName);
  1533. }
  1534. // Flush remaining buffer
  1535. const canWrite = !res.writableEnded;
  1536. if (buffer.trim()) {
  1537. try {
  1538. const ev = JSON.parse(buffer);
  1539. if (ev.type === "assistant" && ev.message?.content) {
  1540. for (const b of ev.message.content) {
  1541. if (b.type === "text" && b.text && canWrite) res.write(sseChunk(reqId, b.text));
  1542. }
  1543. } else if (ev.type === "result" && ev.result && !sentContent && canWrite) {
  1544. res.write(sseChunk(reqId, ev.result));
  1545. }
  1546. const usage = ev.usage || ev.message?.usage;
  1547. if (usage) {
  1548. const totalInput = (usage.input_tokens || usage.prompt_tokens || 0)
  1549. + (usage.cache_creation_input_tokens || 0)
  1550. + (usage.cache_read_input_tokens || 0);
  1551. reqTokens = { input: totalInput, output: usage.output_tokens || usage.completion_tokens || 0 };
  1552. }
  1553. } catch { /* ignore */ }
  1554. }
  1555. const finalInput = reqTokens.input > 0 ? reqTokens.input : inputEstimate;
  1556. const finalOutput = reqTokens.output > 0 ? reqTokens.output : Math.ceil(outputChars / 4);
  1557. tokenTracker.record(reqId, model, finalInput, finalOutput);
  1558. eventLog.push("complete", {
  1559. reqId, mode: "stream", model, source, exitCode: code,
  1560. inputTokens: finalInput, outputTokens: finalOutput,
  1561. });
  1562. sseBroadcast("complete", { reqId, model, source, exitCode: code, inputTokens: finalInput, outputTokens: finalOutput });
  1563. if (code !== 0 && !sentContent) {
  1564. // All CLI routers failed — fall back to API endpoint
  1565. console.log(`[${ts()}] ALL_CLI_FAILED reqId=${reqId} retryCount=${retryCount} -> falling back to ${FALLBACK_API.name}`);
  1566. streamFromFallbackApi(originalMessages, model, reqId, source, res);
  1567. return; // fallback handles res.end()
  1568. }
  1569. if (canWrite) {
  1570. res.write(sseChunk(reqId, null, "stop"));
  1571. res.write("data: [DONE]\n\n");
  1572. res.end();
  1573. }
  1574. });
  1575. proc.on("error", (err) => {
  1576. clearTimeout(firstByteTimer);
  1577. clearTimeout(heartbeatTimer);
  1578. clearTimeout(execTimer);
  1579. clearInterval(keepaliveInterval);
  1580. workerRelease(worker.name);
  1581. // Quick-fail auto-retry on spawn error too
  1582. if (!sentContent && retryCount < MAX_RETRIES) {
  1583. const untried = _workerPool.find(
  1584. (w) => !triedRouters.has(w.name) && !_workerHealth.get(w.name)?.limited
  1585. );
  1586. const alt = untried || getAlternateWorker(worker.name);
  1587. if (alt) {
  1588. retryCount++;
  1589. console.log(`[${ts()}] STREAM_RETRY reqId=${reqId} failedRouter=${worker.name} error=${err.message} -> retrying on ${alt.name} (attempt ${retryCount}/${MAX_RETRIES})`);
  1590. pipeStream(alt, true);
  1591. return;
  1592. }
  1593. }
  1594. release();
  1595. // All CLI routers errored — fall back to API endpoint
  1596. console.log(`[${ts()}] ALL_CLI_FAILED reqId=${reqId} error=${err.message} -> falling back to ${FALLBACK_API.name}`);
  1597. streamFromFallbackApi(originalMessages, model, reqId, source, res);
  1598. });
  1599. }
  1600. // Start the stream pipeline (first attempt, no retry flag)
  1601. pipeStream(null, false);
  1602. } else {
  1603. try {
  1604. const result = await runCli(prompt, model, systemPrompt, reqId, source, sessionKey);
  1605. release();
  1606. // Estimate tokens for sync: prompt chars/4 for input, result chars/4 for output
  1607. const syncInputTokens = Math.ceil(prompt.length / 4);
  1608. const syncOutputTokens = Math.ceil(result.length / 4);
  1609. tokenTracker.record(reqId, model, syncInputTokens, syncOutputTokens);
  1610. eventLog.push("complete", {
  1611. reqId, mode: "sync", model, source,
  1612. inputTokens: syncInputTokens, outputTokens: syncOutputTokens,
  1613. });
  1614. sendJson(res, 200, completionResponse(reqId, result, model));
  1615. } catch (err) {
  1616. release();
  1617. eventLog.push("error", { reqId, mode: "sync", model, source, error: err.message });
  1618. console.error(`[${ts()}] ERROR src=${source} ${err.message}`);
  1619. sendJson(res, 500, { error: { message: err.message, type: "internal_error" } });
  1620. }
  1621. }
  1622. }
  1623. // ============================================================
  1624. // Other endpoints
  1625. // ============================================================
  1626. function handleModels(req, res) {
  1627. const models = Object.keys(MODEL_MAP).map((id) => ({
  1628. id: `claude-code/${id}`,
  1629. object: "model",
  1630. created: Math.floor(Date.now() / 1000),
  1631. owned_by: "claude-code-proxy",
  1632. }));
  1633. sendJson(res, 200, { object: "list", data: models });
  1634. }
  1635. function handleHealth(req, res) {
  1636. const qs = queue.getStats();
  1637. const rs = registry.getStats();
  1638. const workers = _workerPool.map((w) => {
  1639. const h = _workerHealth.get(w.name);
  1640. return {
  1641. name: w.name,
  1642. bin: w.bin,
  1643. limited: h.limited,
  1644. limitedAt: h.limitedAt || null,
  1645. limitedAgoSec: h.limited ? Math.round((Date.now() - h.limitedAt) / 1000) : null,
  1646. };
  1647. });
  1648. sendJson(res, 200, {
  1649. status: "ok",
  1650. version: "0.5.1",
  1651. claude_bin: CLAUDE_BIN,
  1652. port: PORT,
  1653. redis: redis ? { connected: redis.isReady() } : { connected: false },
  1654. cliRouters: workers,
  1655. primaryRouter: PRIMARY_WORKER,
  1656. queue: { active: qs.active, queued: qs.totalQueued, max: qs.maxConcurrent, sources: qs.sourceCount },
  1657. processes: { tracked: rs.total, byMode: rs.byMode, liveTokens: rs.liveTokens },
  1658. tokens: tokenTracker.getTotals(),
  1659. sessionAffinity: sessionAffinity.getStats(),
  1660. workerStats,
  1661. });
  1662. }
  1663. function handleMetrics(req, res) {
  1664. const qs = queue.getStats();
  1665. const rs = registry.getStats();
  1666. const workers = _workerPool.map((w) => {
  1667. const h = _workerHealth.get(w.name);
  1668. return { name: w.name, limited: h.limited, limitedAt: h.limitedAt || null };
  1669. });
  1670. sendJson(res, 200, {
  1671. rateLimits: RATE_LIMITS,
  1672. rateUsage: rateLimiter.stats(),
  1673. tokens: tokenTracker.getStats(),
  1674. cliRouters: workers,
  1675. loadBalanceMode: _loadBalanceMode,
  1676. primaryRouter: PRIMARY_WORKER,
  1677. queue: qs,
  1678. processes: rs,
  1679. config: {
  1680. useCliAgents: USE_CLI_AGENTS,
  1681. workerCount: _workerPool.length,
  1682. loadBalanceAlgorithm: "least-connections",
  1683. maxConcurrent: MAX_CONCURRENT,
  1684. maxQueueTotal: MAX_QUEUE_TOTAL,
  1685. maxQueuePerSource: MAX_QUEUE_PER_SOURCE,
  1686. queueTimeoutMs: QUEUE_TIMEOUT_MS,
  1687. heartbeatByModel: HEARTBEAT_BY_MODEL,
  1688. defaultHeartbeatMs: DEFAULT_HEARTBEAT_MS,
  1689. streamTimeoutMs: STREAM_TIMEOUT_MS,
  1690. syncTimeoutMs: SYNC_TIMEOUT_MS,
  1691. maxProcessAgeMs: MAX_PROCESS_AGE_MS,
  1692. maxIdleMs: MAX_IDLE_MS,
  1693. reaperIntervalMs: REAPER_INTERVAL_MS,
  1694. sessionAffinityTtlMs: 5 * 60 * 1000,
  1695. sseKeepaliveMs: 30_000,
  1696. maxRetries: MAX_RETRIES,
  1697. retryBaseMs: RETRY_BASE_MS,
  1698. },
  1699. sessionAffinity: sessionAffinity.getStats(),
  1700. workerStats,
  1701. activeConnections: Object.fromEntries(_activeConns),
  1702. });
  1703. }
  1704. function handleZombies(req, res) {
  1705. const zombies = registry.getZombies();
  1706. const qs = queue.getStats();
  1707. sendJson(res, 200, {
  1708. processes: registry.getAll(),
  1709. zombies,
  1710. stats: registry.getStats(),
  1711. activeLeases: qs.activeLeases,
  1712. });
  1713. }
  1714. async function handleKillZombie(req, res) {
  1715. const body = await readBody(req);
  1716. let parsed;
  1717. try {
  1718. parsed = JSON.parse(body);
  1719. } catch {
  1720. return sendJson(res, 400, { error: { message: "Invalid JSON body" } });
  1721. }
  1722. const { pid } = parsed;
  1723. if (!pid) return sendJson(res, 400, { error: { message: "pid required" } });
  1724. const result = registry.kill(Number(pid));
  1725. eventLog.push("kill", { pid: Number(pid), manual: true });
  1726. sendJson(res, 200, { result });
  1727. }
  1728. function handleEvents(req, res, url) {
  1729. const sinceId = parseInt(url.searchParams.get("since_id") || "0", 10);
  1730. const limit = parseInt(url.searchParams.get("limit") || "50", 10);
  1731. const type = url.searchParams.get("type") || null;
  1732. const events = eventLog.getRecent({ sinceId, limit, type });
  1733. sendJson(res, 200, { events, counts: eventLog.getCounts() });
  1734. }
  1735. function handleMetricsHistory(req, res, url) {
  1736. const window = url.searchParams.get("window") || "1h";
  1737. const validWindows = ["1h", "6h", "1d", "7d"];
  1738. if (!validWindows.includes(window)) {
  1739. return sendJson(res, 400, { error: { message: `Invalid window. Use: ${validWindows.join(", ")}` } });
  1740. }
  1741. const points = metricsStore.query(window);
  1742. sendJson(res, 200, { window, points, count: points.length, bufferSize: metricsStore.getBufferSize() });
  1743. }
  1744. async function handlePortal(req, res) {
  1745. try {
  1746. const html = await readFile(join(__dirname, "portal.html"), "utf-8");
  1747. res.writeHead(200, { "content-type": "text/html; charset=utf-8" });
  1748. res.end(html);
  1749. } catch (err) {
  1750. sendJson(res, 500, { error: { message: "Portal file not found: " + err.message } });
  1751. }
  1752. }
  1753. async function handleProxyDashboard(req, res) {
  1754. try {
  1755. const html = await readFile(join(__dirname, "dashboard.html"), "utf-8");
  1756. res.writeHead(200, { "content-type": "text/html; charset=utf-8" });
  1757. res.end(html);
  1758. } catch (err) {
  1759. sendJson(res, 500, { error: { message: "Dashboard file not found: " + err.message } });
  1760. }
  1761. }
  1762. // ============================================================
  1763. // Utilities
  1764. // ============================================================
  1765. function ts() {
  1766. return new Date().toISOString();
  1767. }
  1768. function sendJson(res, status, body, extraHeaders = {}) {
  1769. res.writeHead(status, { "content-type": "application/json", ...extraHeaders });
  1770. res.end(JSON.stringify(body));
  1771. }
  1772. function readBody(req) {
  1773. return new Promise((resolve) => {
  1774. const chunks = [];
  1775. req.on("data", (c) => chunks.push(c));
  1776. req.on("end", () => resolve(Buffer.concat(chunks).toString()));
  1777. });
  1778. }
  1779. function handleSSEStream(req, res) {
  1780. res.writeHead(200, {
  1781. "content-type": "text/event-stream",
  1782. "cache-control": "no-cache",
  1783. "connection": "keep-alive",
  1784. });
  1785. res.write("event: connected\ndata: {}\n\n");
  1786. sseClients = new Set([...sseClients, res]);
  1787. console.log(`[${ts()}] SSE_CLIENT connected (${sseClients.size} total)`);
  1788. req.on("close", () => {
  1789. sseClients = new Set([...sseClients].filter((c) => c !== res));
  1790. console.log(`[${ts()}] SSE_CLIENT disconnected (${sseClients.size} total)`);
  1791. });
  1792. }
  1793. // ============================================================
  1794. // HTTP Server
  1795. // ============================================================
  1796. const server = createServer(async (req, res) => {
  1797. const url = new URL(req.url, `http://0.0.0.0:${PORT}`);
  1798. res.setHeader("access-control-allow-origin", "*");
  1799. res.setHeader("access-control-allow-methods", "GET, POST, OPTIONS");
  1800. res.setHeader("access-control-allow-headers", "content-type, authorization, x-api-key, x-openclaw-source, x-source, x-session-id");
  1801. if (req.method === "OPTIONS") {
  1802. res.writeHead(204);
  1803. res.end();
  1804. return;
  1805. }
  1806. // Auth check (skip for health, dashboard, events)
  1807. const noAuthPaths = ["/health", "/dashboard", "/dashboard/", "/dashboard/proxy", "/dashboard/proxy/", "/events", "/metrics/history", "/stream"];
  1808. if (!noAuthPaths.includes(url.pathname) && !authenticate(req)) {
  1809. return sendJson(res, 401, { error: { message: "Unauthorized" } });
  1810. }
  1811. try {
  1812. if (url.pathname === "/v1/chat/completions" && req.method === "POST") {
  1813. await handleCompletions(req, res);
  1814. } else if (url.pathname === "/v1/models" && req.method === "GET") {
  1815. handleModels(req, res);
  1816. } else if (url.pathname === "/health" && req.method === "GET") {
  1817. handleHealth(req, res);
  1818. } else if (url.pathname === "/metrics" && req.method === "GET") {
  1819. handleMetrics(req, res);
  1820. } else if (url.pathname === "/rate-limits" && req.method === "GET") {
  1821. handleMetrics(req, res); // backward compat
  1822. } else if (url.pathname === "/zombies" && req.method === "GET") {
  1823. handleZombies(req, res);
  1824. } else if (url.pathname === "/zombies" && req.method === "POST") {
  1825. await handleKillZombie(req, res);
  1826. } else if (url.pathname === "/events" && req.method === "GET") {
  1827. handleEvents(req, res, url);
  1828. } else if (url.pathname === "/metrics/history" && req.method === "GET") {
  1829. handleMetricsHistory(req, res, url);
  1830. } else if (url.pathname === "/stream" && req.method === "GET") {
  1831. handleSSEStream(req, res);
  1832. } else if (url.pathname === "/dashboard/proxy" || url.pathname === "/dashboard/proxy/") {
  1833. await handleProxyDashboard(req, res);
  1834. } else if (url.pathname === "/dashboard" || url.pathname === "/dashboard/") {
  1835. await handlePortal(req, res);
  1836. } else {
  1837. sendJson(res, 404, { error: { message: "Not found" } });
  1838. }
  1839. } catch (err) {
  1840. console.error(`[${ts()}] UNHANDLED ${err.message}`);
  1841. sendJson(res, 500, { error: { message: "Internal server error" } });
  1842. }
  1843. });
  1844. // ============================================================
  1845. // Graceful shutdown
  1846. // ============================================================
  1847. function shutdown(signal) {
  1848. console.log(`[${ts()}] SHUTDOWN signal=${signal}`);
  1849. // Kill all tracked processes
  1850. const allProcs = registry.getAll();
  1851. for (const entry of allProcs) {
  1852. console.log(`[${ts()}] SHUTDOWN_KILL pid=${entry.pid} reqId=${entry.requestId} model=${entry.model}`);
  1853. registry.kill(entry.pid);
  1854. }
  1855. registry.destroy();
  1856. queue.destroy();
  1857. metricsStore.destroy();
  1858. sessionAffinity.shutdown();
  1859. // Close Redis connection
  1860. if (redis) {
  1861. redis.quit().catch(() => {});
  1862. }
  1863. server.close(() => {
  1864. console.log(`[${ts()}] SHUTDOWN complete`);
  1865. process.exit(0);
  1866. });
  1867. // Force exit after 5s if graceful close hangs
  1868. setTimeout(() => {
  1869. console.error(`[${ts()}] SHUTDOWN forced after 5s timeout`);
  1870. process.exit(1);
  1871. }, 5000).unref();
  1872. }
  1873. process.on("SIGTERM", () => shutdown("SIGTERM"));
  1874. process.on("SIGINT", () => shutdown("SIGINT"));
  1875. // ============================================================
  1876. // Start server
  1877. // ============================================================
  1878. server.listen(PORT, "0.0.0.0", async () => {
  1879. // Wait for all persistent stores to load from Redis / files
  1880. await Promise.all([
  1881. metricsStore.ready,
  1882. tokenTracker.ready,
  1883. eventLog.ready,
  1884. registry.ready,
  1885. ]);
  1886. eventLog.push("startup", { version: "0.5.0", port: PORT, redis: !!redis });
  1887. // Seed token tracker from all raw metrics snapshots (sums across server restarts)
  1888. const rawSnapshots = metricsStore.getRawBuffer();
  1889. tokenTracker.seedFromHistory(rawSnapshots);
  1890. metricsStore.startSampler(gatherMetricsSnapshot);
  1891. console.log(`Claude Code Proxy v0.5.1`);
  1892. console.log(`Listening on http://0.0.0.0:${PORT}`);
  1893. console.log(`CLI Routers: ${_workerPool.map((w) => `obj${w.name}=${w.bin}`).join(" | ")} | Primary: obj${PRIMARY_WORKER}`);
  1894. console.log(`Auth token: ${AUTH_TOKEN === "local-proxy" ? "(open - no auth)" : "(enabled)"}`);
  1895. console.log(`Concurrent: ${MAX_CONCURRENT} | Queue: ${MAX_QUEUE_TOTAL} total, ${MAX_QUEUE_PER_SOURCE}/source`);
  1896. console.log(`Queue timeout: ${QUEUE_TIMEOUT_MS}ms`);
  1897. console.log(`Models: ${Object.keys(MODEL_MAP).join(", ")}`);
  1898. console.log(`Rate limits: sonnet ${RATE_LIMITS.sonnet.requestsPerMin}/min, opus ${RATE_LIMITS.opus.requestsPerMin}/min, haiku ${RATE_LIMITS.haiku.requestsPerMin}/min`);
  1899. console.log(`Reaper: age=${MAX_PROCESS_AGE_MS}ms idle=${MAX_IDLE_MS}ms interval=${REAPER_INTERVAL_MS}ms`);
  1900. console.log(`Timeouts: sync=${SYNC_TIMEOUT_MS}ms stream=${STREAM_TIMEOUT_MS}ms`);
  1901. console.log(`Heartbeat: opus=${HEARTBEAT_BY_MODEL.opus}ms sonnet=${HEARTBEAT_BY_MODEL.sonnet}ms haiku=${HEARTBEAT_BY_MODEL.haiku}ms`);
  1902. console.log(`Metrics store: ${metricsStore.getBufferSize()} historical snapshots loaded`);
  1903. });

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.