Skip to content
Back to search

deepswarm

SkillWorks with Claude CodeGitHub

Use when running parallel AI workers for any long-running or multi-turn batch API task. Auto-calculates optimal workers + stagger. Supports tiered delegation (V4 Pro orchestrator → V4 Flash workers). 99.95% API success rate at scale.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/amanning3390--deepswarm/badge.svg)](/listings/amanning3390--deepswarm)

HTML

<a href="/listings/amanning3390--deepswarm"><img src="/api/listings/amanning3390--deepswarm/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

hermes skills tap add amanning3390/deepswarm

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 191 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: deepswarm
  3. description: Use when running parallel AI workers for any long-running or multi-turn batch API task. Auto-calculates optimal workers + stagger. Supports tiered delegation (V4 Pro orchestrator → V4 Flash workers). 99.95% API success rate at scale.
  4. version: 2.0.0
  5. author: Hermes Agent
  6. license: MIT
  7. metadata:
  8. hermes:
  9. tags: [orchestration, parallel, workers, api-generation, swarm, batch, multi-turn, delegation, tiered-models]
  10. related_skills: [tmux-agent-orchestrator, hermes-agent]
  11. ---
  12. # DeepSwarm — Task-Agnostic Parallel Worker Orchestration
  13. Spawn N parallel API workers for **any** long-running or multi-turn batch task. Auto-calculates optimal worker count and stagger delay. Supports tiered model delegation: orchestrator plans with a frontier model (V4 Pro), workers execute with a cheaper model (V4 Flash).
  14. ## Overview
  15. DeepSwarm 2.0 generalizes the proven orchestration pattern from the 19,331-trace generation project to any batch API task. You define a task — translations, reasoning traces, code reviews, summarization — and DeepSwarm parallelizes it across optimal workers with the right stagger for your API.
  16. **The core insight:** API rate limits are a function of simultaneous connections, not total volume. Auto-calculated stagger + worker count = 99.95% success.
  17. ## When to Use
  18. - **Any** batch API task: generation, translation, summarization, extraction, classification
  19. - Long-running individual calls (30s+) that benefit from parallelization
  20. - Multi-turn tasks where each worker loops through conversation turns
  21. - Cost optimization via tiered delegation (orchestrator ≠ worker model)
  22. - Crash-resilient batch processing (checkpointed, idempotent)
  23. **Don't use for:**
  24. - Quick calls under 10s (overhead not worth it — just loop)
  25. - Tasks requiring inter-worker coordination (use `delegate_task`)
  26. - Real-time interactive sessions (use tmux-agent-orchestrator)
  27. ## Quick Start
  28. ```bash
  29. # Install
  30. hermes skills tap add amanning3390/deepswarm
  31. # Define your task (task.yaml)
  32. # Generate seeds
  33. python3 scripts/seed.py --task task.yaml
  34. # Launch — auto-optimizes workers, stagger, model routing
  35. python3 scripts/swarm.py --task task.yaml --total 1000
  36. # Filter — repair JSON, validate structure, apply length thresholds
  37. python3 scripts/filter.py --input-dir output/ --output clean.jsonl --errors errors.jsonl
  38. ```
  39. ## Task Definition (task.yaml)
  40. ```yaml
  41. # What to do
  42. task_type: generation # generation | translation | summarization | custom
  43. prompt_template: |
  44. You are an AI assistant. {{seed}}
  45. # Model routing (tiered delegation)
  46. orchestrator_model: deepseek-v4-pro # Plans, monitors, handles errors
  47. worker_model: deepseek-v4-flash # Executes batches (cheaper!)
  48. worker_api_base: https://api.deepseek.com/v1/chat/completions
  49. worker_max_tokens: 4096
  50. # Execution control
  51. multi_turn: true # Workers loop through conversation turns
  52. max_turns: 20 # Max turns per worker conversation
  53. seeds_file: seeds.jsonl # Pre-generated task seeds
  54. # Worker optimization (auto-calculated if omitted)
  55. workers: auto # auto | N
  56. stagger: auto # auto | seconds
  57. batch_size: auto # auto | tasks per worker
  58. # Output
  59. output_dir: output/
  60. output_format: jsonl # jsonl | json | parquet
  61. checkpoint_every: 10 # Save progress every N tasks
  62. # Optional: custom worker logic
  63. worker_script: custom_worker.py # Override default worker behavior
  64. ```
  65. ## Tiered Model Delegation
  66. Orchestrator (V4 Pro) and workers (V4 Flash) can use different models:
  67. ```
  68. User Task → V4 Pro (plans, monitors)
  69. ├─ V4 Flash Worker 0 → API → output/
  70. ├─ V4 Flash Worker 1 → API → output/
  71. ├─ V4 Flash Worker 2 → API → output/
  72. └─ ...
  73. ```
  74. **Why tiered delegation matters:**
  75. - V4 Pro costs ~3× V4 Flash per token
  76. - Orchestrator only plans + monitors (few calls)
  77. - Workers make thousands of calls — use the cheapest model that works
  78. - Typical savings: 60-70% vs using V4 Pro for everything
  79. **When to use same model for both:**
  80. - Task quality requires frontier reasoning at every step
  81. - Worker model doesn't support the required format
  82. - Budget allows it and quality is paramount
  83. ## Auto-Optimization
  84. When `workers: auto` and `stagger: auto`:
  85. 1. DeepSwarm runs a single calibration call to measure call duration
  86. 2. Calculates optimal workers: `min(8, floor(rate_limit / call_duration))`
  87. 3. Sets stagger: `call_duration / workers × 2`
  88. 4. Adjusts batch_size: `total / workers`
  89. **Calibration table (pre-computed):**
  90. | Call Duration | Workers | Stagger | Success | Throughput |
  91. |--------------|---------|---------|---------|------------|
  92. | <10s | 16 | 1s | 99.9% | ~5,760/hr |
  93. | 10-30s | 12 | 2s | 99.9% | ~1,440/hr |
  94. | 30-60s | 8 | 5s | 99.95% | ~440/hr |
  95. | 60-90s | 6 | 10s | 99.9% | ~240/hr |
  96. | >90s | 4 | 15s | 99.9% | ~96/hr |
  97. ## Multi-Turn Task Support
  98. For tasks requiring conversation loops (generation, debugging, interactive work):
  99. ```
  100. Worker loop:
  101. for each seed:
  102. messages = [system_prompt, user_task]
  103. for turn in range(max_turns):
  104. response = api_call(messages, model=worker_model)
  105. messages.append({"role": "assistant", "content": response})
  106. if task_complete(response):
  107. break
  108. if needs_tool_call(response):
  109. messages.append(simulate_tool_response(response))
  110. ```
  111. Each turn is an independent API call. Multi-turn tasks benefit most from parallelization because per-task latency is high.
  112. ## Task-Agnostic Worker Design
  113. The worker (`worker.py`) accepts a YAML task definition and executes any pipeline:
  114. ```python
  115. def run_task(seed, config):
  116. messages = build_messages(seed, config)
  117. for turn in range(config["max_turns"]):
  118. response = call_api(messages, config)
  119. if is_complete(response, config):
  120. return finish(response, messages)
  121. if needs_continuation(response, config):
  122. messages = append_turn(messages, response, config)
  123. return messages
  124. ```
  125. Built-in task types:
  126. - `generation` — Generate content from seed (the trace generation pattern)
  127. - `translation` — Translate each seed text
  128. - `summarization` — Summarize each seed document
  129. - `classification` — Classify each seed input
  130. - `custom` — Uses `worker_script` for completely custom logic
  131. ## Common Pitfalls
  132. 1. **`workers: auto` choosing too many.** If calibration call was fast but actual calls are slow, override manually.
  133. 2. **Forgetting to stagger between workers.** Even 2 workers at same millisecond can trigger rate limits on slow APIs.
  134. 3. **Mixing models without checking format compatibility.** Worker model must support the same prompt format as orchestrator.
  135. 4. **Not checkpointing.** Worker dies at 120/125 = lost work. Checkpoint every 10.
  136. 5. **Shell `&` without `wait`.** Without `wait`, shell exits early and kills child workers.
  137. 6. **Using V4 Pro for workers when V4 Flash would work.** Check quality on 10-sample test before committing to expensive model.
  138. 7. **Not deleting error outputs before restarting.** Error files consume indices and inflate disk.
  139. ## Verification Checklist
  140. - [ ] Task YAML has valid model names and API base URL
  141. - [ ] API key exported for both orchestrator and worker models
  142. - [ ] `workers: auto` or manual count ≤ 8 per batch
  143. - [ ] `stagger: auto` or manual ≥ call_duration / workers × 2
  144. - [ ] Worker model tested on 5-sample run before full batch
  145. - [ ] Output directory exists and is writable
  146. - [ ] Seeds file exists with correct format
  147. - [ ] Checkpointing enabled for runs >100 tasks
  148. - [ ] Orchestrator model is V4 Pro (or equivalent frontier) for planning
  149. - [ ] Worker model is V4 Flash (or cheapest model that handles the task)
README.md · README · 93 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # DeepSwarm 2.0 — Task-Agnostic Parallel Worker Orchestration
  2. Spawn N parallel API workers for **any** batch task. Auto-optimizes worker count + stagger. Tiered model delegation: orchestrator plans (V4 Pro) → workers execute (V4 Flash). 99.95% API success rate.
  3. ## Install
  4. ```bash
  5. hermes skills tap add amanning3390/deepswarm
  6. ```
  7. ## Quick Start
  8. ```bash
  9. # 1. Define your task
  10. cp task.yaml my_task.yaml
  11. # Edit: prompt_template, worker_model, max_tokens
  12. # 2. Generate seeds
  13. python3 scripts/seed.py --count 1000 --template "Generate {{seed}}" > seeds.jsonl
  14. # 3. Launch (auto-optimizes everything)
  15. export DEEPSEEK_API_KEY=sk-...
  16. python3 scripts/swarm.py --task my_task.yaml --total 1000
  17. ```
  18. ## Tiered Delegation
  19. ```yaml
  20. orchestrator_model: deepseek-v4-pro # Plans (few calls, frontier quality)
  21. worker_model: deepseek-v4-flash # Executes (many calls, cheaper)
  22. ```
  23. V4 Pro costs ~3× V4 Flash per token. For batch tasks with thousands of calls, tiered delegation saves 60-70%.
  24. ## Auto-Optimization
  25. | Call Duration | Workers | Stagger | Success | Throughput |
  26. |--------------|---------|---------|---------|------------|
  27. | <10s | 16 | 1s | 99.9% | ~5,760/hr |
  28. | 10-30s | 12 | 2s | 99.9% | ~1,440/hr |
  29. | 30-60s | 8 | 5s | 99.95% | ~440/hr |
  30. | 60-90s | 6 | 10s | 99.9% | ~240/hr |
  31. Omit `workers` and `stagger` in task.yaml — DeepSwarm runs a calibration call and picks optimal values.
  32. ## Task Types
  33. Built-in: `generation`, `translation`, `summarization`, `classification`, `custom`
  34. For multi-turn tasks (tool calling, conversation loops):
  35. ```yaml
  36. multi_turn: true
  37. max_turns: 20
  38. ```
  39. ## Files
  40. ```
  41. deepswarm/
  42. ├── SKILL.md # Hermes skill definition
  43. ├── README.md
  44. ├── task.yaml # Sample task config
  45. ├── architecture.html # Pipeline diagram
  46. ├── scripts/
  47. │ ├── seed.py # Seed generator (simple + rich templates)
  48. │ ├── swarm.py # Orchestrator (auto-optimize + launch)
  49. │ ├── worker.py # Task-agnostic batch processor
  50. │ └── filter.py # Quality filter with JSON repair
  51. ├── templates/
  52. │ └── prompts.py # v2 prompt templates
  53. └── references/
  54. ├── api-rate-limits.md
  55. └── generation-patterns.md
  56. ```
  57. ## Post-Processing
  58. After generation, filter the raw output to remove malformed traces:
  59. ```bash
  60. python3 scripts/filter.py --input-dir output/ --output clean.jsonl --errors errors.jsonl
  61. ```
  62. The filter applies 3 passes: JSON repair (fixes 17% API error rate), structural validation
  63. (think blocks, tag balance), and length thresholds. Typically lifts pass rate from ~28% to ~62%.
  64. ## Provenance
  65. Built from the [DeepSeek Hermes Reasoning Traces](https://huggingface.co/datasets/ning423/deepseek-hermes-reasoning-traces) project:
  66. - 19,331 traces · 192K tool calls
  67. - 96 workers · 31K API calls
  68. - 99.95% success rate
  69. - 8 workers + 5s stagger = the magic formula

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.