deepswarm
Use when running parallel AI workers for any long-running or multi-turn batch API task. Auto-calculates optimal workers + stagger. Supports tiered delegation (V4 Pro orchestrator → V4 Flash workers). 99.95% API success rate at scale.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/amanning3390--deepswarm)HTML
<a href="/listings/amanning3390--deepswarm"><img src="/api/listings/amanning3390--deepswarm/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
hermes skills tap add amanning3390/deepswarm
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 191 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: deepswarm
- description: Use when running parallel AI workers for any long-running or multi-turn batch API task. Auto-calculates optimal workers + stagger. Supports tiered delegation (V4 Pro orchestrator → V4 Flash workers). 99.95% API success rate at scale.
- version: 2.0.0
- author: Hermes Agent
- license: MIT
- metadata:
- hermes:
- tags: [orchestration, parallel, workers, api-generation, swarm, batch, multi-turn, delegation, tiered-models]
- related_skills: [tmux-agent-orchestrator, hermes-agent]
- ---
- # DeepSwarm — Task-Agnostic Parallel Worker Orchestration
- Spawn N parallel API workers for **any** long-running or multi-turn batch task. Auto-calculates optimal worker count and stagger delay. Supports tiered model delegation: orchestrator plans with a frontier model (V4 Pro), workers execute with a cheaper model (V4 Flash).
- ## Overview
- DeepSwarm 2.0 generalizes the proven orchestration pattern from the 19,331-trace generation project to any batch API task. You define a task — translations, reasoning traces, code reviews, summarization — and DeepSwarm parallelizes it across optimal workers with the right stagger for your API.
- **The core insight:** API rate limits are a function of simultaneous connections, not total volume. Auto-calculated stagger + worker count = 99.95% success.
- ## When to Use
- - **Any** batch API task: generation, translation, summarization, extraction, classification
- - Long-running individual calls (30s+) that benefit from parallelization
- - Multi-turn tasks where each worker loops through conversation turns
- - Cost optimization via tiered delegation (orchestrator ≠ worker model)
- - Crash-resilient batch processing (checkpointed, idempotent)
- **Don't use for:**
- - Quick calls under 10s (overhead not worth it — just loop)
- - Tasks requiring inter-worker coordination (use `delegate_task`)
- - Real-time interactive sessions (use tmux-agent-orchestrator)
- ## Quick Start
- ```bash
- # Install
- hermes skills tap add amanning3390/deepswarm
- # Define your task (task.yaml)
- # Generate seeds
- python3 scripts/seed.py --task task.yaml
- # Launch — auto-optimizes workers, stagger, model routing
- python3 scripts/swarm.py --task task.yaml --total 1000
- # Filter — repair JSON, validate structure, apply length thresholds
- python3 scripts/filter.py --input-dir output/ --output clean.jsonl --errors errors.jsonl
- ```
- ## Task Definition (task.yaml)
- ```yaml
- # What to do
- task_type: generation # generation | translation | summarization | custom
- prompt_template: |
- You are an AI assistant. {{seed}}
- # Model routing (tiered delegation)
- orchestrator_model: deepseek-v4-pro # Plans, monitors, handles errors
- worker_model: deepseek-v4-flash # Executes batches (cheaper!)
- worker_api_base: https://api.deepseek.com/v1/chat/completions
- worker_max_tokens: 4096
- # Execution control
- multi_turn: true # Workers loop through conversation turns
- max_turns: 20 # Max turns per worker conversation
- seeds_file: seeds.jsonl # Pre-generated task seeds
- # Worker optimization (auto-calculated if omitted)
- workers: auto # auto | N
- stagger: auto # auto | seconds
- batch_size: auto # auto | tasks per worker
- # Output
- output_dir: output/
- output_format: jsonl # jsonl | json | parquet
- checkpoint_every: 10 # Save progress every N tasks
- # Optional: custom worker logic
- worker_script: custom_worker.py # Override default worker behavior
- ```
- ## Tiered Model Delegation
- Orchestrator (V4 Pro) and workers (V4 Flash) can use different models:
- ```
- User Task → V4 Pro (plans, monitors)
- ├─ V4 Flash Worker 0 → API → output/
- ├─ V4 Flash Worker 1 → API → output/
- ├─ V4 Flash Worker 2 → API → output/
- └─ ...
- ```
- **Why tiered delegation matters:**
- - V4 Pro costs ~3× V4 Flash per token
- - Orchestrator only plans + monitors (few calls)
- - Workers make thousands of calls — use the cheapest model that works
- - Typical savings: 60-70% vs using V4 Pro for everything
- **When to use same model for both:**
- - Task quality requires frontier reasoning at every step
- - Worker model doesn't support the required format
- - Budget allows it and quality is paramount
- ## Auto-Optimization
- When `workers: auto` and `stagger: auto`:
- 1. DeepSwarm runs a single calibration call to measure call duration
- 2. Calculates optimal workers: `min(8, floor(rate_limit / call_duration))`
- 3. Sets stagger: `call_duration / workers × 2`
- 4. Adjusts batch_size: `total / workers`
- **Calibration table (pre-computed):**
- | Call Duration | Workers | Stagger | Success | Throughput |
- |--------------|---------|---------|---------|------------|
- | <10s | 16 | 1s | 99.9% | ~5,760/hr |
- | 10-30s | 12 | 2s | 99.9% | ~1,440/hr |
- | 30-60s | 8 | 5s | 99.95% | ~440/hr |
- | 60-90s | 6 | 10s | 99.9% | ~240/hr |
- | >90s | 4 | 15s | 99.9% | ~96/hr |
- ## Multi-Turn Task Support
- For tasks requiring conversation loops (generation, debugging, interactive work):
- ```
- Worker loop:
- for each seed:
- messages = [system_prompt, user_task]
- for turn in range(max_turns):
- response = api_call(messages, model=worker_model)
- messages.append({"role": "assistant", "content": response})
- if task_complete(response):
- break
- if needs_tool_call(response):
- messages.append(simulate_tool_response(response))
- ```
- Each turn is an independent API call. Multi-turn tasks benefit most from parallelization because per-task latency is high.
- ## Task-Agnostic Worker Design
- The worker (`worker.py`) accepts a YAML task definition and executes any pipeline:
- ```python
- def run_task(seed, config):
- messages = build_messages(seed, config)
- for turn in range(config["max_turns"]):
- response = call_api(messages, config)
- if is_complete(response, config):
- return finish(response, messages)
- if needs_continuation(response, config):
- messages = append_turn(messages, response, config)
- return messages
- ```
- Built-in task types:
- - `generation` — Generate content from seed (the trace generation pattern)
- - `translation` — Translate each seed text
- - `summarization` — Summarize each seed document
- - `classification` — Classify each seed input
- - `custom` — Uses `worker_script` for completely custom logic
- ## Common Pitfalls
- 1. **`workers: auto` choosing too many.** If calibration call was fast but actual calls are slow, override manually.
- 2. **Forgetting to stagger between workers.** Even 2 workers at same millisecond can trigger rate limits on slow APIs.
- 3. **Mixing models without checking format compatibility.** Worker model must support the same prompt format as orchestrator.
- 4. **Not checkpointing.** Worker dies at 120/125 = lost work. Checkpoint every 10.
- 5. **Shell `&` without `wait`.** Without `wait`, shell exits early and kills child workers.
- 6. **Using V4 Pro for workers when V4 Flash would work.** Check quality on 10-sample test before committing to expensive model.
- 7. **Not deleting error outputs before restarting.** Error files consume indices and inflate disk.
- ## Verification Checklist
- - [ ] Task YAML has valid model names and API base URL
- - [ ] API key exported for both orchestrator and worker models
- - [ ] `workers: auto` or manual count ≤ 8 per batch
- - [ ] `stagger: auto` or manual ≥ call_duration / workers × 2
- - [ ] Worker model tested on 5-sample run before full batch
- - [ ] Output directory exists and is writable
- - [ ] Seeds file exists with correct format
- - [ ] Checkpointing enabled for runs >100 tasks
- - [ ] Orchestrator model is V4 Pro (or equivalent frontier) for planning
- - [ ] Worker model is V4 Flash (or cheapest model that handles the task)
README.md · README · 93 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # DeepSwarm 2.0 — Task-Agnostic Parallel Worker Orchestration
- Spawn N parallel API workers for **any** batch task. Auto-optimizes worker count + stagger. Tiered model delegation: orchestrator plans (V4 Pro) → workers execute (V4 Flash). 99.95% API success rate.
- ## Install
- ```bash
- hermes skills tap add amanning3390/deepswarm
- ```
- ## Quick Start
- ```bash
- # 1. Define your task
- cp task.yaml my_task.yaml
- # Edit: prompt_template, worker_model, max_tokens
- # 2. Generate seeds
- python3 scripts/seed.py --count 1000 --template "Generate {{seed}}" > seeds.jsonl
- # 3. Launch (auto-optimizes everything)
- export DEEPSEEK_API_KEY=sk-...
- python3 scripts/swarm.py --task my_task.yaml --total 1000
- ```
- ## Tiered Delegation
- ```yaml
- orchestrator_model: deepseek-v4-pro # Plans (few calls, frontier quality)
- worker_model: deepseek-v4-flash # Executes (many calls, cheaper)
- ```
- V4 Pro costs ~3× V4 Flash per token. For batch tasks with thousands of calls, tiered delegation saves 60-70%.
- ## Auto-Optimization
- | Call Duration | Workers | Stagger | Success | Throughput |
- |--------------|---------|---------|---------|------------|
- | <10s | 16 | 1s | 99.9% | ~5,760/hr |
- | 10-30s | 12 | 2s | 99.9% | ~1,440/hr |
- | 30-60s | 8 | 5s | 99.95% | ~440/hr |
- | 60-90s | 6 | 10s | 99.9% | ~240/hr |
- Omit `workers` and `stagger` in task.yaml — DeepSwarm runs a calibration call and picks optimal values.
- ## Task Types
- Built-in: `generation`, `translation`, `summarization`, `classification`, `custom`
- For multi-turn tasks (tool calling, conversation loops):
- ```yaml
- multi_turn: true
- max_turns: 20
- ```
- ## Files
- ```
- deepswarm/
- ├── SKILL.md # Hermes skill definition
- ├── README.md
- ├── task.yaml # Sample task config
- ├── architecture.html # Pipeline diagram
- ├── scripts/
- │ ├── seed.py # Seed generator (simple + rich templates)
- │ ├── swarm.py # Orchestrator (auto-optimize + launch)
- │ ├── worker.py # Task-agnostic batch processor
- │ └── filter.py # Quality filter with JSON repair
- ├── templates/
- │ └── prompts.py # v2 prompt templates
- └── references/
- ├── api-rate-limits.md
- └── generation-patterns.md
- ```
- ## Post-Processing
- After generation, filter the raw output to remove malformed traces:
- ```bash
- python3 scripts/filter.py --input-dir output/ --output clean.jsonl --errors errors.jsonl
- ```
- The filter applies 3 passes: JSON repair (fixes 17% API error rate), structural validation
- (think blocks, tag balance), and length thresholds. Typically lifts pass rate from ~28% to ~62%.
- ## Provenance
- Built from the [DeepSeek Hermes Reasoning Traces](https://huggingface.co/datasets/ning423/deepseek-hermes-reasoning-traces) project:
- - 19,331 traces · 192K tool calls
- - 96 workers · 31K API calls
- - 99.95% success rate
- - 8 workers + 5s stagger = the magic formula
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.