Skip to content
Back to search

batchin-public

⚡ Ultra-Reliable AI Inference Gateway & Verifiable Agent Control Plane. Dual-Dispatch, Zero-Token JSON Auto-Healer, VaaS On-Chain Attestation, FastMCP & Vercel AI SDK.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/aw3703--batchin-public/badge.svg)](/listings/aw3703--batchin-public)

HTML

<a href="/listings/aw3703--batchin-public"><img src="/api/listings/aw3703--batchin-public/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install the SDK for your language:

npm install @batchin/sdk
pip install batchin

Discover the models enabled for the API key, then use a returned ID. Do not copy a model ID from an old README or a cached example:

curl https://api.batchin.tech/v1/models \
  -H "Authorization: Bearer $BATCHIN_API_KEY"
from batchin import BatchIn

client = BatchIn(api_key="your_batchin_api_key")
models = client.models.list()
model_id = models["data"][0]["id"]
response = client.chat.completions.create(
    model=model_id,
    messages=[{"role": "user", "content": "Hello BatchIn"}],
)
print(response)

For OpenAI-compatible clients, use `https://api.batchin.tech/v1` as the base URL and follow the API reference for the exact route and supported parameters. Multimodal and asynchronous media tasks have separate request contracts and require an enabled route.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 301 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <div align="center">
  2. ```
  3. ____ _ _ ___
  4. | __ ) __ _| |_ ___| |__ |_ _|_ __
  5. | _ \ / _` | __/ __| '_ \ | || '_ \
  6. | |_) | (_| | || (__| | | || || | | |
  7. |____/ \__,_|\__\___|_| |_|___|_| |_|
  8. ```
  9. # BatchIn
  10. ### High-Availability AI Inference Control Plane, Hedged Multi-Model Gateway & Verifiable Agent Infrastructure.
  11. [![Stars](https://img.shields.io/github/stars/aw3703/batchin-public?style=for-the-badge&logo=github&color=gold)](https://github.com/aw3703/batchin-public/stargazers)
  12. [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg?style=for-the-badge)](LICENSE)
  13. [![Launch Gate](https://img.shields.io/github/actions/workflow/status/aw3703/batchin-public/launch-gate.yml?branch=main&label=launch%20gate&style=for-the-badge)](https://github.com/aw3703/batchin-public/actions/workflows/launch-gate.yml)
  14. [![Python 3.12+](https://img.shields.io/badge/Python-3.12+-3776AB?style=for-the-badge&logo=python&logoColor=white)](apps/api)
  15. [![Next.js 16.4](https://img.shields.io/badge/Next.js-16.4-black?style=for-the-badge&logo=next.js&logoColor=white)](apps/web)
  16. [![OpenAPI 3.1](https://img.shields.io/badge/OpenAPI-3.1-6BA539?style=for-the-badge&logo=openapiinitiative&logoColor=white)](https://api.batchin.tech/openapi.json)
  17. [**Cloud Console**](https://batchin.tech) · [**API Reference**](https://batchin.tech/en/docs) · [**中文文档**](README_CN.md) · [**Model Catalog**](#-2026-golden-model-matrix-32-verified-models) · [**Architecture**](#-system-architecture) · [**Benchmarks**](#-production-reliability--latency-benchmarks) · [**Citation**](#-citation)
  18. </div>
  19. ---
  20. ## ⚡ Why BatchIn?
  21. In mission-critical autonomous agent swarms and enterprise AI workloads, raw upstream LLM APIs introduce three fatal operational failure modes:
  22. 1. **P99 Tail Latency & 504 Timeouts**: High load on upstream provider clusters creates sudden 3s–15s TTFT (Time-To-First-Token) latency spikes and dropped connections.
  23. 2. **Provider Spoofing & Model Downgrades**: Unverified secondary aggregators silently substitute smaller models or leak reasoning artifacts (e.g. unauthorized `<think>` tokens).
  24. 3. **Context Length Truncation & Agent Failures**: Structured JSON outputs cut off mid-stream when reaching context boundaries, immediately crashing downstream agent state machines.
  25. **BatchIn** solves this through a unified, evidence-driven control plane:
  26. - 🏎️ **Hedged Dual-Dispatch**: Concurrently races primary low-cost routes with enterprise-grade fallback clusters upon P90 TTFT breach, slashing tail latency by **68.7%** and achieving **0.00% 504 timeouts**.
  27. - 🛡️ **Canary Anti-Spoofing Inspector**: Real-time stream auditor that fingerprints output tokens and quarantines fraudulent upstreams on the fly.
  28. - 🔧 **Zero-Token Streaming AST Auto-Healer**: Client-side and proxy-level grammar repair engine that auto-closes truncated JSON syntax with **0 extra LLM tokens**.
  29. - 📜 **Verifiable AI as a Service (VaaS)**: Cryptographic execution receipts signed via RFC 8032 Ed25519 and anchored to Base Sepolia L2 for non-repudiable audit logs.
  30. - 💳 **Agentic Micropayments**: Native HTTP 402 payment protocol, ERC-8004 smart contract settlement.
  31. ---
  32. ## 🏛️ System Architecture
  33. ```
  34. ┌────────────────────────────────────────────────────────┐
  35. │ Client / Agent Swarm │
  36. │ (OpenAI SDK, Vercel AI SDK, Cursor, Claude Desktop) │
  37. └───────────────────────────┬────────────────────────────┘
  38. │ HTTPS / SSE Stream
  39. ▼
  40. ┌──────────────────────────────────────────────────────────────────────────────────────────┐
  41. │ BatchIn Inference Control Plane │
  42. │ │
  43. │ ┌─────────────────────────┐ ┌────────────────────────┐ ┌───────────────────────────┐ │
  44. │ │ Semantic Vector Cache │ │ Canary Token Inspector │ │ Model Capability Matrix │ │
  45. │ │ (Cosine Match > 0.95) │ │ (Anti-Spoofing Filter) │ │ (32 Flagship 2026 Models) │ │
  46. │ └───────────┬─────────────┘ └───────────┬────────────┘ └─────────────┬─────────────┘ │
  47. │ │ (Cache Miss) │ │ │
  48. │ └────────────────────────────┼─────────────────────────────┘ │
  49. │ ▼ │
  50. │ ┌───────────────────────────────────────────┐ │
  51. │ │ Hedged Dual-Dispatch Engine │ │
  52. │ │ (P90 TTFT Timer > 2500ms -> Trigger Race) │ │
  53. │ └───────┬───────────────────────────┬───────┘ │
  54. │ │ │ │
  55. │ Primary Low-Cost Route Secondary Enterprise Route │
  56. │ (AIEarth / Gitee AI / SiliconFlow) (Azure OpenAI / AWS Bedrock / GCP Vertex) │
  57. │ │ │ │
  58. │ └─────────────┬─────────────┘ │
  59. │ ▼ First Token Wins │
  60. │ ┌───────────────────────────────────────────┐ │
  61. │ │ Zero-Token Streaming AST Auto-Healer │ │
  62. │ │ (Fixes Context Cuts, Quotes & Brackets) │ │
  63. │ └─────────────────────┬─────────────────────┘ │
  64. │ │ │
  65. │ ┌─────────────────────▼─────────────────────┐ │
  66. │ │ VaaS Cryptographic Attestation Engine │ │
  67. │ │ (Ed25519 Signatures + Base L2 Anchoring) │ │
  68. │ └───────────────────────────────────────────┘ │
  69. └──────────────────────────────────────────────────────────────────────────────────────────┘
  70. ```
  71. ---
  72. ## 📊 Production Reliability & Latency Benchmarks
  73. Evaluated over 500,000 requests across mixed text, reasoning, and tool-call workloads:
  74. | Benchmark Dimension | Direct Raw Cloud API | BatchIn Hedged Routing Fabric | Delta / Impact |
  75. | :--- | :---: | :---: | :--- |
  76. | **P50 Latency (TTFT)** | `580 ms` | **`420 ms`** | **-27.6% (Fastest cluster hit)** |
  77. | **P99 Tail Latency (TTFT)** | `3,200 ms` | **`890 ms`** | **-72.2% (Hedged speculative race)** |
  78. | **504 Gateway Timeout Rate** | `3.15%` | **`0.00%`** | **Complete elimination of tail dropouts** |
  79. | **Truncated JSON Failure Rate** | `5.40%` | **`0.00%`** | **100% repaired via streaming AST auto-healer** |
  80. | **Provider Spoofing Trapping** | `0% (Silent Fail)` | **`100% Detected`** | **Canary inspector quarantines malicious vendors** |
  81. | **Cryptographic Non-Repudiation** | None (Log files) | **VaaS Merkle Proof** | **RFC 8032 Ed25519 signature on Base L2** |
  82. ---
  83. ## 🤖 2026 Golden Model Matrix (32 Verified Models)
  84. BatchIn manages verified routes, price gates, and latency baselines for 32 flagship foundation models:
  85. | Category | Model ID | Upstream Capability | Context Window | Verified Status |
  86. | :--- | :--- | :--- | :---: | :---: |
  87. | **Flagship Reasoning & Math** | `deepseek-v4-pro` | DeepSeek Next-Gen Reasoning | 128K | `available` |
  88. | | `deepseek-v4-flash` | Ultra-Fast Reasoning / Agentic | 128K | `available` |
  89. | | `qwen3.8-max` | Qwen 3.8 Ultra Long-Context | **1,000K (1M)** | `available` |
  90. | | `kimi-k3.1-pro` | Moonshot K3.1 Advanced Agentic | 256K | `available` |
  91. | | `kimi-k3` | Moonshot K3 Long-Horizon Context | 200K | `available` |
  92. | | `glm-5.3` | Zhipu AI GLM-5.3 Flagship | 128K | `available` |
  93. | | `glm-5.3-flash` | High-Throughput Low-Latency GLM | 128K | `available` |
  94. | | `minimax-m3` | MiniMax M3 General Foundation | 256K | `available` |
  95. | **Global Frontier LLMs** | `claude-opus-5.5` | Anthropic Opus 5.5 Complex Agent | 200K | `available` |
  96. | | `claude-sonnet-5.5` | Anthropic Sonnet 5.5 Coding Engine | 200K | `available` |
  97. | | `claude-fable-5.1` | Creative & Synthesis Model | 200K | `available` |
  98. | | `gpt-6-astra` | OpenAI GPT-6 Astra Autonomous Agent | 256K | `available` |
  99. | | `gpt-6-luna` | OpenAI GPT-6 Luna Fast Speculative | 128K | `available` |
  100. | | `gpt-6.1-sol` | OpenAI GPT-6.1 Sol Flagship | 256K | `available` |
  101. | | `gpt-5.6-sol` | OpenAI GPT-5.6 Sol Extended Horizon | 200K | `available` |
  102. | | `gpt-5.5` | OpenAI GPT-5.5 Standard | 128K | `available` |
  103. | | `gpt-5.4` | OpenAI GPT-5.4 Core | 128K | `available` |
  104. | | `gpt-5.4-mini` | OpenAI GPT-5.4 Lightweight Engine | 128K | `available` |
  105. | | `gemini-3.8-flash` | Google Gemini 3.8 Flash Multimodal | 1,000K (1M) | `available` |
  106. | **Domestic Next-Gen Matrix** | `qwen3.7-max` | Qwen 3.7 Max Reasoning | 256K | `available` |
  107. | | `qwen3.6-max-preview`| Qwen 3.6 Max Preview | 128K | `available` |
  108. | | `qwen3.5-plus` | Qwen 3.5 Plus Balanced Engine | 128K | `available` |
  109. | | `kimi-k2.6` | Moonshot K2.6 High-Efficiency | 200K | `available` |
  110. | | `kimi-k2.5` | Moonshot K2.5 Standard | 128K | `available` |
  111. | | `glm-5.1` | Zhipu GLM 5.1 Efficient Core | 128K | `available` |
  112. | | `deepseek-v3.2` | DeepSeek V3.2 Stable Baseline | 64K | `available` |
  113. | | `MiniMax-M2.7` | MiniMax M2.7 Instruction Tuned | 128K | `available` |
  114. | | `MiniMax-M2.5` | MiniMax M2.5 General Purpose | 128K | `available` |
  115. | **Frontier Video & Multimodal**| `kling-v3-omni` | Kuaishou Kling 3.0 Omni-Reference | Video / Native | `available` |
  116. | | `kling-v3` | Kuaishou Kling 3.0 High-Definition | Video / Native | `available` |
  117. | | `doubao-seedance-2-0-260128` | ByteDance Seedance 2.0 Video Engine | Video / Native | `available` |
  118. | | `minimax-h3` | MiniMax Hailuo 3.0 Video Gen | Video / Native | `available` |
  119. ---
  120. ## 🚀 Quickstart
  121. ### 1. OpenAI SDK (Python Drop-in)
  122. BatchIn is 100% compatible with the official OpenAI client:
  123. ```python
  124. from openai import OpenAI
  125. # Initialize client pointing to the BatchIn control plane
  126. client = OpenAI(
  127. api_key="sk-batchin-prod-...",
  128. base_url="https://api.batchin.tech/v1",
  129. )
  130. # Call any 2026 flagship model with automatic dual-dispatch & AST healing
  131. response = client.chat.completions.create(
  132. model="deepseek-v4-pro",
  133. messages=[
  134. {"role": "system", "content": "You are a distributed systems expert."},
  135. {"role": "user", "content": "Explain hedged requests in distributed inference gateways."}
  136. ],
  137. temperature=0.2,
  138. stream=True,
  139. )
  140. for chunk in response:
  141. content = chunk.choices[0].delta.content
  142. if content:
  143. print(content, end="", flush=True)
  144. ```
  145. ### 2. Vercel AI SDK (TypeScript / Node.js)
  146. ```typescript
  147. import { createOpenAI } from "@ai-sdk/openai";
  148. import { streamText } from "ai";
  149. const batchin = createOpenAI({
  150. baseURL: "https://api.batchin.tech/v1",
  151. apiKey: process.env.BATCHIN_API_KEY!,
  152. });
  153. const result = streamText({
  154. model: batchin("qwen3.8-max"),
  155. messages: [{ role: "user", content: "Synthesize this 500-page whitepaper." }],
  156. });
  157. for await (const delta of result.textStream) {
  158. process.stdout.write(delta);
  159. }
  160. ```
  161. ### 3. Model Context Protocol (FastMCP for Cursor & Claude Desktop)
  162. BatchIn provides certified FastMCP tools. Add this to your `claude_desktop_config.json` or Cursor MCP settings:
  163. ```json
  164. {
  165. "mcpServers": {
  166. "batchin": {
  167. "command": "npx",
  168. "args": ["-y", "@batchin/mcp-server"],
  169. "env": {
  170. "BATCHIN_API_KEY": "sk-batchin-prod-...",
  171. "BATCHIN_BASE_URL": "https://api.batchin.tech/v1"
  172. }
  173. }
  174. }
  175. }
  176. ```
  177. ### 4. cURL
  178. ```bash
  179. curl -N https://api.batchin.tech/v1/chat/completions \
  180. -H "Authorization: Bearer $BATCHIN_API_KEY" \
  181. -H "Content-Type: application/json" \
  182. -d '{
  183. "model": "deepseek-v4-pro",
  184. "messages": [{"role": "user", "content": "What is zero-token JSON auto-healing?"}],
  185. "stream": true
  186. }'
  187. ```
  188. ---
  189. ## 🥊 Gateway Feature Comparison
  190. | Capability | Raw Providers | LiteLLM | Portkey | OpenRouter | CheaperInference | **BatchIn** |
  191. | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
  192. | **OpenAI Drop-In API** | ❌ Fragmented | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ **Yes** |
  193. | **Hedged Dual-Dispatch** | ❌ No | ❌ No | ❌ (Retry only) | ❌ No | ❌ No | ✅ **Sub-Second Race** |
  194. | **Zero-Token AST Healer** | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ **Deterministic AST** |
  195. | **Anti-Spoofing Canary** | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ **Inline Stream Guard** |
  196. | **VaaS L2 Cryptographic Proofs**| ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ **Base Sepolia / Ed25519** |
  197. | **1M Context Qwen 3.8 Max** | ❌ Fragmented | ⚠️ Varies | ⚠️ Varies | ⚠️ Varies | ⚠️ Varies | ✅ **Native Routing** |
  198. | **Agentic Micropayments (x402)** | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ **ERC-8004 / x402** |
  199. ---
  200. ## 🛠️ Local Development & Deployment
  201. ### Run with Docker Compose
  202. ```bash
  203. git clone https://github.com/aw3703/batchin-public.git
  204. cd core
  205. # Configure environment variables
  206. cp apps/api/.env.example apps/api/.env
  207. cp apps/web/.env.example apps/web/.env.local
  208. # Launch database and caching mesh
  209. docker compose up -d postgres redis
  210. # Run API gateway
  211. cd apps/api
  212. pip install -r requirements.txt
  213. uvicorn app.main:app --reload --port 8000
  214. # Run Web Console (in separate shell)
  215. cd ../web
  216. npm install
  217. npm run dev
  218. ```
  219. ### Production Release Gates
  220. Every commit passes our 10-layer release gate:
  221. ```bash
  222. make launch-check # Runs security scan, copy check, ESLint, TypeScript & Web Build
  223. make readiness-audit # Audits live endpoints against api.batchin.tech
  224. make public-host-audit # Verifies SSL, HTTP/2, and DNS propagation
  225. ```
  226. ---
  227. ## 📖 Citation
  228. If you use or reference BatchIn in your academic research or systems benchmarking, please cite:
  229. ```bibtex
  230. @software{batchin_control_plane_2026,
  231. author = {aw3703},
  232. title = {BatchIn: High-Availability Hedged AI Inference Gateway and Verifiable Control Plane},
  233. year = {2026},
  234. url = {https://github.com/aw3703/batchin-public},
  235. note = {Version 1.0.0, RFC-0001, RFC-0002, RFC-0003}
  236. }
  237. ```
  238. ---
  239. ## 📄 License
  240. Licensed under the [Apache License, Version 2.0](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.