Skip to content
Back to search

ContextPilot

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/EfficientContext/ContextPilot with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/EfficientContext/ContextPilot'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/efficientcontext--contextpilot/badge.svg)](/listings/efficientcontext--contextpilot)

HTML

<a href="/listings/efficientcontext--contextpilot"><img src="/api/listings/efficientcontext--contextpilot/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

**Requirements:** Python >= 3.10

---

OpenClaw

**Option A: Native Plugin** (recommended — zero external dependencies)

openclaw plugins install @contextpilot-ai/contextpilot

Then enable in `~/.openclaw/openclaw.json`:

{
  "plugins": {
    "slots": { "contextEngine": "contextpilot" },
    "entries": { "contextpilot": { "enabled": true } }
  }
}

Restart OpenClaw. Done — ContextPilot runs in-process, no proxy needed.

**Option B: HTTP Proxy** (for self-hosted models or custom backends)

pip install contextpilot

python -m contextpilot.server.http_server \
  --port 8765 --infer-api-url http://localhost:30000   # SGLang
  # or: --infer-api-url https://api.anthropic.com      # Anthropic
  # or: --infer-api-url https://api.openai.com         # OpenAI

Then set OpenClaw's base URL to `http://localhost:8765/v1`. See the [full OpenClaw integration guide](docs/guides/openclaw.md) for details.

---

Hermes Agent

Native plugin — zero Hermes source changes, zero external dependencies:

hermes plugins install EfficientContext/ContextPilot
hermes plugins   # select: Context Engine → contextpilot

Typical savings: **40–50% input tokens** on agentic workloads with repeated file reads. See the [Hermes integration guide](docs/guides/hermes.md) for details.

---

vLLM / SGLang

**From PyPI** — the vLLM and SGLang hooks are installed automatically:

pip install contextpilot          # CPU index computation
pip install "contextpilot[gpu]"   # GPU index computation (CUDA 12.x)

**From source** — run `install_hook` manually after install, since editable installs do not copy the `.pth` file to site-packages:

git clone https://github.com/EfficientContext/ContextPilot.git
cd ContextPilot
pip install -e .                  # CPU
pip install -e ".[gpu]"           # GPU (CUDA 12.x)
python -m contextpilot.install_hook   # one-time: enables automatic vLLM / SGLang integration

The `install_hook` step writes a `.pth` file into your site-packages so the vLLM and SGLang hooks load automatically at Python startup — no code changes required. To uninstall: `python -m contextpilot.install_hook --remove`.

---

Mac / Apple Silicon — llama.cpp

**From PyPI:**

pip install contextpilot
xcode-select --install    # one-time: provides clang++ to compile the native hook

**From source:**

git clone https://github.com/EfficientContext/ContextPilot.git
cd ContextPilot
pip install -e .
xcode-select --install    # one-time: provides clang++ to compile the native hook

> **Why `xcode-select`?** The llama.cpp integration uses a small C++ shared library injected into `llama-server` via `DYLD_INSERT_LIBRARIES`. It is compiled automatically on first use and requires `clang++` from Xcode Command Line Tools.

---

More [detailed installation instructions](https://efficientcontext.github.io/contextpilot-docs/getting_started/installation) are available in the docs.

Docker images are also available for both all-in-one and standalone deployment. See the [Docker guide](https://efficientcontext.github.io/contextpilot-docs/getting_started/docker).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 289 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <img src="assets/about.png" alt="ContextPilot Logo" width="600"/>
  3. <h2><strong>ContextPilot: Fast Long-Context Inference via Context Reuse</strong></h2>
  4. [![Python](https://img.shields.io/badge/python-≥3.10-blue)](https://www.python.org/)
  5. [![PyPI](https://img.shields.io/pypi/v/contextpilot)](https://pypi.org/project/contextpilot/)
  6. [![License](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE)
  7. <p><strong>4–12× cache hits | 1.5–3× faster prefill | ~36% token savings</strong> across vLLM, SGLang, RAG, AI Agents, and more.</p>
  8. </div>
  9. --------------------------------------------------------------------------------
  10. | [**Documentation**](https://efficientcontext.github.io/contextpilot-docs/) | [**Examples**](examples/) | [**Benchmarks**](https://efficientcontext.github.io/contextpilot-docs/reference/benchmarks) | [**Docker**](https://efficientcontext.github.io/contextpilot-docs/getting_started/docker) | [**Paper**](https://arxiv.org/abs/2511.03475) |
  11. ## News
  12. - [2026/09] Supports **multimodal contexts**: reorder retrieved video frames / images for prefix-cache sharing while preserving accuracy — [guide](docs/guides/multimodal.md) | [video RAG](examples/video_rag/)
  13. - [2026/05] Supports [Hermes Agent](https://github.com/NousResearch/hermes-agent) as a native context engine plugin — [guide](docs/guides/hermes.md)
  14. - [2026/03] Supports [OpenClaw](https://openclaw.ai) — [guide](docs/guides/openclaw.md) | [benchmark](docs/benchmarks/openclaw.md)
  15. - [2026/03] Supports cloud APIs (OpenAI, Anthropic, MiniMax) — [cache sync](docs/guides/cache_sync.md)
  16. - [2026/03] ContextPilot now can run on **macOS / Apple Silicon** via [llama.cpp](docs/guides/mac_llama_cpp.md).
  17. - [2026/02] ContextPilot v0.3.2 released, supporting [PageIndex](https://github.com/VectifyAI/PageIndex) and [Mem0](https://github.com/mem0ai/mem0).
  18. - [2026/01] ContextPilot has been accepted to MLSys 2026 🎉! See you in Bellevue, WA, USA.
  19. ## About
  20. Long-context workloads (RAG, memory chat, tool-augmented agents) prepend many context blocks. Across requests, these blocks often overlap but get reordered or duplicated, changing token prefixes and triggering cache misses and redundant KV recomputation. Common examples include (1) Trending Topic QA, (2) Closed-Domain Long-Context QA, (3) Batched Long-Context Inference, (4) multi-turn conversations with long-term memory and many more.
  21. ContextPilot sits between context assembly and inference to maximize prefix reuse and remove duplicates:
  22. 1. **Higher throughput & cache hits** — boosts prefill throughput and prefix cache hit ratio via context reuse.
  23. 2. **Drop-in solutions** — supports [OpenClaw](https://openclaw.ai) ([guide](docs/guides/openclaw.md)), [Hermes Agent](https://github.com/NousResearch/hermes-agent) ([guide](docs/guides/hermes.md)), [PageIndex](https://github.com/VectifyAI/PageIndex), [Mem0](https://github.com/mem0ai/mem0), [LMCache](https://github.com/LMCache/LMCache), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [llama.cpp](docs/guides/mac_llama_cpp.md), and cloud APIs (OpenAI, Anthropic).
  24. 3. **No compromise in reasoning quality** — can even improve with extremely long contexts.
  25. 4. **Text and multimodal** — reorders retrieved video frames / images as well as text blocks, raising the cache hit ratio while keeping answer accuracy ([guide](docs/guides/multimodal.md)).
  26. 5. **Widely tested** — validated across diverse RAG and agentic workloads.
  27. It maintains a **Context Index** of cached content, then per request applies **Reorder** (align shared blocks into a common prefix) and/or **Deduplicate** (replace repeats with reference hints), plus **cache-aware scheduling** to maximize prefix sharing. The optimized prompt is sent via the OpenAI-compatible API; `POST /evict` keeps the index synced when KV cache is reclaimed. See its design overview below.
  28. <div align="center">
  29. <img src="assets/system_description.jpg" alt="ContextPilot Architecture" width="600"/>
  30. </div>
  31. > For more design details, see [Paper](https://arxiv.org/abs/2511.03475) and [Documentation](https://efficientcontext.github.io/contextpilot-docs/).
  32. ## Performance at a Glance
  33. **OpenClaw Agent on RTX 5090** — 60 enterprise document analysis tasks ([claw-tasks](https://github.com/EfficientContext/ClawTasks)), Qwen3-4B-Instruct via SGLang. [Full results →](docs/benchmarks/openclaw.md)
  34. | Metric | OpenClaw + SGLang | + ContextPilot | Δ |
  35. |--------|-------------------|----------------|---|
  36. | Prompt tokens / request (avg) | 45,771 | 33,622 | **-26.5%** |
  37. | Prompt tokens / request (P99) | 92,785 | 51,581 | **-44.4%** |
  38. | Wall time (avg) | 26.1s | 20.8s | **-20.4%** |
  39. | Wall time (P99) | 68.8s | 50.4s | **-26.6%** |
  40. | Accuracy | 245/245 | 245/245 | ✓ |
  41. **Qwen3-4B on 1×A6000** — multi-turn memory chat with [Mem0](https://github.com/mem0ai/mem0) on the [LoCoMo](https://github.com/snap-research/locomo) benchmark.
  42. | Context Size | Method | TTFT (s) | LLM Judge |
  43. |--------------|--------|----------|-----------|
  44. | 5 (long context memory) | SGLang | 0.1051 | 0.418 |
  45. | | **SGLang + ContextPilot** | **0.0548** | 0.414 |
  46. | 100 memories | SGLang | 0.1012 | 0.437 |
  47. | | **SGLang + ContextPilot** | **0.0554** | 0.420 |
  48. >ContextPilot results in mem0 table are without context annotation — an optional feature that adds original importance ranking to reordered context blocks, which can further improve answer quality (see [Paper](https://arxiv.org/abs/2511.03475)).
  49. **Llama-3.2-1B on Apple Silicon** — MultihopRAG with llama.cpp, no GPU server required.
  50. | Device | Method | Avg Latency (ms) |
  51. |--------|--------|-----------------|
  52. | M3 (MacBook Air, 16 GB) | llama.cpp | 3,315 |
  53. | | **llama.cpp + ContextPilot** | **1,378** |
  54. | M5 (MacBook Air, 32 GB) | llama.cpp | 2,157 |
  55. | | **llama.cpp + ContextPilot** | **911** |
  56. Settings: `Llama-3.2-1B-Instruct-Q4_K_M.gguf`, Metal offload (`-ngl 99`), `--cache-reuse 256`, `--parallel 4`, context 32768 tokens. See the [Mac + llama.cpp guide](docs/guides/mac_llama_cpp.md).
  57. We also evaluated on academic RAG (Qwen3-32B, 4×A6000) and production MoE inference (DeepSeek-R1-671B, 16×H20) — see [RAG benchmarks](docs/benchmarks/rag.md) and [paper](https://arxiv.org/abs/2511.03475).
  58. ## Installation
  59. **Requirements:** Python >= 3.10
  60. ---
  61. ### OpenClaw
  62. **Option A: Native Plugin** (recommended — zero external dependencies)
  63. ```bash
  64. openclaw plugins install @contextpilot-ai/contextpilot
  65. ```
  66. Then enable in `~/.openclaw/openclaw.json`:
  67. ```json
  68. {
  69. "plugins": {
  70. "slots": { "contextEngine": "contextpilot" },
  71. "entries": { "contextpilot": { "enabled": true } }
  72. }
  73. }
  74. ```
  75. Restart OpenClaw. Done — ContextPilot runs in-process, no proxy needed.
  76. **Option B: HTTP Proxy** (for self-hosted models or custom backends)
  77. ```bash
  78. pip install contextpilot
  79. python -m contextpilot.server.http_server \
  80. --port 8765 --infer-api-url http://localhost:30000 # SGLang
  81. # or: --infer-api-url https://api.anthropic.com # Anthropic
  82. # or: --infer-api-url https://api.openai.com # OpenAI
  83. ```
  84. Then set OpenClaw's base URL to `http://localhost:8765/v1`. See the [full OpenClaw integration guide](docs/guides/openclaw.md) for details.
  85. ---
  86. ### Hermes Agent
  87. Native plugin — zero Hermes source changes, zero external dependencies:
  88. ```bash
  89. hermes plugins install EfficientContext/ContextPilot
  90. hermes plugins # select: Context Engine → contextpilot
  91. ```
  92. Typical savings: **40–50% input tokens** on agentic workloads with repeated file reads. See the [Hermes integration guide](docs/guides/hermes.md) for details.
  93. ---
  94. ### vLLM / SGLang
  95. **From PyPI** — the vLLM and SGLang hooks are installed automatically:
  96. ```bash
  97. pip install contextpilot # CPU index computation
  98. pip install "contextpilot[gpu]" # GPU index computation (CUDA 12.x)
  99. ```
  100. **From source** — run `install_hook` manually after install, since editable installs do not copy the `.pth` file to site-packages:
  101. ```bash
  102. git clone https://github.com/EfficientContext/ContextPilot.git
  103. cd ContextPilot
  104. pip install -e . # CPU
  105. pip install -e ".[gpu]" # GPU (CUDA 12.x)
  106. python -m contextpilot.install_hook # one-time: enables automatic vLLM / SGLang integration
  107. ```
  108. The `install_hook` step writes a `.pth` file into your site-packages so the vLLM and SGLang hooks load automatically at Python startup — no code changes required. To uninstall: `python -m contextpilot.install_hook --remove`.
  109. ---
  110. ### Mac / Apple Silicon — llama.cpp
  111. **From PyPI:**
  112. ```bash
  113. pip install contextpilot
  114. xcode-select --install # one-time: provides clang++ to compile the native hook
  115. ```
  116. **From source:**
  117. ```bash
  118. git clone https://github.com/EfficientContext/ContextPilot.git
  119. cd ContextPilot
  120. pip install -e .
  121. xcode-select --install # one-time: provides clang++ to compile the native hook
  122. ```
  123. > **Why `xcode-select`?** The llama.cpp integration uses a small C++ shared library injected into `llama-server` via `DYLD_INSERT_LIBRARIES`. It is compiled automatically on first use and requires `clang++` from Xcode Command Line Tools.
  124. ---
  125. More [detailed installation instructions](https://efficientcontext.github.io/contextpilot-docs/getting_started/installation) are available in the docs.
  126. Docker images are also available for both all-in-one and standalone deployment. See the [Docker guide](https://efficientcontext.github.io/contextpilot-docs/getting_started/docker).
  127. ## Getting Started
  128. ### Quick Start with OpenClaw
  129. ```bash
  130. # Ask OpenClaw to analyze vendor contracts (ContextPilot deduplicates shared content automatically)
  131. openclaw agent --message "Read contracts/contract_alpha_cloud.txt and summarize the liability terms."
  132. openclaw agent --message "Read contracts/contract_beta_ai.txt and compare its liability with Alpha."
  133. openclaw agent --message "Read contracts/contract_gamma_security.txt. Rank all three by liability exposure."
  134. ```
  135. When the agent reads multiple documents sharing content (contracts from the same template, proposals with shared methodology), ContextPilot automatically deduplicates identical blocks — reducing prefill tokens by ~27% with zero accuracy loss. See the [integration guide](docs/guides/openclaw.md) and [benchmark](docs/benchmarks/openclaw.md).
  136. ---
  137. ### Quick Start with Context Ordering
  138. Add **one call** (`cp_instance.optimize()`) before inference to rearrange context blocks so that shared content aligns into a common prefix, enabling cache reuse. An importance ranking in the prompt preserves accuracy.
  139. | Mode | When to Use | How It Works |
  140. |------|-------------|--------------|
  141. | **Online** | Multi-turn (e.g., chatbot + [Mem0](https://github.com/mem0ai/mem0)) | Tracks previously cached blocks; moves overlapping ones to the prefix each turn |
  142. | **Offline** | Batch / single-shot | Globally reorders and schedules all requests for maximum prefix sharing |
  143. Both modes work with any OpenAI-compatible endpoint (vLLM, SGLang, etc.) — no changes to your inference deployment. They support both direct API calls (shown below) and HTTP server deployment (see the [online usage guide](https://efficientcontext.github.io/contextpilot-docs/guides/online_usage)).
  144. ---
  145. #### Accelerating Online Inference
  146. Multi-turn chatbot with Mem0 or RAG where each turn's context blocks partially overlap. `cp_instance.optimize()` moves shared blocks to the prefix so the engine reuses cached KV states.
  147. ```python
  148. from openai import OpenAI
  149. # Step 1: Import ContextPilot
  150. import contextpilot as cp
  151. client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
  152. # Step 2: Create a ContextPilot instance
  153. cp_instance = cp.ContextPilot(use_gpu=False)
  154. for query in queries:
  155. contexts = get_contexts(query) # Mem0, Retriever, ...
  156. # Step 3: Optimize context ordering and build ready-to-use messages
  157. messages = cp_instance.optimize(contexts, query)
  158. response = client.chat.completions.create(
  159. model="Qwen/Qwen3-4B",
  160. messages=messages,
  161. )
  162. print(f"Q: {query}\nA: {response.choices[0].message.content}\n")
  163. ```
  164. > **Note:** When the engine evicts KV-cache entries under memory pressure, ContextPilot's index can go stale. Set `CONTEXTPILOT_INDEX_URL` when launching [SGLang or vLLM](https://efficientcontext.github.io/contextpilot-docs/guides/online_usage#inference-engine-integration) to enable automatic eviction sync. For distributed setups, see [Distributed Setup](https://efficientcontext.github.io/contextpilot-docs/getting_started/installation#distributed-setup).
  165. ---
  166. #### Accelerating Offline Inference
  167. Batch of requests with overlapping context blocks. `cp_instance.optimize_batch()` globally reorders blocks and schedules execution order so queries with similar contexts run consecutively, maximizing cache reuse. See the [offline usage guide](https://efficientcontext.github.io/contextpilot-docs/guides/offline_usage) for details. Offline mode can also be deployed as an HTTP server without eviction sync — see [Stateless Mode](https://efficientcontext.github.io/contextpilot-docs/guides/online_usage#stateless-mode).
  168. ```python
  169. import asyncio
  170. import openai
  171. # Step 1: Import ContextPilot
  172. import contextpilot as cp
  173. BASE_URL = "http://localhost:30000/v1"
  174. # Step 2: Create a ContextPilot instance
  175. cp_instance = cp.ContextPilot(use_gpu=False)
  176. all_contexts = [get_contexts(q) for q in queries] # Mem0, Retriever, ...
  177. # Step 3: Optimize — reorder, schedule, and build prompts in one call
  178. messages_batch, order = cp_instance.optimize_batch(all_contexts, queries)
  179. # Send all requests concurrently
  180. async def generate_all():
  181. ac = openai.AsyncOpenAI(base_url=BASE_URL, api_key="EMPTY")
  182. return await asyncio.gather(*[ac.chat.completions.create(
  183. model="Qwen/Qwen3-4B", messages=m
  184. ) for m in messages_batch])
  185. for resp, idx in zip(asyncio.run(generate_all()), order):
  186. print(f"Q: {queries[idx]}\nA: {resp.choices[0].message.content}\n")
  187. ```
  188. For a detailed walkthrough with concrete examples, see the [Quick Start Guide](https://efficientcontext.github.io/contextpilot-docs/getting_started/quickstart). For more fine-grained control, you can also use `cp_instance.reorder()` and `cp_instance.deduplicate()` directly — see the [API reference](https://efficientcontext.github.io/contextpilot-docs/reference/api) and [multi-turn deduplication guide](https://efficientcontext.github.io/contextpilot-docs/guides/multi_turn).
  189. ### Adoption Examples
  190. See many useful adoption examples: [Mem0 integration](https://efficientcontext.github.io/contextpilot-docs/guides/mem0), [PageIndex RAG](https://efficientcontext.github.io/contextpilot-docs/guides/pageindex), [offline batch scheduling](https://efficientcontext.github.io/contextpilot-docs/guides/offline_usage), and [multi-turn deduplication](https://efficientcontext.github.io/contextpilot-docs/guides/multi_turn).
  191. ## Citation
  192. ```bibtex
  193. @inproceedings{contextpilot2026,
  194. title = {ContextPilot: Fast Long-Context Inference via Context Reuse},
  195. author = {Jiang, Yinsicheng and Huang, Yeqi and Cheng, Liang and Deng, Cheng and Sun, Xuan and Mai, Luo},
  196. booktitle = {Proceedings of the 9th Conference on Machine Learning and Systems (MLSys 2026)},
  197. year = {2026},
  198. url = {https://arxiv.org/abs/2511.03475}
  199. }
  200. ```
  201. ## Contributing
  202. We welcome and value all contributions! Please feel free to submit issues and pull requests.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.