Skip to content
Back to search

pixelrag-plugins

MarketplaceWorks with Claude CodeGitHub

PixelRAG Claude Code plugins

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/startrail-org/pixelrag with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/startrail-org/pixelrag'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/startrail-org--pixelrag/badge.svg)](/listings/startrail-org--pixelrag)

HTML

<a href="/listings/startrail-org--pixelrag"><img src="/api/listings/startrail-org--pixelrag/badge.svg" alt="findsafeskills safety rating" /></a>

Install

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 16 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "pixelrag-plugins",
  3. "owner": {
  4. "name": "StarTrail-org"
  5. },
  6. "description": "PixelRAG Claude Code plugins",
  7. "plugins": [
  8. {
  9. "name": "pixelbrowse",
  10. "source": "./plugin",
  11. "description": "Give Claude eyes — screenshot any URL with pixelshot and read it visually",
  12. "homepage": "https://github.com/StarTrail-org/PixelRAG",
  13. "repository": "https://github.com/StarTrail-org/PixelRAG"
  14. }
  15. ]
  16. }
README.md · README · 404 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <img src="docs/assets/banner.png" alt="PixelRAG — Visual Retrieval-Augmented Generation" width="100%">
  3. </p>
  4. <p align="center">
  5. Official codebase for <b><a href="https://arxiv.org/abs/2606.28344">PIXELRAG: Web Screenshots Beat Text for
  6. Retrieval-Augmented Generation</a></b>
  7. </p>
  8. <p align="center">
  9. <a href="https://yichuan-w.github.io/">Yichuan Wang</a>*,
  10. <a href="https://zhifei.li/">Zhifei Li</a>*,
  11. <a href="https://zwcolin.github.io/">Zirui Wang</a>,
  12. <a href="https://www.linkedin.com/in/paul-teiletche/">Paul Teiletche</a>,
  13. <a href="https://www.linkedin.com/in/lesheng-jin-9618b0201/">Lesheng Jin</a>
  14. <br>
  15. <a href="https://people.eecs.berkeley.edu/~matei/">Matei Zaharia</a>†,
  16. <a href="https://people.eecs.berkeley.edu/~jegonzal/">Joseph E. Gonzalez</a>†,
  17. <a href="https://www.sewonmin.com/">Sewon Min</a>†
  18. </p>
  19. <p align="center"><sub>* Equal contribution &nbsp; † Equal advising</sub><br><sub>Work done at <a href="https://sky.cs.berkeley.edu/">Berkeley SkyLab</a> &amp; <a href="https://bair.berkeley.edu/">BAIR</a> &amp; <a href="https://nlp.cs.berkeley.edu/">Berkeley NLP</a></sub></p>
  20. <p align="center">Search any document by how it <em>looks</em>, not just the text it contains.</p>
  21. <p align="center">
  22. <a href="https://github.com/StarTrail-org/PixelRAG/actions/workflows/ci.yml"><img src="https://github.com/StarTrail-org/PixelRAG/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  23. <a href="https://pixelrag.ai"><img src="https://img.shields.io/badge/demo-pixelrag.ai-7c3aed" alt="Live demo"></a>
  24. <a href="https://status.pixelrag.ai"><img src="https://img.shields.io/badge/status-live-22c55e" alt="Status"></a>
  25. <a href="https://join.slack.com/t/leann-e2u9779/shared_invite/zt-3ol2ww9ic-Eg_kB8omwe6xmYVd0epr4Q"><img src="https://img.shields.io/badge/Slack-join-4A154B?logo=slack&logoColor=white" alt="Slack"></a>
  26. <img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="License">
  27. </p>
  28. <p align="center">
  29. <a href="#what-it-is">What it is</a> &middot;
  30. <a href="#give-claude-eyes">Give Claude eyes</a> &middot;
  31. <a href="#how-it-works">How it works</a> &middot;
  32. <a href="#pipelines">Pipelines</a>
  33. </p>
  34. ---
  35. ```bash
  36. pip install pixelrag
  37. ```
  38. The two core operations — **render** a page to screenshots, **search** a visual index:
  39. ```bash
  40. # Render any page or document to screenshot tiles
  41. pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles
  42. # Search a hosted index of 8.28M Wikipedia pages — no setup, runs against the live API
  43. curl -X POST https://api.pixelrag.ai/search \
  44. -H "Content-Type: application/json" \
  45. -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
  46. ```
  47. > **Live, hosted endpoint** — [`https://api.pixelrag.ai`](https://api.pixelrag.ai/status) serves a
  48. > pre-built index of **8.28M Wikipedia pages**. No setup, no API key. It even takes an image as the query
  49. > ([visual search](https://pixelrag.ai/docs#search)) — see the **[API reference →](https://pixelrag.ai/docs)**.
  50. Or try it in the browser at **[pixelrag.ai](https://pixelrag.ai)**, or run the demo notebook in
  51. Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/StarTrail-org/PixelRAG/blob/main/demos/quickstart.ipynb) — it
  52. renders a page and searches the hosted index, with the images inline.
  53. ## What it is
  54. PixelRAG renders documents — web pages, PDFs, images — as screenshots and retrieves over the
  55. images directly. Visual structure that HTML parsing throws away — tables, charts, layout,
  56. infographics — stays intact, so the reader model can actually answer questions about it.
  57. Wikipedia's 8.28M articles ship as a pre-built index; the pipeline itself is general-purpose.
  58. ## Give Claude eyes
  59. The renderer also ships as a Claude Code plugin — the **pixelbrowse** skill. Instead of fetching
  60. raw HTML, Claude screenshots a page with `pixelshot` and _reads the image_, so it sees
  61. charts, diagrams, tables, and layout the way a person does.
  62. Install it — no clone needed. Install the `pixelshot` CLI so it's on your `PATH`
  63. (use `uv tool` or `pipx` to keep it isolated yet always available to Claude — a
  64. plain `pip install` into a project venv may leave `pixelshot` off `PATH`):
  65. ```bash
  66. uv tool install pixelrag # pixelshot on PATH (or: pipx install pixelrag)
  67. claude plugin marketplace add StarTrail-org/PixelRAG
  68. claude plugin install pixelbrowse@pixelrag-plugins
  69. ```
  70. Then just ask Claude to look at a page:
  71. ```bash
  72. claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
  73. claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"
  74. ```
  75. Or use the slash command in an interactive session: `/screenshot https://example.com`.
  76. No MCP server, no backend: the skill just calls `pixelshot` (Playwright/CDP) on your machine.
  77. **opencode** users get the same `screenshot` tool via the `@startrail/pixelbrowse` npm package
  78. — with `pixelshot` on `PATH`, add it to `opencode.json`:
  79. ```json
  80. {
  81. "$schema": "https://opencode.ai/config.json",
  82. "plugin": ["@startrail/pixelbrowse"]
  83. }
  84. ```
  85. See [`plugin/opencode/`](plugin/opencode/) for details.
  86. ## How it works
  87. <p align="center">
  88. <img src="docs/assets/pipeline.png" alt="Text-based RAG parses to text and loses the table; PixelRAG renders to screenshot tiles and keeps it" width="100%">
  89. </p>
  90. Text-based RAG parses the page to text chunks and **loses the table** — the reader can't find the
  91. answer. PixelRAG renders the page to **screenshot tiles**, retrieves the right tile, and the reader
  92. reads the number straight off the image.
  93. Two pieces make this work: (1) rendering documents to images instead of parsing them to text, and
  94. (2) a `Qwen3-VL-Embedding` model, LoRA-fine-tuned on screenshot data, that embeds page images into
  95. a space where visual content is retrievable.
  96. ## Pipelines
  97. Capture is the standalone `pixelshot` command; the rest of the pipeline runs through the
  98. `pixelrag` umbrella — `pixelrag <stage>`. Install only the stages you need:
  99. | Command | What it does | Install |
  100. | ------------------------------------------ | --------------------------------------------------------------- | ------------------------------- |
  101. | `pixelshot` | Document → image tiles (Playwright CDP, PDF) | `pip install pixelrag` |
  102. | `pixelrag chunk` · `embed` · `build-index` | Tiles → vectors → FAISS index | `pip install 'pixelrag[embed]'` |
  103. | `pixelrag index` | Orchestrates the full pipeline: source → ingest → embed → index | `pip install 'pixelrag[index]'` |
  104. | `pixelrag serve` | FAISS search API (FastAPI, CPU or GPU) | `pip install 'pixelrag[serve]'` |
  105. ```
  106. render ←── index ──→ embed serve (independent) train → serve (HTTP)
  107. ```
  108. **`train` is a separate uv project** with its own pinned env (`torch==2.9.1+cu129`,
  109. `transformers==4.57.1`, cuDNN 9.20) — install it from inside `train/`, not from the root.
  110. ### Search a pre-built index
  111. ```bash
  112. pip install 'pixelrag[serve]'
  113. # Download a pre-built index from Hugging Face. The dataset repo holds four FAISS indexes
  114. # (base/LoRA Wikipedia pixel, Wikipedia text, news pixel); grab just the base one (~217G) here.
  115. huggingface-cli download StarTrail-org/pixelrag-faiss-indexes \
  116. --repo-type dataset --include "search_index_normed_v2/*" --local-dir ./index
  117. # Serve, then query
  118. pixelrag serve --index-dir ./index/search_index_normed_v2 --port 30001
  119. curl -X POST http://localhost:30001/search \
  120. -H "Content-Type: application/json" \
  121. -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
  122. ```
  123. ### Build an index from your own documents
  124. Works on **Linux (CUDA)** and **macOS (Apple Silicon / MPS)** — `device: auto` picks the best backend.
  125. ```bash
  126. pip install 'pixelrag[index]'
  127. # Create pixelrag.yaml
  128. cat > pixelrag.yaml << 'EOF'
  129. source:
  130. type: local
  131. path: ./my_docs
  132. embed:
  133. model: Qwen/Qwen3-VL-Embedding-2B
  134. device: auto # cuda on Linux, mps on macOS, cpu as fallback
  135. output: ./my_index
  136. EOF
  137. # Build, then serve
  138. pixelrag index build
  139. pixelrag serve --index-dir ./my_index --port 30001
  140. ```
  141. <details>
  142. <summary><strong>Try it: index a PDF and search it locally</strong></summary>
  143. No GPU required — runs on macOS (Apple Silicon) or any machine with Python 3.10+.
  144. PDF rendering needs the `pdf` extra and poppler (`brew install poppler`, or
  145. `apt-get install poppler-utils` on Debian/Ubuntu).
  146. ```bash
  147. pip install 'pixelrag[index,pdf]'
  148. # 1. Grab a sample PDF (or use your own)
  149. curl -L -o paper.pdf https://raw.githubusercontent.com/StarTrail-org/PixelRAG/main/assets/pixelrag-paper.pdf
  150. # 2. Create config (device: auto picks MPS on Mac, CUDA on Linux)
  151. cat > pixelrag.yaml << 'EOF'
  152. source:
  153. type: local
  154. path: ./paper.pdf
  155. embed:
  156. model: Qwen/Qwen3-VL-Embedding-2B
  157. device: auto
  158. output: ./paper_index
  159. EOF
  160. # 3. Build the index (~3 min on Apple M-series, ~1 min on GPU)
  161. pixelrag index build
  162. # 4. Serve it
  163. pixelrag serve --index-dir ./paper_index --port 30001
  164. # 5. Search — should return page 2 (the overview diagram)
  165. curl -X POST http://localhost:30001/search \
  166. -H "Content-Type: application/json" \
  167. -d '{"queries": [{"text": "Overview of PixelRAG and the diagram"}], "n_docs": 1}'
  168. ```
  169. </details>
  170. ### Render a page programmatically
  171. To find pages on the open web before capturing them, try the optional
  172. [Parallel Search MCP demo](demos/parallel_search/). It saves source URLs and text
  173. excerpts, then can render selected pages as screenshot tiles without a search API key.
  174. ```python
  175. from pixelrag_render import render_url
  176. # render a single page to tiles — e.g. for an agent to read
  177. tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")
  178. ```
  179. The same rendering is available as a CLI — `pixelshot` ships with `pip install pixelrag`:
  180. ```bash
  181. # Web page → tiles (headless Chromium via CDP)
  182. pixelshot https://en.wikipedia.org/wiki/Python -o ./tiles
  183. # PDF → tiles (requires poppler; install the pdf extra: pip install 'pixelrag[pdf]')
  184. curl -sL -o paper.pdf https://arxiv.org/pdf/2503.09516
  185. pixelshot paper.pdf -o ./tiles --dpi 200
  186. # URLs and local files can be mixed freely
  187. pixelshot https://github.com/StarTrail-org/PixelRAG paper.pdf -o ./tiles
  188. ```
  189. > **Chrome on Windows/macOS** — the bundled turbo `headless_shell` auto-installs on
  190. > **linux-x64** only. Elsewhere, `pixelshot` uses your system Chrome/Chromium (or
  191. > Playwright's Chromium), auto-detected from the standard install locations. Point it at
  192. > a specific binary with `CHROME_PATH=/path/to/chrome` if it isn't found automatically.
  193. > Each render runs in an isolated, throwaway Chrome profile, so it works even while you
  194. > have Chrome open.
  195. ### Embed tools (standalone)
  196. Each stage runs independently, without the orchestrator:
  197. ```bash
  198. pip install 'pixelrag[embed]'
  199. pixelrag chunk --tiles-dir ./tiles
  200. pixelrag embed --shard-dir ./tiles --output-dir ./embeddings --gpu-ids 0,1
  201. pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index
  202. ```
  203. ### Qdrant backend
  204. [Qdrant](https://qdrant.tech) is an open-source vector search engine for high-performance and massive scale. [FAISS](https://ai.meta.com/tools/faiss/) remains the default for local indexes. Use Qdrant for configurable quantization, disk-backed vectors, payload filtering, and one collection shared by multiple PixelRAG servers.
  205. Quantization compresses vectors to reduce memory use and speed up search, with a recall tradeoff that depends on the method and settings.
  206. To configure quantization, pass any Qdrant `quantization_config` object in a JSON file.
  207. See Qdrant's [quantization guide](https://qdrant.tech/documentation/manage-data/quantization/#setting-up-quantization-in-qdrant) for supported methods and parameters.
  208. ```json
  209. {
  210. "scalar": {
  211. "type": "int8",
  212. "quantile": 0.99,
  213. "always_ram": true
  214. }
  215. }
  216. ```
  217. ```bash
  218. pip install 'pixelrag[serve,qdrant]' # or 'pixelrag[index,qdrant]'
  219. # Build against a Qdrant server.
  220. # Start one locally with: docker run -p 6333:6333 qdrant/qdrant
  221. pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
  222. --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag \
  223. --qdrant-quantization-config ./quantization.json
  224. # Add documents to an existing collection.
  225. pixelrag build-index --embeddings-dir ./more --output-dir ./index \
  226. --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --append
  227. # Replace an existing collection and its configuration.
  228. pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
  229. --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --recreate
  230. # Serve the collection. PixelRAG reads the backend from summary.json.
  231. pixelrag serve --index-dir ./index --qdrant-url http://localhost:6333 \
  232. --qdrant-client-config ./qdrant-client.json --port 30001
  233. ```
  234. Configure the orchestrator in `pixelrag.yaml`:
  235. ```yaml
  236. index:
  237. backend: qdrant
  238. qdrant_url: http://localhost:6333
  239. collection: pixelrag
  240. client_config: ./qdrant-client.json
  241. quantization_config: ./quantization.json
  242. # Set append: true or recreate: true when the collection already exists.
  243. ```
  244. ### Run the API in Docker
  245. `Dockerfile` builds a two-stage image that installs the `serve` extra and a pinned Chromium,
  246. and `docker-compose.yml` runs `pixelrag serve` on port 8000 against the index and tiles you
  247. mount from the host.
  248. **Build an index first.** The container mounts `./data/index` read-only and cannot create one,
  249. so `docker compose up` against an empty directory exits before the server starts — `pixelrag
  250. serve` opens the index and `articles.json` during startup. Point `output:` at it in your
  251. `pixelrag.yaml` (as above), then:
  252. ```bash
  253. mkdir -p data/index data/tiles
  254. pixelrag index build # writes index.faiss, metadata.npz and articles.json to ./data/index
  255. docker compose up --build
  256. # then query it
  257. curl -X POST http://localhost:8000/search \
  258. -H "Content-Type: application/json" \
  259. -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
  260. ```
  261. Three things to know before using it:
  262. - **Nothing in CI builds this image.** It is best-effort and can drift when dependencies or
  263. source paths move, so build it yourself before relying on it.
  264. - **On Apple Silicon the image builds without a renderer.** The pinned `headless_shell` is
  265. published for linux-x64 only, so the build skips it on other architectures: search over a
  266. prebuilt index works, on-demand rendering does not. For a renderer-capable image, build for
  267. amd64 (emulated, and slower) by adding `platform: linux/amd64` to the `pixelrag-api` service.
  268. - **The container runs as root, so Chrome renders with its sandbox disabled.** Chrome's
  269. sandbox inside a container requires `--cap-add=SYS_ADMIN` or a tailored seccomp profile,
  270. and a non-root `USER` without those stops Chrome launching entirely. Container isolation
  271. is therefore the only boundary around the renderer — keep the container isolated and do
  272. not point it at untrusted pages otherwise.
  273. ### Training
  274. Fine-tuning lives in `train/` — a **separate uv project** (`wiki-screenshot-training`) with its own
  275. pinned env. It LoRA-fine-tunes `Qwen/Qwen3-VL-Embedding-2B` for webpage retrieval; run it from
  276. inside `train/` (`cd train && uv sync`). See [`train/README.md`](train/README.md) for the full recipe.
  277. You don't need to retrain to use the model — the trained adapters are published at
  278. [`Chrisyichuan/wiki-screenshot-embedding-lora`](https://huggingface.co/Chrisyichuan/wiki-screenshot-embedding-lora/tree/main/lora_vit/ckpt200).
  279. We also release the full training set
  280. ([`Chrisyichuan/screenshot-training-natural-filtered-v2`](https://huggingface.co/datasets/Chrisyichuan/screenshot-training-natural-filtered-v2)),
  281. so you can adapt other backbones yourself — a larger Qwen, or any other embedding model.
  282. The data curation pipeline (LLM-augmented query generation, filtering, hard-negative mining)
  283. is documented in [`train/docs/synthetic_data_pipeline.md`](train/docs/synthetic_data_pipeline.md).
  284. ## Citation
  285. If you find PixelRAG useful, please cite our paper:
  286. ```bibtex
  287. @misc{wang2026pixelragwebscreenshotsbeat,
  288. title={PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation},
  289. author={Yichuan Wang and Zhifei Li and Zirui Wang and Paul Teiletche and Lesheng Jin and Matei Zaharia and Joseph E. Gonzalez and Sewon Min},
  290. year={2026},
  291. eprint={2606.28344},
  292. archivePrefix={arXiv},
  293. primaryClass={cs.IR},
  294. url={https://arxiv.org/abs/2606.28344},
  295. }
  296. ```
  297. ## Acknowledgments
  298. Thanks to [Rulin Shao](https://rulinshao.github.io/) for support.
  299. Thanks also to [Claude Code](https://github.com/anthropics/claude-code) and
  300. [OpenAI Codex](https://github.com/openai/codex) for supporting open-source contributors with credits and plans,
  301. which we earned by working on [LEANN](https://github.com/StarTrail-org/LEANN).
  302. This work is done by the [Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/),
  303. [BAIR](https://bair.berkeley.edu/), and the [Berkeley NLP Group](https://nlp.cs.berkeley.edu/).
  304. ## License
  305. Apache-2.0

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.