pixelrag-plugins
PixelRAG Claude Code plugins
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/startrail-org/pixelrag with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/startrail-org/pixelrag'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/startrail-org--pixelrag)HTML
<a href="/listings/startrail-org--pixelrag"><img src="/api/listings/startrail-org--pixelrag/badge.svg" alt="findsafeskills safety rating" /></a>Install
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 16 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- {
- "name": "pixelrag-plugins",
- "owner": {
- "name": "StarTrail-org"
- },
- "description": "PixelRAG Claude Code plugins",
- "plugins": [
- {
- "name": "pixelbrowse",
- "source": "./plugin",
- "description": "Give Claude eyes — screenshot any URL with pixelshot and read it visually",
- "homepage": "https://github.com/StarTrail-org/PixelRAG",
- "repository": "https://github.com/StarTrail-org/PixelRAG"
- }
- ]
- }
README.md · README · 404 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <p align="center">
- <img src="docs/assets/banner.png" alt="PixelRAG — Visual Retrieval-Augmented Generation" width="100%">
- </p>
- <p align="center">
- Official codebase for <b><a href="https://arxiv.org/abs/2606.28344">PIXELRAG: Web Screenshots Beat Text for
- Retrieval-Augmented Generation</a></b>
- </p>
- <p align="center">
- <a href="https://yichuan-w.github.io/">Yichuan Wang</a>*,
- <a href="https://zhifei.li/">Zhifei Li</a>*,
- <a href="https://zwcolin.github.io/">Zirui Wang</a>,
- <a href="https://www.linkedin.com/in/paul-teiletche/">Paul Teiletche</a>,
- <a href="https://www.linkedin.com/in/lesheng-jin-9618b0201/">Lesheng Jin</a>
- <br>
- <a href="https://people.eecs.berkeley.edu/~matei/">Matei Zaharia</a>†,
- <a href="https://people.eecs.berkeley.edu/~jegonzal/">Joseph E. Gonzalez</a>†,
- <a href="https://www.sewonmin.com/">Sewon Min</a>†
- </p>
- <p align="center"><sub>* Equal contribution † Equal advising</sub><br><sub>Work done at <a href="https://sky.cs.berkeley.edu/">Berkeley SkyLab</a> & <a href="https://bair.berkeley.edu/">BAIR</a> & <a href="https://nlp.cs.berkeley.edu/">Berkeley NLP</a></sub></p>
- <p align="center">Search any document by how it <em>looks</em>, not just the text it contains.</p>
- <p align="center">
- <a href="https://github.com/StarTrail-org/PixelRAG/actions/workflows/ci.yml"><img src="https://github.com/StarTrail-org/PixelRAG/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
- <a href="https://pixelrag.ai"><img src="https://img.shields.io/badge/demo-pixelrag.ai-7c3aed" alt="Live demo"></a>
- <a href="https://status.pixelrag.ai"><img src="https://img.shields.io/badge/status-live-22c55e" alt="Status"></a>
- <a href="https://join.slack.com/t/leann-e2u9779/shared_invite/zt-3ol2ww9ic-Eg_kB8omwe6xmYVd0epr4Q"><img src="https://img.shields.io/badge/Slack-join-4A154B?logo=slack&logoColor=white" alt="Slack"></a>
- <img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="License">
- </p>
- <p align="center">
- <a href="#what-it-is">What it is</a> ·
- <a href="#give-claude-eyes">Give Claude eyes</a> ·
- <a href="#how-it-works">How it works</a> ·
- <a href="#pipelines">Pipelines</a>
- </p>
- ---
- ```bash
- pip install pixelrag
- ```
- The two core operations — **render** a page to screenshots, **search** a visual index:
- ```bash
- # Render any page or document to screenshot tiles
- pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles
- # Search a hosted index of 8.28M Wikipedia pages — no setup, runs against the live API
- curl -X POST https://api.pixelrag.ai/search \
- -H "Content-Type: application/json" \
- -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
- ```
- > **Live, hosted endpoint** — [`https://api.pixelrag.ai`](https://api.pixelrag.ai/status) serves a
- > pre-built index of **8.28M Wikipedia pages**. No setup, no API key. It even takes an image as the query
- > ([visual search](https://pixelrag.ai/docs#search)) — see the **[API reference →](https://pixelrag.ai/docs)**.
- Or try it in the browser at **[pixelrag.ai](https://pixelrag.ai)**, or run the demo notebook in
- Colab [](https://colab.research.google.com/github/StarTrail-org/PixelRAG/blob/main/demos/quickstart.ipynb) — it
- renders a page and searches the hosted index, with the images inline.
- ## What it is
- PixelRAG renders documents — web pages, PDFs, images — as screenshots and retrieves over the
- images directly. Visual structure that HTML parsing throws away — tables, charts, layout,
- infographics — stays intact, so the reader model can actually answer questions about it.
- Wikipedia's 8.28M articles ship as a pre-built index; the pipeline itself is general-purpose.
- ## Give Claude eyes
- The renderer also ships as a Claude Code plugin — the **pixelbrowse** skill. Instead of fetching
- raw HTML, Claude screenshots a page with `pixelshot` and _reads the image_, so it sees
- charts, diagrams, tables, and layout the way a person does.
- Install it — no clone needed. Install the `pixelshot` CLI so it's on your `PATH`
- (use `uv tool` or `pipx` to keep it isolated yet always available to Claude — a
- plain `pip install` into a project venv may leave `pixelshot` off `PATH`):
- ```bash
- uv tool install pixelrag # pixelshot on PATH (or: pipx install pixelrag)
- claude plugin marketplace add StarTrail-org/PixelRAG
- claude plugin install pixelbrowse@pixelrag-plugins
- ```
- Then just ask Claude to look at a page:
- ```bash
- claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
- claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"
- ```
- Or use the slash command in an interactive session: `/screenshot https://example.com`.
- No MCP server, no backend: the skill just calls `pixelshot` (Playwright/CDP) on your machine.
- **opencode** users get the same `screenshot` tool via the `@startrail/pixelbrowse` npm package
- — with `pixelshot` on `PATH`, add it to `opencode.json`:
- ```json
- {
- "$schema": "https://opencode.ai/config.json",
- "plugin": ["@startrail/pixelbrowse"]
- }
- ```
- See [`plugin/opencode/`](plugin/opencode/) for details.
- ## How it works
- <p align="center">
- <img src="docs/assets/pipeline.png" alt="Text-based RAG parses to text and loses the table; PixelRAG renders to screenshot tiles and keeps it" width="100%">
- </p>
- Text-based RAG parses the page to text chunks and **loses the table** — the reader can't find the
- answer. PixelRAG renders the page to **screenshot tiles**, retrieves the right tile, and the reader
- reads the number straight off the image.
- Two pieces make this work: (1) rendering documents to images instead of parsing them to text, and
- (2) a `Qwen3-VL-Embedding` model, LoRA-fine-tuned on screenshot data, that embeds page images into
- a space where visual content is retrievable.
- ## Pipelines
- Capture is the standalone `pixelshot` command; the rest of the pipeline runs through the
- `pixelrag` umbrella — `pixelrag <stage>`. Install only the stages you need:
- | Command | What it does | Install |
- | ------------------------------------------ | --------------------------------------------------------------- | ------------------------------- |
- | `pixelshot` | Document → image tiles (Playwright CDP, PDF) | `pip install pixelrag` |
- | `pixelrag chunk` · `embed` · `build-index` | Tiles → vectors → FAISS index | `pip install 'pixelrag[embed]'` |
- | `pixelrag index` | Orchestrates the full pipeline: source → ingest → embed → index | `pip install 'pixelrag[index]'` |
- | `pixelrag serve` | FAISS search API (FastAPI, CPU or GPU) | `pip install 'pixelrag[serve]'` |
- ```
- render ←── index ──→ embed serve (independent) train → serve (HTTP)
- ```
- **`train` is a separate uv project** with its own pinned env (`torch==2.9.1+cu129`,
- `transformers==4.57.1`, cuDNN 9.20) — install it from inside `train/`, not from the root.
- ### Search a pre-built index
- ```bash
- pip install 'pixelrag[serve]'
- # Download a pre-built index from Hugging Face. The dataset repo holds four FAISS indexes
- # (base/LoRA Wikipedia pixel, Wikipedia text, news pixel); grab just the base one (~217G) here.
- huggingface-cli download StarTrail-org/pixelrag-faiss-indexes \
- --repo-type dataset --include "search_index_normed_v2/*" --local-dir ./index
- # Serve, then query
- pixelrag serve --index-dir ./index/search_index_normed_v2 --port 30001
- curl -X POST http://localhost:30001/search \
- -H "Content-Type: application/json" \
- -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
- ```
- ### Build an index from your own documents
- Works on **Linux (CUDA)** and **macOS (Apple Silicon / MPS)** — `device: auto` picks the best backend.
- ```bash
- pip install 'pixelrag[index]'
- # Create pixelrag.yaml
- cat > pixelrag.yaml << 'EOF'
- source:
- type: local
- path: ./my_docs
- embed:
- model: Qwen/Qwen3-VL-Embedding-2B
- device: auto # cuda on Linux, mps on macOS, cpu as fallback
- output: ./my_index
- EOF
- # Build, then serve
- pixelrag index build
- pixelrag serve --index-dir ./my_index --port 30001
- ```
- <details>
- <summary><strong>Try it: index a PDF and search it locally</strong></summary>
- No GPU required — runs on macOS (Apple Silicon) or any machine with Python 3.10+.
- PDF rendering needs the `pdf` extra and poppler (`brew install poppler`, or
- `apt-get install poppler-utils` on Debian/Ubuntu).
- ```bash
- pip install 'pixelrag[index,pdf]'
- # 1. Grab a sample PDF (or use your own)
- curl -L -o paper.pdf https://raw.githubusercontent.com/StarTrail-org/PixelRAG/main/assets/pixelrag-paper.pdf
- # 2. Create config (device: auto picks MPS on Mac, CUDA on Linux)
- cat > pixelrag.yaml << 'EOF'
- source:
- type: local
- path: ./paper.pdf
- embed:
- model: Qwen/Qwen3-VL-Embedding-2B
- device: auto
- output: ./paper_index
- EOF
- # 3. Build the index (~3 min on Apple M-series, ~1 min on GPU)
- pixelrag index build
- # 4. Serve it
- pixelrag serve --index-dir ./paper_index --port 30001
- # 5. Search — should return page 2 (the overview diagram)
- curl -X POST http://localhost:30001/search \
- -H "Content-Type: application/json" \
- -d '{"queries": [{"text": "Overview of PixelRAG and the diagram"}], "n_docs": 1}'
- ```
- </details>
- ### Render a page programmatically
- To find pages on the open web before capturing them, try the optional
- [Parallel Search MCP demo](demos/parallel_search/). It saves source URLs and text
- excerpts, then can render selected pages as screenshot tiles without a search API key.
- ```python
- from pixelrag_render import render_url
- # render a single page to tiles — e.g. for an agent to read
- tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")
- ```
- The same rendering is available as a CLI — `pixelshot` ships with `pip install pixelrag`:
- ```bash
- # Web page → tiles (headless Chromium via CDP)
- pixelshot https://en.wikipedia.org/wiki/Python -o ./tiles
- # PDF → tiles (requires poppler; install the pdf extra: pip install 'pixelrag[pdf]')
- curl -sL -o paper.pdf https://arxiv.org/pdf/2503.09516
- pixelshot paper.pdf -o ./tiles --dpi 200
- # URLs and local files can be mixed freely
- pixelshot https://github.com/StarTrail-org/PixelRAG paper.pdf -o ./tiles
- ```
- > **Chrome on Windows/macOS** — the bundled turbo `headless_shell` auto-installs on
- > **linux-x64** only. Elsewhere, `pixelshot` uses your system Chrome/Chromium (or
- > Playwright's Chromium), auto-detected from the standard install locations. Point it at
- > a specific binary with `CHROME_PATH=/path/to/chrome` if it isn't found automatically.
- > Each render runs in an isolated, throwaway Chrome profile, so it works even while you
- > have Chrome open.
- ### Embed tools (standalone)
- Each stage runs independently, without the orchestrator:
- ```bash
- pip install 'pixelrag[embed]'
- pixelrag chunk --tiles-dir ./tiles
- pixelrag embed --shard-dir ./tiles --output-dir ./embeddings --gpu-ids 0,1
- pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index
- ```
- ### Qdrant backend
- [Qdrant](https://qdrant.tech) is an open-source vector search engine for high-performance and massive scale. [FAISS](https://ai.meta.com/tools/faiss/) remains the default for local indexes. Use Qdrant for configurable quantization, disk-backed vectors, payload filtering, and one collection shared by multiple PixelRAG servers.
- Quantization compresses vectors to reduce memory use and speed up search, with a recall tradeoff that depends on the method and settings.
- To configure quantization, pass any Qdrant `quantization_config` object in a JSON file.
- See Qdrant's [quantization guide](https://qdrant.tech/documentation/manage-data/quantization/#setting-up-quantization-in-qdrant) for supported methods and parameters.
- ```json
- {
- "scalar": {
- "type": "int8",
- "quantile": 0.99,
- "always_ram": true
- }
- }
- ```
- ```bash
- pip install 'pixelrag[serve,qdrant]' # or 'pixelrag[index,qdrant]'
- # Build against a Qdrant server.
- # Start one locally with: docker run -p 6333:6333 qdrant/qdrant
- pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
- --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag \
- --qdrant-quantization-config ./quantization.json
- # Add documents to an existing collection.
- pixelrag build-index --embeddings-dir ./more --output-dir ./index \
- --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --append
- # Replace an existing collection and its configuration.
- pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
- --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --recreate
- # Serve the collection. PixelRAG reads the backend from summary.json.
- pixelrag serve --index-dir ./index --qdrant-url http://localhost:6333 \
- --qdrant-client-config ./qdrant-client.json --port 30001
- ```
- Configure the orchestrator in `pixelrag.yaml`:
- ```yaml
- index:
- backend: qdrant
- qdrant_url: http://localhost:6333
- collection: pixelrag
- client_config: ./qdrant-client.json
- quantization_config: ./quantization.json
- # Set append: true or recreate: true when the collection already exists.
- ```
- ### Run the API in Docker
- `Dockerfile` builds a two-stage image that installs the `serve` extra and a pinned Chromium,
- and `docker-compose.yml` runs `pixelrag serve` on port 8000 against the index and tiles you
- mount from the host.
- **Build an index first.** The container mounts `./data/index` read-only and cannot create one,
- so `docker compose up` against an empty directory exits before the server starts — `pixelrag
- serve` opens the index and `articles.json` during startup. Point `output:` at it in your
- `pixelrag.yaml` (as above), then:
- ```bash
- mkdir -p data/index data/tiles
- pixelrag index build # writes index.faiss, metadata.npz and articles.json to ./data/index
- docker compose up --build
- # then query it
- curl -X POST http://localhost:8000/search \
- -H "Content-Type: application/json" \
- -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
- ```
- Three things to know before using it:
- - **Nothing in CI builds this image.** It is best-effort and can drift when dependencies or
- source paths move, so build it yourself before relying on it.
- - **On Apple Silicon the image builds without a renderer.** The pinned `headless_shell` is
- published for linux-x64 only, so the build skips it on other architectures: search over a
- prebuilt index works, on-demand rendering does not. For a renderer-capable image, build for
- amd64 (emulated, and slower) by adding `platform: linux/amd64` to the `pixelrag-api` service.
- - **The container runs as root, so Chrome renders with its sandbox disabled.** Chrome's
- sandbox inside a container requires `--cap-add=SYS_ADMIN` or a tailored seccomp profile,
- and a non-root `USER` without those stops Chrome launching entirely. Container isolation
- is therefore the only boundary around the renderer — keep the container isolated and do
- not point it at untrusted pages otherwise.
- ### Training
- Fine-tuning lives in `train/` — a **separate uv project** (`wiki-screenshot-training`) with its own
- pinned env. It LoRA-fine-tunes `Qwen/Qwen3-VL-Embedding-2B` for webpage retrieval; run it from
- inside `train/` (`cd train && uv sync`). See [`train/README.md`](train/README.md) for the full recipe.
- You don't need to retrain to use the model — the trained adapters are published at
- [`Chrisyichuan/wiki-screenshot-embedding-lora`](https://huggingface.co/Chrisyichuan/wiki-screenshot-embedding-lora/tree/main/lora_vit/ckpt200).
- We also release the full training set
- ([`Chrisyichuan/screenshot-training-natural-filtered-v2`](https://huggingface.co/datasets/Chrisyichuan/screenshot-training-natural-filtered-v2)),
- so you can adapt other backbones yourself — a larger Qwen, or any other embedding model.
- The data curation pipeline (LLM-augmented query generation, filtering, hard-negative mining)
- is documented in [`train/docs/synthetic_data_pipeline.md`](train/docs/synthetic_data_pipeline.md).
- ## Citation
- If you find PixelRAG useful, please cite our paper:
- ```bibtex
- @misc{wang2026pixelragwebscreenshotsbeat,
- title={PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation},
- author={Yichuan Wang and Zhifei Li and Zirui Wang and Paul Teiletche and Lesheng Jin and Matei Zaharia and Joseph E. Gonzalez and Sewon Min},
- year={2026},
- eprint={2606.28344},
- archivePrefix={arXiv},
- primaryClass={cs.IR},
- url={https://arxiv.org/abs/2606.28344},
- }
- ```
- ## Acknowledgments
- Thanks to [Rulin Shao](https://rulinshao.github.io/) for support.
- Thanks also to [Claude Code](https://github.com/anthropics/claude-code) and
- [OpenAI Codex](https://github.com/openai/codex) for supporting open-source contributors with credits and plans,
- which we earned by working on [LEANN](https://github.com/StarTrail-org/LEANN).
- This work is done by the [Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/),
- [BAIR](https://bair.berkeley.edu/), and the [Berkeley NLP Group](https://nlp.cs.berkeley.edu/).
- ## License
- Apache-2.0
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.