openreality
Open Reality for Claude Code: phone video to an AI-queryable 3D scene (measure, plan paths, export robot-training data).
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/reality-opened--openreality)HTML
<a href="/listings/reality-opened--openreality"><img src="/api/listings/reality-opened--openreality/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Add the tools to your AI assistant:
claude mcp add openreality -- npx -y openreality-mcp serve codex mcp add openreality -- npx -y openreality-mcp serve
<details> <summary><b>Claude desktop / Cursor</b> (click to expand)</summary>
Add this to `claude_desktop_config.json` (Settings → Developer → Edit Config) or `~/.cursor/mcp.json`:
{
"mcpServers": {
"openreality": {
"command": "npx",
"args": ["-y", "openreality-mcp", "serve"]
}
}
}</details>
<details> <summary><b>Claude Code plugin / Claude desktop one-click extension</b> (click to expand)</summary>
Claude Code plugin (the MCP server plus the workflow skill in one step):
/plugin marketplace add reality-opened/openreality /plugin install openreality@openreality
Claude desktop extension: download `openreality-mcp-<version>.mcpb` from the [latest release](https://github.com/reality-opened/openreality/releases/latest) and double-click it, or drag it into the Claude desktop window. No terminal needed.
</details>
Then sign in once (opens your browser, stores a revocable API key on your machine):
npx -y openreality-mcp login
Scan a room with your phone at [open-reality.io](https://open-reality.io), or just ask your assistant to upload a video file. Full per-client setup: [open-reality.io/mcp](https://open-reality.io/mcp).
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 17 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- {
- "name": "openreality",
- "owner": {
- "name": "reality-opened",
- "url": "https://open-reality.io"
- },
- "description": "Open Reality for Claude Code: phone video to an AI-queryable 3D scene (measure, plan paths, export robot-training data).",
- "version": "0.4.0",
- "plugins": [
- {
- "name": "openreality",
- "source": "./mcp",
- "description": "The openreality MCP server (41 tools) plus the workflow skill. Install, then run `npx -y openreality-mcp login` once.",
- "version": "0.4.0"
- }
- ]
- }
README.md · README · 220 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- <p align="center">
- <a href="https://open-reality.io"><img src="docs/assets/hero.svg" alt="Open Reality: phone video to AI-queryable 3D scenes" width="100%" /></a>
- </p>
- <p align="center">
- <b>The Agentic OS for any hardware</b><br/>
- Open Reality gives your AI agent the ability to use any hardware. Vision + Action.
- </p>
- <p align="center">
- <a href="https://www.npmjs.com/package/openreality-mcp"><img src="https://img.shields.io/npm/v/openreality-mcp?color=cb3837&logo=npm&label=openreality-mcp" alt="npm" /></a>
- <!-- <a href="https://github.com/reality-opened/openreality/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/reality-opened/openreality/ci.yml?branch=main&logo=github&label=CI" alt="CI" /></a> -->
- <a href="LICENSE"><img src="https://img.shields.io/badge/license-BSD--2--Clause-blue" alt="license" /></a>
- <img src="https://img.shields.io/badge/MCP-41%20tools-8A2BE2" alt="MCP tools" />
- <img src="https://img.shields.io/badge/self--host-your%20GPU%20or%20Modal-0fa573" alt="self-host" />
- <!-- <img src="https://img.shields.io/badge/node-%E2%89%A520-339933?logo=node.js&logoColor=white" alt="node" />
- <img src="https://img.shields.io/badge/python-3.11-3776AB?logo=python&logoColor=white" alt="python" /> -->
- <a href="#-contributing"><img src="https://img.shields.io/badge/PRs-welcome-e07070" alt="PRs welcome" /></a>
- </p>
- <p align="center">
- <a href="README.md"><img src="https://img.shields.io/badge/English-README-d4a017" alt="English" /></a>
- <a href="README.zh.md"><img src="https://img.shields.io/badge/简体中文-点击查看-e07070" alt="简体中文" /></a>
- </p>
- <p align="center">
- <a href="https://open-reality.io">Website</a> ·
- <a href="#-get-started-in-60-seconds">Get started</a> ·
- <a href="#-things-you-can-ask">Things you can ask</a> ·
- <a href="#-run-it-yourself">Self-host</a> ·
- <a href="mcp/README.md">MCP docs</a> ·
- <a href="#-contributing">Contributing</a>
- </p>
- ---
- <p align="center">
- <img src="docs/demo/demo-gif-v1.gif" alt="Demo v1: a Claude Code session driving Open Reality beside a synchronized 3D scene panel: install, upload, honest relative units, calibration to metres, robot-data export" width="900" />
- <br/>
- <sub>Demo v1: a real Claude Code (Opus) session against the built-in offline simulator, beside a synchronized scene panel. Video: <a href="docs/demo/demo-video-v1.mp4">docs/demo/demo-video-v1.mp4</a>. Rendered from <a href="docs/demo/">docs/demo/</a> with <a href="https://github.com/charmbracelet/vhs">VHS</a>.</sub>
- </p>
- <p align="center">
- <img src="https://github.com/reality-opened/web-assets/releases/download/v1/vggt_slam_demo.gif" alt="Live 3D reconstruction from a handheld phone video (VGGT-SLAM)" width="680" />
- <br/>
- <sub>Live reconstruction from a handheld video: the camera path and the 3D scene build up together. Scan demo and the initial technical idea: <a href="https://github.com/MIT-SPARK/VGGT-SLAM">VGGT-SLAM</a> by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). Open Reality's reconstruction core builds on their work.</sub>
- </p>
- ## Utilities
- | | |
- |---|---|
- | 🎥 **Video in, 3D scene out** | Upload a phone video. A few minutes later you have a persistent 3D scene. |
- | 📏 **Measurement** | Distances and angles between any points. Numbers are only called metres after you calibrate with real distance; otherwise relative.|
- | 🧭 **Path planning** | Plan a route through the scanned free space to an object or a point. |
- | 🤖 **Robot-training exports** | Turns a scan into LeRobot / GR00T style datasets or an Isaac Sim or MuJoCo scene (hosted service). |
- | 🕵️ **Scene agents** | Server-side agents that survey, label, and answer questions about a scene. |
- | 🛠️ **42 tools for your AI** | Everything is exposed through MCP. |
- | 🧪 **Offline simulator** | A mock backend fakes the entire workflow with fixture data, so you can develop and demo with no account and no GPU. |
- | 🏠 **Self-hostable** | The full server runs on your own GPU box or your own Modal account, no account with us needed. |
- ## Quickstart
- Add the tools to your AI assistant:
- ```bash
- claude mcp add openreality -- npx -y openreality-mcp serve
- codex mcp add openreality -- npx -y openreality-mcp serve
- ```
- <details>
- <summary><b>Claude desktop / Cursor</b> (click to expand)</summary>
- Add this to `claude_desktop_config.json` (Settings → Developer → Edit Config)
- or `~/.cursor/mcp.json`:
- ```json
- {
- "mcpServers": {
- "openreality": {
- "command": "npx",
- "args": ["-y", "openreality-mcp", "serve"]
- }
- }
- }
- ```
- </details>
- <details>
- <summary><b>Claude Code plugin / Claude desktop one-click extension</b> (click to expand)</summary>
- Claude Code plugin (the MCP server plus the workflow skill in one step):
- ```
- /plugin marketplace add reality-opened/openreality
- /plugin install openreality@openreality
- ```
- Claude desktop extension: download `openreality-mcp-<version>.mcpb` from the
- [latest release](https://github.com/reality-opened/openreality/releases/latest)
- and double-click it, or drag it into the Claude desktop window. No terminal needed.
- </details>
- Then sign in once (opens your browser, stores a revocable API key on your machine):
- ```bash
- npx -y openreality-mcp login
- ```
- Scan a room with your phone at [open-reality.io](https://open-reality.io), or just ask
- your assistant to upload a video file. Full per-client setup:
- [open-reality.io/mcp](https://open-reality.io/mcp).
- ## Examples
- Once connected, talk to your assistant like this:
- > "Upload `~/Videos/kitchen.mp4` and reconstruct it."
- > "What objects are in my latest scan, and how big is the room?"
- > "The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."
- > "Plan a path from the door to the desk and describe it."
- > "Export this scan as robot-training data and save the zip locally."
- ## Self Hosting
- The whole workflow is self-hostable. Read
- [`server/docs/self-hosting.md`](server/docs/self-hosting.md); the short version:
- ```bash
- git clone https://github.com/reality-opened/openreality
- cd openreality/server
- # Path A: your own GPU box (one process, local disk)
- python -m server.selfhost --data-dir ~/openreality-data
- # Path B: your own Modal account (CPU web server + GPU worker)
- modal deploy modal_selfhost.py
- ```
- Self-hosted servers need no account: a single token printed at first start is
- your login, and the MCP client connects with `OPENREALITY_URL` plus that token.
- > [!IMPORTANT]
- > **Licensing.** This repo is BSD-2-Clause, but the 3D reconstruction model a
- > self-hosted server downloads (VGGT-1B and the VGGT code it runs on) is
- > licensed by its owners as **CC BY-NC 4.0, non-commercial use only**. Nothing
- > here redistributes it; your server fetches it from the source under their
- > terms. For commercial use, use the hosted service (which runs a commercially
- > licensed model) or get your own license from the model owners.
- ## Repo index
- | Directory | What it is | Ships as |
- |---|---|---|
- | [`mcp/`](mcp/) | The MCP server: 42 tools, scene resources, the offline simulator, and a full test suite. Developed here directly. | npm [`openreality-mcp`](https://www.npmjs.com/package/openreality-mcp) |
- | [`server/`](server/) | The backend: turns videos into persistent scenes and serves measurement, planning, agents, and exports over a plain REST API. | source (public mirror) |
- | [`core/`](core/) | The 3D reconstruction library: camera tracking and dense geometry from ordinary video (the VGGT-SLAM 2.0 line), plus metric calibration, object detection, and splat export. | source (public mirror) |
- `server/` and `core/` are curated mirrors of our private working repos, synced
- by hand; each carries a `MIRROR.md` that says exactly what is included and how
- it is synced. `mcp/` is developed in this repo directly.
- ## Technical details
- ```mermaid
- flowchart LR
- A["📱 phone video"] --> B["server: 3D reconstruction<br/>(GPU job)"]
- B --> C[("persistent scene:<br/>geometry + objects + report")]
- C --> D["41 MCP tools"]
- D --> E["🤖 Claude / Codex / Cursor"]
- E -->|"measure · navigate · export"| D
- ```
- The MCP process always runs on your machine and holds your credentials; every
- tool call is a typed REST call to a server (ours or yours). Big files are
- written to your disk, never pasted into the AI's context. Server refusals and
- uncertainty labels reach the AI unedited, so it cannot pretend a relative
- number is metres.
- The measurement rule is explained in
- [Why our 3D tools refuse to say "metres" until you prove it](docs/posts/relative-units-are-not-metres.md).
- ## 🙏 Acknowledgements
- Open Reality stands on other people's research and code. Thank you to:
- - [VGGT-SLAM](https://github.com/MIT-SPARK/VGGT-SLAM) by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). The reconstruction core in `core/` is the VGGT-SLAM 2.0 line, and the scan demo above is theirs.
- - [VGGT](https://github.com/facebookresearch/vggt) (Meta AI), the feed-forward 3D backbone a self-hosted server fetches under its own licence (see the note above).
- - [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2), the metric-depth model behind our scale-drift correction between map segments.
- - [gsplat](https://github.com/nerfstudio-project/gsplat) from the Nerfstudio project, the optional GPU refinement in our splat export.
- - [SAM 3](https://github.com/facebookresearch/sam3) (Meta AI) for segmentation, and SAM 3D Objects and [TRELLIS](https://github.com/microsoft/TRELLIS) (Microsoft) for object completion and variants on the hosted service.
- - [LeRobot](https://github.com/huggingface/lerobot) (Hugging Face), NVIDIA [GR00T](https://github.com/NVIDIA/Isaac-GR00T) and Isaac Sim, whose dataset and scene formats our robot-training exports target.
- - [GTSAM](https://github.com/borglab/gtsam) for factor-graph optimization, [cordis](https://github.com/cordiverse/cordis) for the plugin runtime, and the [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk).
- ## 🤝 Contributing
- Issues and pull requests are welcome on any component. Changes to `mcp/` land
- here directly; fixes to `server/` and `core/` are folded back into the private
- working repos and re-synced out. If you self-host and something breaks, an
- issue with your logs is a gift: the self-host paths are young.
- ## License
- [BSD-2-Clause](LICENSE) for everything in this repository. Third-party models
- are fetched from their owners under their own licenses (see the licensing note
- above and [`server/docs/self-hosting.md`](server/docs/self-hosting.md)).
- ---
- <p align="center">
- <sub>Built by <a href="https://github.com/reality-opened">reality-opened</a> · <a href="https://open-reality.io">open-reality.io</a></sub>
- </p>
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.