Skip to content
Back to search

openreality

Open Reality for Claude Code: phone video to an AI-queryable 3D scene (measure, plan paths, export robot-training data).

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/reality-opened--openreality/badge.svg)](/listings/reality-opened--openreality)

HTML

<a href="/listings/reality-opened--openreality"><img src="/api/listings/reality-opened--openreality/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Add the tools to your AI assistant:

claude mcp add openreality -- npx -y openreality-mcp serve

codex mcp add openreality -- npx -y openreality-mcp serve

<details> <summary><b>Claude desktop / Cursor</b> (click to expand)</summary>

Add this to `claude_desktop_config.json` (Settings → Developer → Edit Config) or `~/.cursor/mcp.json`:

{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": ["-y", "openreality-mcp", "serve"]
    }
  }
}

</details>

<details> <summary><b>Claude Code plugin / Claude desktop one-click extension</b> (click to expand)</summary>

Claude Code plugin (the MCP server plus the workflow skill in one step):

/plugin marketplace add reality-opened/openreality
/plugin install openreality@openreality

Claude desktop extension: download `openreality-mcp-<version>.mcpb` from the [latest release](https://github.com/reality-opened/openreality/releases/latest) and double-click it, or drag it into the Claude desktop window. No terminal needed.

</details>

Then sign in once (opens your browser, stores a revocable API key on your machine):

npx -y openreality-mcp login

Scan a room with your phone at [open-reality.io](https://open-reality.io), or just ask your assistant to upload a video file. Full per-client setup: [open-reality.io/mcp](https://open-reality.io/mcp).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 17 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. {
  2. "name": "openreality",
  3. "owner": {
  4. "name": "reality-opened",
  5. "url": "https://open-reality.io"
  6. },
  7. "description": "Open Reality for Claude Code: phone video to an AI-queryable 3D scene (measure, plan paths, export robot-training data).",
  8. "version": "0.4.0",
  9. "plugins": [
  10. {
  11. "name": "openreality",
  12. "source": "./mcp",
  13. "description": "The openreality MCP server (41 tools) plus the workflow skill. Install, then run `npx -y openreality-mcp login` once.",
  14. "version": "0.4.0"
  15. }
  16. ]
  17. }
README.md · README · 220 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <a href="https://open-reality.io"><img src="docs/assets/hero.svg" alt="Open Reality: phone video to AI-queryable 3D scenes" width="100%" /></a>
  3. </p>
  4. <p align="center">
  5. <b>The Agentic OS for any hardware</b><br/>
  6. Open Reality gives your AI agent the ability to use any hardware. Vision + Action.
  7. </p>
  8. <p align="center">
  9. <a href="https://www.npmjs.com/package/openreality-mcp"><img src="https://img.shields.io/npm/v/openreality-mcp?color=cb3837&logo=npm&label=openreality-mcp" alt="npm" /></a>
  10. <!-- <a href="https://github.com/reality-opened/openreality/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/reality-opened/openreality/ci.yml?branch=main&logo=github&label=CI" alt="CI" /></a> -->
  11. <a href="LICENSE"><img src="https://img.shields.io/badge/license-BSD--2--Clause-blue" alt="license" /></a>
  12. <img src="https://img.shields.io/badge/MCP-41%20tools-8A2BE2" alt="MCP tools" />
  13. <img src="https://img.shields.io/badge/self--host-your%20GPU%20or%20Modal-0fa573" alt="self-host" />
  14. <!-- <img src="https://img.shields.io/badge/node-%E2%89%A520-339933?logo=node.js&logoColor=white" alt="node" />
  15. <img src="https://img.shields.io/badge/python-3.11-3776AB?logo=python&logoColor=white" alt="python" /> -->
  16. <a href="#-contributing"><img src="https://img.shields.io/badge/PRs-welcome-e07070" alt="PRs welcome" /></a>
  17. </p>
  18. <p align="center">
  19. <a href="README.md"><img src="https://img.shields.io/badge/English-README-d4a017" alt="English" /></a>
  20. <a href="README.zh.md"><img src="https://img.shields.io/badge/简体中文-点击查看-e07070" alt="简体中文" /></a>
  21. </p>
  22. <p align="center">
  23. <a href="https://open-reality.io">Website</a> ·
  24. <a href="#-get-started-in-60-seconds">Get started</a> ·
  25. <a href="#-things-you-can-ask">Things you can ask</a> ·
  26. <a href="#-run-it-yourself">Self-host</a> ·
  27. <a href="mcp/README.md">MCP docs</a> ·
  28. <a href="#-contributing">Contributing</a>
  29. </p>
  30. ---
  31. <p align="center">
  32. <img src="docs/demo/demo-gif-v1.gif" alt="Demo v1: a Claude Code session driving Open Reality beside a synchronized 3D scene panel: install, upload, honest relative units, calibration to metres, robot-data export" width="900" />
  33. <br/>
  34. <sub>Demo v1: a real Claude Code (Opus) session against the built-in offline simulator, beside a synchronized scene panel. Video: <a href="docs/demo/demo-video-v1.mp4">docs/demo/demo-video-v1.mp4</a>. Rendered from <a href="docs/demo/">docs/demo/</a> with <a href="https://github.com/charmbracelet/vhs">VHS</a>.</sub>
  35. </p>
  36. <p align="center">
  37. <img src="https://github.com/reality-opened/web-assets/releases/download/v1/vggt_slam_demo.gif" alt="Live 3D reconstruction from a handheld phone video (VGGT-SLAM)" width="680" />
  38. <br/>
  39. <sub>Live reconstruction from a handheld video: the camera path and the 3D scene build up together. Scan demo and the initial technical idea: <a href="https://github.com/MIT-SPARK/VGGT-SLAM">VGGT-SLAM</a> by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). Open Reality's reconstruction core builds on their work.</sub>
  40. </p>
  41. ## Utilities
  42. | | |
  43. |---|---|
  44. | 🎥 &nbsp;**Video in, 3D scene out** | Upload a phone video. A few minutes later you have a persistent 3D scene. |
  45. | 📏 &nbsp;**Measurement** | Distances and angles between any points. Numbers are only called metres after you calibrate with real distance; otherwise relative.|
  46. | 🧭 &nbsp;**Path planning** | Plan a route through the scanned free space to an object or a point. |
  47. | 🤖 &nbsp;**Robot-training exports** | Turns a scan into LeRobot / GR00T style datasets or an Isaac Sim or MuJoCo scene (hosted service). |
  48. | 🕵️ &nbsp;**Scene agents** | Server-side agents that survey, label, and answer questions about a scene. |
  49. | 🛠️ &nbsp;**42 tools for your AI** | Everything is exposed through MCP. |
  50. | 🧪 &nbsp;**Offline simulator** | A mock backend fakes the entire workflow with fixture data, so you can develop and demo with no account and no GPU. |
  51. | 🏠 &nbsp;**Self-hostable** | The full server runs on your own GPU box or your own Modal account, no account with us needed. |
  52. ## Quickstart
  53. Add the tools to your AI assistant:
  54. ```bash
  55. claude mcp add openreality -- npx -y openreality-mcp serve
  56. codex mcp add openreality -- npx -y openreality-mcp serve
  57. ```
  58. <details>
  59. <summary><b>Claude desktop / Cursor</b> (click to expand)</summary>
  60. Add this to `claude_desktop_config.json` (Settings → Developer → Edit Config)
  61. or `~/.cursor/mcp.json`:
  62. ```json
  63. {
  64. "mcpServers": {
  65. "openreality": {
  66. "command": "npx",
  67. "args": ["-y", "openreality-mcp", "serve"]
  68. }
  69. }
  70. }
  71. ```
  72. </details>
  73. <details>
  74. <summary><b>Claude Code plugin / Claude desktop one-click extension</b> (click to expand)</summary>
  75. Claude Code plugin (the MCP server plus the workflow skill in one step):
  76. ```
  77. /plugin marketplace add reality-opened/openreality
  78. /plugin install openreality@openreality
  79. ```
  80. Claude desktop extension: download `openreality-mcp-<version>.mcpb` from the
  81. [latest release](https://github.com/reality-opened/openreality/releases/latest)
  82. and double-click it, or drag it into the Claude desktop window. No terminal needed.
  83. </details>
  84. Then sign in once (opens your browser, stores a revocable API key on your machine):
  85. ```bash
  86. npx -y openreality-mcp login
  87. ```
  88. Scan a room with your phone at [open-reality.io](https://open-reality.io), or just ask
  89. your assistant to upload a video file. Full per-client setup:
  90. [open-reality.io/mcp](https://open-reality.io/mcp).
  91. ## Examples
  92. Once connected, talk to your assistant like this:
  93. > "Upload `~/Videos/kitchen.mp4` and reconstruct it."
  94. > "What objects are in my latest scan, and how big is the room?"
  95. > "The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."
  96. > "Plan a path from the door to the desk and describe it."
  97. > "Export this scan as robot-training data and save the zip locally."
  98. ## Self Hosting
  99. The whole workflow is self-hostable. Read
  100. [`server/docs/self-hosting.md`](server/docs/self-hosting.md); the short version:
  101. ```bash
  102. git clone https://github.com/reality-opened/openreality
  103. cd openreality/server
  104. # Path A: your own GPU box (one process, local disk)
  105. python -m server.selfhost --data-dir ~/openreality-data
  106. # Path B: your own Modal account (CPU web server + GPU worker)
  107. modal deploy modal_selfhost.py
  108. ```
  109. Self-hosted servers need no account: a single token printed at first start is
  110. your login, and the MCP client connects with `OPENREALITY_URL` plus that token.
  111. > [!IMPORTANT]
  112. > **Licensing.** This repo is BSD-2-Clause, but the 3D reconstruction model a
  113. > self-hosted server downloads (VGGT-1B and the VGGT code it runs on) is
  114. > licensed by its owners as **CC BY-NC 4.0, non-commercial use only**. Nothing
  115. > here redistributes it; your server fetches it from the source under their
  116. > terms. For commercial use, use the hosted service (which runs a commercially
  117. > licensed model) or get your own license from the model owners.
  118. ## Repo index
  119. | Directory | What it is | Ships as |
  120. |---|---|---|
  121. | [`mcp/`](mcp/) | The MCP server: 42 tools, scene resources, the offline simulator, and a full test suite. Developed here directly. | npm [`openreality-mcp`](https://www.npmjs.com/package/openreality-mcp) |
  122. | [`server/`](server/) | The backend: turns videos into persistent scenes and serves measurement, planning, agents, and exports over a plain REST API. | source (public mirror) |
  123. | [`core/`](core/) | The 3D reconstruction library: camera tracking and dense geometry from ordinary video (the VGGT-SLAM 2.0 line), plus metric calibration, object detection, and splat export. | source (public mirror) |
  124. `server/` and `core/` are curated mirrors of our private working repos, synced
  125. by hand; each carries a `MIRROR.md` that says exactly what is included and how
  126. it is synced. `mcp/` is developed in this repo directly.
  127. ## Technical details
  128. ```mermaid
  129. flowchart LR
  130. A["📱 phone video"] --> B["server: 3D reconstruction<br/>(GPU job)"]
  131. B --> C[("persistent scene:<br/>geometry + objects + report")]
  132. C --> D["41 MCP tools"]
  133. D --> E["🤖 Claude / Codex / Cursor"]
  134. E -->|"measure · navigate · export"| D
  135. ```
  136. The MCP process always runs on your machine and holds your credentials; every
  137. tool call is a typed REST call to a server (ours or yours). Big files are
  138. written to your disk, never pasted into the AI's context. Server refusals and
  139. uncertainty labels reach the AI unedited, so it cannot pretend a relative
  140. number is metres.
  141. The measurement rule is explained in
  142. [Why our 3D tools refuse to say "metres" until you prove it](docs/posts/relative-units-are-not-metres.md).
  143. ## 🙏 Acknowledgements
  144. Open Reality stands on other people's research and code. Thank you to:
  145. - [VGGT-SLAM](https://github.com/MIT-SPARK/VGGT-SLAM) by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). The reconstruction core in `core/` is the VGGT-SLAM 2.0 line, and the scan demo above is theirs.
  146. - [VGGT](https://github.com/facebookresearch/vggt) (Meta AI), the feed-forward 3D backbone a self-hosted server fetches under its own licence (see the note above).
  147. - [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2), the metric-depth model behind our scale-drift correction between map segments.
  148. - [gsplat](https://github.com/nerfstudio-project/gsplat) from the Nerfstudio project, the optional GPU refinement in our splat export.
  149. - [SAM 3](https://github.com/facebookresearch/sam3) (Meta AI) for segmentation, and SAM 3D Objects and [TRELLIS](https://github.com/microsoft/TRELLIS) (Microsoft) for object completion and variants on the hosted service.
  150. - [LeRobot](https://github.com/huggingface/lerobot) (Hugging Face), NVIDIA [GR00T](https://github.com/NVIDIA/Isaac-GR00T) and Isaac Sim, whose dataset and scene formats our robot-training exports target.
  151. - [GTSAM](https://github.com/borglab/gtsam) for factor-graph optimization, [cordis](https://github.com/cordiverse/cordis) for the plugin runtime, and the [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk).
  152. ## 🤝 Contributing
  153. Issues and pull requests are welcome on any component. Changes to `mcp/` land
  154. here directly; fixes to `server/` and `core/` are folded back into the private
  155. working repos and re-synced out. If you self-host and something breaks, an
  156. issue with your logs is a gift: the self-host paths are young.
  157. ## License
  158. [BSD-2-Clause](LICENSE) for everything in this repository. Third-party models
  159. are fetched from their owners under their own licenses (see the licensing note
  160. above and [`server/docs/self-hosting.md`](server/docs/self-hosting.md)).
  161. ---
  162. <p align="center">
  163. <sub>Built by <a href="https://github.com/reality-opened">reality-opened</a> · <a href="https://open-reality.io">open-reality.io</a></sub>
  164. </p>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.