reality-opened/openreality

The Agentic OS for any hardware
Open Reality gives your AI agent the ability to use any hardware. Vision + Action.

Website · Get started · Things you can ask · Self-host · MCP docs · Contributing

_{Demo v1: a real Claude Code (Opus) session against the built-in offline simulator, beside a synchronized scene panel. Video: docs/demo/demo-video-v1.mp4. Rendered from docs/demo/ with VHS.}

_{Live reconstruction from a handheld video: the camera path and the 3D scene build up together. Scan demo and the initial technical idea: VGGT-SLAM by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). Open Reality's reconstruction core builds on their work.}

Add the tools to your AI assistant:

claude mcp add openreality -- npx -y openreality-mcp serve

codex mcp add openreality -- npx -y openreality-mcp serve

{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": ["-y", "openreality-mcp", "serve"]
    }
  }
}

/plugin marketplace add reality-opened/openreality
/plugin install openreality@openreality

Then sign in once (opens your browser, stores a revocable API key on your machine):

npx -y openreality-mcp login

Scan a room with your phone at open-reality.io, or just ask your assistant to upload a video file. Full per-client setup: open-reality.io/mcp.

Once connected, talk to your assistant like this:

"Upload ~/Videos/kitchen.mp4 and reconstruct it."

"What objects are in my latest scan, and how big is the room?"

"The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."

"Plan a path from the door to the desk and describe it."

"Export this scan as robot-training data and save the zip locally."

The whole workflow is self-hostable. Read server/docs/self-hosting.md; the short version:

git clone https://github.com/reality-opened/openreality
cd openreality/server

# Path A: your own GPU box (one process, local disk)
python -m server.selfhost --data-dir ~/openreality-data

# Path B: your own Modal account (CPU web server + GPU worker)
modal deploy modal_selfhost.py

Self-hosted servers need no account: a single token printed at first start is your login, and the MCP client connects with OPENREALITY_URL plus that token.

server/ and core/ are curated mirrors of our private working repos, synced by hand; each carries a MIRROR.md that says exactly what is included and how it is synced. mcp/ is developed in this repo directly.

flowchart LR
    A["📱 phone video"] --> B["server: 3D reconstruction<br/>(GPU job)"]
    B --> C[("persistent scene:<br/>geometry + objects + report")]
    C --> D["41 MCP tools"]
    D --> E["🤖 Claude / Codex / Cursor"]
    E -->|"measure · navigate · export"| D

The MCP process always runs on your machine and holds your credentials; every tool call is a typed REST call to a server (ours or yours). Big files are written to your disk, never pasted into the AI's context. Server refusals and uncertainty labels reach the AI unedited, so it cannot pretend a relative number is metres.

The measurement rule is explained in Why our 3D tools refuse to say "metres" until you prove it.

Open Reality stands on other people's research and code. Thank you to:

• VGGT-SLAM by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). The reconstruction core in core/ is the VGGT-SLAM 2.0 line, and the scan demo above is theirs.
• VGGT (Meta AI), the feed-forward 3D backbone a self-hosted server fetches under its own licence (see the note above).
• Depth Anything V2, the metric-depth model behind our scale-drift correction between map segments.
• gsplat from the Nerfstudio project, the optional GPU refinement in our splat export.
• SAM 3 (Meta AI) for segmentation, and SAM 3D Objects and TRELLIS (Microsoft) for object completion and variants on the hosted service.
• LeRobot (Hugging Face), NVIDIA GR00T and Isaac Sim, whose dataset and scene formats our robot-training exports target.
• GTSAM for factor-graph optimization, cordis for the plugin runtime, and the MCP TypeScript SDK.

Issues and pull requests are welcome on any component. Changes to mcp/ land here directly; fixes to server/ and core/ are folded back into the private working repos and re-synced out. If you self-host and something breaks, an issue with your logs is a gift: the self-host paths are young.

BSD-2-Clause for everything in this repository. Third-party models are fetched from their owners under their own licenses (see the licensing note above and server/docs/self-hosting.md).

_{Built by reality-opened · open-reality.io}