CopilotKit/openmuse

Alpha, for self-hosting and building on. Open-ended reasoning, live Google accounts, and CopilotKit Rich Threads require their own configuration. See what is verified and the roadmap.

On iPhone, ask OpenMuse to find interesting stories on Hacker News and summarize CopilotKit. On desktop, ask it to check the school-trip email, open the message, and research exhibits at Monterey Bay Aquarium. The agent shows email and browser results inline. Take control opens that same browser session when you need it.

The 38-second iPhone and 42-second desktop web demos show the current interface, framed in 16:9. The send arrow becomes a stop square inside the input pill while the agent replies, then switches back. Stopping keeps your draft intact. See the recording notes for the model setup and reproduction steps.

Mobile MP4 · Web MP4 · Recording details and reproduction

The Jev aquarium-trip demo walks through a fictional school email, clarification choices, sourced exhibit cards, hands-on preference refinement, and a confirmed selection. Watch the 83-second live Jev recording, where TypeSafe Jev makes the decisions and a scripted agent keeps the trip scenario repeatable. A scripted-decision sample recording is also available.

OpenMuse is a personal-agent application with an agent computer, visible work, and rich results. It runs its own server, task worker, and browser worker. You can inspect and change the source under the MIT license.

The computer combines persistent Chromium and an optional Linux workspace. The agent can browse public pages, run commands in its own container, work with files, and move PDFs between the computer and the app. You can open its browser or terminal and continue the work. Graphical desktops and autonomous checkout remain future work.

The feature inventory describes implemented capabilities and planned extensions. Health/bank/social connectors, device push, voice, generated executable tools, and automatic reservations/payments are on the roadmap.

Requirements: Node 24 LTS, pnpm 11.19.0, and a CopilotKit Intelligence project key. The local sample app needs no model, Google account, or Docker.

git clone https://github.com/CopilotKit/OpenMuse.git openmuse
cd openmuse
pnpm install --frozen-lockfile
cp .env.example .env
npx copilotkit@latest login
npx copilotkit@latest project select
# Set CPK_INTELLIGENCE_API_KEY in .env to the generated server-only project key.
# Keep the CLI-generated CPK_TELEMETRY_ID too, including in deployed server environments.
pnpm dev

In another terminal:

pnpm dev:web

Open localhost:8081. The API runs at localhost:8787/api/health.

1. In Chat, send “Complete the permission slip”. Open the task, supply fictional form values, inspect the saved PDF, and review the prepared reply. This writes only to the local mailbox.
2. In Goals → Track, create a built-in availability watch, then change the built-in test page to trigger an alert.
3. In Menu → Delegate task → Finance, use Try example transactions to create an interactive spending tracker.
4. Start the browser worker and configure a model, then ask “Check out Hacker News for cool stuff” or “Summarize copilotkit.ai”. Follow the browser inline and use Take control to open its session. For a model-free version of this flow, follow the AI Mock demo setup.

For iOS or Android, use pnpm --dir apps/mobile ios or pnpm --dir apps/mobile android. Xcode or Android tooling is required. The PDF reader needs an Expo development build; use native setup.

render.yaml deploys three services: the API, the web app, and a private browser. The API answers / with JSON, so the UI is its own static site.

1. Click Deploy to Render. Wait until openmuse-api, openmuse-web, and openmuse-browser are live.
2. On openmuse-api, open Environment and copy OPENMUSE_ACCESS_KEY.
3. Open the openmuse-web URL and sign in with that key.
4. Send a message.

The deploy form asks for three values you provide. Render generates the other two.

Health check: https://<openmuse-api>/api/health.

Standard is the smallest plan that stays up. At 512 MB the process runs out of memory before it binds a port, because PGlite loads an embedded Postgres build.

The disk holds the database, PDFs, and the signing key. A redeploy without it wipes that data. Chat threads are stored by CopilotKit Intelligence, so a thread can still load after you sign back in even when the disk was never attached.

Live mode is required. Render binds 0.0.0.0, and sample mode rejects any host that is not loopback. The Blueprint sets WORKSPACE_MODE=live.

Browsing is included, and you can take it out. openmuse-browser is a private service, so it has no public URL. The API reaches it at http://openmuse-browser:8790 with a token Render generates. If the private hostname is not openmuse-browser, set BROWSER_WORKER_URL to http://<that-host>:8790. To deploy without it, delete the openmuse-browser service and the BROWSER_WORKER_URL and WORKER_TOKEN entries on openmuse-api. Chat, drafts, and tasks still run. Page reads, screenshots, and Take control do not.

The Docker computer and Google mail or calendar need the setup in the sections below. This Blueprint does not start them.

Copy the commented settings in .env.example into your private .env:

1. Set AGENT_BACKEND=model, MODEL=provider/model-id, and the matching provider key. CopilotKit supports the configured OpenAI, Anthropic or Google provider. Fictional data can still be used with a real model. Provider keys stay on the server.
2. Create or select a CopilotKit Intelligence project with npx copilotkit@latest login and npx copilotkit@latest project select. Keep the generated CPK_INTELLIGENCE_API_KEY on the server.
3. For personal mail/calendar, set WORKSPACE_MODE=live, the generated CPK_INTELLIGENCE_API_KEY, a random OPENMUSE_ACCESS_KEY of at least 24 characters, and TOKEN_ENCRYPTION_KEY containing 32 random bytes encoded as base64. Restart the API.
4. Configure a Google OAuth web client with Gmail and Calendar APIs enabled. Set GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET; register ${PUBLIC_API_URL}/api/google/callback as its redirect URI. Configure consent/test-user access in your Google project.
5. Open Apps → Gmail (or Google Calendar), connect read access, and grant write access when needed. Every send or calendar change still requires its own stored review. Changing/disconnecting the account invalidates pending connection-bound work.

Google credentials are encrypted at rest. File URLs and browser consoles use short-lived signatures. This deployment uses one owner protected by a shared access key; it is not a multi-tenant authentication system. Use HTTPS and restricted network access for a remote host. Keep the default local-data mode on loopback.

Web search is enabled by default. Set WEB_SEARCH_ENABLED=false on the API and any separate task worker to disable it. With a configured model, delegated tasks and built-in model chat (AGENT_BACKEND=model) use Parallel's free, keyless Search MCP as their built-in search provider. Ask, for example, “Find the official CopilotKit React Native setup instructions and cite the sources.” Results include source URLs and excerpts; delegated tasks save them as evidence. Scripted sample chat and external AG-UI conversations keep their existing tools.

Search sends model-generated queries and context, which may include information from your conversation or task, plus a random per-chat/task session identifier to Parallel. See Parallel's privacy policy. Free access is rate limited; failures are reported without a paid fallback. Stopping chat or interrupting a task cancels its search.

Set BROWSER_WORKER_URL=http://127.0.0.1:8790 and a random WORKER_TOKEN of at least 32 characters in .env.

pnpm --dir apps/worker exec playwright install chromium
pnpm dev:browser

Or use docker compose --env-file .env -f infra/compose.yaml up --build -d. The same token must reach the API and worker. Sessions have persistent Chromium profiles; the app can open a live screenshot console and import PDF downloads. Agent tools can read public pages and hand interactive work to the person. Worker setup and boundaries.

Build the computer image, enable it on the API, then open Computer → Terminal → Start computer:

docker build -t openmuse-computer:local apps/computer
COMPUTER_ENABLED=true pnpm dev

The API needs access to the Docker CLI and engine. Commands run in a nonroot container with no host-directory mounts or credentials. A named /workspace volume retains files when stopped. Terminal networking is disabled; public web access uses the browser worker. Commands have a 30-second limit and saved output/exit receipts. Files supports folders, text editing, and PDF transfer to/from Documents. This is a Linux container, not a full operating-system VM. Setup, Colima option, and boundaries.

By default, embedded PGlite, documents and the signing key live in .openmuse/; browser profiles live in .openmuse/browser-profiles/. Keep that directory private and back it up. The API hosts the task worker. The host must remain running for background work.

For a separate task worker, configure the same DATABASE_URL, secrets and shared DATA_DIR for both processes, then set TASK_WORKER_ENABLED=false on the API and run pnpm dev:worker. PGlite cannot be opened by separate processes. Production commands are pnpm build:server, pnpm start and pnpm start:worker. Run one API instance; task workers coordinate through SQL leases.

No hidden retry occurs after an uncertain external write. Review its provider outcome before creating a replacement. Pausing/cancelling prevents subsequent task steps; an already approved in-flight provider request may finish.

Every deployment requires CPK_INTELLIGENCE_API_KEY on the API server for CopilotKit Intelligence conversation persistence and replay. Create or select a project with npx copilotkit@latest login and npx copilotkit@latest project select, set the generated server-only key, and restart the API. The native menu uses useThreads; rich tool results link back to saved tasks, documents, and browser sessions.

Intelligence is a separate service and is not included in this repository's MIT license. No project key is shipped. Configuration and validation boundaries.

The CopilotKit SDK sends runtime telemetry through the shared CopilotKit sink to PostHog with accessibility_title: "OpenMuse". Telemetry defaults to full capture; explicit SDK opt-outs and sampling settings take precedence. Keep the CLI-generated CPK_TELEMETRY_ID alongside the server-only project key when deploying. Intelligence records completed account signups separately; OpenMuse runtime usage can be joined to those accounts after the sink resolves the identity. Setup, opt-out and PostHog validation.

flowchart TD
  Client[Expo / React Native / Web] -->|AG-UI and authenticated API| API[Hono + CopilotKit runtime]
  API --> Tasks[Durable task worker]
  API --> Threads[CopilotKit Intelligence required in every mode]
  API --> Store[(PGlite or PostgreSQL)]
  Tasks --> Store
  Tasks --> Review[Stored action review]
  Review --> Google[Gmail / Calendar adapters]
  Tasks --> Browser[Chromium worker + persistent profiles]
  API --> Browser
  API --> Computer[Optional Docker Linux computer]
  Tasks --> Computer
  Computer --> Volume[(Persistent workspace volume)]
  Tasks --> Files[PDF files + structured artifacts]
  API -. future adapter .-> OpenBot[OpenBot]

OpenMuse's native client and personal-agent workflows are independent of OpenBot. The disabled OpenBot adapter is pinned and contract-tested against upstream interfaces. Live user/session bridging, routine mapping, and computer backend wiring remain future work. OpenBot's Intelligence runtime is not a raw AG-UI endpoint. Integration contract.

pnpm lint
pnpm typecheck
pnpm test
pnpm build:server
pnpm build:web
pnpm build:ios
pnpm build:android
pnpm --dir apps/worker typecheck
pnpm test:browser
pnpm test:computer

Platform build scripts export JavaScript/Hermes bundles; they do not produce signed app binaries. Browser checks use controlled HTTP fixtures; the local check requires installed Chromium, and the container check requires Docker and registry access to build its image. CI also exercises the browser and Linux computer containers. See contribution guidance and verification results.

Issues and pull requests are welcome. Start with CONTRIBUTING.md, ROADMAP.md, and the security policy.

MIT licensed. Built by CopilotKit. Its original interface and fictional assets are included. Website, email, and document content supplies evidence, not permission to act.