dora-rs/dora

English | 简体中文

dora 1.0 is out. Read the 1.0 release post.

Agentic Dataflow-Oriented Robotic Architecture -- a 100% Rust framework for building real-time robotics and AI applications.

User Guide | 用户指南 (中文)

Built and maintained with agentic engineering -- AI agents do the heavy lifting on code generation, reviews, refactoring, and testing; humans set direction and gate every merge.

• Features
• Installation
• Quick Start
• CLI Commands
• Dataflow Configuration
• Architecture
• Language Support
• Examples
• Development
• Contributing
• License

• 10-17x faster than ROS2 Python -- 100% Rust internals with zero-copy shared memory IPC for messages >4KB, flat latency from 4KB to 4MB payloads
• Zenoh SHM data plane -- nodes publish directly via Zenoh shared memory, bypassing the daemon for 35% lower latency and 3-10x higher throughput on large payloads; automatic network fallback for cross-machine
• Apache Arrow native -- columnar memory format end-to-end with zero serialization overhead; optional Arrow IPC framing for self-describing wire format; shared across all language bindings
• Non-blocking event loop -- Zenoh publishes offloaded to a dedicated drain task; control commands respond in <500ms even under high data throughput

• Single CLI, full lifecycle -- dora run for local dev, dora up/start for distributed prod, plus build, logs, monitoring, record/replay all from one tool
• Declarative YAML dataflows -- define pipelines as directed graphs, connect nodes through typed inputs/outputs, optional type annotations with static validation, override with environment variables
• Multi-language nodes -- write nodes in Rust, Python, C, or C++ with native APIs (not wrappers); mix languages freely in one dataflow
• Reusable modules -- compose sub-graphs as standalone YAML files with typed inputs/outputs, parameters, optional ports, and nested composition (compile-time expansion, zero runtime overhead)
• Hot reload -- live-reload Python operators without restarting the dataflow
• Programmatic builder -- construct dataflows in Python code as an alternative to YAML

• Fault tolerance -- per-node restart policies (never/on-failure/always), exponential backoff, health monitoring, circuit breakers with configurable input timeouts
• Distributed by default -- local shared memory between co-located nodes, automatic Zenoh pub-sub for cross-machine communication, SSH-based cluster management with label scheduling, rolling upgrades, and auto-recovery
• Coordinator HA -- persistent redb-backed state store (default), daemon auto-reconnect with exponential backoff, dataflow records survive coordinator restart (running dataflow reclaim-across-restart is partial, see the open issue tracker)
• Dynamic topology -- add and remove nodes from running dataflows via CLI (dora node add/remove/connect/disconnect) without restarting
• Soft real-time -- optional --rt flag for mlockall + SCHED_FIFO; per-node cpu_affinity pinning in YAML; comprehensive tuning guide for memory locking, kernel params, and container deployment
• OpenTelemetry -- built-in structured logging with rotation/routing, metrics, distributed tracing, and zero-setup trace viewing via CLI

• Record/replay -- capture dataflow messages to .drec files, replay offline at any speed with node substitution for regression testing
• Topic inspection -- topic echo to print live data, topic hz TUI for frequency analysis, topic info for schema and bandwidth
• Resource monitoring -- dora top TUI showing per-node CPU, memory, queue depth, network I/O, restart count, and health status across all machines; --once flag for scriptable JSON snapshots
• Trace inspection -- trace list and trace view for viewing coordinator spans without external infrastructure
• Dataflow visualization -- generate interactive HTML or Mermaid graphs from YAML descriptors

• Communication patterns -- built-in service (request/reply), action (goal/feedback/result), and streaming (session/segment/chunk) patterns via well-known metadata keys; no daemon or YAML changes required
• ROS2 bridge -- bidirectional topics, services, and actions over DDS or native rmw_zenoh_cpp-compatible Zenoh; QoS mapping; Arrow-native type conversion
• Node Hub (package manager) -- pull a reusable node into a dataflow with one line -- hub: dora-yolo@^0.5 -- with cargo-style versioned resolution, reproducible lockfiles (--locked), and typed contracts checked at build time; backed by a git-based public catalog of ready-made nodes for cameras, YOLO, LLMs, TTS, and more. See the Hub guide (unstable)
• In-process operators -- lightweight functions that run inside a shared runtime, avoiding per-node process overhead for simple transformations

cargo install dora-cli           # CLI (dora command)
pip install dora-rs              # Python node/operator API

git clone https://github.com/dora-rs/dora.git
cd dora
cargo build --release -p dora-cli
PATH=$PATH:$(pwd)/target/release

# Python API (requires maturin >= 1.8: pip install maturin)
# Must run from the package directory for dependency resolution
cd apis/python/node && maturin develop --uv && cd ../../..

macOS / Linux:

curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/dora-rs/dora/releases/latest/download/dora-cli-installer.sh | sh

Windows:

powershell -ExecutionPolicy ByPass -c "irm https://github.com/dora-rs/dora/releases/latest/download/dora-cli-installer.ps1 | iex"

cargo install dora-cli --features redb-backend

Important: The PyPI package is dora-rs, not dora. The import name is dora (from dora import Node), but pip install dora installs an unrelated package.

cargo install dora-cli            # or use install script below
pip install dora-rs numpy pyarrow
git clone https://github.com/dora-rs/dora.git && cd dora
dora run examples/python-dataflow/dataflow.yml

This runs a sender -> transformer -> receiver pipeline. Here's what the Python node code looks like:

# sender.py -- sends messages and polls for STOP
from dora import Node
import pyarrow as pa
import time

node = Node()
sent = 0
while sent < 100:
    event = node.try_recv()
    if event is not None and event["type"] == "STOP":
        break
    node.send_output("message", pa.array([sent]))
    sent += 1
    time.sleep(0.1)

# receiver.py -- receives and prints messages
from dora import Node

node = Node()
for event in node:
    if event["type"] == "INPUT":
        print(f"Got {event['id']}: {event['value'].to_pylist()}")
    elif event["type"] == "STOP":
        break

See the Python Getting Started Guide for a full tutorial, or the Python API Reference for complete API docs.

cd examples/rust-dataflow
dora run dataflow.yml

# Terminal 1: start coordinator + daemon
dora up

# Terminal 2: start a dataflow (--debug enables topic inspection)
dora start dataflow.yml --attach --debug

# Terminal 3: monitor
dora list
dora logs <dataflow-id>
dora top

# Stop or restart
dora stop <dataflow-id>
dora restart --name <name>
dora down

# Bring up a multi-machine cluster from a config file
dora cluster up cluster.yml

# Start a dataflow across the cluster
dora start dataflow.yml --name my-app --attach

# Check cluster health
dora cluster status

# Tear down
dora cluster down

See the Distributed Deployment Guide for cluster.yml configuration, label scheduling, systemd services, rolling upgrades, and operational runbooks. The network side — one LAN, a VPN mesh, or isolated subnets joined by zenoh routers — is the Multi-machine Guide.

Reference a node with one line of YAML -- hub: dora-yolo@^0.5 -- and dora build resolves, pins, and type-checks it. See the Hub guide.

For full CLI documentation, see docs/cli.md. For running a dataflow across several machines, start with docs/multi-machine.md; for cluster management, see docs/distributed-deployment.md.

Dataflows are defined in YAML. Each node declares its binary/script, inputs, and outputs:

nodes:
  - id: camera
    build: pip install opencv-video-capture
    path: opencv-video-capture
    inputs:
      tick: dora/timer/millis/20
    outputs:
      - image
    env:
      CAPTURE_PATH: 0
      IMAGE_WIDTH: 640
      IMAGE_HEIGHT: 480

  - id: object-detection
    build: pip install dora-yolo
    path: dora-yolo
    inputs:
      image: camera/image
    outputs:
      - bbox

  - id: plot
    build: pip install dora-rerun
    path: dora-rerun
    inputs:
      image: camera/image
      boxes2d: object-detection/bbox

Built-in timer nodes: dora/timer/millis/<N> and dora/timer/hz/<N>.

Input format: <node-id>/<output-name> to subscribe to another node's output. Long form supports queue_size, queue_policy (drop_oldest or backpressure), and input_timeout. See the YAML Specification for details.

Type annotations: Optionally annotate ports with type URNs for static and runtime validation. See the Type Annotations Guide for the full type library.

nodes:
  - id: camera
    path: camera.py
    outputs:
      - image
    output_types:
      image: std/media/v1/Image

dora validate dataflow.yml                        # static check (warnings)
dora validate --strict-types dataflow.yml         # fail on warnings (CI)
dora build dataflow.yml --strict-types            # type check during build
DORA_RUNTIME_TYPE_CHECK=warn dora run dataflow.yml  # runtime check

Modules: Extract reusable sub-graphs into separate files with module: instead of path:. See the Modules Guide for details.

nodes:
  - id: nav_stack
    module: modules/navigation.module.yml
    inputs:
      goal_pose: localization/goal

• Coordinator -- orchestrates dataflow lifecycle across daemons. Persistent redb state store by default; daemons auto-reconnect on coordinator restart.
• Daemon -- spawns and manages nodes on a single machine. Routes messages and manages Zenoh SHM data plane.
• Runtime -- in-process operator execution engine. Operators run inside the runtime process, avoiding per-operator process overhead.
• Nodes -- standalone processes that communicate via inputs/outputs. Written in Rust, Python, C, or C++.
• Operators -- lightweight functions that run inside the runtime. Faster than nodes for simple transformations.

binaries/
  cli/                  # dora CLI binary
  coordinator/          # Orchestration service
  daemon/               # Node manager + IPC
  runtime/              # In-process operator runtime
  ros2-bridge-node/     # ROS2 bridge binary
  mavlink2-bridge-node/ # MAVLink 2 bridge binary
  record-node/          # Dataflow message recorder
  replay-node/          # Recorded message replayer
libraries/
  core/                 # Descriptor parsing, build utilities
  message/              # Inter-component message types
  arrow-convert/        # Arrow data conversion
  recording/            # .drec recording format
  log-utils/            # Log parsing, merging, formatting
  coordinator-store/    # Persistent coordinator state (redb)
  extensions/
    telemetry/          # OpenTelemetry tracing + metrics
    ros2-bridge/        # ROS2 interop (bridge, msg-gen, arrow, python)
    mavlink2-bridge/    # MAVLink 2 interop (Arrow ↔ MAVLink, TCP/UDP/serial)
    download/           # Download utilities
apis/
  rust/node/            # Rust node API (dora-node-api)
  rust/operator/        # Rust operator API (dora-operator-api)
  python/node/          # Python node API (PyO3)
  python/operator/      # Python operator API (PyO3)
  python/cli/           # Python CLI interface
  c/node/               # C node API
  c/operator/           # C operator API
  c++/node/             # C++ node API (CXX bridge)
  c++/operator/         # C++ operator API (CXX bridge)
examples/               # Example dataflows

Gate meanings (#1716):

• PR-gated — every PR to main runs these tests; merge is blocked on failure.
• Nightly-gated — the daily scheduled run (.github/workflows/nightly.yml) runs these. A failure auto-files a nightly-regression issue but does NOT block PRs.
• Not gated — no automated CI coverage. Regressions surface via user reports.

dora new --lang rust/python template tests run in nightly across all three platforms; C/C++ variants run in nightly on Linux only. Developers who need cross-platform verification before merge can run make qa-test / make qa-examples / make qa-nightly locally. See docs/testing-matrix.md for the full rationale.

See docs/patterns.md for the full guide.

Rust edition 2024; MSRV and default workspace package metadata are tracked in [workspace.package] of the root Cargo.toml. Most crates inherit the workspace version via version.workspace = true; a handful (e.g. apis/rust/operator/types, the examples/error-propagation/* samples) pin their own version independently.

# Build all (excluding Python packages which require maturin)
cargo build --all \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python

# Build specific package
cargo build -p dora-cli

# Run all tests
cargo test --all \
  --exclude dora-runtime-python \
  --exclude dora-node-api-python \
  --exclude dora-operator-api-python \
  --exclude dora-ros2-bridge-python

# Test single package
cargo test -p dora-core

# Smoke tests (requires coordinator/daemon)
cargo test --test example-smoke -- --test-threads=1

cargo clippy --all
cargo fmt --all -- --check

cargo run --example rust-dataflow
cargo run --example python-dataflow
cargo run --example benchmark --release

Dora ships with a three-tier QA system designed for AI-authored code. Everything runs locally first; CI mirrors the same scripts.

make qa-install        # one-time: install cargo-audit, cargo-deny, cargo-llvm-cov, cargo-mutants, cargo-semver-checks
make qa-fast           # ~15s    -- fmt + clippy + audit + unwrap-budget + secret-files + typos + publish-graph (pre-commit)
make qa-full           # ~5-10m  -- qa-fast + tests + coverage (pre-push)
make qa-deep           # ~15m    -- qa-full + mutation testing + semver (target Tier 1 gate, stronger than today's CI; alias: qa-tier1)
make qa-nightly        # ~3-4h -- qa-deep + proptest@1000 + miri + example-smoke + ci-nightly-jobs (full parity with .github/workflows/nightly.yml)
make qa-release-gate   #         -- qa-deep + semver (Tier 3 automatable; audit/dogfood are human)
make qa-mutation-audit # ~10-18h -- full-repo cargo-mutants; deliberate test-quality audit
make qa-examples       # ~15-20m -- run all smoke-eligible example dataflows end-to-end (skips CUDA/ROS2/C++/interactive)

On Ubuntu, install ripgrep separately and install typos-cli with Cargo:

sudo apt update
sudo apt install ripgrep
cargo install typos-cli

Gates in place:

• Supply chain -- cargo-audit + cargo-deny for CVEs, license policy, dependency bans
• Unwrap ratchet -- counts .unwrap() / .expect( in production code; can only go down (.unwrap-budget)
• Coverage -- cargo-llvm-cov with diff-coverage gate (70% on PR-touched lines)
• Mutation testing -- cargo-mutants against critical crates (library crates at package scope, binary crates with test_workspace = true)
• Property testing -- proptest on wire-protocol types; catches edge cases unit tests miss
• Miri -- UB detection on pure-Rust unsafe hotspots (e.g., dora-core::metadata)
• SemVer check -- cargo-semver-checks against the last git tag
• Breaking-change gate -- every surface dora 1.x freezes, diffed against the last release tag: the C header, the cxx bridge, the dataflow YAML schema, the postcard wire format, the dora command, the Python floor, and the Rust API of the covered crates. Runs on every PR; the surface half needs no build (make qa-breaking)
• Adversarial LLM review -- scripts/qa/adversarial.sh runs a different model on your diff to catch single-model blind spots (local today; CI pending API secret)

Reference docs:

• Contributor QA Cheat Sheet -- contributor-oriented setup, day-to-day commands, and PR validation checklist
• QA Runbook -- day-to-day command reference, failure modes, and fixes
• Agentic QA Strategy -- full three-tier design and rationale
• POC Report -- case studies, metrics, lessons learned, recommendations for the wider ecosystem

We welcome contributors of all experience levels. See the contributing guide to get started.

For non-trivial work, discuss the approach in a GitHub issue, discussion, or Discord thread before implementing it. Before opening or updating a PR, run the QA level appropriate for the change and include the validation you ran in the PR description. The Contributor QA Cheat Sheet is the fastest day-to-day reference; the stricter per-change policy lives in docs/agentic-qa-policy.md.

• Discord
• GitHub Discussions

This repository is built with agentic engineering. AI agents collaborate on day-to-day work -- code generation, reviews, refactoring, testing, drafting PR comments, triaging nightly regressions -- while maintainers set direction, review judgments, and authorize what ships. The two roles compound: AI agents move fast on mechanical work; humans catch the things that matter.

Apache-2.0. See NOTICE.md for details.