CarterT27/scion

Source control for always-online clients, coding agents, and CI.

Scion is an independent, clean-room Rust implementation inspired by Anthropic's Redesigning source control. It explores the architecture described in that post: keep history and expensive operations on the server, fetch immutable objects as needed, and check out only the paths a job needs. This repository is not Anthropic's implementation and is not affiliated with Anthropic.

The benchmark runner compares Scion with Git on the four synthetic workloads used by OpenScion. These are local measurements of this implementation, using its own deterministic fixture.

Median wall-clock times, measured 2026-10-14 on an Apple M4 Pro with 24 GiB RAM, macOS 26.6.2, Rust 1.98.0, and Git 2.56.0. Scion was built in release mode from 9d27736 plus the benchmark runner. Each file contains 4 KiB of seeded pseudorandom bytes, spread across 10 directories; the 1,000-file case uses seven runs and the 10,000-file case uses five. All times are milliseconds; lower is better.

Scion imports and materializes the full tree faster in this setup. Git commits faster at both sizes and is slightly faster for the 10k-file sparse case. Sparse checkout materializes 100 or 1,000 files respectively and fetches exactly that many blobs plus two trees.

Reproduce the measurements (requires Git and the build prerequisites below):

cargo run --release -p scion-conformance --example benchmark -- 100 7
cargo run --release -p scion-conformance --example benchmark -- 1000 5
# Quick correctness check: 10 directories × 2 files, one run
cargo run --release -p scion-conformance --example benchmark -- 2 1

The positional arguments are files per directory and run count (defaults: 100 5). The runner prints each sample to stderr and a median table to stdout, uses a fresh temporary repository and server per run, and removes its scratch files afterward. Setup, fixture generation, authentication, and byte-for-byte checkout verification are outside the timers.

• Import: Git runs init, add -A, and commit. Scion reads the fixture files, hashes and uploads blobs in batches of 256, creates a parentless changeset, and creates a ref through the write APIs. This measures directory ingestion, not scion-import's Git-history migration; fixture path enumeration is prepared beforehand.
• Commit: Change one byte in one file of the full working copy, then time Git commit -a or Scion's working-copy commit, including status scanning and blob upload. Neither side includes a subsequent push or land.
• Full checkout: Git uses clone --no-local to build and transfer a pack; Scion uses WorkingCopy::checkout with its default 256-object fetch batches.
• Sparse checkout: Git uses clone --no-local --filter=blob:none --sparse followed by sparse-checkout set dir0000, with filtering enabled on the source. Scion checks out only dir0000. Both outputs are checked against the fixture, including absent paths outside the selected directory.

These measurements use an in-process Scion server and client over real loopback TLS/gRPC, with in-memory metadata and object storage, elevated test rate limits, and no daemon. Scion still performs authorization, BLAKE3 verification, and production zstd compression, but does not pay CockroachDB/S3 latency or durable server writes. Git uses disk-backed storage, compression, packing, and subprocesses; Scion's CLI process startup is excluded. OS caches are not flushed, and recently written working copies can trigger timestamp-related rehashing during commit. At 10k files, Git's commit samples ranged from 67.5 to 253.8 ms. These are small, single-client, single-commit workloads, not production throughput results or a direct comparison with OpenScion's published timings.

scion CLI ── Unix socket ──▶ scion-daemon ── TLS/gRPC ──▶ scion-server
    └─────────────────────── TLS/gRPC ──────────────────▶     │
                                                            ├── CockroachDB
                                                            └── S3-compatible storage

• Server-side writes. Create commits, rebase, merge, and land through APIs. Commits are immediately stored on the server; there is no separate push step.
• Incremental history. Changesets record file deltas against their first parent, alongside a root-tree pointer. Objects use BLAKE3 identifiers and zstd compression.
• Sparse checkouts. Fetch missing trees and blobs by identifier, pruning directories outside the requested scope. A bounded per-node daemon cache shares objects across jobs.
• History without a clone. Query diff, log, blame, ancestry, and merge bases over gRPC.
• Landing without a global queue. Rebase a linear stack onto a ref, then update it with a versioned compare-and-swap. Concurrent attempts retry with backoff.
• Derived indexes. Eager and lazy derived data support path history, blame, commit-graph queries, and Git mappings.
• Access control and redaction. GitHub device-flow login, service principals, path/ref policies, and quarantine/purge of blob or metadata contents without rewriting identifiers.
• Git migration. A one-way importer preserves history and records Git-to-Scion mappings.

The current v1 targets one monorepo and Unix clients (Linux and macOS). It implements the core source-control service, CLI, daemon, importer, and test infrastructure. Production throughput and resilience at Anthropic's scale have not been established.

Several features from the blog are deferred: regional cache servers, daemon snapshot bootstrap, virtualized checkouts, filesystem-watcher status, and copy detection. scion-cache is a library used by the daemon and client, not a regional cache server. Working copies materialize the files in their sparse scope, and status uses a stat-cache-based filesystem walk.

The storage stack uses CockroachDB and S3-compatible storage rather than Spanner. Code review, code search, and CI orchestration are separate systems outside this repository. Offline operation, Git protocol compatibility, and two-way Git sync are out of scope.

See the specification for contracts and deferred work and known edge cases for current limits.

Install Rust through rustup and ensure the Protocol Buffers compiler (protoc) is on your PATH. The repository pins Rust 1.98.0 in rust-toolchain.toml.

cargo build --workspace --all-targets
./target/debug/scion --help

The workspace builds four executables: scion, scion-server, scion-daemon, and scion-import.

Running the service requires CockroachDB, an S3-compatible bucket, TLS certificates, an Ed25519 signing key, and an access policy. Configure the server using SPEC section 6, then start it:

./target/debug/scion-server --config /path/to/scion-server.toml

With a configured server and an authorized GitHub account, a typical workflow is:

export SCION_ENDPOINT=https://scion.example.internal:7443
# For a private CA, also set SCION_CA_FILE to its PEM file.

./target/debug/scion login
./target/debug/scion checkout main /tmp/scion-work --sparse crates/scion-core
./target/debug/scion -C /tmp/scion-work status

# Edit files in the working copy, then:
./target/debug/scion -C /tmp/scion-work diff
./target/debug/scion -C /tmp/scion-work commit -m "Update core"
./target/debug/scion -C /tmp/scion-work land --ref main

Use login --service NAME with SCION_SERVICE_SECRET for a configured service principal. To use a node daemon, start scion-daemon --config /path/to/scion-daemon.toml and point SCION_ENDPOINT at its unix:///path/to/daemon.sock endpoint.

Commands support --json with versioned output schemas in schemas/v1. Run each command with --help for options.

The default suite uses in-memory storage, temporary working copies, and loopback services. It requires no running database or Docker. Tests include an independent reference model, Git differentials, fault injection, concurrency checks, and storage-access complexity gates.

cargo test --workspace
scripts/check.sh                 # Format, build, test, Clippy, and dependency-layer checks
scripts/check.sh --full          # Also run the larger, ignored test cases

To exercise CockroachDB and MinIO, install Docker with Compose and run:

scripts/test-external.sh --up --all
docker compose -f docker/compose.test.yml down -v

The final command removes the disposable test containers and their volumes. External-backend tests skip when their environment variables are unset; some Git differential tests skip when the Git CLI is unavailable. See docs/testing.md for details.

For design details, start with SPEC.md, implementation-defined behavior, and storage semantics.

Licensed under the Apache License, Version 2.0. Third-party Git fixtures retain their own license notices in corpus/git.