siglake/siglake

Open-source log and trace analytics on your object storage.

Siglake collects telemetry through OpenTelemetry, stores it as Apache Parquet in Apache Iceberg tables, and lets you explore it with SQL. Built in Rust on DataFusion, it separates ingestion, compaction, and querying so each can scale independently.

Designed for agents. Siglake gives agents direct access to telemetry through SQL, a command-line client, and a read-only MCP server. Tools such as Claude and Codex can discover schemas, estimate query costs, and work with structured results to investigate incidents and answer operational questions.

An open stream for your detection pipelines. Siglake's write-ahead log (WAL) is also a supported integration point. External services can consume sealed Arrow batches through the Rust consumer interface, with durable cursors and at-least-once delivery. Build your own detectors, alerting pipelines, or routers alongside Siglake's storage pipeline.

Documentation · Benchmarks · Releases · Changelog

• Use your existing collectors. Ingest logs and traces over OTLP/HTTP or OTLP/gRPC, or send logs through the Elasticsearch-compatible bulk API.
• Keep your data open. Read committed Iceberg snapshots directly with DuckDB, Spark, Trino, and other compatible engines.
• Run it on your infrastructure. Start locally with Docker Compose, or deploy on Kubernetes with Helm, the operator, and AWS deployment tooling.

You'll need Git, Bash, curl, and Docker with the Compose plugin. Clone the repository and start the local stack:

git clone https://github.com/siglake/siglake.git
cd siglake
scripts/up.sh

The first run builds the images from source, then starts the ingester, compactor, query server, PostgreSQL catalog, and MinIO object store.

Send a log record over OTLP/HTTP:

curl -fsS http://localhost:8088/v1/logs \
  -H 'Content-Type: application/json' \
  --data-binary @- <<EOF
{
  "resourceLogs": [{
    "scopeLogs": [{
      "logRecords": [{
        "timeUnixNano": "$(date +%s)000000000",
        "severityText": "INFO",
        "body": {"stringValue": "hello siglake"}
      }]
    }]
  }]
}
EOF

Query it with SQL:

curl -fsS http://localhost:8089/api/v1/sql \
  -H 'Content-Type: application/json' \
  -d '{"query":"SELECT timestamp, raw FROM events ORDER BY timestamp DESC LIMIT 10"}'

Events become queryable asynchronously; if the first query returns no rows, try it again shortly. The quickstart guide covers sending more data and exploring it.

Stop the local stack while keeping its data:

scripts/down.sh --keep-volumes

• Helm and the Kubernetes operator
• AWS deployment
• HTTP API
• Claude and Codex via MCP
• Grafana and Prometheus monitoring

See benchmarks.siglake.dev for query results, engine comparisons, and resource measurements. The methodology explains datasets, hardware, cache settings, and comparison limits so you can evaluate the results in context.

The architecture guide explains how ingestion, the write-ahead log, Iceberg storage, and distributed queries fit together. Design notes cover individual subsystems, and scope and limitations describe supported behavior and tradeoffs.

See CONTRIBUTING.md to build from source or submit a change, SECURITY.md to report a vulnerability, and the code of conduct for community guidelines.

Apache License 2.0. Vendored dependencies retain their upstream licenses and notices; see third_party/README.md and NOTICE.