postgres-ai/database-lab-engine

_{Deutsch | Português (BR) | Русский | Español | Українська}

_{^*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren't possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.}

• Build dev/QA/staging environments using full-scale, production-like databases.
• Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
• Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
• Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.

For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.

Try it yourself right now:

• Visit Postgres.ai Console, set up your first organization, and provision a DBLab Standard Edition (DBLab SE) to any cloud or on-premises environment.
  • Pricing (starting at $62/month)
  • Documentation: How to install DBLab SE
• Demo: https://demo.dblab.dev (use the token demo-token to access)
• Looking for a free version? Install the DBLab Community Edition by following this tutorial.

Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.

Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.

Read more:

• How it works
• Database Migration Testing
• SQL Optimization with Joe Bot
• Questions and answers

• DBLab tutorial for any PostgreSQL database
• DBLab tutorial for Amazon RDS
• How to install DBLab SE using Postgres.ai Console
• How to install DBLab SE using AWS Marketplace

• GitLab: How GitLab iterates on SQL performance optimization workflow to reduce downtime risks

• Speed & scale
  • Blazing-fast cloning of PostgreSQL databases – clone in seconds, irrespective of database size
  • Theoretical max of snapshots/clones: 2⁶⁴ (ZFS, default)
  • Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2¹²⁸ bytes (ZFS, default)
• Support & technologies
  • Supported PostgreSQL versions: 10–18
  • Thin cloning (CoW) technologies: ZFS and LVM
  • UI for manual tasks and API & CLI for automation
  • Packaged in Docker containers for all components
• PostgreSQL containers
  • Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
  • Customization capabilities for containers (docs)
  • Docker container and PostgreSQL configuration parameters in the DBLab config
• Source database requirements
  • Location flexibility: self-managed PostgreSQL, AWS RDS, GCP Cloud SQL, Azure, etc.—no source adjustments needed.
  • No ZFS or Docker requirements for source databases
• Data provisioning & retrieval
  • Simple-mode setup: paste source URL + password, get a probed configuration in one click (UI Configuration → Simple tab)
  • Expert-mode form for both logical (dump/restore) and physical (WAL-G, pgBackRest) retrieval
  • Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
  • Partial data retrieval in logical mode (specific databases/tables)
  • Database renaming during snapshot creation (databaseRename option)
  • Continuous update in physical mode
  • Periodic full refresh in logical mode without downtime
• Recovery & management
  • Fast Point in Time Recovery (PITR) for physical mode
  • Auto-deletion of unused clones
  • Time-based auto-deletion of unused branches and snapshots (retention config; safe-only, never force-deletes dependents)
  • Snapshot retention policies in DBLab configuration
• Clones, branches & snapshots
  • "Deletion protection" for clones, branches, and snapshots (blocks manual and auto-deletion, plus count-based retention for snapshots)
  • Persistent clones withstand DBLab restarts
  • "Reset" command for data version switching
  • "Upgrade" command to move a single clone to a newer PostgreSQL major version in place
  • Resource quotas: CPU, RAM
• Monitoring & security
  • /healthz API endpoint (no auth), extended /status endpoint (API docs)
  • Prometheus metrics endpoint (/metrics) for monitoring
  • Netdata module for insights

A single clone can be moved to a newer PostgreSQL major version without touching the rest of the instance, which makes it practical to test an upgrade against production-like data:

dblab clone upgrade my-clone

The target version is not a parameter: it follows from the upgrade image the instance is configured with, which is built on that major's binaries. The engine reports it in GET /status as cloneUpgrade.targetVersion, the CLI prints it, and the UI shows it on the confirmation dialog.

The same action is available in the UI on the clone page and over the API as POST /clone/{id}/upgrade.

The clone's Postgres is shut down cleanly, pg_upgrade --link converts its data directory in place, and the clone restarts on an image of the target major. Because --link hard-links the data files instead of copying them, the upgrade is fast and consumes almost no extra space. The request returns as soon as the clone enters the UPGRADING state; watch the clone status for the result.

Enabling it. Set provision.pgUpgradeImage to an upgrade image of the major you want clones upgraded to. That image is the single source of the target: the engine reads the major out of a release tag (:17, :17-0.8.0, :17-0.8.0-glibc236), and re-reads it from the image's own PG_MAJOR before every upgrade, so a tag repointed at another build is caught before any data moves. A tag outside that grammar — a digest pin, latest, a CI build — states no major, so the engine fetches the image in the background at startup and reads it from there; until that lands the upgrade reports itself unavailable. When pgUpgradeImage is unset the feature is simply unavailable and the endpoint says so; nothing else about the instance changes. The UI keeps the "Upgrade clone" button visible but disabled in that state, and its tooltip carries the reason reported on /status, so an unset image is one hover away from being diagnosed.

provision:
  pgUpgradeImage: "postgresai/pg-upgrade:17"
  pgUpgradeTimeout: 3h
  pgUpgradePullTimeout: 1h

pgUpgradeTimeout bounds a single upgrade run; it defaults to three hours. When the budget runs out, the upgrade container is removed and the clone is put back on the version its data directory still holds, exactly as it would be after any other failure.

pgUpgradePullTimeout bounds each of the two image pulls that precede it, and defaults to one hour. The upgrade image carries the target major plus the server packages of the four preceding ones, so the first upgrade on an instance downloads several gigabytes; pre-pulling both images makes the step a no-op. Both pulls happen while the clone is still serving traffic, so exceeding the budget leaves the clone untouched and running. The two budgets are separate on purpose — a slow pull must not eat into the time pg_upgrade gets once the clone is already stopped.

An upgrade request may name an image explicitly instead of letting the engine substitute the major in the current tag. provision.upgradeImageAllowList says which repositories such a request may name. When it is empty, as it is by default, only the repository of the clone image is accepted: the one databaseContainer.dockerImage points at, plus the repository of the clone's own image when the clone has already been upgraded to another one. That default accepts exactly the image the engine would derive itself. List repositories to allow other ones, or "*" to allow any repository the instance can reach.

provision:
  upgradeImageAllowList:
    - "postgresai/extended-postgres"

Releases up to v4.2.0 accepted any repository when the list was empty. An instance that relied on that must now set the list to ["*"].

How it can end. Almost every outcome leaves the clone running:

Rolling back. Resetting the clone returns it to the version the instance is configured with, because a reset re-provisions from the origin snapshot using the engine-wide image.

Things to know.

• The target must be newer than the clone's current major and at most four majors ahead — the upgrade image carries binaries for the four preceding versions. A clone outside that range is refused; upgrading it means pointing pgUpgradeImage at a different major.
• Only majors DBLab ships a default configuration for can be targeted (10–18 today).
• The image tag keeps its extension bundle and glibc suffix; only the major changes (…:16-0.8.0-glibc236 → …:17-0.8.0-glibc236). Changing the glibc build across an upgrade would change collation behaviour, and pg_upgrade does not reindex. Pass --docker-image to override the choice; when its tag names a major, that major has to be the one being upgraded to.
• pg_upgrade verifies that every extension in the source database has a matching library for the new major. An extension outside the image's set fails this check, and the clone rolls back untouched.
• Clones using non-default tablespaces are refused: those live outside the clone dataset.
• In physical mode the sync instance and the pool are untouched — the upgrade only ever acts on a clone's own dataset. The instance-wide dockerImage must still match the source major.
• Iteration 1 limitation: a snapshot created from an upgraded clone does not record its major version, so clones created from that snapshot start on the instance-wide image and will fail to start. Avoid snapshotting upgraded clones until snapshot-level version resolution lands.

The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐

• Tweet about DBLab and mention @Database_Lab.
• Share a link to this repository on your favorite social media platform.

If DBLab has been a vital tool for you, tell the world about your journey. Use the logo from the ./assets folder for a visual touch. Whether it's in documents, presentations, applications, or on your website, let everyone know you trust and use DBLab.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

For darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

For proposals, bug reports, and participation in development, see our Contributing Guide.

• DBLab components
• Client CLI reference
• DBLab API reference
• DBLab configuration reference

• How to install and initialize Database Lab CLI
• How to manage DBLab
• How to work with clones
• How to work with branches
• How to integrate DBLab with GitHub Actions – TBD
• How to integrate DBLab with GitLab CI/CD – TBD

You can find more in the "How-to guides" section of the documentation.

• DBLab Docker images
• Extended Docker images for PostgreSQL (with plenty of extensions)
• SQL Optimization chatbot (Joe Bot)
• DB Migration Checker

The DBLab source code is licensed under the OSI-approved open source license Apache 2.0.

Reach out to the PostgresAI team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

• Database Lab Engine Community Covenant Code of Conduct
• Where to get help: Contact page.
• Community Slack
• If you need to report a security issue, follow the instructions in Database Lab Engine Security Guidelines.

Many thanks to our amazing contributors!

Making DBLab more accessible to engineers around the globe is a great help for the project. Check details in the translation section of contributing guide.

This README is available in the following translations:

• German / Deutsch (by @ane4ka)
• Brazilian Portuguese / Português (BR) (by @Alexand)
• Russian / Русский (by @Tanya301)
• Spanish / Español (by @asotolongo)
• Ukrainian / Українська (by @denis-boost)

👉 How to make a translation contribution