T-Crypt/berth

Quick start  ·  How it works  ·  Install  ·  Gateway  ·  Harbor  ·  Roadmap

Berth makes the programs on your PC take turns with the GPU's memory, so they stop crashing each other, and tells you who is holding it.

Berth decides who gets your GPU. Chat models, image pipelines, voice engines, games and GPU apps lease the card instead of grabbing it, so two of them never load at once and crash each other with an out-of-memory error. One Go binary, no cloud account, nothing else to install.

berth daemon &                                        # one broker per card
berth run --as llama -- llama-server -m model.gguf    # this engine now takes turns
berth status                                          # who holds the card, who waits, and why

Every engine asks berthd for the card: through berth run, the gateway, ComfyUI's API or a game claim. When another engine holds it, Berth:

1. queues the request by priority: hold, foreground, interactive, batch;
2. lets the holder finish its reply or image job, then unloads it;
3. waits until the driver shows the memory free;
4. grants the lease and starts the next engine, and the request that waited streams.

The numbered markers on the timeline are those four steps. A process that holds VRAM without a lease shows up as a stowaway: Berth reports its PID and memory and leaves it running.

Status: alpha (v0.2). Runs daily on an RTX 4090 workstation beside llama-swap, NInfer, Strata, ComfyUI and games. Capabilities lists what works and has tests; the Roadmap lists the phases and what is still missing.

Use both. llama-swap swaps the models in its own config, and does it well. Berth governs the whole card: llama-swap's models, ComfyUI, Ollama, vLLM, games, and any process that holds VRAM, including the ones neither tool started. With Berth beside it, llama-swap's next model waits for ComfyUI or the engine before it to give the memory back, and when a crashed process still holds part of the card, Berth names its PID and how much it holds, so you aren't hunting through nvidia-smi while a load fails. berth wire llama-swap adds the lease to every model in your config in one pass. An upstream integration with llama-swap is planned, so the wrapper line goes away.

Shares one card between everything

• Engines take turns. You start each one through berth run. When another needs the card, Berth waits for the holder's current reply to end, unloads it (its own unload, then SIGTERM, then SIGKILL) and lets the next one in only after the driver shows the memory free.
• Priorities. foreground, interactive, batch, and hold above them all. A lower request waits and gets a reason naming the holder, never an OOM crash.
• No ping-pong. When two clients ask for different engines at once, a ranking picks one and the other waits, so the card stays put.
• ComfyUI at a job boundary. Berth waits for the running image job to finish before the handoff.
• Survives restarts. Upgrade or restart the daemon and your running engines stay up and re-attach. The gateway's engines survive it too: it starts each one as its own berth run process (in its own systemd scope on Linux) and adopts them again on its next start.

Games and other apps

• Games take the card while you play. Launch a Steam, Lutris or gamescope game on Linux, or a Steam, Epic, GOG, EA, Ubisoft, Xbox or Battle.net game on Windows, and Berth drains your AI engines until you quit.
• Creative apps keep their memory. Berth names what Blender, DaVinci Resolve, OBS, Kdenlive, Premiere and After Effects hold and counts it against the card, without a stowaway report.
• Your own rules. [[claim]] in berth.toml covers anything else: a training run, a render farm job, a cracking session, by name, command line or launcher.
• berth hold frees the card for a call or a game you start yourself. Ctrl-C gives it back.
• berth run --pinned for an app that must never be stopped mid-work.

One API for your engines (preview)

• berth gateway: one OpenAI-compatible port. Name a model and Berth starts its engine, unloads whatever holds the card, waits for the load and streams the answer.
• It proxies ComfyUI's workflow API too, so an image request queues behind the same lease.
• Wake pages (--gui-listen): a bookmark to a stopped engine's web UI shows who holds the card and a Start button, says what the start waits for, and opens the UI once it answers.
• Bearer key on every route, loopback only, browser origins refused.

Models

• berth pull hf:org/repo:Q4_K_M downloads from Hugging Face, resumes where it stopped, and checks every file against the size and sha256 the Hub publishes before using it.
• berth pull civitai:VERSION_ID does the same from CivitAI, with the hash the site publishes, and berth models shelves what it finds as chat, image or audio so a machine reading a shared root knows what a folder is.
• berth storage shows each model root's filesystem, free space against its floor, and the other operating systems it is shared with.
• berth verify re-checks pulled models offline and names any file that changed.
• berth models lists every model on the machine (GGUF, safetensors, the Hugging Face cache, LM Studio and Ollama's stores) with quant, size, parameters, context and when each was last used, so you can see what earns its disk space.
• berth models export ollama turns a GGUF you already have into an Ollama model.

Keeps itself current

• berth update asks the release page whether a newer berth exists and, when one does, downloads this machine's binary and the release's SHA256SUMS, checks the hash, and replaces the running binary in place after you confirm. It then names what to restart; it never restarts anything itself.
• Update check. The harbor asks GitHub at most once every 24 h, one GET to the API's releases/latest, cached in the state directory, and shows "Update available" in its header when the answer is a newer release. [update] check = false in berth.toml turns the check off, and with it off Berth never contacts GitHub on its own; typing berth update is the operator asking, and it still does.

Sees everything on the card

• Finds your engines. Berth reads llama.cpp, llama-swap, Ollama, vLLM, SGLang, ComfyUI, Kokoro, LM Studio, KoboldCpp, TabbyAPI, LocalAI and more on their default ports, plus any OpenAI-compatible or custom server you name.
• Stowaways. Berth reports any process holding VRAM without a lease, with its PID, name and memory, and leaves it running.
• Built-in telemetry. Per-card memory, load, temperature, power and clocks; tokens per second, queue, latency percentiles, KV cache use, a token ledger and a per-model leaderboard; 30 days of history on disk.
• The harbor web UI (berth harbor): a departures board of leases, a collisions-avoided counter, charts and the event log. Read-only, loopback by default.
• Alerts to the desktop or a webhook, /metrics for Prometheus, and berth export for CSV or JSON.

Built for agents and scripts

• berth skill install teaches Claude Code, Codex, OpenCode and other agents to ask Berth for the card instead of killing GPU processes.
• Every command takes --format json|yaml, keeps results on stdout and diagnostics on stderr, and exits 0, 1 or 2 by one rule.

Runs on Linux with an NVIDIA GPU (supported) and Windows 10/11 with an NVIDIA GPU (experimental). Apache-2.0.

go install github.com/T-Crypt/berth/cmd/berth@latest      # or a release binary, see Install
berth service install --harbor                            # berthd and the harbor, as user services
berth status                                              # the card, leases, the engines and their estimated VRAM
berth run --as llama --priority foreground -- llama-server -m model.gguf --port 8080

On Linux the services are systemd user units; on Windows they are Task Scheduler logon tasks that need no admin and run in your session, where your GPU apps are. The harbor answers on http://127.0.0.1:9798/. berth service status shows them and berth service uninstall removes them; the manual steps in Install do the same by hand.

The harbor UI (basic mode, served from the built-in demo fixture - every number on the screen comes from demo.json). The tide shows the card's memory by holder over six hours; the hatched stretch is a stowaway the fixture replays: a model that ran without a lease after a daemon restart, reported and left alone.

The Engines tab (advanced mode): the engine cards with their learned footprints as EST. VRAM, the library (shelf, quant, size, the cached tier, the serving engine), decode throughput, the token ledger, the per-model leaderboard, storage and residency.

Two ways: hand it to your coding agent, or do it yourself. Both end in the same place: the daemon running as a user service, your engines wrapped in berth run, and the harbor UI on http://127.0.0.1:9798/.

Paste this into Claude Code, Codex, Cursor, OpenCode or any coding agent that can run shell commands on the machine with the GPU:

Install Berth (https://github.com/T-Crypt/berth), a VRAM lease broker for NVIDIA GPUs, on this
Linux machine, following its README and contrib/INTEGRATION.md. Steps:

1. Check prerequisites: Linux, an NVIDIA driver with `nvidia-smi` working, systemd user
   services. Stop and tell me if any is missing.
2. Download the latest release binary for this CPU (berth-linux-amd64 or berth-linux-arm64)
   and its SHA256SUMS from https://github.com/T-Crypt/berth/releases/latest, verify the
   checksum, and install it as ~/.local/bin/berth. (If Go 1.27+ is installed you may instead
   run: go install github.com/T-Crypt/berth/cmd/berth@latest.) Run `berth version`.
3. Run `berth service install --harbor`. It writes the systemd user units
   (the files in contrib/) with your binary's path in place of the one they
   name, and starts them; a unit file that already differs is kept as .bak.
   Confirm with `berth status`.
4. Find the inference engines on this machine (llama.cpp, llama-swap, Ollama, vLLM, SGLang,
   ComfyUI, LM Studio, KoboldCpp, TabbyAPI, LocalAI and so on): what runs them (systemd unit,
   llama-swap config, script) and on which port. Show me the list before changing anything.
5. For each engine I approve, wrap its start command in `berth run` as contrib/INTEGRATION.md
   shows (ComfyUI: contrib/comfyui.service with --drain-comfyui; Ollama: a systemd override of
   `ollama serve`). For llama-swap, wire the whole roster in one pass instead of one model at a
   time: `berth wire llama-swap --config <llama-swap config> --write` keeps a backup beside the
   file, keeps each model's original command, and `--check` exits 1 when a model is unwired.
   Ask me which engine wins the card (--priority foreground) and which waits (interactive or
   batch). Back up every file before you edit it and tell me how to undo each change.
6. Engines on non-default ports or other machines: add [[engine]] entries or
   [engines] discover_ports / discover_hosts to ~/.config/berth/berth.toml, then run
   `berth config check` and `berth engine check`.
7. Restart what you changed, then show me `berth status` with each engine leased and read,
   and the harbor at http://127.0.0.1:9798/. Do not expose the harbor beyond loopback unless
   I ask; it has no authentication.

The agent asks before it wraps anything, and every change it makes has an undo.

1. The binary. Download berth-linux-amd64 (or -arm64) and SHA256SUMS from the latest release:

cd /tmp
curl -fLO https://github.com/T-Crypt/berth/releases/latest/download/berth-linux-amd64
curl -fLO https://github.com/T-Crypt/berth/releases/latest/download/SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS
install -Dm755 berth-linux-amd64 ~/.local/bin/berth
berth version

Or build it: go install github.com/T-Crypt/berth/cmd/berth@latest (Go 1.27+, no C toolchain).

2. The daemon and the UI (one per machine). berth service install --harbor does this on both platforms; by hand, on Linux, they are systemd user units from contrib/:

mkdir -p ~/.config/systemd/user
cp contrib/berthd.service contrib/berth-harbor.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now berthd berth-harbor
berth status            # the card, leases, stowaways, the engines and their estimated VRAM

Open http://127.0.0.1:9798/. Engines on their default ports appear on their own.

3. Wrap your engines so they lease the card instead of grabbing it. The pattern:

berth run --as <name> --priority <batch|interactive|foreground> -- <the engine's command>

Per-engine recipes (llama-swap, llama.cpp, Ollama, ComfyUI, vLLM, SGLang, custom) are in contrib/INTEGRATION.md.

4. Optional configuration in ~/.config/berth/berth.toml; validate with berth config check:

[metrics]
listen = "127.0.0.1:9797"      # Prometheus text, loopback only

[engines]
# discover = true              # default: probe the well-known engine ports every minute
# discover_ports = [9123]      # extra ports to probe
# discover_hosts = ["192.168.1.40"]

[[engine]]                     # anything discovery does not find, on any port
name = "llama-swap"
type = "llama-swap"
url  = "http://127.0.0.1:9090"

[alerts]
[alerts.queue]
warn = 4
crit = 10

[models]
# dirs = ["/mnt/d/models"]   # the roots berth models scans and pull writes into

[library]
# free_floor = 0.10          # keep this fraction of every root free; 0 turns it off

[[library.root]]             # what a root holds and shares
path   = "/mnt/d/models"
shelf  = "image"             # chat | image | audio; beats what the files say
shared = "windows"           # the other side of a shared root, shown by berth storage

Uninstall: systemctl --user disable --now berthd berth-harbor, remove the two unit files and ~/.local/bin/berth, and take berth run ... -- off any engine command you wrapped. History lives in ~/.local/state/berth/.

Every reader takes any URL, so any port and any host works. Engines that need a key name the environment variable that holds it (api_key_env); the key never goes in the file. An engine's figures are its own claims and are shown apart from what the driver measures.

berth engine install|show|update|rollback|remove manages ComfyUI as a suite: the core, ComfyUI-Manager and every git-based node pack are separate components, each pinned to a git commit with one previous commit kept as the rollback snapshot. Several installs run side by side on separate ports; each install's derived engine appears in berth engine check, berth status and the gateway's wake page, and its start line is the exact berth run shape an operator's unit already uses (the only alignment needed is --as <install-name>).

• install NAME --adopt PATH records an existing checkout in place; --path clones a fresh one (PyTorch is your step: the install is registered either way, and ComfyUI's own start error names the missing piece).
• update NAME pins the core and each pack to a newer commit; rollback NAME --component C swaps back. A dirty component, or an install with jobs running, is refused: the update happens at the job boundary.
• remove NAME drops the state entry; --delete --yes also removes the directory.

The state lives in engines.toml beside berth.toml. Berth writes each install an extra_model_paths.yaml naming the node's model roots from berth.toml, so an install's models come from the same library the rest of the node uses; the daemon rewrites the file at startup, so a change to the roots applies on the next restart.

• Explicit leases. Engines start through berth run, which waits for a berth, starts the engine in its own process group, and gives the berth back when the engine exits.

• Priorities. foreground > interactive > batch. An equal or higher request asks the holder to drain; a lower one waits and is refused with a reason naming the holder (exit 75), never an out-of-memory crash.

• Handoff in steps. A drain tries the engine's own unload first (for llama-swap, its unload URL), then SIGTERM to the process group, then SIGKILL, each after a grace period.

• Measured release. Berth grants the next engine only after the driver shows the old holder's processes gone, since an exited process can still hold memory.

• Stuck releases are caught. A released lease whose memory never frees turns aground and keeps blocking, with a reason naming the process and how much it holds.

• A way out for a stuck release. berth reclaim [lease] lists what the stuck lease recorded, PID, start time, memory and command, and anything on the card no lease recorded ("not Berth's", report only). With --yes it stops exactly the recorded ones, SIGTERM first and SIGKILL to the survivors after --grace, re-checking each PID and start time immediately before every signal.

• Process identity is PID plus start time, so a reused PID never inherits a lease.

• Whole-card or sized leases. --needs-mib N lets engines that fit share a card.

• Fail-open, never silent. If the daemon is down, berth run still starts the engine but records it as an unprotected run that status shows (--fail-closed refuses).

• Survives a daemon restart. A running engine stays up when berthd restarts or is upgraded; berth run dials the new daemon and re-attaches by PID plus start time. For 15 s after a start the daemon grants nothing new while such engines are still due back, so nothing is loaded onto a card an engine already holds.

• ComfyUI at a job boundary. --drain-comfyui waits until ComfyUI's queue is empty before stopping it; a job in progress is never cut off.

• No juggling. Two engines at the same priority drain each other: the last one asked for wins, which is what you want when you switch models. When two clients want different engines at once, that flips the card back and forth and every request fails. Berth calls it a juggle when an engine is drained within juggle_window (15 s) of getting the card while it is still busy (a request in flight, a ComfyUI job queued, a model loading). The engine ranking decides a juggle: the winner keeps or takes the card, and the loser waits for it instead of draining it back. An idle holder hands over at once, so chat, then an image, then chat again is three clean handoffs.

  [priority]
  order = ["llama-swap", "strata", "comfyui"]  # first is best; omit to rank by use
  juggle_window = "15s"                        # a negative value turns it off

  Without an order, the engine with the most hours on the card through Berth over the last 7 days ranks first (kept in ~/.local/state/berth/usage.json); a tie leaves the holder in place. berth run --group NAME files a lease under an engine family: llama-swap runs each model under its own name, so give each --group llama-swap. berth status shows the ranking and a count of juggles stopped; the harbor log shows each one with the reason.

• Claims cover work Berth does not start. berthd scans the process table every 2 s. A hold claim keeps the card while its process runs: engines are drained and wait, with a 15 s linger so a launcher restarting the game does not bounce the card back. A track claim names an app's card memory so it counts against the card without being reported as a stowaway; nothing waits on it.

• Built-in pack, on by default. Holds: Steam games (anything under Steam's reaper), Lutris and gamescope on Linux; Steam libraries (steamapps\common), Epic, GOG, EA, Ubisoft, Xbox and Battle.net games on Windows, with launchers, Wallpaper Engine and Lossless Scaling left out. Tracks: Blender, DaVinci Resolve, OBS, Kdenlive, Premiere, After Effects. berth claims shows every rule and what it matches now.

• Your own rules:

  [[claim]]
  name = "training"
  cmdline = 'train\.py'      # regex; on Windows it sees the exe's full path
  action = "hold"            # or "track"
  linger = "30s"
  
  [claims]
  builtin = false            # turn the built-in pack off

  A rule can also match exe names (globs, .exe optional), a launcher parent, and exclude with not_cmdline.

• berth hold [--for 2h] [--reason game] takes the card for you. It is a pinned lease at the top priority, so nothing drains it, and it ends on Ctrl-C, after --for, or when the process dies.

• berth run --pinned gives an app a lease that is never drained (it keeps the card until it exits).

• A hold cannot free memory held by something outside Berth: that memory is still reported, as contested.

• berth pull hf:ORG/REPO[:QUANT|FILE|GLOB][@REV] resolves the revision to one commit, picks the GGUF for the quant (every part of a split model, and the vision projector when there is one), and downloads to ~/models/<org>/<repo>/.
• berth pull civitai:VERSION_ID resolves a model version (or civitai:model:MODEL_ID for its latest, or the site's own URLs), downloads the files it lists to ~/models/civitai/<model>/, and writes the same manifest. Public models need no key; gated files read $CIVITAI_API_KEY.
• berth civitai search QUERY and berth civitai versions MODEL_ID find the ids a pull takes: models by name and asset kind, then a model's versions newest first.
• Integrity first. Each file lands as .part and is moved into place only after its size and sha256 match the source's published hash. A mismatch deletes it and fails.
• Resumes. Run the same command again after an interruption and it continues with a range request. A file already present and correct is skipped.
• The floor. Before it downloads, a pull measures the destination root and refuses when its files would leave the root under [library] free_floor (a tenth by default, 0 turns it off). A root whose space this platform cannot measure says the floor was not checked, and goes on: a guard that cannot measure does not claim to guard.
• Shelves. berth models reads each model for which shelf it is: chat, image, audio, or unshelved when the file states nothing. A root declared in [library] (say shelf = "image") declares what it holds, and that beats a file's header. The SHELF column shows it, --shelf image lists one shelf, and the totals under the listing say how much of the library each shelf holds.
• The floor's view: berth storage. Each root's filesystem (an ntfs3 root shared with Windows reads as ntfs3, on both sides of the share), the space it has left against its floor, what it can share (hard links, symlinks, reflinks), the other operating systems it is shared with, the shelves it holds, and when a berth last scanned it. A root under its floor fails the check.
• The in-root catalog. A scan leaves <root>/.berth/catalog.json: what it resolved, from which machine, and when. On a root two operating systems share, the second boot reads the first's resolutions instead of re-guessing them. The scan always recomputes; the file is a snapshot for the other side, never a cache this side trusts.
• berth verify re-hashes pulled models against the manifest the pull wrote (.berth-manifest.json: source, name, ref, every hash), offline, and exits 1 on any change.
• Gated and private models read $HF_TOKEN (Hugging Face) or $CIVITAI_API_KEY (CivitAI).
• berth models reads every GGUF header (never the weights) in the tools' usual places and any --dir or [models] dirs. Split models list as one; vision projectors and MTP or draft heads list with the model they belong to (--all shows them on their own). --usage 30d adds requests and tokens per file from the harbor's request log, matched through llama-swap's config when that is how the model is served.
• The CACHED column. How much of a model's files (its parts, and the vision projector and draft that go with it) sit in the kernel page cache, and the tier that share reads as: COOL at half or more, COLD below it. The check reads the kernel's page table (mincore), never a byte of a file, so it cannot pull a page in, and --no-cache-check skips it for a listing of a big library. The page cache is the kernel's: it keeps the pages until the machine needs the RAM, so the column says a restart is likely to read from memory, never that any RAM is held.
• berth prefetch MODEL_OR_PATH asks the kernel to read a model's files into the page cache (posix_fadvise WILLNEED). The ask is made and the command returns at once; the kernel does the reading in the background, at its own pace, and takes the pages back whenever the machine needs the RAM. This is the manual form: while an engine that is about to run is still waiting for the card, its model's files can be warmed so the start reads from memory. The harbor's LIBRARY panel carries the same ask as a copy button on every model row.
• berth models show NAME prints one model's header: architecture, context, tensor types, chat template, and what of the model sits in the page cache.
• berth models export ollama NAME [--dry-run] writes a Modelfile and runs ollama create; a split model needs llama-gguf-split --merge first.

• One port for every engine. berth gateway proxies OpenAI-style requests to the engine that serves the requested model, streaming.
• Cold starts under a lease. An engine with start and serves in berth.toml starts on the first request that names one of its models, through the berth run path: lease, drain of the holder, measured release, then the start. One start per engine, however many requests arrive while it loads.
• Last switch wins. Give gateway-started engines the priority of the engine they replace (priority = "foreground" next to a foreground llama-swap) and each side drains the other on demand. A lower priority waits behind the holder instead.
• Clear failures. Unknown model: 404. A start the broker refuses or an engine that exits before it answers: 503 with the reason, the engine's own failure line, a Retry-After and an error code a client can branch on: outranked (lost a juggle), drained_while_starting, no_berth, not_ready, start_failed, engine_oom (the engine could not get the GPU memory it asked for) or no_start. An engine that crashes on start isn't started again on every retry: for 15 s, doubling to 5 min, the gateway answers at once with the same cause, and Retry-After says when the next start is allowed. A start that answers clears it. A model listed under serves by two engines fails berth config check.
• API key. Bearer token on every route; berth key prints it (created on first use, mode 600). Loopback only; browser origins and non-loopback Host names are refused.

Two engines that each want the whole card can't run together, and most harnesses can't start an engine before they call it. berth gateway starts it for them. Give an engine a start command and the model names it serves in berth.toml:

[[engine]]
name     = "strata"
type     = "llamacpp"
url      = "http://127.0.0.1:8080"
start    = ["/path/to/strata/.venv/bin/python", "serve/server.py", "--engine", "strata",
            "--config", "strata-262k.json", "--port", "8080"]
serves   = ["router-strata", "qwen3.8-flash-next-iq2_xs"]
priority = "foreground"   # equal to the engine it replaces, so the last model you ask for wins

Run berth gateway (loopback 127.0.0.1:9799; a user unit is in contrib/berth-gateway.service) and send your requests there with the node's API key as the bearer token. The gateway creates the key on first start (~/.config/berth/api.key, mode 600) and berth key prints it, so a client that runs commands for its key never has the secret written into its config:

{ "baseUrl": "http://127.0.0.1:9799/v1", "apiKey": "!berth key" }

For each request the gateway:

1. reads the model name from the body;
2. hands the request straight to an engine that is up and lists that model (llama-swap and Ollama load their own models under their own leases);
3. otherwise starts the engine that serves it: as its own berth run process, under the same lease path (Berth drains the holder: llama-swap through its unload URL once its slots are idle, then SIGTERM, then SIGKILL), and the gateway waits until the new engine answers /v1/models;
4. proxies the request and streams tokens as the engine sends them.

Requests that arrive during a start wait on that one start. A model no engine serves gets a 404, and a start the broker refuses comes back as a 503 with the broker's reason. A start waits for the card no longer than --ready (10 minutes); if the holder has not left by then, the request fails and the start is dropped, so it cannot take the card hours later. Asking llama-swap for one of its models later drains the gateway's engine the same way, so either side can take the card back.

Measured on the reference RTX 4090 with an agent harness switching models between NInfer (through llama-swap) and Strata (262K context):

At no point did the two engines hold the card at the same time.

A gateway restart keeps its engines. Each engine runs as its own berth run process, in a systemd user scope on Linux (systemd-run --user --scope, when the user manager is reachable) so it is outside the gateway service's cgroup, else in a new session. The gateway writes each started run, with its PID, start time and lease, to a small file under the state directory (~/.local/state/berth/gateway-engines.json), and the run's own output goes to ~/.local/state/berth/engines/<name>.log (mode 600), never to a pipe the gateway holds, so a gateway that stops cannot break an engine's writes. While the gateway runs it copies the log to its own journal as well. When the gateway comes back it reads that file and adopts every engine whose process is still the same one, answering or still loading: the next request is proxied, not started again. Records for processes that left are dropped. Restarting during a 40-110 s cold start costs nothing but the restart itself.

Stopping the gateway keeps its engines too. Each engine runs outside the gateway's cgroup (above), so stopping the gateway service does not stop the engines: they stay on the card under their own leases and keep answering. berth stop ENGINE is how one is stopped on purpose: the daemon asks the engine's leases to give the card back through the normal stop ladder, as when another engine asks for it, and a lease frees only after a card read shows the engine's memory gone. berth stop ENGINE --wait follows the daemon's event log until the leases are freed (exit 0) or stuck (exit 2, pointing at berth reclaim for the stuck ones).

Long waits and idle clients. Cold starts take 41-110 s (table above), longer than many OpenAI clients wait before giving up. Streaming requests get the answer early: the gateway sends its SSE headers (Content-Type: text/event-stream, Cache-Control: no-cache) right away and writes an SSE comment every 10 s while the start runs, : berth: loading strata, 34 s, so proxies and clients keep the connection and a reader sees the wait moving. Once the engine answers, its body streams on; a start that fails sends one OpenAI-shaped error event with the same reason code the 503 path carries, then data: [DONE]. Non-streaming requests are unchanged; if a client drops a long non-streamed wait, raise its read timeout or make the request streamed. GET /v1/berth/wait?model=NAME blocks until the engine answers (a failed start is the same 503), so a harness can pre-warm before its first real request. The gateway serves /v1/chat/completions, /v1/completions, /v1/embeddings and a merged /v1/models, plus the ComfyUI surface: POST /prompt queues a workflow (starting ComfyUI under a lease when it is down, holding the request until its /queue answers), while /ws, /queue, /history/, /view and /upload/ proxy to a running ComfyUI. Every route needs the key. The gateway listens on loopback only, refuses a Host that is not a loopback name (DNS rebinding) and refuses any request that carries a browser Origin, so a web page cannot start an engine. ComfyUI already hands off at a job boundary under a lease (--drain-comfyui); a drain first unloads its models via POST /free, and --keep-comfyui-up leaves the server on its port for OpenWebUI instead of stopping it.

A bookmark to ComfyUI or Strata's chat page is a dead link while that engine is stopped. Give the engine a gui URL (where a browser reaches its web UI) next to its start command, and run the gateway with --gui-listen 127.0.0.1:9796:

[[engine]]
name  = "comfyui"
type  = "comfyui"
url   = "http://127.0.0.1:8188"
start = ["comfy", "--workspace", "/path/to/comfy", "launch", "--", "--listen", "127.0.0.1", "--port", "8188"]
gui   = "https://box.tailnet.ts.net:8188/"

Point the bookmark (or a home-page tile) at http://127.0.0.1:9796/gui/comfyui instead:

• ComfyUI answers: the browser goes straight to its UI.
• ComfyUI is stopped: the page says so and lists what holds the card (llama-swap (NInfer-27B) active), with a Start button. Opening the page takes nothing; only Start asks for the card.
• After Start: the page says what the start waits for, in the broker's terms: "Waiting for strata release: finishing its current request", "Waiting for llama-swap (NInfer-27B) release: freeing its memory", then "loading comfyui". It opens the UI once ComfyUI answers. A failed start shows its reason code and a retry button.
• /gui/ lists every engine with a wake page.

A start never runs two engines on the card at once. Berth drains the holder only at its boundary: llama-swap once its slots are idle (drain_post), ComfyUI once its queue is empty (drain_comfyui), and a server with no unload URL once its /slots show idle (drain_slots, below). A pinned holder (a game, a render, berth hold) is never stopped; the page says so and the start waits until it ends.

The wake page needs no API key: it can only start an engine you gave both start and gui. It listens on loopback, accepts loopback Host names plus the gateway's --allow-host names (for tailscale serve in front), and takes a start only as a POST from its own page, so another site cannot make a visitor's browser drain the card.

An engine with /slots but no unload URL (Strata, a bare llama-server) would otherwise get SIGTERM mid-request when another engine asks for the card. drain_slots (or berth run --drain-slots URL) names its base URL: a drain waits until no slot reports is_processing, then stops it.

[[engine]]
name        = "strata"
type        = "llamacpp"
url         = "http://127.0.0.1:8080"
drain_slots = "http://127.0.0.1:8080"

The wait is unbounded by default, so a long generation is never cut off. drain_max (or berth run --drain-max DURATION, which wins) sets a ceiling on it: while the boundary is busy the wrapper prints a line each minute naming the slots and how long they have been busy, and once the ceiling passes it runs the normal stop ladder anyway and says so. The same ceiling applies to a ComfyUI queue that will not empty.

A drain today ends with the engine's weights on disk, and the next request reloads them: tens of seconds every switch. Parking keeps the warm copy in host RAM instead:

Config, one line per engine plus the node's budget:

[[engine]]
name       = "vllm-chat"
type       = "vllm"
url        = "http://127.0.0.1:8000"
park       = "hook"
park_url   = "http://127.0.0.1:8000/sleep?level=1"   # optional: each beats the built-in
resume_url = "http://127.0.0.1:8000/wake_up"

[park]
floor_gib = 4.0        # RAM a park must leave the machine (default 4)
max_gib   = 48         # cap on all parked copies together (default: no cap)
ttl       = "30m"      # a parked copy is stopped after this long
residual_max_gib = 2   # card memory a parked engine may keep

• How it runs. A drain first waits for the engine's boundary (idle slots, or an empty ComfyUI queue), then asks berthd to admit the park: admission measures the machine's available RAM, and a park that would leave less than the floor is refused quietly, so the drain runs as today. On admission the wrapper calls the engine's own offload hook and records the park; the lease stays parked, the card's residual counts, and a waiter builds beside it. The next request sends the resume, and the engine serves again in seconds.
• Adapters. ComfyUI needs nothing: its unload keeps the models in RAM until memory pressure. vLLM needs --enable-sleep-mode and VLLM_SERVER_DEV_MODE=1 in the start command. SGLang needs --enable-memory-saver --enable-weights-cpu-backup. Any other engine can speak the park protocol (contrib/PARK-PROTOCOL.md): Berth probes GET /berth/state once and uses it when the endpoint answers.
• Eviction. A parked copy is stopped, least recently parked first, when it passes its ttl, when the machine's available RAM falls under the floor, or one step at a time under memory pressure. Stopping a parked engine is a drain, and its run is Berth's own, so the stop is graceful.
• Who sits out. llama.cpp and llama-swap have no offload call upstream yet, so they run today's drain; a park hook there waits for an upstream endpoint or a patched route.

• Stowaways: every process holding VRAM without a lease, with memory, identity and first-seen time; optional desktop notification and webhook. Report-only by design.
• Contested grants: a grant made while stowaways hold memory is flagged, not hidden.
• Event log: every request, wait, drain, release, free, stuck lease, stowaway and probe change, with gap-free sequence numbers; berth events -f follows it live.
• The collisions board: collisions avoided, VRAM handed off, drains, stuck releases caught, stowaways caught, contested grants, refusals. Each count links back to a recorded event. berthd keeps the board in ~/.local/state/berth/counters.json (--counters), so a restart or a reboot carries the counts on. The harbor stores a reading a minute and its ticket counter shows the last 5 min, 15 min, hour, 24 h or 7 days next to the all-time board (GET /v1/counters?range=5m|15m|1h|24h|7d).
• The engines' estimated VRAM: a run measures how much VRAM its leases held at their peak, and the broker keeps the last figure per engine in ~/.local/state/berth/footprint.json (--footprint), across restarts. berth status shows it as EST. VRAM beside the engine's own figures, and the JSON names it estimated_vram_bytes. It is an estimate of what a run needs, never a reservation: a run that holds far more than the learned figure (1.5x, once the engine has three runs) logs an engine.footprint event, so a grown engine is seen instead of mis-sized.
• The queue in every wait reason: every reason a waiter is sent starts with its place in the queue, "next in line" or "2nd in line (behind interactive comfyui)", so a client behind the head of the queue hears where it stands instead of silence. The waiting table of berth status shows the same place beside a start estimate, in seconds after the card frees: the measured starts in the waiter's chain, one per seat ahead plus its own. An unmeasured start says unmeasured and adds no number, and the holder's drain is never in the figure, because a request runs as long as it runs.
• Alerts: one threshold table in berth.toml ([alerts]): warn and crit per key (queue, ttft, tpot, p95, cache, reingest, vram, temp, power, kv). The node evaluates it on its 5 s sample cycle, fires a desktop notification and webhook on entering a level, and logs alert.raised / alert.cleared events (berth events -f, the harbor log); a level stays until the value stays clear for clear_polls samples (default 3). GPU probe failure and aground leases are built-in danger alerts, always on. Default posture: no [alerts] table means no KPI alerts, and the notification channels are separately opt-in (--notify, --webhook). The harbor colors the same state the node notifies on: KPI tiles and GPU cells, a strip of the currently-breaching keys, and an overall LIVE / DEGRADED / OFFLINE state beside the connection pill.

Everything the node collects shows in one of three tabs, and every panel answers a question an operator actually asks.

The CARD tab.

• The ticket counter answers "has anything collided?" with the collisions-avoided odometer, the seven collision tiles and the request KPIs, over the all-time board or a stored window.
• The tide answers "who holds the card, and when?" with card memory by holder and the lease events overlaid, over a 1H / 6H / 24H / 7D / 30D picker.
• The model timeline answers "when was each model loaded on each engine?"
• The departures board answers "what is on the card right now?" with one row per lease: the ticket number, the engine, the class, the hold (measured once docked, asked-for while boarding, the parked engine's RAM copy while WARM), the state word, and the age. A WARM flag marks a parked engine; PARKING is the transitional state while the weights move to RAM. A boarding row carries a Q flag with its queue position and the node's completion estimate when the node sends them. Clicking a row opens the lease's ticket: the tier the weights live in (HOT on the card, WARM in RAM), the queue rows, the container a run is in, and the exact terminal order for the lease beside it as a copy button (berth stop ENGINE, berth resume ENGINE, berth reclaim LEASE).
• The berths, the machine, stowaways and no ticket answer "what else is on the card and the host, and who holds memory without a ticket?"

The ENGINES tab.

• The engine cards answer "is this engine up and serving?" with the state the engine reports, its loaded models, decode and prefill rates, and its learned footprint as EST. VRAM: what its past runs measured, never "reserved".
• The library answers "is this model warm?" with every model in the configured roots, each row showing shelf, quant, size, the CACHED share with its tier (COOL or COLD), the engine serving it when one reports it resident, that engine's EST. VRAM, and copy buttons for berth prefetch NAME and berth models show NAME. The listing is read-only here; the terminal does the acting.
• Card ranking, the KV pool, throughput, the token ledger, the leaderboard, the recent requests and context residency answer the questions behind the request log, and the storage panel (advanced) says what the harbor's own store keeps.

The ABOUT tab is this page's own documentation: the modes, the shortcut keys, and what every state word and figure means.

• GPU: memory, utilisation, temperature, power against its limit, fan, SM and memory clocks, PCIe link, for every card in the machine with its real limits.
• Host: CPU, RAM, swap, load.
• Engines: every engine in the table above, found on its default port or configured on any other. Loaded models, decode and prefill tok/s, token totals, queue, draft acceptance, the context window per model (llama.cpp/NInfer /props, Strata /health, llama-swap per model proxy), and the live KV slot pool (/slots: per-slot context tokens against the window). An engine without those endpoints degrades to "no window / no slot data", never an error. An engine that was up and stops answering is shown as unresponsive (latched), distinct from one that never ran.
• Requests: a 24-hour request log from llama-swap; requests per minute, p50 and p95 latency, prefill time, time per output token, cache and re-ingest share, a token ledger (1 h, 24 h, since start), a per-model leaderboard, and the most recent requests.
• History: the node keeps the last hour at 5-second resolution in memory. The harbor stores it on disk (SQLite, no C toolchain needed): 1-minute rows for 30 days, per engine the KV pressure (peak slot fill against the window) and the resident models' windows, hourly rows for 400 days, and the request log for 30 days, de-duplicated by request id. Databases written before the KV column is added are migrated in place on the first open; nothing is ever rewritten.
• Model timeline: when each model was loaded and unloaded on each engine, kept across restarts. A gap in the readings over 3 minutes is never drawn as loaded.
• The UI labels engine-reported figures as the engine's own, apart from what the driver measures.

• CLI: one command tree with one output contract. berth status, berth events -f, berth run, berth hold, berth stop, berth claims, berth pull, berth verify, berth models, berth gateway, berth wire llama-swap, berth engine check, berth config check, berth config show, berth export, berth key, berth skill install, berth update, berth version. Berth runs fully headless.

  berth wire llama-swap --config cfg.yaml          # plan: which models lack a wrapper
  berth wire llama-swap --config cfg.yaml --write  # apply; a backup stays beside the file
  berth wire llama-swap --config cfg.yaml --check  # exit 1 when a model is unwired
  berth config check --strict          # a warning fails the check, exit 1
  berth config show --format toml      # the evaluated config, ready to save
  berth engine check -v                # what each engine reports, right now
  berth -q status; echo $?             # exit codes only, for CI
  berth status --format json | jq .leases

  Results go to stdout, diagnostics go to stderr. --format is human (the default), json or yaml, and berth config show also takes toml, the config's own file format. -v adds detail to the human view, and -q suppresses stdout and leaves the exit code as the answer. Exit codes are 0 (done), 1 (validation or logic: an invalid config, a warning under --strict, an engine that did not answer, a lease the broker refused) and 2 (berth could not do its job: unknown flag, missing file, berthd unreachable). berth run passes the engine's own status through and exits 75 when no berth was granted. Every read-only command changes nothing.

• Harbor web UI (berth harbor): a departures board of every lease (the HOT / WARM tiers included: WARM marks a parked engine, with its parked RAM and age), a boat ticket per lease, the collisions counter, berths with measured memory, stowaways, the event log, and engine metrics. The tide (card memory by holder) and the model timeline share a 1H / 6H / 24H / 7D / 30D picker; the Engines tab adds the library (every model on the machine with its shelf, quant, size, cached tier, the engine serving it and that engine's EST. VRAM, with the terminal commands one row of copy buttons away), a queue KPI (the berth's own queue; each engine's own queue stays on its card), the KV pool (the live slots of each engine against their model windows, stored ranges the per-minute peak slot fill), context residency (the last requests' token figures against the model's window, plus the live KV sessions), a throughput chart, a 24H / 7D / 30D leaderboard and, in advanced mode, a storage panel. The top bar shows the overall state (LIVE / DEGRADED / OFFLINE) and a strip of the alerts currently breaching, colored from the node's own threshold state. Basic and advanced modes. While its update check is on and a newer release exists, the header shows it: the release, a link to it, and a button that copies berth update, the command that does the replacing. Read-only, strict content security policy. It listens on loopback by default (127.0.0.1:9798); passing a non-loopback --listen address (e.g. 0.0.0.0:9798; a specific interface IP works too) is the explicit opt-in to expose it on your network: the startup logs a warning and the page shows a banner on every screen while exposed. There is no authentication in this release: expose it only on a network you trust; phones on that network get the full read-only dashboard. To reach it from elsewhere without exposing it, keep the loopback listener and put a same-machine reverse proxy in front (tailscale serve, caddy), passing the proxy's hostname with --allow-host. A systemd user unit is in contrib/berth-harbor.service.

• Prometheus /metrics (loopback only) for Grafana and friends.

• Footprint: the node uses about 0.2% of one CPU core and ~21 MB of memory on the reference box; the harbor about 0.4% and ~23 MB, writing to disk once a minute.

The scenarios in contrib/chaos and the numbers below are how Berth's failure handling gets checked outside this tree. Run them on your card and add a row with the result, or report what happened with the drain report issue form and the table credits you here.

contrib/bench/swap through the gateway on the RTX 4090 workstation (driver 615.78.08, PCIe 4.0 x16), 2026-10-20: an opening request to A, then two cycles of B and A, a fixed prompt, greedy.

Switching on every request is the ping-pong Berth's juggle guard exists to hold back: the switches to Strata waited behind the guard for llama-swap and measured 377-382 s end to end, of which the engine's own load (its log) was about 80 s. Park was off for both engines; parked resume numbers and cuda-checkpoint follow in v0.3.x. Run the bench on your card and add a row.

v0.3 goes after the cost you feel most, reloading every engine from disk on every switch, and proves Berth holds up when an engine misbehaves. It ships when the swap benchmark and the failure scenarios pass, not on the date.

In v0.3:

• Park in RAM. On a drain, an engine that can offload itself moves its weights to system memory through its own park and resume URLs, and comes back in seconds instead of reloading from disk. ComfyUI gets a built-in adapter. Berth parks only when the free RAM it measures covers the engine plus a floor, and stops parked engines first when memory runs short. Without the RAM, the drain runs as it does today.
• Measured. A swap benchmark (cold, warm and parked, per engine) with the numbers published here.
• berth reclaim. Lists the processes Berth recorded under a lease that still hold memory (a crashed run's child, a stuck release) and stops them when you confirm. Processes Berth never started stay report-only.
• Handoff fixes. A lease frees only after a card read shows its memory gone; a gateway restart keeps the engines it started; a streaming request gets keep-alive lines while its engine loads.
• A place and an estimate in every wait reason. Every reason a waiter is sent starts with its place in the queue, "next in line" or "2nd in line (behind interactive comfyui)", and the waiting table of berth status shows the place with a start estimate in seconds after the card frees. The estimate prices only what the node's book of measured switch times has measured, one start per waiter ahead plus its own; an unmeasured start says unmeasured, and the holder's drain is never in the figure, because a request runs as long as it runs.
• berth update. Asks the release page (one GET, cached 24 h) whether a newer release exists, and on your say-so downloads this machine's binary and the release's SHA256SUMS, checks the hash, and replaces the running binary in place, then names what to restart. The harbor's header shows a new release the same way; [update] check = false in berth.toml keeps Berth from ever contacting GitHub on its own.
• Proof. Tagged releases with changelogs; contrib/chaos, the failure scenarios anyone can run (an engine that ignores SIGTERM, a child that escapes, a stowaway, a daemon restart mid-lease); a public table of tested GPUs, drivers and engines.

In v0.3 if ready, else a v0.3.x release: NVIDIA's cuda-checkpoint for unmodified CUDA engines on Linux; reading the next engine's files ahead while the holder drains, and showing how much of each model the page cache still holds; berth fit and scenes, so engines that fit together never swap.

On hold until v0.3 ships: engine installers beyond the ComfyUI suite, the migration manager, and Berth swapping models itself. llama-swap already swaps models well; Berth works beside it, and a proposal to llama-swap for generic load hooks is planned, so Berth needs no wrapper line.

• Kill a process it did not start, or any process by default.
• Grant a card because a process exited, before the driver shows the memory free.
• Claim memory is reserved when it is only estimated to be free.
• Take a card from another user's machine without their consent (on one machine, only the priorities you set decide).
• Require a cloud account, telemetry upload, or any other software to work.

What is an engine? A program that runs a model on your GPU: llama.cpp, llama-swap, Ollama, vLLM, ComfyUI, a voice server. Berth doesn't run models; it decides which engine has the card.

Why not just llama-swap? Use both. llama-swap swaps the models in its own config. Berth covers the whole card, including ComfyUI, games and the stray process holding memory that neither tool started. See Berth and llama-swap.

Can a 700B model take turns on a 24 GB card? No. Berth gives each engine the whole card in turn; it doesn't make a model fit that doesn't.

How reliable is it? Alpha. It runs every day on the author's RTX 4090 workstation, and v0.3 adds contrib/chaos and a public table of tested hardware so you can check it on yours.

AMD? Windows? AMD on Linux and a full Windows node are planned for v0.4. Windows with NVIDIA works today as experimental; see the support matrix.

Can Berth handle ComfyUI on the host and vLLM in Docker? Yes. Start each one under its own lease: berth run --as comfyui -- python ... for ComfyUI, and berth run --as vllm --container vllm -- docker run --rm --name vllm --gpus all ... vllm/vllm-openai for vLLM. The container's GPU processes are parented by containerd-shim, so the process tree cannot reach them; the lease names the container instead, Berth matches the processes by container ID, and the drain runs docker stop -t 30 vllm (podman's when podman resolved it) instead of signalling the wrapper. The handoff waits for the driver to show the container's memory gone, as with a host process. A container Berth did not start is still a stowaway: reported with its container ID, report only, never signalled. Details in contrib/INTEGRATION.md.

Issues and pull requests are welcome. AGENTS.md has the one command that proves a change (build, format, vet, tests, lint, as CI runs them) and the invariants a change must keep.

Apache License 2.0. Copyright 2026 Trevin Tindall (T-Crypt), who created Berth and maintains it here. Forks are welcome: keep the NOTICE file and the copyright header in each source file, mark the files you change, and release your fork under its own name. The Berth name, the Pilot mascot and the artwork in assets/ are not covered by the license. Third-party components: THIRD_PARTY_NOTICES.