Geramy/LSE

LLM inference engine for AMD GPUs on Linux, macOS and iPadOS.

LSE records a model's forward pass as a graph, fuses operations, and generates kernels for the GPU it runs on. Kernels are emitted in the Loom dialect, compiled by loomc for the device's gfx target, launched through HRX, and kept in an on-disk cache. Kernel shapes come from device facts (compute units, LDS, wave size, matrix units) and per-architecture tables for gfx1201 and gfx1151. Where a kernel has several layouts, LSE runs each on the GPU, drops any whose output does not match, keeps the fastest, and saves the choice. MTP and DFlash2 speculative decoding, with draft trees, pick how many draft tokens to verify at each step from measured acceptance and the verify cost measured on that GPU. FlashPrefill V2 makes long prompt prefill sparse. LSE runs Qwen3.5-family models from MLX 4-bit and 8-bit checkpoints, with FP16, BF16 or FP8 KV caches, on the amdgpu KFD driver on Linux and the mac_linuxgpu driver on macOS and iPadOS. It ships an OpenAI-compatible HTTP server, a CLI and a C library (libLSE).

Multi-GPU and multi-node support is in progress. --pool can already name several GPUs: the loader splits each layer across them, or gives each a run of layers, and the scheduler copies between them peer to peer. This path has no published results yet. TCP and Unix-socket transports exist and are tested, but do not carry inference traffic yet. The next step extends kernel generation and placement across several GPUs and machines, choosing how to split work from measured link latency and bandwidth so that adding devices never slows a single-device run.

Prefill, tok/s (prompt tokens / time to first token, measured by the client on a new prompt length):

Decode, code prompt, tok/s:

Model: Qwen3.8-27B, 4-bit (MLX); drafts: dMTP (dynamic MTP) 8-bit, DFlash2 8-bit. Temperature 0.6, median of 3, after a warm-up that completes kernel variant selection.

Download from Releases, check, extract and list devices:

asset=lse-v0.6.2-linux-x86_64        # or lse-v0.6.2-macos-arm64
url=https://github.com/Geramy/LSE/releases/download/v0.6.2
curl -fL -O "$url/$asset.tar.gz" -O "$url/$asset.tar.gz.sha256"
shasum -a 256 -c "$asset.tar.gz.sha256"
tar -xzf "$asset.tar.gz" && cd "$asset"
./lse --devices                       # macOS: programs are in bin/

The archives bundle HRX, Loom and the HSA runtime. No ROCm install is needed. On Linux, add your user to the render and video groups if --devices shows no HRX device.

./lse-server pull mlx-community/Qwen3.5-0.8B-4bit
./lse-server --model mlx-community/Qwen3.5-0.8B-4bit --pool hrx:0 --kv-len 32768

The server listens on http://127.0.0.1:8080/v1 (Chat Completions and Completions). See the API reference.

lse-server flags. lse takes the same model, device, KV and decoding flags; run lse --help for its sampling flags.

Model commands: lse-server models lists cached MLX models, lse-server pull ORG/NAME[@REV] downloads one, lse-server models rm ORG/NAME --yes removes one. Add --help to each for options.

• Build from source; CMake options and the iOS XCFramework: build instructions
• Docs: API, client compatibility, DFlash2, KV cache, perplexity, benchmarks, release history, C API include/lse/lse.h
• Related: mac_linuxgpu, amdgpu_mtopg, optional RDNA4 KFD patch for Linux
• FlashPrefill V2: research by shcherbakov22, paper
• License: MIT