Soulmate-Halo/heterogeneous-gpu-pd-lab

中文

⬇ Download the HD original (1080p · 60MB): GPU Heterogeneous Build Explainer v24 · English — inline playback above is a 480p compressed preview (12 min 22 s, same content). The Chinese-narration original is still available: v24 中文版

Ten minutes on GPU heterogeneity: MoE mixture-of-experts basics, five build patterns, three real-hardware benchmarks (3080+Spark PD split, Qwen3.8 Flash EP, DS4.1 comparison), and the production-grade deployment takeaway.

📌 Data methodology note: All experiment results in this repository are untuned raw numbers — only the base stack's built-in optimizations are used, with no additional tuning applied. Comparing tuned figures from other open-source projects against these raw numbers is meaningless: those techniques are equally applicable to heterogeneous setups and are left off only to keep the measurements clean.

DGX Spark handles Prefill alone; Spark and the RTX 3080 cooperate on Decode to accelerate decoding. Deployment guide · Bundle

The experimenter supplied the V2 Decode results and solo controls on 2026-10-06; raw benchmark logs, concurrency and workload details were not included in this revision. These reported values give approximately 2.25x aggregate and 2.78x single-stream Decode. The three public Prefill runs use 3600 prompt tokens; the solo Prefill control uses 2047, so they do not establish a strict Prefill speedup.

flowchart LR
    A["API request"] --> P["DGX Spark / GB10<br/>Solo Prefill"]
    P --> D["Decode work distribution"]
    D --> G["RTX 3080 20GB<br/>Hot experts and main compute"]
    D --> S["DGX Spark / GB10<br/>Cold expert compute"]
    G <-->|"RDMA expert requests and results"| S
    G --> O["Output tokens<br/>Faster joint Decode"]

The diagram shows the experimental division of work. The frozen HTTP router selects a backend per request; it has no cross-host KV export/import interface and cannot seamlessly hand one request from Spark Prefill to 3080 Decode. Joint Decode uses RDMA compute cooperation between hot experts on the 3080 and cold experts on Spark, rather than polling two independent HTTP services. The container diagnostics disclose this reproduction limit.

Docker image: ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/qwen38-flash-spark-3080:latest. It includes CUDA 13 build tools, RDMA dependencies, pinned TVM-FFI, public sources, role commands and startup diagnostics. The first native build is cached on each host. Matching model assets, GPU drivers and RDMA must be prepared first; see Docker deployment.

V2 metrics · Performance notes

RTX 3080 20GB (OCuLink) computes hot experts and the dense main path; the AI Max 395 (8060S iGPU + Zen5 CPU with AVX-512) computes cold experts. The 3080 accelerates prefill about 3x. Deployment guide & bundle

Exact figures: prefill 843.70 tok/s is the measured 16K-tier peak (C1_ts55_ub1024, 3080 holds 55% of layers, ub 1024; pp1K 612.40 / pp6K 797.00 / pp16K 843.70); decode 40.55 tok/s is the client-measured median (range 28~56 tok/s, rising with expert-cache hit rate 47% to 83%); solo baselines are exact. Decode runs mostly on the AI Max 395 in both deployments, so it stays flat; the gain is in prefill (~3.1x, 273.04 to 843.70 tok/s). The model is the 125B Qwen3.8-Flash-Next hybrid MoE (NVFP4), served by the Strata NVFP4 engine on a single host (mx7).

flowchart LR
    A["API request"] --> E["Strata NVFP4 engine<br/>Port 8095"]
    E --> D["Expert work distribution"]
    D --> G["RTX 3080 20GB / OCuLink<br/>Hot experts and main compute"]
    D --> S["AI Max 395 / 8060S iGPU + Zen5 AVX-512<br/>Cold expert compute"]
    G <-->|"Expert requests and results"| S
    G --> O["Output tokens"]

The reproducibility image is published at ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/ds41-flash-v8:latest. It contains the frozen V8 cross-engine patches, final DSpark delta, runtime environment, and SHA256 evidence; it deliberately does not redistribute the 510 GB official weights or private role images.

• P stage: two RTX 6000D cards in vLLM TP2 (RTX 5500 Pro or RTX 6000 Pro are documented substitutes when validated).
• D stage: four DGX Spark/GB10 nodes in SGLang TP4/EP4, connected by NIXL; the overall design is PP2 with PD separation.

Docker configuration and apply procedure ? Dockerfile ? Compose template

docker pull ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/ds41-flash-v8:latest
docker run --rm ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/ds41-flash-v8:latest show-config

The latest seq32 run of our V8 deployment for DeepSeek-V4.1-Flash (DS4.1 Flash) sustains over 16000 tok/s of P-stage batch Prefill throughput at C16, C24 and C32 with 32K inputs. Short-code C32 reaches 442.02 tok/s over the active Decode interval and 436.40 tok/s of end-to-end aggregate output. These are separate workloads. Consistency here means one batch at each of three concurrency levels, not a repeated long-duration stability test.

32K-input Prefill:

Short-code output:

P-stage batch throughput includes P queuing: total input tokens divided by time from batch launch to the first telemetry sample confirming all P requests complete. It excludes D Decode and covers only the specialized Prefill front stage. Sampling every 0.5 seconds plus collection time makes this an approximate metric. Active-Decode throughput excludes the batch's initial first-text wait; end-to-end output includes waiting, handoff and generation. Neither is a sum of individual stream rates.

What changed from V7: the dual-6000D front stage remains vLLM TP2; the back stage changes from vLLM to SGLang TP4/EP4, with cross-engine NIXL cache-page and state adaptation. The tail-replay parameter drops from 1280 to 256 tokens. This V8 follow-up also raises D running slots and Decode CUDA Graph max_bs from 12 to 32, retaining DSpark 5, flashinfer_cutlass, disabled shared-expert fusion and D chunks of 2048. The original frozen V8 snapshot remains seq12; this section records the subsequently tested seq32 configuration.

All cases use an output budget of 1024 tokens, allowing natural EOS; long inputs are exactly 8192 / 32768 tokens and the short-code prompt has 36 input tokens. Every request uses a unique cache salt. All 216/216 requests across nine cases passed, plus four correctness gates and 16 post-warmup requests. Earlier V7 and H20 records use different workloads, output budgets and timing, so this run supplies no matched cross-version or cross-system speedup ratio.

V8 matrix CSV · V8 sanitized timing evidence

Four DGX Sparks plus two RTX 6000Dpro GPUs reach a Prefill peak of 16698.30 tok/s. Per the experimenter’s September 18 correction, the displayed workload is 32768 input / 128 output / C12, with 12/12 requests completed. The existing machine archive records the same numerical rate at C1; the C12 raw batch is pending.

The six GPUs form a two-stage PP2 pipeline: two RTX 6000D GPUs use TP2 for long-input Prefill, and four DGX Sparks use TP4 to receive context and generate the response. DS4.1's dedicated Prefill path does not require the accelerator side to load the full model. Both 6000D GPUs can therefore cooperate on the entire long-input encoding path, concentrating compute where acceleration matters most.

Why does the model allow this? DeepSeek-V4.1-Flash uses a causal encoder-decoder (CED). Long prompts pass through the encoder, and the decoder's global KV is projected from the encoder's final hidden states. The decoder therefore need not process the entire long prompt again. The official active-parameter counts are 8B per Prefill token and 16B per Decode token; generation still requires the complete encoder-and-decoder path. Official model description · Inference framework explanation

flowchart LR
    IN["Long-input request"] --> P
    subgraph FRONT["PP2 front stage · TP2 · two RTX 6000D GPUs"]
        P["Dedicated Prefill path<br/>Complete long-input encoding"]
        G0["6000D 1<br/>Tensor shard"] <--> G1["6000D 2<br/>Tensor shard"]
        P --- G0
        P --- G1
    end
    P -->|"Context cache · NIXL handoff"| D
    subgraph BACK["PP2 back stage · TP4 · four DGX Sparks"]
        D["Receive context<br/>Replay the tail"] --> GEN["Full-model generation<br/>DSpark speculation"]
        POOL["Spark 1 · Spark 2 · Spark 3 · Spark 4<br/>Four-way tensor parallelism"] --- GEN
    end
    GEN --> OUT["Output tokens"]

TP2 means that two GPUs cooperate on the front stage; TP4 means that four Sparks cooperate on the back stage. PP2 counts stages, while TP counts devices within each stage: 2 + 4 = 6 GPUs. PP2 here describes the experimenter's two-stage organization. V7 connects same-engine P/D services through NIXL, while V8 adapts NIXL between vLLM and SGLang; generation runs the full model on the back stage without sending each new token back to the front stage.

Figure 2 · Why Prefill can use the encoder path

flowchart LR
    subgraph INPUTPATH["Long-prompt Prefill path"]
        PROMPT["All input tokens"] --> ENC["Causal encoder<br/>Accelerated by front-stage TP2"]
        ENC --> H["Final hidden states"] --> KV["Project context KV"]
        KV --> REPLAY["Receive cache on back stage<br/>Reconstruct bounded tail state"]
    end
    subgraph OUTPUTPATH["Decode path for new tokens"]
        NEW["New token"] --> FULL["Back-stage TP4<br/>Encoder + decoder"] --> NEXT["Predict next token"]
    end
    REPLAY --> FULL
    KV -.->|"Reuse context"| FULL

The 6000D pair carries the main computation over the full input. The Sparks reuse the prepared context and perform the implementation's required tail replay before generation. Large-memory hosts hold the complete generation path while compute GPUs handle the long-input matrix work.

Figure 3 · Weight placement and memory roles

flowchart TB
    MODEL["Complete DS4.1 model"] --> PW["Dedicated front-stage Prefill instance<br/>Encoder and required shared components"]
    MODEL --> DW["Back-stage generation instance<br/>Weights required for full generation"]
    PW --> P0["6000D 1<br/>TP2 shard 1"]
    PW --> P1["6000D 2<br/>TP2 shard 2"]
    DW --> DPOOL["Four Sparks · TP4<br/>Model shards + KV + generation workspace"]
    OMIT["Decoder weights unused by front-stage Prefill<br/>Need not reside on the 6000D pair"] -.-> PW
    E["Engram<br/>On-demand lookup and GPU staging"] -.-> PW
    E -.-> DW

Fully using the pair for Prefill means assigning long-input encoding to both 6000D GPUs, without splitting that main path back onto the Sparks merely to fit the complete model. Actual speed still depends on intra-stage communication, Engram lookup, chunking and cache handoff; this is not a measured claim of 100% GPU utilization. The 8B / 16B figures describe active parameters per token, not resident weight memory. TP sharding, external Engram storage and KV budgeting remain necessary in this experiment.

The acceleration chain is: CED enables a specialized Prefill path → TP2 fits and computes the required weights jointly → the 6000D pair handles long-input batches → the large-memory TP4 back stage receives context and generates. This explains why Prefill is the main improvement; measured peaks and matched controls follow below.

The six-GPU entries include the new V8 32K/C32 result and the historical 16698.30 tok/s Prefill peak, alongside standalone TP4 and eight-H20 references. All prices are USD references checked or converted on 2026-09-18; this update does not refresh quotations.

V8 uses sampled P-stage batch throughput; the historical H20 entries use input throughput over complete-request wall time, with different workloads. These are individual measured references, not a common-metric performance ranking.

Pricing basis (2026-09-18): All prices in the table are USD. The Bank of Russia daily rates are 1 USD = 84.5093 RUB and 1 CNY = 12.5788 RUB, implying 1 USD ≈ 6.718391 CNY. Calculations use the unrounded cross rate; displayed amounts are rounded to whole dollars. Converted China quotes and overseas seller listings are identified separately. Taxes, shipping and cluster networking are not normalized.

Six-GPU cost breakdown: Four Sparks at the official US$4,699 unit price total US$18,796, each including 128GB unified memory and a 4TB SSD. The experimenter quotes CNY 65,000 per 6000D, approximately US$9,675 each or US$19,350 for two. The system cost is therefore US$38,146 plus the dual-GPU host and RAM. The latter covers CPU, motherboard, RAM, storage, chassis, PSU and cooling; its actual quote is still missing. US$38,146 is the known subtotal, not a complete system price including the host and RAM.

Overseas H20 system and architecture: ServerICT lists a Dell PowerEdge XE9680 from RUB 21,703,313, approximately US$256,816 and up. Its configuration page lists 8×96GB H20 (768GB total), dual Xeon CPUs and 16×64GB DDR5 RDIMMs (1TB total). The entry SKU D-XE9680-H20-1-64GB lists 2×7.68TB SAS SSDs. The Dell specification confirms a 6U chassis, 500W H20 SXM5 GPUs and full NVLink connectivity; the official NVSwitch support table places H20 96GB in the Hopper family and confirms NVSwitch support. This is a family listing; exact CPUs and delivered configuration require the seller quotation. The experimenter’s China quote of CNY 1,300,000 for a complete 8×96GB H20 server including RAM converts to approximately US$193,499 and is not assumed to have the same configuration as the overseas system.

Benchmark versus purchase model: The community benchmark used H20-3e reporting 143,771 MiB per GPU. As requested, the prices refer to an 8×96GB H20 purchase reference, not to that benchmark hardware.

The six-GPU peak measures only the P stage; TP4 and H20 input rates include generation time, so these rows cannot be used to calculate hardware speedup ratios. Input/output lengths, concurrency and speculation settings also differ. These are measured results, not hardware performance limits.

Scope: C12 and 12/12 in the six-GPU row follow the experimenter’s 2026-09-18 correction. The same numerical rate in the old archive is a C1 P-stage record from 2026-09-17, with zero additional P-side prefix-cache hits; TP4 has 12 formal Prefill batches across chunk sizes 2048 / 4096 / 8192, excluding warmup; H20 has 36 core rounds per engine, excluding warmup, historical-length and arrival-rate experiments. The TP4 peak batch contains 8194 + 8018 + 8019 + 7895 = 32126 input tokens.

Peak CSV · Per-batch metrics, formulas and source evidence. Historical C1 archive calculation: 32768 / 1.962356 ≈ 16698.30 tok/s; this single-request formula does not recompute the corrected C12 result. H20 input rates are recomputed from original round timings: SGLang = 524288 / 106.588177; vLLM = 1572864 / 225.512473. Failed requests remain in the full-wall denominator.

This is aggregate input throughput across the Prefill serving stage: all input tokens / (latest first text − earliest request start), including Prefill, transfer, replay and queuing. Both paths reuse the same D service with 512 output tokens, C8 and two-batch means; code templates match but random prefixes differ. This measures the complete input-serving interval, not isolated GPU kernels.

12K+ is verified at three concurrency levels: a separate formal code matrix with 32K input and 512 output reaches 12804.59 / 13173.51 / 13814.56 tok/s at C4 / C8 / C12. The matrix completed 100/100 requests; the matched C8 comparison above uses independent batches.

V8 seq32 settings: PP2 organization with vLLM TP2 Prefill and SGLang TP4/EP4 Decode; cross-engine NIXL; D running slots and Decode Graph max_bs 32, D chunks 2048, tail-replay parameter 256; DSpark 5, flashinfer_cutlass, shared-expert fusion disabled. V7 settings and failures remain in the historical report sections.

V1–V8 evolution, full comparisons and operating limits · Data · Sanitized evidence. Decode results and failure records remain in the detailed report.

The fastest pair depends on how compute and memory are allocated. Experiments show a relationship: capacity constrains placement, while compute and transfer time determine which placement is fast. The fastest measured profiles for different models and accelerators have an explainable structure. The selection method can be calculated, reasoned about and reused; exact ratios still need calibration, and no universal cross-model formula has been established.

September 14 result: Qwen3.8-Flash-Next NVFP4, RTX 6000D + DGX Spark, 8K Prefill 8157.74 tok/s and C6 aggregate output 414.90 tok/s. These are different workloads on the same capacity profile. The study also tested specially proportioned PP2 layer splitting: 8K Prefill 8696.94, C6 aggregate output 284.56. Maximum Prefill and maximum generation throughput favor different allocations.

Standalone notes (measured 2026-10): 6000D standalone Prefill measured 7831.64 tok/s with 41 tok/s single-stream decode; the Spark standalone figures come from the 2026-10-06 3080+Spark control experiment (NVFP4 weights / FP8 KV), Prefill 1117.93 tok/s being the mean of three 2047-token runs (1030.80 / 1163.40 / 1161.10). Standalone numbers are single-point representative values, not bucketed by 4K/8K, and differ in methodology from the pair results (input length, concurrency, TTFT inclusion) — treat as order-of-magnitude reference.

Parameters of that pair: vLLM nightly; NVFP4 MoE weights / fp8_e4m3 KV; context 65536; concurrency limit 6; Prefill chunk 2048; MTP 3 + full draft vocabulary; capacity profile GMU 0.99 and KV pool 198332 tokens. Prefill uses C1 at 4K–32K; serving emits 256 tokens/request with 3 measured rounds + 1 warmup, 63/63 successful requests over C1–C6. PP2 has 8 GiB KV per device; exact layer allocation stays private.

The 8157.74 / 414.90 profile concentrates computation on the 6000D and resident PLE lookup on Spark; the 8696.94 / 284.56 profile splits computation across both GPUs. Aggregate output includes TTFT and is not pure-phase Decode throughput; the 4K 8016.63 figure selects the second steady-state run. Full parameters and results · CSV

Put a small-VRAM card and a large-memory host to work together.

• 9B: pair Prefill is 34.0% faster than the 3060 alone and 119.6% faster than the 395 alone.
• 27B: 3080 + DGX Spark Prefill is 44.03% faster than this 3080 standalone. 8K input / 128 output, no speculation.
• 27B long context: pair Prefill at 64K reaches 2.33 times the 395 alone; 98K goes from timeout to completion.

A small-VRAM NVIDIA card (RTX 3060 12GB / RTX 3080 20GB) plus a large-memory partner can run large-model inference together. Five routes: RTX 3060 12GB + AMD AI Max+ 395, RTX 3080 20GB + 395, RTX 3080 20GB + DGX Spark GB10, RTX 6000D + DGX Spark GB10, and dual RTX 6000Dpro + four DGX Sparks for six-GPU PD. Dense Acceleration means both devices compute the same stage of the same model at once. Current release v1.6, seven public milestones, plus local Spark addenda dated 2026-09-11, 2026-09-14 and 2026-09-18, without a new public version. Deployment commands, patches, endpoints, and the layer-allocation policy stay private.

C1–C6 means 1 to 6 concurrent requests; C1 is single-stream. Aggregate is the combined speed of the streams that are running together. Speeds are tok/s unless a row names TTFT.

On 9B, the 3060 + 395 pair is the local cell with both a single-card and a single-host control, and the pair is faster than both. On 27B Q4_K_M, DGX Spark is measured only as a 3080 + Spark pair versus this 3080; Spark standalone is not tested (not invented).

llama-bench pp5064 / tg128. Data: benchmark-results.csv. Record: v2.4 fused layer pipeline. Lift = pair ÷ denominator − 1.

Highest Prefill and highest Decode are different working profiles. In the Decode-first profile, the pair and the standalone card each use separately tuned speculation settings. The tables compare each profile and input length separately. Data: qwen27b-spark-3080-profiles.csv. Record: Spark + 3080 profiles.

Highest Prefill confirmed — 8K in / 128 out, no speculation (median of three measured runs after warmup; warmup excluded). Retest of the same profile: 1601.05 / 17.41.

Decode-first, DFlash2 — 2K in / 128 out (3080 standalone uses its own DFlash2 tuning; not the Prefill-confirmed profile).

Decode-first, DFlash2 — 8K in / 128 out (same Decode-confirmed profile as the 2K row; 2K and 8K are not mixed).

Balanced serving C1–C6 has no 3080 standalone and no Spark standalone concurrency control, so this page does not claim a pair gain for those rows. Full C1–C6 tables and timing definitions are under 27B-SPARK-01.

Each row is one experiment: short headline, then the record. Full figures sit in Experiment data.

Three advantages, one bound.

1. In this repository, Dense Acceleration is the 9B cell where the pair beats both standalones. Prefill 2129.69 vs 3060 1589.00 (+34.0%, ÷ 1589.00) and vs 395 970.00 (+119.6%, ÷ 970.00); Decode 50.73 vs 43.87 (+15.6%) and vs 31.27 (+62.2%). The extra is real 395 compute. A second local dense pair is RTX 3080 20GB + DGX Spark GB10, Qwen3.8-27B Q4_K_M: Prefill-confirmed 8K/128 no speculation, pair Prefill 1603.20 vs this 3080 1113.13 (+44.03%, ÷ 1113.13). Only 9B has both standalone baselines; Spark has no Spark standalone baseline, so +44.03% is not a win over every standalone device.

2. The ceiling of phase-separated PD is the accelerator card's own raw compute. 27B-PD-01 pair Prefill 1000.6 reaches 82% of the 3080 llama-bench raw 1228.53, and 1210.6 with the checkpoint off is 98.5% of that ceiling. Those raw and serving figures are not the same method. Decode barely moves in these early single-stream PD tests (9B: 30.28 vs 30.24; 27B: 38.75 vs 36.33). PD schedules the stages separately; DS41-6GPU-01 now demonstrates aggregate serving gains with long-input concurrency, so those early results do not generalize to every PD workload.

3. Dense Acceleration does not require the accelerator card to hold the whole model. Layer-split loading in 27B-LONG-01: pp4096 658.52 vs 395 313.28, pp65536 319.10 vs 136.69, pp98304 from a 900 s timeout to 225.10.

4. Bound: a full single-card plus single-host pair of controls exists only for the 9B Q6_K + RTX 3060 cell (D1). The 27B Spark pair has a matched 3080 standalone at the confirmed Prefill profile, but no Spark standalone baseline. Dense overlapping-pipeline controls on the 395 host for 27B and MoE remain on the roadmap. The dense region needs neighbouring micro-batches to run offset (an overlap window); whatever cannot enter the window belongs to the sparse region, which only has to hold the model.

The early single-stream PD results below are not a universal no-Decode-gain rule: DS41-6GPU-01 now measures aggregate serving gains under long-input concurrency.

The problem first, then the method.

A consumer card computes fast, but its VRAM is small and a larger model does not fit; a large-memory host holds any model, but computes slowly. The fix is to have both devices compute the same stage of the same model at the same time; this is Dense Acceleration. The accelerator card holds the front-stage layers and computes them; the large-memory host holds the rear-stage layers and the intermediate state, and computes that half. The point of the dense region is timing: neighbouring micro-batches run offset, so at any moment both devices have work. Whatever cannot fit into the overlap window belongs to the sparse region: hold the whole model and carry the context capacity.

flowchart LR
    P["Prompt"] --> Q["Async micro-batch queue"]
    subgraph D["Dense Region — concurrent active window"]
        direction LR
        N["Small-VRAM accelerator<br/>front-stage layers"] -->|"current micro-batch"| A["Large-memory host<br/>rear-stage layers and state"]
    end
    Q --> N
    A --> O["Decode and result stream"]
    N -.->|"next micro-batch overlaps"| A

What this is not: it is not running layers one after another, not tensor parallelism, not two devices computing the same layer twice, and not the kind of independent PD that uses the 395 as a remote KV store.

395-host experiments use the five splits below. RTX 3080 20GB + DGX Spark GB10 is the same pair at three working profiles. Short result, then the record.

Only Dense Acceleration is both devices computing the same stage at once in the 395 matrix. The Spark pair also has both devices computing the same model together; its Prefill gain is versus this 3080 standalone only, and there is no Spark standalone baseline.

This is an experiment coverage map across four hardware routes, not a unified ranking under one load. Route 1 is RTX 3060 12GB + AMD AI Max+ 395. Route 2 is RTX 3080 20GB + 395. Route 3 is RTX 3080 20GB + DGX Spark GB10. Route 4 is RTX 6000D + DGX Spark GB10. The six cells split by how much VRAM the running model needs versus the accelerator's VRAM into fits easily, fills the card, and does not fit; each 395-host cell targets five configurations: RTX 3060 alone, RTX 3080 alone, AI Max+ 395 alone, 3060 + 395, and 3080 + 395. Within a cell the model, quantization, prompt, context, concurrency, and metrics must match; a card that cannot hold the model is itself a valid result, and a smaller model or a harsher quantization may not be substituted to manufacture a baseline. The repository holds no RTX 3090 measurements at all; that card was ruled out during selection.

Spark cells with no local measurement are marked not yet tested; 395 figures are not copied into those Spark cells, and community NVFP4 rows are not used as Spark-route evidence.

New fifth route: 2×RTX 6000Dpro + 4×DGX Spark, DeepSeek-V4.1-Flash, V8 cross-engine PD. The matched V7 TP4 controls and new V8 seq32 matrix extend the large-capacity MoE (M3) evidence. Six-GPU results.

One small table per experiment; this is the only place on the front page that holds complete figures. Figures are copied from results/ and data/; if this page and a record disagree, the record and the CSV win. A gain is "pair result ÷ control result − 1", and TTFT is written as how much it dropped. llama-bench raw compute and serving end-to-end are not raced as one method.

Dense Acceleration. 9B-PIPE-01 has both a 3060 single-card control and a 395 single-host control. 27B-SPARK-01 has a matched 3080 standalone at the confirmed Prefill profile and no Spark standalone baseline. Highest Prefill is +44.03% versus this 3080 standalone only.

The latest deployment is V8 seq32, organized as PP2 with vLLM TP2 at the front and SGLang TP4/EP4 at the back. See the opening tables for Prefill consistency across three concurrency levels, C32 output and system references. V1–V8 evolution, all nine cases, historical controls and timing definitions.

The pair completes both capacity balancing and specially proportioned PP2 layer splitting. Respective 8K Prefill results: 8157.74 / 8696.94 tok/s; C6 aggregate output: 414.90 / 284.56 tok/s. Prefill and generation use different input workloads. Full per-stream Decode, TTFT and parameters are in the record.

Each profile completes 63/63 measured requests; C6 alone is 18/18. Aggregate output includes TTFT; output is fixed at 256 tokens. Completion does not certify semantic accuracy of every answer.

Accelerator RTX 3060 12GB; model Ornith 9B, weight quantization Q6_K (the records and CSV label this tier "9B · Q6_K"), llama-bench pp5064 / tg128; data from benchmark-results.csv.

What it verifies: both devices are genuinely working inside the dense region, and the pair beats the fastest single card present. This is the core Dense Acceleration evidence in the repository.

Local measurement on the RTX 3080 20GB + DGX Spark GB10 route, listed on the D2 coverage map. Model Qwen3.8-27B, weight quantization Q4_K_M, KV cache q4_0. "Highest" means the working profile confirmed after this search, not a global optimum. Pair scores are combination scores, not 3080-only. Data from qwen27b-spark-3080-profiles.csv. Record: qwen3.8-27b-spark-3080-profiles.md.

The three confirmed comparison tables are at the top of this page (Prefill-confirmed 8K/128 no speculation; Decode-first DFlash2 2K/128; Decode-first DFlash2 8K/128). Speculation settings on the Decode-confirmed control were tuned on the 3080 standalone and are not a one-variable change. A later live retest of the Prefill-confirmed profile, three repeats at 8K, is 1601.05 / 17.41. Do not quote 1700.

Throughput, RTX 3080 20GB + Spark balanced C1–C6

Speeds below are in tok/s. Prefill comes from separate Prefill waves; Decode and end-to-end output come from complete-request waves generating 256 tokens. The two wave types are summarized by concurrency tier. See the profile record for the timing definitions. No standalone C1–C6 control — no pair-gain column.

Latency, RTX 3080 20GB + Spark balanced C1–C6

36 official waves, 126/126 official requests. What it verifies: pair Prefill at the confirmed Prefill profile is +44.03% versus this 3080 standalone. It does not claim a win over every standalone device, and it does not claim a C1–C6 serving gain.

It runs even when it does not fit. 27B-LONG-01 is layer-split loading: a dense model split by layer across two devices, to solve "does not fit". It is not evidence of overlapping micro-batch pipelining.

Accelerator RTX 3060 12GB; model Qwen3.8-27B, weight quantization UD-IQ3_XXS; pp4096 / pp65536 / pp98304 / tg64, all in tok/s; data from qwen27b-local-results.csv. The 3060 alone cannot hold the whole 27B, which is exactly why the model is split across layers. llama-bench here; not a serving end-to-end race against 27B-PD-01.

What it verifies: when the small card cannot hold the model, the layered split finishes the 98K prompt that the 395 alone could not finish, and at 4K and 64K its Prefill runs more than twice as fast as the 395 alone.

Phase-separated PD. 9B-PD-01, 27B-PD-01, ORNITH-PD-01, and ORNITH-PD-02 are phase-separated PD: the two stages still run one after the other.

Accelerator RTX 3060 12GB; model Ornith 9B, weight quantization Q6_K, serving, 5064 in / 128 out; data from benchmark-results.csv. Host baseline is 395 serving (861.55 / 30.24), not the llama-bench 970.00 / 31.27 used by 9B-PIPE-01.

What it verifies: the state computed on the CUDA side hands over in one piece to the Vulkan side for Decode, the first token comes sooner, and Decode holds. The two phases still run one after the other, so this is phase separation, a different route from Dense Acceleration.

Accelerator RTX 3080 20GB; model Qwen3.8-27B, weight quantization Q4_K_M, KV cache q4_0; all six concurrency tiers C1–C6 passed, tier C1 shown; data from qwen27b-local-results.csv. Host baseline is 395 serving C1 under the v1.0 method (207.2 tok/s). The 3080 llama-bench raw (pp1024 1228.53, pp4096 1203.06, tg64 33.08) is a different method from this serving end-to-end; pair Prefill 1000.6 is 82% of that raw ceiling, not a same-method race.

What it verifies: phase separation works while serving, Prefill does not fall off as concurrency rises, and Decode is unharmed. It verifies that the phases can be separated, not that both devices compute the same phase at once.

The RTX 3080 20GB takes all Prefill and the AI Max+ 395 all Decode; main model Ornith-1.5-35B-A3B with IQ4_XS weight quantization; draft head Qwen3.6-35B-A3B-DFlash with Q4_K_M weight quantization; data from ornith35a3b-local-results.csv. No matched single-host speed control — lifts below are C1 → C6 on the same pair, not pair vs 395.

The same stress run covered two workloads: 1000 in / 128 out, and 100000 in / 128 out; each workload ran 1 to 6 concurrent streams, 1+2+3+4+5+6 = 21 requests, 42/42 succeeded in total, route=pd, n_reuse=0. The 1000-input tier did not separately record the 395 Decode rate, so the table only has Decode for the 100K tier. What it verifies: MoE PD runs stably at 100K context under 1 to 6 concurrent streams, with Prefill attributed to the 3080 and Decode to the 395.

The same model on the same pair of devices, with two additions on top of ORNITH-PD-01's independent PD: Decode runs fused DFlash speculative decode (--spec-type draft-dflash, draft length 6), and the six slots share one unified KV pool (--kv-unified). Main model Ornith-1.5-35B-A3B with IQ4_XS weight quantization; draft head Qwen3.6-35B-A3B-DFlash with Q4_K_M weight quantization; KV cache q4_0; workload 1000 in / 128 out, single stream; data from ornith35a3b-local-results.csv. No matched single-host speed control.

What it verifies: this 3080 + 395 pair measured 4173.47 Prefill and 114.86 single-stream Decode. A different prompt produced 37.2 Decode; the 3.09-fold difference shows sensitivity to the input and draft acceptance, not a same-prompt speedup from enabling the draft head. Draft acceptance on the main row is 107/114 (93.86%). Record

Capacity and serving tiers. 27B-KV-01 is a remote KV pool with two decode routes; FLASH-SPLIT-01 is a single-server split across both devices. Neither has a matched latest standalone speed control for a pair-vs-host claim.

Accelerator RTX 3080 20GB; model Qwen3.8-27B, weight quantization Q4_K_M, KV cache q4_0; data from qwen27b-local-results.csv.

1M context per stream. Configuration C: the 395 decodes; configuration D: the 3080 decodes. On both routes the 395 never runs Prefill and the dense weight compute always stays on the 3080. What it verifies: take C and let the 395 decode when Prefill is the constraint; take D and let the 3080 decode when total Decode throughput is the constraint. This is a capacity and serving route, filed apart from Dense Acceleration.

A single llama-server using the RTX 3080 20GB and the AI Max+ 395 together; model Qwen3.8-Flash, weight quantization Q4, KV cache q4_0; tensor split 0.38 / 0.62, ubatch 1024 / batch 4096, 6 slots at 131072 context; 3080 VRAM peaked at 19129 MiB; data from qwen38flash-q4-local-results.csv. No standalone control — lifts are C1 → C4 on this pair.

C1–C6 aggregate Prefill ran 569.892–633.685 tok/s and aggregate Decode 35.204–71.185 tok/s. 21/21 scored requests succeeded on a workload of about 2077 in / 256 out. What it verifies: the best tier for this configuration is C4, and C5 and C6 no longer rise. It is an operating point, and a different workload must be measured again.

Audit and external reference. 27B-DRAFT-AUDIT-01 is a data audit; EXT-DGX-01 is an external reference, background only. Local Spark Q4_K_M rows above are not merged with those NVFP4 community figures.

Model Qwen3.8-27B, weight quantization Q4_K_M, KV cache q4_0; a data audit run on the AI Max+ 395; data from qwen27b-local-results.csv.

What it verifies: repetitive text and natural language are different workloads. Speculative Decode is sensitive to the input, so a repetitive-text high score does not represent natural-language performance.

This entry contains two external workloads: Qwen3.5 9B with TQ3_4S weight quantization, and Qwen3.8-27B with NVFP4 weights and an FP8 KV cache. Public figures are about 1000 tok/s Prefill, 25–30 tok/s single-stream Decode, and 107 tok/s aggregate Decode, C1–C6. These are public results from someone else's machine, kept as background and never ranked against local data, including the local 27B-SPARK-01 Q4_K_M profiles. Record

Seven releases map onto seven experiments that have data. 27B-SPARK-01 is a 2026-09-11 data addendum on v1.6, not an eighth public version.

Version numbers are experiment milestones, not a count of documentation edits; how they map onto older publication numbers is in VERSION_HISTORY.md.

27B-C and 27B-D are two configurations of the single experiment 27B-KV-01, not two experiments; the natural-language run on the 395 belongs to 27B-DRAFT-AUDIT-01; EXT-DGX-01 remains someone else's public Spark measurement, kept as background; 27B-SPARK-01 is the local 3080 + Spark Q4_K_M measurement.

What conclusion a figure is allowed to support.

• Runs: the request completes, the state hands over, and every metric can be credited to a device. 27B-KV-01, ORNITH-PD-01, ORNITH-PD-02, FLASH-SPLIT-01, and 27B-SPARK-01 belong here, all verified.
• Faster: model, quantization, workload, and metrics all match, and a single-card or single-host control exists. 9B-PIPE-01, 9B-PD-01, 27B-LONG-01, and 27B-PD-01 meet this bar, and their gains are in the tables above. 27B-SPARK-01 meets it only for Prefill versus this 3080 standalone at the confirmed Prefill profile.
• Fits and serves: the workload completes with memory and stability data. That is how 27B-KV-01, FLASH-SPLIT-01, and the 27B-SPARK-01 balanced profile are written.
• Remote KV is a capacity route: a configuration where the 395 only acts as a remote KV pool and never runs dense Prefill compute verifies capacity, with Decode ownership described by the two routes in 27B-KV-01, and is filed apart from Dense Acceleration.
• A speculative-decode point test is not interchangeable with a random-seed stress test: 27B-DRAFT-AUDIT-01 sets a rule and claims no gain.
• External references are background, and unlike conditions are not ranked: EXT-DGX-01 is someone else's public measurement, not a local control; data that differ in model, quantization, engine, prompt, or connection are never joined into one ranking table. A release number is not an experiment ID and not a data source. Every figure must say whether it came from the RTX 3060, the RTX 3080, the 395, or the 3080 + Spark pair.

September 20 addendum: DS41-6GPU-01 V8 seq32. Deployment V1–V8 is a separate evolution line; the public research release remains v1.6. Evolution and controls.

Next: align six-GPU/H20 workload, speculation and timing, and add repeated V8 long-duration runs and cold-restore validation.

The controls that are still missing.

New DS41-6GPU-01: V1–V8 and system comparisons · CSV · Evidence JSON.

This page quotes only the few key figures per experiment; the complete data rows, metric definitions, and field notes are in the records and CSV files below; use the matching CSV for your own calculations, and do not combine data from different experiment IDs unless the record states that a comparable control exists.

The mapping from experiment IDs to legacy labels is in data/experiment-index.csv; all records are under results/. The changelog records what each public experiment version measured. Version mapping: VERSION_HISTORY.md.

公开可复现包：qwen38-flash-spark-3080。3080 承担 NVFP4 dense/prefill 与 FP8 KV，DGX Spark 承担 MoE 冷专家、PLE/KV worker；启动顺序是先 worker，再 engine，模型权重、hot/cold pack、PLE 表和 role 镜像由使用者挂载，镜像不含 510 GB 官方权重。

V2 metrics: DGX Spark performs Prefill alone; Spark + RTX 3080 joint Decode reports 242.37 tok/s aggregate (Spark solo 107.60) and 62.00 tok/s single stream (Spark solo 22.30). The Decode figures were supplied on 2026-10-06; raw logs were not attached.

docker pull ghcr.io/soulmate-halo/heterogeneous-gpu-pd-lab/qwen38-flash-spark-3080:latest
python qwen38-flash-spark-3080/verify_bundle.py