ComfyUI-Ref2VA-VSA: Ultra-Fast Character Video Generation (72s on RTX 4090)
Breakthrough inference speed for MiniMax H3 Reference-to-Video (Ref2VA / R2VA): Generate a full 5-second, 24 fps video conditioned on reference character images in ~72s (warm) / 95s (first run) on a single consumer NVIDIA GeForce RTX 4090 (24GB).
🎬 Empirical Benchmark & Results
Both videos were generated on the exact same NVIDIA GeForce RTX 4090 (24GB), using the exact same prompt, character reference image, seed (981445682258077), and resolution (1344×768, 5.16s @ 24 fps / 124 frames).
Input Character Reference Image
🚀 Ref2VA VSA (Ours - 4 Steps) |
🔬 Video Delta Net / VDN-H3 (8 Steps) |
|---|---|
▶ Download Full 1344x768 Video (MP4) |
▶ Download Full 1344x768 Video (MP4) |
|
|
📊 Comprehensive Performance Comparison (RTX 4090)
| Pipeline | Attention Mechanism | Steps / Sampler | 1st Run Latency | Warm Wall Time | Peak VRAM |
|---|---|---|---|---|---|
| MiniMax H3 Native Dense | Dense Global Softmax | 50 steps / Euler | ~650s | ~650s (10.8 min) | ~22.5 GB |
| H3 Turbo LoRA (Dense) | Dense Global Softmax | 4 steps / Euler | ~210s | ~190s (3.1 min) | ~21.0 GB |
| Video Delta Net (VDN-H3) | Windowed + Linear Delta | 8 steps / er_sde | 213.0s | ~135s (2.2 min) | ~18.5 GB |
| Ref2VA VSA (This Repo) 🚀 | 75% Video Sparsity + Dense Prefix | 4 steps / Euler | 95.2s | ~72 seconds | ~13.5 GB |
🧠 Technical Architecture: Why Ref2VA was Hard for VSA
FastVideo pioneered Visual Sparse Attention (VSA) for text-to-video (T2V) by clustering video tokens into 3D spatial-temporal tiles (4×4×4) and dynamically pruning 75–90% of tiles via a learned gating projection (to_gate_compress).
However, Ref2VA (Reference-to-Video) was widely considered incompatible with VSA because:
- Ref2VA prepends dynamic multimodal condition segments (reference image latents, reference audio latents, and text prompt tokens) ahead of the generated video sequence.
- Naive 3D tiling causes tokens from different modalities to straddle the same tile, corrupting the conditioning masks and breaking identity fidelity.
The Solution: Ref2VAVSAGatePatch
Our patch resolves this with an engineered two-tier attention layout:
- Multimodal Dense-Exempt Prefix: The geometry mapper isolates text, reference image, and reference audio tokens into segment-pure tiles that are completely exempt from top-k pruning. Reference tokens always remain 100% dense, guaranteeing strict character identity adherence.
- Video-Only Sparse Attention: VSA top-k pruning is applied strictly to generated-video key tiles.
- Gate Transplant: The 50 learned
to_gate_compressprojection matrices are transplanted directly onto the Ref2VA base model (minimax_h3_ref2va_pruned_int8_convrot.safetensors).
📦 Requirements
- ComfyUI (latest version with native MiniMax-H3 support).
- comfy-kitchen installed with CUDA
sol_attnsupport. - PyTorch 2.4+ and CUDA 12.1+.
- NVIDIA GPU with 24GB VRAM (RTX 3090, RTX 4090, A5000, L40S, etc.).
Checkpoints & Direct Download Links
Place the following files in your ComfyUI/models/ directories:
| Model Type | File Name | Destination Directory | Download Link |
|---|---|---|---|
| Diffusion Model | minimax_h3_ref2va_pruned_int8_convrot.safetensors |
models/diffusion_models/ |
Download from Comfy-Org |
| VSA Gate | fasth3_vsa_gate.safetensors |
models/loras/ |
Download from Hugging Face |
| Turbo LoRA (4-Step) | minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors |
models/loras/ |
Download from LightX2V |
| Text Encoder | qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors |
models/text_encoders/ |
Download from Comfy-Org |
| Video VAE | minimax_h3_video_vae_fp16.safetensors |
models/vae/ |
Download from Comfy-Org |
| Audio VAE | minimax_h3_audio_vae_fp32.safetensors |
models/vae/ |
Download from Comfy-Org |
💡 Tip: The VSA Gate can also be extracted locally from any official FastH3 base checkpoint using the included
tools/extract_vsa_gate.pyscript.
🚀 Quick Start
-
Clone or copy this repository into your ComfyUI
custom_nodesdirectory:cd ComfyUI/custom_nodes git clone https://github.com/Kablex/ComfyUI-Ref2VA-VSA -
Copy the example character image into your ComfyUI input folder:
cp ComfyUI-Ref2VA-VSA/assets/example_character.jpg ComfyUI/input/
-
Restart ComfyUI. The node appears under:
FastH3/VSA -> Ref2VA VSA Gate Transplant (EXPERIMENTAL) -
Load the ready-to-use workflow from
workflows/ref2va_vsa_4step_rtx4090.jsonand click Queue Prompt.
🛠️ Node Wiring
[UNETLoader (Ref2VA INT8)]
│
▼
[LoraLoaderModelOnly (Turbo 4-Step LoRA, strength=1.0)]
│
▼
[Ref2VAVSAGatePatch (fasth3_vsa_gate.safetensors, sparsity=0.75)]
│
▼
[MiniMaxH3SigmaShift (shift_video=12, shift_audio=3)]
│
├──► [BasicScheduler (steps=4, scheduler=simple)]
│
└──► [BasicGuider] ──► [SamplerCustomAdvanced (sampler=euler)]
🧰 Extracting the Gate File (tools/extract_vsa_gate.py)
If you have a FastH3 VSA base checkpoint (minimax_h3_fastvideo_vsa_datafree_1300step_4step_int8_convrot.safetensors), you can extract fasth3_vsa_gate.safetensors yourself:
python tools/extract_vsa_gate.py \
--input /path/to/minimax_h3_fastvideo_vsa_datafree_1300step_4step_int8_convrot.safetensors \
--output /path/to/ComfyUI/models/loras/fasth3_vsa_gate.safetensors📄 License & Acknowledgements
- Licensed under the Apache License 2.0.
- Special thanks to the FastVideo team for the VSA-H3 attention concept and to the ComfyUI / comfy-kitchen developers for the CUDA Sol-Attention kernel primitives.
🛠️ Professional help
Need this tuned for your own pipeline, GPU or model? We do speed and VRAM audits for ComfyUI image and video workflows, measured before and after. hello@grawwww.xyz