VAST-AI-Research/EdenLive

TL;DR:

• For interactive world models, we decouple state evolution (maintained by coding agents or engines) and appearance (rendered by EdenLive).
• EdenLive renders reference videos to enhanced videos in real time at 24 fps or higher, even on a consumer GPU (RTX 5090).

EdenLive is a real-time generative renderer that treats the reference video as the interface between an interactive program and neural visual synthesis: the upstream program executes the rules and maintains the world state, whereas EdenLive renders the resulting reference video, guided by a target first frame and a text prompt, into detailed frames.

The project page demonstrates over 24 FPS, minute-scale generation, and 480P/720P output. The 5B model supports single-GPU deployment at 720P.

We have open-sourced:

• 5B and 14B model weights, for both bidirectional and few-step (chunk-AR) inference.
• Real-time streaming inference code.
• game-prototype-turbo skill that turns a text prompt into a coarse playable game.
• A frontend to try the full prompt-to-game pipeline.

One Python environment runs both the offline inference in edenlive/ and the real-time service in realtime/.

git clone https://github.com/VAST-AI-Research/EdenLive.git EdenLive
cd EdenLive

conda create -n edenlive python=3.10 -y
conda activate edenlive

# 1. PyTorch for a CUDA 13 driver (use .../whl/cu128 for CUDA 12.8).
pip install torch==2.13.0 torchvision==0.28.0 --index-url https://download.pytorch.org/whl/cu130

# 2. The real-time service and its pinned dependencies.
cd realtime
pip install --no-deps -e ./flashdreams -e ./apps/cam2v -e ./integrations_v2/eden
pip install -r requirements.txt
python -m playwright install chromium
npm install --prefix games
cd ..

# 3. Offline inference and video I/O dependencies.
pip install av==17.1.0 decord==0.6.0 pandas==2.3.3 pyarrow==25.0.1 imageio==2.37.4 imageio-ffmpeg==0.6.0
pip install --no-deps -e .

The default 5B bidirectional profile uses FA4 temporal text attention and requires SM90, BF16, flash-attn-4==4.0.0b31, and nvidia-cutlass-dsl==4.7.1. Use --no-deps when installing kernels into an existing environment to preserve PyTorch. See docs/zh/wan22-fa4.md.

hf download VAST-AI/EdenLive --local-dir ./models/EdenLive

Everything loads from one directory, models/EdenLive under the repository root by default (about 210 GB in total).

models/EdenLive/
  dit-14b/            diffusion_pytorch_model.safetensors, config.json  14B few-step (chunk-AR) DiT
  dit-14b-bidir/      diffusion_pytorch_model.safetensors, config.json  14B bidirectional DiT
  dit-5b/             diffusion_pytorch_model.safetensors, config.json  5B few-step (chunk-AR) DiT
  dit-5b-bidir/       diffusion_pytorch_model.safetensors, config.json  5B bidirectional DiT
  vae-14b/            Wan2.1_VAE.pth, config.json
  vae-5b/             Wan2.2_VAE.pth, config.json
  decoder-14b/        lighttaew2_1.pth                    TAEHV decoder (real-time only)
  decoder-5b/         lighttaew2_2.pth                    TAEHV decoder (real-time only)
  scheduler-14b/      scheduler_config.json               offline only
  scheduler-5b/       scheduler_config.json               offline only
  umt5/               text_encoder/, tokenizer/           UMT5 text encoder
  clip/               image_encoder/, image_processor/    CLIP, 14B I2V only
  vision-encoder/     backbone/, projection.safetensors   proxy-DINO encoder
  flux-c2f/           Diffusers FLUX tree + c2f_lora.safetensors   workbench first frame (real-time only)

The default weight directory is models/EdenLive, relative to the working directory. To specify another directory:

python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLive

The real-time service reads the same tree through MODEL_ROOT in realtime/deploy.conf.

Two proxy recordings ship with the repo under examples/samples/, with a manifest that points at them:

examples/samples/
  manifest.csv        id, prompt, proxy_video, first_frame
  cryo-crossing/       cover.png, first-frame.png, prompts.json, prompts.txt, recording.mp4
  pulse-skirmish/      same five files

recording.mp4 is the greybox proxy (1280x704), and first-frame.png is its first frame. cover.png is a photoreal render of that frame; the manifest's first_frame points to this file. prompts.json contains staticPrompt and timestamped dynamicPrompts; prompts.txt is the corresponding text version.

Bidirectional inference: all frames are denoised jointly.

torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_bidir.yaml \
  runtime.sequence_parallel.size=2

python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_bidir.yaml

Few-step inference: chunk-autoregressive sampling, 4 steps per chunk.

torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_few_step.yaml \
  runtime.sequence_parallel.size=2

python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_few_step.yaml

Real-time service: uses game frames as input and streams video over WebRTC. Configure the weight directory, service addresses, ports, and GPU count in realtime/deploy.conf. 5B uses one GPU by default. Set NPROC for multi-GPU inference.

Set TURN_HOST (the relay address reachable by the browser), TURN_IFACE (the relay interface), and TURN_SECRET (your own password) in realtime/deploy.conf. To run existing games, execute on the GPU host:

cd realtime
apt-get install coturn  # First-time installation
bash turn_server.sh    # Start TURN first
bash start.sh 14b       # Model service; or use 5b

The workbench — a game library, a chat that builds new games from a text description, and a play window — needs all four services. Its first-frame renderer (the FLUX worker) runs FLUX.2 [klein] 9B under the FLUX Non-Commercial License, so the workbench is for non-commercial use only, and that license requires content filtering on the images it generates (see License). Playing registered games with start.sh does not load FLUX.

The bridge host requires the claude CLI, claude auth login, and first-run trust confirmation. Game generation uses --dangerously-skip-permissions. For separate hosts, copy the bridge's .workbench/token to the GPU host's realtime/ directory and set BRIDGE_HOST; leave it empty for a single-host deployment.

cd realtime

# 1. TURN relay, GPU host; skip if already running
bash turn_server.sh

# 2. Workbench bridge, Claude CLI host
bash workbench_bridge.sh

# 3. FLUX first-frame service, GPU host
bash workbench_flux.sh

# 4. Model service, GPU host
bash launch.sh

# Stop services on this host
bash stop.sh

Open http://<host>:8089 (APP_PORT in deploy.conf).

The real-time service uses the TAEHV decoder. See realtime/README.md for the service topology and developer documentation.

The default profiles read a recording manifest (CSV/JSONL/JSON/parquet), one recording per request. Each directory contains recording.mp4, cover.png, and prompts.json; text comes from staticPrompt and dynamicPrompts:

id,proxy_video,first_frame
shot_001,shot_001/recording.mp4,shot_001/cover.png

python -m edenlive.scripts.infer --config CONFIG.yaml \
  data.dataset.params.path=/data/my_manifest.csv \
  data.dataset.params.root=/data

LightX2V provides accelerated inference for EdenLive's 14B and 5B models in both bidirectional and four-step autoregressive modes. Optimizations include low-bit quantization, efficient kernels and operator fusion, KV cache optimizations, torch.compile, CUDA Graphs, and streaming LightTAE decoding.

Real-time inference on a single RTX 5090: EdenLive 5B achieves approximately 24.7 FPS inference throughput at 1280×704, meeting the throughput requirement for real-time 24 FPS playback.

See the EdenLive deployment guide for environment setup, model conversion, quantization, and launch commands.

@misc{edenlive2026,
  title  = {EdenLive: Real-Time Generative Rendering from Any Visual Reference},
  author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Zeyi Zheng and Yifei Song and Zekai Gu and Zhipeng Yu and Peng Wang and Yanpei Cao},
  year   = {2026},
  url    = {https://vast-eden.ai/?p=edenlive}
}

Built on Wan2.1 / Wan2.2, 🤗 Diffusers, DINOv3, FLUX.2 [klein] by Black Forest Labs, and the LightX2V LightTAE decoders. The game renders use Three.js, and the workbench's game-generation skill builds on GameBlocks.

The real-time streaming service under realtime/ builds on NVIDIA FlashDreams.

First-party code is licensed under Apache 2.0 — see LICENSE and NOTICE. realtime/ is also Apache 2.0 and keeps its own LICENSE, NOTICE and THIRD-PARTY-NOTICES. The vendored html-to-image bundle and GameBlocks-derived skill modules retain their MIT licenses and original copyright notices; per-file scope is recorded in REUSE.toml.

The model weights are distributed separately and are not all Apache 2.0:

Each component ships its own license and notice files alongside the weights; see the model repository's README for details.