TaoLiveAIGC/TLive-Omni

🌟 TLive-Omni 🌟: An Omni-Modal Understanding Model for E-Commerce Live Streaming

• 📄 [2026-08-24] Technical Report — TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming is now available on arXiv!
• 🔥 [2026-08-20] Model Release — TLive-Omni is now available on Hugging Face! With TLive-Omni-4B and TLive-Omni-9B.

TLive-Omni is an omni-modal understanding model for e-commerce live-stream, mapping image, video, audio, and text into a unified text-output interface. Built on a Qwen3.5 backbone with a grafted AuT audio encoder, it supports 256K tokens of context, trained via a three-stage SFT recipe followed by Faithful-RFT reinforcement fine-tuning.

• Timestamped Per-vGrid layout — Audio and video tokens are organized into timestamped grid with explicit boundaries, keeping audio segments adjacent to their corresponding visual content for fine-grained temporal alignment over long streams.
• Three-stage SFT recipe — Progressive training from audio-language alignment to full multimodal SFT, developing live-commerce understanding from omni-modal perception to instruction-following responses.
• Faithful-RFT — A reinforcement fine-tuning stage for faithful and real-time live-stream demands, suppressing explicit reasoning traces and directly optimizing answer quality for live-commerce tasks.
• Rich atomic capabilities — A scenario-oriented taxonomy covering speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc, supported by a compact data production engine.
• Strong live-commerce performance with competitive generalization — 4B and 9B variants demonstrate strong results across live-commerce audio, image, and video tasks, together with excellent generalization on general benchmarks.

TLive-Omni is built on a Qwen3.5 backbone and extends it with a audio encoder through a lightweight MLP aligner, forming a unified text-output omni-modal understanding model. For video inputs with audio, each temporal grid is organized into a timestamped grid that interleaves video and audio token blocks, keeping audio segments adjacent to their corresponding visual content. The model supports up to 256K tokens of context at inference.

We evaluate TLive-Omni-4B and TLive-Omni-9B on both live-commerce tasks and general benchmarks. Dash (-) denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined.

Notes: Live ASR and Spk. ASR denote live-commerce ASR and speaker-attributed ASR.

Notes: MMBench results are reported on the EN-DEV-v1.1 split. VAB and RWQA denote VLMsAreBlind and RealWorldQA.

Notes: CharXiv results are on the RQ split.

Notes: LongVB denotes LongVideoBench. V-MME and V-MMMU denote Video-MME and VideoMMMU.

Notes: OmniVB denotes OmniVideoBench. V-Holmes denotes VideoHolmes.

1. Live-Commerce Video QA — Links audio cues about a hidden 4-cm height boost with visual evidence.
2. Temporal Grounding — Localizes repeated appearances of a queried product badge.
3. Product Visual Grounding — Predicted vs. ground-truth bounding boxes for a queried product.
4. OCR — Text extraction with commerce-oriented semantic labels.
5. Dense Video Captioning — Temporally segmented descriptions of a clothing demonstration.
6. Multi-Dimensional Shot Tagging — Structured labels for shot size, camera, layout and content category.

1. Dense Video Captioning — Time-aware segment descriptions with visual frames.
2. Temporal Grounding — Localizes a specific action moment with matched intervals.
3. Visual Grounding — Bounding box prediction for queried regions.
4. OCR — Text extraction from nutrition labels at line-level precision.
5. Omni-Modal QA — Cross-modal reasoning combining audio cues with visual evidence.
6. Multi-Dimensional Shot Tagging — Structured labels for shot size, camera, layout and content category.

This release targets Python 3.10 on Linux x86_64 with CUDA 12.8 and PyTorch 2.10.0.

conda create -n tlive python=3.10 -y
conda activate tlive
pip install -r environments/requirements.txt

environments/requirements.txt includes custom wheels for the supported environment and model. If any wheel does not match your hardware, CUDA version, or Python version, replace it with a compatible build for your setup.

# Text
python examples/inference_five_modes.py \
  --model /path/to/model --mode text

# Image
python examples/inference_five_modes.py \
  --model /path/to/model --mode image --image data/image.jpg

# Standalone audio
python examples/inference_five_modes.py \
  --model /path/to/model --mode audio --audio data/audio.mp3

# Video frames + audio track
python examples/inference_five_modes.py \
  --model /path/to/model --mode vocal-video --video data/vocal_video.mp4

# Video frames only
python examples/inference_five_modes.py \
  --model /path/to/model --mode silence-video --video data/silence_video.mp4

For temporal localization outputs, we recommend the MM:SS - MM:SS interval format, for example 01:23 - 01:35.

For direct processor calls, set use_audio_in_video=True to use the video's audio track, or False for visual-only video:

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    use_audio_in_video=True,
)

First install the pre-built wheel (Python 3.10 + CUDA 12.8 + Linux x86_64), built and tested on NVIDIA H20 GPUs (Hopper, sm_90):

pip install https://github.com/TaoLiveAIGC/TLive-Omni/releases/download/v1.0.0-rc1/vllm-0.19.0+cu128-cp310-cp310-linux_x86_64.whl

If your GPU, driver, or CUDA setup is not compatible with this wheel, build vLLM from source using the customized code in vllm/.

# Text
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode text

# Image
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode image --image data/image.jpg

# Standalone audio
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode audio --audio data/audio.mp3

# Video frames + audio track
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode vocal-video --video data/vocal_video.mp4

# Video frames only
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode silence-video --video data/silence_video.mp4

TLive-Omni is built with reference to the following open-source projects: Qwen3.5, Qwen3-Omni, Transformers, ms-swift, DeepSpeed, and vLLM. We sincerely thank these projects and the Qwen team for their outstanding open-source models.

If you find our work helpful, please consider citing our paper:

@article{hu2026tliveomni,
  title={TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming},
  author={Hu, Yibo and Qian, Yu and Gu, Mao and Tao, Yingfan and Chen, Yuhao and Luo, Yongdong and Liu, Zhuoqun and Jin, Meiguang and Ma, Junfeng},
  journal={arXiv preprint arXiv:2608.20958},
  year={2026}
}

This project is released under the Apache License 2.0.