BQ-Wang0511/TalkLikeYou

Official implementation of Talk Like You: Imitating How You Speak in Real-Time Talking Head Generation.

Baiqin Wang, Zhixing Ding, Jijie Li, Jiankuo Zhao, Zhen Lei, Xiangyu Zhu

MAIS, Institute of Automation, Chinese Academy of Sciences; University of Chinese Academy of Sciences; CAIR, HKISI, Chinese Academy of Sciences; Macau University of Science and Technology

TalkLikeYou imitates a target person's speaking habits for real-time audio-driven talking-head generation. It uses a one-step Flow Matching Motion Generator in an 18-dimensional lip-motion space and supports either a preset habit ID or a reference video.

The separate Streaming RealTime Demo provides a browser interface for live microphone animation, video dubbing, and interactive chat-driven avatars. On an NVIDIA RTX 3090, the demo achieves 30+ FPS generation throughput while using less than 4 GB of GPU memory, with high-clarity facial details.

• Release Inference Code
• Release Checkpoint
• Release Streaming RealTime Demo
• PLAD Evaluation

The recommended environment is Linux, Python 3.10, CUDA 12.1, cuDNN 9, and an NVIDIA GPU with at least 12 GB of memory. FFmpeg must be available on PATH.

conda env create -f environment.yaml
conda activate talklikeyou

Alternatively, install a CUDA-compatible PyTorch build and then run:

pip install -r requirements.txt

Download the inference checkpoints from Hugging Face into the repository root:

hf download doubi-killer/TalkLikeYou --include "checkpoints/**" --local-dir .

Keep the downloaded checkpoints/ directory structure unchanged. Model weights are not included in this code repository; see THIRD_PARTY_NOTICES.md for their usage terms.

Run commands from the repository root. Recommended preset IDs are 192, 166, and 202.

CUDA_VISIBLE_DEVICES=0 python -m talklikeyou \
  --source examples/source.jpg \
  --audio examples/audio.wav \
  --person-id 192 \
  --stage0 \
  --neutral \
  --motion-scale 1.0 \
  --guidance-scale 1.3 \
  --sampling-steps 1 \
  --pose-mode audio \
  --pose-sampling-steps 1 \
  --output outputs/preset_192.mp4

CUDA_VISIBLE_DEVICES=0 python -m talklikeyou \
  --source examples/source.jpg \
  --audio examples/audio.wav \
  --habit-reference examples/habit_reference.mp4 \
  --stage0 \
  --neutral \
  --motion-scale 1.0 \
  --guidance-scale 1.3 \
  --sampling-steps 1 \
  --output outputs/reference_habit.mp4

The reference video must contain at least 100 frames. It overrides --person-id when provided.

A source video does not require a habit-reference video.

CUDA_VISIBLE_DEVICES=0 python -m talklikeyou \
  --source path/to/source.mp4 \
  --audio path/to/audio.wav \
  --person-id 166 \
  --stage0 \
  --neutral \
  --motion-scale 1.0 \
  --guidance-scale 1.3 \
  --sampling-steps 1 \
  --pose-mode source \
  --output outputs/dubbed.mp4

Each command produces only one final MP4. If --output is omitted, a randomly named video is written to outputs/. Input and reference videos should use 25 FPS.

--stage0 generates lip motion relative to the built-in neutral mouth expression. --neutral first converts the source mouth to the same stable, slightly open configuration so that teeth and lip structure remain consistent. Using both options is recommended for an image source. For a video source, --neutral automatically uses relative lip motion while preserving the source video's native head pose. No separate neutral image is required.

Pose mode is selected automatically from the source type:

• An image source defaults to --pose-mode audio: TalkLikeYou generates habit-aware lip motion with its one-step Flow Matching Motion Generator, while Ditto generates the accompanying audio-driven head pose. The default Ditto sampling setting is --pose-sampling-steps 1.
• A video source defaults to --pose-mode source: it preserves the video's native head pose and does not load Ditto's pose-generation model.

To preserve the pose of an image and skip Ditto's audio-driven pose generation, select source-pose mode explicitly:

CUDA_VISIBLE_DEVICES=0 python -m talklikeyou \
  --source examples/source.jpg \
  --audio examples/audio.wav \
  --person-id 192 \
  --pose-mode source \
  --output outputs/source_pose.mp4

Either automatic choice can be overridden explicitly with --pose-mode audio or --pose-mode source. Model initialization, face registration, optional neutralization, and audio muxing are one-time preprocessing/postprocessing costs and are not included in steady-state rendering FPS.

@misc{wang2026talklikeyouimitating,
  title={Talk Like You: Imitating How You Speak in Real-Time Talking Head Generation},
  author={Baiqin Wang and Zhixing Ding and Jijie Li and Jiankuo Zhao and Zhen Lei and Xiangyu Zhu},
  year={2026},
  eprint={2610.06658},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.06658}
}

The TalkLikeYou code is released under the MIT License. Third-party code and model files remain subject to their original licenses and usage terms; see THIRD_PARTY_NOTICES.md.

For questions, contact wangbaiqin0511@gmail.com.