pnnbao97/VieNeu-TTS

🦜 VieNeu-TTS

Important

🦜 VieNeu-TTS v4 — available on vieneu.io

VieNeu-TTS v4 delivers near-original voice cloning fidelity, allowing a short reference clip to be reproduced with very high speaker similarity.

Due to the strength of its voice-cloning capabilities and the potential for misuse, v4 is proprietary and will not be open-sourced. It is available exclusively through the VieNeu API / vieneu.io.

VieNeu-TTS v3 Turbo remains the latest open-source version available in this repository. Future open-source releases, including potential v3.x updates, may also be published here.

Note

🦜 VieNeu-TTS v3 Turbo is officially released! A brand-new architecture designed and trained from scratch by Phạm Nguyễn Ngọc Bảo (codec: MOSS-Audio-Tokenizer-Nano; phonemizer: sea-g2p):

• 48 kHz high-fidelity audio (up from 24 kHz).
• 25 built-in preset voices (10 editors' picks) — stable and consistent, no reference clip needed.
• Natural reading style everywhere — the style follows the reference voice (the style argument is deprecated and ignored).
• Emotion / non-verbal cues (experimental): drop [cười], [thở dài], [hắng giọng] straight into the text.
• Batched generation (batch size up to 32), including a multi-speaker Conversation mode that batches the whole script regardless of speaker.
• Instant voice cloning from a 3–8s clip, with automatic reference denoising.
• Real-time streaming + OpenAI-compatible API + Docker — POST /v1/audio/speech drop-in for the OpenAI SDK / Pipecat / LiveKit; first audio in ~115 ms and 16 concurrent streams under 200 ms on a single RTX 3060 (32 max), CPU-only streaming too. See §3, §4 Benchmarks and docs/streaming.md.
• LoRA fine-tuning — train your own voice or reading style on one consumer GPU (§5).

Try it in the Web UI (backbone "VieNeu-TTS-v3-Turbo") or the SDK (Vieneu(mode="v3turbo"), the default).

🎬 Demos

Tip

Dubbing, Lecture, Tiktok Reel and Audiobook features are exclusively available in the official VieNeu App. This GitHub repository only provides a simplified Gradio demo interface and the core SDK for developers.

📌 Table of Contents

🦜 1. Installation & Web UI

Tip

On Windows? The fastest way to get started is the standalone installer at vieneu.io/#/download — no need to install uv or clone the repo manually. macOS: a similar installer is coming in an upcoming release; for now, please use the uv sync steps below. Docker? Skip the steps below: --profile api-gpu / api-cpu (OpenAI-compatible streaming API) — see §3 API Server & Docker.

Setup with uv (Recommended)

# Windows:
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Linux/macOS:
curl -LsSf https://astral.sh/uv/install.sh | sh

📦 2. Using the Python SDK (vieneu)

Quick Start

pip install vieneu

pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install "transformers==4.57.6"   # pinned — most stable transformers for the GPU SDK
pip install vieneu

import time
from vieneu import Vieneu

# Default = v3 Turbo (48 kHz). GPU → PyTorch (auto-detected).
vieneu = Vieneu() # On a GPU machine you can still switch to ONNX/CPU if you prefer: Vieneu(backend="onnx")

# 1. Built-in voice by name — no reference clip needed
print("🔊 Generating speech...")

start_time = time.time()
audio = vieneu.infer("[cười] Trời ơi, cái giọng nó tự nhiên mà nó mượt mà dã man, nghe không khác gì người thật luôn. Giờ thì tha hồ mà quẩy content với cả kho giọng nói đa dạng, đủ mọi sắc thái biểu cảm. Mọi người bật loa lên rồi cùng trải nghiệm thử với mình nhé!", voice="Phạm Tuyên")
elapsed_time = time.time() - start_time

vieneu.save(audio, "output.wav")
print("✅ Saved to output.wav")

# Tính RTF (Real-Time Factor)
sample_rate = 48000
audio_duration = len(audio) / sample_rate
rtf = elapsed_time / audio_duration

print(f"\n⏱️  Thời gian xử lý: {elapsed_time:.3f}s")
print(f"🎵 Thời lượng audio: {audio_duration:.3f}s")
print(f"📊 RTF: {rtf:.4f}  ({'nhanh hơn' if rtf < 1 else 'chậm hơn'} real-time {1/rtf:.2f}x)" if rtf > 0 else "")

# List the built-in voices
voices = vieneu.list_preset_voices()
print(f"\n🎙️  {len(voices)} built-in voices available:")
for label, voice_id in voices:
    print(f"  - {label} ({voice_id})")

# 2. ⚡ Batch on GPU: infer_batch() runs many texts in ONE batched forward — same API.
#    On a CUDA GPU the chunks from every text share each forward step (big throughput
#    win). On CPU it still WORKS (no error) — just sequentially, so there's no batch
#    gain. Batch caps at max_batch_size (default 32; tune via Vieneu(max_batch_size=64)
#    or infer_batch(..., batch_size=64), or batch_size=1 to disable). A single long
#    infer() also auto-batches its own chunks. For real-time use, infer_stream() is the
#    streaming twin (GPU: 16 concurrent streams — see "Streaming" below). Uncomment to
#    try (GPU recommended):
#
# import time
# texts = [
#     "Chào cả nhà, hôm nay mình sẽ hướng dẫn các bạn cách cài đặt và sử dụng bộ giọng đọc mới.",
#     "Giọng nghe cực kỳ tự nhiên và truyền cảm, lại có thể chuyển đổi biểu cảm một cách linh hoạt.",
#     "Nếu thấy hữu ích, các bạn nhớ để lại một lượt thích và chia sẻ video này cho mọi người nhé!",
# ] * 10   # 30 texts — enough to fill the batch and really show the GPU throughput win
# t0 = time.time()
# audios = vieneu.infer_batch(texts, voice="Hải Đăng")
# elapsed = time.time() - t0
# total_audio = sum(len(a) for a in audios) / 48_000
# print(f"⚡ {len(texts)} texts | audio {total_audio:.1f}s | wall {elapsed:.1f}s | RTF {elapsed/total_audio:.3f}")
# for i, a in enumerate(audios):
#     vieneu.save(a, f"batch_{i}.wav")

Streaming (real-time) 🔊

from vieneu import Vieneu
vieneu = Vieneu()                                  # GPU → PyTorch + stream scheduler; no GPU → ONNX/CPU
for chunk in vieneu.infer_stream("Xin chào các bạn!", voice="Mai Anh"):
    play(chunk)                                   # np.float32 @ 48 kHz — play/write as it arrives

# Pick ONE of these — all serve http://localhost:8000/v1/audio/speech
uv run python -m apps.openai_speech                                  # from the repo (auto-detects GPU/CPU)
docker compose -f docker/docker-compose.yml --profile api-gpu up     # or: Docker, GPU
docker compose -f docker/docker-compose.yml --profile api-cpu up     # or: Docker, CPU only

Available Voices

Reading style — deprecated ⚠️

Warning

style is deprecated on v3 Turbo and has no effect. The reading style is already baked into the reference itself (the speaker embedding + reference codes of the preset voice or of your cloned clip), so every generation follows the reference and comes out in its natural reading style.

The style argument is still accepted by infer, infer_stream, infer_batch and add_voice so existing code keeps running — whatever you pass ("tin_tuc", "doc_truyen", …) is simply ignored. New code should just omit it.

# Old code — still runs, but `style` is ignored
audio = vieneu.infer("Bản tin sáng nay.", voice="Hải Đăng", style="tin_tuc")

# New code — pick the reading character through the voice / reference clip instead
audio = vieneu.infer("Bản tin sáng nay.", voice="Hải Đăng")

Emotion cues (experimental)

audio = vieneu.infer("Nghe hay quá đi [cười]. Để mình nói tiếp [hắng giọng].", voice="Hải Đăng")

Voice cloning

audio = vieneu.infer(
    "Đây là giọng được nhân bản tức thì.",
    ref_audio="my_voice.wav",   # a 3–8s reference clip
    denoise=True,               # default; set False if the clip is already clean
)
vieneu.save(audio, "cloned.wav")

Save & reuse a cloned voice

# Enroll a voice (denoises + extracts the speaker profile once)
vieneu.add_voice("Giọng của tôi", "my_voice.wav")

# Now reuse it anywhere, including the conversation mode
audio = vieneu.infer("Câu này dùng giọng đã lưu.", voice="Giọng của tôi")

# Persist your voices so they load next session
vieneu.save_voices()                 # writes to the default voices file
# vieneu.remove_voice("Giọng của tôi")

# Add a voice you already cleaned yourself → skip denoising
vieneu.add_voice("Giọng sạch", "already_clean.wav", denoise=False)

Clean up a clip on its own

wav, sr = vieneu.denoise("noisy.wav", out_path="clean.wav")   # 44.1 kHz mono

v3 Nano (preview) — for edge devices / weak CPUs only 🪶

Warning

v3 Turbo remains the default and the recommended model. Use v3 Nano only when Turbo is too slow on your hardware (old laptops, mini PCs, ARM boards, CPUs without AVX-512/VNNI where the int8 Turbo build produces garbled audio). Nano is a 48M-parameter flow-matching model (ONNX, CPU, torch-free) and it trades quality for speed:

• Lower quality than v3 Turbo — most noticeably on English and code-switched (En-Vi) text. Vietnamese is close; English words come out with a Vietnamese accent and are less stable.
• 24 kHz output (Turbo: 48 kHz).
• 11 preset voices + voice cloning (ref_audio, add_voice, encode_reference work like Turbo; the three cloning graphs, ~110 MB, download on first use).
• No frame-level streaming — infer_stream yields one finished chunk at a time.

from vieneu import Vieneu

tts = Vieneu(mode="v3nano")                      # ONNX, CPU, torch-free
audio = tts.infer("Xin chào, mình là giọng đọc của VieNeu Nano.", voice="Minh Quân")
tts.save(audio, "nano.wav")                      # 24 kHz

tts.list_preset_voices()                         # Adam, Ái Hân, Mỹ Duyên, Đức Trí, Hữu Quân, Xuân Tiên, Mai Anh, Trúc Ly, Anh Khôi, Minh Quân, Mạnh Dũng
audio = tts.infer("Bản nhanh cho máy rất yếu.", voice="Ái Hân", steps=8, sway=-1)   # ~2× faster

🐳 3. API Server & Docker

Streaming API — OpenAI-compatible (v3 Turbo, CPU or GPU)

# Start the server — pick ONE of these three (all listen on http://localhost:8000):
uv run python -m apps.openai_speech                                  # from the repo (auto-detects GPU/CPU)
docker compose -f docker/docker-compose.yml --profile api-gpu up     # or: Docker, GPU container
docker compose -f docker/docker-compose.yml --profile api-cpu up     # or: Docker, CPU container (torch-free)

# Then, in another terminal: measure TTFA / RTF on your machine
uv run python examples/openai_speech_client.py --bench 8

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
with client.audio.speech.with_streaming_response.create(
    model="vieneu-v3-turbo", voice="Mai Anh", input="Xin chào! Đây là chế độ streaming của VieNeu, phát tới đâu nghe tới đó.", response_format="pcm",
) as r:
    for chunk in r.iter_bytes(4096):   # s16le 48 kHz mono, as it is generated
        play(chunk)

Web UI in Docker

docker compose -f docker/docker-compose.yml --profile gpu up   # or --profile cpu → http://localhost:7860

Legacy v2 API server (LMDeploy) — deprecated

Warning

Deprecated. This LMDeploy server and the remote mode only work with VieNeu-TTS v2, which is no longer updated. They are kept for existing deployments. For v3 Turbo use the streaming API above.

📊 4. Benchmarks

Throughput — how fast a whole text is synthesized

Streaming — first-audio latency and concurrency (infer_stream, OpenAI API)

🎓 5. Fine-tuning (LoRA)

uv sync --extra finetune
uv run python finetune/prepare_dataset.py --dataset-dir finetune/dataset   # CPU, torch-free; all clips = one speaker
uv run python finetune/train_lora.py --data finetune/dataset/train.parquet --run my_voice --merge
uv run python finetune/make_voice.py --audio ref.wav --name "My voice" --out finetune/output/my_voice/merged

tts = Vieneu(mode="v3turbo", backbone_repo="finetune/output/my_voice/merged")   # or your Hub repo
audio = tts.infer("Xin chào!", voice="My voice")        # packed voice — no reference audio needed

🔬 6. Model Overview

🚀 7. Roadmap

🤝 8. Support & Contact

📑 9. Citation

@misc{vieneutts2026,
  title        = {VieNeu-TTS: Advanced Vietnamese Text-to-Speech with Instant Voice Cloning},
  author       = {Pham Nguyen Ngoc Bao},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/pnnbao-ump/VieNeu-TTS}}
}

🌟 Star History

🤝 Contributors

🙏 Acknowledgements