Edge0-AI/Audio8_TTS

Audio8_TTS Preview

A 0.6B-parameter multilingual text-to-speech model with zero-shot voice cloning.

中文文档: README_zh.md

🚀 How small a zero-shot cloning TTS can be? Meet our latest release, Audio8-TTS-0.1B: a powerful, compact, and even portable text-to-speech model!

This repository provides the audio8_tts Preview checkpoint, Hugging Face remote code, inference tools, and an independent SFT pipeline for multilingual speech generation and zero-shot voice cloning.

Preview status: language coverage is intentionally limited in this release. Use the model primarily with the 11 recommended languages below. Multilingual coverage and Chinese dialect support will be expanded in later releases.

The Preview checkpoint performs best in the following languages:

audio8_tts uses a DualAR architecture inspired by Fish Audio S2 Pro.

The slow AR transformer predicts one semantic token for each audio frame. The fast AR transformer then predicts the frame's codec codebooks, conditioned on the slow hidden state and preceding codebooks. Static KV caches are used by both branches during generation. The checkpoint also bundles its neural codec, so reference encoding and waveform decoding require no separate model.

Python 3.10 or newer and a CUDA-capable GPU are recommended.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Download the checkpoint from Hugging Face and place it in the repository's model/ directory. The expected local checkpoint path is model/audio8_tts_0_6B_preview/. All commands also accept a Hugging Face model ID through --model.

For best synthesis quality, keep each input within 150 characters. Longer text may reduce generation quality; split it into shorter segments when needed.

The reference transcript should match the spoken content in the reference audio.

python audio8_tts_infer.py \
  --text "Welcome to audio8_tts." \
  --reference-audio examples/reference.wav \
  --reference-text "Transcript of the reference recording." \
  --output outputs/clone.wav

python audio8_tts_infer.py \
  --text "This utterance does not use a reference voice." \
  --output outputs/no_reference.wav

Each line in the input manifest is an independent JSON object. Relative audio paths are resolved from the manifest directory.

{"id":"sample_001","text":"Target text","reference_audio":"audio/ref.wav","reference_text":"Reference transcript"}
{"id":"sample_002","text":"Text without a reference voice"}

python audio8_tts_infer.py \
  --input-jsonl data/prompts.jsonl \
  --output-dir outputs/batch \
  --batch-size 2

The batch command writes manifest.jsonl and failures.jsonl. Existing WAV files are skipped unless --overwrite is passed. See python audio8_tts_infer.py --help for sampling and code-saving options.

onnx_runtime/ provides a standalone CPU deployment using weight-only INT4 Slow/Fast AR models, FP16 activations and KV caches, and an FP16 codec. It includes CLI inference, a local web and HTTP service, streaming PCM output, and reference-voice registration without PyTorch or Transformers.

The online sessions use about 1 GiB of memory in the tested Apple M2 setup. During voice registration, the online sessions are released before the codec encoder is loaded to keep peak memory low.

Download the ONNX model from Audio8-TTS-Preview-0.6B-ONNX-INT4 and follow the ONNX Runtime guide.

The adapter in sglang_omni/ provides an OpenAI-compatible service with SGLang paged attention, dynamic batching, a fixed KV cache for the fast codebook decoder, reference-audio encoding, and waveform decoding. It is installed as an independent audio8_tts model plugin and does not overwrite SGLang Omni core files.

The adapter uses internal SGLang Omni interfaces, so deploy it with the tested revision instead of the latest main branch.

Audio8-TTS-Preview-0.1b replaces the pure-attention slow AR backbone with a Falcon-H1 hybrid (Mamba 2 SSM + attention). The adapter detects this automatically from config.json (slow_backbone: falcon_h1 or a mamba_d_ssm field) and switches to the eager hybrid path: the slow backbone runs with one FalconHybridMambaAttentionDynamicCache per request, while the fast codebook AR, semantic sampling and vocoder are shared with the 0.6B path. The 0.6B path is unchanged.

export MODEL=/models/Audio8-TTS-Preview-0.1b
export CONFIG=./sglang_omni/configs/audio8_tts_0_1b.yaml
export MODEL_NAME=audio8/tts-0.1b

CUDA_VISIBLE_DEVICES=0 \
SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
MODEL="${MODEL}" \
CONFIG="${CONFIG}" \
MODEL_NAME="${MODEL_NAME}" \
./sglang_omni/scripts/run_server.sh

Because the Mamba backbone keeps its own hybrid caches and does not consume SGLang KV pages, the adapter disables CUDA graphs and caps the static memory fraction for this model. The slow backbone is bit-exact with the Transformers Falcon-H1 implementation for both prefill and cached decode.

Warm single-stream latency was measured on one NVIDIA H20 with BF16 weights, CUDA Graph, greedy decoding, and 128 generated frames. The output WAV was 5.85-5.94 seconds long; cold start and compilation time were excluded. Lower RTF is better.

See the SGLang Omni implementation and evaluation report for the configuration, implementation details, and validation results.

Run these commands from the Audio8 TTS repository root. The example uses Python 3.12 and uv.

export SGLANG_OMNI_ROOT=/opt/sglang-omni
export MODEL=/models/Audio8-TTS-Preview-0.6b

git clone https://github.com/sgl-project/sglang-omni.git "${SGLANG_OMNI_ROOT}"
git -C "${SGLANG_OMNI_ROOT}" checkout 68a572348837f7b004857b4b07993c20ade4c017

uv venv .venv-sglang --python 3.12
source .venv-sglang/bin/activate
uv pip install -v -e "${SGLANG_OMNI_ROOT}"

hf download AutoArk-AI/Audio8-TTS-Preview-0.6b --local-dir "${MODEL}"
./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_ROOT}"
python3 ./sglang_omni/scripts/verify_install.py --model-path "${MODEL}"

For an existing wheel or site-packages installation, resolve the package directory and install the adapter there:

SGLANG_OMNI_PACKAGE="$(python3 -c 'import importlib.util, pathlib; s=importlib.util.find_spec("sglang_omni"); assert s and s.origin; print(pathlib.Path(s.origin).parent)')"
./sglang_omni/scripts/install_adapter.sh "${SGLANG_OMNI_PACKAGE}"

CUDA_VISIBLE_DEVICES=0 \
SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
MODEL="${MODEL}" \
AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
HOST=0.0.0.0 \
PORT=8010 \
./sglang_omni/scripts/run_server.sh

The default fa3 attention backend is intended for Hopper GPUs such as H20 and H100. Consumer Blackwell GPUs such as RTX 5090 report compute capability (12, 0) and have no FA3 kernel image, so the adapter detects them and selects FlashInfer for the SGLang slow-AR path automatically; the short fixed-cache fast head then uses PyTorch SDPA. No configuration is required.

Setting the variable explicitly still overrides the detection on any GPU:

AUDIO8_TTS_ATTENTION_BACKEND=flashinfer \
CUDA_VISIBLE_DEVICES=0 \
SGLANG_OMNI_ROOT="${SGLANG_OMNI_ROOT}" \
MODEL="${MODEL}" \
./sglang_omni/scripts/run_server.sh

The defaults use model name audio8/tts-0.6b, BF16, one GPU, a 0.2 static memory fraction, and up to 32 running requests. The main runtime controls are MODEL_NAME, AUDIO8_TTS_MEM_FRACTION_STATIC, AUDIO8_TTS_MAX_RUNNING_REQUESTS, AUDIO8_TTS_CHUNKED_PREFILL_SIZE, and AUDIO8_TTS_DISABLE_CUDA_GRAPH. When Torch compilation is enabled, the adapter uses SGLang's native batch-size policy. Set AUDIO8_TTS_TORCH_COMPILE_MAX_BS only when an explicit compile limit is needed. AUDIO8_TTS_ATTENTION_BACKEND defaults to fa3, except on GPUs with no FA3 kernel image such as consumer Blackwell, where it defaults to flashinfer; set it explicitly to override. Set SGLANG_OMNI_SITE_PACKAGES when the runtime dependencies are installed in a separate site-packages directory.

• Install the distribution package that provides libnuma if sgl_kernel fails to import (for example, numactl or libnuma1).
• Put the CUDA toolkit bin directory on PATH. If deep_gemm cannot find nvcc during JIT compilation, also set CUDA_PATH to the toolkit root.
• Keep Transformers on the supported 4.x range (>=4.57.0,<5). Transformers 5.x can produce invalid all-zero codes for this custom-code model.

Generate speech without a reference:

curl -sS --fail-with-body \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "audio8/tts-0.6b",
    "input": "Hello from Audio8 TTS.",
    "response_format": "wav",
    "max_new_tokens": 256,
    "temperature": 0.8,
    "top_p": 0.95,
    "top_k": 50
  }' \
  http://127.0.0.1:8010/v1/audio/speech \
  -o audio8.wav

Generate speech with one reference voice:

curl -sS --fail-with-body \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "audio8/tts-0.6b",
    "input": "This sentence uses the reference voice.",
    "response_format": "wav",
    "temperature": 0.8,
    "top_p": 0.95,
    "top_k": 50,
    "references": [{
      "audio_path": "/data/reference.wav",
      "text": "The exact transcript of the reference recording."
    }]
  }' \
  http://127.0.0.1:8010/v1/audio/speech \
  -o audio8_clone.wav

The reference path must be visible inside the service environment. The current adapter supports TP=1 and one reference per request. Set "stream": true to receive SSE audio chunks as they are generated. For the lowest overhead, use "response_format": "pcm"; each event contains Base64-encoded audio in audio.data. Streaming defaults to 12 codec frames per chunk, with 128 frames of decoder context and a one-frame boundary guard. Override these values with AUDIO8_TTS_STREAM_CHUNK_FRAMES, AUDIO8_TTS_STREAM_CONTEXT_FRAMES, and AUDIO8_TTS_STREAM_GUARD_FRAMES.

Streaming is a server-side opt-in. Start the service with AUDIO8_TTS_STREAM_ENABLED=1 to enable SSE streaming; the default is off, and requests are answered with the complete audio after generation finishes (a request-level "stream": true is then ignored). Non-streaming remains the default for maximum throughput and stable memory usage. response_format of codes/codec/npy always returns codec codes without streaming.

Run the smoke test to verify a deployment:

BASE_URL=http://127.0.0.1:8010 ./sglang_omni/scripts/smoke_test.sh
python3 ./sglang_omni/scripts/stream_smoke_test.py \
  --base-url http://127.0.0.1:8010 \
  --output /tmp/audio8_stream.wav

To build the adapter into an existing image, append sglang_omni/Dockerfile.snippet after the SGLang Omni package and its Python dependencies are installed.

Install the training dependencies first:

pip install -r requirements-train.txt

The target audio field is required. reference_audio and reference_text are optional, but must be provided together.

{"id":"utt_001","text":"Target transcript","audio":"audio/target.wav","reference_audio":"audio/reference.wav","reference_text":"Reference transcript"}
{"id":"utt_002","text":"Another transcript","audio":"audio/another.wav"}

python audio8_tts_prepare.py \
  --input-jsonl data/train.jsonl \
  --output-jsonl prepared_data/train.jsonl \
  --batch-size 4

The prepared manifest points to validated [10, T] NumPy arrays using paths relative to the prepared manifest. Existing valid arrays are reused unless --overwrite is passed.

Single GPU:

TRAIN_JSONL=prepared_data/train.jsonl \
NPROC_PER_NODE=1 \
bash audio8_tts_sft.sh

Eight GPUs on one node:

TRAIN_JSONL=prepared_data/train.jsonl \
NPROC_PER_NODE=8 \
BATCH_SIZE=2 \
GRADIENT_ACCUMULATION_STEPS=8 \
bash audio8_tts_sft.sh

For multi-node training, set NNODES, NODE_RANK, MASTER_ADDR, and MASTER_PORT on each node. Common hyperparameters and output paths can be overridden through the environment variables in audio8_tts_sft.sh; additional Transformers arguments may be appended to the command.

SFT optimizes both the slow semantic/EOS objective and the fast codebook teacher-forcing objective. Set FREEZE_SLOW_AR=true or FREEZE_FAST_AR=true when adapting only one branch. The exported directory remains loadable with standard AutoModel and AutoProcessor APIs using trust_remote_code=True.

Audio8 TTS Preview is the smallest model in this comparison at just 0.6B parameters. Despite using only a fraction of the parameters of the other systems, it delivers results in the first tier of industry-leading SOTA TTS models on the benchmarks below. In particular, it achieves the best English WER and competitive Chinese CER on Seed-TTS, while remaining competitive across the CV3 multilingual evaluation.

Lower WER/CER is better; higher SIM is better. Seed-TTS similarity values are shown as percentages.

Parameter counts are calculated directly from the released weight tensors. MOSS-TTS contains 8,489,841,664 parameters. VoxCPM2's main model contains 2,290,004,544 parameters; the separate AudioVAE is not included in the parameter comparison.

Fish S2 Pro was reevaluated because its official evaluation uses its own normalizer. Higgs Audio v2 was evaluated locally because concrete values were unavailable. All other baseline values were collected from their official reports through the VoxCPM repository.

Different normalizers and evaluators make cross-project values reference comparisons rather than a strictly matched ranking. Evaluation coverage does not expand the Preview's supported-language claim beyond the 11 languages listed above.

• This is a Preview checkpoint with limited multilingual and dialect coverage.
• Very long, noisy, or inaccurate reference clips can reduce stability and speaker similarity.
• Generated speech can be misused for impersonation or misinformation. Obtain consent before cloning a voice and clearly disclose synthetic audio where appropriate.
• Test the model for accuracy, safety, and legal compliance before deployment.

Code and model weights in this repository are released under the Apache License 2.0. See NOTICE for attribution details.

We thank the Fish Audio team for publishing the DualAR architecture used in Fish S2 Pro.