Dub Anything
Dub films and shows into another language in each actor's own voice, with the original music and sound effects intact. It runs on your own PC, and an AI coding agent directs the dub line by line, the way a dubbing director would.
You give it a video file and target language. You get the same file back as
my-show (dubbed).mkv: the dub plays by default, the original audio is one
track away, and subtitles in both languages are included.
Demos
| A Japanese anime, dubbed into English | A Japanese film, dubbed into English |
|---|---|
Getting started
-
Install/Open a desktop AI app, such as ChatGPT, Claude or OpenCode, with access to your local files and a terminal. Turn on remote control so you can monitor the dubbing process when you're away from your computer.
-
Ask it to dub your movie or show. Give it this repository, your video's path and the language you want. For example:
Use https://github.com/swax/dub-anything to dub D:\movies\my-show.mkv into English. Download the repository and install the prerequisites for me.
-
Let the agent handle the rest. It reads the project instructions, installs the tools, VoiceStudio and models, then directs, renders and checks the dub. It may ask you to sign in or accept a model license; after that, it's mostly hands off. The first run includes large downloads, and a full movie can take hours.
-
Watch the result. You get the dubbed file and a short list of moments to check, with timestamps. If anything needs fixing, just ask the AI: tell it what went wrong and where, and it re-renders the affected parts.
Each actor keeps their native accent by default, using a faster engine with more accurate voice cloning. You can specify a different accent in your request, such as American English for an English dub.
What makes it different
Most dubbing tools run a fixed pipeline: transcribe, translate, clone each speaker from one clip, speed up whatever runs long. Dub Anything differs where a listener notices:
- An agent directs, line by line. It works out who says each line from voice matching, video frames and the dialogue, fixes what speech recognition misheard, writes a translation timed to the actor, and triages every QA flag. It tells you which lines are guesses instead of hiding them.
- Each line is acted like the original. The cloned voice is prompted with the actor's own delivery of that very line, so a shout stays a shout and a murmur stays a murmur.
- Each line takes the actor's time. Every take is re-timed to end where the actor stops, so turns land where they did in the original, without dead air. Each line's length is predicted before rendering, so the translation is written to fill the time.
- Every line is checked. Each rendered line is transcribed back. A garbled one gets fresh takes, up to three rounds, and the best take is kept.
- It reads the picture. Burned-in captions become the transcript (and often name the speaker), and text on screen, like title cards and signs, can be translated as subtitles placed beside it.
- The rest stays original. Between lines you hear the original soundtrack, so songs, laughs and names stay the actor's own.
- Chosen by ear. The defaults were picked by listening to the alternatives side by side. What was measured is in docs/research.md. Feel free to continue fine tuning, and send back your improvements.
What you need
- macOS (Apple Silicon or Intel), or a Windows PC with an NVIDIA GPU with at least 6 GB of memory. Apple Silicon uses Metal (MPS) acceleration. A 16 GB NVIDIA card runs everything at once. (Linux should be a short port; see below.)
- Disk space: about 40 GB for VoiceStudio, the models and the tools' environments, plus room to work: a movie in both voices used about 85 GB until it was cleaned up.
Your agent installs VoiceStudio, the voice engine, and the other prerequisites as part of the first run.
How long it takes
A short clip takes minutes. A movie with the actors' native accents takes about three to four hours of machine time on a laptop RTX 4090, nearly all of it on the GPU. Adding an optional American English audio track brings the total to about seven hours. The agent directs while the renders run, and on a long video it can split the directing across helper agents.
Privacy
Audio and video processing runs locally. Your AI provider may receive transcripts and selected video frames as the agent directs and checks the dub. To run completely offline you need to use a local LLM for the director.
Responsible use
Only dub media you have the rights to, and only clone voices with the consent the use requires. A dub made from someone else's film or voice is for private use unless you have permission.
How it works
- Split the video into ~3-minute parts at pauses in speech.
- Separate the voices from the music and effects (Mel-Band Roformer).
- Transcribe each part with Whisper large-v3, with two other recognisers as a second opinion, and burned-in captions where there are any.
- Cast each line by matching voices across the whole show, checked against video frames.
- Direct (the agent): correct the transcript, translate, and time every line to the actor.
- Voice each line. The actor's voice: OmniVoice clones the actor, prompted with their own delivery of that line. American: IndexTTS speaks native English following the actor's delivery, and Chatterbox converts it to the actor's voice.
- Fit, mix and check. Each take is re-timed to end where the actor stops, mixed over the original background, and transcribed back to catch garbled words.
- Deliver into your video file. Nothing is re-encoded, and every stream is checked against the original before the file is replaced.
The whole pipeline in one diagram
The diagram is docs/process.jg, a
Jumpgate file. It is rendered with
npm run render -- <path>/docs/process.jg --width 1800 --height 2700 from
Jumpgate's packages/jumpgate.
Documentation
- scripts/README.md: script entry points and shared tools
- docs/setup.md: setup instructions for agents
- AGENTS.md: the agent's instructions, and a dub from start to finish
- docs/directing.md: the directing workflow, every tool, and the gotchas
- docs/languages/: notes for each language, and how to add one
- docs/research.md: what was measured and tried
- docs/roadmap.md: what's next
Running it without an agent
The scripts are plain Python, so you can drive a dub yourself and write the
directed scripts by hand. With VoiceStudio's Python
($py = "C:\git\VoiceStudio\.venv\Scripts\python.exe"),
scripts/dub_anything.py runs the machine work of a whole dub:
& $py scripts\dub_anything.py new D:\movies\my-show.mkv --voices both # a job named my_show & $py scripts\dub_anything.py run my_show --watch # prep, then render each part marked ready # write work\my_show-NN\script.json for each part, then: & $py scripts\dub_anything.py ready my_show 01 & $py scripts\dub_anything.py status my_show
Or one step at a time
With the backend running (scripts/backend.py start):
& $py scripts\split.py videos\episode.mkv # long video -> parts at pauses & $py scripts\dub.py prep videos\episode-parts\episode-01.mp4 # upload, separate, transcribe & $py scripts\draft.py episode-01 # careful transcript, suggested cast # write work\episode-01\script.json and the cast in work\voices.json & $py scripts\dub.py generate episode-01 --match-level # clone, time, mix, export & $py scripts\lineqa.py episode-01 # check each rendered line & $py scripts\assemble.py episode --parts 1-4 # one video with both audio tracks & $py scripts\deliver.py D:\movies\episode.mkv actor=videos\episode-parts\dubbed\episode.dubbed.mp4
Some steps need their own environments (separate.py, subs.py,
second.py; see AGENTS.md).
The workflow, the script format and every tool are in docs/directing.md.
Other languages and hardware
Ask for a language pair that isn't here yet, and your agent adds it as part of
the job. The voices aren't the limit: VoiceStudio clones voices in hundreds of
languages. What's new for each language is what the scripts know about it: the
phrases Whisper hallucinates in it, the recognisers that give second opinions,
and how QA compares a take with its line. That lives in one module,
scripts/languages/<code>.py. The agent writes it, then checks the pair on a
short test dub before adding it, as
docs/languages/ describes.
Share new languages if you'd like. Ask your agent to open a pull request
with its module, its notes (docs/languages/<code>.md) and the pair's entry in
PAIRS, so the next person gets it ready-made.
The models run on Metal MPS (on Apple Silicon macOS) or CUDA (on Windows/Linux) and fall back to the CPU without it, slower. Video is encoded with Apple VideoToolbox on macOS, NVENC on Windows/Linux when NVIDIA is present, else x264. Platform differences (paths, desktop app runtime detection, GPU locks, audio separation environments) are handled automatically across macOS and Windows.
Contributing
Bug reports, ideas and pull requests are welcome, new languages most of all.
If your AI fixes a bug, improves the workflow or adds a language, you can ask it
to send the changes back as a pull request. Sharing is optional.
docs/directing.md explains how the pieces fit. Python is
formatted and linted with ruff (uvx ruff format, uvx ruff check), and
the agent setup guide shows how to turn on the pre-commit check.
videos/ and work/ are git-ignored so that source media, transcripts and
voice references stay out of the repository. Don't force-add them, and use
invented examples in docs and commit messages.
Built on
Dub Anything is direction and glue around excellent open-source work:
- VoiceStudio: the voice backend (dubbing jobs, cloning, mixing)
- OmniVoice: the actor's-voice clones
- IndexTTS: native English for the American voice
- Chatterbox: voice conversion to each actor's timbre
- Whisper, through faster-whisper: transcription
- pyannote and WeSpeaker: who speaks when, and voice matching
- Silero VAD: where speech really starts and stops
- Mel-Band Roformer, through python-audio-separator: voices apart from the music
- sherpa-onnx: the second-opinion recognisers
- RapidOCR with PaddleOCR's PP-OCRv5: reading captions and signs
- FFmpeg: cutting, mixing and muxing
License
Dub Anything is MIT-licensed (LICENSE). It talks to VoiceStudio over
HTTP and doesn't include its code. VoiceStudio is AGPL-3.0, and so is
patches/voicestudio-local-fixes.patch, a change to VoiceStudio's source. The
models have their own licenses: pyannote is gated on Hugging Face, the
Sortformer weights are CC-BY-NC, and Chatterbox is MIT. Check each one before
any commercial use.