Image, video and audio
Local text-to-speech and transcription: what hardware you need
Speech models are the gentlest thing you can run on your own hardware. They are small, they fit on almost any card, and the question is whether they keep up, not whether they load. A voice that takes 10 seconds to produce each second of speech is useless for a conversation and fine for a narration you render overnight.
The real-time factor
The unit that matters is neither gigabytes nor tokens per second. It is how fast audio comes out compared with how fast it plays, and it is called the real-time factor: the seconds of audio you get for each second of compute. At 1 you generate exactly as fast as the sound plays, which is the minimum for anything live. At 2 to 10 you can generate on demand with a short wait. Above 10 you stop noticing, and the model is no longer what limits you.
Two kinds of model behave differently. Autoregressive models, such as XTTS and VibeVoice, emit audio as a stream of tokens one at a time, which makes them memory-bound in the same way a language model is. What decides their speed is the token rate per second of audio, so a model that emits 7.5 tokens per second of speech is far cheaper than one that emits 50 at the same size. Parallel models, such as F5-TTS and Kokoro, generate a whole utterance at once and are limited by compute.
| Device | Memory | Kokoro 82M | XTTS v2 | F5-TTS | VibeVoice Realtime 0.5B | Chatterbox |
|---|---|---|---|---|---|---|
| GeForce RTX 3050 6GB | 6 GB | 7.6x real time BF16 | 2.7x real time BF16 | 1.9x real time BF16 | 3.6x real time BF16 | 0.8x real time GGUF Q8_0 |
| GeForce RTX 3060 12GB | 12 GB | 14x real time BF16 | 5.9x real time BF16 | 3.4x real time BF16 | 7.7x real time BF16 | 0.9x real time BF16 |
| GeForce RTX 4090 | 24 GB | 90x real time BF16 | 17x real time BF16 | 22x real time BF16 | 22x real time BF16 | 2.5x real time BF16 |
| Apple M4 Max 64GB | 64 GB | 5x real time BF16 | 8.5x real time BF16 | 1.2x real time BF16 | 11x real time BF16 | 1.3x real time BF16 |
| Apple M4 16GB | 16 GB | 1.3x real time BF16 | 1.9x real time BF16 | 0.3x real time BF16 | 2.5x real time BF16 | 0.3x real time BF16 |
The table reads as follows. Kokoro and VibeVoice Realtime are fast everywhere, including on the smallest cards, which is what makes them the practical choices for a long narration or a live assistant. XTTS v2 is the common choice for voice cloning and runs comfortably on a mid-range GPU. F5-TTS and Chatterbox are heavier, and they want a stronger card to run faster than real time. We have not timed these ourselves, so treat the factors as estimates good to a factor of two, and the memory as the firmer figure.
Transcription is the easy case
Speech to text is a different job and a lighter one. Whisper reads audio in chunks and is compute-bound, so even a small card transcribes many times faster than the recording. The small model needs about 2 GB and runs at dozens of times real time on any recent GPU. The large-v3 model is far more accurate on accents and noisy audio, and it needs about 7 GB.
| Device | Memory | Whisper Small | Whisper Large v3 |
|---|---|---|---|
| GeForce RTX 3050 6GB | 6 GB | 43x real time BF16 | 3.4x real time GGUF Q8_0 |
| GeForce RTX 3060 12GB | 12 GB | 79x real time BF16 | 6.2x real time BF16 |
| GeForce RTX 4090 | 24 GB | 250x real time BF16 | 40x real time BF16 |
| Apple M4 Max 64GB | 64 GB | 28x real time BF16 | 2.2x real time BF16 |
The practical rule is to use the biggest Whisper model your card can hold. The time saved by a small one is seconds and the accuracy lost is a nuisance to fix by hand. On a 6 GB card, large-v3 does not fit comfortably, and the medium model is the sensible choice.
Music generation
Music models are the heaviest in this group, and the ones where local hardware shows its limits. MusicGen emits 50 tokens for every second of audio, so even the small model needs a strong GPU to run in real time, and the large one needs a card with 8 GB and a lot of patience on anything less than an RTX 3090. Stable Audio Open is a diffusion model and runs in about the time the clip lasts on a mid-range card. Plan on waiting, and use short clips while you iterate.
What to buy, if you are buying
For speech alone you do not need to buy anything. A 6 GB card runs Kokoro, VibeVoice and XTTS comfortably, and a CPU runs Kokoro. If you are building a setup for narration, such as videos or audiobooks, the decision comes down to whether you want voice cloning, which wants a mid-range GPU, and whether you will also run language models on the same machine, which is the better reason to buy a larger card. The calculator shows each model against your card, and the GPU guide covers the purchase.
Questions people ask
Can I run text-to-speech without a GPU?
Yes, for the small models. Kokoro, at 82M parameters, generates speech faster than real time on a modern CPU. Larger voice-cloning models are slow on a CPU and much better on even a modest GPU.
How much VRAM does a TTS model need?
The small models fit in 2 GB and the large ones in 4 to 8 GB. VRAM is rarely the limit for speech, and speed is, since a model that is slower than real time cannot narrate live.
What is a real-time factor?
The seconds of audio produced for each second of computing. A factor of 1 means the model generates exactly as fast as the audio plays. Above about 2, you can generate on demand without waiting.
What do I need to run Whisper locally?
A small Whisper model runs in real time on almost anything. The large-v3 model needs about 7 GB of VRAM, and it transcribes several times faster than the audio plays on a mid-range GPU.