How We Built a Custom On-Device TTS in a Week for Under $150

The Problem

At NxtWave, we had a use case that required a TTS model to run on-device. No cloud calls, no internet dependency, real-time synthesis. The voice had to be a specific custom speaker, not a generic pre-trained voice. And the quality had to be good enough that users wouldn't notice or care that it was running locally on their phone.

Cloud TTS APIs (ElevenLabs, Google Cloud TTS, Amazon Polly) are excellent, but they come with latency from network round-trips, recurring per-character costs that scale with usage, and a hard dependency on internet connectivity. For our use case, none of those trade-offs were acceptable.

So we set out to build our own on-device TTS pipeline from scratch. Custom voice, real-time inference, running entirely on the user's phone. This post is the full story of how we did it, what worked, what didn't, and the exact costs involved.

Why MeloTTS

Choosing the right base model was the most important decision in the entire project. Our constraints were clear: the model had to run at real-time speed on modern mobile CPUs (no GPU), it needed to be fine-tunable on a custom speaker, and the output quality had to be production-grade. We evaluated several options:

MeloTTS hit the sweet spot. Under the hood, it's built on VITS2 with a distilled architecture and uses a BERT model for prosody extraction, which gives it noticeably better intonation and natural emphasis compared to plain VITS-based systems like Piper. The NSF-HiFiGAN decoder is lightweight (~1.5M parameters), and the whole system achieves a real-time factor of 0.4–0.55 on CPU. Critically, it supports fine-tuning on a custom speaker from an existing checkpoint, which meant we could train on our specific voice without starting from scratch.

The Data Problem: Bootstrapping 24 Hours of Voice Data

Here's the challenge with fine-tuning a TTS model on a custom voice: you need a lot of high-quality, single-speaker audio data with accurate transcripts. For MeloTTS fine-tuning, we targeted 24 hours of clean, transcribed speech from our target speaker. Recording 24 hours of studio-quality audio from a real person is expensive and time-consuming. Easily weeks of studio time and tens of thousands of dollars.

We took a different approach: a bootstrapped data generation pipeline that let us go from zero to 24 hours of synthetic voice data in under a week, using a combination of ElevenLabs and GPT-SoVITS.

The pipeline worked in three stages:

Stage 1: ElevenLabs Voice Clone

ElevenLabs' Professional Voice Clone produces some of the best zero-shot voice cloning available commercially. We used it as our quality anchor, the source of truth for what our target speaker should sound like.

Creating the Clone

We provided ElevenLabs with reference recordings from our target speaker (a few minutes of clean audio) and created a Professional Voice Clone through their platform. The clone captures the speaker's timbre, cadence, and speaking style with high fidelity.

Generating the Seed Dataset

With the clone ready, we generated 1 hour of audio. The script selection mattered. We needed phonetic diversity to give downstream models a complete picture of how this voice handles all the sounds in English.

# generate_seed_data.py
import os
import time
from elevenlabs import ElevenLabs

client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])
VOICE_ID = "your_cloned_voice_id"
OUTPUT_DIR = "seed_dataset/wavs"

os.makedirs(OUTPUT_DIR, exist_ok=True)

with open("prompts_phonetically_balanced.txt") as f:
    prompts = [line.strip().split("|") for line in f if line.strip()]

for idx, (prompt_id, text) in enumerate(prompts):
    audio = client.text_to_speech.convert(
        voice_id=VOICE_ID,
        text=text,
        model_id="eleven_multilingual_v2",
        output_format="pcm_22050"
    )

    with open(f"{OUTPUT_DIR}/{prompt_id}.wav", "wb") as out:
        for chunk in audio:
            out.write(chunk)

    if idx % 50 == 0:
        print(f"Generated {idx + 1}/{len(prompts)}")
    time.sleep(0.5)

print(f"Done. {len(prompts)} files in {OUTPUT_DIR}")

Stage 2: GPT-SoVITS Amplification

With 1 hour of high-quality ElevenLabs audio in hand, the next step was to amplify this to 24 hours. Generating all 24 hours through ElevenLabs would have been prohibitively expensive. Instead, we used GPT-SoVITS, an open-source two-stage voice cloning system that can be fine-tuned on minimal data.

Why GPT-SoVITS for Data Generation?

GPT-SoVITS uses a GPT-style language model for semantic prediction and a modified VITS (SoVITS) for acoustic synthesis. It supports fine-tuning from as little as 1 minute of data, but with 1 hour of clean, phonetically diverse audio, the speaker similarity is excellent. Any other model of comparable quality would work here. The key requirement is that it can be fine-tuned quickly and generates audio that's good enough to serve as training data (not necessarily production-quality itself).

Fine-Tuning GPT-SoVITS

# Prepare the dataset in GPT-SoVITS format
# Directory structure:
# seed_dataset/
# ├── wavs/
# │   ├── 0001.wav
# │   ├── 0002.wav
# │   └── ...
# └── metadata.list
#     (format: wav_path|speaker_name|language|text)

# Fine-tune GPT-SoVITS on the seed dataset
# This takes ~1-2 hours on a single GPU
python GPT_SoVITS/s1_train.py --config configs/s1.yaml
python GPT_SoVITS/s2_train.py --config configs/s2.yaml

Generating 23 Hours of Data

With the fine-tuned GPT-SoVITS model, we generated the remaining 23 hours of data. We prepared a large, diverse script (~23 hours worth of text) and ran batch synthesis:

# batch_generate.py
import os
from gpt_sovits import GPTSoVITS

model = GPTSoVITS.load("fine_tuned_checkpoint/")
OUTPUT_DIR = "full_dataset/wavs"

os.makedirs(OUTPUT_DIR, exist_ok=True)

with open("prompts_extended.txt") as f:
    prompts = [line.strip().split("|") for line in f if line.strip()]

for idx, (prompt_id, text) in enumerate(prompts):
    audio = model.synthesize(
        text=text,
        ref_audio="seed_dataset/wavs/0001.wav",
        ref_text="Reference transcript for the audio."
    )
    audio.save(f"{OUTPUT_DIR}/{prompt_id}.wav")

    if idx % 200 == 0:
        print(f"Generated {idx + 1}/{len(prompts)}")

print(f"Done. {len(prompts)} files generated.")

Final Dataset

full_dataset/
├── metadata.csv          # pipe-separated: filename|text|normalized_text
└── wavs/
    ├── 0001.wav          # ← from ElevenLabs (first ~200 files)
    ├── 0002.wav
    ├── ...
    ├── 0200.wav
    ├── 0201.wav          # ← from GPT-SoVITS (remaining ~4600 files)
    ├── ...
    └── 4800.wav

# Total: ~24 hours of single-speaker audio
# Avg clip length: ~18 seconds
# Sample rate: 22050 Hz, mono, 16-bit

Training: Fine-Tuning MeloTTS

With 24 hours of clean, transcribed, single-speaker data ready, we moved to the training phase. We fine-tuned from the pre-trained MeloTTS English checkpoint rather than training from scratch.

Why Fine-Tune, Not Train From Scratch?

We actually tried both approaches. The from-scratch model sounded great on sentences structurally similar to the training data, but when we tested it on novel text (different sentence structures, unusual words, varied prosodic patterns) the quality degraded noticeably. The fine-tuned model, starting from the pre-trained checkpoint, retained all of MeloTTS's existing knowledge of English phonetics, prosody, and text normalization. It only needed to learn our speaker's voice identity, which is a much simpler optimization target.

Infrastructure

We ran the training on an Azure Standard_NC24ads_A100_v4 instance, a single NVIDIA A100 80GB GPU at $3.67/hour. The training ran for 24 hours, for a total compute cost of approximately $88.

Training Configuration

# MeloTTS fine-tuning config
{
  "train": {
    "epochs": 1000,
    "learning_rate": 1e-4,
    "batch_size": 32,
    "fp16": true,
    "seed": 42,
    "eval_interval": 50,
    "save_interval": 100
  },
  "data": {
    "training_files": "/data/full_dataset/metadata.csv",
    "validation_split": 0.02,
    "sample_rate": 22050,
    "max_wav_value": 32768.0,
    "filter_length": 1024,
    "hop_length": 256,
    "win_length": 1024,
    "n_mel_channels": 80,
    "mel_fmin": 0,
    "mel_fmax": null
  },
  "model": {
    "use_bert": true,
    "bert_model": "bert-base-uncased",
    "inter_channels": 192,
    "hidden_channels": 192,
    "filter_channels": 768,
    "n_heads": 2,
    "n_layers": 6
  }
}

# Launch fine-tuning
python melo/train.py \
    --config configs/finetune_en.json \
    --model_dir /data/checkpoints \
    --pretrained_model /data/pretrained/melo_english.pth \
    --data_dir /data/full_dataset

ONNX Export & INT8 Quantization

The trained PyTorch model is too large and too slow for on-device mobile inference. MeloTTS has two key components that both need to be optimized: the BERT prosody model and the VITS2 generator (including the NSF-HiFiGAN decoder). We exported both to ONNX and quantized both to INT8.

Step 1: Export to ONNX

import torch
import onnx

# Export the BERT prosody model
bert_model = load_bert("checkpoints/bert_prosody.pth")
dummy_input_ids = torch.randint(0, 30522, (1, 128))
dummy_attention_mask = torch.ones(1, 128, dtype=torch.long)

torch.onnx.export(
    bert_model,
    (dummy_input_ids, dummy_attention_mask),
    "exports/bert_prosody.onnx",
    input_names=["input_ids", "attention_mask"],
    output_names=["prosody_features"],
    dynamic_axes={
        "input_ids": {0: "batch", 1: "seq_len"},
        "attention_mask": {0: "batch", 1: "seq_len"},
        "prosody_features": {0: "batch", 1: "seq_len"}
    },
    opset_version=17
)

# Export the VITS2 generator
generator = load_generator("checkpoints/generator.pth")
# ... similar export with appropriate dummy inputs

torch.onnx.export(
    generator,
    dummy_generator_inputs,
    "exports/generator.onnx",
    opset_version=17,
    dynamic_axes=dynamic_axes_generator
)

Step 2: Quantize to INT8

from onnxruntime.quantization import quantize_dynamic, QuantType
import os

for model_name in ["bert_prosody", "generator"]:
    input_path = f"exports/{model_name}.onnx"
    output_path = f"exports/{model_name}_int8.onnx"

    quantize_dynamic(
        model_input=input_path,
        model_output=output_path,
        weight_type=QuantType.QInt8,
        optimize_model=True
    )

    original_mb = os.path.getsize(input_path) / (1024 * 1024)
    quantized_mb = os.path.getsize(output_path) / (1024 * 1024)
    print(f"{model_name}: {original_mb:.1f} MB → {quantized_mb:.1f} MB "
          f"({original_mb / quantized_mb:.1f}x compression)")

Size Comparison

On-Device Performance

The moment of truth: does it actually run in real-time on a phone?

Inference Setup

import onnxruntime as ort

def create_session(model_path, num_threads=4):
    opts = ort.SessionOptions()
    opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    opts.intra_op_num_threads = num_threads
    opts.inter_op_num_threads = 1
    opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
    return ort.InferenceSession(
        model_path,
        sess_options=opts,
        providers=["CPUExecutionProvider"]
    )

bert_session = create_session("bert_prosody_int8.onnx")
generator_session = create_session("generator_int8.onnx")

Benchmark Results

We benchmarked on two target devices using 50 sentences of varying lengths (5–30 words):

An RTF (Real-Time Factor) below 1.0 means the model generates audio faster than playback speed. An RTF of 0.9 on Snapdragon means a 10-second utterance takes ~9 seconds to synthesize, just fast enough for real-time playback with a small buffer. On the Apple A17 Pro, an RTF of 0.5 means the same 10-second utterance generates in ~5 seconds, leaving comfortable headroom.

Practical Performance

In real-world usage, the experience feels responsive on both platforms. We use a sentence-level chunking strategy: split the input text into sentences, begin synthesizing the first sentence immediately, and start audio playback as soon as the first chunk is ready. By the time the user hears the first sentence, the second is already being synthesized. This pipelining masks the RTF entirely for conversational-length inputs.

import re
import threading
import queue

def stream_tts(text, bert_session, gen_session):
    sentences = re.split(r'(?<=[.!?])\s+', text)
    audio_queue = queue.Queue()

    def synthesize_chunk(sentence, idx):
        prosody = bert_session.run(None, preprocess_bert(sentence))
        audio = gen_session.run(None, preprocess_gen(sentence, prosody))
        audio_queue.put((idx, audio[0]))

    threads = []
    for i, sent in enumerate(sentences):
        t = threading.Thread(target=synthesize_chunk, args=(sent, i))
        t.start()
        threads.append(t)

    for t in threads:
        t.join()

    results = {}
    while not audio_queue.empty():
        idx, data = audio_queue.get()
        results[idx] = data

    return concatenate_audio([results[i] for i in range(len(sentences))])

Total Cost Breakdown

Here's every dollar we spent on this project:

Compare this to the ongoing cost of cloud TTS APIs for the same use case:

• ElevenLabs: $5–22/month depending on plan, with character limits
• Google Cloud TTS (WaveNet): $16 per 1M characters
• Amazon Polly (Neural): $16 per 1M characters

At any meaningful usage volume, the cloud costs would exceed our total spend within 2–3 months. And we'd still have the latency and connectivity problems. The on-device approach pays for itself almost immediately.

Lessons Learned

1. Synthetic data bootstrapping works

The biggest takeaway is that you don't need to record 24 hours of real audio in a studio. The ElevenLabs to GPT-SoVITS pipeline let us bootstrap a large, high-quality dataset from a small amount of reference audio. The key is using a high-quality source (ElevenLabs) for the seed data, then amplifying with a model (GPT-SoVITS) that's good enough for training data even if it's not production-quality on its own.

2. Fine-tuning beats training from scratch

For single-speaker TTS with a custom voice, fine-tuning from a pre-trained checkpoint consistently outperformed training from scratch, especially on generalization to unseen text. The pre-trained model already knows English; you're just teaching it a new voice.

3. Quantize both components

MeloTTS has two major compute paths: the BERT prosody model and the VITS2 generator. Quantizing only the generator and leaving BERT at FP32 would leave the biggest model on the table. Both need INT8 quantization for viable on-device inference.

4. Apple silicon is significantly faster

The nearly 2x RTF difference between Snapdragon and Apple silicon (0.9 vs 0.5) was larger than we expected. If your primary target is iOS, you have considerably more headroom. For Android, you're closer to the edge of real-time, and sentence-level chunking with pipelined playback becomes essential for a smooth user experience.

5. Quality filtering on synthetic data is non-negotiable

Roughly 8–10% of GPT-SoVITS output had artifacts: clipped audio, repeated words, unnatural pauses, or off-speaker-identity samples. If these make it into the training set, the fine-tuned model learns to reproduce them. Automated quality checks (SNR, duration bounds, silence ratio) plus spot-checking saved us from shipping a model that would occasionally glitch.