Matcha-TTS-PL — Polish Matcha-TTS for a conversational robot

Non-autoregressive Polish text-to-speech (Matcha-TTS, optimal-transport conditional flow matching, 20.9 M parameters) built for a humanoid robot: real-time on a small GPU (≈ 23 ms to first audio on an NVIDIA GB10, 4 ODE steps), 8 blendable reader voices, 18 style tokens (pitch range × speaking rate × question), and a HiFi-GAN vocoder fine-tuned to this model, which removes the phasey layer a stock vocoder adds to predicted mels. Trained from a VCTK warm start on consistency-filtered Wolne Lektury audiobooks (prose only, text/audio agreement verified with Whisper) plus AZON spontaneous speech.

Files

path	what
model/matcha_pl_target.ckpt	the released acoustic model: Lightning checkpoint, 20 speaker rows (ids 0–7 = target readers), 18 style rows
vocoder/hifigan_pl.pt	HiFi-GAN generator fine-tuned on this model's mels (use this one); vocoder/g_02500000_universal = the stock universal vocoder for comparison
onnx/matcha_pl_t2.onnx, onnx/matcha_pl_t4.onnx	acoustic model + fine-tuned vocoder in one ONNX graph, 2 / 4 ODE steps; inputs x (phoneme ids), x_lengths, scales=[temperature, length_scale], spk_emb (float32 [1, 64]); outputs wav, wav_lengths
onnx/voices.json	speaker and style embedding tables for building spk_emb (any blend, any style), plus presets: Wojciech Masiak (default) and a 7-reader blend
data/speaker_map.json, data/speakers.json	speaker id → reader
data/style_map.json, data/style_centroids.json	style token definitions
samples/	synthesised test sentences (manifest.csv: file, voice, text)
RECIPE.md	the full training procedure (data, base, target, vocoder fine-tune)
ATTRIBUTION.md, LICENSE	data attribution (every book, reader, director) and CC BY-SA 4.0
(external)	UI, CLI tools and the Matcha-TTS patch the checkpoint needs: github.com/machinekind/tts-pl-playground

Quick start

git clone https://github.com/machinekind/tts-pl-playground.git && cd tts-pl-playground   # follow its README (Matcha-TTS clone + patch)
python -c "from huggingface_hub import snapshot_download as d; d('machinekind/Matcha-TTS-PL', local_dir='.')"
python playground.py --port 8771            # UI: voices, blends, style tokens, effects, sequence composer, video visualizer
python synth_samples.py --ckpt model/matcha_pl_target.ckpt --vocoder vocoder/hifigan_pl.pt \
    --sentences my_sentences.txt --speakers 2 --steps 4 --temperature 0 --out out/

ONNX Runtime (CPU, GPU or in the browser), no PyTorch needed at run time:

import json, onnxruntime as ort, numpy as np
vj = json.load(open("onnx/voices.json"))
E = lambda i: np.array(vj["speakers"][str(i)]["emb"], np.float32)
spk_emb = E(2)                                         # the default voice: Wojciech Masiak
# a blend: v = mean of E(i); rescale v to the mean length of its parts (a plain average is too short)
# add vj["styles"]["14"]["emb"] for a wider pitch range
sess = ort.InferenceSession("onnx/matcha_pl_t4.onnx")
x = phonemes  # int64 [1, T]: espeak-ng "pl" ids from the playground's polish_cleaners (see RECIPE.md §1)
wav, n = sess.run(None, {"x": x, "x_lengths": np.array([x.shape[1]]), "scales": np.array([0.0, 0.95], np.float32), "spk_emb": spk_emb[None]})

Voices

Speaker ids 0–7 are the target readers (0 Bartosz Bielenia, 1 Katarzyna Faszczewska, 2 Wojciech Masiak, 3 Bartosz Głogowski, 4 Jan Staszczyk, 5 Marek Proszek, 6 Piotr Kopa, 7 Radosław Krzyżowski); ids 8–19 are base-training speakers kept for completeness. Any voice is a weighted blend of speaker embeddings, optionally plus a style embedding: spk_emb = Σ w_i · E_spk[i] / Σ w_i (+ E_style[k]). In PyTorch use --voice "0.33*1+0.33*2+0.34*0" and --style k; the ONNX graphs take spk_emb directly, with both tables in onnx/voices.json, so blends and styles need no re-export. The browser playground exposes this as a mixer.

The default voice is Wojciech Masiak (speaker 2). A 7-reader blend (all except Piotr Kopa) is provided as a preset. Rescale every blend to the average length of its component vectors: a plain average is much shorter than any trained voice and makes short sentences unstable. Blends of several readers are the recommended way to deploy (see Licence and attribution on voice rights).

Style tokens

An extra embedding added to the speaker embedding: pass styles=<id> to synthesise in PyTorch; in ONNX add styles[id].emb from onnx/voices.json to spk_emb. Neutral = 7. Pitch-range tokens move the spread by about ±1.3 semitones; the question tokens give a rising terminal contour for yes/no questions (Polish wh-questions fall — write real punctuation, "?" drives intonation).

id	label
0	flat-range · slow
1	flat-range · slow · question
2	flat-range · normal
3	flat-range · normal · question
4	flat-range · fast
5	flat-range · fast · question
6	mid-range · slow
7	mid-range · slow · question
8	mid-range · normal
9	mid-range · normal · question
10	mid-range · fast
11	mid-range · fast · question
12	wide-range · slow
13	wide-range · slow · question
14	wide-range · normal
15	wide-range · normal · question
16	wide-range · fast
17	wide-range · fast · question

Quality

10 conversational test sentences per voice (4 ODE steps, T 0.5, fine-tuned vocoder; mix0 = 7 readers, equal share, rescaled)

voice	n	WER	CER	UTMOS	F0 spread [st]	chars/s	silence %
Bartosz Bielenia	10	0.012	0.002	3.21	2.59	10.5	21
Katarzyna Faszczewska	10	0.012	0.002	3.35	2.99	10.1	21
Wojciech Masiak	10	0.023	0.006	3.30	3.29	11.7	14
Bartosz Głogowski	10	0.000	0.000	3.26	4.10	10.9	17
Jan Staszczyk	10	0.047	0.061	3.07	3.54	10.7	15
Marek Proszek	10	0.047	0.063	3.19	3.73	11.0	16
Piotr Kopa	10	0.058	0.069	3.27	2.61	10.0	11
Radosław Krzyżowski	10	0.035	0.015	2.81	2.04	10.2	22
mix0	10	0.012	0.002	3.13	2.10	10.2	19

Whisper large-v3 WER/CER, UTMOS (tarepan/SpeechMOS), pitch spread. UTMOS does not capture the vocoder artefacts the fine-tune removes; the vocoder choice was made by listening (see RECIPE.md §4–5).

Latency: NVIDIA GB10, PyTorch bf16 + torch.compile, batch 1, 4 steps ≈ 23 ms to first audio, RTF ≈ 0.006. Apple M-series CPU, ONNX Runtime, 4 steps ≈ 0.4 s for a 4 s sentence. Recommended runtime settings: 4 ODE steps, temperature 0 (the playground default: deterministic and most consistent) up to 0.5 (more variation), length scale 0.9–1.0.

Known limitations

Phrase endings are flatter than a human reader's (the model averages final contours); short one-word replies are less natural than full sentences; only readers' prose style is covered (no shouting, whispering or singing).

Training procedure (summary)

RECIPE.md has the complete, reproducible version. Data: Wolne Lektury prose + AZON, per-clip UTMOS/DNSMOS/F0 statistics, Whisper CER filter, reader ranking by consistency (top 15 → base, top 8 → target), question labels corrected from the measured final pitch, 18 designed style tokens. Base: 40k steps from matcha_vctk (batch 64, bf16, lr 1e-4). Target: 12k steps on the 8 readers (lr 5e-5). Vocoder: HiFi-GAN universal fine-tuned 30k steps on the target model's teacher-forced mels (generator lr 2e-5, discriminators 1e-5, 2k-step generator-only warm-up). About 6 GPU-hours on an RTX 4090.

Licence and attribution

• Weights: CC BY-SA 4.0 (LICENSE). The training audio is CC BY-SA 3.0 PL (Wolne Lektury) and CC BY-SA 4.0 (AZON); ShareAlike propagates to the weights. Every book, reader and director is listed in ATTRIBUTION.md — keep that file with any redistribution or derivative.
• Warm start: Matcha-TTS matcha_vctk.ckpt (MIT; VCTK corpus CC BY 4.0, CSTR, University of Edinburgh). Vocoder: HiFi-GAN universal v1 (MIT), fine-tuned here. Code: Matcha-TTS (MIT), jik876/hifi-gan (MIT), espeak-ng (GPL-3.0, runtime dependency).
• Voices are personal attributes. The CC licence covers the recordings, not the readers' personality rights. The recommended deployment is a blend of two or more readers under a neutral voice name (the seven-reader preset is one); using a single reader's voice commercially should be cleared with the reader or Wolne Lektury. Readers are named here only as data sources.
• Synthetic speech should be disclosed as such where the listener could otherwise take it for a person (EU AI Act, art. 50).