Text-to-Speech & Voice

Synthesizing natural speech, and the ethics of cloning it.

L3 · AdvancedFast-movingKnown~14 min

Reviewed by YuvarajPublished 25 Aug 2026

Text-to-speech (TTS) is speech recognition run backwards: instead of turning audio into text, it turns text into natural-sounding audio. Modern neural TTS is good enough that synthetic voices are routinely mistaken for real ones, which makes it both one of the most useful audio technologies and one of the most fraught. This lesson explains the standard two-stage pipeline and the role of the vocoder, what separates robotic speech from human-sounding speech, and then confronts voice cloning and the consent and misuse questions it forces us to take seriously.

The two-stage pipeline

Almost all neural TTS factors the problem into two learned stages with a spectrogram in the middle, the same mel spectrogram representation used everywhere else in audio:

1Text normalizationExpand the raw text into speakable form: "Dr." to "Doctor", "2026" to "twenty twenty-six", handle punctuation and abbreviations.
2Acoustic model: text to spectrogramA model predicts a mel spectrogram from the (normalized) text, deciding pitch, duration, and emphasis for every sound.

Ask the tutor

Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.

Finished this lesson?

Mark it complete to earn XP, keep your streak, and schedule a review.

3Vocoder: spectrogram to waveformA second model converts the mel spectrogram into an actual audio waveform, sample by sample.

4Audio outThe final waveform, ready to play.

Why two stages? The mel spectrogram is a compact, well-behaved target that the acoustic model can predict reliably, but it is not audio, it has thrown away the fine detail and phase needed to actually make sound (as the spectrograms lesson noted, the mel representation is lossy on purpose). The vocoder exists to invent that missing detail: it is a model trained to reconstruct a realistic waveform from the spectrogram. Early vocoders were the quality bottleneck and sounded buzzy; modern neural vocoders (and end-to-end models that fold both stages together) are what pushed synthetic speech across the line into sounding human.

What makes speech sound human

Getting the words right is the easy part. What separates a natural voice from an obviously robotic one is prosody, the music of speech layered on top of the words:

Intonation, the rise and fall of pitch that turns a statement into a question, or signals sarcasm.
Rhythm and timing, where pauses fall, which syllables are stretched, how fast the speech moves.
Stress and emphasis, "I never said she took it" means something different from "I never said she took it."

The difficulty is that prosody is radically underdetermined: the same sentence has countless valid, natural readings, and text alone rarely says which one is intended. That one-to-many mapping is why expressive, controllable prosody remains an active research frontier even though word-level intelligibility is essentially solved.

Concatenative TTS (older)

Stitches together snippets of recorded human speech
Very natural within its recorded inventory
Rigid, cannot say anything outside what was recorded well
Audible glitches at the seams between units

Neural TTS (modern)

Generates the spectrogram and waveform from scratch
Flexible, any text, adjustable speed, pitch, and style
Can clone a voice from a short reference sample
Quality depends on model and vocoder; can still mispronounce rare words

Voice cloning, consent, and misuse

Modern TTS can clone a specific person's voice. What used to take hours of studio recording now takes, in some systems, only seconds of reference audio to produce speech in someone's voice saying words they never said. That capability is genuinely useful, restoring the voice of someone who has lost it to illness, localizing a narrator across languages, personalizing assistants, and genuinely dangerous.

The misuse surface is real and current

Cloned voices are already used for fraud (fake "family emergency" and executive-impersonation phone scams), for non-consensual and defamatory audio, and to defeat voice-based authentication. This is not a speculative future harm, it is a present one, and it scales cheaply.

Responsible practice treats a person's voice as something that requires consent to reproduce, the same way a photograph or signature does. The main mitigations, none of them complete on their own:

Consent and provenance, cloning only voices whose owner has explicitly agreed, with a verifiable record.
Audio watermarking, embedding an inaudible, robust signal that marks audio as synthetic so it can be detected later.
Synthetic-speech detection, classifiers that flag likely-generated audio (an arms race, since generators improve too).
Disclosure, clearly labeling synthetic voices in products so listeners are not deceived.

The honest summary: detection and watermarking help but are not foolproof, so the first line of defense is norms and policy around consent, not a technical guarantee that fakes can always be caught.

Common mistakes

Skipping text normalization. Numbers, dates, units, and abbreviations must be expanded to words first, or the model will read them in unpredictable ways. - Blaming the acoustic model for buzzy audio. Robotic or metallic output is often the vocoder's limitation, not the spectrogram predictor's, the two stages fail differently. - Treating prosody as solved. Intelligible does not mean natural; flat, wrong-emphasis delivery is the usual tell of synthetic speech, and it is genuinely hard to control. - Cloning a voice without consent. Beyond the ethics, this increasingly carries legal and platform-policy consequences; treat a voice like other personal, protected data. - Assuming watermarks or detectors make cloning safe. They raise the bar but can be stripped or evaded; consent and disclosure, not detection alone, are the real safeguards.