Text To Speech with Tacotron-2 and FastSpeech using ESPnet.

A Beginer’s Guide to End to End Neural Text To Speech..

Sayak Misra

Sayak Misra

Sep 2, 2020·5 min read

Michael Maasen on Unsplash

Text-to-speech (TTS) as the name suggests, reads aloud text. It takes written words as input and converts them into audio. TTS can help anyone who doesn't want to give the effort to read a book, blog or an article. In this article, we will see how we can create a TTS engine considering we don’t know a thing about TTS.

Text-To-Speech Architecture

The above diagram is a simplistic representation of the architecture we are going to follow. We will look into each and every component in detail and we will be using ESPnet framework for implementation purpose.

Front-end

It has mainly three components :

1. POS Tagger: It does the Part Of Speech tagging of the input text.

2. Tokenize: Tokenize a sentence into words.

3. Pronunciation: It breaks the input text into phonemes, based on the pronunciation. e.g. Hello, how are you → HH AH0 L OW, HH AW1 AA1 R Y UW1. This is done by a Grapheme-to-Phoneme convertor, we are using a neural pre-trained G2P(Grapheme to Phoneme) model in this case. This model is designed to convert English graphemes (spelling) to phonemes (pronunciation). To simply illustrate the working of this G2P model, we can say it consults a dictionary if we want to know the pronunciation of some word and if the word is not stored in the dictionary, it uses a seq2seq model based on TensorFlow to predict the phonemes.

Sequence to Sequence Regressor:

We will be using pre-trained seq-to-seq regressor which inputs Linguistic features(Phonemes) and outputs Acoustic features(Mel-spectrogram). Here we will use Tacotron-2(Google’s) and Fastspeech(Facebook’s) for this operation. so let’s quickly look into both of them:

Tacotron-2

Source.

Tacotron is an AI-powered speech synthesis system that can convert text to speech. Tacotron 2’s neural network architecture synthesises speech directly from text. It functions based on the combination of convolutional neural network (CNN) and recurrent neural network (RNN).

FastSpeech

Source.

(a),(b) Feed-Forward Transformer :

FastSpeech adopts a novel feed-forward transformer structure, discarding the conventional encoder-attention-decoder framework, as shown in the above figure. The major component of the feed-forward transformer is the feed-forward transformer block (FFT block, as shown in Figure (b)), which consists of self-attention and 1D convolution. FFT blocks are used for the conversion from phoneme sequence to mel-spectrogram sequence, with N stacked blocks in the phoneme side and mel-spectrogram side, respectively. Uniquely, there is a length regulator in between, which is used to bridge the length mismatch between the phoneme and mel-spectrogram sequences. (Note: Phonemes are the small, distinct sounds of speech.)

(c) Length Regulator:

The model’s length regulator is shown in the above figure. As the length of the phoneme sequence is smaller than that of the mel-spectrogram sequence, one phoneme corresponds to several mel-spectrograms. The number of mel-spectrograms that aligns to a phoneme is called phoneme duration. The length regulator expands the hidden sequence of phonemes according to the duration in order to match the length of a mel-spectrogram sequence. We can increase or decrease the phoneme duration proportionally to adjust the voice speed and can also change the duration of blank tokens to adjust the break between words in order to control part of the prosody.

(d) Duration Predictor :

The duration predictor is very critical for the length regulator to be able to determine duration of each phoneme. As shown in the above figure, the duration predictor consists of a two-layer 1D convolution and a linear layer to predict the duration. The duration predictor stacks on the FFT block in the phoneme side and is jointly trained with FastSpeech through a mean squared error (MSE) loss function. The label of the phoneme duration is extracted from the attention alignment between the encoder and decoder in an autoregressive teacher model.

Waveform-generator / Vocoder :

We will be using pre-trained seq-to-seq model which inputs Acoustic features(Mel-spectogram) and outputs Waveform(Audio). Here we will use parallel WaveGAN vocoder. Here a generative adversarial network (GAN) architechture is used to generate the waveforms from the mel-spectograms, more about this architecture can be found here.

Implementation

We have implemented the above architecture using ESPnet framework. It provides an amazing structure to easily implement all the above pre-trained models, and integrate them. Here is the notebook of the complete Text-to-Speech implementation.

Conclusion

We have implemented a neural TTS system, using various pre-trained models of Tacotron-2, Fastspeech, Parallel WaveGAN etc. We can further try out other models which might produce even better results.

References

1. https://github.com/kan-bayashi/ParallelWaveGAN

2. https://github.com/espnet/espnet

3. https://www.microsoft.com/en-us/research/blog/fastspeech-new-text-to-speech-model-improves-on-speed-accuracy-and-controllability/

4. https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html

5. http://media.speech.zone/images/Interspeech2017_tutorial_Merlin_for_publication_watermarked_compressed_v2.pdf

6. https://arxiv.org/pdf/1910.11480.pdf

Sayak Misra

Data Science enthusiast, working towards solving various NLP problems at Salesken.ai

Follow

Related

Learn how to train image recognition AI with just 5 lines of code

Train Your Image Recognition AI With 5 Lines of Code

The pyzzicato project — Piano Instrumental Music Generation using Deep Neural Networks

How to fail to generate “corridos tumbados” using AI

Accelerated Signal Processing with GPU support