Final final experiment — MLP for each phoneme

In my previous post, I proposed to use the implementation composed of a SdA and a MLP of 2 hidden layers for each phoneme. I will present in this last post the results of this proposal.

The objective is to generate that sentence:

“dont’ask me to carry an oily rag like that”

https://dl.dropboxusercontent.com/u/43075537/final/test_sentence.wav%20

Of course, this sentence won’t be seen by the MLPs. I only trained on the phonemes that are present in this sentence (17 unique phonemes) instead of training on the 40 different phonemes. Therefore, 17 MLPs of 2 hidden layers (300 units each) were trained. One on each phoneme. Their weights were initialized by a SdA.

In order to generate the sentence, I just have to call the right MLP one after another and only input the number of samples needed for the phoneme. My whole implementation can be found here.

The subset of the TIMIT dataset used in this experiment was again composed of the males from the first dialect (for a total of 240 sentences minus the one above). Hence, the number of examples seen by each MLP will not be the same since that some phonemes are more frequent than others.

The hyperparameters of the implementation are almost the same as the ones used in my final experiment post. I just added more epochs (up to 500), and used a smaller batch size (down to 16). I won’t show the 17 learning curves but note that the validation error is between 0.001 and 0.3.

Here is the final result of the generated sentence:

https://dl.dropboxusercontent.com/u/43075537/final/final.wav%20

I also post-processed that signal with a 4th-order Butterworth band-pass filter (cutoff frequency of 300Hz and 4000Hz, as used by Joao). The result is a little bit better:

https://dl.dropboxusercontent.com/u/43075537/final/filtered.wav%20

When generating, the standard deviation is multiplied by a factor of 0.2 (in the same manner as the experiments done by David B, Joao, Hubert or Pierre-Luc). When the constant was set to 1, the acoustic samples were saturating and gave that kind of behaviour:

These late results (one day after the deadline) show that training multiple MLPs – one on each phoneme – could be an alternative method. They are way from being ideal but we can get a sense of what has been generated. Further work on that same avenue would involve a better training of the MLPs because in this experiment I used the same hyperparameters for each MLP.

Last Experiment — Extracting features with SdA

Phonetic information (not working…)

At first, I wanted to add the phonetic information. For some reason, I cannot achieve a good speech generation as those who used a similar architecture. It works for a only few samples (around 20) before exploding, which results in a bunch of NaN for the rest of the generation. There is maybe a normalization that I am missing. The input consisted of the acoustic samples (240) concatenated with the phonetic information (3 one-hot of 40). Here are the results for a 2 hidden layers (800 units each) neural network:

I must have a bug in my code… but time is ticking away, so I decided to come back to my third experiment code (with only the acoustic sample). I will try a stacked denoising auto-encoder (SdA) to see if we can improve the MSE by capturing the distribution of the data.

SdA (working!)

I adapted the code from this tutorial (with rectified linear units and mean squared error) to suit the network of my third experiment. Recall that training is only based on the ‘aw’ phoneme spoken by men of the first dialect in the training set. This subset of 177 418 examples is then split into 50/25/25 for training, validation and test sets.

Due to time constraints, I only tried one architecture which is a 2 hidden layers (300 units each) neural network; without weight decay; a mini-batch of 32 examples; and an initial learning rate of 0.01. The SdA was trained for 20 epochs on each layer with a corruption level of 0.15 and 0.30 respectively. The scripts ran for around 35 min (SdA + MLP) on a GPU.

MSE Without pretraining – best epoch:69, train: 0.012442, valid: 0.057558, test: 0.054483; MSE With pretraining – best epoch: 300 (the last one !!), train: 0.011868, valid 0.056046, test 0.051864

As expected, the training and validation curves with pretraining are slightly better than without pretraining. This confirms Marc-André’s experiment.

I only ran the experiments for 300 epochs. However, the validation error for the MLP pre-trained kept improving through time! Pre-training on the whole TIMIT dataset should provide more meaningful features. I didn’t post the generated speech since that they are equivalent to what I had in my third experiment.

Further work on my code could involve training a MLP for each phoneme but I will stick to that for tonight (I might try that for fun tomorrow).

Third experiment — Adding hidden layers

In this experiment, I will extend my previous post in which I tried to generate a specific phoneme.

I made several modifications to my code (you will find the new version in this repository). The main objective was to go beyond the vanilla MLP that I used during my previous experiments by adding hidden layers.

The hidden layers are stored in a list, so it is very easy to add layers (see the DBN tutorial for an extensive explanation)
The activation of the hidden layers is a rectified linear function.
The learning rate has a decreasing schedule. I used the same implementation as Amjad to decrease linearly the learning rate after 30 epochs.
The code is now adapted to run on GPUs

I found experimentally that the generated sounds were much better without a weight decay (i.e. $\lambda = 0$ ) than with $\lambda = 1e^{-4}$ that I used previously. I also noted that Hubert in his last experiment and Jean-Philippe (if I correctly read his yaml files) also set the weight decay to 0. One solution (that I haven’t tried yet) could be to multiply the weight decay by “one over the number of updates need to go once through the training state” as proposed in the Practical recommendations paper.

The other hyperparameters are:

Mini-batch size: 16
Learning rate = 0.01
2, 3, 4, 5 hidden layers of 300 units each
Number of epochs: 500

The dataset consists of all the phonemes ‘aw’ from the male speakers of the first dialect in the training set. I then split that subset in 50/25/25 for training, validation and test.

Here are the results:

And the audio samples with the learning curves:

2 hidden layers, no weight decay

generated_data

https://dl.dropboxusercontent.com/u/43075537/post3/300x2.wav%20

2 hidden layers, $\lambda = 1e^{-5}$

generated_data

https://dl.dropboxusercontent.com/u/43075537/post3/300x2v2.wav%20

2 hidden layers, $\lambda = 1e^{-6}$

generated_data

https://dl.dropboxusercontent.com/u/43075537/post3/300x2v3.wav%20

3 hidden layers, no weight decay

generated_data

https://dl.dropboxusercontent.com/u/43075537/post3/300×3.wav%20

4 hidden layers, no weight decay

generated_data

https://dl.dropboxusercontent.com/u/43075537/post3/300×4.wav%20

5 hidden layers, no weight decay

generated_data

https://dl.dropboxusercontent.com/u/43075537/post3/300×5.wav%20

What I got at the end sounds mostly like whispering but you actually hear the phoneme ‘aw’! In the next hours, I will include the phonemes as an input for the neural network (and if have time I will also try to use a denoising autoencoder to initialize the weights).

I found that higher regularization tend to give the worst generated sounds even if the MSE is very similar. It is very hard to have some insights on what the neural network is currently learning. Will it give a good generation speech? As proposed by David, we should also implement and output other evaluation metrics because the MSE doesn’t really give a clue on the final results.

Second Experiment — Say ‘aa’ vanilla MLP

In this experiment, I tried to see if it was possible to generate a specific phone (as David did in his post). The results aren’t very good. If the capacity of the model is small, the result (i.e. generated sound) will be noise. If the capacity is larger, the result will sound like electric guitar…

I used the same hyperparameters as my first experiment:

Time window = 240 samples
Learning rate = 0.01
Minibatch size = 20
L2 regularization parameter = 0.0001
tanh activation function for the hidden layer
Linear activation for the output layer

I added more capacity to the model (i.e. more hidden units) and I extended the training time (i.e. more epochs). Gaussian noise was added at the output as explained in my previous post. However I didn’t try to scale the RMSE as David or Joao did.

‘aa’ training set

The objective was to learn the phone ‘aa’ and generate afterward that same phone. I extracted all the phones ‘aa’ from the male speakers of the first dialect, which results in 52280 training examples, 26140 for validation and 26140 for testing. You will find my script to extract those phones here and the MLP here. Note that I used the exact same normalization as the TIMIT class written by Vincent (instead of dividing by a specific scalar value 560 as done in my previous posts).

For those who are curious, I concatenated all the ‘aa’ sequences to hear what it sounds like:

https://dl.dropboxusercontent.com/u/43075537/aa.wav%20

Concatenated phone ‘aa’ from male speakers of the 1st dialect

Results

10 hidden units, 1000 epochs

https://dl.dropboxusercontent.com/u/43075537/post2/10u_1000e.wav%20

Only noise. Best model at epoch 1000: MSE_train = 0.007830, MSE_valid = 0.009223, MSE_test = 0.008822.

100 hidden units, 1000 epochs

https://dl.dropboxusercontent.com/u/43075537/post2/100u_1000e.wav%20

Noise and a final beat. Best model at epoch 760: MSE_train = 0.006946, MSE_valid = 0.008453, MSE_test = 0.008205. Note that we can see the same noise (amplitude of +/- 2000) as the first model above.

500 hidden units, 1000 epochs

https://dl.dropboxusercontent.com/u/43075537/post2/500u_1000e.wav%20

Constant noise at the end. Best model at epoch 728: MSE_train = 0.007135, MSE_valid = 0.008150, MSE_test = 0.008010.

1000 hidden units, 1000 epochs

https://dl.dropboxusercontent.com/u/43075537/post2/1000u_1000e.wav%20

Most realistic sound if we look at the plot but sounds like electric guitar. Best model at epoch 898: MSE_train = 0.007092, MSE_valid = 0.008086, MSE_test = 0.8280.

From the last two results we can see that in this case it requires a lot of epochs to train otherwise it seems to be in an underfitting regime. The best test performance was with the 500 hidden units model (MSE = 0.008010) whereas the 1000 hidden units model has a slightly higher value (MSE = 0.008280). However, the latter has a better generated sound.

If we look closely at the acoustic and compare them to the first figure (the one with all the ‘aa’ sequences concatenated), it seems that the the neural network generates a lot of acoustic samples that have a high amplitude. A postprocessing might help (like a bandpass filter) but I think that the problem is more acute.

Other experiments

The generated sound from the overfitted model (i.e. at the last epoch) is almost the same as the best model (i.e. lowest validation error).

I also changed the L2 regularization parameter (results not shown) by different values (0.00005, 0.00001, 0.000005) and I got almost the same results in terms of MSE and quality of generated sound.

Another point that was discussed in class was to introduce noise in the dataset. To do so, I added gaussian noise to each acoustic sample. Since that the dataset has zero mean and unit variance and that we usually end up with a MSE of around 0.03, the noise was sampled independently from a gaussian distribution with zero mean and [0.05, 0.10, 0.15] standard deviation. The dataset has then 4 times more data.

https://dl.dropboxusercontent.com/u/43075537/post2/noisy_aa.wav%20

Learning curve and acoustic of the generated sound. Best model found at epoch 1945. MLP 1000 hidden units, 2000 epochs, phone ‘aa’.

All the results above are based on the ‘aa’ phone. Here is the same experiment (with noise) on the ‘oy’ phone which is the longest phone.

https://dl.dropboxusercontent.com/u/43075537/post2/noisy_oy.wav%20

Learning curve and acoustic of the generated sound. Best model found at epoch 1716. MLP 1000 hidden units, 2000 epochs, phone ‘oy’

It is hard to conclude that adding noise improves the generated sound since that the resulting sound is not very different. It makes sense to add noise however it increases a lot the computation time.

Quick Experiment – Breaking the sine in one line

As we discussed today in class and as mentioned by Laurent in his post, adding gaussian noise is important to prevent convergence to a constant. In my last post, I found this kind of behaviour which results in a flatline.

Recall Laurent’s post:

$P(x_{t}|x_{t-k},\ldots,x_{t-1})=\mathcal{N}(x_{t}|f(x_{t-k},\ldots,x_{t-1}),\hat{\sigma^{2}})=\mathcal{N}(x_{t}|\hat{x_{t}},\hat{\sigma^{2}})$

where $k$ is the size of the window.

The estimator of the variance is the mean square error (MSE):

$\hat{\sigma^{2}}=\frac{1}{n}\sum_{i=1}^{n}(\hat{x_{t}}-x_{t})^{2}$

where $n$ is the number of training examples.

Compared to my previous post, I only had to modify the output of the neural network to generate speech. Concretely, I sampled from a gaussian distribution which mean equals to the output of the neural network and which variance equals to the MSE of the training examples. You will find that in the same mlp.py script (go to line 380).

Here are the results (note that I used the same previous hyperparameters):

Learning curve. Note that I normalized the samples by dividing them by 560 which is roughly the standard error.

Generation of the acoustic based on a NN trained for 100 epochs. It starts at sample 2500 of the SX397.WAV and then generates the next 30 000 samples.

For those interested in listening the resulting generated speech (in case of the audio player doesn’t work, here is the link to the .wav file):

https://dl.dropboxusercontent.com/u/43075537/generated_data.wav%20

In conclusion it doesn’t sound like Georges Brassens but it does break the undesired behaviour described above!

First experiment — Vanilla MLP with Theano

As a first experiment, I will try to predict the next acoustic sample given a specific number of previous samples. A feedforward neural network (NN) with one hidden layer will be used to perform this task. You will find all my scripts under this repository to generate the dataset (generate_dataset.py) and train a NN (mlp.py).

The work that I will present is mainly inspired by Hubert‘s first experiment and will confirm his results. Please forgive me if everything is not very clean or if it is hard-coded in some places. I never used Theano before and I wanted to give it a try quickly! Please feel free to ask me questions or point out mistakes that I would have made.

Main ideas

I started to code from the script provided in the Theano tutorial for MNIST classification using a NN (here). The NN that I implemented is pretty much the same since they are both NN with one hidden layer.

Nevertheless, there are several differences between my NN and the NN of that tutorial:

The activation function of the output layer is linear instead of a softmax.
The output is a real number instead of an integer.
At the end we only get one unit output ( $n_{out} = 1$ ) instead of 10.
The error function corresponds to the quadratic error ( $\frac{1}{K}\sum_{i}^K(\hat{y_{i}}-y_{i})^2$ where $K$ is the size of the mini-batch) instead of the log-likelihood.

I also used Hubert’s script to generate the dataset. However, the way that he saved the data doesn’t correspond to the input format of the Theano tutorial I mentioned above. That is why I modified his script to save the dataset in a .pkl file à la MNIST with a matrix for the previous samples and a vector for the next samples to predict.

Practical implementation

Generate a .pkl .npz file

Similar to Hubert, I chose the very first subject of the first dialect, in /DR1/FCJF0/. It contains 10 sentences. You will find my script here.

The first nine sentences were split into frames of 241 samples. Remember that 240 samples correspond to 15 ms since the .wav files are samples at 16kHz. The 241th sample is used for prediction given the previous 240 samples. This results in a $343742 \times 240$ matrix plus a vector that contains the sample that we want to predict. Next I randomly shuffled this big matrix and saved 50% for training, 25% for validation and 25% for testing.

What about the tenth sentence? I used it separately since I didn’t want to randomize it for two purposes: reconstruction and generation. The former consists of a window of a fixed size that moves across the tenth sentence to predict the next sample at each iteration. The latter uses the prediction of the next sample as an input for the next moving window.

Another very important thing is the normalization as David Belius pointed out. As he suggested, I just divided all the samples by 560 which corresponds to the standard deviation.

Load the data

Use the load_data2() function if you also want to keep apart one sentence. In that case, the .pkl file will contain 4 datasets instead of 3 (train_set, valid_set, test_set, sentence). If you don’t, just use load_data() like the tutorial. Note that I worked in float64. (Go to utils.py to get the functions)

Hyperparameters and others

Learning rate = 0.01
Minibatch size = 20
Number of epochs = 100
Number of hidden units = 10
L2 regularization parameter = 0.0001
tanh() activation function for the hidden layer
Linear activation for the output layer

Results

Here is the learning curve:

After roughly 30 epochs, both errors only vary slightly…

Speech reconstruction

As I explained above, I tried to reconstruct the 10th sentence. On a practical note, I generated a matrix of $38979 \times 240$ that contains overlapping windows of that sentence and I tried to predict the next value.

It seems that the results look a lot like what Benjamin has. I plotted the acoustic in the same manner as he did (go to affiche.py in my repo).

Speech synthesis

The biggest challenge is to generate a whole sentence from an initial set of 240 samples. I began arbitrarily at the index 2500 and tried to generate the next 30000 samples. At each iteration the window of samples used for prediction is shifted in order to add the previous predicted value.

Generation of the acoustic based on a NN trained for 5 epochs

Generation of the acoustic based on a NN trained for 100 epochs

The results are similar to those obtained by Hubert in his first experiment (a combination of sine waves). However, it is strange that after training the NN for 100 epochs, the generation of the acoustic leads to 0 values.

Next steps

One major drawback of this method is that we have to generate a dataset. For example, the size of the .pkl file was over 700 mo and it only contains 10 sentences… I didn’t try to zip it, but even though that is a lot compared to the 10 raw .npy (or .wav) files that only weight 1 mo in total. There should be a smarter way to run the data through the NN since the information is redondant. As we discussed in class, this issue is important if we want to train on the whole dataset.

I didn’t really look for better hyperparameters. The small changes that I tried (e.g. having more hidden units) lead to the same results.

For the next days, I will try to include the phonemes in the model and I will also try to implement a more complex architecture.

EDIT (Feb 25): For those who want to try out my scripts. I made a few changes since the first commit (Feb 18). I created a utils.py script in which you can find the functions to load the datasets. I also implemented in my main script (mlp.py) functions to save and load the parameters of the best model.

EDIT (March 3rd): The dataset for training is now in a .npz format instead of .pkl.

Some (late) initial thoughts

This year the project in IFT6266 course will focus on a unified approach to speech synthesis based on deep learning algorithms. Speech synthesis consists of mapping a text to the corresponding acoustic signals. This process is also called a text-to-speech system. According to Wikipedia, these systems first appeared in the late 50’s, which is quite old! However, we all noticed that synthetic voice sounds robotic because it lacks naturalness and liveliness (even if Siri or Google Now don’t sound so bad). One drawback is the tone of the speaker which is still and without any emotion in the voice.

As explained by David Krueger, Alex Graves’ application is quite similar to our objectives since it generates handwriting text from a text input given by the user. Wouldn’t it be nice if we could for example synthesize a voice with different intonation or accent from the same text input?

TIMIT dataset will be used for the purpose of the course. It contains audio recordings and the corresponding text/phonemes transcription. David Belius developed a great visualization tool here.

A basic system that doesn’t involve learning would consist of concatenate phones to produce speech sentence. We often listen to them in train stations. However, in such system the available vocabulary is quite small.

Speech can be defined as an acoustic signal with melody, rhythm and harmony that is why a unified approach is preferred to ensure the intelligibility and the naturalness of the voice.

Here is a workflow that explains my perspective on how we should tackle the speech synthesis problem:

Adapted from Analyse, codage et synthèse de la parole book published in 2002

Text Preprocessing

This step is not necessary for the TIMIT dataset since all the sentences have already been preprocessed. Nevertheless if we want to implement a real text-to-speech system, preprocessing will be necessary. Most of the algorithms are rule-based [Reichel and Ptzinger 2006] and consist of normalizing the text by retrieving the litteral form of abbreviations and numbers, e.g. “10 Feb.” becomes “on the twelve of February”.

Text-to-phonemes

Again, the TIMIT dataset already includes a text-to-phonemes conversion. We could rely on a lexicon as the one provided in the TIMIT dataset but a sentence is not a concatenation of words. For example, there are pauses within a sentence. Here is an early work from [Sejnowski and Rosenberg 1987] that uses neural networks to pronounce English text. I haven’t had the time to look at it in details but it seems very interesting.

Prosody analysis

As suggested by Vincent in his post and Yoshua in class, we should also focus on the prosody of the speech. From what I understand, prosody defines the isochrony, the stress and the intonation of a speech. Concretely it means that we will have to predict the duration of sounds, the frequency and the amplitude respectively.

Speech synthesis

This is THE challenging part. How will we process all this information to actually build a framework that will output an acoustic signal that sounds like an intelligible speech?

	Final final experime… on Last Experiment — Extrac…
	Second Experiment… on Quick Experiment – Break…
	TIMIT – FCJF0… on First experiment — Vanil…
	Second MLP architect… on Quick Experiment – Break…
	João Felipe Santos on Quick Experiment – Break…