The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So I want to tell you guys about speech recognition and deep learning.

I think deep learning has been playing an increasingly large role in speech recognition.

And 1 of the things I think is most exciting about this field is that speech recognition is at a place right now where it's becoming good enough to enable really exciting applications that end up in the hands of users.

So for example, if we want to caption video content and make it accessible to everyone, it used to be that we would sort of try to do this, but you still need a human to get really good captioning for something like a lecture.

But it's possible that we can do a lot of this with higher quality in the future with deep learning.

We can do things like hands-free interfaces in cars, make it safer to use technology while we're on the go and keep people's eyes on the road.

Of course, it'd make mobile devices, home devices, much easier, much more efficient and enjoyable to use.

But another actually sort of fun recent study that some folks at Baidu participated in along with Stanford and UW was to show that for even something straightforward that we sort of take for granted as an application of speech, which is just texting someone with voice or writing a piece of text, the studies show that you can actually go 3 times faster with voice recognition systems that are available today.

So it's not just like a little bit faster now, even with the errors that a speech recognition system can make.

And the reason I wanted to highlight this result, which is pretty recent, is that the speech engine that was used for this study is actually powered by a lot of the deep learning methods that I'm gonna tell you about.

So hopefully when you walk away today, you have an appreciation or an understanding of the sort of high-level ideas that make a result like this possible.

So there are a whole bunch of different components that make up a complete speech application.

So for example, there's speech transcription.

So if I just talk, I want to come up with words that represent whatever I just said.

There's also other tasks, though, like word spotting or triggering.

So for example, if my phone is sitting over there and I want to say, hey, phone, go do something for me, it actually has to be listening continuously for me to say that word.

And likewise, there are things like speaker identification or verification so that if I want to authenticate myself or I want to be able to tell apart different users in a room, I've got to be able to recognize your voice, even though I don't know what you're saying.

I'm not going to cover all of them today.

Instead, I'm going to just focus on the bread and butter of speech recognition.

We're going to focus on building a speech engine that can accurately transcribe audio into words.

This is a very basic goal of artificial intelligence.

Historically, people are very, very good at listening to someone talk, just like you guys are listening to me right now.

And you can very quickly turn audio into words and into meaning on your own, almost effortlessly.

And for machines, this has historically been incredibly hard.

So you think of this as like 1 of those sort of consummate AI tasks.

So the goal of building a speech pipeline is, if you just give me a raw audio wave, like you recorded on your laptop or your cell phone, I want to somehow build a speech recognizer that can do this very simple task of printing out hello world when I actually say hello world.

So before I dig into the deep learning part, I wanna step back a little bit and spend maybe 10 minutes talking about how a traditional speech recognition pipeline is working.

If you're out in the wild, you're doing an internship, you're trying to build a speech recognition system with a lot of the tools that are out there, you're going to bump into a lot of systems that are built on technologies that look like this.

So I want you to understand a little bit of the vocabulary and how those things are put together.

And also, this will sort of give you a story for what deep learning is doing in speech recognition today that is kind of special and that I think paves the way for much bigger results in the future.

So traditional systems break the problem of converting an audio wave, of taking audio and turning it into a transcription into a bunch of different pieces.

So I'm going to start out with my raw audio, and I'm just going to represent that by x.

And then usually we have to decide on some kind of feature representation.

We have to convert this into some other form that's easier to deal with than a raw audio wave.

And in a traditional speech system, I often have something called an acoustic model.

And the job of the acoustic model is to learn the relationship between these features that represent my audio and the words that someone is trying to say.

And then I'll often have a language model, which encapsulates all of my knowledge about what kinds of words, what spellings, and what combinations of words are most likely in the language that I'm trying to transcribe.

And once you have all of these pieces, so these different models might be driven by machine learning themselves.

What you would need to build in a traditional system is something called a decoder.

And the job of a decoder, which itself might involve some modeling efforts and machine learning algorithms, is to find the sequence of words W that maximizes this probability.

The probability of the particular sequence W given your audio, that's straightforward, but that's equivalent to maximizing the product of the contributions from your acoustic model and from your language model.

So a traditional speech system is broken down into these pieces, and a lot of the effort in getting that system to work is in developing this sort of portion that combines them all.

So it turns out that if you want to just directly transcribe audio, you can't just go straight to characters.

And the reason is, and it's especially apparent in English, that the way something is spelled in characters doesn't always correspond well to the way that it sounds.

So if I give you the word night, for example, without context, you don't really know whether I'm talking about like a night in armor or whether I'm talking like night like an evening.

And so a way to get around this, to abstract this problem away from a traditional system, is to replace this with a sort of intermediate representation.

Instead of trying to predict characters, I'll just try to predict something called phonemes.

So as an example, if I want to represent the word hello, what I might try to do is break it down into these units of sound.

So the first 1 is like the h, that h sound in hello, and then an ah sound, which is actually only 1 possible pronunciation of an e, and then an l and an o sound.

And that would be my string that I try to come up with using all of my different speech components.

So this, in 1 sense, makes the modeling problem easier.

My acoustic model and so on can be simpler, because I don't have to worry about spelling.

But it does have this problem that I have to think about where these things come from.

So these phonemes are intuitively, they're the perceptually distinct units of sound that we can use to distinguish words.

This might be our imagination that these things actually exist.

There are a bunch of different conventions for how to define these.

And if you end up working on a system that uses phonemes, 1 popular data set is called Timit.

And so this actually has a corpus of audio frames with examples of each of these phonemes.

So once you have this phoneme representation, unfortunately, it adds even more complexity to this traditional pipeline.

Because now, my acoustic model doesn't associate this audio feature with words, it actually associates them with another kind of transcription, with the transcription into phonemes.

And so I have to introduce yet another component into my pipeline that tries to understand how do I convert the transcriptions in phonemes into actual spellings.

So I need some kind of dictionary or a lexicon to tell me all of that.

So this is a way of taking our knowledge about a language and baking it into this engineered pipeline.

And then once you've got all that, again, all of your work now goes into this decoder that has a slightly more complicated task in order to infer the most likely word transcription given the audio.

You'll see a whole bunch of these systems out there.

And we're still using a lot of the vocabulary from these systems.

But traditionally, the big advantage is that it's very tweakable.

If you want to go add a new pronunciation for a word you've never heard before, you can just drop it right in.

But it's also really hard to get working well.

If you start from scratch with this system and you have no experience in speech recognition, it's actually quite confusing and hard to debug.

It's very difficult to know which of these various models is the 1 that's behind your error.

And especially once we start dealing with things like accents, heavy noise, different kinds of ambiguity, that makes the problem even harder to engineer around.

Because trying to think ourselves about, how do I tweak my pronunciation model, for example, to account for someone's accent that I haven't heard, that's a very hard engineering judgment for us to make.

So there are all kinds of design decisions that go into this pipeline, like choosing the feature representation, for example.

So the first place that deep learning has started to make an impact in speech recognition starting a few years ago, is to just take 1 of the core machine learning components of the system and replace it with a deep learning algorithm.

So I mentioned back in this previous pipeline that we had this little model here whose job is to learn the relationship between a sequence of phonemes and the audio that we're hearing.

And There are lots of different methods for training this thing.

So take your favorite machine learning algorithm.

You can probably find someone who has trained an acoustic model with that algorithm, whether it's a Gaussian mixture model or a bunch of decision trees and random forests, anything for estimating these kinds of densities.

There's a lot of work in trying to make better acoustic models.

So some work by George Dahl and co-authors took what was a state-of-the-art deep learning system back in 2011, which is a deep belief network with some pre-training strategies, and dropped it into a state of the art pipeline in place of this acoustic model.

And the results are actually pretty striking, because even though we had neural networks and these pipelines for a while, what ended up happening is that when you replace the Gaussian mixture model in system that already existed, with this deep belief network as an acoustic model, you actually got something between like a 10 and 20% relative improvement in accuracy, which is a huge jump.

And if you compare this to the amount of progress that had been made in preceding years, this is a giant leap for a single paper to make, compared to progress we've been able to make previously.

So this is, in some sense, the first generation of deep learning for speech recognition, which is I take 1 of these components and I swap it out for my favorite deep learning algorithm.

So with these traditional speech recognition pipelines, The problem that we would always run into is that if you gave me a lot more data, you gave me a much bigger computer so that I could train a huge model, that actually didn't help me.

Because all the problems I had were in the construction of this pipeline.

And so eventually, if you gave me more data in a bigger computer, the performance of our speech recognition system would just kind of peter out.

It would just reach a ceiling that was very hard to get over.

And so we just start coming up with lots of different strategies.

We start specializing for each application.

We try to specialize for each user, and try to make things a little bit better around the edges.

And what these deep learning acoustic models did was in some sense moved that barrier a little ways.

It made it possible for us to take a bit more data, much faster computers that let us try a whole lot of models, and move that ceiling up quite a ways.

So the question that many in the research community, including folks at Baidu, have been trying to answer is, can we go to a next generation version of this insight?

Can we, for instance, build a speech engine that is powered by deep learning all the way from the audio input to the transcription itself?

Can we replace as much of that traditional system with deep learning as possible so that over time, as you give researchers more data and bigger computers and the ability to try more models, their speech recognition performance just keeps going up and we can potentially solve speech for everybody.

So the goal of this tutorial is not to get you up here, which requires a whole bunch of things that I'll tell you about near the end.

But what we want to try to do is give you enough to get a point on this curve.

And then once you're on the curve, the idea is that what remains is now a problem of scale.

It's about data and about getting bigger computers and coming up with ways to build bigger models.

So, that's my objective, so that when you walk away from here, you have a picture of what you would need to build to get this point.

And then, after that, it's hopefully all about scale.

So, thanks to Vinay Rao, who's been helping put this tutorial together.

There is going to be some starter code live for the basic pipeline, the deep learning part of the pipeline that we're talking about.

So there are some open source implementations of things like CTC.

But we wanted to make sure that there's a system out there that's pretty representative of the acoustic models that I'm going to be talking about in the first half of the presentation here.

So this will be enough that you can get a simple pipeline going with something called max decoding, which I'll tell you about later.

And the idea is that this is sort of a scale model of the acoustic models that Baidu and other places are powering real production speech engines.

So this will get you that point on the curve.

So here's what we're going to talk about.

The first part, I'm just gonna introduce a few preliminaries, talk about pre-processing.

So we still have a little bit of pre-processing around, but it's not really fundamental.

I think it's probably going to go away in the long run.

We'll talk about what is probably the most mature piece of sequence learning technologies for deep learning right now.

So it turns out that 1 of the fundamental problems of doing speech recognition is how do I build a neural network that can map this audio signal to a transcription that can have a quite variable length.

And so CTC is 1 highly mature method for doing this.

And I think you're actually going to hear about maybe some other solutions later today.

Then I'll say a little bit about training and just what that looks like.

And then finally, say a bit about decoding and language models, which is sort of an addendum to the current acoustic models that we can build that make them perform a lot better.

And then once you have this, that's a picture of what you need to get this point on the curve.

And then I'll talk a little bit about what's remaining.

How do you scale up from this little scale model up to the full thing.

And then time permitting, we'll talk a little bit about production.

How could you put something like this into a cloud server and actually serve real users with it?

This should be pretty straightforward, I think.

Unlike a two-dimensional image where we normally have a 2D grid of pixels, audio is just a 1D signal.

And there are a bunch of different formats for audio, but typically this one-dimensional wave that is actually me saying something like, hello world, is something like 8,000 samples per second, or 16,000 samples per second.

And each wave is quantized into 8 or 16 bits.

So when we represent this audio signal that's going to go into our pipeline, you can just think of that as a one-dimensional vector.

So when I had that box called X that represented my audio signal, you can think of this as being broken down into samples, X1, X2, and so forth.

And if I had a one-second audio clip, this vector would have a length of either, say, 8,000 or 16,000 samples.

And each element would be, say, a floating point number that I'd extracted from this 8 or 16-bit sample.

Now once I have an audio clip, we'll do a little bit of preprocessing.

The first is to just do some vanilla pre-processing, like convert to a simple spectrogram.

So if you look at a traditional speech pipeline, you're going to see things like MFCCs, which are male frequency kepstral coefficients.

You'll see a whole bunch of plays on spectrograms where you take differences in different kinds of features and try to engineer complex representations.

But for the stuff that we're gonna do today, a simple spectrogram is just fine.

And it turns out, as you'll see in a second, we lose a little bit of information when we do this, but it turns out not to be a huge difference.

Now I said a moment ago that I think probably this is going to go away in the long run, and that's because Today you can actually find recent research and trying to do away with even this pre-processing part and having your neural network process the audio wave directly and just train its own feature transformation.

So there's some references at the end that you can look at for this.

How many people have seen a spectrogram or computed a spectrogram before?

So the idea behind a spectrogram is that it's sort of like a frequency domain representation, but instead of representing this entire signal in terms of frequencies, I'm just going to represent a small window in terms of frequencies.

So to process this audio clip, the first thing I'm going to do is cut out a little window that's typically about 20 milliseconds long.

And when you get down to that scale, it's usually very clear that these audio signals are made up of sort of a combination of different frequencies of sine waves.

And then what we do is we compute an FFT.

It basically converts this little signal into the frequency domain.

And then we just take the log of the power at each frequency.

And so if you look at what the result of this is, it basically tells us for every frequency of sine wave, what is the magnitude, what's the amount of power represented by that sine wave that makes up this original signal.

So over here, in this example, we have a very strong low frequency component in the signal.

And then we have differing magnitudes at different differing frequencies.

So we can just think of this as a vector.

So now, instead of representing this little 20 millisecond slice as sort of a sequence of audio samples, instead I'm going to represent it as a vector here where each element represents sort of the strength of each frequency in this little window.

And the next step beyond this is that if I just told you how to process 1 little window, you can of course apply this to a whole bunch of windows across the entire piece of audio.

And that gives you what we call a spectrogram.

And you can use either disjoint windows that are just sort of adjacent, or you can apply them to overlapping windows if you like.

So there's a little bit of parameter tuning there.

But this is an alternative representation of this audio signal that happens to be easier to use for a lot of purposes.

So our goal, starting from this representation, is to build what I'm going to call an acoustic model, but which is really, to the extent we can make it happen, is really going to be an entire speech engine that is represented by a neural network.

So what we would like to do is build a neural net that if we could train it from a whole bunch of pairs, X, which is my original audio that I turned into a spectrogram, and Y star, that's the ground truth transcription that some human has given me.

If I were to train this big neural network off of these pairs, what I'd like it to produce is some kind of output that I'm representing by the character C here, so that I could later extract the correct transcription, which I'm going to denote by y.

So if I said hello, the first thing I'm going to do is run preprocessing to get all these spectrogram frames.

And then I'm going to have a recurrent neural network that consumes each frame and processes them into some new representation called c.

And hopefully, I can engineer my network in such a way that I can just read the transcription off of these output neurons.

So that's kind of the intuitive picture of what we want to accomplish.

So as I mentioned back in the outline, there's 1 obvious fundamental problem here, which is that the length of the input is not the same as the length of the transcription.

So if I say hello very slowly, then I can have a very long audio signal, even though I didn't change the length of the transcription.

Or if I say hello very quickly, then I can have a very short piece of audio.

And so that means that this output of my neural network is changing length, and I need to come up with some way to map that variable length neural network output to this fixed length transcription.

And also do it in a way that we can actually train this pipeline.

So, The traditional way to deal with this problem, if you were building a speech engine several years ago, is to just try to bootstrap the whole system.

So I'd actually train a neural network to correctly predict the sounds at every frame using some kind of data set like Timit, where someone has lovingly annotated all of the phonemes for me.

And then I try to figure out the alignment between my saying hello in a phonetic transcription with the input audio.

And then once I've lined up all of the sounds with the input audio, now I don't care about length anymore, because I can just make a one-to-one mapping between the audio input and the phoneme outputs that I'm trying to target.

But this alignment process is horribly error prone.

You have to do a lot of extra work to make it work well.

We really want to have some kind of solution that lets us solve this straight away.

And As I mentioned, there's some current research on how to use things like attentional models, sequence to sequence models that you'll hear about later, in order to solve this kind of problem.

But as I said, we'll focus on something called connectionist temporal classification, or CTC, that is sort of current state of the art for how to do this.

So our recurrent neural network has these output neurons that I'm calling C.

And the job of these output neurons is to encode a distribution over the output symbols.

So because of the structure of the recurrent network, the length of this symbol sequence C is the same as the length of my audio input.

So if my audio input, say, was 2 seconds long, that might have 100 audio frames, and that would mean that the length of C is also 100 different values.

So if we were working on a phoneme-based model, then C would be some kind of phoneme representation.

We would also include a blank symbol, which is special for CTC.

But if, as we'll do in the rest of this talk, we're trying to just predict the graphemes, trying to predict the characters in this language directly from the audio, then I would just let C take on a value that's in my alphabet, or take on a blank, or a space, if my language has spaces in it.

And then the second thing I'm going to do, once my RNN gives me a distribution over these symbols C, is that I'm going to try to define some kind of mapping that can convert this long transcription C into the final transcription Y.

And now, recognizing that C is itself a probabilistic creature, there's a distribution over choices of C that correspond to the audio.

Once I apply this function, that also means that there's a distribution over Y.

There's a distribution over the possible transcriptions that I could get.

And what I'll want to do to train my network is to maximize the probability of the correct transcription given the audio.

So those are the 3 steps that we have to accomplish in order to make CTC work.

So we have these output neurons C, and they represent a distribution over the different symbols that I could be hearing in the audio.

You can see the spectrogram frames poking up.

And this is being processed by this recurrent neural network.

And the output is a big bank of softmax neurons.

So for the first frame of audio, I have a neuron that corresponds to each of the symbols that C could represent.

And this set of softmax neurons here, with the output summing to 1, represents the probability of, say, C1 having the value ABC, and so on, or this special blank character.

So for example, if I pick 1 of the neurons over here, then the first row, which represents the character B, and the 17th column, which is the 17th frame in time, This represents the probability that C17 represents the character B given the audio.

So once I have this, that also means that I can just define a distribution not just over the individual characters, but if I just assume that all of the characters are independent, which is kind of a naive assumption, but if I bake this into the system, I can define a distribution over all possible sequences of characters in this alphabet.

So if I gave you a specific instance, a specific character string using this alphabet, for instance, I represent the string hello as H-H-H-E blank E blank blank L-L blank L-O, and then a bunch of blanks.

See all Lex Fridman transcripts on Youtube

Deep Learning for Speech Recognition (Adam Coates, Baidu)