- Adam Coates runs the AI lab at Baidu in Silicon Valley, which is the largest search engine in China.
- Baidu has remade itself as an AI company and recognizes the value of AI for various applications.
- The AI Lab focuses on understanding current research and translating it into business and product impact.
- The lab aims to create AI technologies that can have a significant impact on at least 100 million people.
- They work on both basic research and implementing AI into products.
- An example is their speech recognition project called DeepSpeech, which started as a basic research project and now has superhuman performance in Mandarin.
- The challenge is the need for large amounts of data to train these systems, but they are working on developing systems that require less data.
- They are also working on text-to-speech projects, exploring ways to share data across applications and move from supervised to unsupervised learning.
- The process of speech recognition involves collecting audio clips, obtaining ground truth transcription, and training a deep learning algorithm to predict the correct text transcript.
- Traditionally, speech recognition required breaking down the problem into different stages and engineering each stage with expert knowledge.

- Deep learning systems like DeepSpeech can solve speech recognition in one step, eliminating the need for hand-engineered representations.
- Data from sources like crowdsourcing services and audio books can be used to train speech systems, even with unusual words and variations in pronunciation.
- Deep learning techniques can also be applied to other domains, such as text-to-speech, to build more efficient and effective systems.
- Baidu's AI lab is focused on developing immersive AI products, like voice-first keyboards, that can have a significant impact on user habits and experiences.
- The goal is to make speech and text-to-speech interfaces work for a wide range of users and improve latency for a more seamless user experience.
- Baidu is working on making these technologies run locally and addressing challenges like latency to meet user expectations.

- The speaker discusses the challenge of speech recognition systems needing to process an entire audio clip before giving a response, which is not ideal for real-time interactions.
- The speaker mentions the need to alter neural networks to provide good answers based on limited context and quickly update as more context is provided.
- The speaker highlights the difficulty current speech systems have with handling cross-talk, background noise, and long-form transcriptions.
- The speaker mentions the importance of critical thinking and adapting to the implications of AI technology.
- The speaker believes that AI will address various challenges and have positive impacts, particularly for individuals with disabilities.
- The speaker discusses the job turnover rate and the need for continual learning in the field of AI.
- The speaker describes the characteristics of a "full stack machine learning engineer" and the importance of being self-directed and adaptable.
- The speaker recommends exploring startup books for influential insights and emphasizes the importance of learning and connecting with real-world pain points in AI research.

- It is important to be clear-eyed and focus on learning in order to find the most important parts of AI research and address the most important pain points in the real world.
- This mindset is influenced by the startup world.

- Baidu's AI Lab, led by Adam Coates, focuses on translating current research into impactful AI technologies.
- They work on both basic research and implementing AI into products, such as their speech recognition project DeepSpeech.
- The lab aims to create AI technologies that can impact at least 100 million people.
- Baidu's AI Lab is also working on text-to-speech projects and exploring ways to improve data sharing and move towards unsupervised learning.
- They are focused on developing immersive AI products, like voice-first keyboards, to improve user experiences.
- Baidu is addressing challenges like latency and real-time interactions in speech recognition systems.
- The speaker emphasizes the importance of critical thinking, adaptability, and continual learning in the field of AI.
- Learning from real-world pain points and understanding the implications of AI technology is crucial.
- The startup world has influenced the mindset of focusing on learning and addressing important pain points in AI research.

Adam Coates is the Director of Baidu's Silicon Valley AI Lab - https://twitter.com/adampaulcoates

http://research.baidu.com

Read the transcript: http://blog.ycombinator.com/

YC's podcast is hosted by Craig Cannon - http://twitter.com/craigcannon

Today we have Adam Coates here for an interview.

Adam, you run the AI lab at Baidu in Silicon Valley.

Could you just give us a quick intro and explain what Baidu is for people who don't know?

Yeah, so Baidu is actually the largest search engine in China.

So it turns out the internet ecosystem in China is this incredibly dynamic environment.

And so Baidu, I think, sort of turned out to be an early technology leader and really established itself in PC search, but then also has sort of remade itself in the mobile revolution and increasingly today is becoming an AI company, recognizing the value of AI for a whole bunch of different applications, not just search.

Okay, and so yeah, what do you do exactly?

So I'm the director of the Silicon Valley AI Lab, which is 1 of 4 labs within Baidu Research.

So especially as Baidu's becoming an AI company, the need for a team to sort of be on the bleeding edge and understand all of the current research, be able to do a lot of basic research ourselves, but also figure out how we can translate that into business and product impact for the company.

And so that's what Baidu Research is here for.

And the AI Lab in particular, we kind of founded recognizing how extreme this problem was about to get.

So I think the deep learning research and AI research right now is flying forward so rapidly that the need for teams to be able to both understand that research, but also quickly translate it into something that businesses and products can use is more critical than ever.

So we founded the AI Lab to try to close that gap and help the company move faster.

And so then how do you break up your time in between like doing basic research for around AI and actually implementing it like bringing it forward to a product?

I think 1 of the things that we try to repeat to ourselves every day is that we're mission oriented.

So the mission of the AI lab is precisely to create AI technologies that can have a significant impact on at least 100 million people.

We chose this to sort of keep bringing ourselves back to the sort of final goal that we want all the research we do to ultimately ends up in the hands of users.

And so sometimes that means that we spot something that needs to happen in the world to really change technology for the better and to help Baidu, but no 1 knows how to solve it.

And there's a basic research problem there that someone has to tackle.

And so we'll sort of go back to our visionary stance and think about the longterm and invest in research.

And then as we have success there, we shift back to the other foot and take responsibility for carrying all of that to a real application and making sure we don't just solve the 90% that you might put in, say, your research paper, but we also solve the last mile.

So maybe the best way to do this, then, is to just explain something that started with research here and how that's been brought on to a full-on product that exists.

We've spent a ton of time on speech recognition.

So speech recognition, a few years ago, was 1 of these technologies that always felt pretty good, but not good enough.

And so traditionally, speech recognition systems have been heavily optimized for things like mobile search.

So if you hold your phone up close to your mouth and you say a short query...

The systems could figure it out and they're getting quite good.

I think the speech engine that we've built at Baidu called DeepSpeech is actually super human for these short queries.

Because you have no context, people can have thick accents.

So that speech engine actually started out as a basic research project.

We looked at this problem, we said, gosh, what would happen if speech recognition were human level for every product you ever used?

So whether you're in your home or in your car or you pick up your phone, whether you hold your phone up close or hold it away.

If I'm in the kitchen and my toddler is yelling at me, can I still use a speech interface?

Could it work as well as a human being understands us?

What is the basic research that moved it forward to put it

So we had the hypothesis that maybe the thing holding back a lot of the progress in speech is actually just scale.

Maybe if we took some of the same basic ideas we could see in the research literature already and scaled them way up, put in a lot more data, invested a lot of time in solving computational problems, and built a much larger neural network than anyone had been building before for this problem.

And lo and behold, with a lot of effort, we ended up with this pretty amazing speech recognition model that, like I said, in Mandarin at least, is actually superhuman.

You can actually sit there and listen to a voice query that someone is trying out and you'll have native speakers sitting around debating with each other, wondering what the heck the person is saying.

And then the speech engine will give an answer and everybody goes, oh, that's what it was because it's just such a thick accent from perhaps someone in rural China.

How much data do you have to give it to train it?

You know, to train it on a new, because I think on the site I saw it was English and Mandarin.

Like if I wanted German, how much would I have to give it?

So 1 of the big challenges for these things is that they need a ton of data.

So our English system uses like 10 to 20,000 hours of audio.

The Mandarin systems are using even more for top end products.

So this certainly means that the technology's at a state where to get that superhuman performance, you've got to really care about it.

So for Baidu voice search maps, things like that, that are flagship products, we can put in the capital and the effort to do that.

But it's also 1 of the exciting things going forward in the basic research that we think about is how do we get around that?

How can we develop machine learning systems that get you human performance on every product and do it with a lot less data?

So what I was wondering then, did you see that Liar Bird thing that was floating around the internet this week.

They claim that they don't need all that much time, all that much data, audio data, to emulate your voice or simulate, whatever they call it.

You guys have a similar project going on, right?

Why can they achieve that with less data?

I think the technical challenge behind all of this is there's sort of 2 things that we can do.

1 is to try to share data across many applications.

So to take text to speech as 1 example, If I learn to mimic lots of different voices, and then you give me the 1,001st voice, you'd hope that the first 1,000 taught you virtually everything you need to know about language, and that what's left is really some idiosyncratic change that you could learn from very little data.

The other side of it is that a lot of these systems, this is much more important for things like speech recognition that we were talking about, is We want to move from using supervised learning, where a human being has to give you the correct answer in order for you to train your neural network, but move to unsupervised learning, where I could just give you a lot of raw audio and have you learn the mechanics of speech before I ask you to learn a new language.

And hopefully that can also bring down the amount of data that we need.

And so then on the technical side, like could you give us just a somewhat of an overview of how that actually works?

Let's do both actually, cause I'm super interested.

Yeah, let's start with speech recognition.

Before we go and train a speech system, what we have to do is collect a whole bunch of audio clips.

So for example, if we wanted to build a new voice search engine, I would need to get lots of examples of people speaking to me, giving me little voice queries.

And then I actually need human annotators, or I need some kind of system that can give me ground truth, that can tell me for a given audio clip what was the correct transcription.

And so once you've done that, you can ask a deep learning algorithm to learn the function that predicts the correct text transcript from the audio clip.

We're really good with this for lots of different applications.

But the big challenge there is those labels.

That someone has to be able to sit there and give you say 10,000 hours worth of labels, which can be really expensive.

So, for- Yeah, how is it actually recognize, like what is the software doing to recognize the intonation of a word?

Well, traditionally what you would have to do is break these problems down into lots of different stages.

So I, as a speech recognition expert, would sit down and I would think a lot about what are the mechanics of this language?

So for Chinese, you would have to think about tonality and how to break up all the different sounds into some intermediate representation.

And then you would need some sophisticated piece of software we called a decoder that goes through and tries to map that sequence of sounds to possible words that it might represent.

And so you have all these different pieces and you'd have to engineer each 1 often with its own expert knowledge.

But deep speech and all of the new deep learning systems we're seeing now try to solve this in 1 fell swoop.

So the, really the answer to your question is kind of the vacuous 1, which is that once you give me the audio clips and the characters that it needs to output, a deep learning algorithm can actually just learn to predict those characters directly.

And in the past, it always looked like there was some fundamental problem that maybe we could never escape this need for these hand-engineered representations.

But it turns out that once you have enough data, all of those things go away.

We actually do a lot of clever tricks in English where we don't have a lot of, a large number of English language products.

So for example, it turns out that if you go onto, say, a crowdsourcing service, you can hire people very cheaply to just read books to you.

And it's not the same as the kinds of audio that we hear in real applications.

But it's enough to teach a speech system all about, you know, liaisons between words and you get some speaker variation and you hear strange vocabulary where English spelling is totally ridiculous.

And in the past, you would hand engineer these things.

You'd say, well, I've never heard that word before, so I'm going to bake the pronunciation into my speech engine.

So if I hear enough of these unusual words, you see these neural networks actually learn to spell on their own, even considering all the weird exceptions of English.

Because I've heard of people doing it with like a YouTube video, but then you need a caption as well with the audio, so it's twice as much, if not more, work.

And so then what about the other way around?

How does that work on the technical side?

Right, so that's 1 of the really kind of cool parts of deep learning right now is that a lot of these insights about what works in 1 domain keep transferring to other domains.

So with text-to-speech, you could see a lot of the same practices.

So you would see that a lot of systems were hand-engineered combinations of many different modules.

And each module would have its own set of machine learning algorithms with its own little tricks.

And so 1 of the things that our team did recently with a piece of work that we're calling Deep Voice was to just ask, what if I rewrote all of those modules using deep learning for every single 1?

To not put them all together just yet, but even just ask, can deep learning solve all of these adequately to get a good speech system?

That you can basically abandon most of this specialized knowledge in order to build all of the subsequent modules.

And in more recent research that's in the deep learning community, we're seeing that, of course, everyone is now figuring out how to make these things work end to end.

And that's the same story we saw for DeepSpeech.

And so do you have a team just dedicated to parsing like research coming out of universities and then figuring out how to apply it?

Or are you testing everything that comes out?

It's definitely our role to not only think about AI research, but to think about AI products and how to get these things to impact.

I think there is clearly so much AI research happening that it's impossible to look through everything.

But 1 of the big challenges right now is to not just digest everything, but to identify the things that are truly important.

So what's like a 90 million person product?

the speech recognition we chose because we felt in aggregate it had that potential.

So as we have the next wave of AI products, I think we're gonna move from these sort of bolted on AI features to really immersive AI products.

So if you look at how keyboards were designed a few years ago for your phone, you see that everybody just bolted on a microphone and they hooked it up to their speech API.

And then that was fine for that level of technology.

But as the technology's getting better and better, we can now start putting speech up front.

We can actually build a voice-first keyboard.

So it's actually something we've been prototyping in the AI lab.

You can actually download this for your Android phone.

So it's called TuckType in case anybody wants to try it.

But it's remarkable how much it changes your habits.

I use it all the time and I never thought I would do that.

And So it emphasized to me why the AI lab is here, that we can discover these changes in user habits.

We can understand how speech recognition can impact people much more deeply than it could when it was just bolted onto our product.

And that sort of spurs us on to start looking at the full range of speech problems that we have to solve to get you away from this sort of close talking voice search scenario and to 1 where I can just talk to my phone or talk to a device and have it always work.

So as you've like, you know, given this to a bunch of users, I assume and gotten their feedback, have you been surprised with the like voice as interface?

Some people say like, ah, it doesn't really make sense.

You know, for example, you see like Apple transcribing voicemails now.

Are there certain use cases where you've been surprised at how effective it is and others where you're like, I don't know if this will ever play out?

You know, I think, you know, the really obvious ones like texting seem to be the most popular.

I feel like the feedback that is maybe the most fun for me is when people with thick accents post a review, they say, oh, I have this crazy accent I grew up with, and nothing works for me.

But I tried this new keyboard, and it works amazingly well.

I have a friend who has a thick Italian accent and he complains all the time that nothing works.

And all of this stuff now works for different accents because it's all data driven.

We don't have to think about how we're going to serve all these different users.

If they're represented in the data sets and we get some transcriptions, we can actually serve them in a way that really wasn't possible when we were trying to do it all by hand.

And have you gone through the whole system?

In other words, if I want to give myself an Italian-American accent, can I do that yet with Baidu?

We can't do that yet with our TTS engine, but it's definitely on the way.

I think these are part of a big effort to make this next generation of AI products really fly.

Once text-to-speech and speech are your primary interface to a new device, they have to be amazingly good and they have to work for everybody.

And so I think there's actually still quite a bit of room to run on those topics, not just making it work for a narrow domain, but making it work for really the full breadth of what humans can do.

Do you see a world where you can run this stuff locally or will they always be calling an API?

1 kind of funny thing is that if you look at folks who maybe have a lot less technical knowledge and don't really have the sort of instinct to think through how a piece of technology is working on the backend.

I think the response to a lot of AI technologies now because they're reaching this sort of uncanny valley is that we often respond to them as though they're sort of human.

Our expectations for how delightful a product should be is now being set by our interactions with people.

And 1 of the things we discovered as we were translating deep speech into a production system was that latency is a huge part of that experience.

That the difference between 50 or 100 milliseconds of latency and 200 milliseconds of latency is actually quite perceptible.

And it really, anything we can do to bring that down actually affects user experience quite a bit.

We actually did a combination of research, production hacking, working with product teams, thinking through how to make all of that work.

And that's a big part of the sort of translation process that we're here for.

And so, yeah, what happens on the technical side to make it run faster?

So when we first started the basic research for DeepSpeech, like all research papers, we choose the model that gets the best benchmark score, which turns out to be horribly impractical for putting online.

And so after sort of the initial research results, team sat down with just a set of what you might think of as product requirements and started thinking through what kinds of neural network models will allow us to get the same performance, but don't require so much sort of future context.

They don't have to listen to the entire audio clip before they can give you a really high accuracy response.

So kind of doing that, like, you know, the language prediction stuff, like the open AI guys were doing with the Amazon reviews, like predicting what's coming next?

Maybe not even predicting what's coming next.

But 1 thing that humans do without thinking about it is if If I misunderstand a word that you've said to me and then a couple of words later I pick up context that disambiguates it.

I actually don't skip a beat I I just understand that as 1 long stream And so 1 of the ways that our speech systems would do this is that they would listen to the entire audio clip first, process it all in 1 fell swoop, and then give you a final answer.

And that works great for getting the highest accuracy, but it doesn't work so great for a product where you need to give a response online, give people some feedback that lets them know that you're listening.

And so you need to alter the neural network so that it tries to give you a really good answer using only what it's heard so far, but can then update it very quickly as it gets more context.

So I've noticed over the past few years, people have gotten quite good at structuring sentences so Siri understands them.

You know, they put the noun in the correct position so it feeds back the data correctly.

I found this when I was traveling, I was using Google Translate, and I, after like 1 day, recognized that I couldn't give it a sentence.

But if I gave it a noun, I could just show it to someone.

And like, if I just show like, you know, bread, it will translate it perfectly and give it.

Do you find that like, we're gonna have to slightly adapt how we communicate with machines, or your goal is to communicate like perfectly as we would?

And I don't see a serious barrier to getting there, at least for really high valued applications.

I think there's a lot more research to do, but I sincerely think there's a chance that over the next few years, we're going to regard speech recognition as a solved problem.

So what are the really hard things happening right now?

So I think we were talking earlier about getting all this data.

So for problems where we can just get gobs of labeled data, I think we've got a little bit more room to run there.

But we can certainly solve those kinds of applications.

But there's a huge range of what humans are able to do, often without thinking that current speech engines just don't handle.

We can deal with cross-talk and a lot of background noise.

If you talk to me from the other side of a room, even if there's a lot of reverberation and things going on, it usually doesn't bother anybody that much.

And yet current speech systems often have a really hard time with this.

But for the next generation of AI products, they're going to need to handle all of this.

And so a lot of the research that we're doing now is focused on trying to go after all of those other things.

How do I handle people who are talking over each other or handle multiple speakers who are having a conversation very casually?

How do I transcribe things that have very long structure to them, like a lecture, where over the course of the lecture, I might realize I misunderstood something or some piece of jargon gets spelled out for me and now I need to go and transcribe it.

So this is 1 place where our ability to innovate on products is actually really useful.

We've just launched recently a product vision called SwiftScribe to help transcriptionists be much more efficient.

And that's targeted at understanding all of these scenarios where the world wants this long-form transcription.

We have all of these conversations that we're having that are just sort of lost and we wish we had written down, but it's just too expensive to transcribe all of it for everyday applications.

So do you, so in terms of emulating someone's voice, do you have any concerns for faking it?

Cause I, did you see the face simulation?

I forget the researcher's name, so I'll have to link to it, but you know what I'm talking about.

So essentially you can like feed it both video and audio and you can recreate, you know, Adam talking.

Do you have any thoughts on like how we can prepare for that world?

No, I think in some sense this is a social question, right?

I think culturally we're all going to have to exercise a lot of critical thinking.

We've always had this problem in some sense that I can read an article that has someone's name on it, and notwithstanding understanding writing style, I don't know for sure where that article came from.

And so I think we have habits for how to deal with that scenario.

And I think we're going to have to come up with ways to adapt that to this sort of brave new world.

I think those are big challenges coming up.

But I also think a lot about just all the positives that AI is going to have.

My mother actually has muscular dystrophy.

And so things like speech and language interfaces are just incredibly valuable for someone who cannot type on an iPad because the keys are too far apart.

And so these are just all these things that you don't really think about that these technologies are going to address over the next few years.

And on balance, I know that we're gonna have a lot of big challenges of how do we use these, how do we as users adapt to all of the implications.

But I think we've done really well with this in the past and we're gonna keep doing well with it in the future.

See all Y Combinator transcripts on Youtube

Baidu's AI Lab Director on Advancing Speech Recognition and Simulation