This is a clip from a conversation with Francois Chollet from Sep 2019. New full episodes every Mon & Thu and 1-2 new clips or a new non-podcast video on all other days. You can watch the full conversation here: https://www.youtube.com/watch?v=Bo8MY4JpiXE
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on iTunes:
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Note: I select clips with insights from these much longer conversation with the hope of helping make these ideas more accessible and discoverable. Ultimately, this podcast is a small side hobby for me with the goal of sharing and discussing ideas. For now, I post a few clips every Tue & Fri. I did a poll and 92% of people either liked or loved the posting of daily clips, 2% were indifferent, and 6% hated it, some suggesting that I post them on a separate YouTube channel. I hear the 6% and partially agree, so am torn about the whole thing. I tried creating a separate clips channel but the YouTube algorithm makes it very difficult for that channel to grow unless the main channel is already very popular. So for a little while, I'll keep posting clips on the main channel. I ask for your patience and to see these clips as supporting the dissemination of knowledge contained in nuanced discussion. If you enjoy it, consider subscribing, sharing, and commenting.

François Chollet is the creator of Keras, which is an open source deep learning library that is designed to enable fast, user-friendly experimentation with deep neural networks. It serves as an interface to several deep learning libraries, most popular of which is TensorFlow, and it was integrated into TensorFlow main codebase a while back. Aside from creating an exceptionally useful and popular library, François is also a world-class AI researcher and software engineer at Google, and is definitely an outspoken, if not controversial, personality in the AI world, especially in the realm of ideas around the future of artificial intelligence.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

Let's go from the philosophical to the practical.

Can you give me a history of Keras and all the major deep learning frameworks that you kind of remember in relation to Keras and in general, TensorFlow, Theano, the old days.

Can you give a brief overview, Wikipedia style history and your role in it before we return to AGI discussions?

I actually picked the name like just the day I was going to release it.

And so at the time, there weren't too many people working on deep learning, maybe like fewer than 10,000.

The software tooling was not really developed.

So the main deep learning library was Caffe, which was mostly C++.

Caffe was vastly more popular than Theano in late 2014, early 2015.

Caffe was the 1 library that everyone was using for computer vision.

And computer vision was the most popular problem.

Like Covenants was like the subfield of deep learning that everyone was working on.

I was actually interested in RNNs, in Recurrent Neural Networks, which was a very niche topic at the time, right?

I had used Torch7, I had used Tiano, used Tiano a lot in Kaggle competitions.

And there was no good solution for RNNs at the time.

There was no reusable open source implementation of an LSTM, for instance.

And at first, the pitch for that was, it was going to be mostly around LSTM, Merkle neural networks.

An important decision at the time that was kind of not obvious is that the models would be defined via Python code, which was kind of like going against the mainstream at the time because Caffe, Pylon 2 and so on, like all the big libraries were actually going with the approach of having static configuration files in YAML to define models.

So some libraries were using code to define models like Torch 7, obviously, but that was not Python.

Lasagne was like a Tiano-based, very early library that was, I think, developed, I'm not sure exactly, probably late

And so I started working on something and the value proposition at the time was that not only that the, what I think was the first reusable open source implementation of LSTM, you could combine RNNs and Covenants with the same library, which is not really possible before.

And it was kind of easy to use because, so before I was using Therian, I was actually using scikit-learn and I loved scikit-learn for its usability.

So I drew a lot of inspiration from scikit-learn when I met Keras.

It's almost like scikit-learn for neural networks.

Exactly, the fit function, like reducing a complex string loop to a single function call.

And of course, some people will say, this is hiding a lot of details, but that's exactly the point.

It's magical in the sense that it's delightful, right?

I didn't know that it was born out of desire to implement RNNs and LSTMs.

So you were actually 1 of the first people to really try to attempt to get the major architecture together.

And it's also interesting, you made me realize that that was a design decision at all, is defining the model and code.

Just, I'm putting myself in your shoes, whether the YAML, especially if Cafe was the most popular.

I didn't like the YAML thing, but it makes more sense that you will put in a configuration file the definition of a model.

That's an interesting gutsy move to stick with defining it in code?

Other libraries were doing it as well, but it was definitely the more niche option.

So I released Keras in March, 2015, and it got users pretty much from the start.

So the deep learning community was very, very small at the time.

Lots of people were starting to be interested in LSTM.

So it was going to release it at the right time because it was offering an easy to use LSTM implementation.

Exactly at the time where lots of people started to be intrigued by the capabilities of RNN, RNN, so NLP.

Then I joined Google about 6 months later, And that was actually completely unrelated to Keras.

I actually joined a research team working on image classification, mostly like computer vision.

So I was doing computer vision research at Google initially.

And immediately when I joined Google, I was exposed to the early internal version of TensorFlow.

And the way it appeared to me at the time, and that was definitely the way it was at the time, is that this was an improved version of Theano.

So I immediately knew I had to port Keras to this new TensorFlow thing.

And I was actually very busy as a new Googler.

But then in November, I think it was November

And it was kind of like my wake up call that, hey, I had to actually go and make it happen.

So in December, I ported Keras to run on top of TensorFlow.

It was more like a refactoring where I was abstracting away all the backend functionality into 1 module so that the same code base could run on top of multiple backends.

And for the next year, Theano stayed as the default option.

It was easier to use, somewhat less buggy.

It was much faster, especially when it came to Ornans.

But eventually, you know, TensorFlow overtook it, right?

And TensorFlow, the early TensorFlow, has similar architectural decisions as Theano, right?

So what, I mean, that's still Keras as a side, almost fun project, right?

So, And even though it grew to have, you know, a lot of users for deep learning library at the time, like throughout 2016, but I wasn't doing it as my main job.

So things started changing in, I think it's maybe October

So Rajat, who was the lead on TensorFlow, basically showed up 1 day in our building.

I was doing research and things like, So I did a lot of computer vision research, also collaborations with Christian Zugedi and deep learning for theorem proving.

It was a really interesting research topic.

And so Rajat was saying, hey, we saw Keras, we like it, we saw that you're at Google, why don't you come over for like a quarter and work with us?

And I was like, yeah, that sounds like a great opportunity, let's do it.

And so I started working on integrating the Keras API into TensorFlow more tightly.

So what followed up is a sort of like temporary TensorFlow-only version of Keras that was in TensorFlow.com Trib for a while, and finally moved to TensorFlow core.

And, you know, I've never actually gotten back to my old team doing research.

Well, it's kind of funny that somebody like you who dreams of, or at least sees the power of AI systems that reason and theorem proving, we'll talk about, has also created a system that makes the most basic kind of Lego building that is deep learning, super accessible, super easy, so beautifully so.

It's a funny irony that you're both, you're responsible for both things.

But so TensorFlow 2.0 is kind of, there's a sprint.

I don't know how long it'll take, but there's a sprint towards the finish.

What do you look, what are you working on these days?

There's so many things that just make it a lot easier to work.

So I've spent the past year and a half working on TensorFlow 2.

It's a delightful product compared to TensorFlow 1.

So on the Keras side, what I'm really excited about is that, So, you know, previously Keras has been this very easy to use high-level interface to do deep learning.

But if you wanted to, you know, if you wanted a lot of flexibility, the Keras framework, you know, was probably not the optimal way to do things compared to just writing everything from scratch.

So in some way, the framework was getting in the way.

And in TensorFlow 2, you don't have this at all, actually.

You have the usability of the high level interface, but you have the flexibility of this lower level interface.

And you have this spectrum of workflows where you can get more or less usability and flexibility trade-offs, depending on your needs, right?

You can write everything from scratch and you get a lot of help doing so, by, you know, subclassing models and writing some train loops using ego execution.

It's very flexible, it's very easy to debug, it's very powerful.

But all of this integrates seamlessly with higher level features up to, you know, the classic Keras workflows, which are very scikit-learn like, and, you know, are ideal for a data scientist, machine learning engineer type of profile.

So now you can have the same framework offering the same set of APIs that enable a spectrum of workflows that are more or less low level, more or less high level that are suitable for profiles ranging from researchers to data scientists and everything in between.


Speaker 0: Let's go from the philosophical to the practical. Can you give me a history of Keras and all the major deep learning frameworks that you kind of remember in relation to Keras and in general, TensorFlow, Theano, the old days. Can you give a brief overview, Wikipedia style history and your role in it before we return to AGI discussions?

Speaker 1: Yeah, that's a broad topic. So I started working on Keras. It was a name Keras at the time. I actually picked the name like just the day I was going to release it. So I started working on it in February

Speaker 0: 2015.

Speaker 1: And so at the time, there weren't too many people working on deep learning, maybe like fewer than 10,000. The software tooling was not really developed. So the main deep learning library was Caffe, which was mostly C++.

Speaker 0: Why do you say Caffe was the main 1?

Speaker 1: Caffe was vastly more popular than Theano in late 2014, early 2015. Caffe was the 1 library that everyone was using for computer vision.

Speaker 0: And computer vision was the most popular problem. Absolutely.

Speaker 1: Like Covenants was like the subfield of deep learning that everyone was working on. So myself, so in late

Speaker 0: 2014,

Speaker 1: I was actually interested in RNNs, in Recurrent Neural Networks, which was a very niche topic at the time, right? It really took off around 2016. And so I was looking for good tools. I had used Torch7, I had used Tiano, used Tiano a lot in Kaggle competitions. I had used Caffe.

And there was no good solution for RNNs at the time. There was no reusable open source implementation of an LSTM, for instance. So I decided to build my own. And at first, the pitch for that was, it was going to be mostly around LSTM, Merkle neural networks. It was going to be in Python.

An important decision at the time that was kind of not obvious is that the models would be defined via Python code, which was kind of like going against the mainstream at the time because Caffe, Pylon 2 and so on, like all the big libraries were actually going with the approach of having static configuration files in YAML to define models. So some libraries were using code to define models like Torch 7, obviously, but that was not Python. Lasagne was like a Tiano-based, very early library that was, I think, developed, I'm not sure exactly, probably late

Speaker 0: 2014.

Speaker 1: It's Python as well. It's Python as well. It was like on top of Tano. And so I started working on something and the value proposition at the time was that not only that the, what I think was the first reusable open source implementation of LSTM, you could combine RNNs and Covenants with the same library, which is not really possible before. Like, Caffe was only doing Covenants.

And it was kind of easy to use because, so before I was using Therian, I was actually using scikit-learn and I loved scikit-learn for its usability. So I drew a lot of inspiration from scikit-learn when I met Keras. It's almost like scikit-learn for neural networks.

Speaker 0: Yeah. The fit function.

Speaker 1: Exactly, the fit function, like reducing a complex string loop to a single function call. And of course, some people will say, this is hiding a lot of details, but that's exactly the point. The magic is the point. So it's magical, but in a good way. It's magical in the sense that it's delightful, right?

Speaker 0: Yeah, I'm actually quite surprised. I didn't know that it was born out of desire to implement RNNs and LSTMs.

Speaker 1: It was.

Speaker 0: That's fascinating. So you were actually 1 of the first people to really try to attempt to get the major architecture together. And it's also interesting, you made me realize that that was a design decision at all, is defining the model and code. Just, I'm putting myself in your shoes, whether the YAML, especially if Cafe was the most popular.

Speaker 1: It was the most popular by far. If I was...

Speaker 0: At the time. If I were... Yeah, I don't... I didn't like the YAML thing, but it makes more sense that you will put in a configuration file the definition of a model. That's an interesting gutsy move to stick with defining it in code?

Just if you look back.

Speaker 1: Other libraries were doing it as well, but it was definitely the more niche option. Yeah. Okay, Keras and then- Keras. So I released Keras in March, 2015, and it got users pretty much from the start. So the deep learning community was very, very small at the time.

Lots of people were starting to be interested in LSTM. So it was going to release it at the right time because it was offering an easy to use LSTM implementation. Exactly at the time where lots of people started to be intrigued by the capabilities of RNN, RNN, so NLP. So it grew from there. Then I joined Google about 6 months later, And that was actually completely unrelated to Keras.

I actually joined a research team working on image classification, mostly like computer vision. So I was doing computer vision research at Google initially. And immediately when I joined Google, I was exposed to the early internal version of TensorFlow. And the way it appeared to me at the time, and that was definitely the way it was at the time, is that this was an improved version of Theano. So I immediately knew I had to port Keras to this new TensorFlow thing.

And I was actually very busy as a new Googler. So I had not time to work on that. But then in November, I think it was November

Speaker 0: 2015,

Speaker 1: TensorFlow got released. And it was kind of like my wake up call that, hey, I had to actually go and make it happen. So in December, I ported Keras to run on top of TensorFlow. But it was not exactly a port. It was more like a refactoring where I was abstracting away all the backend functionality into 1 module so that the same code base could run on top of multiple backends.

So on top of TensorFlow or Theano. And for the next year, Theano stayed as the default option. It was easier to use, somewhat less buggy. It was much faster, especially when it came to Ornans. But eventually, you know, TensorFlow overtook it, right?

Speaker 0: And TensorFlow, the early TensorFlow, has similar architectural decisions as Theano, right? So it was a natural transition.

Speaker 1: Yeah, absolutely. So what, I mean, that's still Keras as a side, almost fun project, right? Yeah. So it was not my job assignment. It was not.

I was doing it on the side. So, And even though it grew to have, you know, a lot of users for deep learning library at the time, like throughout 2016, but I wasn't doing it as my main job. So things started changing in, I think it's maybe October

Speaker 0: 2016.

Speaker 1: So 1 year later. So Rajat, who was the lead on TensorFlow, basically showed up 1 day in our building. I was doing research and things like, So I did a lot of computer vision research, also collaborations with Christian Zugedi and deep learning for theorem proving. It was a really interesting research topic. And so Rajat was saying, hey, we saw Keras, we like it, we saw that you're at Google, why don't you come over for like a quarter and work with us?

And I was like, yeah, that sounds like a great opportunity, let's do it. And so I started working on integrating the Keras API into TensorFlow more tightly. So what followed up is a sort of like temporary TensorFlow-only version of Keras that was in TensorFlow.com Trib for a while, and finally moved to TensorFlow core. And, you know, I've never actually gotten back to my old team doing research.

Speaker 0: Well, it's kind of funny that somebody like you who dreams of, or at least sees the power of AI systems that reason and theorem proving, we'll talk about, has also created a system that makes the most basic kind of Lego building that is deep learning, super accessible, super easy, so beautifully so. It's a funny irony that you're both, you're responsible for both things. But so TensorFlow 2.0 is kind of, there's a sprint. I don't know how long it'll take, but there's a sprint towards the finish. What do you look, what are you working on these days?

What are you excited about? What are you excited about in 2.0? I mean, eager execution. There's so many things that just make it a lot easier to work. What are you excited about?

And what's also really hard? What are the problems you have to solve?

Speaker 1: So I've spent the past year and a half working on TensorFlow 2. It's been a long journey. I'm actually extremely excited about it. I think it's a great product. It's a delightful product compared to TensorFlow 1.

We've made huge progress. So on the Keras side, what I'm really excited about is that, So, you know, previously Keras has been this very easy to use high-level interface to do deep learning. But if you wanted to, you know, if you wanted a lot of flexibility, the Keras framework, you know, was probably not the optimal way to do things compared to just writing everything from scratch. So in some way, the framework was getting in the way. And in TensorFlow 2, you don't have this at all, actually.

You have the usability of the high level interface, but you have the flexibility of this lower level interface. And you have this spectrum of workflows where you can get more or less usability and flexibility trade-offs, depending on your needs, right? You can write everything from scratch and you get a lot of help doing so, by, you know, subclassing models and writing some train loops using ego execution. It's very flexible, it's very easy to debug, it's very powerful. But all of this integrates seamlessly with higher level features up to, you know, the classic Keras workflows, which are very scikit-learn like, and, you know, are ideal for a data scientist, machine learning engineer type of profile.

So now you can have the same framework offering the same set of APIs that enable a spectrum of workflows that are more or less low level, more or less high level that are suitable for profiles ranging from researchers to data scientists and everything in between.

See all Lex Fridman transcripts on Youtube

François Chollet: History of Keras and TensorFlow | AI Podcast Clips