- The speaker is Wojciech Zaremba, a founder at OpenAI, who is working on robotics and AI.
- OpenAI focuses on building AI for the good of humanity and has received a $1 billion investment.
- One of the main projects at OpenAI is robotics, specifically focusing on manipulation, which is the most difficult aspect of robotics.
- Another project involves playing complicated computer games, while a third project involves playing a large number of computer games to understand human learning capabilities.
- The speaker discusses the challenges of using reinforcement learning in AI systems and the difficulty in determining rewards in the real world.
- The speaker believes that working on robotics is closer to real-world applications and highlights the challenges of building a reward system for tasks like cooking scrambled eggs.

- The speaker discusses the challenges of quantifying and resetting a system in robotics.
- Artificial intelligence (AI) is a broad domain, with machine learning being a sub-part of AI.
- Machine learning involves using data to generate a program with specific behavior.
- Deep learning is a paradigm within machine learning that involves multiple steps of computation.
- Deep learning has become popular because it recognizes that true intelligence requires multiple steps of computation.
- Deep learning often involves neural networks, which consist of layers represented by matrices and involve nonlinear operations.
- Nonlinearities, such as the sigmoid function, are used in neural networks to introduce complexity and improve performance.

- The sigmoid function and rectify linear unit (ReLU) are commonly used activation functions in deep learning.
- Deep learning has been successful in solving problems in supervised learning and often outperforms other methods.
- Stochastic gradient descent is a commonly used learning algorithm for neural networks.
- The increase in computational power and advancements in training neural networks have contributed to the recent interest in deep learning.
- Convolutional neural networks (CNNs) are a critical advancement in deep learning, particularly for image processing.
- CNNs apply the same computation to different parts of an image, allowing for faster and more efficient processing.
- Proper weight initialization is important for training deep neural networks.
- Stochastic gradient descent is a simple and effective algorithm for training neural networks.

- The video discusses the importance of adjusting the magnitudes of values in neural networks to prevent them from blowing up or shrinking down.
- The discovery of this concept happened at the University of Toronto in the Jeffrey Hinton Lab.
- The video also mentions the excitement surrounding AI in the past five years, particularly due to the ImageNet results.
- ImageNet is a large dataset of images that helped advance deep learning in computer vision.
- Teams participating in the ImageNet competition were able to achieve impressive error rates in object recognition.
- These advancements in computer vision also had implications for other fields such as speech recognition and translation.

- The video discusses the surprising nature of translation and the challenges it presents for neural networks.
- Recurrent neural networks are introduced as a solution for handling variable length input and output in translation.
- The paper called "Sequence to Sequence" is mentioned as a model that uses recurrent neural networks to translate word by word.
- Neural network based solutions are found to outperform other systems, but are not widely deployed due to computational costs.
- The difference between narrow AI, general AI, and super intelligence is explained.
- Currently, supervised learning is the most successful paradigm in machine learning for business applications.
- The field of AI is both under hyped and over hyped, as it has shown success in certain areas but still faces challenges in others.

- The problem of how to grip an apple or place fingers on it is not scientifically solved.
- To learn more about AI, Coursera and TensorFlow tutorials are recommended.
- Andrej Karpathy's class at Stanford is accessible and can be found on YouTube.
- Exercises can be done through homework or by trying out random thought experiments.
- Basic income may be necessary in the future due to automation of low-level jobs.
- People may need help finding purpose in a world with automation.
- The book "Homo Deus" and movies like "Hair" and "Ex Machina" are recommended for inspiration.

- OpenAI, a company focused on building AI for the benefit of humanity, has received a $1 billion investment.
- One of their main projects is robotics, particularly manipulation, which is a challenging aspect of robotics.
- Another project involves playing complex computer games, while a third project aims to understand human learning capabilities through playing a large number of computer games.
- The speaker discusses the challenges of using reinforcement learning and determining rewards in real-world AI systems.
- Deep learning, a paradigm within machine learning, has become popular due to its recognition that true intelligence requires multiple steps of computation.
- Neural networks, with layers represented by matrices and involving nonlinear operations, are commonly used in deep learning.
- Stochastic gradient descent is a commonly used learning algorithm for neural networks.
- The recent interest in deep learning is due to advancements in computational power and training techniques.
- Convolutional neural networks (CNNs) have significantly advanced image processing in deep learning.
- Proper weight initialization and adjusting value magnitudes are important for training deep neural networks.
- AI has shown success in certain areas, but still faces challenges in others.
- The field of AI includes narrow AI (specific tasks), general AI (human-like intelligence), and super intelligence.
- Coursera, TensorFlow tutorials, and Andrej Karpathy's class at Stanford are recommended resources to learn more about AI.
- The automation of low-level jobs may require the implementation of basic income and assistance in finding purpose in a world with automation.

Wojciech Zaremba is a cofounder of OpenAI: https://openai.com

OpenAI is a non-profit AI research company, focused on discovering and enacting the path to safe artificial general intelligence.

Read the transcript here: http://blog.ycombinator.com/an-ai-primer-with-wojciech-zaremba

YC's podcast is hosted by Craig Cannon - http://twitter.com/craigcannon

Hey, today we have Wojciech Zaremba and we're going to talk about AI.

So Wojciech, could you give us a quick background?

I think that deep learning and AI is a great application for robotics.

Prior to that I spent a year at Google Brain and I spent a year at Facebook AI Research and same time I graduated from...

So the great thing about both of these organizations is that they are focused on research.

So throughout my PhD, I was actually publishing papers over there.

I highly recommend both organizations, as well as, of course, OpenAI.

Yeah, okay, so most people probably don't know what OpenAI is, so could you just give that

OpenAI focuses on building AI for the good of humanity.

We are a group of researchers and engineers collaborating together who essentially try to figure out what are the missing pieces of general artificial intelligence and how to build it in a way that would be maximally beneficial to humanity as a whole.

Open AI is greatly supported by Elon Musk and Sam Altman.

In total, we gather an investment of $1 billion

And so what are, I mean, I know some, but what are the OpenAI projects?

So there is several large projects going on.

Simultaneously we have also, we are doing also basic research.

So let me first enumerate large projects.

So in terms of robotics, we are working on manipulation.

We think that manipulation is the 1 of the parts of robotics which is the most unresolved.

Sorry, just to clarify, what does that mean exactly?

It means that, so in robotics, there are essentially 3 major families of tasks.

1 is locomotion, which means how to move from, let's say how to walk, how to move from point A to point B.

Let's say you are moving in the complicated environment such as, for instance, a flat or a building and you have to figure out actually to which rooms have you visited before, which not and where to go.

So it means you want to grasp an object, let's say open an object, place objects in various locations.

And the third 1 is the 1 which is currently the most difficult.

So it turns out that when it comes to arbitrary objects, current robots are unable to just grasp an arbitrary object.

For any object it's possible to hand code a single solution.

So say as long as, let's say in Factory, if you have the same object, like, I don't know, you are producing glasses, And there exists a hand-coded solution to it.

There is a way by code to write the program saying, let's place a hand in the middle of the glass and then let's close it.

But there is no way so far to write the program such that it would be able to grasp an arbitrary object.

And then just very quickly, the other OpenAI projects going on?

So another 1 has to do with playing a complicated computer game.

And the third 1 has to do with playing a large number of computer games.

That so human is has an incredible skill of being able to learn extremely quickly.

And it has to do with a prior experience.

So let's say even if you haven't played ever a volleyball, if you try it out for the first time within 10 or 15 minutes, you would be able to grasp how to actually, how to play.

And it has to do with all the prior experience that you have from different games.

If you would put the child, or like if you would put an infant on the volleyball court and ask him or her to play, it would fail miserably.

But I mean, due to the fact that it has experience coming from large number of other games, or let's say other life situations, it's able to actually transfer all the knowledge.

So at OpenAI, we're able to pull together a large number of computer games and computer games can be, it's quite easy to quantify how good they are in the computer game.

So first of all it's possible for many computer games to write a program that solves it pretty well or plays it well.

And there are also results from, let's, there are results in terms of reinforcement learning or in terms of so-called deep reinforcement learning, showing that it's possible to learn how to play a computer game.

These are, these are the initial results are coming from DeepMind.

And, but simultaneously, simultaneously it takes extremely long time, like in terms of real time execution to learn to play computer games.

So for instance, Atari games, for instance, in terms of real-time execution it takes something around 3 years of play to learn to play simple games.

I mean, it can be hugely paralyzed, therefore it takes a few days to train it on current computers, but it's way shorter for humans.

And is that through you giving it feedback?

So the way how it works in case of computer games, the feedback comes from the score.

So it looks at the score in the game and tries to optimize it and I would say that's kind of reasonable but I would say simultaneous is not that satisfying to me so the reason why it's not that satisfying to me So the assumption underlying reinforcement learning is that there is some environment and environment, you are an agent and you are acting in environment by executing actions and getting rewards from the environment.

And the rewards might be thought as, let's say, pleasure or so.

And the main issue is that it's actually not that easy to figure out what are the rewards in the real world.

Further on, other underlying assumption is in being able to reset environment to kind of get to repetitive the same situation and So the system can try Thousands or millions of times to actually finish a game so so there are some Small discrepancies People also believe that it might be possible somehow to hard code into system rewards, but I would say that's actually 1 of the big issues that it's kind of unresolved.

Like when I look how my nephew plays computer game, he actually doesn't look on score because he cannot read.

And still, my nephew, yeah, they can play pretty well.

So I mean, you can say maybe reward is somewhat different.

Maybe reward comes from seeing a nice, hearing a nice voice in the game or so.

But I would say that's something what is very unclear how to build a system and what system should optimize.

So in some sense, if we have a metric that we want to optimize, it's possible to build a system that could optimize for it.

But it turns out that in many cases it's not that easy.

And I would say that's actually 1 of the motivations why I wanted to work on robotics.

Because in case of robotics, it's way closer to the system that we care about.

So what I mean by that, For instance, let's say you would like your robot to prepare scramble eggs for you.

And so the question is, so how should I build a reward?

And in computer games actually the nice thing is they are getting reward extremely frequently.

So let's say anytime you kill an enemy or let's say won't die, it's quite great.

But in case of scrambling eggs, it would mean or the way how people write rewards for systems, it would mean distance from hand to a pan.

Then let's say somehow you have to quantify what's the, if the egg is, if you were able to crack open an egg, or let's say if you fried it sufficiently and how to kind of quantify it turns out to be extremely difficult.

And also there is no way even to reset the system.

How to reset the system to the same place.

So, okay, these are like fundamental issues and the reason why I'm personally interested in robotics is think that actually these challenges will tell us how to solve.

So let's start by defining a couple things.

So artificial intelligence is actually extremely broad.

It's extremely broad domain and the machine learning is a sub-part of this domain.

And in a sense, artificial intelligence consists of any writing, any software that tries to solve some problems through some intelligence.

It might be hand-coded solution, rules, rule-based system.

Yeah, so pretty much it's actually very hard to say what is not artificial intelligence.

So initial version for instance of Google search was based on...

It was avoiding any machine learning and it was...

There was like a well-defined algorithm called PageRank and essentially PageRank counts how many incoming links are from other websites and that's artificial intelligence.

It's essentially a system that does intelligent things for you.

Then over the time, Google search started to use machine learning because it was, it helps to improve results.

At the same time, they wanted to avoid it for some time as it's more difficult to interpret the results and it's more difficult to actually understand what system does.

Machine learning, it's essentially way of building, or let's say, essentially you have data and you would like to generate based on data program with some behavior.

So like the most common example, which is still sub-branch of machine learning, so-called supervised learning.

So you have pairs of examples, x comma y, which means I would like to map x to y, for instance, either if given email is spam or not spam, or let's say, if an image, what is the category of an image, or for instance, to whom should I recommend given product?

And based on these data, I would like to generate a program, some sort of the black box or some function that for new examples would be able to give you similar answers.

And that's an example of supervised learning.

But this says Machine learning means that you would like to generate program from data.

And this usually uses statistical machine learning method.

So somehow you can't somebody, how many times given events, given events occurred or so.

So deep learning, that's 1 paradigm in terms of machine learning.

So People realize that if you want to, as I said, machine learning means that you get data as an input and program as an output.

And deep learning says that the computation of the program, what I'm actually doing with this data should involve many steps.

And pretty much that's it in terms of meaning of deep learning.

So you might ask why it's so popular now and how it's so different from what was there before.

So it turns out that if you assume that you do 1 step of computation, let's say that you take your data and you kind of have single if statement or a small number of if statements, then I can, like for instance, say if you have a, I don't know, let's say your data is a recording from a stock market and you're saying you're gonna sell or buy depending on values bigger or smaller than something or if let's say depending on who is the new president or so you are making some decisions.

So in a sense, there's that in case of models that are based on single step, people are able to prove plenty of stuff mathematically.

And in terms of models that require multiple steps of computation, mathematical proofs are extremely weak.

And for a long time, models that do single step of computation, they were outperforming models that do many steps of computation.

And it was for many people, it was obvious for a long time that true intelligence cannot be done in single step, but it would require many steps.

But so far many systems actually they worked in the way that they had kind of very, very shallow.

They were very shallow, but simultaneously extremely gigantic.

So what I mean by that, you could generate, let's say, for the task of interest, let's say the recommendation, you could generate large number of features, let's say thousands of them.

These are features saying, for instance, from, let's say you want to do movie recommendation, You can say, is movie longer or shorter than 2 hours?

And then, or let's say 100,000, that's actually quite reasonable value.

And then your shallow classifier can essentially determine based on the combination of these features either to recommend it to you or not.

In case of deep learning, you would say, let's kind of combine it for multiple steps.

And that's essentially, that's entire difference.

And in case of deep learning, the most successful embodiment of deep learning is in terms of neural networks.

So neural networks, it's also extremely simple concept and that's something that people came up with a long time ago.

You have an input, This might be say vector or it might have some additional structure like let's say image.

So it's kind of a matrix 2 dimensional and you neural network.

And what you do is you multiply your input by a matrix and apply some nonlinear operation and multiply it again by a matrix and apply nonlinear operation.

You might ask why would I even need to apply this nonlinear operation?

Turns out that if you would multiply by 2 matrices, it can be reduced to multiplication by single matrix.

Like a composition of 2 linear operators can be written as single linear operator.

You could multiply these matrices together and the result of the, and you could condense it into single matrix.

And non-linearity is something like, the classical non-linearity, But there are, so say, there are extremely large number of variants in terms of what I said.

But What I just described is so-called feed-forward neural network.

So it essentially takes input, multiplies it by matrix, non-linearity multiplies it by matrix.

Examples of non-linearities, there is something, 1 which is classical, something called sigmoid.

So sigmoid is a function that it has a shape of s character, s letter.

It's kind of close to 0 for negative values.

It grows to half at 0 and then goes up to 1 when the values are larger.

It kind of modulates the input and that's the most classical version of activation function it turns out that the 1 which is even simpler empirically works way better which is called relu rectify linear unit and this 1 it's ridiculously simple.

So when you have negative value, you set 0.

You have positive value, just copy the value and that's it.

So you might ask, so first of all, what are the successes of deep learning?

Why, what changed and why it's so much different than it was before.

No, it's exactly where I was gonna go, but I was gonna ask beforehand.

Yeah, why neural networks are a thing now as opposed to in the past?

The main difference is all of a sudden we can train them to solve various problems.

And let's say 1 family of problems, these are problems in supervised learning.

So better than any other method, they can map these examples to labels and then on the holdout data, on test data, they outperform anything else.

And in many cases, they get superhuman results.

And is that just a function of like computational power that we have access to?

When it comes to models and neural networks is an example of model, there is always a question.

So how to figure out parameters of a model?

So there is some training procedure and the most common procedure for neural networks is so-called stochastic gradient descent.

It's also a ridiculously simple procedure.

And it turns out that empirically it works very well.

So people came out with a vast number of learning algorithms.

Stochastic gradient descent is an example of 1 learning algorithm.

Let's say there is something called Hebbian learning that's motivated by the way how neurons in human brain learn.

But this 1 so far empirically is working the best.

Okay, so then let's go to the question you asked yourself, which is why now?

What's happening to make people care about it right now?

So since 20 years ago, there were several small differences in terms of how people train neural networks.

And there is a large increase in computational power.

So number 1 advance, I would say that's even, the 1 advance that's actually an old 1, but it seems to be extremely critical, something called convolutional neural network.

Yeah, so It's actually a very simple concept.

So let's say your input is an image and let's say your image is of a size 200 by 200.

The number of values in total is actually 120 000 so if you would actually squash it into a vector this vector would be of this size okay And then I can think that if you would like, let's say, to apply neural network to essentially multiply it by a matrix and let's say if you would like to have output of the multiplication of similar size, let's say 120 000, then all of a sudden the matrix to multiply it would be of a gigantic size.

And learning consists of estimating parameters for neural network.

It turns out that empirically that wouldn't essentially work.

That if you would use algorithm of backpropagation you would get quite poor results.

And people realized that in case of images you might want to multiply by a little bit special matrix that also allows to do way faster computations.

So you can think that neural network as it applies some computation to the input.

So neural network applies some computation to the to the input.

You might want to constrain this computation in some sense.

So you might think as you will have several layers, maybe initially you would like to do very local computation.

And it should be pretty much similar in every location.

So you would like to apply the same computation in the center as in the corners.

Maybe later on you need some diversification.

But you want to pre-process image the same way.

So the idea is that when you take an image or any actually two-dimensional structure, so the other example is you can take voice.

And it turns out that you can, by applying Fourier transform, you turn voice into image.

And all the, it's like a 2 dimensional image.

It's a, yeah, so you take a waveform and you apply Fourier transform.

And essentially on the X axis you have time as the speech goes on.

And on Y-axis, you have different frequencies.

And that's an image and speech recognition systems, they also, they treat sound as it would be an image.

So that's why I'm saying that the technique that, also, like a kind of as a sidetrack, The cool thing about neural networks is it used to be the case that people specialized in processing text, images, sound, and these days this is the same group of people.

So coming back to what is convolutional neural network Yeah, as I mentioned you would like to apply the same computation All over the placing image and essentially convolutional neural network says when we take an image, let's just connect a neuron with local values on the image And let's copy the same weights over and over again.

So this way you will multiply kind of, multiply values in the center in the corners by the same values in the matrix.

And so input to the convolution is an image, and output is kind of also an image.

You can think that there is also some specific vocabulary.

So this kind of three-dimensional image is like you have height and you have also depth.

let's say in case of image that's 3 dimensions and then you apply convolution, you can kind of change number of that dimensions.

Usually people go to, let's say, I don't know, 100 dimensions or so.

And then you kind of have several of these layers and then there are so-called fully connected layers which are just conventional matrices.

So I would say that's 1 of advances that actually happened 20 years ago already.

Another 1 which is, it might sound kind of funny, but for a long time people didn't believe that it's possible to train deep neural networks and they were thinking quite a lot about what are the proper learning algorithms.

And it turns out that, so let's say when you train a neural network, you start off by initializing weights to some random values.

And it turns out that it's very important to be careful to what magnitudes you initialize weights.

And if you set it to right values, and I can even give you some, let's say, intuition of what it means.

Turns out that then simplest algorithm, which is called stochastic gradient descent, actually works pretty well.

So, some sense, as I said, let's say, layers of neural network, they kind of multiply, they multiply input by matrices.

And a property that you would like to retain, it's you don't want the magnitude of values to blow up and also you don't want it to shrink down.

And if you kind of multiply, if you choose random initialization, it's easy to choose some initialization that will kind of, you know, turn the magnitude to keep on increasing And then if you have 10 layers and let's say in each of them you multiply by 2 2 2 2 2

then the output all the Sun is of completely different magnitude and Learning is not happening anymore And if you kind of just choose them and it's a matter of choosing variants are like a magnitude of initial weights and if you set it such that let's say output is of the same magnitude as input and everything works.

So basically just adjusting those magnitudes was what proved that you could do this with a neural network?

That's kind of ridiculous that let's say people haven't realized it for a long time, but that's what it is.

It happened actually at the University of Toronto.

See all Y Combinator transcripts on Youtube

An AI Primer with Wojciech Zaremba