Boris Sofman is the Senior Director Of Engineering and Head of Trucking at Waymo, formerly the Google Self-Driving Car project. He was also the CEO and co-founder of Anki, a home robotics company. Please support this podcast by checking out our sponsors:
- LMNT: https://drinkLMNT.com/lex to get free sample pack
- Athletic Greens: https://athleticgreens.com/lex and use code LEX to get 1 month of fish oil
- ROKA: https://roka.com/ and use code LEX to get 20% off your first order
- Indeed: https://indeed.com/lex to get $75 credit
- BetterHelp: https://betterhelp.com/lex to get 10% off

EPISODE LINKS:
Boris's Twitter: https://twitter.com/bsofman
Boris's LinkedIn: https://www.linkedin.com/in/bsofman
Waymo's Twitter: https://twitter.com/waymo
Waymo's YouTube: https://www.youtube.com/waymo
Waymo's Website: https://waymo.com/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
1:08 - Robots in science fiction
6:49 - Cozmo
32:04 - AI companions
38:59 - Anki
1:04:33 - Waymo Via
1:36:10 - Sensor suites for long haul trucking
1:46:06 - Machine learning
2:04:03 - Waymo vs Tesla
2:14:38 - Safety and risk management
2:23:42 - Societal effects of automation
2:34:47 - Amazon Astro
2:39:12 - Challenges of the robotics industry
2:43:39 - Humanoid robotics
2:50:42 - Advice for getting a PhD in robotics
2:58:13 - Advice for robotics startups
3:09:19 - Advice for students

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Boris Sofman, who is the Senior Director of Engineering and Head of Trucking at Waymo, the autonomous vehicle company, formerly the Google Self-Driving Car Project.

Before that, Boris was the co-founder and CEO of Anki, a robotics company that created Cosmo, which in my opinion, is 1 of the most incredible social robots ever built.

It's a toy robot, but 1 with an emotional intelligence that creates a fun and engaging human robot interaction.

It was truly sad for me to see Anki shut down when he did.

I had high hopes for those little robots.

We talk about this story and the future of autonomous trucks, vehicles, and robotics in general.

I spoke with Steve Viscelli recently on episode 237 about the human side of trucking.

This episode looks more at the robotics side.

To support it, Please check out our sponsors in the description.

And now, here's my conversation with Boris Sofman.

Who is your favorite robot in science fiction, books or movies?

WALL-E and R2-D2, where they were able to convey such an incredible degree of intent, emotion, and kind of character attachment without having any language whatsoever.

And just purely through the richness of emotional interaction.

And then the Terminator series just like really pretty wide range, right?

But I kind of love this dynamic where you have this like incredible Terminator itself that Arnold played.

But, and then he was kind of like the inferior like previous generation version that was like totally outmatched, you know, in terms of kind of specs by the new 1, but you know, still kind of like held his own.

And so it was kind of interesting where you, you realize how many, how many levels there are on the spectrum from human to kind of potentials and AI and robotics to futures.

And so, yeah, that movie really, as much as it was like kind of a direct world in a way, was actually quite fascinating, gets the imagination going.

Well, from an engineering perspective, both the movies you mentioned, WALL-E and Terminator, the first 1, is probably achievable.

You know, humanoid robot, maybe not with like the realism in terms of skin and so on, but that humanoid form, we have that humanoid form, it seems like a compelling form.

Maybe the challenge is it's super expensive to build, but you can imagine, maybe not a machine of war, but you can imagine Terminator type robots walking around.

And then the same, obviously, with WALL-E, you've basically, so for people who don't know, you created the company Anki that created a small robot with a big personality called Cosmo that just does exactly what Wally does, which is somehow with very few basic visual tools is able to communicate a depth of emotion.

But then again, the humanoid form is super compelling.

So like Cosmo is very distant from a humanoid form, and then the Terminator has a humanoid form, and you can imagine both of those actually being in our society.

It's true, and it's interesting because it was very intentional to go really far away from human form when you think about a character like Cosmo or like Wally, where you can completely rethink the constraints you put on that character, what tools you leverage, and then how you actually create a personality and a level of intelligence interactivity that actually matches the constraints that you're under, whether it's mechanical or sensors or AI of the day.

This is why I was always really surprised by how much energy people put towards trying to replicate human form in a robot because you actually take on some pretty significant kind of constraints and downsides when you do that.

The first of which is obviously the cost, where it's just the articulation of a human body is just so magical in both the precision as well as the dimensionality, that to replicate that even in its reasonably close form takes a giant amount of joints and actuators and motion and sensors and encoders and so forth.

But then you're almost like setting an expectation that the closer you try to get to human form, the more you expect the strengths to match.

And that's not the way AI works, is There's places where you're way stronger and there's places where you're weaker.

And by moving away from human form, you can actually change the rules and embrace your strengths and bypass your weaknesses.

And at the same time, the human form like has way too many degrees of freedom to play with.

It's kind of counterintuitive, just as you're saying, but when you have fewer constraints, it's almost harder to master the communication of emotion.

Like you see this with cartoons, like stick figures.

You can communicate quite a lot with just very minimal, like 2 dots for eyes and a line for a smile.

I think you can almost communicate arbitrary levels of emotion with just 2 dots and a line.

And if you focus on just that, you can communicate the full range.

And then if you do that, then you can focus on the actual magic of human and dot line interaction versus all the engineering mess.

Dimensionality, voice, all these sort of things, they actually become a crutch where you get lost in a search space almost.

And so some of the best animators that we've worked with, they almost like study when they come up, you know, kind of in building their expertise by forcing these projects where all you have is like a ball that can like kind of jump and manipulate itself or like really, really like aggressive constraints where you're forced to kind of extract the deepest level of motion.

And so in a lot of ways, when we thought about Cosmo, I was like, you're right.

If we had to describe it in 1 small phrase, it was bringing a Pixar character to life in the real world.

And in a lot of ways, what was interesting is that with WALL-E, which we studied incredibly deeply, and in fact, some of our team were, you know, kind of had worked previously at Pixar and on that project, they intentionally constrained WALL-E as well, even though in an animated film, you could do whatever you wanted to, because it forced you to like, really saturate the smaller amount of dimensions, but you sometimes end up getting a far more beautiful output because you're pushing at the extremes of this emotional space in a way that you just wouldn't because you get lost in the surface area if you have like something that is just infinitely articulable.

So if we backtrack a little bit, you thought of Cosmo in 2011, in 2013 actually designed and built it.

What was the vision behind this incredible little robot?

We started Anki back in, like while we were still in graduate school.

So myself and my 2 co-founders, we were PhD students in the Robotics Institute at Carnegie Mellon.

And so we were studying robotics, AI, machine learning, kind of different areas.

1 of my co-founders was working on walking robots for a period of time.

And so we all had a bit of a really deep, kind of a deeper passion for applications of robotics and AI, where there's like a spectrum where there's people that get like really fascinated by the theory of AI and machine learning robotics, where whether it gets applied in the near future or not is less of a kind of factor on them, but they love the pursuit of the challenge.

And there's a lot of incredible breakthroughs that happen there.

We're probably closer to the other end of the spectrum where we love the technology and all the evolution of it, but we were really driven by applications.

Like, how can you really reinvent experiences and functionality and build value that wouldn't have been possible without these approaches.

And we had a kind of some experiences through previous jobs and internships where we like got to see the applied side of robotics.

And At that time, there was actually relatively few applications of robotics that were outside of pure research or industrial applications, military applications and so forth.

So maybe iRobot was like 1 exception and maybe there were a few others, but for the most part, there weren't that many.

And so we got excited about consumer applications of robotics where you could leverage way higher levels of intelligence through software to create value and experiences that were just not possible in those fields today.

And we saw kind of a pretty wide range of applications that varied in the complexity of what it would take to actually solve those.

And what we wanted to do was to commercialize this into a company, but actually do a bottoms up approach where we could have a huge impact in a space that was ripe to have an impact at that time, and then build up off of that and move into other areas.

Entertainment became the place to start because you had relatively little innovation in a toy space, an entertainment space.

You had these really rich experiences in video games and movies, but there was like this chasm in between.

And so we thought that we could really reinvent that experience and there was a really fascinating transition technically that was happening at the time where the cost of components was plummeting because of the mobile phone industry and then the smartphone industry.

And so the cost of a microcontroller, of a camera, of a motor, of memory, of microphones, cameras, was dropping by orders of magnitude.

And then on top of that, with the iPhone coming out in

It started to become apparent within a couple of years that this could become a really incredible interface device and the brain with much more computation behind a physical world experience that wouldn't have been possible previously.

And so we really got excited about that and how we push all the complexity from the physical world into software by using really inexpensive components, but putting huge amounts of complexity into the AI side.

And so Cosmo became our second product, and then the 1 that we're probably most proud of.

The idea there was to create a physical character that had enough understanding and awareness of the physical world around it and the context that mattered to feel like he was alive.

And to be able to have these emotional connections and experiences with people that you would typically only find inside of a movie.

We had an incredible respect and appreciation for what they were able to build in this like really beautiful fashion and film.

But it was always like a, you know, 1 it was virtual and 2, it was like a story on rails that had no interactivity to it.

It was very fixed and it obviously had a magic to it, but where you really start to hit a different level of experiences when you're actually able to physically interact with that robot.

And then that was your idea with Anki, like the first product was the cars.

So basically you take a toy, you add intelligence into it in the same way you would add intelligence into AI systems within a video game, but you're not bringing it into the physical space.

So the idea is really brilliant, which is you're basically bringing video games to life.

We literally use that exact same phrase because in the case of Drive, this was a parallel of the racing genre.

And the goal was to effectively have a physical racing experience, but have a virtual state at all times that matches what's happening in the physical world.

And then you can have a video game off of that and you can have different characters, different traits for the cars, weapons and interactions and special abilities and all these sorts of things that you think of virtually, but then you can have it physically.

And 1 of the things that we were really surprised by that really stood out and immediately led us to really like kind of accelerate the path towards Cosmo is that things that feel like they're really constrained and simple in the physical world, they have an amplified impact on people where the exact same experience virtually would not have anywhere near the impact, but seeing it physically really stood out.

And so effectively with Drive, we were creating a video game engine for the physical world.

And then with Cosmo, we expanded that video game engine to create a character and kind of an animation and interaction engine on top of it that allowed us to start to create these much more rich experiences.

And a lot of those elements were almost like a proving ground for what would human robot interaction feel like in a domain that's much more forgiving, where you can make mistakes in a game.

It's okay if like, if, you know, car goes off the track or if Cosmo makes a mistake.

And what's funny is actually, we were so worried about that.

In reality, we realized very quickly that those mistakes can be endearing, and if you make a mistake, as long as you realize you made a mistake and have the right emotional reaction to it, it builds even more empathy with the character.

So, when the thing you're optimizing for is fun, you have so much more freedom to fail, to explore, And also in the toy space, like all of this is really brilliant.

And I got to ask you, backtrack, it seems for roboticist to take a jump in into the direction of fun is a brilliant move.

Because 1, you have the freedom to explore and to design all those kinds of things.

Like you don't have to, like if you're not chasing perfection and like toys, it's understood that you can go cheaper, which means in robot, it's still expensive, but it's actually affordable by a large number of people.

So it's a really brilliant space to explore.

And in fact, we realized pretty quickly that like perfection is actually not fun because like in a traditional robotic, roboticist sense, the first kind of path planner, and this is the part that I worked on out of the gate, was a lot of the AI systems where you have these vehicles and cars racing, making optimal maneuvers to try to get ahead.

And you realize very quickly that like, that's actually not fun because you want the like, chaos from mistakes and the, and so you start to kind of intentionally almost add noise to the system in order to kind of create more of a realism in the exact same way the human player might start really ineffective and inefficient and then start to kind of increase their quality bar as they progress.

And there is a really, really aggressive constraint that's forced on you by being a consumer product where the price point matters a ton, particularly in kind of an entertainment where you can't make a $1,000 product unless you're going to meet the expectations of a $1,000 product.

And so in order to make this work, like your cost of goods had to be like, like, you know, well under a hundred dollars.

In the case of Cosmo, we got it under $50 and fully packaged and delivered.

So, okay, if we sit down like at the early stages, if we go back to that and you're sitting down and thinking about what Cosmo looks like

a design perspective and from a cost perspective, I imagine that was part of the conversation.

Is there a target you're trying to chase?

Did you have a vision in mind, like size?

Did you have, because there's a lot of unique qualities to Cosmos, so for people who don't know, they should definitely check it out.

There's a display, there's eyes on the display,

and those eyes can, it's pretty low resolution eyes, right?

And there's this arm, like that, lift stuff, but there's something about arm movement that adds even more kind of depth.

It's like the face communicates emotion and sadness and disappointment and happiness.

And then the arms kind of communicates, I'm trying here.

I'm doing my best in this complicated world.

Exactly, so it's interesting because like, all of Cosmos is only 4 degrees of freedom.

And 2 of them are the 2 treads, which is for basic movement.

And so you literally have only a head that goes up and down, a lift that goes up and down, and then your 2 wheels.

And you have sound and a screen, and a low resolution screen.

And with that, it's actually pretty incredible what you can come up with, where like you said, it's a really interesting give and take because there's a lot of ideas far beyond that, obviously, as you can imagine, where, Like you said, how big is it?

It's a formula that actually scales way beyond entertainment.

This is the formula for human kind of robot interface more generally is you almost have this triangle between the physical aspects of it, the mechanics, the industrial design, what's mass-producible, the cost constraints, and so forth.

You have the AI side of how do you understand the world around you, interact intelligently with it, execute what you want to execute, so perceive the environment, make intelligent decisions, and move forward.

And then you have the character side of it.

Most companies that have done anything in human-robot interaction really miss the mark or underinvest in the character side of it.

They overinvest in the mechanical side of it, and then varied results on the AI side of it.

And so the thinking is that you put more mechanical flexibility into it, you're going to do better.

You don't necessarily, you actually create a much higher bar for a higher ROI, because now your price point goes up, your expectations go up.

And if the AI can't meet it or the overall experience isn't there, you missed the mark.

So who, like, how did you, through those conversations, get the cost down so much and make it, made it so simple?

Like that, there's a big theme here because you come from the mecca of robotics, which is Carnegie Mellon University, robotics.

For all the people I've interacted with that come from there, or just from the world experts at robotics, they would never build something like Cosmo.

It came from this combination of a team that we had.

It was quite cool because like we, and by the way, you ask anybody that's like experienced in the like, kind of, you know, toy entertainment space, you'll never sell product over $99.

That was fundamentally false and we believed it to be false.

It was because experience had to kind of meet the mark.

And so we pushed past that amount, but there was a pressure where the higher you go, the more seasonal you become and the tougher it becomes.

And so on the cost side, we very quickly partnered up with some previous contacts that we worked with where, just as an example, our head of mechanical engineering was 1 of the earliest heads of engineering at Logitech and has a billion units of consumer products in circulation that he's worked on.

So, like crazy low cost, high volume consumer product experience.

We had a really great mechanical engineering team and just a very practical mindset where we were not going to compromise on feasibility in the market in order to chase something that would be an enabler.

And we pushed a huge amount of expectations onto the software team where yes, we're gonna use cheap, noisy motors and sensors, but we're gonna fix it on the software side.

Then we found on the design and character side, there was a faction that was more from a game design background that thought that it should be very games-driven Cosmo, where you create a whole bunch of games experiences and it's all about game mechanics.

And then there was a faction which my co-founder and I are the most involved in this, really believed in, which was character-driven.

And the argument is that you will never compete with what you can do virtually from a game standpoint, but you actually on a character side, put this into your wheelhouse and put it more towards your advantage because a physical character has a massively higher impact physically than virtually.

When I, for people who don't know, Cosmo plays games with you.

And I actually, when I was, you know, playing with it, I wondered exactly what is the compelling aspect of this?

Because to me, obviously I'm biased, but to me the character, what I enjoyed most, honestly, or what got me to return to it as the character.

That's a fascinating discussion of, you're right.

Ultimately, you cannot compete on the quality of the gaming experience.

The physical world is just too restrictive and you don't have a graphics engine, it's like all this.

But on the character side, we, and clearly we moved in that direction as like kind of the winning path.

And we partnered up with this really, we immediately like went towards Pixar and Carlos Bena, he was 1 of, like had been at Pixar for 9 years.

He'd worked on tons of the movies, including WALL-E and others, and just immediately kind of spoke the language and it just clicked on how you think about that, like kind of magic and drive.

And then we built out a team with him as a really prominent driver of this, with different types of backgrounds and animators and character developers, where we put these constraints on the team, but then got them to really try to create magic despite that.

And we converged on this system that was at the overlap of character and the character AI that were, if you imagine the dimensionality of emotions, happy, sad, angry, surprised, confused, scared, like you think of these extreme emotions, we almost like kind of put this challenge to kind of populate this library of responses on how do you show the extreme response that like goes to the extreme spectrum on angry or frustrated or whatever.

And so that gave us a lot of intuition and learnings.

And then we started parameterizing them where it wasn't just a fixed recording, but they were parameterized and had randomness to them where you could have infinite permutations of happy and surprised and so forth.

And then we had a behavioral engine that took the context from the real world and would interpret it and then create kind of probability mappings on what sort of responses you would have that actually made sense.

And so if Cosmos saw you for the first time in a day, he'd be really surprised and happy in the same way that the first time you walk in and like your toddler sees you, they're so happy, but they're not going to be that happy for the entirety of your next 2 hours.

But like you have this like spike in response, or if you leave him alone for too long, he gets bored and starts causing trouble and like nudging things off the table.

Or if you beat him in a game, the most enjoyable emotions are him getting frustrated and grumpy to a point where our testers and our customers would be like, I had to let him win because I don't want him to be upset.

And so you start to create this feedback loop where you see how powerful those emotions are.

And just to give you an example, something as simple as eye contact.

You don't think about it in a movie, just like it kind of happens, like, you know, camera angles and so forth.

But that's not really a prominent source of interaction.

What happens when a physical character like Cosmo, when he makes eye contact with you, it built universal kind of connection, kids all the way through adults.

It was not like people stopped caring after 10, 12 years old.

And so we started doing experiments and we found something as simple as increasing the amount of eye contact, like the amount of times in a minute that he'll look over for your approval to like kind of make eye contact, just by I think doubling it, we increased the playtime engagement by 40%.

Like, you see these sort of, like, kind of interactions where you build that empathy, and so we studied pets, we studied virtual characters.

There's like, a lot of times, actually, dogs are 1 of the perfect, most perfect influencers behind these sort of interactions.

And what we realized is that the games were not there to entertain you.

The games were to create context to bring out the character.

And if you think about the types of games that you played, they were relatively simple, but they were always ones to create scenarios of either tension or winning or losing or surprise or whatever the case might be.

And they were purely there to just like create context to where an emotion could feel intelligent and not random.

And in the end, it was all about the character.

So yeah, there's so many elements to play with here.

What lessons do we draw from cats who don't seem to give a damn about you?

And so you could almost like in early expirations, we thought that it would be really incredible if you had a diversity of characters where you almost help encourage which direction it goes, just like in a role-playing game.

Like, think of, like, the 7 dwarves, sort of.

And initially, we even thought that it would be amazing if their characters actually help them have strengths and weaknesses in whatever they end up doing.

Like some are scared, some are arrogant, some are super warm and kind of friendly.

And in the end, we focused on 1 because it made it very clear that, hey, we got to build out enough depth here because you're kind of trying to expand.

It's almost like, how long can you maintain a fiction that this character is alive to where the person's explorations don't hit a boundary, which happens almost immediately with typical toys.

And even with video games, how long can we create that immersive experience to where you expand the boundary?

And 1 of the things we realized is that you're just way more forgiving when something has a personality and it's physical.

That is the key that unlocks robotics interacting in the physical world more generally, is that When you don't have a personality and you make a mistake as a robot, the stupid robot made a mistake, why is it not perfect?

When you have a character and you make a mistake, you have empathy and it becomes endearing and you're way more forgiving.

And that was the key that was like, I think goes far, far beyond entertainment.

It actually builds the depth of the personality, the mistakes.

So let me ask the movie, her question then.

How, so Cosmos feels like the early days of something that will obviously be prevalent throughout society at a scale that we cannot even imagine.

My sense is it seems obvious that these kinds of characters will permeate society and we'll be friends with them, we'll be interacting with them in different ways.

In a way we, I mean, you don't think of it this way, but when you play video games, they're often cold and impersonal, but even then, you think about role-playing games, you become friends with certain characters in that game.

But if they acknowledge that you exist in this physical world, if the characters in the game remember that you exist, that you, like for me, like Lex, they understand that I'm a human being who has like hopes and dreams and so on.

It seems like there's going to be like billions, if not trillions of cosmos in the world.

So if we look at that future, there's several questions to ask.

How intelligent does that future cosmos need to be to create fulfilling relationships like friendships?

And part of it was the recognition that it's gonna take time to get there, because it has to be a lot more intelligent.

Because what's good enough to be a magical experience for an eight-year-old, It's a higher bar to do that via like a pet in the home or to help with functional interface in an office environment or in a home or in so forth.

And so, and the idea was that you build on that and you kind of get there and as technology becomes more prevalent and less expensive and so forth, you can start to kind of work up to it.

But you're absolutely right at the end of the day, we almost equated it to how the touchscreen created like this really novel interface to physical kind of devices like this.

This is the extension of it where you have much richer physical interaction in the real world, this is the enabler for it.

And it shows itself in a few kind of really obvious places.

So just take something as simple as a voice assistant.

You will never, most people will never tolerate an Alexa or a Google Home just starting a conversation proactively when you weren't kind of expecting it, because it feels weird.

It's like you were listening and like, and then now you're kind of, it feels intrusive.

But if you had a character, like a cat that touches you and gets your attention, or a toddler, like you never think twice about it.

And what we found really kind of immediately is that these types of characters like Cosmo, and they would like roam around and kind of get your attention.

And we had a future version that was always on, kind of called Vector.

And so you could initiate interaction in a way that is not acceptable for machines.

See all Lex Fridman transcripts on Youtube

Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics | Lex Fridman Podcast #241