The following is a conversation with Gavin Miller.

Adobe has empowered artists, designers, and creative minds from all professions working in the digital medium for over 30 years with software such as Photoshop, Illustrator, Premiere, After Effects, InDesign, Audition, software that work with images, video, and audio.

Adobe Research is working to define the future evolution of these products in a way that makes the life of creatives easier, automates the tedious tasks, and gives more and more time to operate in the idea space instead of pixel space.

This is where the cutting edge, deep learning methods of the past decade can really shine more than perhaps any other application.

Gavin is the embodiment of combining tech and creativity.

Outside of Adobe Research, he writes poetry and builds robots, both things that are near and dear to my heart as well.

This conversation is part of the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Gavin Miller.

You're head of Adobe Research, leading a lot of innovative efforts in applications of AI, creating images, video, audio, language, but you're also yourself an artist, a poet, a writer, and even a roboticist.

So, while I promise to everyone listening that I will not spend the entire time we have together reading your poetry, which I love, I have to sprinkle it in at least a little bit.

So some of them are pretty deep and profound and some are light and silly.

Let's start with a few lines from the silly variety.

You write in Je ne vinaigrette rien, a poem that beautifully parodies both Edith Piaf's Je Ne Venais Gratien and my way by Frank Sinatra.

So it opens with, and now dessert is near.

I've tried to slim all year, but my diets have been anecdotal.

So where does that love for poetry come from for you?

And if we dissect your mind, how does it all fit together in the bigger puzzle of Dr.

That was a poem I wrote when I'd been to my doctor and he said you really need to lose some weight and go on a diet.

And whilst the rational part of my brain wanted to do that, the irrational part of my brain was protesting and sort of embraced the opposite idea.

Taken to an extreme, I thought it would be funny.

Obviously it's a serious topic for some people.

But I think for me, I've always been interested in writing since I was in high school, as well as doing technology and invention.

And sometimes there are parallel strands in your life that carry on.

And, you know, 1 is more about your private life and one's more about your technological career.

And then at sort of happy moments along the way, sometimes the 2 things touch.

think your writing, the art, the poetry contribute indirectly or directly to your research, to your work in Adobe?

Well sometimes it does if I say imagine a future in a science fiction kind of way and then once it exists on paper I think, well why shouldn't I just build that?

There was an example where when realistic voice synthesis first started in the 90s at Apple where I worked in research, it was done by a friend of mine.

I sort of sat down and started writing a poem which each line I would enter into the voice synthesizer and see how it sounded and sort of wrote it for that voice.

And at the time, the agents weren't very sophisticated.

And I kind of made up the poem to sort of match the tone of the voice.

And it sounded slightly sad and depressed.

So I pretended it was a poem written by an intelligent agent, sort of telling the user to go home and leave them alone, but at the same time, they were lonely and wanted to have company and learn from what the user was saying.

And at the time it was way beyond anything that AI could possibly do.

But, you know, since then it's becoming more within the bounds of possibility.

And then At the same time, I had a project at home where I did sort of a smart home.

And I had the talking voice, it would remind me when I walked in the door of what things I had to do.

I had buttons on my washing machine because I was a bachelor and I'd leave the clothes in there for 3 days and they'd go moldy.

So as I got up in the morning, it would say, don't forget the washing and so on.

I made photo albums that used light sensors to know which page you were looking at, would send that over wireless radio to the agent who would then play sounds that match the image you were looking at in the book.

So I was kind of in love with this idea of magical realism and whether it was possible to do that with technology.

So that was a case where the agent sort of intrigued me from a literary point of view and became a personality.

I think more recently, I've also written plays and when in plays you write dialogue and obviously you write a fixed set of dialogue that follows a linear narrative.

But with modern agents as you design a personality or a capability for conversation, you're sort of thinking of I kind of have imaginary dialogue in my head and then I think, what would it take not only to have that be real, but for it to really know what it's talking about?

So it's easy to fall into the uncanny valley with AI where it says something it doesn't really understand but it sounds good to the person but you rapidly realize that it's kind of just stimulus response it doesn't really have real-world knowledge about the thing it's describing And so when you get to that point, it really needs to have multiple ways of talking about the same concept so it sounds as though it really understands it.

Now what really understanding means is in the eye of the beholder, right?

But if it only has 1 way of referring to something, it feels like it's a canned response, but if it can reason about it or you can go at it from multiple angles and give a similar kind of response that people would then it starts to seem more like there's something there that's sentient.

You can say the same thing, multiple things from different perspectives.

I mean, with the automatic image captioning that I've seen the work that you're doing, there's elements of that, right?

Being able to generate different kinds of

So, in my team, there's a lot of work on turning a medium from 1 form to another, whether it's auto tagging imagery or making up full sentences about what's in the image, then changing the sentence, finding another image that matches the new sentence or vice versa.

And in the modern world of GANs, you sort of give it a description and it synthesizes an asset that matches the description.

My early days in my career were about 3D computer graphics, the sort of pioneering work, sort of before movies had special effects done with 3D graphics and sort of rode that revolution.

And that was very much like the Renaissance where people would model light and color and shape and everything.

And now we're kind of in another wave where it's more impressionistic and it's sort of the idea of something can be used to generate an image directly, which is Sort of the new frontier in computer image generation using AI algorithms So

so the creative process is more in the space of ideas or becoming more in the space of ideas versus in the raw pixels?

I think at Adobe, we really want to span the entire range from really, really good, what you might call low level tools by low level, as close to say, analog workflows as possible.

So what we do there is we make up systems that do really realistic oil paint and watercolor simulation.

So if you want every bristle to behave as it would in the real world and leave a beautiful analog trail of water and then flow after you've made the brushstroke.

You can do that, and that's really important for people who want to create something really expressive or really novel because they have complete control.

And then as certain other tasks become automated, it frees the artists up to focus on the inspiration and less of the perspiration.

So, thinking about different ideas, obviously.

Once you finish the design, there's a lot of work to say, do it for all the different aspect ratio of phones or websites and so on.

And that used to take up an awful lot of time for artists.

It still does for many what we call content velocity.

And 1 of the targets of AI is actually to reason about from the first example of what are the likely intent for these other formats.

Maybe if you change the language to German and the words are longer, how do you reflow everything so that it looks nicely artistic in that way?

And so the person can focus on the really creative bit in the middle, which is what is the look and style and feel and what's the message and what's the story and the human element.

So that's 1 way in which we're trying to just make it easier and faster and cheaper to do so that There can be more of it, more demand, because it's less expensive.

So everyone wants beautiful artwork for everything from a school website to Hollywood movie.

On the other side, as some of these things have automatic versions of them, people will possibly change role from being the hands-on artisan to being either the art director or the conceptual artist and then the computer will be a partner to help create polished examples of the idea that they're exploring.

Let's talk about Adobe products first, about AI and Adobe products.

Just so you know where I'm coming from, I'm a huge fan of Photoshop for images, Premiere for video, Audition for audio.

I'll probably use Photoshop to create the thumbnail for this video, Premiere to edit the video, Audition to do the audio.

That said, everything I do is really manually and I set up, I use this old school Kinesis keyboard and I have AutoHotKey that just, it's really about optimizing the flow of just making sure there's as few clicks as possible, so just being extremely efficient.

So before we get into the fun, sort of awesome deep learning things, Where does AI, if you could speak a little more to it, AI or just automation in general, do you see in the coming months and years, or in general, prior in 2018, fitting into making the life, the low level pixel work flow easier?

So we have a very rich array of algorithms already in Photoshop, just classical procedural algorithms as well as ones based on data.

In some cases, they end up with a large number of sliders and degrees of freedom.

So 1 way in which AI can help is just an auto button which comes up with default settings based on the content itself rather than default values for the tool.

So that's a very kind of make life easier for people whilst making use of common sense from other example images.

Another 1 is something we've spent a lot of work over the last 20 years I've been at Adobe, well 19, thinking about selection, for instance, where, you know, with a quick select, you would look at color boundaries and figure out how to sort of flood fill into regions that you thought were physically connected in the real world.

But that algorithm had no visual common sense about what a cat looks like or a dog.

It would just do it based on rules of thumb, which were applied to graph theory.

And it was a big improvement over the previous work where you had sort of almost click everything by hand, or if it just did similar colors, it would do little tiny regions that wouldn't be connected.

But in the future, using neural nets to actually do a great job with, say, a single click, or even in the case of well-known categories like people or animals, no click, where you just say, select the object, and it just knows the dominant object is a person in the middle of the photograph.

Those kinds of things are really valuable if they can be robust enough to give you good quality results.

Or they can be a great start for tweaking it.

Like 1 thing I'll, in a thumbnail, I'll take a picture of you right now and essentially remove the background behind you.

And I want to make that as easy as possible.

Like you don't have flowing hair, like rich, rich sort of.

in the past, it may come again in the future, but for now.

So that sometimes makes it a little more challenging to remove the background.

How difficult do you think is that problem for AI for basically making the quick selection tool smarter and smarter and smarter?

Well, we have a lot of research on that already.

If you want a sort of quick, cheap and cheerful, look, I'm pretending I'm in Hawaii, but it's sort of a joke, then you don't need perfect boundaries.

And you can do that today with a single click with the algorithms we have.

We have other algorithms where with a little bit more guidance on the boundaries, like you might need to touch it up a little bit.

We have other algorithms that can pull a nice mat from a crude selection.

So we have combinations of tools that can do all of that.

And at our recent Max conference, Adobe Max, we demonstrated how very quickly, just by drawing a simple polygon around the object of interest, we could not only do it for a single still, but we could pull a mat, well, pull at least a selection mask from a moving target, like a person dancing in front of a brick wall or something.

And so it's going from hours to a few seconds for workflows that are really nice.

And then you might go in and touch up a little.

You know, there's like a journey for an idea, right?

And what you presented probably at max has elements of just sort of, it inspires the concept.

It can work pretty well in majority of cases.

But how do you make something that works, well, in majority of cases, how do you make something that works maybe in all cases or it becomes a robust tool

So that really touches on the difference between academic research and industrial research.

So in academic research, it's really about who's the person to have the great new idea that shows promise.

And we certainly love to be those people too.

But we have sort of 2 forms of publishing.

1 is academic peer review, which we do a lot of, and we have great success there as much as some universities.

But then we also have shipping, which is a different type of peer.

And then we get customer review, as well as product critics.

And that might be a case where it's not about being perfect every single time, but perfect enough of the time, plus a mechanism to intervene and recover where you do have mistakes.

So we have the luxury of very talented customers.

We don't want them to be overly taxed doing it every time, but if they can go in and just take it from 99 to 100 with the touch of a mouse or something, then For the professional end, that's something that we definitely want to support as well.

And for them, it went from having to do that tedious task all the time to much less often.

If it had to be 100% automatic all the time, then that would delay the time at which we could get to market.

So on that thread, maybe you can untangle something.

Again, I'm sort of just speaking to my own experience.

So I think Photoshop, as an example, or Premiere, has a lot of amazing features that I haven't touched.

And so what's the, in terms of AI helping make my life or the life of creatives easier.

How this collaboration between human and machine, how do you learn to collaborate better?

Is it something that where you have to watch tutorials and you have to watch videos and so on?

Or do you ever think, do you think about the experience itself through exploration being the teacher?

So we, I'm glad that you brought this up.

1 is helping the person in the moment to do the task that they need to do, but the other is thinking more holistically about their journey learning a tool.

And we're just like, think of it as Adobe University where you use the tool long enough, you become an expert.

And not necessarily an expert in everything.

You don't necessarily know every street, but you know the important ones you need to get to.

So we have projects in research, which actually look at the thousands of hours of tutorials online and try to understand what's being taught in them.

And then we had 1 publication at CHI where it was looking at, given the last 3 or 4 actions you did, what did other people in tutorials do next?

So if you want some inspiration for what you might do next, or you just want to watch the tutorial and see, learn from people who are doing similar workflows to you, you can without having to go and search on keywords and everything.

So really trying to use the context of your use of the app to make intelligent suggestions, either about choices that you might make, or in a more assistive way where it could say, if you did this next, we could show you.

And that's basically the frontier that we're exploring now, which is If we really deeply understand the domain in which designers and creative people work, can we combine that with AI and pattern matching of behavior to make intelligent suggestions, either through verbal possibilities or just showing the results of if you try this.

That's really the, I was in a meeting today thinking about these things.

We'd all love an artist over 1 shoulder and a teacher over the other.

The right thing to do is to give enough at each stage that it's useful in itself, but it builds a foundation for the next level of expectation.

Are you aware of this gigantic medium of YouTube that's creating just a bunch of creative people, both artists and teachers of different kinds.

and the more we can understand those media types, both visually and in terms of transcripts and words, the more we can bring the wisdom that they embody into the guidance that's embedded in the tool.

That would be brilliant to remove the barrier from having to yourself type in the keyword searching, so on.

And then in the longer term, an interesting discussion is, does it ultimately not just assist with learning the interface we have, but does it modify the interface to be simpler?

Or do you fragment into a variety of tools, each of which has a different level of visibility of the functionality.

I like to say that if you add a feature to a GUI, you have to have yet more visual complexity confronting the new user.

Whereas if you have an assistant with a new skill, if you know they have it, so you know to ask for it, then it's sort of additive without being more intimidating.

So we definitely think about new users and how to onboard them.

Many actually value the idea of being able to master that complex interface and keyboard shortcuts like you were talking about earlier because with great familiarity it becomes a musical instrument for expressing your visual ideas and other people just want to get something done quickly in the simplest way possible And that's where a more assistive version of the same technology might be useful, maybe on a different class of device, which is more in context for capture, say.

Whereas somebody who's in a deep post-production workflow, maybe want to be on a laptop or big screen desktop and have more knobs and dials to really express the subtlety of what they want to do.

So there's so many exciting applications of computer vision and machine learning that Adobe is working on, like scene stitching, sky replacement, foreground, background removal, spatial object-based image search, automatic image captioning like we mentioned, Project Cloak, Project Deep Fill, filling in parts of the images, Project Scribbler, Style Transfer Video, Style Transfer on Faces and Video with Project Puppetron, best name ever.

Can you talk through a favorite or some of them or examples that popped in mind.

And I'm sure I'll be able to provide links to other ones we don't talk about because there's visual elements to all of them that are exciting.

Why they're interesting for different reasons might be a good way to go.

So I think a Sky Replace is interesting because, you know, we talked about selection being sort of an atomic operation.

It's almost like a, if you think of an assembly language, it's like a single instruction.

Whereas Sky Replace is a compound action where you automatically select the sky.

You look for stock content that matches the geometry of the scene.

You try to have variety in your choices so that you do coverage of different moods.

It then mattes in the sky behind the foreground, but then importantly, it uses the foreground of the other image that you just searched on to recolor the foreground of the image that you're editing.

So if you say go from a midday sky to an evening sky, it will actually add sort of an orange glow to the foreground objects as well.

I was a big fan in college of Magritte and he has a number of paintings where it's surrealism because he'll like do a composite but the foreground building will be at night and the sky will be during the day.

There's 1 called the Empire of Light, which was on my wall in college.

It can be a choice, but we'd rather have it be natural by default rather than it looking fake, and then you have to do a whole bunch of post-production to fix it.

So that's a case where we're kind of capturing an entire workflow into a single action and doing it in about a second rather than a minute or 2.

And when you do that, you can not just do it once, but you can do it for say like 10 different backgrounds and then you're almost back to this inspiration idea of I don't know quite what I want, but I'll know it when I see it.

And you can just explore the design space as close to final production value as possible.

And then when you really pick 1, you might go back and slightly tweak the selection mask just to make it perfect and do that kind of polish that professionals like to bring to their work.

So then there's this idea of, you mentioned the sky, replacing it to different stock images of the sky.

But making even more intelligent choices about ways to search stock images, which is really interesting.

Right, So that was something we called concept canvas.

So normally when you do a, say an image search, you would, I assume it's just based on text.

You would give the keywords of the things you want to be in the image and it would find the nearest 1 that had those tags.

For many tasks, you really want, you know, to be able to say I want a big person in the middle or in a dog to the right and umbrella above the left because you want to leave space for the text or whatever.

And so Concept Canvas lets you assign spatial regions to the keywords.

Then we've already pre-indexed the images to know where the important concepts are in the picture.

So we then go through that index matching to assets.

Even though it's just another form of search, because you're doing spatial design or layout, it starts to feel like design.

You sort of feel oddly responsible for the image that comes back as if you invented it.

So it's a good example where giving enough control starts to make people have a sense of ownership over the outcome of the event.

And then we also have technologies in Photoshop where you physically can move the dog in post as well.

But for concept canvas it was just a very fast way to sort of loop through and be able to lay things out.

And in terms of being able to remove objects from a scene and fill in the background automatically.

So that's extremely exciting And that's so neural networks are stepping in there.

I just talked this week with Ian Goodfellow.

the GANs for doing that is definitely 1 approach.

Is it as difficult as it looks, again, to take it to a robust product level?

are certain classes of image for which the traditional algorithms like content-aware fill work really well.

Like if you have a naturalistic texture like a gravel path or something, because it's patch-based, it will make up a very plausible looking intermediate thing and fill in the hole.

And then we use some algorithms to sort of smooth out the lighting so you don't see any brightness contrast in that region.

Or you've gradually ramped from 1 from dark to light if it straddles a boundary.

Where it gets complicated is if you have to infer invisible structure behind the person in front.

That really requires a common sense knowledge of the world to know what If I see 3 quarters of a house, do I have a rough sense of what the rest of the house looks like.

If you just fill it in with patches, it can end up sort of doing things that make sense locally, but you look at the global structure and it looks like it's just sort of crumpled or messed up.

And so what GANs and neural nets bring to the table is this common sense learned from the training set.

And the challenge right now is that the generative methods that can make up missing holes using that kind of technology are still only stable at low resolutions.

And so you either need to then go from a low resolution to a high resolution using some other algorithm Or we need to push the state of the art and it's still in research to get to that point Of course if you show it something say it's trained on houses and then you show it an octopus, it's not gonna do a very good job of showing common sense about octopuses.

So again, you were asking about how you know that it's ready for prime time.

You really need a very diverse training set of images.

And ultimately, that may be a case where you put it out there with some guardrails where you might do a detector which looks at the image and estimates its own competence of how well a job could this algorithm do.

So eventually there may be this idea of what we call an ensemble of experts where any particular expert is specialized in certain things and then there's sort of a, Either they vote to say how confident they are about what to do.

Or there's some dispatcher which says, you're good at houses, you're good at trees.

See all Lex Fridman transcripts on Youtube

Gavin Miller: Adobe Research | Lex Fridman Podcast #23