- The interview is with Dr. Werner Vogels, the CTO of Amazon, who has experience with startups and technical topics.
- Dr. Vogels previously worked as an academic and did two startups before joining Amazon.
- He was invited to give a talk at Amazon and was impressed by the massive technology operation and the challenges they faced.
- Amazon was ahead of the curve in terms of technology and operational scale, and Dr. Vogels was offered a job.
- He became the CTO of Amazon six months after joining, with a focus on scaling and inserting academic rigor into the company's approach.
- The role involved addressing performance, measurement, single points of failure, and improving reliability.

- The speaker discusses the process of isolating data centers and the challenges faced during manual processes.
- The investment in hardware and IT staff was a major obstacle for e-commerce companies using the Amazon catalog.
- The speaker mentions the shift with AWS and cloud, where investment goes into hiring relevant people rather than hardware companies.
- The role of a CTO varies depending on the type of company, such as infrastructure manager, technical co-founder, big thinker, or external facing technologist.
- The speaker emphasizes the importance of building independent teams and removing hierarchy for a fast-moving engineering culture.
- The Amazon culture is driven by leadership principles, and hiring is focused on culture fit.
- Small teams of 10-12 people are preferred, with strong ownership over their tasks.
- The role of a CTO is different from a VP of Engineering, with the former focusing on technology and the latter on managing teams.

- The VP of Engineering at Amazon, Michael Lobb, is a well-known writer on effective team management.
- Amazon's move towards a service-oriented architecture led to the creation of AWS.
- Initially, teams were responsible for managing their own databases, leading to a decrease in innovation.
- The introduction of microservices architecture helped to improve scalability and reliability.
- The development of AWS was driven by the need to manage hardware provisioning and capacity.
- Amazon S3 and EC2 were the first services launched, initially targeting internet-scale companies.
- The growth of AWS exceeded expectations, but the belief in its potential was there from the start.
- Amazon encourages both team-driven innovation and significant capital investment for successful projects.

- The speaker emphasizes the need for capital investment in big opportunities for success.
- They discuss the importance of developing new processes and technologies for success.
- The speaker mentions the challenges of evolving and scaling software architecture.
- They highlight the need for additional roles like solution architects and customer support for a successful company.
- The speaker discusses how customer requests and feedback drive the development of new features and services.
- They mention the importance of launching products with a minimum feature set and iterating based on customer usage.
- The speaker talks about the changing landscape of software development and the need to work closely with customers for innovation.
- They mention the concept of "working from the customer backwards" and how it is a key focus at Amazon.
- The speaker emphasizes the importance of building products that solve customer problems rather than focusing solely on technology.

- Amazon uses a process called working backwards to ensure clarity and focus in their projects.
- The process involves writing a press release, answering frequent questions, creating a UX document, and a user manual.
- Amazon avoids using slides in meetings and instead uses six-page narratives to ensure everyone is on the same page.
- Containers have become popular in development, but Amazon is looking towards serverless environments in the future.
- Security should be a top priority for all businesses, and Amazon encourages everyone to become security conscious and protect customer data.

- Development processes need to change to include security as a default part of continuous integration and continuous deployment pipelines.
- Automation tools can be used to continuously test for vulnerabilities and compliance with regulatory requirements.
- Continuous deployment can be better for security as it allows for testing of small code changes.
- Treating AWS as just a data center and not utilizing higher-level services can limit productivity improvements.
- Startups should determine their growth strategy (high growth or sustainable) and build architectures accordingly.
- AWS supports both "mercenaries" (in it for the money) and "missionaries" (in it for the love of the product) approaches to entrepreneurship.

Dr. Werner Vogels, the CTO of Amazon, discusses his experience and the challenges faced by the company in a recent interview. He highlights the importance of scaling, reliability, and academic rigor in technology operations. Dr. Vogels emphasizes the need for independent teams and a fast-moving engineering culture. He also mentions the role of a CTO and the difference from a VP of Engineering. The interview covers the development of AWS and the importance of customer feedback in driving innovation. Amazon's approach involves working backwards, prioritizing customer problem-solving, and focusing on security. The interview concludes with insights on development processes, architecture, and entrepreneurship.

Werner Vogels, CTO of Amazon, joins YC Software Engineer Kyle Corbitt to share his experience and insights on technology and startups.

Learn more at https://startupschool.org.

He is the CTO of Amazon, and of course, has a lot of really exciting experience with that.

And so we're gonna be talking to him today about his experience with Amazon, about his experience with startups, and about lots of technical topics as well that will be relevant to many of us.

So thank you all, and let's give it up for Dr.

So we're going to be talking, of course, about Amazon today and about your role there, but I'd like to start with a little bit of background.

Would you mind telling us about your career before you started at Amazon and sort of what brought you to that point?

So I'm, so yeah, I was an academic before I joined Amazon.

And so I'd been a research scientist at Cornell for 10 years, building very large scale distributed systems.

And as in common in American academia, you're being motivated to do start-ups on the side.

1 of them that already existed when I joined them and they got sold off and were successful to a company called Stratus.

And another company that actually failed.

So before that, so I'm not the typical sort of computer scientist.

I wasn't until 28 when I decided to actually go back to school.

I worked in hospitals before that, radiotherapy in the Dutch Cancer Research Institute, doing radiotherapy in cancer patients.

And at 1 moment I realized that I really hated all these people dying around me.

And so I decided to go do something that had no humans involved whatsoever.

So computer science seemed like a really good thing to go into.

And this is mid-80s, so computer science is nowhere where it is now to today.

So, and from there I went into research because I worked with the kinds of things that I really was interested in.

Got my PhD, worked in Portugal for a number of years in a research institute, and then was invited to come to Cornell.

And then at 1 moment, so what I did do around that time also when I was still at Cornell was actually either consult for large companies like the HPs and the Suns and whatever of this world.

And so at 1 moment, Amazon invites me to come give a talk about sort of some of the material I was working on.

1 glimpse in that kitchen, however, and I realized that this is a massive technology operation.

It's not a retailer, it's a technology company.

And operating at a scale that I had never seen before, definitely not at other companies that I'd consulted for.

And so the challenges that they were faced with from a distributed systems researcher perspective were amazing.

And so I didn't need to think very hard when they offered me a job.

Like today, sort of like the interesting distributed systems problems are kind of like huge companies and maybe before it was academia or was that just that's the way you're having to go?

I think most distribution researchers have become more aware of sort of the kind of scale that these very large companies need to operate at, or not even these very large companies.

I think if you think about, you know, any successful internet company or digital-only company needs to operate at a scale that is unparalleled.

You know, in 2004 when I joined Amazon, I mean, many of you, if you're going to be successful, will be easily operating at that particular scale that Amazon was at in 2004.

But there was no body of work that you could really rely on.

There was no infrastructure that you had to rely on.

So a lot of effort went into basically keeping the lights on, on things that cloud and other technologies now gives you.

And so Amazon had gotten it to 2004 to a scale purely by being practical.

And it's not that there was a book or so that you could read how to build a scalable organisation or a scalable company.

And so everything, Amazon was sort of, in my eyes, 5 to 10 years ahead of the curve, both in terms of the usage of technology, of the development of technology, but also the operational scale.

And especially if you want to be a fast-moving company, it's not like you could look at traditional enterprises because they all sort of suffer from innovators' dilemma and other things are really slow at 1 moment once it becomes successful.

How to build a company that needs to continue to move fast is a whole different story.

And so you may be doing things that, you may be making business trade-offs, for example, the creation of technical debt or the allowing duplication to happen and things like that, that would be out of the question in a traditional enterprise because efficiency is their main goal.

At Amazon, moving fast and innovating fast and having a very long pipeline of experiments.

So you're willing to allow duplication to happen, you're allowed to create technical debt as long as you know that you have to pay it off.

So there's many trailers that Amazon has been willing to make over time that you can't find in a traditional MBA book.

And so in most of these things, Amazon had to develop for themselves, and whether it's technology or whether it's processes or business processes.

It helps, of course, that with Jeff Bezos at the helm, you have someone who's a true visionary and truly understands sort of how the next modern world will look like or should look like.

So A couple of years after you joined, you were appointed the CTO of Amazon.

Someone asks you, do you want to be the CTO?

You think, he's not going to ask next year, is he?

No, So Alfie Mullen, who was still at the company by the way, didn't really, he really wanted to be a developer and they had been looking for a replacement for him.

And as I said earlier, sometimes you have gifts that you don't know about until you start doing them.

And so I think I joined in September and I think in January I became a CTO.

So tell me what that was like at Amazon and then sort of what that role has shifted into as we get till today.

So if you've, with an agreement, so 1 of the things that Amazon, so 1 of the reasons for hiring me, and actually a few of my, I brought actually, what was it, I brought 5 of my students, former students with me when I came there, was really to insert more sort of academic rigor into the way that we wanted to approach scaling.

Where Amazon had become really good at scale, but the challenge was that we wanted to do orders of magnitude more.

And in essence, we go from order of magnitude growth, You have to revisit almost everything that you do, whether it's your processes but also your technologies.

And so to really be on a solid footing to do sort of the next 1 or 2 orders of magnitude of growth that were coming, we needed to sort of insert way more rigor into our thinking.

And it was, for example, around performance.

What kind of infrastructure do we need for measurements?

How do you, if you want to be a truly data-driven company that sort of makes data-driven decisions, you need to first of all have the data, but you also need to have a culture around how do you measure, and how do you interpret those measurements.

And I mean, a medium latency of, let's say, 1.2 seconds of your webpage to your customers doesn't say anything.

Just says that 50% of your customers have a worse experience.

And then, from an engineering, so for example, the 99 percentile or the 99.9 percentile is way more important often in those cases.

And then, how do you create an engineering discipline that has control over that 99 percentile such that you can actually start pulling it in, if that's what you want to do.

And then be able to associate that with business decisions.

For example, I mean, the common knowledge is that if you improve latencies of your web pages, conversion goes up.

And so the question then is how much are you willing to spend to drive that conversion up?

You need to have more capacity, you need to do more engineering.

And so when you have a diminishing return on sort of the investments that you have to make that you get that control.

So the whole year we focused really on performance and measurement and things like that.

There was a bit in progress already when I joined.

We did a whole year on focusing on single points of failure.

Really I think in 2004 we were pretty good in terms of reliability.

There were rules around, so we had to use 3 data centers, we had to replicate whatever you did over these 3 data centers, you'd be able to lose a data center, customer should not be impacted, so you could lose 2 data centers, customer can be impacted in terms of latency, but not in functionality.

All of those rules were there, and I think we were pretty good at it, until at 1 moment we decided, so why don't we pull the plug out of 1 of these data centers and see what happens?

And the first time when we did this was not like we made it as a surprise.

No, we gave everybody a heads up that this is what we're going to do.

Turns out all these things that look great on paper didn't really look that great in practice, until the first time when you do it.

Lots of manual processes, manual database failovers, things like that.

And so That's sort of a whole year focusing on that.

And so by the time you do the third or the fourth of these, what we call game days, you actually get to a point where these things become really well automated and that you can actually do them without human intervention.

1 biggest surprise when we did the first time was, and then you bring the other data centre back online.

And then, certainly, that data centre has to sync with the other data centres.

That was a scenario that nobody had thought about.

But until you do these things, you don't know them, and especially at scale.

And then we did a year on efficiency, we did some other things.

So mostly the role of a CTO is to drive really big programs and think about sort of what is the technology that we need for the future to be able to be on solid footing as a business?

Or in the case of Amazon, what are the kind of unique technologies that we've developed that we may be able to turn into a business.

Then things change at 1 moment because in early 2000 we had opened up the catalogue, we put an API on the catalogue, that was popular in those days.

You just put an API on something, see what innovation happens.

And it turns out, lots of great companies were being built.

So you would have access to the catalog, search, shopping cart, and I think 1 more, I forget.

Anyway, so lots of new e-commerce companies were being built based on the Amazon catalog.

They would do comparison shopping and have wonderful new UIs towards the whole site.

But as soon as 1 of those companies became successful, they all started to stutter in the execution.

They all needed to get investment, not because they needed to do the business operations, they all needed to start buying hardware.

And so all this investment was in essence an indirect funding of the American hardware industry.

Now because, no, and that was where all the...

And so all of these companies had a hard time doing that.

They had a hard time getting the investment.

They had a hard time getting the hardware.

Because, you know, you might order 50 servers.

It's not like driving up to Costco and actually getting them in your shopping cart.

And then hire IT people and things like that.

So most of those companies that were initially successful around the Amazon catalog all failed.

And they didn't fail because they didn't have great ideas.

They failed because they couldn't get their IT staff ready or get the money to actually do that.

And I think 1 of the things that has shifted with AWS and cloud is that the investment that you get no longer goes in large numbers to hardware companies.

It goes into hiring more people that are relevant for your company and actually build your products better.

So, then we become, so we started AWS, and we can talk more about that later, but When you become a technology provider, your role as a CTO changes.

I think there's sort of, I wrote a blog post once about this.

In my eyes, there's sort of 4 different types of CFOs, so ask any large company, every large company in the world for CTO will be different, but I think there's sort of 4 large categories.

1 is in enterprises, they're often sort of the infrastructure manager.

Yeah, they'll report to the CIO, manages large amounts of infrastructure.

And then there is sort of the second role, I think is that of what you often see in younger businesses where the CTO is the technical co-founder, the 1 with the technical vision.

I do think that that role is a dangerous 1 because there's so many other things that actually fall into that bucket in terms of VP of engineering teams and things that CTOs may not necessarily be really good at, but we can talk about that later.

This really where you're sort of driving innovation.

If you look at companies like AT&T and Lucent and others, they all had CTOs that had an office of the CTO that would be really building next generation technologies or experimentation there.

And then there's the role of the external facing technologists.

If you're a technology provider to other companies, there's a role for an executive to actually interact at a deep technical level with your customers to understand how are they using my products?

Yeah, and looking for deeper, bigger patterns among your customers.

What are the bigger pain points that they still have?

Not only with our technology, but just in general.

If, often jokily said, so we're in the business of pain management, AWS.

Yeah, so what kind of pain do customers still have that they have to work on that actually don't contribute to the products that they want to build.

And so that's sort of, on 1 hand, helping customers understand things, sort of being a bit of the evangelist for the whole notion of cloud, which definitely, let's say 10 years ago, was much more important than it is now.

But then most importantly, take the feedback from your customers back and start thinking about sort of what new features or products you have to do or what kind of processes we need to change to make sure we serve our customers better.

And so it's a much more customer-focused position than a technology-focused 1.

I'm curious, you were talking earlier about, especially in the early days, There were lots of tasks like the game day you were talking about to make sure that the infrastructure was up for scale.

I'm curious if you've noticed in your time and Amazon as it's grown and the engineering culture has grown so much, are there any tips or tricks you've seen on how to grow an engineering culture and keep it moving quickly, you know, keep it effective?

Well, given that Amazon is in reality my first real job, for a long time I thought that the rest of the world was just like Amazon.

There's a very unique culture at Amazon, I think, that works really well for fast-moving companies, and that is to build your teams as independent as possible, to remove as much hierarchy and structure from your organization as that is possible.

Hierarchy, in my eyes, is totally unnatural.

I mean, you need to have some form of hierarchy maybe for reporting or things like that or for some management pieces.

But, you know, if you look at nature, there's a head monkey and all the other monkeys.

I mean, you may have multiple groups of this, all with their head monkey, but there is...

I mean, if you look at ants and all sorts of other patterns in nature, if you're able to build a self-organizing organization, which means basically that you hire people that really want to be independent, not want to be followers, that actually want to own a piece of the product and actually take ownership on that.

I think that's extremely important when you're a younger business, where you don't need followers, you don't need just coders.

You need people that want to have ownership over the piece that you want.

There is something at Amazon that we call the leadership principles.

There's 14 of them, and basically it's all around culture.

And so it's customer obsession, ownership, dive deep.

If you Type them in your favorite search engine, you'll find them with a deep-dug explanation of all of them.

But that really drives what the Amazon culture is.

And so when we hire, let's say when you bring in someone for an interview, you probably have really figured out whether he or she is a good enough engineer for the phone interviews and things like that.

It's really do you have a real good culture fit because there's nothing worse than a bad hire when it comes to sort of the culture kind of environment because that sort of disrupts your small teams tremendously.

So at Amazon we're a strong believer in very small teams, so 10 to 12 people.

Because with 10 to 12 people, everybody knows of each other what you're doing.

And I mean, by the way, if you run Scrum, if you want to stand in the corridor in the morning with 25 people, that doesn't work.

Yeah, and so you're really focusing on small teams and making sure that they have strong ownership over the pieces that they need to do.

And so, Yeah, 1 of the building teams that have, you know, are independent thinkers, have control, want to have control over their own destiny, I think is important.

I think 1 of the challenges is when you grow as a young business, Often, I mean, the role of a CTO, originally, is probably that he does everything that has to do with tech.

It's basically think about technology, but also manage the teams.

Now, I think managing teams is a completely different discipline.

Now, there's a major difference between what I would call a VP of engineering and a CTO.

A VP of engineering every morning wakes up thinking, do I have the absolute best team?

Is my team in the best position to deliver what they need to deliver?

I'm really thinking about making sure that your engineers are in the right position to deliver the technology, the products that they need to deliver.

So it's a very different job in my eyes than that of a VP of engineering.

Rants, I think he's well-known as sort of the blogger.

If you're a manager, if you think about managing, I think in terms of technology management, he's probably sort of the most well-known modern writer that really thinks deeply about how do you make teams effective?

And so I think that sort of really embodies that.

Michael Lobb, Rants and Respose, I think is the blog.

So you mentioned earlier that in sort of the early days of Amazon, you had those companies integrating with you and they were failing for the infrastructure reasons.

And then, so Is that where AWS came from?

Or tell me, how did the idea for AWS happen?

Well, so internally, we're the Amazon retailer, and AWS as well now, are more or less organized in such independent teams that most of those teams look like startups as well.

Or they have complete control over their own destination.

And believe me, the recommendation engine for books is very different than the recommendation engine for shoes.

And So they have a different team, they have different goals, they have their own innovation agenda, all developed by the team themselves, by the way.

So we'd gone through a number of architectural changes.

So end of the 90s, so Amazon's goal was to get big fast.

And you were basically violating all sorts of architectural principles to get to that point.

That meant that they ended up with some sort of monolith and a massive database infrastructure in the back end that was very brittle and basically couldn't grow anymore from an architectural point of view.

So, but remember that nobody had done this before, so they can't be blamed.

It's not like you're violating the textbook.

And so then Amazon moves to what's called a service-oriented architecture.

That was not a word that existed around that time either.

So carving off pieces of the monolith, bringing them together, the data that they operated on, putting an API on it, and then have a network infrastructure that you can call these what we call services.

And basically that took 2 to 3 years to get that done.

Mostly because there was, again, no history about how do you run services.

And so basically what we were doing, we were swinging the pendulum all the way to the other end.

So in this 1, with the monolith, the whole set of databases were shared resource and basically the constraint.

Yeah, because shared resources are the thing that sort of make you move slower.

What we saw, 1 of the reasons for changing architecture was that we saw that the effectiveness of engineers was dropping off.

Basically, deployments were going slower, new features were getting slower, all these kind of things.

Because the backend databases were managed by a group of DBAs, so database administrators, the DBA cabal.

And to make any schema changes, you had to go through them.

And these guys were responsible for these databases.

So they were as conservative as you can imagine, because the reliability of the site depended on them.

So that saw that we sort of, our effectiveness in innovation was really dropping off.

And so moving to this new architecture, we created all these teams that were going to be independent, that they would have their own data stores, there's no longer a centralized data store, there's no longer a centralized resource, things like that.

Except for that, we made some mistakes there.

So, 2, 3 years later, we carved off all these pieces, the monolith is largely gone, all services.

We made a mistake in the 3 very large data sets, customers, items, that's the catalog, and orders.

See all Y Combinator transcripts on Youtube

A Conversation with Werner Vogels