- Amin Bartram, co-founder of SocialCam and TripleByte, discusses hiring in startups
- Companies should define what they are looking for in an engineer and effectively search for those qualities
- Different companies have different definitions of a "good" engineer, leading to hiring noise
- Smaller companies may prioritize productivity while larger companies may want a mix of skills
- Hiring based on the interviewer's strengths can lead to biased hiring and a skewed company culture
- Credentials and experience (boot camps vs. CS degrees) are not significantly different, experience matters more
- Interviews assess problem-solving ability, but real-world experience is needed for solving larger problems
- TripleByte evaluates programming productivity, academic computer science skills, and debugging abilities in their pre-screening process

- The interview process for engineering positions at tech companies often relies too heavily on brain teaser questions, which can introduce a lot of noise and may not accurately predict a candidate's skills.
- Larger companies tend to rely more on brain teaser questions, while smaller companies may focus more on specific skills and experience.
- Asking relatively easy interview questions can be more predictive of a candidate's success as an engineer than asking harder questions.
- New graduates should focus on solidifying their knowledge of classic computer science concepts and practicing coding under stress.
- Resources such as "Cracking the Coding Interview" and online sites can provide good practice questions for interview preparation.

- Many companies want candidates to be well-prepared and have a strong understanding of the material rather than just potential
- Facebook provides an interview prep class for applicants
- Brain teasers are rare in interviews, practical application of computer science ideas is more common
- Side projects can help with resume screening, but have limited importance in the actual interview
- Companies use crude filters due to overwhelming number of applicants, but are trying to find better ways to measure skill
- Location is not a major factor for big companies, but smaller companies may prefer local candidates
- Different interview methods can work, but consistency is important to reduce noise in the process

- Consistency is key in the hiring process, regardless of the specific questions asked
- Companies often have different preferences when hiring, but it's important to make conscious decisions about what qualities are truly necessary for the role
- Whiteboard coding interviews may not accurately assess a candidate's skills, so using an actual computer may be more effective
- Staying up to date with new technologies is not crucial on day one, but neglecting to keep up with industry changes can have negative consequences in the long run
- Employee retention is often higher when candidates are genuinely interested in the company and its mission
- There are no specific books recommended for conducting interviews, as many common ideas about interviewing are not accurate
- Standardizing the interview process is important to minimize bias and ensure fairness
- Interviewing should not turn into hazing or a demonstration of power, and interviewers should avoid being mean or aggressive towards candidates.

- The speaker discusses the challenges of conducting interviews and dealing with candidates who perform poorly.
- They suggest switching from evaluation mode to teaching mode when a candidate is failing the interview, focusing on explaining the answer rather than eliciting responses.
- The speaker recommends a maximum of 2 interviewers per interviewee to reduce stress and maintain consistency.
- They provide their email address for further inquiries.

Amin Bartram, co-founder of SocialCam and TripleByte, discusses the hiring process in startups. He emphasizes the importance of defining what qualities are desired in an engineer and effectively searching for those qualities. Bartram highlights the hiring noise caused by different definitions of a "good" engineer among companies. He notes that smaller companies may prioritize productivity while larger companies may value a mix of skills. Bartram warns against biased hiring and skewed company culture resulting from hiring based on the interviewer's strengths. He states that credentials and experience (boot camps vs. CS degrees) are not significantly different, with experience being more important. Bartram mentions that interviews assess problem-solving ability but real-world experience is needed for solving larger problems. He explains that TripleByte evaluates programming productivity, academic computer science skills, and debugging abilities in their pre-screening process. Bartram criticizes the reliance on brain teaser questions in engineering interviews, suggesting that asking relatively easy questions can be more predictive of a candidate's success. He advises new graduates to focus on solidifying their knowledge of classic computer science concepts and practicing coding under stress. Bartram recommends resources like "Cracking the Coding Interview" and online sites for interview preparation. He mentions that many companies want candidates to be well-prepared and have a strong understanding of the material. Bartram notes that location is not a major factor for big companies, but smaller companies may prefer local candidates. He emphasizes that consistency is key in the hiring process, regardless of the specific questions asked. Bartram suggests that whiteboard coding interviews may not accurately assess a candidate's skills and using an actual computer may be more effective. He highlights the importance of staying up to date with new technologies to avoid negative consequences in the long run. Bartram mentions that employee retention is higher when candidates are genuinely interested in the company and its mission. He suggests standardizing the interview process to minimize bias and ensure fairness. Bartram advises against hazing or aggressive behavior towards candidates during interviews. He recommends switching to teaching mode when a candidate is failing the interview and focusing on explaining the answer. Bartram recommends a maximum of 2 interviewers per interviewee to reduce stress and maintain consistency. He provides his email address for further inquiries.

Ammon Bartram is the cofounder of Triplebyte (YC S15) and Socialcam (YC W12) - https://twitter.com/ammonbartram

Triplebyte connects software engineers with companies that are hiring: https://triplebyte.com/

Read the transcript here: http://blog.ycombinator.com/hiring-engineers-with-ammon-bartram

YC's podcast is hosted by Craig Cannon - http://twitter.com/craigcannon

Hey guys, today we have Amin Bartram, co-founder of SocialCam, TripleByte, and he is here to talk to us about hiring.

So could you just give us a quick intro about what you've worked on?

So I joined Justin.tv fresh out of school in 2009 when it was just 25 folks and kind of went through the the roller coaster of the early days of Justin TV and There I work mostly on the the video system and I think that was where I got my first taste of hiring.

At the end of that we were hiring pretty aggressively and that was when I first realized how much noise there is in the hiring process.

And Then I was part of the spin-off of SocialChem, and that was a video-sharing app.

I did that for about 3 and a half years, and we were acquired by Autodesk in 2012.

I worked there until 2014, And then took a bit of time off and started a triple byte Cool, and so triple byte just for context for people.

Can you explain sure so yeah, so we're a recruiting startup So we help startups hire engineers,

And so and engineers apply to us, and then we do sort of a full interview with them, you know, past the engineers who are good and match them with companies where they have a high probability of doing well.

So people ask us a million questions about hiring, recruiting, all of it.

In general, let's assume that you're, you know, an early stage startup.

What should companies be looking for in an engineer?

I think The pithiest answer I can give to that is you have to decide what it is you want to look for and then you have to effectively look for that.

So the status quo, actually I think there's an important truth here.

Most companies think that they are trying to hire good engineers.

And what they don't realize is that company A's definition of a good engineer is significantly different from company B's.

And so what you have is a situation where everyone has this definition in mind and they're all different and this is the big source of noise.

So for example, if 1 company thinks that engineers need to be very fast and productive and be able to show that in an interview.

And a different company thinks that engineers need to be, you know, very intellectual and be able to deeply understand computer science topics and talk crisply about that.

What happens is sort of all of the awesomely productive, you know, engineers, very practical, not necessarily strong academics who applied to the second company, fail.

All of the very academic, could totally solve all your hard problems, but maybe aren't quite as flashy in a web environment, engineers who applied to the first company also fail.

And so it's, you know, for companies that are a bit of a larger stage, I think the obvious answer is you want to hire both those people.

And so it's about building a process that can identify more broadly different types of skill.

For smaller companies, I think you're more in a situation where you may well actually only want 1 of those folks.

You may well, you know, the thing that's holding your company back may well be productivity.

And we need someone who's going to come in and be productive and sort of bang out code.

And if that's the case, you need to realize that it's not important that everyone you hire be strong in computer science.

And if you are a company where you're facing security issues and really clean code, really precise code and solving hard problems is important to you at an early stage.

It probably makes sense to have a process that skews more in that direction.

So do you have general advice for people that come to you guys at TripleByte, or just you as a friend advisor, for diagnosing what kind of engineer is a good engineer for your company?

We're rarely in that situation, actually.

I think most people have strong preconceptions.

So we're more often in the situation of sort of broadening people's vision of what a skilled engineer can be.

Yeah, let me circle back to the question.

I think what happens a lot, this is a mistake that's easy to fall into, is when people are interviewing an engineer, they tend to ask about the things that they are the best at.

There's this overlap between the things that you're the best at and the things that you think are the most important.

So every engineer thinks the things that they know are kind of the core of the discipline.

And so you ask everyone who you interview those questions, you bias yourself toward hiring engineers who have those skills, they join your team, they interview the same type of question.

And so the whole organization can grow in a direction that might not make sense.

It's very complicated because there are plenty of examples of companies with defined engineering cultures that have worked out very well.

So for example, Google, intentionally or unintentionally, has grown very much in a computer science direction and that's obviously worked out very well for them.

There are other companies in YC that, I just don't want to say names, but there are companies that take a complete opposite, take a very human productivity friendly approach and are also extremely successful, excellent companies.

And so, answering that question is not, basically there are success cases on both sides.

But I think as, you know, when you're hiring your first employers, I think you need to just basically, you know, try to decide what's holding us back, you know.

And so, yeah, and so say like, Say I'm trying to vet this pool of engineers and they all fit the certain rubric that I've created, right?

But maybe 1 of them did a bootcamp and has some projects and then 1 of them went to a great school, has a CS degree.

So How should I think about credentials and experience?

Boot camps versus CS degree, I don't think are all that different.

Now, okay, that's obviously a forceful statement.

So I think experience matters more than where you got your education.

So someone fresh out of a CS program who doesn't have internships is still essentially a junior engineer.

And perhaps they may have a more academic slant to what they've studied than someone out of a boot camp.

But both those people lack real world experience.

I think categorizing that differently than someone who has worked in the industry for 5 years and can own projects.

So the skill that you can most easily measure in an interview is ability to think through relatively small problems quickly.

That's really what interviews can evaluate.

The skill that you need in an employee is quite different.

The ability to solve large sprawling problems well, but over a long period of time.

And there's obviously a correlation there.

We use interviews as a proxy of evaluating actual skill because there is a correlation there.

And an interesting observation is that people who are fresh out of university and boot camps actually in many cases, because they've been practicing, are better at the kind of problem that gets asked to interview than your very senior 10 years of experience at large company engineer.

What the senior engineer has typically is experience making architecture decisions, owning projects, gathering requirements, carrying that whole process through.

And so, okay, how do we evaluate for that?

Basically, what ends up happening, and this is honestly unfair, is that experience is used as a proxy for that.

This is something that we're focusing a lot at Trailbite.

But what, you know, this is very strange.

Actually, if you are, if you're, if you have 5 years of experience, it's just flat out easier to pass an interview.

You will get a job offer, you know, after a worse performance.

People think maybe that senior engineers, you know, you know, you expect their senior, they should, they should, they should perform better in the interview and that's actually not generally true the bar to getting a job offer goes down as you have sort of a present-looking resume and and This is not this is you know, it's not irrational on the part of the companies.

sure Okay, And so then when you guys are pre-screening these people for TripleByte, for example, what are you looking at?

So the approach that we take is to evaluate as much as we can in isolation and be aware of what we're evaluating.

So we explicitly evaluate programming productivity.

Given a relatively spec'd out problem, so for example, describing an algorithm to solve a problem, it's not super mathy, it's just here's the set of steps that you have to do.

Can the candidate take that and render it into well-working, well-structured code?

And interestingly, junior folks actually often do better than senior folks at that sort of problem.

We then separately do an evaluation of academic computer science skills.

Is the engineer knowledgeable about computer science and about that approach to problem solving?

1 thing that we took from Stripe actually is we do a debugging section.

And so we give the candidate a large code base and that has some bugs and we ask them to sort of, take some time, dig into the code base and then try to find and fix these bugs.

And I think this does a great job of solving some of those problems, basically, because this is a skill that comes from experience that is often missed by more traditional engineers.

And then finally, we do a system design section.

So Here's some requirements to design a production web system to satisfy these requirements.

Okay, now we're going to change the requirements.

And all of that's done remotely because the person's at home, right?

Yes, we do this all over Google Hangouts.

Okay, and so what's a reasonable amount of time for someone to just go through 1 of these exercises?

Our interviews are about 2 hours in length, And so we spend about 30 minutes on each section.

And you find that's like a very strong data set in terms of correlating how successful they are?

Yeah, well, we've done about 2000 of these interviews over the last year and a half.

And so we've been able to sort of drill in on the parts that are most predictive, and cut time off and shorten it.

I think if you're starting from scratch, you would probably need about twice the amount of time to get through all that stuff.

And so what, like, so having gone through all these interviews at this point, was there anything that you thought was really important in the beginning, or something that's like very common in the valley that many people think is important that isn't really important?

So the classic interview format in tech companies is a number of 45 minute to 1 hour sessions.

And often engineers pick the questions themselves.

They're sometimes pejoratively called brain teasers.

They're more like legitimate programming problems.

Like how do you, given a string of parentheses, how do you turn off the well match?

Okay, given multiple types, how do you turn off the well match?

If you don't, that's a classic interview question.

And it ends up, there's just a huge amount of noise.

If you just take a bunch of candidates and in a controlled setting have them all answer 3 or 4 of these questions, you'll see there's some correlation, but there's way less correlation than you would think.

I mean, honestly, I believe this in my previous...

You have this question, you ask them the question, and you see this variation, and you assume, oh, the people who answer this question well must be smart and good programmers, and the people who get totally tripped up must not be and then when you actually like inspect that you see that there's this huge amount of noise and like We have this pretty incredible lens on this this because we evaluate engineers pretty rigorously and then send out to multiple companies and see what happens, get detailed feedback.

And so yeah, is that feedback from the actual interview process or then once they're placed you actually know as well how they're doing?

But I'm currently talking about the interview process.

So we screen engineers, we send them to companies, and then they do the interview there and we get feedback.

And it's just pretty incredible how much disagreement there is.

So a candidate who does really well at 1 company and we get told, oh, this is the best person we've interviewed in months, this is a rock star, goes on to fail an interview at somewhere else.

And a pretty interesting stat we dug up is I compared the rate of agreement between interviewers at companies with a data set of users reviewing movies online.

And the numbers were actually, basically, and the inter-rater agreement was equivalent.

So basically, knowing that an engineer did well at 1 company gives you about as much information about whether that engineer is skilled as knowing whether the New York Times film critic rated 12 Years a Slave as excellent or terrible.

So okay, Maybe you don't have an answer to this, but say I'm really good at brain teasers.

So and this makes a certain amount of sense.

So bigger companies, so brain teasers always introduce noise.

But we've found that bigger companies rely more on that type of interview.

And they do that partly for some rational reasons.

So bigger companies care more about measuring your innate ability and less about measuring whether you can jump into their particular code base and be productive on day 1.

It's way more likely that a smaller company is going to say, we're using Ruby on Rails, we need a very productive Rails developer, come take this interview and we're going to ask you how well you can do this.

Maybe how well you can work on our actual code base.

Whereas the big companies, Facebook, Dropbox, Apple, Google, are more likely to say, we care about smart people and within the confines of the noise of the interview process, they're trying to identify intelligence rather than specific experience.

I mean, they also have like capacity time to train people.

Okay, and then prior, 1 of the questions, what about skills that people don't think is correlated, that are strongly correlated to a successful engineer?

We have found that asking pretty easy interview questions is often more generally predictive than asking harder interview questions.

And so to break this down, there are 2 sources of signal from asking a question.

You can get signal on whether the candidate comes up with the right answer, and you can get signal on whether they struggle, how easy or hard it is for them to solve the problem.

And so We score both these things for a bunch of questions, and we've done this for, again, thousands of candidates.

And what we found is that if you go in and look at how much the individual score on 1 question we're asking correlates with how the candidate does on the job.

And what we've found is that, As you would expect, getting a question right is correlated

And as you would expect, being able to answer a question easily is correlated with being a good engineer.

But there's also, of course, false negatives.

So there are great engineers who fail in real questions and there are great engineers who struggle with questions.

And if you look at the actual predictive ability, rather than just the correlation of getting on the right side, the sweet spot is actually far lower on the scale than most people intuitively think.

And so, yeah, can you give a couple examples of what those easier questions might be?

Just like saying, like, you know, we want you to create a checkers game.

Absolutely no logic, nothing complicated, just a class that has a board, it has a grid, it's got pieces, pieces move around.

This is really a pretty mundane, straightforward task.

That actually ends up being, how well can they do that, ends up being a more stable predictor of engineering skill than sort of, you know, here's a sentence that consists of a list of words all glammed together and you had to find the optimal way to give a dictionary to break this apart into words.

That second problem ends up being a graph search problem that can be optimized with memorization or dynamic programming.

Again, the second problem right carries more information than getting the first problem right, But with a really high false negative rate.

so the first problem ends up actually being a better general predictor of engineering skill

And so is there a way if I'm like getting ready for a new, you know, I'm gonna go to another company I'm gonna get ready to interview Do you recommend people train in any particular way or is because you're going for that sweet spot of easy questions, you just have to be smart enough to do it?

Yeah, if I'm going to prep to do some interviews, what would I do?

1 is where I think, for companies that I think are doing a good job interviewing and then maybe for where I think some of the status quo is.

In general, it depends where you're coming from.

So very different advice for new grads and for experienced folks.

For new grads, I would say make sure you're solid with the classic stuff, so breadth-first search, hash tables, heaps.

A surprising percentage of a new question ends up being slightly obscured applications of those, especially hash tables and breadth-first search.

Those 2 things by themselves represent probably 40% of the questions that are asked in most companies.

But actually, many new grads are already pretty solid on that because they've been being drilled that throughout school.

The second thing is practicing writing code under stress.

So working out a big problem over time is very different than you have 10 minutes or 30 minutes, here's a marker, writing a whiteboard, or even here's a laptop program on it.

And so just the things correlate, The skills correlate, but you can improve your performance by practicing.

So totally put in 30 minutes a day, finding some questions online, and giving yourself a time limit, and trying to solve them under stressful situations.

And are there good resources that people can look for?

So Cracking the Coding Interview has a pretty good list of questions.

It's, the other advice in that book I don't think really applies to startups very much, but the questions are good.

And then there are a bunch of sites online that have lists.

Interview Cake is 1 that I've seen that I think is high quality.

An interesting aside to this though is that most companies actually want you to do these things.

It's not companies would prefer that all their candidates, we prefer, we totally prefer.

We try to design our interview in such a way that there's no impact.

We don't really want to be measuring if you've been cramming on algorithms.

But what companies want to measure is like max scale, max potential.

They would actually much rather see you in a state where you're well prepared in the material rather than a state when you have the potential to understand it but forgot about it.

And an interesting trend that's happening in the industry is companies being more upfront about what they're asking.

So Facebook, for example, has started providing a sort of interview prep class to everyone who applies so that they're sort of going over the material.

It's part of me finds it encouraging, because it is moving it in a better direction.

But that's also discouraging, because it's Really sucky that you have to like take a class

I just wonder if it like if it's filtering for those types of people, right?

We're just like looking for like I don't know what to do.

I don't know what to do It's like well if you apply here you can take the class and then like take it like let me hold your hand the whole way through your life

Okay, So say I am going to interview at a bigger company.

Is there a way to prep to do well with the brain teaser stuff?

So, again, there are some words that are thrown around describing any of your questions that aren't.

Some companies probably have asked things like, you know, the golf balls in a 747, but that's really very rare.

Much more common is application of a computer science idea to a practical problem.

there still is this leap of insight required.

In many cases, I think those are bad interview questions.

You don't like, you know, Companies should try very hard to ask questions where it's not like there's 1 thing that has to be grasped until that's grasped, the problem feels impossible.

But practical application of a computer science topic represents the significant majority of questions at big companies.

And so when I was in college, I interviewed at 1 of those big management consulting firms, and they did have all those questions.

We spent 2 months prepping for it, and I didn't get the job.

And I did okay on the stupid ping pong ball questions.

1 thing, 1 number we have that's interesting is that the engineers who do the best at companies go on to pass about 80% of their interviews, but not 100.

Almost no 1 passes more than 80% of the way to visit companies.

And yeah, so 1 big advice to everyone is just don't feel bad if you fail the interview.

It is really not a referendum on your skill.

I'm very happy to have not gone in that direction.

Are there certain types of side projects that across the board are attractive to companies?

Or is it like, Say I'm applying for a job at Stripe and I did x payment type project and that would be more attractive to them.

So across the board are there things that are interesting?

Let me answer that question and then we'll talk about what I think the right thing for companies to do is.

So companies don't actually pay very much attention to side projects, except for at the screening stage.

So resume screen, candidate applies to a company, company decides if they're going to interview the person at all.

See all Y Combinator transcripts on Youtube

Hiring Engineers with Ammon Bartram