Are AI referees ruining sports? Or making them better?
Check out Storyblocks and sign up here for great footage: https://storyblocks.com/cleoabram 

Subscribe to support optimistic tech content: https://www.youtube.com/cleoabram?sub_confirmation=1

If you’re watching the 2023 US Open, you’re seeing automated referee calls. After years of controversy about human accuracy, tennis has used a system called Hawkeye to not only check human calls… but now to replace human referees. And this robot ref revolution isn’t just happening in tennis. It’s taking over the sports world. Soccer, basketball, baseball, they are all using machines to improve the calls being made on the field. The World Cup in particular has invested in these tools, from VAR (video assistant referee) to semi-automated offside and goal-line technology.

The stakes are high. The difference between a right or wrong call can be a championship, or millions of dollars in prizes, endorsements, ticket sales. For the most part, fans like tech that makes calls more objective. HOWEVER… there is a big chunk of sports fans who would strongly disagree, who think that robot refs are ruining the game. 

Thing is, this tech isn’t going back in the bottle! Stadiums are now being decked out with dozens of cameras that can track balls, people, and even use AI to make calls so accurate that one day human refs might be a thing of the past…

Chapters: 
00:00 How tennis started a robot ref takeover
00:50 Soccer, basketball, and baseball automating refs
01:53 High tech World Cup balls
02:17 Semi-automated offsides technology
02:58 Video assistant referee (VAR)
04:25 Why some fans HATE robot referees
05:17 Do these tools remove human nuance?
06:27 Why some fans LOVE robot referees
07:02 How Hawkeye works in tennis
07:40 NBA and MLB announcements
08:14 Why these tools are huge if true

I talked about this episode with @mkbhd on the @Waveform podcast here: https://youtu.be/fmIvzU115y0?si=4FbSeNZFU4sAOBER

You can find me on TikTok here for short, fun tech explainers: https://www.tiktok.com/@cleoabram 
You can find me on Instagram here for more personal stories: https://www.instagram.com/cleoabram 
You can find me on Twitter here for thoughts, threads and curated news: https://twitter.com/cleoabram 

Bio:
Cleo Abram is an Emmy-nominated independent video journalist. On her show, Huge If True, Cleo explores complex technology topics with rigor and optimism, helping her audience understand the world around them and see positive futures they can help build. Before going independent, Cleo was a video producer for Vox. She wrote and directed the Coding and Diamonds episodes of Vox’s Netflix show, Explained. She produced videos for Vox’s popular YouTube channel, was the host and senior producer of Vox’s first ever daily show, Answered, and was co-host and producer of Vox’s YouTube Originals show, Glad You Asked. 

Additional reading and watching:
- Semi-automated offside technology, FIFA: https://www.fifa.com/technical/football-technology/football-technologies-and-innovations-at-the-fifa-world-cup-2022/semi-automated-offside-technology 
- Video assistant referee, FIFA: https://www.fifa.com/technical/football-technology/football-technologies-and-innovations-at-the-fifa-world-cup-2022/video-assistant-referee-var 
- The TRUTH behind Hawk Eye Accuracy, by Cult Tennis: https://youtu.be/ot3GGJ3tWbw?si=0XJTbjkEFODD-aZV
- How the World Cup’s AI instant replay works, by Vox: https://youtu.be/C164kYMGV1A?si=ataWZyJ2jsWYf4Ql
- Breaking down Goal-Line Technology, by FIFA: https://youtu.be/m17ERc-kQhQ?si=R80YH_7U81n8I_SX
- Automated Line Calls Will Replace Human Judges at U.S. Open, New York Times: https://www.nytimes.com/2020/08/03/sports/tennis/us-open-hawkeye-line-judges.html
- At the U.S. Open, line judges are out. Automated calls are in., NPR: https://www.npr.org/2022/09/10/1121801484/us-open-tennis-human-line-judges-replaced 
- Hawk-Eye Innovations and MLB Introduce Next-Gen Baseball Tracking and Analytics Platform: https://pro.sony/ue_US/press/hawk-eye-mlb-tracking 
- NBA to use Hawk-Eye tracking system to follow players and ball, ESPN: https://www.espn.com/nba/story/_/id/35818363/nba-use-hawk-eye-tracking-system-follow-players-ball 
- The World Cup’s new high tech balls: https://www.youtube.com/shorts/kTNoXCP77Fo 
- Worst line calls and umpire decisions in Tennis: https://youtu.be/IsyXG8Xhcsg?si=CRSXqFf_m9JseWCB 

Vox: https://www.vox.com/authors/cleo-abram
IMDb: https://www.imdb.com/name/nm10108242/ 

Gear I use: 
Camera: Sony A7SIII
Lens: Sony 16–35 mm F2.8 GM and 35mm prime
Audio: Sennheiser SK AVX

Music: Musicbed

Follow along for more episodes of Huge If True: https://www.youtube.com/cleoabram?sub_confirmation=1

—
Welcome to the joke down low:

What do you serve but never eat?

… Tennis balls.

Use the word “serve” in a comment to let me know you’re a real one who read to the end :)

Okay, so it's 2004 and Serena Williams is playing Jennifer Capriati in the US Open.

What you're about to see is a referee call so bad, it changed a sport forever.

So she hits the ball, it bounces just inside the line, the crowd cheers, and then suddenly the umpire calls it out.

You can clearly see on this review, the ball is in.

But the umpire wasn't using that review system.

The system is called Hawkeye, and you can hear the commentators begging them to use it.

Within just a year from this controversial game, Hawkeye was added as an official part of the US Open review process.

And Today, it's replaced over 200 of their human judges.

And this whole robot-ref revolution, it isn't just happening in tennis.

Soccer, basketball, baseball, they are all using machines to try to improve the calls being made on the field.

The difference between a right and a wrong call could be a championship, or millions of dollars in prizes or endorsements or ticket sales.

And for the most part, fans like tech that makes calls more objective.

However, there is a big chunk of sports fans who would strongly disagree, who think that this tech is ruining the game, who believe that human error is key to sports.

The thing is, though, this tech isn't going back in the bottle.

Stadiums are now being decked out with dozens of cameras that can track people and balls and even use AI to make calls so accurate that 1 day, human refs might be a thing of the past.

Footballers and coaches have complained about referees and lack of accuracy.

I don't think people are enjoying football as much.

Incredibly frustrated with the Hawkeye system.

The receiver did not maintain possession.

This is a ball from the most recent men's World Cup.

It looks pretty normal, but inside it's anything but.

For the first time ever, hidden inside this ball are special sensors that collect positioning data in real time.

The sensors are hung on a suspension system, so the ball doesn't actually feel any different to the players.

All the while, it's sending data 500 times a second back to a room filled with refs you never see.

That room also receives data from 12 cameras mounted around the stadium that track 29 data points for each player on the field, and can automatically alert the refs the moment a player is off sides, here meaning past the last defender.

They called this the new semi-automated offsides technology, and it uses that tracking data and AI to build a virtual 3D playing field, so the refs can see the play from basically any angle.

This is the cutting edge of live sports tracking tech, And it was featured in the World Cup with lots of hype.

In this system, there are still human refs involved, but they only need to validate the calls being made by these robots.

A ton of amazing engineering and money went into this system.

And it built upon the video assistant ref system, which requires refs to draw their own lines while looking at multiple camera angles and lots of slow-mo footage.

Like in the recent World Cup match between the US and Sweden, the video review system decided the game.

The ruling was too close to call with the human eye.

With the help of the automated system, you can see that the ball just barely makes it fully across the line.

These robot ref systems are clearly helping make better calls and taking more human error out of the game.

And while many see that as a good thing, some fans hate it.

But Before we get to that, if you watch my show, Huge If True, you've seen Storyblocks footage, and they're sponsoring this video.

All of these shots are from Storyblocks, and all of these shots as well.

I just go to storyblocks.com slash Cleo Abram.

With my subscription, I can download as many clips as I want and they're all royalty free.

There's even a Creative Cloud plugin so I can access all of the templates and the audio and the footage from my editing software.

If you want to check it out, you can go to the link in my description or storyblocks.com slash Cleo Abram.

Now, back to why some fans really hate robot refs.

Why wouldn't you want a system that makes the right call every time?

Because while people do voice concerns with specific calls with this automated tech, it's not accurate doesn't seem to be the main complaint.

Do people just like fighting with the ref?

I get the fun of getting all riled up, but I also just don't think that I could look an athlete that I admire in the face and say, we think this tech will judge you more fairly, but we don't want to use it because injustice for you is entertainment for me.

And besides, if I think a call is wrong, I can totally yell at a machine.

It turns out, though, that wanting to argue with the ref isn't actually the biggest issue here either.

The main complaint is something that I kind of actually agree with.

AI-powered robot ref systems don't understand the natural gray areas of sports.

To really see this in action, watch this clip.

In the second half of the game, Leverkusen scores.

But the play gets flagged by the semi-automated system because an inch of his toe was over the line.

So the goal gets disallowed and the other team wins the game.

The player probably reasonably believed that he was on sides.

With a more human system, we had a natural buffer, because we couldn't be more accurate.

Now that we can, we have to decide if we actually want to be, or if we actually like the sort of human nuance.

You totally could, in theory, build the buffer into an automated system.

But you'd have to admit that you want it first.

This is where it gets complicated, because at the start of this story I was like, yeah, 100% accurate, that's the way to go, and then you give me 1 example and all of a sudden I'm feeling conflicted.

But on the other hand, not having precise, consistent calls affects the game as well.

Remember that tennis match with Serena Williams?

Here's what she had to say about it years later.

The reason Hawkeye became like a thing was because they were calling my balls out and they weren't even close to the line and it became impossible to play.

We ask athletes to spend their whole lives training to perform better than anybody else and then give their all in front of us in every game.

I think we owe it to them to use the tools that we can to judge them fairly, and to try to do better at that all the time.

Tennis is a good example of that effort going well.

Today, The Hawkeye system at the US Open tracks the ball by using 12 cameras capturing 340 frames every second.

For each frame, it identifies the ball, triangulates its position, and creates a combined model of exactly where it's been to predict where it's going.

Hawkeye boasts a margin of error less than 2 millimeters.

The best study that I could find comparing human refs showed that the margin of error for line judges is around 40 millimeters.

In tennis, it's gotten to the point where championships not using Hawkeye to the fullest are the ones becoming controversial instead of the other way around.

Like, I really like seeing the Hawkeye replay and trying to figure out if I was right.

And while tennis and soccer have really helped pioneer some of these new systems, a lot of other sports are coming on board as well.

The NBA just announced that it's gonna use Hawkeye tech to track the ball and the players in upcoming basketball games.

Other sports like minor league baseball are already using it to let players challenge human calls.

It's clear that we're entering a new world of refereeing.

With these tracking tools, what we're really doing is creating super accurate models of the world.

We can understand what's going on much better than we can with just our eyes.

That superpower brings with it questions that extend far beyond sports.

Like, how much better does a technology need to be than humans for us to like it?

That's a question that we're exploring with technologies like self-driving cars and AI medical diagnoses.

Or how precisely do we actually want our rules enforced?

Or when do we actually want human judgment and why?

Especially if we can measure that it's less reliable.

But for all of the talk about tech replacing humans, what struck me most about this story was that the cutting-edge tech in sports is actually going into just measuring humans.

We're not interested in replacing the athletes with robots.

I mean, a robot basketball game sounds fun, but it's no NBA.

Sports are fundamentally about pushing forward what humans can do.

And all of this tech, it's just a new way to appreciate that.


Speaker 0: Okay, so it's 2004 and Serena Williams is playing Jennifer Capriati in the US Open. What you're about to see is a referee call so bad, it changed a sport forever. So she hits the ball, it bounces just inside the line, the crowd cheers, and then suddenly the umpire calls it out. What? No way.

It was way in. The ball was so in! That's 1 of the worst calls I've seen. And Serena's right. You can clearly see on this review, the ball is in.

But the umpire wasn't using that review system. Only the TV stations could see it. The system is called Hawkeye, and you can hear the commentators begging them to use it. Oh no. Hawkeye please.

Within just a year from this controversial game, Hawkeye was added as an official part of the US Open review process. And Today, it's replaced over 200 of their human judges. And this whole robot-ref revolution, it isn't just happening in tennis. It's taking over the sports world. Soccer, basketball, baseball, they are all using machines to try to improve the calls being made on the field.

The stakes here are high. The difference between a right and a wrong call could be a championship, or millions of dollars in prizes or endorsements or ticket sales. And for the most part, fans like tech that makes calls more objective. However, there is a big chunk of sports fans who would strongly disagree, who think that this tech is ruining the game, who believe that human error is key to sports. The thing is, though, this tech isn't going back in the bottle.

Stadiums are now being decked out with dozens of cameras that can track people and balls and even use AI to make calls so accurate that 1 day, human refs might be a thing of the past. Footballers and coaches have complained about referees and lack of accuracy. The fans hate it. I don't think people are enjoying football as much. Incredibly frustrated with the Hawkeye system.

You cannot be serious! The receiver did not maintain possession. Could Hawkeye technology replace humans? Okay, look at this. This is a ball from the most recent men's World Cup.

It looks pretty normal, but inside it's anything but. For the first time ever, hidden inside this ball are special sensors that collect positioning data in real time. The sensors are hung on a suspension system, so the ball doesn't actually feel any different to the players. All the while, it's sending data 500 times a second back to a room filled with refs you never see. That room also receives data from 12 cameras mounted around the stadium that track 29 data points for each player on the field, and can automatically alert the refs the moment a player is off sides, here meaning past the last defender.

They called this the new semi-automated offsides technology, and it uses that tracking data and AI to build a virtual 3D playing field, so the refs can see the play from basically any angle. This is the cutting edge of live sports tracking tech, And it was featured in the World Cup with lots of hype. In this system, there are still human refs involved, but they only need to validate the calls being made by these robots. That's why they call it semi-automated. A ton of amazing engineering and money went into this system.

And it built upon the video assistant ref system, which requires refs to draw their own lines while looking at multiple camera angles and lots of slow-mo footage. These systems work surprisingly well. Like in the recent World Cup match between the US and Sweden, the video review system decided the game. Just watch this. The ruling was too close to call with the human eye.

With the help of the automated system, you can see that the ball just barely makes it fully across the line. And Sweden won. These robot ref systems are clearly helping make better calls and taking more human error out of the game. And while many see that as a good thing, some fans hate it. But Before we get to that, if you watch my show, Huge If True, you've seen Storyblocks footage, and they're sponsoring this video.

Let me show you. All of these shots are from Storyblocks, and all of these shots as well. Here's how this works. I just go to storyblocks.com slash Cleo Abram. I type in whatever I wanna show.

So for example, meteor shower, Ta-da! With my subscription, I can download as many clips as I want and they're all royalty free. There's even a Creative Cloud plugin so I can access all of the templates and the audio and the footage from my editing software. I use Adobe Premiere. I think you might really like them.

If you want to check it out, you can go to the link in my description or storyblocks.com slash Cleo Abram. Now, back to why some fans really hate robot refs. I was surprised by all of the hate here. Why wouldn't you want a system that makes the right call every time? Because while people do voice concerns with specific calls with this automated tech, it's not accurate doesn't seem to be the main complaint.

Do people just like fighting with the ref? Well, maybe. I get the fun of getting all riled up, but I also just don't think that I could look an athlete that I admire in the face and say, we think this tech will judge you more fairly, but we don't want to use it because injustice for you is entertainment for me. I don't think that's right. And besides, if I think a call is wrong, I can totally yell at a machine.

We're a f***ing embarrassment, mate. That's some rubbish! Rubbish! It turns out, though, that wanting to argue with the ref isn't actually the biggest issue here either. The main complaint is something that I kind of actually agree with.

AI-powered robot ref systems don't understand the natural gray areas of sports. To really see this in action, watch this clip. This is Leverkusen... Leverkusen? Let me check that.

This is Leverkusen... Leverkusen. Leverkusen. This is Leverkusen versus Club Brugge. Club Brugge.

Yeah, I got that 1 right. In the second half of the game, Leverkusen scores. Oh, it's all the way in! But the play gets flagged by the semi-automated system because an inch of his toe was over the line. Like, just look at it.

You can hardly see this. So the goal gets disallowed and the other team wins the game. I get why people dislike this. The player probably reasonably believed that he was on sides. With a more human system, we had a natural buffer, because we couldn't be more accurate.

Now that we can, we have to decide if we actually want to be, or if we actually like the sort of human nuance. You totally could, in theory, build the buffer into an automated system. But you'd have to admit that you want it first. This is where it gets complicated, because at the start of this story I was like, yeah, 100% accurate, that's the way to go, and then you give me 1 example and all of a sudden I'm feeling conflicted. But on the other hand, not having precise, consistent calls affects the game as well.

Remember that tennis match with Serena Williams? Here's what she had to say about it years later.

Speaker 1: The reason Hawkeye became like a thing was because they were calling my balls out and they weren't even close to the line and it became impossible to play.

Speaker 0: We ask athletes to spend their whole lives training to perform better than anybody else and then give their all in front of us in every game. I think we owe it to them to use the tools that we can to judge them fairly, and to try to do better at that all the time. Tennis is a good example of that effort going well. Today, The Hawkeye system at the US Open tracks the ball by using 12 cameras capturing 340 frames every second. For each frame, it identifies the ball, triangulates its position, and creates a combined model of exactly where it's been to predict where it's going.

Oh! Hawkeye boasts a margin of error less than 2 millimeters. The best study that I could find comparing human refs showed that the margin of error for line judges is around 40 millimeters. In tennis, it's gotten to the point where championships not using Hawkeye to the fullest are the ones becoming controversial instead of the other way around. And besides, it's just fun.

Like, I really like seeing the Hawkeye replay and trying to figure out if I was right. And while tennis and soccer have really helped pioneer some of these new systems, a lot of other sports are coming on board as well. The NBA just announced that it's gonna use Hawkeye tech to track the ball and the players in upcoming basketball games. Other sports like minor league baseball are already using it to let players challenge human calls. It's clear that we're entering a new world of refereeing.

And a new world of being a fan, too. With these tracking tools, what we're really doing is creating super accurate models of the world. And those models give us a superpower. We can understand what's going on much better than we can with just our eyes. That superpower brings with it questions that extend far beyond sports.

Like, how much better does a technology need to be than humans for us to like it? That's a question that we're exploring with technologies like self-driving cars and AI medical diagnoses. Or how precisely do we actually want our rules enforced? Or when do we actually want human judgment and why? Especially if we can measure that it's less reliable.

But for all of the talk about tech replacing humans, what struck me most about this story was that the cutting-edge tech in sports is actually going into just measuring humans. We're not interested in replacing the athletes with robots. I mean, a robot basketball game sounds fun, but it's no NBA. Sports are fundamentally about pushing forward what humans can do. And all of this tech, it's just a new way to appreciate that.

See all Cleo Abram transcripts on Youtube

Why Human Referees are Getting Replaced