Skip to content
Naked Security Naked Security

Self-driving car dataset missing labels for pedestrians, cyclists

Udacity Dataset 2, used to train thousands of engineers, contained thousands of unlabeled vehicles and hundreds of unlabeled pedestrians.

A popular self-driving car dataset for training machine-learning systems – one that’s used by thousands of students to build an open-source self-driving car – contains critical errors and omissions, including missing labels for hundreds of images of bicyclists and pedestrians.
Machine learning models are only as good as the data on which they’re trained. But when researchers at Roboflow, a firm that writes boilerplate computer vision code, hand-checked the 15,000 images in Udacity Dataset 2, they found problems with 4,986 – that’s 33% – of those images.
From a writeup of Roboflow’s findings, which were published by founder Brad Dwyer on Tuesday:

Amongst these [problematic data] were thousands of unlabeled vehicles, hundreds of unlabeled pedestrians, and dozens of unlabeled cyclists. We also found many instances of phantom annotations, duplicated bounding boxes, and drastically oversized bounding boxes.
Perhaps most egregiously, 217 (1.4%) of the images were completely unlabeled but actually contained cars, trucks, street lights, and/or pedestrians.

Junk in, junk out. In the case of the AI behind self-driving cars, junk data could literally lead to deaths. This is how Dwyer describes how bad/unlabelled data propagates through a machine learning system:

Generally speaking, machine learning models learn by example. You give it a photo, it makes a prediction, and then you nudge it a little bit in the direction that would have made its prediction more ‘right’. Where ‘right’ is defined as the ‘ground truth’, which is what your training data is.
If your training data’s ground truth is wrong, your model still happily learns from it, it’s just learning the wrong things (eg ‘that blob of pixels is *not* a cyclist’ vs ‘that blob of pixels *is* a cyclist’)
Neural networks do an Ok job of performing well despite *some* errors in their training data, but when 1/3 of the ground truth images have issues it’s definitely going to degrade performance.

Self-driving car engineers, please use the fixed dataset

Thanks to the permissive licensing terms of the open-source data, Roboflow has fixed and re-released the Udacity self-driving car dataset in a number of formats. Dwyer is asking those who were training a model on the original dataset to please consider switching to the updated dataset.
Dwyer hasn’t looked into any other self-driving car datasets, so he’s not sure how much bad data is sitting at the base of AI training in this nascent industry. But he has looked at datasets in other domains, finding that Udacity’s Dataset 2 was particularly bad in comparison, he told me:

Of the datasets I’ve looked at in other domains (eg medicine, animals, games), this one stood out as being of particularly poor quality.

Could crappy data quality like this have led to the death of 49-year-old Elaine Herzberg? She was killed by a self-driving car as she walked her bicycle across a street in Tempe, Arizona in March 2018. Uber said that her death was likely caused by a software bug in its self-driving car technology.
Dwyer doesn’t think bad data quality had anything to do with the tragic crash. According to a federal report released in November, the self-driving Uber SUV involved in the crash couldn’t figure out if Herzberg was a jaywalking pedestrian, another vehicle, or a bicycle, and it failed to predict her path’s trajectory. Its braking system wasn’t designed to avoid an imminent collision, the federal report concluded.
I’ve reached out to Vincent Vanhoucke, principal scientist and Director of Robotics at Google, who teaches the Udacity course on becoming a self-driving car engineer, to get his take on the bad data and to find out if he plans to update to the fixed dataset. I’ll update the article if I hear back.
Over the coming weeks, Roboflow will be running some experiments with the original dataset and the fixed dataset to see just how much of a problem the bad data would have been for training various model architectures.
For now, Dwyer’s hoping that Udacity updates the data set it’s feeding self-driving car engineering students and that the companies actually putting cars on the road are more diligent at cleaning up their AI training materials than what this open-source dataset might suggest:

I would hope that the big companies who are actually putting cars on the road are being much more rigorous with their data labeling, cleaning, and verification processes.

UPDATE 20 February 2020: Since this story was first published, Udacity got in touch to reassure us that Roboflow was looking at a dataset that was created years ago, for educational purposes only, to train engineers on how to collaborate on building a self-driving car. Udacity’s project died down three years ago, a spokesperson said, meaning that its self-driving car hasn’t been a menace to anybody on the streets:

Udacity created this dataset years ago as a tool purely for educational purposes, back when self-driving car datasets were very hard to come by, and those learning the skills needed to develop a career in this field lacked adequate training resources. At the time it was helpful to the researchers and engineers who were transitioning into the autonomous vehicle community. In the intervening years, companies like Waymo, nuTonomy, and Voyage have published newer, better datasets intended for real-world scenarios. As a result, our project hasn’t been active for three years.
We make no representations that the dataset is fully labeled or complete. Any attempts to show this educational data set as an actual dataset are both misleading and unhelpful. Udacity’s self-driving car currently operates for educational purposes only on a closed test track. Our car has not operated on public streets for several years, so our car poses no risk to the public.

Latest Naked Security podcast


Click-and-drag on the soundwaves below to skip to any point in the podcast. You can also listen directly on Soundcloud.


Very interesting, so the AI is not so much AI as it is predetermined image recognition and a decision tree. That leaves a lot to be desired.
If you get hit by a self driving car, who is responsible? Program company, programmer, photographer for a bad angle of a photo or a typo on a label, owner, seller?
Will it take a 5 year investigation to find a leaky memory capacitor and shift the blame to a unknown generic component plant, or they put in 5 kinds of bikes but forgot to put in images of a bigwheel? Should it be detecting heat signatures? what about dogs, deer and moose? (Moose is a common issue where I live, often bigger than a horse, walk out into the highway, usually killing the occupants) Who will go to jail when people are killed, or will the victims be SOL?
People are bad drivers, but machines driving in the chaotic real world without responsibility is worse.


It’ll be until some kind of incidents happened that some of these will be the cases to based on; responsibilities, software/AI, company, legislation, …
Just remember 737MAX, were the software, sensors, pilots, company, …?
A lot to think…


I love technology as both user and developer, especially robotics, but just because you can do something doesn’t mean you should. You can use machine learning all you want, but in the end you still have a machine that is incapable of FEELING empathy. That handicap is the difference between hitting a gray sweater wearing grandma and her walker, or a gray garbage can of about the same size and stature. We can make choices between the lesser of two evils because we feel true empathy, something I doubt a machine will truly ever be capable of. Maybe at some point in the future we will have the technology to make machines that feel empathy, but on that day we have bigger problems – machines that can feel empathy, will likely also feel hate, envy, greed, and the rest all of those emotions from our seedy underbelly.
Thank you, but I’ll keep a human in the driver seat for the time being ;)


Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to get the latest updates in your inbox.
Which categories are you interested in?
You’re now subscribed!