When should a car start braking at a stop sign when it’s snowing outside? When should you turn on your turn signal? How can you tell the difference between a traffic light and a full moon?
These are just some of the questions Tesla’s Autopilot team tackles every day.
Tesla’s driver-assistance software relies on a small army of data annotators who review thousands of hours of footage shot by Tesla owners and its in-house test drivers. The annotators create individual 30-second clips, gradually teaching the company’s AI how to behave like a human driver. Tesla employs dozens of annotators full-time, paying them about $20 an hour.
Business Insider interviewed 17 current and former employees of Tesla’s data annotation team, which is split across three Tesla facilities: Buffalo, New York; Palo Alto, California; and Draper, Utah, to understand what it’s like to develop the company’s Autopilot and Full Self-Driving software.
The team’s projects can take anywhere from a few months to a few days, and workflows range from labeling short clips to tracing still images and overlaying satellite data.
“It can get tedious,” one former employee said. “You can end up spending eight hours a day for months just labeling lanes and curbs on thousands of videos.”
“An intimate look into someone’s life”
The clips offer a unique glimpse into the daily lives of Tesla drivers, and at one point, a project called for employees to label data captured by Tesla’s Sentry Mode feature from inside owners’ garages, five employees said.
Another project, called “Selfie,” involved workers labeling data captured by cameras inside Tesla cars, according to two employees who witnessed employees working on the project. Four more employees were aware of the program, the employees said. The Selfie program was designed to teach Tesla’s systems how to identify when a driver isn’t paying attention to the road while using Autopilot, the employees said.
Tesla says that, according to its owner’s manual, the vehicle’s interior camera “shares short interior camera video clips with Tesla to help develop future safety improvements and to continually improve the intelligence of features that rely on the interior camera.” Tesla owners must first opt-in to data sharing before Tesla labelers can access the videos, the company said.
In other cases, workers ended up labelling routes associated with YouTubers and Elon Musk himself, BI previously reported.
“It’s very strange to have such an intimate look into someone’s life,” a current employee said. “It’s strange to see someone’s daily driving, but it’s also an important part of refining and improving the program.”
The videos came from across the U.S., as well as parts of Europe and South America, according to 15 workers. Two workers recalled labeling a video that appeared to have been taken from a customer’s car in Ukraine during the Russian invasion.
Business Insider reached out to Tesla, Musk and his legal team for comment but did not receive a response before publication.
Tesla’s driver assistance software can automatically change lanes and stop at traffic lights. Mark Matousek / Business Insider
Because workers may encounter data from multiple countries in a single workflow, they must stay aware of different local road rules. Tesla seemed to take a more relaxed stance on the rules, seven former and current employees said. For example, some workers were instructed to ignore “no right turns on red” and “no U-turns” signs, which meant not training the system to obey those signs.
“It’s a driver-first mentality,” one former employee said. “I think the idea is we want to train people to drive like humans, not just robots that follow the rules.”
At times, the job also requires labeling videos of accidents or near misses. Seven employees recalled labeling videos that included Tesla accidents or accidents involving nearby vehicles. At one point, four employees said, some employees circulated among themselves a video of a boy on a bicycle being hit by a Tesla. It was one of many videos and memes employees were exchanging, they said.
Reuters first reported on potential privacy issues on the bike clip and annotation site last year. Shortly after the story was published, Tesla began restricting employees’ access to the clips outside of their designated projects and adding watermarks to some videos and images to make it easier to track which employees were sharing images, nine employees told BI.
Tesla’s Employee Monitoring System
Tesla has instituted some pretty strict employee monitoring systems at its Buffalo factory, with 11 workers telling Business Insider that the facility is equipped with numerous security cameras that overlook workspaces.
Employees are closely monitored using two different software systems.
The system, called Humans, measures how much time people spend on each clip, according to four employees. Annotators who consistently spend more time than their allotted time are more likely to receive a lower performance rating or be placed on a performance improvement plan (PIP). The software was originally designed to help U.S. Air Force pilots, according to the company’s website, and it also tracks employees’ eye movements and records audio. It’s unclear, however, whether Tesla uses the software to track its staff’s eye movements.
The company also uses a measurement called “Flide Time” to track how long annotators are active on the labeling software, 17 employees said. The method tracks keystrokes and the time the labeling software is open, but not how long employees spend using other tools on their computers. Depending on the employee level, employees can expect to log between five and seven and a half hours of Flide Time, and must be active on the software for at least that long.
Missing even five minutes from a designated flight time could result in disciplinary action, the six workers said. Missing three flight times within a six-month period could mean being fired, they added.
Some Tesla employees have tried to challenge the company’s employee evaluation standards, with little success.
In February 2023, some workers at Tesla’s Buffalo factory attempted to unionize. A union organizer at the Buffalo plant told Bloomberg that the company tracks workers’ keystrokes and that some workers were tired of being “treated like robots.”
That same month, Tesla fired dozens of workers at its Buffalo factory. At the time, the National Labor Relations Board (NLRB) filed a complaint alleging that Tesla illegally fired some workers in “retaliation for and to stifle union activity.” Tesla denied the allegations, saying the workers were fired for poor performance. The NLRB did not respond to inquiries about the status of the complaints.
When Tesla began building its driver assistance program in 2016, it outsourced data labeling to a California-based company with an office in Kenya, but began running the program in-house in 2019, according to Reuters.
More recently, Tesla’s Autopilot team was hit by company-wide layoffs in April, with Tesla firing about 300 staff in Buffalo, according to a WARN notice.
Tesla says its neural networks will be able to self-learn in the future, but for now they rely on human labor.
The job is crucial to Musk’s vision for the car company.
Tesla’s CEO has repeatedly stressed the importance of Tesla’s work toward self-driving cars over the years: In 2022, Musk said Tesla’s self-driving technology was “the difference between Tesla being worth a ton of money or essentially nothing.”
Tesla is set to unveil an autonomous robotaxi service later this year, which will be built on the same self-driving software and, of course, tedious clip-by-clip analysis by human labelers.
Do you work for Tesla or have information? Please contact us using a non-work email and device. email address Or contact us via our secure messaging app Signal at 248-894-6012.