The race for self-driving cars began in all earnest in 2015 with Tesla aggressively leading the pack at the end of 2016. Tesla’s CEO, Elon Musk, is of the opinion that current hardware is already sufficient to allow Level 5 automation (full automation) as set by SAE International, an engineering association. KPMG in its 2017 Global Automotive Executive Outlook found that 37% of auto executives rated the self-driving trend as extremely important. The initial front-runner for the technology, Google Auto (now Waymo) has changed strategy from developing a car to developing systems and rather partnering with automakers than building its own car. Other than Tesla, who develop and test on the go with incremental software upgrades. Google and most other contenders are developing solutions in closed environments while being permitted to test on public roads in States such as California, Michigan, and Illinois. By late 2016 a total of 20 companies has received permits from the California Department of Motor Vehicles (CA DMV).
According to the regulations, each permit holder must annually file a disengagement report, reflecting the number of events where a driver essentially has to take over from the vehicle’s autonomous mode to either prevent a traffic incident or where the system fails. These submissions allow the public who is interested what is happening in the automaker’s self-driving test some insights, outside from what the marketing departments feed the media. Although a total of 20 permits have been granted by the end of 2016, only ten companies have conducted public tests or filed reports for the period. Here follows a summary of the reports filed by the various companies for their testing on Califonia’s public roads for the period December 2015 to November 2016. For comparison purposes, we converted the result to indicate the amount of disengagements per 1,000 miles, a measurement used by Waymo in a recent post by its Head of Self-Driving Technology, Dmitri Dolgov.
[supsystic-tables id=’88’]
Comparing disengagement results per 1,000 miles above shows, bar Tesla’s customers who have clocked over 300 million miles on Autopilot (Level 2 Automation) as at November 2016, it is clear that Waymo is aggressively accelerating its learning and showing the results for it compared to the other brands with projects on Californian roads. One must, however, take into consideration that comparing the different disengagement reports is not truly comparing apples to apples. Ford’s testing, for instance, was only on the stretch of Interstate 10 between Los Angeles and Arizona. The companies also have different strategies. Tesla aims to have level 5 automation as soon as possible while companies such as Ford aims to have autonomous vehicles for ride-hailing and sharing services only from 2021 onwards, influencing the difficulty grade of testing. To further try and unravel each company’s testing program we delve deeper into their submitted reports below.
READ THE BLOG FOR A COMPLETE BREAKDOWN OF DISENGAMENTS REPORTS
BMW Autonomous Disengagement report
[supsystic-tables id=’90’]
BMW’s two-page report is not very telling. The company only conducted tests over March and April with one disengagement on Highway 101 where the road markings were too faint. Can we deduce from this that the company only tested on highways and freeways?
Bosch Autonomous Disengagement report
[supsystic-tables id=’91’]
The automotive parts company have conducted ongoing tests with three vehicles from the start of 2016, but in total the tests were only over 983 miles. The company tested on interstates, freeways, state highways, public rural roads or streets and parking facilities. Most of Bosch’s disengagements were on interstates and freeways with only two on rural roads or other streets. Bosch lists disengagements as:
- Due to weather conditions;
- Road surface conditions;
- Construction emergencies;
- Accidents or collisions; and
- Planned test of the technology.
In total, the company classified all its disengagements as “Planned test of Technology.”
Delphi Automotive Autonomous Disengagement report
[supsystic-tables id=’92’]
Delphi Automotive, like its competitor Bosch, have also conducted ongoing test over the 12 month period but put more miles under its belt, completing testing over 3,090 miles. Delphi tested with two 2014 Audi SQ5’s, one black (1583 miles) and one white (1542.3 miles). Delphi’s conducted tests on US 101, Highway 85, Central Expressway I280 in the San Francisco Bay area and streets in Mountain View and Palo Alto. Delphi indicated that its driver always has one hand on the steering wheel, allowing him/her to execute a disengagement in less than a second for the following scenarios:
[supsystic-tables id=99]
Ford Autonomous Disengagement report
[supsystic-tables id=’98’]
For the period Ford had only two identically equipped Ford Fusion Hybrid’s and as mentioned above testing occurred only on Interstate 10 between Los Angeles and Arizona during March 2016. The company announced in December 2016 that it converted 20 Ford Fusion Hybrid Autonomous vehicles for testing in California and Michigan. During the testing, there were only three disengagements, two due to other drivers overtaking at high speed and one due to its self-driving technology malfunctioning.
Honda Disengagement Report
Honda’s report indicates that the company did not test any vehicles on California’s public road, only on a closed circuit, non-public road in Contra Costa County, CA. Honda approached Waymo in December 2016 to explore a partnership similar to Waymo’s successful partnership with Chrysler on the Pacifica Minivan entered into in May 2016.
GM Cruise Autonomous Disengagement report
[supsystic-tables id=’93’]
GM tested it’s self-driving technology on San Francisco’s city streets using twenty 2017 model Chevrolet Bolt’s, two 2012 Nissan Leaf and three 2016 Nissan Leafs. The program kicked off slowly with only about 2,000 miles done in the Nissan Leafs during the first eight months of the period with around 7,000 miles in the remaining four months done with the new Bolt’s. The bulk of GM’s 181 disengagement events were done to avoid unexpected behavior. Although this resulted in higher incidents than Ford, Nissan or BMW we salute the company for testing in city streets which is ultimately more difficult than testing on interstates and should be taken into consideration when comparing disengagements.
Mercedes Autonomous Disengagement report
[supsystic-tables id=’94’]
Mercedes had only one vehicle doing public testing while the rest of its captured fleet did so on a closed circuit. Mercedes captured automatic disengagements which were technical in nature and manual disengagements where the driver felt uncomfortable. The automaker clearly identified technical issues during the first round of test which they improved in a second round at the end of the year.
Nissan Autonomous Disengagement report
[supsystic-tables id=’95’]
Nissan’s test fleet included three Nissan Leaf’s and two Infinity Q50’s, the automaker’s luxury brand. The company concluded only around 4,000 miles of testing, with over half of it in the first and last month of the test. The automaker claims to have tested on all manner of roads, with 19 incidents on what the classify as Highway and City and ten incidents on what is qualified as City Street and Highway. All of the disengagements were as a result of system failure to identify an incident according to Nissan’s classifications.
[supsystic-tables id=100]
Tesla Autonomous Disengagement report
[supsystic-tables id=’96’]
Tesla’s captured fleet for testing on Califonia’s public road system consisted of four Tesla Model X’s. The automaker only tested in October and for a short distance during the month of November. Tesla’s report is quite secretive and not very telling as it does not classify different causes of disengagement other than marking it as “Follower Output Invalid” or “Planner Input Invalid.” Tesla marked a limited amount of entries as “Health Monitor,” “ACC Cancel” or “Cruise Fault.” Over 90% of the disengagements were clocked on wet roads. Tesla’s conducted its tests on freeway’s, suburban and arterial roads. To make any deductions on Tesla based on its disengagement report would be a mistake as the company’s customers have already clocked over 300 million miles in Autopilot, which is operating at Level 2 Automation. Filing as per the DMV requirements is only for Levels 4 and 5 automation. Tesla also divulged in its report that it’s testing in manny other parts of the world.
[supsystic-tables id=101]
Volkswagen Autonomous Disengagement report
Volkswagen filed report indicate that the company did not do any testing on California‘s public road in the period under consideration.
Waymo Autonomous Disengagement report
[supsystic-tables id=’97’]
Wyamo has been testing its self-driving technology in four US cities for the past seven years and clocked over 2.3 million miles in Autopilot on public roads. The company improved its disengagement level from 0.8 per 1,000 miles in 2015 to 0.2 in 2016. The company also conducts around 3 million miles a day in simulation mode. Waymo’s captured fleet for testing on California’s public roads consisted of 66 vehicles, and testing was mostly in a suburban city environment in and around Mountain View California. Google breaks its disengagements down further by distinguishing as per the DMV requirements between:
[supsystic-tables id=104]
Waymo summarized it’s disengagements by cause as follows:
[supsystic-tables id=102]
Waymo conducted its testing on all manner of public roads:
[supsystic-tables id=103]
UBER’s self-driving program
Uber was told to suspend its testing on San Francisco road’s by the CA DMV as they could not agree on workable rules around Uber’s demand for its technology to be classified as SAE Level 3 automation. It is not the end of Uber’s efforts though as it is piloting retrofitted Volvo XC90 SUV’s and Ford Focus vehicles in Pittsburgh.
Picture – Source Waymo
