First demonstration of Mcity's test concept for highly automated vehicles

Highly automated vehicles need a different kind of test to prove they are safe before testing moves to public roads, leaders of Mcity at the University of Michigan said today as they conducted the first demonstration of a protocol they developed to do just that.

The Mcity ABC Test is a concept methodology for an independent and standardized way for automakers and state and federal regulators to validate the safety of Level 4 automated vehicles inside a closed test track, before they’re tested or deployed in the real world. Level 4 vehicles, as defined by SAE, are truly “driverless.” No human interaction is needed, but unlike fully autonomous Level 5 vehicles, they are limited in some way—by speed, weather conditions or location, for example.

Today, such vehicles are in some cases operating on public roads, either during testing or actual use. Several fatal crashes in 2018, including one in a Level 4 AV, illustrate the hazards of those practices. One consequence of these fatalities has been a steep decline in public trust in AVs, which persists today. According to a AAA survey earlier this year, only 14% of drivers would trust riding in a vehicle that drives itself.

Standardized safety testing protocols are essential to realize the benefits of AVs and increase public trust, said Greg McGuire, associate director of Mcity.

“The very existence of autonomous vehicles is based on the promise to reduce roadway deaths and injuries, while also saving energy and increasing access to transportation,” McGuire said. “The 2018 fatalities were a jolt to the industry, and a core question we’re facing right now is: How do you prove an AV is safe enough to operate on public roads? The Mcity ABC Test could serve as a blueprint to address this challenge.”

Today, Mcity leaders publicly unveiled key components of the ABC Test in a remote demonstration for transportation industry representatives and policymakers. Attendees saw how the protocol would work for seven of the 50 common crash-causing situations Mcity compiled into a library of “behavior competence scenarios” AVs should demonstrate they can safely navigate before testing or deploying on public roads. This comprises the “B” in “Mcity ABC Test.”

The “A” stands for accelerated evaluation, as the approach speeds up testing by concentrating on the most common risky driving situations. And the “C” stands for corner-case testing, which focuses on situations that push the limits of automated vehicle performance and technology. Any of the 50 behavioral competence scenarios can include accelerated testing or corner cases.

“We understand that not all 50 scenarios should be tested for every autonomous vehicle,” said Huei Peng, Mcity director and professor of mechanical engineering. “In fact, the scenarios and test cases should be selected based on where and how an AV will operate.”

The seven scenarios presented were: merging onto a highway, merging into a roundabout, one vehicle cutting in front of another on a highway, a vehicle door opening in the path of another vehicle, a deer in the path of a vehicle coming around a blind curve, an unprotected left turn, and a pedestrian crossing an intersection at a crosswalk without following traffic signals.

In an example during the pedestrian scenario demonstration, the vehicle does not hit the pedestrian, but it comes close.

“Let’s talk about the concept of scoring,” Peng said. “After every test, we must judge whether the vehicle passes the test or not. And being safe is not enough. For example, a vehicle might avoid a crash by applying hard braking, but that could cause trouble for the car behind it. So even if there’s no crash, there may be several reasons we don’t believe it is a perfect execution of the scenario.”

In addition to safety, Peng recommends scoring based on several other considerations, including:

Efficiency, meaning the vehicle drives fast enough;
Compliance, meaning it follows traffic laws;
Path tracking, meaning the vehicle refrains from weaving; and
Smoothness, meaning the vehicle steers appropriately and does not unnecessarily brake hard.

“We fully believe that AVs can make mobility safer, cleaner, and more equitable,” Peng said. “But in order to realize their full potential as soon as possible, we need to establish and maintain public trust and confidence. A standardized testing protocol can do that in an effective and transparent way.”

White paper: Conducting the Mcity ABC Test: A Testing Method for Highly Automated Vehicles (April 2020)

White paper (pdf): Mcity ABC Test: A Concept to Assess the Safety Performance of Highly Automated Vehicles (January 2019)

First demonstration of Mcity’s test concept for highly automated vehicles