Putting the AI in aviation
Persuading a field to face its future
Michigan Engineering News
Somewhere, in a parallel universe, there is an Ella Atkins who happily does her research on autonomy in aerial vehicles, licenses software to aircraft companies and generally makes airspace a safer place.
In our universe, aerospace hasn’t been ready for her. Just doing her research has been a struggle. But she’s not one to succumb to prevailing winds.
“She is strong-willed in a good way,” said Ed Durfee, a professor of computer science and engineering at U-M, recalling Atkins as a PhD student. “She had direction, she had purpose, she had strong opinions as to how to do things, and she was very good at digging deeper into her rationale. She was very often able to convince us that her idea was worth pursuing.”
From graduate school on, she’s had to keep making her case both in academia and in government. There are changes coming to aerospace whether we like it or not, both in the form of increasingly autonomous airplanes and in the proliferation of unmanned aerial vehicles. Done right, she sees the potential for autonomy to raise the already high standard of safety in the skies.
Still, she thinks about it sometimes, the status she would have had if she had spent her time pursuing the incentives of academia. But in this, too, she’s been ahead of her time: changes are afoot in leading universities to encourage professors to choose paths more like Atkins’.
As one of U-M’s early efforts in this, President Mark Schlissel introduced a new honor in 2017: the President’s Award for National and State Leadership. Atkins was one of two inaugural recipients.
“Professor Atkins has more than a decade of engagement with aerospace committees and posts at the national level, including NASA’s Jet Propulsion Lab and the National Research Council,” Schlissel said.
For Atkins, this engagement was not merely dutiful. It was necessary. She wanted to realize the potential to improve flight safety through autonomy, and she has had to fight on two fronts: regulation and education. In another parallel universe, she might be winning. But in this one, she’s had to throw every ounce of her weight just to nudge aerospace in the right direction.
Takeoff
Like many aerospace engineers, Atkins became interested in planes as a kid. Living near military practice airspace, she saw more than just the run-of-the-mill airliners. She earned her bachelor’s and master’s degrees in aeronautics and astronautics engineering from the Massachusetts Institute of Technology and moved to California, taking a job in industry.
After six years of learning about airplanes, she decided it was time to learn how to fly one, taking lessons from instructor Cindy Rice.
“We flew everywhere ranging from Palm Springs for dinner, to Long Beach – a crowded airport, to various places in the desert to just practice landings,” said Atkins.
At the time, she carried a hard copy of the Federal Aviation Administration (FAA) rulebook FAR/AIM (Federal Aviation Regulations and Aeronautical Information Manual).
“I thought it was like the dictionary. You just look stuff up,” she said. She didn’t yet understand that the world around the FAA was changing, and these rules wouldn’t be able to keep up.
She did, however, recognize her own limitations. As a single pilot in a single-engine aircraft, she had a lot to think about if something went wrong: where to go, how to keep the aircraft from stalling on the way, how to land in spite of the problem. She became interested in how software could help in these situations.
I realized that even somebody who tended to make the right decisions when they flew could get into situations that they couldn’t escape.
Ella Atkins
By this time, in the early 1990s, commercial aircraft had been transitioning from hydraulic controls to electronic fly-by-wire systems. While you’d need a full-on robot to handle hydraulic controls, a computer was already sending the electrical signals on the newer planes. It would just need to be smarter. She wanted to design autonomy that could handle problems as well as ordinary flights.
And she wasn’t happy as a workaday engineer, clocking off at 5 p.m. and leaving the puzzles behind. She wanted to keep thinking and talking about them, and she wanted to be around others who felt the same. So, she applied for a PhD in computer science.
In 1993, she came to U-M to study under Durfee and Kang Shin, also a professor of computer science and engineering. Together, Durfee and Shin co-advised students interested in real-time intelligent systems, with Durfee bringing the AI and Shin focusing on speed.
A year into her studies, Atkins received news that Rice had died, caught by a waterspout off the coast of San Diego while flying a cargo run.
“I realized that even somebody who tended to make the right decisions when they flew could get into situations that they couldn’t escape,” said Atkins.
Atkins’ predecessor under Durfee and Shin, David Musliner, had designed a generic architecture for real-time intelligent control. Her mission was to improve its ability to handle unexpected situations and apply it to aircraft.
“A big chunk of the work Ella did was designing for what happens as capabilities of an aircraft degrade, operating within the limits of the available processing power, sensors and actuators,” said Durfee.
Here, her piloting experience merged well with the needs of real-time systems. If the system has too much to do, it needs to slow the world down. When a pilot needs time to think about how to land missing one engine, or with some of the wing flaps not working, he or she can slow the world down by circling. Or if the world can’t be slowed down, as in a case when an aircraft has no functioning engines, the computer has to quickly decide which ongoing tasks to ignore, instead focusing its limited time and resources on landing safely.
Stalling
Atkins develops software that, unlike conventional autopilot, can overcome problems. Autopilot already handles routine climb, cruise and descent, but it can’t navigate abnormal situations like storms or equipment failures. Faced with a challenge, conventional automation hands control to a human.
The resilient autonomy software developed by Atkins and her colleagues is more self-reliant. It can look at the data from onboard databases, the cloud and its sensors. It can test whether components are behaving as expected and change plans accordingly. But, as with a human, there’s no guarantee that it will make the right decision every time.
The crucial question is will the software make the right decision more often than a trained human pilot? Aviation has been unwilling to ask that question seriously. While we all appreciate that the road is full of bad drivers, we put pilots on a pedestal, said Atkins.
Some pilots, like Tammie Jo Shults, who landed Southwest flight 1380 after an engine exploded and blew a hole in the fuselage, remain composed and skillful in the face of disaster. But many black box recordings also reveal panic and confusion in the cockpit before crashes.
“Those people are not going to save the day. In fact, autonomy needs to save them from their airplane,” said Atkins.
Airplane manufacturers don’t have much in the way of incentives for improving safety. In the U.S., Atkins says, juries are likely to punish an aircraft manufacturer if an older plane crashes because it didn’t have a feature that its new planes have. So if they add the new feature, they also need to consider the cost of retrofitting their older planes that are still in service.
She is encouraged to see Airbus pushing automation with technologies akin to driver assist, but nothing like the emergency landing planning software that she and colleagues at NASA (the National Aeronautics and Space Administration) have developed is currently available on commercial airplanes. However, emergency landing aids may soon crack the amateur pilot market with the rise of tablet apps for flying. And emergency landing software will be critical to safe, autonomous drones.
Pilotless flight
While Atkins was originally inspired to improve the safety of flights with humans onboard, her thesis opened a new avenue. During her dissertation research, she got hooked on remote-controlled flight, and to this day, she continues to use these vehicles to test her research group’s control algorithms.
Late afternoon in the Francois-Xavier Bagnoud Building, or FXB, on May 24, 2018, Atkins’ students gathered in the atrium. They cordoned off a space about fifteen feet square, where they tested how well their quadcopter can return to a place where it has been programmed to hover. They exchanged wisecracks and quips while calibrating the equipment and taking their places – two at the computers, two holding tethers connected to the quadcopter, and one on the remote control.
The quadcopter knew where it was with the help of four cameras on stands, about ten feet up in the air. GPS isn’t accurate enough for what they plan to do with the drones: fly five information, carrying power tools for a roofing experiment later in the summer.
“If a small quadcopter falls, it usually won’t send anyone to the hospital,” said Atkins, explaining why they’re choosing the complications of formation flight. A single drone big enough to carry a nailgun could.
Even flying a relatively small quadcopter, with rotors about six inches across, the students were safety-conscious, following a protocol of calls and responses.
Then, Owen Marshall, holding one of the tethers, gave the quadcopter a tug. Displaced a foot or two, the quadcopter brought itself back to where Mathew Romano, at the controls, had placed it.
“Aw, yeah, it works!” Romano crowed.
Atkins called for a tougher test, and Marshall put his body into it, yanking the copter about four feet while Mia Stevens stepped in to keep slack in her tether.
“You can protect your partner, but you can kill yourself,” Romano observed. Yet the quadcopter stopped heading toward Marshall as soon as he stopped pulling. It wasn’t in a rush, but it made its way back to the spot where Romano had stationed it.
Indoors after hours, over farmers’ fields or in military airspace, Atkins is used to testing flight algorithms wherever she can. When she started out, she followed the rules for model aircraft hobbyists. She and her students flew at designated airfields or over the agricultural fields at the University of Maryland, near Washington DC, where she started as an assistant professor in 1999.
Then 9/11 happened, and regulation of aircraft near Washington DC tightened. She could no longer fly her Cessna from the grass strip runway in her backyard without special permission from air traffic control.
“Between the College Park airport closing and our peaceful grass strip runway no longer being a simple place to punch a hole in the sky, I recognized that aviation was changing,” she said.
Model aircraft were also suspect, as drones were already established military weapons. Even now, Atkins prefers to use other terms, like unmanned aircraft system (UAS). She and her students had to drive about an hour away from the university to run tests. If something broke in the first few minutes, it was a huge waste of time.
So she got out of the DC area. She was hired in as an associate professor at the University of Michigan in aerospace engineering. Soon, she took on the project “Flying Fish”, funded by the Defense Advanced Research Projects Agency (DARPA). The autonomous pontoon boat with a seven-foot wingspan ran its sea trial for DARPA officials off the coast of Monterey, Calif., in 2007.
It was a roaring success – the boat took off and landed 22 times on the undulating surface of the Pacific. Atkins and her colleagues were happy, DARPA was happy, LiveScience did a news story on the experiment, and when the FAA saw the news story, they were…not happy. Apparently, even with land out of range for the battery-powered plane, they needed a Certificate of Waiver or Authorization.
The regulatory trouble was that Atkins is not a hobbyist. She’s a professional. And the FAA has different rules for professionals. In fact, if she hadn’t been a member of a public institution, she wouldn’t have been allowed to fly at all.
“This was the first time that I really stepped back and said, ‘Maybe this book, this dictionary that I have used since I was first learning to fly planes, isn’t quite as irrefutable as I had originally thought,’” said Atkins.
Still, she applied for a waiver and waited. Fall turned to winter, and DARPA wanted more tests in the spring. The FAA hadn’t authorized her. Then Atkins’ program manager at DARPA got in touch, and the waiver came through.
Now people without skill or appreciation of the dangers can pull a drone out of their backpack or car trunk and fly wherever they want to.
Ella Atkins
Atkins stayed on top of the paperwork as best she could, but the rules kept changing. She has flown under three different sets of FAA regulations in the last decade. Sometimes her certifications came through quickly, sometimes they were held up with no explanation.
By the time the FAA came out with a consistent framework for approving UAS flight in 2016, industry had been clamoring for it for years. And now, to Atkins, it all seems too easy.
“The pendulum swung too far,” she said. “All of a sudden, after you pass a multiple choice test, there are no restrictions to stop you as long as you aren’t near an airport. Now people without skill or appreciation of the dangers can pull a drone out of their backpack or car trunk and fly wherever they want to.
“We have some in our lab that can carry a 15 pound movie camera. Unshielded propellers made of carbon fiber – if they fall from the sky, they can kill if they’re over someone. No question in my mind.”
And unlike the earlier days of the hobbyists, when remote pilots had to be reasonably skilled to avoid crashing their planes, autonomy managing the motors has made off-the-shelf quadrotors relatively easy to fly.
Safety net
U-M’s rules on flight are more restrictive. In order to protect humans and still enable convenient outdoor flight testing, U-M built M-Air on North Campus, putting the finishing touches on early this summer.
It is an enclosed space – four stories high, and with about half the footprint of a football field – so it doesn’t matter that they operate near the U-M hospital helipad. They can test without tethers even though there are roads and sidewalks just a few feet away. It’s rigged with lights for night flights, and they can mount tracking systems like the one they used inside FXB. The walls and ceiling of netting expose drones to all manner of weather, but there’s a pavilion to shelter researchers when they run experiments in the rain or under the hot July sun.
Her integrity and characteristic frankness were vital as the committee debated not only safety, but also legal and regulatory issues for the individual and university.
James Ashton-Miller
Atkins and Pete Washabaugh, a professor of aerospace engineering, championed M-Air in part to provide a safe place for students to test out their ideas. James Ashton-Miller, Associate Vice President for Research Policy and Compliance and also the Albert Schultz Collegiate Research Professor & Distinguished Research Scientist in mechanical engineering, helped to shepherd the project toward approval.
Ashton-Miller had this to say of Atkins: “Her integrity and characteristic frankness were vital as the committee debated not only safety, but also legal and regulatory issues for the individual and university.”
Maya Kobersy, Associate General Counsel at U-M, added that Atkins helped to test the site, ensuring that the noise would not pose a problem for the neighboring buildings. This highlights another of Atkins’ virtues: She’s happy to do the legwork needed to get the job done.
Reaching out
If Atkins has learned anything from her challenges with the FAA, it is that exchange among technologists (like herself), operators (like pilots), and regulators is critical. So when the National Academy of Sciences looked for groups of experts to draft recommendations for NASA, she accepted their invitations.
The first report, on NASA’s aviation safety research, was released in 2010. The big problem? NASA technologists were running into the same wall Atkins had: you can develop a great new safety feature, but no company incorporates it into an airplane without demand from customers or pressure from regulators.
The second report, released in 2014 and which Atkins describes as more fun, was an imagining of the future of autonomy in civil aviation.
“It was almost a blank page to allow us to identify critical areas where new research for increasingly autonomous systems should be instigated and followed through to maturity,” said Atkins. “The people around the table ranged from aerospace engineers to computer scientists to human factors experts to pilots and lawyers. It wasn’t a huge group, but the idea was balance.”
They agreed on four top-priority research areas, the two most relatable being the behavior of adaptive and nondeterministic systems (or what is your computer pilot going to do in different situations?) and operation without continuous oversight (or figuring out when you can trust the computer pilot).
Atkins fought hard for the latter. The regulatory assumption has been – and still is – that a human operator must manage an autonomous vehicle. The UAS has to be within line of sight. But for a future of urban air taxis like companies such as Airbus and Uber Elevate propose, or even low-risk scenarios like overnight crop monitoring, we’re going to have to let the computers off the leash.
In addition to raising the importance of human-independent operation, she was able to craft this section of the report.
“If you’re willing to do work – as opposed to just chatting – and the committee agrees with you, you can contribute text directly,” she observed.
And so she disappeared for a few hours, emerged with the draft, the group made some revisions, and that was that. The report has been downloaded more than 4,200 times.
Now she is in the quiet phase of working with the National Research Council, reviewing reports, but if they tap her to lead in the coming years, she intends to step up.
Room for autonomy
Atkins is a firm believer in computer programming as a core competency of modern aerospace engineers. When she arrived at the University of Maryland, they were ready for her. They knew they were hiring a computer science PhD, and they expected her to start computer science courses. And she did: she developed an aerospace computing course for freshmen as well as a flight software course for juniors. Both are still requirements.
Things were different at Michigan. Part of the problem, Atkins later realized, was the generic first-year engineering curriculum at Michigan. A freshman aerospace computing course was impossible, and with only three years’ worth of flexibility, the department was locked into its required courses.
She developed an accelerated computing course for freshmen, but it mostly serves computer science majors. She has succeeded in establishing flight software systems as a technical elective, and she is still pressing to make it a requirement.
Facing a relentless series of regulatory and institutional challenges, it would be easy to become resentful. Yet Atkins recounts it all with humor (some wry), empathy for why the status quo is what it is and gratitude, even, for the challenge of bringing ideas to a table that didn’t always have a place for her. She has an easy smile and enjoys the whimsy that her students bring to the lab, such as the “Disco Machine.” This large octocopter is fitted with bumpers of white plastic tubing for safety – and inside the tubing, her students are adding a string of lights for more spectacular night flights.
It helps that the students who do learn from her recognize that developing increasingly autonomous systems is critical to the future of aviation.
“Professor Atkins’ work builds an essential bridge between computer science and engineering and aviation,” said Pedro Di Donato (MS Robotics ’16, PhD Aero ’17), now a Regulation Specialist in the engineering division of the Brazilian National Civil Aviation Agency. “This bridge between different research fields is fundamental considering that a modern airplane has millions of lines of code. I thank Professor Atkins for expanding my horizons beyond more ‘classical’ aerospace areas such as flight dynamics, structures and controls.”
Swee Balachandran (MSE AERO ’13, PhD ’16), now an engineer at NASA Langley, said, “She instilled in me the idea that algorithms and software architectures can help oversee the execution of complex aerospace systems and provide real-time decisions to make them safer.”
But if there isn’t enough room for autonomy in aerospace, Atkins has found that there is plenty of room for aerospace in robotics. She helped develop Michigan’s robotics program from the beginning, directing graduate studies from the first admissions in 2014.
And perhaps this is also true more broadly. Full autonomy will probably come to small UASs long before it reaches passenger jets. It is the robots operating in airspace that are pushing the boundaries in aerospace. And Atkins, as ever, is on the front lines.
Update since going to print: Incoming chair Anthony Waas on the future of Michigan Aero and the prominent role of autonomy