Title: CERTIFYING AN AUTONOMOUS SYSTEM TO COMPLETE TASKS CURRENTLY RESERVED FOR QUALIFIED PILOTS
Author: Donald “Bucket” Costello
Date/Time: Oct 29, 2020 07:00 PM Eastern Time (US and Canada)
Examining Committee:
- Asst. Prof. Huan Xu, Chair/Advisor
- Prof. Adam Porter, Dean’s Representative
- Prof. Jeffrey Herrmann
- Prof. Miao Yu
- Asst. Prof. Sarah Bergbreiter
Abstract: When naval certification officials issue a safety of flight clearance, they are certifying that when the vehicle is used by a qualified pilot they can safety accomplish their mission. The pilot is ultimately responsible for the vehicle. While the naval safety of flight clearance process is an engineering based risk mitigation process, the qualification process for military pilots is largely a trust process. When a commanding officer designates a pilot as being fully qualified, they are placing their trust in the pilot’s decision making abilities during off nominal conditions. The advent of autonomous systems will shift this established paradigm as there will no longer be a human in the loop who is responsible for the vehicle. Yet, a method for certifying an autonomous vehicle to make decisions currently reserved for qualified pilots does not exist. We propose and exercise a methodology for certifying an autonomous system to complete tasks currently reserved for qualified pilots. First, we decompose the steps currently taken by qualified pilots to the basic requirements. We then develop a specification which defines the envelope where a system can exhibit autonomous behavior. Following a formal methods approach to analyzing the specification, we developed a protocol that software developers can use to ensure the vehicle will remain within the clearance envelope when operating autonomously. Second, we analyze flight test data of an autonomous system completing a task currently re-served for qualified pilots while focusing on legacy test and evaluation methods to determine suitability for obtaining a certification. We found that the system could complete the task under controlled conditions. However, when faced with conditions that were not anticipated (situations where a pilot uses their judgment) the vehicle was unable to complete the task. Third, we highlight an issue with the use of onboard sensors to build the situational awareness of an autonomous system. As those sensors degrade, a point exists where the situational awareness provided is insufficient for sound aeronautical decisions. We demonstrate (through modeling and simulation) an objective measure for adequate situational awareness (subjective end) to complete a task currently reserved for qualified pilots.