News. Presentation "Ethical Control of Autonomous Unmanned Systems: A Practical Approach" 12 April 2017 at NPS as part of the annual CRUSER Technical Conference (TechCon) 2017.

The backdrop for this work:

• Roboticists tend to build software systems in unique and dissimilar ways, but nevertheless share a common repertoire of directable capabilities.
• A number of philosophers view unmanned systems as inherently uncontrollable, and therefore propose international protocols banning their existence.
• Potential opponents are likely to use such systems as weapons of war regardless, and do not particularly care about ethical command and control.

The gist of our work:

• Releasing uncontrolled robots at sea with potential for lethal force is not permissible under Law of Armed Conflict (LOAC).
• Naval officers must act ethically and maintain supervisory control of such devices even if direct communications might be lost.
• Formal mission orders can describe both tasking and constraints on action that are logically validatable and executable by a wide variety of robots.
• Identification Friend Foe Neutral (IFFN) is much simpler on the ocean than it is on land.
• A feasible path forward exists that allows Navy commanders to similarly task and trust unmanned systems to act - and appropriately avoid acting - just as they might with other trusted human partners.

Abstract.  Autonomous systems can be ethically supervised by humans without constant communications.  Ships have the potential to direct autonomous systems effectively, as trusted partners that do not require constant supervisory control, if robot mission orders clearly complement the mission tasking followed by humans.  Such a capability can make robot operations safer - and Network Optional Warfare achievable - for maritime unmanned systems. one

• Semantic coherence is the primary key for achieving success: consolidating a vast variety of robot dialects into a common set of C2 definitions for task orders and constraints. 
• Efficient messaging is also necessary since maritime systems can easily lose communication links due to long ranges and severe environmental changes. 
• Optical signaling can help unmanned systems avoid revealing presence, even acting as "data mules" when important messages need to be delivered covertly.

Adding constraints such as no-fly zones, time limitations, permission prerequisites etc. to mission orders allows operators to legally and ethically control mobile systems that have the potential for deliberate (or unintentional) lethal force. The following papers present results from many work-years of effort, showing that ethical control can be practically achieved by providing parsable (and ethically validatable) orders to diverse unmanned systems.  Continuing work intends to demonstrate such capabilities in full-fidelity simulations and at-sea experimentation.

Short-form and long-form papers, presentations:

Abstract. Many types of robotic vehicles are increasingly utilized in both civilian and military maritime missions. Some amount of human supervision is typically present in such operations, thereby ensuring appropriate accountability in case of mission accidents or errors. However, there is growing interest in augmenting the degree of independence of such vehicles, up to and including full autonomy. A primary challenge in the face of reduced operator oversight is to maintain full human responsibility for ethical robot behavior.

Informed by decades of direct involvement in both naval operations and unmanned systems research, this work proposes a new mathematical formalism that maintains human accountability at every level of robot mission planning and execution. This formalism is based on extending a fully general model for digital computation, known as a Turing machine. This extension, called a Mission Execution Automaton (MEA), allows communication with one or more "external agents" that interact with the physical world and respond to queries/commands from the MEA while observing human-defined ethical constraints.

An important MEA feature is that it is language independent and results in mission definitions equally well suited to human or robot execution (or any arbitrary combination). Formal description logics are used to enforce mission structure and semantics, provide operator assurance of correct mission definition, and ensure suitability of a mission definition for execution by a specific vehicle, all prior to mission parsing and execution. Computer simulation examples show the value of such a Mission Execution Ontology (MEO).

The flexibility of the MEA formalism is illustrated by application to a prototypical multiphase area search and sample mission. This paper presents an entirely new approach to achieving a practical and fully testable means for ethical mission definition and execution. This work demonstrates that ensuring ethical behavior during mission execution is achievable with current technologies and without requiring artificial intelligence abstractions for high-level mission definition or control.

To learn more: NPS AUV Workbench: Ethical Control of Unmanned Systems provides additional references and related resources for ongoing work. Feedback on these important topics are always welcome.

Executive Summary. One of the most rapidly growing areas of innovative technology adoption involves unmanned systems. The U.S. military’s use of these systems—especially armed unmanned systems—is not only changing the face of modern warfare, but is also altering the process of decision-making in combat operations. These systems are evolving rapidly to deliver enhanced capability to the warfighter and seemed poised to deliver the next “revolution in military affairs.” However, there are increasing concerns regarding the degree of autonomy these systems—especially armed unmanned systems—should have. Until these issues are addressed, military unmanned systems may not reach their full potential.

The Department of Defense has evolved a comprehensive Unmanned Systems Integrated Roadmap that forecasts the evolution of military unmanned systems over the next quarter-century. Concurrently, funding for unmanned systems is predicted to rise year-over-year for the foreseeable future. Indeed, as the DoD has rolled out a “Third Offset Strategy” to evolve new operational concepts and technologies to deal with emerging peer competitors, autonomous systems and artificial intelligence have emerged as key—even critical—components of that strategy. One of the operational and technical challenges of fielding even more capable unmanned systems is the rising cost of military manpower—one of the fastest growing military accounts—and the biggest cost driver in the total ownership cost (TOC) of all military systems. Because of this, the U.S. military has sought to increase the autonomy of unmanned military systems in order to drive down total ownership cost.

As military unmanned systems have become more autonomous, concerns have surfaced regarding a potential “dark side” of having armed unmanned systems make life-or-death decisions. Some of these concerns emerge from popular culture, such as movies like 2001: A Space Odyssey, Her, and Ex Machina. Whether the movies are far-fetched or not isn’t the point, what is important is that the ethical concerns regarding employing armed unmanned systems are being raised in national and international media. While the DoD has issued guidance regarding operator control of autonomous vehicles, rapid advances in artificial intelligence (AI) have exacerbated concerns that the military might lose control of armed autonomous systems. The challenge for autonomous systems designers is to provide the military not with completely autonomous systems, but with systems with augmented intelligence that provides the operator with enhanced warfighting effectiveness.

The DoD can use the experience of the automotive industry and driverless cars to help shape the degree of autonomy in future unmanned systems. As testing of these vehicles has progressed, and as safety and ethical considerations have emerged, carmakers have tempered their zeal to produce completely autonomous vehicles and have looked to produce cars with augmented intelligence to assist the driver. Harnessing AI to provide warfighters with unmanned systems with augmented intelligence—vice fully autonomous systems—may hold the key to overcoming the ethical concerns that currently limit the potential of military unmanned systems.

George Galdorisi works at Space and Naval Warfare Systems Center Pacific (SSCPAC), San Diego, California.  Full paper and slideset are available online. 

This work was presented during the CRUSER Technical Continuum (TechCon) at NPS 11-12 April 2017.  The Consortium for Robotics and Unamanned Systems Education and Research (CRUSER) provides a collaborative environment for the advancement of educational and research endeavors across the Navy and Marine Corps. The Consortium seeks to capitalize efforts, both internal and external to NPS, by facilitating active means of collaboration, providing a portal for information exchange among researchers and educators with collaborative interests, fostering innovation through directed programs of operational experimentation, and supporting the development of an array of educational ventures.

This important essay includes command and control (C2) analytic thinking regarding distributed lethality, decentralized netted fires, adapting mesh networks, and network enabled vice network dependendent.  The author notes "Most of the article’s observations are culminations of several years of analysis by our (NPS) students, faculty, and fellow naval design strategists." 

Growing controversy surrounds the rapid development of artificial intelligence (AI) in weapon systems, with little consideration of intent or the variety of potential risks involved.  The following papers provide significant detail and insight regarding actual legal aspects with respect to International Humanitarian Law (IHL).  Key insights include recognition of the temporal aspect associated with naval missions.  Long time intervals may occur between direction and execution, without frequent communication, but the need for human control remains essential throughout.

• Alan L. Schuller, "At the Crossroads of Control: The Intersection of Artificial Intelligence in Autonomous Weapon Systems with International Humanitarian Law," Harvard National Security Journal, vol. 8 no. 2, 30 May 2017, pp. 379-425.  (online, pdf)

Abstract. Lawyers and scientists have repeatedly expressed a need for practical, substantive guidance on the development of Autonomous Weapons Systems (AWS) consistent with the principles of IHL. Less proximate human control in the context of machine learning poses challenges for IHL compliance, since this technology carries the risk that subjective judgments on lethal decisions could be delegated to artificial intelligence (AI). Lawful employment of such technology depends on whether one can reasonably predict that the AI will comply with IHL in conditions of uncertainty. With this guiding principle, the article proposes clear, objective principles for avoiding unlawful autonomy: the decision to kill may never be functionally delegated to a computer; AWS may be lawfully controlled through programming alone; IHL does not require temporally proximate human interaction with an AWS prior to lethal action; reasonable predictability is only required with respect to IHL compliance; and close attention should be paid to the limitations on both authorities and capabilities of AWS.

• Alan L. Schuller, "Inimical Inceptions of Imminence: A New Approach to Anticipatory Self-Defense Under the Law of Armed Conflict," UCLA Journal of International Law and Foreign Affairs, vol. 18, no. 2, 2014, pp. 161-206.   (online, pdf)

Abstract. The Law of Armed Conflict (LOAC) has historically incorporated the term “imminence” across the bodies of law governing resort to armed force (jus ad bellum) and those which govern during an armed conflict (jus in bello), as an integral part of evaluating the legality of responding to a threat. Since these areas of the LOAC have traditionally been considered separate and distinct, the meaning of imminence within them has likewise been treated as distinguishable. But the modern threat environment, especially following the terrorist attacks of September 11, 2001, has proven that this division of imminence ad bellum and in bello is no longer tenable. Application of the concept of an imminent threat has been incoherent and inconsistent. This Article argues that imminence should be a singular concept that applies logically in any situation and given any threat of armed attack. In making this argument, the Article presents a simple and flexible framework that can be applied by any person or entity even in light of crisis and imperfect information. Finally, it proposes three principles of imminence that can be applied in evaluating the legality of actions in self-defense across the spectrum of armed conflict.

Proper control of remote unmanned systems with weapons capabilities is of course fundamentally important for achieving Network Optional Warfare (NOW) goals, namely naval forces operating with far less communications-related vulnerabilities. 

Current press reports how some industrialists - many themselves producing commercial autonomous vehicles with potential for lethal force - to nevertheless call for outlawing any form of autonomous weapons.  For military forces, the other team doesn't necessarily "read the same memos" or follow the same rules regarding IHL. Perhaps pushing such notions to their logical conclusion:  If AI is outlawed, will only outlaws have AI?

• Jamie Smith, Tech founders warn on ‘killer robots’ as UN talks delayed, Financial Times, 20 August 2017
• Daniel Victor, "Elon Musk and Stephen Hawking Among Hundreds to Urge Ban on Military Robots," NY Times, 27 July 2015 
• (Many similar articles exist.  Note apparently obligatory reference to Terminator movie in most press coverage of this topic.)

A further insight emerged from recent group discussions together at the Stockton Center for the Study of International Law, Naval War College (NWC) in Newport RI.  For at least the past century, the operational effectiveness of Naval forces has improved in direct relation to the ability of ships to communicate and coordinate, both internally and externally.  Thus more-effective supervised teaming of humans with unmanned systems is not "just" a moral or ethical imperative, not "just" superior control of numerous diverse robotic systems, but also better warfighting capabilities for our forces that must operate in harm's way. 

Humans - qualified military professionals - are trained and committed to meet difficult moral, legal and ethical challenges in modern warfare.  Hybrid human-machine approaches are increasingly necessary for successful defense.  Colonel Schuller's important papers clarify International Humanitarian Law (IHL) regarding autonomous lethality, examine key issues relevant to evolving naval operations, and explore the critical thinking behind these fundamental legal principles.

Submitted to IEEE Journal of Oceanic Engineering for forthcoming special issue on Cutting Edge AUV Technology, planned publication early 2018.

Abstract.  Experts and practitioners have worked long and hard towards achieving functionally capable robots. While numerous areas of progress have been achieved, ethical control of unmanned systems meeting legal requirements has been elusive and problematic.  Common conclusions that treat ethical robots as an always-amoral philosophical conundrum requiring undemonstrated morality-based artificial intelligence (AI) are simply not sensible or repeatable. Patterning after successful practice by human teams shows that precise mission definition and task execution using well-defined, syntactically valid vocabularies is a necessary first step.  Addition of operational constraints enables humans to place limits on robot activities, even when operating at a distance under gapped communications.  Semantic validation can then be provided by a Mission Execution Ontology (MEO) to confirm that no logical or legal contradictions are present in mission orders.  Thorough simulation, testing and certification of qualified robot responses are necessary to build human authority and trust when directing ethical robot operations at a distance.  Together these capabilities can provide safeguards for autonomous robots possessing the potential for lethal force. This approach appears to have broad usefulness for both civil and military application of unmanned systems at sea.

Index Terms — autonomous vehicles, robot ethics, Mission Execution Automata (MEA), Mission Execution Ontology (MEO)

Available: publication draft paper and summary flyer.

“Ethical constraints on robot mission execution are possible today.  There is no need to wait for future developments in Artificial Intelligence (AI). It is a moral imperative that ethical constraints in some form be introduced immediately into the software of all robots that are capable of inflicting unintended or deliberate harm to humans or property.”  Robert B. McGhee, April 2016.

Consideration and feedback are welcome. Contact: brutzman@nps.edu