Designing Autonomous Systems for Warfighters: Keeping Humans in the Loop

Designing Autonomous Systems for Warfighters: Keeping Humans in the Loop

George Galdorisi

Perspective

In his best-selling book, War Made New, military historian Max Boot notes, “My view is that technology sets the parameters of the possible; it creates the potential for a military revolution.”[1] He supports his thesis with historical examples to show how technological-driven “Revolutions in Military Affairs” have transformed warfare and altered the course of history. The U.S. military has embraced a wave of technological change that has constituted a true revolution in the way that war is waged.

One of the most rapidly growing areas of innovative technology adoption involves unmanned systems. In the past decade, the military’s use of unmanned aerial vehicles (UAVs) has increased from only a handful to more than 10,000, while the use of unmanned ground vehicles (UGVs) has exploded from zero to more than 12,000. The use of unmanned surface vehicles (USVs) and unmanned underwater vehicles (UUVs) is also growing, as USVs and UUVs are proving to be increasingly useful for a variety of military applications. The exploding use of unmanned systems (UxS) is already creating strategic, operational, and tactical possibilities that did not exist a decade ago.

The expanding use of armed, unmanned systems is not only changing the face of modern warfare, but is also altering the process of decision-making in combat operations.  Indeed, it has been argued that the rise in drone warfare is changing the way we conceive of and define “warfare" itself. These systems have been used extensively in the conflicts in Iraq and Afghanistan, and will continue to be equally relevant—if not more so—as the United States’ strategic focus shifts towards the Asia-Pacific region and the high-end warfare this strategy requires.

While these unmanned systems are of enormous value today and are evolving to deliver better capabilities to the warfighter, it is their promise for the future that causes the most excitement.  Indeed, these systems have created a substantial anticipation in policy, military, industry and academic circles.  However, an increasing amount of this buzz involves concerns—many legitimate—regarding the extent of autonomy military unmanned systems ought to have. Unless or until these concerns are addressed, the enormous potential of these technological marvels may never be realized.

The Plan for Military Autonomous Systems

At the highest levels of U.S. policy and strategy documents, unmanned systems are featured as an important part of the way the Joint force will fight in the future. The most recent Quadrennial Defense Review notes, “Continuing a trend that began in the late 1990s, U.S. forces will increase the use and integration of unmanned systems.” Elsewhere in the QDR, unmanned systems are identified as: “Maintaining our ability to project power.” Importantly, the QDR highlights unmanned systems as a key part of the DoD’s commitment to innovation and adaptation.[2]

The Department of Defense’s vision for unmanned systems is to integrate these systems into the Joint force for a number of reasons, but especially to reduce the risk to human life in high threat areas, to deliver persistent surveillance over areas of interest, and to provide options to warfighters that derive from the inherent advantages of unmanned technologies—especially their ability to operate autonomously.

Because unmanned systems are used by all the military Services, the Department of Defense publishes a biennial roadmap to provide an overarching vision for the military’s use of unmanned systems. The most recent roadmap, the FY 2013-2038 Unmanned Systems Integrated Roadmap, singled out the need for enhanced UxS autonomy, noting, “DoD envisions unmanned systems seamlessly operating with manned systems while gradually reducing the degree of human control and decision making required for the unmanned portion of the force structure.”[3] As Dyke Weatherington, the DoD Director for Unmanned Warfare and Intelligence, Surveillance and Reconnaissance noted, “The roadmap articulates a vision and strategy for the continued development, production, test, training, operation and sustainment of unmanned systems technology across DoD…This roadmap establishes a technological vision for the next 25 years.”[4]

As the QDR and Unmanned Systems Integrated Roadmap both note, unmanned systems are especially important assets in those areas where the U.S. military faces a strong anti-access/area denial (A2/AD) threat. The Joint Operational Access Concept identifies, “Unmanned systems, which could loiter to provide intelligence collection or fires in the objective area,” as a key counter-area denial capability.[5] And unmanned systems are a key component in executing the AirSea Battle Concept (now retitled the Joint Concept for Access and Maneuver in the Global Commons (JAM-GC)) in high threat areas such as the Western Pacific, where adversary A2/AD systems pose an unacceptably high risk to manned aircraft.

The Need for “Offset” Strategies

More recently, the Department of Defense has initiated a “Third Offset Strategy,” to ensure that the United States retains the military edge against potential adversaries. An “offset” strategy is an approach to military competition that seeks to asymmetrically compensate for a disadvantaged position. Rather than competing head to head in an area where a potential adversary may also possess significant strength, an offset strategy seeks to shift the axis of competition—through the introduction of new operational concepts and technologies—toward one in which the United States has a significant and sustainable advantage.

The United States was successful in pursuing two distinct offset strategies during the Cold War. These strategies enabled the U.S. to “offset” the Soviet Union’s numerical advantage in conventional forces without pursuing the enormous investments in forward-deployed forces that would have been required to provide overmatch soldier-for-soldier and tank-for-tank. These offset strategies relied on fundamental innovation in technology, operational approaches, and organizational structure to compensate for Soviet advantage in time, space, and force size.

The first of these offset strategies occurred in the 1950’s, when President Eisenhower sought to overcome Warsaw Pact’s numerical advantage by leveraging US nuclear superiority to introduce battlefield nuclear weapons—thus shifting the axis of competition from conventional force numbers to an arena where the United States possessed an asymmetrical advantage. This approach provided stability and offered the foundation for deterrence. The second of these offset strategies arose in the late 1970’s and 1980’s with the recognition that the Soviet Union had achieved nuclear parity. The Second Offset Strategy sought to create an enduring advantage by pursuing a new approach to joint operations—leveraging the combined effects of conventional precision weapons, real-time, long-range, ISR sensor capabilities that supported real-time precision targeting, and the joint battle networks that permitted these capabilities to be synchronized and executed over the full breadth of the battlespace.

At the time of the introduction of the Second Offset Strategy in the early 1980’s, the United States was the only nation with the knowledge and capacity to develop, deploy, and successfully execute the intelligence, surveillance and reconnaissance capabilities, the space-based systems, and the precision weapons that supported this approach. Today, competitors such as Russia and China (and countries to which these nations proliferate advanced capabilities) are pursuing and deploying advanced weapons and capabilities that demonstrate many of the same technological strengths that provide the high-tech basis for U.S. advantage.

The emergence of increasing symmetry in national security environment suggests that it is again time to begin considering the mix of technologies, system concepts, military organizations, and operational concepts that might shift the nature of the competition to U.S. advantage. This set of capabilities will provide the basis for a Third Offset Strategy. As was true of previous offset strategies, a Third Offset Strategy seeks to maintain and extend United States competitive technological and operational advantage by identifying asymmetric advantages that are enabled by unique U.S. strengths and capabilities.

In explaining the technological elements of the Third Offset Strategy, Deputy Secretary of Defense Robert Work has emphasized the importance of emerging capabilities in artificial intelligence and autonomy. These technologies offer significant advantage to the Joint Force—enabling the future force to develop and operate advanced joint, collaborative human-machine battle networks that synchronize simultaneous operations in space, air, sea, undersea, ground, and cyber domains. Artificial intelligence will allow new levels of autonomy–the limited delegation of decision-making authority—within joint battle networks, leading to entirely new opportunities for human-machine collaboration and combat teaming.[6]

It is difficult to overstate the prominence of technologies such autonomous systems and artificial intelligence in the Third Offset Strategy and especially in the Strategy’s Long Range Research and Development Plan (LRRDP). That said, there is strong component of this strategy that emphasizes keeping humans in the loop when using autonomous systems with increasingly sophisticated artificial intelligence capabilities. Indeed, human-machine collaboration is an imperative that is emphasized in extant Third Offset Strategy documentation as well as in speeches and interviews with senior DoD officials.[7] The Strategy’s primary technical line of effort is focused on the concept of Human-Machine Collaboration and Combat Teaming. The five basic building blocks of this concept are:[8]

• Autonomous deep learning systems, which will leverage machine learning to operate “at the speed of light” in areas where human reaction time is too slow, such as cyber-attacks, electronic warfare attacks, or large missile raid attacks.
• Human-machine collaboration, which will allow machines to help humans make better decisions faster; as examples, Secretary Work cited the F-35 Joint Strike Fighter and the Naval Integrated Fire Control Counter-Air (NIFC-CA).
• Assisted human operations, which will focus on the ways in which man and machines can operate together, through tools such as wearable electronics, exoskeletons, and combat applications to assist warfighters in every possible contingency.
• Advanced human-machine combat teaming, which will focus on humans working with unmanned systems in cooperative operations; one example is the operation of the Navy’s P-8 Poseidon with an MQ-4C Triton. Going forward, the next level of teaming will examine swarming tactics and cooperative autonomy.
• Network-enabled, cyber-hardened autonomous weapons, which will be resilient to operate in an EW and cyber environment.  A current example includes the tactical Tomahawk Block IX, whose targets can be updated in-flight. Secretary Work also noted that DoD is exploring a concept that the Russians pioneered, namely using weapons in a salvo launched at a surface action group; one weapon would go high and direct the others to their targets, based on the electronic emissions that it senses.

Knowledgeable outside observers have referenced the Third Offset Strategy and have highlighted the importance of unmanned systems in achieving U.S. strategic goals. In his article in Foreign Policy, “The New Triad,” Admiral James Stavridis, former SACEUR, identified unmanned systems as one of the three pillars of this New Triad, noting, “The second capability in the New Triad is unmanned vehicles and sensors. This branch of the triad includes not only the airborne attack "drones"…but unmanned surveillance vehicles in the air, on the ground, and on the ocean's surface…Such systems have the obvious advantage of not requiring the most costly component of all: people.”[9]

The Challenges for Autonomous Systems

Well over a decade ago, in their report, Roles of Unmanned Vehicles, the Naval Research Advisory Committee highlighted the bright future and enormous potential for autonomous systems, noting, “The combat potential of UVs (unmanned vehicles) is virtually unlimited…There is no question that the Fleet/Forces of the future will be heavily dependent upon UVs.”[10] In the years following the NRAC report, the U.S. military has been working with industry and academia to make unmanned vehicles more-and-more autonomous. There are compelling reasons for this effort.

As described in the most recent Unmanned Systems Roadmap, there are four levels of autonomy: Human Operated, Human Delegated, Human Supervised, and Fully Autonomous. However, the Roadmap notes that in contrast to automatic systems, which simply follow a set of preprogrammed directions to achieve a predetermined goal, autonomous systems “are self-directed towards a goal in that they do not require outside control, but rather are governed by laws and strategies that direct their behavior.”[11]

One of the most pressing challenges for the DoD is to reduce the prohibitively burdensome manpower requirements currently necessary to operate unmanned systems.  Military manpower makes up the largest part of the total ownership cost of systems across all the Services.[12]  But how expensive is military manpower?  To better understand this compelling need to reduce these manpower requirements, it is important to understand the costs of manpower to the U.S. military writ large.

Military manpower accounts comprise the largest part of the total ownership cost (TOC) of military systems across all the Services.  Additionally, military manpower costs are the fastest growing accounts, even as the total number of military men and women decrease.  According to an Office of Management and Budget report, military personnel expenditures have risen from $74 billion dollars in 2001 to $159 billion dollars in 2012, an increase of almost 115 percent.[13]  Mackenzie Eaglen and Michael O’Hanlon have noted that between fiscal year 2001 and 2012, the compensation cost per active-duty service member increased by 56%, after being adjusted for inflation.[14]

Lessons learned throughout the development process of most unmanned systems—especially unmanned aerial systems—demonstrate that unmanned systems can actually increase manning requirements.  Indeed, the Air Force has estimated that the MQ-1 Predator requires a crew of about 168 personnel, while the MQ-9 Reaper requires a crew of 180 and the RQ-4 Global Hawk relies on 300 people to operate it.[15]  As General Philip Breedlove, Vice Chief of Staff of the Air Force, has emphasized, “The number one manning problem in our Air Force is manning our unmanned platforms.”[16]

Compounding the TOC issue, the data overload challenge generated by the proliferation of unmanned aircraft and their sensors has created its own set of manning issues.  In fact, the situation has escalated so quickly that many doubt that hiring additional analysts will help ease the burden of sifting through thousands of hours of video. A former Vice Chairman of the Joint Chiefs of Staff complained that a single Air Force Predator can collect enough video in one day to occupy nineteen analysts, noting, “Today an analyst sits there and stares at Death TV for hours on end, trying to find the single target or see something move.  It’s just a waste of manpower.”[17] The data overload challenge is so serious that it’s widely estimated that the Navy will soon face a “tipping point,” after which the Navy will no longer be able to process the amount of data that it’s compiling.[18]

With the prospect of future flat or declining military budgets, the rapidly rising costs of military manpower, and the increased DoD emphasis on total ownership costs, the mandate to move beyond the “many operators, one-joystick, one-vehicle” paradigm for UxS that has existed during the past decades for most unmanned systems is clear and compelling. The DoD and the Services are united in their efforts to increase the autonomy of unmanned systems as a primary means of reducing manning and achieving acceptable TOC. But this drive for autonomy begs the question as to what this imperative to increase autonomy comports and what, if any, downside occurs if we push UxS autonomy too far.  Is there an unacceptable “dark side” to too much autonomy?

The Dark Side of Unmanned Systems Autonomy

One of the most iconic films of the last century, Stanley Kubrick’s 2001: A Space Odyssey had as its central theme, the issue of autonomy of robots (the unmanned vehicles of the time). Few who saw the movie can forget the scene where astronauts David Bowman and Frank Poole consider disconnecting HAL's (Heuristically programmed ALgorithmic computer) cognitive circuits when he appears to be mistaken in reporting the presence of a fault in the spacecraft's communications antenna. They attempt to conceal what they are saying, but are unaware that HAL can read their lips. Faced with the prospect of disconnection, HAL decides to kill the astronauts in order to protect and continue its programmed directives.

While few today worry that a 21^st century HAL will turn on its masters, the issues involved with fielding increasingly-autonomous unmanned systems are complex, challenging and increasingly contentious. Kubrick’s 1968 movie was prescient. Almost half-a-century later, while we accept advances in other aspects of UxS improvements such as propulsion, payload, stealth, speed, endurance and other attributes, we are still coming to grips with how much autonomy is enough and how much may be too much. This is arguably the most important issue we need to address with unmanned systems over the next decade.[19]

Unmanned systems become more autonomous in direct proportion to their ability to sense the environment and adapt to it.  This capability enables unmanned systems to achieve enhanced speed in decision making and allows friendly forces to act within an adversary’s OODA (Observe, Orient, Decide, and Act) loop.[20] As the environment or mission changes, the ability to sense and adapt will allow unmanned systems to find the optimal solution for achieving their mission, without the need to rely on constant human operator oversight, input and decision-making. But while we need unmanned systems to operate inside the enemy’s OODA loop, are we ready for them to operate without our decision-making, to operate inside our OODA loops?

In an article entitled, “Morals and the Machine,” The Economist addressed the issue of autonomy and humans-in-the-loop this way:

As they become smarter and more widespread, autonomous machines are bound to end up making life-or-death decisions in unpredictable situations, thus assuming—or at least appearing to assume—moral agency. Weapons systems currently have human operators “in the loop,” but as they grow more sophisticated, it will be possible to shift to “on the loop” operation, with machines carrying out orders autonomously.

As that happens, they will be presented with ethical dilemmas. Should a drone fire on a house where a target is known to be hiding, which may also be sheltering civilians? Should a driverless car swerve to avoid pedestrians if that means hitting other vehicles or endangering its occupants? Should a robot involved in disaster recovery tell people the truth about what is happening if that risks causing a panic?

Such questions have led to the emergence of the field of “machine ethics,” which aims to give machines the ability to make such choices appropriately—in other words—to tell right from wrong. More collaboration is required between engineers, ethicists, lawyers and policymakers, all of whom would draw up very different types of rules if they were left to their own devices.[21]

More recently, while it may seem counterintuitive, concerns about autonomous machines and artificial intelligence are also coming from the very industry that is most prominent in developing these technological capabilities. The author of a New York Times article entitled, “Robot Overlords? Maybe Not,” Alex Garland of the movie “Ex Machina” talks about artificial intelligence and quotes several leaders of the tech industry.

The theoretical physicist Stephen Hawking told us that “the development of full artificial intelligence could spell the end of the human race.” Elon Musk, the chief executive of Tesla, told us that A.I. was “potentially more dangerous than nukes.” Steve Wozniak, a co-founder of Apple, told us that “computers are going to take over from humans” and that “the future is scary and very bad for people.”[22]

The Department of Defense is addressing the issue of human control of unmanned systems as a first-order priority and is beginning to issue policy guidance to ensure that humans do remain in the OODA loop. A November 2012 directive by then-Deputy Secretary of Defense Ashton Carter issued the following guidance:

Human input and ongoing verification are required for autonomous and semi-autonomous weapon systems to help prevent unintended engagements. These systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force. Humans who authorize the use of, or operate these systems, must do so with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules and applicable rules of engagement. An autonomous system is defined as a weapon system that, once activated, can select and engage targets without further intervention by a human operator.[23]

These are the kinds of directives and discussions that are—and should be—part of the dialogue between and among policy makers, military leaders, industry, academia and the science and technology community as the design and operation of tomorrow’s UxS are thoughtfully considered. As Deputy Secretary of Defense Robert Work noted during his remarks at the Center for New American Security Defense Forum last year, “We believe, strongly, that humans should be the only ones to decide when to use lethal force.  But when you're under attack, especially at machine speeds, we want to have a machine that can protect us.[24]

But it is one thing to issue policy statements, and quite another to actually design UxS to carry out the desired policy. This is not a trivial undertaking and—in Albert Einstein’s words—will require a new way of “figuring out how to think about the problem.”[25] And importantly, most informed discussion begins with the premise that adversaries who intend to use UxS against our interests will not be inhibited by the kinds of legal, ethical and moral strictures the United States adheres to.  Designing the right degree of autonomy into our unmanned systems is the central issue that will determine their success or failure.

Designing in the Right Degree of Autonomy

As the DoD and the Services look to achieve the right balance of autonomy and human interaction—to balance these two often-opposing forces and get them just right—designing this capability into tomorrow’s unmanned systems at the outset rather than trying to blot it on after the fact may be the only sustainable road ahead. If we fail to do this, it is almost inevitable that concerns our armed unmanned systems will take on “HAL-like” powers and be beyond our control will derail the promise of these technological marvels. 

The capabilities required to find this “just right” balance must leverage many technologies that are still emerging. The military knows what it wants to achieve, but often not what technologies or even capabilities it needs in order to field UxS with the right balance of autonomy and human interaction. A key element of this quest is to worry less about what attributes—speed, service ceiling, endurance, and others—the machine itself possess and instead focus on what is inside the machine. The Defense Science Board report, The Role of Autonomy in DoD Systems, put it this way:

Instead of viewing autonomy as an intrinsic property of unmanned systems in isolation, the design and operation of unmanned systems needs to be considered in terms of human-systems collaboration…A key challenge for operators is maintaining the human-machine collaboration needed to execute their mission, which is frequently handicapped by poor design…A key challenge facing unmanned systems developers is the move from a hardware-oriented, vehicle-centric development and acquisition process to one that emphasizes the primacy of software in creating autonomy.[26]

One need only go to an industry conference where UxS are being displayed at multiple booths to understand that today the emphasis is almost completely on the machine itself; what is inside is not a primary consideration. But as the Defense Science Board notes, it is software that is the primary driver of capabilities. For example, the manned F-35 Lightning has ten billion lines of computer code and there is human supervision by the pilot. How many lines of code will need to be built into an unmanned system to get the balance of autonomy and human interaction just right?

For the relatively small numbers of UxS that will engage an enemy with a weapon, this balance is crucial. Prior to firing a weapon, the unmanned platform needs to provide the operator—and there must be an operator in the loop—with a “pros and cons” decision matrix regarding what that firing might entail. When we build that capability into unmanned systems we will, indeed, have gotten it just right and the future of military autonomous will be bright.

Technology has driven mankind's progress, but each new advance has posed troubling new questions. Autonomous machines and artificial intelligence are no different. The sooner the questions of moral agency discussed here are answered, the easier it will be for mankind to enjoy the benefits that autonomous systems will undoubtedly bring. More collaboration is required between engineers, ethicists, lawyers and policymakers, all of whom would draw up very different types of rules if they were left to their own devices.

Into the Future with Autonomous Systems and Artificial Intelligence

We began this paper with the thesis that autonomous systems and artificial intelligence have the potential to drive a true revolution in military affairs. But this can only happen if the rapid—some would say galloping—technological advances in autonomous systems and artificial intelligence take into account valid moral and ethical considerations regarding their use. Perhaps most importantly, as senior U.S. Department of Defense officials have emphasized, from the U.S. DoD perspective, autonomous systems must always have the option for human control and verification, especially when it comes to the use of lethal force by autonomous systems with artificial intelligence.

Those responsible for the concepts, research, development, building, fielding and use of autonomous systems with artificial intelligence might be well-served to look into the commercial trade space, to the automobile industry, for best-practices examples. It is here that we may well find that vital customer feedback that indicates what drivers really want. And while not a perfect one-to-one match, this analogy can suggest what kinds of autonomous systems with artificial intelligence industry should offer to the military.

Automobiles are being conceived, designed, built and delivered with increasing amounts of artificial intelligence. It is worth examining where these trend lines are going? Put a bit simplistically, automobiles can be broken down into three basic categories:

• A completely manual car – something your parents drove
• A driverless car that takes you where you want to go via artificial intelligence
• A car with augmented intelligence

The initial enthusiasm for driverless cars has given way to second thoughts regarding how much a driver will be willing to be taken completely out of the loop. One article in particular, in the New York Times Business Day section, “Riding Down the Highway, with Tesla’s Code at the Wheel,” captures the concerns of many.[27] A growing consensus among consumers is that drivers want to be “in the loop” and that they want semi- and not fully-autonomous cars. That may change in the future, but perhaps not.

Extrapolating this every-day example to the military unmanned systems, we believe–and we think the available evidence, including some of the most cutting-edge work going on today–strongly suggests that warfighters want augmented intelligence in their autonomous machines. That will make these machines more useful and allow us to address and control them in a way that will go a long way toward resolving many of the moral and ethical concerns related to their use.

The Challenge and Way Ahead

The sooner the U.S. military signals this desire for unmanned systems with augmented intelligence the more likely it will be that the Services will have the optimal autonomous systems that will function in concert with manned systems with just the right degree of human oversight and control. The ultimate beneficiary will be the U.S. Joint warfighters who will then able to employ these systems within the constructs of the Law of War and Rules of Engagement.

The DoD would also be well-served to expand efforts to ensure that all those with a stake in the use of autonomous systems by the U.S. military, not just those in uniform, but engineers, ethicists, lawyers and policymakers, can have input into the design process of autonomous systems. This will ensure that all stakeholders in the often-controversial issue of armed autonomous systems can be fully engaged at the outset. As one way of establishing and sustaining this kind of informed dialogue, in 2011, then-Undersecretary of the Navy (now-Deputy Secretary of Defense) Robert Work established the Naval Postgraduate School Consortium for Robotics and Unmanned Systems Education and Research (CRUSER), to establish a wide-ranging community of interest for unmanned systems. More efforts like CRUSER will help ensure that the autonomous systems we provide U.S. warfighters are the right ones.[28]

End Notes