In November 2012, United States Deputy Defense Secretary Ashton Carter signed directive 3000.09, establishing policy for the “design, development, acquisition, testing, fielding, and … application of lethal or non-lethal, kinetic or non-kinetic, force by autonomous or semi-autonomous weapon systems.” Without fanfare, the world had its first openly declared national policy for killer robots.
The policy has been widely misperceived as one of caution. According to one account, the directive promises that a human will always decide when a robot kills another human. Others even read it as imposing a 10-year moratorium to allow for discussion of ethics and safeguards. However, as a Defense Department spokesman confirmed for me, the 10-year expiration date is routine for such directives, and the policy itself is “not a moratorium on anything.”
A careful reading of the directive finds that it lists some broad and imprecise criteria and requires senior officials to certify that these criteria have been met if systems are intended to target and kill people by machine decision alone. But it fully supports developing, testing, and using the technology, without delay. Far from applying the brakes, the policy in effect overrides longstanding resistance within the military, establishes a framework for managing legal, ethical, and technical concerns, and signals to developers and vendors that the Pentagon is serious about autonomous weapons.
Did soldiers ask for killer robots? In the years before this new policy was announced, spokesmen routinely denied that the US military would even consider lethal autonomy for machines. Over the past year, speaking for themselves, some retired and even active duty officers have written passionately against both autonomous weapons and the overuse of remotely operated drones. In May 2013, the first nationwide poll ever taken on this topic found that Americans opposed to autonomous weapons outnumbered supporters by two to one. Strikingly, the closer people were to the military—family, former military, or active duty—the more likely they were to strongly oppose autonomous weapons and support efforts to ban them.
Since the 1990s, the military has exhibited what autonomy proponent Barry Watts has called “a cultural disinclination to turn attack decisions over to software algorithms.” Legacy weapons such as land and sea mines have been deemphasized and some futuristic programs canceled—or altered to provide greater capabilities for human control. Most notably, the Army’s Future Combat Systems program, which was to include a variety of networked drones and robots at an eventual cost estimated as high as $300 billion, was cancelled in 2009, with $16 billion already spent.
At the same time, calls for autonomous weapons have been rising both outside and from some inside the military. In 2001, retired Army lieutenant colonel T. K. Adams argued that humans were becoming the most vulnerable, burdensome, and performance-limiting components of manned systems. Communications links for remote operation would be vulnerable to disruption, and full autonomy would be needed as a fallback. Furthermore, warfare would become too fast and too complex for humans to direct. Realistic or not, such thinking, together with budget pressures and the perception that robots are cheaper than people, has supported a steady growth of autonomy research and development in military and contractor-supported labs. In March 2012, the Naval Research Lab opened a new facility dedicated to development and testing of autonomous systems, complete with simulated rainforest, desert, littoral, and shipboard or urban combat environments. But the killer roboticists’ brainchildren have continued to face what a 2012 Defense Science Board report, commissioned by then-Undersecretary Carter, called “material obstacles within the Department that are inhibiting the broad acceptance of autonomy.”
The discrimination problem. Navy scientist John Canning recounts a 2003 meeting at which high-level lawyers from the Navy and Pentagon objected to autonomous weapons. They assumed that robots could not comply with international humanitarian law, core principles of which include a responsibility to distinguish civilians from combatants and to refrain from attacks that would cause excessive harm to civilians. These principles, and the military rules of engagement intended to implement them, assume a level of awareness, understanding, and judgment that computers simply don’t have. Weapons are also subject to mandated legal review, and indiscriminate weapons—that is, weapons that cannot be selectively directed to attack lawful targets and avoid civilians—are forbidden. The lawyers did not think they would ever be able to sign off on autonomous weapons.
Georgia Tech roboticist Ron Arkin has argued that unemotional robots, following rigid programs, could actually be more ethical than human soldiers. But his proposals fail to solve the hard problems of distinguishing civilians, understanding and predicting social and tactical situations, or judging the proportionality of force. Others argue, philosophically, that only humans can make such targeting judgments legitimately. In a world getting used to talking about virtual assistants and self-driving cars, it may not be obvious what the limits of artificial intelligence will be, or what people will accept, in 10, 20, or 40 years. But for now, and for the immediate future, the robot discrimination problem is hard to dispute.
To break the legal deadlock, Canning suggested that robots might normally be granted autonomy to attack materiel, including other robots, but not humans. Yet in many situations it might be impossible to avoid the risk—or the intent—of killing or injuring people. For such cases, Canning proposed what he called “dial-a-level” autonomy; that is, the robot might ordinarily be required to ask a human what to do, but in some circumstances it could be authorized to take action on its own. More
