'Killer robots' to be debated at UN

Killer robots will be debated during an informal meeting of experts at the United Nations in Geneva.

Two robotics experts, Prof Ronald Arkin and Prof Noel Sharkey, will debate the efficacy and necessity of killer robots.

The meeting will be held during the UN Convention on Certain Conventional Weapons (CCW).

A report on the discussion will be presented to the CCW meeting in November.

This will be the first time that the issue of killer robots, or lethal autonomous weapons systems, will be addressed within the CCW.

Autonomous kill function

A killer robot is a fully autonomous weapon that can select and engage targets without any human intervention. They do not currently exist but advances in technology are bringing them closer to reality.

Those in favour of killer robots believe the current laws of war may be sufficient to address any problems that might emerge if they are ever deployed, arguing that a moratorium, not an outright ban, should be called if this is not the case.

However, those who oppose their use believe they are a threat to humanity and any autonomous "kill functions" should be banned.

"Autonomous weapons systems cannot be guaranteed to predictably comply with international law," Prof Sharkey told the BBC. "Nations aren't talking to each other about this, which poses a big risk to humanity."

Prof Sharkey is a member and co-founder of the Campaign Against Killer Robots and chairman of the International Committee for Robot Arms Control.

Side events at the CCW will be hosted by the Campaign to Stop Killer Robots.

Automation of warfare

Prof Arkin from the Georgia Institute of Technology told the BBC he hoped killer robots would be able to significantly reduce non-combatant casualties but feared they would be rushed into battle before this was accomplished.

"I support a moratorium until that end is achieved, but I do not support a ban at this time," said Prof Arkin.

He went on to state that killer robots may be better able to determine when not to engage a target than humans, "and could potentially exercise greater care in so doing".

Prof Sharkey is less optimistic. "I'm concerned about the full automation of warfare," he says.

Drones

The discussion of drones is not on the agenda as they are yet to operate completely autonomously, although there are signs this may change in the near future.

The UK successfully tested the Taranis, an unmanned intercontinental aircraft in Australia this year and America's Defense Advanced Research Projects Agency (Darpa) has made advances with the Crusher, an unmanned ground combat vehicle, since 2006.

The MoD has claimed in the past that it currently has no intention of developing systems that operate without human intervention.

On 21 November 2012 the United States Defense Department issued a directive that, "requires a human being to be 'in-the-loop' when decisions are made about using lethal force," according to Human Rights Watch.

The meeting of experts will be chaired by French ambassador Jean-Hugues Simon-Michel from 13 to 16 May 2014. More

 

© 2014

Confessions of a Drone Warrior

From the darkness of a box in the Nevada desert, he watched as three men trudged down a dirt road in Afghanistan.

The box was kept cold—precisely sixty-eight degrees—and the only light inside came from the glow of monitors. The air smelled spectrally of stale sweat and cigarette smoke. On his console, the image showed the midwinter landscape of eastern Afghanistan’s Kunar Province—a palette of browns and grays, fields cut to stubble, dark forests climbing the rocky foothills of the Hindu Kush. He zoomed the camera in on the suspected insurgents, each dressed in traditional shalwar kameez, long shirts and baggy pants. He knew nothing else about them: not their names, not their thoughts, not the thousand mundane and profound details of their lives.

He was told that they were carrying rifles on their shoulders, but for all he knew, they were shepherd’s staffs. Still, the directive from somewhere above, a mysterious chain of command that led straight to his headset, was clear: confirmed weapons. He switched from the visible spectrum—the muted grays and browns of “day-TV”—to the sharp contrast of infrared, and the insurgents’ heat signatures stood out ghostly white against the cool black earth. A safety observer loomed behind him to make sure the “weapon release” was by the book. A long verbal checklist, his targeting laser locked on the two men walking in front. A countdown—three…two…one…—then the flat delivery of the phrase “missile off the rail.” Seventy-five hundred miles away, a Hellfire flared to life, detached from its mount, and reached supersonic speed in seconds.

It was quiet in the dark, cold box in the desert, except for the low hum of machines.

He kept the targeting laser trained on the two lead men and stared so intently that each individual pixel stood out, a glowing pointillist dot abstracted from the image it was meant to form. Time became almost ductile, the seconds stretched and slowed in a strange electronic limbo. As he watched the men walk, the one who had fallen behind seemed to hear something and broke into a run to catch up with the other two. Then, bright and silent as a camera flash, the screen lit up with white flame.

Airman First Class Brandon Bryant stared at the scene, unblinking in the white-hot clarity of infrared. He recalls it even now, years later, burned into his memory like a photo negative: “The smoke clears, and there’s pieces of the two guys around the crater. And there’s this guy over here, and he’s missing his right leg above his knee. He’s holding it, and he’s rolling around, and the blood is squirting out of his leg, and it’s hitting the ground, and it’s hot. His blood is hot. But when it hits the ground, it starts to cool off; the pool cools fast. It took him a long time to die. I just watched him. I watched him become the same color as the ground he was lying on.”

That was Brandon Bryant’s first shot. It was early 2007, a few weeks after his twenty-first birthday, and Bryant was a remotely-piloted-aircraft sensor operator—a “sensor” for short—part of a U.S. Air Force squadron that flew Predator drones in the skies above Iraq and Afghanistan. Beginning in 2006, he worked in the windowless metal box of a Ground Control Station (GCS) at Nellis Air Force Base, a vast sprawl of tarmac and maintenance hangars at the edge of Las Vegas.

The airmen kept the control station dark so they could focus on controlling their MQ-1B Predators circling two miles above the Afghan countryside. Bryant sat in a padded cockpit chair. He had a wrestler’s compact build, a smooth-shaved head, and a piercing ice blue gaze frequently offset by a dimpled grin. As a sensor, his job was to work in tandem with the drone’s pilot, who sat in the chair next to him. While the pilot controlled the drone’s flight maneuvers, Bryant acted as the Predator’s eyes, focusing its array of cameras and aiming its targeting laser. When a Hellfire was launched, it was a joint operation: the pilot pulled a trigger, and Bryant was responsible for the missile’s “terminal guidance,” directing the high-explosive warhead by laser to its desired objective. Both men wore regulation green flight suits, an unironic Air Force nod to the continuity of military decorum in the age of drone warfare.

Since its inception, the drone program has been largely hidden, its operational details gathered piecemeal from heavily redacted classified reports or stage-managed media tours by military public-affairs flacks. Bryant is one of very few people with firsthand experience as an operator who has been willing to talk openly, to describe his experience from the inside. While Bryant considers leakers like Chelsea Manning and Edward Snowden heroes willing to sacrifice themselves for their principles, he’s cautious about discussing some of the details to which his top-secret clearance gave him access. Still, he is a curtain drawn back on the program that has killed thousands on our behalf.

Despite President Obama’s avowal earlier this year that he will curtail their use, drone strikes have continued apace in Pakistan, Yemen, and Afghanistan. With enormous potential growth and expenditures, drones will be a center of our policy for the foreseeable future. (By 2025, drones will be an $82 billion business, employing an additional 100,000 workers.) Most Americans—61 percent in the latest Pew survey—support the idea of military drones, a projection of American power that won’t risk American lives. More

======================

Chatham House Conference - Stop Killer Robots

 

© 2014

Campaign to stop killer robots: Chatham House conference

Chatham House conference

The first Chatham House conference on autonomous military technologies in London on 24-25 February brought together individuals from different constituencies to contemplate autonomous weapons and the prospect of delegating human control over targeting and attack decisions to machines. The Campaign to Stop Killer Robots was pleased to be able to attend this well-organized and timely conference held under the Chatham House rule, which permits participants to use information received but not to reveal the identity or affiliation of the speaker or participants. The conference was a useful opportunity to discuss our concerns with fully autonomous weapons, provide clarifications, and answer questions about our coalition’s focus and objectives.

Some participants have since publicly provided their views on the conference, including Charles Blanchard on Opinio Juris (4 March) and Paul Scharre on the Lawfare blog (3 March).

Several of the Campaign to Stop Killer Robots representatives who attended the Chatham House conference have provided input for this web post, including on the reflections published by Blanchard and Scharre. The campaign’s principal spokespersons Nobel Peace laureate Jody Williams, roboticist Professor Noel Sharkey, and Human Rights Watch arms director Steve Goose addressed the conference, while campaigners were present from the non-governmental organizations Action on Armed Violence,Amnesty International, Article 36, Human Rights Watch, International Committee for Robot Arms Control, and PAX (formerly IKV Pax Christi).

The perspective of the Campaign to Stop Killer Robots and its call for a ban on fully autonomous weapons were heard throughout the conference, but to ensure that key concerns are not downplayed and in the spirit of furthering common understanding on this emerging issue of international concern, we have the following comments on the reflections by Blanchard and Scharre.

Blanchard, a former US Air Force general counsel, gave a public talk on the topic “Autonomous Technologies: A Force for Good?” at Chatham House together with our campaign spokesperson Jody Williams, who received the 1997 Nobel Peace laureate together with the International Campaign to Ban Landmines (ICBL). He is now a partner at Arnold & Porter LLP, a Washington DC law firm that actively supported the negotiation of the 2006 Disability Rights Treaty as well as efforts to include victim assistance provisions in the 1997 Mine Ban Treaty.

Blanchard considers “deep philosophical viewpoints” in his piece, which looks at some of the “disputes” at the Chatham House conference over the call for a ban on fully autonomous weapons to enshrine the principle that only humans should decide to kill other humans. Blanchard is concerned that “more death” may result from a ban because autonomous weapons might be “more capable than humans” of complying with the laws of war.

While we do not agree with Blanchard’s skeptical position as to the benefits that a ban on fully autonomous weapons could provide, we welcome his acknowledgement of the counter-argument that letting a machine decide whom to kill would violate notions of human dignity. Blanchard’s assessment of the viability of a ban illustrates how the debate has advanced far in recent months to the point that a ban is being seriously contemplated.

Paul Scharre heads the 20YY Warfare Initiative at the Center for a New American Security in Washington DC and previously worked for the US Department of Defense, where he led a working group that drafted the 2012 policy directive 3000.09 on autonomy in weapon systems. His comprehensive presentations at the Chatham House conference were well-received, and his rational, measured and well-written reflections on the conference contain many useful observations.

Yet Scharre’s “key takeaways” oversimplify the “areas of agreement” and make it sound as if participants agreed more often than they actually did. His commentary attempts to reflect the conference speakers’ views and areas of convergence, but the same cannot be done for the audience—comprising approximately 150 participants from government, military, industry, think tanks, academia, civil society, and media.

With respect to the scope of what was discussed at the Chatham House, Scharre’s depiction of the conference being focused only on “anti-materiel” autonomous weapons systems is confusing as the conference addressed all types of autonomous weapons systems, including “anti-personnel.” The conference was also not specifically limited to “lethal” autonomous weapons as opposed to “non-lethal” or “less-than-lethal.” That said, we welcome the comments by Scharre indicating that he is not in favor fully autonomous anti-personnel weapon systems.

There was indeed convergence by the technologists who spoke to the capabilities of current autonomous technologies and the notion that precursors indicate something more dangerous to come.

Throughout the conference there did appear to be “universal agreement that humans should remain in control of decisions over the use of lethal force.” Consensus on this point was, however, qualified by a number of speakers who suggested that systems with no meaningful human control could be legal and have military utility. Such views illustrate why policy-level restraints will not suffice in addressing the challenges posed by fully autonomous weapons and should be supplemented with new law.

Indeed, this debate is happening because many are contemplating a future with no human control. Yet Scharre gave minimal consideration to proliferation concerns—development, production, transfer, stockpiling—in the “objections” section of his reflection. Concerns over an arms race were raised several times in the course of the Chatham House conference, which was sponsored by BAE Systems, manufacturer of the Taranis autonomous aircraft, the prime example of a UK precursor to autonomous weapons technology. As has been learned from experience with nuclear weapons, proliferation concerns cannot be addressed permanently through regulation and existing international humanitarian law.

Scharre claims that “a major factor in whether autonomous weapons are militarily attractive or even necessary may be simply whether other nations develop them,” but he seems to misunderstand the point of stigmatization in the “endgame” section of his reflections. By proposing that that the answer to concerns about “cheating” is an “even playing field” where everyone can have them (and presumably all can be “cheaters”), Scharre dismisses the power of an international, legally binding ban to stigmatize a weapon and ensure respect for the law. A global ban could succeed in stigmatizing autonomous weapons to the extent that no major military power uses them, as has been the case for the Mine Ban Treaty where major powers have not used antipersonnel landmines in years.

Scharre views the commercial sector as driving the “underlying technology behind autonomy” but that ignores that fact that industry is regulated by the state. Governments won’t prevent industry from developing the underlying technology nor–as Blanchard notes–is the campaign seeking to do that because the same technology that will be used in autonomous robotics and AI systems has many non-weapons and non-military purposes. But research and development activities should be banned if they are directed at technology that can only be used for fully autonomous weapons or that is explicitly intended for use in such systems.

Scharre downplays legal concerns in several sections of his reflections. This is in part because the conference panel on international law was dominated by legal advocates of autonomous weapons. Several of the law panelists may have agreed with each other that autonomous weapons are “not illegal weapons prohibited under the laws of armed conflict” but this was not a view shared by all participants at the conference. In particular, serious concern was expressed about the nature of fully autonomous weapons and their likely inability, in making attack decisions, to distinguish noncombatants and judge the proportionality of expected civilian harms to expected military gains. Although no one can know for sure what future technology will look like, the possibility that fully autonomous weapons would be unable to comply with the laws of war cannot be dismissed at this point.

One speaker argued that if fully autonomous weapons could lawfully be used in any circumstance, they could not be considered per se unlawful. This point may be correct legally, but the case can be made that any weapon can be used legally in some carefully crafted scenario. The possibility of such limited use should not be used to legitimize fully autonomous weapons. History has well demonstrated that once a weapon is developed and fielded, it will not only be used in limited, pre-determined ways. The potential for harm is so great as to nullify the argument for legality.

Scharre claims agreement about “lawful limited uses,” citing three examples of his own. We certainly don’t agree.

Accountability is another area where there was less agreement than depicted in Scharre’s reflections. As he states, machines, as currently envisioned, can’t be held responsible under laws of war, and it makes sense that programmers or operators not be held liable for war crimes unless they intended the robot to commit one.

The notion of accountability for operators was touched on during the Chatham House conference, but it was not considered in depth and it is important to note the lingering concerns of some audience members. For example, the “fixes” that Scharre cites from the US Department of Defense directive fall far short. Under the directive, human decision makers are charged with responsibility for ensuring compliance with laws of war when the machines they set in motion are unable to ensure this. However, it is unlikely that commanders will be held liable for war crimes if unintended technical failures can be blamed, while programmers, engineers and manufacturers are unlikely to be held liable if they have acted in good faith.

Scharre’s apparent answer to the issue of accountability is a “completely predictable and reliable system,” but how is that possible? Even with rigorous test and evaluation procedures, autonomy will make it significantly harder to ensure predictability and reliability. In fact, one definition of autonomy is that the system, even when functioning correctly, is not fully predictable (due to its complexity and that of the environment with which it is interacting).

In addition, some question whether operators should be held directly responsible for the consequences of fully autonomous weapons’ actions. Can these operators be treated in the same way as operators of a “normal” weapon when fully autonomous weapons are able to make choices on their own?

Scharre seems to dismiss the Martens Clause as only an ethical issue, but it’s a legal one as well. Although its precise meaning is debated, the clause is a fixture of international humanitarian law that appears in several treaties. It implies that when there is no existing law specifically on point, weapons that “shock the human conscience” can be regarded as unlawful in anticipation of an explicit ban. It also supports adoption of an explicit ban of weapons that violate the “principles of humanity and dictates of public conscience.”

Scharre’s post raises a “practical” objection to fully autonomous weapons that was not considered by the conference: “A weapon that is uncontrollable or vulnerable to hacking is not very valuable to military commanders. In fact, such a weapon could be quite dangerous if it led to systemic fratricide.” This concern about “large-scale,” accidental killing is valid, but the same practical argument applies to civilian casualties and not just military ones.

As Scharre notes, there are many concerns with fully autonomous weapons that exist on several fundamentally different levels. We agree that discussions about where the technology is headed are critical, but finding a permanent solution is even more urgent.

The Chatham House event was the first of several important meetings due to be held on killer robots in2014. The International Committee of the Red Cross (ICRC) will convene its first experts meeting on autonomous weapons systems on 26-28 March. The first Convention on Conventional Weapons (CCW) meeting on lethal autonomous weapons systems will be held at the UN in Geneva on 13-16 May. UN Special Rapporteur Christof Heyns is due to report on lethal autonomous robots and other matters to the Human Rights Council in Geneva during the week of 10 June.

The fact that conferences like the one held by Chatham House are happening shows how the challenge of killer robots has vaulted to the top rank of traditional multilateral arms control and humanitarian disarmament, validating the importance and urgency of the issue and undercutting arguments that fully autonomous weapons are “inevitable” and “nothing to worry about.” The strong and diverse turn-out means it is unlikely to be the last Chatham House conference on this topic.

Immediately after the Chatham House conference, the Campaign to Stop Killer Robots held a strategy meeting that 50 NGO representatives attended. The meeting focused on planning the campaign’s strategy for year ahead at CCW and the Human Rights Council as well as how to initiate national campaigning to influence policy development and secure support for a ban.

For more information see:

• “Reflections on the Chatham House Autonomy Conference” by Paul Scharre, Lawfare (3 March 2014)
• “Autonomous Weapons at Chatham House: It’s Bentham versus Kant” by Charles Blanchard, Opinio Juris (4 March 2014)
• Chatham House conference agenda and list of speakers
• Campaign to Stop Killer Robots YouTube playlist of video recordings of the evening event with Jody Williams & Charles Blanchard and interviews with Amb. Simon-Michel of France, Steve Goose of Human Rights Watch, former US Air Force general counsel Charles Blanchard, and Paul Scharre of the Center for a New American Security. More

Photo: Patricia Lewis, research director for international security at Chatham House (center) introduced the first panel of the Chatham House conference on autonomous military technologies. (c) Campaign to Stop Killer Robots, 24 February 2014

 

© 2014

US killer robot policy: Full speed ahead

In November 2012, United States Deputy Defense Secretary Ashton Carter signed directive 3000.09, establishing policy for the “design, development, acquisition, testing, fielding, and … application of lethal or non-lethal, kinetic or non-kinetic, force by autonomous or semi-autonomous weapon systems.” Without fanfare, the world had its first openly declared national policy for killer robots.

The policy has been widely misperceived as one of caution. According to one account, the directive promises that a human will always decide when a robot kills another human. Others even read it as imposing a 10-year moratorium to allow for discussion of ethics and safeguards. However, as a Defense Department spokesman confirmed for me, the 10-year expiration date is routine for such directives, and the policy itself is “not a moratorium on anything.”

A careful reading of the directive finds that it lists some broad and imprecise criteria and requires senior officials to certify that these criteria have been met if systems are intended to target and kill people by machine decision alone. But it fully supports developing, testing, and using the technology, without delay. Far from applying the brakes, the policy in effect overrides longstanding resistance within the military, establishes a framework for managing legal, ethical, and technical concerns, and signals to developers and vendors that the Pentagon is serious about autonomous weapons.

Did soldiers ask for killer robots? In the years before this new policy was announced, spokesmen routinely denied that the US military would even consider lethal autonomy for machines. Over the past year, speaking for themselves, some retired and even active duty officers have written passionately against both autonomous weapons and the overuse of remotely operated drones. In May 2013, the first nationwide poll ever taken on this topic found that Americans opposed to autonomous weapons outnumbered supporters by two to one. Strikingly, the closer people were to the military—family, former military, or active duty—the more likely they were to strongly oppose autonomous weapons and support efforts to ban them.

Since the 1990s, the military has exhibited what autonomy proponent Barry Watts has called “a cultural disinclination to turn attack decisions over to software algorithms.” Legacy weapons such as land and sea mines have been deemphasized and some futuristic programs canceled—or altered to provide greater capabilities for human control. Most notably, the Army’s Future Combat Systems program, which was to include a variety of networked drones and robots at an eventual cost estimated as high as $300 billion, was cancelled in 2009, with $16 billion already spent.

At the same time, calls for autonomous weapons have been rising both outside and from some inside the military. In 2001, retired Army lieutenant colonel T. K. Adams argued that humans were becoming the most vulnerable, burdensome, and performance-limiting components of manned systems. Communications links for remote operation would be vulnerable to disruption, and full autonomy would be needed as a fallback. Furthermore, warfare would become too fast and too complex for humans to direct. Realistic or not, such thinking, together with budget pressures and the perception that robots are cheaper than people, has supported a steady growth of autonomy research and development in military and contractor-supported labs. In March 2012, the Naval Research Lab opened a new facility dedicated to development and testing of autonomous systems, complete with simulated rainforest, desert, littoral, and shipboard or urban combat environments. But the killer roboticists’ brainchildren have continued to face what a 2012 Defense Science Board report, commissioned by then-Undersecretary Carter, called “material obstacles within the Department that are inhibiting the broad acceptance of autonomy.”

The discrimination problem. Navy scientist John Canning recounts a 2003 meeting at which high-level lawyers from the Navy and Pentagon objected to autonomous weapons. They assumed that robots could not comply with international humanitarian law, core principles of which include a responsibility to distinguish civilians from combatants and to refrain from attacks that would cause excessive harm to civilians. These principles, and the military rules of engagement intended to implement them, assume a level of awareness, understanding, and judgment that computers simply don’t have. Weapons are also subject to mandated legal review, and indiscriminate weapons—that is, weapons that cannot be selectively directed to attack lawful targets and avoid civilians—are forbidden. The lawyers did not think they would ever be able to sign off on autonomous weapons.

Georgia Tech roboticist Ron Arkin has argued that unemotional robots, following rigid programs, could actually be more ethical than human soldiers. But his proposals fail to solve the hard problems of distinguishing civilians, understanding and predicting social and tactical situations, or judging the proportionality of force. Others argue, philosophically, that only humans can make such targeting judgments legitimately. In a world getting used to talking about virtual assistants and self-driving cars, it may not be obvious what the limits of artificial intelligence will be, or what people will accept, in 10, 20, or 40 years. But for now, and for the immediate future, the robot discrimination problem is hard to dispute.

To break the legal deadlock, Canning suggested that robots might normally be granted autonomy to attack materiel, including other robots, but not humans. Yet in many situations it might be impossible to avoid the risk—or the intent—of killing or injuring people. For such cases, Canning proposed what he called “dial-a-level” autonomy; that is, the robot might ordinarily be required to ask a human what to do, but in some circumstances it could be authorized to take action on its own. More