This post is the latest installment of our “Monday Reflections” feature, in which a different Just Security editor examines the big stories from the previous week or looks ahead to key developments on the horizon.
It’s no secret that the United States and other nations are pouring billions of dollars into developing lethal robots capable of making their own targeting decisions that operate everywhere from aircraft carriers to land and under the oceans. These machines have been developed in conjunction with a suite of top-secret digital weapons and intelligence gathering tools that do everything from turn domestic cellphones into surreptitious recording devices to wreak havoc inside foreign nuclear facilities and much more. Such secret digital weapons and tools have given the world’s governments the ability to learn immense amounts about their citizens’ lives with unprecedented ease. Yet these capabilities, and the methods by which they are used, were developed in almost total obscurity, with little input from the societies they affect. The new weapon systems being imagined by Pentagon planners have the potential to amplify this power by an unknown amount. We cannot afford to let the next generation of digital-age weapons be developed under similar secrecy.
The world is heading into uncharted territory. The technology behind increasingly sophisticated autonomous military drones and cyber weapons powered by massive increases in artificial intelligence are all on track to converge (though estimates of when and how intensely this will happen vary wildly). We can’t forget to add rapid advances in network connectivity, sensor power, data processing, additive manufacturing (3D printing), and synthetic biology to that mix.
As the 2014 report, War in the Robotic Age, written by Bob Work, the Defense Department’s number two official, explains:
While defense companies are pursuing advanced stealth systems, electric weapons and protected communications, companies focused on producing consumer goods and business-to-business services are driving many other key enabling technologies, such as advanced computing and “big data,” autonomy, artificial intelligence, miniaturization, additive manufacturing and small but high density power systems. All of these technologies – largely evolving in the thriving commercial computing and robotics sectors – could be exploited to build increasingly sophisticated and capable unmanned and autonomous military systems.
The fusing of these technologies is likely to lead to an incredible increase in capability from a military or intelligence agency’s perspective. (This is tacitly acknowledged in a November 2014 memo from the Pentagon’s top weapons buyer commissioning a study to find ways to “reduce or eliminate” barriers to DOD’s use of autonomous systems.) Therefore, anyone developing such weapons will likely want to keep their most potent new capabilities as secret as possible out of the natural desire not to reveal them to potential enemies, at least until there is little anyone can do about them.
Yet it’s imperative that such new weapons are developed with the highest possible levels of transparency and public scrutiny. If not, the first time we learn of them and their true potential could make Edward Snowden’s revelations seem quaint.
The weapons incorporating all of the technologies above will probably be linked, to some degree, to the same networks we conduct our civilian lives on. Indeed, these weapons will likely use and exploit civilian networks in combat since “cyberspace,” or the entire electromagnetic spectrum including the Internet, has been famously been defined by the Pentagon as its newest “warfighting domain,” on par with land, sea, air, and space. What happens when all of our homes, cars, clothes (the Internet of Things), and even our bodies are connected to a network that autonomous weapons ranging from robots to malware will also have access to?
Laws and regulations governing the development and use of new weapons cannot be properly debated and shaped if the public does not fully understand what is truly under discussion. The types of weapons being developed and the ways they would be used and controlled must be explained in the open, with language that is deliberately accessible and unambiguous (unlike the laws that have come to govern mass surveillance).
Former Defense Secretary Chuck Hagel hinted at the scope of the Pentagon’s ambition regarding such new technologies in a November 2014 speech announcing the Defense Innovation Initiative (DII) — the official title of DOD’s effort to develop wonder weapons to maintain America’s battlefield supremacy in the face of declining budgets and ever-shrinking military ranks:
America does not believe in sending our troops into a fair fight. Bob Gates, Leon Panetta, all of our predecessors believe that – and were responsible for ensuring that didn’t happen.
But that is a credo we will not be able to honor if we do not take the initiative and address these mounting challenges now. DoD must continue to modernize our nation’s capabilities and sustain its operational and technological edge. And we must do so by making new, long-term investments in innovation.
…
Today I’m announcing a new Defense Innovation Initiative – an initiative that we expect to develop into a game-changing third ‘offset’ strategy.
This new initiative is an ambitious department-wide effort to identify and invest in innovative ways to sustain and advance America’s military dominance for the 21st century.
…
Our technology effort will establish a new Long-Range Research and Development Planning Program that will help identify, develop, and field breakthroughs in the most cutting-edge technologies and systems – especially from the fields of robotics, autonomous systems, miniaturization, big data, and advanced manufacturing, including 3D printing. This program will look toward the next decade and beyond.
In the near-term, it will invite some of the brightest minds from inside and outside government to start with a clean sheet of paper, and assess what technologies and systems DoD ought to develop over the next three to five years and beyond. (Emphasis added.)
New information-age weapons of unknown power will change the world in the same way the new weapons of the industrial age — tanks, radios, airplanes, submarines, machine guns, guided missiles, weapons of mass destruction — changed the world. Those innovations helped to destroy centuries-old political and economic orders and helped fuel one of the deadliest eras in history.
While we know the world will change with the advent of new digital weapons, we don’t know nearly enough about how they are going to be used. For example, we don’t know the details of what will happen if truly autonomous robots capable of collecting massive amounts of data and making their own lethal targeting decisions are tied into the computer networks that govern every aspect of our lives. We don’t know what weapons they will use, we don’t know how much autonomy they will have, we don’t know how interconnected with our civilian networks they will be, we don’t know which rules of engagement they will operate under, we don’t know who will be allowed to develop and use them, and in the long run, we don’t know how they will be kept subservient to humans.
The Pentagon’s current policy on autonomous weapons dictates that a human must supervise the decisions made by autonomous robots, and that such weapons cannot make the decision to kill. This is merely a policy, not law. There’s nothing preventing this policy from being changed (or interpreted narrowly) by the next secretary of defense.
What could possibly go wrong?
We have the chance to resolve these questions now, before we wake up one day to find ourselves living with a fear that combines some of the discomfort we feel about today’s mass surveillance programs with the existential fears felt during the height of the Cold War, albeit with added twists that are impossible to predict at present.
TRANSPARENCY INITIATIVES
Preventing this scenario means that the Pentagon, intelligence agencies, contractors, lawmakers, NGOs, and the media must foster vigorous and open discussions of weapons being developed and the rules that will govern their use. It’s not good enough to write broad speeches and policy papers outlining governments’ desire to develop such weapons. We must decide, as a society, with as much clarity as possible how these technologies will be developed and used.
Suggestion 1: DOD
The Pentagon must use the DII as an opportunity to develop its next generation of weapons in the most public manner possible. This has been proven possible with numerous weapons systems from nuclear-powered ballistic missile submarines in the 1950 and 1960s to the F-35 Joint Strike Fighter today. These weapons were developed publicly in ways that didn’t reveal secrets benefitting potential enemies. Recent weapons that were developed in utter secrecy — like the first generation of stealth jets and the Army’s stealth helicopters used in the bin Laden raid — were refinements of existing weapons rather than inventions with the potential to disrupt massive segments of society.
The Pentagon must go out of its way to discuss publicly the ways in which wants to develop and use its new weapons by engaging the media, NGOs, and lawmakers in developing the Concepts of Operations (CONOPS) that will govern their use. Once again, this is possible to do without revealing specific tactics that will provide an enemy with an edge.
Suggestion 2: Contractors
Businesses ranging from traditional defense contractors to technology companies and telecommunications giants must foster open and public discussion about the technologies they are developing and how they could be used most appropriately. This could be done via media engagement, participation in interdisciplinary conferences with the NGO and academic world, and even by reporting on possible military uses for technology through a mechanism modeled after the transparency reports being generated by some technology firms in response to mass government surveillance.
Suggestion 3: Congressional oversight
Lawmakers must take their role of oversight seriously in this arena, holding public hearings that challenge the Pentagon and defense industry instead of serving as a rubber stamp legislature handing over unlimited, unsupervised money for potentially world-changing projects. Laws and regulations governing the development and use of such weapons must be vigorously debated in the same vein as net neutrality or cybersecurity legislation. And Congress should certainly avoid the hurried and overlooked manner in which bills like the Patriot Act or FISA were passed.
Suggestion 4: NGOs
DOD, private businesses, and Congress much actively court privacy, technology, civil liberties, arms control, and other concerned NGOs, along with the academic community throughout the development of the new generation of weapons. All four groups must work together to foster a vision of how such technology should be developed. This should be done, in part, via public fora and events, designed to engage the media and public. Too many important events and discussions surrounding national security matters incorporate only the voices of the military and its weapons suppliers. Autonomous weapons are too critical to exclude or marginalize the perspectives of the NGO and academic world.
Suggestion 5: Arms control and monitoring regimes
As we have seen with nuclear weapons the previous class of world-changing arms that are often developed in total secrecy, domestic laws and international agreements aren’t always enough to limit the spread of incredibly potent weapons. An international regime designed to restrict or guide the development and use of autonomous super weapons may be needed to give teeth to whatever decisions result from a public debate about the use of such tools. Such regimes could be modeled on the Organisation for the Prohibition of Chemical Weapons or the nuclear non-proliferation treaty. While not perfect, these instruments have helped check the uncontrolled proliferation of weapons of mass destruction.
Suggestion 6: Media
Finally, the media must be engaged with all of the groups above. The media cannot abdicate its duty to cover, oversee, and challenge what is said and done by all entities involved in the development of new weapons and the rules surrounding their use. Journalists covering military technology and policy must take extra care to lend a critical eye toward new weapons and their planned uses. At the same time, major outlets must be sure to give plenty of space and promotion to stories covering such issues. Avoiding media scrutiny is the first step toward these capabilities being developed in secret.
A future full of all of the technologies discussed above is all but inevitable. Such innovations will likely bring countless benefits to our society, but they also present significant risks. Putting serious thought into their development today may be far easier than trying to close the barn door in 25 or 50 years.