The CNA Corporation, a non-profit research and analysis organization which runs the Center for Naval Analyses, recently published an important report by Larry Lewis, entitled Drone Strikes in Pakistan: Reasons to Assess Civilian Casualties. Below is my summary of its major conclusions, and some of my concerns and criticisms. [For other (more positive) reactions to the study, see Professor Charli Carpenter’s and Professor Derek Gregory’s posts.]
I. Summary of major conclusions
Conclusion 1: The US government underestimates civilian casualty rates in Pakistan.
The report identifies two types of mistakes—legal errors and factual errors—that are part of current US policy.
Legal error: According to the report, the administration classifies some casualties as combatants when they should instead be presumed civilian unless proven otherwise. The source of this error includes: the legal definition of civilians, the presumption that should apply in cases of doubt, and the factors that should be used to rebut or uphold that presumption.
Factual error: According to the report, the US government tends to underestimate the actual count of civilian casualties because of flawed methods of verification (e.g., insufficient HUMINT and follow-up investigations on the ground).
Add-on no. 1: The study makes an interesting analytic point: the failure to count civilian casualties properly makes it difficult for the administration to meet its legal obligations to avoid harms to civilians. The report states:
“[T]he ability of a military to do everything possible to avert civilian harm is limited by its ability to consistently recognize instances of civilian harm. If the problem of civilian harm is not recognized and well-understood, then the actual scale of civilian harm will be misunderstood and measures will not be put in place to address it effectively.”
Added-on no. 2: The study also critiques counts by organizations such as the Brookings Institution (footnote 45) because they disregard drone casualties who are categorized as “unknown” status and rely instead only on cases of confirmed civilians.
Conclusion 2: Armed drones have resulted in greater civilian casualty rates than manned aircraft under certain conditions.
This conclusion, if accurate, explodes a myth about the accuracy of drones.
Relying on an earlier classified study by Lewis (for which only the executive summary is available), the new CNA report states that engagement by drones in Afghanistan (2010-2011) were “ten times more likely to result in civilian casualties” than engagements by manned platforms. The report explains that these statistics “highlight[] the fact that platform characteristics alone are not the driver of a decreased likelihood of civilian casualties.” The take-home point is that an integrated process that properly assesses civilian casualties and feeds that information back into training and planning is the key.
Conclusion 3: The CIA-directed drone program (in Pakistan) has involved a higher civilian casualty rate than the Pentagon’s drone program (in Afghanistan).
This conclusion, if accurate, also explodes a myth about the accuracy of the CIA-directed drones operations compared to the Pentagon-directed drone operations. That said, the author is less explicit about this finding and, indeed, never refers to the “CIA.” But it is fairly easy to piece together the conclusion from the following three statement in the report.
1) Congress is debating whether the drone campaign should be “shifted entirely to the U.S military” or “continue to be conducted in part by another element of the U.S. government.”
2) That policy question can be usefully addressed by asking: “how well is the current drone campaign in Pakistan doing in minimizing civilian harm?”
3) The answer to that question is: The Pakistan drone program’s “civilian casualty rates are significantly higher than those seen for drone and overall counterterrorism operations in Afghanistan conducted by U.S. and international forces. While rates for the two countries are not necessarily directly comparable, the operations in Afghanistan illustrate that lower rates can be achieved during counterterrorism operations in general.” And the Executive Summary puts it more bluntly: “drone strikes in Pakistan were more likely to cause civilian casualties on average than drone strikes by military forces in Afghanistan.”
II. My disagreements and quibbles
1. An elephant in the room is the rate of civilian casualties from US drone strikes in Pakistan in year 2013. As Derek Jinks and I pointed out in an earlier post on a related topic, it can be misleading to aggregate the number of drone civilian casualties across multiple years—because that total masks the 2013 number of civilian casualties in Pakistan, which is close to zero, if not zero. Yes, zero, naught, nil, nada. The CNA study admittedly includes one Table that shows that the number of civilian casualties plummeted in 2013. The study, however, does not raise the implications of this aspect of the data, or grapple with how the data undercut some of the conclusions reached in the report. Moreover, the civilian deaths from drone operations in Afghanistan reportedly increased by 300% in 2013.
How then should we think about the relative performance of the CIA-directed ops (in Pakistan) versus the Pentagon ops (in Afghanistan) given the recent trends lines (a drop to zero versus a threefold increase)? The report doesn’t tell us. And how should we think about the report’s statement that “there remains room for improvement” in the Pakistan operations — when viewed specifically in light of the 2013 record of success?
2. A key statement of the report is that “U.S.-reported levels of civilian casualties for operations in Pakistan differ significantly from nearly every other estimate available” (my emphasis added). Indeed, that is arguably the thesis of the study. But that statement is a bit misleading. The US does not report levels of civilian casualties in Pakistan. Indeed, the US government does not even acknowledge that a drone operation in Pakistan exists.
3. The administration’s legal errors?
In a New York Times Op-ed, I have written that the administration’s definition of lawful military targets is flawed. But my point was different than the one Lewis makes. So, what is the specific basis for Lewis’s claim that the administration is using the wrong legal criteria? First, he cites the 2012 New York Times report that the administration “counts all military-age males in a strike zone as combatants… unless there is explicit intelligence posthumously proving them innocent.” But it was never clear whether that news report was accurate. What made it especially newsworthy at the time was the administration’s failure to disavow the report. That said, the White House, belatedly but eventually, did disavow it. Also, Lewis suggests that the administration is not using the principle of precaution during attacks (pp. 21-22), but his reference to press reports does not prove this. The best evidence to support (and interrogate) that proposition is unfortunately classified.
4. The administration’s factual errors?
It is unclear whether Lewis has hit the mark about efforts that the administration currently undertakes to verify civilian casualties after a strike (e.g., lack of HUMINT). Once again, we don’t know because the program (including any post-strike assessment) is covert. And it is unclear from the CNA study how much is accessible to Lewis. It should be noted that a recent report by Glenn Greenwald and Jeremy Scahill criticized the administration for the opposite: using more thorough methods, including HUMINT, after deadly strikes but not beforehand.
5. The report potentially overestimates the number of civilian casualties in one respect.
The report calls for counting all casualties of “unknown” status as civilian – and criticizes other organizations for failing to use that approach. The basis for the approach, according to the report, is that the law of armed conflict requires a presumption that an individual is civilian when their status is in doubt (pp. 9 & 13).
However, it is one thing for such a legal presumption to operate as an ex ante duty of military operators conducting attacks. It is another thing for such a presumption to apply to statisticians and analysts measuring civilian casualties after the fact. Indeed, it is highly unlikely, if not statistically impossible, that 100% of the unknown cases were all civilians.
6. The report potentially underestimates the number of civilian casualties in another respect.
The report explains that the leading organizations tracking casualty rates include a minimum estimate and a maximum estimate. The report is explicitly conservative: it relies only on the minimum estimates in drawing its major conclusions. As justification for this methodological choice, the report explains that organizations such as the Bureau of Investigative Journalism “regards the lower end of its estimates to be more likely to approach actual values.” That sounds right. Nevertheless, presumably the actual value lies somewhere between the lower end and the maximum (even if it is closer to the former).
[What would result if we corrected the overestimation flaw and the underestimation flaw? I wish I knew the answer, but I know it’s complicated. Because these are two very different kinds of errors, it is unlikely that they simply cancel each other out.]
* * *
The CNA study is a balanced report in many respects. Indeed, the fact that it may include an overestimation and an underestimation civilian casualties is, ironically, evidence of that fact. Unfortunately the best evidence for analyzing the report remains classified, and it is unclear to what degree the study itself relies on classified information. As Charli Carpenter said in reaction to the earlier study by Lewis, “[t]he government’s failure to release the information is the real story here.”