(Editor’s Note: This article is part of the Just Security symposium “Thinking Beyond Risks: Tech and Atrocity Prevention,” organized with the Programme on International Peace and Security (IPS) at the Oxford Institute for Ethics, Law and Armed Conflict. Readers can find here an introduction and other articles in the series as they are published.)
The downing of Malaysian Airlines Flight 17 over Ukraine in the early months of Russia’s 2014 invasion marked a turning point for technology’s role in accountability for violations of international law and human rights. A decentralized, international movement of volunteer online investigators drew attention to the potential of digital open-source information– specifically, material available on websites and social media – to help determine who was responsible for the incident, which killed all 298 people on board. The combined efforts of these investigators, alongside formal law enforcement agencies, inspired new ideas about what might be possible in a digital world. The proliferation of mobile devices, growing availability of high-speed internet access, and popularity of social media platforms made serious crimes increasingly difficult to hide.
Around the same time, a growing ambition for formal, legal accountability for atrocities was guiding international human rights bodies toward “collection, archival, and preservation of information relevant to international crimes.” Crucially, United Nations General Assembly Resolution A/71/248 in December 2016 established the International, Impartial, and Independent Mechanism for Syria (the IIIM), with a mandate to collect, consolidate, preserve, and analyze information and evidence, and to share it with competent jurisdictions. While the IIIM was the first U.N. entity to receive this mandate, it was not the last. Since 2016, the U.N. has continued to establish (through the Security Council, the General Assembly, and the Human Rights Council) new, situation-specific mandates with similar language on legal accountability.
Taken together, the turn toward accountability and the rise of internet-enabled digital investigations over the past decade have helped create an “accountability ecosystem” permeated by digital data. Within this ecosystem, documenters (including civil society actors, victims and survivors, advocates, and others) identify and gather information about events, frequently and increasingly aiming to submit it to a formal, mandated accountability entity (such as a U.N. mechanism, commission, tribunal, or national judicial proceeding).
U.N. accountability entities are a vital component of this broader ecosystem, connecting those who have gathered information and evidence of atrocity crimes with those responsible for investigating and prosecuting those crimes. Powering this system is an “evidentiary pipeline” through which information and evidence, in the form of digital data, flow from the point of identification and preservation to its eventual use in accountability proceedings. Though it is not their primary purpose, at an operational level, U.N. accountability entities are, in effect, data enterprises.
At the IIIM, I lead the teams responsible for information systems and evidence management. As the first U.N. accountability mechanism of its kind, and the first to confront the operational requirements for an “evidentiary pipeline,” the IIIM built the necessary capacities in terms of skills, processes, and infrastructure, starting largely from scratch. Along the way, we learned a number of lessons about what tech can do to help accountability efforts, as well as what it can’t.
What Tech Can Do
A colleague once defined “technology” as “stuff that didn’t exist when I was in high school.” (Email still feels like “tech” to me, but I believe that is now the minority view.) Our human tendency to direct attention to novelty can obscure opportunities that might otherwise be within reach. Much has been written about the potential of emerging technology to promote accountability, with significant focus on internet investigations and, more recently, artificial intelligence (AI). But what is often missing is discussion about less novel (and possibly less “exciting”) technologies. To reap the benefits of emerging technology, and before talking about harnessing “AI for accountability,” any organization first must build a foundation.
U.N. accountability entities, like the IIIM, must be able to collect and preserve information and evidence, search and retrieve it when needed, perform analytical activities (such as annotating, tagging, drafting, creating charts, and so on), and finally, transfer selected material to appropriate authorities. And there’s a further requirement: while performing each of those activities, an accountability entity must have the capacity to maintain information governance over the material. This includes establishing information segmentation and protection, ensuring its confidentiality, integrity, and availability, and actively monitoring it for threats or anomalies.
Information Governance
Often overlooked in human rights and accountability domains, information governance refers to establishing and maintaining security, control, and optimization of information. Information security is an obvious and essential facet of information governance, but so are records management, privacy, information retrieval, risk management, and other areas of responsibility.
An effective accountability entity must practice effective information governance. This means knowing where its information is located, how many copies exist, what purpose the information serves, how long it will be required, who has access to it, and whether and how it has been altered, among other things. In my current role, over the past six years, I have been asked dozens of times for advice on getting started. Almost always, the first question posed is: “What software do we need?” The reality is that merely deploying a piece of software is insufficient for achieving information governance. There is no magic button.
Rather, information governance must be built by investing in three key areas: identifying and retaining people with the necessary skills and qualifications; developing core processes for information management and monitoring compliance; and, yes, implementing the right tools, including software. Organizations that execute only on the latter will struggle with data, and the volume and complexity will eventually overwhelm them.
The good news is that help is available. The IIIM recruited credentialed experts in information security (including two CISSPs), data protection (multiple CIPP/Es), and evidence management (RCA, CEDS, IAPE, etc.). These colleagues work side by side in integrated teams with investigators, analysts, legal officers, and operational support specialists. Together, they have developed processes for information governance, and, eventually, selected and implemented the right software for the job.
Scanning and Optical Character Recognition
Acquiring text data, either by scanning hardcopy documents or by processing existing images of documents using optical character recognition (OCR), is hardly a new technology. But it remains an essential and often undervalued part of an evidentiary pipeline. Occasionally we are asked: “What’s the best software for OCR?” As with information governance, there is no one right answer to this question.
First, it misses a more important point: the best OCR software in the world cannot make up for poor quality scanned images. A smart, holistic approach to text acquisition starts with the scanning process, and the hardware and settings used for the initial effort. A little planning in the beginning can avoid an eventual need to rescan the document collection (a too-frequent – and expensive – occurrence).
With scanned images of sufficient quality, the OCR selection question becomes more relevant. But there’s still no single answer. Rather, which tool is “best” will depend on a variety of factors.
In the early days of the IIIM, a team of evidence officers set out to answer this question by developing a methodology for assessing OCR quality across several different techniques, and across different data sets (including documents in multiple languages, different fonts, of varying quality, etc.). The team’s work not only helped us select the right tools in that case, but also established a methodology that could be used in other contexts, where different tools might be better. Sharing such lessons with other accountability actors is particularly valuable because improvements in the quality of scanning and OCR upstream in the evidentiary pipeline affect everyone downstream. By working collaboratively, the IIIM has significantly improved text acquisition from major collections of text documents, making their contents both more accessible and searchable.
Machine Translation
Machine translation is another “old” technology, albeit one experiencing significant evolution and innovation today. For teams working on accountability issues, navigating different languages is a requirement. Obviously, the gold standard for document translation remains a qualified, experienced translator and reviser. But that is not cost-effective for very large volumes of documents in multiple languages.
Again, there is good news. Machine translation is available for many languages and can even be affordable. The guidelines for effective machine translation are similar to OCR: everything depends on the quality of the acquired text. Poorly scanned documents, resulting in a garbled or unintelligible text layer, cannot be effectively translated by any technology existing today. And like OCR, the right tool will depend on several factors including the quality of the text layer, the source and destination languages, the sensitivity of the material and so forth.
Many of the latest, next generation AI-based translation tools are cloud-driven, meaning the text must be shared with a vendor or cloud service provider. While this may be acceptable for some purposes, in confidential international criminal investigations, this is not always the case. Fortunately, previous generation, “offline” machine translation tools can often produce better results, without the need to share the data. As with OCR, accountability actors must develop a methodology for comparing and assessing translation technologies and develop a “toolkit” for translation, rather than locking in a single tool.
Finally, as with OCR, machine translation workflows are at least as important as the tools. Simply pointing translation software at a large collection of documents is insufficient. Instead, incorporating it into an integrated workflow – for example, by exporting translated text layers into a document-review platform for analysis alongside the original documents – can result in major efficiency gains.
Transcription
OCR and machine translation are essential for working with text data, but multimedia data, like audio and video, pose additional challenges. One way to make multimedia data more accessible is machine transcription. Again, this is an older, existing technology, but recent developments in AI large-language models (LLM) have created new possibilities.
Previously, transcription technologies required pristine audio quality, worked best for only one pre-identified language at a time, and ideally only one speaker at a time. Such technologies are inadequate for today’s large volumes of complex multimedia data. The latest generation of LLMs address most of these problems. They can automatically detect languages used in the audio, isolate speech even in noisy audio tracks, and identify multiple speakers. This means transcription for high volumes of complex multimedia data is no longer beyond reach.
But is it feasible? The answer depends on the operating environment. For a turnkey solution, options may require sharing the data with a cloud-service provider and may be costly. But for organizations with a strong information governance foundation and the right skills on staff, implementing transcription can be an affordable and effective.
At the IIIM, automated transcription of multimedia materials remained on our “to-do” list for years, postponed due to cost and data-sharing concerns. Recently, with the public availability of next-generation open-source AI systems, we were able to rapidly adapt and deploy effective, multi-lingual transcription systems in our pipeline. These systems do not connect to the cloud, do not require any additional software costs, and run on our existing hardware infrastructure.
Our ability to deploy this tech effectively depended not only on the availability of the software, but also on the existence of a solid foundation: the wholly non-novel technology of a set of offline processing machines, a team with the right skills, and a flexible, modular workflow that could anticipate, adapt to, and incorporate advances in technology as they arrived.
What Tech Can’t Do
Technology can do a lot to aid accountability efforts. It can make an evidentiary pipeline possible and the accountability ecosystem more effective. But there are some things technology can’t do, at least not yet.
First, there are limitations even for the examples above. Crucially, for cultural, economic, and technical reasons, some languages are better supported than others. This has major implications for accountability efforts. Technological innovation in text and language tools has been focused primarily on a relatively small number of languages, usually those with large, digital, readily accessible datasets of existing text, and those presenting a potentially lucrative market opportunity. These innovations underperform, or may not work at all, when applied to language data outside of the major datasets. The situation is much better than it used to be, but this remains a major limitation. As a result, conflicts where lesser-represented languages predominate the evidence base may be under-investigated.
To improve accountability efforts for a broad, truly global range of situations, funders and decisionmakers must go beyond simply mandating accountability initiatives to use advanced technology. They should also promote the more inclusive, global development of existing technologies to increase linguistic reach and effectiveness.
Second, while cloud-based technologies are rapidly evolving and conquering the market, the implications for accountability mechanisms are not certain. Cloud processing of data is a powerful tool and may be the right solution, but this should never be a foregone conclusion. Often, and increasingly, organizations are led to believe that cloud systems are more secure and economical than “on-premises” systems. This is sometimes true, but not always. Funders and decisionmakers should resist the urge to require accountability actors, in the name of innovation and cost-saving, to widely adopt cloud technologies. Instead, they should encourage methodical, realistic assessments of their needs and capacities.
Finally, AI isn’t magic, nor is it a substitute for investigative strategy. At the IIIM, we are frequently asked if AI can help make sense of complex data, like large collections of videos. Advances in computer vision, combined with AI-enhanced transcription, certainly open up promising new approaches to exploit and comprehend large video datasets (like glyph recognition and clustering). But such approaches must be driven by a clear investigative strategy, with well-defined goals and parameters, and experienced, human investigators behind the wheel.
Funders and decisionmakers should temper their expectations for accountability efforts that entail large-scale collection of complex data, like video, grounding them in expert advice and pragmatic thinking. If investigators expect AI technology to examine a set of, say, 100,000 videos and then identify the most “relevant” items, they will be disappointed. AI and other investigative and analytical technologies can do a lot of things, but they can’t yet read our minds, and while they can make some passable guesses, they are not able to do the strategic thinking necessary for an investigation.
Beyond Hype: Real Progress Requires Mastering Basics
Over the past decade, the global accountability ecosystem has evolved significantly. With the proliferation of open-source investigations uncovering digital evidence of atrocity crimes alongside the establishment of U.N. mechanisms to centralize, govern, and share such evidence, the “evidentiary pipeline” connecting decentralized online investigators to courtrooms is finally taking shape.
But each stage in this pipeline, and the technology that powers it, needs to work effectively. Today’s preoccupation with AI and emerging tech risks overshadowing the operational needs and strategic objectives of global criminal justice. To achieve those objectives, the ecosystem needs smart investments in three areas: (1) people with the data skills required to establish and maintain information governance, (2) appropriate tools and infrastructure, especially non-novel, established tech, and (3) workflows to integrate them. With coordinated investments in each of these areas, funders and decisionmakers can move beyond the hype and ensure that technological advances more effectively serve the real needs of accountability efforts.