Statement on ISW Methodology

Statement on ISW Methodology

May 4, 2023

The Institute for the Study of War (ISW) welcomes robust and rigorous debate on the issues ISW covers and is especially grateful for the interaction with and feedback from those who engage seriously with the methodological challenges of performing intelligence from publicly available information. ISW’s entire team shares a passion for sound tradecraft and a desire to engage openly regarding our methodology, including our analytic and forecasting successes as well as our limitations. We seek to raise the general level of discussion of national security and especially military issues and are grateful for the aid of all who contribute to that effort.

A few key takeaways regarding ISW’s approach can help inform readers about ISW’s work and establish constructive topics for engagement:

• ISW applies rigorous intelligence tradecraft that reflects best practices drawn from the US intelligence community. ISW uses technical language from the military and intelligence community in its products in order to maintain the rigor of its tradecraft and communicate with the greatest possible precision.
• ISW’s extensive, daily collection and analysis provides a highly tested baseline of existing patterns, enabling relatively fast identification of anomalies and inflections. ISW’s ability to make a judgment on a given piece of data is informed by its months of prior study.
• ISW has applied its proven methodology for analyzing conflicts for over 15 years. ISW’s Ukraine work applies the same tradecraft ISW has used to analyze and map other wars, including ISW’s extensive coverage of the war against ISIS in Iraq and Syria.
• ISW is fully independent of the US and foreign governments, takes no money from these entities, and is committed to its independence as a key source of objectivity.
• ISW aims to inform, not convince, audiences and makes available the sourcing of each product to ensure readers can engage directly with the underlying data and form their own conclusions, especially amidst disinformation campaigns. ISW uses no classified or privileged information in its products.
• ISW analysts interpret available data to form assessments – they do not simply report facts. ISW explicitly states its specific confidence level in each assessment whenever applicable and encourages our readers to include it when reporting our assessments in order to minimize the risk of misinterpretation. ISW will publish low-confidence assessments if its analysts judge the issue being assessed to be sufficiently important, timely, or risky to warrant a flag even as ISW continues to evaluate new information.
• ISW innovates new ways of collecting, analyzing, and visualizing public data. ISW established a new geospatial program to scale its mapping effort after the Russian invasion and to drive mapping innovation, which was generously enabled by the Dr. Jack London Geospatial Fund.

ISW is committed to providing accurate, granular, and timely intelligence from open sources on threats to US national security interests, including Russia’s brutal invasion of Ukraine. The purpose of ISW’s work is to inform US and allied decision-makers and to educate public audiences regarding these threats. ISW products help to visualize and explain fast-moving conflicts as well as to interpret their trajectory and significance. ISW’s publication series on the war in Ukraine is an exemplar of ISW’s institutional values and methodology. As such, it is a useful means of understanding ISW’s approach to intelligence analysis.

ISW applies industry-leading tradecraft. ISW’s analytic program trains and applies intelligence methodology that emphasizes source evaluation and characterization, structured analytical techniques, and other foundational principles. Retired members of the intelligence community have informed this tradecraft over many years and ISW welcomes engagement with current intelligence experts regarding ISW’s methodologies. ISW extends this tradecraft into its written product. ISW uses military and intelligence technical language and concepts with precision not only to communicate within a framework that US and allied governments can easily interpret but also, more importantly, to maintain the rigor and discipline of ISW’s analysis even in external communication.

Independence and transparency are core ISW values and are applied to all ISW products. ISW accepts no US or foreign government money and uses no classified material from any source to produce our analysis. ISW’s independence is a critical enabler of its objectivity. ISW also aims to demonstrate what is possible to achieve in the public domain through the careful application of rigorous tradecraft, even though it is often more difficult than alternative methods. The proven utility of ISW’s products to US and allied governments demonstrates the value of ISW’s approach. ISW’s analytic program is attractive to both aspiring and former members of the intelligence community for this reason.

ISW’s commitment to informing audiences – not convincing them – is demonstrated by ISW’s extensive sharing of its sourcing. ISW makes available extensive footnotes detailing the evidence used to inform each assessment and has done so for Ukraine since Russia’s current invasion began in February 2022. The investment in this level of evidence-sharing reflects ISW’s commitment to ensuring that readers can form their own judgments on the basis of the evidence and ISW’s reasoning and do not need to take ISW’s word for anything. ISW also analyzes the various information operations directed at and interacting with the war in Ukraine and makes its sourcing available to provide transparency into ISW’s own conclusions.

ISW is not a news organization, however. ISW products do not simply report facts but rather interpret available data to form assessments regarding what is occurring, has occurred, and could occur and the implications. Understanding the distinction between fact and assessment is critical for responsibly using ISW products. Assessments can be wrong. In fact, if an analyst is never wrong it is typically a sign that the analyst is doing more reporting than assessing. ISW analysts make calls about what they are seeing and its significance on the basis of data which is invariably incomplete and often contradictory. That data includes but is not limited to the data layers behind ISW’s Ukraine map. ISW analysts also form and continually revise strategic assessments of key actors’ intent, decision-making, capacity, and other factors. This holistic approach is intensive and comprises an analytic backend that is less visible than ISW’s maps but no less vital. ISW’s assessments are sometimes wrong, of course. Whenever ISW concludes that it has made a mistake it publishes a formal correction.

Moreover, analytically sound mapping of war itself requires more than mere data aggregation, as maps are also analytic judgments in visual form. More information on ISW’s mapping methodology is available here.

ISW explicitly states its specific confidence level whenever applicable and encourages our readers to include it when reporting our assessments in order to minimize the risk of misinterpretation. ISW does not refrain from publishing low-confidence assessments, which often distinguishes ISW from other organizations. Instead, ISW will publish at low confidence if its analysts judge the issue being assessed to be sufficiently important, timely, or risky to warrant a flag, which ISW then caveats as necessary and then re-evaluates as new information emerges. This practice enables early warning that provides a valuable means of scoping the aperture of decision-makers even as the situation develops. It is particularly important amidst highly contested information environments like the one in Ukraine.

ISW’s diligence in performing daily analysis of the available data regarding the war in Ukraine is a powerful enabler of ISW’s accuracy. ISW’s teams maintain a constant stare at not only the battlespace but also the information environment. This approach often enables analysts to make relatively fast judgments of new information based on the established patterns ISW monitors closely. It also enables rapid detection of change, anomalies, and inflections which is a key marker of ISW’s unique success in understanding and explaining the war in Ukraine and other conflicts. It has been a critical enabler of ISW’s forecasting accuracy for many years. For example, ISW warned that ISIS would attempt to seize territorial control in Iraq and that Mosul was vulnerable seven months before ISIS launched its blitz offensive in June 2014. 

ISW has applied this proven methodology for analyzing conflicts for over 15 years. ISW products, including detailed maps, have consistently been utilized by decision-makers and news organizations covering fast-moving and complex national security challenges because of their accuracy and consistency. ISW’s maps of ISIS control in Iraq and Syria provided the leading source of insight and visualization of the war against ISIS after Mosul’s fall and were broadcast widely by international media organizations and leveraged directly by US Central Command in public communications to explain the status of the fight. ISW has extensively mapped the war in Syria, produced detailed maps of the war in Afghanistan at key stages, and has mapped the fight in Ukraine in various ways since the Russians first invaded in 2014. 

ISW continues to innovate new ways of collecting, analyzing, and visualizing public data in order to answer intelligence questions that provide US and allied decision-makers with timely and reliable information. New social media platforms, commercially available data, censorship and disinformation efforts, automation tools and other software all pose new and interesting analytic challenges and opportunities for studying conflict. ISW routinely grapples with these and other opportunities to enhance our tradecraft and reach new insight.

ISW scaled up its mapping effort to meet the demands posed by the Ukraine war in 2022 and established a new geospatial program at ISW, generously enabled by the Dr. Jack London Geospatial Fund. ISW’s Geospatial team is now a discrete entity that works in support of ISW’s analytic team, providing granular tracking and visualization of the war that informs the assessments ISW analysts produce. This configuration – a mapping team in support of an analytic team – represents a consolidation of ISW’s traditional approach to analyzing conflicts and will enable ISW to continue to achieve greater scale by applying this tradecraft to new portfolios, including China. It also enables ISW’s ongoing innovation of new mapping approaches.

ISW is extraordinarily proud of what its dedicated team has achieved in covering the war in Ukraine for over 430 days. ISW remains grateful to every reader as well as to our generous supporters who enable our work. We also appreciate the hard work of the many other responsible organizations and experts who are adding important insights to the public debate and discussion. ISW’s teams are available, as ever, for inquiries into ISW’s methodology or regarding its products and can be reached via [email protected].