Boosting intelligence analysts’ judgment accuracy: what works, what fails?

Mandel, David R., Karvetski, Christopher W. and Dhami, Mandeep K. ORCID logoORCID: (2018) Boosting intelligence analysts’ judgment accuracy: what works, what fails? Judgment and Decision Making, 13 (6) . pp. 607-621. ISSN 1930-2975 [Article]

PDF - Published version (with publisher's formatting)
Download (284kB) | Preview


A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.

Item Type: Article
Research Areas: A. > School of Science and Technology > Psychology
Item ID: 25593
Notes on copyright: Copyright: © 2018. The authors license this article under the terms of the Creative Commons Attribution 3.0 License.
Useful Links:
Depositing User: Mandeep Dhami
Date Deposited: 13 Nov 2018 12:13
Last Modified: 29 Nov 2022 19:28

Actions (login required)

View Item View Item


Activity Overview
6 month trend
6 month trend

Additional statistics are available via IRStats2.