Boosting intelligence analysts’ judgment accuracy: what works, what fails?

Mandel, David R., Karvetski, Christopher W. and Dhami, Mandeep K. (2018) Boosting intelligence analysts’ judgment accuracy: what works, what fails? Judgment and Decision Making, 13 (6). pp. 607-621. ISSN 1930-2975

[img]
Preview
PDF - Published version (with publisher's formatting)
Download (284kB) | Preview

Abstract

A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence
organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH)
to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments.

Item Type: Article
Research Areas: A. > School of Science and Technology > Psychology
Item ID: 25593
Notes on copyright: Copyright: © 2018. The authors license this article under the terms of the Creative Commons Attribution 3.0 License.
Useful Links:
Depositing User: Mandeep Dhami
Date Deposited: 13 Nov 2018 12:13
Last Modified: 05 Apr 2019 10:31
URI: https://eprints.mdx.ac.uk/id/eprint/25593

Actions (login required)

Edit Item Edit Item

Full text downloads (NB count will be zero if no full text documents are attached to the record)

Downloads per month over the past year