Using machine learning to infer reasoning provenance from user interaction log data: based on the data/frame theory of sensemaking

Kodagoda, Neesha, Pontis, Sheila, Simmie, Donal, Attfield, Simon ORCID: https://orcid.org/0000-0001-9374-2481, Wong, B. L. William ORCID: https://orcid.org/0000-0002-3363-0741, Blandford, Ann and Hankin, Chris (2017) Using machine learning to infer reasoning provenance from user interaction log data: based on the data/frame theory of sensemaking. Journal of Cognitive Engineering and Decision Making, 11 (1) . pp. 23-41. ISSN 1555-3434 [Article] (doi:10.1177/1555343416672782)

Preview

PDF - Final accepted version (with author's formatting) Download (980kB) | Preview

Abstract

The reconstruction of analysts’ reasoning processes (reasoning provenance) during complex sensemaking tasks can support reflection and decision making. One potential approach to such reconstruction is to automatically infer reasoning from low-level user interaction logs. We explore a novel method for doing this using machine learning. Two user studies were conducted in which participants performed similar intelligence analysis tasks. In one study, participants used a standard web browser and word processor; in the other, they used a system called INVISQUE (Interactive Visual Search and Query Environment). Interaction logs were manually coded for cognitive actions based on captured think-aloud protocol and posttask interviews based on Klein, Phillips, Rall, and Pelusos’s data/frame model of sensemaking as a conceptual framework. This analysis was then used to train an interaction frame mapper, which employed multiple machine learning models to learn relationships between the interaction logs and the codings. Our results show that, for one study at least, classification accuracy was significantly better than chance and compared reasonably to a reported manual provenance reconstruction method. We discuss our results in terms of variations in feature sets from the two studies and what this means for the development of the method for provenance capture and the evaluation of sensemaking systems.

Actions (login required)

View Item

Statistics

DownloadsShow export options

Export as CSVXMLJSON

Activity Overview

6 month trend

258Downloads

6 month trend

370Hits

Additional statistics are available via IRStats2.