Using Paradata to Evaluate Online Performance

Abstract
Computer aided data collection systems produce a variety of data alongside the actual data collection. In the survey field such data are called paradata and are used to monitor fieldwork, drive responsive design decisions, and help detect problematic cases. In web surveys, paradata can help guide respondents through a survey, and can be used to target specific respondents who may need assistance. One such measure that has been used to identify individual respondents that may need attention is mouse movements. Mouse movements have been used in e-learning and web design to measure interest and uncertainty and in survey research to measure data quality and respondent difficulty answering questions. Specifically, researchers have measured the total distance traveled, the cursor’s trajectory, and specific patterns of movement. The study presented here aims to develop automated procedures for detecting and quantifying difficulty indicators. We use participant’s mouse movements, logged as they complete an online questionnaire. The questionnaire includes factual, opinion, and problem-solving questions with a variety of response formats, such as radio buttons and slide bars. We vary experimentally the difficulty and complexity of items between respondents and draw upon administrative data to validate the veracity of the responses given. We show how complexity affects response behavior and the degree to which indicators of insecurity are related to the veracity of answers.

Frauke Kreuter is Professor in the Joint Program in Survey Methodology at the University of Maryland, Professor of Methods and Statistics at the University of Mannheim, and head of the statistical methods group at the German Institute for Employment Research in Nuremberg. Previously, she held positions in the Department of Statistics at the University of California, Los Angeles, and the Department of Statistics at the Ludwig-Maximillian’s University of Munich. Frauke serves on several advisory boards for National Statistical Institutes around the world, and within the Federal Statistical System in the United States. Frauke is also a Gertrude Cox Award winner, which recognizes statisticians in early- to mid-career who have made significant breakthroughs in statistical practice, winner of the inaugural Links Lecture Award and elected fellow of the American Statistical Association. Additionally, she is co-founder of the Coleridge Initiative (https://coleridgeinitiative.org), founder of the International Program in Survey and Data Science (sruvey-data-science.net), and co-host of the digitalization podcast www.digdeep.de

Date: 
Tuesday, March 12, 2019 - 2:00pm
Building: 
2121 Berkeley Way West
Room: 
1212