News & Events

Subscribe to email list

Please select the email list(s) to which you wish to subscribe.

You are here

Semi-automated categorization of open-ended questions

Thursday, February 6, 2014 - 16:00
Matthias Schonlau, Professor, Department of Statistics and Actuarial Science, University of Waterloo
Room 4192, Earth Sciences Building (2207 Main Mall)

Text data from open-ended questions in surveys are difficult to analyze and are frequently ignored. Yet open-ended questions are important because they do not constrain respondents’ answer choices. Where open-ended questions are necessary, sometimes multiple human coders hand-code answers into one of several categories. At the same time, computer scientists have made impressive advances in text mining that may allow automation of such coding.  Automated algorithms do not achieve an overall accuracy high enough to entirely replace humans.  We categorize open-ended questions using text mining for easy-to-categorize answers and humans for the remainder using expected accuracies to guide the choice of the threshold delineating between “easy” and “hard”. We illustrate this approach with examples from open-ended questions related to respondents’ advice to a patient in a hypothetical dilemma, a follow-up probe related to respondents’ perception of disclosure/ privacy risk, and from a follow-up survey from the Ontario Smoker’s Helpline. Targeting 80% combined accuracy, we found that 59%-80% of the data could be categorized automatically.