Insights on asking better survey questions
Suppose you want to find out what your students think about your course; maybe not the content, so much, but how you’re teaching it. It sounds simple enough, but even the simplest of surveys requires forethought.
“Writing good questions is not easy,” warns Don Dillman, WSU Regents Professor and expert in survey methodology.
“The first thing I usually suggest to people is to keep questions short and simple,” Dillman said. “Sometimes people use several sentences when one sentence will do.”
“I always ask myself for each question in a survey, ‘Will it make sense to the respondent?’ ” he said. “If it doesn’t make sense to them, it probably isn’t going to produce a good answer.”
According to Dillman, long or multi-part questions have two disadvantages. One is that some readers simply won’t read it all; they’ll answer the first part without ever considering the rest. Second, long questions provide more grist for various interpretations.
Dillman’s second tip is to stay “within the culture.” Understand who your respondents are and what their expectations are. For instance, if you want students to rank class objectives by relative importance, give them a scale that they are comfortable with, not one that seems overly complicated and abstract.
“Recently I saw someone write a question that asked people to allocate 100 points to the five most important objectives the respondent hoped to realize by taking a particular course,” Dillman said. “Most people don’t have five distinct objectives, and allocating points among abstract objectives was hard for them to do.” Some didn’t answer the question, he said, and others thought it was silly.
A third suggestion, Dillman said, is to verbally test drive your survey question, particularly if it is innovative. If you ask the question and then listen while people talk through their answers, he said, you’ll have a much better idea whether the question elicits the type of information you want or need. If a question provokes more questions than answers, he said, you know you need to rework it.