Research answers questions about questions and answers

In Don Dillman’s world, there is no such thing as a simple question. Context matters. Context always matters.

How else do you explain why 70.2 percent of WSU students in one Web survey said they typically study 2.5 hours or less per day, while in answer to exactly the same question, only 28.9 percent of students in a separate Web survey conducted simultaneously said they study that much or less? A third Web survey of students, answering the identical question, revealed that 42.2 percent of respondents studied 2.5 hours or less.

The answer, says Dillman, an expert in survey methodology, is in the answer. Or rather, the choices respondents were given when deciding their answer. In the first survey, students were given six possible responses in a low range, from half an hour or less to more than 2.5 hours. In the second survey, respondents were also given six choices, but in a high range, starting at 2.5 hours or less and going up to 4.5 hours or more. In the third survey respondents were not given choices, but simply supplied their own answer.

Range as an influence
In a 2005 paper titled, “Context Effects in Web Surveys: New Issues and Evidence,” Dillman and his co-authors, graduate students Jolene Smyth and Leah Melani Christian, argue that the range of possible answers provided a context for students to determine their own answers. For instance, students who believed their study habits were about average would choose an answer in the mid range, while students who believed they studied above or below average would choose answers at the high or low end of the scale, regardless of what the scale was.

“This study is about a fairly harmless type of activity,” said Smyth, but understanding how context affects the survey process is crucial to obtaining useful information. “It speaks to the science of surveying and the need to be more rigorous about it,” she said. “Are we getting the information we need?”

Open-ended questions
While asking open-ended questions would seem to elicit the least biased information, Smyth said that isn’t always practical.

Open-ended questions “are often incredibly difficult to code and summarize,” she said. If public opinion polls had relied on open-ended questions during the 2004 election season, she said, “The election would have been long gone before we ever figured out what was going on.”

Dillman, a Regents’ Professor and the Thomas S. Foley Distinguished Professor of Government and Public Policy, is one of the country’s leading authorities on the development of mail, telephone and Internet survey methods. During his 36-year career at WSU, Dillman has authored hundreds of books and publications, including what some consider the bible for conducting public opinion surveys: Mail and Telephone Surveys: The Total Design Method (1978.)

But Dillman doesn’t consider his book immutable. There is always something else to consider.

Next Story

Provost selection process ongoing

WSU expects to name its next provost before the end of April. President Kirk Schulz is actively considering two finalists, with feedback provided by the university community being a key factor in the decision.

Recent News

E-tongue can detect white wine spoilage before humans can

While bearing little physical resemblance to its namesake, the strand-like sensory probes of the “e-tongue” still outperformed human senses when detecting contaminated wine in a recent WSU-led study.

Employee Assistance Program hosts special sessions, April 17

Washington State Employee Assistance Program Director Jennifer Nguyen will lead two discussions tomorrow on the topics of change and personal wellbeing. Both presentations will be livestreamed.

The 2024 Seattle Experience highlights resiliency

The faculty-led alternative spring break program hosted by the College of Arts and Sciences helps students develop professional skills while networking with business leaders in the region.

Deadly bacteria show thirst for human blood

A WSU-led study has found the some of the world’s deadliest bacteria seek out and feed on human blood, a phenomenon researchers are calling “bacterial vampirism.”