In a radically disrupted consumer marketplace, understanding what people want and need has become more crucial than ever – and much more difficult. Surveys still play a key role in getting to the why’s of consumer behavior; but new research from GfK shows how challenging it has become to find trustworthy consumer research samples.
A variety of factors have placed growing pressure on respondent quality, from survey-taking bots and click farms to “professional” or simply inattentive respondents. Consolidation among sample providers has also narrowed options for buyers and contributed to a lack of transparency about response rates and other key quality metrics.
The new GfK study compared 21 sample sources from three major industry providers, assessing everything from double-opt-in panels and communities to “river” samples that recruit respondents directly from web traffic.
Click here to read an article based on GfK’s study
GfK found that no one sample supplier dominated in terms of quality, with all three posting fail rates of within one or two percentage points of each other on many measures. GfK also found similar levels of overstatement on three characteristics – had lasik surgery, suffers from gout, and has changed banks – that have proven to be markers of sample representativeness.
The 21 individual sample sources also turned out to be largely comparable on quality, with some river samples posting surprisingly high scores and some panels and communities delivering lesser results.
In addition, GfK found that certain demographic groups – including males, Hispanics, and those from low-income households – are more frequently under-represented in sample sources or drive higher survey fail rates.
Drawing on the new research, GfK has updated its methodology for minimizing bias among third-party sample, recommending a combination of
- Pre-survey tools (e.g., Research Defender),
- Quality Control (QC) metrics, and
- Plausibility checks
GfK found that, when used in tandem, these systems can deliver a 33% reduction in bias.
“Maintaining strong sample quality even as the marketplace changes dramatically is a journey we are taking with our suppliers,” said Stacy Baker, VP of Operations for the Americas. “Our research shows that bias and reliability issues demand our constant attention – but significant improvements are possible. We will continue to pursue this essential topic to be sure that our clients have continued access to a reliable, top-quality understanding of consumers’ wants and needs.”
