Survey Data Quality in an Evolving Market Research Ecosystem

As an industry, are our metrics for what constitutes acceptable data quality changing? At a minimum, the market research industry is determining if metrics for survey data quality need to adapt across device types as smartphones and tablets comprise higher and higher percentages of market research traffic. Chuck Miller (DM2) and Mark Menig (TrueSample) recently presented a study on a wide variety data quality issues as influenced by the evolving market research landscape. The findings were shared at CASRO’s Digital Conference last month, and the study touched on many broad aspects of data quality. Below are a few of the findings that we found most interesting.

The study utilized ratings in the form of a concept study, where ratings were requested for the desirability of owning Google Glass. The study used grids of varying scales; 3 points, 5 points, and 7 point. All of grids could render across devices without scrolling, and all were optimized to display uniquely to each device. The study looked at the concept of viewing time, answer consistency, open end quality and length, and other factors in order to determine an overall measure of data quality. The overall sample size was 1,800 respondents who were evenly split into short, medium, or long surveys with quotas established for type of device in each group. TrueSample was used for data validation.

Influence of Sources

Samples were provided by an increasing array of sources and types including; enterprise and customer panels, traditional access panels, river sample, social media, and lists. One topic offered up was data quality as correlated with the type of sample utilized.

  • There were few differences in data quality between a traditional online panel and a virtual or managed panel.

  • Respondents from river sampling (which consists of individuals who do not opt in to a panel, but only opt in to complete a project) showed significantly lower data quality than a traditional OR a virtual / managed panel.

Intuitively, it makes sense that a double opt-in panelist is going to be a more cooperative individual with most respondents also consenting to profiling and some to downloading of apps. Most project managers have little control, however, since river sampling is frequently used by the sample provider to help fill unmet quotas and to help reduce bias of a double opt-in approach. For many studies, river sampling may be an unavoidable and unseen part of the sampling process. Limiting sample to a DOI (double opt-in) managed panel approach is not feasible for many projects. This means that quality metrics should and more than likely will prevail for the forseeable future.

Mobile Data Quality

But what about the impact of mobile on data quality? Mobile respondents can come in from sources like traditional or managed panels, river sample, or even customer lists. One area of controversy is around the use of large-scale rating scales, both in terms of displaying large rating grids as well as maintaining benchmarks. Net Promoter scores, a standard in the industry for measurement of customer loyalty, utilize 11-point scales that require scrolling on most smartphones. Some research-on-research has demonstrated that such ratings, both NetPromoter and other types of rating scales, do not benchmark. Instead, respondents rate them lower correlating to the decreased size of their device.  This means that a rating of 10 on customer satisfaction might occur across desktop devices while it might achieve only an 8.5 (example only) across smartphone respondents. Any discrepancies could be due to a variety of issues — among them reduced accuracy in data entry or general inattentiveness when using a mobile device. Thus, this study also attempted to determine if data quality differences could be measured when grids of various scale sizes were rendered across device types (desktop, tablets, and smartphones).

The time spent viewing a concept was inversely proportionate to screen size. This again makes intuitive sense, as a respondent using a mobile device is less likely to be using the device for lengthy exercises. However, this did not necessarily impact data quality. Mobile respondents otherwise showed the same general data quality as other types of online respondents. Data quality issues varied across specific demographic and behavioral respondent groups.

  • 18-34 year olds yielded lower response quality data than did other age ranges.
  • Those shopping for the concept provided the highest quality data, while those who were deemed “influencers” or “enthusiasts” of technology provided relatively lower quality data.
  • The lowest quality data came from individuals who “preferred not to answer” on questions where this option was presented.
  • There were no data quality differences between those taking shorter versus longer versions of the same study (although in this context it should be noted that the dropout rate is significantly higher on longer studies, as reported from a multitude of sources).
  • Those with college degrees provided the same quality of data (slightly worse, but not significantly so) than those with high school degrees.
  • The best data overall came from female shoppers who were responding via tablets on short versions of the survey.
  • Across browsers, the lowest quality data came from respondents who were using Windows. Fast concept viewing time was highly correlated with lower quality data, but slower viewing times did not impact data quality.

Overall, it is clear that as our benchmarks change, our measures of acceptable data quality may also change. Increasingly, data is based upon behavioral attributes which do not bring quality into question. A TV show was viewed or it was not, a shopper visited a particular retailer after viewing online or they did not. As we merge “big data” with other data sources, the quality measures that are used may be modified.

– Erica Dent Google+

Posted on April 11, 2014 in Blog

Share the Story

About the Author

Erica is the Marketing Manager for Kinesis Survey Technologies. She loves bringing you the latest from the industry and engaging on social media! You can even reach Erica directly, to discuss this blog on
Back to Top
Learn more about the products and get a free demo!CONTACT US