By Bob Fawson, Data Quality Co-op
The market research industry is putting considerable effort into data quality. Sample suppliers, agencies and research buyers are all applying checks, removing respondents and trying to keep poor-quality data out of research. Yet buyers still have relatively little confidence that poor-quality respondents have been removed before data reaches them.
That gap between effort and trust was the focus of our recent panel discussion at the Insights Association’s Corporate Researchers Conference (CRC). The discussion brought together three views of the problem: a Global Data Quality (GDQ) study of 173 research buyers across 29 countries, quality benchmarking from 46 companies across 13 countries representing 1.8 million records from the Insights Association and Data Quality Co-op’s cross-supplier respondent history.
Looking across those sources, five themes emerge about where the industry is making progress, where practices remain uneven and what buyers are asking to see.
Theme 1: Industry effort is high, trust remains low
There is a lot of quality work happening. The benchmark data shows agencies removing 28.5% of records before delivery, after suppliers have already done their own cleaning. Buyers then run another layer of checks themselves, an average of 4.9 per buyer. Seventy-one percent look for poor or nonsense open text, 67% compare results with what they expect and 50% check for straightlining.
Despite all that effort, only 33% of buyers are confident poor-quality respondents have been removed before the data reaches them.
One research buyer in Switzerland said vendors “don’t like going into details about the quality checks they do” and tend to provide jargon and the broad picture. The cleaning is happening, but buyers don’t necessarily know what was done or have a way to verify it.
Theme 2: Procurement is starting to pay attention to trust
Buyers see the main risks from poor quality as loss of trust in insights and making the wrong decision. Together, those account for 82% of responses. Only 2% selected wasted budget or rework.
We are also starting to see that concern show up in purchasing. Thirty-nine percent of buyers have paid more for stronger quality controls and 45% have eliminated a supplier over quality.
The 39% may be the number worth watching. When that many buyers say they are paying more for better quality controls, it suggests that concerns about quality are making their way into procurement decisions.
The benchmark data found a 59.6% post-survey cleanout rate for general B2B work. In a low-incidence study looking for people who make B2B software purchase decisions, perhaps 95% of legitimate entrants will screen out. Fraudsters, meanwhile, know how to qualify, which helps explain why B2B can have such a pronounced quality problem.
Theme 3: Data quality process is uneven across the industry
The number of automated in-survey quality checks being used by agencies varies considerably. In DQC data, five of the 15 agencies examined averaged fewer than two checks per survey, while three averaged between six and eight.
That variation matters given what research tells us about how quality checks work together. A recent academic study evaluated 31 individual fraud indicators along with six different combinations, or “ensembles,” of those indicators. Individual techniques performed differently, but combinations of evidence consistently performed better than relying on a single indicator. Some agencies are already using a broad set of checks, while others are operating with far fewer.
There are signs that buyers are concerned about whether current quality practices will keep pace with new forms of fraud. The GDQ buyer study ended with an open-ended question asking whether respondents had anything else to say about data quality. One in four responses mentioned AI, bots or synthetic responses without being prompted on any of those topics. Buyers are thinking about what is going to happen to quality in the future while parts of the supply side are still catching up to the present.
Theme 4: Buyers are asking for something that exists
Almost two-thirds of buyers want providers to disclose cleaning rules and counts of responses removed. Sixty-two percent want information about sample source and recruitment and 60% want information about fraud and bot detection. By comparison, only 15% identified independent certification as something they want providers to disclose.
Much of what buyers are asking for can already be provided. The GDQ feedback loop, for example, establishes common language and definitions for sharing quality information between buyers and suppliers. Other systems and processes can provide similar visibility.
The findings raise a fairly basic question: if buyers are clear about the information they want and the industry has ways to provide it, why is this level of disclosure still not routine?
Theme 5: Quality risk also exists outside of vendor relationships
The conversation about quality is often dominated by finger-pointing over who is responsible for what. The buyer data shows that companies have plenty to look at inside their own organizations as well.
Eighty-four percent of buyers field at least some surveys in-house, yet only 47% trust their own quality checks. Only 57% always or often put minimum quality requirements in their briefs, 39% include the right to reject poor data, 38% specify sample-source requirements and 24% include specific quality metrics or benchmarks.
Buyers can change those practices without waiting for anyone else. They can set specific benchmarks when they ask for bids, establish minimum quality requirements and make those expectations clear before fieldwork begins.
With AI transformation happening across organizations at different stages, companies also have to get their data provenance sorted. Research data is going to be used in more systems and in more ways. Insights teams need to know where that data came from, what quality controls were applied to it and what happened to it before it became part of the organization’s record.
ABOUT THE AUTHOR
Bob Fawson is Co-founder and CEO of the Data Quality Co-op, a new platform promoting transparency and data quality for consumer insights. An experienced executive, strategic advisor, and consumer insights expert, Bob has contributed to the evolution of insights data through executive Strategy, Operational, and Product roles at Numerator, Dynata, SSI, and Opinionology. As Board Chairman at Samplecon, Bob enjoys contributing to the next wave of insights innovation. He lives in Salt Lake City with his family, and is an avid mountain biker and skier.