"Three or four studies where I paid for prospect data, got the data back, got reports back from the vendors. And it was so atrocious, it did not pass the smell test. I just ended up removing it completely."
Gun fights, credit scores, good looking fraud? Deep discussion on data quality in the research industry in today’s episode of the Data Gurus Podcast by Sima Vasa. The panel explores the growing challenges of fraud, trust and integrity in research.
"It's about 10% of respondents in a typical consumer study that I see are ugly fraud, but a much larger proportion, about a quarter of respondents in some studies are good looking fraud."
Steven Snell, EVP, Head of Research of Rep Data.
"What we're trying to create is something conceptually like a credit score that follows respondents and suppliers around."
Bob Fawson, Co-Founder of Data Quality Co-Op
It’s not just an arms race with vendors trying to algorithmically identify fraudulent respondents while the respondents can use increasingly sophisticated tools to answer surveys (agentic AI based browsers!). It’s also a subjective question: does one open-end pasted from ChatGPT invalidate an otherwise solid response?
"You can't just have one shield, right? It's like taking a knife to a gunfight"
Dyna Boen, EVP/Managing Director of Escalent
There is unlikely a one size fits all approach to dealing with data quality. It’ll require multiple layers, continual improvements as well as some judgement.
Listen for yourself here.
FULL TRANSCRIPT via Deepgram nova-3 all the names are misspelled
SPEAKER 0 (00:00 - 00:06): And what we're trying to create is something conceptually like a credit score that follows respondents and suppliers around. I feel
SPEAKER 1 (00:06 - 00:14): like the more it can be brought to people's attention, the more we can talk about it, the more likely we are to make progress and start forcing progress a little bit.
SPEAKER 2 (00:14 - 00:23): Technology evolves and with that, the threats and how people use technology evolve typically to make money because of some of the incentives.
SPEAKER 3 (00:23 - 00:28): We want to make sure that we are monitoring survey quality after, but we still need to clean data out the end.