Online Sample Fraud: Causes, Costs & Cures - Continuing the Conversation, March 25, 2022
Transcript Courtesy of Focus Forward & FF Transcription
Panelists: Charles Guilbeau, Vice President, Consumer and Shopper Understanding, Ferrero North America; Tia Maurer, Group Scientist, Products Research, Procter and Gamble; Joanne Mechling, Operations Research Lead, Investor Research and Insight, Vanguard; Nicole Mitchell, Senior Knowledge Specialist, Dynata; Steve Valigosky, Senior Director, Growth and Strategy, Kantar Profiles; Lisa Weber-Raley, Chief Research Officer, Greenwald Research.
Moderated By: Melanie Courtright, CEO, Insights Association
Melanie Courtright: Hello and welcome everyone, great to see you all. Very excited to be having this Online Sample Fraud discussion. We would really encourage you to be very active in the chat as we go through this. If you have a differing opinion, a resource that you'd like to give a link, please be very active in that, put as much as you want. Well, it is two after and I know we have a lot to cover today, so I'm gonna get started with just a little bit of housekeeping. First of all, welcome, thank you all for being here. This is the Town Hall on Online Sample Fraud - Causes Costs and Cures. And it is a continuing discussion from the conversation that we had a couple- On February 11th, I believe it was, or March 11th. February 11th? Something like that. So, as you know, we had that conversation and it was very active. And, so, we've scheduled this conversation to continue with some very active people in the chat from last wave. We are recording this town hall. It will be both recorded and made available on our virtual town hall archive, but it will also be transcribed, and the transcription will also be available. So, if you need to go back and re-listen or share something with others, you'll have an opportunity to do that. We'd, again, love to hear from you. Please, please be very active in the chat. We want you to express your point of view. And last time when we did this, we chose some of the most active people who had the sort of the best contributions. And, so, it also gives you an opportunity for us to see this is something you're passionate about, maybe we'll tap you for a conversation down the line. We will leave time for Q&A at the end from you. So, again, if you don't have your Q&A pod up, bring it up. And, if you have a specific question for the speakers and the panelists, please drop it in there. So, now, just a little bit of background from me before I introduce your panel. On February 11th, we held a very active town hall. It was presented on- With CASE, Coalition for Advancing Sampling Excellence, based on some really great work that CASE had done, presented the research, the results of their research into online sample fraud and quality. If you were unable to attend that, we encourage you again to go to our town hall archives, listen to that or read the transcripts. The results were provocative in three key areas. First, we talked about de-duplication technology, the kind of summary there was that de-duplication is relatively mature, but there remain some issues around mobile devices. Then we moved on to fraud technology, the summary of the fraud technology, and actually looking at finding and flagging true fraud was that the technology is still relatively immature. And you can get some very different results based on which fraud technology you're using, with across the fraud technology resources at the time that we ran the survey, you would get different people flagged as fraud based on which partner you were using. And, so, there's an opportunity for us as a profession to build out a more mature product there. And then, finally, there was a large discussion around participant frequency. Results indicate a resurgence of some very active participants and a consequential impact on data quality. And, so, as you can imagine, those are three big topics, and so the conversation was very passionate. And you guys said you wanted to continue the conversation. Many of the people here on your panel were really active and wanted to continue the conversation, so that's what we're doing today. For each of you, would you tell me, what stuck out to you as one of the key areas that sort of piqued your interests of the results from last- From the February 11th town hall? Tia, why don't I start with you?
Tia Maurer: Sure, I'm happy to go. So it was surprising to me as we were analyzing the data. And I know that we call them affectionately professional panelists at Procter & Gamble, and I'm sure I've heard that term other places as well. But, for us, there were just so many percentage of our study that were people who have taken 20 plus surveys in a day. We asked that question and we also track on the back side how many surveys they had entered and completed. And it was just shocking to me to know that people were completing that many surveys. And that leads me to wonder if those professional survey takers are biased in some way. And then, I'm wondering if the industry as a whole knows, in terms of the brands, did they know that the people who are entering their surveys are taking 20 plus surveys a day? And the question is, is that even humanly possible for somebody to spend that much time taking surveys if they have other day jobs and other things to do when most of our surveys are 15, 20 minutes a piece in length.
Melanie Courtright: Yeah. So, Charles, I know you were very active with CASE and you were- Ferrero was actually a sponsor for the CASE work, what about you? What stuck out to you?
Charles Guilbeau: Well, for me, it was the findings of the fraud detection softwares. One is that the prevalence of actual identified fraud, but also at the same time, that the different companies did identify different respondents in each of those cases. So, while it is, on the one side, reassuring that there is a- There are tools in place to manage fraud, what's a little bit disturbing at the same time was that the results you get are very much gonna depend on the tool you use. And I think it then triggers a conversation that a client and a vendor need to have about what are their fraud control softwares in place?
Melanie Courtright: Yeah. Great. Joanne, I think you raised your hand a moment ago, you were gonna speak.
Joanne Mechling: Yeah. I'd have to say I would agree with Charles, because I was surprised to hear that there was such a disagreement among the different fraud provider, fraud detection providers, as to what constitutes a fraudulent respondent. So I really hope that we can come- That the industry can come to some, at least, consistent meaning to different things we might want to think about as buyers of sample. And so we can-
Melanie Courtright: If it encourages- If it encourages you all, I remember when that was the case for duplication software, depending on who, what duplication software you used back in the day, you would get different results. But that has matured. And, so, that's why I tend to use the words immature, that it needs to mature. And what- Lisa, what about you? And then, Nicole, and then Steve.
Lisa Weber-Raley: I think I as a vendor, sort of sitting between panel and client side, got a little sick about the participant frequency because, sometimes, that's invisible to us. And we do our best to try and make sure we've got quality data coming in the door and we're handing quality data out to our clients. But when you think about somebody taking 20, 30 surveys in a day, the impact that that could have on your data is a little scary.
Melanie Courtright: Do you think there are- Are there things we need to consider when we think about asking participants how many surveys they're taking? Do they understand what we mean by a survey? Do they-? Does a screener seem like a survey to them? What does our technology think it means when we talk about surveys? Do you think there are- I'm jumping a little bit because- But because it's been mentioned, do you think there's some context that we need when we think about participation rates? And, maybe, Nicole and Steve, you guys could jump in too here, are there- And Tia- Are there things we should consider?
Nicole Mitchell: Yeah. I think that we're asking people to kind of recollect something that, like how many surveys have you taken? And it's- That question in itself is gonna have its bias, right? Because people don't remember things, it's just kind of a guesstimate. So I think, maybe, there should be a different way to kind of get that information that's not necessarily straight from the respondent because they may not necessarily actually know what the true answer is. Because, like you said, the- A screener could- They may have taken a screener and thought it was a questionnaire, and they haven't even gotten to the actual main questionnaire yet, so.
Charles Guilbeau: Yeah. Well, I think we got that from the browsing history though. I think that was the source of the data, it was actual behavior rather than recall from the respondent.
Melanie Courtright: Well, it was both, right? It was both stated and then looked at from the data from the partner, is that right, Tia?
Tia Maurer: Yeah. So we actually had partners who were able to track the surveys that folks had gone into and give us a number on that. And then, we also asked the panelist if this was their first survey of the day, and if it was not, how many had they taken. Interestingly enough, if I had only taken one or two surveys, I could remember it. And I only took one or two surveys. If I actually said I took ten, I probably did take quite a few surveys, and I'm not lying about that. So, it's easy to remember if you only did a couple of things. It's harder to remember when you've done a lot. But the question becomes, is who's really benefiting from having no limits? Is it the person who's gathering the data or is it the supplier? Because, if I thought about this as an analogy and I'm a pimp and I have a prostitute that's willing to turn a lot of tricks, I'm making a lot of money.
Steve Valigosky: No doubt. No doubt. That's it.
Melanie Courtright: Oh my goodness.
Nicole Mitchell: I agree.
Steve Valigosky: I mean, this is my 30-year perspective on this, because I know that this has been an issue for- Professional panelist has been an issue for as long as there's been online panels. I mean, I- So for Greenfield Online in 2000, and that was the first thing, was like, how do you know these people who are- Are who they say they are? When everybody doubted that online was valid. As far as the number of surveys they do, I think there's a number of challenges that go into us being able to manage that effectively. Back in the day, I mean, we used to manage how many surveys people got invitations to. What you've got now is exchanges in the way that sites operate, where it's not just an invitation only type access to the survey portal, so to speak. And we're trying to do things at Kantar to sort of pre-select people to get served up survey, not necessarily invitations, but the opportunity to fill out a survey, and not necessarily just put everything out wholesale. But you still have a group of people that are out there and, I think, to go back to your question about the most alarming thing, it was the statistic that 20 percent of all surveys are completed by three percent of the people. And that, to me, reminds me of, yikes. How much faith do we have in sampling? Because this is a small universe that's doing an awful lot of surveys. So I do think that there's responsibility for all of us to try to figure out how to do that, but the challenges are great just given the different ways that people can access surveys.
Melanie Courtright: Yeah. So let's do a poll really quick for the audience. Jen, if you don't mind bringing up a poll. Let's try to give a little bit of context to this. In the past three months, have you experienced a research project with a data quality issue that resulted in a delay in fielding, reporting, or acting on a business decision? Yes, no. In the past three months, have you experienced a research project with a data quality that resulted in a delay? Click yes, no. We'll wait just another second or two, and then we'll get some results. So from our- We've got 250 people, 260 people. This will be a good indicator, at least of the biased group that's interested in this topic, have you personally experienced a research project? Let's close the poll, Jen, and show the results. Looks like 73 percent said, yes, they have experienced a problem. Let's do one more. And let's bring it down to a month. Have you personally experienced a problem that resulted in a delay in fielding, a delay in making a business decision, or a delay in reporting in just the past month. Have you personally experienced a research project with that kind of data quality issue? And, if it's still high, I'm gonna ask you about the next week, so I'm just- So 73 percent in the past three months. All right. Jen, let's close that and get results. We're still at 51 percent. Now, let's do the last week. In the last week. It's gonna bring up in just a second. In these polls, it's hard to do multiple choice and it would be hard to calculate them just running them sequentially. In the past week, have you experienced a research project with a data quality issue that resulted in a delay in either your fielding, your reporting, or your business decision-making. I will tell you, it's very challenging to write questions when you're gonna put them in- Polls when you're gonna put them in front of this group, because if you write it wrong, you're gonna get called out. So I labored over these questions. All right. Let's close it and see where we are. So 73 percent in the past three months, at a quarter even within just the past week. So this is a real problem affecting people, some sort of data quality issue that's resulting in a delay in our ability to get the data quickly and slowing down their fielding, reporting, or acting. So what are people doing? What are each of you doing in this area to sort of move the needle in data quality? Who wants to go first? So next-
Charles Guilbeau: I'm happy to go, to go first.
Melanie Courtright: Go ahead, Charles. Go ahead.
Charles Guilbeau: Yeah. We, with Ferrero being a sponsor of the project, because we saw some of the first results, we had a chance to do something about it. And I thought the results were too compelling not to take action. So one thing is we launched an audit across eight agencies around the world, 25,000 respondents, 20 projects to actually measure quality of data, even after it had been cleaned by our vendors, to measure the quality. We still found about seven percent quality issues of various kinds, that's for- A topic for another time. But I think there's basically five things, which I'll tell you quickly, that everybody could do. And I would encourage especially anybody who's in the business of hiring research vendors to adopt some of these practices. It really begins, first of all, with who you hire. If the vendor you hire is not able to have an intelligent discussion about quality, then you probably want to move on right there. And I find especially some of the more technology plays, newer technology plays, they're not able to have that discussion. The second is, before you begin a project, actually already have discussed what quality controls are gonna be in place. Thirdly, after the project- Well, actually, during- Also, implement your own quality controls that you can look at in parallel. After a project, make a point to, not just jump right into the analysis of the data, but take a step back and look at the quality diagnostics that came through, how many surveys were tossed out? Why were they tossed out, etc. And then, I would say, finally, maintain a database of that over time so you can see how well your vendors are doing and whether they're improving or degrading over time. And those are all easy things I think people can implement. Unfortunately, as a client, we don't want to have to think about those things, but now we have to.
Melanie Courtright: I dropped those into the chat, I was a fast typer. So, Nicole.
Nicole Mitchell: So Dynata employs various methods to ensure sample quality across all respondent touch points. So it's important to take a holistic approach and guard- To guard against threats at every point in the system. So we use multiple techniques at multiple touch points so that it reduces the risk that you'll miss someone and they'll end up in your survey. So, for example, we use multiple techniques to confirm participants' identity. So these techniques range from authentication at the reward redemption to using third party databases to confirm real world existence. So we use new sophisticated techniques like machine learning to detect patterns that we might not otherwise catch. And then, in addition, because we on average have about 150 data points for our respondents, we can monitor behaviors. And this allows us to quickly identify any anomalies, such as unreasonable number of devices or used in a short amount of time or any unusual key stroke speeds or patterns.
Melanie Courtright: That's great. Thank you. Lisa, and then Joanne.
Lisa Weber-Raley: Yeah. I think about this from a, sort of- And we've all lived this because we're sort of, what, year- Almost going to year three of pandemic. The Swiss cheese model of health protection, right? We've got to have that conversation between the sample providers, the vendors, and the clients. And we all need to layer in pieces to help each other. I am very much aligned with the idea that if my vendors, my sample providers, are not willing to have a conversation about the things that they're doing like Nicole just outlined, they're not going to be my sample provider. And they need to be transparent and have a conversation, answer basic questions, how your panelists come to be, how do you invite them, how- Just basic questions that you can ask your panel providers. Then I am in the next layer, where I'm doing things like looking at the data, building a good questionnaire, adding in those trap questions, looking at the open end. But I'm expecting that there are multiple things happening along the way so that we can improve the quality.
Melanie Courtright: Great. Joanne.
Joanne Mechling: Yeah, so, Vanguard, I don't know how unique it is. But we both engage with research vendors to- With projects that we do. But we also have our own department that does our own research using a data collection platform, and we- So we engage sample providers directly as well. So we are thinking about for our DIY research, kind of buying the services of one of the fraud technology identifying companies so that we can do a better job with that. It's really impressive, all the- A lot of this stuff is really new over the last few years, so it's kind of exciting to learn about and apply it. We also, of course, want to make sure, and we'll probably start having conversations with some of our key vendors about making sure that they're using some of these tools to do the things that we want to do in-house. And we also are, even though we might have those tools in place, we're not gonna necessarily just trust the data that come into us at face value. So we're doing a lot of experimentation right now about the types of survey questions that we want to include that do a good job of helping us identify people who probably- Who are contradicting themselves, giving answers that just aren't realistic. And, so, we're kind of using- We're taking a look over a few different kinds of surveys where we've incorporated these questions. We take people all the way to the end of the screener, even if they kind of terminate out at the beginning so that we have this dataset that we can analyze. So we're doing some things like TURF analysis to understand what subset of questions are really gonna give us the biggest bang for our buck when- And we should- That we should move forward with and include those in kind of all of the research we do, whether it's in-house or vendor assisted.
Melanie Courtright: Great. Tia, and then we'll move on to the next question.
Tia Maurer: I've seen a lot of the chat here talking about the data cleaning. So it sounds like a lot of folks are doing a lot of data cleaning. And what you're finding, even what we saw in our study, even with all of the fraud detection services in place and them throwing up flags that then we had to interpret what that meant and whether we should take somebody out of the data or not. On top of that, we still had a good 26 percent that I had removed as being disengaged within the survey or whatnot for various reasons using different cleaning techniques. One of the things we've done here at Procter & Gamble, we've actually created an algorithm to detect the fraud, so that I'm not going line by line. And it flags the fraud. We've written kind of a program that will flag straight liners and flag gobbledygook in the open ends and so on. And, so, that, at least, throws a flag in your data, then you can go back and filter by the flag and then look at those more specifically, versus seeing every single row line by line, and then take things out. Another thing that we've done is we've dialed back on doing a lot of the research with external suppliers and the big studies that we've done. And we've said, "You know what? If our employees don't like our idea, then it's not a good idea." And, so, we do a lot of team tests in-house. So we've dialed back our funding because we're not gonna pay for garbage. And, so, until the situation gets cleaned up, we're bringing stuff in-house. And another thing that we've done is we've created some of our own panels and gone out and done the work ourselves because we can control that and we can quarantine or kick the people out who are fraudulent and only keep in the people that we know are actually engaged in telling us the truth. So that's some of the stuff that we've done.
Melanie Courtright: And Steve?
Steve Valigosky: Yep. Yeah. And just quickly, I mean, to kind of piggyback on what my colleague from Dynata said. We take a multi-pronged approach too. And that doesn't just include looking at them at the survey phase, but even in the enrollment phase to try to weed out some of these people that are just signing up and providing all sorts of wacky answers and what have you. The other thing we're doing on the back end of the data is we have- With the fact that we have a lot of tracking data and a lot of normative stuff, we can actually kind of use that information to sort of say, "These are the responses that make sense." If you look at the entire dataset, does this line up with what we've seen before? So we're kind of- And we use a ton of different tools, we don't just use one or two. And it's kind of like, what is that right mix? We're still working on it because, quite frankly, the fraudsters, the speedsters, and the bots are constantly evolving their technology. And it's just a game of adaptation and trying- You can rarely, if ever, get ahead of those folks. And, I mean, Tia's idea of building your own panel and maintaining that that is helpful, but then that leads to other biases in research. So there's no one great solution, there's just a lot of different ways to attack it. And, I think that, as we come together and discuss it, we can compare notes, pick the best solutions, and all sort of move forward together with a unified concept that keeps things cleaner for everybody.
Melanie Courtright: Yeah. The chat is full of this sort of theme around Dynata, when Dynata's all by itself is one story, and Kantar, when Kantar's working on a project all by itself is a story. But then, the realities of using multiple people, let's talk about those realities for a second. What are the realities of participation limits and of duplication and of fraud control in a multi-sourcing environment, and even when Kantar or Dynata or a partner of any type is using a network, of an affiliate network, to draw a sample in. What are the realities of participation limits and fraud? Nicole or Steve, do you guys-
Nicole Mitchell: Participation limits, there's an upside, right? There's you remove the possibility of overburdening respondents, but at the same time, you take away control from the respondent, which in itself can lead to then becoming disengaged with survey taking as a whole, right? So- If you want to take more surveys and you can't because there's a limit there, that may make you not want to take anymore surveys at all because, here's your opportunity to share your voice, but we're not allowing it. So there's a delicate balance in trying to figure out what should be the participation limit. There's a debate over that, right? So it could be total time of session, total time per day, number of surveys. But, I think, moving away from the invitation-based model, where invitations were sent per study, that kind of removes the potential of over soliciting. And then, instead, giving respondents the opportunity to decide on how much or how little they wish to participate. So they could take four five-minute studies, one 20-minute study. There's been a lot of talk about people taking 20 surveys, and- But our internal data shows something different in terms of people typically don't complete more than three surveys a day. Now, this gets into the debate about what are people thinking are surveys? And it also is, what actually is survey taking? Is it them completing a survey or is it them taking a survey and they get screened out? So I think that that's important to decipher as we have this conversation about people taking so many surveys.
Melanie Courtright: Well, and then to the question, when you multi-source, if they took three for Dynata and three for Kantar and three for EMI-
Melanie Courtright: And three for Protege and three for Paradigm and three for Toluna, so you can get to 20. And, so, that's kind of the point I was getting to with the affiliate networks and the sample sharing ecosystem. Maybe, Steve, you want to speak to that?
Steve Valigosky: Yeah. I mean, I- It is a challenge because we have our own proprietary panel and we have the PII for those people that allows us to really more tightly control that. Once we start talking about lower incidents surveys or broader, large sample sizes, we're gonna be going out to our partner network where we don't have their PII. So we either have to rely on them sending us samples that they've done the right way and are monitoring. And the other challenges that we have against that is just the fact that people belong to multiple panels. And, because of the differences in technology sign-up, whatever, there are challenges in kind of trying to manage that as well. So these are some of the complicating factors in that space. We're, obviously, trying to do what we can to make that right, but it is- There's just a lot of challenges inherent in that, and the multiple panel piece also becomes quite a challenge.
Melanie Courtright: Well, let's do another quick poll. Jen, pull up the poll on removal rates, if you don't mind. We're gonna do a quick poll, and then I'm gonna ask the question that has been asked that we got in Q&A. We got it last time, we got it in the Q&A this time, can I trust my panel data? So the question here is, from your experience, what's the average percent of completed surveys that are being thrown out due to quality issues? From your experience, the average percent of completed surveys being thrown out due to quality issues. And I was very careful here not to let my categories overlap if you'll notice. Zero to ten, 11 to 20, 21 to 30, 31 to 50, more than 50. One more second. All right. Jen, close the poll. Let's see what we got. And zero to ten percent is 30 percent. So this is not bad. This is better than what I was afraid it was gonna be based on some of the comments I was seeing.
Lisa Weber-Raley: That's got to be consumer answers there.
Melanie Courtright: Yeah.
Lisa Weber-Raley: Because B2B would be at the upper end of that.
Melanie Courtright: That's a good point. So I didn't specify B2B or B2C, I just asked them to give an overall average. So it may be skewed B2C. So the question though is kind of a two-pronged question for the panel here. Is toss rates a good quality signal and can I trust my panel data? I don't know. Go ahead-
Steve Valigosky: Yeah. And I'll jump in there because the toss rate is an interesting one because the- It could be dependent on the toss rate that we're doing with our quality checks before you guys get the data in your hand. So it could be-
Melanie Courtright: Well, but it-
Steve Valigosky: Far higher than what you're looking at.
Melanie Courtright: But that's fine. I mean, if the buyer was confident because they're not having to throw out as much, I mean.
Steve Valigosky: Yeah.
Melanie Courtright: So, right now, we don't have a lot of quality signals in our- In this space, right? There's not a lot of- So what I have sensed over my experience is that buyers base it on, how much am I having to toss? That's the clear quality signal that they have. And so, as toss rates go up, their confidence goes down. It's completely-
Steve Valigosky: Correct.
Melanie Courtright: Correlated. And as toss rates go down, their confidence comes back up. So, if you're doing things on your side to take that burden off them, good for you. Because then, they're feeling like they're getting what they pay for.
Steve Valigosky: Indeed. Yeah.
Melanie Courtright: Yeah. Nicole, you raised your hand, and Lisa, you-
Nicole Mitchell: Yeah. So I was thinking, agreeing with Steve in terms of we remove- Have quality removals before they- The data is sent to the client. But I think it's important here to think about what you're doing to toss these people out, because there's, a lot of times, there's these in quality control questions that are not fair to the respondent and they get thrown out of the survey. So I think that we have to be very careful when thinking about how to judge if someone is a quality respondent. And then, I think it's important to think about, what's the difference between fraud and what's the difference between an unengaged respondent? Because those are two different things and those need two different types of tools to address that. And, a lot of times, if it's an unengaged respondent, sometimes we need to step back and look, what can we do to make this survey more engaging? Because we have a limited supply, right? We don't have tons and tons of people available to us to take surveys anymore. So we need to be able to hold onto those people and make sure that we're treating them well. And I think part of that is being fair when you're tossing people out, their- That we're allowing them to make mistakes. Are we being fair in our judgment of them? So I think that's an important piece to think about when tossing people out.
Melanie Courtright: Yeah. So it's to fraud, which we know is a bucket, and then it's unengaged or sloppy, and then, really, it's just human beings taking surveys, making errors that might result in a skewed data point. And then, it's the impact of survey design and the survey instrument itself. So it's really sort of four things that- And we sort of all lump them together in this one number called toss rate. And, so, it- But why not, because that's the only metric we have, right? Sort of putting the buyer hat on a minute, the only metric I've got, so gonna- Yeah. But let's talk about that for a second, this interplay of survey design and quality, fraud set aside for a moment. Fraud is fraud is fraud. But what, within the removal rate- And, maybe, Tia you want to talk about that first because I know you've got a lot that you do in this space. What about survey design and its impact on quality engagement and then in just survey mistakes?
Tia Maurer: Survey design and survey length are important. I actually had a supplier- I was doing some testing. I've done lots of testing. And one of them, I did a five-minute survey. And it was very, very easy to do. And it had one question in it, and I don't remember if it said, "For quality purposes, select this." Or if it said, "To prove that you're not a robot." There was something like that and it told them what to select. And I had a good number of people who failed it. So, from my point of view, it doesn't matter how short the survey is, you still are gonna have a number of people who are fraudulent. And that supplier, I called them out on it because I think that's the best thing that you can do, is actually talk to them and say, "Hey, look, if you don't close the feedback loop, then they don't know." And I was told, "Well, there was a better way that you should've asked that. And that just probably pissed people off, so they picked the wrong answer. Blah, blah, blah, blah." And, so, they said a better way to ask it would be- And they read some statement, not knowing that the statement that they read was also my study that was put in the field by another P&Ger for me. And I said, "Well, as a matter of fact, that was mine as well. And here's what your failure was on that one." And, so, what I did the next day is I went down, I had access to 50 panelists who came into our building. And I set my survey, my five-minute survey up with the to prove you're human or to prove you're not a robot or whatever it was in the five-minute survey. And I handed the iPads around the table and had our sensory panelists fill out the survey. Didn't tell them what I was looking for. And, when I was finished, I said, "How many of you noticed that there was a question in there to check to see if you were paying attention?" And most of the people raised their hand. And I said, "How many of you answered it correctly?" And they raised their hands. And I said, "Was anybody offended by that question?" And only one woman out of the 50 raised her hand. And I said, "Did you answer it correctly or did you put something in because you were pissed?" And she said, "I answered it correctly." So, then, I went back to that supplier and said, "You're wrong, because I did the research with 50 people." Right? And, so, they're gonna push back on me and tell me my survey needs to be shorter in order for me to get better attention, it can't be much shorter than five minutes, folks. So I do think the way that we ask the questions is probably important, but you are just gonna get a number of people who are just disengaged or who are just trying to get the end- To the end of the maze to get the cheese. And I don't know what you can do about it other than strip them out. And I think somebody had mentioned in the chat here, if I strip out one person who had the best intentions and just happened to blink and pick the wrong answer, I can't take on a billion-dollar decision, I can't take the chance of leaving that person in and think, "They just made a mistake." So, for me, I would rather throw out one good person and know that my data is good all the way across the board than to keep them in and screw up my business decision. Because we made billion-dollar business decisions where I went back postmortem, scrapped all of the people out that shouldn't have been in that dataset, and it changed the answer and the answer aligned to our business decision. So it's tough. It's a tough call.
Melanie Courtright: Yeah. I mean, I think we might need a quick, I don't know, commercial break. So we're not- Can I trust my panel data? Do we have problems in panel data? Yes. But is it fair to say that there is good data out there we can trust it and we are making million-dollar decisions when it's done well and done right. I mean, what's- When we have these conversations, they're important, but do we- Are we saying that we can't trust panel data at all? I don't think anyone here is saying that, right?
Lisa Weber-Raley: I don't think the quality issues are unique to panels either, right? I mean, you see some of these quality issues in client listed studies, employee listed studies. If you're writing a bad survey, you're gonna get bad data. But I do sort of disagree with Nicole a little bit because I think there are people that are incentivized to stretch and lie in surveys just to get that carrot at the end. And that's terrifying because you are making million-dollar decisions. And, in my kind of work where we're doing healthcare and financial services work, I'm sorry, it's a little boring. I'm not gonna be able to spice that up that much when we're talking about health insurance. And you have to know your audience. You can't- You're not trying to gotcha on real people who are trying to give you feedback. But, if I'm doing a survey with health insurance brokers and they tell me, "Yeah. I've got HSAs with my employers and they have a deductible less than $1,000." That can't happen. They need to get tossed. That is not a legitimate response. That's someone lying or stretching their role. That's not good quality data. They're not a broker. An insurance broker would never answer that. And, so, I think there's a delicate balance. There has to be a conversation, but you have to know your audience. You have to have the subject matter expertise to know who you're going after. If you sent me an email tomorrow and you're like, "Hey, Lisa, I really need to do this cat litter study." I'd be like, "I'm not your woman." Right? I don't know that industry. I'm not gonna be able to catch someone faking their way through that study. But, if you're coming to me with a health insurance project or financial services project, that I can do because I know those industries. So I think it's really understanding who you're going after to get the right questionnaire built, the right checks built, the right targeting, the right conversation between everyone involved.
Melanie Courtright: Yeah. There's a really interesting conversation going on in the chat about, can't we just recruit more people? What do you say to that?
Lisa Weber-Raley: When was the last time you took a study? I mean, how many surveys- I went, got service on my car, right? And you get the survey. Sorry, bad karma, I did not take it.
Melanie Courtright: Yeah. I didn't either. But I'm in the business, so you can't make me. You can't make a plumber fix his own plumbing. You can't make a researcher take a survey. Nicole?
Tia Maurer: I think it's interesting to take those surveys though because you learn. I've taken a few and some of the scales that they put in- One of them was a legal survey and it was ridiculous. I wish I could bring it up right now because you guys would all laugh your heads off. It was just really ridiculous, the answers that they had listed. But I think you learn from those too, like how do people do it and could I do mine better? Or, hey, mine looks pretty good in comparison to the other guy. And I feel like I should help out other people because I'm the one always asking people to take mine. I should probably take theirs a time or two here and there.
Melanie Courtright: Yeah. But what do we say? What? Why can't we recruit more people?
Steve Valigosky: As far as adding more?
Melanie Courtright: Yeah. Why can't we just recruit more people? And what's the participation limit by having more people in the system?
Nicole Mitchell: I think that it's just becoming more difficult to- It's difficult and it's expensive, right? To find engaged and willing participants. The Internet has changed from when we first started doing online. So that's going to- There's so many other things that you can be doing now than taking a survey. And I think that we have to be reasonable about recruiting people because we've had decades of having people to be able to send surveys to or to participate in surveys. And that's just not true anymore. I mean, people, privacy concerns, people don't trust businesses anymore to hold their data. So we have to accept that it's not going to- It's not gonna get easier to find people, right? It's just going to get harder. And we need to be better stewards of their time and set expectations that are reasonable.
Melanie Courtright: Yeah.
Steve Valigosky: Yeah. And, I mean, I totally get what she- What you're saying, Nicole, because you- It's there's a certain portion of the population that's willing to take surveys. So you're already- We've tried to find multiple ways to get to those people, whether it be initially, we used to send out something in the mail to get them to sign up. That's gone to find people where they are, figure out who we're targeting, go to a website, open up, ask them to fill out a survey that gets them into this, then the fact that they need to be double opted in. So that process weeds people out as well. Obviously, I think it's a challenge for all of us to figure out. It's like, yeah, we want more people to do these surveys, so we've got to make the surveys better. Yeah, we want the experience to be better, so we need to build better tools and create more engaging surveys. And there's just a lot that we probably can do as a community to make the experience better. But, at the same time, there's always gonna be somebody who says, "I don't really need to build a panel. I'm just gonna put together a database of people." Or "I've got these email addresses and I'm just gonna start a survey company." And that's that LCD, that lowest common denominator of somebody who's just throwing it out there and not putting any controls against it is ruining that experience for those people, who then say, "I'm not gonna do that anymore." So we kind of all own making it a good experience and keeping it clean. And that is super hard.
Melanie Courtright: And, so, now in the chat, they've given me a layup, thank you to Brent Whitesell and to Jared Huizenga talking about price. So let's put up our last poll, and then I'm gonna ask you guys. We've talked a little bit of all the things we could do, recruiting, but recruiting is expensive. Experience, but experience is expensive. Changing the incentive model, but that's expensive. So let's put up our- Jen, our last poll question. Do I see it? What is your opinion of the interplay between price and data quality as it relates to online data collection? And really think about this, because if you say you're willing to pay more, we're probably gonna give you a call. No, I'm kidding. I don't sell sample anymore. But what is your opinion? We should expect to pay more for quality protocols that reduce toss rates, if that's a good buying signal. Or we should not expect to pay more for quality protocols that reduce toss rates because that's kind of expected. So is price a driver? Because all these things that we're talking about wanting to do potentially have price impacts. All right. One more second. And-
Steve Valigosky: 60/40.
Melanie Courtright: Jen. 60/40. Let's guess.
Steve Valigosky: That's my guess.
Melanie Courtright: Pretty close. 70/30.
Melanie Courtright: 68 percent, we should expect to pay more. For those of you in the session, I'd love for you to put in a couple of comments about why you voted the way you did. If you voted the way you did to pay more, why? And, if you voted to not pay more, why? Well, let's talk about that for you guys. Are we going to pay more for quality? What are the realities of this interplay between price and quality? Charles, you've been quiet for a moment. What do you think about pricing? And then, Tia.
Charles Guilbeau: Yeah. Well, I mean, the very- I remember the very first company I worked at. And we had the most expensive vendor. And I asked my boss why when I was a young, naive researcher. And he said, "Because that's- the quality is highest." And I think I've carried that with me my whole life. I'm happy to pay more for quality every day of the week. However, I actually don't find any- Or very few vendors- Who actually discuss it as a point of difference. So, to me, if it was- If it's brought out as a point of difference, you already have a huge advantage in your initial discussion with me.
Melanie Courtright: Do you work directly with sample providers very often or do you work mostly with agencies? So, do you think that the disconnect between the language breaks between the Dynata to the agency to Ferrero or-
Charles Guilbeau: I work directly with the agencies. And I expect them to be able be fully versed and have protocols in place and who- And knowing and have processes in place for sample management. So, but they also need to be able to share what that is. It's not something that's behind the scenes that I don't have interest in. I certainly would love to have that discussion with vendors.
Melanie Courtright: Yeah. So, I mean, that might be one of the places that this transparency and quality and all of that is sort of breaking, is all down- Up and down the value chain. Tia, did you have a point of view here?
Tia Maurer: Yeah. This is an interesting one. It's like a double-edged sword.
Melanie Courtright: Yeah, it is.
Tia Maurer: And I'm willing to pay more, right? To get quality. Because it's like, if you want to buy a quality car versus a cash for clunkers, right? So, if you want a clunker or you want a quality car, you're gonna pay more for a quality car, right? But, if we have three percent of the people taking 19 percent of the surveys, I don't just want my supplier to clean out the bad apples before they give it to me and do all that cleaning that I've done on my own part on the back side of it. I would like to have fresh respondents instead of recycled respondents. So I think there's that piece. And then, I think, who's getting the money? So are we paying more money to the respondent so that we're getting better quality respondents or- There's a piece of this, and this is the double-edged sword part of it. I did some surveys where I offered $5 out as the incentive. And that's just kind of unheard of. People don't pay that kind of [AUDIO SKIPS] it's like a $1 to the respondent, or- So I offered $5 and I got a guy from China who got into my survey, created different accounts, and had- I had a white hat hacker who sent me a picture of his screenshot saying, "He is China's biggest liar. He is scamming your survey and trying to get $5 incentives." So 30 percent of my 1,000 people who filled it out were this dude from China who was creating fake accounts to try to get- So it's like, if we're offering more money to the panelists, is it increasing the fraud? And, I think, during the survey people were saying in the B2B, which we offer more money, then that gets increased fraud. So it's like there's a double-edged sword. So what does the money go to? I'm willing to pay it, but it feels like if it goes to the panelists, which I hate for them not to be getting what is considered fair for their time, does that just make the issue worse?
Melanie Courtright: Any pop.
Tia Maurer: I think it's hard.
Melanie Courtright: Yeah. I agree.
Melanie Courtright: Who else? Joanne?
Steve Valigosky: And, to go to- Well, to go to Tia's double-edged sword, I think that the pandemic- I saw somebody's comment about since November of 2020. And I think that the pandemic accelerated this trend. I don't have any data on this, but if you think about the situation and people not having jobs and trying to figure out some way to make money, if you up incentives, you do open yourself up to people who are just trying to make money. I mean, there's- You go to the Internet and say, "Make money doing surveys." I mean, there's a website and they tell you what- Which panels to go to because they pay the most. So this, I mean, there's a game afoot that we have very little control over other than to try to figure out, like I said, those ways to screen people at multiple points. But you're still gonna get- If they're a novice at the process, they look just like somebody who's signing up, which then makes it- I don't know how to unravel that, except on the back end. And I think that, as I said, we're working on what is that perfect recipe? And I don't really think there is such a thing. But trying to figure it out, and this probably goes into the AI discussion, where I think that we can use some of those tools and technology to help us get there. I just don't know what that is, but that's sort of where we think the next step in this process is because it is- There's so many variables that go into how to fix this, that we have to kind of use something that's smarter.
Melanie Courtright: Yep. Joanne, and then probably need to move on to our last question and do a quick question from the group. Joanne?
Joanne Mechling: I would say, at Vanguard, and something that Tia mentioned is that she's bringing more research in-house, and so we can control it more. And that's the case, certainly, at Vanguard. We've been adding market research online communities that we have developed and we manage. So just I think that's certainly what research buyers, that's one of the things, the responses we have to this problem. But we do need to do quantitative surveys. And, right now, panels are really the only legitimate and potential source of respondents for us. So we just have to be really careful in how we do it. And this question about, are we gonna pay more for what we think of better quality sample is certainly at the floor right now. And at some, probably in the next couple of weeks, we're gonna go to a recommendation to our department that, yes, we do pay more. We recently had a situation where we'd been using an online sample provider, pretty much that was the only one that was being used. And we tried another, and the differences were stark and in a very good way. So the issue is that the panel provided us responses that we- That our toss rate was very low, which is that is our quality indicator, costs more. But there's a good reason because we, on the back end, our soft costs were reduced. We spent less time having to look at the data ourselves because we did use our own kind of quality screening questions. And we spent a lot less time dealing with the dataset because the bad fraudsters were kind of removed. So it's worth it to us, we'd rather spend our time working with quality datasets than cleaning them, so.
Tia Maurer: Well, and I think that the piece that is important here is, if you do have good quality data, you can probably have a smaller base size.
Joanne Mechling: Exactly.
Melanie Courtright: Yeah. All right. So, then, there's this question that was a theme last time and a quick theme this time, niche audiences in B2B. What does all this mean for B2B and these low incident segments? I mean, we could probably hit it quickly, right? Is the answer that-
Lisa Weber-Raley: So much worse.
Melanie Courtright: It's the exact same problem, but it's so much more work and so much more to watch. I mean, it's- Right, badly-.
Lisa Weber-Raley: Yeah. It's just, it's everything exacerbated. And I think one of the core problems is, a lot of times, what it comes down to is these are not probability-based samples, these are convenient samples. And you're trying to get B2B-
Melanie Courtright: Some of them. Many of them.
Lisa Weber-Raley: Right. Many of them are.
Melanie Courtright: I mean, there are quality based panels out there now, but yes.
Lisa Weber-Raley: Right. For the most part. Yeah. And what we're trying to do is go for a tiny, tiny, very small group, right? In B2B, from a very large, basically built for consumer source. And, so, it just creates lots of issues. I think the incentive issue is one, right? Because you're motivating people to do it for the incentive, as opposed to sort of the social contracts of wanting to improve something that is important to you, which is really eroded, I think, over the last few decades. So it just exacerbates all the issues, right? You see not just the bots and the survey farms and the sort of technical fraud, but then you see people stretching their role like, "Yeah, I made that decision about what health insurance plan to offer." When they actually have no say in that decision and you can tell from their answers. But it then makes the work that much more to clean the data, because you have to have people involved and there's no one tech solution. So I just think it just exacerbates everything that we've talked about in the last session and then this one for B2B.
Melanie Courtright: Joanne?
Joanne Mechling: I think, yes, something that we've run across recently is that we don't have B2B. Well, we do sometimes, but we're looking for a very low incidence segment called ultra-high net worth. And, basically, we've come to the conclusion that we cannot go to panels, the commercial panels that are available for these kinds of folks. They're just not on those panels. So I think all researchers have to think about fit for purpose.
Melanie Courtright: Fit for purpose. It's not a quality conversation if we don't say fit for purpose. Thank you, Joanne.
Joanne Mechling: Yeah. I mean, it's just we're gonna tell our internal clients that, if they want to- If they want to talk to these people, they're gonna have to do qualitative research and they're gonna have to pay more because it's just not worth our time to try to go to panels for these people where we're tossing 85 percent of them. And, really, I don't even believe that some, most of the 15 percent are really ultra-high net worth panelists. I think we really have to kind of have these hard conversations with our business partners.
Melanie Courtright: Yeah. So you don't go to the 7-Eleven to find fancy wine buyers. You just, you have to think about, is the audience that I'm trying to reach, is it the- Is that the right place for it? So I'm with that.
Lisa Weber-Raley: Right. Methodology.
Melanie Courtright: Well, I- Before anybody goes, don't leave, we've got two minutes left. I want to just present some- An opportunity for everyone here on the panel and everyone on the session to get involved, to put your passion where your- To put your time where your passion is.
Melanie Courtright: We're gonna put together a data quality task force. And we're working on this in partnership with CASE again. Here are some, just an initial look at some possible resources. In the chat, you're gonna see a link that you can sign up to be an expert. We're putting out a call for experts. Some of the things we're talking about doing, a tech company partnership, industry report with a barometer of quality, toss rate, flag rate, something that would give buyers and sellers of sample, a barometer to know, are things getting better, are things getting worse, instead of it being a squishy conversation. Some sample or metrics audit programs. A sample audit check list, just a check list, building off ESOMAR's Questions for Users and Buyers of Online Sample. Some sample metrics, reporting tools, again, building on that metrics section that's in the new ESOMAR. ISO Advancement and Usage. Recommendations on Appends for Transparency, I saw some of that in the chat. What are we appending? How could we append? How could we use appends? And the passing of data within the link to get a little bit more information. A tech company fraud detection improvement effort. A tech company duplication method. Partnership to someone to figure out the mobile issue. These are some of the things we're thinking about putting a task force together to build. We don't want to say we're gonna do a bunch more research, because CASE is gonna do that. But we need to take the research and do something with it, other than have the town hall, which I'm happy we're doing. But we want to see if we can roll out a couple of things to actually make an improvement. So I'm hoping that what you're seeing now in the chat is a link to jump in, I haven't seen it yet though. Is it there? Art or Jen. There it is. Get involved. Click that link if you are an expert. A call for experts, new challenges demand new solutions. We want people who want to give up their time, their energy, and their passion, and actually help us craft some of the things we showed on the last slide. And, so, one or two more minutes while people do that. Do any of you have any just last minute things you want to say that you just really wanted to say here that you didn't get a chance to? Anyone? Charles, Steve, Lisa, Nicole?
Nicole Mitchell: I just wanted to say, I think that a quality revolution isn't required, but a quality evolution is in the form of the participant experience and more education and sharing of best practices across the industry and across every part of the research process.
Melanie Courtright: Yeah. I agree. So you'll notice on this call, one- Some of the things we're talking about is actually tech company partnerships to share things about fraud, to share things about duplication and mobile, to share things about metrics, and actually do some partnering to advance the whole profession and not make- Keep it so proprietary and under lock and key.
Nicole Mitchell: Right. Yes. Exactly.
Melanie Courtright: All right. Well, thank you all very much. If you are not going to our annual conference in Philly, we're gonna continue the conversation there as well, of course. And, so, we hope to see you there. Thank you, Steve. Thank you, Tia. Thank you, CASE. Thank you, Lisa. Thank you, Nicole. Thank you, Charles. And thank you, Joanne. I really appreciate it. This was another great conversation. And for those of you who signed up, we'll look to- Look to hear from us more. We'll continue the conversation again soon. Thank you. Have a good rest of the day.