Online Sample Fraud: Causes, Costs & Cures, February 11, 2022

Transcript Courtesy of Focus Forward & FF Transcription

Presenters: Mary Beth Weber, Senior Account Executive, Lucy.ai; Founder, CASE; Tia Maurer, Group Scientist, Products Research Testing Group, Procter & Gamble; Carrie Campbell, Vice President, Analytics, Ketchum; Efrain Ribeiro, Sampling & Panel Online Access Advisor, Zinklar; Moderated by: Melanie Courtright, CEO, Insights Association

Melanie Courtright: Hello, everyone. I see a great group already. Super exciting. Welcome, welcome, welcome. We are very glad to have you here with us today. We're going to be talking about Online Sample Fraud: Causes, Cost & Cures. A labor of love by the people on this call. And we're really excited to have you. For those of you who have not been on a session before, we encourage you to pull up your chat window, say hello, make some friends if you want to make some connections. It's 12:02, so I am going to get started. I'd like to start today with just a little bit of housekeeping. First, welcome to the Virtual Town Hall Online Sample Fraud: Causes, Costs & Cures. We are very glad that you are here with us today. This town hall will be recorded in its entirety. It will also be transcribed. Thank you very much to our partner, Focus Forward, who is our transcription partner. It'll be transcribed and both the recording and the transcription will be placed in our Town Hall Library, and you will also be emailed those, access to that. We're going to have that up there just as quickly as we can. We would love to hear from you, as I've mentioned already. If you don't have your chat and your Q and A pod up, chat is for chat, Q and A is for actual questions to the speakers. Help us out by putting speaker questions right into the chat window – into the Q and A window for us so we can find them easily. There will be plenty of time at the end for questions. And we will have a Q and A, about ten minutes at the end of this. And so you can – we'll save all of those questions for that time so that we can make sure we get through all of the material. Next, I'm just going to present our presenters. Again, this is a labor of love by some people who have been around this space for a really long time. And so, first – we have four speakers today. Our first speaker is Mary Beth Weber. Many of you may know her as the founder and advocate for CASE, which is the Coalition for the Advancement of Sampling Excellence. She has assisted corporate buyers in ensuring provider compliance to research standards for more than two decades. Leading spokeswoman for the movement toward greater transparency and accountability and the quality of marketing and marketing research data and intelligence. In her senior role – in her role as Senior Executive at Lucy. ai, she also works with brands to democratize and capitalize on enterprise-wide knowledge and insights. She's going to be followed by Tia Maurer, group scientist of Products Research Testing Group at Procter & Gamble. She's a group scientist there, and with over 20 years of experience in consumer-centric research and product development, a real friend to the profession and the association in the work that she has contributed. We're super grateful for everything that she and that Procter & Gamble are doing with us in this area. Tia will be followed by Carrie Campbell, Vice President of Analytics at Ketchum. For more than 25 years, she's been a leader in the media and marketing research, working in both the supplier and client side. She's also worked at National Geographic, Roper, The Walt Disney Company, and The New York Times. As a member of CASE, she's been really involved in the data and ensuring data quality in our industry is a passion. And then she will be followed by Efrain Ribeiro, Sampling and Panel Online Access Advisor at Zinklar. Many of you know Efrain. He almost needs no introduction. Leading panel expert, in-depth knowledge of respondent sourcing, online sampling, and data quality best practices. I've worked with Efrain for many – on many occasions and on lots of great work. He's served as a global head of TNS access panels, Ipsos access panels, COO of Lightspeed in Kantar access panels, and CRO of Kantar. Founding member of the Fundamentals of Qualities, if you guys will remember that, developed the initial measures. And then, also on the committee that established ESOMAR's 28, which is now the questions to help buyers of online sample and the metrics at the back of that. And Efrain advises start-up research companies on how best to leverage automation and existing respondent sources. You can see a gigantic panel of experience, passion, and knowledge of this space. And so, with that, Mary Beth, I am going to hand it off to you. 

Mary Beth Weber: Thank you very much for having us. Happy Friday to everyone. I want to thank all of you for investing time today on what is a very important topic, which is the online sample fraud. CASE was created on behalf of client-side researchers who were questioning the accuracy of their research findings. In some cases, these clients would use the same supplier, same survey, same sample, but wound up with different findings, which led to different decisions. That's how we started CASE. I also want to start by really thanking Melanie because she's been instrumental to CASE over the years. She even convinced Research Now's law firm to donate space to us a few years back, where we held the coalition's kickoff meeting. During that, we had very candid and open discussions between client-side researchers, providers, and agencies. And Melanie asked the brands a point-line question. Are you willing to pay more for quality sample? Now, I'm not sure if anyone actually ever answered her, however, I believe that brands are willing to pay more if they have clear and objective ways to measure quality. The good news is that there are tools now available that are going to assist brands and research buyers to have these kinds of measures in place. The study we're presenting today focused on exploring these tools and their effectiveness for detecting fraud and minimizing its impact. This was a collaboration between CASE, the Insights Association, the ARF, and the industry thought leaders. And it's one of the few objective studies that has been conducted on this topic, probably, in at least the last 20 years. Next page. The first step to ensuring quality foundation for research is to make sure that your sample consists of real, unique, and engaged respondents. And this is why we decided to focus on fraud. Over the past few years, the options available for fraud detection have grown significantly. The key for researchers, however, is to really understand if and how these tools work and then to what extent they're actually being utilized on your studies and on your sample. CASE is very grateful for all of those who made this study possible, and especially for the presenters that Melanie so very well introduced today. These three experts have decades of experience in our industry, and they've invested an exhaustive amount of time and energy to analyze the data and findings. I consider Efrain as the godfather of sampling. He's been working, as you know, as Melanie just shared, to ensure the quality of online panel since their inception back in the mid-90s. Many of you who follow CASE, you know Tia. She has presented about fraud on our behalf using her beloved Grinch Poem, which teaches us in a very memorable way how to properly clean our studies of bad data. And Carrie, well, she had her initiation to CASE through fire as she jumped right in, hands on, and was really instrumental to this analysis. CASE also wants to acknowledge the generous support and the resources that were provided by Quester, and by EMI, and the extensive cooperation of the four prominent fraud detection firms in our industry. And those include Imperium, Bizpinion SurValidate, OpinionRoute's CleanID, and Research Defender. Next slide. Client-side researchers hold the key to raising the bar on data quality. And that's especially because they're the ones that control the purse strings. But without tools to understand quality differences, the clients will continue to base their decisions on price and time. And this leads to downward pressures on quality. Our study shines a light on some of these tools that are available today. And this study would not have been possible without the support and commitment from our client-side sponsors at Colgate, CVS Aetna, Ferrero, Flowers Foods, L’Oréal, Pepsi, Pernod Ricard, and Procter & Gamble. Without further delay, I'd like to turn the discussion over to Tia, who will begin with a summary of what we learned from the study.

Tia Maurer: Good afternoon. Sharing kind of what we learned in broad brush strokes before we get into the nitty-gritty details. On fraud, fraud is really widespread in a sample environment. It ranges from study to study and region to region. I've been doing quantitative and qualitative research for years, and I see fraud in a lot of the studies. But it does range based on the study length, and who you're looking for, and also from region to region. And so, we recommend using fraud detection services on each study to really help control fraud. However, on top of – layering on top of the fraud detection services, you should also review your data with a keen eye before you commence your analysis. In our study, what we observed were frequent survey takers. In fact, we had a small subset of these respondents, which was close to about 20 percent of our survey completes, that appear to be taking more surveys in one day than is humanly possible. And they sometimes can fly under the radar of most of the fraud detection services. It's very important that you do that cleaning with a keen eye before you commence analysis and don't solely rely on the fraud detection services. In combination, they make a one-two punch that generally takes most of the fraud or the disengaged respondents out of your surveys before you do your analysis. Additional research is needed to really understand the impact of the professional survey takers, is what I like to call them, on data quality. We didn't get into that in this study. However, we would need to do some additional work. Next slide. The history of fraud. How did this kind of get started? And I used to run a research facility within Procter & Gamble, and so I did a lot of recruiting for qualitative-type or semi-quantitative type research. And so this isn't new. This has been going on for a long time, even in qualitative research, people will try to – we call it running the screener and giving you the answers that they think are going to make them qualify. However, in the case of this quantitative research, this kind of started back in 1997 with the inception of online research. When we started digitizing incentives, thus making it easy to hide behind the curtain of the internet and collect incentives for participating. And, in some cases, respondents aren't even validated that they're actually real people. And so, they're basically doing what I think is akin to survey catfishing. You've probably heard the term catfishing, where people hide behind the internet, claim to be someone, and try to find a date, and someone who will give them all of their money, and they catfish that person. This is kind of survey catfishing. And since there's no validation of the respondents, they can be whoever they want to be and do number of surveys in order to get the incentive for doing so. Next slide, please. What did we do in our approach here? This is a groundbreaking study. We originally designed this as one phase. And we were thinking, "We will just figure out how much fraud exists in the survey sampling ecosystem today." And as we took a step back and we kind of thought about it, we poked around and said, "You know what? We're going to use these fraud detection vendors to try to determine what the percentage of sample, taking a survey, is fraud. But did anybody ever think about testing the fraud detection software to determine if they're giving us false positives or false negatives?" We brought this into a two-phased approach. And, in phase one, what we wanted to do was have a security firm emulate the fraud. We got the fraud detection software hooked up to our survey, and then we hired a security firm to go in and pretend that they were trying to trip the traps. And then we looked at the data to understand if the traps were in fact tripped. If they tried to take the survey numerous times, did they trip the duplicate flag? If they tried to sneak in with VPN or from the dark web on a Tor device, were we able to trip the trap on that as well? We really wanted to look at that. And no one has ever utilized a third-party security firm to test the capabilities of industry technologies. These technologies are constantly evolving to keep up with fraud, so this analysis helped us to identify the areas that needed tweaking. And the participating firms really proved to be highly amendable and adaptable to change, so from what we learned and presented to them, they were making changes to their software. As we moved on into phase two after we learned whether the fraud detection flags did in fact trip when they were supposed to or not trip when they were supposed to, we actually tested it on a live study with sample. We allowed the fraudsters to go through the entire study versus kicking them out at the beginning. They were able to take the entire survey, so we didn't block their entry. And that is – which is standard for fraud suppliers to do so, but we wanted them to go through the entire survey because we wanted to understand the impact on the data. We did select eight different survey suppliers, both large and small, who are part of the sample industry. And we went through an intermediary to manage the study so that we could ensure a normal sample. Next slide. The fraud vendors used proprietary technology to predict who may be fraudsters before they enter the survey. What they're doing here is they're employing a digital fingerprinting approach, so it looks at several things that are on the person's device as they go into the survey. They're looking to – and I'll just give some examples. They're looking to say, what is the time zone? What is the geolocation? Where is the person located? Does the geolocation and the time zone match? They're looking at, if they're coming in on VPN, are they coming in on Tor? Does the browser match the operating system? Those sorts of things. They're collecting that digital fingerprint evidence to look at, and analyze, and determine whether somebody is a probable fraud or not. They also have – what we learned was we had average de-duplication rates across the four fraud detection firms that were in the study of about 11 percent. And we did have an agreement overlap on that duplication of 65 to 80 percent between three of the firms. One of them had some trouble and struggled a little bit, but between the other three firms, we had a 65 to 80 percent overlap in detecting that. Next slide. The fraud detection can detect most – the most egregious fraud, so the people who are hiding on the dark web, and using Tor or using a VPN, and trying to hide who they were and where they were coming from. And also, the duplicate responses, in most cases, those they were able to find out and flag them as fraud or duplicates. The providers though, however, give a numeric score and some fraud triggers as their output. And the user, or the person who's created the survey, is the person who has to make the elimination decision. It's not cut and dry where the fraud detection firms say, "This is fraud. Throw this person out." Instead, you have to move the lever and set the threshold yourself to determine what you think is fraud and what should be removed. The end clients may want to request transparency in terms of the settings being used by the partners and in terms of the duplicates and frauds being flagged and removed so that they can best understand what threshold they want to set in order to determine who they want to throw out of the survey. Next slide. We have multiple companies that are available to work within this fraud detection. Imperium kind of started this in 2007, so they were there for a long time. And then, just more frequent – more recently – sorry. In 2019, we had new entrants come in. Accertify, which was formerly iOvation, Opinion Route, which has a service called CleanID, and then Research Defender, which was formerly SampleChain, and Bizpinion, which is Survalidate, formerly SurValidate, or has SurValidate as the name of their software. These new vendors are common – use common techniques and also methods in the financial tech industry to anticipate fraud. The word of caution here is anticipate fraud. They're trying to stay ahead of the fraudsters, but often, they find themselves behind the eight ball or behind the fraudsters and they're playing catch up because the – and I liken this to the analogy or the true occurrence of we could never have dreamed up three years ago that people would put a skimmer, a credit card skimmer, on a gas station pump. And then you go and pay at the pump, and they've just stolen your credit card information and now they're running around buying things with your credit card, right? These fraud detection firms aren't able to always anticipate, what is the fraudster going to do next? Instead, the fraudster creates the fraud, and then they have to retrofit to that to try to keep up and stop that from happening. With that, what did we learn in phase two? I'm going to pass that on to Carrie and she is going to take you through the learnings for phase two.

Carrie Campbell: Thank you, Tia, and everybody. Hello. Welcome to phase two of study. We're going to get into a little bit of the nuts and bolts here of what the study was all about, how it was constructed, and the purpose of it. Pardon me. I'm a little bit hoarse with a cold today. The plan for phase two, as Tia mentioned, was to put what we learned in phase one into the field to test it in real time and in real-life conditions. We constructed a personal care survey targeting the general population, nationally representative quotas for it. Just a basic survey, nothing fancy, no low instance targeting that we were going for. Sample plan required 500 completes from eight separate typical sources, such as opt-in panels, sample exchanges, rewards communities, et cetera. Due to several of the sample sources not being able to deliver 500 completes within the originally allotted seven days, we extended the field work to nearly three weeks in order to meet source quotas, which was a little bit of a surprise to us that it took so long to just get the – fill those quotas. We also had demographic quotas to ensure that there was nationally representative. It wasn't a long survey, only about 11 minutes, and it was mobile compatible. As Tia also mentioned, we accepted all the respondents into the survey so we could analyze any impact of suspected fraudulent respondents. And we intentionally left in the dupes, as well, for the same reasons. We wanted to see what their impact was, how they were similar, how they were different, if we found any because we weren't sure at that point, but we wanted to see how they differed from those who were not fraudulent or not dupes. Additionally, all four vendors had adjustable levels in terms of fraud sensitivity on their tools. In order to keep the playing field as level as possible, we had all four vendors use their most stringent fraud detection settings. That essentially meant they were pulling out more people than they typically would. Now, let me preface the next slide by saying our objective for the study was not to evaluate the effectiveness of the four firms who participated in it with us. We are grateful they donated their time, they donated their expertise, they worked with us in explaining how their methodologies work, and how they're different, and what the pros and cons are, and where their pain points are in the whole process. We're really grateful to their participation in it. We did not and we will not be making any recommendations as to which firm to use at all, that's – if you want to use a firm, you need to assess them. Assess their pros and cons and decide what suits – best fits your needs. All four vendor names are masked so you can't tell who is who in there and what the results are. Additionally, the purpose of this study was to evaluate the amount of fraud in the sample ecosystem and its impact, if any, on survey results. This study is not meant to be any kind of benchmark in terms of the levels of fraud that we discovered or anything else. It was one snapshot in time. What we found is that the sample screen in the ecosystem is really, essentially, a living, breathing thing. It changes day to day. There'll be bot attacks one day that will go and wipe out a bunch of surveys, and then it'll be calm for a while, or you'll get a big – if there's a major event going on, there'll be a big rush of people getting in to tap into it. It's really changing day to day. This is nothing more than a snapshot in time between March and April of this year. Next slide, please. Here's what we found. We wound up with a total of 3,600 completes, including the dupes and frauds, as both Tia and I have already mentioned. The number of frauds that were found varied quite a bit between the four vendors with one, Detect 4, all the way over in the right-hand column identifying only 48 potentially fraudulent respondents, and Detect 2 flagging 364 potentially fraudulent respondents. Now, I'm just going to take a look at the chat, oddly quiet. I figured that would be lighting up and the question box would be lighting up is why the difference? Why is that? And that's really easy to understand if you take a look at the lines below on the chart here. These are different tools and different variables that the different firms used in determining the level of fraudulence or potential fraud from each respondent. And you can see device obfuscation is used – was used universally by the four vendors with almost all of the – almost all of the frauds Detect Company 4 found, 93 percent were kicked out, 93 percent of the 48 were kicked out because of – or flagged, rather, because of device obfuscation. That's the use of Tor networks, private browsers, VPNs, or anything that will hide the identity of that device. They also used geo time mismatch. If you've got the survey being taken at three o'clock in the morning, wherever thing is, and it's just – it doesn't make sense for a lot of people to be taking surveys at that time in that area. Browsing anomalies were used by two of the vendors. And you can see how bots and web crawlers were used extensively, more extensively by Detect Firm 2 as well as the blacklist they keep. You can see here, this clearly shows the different variables that go into each of the firm's detection score, I guess, is the best way to put it because they do have the scores. And, as also was mentioned earlier, all four firms provide recommendations to clients on who to remove for fraud consideration, and it's up to the client to do the actual removal. That's really the key takeaway on this slide here is it's up to you, the user, to decide how stringent you want to be and if you want to keep in or remove those respondents. There's also one of the things that we found on this – excuse me – fraud and dupe data is the variance clearly due to the methodological differences. The agreement of which respondents were fraudulent across the four firms was only about 25 percent. That was something that was a little bit surprising to us, that they couldn't all agree on exactly who was a fraud. And that, again, we noted goes back to the differences in methodologies. Some are going to get picked up for one thing, while some will get picked up for something else. All the firms provide guidance and recommendations as to the level of protection that a person should be using. However, it's up solely to client discretion as to use it. We even, in speaking with one of the vendors, reported clients who purchased their service, yet never actually turned it on. They claimed they have it. They claimed they use it. But they didn't actually use it, so it's a tool that you have that you can use. And our recommendation, as you'll see shortly, is to use it. On to the next slide, please. The results also showed a wide range of identified dupes as well, averaging about 11 percent dupe rate across the four vendors. However, this is deceptively high because one of the vendors had a dupe rate of over 25 percent. And so that totally skewed the rest of it. Three remaining firms all reported dupe rates under four and a half percent. Overall, there were more dupes and frauds in this survey. Dupes are most typically automatically removed pre-survey, as we all know. Frauds, not so much. It drives up costs for everybody to remove them, as you have to go through more sample to replace them in your final results. In looking at the study across all four vendors, the average fraud rate was about seven percent. Netting them all together, you're looking at about 18 percent of respondents IDed as either dupes or frauds, or I should say potential frauds. That's a lot to consider that it could – if you don't your vendor cleaning out your potential frauds and cleaning out your dupes, that's enough to impact results that you have. I also want to say that the lower dupe, the – excuse me – the lower fraud rate was also driven primarily by two firms who had much lower incidence of ID frauds in the three and four percent range. Again, there was very little duplication of – or agreement amongst those firms as to who was a fraud and who was not, and even amongst who was a dupe and who was not. Next slide, please. Now, this slide, I think, is a key one to take a look at and really internalize and see what we're talking about here. We're throwing around, while there's X, 11 percent average fraud rate here, another tool that we have, that we all have and we all should be using anyway, is manual cleaning of our survey data. We also employed that on this particular survey as well. That manual cleaning referred to, we inserted trap questions, open-end analysis, mismatched answers, survey completion time analysis. And that alone removed around 20 percent of all the completes that were there. When you net it all down for taking everything out, worst case, or the lowest amount of net completes that you have, is about 57 percent of the total. You're taking out anywhere between 30 to 40 percent of completes. If you look at your end tab of, say, 1,000, and you're taking out 30 to 40 percent, your end tab right there is going to go down to – excuse me, again – 60 to 70, 600 to 700. That's a huge impact on your ability to drill down and get more granular with the results. That said, it's also large enough to impact any conclusions you may draw from it and any recommendations that you make. I think this slide, more than all the others, really highlights the need for a layered approach to maintaining the integrity of your data. Next slide, please. Sorry about that. Now, some of the findings for the end users that we have. Use all the tools available because the fraudsters are out here. We don't recommend any one of the vendors over another, as we've said, they're changing every day. I'm sure now they're doing things, and measuring things, and finding things that they weren't when we employed them in this study back in March and April. As I mentioned earlier, one of them changed in mid-fielding of the data, so we had to change our results to accommodate that. It's an ever-changing ecosystem out there, fraudsters are coming up with new ways to get in. If they can hack blockchain, they can hack this in the end. I think the big takeaway, anything is hackable. Nothing, if it's online anywhere, is really, truly safe. We recommend that you use a fraud detection tool and additional measures to reduce data-quality issues. The manual cleaning, the trap questions, removed more questionable respondents than the detection vendors. But, even after that, there were still a number of vendor ID dupes and frauds that the trap questions missed. Some of the frauds got the trap questions right. Some of them got them wrong. It really behooves us to use all the tools, not just one or two, and feel safe about it. As we showed in the last slide, the number of dupe, frauds, and trap fails combined was more than large enough that non-removal could impact survey results enough to cause incorrect, misleading conclusions to be drawn, which means you're making multi-million-dollar business decisions on bad data. And it's not good for anyone. It's not good for the industry. It's not good for us researchers, our personal reputations within a company, it's not good for anybody. I would also – another recommendation in determining sample size you should anticipate, if you do go the route of engaging a fraud detection service and doing the manual cleaning that we were mentioning, you probably want to bump up your anticipated – your sample sizes by 15 to 25 percent to accommodate over the loss of completes due to quality issues. Next slide, please. Now, some – a little – stepping back to a little more, few more tactical things that you can do. I know everybody hates the Captchas, doesn't matter whether it's to start a survey, whether it's to get behind a pay wall. It doesn't matter. Everybody hates them. But they work for fraud detection pre-survey, so that's a recommendation that we suggest using. As much as they're hated, they do help. And you can even do the double Captcha that they have now. Review and analysis of open ends in survey and post-survey. Look for garbage. Look for things, responses that don't match the questions. Look for responses, open-end responses that are copy and pasted from question, to question, to question. Look for garbage, profanity, gobbledygook in there. Use low-incidence survey trap questions. We were fortunate it was easy for us to do a low-incidence survey trap question when this was in the field. My personal favorite that was asked was, "In the last 12 months, did you go to the opera?" Well, there was no opera anywhere open in the United States 12 months prior to us fielding the survey, so anyone who answered, yes, they went to the opera, they were out because it was impossible to go. Get creative with the trap questions that you put in or the red herrings, as they're referred to, as well. Post-survey respondent engagement validation. This is really referring to speeding through a survey, and not just speeding through the survey start to finish, completing a 20-minute survey in three minutes, it's also time spent on each page of your survey. It's the dwell time. You don't want someone spending 30 seconds or 15 seconds on a page for 20 pages and then sitting on the last page for ten minutes, so it winds up being a 15- or 20-minute survey. Looking at that, the velocity, is one way of describing it, that can be an indication of fraud. And throughout your study, begin collecting fraud and dupe incidence rates. One of the hopes that we would like to do at some point is to create a benchmarking system of fraud rates and dupe rates, incidence rates that we're finding in the survey, and it getting to a point where that can be published somewhere, we can go and see what it is and see how your study is doing compared to others in your category, et cetera. Next slide. And now, I'm going to pass it off to our godfather of samples, Efrain.

Efrain Ribeiro: Thanks, Carrie. Good afternoon, everybody. I'm going to talk about another aspect of the study. In addition to the product survey questions that we included. We also put in a number of questions to help us understand the respondent's survey-taking environment. We were thinking that if we could get an understanding of this, it could give us some clues about why fraud happens. The topics that we covered were we asked the respondents about the type of incentives they were offered to do the survey, panel website memberships where they get invited to take surveys, how often they're doing surveys, and how they actually came upon our specific survey. Always interested in understanding how they end up on the specific survey that we're providing. And we definitely learned a lot, but because of the time that we have, we're going to focus on some specific insights related to the survey-taking habits of these respondents. Here's what our respondents told us when we asked them about how often they take surveys. And, by the way, this is from the clean sample, after we removed all the frauds, dupes, and other quality issues. More than half, 58 percent, indicated that they were taking surveys on a daily basis. 90 percent of them said at least weekly. And the occasional participants, those people that are doing surveys once a month, once every two months, supposedly, it's a small minority. This is self-reported information. If it's accurate, we're actually dealing, in our sample, with very regular, frequent survey takers as the core of our participants in this particular survey. We also wanted to understand more precisely what these regular survey taking activities look like, so we asked them if the survey they were taking was their first of the day. And, as you can see here, a third of them indicated that, yes, this in fact was the first survey of the day, which made us feel good about it because you always want your survey to be the first one of the day. Two-thirds indicated they had taken other surveys. And, of those who indicated they had taken other surveys, you can see that three-quarters of them, 75 percent, had done between, somewhere between one and five. The remaining 25 percent of the respondents said they had done six or more. And there's an eight percent there that indicated double digit range. From this, it looks like survey respondents are telling us that they are taking a lot of surveys daily, certainly more than at least what I had thought previously. We went back to the fraud detection firms at this point to see if they could provide us with some visibility on this issue. And, fortunately, one of the detection firms already tracks a helpful metric. And it's the frequency of entering – of surveys entering – devices entering surveys within their ecosystem in the prior 24 hours. I hope that made some sense there. And this is developed from the clients they service, so it's not an entire US survey ecosystem. But what they can see is they can see – it's more than one client. They can actually see all the different surveys that a particular device is entering during a particular timespan. In this case, it's 24 hours. This is actual, behavioral information that indicates how many surveys a device has entered to in the 24 hours prior to our survey. It's not completed surveys, but it can provide a sense of the level of activity by these respondents, by these devices. This we thought would be a good corroboration of what respondents were telling us about their frequent survey activities. Just to clarify this one last time, the fraud detection companies create unique IDs to do the de-duping, which is part of their service they provide. They're able to track these devices across the surveys that they are monitoring for their clients. This is an ability that they have, and this particular firm actually takes this information and provides it to their clients. Next slide, please. This information, the way the client can use this information, obviously, is they can filter out who they would consider to be professional or frequent survey responders. And I believe that most of their clients are sample suppliers as opposed to end clients. Now, the key – I think some of – the two key numbers that I would like to highlight is that 24 percent of our study's respondents, according to this data, have entered over 25 different surveys in the preceding 24 hours. And this information includes those people who told us they had done no surveys before our survey; our survey was the first. The average number of surveys entered by all of the respondents in our study was 21.5. And just a reminder, this would be across the eight different suppliers who provided us with sample. Excuse me. In looking at this, we went back to the detection companies and said, "Hey, is this – is what we're seeing here, is this actually a normal amount of surveys that respondents are participating, going into?" And three out of the four firms said that, yes, this is what they normally, regularly see from their daily monitoring of activity within their clients. The types of questions that you ask when you look at this then is, with an average LOI of 15 to 20 minutes, how could someone actually complete 30 surveys a day and provide useful, thoughtful answers? And that's just 30. We're talking here about some individuals, some devices exceeding 50, exceeding 100. And I think the question gets even more complex when you think about the fact that what are these surveys about? There are multiple different categories. But what if they're doing five beverage studies before your beverage study? It makes you concerned about the potential results. And, also, the other question that this brings to mind is are there any daily effective survey limits on survey taking today, in today's industry? But let's not jump ahead. This is information on surveys entered. And we do know that lots of times, respondents go into a survey, they get disqualified, they never complete the survey. And that happens a lot. We know that. We know that because people are always looking for lower-incidence targets. Fortunately, we had another of the detection firms that also provides in-survey quality evaluation services for their clients. And they said, "Well, we can actually give you completed – we can look at completed surveys from our – from the database that we have." They provided us with single-session data from a client who runs all their surveys with them. The single session is measured by the length the browser is open at the survey site. And these sessions can be as short as minutes, or they can be as long as days. The data they provided – and I think I want the slide changed here, please. The data they provided was extremely rich. But we used it for one thing, to highlight one thing, that I think is important. And that is, if you look at this slide of the single-session survey activity, even though a very small percent of the respondents are doing many, many surveys, and that's three percent of the respondents who did over 21 completed surveys per session, that group actually completed more surveys than the 57 percent who did between one and three surveys. The three percent of respondents in this dataset completed one-fifth of all the surveys. Next slide, please. What to do? This situation – and this is my opinion. I'm giving you an opinion right here. It's not – I don't have the data to prove this. But this situation, I don't think has happened overnight. It's been in the making for a number of years. In fact, those of us who were around in 2005 can maybe remember Comscore reporting at an ARF conference these – on these similar trends. That there was a small group of respondents who were doing a lot of surveys. And you can still see that trend in this data. Now, this particular information raises a series of basic questions. And I think that it's important to try and answer these questions this time. And we're not going to do it. I think the industry has to do this. But what is creating this situation? What is the impact of these frequent survey-taking devices on the actual research being conducted? Is this behavior prevalent across the different sample suppliers? Are some better? Are there others worse? And, if this is impacting the research, is there a way to control the participation of these type of devices? Then, of course, just a regular survey. I think there's got to be a level set at some point, but is it really possible for a person to complete over 50 surveys in a day and provide thoughtful answers? And I think the – a big, big question in terms of the industry in general is, if you start excluding these devices from surveys, will they impact feasibility? And I think the answer there is most likely. And it'll probably impact pricing, too. Those are things that – questions that need to be thought about and addressed by the industry. Now, over on this other side here, we put up some recommendations for – and these recommendations are specifically geared for those clients running and using the research. But the advice here is to begin collecting this survey activity information from your suppliers for each device that participates in your surveys. And we would recommend specifically collecting it from the fraud detection vendors who track this type of information on a constant basis. And you should be able to establish a direct relationship with the specific detection vendor of your choice to get this information. And, on top of that, many of them have incredibly useful, user-friendly dashboards to track this information. Now, we would recommend that you treat this data as another datapoint, and your researchers can use it to evaluate the impact of those respondents who conduct sizable amount of surveys on your studies. It would be just like – I don't know if this is still done. I would think it may be still done, but we used to collect panel tenure. And you used to analyze data on panel tenure from respondents and control your sample that way. You'd want to do it in a similar way, I would think, initially. And then, we would recommend that the learnings from this should be shared across the industry so that we can collectively develop a better understanding of how to deal with this issue, which at this point is new to us. I don't know how many people were aware of this particular issue, but it's something that I think the industry would need to get a handle on. Next slide, please. Let's look into the future a little bit for CASE. There're several areas that we would like to move forward with, but first, I think this type of fraud study, because of the rapidly changing nature of fraud and the capabilities of the detection firms, which are growing, becoming better, should be conducted on a regular basis in the US. It'll give the industry visibility on what's happening. And it also could be a way of monitoring success in terms of controlling this. Right now, we have no idea what – whether we've improved the situation from five years ago with the work that these detection firms are doing. We think, also, that this type of study should be done internationally. And, interestingly enough, we're working with researchers in Australia to set them up to do a similar type of study. And this study focused on general – gen pop. And there was a reason for that. We didn't want to go in and do a low-incidence study and actually have an even higher fraud rates. We wanted to get a broad view to get a basic understanding. The next step is to actually look at targeted populations and populations where you're paying a higher incentive because, anecdotally, that's really where a lot of these fraudsters are attracted to. That's something that needs to be done. B2B populations deserve similar study, but B2B populations, as everybody who works with them, is a very different animal. The way that they're cultivated and built, et cetera, are extremely – are very different. Excuse me for the phone ringing in the background. In terms of frequent responders, we're definitely eager to see where the industry takes the next steps in terms of investigation. For us, it appears to be a broad problem from what we've seen in the data. CASE is a relatively small organization. We have four people actually doing this investigation part time. And it's beyond our resources to really dive into solving this issue. This issue really needs the entire attention of the industry and its investigative powers to resolve and understand what's happening. Finally, we would definitely welcome the opportunity to present more detailed versions of our results as there were additional learnings that we uncovered that we haven't even touched on in this discussion. Next slide, please. We're going to be providing you with the contact information. This is the contact information for the four fraud detection firms that participated in the study. We really encourage you to contact them, and learn more about online fraud, and how we can collectively begin to control this issue. They are experts in this area, and they have a lot of valuable tools to help attack this issue. With that, I think we will open it up to questions now.

Melanie Courtright: Thank you very much. Wonderful. There are a ton of questions and a really engaged group. I'm going to just try to hit a couple of the themes, and then we'll probably have to take some of them offline. One of the themes was B2B. You talked about it a little bit, but just to confirm, this study was consumer only. It was not B2B.

Efrain Ribeiro: Correct.

Melanie Courtright: And we have a goal to do B2B very soon, as well as a low-incidence group. But so, just pontificating, what would you expect if you were to look at a B2B or a low-incidence crowd?

Efrain Ribeiro: I think B2B and, I mean, I don't have to say this. Everybody, I think, a lot of your audience here has – does B2B research. I think B2B research is a bigger target for fraud and, therefore, I believe many of the people that provide B2B services employ special techniques, cleaning techniques, from – of the sample. I would think that it would be definitely a very different type of study than the one that we just – that you just saw. Because you would want to understand how the sample has been built, how the panel or the sample source has been built, and the measures that have gone into validating it. And, once you understand that, you probably can have a slightly different – you'd be able to apply the same tools, but I think you would have to do a different kind of post-validation. That's – and there's a ton of – more on that, but just –

Melanie Courtright: Yeah. It's a whole other session.

Efrain Ribeiro: Yes.

Melanie Courtright: Tia, a question for you. A lot of people asking about the in-survey questions and traps that were used and a little bit of your opinion on the right ones, the best ones. Like one mistake or multiple mistakes, what kind of questions were used here, and what's your opinion on the best cleaning practice?

Tia Maurer: Yeah. I actually put some of those questions in the answer box for the question and answer. But the traps were set throughout the survey and there were a number of them. For example, I know that a lot of times people like to lie about their age to qualify, so at the beginning we ask open ended, "May I have your age, please?" And, at the end, with the other demographics, we ask them for their birth year. And then you are looking for a match or a mismatch there to see if people were trying to lie to qualify because you can say, "But they were honest throughout the rest of the survey." The problem is, if you're trying to hit a certain quota, then they're counting as a quota and they're not the age that you think they are, first of all. That's one. Some of them, we would say, "For quality purposes, please select." And we would put what answer option we wanted them to select in the survey, so that was one. I've used ones that say, "To prove you're a human." Or "To prove you're not a robot." Or something, "Please click this box." Or whatnot. We've put questions in past surveys where we say, "Which of the following has wheels?" And we give them different things that don't have wheels and only one of the things would have wheels. They're not complicated math problems or anything like that. Carrie had mentioned we ask what people had done in the past year, one was attend the opera or walk on the moon. Clearly, nobody walked on the moon in the last year that's taking our surveys. Those are just some of the things that we tried. We went from wacky to very – and most of them were very easy to answer, so it wasn't requiring math or any sort of logic.

Melanie Courtright: Thank you. There's some energy here for a group of people to get together and start talking about actual tactics for solving some of the problems. I know this group is very much about measuring transparency awareness, but there's some energy here, so the – I'll pull this chat and reach out to some people. The Insights Association and CASE would be happy to set up a separate task force that would be more about the actions that the industry at large could take. But this group is very much about the measurement. And we want to keep measuring different crowds and, over time, longitudinally and see if we can make improvements, right? And watch the technology mature. Anything else you guys want to sort of add to that statement before we move on? No? All right. There are so many questions. I promise to pull the questions out and answer as many as we can to the greater population of questions that we have here. We can't stay much longer today, so I just want to remind everyone that we do have our annual conference in April, April 4th through 6th in Philadelphia. We're going to continue this conversation there, as well as have conversations about inclusion, diversity, equity, and access, and compensation trends, and the size of the industry, and information you can only get about the profession from an event like this. I really hope you'll come to that. And then, I thank you all for being here. Again, yes, this will be recorded. It will be transcribed. It will be made available to you. And we will pull everyone's names that said they're interested in more of an action-oriented group and try to establish something and be in touch with you soon. Thank you, Carrie. Thank you, Tia. Thank you, Mary Beth. And thank you, Efrain.

Efrain Ribeiro: Thank you.

Melanie Courtright: Everyone loved this conversation and doesn't want it to stop here. Thank you very much.