By Crispin Beale, CEO, Insight250
With Halloween quickly approaching, it’s a time associated with ghosts, ghouls, and goblins. And while these fearful beings can cause a stir during the season, so can a spectrum of elements that haunt market researchers around the world.
To learn about what aspects of research scare the researchers, I spoke with insight leaders and legends to find out what research elements keep them up at night. Here’s a look at some of the research horrors to avoid.
Jasper Grosskurth, Managing Director at Dalberg Research, Kenya 
“I fear Gresham’s Law: bad research drives out good research. If the client has the choice between a great research project for USD $50,000 or a somewhat weaker one for USD $5,000 (an unbalanced online sample with automated analysis) or something that looks like research but is not for USD $500 (I asked ChatGPT and sent you the answer), then the good research is a hard one to pick for any client.”
Sir Martin Sorrell, Founder and Executive Chairman, S4 Capital plc, UK 
“I suppose my biggest fear is bad actors’ use of AI or AGI. After all, Geoffrey Hinton left Google because he was worried about the existential risk posed by AI systems to humans. He thinks AI systems will be as intelligent as humans. ChatBots will know instantly what other ChatBots know, which is different from humans. An enormous opportunity for the democratization of information, but may be an existential threat in the wrong hands.”
Ben Page, CEO, Ipsos, France 
“Horrors continue to mount - researchers who have no idea what a representative sample looks like because they rely on online samples from elsewhere and think they can deal with any problems by weighting - when you can’t weigh what isn’t there. No understanding of statistical reliability or the impact of design effects. Ghastly presentations that take a report and try to present too many slides, full of too many numbers. I could go on…”
Pam Harrison, Senior Consumer Insights Manager, Beiersdorf, Inc., U.S. 
“When at the Insights Association’s Annual Conference earlier this year, after several AI-related presentations, it occurred to me, as an end client, I needed to be very concerned about my vendors using open AI platforms with our proprietary data. I posed the question to the final presenter and asked the audience if I was the only one concerned about this - only one other client-side person spoke up. This surprised me. I thought – perhaps I’m too old-fashioned. I started conversations with my teams that week about adjusting our MSAs and protocols with vendors to avoid our proprietary learning from being uploaded into the public domain via an open AI platform. Unfortunately, our move to prevent this from happening was just a wee bit slow. As the legal document process unfolded, a well-intended, somewhat junior person at a core vendor partner decided to see what ChatGPT could do to summarize our concepts to fewer words. No malice was intended, but this person didn’t think about how loading the concept up to this open platform would put our innovation idea (which was branded) ‘out there’ for all to see. So the issue goes deeper than updating MSAs because the corner office is the only one that truly sees the MSA…the support staff must also be coached through these rules.”
Mariela Mociulsky, CEO, Trendsity, Argentina 
“In the tenuous tangle of market research and artificial intelligence, several potential "horrors" lurk. The phantom of data quality, with its distorted figures and misleading correlations, can lead professionals astray or conjure false narratives. The ghoul of AI technology's complexity and inscrutability can shroud insights, obstructing the clear view necessary for decision-making. Beware of sampling specters, representing not the entire population but a skewed subset, whispering biased conclusions. Last, dread the reanimated corpse of obsolete tactics, lurching towards outdated goals. Let's illuminate these horrors in this spooky season, not with a lantern's flickering uncertainty but with the sharp, discerning beam of informed strategy.”
As you can see, several horrors can spring up on researchers across various dimensions. So, mind your methodologies, strategies and tactics this Halloween and throughout the year since the research ghosts, goblins, and ghouls can appear anytime.”
Alex Hunt, CEO, Behaviorally, U.S.
“Any insights provider or professional who fails to embrace the inescapable and disruptive power of technology, including the latest breakthroughs in AI, which have the potential to unlock exponentially greater value from high-quality datasets. But equally, it will be a horror show to see any insights provider declaring itself purely a technology provider and marketing only a platform product to corporate buyers: it’s not how brands buy insights, at least today. It’s why, at Behaviorally, we’ve walked toward technology, embedding AI across 83% of our fully tech-enabled product stack thus far, but in a way that enhances the intelligence data brings to our own team of domain experts and our customers.”
Jane Frost, CBE, CEO, MRS, UK
“A few years ago, the two words “big data” were slapped on everything, and so many Emperor’s New Clothes were bought. For some reason, “smart data” just didn’t catch on. Now, the two dread words are Artificial Intelligence, again, as confirmed by my more technical colleagues, slapped on everything and many nothings. Artificial Intelligence isn’t my horror; it’s jargon-based labeling encouraging the purchase of too many pigs in pokes.”
Mark Langsfeld, CEO, mTab, U.S.
“One of the growing horrors is seeing brands have access to a wealth of customer data that can elevate their business, only to ignore it due to intimidation. Many brands feel they lack the technology or expertise to manage and extract value from the monstrous size of information. For example, as the streaming entertainment industry transitions from B2B to B2C, studios are acquiring unprecedented access to their audiences. This new connectivity has some streaming services terrified of change in order to understand their audience. Meanwhile, innovative entertainment brands are relying on mTab's platform to provide show insights to their entire team, so they can craft dynamic storylines and develop engaging characters that resonate with viewers.
“The bottom line is to harness the opportunities that insights offer rather than becoming paralyzed by the sheer volume of data.”
Urpi Torrado, CEO, Datum Internacional, Peru 
“From my perspective, some of the research horrors we should be most wary of are:
1. Shiny Object Syndrome: Clients solely pursuing new methodologies or technologies without a clear connection to research objectives, leading to wasted resources and misguided results. Avoiding these pitfalls promotes sound, ethical research and advances meaningful knowledge.
2. Confirmation Bias: Cherry-picking data to support preconceived notions, distorting objectivity or even trying to get positive results to publish them.
3. Poor Methodology: Using inadequate sample sizes or not representative ones. Additionally, calculating the margin of error for non-representative samples and/or reading results for sub-groups with an insufficient sample.
4. Lack of Transparency: Insufficiently detailing methods and data, hindering reproducibility and peer review.”
Alexander Edwards, President of Strategic Vision, U.S.
“While I have experienced research misinterpretations based on graphics rather than data, language mistranslations of “burning a hole in your pocket,” and I am forbidden to share the discussion I had in a one-on-one research interview about dentures which turned into how blessed it is to be the only male in the retirement home; I think the most horrific story one shared with me about men’s shaving.
“Individuals entered one at a time into a focus group room where a working sink was set close to the view room mirror. Respondents were asked to shave their faces as they "did at home" while those behind the mirror took notes about shaving patterns. One man entered, looked confused, and began removing his clothes. In moments, he was naked and shaving. The room was quiet, stunned, until catering entered the room behind the mirror and turned on the lights.
“Seeing the researchers look at him, he put his clothes back on, informed the receptionist that he shaved “just as he did at home,” took double the respondent fees, and left for a nice NY lunch.”
Monica Tenorio, VP, Insights, Analytics & Marketing Capabilities, PepsiCo, Switzerland 
“One of the worst nightmares I had since I was a little girl was the thought of being chased. And that nightmare is materialized today in real life with a current practice I see in our Industry. With the creation of many new consultancy companies, analytics organizations, and other market research services. I have been inundated for the last two years with emails and calls requesting time to get to know their organizations - not only a few - every day, I have to delete many emails from organizations that use this tactic. Many of them use all sorts of tactics to get my attention, some good ones, but others are terrible. I literally feel chased in real life. It almost feels to me that we are degrading the importance of the research industry's services, like selling encyclopedias vs Google.
“I don’t have an answer for the solution, but as an Industry, I would like us to be a bit more disciplined on how value is offered to clients.”
Walid Benchama, Global Director, Shopper & Channel Insights, Coca-Cola, US
“Don't expect too much from intent metrics; you will rarely be disappointed or out of business.”
Lucy Davison, Founder and CEO, Keen as Mustard Marketing, UK
“What really sends chills down my spine? What makes the blood drain from my body? It’s the horrifying way researchers insist on emailing PowerPoint reports to their stakeholders to share results. Often, these reports are ‘bar chart hell’ without structure, story, or design. We still see many examples from leading brands of ‘insights’ shared in this way. There are so many better ways to share insights. It’s terrifying. Stop it. Please.”
Justine Clements, Consumer Insights Manager, Samsung Electronics, Australia 
“More recently, and likely in reaction to budget cuts, I have seen a growth in what could be termed ‘decorative qualitative,’ namely people using very small-scale qualitative research to provide what they call ‘colour.’ Sounds festive, but in reality, it is a return to the bad old days when qualitative was not considered a discipline worthy of significant investment and outcomes were limited to consumer quotes rather than in-depth analysis and genuine insight.”
Fiona Blades, President and Chief Experience Officer, MESH Experience, U.S.
“I am frightened about this: it may be easy to use AI/ChatGPT to pull together something that looks interesting but is absolutely untrue. Fact check, fact check, fact check! Our data is the bedrock of our industry. Our interpretation is what our credibility depends upon. Use AI and ChatGPT, but don’t be lazy and get caught out.”
Annie Pettit, Annie Pettit Consulting, Canada 
“I am most fearful of researchers choosing to ghost the fact that research quality starts with ME. It’s not lazy participants trying to trick us, low-quality sample suppliers, or mysterious results from AI. It’s our overly formal questionnaires with inadequate screeners, revoking promised candy treats when they don’t meet our random requirements, and our choice to instantly accept automated results rather than validate and improve them. It’s scary that we waste time and money researching how to fix the data quality problem when we already know it’s an industry nightmare which can be solved by ME and YOU right now.”
Humayun Rashid, Consumer Insights Director, Microsoft, U.S.
“A few years back, when Windows Phone was still a thing, we used to travel worldwide to run simulated shopping environments. We would test prototypes of Windows Phones in a retail setting against in-market competitors. Schedules were always crazy, and any delay could compromise our team's deadlines. Also, the team and agency were traveling with very confidential new phone models, along with some very expensive competitor devices, so they would keep all of the devices in our carry-on. Think 30 to 50 devices in our backpack.
“This always led to some very interesting conversations with folks at security and customs, but they had a signed note from home (aka management at the time), so for the most part, things went smoothly. Until they didn’t, the Vietnam Custom Officials would not let our team through. They were convinced we were importing devices, intending to sell them and do bad stuff! They would just not budge until we paid the customs fee. Then, they couldn’t take any electronic payments, so we had to leave the terminal, wander around looking for an ATM, and withdraw the necessary cash to give to customs. It was a crazy few hours of being held captive at an airport. Then, a mad dash to the facility just in time for the first respondent to walk in. After that, we would always travel with pre-filled customs forms and some "extra cash" in our pocket for those everyday emergencies!”
Seyi Adeoye, CEO, Pierrine Consulting, Nigeria 
“The fundamentals will always be sacrosanct. One of the things I find concerning is the push at times by clients to use one single research program to answer a plethora of business questions – yes, there may be budget concerns, but trading quality and clarity for a verbose dataset should be avoided by all means!
“Lack of crisp objectives then consequently leads to long surveys, disjointed discussion guides, or poorly structured ones. Additionally, clients sometimes ask for a last-minute infusion of ‘nice-to-know’ questions, which additionally wreck logic flow as there might not be sufficient time to reappraise the whole logic flow again.
“Furthermore, once the data is out, there is always the temptation to write out 100+ slides to show how awesome our work is but clients are looking for growth-oriented thoughts backed by data presented in a story-easily digestible format. Clients are more focused on getting answers to business questions.
“There is a need always to keep the fundamentals in view while powering up consultative and storytelling approaches to reporting.”
ABOUT THE AUTHOR
Crispin Beale is a marketing, data and customer experience expert. Crispin spent over a decade on the Executive Management Board of Chime Communications as Group CEO of leading brands such as Opinion Leader, Brand Democracy, Facts International and Watermelon. Prior to this Crispin held senior marketing and insight roles at BT, Royal Mail Group and Dixons. Crispin originally qualified as a chartered accountant and moved into management consultancy with Coopers & Lybrand (PwC). Crispin has been a Board Director (and Chairman) of the MRS for c15 years and UK ESOMAR Representative for c10 years. As well as being CEO of Insight250, Crispin is currently Group President of Behaviorally with responsibility for the client and commercial teams globally and the Senior Strategic Advisor at mTab.