qual and quant methodologies don’t need to be exclusive
Yesterday, a prospective client asked me, “Are you a qual person or a quant person?” He was shocked (and delighted, I think) when I told him I was both.
Some researchers see qual and quant as two distinct (and sometimes warring) camps; I’ve heard colleagues complain that “qual is too squishy” or that “quant is too dry.” Sometimes this even devolves into personal attacks: “Qual people are scared of numbers!” “Quant researchers think respondents are just rows in their data sets!”
I’d like to call for an end to the segregation of qual and quant, and encourage you to start thinking of the two as peanut butter and bananas: Each is viable on their own, but the two can be even better in combination!
Often the smartest research involves a combination of qual and quant methodologies to address clients’ objectives; I frequently recommend approaches
Using qual to inform quant. I often work with clients who have vague ideas around a topic they want to explore, but no idea where to start. Any questionnaire designed at this stage is likely to be a disaster and sure to leave you (and your clients) frustrated and disappointed. When interest is there but specifics aren’t, I usually recommend that they start with some type of qual to understand the lay of the land before diving in with a quantitative survey. What are some of the issues respondents bring up in focus groups or interviews (and what kinds of things never come up)? What words do they use to talk about the topic? Open-ended conversations can provide answers that researchers didn’t even know they were looking for, and if these answers come up often enough, they can become good candidates for inclusion in a response list.
Right now I’m working on a project that is starting with IDIs among non-profit leaders, gauging their interest in my client’s educational offerings. While the interviews are going on, I’m designing a quantitative survey. The moderator has been my best source of questions and response options, feeding me ideas that are top of mind for respondents, in their own language. When I toss out ideas, she’s quick to tell me that “They don’t care about that” or “People don’t talk about it in that way.” I’m positive that the survey wouldn’t be nearly as comprehensive if it hadn’t been informed by the interview results.
Like many MRA members, I’m a content-agnostic researcher. This means I’ll ask questions about any subject – often a subject I know nothing about. As a result, in addition to developing full research programs for clients that use qual methodologies like IDIs or groups to inform quant studies, I do a lot of informal questioning to get myself up to speed on topics about which I’m writing surveys. I call these conversations “expert interviews”– even if the “expert” is my dad talking to me about statins or my hairdresser weighing in on the difference between hair serum and gel for my recent project on “bad hair days.” My facebook status is often a request for people to talk to me about various and sundry topics: “Anyone willing to talk to me for 10 minutes about what it’s like to have a broken bone?” “What are the biggest headaches when you’re grocery shopping with toddlers?” “Any friends known anyone with diabetes? Please have them call me – I’m writing a survey and need some info!” It’s important to note, of course, that a sample of my Facebook friends doesn’t give a thorough understanding of how people think about a certain topic – but these conversations certainly give me a place to start when I’m writing questions on things I know nothing about, like cat food, power tools, or American Idol.
Using qual to perfect quant. When I teach classes on questionnaire design, I like to remind students that people will answer any question you ask them – but it’s up to you as the survey designer to make sure that all your respondents are answering the question that you think you are asking. I use the real life example of a question one of my clients wanted to ask: “How safe do you feel in the area where you live?” I was concerned that the same word could have different meanings to different people. Pre-testing this question led to the following descriptions of “area” from respondents:
“My area is just my block.”
“Area means the neighborhood, the part of the city I live in.”
“I thought “area” meant this part of the country. I feel safer in the Midwest than I would on the west coast; at least we don’t have to worry about earthquakes.”
This example usually opens people’s eyes to the importance of making sure survey questions are clear and have the same meaning to all respondents. A good way to work towards this uniform understanding is pre-testing. I suggest one-on-one interviews, sitting with a few people while they go through a survey and asking questions like “What does this word mean to you?” or “How would you ask this question in your own words?” or “Why did you choose that response?” as well as using exercises like “think-alouds” to get inside respondents’ heads while they read and respond to questions can show researchers where there may be confusion or different interpretations of words or phrases. The survey can then be designed with language included that clarifies any “fuzzy” terms.
An important note: pre-test respondents should mirror the actual audience being surveyed as much as possible. I worked for several years in public relations agencies in New York, and was always fighting account teams who wanted to do informal pre-tests of surveys with people around the office. Pre-testing questions about grocery shopping of people who work in PR and live in Manhattan, when your actual respondents will include stay-at-home moms in Middle America, could lead to an ineffective survey.
Using qual to enhance quant...I have to admit, I’ve read (and probably written) more than one research report that lent some truth to the stereotype that “quant can’t tell compelling stories.” This isn’t entirely true, of course, but slides announcing that 62 percent of moms like a certain brand of peanut butter, or that 28 percent try to limit their kids’ intake of certain foods, can only be so interesting. Storytelling is key; nobody wants their clients to fall asleep during a presentation, or be overwhelmed with a “data dump” of charts and graphs.
On the other hand, interspersing these numbers with the story of Linda from Baton Rouge, who gives her kids peanut butter sandwiches for breakfast, makes them come alive a little more. Add some pictures or video clips from our visit to Linda’s home to watch her make the sandwiches for her adorable kids, and now we have a research report that will engage even the most uninterested client. Layering qual and quant approaches can be a great way to show the people behind the numbers and make important data points come alive.
…and quant to supplement qual. The comeback to “quant can’t tell stories” is, of course, “my client can’t make a decision based on qual.” Of course they can’t. When I learn that a client is trying to use two focus groups to select a logo or message, I’m quick to suggest that they consider an online methodology that incorporates both open-ended responses or discussions and quantitative questions. Rather than two groups of eight respondents in one market, I like to steer them towards four (or even six) online groups or bulletin boards of 25 respondents each, drawn from across the country. For a similar investment, we can get a lot of rich qualitative reactions to concepts, but substantiate them with analysis of some closed-ended questions (based on a small sample, admittedly, but still one that is more robust than the participants in two focus groups in one city). This approach keeps everyone happy: It’s quantifiable enough to inform decisions (though I would still not recommend making major business decisions based on the input from 100 people), but rich enough to give the client a sense of how consumers are talking about their message. As an added bonus, clients can view these groups from the comfort of their home or office; nobody has to travel or spend hours in the back room of a focus group facility.
I hope this piece has convinced you that qual and quant methodologies don’t need to be exclusive. Each one has strengths, and often the whole can be greater than the sum of parts – just like a peanut butter and banana sandwich.