You said you wanted innovative options for the qualitative phase, but now have a new problem. How will you choose among the focus groups, webcam groups, individual mobile activities, online community, or many other potential options? Each option may have a different number of participants, durations, incentives, and professional fees. If you find yourself wishing the whole thing were simpler, you are surely not alone.
Faced with such a challenge, you might decide to compare the Cost Per Interview (CPI) for each method, doing a few quick calculations for each option. If so, you used an inappropriate metric, and may have shortchanged your project as a result.
As new methodologies for recruiting and conducting qualitative research appear and change, we all need some basis for looking at comparative value.
Unfortunately, we don’t yet have Cost Per Insight, the metric we would all like to have, but there are alternatives to shoehorning survey metrics onto qualitative projects. Here are a few that will help you get behind the quote, and get past CPI.
Real World Comparison of Data
Let’s start by looking at what a variety of methods can be expected to yield in terms of data. Table 1 shows the data results from a sample of real-world consumer research projects.1
In a face-to-face focus group lasting two hours, shown in column A, the high-level transcript for two groups had 11,000 words in it, amounting to 733 words per participant.
A text chat group, shown in column B, has more people, but a shorter duration, with a single 90 minute research event. Because text chat is multi-synchronous, with everyone talking at the same time, there is more data collected: 1,000 words per participant.
The discussion forum shown in column C was a three-day event, where participants were asked to contribute two hours over the course of the three days. There were 60 participants in five segment-based groups, generating an astonishing 88,000 words of transcript, almost 1,500 words per person.
Columns D and E were two phases of the same project. In E, the 10 participants were providing an individual mobile diary, taking photos and videos as well as providing text commentary. In addition to all the text, a large number of photos and videos were provided. Column D shows the second phase of the project, where online group discussions occurred over a five day period.
In this situation, a relatively small number of participants generated an astonishing 3,300 words per person!
The last two had a highly engaging topic, dogs, versus the project in column C, which was related to OTC pharmaceuticals. Moderation quality would also be a factor, with more active moderation generating higher participant engagement, and more data.
You don’t know how much data you will get in advance of the project, but you can compare other elements as proxies.
Consider Participant Time Commitment
One thing you should look at is the amount of time any given participant in the project is contributing. More time from individual participants means you getting deeper insights. The projects in Table 1 involve established approaches.
Some new methods offered today gather qualitative insights from a large number of people relatively cheaply and rapidly. The participants in the research may be drawn from a pre-existing group (e.g. a panel, or loyalty program members).
When comparing these approaches to other qualitative methods, a key question to ask is how much time any given participant is spending on the research task. In a focus group, a participant will likely be asked to do an assignment in advance and arrive early at the facility. They may be thinking about the topic before the research event. Then they will spend two hours listening and contributing on the topic.
For extended online projects, you need to look at the average weekly time commitment expected of a participant. It might be five minutes, or 45 minutes, but this will give you a good sense of the nature of the participation you can expect, as well as the types of questions and activities that can be successfully executed.
Hours of Data Collected
In real-time qualitative, the amount of data you collect is limited to the amount of time for the interview or discussion. Conducting twenty 30-minute interviews will yield 10 hours of data. Conducting two 2-hour focus groups will yield four hours of data.
Extended methods (asynchronous), such as mobile qualitative and online discussions, create a situation where all participants can contribute equally to all topics. If the participant time commitment is 60 minutes, and there are 15 participants, you will collect 15 hours of data.
Text chat groups are multi-synchronous, so they sit between the two ends. They occur in real time, but everyone can “talk” at once, leading to more data collected.
You should be able to approximate the hours of data collection, even if the researcher has not explicitly provided this information. And an approximation is all you need; this is not something to get precise about.
More data is not necessarily better, but it is a consideration, because it allows more chances to get the critical nugget. Where participants are engaged, they will often far exceed the required time commitment, generating even more. Which creates a challenge in the analysis phase.
Active Moderation, Active Analysis
Simply bringing the right people together in a nice facility does not create an insight-generating discussion. We can’t just leave the participants with a copy of the moderator’s guide and hope the work gets done. Skilled moderation makes a difference regardless of the method.
Some methods of research require more moderation time or analysis time to get through all that data! There have been many times I have wished that data could analyze themselves, but this is also not the case (at least not yet).
As a buyer, you may not be able to find out how many hours of time your practitioner has budgeted for active moderation or analysis, but you should try to get a feel for this, because it will directly bear on the quality of the final result.
Data Quality Considerations
Your qualitative partner will have considered the pros and cons in deciding which options to suggest, and this should also be your first consideration.
- Will some methods do a better job of reaching the right target (e.g., gain access to low incidence population, smaller markets, targets with mobility issues)?
- Are the effects of the group something to be controlled, or something to be studied?
- Which method will generate richer data on this project?
Having great data from exactly the right target should always be the main consideration in method selection.
Some approaches may also be more efficient or engaging for your internal team. Although not strictly a data quality concern, your ultimate goal is insights for the business, and some methods may get you there faster or more easily than others.
Real World Examples
In a recent consumer project involving concept development, I was initially asked to bid on six focus groups. For roughly the same budget, I offered the client a much improved project, starting with 3-hour co-creation labs followed by an online discussion in a second phase. The combination of the two methods gave us better geographic coverage, as well as richer, more abundant, data.
Table 2 outlines a B2B example, also involving concept development. The budget was quite limited, and the target, investment professionals, can be difficult to get into a focus group room.
To control group effects, and cover the needed segments, the best face-to-face option was four short micro groups of 1-hour duration, which probably would have turned into triads in the field.
Although the second option was somewhat costlier due to more professional time, it enabled the client to evaluate 10 concepts, and obtain richer information. Looking at the quantity of data collected helped the client choose the online option.
In both of these cases, a CPI approach would likely have led the client to make a different choice, with negative effects on the insights obtained (see chart above.)
What to Use Instead of Cost Per Interview
There are four considerations when comparing method options and qualitative proposals:
- Data quality
- Participant time commitment: individual total for short projects, or average weekly time commitment for extended projects
- Hours of data collected
- Active moderation and analysis time
Once you know these elements, you are in a much better position to choose options. Forget CPI, it’s a survey metric that won’t help you make good choices in the new world of qualitative options.
- Project B was provided by Jennifer Dale of Inside Heads. All other projects came from Abbott Research + Consulting.