Keeping surveys relatively short is important to ensure data quality and to save money. In many cases, the quality of responses starts to diminish after approximately 15 minutes, and every extra 5 to 10 minutes added to the survey costs more for interviewing, fielding, analyzing and reporting. The pressure to keep surveys briefer is increasing because more people are taking surveys on their smartphones and, as we know, attention spans aren’t exactly increasing.

Furthermore, we have seen declining response rates over the years. But declining response rates are not just functions of survey length; they have a lot to do with poorly written questionnaires. When surveys include questions that are unclear, unanswerable, or biased, some respondents drop out or don’t take the rest of the questions seriously. Poorly written questionnaires have a corrosive effect on the industry, making it harder and harder to get and keep respondents.

These are not new issues. The benefits of shorter, well-written questionnaires have always been better data at lower cost. We have been talking about these issues for at least three decades without much progress. How do we solve these problems? As Albert Einstein said, “The significant problems we face today cannot be solved at the same level of thinking we were at when we created them.”

Figure 1. The Framework for Writing Questionnaires

The fundamental problem is that we have not treated questionnaire design as a professional discipline. We would only ask lawyers with years of legal training to write legal documents. Yet with questionnaire design, almost anyone in the office is asked to write the first draft of the questionnaire and email it around to almost anyone else to edit. Lawyers have years of professional training and have passed the Bar examination. What training do our questionnaire writers have? What tests have they passed? 

Few, in reality, have much training on the topic, have books on their desk about questionnaire design or have studied how to write questionnaires. We have a 75-year history of scientific inquiry and research on survey response. There are hundreds of published studies that inform us on how to write better questionnaires. Who among us can cite three or four experiments that have changed how we do so? We can do better. 

Below are key themes associated with treating questionnaire design as a professional discipline. These are also represented in Figure 1, The Framework for Writing Questionnaires.

When we raise our expertise to the level of a professional discipline, we change the relationship with our clients. We improve consulting stature by working with clients to clearly define decisions and determine ways by which to identify and obtain the information needed. When it comes to writing questionnaires, our knowledge and expertise should be so obvious to our clients that they defer to us.

Plan Research to Support Decision-Making

Planning research to support decisions, sometimes referred to as action-oriented marketing research, is a common refrain, but far more challenging than meets the eye. In order to do it, we need to be well-versed in the key decisions marketers need to make and the information they need to make those decisions. This is critical for questionnaire design — but note that this happens well before questionnaires are written and before we even know if a survey is needed.

The key decisions marketers make for their brands are not so obvious. Both marketers and marketing researchers think in terms of either information requests (e.g., “Why are customers setting up accounts but not ordering? What are the emotional needs in this category?”) or study-types (e.g., “We need to do focus groups. We need to do a survey with users and non-users”). One or two decision always underlie both information and study-type requests. It is important for us to work with clients to articulate those decisions precisely.

When asked for decisions, people almost always give an information request. I have found that it helps to write decisions down as declarative statements to help distinguish decisions from information needs, which I write as questions by utilizing the practice of filling in the blank as follows: 

We will decide _______________________.

It helps to offer possibilities. You might ask, “Are we going to decide to stop running the program, adjust it, or maybe roll it out to a new target market?” A clearly specified decision helps focus the research efforts.

Here are six major decisions marketers need to make for the success of their brands:

  1. Product Benefits: We will decide what benefits to offer.
  2. Targeting & Segmentation: We will decide whom to target and whether a segmentation approach makes sense. (Does it make economic sense to offer a different product benefit or different approach to defined, reachable groups?)
  3. Positioning: We will decide what we need customers within each target group to think and feel for them to buy our product or service.
  4. Market Approach: We will decide whether to steal share, grow the market or create a new market.
  5. “Story” to Establish Positioning: We will decide what story we need to tell to establish that positioning (e.g., messages, creative, customer service, etc.).
  6. Delivery: We will determine the campaign to deliver the story to establish the positioning within each target.

The decisions are always more detailed versions of these six key areas. For example, we might need to decide whether the benefit customers are seeking in a particular skin care category is medicinal or aesthetic. Or we might decide whether to steal share from a particular brand. (See Key Decisions and Information Needed, below.)

Clearly defining the decisions and the information needed is the first step in the research planning process. When planning is not practiced to support decisions, we end up with scope-creep. Without the focus on information needed to support decisions, what is stopping people from adding “nice to know” questions? 

Ultimately, we need to be really good at consulting with clients to specifically define the decisions they need to make. Only then do we know what information is needed. 

This is the first and most important step in keeping our questionnaires shorter because most decisions are made based on a small set of data. 

Conduct Qualitative Research

To plan research, and write good questionnaires, we need to be well-versed in qualitative research. Most serious research efforts involve both qualitative and quantitative needs and approaches. Qualitative research offers numerous ways to explore topics that are both different from and complementary to those of quantitative research. Knowing the benefits of qualitative research and how to apply them to our quantitative work greatly enhances both the understanding of what’s being researched and our ability to write an effective questionnaire.

Performing qualitative research is almost always the first step in any research process. To develop a thorough understanding of a topic, we need the insights that only it can provide. When it comes to writing the quantitative questionnaire, we need insights and information qualitative research provides to know what to ask and how to ask it. Imagine trying to write a question that asks respondents to select which of several items constituted their reasons for cancelling their service with your company. How would you know what to put on the list without first talking to people? Even if you know the issues, you need to learn (or validate) the words and phrases they use to describe these issues.

One of the biggest mistakes in questionnaire design is to ask for information that is inherently qualitative in nature, which tend to be long and skip around from topic to topic, causing error and bias. Another, as previously mentioned, is to write the questionnaire without the information and insights that qualitative research provides. These contain questions that don’t make sense to respondents because the key issues are not understand or because we don’t use the words and phrases respondents’ use. 

Being well-versed in the differences between qualitative and quantitative information needs is a must. Likewise, it’s necessary to have a deep understanding of how qualitative research will enhance our ability to write a good questionnaire. Doing qualitative research first gets the qualitative issues out of the questionnaire and it helps us to write a questionnaire that is more conversational and to the point because we know how respondents think and feel about the issues.

Key Decisions and Information Needed

As an exercise, have everyone on the team write, in declarative statements, the key decisions they need to make for the brand over the next 18 months.

We will decide ______________________________.

For each decision, write, in question format, the information needed to make each decision.

Here is a partial example:

Decision: We will decide which of seven concepts to progress to volumetric testing.

Information needed:

What are the purchase intention scores for each concept?

What are the interest scores for each concept?

What are the likes and dislikes for each concept?

Question: Might you take these concepts through a brief round of qualitative exploration to adjust concepts before quantitative testing or to possibly delete concepts that seem to fail miserably? What would be the decisions and information needed in that case?

Plan the Questionnaire

While we are doing qualitative research, it is extremely helpful to write a questionnaire plan.

A questionnaire plan helps outline the categories of information needed. To keep us focused, we write the decisions at the top and then outline the categories of information needed (see an example in Figure 2). Mapping out the categories of information needed helps to keep questionnaires shorter and avoid order bias and order misunderstanding.

Once you get the categories of information mapped out, talk with clients not about how to write questions, but about the information needed. When they say, “Ask respondents what percent of their brand purchases are their priority choice,” you can discuss what they really mean and need. Do they want to know which brands they buy the most, or which brands they like the most? And what time frame makes sense (e.g., past 30 days, past 7 days, etc.)?

The point is that we need to talk with clients about the information needed, not how to write questions. Later, we will write questions. We will of course check with our clients to ensure that we are capturing the right information and using the right words and phrases, but question form and structure are part of our training as questionnaire design professionals.

In column three of the Questionnaire Plan, we document how we plan to analyze the data. We might write: “See if message reception is correlated with purchasing and intent to purchase.”

In the middle column, we capture how to ask the questions. Sometimes I write different versions of questions that I might discuss with my client or that I might test in pretesting. It is also helpful to make notes about issues to explore in pretesting. For example, suppose you have the question: “In the past 30 days, how many times have you yourself taken a vitamin?” I might add the following notes for pretesting:

Figure 2. The Questionnaire Plan

Decisions

Information Needed

How to As the Questions

How to Analyze the Data

Screener Information

 

 

 

 

 

 

 

 

 

 

 

Unaided Advertising & Brand Recall

 

 

 

 

 

 

 

 

 

 

 

Prompted Brand Awareness

 

 

 

 

 

 

 

 

 

 

 

Action & Intention

 

 

 

 

 

 

 

 

 

 

 

Demographic Questions

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 3.

Make Questions Clear
  1. State the unit of measurement.
  2. Use the vocabulary of respondents.
  3. Use precise words and phrases.
  4. When using the word “you,” make sure respondents know to whom you are referring.
  5. Make sure the question is really asking only one question.
  6. When asking for percentages, make sure the base is clear.
  7. Make sure the question stem and the answer choices match each other.
  8. Use bold, underlining, italics, and/or capitalization to highlight key words and phrases.

Make Questions Answerable

  1. State time frames in which people can recall the information you need.
  2. Don’t assume regularity of behavior.
  3. Don’t ask people for information they simply don’t have.
  4. Screen respondents to make sure each question applies to them.
  5. Make “Don’t know” an answer choice if some respondents simply don’t know the answer to your question.

Make Questions Easy

  1. Keep the question stem fewer than 25 words.
  2. When writing questions, say the question out loud as if you were talking to someone.
  3. Limit the length of the questionnaire.
  4. Don’t ask for more detail than you really need.
  5. Soften questions with phrases such as “approximately,” “your best estimate,” or “as best you remember.”
  6. Don’t ask questions in the form of complex grids.
  7. Add labels to answer categories.

Make Questions Unbiased

  1. Do not introduce ideas or opinions in questions that will influence responses.
  2. Make sure that none of the answer choices is more loaded than any of the others.
  3. Make clear that either a positive or a negative answer is equally acceptable.
  4. Randomize answer choices if there is a possibility of order bias.
  5. To get sensitive information, consider disguising the question, shifting the focus away from the respondent, softening the question, or collecting correlated data.

The think-aloud method for pretesting asks respondents to verbalize their thought process as they try to answer each question.

In this example, the interviewer read the question and the respondent verbalized his thought process as he tried to answer. 

Interviewer: “How many cars do you currently own?”

Respondent: “We have three cars, in that I have a car, my wife has a car, and we have a car for our 19-year-old son, but he is away at college, so I am not sure if you want me to count that one. We only own one of them, though, and that would be the one that my son drives. The car that I drive is actually a truck, but I assume you want to know how many vehicles we drive, right? The car my wife drives is actually a company car, but we use it as a family car as well, so I am not sure if you want me to count that one. So I guess I’ll just say two, but I don’t know if that is the right answer.”

The think-aloud method is useful for peeking into mental processes. This example reveals several problems with the question. Both the concepts of “car” and “ownership” are ambiguous and need clarification. The term “you” is also unclear to this respondent. He doesn’t know whether to count his son’s car. Clearly, pretesting revealed a host of problems with a seemingly simple question.

 

 

Checklist for Questionnaire Design Competency

What are two or three books that you have used over the years to learn about how to write questionnaires? What did you learn from each?

Tell me about three or four experiments or experimental effects that have shaped how you write questions?

Tell me about an issue in questionnaire design that you think remains unanswered? In other words, tell me about an issue that you think should be studied scientifically so that we can write better questionnaires.

How do you pretest questionnaires? Walk me through your process.

Give me examples of common problems you see in questionnaires and how you would fix them.

What might be wrong with using agree/disagree scales?

Under what circumstances would you use an even- versus odd-numbered bipolar scale?

What are your guidelines for whether or not to name the middle points in a unipolar versus a bipolar scale? 

You can add others, but this is a good start.

 

  • Is a 30-day time frame too long?
  • Is the concept “vitamin” difficult? Are respondents including supplements and minerals? What about fish oil, calcium or iron, for example?

Don’t let anyone tell you they don’t have time to write a questionnaire plan. Using this process makes writing the questionnaire faster because you have fewer rounds of edits. 

Questionnaire planning also generates shorter questionnaires which translates to faster fielding and lower costs.

Learn the Guidelines for Writing Questionnaires

There are many guidelines for writing questions—too many to include in this article. Figure 3 displays a selection of guidelines on how to make questions clear, answerable, easy and unbiased.

As professional questionnaire designers, we need to have a solid understanding of these guidelines and and many others not listed here. For example, not listed: asking respondents to select from a list, rating things on scales, answering open-ended questions, and visually displaying questions.

When we have a working knowledge of guidelines for how to write questionnaires, we are in a much better position to create questionnaires that make sense to respondents, that feel conversational, that don’t drone on too long and that yield unbiased data.

Please note that there are details and nuances associated with each guideline. It isn’t enough to know the guideline. Applying them takes lots of practice.

Conduct Proper Pretesting (Cognitive Interviewing)

One of the most critical steps in questionnaire design is pretesting — known in academic settings as cognitive interviewing — but it is rarely done properly.

Proper pretesting involves interviewing respondents to find out what questions mean to them, how they came up with their answers and to uncover any problems with the questionnaire. We use techniques such as:

Asking respondents to tell you in their own words what they think each question is asking. This way, you will find out if the questions are working as intended.

Asking respondents how they came up with their answers to specific questions. This helps us see if they struggled with the question or if there was ambiguity about how to answer it.

Finding out what comes to mind when respondents think about certain terms and phrases in the questionnaire. For example, ask, “What does the term ‘family member’ mean to you?” Often, we find that respondents have different definitions for terms and phrases than we do.

Probing to find out if questions were hard to answer or why they took longer than expected to answer.

Proper pretesting reveals all sorts of problems with questions that we had no idea existed, but many companies’ version of pretesting involves sending questionnaires out and seeing if they come back with answers. That is not proper pretesting.

Summary

We started with the goal of writing shorter questionnaires, but that is really not the target. When we treat questionnaire design as a professional discipline, we get shorter questionnaires and much more.

When we treat questionnaire design as a professional discipline, we write shorter questionnaires that make sense to respondents. We reduce the length of most questionnaires by 20 to 50 percent, and we reduce dropout rates.

The benefits of treating questionnaire design as a professional discipline include saving money, getting more accurate data, understanding the subject matter better and, ultimately, helping our organizations make better decisions. 

It will require some work, but let’s commit ourselves to doing it.