Case Study 3: Reviewing Your Question Wording

• Tweet

So far I have looked at two case studies involving clients that did not have much survey experience. This case study involves a client who is very familiar with doing surveys. One lesson to be learned is that even if you have experience designing surveys you can benefit greatly from having an impartial third party review your survey and offer suggestions.

Wildflower Marketing (real name changed) uses surveys frequently to test marketing ideas, profile target markets and evaluate new product ideas. They usually come to me with a draft survey and ask me to review and comment on their questions.

This case study is a recent request from this client. I’ll go through a sample of the input I gave on their survey questions. Along the way I will highlight the general principles that specific examples help illustrate.

I was sent their draft questionnaire and asked to comment. My first question back to the client was, “What is the purpose of your survey, what are your main objectives?”

General Take-away #1: One can’t make meaningful comments about a survey unless he or she knows exactly what the surveyor plans to do with the information.

In this case, the main purpose of the survey was to obtain information about the target market, herbal tea drinkers, in order to help establish marketing strategies for selling herbal tea.

The survey’s first question was, “What do you consider to be herbal tea?” which had multiple-choice selections. The second question was, “Do you drink herbal tea?” I recommended that in between question one and question two my client include a definition of “herbal tea” for all respondents to use for the remainder of the survey.

General Take-away #2: Make sure terms and jargon are defined unambiguously. A survey question depends on the fact that all the respondents have the SAME understanding of it. It is not enough that everyone understand it.

Here are the next couple of questions and my input (in bold).

Q. Do you MAINLY use herbal tea bags or loose tea? * [Select one]
* Tea bags
* Loose tea
* Both (add this option)
* I don’t know

This simple change makes it easier for the respondent to answer and actually provides more information. No evaluation of “mainly” by the respondent is needed, and information about how many use both methods is added.

Below is an example where use of a similar type of word, “typically,” can actually make it easier to answer for the respondent because it encourages the respondent to go with their gut feeling.

Q. When you drink herbal tea, how do you typically prepare your herbal tea? *
[Select up to two]
* Iced
* Hot in teapot
* Hot in cup
* Sun tea
* Other (please specify)

Q. What is your MAIN reason for drinking herbal tea? *

[Assign no more than 100 points in total across the categories below]
* Flavor
* Health benefit (e.g. digestive, detox, laxative, cold care)
* Comfort (e.g. soothing)
* Habit
* Morning ritual
* For a “pick me up”
* Boredom
* To warm up
* To be sociable
* Instead of coffee
* Doctor’s orders
* Don’t know

My client has his own ideas about why people drink tea, but it doesn’t take much thought to think of possible other reasons. Above, you can see the items I added as suggestions. If a respondent sees his thoughts in the question options, it makes it easier for him or her to answer and it produces better data.

General Take-away #3: Don’t bias your survey questions toward your specific objectives, work to expand your thinking toward how a respondent might think. You can also accomplish the same thing by having someone else review your survey and provide his or her input!

At this point I noticed all of the questions on the survey were required. At best this is totally unnecessary and at worst it is extremely annoying for the respondent.

General Take-away #4: Only make a question required if absolutely necessary like when it is a branching or skipping question.

The next question on the survey was, “How often do you buy herbal tea?” This question seems harmless enough but I asked how this information would be used. It would be impossible to correlate the answers with the amount of tea one drinks or how much tea they buy or even how often they drink tea since there is no mention of how much they buy or when they buy. I encouraged my client to more fully explore the reasons behind this question and what he was really looking for.

Here were the next couple of questions.

Q. Have you ever had any of the following health conditions? [Select all that apply]
* Anxiety disorders
* High blood pressure
* Back Pain
* Cancer
* Sexual conditions
* Migraines
* Irritable bowel syndrome
* Diarrhea
* Obesity
* Depression
* Skin Problems
. . .
Etc. (the list goes on!)
* None

Q. Which alternative medicines or therapies have you used? [Select all that apply]
* Yoga
* Massage
* Homeopathy
* Meditation
* Martial arts
* Acupuncture
. . .
* Etc.
* None

At this point in the survey my questions were: Are these questions necessary? Where did the lists come from? Where are you going with these questions?

The questions hit me as coming out of the blue. If these questions were necessary, then it would be important to prepare the respondent for them and let the respondent know where you are going. Something as simple as – /m=“Some people use herbal teas for medicinal purposes, have you considered their use for any of the following conditions? [Select all that apply]” – could work.

General Take-away #5: Think about how your questions are going to make your respondent feel. In general people will answer personal questions or questions about sensitive topics but they need to feel it is both necessary and critical information for you, the surveyor, to have. Throwing in a question totally out of the blue (of a personal nature or not) can be a turnoff especially when the purpose for the question has not been made clear.

My next recommendation was for my client to change their ranking question:

What are your TOP criteria for selecting a brand of herbal tea? [Place in rank order of importance]
* Clinically tested
* Price per container
* Environmental practices of manufacturer
* Price per bag
* Flavor
* Certified organic
* Effectiveness
* Fair trade certified

I prefer to use checkbox questions as opposed to the ranking question because I feel that ranking questions can be time consuming for respondents. (Next month’s blog will explore this issue in depth) I recommended that the client change this question to:

Q. What are your TOP criteria for selecting a brand of herbal tea? [Select no more than 3]

This will give them a natural ranking of the items and not increase the burden on the respondent.

General Take-away #6: Ranking questions can be time consuming for a respondent to answer. A good alternative is to ask them to select NO MORE THAN their top X items where X is roughly one-third of the total number of items (see next month’s article for the details behind this recommendation).

My final recommendation was for the client to eliminate the following question appearing at the end of the survey.

Q. Please share any additional information you feel would be relevant to this study.

To me you might as well reword this question as, “I don’t have time to think about whether I’ve included everything. Can you help me out and tell me what might be missing, even though you don’t really know what my objectives are or how I plan to use the information you provide?” Not a good question to ask.

General Take-away #7: Don’t burden your respondent with work you are unwilling to do yourself. They typically do not have as much invested in your survey as you do.

This is the third in a series of case studies focusing on Survey Design. Next time I will talk about different methods for obtaining a ranking of a list of items.

Share This: