“Dear Boulder Resident,

[...] To get a representative sample of people living in Boulder, this questionnaire should be completed by the adult (anyone 18 years or older) in your household who most recently had a birthday.

Please have this person take a few minutes to answer all the questions and return the survey in the postage-paid envelope [...].”

Hmmm. A few minutes? Are they kidding? The survey had 170 questions and 8,500 words on it. Spending just 6 seconds on each question (and how important it is to the City of Boulder) means the survey would take more than 15 minutes. Thoughtfully considering each question could take hours.

A survey like this undoubtedly creates “survey fatigue,” but that alone is probably not its biggest problem. If done correctly there are situations where a 170-question survey can be appropriate – but that’s a topic for another article.

The biggest problem with this survey is that the authors made a number of mistakes when putting it together. Here’s the list of the most common mistakes survey authors make from my previous blog entries:

10 Common Mistakes Made When Writing Surveys

1. Having little or no understanding of the target audience
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions (online surveys only)
4. Asking too many open-ended questions
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions
7. Asking too many questions
8. Asking two questions in one
9. Making questions too general
10. Putting too little thought and planning into writing the survey, period

The survey above has been useful in helping me refine the list of common mistakes (which supports the old adage that nothing is ever a complete failure, as it can often serve admirably as a bad survey example). To wit, I have refined the list (refinements shown in red below):

10 Common Mistakes Made When Writing Surveys

1. Having little or no understanding of the target audience and what information they will be able to provide
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions (online surveys only)
4. Asking too many open-ended questions (or asking open-ended questions that are not useful)
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions or ones that won’t produce usable information
7. Asking too many questionsand/or including disjointed laundry-list questions
8. Asking two questions in one
9. Making questions too general
10. Putting too little knowledgeable thought and planning into writing the survey, period

Below are some examples from the survey that support this refinement of my list of mistakes.

First, an example of the type of open-ended question that is not useful:

What do you think should be the top three priorities of the Boulder City Council in 2012?

This type of question will never lead to gathering useful information, except by accident. The reason for this is the respondent has not been given any context within which to answer the question, such as: What’s on the City’s docket? What kinds of things can and will the Council address? Are there budget constraints? It is not fair to respondents to require them to guess at context. This type of question is what gives open-ended questions a bad name.

Next is an example of one of several “laundry-list” questions that were included in the City of Boulder 2011 Community Survey:

Give your respondents a break, literally. At least break the list up into smaller pieces. A more thoughtful way to present the question would be to group items into topic areas.

In summary, the main mistakes in the City of Boulder 2011 Community Survey were:

Mistakes 4 and 7 were the result of survey mistake #10: the authors did not put enough knowledgeable thought and planning into writing the survey. As a result, the data collected by this survey will not necessarily provide the information they want. There is really no way to know exactly how survey fatigue has affected the data collected. A survey of this length will certainly exclude a significant portion of the target audience.

I will address the issue of appropriate survey length in my next blog post. Happy Survey-Taking!

Image courtesy of opensourceway – Flickr, Creative Commons (Attribution)

The topic was political. The purpose seemed to be to poll the opinions of voters. As I first answered the call, I hesitated to continue, but given that I’m in the survey business I went forward. (I usually try to participate in surveys at least until my patience runs thin.)

There was a slight introduction asking if I was registered to vote, etc., but not much else. The survey robot seemed friendly but clearly refused to laugh at any of my jokes. (That should have been a clue.)

I answered the first few survey questions without much trouble. All required Yes-No responses. Questions like: “Did you vote in the last election?” “Do you think Boulder’s City Council is doing a good job?” Then things started to get more involved. Soon there was a question I couldn’t answer without some thought. “Hmmm, I’m not sure,” I said. The robot responded, “This question requires a ‘Yes’ or a ‘No’ answer.” I hesitated again, not knowing how to answer and the robot seemed to get annoyed, replying, “This survey will end unless I receive a ‘Yes’ or a ‘No’ answer.” Suddenly, I panicked. “Yes,” I said.

Whew! That seemed to calm down the robot and the survey continued with a few more questions before there was another for which my real answer was not “Yes” or “No.” It was probably something like, “I’m not sure” or “I haven’t decided yet” or “It depends” or “I don’t understand the question.” Once again the robot insisted, “You must answer ‘Yes’ or ‘No’.” As I tried to explain to the robot why I couldn’t answer “Yes” or “No,” the line went dead and I assumed that meant the survey was over.

Lots of Questions

As I hung up the phone, a host of questions rattled through my head:

• How do the survey sponsors expect to use the data from a survey like this?
• Do the sponsors really think the data they get is unbiased?
• Who writes these surveys, anyway?
• Do the survey authors really think it doesn’t matter what frame of mind people are in when they respond to their survey? Do they even think about that?
• Do the survey authors really think all of their survey questions only have the possibility of a Yes-No response?
• Do the survey authors really think that a respondent’s true feelings, opinions, and attitudes are like items on a shelf, ready for a robot to pick up and put in a box?
• What were they really trying to do with their survey?
• Isn’t it likely that the analysis of the data will ignore the method and circumstances under which the data were collected?
• Will the survey authors give any thought to the data’s validity? Or will the survey data instantly become meaningful (in their minds) once they see numbers or graphs on a page?

That is a lot of questions that went through my mind in the few seconds it took me to put down the phone.

The Mistakes This Survey Made

Clearly, mistakes were made in putting this survey together. I think it would be instructive to go back to my last article, the 10 Common Mistakes Made When Writing Surveys, and see which mistakes from the list were made here.

Let’s take a look. Here’s my list of the common survey mistakes:

1. Having little or no understanding of the target audience
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions
4. Asking too many open-ended questions
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions*
7. Asking too many questions*
8. Asking two questions in one
9. Making questions too general
10. Putting too little thought and planning into writing the survey, period.

(I’ve bolded the ones I think the authors of the robotic survey made. The ones with stars are the ones about which my premature exit from the survey preclude me from making a clear determination.)

Items 1-3 and 9 are all closely linked. The authors’ understanding of their target audience seems minimal because they didn’t allow for more response choices and the choices they did provide were too restrictive, even to the point of requiring every question to have a “Yes” or a “No” response. The survey questions become too general from the restrictive responses.

The end result of these four mistakes is mistake number 10. The authors did not put enough thought into who would answer the survey, how their survey method would affect the response, and how the information was to be used.

Happy Survey Taking! I’ll talk about the other survey I attempted to take in my next blog post.

Photo courtesy of Jonty Wareing – Flickr, Creative Commons (Attribution)

String Variables to Numeric

While SurveyGizmo allows for changing of variable format in SPSS, sometimes I end up with string variables that really want to be numeric. This can easily be accomplished with a short line of SPSS syntax.

Let’s say you wish to convert the variable named “String_Zip” to a numeric variable entitled “Numeric_zip”. The following code achieves this:

RECODE String_Zip (Convert) INTO Numeric_zip.

EXE.

While this is simple, the default numeric value in SPSS is a scale variable.

To help remind yourself not to take the “average” of zip code you might want to change the variable level to Nominal. This can be achieved with the following line of syntax:

Variable Level Numeric_Zip (Nominal).

As a reference, below are the other variable levels supported in SPSS

• Variable Level Numeric_Zip (Ordinal)
• Variable Level Numeric_Zip (Scale)

Numeric to String

If you need to convert a numeric value to a string there is one additional step. You must first declare the string variable. The following creates a string variable named String_Zip2 with a width of 5 characters.

STRING String_Zip2(A5).

Now that you’ve created the variable, you can convert the numeric value to it’s string counterpart.

COMPUTE String_zip2=STRING(Numeric_zip,N5).

EXE.

Dropping unused variables

Want to drop the old numeric variable from the data set? Simply use the match files command with the drop statement.

Match files file=* / DROP Numeric_Zip .

EXE.

Thank you again, Joe, for teaching us a few quick and easy tricks using SPSS. If you’re out there reading this and would be interested in becoming a SurveyGizmo survey expert, please send us an email at marketing@sgizmo.com and we would be more than happy to start working with you.

Now, we’ll look at what happens when we add another dimension, time. (I’ll look at yearly data here, but it could just as easily be monthly, quarterly or any time period.) With the added dimension, things get cluttered pretty quickly. Let’s look at the Stacked Bar Chart when we add just one year.

All the information is there but it is cluttered. I.E – it is not easy to take it all in at a glance. This compromises the real value of the graphical display. Further, it won’t stay this way as time marches on! The problem quickly becomes untenable.

One thing we may decide to do is to drop the individual attributes and look at just Overall Satisfaction. One could argue that it is the most important anyway. See figure 2 below.

This is not too bad for looking at ten years, but it isn’t exactly crisp.

Enter One Number Reporting (ONR). Wouldn’t it be nice if there were a single number that we could track that tells us how we are doing? This is a frequent request of management and has been for many years.

Since the goal is to reduce the five levels of our Likert scale down to one, we won’t expect any measure to work perfectly.

Here are four ONR options:
1. Averages – The average rating after assigning numbers to the Likert scale levels; 5=Very satisfied, 4=Satisfied, 3=Neutral, 2=Dissatisfied, 1=Very unsatisfied.
2. Top two box percent – The sum of the top levels in the Likert scale (in Figure 2, the sum of the orange and blue boxes).
3. Net Promoter* Score 1 (NPS1) – The top box (blue) minus the bottom three boxes. [This was originally designed for the one question, “Would you recommend us to a colleague with similar needs?” the Net Promoter Score is the total who are “Very likely” to recommend you minus those who are neutral or unlikely to recommend you. The theory is that gives you “Promoters minus Detractors”.]
4. Modified Net Promoter* Score 2 (NPS2) – Percent satisfied (Very satisfied or Satisfied) minus (Very dissatisfied or Dissatisfied).

Below are charts of each option using the same data that appears in figure 2 above.

All four options yield similar pictures, so that’s good! As I mentioned in my last article, I’m not a big fan of the using averages because the number itself is hard to interpret, but any of the other three seem equally good.

So what does all this get us? For one, we have that “one number” for management that we can use to track progress over time with relative ease.

Secondly, we can add back the attributes we dropped from the attribute satisfaction we looked at in Figure 1. We now have a way to look at all the attributes over time using the so-called “spaghetti chart” (see below).

This chart let’s you look at trend data for several attributes at once. It can get a little cluttered but proves to be a useful data display tool. I am a big believer in looking at data over time. Looking at data over time allows you to build knowledge for improving your decision-making. Next time I’ll take a close look at analyzing trend data.

*Net Promoter is a registered trademark of Satmetrix Systems.

But when these test are not well designed, this type of research tends to yield general information that is not very actionable or worst yet, could be misleading. Creating a concept test correctly is key to making informed decisions.

The first step in concept testing is to create product concepts that clearly define the “reasons-to-believe” in the product and the experience the customer may expect from it.

The best source for good product concept descriptions is the potential customer.

Conducting qualitative market research as the first step in the concept testing process can bring a lot of value by uncovering what is important to customers and how customers talk about the product.

I recently was invited to take an online survey which was used to conduct a concept test to measure the appeal of online marketing research courses.  I think back now and wonder if the sponsor of the survey did his homework in trying to identify what features really matter to potential end-users of online marketing research courses.

The image below is a snapshot of how the concept was presented in the online survey. Of the four features described in the concept, the only one I found relevant is the 24/7 accessibility. The other three are quite vague and somewhat immaterial.

 What does it matter if the courses will have 3 to 8 hours of total instruction if I don’t know the length of each class? Is it 30 minutes or 60 minutes?

My work schedule and home life may only allow me to sit for 20 minutes at a time to attend an online class.

Furthermore, I don’t know what they mean by “comprehensive instructions.”

Are the classes task-oriented, will I need to do a lot in preparation to take the class? I associate the word “instructions” with manuals, so when I read this I feel exhausted already. Who reads manuals these days?

“Dynamic interfaces” and the image they use don’t tell me much either.

A more concrete example would have helped to set my expectations. Without any direction, my expectations could be too high, which can lead to deep disappointment if the courses don’t align with my notions of what a dynamic interface is in this context.

A short video showing different screens would have provided a better illustration of what the user can expect from the interface.

Finally, the most relevant feature to me is the course topic and this is absent from the product concept.

One may contend that there are many topics of interest and there is interest in testing only the format. On the hand, I would argue that the format is irrelevant if the topic is not of interest. The approach followed here doesn’t really reflect how the end-user thinks about this product.

If topic is the most important feature when selecting an online class, then excluding it creates an artificial product concept to which it is difficult to give an accurate answer. When they asked if I would be interested in these courses, my only plausible answer was “might or might not.”

I have a feeling that that study produced misleading results for them, if I had so much trouble with filling out the answers. That was an example of what not to do.

So what are the key steps in conducting product concept tests that yield accurate results?

1. Conduct qualitative research with members of the target audience.  Avoid “inbred” concepts based only on input from internal staff – A few interviews with marketing research professionals would have provided a richer and more realistic foundation on which to base the concept.
2. Create concepts that are concrete and put them in relevant context – An online course on a particular topic would have allowed me to give a more realistic response. A video could have shown me what to expect from the “dynamic interface.”
3. Write concepts in the target audience’s language – No “instructions” please!
4. Focus on Reasons-to-Believe arguments that highlight what is important to your audience – I need more than 24/7 accessibility to assess the value of this service. Topics are a good start.

If you’d like more help with concept testing, please head over to Relevant Insights’ and see our Concept Testing and Product Optimization services.

1. Data collection mode: While a respondent’s words taken during phone surveys or in-person interviews take more importance given the conversational format, online surveys’ visual design elements have a bigger impact on how questions are read and interpreted.  When designing a survey online, it is important to be aware of the question types that are a good fit.
2.  Respondent effort:  There are questions that put a heavier burden on the respondent’s working memory and comprehension or are likely to elicit higher non-response if asked in different data collection modes.
3.  Question wording: Formulating questions with the right wording so it accurately reflects the issue of interest is one of the hardest parts in writing questionnaires. Data errors can sift through a survey if you use unfamiliar, complex, or technically inaccurate words, ask more than one question at a time, use incomplete sentences, use abstract or vague concepts, make the questions too wordy, ask questions without a clear task or ask questions that lead respondents to a particular answer.
4.  Question sequence: Questions should follow a logical flow. Order inconsistencies can confuse respondents and bias the results.
5.  Question format: Questions can be closed-ended or open-ended. Closed-ended questions provide answer choices, while open-ended questions ask respondents to answer in their own words. Each type of question serves different research objectives and has its own limitations. The key issues here are related to the level of detail and information richness we need, our previous knowledge about the topic, and whether to influence respondents’ answers.  
6.  Information accuracy: Some questions yield more accurate information than others.   Respondents can answer questions about their gender and age pretty accurately, but when it comes to attitudes and opinions in a particular issue, many may not have a clear answer.  Overall, attitudes and opinion questions should be worded in a way that best reflects how respondents think and talk about a particular issue.
7.  Measured behaviors: People tend to have less precise memories of mundane behaviors they engage in on a regular basis, and usually they do not categorize events by periods of times (e.g. week, month, and year).  We need to consider appropriate reference periods for the type of behavior we want to measure. Measured behavior should be relevant to the respondent and capture his or her potential state of mind.
8.  Question structure: Questions have different parts that must work in harmony to capture high-quality data. These are the question stem (e.g. what is your age?), additional instructions (e.g. select one answer) and response options, if any (e.g. Under 18, 19 to 24, 25 +). The wrong combination can leave respondents baffled about how to answer a question.
9.  Visual layout: Using design elements in an inconsistent way can increase the burden put on the respondent in trying to understand the meaning of what is asked. Different font sizes, colors, and strengths across questions, forces the respondent to relearn their meaning every time they are used. Also presenting scales with different directions (positive to negative or vice versa) in rating questions within the same survey increases measurement error as respondents often assume all rating questions have the same scale direction even when the instructions explain the meaning of the end points of the scale.
10.  Analytical plan: Based on the research object, both the type of information requested and the question format are important for the type of analysis we plan to perform once the data is collected. There is also the question of whether you want to replicate the results, track certain events or just run a one-time ad-hoc analysis. If the goal is to track certain metrics, time and care should be dedicated to craft tracking questions, as slight changes in wording can change the meaning of a question and thus its results.

 If you take each of these aspects of survey writing into consideration, you will be on your way to creating surveys that produce valid data and support in order to make tactical and strategic business decisions with confidence.

If you would like to learn more about survey design, read Intelligent (Survey) Design, the full version of this article as published in Quirk’s Marketing Research Review, July 2010 issue.

To learn more about Relevant Insights’ consumer data service visit Consumer Shopping Behavior Insights. To request consumer shopping behavior data and insights don’t hesitate to contact me, Michaela Mora.

• Bulletin boards
• Immersion IDI’s
• Mobile qualitative
• Quantitative-Qualitative hybrids
• Real time chats
• Research blogs

As well as:

• Research communities
• Social network monitoring
• Video journals
• Webcam focus groups
• Web-Intercepts/chats

 These new qualitative research techniques have often made qualitative research better, faster and cheaper. Studies using these online qualitative research techniques can be deployed pretty quickly and provide immediate access to transcripts or videos for review.

Below are some of the most often used online qualitative techniques, their advantages, disadvantages, and applications.

Another approach that is getting popular is hybrid research where quantitative and qualitative research are combined in one data collection opportunity. Hybrid research can also be done using SurveyGizmo, which has the capability to integrate chat sessions from iModerate. 

The main of a hybrid approach are:

• Immediacy since there isn’t a lag between quantitative and qualitative data collection
• Can be very cost effective since cost incurred in recruitment, travel and focus group facilities can be eliminated
• Ability to dive deep into the story behind the numbers
• Ability to segment qualitative data together with quantitative results

Mobile qualitative research is also being considered as a data collection tool. This can be easy and comfortable for the participants, although it is currently limited to texting only. This technique can be used to reach difficult groups, send reminders about “homework” given to study participants, and do research at the point of consumption.

All these new techniques make the qualitative research field exciting and promising, but before jumping in, it is always important to have clear research objectives and then you can evaluate if these techniques are a good fit for what you want to accomplish.

If you need help in determining the best research approach for your business objectives, don’t hesitate to contact us at Relevant Insights.

In fact, it is rare that I don’t have to create the same exact report displaying the data in the previous mentioned ways. Thankfully by using SPSS syntax, and some planning, this can be painless.

The Temporary command allows you to temporarily transform the data without making permanent changes. The transformation works for the very first procedure, then it reverts back to the original values.

Here is the data file created from this questionnaire within SurveyGizmo 3.0. (I took advantage of the new “Generate Test Data” feature in 3.0).

If you simply run Descriptive statistics on the data file you will get the average for each variable. Here is the syntax for obtaining the means sorted lowest to highest:

DESCRIPTIVES Hot to Elephant /STATS=MEAN /SORT (a).

You should see a table similar to the following table (Mine will look slightly different as I have scripts that automatically run and tweak my output)

Now what if your client wanted the top-3 box instead of mean? TEMP to the rescue! Simply Add two lines in front of the Descriptives command as I have below:

TEMP.
RECODE hot to elephant (1 thru 4=0) (5 thru 7=1).
DESCRIPTIVES hot to elephant /STATS=MEAN /SORT (a).

Make sure that, when you run the code, you run all three lines. If you only run the 2nd and 3rd line the transformation will not be temporary and you’ll end up transforming your data. (This is probably okay as you haven’t saved it, however, you’ll have to reload the data file if you wish to have the original values)

If your client were to want the top two box, simply change the recode line to the following:

RECODE hot to elephant (1 thru 5=0) (6 thru 7=1).

OTHER USES OF TEMPORARY

Does Temporary have other uses? Why, yes it does. Let’s say that for a particular graph you wanted to tweak the variable labels. Below is an example using Temp with variable labels and recoding to show top box.

TEMP.
RECODE hot to elephant (1 thru 6=0) (7=1).

VAR LABEL elephant “No idea elephant…”.
DESCRIPTIVES hot to Elephant /STATS=MEAN /SORT (a).

Another common way I use the Temp command is when I wish to suppress a small sized group when reviewing the ANOVA test via the Means command. The following syntax runs the Means command with that ANOVA test to see if there is a statistical difference on the elephant statement by gender.

MEANS elephant BY gender /CELLS MEAN COUNT /STATS ANOVA.

As you’ll see there were 13 respondents that did not know their Gender. (Perhaps the crying game fan club?) Anyway, the ANOVA test (not shown here) comes up with a Significance value of .001 (meaning 1 time in 1,000 this difference would have been found simply due to chance) Looking a little closer at the scores, the average of the “Don’t know” group is very low. What would happen if we were to exclude them out from the test?

A quick way to do this is to temporarily define them as missing. The following code does this as well as runs the means test.

TEMP.
MISSING VALUES Gender (3).
MEANS elephant BY gender /CELLS MEAN COUNT /STATS ANOVA.

Notice that now when only comparing between Men and Women, the significance level is .115 (We probably would not state there is a difference between them)

USING WITH SELECT IF STATEMENTS

On my larger studies, I frequently have to run the report for various business departments, Regions, Countries, etc. On shorter reports, combining the TEMP command with SELECT IF is very helpful. For instance, if we wanted to show the top-2 box scores for Males, we would use the following:

TEMP.
SELECT IF gender=1.
DESCRIPTIVES Hot to Elephant /STATS=MEAN /SORT (a).

On longer reports, where I’m creating an entire sub-report on a given country, region, etc., I typically use the SELECT IF statement without the TEMP command. This allows me to filter the data once and run the entire report. I just make certain not to save the data after I’ve filtered it.

Those were some quick tips on using the TEMP command within SPSS so you can analyze your survey data even further.

Wildflower Marketing (real name changed) uses surveys frequently to test marketing ideas, profile target markets and evaluate new product ideas. They usually come to me with a draft survey and ask me to review and comment on their questions.

This case study is a recent request from this client. I’ll go through a sample of the input I gave on their survey questions. Along the way I will highlight the general principles that specific examples help illustrate.

I was sent their draft questionnaire and asked to comment. My first question back to the client was, “What is the purpose of your survey, what are your main objectives?”

General Take-away #1: One can’t make meaningful comments about a survey unless he or she knows exactly what the surveyor plans to do with the information.

In this case, the main purpose of the survey was to obtain information about the target market, herbal tea drinkers, in order to help establish marketing strategies for selling herbal tea.

The survey’s first question was, “What do you consider to be herbal tea?” which had multiple-choice selections. The second question was, “Do you drink herbal tea?” I recommended that in between question one and question two my client include a definition of “herbal tea” for all respondents to use for the remainder of the survey.

General Take-away #2: Make sure terms and jargon are defined unambiguously. A survey question depends on the fact that all the respondents have the SAME understanding of it. It is not enough that everyone understand it.

Here are the next couple of questions and my input (in bold).

Q. Do you MAINLY use herbal tea bags or loose tea? * [Select one]
* Tea bags
* Loose tea
* Both (add this option)
* I don’t know

This simple change makes it easier for the respondent to answer and actually provides more information. No evaluation of “mainly” by the respondent is needed, and information about how many use both methods is added.

Below is an example where use of a similar type of word, “typically,” can actually make it easier to answer for the respondent because it encourages the respondent to go with their gut feeling.

Q. When you drink herbal tea, how do you typically prepare your herbal tea? *
[Select up to two]
* Iced
* Hot in teapot
* Hot in cup
* Sun tea
* Other (please specify)

Q. What is your MAIN reason for drinking herbal tea? *

[Assign no more than 100 points in total across the categories below]
* Flavor
* Health benefit (e.g. digestive, detox, laxative, cold care)
* Comfort (e.g. soothing)
* Habit
* Morning ritual
* For a “pick me up”
* Boredom
* To warm up
* To be sociable
* Instead of coffee
* Doctor’s orders
* Don’t know

My client has his own ideas about why people drink tea, but it doesn’t take much thought to think of possible other reasons. Above, you can see the items I added as suggestions. If a respondent sees his thoughts in the question options, it makes it easier for him or her to answer and it produces better data.

General Take-away #3: Don’t bias your survey questions toward your specific objectives, work to expand your thinking toward how a respondent might think. You can also accomplish the same thing by having someone else review your survey and provide his or her input!

At this point I noticed all of the questions on the survey were required. At best this is totally unnecessary and at worst it is extremely annoying for the respondent.

General Take-away #4: Only make a question required if absolutely necessary like when it is a branching or skipping question.

The next question on the survey was, “How often do you buy herbal tea?” This question seems harmless enough but I asked how this information would be used. It would be impossible to correlate the answers with the amount of tea one drinks or how much tea they buy or even how often they drink tea since there is no mention of how much they buy or when they buy. I encouraged my client to more fully explore the reasons behind this question and what he was really looking for.

Here were the next couple of questions.

Q. Have you ever had any of the following health conditions? [Select all that apply]
* Anxiety disorders
* High blood pressure
* Back Pain
* Cancer
* Sexual conditions
* Migraines
* Irritable bowel syndrome
* Diarrhea
* Obesity
* Depression
* Skin Problems
. . .
Etc. (the list goes on!)
* None

Q. Which alternative medicines or therapies have you used? [Select all that apply]
* Yoga
* Massage
* Homeopathy
* Meditation
* Martial arts
* Acupuncture
. . .
* Etc.
* None

At this point in the survey my questions were: Are these questions necessary? Where did the lists come from? Where are you going with these questions?

The questions hit me as coming out of the blue. If these questions were necessary, then it would be important to prepare the respondent for them and let the respondent know where you are going. Something as simple as – /m=“Some people use herbal teas for medicinal purposes, have you considered their use for any of the following conditions? [Select all that apply]” – could work.

General Take-away #5: Think about how your questions are going to make your respondent feel. In general people will answer personal questions or questions about sensitive topics but they need to feel it is both necessary and critical information for you, the surveyor, to have. Throwing in a question totally out of the blue (of a personal nature or not) can be a turnoff especially when the purpose for the question has not been made clear.

My next recommendation was for my client to change their ranking question:

What are your TOP criteria for selecting a brand of herbal tea? [Place in rank order of importance]
* Clinically tested
* Price per container
* Environmental practices of manufacturer
* Price per bag
* Flavor
* Certified organic
* Effectiveness
* Fair trade certified

I prefer to use checkbox questions as opposed to the ranking question because I feel that ranking questions can be time consuming for respondents. (Next month’s blog will explore this issue in depth) I recommended that the client change this question to:

Q. What are your TOP criteria for selecting a brand of herbal tea? [Select no more than 3]

This will give them a natural ranking of the items and not increase the burden on the respondent.

General Take-away #6: Ranking questions can be time consuming for a respondent to answer. A good alternative is to ask them to select NO MORE THAN their top X items where X is roughly one-third of the total number of items (see next month’s article for the details behind this recommendation).

My final recommendation was for the client to eliminate the following question appearing at the end of the survey.

Q. Please share any additional information you feel would be relevant to this study.

To me you might as well reword this question as, “I don’t have time to think about whether I’ve included everything. Can you help me out and tell me what might be missing, even though you don’t really know what my objectives are or how I plan to use the information you provide?” Not a good question to ask.

General Take-away #7: Don’t burden your respondent with work you are unwilling to do yourself. They typically do not have as much invested in your survey as you do.

This is the third in a series of case studies focusing on Survey Design. Next time I will talk about different methods for obtaining a ranking of a list of items.

Since the advent of online panels and the increase of online surveys using panel-provided samples, the issue of testing for significant differences using standard parametric tests has become a moot point in many research studies.

Nowadays many of the surveys conducted online use samples provided by online panels, but these are mostly convenience samples (non-probability). The populations of online panels include respondents who are willing to participate in studies, excluding those unwilling to be part of the panel who may be members of the target population we are after.

In probability sampling, each possible respondent from the target population has a known probability to be chosen. Probability sampling helps us avoid some of the selection biases that can make a sample not representative of the target population. For more on this read Representative Samples – Does Sample Size Really Matter?

Unfortunately, taking a probability sample is hard and costly. For most consumer research studies and social behavior studies, we really don’t know the size of the actual population of consumers behaving in certain ways or consuming certain product. Trying to find this out would make the research prohibitively expensive. This is why we often have to settle for convenience samples like the ones offered by online panels. They still can offer valuable insights if designed with care, but again doing statistical testing in a convenience sample is pointless since the assumptions about probability sampling are violated.

Online panels are here to stay, and they will continue to be a source for affordable sample for market research. Research using convenience sample is often better than no research at all if the survey is well designed and screening criteria are used to define the target population.

A more appropriate case for testing statistically significant differences are random samples taken from a customer database, since this is essentially the population frame where we can count all members and estimate their probability to be chosen.

However, if you don’t have a customer database or are interested in surveying non-customers, then use a convenience sample. You may feel more confident in your sample if you are able to replicate the results in repeated surveys, but always be cautious about inferences made from convenience samples since there could be a hidden systematic bias in the data.

It is always important that whenever you use convenience samples  you consider the following when analyzing the results:

1. Who is systematically excluded from the sample?

2. What groups are over or under represented in the sample?

3. Have the results been replicated with different samples and data collection methods?

 If testing for significant difference gives you peace of mind, even when using convenience samples, do it to confirm the “direction” of the data, but restrain yourself from doing inferences to a larger population.

For help on sample size calculation check out Survey Sample Size – What Should It Be?

To calculate sample size and margins of error, use the Sample Size and Margin of Error Calculators from Relevant Insights.

Photo courtesy of lrargerich – Flickr, Creative Commons (Attribution)