“Dear Boulder Resident,

[...] To get a representative sample of people living in Boulder, this questionnaire should be completed by the adult (anyone 18 years or older) in your household who most recently had a birthday.

Please have this person take a few minutes to answer all the questions and return the survey in the postage-paid envelope [...].”

Hmmm. A few minutes? Are they kidding? The survey had 170 questions and 8,500 words on it. Spending just 6 seconds on each question (and how important it is to the City of Boulder) means the survey would take more than 15 minutes. Thoughtfully considering each question could take hours.

A survey like this undoubtedly creates “survey fatigue,” but that alone is probably not its biggest problem. If done correctly there are situations where a 170-question survey can be appropriate – but that’s a topic for another article.

The biggest problem with this survey is that the authors made a number of mistakes when putting it together. Here’s the list of the most common mistakes survey authors make from my previous blog entries:

10 Common Mistakes Made When Writing Surveys

1. Having little or no understanding of the target audience
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions (online surveys only)
4. Asking too many open-ended questions
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions
7. Asking too many questions
8. Asking two questions in one
9. Making questions too general
10. Putting too little thought and planning into writing the survey, period

The survey above has been useful in helping me refine the list of common mistakes (which supports the old adage that nothing is ever a complete failure, as it can often serve admirably as a bad survey example). To wit, I have refined the list (refinements shown in red below):

10 Common Mistakes Made When Writing Surveys

1. Having little or no understanding of the target audience and what information they will be able to provide
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions (online surveys only)
4. Asking too many open-ended questions (or asking open-ended questions that are not useful)
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions or ones that won’t produce usable information
7. Asking too many questionsand/or including disjointed laundry-list questions
8. Asking two questions in one
9. Making questions too general
10. Putting too little knowledgeable thought and planning into writing the survey, period

Below are some examples from the survey that support this refinement of my list of mistakes.

First, an example of the type of open-ended question that is not useful:

What do you think should be the top three priorities of the Boulder City Council in 2012?

This type of question will never lead to gathering useful information, except by accident. The reason for this is the respondent has not been given any context within which to answer the question, such as: What’s on the City’s docket? What kinds of things can and will the Council address? Are there budget constraints? It is not fair to respondents to require them to guess at context. This type of question is what gives open-ended questions a bad name.

Next is an example of one of several “laundry-list” questions that were included in the City of Boulder 2011 Community Survey:

Give your respondents a break, literally. At least break the list up into smaller pieces. A more thoughtful way to present the question would be to group items into topic areas.

In summary, the main mistakes in the City of Boulder 2011 Community Survey were:

Mistakes 4 and 7 were the result of survey mistake #10: the authors did not put enough knowledgeable thought and planning into writing the survey. As a result, the data collected by this survey will not necessarily provide the information they want. There is really no way to know exactly how survey fatigue has affected the data collected. A survey of this length will certainly exclude a significant portion of the target audience.

I will address the issue of appropriate survey length in my next blog post. Happy Survey-Taking!

Image courtesy of opensourceway – Flickr, Creative Commons (Attribution)

As I said last time, although it’s nice to know the most common survey mistakes made, the real value is in understanding how to avoid them. You can do that by understanding why each is a mistake.

6. Asking unnecessary survey questions

It is easy when writing surveys to fall into the trap of wanting to know everything. However, as a survey writer you owe it to your respondents to only ask questions from which the resulting data will be used to take action or make a decision. Respondents can sense when you are asking a question that isn’t needed and will not be used. The two most common types of unnecessary questions are asking about something that has already been decided and asking about things over which you have no control.

7. Asking too many questions

This survey mistake appears straightforward, but is often misunderstood. I’m often asked how many questions can be asked on a survey; however, there is no magic number for the right number of survey questions. The two limiting factors are: 1) The commitment and attention span of the target audience, and 2) the resources and time the survey owner has for acting on the information received.

For example, when it comes to commitment to a survey, you can’t ask as much time of a general audience without an investment in your subject than you can of a loyal customer or a dedicated employee. If respondents are committed to your subject and are kept engaged by the survey instrument, they will spend the time it takes to complete a long survey.

The second limiting factor, the resources and time the survey owner has for acting on the information received, usually overrides the first. The only reason to do a survey is to use the information obtained. Thoughtfully using this information and making changes or improvements takes a considerable amount of time and effort. It is better to implement shorter surveys more often than to implement one survey that produces five years of work.

8. Asking two survey questions in one

This is a great way to frustrate your respondents and give you ambiguous data. For clarity, let’s look at two examples. The first is the question: “Please rate the technician’s knowledge and professionalism.” This is clearly two questions. The technician’s knowledge may be great and his professionalism lousy.

The second example, asked of a high school counselor, illustrates a more subtle way of making this survey mistake. “Do you interact with your students’ parents about college?” Again, this is really two questions: “Do interact with parents?” and, if so, “Do you talk about college?” This mistake can be fixed in either of two ways. You can ask both questions separately or you can ask the one question and adjust your response choices to include both “I don’t interact with students’ parents” and “I interact with students’ parents but not about college.”

9. Making questions too general

The problem with questions that are too general is that two respondents can sometimes answer the question the same but for completely different reasons. For example, “Do you believe wireless devices can cause health problems?” Clearly, there are many ways for people with very different views to answer this question “Yes”. One person may feel it is a remote chance while another may think it is an absolute certainty. The quality of the information obtained from a survey depends on asking focused, unambiguous questions specific to the survey objectives. A better approach for determining people’s beliefs about the dangers of wireless devices might be to ask, “Do you curtail your use of wireless devices specifically to avoid risk to your health?”

10. Putting too little time and effort into writing the survey, period

The fact is, it is easy to write a survey with lot of questions and send it out to a broad group of people. The difficulty is getting usable information that can help with solid decision-making. Every question in a survey needs to be well thought out and evaluated against the survey objectives and the target audience. Too often the results from a survey raise more questions than they answer because the questions weren’t well thought out, reviewed, tested and reviewed again. Extra effort spent writing your survey will pay big dividends when using the data.

With that in mind, I thought I’d share with you the ten most common mistakes made when writing surveys.

Keep in mind that while it is nice to know the most common mistakes made, the real value is in understanding how to avoid them. In order to do that it is important to understand why each is a mistake. I will address them one by one in this and my next article. To start, here are the first five.

1. Having little or no understanding of the target audience

This seems straightforward: how can you write an effective survey if you don’t understand much about who will be completing it? The issue is making a connection with your respondent. The survey writer should know as much as possible about the attitudes and beliefs of the potential respondent. The wrong wording can offend respondents or just steer them away from what you intended. Often, too much focus is placed on what information the survey writer wants to get back, and not enough on what information respondents can provide.

2. Providing multiple choice lists that are too restrictive

It is always a good idea to include answer options that include “don’t know” or “uncertain,” “not applicable” and “other.” Respondents become frustrated when they don’t see their response in a multiple-choice list. The idea is to weed out respondents that don’t have a clear opinion from those that do. Otherwise you risk contaminating the good responses.

3. Requiring answers to all questions (online surveys only)

Nothing is more annoying to respondents than having offered their time to complete a survey and then having trouble progressing through the survey. A few skipped responses is not going to change your results – and ultimately you cannot force respondents to answer a question. If they want, a respondent can just close their browser and forget about your survey.

4. Asking too many open-ended questions

It is good to have comment fields, but too many open-ended questions makes it appear that the survey writer did not want to put in the effort to create easy-to-answer questions focused on survey objectives. The main purposes for open-ended questions in a survey are to provide respondents an outlet for thoughts and opinions that may otherwise distract them from thinking about the questions asked, and to add richness and understanding to the quantitative results obtained.

5. Using ranking questions incorrectly (or overusing them)

The inexperienced survey writer will often overlook the fact that ranking questions are difficult for a respondent to answer and even more difficult for them to analyze and interpret. The mistake made is to assume that the best way to ask a question of one person is the best way to ask the same question of many people. If I have only one customer, then I would want that customer to rank their priorities (one to whatever). That all changes, however, when I have to consider the priorities of many customers together. Asking respondents to select their top three priorities (or two or four, etc.) creates a natural ranking when the data is summarized.

In my next post, I’ll finish my list and address the last five most common mistakes made when writing surveys.

The interested reader might want to refer to Edward R. Tufte’s 2001 book, The Visual Display of Quantitative Information for a more detailed account on how to make sure your reports have “graphical integrity.” It is an excellent book.

Tips for Maintaining Graphical Integrity

1. Always include the number, N, of observations on your chart.

This simple addition to any chart (see example below*) is critical to understanding the data presented. Including the number of observations on your chart let’s the reader reconstruct your data from the percentages given. It also provides an indirect measure of the variability one might expect to see.

2. Use counts instead of percents when N is small, fewer than 10 to 15.

When you have a very small number of survey respondents it is better to plot counts than the percentage of counts (see following two charts). Even though it is possible for the reader to calculate the counts on his or her own, going to counts for small numbers helps the readability of a chart by eliminating the extra step. The human brain can read and interpret small numbers quickly and without ambiguity.

3. Don’t overload your chart or graph.

It is important that your reader be able to get information quickly and efficiently from your chart of graph. There is often a tendency to add too many items to your graph that do not add to the data’s message. Often “non-data” items (e.g. 3-D images) are included that can detract from or mask the true information. Even if you include only data on the chart, it is easy to overload it will information.

The following example shows a common mistake, which leads to a chart that is difficult interpret.

The most critical piece of the data from the chart above is the trend information. Therefore it is best to emphasize that element of the data. If, in addition, you combine the percentages for “Much better” and “Better” as well as for “Worse” and “Much worse,” the chart becomes much easier to read and interpret (see below).

4. Show all of your data.

The complete truth about a data set usually extends beyond what you can fit on a single chart or graph, but you need to strive to show all available data. For example, suppose a manufacturer comes out with a new product release and wants to see what effect the new release has had on service calls. The chart below shows the number of product service calls for the year before the release compared to the same number after the new release.

This seems like a reasonable approach but the two years of data do not begin to tell the whole story. Showing all of your available data better represents the true situation (see chart below). The two charts give substantially different pictures.

As I’ve talked about many times, graphical displays of data can be powerful communication tools. If you follow these few tips it should help you provide clear and concise information and avoid charts that are confusing or misleading to your readers. In my next article I’m going to return to the topic of writing surveys, by going over the top ten most common mistakes made when writing surveys.

*NOTE: The charts and graphs in this blog post were generated by exporting a CSV of SurveyGizmo survey data into Excel.

Wildflower Marketing (real name changed) uses surveys frequently to test marketing ideas, profile target markets and evaluate new product ideas. They usually come to me with a draft survey and ask me to review and comment on their questions.

This case study is a recent request from this client. I’ll go through a sample of the input I gave on their survey questions. Along the way I will highlight the general principles that specific examples help illustrate.

I was sent their draft questionnaire and asked to comment. My first question back to the client was, “What is the purpose of your survey, what are your main objectives?”

General Take-away #1: One can’t make meaningful comments about a survey unless he or she knows exactly what the surveyor plans to do with the information.

In this case, the main purpose of the survey was to obtain information about the target market, herbal tea drinkers, in order to help establish marketing strategies for selling herbal tea.

The survey’s first question was, “What do you consider to be herbal tea?” which had multiple-choice selections. The second question was, “Do you drink herbal tea?” I recommended that in between question one and question two my client include a definition of “herbal tea” for all respondents to use for the remainder of the survey.

General Take-away #2: Make sure terms and jargon are defined unambiguously. A survey question depends on the fact that all the respondents have the SAME understanding of it. It is not enough that everyone understand it.

Here are the next couple of questions and my input (in bold).

Q. Do you MAINLY use herbal tea bags or loose tea? * [Select one]
* Tea bags
* Loose tea
* Both (add this option)
* I don’t know

This simple change makes it easier for the respondent to answer and actually provides more information. No evaluation of “mainly” by the respondent is needed, and information about how many use both methods is added.

Below is an example where use of a similar type of word, “typically,” can actually make it easier to answer for the respondent because it encourages the respondent to go with their gut feeling.

Q. When you drink herbal tea, how do you typically prepare your herbal tea? *
[Select up to two]
* Iced
* Hot in teapot
* Hot in cup
* Sun tea
* Other (please specify)

Q. What is your MAIN reason for drinking herbal tea? *

[Assign no more than 100 points in total across the categories below]
* Flavor
* Health benefit (e.g. digestive, detox, laxative, cold care)
* Comfort (e.g. soothing)
* Habit
* Morning ritual
* For a “pick me up”
* Boredom
* To warm up
* To be sociable
* Instead of coffee
* Doctor’s orders
* Don’t know

My client has his own ideas about why people drink tea, but it doesn’t take much thought to think of possible other reasons. Above, you can see the items I added as suggestions. If a respondent sees his thoughts in the question options, it makes it easier for him or her to answer and it produces better data.

General Take-away #3: Don’t bias your survey questions toward your specific objectives, work to expand your thinking toward how a respondent might think. You can also accomplish the same thing by having someone else review your survey and provide his or her input!

At this point I noticed all of the questions on the survey were required. At best this is totally unnecessary and at worst it is extremely annoying for the respondent.

General Take-away #4: Only make a question required if absolutely necessary like when it is a branching or skipping question.

The next question on the survey was, “How often do you buy herbal tea?” This question seems harmless enough but I asked how this information would be used. It would be impossible to correlate the answers with the amount of tea one drinks or how much tea they buy or even how often they drink tea since there is no mention of how much they buy or when they buy. I encouraged my client to more fully explore the reasons behind this question and what he was really looking for.

Here were the next couple of questions.

Q. Have you ever had any of the following health conditions? [Select all that apply]
* Anxiety disorders
* High blood pressure
* Back Pain
* Cancer
* Sexual conditions
* Migraines
* Irritable bowel syndrome
* Diarrhea
* Obesity
* Depression
* Skin Problems
. . .
Etc. (the list goes on!)
* None

Q. Which alternative medicines or therapies have you used? [Select all that apply]
* Yoga
* Massage
* Homeopathy
* Meditation
* Martial arts
* Acupuncture
. . .
* Etc.
* None

At this point in the survey my questions were: Are these questions necessary? Where did the lists come from? Where are you going with these questions?

The questions hit me as coming out of the blue. If these questions were necessary, then it would be important to prepare the respondent for them and let the respondent know where you are going. Something as simple as – /m=“Some people use herbal teas for medicinal purposes, have you considered their use for any of the following conditions? [Select all that apply]” – could work.

General Take-away #5: Think about how your questions are going to make your respondent feel. In general people will answer personal questions or questions about sensitive topics but they need to feel it is both necessary and critical information for you, the surveyor, to have. Throwing in a question totally out of the blue (of a personal nature or not) can be a turnoff especially when the purpose for the question has not been made clear.

My next recommendation was for my client to change their ranking question:

What are your TOP criteria for selecting a brand of herbal tea? [Place in rank order of importance]
* Clinically tested
* Price per container
* Environmental practices of manufacturer
* Price per bag
* Flavor
* Certified organic
* Effectiveness
* Fair trade certified

I prefer to use checkbox questions as opposed to the ranking question because I feel that ranking questions can be time consuming for respondents. (Next month’s blog will explore this issue in depth) I recommended that the client change this question to:

Q. What are your TOP criteria for selecting a brand of herbal tea? [Select no more than 3]

This will give them a natural ranking of the items and not increase the burden on the respondent.

General Take-away #6: Ranking questions can be time consuming for a respondent to answer. A good alternative is to ask them to select NO MORE THAN their top X items where X is roughly one-third of the total number of items (see next month’s article for the details behind this recommendation).

My final recommendation was for the client to eliminate the following question appearing at the end of the survey.

Q. Please share any additional information you feel would be relevant to this study.

To me you might as well reword this question as, “I don’t have time to think about whether I’ve included everything. Can you help me out and tell me what might be missing, even though you don’t really know what my objectives are or how I plan to use the information you provide?” Not a good question to ask.

General Take-away #7: Don’t burden your respondent with work you are unwilling to do yourself. They typically do not have as much invested in your survey as you do.

This is the third in a series of case studies focusing on Survey Design. Next time I will talk about different methods for obtaining a ranking of a list of items.

CASE STUDY 1

A leading price optimization and management solutions company, TriBlend Solutions, needs information to guide its marketing efforts. They plan to do a survey of customers and prospects in order to better understand their market and their target audience.

INTRODUCTION

TriBlend Solutions (TS) partnered with the Professional Pricing Society (PPS) to do their survey. TS was considering doing the survey themselves. They had a list of questions and PPS had an account for doing surveys. (This would change!)

The main players for this project were Lori, Lori’s co-worker, Eric and their manager Andy, all with TS. TS was looking to get some help with the design and structure of their survey. Lori found Survey Design and Analysis from a web search and that’s how she came into contact with me. Lori and her team had most of their questions written and suspected it wouldn’t take much time to have someone help them create the final survey.

Lori’s original request was on June 24. They planned to deploy in the fall.

On June 27 I responded to Lori with a request for more information on the following:

• Your survey objectives and reasons for starting this undertaking
• The number of names (email addresses) on your survey list
• Whether you’ve done previous studies that were similar
• The proposed size of the survey (number of questions)

[The design process actually starts in the proposal phase!]

The information she provided was used to create a proposal. After several delays through July we had a conference call on August 2 to discuss the details of our proposal. On August 8th our proposal to design their questionnaire and make recommendations for executing the survey was accepted. A few days later the work was expanded to include the complete execution of the survey. The survey design always includes considerations for execution and Lori realized the complete execution would be a natural extension of the design process.

SURVEY DESIGN PROCESS

A kickoff meeting (conference call) was scheduled for August 14. Prior to the meeting the following survey objectives were enumerated:

• Learn what they can do better
• Look at challenges, prepare for future
• Assess the momentum in industry
• Understand current pricing capabilities

Kickoff Meeting (conference call), August 14 – Ed with Lori, Andy and Eric

This first meeting lasted over two hours on the phone. It went long because it was productive. Everyone felt that we made good progress. There was that “ah-ha” moment when Lori and Eric realized that designing a good survey meant thinking through EVERY question completely. Comment from Lori:

“I know it was a long meeting but we’re so happy to be working with you. We’re really confident that in the end we’re going to have a fabulous survey.”

There were 35 questions in the original list and we talked about each question. What are we trying to get from this question? Will it be ambiguous to the respondent? Is it clear? Here are a couple of examples from our discussions:

1. Has your company’s pricing process been discussed or addressed by your company’s board, investors or financial analysts?
   Yes – answer 5 then go to 6
   No – go to 6
   What does “discussed or addressed,” mean? Are they the same thing or different? And what does “pricing process,” mean? Is the meaning the same for every company who will respond?
2. What is their primary focus or directive?
   “Primary focus or directive”? Are we trying to get at “business goals”? The wording is too vague and will probably mean something different for each respondent.
3. What functional group is driving the pricing improvement initiative?
   Driving or leading? Which are we looking for?
4. Do you plan on implementing price management software?
   Is implementing the right word or is deploying more appropriate for software?

This is a small subset. Every question (all 35) were discussed and evaluated from the viewpoint of the person who would be answering the question. As I’ve mentioned on this blog before, it isn’t so critical that everyone understand every question. What is critical is that everyone has the same understanding of every question. Questions need to be so clear that there can be only one way to view them.

Discussion about the wording for each question led to work on the structure of the questions. How can we order the questions so that they flow evenly and the information builds in a way that makes it easier for the respondent? Are there background or qualifying questions needed to help the respondent see what is coming?

At the end of this first conference call, we realized that we needed an introduction and a few qualifying questions in order to not only help the respondent but also for us to know who (what type of person) was answering the survey.

Our next steps were for TriBlend to put together definitions or explanations for the following terms (this list started out differently and changed several times over the course of the design process):

• Pricing team
• Pricing technology
• Pricing process
• Pricing tool
• Pricing function
• Pricing effectiveness
• Price management software

We also determined that we needed to understand the “pricing activities” that typical companies go through. This would help us have a basic understanding of whom we were dealing with and the perspective they have for answering the questions. TriBlend took on this task.

I created the next draft of the survey, which I’ll call version 4 (TriBlend had gone through a couple of revisions at the onset).

Version 4 included:

The following introduction:

“Thank you for participating in our pricing study. The information you provide will be used to better understand the state of the current technology, challenges and trends in our industry. These questions should take 10-15 minutes of your time. If you are interested in receiving a summary of the results of this study be sure to indicate so on the last page of this questionnaire.”

A list of definitions to which the respondent could refer throughout the survey:

Definitions
In what follows several terms will be used. Please refer to these definitions as needed.

I. Pricing – Any and all activities involved with understanding and or setting prices for a company’s products

II. Pricing Process – Any sequence of pricing-related activities

III. Pricing Tool – Any software, commercial or otherwise, used to support a pricing activity

IV. Price Management Software – A commercial pricing tool specifically designed to improve pricing effectiveness

V. Pricing Effectiveness – The degree to which your pricing processes and or tools produce the desired results in terms of revenue, win rates, margins and predictability

And a set of introductory or qualifying questions:

1. How familiar are you with the “pricing” at your company?
2. What is your role with regard to pricing at your company? (Select all that apply)
3. Overall, how would rate the effectiveness of the pricing processes at your company?
4. How would rate the effectiveness of the pricing tools at your company?
5. What method do you use to set and negotiate prices?
6. Comments On Introductory Questions

[Note the first five of these questions were multiple choice but the choices have been left off for brevity sake.]

Version 4 was sent to Lori, Andy and Eric on August 18 for their review. I asked them to focus on the following points:

• Addition and deletion of questions
• Placement of questions in sections
• Placement of Sections
• Section titles
• Length of the survey and flow of the questions
• Question applicability by type of respondent (managers versus workers)
• Executive only questions (i.e. do some questions apply only to executives)
• Understandability and definitions

As part of the next review we were able to focus on which questions could be answered by which type of customer or prospect. This led to adding branching and question skipping (and a chance for SurveyGizmo features to shine). We also expanded the review to include more people identified by Lori and Eric.

The survey was shaping up but it was a long way from being finalized and ready to launch, as we will see in my next blog entry. The survey still needed detailed refinement, deployment online, more reviews of the online version and testing before it would be ready to launch. As I continue this case study we will watch the survey go to version 12, see the final product and see how all the design work paid off.

To be continued . . .

We’ve all participated in customer satisfaction surveys. Some may have been okay, most probably not. It can be very frustrating if you want to give feedback to a company and their survey asks about only things for which you have no concern. It can also be frustrating if the online survey is hard to understand. Stopping to figure out what is being asked on a survey adds insult to injury. Have you ever taken a customer satisfaction survey and thought, “Was this survey meant for me? Did they send it to the right person?” I know I have.

Getting a customer satisfaction survey right doesn’t have to be that hard. Just keep a few things in mind. A customer satisfaction survey should enhance the customer-supplier relationship, not compromise it. A survey’s quality should, at least, match that of one’s own products and services. I’ve seen a situation where a company with very satisfied customers sent out a customer satisfaction survey that was disappointing. That company’s customers were used to seeing only the highest quality from this company and thus reacted negatively to a satisfaction survey that was poorly thought out and poorly executed.

The goal of a customer satisfaction survey should be to establish a lasting program that will continue to foster good will with customers and provide valuable information for company decision-making for years going forward. Here are some things to think about before undertaking a customer satisfaction survey:

• It is easy to write questions and send them out to customers. It is challenging to design and execute a customer survey that will actually provide useful information for improving your business. If you need help designing your survey questions get the assistance of a SurveyGizmo Survey Expert.
• A customer satisfaction program will raise customer expectations. Most of the effort spent on a customer satisfaction program should be spent after the survey is done and the data is collected. It is important that the budget and resources be there for taking action afterwards.
• The ownership and commitment for the program should be integrated and coordinated across all departments. (Will a sales or service representative know the survey is happening and why?). Spread the word about your customer satisfaction survey around the organization.

Plan for the long haul. Starting and stopping customer satisfaction efforts can do more harm than good. Your customer satisfaction program should be seen as ongoing — not just a one time event.

• Time spent up front to develop the program and clearly understand customers’ wants and needs, will pay dividends for years to come.

We recommend the following two activities for anyone initiating a customer satisfaction program. They will complement the standard techniques for designing any survey.

1. A Customer Needs Assessment – This involves conducting in-depth interviews (face-to-face or over the phone) with a few select customers to make sure you understand what is most important to them, what their needs are, what they think of your company and what they think about your planned customer satisfaction survey. This two-way communication early in the process has been found to be a critical success factor for obtaining actionable information and ensuring that your survey asks the right questions of the right people in the right way.
2. A Stakeholders Strategy Session – The success of a customer satisfaction survey is measured by the actions it drives. It is critical that the survey ask the right questions and address the needs of customers, but it also needs to be integrated into the operations of your company. Since your customer survey is going to be an ongoing part of running your business, it becomes very important for the survey to have a broad base of support and understanding. A strategy session should have key people from sales, marketing, development and operations who get together for a four to six-hour meeting to work on the survey objectives and how they tie to the company mission and values. The session might also include discussions on the appropriate customer segments to target, training from a customer satisfaction expert (internal or external) on best practices and/or defining the scope of the customer feedback you will solicit.

Do yourself and your customers a favor — don’t survey your customers without the appropriate commitment! If you need help in any of these areas a SurveyGizmo Survey Expert can help.

]]> http://www.surveygizmo.com/survey-blog/customer-satisfaction-surveys/feed/ 1 http://www.surveygizmo.com/survey-blog/ed-halteman-a-sampling-attitude/ http://www.surveygizmo.com/survey-blog/ed-halteman-a-sampling-attitude/#comments Fri, 02 Jan 2009 10:00:19 +0000 SurveyGizmo Admin http://wwwjase.sgizmo.com/?p=2058 Read More »]]> “Attitude is everything” has significance for more than just football or other sports! Cultivating a sampling attitude can help you “win” improvements to your survey process.

We have already seen the power of sampling in predicting election results in my prior post on election sampling and intuition tells us that sampling should save time and money. However, having a sampling attitude goes beyond this. A sampling attitude stems from customer focus and leads to better decision-making, improved relationships with customers, employees and/or prospects, and savings in time and money. Why not get the biggest bang from your survey dollar?

A sampling attitude starts with keeping two questions at the forefront:

1. How can I perform my survey in the most efficient manner possible?
2. How can I make my survey as respondent-friendly as possible (so that my survey respondents can provide good information without being inconvenienced)?

Clearly, if we can get the same information from 300 responses to our survey as from 2000 responses then it would be much more efficient to aim for obtaining the 300 responses. However, not only do YOU save time and money going to a smaller sample, but think of all the “respondent time” that has been saved. Customers, especially, appreciate suppliers that show concern for their time!

If we can be more efficient with sample size, let’s keep the attitude going. We can be more efficient in the way we word questions – creating questions that are clear and easy to understand. We can be more efficient in the number of questions we include – including only questions that are absolutely necessary for making our decisions, leaving nice-to-know questions out.

So how do we do all this? Change your attitude and follow these general guidelines:

• Aim for receiving between 150 and 300 responses to your survey.
• Test and re-test your questions for clarity and understanding.
• Spend time to make your survey objectives clear, and then test each of your questions against the objectives to make sure they will add to meeting that objective.

For 2009 improve your attitude and improve your survey process at the same time.

Happy New Year!

]]> http://www.surveygizmo.com/survey-blog/ed-halteman-a-sampling-attitude/feed/ 0 http://www.surveygizmo.com/survey-blog/election-sampling/ http://www.surveygizmo.com/survey-blog/election-sampling/#comments Tue, 11 Nov 2008 23:13:05 +0000 Ed Halteman - A SurveyGizmo Survey Expert http://www.surveygizmo.com/?p=813 Read More »]]> Election season is the time for lots of polls, which makes it a good time to highlight what they can teach us about our surveys! Basically a poll is just a type of survey and so the principles that make polling powerful apply to surveys.

Can the results of polls be trusted? I just read an article today on usatoday.com that stated: “[Sarah] Palin said she did not realize until the night of the election that the ticket would probably lose and was initially surprised at the margin of the loss. Palin said she had felt that voters would in large measure pick the Republicans, despite the polls.” The fact is the pre-election polls were spot on and exit polling enabled the news rooms to call Pennsylvania for Obama before even 20% of the precincts there had reported, thus ending any hope for the McCain-Palin ticket.

I, for one, wasn’t surprised by the results. Polls have been predicting elections for decades and the good pollsters have perfected their methods. The same methods you and I should be using for surveys.

So what makes polls work? First, it takes good survey design to get good results. There are three kinds of bias to avoid when designing your poll (or survey):

• Selection bias
• Non-response bias
• Questions bias

Selection bias occurs when you select participants in such a way that you end up with a disproportionate number from a group with known preferences on the poll topic. For example, for a political poll you chose your random sample from big cities (New York, Chicago and Los Angeles). Big cities are known to favor democrats and thus are not representative of the country as a whole.

Non-response bias is bias associated with the group that does not respond to your poll. If that group tends to have different preferences on the poll topic from those that do, then bias is introduced. For example, if we wanted to poll the public about their reactions to responding to polls, those that answer will most likely have different views than those that choose not to respond. There is always non-response, but telephone surveys often avoid bias because whether someone answers the phone is independent of the survey topic.

Question bias is the bias introduced by the way your questions are asked. Leading or misleading questions bias your results (question wording is a foolproof way to weed out surveys that lack credibility). For example, “National health care in Canada has led to substandard care; do think the USA should have national health care?”

Second, there are a couple of real world issues that pollsters think about to make sure they hit the target. One, polls are snapshots in time and people can change their views or shift their opinions, so pollsters take this into account by setting the frequency of polling to match the likelihood for changing views. For presidential elections they even do daily tracking polls! Since most polls try to predict the future, change can cause problems. This is why “Exit Polls” are so much more accurate. An exit poll asks participants how they DID vote not how they WILL vote. The issue of evolving viewpoints is eliminated for an exit poll. Two, pollsters use chance to their benefit. If hundreds of polls are done, then just by chance you might expect a small number of them to be off. That is why pollsters will look at polls over time.

Third and finally, polls rely heavily on sampling to get the results they do! One of the hardest things for a layperson not familiar with polling methods to understand is how a poll of 1200 voters can accurately predict the votes of 20 million people. The fact is; that is the power of sampling.

If the biases mentioned above are avoided then the error in the sampling is the only thing left to consider. First of all, the accuracy of a sample does not depend on the number in the target population (20 million), it only depends on the sample size (1200). The margin of error for a sample of 1200 is 3-4% which is the accuracy needed for political polls. The margin of error for a sample of 250 is 7-9%. Most surveys do not warrant the level of accuracy needed for polls. That is why I usually recommend sample sizes between 200 and 300 for surveys. Increasing accuracy beyond what is needed increases costs (and resources) without increasing information.

In order to illustrate why a poll needs a sample of 1200 and your survey may only needs a sample of 300; consider a political poll where the actual population is divided 52% to 46% (the final actual percentages in this year’s election). A margin of error in the range of 3-4% is clearly needed to predict a winner when sampling. Now consider your satisfaction survey that shows 52% satisfied customers and 46% dissatisfied customers. Does our reaction to these results change when we switch the numbers; 46% satisfied and 52% dissatisfied? Hardly, it is inconsequential, either way your action is the same; you need to improve customer satisfaction! In the case of the political poll 52% to 46% is vastly different than the reverse!

So, think about some of these issues for your next survey and you might want to consider using the power of sampling!

]]> http://www.surveygizmo.com/survey-blog/election-sampling/feed/ 0 http://www.surveygizmo.com/survey-blog/value-added-surveys/ http://www.surveygizmo.com/survey-blog/value-added-surveys/#comments Thu, 25 Sep 2008 10:00:49 +0000 Ed Halteman - A SurveyGizmo Survey Expert http://www.surveygizmo.com/value-added-surveys/ Read More »]]> It’s been awhile and 2008 has been a busy year. It has also been a year where people have been looking to get more out of their surveys. I have had a steady stream of people coming to me with questions like:
• I have all this data, but I’m not sure what it is telling me?
• How do I get better information from my survey, for making my decisions?
• How do I get more people to respond to my survey?
• I have thousands of responses but how do I present the results?
• Is my survey too long?

Sound familiar? SurveyGizmo has made it easy for thousands to build their own surveys and naturally they want to get the most out of their efforts.

Here are some tips for adding value to your surveys.

No survey is too long or too short. Respondents have the time for what’s important to them and have no patience if their time is being wasted. Respondents can tell how well thought out your survey is, whether they are providing useful information and whether your survey is “talking” to them. The bottom line is: know your audience and engage them.

Do unto others as you would have them do unto you. Take your own survey. Don’t just read it, actually take it as if you are one of those receiving it. This will give you your first clue for how “user friendly” your survey is.

The time to figure out what your data is telling you is before you send out your survey. This may sound funny but it is a valuable exercise. Go through your survey questions. Then imagine (or make up) what your responses are going to look like. Try different scenarios and ask what this imaginary data is telling you while you still have the chance to modify your questions if needed.

Test drive your decisions before you send out your survey. You start your survey with some decision that you want to make (if not, you need to rethink why you are doing the survey). Once you design your questions, go back and check to make sure you will get the information you need for making your decision.

Design your questions with your reports in mind. The time to think about how you are going to report your results is when you are designing your questions. This will give you a chance to see whether the information will be easy to present. If the presentation of the results is going to be difficult the information is not likely to be useful.

]]> http://www.surveygizmo.com/survey-blog/value-added-surveys/feed/ 0