These are all good questions, but unfortunately there is no ONE good answer. The “right” number of questions for a survey depends on the survey, the survey audience, and your resources for acting on the information received.

This means we need to look at the factors that determine an acceptable length for a survey and use those to guide our thoughts on survey length (for both online surveys and offline surveys).

The main factors that determine an acceptable length for your survey are:

1. The relationship between you and the respondent.
2. The relevance of your survey’s subject matter to the respondent.
3. The thought put into the survey questions and how the survey flows.
4. The likelihood that you will use the information from the survey to make a decision.
5. The resources (time, money, and people) you have to implement the results of the study in a timely fashion.

Let me explain each factor so that you can understand how each affects the acceptable survey length.

1. The relationship between you and the respondent.

As an example, a very loyal customer with extensive knowledge about your product can stay engaged through a longer survey. The stronger the relationship, the longer a survey can potentially be. On the other hand, if a marketer is reaching out to the general public for opinions on a new product with no prior relationship to the target audience, he or she needs to be looking at a keeping the survey short and focused.

2. The relevance of your survey’s subject matter to the respondent.

If you are doing a survey to get opinions on golf products from golf enthusiasts then the survey certainly can be longer than a community survey on the workings of its local government. In general, people don’t mind spending a few minutes on a survey of passing interest, but will spend quite a bit longer on a survey that covers a topic near and dear to them.

3. The thought put into the survey questions and how the survey flows.

Respondent-friendly surveys are easier to complete and thus reduce survey fatigue. While writing the survey you need to think about how much effort it will take a respondent to answer the questions. The question type (e.g. ranking questions), the content involved, and the length of the survey questions all deserve consideration when aiming for a respondent-friendly survey.

4. The likelihood that you will use the information from the survey to make a decision.

It may seem strange to think that your actions after the survey can affect how a respondent feels about your survey, but a respondent can usually tell how useful the information they are giving will be. If your respondent is wondering how the information they provide can possibly be used or is wondering why certain questions are being asked, they will likely lose interest quickly.

5. The resources (time, money, and people) you have to implement the results of the study in a timely fashion.

If a respondent sees action taken as a result of his or her input then they are more likely to complete a future survey thoughtfully. Many surveys are used to identify improvement opportunities, but budgets and/or resources may limit the ability to implement the opportunities identified. It might take years to act meaningfully on information gathered from 100-question survey, for example, even if they were all essential questions and engaging for the respondent. In a case like this, the survey should be broken into a number of shorter surveys and the results of each can then be acted on in a timely manner. This will keep your respondents interested in taking part in each of your surveys.

These are five factors to consider when deciding whether your survey is too long or not. You should also obtain feedback by surveying a pilot group from the target audience to help with the same decision.

In my next blog post, I’ll look at things you can do to enhance survey response rate.

Image courtesy of lisalou66 – Flickr, Creative Commons (Attribution)

“Dear Boulder Resident,

[...] To get a representative sample of people living in Boulder, this questionnaire should be completed by the adult (anyone 18 years or older) in your household who most recently had a birthday.

Please have this person take a few minutes to answer all the questions and return the survey in the postage-paid envelope [...].”

Hmmm. A few minutes? Are they kidding? The survey had 170 questions and 8,500 words on it. Spending just 6 seconds on each question (and how important it is to the City of Boulder) means the survey would take more than 15 minutes. Thoughtfully considering each question could take hours.

A survey like this undoubtedly creates “survey fatigue,” but that alone is probably not its biggest problem. If done correctly there are situations where a 170-question survey can be appropriate – but that’s a topic for another article.

The biggest problem with this survey is that the authors made a number of mistakes when putting it together. Here’s the list of the most common mistakes survey authors make from my previous blog entries:

10 Common Mistakes Made When Writing Surveys

1. Having little or no understanding of the target audience
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions (online surveys only)
4. Asking too many open-ended questions
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions
7. Asking too many questions
8. Asking two questions in one
9. Making questions too general
10. Putting too little thought and planning into writing the survey, period

The survey above has been useful in helping me refine the list of common mistakes (which supports the old adage that nothing is ever a complete failure, as it can often serve admirably as a bad survey example). To wit, I have refined the list (refinements shown in red below):

10 Common Mistakes Made When Writing Surveys

1. Having little or no understanding of the target audience and what information they will be able to provide
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions (online surveys only)
4. Asking too many open-ended questions (or asking open-ended questions that are not useful)
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions or ones that won’t produce usable information
7. Asking too many questionsand/or including disjointed laundry-list questions
8. Asking two questions in one
9. Making questions too general
10. Putting too little knowledgeable thought and planning into writing the survey, period

Below are some examples from the survey that support this refinement of my list of mistakes.

First, an example of the type of open-ended question that is not useful:

What do you think should be the top three priorities of the Boulder City Council in 2012?

This type of question will never lead to gathering useful information, except by accident. The reason for this is the respondent has not been given any context within which to answer the question, such as: What’s on the City’s docket? What kinds of things can and will the Council address? Are there budget constraints? It is not fair to respondents to require them to guess at context. This type of question is what gives open-ended questions a bad name.

Next is an example of one of several “laundry-list” questions that were included in the City of Boulder 2011 Community Survey:

Give your respondents a break, literally. At least break the list up into smaller pieces. A more thoughtful way to present the question would be to group items into topic areas.

In summary, the main mistakes in the City of Boulder 2011 Community Survey were:

Mistakes 4 and 7 were the result of survey mistake #10: the authors did not put enough knowledgeable thought and planning into writing the survey. As a result, the data collected by this survey will not necessarily provide the information they want. There is really no way to know exactly how survey fatigue has affected the data collected. A survey of this length will certainly exclude a significant portion of the target audience.

I will address the issue of appropriate survey length in my next blog post. Happy Survey-Taking!

Image courtesy of opensourceway – Flickr, Creative Commons (Attribution)

The topic was political. The purpose seemed to be to poll the opinions of voters. As I first answered the call, I hesitated to continue, but given that I’m in the survey business I went forward. (I usually try to participate in surveys at least until my patience runs thin.)

There was a slight introduction asking if I was registered to vote, etc., but not much else. The survey robot seemed friendly but clearly refused to laugh at any of my jokes. (That should have been a clue.)

I answered the first few survey questions without much trouble. All required Yes-No responses. Questions like: “Did you vote in the last election?” “Do you think Boulder’s City Council is doing a good job?” Then things started to get more involved. Soon there was a question I couldn’t answer without some thought. “Hmmm, I’m not sure,” I said. The robot responded, “This question requires a ‘Yes’ or a ‘No’ answer.” I hesitated again, not knowing how to answer and the robot seemed to get annoyed, replying, “This survey will end unless I receive a ‘Yes’ or a ‘No’ answer.” Suddenly, I panicked. “Yes,” I said.

Whew! That seemed to calm down the robot and the survey continued with a few more questions before there was another for which my real answer was not “Yes” or “No.” It was probably something like, “I’m not sure” or “I haven’t decided yet” or “It depends” or “I don’t understand the question.” Once again the robot insisted, “You must answer ‘Yes’ or ‘No’.” As I tried to explain to the robot why I couldn’t answer “Yes” or “No,” the line went dead and I assumed that meant the survey was over.

Lots of Questions

As I hung up the phone, a host of questions rattled through my head:

• How do the survey sponsors expect to use the data from a survey like this?
• Do the sponsors really think the data they get is unbiased?
• Who writes these surveys, anyway?
• Do the survey authors really think it doesn’t matter what frame of mind people are in when they respond to their survey? Do they even think about that?
• Do the survey authors really think all of their survey questions only have the possibility of a Yes-No response?
• Do the survey authors really think that a respondent’s true feelings, opinions, and attitudes are like items on a shelf, ready for a robot to pick up and put in a box?
• What were they really trying to do with their survey?
• Isn’t it likely that the analysis of the data will ignore the method and circumstances under which the data were collected?
• Will the survey authors give any thought to the data’s validity? Or will the survey data instantly become meaningful (in their minds) once they see numbers or graphs on a page?

That is a lot of questions that went through my mind in the few seconds it took me to put down the phone.

The Mistakes This Survey Made

Clearly, mistakes were made in putting this survey together. I think it would be instructive to go back to my last article, the 10 Common Mistakes Made When Writing Surveys, and see which mistakes from the list were made here.

Let’s take a look. Here’s my list of the common survey mistakes:

1. Having little or no understanding of the target audience
2. Providing multiple choice lists that are too restrictive
3. Requiring answers to all questions
4. Asking too many open-ended questions
5. Using ranking questions incorrectly, or overusing them
6. Asking unnecessary questions*
7. Asking too many questions*
8. Asking two questions in one
9. Making questions too general
10. Putting too little thought and planning into writing the survey, period.

(I’ve bolded the ones I think the authors of the robotic survey made. The ones with stars are the ones about which my premature exit from the survey preclude me from making a clear determination.)

Items 1-3 and 9 are all closely linked. The authors’ understanding of their target audience seems minimal because they didn’t allow for more response choices and the choices they did provide were too restrictive, even to the point of requiring every question to have a “Yes” or a “No” response. The survey questions become too general from the restrictive responses.

The end result of these four mistakes is mistake number 10. The authors did not put enough thought into who would answer the survey, how their survey method would affect the response, and how the information was to be used.

Happy Survey Taking! I’ll talk about the other survey I attempted to take in my next blog post.

Photo courtesy of Jonty Wareing – Flickr, Creative Commons (Attribution)

As I said last time, although it’s nice to know the most common survey mistakes made, the real value is in understanding how to avoid them. You can do that by understanding why each is a mistake.

6. Asking unnecessary survey questions

It is easy when writing surveys to fall into the trap of wanting to know everything. However, as a survey writer you owe it to your respondents to only ask questions from which the resulting data will be used to take action or make a decision. Respondents can sense when you are asking a question that isn’t needed and will not be used. The two most common types of unnecessary questions are asking about something that has already been decided and asking about things over which you have no control.

7. Asking too many questions

This survey mistake appears straightforward, but is often misunderstood. I’m often asked how many questions can be asked on a survey; however, there is no magic number for the right number of survey questions. The two limiting factors are: 1) The commitment and attention span of the target audience, and 2) the resources and time the survey owner has for acting on the information received.

For example, when it comes to commitment to a survey, you can’t ask as much time of a general audience without an investment in your subject than you can of a loyal customer or a dedicated employee. If respondents are committed to your subject and are kept engaged by the survey instrument, they will spend the time it takes to complete a long survey.

The second limiting factor, the resources and time the survey owner has for acting on the information received, usually overrides the first. The only reason to do a survey is to use the information obtained. Thoughtfully using this information and making changes or improvements takes a considerable amount of time and effort. It is better to implement shorter surveys more often than to implement one survey that produces five years of work.

8. Asking two survey questions in one

This is a great way to frustrate your respondents and give you ambiguous data. For clarity, let’s look at two examples. The first is the question: “Please rate the technician’s knowledge and professionalism.” This is clearly two questions. The technician’s knowledge may be great and his professionalism lousy.

The second example, asked of a high school counselor, illustrates a more subtle way of making this survey mistake. “Do you interact with your students’ parents about college?” Again, this is really two questions: “Do interact with parents?” and, if so, “Do you talk about college?” This mistake can be fixed in either of two ways. You can ask both questions separately or you can ask the one question and adjust your response choices to include both “I don’t interact with students’ parents” and “I interact with students’ parents but not about college.”

9. Making questions too general

The problem with questions that are too general is that two respondents can sometimes answer the question the same but for completely different reasons. For example, “Do you believe wireless devices can cause health problems?” Clearly, there are many ways for people with very different views to answer this question “Yes”. One person may feel it is a remote chance while another may think it is an absolute certainty. The quality of the information obtained from a survey depends on asking focused, unambiguous questions specific to the survey objectives. A better approach for determining people’s beliefs about the dangers of wireless devices might be to ask, “Do you curtail your use of wireless devices specifically to avoid risk to your health?”

10. Putting too little time and effort into writing the survey, period

The fact is, it is easy to write a survey with lot of questions and send it out to a broad group of people. The difficulty is getting usable information that can help with solid decision-making. Every question in a survey needs to be well thought out and evaluated against the survey objectives and the target audience. Too often the results from a survey raise more questions than they answer because the questions weren’t well thought out, reviewed, tested and reviewed again. Extra effort spent writing your survey will pay big dividends when using the data.

With that in mind, I thought I’d share with you the ten most common mistakes made when writing surveys.

Keep in mind that while it is nice to know the most common mistakes made, the real value is in understanding how to avoid them. In order to do that it is important to understand why each is a mistake. I will address them one by one in this and my next article. To start, here are the first five.

1. Having little or no understanding of the target audience

This seems straightforward: how can you write an effective survey if you don’t understand much about who will be completing it? The issue is making a connection with your respondent. The survey writer should know as much as possible about the attitudes and beliefs of the potential respondent. The wrong wording can offend respondents or just steer them away from what you intended. Often, too much focus is placed on what information the survey writer wants to get back, and not enough on what information respondents can provide.

2. Providing multiple choice lists that are too restrictive

It is always a good idea to include answer options that include “don’t know” or “uncertain,” “not applicable” and “other.” Respondents become frustrated when they don’t see their response in a multiple-choice list. The idea is to weed out respondents that don’t have a clear opinion from those that do. Otherwise you risk contaminating the good responses.

3. Requiring answers to all questions (online surveys only)

Nothing is more annoying to respondents than having offered their time to complete a survey and then having trouble progressing through the survey. A few skipped responses is not going to change your results – and ultimately you cannot force respondents to answer a question. If they want, a respondent can just close their browser and forget about your survey.

4. Asking too many open-ended questions

It is good to have comment fields, but too many open-ended questions makes it appear that the survey writer did not want to put in the effort to create easy-to-answer questions focused on survey objectives. The main purposes for open-ended questions in a survey are to provide respondents an outlet for thoughts and opinions that may otherwise distract them from thinking about the questions asked, and to add richness and understanding to the quantitative results obtained.

5. Using ranking questions incorrectly (or overusing them)

The inexperienced survey writer will often overlook the fact that ranking questions are difficult for a respondent to answer and even more difficult for them to analyze and interpret. The mistake made is to assume that the best way to ask a question of one person is the best way to ask the same question of many people. If I have only one customer, then I would want that customer to rank their priorities (one to whatever). That all changes, however, when I have to consider the priorities of many customers together. Asking respondents to select their top three priorities (or two or four, etc.) creates a natural ranking when the data is summarized.

In my next post, I’ll finish my list and address the last five most common mistakes made when writing surveys.

The interested reader might want to refer to Edward R. Tufte’s 2001 book, The Visual Display of Quantitative Information for a more detailed account on how to make sure your reports have “graphical integrity.” It is an excellent book.

Tips for Maintaining Graphical Integrity

1. Always include the number, N, of observations on your chart.

This simple addition to any chart (see example below*) is critical to understanding the data presented. Including the number of observations on your chart let’s the reader reconstruct your data from the percentages given. It also provides an indirect measure of the variability one might expect to see.

2. Use counts instead of percents when N is small, fewer than 10 to 15.

When you have a very small number of survey respondents it is better to plot counts than the percentage of counts (see following two charts). Even though it is possible for the reader to calculate the counts on his or her own, going to counts for small numbers helps the readability of a chart by eliminating the extra step. The human brain can read and interpret small numbers quickly and without ambiguity.

3. Don’t overload your chart or graph.

It is important that your reader be able to get information quickly and efficiently from your chart of graph. There is often a tendency to add too many items to your graph that do not add to the data’s message. Often “non-data” items (e.g. 3-D images) are included that can detract from or mask the true information. Even if you include only data on the chart, it is easy to overload it will information.

The following example shows a common mistake, which leads to a chart that is difficult interpret.

The most critical piece of the data from the chart above is the trend information. Therefore it is best to emphasize that element of the data. If, in addition, you combine the percentages for “Much better” and “Better” as well as for “Worse” and “Much worse,” the chart becomes much easier to read and interpret (see below).

4. Show all of your data.

The complete truth about a data set usually extends beyond what you can fit on a single chart or graph, but you need to strive to show all available data. For example, suppose a manufacturer comes out with a new product release and wants to see what effect the new release has had on service calls. The chart below shows the number of product service calls for the year before the release compared to the same number after the new release.

This seems like a reasonable approach but the two years of data do not begin to tell the whole story. Showing all of your available data better represents the true situation (see chart below). The two charts give substantially different pictures.

As I’ve talked about many times, graphical displays of data can be powerful communication tools. If you follow these few tips it should help you provide clear and concise information and avoid charts that are confusing or misleading to your readers. In my next article I’m going to return to the topic of writing surveys, by going over the top ten most common mistakes made when writing surveys.

*NOTE: The charts and graphs in this blog post were generated by exporting a CSV of SurveyGizmo survey data into Excel.

Let me change the age-old saying “a picture is worth a thousand words” to “a picture is worth a thousand numbers”.

Anytime you have survey data, you have the problem of organizing them for interpretation and/or distribution. Usually this means tabulating, summarizing or charting the survey data.

The whole process can be fraught with traps that can mislead, or, at best, confuse those looking to take information away from the data. It pays to make sure you know how to look at your survey data.

As a statistician who is often messing around with survey data, I frequently have to go back and correct an analysis because I forget to first ascertain the context for my analysis or interpretation. The best way to do this is by “looking” at the survey data ahead of time through the use of graphical displays.

Things to look for!
Some things to look for in your survey data are:

• Typos or data entry errors
• Unusual patterns or anomalies
• Does the survey data make sense in light of the context?

Let’s look at a few examples.

1. This happened to me recently. I was summarizing data from a very extensive survey of grocery store executives. There were many questions, each with many parts to be summarized using averages. The table below shows the averages resulting from the question: What proportion of your company’s total loss comes from each of the following areas?

This seemed fine until I looked at a histogram for each area. Below is the histogram for Shoplifting. It quickly and easily showed someone entered a 6 instead of .06 or 6%!

The corrected histogram is below. The corrected average was 24% instead of 44%.

2. Now let’s look at a well known data set that illustrates the same problem of just trusting statistical calculations without LOOKING at the data. A statistician named F. J. Anscombe published the paper “Graphs in Statistical Analysis” with this data in 1973. Below are the data; I’ve added a context by imagining them representing sales by quarter for three regions.

The data look harmless enough and interestingly they all have the same mean and standard deviation. Now let’s look at a few pictures.

Clearly each graph tells a completely different story, while the means and the standard deviations all tell the same story. (As a side note, for those that like to use more sophisticated statistical tools, the linear regression lines for each of the three groups are identical as well!)

3. As a final example I want to show the power of a picture for making comparisons. Consider the average customer satisfaction levels by city for two quarters. See the table below.

Natural questions that might be behind this data are; How do the cities compare on satisfaction level? Which cities improved from last quarter? What city has the highest satisfaction level? What city has the lowest satisfaction level? What city has shown the most improvement since last quarter?

See if looking at the graph in Figure 4 doesn’t make answering these questions easier than pouring through the table above.

I encourage everyone to get used to “looking” at the survey data before going too far with your analysis and interpretation. That’s what makes using SurveyGizmo’s Summary graphs so great. They’re already built in, and ready for you to use. Practice looking at graphical displays of your survey data, and it will pay big dividends.

Next time I’ll continue to look at graphical displays by providing some tips and cautions based on information from the bible on the subject, Edward Tufte’s book, “The Visual Display of Quantitative Information”.

The one question to ask when interpreting trend data
There is really only one question to ask when interpreting trend data and it is, “Has a change occurred?”

After that question is answered, there are a host of follow-on questions to ask, such as, “Why haven’t we improved?” or “Why is customer service satisfaction declining?” and so on. None of these questions can be addressed until we have determined whether or not a change has occurred.

Anytime we measure something more than once, we will see variation. Variation always exists. It is neither bad nor good.

That is why rather than asking the question, “Is there a variation?” the more pertinent question is, “Has a change occurred?” Even better, we can phrase the question slightly differently to read, “Is the variation we see in the data just normal variation or has something significant changed?”

Practice looking at trend data!
We don’t expect our measurements to always be the same even when nothing has changed, but how much do we expect them to vary?

There are statistical tools that can help to answer this question, but ultimately it takes practice looking at data; looking for and recognizing patterns, asking questions, investigating the numbers behind the data, and understanding how the data were collected. There is no substitute for experience when it comes to looking at trend data, or any data for that matter.

Let’s start by looking at the three examples in Figures 1, 2, and 3 below. Tips:

1. Ask “Am I looking at normal variation or is something significant going on?”
2. Try not to focus on individual points. Instead, look at the graph as a complete picture.
3. Look for patterns or anomalies.

There doesn’t seem to be a lot going on with this data. After three increases in a row starting in 2002 we might have been excited, but would have wanted to withhold the celebrating. The satisfaction level over the ten years looks fairly flat, hovering around 80%.

This example appears straightforward. Something changed for the better after 2007! The data from years 2001-2007 look relatively flat hovering around 58%, but the satisfaction levels after 2007 clearly increased.

This data look erratic, especially when compared to the previous examples. Prior to the third quarter the response rate may have been higher and dropped, but we only have two data points to consider so not much can be said about that. Next, the variability across quarters 3 through 8 seems large next to that for quarters 9 through 14. That should be investigated. The level of the response rate does not seem to have changed.

Optional: The Has-A-Change-Occurred (HACO) Chart
For those who want a more rigorous way to answer the question about whether a change has occurred, one can look at a HACO chart, perhaps better known as a control chart. The theory and application of HACO charts is quite extensive and complicated, but I’ve found one type of chart that is both easy to use and versatile. That chart is an Individuals Moving Range (IMR) chart.

An IMR chart, like all HACO charts, has an average line, an upper limit, and a lower limit. Calculating these values is straightforward, but it is never a good idea to make blind calculations without some understanding of the tool you are using. That is a discussion for another time.

To create an IMR chart, follow these steps:

1. Sum the absolute differences between one data point and the previous point (moving ranges of size 2).
2. Divide the sum obtained above by the number of data points minus one to create the average moving range, MR_ave.
3. Calculate the average of the points, X_ave.
4. Compute the lower limit as X_ave – 2.66 * MR_ave and the upper limit as X_ave + 2.66 * MR_ave.
5. Plot the data and add the average, upper, and lower limit lines to the chart.

Let’s see how the IMR chart looks for the three trend data examples given above. They appear in succession below.

The HACO chart tells us that nothing has changed with the data in Figure 4, that a change did occur in the data in Figure 5 and that the response rate has not changed in Figure 6. What about the issue of reduced variability that we saw in Figure 3? This is an important issue, but needs to be resolved by a different type of control chart, the Moving Range (MR) chart (which is a subject for another article).

In summary, let me remind you that it takes practice to read and interpret trend data. If you are not used to looking at graphical displays of data, it can be hard to interpret what you are seeing. Next time I’ll continue to look at the power of graphical displays in interpreting data.

Now, we’ll look at what happens when we add another dimension, time. (I’ll look at yearly data here, but it could just as easily be monthly, quarterly or any time period.) With the added dimension, things get cluttered pretty quickly. Let’s look at the Stacked Bar Chart when we add just one year.

All the information is there but it is cluttered. I.E – it is not easy to take it all in at a glance. This compromises the real value of the graphical display. Further, it won’t stay this way as time marches on! The problem quickly becomes untenable.

One thing we may decide to do is to drop the individual attributes and look at just Overall Satisfaction. One could argue that it is the most important anyway. See figure 2 below.

This is not too bad for looking at ten years, but it isn’t exactly crisp.

Enter One Number Reporting (ONR). Wouldn’t it be nice if there were a single number that we could track that tells us how we are doing? This is a frequent request of management and has been for many years.

Since the goal is to reduce the five levels of our Likert scale down to one, we won’t expect any measure to work perfectly.

Here are four ONR options:
1. Averages – The average rating after assigning numbers to the Likert scale levels; 5=Very satisfied, 4=Satisfied, 3=Neutral, 2=Dissatisfied, 1=Very unsatisfied.
2. Top two box percent – The sum of the top levels in the Likert scale (in Figure 2, the sum of the orange and blue boxes).
3. Net Promoter* Score 1 (NPS1) – The top box (blue) minus the bottom three boxes. [This was originally designed for the one question, “Would you recommend us to a colleague with similar needs?” the Net Promoter Score is the total who are “Very likely” to recommend you minus those who are neutral or unlikely to recommend you. The theory is that gives you “Promoters minus Detractors”.]
4. Modified Net Promoter* Score 2 (NPS2) – Percent satisfied (Very satisfied or Satisfied) minus (Very dissatisfied or Dissatisfied).

Below are charts of each option using the same data that appears in figure 2 above.

All four options yield similar pictures, so that’s good! As I mentioned in my last article, I’m not a big fan of the using averages because the number itself is hard to interpret, but any of the other three seem equally good.

So what does all this get us? For one, we have that “one number” for management that we can use to track progress over time with relative ease.

Secondly, we can add back the attributes we dropped from the attribute satisfaction we looked at in Figure 1. We now have a way to look at all the attributes over time using the so-called “spaghetti chart” (see below).

This chart let’s you look at trend data for several attributes at once. It can get a little cluttered but proves to be a useful data display tool. I am a big believer in looking at data over time. Looking at data over time allows you to build knowledge for improving your decision-making. Next time I’ll take a close look at analyzing trend data.

*Net Promoter is a registered trademark of Satmetrix Systems.

Please indicate your level of satisfaction with each of the following aspects of our service.

This type of question is popular for surveys because it is easy to design and easy for the respondent to complete, both good reasons to use it. In this article, I look at the best way to report the data obtained from this question type.

Alternatives
1. Table with Counts and Percentages (plus)

SurveyGizmo adds “Average %” at the bottom of the table.

2. Individual Bar Charts

This requires five charts, one for each attribute. I have shown an example of what one would look like above.

3. Stacked Bar Charts

This option essentially puts all five bar charts on the same chart. This chart was created in Microsoft Excel using the percentages SurveyGizmo gives in the original table. It is one of the chart type options Microsoft products offer and is undoubtedly a part of other charting software as well.

4. Average* Attribute Ratings Bar Chart

This chart was created by first assigning numbers to the Likert scale levels; 5=Very satisfied, 4=Satisfied, 3=Neutral, 2=Dissatisfied, 1=Very unsatisfied and then averaging all the ratings for each attribute. The five averages are then plotted using a horizontal bar chart not unlike the vertical bar chart in alternative 2.

Advantages and Disadvantages

Alternatives 1, 2 and 3 all have a key advantage over alternative 4. From the averages in number 4, one can’t recreate the raw data as you can with the other three alternatives. In addition, averages can be difficult to interpret. The percentages in the first three alternatives have very specific meanings.

If we look at just alternatives 1, 2 and 3, alternative 3, the stacked bar charts, has some distinct advantages. It is less cumbersome than the five charts required for alternative 2 and it is easier to read than the 61 numbers in the alternative 1 table.

With alternative 3 one can make quick comparisons between the different attributes. One measure I like to use when comparing attributes is the percentage of respondents that were either “Very satisfied” or “Satisfied” (the first two sections in the stacked bar chart).

So reporting matrix data can be pretty straight forward, but the problem gets more complicated when you want to look at the same data over time. This leads to the goal of “one number reporting”. By “one number reporting” I mean developing a single number that best depicts, for example, your client satisfaction. One number reporting is always desirable (for simplicity and tracking purposes) and always challenging with the tradeoffs that are required. I’ll leave that discussion for my next blog entry.