In my last post (Increasing Response Rates: Part I), I talked about increasing your survey response rates. I suggested that you:

1. Tell your respondents you will share the results with them.
2. Identify allies who will urge potential respondents to participate.
3. Use reminder notices.
4. Offer charitable gifts for each response.
5. Offer thank-you gifts as incentives.

Which incentives are most effective is a question I get a lot – and I think I admitted in that last post that, in 30 years, I still haven’t figured that out. If there’s a science to it, it’s a science that I’ve never learned.

Is this something we can help each other figure out? I’ve had some interesting things happen regarding incentives that I’ll tell you about below. When you’ve read through them, maybe you can add some stories of your own. I’d like to know what incentives you’ve tried with what kinds of audiences and how they worked. Maybe we can figure this science out together!

• Back in the old paper-and-pencil days – after receiving a survey with a dollar bill attached to it — I attached two crisp one-dollar bills to 150 surveys being sent to people in state departments of education. I sent out 150 surveys – and got 3 back. In retrospect, I think the $2 was presumptive and insulting. The survey required more than $2 worth of the respondents’ time. A brief cover letter describing the purpose of the study and offering to share the results would have been more effective.




• A client of mine once offered $25 gas cards to all those who responded to a particular lead-generation survey. Shortly after we fielded the survey, the average price for a gallon of gas went over $3 and we had a lot of responses. I don’t remember the response rate, but I do remember my client was pretty excited.
  
  Of course, $25 is a steep incentive – but the leads were worth the cost. There was a lesson in this though – when we started going through the file, we found a few duplicates – people who thought they could game the system for two or three gas cards. So, if your incentive is really exciting – Super Bowl tickets anyone? – make sure you’ve got the quality control in place to protect yourself from respondents you don’t want and from false data.




• Once, a client offered a little desk puzzle that had the client’s company logo on the side. When I held the puzzle in my hand, I wanted one. It was a clever puzzle and looked nice on my desk – but it didn’t generate many responses.
  
  In fact, some respondents added comments saying don’t send me that puzzle with your logo on it. (Although mine still looks good on my desk and is a conundrum for anyone entering my office!) Any incentive that seems self-serving in the least, however, may not be effective.




• Here’s the experience that mystifies me the most…
  
  This story is from another lead-generation survey. I had a client who said to me, “I can use either of two incentives – a $10-coffee-shop-card or a USB drive. Which would you recommend?” I said that it was hard to say for sure but that, given that all of his respondents were in the technology industry, I suspected they had all of the USB drives they wanted and that the $10-coffee-card would be more effective.
  
  He wasn’t so sure, so we copied his survey (making two of them), split the list of e-mail invites in half randomly and sent the survey link out offering the $10-coffee-card to half the invitees and the USB drive to the other half. Responses from the USB group outnumbered respondents from the coffee group about 6-to-1!
  
  One week later, we sent our reminder out to both of the e-mail lists offering only the USB drive. The response rate from what had originally been the coffee group increased significantly – and this was all from people who had to have USB drives lying around like paper clips.

I think the message in these stories is that in order to work, a tangible incentive does not have to have a lot of monetary worth. But it has to show that you know and respect your respondents and that you are at least trying to say “thanks for your time.”

Please add your own stories about your experiences with incentives in the comments below!

I’ll let the staff at SurveyGizmo select the best story. The Winner receives 4-ounces of Greenwell Estate Private Reserve 100% Kona coffee from my private stash. (Let’s see how this incentive works.)

So, good luck with your response rates. And remember:

1. SHARE THE RESULTS
2. ENLIST THE SUPPORT OF ALLIES (with whom you will SHARE THE RESULTS)
3. TAKE TIME TO SEND REMINDERS (but no more than your purpose requires)
4. USE CHARITABLE GIFTS AS INCENTIVES
5. USE INCENTIVES – wisely, respectfully and in ways that demonstrate you know your respondents

Many of us continually fight the battle of low-response rates.

My first professional experience with surveying occurred more than 30 years ago. I simply wanted to get the opinions of particular professionals at eleven different colleges in my state and I developed a two or three page survey for them to respond to and send back to me. I thought it was going to be easy because:

• I was asking pretty good questions that I thought even the respondents would want the answers to.
• I assumed professional courtesy would compel people to respond.

My boss scoffed and said, “You might get two back.”

Well, I put my survey together with a cover letter promising to share the aggregate results with all respondents. Then I followed-up with a reminder postcard about a week later.

Low-and-behold – I received 10 out of 11. Being naïve about my success, I actually called the 11th respondent asking him to complete the survey and respond (I probably even told him that everyone was waiting on him!). I remember he was rather curmudgeonly and really wanted nothing to do with my survey, but I persisted and he relented.

I had a 100% response rate!

It was several years before my next professional survey experience. I was Director of Research at an education technology company.

We wanted any one-of-three people in each of the fifty state departments of education to respond to a survey about curricula in economics. I had just responded myself to a survey which I had received in the mail with a crisp one-dollar bill, so I thought, “I’ll make sure to get a good response by including TWO crisp one-dollar bills.” I went to the bank, purchased 300 crisp one-dollar bills and included two with each survey.

I think I got 3 surveys back – 3 out of 150.

Motivating people to respond to a survey is a curious thing. There is some well-accepted common knowledge about incentives to respond to surveys. Here are five tips:

1. Share the results.
2. Identify other people who care about the results.
3. Send a reminder.
4. Charitable giving.
5. Tangible incentives do work.

SHARE THE RESULTS

Offer your respondent the opportunity to see the survey results. The simplest way to do this is to generate a SurveyGizmo summary report in PDF format and post it on your website or (an even easier option) is at the Pro level and above, you can embed a report into your website. Sometimes, an organization may even add value to their survey by sending out a PowerPoint summary.

I have had clients tell me they don’t want to share the results of their surveys because they believe the information they are gathering gives them a competitive advantage. The silliness of thinking that competitive advantage lies in survey results aside; the half-life of survey information is too brief to convey much market power. There is more to be gained by sharing the results and being seen as the organization to generate that information, than believing you’ll have a competitive advantage by holding it back.

I have a client who sponsors a market information survey annually. Five years ago they were reluctant to share the information. Today, they get higher response rates because people in their marketplace anticipate the survey and want to make sure their company is included in the published results.

IDENTIFY OTHERS WHO CARE ABOUT THE SURVEY RESULTS

You are not the only person who wants to know the answers to the survey you are building. Your pastor, professional colleagues, the school board, the president of the local chapter of your professional organization, or even a magazine/newsletter editor might care about your results.

All of the people who care about your survey results can help prepare your respondents by announcing your survey at meetings, in newsletters or in e-mails. You don’t have to be alone in generating survey responses.

SEND A REMINDER

We don’t send post card reminders very much (at all?) anymore, but reminders work just as well via e-mail as they do via snail mail.

Use SurveyGizmo’s email system to send reminders out to all those who have not responded.

You will probably receive a third to a half as many responses to the first reminder as you did to the initial invitation. Subsequent reminders will generate fewer and fewer responses so, after that first reminder, consider your purpose before sending more reminders.

If you’re doing a lead-generation survey where every response can increase your revenue stream, subsequent reminders may be a good idea. If you’re doing research where you need a particular sample size to represent central tendencies, reminders after your first reminder will probably not substantially change your results. It would not be worth the time of leaving your survey in the field, but that first reminder is almost always worth the effort.

CHARITABLE GIVING

Charitable gifts can be particularly effective if your target audience is affluent business executives or other professionals.

Charitable gifts can also be effective if you’re trying to gather responses from a group that you know has a singular mission in mind.

For example, I once pilot-tested a marketing survey for a transportation components company. In order to make sure we got enough responses to the pilot test, we offered $10 to a local nonprofit bicycle shop that trained neighborhood kids in bicycle mechanics for every person who responded to the pilot test and helped us make survey changes.

TANGIBLE INCENTIVES

I have seen tangible incentives positively affect survey response rates and I have seen them crash and burn. After years in the business, I still haven’t figured out the science of incentives.

Here are three approaches you can take:

• Offer a “thank-you” gift – a coffee card, a USB drive, desk knick-knack – to those who respond.
• Offer “points” for each response. Most online panels do this. A respondent gets 250 points for each survey they respond to. As they respond to more and more surveys, they accumulate more and more points, which they can redeem for other gifts.
• Offer a lottery. Take the total cash you have budgeted for “thank-you” gifts and offer it as a cash prize to one randomly selected winner.

My next article will have more to say about tangible thank-you gifts as incentives since, although they may not be the most effective way to increase response rates, they are easy to administer and used very often – so they do deserve a longer discussion.

In review, there are several things you can do to increase your response rates. Components of your strategy might include:

• Offering to share the survey’s results.
• Engaging allies who are also invested in your survey content.
• Use reminder notices.
• Offer charitable gifts.
• Offer tangible incentives.

But analyzing comments can intimidate us. We think that to analyze comments we have to read all of them. And there is no software that will tell us, “Your average comment was . . . “

If we read just 10 comments though, we might summarize them pretty easily. We could report that 3 were about product improvements, 5 simply said the product was “fine,” and 2 were requests for added features. Then we could report the 3 problems and the 2 additional features requested.

One of my clients is a nonprofit umbrella organization. They have a program providing services to homeless youth. The director of that program bemoaned once that she had over 300 surveys from youth who were living on the streets and she had no idea how to summarize their comments and, no less, do it quickly. She had a board meeting in three days and had to report on what the youth had to say.

I asked my client to give me the spreadsheet with the survey responses and I would provide a useful summary for her at least one full day before her board meeting.

You too can do it – reliably and quickly. Here’s how:

1. Randomly select 50 comments as a representative sample for all of the comments. (To do this, we’re going to invoke Excel’s very easy to use RAND function.)
2. Insure the representativeness of those 50 randomly selected comments.
3. Develop categories from the first 10 comments.
4. Apply those categories to all 50 comments.

WHEN NOT TO USE THIS PROCESS ALONE

This process is for giving an audience a trustworthy and useful description of comments overall. If your survey data includes key respondents or your audience has particularly important decisions to make, you may choose to analyze and describe some comments separately.

Create your spreadsheet

If you’ve gathered your survey responses with SurveyGizmo, you can download your data into a spreadsheet and, after you’ve cleaned it up, a portion of it might look like this:

Of course, our spreadsheet had many more columns than this but for analyzing the comments responding to the question, “What is your biggest concern living on the streets?” this is what we need to get started. Note that we have saved the comments as well as enough identifying information to see what age, gender and experience (measured in months homelessness) are represented by each comment.

We’re going to use the random number function to guarantee that we’re getting a completely random selection of comments. (This example assumes the use of an Excel spreadsheet.) To do that, we add a column and enter =RAND( ) in each row of the spreadsheet. Use of this random number function gives each comment a unique identifier and allows us to sort them randomly.

STOP – before you go any further, preserve your random numbers! (If you do not do this, your numbers will update when you manipulate a cell in the RNUM column, the number in the cell will change and you will lose your randomness.) To preserve the random values:

1. Highlight the random number (RNUM) column.
2. Type CTRL-C.
3. Type:
   ALT-E
   S
   V

Then click “OK” at the bottom of the “Paste Special” box.

NOW – sort the file using RNUM as the only sort key.

Then, take the first 50 rows as your randomly selected comments.

Check for representativeness

In the case at hand, we knew approximately how long, how many youth stayed homeless, how old they were, and what their distribution was by gender.

In your file of comments, look at those identifiers you selected and saved in the columns to the left of the comments. Based on those identifiers, do your first 50 comments represent all of those who took your survey? Most likely they do. As long as I select at least 50 comments, the randomization process has always given me a representative sample. It would probably work with 30 comments, but the difference in effort between summarizing 50 comments versus summarizing 30 comments is pretty small. After 30, you’re “on a roll” and the increased reliability you get by boosting your sample from 30 to 50 consumes very little time.

If your first 50 randomly selected comments, however, are not representative of your population as a whole on any of your selected identifiers, just reinsert your =RAND( ) function and resort the file. You’ll get another 50 and you can check them again.

Categorize your comments

Read the first 10 comments only and enter a word or phrase that summarizes what the comment is about. Don’t dwell on this step too long or you’ll drive yourself nuts. Think about your audience and topical areas they need to know about. Try to use those topical areas as categories in this step.

You are completely normal (at least insofar as this activity goes) if you have trouble deciding on the categories or themes that will most effectively capture the essence of the first ten comments in your file. That’s exactly why you’re stopping after 10 comments! Do not look at comment-11 or any of the comments after that.

Maybe your spreadsheet now looks like this:

Now, use the categories that worked for those first 10 comments and apply them to comments 11 through 50. Some comments are going to be easy to categorize and some are going to be difficult. Follow these rules:

Once you’re done with all 50 comments, go back to that “something else” column and put asterisks next to each comment that is about that “something else.” You may have 2 or 3 “something else” columns.

Now your spreadsheet might look like this:

Quality control

OK, so far so good. You went quickly and most of your comments are categorized – plus you’ve also identified those comments within each category that are about money (or about product improvements or that compliment a person in your organization or . . . ). The trouble is, you went so quickly that you have too little confidence in what you did and you’d be very nervous about reporting your summary to anyone who mattered! Not to worry . . .

Let’s do some quality control. This step is easy and will make you feel great – increased confidence and all that. Sort the file by Category. Read through the comments within each category. Do they hold together as a group? Re-label them if they don’t. Are all of the concepts captured that you want to capture? You can always add other “something else” columns to make sure you’ve got everything.

It is common during quality control to tweak labels – maybe to find a new category or to merge two categories together. You will also find a category for the comments that went unlabeled your first time through the comments. You should have about ten percent unlabeled comments. That would be normal if you have categories that fit your data.

Summary

All-in-all, analyzing survey comments is like anything else. There is a learning curve and you will develop your own tricks after you’ve done it once or twice, and then you will be able to enjoy the paradox of being able to do it more quickly AND with higher quality.

One last tip – this is an extraordinarily appropriate application for Excel data filtering. Excel “data filtering” will allow us to sort our file by categories or it will allow us to isolate our attention on only those comments within a given category. If you’re not familiar with Excel data filtering, highlight the top row of your spreadsheet, go to “Data” and then “Automatic filter.” Now let the computer be your teacher. Play with the selection options in each column and you’ll see what I mean!

So, we have answered the question, “How in the world do we make sense of all these comments?” We did it by:

1. Creating a spreadsheet with comments and adding identifying information for each comment.
2. Randomizing the comments and using the first 50.
3. Creating categories from the first 10 comments.
4. Labeling the next 40 quickly and not driving ourselves nuts over getting everything perfect.
5. Sorting and doing some quality control, re-labeling as needed.
6. Finishing everything within 90 minutes when we had planned 5 hours, we left work early and stopped for a beer on the way home.

Many respondents want the opportunity to add comments to their surveys and we can quickly analyze them and make them useful to our audiences.

I hope this was helpful. Although maybe an even more useful question to have answered would have been, “How in the world did they get three hundred surveys from homeless teenagers in the first place?”Well, that would be a story around a different campfire.

That’s easy, isn’t it? Three months ago, your survey began with a set of questions. Now you’ve got your data. Look up the answers!! Oh – and don’t forget the PowerPoint slides. Make them pretty.

But there must be more to it than that. After all, you received 419 responses to 26 survey items. That’s 10,894 responses to survey items. Surely we can get more than 18 PowerPoint slides and an internal memo out of them!

How well you explore your data is key to effectively delivering value to your audience. To do this, eyeball your data.

What does this mean? Well, it doesn’t mean export to Excel and then stare at the computer screen (although I’ve tried that).

We’re going to look at using SurveyGizmo reports in combination with a couple of uses of Excel to discover more of what our data are telling us. Within SurveyGizmo we will use summary reporting with bar charts; we will use crosstabs; and we will use the data export function. Then, in Excel, we’ll look at using some very simple functions, a sorting method and a couple of charts.

Your first step in eyeballing (rather than analyzing) your data is to run a SurveyGizmo summary report. I suggest a bar-chart format. Bar charts make differences and patterns easier to find. Scroll through it. You’re not looking for answers to questions. You’re looking for charts that don’t look like the other charts – for bars that are longer than others or for charts with bars of similar length. Identify those charts that represent survey items of particular interest.

Print the report. Put *’s next to items of particular interest to your audience. Look for charts that don’t make sense or, if you have a baseline, that are different from the baseline. Are there other charts that might explain the charts that surprise you? For example, if you’re surveying customer satisfaction and the overall value ratings are down, are the ratings lower on product quality or responsiveness also? If they are, don’t assume that one causes the other. Do a SurveyGizmo crosstab with “value” in your rows and “product quality” and “responsiveness” in your columns? Do the percentages show a pattern? If not, that may be a finding also and you might want to report that there is no apparent relationship between “value” and either “product quality” or “responsiveness.”

Any hypotheses confirmed? Any surprises? Make note.

Now let’s use Excel to explore further. Do a SurveyGizmo export to an Excel spreadsheet. Each row is a respondent and each column is a survey item.

Before building new charts, we’ll use some of the simplest functions and sort our data to see what we can find.

Go to the blank row at the bottom of the data and use the =AVERAGE function to enter each item’s mean score. Go to the next blank row and use the =MEDIAN function. Now, sort the file HORIZONTALLY on the means. [Note: Many people are unaware that data can be sorted horizontally. To find this option, click on DATA-SORT-OPTIONS-SORT LEFT TO RIGHT.] Now, your means are sorted from left-to-right. Most of your medians probably are also. But which medians are out of order? But take another look at that row of medians. Are some out of sequence? Wherever a median is out of sequence, you have found a distribution that is unusual compared to the dataset as a whole.

In and of itself, an unusual distribution may not mean anything – but once you’ve found one, look more closely at it to see if it offers anything you want to report or if it prompts questions you had not thought of before. Can you explain to yourself why respondents would be different for this item?

Now let’s make two charts.

The first chart is one you can use to compare several items to each other in one picture and is useful with scaled items (for example, “on a scale to 1-to-5″). Highlight the appropriate data and open the chart wizard. Click BAR and select the “100% stacked bar” as the chart sub-type.

[Note: If you're chart does not look like this, you may have your series set up in rows when you want columns (or vice versa). No need to rearrange your data. In the chart wizard there is a Rows/Columns button that will change the orientation of the chart for you. Also, this chart is not in the default colors but has been enhanced for your viewing pleasure.]

One of the nice features of this chart is that you can compare items with different response rates to each other. The chart will standardize the distributions to make them comparable.

What can we see in this chart? Well, product quality is the attribute that satisfies the most customers. One-in-ten being very dissatisfied seems like a lot for an attribute that customers are generally so satisfied with. You might want to isolate those ten percent and see if they all have the same product or share some demographic. Also, if you’re combining “Very satisfied” and “satisfied” as one category, you are missing the huge difference in customer responses between “customer service” and “value.”

You may have quantitative items in your data that are not scaled. Average ratings from a teacher evaluation survey, for instance, may show that average ratings in one department range from 15 to 85 while in another department they range from 40 to 70. To explore the ratings across departments at your university, create a SurveyGizmo form asking each department to enter the average overall rating for each professor in the department. Your SurveyGizmo Excel download will look like:

Now go to the first blank row, as we did above, and enter the averages, then to the next blank row and capture the maximum rating (=MAXIMUM). In the next blank row capture the lowest ratings (=MINIMUM) and finally, in the next row enter the median functions again.

Excel offers chart types that are designed for showing stock prices – but they have other uses as well. Highlight the mean, maximum, minimum and median ratings. [Note: They must be in this order.] Open the chart wizard again and click STOCK. Select “Open-High-Low-Close” as your chart type.

From the average teacher ratings alone, you may have thought that the English and Math departments were similar in teacher evaluations but, as you can see from the chart, they are really quite different. Also, the height of the boxes shows how far your mean and median are from each other. Over half of the teachers are above the median and half are below in each department. The height of that box in biology indicates that, while over half of the teachers are above the top line of the box, those that are low are so low they are dragging the average score way down. You have found some true outliers.

As you go through these steps, expect to find many, many more questions than answers. That’s the whole point of data exploration – to be able to help your audience know all of the questions which their data can begin to answer. These steps are a way to explore data; not THE way to explore data. Starting here may lead you to idiosyncratic strategies that work for you and your audience. The steps we have reviewed here are:

• Run a SurveyGizmo report with bar charts. Look for items of interest to your audience or that are unique.
• Run SurveyGizmo crosstabs whenever you think items may be related to each other.
• Export your data to Excel.
• Use =AVERAGE and =MEDIAN functions to find items with unusual distributions.
• Chart your data. Compared scaled items to each other with 100%-stacked-bars. Compare non-scaled distributions using variations of Excel stock charting tools.

In doing this, you’ve learned a lot about your data and never had to sit through a class in statistics. No statistical analysis and you did it all with your eyeball. Good work!

What’s that you say? Your survey also had 4 open-ended questions and you don’t know what to do about that? That’s an article for December . . . or maybe January.

We can increase our understanding of polls by answering two questions:

1. How can polls be accurate with such seemingly small samples?
2. How do we align popular polls with reality? (The general form of this question — How do we align our survey research with the real world? is important in all survey research.)

Political polls rely on two statistical principles for their trustworthiness — randomness and weighting.

Think back to high school. Picture yourself in your cafeteria at lunch time. Your task is to pick one person at a time and ask the question, “Who do you want to win the election for student body president?”

How many students do you think you will have to poll before your overall results no longer change very much? Eight people? 15? 25? 50?

As a rule of thumb, your results are going to be reasonably stable after you’ve queried 30 people, provided your selections have been truly random.

How do we achieve random results? It’s difficult and it is critical to the reliability of small sample sizes. Obviously, in the cafeteria example above, you’re not going to query six of your friends all seated at the same table. But what else will affect the randomness of your sample? Here are some questions to consider in the current example:

1. Is there only one cafeteria in your school and is every student equally likely to use it?
2. How many lunch periods are there during a day? Is there any bias regarding which students go to lunch at which hours of the day? For instance, do freshmen all go to Lunch I; sophomores to Lunch II; etc.?
3. What affects seating patterns? Is there a salad bar? A hot lunch queue? Do students who bring their lunches to school eat in the cafeteria or outside on the campus?
4. Can polling at lunch time in the cafeteria account for students who take post-secondary options in the afternoon? Or do they leave the building before lunch and head to their other classes?
5. In selecting students to poll, did you inadvertently avoid people you’re uncomfortable with? (Maybe you should have had someone from another school select respondents and conduct the poll.)

Let’s say you’ve satisfied yourself that polling in the cafeteria is going to work. You go ahead and conduct your poll and you find that, after you’ve polled 30 students you have talked with 9 freshmen, 8 sophomores, 5 juniors and 8 seniors. Here are your data:

Overall, it looks close — maybe a tie!

One of the downsides of randomness is it is not guaranteed to generate representative samples. You know that, in your school, 30 percent of the students are freshmen, 27 percent are sophomores, 23 percent are juniors and 20 percent are seniors.

So your random sample does not represent your electorate on a key variable — class. Therefore, we weight.

Here is the distribution of your students by class:

And here is how we adjust our polling data to represent what we think will happen on Election Day. (There are several methods for weighting our poll results but they all arrive at the same result.)

So, what looked like a pretty close race, may actually be an 8-point victory for Henry. Of course, how well your poll reflects reality will depend partly on the validity of your weighting assumption, that is, that the class you are in affects for whom you will vote.

So this is where polling starts. Of course, there’s a lot more to it, such as:

• Confidence intervals
• Timing of the poll
• Whether or not those that we poll will actually vote
• How to represent students who may vote but never go in the cafeteria
• Controlling for students who lied to us about their choice.

Also, we know that with weighting we have made our sample more representative of the school overall, but how do we know that we’ve represented each class reliably? All of these things affect poll results also. Randomness and weighting, however, are what give us a firm methodological beginning.

Randomness and weighting are part of how polls generate reasonable results with what appear to be small sample sizes. But there’s another important question to pose to align polling with reality. That is, how do we align popular polls with the Electoral College? For that, see http://www.electoral-vote.com. It is an excellent site. It aligns popular polls with the realities of the Electoral College and you can learn how they do it by reading the FAQs.

Yes, it matters.

Use an odd-numbered scale and, if you expect your results to be fully distributed from the positive to the negative end of the continuum, use a 5-point scale. If you expect your results to be either positively or negatively skewed, use a 7-point scale.

Odd or even-numbered scale

There are two purposes of a survey item. 1) Learn what people think. 2) Distribute respondents along a continuum in order to compare them individually or in groups to each other. (As in, “Will 25 year-old buyers have as much interest in my product as 20-year old buyers?”)

With an even number of alternatives, the survey developer is telling people, “You have to agree or disagree with this item. You can’t be neutral or undecided”. Maybe psychologists sometimes need to present respondents with forced-choices, but most of us are not psychologists. Why some survey developers think they should control respondents with even-numbered scales is something I have never understood and as a respondent, resent.

Moreover, even numbered scales never, never, never gather more information than odd-numbered scales. In fact, they impoverish our data sets. Ah, you say, but I want to know how those people who are neutral would respond if they were forced to choose. Why? What difference will it make? Wouldn’t you rather know how many of those people who are near the middle of the scale really can’t tell the difference between the choices, or haven’t made up their minds? Besides, I can tell you within a couple of percentage points how those neutrals will respond if the middle choice is taken away from them. They will respond the same as the other respondents in the survey. Try it some time. Ask the same question in two different parts of your survey once with an odd-numbered scale and once with an even-numbered scale and you can prove this point to yourself.

Scale length

The human mind can embrace 7-plus-or-minus-2 data bits at any given time. Those of you old enough to remember life before cell phones and the ubiquitous use of area codes remember that phone numbers could be easy to remember or peculiarly difficult.

Obviously those numbers used more often were easier to remember, but what else made some numbers easy to remember and others difficult? Patterns! Patterns reduced the number of data bits your mind had to retain. The number 222-4466 is much easier to remember than 497-5031. In the first number, the pattern is apparent. In the second number you have to retain all seven numbers as separate bits.

The application to surveys of the 7-plus-or-minus-2 rule is this. More than seven levels of “agreement” cannot be considered in one instant which makes scales with more than 7 levels difficult to respond to. They are fatiguing. They deprive the mind of the chance to embrace the scale as one and accurately make a selection from an array of balanced alternatives. What the mind does with a ten-point scale is first, split it into a positive and a negative half — an especially frustrating decision if the respondent is in the middle.

In sum, using scales that provide a full array of alternatives within an appropriate length permits respondents to complete your survey in a shorter amount of time and with greater accuracy.

• 4-point scales — an impoverished and inaccurate data set
• 5-point scales — accurate; MEETS YOUR NEEDS 90% OF THE TIME
• 7-point scales — accurate; necessary for skewed data
• 10-point scales — You deserve all of the “abandons” you get