Building Better Surveys – Effective Scales

A question I have been asked from time-to-time is, “How many choices should I give respondents when I’m asking them how much they agree or disagree with something?” As in . . .

Which is correct? Does it matter?

Yes, it matters.

Use an odd-numbered scale and, if you expect your results to be fully distributed from the positive to the negative end of the continuum, use a 5-point scale. If you expect your results to be either positively or negatively skewed, use a 7-point scale.

Odd or even-numbered scale

There are two purposes of a survey item. 1) Learn what people think. 2) Distribute respondents along a continuum in order to compare them individually or in groups to each other. (As in, “Will 25 year-old buyers have as much interest in my product as 20-year old buyers?”)

With an even number of alternatives, the survey developer is telling people, “You have to agree or disagree with this item. You can’t be neutral or undecided”. Maybe psychologists sometimes need to present respondents with forced-choices, but most of us are not psychologists. Why some survey developers think they should control respondents with even-numbered scales is something I have never understood and as a respondent, resent.

Moreover, even numbered scales never, never, never gather more information than odd-numbered scales. In fact, they impoverish our data sets. Ah, you say, but I want to know how those people who are neutral would respond if they were forced to choose. Why? What difference will it make? Wouldn’t you rather know how many of those people who are near the middle of the scale really can’t tell the difference between the choices, or haven’t made up their minds? Besides, I can tell you within a couple of percentage points how those neutrals will respond if the middle choice is taken away from them. They will respond the same as the other respondents in the survey. Try it some time. Ask the same question in two different parts of your survey once with an odd-numbered scale and once with an even-numbered scale and you can prove this point to yourself.

Scale length

The human mind can embrace 7-plus-or-minus-2 data bits at any given time. Those of you old enough to remember life before cell phones and the ubiquitous use of area codes remember that phone numbers could be easy to remember or peculiarly difficult.

Obviously those numbers used more often were easier to remember, but what else made some numbers easy to remember and others difficult? Patterns! Patterns reduced the number of data bits your mind had to retain. The number 222-4466 is much easier to remember than 497-5031. In the first number, the pattern is apparent. In the second number you have to retain all seven numbers as separate bits.

The application to surveys of the 7-plus-or-minus-2 rule is this. More than seven levels of “agreement” cannot be considered in one instant which makes scales with more than 7 levels difficult to respond to. They are fatiguing. They deprive the mind of the chance to embrace the scale as one and accurately make a selection from an array of balanced alternatives. What the mind does with a ten-point scale is first, split it into a positive and a negative half — an especially frustrating decision if the respondent is in the middle.

In sum, using scales that provide a full array of alternatives within an appropriate length permits respondents to complete your survey in a shorter amount of time and with greater accuracy.

  • 4-point scales — an impoverished and inaccurate data set
  • 5-point scales — accurate; MEETS YOUR NEEDS 90% OF THE TIME
  • 7-point scales — accurate; necessary for skewed data
  • 10-point scales — You deserve all of the “abandons” you get
Join the Conversation
  • http://twitter.com/FAPhoenix Tyler Totman

    What is your opinion of the “Net Promotor Score” method with 10 point scales?  For example:
    “How likely are you to refer our product/service to a friend?”
    1 – 10Then they use the following:    [ Sum (1-5) ] [ discard 6,7]  [Sum (8,9,10) ]  [Sum (8,9,10) ] minus [ Sum (1-5) ]  The result is your promoters less your attractors, for a solid measurement of how people think/promote your brand?

    I see how it could be modified for a 5 point scale (or 7), but not sure what that would do to effectiveness of the simple answer (especially if it is being used in a balance scorecard etc.) 

    • sgizmo

      Although Bill recommends the 7-point scale, Fred Reicheld, the author of The Ultimate Question, recommends the 11-point scale rather than a 10 point scale. The 11-point scale gives you the most variability for selecting the likelihood to recommend or not. The importance of the 0 is so a respondent that would not recommend a product or service at all has a valid answer to select. A 1 may indicate that they see some, although limited value in the product or service and may lead the respondent to not answer the question, thus tainting the overall data.

  • SS

    hi!  i want to ask government officials  how important “issue x” is for them.  Do you suggest using a 10 point scale (ie “how important is x issue” and they would pick a number b/n 1 and 10  or can i use a 7 point likert.  I guess more importantly, when can i use a likert scale and when i use a regular scale? 
    Thanks so much for your time.

  • Robin

    I don’t agree with you 100%. I think the value of practice is really important, but there is research out there that has demonstrated (given, in their limited settings) that a mid-point can bias results (see for example Garland (1991) The Mid-Point on a Rating Scale: Is it Desirable? – there are other more recent studies that I don’t have on hand). As a survey researcher myself, there have been contexts where it is absolutely necessary to force a choice, and others where, as you suggest, it makes no sense to do so. It is highly context dependent. I don’t see why you come so strong on one end of the spectrum. If you have empirical research (other than gut feeling or past experience in an uncontrolled environment) I think it is your responsibility as a researcher to cite that when giving technical advice to others.

    • sgizmo

      You are absolutely correct: there are reasons why a survey might benefit from including a neutral mid-point or not. It really depends on what the researcher is trying to accomplish. Thanks for pointing out that methodology can (and should) shift depending on a survey’s intent.

';