Alternatives to Ranking Question Types in Surveys

In my last article about survey question wording, I mentioned I would talk about the different methods in a survey for obtaining a ranking of a list of items. Obtaining such a ranking has wide application from marketing (when trying to decide what product features are most important), to customer satisfaction (when trying to decide what improvements should be worked on first) to employee satisfaction (when trying to decide what is most important part of job satisfaction).

In “Case Study 3: Reviewing your Question Wording”, the marketer wanted a rank ordering of the criteria herbal tea buyers use to make their purchase decision.

Before we dive into that, let’s first explore briefly what “ranking” means in the context of a survey. Clearly, if you are my only customer and I ask you to rank five possible criteria in order of their importance then I’m done. I have all the information I need to evaluate each criterion. However, suppose I have two customers. Is it still best to ask each to rank order the five criteria? What if I have three customers or more?

How do I use this data to “rank” my criteria? When I have one customer or one critical party then ranking gives an unambiguous ordering. We see that changes when you are looking for a ranking from a group of survey respondents. The result of rank ordering doesn’t translate directly from one to many. In the above example we could make a case for ranking criterion D number one and Criterion C number two, but beyond that things gets pretty murky.

Now let’s go back to our ranking question from last month’s blog. We want to rank the criteria herbal tea buyers use when buying tea. We want to know the best ordering of the following buying criteria for herbal tea drinkers:

  • Clinically tested
  • Environmental practices
  • Price
  • Flavor
  • Certified organic
  • Effectiveness
  • Fair trade certified

We assume here that knowing this will help guide our engineering and marketing efforts.

We have already talked about one way to approach this problem; we can ask each respondent to rank order the seven criteria.

Now let’s look at two other alternatives.

The first alternative is to ask respondents to select their top two or three (a good rule of thumb is to have them limit their selection to about one-third of the total number of items) criteria out of the list. This requires the respondent to do a similar evaluation as he or she would in rank ordering the criteria, however the evaluation is not as extensive. The results yield a natural ranking by looking at the criteria according to how many “votes” each item received.

The second alternative is to give respondents 100 points to distribute among the seven choices as they please. This option, too, requires a similar evaluation by the respondent however it provides the respondent additional freedom by allowing him or her the chance to divide up the points among all the items. The only requirement is that the sum of points assigned adds to 100.

Here are the three choices:

  1. Rank order the criteria from 1 to 7, based on the importance they have on your buying decision
  2. Select the three most important criteria for your buying decisions (Select NO MORE THAN 3)
  3. Distribute 100 points among the seven criteria based on the importance each has on your buying decision.

Now let’s compare the three options.

Comparison from the respondent’s viewpoint:

#2 is clearly the easiest to answer for the respondent. The respondent merely has to divide the seven criteria into two groups, important or not important.

#3 is the next most friendly to the respondent, because the respondent has the flexibility to decide the level of effort they want to put into the selections. If a respondent is so inclined, he or she can select his or her top three choices like alternative #2 and then just divide the 100 points among them. However, with this alternative the respondent has the flexibility to assign different point totals to the one or ones he thinks are important.

#1 requires the most effort from the respondent since each criterion must be given a separate rank. This can be difficult (try it, if you haven’t tried it lately) when two things seem equally important or when the bottom of the list has three or four items that are equally unimportant but still must be ordered (i.e. given a rank).

Analysis and Interpretation Comparison

Suppose we have 100 responses to the ranking question. For alternative #1 this yields 100 sets of 7 numbers (1 to 7). For #2 we have up to 300 “votes” and for #3 we have 100 divisions of 100 points into seven groups. #2 and #3 are easily summarized and interpreted. One would just total the “votes” or “points” for each criterion to create a natural ranking of the seven criteria.

#1 is different. The summary is not so straightforward. I could average the rankings for each criterion but I have to decide if I can live with a scheme where 100 4′s is the same as 50 1′s and 50 7′s. I could total the number of times each criterion got a first place rank, but then I have to be happy with a scheme that gives a higher ranking to a criterion with 10 first place ranks and no others in the top 3, than to one that got 9 first place ranks and 91 second place ranks. Analysis and interpretation for option #1 takes some thought and rarely has an easy answer.

So which method should we use? I like the simplicity and ease of analysis and interpretation of the select-your-top-one-third method (#2). It provides a natural ranking and is unlikely to overwhelm or frustrate the respondent. It is important to keep the respondent in a good frame of mind for answering the rest of the survey.

Clearly the distribute-100-points method (#3) provides more precision. If I feel like I need more precision in distinguishing between items of a list I might use this method, but in most cases it is not really needed. Remember, it is always a good idea to consider how you are going to use the information you get back when deciding on a question to ask. In this case, one might want to consider what kind of precision is warranted. Maybe we should rank order the alternatives!

In my next blog article, I will present another case study that looks at an expanded process for designing a market research survey.

Join the Conversation
  • Amjad Ata

    Thanks for the topic, very helpful and I’m going to cite it in my research :)

  • Amjad Ata

    Thanks for the topic, very helpful and I’m going to cite it in my research :)

  • luis saldana

    I know this is an old post but just read it… What about when the list of features is longer… say 20 or 30?

    • sgizmo

      Hi Luis, thanks for your comment. We would highly recommend providing no more than 10 options for your respondent to rank, otherwise, you may cause survey fatigue and may get a poor response rate.

      -SurveyGizmo Team