The Checkbox Question As Checkdown Pass

The checkdown pass is the last option of the quarterback, after he’s scanned the field and seen that his wide receivers and tight ends are covered. Similarly, there are times that the checkbox question should be a last resort.

For example, we recently used SurveyGizmo to survey over 1,000 U.S. adults about attitudes towards the NFL, the Super Bowl, and their sports superstitions. The study clearly illustrated some of the potential pitfalls of the oft-used checkbox question.

Here’s what we learned about the NFL’s audience and how to get the most out of your checkboxes.

Understanding Checkbox Questions

The checkbox is typically the “Select all that apply” question. Unlike radio buttons and dropdowns, respondents can select every choice that is appropriate.

And that’s great when you know the list.

If you want to ask, “Which NFL teams do you root for?” then you know the list of all 32 possible options.

But survey question trouble often crops up when you think you know the list.

You might hope that if you add an, “Other (please specify)” choice, you’ve covered yourself. But respondents are busy people taking their cues from you.

If you leave a choice out, they might very well just select another choice that is, “close enough.”

Checkbox Questions and Broncos Country

One of the questions we wanted to ask in our recent survey was, “Why are you rooting for the Broncos? (Select all that apply.)”

I started writing the question and came up with this initial list:

  • Always root for my team’s conference
  • Always root against my team’s conference since the conference champ was better than my team
  • Like Peyton Manning
  • Like the players
  • Long-time fan
  • My home team
  • Dislike Carolina Panthers’ players
  • Dislike Carolina Panthers more
  • Other (please specify)
  • None of the above

But as soon as I finished the list I realized that I didn’t really know why people would root for the Broncos.

(And that’s not just because I’m originally from Cleveland and still haven’t forgiven them for winning 2 AFC Championship games in a row in the 1980s!)

I decided to hide the question from within SurveyGizmo. And instead I created another question, “Why are you rooting for the Broncos?” and made it an Essay/Long Answer Question instead.

I did this with each of my questions where I realized I did not know enough to come up with a list of choices that was comprehensive but not tedious: for instance, when asking why people were as satisfied or unsatisfied as they were with the NFL.

When the questions that we were confident in were all programmed, we tested the survey and shared it. After we collected 100 responses, we paused it.





guide to choosing survey questions




Benefits of Testing Checkbox Options First

After our test period we read the answers to our essay questions.

In some cases the written responses revealed enough about our audience to let us create a checkbox list with confidence. Other times we found the patterns weren’t clear enough.

We got enough responses to our essay question about the NFL that we were able to hide it and replace it with a Radio Button Grid.

But we needed more responses to our question about the Broncos, as we were hiding or showing the question based on whether respondents were rooting for the Broncos (showing), the Panthers (hiding), or neither (hiding).

So we opened the survey back up until we had somewhat more than 300 responses altogether. At that point, we had about 100 responses per skip pattern.

In most cases, you only need to gather 100 or 200 answers to an open-ended verbatim question to confidently create a checkbox list. It’s too much to read and process otherwise, and you get diminishing returns for each added response.

But 100 answers per question were great. After reading all the verbatim replies, we ended up writing a new list of choices:

  • Like Peyton Manning
  • Want Peyton to win 1 more Super Bowl before he retires
  • Like the team and its players
  • Live in Colorado
  • Am a Denver Broncos fan
  • Like Broncos more than Panthers
  • Hate the Carolina Panthers
  • To support friends who are fans
  • Other (please specify)
  • None of the above

Here it is as a marked up version of our original:

  • Always root for my team’s conference
  • Always root against my team’s conference since the conference champ was better than my team
  • Like Peyton Manning
  • Want Peyton to win 1 more Super Bowl before he retires
  • Like the team and its players
  • Long-time fan
  • My home team Live in Colorado
  • Hate the Carolina Panthers Dislike Carolina Panthers’ players
  • Like Broncos more than Panthers Dislike Carolina Panthers more
  • To support friends who are fans
  • Other (please specify)
  • None of the above

Had we not used the open-ended question first, we would have missed how important this choice was: “Want Peyton to win 1 more Super Bowl before he retires.”

It turned out to be the second biggest reason to root for the Broncos and was selected by the majority of respondents (51%), coming just behind, “Like Peyton Manning” (58%).

Checkbox Question Results About the NFL
Want to see some of the other answers? Check out The Most Superstitious NFL Fan Bases before you decide what to wear, where to sit, or what to eat Sunday!

And remember, the checkbox is a great checkdown. But get a read on your other options first.


About the Author:
Jeffrey Henning is president of Researchscape International, a Friend of SurveyGizmo. You can follow him on Twitter @jhenning. He holds sports grudges against teams going back decades. So he can’t root for the Broncos.

Join the Conversation