I recently heard from a friend who wanted some help with a survey. He was working with a group of volunteers whose main goal was to influence the hospitality industry to be “greener.” As volunteers, they had no budget. They wanted help formulating a questionnaire that would give them information they could use to help “build tools” and write “green papers” for the hospitality industry.
I like to help when I can for a worthy cause so I answered his questions as best I could. I think it makes a good case study looking at the early stages of survey design.
My friend’s questions were typical of those I hear often when someone doesn’t have a lot of experience with surveys. This is a great starting point for understanding how the survey process works. As such, I tried to do more than just answer his questions; in addition I offered details on the best way to approach his survey design process.
Our exchange started with the following email.
“I appreciate your willingness to help us formulate a questionnaire that will help us gather valuable information so we can build tools and write “green papers” for the industry.
My first few questions, and I hope I communicate them clearly, are:
- Is there a theory on how many questions are too many?
- Is it better to have an even or odd number of “answers” to a question?
- How brief do we want to keep the wording for the answer options?
If I am not asking a question I need to be, would you point that out to me, and then answer it? :~)”
I got him started with the following answers and other thoughts on his design.
Is there a theory on how many questions are too many?
If you are asking me if there is a dominant theory that dictates the correct number of questions to have on a survey in order to get “superior” results, the answer is, in a word, “no.” As with most disciplines, there are lots of theories on what “should” be done, as in this case, what the “correct number” of questions is, but there is no simple answer.
I’ll answer your question by telling what question you should be asking. The best question you can ask at the beginning of your survey is, “What information do I need and from whom do I need it in order to help me make my decisions?”
NOTE: If you aren’t making any decisions you don’t need to do a survey!
The follow-up question to the “what and who” is; “How do I go about creating a survey that engages my target audience, respects their time and provides me with the information I need?”
Over the years my experience has shown me that if a survey is well thought out, respects the respondent’s time and is focused on clear objectives then the limiting factor is not the number of questions but the surveyor’s ability to act on all of the information he or she will receive from the survey.
So, while your initial question appeared straightforward, my response aims to shift your focus to the following:
- Plan on putting time up front in the design
- Work to understand your target audience and how they will view your survey; this will help you view it from their perspective and let you write a survey in their language
- Have lots of people review your survey and give you input about length, clarity, language and understanding
- Pilot the survey with a small friendly group from the target audience
- Look at the data that comes back from a pilot and simulate using it to make your decisions. Ask whether any information is missing or if something isn’t needed.
Is it better to have an even or odd number of “answers” to a question?
I assume you are talking about rating scale questions. I prefer having a middle position because I think it is more realistic and thus makes it easier for the respondent to answer.
How brief do we want to keep the wording for the answer options?
There is no rule here. The driving factor is that the answer options are clear, mutually exclusive, unambiguous and descriptive. I know that this is a lot to consider; yet keeping all of these in mind as you design the survey will give data that is useful.
If I’m not asking a question I need to be, would you point that out to me, and then answer it?
I tried to do that above. Here are some others questions that you should work to answer yourself.
- What is it, exactly, that I want to get from the survey data?
- What process will I use to make my decisions?
- What can and can’t I do with the survey information?
Remember, before you decide to put a question on your survey ask yourself how you will use the information and whether you will have the resources you need to act on the information. It is important that you be able to act on all the information you receive otherwise you are wasting your respondent’s time and yours by gathering it!
This is the second in a series of case studies focusing on Survey Design. Next time I will present a case study that focuses on question wording.