There are many methodological issues to consider when creating a questionnaire, if you want to gather high-quality data in a survey. Among these issues are:
- Data collection mode: While a respondent’s words taken during phone surveys or in-person interviews take more importance given the conversational format, online surveys’ visual design elements have a bigger impact on how questions are read and interpreted. When designing a survey online, it is important to be aware of the question types that are a good fit.
- Respondent effort: There are questions that put a heavier burden on the respondent’s working memory and comprehension or are likely to elicit higher non-response if asked in different data collection modes.
- Question wording: Formulating questions with the right wording so it accurately reflects the issue of interest is one of the hardest parts in writing questionnaires. Data errors can sift through a survey if you use unfamiliar, complex, or technically inaccurate words, ask more than one question at a time, use incomplete sentences, use abstract or vague concepts, make the questions too wordy, ask questions without a clear task or ask questions that lead respondents to a particular answer.
- Question sequence: Questions should follow a logical flow. Order inconsistencies can confuse respondents and bias the results.
- Question format: Questions can be closed-ended or open-ended. Closed-ended questions provide answer choices, while open-ended questions ask respondents to answer in their own words. Each type of question serves different research objectives and has its own limitations. The key issues here are related to the level of detail and information richness we need, our previous knowledge about the topic, and whether to influence respondents’ answers.
- Information accuracy: Some questions yield more accurate information than others. Respondents can answer questions about their gender and age pretty accurately, but when it comes to attitudes and opinions in a particular issue, many may not have a clear answer. Overall, attitudes and opinion questions should be worded in a way that best reflects how respondents think and talk about a particular issue.
- Measured behaviors: People tend to have less precise memories of mundane behaviors they engage in on a regular basis, and usually they do not categorize events by periods of times (e.g. week, month, and year). We need to consider appropriate reference periods for the type of behavior we want to measure. Measured behavior should be relevant to the respondent and capture his or her potential state of mind.
- Question structure: Questions have different parts that must work in harmony to capture high-quality data. These are the question stem (e.g. what is your age?), additional instructions (e.g. select one answer) and response options, if any (e.g. Under 18, 19 to 24, 25 +). The wrong combination can leave respondents baffled about how to answer a question.
- Visual layout: Using design elements in an inconsistent way can increase the burden put on the respondent in trying to understand the meaning of what is asked. Different font sizes, colors, and strengths across questions, forces the respondent to relearn their meaning every time they are used. Also presenting scales with different directions (positive to negative or vice versa) in rating questions within the same survey increases measurement error as respondents often assume all rating questions have the same scale direction even when the instructions explain the meaning of the end points of the scale.
- Analytical plan: Based on the research object, both the type of information requested and the question format are important for the type of analysis we plan to perform once the data is collected. There is also the question of whether you want to replicate the results, track certain events or just run a one-time ad-hoc analysis. If the goal is to track certain metrics, time and care should be dedicated to craft tracking questions, as slight changes in wording can change the meaning of a question and thus its results.
If you take each of these aspects of survey writing into consideration, you will be on your way to creating surveys that produce valid data and support in order to make tactical and strategic business decisions with confidence.
To learn more about Relevant Insights’ consumer data service don’t hesitate to contact me, Michaela Mora.