How to Use Logic To Eliminate Online Survey Bias
After building and testing your survey, and finally getting buy-in from your stakeholders, you may want to use your survey software’s logic and validation features to get the best quality data and reduce the length of the survey for your survey takers.
Survey logic is a term used to describe any feature that alters the flow and order of your survey. Logic features have two basic goals:
- To eliminate survey bias by protecting against unqualified and duplicate survey takers.
- To fight fatigue by routing survey takers to the smallest set of questions possible.
Today, we are going to talk about using logic in order to eliminate bias.
If a respondent answers your survey more than once, it can lead to skewed data. Duplicate protection keeps a respondent from returning to a survey and answering it more than once. This is also called vote protection.
The most common is cookie-based vote protection.
It places a small tag or “cookie” in the respondent’s browser, which flags them as already having taken the survey. When the respondent returns to the survey it will redirect them to a thank you message.
The problem with cookie-based vote protection is that the cookie is placed on the survey taker’s computer and is specific to the web browser they used to respond to the survey. This means if they use a different web browser or delete the cookie, they will be allowed to take the survey again.
Vote protection based on IP address does not require storage on the survey taker’s computer and cannot be easily avoided. However, many corporate environments share a single IP address, which can result in only one person in the organization being able to take the survey.
When fielding a survey you’ll have a list of characteristics that represents the population you are surveying, also known as your target audience. The results of your survey and the quality of the data you collect will rely on the extent to which your survey respondents match these predefined characteristics.
How certain are you that the people taking your survey are in your target audience?
You can use disqualification questions and logic to filter out unwanted survey takers. This technique requires you to ask several qualifying questions at the beginning of your survey designed to confirm that the survey respondent matches your target audience.
You then setup disqualification rules that trigger when the respondent does not match your desired qualifications. Most often, your survey software will have a place where you can create a customized disqualify message to the disqualified survey taker telling them they are not part of your target audience and thanking them for their time.
Randomization allows you to randomly reorder pages, questions and options in your survey.
Randomization helps eliminate order bias. That is, the bias that people have when they read items in a certain order.
Order has a large influence on survey-taker behavior, particularly with a larger list of options.
Another benefit of page randomization is to distribute the potential effect of survey abandonment across the entire survey. A certain amount of abandonment will happen in any survey. By using page randomization you can help ensure that you get enough responses to questions “deeper” in the natural order of your survey.
Use these relevant forms of logic to reduce or eliminate bias in your surveys.