Deciding on sample size is one of the steps we have to take at Relevant Insights in the early planning stages of a survey. In order to determine a survey sample size, we need to consider the following:
Analytical plan: The planned analysis for the results should be considered first. There are multivariate analysis techniques (e.g. regression analysis) that require a certain number of observations per variable. Furthermore, if there is an interest in finding statistically significant differences between subgroups in the sample, the sample size should be adjusted for it.
Population variability: This refers to variability in the target population. If there is a lot of variability in the issue of interest, a large sample might be needed. A behavior occurring 20% or 80% of the time has less variability than if it were to occur 50% of the time.
Level of confidence: This is the level of risk we are willing to tolerate, usually expressed as a percentage (e.g. 95% confidence level or interval). Although survey results are reported as point estimates (e.g. 65% of respondents like this product), the fact is that since we are working with a sample of the target population, we can only be confident that the true value of the estimate in that population falls within a particular range or what is called confidence interval. This percentage indicates the probability that the true value falls in fact within the confidence interval boundaries. The confidence level is inversely proportional to estimate accuracy, so the more confident you want to be, the larger the interval that will contain the true value of the estimate is needed, which leads to lower levels of precision
Margin of error: Also called sampling error, it indicates the level of precision of the estimates that is desirable. This is what is often quoted in the media when poll results are presented (e.g. +/-3%). This percentage defines the lower and upper bounds of the confidence interval likely to contain the parameter estimate, and it is an indicator of the estimate’s reliability. The larger the sample, the smaller the margin of error and the greater the estimate precision.
Below is a table illustrating how the margin of error and level of confidence interact with sample size. To get the same level of precision (+/-3.2%), larger samples are needed as the confidence level increases. For example, if we want to be certain that in 95 out of 100 times the survey is repeated the estimate will be +/- 3.2%, we need a sample of 950.
For more sample size and margin of error estimates, use Relevant Insights’ Sample Size and Margin of Error Calculators.
- Cost: It is common in research to have to make a trade-off between statistical accuracy and research costs. Larger samples mean higher cost. Factors such as low incidence and low response rates can also increase sample cost. Not long ago, I received a call from a client who wanted to conduct an online survey with a sample of 1,000 respondents, which would give a statistical accuracy of +/-3.1% at the 95% confidence level, but would cost them $8,000. At the same time, sample of 400 respondents would give a statistical accuracy of +/-4.9% and cost $3,400. In this case, a 135% increase in sample cost would only yield a 60% gain in statistical accuracy. The client decided to conduct the study on a smaller sample.
- Population size: Most of the time the size of the total target population is unknown, and it is assumed to be large (>100,000), but in studies where the sample is a large fraction of the population of interest, some adjustments are needed.
SAMPLE SIZE ESTIMATION CHECK LIST
So, when calculating sample size, be prepared to answer the following questions:
- What type of data analysis will be conducted? Will subgroups be compared?
- What is the probability of the event occurring? If no previous data exists, use 50% for a conservative estimate.
- How much error is tolerable (confidence interval)? How much precision do we need?
- How confident do we need to be that the true population value falls within the confidence interval?
- What is the research budget? Can we afford the desired sample?
- What is the population size? Large? Small/Finite? When the population size is unknown, it is assumed to be large.
As you may have figured out, there is no magic solution to determining the right sample size for a survey, but there is a lot of judgment. I hope at least I gave you some guidance on things to consider. To calculate sample size and margins of error, use the Sample Size and Margin of Error Calculators from Relevant Insights.