Your survey sample size, or the number of people from your target population that you distribute your survey to, has a big impact on both your costs and your data.
If you pay to distribute your survey to every single college graduate in a particular class, your costs will be astronomical. But if you just give it to the ones in your hometown that you happen to meet on a Saturday afternoon, you won’t be able to draw any accurate conclusions from it.
6 Things That Affect Your Sample Size
Market research expert Michaela Mora of Relevant Insights gave us these six crucial things to consider when choosing how far and wide to disseminate your survey, along with a handy checklist to use with your next project.
1. Analytical Plan
The planned analysis for the results should be considered first. In other
words, consider how you want to use the data that you’re going to gather.
There are multivariate analysis techniques, such as regression analysis, that require a certain number of observations per variable in order to be accurate. If you’re planning to use these types of analysis, you’ve got to make sure you’re getting the requisite number of responses for each variable you plan to analyze.
Further more, if there is an interest in finding statistically significant differences between subgroups in the sample, the sample size needs to be adjusted accordingly.
This means if you want to determine whether male or female college graduates feel more prepared for careers, you’ve got to be sure and have enough responses from both groups that you can compare them accurately.
Guide to Survey Sample Size
Download our guide to your survey sample size, complete with a handy chart and checklist, so you get the right sample size every time.
Get the Guide
2. Population Variability
This refers to the level of variation among your target population. If there is a lot of variability in the issue of interest, a large sample might be needed.
For instance, if between 20% and 80% of your college graduate audience reports that they plan to take a vacation right after graduation, that’s a high level of variability. It means you need a larger sample size.
But if around 50% of your audience exhibits a particular behavior, then you don’t need as large of a sample to ensure statistical relevance.
3. Level of Confidence
How much risk you’re willing to tolerate in your survey results is referred to as the level of confidence. It’s typically expressed as a percentage, e.g. 95% confidence level or confidence interval.
And although survey results are reported as point estimates (65% of college graduates like breakfast cereal), the fact is that since we are working with only a sample of the population we can only be confident that 65% of the people who took the survey feel this way.
How confident we are that this percentage correlates to the responses that would be given by the entire target population is our level of confidence.
This percentage indicates the probability that the true value falls within the confidence interval boundaries.
The confidence level is inversely proportional to estimate accuracy. This means that the more confident you want to be in your data, the larger the interval that will contain the true value of the estimate, which leads to lower levels of precision.
4. Margin of Error
Also known as sampling error, this indicates the level of precision of the estimates that you want in your data.
When you see poll results presented in the media as +/- 3%, this refers to their margin of error.
This percentage defines the lower and upper bounds of the confidence interval likely to contain the parameter estimate, and it is an indicator of the estimate’s reliability.
The larger the sample, the smaller the margin of error and the greater the estimate precision.
It’s common in research to have to make a trade-off between statistical accuracy and research costs. Larger samples mean higher cost.
Factors such as low incidence and low response rates can also increase sample costs.
For example, let’s say you wanted to conduct an online survey with a sample of 1,000 respondents, which would give a statistical accuracy of +/- 3.1% at the 95% confidence level, and your budget was $8,000.
At the same time, a sample of 400 respondents would give a statistical accuracy of +/- 4.9% and cost $3,400.
In this case, a 135% increase in sample cost would only yield a 60% gain in statistical accuracy. A smaller sample size would be the way to go.
6. Population Size
Most of the time the size of the total target population is unknown, and it’s assumed to be very large (i.e. greater than 100,000). But in studies where the sample is a large fraction of the population of interest, some adjustments will need to be made to the sample size to ensure it’s representative of the target population’s size and proportions.
Sample Size Estimation Checklist
When calculating sample size, ask yourself the following questions:
- What type of data analysis will I be doing with this data? Will I want to compare subgroups?
- What’s the probability of the event I’m investigating occurring in this population? If no previous data exists, use 50% for a conservative estimate.
- How much error can I tolerate (your confidence interval)? How much precision do I need?
- How confident do I need to be that the true population value falls within my confidence interval?
- What’s my budget? Can I afford the sample I really want/need?
- What’s the population size? Large? Small? Finite? When the population size is unknown, assume it’s very large.
As you may have deduced, there’s no magic solution to determining the right sample size for a survey, but there are a lot of factors to consider carefully.
Hopefully these factors will give you a place to start, and you can also use this calculator from Relevant Insights to help you determine your ideal sample size and margins of error.
Note: This post was originally published in May, 2010. It has been updated and expanded and republished.