Survey Sample Size – What Should It Be?

Deciding on sample size is one of the steps we have to take at Relevant Insights in the early planning stages of a survey. In order to determine a survey sample size, we need to consider the following:

Analytical plan: The planned analysis for the results should be considered first. There are multivariate analysis techniques (e.g. regression analysis) that require a certain number of observations per variable. Furthermore, if there is an interest in finding statistically significant differences between subgroups in the sample, the sample size should be adjusted for it.

Population variability: This refers to variability in the target population. If there is a lot of variability in the issue of interest, a large sample might be needed. A behavior occurring 20% or 80% of the time has less variability than if it were to occur 50% of the time.

Level of confidence: This is the level of risk we are willing to tolerate, usually expressed as a percentage (e.g. 95% confidence level or interval). Although survey results are reported as point estimates (e.g. 65% of respondents like this product), the fact is that since we are working with a sample of the target population, we can only be confident that the true value of the estimate in that population falls within a particular range or what is called confidence interval. This percentage indicates the probability that the true value falls in fact within the confidence interval boundaries. The confidence level is inversely proportional to estimate accuracy, so the more confident you want to be, the larger the interval that will contain the true value of the estimate is needed, which leads to lower levels of precision

Margin of error: Also called sampling error, it indicates the level of precision of the estimates that is desirable. This is what is often quoted in the media when poll results are presented (e.g. +/-3%). This percentage defines the lower and upper bounds of the confidence interval likely to contain the parameter estimate, and it is an indicator of the estimate’s reliability. The larger the sample, the smaller the margin of error and the greater the estimate precision.

Below is a table illustrating how the margin of error and level of confidence interact with sample size. To get the same level of precision (+/-3.2%), larger samples are needed as the confidence level increases. For example, if we want to be certain that in 95 out of 100 times the survey is repeated the estimate will be +/- 3.2%, we need a sample of 950.

For more sample size and margin of error estimates, use Relevant Insights’ Sample Size and Margin of Error Calculators.

  • Cost: It is common in research to have to make a trade-off between statistical accuracy and research costs. Larger samples mean higher cost. Factors such as low incidence and low response rates can also increase sample cost. Not long ago, I received a call from a client who wanted to conduct an online survey with a sample of 1,000 respondents, which would give a statistical accuracy of +/-3.1% at the 95% confidence level, but would cost them $8,000. At the same time, sample of 400 respondents would give a statistical accuracy of +/-4.9% and cost $3,400. In this case, a 135% increase in sample cost would only yield a 60% gain in statistical accuracy. The client decided to conduct the study on a smaller sample.
  • Population size: Most of the time the size of the total target population is unknown, and it is assumed to be large (>100,000), but in studies where the sample is a large fraction of the population of interest, some adjustments are needed.

SAMPLE SIZE ESTIMATION CHECK LIST

So, when calculating sample size, be prepared to answer the following questions:

  • What type of data analysis will be conducted? Will subgroups be compared?
  • What is the probability of the event occurring? If no previous data exists, use 50% for a conservative estimate.
  • How much error is tolerable (confidence interval)? How much precision do we need?
  • How confident do we need to be that the true population value falls within the confidence interval?
  • What is the research budget? Can we afford the desired sample?
  • What is the population size? Large? Small/Finite? When the population size is unknown, it is assumed to be large.

As you may have figured out, there is no magic solution to determining the right sample size for a survey, but there is a lot of judgment. I hope at least I gave you some guidance on things to consider. To calculate sample size and margins of error, use the Sample Size and Margin of Error Calculators from Relevant Insights.

Join the Conversation
  • Pingback: Representative Samples – Does Sample Size Really Matter? | SurveyGizmo

  • jessief29812

    Thanks :) this really helped me out on my research project im conducting… I also found that using a sample size calculator could be an easier way :) http://www.surveysystem.com/sscalc.htm

  • http://www.surveygizmo.com/ bri

    @jessief29812:disqus: Thanks for your post! I’m glad this topic was helpful to you!

  • http://surveygizmov4.helpgizmo.com/help bri

    @jessief29812:disqus: Thanks for your post! I’m glad this topic was helpful to you!

  • Muhammad jabbar

    I am working on a comparative study of universities according to different attributes of two different areas, say A and B.

    A having 10 universities.

    B having 4 universities.

    i have selected 5 universities from A and 2 universities from B, we may consider that total students of all 7 selected universities are greater than 1,00,000.

    I would like to compare different attributes according to natural sciences group, bio sciences group, social sciences group.

    Now my question is that how much sample size i require over all?

    how much from each university?

    and how much from each area of study ( natural sciences group, bio sciences group, social sciences group) ??

    waiting for your kind response.

  • Muhammad jabbar

    I am working on a comparative study of universities according to different attributes of two different areas, say A and B.

    A having 10 universities.

    B having 4 universities.

    i have selected 5 universities from A and 2 universities from B, we may consider that total students of all 7 selected universities are greater than 1,00,000.

    I would like to compare different attributes according to natural sciences group, bio sciences group, social sciences group.

    Now my question is that how much sample size i require over all?

    how much from each university?

    and how much from each area of study ( natural sciences group, bio sciences group, social sciences group) ??

    waiting for your kind response.

  • sgizmo

    I wish I could just post a number and that could be your goal, but unfortunately, it is not that easy. For each sub-category, you will need to determine how large the estimated population is in order to determine the statistically sound number of responses you need, for say the natural sciences group. Feel free to give us a call and we would be happy to work with you on this problem.

  • sgizmo

    I wish I could just post a number and that could be your goal, but unfortunately, it is not that easy. For each sub-category, you will need to determine how large the estimated population is in order to determine the statistically sound number of responses you need, for say the natural sciences group. Feel free to give us a call and we would be happy to work with you on this problem.

';