Statistical Significance in Market Research

I’ve been doing survey design and market research for 25 years and I’m still amazed when I’m asked, “How many responses do I need to make my study ‘statistically significant’?”

Hearing those words “statistically significant” in a market research context always confuses me. My usual response is, “What exactly do you mean by that?”

I either get a blank stare or a variety of responses, but most people are intending to say “You know, ‘statistically significant,’ the magic label that makes my results better.”

Why The Confusion Over Statistical Significance?

This term gets thrown around out of context quite often, and it also gets applied to to things that cannot themselves be statistically significant (like a survey). Here are the most common sources of confusion:

  • A survey can’t be “statistically significant,” nor can a certain number of responses. Only a test statistic (e.g., a calculated statistical quantity) can be statistically significant.
  • To a statistician, “statistically significant” has a very particular meaning related to hypothesis testing requiring a specific set of assumptions, which are rarely if ever true in a market research setting.
  • Researchers’ main focus should be finding out about the significance of the survey results in the context of their objectives.

A Closer Look at Statistical Significance

As an example, suppose someone comes to me for help with a survey to test the market’s preferences for skin care products. Often the first question they ask me is how many responses are needed to get “statistically significant” results.

That is where the confusion starts, because that question only makes sense in the context of a statistical hypothesis test. A survey may involve many hypotheses that we want to test.

A statistical hypothesis test requires both a hypothesis (e.g. women buy more skin care products than men), and a test statistic (e.g. the percent of women who buy skin care products minus the percent of men who buy skin care products).

Now we can ask if the test statistic (the difference between the two percentages) is “statistically significant.” That’s a legitimate question.

But a more meaningful question might be whether the difference is “practically significant.”

Practical Significance Defined

A calculated difference is practically significant if the actual difference it is estimating will affect a decision to be made.

If the reason for running our survey is to find out if we should we focus more marketing dollars on women than we do on men, then the difference will determine if the results are practically significant.

Statistical significance, on the other hand, depends on the sample size.

A difference of 3% (58% for women minus 55% for men) can be statistically significant if the sample size is big enough, but it may not be practically significant. Three percent hardly seems big enough to warrant focusing on one market over the other.

A difference of 30% (65% for women minus 35% for men) may be practically significant (i.e., warrant a decision to focus more dollars on the women’s market), but if the difference isn’t statistically significant (that depends on sample size) then you can’t be sure the difference you see (30%) is real.

Therefore you either need to get more data or treat the two groups as the same.

Complete Survey Reports Guide

Find out if your results and significant and actionable with our detailed guide to survey reporting.

Get Ebook

Final Considerations For Determining Statistical Significance

There are additional statistical issues with statistical significance that I haven’t addressed here, such as all the assumptions needed to perform the testing correctly.

The assumptions that data is distributed totally evenly and that we’ll get 100% response rates are rarely, if ever, met in a market research setting. Both of these impact the validity of your sample and responses, but I’ll save those for another time.

So, how “statistically significant” is your survey data, and is that even the right question to ask?

Join the Conversation
  • Hi Ed!
    Thanks for writing this. We get this question a ton over at SurveyGizmo and it’s always great to get some insight.
    Cheers,
    Brittany

  • You are welcome Brittany.

    I received a comment early that is gone. I’ll repeat it here:

    From Rags Srinivasan:
    “I wonder if they mean level of confidence and margin of error when they say stat significance. A decision maker may not know the technical nuances, and that is probably why they want to hire an expert do the survey for them. I think an appropriate reply to them is not what do they mean by stat significance but what are they trying to do differently with the data and walk them thru the required confidence level or statistical significance.”

    My reply was:
    Thanks for your comment.

    Yes, most of the time they are asking about level of confidence and margin of error, but my point is those concepts only make sense in the context of an individual survey question and in particular a test statistic and a hypothesis. There is a misconception out there that a survey can be “statistically significant” and that there is a certain sample size that will make things “statistically significant”.

  • I received another comment that didn’t appear here so I want to share it.

    “Ed
    Confidence interval and margin of error do correspond to specific metric (e.g., customer satisfaction or likelihood to recommend) as you point out but level of significance applies for a hypothesis test.
    -rags”

    Thanks Rags, you make a good distinction. Perhaps I can cover in a future article the topic of confidence intervals, margin of error and level of significance in a market research setting.

  • This is very funny Scott – not in the comical sense but in the sense of it being pretty much a transcript of conversations I have regularly. Thank you for putting it all together!

  • Pingback: Shoulda Coulda Paid Attention in Stats Class « Fail Forward()