Statistical Significance in Market Research

It’s statistically significant! Wow! But is it significant?
Or even better, is it meaningful?

I’ve been doing survey design and market research for 25 years and I’m still amazed when I’m asked: “How many responses do I need to make my study ‘statistically significant’?” Hearing those words “statistically significant” in a market research context always confuses me. My usual response is “What exactly do you mean by that?” I either get a blank stare or a variety of responses, but most people are intending to say “You know, ‘statistically significant,’ the magic label that makes my results better.”

So what’s the confusion?

  • A survey can’t be “statistically significant,” nor can a certain number of responses. Only a test statistic (e.g., a calculated statistical quantity) can be statistically significant.
  • To a statistician “statistically significant” has a very particular meaning related to hypothesis testing requiring a specific set of assumptions which are rarely if ever true in a market research setting.
  • Researchers aren’t asking about the significance of the results in the context of their objectives. This should be their main focus.

Let’s take a closer look:

For example, suppose someone comes to me for help with a survey to test the market’s preferences for skin care products. Often the first question they ask me is how many responses are needed to get “statistically significant” results. That is where the confusion starts, because that question only makes sense in the context of a statistical hypothesis test. A survey may involve many hypotheses that we want to test.

A statistical hypothesis test requires both a hypothesis: women buy more skin care products than men, and a test statistic: the percent of women who buy skin care products minus the percent of men who buy skin care products. Now we can ask if the test statistic (the difference between the two percentages) is “statistically significant.” That’s a legitimate question. A more meaningful question might be whether the difference is “practically significant.”

Practical significance: A calculated difference is practically significant if the actual difference it is estimating will affect a decision to be made. (Should we focus more marketing dollars on women than we do on men?)

Statistical significance depends on the sample size. A difference of 3% (58% for women minus 55% for men) can be statistically significant if the sample size is big enough, but it may not be practically significant. 3% hardly seems big enough to warrant focusing on one market over the other.

A difference of 30% (65% for women minus 35% for men) may be practically significant (i.e., warrant a decision to focus more dollars on the women’s market) but if the difference isn’t statistically significant (that depends on sample size) then you can’t be sure the difference you see (30%) is real, so you either need to get more data or treat the two groups as the same.

There are additional statistical issues with statistical significance that I haven’t addressed here, such as all the assumptions needed to perform the testing correctly. The assumptions of normally distributed data and 100% response rates are rarely, if ever, met in a market research setting. I’ll save those issues for another article.

So, how “statistically significant” is your survey data and is that the right question?

More: Know How
Join the Conversation
  • Brittany Heidtke

    Hi Ed!
    Thanks for writing this. We get this question a ton over at SurveyGizmo and it’s always great to get some insight.

  • Ed Halteman

    You are welcome Brittany.

    I received a comment early that is gone. I’ll repeat it here:

    From Rags Srinivasan:
    “I wonder if they mean level of confidence and margin of error when they say stat significance. A decision maker may not know the technical nuances, and that is probably why they want to hire an expert do the survey for them. I think an appropriate reply to them is not what do they mean by stat significance but what are they trying to do differently with the data and walk them thru the required confidence level or statistical significance.”

    My reply was:
    Thanks for your comment.

    Yes, most of the time they are asking about level of confidence and margin of error, but my point is those concepts only make sense in the context of an individual survey question and in particular a test statistic and a hypothesis. There is a misconception out there that a survey can be “statistically significant” and that there is a certain sample size that will make things “statistically significant”.

  • Ed Halteman

    I received another comment that didn’t appear here so I want to share it.

    Confidence interval and margin of error do correspond to specific metric (e.g., customer satisfaction or likelihood to recommend) as you point out but level of significance applies for a hypothesis test.

    Thanks Rags, you make a good distinction. Perhaps I can cover in a future article the topic of confidence intervals, margin of error and level of significance in a market research setting.

  • Judy Fishman

    This is very funny Scott – not in the comical sense but in the sense of it being pretty much a transcript of conversations I have regularly. Thank you for putting it all together!

  • Pingback: Shoulda Coulda Paid Attention in Stats Class « Fail Forward