Statistical significance means that a result observed in your data is unlikely to have occurred by random chance alone. In practice, a result is called statistically significant when its p-value is less than or equal to a pre-set threshold called the significance level (alpha), most commonly 0.05. It is the standard way researchers decide whether the patterns they see are likely to reflect a genuine effect rather than noise.
This guide explains what statistical significance is, how p-values and the alpha level work together, what “significant” does and (importantly) does not mean, how significance differs from effect size and practical importance, and how to interpret a test correctly with a fully worked example. Let’s begin.
What Is Statistical Significance?
Statistical significance is a determination that the relationship or difference observed in a sample is unlikely to have arisen from random sampling variation, assuming the null hypothesis is true.
To unpack that, you need two ideas: the null hypothesis and the significance level.
The null hypothesis (written H0) is the default position that there is no real effect, no difference, or no relationship between the variables being studied. The alternative hypothesis (H1 or Ha) is the claim that there is an effect or relationship.
A statistical test does not prove either hypothesis. Instead, it assumes the null hypothesis is true and asks: if there really were no effect, how likely would it be to see data at least as extreme as what I actually observed? That probability is the p-value.
The significance level, denoted by the Greek letter alpha (α), is the threshold you set in advance for how much risk of a false alarm you are willing to accept. It is the probability of wrongly rejecting a true null hypothesis (a Type I error). By convention, α is usually set at 0.05 (5%), although stricter fields use 0.01 (1%) or even smaller.
The confidence level is simply 1 − α. With α = 0.05, the confidence level is 0.95, or 95%.
“The smaller the p-value, the greater the statistical incompatibility of the data with the null hypothesis… A p-value does not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” — American Statistical Association, Statement on p-Values (2016)
How to Test for Statistical Significance
In inferential statistics, you assess data through hypothesis testing (also called null hypothesis significance testing). It is a structured procedure for deciding whether an observed relationship between variables is statistically significant. The process follows five steps:
- State the hypotheses. Define the null hypothesis (H0: no effect) and the alternative hypothesis (H1: there is an effect).
- Set the significance level (α). Choose your threshold before collecting or analysing data — typically 0.05.
- Collect data and run the appropriate test. Use a statistical test suited to your data, such as a t-test, chi-square test or ANOVA.
- Obtain the test statistic and p-value. Every test produces these two numbers.
- Make a decision. Compare the p-value with α and either reject or fail to reject H0.
The significance decision
Two outputs come from every statistical test:
- The p-value — the probability of obtaining a result at least as extreme as the one observed, if the null hypothesis were true.
- The test statistic — a standardised value (such as t or z) that measures how far your sample result sits from what the null hypothesis predicts.
What Influences Whether a Result Is Significant
Whether a real effect shows up as statistically significant depends on three things working together:
| Factor | What it is | Effect on significance |
|---|---|---|
| Sample size (n) | How many observations you collect | Larger samples make it easier to detect even tiny effects as significant |
| Effect size | How large the true difference or relationship is | Larger effects are easier to detect with a smaller sample |
| Significance level (α) | Your chosen threshold for “unlikely” | A stricter α (e.g. 0.01) makes significance harder to reach |
This trio also relates to statistical power — the probability that a test will detect an effect that genuinely exists. Underpowered studies (often due to small samples) frequently miss real effects and produce non-significant results.
Worked Example: A One-Sample T-Test
Step 1 — Hypotheses. H0: μ = 70 (no difference). H1: μ ≠ 70 (two-tailed).
Step 2 — Significance level. α = 0.05.
Step 3 — Test statistic. The standard error is s ÷ √n = 10 ÷ √25 = 10 ÷ 5 = 2. So t = (x̄ − μ) ÷ SE = (74 − 70) ÷ 2 = 2.00, with df = n − 1 = 24.
Step 4 — Compare. For a two-tailed test at α = 0.05 with 24 degrees of freedom, the critical t-value is approximately ±2.064. The corresponding p-value for t = 2.00 is about 0.057.
Step 5 — Decision. Because |t| = 2.00 is just below the critical value 2.064 (equivalently, p ≈ 0.057 > 0.05), we fail to reject H0. The result is not statistically significant at the 5% level — it falls just short of the threshold.
Notice how close this is. Had the sample been slightly larger, the same 4-point gap could easily have crossed the threshold. This is exactly why a p-value should never be read as a simple yes/no verdict on whether an effect is real or important.
What Statistical Significance Does NOT Mean
Statistical significance is one of the most widely misinterpreted concepts in research. Here is what a significant result (p ≤ 0.05) does not tell you:
- It does not mean the result is important or large. Significance only concerns chance, not magnitude. A trivially small effect can be statistically significant if the sample is large enough.
- It does not mean there is a 95% chance the alternative hypothesis is true. The p-value is calculated assuming the null hypothesis is true; it is not the probability that any hypothesis is correct.
- It does not mean p = 0.05 is the probability the result was due to chance. A p-value of 0.04 does not mean there is a 4% chance the finding is a fluke. It means that if there were no real effect, data this extreme would occur 4% of the time.
- It does not prove the null hypothesis when p > 0.05. Failing to reject H0 means the evidence was insufficient — “absence of evidence is not evidence of absence.”
- It does not guarantee the result will replicate. A single significant study can still be a false positive; replication matters.
Statistical Significance vs Effect Size and Practical Importance
Because significance is so sensitive to sample size, it must always be reported alongside an effect size — a measure of how large the difference or relationship actually is (for example, Cohen’s d or a correlation coefficient r).
Practical (or clinical) significance goes one step further and asks whether the effect is large enough to matter in the real world. A result can be statistically significant yet practically meaningless, or practically important yet not (quite) statistically significant.
| Aspect | Statistical significance | Practical significance |
|---|---|---|
| Question it answers | Is the effect unlikely to be due to chance? | Is the effect big enough to matter? |
| Measured by | P-value vs alpha | Effect size in real-world units |
| Sensitive to sample size? | Yes — strongly | No |
| Tells you the effect is large? | No | Yes |
Why Statistical Significance Matters in Research
Statistical significance gives researchers a transparent, agreed-upon rule for separating likely-real findings from random noise. It is central to fields where false conclusions are costly — for instance, pharmaceutical drug trials, vaccine studies and pathology research routinely require results to clear a significance threshold before findings are accepted or published.
Its importance, however, is context-dependent. In academic and scientific publishing, significance testing is a near-universal gatekeeper. In business, by contrast, decision-makers are usually more interested in the practical impact of a finding — the projected revenue, conversion uplift or cost saving — than in whether a p-value crossed 0.05.
“Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. A conclusion does not immediately become true on one side of the divide and false on the other.” — American Statistical Association, Statement on p-Values (2016)
The sound approach is to treat statistical significance as the first filter — a check that an effect is unlikely to be chance — and then report an effect size and confidence interval so readers can judge how large and reliable the effect is. Used this way, significance, effect size and statistical power together give a far more honest picture than a lone p-value ever could.
Struggling to interpret your p-values and effect sizes?
ResearchProspect to the rescue!
Our expert statisticians run the correct tests, report significance and effect sizes properly, and explain exactly what your results mean — see our statistical analysis service.
