MerQuR+Statistics Glossary

Statistics Glossary → Significance level (α)

Significance level (α)concept

The upper limit, set by the researcher in advance, on the probability of rejecting a null hypothesis that is actually true.

The significance level is a threshold chosen before the test is run; the most common values are 0.05, 0.01 and 0.10. If the p value from the test is less than or equal to α, the null hypothesis is rejected and the result is called 'statistically significant'. Choosing α = 0.05 means accepting that, when the null hypothesis is true, about 5% of tests will wrongly declare an effect in the long run (a Type I error).

α is often confused with the p value: α is a fixed criterion chosen by the researcher, whereas p is computed from the data. Lowering α reduces Type I errors but increases the chance of missing a real effect (a Type II error), so power falls. When many tests are run on the same data, using α = 0.05 for each one inflates the overall false-positive risk, which is why multiple-comparison corrections such as Bonferroni, Holm or FDR are needed. It is also tied to the confidence level: α = 0.05 corresponds to a 95% confidence interval.

An everyday example

Think of setting the sensitivity of a smoke alarm. Set it too sensitive and it goes off whenever you make toast (a false alarm); set it too lax and it notices a real fire late. α is the setting that says how many false alarms you are willing to tolerate: α = 0.05 means accepting that, when there is no fire, the alarm will sound in about 5 out of every 100 situations.

In MerQur

It is set as the 'α (alpha)' parameter in Normality Tests, Moran's I and the Getis-Ord Gi* Hotspot analysis, as the 'α (0.001-0.999)' parameter in Multiple Comparison Correction, and as 'Grubbs/ESD α and k' and 'Mahalanobis/MCD threshold (α)' in Anomaly Detection. In analyses with a confidence level option (the t-tests, the ANOVAs, MANOVA, ANCOVA, Mann-Whitney U, Wilcoxon, Kruskal-Wallis, Friedman, the Sign, Runs, Binomial and chi-square tests, Cochran's Q, Kaplan-Meier, Time Series) α = 1 - confidence level, and the H0 decision is made against this threshold.

Analyses where this concept appears

Normality TestsOne-Sample t-TestIndependent-Samples t-TestPaired t-TestOne-Way ANOVATwo-Way ANOVARepeated Measures ANOVAMANOVAANCOVABootstrap CIMultiple Comparison CorrectionMann-Whitney UWilcoxon Signed-RankKruskal-WallisFriedmanBinomial TestSign TestRuns TestChi-Square IndependenceChi-Square Goodness of FitMR × Categorical CrosstabMR × MR CrosstabCochran's Q TestKaplan-Meier Survival AnalysisTime SeriesAnomaly DetectionMoran's I (Spatial Autocorrelation)Getis-Ord Gi* Hotspot

Related concepts

p-valueConfidence level and confidence intervalType I and Type II errors