Significance level is the probability that the results observed in a study (or more extreme results) could have occurred by chance alone. It is closely associated with Type I error (see Type I and Type II errors): incorrectly rejecting the null hypothesis (false positive result). Based on the distribution of the test statistic used, a p value corresponding to the study results is estimated and compared with the pre-specified significance level, to determine whether or not to reject the study’s null hypothesis. Usually a 2-sided test is required, except in the case of non-inferiority studies. A threshold level of 5% (a = 0.05) for statistical significance is commonly used, which corresponds to a 1 in 20 chance of a positive result occurring by chance if there is no true underlying difference in the quantity of interest in the population. This level is based on convention rather than statistical theory. If many separate comparisons are being made in the analysis of a study, it is more likely that a significant result for one of these individual comparisons can occur by chance. In this case a tighter significance threshold (e.g. a = 0.01) may be used or adjustments such as Bonferroni correction may be made to p values associated with achieving the conventional threshold for each comparison.
How to cite: Significance Level [online]. (2016). York; York Health Economics Consortium; 2016. https://yhec.co.uk/glossary/significance-level/