MerQuR+Statistics Glossary

Statistics Glossary → p-value

p-valueconcept

The probability, assuming the null hypothesis is true, of obtaining a result at least as extreme as the one observed purely by chance.

A test computes a test statistic from the data (such as t, F or χ²) and compares it with the distribution expected under the null hypothesis. The p-value is the probability of seeing a statistic this extreme or more extreme if the null hypothesis were true. A small p means the data are hard to reconcile with the null hypothesis; if it falls below the significance level chosen in advance (usually α = 0.05), the null hypothesis is rejected.

The p-value is not the probability that the null hypothesis is true, and it says nothing about the size or importance of an effect. With very large samples, trivially small differences can give tiny p-values; with small samples, a real and important effect can fail to reach significance. For this reason p should be reported together with an effect size and a confidence interval. When many tests are run, some small p-values appear by chance alone, which calls for a multiple-comparison correction.

An everyday example

A friend claims their coin is fair, but it lands heads 16 times in 20 tosses. If the coin really is fair, the chance of 16 or more heads in 20 tosses (or an equally extreme 4 or fewer) is about 0.012. This small p says the result sits poorly with the claim that the coin is fair; it does not mean there is a 98.8% chance that the coin is biased.

In MerQur

Nearly every hypothesis test in MerQur reports a p-value, including the normality tests, t-tests, the permutation and binomial tests, crosstabs, correlation, regression and mixed models. In Multiple Comparison Correction, the raw p-values entered in the 'p list' field are adjusted; in Mann-Kendall + Sen's Slope, the 'Additional output: Change point, Exact p, Save' option gives a p-value from the exact distribution of S for tie-free series with n <= 60.

Analyses where this concept appears

Normality TestsOne-Sample t-TestPermutation TestMultiple Comparison CorrectionBinomial TestLog-Linear ModelCrosstabMR × Categorical CrosstabMR × MR CrosstabCochran's Q TestCorrelation AnalysisMultiple Linear RegressionLogistic RegressionNested Mixed Model (Nested LMM, Multi-Level Hierarchical Mixed Model)Crossed Mixed Model (Crossed LMM, Crossed Random Effects Mixed Model)Panel Data RegressionConfusion Matrix MetricsExploratory Factor Analysis (EFA)Variance Components Analysis (VARCOMP)Generalized Additive Model (GAM, Nonlinear Regression)Bayesian t-Test (JZS Bayes Factor)Bayesian Linear RegressionGWR (Geographically Weighted Regression)Kaplan-Meier Survival AnalysisTime SeriesMann-Kendall + Sen's SlopePLS-SEM (Partial Least Squares Path Modeling)Moran's I (Spatial Autocorrelation)Getis-Ord Gi* HotspotPoint Pattern Analysis (Ripley K/L, G/F/J, CSR)

Related concepts

Significance level (α)Null hypothesis (H₀) and test valueExact test and exact p-value