Statistics Glossary → Monte Carlo p-value
Monte Carlo p-valueconcept
An approximate p value obtained by mimicking the distribution of a test statistic under the null hypothesis through a large number of random rearrangements or simulations.
Computing an exact p value requires enumerating every possible permutation or table, which becomes impossible as the sample grows. Asymptotic (large-sample) approximations, on the other hand, can be unreliable for small samples or for tables with low expected counts. In the Monte Carlo approach the data are regenerated B times (e.g. 10,000) at random under the null hypothesis, the test statistic is computed each time, and the number b of simulations that are at least as extreme as the observed value is counted. The formula p = (b + 1) / (B + 1) treats the observed data as one of the possible arrangements and prevents p from being exactly zero.
Because the result rests on random simulation it carries a small sampling error; a confidence interval for p (e.g. Clopper-Pearson) is therefore reported, and the random seed is fixed so that results can be reproduced. The error shrinks as B grows. A Monte Carlo p value is an approximation that stands in for an exact test; it does not change the null hypothesis or the assumptions of the test.
An everyday example
You wonder whether a die is loaded: in 60 rolls, six came up 18 times. Instead of working out the exact probability by hand, you have a computer roll a fair die 60 times and repeat that experiment 10,000 times. If 250 of those runs show 18 or more sixes, the Monte Carlo p value is about (250 + 1) / (10,000 + 1) ≈ 0.025.
In MerQur
It is offered through 'p computation (Monte Carlo, Exact, Automatic)' in the Permutation Test, the 'Monte Carlo p' extra output in the Kruskal-Wallis test (B = 10000), the 'Monte Carlo p' checkbox in the Friedman test, 'Permutation p' in the Mann-Whitney U and Wilcoxon tests, 'Exact / Monte Carlo p (None, Monte Carlo χ², Fisher)' in Chi-Square Independence and 'Exact / Monte Carlo p' in Chi-Square Goodness of Fit, the 'Monte Carlo p' extra output in Crosstab and the 'p computation' option in Cochran's Q test. In Normality Tests the one-sample K-S p (parameters from the data) is computed by Monte Carlo by default, and MR × Categorical Crosstab applies a Monte Carlo Fisher test in 2xk tables. Fisher's Exact Test applies the Fisher-Freeman-Halton test by Monte Carlo for more than two levels; GWR gives the 'Monte Carlo test of variability', and Point Pattern Analysis gives the MAD p value of the CSR simulation envelopes.
Analyses where this concept appears
Normality TestsPermutation TestMann-Whitney UWilcoxon Signed-RankKruskal-WallisFriedmanChi-Square IndependenceChi-Square Goodness of FitFisher's Exact TestCrosstabMR × Categorical CrosstabCochran's Q TestGWR (Geographically Weighted Regression)Point Pattern Analysis (Ripley K/L, G/F/J, CSR)