Statistics Glossary → Bayesian t-Test (JZS Bayes Factor)
Bayesian t-Test (JZS Bayes Factor)analysis
A test of whether a mean, or the difference between two means, is zero, using a Bayes factor that shows how strongly the data support the 'difference' and 'no difference' hypotheses relative to each other.
Concept
The classical t-test only says whether we can reject the 'no difference' hypothesis; a non-significant p value does not prove that there is no difference. The Bayesian t-test compares the two hypotheses directly and measures evidence in both directions: the data can support 'no difference', support 'a difference', or fail to discriminate between them.
The key quantity is the Bayes factor: BF₁₀ = P(data | H₁) / P(data | H₀). BF₁₀ = 6 means the observed data are 6 times more likely under H₁ than under H₀; BF₁₀ = 0.2 (that is, BF₀₁ = 5) means the data support H₀ five times more. Roughly, values of 1-3 are read as weak, 3-10 as moderate and above 10 as strong evidence. Under H₁ the effect size δ is given a prior distribution; the common default is a Cauchy prior centred on zero whose width is set by r (the JZS prior). The test also gives the posterior distribution of δ and its credible interval.
The assumptions are those of the classical t-test: independent observations and approximate normality (plus similar variances for two independent samples). The Bayes factor is sensitive to the width of the prior, so a sensitivity analysis with different r values should be reported. A credible interval states directly that, given the data and the prior, δ lies in the interval with the stated probability.
When to use it
- When the strength of evidence about a mean difference is wanted rather than just 'significant / not significant', especially when support for 'no difference' is needed (e.g. showing that two methods are equally good).
- When comparing one group's mean with a reference value, two independent groups, or two measurements on the same people; for three or more groups, use Bayesian ANOVA.
- If the distribution is clearly non-normal and the sample is small, rank-based tests (Mann-Whitney or Wilcoxon) or their Bayesian versions should be considered; if a pre-specified equivalence range exists, TOST is an option in the classical framework.
An everyday example
In a hospital's physiotherapy unit, 40 knee-surgery patients were split between two exercise programmes, and their range of motion is measured after six weeks. When the classical t-test gives p = 0.08, saying 'the programmes do not differ' would be misleading. If the Bayesian t-test gives BF₁₀ = 1.1, the data support the two hypotheses almost equally, so more patients are needed. Had it given BF₁₀ = 0.15, we could say the data support 'no difference' about 7 times more.
Parameters in MerQur
Mode (One-sample / Independent two-sample / Paired)
Concept: Sets which comparison the test makes: one mean against a reference value, two independent groups against each other, or two measurements on the same people. With paired measurements the differences between people are removed, so the same effect can yield stronger evidence.
Everyday example: Comparing the range of motion of two separate patient groups uses the independent two-sample mode; comparing the same patients before and after surgery uses the paired mode; comparing the group mean with the 'normal' 120 degrees uses the one-sample mode.
What it does: Determines the type of test: H0: μ = μ0 in one-sample, H0: μ1 = μ2 in independent two-sample, and H0: μ_difference = 0 in the paired mode.
Effect on the result: The t statistic and the effective sample size of δ change (n in one-sample and paired, n1·n2/(n1+n2) in independent).
On the validation data: sleep paired: t = -4.062, BF10 = 17.259. The same first measurement tested as one-sample (μ0 = 0) gives t = 1.326, BF10 = 0.617; the same 20 values treated as two independent groups give BF10 = 1.266. For ToothGrowth len ~ supp, BF10 = 1.199.
Dependent variable, Group column, 2nd measurement
Concept: The dependent variable is the continuous measurement being compared; the group column defines the two independent groups, and the second-measurement column gives the same person's other measurement in a paired design.
Everyday example: Range of motion is the dependent variable; the 'programme A / programme B' column is the group column. If pre- and post-surgery measurements are in two separate columns, the second is chosen for the paired mode.
What it does: Variable selection. The group column is active only in independent mode, and the 2nd measurement only in paired mode.
Effect on the result: In independent mode, leaving the 2nd measurement selected does not change the result. The order of the group levels is the data order (the first one seen is group 1; in ToothGrowth VC - OJ).
On the validation data: In the sleep two-group version no value changes when the 2nd measurement stays selected; no additional numerical difference.
μ₀ (one-sample reference)
Concept: The value stated by the null hypothesis in the one-sample test (H₀: μ = μ₀). The Bayes factor depends on how far the data move away from this value.
Everyday example: If the question is whether patients' mean range of motion differs from the 120 degrees expected in healthy knees, μ₀ = 120 is entered. A mean of 118 would likely support H₀; a mean of 100 would strongly support H₁.
What it does: The test value used in the one-sample test.
Effect on the result: t = (x̄ - μ0)/(s/√n) and the posterior of δ change with μ0; the confidence interval is computed for the difference (x̄ - μ0). No effect in the paired mode.
On the validation data: sleep extra_1: BF10 = 0.617 for μ0 = 0, 0.336 for μ0 = 1, 5.071 for μ0 = -1. In the paired mode μ0 = 5 does not change the result.
JZS prior scale (r)
Concept: The width of the Cauchy prior for the effect size δ under H₁; r expresses the typical size of effects the researcher expects. r = 0.707 means half of the effects are expected within |δ| < 0.707.
Everyday example: If the physiotherapists expect a small-to-medium difference between programmes, r = 0.707 is a reasonable default. Making r larger lets H₁ treat very large effects as plausible too; when the observed difference is small, this usually lowers BF₁₀. Trying the result with several r values is therefore good practice.
What it does: The width of the δ ~ Cauchy(0, r) prior (Rouder et al. 2009). The named scales are medium √2/2, wide 1, ultrawide √2. The default box value (0.707) is treated as exactly √2/2.
Effect on the result: BF10 and the posterior of δ change; the frequentist output does not change.
On the validation data: sleep paired BF10: 15.030 for r = 0.5; 17.259 for √2/2 (17.258880); 18.415 for 1; 18.015 for √2. chickwts: 5.976 for r = 0.707, 6.193 for r = 1.
Prior type (Cauchy r box / medium / wide / ultrawide / Informed t or normal)
Concept: A prior is a probability distribution expressing what we believe about the effect size before seeing the data. Default (objective) priors are wide and centred on zero; informed priors carry information from earlier studies.
Everyday example: If there is no earlier research on the question, a standard width such as 'medium' is chosen. If many studies show the effect of similar programmes, an informed prior reflecting that knowledge can be used.
What it does: Selects the family of the prior. "Cauchy (value in the r box)" uses the box above; the three named options are fixed r values; the informed option carries prior information.
Effect on the result: The scale in the r box applies only to the first option; in the others BF10 is computed with the fixed prior scale.
On the validation data: For sleep paired, the same numbers as the r values above.
Informed prior (location, scale, df)
Concept: An informed prior centres the effect size not on zero but on a location chosen from earlier knowledge; the location sets the expected effect, the scale the uncertainty, and the degrees of freedom how heavy the tails are. If the prior's direction contradicts the data, the Bayes factor can shift in favour of H₀.
Everyday example: If earlier studies show programme B raising range of motion by about 0.35 standard deviations, the prior is built around that value. If the data instead show B doing worse, the evidence shifts towards H₀, because this prior carries the expectation that B is better.
What it does: A δ ~ t_ν(location, scale) prior (normal if ν is empty) (Gronau, Ly and Wagenmakers 2020). The default "0.350, 0.102, 3" is the Oosterwijk prior.
Effect on the result: The headline BF10, the BF for the selected H1, BF±0, the posterior of δ, the APA sentence and the method line are computed with this prior; the Cauchy BF is given in a separate line for comparison.
On the validation data: sleep paired (negative effect): with the Oosterwijk prior BF10 = 0.252, posterior median -0.360; with an N(0, 1) prior BF10 = 25.475. A prior whose direction contradicts the data comes to support H0.
Alt. hypothesis (Two-sided / δ > 0 / δ < 0)
Concept: Sets the direction of H₁. Under a directional hypothesis the prior allows only effects in the expected direction; if the data point that way the evidence strengthens, and if they point the other way it turns towards H₀. The direction must be chosen before looking at the data.
Everyday example: If programme B can only plausibly be better, 'δ > 0' is chosen. If the data really favour B, BF₊₀ comes out larger than the two-sided BF₁₀; if they favour A, BF₊₀ becomes very small.
What it does: Computes BF+0 or BF-0 for a directional H1; the posterior summaries are given with the truncated prior.
Effect on the result: Directional BFs and their truncated posterior summaries; the one-sided p value.
On the validation data: sleep paired: BF-0 = 34.415, BF+0 = 0.101; one-sided p = 0.0014.
Confidence Level (%90 / %95 / %99 / Other, custom value)
Concept: Sets the level of the frequentist confidence interval and the Bayesian credible interval. A credible interval directly gives the probability that the parameter lies in the interval given the data; the HDI is the narrowest interval carrying that probability.
Everyday example: If the 95% credible interval for δ is [0.1, 0.9], then, taking the data and prior together, the true effect between the programmes lies in this range with 95% probability. Choosing 90% narrows it.
What it does: The level of the frequentist confidence interval, the credible interval of δ, the highest posterior density interval (HDI) and the descriptive credible interval.
Effect on the result: Interval limits; the Bayes factors do not change. While the custom box is off, a value entered in it has no effect.
On the validation data: sleep paired CI: 90% [-2.293, -0.867], 95% [-2.460, -0.700], 99% [-2.844, -0.316]; 90% credible interval for δ [-1.829, -0.401].
Additional output: Descriptives, Assumption checks, Sequential analysis
Concept: Descriptives summarise the groups, assumption checks cover normality and equal variances, and the sequential analysis shows how the Bayes factor changes as observations are added.
Everyday example: If in the sequential plot BF₁₀ hovers around 1 for the first 10 patients and climbs to 8 by 40 patients, the evidence is seen to grow steadily towards 'a difference' as data accumulate. The Shapiro-Wilk and Levene results show whether the t-test assumptions are reasonable.
What it does: Descriptives give n, mean, SD, SE and the credible interval of the mean with a Jeffreys prior. Assumption checks include the Shapiro-Wilk and Levene tests per group (on the difference in the paired case). Sequential analysis gives the BF10 sequence at the first i observations.
Effect on the result: Only adds output; it does not change the main BF.
On the validation data: chickwts horsebean mean 160.2, 95% credible interval [132.57, 187.83]; Levene F = 1.469, p = 0.240.
Example on real data
Data: Crampton 1947 ToothGrowth, R datasets
Variables: independent two-sample mode, dependent len (tooth length), group supp (VC / OJ)
- Question: Does the source of vitamin C (orange juice OJ versus ascorbic acid VC) change tooth length in guinea pigs? We compare H₀ (no difference) with H₁ by the Bayes factor.
- Frequentist counterpart: t(58) = -1.92, p = .060, 95% CI [-7.57, 0.17]; not significant at the 5% level, d = -0.49 (a medium-sized difference).
- Result: BF₁₀ = 1.199 (BF₀₁ = 0.834); the data support H₁ over H₀ only 1.2-fold, i.e. "weak / inconclusive evidence". Rather than concluding "no difference" from p = .06, one should say that the data do not discriminate between the two hypotheses.
- Posterior (effect size): δ median = -0.42, 95% credible interval [-0.92, 0.05]; the interval contains 0, so the tendency toward shorter teeth with VC is uncertain.
- Sensitivity: BF₁₀ decreases as the prior widens (r = 1: 0.993; r = 1.414: 0.778); for the directional hypothesis VC < OJ, BF₋₀ = 2.297.
- APA: Bayesian independent t test (JZS, r = 0.707): BF₁₀ = 1.20, d = -0.49, δ median = -0.42, 95% CrI [-0.92, 0.05]; t(58) = -1.92, p = .060.
Related concepts
Null hypothesis (H₀) and test valueEffect sizePrior distributionJZS (Cauchy) prior and the scale parameter rBayes factor (BF₁₀)Posterior distribution and posterior predictive checkOne-sided and two-sided tests (hypothesis direction)Credible interval and highest density interval (HDI)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.