Statistics Glossary → Bayesian Linear Regression
Bayesian Linear Regressionanalysis
A method that estimates a linear regression explaining an outcome by several predictors, combining prior information about the coefficients with the data and giving a posterior distribution and credible interval for each coefficient.
Concept
Classical linear regression gives a single estimate, a standard error and a p value for each coefficient. Bayesian regression produces the whole distribution of plausible values for each coefficient (the posterior distribution). This allows direct probability statements such as 'the probability that the coefficient is positive is 95%' or 'the coefficient lies between 0.4 and 0.9 with 95% probability', and knowledge from earlier studies can enter the model through the prior.
The model is the same as in classical regression: y = β₀ + β₁x₁ + … + ε, ε ~ N(0, σ²). The difference is that a prior distribution is specified for the βs and σ². By Bayes' rule the posterior is proportional to the likelihood times the prior: the vaguer (wider) the prior, the closer the result is to the classical estimate (OLS); the more precise the prior, the more the coefficients are pulled towards the prior mean (shrinkage). The evidence for the model against an intercept-only model can also be reported as a Bayes factor.
The assumptions of classical regression still hold: linearity, independent observations, and residuals with constant variance that are roughly normal. Because the choice of prior can affect the result, a prior sensitivity analysis should be done, especially with small samples. A credible interval gives the range in which a coefficient lies with the stated probability; with strong multicollinearity among the predictors, the coefficients' posteriors are wide and dependent on one another.
When to use it
- When explaining a continuous outcome with one or more predictors and the uncertainty of the coefficients should be expressed in terms of probability, or prior knowledge should be built into the model.
- With a small sample and reliable information from earlier studies, use an informed prior; without prior information, use the reference (Jeffreys) prior or Zellner's g-prior.
- If observations are grouped in clusters such as schools or hospitals, a hierarchical (mixed) model is preferred; if the outcome is binary or a count, generalised linear models such as logistic or Poisson regression are preferred.
An everyday example
An estate agency wants to explain the sale prices of 40 flats in a district by floor area and building age. Bayesian regression might put the posterior mean of the floor-area coefficient at 28 thousand TL per square metre, with a 95% credible interval of 22 to 34 thousand TL. If the interval for building age includes zero but the probability that this coefficient is negative is 90%, the agency can say 'older buildings are probably somewhat cheaper, but more sales are needed to be sure'.
Parameters in MerQur
Target (Y) and Predictors (X)
Concept: The target is the continuous outcome to be explained; the predictors are the variables used to explain it. A categorical predictor is expanded into dummy variables and each level is read relative to the base level.
Everyday example: In the estate-agency example, the target is sale price and the predictors are floor area and building age. If a variable such as 'lift / no lift' is added, its coefficient gives the price difference of flats with a lift compared with those without.
What it does: Variable selection. Text columns are dummy-coded (the first level is the baseline level), missing rows are dropped, and an intercept is always added.
Effect on the result: All outputs depend on the selected model.
On the validation data: attitude rating ~ complaints + learning (n = 30; main model), complaints only, 6 predictors and swiss Fertility ~ 5 predictors.
Prior precision (prior precision τ)
Concept: Prior precision expresses how strong the prior belief about the coefficients is. With low precision the prior is vague and the result is close to the classical estimate; as precision grows, the coefficients shrink towards the prior mean.
Everyday example: A very small τ amounts to saying 'I have no idea about the floor-area coefficient', and the result is practically equal to OLS. A very large τ behaves like 'I am sure it is 0' and pulls the coefficient towards zero whatever the data say, so τ should be chosen deliberately.
What it does: In the conjugate prior, β | σ² ~ N(μ0·1, σ²/τ·I); all coefficients, including the intercept, are pulled toward μ0 with the same precision.
Effect on the result: When τ is small, the result is close to OLS; as it grows, the coefficients (especially the intercept, which has a large scale) shrink toward μ0. The posterior of σ² and the log marginal likelihood also change.
On the validation data: Intercept: 9.870 for τ = 0.0001, 9.766 for 0.01, 4.766 for 1, 0.099 for 100; complaints coefficient 0.6435; 0.6444; 0.6843; 0.7132; log marginal likelihood -126.47; -119.58; -113.62; -109.92.
Prior hyperparameters (μ0, a0, b0)
Concept: μ₀ is the prior mean of the coefficients, the value expected before seeing the data. a₀ and b₀ set the shape of the inverse-gamma prior for the error variance σ²; the larger these values, the more precise the prior information about σ².
Everyday example: If earlier years' data suggest each coefficient is about 1, μ₀ = 1 can be chosen. If there is strong knowledge of how far prices typically deviate from the model, a₀ and b₀ are set to reflect it; without such knowledge, small values are kept.
What it does: μ0 is the prior mean for all coefficients; σ² ~ inverse gamma(a0 shape, b0 scale). Active only in the conjugate prior.
Effect on the result: The coefficient posteriors and the σ² posterior change; in the reference prior these boxes have no effect.
On the validation data: Intercept 9.777 for μ0 = 1; σ² mean 26.20 for a0 = 10 (default 41.92); 44.88 for a0 = 0.01; 42.52 for b0 = 10. Combined (μ0 = 1, a0 = b0 = 10, τ = 1): intercept 5.275, σ² 27.36.
Prior type (Conjugate (NIG) / Reference (Jeffreys) / Zellner g-prior) and g
Concept: The prior type sets the form in which prior information is given. The conjugate prior is tuned explicitly by a mean and a precision; the reference (Jeffreys) prior carries no information and gives results numerically equal to OLS; Zellner's g-prior shrinks the slopes towards zero by a single g value, in a way suited to the predictors' scales.
Everyday example: If the agency has no earlier knowledge, the reference prior means 'let the data speak alone'. With Zellner's g-prior and g = 40 (the number of flats), the slopes shrink to 40/41 of their OLS values, that is, only very slightly.
What it does: Conjugate NIG; reference prior (Jeffreys, p(β, σ²) ∝ 1/σ²: the posterior mean equals OLS and the interval equals the t interval); Zellner g-prior (flat intercept, slopes shrunk to g/(1+g) times OLS; if g = 0 is entered, g = n). g is active only in the g-prior.
Effect on the result: Coefficient posteriors, credible intervals and the log BF change.
On the validation data: Reference prior: complaints 0.6435 [0.4004, 0.8866]. g = n = 30: 0.6228, log BF10(g) = 13.324; g = 10: 0.5850, 12.564; g = 1000: 0.6429, 10.906.
Bayes factor (JZS) + R² and JZS prior scale (r)
Concept: The Bayes factor shows how many times more probable the data are under the model with the predictors than under the intercept-only model. The prior scale sets the width of the effects expected under H₁.
Everyday example: If BF₁₀ = 5000, the evidence that floor area and building age together explain price is very strong. R² says how much of the variation in prices these two variables explain.
What it does: Gives the JZS (Zellner-Siow) Bayes factor of the full model against the intercept-only model (Liang et al. 2008; Rouder and Morey 2012) and R². The scale is the JZS prior scale for continuous predictors (medium √2/4, wide 1/2, ultrawide √2/2) and is effective only when the Bayes factor is on.
Effect on the result: BF10 changes; the posterior coefficients are not affected.
On the validation data: attitude complaints only BF10 = 417 938.6; complaints + learning 207 271.9, wide 265 544.5, ultrawide 315 512.9; swiss 2.408e7.
Confidence Level (%90 / %95 / %99 / Other (custom))
Concept: Sets the level of the credible interval: a 95% credible interval is the range in which the coefficient lies with 95% probability, given the data and the prior.
Everyday example: If the 95% credible interval for the floor-area coefficient is 22 to 34 thousand TL, choosing 90% narrows it and choosing 99% widens it.
What it does: The level of the credible interval (t quantiles).
Effect on the result: Only the interval limits change.
On the validation data: complaints coefficient: 90% [0.4600, 0.8287], 95% [0.4227, 0.8660], 99% [0.3463, 0.9424], 80% [0.5019, 0.7868].
Extra output: Standardized β + HDI; Prior sensitivity; Posterior predictive check
Concept: Standardised coefficients make predictors in different units comparable; prior sensitivity shows how the results change with the prior precision; the posterior predictive check tests whether artificial data generated from the model resemble the real data.
Everyday example: Because floor area and building age are in different units, the standardised β shows which affects price more. If, in the posterior predictive check, the prices the model generates fail to match the distribution of real prices, the model is missing something.
What it does: Standardized coefficients β·sd(X)/sd(Y) with the narrowest (HDI) and equal-tailed intervals for σ²; posterior means and the log marginal likelihood on a τ grid (between 0.001 and 100); the coverage of the analytic 95% prediction interval and a posterior predictive p value from 2000 simulations.
Effect on the result: Only adds output.
On the validation data: Main model σ² HDI [23.194, 64.274]; PPC coverage 100%. Standardization and sensitivity are computed in closed form, the PPC by Monte Carlo simulation.
Flat prior for the intercept
Concept: The intercept is the value of the outcome when all predictors are zero, and its size depends on the scale of the data. Giving it a flat prior means applying prior information only to the slopes, without pulling the intercept towards the prior mean.
Everyday example: Since flat prices are in the millions of TL, pulling the intercept towards 0 makes no sense. The flat prior leaves the intercept free, and shrinkage applies only to the floor-area and building-age coefficients.
What it does: In the conjugate prior, a flat (improper) prior p(β0) ∝ 1 for the intercept; the other coefficients β_j | σ² ~ N(μ0, σ²/τ). Active only in the conjugate prior.
Effect on the result: The intercept is not shrunk toward μ0; the degrees of freedom of the σ² posterior decrease by one (a_n = a0 + (n-1)/2); the log marginal likelihood is undefined.
On the validation data: Main model: intercept 9.871 (OLS 9.871; default conjugate 9.766), SD 6.819; with τ = 1, 9.878 (4.766 without the flat prior). Ticking it in the reference prior has no effect.
Example on real data
Data: Chatterjee and Price 1977 attitude, R datasets
Variables: target rating (overall rating), predictors complaints and learning
- Question: In 30 departments, how is the overall rating of the manager (rating) related to the handling of complaints (complaints) and learning opportunities (learning)? We estimate the posterior distribution of the coefficients with a conjugate prior (τ = 0.01, vague prior).
- Result: The posterior mean of complaints is 0.644 (SD 0.112), 95% credible interval [0.423, 0.866], P(β > 0) ≈ 1.00: a 1-unit increase in the complaints score raises the rating by about 0.64 points.
- Second predictor: The posterior mean of learning is 0.212, 95% credible interval [-0.040, 0.464]; the interval contains 0 but P(β > 0) = .952; a positive effect is likely but uncertain.
- Model fit: The posterior mean of σ² is 41.92 (SD 11.20); the log marginal likelihood is -119.58 (used in model comparison).
- Effect of the prior: Because τ = 0.01 is vague, the result is very close to OLS (OLS complaints 0.6435, posterior 0.6444); the intercept is shrunk slightly toward 0 (OLS 9.871, posterior 9.766).
- APA: Bayesian linear regression (conjugate NIG, τ = 0.01): complaints β = 0.64, 95% CrI [0.42, 0.87]; learning β = 0.21, 95% CrI [-0.04, 0.46]; N = 30.
Related concepts
Conjugate prior (normal-inverse-gamma, Zellner's g)Prior distributionBayes factor (BF₁₀)Non-informative (flat, Jeffreys) priorPosterior distribution and posterior predictive checkStandardized regression coefficient (β)JZS (Cauchy) prior and the scale parameter rCredible interval and highest density interval (HDI)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.