MerQuR+Statistics Glossary

Statistics Glossary → Ordinary least squares (OLS)

Ordinary least squares (OLS)concept

A method that estimates regression coefficients by minimizing the sum of squared differences between the observed values and the model's predictions (the residuals).

In linear regression the coefficients are chosen to minimize Σ (y_i - ŷ_i)²; in matrix notation the solution is b = (X'X)⁻¹ X'y. Squaring stops positive and negative errors from cancelling out and gives more weight to large errors. This is why OLS is sensitive to outliers and influential observations.

By the Gauss-Markov theorem, if the model is correctly specified and the errors are uncorrelated with the regressors, have constant variance and are uncorrelated with each other, OLS has the smallest variance among linear unbiased estimators (BLUE). Normality is needed not for unbiasedness but for exact t and F tests in small samples. Under heteroscedasticity the coefficients stay unbiased but the standard errors are wrong, so robust standard errors or weighted least squares are used. If a regressor is endogenous, OLS is biased and instrumental-variable methods are needed.

An everyday example

A taxi company wants to explain fares by distance. Among hundreds of points on the chart, a line is drawn such that the sum of the squared vertical distances from every point to the line is as small as possible. The slope of this line estimates the price per kilometre, and the intercept the starting fare.

In MerQur

The estimation method of Multiple Linear Regression; its 'Standard error type' option offers the Classic (OLS) error alongside the HC0-HC3 robust errors, and the 'WLS weight variable' applies weighted least squares. In Lasso Regression, 'Extra output: OLS on selected' refits an unpenalized model on the selected variables; SAR reports 'OLS assumption diagnostics'. The 'Estimator' of Instrumental Variables Regression (2SLS, two-stage least squares), pooled OLS in Panel Data Regression and conditional and unconditional least squares estimation in ARIMA Forecast also rest on this method.

Analyses where this concept appears

Two-Way ANOVABland-Altman Agreement AnalysisMultiple Linear RegressionLogistic RegressionRidge RegressionLasso RegressionNested Mixed Model (Nested LMM, Multi-Level Hierarchical Mixed Model)Panel Data RegressionInstrumental Variables Regression (2SLS)Weighted Linear Regression (Survey Regression)Nonlinear RegressionQuantile RegressionBayesian Linear RegressionSAR (Spatial Autoregressive Lag)Spatial Error Model (SEM)GWR (Geographically Weighted Regression)Mediation AnalysisMultiple ImputationARIMA ForecastLOESS (Local Regression)Sample Selection and Truncated Regression (Heckman)Variogram + Kriging (Spatial Interpolation)

Related concepts

Weighted least squares (WLS)Coefficient of determination (R²) and adjusted R²