MerQuR+Statistics Glossary

Statistics Glossary → Prediction error measures (RMSE, MAE, MAPE, MASE)

Prediction error measures (RMSE, MAE, MAPE, MASE)concept

Measures that summarize how far a model's predictions deviate from the actual values.

MAE (mean absolute error) is the average of the absolute errors; it is in the units of the data and easy to interpret. RMSE (root mean squared error) penalizes large errors more heavily because it squares them; it is always equal to or larger than MAE, and the gap shows how unevenly the errors are spread. MSE is the square of RMSE.

MAPE (mean absolute percentage error) expresses the error as a percentage of the actual value and helps compare series on different scales; but it becomes undefined or explodes when actual values are zero or near zero, and it does not penalize over- and under-forecasts equally. MASE divides the forecast MAE by the in-sample MAE of a simple (where needed, seasonal) naive forecast (Hyndman and Koehler); a value below 1 means the model beats the naive forecast. These measures should be computed on test data or by cross-validation, not on the training data.

An everyday example

Forecasts of a bakery's daily bread sales miss by 2, 3, 1, 4 and 15 loaves over 5 days. The MAE is 5 loaves, while the RMSE is about 7.1 loaves, because the single large error of 15 gains weight when squared. If occasional big misses matter to the baker, RMSE is the more meaningful measure.

In MerQur

'Extra output: CV performance' in Ridge regression (MSE, RMSE, MAE, R²_CV) and the same output in Lasso regression (MSE, R²_CV); test accuracy (RMSE, MAPE, MASE) with the 'Holdout set (last k observations)' in ARIMA Forecast. RMSE(CV) in regularized regression, RMSECV in partial least squares, RMSE and MAE in k-nearest neighbours regression, and ME, MAE and RMSE from leave-one-out cross-validation in Variogram + Kriging are also reported.

Analyses where this concept appears

Ridge RegressionLasso RegressionRegularized Regression (LASSO / Ridge / ElasticNet)Partial Least Squares Regression (PLS)Path AnalysisARIMA Forecastk-Nearest Neighbors (KNN)Variogram + Kriging (Spatial Interpolation)

Related concepts

Train-test split (holdout)k-fold cross-validation