MerQuR+Statistics Glossary

Statistics Glossary → Anomaly Detection

Anomaly Detectionanalysis

Methods that use statistical or machine-learning criteria to identify and flag unusual (outlying or anomalous) observations that do not fit the general structure of a data set.

Concept

Outlying observations may come from a measurement or data-entry error, from an unusual but genuine event, or from a separate subgroup. If they go unnoticed, they can badly distort results such as the mean, standard deviation, correlation and regression. Anomaly detection finds these observations and shows why they were flagged; but whether an observation should be removed is a subject-matter decision, not a statistical one.

The methods fall into three groups. Univariate methods examine each variable separately: the IQR method flags values lying a certain distance beyond the quartiles, the z-score looks at how many standard deviations a value lies from the mean, and the modified z-score uses robust measures such as the median and MAD; the Grubbs and Rosner ESD tests formally test for outliers under normality. Multivariate methods assess the variables together: the Mahalanobis distance takes the correlation between variables into account, while Isolation Forest and LOF score points by how isolated they are or by local density, without assuming a distribution. The time-series method first removes trend and season and then looks for outliers in the remainder.

An important trap is masking: when several outliers pull the mean and standard deviation towards themselves, classical methods fail to see them. Robust methods such as the median/MAD and robust Mahalanobis (MCD) are therefore more reliable. In most methods the threshold is a choice, and the number of flagged observations changes with it. If a large share is flagged, these may be a separate cluster in the data rather than 'errors'.

When to use it

  • For data cleaning and quality control before analysis, to flag entry errors, measurement faults or suspicious records.
  • To look for extreme values in a single variable, use the IQR or the modified z-score; if values may be ordinary one by one but unusual together, use Mahalanobis/MCD, Isolation Forest or LOF.
  • To look for outlying periods in a seasonal time series, use the STL remainder + IQR; to find the observations that most influence a model's results, regression influence diagnostics (Cook's distance, leverage) are preferred.

An everyday example

A supermarket chain is examining the amount, number of items and discount of every transaction through its tills over a month. A receipt where the cashier typed 1,250 instead of 12.50 stands out at once as an extreme value in the amount variable alone. A subtler case is a 400-unit discount on a three-item basket: neither the number of items nor the discount is unusual by itself, but the combination is, and only a multivariate method such as Mahalanobis or Isolation Forest catches it.

Parameters in MerQur

Feature(s) (variable selection)

Concept: Sets which variables the anomaly search is run on. Univariate methods examine each variable separately and flag a row that crosses the limit in any of them; multivariate methods look at the pattern the variables form together.

Everyday example: With only the transaction amount selected, the faulty receipt of 1,250 is found. With the number of items and the discount selected together, oddities that only appear in combination, such as a large discount on a small basket, can also be sought.

What it does: Univariate methods (IQR, z, MAD, Grubbs, ESD) are applied per variable; a row exceeding the bound in any of them is an anomaly. Isolation Forest, LOF, Mahalanobis and MCD are multivariate (Isolation Forest and LOF need at least 2 features).

Effect on the result: The set of variables examined.

On the validation data: hbk with X1 alone: IQR still finds cases 1-14.

Method (IQR / Z-Score / Modified Z-Score MAD / Isolation Forest / LOF / Mahalanobis D² / Robust Mahalanobis MCD / Grubbs / Rosner ESD / Time series: STL remainder + IQR)

Concept: Chooses the criterion that defines an outlier: distance from the quartiles (IQR), from the mean (z), from a robust centre (MAD), isolation (Isolation Forest), local density (LOF), distance accounting for correlation (Mahalanobis, MCD), a formal test under normality (Grubbs, ESD) or a time-series remainder. Classical criteria are prone to masking; robust ones are not.

Everyday example: If the same cashier made the same mistake five times in a day, those five extreme receipts pull the mean and standard deviation upwards, and the z-score may flag none of them. A method based on the median and MAD catches all five.

What it does: Selects the anomaly criterion.

Effect on the result: The set of flagged observations.

On the validation data: hbk X1-X3, default thresholds: IQR 14 (1-14), z 1 (14), MAD 14, Isolation Forest 4 (11-14), LOF 4, Mahalanobis 1 (14), MCD 14, Grubbs 1, ESD 12. Classical methods suffer masking; robust methods (MAD, MCD) find all of 1-14.

→ Outlier

IQR multiplier (k)

Concept: In the IQR method the limits (Tukey fences) are drawn k·IQR below the first quartile and k·IQR above the third. k = 1.5 is the usual limit; k = 3 is a strict limit that catches only very extreme values.

Everyday example: If the transaction amounts have quartiles of 80 and 320, the IQR is 240; with k = 1.5 the upper fence is 680, and with k = 3 it is 1,040. The faulty receipt of 1,250 is flagged in both cases, while a large but ordinary 700 purchase is flagged only at k = 1.5.

What it does: Fences Q1 - k·IQR and Q3 + k·IQR (quartiles by linear interpolation, Hyndman-Fan type 7). In the time series method it is applied to the STL remainder.

Effect on the result: The fences and the number of flags.

On the validation data: hbk: 18 anomalies for k = 0.5; 14 for k = 1.5, 3.0 and 5.0. AirPassengers STL: 23 for k = 1.5, 15 for k = 3.0.

→ Interquartile range (IQR) and Tukey fences

Z threshold

Concept: The z-score shows how many standard deviations a value lies from the mean; observations whose absolute value exceeds the threshold are counted as outliers. Because the mean and standard deviation are themselves affected by outliers, the method is prone to masking.

Everyday example: If the mean transaction is 210 with a standard deviation of 150, a threshold of 3 flags receipts above 660. Lowering it to 2 makes every purchase above 510 suspicious, and many ordinary large purchases join the list.

What it does: |z| > threshold, z = (x - x̄)/s.

Effect on the result: Number of flags. The label in the method list shows the chosen threshold.

On the validation data: hbk: 9 anomalies for threshold 2.0, 1 for 3.0, 0 for 5.0.

→ z-score (standardization)

MAD threshold

Concept: The modified z-score uses the median instead of the mean and the median absolute deviation (MAD) instead of the standard deviation; because these measures are hardly affected by outliers, it resists masking. A commonly recommended threshold is 3.5.

Everyday example: Even if a few huge mis-typed receipts inflate the mean, the median transaction amount barely changes. The MAD method therefore also catches the faulty receipts that the z-score misses.

What it does: |M| > threshold, M = 0.6745 (x - median)/MAD (Iglewicz and Hoaglin 1993).

Effect on the result: Number of flags.

On the validation data: hbk: because 1-14 are very far, 2.0, 3.5 and 5.0 all give 14 anomalies.

→ Median absolute deviation (MAD) and modified z-score

Contamination (Isolation Forest and LOF)

Concept: Contamination is a prior guess at the proportion of anomalies in the data: scores are ranked and the most extreme fraction c of observations is counted as anomalous. It is an assumption made by the user, not a property of the data; the number flagged is about c·n.

Everyday example: If management believes about 1% of transactions are erroneous, it enters c = 0.01, and of 10,000 transactions the roughly 100 that look strangest are reviewed. With c = 0.05, 500 transactions would be reviewed.

What it does: The (1 - c) percentile of the scores becomes the threshold; the number of anomalies is about c·n.

Effect on the result: Number of flags.

On the validation data: hbk: 1 for c = 0.001, 4 for 0.05, 8 for 0.10, 15 for 0.20.

→ Isolation forest and contamination rate

LOF neighbors

Concept: LOF compares the density around a point with the density around its neighbours; a point lying in a region much sparser than its neighbours' scores high. The number of neighbours k sets how wide 'local' is defined.

Everyday example: If small weekday baskets and large weekend family shops form two separate dense regions, LOF counts a large basket as an outlier not simply because it is large but when it is close to neither group. A very small k may treat a handful of similar odd transactions as a 'normal' cluster.

What it does: Number of neighbours k for LOF.

Effect on the result: LOF scores.

On the validation data: hbk: the flags (11-14) do not change for k = 5, 20 and 50; the LOF scores are computed from the k nearest neighbours of each point.

→ Number of nearest neighbours (k) and the neighbourhood graph

Mahalanobis/MCD threshold (α)

Concept: The Mahalanobis distance (D²) measures how far an observation lies from the centre while accounting for the variables' variances and their correlation. Under multivariate normality D² follows a chi-square distribution, so α sets the cut-off in that distribution; MCD computes the same distance with a robust centre and covariance that outliers do not distort.

Everyday example: It is normal for the discount to grow with the number of items; a 400 discount on three items does not fit this relationship. The Mahalanobis distance of this receipt is large; the smaller α, the more only the most extreme combinations are flagged.

What it does: Threshold D² > χ²(p; 1 - α).

Effect on the result: Number of flags.

On the validation data: hbk Mahalanobis: 1 (14) for α = 0.0001 and 0.001; 2 (12, 14) for 0.05. MCD gives 1-14 at all three α. The classical D² is computed with the sample mean and covariance.

→ Mahalanobis distance

Grubbs/ESD α and k

Concept: Grubbs' test assumes normality and tests whether the single most extreme value is an outlier. Rosner's ESD test repeats this step by step for up to k outliers and is less prone than Grubbs to the masking trap when there are several outliers; α is the significance level at each step.

Everyday example: If a branch's daily takings over a month are normally distributed, Grubbs says whether the highest day is unusual. If the month may contain several odd days, ESD looks for up to k = 5 outliers.

What it does: Significance level of the Grubbs and Rosner ESD tests and the maximum number of outliers k sought by ESD.

Effect on the result: Number of outliers detected.

On the validation data: Rosner data: ESD 0 for α = 0.01, 3 for 0.05 (cases 52-54), 6 for 0.20; sequential Grubbs 0 for α = 0.05 (masking). The R_i and λ_i values of each step are reported.

→ Significance level (α)

Order/date column and period (time series method)

Concept: The time-series method first splits the series into trend and seasonal components and then looks for outliers in the remainder using the IQR rule. Periods that are high or low simply because of the season are therefore not flagged, only those that are unusual for their season; the ordering column and the period define this decomposition.

Everyday example: Supermarket takings rise every year in the days before a holiday; because the seasonal component explains this rise, it is not counted as an anomaly. If takings suddenly halve on an ordinary Tuesday, the remainder is large and that day is flagged; perhaps a till broke down.

What it does: Ordering column and seasonal period m in the STL + IQR method (0 = automatic from the date frequency). For sub-hourly frequency the daily cycle 1440/k is used (48 for 30 minutes).

Effect on the result: Only in this method, the STL remainders and flags; the order column has no effect in the other methods.

On the validation data: AirPassengers: 23 anomalies with the automatic period 12, 20 with m = 6. nyc_taxi (30-minute) period 48: 2049 anomalies (19.9 percent; STL, m = 48).

Isolation Forest trees and seed

Concept: Isolation Forest builds many trees that split the data at random; a point that can be separated from the rest in few splits is counted as an anomaly. The more trees, the more stable the scores; the seed is the starting point of the random splits, and the same seed gives the same result.

Everyday example: A transaction such as a 400 discount on a three-item basket ends up alone in a box after just a few random cuts, while an ordinary basket is isolated only after many cuts. If the auditor uses the same seed, the same list is obtained again.

What it does: Number of trees in the forest and the random seed.

Effect on the result: Scores; the flag set does not change in these data.

On the validation data: hbk: 50, 200 and 1000 trees, seeds 7 and 42: the same 4 flags (11-14). The same seed gives the same result; a different seed changes the scores.

→ Isolation forest and contamination rate

Additional output: Anomaly reason, Save to dataset

Concept: The reason explains because of which variable(s), and by how much, an observation was counted as an outlier. The save option adds flag and score columns to the data set, so that what is done with the outliers in later analyses (review, correction, exclusion, sensitivity analysis) is documented.

Everyday example: The list does not just tell the auditor 'receipt 57 is an outlier'; it says 'the discount is unusually high for the number of items'. With the flag column added, management can run its analyses with and without these receipts and see whether the conclusions change.

What it does: The reason gives the variable(s) that flagged the anomaly and the z value. Save adds anomaly_flag and, where available, score, distance, p and reason columns to the dataset.

Effect on the result: Only additional output.

On the validation data: hbk case 14: "X2 (z=+3.45)". The saved row count and values match the table.

Example on real data

Data: hbk, Hawkins, Bradu and Kass 1984, R robustbase

Variables: features X1, X2, X3 (artificial data; cases 1-14 are outliers)

  • Question: Which observations are outliers in the Hawkins-Bradu-Kass data (n = 75; X1, X2, X3)?
  • Method: IQR (Tukey fences, k = 1.5, type 7 quartiles); a row outside Q1 - 1.5 IQR or Q3 + 1.5 IQR on any variable is an anomaly.
  • Result: 14 anomalies (18.67%), cases 1-14; the same 14 rows lie outside the fences on each of the three variables (X1 fences -2.525 and 6.475).
  • Descriptive: Q1 = 0.85, Q3 = 3.10 for X1; 1.10 and 3.25 for X2; 0.90 and 3.00 for X3.
  • Limitation: The anomaly rate exceeds 10%; this set may be part of the data structure (a separate cluster). Classical z and Mahalanobis methods find only case 14 because of masking, while robust methods (MAD, MCD) find 1-14.
  • APA: By the IQR criterion (k = 1.5), 14 of 75 observations (18.7%) were flagged as outliers.

Related concepts

Median absolute deviation (MAD) and modified z-scoreInterquartile range (IQR) and Tukey fencesIsolation forest and contamination rateMahalanobis distanceLocal outlier factor (LOF)OutlierGrubbs' and generalized ESD testsMinimum covariance determinant (MCD) and robust distance

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.