Statistics Glossary → Confusion Matrix Metrics
Confusion Matrix Metricsanalysis
An analysis that computes performance measures such as accuracy, sensitivity, specificity, predictive values, F1, MCC and kappa from the confusion matrix, which cross-tabulates a classifier's or diagnostic test's predictions against the true classes.
Concept
When a test or model predicts 'positive' or 'negative' there are four possible outcomes: correctly detecting a true positive (true positive, TP), missing a true positive (false negative, FN), wrongly calling a true negative positive (false positive, FP) and correctly ruling out a true negative (true negative, TN). The 2×2 table formed by these four counts is the confusion matrix, and every performance measure is derived from it.
The measures answer different questions. Sensitivity, TP / (TP + FN), shows how many true positives are detected; specificity, TN / (TN + FP), how many true negatives are correctly ruled out. The positive predictive value (PPV), TP / (TP + FP), shows how many positive predictions are correct; the negative predictive value (NPV) says the same for negative predictions. Unlike sensitivity and specificity, PPV and NPV depend strongly on how common the condition is (prevalence). Likelihood ratios (LR+, LR−) show how much a test result changes the probability of disease, and the diagnostic odds ratio summarises the test's overall discrimination in a single number.
With imbalanced classes accuracy is misleading: if a disease has 1% prevalence, a test calling everyone 'healthy' is 99% accurate. Balanced accuracy (the mean of sensitivity and specificity), F1 (the harmonic mean of PPV and sensitivity), the Matthews correlation coefficient (MCC; a measure using all four cells and ranging from −1 to +1) and Cohen's kappa, which measures agreement beyond chance, should therefore be reported alongside. All these measures describe performance at a particular threshold; if the threshold changes, so do the values.
When to use it
- When the performance of a classifier or diagnostic test at a given threshold is to be reported in detail (diagnostic test validation, machine-learning model evaluation, quality control).
- If the threshold is not yet fixed and the test's discrimination across all thresholds is to be studied, use the ROC curve and AUC; to summarise threshold-dependent skill of species distribution models in one measure, use the TSS.
- When comparing the results of two tests or two raters on the same individuals, use the McNemar test and Cohen's kappa; with more than two raters, agreement measures such as Fleiss' kappa are preferred.
An everyday example
A company's email filter examines 1000 messages, 100 of which are genuinely unwanted (spam). The filter catches 85 spam messages (TP), misses 15 (FN), wrongly flags 45 normal messages as spam (FP) and correctly lets 855 normal messages through (TN). Accuracy is 94%, sensitivity 85% and specificity 95%. Yet only 85 of the 130 messages in the spam folder are really spam (PPV ≈ 65%), so a user who never checks the spam folder may miss some important messages. MCC ≈ 0.71 and F1 ≈ 0.74 summarise the filter's success in this imbalanced situation more realistically than accuracy.
Parameters in MerQur
Target (actual) and Prediction
Concept: The target holds each observation's true class and the prediction the class given by the test or model. The confusion matrix is the cross-tabulation of these two columns; rows with missing values are not counted.
Everyday example: In the email example the target column holds whether each message really is spam (labelled by hand) and the prediction column the filter's decision. Comparing these two columns for the 1000 messages gives the TP, FN, FP and TN counts.
What it does: Determines the actual and predicted class columns; rows with a missing actual or predicted value are dropped.
Effect on the result: Binary reduction relative to the positive class (1 if the actual or predicted value equals the positive value, otherwise 0; one-vs-rest in the multi-class case).
On the validation data: (FOB): TP 20, FN 10, FP 180, TN 1820; accuracy 0.9064, Se 0.6667, Sp 0.91, PPV 0.10, NPV 0.9945, F1 0.1739, MCC 0.2335, BA 0.7883, κ 0.1521, J 0.5767.
Positive class
Concept: Determines which class counts as 'positive' (the condition being sought). If the positive class changes, sensitivity and specificity swap, as do PPV and NPV, and positive-class measures such as F1 change; symmetric measures such as accuracy, kappa and MCC stay the same.
Everyday example: If 'spam' is the positive class, sensitivity is the proportion of spam caught (85%). If 'normal message' is positive, sensitivity becomes the proportion of normal messages correctly let through (95%), and F1 comes out much higher because normal messages are the majority.
What it does: The class that TP/FN count as "positive".
Effect on the result: Se and Sp, and PPV and NPV, swap; LR+/LR-, prevalence, detection rates and F1 change. Accuracy, κ, MCC, BA, J and DOR do not change.
On the validation data: (FOB, positive 1 and 0): Se 0.6667 and 0.91; PPV 0.10 and 0.9945; F1 0.1739 and 0.9504; LR+ 7.4074 and 2.73; LR- 0.3663 and 0.135; κ 0.1521 and DOR 20.2222 are the same in both.
Confidence Level (90% / 95% / 99% / custom)
Concept: Sets the level of the confidence intervals given for the measures. Proportions such as sensitivity or PPV rest on a limited number of observations and carry uncertainty; the higher the confidence level, the wider the interval.
Everyday example: Because sensitivity is computed from 100 spam messages, its 95% confidence interval runs from about 77% to 91%. With only 20 spam messages, the same 85% sensitivity would have a much wider interval.
What it does: Sets the level for the Wilson intervals (accuracy, Se, Sp, PPV, NPV, prevalence), the LR+/LR-/DOR log intervals, the κ interval, the exact accuracy interval and the bootstrap percentiles.
Effect on the result: Only the interval width; point estimates and p values do not change.
On the validation data: (FOB, Se interval): 80% [0.5503, 0.7658]; 90% [0.5166, 0.7892]; 95% [0.4878, 0.8077]; 99% [0.4336, 0.8394]; 99.9% [0.3757, 0.8692]; LR+ 95% [5.549, 9.888].
Additional output: Kappa SE/CI + McNemar + NIR test
Concept: The standard error and confidence interval of kappa show whether agreement differs from chance. The McNemar test checks whether the two kinds of error (false negatives and false positives) are balanced in number, i.e. whether the test leans systematically in one direction. The NIR test checks whether accuracy exceeds what would be achieved by predicting the majority class for everyone (the no-information rate).
Everyday example: Because 90% of messages are normal, a filter calling everything 'normal' would be 90% accurate; the NIR test says whether the 94% accuracy is significantly above this level. The McNemar test shows whether the imbalance between the filter's 45 false alarms and 15 misses can be explained by chance.
What it does: The asymptotic SE of κ (Fleiss-Cohen-Everitt 1969) and its interval, z and p for H0: κ = 0; McNemar (systematic error; continuity-corrected χ² and exact binomial p; b = FN, c = FP; the uncorrected χ² is also given); the Clopper-Pearson interval for accuracy and the one-sided binomial test of H1: accuracy > NIR.
Effect on the result: The matrix and basic metrics do not change; the SE, CI and z test of κ, the McNemar χ² and p, the exact accuracy interval and the NIR test are added.
On the validation data: Wikipedia example κ = 0.4000, ASE 0.1270, z 2.887; Drugs A-F data κ -0.0328, McNemar χ² 9.389 (corrected); uncorrected 10.8889, p 0.00097; the exact p is also given; NIR 0.75 and the p testing whether accuracy exceeds the NIR are reported.
Additional output: Multi-class matrix
Concept: With more than two classes, builds the confusion matrix with all classes and computes measures for each class by comparing it with all the others. The macro average gives each class equal weight; micro and weighted averages give more weight to large classes.
Everyday example: If the filter sorts messages into four classes, 'work', 'personal', 'advertising' and 'spam', separate sensitivity and PPV are computed for each. Macro F1 reveals failure in the rare but important 'personal' class more clearly than micro F1.
What it does: Confusion matrix with all classes; one-vs-rest Se, Sp, PPV, NPV, F1, BA per class; macro, micro and weighted F1; multi-class κ (ASE, interval); accuracy interval and NIR test.
Effect on the result: All classes are reported instead of the binary reduction; per-class metrics and macro, micro and weighted averages are added.
On the validation data: SkinCondition (4 classes) κ 0.3449, ASE 0.0724, interval [0.2030, 0.4868]; per-class metrics are reported separately.
Additional output: F1/MCC/BA/Youden bootstrap CI
Concept: Because simple formula-based standard errors are not common for measures such as F1, MCC, balanced accuracy and the Youden index, their confidence intervals are computed by resampling the data (bootstrap). The result may vary slightly with the random seed.
Everyday example: Thousands of new 'message sets' are drawn with replacement from the 1000 messages and the MCC is recomputed in each. The MCC wandering between 0.64 and 0.78 across these sets shows the uncertainty of the value 0.71.
What it does: Percentile bootstrap interval (2000 repetitions, seed 42; a sample in which a class is empty is skipped). A closed-form SE is not common for F1 and MCC.
Effect on the result: Point values do not change; intervals are added for F1, MCC, BA and J, and the result depends on the seed.
On the validation data: (FOB): F1 [0.109, 0.242], MCC [0.154, 0.307]; the intervals carry seed-dependent Monte Carlo variation.
Example on real data
Data: Wikipedia, faecal occult blood (FOB) screening example (n = 2030)
Variables: actual class gercek (cancer present/absent), prediction tahmin (FOB test result)
- Question: How accurately does the faecal occult blood (FOB) test detect colorectal cancer (actual = 1) in 2030 people?
- Matrix: TP = 20, FN = 10, FP = 180, TN = 1820; prevalence 1.48% (30/2030), the classes are highly imbalanced.
- Result: Accuracy = .906 [.893, .918], sensitivity = .667 [.488, .808], specificity = .910 [.897, .922]; the test detects two thirds of the cancers.
- Meaning of a positive result: PPV = .10 [.066, .149], NPV = .995; only 1 in 10 people with a positive result has cancer, while a negative result largely rules out cancer. LR+ = 7.41 [5.55, 9.89], LR- = 0.37.
- Imbalance-sensitive measures: F1 = .17, MCC = .23, balanced accuracy = .79, Cohen κ = .15 (slight agreement); the high accuracy comes largely from the size of the negative class, so MCC and balanced accuracy should be used.
- APA: FOB test (N = 2030): sensitivity = .67, 95% CI [.49, .81]; specificity = .91, 95% CI [.90, .92]; PPV = .10; MCC = .23; κ = .15.
Related concepts
Likelihood ratio test (LRT)Bootstrap (resampling)F1 score, MCC and balanced accuracyKappa statistic (Cohen's κ)Positive and negative predictive values (PPV, NPV)Classification table (confusion matrix)Youden's J index (optimal cut-off)Sensitivity and specificity
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.