Statistics Glossary → ROC Curve
ROC Curveanalysis
A curve showing how well a continuous test value or model probability distinguishes two states (ill/healthy, event/no event) across all possible thresholds, and the area under the curve (AUC) that summarises this discrimination in a single number.
Concept
Many diagnostic tests and prediction models produce not a 'yes/no' but a continuous value: the level of a marker in the blood, a risk probability computed by a model. Deciding above which threshold the value counts as 'positive' creates a trade-off between sensitivity (the proportion of ill people detected) and specificity (the proportion of healthy people correctly ruled out): lowering the threshold catches more ill people but also wrongly flags more healthy ones. The ROC curve plots sensitivity (vertical axis) against 1 − specificity (horizontal axis) for every possible threshold.
The area under the curve (AUC) is the probability that a randomly chosen positive individual has a higher test value than a randomly chosen negative one. AUC = 0.5 means the test is no better than tossing a coin and AUC = 1 means perfect discrimination; as a rough guide, 0.7-0.8 is considered acceptable, 0.8-0.9 good and above 0.9 excellent. The AUC does not depend on a threshold; the threshold used in practice is chosen with a criterion such as the Youden index (the point maximising sensitivity + specificity − 1) or according to the relative costs of false negatives and false positives.
The AUC measures only discrimination; it does not show whether the predicted probabilities are accurate (calibration). When the threshold and AUC are chosen and evaluated on the same data the results are optimistic, so validation on independent data or a bootstrap optimism correction is needed. When the positive class is very rare, the AUC can look high while most positive predictions are wrong; the precision-recall (PR) curve should then be examined too. The AUCs of two tests measured on the same patients are compared with the paired DeLong test.
When to use it
- When you want to assess how well a continuous test value or a model's predicted probability separates two states and to choose a suitable cut-off (diagnostic tests, biomarkers, risk scores, classification models).
- If the threshold is already fixed and only performance at that threshold is to be reported, use confusion matrix metrics; for summarising threshold-dependent skill in species distribution models, use the TSS.
- If the classes are highly imbalanced and correctly predicting positives matters, add the PR curve and average precision to the ROC; if the accuracy of the probabilities matters too, assess calibration plots or the Hosmer-Lemeshow test.
An everyday example
A paediatric emergency department examines whether the blood CRP level can identify bacterial infection in children with fever. Of 300 children, 60 are later confirmed to have a bacterial infection. If the ROC analysis gives AUC = 0.84, a randomly chosen infected child has a higher CRP value than a randomly chosen uninfected child with probability 84%. The Youden criterion might suggest a threshold of 40 mg/L: at this threshold 80% of infections are detected and 75% of uninfected children are correctly ruled out. If doctors consider not missing an infection more important, they may choose a lower threshold.
Parameters in MerQur
Features and Target (binary)
Concept: The test variable is the continuous measurement whose discrimination is assessed; the target is the binary variable showing the true state. When several test variables are chosen they can be combined in a logistic model, and the model's predicted probability is used as the score.
Everyday example: In the CRP example the test variable is the CRP level and the target is whether bacterial infection was confirmed. If duration of fever and white cell count are added to CRP, the three are combined in a logistic model and the ROC curve of a single risk probability is drawn.
What it does: Test variable(s) and the binary status variable. With a single variable the score is the variable itself; with several variables, in "combined" mode the predicted probability of a logistic model is the score.
Effect on the result: The score, and therefore the AUC, SE, threshold and Se/Sp, depend on this choice; with several variables the combined mode fits an unpenalized maximum likelihood (ML) logistic model.
On the validation data: aSAH s100b AUC 0.7314, DeLong SE 0.0517, 95% CI [0.6301, 0.8326], Youden threshold 0.205 (Se 0.6341, Sp 0.8056). Combined model (s100b + ndka + wfns + age): unpenalized ML logistic, AUC 0.8621; in the remission data step 2 c 0.8735.
Positive class (Automatic / 0 / 1)
Concept: Determines which value of the target counts as 'positive' (the condition being sought). Sensitivity and specificity are defined by this choice; if the positive class changes, the two swap places, while the AUC stays the same.
Everyday example: If 'infection present' is the positive class, sensitivity is the proportion of infected children detected. If 'no infection' is chosen as positive, the same figure appears as specificity.
What it does: Value of the positive status; automatic = the second value in sort order (Poor for Good/Poor; 1 for 0/1).
Effect on the result: With the automatic direction the AUC stays the same; Se/Sp and the threshold direction change.
On the validation data: aSAH with positive Good: direction ">" (smaller value positive), AUC 0.7314, Youden Se 0.8056 and Sp 0.6341 (swapped). With positive 0, alb, totscore, tp AUC 0.7366, 0.7258, 0.6478.
Test direction (Automatic / Larger value positive / Smaller value positive)
Concept: Determines whether larger or smaller test values indicate the positive state. If the direction is reversed, the curve falls below the diagonal and the AUC comes out as 1 − AUC.
Everyday example: Because CRP rises with infection, 'larger value positive' is the correct direction. Some markers fall with disease; low blood albumin, for instance, may indicate a poor course, and 'smaller value positive' should then be chosen.
What it does: Which direction of the raw test variable is counted as positive; automatic = larger value positive if median(negative) <= median(positive) (the pROC direction = "auto" rule). Ignored for the combined logistic score.
Effect on the result: The AUC and the threshold direction change: when the reverse direction is forced, the AUC becomes 1 - AUC.
On the validation data: s100b automatic direction is the same as "<"; when ">" is forced, AUC 0.2686, CI [0.1674, 0.3699], maximum Youden 0.0244.
Multiple variables (One combined logistic-model curve / Separate curve per test variable (+ paired DeLong))
Concept: Determines whether several test variables are combined and assessed with one curve or each is assessed with its own curve. With separate curves, the AUCs of tests measured on the same individuals are compared with the paired DeLong test, which accounts for the measurements being correlated.
Everyday example: If CRP and procalcitonin were measured in the same children, separate curves are drawn and 'which test better identifies infection?' is answered with the paired DeLong test. The combined model addresses 'are the two together better than either alone?'.
What it does: Combined logistic-model score, or a separate curve for each variable with paired DeLong tests.
Effect on the result: Combined mode gives one curve and one AUC; separate mode gives one curve per variable and a paired DeLong z and p for each pair.
On the validation data: aSAH separate curves: wfns - s100b z 2.2090, p 0.0272; wfns - ndka z 2.7978, p 0.0051; s100b - ndka z 1.3908, p 0.1643. CT2: mod1 - mod2 z² = 2.3146. In the combined model the threshold is reported as an observed probability value.
AUC CI method (DeLong / Bootstrap (stratified, 2000) / Hanley-McNeil)
Concept: Determines how the confidence interval of the AUC is computed. The DeLong method is the standard approach that needs no distributional assumption; the bootstrap builds the interval by resampling the data; Hanley-McNeil is a simple approximate formula.
Everyday example: For AUC = 0.84, the DeLong method might give an interval such as [0.79, 0.89]. The bootstrap draws 2000 new 'groups of children' from the same data, computes the AUC in each and produces a similar interval; the result may vary slightly with the random seed.
What it does: DeLong; stratified percentile bootstrap (2000, seed 42); Hanley-McNeil (1982) negative exponential approximation Q1 = A/(2-A), Q2 = 2A²/(1+A).
Effect on the result: The AUC does not change; the SE and the confidence interval change. The bootstrap result depends on the seed.
On the validation data: (s100b): DeLong [0.6301, 0.8326]; Hanley [0.6309, 0.8318] (SE 0.0512); bootstrap [0.6175, 0.8288] (mean lower limit over 30 seeds 0.6268; the difference is seed-dependent Monte Carlo variation). In the Hanley rating data the Hanley SE is 0.0325 (Q1 and Q2 are computed from the AUC with the Hanley-McNeil formula).
Confidence Level (90% / 95% / 99% / custom)
Concept: Sets the confidence level of the intervals for the AUC and the measures at the threshold; the higher the level, the wider the interval.
Everyday example: If the 95% interval for the AUC is [0.79, 0.89], at 99% it widens to something like [0.77, 0.91]. As long as the lower limit stays above 0.5, CRP can be said to discriminate better than chance.
What it does: Level of the additional CIs and the Wilson CIs in the cut-off detail (the main 95% DeLong row does not change).
Effect on the result: The AUC and the point estimates do not change; only the CI width changes.
On the validation data: (s100b DeLong): 80% [0.6652, 0.7976]; 90% [0.6464, 0.8163]; 97.5% [0.6156, 0.8472]; 99% [0.5983, 0.8644]; 99.9% [0.5614, 0.9014].
Cut-off criterion, Cost ratio (FN/FP), Target sensitivity/specificity
Concept: Determines how the threshold for practical use is chosen. The Youden index maximises the sum of sensitivity and specificity; the cost ratio accounts for the relative price of false negatives and false positives; a target sensitivity or specificity guarantees that one of them does not fall below a set level.
Everyday example: If missing an infection is judged four times worse than giving an unnecessary antibiotic, a cost ratio of 4 is entered and the threshold moves down. If doctors say 'we must catch at least 95% of infections', a target sensitivity of 0.95 is chosen and the threshold with the highest specificity meeting this condition is found.
What it does: Youden J; closest point to (0,1); weighted Youden (by cost and prevalence, r = (1 - prev)/(cost·prev), pROC best.weights); sensitivity >= target (highest specificity) or specificity >= target (highest sensitivity). Cost is effective only for the closest-point and weighted criteria, and the target only for the sensitivity/specificity criteria.
Effect on the result: The AUC does not change; the selected threshold and the Se, Sp, PPV and NPV at that threshold change.
On the validation data: (s100b): Closest point 0.205; weighted (cost 1) 0.51 (Se 0.2927, Sp 1.0), with cost 4 it is 0.065 (Se 0.9756), with 0.25 it is 0.51, with 100 it is -infinity (all positive); Se >= 0.90 gives 0.075 (Sp 0.2222), Se >= 0.50 gives 0.29, Se >= 0.99 gives -infinity; Sp >= 0.90 gives 0.435 (Se 0.3902).
Additional output: PPV/NPV at the threshold and CI
Concept: The positive predictive value (PPV) is the probability that someone testing positive is truly positive; the negative predictive value (NPV) is the probability that someone testing negative is truly negative. Unlike sensitivity and specificity, they depend on how common the condition is (prevalence).
Everyday example: If, at the 40 mg/L threshold, 44% of children with a high CRP really have an infection, the PPV is 0.44; if 94% of those with a low CRP are free of infection, the NPV is 0.94. In a general practice setting where infection is rarer, the same test gives a lower PPV.
What it does: Se, Sp, PPV, NPV and Wilson CIs at the selected threshold.
Effect on the result: Adds a threshold-specific table; the AUC and the threshold selection do not change.
On the validation data: s100b Youden: TP 26, FP 14, TN 58, FN 15; Se 0.6341 [0.4812, 0.7641], PPV 0.65 [0.4951, 0.7787] (Wilson interval without continuity correction). Hanley data ">= 4" threshold Se 86.27%, Sp 77.59%.
Partial AUC (specificity >=)
Concept: The area under the ROC curve in the high-specificity region only. The full AUC weights all thresholds equally; if in practice only the region with few false positives matters, the partial AUC shows performance in that region separately.
Everyday example: If a screening programme must correctly rule out at least 90% of healthy children, only the part of the curve where specificity exceeds 0.90 matters. Two tests with similar full AUCs may turn out to differ clearly in this region.
What it does: Area under the ROC in the specificity range [lower, 1] (linear interpolation) and the McClish (1989) correction; 0 = off.
Effect on the result: Adds the area of the high-specificity region and its corrected value next to the full AUC; the full AUC does not change.
On the validation data: wfns specificity >= 0.90: 0.03344, corrected 0.64969; ndka 0.01070 and 0.53002; s100b specificity >= 0.80: 0.0806 and 0.6683.
Optimism correction (bootstrap)
Concept: When a model is both fitted and evaluated on the same data, the AUC comes out too high. The bootstrap optimism correction refits the model on resamples to estimate the size of this optimism and subtracts it from the apparent AUC.
Everyday example: A model built from CRP, duration of fever and white cell count might give AUC = 0.88 on its own data; if the corrected value is 0.85, the discrimination to expect in new children is closer to 0.85.
What it does: Harrell bootstrap optimism correction in the combined logistic model (B samples, seed 42); not applied with a single variable, and this is stated in the output.
Effect on the result: The bootstrap optimism is subtracted from the apparent AUC to give a corrected AUC; the result depends on the seed and on B.
On the validation data: aSAH B = 200: apparent 0.8621, optimism 0.0198, corrected 0.8423; B = 2000: 0.0176; remission (n = 27) B = 200: 0.0415. The optimism carries seed-dependent Monte Carlo variation.
Additional output: PR curve and average precision (AP)
Concept: The precision-recall curve plots, at each threshold, how many positive predictions are correct (precision) against the proportion of positives detected (recall). Average precision (AP) summarises the curve; its chance level equals the proportion of positives. It is more informative than the ROC when positives are rare.
Everyday example: If infection occurred in only 5% of children, CRP could have a high AUC while most children testing positive were uninfected. The PR curve shows this plainly and lets the AP be compared with the 5% chance level.
What it does: Precision-recall curve and average precision as a step sum.
Effect on the result: Does not change the ROC results; adds a summary (AP) that is useful with imbalanced classes.
On the validation data: s100b AP 0.6856 (chance level 0.3628).
Example on real data
Data: aSAH (subarachnoid hemorrhage), R pROC package (Robin et al., 2011)
Variables: target outcome (Good/Poor clinical outcome), test variable s100b (biomarker)
- Question: Can the S100B biomarker (s100b) separate a poor clinical outcome (Poor) from a good outcome (Good) in patients with subarachnoid hemorrhage?
- Result: AUC = 0.731 (DeLong SE = 0.052), 95% CI [0.630, 0.833]; z = 4.48, p < .001 for H0: AUC = 0.5. The discrimination is acceptable.
- Cut-off: The threshold that maximizes Youden J is s100b > 0.205; at this threshold sensitivity = .634 and specificity = .806 (1 - specificity = .194).
- Descriptive: n = 113 (41 poor, 72 good outcomes); a high s100b value is associated with a poor outcome (automatic direction: larger value positive).
- Limitation: Because the threshold is selected on the same data, it is optimistic; with sensitivity around 63% the marker alone is insufficient for screening and should be evaluated together with clinical variables.
- APA: AUC = .73, 95% CI [.63, .83], N = 113 (n+ = 41, n- = 72); Youden threshold 0.205 (sensitivity = .63, specificity = .81).
Related concepts
Area under the ROC curve (AUC)DeLong test and AUC confidence intervalPartial AUCYouden's J index (optimal cut-off)Positive and negative predictive values (PPV, NPV)Bootstrap optimism correctionPrecision-recall curve and average precisionClassification threshold (cut-off)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.