Statistics Glossary → True Skill Statistic (TSS)
True Skill Statistic (TSS)analysis
A measure that summarises how skilfully a classifier or species distribution model separates presence from absence at a given threshold, using the formula sensitivity + specificity − 1.
Concept
Species distribution models learn from places where a species is present and absent and produce a suitability probability for every location. To use this probability, a threshold must be chosen to turn it into 'present' and 'absent' predictions; then the agreement between predictions and actual observations is measured. Simple accuracy can mislead: if the species occurs at only 10% of sites, a model that says 'absent' everywhere still achieves 90% accuracy.
The true skill statistic (TSS) avoids this by using sensitivity (the proportion of presence sites predicted correctly) and specificity (the proportion of absence sites predicted correctly) together: TSS = sensitivity + specificity − 1. It ranges from −1 to +1; 0 indicates no better than chance and 1 a perfect prediction. The formula is the same as the Youden index. Unlike Cohen's kappa, the TSS is largely unaffected by how common the species is (prevalence), which is why it is widely used to compare models for species of differing prevalence.
The TSS depends on the threshold; when the threshold changes, sensitivity and specificity change and so does the TSS. When the threshold is chosen and evaluated on the same data the value is optimistic; for a reliable assessment an independent test set or cross-validation should be used. The TSS weights false negatives and false positives equally; if the two errors have different costs in practice it may not be enough on its own. The AUC is reported alongside for threshold-independent discrimination.
When to use it
- When the threshold-dependent performance of species distribution models or binary classifiers is to be summarised in a single measure, especially when the classes are imbalanced.
- To assess threshold-independent discrimination use the ROC curve and AUC; to report all measures at a single threshold in detail (PPV, NPV, F1, MCC), use confusion matrix metrics.
- If agreement between prediction and observation beyond chance is of interest, Cohen's kappa can also be given; but kappa is sensitive to prevalence, which makes comparisons between species of differing prevalence harder.
An everyday example
A forestry office has built a model predicting where a rare orchid may grow. The orchid was found at 80 of 400 sampling points. At a suitable threshold, the model correctly predicts 64 of the orchid points (sensitivity 0.80) and correctly marks 256 of the 320 orchid-free points as 'absent' (specificity 0.80). TSS = 0.80 + 0.80 − 1 = 0.60, so the model is clearly better than chance. A model saying 'absent' at every point would reach 80% accuracy, but its TSS would be 0.
Parameters in MerQur
Features and Target (binary)
Concept: The target is the observed binary state of the species being present or absent; the features are the explanatory variables of the logistic model predicting this state. The predicted probability the model gives for each point is the score on which the TSS is computed.
Everyday example: In the orchid example the target is whether the orchid was found at each point, and the features are altitude, slope, soil moisture and shading. The logistic model uses them to compute an 'orchid presence probability' for each point.
What it does: Explanatory variables of the logistic model if no ready-made probability column is selected; the target is the binary observed status.
Effect on the result: The score is the logistic predicted probability (unpenalized maximum likelihood, IRLS).
On the validation data: UCLA binary (n = 400) in-sample maximum TSS 0.3087; at threshold 0.5, FP 20, TSS 0.1551, κ 0.1856.
Evaluation (On all data (in-sample) / Independent test set / k-fold cross-validation)
Concept: Determines on which data the TSS is computed. The value on all data is optimistic, because the model and threshold were chosen from the same data. An independent test set or k-fold cross-validation shows more honestly how well the model performs at new points.
Everyday example: The model might give TSS = 0.60 on all 400 points but TSS = 0.50 when fitted on 280 points and tested on the remaining 120. The success of predictions for a new valley will be closer to the second value.
What it does: In-sample (optimistic), independent test set (stratified split; model and threshold on the training part, TSS on the test part) or stratified k-fold CV (mean fold TSS ± SD, confusion matrix as the sum over folds).
Effect on the result: The reported TSS, Se, Sp and counts; in the additional card the CI, κ and AUC are computed from test or out-of-fold scores.
On the validation data: (UCLA): In-sample 0.3087; test set (0.30, seed 42) 0.2336; 5-fold CV 0.2853 ± 0.0218.
Test proportion, CV folds (k), Random seed
Concept: The test proportion sets how much of the data is set aside for testing, the number of folds how many parts the data are split into for cross-validation, and the random seed which random arrangement is used for the split. With small test sets the TSS can vary markedly from one split to another.
Everyday example: If only 10% of the 400 points (40 points, about 8 with orchids) are held out for testing, the correct or incorrect prediction of a single orchid can swing the TSS a lot. Repeating with different seeds shows how stable the result is.
What it does: Test set proportion; number of CV folds; seed for splitting, shuffling and bootstrap.
Effect on the result: Only in the relevant evaluation mode; the seed also affects the bootstrap CI. It has no effect in-sample, and the fields are disabled in an unrelated mode.
On the validation data: Test proportion 0.10, 0.30, 0.50 gives TSS 0.4302, 0.2336, 0.2477 (n_test 40, 120, 200); k = 2, 5, 10, 20 gives 0.2253, 0.2853, 0.2809, 0.2689 (SD 0.017, 0.022, 0.098, 0.196); seed 0, 42, 123456 (test set) gives 0.2632, 0.2336, 0.0789. The sampling variability of TSS is large in a small test set and in CV with many folds.
Ready-made probability column
Concept: Uses probabilities or suitability values already computed by another method (such as MaxEnt, random forest or GAM) as the score instead of fitting a logistic model, so that the TSS of different models can be compared with the same measure.
Everyday example: If a suitability map for the orchid has already been produced with MaxEnt, each point's suitability value goes into this column and the TSS is computed directly from these values.
What it does: Directly uses the predicted probability or suitability score of another model (MaxEnt, RF, GAM); no logistic model is fitted.
Effect on the result: The score is this column instead of a model; TSS, threshold, κ and AUC are computed directly from these probabilities.
On the validation data: ABCO GAM threshold 0.5: Se 0.5000, Sp 0.9474, TSS 0.4474, κ 0.4658, AUC 0.8809. See5: TSS 0.3461; SGB: 0.3292.
Positive class
Concept: Determines which value of the target counts as 'present' (positive). If the positive class changes, sensitivity and specificity swap places, while the TSS, which is based on their sum, stays the same.
Everyday example: If 'orchid present' is the positive class, sensitivity is the proportion of orchid points detected. If 'orchid absent' is chosen, the same figure appears as specificity, but TSS = 0.60 does not change.
What it does: Presence (positive) value; automatic = the second value in sort order. The ready-made probability column is taken to be the "probability of class 1"; if the other value is chosen as positive the score is taken as 1 - p and a note is written.
Effect on the result: TSS stays the same, sensitivity and specificity swap; with a ready-made probability column the score is taken as 1 - p.
On the validation data: UCLA logistic with positive 0: TSS the same (0.3087), Se/Sp swap. ABCO GAM with positive 0: TSS 0.6326 (the probability is converted to 1 - p for the selected positive class).
Threshold criterion (Max sensitivity+specificity (Youden) / Sensitivity = specificity / Predicted = observed prevalence / Max Kappa / Sensitivity >= 0.90)
Concept: Determines the rule for choosing the threshold that turns probabilities into 'present' and 'absent' predictions: maximising the sum of sensitivity and specificity (Youden, the threshold that maximises the TSS), equalising the two, matching predicted to observed prevalence, maximising kappa, or guaranteeing a given sensitivity.
Everyday example: A conservation study may want to miss almost none of the places the orchid might grow; the rule 'sensitivity ≥ 0.90' is then chosen. The threshold falls, 90% of orchid points are detected, but more orchid-free points also appear 'suitable'.
What it does: Chooses the threshold when there is no manual threshold (PresenceAbsence optimal.thresholds definitions).
Effect on the result: The threshold and all counts that depend on it. MerQur candidate thresholds are the observed score values (the highest in a tie); PresenceAbsence uses a grid of 101 thresholds between 0.00 and 1.00 and takes the mean in a tie; pROC uses the midpoint of adjacent values.
On the validation data: (QUGA GAM): Youden 0.3059 (TSS 0.7813); Se=Sp 0.1342 (TSS 0.7526); PredPrev 0.4432 (TSS 0.6707); MaxKappa 0.3059; ReqSens 0.0507. The candidate thresholds are the observed scores.
Manual threshold
Concept: Uses a fixed value set by the researcher as the threshold instead of choosing it by a rule; if the score is equal to or greater than this value, the prediction is 'present'.
Everyday example: If the field guidelines say 'places with suitability of 0.3 or more will be surveyed', the threshold is entered manually as 0.3 and the TSS is computed at this practice-specific threshold.
What it does: Fixed threshold (positive if score >= threshold); when automatic, the criterion is used.
Effect on the result: Replaces the threshold criterion; TSS, Se, Sp, κ and the counts are computed at this threshold.
On the validation data: ABCO GAM threshold 0.2: TSS 0.5811; 0.5: 0.4474; Youden (0.1001): 0.6326 (counts, κ and ASE are reported).
TSS CI + Kappa + AUC and TSS CI method (Wald / Bootstrap (stratified, 2000))
Concept: Gives a confidence interval for the TSS together with Cohen's kappa, which measures agreement between prediction and observation beyond chance, and the AUC, which shows threshold-independent discrimination. The interval can be computed by formula (Wald) or by resampling the data (bootstrap).
Everyday example: If the 95% interval for TSS = 0.60 is [0.50, 0.70], the model's skill is fairly precisely moderate to good. If the AUC is 0.88, the model ranks orchid and orchid-free points well whatever the threshold.
What it does: Wald CI for TSS (Var = Se(1-Se)/n1 + Sp(1-Sp)/n0; Youden 1950) or row bootstrap CI (1000, threshold fixed); Cohen κ (Fleiss-Cohen-Everitt ASE, CI, z with H0 SE); AUC (DeLong). In CV mode Wald is used even if bootstrap is selected (documented).
Effect on the result: The TSS point value does not change; the CI, κ and AUC are given in an additional card. The bootstrap CI depends on the seed.
On the validation data: (QUGA GAM Youden): TSS 0.7813 [0.6832, 0.8794], κ 0.7236 (ASE 0.0483), AUC 0.9381 [0.9033, 0.9729]. Bootstrap: PICO SGB [0.8720, 0.9370].
Example on real data
Data: UCLA OARC binary.csv (graduate admission, n = 400)
Variables: target admit (admission 0/1), logistic model predictors gre, gpa and rank
- Question: How skillfully does a logistic model with GRE, grade point average (gpa) and school rank (rank) predict graduate admission (admit) (TSS = sensitivity + specificity - 1)?
- Model: In the logistic regression rank (b = -0.56, p < .001), gpa (b = 0.78, p = .018) and gre (b = 0.002, p = .036) are significant; the predicted probability is used as the score (all data, n = 400).
- Result: At the Youden threshold 0.335, TSS = 0.309; sensitivity = .646, specificity = .663. Confusion matrix: TP = 82, FN = 45, FP = 92, TN = 181.
- Interpretation: TSS = 0.31 is above the chance level (0) but shows limited skill (acceptable); the model captures two thirds of the admitted applicants and counts one third of the rejected ones as false positives.
- Limitation: The evaluation is in-sample and the threshold is selected on the same data, so TSS is optimistic; it should be recomputed with an independent test set or k-fold cross-validation.
- APA: TSS = .31, sensitivity = .65, specificity = .66, N = 400 (threshold = 0.335).
Related concepts
Train-test split (holdout)Kappa statistic (Cohen's κ)Classification threshold (cut-off)Area under the ROC curve (AUC)Youden's J index (optimal cut-off)k-fold cross-validationSensitivity and specificityDeLong test and AUC confidence interval
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.