HABITUSReference Book

7. Evaluation (④ Evaluation)

Tab: ④ Evaluation · Code: tabs/tab_evaluation.py (UI); sdm_core.py evaluate_all, boyce_ecospat, compute_calibration, cv_roc_curves, cv_omission_pr_calibration, cv_boyce_curves, cv_threshold_performance, compute_response_curves, evaluation_scores_table (computation)

7.1 What this tab does

④ Evaluation shows the performance of the models built in ③ Models in charts and tables. It does not build new models or re-split the data: all curves and scores are produced from the out-of-fold (CV hold-out) predictions saved during the ③ Models run (sdm_core.py:5547-5556, 6826-7037). Only two things are computed from the full-data (Full) models: the response curves (§7.7) and the map thresholds in the Classification Thresholds table (§7.9).

7.2 Screen layout

Figure 7.1. ④ Evaluation tab with the Model Scores sub-tab open: the layer name guide at the top, seven sub-tabs in the middle, ROC, TSS and Boyce panels with Group by = Algorithm and Chart = Scatter + range (including the EMca and EMwmean ensemble rows), and the Refresh All Charts, Save All Charts to Output Folder and Save Scores (CSV) buttons at the bottom.

Figure 7.1. ④ Evaluation tab with the Model Scores sub-tab open: the layer name guide at the top, seven sub-tabs in the middle, ROC, TSS and Boyce panels with Group by = Algorithm and Chart = Scatter + range (including the EMca and EMwmean ensemble rows), and the Refresh All Charts, Save All Charts to Output Folder and Save Scores (CSV) buttons at the bottom.
Figure 7.1. ④ Evaluation tab with the Model Scores sub-tab open: the layer name guide at the top, seven sub-tabs in the middle, ROC, TSS and Boyce panels with Group by = Algorithm and Chart = Scatter + range (including the EMca and EMwmean ensemble rows), and the Refresh All Charts, Save All Charts to Output Folder and Save Scores (CSV) buttons at the bottom.
Area Content In this chapter
Top strip Layer name guide: meaning of the layer names (RF_PA1, EMwmean, _prob, _bin …) §7.2
Model Scores ROC, TSS, Boyce distributions §7.3
ROC Curves Out-of-fold ROC curves §7.4
Omission & PR Omission, precision-recall, calibration §7.5
Variable Importance Permutation importance §7.6
Response Curves Response curves (marginal / partial / ice) §7.7
Boyce Detail Boyce P/E curves per fold §7.8
Classification Thresholds Thresholds of eleven threshold rules and their out-of-fold performance §7.9
Bottom buttons ↻ Refresh All Charts, 💾 Save All Charts to Output Folder, 💾 Save Scores (CSV) §7.10

In the sub-tab strip, the name of the third sub-tab appears on screen as Omission PR; in the code its name is Omission & PR (tabs/tab_evaluation.py:201). This chapter uses the name in the code.

Multi-row figures (scores, ROC, omission, response, Boyce) sit in their own scroll area; if the figure continues below, the hint ▼ Scroll down for more of this figure appears (tabs/tab_evaluation.py:69-101).

7.3 Model scores (Model Scores)

This sub-tab (Figure 7.1) plots the ROC, TSS and Boyce columns of the evaluation_scores.csv table in three panels (tabs/tab_evaluation.py:229-353). The table itself contains every metric; the metrics are defined in §7.3.2.

Name in the UI What it does Default Options Code
Group by: Column used to group the points Algorithm Algorithm / PA Set / CV Run tabs/tab_evaluation.py:233-235
Chart: Chart type Scatter + range Scatter + range / Box plot / Bar mean±SD tabs/tab_evaluation.py:237-239
Ensemble note Writes the hold-out scores and fold groups of the ensemble rows — — tabs/tab_evaluation.py:261-281

7.3.1 Chart and grouping

What it does. Scatter + range draws each row as a point, the group's minimum–maximum range as a line and the mean as a diamond; Box plot draws a box plot; Bar mean±SD draws a mean and standard deviation bar (tabs/tab_evaluation.py:315-331). The ROC and TSS panels have dashed lines at 0.8 (green) and 0.6 (orange), the Boyce panel lines at 0.5 and 0. If there are more than six groups, the labels are written vertically.

When and how to set it. Algorithm compares algorithms; PA Set shows the difference between pseudo-absence sets (the effect of background sampling); CV Run shows the difference between folds (whether some folds are systematically difficult).

Effect on results. Changes only the view; the table and files do not change.

Caution. The chart uses all rows of the table: CV runs, Full rows carrying the CV mean, and ensemble rows (PA Set = Ensemble). Because Full rows are copies of the mean, in Algorithm grouping one extra mean point per PA set is added to an algorithm's distribution; when reading the spread, rely on the Score_source = cv_fold rows in evaluation_scores.csv. The lower axis limits are 0.3 for ROC and TSS and −0.15 for Boyce (tabs/tab_evaluation.py:314); values below these are not visible in the panel.

7.3.2 Metrics in the score table

Each row is computed by evaluate_all from a model's predictions (sdm_core.py:1122-1257). Threshold-dependent metrics are computed at the threshold found with the threshold rule and threshold source selected in ③ Models (train_cv by default) (Chapter 6, §6.5). TP, FN, FP, TN are the cells of the confusion matrix; if a denominator is zero, the metric becomes NaN, not 0 (sdm_core.py:693-719).

Column Definition Threshold-dependent Code
ROC Area under the ROC curve (AUC) no sdm_core.py:951-959
TSS Sensitivity + specificity − 1 (Allouche et al. 2006) yes sdm_core.py:671-679
Boyce Continuous Boyce index (ecospat or legacy definition) no sdm_core.py:475-664
Threshold Binary threshold used — sdm_core.py:1242
Prevalence, N_pres, N_abs Proportion and numbers of presences in the evaluation set no sdm_core.py:1249-1252
PredPresFrac Proportion of rows above the threshold yes sdm_core.py:1235-1236
Brier \overline{(p-y)^2}; smaller is better no sdm_core.py:998
CalSlope Slope of the logistic regression of y on \mathrm{logit}(p); 1 good calibration, <1 overconfident (Pearce and Ferrier 2000) no sdm_core.py:1000-1013
Sensitivity, Specificity TP/(TP+FN), TN/(TN+FP) yes sdm_core.py:709-710
Kappa Cohen's kappa (Fielding and Bell 1997) yes sdm_core.py:711-712
MCC Matthews correlation coefficient yes sdm_core.py:713-714
Jaccard, Sorensen, FPB TP/(TP+FP+FN); 2TP/(2TP+FP+FN); 2 × Jaccard (Li and Guo 2013) yes sdm_core.py:715-719
OR_MTP, OR_10p Proportion of test presences below the lowest / 10th percentile prediction of the training presences depends on reference threshold sdm_core.py:1095-1099
AUC_train, dAUC AUC on the training data; AUC_train − ROC (overfitting gap) no sdm_core.py:1074-1084
OR_MTP_degenerate, OR_10p_degenerate 1 if the omission threshold is equal to or below the smallest prediction (omission is 0 by construction) — sdm_core.py:1100-1114
Threshold_source, OR_reference, Boyce_formula Source of the threshold, the omission reference and the Boyce definition — sdm_core.py:5877-5879
Score_source cv_fold, cv_mean, random_split, cv_holdout — sdm_core.py:1038-1053

Score_source values: cv_fold is the score of a CV run on its own held-out fold; cv_mean is the CV mean carried over to the full-data model; random_split is the random-split score of a full-data model for which no CV score was produced (optimistic); cv_holdout is an ensemble row (Chapter 6, §6.6.7).

Note: compute_calibration also computes the calibration intercept (CalIntercept) and the maximum and expected calibration errors (MCE, ECE) (sdm_core.py:981-1033); these three values are not written to evaluation_scores.csv.

A row from the example run (GLM, PA1, RUN1): ROC 0.941, TSS 0.865, Boyce 0.399, threshold 0.177, prevalence 0.148 (43 presences, 248 absences), sensitivity 0.953, specificity 0.911, calibration slope 1.28, AUC_train 0.900 (dAUC = −0.041). In the RUN2 fold of the same algorithm, ROC is 0.790 and TSS 0.457; the difference between folds is larger than the difference between algorithms.

7.4 ROC curves (ROC Curves)

Figure 7.2. ROC Curves sub-tab: a separate panel for each algorithm and ensemble rule; thin lines are individual folds, the thick line is the mean curve, the shading is the ±1 SD band.

Figure 7.2. ROC Curves sub-tab: a separate panel for each algorithm and ensemble rule; thin lines are individual folds, the thick line is the mean curve, the shading is the ±1 SD band.
Figure 7.2. ROC Curves sub-tab: a separate panel for each algorithm and ensemble rule; thin lines are individual folds, the thick line is the mean curve, the shading is the ±1 SD band.

What it does. For each algorithm, one ROC curve is drawn from the out-of-fold predictions of each PA set × fold group; the curves are interpolated onto a common 101-point false positive rate grid, and the mean curve and ±1 SD band are computed; the legend has the form AUC = mean ± SD (n = number of fold groups) (cv_roc_curves, sdm_core.py:6826-6891; tabs/tab_evaluation.py:372-435). Ensemble rules (EMwmean (ensemble) …) are drawn from the ensemble's hold-out surfaces (Chapter 6, §6.6.7). There are no settings.

Effect on results. Screen and evaluation_roc_curves.png; the scores do not change. In the example run, AUC over eight fold groups: GAM 0.872 ± 0.050, GLM 0.889 ± 0.047, MaxEnt 0.895 ± 0.051.

Caution. A single-class hold-out is skipped; an algorithm with no out-of-fold predictions at all is listed below the figure with the note Not shown: … (sdm_core.py:6872-6876; tabs/tab_evaluation.py:427-429). Because the absence class consists of pseudo-absences, AUC is the ability to "discriminate presence points from the background"; it is not a measure of performance against true absences.

7.5 Omission, precision-recall and calibration (Omission & PR)

Figure 7.3. Omission & PR sub-tab: omission rate, precision-recall and calibration panels in one row for each algorithm, thin curves per fold and the mean.

Figure 7.3. Omission & PR sub-tab: omission rate, precision-recall and calibration panels in one row for each algorithm, thin curves per fold and the mean.
Figure 7.3. Omission & PR sub-tab: omission rate, precision-recall and calibration panels in one row for each algorithm, thin curves per fold and the mean.

What it does. For each algorithm, three panels are drawn from the out-of-fold predictions (cv_omission_pr_calibration, sdm_core.py:6926-6995; tabs/tab_evaluation.py:459-571):

Effect on results. Screen, evaluation_omission_pr_calibration.png, and figures/evaluation_omission_pr_calibration.csv, which contains for each algorithm the mean omission at the 0.5 threshold, AP, baseline precision and mean calibration slope (tabs/tab_evaluation.py:573-597).

In the example run, AP is 0.556 ± 0.103 (baseline 0.12) with a mean calibration slope of 0.60 for GAM, and AP 0.547 ± 0.109 with a slope of 1.03 for GLM: the two models discriminate similarly, but GAM's probabilities are overconfident.

Caution. With pseudo-absence data, precision and calibration cannot be interpreted as absolute values: the "absence" class is background, and the probability scale depends on the PA:presence ratio (7:1 in the example run, baseline precision about 0.12). These panels are for comparing algorithms and folds with each other. Because the hold-out sets are small (32–43 presences per fold in the example run), the ±1 SD band is not a confidence interval.

7.6 Variable importance (Variable Importance)

Figure 7.4. Variable Importance sub-tab: bars of mean permutation importance per algorithm for each variable; the legend box is to the right of the chart.

Figure 7.4. Variable Importance sub-tab: bars of mean permutation importance per algorithm for each variable; the legend box is to the right of the chart.
Figure 7.4. Variable Importance sub-tab: bars of mean permutation importance per algorithm for each variable; the legend box is to the right of the chart.

What it does. The permutation importance computed on the test fold in each CV run in ③ Models (the drop in AUC when a predictor is shuffled) is averaged for each algorithm × variable over all folds and PA sets and plotted in a grouped bar chart (tabs/tab_evaluation.py:607-647; computation: sdm_core.py:5562-5636). There are no settings; the number of repetitions is the Variable importance reps: field in ③ Models (Chapter 6, §6.4.7).

When and how to set it. Importance values are on the same scale across algorithms (AUC drop); nevertheless, tree-based and linear models may rank importance differently, so look for consistency across algorithms for a common ranking.

Effect on results. Screen and evaluation_variable_importance.png. In the example run, the most important variable in GAM and GLM is bio_19 (AUC drop of about 0.27 and 0.21, respectively), and in MaxEnt bio_15 (about 0.16); in RF no variable exceeds 0.06.

Caution. Correlated predictors share importance: when one is shuffled, the other continues to carry the information, and both may appear unimportant. Importance can be negative (shuffling increased AUC by chance). If Variable importance reps: = 0, the chart stays empty. Importance is not computed for full-data models; the chart comes only from CV runs.

7.7 Response curves (Response Curves)

Figure 7.5. Response Curves sub-tab: Algorithm, Method and Plot controls; one panel per continuous predictor for the selected algorithm, the thick line is the mean response, the shading is the standard deviation across replicates; only the upper part of the second-row panels is visible, the curves lie in the region below 0.4 that becomes visible on scrolling.

Figure 7.5. Response Curves sub-tab: Algorithm, Method and Plot controls; one panel per continuous predictor for the selected algorithm, the thick line is the mean response, the shading is the standard deviation across replicates; only the upper part of the second-row panels is visible, the curves lie in the region below 0.4 that becomes visible on scrolling.
Figure 7.5. Response Curves sub-tab: Algorithm, Method and Plot controls; one panel per continuous predictor for the selected algorithm, the thick line is the mean response, the shading is the standard deviation across replicates; only the upper part of the second-row panels is visible, the curves lie in the region below 0.4 that becomes visible on scrolling.
Name in the UI What it does Default Options Code
Algorithm: Algorithm whose curves are drawn first algorithm algorithms in the run tabs/tab_evaluation.py:654-655, 885-892
Method: Response curve method marginal (median-hold, fast) marginal / partial / ice tabs/tab_evaluation.py:656-671
Plot Computes and draws the curves — — tabs/tab_evaluation.py:672-673

7.7.1 Response curve method (Method)

What it does. Each continuous predictor is varied at 100 points between its 5th and 95th percentiles in the pooled data of all PA sets (presences and pseudo-absences) (compute_response_curves, sdm_core.py:8354-8510):

In every method, all fitted models of the algorithm (the CV runs of each PA set and the full-data model) are predicted separately; the thick line is their mean and the shading their standard deviation (sdm_core.py:8461-8506). For categorical predictors, one bar is drawn per class (other predictors at the median; variables with 2–60 classes) (sdm_core.py:8512-8553).

When and how to set it. marginal is fast and comparable with biomod2/maxnet outputs. If there are strong interactions or correlations between predictors, the median row may be an unrealistic combination of predictors; partial averages over the real data distribution. ice is used to see interactions: thin curves of differing shapes indicate that the effect of the focal variable depends on other variables.

Effect on results. Screen and evaluation_response_<ALGO>.png only. partial requires about 500 × 100 predictions, ice 60 × 100 predictions × the number of models; it is noticeably slow for MaxEnt and RF. The result is cached for the same run, method and algorithm; redrawing does not recompute (tabs/tab_evaluation.py:690-712).

Caution. The curves are computed without applying the Clamp predictors … setting of ③ Models (the prediction functions are called directly, sdm_core.py:8419-8437). The y axis is fixed to the range 0–1; with pseudo-absence data the probability scale is low, so the curves may look compressed (in the example run the GLM curves stay below 0.25). The shading of a curve shows the difference between CV and full-data models; it is not a confidence interval. The figure has 3 columns; with more than nine predictors, scroll the figure for the lower rows.

7.8 Boyce detail (Boyce Detail)

Figure 7.6. Boyce Detail sub-tab: one panel per PA set × fold group for the selected algorithm; P/E (F-ratio) curve, dashed line at level 1 and the CBI value of that fold in the title.

Figure 7.6. Boyce Detail sub-tab: one panel per PA set × fold group for the selected algorithm; P/E (F-ratio) curve, dashed line at level 1 and the CBI value of that fold in the title.
Figure 7.6. Boyce Detail sub-tab: one panel per PA set × fold group for the selected algorithm; P/E (F-ratio) curve, dashed line at level 1 and the CBI value of that fold in the title.
Name in the UI What it does Default Options Code
Algorithm: Algorithm first algorithm algorithms in the run tabs/tab_evaluation.py:774
PA set: PA set to display (all) (all), PA1, PA2, … tabs/tab_evaluation.py:775, 893-897
Plot Boyce Draws the panels — — tabs/tab_evaluation.py:776-777

What it does. For each PA set × fold, a Boyce curve is drawn from the hold-out predictions: the x axis is the window midpoint, the y axis the ratio F=P_i/E_i; the region above 1 is shaded; the title colour is green for CBI ≥ 0.5, orange for 0–0.5 and red below 0 (cv_boyce_curves, sdm_core.py:6998-7015; tabs/tab_evaluation.py:791-847). The function called is the same call that produces the Boyce value in the score table; the CBI in the panel title matches the value in the table.

When and how to set it. In a good model, the P/E curve rises monotonically with suitability and lies below 1 at low suitability and above 1 at high suitability (Hirzel et al. 2006). A curve that falls after a peak indicates that the model assigns its highest values to too large an area; a flat curve indicates weak discrimination.

Effect on results. Screen and evaluation_boyce_<ALGO>.png. In the example run, the fold CBI values of GLM range from 0.274 (PA1 RUN2) to 0.943 (PA1 RUN3); in the RUN1 and RUN2 panels the P/E curve peaks at intermediate suitability and then falls.

Caution. The window width is one tenth of the prediction range (ecospat definition); in models with a very narrow prediction range (e.g. BIOCLIM) the windows fall very close to each other. If the number of presences per fold is small, the curve is stepped; read the CBI as the mean over folds, not from a single fold.

7.9 Classification thresholds (Classification Thresholds)

Figure 7.7. Classification Thresholds sub-tab: descriptions of the threshold rules and, for each full-data model, the threshold of each of the eleven rules (upper line) with the out-of-fold TSS at that threshold (lower line); the last column is the threshold used in the map.

Figure 7.7. Classification Thresholds sub-tab: descriptions of the threshold rules and, for each full-data model, the threshold of each of the eleven rules (upper line) with the out-of-fold TSS at that threshold (lower line); the last column is the threshold used in the map.
Figure 7.7. Classification Thresholds sub-tab: descriptions of the threshold rules and, for each full-data model, the threshold of each of the eleven rules (upper line) with the out-of-fold TSS at that threshold (lower line); the last column is the threshold used in the map.
Name in the UI What it does Code
Threshold Methods box Short definitions of the eleven rules tabs/tab_evaluation.py:1132-1160
Table Rows: full-data model (algorithm × PA set); columns: 11 rules and Used (current) tabs/tab_evaluation.py:1162-1180
↻ Compute All Thresholds Recomputes the table tabs/tab_evaluation.py:1183-1185
💾 Save Threshold Table (CSV) Writes figures/classification_thresholds.csv tabs/tab_evaluation.py:1316-1342
💾 Save Scores (CSV) Writes evaluation_scores.csv (§7.10) tabs/tab_evaluation.py:1188-1194

What it does. For each full-data model, a map threshold is found with each of the eleven rules (Chapter 6, §6.5.1), and two lines are written in the cell: the threshold and the mean over folds of the TSS this threshold gives on the out-of-fold predictions (tabs/tab_evaluation.py:1200-1314; sdm_core.py:7018-7037). The tooltip additionally shows the out-of-fold sensitivity and specificity at that threshold, the criterion value on the training data and the rule "pixel >= t -> Present (1)". The Used (current) column is the threshold actually used in the map. The CSV contains, for each rule, the threshold and the _holdout_Sens, _holdout_Spec, _holdout_TSS columns, the threshold used, the name of the rule and the number of folds.

When and how to set it. To support the choice of threshold rule: you see which rule gives a wider or narrower map while preserving out-of-fold TSS. The sensitivity column is always computed with a target of 0.90 (method description).

Effect on results. This table does not change the maps; the maps change when the rule is changed in ③ Models and the models are rerun.

Caution. The thresholds in this table and the threshold used in the map are found in different ways:

For this reason the max_tss column and Used (current) differ even for the same rule. In the example run, for GAM PA1 the max_tss column is 0.2562 (out-of-fold TSS 0.59) and the threshold used is 0.0605 (out-of-fold TSS 0.64). The difference grows for algorithms that memorise the training data: for RF PA1 the max_tss column is 0.4532 (out-of-fold TSS 0.34) and the threshold used is 0.1540 (out-of-fold TSS 0.63). To compare rules, look at the out-of-fold TSS in the lower line; do not read the threshold values as the map threshold.

7.10 Bottom buttons: refresh, save

Name in the UI What it does Code
↻ Refresh All Charts Updates the drop-down lists and recomputes all seven sub-tabs tabs/tab_evaluation.py:851-872
💾 Save All Charts to Output Folder Writes all figures and the score table to the figures/ folder; can be cancelled tabs/tab_evaluation.py:939-1058
💾 Save Scores (CSV) Writes evaluation_scores.csv to the output root and a copy to figures/ tabs/tab_evaluation.py:1064-1108

What it does. Save All Charts … writes, in order, the score, ROC, omission/PR/calibration and importance figures; then, for each algorithm with CV models, the response curve figure (with the currently selected Method:) and the Boyce figure; and finally the score table (tabs/tab_evaluation.py:993-1030). Each figure is saved once as a 300 dpi PNG (tabs/tab_evaluation.py:923-937). When a figure is drawn on screen, it is also saved under the same name; therefore, before the button is pressed, the figures/ folder contains only the response and Boyce figures of the displayed algorithm (in the example run only evaluation_response_GLM.png and evaluation_boyce_GLM.png).

Caution. evaluation_scores.csv is written automatically when ③ Models finishes, when the tab is first opened and when the ensemble is rebuilt (tabs/tab_evaluation.py:1071-1081; tabs/tab_models.py:1840-1845); Save Scores (CSV) is only for repeating this manually. If there is no output folder, figures cannot be saved and "Output directory not set — cannot save figure." is written to the log. The information window after saving mentions a figures/response_curves/ subfolder; no such folder is created, and a single combined figure is written per algorithm.

7.11 Outputs and what passes to the next step

In the <output>/figures/ folder:

evaluation_scores.csv in the output root. This tab does not change models, thresholds or maps; ⑧ Report uses the same score table and figures (Chapter 11).

7.12 Expert questions

Q1. Are the ROC, omission and Boyce curves on this tab based on independent data? Is the model tested on its training data? Answer: The curves are drawn only from the CV hold-out predictions saved in ③ Models (modeler.cv_holdout; sdm_core.py:5547-5556); no new split is made and no predictions are made on training rows (sdm_core.py:6926-6932). The degree of independence depends on the CV design in ③ Models: with spatial blocks, the tests are geographically separate from training; with the random design they are not. The exceptions are the response curves and the threshold columns of the Classification Thresholds table.

Q2. Is the AUC reported for the ensemble (EMwmean) computed from the ensemble map? Answer: No. The ensemble rows and ROC panels are produced by combining, in each fold, the CV models that have not seen that fold with the ensemble rule and scoring them only on that fold; member selection and weights come from the scores of the other folds (sdm_core.py:6516-6696). The note on the scores sub-tab states this explicitly: "… they are NOT computed from the ensemble map." (tabs/tab_evaluation.py:269-274).

Q3. At how many thresholds and on which data is TSS computed; is it comparable with ecospat or biomod2? Answer: TSS is computed on the test fold, at the threshold that the selected rule (default max_tss) finds on the inner out-of-fold predictions of the training rows (sdm_core.py:1193-1217). biomod2 and ecospat.max.tss usually optimise the threshold on the evaluation data itself; this "oracle" threshold raises TSS. If Test-fold threshold: = test is selected in ③ Models, HABITUS follows the same path and the numbers become comparable; however, this choice is optimistic (Chapter 6, §6.5.2).

Q4. Is the Boyce index exactly the same as in ecospat? Answer: The default ecospat formula is a line-by-line port of the ecospat.boyce function (ecospat 4.1.4): window width 1/10 of the prediction range, 101 positions, closed windows, removal of consecutive repeated P/E values, and Spearman correlation (sdm_core.py:475-543). The difference lies in the choice of the fit set: HABITUS uses all rows of the evaluation fold (test presences and test pseudo-absences) (sdm_core.py:605-610); in ecospat, fit is in most applications the raster predictions of the whole study area. The values therefore match exactly for the same predictions, and approximately for the full map.

Q5. How is the calibration slope computed, and what does it mean with pseudo-absence data? Answer: It is the slope of the logistic regression (practically unpenalised, C = 1e12) of the observation on the logit of the prediction; predictions are clipped to the range [10^{-6}, 1-10^{-6}], and at least 10 rows and two classes are required (sdm_core.py:989-1013; Pearce and Ferrier 2000). With pseudo-absence data, the scale of the probability depends on the PA:presence ratio; the slope is an indicator not of the true probability of presence but of the model's over- or underconfidence on out-of-fold data. A slope much smaller than 1 (overfitting) is meaningful in comparisons between algorithms.

Q6. Why does OR_10p sometimes come out as 0, and is this a good result? Answer: Not always. If the omission threshold is equal to or below the lowest prediction of the evaluation set, no presence can fall below the threshold; omission is 0 by construction. This case is flagged with OR_10p_degenerate = 1; if it occurs in at least half of the folds, a warning is written for the algorithm (sdm_core.py:1100-1114, 5787-5816). It is usually seen with BIOCLIM, shallow CART and strongly regularised GLMNET; the omission of these algorithms should not be compared with that of the others.

Q7. Which method is used to compute variable importance; is it like MaxEnt's percent contribution? Answer: No, it is permutation importance: a predictor is shuffled on the test fold and the drop in AUC is measured (sdm_core.py:5564-5636). MaxEnt's "percent contribution" and "permutation importance" outputs are normalised to percentages and computed on the training data; HABITUS does not normalise and computes on the test fold. For scikit-learn models, because of the fixed seed, the Variable importance reps: repetitions give the same result; the effective repetition is n_repeats = 3.

Q8. From which models and which data are the response curves produced? Answer: From all fitted models of the algorithm (the CV runs of each PA set and the full-data model); the thick line is their mean, the shading their standard deviation (sdm_core.py:8461-8506). The range of the variable and the median row are taken from the pooled presence and pseudo-absence rows of all PA sets; the range is the 5th–95th percentile, and there is no extrapolation at the ends (sdm_core.py:8405-8457). The default approach in biomod2's bm_PlotResponseCurves function is also to hold the other variables at a fixed value (the median); because the variable range and summarising details are not the same in the two programs, the curves may be similar in shape but not numerically identical.

Q9. When do marginal, PDP and ICE give different results? Answer: When there are interactions or strong correlations between predictors. Marginal shows only the neighbourhood of the median row; PDP averages over real data rows (Friedman 2001); ICE shows the curve of each row separately and reveals the heterogeneity that PDP hides by averaging (Goldstein et al. 2015). In a linear additive model (e.g. a GLM without interactions) all three give the same shape, though their levels may differ; they diverge in tree models and in GLMs with interactions terms.

Q10. Why is max_tss in the Classification Thresholds table not the same as the threshold in the map? Answer: The table finds the threshold from the full-data model's predictions (seen in training) on the pooled data of all PA sets (tabs/tab_evaluation.py:1217-1259); the threshold in the map, in contrast, is the mean of the thresholds found in the CV folds according to the threshold source (sdm_core.py:5852-5853). In the example run, for RF PA1 the table gives 0.4532 and the map 0.1540; because RF's training predictions are close to 1 at presences, the table threshold comes out high and out-of-fold TSS drops to 0.34, whereas the map's threshold gives 0.63. The reliable part of the table is the out-of-fold TSS in the lower line.

Q11. Is the difference between algorithms significant? Is a test provided? Answer: No, there is no statistical comparison test. The program gives the mean, SD and range over folds and PA sets (sdm_core.py:5883-5919), and the charts show the distribution. Because the folds derive from the same data, they are not independent; a simple t test would not be valid either. In the example run, the difference between folds of the same algorithm (GLM PA1 ROC 0.790–0.941) is larger than the difference between algorithms; in this situation it is more defensible to include several algorithms in the ensemble than to rank them.

Q12. How does an algorithm run without ESM or CV appear on this tab? Answer: Algorithms without out-of-fold predictions are listed with a "Not shown" note in the ROC, omission and Boyce figures (sdm_core.py:6872-6876, 6974-6977); the Classification Thresholds cells show TSS n/a. ESM's hold-out predictions are saved and appear normally; only because it does not produce inner out-of-fold training predictions may its threshold source be labelled differently (Chapter 6, §6.5.2).

7.13 Common mistakes and pitfalls

7.14 References

Allouche, O., Tsoar, A., Kadmon, R., 2006. Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS). Journal of Applied Ecology 43, 1223–1232. https://doi.org/10.1111/j.1365-2664.2006.01214.x

Fielding, A.H., Bell, J.F., 1997. A review of methods for the assessment of prediction errors in conservation presence/absence models. Environmental Conservation 24, 38–49. https://doi.org/10.1017/S0376892997000088

Friedman, J.H., 2001. Greedy function approximation: A gradient boosting machine. The Annals of Statistics 29, 1189–1232. https://doi.org/10.1214/aos/1013203451

Goldstein, A., Kapelner, A., Bleich, J., Pitkin, E., 2015. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics 24, 44–65. https://doi.org/10.1080/10618600.2014.907095

Hirzel, A.H., Le Lay, G., Helfer, V., Randin, C., Guisan, A., 2006. Evaluating the ability of habitat suitability models to predict species presences. Ecological Modelling 199, 142–152. https://doi.org/10.1016/j.ecolmodel.2006.05.017

Li, W., Guo, Q., 2013. How to assess the prediction accuracy of species presence–absence models without absence data? Ecography 36, 788–799. https://doi.org/10.1111/j.1600-0587.2013.07585.x

Pearce, J., Ferrier, S., 2000. Evaluating the predictive performance of habitat models developed using logistic regression. Ecological Modelling 133, 225–245. https://doi.org/10.1016/S0304-3800(00)00322-7