HABITUSReference Book

2. Data I: records, layers and study boundary (① Data)

Tab: ① Data · Code: tabs/tab_data.py (UI), sdm_inputs.py (CSV reading), sdm_mask.py (folder scanning, clipping to the boundary), sdm_core.py DataFormatter (loading and record filtering)

2.1 What this tab does

① Data is the first and mandatory step of the workflow. It reads the species records (presence points), the predictor raster layers and an optional study boundary; it filters the records; it then generates the pseudo-absence sets for the machine learning models, the background points for MaxEnt and a separate background sample for ENFA. This chapter covers the "data" half of the tab: the record file, the layers, the boundary, the seed, the output folder and every operation that Load Data performs on the records. The background, pseudo-absence, accessible area and thinning settings are in Chapter 3.

2.2 Screen layout

The tab consists of two columns (tabs/tab_data.py:78-82). The left column holds the record file, the raster lists and the study boundary; the right column holds the background/pseudo-absence settings, reproducibility and the output folder. At the bottom is the green button that starts the whole step, with the progress bar and the session log below it.

Figure 2.1. The ① Data tab with sample data loaded, before the button is pressed. Left: Occurrence Data (CSV), the Continuous/Categorical Rasters sub-tabs and Study Boundary; right: Background & Pseudo-Absence Settings (here the Study Area & Thinning sub-tab is open), Reproducibility and Output Directory; bottom: the Load Data button and the log.
Figure 2.1. The ① Data tab with sample data loaded, before the button is pressed. Left: Occurrence Data (CSV), the Continuous/Categorical Rasters sub-tabs and Study Boundary; right: Background & Pseudo-Absence Settings (here the Study Area & Thinning sub-tab is open), Reproducibility and Output Directory; bottom: the Load Data button and the log.
Region Name in the UI In this book
Top left Occurrence Data (CSV) §2.3
Middle left Continuous Rasters, Categorical Rasters §2.4
Bottom left Study Boundary (mask, optional) §2.5
Top right Background & Pseudo-Absence Settings (6 sub-tabs) Chapter 3
Middle right Reproducibility §2.6
Bottom right Output Directory §2.7
Bottom bar Load Data & Generate Background/Pseudo-Absence Points §2.8, §2.9

2.3 Species records: Occurrence Data (CSV)

This group reads the table holding the coordinates of the places where the species was observed. When a file is selected, the program inspects the first rows, identifies the columns, states in a green information box which column will be used as what and, if the file contains more than one species, opens a species list.

Figure 2.2. Left column: the selected record file and the "Detected columns (by name)" information box (species, longitude, latitude found by name); below it the 19 WorldClim bioclimatic layers in the Continuous Rasters list and the Add Continuous Rasters…, Add Folder…, → Categorical, Remove Selected buttons.
Figure 2.2. Left column: the selected record file and the "Detected columns (by name)" information box (species, longitude, latitude found by name); below it the 19 WorldClim bioclimatic layers in the Continuous Rasters list and the Add Continuous Rasters…, Add Folder…, → Categorical, Remove Selected buttons.
Name in the UI What it does Default Range / options Code
CSV file: + Browse… Selects the record table; on selection scans the columns and species and suggests an output folder empty .csv, .tsv, .txt tabs/tab_data.py:92-96, 972-979
Information box Shows the detected species, longitude and latitude columns and the columns that will be ignored — — tabs/tab_data.py:99-108, 1008-1023
Species: list In a multi-species file, one checkbox per species (with its record count) all ticked All / None tabs/tab_data.py:112-149, 1045-1072
Analyse now: Selects the species for which the ①–⑧ workflow will now run first ticked species ticked species tabs/tab_data.py:132-140, 1085-1095

2.3.1 File format, delimiter and decimal mark

2.3.2 Column detection (by name, by position, GBIF)

2.3.3 Multi-species file and Analyse now

2.3.4 Species name and suggested output folder

The species name used in reports and file names is determined as follows (sdm_core.py:1566-1593): if the first column is neither a coordinate nor the species column and is not numeric, the most frequent text in it; otherwise the most frequent text in the species column; failing that, the name of the CSV file (underscores and hyphens converted to spaces). In the sample run the name is "abies nordmanniana". If Output Directory is empty, a folder Documents/HABITUS/<species>_YYYYMMDD_HHMMSS is created as soon as the file is selected and written into the field (tabs/tab_data.py:1125-1160, main_dialog.py:832-853); for a multi-species file the folder name is taken from the CSV name.

2.4 Predictor layers: Continuous Rasters and Categorical Rasters

Predictors are kept in two separate lists. Continuous layers are measured quantities: temperature, precipitation, elevation, slope, NDVI. Categorical layers are class codes: land cover, soil type, geology. This distinction affects every subsequent step: continuous layers enter the VIF and correlation analysis in ② Variables, while categorical layers are excluded from that analysis and are passed to the models as classes.

Name in the UI What it does Default Range / options Code
Add Continuous Rasters… Adds the selected files to the continuous list (the same path is not added twice) — .tif .tiff .asc .img .nc .vrt tabs/tab_data.py:911-912, 1183-1191
Add Folder… Reads all rasters in the folder and classifies each as continuous/categorical — folder tabs/tab_data.py:913-919, 1205-1290
→ Categorical / → Continuous Moves the selected files to the other list — — tabs/tab_data.py:920-922, 948-950, 1197-1203
Remove Selected Removes the selected files from the list — — tabs/tab_data.py:923-924, 1193-1195
Add Categorical Rasters… Adds the selected files to the categorical list — same extensions tabs/tab_data.py:946-947
Encoding note States that the categorical encoding is label (integer class code) label label only tabs/tab_data.py:959-968, sdm_core.py:1428-1440

2.4.1 Adding files individually and variable names

2.4.2 Add Folder…: continuous/categorical classification and its rationale

2.4.3 How categorical layers enter the model

2.4.4 Converting large ASCII grids to GeoTIFF

2.4.5 Grid alignment and coordinate system

2.5 Study boundary: Study Boundary (mask, optional)

This group clips all rasters with a polygon or with a rectangle in degrees. Global layers can thus be used without being cut beforehand, and every cell outside the boundary becomes NoData.

Name in the UI What it does Default Range / options Code
Mode box Selects the type of boundary None (whole raster extent) None (whole raster extent) / Polygon layer (SHP, GPKG, KML, GeoJSON) / Corner coordinates (degrees) tabs/tab_data.py:173-181
File field + Browse… / Clear Polygon layer (visible only in the second mode); selecting a file switches the mode to polygon automatically empty .shp .gpkg .geojson .json .kml .gml tabs/tab_data.py:161-172, 1332-1339
West / South / East / North Corners of the rectangle, WGS84 degrees (third mode only) 25 / 35 / 45 / 43 longitude −180…180, latitude −90…90; 4 decimals tabs/tab_data.py:186-197

2.5.1 Modes and the scope of clipping

2.5.2 How clipping is done: single window, common grid, masked_inputs

2.6 Reproducibility: Reproducibility

Figure 2.3. The Reproducibility group (Random seed 42) and the Output Directory field; the explanatory text states what the seed is applied to and that it is written to the habitus_run_config.json file.
Figure 2.3. The Reproducibility group (Random seed 42) and the Output Directory field; the explanatory text states what the seed is applied to and that it is written to the habitus_run_config.json file.
Name in the UI What it does Default Range / options Code
Random seed: Starting number for all random operations 42 0 – 2147483647 tabs/tab_data.py:242-249, run_config.py:28

2.6.1 Random seed

2.7 Output folder: Output Directory

Name in the UI What it does Default Range / options Code
Folder field + Browse… Folder to which all outputs are written when a CSV is selected, Documents/HABITUS/<species>_<time> any folder tabs/tab_data.py:253-260, 1125-1160

It does not affect the results; if left empty, the run stops with the message "Please specify an output directory.". In a multi-species file each species is written to the <output folder>/<species> subfolder (tabs/tab_data.py:1115-1123); the session log is also moved to this folder at the start of the run, so in a batch run each species' log is in its own folder (tabs/tab_data.py:1398-1408).

2.8 The Load Data button: order of operations and record filtering

The ▶ Load Data & Generate Background/Pseudo-Absence Points button runs the whole step in the background, without freezing the UI (tabs/tab_data.py:1363-1574). The order of operations matters for understanding which record was dropped and why:

  1. Pre-check: does the CSV exist, is there at least one continuous raster, is the output folder filled in, do the target-group CSV and the bias raster (if given) exist, is the boundary valid.
  2. If there was a previous data step, all results of the later tabs are reset.
  3. In a multi-species file, only the records of the selected species are separated into occurrences_used.csv.
  4. The seed is set (§2.6).
  5. If a boundary is given, all layers are clipped (§2.5).
  6. load_data: the CSV and the layers are read, the species name is determined, grid alignment is checked (sdm_core.py:1549-1681).
  7. generate_pa_datasets: coordinates are converted to numbers, records are filtered (this chapter), thinning and the accessible area are applied, pseudo-absence sets and backgrounds are drawn (Chapter 3), and the collinearity pre-computation is performed (sdm_core.py:2911-3249).
  8. The accessible area is written as accessible_area.tif; if requested, the MaxEnt background is saved to CSV.

2.8.1 Steps of record filtering

Presence records are filtered in the following order (sdm_core.py:2931-2968, 1824-2037). Every dropped record enters the dropped_points list with its reason; the list appears in the warning window, the log, the report and habitus_run_config.json (points_dropped_by_reason).

Order Reason (text in the UI) Rule Can it be turned off Code
1 missing coordinate Longitude or latitude cell is empty no sdm_core.py:2944-2953
2 duplicate coordinate Identical to an earlier record when the coordinates are rounded to 8 decimals no sdm_core.py:1844-1856
3 outside raster extent Cell row/column is outside the grid no sdm_core.py:1864-1876
4 falls on NoData cell NaN in any continuous layer or NoData in any categorical layer no sdm_core.py:1878-1905
5 same raster cell as another record A record earlier in file order exists in the same predictor cell yes: Keep one record per raster cell sdm_core.py:1913-1936
6 spatially thinned A retained record exists within the thinning distance yes (Chapter 3) sdm_core.py:2960-2968

2.8.2 Duplicate coordinates, outside the raster and NoData

2.8.3 One record per cell

2.8.4 At least five presences

2.8.5 Warnings on record count and the record/predictor ratio

2.8.6 Messages after the data step

When the step finishes, the following windows appear in order (tabs/tab_data.py:1576-1761):

  1. "Invalid Occurrence Points Detected": if at least one record was dropped; one heading per reason and a list of coordinates (under the cell and thinning headings the first 15 records, the rest as "… and N more"), ending with "✔ N valid point(s) will be used for modelling." and a CRS reminder. The same text is written to the log.
  2. "Points worth knowing before you interpret this run": warnings that affect the interpretation of the result (resolution, record/predictor ratio, accessible area, thinning, absence records…). If the only warning is the sampling bias reminder, this window does not open and the warning is written only to the log with the prefix "ADVISORY:".
  3. "Data ready!" summary: number of presences used/loaded, breakdown of dropped records by reason, the pseudo-absence or true absence row, number of MaxEnt background points, the accessible area and the share it covers, the thinning distance and how it was chosen, the autocorrelation range, prevalence, the note that sampling bias has not been corrected, the seed, and the names of the continuous and categorical variables.

The summary of the sample run, compiled from the session log, is as follows: 626 records loaded; 405 dropped as same cell and 78 by thinning; 143 presences used; 2 sets × 1000 pseudo-absences; 10,000 MaxEnt background points; accessible area buffer, 200 km, 40% of the raster extent; thinning 4.7 km (automatic); autocorrelation range 151.5 km; prevalence 0.125.

Figure 2.4. The bottom bar after the data step has finished: "Ready: 143 presences, 2 PA sets, 19 cont + 0 cat vars" in the progress bar, and in the log the end of the record/predictor warning and the line for the correlation heat map saved on moving to ② Variables.
Figure 2.4. The bottom bar after the data step has finished: "Ready: 143 presences, 2 PA sets, 19 cont + 0 cat vars" in the progress bar, and in the log the end of the record/predictor warning and the line for the correlation heat map saved on moving to ② Variables.

2.9 Outputs and what passes to the next step

2.10 Expert questions

Q1. Are the coordinates transformed to the raster's coordinate system, or are they expected to be in the same system? Answer: They are expected to be in the same system. Records are converted directly to row-column with the raster's transform matrix (sdm_core.py:1763-1765, 1866); there is no coordinate system field or reprojection for the CSV. Reprojection is done only for the boundary and calibration polygons (sdm_mask.py:219-227, sdm_spatial.py:1194-1225). With projected rasters the CSV must also be in metres; with geographic rasters values outside ±90/±180 produce a warning.

Q2. Which operation in R does the one-record-per-cell rule correspond to, and why is it applied before thinning? Answer: It is de-duplication by cell identity on the predictor grid; it resembles selecting one point per grid cell with gridSample in dismo, or filtering with duplicated on cell numbers. HABITUS keeps the first record in file order in each cell (sdm_core.py:1919-1936). The reason for applying it first is that records in the same cell are copies of each other in terms of predictors, and thinning would waste time eliminating these copies in random order; moreover, distance-based thinning is meaningless at the sub-cell scale. If a thinning distance smaller than the cell is entered, the program reports it (sdm_core.py:2526-2534).

Q3. How precisely are duplicate coordinates compared? Answer: Longitude and latitude are rounded to 8 decimals and compared (sdm_core.py:1848); in degrees this is about 1 mm. That is, only a truly identical point counts as a duplicate; records entered a few metres apart pass this filter and are left to the one-per-cell rule and thinning.

Q4. How is it detected that longitude and latitude are swapped? Answer: If more than 20% of the non-duplicate records fall outside the raster or on NoData, the records are placed on the grid once more with latitude and longitude swapped; if the share falling inside in this state is at least 0.5 and greater than the actual share inside, the message says that the columns are most likely reversed (sdm_core.py:1972-1990, sdm_inputs.py:207-216). The program does not swap the columns automatically; the decision is left to the user.

Q5. What is the minimum number of records, where does this number come from, and what is recommended with few records? Answer: At least 5 records after filtering are mandatory (sdm_core.py:1269, 1993-2013); the rationale is the studies in which SDM methods were tested with small samples (Hernandez et al. 2006; Pearson et al. 2007). Warnings are written that below 10 no reliable model can be produced and below 25 jackknife validation should be selected (sdm_core.py:2864-2875). For species with very few records, the ESM (ensemble of small models) option in ③ Models is described separately (Chapter 5).

Q6. With which number of predictors is the "10 records / predictor" rule applied? Answer: With the number of all layers loaded in ① Data (sdm_core.py:2858); this number is usually larger than the number in the final model. When the selection is confirmed in ② Variables, the same check is repeated with the selected number and the new message replaces the previous one (sdm_core.py:2889-2909). The rule is advisory; it does not stop the run. Source: Wisz et al. (2008) and the events-per-variable tradition in logistic regression.

Q7. Can layers of different resolution or extent be used together? Answer: Without a boundary, no: the number of rows and columns, cell size, origin and CRS must be the same, otherwise the run names the non-matching layer and stops (sdm_core.py:1698-1744). With a boundary, yes: all layers are brought by nearest neighbour to the common grid defined by the cells of the first layer of the list (sdm_mask.py:283-301). The program does not aggregate by averaging; to bring a fine layer down to a coarse grid, aggregating it beforehand with a GIS tool is more appropriate.

Q8. Why is a categorical layer not used as a number in linear models, and which algorithms leave it out? Answer: The layer is carried in a single column with its class code, but the first step of the GLM, GLMNET, SVM and ANN pipeline (_CatPrep) produces a separate 0/1 column for each class seen in training; the code is never used as a number that is squared, scaled or multiplied (sdm_core.py:3364-3384). GAM uses a factor term and MaxEnt a categorical feature; tree methods split on the code; BIOCLIM, Domain, ENFA and Mahalanobis leave the layer out (sdm_core.py:1781-1787). If a class not seen in training is encountered in the projection area, all 0/1 columns of that variable become zero.

Q9. Do two runs with the same seed give bit-for-bit identical results? Answer: That is the aim: the seed is applied to all generators at the start of the data step (tabs/tab_data.py:1510, run_config.py:31-50), and the background draw, thinning, variogram subsample, k-means and folds come from their own seeded generators. The order in which layers are added does not change the result either, because the variables are sorted in natural order (sdm_core.py:1617-1620). Bit-for-bit equality across different operating systems or library versions is not guaranteed in the code and has not been tested; for this reason the run record also writes the library versions (habitus_run_config.json, environment).

Q10. What is the difference between the study boundary and the accessible area; can one be used in place of the other? Answer: The boundary clips all layers: cells outside become NoData and the map also ends at the boundary (sdm_mask.py:304-418). The accessible area only restricts the cells from which background and pseudo-absences are drawn; the map is still produced for the whole boundary or raster extent (sdm_core.py:2181-2203). To define the M area in the sense of Barve et al. (2011), the accessible area should be used; using the boundary as M also reduces the model's projection area to M.

Q11. How is the memory requirement determined? Answer: Each continuous layer is read fully into memory; sources that fit losslessly into float32 (float32, 16-bit integers) are held as float32, the others as float64 (sdm_core.py:1367-1379). According to the code comment, 19 global layers at 2.5′ resolution occupy 2.6 GB. The way to reduce memory is to give a study boundary; clipped layers cover only the bounding box of the boundary.

2.11 Common mistakes and pitfalls

2.12 References

Barve, N., Barve, V., Jiménez-Valverde, A., Lira-Noriega, A., Maher, S.P., Peterson, A.T., Soberón, J., Villalobos, F., 2011. The crucial role of the accessible area in ecological niche modeling and species distribution modeling. Ecological Modelling 222, 1810–1819. https://doi.org/10.1016/j.ecolmodel.2011.02.011

Hernandez, P.A., Graham, C.H., Master, L.L., Albert, D.L., 2006. The effect of sample size and species characteristics on performance of different species distribution modeling methods. Ecography 29, 773–785. https://doi.org/10.1111/j.0906-7590.2006.04700.x

Pearson, R.G., Raxworthy, C.J., Nakamura, M., Peterson, A.T., 2007. Predicting species distributions from small numbers of occurrence records: a test case using cryptic geckos in Madagascar. Journal of Biogeography 34, 102–117. https://doi.org/10.1111/j.1365-2699.2006.01594.x

Wisz, M.S., Hijmans, R.J., Li, J., Peterson, A.T., Graham, C.H., Guisan, A., NCEAS Predicting Species Distributions Working Group, 2008. Effects of sample size on the performance of species distribution models. Diversity and Distributions 14, 763–773. https://doi.org/10.1111/j.1472-4642.2008.00482.x