Default vs. spatial

To assess which method performs best we will use four different evaluation criteria. Then we will create plots of the AUC (area under the receiver operating characteristic curve) together with the MAE (mean absolute error) as proposed by Konowalik & Nosol (2021). We will calculate both metrics on the independent test data from the presence records and the background points using the R package Metrics. As a third metric independent from the background points we will also calculate the Boyce-Index (Boyce et al., 2002) using the R package ecospat with the prediction raster and spatial test folds. To assess how complex the models are we will furthermore have a look at the number of parameters of each model. We will determine all four metrics for each fold of the FFME and an calculate the median of all these values to get a comprehensive picture of the performance of each method.

We will start by calculating the metrics for each fold:

Forward Fold Metric Estimation

Once we have the metrics for all folds we will calculate the median for each species using the script below. These are our final results.

To have a look at our final results and compare the methods with each other we we will now plot the different metrics. The first plot shows for how many of the species which method provided the better results compared to the other method:

Figure: Number of species (y-axis) for which the maximum Boyce-Index, the maximum AUC value, the minimum MAE or the minimum number of parameters was measured. Sums up to 20 species for each metric.

The next plots are inspired by the plots proposed by Konowalik & Nosol (2021). Therefore we plot the MAE on the x-axis and the AUC and the Boyce-Index on the y-axis. The method that is closer to the upper right corner performed better.

Figure: Results of region Ontario, Canada modeled with the methods spatial (yellow) and standard (purple). AUC (point) and Boyce-Index (triangle) on the y-axis and the MAE on the x-axis. For each Species the median value over all training folds is given.

We repeat the same plots again this time with the number of parameters instead of the MAE on the x-axis. Again the method that is closer to the upper right corner performed better.

Figure: Results of region Ontario, Canada modeled with the methods spatial (yellow) and standard (purple). AUC (point) and Boyce-Index (triangle) on the y-axis and the Number of parameters on the x-axis. For each Species the median value over all training folds is given.