cito: An R package for training neural networks using torch
Deep Neural Networks (DNN) have become a central method in ecology. Most
current deep learning (DL) applications rely on one of the major deep learning
frameworks, in particular Torch or TensorFlow, to build and train DNN. Using
these frameworks, however, requires substantially more experience and time than
typical regression functions in the R environment. Here, we present 'cito', a
user-friendly R package for DL that allows specifying DNNs in the familiar
formula syntax used by many R packages. To fit the models, 'cito' uses 'torch',
taking advantage of the numerically optimized torch library, including the
ability to switch between training models on the CPU or the graphics processing
unit (GPU) (which allows to efficiently train large DNN). Moreover, 'cito'
includes many user-friendly functions for model plotting and analysis,
including optional confidence intervals (CIs) based on bootstraps for
predictions and explainable AI (xAI) metrics for effect sizes and variable
importance with CIs and p-values. To showcase a typical analysis pipeline using
'cito', including its built-in xAI features to explore the trained DNN, we
build a species distribution model of the African elephant. We hope that by
providing a user-friendly R framework to specify, deploy and interpret DNN,
'cito' will make this interesting model class more accessible to ecological
data analysis. A stable version of 'cito' can be installed from the
comprehensive R archive network (CRAN).
Christian Amesoeder, Florian Hartig, Maximilian Pichler
cito: An R package for training neural networks using torch

###### Abstract

1. Deep neural networks (DNN) have become a central class of algorithms for regression and classification tasks. Although some packages exist that allow users to specify DNN in R, those are rather limited in their functionality. Most current deep learning applications therefore rely on one of the major deep learning frameworks, PyTorch or TensorFlow, to build and train DNN. However, using these frameworks requires substantially more training and time than comparable regression or machine learning packages in the R environment.
2. Here, we present 'cito', an user-friendly R package for deep learning. 'cito' allows R users to specify deep neural networks in the familiar formula syntax used by most modeling functions in R. In the background, 'cito' uses 'torch' to fit the models, taking advantage of all the numerical optimizations of the torch library, including the ability to switch between training models on CPUs or GPUs. Moreover, 'cito' includes many user-friendly functions for predictions and an explainable Artificial Intelligence (xAI) pipeline for the fitted models.
3. We showcase a typical analysis pipeline using 'cito', including its built-in xAI features to explore the trained DNN, by building a species distribution model of the African elephant.
4. In conclusion, 'cito' provides a user-friendly R framework to specify, deploy and interpret deep neural networks based on torch. The current stable CRAN version mainly supports fully connected DNNs, but it is planned that future versions will also include CNNs and RNNs.

## Introduction

Deep neural networks (DNNs) are increasingly used in ecology and evolution (Christian et al., 2019; Joseph, 2020; Strydom et al., 2021). For many researchers, however, these methods are still not easily accessible because state-of-the-art deep learning frameworks have steep learning curves, while existing user-friendly R-packages lack important functionalities necessary for training modern DNN architectures, such as the ability to train the models on graphics processing units (Graphic cards, GPUs) or important training techniques (e.g. learning rate scheduler).

Looking at the requirements and user expectations for working with DNNs, we first observe that modern DNNs are almost exclusively implemented and trained in specialized deep learning frameworks such as PyTorch or Tensorflow (Abadi et al., 2016; Paszke et al., 2019). These frameworks are essentially extremely flexible and performant math libraries, consisting of functions and classes to implement and train many different deep learning
architectures, such as large language models (e.g. GPT-3, RoBERTA) (Black et al., 2022; Liu et al., 2019) or complex object detection models (e.g. Mask R-CNN, DeepVit) (He et al., 2017; Zhou et al., 2021).

This flexibility is appealing to "power users", but for many standard applications, even simple fully-connected neural networks (also known as multi-layer perceptron, MLP) can act as a useful predictive models (Strydom et al., 2021) For such applications, the high level of customization offered by the large machine learning frameworks is unnecessary and often prohibitive or at least time-consuming for less experienced users.

As a response to this problem, several simplified frontends for the major machine learning frameworks have been developed, such as Keras for TensorFlow and luz for PyTorch (Allaire and Chollet, 2022; Falbel, 2022). However, building a DNN with these frontends still typically requires a few days of training, which is a lot compared to the time it takes a user to get first results from an R package that relies on the formula syntax for regression or classification that R users are familiar with. Popular packages that follow this syntax include ranger, for training random forests and lme4, for training mixed-effect models (Bates et al., 2015; Wright and Ziegler, 2017).

Some R packages for training DNN using the standard formula syntax already exist, but they often lack crucial functionalities, and most of them do not make use of the state-of-the-art frameworks for model fitting, which limits their use for very large networks because of their numerical inefficiency or their inability to train the models on GPUs. Established R packages such as 'nnet' or 'neuralnet' do not support modern deep learning techniques, such as regularization to control the bias-variance tradeoff (Fritsch et al., 2019; Venables and Ripley, 2002), or more importantly, modern training techniques such as early stopping or learning rate schedulers that can help with otherwise difficult training. The 'h2o' package comes with its own Java backend, and while it allows training with the standard formula syntax, its use in R is cumbersome due to its inability to work with default R objects (LeDell et al., 2022). The 'brulee' R package, which uses 'torch' to train the DNNs specified in standard R syntax, is very similar to the package presented here, but still lacks some critical features (see section "Performance analysis and validation") (Kuhn and Falbel, 2022).

Here, we present 'cito', an R package for training fully-connected neural networks using the standard R formula syntax for model specification. Based on the 'torch' deep learning framework, 'cito' allows flexible specifying of fully-connected neural networks architectures, supports many modern deep learning techniques (e.g. dropout and elastic net regularization, learning rate schedulers), can take advantage of CPU and GPU hardware for parallelization, and yet optionally offers a high degree of customization such as user-defined loss functions. 'cito' furthermore supports many downstream functionalities, such as the possibility to continue the training of existing neural networks with modified training parameters for fine-tuning, or the application of xAI methods to interpret the trained models. As such 'cito' provides an easily useable but nevertheless complete analysis pipeline for building neural networks in R.

In the remainder of the paper, we introduce the design principles of cito in more detail, show validation and performance analysis, and showcase the application of cito using the example of a species distribution model of the African elephant.

## Design of the cito package

### Torch backend

Cito uses 'torch', a variant of PyTorch, as its backend to represent and train the specified neural networks. Until recently, R users who wanted to use PyTorch and Tensorflow had to call their Python bindings through the'reticulate' package. R packages that relied on this pipeline were thus dependent on appropriate Python installations, which often created dependency issues. This issue got solved with the release of 'torch', a native implementation of the torch libraries with a R frontend (Falbel and Luraschi, 2022).

### Building and training neural networks in cito

With 'torch', R users can essentially use PyTorch natively in R, which solves dependency issues, but not the problem that specifying a DNN with 'torch' is complex.

'cito' addresses this problem by providing one simple command, _dnn()_, which combines everything needed to build and train a fully-connected neural network in one line of code. The _dnn()_ function includes options to modify the network architecture, the training process and the monitoring (e.g. by visualization) of the training and validation loss (see Table 1). The function returns an S3 object that can be used, for example, with the _continue_training()_ function to continue training for additional epochs (iterations) with the same or modified training parameters or data. Moreover, many standard R functions such as _summary()_, _predict()_ or _residuals()_ are
implemented for the trained models, and additional specialized explainable Artificial Intelligence (xAI) functions are available for interpreting the fitted networks.

## Performance comparison and validation of cito

After explaining the design of cito, we shortly compare its performance and functionality with the other available packages for implementing neural networks in R. We consider in particular 'nnet' and 'neuralnet', which each have their own backend and are not based on modern DL frameworks (Fritsch et al., 2019; Venables and Ripley, 2002), 'h2o', which possesses a much broader toolkit for training neural networks than the previous two packages (LeDell et al., 2022), and 'brulee' (Kuhn and Falbel, 2022), which, similar to cito, uses the 'torch' DL framework as a backend.

Regarding implemented options, 'cito' offers the most options. 'cito' offers flexible regularization options (to control the bias-variance trade-off), GPU support, the possibility to continue training and custom loss functions and most importantly tools to interpret the trained DNN models (see Table 2).

Looking at the performance of the packages, measured by the time it takes to train the networks, we find that some of the older packages, in particular 'neuralnet', perform better than the torch packages for small networks (see Figure 1). This is probably due to the smaller overhead of these more specialized packages. However, when moving to larger networks (large networks, especially wide networks are important to achieve low generalization

\begin{table}
\begin{tabular}{l l l} \hline \hline Name & Explanation & Default \\ \hline hidden & Quantity and size of hidden Layers & (10,10,10) \\ activation & Activation function for hidden layers & “relu” \\ bias & Should hidden nodes have bias & TRUE \\ \hline \hline
**Training** & & \\ \hline Name & Explanation & Default \\ \hline validation & Split data into test and validation set & 0 \\ epochs & Number of training iterations & 100 \\ device & Set to “cuda” to train on GPU & “cpu” \\ plot & Visualize loss during training & TRUE \\ batchsize & Number of samples used for each training step & 32 \\ shuffle & Shuffle batches in between epochs & TRUE \\ lr & Learning rate & 0.01 \\ early stopping & Stops training early based on validation loss & FALSE \\ \hline \hline \multicolumn{3}{l}{**Controlling bias-variance trade-off (regularization)**} \\ \hline Name & Explanation & Default \\ \hline lambda & Strength of elastic net regularization & 0 \\ alpha & Split of L1 and L2 regularization & 0.5 \\ dropout & Dropout probability of a node & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Changeable parameters for fully-connected neural networks and their default values in ’cito’

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & ‘cito’ & ‘brulee’ & ‘h2o’ & ‘neuralnet’ & ‘nnet’ \\ \hline Customizable network architecture & X & X & X & X \\ Fit a probability distribution & X & & X & \\ GPU support & X & & & \\ Regularization & X & X & & \\ Custom loss function & X & & & X \\ Optimization of additional user-defined parameters & X & & & \\ Continue training & X & & & \\ Class weights for imbalanced data & & X & & \\ Learning rate scheduler & X & X & X & \\ Feature importance (xAI) & X & & & \\ Partial dependency plots (xAI) & X & & & \\ Accumulated local effect plots (xAI) & X & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Feature comparison of R packages used to build fully-connected neural networks
errors (Belkin et al., 2019) 'cito' can play out one the main advantage of modern machine learning frameworks, which is GPU support. On the GPU, training time in city is practically independent of the size of the network, confirming the general consensus that training large networks requires GPU resources. On a CPU, 'cito' performs on par with 'brulee', the other torch-based package, but somewhat worse than 'neuralnet'. We interpret these results as showing that for a simple problem, there is still some overhead of using torch as opposed to a native C implementation. Nevertheless, we argue would argue that the added flexibility and functionality of cityto outweighs this advantage of 'neuralnet'. Moreover, our results suggest that the difference between the torch packages and 'neuralnet' lies mainly in the constant overhead needed to set up the models, and thus for very large models, their performance is roughly equal.

## Workflow and case study

After having discussed the process of model training, which is arguably the core of any machine learning project, we want to comment on the entire workflow of 'cito' in building a predictive model. In general, this workflow consists of model specification, training, and interpretation and predictions (Box 1). To make the discussion of the workflow more accessible to the reader, we show all functions using the example of building a species distribution model for the African elephant (Loxodonta Africana).

As data, we use occurrence records of the African elephant from Angelov, 2020, who compiled data from different studies available on GBIF ((INaturalist Contributors, 2022a, 2022b; Jlegind, 2021; Musila et al., 2019; Navarro, 2022). Those presence-only data were supplemented with randomly sampled background points (pseudo-abscences) to generate a presence-absence signal for the classifier. As predictors, we used standard bioclimatic variables from WorldClim v2 (Fourcade et al., 2018). While it is common in statistical modelling to sample more pseudo-abscences than presences, such unbalanced class numbers can be harmful for machine learning algorithms. We therefore randomly undersampled pseudo-abscences to match the number of observations (another option would be to oversample presences but in our example this resulted in lower accuracy in interim results).

Figure 1: Comparison of different deep learning packages (‘brulee’, ‘h2o’, ‘neuralnet’, and ‘cito’ (CPU and GPU)) on different network sizes on an Intel Xeon 6128 and a Nvidia RTX 2080ti. The networks consist of five equally sized layers (50 to 1000 nodes with a step size of 50) and are trained on a simulated data set with 1000 observations. Panel (A) shows the runtime of the different packages and panel (B) shows the average root mean square error (RMSE) of the models on a holdout of size 1000 observations (RMSE was averaged over different network sizes). Each network was trained 20 times (the dataset was resampled each time).
The building and training can be done in one line of code (Figure 1(a), code provided in Pichler and Amesoder, 2023). The code builds a species-distribution model based on a fully-connected neural network with three hidden layers of 50, 50 and 50 nodes and trains it for 50 epochs. During training, a plot that is not shown here, would visualize the training and validation loss and if the loss does not decrease over time it may be beneficial to stop and restart training with different hyper-parameters (e.g. smaller learning rate). In the second line of code (Figure 1(b)), the resulting object is further trained with a smaller learning rate to achieve a better fit.
The trained models can be interpreted with a range of in-build functions. The _predict()_ can be used to make predict the occurrence probability of the elephant (Figure 3a). _summary()_ function provides a first overview about influential variables by calculating their importance(Fisher et al., 2019). Partial dependency plots (PDP) and averaged local effect plots (ALE) functions can be used to display the effect of specific features on the response, in this case the occurrence probability of the elephant (Figure 3b).

## Conclusion

'cito' is a powerful R package for building and training fully-connected neural networks with a formula syntax. The package seamlessly fits into the R environment and removes many hurdles for inexperienced users, but also saves programming time for experienced users who want to build simple neural networks. Unique features of 'cito', such as training on a GPU, using custom loss functions, modern DL training techniques such as continue training, learning rate scheduler or early stopping cannot be found in other packages. Future releases of 'cito' aim to implement additional functionalities such as internal cross validation for hyperparameter optimization, gradient based methods for hyperparameter tuning and the integration of recurrent and convolutional neural networks.

Figure 3: Predictions for the African elephant from a DNN trained by cito. Panel (A) shows the predicted probability of occurrence of the African elephant. Panel (B) shows the accumulated local effect plot (ALE), i.e. the change of the predicted occurrence probability in response to the Bioclim variable 8 (mean temperature of the wettest quarter).

Figure 2: Training a deep neural network using ’cito’. A shows how ‘cito’ is used to train a model with three hidden layers and B shows how the training can be continued for a trained model.
## Data Availability

The processed datasets for the species distribution model (African elephant) are available at Angelov, 2020. The 'cito' package can be downloaded from CRAN. Scripts to reproduce the analysis and the benchmark are available at Pichler and Amesoder, 2023.

## Authors' Contributions

C.A. implemented the 'cito' software with contributions by M.P. M.P. ran the experiments and analyzed the species distribution model for the present contribution. C.A. provided the first draft of the software note and M.P. and F.H. helped improving it.
