Deep Neural Network Uncertainty Quantification for LArTPC Reconstruction
We evaluate uncertainty quantification (UQ) methods for deep learning applied
to liquid argon time projection chamber (LArTPC) physics analysis tasks. As
deep learning applications enter widespread usage among physics data analysis,
neural networks with reliable estimates of prediction uncertainty and robust
performance against overconfidence and out-of-distribution (OOD) samples are
critical for their full deployment in analyzing experimental data. While
numerous UQ methods have been tested on simple datasets, performance
evaluations for more complex tasks and datasets are scarce. We assess the
application of selected deep learning UQ methods on the task of particle
classification using the PiLArNet [1] monte carlo 3D LArTPC point cloud
dataset. We observe that UQ methods not only allow for better rejection of
prediction mistakes and OOD detection, but also generally achieve higher
overall accuracy across different task settings. We assess the precision of
uncertainty quantification using different evaluation metrics, such as
distributional separation of prediction entropy across correctly and
incorrectly identified samples, receiver operating characteristic curves
(ROCs), and expected calibration error from observed empirical accuracy. We
conclude that ensembling methods can obtain well calibrated classification
probabilities and generally perform better than other existing methods in deep
learning UQ literature.
Dae Heun Koh, Aashwin Mishra, Kazuhiro Terao
# Deep Neural Network Uncertainty Quantification for LArTPC Reconstruction.

###### Abstract

We evaluate uncertainty quantification (UQ) methods for deep learning applied to liquid argon time projection chamber (LArTPC) physics analysis tasks. As deep learning applications enter widespread usage among physics data analysis, neural networks with reliable estimates of prediction uncertainty and robust performance against overconfidence and out-of-distribution (OOD) samples are critical for their full deployment in analyzing experimental data.While numerous UQ methods have been tested on simple datasets, performance evaluations for more complex tasks and datasets are scarce. We assess the application of selected deep learning UQ methods on the task of particle classification using the PiLArNet [1] monte carlo 3D LArTPC point cloud dataset. We observe that UQ methods not only allow for better rejection of prediction mistakes and OOD detection, but also generally achieve higher overall accuracy across different task settings. We assess the precision of uncertainty quantification using different evaluation metrics, such as distributional separation of prediction entropy across correctly and incorrectly identified samples, receiver operating characteristic curves (ROCs), and expected calibration error from observed empirical accuracy. We conclude that ensembling methods can obtain well calibrated classification probabilities and generally perform better than other existing methods in deep learning UQ literature.

+
Footnote †: Corresponding author.
## 1 Introduction

Deep learning has largely established itself as a dominant method for machine learning applications, in part due to its competence in a variety of well-known tasks such as image recognition, natural language processing, and automated control applications. As such, scientists in both artificial intelligence and the physical sciences have been investigating ways to realize deep learning's success in more complex domains of fundamental research. The trend for integrating deep learning for physics data reconstruction has been particularly notable in experimental particle physics, where large data generation from particle detectors such as liquid argon time projection chambers (LArTPCs) and the Large Hadron Collider (LHC) naturally prepare fertile grounds for deep learning models.

Using deep learning for fundamental research, however, presents complications that are often omitted in many common industrial use cases, where practitioners generally attend to achieving state-of-the-art with respect to a family of conventional performance metrics. In particular, one of the most pressing issues with using deep neural networks for fundamental research is developing robust and consistent methods for quantifying uncertainties of its predictions. Deep neural networks are unable to recognize out-of-distribution examples and habitually make incorrect predictions with high confidence for such cases [2; 3]. Uncertainty in predictions has had serious consequences while applying deep learning to high-regret and safety-critical applications such as automated driving [4; 5; 6], law enforcement [7], medical sciences[8], etc. Overconfidence for out-of-distribution examples also
demonstrate the need for deep learning models to acknowledge whether a given prediction is to be trusted or not. Undoubtedly, for deep neural nets to be integrated into the physics measurement process, such characteristics of deterministic neural networks must be addressed by an effective method for uncertainty quantification (UQ).

As demand for UQ gradually escalated in domains such as autonomous driving and medicine, UQ methods diversified into a variety of different approaches under the name of Bayesian Deep Learning (BDL), but with scarce substantial application in the physical sciences. Moreover, most BDL methods have been benchmarked on simplified datasets (MNIST, CIFAR10), which are not representative of the complexity of physics data reconstruction process. Modern accelerator neutrino experiments such as ICARUS and DUNE offer ideal grounds for testing the efficacy of BDL in UQ, due to its recent adaptation and moderate success of deep learning based reconstruction techniques. The benefit derived from a detailed assessment of different UQ algorithms on a complex, multi-objective task such as LArTPC data reconstruction is two-fold: allow practitioners in machine learning to evaluate BDL's applicability in a real-world setting and enable physicists to design neural network that produce well justified uncertainty estimates for rejecting erroneous predictions and detecting out-of-distribution instances.

Practitioners of deep learning in LArTPC reconstruction agree on the need for calibrated uncertainty bounds for deep learning model predictions along with OOD robustness. However, numerous different uncertainty quantification algorithms have been proposed for deep learning. These range from empirical approaches (such as bootstrapped ensembles), Bayesian approaches (such as EDL, HMC) and hybrid approaches (such as MC Dropout). None of these have been tested for complex applications such as LArTPC reconstruction. In this investigation, we select the most promising uncertainty quantification approaches from each of these categories, test and evaluate them with respect to critical intermediate reconstruction tasks: particle classification and semantic segmentation. We first briefly summarize the different methodologies and discuss the apparent advantage/disadvantage of using each of the proposed models in the following section. We describe in detail the monte-carlo generated 3D LArTPC particle image dataset and state any assumptions or additional information that was used to train and evaluate each model. In Section IV, we present quantitative performance evaluation of different UQ models on three different settings of single particle classification, multi-particle classification, and semantic segmentation, using a variety of quantitative metrics to measure UQ fidelity.

## 2 Methods of Uncertainty Quantification in Deep Learning

Among numerous models and studies on uncertainty-quantifying neural networks [9; 10], we focus on methods designed for multi-class classification tasks that require minimal changes to popular neural network architectures. In this paper, we consider three class of UQ methods: model ensembling [11], Monte Carlo Dropout (MCD) [12], and Evidential Deep Learning (EDL) [13; 14].

### Notation

Let \(X=\{x^{(1)},x^{(2)},...,x^{(n)}\}\) and \(Y=\{y^{(1)},y^{(2)},...,y^{(N)}\}\) be data and labels in the training set, and let \(\tilde{X}=\{\tilde{x}^{(1)},\tilde{x}^{(2)},...,\tilde{x}^{(M)}\}\) and \(\tilde{Y}=\{\tilde{y}^{(1)},\tilde{y}^{(2)},...,\tilde{y}^{(M)}\}\) denote the test set. A neural network \(f_{\theta}\), parametrized by weights \(\theta\), is trained on \(D_{train}=\{(x^{(1)},y^{(1)}),...,(x^{(N)},y^{(N)})\}\), with logits
given by \(z^{*}=f_{\theta}(x^{*};X,Y)\) and labels \(\hat{y}^{*}=\operatorname*{argmax}_{c}(f_{\theta}(x^{*};X,Y)_{1},...,f_{\theta}(x ^{*};X,Y)_{c})\), for some \(x^{*}\in X^{*}\subset\tilde{X}\).

### Ensembling Methods

Model ensembling in the context of deep learning models refers to the method of training multiple instances of the same architecture with different random initialization seeds. In Naive Ensembling (NE), one trains each member of the ensemble on the same training dataset, resulting in \(N\) networks with identical architecture but different parameter values. Often, to achieve better generalization and stability, Bootstrapped Ensembling (BE) (or bagging) is preferred over naive ensembling. This is done by training each ensemble member on a dataset reorganized by sampling \(N\) examples from the full training set with replacement. If the size of the resampled dataset is equal to that of the original training set, each ensemble member is expected to see approximately 63% of the original training set. For classification, it is standard to use the most common label among the ensemble members as the final prediction, while for regression one usually computes the empirical mean. When an ensemble consists of a collection of neural networks trained with respect to a _proper scoring rule_[15] and often coupled with an optional adversarial training routine, the ensemble is termed _deep ensembles_[11].

Ensemble methods are the one of the simplest UQ methods that require no additional changes to the underlying model architecture, although the high computational cost in training \(N\) architecturally identical models and performing \(N\) forward passes for one prediction often renders them inapplicable for some memory or time consuming tasks.

### Monte Carlo Dropout

Monte-Carlo Dropout is a bayesian technique introduced in [12], where one approximates the network's posterior distribution of class predictions by collecting samples obtained from multiple forward passes of dropout regularized networks. _Dropout regularization_[16] involves random omissions of feature vector dimension during train time, which is equivalent to masking rows of weight matrices. Inclusion of dropout layers mitigates model overfitting and is empirically known to improve model accuracy [16]. A key observation of [12] is that under suitable assumptions on the bayesian neural network prior and training procedure, sampling \(N\) predictions from the BNN's posterior is equivalent to performing \(N\) stochastic forward passes with dropout layers fully activated. This way, the full posterior distribution may be approximated by monte-carlo integration of the posterior softmax probability vector \(p(\hat{y}^{*}\mid x^{*};X,Y)\):

\[p(\hat{y}^{*}\mid x^{*};X,Y)\approx\frac{1}{T}\sum_{t=1}^{T}\text{Softmax}( \mathbf{f}_{\theta_{t}}(x^{*};X,Y)), \tag{1}\]

where \(T\) denotes the number of stochastic forward passes. As with ensembling methods, the final prediction of MCDropout for classification is given by the majority vote among all stochastic forward passes. For regression, we again compute the empirical mean. As evident from the apparent similarities, MCDropout networks may also be interpreted as a form of ensemble learning [16], where each stochastic forward pass corresponds to a different realization of a trained neural network.

Implementing MCDropout requires one to modify the underlying neural network architecture to include dropout layers and configuring them to behave stochastically during test time. Often
the location of dropout layers can critically affect prediction performance, and for convolutional neural networks the decision is made via trial-and-error [17]. Also, for memory intensive tasks such as semantic segmentation, sample collection by multiple forward passes can accumulate rapidly towards high compuational cost, similar to ensembling methods.

### Evidential Deep Learning

Evidential Deep Learning (EDL) [13, 14], refers to a class of deep neural networks that exploit conjugate prior relationships to model the posterior distribution analytically. For multi-class classification, the distribution over the space of all probability vectors \(\mathbf{p}=(p_{1},...,p_{c})\) is modeled by a Dirichlet distribution with \(c\) concentration parameters \(\alpha=\alpha_{1},...,\alpha_{c}\):

\[D(\mathbf{p}\mid\alpha)=\frac{1}{B(\alpha)}\prod_{i=1}^{c}p_{i}^{\alpha_{i}-1}, \tag{2}\]

where \(\alpha_{i}\geq 1\) for all \(i\), \(B(\cdot)\) denotes the \(c\)-dimensional multinomial Beta function, and \(\mathbf{p}\) is in the \(c\)-unit simplex \(\mathcal{S}_{c}\):

\[\mathcal{S}_{c}=\{\mathbf{v}\in\mathbb{R}^{c}:\sum_{i=1}^{c}v_{i}=1\}. \tag{3}\]

In constrast to deterministic classification neural networks that minimize the cross-entropy loss by predicting the class logits, evidential classification networks predict the concentration parameters \(\alpha=(\alpha_{1},...,\alpha_{c})\). The expected value of the \(k\)-th class probability under the distribution \(D(\mathbf{p}\mid\alpha)\) is then given analytically as

\[\hat{p}_{k}=\frac{\alpha_{k}}{S},\quad S=\sum_{i=1}^{c}\alpha_{i}. \tag{4}\]

To estimate the concentration parameters, several distinct loss functions are available as training criteria. The _marginal likelihood loss_ (MLL) is given by:

\[\mathcal{L}_{MLL}(\theta)=-\log\left(\int\prod_{i=1}^{c}p_{i}^{y_{i}}D( \mathbf{p}\mid\alpha)\ d\mathbf{p}\right). \tag{5}\]

The _Bayes risk_ (posterior expectation of the risk) of the _log-likelihood_ (BR-L) formulation yields:

\[\mathcal{L}_{BR}(\theta)=\int\left[\sum_{i=1}^{c}-y\log\left(p_{i}\right) \right]D(\mathbf{p}\mid\alpha)\ d\mathbf{p}. \tag{6}\]

The _Bayes risk_ of the _Brier score_ (BR-B) may also be used as an alternative optimization objective:

\[\mathcal{L}_{BS}(\theta)=\int\left\|\mathbf{y}-\mathbf{p}\right\|_{2}^{2}\ D( \mathbf{p}\mid\alpha)\ d\mathbf{p}. \tag{7}\]

From Sensoy et. al. [13], analytic integration of the aforementioned loss functions give closed form expressions that are suited for gradient based optimization of the parameters \(\theta\).

EDL methods have the immediate advantage of requiring only one single pass to access the full posterior distribution, at a price of restricting the space of posterior functions onto the appropriate conjugate prior forms. Also, EDL methods only require one to modify the loss function and the final
layer of its deterministic baseline (if necessary), which allows flexible integration with complex, hierarchical deep neural architectures similar to the full LArTPC reconstruction chain. However, due to the strong assumptions made on the posterior analytical form, EDL methods are limited to classification and regression tasks as of now. As we later observe, EDL methods generally fall short on various UQ evaluation metrics compared to ensembling and MCDropout, depending on task specifics.

## 3 Evaluating Uncertainty Quantification Methods

### Evaluation Metrics

As stated in [11], the goal for uncertainty quantification for deep learning models is two-fold: to achieve better alignment of predicted confidence probability with their long-run empirical accuracy and to serve as mis-classification or out-of-distribution alarms that could be used for rejecting unconfident predictions. The first condition, which we term _calibration fidelity_, may be evaluated by plotting the _reliability diagrams_[18], which are constructed by binning the predicted probabilities (often termed _confidence_) into equal sized bins and plotting the bin centers in the \(x\)-axis and the empirical accuracy of the bin members in the \(y\)-axis. The closer the reliability diagram is to the diagonal, the more desirable a given classifer is, in the sense of calibration fidelity. The deviation of a given classifier from the diagonal could be summarized by computing the _adaptive calibration error_ (ACE) [19]:

\[ACE=\frac{1}{K}\frac{1}{R}\sum_{k=1}^{K}\sum_{r=1}^{R}|acc(r,k)-conf(r,k)|. \tag{1}\]

Here, \(K\) denotes the number of unique classes and \(R\) denotes the number of equal-sample bins used to plot the reliability diagram for class \(k\), given by confidence \(conf(r,k)\) and corresponding empirical accuracy \(acc(r,k)\). Although the _expected calibration error_ (ECE) [20] is more widely known, we observed in practice that static binning schemes such as ECE is suboptimal for highly skewed predictive probability distributions common to accurate models.

As calibration fidelity measurements using reliability diagrams are originally designed for binary classifiers, there have been numerous proposals for their extensions to multi-class classifiers [21; 22; 23]. We consider two relatively simple methods; the first is a standard used in Guo et. al. [21], where only the predicted probability for the most confident prediction of each sample is used to plot the reliability diagram. We refer to this mode of assessment as _max-confidence_ calibration fidelity. An alternative method is to evaluate calibration for each of the \(K\) classes separately, as in B. Zadrozny and C. Elkan [23]. We refer to this mode as _marginal_ calibration fidelity.

Another metric of uncertainty quantification measures the model's _discriminative capacity_ to mis-classified or out-of-distribution samples. In practice, uncertainty quantification models have the capacity to reject predictions based on a numerical estimate of the trustworthiness of the prediction in question. For example, in a classification setting the entropy of the predicted softmax probability distribution (_predictive entropy_) could be used as a measure of confusion, as entropy is maximized if the predictive distribution reduces to a uniform distribution over \(K\) classes. In this construction, it is desirable to have the predicted entropy distributions of correctly and incorrectly classified samples to be as separated as possible. To compute the extent of distributional separation, we may use the
first Wasserstein distance [24] between the predictive entropy distributions:

\[W_{1}(u,v)=\inf_{\pi\in\Gamma(u,v)}\int_{\mathbb{R}\times\mathbb{R}}|x-y|\;d\pi(x,y). \tag{10}\]

where \(u\) and \(v\) are two probability distributions, \(\Gamma(u,v)\) is the set of all joint probability measures in \(\mathbb{R}^{2}\). We use the Wasserstein distance with the \(L_{1}\) metric due to its simple computational implementation [24].

Sensitivity may also be measured by computing the area under the receiver operating characteristic curve (AUROC), also known as the concordance statistic (\(c\)-statistic) [25]. Using predictive entropy as the thresholding value, the ROC curve is constructed by plotting the false positive rate (incorrect predictions) in the \(x\)-axis and the true positive rate (correct predictions) in the \(y\)-axis at different threshold levels. In this setting, the AUROC is the probability that a randomly chosen correct prediction will have a lower predictive entropy than that of a randomly chosen incorrect prediction [26].

## 4 Datasets and Network Architectures

**Single Particle Classification**: We first implement and assess the different UQ models on the simpler task of single particle classification. The single particle dataset consists of

Figure 1: Sparse-CNN layer definitions for architecture reference.

Figure 2: Sparse-CNN block definitions for architecture reference.
Figure 4: Sparse-CNN architecture for semantic segmentation networks.

Figure 5: Architecture outline of multi-particle classification network. The geometric node encoder extracts hand-engineered features relevant to particle classification, such as orientation matrix and major PCA axes.

Figure 3: Sparse-CNN architecture for single particle classifiers.
1024 3D images each containing only one particle, where all voxels in the given image belong to the same particle ID. The 3D images have one feature dimension corresponding to the amount of energy deposited in a one-voxel equivalent region of the detector. We use a ResNet [27] type encoder with dropout [16] regularization, where convolution operations are substituted by sparse convolutions implemented in the _MinkowskiEngine_ library [28]. For standard deterministic models, ensembles, and MCDropout, the final prediction probabilities are given by softmax activations, whereas for evidential models the concentration parameters \(\alpha\) are computed from Softplus [29] activations. The single particle dataset contains five particle classes: photon showers (\(\gamma\)), electron showers (\(e\)), muons (\(\mu\)), pions (\(\pi\)), and protons (\(p\)).

**Semantic Segmentation** As segmentation is a classification task on individual pixels, the details of the implementation are mostly identical to those of single particle classification. We employ _Sparse-UResNet_[30] with dropout layers in the deeper half of the network as the base architecture for semantic segmentation networks and use the 768px resolution PILArNet [1] MultiPartRain (MPR) and MultiPartVertex (MPV) datasets for multiple particle datasets. The five semantic labels provided by PiLArNet consists of the following:

* Shower Fragments: connected components of electromagnetic showers that are above a set voxel count and energy deposition threshold.
* Tracks: particle trajectories that resemble stright lines, mostly originating from muon, pion, and protons.
* Michel Electrons: an electron produced by muon decay at rest.
* Delta Rays: electrons produced from muon tracks via hard scattering
* Low Energy Depositions: cloud of low energy depositions of electromagnetic showers which are not labeled as shower fragments.

**Multi Particle Reconstruction** The MPV/MPR dataset also contains particle type labels for each particle instance in a given image. For multi particle classification, we take each cluster of voxels that belong to the same particle and reduce the resulting groups of point cloud into 1-dimensional feature vectors. The node embeddings of each particle consists of geometric features such as its principal component vectors. These feature vectors are then given as input node features to a graph neural network, which performs three message passing operations to incorporate inter-particle relational information.

## 5 Results

### Training Details

The training set consists of 80k images, and the test set were separated to a 2k validation set used for model selection and a 18k test set used for selected model evaluation with high statistics. All models were trained until the validation accuracy plateaued, and the network weights that achieved the highest validation accuracy were selected for further evaluation on a separate test set. To fully account for possible variations in model accuracy and uncertainty quantification quality due to
randomized factors such as parameter initialization, the model selection procedure were repeated for five different random seeds for each model. This results in five independently trained models that share the same architecture but differing in parameter values. We used the Adam optimizer [31] with decoupled weight decay [32].

**Single Particle Classification**: Figure 6 shows the predictive entropy distribution, accuracy, and the \(W_{1}\) distance for single particle classification models. We observe that the distributional separation as measured in \(W_{1}\) is largest for the ensemble methods, while evidential model trained on the Brier score is also competitive. In general, ensemble methods achieve highest accuracy with better distributional separation compared to monte-carlo dropout and evidential models. The AUROC values in figure 8 also reflect the superior discriminative capacity of ensembling.

The calibration curves for single particle classification are shown in the top row of figure 15, and figure 7 illustrates the adaptive calibration error (ACE) values across different subsets of the test set partitioned by true particle id labels. While all UQ models with the possible exception of EDL-BR-B achieve better calibration compared to standard deterministic neural networks, ensembling methods have the least max-confidence and marginal ACE values.

**Semantic Segmentation**: For segmentation, the best distributional separation is achieved by evidential models, which are evident in Figure 9. The ensemble methods have the highest accuracy and AUROC scores as is shown in figure 10. It is interesting to note that while distributional separation measured in \(W_{1}\) is greatest for evidential models, the calibration fidelity falls short even with respect to standard deterministic models. As with single particle classification, the best calibration fidelity is realized by ensemble methods.

**Multi Particle Reconstruction**: Since contextual information which are useful in determining

Figure 6: Predictive entropy distribution, accuracy, and 1-Wasserstein distance for single particle classification.
Figure 8: Single particle ROC and percentage rejection curves.

Figure 7: Single particle classification adaptive calibration errors (ACEs) for each model and class.

Figure 9: Predictive entropy distribution, accuracy, and 1-Wasserstein distance for semantic segmentation.
a given particle's ID can only be used in a multi-particle setting, we expect a gain in accuracy from the single particle datasets. This approach leads to an overall approximate 5% increase in classification accuracy in all models. Again, ensemble methods provide the highest \(W_{1}\) distance, overall accuracy, and AUROC values (figures 12, 14) and the best calibration fidelity (figure 13).

The full reliability plots used to calculate ACE values are provided in figures 15 and 16. A tabular summary of results is available in table 1.
Figure 14: Multi particle ROC and percentage rejection curves.

Figure 12: Predictive entropy distribution, accuracy, and 1-Wasserstein distance for multi particle classification.

Figure 13: Multi particle classification adaptive calibration errors (ACEs) for each model and class.
Figure 16: Reliability plots for semantic segmentation.

Figure 15: Reliability plots for single and multi-particle classification.
## 6 Conclusion

In this paper, we have proposed a new method for constructing the \(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O( }\mathcal{O(\mathcal{O}}(\mathcal{O(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O( }(\mathcal{O(\mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O(\mathcal{O}}(\mathcal{O(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O}( \mathcal{O}(\mathcal{O}(\mathcal{O( }(\mathcal{O(\mathcal{O}}( \mathcal{O(\mathcal{O}}( \mathcal{O(\mathcal{O}}( \mathcal{O(\mathcal{O( }(\mathcal{O( }O(\mathcal{O( }( \mathcal{O( }O( \)( \mathcal{O}(( \mathcal{O( ( ( ( } (\)(\mathcal{(( } ( ( ( ( ( ( ( ( ( )}()}(()((())((((()))
## 6 Conclusions

In this paper, we have presented a new method for computing
We include a brief summary of the required GPU memory and time complexity of each model in tables 3 and 2. The batch size is denoted in the Train/Test subcolumn. The time information corresponds to the CPU time it takes for the model to run one iteration of training or evaluation routine with the denoted batch size. For training, this includes the time required for both model forwarding and gradient backpropagation, while for inference we compute the sum of evaluation mode model forwarding time and other post-processing operations (for example, in MC Dropout we have a sample averaging procedure needed to obtain probability values). The memory value is computed by taking the average of the maximum required GPU memory across 5 different samples. Note that the values for deterministic and naive ensembles are identical, since naive ensembles were constructed from trained deterministic models.

## 6 Categorization of Error Types

Calibration fidelity cannot be examined in a single image basis, as calibration is a collective property that must be measured by appropriate binning of the test set predictions. However, it is possible to assess the discriminative capacity by observing samples with antipodal entropy values. With predictive entropy values in hand, the class predictions may be divided into four categories: 1) confident (low entropy) correct predictions, 2) uncertain (high entropy) correct predictions, 3) confident errors, and 4) uncertain errors. Among the four groups, confident errors are most problematic for robust design of deep learning models. Some representative examples are shown in figures 17, 18, and 19. Figure 17 is a high-entropy misclassification example, in which the network cannot confidently decide whether the set of voxels circled in red is an electron, muon, or a pion, in constrast with the confident predictions it gives for the pair of photons in and the vertex-attached proton. In figures 18 and 19, the network predicts the vertex-attached shower as an electron with high probability, while for the two \(\mu\) and \(\pi\) pair it retains some level of uncertainty. Hence, we observe that the assessment of the network on mis-identifying the shower as an electron is partly justified, as it is difficult to distinguish a photon shower attached to an interaction vertex from an electron shower.

## 7 Discussion

We evaluated three different uncertainty quantification methods for deep neural networks on the task of single particle classification, multi-particle classification, and semantic segmentation using high resolution 3D LArTPC energy deposition images. The various metrics evaluating calibration fidelity and discriminative capacity leads to a notable conclusion: simple ensembling of few independently trained neural networks generally achieve highest accuracy and best calibration of output probability values. Also, we observe that the quality of uncertainty quantification depends greatly on the type of the classifier's task, and often it is possible for Bayesian models to perform worse than deterministic networks in calibration.

Often, the choice made in hyperparameters and neural network architecture significantly affects the classifier's capacity to achieve the desired performance. It is important to note that the UQ methods presented in this paper does not include _structural_ and _hyperparameter_ of our models. Extant deep learning uncertainty quantification approaches can only account for aleatoric uncertainty and the parameter uncertainty component of epistemic uncertainty. Thus, these methods are unable
to account for structural (or model form) uncertainty, that is a component of epistemic uncertainty. While a complete description of epistemic uncertainty is often intractable in practice, it is desirable to assess how much of the variability in a deep classifier's predictions could be attributed to hyperparameter and structural diversity.

While out-of-distribution and mis-classification resilience of uncertainty quantifying neural nets may be used for rejecting unreliable predictions, obtaining calibrated probability estimates would provide further credibility in using deep learning techniques for physical sciences. Post-hoc calibration methods such as temperature scaling [21] train a calibration model (for temperature

Figure 17: Example high-entropy error from a multi-particle evidential GNN (Bayes Risk). The particles that does not originate from a vertex are omitted and are colored in dark navy.

Figure 18: Example low-entropy prediction from a multi-particle evidential GNN (Bayes Risk).
scaling, a single parameter) after training to obtain calibrated probabilities for a deterministic neural network. As post-hoc methods do not require the classifier to be re-modeled and trained from its initial state, such methods may be better suited for ensuring proper calibration of classifiers with lower computational budget. Future work will include evaluation of uncertainty quantifying neural nets and post-hoc calibration methods for a full neutrino physics signal/background classifier, which is built on top of the separate tasks of particle classification and segmentation.

## Acknowledgment

This work was supported in part by funding from Zoox, Inc. This work is supported by the U.S. Department of Energy, Office of Science, Office of High Energy Physics, and Early Career Research Program under Contract DE-AC02-76SF00515.
