On the Temperature of Bayesian Graph Neural Networks for Conformal
  Prediction
Accurate uncertainty quantification in graph neural networks (GNNs) is
essential, especially in high-stakes domains where GNNs are frequently
employed. Conformal prediction (CP) offers a promising framework for
quantifying uncertainty by providing $\textit{valid}$ prediction sets for any
black-box model. CP ensures formal probabilistic guarantees that a prediction
set contains a true label with a desired probability. However, the size of
prediction sets, known as $\textit{inefficiency}$, is influenced by the
underlying model and data generating process. On the other hand, Bayesian
learning also provides a credible region based on the estimated posterior
distribution, but this region is $\textit{well-calibrated}$ only when the model
is correctly specified. Building on a recent work that introduced a scaling
parameter for constructing valid credible regions from posterior estimate, our
study explores the advantages of incorporating a temperature parameter into
Bayesian GNNs within CP framework. We empirically demonstrate the existence of
temperatures that result in more efficient prediction sets. Furthermore, we
conduct an analysis to identify the factors contributing to inefficiency and
offer valuable insights into the relationship between CP performance and model
calibration.
Seohyeon Cha, Honggu Kang, Joonhyuk Kang
# On the Temperature of Bayesian Graph Neural Networks for Conformal Prediction

###### Abstract

Accurate uncertainty quantification in graph neural networks (GNNs) is essential, especially in high-stakes domains where GNNs are frequently employed. Conformal prediction (CP) offers a promising framework for quantifying uncertainty by providing _valid_ prediction sets for any black-box model. CP ensures formal probabilistic guarantees that a prediction set contains a true label with a desired probability. However, the size of prediction sets, known as _inefficiency_, is influenced by the underlying model and data generating process. On the other hand, Bayesian learning also provides a credible region based on the estimated posterior distribution, but this region is _well-calibrated_ only when the model is correctly specified. Building on a recent work that introduced a scaling parameter for constructing valid credible regions from posterior estimate, our study explores the advantages of incorporating a temperature parameter into Bayesian GNNs within CP framework. We empirically demonstrate the existence of temperatures that result in more efficient prediction sets. Furthermore, we conduct an analysis to identify the factors contributing to inefficiency and offer valuable insights into the relationship between CP performance and model calibration.

## 1 Introduction

Graph neural networks (GNNs) have demonstrated their impressive abilities to learn from graph-structured data in a wide range of domains, including social sciences, chemistry, knowledge graphs, and recommendation systems [18; 34; 12; 39; 8]. Meanwhile, with the increasing applications of GNNs in safety-critical tasks, there is a growing demand for trustworthy and reliable uncertainty estimates from GNN outputs [14; 37]. However, many existing uncertainty quantification methods are not applicable to GNNs, since they rely on independent and identically distributed (i.i.d.) data assumption, which is violated in graph-structured data [1; 16]. Recent studies have addressed uncertainty in graph-related tasks, but these approaches introduce additional model architecture for density estimation or knowledge distillation. [31; 43].

Conformal prediction (CP) [36] is a promising framework for obtaining uncertainty estimates under the only assumption of data exchangeability1. CP constructs prediction sets that are guaranteed to contain a ground-truth with a desired coverage level. Formally, consider a training dataset \((x_{i},y_{i})\in\mathcal{X}\times\mathcal{Y},\ i\in\{1,\cdots,n\}\), and a test point \((x_{n+1},y_{n+1})\) drawn exchangeably from an underlying distribution \(P\). Then, for a given a pre-defined coverage \(1-\alpha\), CP generates a prediction set \(\mathcal{C}_{n,\alpha}(x_{n+1})\subseteq\mathcal{Y}\) for a test input \(x_{n+1}\) that satisfies

Footnote 1: The data exchangeability assumes that every permutation of data samples have the same probability, which is a weaker assumption than i.i.d assumption.

\[\mathbb{P}[y_{n+1}\in\mathcal{C}_{n,\alpha}(x_{n+1})]\geq 1-\alpha \tag{1}\]

where the probability is taken over \(n+1\) data samples \(\{(x_{i},y_{i})\}_{i=1}^{n+1}\). A prediction set satisfying the coverage condition in the equation (1) is said to be _valid_. While CP guarantees a valid prediction for
any classifier, the size of a prediction set, called _inefficiency_, is largely affected by the underlying classifier and the data generating process [7, 40].

Meanwhile, Bayesian learning [3, 9, 38] (see also [29, Ch. 12] for an overview) is a traditional way for building a predictive interval. In contrast to frequentist learning, which aims to find an optimal point of model parameters, Bayesian learning produces a posterior estimate for model parameters. Consequently, it can naturally generate credible regions by simply taking \((1-\alpha)\cdot 100\) % confidence region of the posterior distribution. However, the region can only be _valid_ without model misspecification, which means that prior, likelihood, and computational capability should be correctly specified in posterior inference [23, 38]. To address the issue of model misspecification when constructing a posterior credible region, a recent study has introduced a scaling parameter that controls the spread of posterior [33] (see also [19] for related discussion on _generalized Bayesian learning_). Their findings demonstrate that the control of the scaling parameter can yield a credible region with an appropriate size but provides approximate validity, whereas CP offers a rigorous guarantee of validity.

Building upon insights from previous works, we study the benefits of Bayesian GNNs in CP framework to obtain valid and efficient uncertainty estimates. While recent CP approaches for GNNs have focused on ensuring validity and reducing inefficiency, these methods are applicable to specific tasks [6, 41] or require correction dataset for additional GNN training [15]. In our work, we propose the use of a temperature parameter in Bayesian GNNs to allow for flexible control of inefficiency within CP framework. We show that Bayesian GNNs improve the performance of CP, especially the inefficiency, compared to frequentist GNNs while preserving the validity. Furthermore, our experiments on both node classification and graph classification tasks demonstrate the existence of temperatures of Bayesian GNNs that lead to more efficient prediction sets. While previous studies have explored the temperature parameter in Bayesian learning mainly for calibration purposes [33, 42], our study investigates the impact of temperatures on CP performance. Moreover, we provide an analysis that explores the connection between the CP performance and model calibration, providing valuable insights on the varying inefficiency depending on the temperature.

The main contributions of this study are summarized as follows:

* We show that Bayesian GNNs outperforms in CP compared to frequentist GNNs.
* Our experiments on graph-structured data demonstrate the existence of temperatures that result in better efficiency2. Footnote 2: Throughout the paper, we use the terms “lower inefficiency” and “better efficiency” interchangeably.
* We analyze the connection between the inefficiency and model calibration to verify the factors that contribute to the inefficiency.

Figure 1: **Conformal prediction (CP) of Bayesian GNNs with tempered posteriors. In Bayesian GNNs, the temperature parameter \(\beta\) controls the spread of the posterior over model parameters \(\theta\). Once we train Bayesian GNNs with a specific temperature, CP generates prediction sets by comparing softmax output of a test sample to a threshold obtained from calibration set. In the right box of this figure, the blue bar represents the true label, and CP guarantees that prediction sets contain the true label with a pre-defined coverage. We show that Bayesian GNNs with an appropriate temperature produce more efficient prediction sets, reducing inefficiency, while keeping the desired coverage.**
Background

### Conformal Prediction

Conformal prediction (CP) [36] aims to construct a set of candidate labels that is likely to contain the true label. Formally, given a test example \(x\in\mathcal{X}\) and for every candidate label \(y\in\mathcal{Y}\), CP either rejects or accepts the candidate pair \((x,y)\) to be included in a prediction set. This decision is done based on a statistic, called _non-conformity score_\(s((x,y)|\theta_{\mathcal{D}})\), a real-valued function that takes input-output pair \((x,y)\) as an input given a pre-trained model parameter \(\theta_{\mathcal{D}}\). The score \(s((x,y)|\theta_{\mathcal{D}})\) quantifies the disagreement between data sample \(x\) and a candidate label \(y\). One typical example of non-conformity score broadly used in classification task is negative log-likelihood (\(-\log p_{\theta}(y|x)\)).

In this work, we use one most common type of CP, called _split conformal prediction (SCP)_[36], which splits available dataset into training and calibration sets. Given a trained model on training set \(\mathcal{D}\), SCP creates a prediction set \(\mathcal{C}_{n,\alpha}(x)\) for a test input \(x\) using the calibration set \(\{(x_{i},y_{i})\}_{i=1}^{n}\). Specifically, we first compute the non-conformity scores of calibration samples as \(\{s((x_{i},y_{i})|\theta_{\mathcal{D}})\}_{i=1}^{n}\). Then, the \((1-\alpha)\)-quantile of the set \(\{s((x_{1},y_{1})|\theta_{\mathcal{D}}),\ldots,s((x_{n},y_{n})|\theta_{ \mathcal{D}})\}\cup\{\infty\}\) becomes the threshold that determines whether to accept or reject candidate label \(y\in\mathcal{Y}\) to be contained in the set. The threshold can be obtained by selecting \(\lceil(1-\alpha)(n+1)\rceil\)-th smallest value from the set \(\{s((x_{1},y_{1})|\theta_{\mathcal{D}}),\ldots,s((x_{n},y_{n})|\theta_{ \mathcal{D}})\}\cup\{\infty\}\), denoted as \(Q_{\alpha}(\{s((x_{i},y_{i})|\theta_{\mathcal{D}})\}_{i=1}^{n})\). Consequently, the corresponding prediction set for a test sample \(x\) is defined as

\[\mathcal{C}_{n,\alpha}(x)=\{y\in\mathcal{Y}:s((x,y)|\theta_{\mathcal{D}})\leq Q _{\alpha}(\{s((x_{i},y_{i})|\theta_{\mathcal{D}})\}_{i=1}^{n})\}\,. \tag{2}\]

**Theorem 1**: _Given that calibration set \(\{(x_{i},y_{i})\}_{i=1}^{n}\) and a test data point \((x,y)\) are exchangeable random variables, for any coverage level \(1-\alpha\), for any trained model \(\theta_{\mathcal{D}}\), and for any nonconformity score \(s(\cdot|\theta_{\mathcal{D}})\), a conformal prediction set \(\mathcal{C}_{n,\alpha}(x)\) defined in the equation (2) satisfies the equation (1) [36]._

Theorem 1 provides a marginal coverage guarantee for all \(x\) on average. Roughly speaking, the SCP can induce any classifier to produce prediction sets guaranteed to satisfy the desired coverage level. This implies that, on average, these sets have a pre-defined probability of containing the true label.

### Bayesian Learning

We introduce two learning paradigms, _frequentist learning_ and _Bayesian learning_, to find a classifier which predicts an output \(y\in\mathcal{Y}\) for a given input \(x\in\mathcal{X}\). A conventional learning that finds an optimal model parameter \(\theta\) and the corresponding single classifier \(p(y|x,\theta)\) given training set \(\mathcal{D}=\{x_{i},y_{i}\}_{i=1}^{n}\), is called _frequentist learning_[25]. In general, it optimizes the model parameter by minimizing the training loss as

\[\theta^{*}=\operatorname*{argmin}_{\theta}L_{\mathcal{D}}(\theta)=\sum_{(x,y) \in\mathcal{D}}-\log p(y|x,\theta). \tag{3}\]

Meanwhile, _Bayesian learning_[25; 3; 29] updates prior beliefs about model parameter \(\theta\) that fits the dataset \(\mathcal{D}\) while accounting for a prior belief \(p(\theta)\) about the values of \(\theta\). Instead of providing a single point estimate for \(\theta\), it infers a posterior distribution over possible values of parameters \(\theta\) based on Bayes' rule by incorporating likelihood from data and prior knowledge on \(\theta\). Hence, it is readily capable of capturing and quantifying the uncertainty about model parameter \(\theta\). Concretely, the Bayesian posterior can be obtained by minimizing the _free energy criterion_[20; 17],

\[q_{\beta}^{*}(\theta|\mathcal{D})=\operatorname*{argmin}_{q}\mathbb{E}_{q( \theta)}[L_{\mathcal{D}}(\theta)]+\beta\cdot\text{KL}[q(\theta)||p(\theta)], \tag{4}\]

where \(\text{KL}[q(\theta)||p(\theta)]\) is the Kullback-Leibler (KL) divergence between two distributions \(q\) and \(p\), and \(\beta>0\) is the _temperature_ parameter which creates a tempered posterior \(q_{\beta}^{*}(\theta|\mathcal{D})\).

One advantage of posterior inference in Bayesian learning is that it can naturally produce credible regions from the posterior. For example, we simply can take 90 % confidence region of posterior distribution if we want to get a _well-calibrated_ (or _valid_) region which satisfies the pre-defined coverage probability of 90 % to contain a target output in the region, as in the equation (1). However, it fails in common when the model is _misspecified_[10; 23; 19]. Model misspecification occurs from
three assumptions in posterior inference, which are prior belief, likelihood, and computational power required to perform inference on intractable posterior [19]. A model might converge to an incorrect solution if a true model is not included in the prior hypothesis space, or even if the space contains the true solution, a model may struggle to find a good solution unless it benefits from reasonable inductive biases from the likelihood [38]. Hence, if there is model misspecification, the resulting posterior can be poorly calibrated and creates undesirable uncertainty quantification. (refer to Fig. 3 in [19]). There has been a study that introduces a scaling parameter to address the model misspecification when constructing a posterior credible region [33]. The scaling parameter controls the spread of the posterior distribution for the credible region to be _efficient_ in terms of its size and approximately _well-calibrated_. Note that our work differs from previous studies in that we investigate how the temperature parameter \(\beta\) affects the size of prediction sets obtained by CP when applied on top of Bayesian learning.

## 3 Conformal Prediction for Bayesian GNNs

### Bayesian GNNs with Tempered Posterior

We begin with the explanation on how to implement Bayesian graph convolutional networks (GCNs). We use Graph DropConnect (GDC), a stochastic regularization technique with adaptive connection sampling [13]. The adaptive connection sampling is a generalized technique that encompasses a range of other regularization and sampling techniques, including DropOut [30], DropEdge [27], and Node Sampling [5]. It operates by applying binary random masks to the adjacency matrix, randomly masking edges, nodes, and channels. They show that connection sampling in GDC can be interpreted as Bayesian extensions of GNNs by transferring randomness from sampling to the model parameter space. Consequently, by implementing GDC within GNNs, we are able to get an appropriate posterior estimate for GNN model parameters and further adjust its temperature.

In detail, for every edge \(e=(v,u)\in\mathcal{E}\), we generate a random mask vector denoted as \(\mathbf{z}_{v,u}^{(l)}\in\{0,1\}^{1\times f_{l}}\) by sampling from Bernoulli distribution with a drop rate \(\pi_{l}\), denoted as \(\text{Bern}(\pi_{l})\). Here, \(f_{l}\) represents the number of features in the \(l\)-th layer of GNN. We define the degree of node \(v\) as \(c_{v}\) and its neighborhoods as \(\mathcal{N}(v)\). Then, for every \(l\)-th layer with the weight parameter matrix \(\mathbf{w}^{(l)}\) and activation function \(\sigma(\cdot)\), we can represent its output feature of each node \(v\) when connection sampling is applied as the equation (5) where \(\mathbf{w}_{v,u}^{(l)}:=\text{diag}(\mathbf{z}_{v,u}^{(l)})\mathbf{w}^{(l)}\) and \(\mathbf{h}_{u}^{(l)}\) denote output feature of node \(u\) at \(l\)-th layer, resulting in learning different random weight parameter \(\mathbf{w}_{v,u}^{(l)}\) for each edge \((v,u)\in\mathcal{E}\), i.e.,

\[\mathbf{h}_{v}^{(l+1)}=\sigma\left(\frac{1}{c_{v}}(\sum_{u\in\mathcal{N}(v) \cup\{v\}}\mathbf{z}_{v,u}^{(l)}\odot\mathbf{h}_{u}^{(l)})\mathbf{w}^{(l)} \right)\,=\sigma\left(\frac{1}{c_{v}}\sum_{u\in\mathcal{N}(v)\cup\{v\}} \mathbf{h}_{u}^{(l)}\mathbf{w}_{v,u}^{(l)}\right). \tag{5}\]

In Bayesian GNNs, we estimate the posterior distribution of GNN weight parameters considering them as random parameters. Since GDC incorporates random parameters including weight parameters \(\mathbf{W}^{(l)}=\{\mathbf{w}_{c}^{(l)}\}_{e=1}^{|\mathcal{E}|}\), random masks \(\mathbf{Z}^{(l)}=\{\mathbf{z}_{e}^{(l)}\}_{e=1}^{|\mathcal{E}|}\), and the corresponding dropout rate \(\pi_{l}\), we infer posteriors on these parameters, denoted as \(\boldsymbol{\theta}=\{\mathbf{W}^{(l)},\mathbf{Z}^{(l)},\pi_{l}\}_{l=1}^{L}\). Note that the drop rates are also learnable to enhance the flexibility of the model [4]. Then, the free energy criterion for training Bayesian GNNs is defined as

\[F_{\beta}(q)=\mathbb{E}_{q(\boldsymbol{\theta})}\left[\sum_{(x,y)\in\mathcal{D }}-\log p(y|x,\boldsymbol{\theta})\right]\,+\beta\cdot\sum_{l=1}^{L}\text{KL} \left[q(\boldsymbol{\theta})||p(\boldsymbol{\theta})\right] \tag{6}\]

where the _temperature_ parameter \(\beta\) controls the spread of posterior distribution, as illustrated in Fig. 1. The posterior with \(\beta=1\) corresponds to standard Bayesian posterior, and the \(\beta\gg 1\) and \(\beta\ll 1\) depend more on the prior assumption and information from data, respectively. Consequently, the _tempered posterior_ can be obtained by minimizing the equation (6) as

\[q_{\beta}(\boldsymbol{\theta}|\mathcal{D})\propto p(\boldsymbol{\theta})\prod_{ (x,y)\in\mathcal{D}}p(y|x,\boldsymbol{\theta})^{1/\beta}. \tag{7}\]

Though estimating the posterior distribution in the equation (7) requires intractable computation, _variational inference (VI)_[9] suggests a good approximation for it, where the variational distribution
from a defined family of distributions (e.g., Gaussian distributions) approximates true posterior distribution. Details on prior assumptions and VI algorithm for GDC are stated in Appendix D. Suppose we have the tempered posterior \(q_{\beta}(\mathbf{\theta}|\mathcal{D})\) or its VI approximate, then the predictive distribution on a test sample \(x\) can be obtained by averaging over all likely models as

\[p_{\beta}(y|x,q)=\int p(y|x,\mathbf{\theta})q_{\beta}(\mathbf{\theta}|\mathcal{D})d\mathbf{ \theta}\approx\frac{1}{T}\sum_{t=1}^{T}p(y|x,\hat{\mathbf{\theta}}_{t}) \tag{8}\]

where the distribution can be approximated by the average over \(T\) sets of i.i.d. samples \(\hat{\mathbf{\theta}}_{t}\) drawn from \(q_{\beta}(\mathbf{\theta}|\mathcal{D})\). Since we use negative log-loss as non-conformity score for CP, i.e., \(-\log p_{\beta}(y|x,q)\), the prediction sets, especially the size of sets is directly affected by the temperature. Accordingly, we discuss the adjustment of temperature of Bayesian GNNs for CP in the following.

### Temperature Control for Conformal Prediction

We now present how to exploit the benefits of Bayesian GNNs with tempered posterior in split conformal prediction (SCP). As illustrated in Fig. 1, we first train Bayesian GNNs using loss function in the equation (6) with a pre-determined temperature \(\beta\) using training set. Afterward, for a given calibration set \(\{(x_{i},y_{i})\}_{i=1}^{n}\), we compute the non-conformity score of calibration data as negative log-loss based on the predictive distribution obtained from the tempered posterior as the equation (8). Subsequently, for a given pre-defined coverage level of \(1-\alpha\), the prediction set for a test input \(x\), generated using SCP on top of Bayesian GNNs with tempered posterior, is defined as follows:

\[\mathcal{C}_{n,\alpha,\beta}(x)=\left\{y\in\mathcal{Y}:-\log p_{\beta}(y|x,q) \leq Q_{\alpha}(\{-\log p_{\beta}(y_{i}|x_{i},q)\}_{i=1}^{n})\right\}. \tag{9}\]

This prediction set satisfies the desired coverage condition, as stated in Theorem 2. Additionally, as depicted in Fig. 1, we suggest that employing the tempered posterior in Bayesian GNNs, where the temperature controls its spread as means to overcome model misspecification, can generate an efficient prediction set within CP. While the credible region introduced in [33] provides a coverage guarantee in an asymptotic sense, SCP provides a rigorous guarantee without relying on specific distribution assumptions, as outlined in Theorem 1. Given that conformal predictors alone may not necessarily create the smallest prediction sets, adjusting the temperature provides greater freedom to reduce the size of the prediction sets. Please refer to Appendix B for the proofs of the theorems.

**Theorem 2**: _Given that calibration set \(\{(x_{i},y_{i})\}_{i=1}^{n}\) and a test data point \((x,y)\) are exchangeable random variables, for any coverage level \(1-\alpha\), and for any pre-determined temperature parameter \(\beta\in[0,\infty)\), a prediction set \(\mathcal{C}_{n,\alpha,\beta}(x)\) defined in the equation (9) satisfies the equation (1) with additional marginalization over posterior sampling in the equation (8) when assuming a finite number of sampling \(T\)._

Figure 2: **Example of label distribution depending on the temperature of Bayesian GCNs.**
To suggest a brief insight into how the temperature affects the inefficiency, we provide an example of label distribution from Bayeisan GNNs at different temperatures, extracted from real data samples of Cora and Citeseer. In Fig. 2, note that there are some cases where the true label is included in prediction set, but not selected as a point prediction. While prediction sets from all models include the true label, the size largely depends on the temperature of Bayesian GNNs. As depicted in Fig. 1, when the temperature parameter \(\beta\) is set too low, the label distribution tends to reflect an over-confident behavior compared to other models. This leads to extremely low quantiles and prediction set containing a large number of candidate labels. Conversely, if the temperature is set too high, it results in labels having nearly identical confidence levels, leading to prediction sets with less informative labels. However, when an appropriate temperature is selected between these extremes, Bayesian GNNs generate efficient prediction sets that contain fewer number of informative labels, resulting in better efficiency.

## 4 Experimental Results

In this section, we study (i) the conformal prediction performance of tempered posterior in GCN, in terms of empirical coverage and inefficiency, and analyze (ii) the contributing factors of inefficiency, considering the connection between inefficiency and model calibration.

Experimental settingsWe consider two major tasks on graph-structured data, which are graph classification and node classification tasks. We focus on transductive setting in node classification task that ensures the validity [15]. We have selected public graph datasets with enough number of classes that can effectively reflect the change of set size depending on the temperature. We use CIFAR10-superpixel [21; 8] and ENZYMES [24] for graph classification task, and Cora and Citeseer for node classification task [28]. Overall dataset statistics used for model training and CP are stated in Table 1. Our models are based on GCN [18] and Bayesian GCNs are trained with Graph DropConnect (GDC) assuming the same priors for random parameters as [13]. We have selected temperature parameters for each dataset that make reasonable test accuracy. To ensure the exchangeability of split conformal prediction, we randomly split calibration and test set as illustrated in Table 1, and CP results are averaged over 50 runs for CIFAR10, and 100 runs for other datasets. Further details on experimental settings and exchangeability conditions for GNNs can be seen in Appendix A, C.

Evaluation metricsWe use two metrics of conformal prediction, empirical coverage and empirical inefficiency. A conformal predictor is considered to be valid when the empirical coverage over test set \(\mathcal{D}_{\text{test}}\), denoted as \(\widehat{cov}\), satisfies the pre-defined coverage level (we use 0.9). Note that achieving higher coverage is not necessarily advantageous if it leads to an increase in set size, quantified as higher inefficiency. Another metric is empirical inefficiency which measures the average size of prediction sets over test set, denoted as _ineff_. Lower inefficiency indicates that a conformal predictor is more efficient since it outputs smaller number of candidate labels while keeping the same pre-defined coverage level.

\[\widehat{cov}=\frac{1}{|\mathcal{D}_{\text{test}}|}\sum_{(x,y)\in\mathcal{D}_{ \text{test}}}\mathbbm{1}\{y\in C_{n,\alpha}(x)\},\ \widehat{\text{ineff}}=\frac{1}{|\mathcal{D}_{\text{test}}|}\sum_{(x,y)\in \mathcal{D}_{\text{test}}}|C_{n,\alpha}(x)| \tag{10}\]

Calibration measuresTo assess the calibration of GCNs, we employ commonly used calibration measures, along with CP measures. For each data sample \((x,y)\), suppose we have a point prediction from a model \(\theta\) as \(\hat{y}=\operatorname*{argmax}_{x}p(c|x,\theta)\), then accuracy and confidence for every sample are defined as \(\mathbbm{1}\{\hat{y}=y\}\) and \(p(c=\hat{y}|x,\theta)\), respectively. We use reliability diagram, which depicts the average sample accuracy against confidence levels. If a model is perfectly calibrated, where the estimated

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline Dataset & \#Nodes & \#Edges & \#Graphs & \#Feat & \#Classes & \#Train & \#Cal & \#Test \\ \hline Cora & 2708 & 5429 & - & 1433 & 7 & 140 & 500 & 1000 \\ Citeseer & 3327 & 4732 & - & 3703 & 6 & 120 & 500 & 1000 \\ CIFAR10 & - & - & 50000 & 5 & 10 & 10000 & 5000 & 10000 \\ ENZYMES & - & - & 600 & 21 & 6 & 500 & 50 & 50 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset statistics for Cora, Citeseer, CIFAR10, ENZYMES
probability of true label is exactly the same as the confidence for all levels, the reliability diagram takes the form of an identity function. In our experiments, we divide samples into \(M=20\) confidence bins, each denoted as \(B_{m}\), whose samples have confidence within the range \((\frac{m-1}{M},\frac{n}{M}]\). The average accuracy and confidence over samples in \(B_{m}\) are denoted as \(\text{acc}(B_{m})\) and \(\text{conf}(B_{m})\), respectively. We additionally use two measures obtained from reliability diagram, expected calibration error (ECE) and maximum calibration error (MCE) [11], defined as

\[\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|\text{acc}(B_{m})-\text{conf}(B _{m})\right|,\text{ MCE}=\max_{m\in\{1,\cdots,M\}}\left|\text{acc}(B_{m})-\text{ conf}(B_{m})\right|, \tag{11}\]

where \(n\) is the number of total samples. ECE is a weighted average of differences between confidence and accuracy across all bins, while MCE represents the largest of these differences among all bins.

### Efficiency of Tempered Posteriors

We evaluated our frequentist and Bayesian GCNs using CP evaluation metrics, empirical coverage and empirical inefficiency in Fig. 3. As theoretically guaranteed, Bayesian models of all temperatures satisfy the pre-defined coverage level of 0.9. Since coverage is guaranteed empirically across all models, the empirical inefficiency determines the performance of conformal predictors. A conformal predictor producing a smaller prediction set implies that the set conveys valuable information. In Fig. 3, we have verified that Bayesian GCN gives benefits to the lower inefficiency, while guaranteeing the desired coverage. _Notably, there exists a model with a temperature that makes better inefficiency, compared to other temperatures_, as illustrated as yellow boxes in Fig. 3. One may question that model with high accuracy might create smaller sets, and results in lower inefficiency. However, it is worth noting that while the inefficiency gets affected by the model performance, we have observed that this is not always the case. In the following sections, we will delve into this matter further by examining the relationship between inefficiency and model calibration.

### Contributing Factors to Inefficiency

Our experiments have shown that inefficiency is influenced by not only model performance but also model calibration. For example in Fig. 3(a), the Bayesian model with the temperature \(\beta=2\cdot 10^{-3}\) has lower inefficiency than the model with \(\beta=8\cdot 10^{-3}\), even though the test accuracy is lower. Motivated by this observation, we conducted following analysis to answer to the quenstion _"What factors contribute to inefficiency in CP due to the changes in temperature?"_

Relation between inefficiency and model calibrationTo provide a deeper understanding of the connection between prediction sets and the underlying model, we conducted evaluation of Bayesian GCNs with different temperatures, assessing calibration through reliability diagram, ECE and MCE.

Figure 3: **Empirical coverage and inefficiency of Bayesian GCNs with different temperatures**. Model that produces smallest sized prediction sets on average is denoted as yellow box. The exact average coverage and inefficiency of Bayesian models are represented as red line inside each box.
Overall, we have verified that employing an appropriate temperature parameter \(\beta\) enhances the calibration of GCNs. This improvement is evident in reliabillity diagrams, as shown in Fig. 4(b). Bayesian GCN model which has the lowest inefficiency at \(\beta=10^{-3}\) is comparably well-calibrated than other models.

Furthermore, our experiments imply two major observations. First, _when two models exhibit similar model accuracy levels, the better calibrated model tends to have lower inefficiency, even if the

Figure 4: **Results of inefficiency and calibration on Citeseer**. (a) Model with lower inefficiency implies that it generates prediction sets with smaller size on average. Bayesian GCNs with \(\beta=4\cdot 10^{-3}\) has the lowest inefficiency. (b) The line \(y=x\) on reliability diagrams represents a perfectly calibrated model.

Figure 5: **Results of inefficiency and calibration on CIFAR10**. (a) Model with the lower inefficiency implies that it generates prediction sets with smaller size on average. Bayesian GCNs with \(\beta=10^{-3}\) has the lowest inefficiency. (b) The line \(y=x\) on reliability diagrams represents a perfectly calibrated model.
difference can be slight_. In Fig. 4a, the Bayesian model with \(\beta=4\cdot 10^{-3}\), although having similar accuracy with the model \(\beta=8\cdot 10^{-4}\), exhibits reduced inefficiency. This can be attributed to its more robust calibration, as evidenced by ECE, MCE, and the reliability diagram. Similar case can be observed from Fig. 5 between two Bayesian models with \(\beta=10^{-3}\) and \(\beta=10^{-4}\).

The second finding is that _a well-calibrated model, even when its accuracy is relatively low, yields reasonably efficient prediction sets_. From the comparison of Bayesian models with \(\beta=2\cdot 10^{-3}\) and \(\beta=8\cdot 10^{-3}\), we clearly see that the well-calibrated one with \(\beta=2\cdot 10^{-3}\) results in lower inefficiency, despite its lower test accuracy. Note that the case of \(\beta=2\cdot 10^{-3}\), the two calibration measures ECE and MCE well reflect the reliability diagram as seen in Fig. 4a. However, in the case of a model with \(\beta=10^{-2}\), while MCE reasonably captures the low inefficiency resulting from its poor model quality, whereas ECE does not. Hence, in the following, we explore how previous calibration measures, ECE and MCE, reflect CP inefficiency and propose insights on temperature selection through a new calibration measure.

Combined measure for temperature selectionWhile our previous findings have indicated the existence of temperatures that lead to efficient prediction sets through CP, it is more preferable to find such good temperatures that yield low inefficiency, without adding computational burden from CP. For this purpose, MCE proves to be better than ECE, and this can be attributed to the sensitivity of inefficiency to difficult samples when making prediction sets. However, it is important to note that neither ECE nor MCE perfectly reflects inefficiency, making points such as \(\beta=10^{-2}\) in Fig. 4 and \(\beta=10^{-2}\) in Fig. 5. Consequently, recognizing that both model performance and calibration contribute to set size, we propose that a combined measure involving accuracy and calibration measure could offer a better chance to capture the inefficiency of models with different temperatures. In the rightmost figures of Fig. 4 and Fig. 5, we present the relation between the combined measure, calculated as MCE divided by accuracy, and inefficiency. These figures show that the combined measure better accounts for the inefficiency of models with high temperatures, addressing the limitations of ECE and MCE. While we have provided a preliminary exploration of the combined measure's ability to represent the set size in CP, we plan to further investigate a measure that well explains the inefficiency. Additionally, in our future research, we aim to develop methods for optimizing the temperature through CP-aware loss [15; 40; 32; 26] during training process.

## 5 Conclusion

In this work, we explore the impact of temperature on CP in Bayesian GNNs, specifically focusing on the inefficiency of prediction sets. We show that Bayesian GNNs improve the performance in CP compared to frequentist GNNs while providing more flexibility to control the inefficiency via temperature. Our experiments demonstrate that there exist temperatures of Bayesian GNNs that generate efficient prediction sets within CP. Moreover, our analysis on the connection between inefficiency and model calibration suggests further possibility for a measure capable of accurately capturing the inefficiency. Assessing the CP performance using such measure can reduce the computational burden induced by CP. Furthermore, the investigation of a novel method for training the temperature parameter using CP-aware loss [15; 40; 32; 26] remains as an interesting future work. We anticipate that these findings can provide guidance to trustworthy graph learning that studies reliable uncertainty estimates employing CP.

## Acknowledgement

The authors want to thank Sangwoo Park and Osvaldo Simeone for providing the main idea of this work that studies the impact of temperature in Bayesian learning when used in conjunction with CP under GNN architecture. This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2020-0-01787 and IITP-2023-RS-2023-00259991) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation).