Semi-Supervised Deep Regression with Uncertainty Consistency and
  Variational Model Ensembling via Bayesian Neural Networks
Deep regression is an important problem with numerous applications. These
range from computer vision tasks such as age estimation from photographs, to
medical tasks such as ejection fraction estimation from echocardiograms for
disease tracking. Semi-supervised approaches for deep regression are notably
under-explored compared to classification and segmentation tasks, however.
Unlike classification tasks, which rely on thresholding functions for
generating class pseudo-labels, regression tasks use real number target
predictions directly as pseudo-labels, making them more sensitive to prediction
quality. In this work, we propose a novel approach to semi-supervised
regression, namely Uncertainty-Consistent Variational Model Ensembling (UCVME),
which improves training by generating high-quality pseudo-labels and
uncertainty estimates for heteroscedastic regression. Given that aleatoric
uncertainty is only dependent on input data by definition and should be equal
for the same inputs, we present a novel uncertainty consistency loss for
co-trained models. Our consistency loss significantly improves uncertainty
estimates and allows higher quality pseudo-labels to be assigned greater
importance under heteroscedastic regression. Furthermore, we introduce a novel
variational model ensembling approach to reduce prediction noise and generate
more robust pseudo-labels. We analytically show our method generates higher
quality targets for unlabeled data and further improves training. Experiments
show that our method outperforms state-of-the-art alternatives on different
tasks and can be competitive with supervised methods that use full labels. Our
code is available at https://github.com/xmed-lab/UCVME.
Weihang Dai, Xiaomeng Li, Kwang-Ting Cheng
Semi-Supervised Deep Regression with Uncertainty Consistency and Variational Model Ensembling via Bayesian Neural Networks

###### Abstract

Deep regression is an important problem with numerous applications. These range from computer vision tasks such as age estimation from photographs, to medical tasks such as ejection fraction estimation from echocardiograms for disease tracking. Semi-supervised approaches for deep regression are notably under-explored compared to classification and segmentation tasks, however. Unlike classification tasks, which rely on thresholding functions for generating class pseudo-labels, regression tasks use real number target predictions directly as pseudo-labels, making them more sensitive to prediction quality. In this work, we propose a novel approach to semi-supervised regression, namely Uncertainty-Consistent Variational Model Ensembling (UCVME), which improves training by generating high-quality pseudo-labels and uncertainty estimates for heteroscedastic regression. Given that aleatoric uncertainty is only dependent on input data by definition and should be equal for the same inputs, we present a novel uncertainty consistency loss for co-trained models. Our consistency loss significantly improves uncertainty estimates and allows higher quality pseudo-labels to be assigned greater importance under heteroscedastic regression. Furthermore, we introduce a novel variational model ensembling approach to reduce prediction noise and generate more robust pseudo-labels. We analytically show our method generates higher quality targets for unlabeled data and further improves training. Experiments show that our method outperforms state-of-the-art alternatives on different tasks and can be competitive with supervised methods that use full labels 1.

Footnote 1: Code is available at [https://github.com/xmed-lab/UCVME](https://github.com/xmed-lab/UCVME)

## Introduction

Deep learning has achieved state-of-the-art results on a variety of tasks such as classification [10], segmentation [14], image generation [1], and others. These methods tend to require large amounts of labeled data for training, however, which can be costly to annotate. State-of-the-art image classifiers such as ViT are trained on the JFT-300M dataset, for example, which consists of 300 million images [15]. Labeling can also be prohibitively expensive for medical image analysis, where life-saving tasks such as medical disease diagnoses [11] and tumor segmentation [11] require domain expertise. The ability to train neural networks with reduced labels is therefore highly valuable and an active research area.

Semi-supervised learning uses unlabeled data together with a smaller labeled dataset for model training. These methods reduce reliance on labeled data and sometimes outperform state-of-the-art techniques on fully labeled datasets. Chen _et al_. [14] propose CPS, a semi-supervised algorithm for image segmentation, which is time-consuming to label. Li _et al_. [11] enforce con

Figure 1: Differences between standard pseudo-labeling approaches and our method (UCVME). Classification tasks typically apply thresholding functions to probability predictions, \(\tilde{p}^{i}\), to obtain one-hot pseudo-labels, \(\tilde{y}^{i}\). Regression tasks use real number target predictions, \(\tilde{y}\), directly as pseudo-labels and are therefore more sensitive to prediction quality. Our UCVME improves pseudo-labels for regression by considering pseudo-label uncertainty, \(\sigma^{2}\), and robustness. We use a novel uncertainty consistency loss to improve uncertainty-based loss weighting and a variational model ensembling method to improve pseudo-label quality.
sistency between transformed inputs for medical diagnosis, which requires specialist knowledge for annotation. However, comparatively less attention has been paid to deep regression problems, which cover practical applications such as age estimation [1] and pose estimation [20] from images. Deep regression is particularly important in the medical field as it is used to obtain measurements for disease diagnosis and progression tracking, such as bone mineral density estimation for osteoporosis [14] and ejection fraction estimation for cardiomyopathy [21].

Regression problems are fundamentally different from classification problems because they generate real number predictions instead of class probabilities. Existing semi-supervised classification techniques _cannot be applied to semi-supervised regression_ because they rely on class probabilities and thresholding functions to generate pseudo-labels [15, 16] (see Fig. 1). Limited efforts have been devoted to exploring semi-supervised approaches for deep regression. Recent works by Jean _et al_. [1] propose deep kernel learning for semi-supervised regression, but their method is designed for tabular data. Pretrained feature extractors are used for image inputs, which prevents task-specific feature learning and limits performance. Wetzel _et al_. [20] propose TNNR, which estimates the difference between inputs with deep networks and uses loop consistency for unlabeled data. Loop consistency regulates training, but poor-quality predictions can still reduce the effectiveness of the constraints (see Tables 1 and 4).

Unlike classification tasks, which can smooth predictions using thresholding functions for class pseudo-labeling, regression tasks directly use real number target predictions as pseudo-labels. Therefore, model performance highly depends on the quality of pseudo-labels, _i.e._ predictions. In this paper, we propose a novel Uncertainty-Consistent Variational Model Ensembling method, namely UCVME, that adjusts for the uncertainty of pseudo-labels during training and increases pseudo-label robustness. Our method is based on two key ideas: enforcing uncertainty consistency between co-trained models to improve uncertainty-based loss weighting, and using ensembling techniques to reduce prediction variance for obtaining higher quality pseudo-labels.

We make use of Bayesian neural networks (BNNs), which predict aleatoric uncertainty of observations jointly with the target value. The uncertainty estimates are used for heteroscedastic regression, which assigns sample weightings based on uncertainty to reduce the impact of noisier samples [1]. We observe that aleatoric uncertainty, which by definition is dependant only on input data, _should be equal for the same input,_ and propose a novel consistency loss for uncertainty predictions of co-trained models. Our proposed loss notably improves aleatoric uncertainty estimates on unlabeled data, such that higher quality pseudo-labels are given greater importance through heteroscedastic regression (see Fig. 5). This is non-trivial since unreliable uncertainty estimates can lead to adverse loss-weighting and unstable training. Our proposed method is _the first to address uncertainty estimation quality for regression_.

BNNs also use variational inference during prediction to approximate the underlying distribution of estimates. To improve robustness of pseudo-labels, we introduce variational model ensembling, which uses ensembling methods with variational inference to reduce prediction noise. We analytically show our approach generates higher quality targets for unlabeled data and validate results experimentally (see Table 2). The combined improvements in uncertainty estimation and pseudo-label quality lead to state-of-the-art performance. Fig. 2 illustrates the overall framework.

We demonstrate our method on two regression tasks: age estimation from photographs and ejection fraction estimation from echocardiogram videos. Results show our method outperforms state-of-the-art alternatives and is competitive with supervised approaches using full labels (see Tables 1 and 4). Ablations demonstrate individual contributions from uncertainty consistency and variational model ensembling (Table 2). We summarize our main contributions as follows:

* We propose UCVME, a novel semi-supervised method that improves uncertainty estimates and pseudo-label robustness for deep regression tasks.
* We introduce a novel consistency loss for aleatoric uncertainty predictions of co-trained models, based on the insight that estimates should be equal for the same input.
* We introduce variational model ensembling for generating pseudo-labels on unlabeled data, which we analytically show is more accurate than deterministic methods.
* Results show our method outperforms existing state-of-the-art alternatives on two separate regression tasks.

## Related Works

In this section, we review works on learning from unlabeled data, general approaches for semi-supervised learning, state-of-the-art methods for semi-supervised regression, and existing methods for uncertainty estimation.

### Unsupervised Representation Learning

One way to learn from unlabeled data is to learn unsupervised feature representations, which can then be finetuned for specific tasks using classifiers. Techniques such as

Figure 2: Semi-supervised deep regression framework for our UCVME method. UCVME improves overall pseudo-label quality and assigns greater sample weights to pseudo-labels with low uncertainty.
PCA [1] and data clustering [12] learn intermediate features by reducing input dimensionality. With the increasing effectiveness of deep learning, pre-text tasks such as input reconstruction [13], augmentation prediction [14], and order prediction [15] have been explored for unsupervised training of deep feature extractors. Current state-of-the-art approaches are based on contrastive learning, which has been shown in some cases to outperform supervised learning [16, 17].

### Semi-Supervised Learning

Semi-supervised learning uses both labeled and unlabeled data for training. This reflects realistic settings where raw data is easy to obtain but annotations can be costly. State-of-the-art methods include enforcing consistency on augmented inputs and using pseudo-labels for unlabeled samples. For example, CCT [1] applies prediction consistency after perturbing intermediate features. CPS [17] enforces consistency of segmentation predictions between co-trained models. Temporal ensembling [18] and mean-teacher [19] methods use prediction and model-weight ensembling respectively to generate pseudo-labels. FixMatch [10] and FlexMatch [14] use class probability thresholding for pseudo-labeling to achieve state-of-the-art results on semi-supervised classification. Similar techniques have been applied to video action recognition [20], image generation [10, 11], medical image segmentation [12, 13, 21, 22], and other tasks.

### Semi-Supervised Regression

Regression problems are fundamentally different from classification as they involve predicting real numbers in IR instead of class probabilities. Semi-supervised classification methods, which use thresholding functions to select high-probability class pseudo-labels [10, 14], cannot be adapted to regression tasks _because there is no equivalent to probability thresholding for real number predictions_. Different formulations must be used instead to quantify prediction uncertainty for regression tasks.

Less attention has been paid to semi-supervised deep regression despite its importance [1, 14, 15, 16]. Semi-supervised regression is especially valuable in medical image analysis, since regression tasks are widely used and annotation costs are high [15, 16]. COREG [17] is a semi-supervised regression technique originally proposed in 2005 but still commonly used today. Two KNN regressors are co-trained and used to generate pseudo-labels for unlabeled data. Co-training schemes have also been extended to support vector regression by Xu _et al_. [20]. Graph-based methods proposed in [18] make use of input proximity for pseudo-labeling. More recent works by Jean _et al_. [14] and Mallick _et al_. [15] make use of deep kernel learning for regression.

One major disadvantage of these methods is that they are primarily designed for structured inputs, where samples consist of one-dimensional tabular data. Feature extractors cannot be trained end-to-end for unstructured inputs such as images and video. Jean _et al_. [14] for example rely on feature extractors pretrained on ImageNet [14] to obtain one dimensional embeddings from images. This limits performance as task-specific features cannot be learned (see results in Tables 1 and 4). TNNR [16] is an alternative method that uses deep networks to predict differences between input pairs. Loop consistency is applied to ensure looped differences sum to zero. Although loop consistency helps regularize training on unlabeled data, inaccurate predictions can limit its effectiveness (see Tables 1 and 4).

Unlike previous works, we address semi-supervised deep regression by improving uncertainty estimation and pseudo-label quality for real number targets. Our UCVME method, which proposes a novel uncertainty consistency loss and variational model ensembling, allows training to be focused on high-quality, robust pseudo-label targets and achieves state-of-the-art results on different regression tasks.

### Uncertainty Estimation

Uncertainty estimation is commonly used in semi-supervised learning to adjust for pseudo-label quality of unlabeled samples. UA-MT [21] and UMCT [15] both use Monte Carlo dropout to estimate pseudo-label uncertainty, which is then used to filter pseudo-labels or weight unlabeled samples for segmentation tasks. Yao _et al_. [21] and Lin _et al_. [16] estimate uncertainty based on prediction differences between co-trained models for segmentation of medical images. Semi-supervised classification methods such as FixMatch [10] and FlexMatch [14] implicitly filter out uncertain pseudo-labels by setting confidence thresholds for predictions.

Uncertainty estimation approaches designed for semi-supervised deep regression have not been explored in existing works however. Although methods such as heteroscedastic regression can be used to estimate uncertainty, it can only be done through joint prediction with the target label [15]. Naive implementation using pseudo-labels give unreliable estimates, which can leads to inaccurate pseudo-labels being assigned larger weights (see Fig. 5). In this work, we propose a novel uncertainty consistency loss that significantly improves the quality of uncertainty estimates on unlabeled data. This results in more effective uncertainty-based sample weighting and leads to state-of-the-art performance on different semi-supervised deep regression tasks.

## Methodology

UCVME is based on two novel ideas: enforcing aleatoric uncertainty consistency to improve uncertainty-based loss weighting, and variational model ensembling for generating high-quality pseudo-labels. We make use of Bayesian
neural networks, which differ from regular neural networks by their usage of aleatoric uncertainty prediction and variational inference [2]. We denote \(\mathcal{D}\coloneqq\{(x_{i},y_{i})\}_{i=1}^{N}\) as the labeled dataset consisting of \(N\) samples, where \(x_{i}\) is the input data and \(y_{i}\) is its corresponding label. We denote \(\mathcal{D}^{\prime}\coloneqq\{x_{i^{\prime}}^{\prime}\}_{i^{\prime}=1}^{N^{ \prime}}\) as the unlabeled dataset consisting of input data only. We train two BNNs, \(f_{m}\) where \(m\in\{a,b\}\), in a co-training framework and use Monte Carlo dropout for training and inference. We denote \(\hat{y}_{i,m}\) as model \(m\)'s prediction for target label \(y_{i}\). We denote \(\sigma_{i}^{2}\) as aleatoric uncertainty but predict log-uncertainty \(\ln\sigma_{i}^{2}\) in practice, which is always done to avoid obtaining negative predictions for variance. We denote predicted log-uncertainty using \(\hat{z}_{i,m}\).

### Aleatoric Uncertainty Consistency Loss for Improved Heteroscedastic Regression

Aleatoric uncertainty, \(\sigma_{i}^{2}\), refers to uncertainty relating to input data. It is used in BNNs as the variance parameter for heteroscedastic regression loss:

\[\mathcal{L}_{reg}=\frac{1}{N}\sum_{i=0}^{N}\frac{(y_{i}-\hat{y}_{i})^{2}}{2 \sigma_{i}^{2}}+\frac{\ln\sigma_{i}^{2}}{2}. \tag{1}\]

where \(\hat{y}_{i}\) is the prediction for target label \(y_{i}\). Intuitively, the loss function weighs error values dynamically based on aleatoric uncertainty. Samples with high uncertainty are regarded as having lower quality labels with higher noise, and these are given less importance compared to those with greater certainty [2]. Its formal derivation is based on maximum likelihood estimation, assuming observation errors are distributed with different levels of variance [2]. In contrast, standard mean squared error (MSE) loss assumes homoscedastic errors, _i.e._ uncertainty values \(\sigma_{i}^{2}\) have equal variance, which is a more restrictive and unrealistic assumption. We refer interested readers to Sup-1 of the supplementary materials for a review of formal derivations and comparisons.

Heteroscedastic regression can be beneficial for unlabeled data as it allows samples to be weighted based on pseudo-label uncertainty. In practice however, uncertainty prediction is difficult because uncertainty has no ground truth label and must be jointly predicted with the target value. Unstable predictions that do not reflect label quality can adversely affect training by assigning noisier samples with larger weights. _Stable training is even more difficult for unlabeled data_ because the target ground truth value is also unavailable, which is why heteroscedastic regression has not been successfully used in existing semi-supervised works. We show this effect in Fig. 5, where we see uncertainty predictions obtained using heteroscedastic regression only can be unreliable.

We observe that aleatoric uncertainty for the same input data _should be equal by definition_ and introduce a novel consistency loss to enforce consistent uncertainty predictions between co-trained models. Prediction consistency is known to be an effective regularizer [2] and can be applied to both labeled and unlabeled data to improve estimates. By ensuring uncertainty predictions from co-trained models are consistent, we provide an extra training signal in addition to joint estimation with the target label, which helps the model learn more reliable predictions. For labeled inputs, we introduce consistency loss, \(\mathcal{L}_{unc}^{lb}\):

\[\mathcal{L}_{unc}^{lb}=\frac{1}{N}\sum_{i=1}^{N}(\hat{z}_{i,a}-\hat{z}_{i,b})^ {2}\, \tag{2}\]

which is based on L2 distance. Heteroscedastic regression loss is calculated using the uncertainty predictions:

\[\mathcal{L}_{reg}^{lb}=\frac{1}{N}\sum_{m=a,b}\ \sum_{i=1}^{N}\left(\frac{(\hat{y}_{i,m }-y_{i})^{2}}{2\exp(\hat{z}_{i,m})}+\frac{\hat{z}_{i,m}}{2}\right). \tag{3}\]

For unlabeled data, ground truth target labels for \(y\) are unavailable, which makes joint uncertainty prediction challenging. We instead make use of variational model ensembling to obtain pseudo-labels for log-uncertainty, \(\widetilde{z}_{i}\), which is used as the training target. We describe variational model ensembling for unlabeled samples in the subsection below.

### Variational Model Ensembling for Pseudo-label Generation

BNNs use Monte Carlo dropout and variational inference to estimate the distribution of the predictor \(\hat{y}\). To reduce prediction noise, we can use ensembling techniques that reduce predictor variance, which can be demonstrated through bias-variance decomposition. The performance of predictor \(\hat{y}\) can be evaluated using expected MSE, which we decompose using bias-variance decomposition as follows:

\[E[(\hat{y}_{i}-y_{i})^{2}]=(E[\hat{y}_{i}]-y_{i})^{2}+E[(\hat{y}_{i}-E[\hat{y} _{i}])^{2}]\, \tag{4}\]

where the first right-hand side term is the bias and the second is the variance. If we take individual sample predictions from variational inference, \(\hat{y}_{i}^{\ t}\), and obtain an ensemble to form a new predictor \(\widetilde{y}_{i}\), we have:

\[\widetilde{y}_{i}=\frac{1}{T}\sum_{t=1}^{T}\hat{y}_{i}^{\ t}\, \tag{5}\]

where \(T\) is the number of samples used. The expected MSE loss of the predictor \(\widetilde{y}_{i}\) is then:

\[E[(\widetilde{y}_{i}-y_{i})^{2}]=(E[\widetilde{y}_{i}]-y_{i})^{2}+E[( \widetilde{y}_{i}-E[\widetilde{y}_{i}])^{2}]. \tag{6}\]

The bias terms of the predictors are equal, but the variance term in equation 6 cannot be greater than in equation 4 because more samples are observed (see Sup-2 of supplementary materials for more detailed derivations). This means predictor \(\widetilde{y}_{i}\) will have expected MSE lower than or equal to \(\hat{y}_{i}\) and will always have higher quality.

Based on this effect, we propose variational model ensembling for generating pseudo-labels on both target value \(\widetilde{y}_{i}\) and log aleatoric uncertainty \(\widetilde{z}_{i}\). Whereas pseudo-labels for co-trained models typically rely on cross-supervision in state-of-the-art approaches [2, 2]. The
2021), we ensemble the average estimate of the co-trained models and apply variational inference:

\[\widetilde{y}_{i}=\frac{1}{T}\sum_{t=1}^{T}\frac{\hat{y}_{i,a}^{\ t}+\hat{y}_{i,b}^ {\ t}}{2}\, \tag{7}\]

\[\widetilde{z}_{i}=\frac{1}{T}\sum_{t=1}^{T}\frac{\hat{z}_{i,a}^{\ t}+\hat{z}_{i,b }^{\ t}}{2}. \tag{8}\]

and use this as the pseudo-label for training. Compared to cross-supervision, pseudo-labels calculated using variational model ensembling are _more accurate because of reduced predictive variance_ and better reflect the true target and uncertainty values. This is especially important for regression targets because pseudo-labels directly use real number predictions and do not rely on thresholding functions for smoothing. Uncertainty consistency on unlabeled data is then calculated using \(\widetilde{z}_{i}\) as the training target:

\[\mathcal{L}^{ulb}_{unc}=\frac{1}{N^{\prime}}\sum_{m=a,b}\ \sum_{i=1}^{N^{\prime}}( \hat{z}_{i,m}-\widetilde{z}_{i})^{2}. \tag{9}\]

Heteroscedastic regression loss for unlabeled data is calculated using \(\widetilde{y}_{i}\) as the target and \(\widetilde{z}_{i}\) as the log-uncertainty:

\[\mathcal{L}^{ulb}_{reg}=\frac{1}{N^{\prime}}\sum_{m=a,b}\ \sum_{i=1}^{N^{\prime}} \left(\frac{(\hat{y}_{i,m}-\widetilde{y}_{i})^{2}}{2\exp(\widetilde{z}_{i})}+ \frac{\widetilde{z}_{i}}{2}\right). \tag{10}\]

The improved pseudo-labels lead to more stable heteroscedastic regression, which generates better training signals on unlabeled data.

### Overall Semi-Supervised Framework

During training, we calculate heteroscedastic regression loss \(\mathcal{L}^{lb}_{reg}\) and aleatoric uncertainty consistency loss \(\mathcal{L}^{lb}_{unc}\) using labeled samples. Pseudo-labels for unlabeled data are generated using Eq. 7 and 8 at the start of every training iteration with the most current model weights. The loss values for \(\mathcal{L}^{ulb}_{reg}\) and \(\mathcal{L}^{ulb}_{unc}\) are calculated for the unlabeled data and jointly optimized with labeled data using the total loss:

\[\mathcal{L}= \mathcal{L}^{lb}_{reg}+\mathcal{L}^{lb}_{unc}+w_{ulb}\ (\ \mathcal{L}^{ulb}_{reg}+\mathcal{L}^{ulb}_{unc}\ )\, \tag{11}\]

where \(w_{ulb}\) is the weighting parameter for unlabeled data. Variational model ensembling is also used for test-time inference to obtain \(\widetilde{y}_{i}\) as the final prediction. Pseudo-code is given in S-Algorithm 1 of the supplementary materials.

## Experiments

We demonstrate our method on two semi-supervised deep regression problems: age estimation from photographs and ejection fraction estimation from echocardiogram videos. 2

Footnote 2: Code is available at [https://github.com/xmed-lab/UCVME](https://github.com/xmed-lab/UCVME).

### Age Estimation from Photographs

Age estimation involves predicting a person's age based on their photograph, and is commonly used as a benchmark task for deep regression. Facial images can be easily obtained, but accurate age labels may not always be available given concerns over data privacy. Semi-supervised deep regression methods can provide label-efficient approaches for training.

#### Dataset

We use the UTKFace dataset [15] and follow the train-test split in previous works [11]. A total of 13,144 images are available for training and 3,287 images for testing. We use a subset of the training dataset for validation. Faces have been pre-cropped and age labels range from 21 to 60 (see Figure 3 for examples). For our semi-supervised setting, we use subsets of the training data as labeled data and the remaining as unlabeled data. Label distributions are shown in S-Fig. 1 of the supplementary materials.

#### Settings

We use ResNet-50 [14] as our encoder and add additional dropout layers after each of the four main residual blocks. The model is trained for 30 epochs using learning rate \(10^{-4}\), weight decay \(10^{-3}\), and the Adam optimizer. We use a batch size of 32 for both labeled and unlabeled data. We set dropout probability as 5% and use \(T=5\) for variational inference. We set \(w_{ulb}=10\) which we choose empirically (see S-Table 1 in supplementary materials). Mean absolute error (MAE) and \(R^{2}\) are used for evaluation on the test set. Experiments are run six times and mean results are reported with standard deviation.

#### Comparison with state-of-the-art

We compare our method with alternative state-of-the-art approaches for semi-supervised regression, specifically COREG [13], SSDPKL [12], and TNNR [20], and also adapt mean-teacher [16] and temporal ensembling [16] methods for regression. To highlight the impact of our proposed components, we introduce a baseline method (_Baseline_) that uses two co-trained BNNs with heteroscedastic regression loss, but _without_ aleatoric uncertainty consistency loss and variational model ensembling. We perform training under different semi-supervised settings using only 5%, 10%, and 20% of the available training labels. The remaining samples are treated as unlabeled data. For reference, we also show results using the supervised state-of-the-art method by Berg _et al_. [13] (_RNDB_) for the same settings using reduced labels, as well as on the fully labeled dataset.

Figure 3: Sample data from UTKFace dataset [15] for age estimation. Pre-cropped images are paired with age labels for training.
The same ResNet-50 encoder [16] is used in all methods for fair comparison. We also modify COREG to use co-trained deep regression models instead of KNN regression and use a pretrained feature encoder for SSDPKL to obtain image features. Additional implementation details are included in Sup-3 of supplementary materials. We show results in Table 1 and visually plot them in Fig. 4.

We can see from Fig. 4 that the supervised approach (blue) under-performs semi-supervised approaches in general. Our method (red) gives the best results and achieves the lowest MAE values for all settings. We also note that our method achieves performance competitive with fully supervised results using only 20% of available training labels (MAE 4.85 _v.s._ 4.83). UCVME therefore effectively reduces reliance on labeled data for deep regression.
since higher uncertainty means more prediction noise and lower quality labels. We obtain pseudo-labels and uncertainty predictions for unlabeled samples using the _Baseline_ and _Baseline_ + _Con._ models. Samples are grouped into ten equal bins based on sorted uncertainty predictions. Pseudo-label quality is measured using MSE against the ground truth target value. Average aleatoric uncertainty is calculated for each group. The two values are plotted against each other in Fig. 5 using models trained after five and twenty epochs.

For the _Baseline_ + _Con._ model, we see overall MSE and uncertainty both decrease after training for more epochs (solid red line _v.s._ solid blue line). Pseudo-labels with lower uncertainty have lower MSE and are of higher quality. In contrast, uncertainty predictions from the _Baseline_ model are more extreme (dashed lines) and are not significantly reduced with training (dashed red line _v.s._ dashed blue line). The relationship between uncertainty and pseudo-label quality is not strong, which can lead to noisier samples being assigned higher loss weightings. Aleatoric uncertainty consistency loss therefore _improves uncertainty estimates significantly and helps prevent adverse sample weighting_.

Computational CostWe show in Table 3 the computational cost of different semi-supervised deep regression approaches in gigaFLOPS per image (G). We note that the cost of UCVME is dependant on \(T\), the number of iterations used for variational model ensembling, which is set to 5. For reference, we also show the cost from using a single iteration, \(T=1\), which is equivalent to enforcing uncertainty consistency only without using variational model ensembling.

We can see our UCVME method with \(T=1\) incurs the same cost as COREG. Regression performance outperforms COREG however due to the use of uncertainty consistency, which can be seen from results in Tables 1 and 2 (MAE 5.30 _v.s._ 5.39). Using variational model ensembling by setting \(T=5\) leads to more computation but better predictions. Although mean-teacher, temporal ensembling, and SSDPKL require less computation, they do not perform as well.
data and remaining samples as unlabeled data. Label distributions are given in S-Fig. 2 of the supplementary materials.

SettingsWe use the R2+1D ResNet encoder Tran et al. (2018) pretrained on Kinetics 400 Kay et al. (2017) and add additional dropout layers between the four main residual blocks. We set dropout probability as 5% and use \(T=5\) for variational inference. The model is trained using SGD with \(10^{-4}\) learning rate and 0.9 momentum for 25 epochs. Learning rate is decayed by 0.1 at epoch 15. Clips of 32 frames are sampled from videos at a rate of 1 in every 2 frames for input. Batches of 10 clips are used for labeled and unlabeled videos. We set \(w_{ulb}=10\) which we choose empirically (see S-Table 2 of supplementary materials). We evaluate performance using MAE and \(R^{2}\). Experiments are run five times and mean results with standard deviation are reported.

Comparison with state-of-the-artsWe compare our method with mean-teacher Tarvainen and Valpola (2017), temporal ensembling Laine and Aila (2016), COREG Zhou et al. (2005), SSDPKL Mallick et al. (2021), TNNR Wetzel et al. (2021), and our baseline model (_Baseline_). We perform training under settings where one-sixteenth, one-eighths, and one-quarter of the training labels are used, with the remainder treated as unlabeled data. For reference, we also show results using the supervised method by Ouyang _et al_. Ouyang et al. (2020) on the reduced labels as well as on the fully labeled dataset. The Kinetics pre-trained R2+1D ResNet encoder Tran et al. (2018) is used in all methods for fair comparison. Additional implementation details are given in Sup-4 of the supplementary materials. Results are shown in Table 4 and plotted in Fig. 7.

Our method consistently achieves the best results for all settings by significant margins. We are also able to achieve an MAE of 4.37 using a quarter of the labels, which is only a relative 5.8% higher than the 4.13 MAE achieved by Ouyang _et al_. on fully labeled data. Our proposed method therefore reduces the number of labels required for training which is highly valuable for medical regression tasks.

## Conclusion

In this work, we introduce a novel Uncertainty-Consistent Variational Model Ensembling (UCVME) method for semi-supervised deep regression. Our method improves training on unlabeled data by adjusting for pseudo-label quality and improving pseudo-label robustness. We introduce a novel consistency loss on uncertainty estimates, which we demon

\begin{table}
\begin{tabular}{l|l|l l l l|c}  & \multicolumn{4}{c}{MAE Values \(\downarrow\)} \\ \hline Type & Method & Encoder & 1/16 labeled & 1/8 labeled & 1/4 labeled & All labels \\ \hline Supervised & Ouyang _et al_. Ouyang et al. (2020) & R2+1D & 6.04 \(\pm\) 0.20 & 5.57 \(\pm\) 0.21 & 4.78 \(\pm\) 0.11 & **4.13 \(\pm\) 3.85** \\ \hline \multirow{8}{*}{\begin{tabular}{} \end{tabular} } & Mean-teacher Tarvainen and Valpola (2017) & R2+1D & 6.01 \(\pm\) 0.09 & 5.51 \(\pm\) 0.06 & 4.71 \(\pm\) 0.07 & - \\  & Temporal ensembling Laine and Aila (2016) & R2+1D & 5.97 \(\pm\) 0.08 & 5.52 \(\pm\) 0.06 & 4.67 \(\pm\) 0.06 & - \\  & SSDPKL Jean et al. (2018) & R2+1D & 6.01 \(\pm\) 0.04 & 5.47 \(\pm\) 0.01 & 4.68 \(\pm\) 0.07 & - \\  & TNNR Wetzel et al. (2021) & R2+1D & 5.90 \(\pm\) 0.11 & 5.46 \(\pm\) 0.08 & 4.79 \(\pm\) 0.08 & - \\  & COREG Zhou et al. (2005) & R2+1D & 5.94 \(\pm\) 0.07 & 5.31 \(\pm\) 0.02 & 4.57 \(\pm\) 0.02 & - \\  & Baseline & R2+1D & 5.93 \(\pm\) 0.10 & 5.36 \(\pm\) 0.05 & 4.58 \(\pm\) 0.03 & - \\  & Ours & R2+1D & **5.77 \(\pm\) 0.04** & **5.10 \(\pm\) 0.05** & **4.37 \(\pm\) 0.05** & - \\ \hline \multicolumn{7}{c}{\(\mathbf{R^{2}}\) values \(\uparrow\)} \\ \hline Type & Method & Encoder & 1/16 labeled & 1/8 labeled & 1/4 labeled & All labels \\ \hline Supervised & Ouyang _et al_. Ouyang et al. (2020) & R2+1D & 55.3\% \(\pm\) 2.6 & 62.5\% \(\pm\) 2.2 & 71.6\% \(\pm\) 1.4 & **80.4\% \(\pm\) 1.2** \\ \hline \multirow{8}{*}{
\begin{tabular}{} \end{tabular} } & Mean-teacher Tarvainen and Valpola (2017) & R2+1D & 55.1% \(\pm\) 1.4 & 62.9\% \(\pm\) 0.7 & 72.5\% \(\pm\) 0.4 & - \\  & Temporal ensembling Laine and Aila (2016) & R2+1D & 55.2\% \(\pm\) 1.3 & 62.9\% \(\pm\) 0.7 & 73.2\% \(\pm\) 0.3 & - \\  & SSDPKL Jean et al. (2018) & R2+1D & 56.3\% \(\pm\) 1.0 & 61.2\% \(\pm\) 0.3 & 74.1\% \(\pm\) 1.0 & - \\  & TNNR Wetzel et al. (2021) & R2+1D & 55.9% \(\pm\) 1.2 & 63.4\% \(\pm\) 0.8 & 73.7\% \(\pm\) 0.6 & - \\  & COREG Zhou et al. (2005) & R2+1D & 55.1% \(\pm\) 0.7 & 64.5\% \(\pm\) 0.4 & 74.1\% \(\pm\) 0.1 & - \\  & Baseline & R2+1D & 55.2\% \(\pm\) 1.4 & 64.9\% \(\pm\) 0.3 & 74.5\% \(\pm\) 0.1 & - \\  & Ours & R2+1D & **57.8\% \(\pm\) 0.6** & **66.6\% \(\pm\) 0.5** & **76.3\% \(\pm\) 0.6** & - \\ \hline \end{tabular}
\end{table}
Table 4: Comparison with state-of-the-art methods for ejection fraction estimation from echocardiogram video. We use settings where only 1/16, 1/8, and 1/4 of training labels are available. “Supervised” methods are only able to use labeled data while “Semi-supervised” methods can use labeled and remaining unlabeled data. “Baseline” method uses two co-trained BNNs without uncertainty consistency loss and variational model ensembling. **Bold** numbers represent the best result.

Figure 7: MAE of different state-of-the-art methods for ejection fraction estimation. Our proposed method (red) consistently outperforms alternatives by significant margins.
strate significantly improves heteroscedastic loss weighting, especially for unlabeled samples. We also use variational model ensembling to reduce prediction noise and generate better training targets for unlabeled data. Our method has strong theoretical support and can be applied to different tasks and datasets. We demonstrate this using two deep regression tasks based on image and video data and achieve state-of-the-art performance for both. Results are also competitive with supervised methods using full labels. UCVME is therefore a valuable method for reducing the amount of labels required for deep regression tasks.

## Acknowledgement

The work described in this paper was supported by a grant from Hong Kong Research Grants Council General Research Fund (GRF) (16203319), a grant from HKUST-BICI Exploratory Fund under HCIC-004, and a grant from Hong Kong Innovation and Technology Commission (Project no. ITS/030/21).
