# Self-supervised audio-visual speech representations learning by multimodal
self-distillation

###### Abstract

In this work, we present a novel method, named AV2vec, for learning audio-
visual speech representations by multimodal self-distillation. AV2vec has a
student and a teacher module, in which the student performs a masked latent
feature regression task using the multimodal target features generated online
by the teacher. The parameters of the teacher model are a momentum update of
the student. Since our target features are generated online, AV2vec needs no
iteration step like AV-HuBERT and the total training time cost is reduced to
less than one-fifth. We further propose AV2vec-MLM in this study, which
augments AV2vec with a masked language model (MLM)-style loss using multitask
learning. Our experimental results show that AV2vec achieved comparable
performance to the AV-HuBERT baseline. When combined with an MLM-style loss,
AV2vec-MLM outperformed baselines and achieved the best performance on the
downstream tasks.

Index Terms—  self-supervised learning, audio-visual speech recognition,
lipreading

##  1 Introduction

Humans are known to rely on both audio and visual information to perceive
speech [1]. Presented with corrupted and entangled sounds, the human
perceptual system draws heavily on visual information to reduce ambiguities in
audio [2]. In machine learning, a rich body of research has explored learning
from the multi-modal audio-visual speech data. For instance, visual speech
recognition (VSR), also known as lipreading, aims at predicting speech content
with only visual input [3, 4, 5, 6]. Audio-visual speech recognition (AVSR)
adopts both audio and visual modality input for speech recognition [7, 8, 9,
10, 11]. Because the visual signal is invariant to the acoustic interference,
VSR and AVSR have wide applications such as communication in noisy
environments or improved hearing aids.

Recent years have witnessed rapid development in self-supervised learning and
it has achieved success in both computer vision [12, 13, 14, 15, 16] and
speech processing domain [17, 18, 19, 20]. Different from supervised learning,
where labels provide an explicit discrimination task for learning, self-
supervised learning methods employ pretext tasks for model pretraining. It
enables to learn general representations from rich unlabeled data and to
finetune the model on labeled data. For audio and visual representations
learning, previous studies leverage the natural synergy between the audio and
visual channels of the video for cross-modal self-supervision. AVTS model
proposes to learn general audio-visual representations via self-supervised
temporal synchronization and a contrastive loss [21]. XDC leverages cross-
modal audio-visual clustering for self-supervised learning [22]. A contrastive
learning method for both global and local audio-visual representations has
also been proposed [23]. Particularly, a newly proposed method by Shi et al.
extends HuBERT [19] from speech to audio-visual speech data [24]. AV-HuBERT is
trained iteratively by alternating between an offline feature clustering step
and a masked prediction step until the model performance is no longer
improved. It has achieved state-of-the-art performance on downstream tasks
such as VSR [24] and AVSR [25].

Despite its impressive performance, pretraining an AV-HuBERT model is time-
consuming because of multiple iterations. In this work, we propose AV2vec, a
self-supervised model for learning audio-visual representations without
iterative training. AV2vec employs a student and a teacher module. The student
performs a masked latent feature regression task, in which the targets are
generated online by the teacher module. And the parameters of the teacher are
a momentum update of the student. For the student, we apply three types of
corruption for multimodal input. First, the audio signal input is corrupted by
adding noise randomly. Second, we apply modality-independent span masks for
intermediate hidden features. Third, a modality dropout is employed, namely
that either audio or video features are dropped randomly. For the teacher, it
always accepts full clean audio and visual input. Since our method generates
targets online without iterations, AV2vec can be trained much faster than AV-
HuBERT. Moreover, motivated by the success of AV-HuBERT [24] and HuBERT [19],
this work further presents AV2vec-MLM that augments AV2vec with a masked
language model (MLM)-style loss using multitask learning.

In our experiments, we compared the performance of our proposed AV2vec method
with the AV-HuBERT baseline comprehensively on the ASR, VSR, and AVSR
downstream tasks. For the visual stream, two types of input were investigated
including the entire face and the traditional lip region-of-interest (ROI).
Our experimental results suggest that AV2vec achieved comparable performance
to the AV-HuBERT baseline. When AV2vec is augmented with an MLM-style loss,
AV2vec-MLM achieved the best results on downstream tasks.

Fig. 1: Illustration of our proposed AV2vec model. AV2vec contains a student
and a teacher module. The student performs a masked latent feature regression
task and the targets are generated online by the teacher. The teacher weights
are an exponentially moving average (EMA) of the student parameters.

##  2 Related Works

Many semi-supervised and self-supervised learning methods employ a momentum
teacher for generating continuously refined targets online [26, 27, 14, 28,
29, 20]. Mean teacher [26] penalizes a consistency cost between the student
and the exponentially moving average (EMA) based teacher for semi-supervised
learning. MPL [27] adopts a momentum teacher to generate pseudo-labels for
semi-supervised ASR. For self-supervised learning, BYOL [14] relies on an
online and target network, where the online network is trained to predict the
target network representations of the same image under a different augmented
view. DINO [28] adopts a similar method to BYOL and interprets it as a form of
self-knowledge distillation with no labels. Our work is inspired by data2vec
[20], which predicts latent representations of the full input based on a
masked view of the input. Despite that data2vec is a general framework for
different modality inputs, it’s not designed for learning representations from
multimodal data within a signal model. Different from data2vec, our AV2vec
model learns from audio-visual multimodal data and enables cross-modal
knowledge distillation. Also, a novel multitask learning method that combines
AV2vec with an MLM-style loss is presented in this work.

##  3 Proposed Method

Our method consists of a student and a teacher module, where the student is
trained with a masked latent feature regression task and the targets are
generated online by the teacher. The model is illustrated in Figure 1.

###  3.1 Input of the model

For the audio stream, Mel filter banks feature is used as input. For the
visual stream, two types of input are compared including the conventional lip
region-of-interest (ROI) and the entire face images. Our previous study [30]
suggests the entire faces contain more detailed information than lips which
may benefit visual representation learning. And the experimental results have
shown that adopting faces as visual input led to an improved lipreading
accuracy.

###  3.2 Student module

Let the clean audio be
$\boldsymbol{A}^{c}=[\boldsymbol{a}^{c}_{1},\boldsymbol{a}^{c}_{2},\dots,\boldsymbol{a}^{c}_{T}]$,
the video input be
$\boldsymbol{V}=[\boldsymbol{v}_{1},\boldsymbol{v}_{2},\dots,\boldsymbol{v}_{T}]$.
In order to increase our model’s robustness to acoustic noise, the audio input
of the student
$\boldsymbol{A}=[\boldsymbol{a}_{1},\boldsymbol{a}_{2},\dots,\boldsymbol{a}_{T}]$
is obtained by adding noise to clean audio randomly with a probability
$p_{noise}$. The audio and visual input is processed by modality-specific
feature extractors for producing audio intermediate feature
$\boldsymbol{F}^{a}=[\boldsymbol{f}^{a}_{1},\boldsymbol{f}^{a}_{2},\dots,\boldsymbol{f}^{a}_{T}]$
and visual intermediate feature
$\boldsymbol{F}^{v}=[\boldsymbol{f}^{v}_{1},\boldsymbol{f}^{v}_{2},\dots,\boldsymbol{f}^{v}_{T}]$
respectively. A feed forward network and a 3DCNN-ResNet [9] network are
adopted for audio and visual feature extractor respectively. Then features of
each modality are corrupted independently by span-based masking. Specifically,
let $M_{a}$ be a set of masked indices for audio,
$\tilde{\boldsymbol{F}}^{a}=r(\boldsymbol{F}^{a},M_{a})$ denotes audio feature
$\boldsymbol{F}^{a}$ is corrupted by replacing $\boldsymbol{f}^{a}_{t}$ with
an embedding $\boldsymbol{e}^{a}$ if $t\in M_{a}$. For corrupting visual
features, the same strategy as the audio is used but with different mask
indices $M_{v}$ and a different embedding $\boldsymbol{e}^{v}$. We also tried
to corrupt audio and visual features with the same mask indices in our
preliminary experiments, while the results were worse than the modality-
independent masking. Then a modality dropout similar to that in AV-HuBERT [24]
is employed to train the student in absence of audio or video stream randomly.
Specifically, with a probability $p_{m}$, both modalities are used. The
probability of selecting audio is $p_{a}$ when only one modality is used, and
features of the other modality are set to all zeros. Audio-visual
representations are then obtained as
$\boldsymbol{F}^{av}=Concat(\tilde{\boldsymbol{F}}^{a},\tilde{\boldsymbol{F}}^{v})$,
where $Concat(\cdot)$ denotes channel-wise concatenation.
$\boldsymbol{F}^{av}$ is fed into a stack of Transformer blocks for generating
multimodal contextualized features, which are passed through a linear
projection layer for regressing to latent representations generated by the
teacher. A mean square error loss is used as

$L_{reg}=\sum_{t\in M_{a}\cup
M_{v}}||\boldsymbol{x}_{t}-\boldsymbol{y}_{t}||^{2},$ (1)

where $\boldsymbol{x}_{t}$ is the prediction and $\boldsymbol{y}_{t}$ is the
target produced by the teacher. Notes that we only penalized the predictions
in masked regions.

###  3.3 Teacher module

The teacher module consumes both clean audio $\textbf{A}^{c}$ and video V as
input. Intermediate feature masking or modality dropout is not applied for the
teacher. Therefore, it generates hidden representations with full context and
multimodal input. The teacher weights are updated by an exponentially moving
average (EMA) of the student as

$\boldsymbol{\theta}_{i}=\lambda*\boldsymbol{\theta}_{i-1}+(1-\lambda)*\boldsymbol{\phi}_{i},$
(2)

where $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$ denote the teacher and the
student parameters, $i$ denotes the update step and $\lambda$ is a hyper-
parameter controls the update step. $\lambda$ is linearly increased from
$\lambda_{b}$ to $\lambda_{e}$ at the initial $n$ updates and is kept constant
in the rest training steps. This momentum update is applied only for the
Transformer blocks and the teacher’s feature extractors share the parameters
with the student. No gradients flow to the teacher and only the student is
trained with the back-propagation algorithm. For obtaining targets of the
student, instance normalization [31] is applied for the last $k$ Transformer
blocks’ hidden outputs and then they are averaged following data2vec [20].

###  3.4 Multitask learning

In this section, we further propose to augment AV2vec with a masked language
model (MLM)-style loss like that of HuBERT, which is denoted as AV2vec-MLM. It
employs a separate linear projection layer on the top of the student’s
Transformer blocks for discrete targets classification, which is penalized
with a loss as

$L_{mlm}=\sum_{t\in M_{a}\cup M_{v}}\text{CE}(l_{t},\boldsymbol{p}_{t}),$ (3)

where $l$ and $\boldsymbol{p}_{t}$ denote the category of the discrete target
and the predicted probability distribution respectively, and
$\text{CE}(\cdot)$ denotes the cross entropy loss function. The targets can be
derived from hidden features of a pretrained AV2vec or AV-HuBERT model by
k-means clustering. AV2vec-MLM is optimized with multitask learning and the
total training loss is a summation of the latent feature regression loss and
the MLM-style loss as

$L_{MT}=L_{reg}+L_{mlm}.$ (4)

MLM-style loss serves as an additional supervision signal and is expected to
train the model more efficiently. However, AV2vec-MLM requires a pretrained
model for generating targets of MLM-style loss therefore it costs more
training time than AV2vec. Hence, it provides a trade-off between the training
cost and the model performance.

##  4 Experiments

###  4.1 Implementation details

Experiments were conducted based on the LRS3 dataset [32], which contains over
400 hours of English videos. All training data was used for pretraining and a
subset containing 30 hours of training data or the full training data was used
for finetuning. MUSAN dataset [33] was used and we followed the protocol of
[25] for sampling and adding noise to the audio input of the student and the
test set. We constructed the test sets at five SNR levels from -10 dB to 10 dB
with a 5 dB interval. For extracting lip ROI and the entire face images as
visual input, we followed our previous study [30].

For the network structure, AV2vec adopted the same configuration as the “BASE”
version of AV-HuBERT [24] to achieve the comparable computation complexity and
number of parameters to it. A single feed forward layer and 3DCNN-ResNet are
used for audio and visual feature extractor. It has 12 Transformer blocks and
the embedding dimension/feed forward dimension /attention heads were
768/3072/12 respectively. $p_{noise}$ was set to 0.25. 80% of audio features
and 30% of video features were masked for the student during pretraining.
$p_{m}$ and $p_{a}$ were set to 0.5 and 0.5 respectively. $\lambda_{b}$,
$\lambda_{e}$, and $n$ were set to 0.999, 0.9999, and 30k respectively. By
default, we averaged representations of the last 8 layers for the teacher
following [20]. Adam optimizer was used with a peak learning rate of $5\times
10^{-4}$. The learning rate was first warmup linearly for the first 3% of
updates and then kept constant for 90% of updates followed by exponentially
decay over the remaining 7%.

After pretraining, a 6-layer Transformer decoder was used and the pretrained
model was adopted as the encoder. During finetuning, the encoder was frozen at
the initial $i$ steps and then jointly finetuned with the decoder, where $i$
was determined using the validation set. Our methods were evaluated on three
types of audio-visual downstream tasks, including ASR, VSR, and AVSR. The
noise was added during both pretraining and finetuning stages except
finetuning for VSR task.

Three methods were compared in our experiments and they are described as
follows:

AV-HuBERT [24, 25]: The baseline method. We added noise to the student audio
input during both pretraining and finetuning stage following [25]. The model
was constructed based on the 4th iteration
checkpoint111https://facebookresearch.github.io/av_hubert/ publicly released
by the author to save the training time.

AV2vec: Our proposed method which is described in Section 3.

AV2vec-MLM: Our proposed method which is augmented with an MLM-style loss as
described in Section 3.4. The discrete labels were derived from the pretrained
AV-HuBERT of the 4th iteration to be directly comparable with the baseline.

###  4.2 Training costs

AV2vec and one iteration of AV-HuBERT both cost around 8 days of training time
for 400k updates on 8 Tesla A100 GPU cards. Therefore, training AV-HuBERT for
5 iterations from MFCC features following [24] will spend around 5-6 times as
long as training AV2vec if taking also into account the time costs of the
clustering step. Training AV2vec-MLM from scratch requires more training time
than AV2vec, since it relies on a pretrained model to generate discrete
targets. Despite that, AV2vec-MLM still costs less than a half of training
time than AV-HuBERT if it employs AV2vec for producing targets of MLM loss.
Overall, our proposed method yields a significantly better training efficiency
than AV-HuBERT does.

###  4.3 Main results

Table 1: Word error rate (WER %) results on the LRS3 test set.“Hrs” denotes the amount of labeled training data. “Vt” denotes the visual type for visual stream input, i.e. lips or faces. “ $\infty$” denotes clean audio.  Hrs | Method | Vt | VSR | ASR, SNR (dB)= | AVSR, SNR (dB)=
---|---|---|---|---|---
|  |  |  | -10 | -5 | 0 | 5 | 10 | $\infty$ | -10 | -5 | 0 | 5 | 10 | $\infty$
1.4k |  Afouras et al. [32] | lip | 58.9 | – | – | – | – | – | 8.3 | – | – | – | – | – | 7.2
595 |  Xu et al. [34] | lip | 57.8 | – | – | – | – | – | 7.2 | – | – | – | – | – | 6.8
31k |  Makino et al. [11] | lip | 33.6 | – | – | – | – | – | 4.8 | – | – | – | – | – | 4.5
30 | AV-HuBERT | lip | 47.1 | – | – | – | – | – | – | 30.2 | 18.3 | 10.7 | 7.4 | 6.3 | 5.4
AV-HuBERT | face | 42.5 | 81.5 | 52.8 | 24.7 | 13.2 | 8.5 | 5.7 | 28.0 | 17.4 | 10.9 | 7.5 | 6.4 | 5.4
AV2vec | lip | 45.1 | – | – | – | – | – | – | 30.2 | 18.6 | 10.9 | 7.9 | 6.8 | 5.8
AV2vec | face | 39.4 | 81.6 | 53.1 | 25.1 | 12.6 | 8.8 | 6.1 | 26.9 | 17.2 | 10.5 | 7.8 | 6.8 | 5.6
AV2vec-MLM | face | 39.4 | 81.3 | 51.7 | 23.9 | 12.4 | 8.5 | 5.6 | 26.8 | 16.3 | 10.0 | 7.3 | 6.3 | 5.4
433 | AV-HuBERT | lip | 40.3 | – | – | – | – | – | – | 27.5 | 15.1 | 7.1 | 4.5 | 3.4 | 2.6
AV-HuBERT | face | 34.8 | 71.6 | 46.7 | 19.8 | 8.9 | 5.5 | 2.8 | 25.0 | 14.2 | 7.0 | 4.3 | 3.5 | 2.7
AV2vec | lip | 39.9 | – | – | – | – | – | – | 26.8 | 14.7 | 7.5 | 4.4 | 3.7 | 2.6
AV2vec | face | 34.3 | 72.6 | 48.1 | 20.7 | 9.3 | 6.1 | 2.9 | 23.9 | 13.5 | 6.7 | 4.3 | 3.5 | 2.8
AV2vec-MLM | face | 34.4 | 71.3 | 46.3 | 19.5 | 8.9 | 5.2 | 2.7 | 23.4 | 13.3 | 6.7 | 4.2 | 3.2 | 2.5

Our experimental results are summarized in Table 1. The pretrained models were
finetuned and evaluated in VSR, ASR, and AVSR tasks. In VSR task, it’s
observed that the face-based model outperformed the lip-based counterpart
consistently for both AV-HuBERT and our proposed method. Compared with the AV-
HuBERT baseline, our method obtained better lipreading accuracy. AV2vec and
AV2vec-MLM achieved close performance in this task. In ASR task, the face-
based pretrained models were adopted for finetuning. We observed that AV2vec
achieved close but slightly worse results than the AV-HuBERT baseline. When
AV2vec was augmented with an MLM-style loss, ASR performance can be further
improved and AV2vec-MLM achieved the lowest WER. For AVSR task, we observed
that the face based model achieved better performance than the lip based
counterpart under low SNR ($<$0dB) conditions while they achieved close
performance under high SNR ($>$0dB) conditions. These results also suggest
that faces provide better visual information than lips for speech recognition
because the AVSR model relies more heavily on visual information under a
higher noise condition. In general, AV2vec achieved better performance under
low SNR conditions and worse performance under high SNR conditions compared
with the baseline. AV2vec-MLM model achieved the best performance in AVSR task
at most SNR levels and this proves the effectiveness of multitask learning for
AV2vec for generating better audio-visual representations. We also compared
our method with previous supervised-based models (Table 1, row 2-4) and the
results suggest our method achieved better or competitive performance in three
tasks with less training data.

Table 2: WER (%) results on the LRS3 test set in ablation study of modality dropout. “-m” and “-x” denote dropout the same and the opposite modality for the teacher respectively if one modality is dropped for the student. Method | SNR (dB)=
---|---
-10 | -5 | 0 | 5 | 10 | $\infty$
AV2vec | 26.9 | 17.2 | 10.5 | 7.8 | 6.8 | 5.6
AV2vec-m | 37.3 | 26.0 | 17.1 | 13.4 | 11.7 | 9.9
AV2vec-x | 37.8 | 23.6 | 15.1 | 11.6 | 10.1 | 8.9

###  4.4 Ablation studies

In ablation studies, we investigated the modality dropout used by the teacher.
Three configurations were compared including our proposed method AV2vec which
adopts no teacher modality dropout, AV2vec-m in which the teacher dropouts the
same modality as the student and AV2vec-x dropouts the opposite modality if
one modality is dropped in the student. Ablation studies were conducted based
on the 30-hour labeled data for the AVSR task and results are presented in
Table 2. From the table, our proposed method achieved the best results, which
confirmed the effectiveness of multimodal distillation used in our proposed
method.

Ablation studies that averaged different numbers of teacher layer
representations were also conducted and the results are shown in Table 3. From
the table, it’s observed that multiple layers improved AVSR accuracy, which is
consistent with the previous study [20]. Our method averaging the last 8
layers following [20] achieved the best performance.

Table 3: WER (%) results on the LRS3 test set when averaging the last $k$ layer teacher hidden representations for generating the student’s target. † denotes the proposed method. $k$ | SNR (dB)=
---|---
-10 | -5 | 0 | 5 | 10 | $\infty$
12 | 30.6 | 20.1 | 12.6 | 9.9 | 8.9 | 7.7
8† | 26.9 | 17.2 | 10.5 | 7.8 | 6.8 | 5.6
4 | 32.7 | 20.7 | 13.0 | 9.5 | 7.9 | 6.7
1 | 62.7 | 55.9 | 50.0 | 47.9 | 46.7 | 46.1

##  5 Conclusion

In this work, we propose AV2vec, a self-supervised training method for audio-
visual speech representation learning. It has a student and a teacher module,
where the student performs a masked latent feature regression task and the
multimodal targets are generated online by the momentum-based teacher. Since
our method is trained without iteration, it reduces training time
significantly compared to AV-HuBERT. We also present AV2vec-MLM, in which
AV2vec is augmented with an MLM-style loss by multitask learning. Our
experimental results indicate that AV2vec learned better visual
representations but slightly worse audio representations compared to the Av-
HuBERT baseline. Combining AV2vec with an MLM-style loss further improved the
performance and our proposed AV2vec-MLM method achieved the best results on
downstream tasks including ASR, VSR, and AVSR.

## References

  * [1] H. McGurk and J. MacDonald,  “Hearing lips and seeing voices,”  Nature, vol. 264, no. 5588, pp. 746–748, 1976.
  * [2] T. Rahne, M. Böckmann, H. von Specht, and E. S Sussman,  “Visual cues can modulate integration and segregation of objects in auditory scene analysis,”  Brain research, vol. 1144, pp. 127–135, 2007.
  * [3] M. Wand, J. Koutník, and J. Schmidhuber,  “Lipreading with long short-term memory,”  in ICASSP. IEEE, 2016, pp. 6115–6119.
  * [4] B. Martinez, P. Ma, S. Petridis, and M. Pantic,  “Lipreading using temporal convolutional networks,”  in ICASSP. IEEE, 2020, pp. 6319–6323.
  * [5] Georgios T. Themos S.,  “Combining residual networks with LSTMs for lipreading,”  in INTERSPEECH, 2017, pp. 3652–3656.
  * [6] J.-X. Zhang, K. Richmond, Z.-H. Ling, and L. Dai,  “TaLNet: Voice reconstruction from tongue and lip articulation with transfer learning from text-to-speech synthesis,”  AAAI, vol. 35, no. 16, pp. 14402–14410, May 2021.
  * [7] Y. Mroueh, E. Marcheret, and V. Goel,  “Deep multimodal learning for audio-visual speech recognition,”  in ICASSP. IEEE, 2015, pp. 2130–2134.
  * [8] S. Petridis, T. Stafylakis, P. Ma, G. Tzimiropoulos, and M. Pantic,  “Audio-visual speech recognition with a hybrid CTC/Attention architecture,”  in SLT. IEEE, 2018, pp. 513–520.
  * [9] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman,  “Deep audio-visual speech recognition,”  IEEE transactions on pattern analysis and machine intelligence, 2018\.
  * [10] P. Ma, S. Petridis, and M. Pantic,  “End-to-end audio-visual speech recognition with conformers,”  in ICASSP. IEEE, 2021, pp. 7613–7617.
  * [11] T. Makino, H. Liao, Y. Assael, B. Shillingford, B. Garcia, O. Braga, and O. Siohan,  “Recurrent neural network transducer for audio-visual speech recognition,”  in ASRU. IEEE, 2019, pp. 905–912.
  * [12] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick,  “Momentum contrast for unsupervised visual representation learning,”  in CVPR, 2020, pp. 9729–9738.
  * [13] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton,  “A simple framework for contrastive learning of visual representations,”  in ICML. PMLR, 2020, pp. 1597–1607.
  * [14] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, et al.,  “Bootstrap your own latent-a new approach to self-supervised learning,”  NeurIPS, vol. 33, pp. 21271–21284, 2020.
  * [15] H. Bao, L. Dong, S. Piao, and F. Wei,  “BEiT: BERT pre-training of image transformers,”  in ICLR, April 2022.
  * [16] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick,  “Masked autoencoders are scalable vision learners,”  in CVPR, 2022, pp. 16000–16009.
  * [17] S. Schneider, A. Baevski, R. Collobert, and M. Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  in INTERSPEECH, 2019, pp. 3465–3469.
  * [18] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli,  “Wav2vec 2.0: A framework for self-supervised learning of speech representations,”  NeurIPS, vol. 33, pp. 12449–12460, 2020.
  * [19] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed,  “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [20] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli,  “data2vec: A general framework for self-supervised learning in speech, vision and language,”  in ICML. 2022, vol. 162, pp. 1298–1312, PMLR.
  * [21] B. Korbar, D. Tran, and L. Torresani,  “Cooperative learning of audio and video models from self-supervised synchronization,”  NeurIPS, vol. 31, 2018.
  * [22] H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran,  “Self-supervised learning by cross-modal audio-video clustering,”  NeurIPS, vol. 33, pp. 9758–9770, 2020.
  * [23] S. Ma, Z. Zeng, D. McDuff, and Y Song,  “Contrastive learning of global and local video representations,”  in NeurIPS, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, Eds. 2021, vol. 34, pp. 7025–7040, Curran Associates, Inc.
  * [24] B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed,  “Learning audio-visual speech representation by masked multimodal cluster prediction,”  in ICLR. 2022, OpenReview.net.
  * [25] B. Shi, W.-N. Hsu, and A. Mohamed,  “Robust self-supervised audio-visual speech recognition,”  CoRR, vol. abs/2201.01763, 2022.
  * [26] A. Tarvainen and H. Valpola,  “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,”  in NeurIPS, 2017, pp. 1195–1204.
  * [27] Y. Higuchi, N. Moritz, J. L. Roux, and T. Hori,  “Momentum pseudo-labeling for semi-supervised speech recognition,”  in INTERSPEECH. 2021, pp. 726–730, ISCA.
  * [28] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin,  “Emerging properties in self-supervised vision transformers,”  in CVPR, 2021, pp. 9650–9660.
  * [29] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong,  “Image BERT pre-training with online tokenizer,”  in ICLR, 2022.
  * [30] J.-X. Zhang, G. Wan, and J. Pan,  “Is lip region-of-interest sufficient for lipreading?,”  in ICMI, 2022.
  * [31] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  CoRR, vol. abs/1607.08022, 2016.
  * [32] T. Afouras, J. S. Chung, and A. Zisserman,  “LRS3-TED: a large-scale dataset for visual speech recognition,”  in arXiv preprint arXiv:1809.00496, 2018.
  * [33] D. Snyder, G. Chen, and D. Povey,  “MUSAN: A music, speech, and noise corpus,”  ArXiv, vol. abs/1510.08484, 2015.
  * [34] B. Xu, C. Lu, Y. Guo, and J. Wang,  “Discriminative multi-modality speech recognition,”  in CVPR, 2020, pp. 14433–14442.


11institutetext:  German Research Center for Artificial Intelligence (DFKI),
Germany 22institutetext: RPTU Kaiserslautern-Landau, Germany 33institutetext:
Equal Contribution
33email<EMAIL_ADDRESS>
# Dynamic Attention-Guided Diffusion for
Image Super-Resolution

Brian B. Moser 11 2 2 3 3 0000-0002-0290-7904 Stanislav Frolov 11 2 2 3 3
0000-0003-2700-5031 Federico Raue 11 0000-0002-8604-6207 Sebastian Palacio 11
0000-0002-8656-9569 Andreas Dengel 1122 0000-0002-6100-8255

###### Abstract

Diffusion models in image Super-Resolution (SR) treat all image regions with
uniform intensity, which risks compromising the overall image quality. To
address this, we introduce “You Only Diffuse Areas” (YODA), a dynamic
attention-guided diffusion method for image SR. YODA selectively focuses on
spatial regions using attention maps derived from the low-resolution image and
the current time step in the diffusion process. This time-dependent targeting
enables a more efficient conversion to high-resolution outputs by focusing on
areas that benefit the most from the iterative refinement process, i.e.,
detail-rich objects. We empirically validate YODA by extending leading
diffusion-based methods SR3 and SRDiff. Our experiments demonstrate new state-
of-the-art performance in face and general SR across PSNR, SSIM, and LPIPS
metrics. A notable finding is YODA’s stabilization effect by reducing color
shifts, especially when training with small batch sizes.

##  1 Introduction

The goal of image Super-Resolution (SR) is to enhance Low-Resolution (LR) into
High-Resolution (HR) images [35]. This process has a significant impact in
fields such as medical imaging, remote sensing, and consumer electronics [33,
16, 44]. Despite its long history, image SR remains a fascinating yet
challenging domain due to its inherently ill-posed nature: any given LR image
can lead to several valid HR images, and vice versa [2, 40]. Thanks to deep
learning, image SR has made significant progress [13]. Initial regression-
based methods, such as early convolutional neural networks, work great at low
magnification ratios[27, 7, 43]. However, they fail to produce high-frequency
details at high magnification ratios and generate over-smoothed images [34].

Recently, diffusion models have emerged with better human-rated quality
compared to regression-based methods, but they also introduce new challenges
[21, 38, 46, 9]. Their indiscriminate processing of image regions leads to
computational redundancies and suboptimal enhancements. Some recent methods
have reduced computational demands by working in latent space like LDMs [37],
by exploiting the relationship between LR and HR latent representations like
PartDiff [48], or by starting with a better-initialized forward diffusion
instead of pure Gaussian noise like in CCDF [10]. However, there is still a
lack of strategies that adapt the model capacity based on the spatial
importance of image regions.

In this paper, we question the common approach of diffusion models for image
SR: Is it necessary to update the entire image at every time step in the
reverse diffusion process? We hypothesize that not all image regions require
the same level of detail enhancement. For instance, a face in the foreground
may need more attention than a simple, monochromatic background. Recognizing
this variability in the need for detail enhancement underscores a critical
inefficiency in traditional diffusion methods. Treating all image regions with
uniform intensity risks compromising the overall image quality. To bridge this
gap, we introduce “You Only Diffuse Areas” (YODA), an efficient diffusion
mechanism focusing on detail-rich areas using time-dependent and attention-
guided masking. YODA starts by obtaining an attention map, highlighting
regions that need more refinement. After identification, YODA systematically
replaces the highlighted regions with SR predictions during the denoising
process, depending on the current time step. As a result, detail-rich areas
are refined more often. Our approach is analogous to inpainting methods like
RePaint [31], where only a pre-defined masked region is updated to generate
complementing content. In YODA’s case, however, the selection of regions to
update is time-dependent. We design a dynamic approach that creates expanding
masks, starting from detail-rich regions and converging towards enhancing the
overall image. A key advantage of YODA is its compatibility with existing
diffusion models, allowing for a plug&play application. We integrate YODA with
SR3 [38] for face SR and SRDiff [25] for general SR, achieving notable
improvements in image quality. Interestingly, YODA also improves the training
process. For instance, our face SR experiments require a significantly reduced
batch size due to limited hardware access. When training with smaller batch
sizes, SR3 also suffers from color shifts [42, 8], while YODA produces color
distributions faithfully. In summary, our work:

  * •

introduces YODA, an attention-guided diffusion approach that emphasizes
important image areas through masked refinement. As a result, it refines
detail-rich areas more often, which leads to higher overall image quality.

  * •

demonstrates that attention-guided diffusion results in better training
conditions, accurate color predictions, and better perceptual quality.

  * •

empirically shows that YODA outperforms leading diffusion models in face and
general SR tasks.

  * •

shows that YODA improves the training performance when using smaller batch
sizes, which is crucial in limited hardware scenarios.

##  2 Background

Our method uses attention maps for attention-guided diffusion. Thus, this
section introduces the main components: DDPMs [21] and the DINO framework [6].

###  2.1 DDPMs

Denoising Diffusion Probabilistic Models (DDPMs) employ two distinct Markov
chains [21]: the first models the forward diffusion process $q$ transitioning
from an input $\mathbf{x}$ to a pre-defined prior distribution with
intermediate states $\mathbf{z}_{t}$, $0<t\leq T$, while the second models the
backward diffusion process $p$, reverting from the prior distribution back to
the intended target distribution
$p\left(\mathbf{z}_{0}\mid\mathbf{z}_{T},\mathbf{x}\right)$. In image SR, we
designate $\mathbf{x}$ as the LR image and the target $\mathbf{z}_{0}$ as the
desired HR image. The prior distribution is generally set manually, e.g.,
Gaussian noise.

####  2.1.1 Forward Diffusion

In the forward diffusion phase, an HR image $\mathbf{z}_{0}$ is incrementally
modified by adding Gaussian noise over a series of time steps. This process
can be mathematically represented as:

$q(\mathbf{z}_{t}\mid\mathbf{z}_{t-1})=\mathcal{N}(\mathbf{z}_{t}\mid\sqrt{1-\alpha_{t}}\,\mathbf{z}_{t-1},\alpha_{t}\mathbf{I})$
(1)

The hyperparameters $0<\alpha_{1:T}<1$ represent the noise variance injected
at each time step. It is possible to sample from any point in the noise
sequence without needing to generate all previous steps through the following
simplification [39]:

$q(\mathbf{z}_{t}\mid\mathbf{z}_{0})=\mathcal{N}(\mathbf{z}_{t}\mid\sqrt{\gamma_{t}}\,\mathbf{z}_{0},(1-\gamma_{t})\mathbf{I}),$
(2)

where $\gamma_{t}=\prod_{i=1}^{t}(1-\alpha_{i})$ . The intermediate step
$\mathbf{z}_{t}$ is derived by:

$\mathbf{z}_{t}=\sqrt{\gamma_{t}}\cdot\mathbf{z}_{0}+\sqrt{1-\gamma_{t}}\cdot\varepsilon_{t},\quad\varepsilon_{t}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right)$
(3)

####  2.1.2 Backward Diffusion

The backward diffusion process is where the model learns to denoise,
effectively reversing the forward diffusion to recover the HR image. In image
SR, the reverse process is conditioned on the LR image to guide the generation
of the HR image:

$p_{\theta}\left(\mathbf{z}_{t-1}\mid\mathbf{z}_{t},\mathbf{x}\right)=\mathcal{N}\left(\mathbf{z}_{t-1}\mid\mu_{\theta}(\mathbf{z}_{t},\mathbf{x},\gamma_{t}),\Sigma_{\theta}(\mathbf{z}_{t},\mathbf{x},\gamma_{t})\right)$
(4)

The mean $\mu_{\theta}$ depends on a parameterized denoising function
$f_{\theta}$, which can either predict the added noise $\varepsilon_{t}$ or
the underlying HR image $\mathbf{z}_{0}$. Following the standard approach of
Ho et al. [21], we focus on predicting the noise. Hence, the mean is:

$\mu_{\theta}(\mathbf{x},\mathbf{z}_{t},\gamma_{t})=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{z}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\gamma_{t}}}f_{\theta}\left(\mathbf{x},\mathbf{z}_{t},\gamma_{t}\right)\right)$
(5)

Following Saharia et al. [38], setting the variance of
$p_{\theta}(\mathbf{z}_{t-1}|\mathbf{z}_{t},\mathbf{x})$ to $(1-\alpha_{t})$
yields the subsequent refining step with
$\varepsilon_{t}\sim\mathcal{N}(\textbf{0},\,\textbf{I})$:

$\mathbf{z}_{t-1}\leftarrow\mu_{\theta}(\mathbf{x},\mathbf{z}_{t},\gamma_{t})+\sqrt{1-\alpha_{t}}\varepsilon_{t}$
(6)

####  2.1.3 Optimization

The optimization goal for DDPMs is to train the parameterized model to
accurately predict the noise added during the diffusion process. The loss
function used to measure the accuracy of the noise prediction is:

$\mathcal{L}\left(\theta\right)=\mathop{\mathbb{E}}_{(\mathbf{x},\mathbf{z}_{0})}\mathop{\mathbb{E}}_{t}\bigg{\lVert}\varepsilon_{t}-f_{\theta}\left(\mathbf{x},\mathbf{z}_{t},\gamma_{t}\right)\bigg{\rVert}_{1}$
(7)

###  2.2 DINO

DINO is a self-supervised learning approach for feature extractors on
unlabeled data [6]. It employs a teacher and a student network, where the
student learns to imitate the features learned by the teacher. The student
gets only local views of the image (i.e., $96\times 96$), whereas the teacher
receives global views (i.e., $224\times 224$). This setup encourages the
student to learn “local-to-global” correspondences. The features learned
through self-supervision are directly accessible in the self-attention
modules. These self-attention maps provide information on the scene layout and
important object boundaries. We leverage the generality, availability, and
robustness of these attention maps as a measure of an image’s saliency to
guide the diffusion process. In another context, a similar approach has been
applied to image compression, demonstrating its ability to capture essential
image content in the attention maps [3]. More details can be found in the
supplementary material.

##  3 Methodology

This section describes the components of our proposed method, “You Only
Diffuse Areas” (YODA). YODA is divided into three main phases:

  * •

Identifying Key Regions: Estimate the importance of pixel positions in a LR
image $\mathbf{x}\in\mathbb{R}^{H\times W\times C}$ with an attention map
$\mathbf{A}\in\mathbb{R}^{H\times W}$.

  * •

Time-Dependent Masking: Using $\mathbf{A}$, define time-dependent, binary
masks $\mathbf{M}:\mathbb{N}_{0}\to\\{0,1\\}^{H\times W}$ to identify salient
areas at time step $t\in\mathbb{N}_{0}$ of the diffusion process.

  * •

Guided Backward Diffusion: Condition the diffusion process on the regions
identified by the time-dependent masking $\mathbf{M}(t)$.

###  3.1 Identifying Key Regions

Super-resolving with YODA begins by identifying areas of an image with greater
details. This is achieved by generating an attention map $\mathbf{A}$ with
$0\leq\mathbf{A}_{i,j}\leq 1$ from the LR image $\mathbf{x}$. The greater the
value of $\mathbf{A}_{i,j}$, the more important it is. Note that $\mathbf{A}$
has to be generated only once for each image.

For generating $\mathbf{A}$, we evaluated several approaches, including innate
methods (i.e., not-learnable) and learnable methods, i.e., ResNet and
Transformer architectures [20, 14]. For the latter, we leverage the DINO
framework for its robustness in self-supervised learning, extracting refined
attention maps directly from LR images without necessitating extra annotated
data [6]. This choice is motivated by DINO’s demonstrated efficacy in
highlighting essential features within images using pre-existing models, e.g.,
for image compression [3]. An additional overview of how DINO and the
attention maps are used is shown in the supplementary materials. Next, we
describe the process of creating time-dependent masks for backward diffusion,
utilizing the attention map $\mathbf{A}$.

###  3.2 Time-Dependent Masking

Given the LR input image $\mathbf{x}$ and the attention map $\mathbf{A}$, we
introduce a strategy to focus the diffusion process to detail-rich areas. Each
position in $\mathbf{A}$ indicates its semantic importance, influencing the
number of refinement steps it receives. Thus, for two positions $(i,j)$ and
$(i^{\prime},j^{\prime})$ with
$\mathbf{A}_{i,j}>\mathbf{A}_{i^{\prime},j^{\prime}}$, YODA applies more
refinement steps to the location $(i,j)$ than to $(i^{\prime},j^{\prime})$.
Since $0\leq\mathbf{A}_{i,j}\leq 1$, the number of diffusion steps employed to
a specific position $(i,j)$ is determined as a proportion of the maximum time
steps, $T$. For instance, $\mathbf{A}_{i,j}=0.7$ means $(i,j)$ is refined
during 70% of all diffusion steps. In addition, a lower bound hyperparameter
$0<l<1$ ensures that every region undergoes a minimum level of refinement. In
other words, the hyperparameter $l$ guarantees that every spatial position is
refined at least $l\cdot T$ times. Note that the backward diffusion process
starts from time step $T$ and ends in time step 0. We define the time-
dependent masking process at time step $0\leq t\leq T$ with $t\to 0$ as
follows:

$\mathbf{M}(t)_{i,j}=\begin{cases}1\text{, if }T\cdot(\mathbf{A}_{i,j}+l)\geq
t\\\ 0\text{, otherwise}\end{cases}$ (8)

This equation ensures that the diffusion process gets applied a variable
number of times for different regions, allowing the salient areas to diffuse
over a longer time span. It is important to highlight that once a spatial
position is marked for refinement, it continues to undergo refinement across
all subsequent steps as $t$ approaches 0:
$\mathbf{M}(t)_{i,j}\geq\mathbf{M}(t-k)_{i,j}\,\forall k>0$. Figure 1 shows an
example of our time-dependent masking. For each time step $t$, we can
determine with $\mathbf{M}(t)_{i,j}=1$ whether a given spatial position
$(i,j)$ should be refined or not.

Figure 1:  Overview of YODA. First, extract an attention map $\mathbf{A}$ from
the LR input. Next, use the values of $\mathbf{A}$ to produce a time-dependent
masking $\mathbf{M}(t)$. For $t:T\to 0$, the area of selected pixels expands
from detail-rich regions to the whole image. Our diffusion process uses these
masks for dynamic and attention-guided refinement, emphasizing important
regions. More specifically, it starts with masked areas that need refinement
(derived from $\mathbf{z}_{t}$ and $\mathbf{M}(t)$) and LR regions, which
retain the noise level needed for the next time step. Finally, the SR and LR
areas are combined to form a whole image with no masked-out regions for the
next iteration.

###  3.3 Guided Backward Diffusion

YODA’s guided diffusion process iteratively refines the image from a noisy
state $\mathbf{z}_{T}$ to a high-resolution state $\mathbf{z}_{0}$. This phase
involves selectively refining areas based on the current time step’s mask,
$\mathbf{M}(t)$, and blending these refined areas with the unrefined,
remaining LR regions. YODA ensures a seamless transition between refined and
unrefined areas, improving image quality with a focus on key regions.

At each time step $t$, the areas that will be refined for transitioning from
$t$ to $(t-1)$ are determined based on the current iteration $\mathbf{z}_{t}$
and the current mask $\mathbf{M}(t)$:

$\mathbf{\widetilde{z}}_{t}\leftarrow\mathbf{M}(t)\odot\mathbf{z}_{t}$ (9)

Next, we divide the image into two components that will later be combined as
the output for the next time step: $\mathbf{z}^{SR}_{t-1}$, which is the
refined image prediction, and $\mathbf{z}^{LR}_{t-1}$, the complementary LR
image. The state $\mathbf{z}^{LR}_{t-1}$ represents unchanged LR areas but is
sampled using $\mathbf{x}$ as the mean. Both components acquire the same noise
level $\Sigma_{\theta}(\mathbf{\widetilde{z}}_{t},\mathbf{x},\gamma_{t})$, and
can be described by:

$\displaystyle\mathbf{z}^{SR}_{t-1}$
$\displaystyle\sim\mathcal{N}\left(\mu_{\theta}(\mathbf{\widetilde{z}}_{t},\mathbf{x},\gamma_{t}),\Sigma_{\theta}(\mathbf{\widetilde{z}}_{t},\mathbf{x},\gamma_{t})\right)$
(10) $\displaystyle\mathbf{z}^{LR}_{t-1}$
$\displaystyle\sim\mathcal{N}\left(\mathbf{x},\Sigma_{\theta}(\mathbf{\widetilde{z}}_{t},\mathbf{x},\gamma_{t})\right)$
(11)

Finally, YODA combines the complementing, non-overlapping image regions111This
formulation is similar to RePaint [31], a method for diffusion-based
inpainting. While RePaint uses a constant mask for all time steps, YODA uses a
time-dependent procedure that enables dynamic control over the refined image
regions.:

$\mathbf{z}_{t-1}\leftarrow\mathbf{M}(t)\odot\mathbf{z}^{SR}_{t-1}+(1-\mathbf{M}(t))\odot\mathbf{z}^{LR}_{t-1}$
(12)

Consequently, the areas refined by $\mathbf{z}^{SR}_{t}$ expand as $t\to 0$,
whereas the areas described by $\mathbf{z}^{LR}_{t}$ shrink in size. The new
state, $\mathbf{z}_{t-1}$, now contains both SR and LR areas and, importantly,
does not have any masked-out regions. As a result, $\mathbf{z}_{t-1}$ can be
used in the next iteration step. This guided refinement is depicted on the
right part of Figure 1.

###  3.4 Optimization

To confine the backward diffusion process to specific image regions as
determined by the current time step $0\leq t\leq T$ and the corresponding mask
$\mathbf{M}(t)$, YODA improves the training objective Equation 7 as follows:

$\mathcal{L}\left(\theta\right)=\mathop{\mathbb{E}}_{(\mathbf{x},\mathbf{y})}\mathop{\mathbb{E}}_{t}\bigg{\lVert}\mathbf{M}(t)\odot\left[\varepsilon_{t}-f_{\theta}\left(\mathbf{x},\mathbf{z}_{t},\gamma_{t}\right)\right]\bigg{\rVert}_{1}$
(13)

The loss function focuses on the highlighted spatial regions within the mask
$\mathbf{M}(t)$. Consequently, YODA optimizes only areas described by
$\mathbf{M}(t)$.

##  4 Experiments

We start by analyzing different methods for obtaining attention maps for YODA.
Then, we evaluate YODA and compare its performance in tandem with SR3 [38] for
face-only, and SRDiff [25] for general SR. We chose SR3 and SRDiff because
they are the most prominent representative diffusion models for image SR in
the respective tasks, where YODA can be integrated straightforwardly. However,
YODA can be theoretically applied to any existing method. We present
quantitative and qualitative results for both tasks. Our method achieves high-
quality results for both tasks and outperforms the baselines using standard
metrics such as PSNR, SSIM, and LPIPS [34]. All experiments were run on a
single NVIDIA A100-80GB GPU. In the supplementary materials, we discuss the
complexity of YODA and explore its potential synergies with other diffusion
model research.

Figure 2:  Comparison of attention maps (blue = low attention; yellow = high
attention). Top row denotes maps derived from ResNet-50 using DINO. It shows
various attention head outputs and the max aggregation of all attention maps
(MAX). Bottom row denotes non-learnable methods, namely Gaussian, Edge-based,
and using SIFT’s points of interest. The maps are subsequently used to produce
time-dependent binary masks.

###  4.1 Analysis of Attention Maps

Since the dynamic masking in YODA relies on attention maps, we thoroughly
evaluate different choices. We considered the pre-trained attention heads from
the last layer of DINO with the respective neural network, i.e., ResNet and
ViT [20, 14]. For ResNet-50, we used a dedicated method to extract the
attention maps from its weights [19]. A qualitative visual comparison of
attention maps generated with DINO and ResNet-50 is shown in Figure 2. It
shows that YODA captures and highlights perceptually essential areas. More
visual results can be found in the supplementary materials. In addition to
different neural networks, we test non-learnable methods to extract attention
maps, also shown in Figure 2:

  * •

Gaussian: Placing a simple 2D Gaussian pattern at the center of the image
provides a straightforward approach, which relies on the assumption that the
essential parts of an image are centered.

  * •

Edge-based: Using the Canny edge detector, the attention maps are defined by
the edges of the image, where adjacent and near edges are connected and
blurred.

  * •

Scale-Invariant Feature Transform (SIFT): Through Gaussian differences, SIFT
[30] provides an attention map characterized by scale invariance. It produces
an attention map by applying 2D Gaussian patterns around the points of
interest.

Table 1:  Results of using different attention maps with SR3+YODA for $16\times 16\rightarrow 128\times 128$ on CelebA-HQ. Aggregating the attention maps extracted with DINO and ResNet-50 backbone under the MAX strategy performs best.  | Attention Maps for YODA |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---|---
| PULSE (No Mask) | 16.88 | 0.440 | n.a.
| FSRGAN (No Mask) | 23.01 | 0.620 | n.a.
| SR3 Reported (No Mask) | 23.04 | 0.650 | n.a.
| SR3 Reproduced (No Mask) | 22.35 | 0.646 | 0.082
| Non-learnable, fixed: Gaussian | 22.13 | 0.602 | 0.260
| Non-learnable, adaptive: Edge-based | 22.93 | 0.648 | 0.151
| Non-learnable, adaptive: SIFT | 22.84 | 0.678 | 0.095
ViT-S/8 | Attention-Head 0 | 22.91 | 0.650 | 0.105
Attention-Head 1 | 22.43 | 0.616 | 0.130
Attention-Head 2 | 22.55 | 0.633 | 0.111
Attention-Head 3 | 22.73 | 0.641 | 0.110
Attention-Head 4 | 22.85 | 0.645 | 0.097
Attention-Head 5 | 22.86 | 0.648 | 0.101
Attention-AVG | 23.25 | 0.663 | 0.122
Attention-MAX | 23.46 | 0.683 | 0.103
ResNet-50 | Attention-Head 0 | 22.82 | 0.649 | 0.115
Attention-Head 1 | 22.54 | 0.627 | 0.117
Attention-Head 2 | 22.84 | 0.650 | 0.107
Attention-Head 3 | 22.78 | 0.645 | 0.105
Attention-Head 4 | 22.38 | 0.620 | 0.127
Attention-Head 5 | 22.50 | 0.630 | 0.119
Attention-AVG | 23.55 | 0.682 | 0.093
Attention-MAX | 23.84 | 0.695 | 0.072

Table 1 presents the results of our study with several baselines and masking
variants for $16\times 16\rightarrow 128\times 128$ scaling on the CelebA-HQ
dataset. The straightforward Gaussian approach performs worst as it does not
adapt to image features. The edge-based segmentation and SIFT methods showed
improved performance over the reproduced baseline using a small batch size.
However, they underperformed relative to the reported SR3 [38] results, which
used a larger batch size. In contrast, using DINO [6] to extract attention
maps improves performance, regardless of the choice of backbone (ResNet-50 and
ViT-S/8). We tested individual attention heads (0 to 5) independently, along
with combination strategies that include averaging (AVG) and selecting the
maximum value (MAX). The MAX combination achieved the best results compared to
individual heads or the AVG combination. DINO [6] with a ResNet-50 backbone
performs best, and we used the MAX aggregation strategy for all main
experiments.

Figure 3:  (Left) Ratio comparison between diffused pixels using our time-
dependent masking approach and the total number of pixel updates in standard
diffusion. On average, DINO with a ResNet-50 backbone leads to more pixel
updates than the VIT-S/8 backbone. The lower bound, defined by $l$, is a
threshold to eliminate areas that would never undergo diffusion. (Right)
Refined image area in percentage across time steps for the MAX combination.
Note that the sampling process goes from $T=500$ to $T=0$. ResNet-50 initiates
the refinement process much earlier, advances more rapidly toward refining the
entire image, and has a higher standard deviation.

The left part of Figure 3 provides more information on the ratio of diffused
pixels using our time-dependent masking. The upper bound is 100%, which is the
case if diffusion is applied across all pixel locations throughout every time
step (as in standard diffusion). Thus, any result under 100% shows that not
all pixels are diffused during all time steps. As can be seen, DINO [6]
coupled with ResNet-50 produces higher attention values, thereby leading to
more total pixel updates. Also, the high variance of the ResNet-50 backbone
indicates a high adaption capability. It can employ 100% of the updates for
some samples, a characteristic not observed with the ViT-S/8 backbone.
Nevertheless, the improved performance with the ViT-S/8 backbone compared to
non-learnable methods and its low ratio of diffused pixels makes it an
attractive candidate for future work regarding inference speed optimization
based on sparser diffusion.

The right part of Figure 3 shows the ratio of diffused pixels depending on
time steps with the MAX aggregation for the ResNet-50 and the ViT-S/8
backbone. As the backward diffusion goes from $T$ to $0$, ResNet-50 backbone
initiates and incorporates the refinement of the whole image areas more
quickly than ViT-S/8. For the first 200 time steps, less than 20% of the image
area is addressed by the attention map derived by ViT-S/8, whereas ResNet-50
has already developed to 100%. Therefore, this observation leads to our
assumption that the superior performance of the ResNet-50 backbone may be
attributed to its faster progression in refining the image. In other words, it
performs better by progressing from 0% to 100% of the image area at an earlier
stage. Intermediate diffusion results can be found in the supplementary
materials.

Table 2:  Results on face SR with 4× scaling ($64\times 64\rightarrow 256\times 256$) and 8× scaling ($64\times 64\rightarrow 512\times 512$) on CelebA-HQ. The models were trained for 1M steps on FFHQ and a reduced batch size of 4 and 8 instead of 256 to fit on a single A100-80GB GPU. Scaling | Model |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---|---
4× | SR3 [38] | 17.98 | 0.607 | 0.138
| SR3 + YODA | 26.33 | 0.838 | 0.090
8× | PartDiff ($K$=25) [48] | - | - | 0.222
| PartDiff ($K$=50) [48] | - | - | 0.217
| SR3 [38] | 17.44 | 0.631 | 0.147
| SR3 + YODA | 25.04 | 0.800 | 0.126

###  4.2 Face Super-Resolution

####  4.2.1 Details

We use the FFHQ dataset [23] for training, which contains 50,000 high-quality
facial images. We adopted the AdamW [29] optimizer, using a weight decay of
0.0001 and a learning rate of 5e-5. The number of sampling steps is set to
$T_{\text{train}}=500$. We use the CelebA-HQ dataset [22] for evaluation,
which contains 30,000 facial images. The number of sampling steps is set to
$T_{\text{eval}}=200$. We trained all models for 1M iterations as in SR3 [38].
We evaluated three scenarios: $16\times 16\rightarrow 128\times 128$,
$64\times 64\rightarrow 256\times 256$, and $64\times 64\rightarrow 512\times
512$. Due to hardware limitations and missing quantitative results in the
original publication of SR3, our experiments required a reduction from the
originally used batch size of 256: we used a batch size of 4 for the $64\times
64\rightarrow 512\times 512$, and 8 for the $64\times 64\rightarrow 256\times
256$ scenario to fit on a single A100-80GB GPU.

####  4.2.2 Results

The results in Table 2 show a significant improvement when SR3 is coupled with
YODA across all examined metrics. We explain the poor performance of SR3 with
a phenomenon that is also observed by other authors [42, 8]. SR3 suffers from
a color shift issue, which we attribute to the reduced batch size necessitated
by our hardware limitations. Examples are shown in Figure 4 and Figure 5. This
color shift manifests in a pronounced deviation in pixel-based metrics such as
PSNR and SSIM but only slightly decreased perceptual quality in terms of
LPIPS. Our results suggest that YODA’s role extends beyond mere performance
enhancement. It actively mitigates the color shift issue and stabilizes
training, especially when faced with hardware constraints. With YODA, SR3 can
be trained with a much smaller batch size and still achieves strong
performance. Figure 5 offers another qualitative comparison between SR3 and
SR3+YODA, highlighting subtle yet important differences, e.g., around the
eyes, mouth, and hair.

LR

SR3
$\uparrow$ PSNR: 11.811
$\uparrow$ SSIM:  0.3311
$\downarrow$ LPIPS: 0.2847

SR3+YODA
$\uparrow$ PSNR: 18.069
$\uparrow$ SSIM:  0.6588
$\downarrow$ LPIPS: 0.2628

HR

Figure 4:  A comparison of LR, HR, SR3, and SR3+YODA images in the $64\times
64\rightarrow 256\times 256$ setting (4x). SR3 suffers from color shift
issues, as also observed by [42, 8]. Our method solves this issue and produces
higher quality reconstructions.

LR

SR3
$\uparrow$ PSNR: 23.061
$\uparrow$ SSIM:  0.6208
$\downarrow$ LPIPS: 0.0504

SR3+YODA
$\uparrow$ PSNR: 23.289
$\uparrow$ SSIM:  0.6334
$\downarrow$ LPIPS: 0.0502

HR

Figure 5:  A comparison of LR, HR, SR3, and SR3+YODA images in the $16\times
16\rightarrow 128\times 128$ setting (8x). The color shift in SR3 can still be
observed (e.g., see upper right corners). YODA produces higher-quality images
without color shift issues.

###  4.3 General Super-Resolution

####  4.3.1 Details

The experimental design follows SRDiff [25] and its hyperparameters, which are
originally based on the experimental design of SRFlow [32]. For training, we
employed 800 2K resolution HQ images from DIV2K [1] and 2,650 images from
Flickr2K [41]. For testing, we used the DIV2K validation set of 100 images.
Furthermore, we evaluated SR3, which was not originally tested on DIV2K. As in
SRDiff, we extracted $40\times 40$ sub-images with a batch size of 16, AdamW
[29], a channel size of 64 with channel multipliers [1, 2, 2, 4] and $T=100$.

Figure 6:  Example zoom-in regions of images (first row) from the DIV2K dataset. LPIPS is reported in the boxes (the lower, the better). YODA consistently produces better texture and more high-frequency details.  Table 3: Quantitative results of 4× general SR on the DIV2K validation set.  Type | Methods |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
---|---|---|---|---
Interpolation | Bicubic | 26.70 | 0.77 | 0.409
Regression | EDSR [27] | 28.98 | 0.83 | 0.270
| LIIF [7] | 29.24 | 0.84 | 0.239
| RRDB [43] | 29.44 | 0.84 | 0.253
GAN | RankSRGAN [47] | 26.55 | 0.75 | 0.128
| ESRGAN [43] | 26.22 | 0.75 | 0.124
Flow | SRFlow [32] | 27.09 | 0.76 | 0.120
| HCFlow [26] | 27.02 | 0.76 | 0.124
Flow + GAN | HCFlow++ [26] | 26.61 | 0.74 | 0.110
VAE + AR | LAR-SR [18] | 27.03 | 0.77 | 0.114
Diffusion | SR3 [38] | 14.14 | 0.15 | 0.753
| SR3 + YODA | 27.24 | 0.77 | 0.127
| SRDiff [25] | 27.41 | 0.79 | 0.136
| SRDiff + YODA | 27.62 | 0.80 | 0.146

####  4.3.2 Results

Table 3 shows the 4× scaling general image SR results on the DIV2K validation
set. The reported values include regression-based methods, which typically
yield higher pixel-based scores than generative approaches [38]. This
disparity is due to PSNR/SSIM penalizing misaligned high-frequency details, a
known and significant challenge in the wider SR field [34].

We observe that results from SR3 underperform compared to YODA. Equipping SR3
with YODA improves image quality even without an extensive hyperparameter
search. We hypothesize that a vanilla SR3 strongly depends on larger batch
sizes and longer diffusion times.

Combining SRDiff with YODA demonstrates improved performance in PSNR (+0.21db)
and SSIM (+0.01), with a minor deterioration in LPIPS (+0.01). Thus, our
approach excels in pixel-based metrics but sees a marginal decline in the
perceptual metric. Nonetheless, when looking qualitatively at the predictions,
one can observe that SRDiff benefits from YODA, as shown in Figure 6. YODA
produces much better LPIPS values for perceptually essential areas, i.e., hair
and cars, but falls short in background areas, i.e., blurry grass or dark
ground. More results can be found in the supplementary materials.

Overall, YODA’s strengths are more significant when combined with SR3 than
with SRDiff. A critical distinction between SR3 and SRDiff is the handling of
conditional information, i.e., the LR image, which we identify as a potential
contributor to the reduced perceptual score LPIPS. SRDiff employs an LR
encoder that generates an embedding during the denoising phase. Meanwhile, SR3
directly uses the LR image during the backward diffusion.

Furthermore, SR3 operates on the full LR images, whereas SRDiff diffuses
residuals. Hence, the attention maps generated beforehand might not accurately
capture the essential regions of the sparse residual inputs. Another possible
reason for the lower perceptual score could be the image size during DINO
training, i.e., $224\times 224$ for the teacher network, which is much smaller
than the 2K resolution images in DIV2K. As such, fine-tuning DINO on larger-
scale images might be essential to capture more meaningful semantic features.

##  5 Conclusion

In this work, we presented “You Only Diffuse Areas" (YODA), an attention-
guided diffusion-based image SR approach that emphasizes specific areas
through time-dependent masking. YODA first extracts attention maps that
reflect the pixel-wise importance of each scene using a self-supervised,
general-purpose vision encoder. The attention maps are then used to guide the
diffusion process by focusing on key regions in each time step while providing
a fusion technique to ensure that masked and non-masked image regions are
correctly connected between two successive time steps. This targeting allows
for a more efficient transition to high-resolution outputs, prioritizing areas
that gain the most from iterative refinements, such as detail-intensive
regions. Beyond performance enhancement, YODA stabilizes training and
mitigates the color shift issue that emerges when a reduced batch size
constrains vanilla SR3. As a result, YODA consistently outperforms strong SR
baselines such as SR3 and SRDiff, while requiring less computational resources
by using smaller batch sizes.

##  6 Limitations & Future Work

A notable constraint of this study is its dependence on a good saliency
estimation. Even though DINO is trained to be a generic vision encoder, it has
known limitations that will reflect on the quality of YODA [11]. Meanwhile,
the modularity of YODA allows for the saliency model to be switched once a
better one becomes available. Additionally, DINO is explicitly trained on
input image resolutions such as $224\times 224$, which may not suffice for
image SR applications with much larger spatial sizes of the LR image. An ideal
solution would be a scale-invariant extraction of attention maps, e.g., a more
extended version of our SIFT-adapted approach. Also, we use a linear
correspondence between saliency and the number of steps. We posit that non-
linear mappings may affect the quality of results. Lastly, YODA has to be
extended for other recent state-of-the-art diffusion-based methods, e.g.,
latent-based methods like LDM [37] or unsupervised methods based on singular
value decomposition like DDRM [24] or DDNM [45], which is orthogonal to our
work (see supplementary materials for more details).

## Acknowledgment

This work was supported by the EU project SustainML (Grant 101070408) and by
Carl Zeiss Foundation through the Sustainable Embedded AI project
(P2021-02-009).

## References

  * [1] Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: CVPR Workshop (2017)
  * [2] Anwar, S., Barnes, N.: Densely residual laplacian super-resolution. In: IEEE TPAMI (2020)
  * [3] Baldassarre, F., El-Nouby, A., Jégou, H.: Variable rate allocation for vector-quantized autoencoders. In: ICASSP (2023)
  * [4] Bansal, A., Borgnia, E., Chu, H.M., Li, J., Kazemi, H., Huang, F., Goldblum, M., Geiping, J., Goldstein, T.: Cold diffusion: Inverting arbitrary image transforms without noise. Adv. Neural Inform. Process. Syst. 36 (2024)
  * [5] Bar-Tal, O., Yariv, L., Lipman, Y., Dekel, T.: Multidiffusion: Fusing diffusion paths for controlled image generation (2023)
  * [6] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021)
  * [7] Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: CVPR (2021)
  * [8] Choi, J., Lee, J., Shin, C., Kim, S., Kim, H., Yoon, S.: Perception prioritized training of diffusion models. 2022 ieee. In: CVPR (2022)
  * [9] Chung, H., Lee, E.S., Ye, J.C.: Mr image denoising and super-resolution using regularized reverse diffusion. In: IEEE Transactions on Medical Imaging (2022)
  * [10] Chung, H., Sim, B., Ye, J.C.: Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In: CVPR (2022)
  * [11] Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv preprint arXiv:2309.16588 (2023)
  * [12] Delbracio, M., Milanfar, P.: Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303.11435 (2023)
  * [13] Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. In: IEEE TPAMI (2015)
  * [14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [15] Du, R., Chang, D., Hospedales, T., Song, Y.Z., Ma, Z.: Demofusion: Democratising high-resolution image generation with no $$$. arXiv preprint arXiv:2311.16973 (2023)
  * [16] El-Shafai, W., Ali, A.M., El-Nabi, S.A., El-Rabaie, E.S.M., Abd El-Samie, F.E.: Single image super-resolution approaches in medical images based-deep learning: a survey. Multimedia Tools and Applications pp. 1–37 (2023)
  * [17] Frolov, S., Hinz, T., Raue, F., Hees, J., Dengel, A.: Adversarial text-to-image synthesis: A review. Neural Networks 144, 187–209 (2021)
  * [18] Guo, B., Zhang, X., Wu, H., Wang, Y., Zhang, Y., Wang, Y.F.: Lar-sr: A local autoregressive model for image super-resolution. In: CVPR (2022)
  * [19] Gur, S., Ali, A., Wolf, L.: Visualization of supervised and self-supervised neural networks via attribution guided factorization. In: AAAI. vol. 35, pp. 11545–11554 (2021)
  * [20] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [21] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Adv. Neural Inform. Process. Syst. (2020)
  * [22] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: ICLR (2018)
  * [23] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)
  * [24] Kawar, B., Elad, M., Ermon, S., Song, J.: Denoising diffusion restoration models. Adv. Neural Inform. Process. Syst. 35, 23593–23606 (2022)
  * [25] Li, H., Yang, Y., Chang, M., Chen, S., Feng, H., Xu, Z., Li, Q., Chen, Y.: Srdiff: Single image super-resolution with diffusion probabilistic models. In: Neurocomputing (2022)
  * [26] Liang, J., Lugmayr, A., Zhang, K., Danelljan, M., Van Gool, L., Timofte, R.: Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In: ICCV (2021)
  * [27] Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPR Workshop (2017)
  * [28] Liu, G.H., Vahdat, A., Huang, D.A., Theodorou, E.A., Nie, W., Anandkumar, A.: I 2 sb: Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.05872 (2023)
  * [29] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
  * [30] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. In: IJCV (2004)
  * [31] Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L.: Repaint: Inpainting using denoising diffusion probabilistic models. In: CVPR (2022)
  * [32] Lugmayr, A., Danelljan, M., Van Gool, L., Timofte, R.: Srflow: Learning the super-resolution space with normalizing flow. In: ECCV (2020)
  * [33] Moser, B.B., Frolov, S., Raue, F., Palacio, S., Dengel, A.: Dwa: Differential wavelet amplifier for image super-resolution. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds.) ICANN (2023)
  * [34] Moser, B.B., Raue, F., Frolov, S., Palacio, S., Hees, J., Dengel, A.: Hitchhiker’s guide to super-resolution: Introduction and recent advances. In: IEEE TPAMI (2023)
  * [35] Moser, B.B., Shanbhag, A.S., Raue, F., Frolov, S., Palacio, S., Dengel, A.: Diffusion models, image super-resolution and everything: A survey. arXiv preprint arXiv:2401.00736 (2024)
  * [36] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)
  * [37] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
  * [38] Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., Norouzi, M.: Image super-resolution via iterative refinement. In: IEEE TPAMI (2022)
  * [39] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML (2015)
  * [40] Sun, W., Chen, Z.: Learned image downscaling for upscaling using content adaptive resampler. In: IEEE Transactions on Image Processing (2020)
  * [41] Timofte, R., Gu, S., Wu, J., Van Gool, L.: Ntire 2018 challenge on single image super-resolution: Methods and results. In: CVPR Workshop (2018)
  * [42] Wang, J., Yue, Z., Zhou, S., Chan, K.C., Loy, C.C.: Exploiting diffusion prior for real-world image super-resolution. arXiv:2305.07015 (2023)
  * [43] Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: Enhanced super-resolution generative adversarial networks. In: ECCV Workshop (2018)
  * [44] Wang, X., Yi, J., Guo, J., Song, Y., Lyu, J., Xu, J., Yan, W., Zhao, J., Cai, Q., Min, H.: A review of image super-resolution approaches based on deep learning and applications in remote sensing. Remote Sensing 14(21), 5423 (2022)
  * [45] Wang, Y., Yu, J., Zhang, J.: Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 (2022)
  * [46] Whang, J., Delbracio, M., Talebi, H., Saharia, C., Dimakis, A.G., Milanfar, P.: Deblurring via stochastic refinement. In: CVPR (2022)
  * [47] Zhang, W., Liu, Y., Dong, C., Qiao, Y.: Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In: ICCV (2019)
  * [48] Zhao, K., Hung, A.L.Y., Pang, K., Zheng, H., Sung, K.: Partdiff: Image super-resolution with partial diffusion models. arXiv:2307.11926 (2023)

## Appendix

### Complexity of YODA

We discuss the resource implications of the core components of YODA:
Identifying key regions, time-dependent masking, and the guided diffusion
process. Additionally, we explore potential avenues for future enhancements
aimed at optimizing computational efficiency.

Identifying Key Regions. To avoid the computational burden of on-the-fly
generation, we pre-compute the attention maps prior to training. Table 4 shows
the parameter count and throughput of different DINO backbones.

Time-Dependent Masking and Guided Diffusion Process. The integration of
attention masks within the diffusion framework introduces minimal
computational overhead, thanks to the inherently parallelizable nature of
element-wise multiplication and addition, as demonstrated in Equation 9 and
Equation 12. Consequently, the predominant factor influencing the overall
computational complexity remains the choice of diffusion model, whether it be
SR3, SRDiff, or another variant.

Potential Future Improvements. YODA notably decreases computational
requirements by enabling the use of smaller batch sizes during training, which
in turn reduces VRAM usage without compromising performance. Looking ahead,
YODA paves the way for leveraging sparse diffusion techniques. Such approaches
promise further computational savings by focusing computation efforts on
selectively identified regions (through YODA), thereby streamlining the
diffusion process. Currently, in PyTorch, applying masks to regions within a
matrix does not result in computational savings.

Table 4:  Details of different DINO backbones, values directly extracted from the original work [6]. Throughput was measured with a NVIDIA V100 GPU.  Model | Parameters | Throughput
---|---|---
| [M] | [img/s]
ResNet-50 | 23M | 1237
ViT-S/8 | 21M | 180

This examination of YODA’s complexity highlights its efficiency and the
strategic decisions made to balance performance with computational demands.
The potential for future improvements underscores YODA’s adaptability and its
capacity for integration with emerging sparse diffusion methodologies.

### Pipeline of YODA with DINO

Figure 7:  An Overview of integrating YODA and DINO within SR diffusion
models. Our process begins with using DINO to extract multiple attention maps.
These maps are then combined to form a singular comprehensive attention map,
denoted as $\mathbf{A}$. Subsequently, leveraging $\mathbf{A}$, YODA defines a
unique diffusion schedule through time-dependent masks $\mathbf{M}(t)$ for
every spatial location, as detailed within our method section.

### More Visual Results on Face SR ($16\times 16\rightarrow 128\times 128$)

LR

SR3

SR3+YODA

HR

Figure 8:  A comparison of LR, HR, SR3, and SR3+YODA images for $16\rightarrow
128$.

### More Visual Results on DIV2K Validation

Original

LR

SRDiff
LPIPS: 0.2746

SRDiff+YODA
LPIPS: 0.1683

HR

LPIPS: 0.2579

LPIPS: 0.2276

LPIPS: 0.3701

LPIPS: 0.2681

LPIPS: 0.2768

LPIPS: 0.2281

LPIPS: 0.4244

LPIPS: 0.2999

LPIPS: 0.4663

LPIPS: 0.3433

Figure 9:  Zoomed-in comparison of LR, HR, SRDiff, and SRDiff+YODA on DIV2K.

### Intermediate Results with YODA

Figure 10:  Intermediate results of the YODA’s guided diffusion process on
CelebA-HQ.

### Other State-Of-The-Art Diffusion Models

Table 5:  $\times 4$ upscaling results on ImageNet-Val. ($256\times 256$). Values directly derived from the original work of LDM [37]. Method | IS $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---
Image Regression [38] | 121.1 | 27.9 | 0.801
SR3 [38] | 180.1 | 26.4 | 0.762
LDM-4 (100 steps) [37] | 166.3 | 24.4$\pm$3.8 | 0.69$\pm$0.14
LDM-4 (big, 100 steps) [37] | 174.9 | 24.7$\pm$4.1 | 0.71$\pm$0.15
LDM-4 (50 steps, guiding) [37] | 153.7 | 25.8$\pm$3.7 | 0.74$\pm$0.12

As shown in the study of Moser et al. [35], many approaches apply to image SR.
In this section, we want to discuss their potential in combination with YODA
and possible limitations for future work.

Latent Diffusion Models. Despite the significant advancements brought by
Latent Diffusion Models (LDMs) [37], their efficacy in the realm of image SR
competes closely with that of SR3 [38], as shown in Table 5. Unfortunately,
recent research in this direction focused primarily on text-to-image tasks
[17], which makes further comparisons with image SR methods challenging, e.g.,
SDXL [36], MultiDiffusion [5], or DemoFusion [15]. Nevertheless, their
potential for image SR is undeniable. Concerning YODA, we also see great
research avenues in combination with LDMs. A critical prerequisite for this
synergy is the conversion of attention maps from pixel to latent
representations. This aspect has to be investigated in more detail in future
work.

Unsupervised Diffusion Models. Another interesting research avenue is
unsupervised diffusion models for image SR, exemplified by DDRM [24] or DDNM
[45]. Interestingly, they use a pre-trained diffusion model to solve any
linear inverse problem, including image SR, but they rely on singular value
decomposition (SVD). Similar to the challenges faced with LDMs, integrating
YODA into unsupervised diffusion models presents another interesting research
avenue. The core of this challenge lies in devising a method for effectively
translating attention maps into a format compatible with the SVD process used
by these models. This transformation is crucial for harnessing the power of
attention-based enhancements in unsupervised diffusion frameworks for image
SR. Future work needs to conceptualize and implement a seamless integration
strategy that combines the dynamic attention modulation offered by YODA with
SVD.

Alternative Corruption Spaces. Applying YODA with alternative corruption
spaces (not pure Gaussian noise), such as used in InDI [12], I2SB [28], CCDF
[10], or ColdDiffusion [4] is also an interesting future research direction.
Although our primary focus has been refining and enhancing specific models’
diffusion process through attention-guided masks, we acknowledge the
orthogonal potential these approaches represent within the broader context of
SR.

### More Details on DINO

DINO [6] is a self-supervised learning approach. involving a teacher and
student network. While both networks share the same architecture, their
parameters differ. The student network is optimized to match the teacher’s
output via cross-entropy loss. During training, both receive two random views
of the same input image: the teacher is trained on global views, i.e.,
$224\times 224$ crops, while the student receives local views, i.e., $96\times
96$ crops. This setup encourages the student to learn “local-to-global”
correspondences. In other words, by predicting the teacher’s output, the
student learns to infer global information from local views. To prevent mode
collapse due to identical architectures, the teacher’s parameters are updated
as a moving average of the student’s parameters.

### More DINO Attention Maps for YODA

Input

Att.-Head 0

Att.-Head 1

Att.-Head 2

Att.-Head 3

Att.-Head 4

Att.-Head 5

MAX

Figure 11:  Comparison of attention maps derived from different DINO layers.