# Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction

Yingzi Fan  Xidian UniversityXi’anChina<EMAIL_ADDRESS>,  Longfei Han∗
Beijing Technology and Business UniversityBeijingChina<EMAIL_ADDRESS>,
Yue Zhang  Xi’an Jiaotong UniversityXi’anChina<EMAIL_ADDRESS>,  Lechao
Cheng  Zhejiang LabHangzhouChina<EMAIL_ADDRESS>,  Chen Xia
Northwestern Polytechnical UniversityXi’anChina<EMAIL_ADDRESS>and  Di Hu
Renmin University of ChinaBeijingChina<EMAIL_ADDRESS>
(2022)

###### Abstract.

Both visual and auditory information are valuable to determine the salient
regions in videos. Deep convolution neural networks (CNN) showcase strong
capacity in coping with the audio-visual saliency prediction task. Due to
various factors such as shooting scenes and weather, there often exists
moderate distribution discrepancy between source training data and target
testing data. The domain discrepancy induces to performance degradation on
target testing data for CNN models. This paper makes an early attempt to
tackle the unsupervised domain adaptation problem for audio-visual saliency
prediction. We propose a dual domain-adversarial learning algorithm to
mitigate the domain discrepancy between source and target data. First, a
specific domain discrimination branch is built up for aligning the auditory
feature distributions. Then, those auditory features are fused into the visual
features through a cross-modal self-attention module. The other domain
discrimination branch is devised to reduce the domain discrepancy of visual
features and audio-visual correlations implied by the fused audio-visual
features. Experiments on public benchmarks demonstrate that our method can
relieve the performance degradation caused by domain discrepancy.

Audio-visual saliency prediction, Cross-modal self-attention, Unsupervised
domain adaptation.

∗ Corresponding author

††copyright: acmcopyright††journalyear: 2022††copyright:
acmlicensed††conference: Proceedings of the 3rd International Workshop on
Human-Centric Multimedia Analysis; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 3rd International Workshop on Human-
Centric Multimedia Analysis (HCMA ’22), October 10–14, 2022, Lisboa,
Portugal††price: 15.00††doi: 10.1145/3552458.3556447††isbn:
978-1-4503-9492-5/22/10

##  1\. Introduction

Visual saliency prediction (VSP) aims to extract important regions from input
images, which is inspired by the selective attention mechanism of humans. It
has a very wide range of application in computer vision tasks, such as video
summarization (marat, ), stream compression (2013saliency, ), video monitoring
(2010predictive, ), etc. In real world, visual signals are usually accompanied
by audio signals, which can also provide valuable clues for locating salient
regions. Recently, the audio-visual saliency prediction (AVSP) task has
attracted extensive research interest. Benefiting from their strong
representation capacity, deep convolutional neural networks (CNN) bring huge
development to this field (tavakoli2019dave, ; tsiami2020stavis, ). However,
CNN models trained with source data usually do not perform well on target data
due to the data distribution shift. This paper focuses on studying the
unsupervised domain adaptation problem for AVSP.

Existing AVSP methods (tavakoli2019dave, ; tsiami2020stavis, ; yuan2021bio, )
mainly concentrate on modeling the feature representations of auditory and
visual inputs and the interactions between the two kinds of inputs. They all
employ a two-branch framework to encode the audio and video inputs. A 3D
convolution branch is leveraged to extract visual features from video inputs.
(tavakoli2019dave, ) utilizes a 3D convolution branch to extract auditory
feature representations from the log mel-spectrogram frames of the audio
inputs, while (tsiami2020stavis, ; yuan2021bio, ; jain2020avinet, ) implement
the audio representation modeling with SoundNet (aytar2016soundnet, ). Fusing
the visual and auditory features is the other critical point in AVSP.
(tsiami2020stavis, ) simply uses the concatenation operation to fuse the two
kinds of features. (yuan2021bio, ) devises attention mechanisms to explore the
spatial and temporal dependencies between the two features. These methods
achieve excellent performance under the circumstance that testing data obeys
the same distribution with training data. However, the unsupervised adaptation
of them to unseen testing data is still under research.

Figure 1. Examples for visualizing the variance of audio and video data.

The exploration of visual contrast information and audio-visual interaction is
critical to settle the AVSP task. However, these factors can be easily
influenced by the shooting conditions. Objects having relatively higher
contrast to background stuffs are more prone to attract attention of humans.
The identities of such objects are flexible and vary as the change of shooting
scenes. For example, in a speaking video, the mouth region usually draws the
highest attention, while the attractive regions are completely different in
other social activities such as sport and dance scenes. Visual data
differences induced by such factors can degrade the performance of AVSP
methods learned with stationary source data. It is likely to contain crucial
cues for recognizing salient regions that the audio information accompanying
video data. However, the intrinsic attributes of sound data such as pitch,
loudness, and timbre are diversified among videos. On the other hand, the
relevance between audio and salient regions is variant according to different
types of videos. For example, the sound resource locations are usually
consistent with salient regions in speaking videos as indicated by the ‘news’
example of Fig. 1; salient regions can not be thoroughly identified out
according to sounds in more general social videos like the ‘soccer’ example of
Fig. 1; in natural scene videos (‘scenery’ and ‘kids playing in leaves’ of
Fig. 1), the audio data is generated by multiple sound resources, and the
connection between audio and salient regions is sometimes ambiguous. Due to
the above factors, the performance of AVSP methods can be hampered by the
domain discrepancy caused by variances of the audio data and the interaction
between the audio and the video data.

To deal with the issue of domain discrepancy, we propose a dual domain-
adversarial learning framework on the basis of the unsupervised domain
adaptation paradigm (Fig. 2 (a)). Following existing methods
(tsiami2020stavis, ; yuan2021bio, ), we set up independent branches for
extracting features from audio and video inputs respectively. First, an
auditory feature domain discriminator is set up to eliminate the impacts of
audio data variance. It guides the optimization of the audio feature
extraction branch via adversarial training and helps to produce domain-
invariant audio features. Then, a cross-modal self-attention module as shown
in Fig. 2 (b) is devised to fuse auditory features into visual features
extracted by a 3D convolution backbone. The other domain discriminator is
adopted to learn fused audio-visual features having a uniform distribution
across source and target domains. This is helpful for mitigating the domain
discrepancy caused by the variance of visual features and the relation between
auditory and visual features. To synthesize the effect of domain discrepancy,
we set up two experimental settings: 1) videos captured under different types
of scenes are regarded as samples of source and target domains; 2) datasets
collected from different sites are regarded as source and target domains.
Extensive experiments on two settings demonstrate that our proposed method is
effective in making up the performance reduction caused by domain discrepancy.

In summary, the main contributions of this paper are as follows.

  * 1)

We make an early attempt to tackle the unsupervised domain adaptation problem
for audio-visual saliency prediction.

  * 2)

A dual domain adversarial learning framework is proposed for aligning the
audio-visual feature distributions between source and target domains.

  * 3)

We conduct extensive experiments on cross-scene and cross-dataset adaptation
settings, validating the efficacy of our proposed method.

Figure 2. A brief introduction to our proposed method.

##  2\. Related Work

###  2.1. Video Saliency

With the development of deep convolutional neural networks, a vast number of
saliency prediction methods are proposed, and they achieve excellent
performance on the image saliency prediction (huang2015salicon, ;
zhang2015exploiting, ; pan2017salgan, ; wang2017deep, ) and video saliency
prediction (bak2017spatio, ; jiang2018deepvs, ; wang2018revisiting, ;
min2019tased, ) task. In video saliency prediction, many recent methods
utilize an LSTM to extract spatial and temporal visual features. Wang et al.
(wang2018revisiting, ) propose a CNN-LSTM network architecture with an
attention mechanism. Linardos et al. (linardos2019simple, ) add a ConvLSTM to
an existing neural network and wrap a convolutional layer by using a temporal
exponential moving average(EMA) operation. Some existing models have explored
3D-CNN architecture to learn visual features. TASED-Net (min2019tased, ) is a
3D fully-convolutional network, including an encoding network to extract
spatiotemporal features and a prediction network to produce a single saliency
map. STSConvNet (bak2017spatio, ) extract the temporal and spatial information
by using a two-stream network architecture, then fusing them to get a saliency
map. Fang et al. (fang2022densely, ) proposed a densely nested top-down flows
(DNTDF)-based framework to predict pixel-wise salient object regions.
(zhuge2022salient, ) designed an ICON to integrate the features from the micro
level and macro level. In (cong2021rrnet, ), a relational reasoning network is
proposed for SOD in optical RSIs. (cong2018review, ) review some visual
saliency detection algorithms with comprehensive information. Liu et al.
(liu2020learning, ) incorporated multi-modal information to learn attention by
integrating self-attention and each other’s attention. (zhang2020uc, ) is the
first work to employ uncertainty for RGB-D saliency detection, which generates
multiple saliency maps by modeling human annotation uncertainty. A survey of
RGB-D salient object detection is provided by Zhou et al.(zhou2021rgb, ).

###  2.2. Audio-Video Saliency

Some early audio-visual saliency prediction methods have been explored for
application-specific(ruesch2008multimodal, ; schauerte2011multimodal, ;
chen2014audio, ; liao2015audeosynth, ). They employ traditional signal
processing techniques for visual saliency and audio localization. Coutrot et
al. (coutrot2014audiovisual, ) extract static and dynamic video features by
using Gabor filters, then weighing the face regions appropriately and
combining the visual saliency map to generate the final audio-visual saliency
map. Also, Min et al. (min2016fixation, ) propose an audio-visual model to
predict the salient region in the scenes containing moving and sound-
generating objects. They generate the saliency map by fusing the audio,
spatial, and temporal attention maps.

However, only a few works have explored the end-to-end network for audio-
visual saliency fixation prediction based on deep learning. Tavakoli et al.
(tavakoli2019dave, ) adopt two 3D-ResNet as the backbone of the two modalities
respectively, and their outputs are catenated for encoding. Tsiami et al.
(tsiami2020stavis, ) propose STAViS, which employs a multi-modal network that
combines visual and auditory information at multiple stages. Furthermore, they
investigate the fusion approaches such as computing the cosine similarity,
taking the weighted inner product, and bilinear transformation to fuse the
audio modality. Jain et al. (jain2020avinet, ) propose a hierarchical
structure for audio-video saliency prediction. They analyze three fusion
mechanisms, including simple concatenation, bilinear fusion, and transformer-
based fusion, to fuse video and audio cues. Zhu et al. (zhu2021lavs, ) propose
a lightweight audio-visual saliency model for the audio-visual saliency
fixation prediction task. Although these methods show excellent performance,
they do not involve the study of unsupervised domain adaptation when the
distribution of unlabeled testing data differs from training data.

###  2.3. Domain Adaptation

Unsupervised domain adaptation(UDA) is targeted at aligning the data
distribution of labeled source domain and unlabeled target domain. Thus, the
model trained in the source domain can be directly migrated to the target
domain without significant performance degradation. The core issue in UDA is
how to align the distributions of source and target data. A large number of
UDA algorithms have been proposed for classical computer vision tasks such as
image classification and semantic segmentation, based on image translation
(hoffman2018cycada, ; murez2018image, ), maximum mean discrepancy minimization
(wang2020rethink, ), or domain adversarial learning (ganin2015unsupervised, ;
liu2021adversarial, ). In this paper, we concentrate on the unsupervised
domain adaptation for audio-visual saliency prediction which has not been
studied in previous literature.

Figure 3. Overall architecture of the proposed model.

##  3\. Method

This paper aims to tackle the unsupervised domain adaptation for AVSP. Assume
the source domain data be
$\\{(\mathbf{A}^{s}_{i},\mathbf{V}^{s}_{i})\\}_{i=1}^{N_{s}}$, where
$\mathbf{A}^{s}_{i}$ and $\mathbf{V}^{s}_{i}$ denotes the video and audio of
the $i$-th sample respectively. $N_{s}$ represents the number of source
samples. The ground-truth saliency map for the $i$-th sample in the source
dataset is denoted as $\mathbf{G}^{s}_{i}$. For the target data, only a subset
of unlabeled samples
$\\{(\mathbf{A}^{t}_{j},\mathbf{V}^{t}_{j})\\}_{j=1}^{N_{t}}$. $N_{t}$
represents the number of target samples. The goal of this paper is to learn a
CNN model capable of achieving high AVSP performance on the target domain
data. A dual domain adversarial training pipeline is proposed for addressing
the above problem. Technical details are introduced in following subsections.

###  3.1. Architecture Overview

Fig. 3 illustrates the proposed architecture for cross-modal audio-visual
saliency prediction. The proposed architecture consists of a visual branch to
extract visual features, an audio branch to extract audio features, a cross-
modal self-attention module to obtain correlated meaningful audio-visual
features for saliency prediction, and a decoder module to decode the audio-
visual features and get the final saliency map. Domain adaptation modules are
added to make the model learn generalizable features. We input consecutive
video frames $\mathbf{V}$ and the corresponding sampled audio sequences
$\mathbf{A}$ into the visual and audio branches, respectively, and the final
output is a saliency prediction map $\mathbf{P}$. Each domain classifier
outputs a binary result to indicate whether the input video comes from the
target domain or the source domain.

For visual branch, we use S3D (xie2018rethinking, ) as the backbone to extract
visual features. The size of input video frames $\mathbf{V}$ is $C_{0}\times
F_{0}\times H_{0}\times W_{0}$, where $F_{0}=32$ is the number of frames and
$C_{0}=3$ is the number of frame channels. $H_{0}$ and $W_{0}$ represents the
height and width of each frame, respectively. The output of the visual branch
is $\mathbf{O}_{\rm V}$. The size of $\mathbf{O}_{\rm V}$ is $C\times F\times
H\times W$, where $C=1024$, $F=\frac{F_{0}}{8}$, $H=\frac{H_{0}}{32}$, and
$W=\frac{W_{0}}{32}$.

For the audio branch, we adopt SoundNet (aytar2016soundnet, ) to encode the
representation of the audio sequence, which is a 1-D fully convolutional
network and is initially proposed for audio classification. The size of pre-
processed audio sequence $\mathbf{A}$ is $1\times T_{a}\times 1$ , and the
size of the output features $\mathbf{O}_{\rm A}$ is $1024\times 3\times 1$.

For audio-visual fusion, inspired by (tsai2019multimodal, ), we use the cross-
modal self-attention mechanism to fuse two modalities. $\mathbf{O}_{\rm A}$
and $\mathbf{O}_{\rm V}$ through dimensional changes are input to the cross-
modal self-attention fusion module. For the details, see Section 3.2. Then the
fusion information goes through a fully-connected layer and is added with
$\mathbf{X}_{\rm V}$ to obtain the salient audio-visual features
$\mathbf{O}_{\rm AV}$.

Afterwards, $\mathbf{O}_{\rm AV}$ are fed into a decoder consisting of six
decoding layers. The first decoding layer contains a 3D convolution and a
trilinear interpolation layer. Each of the next three decoding layers contains
a concatenation, a 3D convolutional, and a trilinear upsampling layer. The
concatenation operation combines the output of the visual block and the output
of the last decoder layer along the temporal dimension. The fourth decoding
layer is similar to the first layer. The last decoding layer is composed of
two 3D convolutional layers, which project the channel and temporal dimension
to 1. Finally, the single map goes through a sigmoid activation function to
obtain a saliency map $\mathbf{P}$.

Domain adaptation modules are applied to make the knowledge acquired in the
labeled source domain to be better generalized in the unlabeled target domain.
Specifically, the branch consisting of a gradient reversal layer and a domain
classifier is added after the audio output and the output of the fusion
module, respectively.

In the following, we mainly describe the cross-modal self-attention mechanism
of the fusion module(see Section 3.2) and domain adaptation(see Section 3.3).

###  3.2. Cross-modal Self-attention

For the fusion of two modalities, previous methods such as computing the
cosine similarity between two modalities, simple concatenation, bilinear
fusion and so on, can not fully explore the correlation between two
modalities. In order to obtain meaningful audio-visual correlation
information, we adopt a cross-modal self-attention mechanism, which enables
visual modality to receive information from audio modality selectively. The
source audio modality is used to reconstruct the information of visual
modality and the cross-modal interaction between vision and audio is realized.
Details about the cross-modal self-attention are as follows.

The output features of the visual branch $\mathbf{O}_{\rm V}$ are first
processed with a max pooling layer and are flattened to $\mathbf{x}_{\rm
V}\in{\mathbf{R}^{C\times(HW)}}$. Then the $\mathbf{x}_{\rm V}$ passes through
a 1-D convolution and then are collapsed as a vector $\mathbf{X}_{\rm
V}\in{\mathbf{R}^{T_{V}\times d_{V}}}$, and the audio features
$\mathbf{O}_{\rm A}$ are passed through a 1-D convolution and then are
collapsed as a vector $\mathbf{X}_{\rm A}\in{\mathbf{R}^{T_{A}\times d_{A}}}$.
Where $T_{V}=H\times W=42$, $T_{A}=3$, and $d_{V}=d_{A}=512$. $T_{(\cdot)}$ is
the length of the sequence and $d_{(\cdot)}$ is the feature dimension. The
cross-modal attention from $\mathbf{X}_{\rm A}$ to $\mathbf{X}_{\rm V}$ is
defined as $\mathbf{Y}_{\rm{A\rightarrow V}}$. The calculation process is as
below.

(1)
$\mathbf{Y}_{\rm{A}\rightarrow\rm{V}}=softmax(\frac{\mathbf{Q}_{\rm{V}}\mathbf{K}_{\rm{A}}^{\rm{T}}}{\sqrt{d_{k}}})\mathbf{V}_{\rm{A}}$

where $\mathbf{Q}_{\rm V}={\mathbf{X}_{\rm
V}}{\mathbf{W}_{\rm{Q_{V}}}},\mathbf{\mathbf{K}_{\rm A}={\mathbf{X}_{\rm
A}}{\mathbf{W}_{\rm{K_{A}}}}}$ and $\mathbf{V}_{\rm A}={\mathbf{X}_{\rm
A}}{\mathbf{W}_{\rm{V_{A}}}}$ represent query, key and value in the cross-
modal self-attention module.
$\mathbf{W}_{\rm{Q_{V}}}\in{\mathbf{R}^{{d_{V}}\times{d_{k}}}},\mathbf{W}_{\rm{K_{A}}}\in{\mathbf{R}^{{d_{A}}\times{d_{k}}}}$
and $\mathbf{W}_{\rm{V_{A}}}\in{\mathbf{R}^{{d_{A}}\times{d_{v}}}}$ are
weights.

In this way, we get the correlation matrix between the audio sequence and the
corresponding video, which plays a great role in determining the saliency
region of the video. Then, $\mathbf{Y}_{\rm{A\rightarrow V}}$ is added back
into $\mathbf{X}_{\rm V}$, and a residual multi-layer perceptron is attached
for further feature enhancement. The complete calculation process is
illustrated in Fig. 2 (b).

###  3.3. Domain Adaptation

Our unsupervised domain adaptation module is constituted by a gradient
inversion layer and a domain classifier. The gradient inversion layer is
located between the feature extraction module and the domain classifier.
During the error backpropagation process, this layer reverses the gradient
direction, namely making the feature extraction module confuse the domain
classifier. Such a manner avoids the two-stage training process of adversarial
training. For the domain classifier, three $1\times 1$ spatial convolution
layers are adopted to compress the feature dimension. Afterwards, three fully-
connected layers are attached, producing the domain classification score.

We setup two separate domain classifiers for audio features and fused audio-
visual features respectively. The domain classifier of audio features is
denoted as $D_{\rm{A}}$, and that of audio-visual features is denoted as
$D_{\rm{AV}}$. Suppose the audio feature and the audio-visual feature
extracted from a source domain sample ($\mathbf{A}^{s}$, $\mathbf{V}^{s}$) be
$\mathbf{O}^{s}_{\rm{A}}$ and $\mathbf{O}^{s}_{\rm{AV}}$, respectively.
$\mathbf{O}^{t}_{\rm{A}}$ and $\mathbf{O}^{t}_{\rm{AV}}$ denotes the audio
feature and the audio-visual feature of a target domain sample
($\mathbf{A}^{t}$, $\mathbf{V}^{t}$), respectively. $\mathbf{O}^{s}_{\rm{A}}$
and $\mathbf{O}^{t}_{\rm{A}}$ are fed into the audio domain classifier
$D_{\rm{A}}$, resulting in prediction scores $d^{s}_{\rm{A}}$ and
$d^{t}_{\rm{A}}$, respectively. On the other hand, $D_{\rm{AV}}$ predicts
domain classification scores $d^{s}_{\rm{AV}}$ and $d^{t}_{\rm{AV}}$ from
$\mathbf{O}^{s}_{\rm{AV}}$ and $\mathbf{O}^{t}_{\rm{AV}}$, respectively. The
domain classification loss for $D_{\rm{A}}$ is defined as $L_{\rm{A}}$ and the
loss for $D_{\rm{AV}}$ is $L_{\rm{AV}}$ :

(2) $\displaystyle L_{\rm{A}}(d^{s}_{\rm{A}},d^{t}_{\rm{A}})$
$\displaystyle=-\log(d^{s}_{\rm{A}})-\log(1-d^{t}_{\rm{A}}),$ (3)
$\displaystyle L_{\rm{AV}}(d^{s}_{\rm{AV}},d^{t}_{\rm{AV}})$
$\displaystyle=-\log(d^{s}_{\rm{AV}})-\log(1-d^{t}_{\rm{AV}}).$

###  3.4. Loss Function

The source domain images are employed for training the saliency prediction
model with the Kullback-Leibler (KL) divergence. The training loss $L_{s}$ on
source domain images is as below:

(4)
$L_{s}(\mathbf{P}^{s},\mathbf{G}^{s})=\sum_{i=1}^{|\mathbf{P}^{s}|}\mathbf{G}^{s}(i)\log(\epsilon+\frac{\mathbf{G}^{s}(i)}{\mathbf{P}^{s}(i)+\epsilon}),$

where $\mathbf{P}^{s}$ is the saliency map predicted from $\mathbf{A}^{s}$ and
$\mathbf{V}^{s}$, $\mathbf{G}$ is the corresponding ground-truth map, and
$\epsilon$ is a regularization parameter. $\mathbf{G}^{s}(i)$ and
$\mathbf{P}^{s}(i)$ represents the $i$-th pixel of $\mathbf{G}^{s}$ and
$\mathbf{P}^{s}$, respectively. $|\mathbf{P}^{s}|$ denotes the number of
pixels in $\mathbf{P}^{s}$. The final loss function is formed by summing up
(2), (3), and (4):

(5) $L=L_{s}+L_{\rm{A}}+L_{\rm{AV}}.$

##  4\. Experiments

In this section, we validate the effectiveness of our method on multiple
benchmarks. For evaluating the performance of unsupervised domain adaptation,
we synthesize different data domains according to shooting scenarios or
dataset sources.

###  4.1. Datasets and Evaluation Metrics

We carry out the experiments on six audio-visual eye-tracking datasets: DIEM
(mital2011clustering, ), AVAD (min2016fixation, ), Coutrot1
(coutrot2014saliency, ), Coutrot2 (coutrot2016multimodal, ), SumMe
(gygli2014creating, ), and ETMD (koutras2015perceptually, ). DIEM contains 84
video clips, including commercials, documentaries, game trailers, movie
trailers, etc. Coutrot1 contains 60 dynamic nature scene clips such as moving
objects, landscapes and faces. Coutrot2 contains 15 clips of 4 people in the
meeting. AVAD contains 45 short clips with audio-visual scenes such as
dancing, guitar playing, birds singing, etc. ETMD contains 12 videos from six
different Hollywood films. SumMe contains 25 unstructured videos. These eye-
tracking data are all recorded via the Eyelink eye-tracker from viewers.

Table 1. Number of videos in each dataset. Dataset | Train | Valid./Test
---|---|---
Faces | 30 | 14
Nature | 18 | 10
Social | 58 | 15

For the evaluation of the domain adaptation, we divide these datasets into two
settings based on their shooting scenes. One of the settings is ‘DIEM, AVAD
and Coutrot1’, which consists of parts of AVAD, Coutrot1 and DIEM datasets.
The three datasets contain multiple scenarios. The other is ‘Faces, Nature and
Social’. The three datasets are constructed by selecting data from the
existing six datasets according to the content of video frames. ‘Faces’
contains 30 clips with faces, and the corresponding audio is human voice.
‘Nature’ contains 18 clips with landscapes and scenery. ‘Social’ contains 58
clips, including dancing, guitar playing, playing football, etc. The number of
videos in each dataset is summarized in Table 1.

Figure 4. Sample frames from Coutrot1 dataset, and the corresponding ground
truth, and saliency maps without adding DA, and adding DA after the audio
branch and fusion module. Table 2. Performance of our proposed method under
various domain adaptation settings. Target data: Nature
---
Source data | Approach | CC | NSS | sAUC | AUC-J | SIM
Faces | w/o DA | 0.4099 | 1.670 | 0.5981 | 0.8442 | 0.3587
DA(audio) | 0.4352 | 1.785 | 0.6021 | 0.8591 | 0.3574
DA(audio + fusion) | 0.4488 | 1.843 | 0.5988 | 0.8625 | 0.3608
Social | w/o DA | 0.477 | 1.9711 | 0.6019 | 0.8717 | 0.3742
DA(audio) | 0.4951 | 2.064 | 0.6016 | 0.8721 | 0.3964
DA(audio + fusion) | 0.4985 | 2.084 | 0.6083 | 0.8727 | 0.3833
Source data: AVAD
Target data | Approach | CC | NSS | sAUC | AUC-J | SIM
Coutrot1 | w/o DA | 0.4685 | 2.274 | 0.5881 | 0.8673 | 0.3962
DA(audio) | 0.4749 | 2.298 | 0.5926 | 0.8716 | 0.3962
DA(audio + fusion) | 0.4891 | 2.364 | 0.5927 | 0.8723 | 0.4057
DIEM | w/o DA | 0.5640 | 2.281 | 0.6744 | 0.8857 | 0.4710
DA(audio) | 0.5692 | 2.300 | 0.6782 | 0.8898 | 0.4712
DA(audio + fusion) | 0.5758 | 2.315 | 0.6797 | 0.8911 | 0.4714

We use five evaluation metrics (bylinskii2018different, ), including Linear
Correlation Coefficient (CC), AUC-Judd (AUC-J), Similarity Metric (SIM),
shuffled AUC (s-AUC), and Normalized Scanpath Saliency (NSS) to evaluate the
performance of our model.

###  4.2. Experimental Setup

In our experiment, for the visual branch, we use the weights pre-trained on
DHF1K (wang2018revisiting, ) to initialize the model. For the audio feature
extraction network, we initialize the weights from the SoundNet pre-trained in
the sound localization task (aytar2016soundnet, ). The learning rate is set to
be $10^{-4}$, and the batch size is 8 when training the model without domain
adaptation. When adding the domain adaptation, the learning rate is set to be
$10^{-5}$, and the batch size is set to 6. In all experiments, we adopt the
Adam optimizer to update network parameters.

###  4.3. Results

Figure 5. Sample frames from Coutrot2 and Coutrot1 datasets, and the corresponding ground truth, and saliency maps generated by our cross-model self-attention fusion and bilinear fusion for comparisons. Table 3.  Experimental results of using different auditory and visual feature fusion methods. Evaluation metrics are calculated by averaging results on ETMD, SumMe, Coutrot1, Coutrot2, DIEM and AVAD. Methods | CC | NSS | sAUC | AUC-J | SIM
---|---|---|---|---|---
C | 0.6167 | 3.367 | 0.6993 | 0.9155 | 0.4422
B | 0.6090 | 3.412 | 0.6965 | 0.9152 | 0.4427
CM | 0.6138 | 3.428 | 0.6757 | 0.9191 | 0.4735
Table 4.  Comparision results. Evaluation metrics are calculated by averaging results on ETMD, SumMe, Coutrot1, Coutrot2, DIEM and AVAD. | CC | NSS | sAUC | AUC-J | SIM
---|---|---|---|---|---
STAViS | 0.5644 | 2.9683 | 0.6584 | 0.854 | 0.4345
AViNet | 0.6118 | 3.3683 | 0.6998 | 0.9178 | 0.4445
Ours | 0.6138 | 3.428 | 0.6757 | 0.9191 | 0.4735

We conduct domain adaptation experiments on several different combinations of
datasets. Specifically, we take Faces/Social datasets as the source domain,
and Nature dataset as the target domain, forming domain adaptation settings:
Faces$\rightarrow$Nature and Social$\rightarrow$Nature. The result of not
adding the domain adaptation(DA) is tested, denoted by w/o DA, which means the
model is trained on the source domain and is tested directly on the target
domain. DA(audio) and DA(audio + fusion) represent adding DA after the audio
branch and adding DA after the audio branch and fusion module. The results are
shown in Table2. Among DIEM, Coutrot1 and AVAD datasets, we take AVAD dataset
as the source domain, and DIEM/Coutrot1 datasets as the target domain, forming
another two domain adaptation settings: AVAD$\rightarrow$DIEM and
AVAD$\rightarrow$Coutrot1. The experimental results are shown in Table 2.

The experimental results show that the performance of the model on the
unlabeled target domain datasets is improved by adding DA module. This
indicates that the model with DA has learned the domain-invariant
characteristics, and the discrepancy between source and target domain is
mitigated. We also visualize the results in Fig.4, where the model with DA
module performs better.

###  4.4. Ablation Study

Efficacy of Dual Domain-Adversarial Learning. We analyze the improvement of
the model’s performance by adding DA (domain adaptation) in different parts of
the model so as to determine the most appropriate position for adding DA. We
perform ablation experiments in the following two cases: (1) adding DA after
the audio branch (DA(audio)); (2) adding DA after the audio branch and fusion
module (DA(audio + fusion)). Table 2 shows the test results of the model in
the above two cases. The results show that adding DA after the audio branch
and fusion module performs better.

Efficacy of Cross-Modal Self-Attention. We verify the effectiveness of our
proposed fusion strategy on six audio-visual datasets by comparing it with
existing fusion methods. One is concatenating visual and auditory features,
and the other is bilinear fusion. Table 3 shows the averaged evaluation
metrics. ‘C’ represents simple concatenation, ‘B’ represents bilinear fusion,
and ‘CM’ represents our proposed fusion strategy based on cross-modal self-
attention mechanism. It can be seen from Table 3 that the proposed fusion
method is superior to other methods on NSS, AUC-J, and SIM, demonstrating the
effectiveness of our proposed fusion method. Furthermore, we compare the
fusion methods qualitatively. In Fig.5, the first row depicts the sample
frames from Coutrot2 and Coutrot1. The ground-truth saliency maps are
presented in the second row. The final two rows include the saliency maps
generated by bilinear fusion and our cross-model self-attention fusion. As
shown in the Figure 5, our results are closer to the ground truth. What’s
more, to prove our model’s efficacy without DA for AVSP task, we compare our
model with other AVSP methods. Table 4 shows the averaged evaluation metrics
on six audio-visual datasets. As is shown, our method obtains better
performance on several metrics.

##  5\. Conclusion

In this work, we propose a dual domain-adversarial learning algorithm to solve
the problem of unsupervised domain adaptation in audio-visual saliency
detection. Specifically, we establish two domain discrimination branches to
align the distribution of auditory features and to reduce the domain
differences in visual features and audio-visual correlation. Experimental
results show that our method can relieve the performance degradation caused by
domain discrepancy on the audio-visual saliency prediction task.

###### Acknowledgements.

This work is supported in part by the National Major Scientific Instruments
and Equipments Development Project of National Natural Science Foundation of
China No. 62027813 and in part by the Key Program of the National Natural
Science Foundation of China under Grant No. 62036005. And it is partially
supported by the National Natural Science Foundation of China (Grant No.
62106235), by the Exploratory Research Project of Zhejiang Lab(2022PG0AN01),
by the Zhejiang Provincial Natural Science Foundation of China (LQ21F020003).

## References

  * [1] Sophie Marat, Mickäel Guironnet, and Denis Pellerin.  Video summarization using a visual attention model.  In 2007 15th European Signal Processing Conference, pages 1784–1788. IEEE, 2007.
  * [2] Hadi Hadizadeh and Ivan V Bajić.  Saliency-aware video compression.  IEEE Transactions on Image Processing, 23(1):19–33, 2013.
  * [3] Fahad Fazal Elahi Guraya, Faouzi Alaya Cheikh, Alain Tremeau, Yubing Tong, and Hubert Konik.  Predictive saliency maps for surveillance videos.  In 2010 Ninth International Symposium on Distributed Computing and Applications to Business, Engineering and Science, pages 508–513. IEEE, 2010\.
  * [4] Hamed R Tavakoli, Ali Borji, Esa Rahtu, and Juho Kannala.  Dave: A deep audio-visual embedding for dynamic saliency prediction.  arXiv preprint arXiv:1905.10693, 2019.
  * [5] Antigoni Tsiami, Petros Koutras, and Petros Maragos.  Stavis: Spatio-temporal audiovisual saliency network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4766–4776, 2020.
  * [6] Yuan Yuan, Hailong Ning, and Bin Zhao.  Bio-inspired audio-visual cues integration for visual attention prediction.  arXiv preprint arXiv:2109.08371, 2021.
  * [7] Samyak Jain, Pradeep Yarlagadda, Ramanathan Subramanian, and Vineet Gandhi.  Avinet: Diving deep into audio-visual saliency prediction.  arXiv e-prints, pages arXiv–2012, 2020.
  * [8] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  Advances in neural information processing systems, 29, 2016.
  * [9] Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao.  Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks.  In Proceedings of the IEEE international conference on computer vision, pages 262–270, 2015.
  * [10] Jianming Zhang and Stan Sclaroff.  Exploiting surroundedness for saliency detection: a boolean map approach.  IEEE transactions on pattern analysis and machine intelligence, 38(5):889–902, 2015.
  * [11] Junting Pan, Cristian Canton Ferrer, Kevin McGuinness, Noel E O’Connor, Jordi Torres, Elisa Sayrol, and Xavier Giro-i Nieto.  Salgan: Visual saliency prediction with generative adversarial networks.  arXiv preprint arXiv:1701.01081, 2017.
  * [12] Wenguan Wang and Jianbing Shen.  Deep visual attention prediction.  IEEE Transactions on Image Processing, 27(5):2368–2378, 2017.
  * [13] Cagdas Bak, Aysun Kocak, Erkut Erdem, and Aykut Erdem.  Spatio-temporal saliency networks for dynamic saliency prediction.  IEEE Transactions on Multimedia, 20(7):1688–1698, 2017.
  * [14] Lai Jiang, Mai Xu, Tie Liu, Minglang Qiao, and Zulin Wang.  Deepvs: A deep learning based video saliency prediction approach.  In Proceedings of the european conference on computer vision (eccv), pages 602–617, 2018.
  * [15] Wenguan Wang, Jianbing Shen, Fang Guo, Ming-Ming Cheng, and Ali Borji.  Revisiting video saliency: A large-scale benchmark and a new model.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4894–4903, 2018.
  * [16] Kyle Min and Jason J Corso.  Tased-net: Temporally-aggregating spatial encoder-decoder network for video saliency detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2394–2403, 2019.
  * [17] Panagiotis Linardos, Eva Mohedano, Juan Jose Nieto, Noel E O’Connor, Xavier Giro-i Nieto, and Kevin McGuinness.  Simple vs complex temporal recurrences for video saliency prediction.  arXiv preprint arXiv:1907.01869, 2019.
  * [18] Chaowei Fang, Haibin Tian, Dingwen Zhang, Qiang Zhang, Jungong Han, and Junwei Han.  Densely nested top-down flows for salient object detection.  Science China Information Sciences, 65(8):1–14, 2022.
  * [19] Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, and Ling Shao.  Salient object detection via integrity learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [20] Runmin Cong, Yumo Zhang, Leyuan Fang, Jun Li, Yao Zhao, and Sam Kwong.  Rrnet: Relational reasoning network with parallel multiscale attention for salient object detection in optical remote sensing images.  IEEE Transactions on Geoscience and Remote Sensing, 60:1–11, 2021\.
  * [21] Runmin Cong, Jianjun Lei, Huazhu Fu, Ming-Ming Cheng, Weisi Lin, and Qingming Huang.  Review of visual saliency detection with comprehensive information.  IEEE Transactions on circuits and Systems for Video Technology, 29(10):2941–2959, 2018.
  * [22] Nian Liu, Ni Zhang, and Junwei Han.  Learning selective self-mutual attention for rgb-d saliency detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13756–13765, 2020.
  * [23] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saeed Anwar, Fatemeh Sadat Saleh, Tong Zhang, and Nick Barnes.  Uc-net: Uncertainty inspired rgb-d saliency detection via conditional variational autoencoders.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8582–8591, 2020.
  * [24] Tao Zhou, Deng-Ping Fan, Ming-Ming Cheng, Jianbing Shen, and Ling Shao.  Rgb-d salient object detection: A survey.  Computational Visual Media, 7(1):37–69, 2021.
  * [25] Jonas Ruesch, Manuel Lopes, Alexandre Bernardino, Jonas Hornstein, José Santos-Victor, and Rolf Pfeifer.  Multimodal saliency-based bottom-up attention a framework for the humanoid robot icub.  In 2008 IEEE International Conference on Robotics and Automation, pages 962–967. IEEE, 2008.
  * [26] Boris Schauerte, Benjamin Kühn, Kristian Kroschel, and Rainer Stiefelhagen.  Multimodal saliency-based attention for object-based scene analysis.  In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1173–1179. IEEE, 2011.
  * [27] Yanxiang Chen, Tam V Nguyen, Mohan Kankanhalli, Jun Yuan, Shuicheng Yan, and Meng Wang.  Audio matters in visual attention.  IEEE Transactions on Circuits and Systems for Video Technology, 24(11):1992–2003, 2014.
  * [28] Zicheng Liao, Yizhou Yu, Bingchen Gong, and Lechao Cheng.  Audeosynth: music-driven video montage.  ACM Transactions on Graphics (TOG), 34(4):1–10, 2015.
  * [29] Antoine Coutrot and Nathalie Guyader.  An audiovisual attention model for natural conversation scenes.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 1100–1104. IEEE, 2014.
  * [30] Xiongkuo Min, Guangtao Zhai, Ke Gu, and Xiaokang Yang.  Fixation prediction through multimodal analysis.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 13(1):1–23, 2016.
  * [31] Dandan Zhu, Defang Zhao, Xiongkuo Min, Tian Han, Qiangqiang Zhou, Shaobo Yu, Yongqing Chen, Guangtao Zhai, and Xiaokang Yang.  Lavs: A lightweight audio-visual saliency prediction model.  In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2021.
  * [32] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998. PMLR, 2018.
  * [33] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim.  Image to image translation for domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  * [34] Wei Wang, Haojie Li, Zhengming Ding, and Zhihui Wang.  Rethink maximum mean discrepancy for domain adaptation.  arXiv preprint arXiv:2007.00689, 2020.
  * [35] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189. PMLR, 2015.
  * [36] Xiaofeng Liu, Zhenhua Guo, Site Li, Fangxu Xing, Jane You, C-C Jay Kuo, Georges El Fakhri, and Jonghye Woo.  Adversarial unsupervised domain adaptation with conditional and label shift: Infer, align and iterate.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10367–10376, 2021.
  * [37] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In Proceedings of the European conference on computer vision (ECCV), pages 305–321, 2018.
  * [38] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
  * [39] Parag K Mital, Tim J Smith, Robin L Hill, and John M Henderson.  Clustering of gaze during dynamic scene viewing is predicted by motion.  Cognitive computation, 3(1):5–24, 2011.
  * [40] Antoine Coutrot and Nathalie Guyader.  How saliency, faces, and sound influence gaze in dynamic social scenes.  Journal of vision, 14(8):5–5, 2014.
  * [41] Antoine Coutrot and Nathalie Guyader.  Multimodal saliency models for videos.  In From Human Attention to Computational Attention, pages 291–304. Springer, 2016.
  * [42] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool.  Creating summaries from user videos.  In European conference on computer vision, pages 505–520. Springer, 2014.
  * [43] Petros Koutras and Petros Maragos.  A perceptually based spatio-temporal computational framework for visual saliency estimation.  Signal Processing: Image Communication, 38:15–31, 2015.
  * [44] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand.  What do different evaluation metrics tell us about saliency models?  IEEE transactions on pattern analysis and machine intelligence, 41(3):740–757, 2018.


# WaveFlow – Towards Integration of Ultrasound Processing with Deep Learning

Piotr Jarosik1, Michal Byra2, Marcin Lewandowski34  1Department of Information
and Computational Science,
2Department of Ultrasound,
3Laboratory of Professional Electronics,
Institute of Fundamental Technological Research PAS,
Warsaw, Poland;
4us4us Ltd., Warsaw, Poland;
email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The ultimate goal of this work is a real-time processing framework for
ultrasound image reconstruction augmented with machine learning. To attain
this, we have implemented WaveFlow – a set of ultrasound data acquisition and
processing tools for TensorFlow. WaveFlow includes: ultrasound Environments
(connection points between the input raw ultrasound data source and
TensorFlow) and signal processing Operators (ops) library. Raw data can be
processed in real-time using algorithms available both in TensorFlow and
WaveFlow. Currently, WaveFlow provides ops for B-mode image reconstruction
(beamforming), signal processing and quantitative ultrasound. The ops were
implemented both for the CPU and GPU, as well as for built-in automated tests
and benchmarks. To demonstrate WaveFlow’s performance, ultrasound data were
acquired from wire and cyst phantoms and elaborated using selected sequences
of the ops. We implemented and evaluated: Delay-and-Sum beamformer, synthetic
transmit aperture imaging (STAI), plane-wave imaging (PWI), envelope detection
algorithm and dynamic range clipping. The benchmarks were executed on the
NVidia® Titan X GPU integrated in the USPlatform research scanner (us4us Ltd.,
Poland). We achieved B-mode image reconstruction frame rates of 55 fps, 17 fps
for the STAI and the PWI algorithms, respectively. The results showed the
feasibility of real-time ultrasound image reconstruction using WaveFlow
operators in the TensorFlow framework. WaveFlow source code can be found at
github.com/waveflow-team/waveflow.

###### Index Terms:

beamforming, deep learning, machine learning, tensorflow

##  I Introduction

In recent years, we can observe a growing interest in the use of deep learning
methods and GPU processing in ultrasound imaging. Various machine learning
tasks have been considered for this modality, eg. classification, segmentation
(for CAD systems) and image enhancement [4, 5, 6]. What is more, the
implementation of ultrasound imaging pipeline directly on GPU becomes more
popular and realistic to perform, mostly due to increasing power of the
available technology [7, 9]. The Software Defined Ultrasound paradigm
naturally opens the possibility to apply machine learning tools at any stage
of the imaging pipeline, especially to process raw ultrasound radio-frequency
(RF) data. This idea is highly promising and becomes well supported by
hardware, however the appropriate software architecture still has to be
developed.

Ultrasound image reconstruction algorithms can be defined and implemented as a
sequence of signal processing operations like: filtering, Delay-and-Sum (DAS)
beamforming, envelope detection and dynamic range clipping. These operations
have a clear functional definition and constitute the data processing
pipeline. The target implementation of the pipeline should be easy to
rearrange depending on the needs, e.g. the software should provide a
convenient way to introduce new signal filtering operations. The idea of
ultrasound imaging pipeline has been already considered and evaluated in
previous publications [8, 9]. Each of this work considers own, different
implementation of the pipeline framework, with operations implemented
specifically for one type of the processing device.

In this work we present WaveFlow [1] – a set of ultrasound data acquisition
and processing tools for TensorFlow. TensorFlow is a framework, which allows
to define a dataflow graph, whose nodes represent operators (units of
computation, _ops_), and edges represent ops input/output data (tensors) [2].
Each op can have an implementation for multiple processing devices, like CPU
or GPU. This software gained the most popularity, thanks to a broad library of
available machine learning algorithms, however the framework can be used not
only for AI training and inference. In WaveFlow, we provide a collection of
high-level signal processing ops and tools, which can constitute imaging
pipeline. Those operators can be added to the graph, and process RF data to
reconstruct B-mode frames. Furthermore, WaveFlow and TensorFlow ops can be
placed in the same graph what provides a convenient way to augment ultrasound
image reconstruction with machine learning.

##  II Methodology

WaveFlow includes ultrasound Environments and signal processing Operators
(ops) library for TensorFlow.

###  II-A Ultrasound Environment

UltrasoundEnvTensorFlowAgentRF, metadatarewardaction Figure 1: Ultrasound
Environment provides observations (RF data and metadata), which can be
processed by graph defined in TensorFlow. Currently, Ultrasound Environment is
an _read-only environment_ specialized for ultrasound imaging.

The Ultrasound Environment is a connection point between the input raw RF data
source (i.e. dataset, simulation or hardware scanner) and TensorFlow
framework. It feeds the graph with:

  * •

raw RF data stored as a TensorFlow tensor,

  * •

metadata, e.g.:

    * –

examined physical environment parameters, like speed of sound,

    * –

ultrasound probe parameters, like aperture size and sampling frequency.

The name _Ultrasound Environment_ loosely refers to reinforcement learning
task environments, where some software agent takes some actions so as to
maximize some cumulative reward [10]. WaveFlow currently provides only
observations from the environment, so it is possible to reconstruct ultrasound
image on further steps (see figure 1). However, this idea can be extended in
future e.g. with actions to perform.

###  II-B Signal Processing Operators

The data provided by the Ultrasound Environment can be processed by operators
registered in TensorFlow framework. WaveFlow includes implementations of the
following ops:

  * •

pre-processing: e.g. FIR filter;

  * •

mid-processing: Delay-and-Sum (DAS) beamformer, including the Synthetic
Transmit Aperture Imaging (STAI) and the Plane Wave Imaging (PWI);

  * •

post-processing: e.g. envelope detection, dynamic adjustment.

These ops can be chained into a classical ultrasound image reconstruction
pipeline, as presented in figure 2. The pipeline can be augmented with machine
learning tools at any step of processing. Here we present two simple use cases
as an example: applying neural network to estimate homodyned K distribution
parameters from beamformed data (figure 3), and applying convolutional network
to classify beamformed image (figure 4). Our public library of ops will be
extended with the quantitative ultrasound (qUS) estimators in future.

Whenever possible, convenient and efficient, we implemented WaveFlow
operations as a composition of ops available in the TensorFlow core library
(e.g. Hilbert transform using the FFT). For other cases (e.g. DAS) we have
provided our own implementations for the CPU and GPU.

UltrasoundEnvImaging PipelineBeamformingAnalytic SignalAbsolute Value Dynamic
Adjustment RF, metadataframe

Figure 2: Pipeline illustrating B-mode image reconstruction. Blue boxes
represent operators defined in WaveFlow, orange – TensorFlow.

UltrasoundEnvBeamformingAnalytic SignalAbsolute
ValueEstimatorSlice$E[X]$$E[X^{2}]$$E[X^{3}]$FC… Dynamic Adjustment RF,
metadata[(u,k)]frame

Figure 3: Example use case: we can estimate the u and k parameters of the
Homodyned K distribution, which is widely used to model backscattered echo
statistics. The estimation can be performed using fully connected multi-layer
neural network. As the result, quantitative parametric maps of distribution’s
parameters are obtained. Blue boxes represent operators defined in WaveFlow,
orange – TensorFlow.

UltrasoundEnvImaging PipelineClassifierConv…Fully ConnectedRF,
metadataframeframeclass

Figure 4: Example use case: we can classify B-mode frame using convolutional
neural network. As the result, we obtain B-mode frame and its class. Blue
boxes represent operators defined in WaveFlow, orange – TensorFlow.

##  III Evaluation

The most recent performance evaluations are available at [1]. Here we present
the imaging pipeline throughput (figure 2), evaluated using the following
software and hardware:

  * •

WaveFlow ver. 0.1 (March 28, 2018);

  * •

TensorFlow ver. 1.6.0;

  * •

CUDA ver. 9.0;

  * •

NVidia Titan X GPU integrated in the USPlatform research scanner (us4us Ltd.,
Poland).

Experiments were performed according to the following assumptions:

  * •

STAI: input tensor with a shape (128, 64, 2048) [number of acquisitions,
number of channels, number of samples] per frame, data type: float64, output
frame: (2048, 128), single focal point, wire/cyst phantom data examined,
available here [3];

  * •

PWI: input tensor with a shape (11, 192, 2048), data type: float64, output
frame (512, 128), wire/cyst phantom data examined, available here [3];

  * •

dynamic clipped to range [0, 30] dB.

TABLE I: Results achieved on each step of ultrasound imaging pipeline. Step | STAI [ms/frame] | PWI [ms/frame]
---|---|---
Beamforming | 11 | 55
Envelope Detection | 5 | 4
Dynamic Adjustment | 2 | 1

Table I presents the number of miliseconds required to process an input data
frame at each step of the pipeline. In total, 18 ms/frame was required for the
STAI (55 FPS), and 60 ms/frame (17 FPS) for the PWI.

##  IV Conclusion

In this work, we implemented and verified the feasibility and efficiency of
WaveFlow – a set of ultrasound data acquisition and processing tools for
TensorFlow on a GPU-based ultrasonic research scanner. The results of the
experiments show, that our software can be successfully used to reconstruct
ultrasound B-mode images in real-time.

Tight integration with TensorFlow opens a possibility to conveniently
integrate the imaging pipeline with machine learning algorithms. It also
supports execution on multiple processing units, for the price of accepting
limitations of this software (e.g. static dataflow graph). WaveFlow extends
the TensorFlow framework with a collection of general purpose, signal
processing algorithms. It can be used to reconstruct ultrasound images in
particular, and can be executed on GPU or CPU, in experimental real-time
environment or for research purposes.

Waveflow is an open-source effort (github.com/waveflow-team/waveflow) and
contributors are welcome.

## References

  * [1] https://github.com/waveflow-team/waveflow/
  * [2] https://www.tensorflow.org/
  * [3] https://brain.fuw.edu.pl/edu/index.php/USG
  * [4] Byra Michał, et al. ”Combining Nakagami imaging and convolutional neural network for breast lesion classification.” _Ultrasonics Symposium (IUS), 2017 IEEE International. IEEE_ , 2017.
  * [5] Smistad Erik, Østvik Andreas. ”2D left ventricle segmentation using deep learning.”, _Ultrasonics Symposium (IUS), 2017 IEEE International. IEEE_ , 2017
  * [6] Perdios Dimitris, et al. ”A deep learning approach to ultrasound image recovery.” _Ultrasonics Symposium (IUS), 2017 IEEE International. IEEE_ , 2017.
  * [7] Lewandowski Marcin. ”Ultrasound Medical Imaging in the GPU Era.” _GPU Technology Conference, San Diego_ , 2018.
  * [8] Lewandowski Marcin, et al. ”Modular & scalable ultrasound platform with GPU processing.” _Ultrasonics Symposium (IUS), 2012 IEEE International. IEEE_ , 2012.
  * [9] Hyun Dongwoon. ”Customizable Ultrasound Imaging in Real-Time Using a GPU-Accelerated Beamformer.” _GPU Technology Conference, San Diego_ , 2018.
  * [10] Szepesvari Csaba. ”Algorithms for reinforcement learning.” _Synthesis lectures on artificial intelligence and machine learning_ 4.1 (2010): 1-103.


# Commensurate-Incommensurate Transitions of the 1D Disordered Chiral Clock
Model

Pengfei Liang  Beijing Computational Science Research Center, Beijing 100193,
China  Rosario Fazio  The Abdus Salam International Center for Theoretical
Physics, Strada Costiera 11, 34151 Trieste, Italy  Dipartimento di Fisica,
Università di Napoli “Federico II”, Monte S. Angelo, I-80126 Napoli, Italy
Beijing Computational Science Research Center, Beijing 100193, China  Stefano
Chesi<EMAIL_ADDRESS>Beijing Computational Science Research Center,
Beijing 100193, China  Department of Physics, Beijing Normal University,
Beijing 100875, China

###### Abstract

We study the effects of quenched disorder on the commensurate-incommensurate
transitions in the 1D $\mathbb{Z}_{N}$ chiral clock model. The interplay of
domain walls and rare regions rounds the sharp transitions of the pure model.
The density of domain walls displays an essential singularity, while the order
parameter develops a discontinuity at the transition. We perform extensive
density-matrix renormalization group calculations to support theoretical
predictions. Our results provide a distinct rounding mechanism of continuous
phase transitions in disordered systems.

Understanding phase transitions in disordered systems is a long-standing
problem in condensed matter physics. Quenched disorder that only causes
fluctuations of the local critical point without explicitly breaking the
underlying symmetries is of random-mass type Vojta (2006, 2019). The Harris
criterion Harris (1974) states that weak random-mass disorder is irrelavent to
a clean critical point if $\nu\geq 2/d$, where $\nu$ is the correlation length
exponent and $d$ is the spatial dimensionality. If disorder is relevant, the
clean critical point flows to a new dirty fixed point characterized by either
finite Vajk and Greven (2002); Sknepnek  _et al._ (2004); Vojta and Sknepnek
(2006); Vojta  _et al._ (2016); Iyer  _et al._ (2012); Vojta  _et al._ (2016)
or infinite McCoy and Wu (1968); Fisher (1992, 1995) randomness. Another
crucial aspect of disordered phase transitions is the role played by rare
regions, such as large spatial regions which are locally ordered. Recently, a
classification of critical points has been proposed Vojta (2006, 2019),
depending on the effective dimensionality $d_{r}$ of rare regions and the
lower critical dimension $d_{c}^{-}$ of the phase transition. If $d_{r}\leq
d_{c}^{-}$, a (quantum) Griffiths region is formed on both sides of the dirty
critical point, where some physical quantities display essential or power-law
Griffiths singularities Griffiths (1969); while if $d_{r}>d_{c}^{-}$, local
static order can develop in rare regions, rounding the original sharp
transition Vojta (2003); Sknepnek and Vojta (2004). Disorder can also round
classical or quantum first-order transitions to continuous ones Aizenman and
Wehr (1989); Goswami  _et al._ (2008). In the context of 1D $N$-state clock
models, the strong disorder renormalization group (SDRG) method has been
applied to the _achiral_ case Iglói and Monthus (2005), showing that the
critical point flows either to infinite randomness fixed points for $N=3,4$,
where disorder dominates over quantum fluctuations at large distance, or to
clean fixed points for $N\geq 5$, where weak disorder is irrelevant. This is
consistent with the Harris criterion since for $N=3,4$ phase transitions in
the corresponding clean model are in the $3$-state Potts and Ashkin-Teller
universality classes respectively, both of which satisfy $\nu<2$.

On the other hand, to the best of our knowledge, disorder in the chiral clock
model (CCM), a variant breaking charge conjugation symmetry, remain
unexplored. Interest in this model comes from the fact that a new
commensurate-incommensurate (C-IC) transition occurs Schulz (1980); Ostlund
(1981); Zhuang  _et al._ (2015); Samajdar  _et al._ (2018); Whitsitt  _et al._
(2018), which is known to violate the Harris criterion. More specifically, the
transition is of the Pokrosky-Talapov (PT) type Pokrovsky and Talapov (1979)
with $\nu=1/2$ for $N\geq 4$, and always happens through an intermediate
incommensurate (IC) critical phase. For $N=3$, both numerical simulations and
field-theoretical analysis support a direct transition with a continuous
varying exponent $\nu<5/6$ between ordered and disordered phases in the
vicinity of the Potts point Zhuang  _et al._ (2015); Samajdar  _et al._
(2018); Whitsitt  _et al._ (2018), which then crosses over to the PT
transition. The magnetic ordered phase is a domain-wall-free commensurate (C)
phase and there might be both clockwise (CC) and counterclockwise (CW) domain
wall excitations on top of it near the C-IC transition. Breaking of charge
conjugation makes one species of domain walls energetically more favored,
thereby an excess amount of domain walls destroys the magnetic long-range
order (LRO) in the IC phase.

In this work, we aim to study the effects of random-mass disorder on the C-IC
transitions in the 1D $\mathbb{Z}_{N}$ CCM by means of various theoretical
arguments and density-matrix renormalization group (DMRG) simulations. Our
main findings can be summarized as follows: excess domain-wall excitations
localized in rare regions of weakly-coupled sites (illustrated in the inset of
Fig. 1) round the sharp transition and, interestingly, induce a nonuniversal
essential singularity in the density of domain walls, while the
$\mathbb{Z}_{N}$ order parameter develops a discontinuity at the transition.
The latter does not mean that the transition is first-order as there is no
phase coexistence. Our results are in sharp contrast with the corresponding
achiral model and show a different rounding mechanism compared to Refs. Vojta
(2003); Sknepnek and Vojta (2004).

The $\mathbb{Z}_{N}$ random chiral clock model in 1D is given by the
Hamiltonian,

$\displaystyle
H(\theta,\phi)=-\sum_{j}\left(J_{j}\sigma_{j}^{\dagger}\sigma_{j+1}e^{i\theta}+h_{j}\tau_{j}^{\dagger}e^{i\phi}+{\rm
H.c.}\right),$ (1)

defined on a chain of length $L$. The bonds $J_{j}$ and transverse fields
$h_{j}$ are taken to be independent identically distributed (i.i.d.) random
variables and we assume they are positive, $J_{j},h_{j}>0$. The $N$-state spin
operators $\tau$ and $\sigma$ on each site obey the algebra
$\tau^{N}=\sigma^{N}=1,\;\sigma\tau=\omega\tau\sigma,\;\tau^{\dagger}=\tau^{-1},\;\sigma^{\dagger}=\sigma^{-1}$,
with $\omega=e^{i2\pi/N}$. Unlike the Pauli operators for a spin-half, these
operators are non-Hermitian with complex eigenvalues
$\omega^{0},\omega^{1},\cdots,\omega^{N-1}$. In the $\sigma$-representation,
$\sigma$ indicates the direction of spin while $\tau$ rotates the spin
clockwise by a discrete angle $2\pi/N$. A nonzero $\theta$ or $\phi$ breaks
the charge conjugation symmetry 111See Supplemental Material for discussion of
the symmetries, the C-IC transition and its low-energy effective Hamiltonian,
the decimation rules as well as for more DMRG results, which turns out to be
vital in the description of the transitions. The clean model possesses a
$\mathbb{Z}_{N}$ symmetry-broken phase with finite magnetization, whose phase
boundary for $N=3$ and $\phi=0$ is shown in Fig. 1.

Figure 1: Phase diagram of the $N=3$ model, with $J_{w}=1/4$, $J_{s}=1$, and
$\phi=0$. The dashed line is the phase boundary $h_{c}^{w}$ of the clean model
without strong links, i.e., $p=0$ in Eq. (2). Red and blue symbols are
critical points with finite disorder ($p=0.2$), see the dashed lines in Fig.
2(c). The green crosses at $\theta=0$ indicate the presence of a Griffith
region around the critical point. Inset: for $h>h_{c}^{w}$ rare regions of
weak links support a finite density of domain walls, destroying long-range
order. $\xi_{0}$ is the correlation length at $p=0$. Figure 2: (a)
$\overline{Q_{2}}$ and (b) $\overline{\langle\mathbf{m}^{2}\rangle}$ as a
function of $h$ at $\theta=13\pi/48$ and different values of $L$. The dashed
lines mark $h_{c}^{w}$. Inset of panel (b):
$\overline{\langle\mathbf{m}^{2}\rangle}$ plotted against $1/L$ at
$h=h_{c}^{w}$. Panel (c) shows the crossing points $h_{cr}$ between
$\overline{Q_{2}}$ curves of neighboring system size ($L$ and $2L$), at
several values of $L$ and $\theta$. For clarity, the clean critical value
$h_{c}^{w}$ has been subtracted from $h_{cr}$. In all panels we take
$J_{s}=1$, $J_{w}=1/4$, and $p=0.2$.

In the rest of the paper we assume $\phi=0$ and, by symmetry considerations
Note (1), restrict the range of $\theta$ from $0$ to $\pi/N$. We also consider
uniform $h_{j}=h$ and, initially, the binary probability distribution:

$P(J)=p\delta(J-J_{s})+(1-p)\delta(J-J_{w}),$ (2)

where $J_{s(w)}$ indicates strong (weak) bonds and $p$ is the concentration of
strong bonds. In analyzing the effects of disorder through DMRG Fishman  _et
al._ (2020), we have chosen a rather small truncation parameter $\sim
10^{-11}$ to ensure convergence in all calculations, and average over a
sufficient number of disorder realizations (up to $10,000$ in selected cases),
to achieve small statistical error bars. We address numerically the C-IC
transition through the disorder-averaged
$\overline{\langle\mathbf{m}^{2}\rangle}$, with
$\mathbf{m}=\sum_{j}\mathbf{m}_{j}/L$ and
$\mathbf{m}_{j}=((\sigma_{j}+\sigma_{j}^{\dagger})/2,(\sigma_{j}-\sigma_{j}^{\dagger})/2i)^{T}$,
as well as the Binder ratio
$\overline{Q_{2}}=\overline{\langle\mathbf{m}^{4}\rangle}/\overline{\langle\mathbf{m}^{2}\rangle}^{2}$,
where disorder average is done first to correctly detect phase transitions in
disordered systems Lin  _et al._ (2017). Figure 2 shows values of
$\overline{\langle\mathbf{m}^{2}\rangle}$ and $\overline{Q_{2}}$ computed at
$N=3$ and $\theta=13\pi/48$. As expected, the system remains ordered for
$h<h_{c}^{w}$, i.e., below the clean critical point associated with the weak
links. In this region, the magnetization extrapolates to a finite value (see
inset). On the other hand, as we will see shortly, the numerical data support
the occurrence of a disordered phase with the same phase boundary $h_{c}^{w}$
of the clean model.

To obtain the disordered critical point, we analyze the $\overline{Q_{2}}$
curves of Fig. 2(a), whose crossing points shift towards $h_{c}^{w}$ at large
$L$. As shown in Fig. 2(c), we have examined other values of $\theta$ and
observed a similar behavior: By plotting the difference between crossing
points $h_{cr}$ and $h_{c}^{w}$ as function of inverse length $1/L$, we find
that in all cases (if $\theta>0$) $h_{cr}$ shifts towards $h_{c}^{w}$ beyond a
certain crossover length scale $L^{*}$. Remarkably, for $\theta\geq\pi/8$
extrapolation of data with $L>L^{*}$ (dashed lines) gives values in close
agreement with $h_{c}^{w}$, also shown by the blue triangles in Fig. 1. For
$\theta=\pi/16$, the large size of the crossover scale $L^{*}$ does not allow
for an accurate extrapolation, but the behavior in Fig. 2(c) is still
consistent with the $h_{c}^{w}$ phase boundary.

Figure 3: Distribution of the surface magnetization ${\bf m}_{s}$ at two
different values of $h$. Other parametrs are as in Fig. 2.

For $\theta=0$, the monotonic behavior of $h_{cr}$ extrapolates to
$h_{c}^{*}\approx 0.33$, close to the theoretical predication for the random
transverse field Ising chain (TFIC), $h_{c}^{\rm TFIC}=\exp[p\log
J_{s}+(1-p)\log J_{w}]$ Iglói  _et al._ (2001). As the latter is controlled by
an infinite randomness fixed point (IRFP), we test if the same is true in our
model. Under this scenario, an independent assessment of $h_{c}^{*}$ is
through the surface magnetization, computed by fixing the state of spin $L$ to
one of the eigenstates of $\sigma_{L}$ (say, $|0\rangle$) and defining
$\mathbf{m}_{s}=\mathbf{m}_{1}$. For an IRFP, the distribution
$p(\log|\mathbf{m}_{s}|)$ features an exponential tail in the ordered
Griffiths region Iglói  _et al._ (2001):

$p(\log|\mathbf{m}_{s}|)\sim e^{1/z\log|\mathbf{m}_{s}|},$ (3)

where $z$ is the dynamical exponent and satisfies $z>1$. As shown in Fig. 3,
convergence to an exponential scaling is indeed achieved at $L=128$ for values
of $h$ as large as $h\approx 0.29$, while broadening and drifting of
$p(\log|\mathbf{m}_{s}|)$ towards smaller magnetization are found when
$h>h_{c}^{\rm TFIC}$. Such behavior provides clear evidence of an IRFP with
$h_{c}^{*}>0.29$ but a precise estimate of the critical point through
$p(\log|\mathbf{m}_{s}|)$ is challenging. In Note (1) we perform a detailed
comparison of the CCM and TFIC, showing that much larger values of $L$ (only
available in the TFIC) are necessary to obtain exponential scaling when
$h\approx h_{c}^{\rm TFIC}$. On the other hand, the close similarity of
$p(\log|\mathbf{m}_{s}|)$ between the two models at accessible system sizes
Note (1), together with the finite-size scaling analysis of Fig. 2(c), support
the conjecture $h_{c}^{*}=h_{c}^{\rm TFIC}$.

In the phase diagram of Fig. 1, the value $h_{c}^{*}\approx h_{c}^{\rm TFIC}$
is marked as a red star. Physically, the singular behavior at $\theta=0$ is
allowed by the divergence of the crossover scale $L^{*}$ at vanishing
$\theta$. An argument based on the flow of the SDRG, indicating that the
limits $L\to\infty$ and $\theta\to 0$ do not commute, will be discussed
towards the end.

Figure 4:  Average angle $\overline{\varphi_{ij}}$ vs distance $|j-i|$ for
different values of $h$. The inset shows the scaling of $k_{ave}/k_{0}^{w}$
with $\xi_{0}\propto\delta^{-\nu}$, where $k_{0}^{w}$ is obtained from
simulations of the clean model (see Fig. 5) and $\nu$ is from Ref. Samajdar
_et al._ (2018). We used $J_{s}=1$, $J_{w}=1/4$, $p=0.2$, $L=128$, and
$\theta=13\pi/48$.

We now return to the chiral case, recalling that at $\theta\neq 0$ the phase
transition features two complementary order parameters: magnetization and
domain wall density $\rho$. In the clean model, a finite amount of excess CC
(or CW) domain walls will completely destroy the magnetic order, and the
transitions of $\langle\mathbf{m}^{2}\rangle$ and $\rho$ happen
simultaneously. Domain walls cause winding of spins in space, thus $\rho$ is
accessible through the angle $\varphi_{ij}$ between two spins $i,j$. By using
$\cos\varphi_{ij}\propto\langle\mathbf{m}_{i}\cdot\mathbf{m}_{j}\rangle=\mathrm{Re}\langle\sigma_{i}\sigma_{j}^{\dagger}\rangle$,
where the expectation is taken over a $\mathbb{Z}_{N}$ symmetric state, we
obtain $\varphi_{ij}$ by calculating the two-points correlator
$\langle\sigma_{i}\sigma_{j}^{\dagger}\rangle=A_{ij}e^{i\varphi{ij}}$, where
$A_{ij}$ is the amplitude. In the clean case, a linear behavior of type
$\varphi_{ij}\simeq k_{0}(i-j)$ is obtained in the bulk of the chain, where
the slope $k_{0}$ is a measure of the density of excess domain walls. It
vanishes in the absence of domain walls (as in the ordered phase) or if the
two species of domain walls are excited equally (at $\theta=0$, guaranteed by
the charge conjugation symmetry). As shown in Fig. 4, in the disordered model
we find a similar linear dependence for the average angle
$\overline{\varphi_{ij}}\simeq k_{ave}(i-j)$.

Consistent with previous findings, $k_{ave}$ approaches zero at the clean
phase boundary $h_{c}^{w}$, where the leading singularity may be inferred
through Lifshitz-tail arguments. As illustrated in Fig. 1 (inset), domain
walls are most likely to live in extended regions of length $l\gg 1$ composed
of only weak links. Since $J_{j}$ are independent random variables, the
probability that a site belongs to one of such regions follows the Poisson
distribution $f(l)=e^{-\alpha l}(1-e^{-\alpha})$ with $\alpha=-\ln(1-p)$, and
is exponentially suppressed with $l$. In these rare regions, the linear
increase of $\overline{\varphi_{ij}}$ is controlled by $k_{0}^{w}$, i.e., the
slope of the clean model with $J_{j}=J_{w}$. We thus estimate:

$k_{ave}\simeq k_{0}^{w}\sum_{l=l_{0}}^{\infty}f(l)=k_{0}^{w}e^{-\alpha
l_{0}},$ (4)

where $l_{0}$ is a suitable cutoff. It is natural to identify $l_{0}\sim
c\xi_{0}$, with $\xi_{0}$ the clean correlation length. Indeed, by considering
individual disorder realizations at $h>h_{c}^{w}$, we find that ordered
domains are formed where strong links are dense. For a given $i$, the angle
$\varphi_{ij}$ is basically pinned to a constant when $j$ belongs to such an
ordered domain Note (1). Instead, as expected, magnetic correlations decay
exponentially between two distant $J_{s}$ links, where $\varphi_{ij}$ follows
approximately the linear dependence of the clean model. From these arguments
and Eq. (4), we conclude that $k_{ave}$ is given by:

$k_{ave}/k_{0}^{w}\sim e^{-\alpha c\xi_{0}}=e^{-\alpha
c^{\prime}\delta^{-\nu}},$ (5)

where $\delta=|h-h_{0}^{w}|$ measures the distance to $h_{0}^{w}$ and $\nu$ is
the corresponding clean correlation length exponent 222Note that Eq. (5) is
consistent with the two clean limits: when $p\to 0$ there are no strong links
and $k_{ave}\to k_{0}^{w}$ while in the limit $p\to 1$, the system is in the
ordered phase at $h_{c}^{w}$ and $k_{ave}\to 0$.. In the inset of Fig. 4 we
show that the numerically evaluated ratio $k_{ave}/k_{0}^{w}$ scales
exponentially with $\xi_{0}$, confirming the presence of an essential
singularity. We have also checked Eq. (5) is insensitive to system size $L$
and holds for other values of $\theta$ Note (1).

Figure 5:  Plot of $\langle\mathbf{m}^{2}\rangle$ and $k_{0}^{w}$ (solid
lines) for the clean model with $J_{j}=J_{w}$. In the disordered case we show
extrapolated values of $\overline{\langle\mathbf{m}^{2}\rangle}$ (blue filled
dots), obtained as in the inset of Fig 2(b), and $k_{ave}$ (red empty
circles), obtained from Fig. 4. Dashed lines follow the theoretical
predictions for the disordered case. All parameters are the same of Fig. 4.

Figure 1 shows the remarkable difference in the critical properties of the two
models (with and without disorder). Despite the jump in the averaged squared
magnetization $\overline{\langle\mathbf{m}^{2}\rangle}$, the domain wall
density has a smooth dependence, thus the disordered phase transition is still
continuous. In fact, as in a mixed-order phase transition Puel  _et al._
(2019, 2021), the correlation length $\xi$ of the diordered model diverges
when approaching the critical point. By an argument similar to Eq. (5), we
find an essential singularity also for $\xi$ Note (1). Below $h_{c}^{w}$ the
presence of strong bonds implies that (as in Fig. 5)
$\overline{\langle\mathbf{m}^{2}\rangle}$ is larger than in the clean model.
That $\overline{\langle\mathbf{m}^{2}\rangle}>0$ at $h_{c}^{w}$ is a result of
aligned locally ordered domains. For illustration, we can consider the simple
limit of small $p$ (when strong links are rare). An isolated strong link
nucleates a total magnetization $\Delta{\bf m}$ around it, thus a nonzero
magnetization density $\sim p\Delta{\bf m}$ survives at $h\to h_{c}^{w}$.
Importantly, when $h\lesssim h_{c}^{w}$ the intermediate regions formed by
weak links support long-range order. Therefore, differently from the
schematics in Fig. 1 (where $h>h_{c}^{w}$), two distant strongly ordered
regions remain correlated.

The main features obtained using Eq. (2) are still valid for general $N$.
Furthermore, the above arguments impose no constraint on the strength and
functional form of disorder and apply to random $h_{j}$ as well. We have also
analyzed Eq. (1) through a rigorous mapping of chiral domain-wall excitations
to spinless fermions, valid for general $N$ and small $\delta=\pi/N-\theta$
Note (1). A main conclusion of that approach is that $J_{j}$ mimics a random
potential with strength $\propto J_{j}$, thus excess domain walls are most
likely localized in the rare regions while strong links act like energy
barriers for the domain walls. This physical picture is consistent with the
critical behavior discussed so far. We note, however, that the functional form
of the singular part of $k_{ave}/k_{0}^{w}$ is nonuniversal and may depend on
the details of the distribution. For distributions of $J_{j}$ with
infinitesimal small bonds (e. g., $J_{w}\to 0$) we may expect that long range
order is completely washed out, as rare regions exist for any finite $h$. We
have studied explicitly the uniform distribution:

$P(J)=\left(\Theta(J)+\Theta(J_{s}-J)\right)/2J_{s},$ (6)

with $\Theta(J)$ the Heaviside step function. Indeed, now the crossings in
$\overline{Q_{2}}$ shift all the way down to zero Note (1).

Finally, it is interesting to look at the phase transition form the
perspective of the SDRG method. The decimation rules of the random
$\mathbb{Z}_{N}$ CCM read Note (1):

$\displaystyle
h_{j}^{\prime}=\kappa_{\theta_{j}}\frac{h_{j}h_{j+1}}{J_{j}},\quad\phi_{j}^{\prime}=\phi_{j}+\phi_{j+1},$
(7) $\displaystyle
J_{j-1}^{\prime}=\kappa_{\phi_{j}}\frac{J_{j-1}J_{j}}{h_{j}},\quad\theta_{j-1}^{\prime}=\theta_{j-1}+\theta_{j},$
(8)

where $2\kappa_{\alpha}=\sum_{s=\pm 1}[\cos\alpha-\cos(\alpha+2s\pi/N)]^{-1}$
and we have introduced a position dependence of the chiral phases
$\theta_{j},\phi_{j}$, to incorporate the fact that they will change during
the renormalization process. In the achiral case ($\theta_{j}=\phi_{j}=0$) we
recover the constant prefactor $\kappa_{0}=1/[1-\cos(2\pi/N)]$, thus the SDRG
flow proceeds as usual. For $N=3,4$, we have $\kappa_{0}<1$, meaning that the
critical point flows to IRFP; while for $N\geq 5$, we have $\kappa_{0}>1$,
meaning that weak disorder is irrelevant Iglói and Monthus (2005). If the
initial values of $\theta,\phi$ are nonzero, however, the chiral phases evolve
in a nontrivial manner since they accumulate over progressively larger domains
through Eqs. (7) and (8). Inevitably, resonances will occur at some
renormalization step, indicating a breakdown of the renormalization group
flow. For small starting values, $\theta,\phi\ll 1$, the initial flow follows
closely the achiral case. Significant deviations and the appearance of
resonances only occur above a certain scale which, we suggest, is related to
the large crossover length $L^{*}$ observed in numerical simulations.

In conclusion, we have shown that the sharp C-IC transitions in the disordered
$\mathbb{Z}_{N}$ CCM are rounded by the interplay of excess domain wall
excitations and rare regions. Compared to the known rounding mechanism of
continuous phase transitions in Ref. Vojta (2003), our results reveal an
important role played by low-energy excitations in certain disordered
continuous phase transitions.

###### Acknowledgements.

P.L. acknowledges computational resources from the Beijing Computational
Science Research Center. R.F. acknowledges partial financial support from the
Google Quantum Research Award. R.F. research has been conducted within the
framework of the Trieste Institute for Theoretical Quantum Technologies (TQT).
S.C. acknowledges support from NSFC (Grants No. 11974040 and No. 12150610464),
and NSAF (Grant No. U1930402).

## References

  * Vojta (2006) T. Vojta, Journal of Physics A: Mathematical and General 39, R143 (2006).
  * Vojta (2019) T. Vojta, Annual Review of Condensed Matter Physics 10, 233 (2019), https://doi.org/10.1146/annurev-conmatphys-031218-013433 .
  * Harris (1974) A. B. Harris, Journal of Physics C: Solid State Physics 7, 1671 (1974).
  * Vajk and Greven (2002) O. P. Vajk and M. Greven, Phys. Rev. Lett. 89, 177202 (2002).
  * Sknepnek  _et al._ (2004) R. Sknepnek, T. Vojta, and M. Vojta, Phys. Rev. Lett. 93, 097201 (2004).
  * Vojta and Sknepnek (2006) T. Vojta and R. Sknepnek, Phys. Rev. B 74, 094415 (2006).
  * Vojta  _et al._ (2016) T. Vojta, J. Crewse, M. Puschmann, D. Arovas, and Y. Kiselev, Phys. Rev. B 94, 134501 (2016).
  * Iyer  _et al._ (2012) S. Iyer, D. Pekker, and G. Refael, Phys. Rev. B 85, 094202 (2012).
  * McCoy and Wu (1968) B. M. McCoy and T. T. Wu, Phys. Rev. Lett. 21, 549 (1968).
  * Fisher (1992) D. S. Fisher, Phys. Rev. Lett. 69, 534 (1992).
  * Fisher (1995) D. S. Fisher, Phys. Rev. B 51, 6411 (1995).
  * Griffiths (1969) R. B. Griffiths, Phys. Rev. Lett. 23, 17 (1969).
  * Vojta (2003) T. Vojta, Phys. Rev. Lett. 90, 107202 (2003).
  * Sknepnek and Vojta (2004) R. Sknepnek and T. Vojta, Phys. Rev. B 69, 174410 (2004).
  * Aizenman and Wehr (1989) M. Aizenman and J. Wehr, Phys. Rev. Lett. 62, 2503 (1989).
  * Goswami  _et al._ (2008) P. Goswami, D. Schwab, and S. Chakravarty, Phys. Rev. Lett. 100, 015703 (2008).
  * Iglói and Monthus (2005) F. Iglói and C. Monthus, Physics Reports 412, 277 (2005).
  * Schulz (1980) H. J. Schulz, Phys. Rev. B 22, 5274 (1980).
  * Ostlund (1981) S. Ostlund, Phys. Rev. B 24, 398 (1981).
  * Zhuang  _et al._ (2015) Y. Zhuang, H. J. Changlani, N. M. Tubman, and T. L. Hughes, Phys. Rev. B 92, 035154 (2015).
  * Samajdar  _et al._ (2018) R. Samajdar, S. Choi, H. Pichler, M. D. Lukin, and S. Sachdev, Phys. Rev. A 98, 023614 (2018).
  * Whitsitt  _et al._ (2018) S. Whitsitt, R. Samajdar, and S. Sachdev, Phys. Rev. B 98, 205118 (2018).
  * Pokrovsky and Talapov (1979) V. L. Pokrovsky and A. L. Talapov, Phys. Rev. Lett. 42, 65 (1979).
  * Note (1) See Supplemental Material for discussion of the symmetries, the C-IC transition and its low-energy effective Hamiltonian, the decimation rules as well as for more DMRG results.
  * Fishman  _et al._ (2020) M. Fishman, S. R. White, and E. M. Stoudenmire, “The ITensor software library for tensor network calculations,” (2020), arXiv:2007.14822 .
  * Lin  _et al._ (2017) Y.-P. Lin, Y.-J. Kao, P. Chen, and Y.-C. Lin, Phys. Rev. B 96, 064427 (2017).
  * Iglói  _et al._ (2001) F. Iglói, R. Juhász, and P. Lajkó, Phys. Rev. Lett. 86, 1343 (2001).
  * Note (2) Note that Eq. (5) is consistent with the two clean limits: when $p\to 0$ there are no strong links and $k_{ave}\to k_{0}^{w}$ while in the limit $p\to 1$, the system is in the ordered phase at $h_{c}^{w}$ and $k_{ave}\to 0$.
  * Puel  _et al._ (2019) T. O. Puel, S. Chesi, S. Kirchner, and P. Ribeiro, Phys. Rev. Lett. 122, 235701 (2019).
  * Puel  _et al._ (2021) T. O. Puel, S. Chesi, S. Kirchner, and P. Ribeiro, Phys. Rev. B 103, 035108 (2021).

See pages 1 of supp.pdf

See pages 2 of supp.pdf

See pages 3 of supp.pdf

See pages 4 of supp.pdf

See pages 5 of supp.pdf

See pages 6 of supp.pdf

See pages 7 of supp.pdf

See pages 8 of supp.pdf