(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:  University of Texas at Austin 22institutetext: FAIR, Meta

# Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric
Videos

Changan Chen∗ 11 Puyuan Peng∗ 11 Ami Baid 11 Zihui Xue 11
Wei-Ning Hsu 22 David Harwath 11 Kristen Grauman 11

###### Abstract

Generating realistic audio for human interactions is important for many
applications, such as creating sound effects for films or virtual reality
games. Existing approaches implicitly assume total correspondence between the
video and audio during training, yet many sounds happen off-screen and have
weak to no correspondence with the visuals—resulting in uncontrolled ambient
sounds or hallucinations at test time. We propose a novel _ambient-aware_
audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism
to learn to disentangle foreground action sounds from the ambient background
sounds in in-the-wild training videos. Given a novel silent video, our model
uses retrieval-augmented generation to create audio that matches the visual
content both semantically and temporally. We train and evaluate our model on
two in-the-wild egocentric video datasets Ego4D and EPIC-KITCHENS. Our model
outperforms an array of existing methods, allows controllable generation of
the ambient sound, and even shows promise for generalizing to computer
graphics game clips. Overall, our work is the first to focus video-to-audio
generation faithfully on the observed visual content despite training from
uncurated clips with natural background sounds.

###### Keywords:

audio-visual learning egocentric video understanding action sound generation

${}^{*}$${}^{*}$footnotetext: indicates equal contribution.

##  1 Introduction

We interact with objects around us in our daily lives and these actions often
produce sound as a result of physical interactions, e.g., clicking on a mouse,
closing a door, or cutting vegetables. The distinct characteristics of these
_action sounds_ depend upon the type of action being performed, the shape and
material composition of the objects being acted upon, the amount of force
being applied, and so forth. Vision not only captures _what_ physical
interaction happens but also informs us _when_ the interaction happens,
suggesting the possiblity of synthesizing semantically plausible and
temporally synchronous action sounds from silent videos alone. This capability
would accelerate many real-world applications, such as text-to-video
generation, generating sound effects for films (Foley), or sound effect
generation for virtual reality (VR) and video games.

Figure 1: Real-world audio consists of both foreground action sounds (whose
causes are visible in the FoV) and background ambient sounds that are
generated by sources offscreen. Whereas prior work is agnostic to this
division when performing generation, our method is ambient-aware and
disentangles action sound from ambient sound. Our key technical insight is how
to train with in-the-wild videos exhibiting natural ambient sounds, while
still learning to factor out their effects on generation. The green arrows
reference how we condition generation on sound from a related, but time-
distinct, video clip to achieve this.

Some prior work studies impact sound synthesis from videos [42, 48] while
others target more general video-to-audio generation [25, 38]. All these
methods _implicitly assume total correspondence between the video and audio_
and aim to generate the whole target audio from the video. However, this
strategy falls short for in-the-wild training videos, which are rife with off-
screen ambient sounds, e.g., traffic noise, people talking, or A/C running.
While some of these ambient sounds are weakly correlated with the visual
scene, such as the wind blowing in an outdoor environment, many of them have
no visual correspondence, such as off-screen speech or a stationary buzzing
noise from the fridge. Existing methods are not able to disentangle action
sounds from ambient sounds and treat them as a whole, leading to uncontrolled
generation of ambient sounds at test time and sometimes even hallucination,
e.g., random action or ambient sounds. This is particularly problematic for
generating action sounds because they are often subtle and transient compared
to the ambient sounds. For example, trained in the traditional way, a model
given a scene that looks like a noisy restaurant risks generating “restaurant-
like" ambient sounds, while ignoring the actual movements and activities of
the foreground actions, such as a person stirring their coffee with a metal
spoon.

How can we disentangle the foreground action sounds from background ambient
sounds for in-the-wild video data _without_ ground truth separated streams?
Simply applying a noise removal algorithm on the target audio does not work
well since in-the-wild blind source separation of general sounds from a single
microphone is still an open challenge [49]. The key observation we have is
that while action sounds are highly localized in time, ambient sounds tend to
persist across time. Given this observation, we propose a simple but effective
solution to disentangle ambient and action sounds: during training, in
addition to the input video clip, we also condition the generation model on an
audio clip from the same long video as the input video clip but from different
timestamps. See Fig. 1. By doing so, we lift the burden of generating energy-
dominating ambient sounds and encourage the model to focus on learning action
cues from the visual frames to generate action sounds. At test time, we do not
assume access to (even other clips of) the ground truth video/audio. Instead,
we propose to retrieve an audio segment from the training set with an audio-
visual similarity scoring model, inspired by recent ideas in retrieval-
augmented generation (RAG) [29, 19, 34]. This benefits examples where the
visual scene has a weak correlation with the ambient sound that is appealing
to capture, e.g., outdoor environments.

Existing action sound generation work relies on either clean, manually-
collected data that has a limited number of action categories [42, 48, 8], or
videos crawled from YouTube based on predefined taxonomies [15, 5, 25]. To
expand the boundary of action sound generation to in-the-wild human actions,
we take advantage of recent large-scale egocentric video datasets [18, 9].
Though our model is not tailored to egocentric video in any way, there are two
main benefits of using these datasets: 1) egocentric videos provide a close
view of human actions compared to exocentric videos, where hand-object
interactions are much smaller from a distance and often occluded, and 2) these
datasets have timestamped narrations describing atomic actions. We design an
automatic pipeline to extract and process clips from Ego4D, and curate
Ego4D-Sounds with 1.2 million audio-visual action clips.

Our idea of disentangling action and ambient sounds implicitly in training is
model-agnostic. In this paper, we instantiate it by designing an audio-visual
latent diffusion model (AV-LDM) that conditions on both modality streams for
audio generation. We evaluate our AV-LDM against recent works on a wide
variety of metrics and show that our model outperforms the existing methods
significantly on both Ego4D-Sounds and EPIC-KITCHENS. We conduct a human
evaluation study that shows our model synthesizes plausible action sounds
according to the video. Please see/listen for yourself in our supplementary
video! We also show promising preliminary results on virtual reality game
clips. To the best of our knowledge, this is the first work that demonstrates
the disentanglement of foreground action sounds from background sounds for
action-to-sound generation on in-the-wild videos.

##  2 Related Work

###  2.1 Action Sound Generation

A pioneering work for capturing human-generated action sounds collects videos
where people hit, scratch, or prod objects with a drumstick [42]. This is an
early inspirational effort, though it is by design limited in the type of
actions. The robotics community also studies this problem by using robotic
platforms to collect collision sounds and analyze or synthesize them from
video [14, 8]. Other works approach this problem by building a simulation for
collision events [13]; however, it is hard for computational approaches to
simulate the impact or general action sounds due to the complexity of the
physical interactions. Existing methods demonstrate good synthesis results
when the data are noise-free. However, they are not equipped to learn from in-
the-wild action videos, where the action sound is always coupled with ambient
sound. We propose an ambient-aware model to deal with this issue head-on and
also introduce the Ego4D-Sounds dataset to expand action sound synthesis to
in-the-wild actions.

###  2.2 Egocentric Video Understanding with Audio

Understanding human activities in videos has long been a core challenge of
computer vision. Early research studies activity recognition from exocentric
videos such as UCF101 [47], Kinetics [27], or ActivityNet [12]. Recent work
explores the egocentric setting and introduces large egocentric datasets such
as Ego4D [18] or EPIC-KITCHENS [9]. Leveraging both the video and audio
streams in egocentric videos, many interesting tasks are enhanced, such as
action recognition [28], localization [44], active speaker localization [26],
sounding object localization [22], and state-aware visual representations from
audible interactions [39]. Most related to our work is SoundingActions [4]
that learns visual representations of actions that make sounds, which is
valuable for indexing and recognition problem settings, but ill-equipped for
generation, as we show later when retrieval models [16] perform poorly for
generating synchronized audio. All existing audio-visual learning for
egocentric video focuses on perception, i.e., understanding what happens in
the video. In contrast, we target the video-to-audio generation problem.
Furthermore, relative to any of the above, our idea to implicitly learn to
disentangle the action sound from ambient sounds is novel.

###  2.3 Diffusion Models and Conditional Audio Generation

Diffusion models have attracted significant attention recently because of
their high fidelity generation [10, 40, 45]. Initially proposed for image
generation [46, 20], dthey have also been successfully applied to speech and
audio generation [33, 43, 51, 36, 23]. Benefitting from classifier-free
guidance [21] and large-scale representation learning, AudioLDM [36] and Make-
An-Audio [23] have demonstrated successful diffusion-based _text_ -to-audio
generation. More recently, Diff-Foley [38] adapts latent diffusion models for
video-to-audio generation by first conducting audio-video contrastive learning
and then video-conditioned audio generation. While this approach demonstrates
promising results, it does not address the background ambient sound problem.
Inspired by recent work on retrieval-augmented generation (RAG) for text [29,
19, 34, 3] and image generation [6, 2], we show how our audio-conditioning
insight carries over to inference time via a retrieval component of the model.
Conditional video-to-audio generation conditions on either a physics prior to
guide diffusion-based impact sound generation [48] or, in CondFoleyGen [11],
another video clip to modify the characteristics of the action sound. Our
method also considers additional conditioning signals to control the output,
but for a very different purpose; our model is the first to address
foreground/background sound disentanglement in generation.

##  3 Ambient-aware Action Sound Generation

Figure 2: Illustration of the harm of ambient sound in video-to-audio
generation. In this example, this person is closing a packet of ginger powder,
which makes some rustling sound (red circled in the middle). There is also
some buzzing sound semantically irrelevant to the visual scene in the
background, which dominates the energy of the spectrogram. On the right-hand
side, we show a prediction made by a vanilla model that misses the action
sound but predicts the ambient sound.

We first discuss our high-level idea of how to guide the generation model to
disentangle action sounds from ambient sounds. We then extend the latent
diffusion models (LDM) to accommodate both audio and video conditions, which
we name AV-LDM. We also discuss our pretraining stage.

###  3.1 Action-to-Sound Generation

Given a video $V\in\mathbb{R}^{(T*S_{V})\times H\times W\times 3}$, where $T$
is the duration of the video and $S_{V}$ is the video sample rate, and the
accompanying audio waveform $A\in\mathbb{R}^{1\times(T*S_{A})}$, where $S_{A}$
is the audio sample rate, our goal is to model the conditional distribution
$p(A|V)$ for video-to-audio generation. During training we observe natural
video coupled with its audio, whereas at inference time we have only a silent
video—e.g., could be an output from text-to-video generation, or a VR/video
game clip, or simply a real-world video for which we want to generate new
plausible sounds.

###  3.2 Disentangling Action and Ambient Sounds

Learning a video-to-audio generation model using in-the-wild egocentric videos
is challenging because of entangled foreground action and background ambient
sounds, as illustrated in Fig. 2. More specifically, the reasons are two-fold:
1) while action sounds are usually of very short duration, ambient sounds can
last the entire clip, and therefore dominate the loss, leading to low-quality
action sound generation; 2) while some ambient sounds might be semantically
related to the visual scene such as bird chirping in the woods, in many cases,
ambient sounds are difficult to infer from the visual scene because they are
the results of the use of certain microphones, recording conditions, people
speaking, off-screen actions, etc. Forcing a generation model to learn those
background sounds from video results in hallucinations during inference (see
examples in Fig. 6).

Therefore, it’s beneficial to proactively disentangle action sounds and
ambient sounds during training. However, separating in-the-wild ambient sounds
is still an open challenge as recent models rely on supervised training on
artificially mixed sounds, for which the ground truth complex masks can be
obtained [49]. Simply applying off-the-shelf noise reduction methods to
training data leads to poor performance, as we will show in Sec. 5.

While it is difficult to _explicitly_ separate the ambient and action sound in
the target audio, our key observation is that ambient sounds are usually
fairly stationary across time. Given this observation, we propose a simple but
effective method to achieve the disentanglement. During training, in addition
to video clip $V$, we also provide the model an audio clip $A_{n}$ that comes
from the same training video but a different timestamp as the input video clip
(see Fig. 3). Therefore, instead of modeling $p(A|V)$, we model
$p(A|V,A_{n})$. Given the hypothesis that $A_{n}$ is likely to share ambient
sound characteristics with $A$, it can take away the burden of learning weakly
correlated or even uncorrelated ambient sounds from visual input alone, and
encourages the model to focus on learning action features from the visual
input. For the selection of $A_{n}$, we randomly sample one audio clip from
the nearest $X$ clips in time. While there is no guarantee that the sampled
audio shares exactly the same ambient sound with the target audio, their
ambient sounds should largely overlap since they are close in time, which
provides a consistent learning signal to help the model learn the
disentanglement.

Figure 3: Audio condition selection and the model architecture. Left: During
training, we randomly sample a neighbor audio clip as the audio condition. For
inference, we query the training set audio with the (silent) input video and
retrieve an audio clip that has the highest audio-visual similarity with the
input video using our trained AV-Sim model (Sec. 3.5). Right: We represent
audio waveforms as spectrograms and use a latent diffusion model to generate
the spectrogram conditioned on both the input video and the audio condition.
At test time, we use a trained vocoder network to transform the spectrogram to
a waveform.

###  3.3 Retrieval Augmented Generation and Controllable Generation

While during training we have access to the clips in the same long video as
the input clip, we of course cannot access that information at test time. How
we select $A_{n}$ at test time depends on the purpose of the generation. We
consider two use cases: action-ambient joint generation and action-focused
generation. In the first scenario, we would like the model to generate both
the action sound and the ambient sound that is plausible for the visual
environment. This is, for example, useful for generating sound effects for
videos. In the latter scenario, we would like the model to focus the
generation on action sounds and _minimize_ ambient sounds, which is useful,
for example, for generating sounds for games. Fig. 4 depicts the two
scenarios.

For action-ambient joint generation, we want $A_{n}$ to be semantically
relevant to the visual scene. Inspired by recent work in retrieval augmented
regeneration, we propose to retrieve audio such that:

$A_{n}=\operatorname*{arg\,max}_{A_{i}\in\mathcal{D}}{\text{AV-
Sim}(A_{i},V)},$ (1)

where $\mathcal{D}$ is the dataset of all training audio clips and $V$ is the
(silent) input video. $\text{AV-Sim}(A,V)$ is a similarity scoring function
that measures the similarity between $A$ and $V$, which we will cover in Sec.
3.5.

For action-focused generation, we want $A_{n}$ to have minimal ambient level.
We find simply filling $A_{n}$ with all zeros results in poor performance,
likely because it is too far out of the training distribution. Instead, we
find conditioning the generation on a low-ambient sound will hint the model to
focus on action sound generation and generate minimal ambient sound. See Sec.
5.3.

Figure 4: Two inference settings: “action-ambient joint generation” and
“action-focused generation”. In the first setting, we condition on audio
retrieved from the training set and aim to generate both plausible action and
ambient sounds. In the second setting, we specify an audio file with low
ambient sound and the model focuses on generating plausible action sounds
while minimizing the ambient sounds.

###  3.4 Audio-Visual Latent Diffusion Model

While the above idea of disentanglement is universal and not specific to any
model architecture, here we instantiate this idea on diffusion models due to
their success in audio generation [36, 38]. We extend the latent diffusion
model to accommodate our audio-visual conditions, thus yielding an audio-
visual latent diffusion model (AV-LDM).

Fig. 3 (right) shows the architecture of our model. During training, given
audio waveform target $A$, we first compute the mel-spectrogram
$x_{0}\in\mathbb{R}^{T\times D_{\text{mel}}}$, where $D_{\text{mel}}$ is the
number of mel bins. We then use a pretrained Variational Autoencoder (VAE) to
compress the mel-spectrogram $x_{0}$ to a latent representation
$z_{0}\in\mathbb{R}^{C^{\prime}\times H^{\prime}\times W^{\prime}}$, where
$z_{0}$ is the generation target of the LDM. We condition the generation on
both the video feature $c_{v}\in\mathbb{R}^{T_{v},D_{c}}$ and audio feature
$c_{a}\in\mathbb{R}^{T_{a},D_{c}}$. We extract the video feature with a
pretrained video encoder (see Sec. 3.5) from $V$. We extract the audio feature
from the audio condition $A_{n}$ with the same VAE encoder and then transform
the feature into 1-d vector with a multilayer perceptron (MLP).

Following [38], we use cross attention where the query is produced by $z_{t}$,
which is the sample diffusion step $t$, and key and value are produced by
$\text{concat}([\text{Pos}_{v}+c_{v};\text{Pos}_{a}+c_{a}])$, where Pos
denotes learnable positional embeddings. The model is trained with the
denoising objective:

$\mathcal{L}=\mathbb{E}_{t\sim\text{uniform}(1,T),z_{0},\epsilon_{t}}\|\epsilon_{t}-\epsilon_{\theta}(\mathbf{x}_{t},t,c_{v},c_{a})\|^{2},$

where $\epsilon_{t}$ is the standard Gaussian noise sampled for diffusion step
$t$, and $\epsilon_{\theta}(\mathbf{x}_{t},t,c_{v},c_{a})$ is the model
estimation of it ($\theta$ represents model parameters).

The reverse process can be parameterized as:

$\displaystyle p(z_{T})=\mathcal{N}(0,I),$ $\displaystyle
p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\frac{1}{\sqrt{\alpha_{t}}}\Big{(}z_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(z_{t},t,c_{v},c_{a})\Big{)},\sigma_{t}^{2}I),$

where $\alpha_{t}$ and $\sigma_{t}$ are determined by noise schedule of the
diffusion process. To generate audio during inference, we first sample
standard Gaussian noise $z_{T}$, and then apply classifier free guidance [21]
to estimate $\tilde{\epsilon}_{\theta}$ as

$\tilde{\epsilon}_{t}(z_{t},t,c_{v},c_{a})=\omega\epsilon_{\theta}(z_{t},t,c_{v},c_{a})+(1-\omega)\epsilon_{\theta}(z_{t},t,\emptyset,\emptyset),$

where $\emptyset$ denotes zero tensor. For the above estimation to be more
precise, during training, we randomly replace $c_{v}$ with $\emptyset$ with
probability $0.2$. As for $c_{a}$, we found dropping it even with even a small
probability harms the performance, and therefore we always condition the LDM
with $c_{a}$.

During inference, we use DPM-Solver [37] on LDM to sample a latent
representation, which is then upsampled into a mel-spectrogram by the decoder
of VAE. Lastly, we use a vocoder (HiFi-GAN [32]) model to generate waveform
from the mel-spectrogram.

###  3.5 Audio-Visual Representation Learning

Generating semantically and temporally synchronized action sounds from video
requires the video encoder to capture these relevant features. In addition, we
would like to train a video model and an audio model whose representations
align in the embedding space to support retrieval-augmented generation
discussed in Sec. 3.3. For this purpose, we train a video encoder and audio
encoder contrastively to optimize the following objective:

$\text{AV-
Sim}(A,V)=-\frac{1}{|\mathcal{B}|}\sum_{t\in\mathcal{B}}\log\frac{\exp(e_{A}^{t}e_{V}^{t}/\tau)}{\sum_{l\in\mathcal{B}}\exp(e_{A}^{t}e_{V}^{l}/\tau)},$

where $\mathcal{B}$ is the current batch of data, $e_{A}^{t}$ and $e_{V}^{t}$
are normalized embeddings of the audio and video features, $\tau$ is a
temperature parameter. To leverage the full power of narrations on Ego4D, we
initialize the video encoder weights from models pre-trained on video and
language from [35].

###  3.6 Implementation Details

We use Ego4D-Sounds (see Sec. 4) to train our AV-LDM. Video is sampled at 5FPS
and audio is sampled at 16kHz. Video is passed through the pre-trained video
encoder to produce condition features $c_{v}\in\mathbb{R}^{16\times 768}$. The
audio waveform is transformed into a mel-spectrogram with a hop size of 256
and 128 mel bins. The mel-spectrogram is then passed to the VAE encoder with
padding in the temporal dimension to produce target
$z_{0}\in\mathbb{R}^{4\times 16\times 24}$. The audio condition is processed
the same way except that we use an additional MLP to process VAE’s output to
produce $c_{a}\in\mathbb{R}^{24\times 768}$. We load the weights of VAE and
LDM from the pretrained Stable Diffusion to speed up training, similar to
[38], and VAE is kept frozen during training. LDM is trained for $8$ epochs
with batch size $720$ on Ego4D-Sounds with the AdamW optimizer with learning
rate $1e-4$. During inference, we use $25$ sampling steps with classifier-free
guidance scale $\omega=6.5$. For HiFi-GAN, we train it on a combination of
0.5s segments from Ego4D[18], Epic-Kitchens [24], and AudioSet [15]. We use
AdamW to train HiFi-GAN with a learning rate of $2e-4$ and batch size of $64$
for 120k steps. We set the number of random nearby audio samples $X=6$. See
more details in Supp.

##  4 The Ego4D-Sounds Dataset

Figure 5: Example clips in Ego4D-Sounds. We show one video frame, the action
description, and the sound for each example. Note how these actions are subtle
and long-tail, usually not present in typical

video datasets.

Datasets | Clips | Language | Action Types
---|---|---|---
The Greatest Hits [42] | 46.6K | ✗ | Hit, scratch, prod
VGG-Sound [5] | 200K | Video tags | Not action-specific
EPIC-SOUNDS [24] | 117.6K | Audio labels | Kitchen actions
Ego4D-Sounds | 1.2M | Action narrations | In-the-wild actions
Table 1: Comparison with other audio-visual action datasets. Ego4D-Sounds not
only has one order of magnitude more clips, but it is also coupled with
language descriptions, supporting evaluation of sound generation based on
semantics.

Next we describe our efforts to curate Ego4D-Sounds, an audio-video dataset
for human action sound generation. Our goal is to curate a high-quality
dataset for action-audio correspondence for action-to-sound generation,
addressing the issue of limited action types in the existing impact sound
datasets [42, 7].

Ego4D [18] is an existing large-scale egocentric video dataset that has more
than 3,600 hours of video recordings depicting hundreds of daily activities;
2,113 of those hours have audio available. It also has time-stamped narrations
that are free-form sentences describing the current activity performed by the
camera-wearer. We first utilize the narration timestamps in Ego4D to extract
clips. However, not all clips have meaningful action sounds and there are many
actions like “talk with someone", “look around", “turn around" that have low
audio-visual correspondence. We then use an automatic pipeline to process all
extracted clips to create the Ego4D-Sounds dataset, which has 1.2 million
audio-visual action clips. Similarly, for the test set, we curate 11k clips
for evaluation. See Supp. for more details on the data processing pipeline. We
show examples in Fig. 5 and comparison with other datasets in Tab. 1.

For all resulting clips, we extract them as 3s clips with $224\times 224$
image resolution at $30$ FPS. For audio, we extract them as a single channel
with a $16000$ sample rate.

##  5 Experiments

###  5.1 Evaluation

To evaluate the performance of our model, we use the following metrics:

  1. 1.

Fréchet Audio Distance (FAD) [30]: evaluates the quality of generated audio
clips against ground truth audio clips by measuring the similarity between
their distributions. We use the public pytorch implementation.
111https://github.com/gudgud96/frechet-audio-distance

  2. 2.

Audio-visual synchronization (AV-Sync) [38]: a binary classification model
that classifies whether the video and generated audio streams are
synchronized. Following [38], we create negative examples by either shift
audio temporally or sample audio from a different video clip. See more details
in Supp.

  3. 3.

Contrastive language-audio contrastive (CLAP) scores [50]: evaluates the
semantic similarity between the generated audio and the action description. We
finetune the CLAP model 222https://github.com/LAION-AI/CLAP on the
Ego4D-Sounds data and compute scores for the generated audio and the narration
at test time.

These metrics measure different aspects of generation collectively, including
the distribution of generated samples compared to the ground truth clips,
synchronization with the video, and the semantic alignment with the action
description.

We compare with the following baseline methods:

  1. 1.

Retrieval: we retrieve the audio from the training set using the AV-Sim model
introduced in Sec. 3.5. This method represents retrieval-based generation
models such as ImageBind [16].

  2. 2.

Spec-VQGAN [25]: a video-to-audio model that generates audio based on a
codebook of spectrograms. We run their pre-trained model on our test set.

  3. 3.

Diff-Foley [38]: a recent LDM-based model. We follow their fine-tuning steps
on egocentric videos to train on our dataset.

Neither learning-based model has the ability to tackle the ambient sound,
whereas our model disentangles it from the action sound.

In addition, we also evaluate the following ablations: “w/o vocoder": we
replace the trained HiFi-GAN vocoder with Griffin-Lim; “w/o cond": we remove
the audio condition at training time; “w/o cond + denoiser": we use an off-
the-shelf model to denoise the target audio
333https://github.com/timsainb/noisereduce; “w/ random test cond": we use
random audio from the training set as the condition instead of retrieving
audio with the highest AV-Sim score.

| FAD $\downarrow$ | AV-Sync (%)$\uparrow$ | CLAP$\uparrow$
---|---|---|---
Ground Truth (Upper Bound) | 0.0000 | 77.69 | 0.2698
Retrieval | 1.8353 | 11.84 | 0.0335
Spec-VQGAN [25] | 3.9017 | 7.12 | 0.0140
Diff-Foley [38] | 3.5608 | 5.98 | 0.0346
Ours w/o vocoder | 4.9282 | 29.60 | 0.1319
Ours w/o cond + denoiser | 1.4676 | 1.09 | 0.0009
Ours w/o cond | 1.4681 | 39.63 | 0.1418
Ours w/ random test cond | 1.0635 | 28.74 | 0.1278
AV-LDM (Ours) | 0.9999 | 45.74 | 0.1435
Table 2: Results on Ego4D-Sounds test set. We also report the performance of
the ground truth audio, which gives the upper bound value for each metric.
Figure 6: Qualitative example. We show the frames of each video followed by
the waveform/spectrogram of various baseline methods. Our model generates the
most synchronized sounds.

###  5.2 Results on Ego4D-Sounds

In this section, we evaluate the ambient-sound joint generation setting with
retrieval augmented generation. The results are shown in Tab. 2. Compared to
all three baselines, we outperform them on all three metrics by a large
margin. While the Retrieval baseline retrieves natural sounds from the
training set and has a low FAD score compared to Spec-VQGAN and Diff-Foley,
both its AV-Sync accuracy and CLAP scores are very low. Diff-Foley has a
higher performance than Spec-VQGAN since it has been trained on this task, but
it still largely underperforms our model w/o cond, likely because their video
features do not generalize to the egocentric setting well.

For ablations, “Ours w/o cond" has a much worse FAD score compared to the full
model, showing the importance of our ambient-aware training. As expected,
“Ours w/o cond + denoiser" has very low scores on AV-Sync and CLAP since
existing noise reduction algorithms are far from perfect. We also test our
model by conditioning it on a random audio segment at test time instead of the
one retrieved with the highest audio-visual similarity and its performance
also gets worse, verifying the effectiveness of our retrieval-based solution.

We show two qualitative examples in Fig. 6 comparing our model with several
baselines and we show that our model synthesizes both more synchronized and
more plausible sounds. To fully evaluate our results, it is important to view
the supplementary video.

(a) Varying ambient level condition

(b) Audio generation accuracy (FAD)

Figure 7: The achieved ambient level and accuracy of the prediction as a
function of the input ambient levels. (a): we show the ambient level of our
model changes according to the ambient level in the audio condition while the
ambient level of “Ours w/o cond" and the original audio stay constant,
illustrating the controllability of our model. (b) FAD is low for most input
ambient levels unless it goes too extreme (too low or too high), showing our
model generates high-quality action sounds even when varying output ambient
levels.

###  5.3 Ambient Sound Control

By disentangling action sounds from ambient sounds, our model allows taking
any given sound as the condition at test time. To examine whether our model
truly relies on the audio condition to learn the ambient sound information, we
test the model by providing audio conditions of various ambient levels and
then calculate the ambient level in the generated audio. The ambient level is
defined as the lowest energy of any 0.5s audio segment in a 3s audio.

The results are shown in Fig. 7, where we also plot the ambient levels of
“Ours w/o cond" and the original audio. Our model changes the ambient sound
level according to the input ambient (shown in Fig. 7(a)) while still
synthesizing plausible action sounds (shown in Fig. 7(b)). FAD spikes when the
condition ambient is too low or too high, most likely because the generated
ambient sound is out of distribution since the original audio always has some
ambient sounds.

Fig. 8 shows example outputs from our model and several baselines. The
examples show how our model generates plausible action sounds when conditioned
on a low-ambient sound for action-focused generation. We can see that the
action-focused setting generates similar action sounds as the action-ambient
setting while having a minimal ambient level. While by definition we lack a
good evaluation of this setting (there is no ground truth audio source
separation for the data), our model shows an emerging capability of generating
clean action sounds although it has never been explicitly trained to do so.

Figure 8: Visualization of action-focused generation. For both examples, Diff-
Foley [38], Ours w/o cond or Ours (action-ambient generation) generate
plausible action sounds along with ambient sounds. In contrast, our model
conditioned on a low ambient sound generates plausible action sounds (see
green boxes) with minimal ambient sound.

###  5.4 Human Evaluation

To further validate the performance of various models, we conduct a subjective
human evaluation. In each survey, we provide 30 questions and each question
has 5 videos with the same visuals but different audio samples. For each
video, we ask the participant to select the video(s) whose audio 1) is most
semantically plausible and temporally synchronized with the video and 2) has
the least ambient sounds. We invite 20 participants to complete the survey and
compute the average voting for all 30 examples. See the survey interface and
guidelines in Supp.

Tab. 3 shows the results. Overall, all learning-based methods generate
reasonable action sounds while our model (action-ambient) has the highest
score for action-sound quality compared to other methods. Although ours
(action-focused) has a slightly lower action-sound score, it has significantly
less ambient sound. This is likely because sometimes the low-ambient condition
can lead the model to suppress some minor action sounds.

| Action sound quality | Least ambient sound
---|---|---
Retrieval | 12.5% | 12.5%
Diff-Foley [38] | 47.5% | 12.5%
AV-LDM w/o cond | 55.0% | 17.5%
AV-LDM (action-focused) | 60.0% | 97.5%
AV-LDM (action-ambient) | 72.5% | 22.5%
Table 3: Survey results showing user preferences. Higher is better. Our model
in the action-ambient joint generation setting scores highest for action sound
quality, showing its ability to produce action-relevant sounds despite
training with in-the-wild data. Ours in the action-focused generation setting
scores highest for the least ambient sound, at a slight drop in action sound
quality score, showing the ability to eliminate background sounds when
requested by the user.

###  5.5 Results on EPIC-KITCHENS

To evaluate whether our model generalizes to other datasets, we also test our
model on the EPIC-KITCHENS dataset. We first sample 1000 3s clips on EPIC-
KITCHENS and then evaluate the retrieval baseline, Diff-Foley, Ours w/o cond,
and our full model on these data and then compute the FAD and AV-Sync scores
for them.

Results are shown in Tab. 4. Similar to what we observe on Ego4D-Sounds, our
model outperforms other models on FAD and AV-Sync by a large margin, showing
ours learns better to generate action sounds from visuals, which also transfer
to other datasets.

| GT | Retrieval | Diff-Foley | Ours w/o cond | AV-LDM (Ours)
---|---|---|---|---|---
FAD $\downarrow$ | 0.0000 | 1.9618 | 3.4649 | 1.4731 | 1.3200
AV-Sync (%) $\uparrow$ | 73.94 | 13.84 | 14.19 | 50.42 | 59.26
Table 4: Results on Epic-Kichens. GT stands for Ground Truth.

###  5.6 Demo on VR Cooking Game

One compelling application of action-to-sound generation is to generate sound
effects for games in virtual reality, where simulating complex hand-object
interactions is non-trivial. To examine whether our learned model generalizes
to VR games, we collect game videos of a cooking VR game “Clash Of Chefs” from
YouTube and test our model without fine-tuning. Preliminary results suggest
our model can generate synced action sounds (see Fig. 9 and Supp). This
suggests promising future in learning action-to-sound models from real-world
egocentric videos and applying them to VR games to give a game user an
immersive audio-visual experience that dynamically adjusts to their own
actions.

Figure 9: We apply our model on a VR cooking game clip where the person cuts a
sushi roll three times. Our model successfully predicts the 3 cutting sounds.

##  6 Conclusion

We investigate the problem of generating sounds for human actions in
egocentric videos. We propose an ambient-aware approach that disentangles the
action sound from the ambient sound, allowing successful generation after
training with diverse in-the-wild data, as well as controllable conditioning
on ambient sound levels. We show that our model outperforms existing methods
and baselines—both quantitatively and through human subject studies. Overall,
it significantly broadens the scope of relevant training sources for achieving
action-precise sound generation. In future work we aim to explore the
possibilities for sim2real translation of our learned audio generation models
to synthetic imagery inputs, e.g., for VR game applications.

#### Acknowledgments:

UT Austin is supported in part by the IFML NSF AI Institute. Wei-Ning Hsu
helped advise the project only, and all the work and data processing were done
outside of Meta.

## References

  * [1] Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: Proceedings of the 38th International Conference on Machine Learning. pp. 813–824 (2021)
  * [2] Blattmann, A., Rombach, R., Oktay, K., Ommer, B.: Retrieval-augmented diffusion models. ArXiv abs/2204.11824 (2022), https://api.semanticscholar.org/CorpusID:248377386
  * [3] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J.B., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T.W., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J.W., Elsen, E., Sifre, L.: Improving language models by retrieving from trillions of tokens. In: International Conference on Machine Learning (2021), https://api.semanticscholar.org/CorpusID:244954723
  * [4] Chen, C., Ashutosh, K., Girdhar, R., Harwath, D., Grauman, K.: Soundingactions: Learning how actions sound from narrated egocentric videos. In: CVPR (2024)
  * [5] Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: ICASSP (2020)
  * [6] Chen, W., Hu, H., Saharia, C., Cohen, W.W.: Re-imagen: Retrieval-augmented text-to-image generator. ArXiv abs/2209.14491 (2022), https://api.semanticscholar.org/CorpusID:252596087
  * [7] Clarke, S., Gao, R., Wang, M., Rau, M., Xu, J., Wang, J.H., James, D.L., Wu, J.: Realimpact: A dataset of impact sound fields for real objects. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (2023)
  * [8] Clarke, S., Heravi, N., Rau, M., Gao, R., Wu, J., James, D., Bohg, J.: Diffimpact: Differentiable rendering and identification of impact sounds. In: 5th Annual Conference on Robot Learning (2021)
  * [9] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T.: Scaling egocentric vision: The epic-kitchens dataset. In: ECCV (2018)
  * [10] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. ArXiv abs/2105.05233 (2021), https://api.semanticscholar.org/CorpusID:234357997
  * [11] Du, Y., Chen, Z., Salamon, J., Russell, B., Owens, A.: Conditional generation of audio from video via foley analogies. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2426–2436 (2023)
  * [12] Fabian Caba Heilbron, Victor Escorcia, B.G., Niebles, J.C.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 961–970 (2015)
  * [13] Gan, C., Schwartz, J., Alter, S., Mrowca, D., Schrimpf, M., Traer, J., De Freitas, J., Kubilius, J., Bhandwaldar, A., Haber, N., Sano, M., Kim, K., Wang, E., Lingelbach, M., Curtis, A., Feigelis, K., Bear, D.M., Gutfreund, D., Cox, D., Torralba, A., DiCarlo, J.J., Tenenbaum, J.B., McDermott, J.H., Yamins, D.L.K.: Threedworld: A platform for interactive multi-modal physical simulation. In: NeurIPS Datasets and Benchmarks Track (2021)
  * [14] Gandhi, D., Gupta, A., Pinto, L.: Swoosh! rattle! thump! - actions that sound. In: RSS (2022)
  * [15] Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 776–780 (2017)
  * [16] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: CVPR (2023)
  * [17] Gong, Y., Chung, Y.A., Glass, J.: Ast: Audio spectrogram transformer. In: InterSpeech (2021)
  * [18] Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., Martin, M., Nagarajan, T., Radosavovic, I., Ramakrishnan, S.K., Ryan, F., Sharma, J., Wray, M., Xu, M., Xu, E.Z., Zhao, C., Bansal, S., Batra, D., Cartillier, V., Crane, S., Do, T., Doulaty, M., Erapalli, A., Feichtenhofer, C., Fragomeni, A., Fu, Q., Gebreselasie, A., Gonzalez, C., Hillis, J., Huang, X., Huang, Y., Jia, W., Khoo, W., Kolar, J., Kottur, S., Kumar, A., Landini, F., Li, C., Li, Y., Li, Z., Mangalam, K., Modhugu, R., Munro, J., Murrell, T., Nishiyasu, T., Price, W., Puentes, P.R., Ramazanova, M., Sari, L., Somasundaram, K., Southerland, A., Sugano, Y., Tao, R., Vo, M., Wang, Y., Wu, X., Yagi, T., Zhao, Z., Zhu, Y., Arbelaez, P., Crandall, D., Damen, D., Farinella, G.M., Fuegen, C., Ghanem, B., Ithapu, V.K., Jawahar, C.V., Joo, H., Kitani, K., Li, H., Newcombe, R., Oliva, A., Park, H.S., Rehg, J.M., Sato, Y., Shi, J., Shou, M.Z., Torralba, A., Torresani, L., Yan, M., Malik, J.: Ego4d: Around the world in 3,000 hours of egocentric video. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18973–18990 (2022)
  * [19] Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.W.: Realm: Retrieval-augmented language model pre-training. ArXiv abs/2002.08909 (2020), https://api.semanticscholar.org/CorpusID:211204736
  * [20] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
  * [21] Ho, J., Salimans, T.: Classifier-free diffusion guidance (2022)
  * [22] Huang, C., Tian, Y., Kumar, A., Xu, C.: Egocentric audio-visual object localization. In: CVPR (2023)
  * [23] Huang, R., Huang, J.B., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., Zhao, Z.: Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. ArXiv abs/2301.12661 (2023), https://api.semanticscholar.org/CorpusID:256390046
  * [24] Huh, J., Chalk, J., Kazakos, E., Damen, D., Zisserman, A.: Epic-sounds: A large-scale dataset of actions that sound. In: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5 (2023)
  * [25] Iashin, V., Rahtu, E.: Taming visually guided sound generation. In: BMVC (2021)
  * [26] Jiang, H., Murdock, C., Ithapu, V.K.: Egocentric deep multi-channel audio-visual active speaker localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10544–10552 (2022)
  * [27] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., Suleyman, M., Zisserman, A.: The kinetics human action video dataset. CoRR abs/1705.06950 (2017), http://arxiv.org/abs/1705.06950
  * [28] Kazakos, E., Nagrani, A., Zisserman, A., Damen, D.: Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5491–5500 (2019)
  * [29] Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., Lewis, M.: Generalization through memorization: Nearest neighbor language models. ArXiv abs/1911.00172 (2019), https://api.semanticscholar.org/CorpusID:207870430
  * [30] Kilgour, K., Zuluaga, M., Roblek, D., Sharifi, M.: Fréchet audio distance: A metric for evaluating music enhancement algorithms. arxiv (2018)
  * [31] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  * [32] Kong, J., Kim, J., Bae, J.: Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems 33, 17022–17033 (2020)
  * [33] Kong, Z., Ping, W., Huang, J., Zhao, K., Catanzaro, B.: Diffwave: A versatile diffusion model for audio synthesis. ArXiv abs/2009.09761 (2020), https://api.semanticscholar.org/CorpusID:221818900
  * [34] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv abs/2005.11401 (2020), https://api.semanticscholar.org/CorpusID:218869575
  * [35] Lin, K.Q., Wang, J., Soldan, M., Wray, M., Yan, R., Xu, Z., Gao, D., Tu, R.C., Zhao, W., Kong, W., Cai, C., HongFa, W., Damen, D., Ghanem, B., Liu, W., Shou, M.Z.: Egocentric video-language pretraining. In: Advances in Neural Information Processing Systems (2022)
  * [36] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D.P., Wang, W., Plumbley, M..: Audioldm: Text-to-audio generation with latent diffusion models. In: International Conference on Machine Learning (2023), https://api.semanticscholar.org/CorpusID:256390486
  * [37] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., Zhu, J.: Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 (2022)
  * [38] Luo, S., Yan, C., Hu, C., Zhao, H.: Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. In: NeurIPS (2023)
  * [39] Mittal, H., Morgado, P., Jain, U., Gupta, A.: Learning state-aware visual representations from audible interactions. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022), https://openreview.net/forum?id=AhbTKBlM7X
  * [40] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In: International Conference on Machine Learning (2021), https://api.semanticscholar.org/CorpusID:245335086
  * [41] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [42] Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E.H., Freeman, W.T.: Visually indicated sounds. In: CVPR (2016)
  * [43] Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., Kudinov, M.A.: Grad-tts: A diffusion probabilistic model for text-to-speech. In: International Conference on Machine Learning (2021), https://api.semanticscholar.org/CorpusID:234483016
  * [44] Ramazanova, M., Escorcia, V., Heilbron, F.C., Zhao, C., Ghanem, B.: Owl (observe, watch, listen): Localizing actions in egocentric video via audiovisual temporal context (2022)
  * [45] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding. ArXiv abs/2205.11487 (2022), https://api.semanticscholar.org/CorpusID:248986576
  * [46] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. In: Neural Information Processing Systems (2019), https://api.semanticscholar.org/CorpusID:196470871
  * [47] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. CoRR (2012)
  * [48] Su, K., Qian, K., Shlizerman, E., Torralba, A., Gan, C.: Physics-driven diffusion models for impact sound synthesis from videos. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 9749–9759 (2023), https://api.semanticscholar.org/CorpusID:257805229
  * [49] Wang, D., Chen, J.: Supervised speech separation based on deep learning: An overview. arxiv (201)
  * [50] Wu*, Y., Chen*, K., Zhang*, T., Hui*, Y., Berg-Kirkpatrick, T., Dubnov, S.: Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP (2023)
  * [51] Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., Yu, D.: Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 1720–1733 (2022), https://api.semanticscholar.org/CorpusID:250698823

##  7 Supplementary

In this supplementary material, we provide additional details about:

  1. 1.

Supplementary video for qualitative examples (referenced in Sec. 1).

  2. 2.

Additional implementation details (referenced in Sec. 3).

  3. 3.

Dataset details (referenced in Sec. 4).

  4. 4.

Evaluation metric details (referenced in Sec. 5).

  5. 5.

Human evaluation details (referenced in Sec. 5).

###  7.1 Supplementary Video

In this video, we include examples of Ego4D-Sounds clips, qualitative examples
on unseen Ego4D clips, and qualitative examples on VR games. Wear headphones
to hear the sound.

###  7.2 Additional Implementation Details

Audio-Visual LDM. Our Ego4D-Sounds clips are 3 seconds long. For model
training and inference, we sample audio waveform at 16000Hz. We use FFT size
1024, mel bins 128, hop size 256 to transform the 3-second audio waveform into
a mel-spectrogram of length 188, which we then pad in the temporal dimension
to $192$. To speed up training, similar to [38], we load VAE and diffusion
model weights from the pre-trained Stable Diffusion model. Note that Stable
Diffusion expect image as the input/target, and therefore we duplicate the
mel-spectrogram in the channel dimension and to achieve size
$x_{0}\in\mathbb{R}^{3\times 128\times 192}$, passing $x_{0}$ to the VAE
encoder, we get compressed latent representation $z_{0}\in\mathbb{R}^{4\times
16\times 24}$. For conditioning, videos are sampled at 5 FPS, passed through
the video encoder and a linear projection layer that produces features of size
$c_{v}\in\mathbb{R}^{16\times 768}$. Audio condition is also a 3-second clip
and is processed the same way as the target audio to get
$\tilde{c}_{a}\in\mathbb{R}^{4\times 16\times 24}$, it is then projected to a
$2$ dimensional tensor of shape $c_{a}\in\mathbb{R}^{24\times 768}$. For
classifier-free guidance, we set the scale $\omega=6.5$, and use DPM-Solver
[37] for accelerated inference using only $25$ sampling steps. For the mel-
spectrogram to waveform vocoder HiFi-GAN [32], we train the model from scratch
with the mel-spectrogram processing hyperparameters aligned with that of our
AV-LDM. During training, we freeze the pre-trained VAE, and train the LDM
model on Ego4D-Sounds for 8 epochs with batch size 720. We use the AdamW
optimizer with a learning rate of $1e-4$. HiFi-GAN is trained on a combination
of 0.5s clips from Ego4D[18], Epic-Kitchens [24], and AudioSet [15]. We use
AdamW to train HiFi-GAN with a learning rate of $2e-4$ and a batch size of 64
for 120k steps.

Audio-visual representation learning. We use Timesformer [1] as the video
encoder, and AST [17] as the audio encoder. We train video and audio encoders
for 5 epochs with batch size $256$. We use the InfoNCE [41] loss and Adam
optimizer [31] with a learning rate $1e-4$.

###  7.3 Dataset Details

To evaluate the effectiveness of our proposed ambient-aware action sound
generation model, we leverage Ego4D [18], a large-scale egocentric video
dataset for daily human activities. While our model is capable of
disentangling action sound from ambient sound, there is little value in
learning on data that only contain ambient sounds or background speech. Our
goal is to curate an in-the-wild action dataset that has meaningful action
sounds. We design a four-stage pipeline consisting of both learning-based
tagging tools and rule-based filters to curate the Ego4D-Sounds dataset. To be
consistent with the public splits of Ego4D benchmarks, we use all 7.5K videos
in the training set, where we extract 3.8M clips centered at the narrations’
timestamps with the left and right margins being 1.5s, i.e. each clip is 3s
long, which we find to be sufficiently long enough to capture the narrated
action.

We first remove all clips without sounds, resulting in 2.5M clips. We then
filter the above clips based on the scenarios. Each Ego4D video has a scenario
label, categorizing the activity depicted in the video. We go through all
scenario labels and pick 28 scenarios that are mainly social scenarios, e.g.,
"playing board games", "attending a party", "talking with friends", where
majority of the sounds are speech or off-screen sounds with no on-screen
actions. We remove videos with these tags, resulting in 3.1K videos and 1.7M
clips.

While the previous stage removed videos for social scenarios as a whole, there
are still many clips that have only speech or background music. To detect
these clips, we use an off-the-shelf audio tagging tool to tag the remaining
clips. The goal is to remove clips that have solely off-screen sounds, i.e.,
speech and music. So we threshold the tagged probability at $0.5$, i.e.,
removing clips that most likely only contain off-screen sounds and not action
sounds. This filtering process further removed 0.5M clips, with 1.23M clips
remaining.

Lastly, we also observe that in a long video clip, there are silent periods
when no sounding action occurs. Based on this observation, we devise an
energy-based filtering process, i.e. we normalize the amplitude of each clip
with respect to the maximum amplitude of audio in the video. We then convert
the amplitude to dB and remove clips with energy below -60 dB. This results in
1.18M clips.

###  7.4 Evaluation Metric Details

For the audio-visual synchronization (AV-Sync) [38] metric, we train a
synchronization binary classification model on Ego4D-Sounds, and use it to
judge whether the generated audio is synchronized with the video. Following
[38], to construct the input to the classification model, we input paired and
synced video and audio, unpaired video and audio, and paired but unsynced
video and audio 50%, 25%, and 25% of the time respectively. The model uses
Timesformer [1] as the video encoder, AST [17] as the audio encoder, and a 3
layer MLP as the classification head which takes the CLS tokens from the two
encoders, concatenates them in the feature dimension, and produces a
probability indicating the synchronization. We train this model for 30k steps
with AdamW optimizer, which achieves a classification accuracy of 70.6% on the
validation set.

###  7.5 Human Evaluation Details

For the human evaluation, we first compile a guideline document, clarifying
and defining the objectives of the survey and what the participant should be
looking for. There are two main objectives: 1) select video(s) with the most
plausible action sounds (e.g., object collisions, water running) that are
semantically and temporally matching with the visual frames, and 2) select
video(s) with the least ambient noise. We also provide multiple positive and
negative examples for each criterion in the guideline document. We ask
participants to read the guidelines before doing the survey. We show one
example of the survey interface in Fig. 10.

Figure 10: Survey interface. For each example, we ask participants to select
video(s) that are most plausible with the video and the video(s) that have the
least ambient sound.


# A Novel Actor Dual-Critic Model for Remote Sensing Image Captioning

Ruchika Chavhan 1, Biplab Banerjee1, Xiao Xiang Zhu2, Subhasis Chaudhuri1  1
Indian Institute of Technology Bombay, India
2Signal Processing in Earth Observation, Technical University of Munich,
Germany

###### Abstract

We deal with the problem of generating textual captions from optical remote
sensing (RS) images using the notion of deep reinforcement learning. Due to
the high inter-class similarity in reference sentences describing remote
sensing data, jointly encoding the sentences and images encourages prediction
of captions that are semantically more precise than the ground truth in many
cases. To this end, we introduce an Actor Dual-Critic training strategy where
a second critic model is deployed in the form of an encoder-decoder RNN to
encode the latent information corresponding to the original and generated
captions. While all actor-critic methods use an actor to predict sentences for
an image and a critic to provide rewards, our proposed encoder-decoder RNN
guarantees high-level comprehension of images by sentence-to-image
translation. We observe that the proposed model generates sentences on the
test data highly similar to the ground truth and is successful in generating
even better captions in many critical cases. Extensive experiments on the
benchmark Remote Sensing Image Captioning Dataset (RSICD) and the UCM-captions
dataset confirm the superiority of the proposed approach in comparison to the
previous state-of-the-art where we obtain a gain of sharp increments in both
the ROUGE-L and CIDEr measures.

##  I Introduction

Image Captioning is the task of generating a natural language description for
a given image. Describing the images in properly framed sentences is a task
that humans can perform with utmost ease [1], but was an inconceivable task
for computers before the advent of deep learning. The primary requirements to
automatically generate a caption are capturing the essence of the image and
describing the correlations between objects to localize them in the image.
Image captioning is more difficult than other tasks [2] that the computer
vision community has concentrated on, as it involves the integration of
semantic interpretation of an image into textual explanations.

Image captioning data is derived from different modalities: images that are
represented by pixel intensities and textual descriptions represented as
discrete word count vectors. The center of image captioning is recognizing the
collective interpretation of these various modalities. The problem setting
requires studying an abstract interpretation of the contents of the image and
the semantic associations between the objects that can be constructed as a
natural language description. Therefore, the problem of image captioning was
defined as: Given an image $I$, a model is trained to maximize the likelihood
$p(W|I)$ where $W={w_{1},w_{2},w_{3},...w_{n}}$ where all $w_{i}$ are words
from a pre-defined vocabulary. One of the first methods to describe images in
human interpretable language [3] employed a Convolutional Neural Network (CNN)
as an encoder to extract meaningful features from an image in the form of a
fixed vector and a recurrent neural network (RNN) as a decoder to generate
sentences. The primary inspiration of this model arises from the intuition
that images can be translated to sentences, similar to the problem of machine
translation [4] which follows an encoder-decoder architecture to translate the
source sentence into target format through a bottleneck latent space. The goal
of subsequent works in image captioning [5, 6] has been to produce more
diverse captions that also accurately represent the image content. The aim of
most image captioning works is to produce multiple contextual explanations
that novelly manifest the local associations of objects in the image.

Reinforcement Learning [7] is a domain of Machine Learning that enables an
agent to explore an environment by performing an action determined by a
policy. While Deep Learning offers the best set of models for learning
representations of multi-modal data, Reinforcement Learning is a framework for
learning sequential decision-making tasks. Therefore, RL is the mainstream
algorithm used to solve complex environments in different games to achieve
super-human performance[8], robotics[9], and recommendation systems. Most RL
agents acquire a stochastic mapping of states and actions, called the policy
and pursue a trajectory by carrying out actions determined by this policy to
maximize the total expected return in a given time phase. A model-free RL
agent does not pre-specify a structural model of the environment, instead, it
gradually learns the best policies based on trial and error and adequate
exploration of the environment [10]. Most environments consist of multi-modal
data and a single joint representation is learnt by an RL agent as a decision-
making framework.

All reinforcement learning-based approaches are an exploration-based approach
in which the agent first collects knowledge about the entire environment and
state-space to anticipate reward-maximizing behavior. The agent is able to
make highly optimized decisions in that way. Therefore, addressing the task of
image captioning in the Reinforcement learning domain enables the generation
of semantically more coherent captions. All supervised learning methods aim to
generate sentences that are exactly identical to ground truth, while policies
trained in an RL setup allow one to predict sentences much better than ground
truth. Policy-generated natural language descriptions can be dramatically
enhanced by increasing the degree of environmental analysis consisting of
paired images and captions.

Recently, acquisition of an unprecedented volume of satellite images from
different sensors is observed. The major product derived from the satellite
images has been the land-cover maps which assigns semantic class-labels at the
pixel locations. However, the task of pixel-labeling is redundant for some
emergency applications where an overall description of the scene under
consideration is encouraged. Under this premise, the task of generating
automatic captions for satellite scenes holds much potential. However, there
does not exist extensive literature in this front specifically for remote
sensing data.

In this paper, we implement an Actor Dual-Critical (ADC) training setup to
address the issue of high inter-class image and caption similarity in
satellite data. We have performed our proposed experiment on the Remote
Sensing Image Captioning Dataset (RSICD) [11] and the UCM-captions dataset
[12, 13]. The key problem in the datasets for many images is the strong inter-
class similarity and the identical reference sentences. The contributions and
results of the paper are summarized as follows:

  * •

To the best of our knowledge, our approach is the first to use Reinforcement
Learning to produce captions on remote sensing images. Unlike all existing
methods, we employ an additional encoder-decoder RNN as a critic for Actor
Dual-Critic (ADC) training setup. This critic plays a key role in creating a
variety of different sentences that represent identical visual perception.

  * •

The captions predicted by our proposed model are more semantically related to
the objects in the image, explicitly describe object localization, and
specifically focus on the existence of the entities in the image.

  * •

We also perform cross-dataset captioning and obtain superior results on the
RSICD dataset by models trained on the UCM-captions data set to demonstrate
the generalization capability of a policy trained using the ADC setup as
compared to previous state-of-the-art methods.

##  II Related Work

Actor-critic methods in the Reinforcement Learning aim to train the
participant in choosing actions taking into account the critic’s reward. In
the context of image captioning implemented in an actor-critic training
strategy [14], given an image $I$ and partially generated sentence
$S=(w_{1},w_{2},...,w_{t})$ , $w_{t+1}$ is viewed as an action that the policy
predicts. An actor’s job is to learn a policy $\pi(a_{t}|s_{t-1})$, where
$a_{t}$ is the action performed and $s_{t}$ is the state at time $t$. The
critic provides the value function $v_{\theta}^{\pi}$, where $\theta$ denotes
the parameters of the value network. The reward metric for the generated
caption is the score evaluated by ROUGE-L and $v_{\theta}^{\pi}$ is used as an
advantage baseline for the advantage factor
$A^{\pi}(s_{t}|a_{t+1})=(\gamma^{T-t-1}r_{T}-v_{\theta}^{\pi})$. Subsequently,
the actor is trained to optimize expected value of reward over the trajectory
using a cross-entropy loss and the advantage factor using the REINFORCE
Algorithm where gradient are given by
$\mathbf{E}[\sum_{t=0}^{T}A^{\pi}(s_{t}|a_{t+1})\nabla\log\pi(a_{t}|s_{t-1})]$.
Hence, this training method is named Advantage-Actor Critic (A2C). Instead of
calculating a value function for baseline, the test-time reward metric can
also be used a baseline to normalize the rewards experienced during training,
thus eliminating the need for a critic, making the training setup self-
critical [15, 16].

Recent work on remote sensing image captioning [12, 17] involves using deep
multi-modal networks and analyzing the quality of generated captions by
experimenting with various types of CNN, RNN, and LSTM combinations. Similar
experiments were performed on the RSICD dataset [11] to produce accurate
captions. Because of the wide area of the Earth’s surface covered by remote
sensing images, the main criteria for caption generation are to recognize
semantic uncertainty in remote sensing images by analyzing key instances [18]
and performing image context and landscape analysis.

(a) Working of the actor. The words are converted into the embedding space
(not shown) before being fed into the GRU (denoted by blue) and LSTM (denoted
by grey).

(b) Working of the proposed Encoder-Decoder RNN critic. The outputs and the
hidden states of the encoder RNN are transformed by $\psi_{1}$ and $\psi_{2}$
(not shown) and serve as input to the decoder RNN.

Figure 1: Working Principle of the proposed ADC setup

##  III Proposed Methodology

We first discuss the working principles of our proposed Actor Dual-Critic
(ADC) training and elaborate on the three components employed in the process.
We formulate the problem statement as follows: Given an image $I$, the model
is expected to generate a natural language description
$S=(w_{1},w_{2},...,w_{n})$ where $w_{i}$ is a word from a pre-defined
vocabulary. The actor generates a sentence given the image and the critics
provide rewards based on the quality and relevance of this sentence. We
utilize two critics: an RNN critic and an encoder-decoder RNN critic to
provide two rewards to update parameters of the actor using the REINFORCE
Algorithm. The training algorithm is mentioned in Algorithm 1.

Learning the Policy: We propose a three-component model for the actor to learn
a policy. The actor provides a measure of confidence $q_{\pi}(a_{t}|s_{t})$ to
predict the next action $\mathbf{a_{t}}=\mathbf{w_{t+1}}\in\mathbb{R}^{d}$
according to the current state. For the extraction of features from an image,
we utilize a pre-trained CNN. The extracted features
$\mathbf{f}\in\mathbb{R}^{n}$ are passed as an input to a Gated Recurrent Unit
(GRU). The hidden state of GRU $\mathbf{h_{t}^{g}}\in\mathbb{R}^{n}$ evolves
with time as the predicted words $w_{t-1}$ are fed into it. The GRU acts
indirectly as the generator of a context vector [19] to assist the decoder to
surmount difficulties in learning due longer sentences. We observed a
significant increase in performance by replacing the normal LSTM with a Layer
Norm LSTM [20] with dropout. The following equations explain the functionality
of the actor:

$\displaystyle f=W_{x}(\text{CNN}(I))$ (1) $\displaystyle\phi_{0}=f$
$\displaystyle o_{t}^{g},h_{t}^{g}=\text{GRU}(\phi_{t-1},h_{t-1}^{g})$
$\displaystyle o_{t}^{l},h_{t}^{l}=\text{LSTM}(o_{t}^{g},h_{t-1}^{l})$
$\displaystyle q_{\pi}(a_{t}|s_{t})=\psi({o_{t}^{l}})$
$\displaystyle\phi_{t}=\zeta(w_{t-1})$

Here, $W_{x}$ is the weight of the linear embedding model of the CNN,
$o_{t}^{g}$ and $o_{t}^{l}$ are the outputs of the GRU and LSTM respectively
at time step $t$. $\mathbf{\psi}\colon\mathbb{R}^{n}\mapsto\mathbb{R}^{d}$ is
a non-linear function that transforms the output of the LSTM to a space whose
dimension is equal to the dimension of the vocabulary.
$\mathbf{\zeta}\colon\mathbb{R}^{d}\mapsto\mathbb{R}^{n}$ denotes the
embedding model to represent words in a common embedding space. We denote the
policy network by $\pi(a_{t}|s_{t-1})$. Please refer to Figure 1(a) for
complete description and pipeline of the model.

The model is trained to optimize the objective:

$\min_{\pi}\sum_{t=0}^{T}\log(q_{\pi}(a_{t}|s_{t}))$ (2)

Value Network: This critic consists of an RNN which outputs a value function
$v_{\theta}^{\pi}(s_{t})$ given the words predicted by the current policy and
features extracted by the CNN in Figure 1(a). We initialize the hidden state
of the RNN by these extracted features. We will denote this critic as
$V(s_{t})$. The output of this network is directed to be the expected value of
future rewards
$\mathbb{E}[\sum_{l=0}^{T-t-1}\gamma^{l}r_{t+l+1}|a_{t+1},...,a_{T}\sim\pi,I]$
for choosing a state $s_{t}$ given the current policy $\pi$. We set $\gamma=1$
similar to [14] and calculate the reward $r_{T}$ after the prediction of the
entire sequence implying $r_{t}=0$, $\forall$ $t<T$.

The reward $r_{T}$ for the entire generated sentence is obtained by using the
evaluation scores of ROUGE-L or BLEU. We observed more stable training using
the Huber Loss instead of the norm of the difference between the reward and
$v_{\theta}$:

$L=\begin{dcases}||v_{\theta}^{\pi}-r_{T}||^{2}&||v_{\theta}^{\pi}-r_{T}||\leq\delta=0.5\\\
\delta||v_{\theta}^{\pi}-r_{T}||-\frac{1}{2}\delta^{2}&\text{otherwise}\\\
\end{dcases}$ (3)

Encoder-Decoder LSTM Critic: The addition of this critic is the main
contribution of this paper. The intuition behind this critic is as follows:

  * •

Theoretically, image captioning is defined as translation of images into
sentences that aptly describe the images. We hypothesize that the sentences
translated back into images should generate a quantity which closely resembles
the features extracted from images.

  * •

This procedure ensures maximum accumulation of information about the image
through a textual description relatively similar to the information captured
by ground truth sentences.

Please refer to Figure 1(b) for the entire pipeline of this model. The working
principle of this critic denoted by $D(S)$ is governed by the following
equations:

$\displaystyle h_{0}^{enc}=W_{x}(\text{CNN}(I))$ (4)
$\displaystyle\eta_{t}=\zeta(S)$ $\displaystyle
o_{t}^{enc},h_{t}^{enc}=\text{RNN}_{enc}(\eta_{t},h_{t-1}^{enc})$
$\displaystyle h_{0}^{dec}=\psi_{2}(h_{T}^{enc})$ $\displaystyle
i_{1}^{dec}=\psi_{1}(o_{T}^{enc})$ $\displaystyle
o_{t}^{dec},h_{t}^{dec}=\text{RNN}_{dec}(i_{t}^{enc},h_{t-1}^{dec})$

Here, RNNenc and RNNdec are the encoder-decoder RNN respectively.
$S=(w_{1},w_{2},...,w_{T})$ denotes a natural language description of the
image. We have used
$\psi_{1},\psi_{2}\colon\mathbb{R}^{n}\mapsto\mathbb{R}^{n}$ as a linear
function with dimension equal to that of the embedding space of sentences,
along with the ReLU activation function.

This critic is trained by optimizing the mean squared error (MSE) between the
output of the decoder and the features:

$L=(\frac{\sum_{t=0}^{T}o_{t}^{dec}}{|S|}-f)^{2}$ (5)

Accuracy for the decoder output is given by the cosine similarity between the
output of the decoder and features:

$A_{gen}=\frac{\frac{\sum_{t=0}^{T}o_{t}^{dec}}{|S|}f}{\left\lVert
f\right\rVert\left\lVert\frac{\sum_{t=0}^{T}o_{t}^{dec}}{|S|}\right\rVert}$
(6)

$A_{gen}$ and $A_{orig}$ are the accuracies of the network when captions
generated by the actor and ground truth captions are fed into the encoder
respectively. We defined an advantage factor for this critic to be:

$A_{ed}=A_{gen}-\delta_{t}A_{orig}$ (7)

The encoder-decoder critic is pre-trained on features extracted and
corresponding original sentences to learn a ground truth latent
representation. We observed that $\delta_{t}=1$ initially leads to exploding
gradients and a non-converging policy. Therefore, we begin with
$\delta_{t}=0.01$ and increase it linearly to 1.0 over the epochs. For
generating captions for a test image, we pass the image $I_{test}$ to the
actor to generate sentences till the $<$end$>$ token is encountered.

##  IV Experiments and Results

In this section, we study the performance of our model and it’s results on two
remote sensing image captioning datasets: RSICD and UCM-captions. We also
visualise the output of our novel critic and analyze the validity of it’s
working principle.

Algorithm 1 Training Algorithm

0: Pre-trained models $\pi(a_{t}|s_{t-1})$, $V(s_{t})$ and $D(S)$ using the
objectives given by the equations 2, 3 and 5 respectively as done in [14].

1: for $episode=1$ to total episodes do

2: Given an Image $I$ sample action $(a_{1},a_{2},...,a_{T})$ from the current
policy using a multinomial distribution given by $q_{\pi}(s_{t}|a_{t})$;

3: Calculate advantage factor $A^{\pi}$ using the reward $r_{T}$ for the value
network;

4: Update the parameters of the policy using $A^{\pi}$ by the REINFORCE
Algorithm;

5: Update parameters of the critic by optimising Eq. 3;

6: Calculate advantage factor $A_{ed}$ using the encoder-decoder critic;

7: Update the parameters of the policy using $A_{ed}$ by the REINFORCE
Algorithm;

8: Update parameters of the critic using $A_{orig}.$

9: end for

TABLE I: Results of ADC setup on the RSICD dataset Metric |  B-1 |  B-2 |  B-3 |  B-4 |  METEOR [21] |  ROUGE-L |  CIDEr[22]
---|---|---|---|---|---|---|---
MM [11] |  0.57905 |  0.41871 |  0.32628 |  0.26552 |  0.26103 |  0.51913 |  2.05261
SA [11] |  0.65638 |  0.51489 |  0.41764 |  0.34464 |  0.32924 |  0.61039 |  1.87415
HA [11] |  0.68968 |  0.5446 |  0.44396 |  0.36895 |  0.33521 |  0.62673 |  1.98312
A2C [14] |  0.60157 |  0.41991 |  0.364516 |  0.28788 |  0.19382 |  0.63185 |  2.098
Ours |  0.73973 |  0.55259 |  0.46353 |  0.41016 |  0.22126 |  0.71311 |  2.243
TABLE II: Results of ADC setup on the UCM dataset Metric |  B-1 |  B-2 |  B-3 |  B-4 |  METEOR [21] |  ROUGE-L |  CIDEr[22]
---|---|---|---|---|---|---|---
MM [11] |  0.37066 |  0.32344 |  0.32346 |  0.23259 |  0.40476 |  0.4236 |  1.708
SA [11] |  0.79693 |  0.71345 |  0.6514 |  0.59895 |  0.74952 |  0.41676 |  2.12846
HA [11] |  0.78498 |  0.70929 |  0.65182 |  0.60167 |  0.77357 |  0.43058 |  2.19594
A2C [14] |  0.373089 |  0.23776 |  0.15857 |  0.12222 |  0.39645 |  0.35989 |  2.381
Ours |  0.85330 |  0.75679 |  0.67854 |  0.61165 |  0.83242 |  0.80872 |  4.865
TABLE III: Results of cross dataset captioning on the RSICD dataset  Metric |  B-1 |  B-2 |  B-3 |  B-4 |  METEOR [21] |  ROUGE-L |  CIDEr[22]
---|---|---|---|---|---|---|---
MM [11] |  0.19618 |  0.01481 |  0.00721 |  0.00445 |  0.07416 |  0.2457 |  0.08015
A2C [14] |  0.19405 |  0.04137 |  0.00714 |  0.00175 |  0.18855 |  0.1846 |  0.961
Ours |  0.38810 |  0.08643 |  0.019065 |  0.00608 |  0.23964 |  0.2888 |  2.013

(a) Original Image with ground truth and predicted sentences. The image is
from the class ’desert’. A test image from the same class but with the same
captions as the test input image are passed to the encoder for this
experiment. The test image used for this case belongs to the class ’desert’.

(b) Reward during Training Process.

Figure 2:

Datasets: The vocabulary of words obtained from the RSICD dataset contains a
collection of 1653 words including the $<$start$>$, $<$unk$>$ and $<$end$>$
tokens. The dataset contains image-caption pairs of 30 classes namely airport,
bare-land, baseball-field, beach, church, commercial, dense-residential,
meadow, river, bridge etc. The dataset contains a total of 10921 images, split
into 8734 training, 1094 validation, and 1093 testing images. The UCM-captions
dataset contains identical captions for images from the same class and thus
spans a vocabulary of only 210 words. It contains 21 classes land use image,
including agricultural, airplane, buildings, chaparral, overpass, parking lot,
river, runway etc and with 100 images for each class. Because of the broad
variety of image classes they provide, the above two datasets are widely used
in remote sensing image captioning research.

Experiment Details: We observed a substantial improvement in performance for
extracting features from images by employing AlexNet [23]. A fully connected
layer with dimension of 256 and batch normalization along with the ReLU
activation function is applied to the output of the feature extractor. We use
an embedding module of dimension 256 to encode each word into an embedding
space. For the value Network, a fully connected layer of dimension $256\times
1$, with hyperbolic tan has been applied as the activation function to obtain
a value function in the range [-1, 1]. We also observed that by employing a
GRU in the encoder-decoder critic instead of an RNN results in a sharp
increment in training speed. The functions $\psi_{1}$ and $\psi_{2}$ in
Equation 4 is a fully connected layer with dimension 256. We used the Adam
Optimiser [24] with a learning rate of 5e-4 and is decreased by factor of 0.9
after every 10 iterations. The networks are trained for a total of 100 epochs.

Metrics for Comparison of results: We compare our results qualitatively and
quantitatively with the [11] who trained a deep multi-modal neural network
(referred to as MM) with different types of CNNs, RNNs, and LSTMs for semantic
understanding of high-resolution remote sensing images in the RSICD and UCM-
captions dataset. The authors also performed experiments on their dataset
using the ”hard” and ”soft” attention mechanisms proposed by [6] denoted by HA
and SA respectively. These three methods are not methods based on
reinforcement learning, except for the mechanism of hard attention which uses
the REINFORCE algorithm for attention but not for the prediction of the
caption. Table I and Table II quantitatively compares the three methods
(denoted by MM, SA, HA respectively) and the results of the A2C training setup
with our proposed method. In Figure 4 and Figure 5, we compare the results of
the A2C training setup replacing their LSTM by a Layer Norm LSTM like in our
ADC training setup. Figure 2(b) shows the values of the reward metrics
ROUGE-L, BLEU-1, BLEU-2, BLEU-3 and BLEU-4 during training on the RSICD
dataset.

Demonstrating validity of proposed critic: The aim of this experiment is to
visualise the relevance of the output of the decoder with respect to the
output of the feature extractor CNN for different image-sentence pairs. To
verify if the critic can distinguish between correct and incorrect captions,
we also input a test image of the same class with a different sentence from
the reference sentence of the ground truth (Figure 2(a)). The figures in this
section are normalized vector representation resized for the sake of
visualization. Figure 3c represents the difference between the output of the
decoder for a reference sentence and the features extracted by the CNN. The
difference between the output of the decoder for the ground truth and
predicted sentence is shown in Figure 3e. As expected, this difference is
negligible (faint grey lines). This means that a generalized high-level
correspondence between images and sentences that defines it semantically has
been learned by this critic. Figure 3g is the difference between the
representations learnt by the encoder-decoder critic for an image from the
same class on a different ground truth caption. As expected, this difference
is high implying that a sentence not describing an image does not get
translated into abstract features corresponding to that image. We deduce that
this critic does not trivially learn the identity function with respect to
input features, due to the presence of non-invertible functions $\psi_{1}$ and
$\psi_{2}$ in Equation 4. It means that the critic takes into account the
correspondences of both image and sentence for two sentences with entirely
different word2vec representations. We may infer from the above experiment
that the critic effectively learns the correlations between images and
reference sentences and encourages the generation of different sentences that
encapsulate the same correlations.

Figure 3: Qualitative results of the experiment comparing output of the
decoder for different image-sentence pairs

(a) A2C: It is a piece of green meadow.
Ours: A dirt lines are in this meadow.

(b) A2C: It is a piece of yellow desert.
Ours: It is a rather flat desert stained with several black stains

(c) A2C: Many rectangular buildings and green trees are in a dense area.
Ours: Houses with red roofs on both sides of the road.

(d) A2C: Many white boats are in the port.
Ours: Two rows of white boats are in port

Figure 4: Qualitative results of image captioning of the ADC setup on the
RSICD dataset

(a) A2C: There are many airports at the airport.
Ours: There is a red airplane with lots of cars.

(b) A2C:There are many airports at the airport.
Ours: There is a red airplane in the airport.

(c) A2C: There are many buildings.
Ours: There is one road next to many buildings.

(d) A2C: There are lots of cars with some buildings.
Ours: Lots of cars are rectangular and close to each other in the parking lot.

Figure 5: Qualitative results of image captioning of the ADC setup on the UCM-
captions dataset

(a) Original: Many tall buildings are in a commercial area.
Ours:There is one road next to many buildings.

(b) Original: Many cars are parked in the parking lot.
Ours: Lots of cars are parked neatly in a parking lot.

(c) Original: Yellow beach is between green ocean and green trees.
Ours: This is a beach with blue sea and white sands.

(d) Original:On the ground, there are two spherical storage tank.
Ours: Two small storage tanks are on the ground.

Figure 6: Qualitative results of cross dataset image captioning of the ADC
setup on the RSICD dataset

Figure 7: Qualitative comparision of results of image captioning of the ADC
setup on the RSICD dataset.

Qualitative Analysis of Results: As observed from the result of the sentence
generation, our proposed method can generate more complicated yet explainable
sentences having longer lengths, more words with low frequency in caption
labels. From the examples in Figure 7, it is evident that the generated
captions contain phrases that provide a highly accurate semantic explanation
of the nature and localization of objects in the scene. We note, however, that
these phrases are absent in the caption of ground truth sentences. We analyzed
ground-truth captions of other images of the same class containing such
phrases to explain the occurence of these phrases, and compared the test image
with the corresponding images in the data set. From Figure 7, it can be
deduced that the addition of an encoder-decoder RNN critic has significantly
improved the quality of the policy’s performance on remote sensing images as
compared to the baseline method (A2C). This demonstrates that the policy has
successfully investigated the environment consisting of images and captions
and has gained more knowledge compared to the baseline approach due to this
critic’s extra upgrade step in the optimization of policy objective. As
observed in Table I, our proposed approach provides a rapid increase in six
out of seven scores used for comparison for the RSICD dataset. However, our
approach proves superior for the UCM-captions dataset than previous methods
for all the seven scores. CIDEr captures consensus-based human judgment better
than established metrics through sentences created from different sources. The
implementation of an encoder-decoder LSTM critic has resulted in strong
increments in CIDEr for both the datasets.

Cross Dataset Captioning: Testing trained models across datasets with similar
domains gives an understanding of the model’s ability to generalize and
utilise it for real time predictions. We have therefore tested a model trained
on the UCM-captions dataset on the RSICD dataset and made qualitative as well
as quantitative comparisons with previous methods [11]. The models trained on
another dataset experience a rapid decline in metrics compared with the
outcome of model trained on the corresponding dataset. From Table III, it is
noted that the inclusion of our proposed critic has resulted in a substantial
improvement in all 7 scores for the captioning of the cross datasets. We also
note from Figure 6 that our policy generates the captions which convey the
same meaning as the sentences of ground truth. This ensures that the policy
trained using the ADC system can be used in Remote Sensing applications in
real life.

##  V Conclusion

In this paper, we proposed an Actor Dual-critic (ADC) method for Image
Captioning for the Remote Sensing Image Captioning Dataset. We are introducing
another critic to the A2C training setup to encourage the prediction of
sentences capturing relevant details along with sentence diversity. In the
sense of converting sentences back to original images we suggest an encoder-
decoder model for this critic. We also proposed a training strategy with an
advantage factor based on a weighted difference of cosine similarities to
update the policy parameters. We prove the superiority of our method
quantitatively using the metrics BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and
CIDEr. Ultimately, we use a model trained over the UCM-captions dataset and
validate its superiority over other approaches to generate textual
descriptions of images in the RSICD dataset.

## References

  * [1] L. Fei-fei, A. Iyer, C. Koch, and P. Perona, “What do we perceive in a glance of a real-world scene?” _Journal of vision_ , vol. 7 1, p. 10, 2007.
  * [2] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [3] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015, pp. 3156–3164.
  * [4] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation.” in _EMNLP_ , A. Moschitti, B. Pang, and W. Daelemans, Eds. ACL, 2014, pp. 1724–1734.
  * [5] A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015, pp. 3128–3137.
  * [6] K. Xu, J. L. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ , ser. ICML’15. JMLR.org, 2015, p. 2048–2057.
  * [7] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. Cambridge, MA, USA: A Bradford Book, 2018.
  * [8] S. D. Holcomb, W. K. Porter, S. V. Ault, G. Mao, and J. Wang, “Overview on deepmind and its alphago zero ai,” in _Proceedings of the 2018 International Conference on Big Data and Education_ , ser. ICBDE ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 67–71. [Online]. Available: https://doi.org/10.1145/3206157.3206174
  * [9] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _J. Mach. Learn. Res._ , vol. 17, no. 1, p. 1334–1373, Jan. 2016.
  * [10] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” in _Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3_ , ser. AAAI’08. AAAI Press, 2008, p. 1433–1438.
  * [11] X. Lu, B. Wang, X. Zheng, and X. Li, “Exploring models and data for remote sensing image caption generation,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 56, no. 4, pp. 2183–2195.
  * [12] B. Qu, X. Li, D. Tao, and X. Lu, “Deep semantic understanding of high resolution remote sensing image,” in _2016 International Conference on Computer, Information and Telecommunication Systems (CITS)_ , 2016, pp. 1–5.
  * [13] Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions for land-use classification,” 01 2010, pp. 270–279.
  * [14] L. Zhang, F. Sung, F. Liu, T. Xiang, S. Gong, Y. Yang, and T. Hospedales, “Actor-critic sequence training for image captioning,” 06 2017.
  * [15] S. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1179–1195, 2017.
  * [16] J. Gao, S. Wang, S. Wang, S. Ma, and W. Gao, “Self-critical n-step training for image captioning,” 04 2019.
  * [17] X. Zhang, Q. Wang, S. Chen, and X. Li, “Multi-scale cropping mechanism for remote sensing image captioning,” in _IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium_ , 2019, pp. 10 039–10 042.
  * [18] Z. Shi and Z. Zou, “Can a machine generate humanlike language descriptions for a remote sensing image?” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 55, no. 6, pp. 3623–3634, June 2017.
  * [19] Y. Bengio and Y. LeCun, Eds., _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015. [Online]. Available: https://iclr.cc/archive/www/doku.php%3Fid=iclr2015:accepted-main.html
  * [20] J. Ba, J. Kiros, and G. Hinton, “Layer normalization,” 07 2016.
  * [21] A. Lavie and A. Agarwal, “Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments,” in _Proceedings of the Second Workshop on Statistical Machine Translation_ , ser. StatMT ’07. USA: Association for Computational Linguistics, 2007, p. 228–231.
  * [22] R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4566–4575, 2014.
  * [23] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems 25_ , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105. [Online]. Available: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
  * [24] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations_ , 12 2014.