# Towards Domain-Independent Supervised Discourse Parsing Through Gradient
Boosting

Patrick Huber and Giuseppe Carenini
Department of Computer Science
University of British Columbia
Vancouver, BC, Canada, V6T 1Z4
{huberpat<EMAIL_ADDRESS>
##  1 Introduction

Discourse analysis and discourse parsing have shown great impact on many
important problems in the field of Natural Language Processing (NLP) (e.g., Ji
and Smith (2017); Bhatia et al. (2015); Nejat et al. (2017); Gerani et al.
(2014)). Given the direct impact of discourse annotations on model performance
and interpretability, robustly extracting discourse structures from arbitrary
documents is a key task to further improve computational models in NLP. To
this end, a variety of complementary discourse theories have been proposed in
the past, such as the lexicalized discourse framework Webber et al. (2003),
the Segmented Discourse Representation Theory (SDRT) Asher (1993); Asher et
al. (2003), and the Rhetorical Structure Theory (RST) Mann and Thompson
(1988), with RST focusing on the semantic and pragmatic structure of complete
monologue documents, as used in this work.

Figure 1: Schematic of our proposed gradient boosted discourse parsing model.
Grey=Inputs/Outputs, Green=Gradient Boosting component,
Red/Orange/Yellow=Trained models. A12 refers to the second shift-reduce action
of the first step. Please note that the models are independent and only
connected by the aggregation in green.

Despite the importance of discourse analysis and discourse parsing for the
field of NLP and the obvious value of the RST discourse theory for many
downstream applications, one major limitation for a wider application of
discourse information is the severe data sparsity issue (for instance, the
popular RST-DT Carlson et al. (2002) and GUM Zeldes (2017) treebanks do not
exceed a minuscule number of 400 documents). Furthermore, while the data
sparsity issue has been a long-standing problem, modern, data-intensive
machine learning approaches further reinforce its severity.

In general, three modelling alternatives have been established in the current
landscape: (i) Supervised approaches (e.g., Ji and Eisenstein (2014); Feng and
Hirst (2014); Joty et al. (2015); Li et al. (2016); Wang et al. (2017); Guz et
al. (2020)), performing well in the domain in which they are trained, however,
obtain severely reduced performance if a domain shift is present, as shown in
Huber and Carenini (2019, 2020). (ii) Distantly supervised models (e.g., Huber
and Carenini (2019, 2020); Nishida and Nakayama (2020); Karimi and Tang
(2019); Liu et al. (2019); Huber et al. (2021); Xiao et al. (2020)), aiming to
overcome the domain adaptation problem by exploiting large-scale supervised
datasets from context-sensitive auxiliary tasks (e.g., sentiment analysis).
(iii) Self-supervised/unsupervised methods (e.g., Zhu et al. (2020); Koto et
al. (2021); Wu et al. (2020); Kobayashi et al. (2019); Huber and Carenini
(2021, 2022)), predicting discourse from either pre-trained language models,
auto-encoder style frameworks, or by recursively computing dissimilarity
scores.

In this landscape of models aiming to overcome the data sparsity and domain
dependency of current discourse parsers, we present a new, supervised paradigm
directly tackling the domain adaptation issue. Specifically, we introduce the
first fully supervised discourse parser designed to alleviate the domain
dependency through a staged model of weak classifiers by introducing the
gradient boosting framework Schaal and Atkeson (1995); Drucker et al. (1994);
Schwenk and Bengio (1997); Badirli et al. (2020) into the process of discourse
parsing. Using the underlying assumption that any discourse treebank contains
a mix of frequently appearing, general discourse features (applicable to any
domain) as well as a number of dataset-related nuances (which are domain-
specific), we postulate that a set of weak classifiers is likely to learn
increasingly specific and rare features of the training data. Using this
assumption, we can reasonably assume that there exists a threshold of weak
classifiers, which effectively separates the general features of discourse
from domain-specific characteristics introduced by the dataset. As a result,
we aim to separate this mixture of features using the gradient boosting
approach with the goal to generate a more domain-independent discourse parser.

##  2 Approach

Our approach to introducing neural gradient boosting into the domain of
discourse parsing builds on top of the state-of-the-art (SOTA) neural shift-
reduce parser by Guz et al. (2020); Guz and Carenini (2020)111For more
information on the underlying approach, we refer interested readers to Guz et
al. (2020).. In this work, we aim to extend this previous line of research in
three meaningful directions:

###  2.1 Added Discourse Relations

The method proposed in Guz et al. (2020); Guz and Carenini (2020) reaches SOTA
performance on the RST-DT structure and nuclearity prediction, however, does
not consider the important relation attribute. In this work, we aim to
generate complete discourse trees with all three components, introducing an
additional relation-prediction component besides the structure and nuclearity
predictor.

###  2.2 Linguistically Inspired Stack Representations

In the current SOTA method, spans on the stack are truncated by removing
tokens from the center of the textual representation (e.g., a sequence of
$\\{t_{1},t_{2},t_{3},t_{4}\\}$ and a maximal length of $2$ results in
$\\{t_{1},t_{4}\\}$). As shown by the promising performance of the approach,
this heuristic assumption seems reasonable, however, lacks linguistic
justification. To this end, we propose a new method to reduce stack elements
according to the sub-tree nuclearity. This directly follows the argument in
Morey et al. (2018), stating that the relation between constituents in an RST-
style discourse tree holds between the respective nuclei of the sub-trees.

###  2.3 Gradient Boosting Approach

Gradient boosting refers to a classical machine learning approach using an
ensemble of weak classifiers initially developed for decision trees Schaal and
Atkeson (1995) and later adopted for neural architectures Schwenk and Bengio
(1997); Badirli et al. (2020) (oftentimes called neural/deep gradient
boosting), which has been shown to benefit important NLP tasks, such as part-
of-speech tagging Abney et al. (1999), sentiment analysis Athanasiou and
Maragoudakis (2017), and text classification Kudo and Matsumoto (2004),
delivering robust models when data is scarce. Our model architecture
envisioned in this work is presented in Figure 1. Following the gradient
boosting paradigm, we start with a single weak classifier (left side in Figure
1) and train a standard shift-reduce model to predict RST-style discourse
trees. The number of free parameters, purposely chosen to be small for
individual, weak classifiers, is thereby likely to limit the ability of the
model to learn complex features and relations, resulting in the initial
training step to exploit simple structures, e.g., resembling purely right-
branching trees. After convergence of the initial weak classifier, a second
step is introduced with a similar-sized set of free parameters in the shift-
reduce parsing component. This time (see the center in Figure 1), the parsing
component is trained to improve the performance of the combined prediction
consisting of the initial parser in step 1 and the currently trained
component. With the parameter-frozen prediction from the first step being
combined with the output of the parser in step 2, the combined model is bound
to learn more nuanced relationships in the data. Following the gradient-
boosting methodology, the second step thereby improves (i.e., boosts) the
performance of the joint classifier for samples that the first step did not
capture. Repeating this process for $n$ times, an increasingly specific parser
is built.

To summarize, we believe that the gradient boosted method in combination with
our extensions of the SOTA work by Guz and Carenini (2020) should improve the
domain-independence of supervised discourse parsers when trained on small-
scale, human-annotated discourse treebanks. With the iterative modelling
strategy, our gradient boosted method can likely utilize the limited training
data more efficiently, achieving more domain-independent models, while still
reaching high performance in-domain.

##  3 Planned Evaluation

Following our novel extensions proposed in section 2, we plan to evaluate the
model along four dimensions:

#### Performance Comparison to Single-Step Models,

namely Guz et al. (2020) and Guz and Carenini (2020), focusing on the
potential of multiple, weak classifiers compared to a single-step, strong
classifier.

#### Training Time and Size Requirements:

With large models requiring increasingly restrictive training time and
resources, the linear combination of weak classifiers allows for more
efficient training, making models more accessible, even with severe hardware
restrictions.

#### Number of Free Parameters:

We plan to investigate the size of weak classifiers in regard to the number of
gradient boosting steps and performance. We believe that this detailed
investigation can shed further light onto the potential of gradient boosted
approaches for discourse parsing.

#### Domain Independence Across Steps:

Here, we aim to evaluate whether a larger number of gradient boosting steps
leads to increased domain-specificity. This evaluation will compare the model
performance of the first $m$ gradient boosting steps (with $1\leq m\leq n$) to
gold-standard discourse structures in different domains, aiming to quantify
the correlation of gradient boosted modelling steps with increased domain
dependency.

##  4 Conclusion

In this work, we aim to improve current supervised discourse parsers through a
gradient-boosted modelling approach and linguistically inspired model
adaptions. Compared to previously proposed models, we try to overcome the
domain dependency through a staged model capturing increasingly domain-
specific information, making better use of the limited amount of gold-standard
discourse data. Using more linguistically inspired stack representations and
adding a relation classification component, we hope to create a general and
domain-independent, fully supervised discourse parser.

## References

  * Abney et al. (1999) Steven Abney, Robert E. Schapire, and Yoram Singer. 1999.  Boosting applied to tagging and PP attachment.  In _1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_.
  * Asher (1993) Nicholas Asher. 1993.  _Reference to abstract objects in discourse_, volume 50.  Springer Science & Business Media.
  * Asher et al. (2003) Nicholas Asher, Nicholas Michael Asher, and Alex Lascarides. 2003.  _Logics of conversation_.  Cambridge University Press.
  * Athanasiou and Maragoudakis (2017) Vasileios Athanasiou and Manolis Maragoudakis. 2017.  A novel, gradient boosting framework for sentiment analysis in languages where nlp resources are not plentiful: a case study for modern greek.  _Algorithms_ , 10(1):34.
  * Badirli et al. (2020) Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya S Keerthi. 2020.  Gradient boosting neural networks: Grownet.  _arXiv preprint arXiv:2002.07971_.
  * Bhatia et al. (2015) Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015.  Better document-level sentiment analysis from rst discourse parsing.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2212–2218, Lisbon, Portugal. Association for Computational Linguistics.
  * Carlson et al. (2002) Lynn Carlson, Mary Ellen Okurowski, and Daniel Marcu. 2002.  _RST discourse treebank_.  Linguistic Data Consortium, University of Pennsylvania.
  * Drucker et al. (1994) Harris Drucker, Corinna Cortes, L.D. Jackel, Yann LeCun, and Vladimir Vapnik. 1994\.  Boosting and other machine learning algorithms.  In William W. Cohen and Haym Hirsh, editors, _Machine Learning Proceedings 1994_ , pages 53–61. Morgan Kaufmann, San Francisco (CA).
  * Feng and Hirst (2014) Vanessa Wei Feng and Graeme Hirst. 2014.  A linear-time bottom-up discourse parser with constraints and post-editing.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 511–521, Baltimore, Maryland. Association for Computational Linguistics.
  * Gerani et al. (2014) Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T. Ng, and Bita Nejat. 2014\.  Abstractive summarization of product reviews using discourse structure.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1602–1613, Doha, Qatar. Association for Computational Linguistics.
  * Guz and Carenini (2020) Grigorii Guz and Giuseppe Carenini. 2020.  Coreference for discourse parsing: A neural approach.  In _Proceedings of the First Workshop on Computational Approaches to Discourse_ , pages 160–167, Online. Association for Computational Linguistics.
  * Guz et al. (2020) Grigorii Guz, Patrick Huber, and Giuseppe Carenini. 2020.  Unleashing the power of neural discourse parsers - a context and structure aware approach using large scale pretraining.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3794–3805, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Huber and Carenini (2019) Patrick Huber and Giuseppe Carenini. 2019.  Predicting discourse structure using distant supervision from sentiment.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2306–2316, Hong Kong, China. Association for Computational Linguistics.
  * Huber and Carenini (2020) Patrick Huber and Giuseppe Carenini. 2020.  MEGA RST discourse treebanks with structure and nuclearity from scalable distant sentiment supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7442–7457, Online. Association for Computational Linguistics.
  * Huber and Carenini (2021) Patrick Huber and Giuseppe Carenini. 2021.  Unsupervised learning of discourse structures using a tree autoencoder.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(14):13107–13115.
  * Huber and Carenini (2022) Patrick Huber and Giuseppe Carenini. 2022.  Towards understanding large-scale discourse structures in pre-trained and fine-tuned language models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2376–2394, Seattle, United States. Association for Computational Linguistics.
  * Huber et al. (2021) Patrick Huber, Linzi Xing, and Giuseppe Carenini. 2021.  Predicting above-sentence discourse structure using distant supervision from topic segmentation.  _arXiv preprint arXiv:2112.06196_.
  * Ji and Eisenstein (2014) Yangfeng Ji and Jacob Eisenstein. 2014.  Representation learning for text-level discourse parsing.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13–24, Baltimore, Maryland. Association for Computational Linguistics.
  * Ji and Smith (2017) Yangfeng Ji and Noah A. Smith. 2017.  Neural discourse structure for text categorization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 996–1005, Vancouver, Canada. Association for Computational Linguistics.
  * Joty et al. (2015) Shafiq Joty, Giuseppe Carenini, and Raymond T. Ng. 2015.  CODRA: A novel discriminative framework for rhetorical analysis.  _Computational Linguistics_ , 41(3):385–435.
  * Karimi and Tang (2019) Hamid Karimi and Jiliang Tang. 2019.  Learning hierarchical discourse-level structure for fake news detection.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3432–3442, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kobayashi et al. (2019) Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019.  Split or merge: Which is better for unsupervised RST parsing? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5797–5802, Hong Kong, China. Association for Computational Linguistics.
  * Koto et al. (2021) Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021.  Discourse probing of pretrained language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3849–3864, Online. Association for Computational Linguistics.
  * Kudo and Matsumoto (2004) Taku Kudo and Yuji Matsumoto. 2004.  A boosting algorithm for classification of semi-structured text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 301–308, Barcelona, Spain. Association for Computational Linguistics.
  * Li et al. (2016) Qi Li, Tianshi Li, and Baobao Chang. 2016.  Discourse parsing with attention-based hierarchical neural networks.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 362–371, Austin, Texas. Association for Computational Linguistics.
  * Liu et al. (2019) Yang Liu, Ivan Titov, and Mirella Lapata. 2019.  Single document summarization as tree induction.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1745–1755, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988.  Rhetorical structure theory: Toward a functional theory of text organization.  _Text - Interdisciplinary Journal for the Study of Discourse_ , 8(3):243–281.
  * Morey et al. (2018) Mathieu Morey, Philippe Muller, and Nicholas Asher. 2018.  A dependency perspective on RST discourse parsing and evaluation.  _Computational Linguistics_ , 44(2):197–235.
  * Nejat et al. (2017) Bita Nejat, Giuseppe Carenini, and Raymond Ng. 2017.  Exploring joint neural model for sentence level discourse parsing and sentiment analysis.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 289–298, Saarbrücken, Germany. Association for Computational Linguistics.
  * Nishida and Nakayama (2020) Noriki Nishida and Hideki Nakayama. 2020.  Unsupervised discourse constituency parsing using Viterbi EM.  _Transactions of the Association for Computational Linguistics_ , 8:215–230.
  * Schaal and Atkeson (1995) Stefan Schaal and Christopher Atkeson. 1995.  From isolation to cooperation: An alternative view of a system of experts.  In _Advances in Neural Information Processing Systems_ , volume 8. MIT Press.
  * Schwenk and Bengio (1997) Holger Schwenk and Yoshua Bengio. 1997.  Training methods for adaptive boosting of neural networks.  In _Advances in Neural Information Processing Systems_ , volume 10. MIT Press.
  * Wang et al. (2017) Yizhong Wang, Sujian Li, and Houfeng Wang. 2017.  A two-stage parsing method for text-level discourse analysis.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 184–188, Vancouver, Canada. Association for Computational Linguistics.
  * Webber et al. (2003) Bonnie Webber, Matthew Stone, Aravind Joshi, and Alistair Knott. 2003.  Anaphora and discourse structure.  _Computational Linguistics_ , 29(4):545–587.
  * Wu et al. (2020) Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020.  Perturbed masking: Parameter-free probing for analyzing and interpreting BERT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4166–4176, Online. Association for Computational Linguistics.
  * Xiao et al. (2020) Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2020.  Do we really need that many parameters in transformer for extractive summarization? discourse can help ! In _Proceedings of the First Workshop on Computational Approaches to Discourse_ , pages 124–134, Online. Association for Computational Linguistics.
  * Zeldes (2017) Amir Zeldes. 2017.  The gum corpus: Creating multilayer resources in the classroom.  _Lang. Resour. Eval._ , 51(3):581–612.
  * Zhu et al. (2020) Zining Zhu, Chuer Pan, Mohamed Abdalla, and Frank Rudzicz. 2020.  Examining the rhetorical capacities of neural language models.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 16–32, Online. Association for Computational Linguistics.


# Egocentric Audio-Visual Object Localization

Chao Huang1, Yapeng Tian1, Anurag Kumar2, Chenliang Xu1
1University of Rochester, 2Meta Reality Labs Research
{chaohuang, yapengtian<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Humans naturally perceive surrounding scenes by unifying sound and sight from
a first-person view. Likewise, machines are advanced to approach human
intelligence by learning with multisensory inputs from an egocentric
perspective. In this paper, we explore the challenging egocentric audio-visual
object localization task and observe that 1) egomotion commonly exists in
first-person recordings, even within a short duration; 2) The out-of-view
sound components can be created when wearers shift their attention. To address
the first problem, we propose a geometry-aware temporal aggregation module
that handles the egomotion explicitly. The effect of egomotion is mitigated by
estimating the temporal geometry transformation and exploiting it to update
visual representations. Moreover, we propose a cascaded feature enhancement
module to overcome the second issue. It improves cross-modal localization
robustness by disentangling visually-indicated audio representation. During
training, we take advantage of the naturally occurring audio-visual temporal
synchronization as the “free” self-supervision to avoid costly labeling. We
also annotate and create the Epic Sounding Object dataset for evaluation
purposes. Extensive experiments show that our method achieves state-of-the-art
localization performance in egocentric videos and can be generalized to
diverse audio-visual scenes. Code is available at
https://github.com/WikiChao/Ego-AV-Loc.

##  1 Introduction

Figure 1: Sounding object localization in egocentric videos. Due to the
wearer’s egomotion, the viewpoint changes continuously across time.
Consequently, audio-visual relations are dynamically changing in egocentric
videos. Our approach tackles challenges in the egocentric audio-visual
sounding object task and learns audio-visual associations from first-person
videos.

The emergence of wearable devices has drawn the attention of the research
community to egocentric videos, the significance of which can be seen from
egocentric research in a variety of applications such as robotics [48, 34,
32], augmented/virtual reality [31, 61, 75], and healthcare [66, 53]. In
recent years, the computer vision community has made substantial efforts to
build benchmarks [12, 13, 57, 15, 69, 40], establish new tasks [17, 37, 39,
60, 36], and develop frameworks [41, 33, 54, 82] for egocentric video
understanding.

While existing works achieve promising results in the egocentric domain, it
still remains an interesting but challenging topic to perform fine-grained
egocentric video understanding. For instance, understanding which object is
emitting sound in a first-person recording is difficult for machines. As shown
in Fig. 1, the wearer moves his/her head to put down the bottle. The frying
pot which emits sound subsequently suffers deformation and occlusion due to
the wearer’s egomotion. Human speech outside the wearer’s view also affects
the machine’s understanding of the current scene. This example reveals two
significant challenges for designing powerful and robust egocentric video
understanding systems: First, people with wearable devices usually record
videos in naturalistic surroundings, where a variety of illumination
conditions, object appearance, and motion patterns are shown. The dynamic
visual variations introduce difficulties in accurate visual perception.
Second, egocentric scenes are often perceived within a limited field of view
(FoV). The common body and head movements cause frequent view changes (see
Fig. 1), which brings object deformation and creates dynamic out-of-view
content.

Although a visual-only system may struggle to fully decode the surrounding
information and perceive scenes in egocentric videos, audio provides stable
and persistent signals associated with the depicted events. Instead of purely
visual perception, numerous psychological and cognitive studies [67, 29, 6,
73] show that integration of auditory and visual signals is significant in
human perception. Audio, as an essential but less focused modality, often
provides synchronized and complementary information with the video stream. In
contrast to the variability of first-person visual footage, sound describes
the underlying scenes consistently. These natural characteristics make audio
another indispensable ingredient for egocentric video understanding.

To effectively leverage audio and visual information in egocentric videos, a
pivotal problem is to analyze the fine-grained audio-visual association,
specifically identifying which objects are emitting sounds in the scene. In
this paper, we explore a novel egocentric audio-visual object localization
task, which aims to associate audio with dynamic visual scenes and localize
sounding objects in egocentric videos. Given the dynamic nature of egocentric
videos, it is exceedingly challenging to link visual content from various
viewpoints with audio captured from the entire space. Hence, we develop a new
framework to model the distinct characteristics of egocentric videos by
integrating audio. In the framework, we propose a geometry-aware temporal
module to handle egomotion explicitly. Our approach mitigates the impact of
egomotion by performing geometric transformations in the embedding space and
aligning visual features from different frames. We further use the aligned
features to leverage temporal contexts across frames to learn discriminative
cues for localization. Additionally, we introduce a cascaded feature
enhancement module to handle out-of-view sounds. The module helps mitigate
audio noises and improves cross-modal localization robustness.

Due to the dynamic nature of egocentric videos, it is hard and costly to label
sounding objects for supervised training. To avoid tedious labeling, we
formulate this task in a self-supervised manner, and our framework is trained
with audio-visual temporal synchronization. Since there are no publicly
available egocentric sounding object localization datasets, we annotate an
Epic Sounding dataset to facilitate research in this field. Experimental
results demonstrate that modeling egomotion and mitigating out-of-view sound
can improve egocentric audio-visual localization performance.

In summary, our contributions are: (1) the first systematical study on
egocentric audio-visual sounding object localization; (2) an effective
geometry-aware temporal aggregation approach to deal with unique egomotion;
(3) a novel cascaded feature enhancement module to progressively inject
localization cues; and (4) an Epic Sounding Object dataset with sounding
object annotations to benchmark the localization performance in egocentric
videos.

Figure 2: An overview of our egocentric audio-visual object localization
framework. In the beginning, our model extracts deep features from the video
and audio streams. Then, the audio and visual features are fed into the
cascaded feature enhancement module to inject localization cues for both
branches. Such a module is additionally trained with “mix-and-separation”
strategy. Next, our geometric-aware temporal modeling block leverages the
relative geometric information between visual frames and performs temporal
context aggregation to get the final visual features for localization.

##  2 Related Work

Audio-visual learning in third-person view videos. Taking the natural audio-
visual synchronization in videos, a large number of studies in the past few
years have proposed to jointly learn from both auditory and visual modalities.
We have seen a spectrum of new audio-visual problems and applications,
including visually guided sound source separation [16, 87, 76, 23, 20, 21, 22,
88, 63], audio-visual representation learning [3, 5, 26, 35, 58, 59, 2],
audio-visual event localization [78, 79, 84, 44], audio-visual video parsing
[77, 83], and sounding object visual localization [65, 4, 11, 62, 38, 28, 71,
52, 51, 27]. Most previous approaches learn audio-visual correlations from
third-person videos, while the distinct challenges of audio-visual learning in
egocentric videos are underexplored. Different from existing works, we propose
an audio-visual learning framework to explicitly solve egomotion and out-of-
view audio issues in egocentric videos.

Egocentric video understanding. In the last decade, video scene understanding
techniques thrived because of the well-defined third-person video datasets [9,
72, 7, 49]. Nevertheless, most of the algorithms are developed to tackle
videos curated by human photographers. The natural characteristics of
egocentric video data, _e.g_., view changes, large motions, and visual
deformation, are not well-explored. To bridge this gap, multiple egocentric
datasets [69, 12, 74, 36, 13, 24] have been collected. These datasets have
significantly advanced investigations on egocentric video understanding
problems, including activity recognition [33, 41, 89], human(hand)-object
interaction [8, 55, 14, 68], anticipation [1, 19, 45, 70], and human body pose
inferring [30, 56]. However, only a handful of audio-visual works [10, 85, 33,
50] is presented for egocentric video understanding. Among those, Kazakos  _et
al_. [33] proposed an audio-visual fusion network for action recognition,
while Mittal  _et al_. [50] used audible interactions as cues to learn state-
aware visual representations in egocentric videos. There are limited studies
in explicit egomotion mitigation and fine-grained audio-visual association
learning in egocentric videos. Unlike past works, we tackle challenges in
egocentric audio-visual data and propose a robust sounding object localization
framework. To enable the research, we propose Epic Sounding Object dataset
based on Epic-Kitchens [12, 13].

##  3 Method

Our goal is to localize sounding objects in egocentric videos visually. We
start by formulating our egocentric audio-visual object localization task in
Sec. 3.1. Our proposed method includes a feature extraction process (described
in Sec. 3.2), a two-stage cascaded feature enhancement pipeline (in Sec. 3.3),
and a geometry-aware temporal aggregation module (explained in Sec. 3.4).
Finally, we summarize the overall training objective in Sec. 3.5.

###  3.1 Problem Formulation and Method Overview

Given an egocentric video clip $V=\\{I_{i}\\}^{T}_{i=1}$ in $T$ frames and its
corresponding sound stream $s$, sounding object visual localization aims at
predicting location maps $\mathcal{O}=\\{O_{i}\\}^{T}_{i=1}$ that represent
sounding objects in the egocentric video. Specifically,
$O_{i}(x,y)\in\\{0,1\\}$ and positive visual regions indicate locations of
sounding objects. In real-world scenarios, the captured sound can be a mixture
of multiple sound sources $s=\sum_{n=1}^{N}s_{n}$, where $s_{n}$ is the $n$-th
sound source and it could be out of view. For the visual input, the video
frames may be captured from different viewpoints. To design a robust and
effective egocentric audio-visual sounding object localization system, we
should consider the above issues in egocentric audio and visual data and
answer two key questions: (Q1) how to associate visual content with audio
representations while out-of-view sounds may exist; (Q2) how to persistently
associate audio features with visual content that are captured under different
viewpoints.

Due to the dynamic nature of egocentric videos, it is difficult and costly to
annotate sounding objects for supervised training. To bypass the tedious
labeling, we solve the egocentric audio-visual object localization task in a
self-supervised manner. The proposed framework is shown in Fig. 2. Our model
first extracts representations from the audio $s$ and video clip $V$. In order
to handle Q1, we develop a cascaded feature enhancement module to disentangle
visually indicated sound sources and attend to visual regions that correspond
to the visible sound sources. To enable the disentanglement, we use on-screen
sound separation task as the proxy and adopt a multi-task learning objective
to train our model where the localization task is solved along with a sound-
separation task. To deal with the egomotion in egocentric videos (Q2), we
design a geometry-aware temporal modeling approach to mitigate the feature
distortion brought by viewpoint changes and aggregate the visual features
temporally. We take the audio-visual temporal synchronization as the
supervision signal and estimate the localization map $\tilde{O_{i}}$.

###  3.2 Feature Extraction

Visual representation. We use a visual encoder network $E_{v}$ to extract
visual feature maps from each input frame $I_{i}$. In our implementation, a
pre-trained Dilated ResNet [86] model is adopted by removing the final fully-
connected layer. We can subsequently obtain a group of feature maps
$v_{i}=E_{v}(I_{i})$, where $v_{i}\in\mathbb{R}^{c\times h_{v}\times w_{v}}$.
Here $c$ is the number of channels, and $h_{v}\times w_{v}$ denotes the
spatial size.

Audio representation. To extract audio representations from the input raw
waveform, we first transform audio stream $s$ into a magnitude spectrogram $X$
with the short-time Fourier transform (STFT). Then, we extract audio features
$a=E_{a}(X),a\in\mathbb{R}^{c\times h_{a}\times w_{a}}$ by means of a CNN
encoder $E_{a}$ in the Time-Frequency (T-F) space.

###  3.3 Cascaded Feature Enhancement

As discussed in Sec. 3.1, a sound source $s_{n}$ in the mixture $s$ could be
out of view due to constant view changes in egocentric videos and the limited
FoV. This poses challenges in visually localizing sound sources and
performance can degrade when the audio-visual associations are not precise. To
address this, we update the features in a cascaded fashion. We first force the
network to learn disentangled audio representations from the mixture using
visual guidance. Then we utilize the disentangled audio representations to
inject the visual features with more localization cues.

Disentanglement through sound source separation. Sound source localization
objective can implicitly guide the system to learn disentangled audio features
as the network will try to precisely localize the sound, and in turn, the on-
screen sound will get disentangled from the rest. However, we formulate our
problem in an unsupervised setting where labels for such localization
objective are not available.

Audio-visual sound separation task [23, 87] uses visual information as
guidance to learn to separate individual sounds from a mixture. Given the
visual guidance, it is expected that the learned representations primarily
encode information from visually indicated sound sources. Hence we argue for a
multi-task learning approach to solve our primary task. Along with the audio-
visual sounding object localization task, the network also learns to
disentangle visible audio representations from the mixture through a source
separation task.

  * •

Training. We adopt the commonly used “mix-and-separate” strategy [23, 87] for
audio-visual sound separation. Given the current audio $s^{(1)}$, we randomly
sample another audio stream $s^{(2)}$ from a different video and mix them
together to generate input audio mixture $\tilde{s}=s^{(1)}+s^{(2)}$. We then
obtain magnitude spectrograms $\tilde{X}$, $X^{(1)}$, $X^{(2)}$ for
$\tilde{s}$, $s^{(1)}$ and $s^{(2)}$ respectively. The audio features is then
modified as $a=E_{a}(\tilde{X})$.

  * •

Inference. During inference, we take the original audio stream as input:
$s=s^{(1)}$ and $X=X^{(1)}$ to extract visually correlated audio
representations. Note that the audio features is $a=E_{a}(X)$.

We define the audio disentanglement network as a network $f(\cdot)$, which
produces the disentangled audio features $\hat{a}\in\mathbb{R}^{c\times
h_{a}\times w_{a}}$. In this network, we want to associate the visual content
with the audio representations to perform disentanglement in the embedding
space. Concretely, we first apply spatial average pooling on each $v_{i}$ and
temporal max pooling along the time axis to obtain a visual feature vector
$g_{v}\in\mathbb{R}^{c}$. Then we replicate the visual feature vector
$h_{a}\times w_{a}$ times and tile them to match the size of $a$. We
concatenate the visual and audio feature maps along the channel dimension and
feed them into the network. Therefore, the audio feature disentanglement can
be formulated as:

$\hat{a}=f(\,\textsc{Concat}[\,a,\;\textsc{Tile}(g_{v})\,]\,).$ (1)

In practice, we implement the disentanglement network $f$ using two 1x1
convolution layers. The audio feature $\hat{a}$ will be used for both
separation mask and sounding object localization map generation.

To separate visible sounds, we add an audio decoder $D_{a}$ following the
disentanglement network to output a binary mask $M_{pred}=D_{a}(\hat{a})$ (at
the bottom of Fig. 2). U-Net architectures [23] are used in the audio encoder
$E_{a}$ and decoder $D_{a}$. We implement the $E_{a}$ and $D_{a}$ in five
convolution and up-convolution layers, respectively. Details of the network
architectures are provided in the Appendix. The ground truth separation mask
$M_{gt}$ can be calculated by determining whether the original input sound is
dominant at locations $(u,v)$ in the T-F space:

$\displaystyle M_{gt}(u,v)=[X^{(1)}(u,v)\geq\tilde{X}(u,v)].$ (2)

To train the sound separator, we minimize the $\ell_{2}$ distance between the
predicted and ground-truth masks as the disentanglement learning objective:

$\mathcal{L}_{dis}=||M_{pred}-M_{gt}||_{2}^{2}.$ (3)

Soft localization. Similar to the out-of-view sounds, the visual frames may
contain sound-irrelevant regions. In order to learn more precise audio-visual
associations, we propose to highlight the spatial regions that are more likely
to be correlated with the on-screen sounds by computing audio-visual
attention. The attention map will indicate the correlation between audio and
visual representations at different spatial locations. Given the output
$\hat{a}$ from disentanglement network $f(\cdot)$, we apply max pooling on its
time and frequency dimensions, obtaining an audio feature vector
$g_{\hat{a}}$. Then at each spatial position (x,y) of visual feature $v_{i}$,
we compute the cosine similarity between audio and visual feature vectors:

$S_{i}:S_{i}(x,y)=\textsc{CosineSim}(v_{i}(x,y),g_{\hat{a}}).$ (4)

Softmax is then used on $S_{i}$ to generate a soft mask that represents the
audio-visual correspondence. Hence, each $v_{i}$ can be attended with the
calculated weights:

$\displaystyle\hat{v}_{i}=\textsc{Softmax}(S_{i})\cdot v_{i}.$ (5)

###  3.4 Geometry-Aware Temporal Modeling

Figure 3:  Overview of our proposed geometry-aware modeling approach. The
visual features $\hat{v}_{j}$ are warped to viewpoint $i$ by homography
transformation.

Given the temporal nature of sounds and the persistence of audio-visual
associations, we incorporate temporal information from neighboring frames to
learn sounding object features. However, temporal modeling is a challenging
problem for egocentric videos due to widespread egomotion and object-
appearance deformations.

Although visual objects are dynamically changing, the surrounding physical
environment is persistent. Hence, temporal variations in egocentric videos
reveal rich 3D geometric cues that can recover the surrounding scene from
changing viewpoints. Prior works have shown that given a sequence of frames,
one can reconstruct the underlying 3D scene from the 2D observations [64, 81].
In our work, rather than reconstructing the 3D structures, we estimate the
relative geometric transformation between frames to alleviate egomotion.
Specifically, we apply the transformation at the feature level to perform
geometry-aware temporal aggregation. Given $\\{I_{i}\\}^{T}_{i=1}$ and their
features ${\\{\hat{v}_{i}\\}^{T}_{i=1}}$, we take $\hat{v}_{i}$ as a query at
a time and use the other features from neighboring frames as support features
to aggregate temporal contexts. For clarity, we decompose the geometry-aware
temporal aggregation into two parts: geometry modeling and temporal
aggregation.

Geometry modeling. This step aims to compute the geometric transformation that
represents the egomotion between frames (see Fig. 3). We found that homography
estimation, which can align images taken from different perspectives, can
serve as a way to measure geometric transformation. We adopt SIFT [46] \+
RANSAC [18] to solve homography. To be specific, a homography is a $3\times 3$
matrix that consists of 8 degree of freedom (DOF) for scale, translation,
rotation, and perspective respectively. Given the query frame $I_{i}$ and a
supporting frame $I_{j}$, we use $h(\cdot)$ to denote the computation process:

$\mathcal{H}_{ji}=h(I_{j},I_{i})_{j\xrightarrow{}i},$ (6)

where $\mathcal{H}_{ji}$ represents the homography transformation from frame
$I_{j}$ to $I_{i}$. With the computed homography transformation, we can then
apply it at the feature level to transform visual features $\hat{v}_{j}$ to
$\hat{v}_{ji}$. The $\hat{v}_{ji}$ is egomotion-free under the viewpoint of
$I_{i}$. Since the resolution of feature maps is scaled down compared to the
raw frame size, the homography matrix $\mathcal{H}$ should also be downsampled
using the same scaling factor. The feature transformation can be written as:

$\hat{v}_{ji}=\mathcal{H}_{ji}\otimes\hat{v}_{j},$ (7)

where $\otimes$ represents the warping operation.

Figure 4: Illustration on the temporal context aggregation process for query
feature $\hat{v}_{i}$ and neighboring frame features $\hat{v}_{i-1}$ and
$\hat{v}_{i+1}$, which is performed independently at each spatial location.

Temporal aggregation. For the query feature $\hat{v}_{i}$, we end up with set
of aligned features $\\{\hat{v}_{ji}\\}^{T}_{j=1}$ corresponding to each frame
viewpoint. To aggregate the temporal contexts, we propose to compute the
correlation between features from different frames at the same locations (see
Fig. 4). The aggregation process can be formulated as:

$z_{i}(x,y)=\hat{v}_{i}(x,y)+\textsc{Softmax}(\frac{\hat{v}_{i}(x,y){\boldsymbol{\hat{v}}(x,y)}^{T}}{\sqrt{d}})\boldsymbol{\hat{v}}(x,y),$
(8)

where $\boldsymbol{\hat{v}}=[\hat{v}_{1i};...;\hat{v}_{Ti}]$ is the
concatenation of frame features; the scaling factor $d$ is equal to the
feature dimension; and $(\cdot)^{T}$ represents the transpose operation [80].
The aggregation operation is applied at all spatial locations $(x,y)$ to
generate the updated visual features $z_{i}$.

Figure 5: Illustration of the Epic Sounding Object dataset statistics. (a)
Example of our video frames and sounding object annotations. Class diversity
(squeeze packaging, close trash can, put the pot, etc.) (b): The distribution
of untrimmed video duration. (c): The number of videos that are annotated as
containing out-of-view sounds. (d): Distribution of bounding box areas in Epic
Sounding Object dataset, the majority of boxes cover less than 20% of the
image area, demonstrating the difficulty of this task.

###  3.5 Training Objective

We take audio-visual synchronization as the “free” supervision and solve the
task in a self-supervised manner using contrastive learning [2, 4, 65, 11].

With the audio feature vector $g_{\hat{a}}$ and the visual features
$\\{z_{i}\\}^{T}_{i=1}$, we can compute an audio-visual attention map $S_{i}$
in Eq. 4 for each frame $I_{i}$. The training objective should optimize the
network such that only the sounding regions have a high response in $S_{i}$.
Since the ground-truth sounding map is unknown, we apply differential
thresholding on $S_{i}$ to predict sounding objectness map
${O}_{i}=sigmoid((S_{i}-\epsilon)/\tau)$ [11], where $\epsilon$ is the
threshold, and $\tau$ denotes the temperature that controls the sharpness.

In an egocentric video clip, a visual scene is usually temporally dynamic.
Sometimes a single audio-visual pair ($I_{i}$,$s$) may not be audio-visually
correlated. To this end, we solve the localization task in the Multiple-
Instance Learning (MIL) [47] setting to improve robustness. Concretely, we use
a soft MIL pooling function to aggregate the concatenated attention maps
$\boldsymbol{S}=[S_{1};...;S_{T}]$ by assigning different weights to $S_{t}$
at different time steps:

$\displaystyle\overline{S}=\sum_{t=1}^{T}(W_{t}\cdot\boldsymbol{S})[:,:,t],$
(9)

where $W_{t}[x,y,:]=\textsc{Softmax}(\boldsymbol{S}[x,y,:])$, $x$ and $y$ are
the indices on spatial dimensions. Subsequently, an aggregated sounding
objectness map $\overline{O}$ is calculated from $\overline{S}$. In this way,
for each video clip $V$ in the batch, we can define its positive and negative
training signals as:

$\displaystyle
P=\frac{1}{|\overline{O}|}\langle\overline{O},\overline{S}\rangle,\quad
N=\frac{1}{hw}\langle\mathbf{1},S_{neg}\rangle,$ (10)

where $\langle\cdot,\cdot\rangle$ is the Frobenius inner product.

We obtain negative audio-visual attention maps $S_{neg}$ by associating the
current visual inputs $I$ with audio from other video clips. $\mathbf{1}$
denotes an all ones tensor with shape $h\times w$. Therefore, the localization
optimization objective is:

$\displaystyle\mathcal{L}_{loc}=-\frac{1}{N}\sum_{k=1}^{N}[\log\frac{\exp(P_{k})}{\exp(P_{k})+\exp(N_{k})}],$
(11)

where $k$ is the video sample index in a training batch. The overall objective
is $\mathcal{L}=\mathcal{L}_{loc}+\lambda\mathcal{L}_{dis}$, where we
empirically set $\lambda=5$ in our experiments.

##  4 The Epic Sounding Object Dataset

Existing sound source visual localization evaluation datasets, such as
SoundNet-Flickr [65], VGG-Sound Source [11], only contain third-person
recordings. To the best of our knowledge, there is no existing dataset that is
suitable for evaluating our model. Thus, we introduce an Epic Sounding Object
Dataset for egocentric audio-visual sounding object localization. Built upon
the well-known Epic-Kitchens [13] dataset, we collect sounding object
annotations on its action recognition test set.
Data preparation. We select 13k test videos from the Epic-Kitchens action
recognition benchmark as our source data. Since these videos are not
originally collected for audio-visual analysis, they vary in length, and not
all of them contain meaningful sounds. To verify the videos for annotations,
we conduct a two-step process: We first determine if a video is silent by
checking its sound-level in decibels relative to full scale. Consequently,
silent videos are filtered out to provide a meaningful data source. Second, we
bin the videos by their duration and show the statistics in Fig. 5 (b). The
majority last less than 2 seconds, and hence we choose to trim the center
1-second clip from each video.

After pre-processing, we obtain 5,089 videos in total for annotation. For each
video, we uniformly select three frames and annotate sounding objects in the
frames. We follow previous works [76, 11] to use bounding boxes to annotate
the objects that emit sounds. To obtain proposals of potential sounding
objects automatically, we follow Epic-Kitchens [13] to use a Mask R-CNN object
detector [25] trained on MS-COCO [43] and a hand-objects detector [68] that is
pretrained with 42K egocentric images [13, 42, 69].

Before voting | After voting
---|---
Video | Frames | Video | Frames | Classes
5,089 | 15,267 | 3,172 | 9,196 | 30
Table 1: Statistics of the Epic Sounding Object dataset.

Annotation collection. Given the pre-processed data, we then annotate the
sounding objects manually. Unlike third-person view videos, the object-sound
associations in the egocentric domain are more complicated. There are two main
challenges: (i) Numerous egocentric videos record wearer-environment
interaction (_e.g_., a human places a dish on the table). The object-sound
associations could be dynamic, and sometimes it is hard to determine what
objects are emitting sounds; and (ii) the objects in egocentric videos are
often missing from the screen, resulting in variations in scale (see Fig. 5
(c)). We address the above issues by taking advantage of human commonsense
knowledge. We ask three or more annotators from the Amazon Mechanic Turk to
annotate the same video (frames). Concretely, they do this by first watching
the 1-second video with three annotated frames to confirm what objects make
sounds in the video. During the annotation course, they are asked to answer
two questions: (1) Does the video contain out-of-view sounds? (2) Which
bounding boxes correspond to the sounding objects? We collect the annotations
in multiple rounds until each video has at least three or more valid
annotations from the Amazon annotators. Finally, we conduct an annotation
verification by voting on all videos. If at least two annotators agree on the
same answer, it will be considered as a correct annotation; If not, we will
simply omit the video. The annotation statistics after voting are shown in
Tab. 1. We obtain 30 classes of sounds by counting the noun (object) classes.
The annotations are evenly split into two sets for validation and testing. The
examples and statistics in Fig. 5 illustrate the diverse and complicated
nature of egocentric audio-visual scenes.

##  5 Experiment

Figure 6: Qualitative comparison on Epic Sounding Object dataset. We show
diverse sounding objects in the kitchen scenes in the first column, sounding
objects are annotated in red boxes. Our method outperforms all the compared
works.

Datasets. In our experiments, we use two egocentric datasets. (1) Epic-
Kitchens [13]: The dataset consists of 100 hours of egocentric recordings from
45 kitchen scenes. Thus, diverse kitchen-relevant events and sounds are in the
dataset. We follow the same data split released in their action recognition
benchmark and select 62,413 training videos. We then filter out the silent
videos by measuring the Decibels relative to full scale. This results in
47,214 training videos in total. For evaluation, we use our annotated Epic
Sounding Object dataset to report the results; (2) Ego4D [24]: Ego4D is the
most recent large-scale egocentric video dataset. Besides kitchen scenes, it
includes diverse daily life scenarios. Specifically, it consists of different
subsets serving different benchmarks. We select the “Bristol” subset as it
contains diverse scenarios (_e.g_., entertainment, sports, commuting, and
more) to test our method. We randomly sample and trim 50,000 1-second videos
from this subset and use 90%/10% as the train/test split. A similar filtering
strategy is applied to obtain 26,858 videos for training. We conduct
experiments in Sec. 5.1.3 on this dataset to showcase the generalization
ability of our method.
Evaluation metric. We follow the prior works [28, 11, 38] and adopt the pixel-
level measurement for evaluating localization performance. Given the ground
truth sounding object bounding boxes, we compute the Consensus Intersection
over Union (CIoU) and Area Under Curve (AUC) between the predicted
localization map and ground truth boxes. We report CIoU over a range of
thresholds to expose the finer aspects of comparison.
Implementation details. To facilitate the training, we cut a 1-second long
video around the center of each raw video. We select the middle frame from the
video clip and its four neighboring frames with an interval of 2 between
frames. Consequently, we get $T=5$ frames as visual input. During training,
the frames are first resized to 256$\times$256 and then randomly cropped to
$224\times 224$. During inference, all the frames are directly resized to the
desired size without cropping. For the audio stream, we extract the
corresponding 1-second audio clip to create the audio-visual pairs. The audio
waveform is sub-sampled at 11kHz and transformed into a spectrogram with a
Hann window of size 254 and a hop length of 64. The obtained spectrogram is
subsequently resampled to $128\times 128$ to feed into the audio network. We
set the number of audio and visual feature channels as 512 and choose
$\epsilon=0.5$ and $\tau=0.03$. All models are trained with the Adam
optimizer, with a learning rate of $10^{-4}$ on the visual encoder and
temporal network, while using a learning rate of $10^{-2}$ for updating the
audio encoder.

| CIoU | AUC
---|---|---
@0.2 | @0.3 | @0.4
Attention [65] | 7.12 | - | - | 6.42
STM [38] | 12.10 | 7.64 | 4.01 | 8.87
Hardway [11] | 24.51 | 13.55 | 6.10 | 13.38
SSPL [71] | 13.62 | 8.10 | 4.45 | 9.56
Mix [28] | 26.01 | 15.25 | 9.90 | 15.39
Our | 38.71 | 19.42 | 10.51 | 18.38

Table 2: Quantitative comparison of localization results on Epic Sounding
Object dataset. All methods are re-trained on Epic-Kitchen. The results of
metrics<EMAIL_ADDRESS>0.3, 0.4} and AUC are reported. The top-1 results are
highlighted.

###  5.1 Results

####  5.1.1 Experimental Comparison

To validate the effectiveness of our framework, we compare it with recent
audio-visual localization methods: Attention [65], STM [38], Hardway [11],
SSPL [71] and Mix [28]. Among all the comparative methods, STM [38] utilizes
weak labels for the training, while the other methods are trained with self-
supervision. We hence adjust STM [38] with our self-supervised localization
loss. As all the methods are developed for third-person view videos, we
retrain their methods on our training data for a fair comparison. The
quantitative results are shown in Tab. 2. We can find that our method
outperforms all the compared approaches by a large margin in all metrics,
indicating the benefits of mitigating out-of-view sounds and explicitly
modeling egomotion in learning egocentric audio-visual localization. Moreover,
we provide a qualitative comparison to visually showcase our localization
results. In Fig. 6, we can see that our model produces localization results
that are tight around the ground truth sounding objects.

Model | TM | SL | $L_{dis}$ |<EMAIL_ADDRESS>| AUC
---|---|---|---|---|---
Avg | Max | GA
a |  |  |  |  |  | 27.41 | 15.36
b | ✓ |  |  |  |  | 31.84 | 15.79
c |  | ✓ |  |  |  | 33.29 | 16.10
d |  |  | ✓ |  |  | 37.38 | 16.59
f |  |  | ✓ | ✓ |  | 38.21 | 17.92
g |  |  | ✓ | ✓ | ✓ | 38.71 | 18.38
Table 3: Ablations on GATM, SL, and audio disentanglement module. The top-1
result in each column is highlighted.

####  5.1.2 Ablation Study

We conduct an ablation study to illustrate how each module affects
localization performance. As shown in Tab. 3, we compare our full model with
different baselines — model a: we remove all the modules and only use the
features from the visual and audio encoders to compute the localization map;
model b-d: we insert Average, Max, and Geometry-Aware Temporal modeling
approaches separately in the framework; in model f, we incorporate the soft
localization (SL) into the pipeline; and in model g, we employ the audio
disentanglement module and train the model with $L_{dis}$. By comparing a and
b-c, we found that it’s crucial to aggregate temporal context, while d
emphasizes the importance of GATM in mitigating the egomotion in egocentric
videos. model d vs. f shows that SL slightly enhances the performance since
some of the unrelated visual content can be reweighted. The comparison between
f and g demonstrates that by incorporating audio feature disentanglement, the
localization performance can be further boosted because it can handle the out-
of-view sounds in videos.

Naive baseline. To assess the difficulty of the task, we provide a center box
method that predicts a gaussian heatmap around the center. This results in a
naive baseline of 16.51 compared to 27.41 (model a) from Tab. 3, showing that
there are various challenging scenarios apart from the object being in the
center that this naive baseline cannot capture.

####  5.1.3 Generalization to More Scenarios

The experiments on Epic Sounding Obejct dataset demonstrate the effectiveness
of our method in localizing sounding objects in the egocentric videos. To
further validate the generalization ability of our method, we train our audio-
visual sounding object localization network on Ego4D [24] and qualitatively
showcase the localization results in Fig. 7. The examples are all selected
from the Ego4D test set. We can see that our model can learn audio-visual
associations and localize the sounding objects in diverse scenes.

Figure 7: Localization results on diverse scenarios in Ego4D [24]. Ref.:
Sounding objects; Pred.: predicted localization results.

##  6  Discussions and Conclusions

In this work, we tackle a fundamental task: egocentric audio-visual
localization to promote the field of study in egocentric audio-visual video
understanding. The uniqueness of egocentric videos, such as egomotions and
out-of-view sounds pose significant challenges to learning fine-grained audio-
visual associations. To address these problems, we propose a new framework
with a cascaded feature enhancement module to disentangle visually indicated
audio representations and a geometry-aware temporal modeling module to
mitigate egomotion. Extensive experiments on our annotated Epic Sounding
Object dataset underpin the findings that explicitly mitigating out-of-view
sounds and egomotion can boost localization performance and learn the better
audio-visual association for egocentric videos.

Limitations. The proposed geometry-aware temporal modeling approach requires
geometric transformation computation. For certain visual scenes with severe
illumination changes or drastic motions, the homography estimation may fail.
Then, our GATM will degrade to a vanilla temporal modeling approach. To
mitigate the issue, we can consider designing a more robust geometric
estimation approach.

Potential Applications. Our work offers potential for several applications:
(a) Audio-visual episodic memory. As egocentric video records what and where
of an individual’s daily life experience, it would be interesting to build an
intelligent AR assistant to localize the object (“where did I use it?”) by
processing an audio query, _e.g_., an audio clip of “vacuum cleaner”; (b)
Audio-visual object state recognition. In egocentric research, it is important
to know the state of objects that human is interacting with, while the human-
object interaction often makes a sound. Therefore, localizing objects by
sounds provides a new angle in recognizing an object state; (c) Audio-visual
future anticipation: following the audio-visual object state recognition task,
it’s natural to predict the trajectory of a sounding object by analyzing the
most recent audio-visual clips.

## References

  * [1] Yazan Abu Farha, Alexander Richard, and Juergen Gall.  When will you do what?-anticipating temporal occurrences of activities.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5343–5352, 2018.
  * [2] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman.  Self-supervised learning of audio-visual objects from video.  In European Conference on Computer Vision, pages 208–224. Springer, 2020.
  * [3] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 609–617, 2017.
  * [4] Relja Arandjelovic and Andrew Zisserman.  Objects that sound.  In Proceedings of the European conference on computer vision (ECCV), pages 435–451, 2018.
  * [5] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  Advances in neural information processing systems, 29, 2016.
  * [6] David A Bulkin and Jennifer M Groh.  Seeing sounds: visual and auditory interactions in the brain.  Current opinion in neurobiology, 16(4):415–419, 2006.
  * [7] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
  * [8] Minjie Cai, Kris M Kitani, and Yoichi Sato.  Understanding hand-object manipulation with grasp types and object attributes.  In Robotics: Science and Systems, volume 3. Ann Arbor, Michigan;, 2016.
  * [9] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [10] Alejandro Cartas, Jordi Luque, Petia Radeva, Carlos Segura, and Mariella Dimiccoli.  How much does audio matter to recognize egocentric object interactions?  arXiv preprint arXiv:1906.00634, 2019.
  * [11] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman.  Localizing visual sounds the hard way.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16867–16876, 2021.
  * [12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 720–736, 2018.
  * [13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Rescaling egocentric vision.  arXiv preprint arXiv:2006.13256, 2020.
  * [14] Dima Damen, Teesid Leelasawassuk, and Walterio Mayol-Cuevas.  You-do, i-learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance.  Computer Vision and Image Understanding, 149:98–112, 2016.
  * [15] Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Krishna Ithapu, and Ravish Mehra.  Easycom: An augmented reality dataset to support algorithms for easy communication in noisy environments.  arXiv preprint arXiv:2107.04174, 2021.
  * [16] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.  arXiv preprint arXiv:1804.03619, 2018.
  * [17] Alireza Fathi, Xiaofeng Ren, and James M Rehg.  Learning to recognize objects in egocentric activities.  In CVPR 2011, pages 3281–3288. IEEE, 2011.
  * [18] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [19] Antonino Furnari and Giovanni Maria Farinella.  Rolling-unrolling lstms for action anticipation from first-person video.  IEEE transactions on pattern analysis and machine intelligence, 43(11):4021–4036, 2020.
  * [20] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba.  Music gesture for visual sound separation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10478–10487, 2020.
  * [21] Ruohan Gao, Rogerio Feris, and Kristen Grauman.  Learning to separate object sounds by watching unlabeled video.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 35–53, 2018.
  * [22] Ruohan Gao and Kristen Grauman.  2.5 d visual sound.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324–333, 2019.
  * [23] Ruohan Gao and Kristen Grauman.  Co-separating sounds of visual objects.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3879–3888, 2019.
  * [24] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  arXiv preprint arXiv:2110.07058, 2021.
  * [25] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [26] Di Hu, Feiping Nie, and Xuelong Li.  Deep multimodal clustering for unsupervised audiovisual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9248–9257, 2019.
  * [27] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou.  Discriminative sounding objects localization via self-supervised audiovisual matching.  Advances in Neural Information Processing Systems, 33, 2020.
  * [28] Xixi Hu, Ziyang Chen, and Andrew Owens.  Mix and localize: Localizing sound sources in mixtures.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10483–10492, 2022.
  * [29] Robert A Jacobs and Chenliang Xu.  Can multisensory training aid visual learning? a computational investigation.  Journal of vision, 19(11):1–1, 2019.
  * [30] Hao Jiang and Kristen Grauman.  Seeing invisible poses: Estimating 3d body pose from egocentric video.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3501–3509. IEEE, 2017.
  * [31] J Adam Jones, J Edward Swan, Gurjot Singh, Eric Kolstad, and Stephen R Ellis.  The effects of virtual reality, augmented reality, and motion parallax on egocentric depth perception.  In Proceedings of the 5th symposium on Applied perception in graphics and visualization, pages 9–14, 2008.
  * [32] Kazuhiko Kawamura, A Bugra Koku, D Mitchell Wilkes, Richard Alan Peters, and Ali Sekmen.  Toward egocentric navigation.  International Journal of Robotics and Automation, 17(4):135–145, 2002.
  * [33] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5492–5501, 2019.
  * [34] Daekyum Kim, Brian Byunghyun Kang, Kyu Bum Kim, Hyungmin Choi, Jeesoo Ha, Kyu-Jin Cho, and Sungho Jo.  Eyes are faster than hands: A soft wearable robot learns user intention from the egocentric view.  Science Robotics, 4(26):eaav2949, 2019.
  * [35] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  Advances in Neural Information Processing Systems, 31, 2018.
  * [36] Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman.  Discovering important people and objects for egocentric video summarization.  In 2012 IEEE conference on computer vision and pattern recognition, pages 1346–1353. IEEE, 2012.
  * [37] Cheng Li and Kris M Kitani.  Pixel-level hand detection in ego-centric videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3570–3577, 2013.
  * [38] Sizhe Li, Yapeng Tian, and Chenliang Xu.  Space-time memory network for sounding object localization in videos.  arXiv preprint arXiv:2111.05526, 2021.
  * [39] Yin Li, Alireza Fathi, and James M Rehg.  Learning to predict gaze in egocentric video.  In Proceedings of the IEEE international conference on computer vision, pages 3216–3223, 2013.
  * [40] Yin Li, Miao Liu, and James M Rehg.  In the eye of beholder: Joint learning of gaze and actions in first person video.  In Proceedings of the European conference on computer vision (ECCV), pages 619–635, 2018.
  * [41] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman.  Ego-exo: Transferring visual representations from third-person to first-person videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943–6953, 2021.
  * [42] Yin Li, Zhefan Ye, and James M Rehg.  Delving into egocentric actions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 287–295, 2015.
  * [43] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [44] Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang.  Dual-modality seq2seq network for audio-visual event localization.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2002–2006. IEEE, 2019.
  * [45] Miao Liu, Siyu Tang, Yin Li, and James M Rehg.  Forecasting human-object interaction: joint prediction of motor attention and actions in first person video.  In European Conference on Computer Vision, pages 704–721. Springer, 2020.
  * [46] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [47] Oded Maron and Tomás Lozano-Pérez.  A framework for multiple-instance learning.  Advances in neural information processing systems, 10, 1997.
  * [48] Roberto Martin-Martin, Mihir Patel, Hamid Rezatofighi, Abhijeet Shenoi, JunYoung Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese.  Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments.  IEEE transactions on pattern analysis and machine intelligence, 2021\.
  * [49] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630–2640, 2019.
  * [50] Himangi Mittal, Pedro Morgado, Unnat Jain, and Abhinav Gupta.  Learning state-aware visual representations from audible interactions.  In Proceedings of the European conference on computer vision (ECCV), 2022.
  * [51] Shentong Mo and Pedro Morgado.  A closer look at weakly-supervised audio-visual source localization.  arXiv preprint arXiv:2209.09634, 2022.
  * [52] Shentong Mo and Pedro Morgado.  Localizing visual sounds the easy way.  arXiv preprint arXiv:2203.09324, 2022.
  * [53] Francesca Morganti, Stefano Stefanini, and Giuseppe Riva.  From allo-to egocentric spatial ability in early alzheimer’s disease: a study with virtual reality spatial tasks.  Cognitive neuroscience, 4(3-4):171–180, 2013.
  * [54] Jonathan Munro and Dima Damen.  Multi-modal domain adaptation for fine-grained action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 122–132, 2020.
  * [55] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman.  Grounded human-object interaction hotspots from video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688–8697, 2019.
  * [56] Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grauman.  You2me: Inferring body pose in egocentric video via first and second person interactions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9890–9900, 2020.
  * [57] Curtis Northcutt, Shengxin Zha, Steven Lovegrove, and Richard Newcombe.  Egocom: A multi-person multi-modal egocentric communications dataset.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [58] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018.
  * [59] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba.  Ambient sound provides supervision for visual learning.  In European conference on computer vision, pages 801–816. Springer, 2016.
  * [60] Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianbo Shi.  Egocentric future localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4697–4705, 2016.
  * [61] Ivan Poupyrev, Tadao Ichikawa, Suzanne Weghorst, and Mark Billinghurst.  Egocentric object manipulation in virtual environments: empirical evaluation of interaction techniques.  In Computer graphics forum, volume 17, pages 41–52. Wiley Online Library, 1998.
  * [62] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin.  Multiple sound sources localization from coarse to fine.  In European Conference on Computer Vision, pages 292–308. Springer, 2020.
  * [63] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba.  Self-supervised audio-visual co-segmentation.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2357–2361. IEEE, 2019.
  * [64] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  * [65] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon.  Learning to localize sound source in visual scenes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366, 2018.
  * [66] Silvia Serino, Francesca Morganti, Fabio Di Stefano, and Giuseppe Riva.  Detecting early egocentric and allocentric impairments deficits in alzheimer’s disease: An experimental study with virtual reality.  Frontiers in aging neuroscience, 7:88, 2015.
  * [67] Ladan Shams and Aaron R Seitz.  Benefits of multisensory learning.  Trends in cognitive sciences, 12(11):411–417, 2008.
  * [68] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey.  Understanding human hands in contact at internet scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9869–9878, 2020.
  * [69] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari.  Charades-ego: A large-scale dataset of paired third and first person videos.  arXiv preprint arXiv:1804.09626, 2018.
  * [70] Krishna Kumar Singh, Kayvon Fatahalian, and Alexei A Efros.  Krishnacam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks.  In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–9. IEEE, 2016.
  * [71] Zengjie Song, Yuxi Wang, Junsong Fan, Tieniu Tan, and Zhaoxiang Zhang.  Self-supervised predictive learning: A negative-free method for sound source localization in visual scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3222–3231, 2022.
  * [72] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [73] Charles Spence and Sarah Squire.  Multisensory integration: maintaining the perception of synchrony.  Current Biology, 13(13):R519–R521, 2003.
  * [74] Yu-Chuan Su and Kristen Grauman.  Detecting engagement in egocentric video.  In European Conference on Computer Vision, pages 454–471. Springer, 2016.
  * [75] J Edward Swan, Adam Jones, Eric Kolstad, Mark A Livingston, and Harvey S Smallman.  Egocentric depth judgments in optical, see-through augmented reality.  IEEE transactions on visualization and computer graphics, 13(3):429–442, 2007.
  * [76] Yapeng Tian, Di Hu, and Chenliang Xu.  Cyclic co-learning of sounding object visual grounding and sound separation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2745–2754, 2021.
  * [77] Yapeng Tian, Dingzeyu Li, and Chenliang Xu.  Unified multisensory perception: Weakly-supervised audio-visual video parsing.  In European Conference on Computer Vision, pages 436–454. Springer, 2020.
  * [78] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu.  Audio-visual event localization in unconstrained videos.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 247–263, 2018.
  * [79] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu.  Audio-visual event localization in the wild.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops, 2019.
  * [80] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [81] Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Katerina Fragkiadaki.  Sfm-net: Learning of structure and motion from video.  arXiv preprint arXiv:1704.07804, 2017.
  * [82] Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang.  Interactive prototype learning for egocentric action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8168–8177, 2021.
  * [83] Yu Wu and Yi Yang.  Exploring heterogeneous clues for weakly-supervised audio-visual video parsing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1326–1335, 2021.
  * [84] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang.  Dual attention matching for audio-visual event localization.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6292–6300, 2019.
  * [85] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer.  Audiovisual slowfast networks for video recognition.  arXiv preprint arXiv:2001.08740, 2020.
  * [86] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser.  Dilated residual networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472–480, 2017.
  * [87] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.  The sound of pixels.  In Proceedings of the European conference on computer vision (ECCV), pages 570–586, 2018.
  * [88] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu.  Sep-stereo: Visually guided stereophonic audio generation by associating source separation.  In European Conference on Computer Vision, pages 52–69. Springer, 2020.
  * [89] Yipin Zhou and Tamara L Berg.  Temporal perception and prediction in ego-centric video.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4498–4506, 2015.

## Appendix

##  Appendix A Video Demo

In our video demo, we illustrate our motivation and method. Next, we show (a)
localization results on the Epic Sounding Object dataset, and (b) on the Ego4D
dataset.

##  Appendix B Details of Our Framework

Our framework consists of a visual branch and an audio branch. We start with
elaborating on the audio branch and then move to the visual one.

Figure 8: Details of audio branch, including an encoder $E_{a}$, a
disentanglement network $f$ and a decoder $D_{a}$.

Audio Encoder $E_{a}$ and Decoder $D_{a}$. To extract the audio feature from
input spectrogram $X$ and predict the separation mask, our audio branch
contains an encoder $E_{a}$ and a decoder $D_{a}$ as shown in Fig. 8. In
practice, we design the encoder and decoder in a U-NET style architecture,
i.e., skin connections are enabled. The encoder-decoder network consists of
five convolution layers and five up-convolution layers, and all layers adopt a
4x4 kernel with stride 2. A BatchNorm (BN) layer and a ReLU activation layer
are appended after each convolution/up-convolution layer. For the last layer
in $D_{a}$, we use a Sigmoid layer instead of the ReLU and remove the BN layer
to output the mask. Skip connections allow the information flow from layer $i$
in $E_{a}$ to layer $n-i$ in $D_{a}$, where $n=5$ is the total number of
layers.

Disentanglement Network. The disentanglement network consists of two 1x1
convolution layers to obtain the visually indicated audio features. Given the
audio input features $a$ with a size of 4x4x512, we first tile the visual
feature $g_{v}\in\mathbb{R}^{512}$ by 4x4 times along the Time and Frequency
axes to match the size of $a$. Then we concatenate audio and visual features
along the channel dimension. The concatenated features will go through two 1x1
convolution layers with Leaky ReLU in between. The output feature maps are
also of size 4x4x512.

In the following, we will explain some details about our visual branch.

Visual Encoder $E_{v}$. The visual branch includes a visual encoder $E_{v}$ to
extract the features in the beginning. It takes $T=5$ frames of dimension
224x224x3 as inputs and outputs 5 feature maps of dimension 28x28x512. We use
a pre-trained Dilated ResNet to implement $E_{v}$.

Geometry-Aware Temporal Modeling Module. Given frames $I_{i}$ and $I_{j}$ and
their corresponding features $v_{i}$ and $v_{j}$, we first estimate the
homography transformation $\mathcal{H}_{ji}$ between frames. The homography is
a 3x3 matrix:

$\mathcal{H}_{ji}=\begin{bmatrix}h_{11}&h_{12}&h_{13}\\\
h_{21}&h_{22}&h_{23}\\\ h_{31}&h_{32}&h_{33}\end{bmatrix}$ (12)

$I_{j}$ can thus be warped to $I_{ji}$ as
$I_{ji}=\mathcal{H}_{ji}\otimes{I}_{j}$. Instead of applying homography at the
image level, we use it in the feature space to warp the features. If the
homography estimation fails due to the poor image condition or drastic
viewpoint change, $\mathcal{H}_{ji}$ will be replaced with an identity matrix:

$\mathcal{H}_{ji}=\begin{bmatrix}1&0&0\\\ 0&1&0\\\ 0&0&1\end{bmatrix}$ (13)

Therefore, our GATM will degrade to a vanilla temporal modeling approach but
still can leverage the temporal contexts. The temporal context aggregation is
implemented as single-head attention at the time dimension.

##  Appendix C Analysis on Audio Disentanglement

Figure 9: Visualizations on localization results generated from the model without (column 2-3) and with (column 4-5) audio disentanglement. Our model trained with $L_{dis}$ can yield consistent results when dealing with and without visually irrelevant audio in the input. Model | Input | @0.2 | @0.3 | @0.4
---|---|---|---|---
w/o $L_{dis}$ | $s^{(1)}+s^{(2)}$ | 38.15 | 18.60 | 10.42
w/ $L_{dis}$ | $s^{(1)}+s^{(2)}$ | 38.70 | 19.42 | 10.51
w/o $L_{dis}$ | $s^{(1)}$ | 38.21 | 18.67 | 10.42
w/ $L_{dis}$ | $s^{(1)}$ | 38.71 | 19.42 | 10.51
Table 4: Analysis of our model’s performance by controlling the out-of-view
sounds. We compare two models (trained w/ and w/o $L_{dis}$) and report
results over<EMAIL_ADDRESS>0.3, 0.4}.  Figure 10: More visualization results of
comparative methods and our approach on both Epic Sounding Object dataset
(column 1-5) and Ego4D [24] dataset (column 6-7). The corresponding sounding
objects are shown in red boxes (columns 1 and 6). Our method can produce more
precise localization results and generalize to diverse daily scenarios.

To further investigate the issue of out-of-view sounds in egocentric videos,
we experimented with analyzing the performance of our model when dealing with
audio mixtures. We compare two different models. One is trained with
disentanglement loss $\mathcal{L}_{ids}$ while the other does not. During
inference, we generate two different audio inputs: one is by mixing the
original audio $s^{(1)}$ with an audio clip $s^{(2)}$ sampled from another
video in the dataset, while the other one takes the original audio $s^{(1)}$
as input. The corresponding results are shown in Table 4. We found that with
audio feature disentanglement, our model achieves stable performance for both
mixed and original audio inputs. Visualizations in Fig. 9 also demonstrate the
effectiveness of disentanglement training.

##  Appendix D More Visualizations

We visualize more localization results for examples in the Epic Sounding
Object and Ego4D [24] datasets (shown in Fig. 10). The figure shows that our
framework can correctly localize various sounding objects, e.g., pan, spoon,
box, vacuum cleaner, scissor, car and etc. Both indoor and outdoor scenarios
are covered. An example of people watching independent activity is also shown
(Row 6 for Ego4D results).

##  Appendix E More Experiments

In this section, we provide extra ablations and hyper-parameter experiments to
add to the thoroughness of the evaluation.

Baseline | Baseline + SL | Baseline + $L_{dis}$
---|---|---
27.41 | 28.83 | 32.96
Table 5: Analysis of the efficacy of Soft Localization (SL) and
disentanglement loss $L_{dis}$. We report results on<EMAIL_ADDRESS>metric.

Detailed ablations. We present a more detailed ablation to complement Table 3
in the main paper. Specifically, we add Soft Localization or $L_{dis}$ to the
baseline model before adding the geometry-aware temporal modeling module.
Results are reported in Table 5. The SL module can slightly increase the
performance to 28.83 (5.2%$\uparrow$). When the disentanglement module is
added to the baseline model without GATM, the performance is boosted to 32.96
(20.2%$\uparrow$), which validates the effectiveness of audio disentanglement
for solving out-of-view sounds problem.

$\lambda=1$ | $\lambda=5$ | $\lambda=10$
---|---|---
30.85 | 32.96 | 32.10
Table 6: Hyper-parameter evaluation on the coefficient that controls the
impact of disentanglement loss $L_{dis}$. The results are reported on model
“Baseline + SL”.  Figure 11: An overview of potential applications following
egocentric audio-visual object localization, including (a) audio-visual
episodic memory, (b) audio-visual object state, and (c) audio-visual future
anticipation. These applications target understanding the past, current, and
future of the wearer’s experience. Such capability is enabled by taking fine-
grained audio-visual perception.

Hyper-parameter $\lambda$. In Table 6, we evaluate the effect of $\lambda$
that is used to balance the losses. When $\lambda$ is small, the model is not
sufficiently trained to remove the visually unrelated contents from the audio
representation, yielding inferior results. However, as $\lambda$ becomes
larger ($\lambda=10$), the training objective will focus more on separation
instead of accurate localization. Therefore, we select $\lambda=5$ empirically
for our main model.

$T=3$ | $T=5$ | $T=7$
---|---|---
36.46 | 37.38 | 35.34
Table 7: Hyper-parameter evaluation on the number of frames $T$. The results
are reported on model “Baseline + GATM”.

Hyper-parameter $T$. The number of frames $T$ used to aggregate the temporal
context is important and could further demonstrate the usefulness of the
geometry-aware temporal modeling module. In Table 7, we show quantitative
results with various frame numbers. We can find that aggregating temporal
information is significant as the performance boost from $T=3$ to $T=5$. When
the frame number becomes even larger, the difficulty in effectively aligning
visual frame features increases as greater viewpoint changes happen.
Therefore, we choose $T=5$ in our main experiments.

Figure 12: Example of our annotation interface. We ask every Amazon Mechanic
Turk worker to watch the video first. Then, they are required to answer two
questions and annotate the sounding objects correspondingly. The rules of
selecting sounding objects ensure the quality of annotations. Besides, we
conduct a voting process to obtain precise annotations for each video.

##  Appendix F Potential Applications

Our work has the potential to facilitate a range of applications, which are
described below:

Audio-Visual Episodic Memory. As egocentric video records what and where of an
individual’s daily life experience, it would be interesting to build an
intelligent AR assistant to search the object that has been presented in the
past. Previously, the episodic memory task could take an image or language
query as input to localize the object. The correlation between visual objects
and sound is less explored. Our egocentric audio-visual object localization
task can extend episodic memory with auditory sense. As shown in Fig. 11 (a),
people can ask “where did I use the vacuum cleaner?”, and the AR assistant can
give the answer by feeding an audio query (a vacuum cleaner audio clip from
the web) to the localization network. Therefore, the vacuum cleaner can be
found in the video.

Audio-Visual Object State. In egocentric research, it is important to know the
state of objects that humans interact with. The recognized object’s state
helps understand the wearer’s actions. Interestingly, human-object interaction
often makes a sound. Therefore, localizing objects by sounds provides a new
angle in recognizing an object’s state. For example, in Fig. 11 (b), both the
laptop and the pot make sounds. By feeding the audio and the frame to the
localization model, the laptop and the pot are localized correspondingly,
which indicates that both objects are in a “working” state. Consequently, the
“watch video” and “cooking food using a pot” actions are easily detected.

Audio-Visual Future Anticipation. As the audio-visual object state task
indicate the human activity at the “current” moment by utilizing the sounding
object results, it is natural to predict the future by analyzing the most
recent audio-visual clips. The sound may change continuously as the object’s
state changes. In Fig. 11 (c), when the cooking is completed, the frying pot
sound will be different and hence indicate the wearer’s future action: the
wearer may move the pot to the table or pour food into a bowl. Thus, by
analyzing the audio-visual object state changes, the future of the wearer can
be anticipated.

##  Appendix G Epic Sounding Object Dataset

Amazon Mechanic Turk Annotation Collection. Annotating sounding objects in
egocentric videos is challenging. Sounds are often correlated with human-
object interactions, and sounding objects are sometimes occluded or under
severe deformation due to frequent viewpoint changes. Therefore, annotating
sounding objects automatically is difficult. To this end, we follow a semi-
automatic labeling process by first generating bounding boxes for potential
sounding objects. We use a Mask R-CNN object detector [25] trained on MS-COCO
[43] and a hand-objects detector [68] that pretrained with 42K egocentric
images [13, 42, 69] to produce bounding boxes. Second, we manually annotate
the sounding objects. Due to the above-described difficulty, people may have
different opinions about what objects are emitting sounds. To reduce this
uncertainty, we ask three or more people to annotate the same video and apply
a voting process to the annotations.

Specifically, we take advantage of the Amazon Mechanic Turk to label the
sounding objects. We develop an interface (as shown in Fig. 12) to support
this process. First, the annotator is required to watch the video to ensure
that sounding objects correspond to the sounds. Then the annotator will answer
the following questions: (1) Use one complete sentence to describe how all the
sounds you hear are produced. Sound sometimes is ambiguous to annotate in
bounding boxes when it is made by the interaction between objects, e.g.,
putting down the dish on the table. Whether “only dish” or “both dish and
table” are the sounding object is hard to determine. In this case, a language
description is suitable to handle the ambiguity; (2) Whether the video
contains sound you can hear but cannot see. The answer hints at whether out-
of-view sounds exist in the video. In egocentric videos, out-of-view sounds
might be created due to the limited FoV. Therefore, we include this question
to provide statistical analysis about the out-of-view sound problem; (3) by
watching the video, the annotators are required to select the bounding boxes
that correspond to the objects that emit sounds. The goal is to select the
bounding box that humans recognize as a sounding object. All the collected
answers will be passed to a voting process to determine the final annotations
for each video (see Fig. 5 in the main paper).