# Semi-Supervised and Unsupervised Sense Annotation via Translations

Bradley Hauer, Grzegorz Kondrak, Yixing Luan, Arnob Mallik, Lili Mou
Alberta Machine Intelligence Institute, Department of Computing Science
University of Alberta, Edmonton, Canada
<EMAIL_ADDRESS>
###### Abstract

Acquisition of multilingual training data continues to be a challenge in word
sense disambiguation (WSD). To address this problem, unsupervised approaches
have been proposed to automatically generate sense annotations for training
supervised WSD systems. We present three new methods for creating sense-
annotated corpora which leverage translations, parallel bitexts, lexical
resources, as well as contextual and synset embeddings. Our semi-supervised
method applies machine translation to transfer existing sense annotations to
other languages. Our two unsupervised methods refine sense annotations
produced by a knowledge-based WSD system via lexical translations in a
parallel corpus. We obtain state-of-the-art results on standard WSD
benchmarks.

##  1 Introduction

Word sense disambiguation, the task of identifying the meaning of a word in
context, is one of the central problems in natural language understanding
Navigli (2018). It is a well-studied benchmark for evaluating contextualized
representations of words Loureiro et al. (2021), and is better understood than
tasks such as WiC Pilehvar and Camacho-Collados (2019). Modern WSD methods can
be divided into supervised and knowledge-based approaches. The former depend
on sense-annotated corpora, such as SemCor Miller et al. (1994), while the
latter rely instead on semantic knowledge bases such as WordNet Miller (1995).

While supervised WSD systems typically outperform knowledge-based systems
Scarlini et al. (2020b), their utility is limited by the availability of
sufficiently large sense-annotated corpora for training. This includes systems
based on contextualized embeddings Bevilacqua and Navigli (2020). In
particular, there is a severe lack of high-quality sense-annotated corpora for
languages other than English. This limitation has motivated the development of
methods aimed at automatically disambiguating a large number of word tokens in
a given unannotated corpus, ideally covering a wide range of word and sense
types, while minimizing noise Pasini and Navigli (2017); Scarlini et al.
(2019); Barba et al. (2020). The automatically tagged corpus can then be used
to train a supervised WSD system, satisfying the dependency on training data
without the need for manual annotation.

Following recent theoretical work Hauer and Kondrak (2020) on establishing the
semantic equivalence of mutual translations, we introduce three translation-
based methods for generating sense-tagged corpora. All three methods make use
of lexical knowledge bases, and semantic information obtained from word-level
translations. Semi-supervised LabelProp creates a synthetic parallel corpus
(bitext) by applying machine translation to a monolingual manually-annotated
corpus, and projecting annotations to the target language. Similarly,
unsupervised LabelGen applies a knowledge-based WSD system to the English side
of a bitext, and projects the resulting sense annotations across bitexts onto
other languages. Finally, unsupervised LabelSync produces sense-annotated
corpora in two languages at once by independently applying a knowledge-based
WSD system to each side of a raw bitext, and then refining the initial
annotations based on the confidence scores and multilingual information.

Our experiments on standard WSD test sets demonstrate that the new methods
achieve state-of-the-art results in both semi-supervised and unsupervised
sense annotation. We train two different reference supervised WSD systems on
the generated data, and apply the resulting models to multilingual WSD
benchmarks. Our results compare favourably to models trained on data produced
by the previous state-of-the-art sense annotation methods. Indeed, some of the
results obtained with our unsupervised methods rival those obtained by
training on a manually sense-annotated corpus.

Our contributions are as follows: We present three novel, scalable methods
that can generate annotated corpora for any language for which a suitable
lexical knowledge base is available. We show that these methods achieve state-
of-the-art results on multiple languages. We make our code and corpora
available.111https://www.cs.ualberta.ca/$\sim$kondrak

##  2 Related Work

The sense tagging systems that we consider in this work, including our three
novel methods, can be divided into four types according to two criteria
(Figure 1). The first criterion is whether the method involves supervision in
the form of a sense-annotated corpus. The second criterion is whether the
method operates as a traditional self-contained WSD system, or instead assigns
sense tags to a subset of the words in a corpus which can then be used to
train a supervised WSD system. In this section, we discuss the most relevant
examples of each of the four resulting types.

Supervised WSD systems rely on sense annotations to train disambiguation
models, which are evaluated on benchmark datasets. Examples include GlossBERT
Huang et al. (2019), EWISE Kumar et al. (2019), and EWISER Bevilacqua and
Navigli (2020). Because they require labelled training data in the target
language, such systems are generally impractical for languages other than
English, nor are they directly comparable to our proposed methods.

Knowledge-Based WSD systems remain important due to the limited coverage of
existing annotated corpora, as well as their English bias. These include
graph-based systems such as UKB Agirre et al. (2014), UKB enhanced with
SyntagNet Maru et al. (2019), and systems based on multilingual BERT Devlin et
al. (2019) and BabelNet Navigli and Ponzetto (2012), such as SensEmBERT
Scarlini et al. (2020a). We compare our unsupervised results to both
UKB+SyntagNet and SensEmBERT.

Figure 1: Typology of relevant sense tagging systems. Our own systems are
shown in double-lined boxes.

Semi-Supervised Corpus Tagging systems depend on sense annotated corpora in
one language to produce sense annotations in other languages. The current
state-of-the-art method in this setting is MuLaN Barba et al. (2020), which
propagates sense annotations from SemCor and WordNet Gloss Corpus (WNG,
Langone et al., 2004) to semantically similar contexts in Wikipedia corpora
using contextual word representations from mBERT. Our LabelProp method differs
by leveraging machine translation to directly propagate sense annotations
across word alignment links.

Unsupervised Corpus Tagging systems produce sense annotations “from scratch”.
Train-O-Matic Pasini and Navigli (2017) annotates Wikipedia in multiple
languages by applying the Personalized PageRank (PPR) algorithm to BabelNet.
OneSeC Scarlini et al. (2019) combines Wikipedia categories and BabelNet
synset representations to produce WSD training data, and outperforms Train-O-
Matic. However, both Train-O-Matic and OneSeC annotate nominal instances only,
and hence are not applicable to all-words WSD. On the other hand, EuroSense
Delli Bovi et al. (2017) jointly disambiguates content words of all parts of
speech in a parallel corpus using a knowledge-based WSD system. Our LabelSync
and LabelGen methods differ in that they explicitly leverage lexical
translation information obtained from a bitext.

Other work on using translations for WSD: Resnik and Yarowsky (1999) propose
to distinguish senses only if a “minimum subset” of languages translate them
differently. Apidianaki (2009) demonstrates how senses can be induced by
clustering lexical translations, and proposes an unsupervised WSD system based
on such induced sense inventory and translation information. Lefever et al.
(2011) frame WSD as translation selection, and propose a method based on
multilingual feature vectors. Finally, Taghipour and Ng (2015) annotate
English words with their Chinese translations using manually crafted sense-to-
translation mappings. These methods are not comparable with our work as they
do not link their sense annotations to the WordNet sense inventory, and
therefore are not applicable to modern WSD datasets.

##  3 Semi-Supervised LabelProp

In this section, we introduce LabelProp, a novel label propagation approach
for constructing multilingual sense-annotated corpora. The idea is to
translate a sense-annotated corpus in order to propagate the sense tags across
the translations. No sense-annotated data is required in the target language.
The method is composed of three steps: translation identification, knowledge-
base filtering, and nearest neighbor filtering (Figure 2).

###  3.1 Translation Identification

Given a sense-annotated source corpus, we first translate the corpus into the
target language using pre-trained neural machine translation models. Each
sentence containing at least one source sense-annotated word is translated
independently. If the translation of an annotated source word can be
identified through word alignment, we annotate the translation with the same
BabelNet synset as the aligned source word. This procedure is based on the
assumption that lexical translations in context are semantically equivalent,
and therefore very likely to express the same concept Hauer and Kondrak
(2020).

For alignment, we use BabAlign Luan et al. (2020), a high-precision alignment
tool which leverages translation information from BabelNet to improve on a
base alignment system. In particular, BabAlign augments the input corpus with
lexical translation pairs to bias the aligner towards aligning words which are
mutual translations. It also corrects alignments to maximize the number of
aligned words that share BabelNet synsets. This emphasis on recovering word-
level translation information makes BabAlign particularly well-suited to our
method.

###  3.2 Knowledge-Based Filtering

The sense-projection procedure in the previous step may annotate a word with a
BabelNet synset which does not actually contain that word. These invalid sense
annotations may occur due to non-literal translation (i.e., the word and its
translation do not express the same concept), errors in translation or
alignment, or omissions in BabelNet. Since each sense of a word must
correspond to a specific synset, such invalid annotations are discarded.

Figure 2: LabelProp propagates senses from language L1 to language L2.

###  3.3 Nearest Neighbor Filtering

In order to further increase the precision, we apply a semi-supervised WSD
method to each target translation that is sense annotated by the previous
steps. For each word, we verify that the annotation propagated from the
source-language corpus matches the annotation assigned by the WSD system;
otherwise, we discard that sense annotation.

Our semi-supervised WSD method uses a one-nearest-neighbor approach with ARES
multilingual synset embeddings Scarlini et al. (2020b). We first obtain
contextual word representations of each sense-annotated target translation by
taking the sum of the last four layers of multilingual BERT Devlin et al.
(2019). Since ARES embeddings have twice the size of the original mBERT
embeddings, we concatenate each obtained word representation with itself. We
then compute the cosine similarity between the mBERT representation of the
word, and the ARES representation of each synset containing the word. The
synset that maximizes the similarity is taken as the output of this WSD
system. To reiterate, we retain only the sense annotations from the previous
step that agree with this WSD system.

##  4 Unsupervised Symmetric LabelSync

The LabelProp method, presented in Section 3, is able to leverage existing
sense annotated corpora, such as SemCor, to create comparable sense annotated
corpora in other languages. However, the availability of sense-annotated
corpora in other domains and languages is very limited. On the other hand,
large bitexts are relatively easy to obtain for many language pairs and
domains.

To further reduce the dependency of WSD systems on _any_ pre-existing
annotated data, we introduce LabelSync, a method which annotates both sides of
a given bitext. This method retains the idea of using word alignment to
validate sense annotations, while eschewing the need for a sense annotated
corpus. It is composed of three steps: monolingual word sense disambiguation,
multilingual post-processing, and translation-based filtering (Figure 3).

###  4.1 Monolingual WSD

Our goal is to enrich both sides of the input bitext with sense tags. Since
LabelSync does not assume access to any sense-annotated corpus, we employ a
language-independent knowledge-based WSD system: a variant of UKB enhanced
with SyntagNet Maru et al. (2019). After each side of the bitext is annotated
independently, we have two sense annotated corpora, one in each of the
languages represented in the bitext.

###  4.2 Multi-Lingual Post-Processing

Now that both sides of the bitext are annotated independently, we leverage the
lexical translation information inherent in the bitext to increase the
accuracy of the sense annotations. To improve the performance of our base WSD
system, we employ the SoftConstraint method of Luan et al. (2020). This method
is applicable to any base WSD system which assigns a numerical score, such as
a probability, to each sense of a disambiguated word. Most modern WSD systems,
including UKB, satisfy this property.

The SoftConstraint method depends on word-level translations of each annotated
word, as well as translation information from BabelNet, which is based on the
hypothesis that the translation of a word token provides semantic information
about its sense Hauer and Kondrak (2020). In our case, translation information
is readily available from the bitext. As with our LabelProp method, we use
BabAlign Luan et al. (2020) to word align the bitext. For each sense-annotated
token, the aligned word or phrase is treated as its translation.

The SoftConstraint method can also incorporate sense frequency information, to
bias the annotations toward more probable senses. However, we exclude sense
frequency information from this step, as it provided no discernible benefit in
our development experiments.

Figure 3: LabelSync assigns and refines sense annotations in two languages
simultaneously.

###  4.3 Translation-Based Filtering

In the final step, we aim to further reduce the noise in our sense-annotated
corpora by employing a BabelNet-based filtering method, similar to the one
described in Section 3.2. As before, the key idea is to impose two constraints
on our sense annotations: (1) a word should only be annotated with a synset
that contains the word, and (2) aligned words should be annotated with the
same synset. LabelProp initially guarantees only the latter constraint, so it
has to discard some annotations to ensure the former. In contrast, LabelSync
initially guarantees the first constraint, as UKB can only annotate a word
with a synset containing it. However, since each side of the corpus is
annotated independently, the second constraint may not hold. The final step of
LabelSync is aimed at resolving this problem by synchronizing the sense
annotations across both sides of the bitext.

Unlike Delli Bovi et al. (2017), who leverage embeddings of concepts to filter
questionable annotations, we adopt a binary alignment-based criteria using the
assumption of semantic equivalence of lexical translations. We retain only
those annotations that refer to the same multilingual synset as the sense
annotations of their translations. We also retain annotations if the token
cannot be aligned, or if its translation is not annotated.

##  5 Unsupervised Asymmetric LabelGen

Unlike LabelSync, our second unsupervised method, LabelGen, assumes that the
source language is English, and treats the two sides of the bitext
differently. The goal is to leverage available English resources to improve
WSD performance on other languages, rather than to generate English sense
annotations.

| LabelProp | LabelSync | LabelGen
---|---|---|---
|  Annotated Tokens |  Annotated Word Types |  Sense Types |  Failed Alignments |  Annotated Tokens |  Annotated Word Types |  Sense Types |  Annotated Tokens |  Annotated Word Types |  Sense Types
EN | - | - | - | - | 1,783,334 | 9,509 | 16,748 | - | - | -
IT | 399,569 | 25,361 | 29,290 | 30,763 | 2,083,741 | 10,910 | 22,211 | 1,372,876 | 8,355 | 16,046
ES | 403,797 | 25,874 | 31,420 | 31,640 | 1,692,232 | 10,549 | 25,181 | 1,326,244 | 7,926 | 17,335
FR | 407,590 | 25,193 | 32,129 | 32,181 | 1,458,588 | 7,776 | 11,529 | 1,433,647 | 7,712 | 17,980
DE | 309,926 | 23,786 | 23,433 | 64,085 | 645,289 | 2,139 | 2,756 | 821,552 | 6,589 | 9,121
Table 1: Statistics of the sense-annotated corpora produced by each of our
methods.

###  5.1 English WSD

Given a bitext, we first apply a knowledge-based WSD system to the English
side _only_ , as described in Section 4.1. The lexical information for other
languages is retrieved from BabelNet multi-synsets which are aligned to
WordNet 3.0 synsets. This automatic candidate retrieval process is noisy,
because most BabelNet lexicalizations are automatically generated from various
resources. In addition, while English WordNet contains the sense frequency
estimates from the manually-annotated SemCor, such information is not readily
available for other languages. Hence, WSD annotations are more accurate for
English compared to other languages.

###  5.2 Label Propagation

Having automatically sense-tagged the English side of the bitext, we propagate
the labels to the non-English side using the procedure described in Section
3.1. In effect, we are applying the first part of LabelProp, treating the
English side as a sense-tagged corpus, and the other side as its translation.
At the end of this process, both sides of the bitext are sense-annotated.

###  5.3 Re-Ranking and Filtering

We further refine the sense annotations on the non-English side of the bitext.
We first apply the SoftConstraint method as described in Section 4.2, which
re-ranks the possible senses for each annotated word using the assigned WSD
scores. We then apply the filtering procedure from Section 3.2, which removes
any sense annotations that do not exist in the BabelNet sense inventory.

##  6 Evaluation

Following prior work, we extrinsically evaluate our corpus construction
approaches by providing the generated annotations as training data for
supervised WSD systems (reference systems), which are then evaluated on
standard multi-lingual WSD benchmarks. While our methods could also be applied
to low-resource languages, the current lack of evaluation datasets precludes
such experiments in this work.

###  6.1 Reference Supervised WSD Systems

We perform experiments with two reference supervised WSD systems: (1) IMS
Zhong and Ng (2010) with the most-frequent-sense (MFS) backoff for English,
and (2) _mBERT_ , a transformer-based method, built on multilingual BERT
Devlin et al. (2019), as described by Barba et al. (2020). We use the default
parameter settings and number of training
epochs222https://github.com/edobobo/transformers-wsd. We train each model on
each set of automatically produced sense annotations.

Following prior work, we use the SemEval-2007 dataset Raganato et al. (2017)
as our validation set for the English experiments. Because of the lack of
standard validation sets for non-English languages, we use random samples of
1000 sentences from our training corpora. The hyperparameters of each system
are held constant throughout all experiments.

###  6.2 Test Data

We test the reference WSD models on standard multilingual benchmark datasets:
SemEval-2013 task 12 Navigli et al. (2013), which contains data for Italian,
Spanish, French, and German, and SemEval-2015 task 13 Moro and Navigli (2015),
which covers Italian and Spanish. The SemEval-2013 datasets contain only
nominal instances, while the SemEval-2015 datasets cover nouns, verbs,
adjectives, and adverbs. We use the latest version of the
datasets333https://github.com/SapienzaNLP/mwsd-datasets, which are annotated
with synsets from BabelNet version 4.0.

For the experiments on English (Section 6.4.3), we use the standardized
benchmarks of Raganato et al. (2017)444http://nlp.uniroma1.it/wsdeval, which
comprise all-words test sets from five shared tasks: Senseval2 Edmonds and
Cotton (2001), Senseval3 Snyder and Palmer (2004), SemEval-2007 Pradhan et al.
(2007), SemEval-2013 Navigli et al. (2013), and SemEval-2015 Moro and Navigli
(2015). We also report the average results on the concatenation of all five
test sets, which we refer to as ALL.

###  6.3 Semi-Supervised Approaches

This section is devoted to the empirical evaluation of the LabelProp method
from Section 3.

####  6.3.1 Experimental Setup

We apply LabelProp to a sense-annotated English corpus comprised of SemCor
Miller et al. (1994) and the WordNet Gloss Corpus (WNG) Langone et al. (2004).
Following Luan et al. (2020), we translate each sentence of our English corpus
with Google Translate independently into Italian, Spanish, French, and German.
As described in Section 3.1, we induce word alignment by applying BabAlign
with FastAlign Dyer et al. (2013) as its base aligner. Table 1 contains the
statistics for the corpora created with our methods. “Sense types” indicates
the number of distinct word senses in the corpus. “Failed alignments” refers
to the number of English sense annotations that could not be propagated.

We compare LabelProp to MuLaN Barba et al. (2020), the current state-of-the-
art system for semi-supervised corpus annotation, which also uses SemCor+WNG
as its manually-annotated base English corpus. Specifically, we apply the same
procedure to train the supervised reference system (IMS or mBERT) on the
annotated data produced by each method. We train a single model for each
system, using only the corpus produced by that system for each language, which
limits the impact of language-specific issues. Nevertheless, due to both
software and hardware variables and hyper-parameters, our MuLaN results differ
from those reported in the original paper.

When reporting the results achieved with mBERT, we also include the results of
two recent WSD systems: ARES, using the reported results from Scarlini et al.
(2020b), and 0-Shot WSD, with results replicated using the code provided by
Barba et al. (2020). Since they are not designed to create annotated training
data for other WSD systems, they are not directly comparable to LabelProp.

####  6.3.2 Results

| SemEval-2013 |  | SemEval-2015 |
---|---|---|---|---
Model | IT | ES | FR | DE |  | IT | ES | AVG
MCS | 44.2 | 37.1 | 53.2 | 70.2 |  | 44.6 | 39.6 | 48.2
MuLaN | 65.6 | 65.6 | 68.1 | 69.7 |  | 63.7 | 59.9 | 65.4
LabelProp | 71.4 | 71.0 | 65.1 | 62.8 |  | 67.1 | 64.0 | 66.9
-NN | 70.5 | 70.1 | 62.7 | 63.5 |  | 66.3 | 61.8 | 65.8
-NN -KB | 66.7 | 68.2 | 60.2 | 56.7 |  | 61.2 | 60.2 | 62.2
Table 2: WSD F-score obtained with IMS trained on the corpora generated by MuLaN and LabelProp. | SemEval-2013 |  | SemEval-2015 |
---|---|---|---|---
Model | IT | ES | FR | DE |  | IT | ES | AVG
ARES | 77.0 | 75.3 | 81.2 | 79.6 |  | 71.4 | 70.1 | 75.7
0-shotSC+WNG | 78.3 | 77.6 | 80.8 | 78.3 |  | 70.5 | 68.6 | 75.7
MuLaN | 76.8 | 78.4 | 80.4 | 78.8 |  | 68.7 | 67.8 | 75.2
LabelProp | 78.4 | 78.5 | 80.4 | 77.8 |  | 72.7 | 68.9 | 76.1
Table 3: WSD F-score obtained with mBERT trained on the corpora generated by
MuLaN and LabelProp. Results of two semi-supervised WSD systems are included
for reference.

Table 2 presents the multilingual WSD results obtained by IMS on standard test
sets. While IMS is no longer a state-of-the-art system, it is still commonly
used as a benchmark for evaluating automatically generated corpora Scarlini et
al. (2019). The results demonstrate the relative quality of the generated
corpora: LabelProp is better than MuLaN on Italian and Spanish, as well as on
average. The difference in performance is found to be statistically
significant across all six datasets ($p<0.05$ with McNemar’s test). Neither of
the approaches outperforms the most common sense (MCS) baseline on German,
which we discuss further in Section 6.3.3. The ablation results in the last
two rows show that both the nearest neighbour WSD filter (NN) and the
translation-based filter (KB) improve the quality of the annotations.

Table 3 presents the corresponding results using the more recent mBERT as the
reference system. Our results are slightly better on average than those of
0-shot and ARES. However, only the MuLaN results are directly comparable to
our LabelProp results, as both systems produce training data for a supervised
WSD reference system. LabelProp matches or outperforms MuLaN on every dataset
except German SemEval-2013, and achieves better results on average compared to
the results we replicated. The difference in F-score between LabelProp and our
replicated MuLaN experiment is significant for the SemEval 2015 Italian
dataset ($p<0.05$ with McNemar’s test).

####  6.3.3 Error Analysis

Error analysis suggests two reasons for the relatively low results on the
German data. First, English multi-word compounds often correspond to single
words in German, which makes it difficult to properly propagate English sense
annotations. For example, the two words in giveaway program, which is a
translation of Werbeprogramm, are separately annotated with different senses.
The second issue is the quality of the BabelNet translation coverage. We
observe that among 69,402 BabelNet synsets, that correspond to word senses
appearing in SemCor+WNG, only 40,490 synsets contain at least one German
translation, compared to over 50,000 synsets in each of the other three
languages.

###  6.4 Unsupervised Approaches

In this section, we evaluate our unsupervised methods, LabelSync and LabelGen,
against comparable systems.

####  6.4.1 Experimental Setup

We adopt UKB Agirre et al. (2014) as the base knowledge-based WSD system used
in the first step of both LabelSync and LabelGen to perform the initial
tagging of a bitext. (This base WSD system is not to be confused with the
reference supervised WSD system that is only used for the purpose of corpus
evaluation.) Following Maru et al. (2019), we use WordNet as a lexical
knowledge base, enriching it with information from WNG, and syntagmatic
information from SyntagNet. BabelNet is the source of multilingual
lexicalization information. When applying UKB, the PPRw2w variant of the
personalized PageRank algorithm is run separately for each word, while
concentrating the initial probability mass in the senses of the context words
rather than the focus word.

Both of our unsupervised methods operate on an unannotated bitext. To keep the
corpus size manageable, we randomly sample 200k sentences with English,
French, German, Italian, and Spanish translations from EuroSense Delli Bovi et
al. (2017) discarding its existing sense annotations. This produces four
bitexts with English as one of the languages, which we align at the word level
using BabAlign. The SoftConstraint method employed by LabelSync to refine the
initial sense annotations leverages the lexical translations. Table 1 presents
the statistics of the produced corpora.

The direct competitor of LabelSync and LabelGen is OneSeC Scarlini et al.
(2019), an unsupervised system which produces sense-annotated data by
leveraging the semantic information within Wikipedia categories. Since OneSeC
can only tag nouns, any model trained on a corpus it produces will likewise
only be able to disambiguate nouns. Therefore, we do not apply models trained
on OneSeC to the SemEval-2015 datasets, which include verb, adjective, and
adverb instances. For our multilingual experiments, we also compare to two
knowledge-based WSD systems described in Section 2: UKB with SyntagNet Maru et
al. (2019), and SensEmBERT Scarlini et al. (2020a).

####  6.4.2 Multilingual Results

| SE-2013 |  | SE-2015 |
---|---|---|---|---
Model | IT | ES | FR | DE |  | IT | ES | AVG
MCS | 44.2 | 37.1 | 53.2 | 70.2 |  | 44.6 | 39.6 | 48.2
UKB+SyntagNet | 72.1 | 74.1 | 70.3 | 76.4 |  | 69.0 | 63.4 | 70.9
SensEmBERT | 69.8 | 73.4 | 77.8 | 79.2 |  | - | - | -
OneSeC | 63.5 | 61.6 | 65.1 | 75.8 |  | - | - | -
LabelSync | 75.7 | 78.2 | 72.4 | 75.3 |  | 70.8 | 66.3 | 73.1
LabelGen | 77.8 | 80.5 | 80.7 | 75.4 |  | 68.7 | 66.1 | 74.9
Table 4: WSD F-score obtained with mBERT trained on the corpora generated by
LabelSync and LabelGen.

Table 4 presents the multilingual WSD results when using mBERT as the
reference WSD system. With the consistent exception of German, the results of
mBERT trained on the annotations produced by LabelSync are substantially
better than those trained on the corpus generated by OneSeC, which is the
previous state-of-the-art for unsupervised corpora tagging. Unlike OneSeC, our
unsupervised methods can annotate tokens representing all parts of speech, and
can therefore be applied to the SemEval 2015 datasets. LabelSync also
outperforms both knowledge-based WSD systems, UKB+SyntagNet and SensEmBERT,
and the most common sense (MCS) baseline. LabelGen further improves on
LabelSync by 1.8% on average. This makes it our best performing system, which
sets a new state-of-the-art on the SemEval-2013 Italian, Spanish, and French
datasets.

####  6.4.3 English Results

| SE2 | SE3 | S07 | S13 | S15 | ALL |
---|---|---|---|---|---|---|---
MFS | 66.8 | 66.2 | 55.2 | 63.0 | 67.8 | 65.2 |
SemCor | 71.3 | 69.1 | 61.5 | 65.5 | 68.3 | 68.3 |
EuroSense + SemCor | - | - | - | 66.4 | 69.5 | - |
LabelSync | 69.4 | 64.5 | 57.4 | 71.7 | 72.9 | 68.4 |
Table 5: WSD F-score on all instances obtained with IMS trained on the corpora generated by LabelSync. | SE2 | SE3 | S07 | S13 | S15 | ALL |
---|---|---|---|---|---|---|---
MFS | 66.8 | 66.2 | 55.2 | 63.0 | 67.8 | 65.2 |
SemCor | 74.8 | 73.1 | 64.2 | 69.9 | 74.7 | 72.6 |
LabelSync | 69.6 | 65.9 | 55.2 | 71.4 | 75.1 | 68.9 |
Table 6: WSD F-score on all instances obtained with mBERT trained on the
corpora generated by LabelSync.

In this section, we evaluate LabelSync on English WSD. We do not test LabelGen
on English, as it was specifically designed to tag non-English corpora.
Furthermore, because OneSeC annotates only nominal instances, we conduct
separate all-words and nouns-only experiments.

Table 5 presents the English WSD results on all test instances with IMS as the
reference system. The corpus annotated by LabelSync is a subset of the corpus
annotated by EuroSense. The results of LabelSync on S13 and S15 are much
better than the results using EuroSense augmented with SemCor as reported by
Delli Bovi et al. (2017), which we attribute to the explicit use of
translation information. On the concatenation of all five test sets, the
unsupervised IMS+LabelSync results rival the supervised results of IMS trained
on SemCor, a manually sense-annotated corpus.

In Table 6, we see mBERT performing much better than IMS when trained on
SemCor. Remarkably, the corpus generated in an unsupervised manner by
LabelSync yields results on S13 and S15 that surpass those obtained by
training mBERT directly on SemCor. These results are impressive because
LabelSync makes no use of manual sense annotation. We speculate that this may
be due to the difference in domain between SemCor and the corpus annotated by
LabelSync.

Tables 7 and 8 present the results of our final set of experiments, in which
the reference systems are tested on English nominal instances only. Here, we
can compare LabelSync directly to its competitor, OneSeC. Both of our
reference WSD systems, IMS and mBERT, clearly perform better across all
datasets when trained on the corpus produced by our LabelSync method, compared
to training on the corpus produced by OneSeC. These results establish
LabelSync as the new state of the art for unsupervised English corpus sense
tagging, and a step towards overcoming the knowledge acquisition bottleneck in
WSD.

| SE2 | SE3 | S07 | S13 | S15 | ALL |
---|---|---|---|---|---|---|---
MFS | 72.1 | 72.0 | 65.4 | 63.0 | 66.3 | 67.6 |
SemCor | 76.8 | 73.8 | 67.3 | 65.5 | 66.1 | 70.4 |
OneSeC | 73.2 | 68.2 | 63.5 | 66.5 | 70.8 | 69.0 |
LabelSync | 76.1 | 70.0 | 68.6 | 71.7 | 72.1 | 72.3 |
Table 7:  English WSD F-score on nominal instances obtained with IMS as the reference WSD system,  | SE2 | SE3 | S07 | S13 | S15 | ALL |
---|---|---|---|---|---|---|---
MFS | 72.1 | 72.0 | 65.4 | 63.0 | 66.3 | 67.6 |
SemCor | 79.7 | 75.4 | 67.9 | 69.9 | 75.0 | 74.0 |
OneSeC | 74.2 | 67.1 | 62.9 | 68.8 | 74.2 | 70.2 |
LabelSync | 76.8 | 70.8 | 66.0 | 71.4 | 75.7 | 73.0 |
Table 8:  English WSD F-score on nominal instances obtained with mBERT as the
reference WSD system.

##  7 Conclusion

We have introduced new methods to address the knowledge acquisition bottleneck
in word sense disambiguation in both the semi-supervised and unsupervised
settings. The methods leverage recent advances in machine translation,
alignment, and contextual embeddings. Extrinsic experiments with a variety of
WSD systems demonstrate that the quality of the corpora created by our methods
is substantially higher compared to those produced by prior work. Our methods
for automatic sense tagging can produce annotated corpora for many languages,
and approach the quality of manual annotation in some cases. We make our
corpora available for further research.

One advantage of our unsupervised methods is that they can be applied to
annotate any bitext involving any languages. We posit that our results could
be further improved by annotating corpora with broader domain coverage, or by
matching the domain of the source corpus to the domain of the data to be
disambiguated. We leave this as a direction for future work.

## Acknowledgments

The authors of this paper are listed in alphabetical order. Yixing Luan and
Arnob Mallik conducted the experiments with the semi-supervised and
unsupervised methods, respectively. Bradley Hauer and Grzegorz Kondrak
prepared the final version of the paper.

This research was supported by the Natural Sciences and Engineering Research
Council of Canada (NSERC), and the Alberta Machine Intelligence Institute
(Amii).

## References

  * Agirre et al. (2014) Eneko Agirre, Oier López de Lacalle, and Aitor Soroa. 2014.  Random walks for knowledge-based word sense disambiguation.  _Computational Linguistics_ , 40(1):57–84.
  * Apidianaki (2009) Marianna Apidianaki. 2009.  Data-driven semantic analysis for multilingual WSD and lexical selection in translation.  In _Proceedings of the 12th Conference of the European Chapter of the ACL (EACL)_ , pages 77–85.
  * Barba et al. (2020) Edoardo Barba, Luigi Procopio, Niccolo Campolungo, Tommaso Pasini, and Roberto Navigli. 2020.  MuLaN: Multilingual label propagation for word sense disambiguation.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3837–3844.
  * Bevilacqua and Navigli (2020) Michele Bevilacqua and Roberto Navigli. 2020.  Breaking through the 80% glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2854–2864.
  * Delli Bovi et al. (2017) Claudio Delli Bovi, Jose Camacho-Collados, Alessandro Raganato, and Roberto Navigli. 2017.  Eurosense: Automatic harvesting of multilingual sense annotations from parallel text.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 594–600.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013.  A simple, fast, and effective reparameterization of IBM model 2.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–648.
  * Edmonds and Cotton (2001) Philip Edmonds and Scott Cotton. 2001.  Senseval-2: Overview.  In _Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems_ , pages 1–5.
  * Hauer and Kondrak (2020) Bradley Hauer and Grzegorz Kondrak. 2020.  Synonymy = translational equivalence.  _arXiv preprint arXiv:2004.13886_.
  * Huang et al. (2019) Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019.  GlossBERT: BERT for word sense disambiguation with gloss knowledge.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3507–3512.
  * Kumar et al. (2019) Sawan Kumar, Sharmistha Jat, Karan Saxena, and Partha Talukdar. 2019.  Zero-shot word sense disambiguation using sense definition embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5670–5681.
  * Langone et al. (2004) Helen Langone, Benjamin R Haskell, and George A Miller. 2004.  Annotating WordNet.  In _Proceedings of the Workshop On Frontiers In Corpus Annotation_ , pages 63–69.
  * Lefever et al. (2011) Els Lefever, Véronique Hoste, and Martine De Cock. 2011.  ParaSense or how to use parallel corpora for word sense disambiguation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 317–322.
  * Loureiro et al. (2021) Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021.  Analysis and evaluation of language models for word sense disambiguation.  _Computational Linguistics_ , pages 1–55.
  * Luan et al. (2020) Yixing Luan, Bradley Hauer, Lili Mou, and Grzegorz Kondrak. 2020.  Improving word sense disambiguation with translations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4055–4065.
  * Maru et al. (2019) Marco Maru, Federico Scozzafava, Federico Martelli, and Roberto Navigli. 2019.  SyntagNet: Challenging supervised word sense disambiguation with lexical-semantic combinations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3534–3540.
  * Miller (1995) George A Miller. 1995.  WordNet: A lexical database for English.  _Communications of the ACM_ , 38(11):39–41.
  * Miller et al. (1994) George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994.  Using a semantic concordance for sense identification.  In _Proceedings of a Workshop on Human Language Technology_ , pages 240–243. Association for Computational Linguistics.
  * Moro and Navigli (2015) Andrea Moro and Roberto Navigli. 2015.  Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking.  In _Proceedings of the 9th International Workshop on Semantic Evaluation_ , pages 288–297.
  * Navigli (2018) Roberto Navigli. 2018.  Natural language understanding: Instructions for (present and future) use.  In _IJCAI_ , pages 5697–5702.
  * Navigli et al. (2013) Roberto Navigli, David Jurgens, and Daniele Vannella. 2013.  Semeval-2013 task 12: Multilingual word sense disambiguation.  In _Proceedings of the Seventh International Workshop on Semantic Evaluation_ , pages 222–231.
  * Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012.  BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network.  _Artificial Intelligence_ , 193:217–250.
  * Pasini and Navigli (2017) Tommaso Pasini and Roberto Navigli. 2017.  Train-o-matic: Large-scale supervised word sense disambiguation in multiple languages without manual training data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 78–88.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of NAACL-HLT_ , pages 1267–1273.
  * Pradhan et al. (2007) Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007.  Semeval-2007 task-17: English lexical sample, srl and all words.  In _Proceedings of the Fourth International Workshop on Semantic Evaluations_ , pages 87–92.
  * Raganato et al. (2017) Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017.  Word sense disambiguation: A unified evaluation framework and empirical comparison.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 99–110.
  * Resnik and Yarowsky (1999) Philip Resnik and David Yarowsky. 1999.  Distinguishing systems and distinguishing senses: New evaluation methods for word sense disambiguation.  _Natural Language Engineering_ , 5(2):113–133.
  * Scarlini et al. (2019) Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2019.  Just “OneSeC” for producing multilingual sense-annotated data.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 699–709.
  * Scarlini et al. (2020a) Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020a.  SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation.  In _Proceedings of the Thirty-Fourth Conference on Artificial Intelligence_.
  * Scarlini et al. (2020b) Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020b.  With more contexts comes better performance: Contextualized sense embeddings for all-round word sense disambiguation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 3528–3539.
  * Snyder and Palmer (2004) Benjamin Snyder and Martha Palmer. 2004.  The English all-words task.  In _Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text_ , pages 41–43.
  * Taghipour and Ng (2015) Kaveh Taghipour and Hwee Tou Ng. 2015.  One million sense-tagged instances for word sense disambiguation and induction.  In _Proceedings of the Nineteenth Conference on Computational Natural Language Learning_ , pages 338–344.
  * Zhong and Ng (2010) Zhi Zhong and Hwee Tou Ng. 2010.  It makes sense: A wide-coverage word sense disambiguation system for free text.  In _Proceedings of the ACL 2010 System Demonstrations_ , pages 78–83.


# Pedestrain detection for low-light vision

Zhipeng Chang(zchang)
Ruiling Ma(ruilin3)
Wenliang Jia(wenliang)

##  1 Short Abstract

The demand for pedestrian detection has created a challenging problem for
various visual tasks such as image fusion. As infrared images can capture
thermal radiation information, image fusion between infrared and visible
images could significantly improve target detection under environmental
limitations. In our project, we would approach by pre-processing our dataset
with image fusion technique, then using Vision Transformer (ViT) [5] model to
detect pedestrians from the fused images. During the evaluation procedure, a
comparison would be made between YOLOv5 and the revised ViT model’s
performance on our fused images.

##  2 Brief summary of what exists

Pedestrian detection for low-light vision has been extensively researched and
developed. There are a lot of image fusion and object detection techniques
that already exist, providing a wide range of options for researchers.

###  2.1 Image fusion

####  2.1.1 Jia et al [8] tested several image fusion techniques in their
work, including GTF, FusionGAN, Densefuse, and IFCNN:

  1. 1.

Gradient Transfer Fusion is a method based on gradient transfer and total
variation minimization. The method involves transferring the gradient
information from the visible image to the infrared image, which is then fused
with the original visible image using total variation minimization. This
process helps to preserve the edges and fine details of the visible image
while enhancing the overall contrast and texture of the fused image. The
technique has been shown to produce high-quality results and outperform other
state-of-the-art fusion methods in terms of both visual quality and
quantitative metrics [12].

  2. 2.

FusionGAN is an end-to-end image fusion technique that uses a generative
adversarial network (GAN) to combine infrared and visible images, which has
two networks: a generator network and a discriminator network. The generator
network is responsible for generating a fused image that combines significant
infrared intensities with additional visible gradients, while the
discriminator network ensures that the fused image has more details from the
visible image. Unlike traditional methods, FusionGAN does not require manually
designing complex activity level measurements or fusion rules. This approach
has been demonstrated to produce high-quality fused images with superior
visual quality and improved detail preservation [13].

  3. 3.

DenseFuse utilizes a dense block architecture and multi-scale fusion to
preserve fine details and enhance image quality. The approach consists of
three parts: an encoder, a fusion layer, and a decoder. The input of the
encoder is the source images, which are infrared and visible images. The
encoder then utilizes CNN layer and dense block to obtain feature maps, which
are fused using the addition and L1-norm fusion strategy. After the fusion
layer, the fused feature maps are integrated into one feature map that
contains salient features from both source images, and the fused image is
reconstructed using the decoder network. Experimental results demonstrate that
DenseFuse exhibits state-of-the-art fusion performance [11].

  4. 4.

IFCNN is a general image fusion framework based on convolutional neural
network. IFCNN consists of three modules: feature extraction, feature fusion,
and image reconstruction. The feature extraction module uses pre-trained CNN
models to extract features from the source images. The feature fusion module
combines the features using an appropriate fusion rule based on the input
images. The image reconstruction module employs two convolutional layers to
generate the fused image. The proposed IFCNN framework outperforms other
state-of-the-art methods in terms of both objective metrics and visual quality
for various image fusion tasks [23].

####  2.1.2 Linfeng et al [19] have also proposed a new image fusion technique
called SeAFusion:

  1. 1.

SeAFusion is a real-time infrared and visible image fusion framework using
semantic awareness. In order to enhance the description ability of the fusion
network for fine-grained details, a gradient residual dense block was
employed, along with an elaborate content loss, it’s able to effectively
maintain salient target intensity and preserve texture detail. Additionally, a
semantic loss was introduced to allow high-level semantic information to flow
back to the image fusion module, improving performance in high-level vision
tasks. A joint low-level and high-level adaptive training strategy was also
introduced in order to achieve impressive performance in both image fusion and
various high-level vision tasks [19].

Figure 1: The overall framework of the proposed semantic-aware infrared and
visible image fusion algorithm [19]

###  2.2 Object detection

####  2.2.1 Jia et al mainly tested YOLOv3 and YOLOv5: [8]

  1. 1.

YOLO (You Only Look Once) is an object detection system that uses a single
convolutional neural network to predict bounding boxes and class probabilities
for objects in an image. YOLOv3 is an improved version of YOLO that uses
darknet-53 as the backbone architecture, resulting in better performance than
previous versions. YOLOv5 is a more recent version of YOLO that uses a new
CSPDarknet53 with a focus on model size reduction and improved speed. YOLOv5
achieves better performance than YOLOv3 [9].

####  2.2.2 A new object detection technique called Vision Transformer (ViT)
has been introduced and extensively studied in recent years

  1. 1.

ViT [5] is an architecture based on the transformer architecture, originally
developed for natural language processing tasks. According to the authors,
this architecture is successful at image recognition tasks and achieves
competitive results on standard benchmarks.

###  2.3 Dataset

We will be using the LLVIP dataset presented by Jia et al [8]. This dataset
consists of 15,488 pairs of images that are precisely aligned in both time and
space. It includes a significant number of pedestrians and most images are
captured under low-light conditions.

##  3 Project Plan identifying what aspects are novel

Pedestrian detection from visible images, infrared images, and fused images is
a well-developed process. However, there is currently no study that has
utilized ViT as the main object detection technique. In our project, we aim to
use ViT as the primary object detection technique to detect pedestrians from
fused images. Therefore, the process of fusing images and then using ViT to
detect pedestrians is novel.

##  4 Milestones

There are two major milestones for this project:

  1. 1.

Selecting an appropriate image fusion technique for the project, and fuse the
images

  2. 2.

Using ViT to detect pedestrians from fused images

##  5 Breakdown of who will do what

  1. 1.

Setting up the necessary infrastructure and reproducing the existing work
described in relevant papers to establish a baseline: Zhipeng Chang

  2. 2.

Selecting an appropriate image fusion technique and fuse testing images:
Wenliang Jia

  3. 3.

Utilizing ViT technique to detect pedestrians: Ruiling Ma

##  6 Literature Review

Medical image registration involves several techniques, including control-
point-based registration, moment-based registration, contour-based
registration, and optimization-based registration. Cheng et al. used the
contour-based registration method to segment the upper airway, as it typically
produces accurate results. To track the contour of the airway, they developed
a method that uses modified GVF snakes with extra edge detection and snake
shifting techniques, along with the contour of an adjacent CT slice to track
the current slice. One key advantage of their method is that it is fully
automated, making it more efficient than other methods. Medical experts
evaluated their results for accuracy and found them to be acceptable,
highlighting the success of their approach [4].

The cost of setting up per-patient based environments is one of the challenges
in current rehabilitation methods. To overcome this issue, Rossol et al.
developed a Virtual Reality solution using a stand-alone open-source 3D
application available on multiple platforms. They also developed an algorithm
based on Bayesian networks that can adapt the training simulation based on
each patient’s skill level. This intelligent adaptation of the training
simulation reduces the high costs associated with traditional rehabilitation
methods. The effectiveness of their solution was evaluated through small-scale
user evaluations, which showed that their proposed application outperforms
traditional power wheelchairs. However, the authors acknowledge that other
factors should also be considered in their study [15].

Accurate localization and tracking of facial features are essential for
developing high-quality model-based coding systems, such as MPEG-4. Bernogger
et al. have introduced a new algorithm that significantly improves the
accuracy of well-known facial feature detection and tracking algorithms.
Additionally, they have presented animation models for eye movements to better
visualize their results. The process involves synthesizing eye movement by
computing the deformation of the 3D model’s eyes using the extracted eye
features. First, they create an individualized 3D face model and map it to the
first frame of the face sequence. Then, they apply deformation parameters to
the 3D model based on the extracted eye features to synthesize eye movement in
successive frames. They evaluate the developed algorithms through experiments
using various color images of different eyes and eye sequences. The results
are promising for synthesizing eye movements, indicating the effectiveness of
the proposed approach [3].

Due to the limitations of existing conventional imaging systems, capturing
images or videos with a large dynamic range often leads to overexposed or
underexposed results. Additionally, current approaches may face artifacts in
their reconstructions when there is a high contrast or colored light source in
the scene. To address these issues, the author proposes a novel neural network
model that combines self-attention mechanisms, adversarial training, and a
tailored loss function. These methods help the network take advantage of
location interdependency and minimize artifacts. The author’s image
enhancement model outperformed other recent methods in terms of image
naturalness and adaptability to various lighting conditions, as demonstrated
through full-reference and non-reference image quality assessments [22].

Camera calibration is essential for the process of relating a 2-D image to a
3-D environment. Existing linear techniques are simpler to implement but
cannot model camera distortions, while non-linear methods can consider
complicated imaging models but require computationally expensive search
procedures. In this paper, the authors present two alternative algorithms. The
first method uses perspective distortions, but it is not reliable for real or
synthetic data with low levels of noise. The second method is introduced to
address this problem, and the article demonstrates that it produces reliable
estimates for synthetic data with high levels of noise and real scenes. The
proposed methods have achieved both mathematical simplicity and high accuracy
results when synthetic data was used. By using these algorithms, an active
camera can automatically calibrate itself [2].

The demand for imaging capabilities in various industries has led to the need
to fuse infrared images with visible light. Although the benefits of infrared
images and visible light imaging techniques can be combined through fusion,
existing techniques struggle to maintain details. To address this issue, this
paper proposes a multi-scale Gaussian rolling guidance filter decomposition-
based method for fusing infrared and visible images. Initially, the source
images are separated into three layers: the detail preservation layer, the
edge preservation layer, and the energy base layer. Based on the unique traits
of each layer, three fusion strategies are employed. The final fusion image is
produced by combining the results of the three separate scales, resulting in
additional valuable scene information. The author demonstrates that the
proposed fusion framework outperforms most evaluations, based on subjective
and objective assessments of multiple sets of fusion outcomes. However, the
method’s comparatively high complexity results in inferior real-time
performance [21].

Motion capture data can effectively produce skeletal animation, and an
efficient compression algorithm can guarantee the quality of the animation
while transmitting the data. However, some coding techniques require high
processing resources for compression or decompression and may not yield good
results if the distortion metric is not dependent on the joint hierarchy. To
address this issue, Firouzmanesh, Cheng, and Basu proposed an efficient
encoding and decoding technique for lossy compression of motion capture data.
This method takes human perception into consideration and focuses on bone
lengths and variation in rotation. It guarantees that the motion data is
preserved and transmitted in an equivalent or better perceptual quality under
limited bandwidth [7].

As smart rooms continue to develop, efficient algorithms are required to
achieve more precise human motion analysis and pose recognition. Currently,
pose recognition can be generally separated into model-based systems and non-
model based systems. To improve the accuracy of pose recognition, Singh,
Mandal, and Basu introduced two pattern recognition methods, the Hough
transform (HT) and the Radon transform (RT), into their algorithm. However,
the use of HT can result in distortion in the shape and location of peaks in
transform space, as well as problems while implementing the discrete HT via
accumulator bins. Therefore, Singh, Mandal, and Basu proposed using the RT in
their human poses and gestures recognition algorithm. The experimental results
indicated that this algorithm guarantees a higher recognition rate for human
arm and leg poses without normalizing the image in the pre-processing stage
[18].

Intraspinal microstimulation (ISMS) involves inserting microwires from the
back of the spinal cord to stimulate nerve cells and help people restore some
leg functions after an injury. However, there are still some potential risks
associated with using ISMS that cannot be ignored. To help clinicians minimize
damage to the spinal cord system, computed tomography or MRI scans are used to
identify an optimal path for microwire implantation. In the preprocessing
stage, Mukherjee et al. proposed an automatic spinal cord image segmentation
technique to assist with surgical planning. Compared to a semi-automatic
spinal cord image segmentation technique, the proposed method uses active
tracing of the segment boundary to provide a better-quality vision of both the
region of interest and its neighbours [14].

Yin and Basu proposed a three-step method for implementing a feature detection
method on facial organ areas. The first step involves limiting the facial
feature detection region to certain areas, and a two-stage region-growing
algorithm is used to make the detection less sensitive to noise. In the first
stage, a large global growing threshold is used to explore more skin areas,
while in the second stage, region information is estimated in the individual
feature areas. The second step involves using pre-defined templates to extract
the shape of the nostril and nose-side. Finally, after detecting the facial
features and extracting the nose’s shape, a 3D wireframe model is used to
track the motion of facial expressions. The experimental results demonstrate
that the proposed method can detect most nose features correctly [20].

The recognition of human action and group activity presents one of the biggest
challenges in video understanding. To achieve a better comprehension of
individual actions and group activity, Actor Relation Graph (ARG)-based group
activity recognition is commonly employed. In order to improve the
understanding of a group’s activity in a single video scene, Kuang and Tie
suggested using MobileNet in the CNN layer and Normalized Cross-Correlation
and Sum of Absolute Differences to construct ARG. Moreover, they introduced a
visualization model that uses bounding boxes on each human object to recognize
and predict human action and group activity. Experimental results indicated
that this method has the potential to improve the accuracy and speed of group
activity recognition. However, Kuang and Tie acknowledged that further
improvements are necessary to precisely identify individual actions [10].

To effectively display panoramic images in a telepresence system, Baldwin,
Basu, and Zhang utilized a predictive Kalman filter. This system integrates a
panoramic imaging system with a viewing direction predicted by the Kalman
filter. The research by Baldwin, Basu, and Zhang presents experimental results
from the prediction process and operator experience. They emphasized that
improving the prediction process and obtaining better quantitative results
require appropriate error modelling and parameter identification. When
transmitting images from a remote location with delays, the system uses local
image information to simulate a continuously flowing sequence of pictures [1].

Detecting straight lines using a pinhole camera model becomes difficult in
cameras where the mirror system does not meet the Single Viewpoint (SVP)
criteria. To solve this problem and detect straight-line features in panoramic
non-SVP images, Fiala and Basu proposed an approach that uses the Panoramic
Hough transform. The transform is specifically designed for catadioptric
panoramic sensors with spherical mirrors. According to experimental results,
this approach can identify horizontal lines to complement the trivial
detection of vertical lines, and it can robustly detect straight-line features
with estimated calibration [6].

Shen, Cheng, and Basu developed an algorithm for multi-exposure fusion using
the hierarchical multivariate Gaussian conditional random field model to
enhance its performance. The model integrates two perceptual quality metrics:
perceived local contrast and color saturation. To preserve details, the model
calculates contrast in the luminance channel of the LHS color space. Moreover,
to ensure satisfactory results for color images, the model uses color
saturation to measure the colorfulness of a pixel, representing the red,
green, and blue components in the RGB color space. The experiment showed that
the proposed algorithm significantly improves the quality of fused images,
providing viewers with a better quality of experience [17].

Due to the projected increase of the senior population to two billion by 2050,
there is a high demand for smart home applications that include fall
detection. Several options for human fall detection exist, such as wearable
devices, floor pressure sensors, or single RGB, multiple cameras, or depth
cameras. Hajari and Cheng proposed a real-time automatic fall detection
technique that uses computer vision to analyze motion deformation and velocity
of the action, with a single RGB camera. This method extracts the human head
and center of mass from each video frame and uses a robust ROI detection
technique to detect areas of interest. To detect falls, a set of thresholds
and the vector of angle size and angular velocity are used. Experimental
results demonstrate that this method is reliable and is not affected by
lighting changes or camera lens distortion [16].

## References

  * [1] Jonathan Baldwin, A. Basu, and Hong Zhang.  Panoramic video with predictive windows for telepresence applications.  volume 3, pages 1922–1927, 01 1999.
  * [2] Anup Basu.  Active calibration of cameras: theory and implementation.  IEEE Transactions on Systems, man, and cybernetics, 25(2):256–265, 1995.
  * [3] S. Bernogger, Lujun Yin, A. Basu, and A. Pinz.  Eye tracking and animation for mpeg-4 coding.  In Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170), volume 2, pages 1281–1284 vol.2, 1998.
  * [4] Irene Cheng, Sharmin Nilufar, Carlos Flores-Mir, and Anup Basu.  Airway segmentation and measurement in ct images.  In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 795–799. IEEE, 2007.
  * [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [6] Mark Fiala and A. Basu.  Hough transform for feature detection in panoramic images.  Pattern Recognition Letters, 23:1863–1874, 12 2002.
  * [7] Amirhossein Firouzmanesh, Irene Cheng, and A. Basu.  Perceptually guided fast compression of 3-d motion capture data.  Multimedia, IEEE Transactions on, 13:829 – 834, 09 2011.
  * [8] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou.  Llvip: A visible-infrared paired dataset for low-light vision.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3496–3504, 2021.
  * [9] Aicha Khalfaoui, Abdelmajid Badri, and Ilham EL Mourabit.  Comparative study of yolov3 and yolov5’s performances for real-time person detection.  In 2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET), pages 1–5, 2022.
  * [10] Zijian Kuang and Xinran Tie.  Improved actor relation graph based group activity recognition.  arXiv preprint arXiv:2010.12968, 2020.
  * [11] Hui Li and Xiao-Jun Wu.  Densefuse: A fusion approach to infrared and visible images.  IEEE Transactions on Image Processing, 28(5):2614–2623, May 2019\.
  * [12] Jiayi Ma, Chen Chen, Chang Li, and Jun Huang.  Infrared and visible image fusion via gradient transfer and total variation minimization.  Information Fusion, 31:100–109, 2016.
  * [13] Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang.  Fusiongan: A generative adversarial network for infrared and visible image fusion.  Information Fusion, 48:11–26, 2019.
  * [14] Dipti Mukherjee, Irene Cheng, Nilanjan Ray, Vivian Mushahwar, Marc Lebel, and A. Basu.  Automatic segmentation of spinal cord mri using symmetric boundary tracing.  Information Technology in Biomedicine, IEEE Transactions on, 14(5):1275 – 1278, 10 2010.
  * [15] Nathaniel Rossol, Irene Cheng, Walter F. Bischof, and Anup Basu.  A framework for adaptive training and games in virtual reality rehabilitation environments.  In Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, VRCAI ’11, page 343–346, New York, NY, USA, 2011. Association for Computing Machinery.
  * [16] Caroline Rougier, Jean Meunier, Alain St-Arnaud, and Jacqueline Rousseau.  Robust video surveillance for fall detection based on human shape deformation.  Circuits and Systems for Video Technology, IEEE Transactions on, 21:611 – 622, 06 2011.
  * [17] Rui Shen, Irene Cheng, and A. Basu.  Qoe-based multi-exposure fusion in hierarchical multivariate gaussian crf.  IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, 22, 12 2012.
  * [18] Meghna Singh, Mrinal Mandal, and Anup Basu.  Pose recognition using the radon transform.  In 48th Midwest Symposium on Circuits and Systems, 2005., pages 1091–1094. IEEE, 2005.
  * [19] Linfeng Tang, Jiteng Yuan, and Jiayi Ma.  Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network.  Information Fusion, 82:28–42, 2022.
  * [20] Lijun Yin and A. Basu.  Nose shape estimation and tracking for model-based coding.  In Proceedings of the Acoustics, Speech, and Signal Processing, 2001\. on IEEE International Conference - Volume 03, ICASSP ’01, page 1477–1480, USA, 2001. IEEE Computer Society.
  * [21] Jiajia Zhang, Pei Xiang, Xiang Teng, Xin Zhang, and Huixin Zhou.  Infrared and visible image fusion based on multi-scale gaussian rolling guidance filter decomposition.  In Smart Multimedia: Third International Conference, ICSM 2022, Marseille, France, August 25–27, 2022, Revised Selected Papers, pages 75–91. Springer, 2022.
  * [22] Shupei Zhang, Kangkang Hu, Zhenkun Zhou, and Anup Basu.  Lighting enhancement using self-attention guided hdr reconstruction.  In Smart Multimedia: Third International Conference, ICSM 2022, Marseille, France, August 25–27, 2022, Revised Selected Papers, pages 409–418. Springer, 2022.
  * [23] Yu Zhang, Yu Liu, Peng Sun, Han Yan, Xiaolin Zhao, and Li Zhang.  Ifcnn: A general image fusion framework based on convolutional neural network.  Information Fusion, 54:99–118, 2020.