# Domain-adaptation of spherical embeddings

Mihalis Gongolidis1, Jeremy Minton1, Ronin Wu1,
Valentin Stauber1, Jason Hoelscher-Obermaier1 and Viktor Botev1
1Iris AI, Bekkestua, Norway

Domain adaptation of embedding models, updating a generic embedding to the
language of a specific domain, is a proven technique for domains that have
insufficient data to train an effective model from scratch. Chemistry
publications is one such domain, where scientific jargon and overloaded
terminology inhibit the performance of a general language model.

The recent spherical embedding model (JoSE) proposed in Meng et al. (2019)
jointly learns word and document embeddings during training on the multi-
dimensional unit sphere, which performs well for document classification and
word correlation tasks. But, we show a non-convergence caused by global
rotations during its training prevents it from domain adaptation.

In this work, we develop methods to counter the global rotation of the
embedding space and propose strategies to update words and documents during
domain specific training. Two new document classification data-sets are
collated from general and chemistry scientific journals to compare the
proposed update training strategies with benchmark models. We show that our
strategies are able to reduce the performance cost of domain adaptation to a
level similar to Word2Vec.

## Embedding space tumbling

A global rotation of the embedding space was observed during training and can
be confirmed by the magnitude of updates between epochs, which converge to a
non-zero value. While the impact this has on the quality of an embedding is
empirically bounded by task performance, it is detrimental to domain
adaptation. Any document or word vector that is not updated will become
misaligned by the global rotation, which is much larger than the typical
adaptation to specific domains.

Investigation revealed a discrepancy between the method described in Meng et
al. (2019) and the provided code. Specifically, the adjustment factor,
$d_{\mathrm{cos}}$, in the gradient retraction onto the spherical space
differs. It can be shown that the factors described in the paper reduce to $0$
or $1$. The $d_{\mathrm{cos}}$ equations implemented in the supplied code do
not, but we could not find justification for those values. It was decided to
drop the $d_{\mathrm{cos}}$ factor entirely and this halved the update
magnitude the model converge to.

This change is a significant improvement, but does not entirely eliminate the
global rotation so a post-processing solution is introduced. A counter-
rotation of the embedding space to minimize the sum of angles between its
vectors and reference embedding can be determined with a singular value
decomposition. This is a feasible approach to eliminate global rotation
relative to a reference embedding, chosen to be after a given number of
epochs.

## Document embedding strategies

The propos A Word2Vec(Mikolov et al., 2013) model is used as a benchmark,
where word vectors are averaged in place of the document vector.

ed loss function in the original spherical embeddings model as discussed in
Meng et al. (2019) does not allow continuation of training for document
vectors. To enable it when adapting the generic embeddings model with a
domain-specific corpus, two approaches are considered:

  1. 1.

In the first approach, we compress each document to $n$ keywords, selected as
the most cosine similar words to the document vector. This compressed
representation is added as cosine similarities to the loss function so the
generic document vectors are updated during domain adaptation. This makes the
loss function

$L_{u\in G\cap D}=L_{Global}+L_{Local}+\alpha\sum_{d\in
P_{u}}\mathrm{max}(0,m-cos(d,u)+cos(d,n))$ (1)

where the nomenclature follows Equation 3 of Meng et al. (2019) as well as the
newly introduced terms $P_{u}$ representing the set of document vectors a word
vector, $u$, is represented with and a weighting parameter $\alpha$.

An advantage of this method, is that keywords that only appear in the original
corpus continue to be updated.

  2. 2.

The second approach constructs the document vectors from an average of word
vectors, replacing, $d_{i}$, in the loss function with
$\frac{1}{|D_{i}|}\sum_{n\in D_{i}}u_{n}$. This is more efficient than the
previous approach because the dynamically constructed document vectors are
efficiently computed and do not need to be persisted and updated.

We will compare these approaches with document classification, using a
k-Nearest Neighbours (kNN) classifier ontop of the document vectors from each
model.

Two new data-sets, from https://core.ac.uk/, were collated to test these
embedding models:

BioChem

Approximately 280 thousand biochemical research abstracts across 18
biochemistry subtopics based on the author’s keywords. The subtopics are
evenly distributed in both train and update corpora.

Core-general

Approximately 210 thousand research abstracts from 14 topics and three
chemistry related topics. The update corpus is composed of the three chemistry
topics and the train corpus is composed of the rest.

These were preprocessed with tokenization, lowercasing and lemmatization. In
addition, three open-source data-sets were used: 20 News Group
(20News)(Rennie, ), Movie Reviews (Reviews)(Pang and Lee, 2005) and Reuters-8
(Reuters-8)(Meng et al., 2020). Each dataset is divided into training, update
and test splits. As a reference, the benchmark models are trained on the
combined training and update splits. Then, to simulate domain adaptation, the
benchmark and newly proposed approaches were trained on the training split,
before separately being updated on the update split.

A Word2Vec(Mikolov et al., 2013) model is used as a benchmark, where word
vectors are averaged in place of the document vector.

Training Approach | 20News | Reuters-8 | Reviews | BioChem | Core-general
---|---|---|---|---|---
Word2vec | 0.69 | 0.90 | 0.75 | 0.62 | 0.76
JoSE | 0.68 | 0.88 | 0.77 | 0.60 | 0.78
Retraining Approach | 20News | Reuters-8 | Reviews | BioChem | Core-general
Word2vec | 0.70 | 0.89 | 0.77 | 0.62 | 0.71
JoSE | 0.62 | 0.78 | 0.72 | 0.54 | 0.59
JoSE (Compression) | 0.66 | 0.86 | 0.71 | 0.59 | 0.73
JoSE (Construction) | 0.65 | 0.89 | 0.74 | 0.60 | 0.76
Table 1: Macro F1 scores for initially training on the training set and
applying subsequent training on the test set for the different update
approaches (top) and different construction approaches (bottom).

Table 1 shows both our proposed approaches improve on naive retraining,
reducing the perofrmance cost of domain-adaptation from 12% to 2% for the
construction approach. This is similar to the performance drop of Word2Vec,
which is around 0.7%.

Of the two proposed approaches, the construction approach performed better,
outperforming the other JoSE models on four of the five datasets. This is
particularly attractive given it is computationally faster than computing the
word-document linkages.

## Conclusion

In this work, we develop the JoSE model presented in Meng et al. (2019) to
accept update training steps for applications such as domain adaptation. This
required the elimination of spinning of the embedding space during training,
which was achieved by correcting a likely error in the original model and
applying a post-processing counter-rotation. Further, two approaches were
proposed to account for the inability to update document vectors during update
training steps: adding compressed document representations to the loss
function and constructing document-vectors as averaged word-vectors. Two novel
data-sets of scientific publications for document classification, specifically
focusing on chemistry were collated for testing against benchmark models.
These approaches improved the JoSE model’s document classification performance
after update training. The performance against a Word2Vec benchmark in this
context is now consistent with the performance presented in Meng et al.
(2019). The results show that our approaches are able to reduce the cost of
domain adaptation to a level similar to Word2Vec and hence this modified JoSE
model is now competitive with Word2Vec in domain adaptation tasks.

## References

  * Meng et al. (2019) Yu Meng, Jiaxin Huang, Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, and Jiawei Han.  Spherical Text Embedding.  _arXiv:1911.01196 [cs, stat]_ , November 2019.  URL http://arxiv.org/abs/1911.01196.  arXiv: 1911.01196.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient Estimation of Word Representations in Vector Space.  _arXiv:1301.3781 [cs]_ , September 2013.  URL http://arxiv.org/abs/1301.3781.  arXiv: 1301.3781.
  * (3) Jason Rennie.  20 news groups.  http://qwone.com/~jason/20Newsgroups/.  Accessed: 2021-03-08.
  * Pang and Lee (2005) Bo Pang and Lillian Lee.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of ACL_ , pages 115–124, 2005.
  * Meng et al. (2020) Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, and Jiawei Han.  Unsupervised Word Embedding Learning by Incorporating Local and Global Contexts.  _Frontiers in Big Data_ , 3, 2020.  ISSN 2624-909X.  doi: 10.3389/fdata.2020.00009.  URL https://www.frontiersin.org/articles/10.3389/fdata.2020.00009/full.

  *[kNN]: k-Nearest Neighbours
  *[20News]: 20 News Group
  *[Reviews]: Movie Reviews
  *[Reuters-8]: Reuters-8