# Label-Free Multi-Domain Machine Translation
with Stage-wise Training

Fan Zhang$1$,$2$, Mei Tu$2$, Sangha Kim$2$, Song Liu$2$, Jinyao Yan$1$
$1$ State Key Laboratory of Media Convergence and Communication, Communication
University of China
$2$ Samsung Research China - Beijing (SRC-B)
{zhang.fan, mei.tu, sangha01.kim<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Most multi-domain machine translation models rely on domain-annotated data.
Unfortunately, domain labels are usually unavailable in both training
processes and real translation scenarios. In this work, we propose a label-
free multi-domain machine translation model which requires only a few or no
domain-annotated data in training and no domain labels in inference. Our model
is composed of three parts: a backbone model, a domain discriminator taking
responsibility to discriminate data from different domains, and a set of
experts that transfer the decoded features from generic to specific. We design
a stage-wise training strategy and train the three parts sequentially. To
leverage the extra domain knowledge and improve the training stability, in the
discriminator training stage, domain differences are modeled explicitly with
clustering and distilled into the discriminator through a multi-classification
task. Meanwhile, the Gumbel-Max sampling is adopted as the routing scheme in
the expert training stage to achieve the balance of each expert in
specialization and generalization. Experimental results on the German-to-
English translation task show that our model significantly improves BLEU
scores on six different domains and even outperforms most of the models
trained with domain-annotated data.

##  1 Introduction

Figure 1: Model architecture. Our model consists of an encoder-decoder-based
backbone model, a discriminator, and a set of experts transplanted in the
backbone model. This model is trained sequentially with a stage-wise training
strategy. We model domain differences by clustering using as few domain-
annotated data as possible and simulate the vague domain boundary with the
routing scheme of the Gumbel-Max sampling.

In recent years, neural machine translation (NMT), as one of the core tasks of
neural language processing (NLP), has already been studied extensively with
great progress (Vaswani et al., 2017; Dabre et al., 2020). Texts of different
domains usually have their own expression styles. Domain diversity leads to
heterogeneous data distribution of a domain-mixed dataset. During the training
of a generic NMT model, data from different domains tend to adjust model
parameters to fitting their own distributions. Such a global optimum
sacrifices domain specialization, especially in those small domains. Multi-
domain machine translation (MDMT) still remains challenging (Pham et al.,
2021).

To help the model distinguish different domains, some works (Kobus et al.,
2017; Britz et al., 2017; Zeng et al., 2018) tried to introduce domain
knowledge during training. Besides, some other works (Bapna and Firat, 2019;
Pham et al., 2020) regarded MDMT as an extension of transfer learning in the
machine translation area, and they tried to handle the domain shift problem by
fine-tuning a generic model with in-domain datasets. However, these two kinds
of works require domain labels in training, and some even need to specify
domains in inference. Domain-label annotation is one of the biggest obstacles
against these efforts. On the one hand, domain labels are usually unavailable
for large-scale bilingual parallel data. On the other hand, the diversity of
domain criteria makes the notion (boundary) of a domain vague, thus making the
domain-label annotation difficult. For instance, a news sentence describing
medicine can belong to both the news domain and the medical domain.

Recently, previous works (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus
et al., 2021; Dai et al., 2022) explored the mixture-of-experts (MoE)
structure in NMT. Since MoE implicitly learns to route data to suitable
experts by the gating network, domain-label annotation is unnecessary when
using MoE to handle MDMT. These works have shown the powerful ability of MoE
on dealing with diverse data distributions, but a training strategy should
always be carefully designed to avoid training instability. For instance,
expert load imbalance may happen during training an MoE model: most data would
have been routed to a small number of experts, meanwhile many other experts do
not get sufficiently trained at all (Lepikhin et al., 2020). Moreover, the
routing fluctuation (Dai et al., 2022) issue, i.e. the same data may be routed
to different experts along with training, is also one of the factors leading
to training instability.

In this work, we propose a label-free MDMT model which requires only a few or
no domain-annotated data in training and no domain labels in inference.
Inspired by the concept of MoE, our model is composed of three parts: (i) a
generic machine translation model as the backbone model; (ii) a domain
discriminator (with respect to the gating network in the MoE structure); and
(iii) a set of experts transplanted in the backbone model. To guarantee
training stability, we design a stage-wise training strategy and train the
three parts individually in their respective stages. When training the domain
discriminator, domain differences are modeled explicitly by clustering using
as few domain-annotated sentences as possible and distilled into the
discriminator through a multi-classification task. In other words, we (i) can
leverage a little accessible extra domain knowledge easily and (ii) train the
discriminator explicitly which enhances the training stability. Meanwhile,
considering that the domain boundary is vague, in the expert training stage,
we adopt the Gumbel-Max sampling as the routing scheme and train the experts
through the standard machine translation task without any domain-annotated
data. With the Gumbel-Max routing scheme, we achieve balance in the domain
knowledge of each expert in specialization and generalization, which further
improves the overall multi-domain translation quality.

Experimental results on the German-to-English translation task show that our
model significantly improves BLEU scores on six different domains and even
outperforms most of the models trained with domain-annotated data. Further
explorations of the domain discrimination ability show the success of the
discriminator training manner. Additionally, the ablation studies in the
expert training stage demonstrate the effectiveness of the Gumbel-Max routing
scheme.

##  2 Related Works

MDMT has been an object of research in the literature on neural language
processing. Some works (Kobus et al., 2017; Britz et al., 2017; Tars and
Fishel, 2018) show that injecting domain information either on the encoder
side or on the decoder side can help the model improve domain translation
quality. Compared with the above works that introduce sentence-level domain
information, some other works (Zeng et al., 2018; Su et al., 2019; Jiang et
al., 2020) inject token-level domain information to cope with multi-domain
sentences.

Another strategy to handle MDMT is to regard it as an extension of transfer
learning. According to the parameter-efficient requirement, the lightweight
adapter modules are well studied on various transfer learning tasks (Houlsby
et al., 2019; Zhu et al., 2021). Some works (Bapna and Firat, 2019; Pham et
al., 2020) transplant adapters into a generic model, and later fine-tune them
with in-domain datasets.

However, all the above works need domain-annotated data, which means they are
suffering from the data collection problem. Actually, this topic has been
widely studied for machine translation (Axelrod et al., 2011; van der Wees et
al., 2017; Aharoni and Goldberg, 2020), but it is not well integrated with the
above works.

The MoE structure (Jacobs et al., 1991) has already been proposed for a long
time. In recent years, it has been widely studied in the machine translation
area (Shazeer et al., 2017; Shen et al., 2019; Lepikhin et al., 2020; Dai et
al., 2022). These works show that the training instability of the gating
network requires a sophisticated training strategy. Besides, high training
cost has also become one of the reasons that prevent MoE from being widely
studied.

##  3 Model Architecture

The core concept of MoE is using multiple experts to divide a problem space
into homogeneous regions (Baldacchino et al., 2016), which perfectly fits the
MDMT task. Following this clue, we design our model to be composed of three
parts: (i) a generic machine translation model as the backbone model; (ii) a
discriminator (with respect to the gating network in the MoE structure, which
responsibility is to route data to suitable experts); and (iii) a set of
experts transplanted in the decoder of the backbone model. Figure 1
illustrates the overall model architecture.

The backbone model is based on the encoder-decoder structure, where the
encoder/decoder block is composed of a stack of several identical layers.
Given a source sentence $x=(x_{1},...,x_{n})$, the encoder block maps it to a
sequence of hidden states $h=(h_{1},...,h_{n})$. Then, $h$ is fed to the
decoder block to generate an output sequence $y=(y_{1},...,y_{m})$ with an
auto-regressive process.

The discriminator makes use of the hidden states $h$ to discriminate different
domain data by scoring predefined domain categories. First,
$h\in\mathbb{R}^{n\times d}$ is condensed to $\hat{h}\in\mathbb{R}^{d}$ by
mean pooling on the sequence length dimension $n$,

$\hat{h}=Pooling(h)$ (1)

Then two linear transformations are introduced with a $tanh$ activation in
between to compute category scores $s$,

$s=tanh(\hat{h}W_{1}+b_{1})W_{2}+b_{2}$ (2)

where $W_{1}\in\mathbb{R}^{d\times d}$, $W_{2}\in\mathbb{R}^{d\times K}$,
$b_{1}\in\mathbb{R}^{d}$ and $b_{2}\in\mathbb{R}^{K}$ are the parameter
matrices of the linear transformations and $K$ is the predefined category
number.

The experts are transplanted in every decoder layer of the backbone model to
transfer the decoded hidden states from generic to specific. In each expert,
the output $z_{i}$ of the $i$-th decoder layer is first normalized with layer
normalization,

$\tilde{z_{i}}=LN(z_{i})$ (3)

Then $\tilde{z_{i}}$ is fed to a fully connected feed-forward network
$FFN(\cdot)$ (Vaswani et al., 2017), followed by a residual connection, to
obtain the expert output,

$o_{i}=FFN(\tilde{z_{i}})+z_{i}$ (4)

In inference, the expert with the biggest category score in each decoder layer
is activated. Unlike inference, the Gumbel-Max sampling is adopted as the
routing scheme in the expert training stage, which will be discussed in the
next section.

##  4 Stage-wise Training

We design a stage-wise training strategy and train the three parts of our
model sequentially. It is worth noting that the parameters of all previous
parts will be frozen when training the latter part. The decoupled training
process has brought us many benefits. First, it is feasible to reuse a well-
pretrained NMT model as the backbone model, which is a huge advantage in
commercial usage since training an NMT model from scratch is relatively time-
consuming. Second, we can easily inject extra domain knowledge to help improve
the performance of the discriminator, thus improving the translation quality.
Third, such a training process avoids the training instability problem caused
by the MoE structure. Next, we will discuss our training process in detail.

###  4.1 Backbone model

The backbone model is trained through a standard machine translation task.
Given a dataset of parallel text $D^{mt}=\\{(x,y^{*})\\}^{N_{t}}_{i=1}$, the
training objective is varying the trainable parameters $\theta$ to minimize
the cross-entropy loss:

$\mathcal{L}_{mt}(\theta)=-\sum_{i=1}^{N_{t}}\sum_{t=1}^{m}logP(y^{*}_{t}|y^{*}_{1:t-1},x;\theta)$
(5)

At this training stage, $\theta$ refers to the parameters of the backbone
model.

###  4.2 Discriminator

Figure 2: Discriminator training set construction.

Sentence features from the same domain are usually closer than those from
different domains, so the domain differences can be modeled by clustering
without any domain information (Aharoni and Goldberg, 2020). Meanwhile, a
small amount of in-domain sentences are readily accessible. These sentences
can be used for small domain augmentation when clustering.

In this training stage, a small sentence set is first prepared by sampling
from $D^{mt}$ at random. Optionally, a few in-domain sentences are injected
into the sentence set as domain anchors if exist. By the encoder of the
backbone model, we convert these sentences into the condensed hidden states
$\hat{h}$ (Eq. 1) as the sentence features and then cluster them into $K$
groups. Each group is regarded as an independent domain category. Figure 2
illustrates the above steps. In the end, we distill this clustering result
into the discriminator through a multi-classification task.

Let $D^{d}=\\{(x,c)\\}^{N_{d}}_{i=1}$ be the training set we construct above
where $c$ is the one-hot vector of the domain category label, the goal in this
training stage is minimizing the multi-classification loss:

$\mathcal{L}_{d}(\theta)=-\sum_{i=1}^{N_{d}}\sum_{j=1}^{K}c_{j}log(p_{j})$ (6)

where

$p_{j}=\frac{e^{s_{j}}}{\sum_{k=1}^{K}e^{s_{k}}}$ (7)

and $\theta$ refers to the parameters in Eq. 2.

###  4.3 Experts

Figure 3: Domain boundary simulation. $k$ controls the boundary width while
$\tau$ controls the domain probability distribution.

Since the domain boundary is vague, routing data to the expert with the
highest score is reckless. In the expert training stage, we propose routing
sentences with the Gumbel-Max sampling scheme (Gumbel, 1954; Maddison et al.,
2014). While ensuring the specialization of domain knowledge, this routing
scheme further improves the knowledge generalization of each expert.

Formally, given the category scores $s$, we focus on the categories with the
top $k$ biggest scores and compute their relative probabilities,

$p=softmax(topk(s)/\tau)$ (8)

where temperature $\tau>0$ is a hyper-parameter controlling the probability
distribution. The higher the $\tau$, the closer the probability distribution
to the discrete uniform distribution. On the contrary, it is closer to the
one-hot distribution. The activated expert is chosen as:

$e=arg\ max(G(p))$ (9)

where

$G(p)=log(p)+g$ (10)

and $g$ is a set of i.i.d samples that are drawn from Gumbel(0,1) distribution
(Gumbel, 1954). Please refer to Appendix A.3 for detailed implementation.

The training objective in this stage is the same as the backbone model (Eq.
5), except that $\theta$ refers to the parameters of the experts.

##  5 Experimental Settings

To evaluate our model, we conduct a set of experiments on MDMT under the
German-to-English translation task. Different from most previous works (Kobus
et al., 2017; Pham et al., 2021) where the data scale of each domain is
comparable, we mix several small domain datasets with a large generic dataset.
The translation quality is measured by the BLEU-4 (Papineni et al., 2002)
score. In the following parts, we will describe our experimental settings in
detail.

###  5.1 Datasets

We collect two datasets of German-English sentence pairs. One is the standard
WMT 2014 German-English dataset consisting of about $4.6$ million sentence
pairs, which can be seen as a large generic domain (WMT). Another one is the
multi-domain dataset from Aharoni and Goldberg (2020) which is originally
provided by Koehn and Knowles (2017), including textual data in five diverse
domains: IT-related text (IT, manuals and localization files of open-source
software), translations of the Koran (KOR), legal text (LAW, legislative text
of the European Union), medical text (MED, PDF documents from the European
Medicines Agency), and subtitles (SUB). To explore model performance under the
label-free scenario, we build a test set with random domain data, namely RND,
by sampling $500$ sentences from each domain test set at random with random
domain labels. In the data processing phase, we first use the Moses tokenizer
(Koehn et al., 2007) for subword tokenization of both English and German
sentences. Then we learn and apply Byte-Pair Encoding (Sennrich et al., 2016)
on all sentences jointly, while the number of the merge operation is set to
$30,000$.

| Train | Dev | Test
---|---|---|---
WMT | $4,603,578$ | $2,001$ | $3,003$
KOR | $17,982$ | $2,000$ | $2,000$
IT | $222,927$ | $2,000$ | $2,000$
MED | $248,099$ | $2,000$ | $2,000$
LAW | $467,309$ | $2,000$ | $2,000$
SUB | $500,000$ | $2,000$ | $2,000$
RND | - | - | $3,000$
Table 1: Dataset statistics. RND test set is built by choosing $500$ sentences
from each domain test set at random with random domain labels.

###  5.2 Implementations

| $Param.$ | WMT | KOR | IT | MED | LAW | SUB | $AVG$ | RND
---|---|---|---|---|---|---|---|---|---
Mixed-nat | $100M$ | 32.08 | 19.65 | 44.79 | 51.47 | 54.34 | 30.60 | 38.82 | 40.66
FT⋆⋄ | $[+500]M$ | 32.14 | 22.31 | 46.76 | 54.34 | 57.06 | 32.02 | 40.77 | 35.91
ADPT⋆⋄ | $[+9.5]M$ | 32.25 | 22.44 | 45.82 | 52.78 | 55.53 | 31.96 | 40.13 | 36.86
DC-tag⋆⋄ | $[+0]M$ | 31.98 | 20.76 | 45.08 | 52.10 | 54.86 | 31.35 | 39.36 | 38.22
DM⋆ | $[+0]M$ | 31.65 | 18.81 | 43.69 | 50.85 | 53.22 | 30.68 | 38.15 | 40.25
ADM⋆ | $[+0]M$ | 31.59 | 18.44 | 43.67 | 51.04 | 53.66 | 30.16 | 38.09 | 40.17
TTM⋆ | $[+0]M$ | 32.26 | 19.11 | 44.34 | 51.63 | 54.23 | 31.54 | 38.85 | 40.57
SG-MoE | $[+9.5]M$ | 31.83 | 21.04 | 45.45 | 53.48 | 56.79 | 31.30 | 39.98 | 41.74
Switch | $[+9.5]M$ | 32.13 | 20.55 | 45.53 | 52.57 | 55.72 | 30.97 | 39.58 | 41.46
Ours-RS | $[+9.7]M$ | 32.78 | 21.97 | 46.20 | 53.15 | 55.91 | 31.63 | 40.27 | 42.16
Ours-DI⋆ | $[+9.7]M$ | 32.45 | 22.27 | 46.29 | 53.54 | 56.10 | 31.72 | 40.40 | 42.22
Table 2:  Multi-domain translation performance. The flags, $\star$ and
$\diamond$, indicate the model needs domain information in training and
inference, respectively. The average BLEU scores of the six domains (WMT, KOR,
IT, MED, LAW, SUB) are listed in $AVG$ column. RND reflects the model
performance without domain labels (or with random domain labels in the label-
required models). The top three BLEU scores in each column are shown in bold.
Please note that compared with other models with flag $\star$, Ours-DI
requires very few domain-annotate data.

We use the standard Transformer (Vaswani et al., 2017) implemented in Fairseq
(Ott et al., 2019) with $6$ layers, $512$ embedding dimensions, and $8$
attention heads as the base model structure. All comparison models are
implemented with the base model structure. Next, we will introduce these
models briefly.

Baseline models. We first train a standard NMT model using the naturally mixed
dataset, referred to as Mixed-nat, which is also our backbone model.
Furthermore, we then fine-tune Mixed-nat for every domain by their in-domain
datasets. The six fine-tuned models are considered as an ensemble, namely FT.

Adapters (ADPT) (Bapna and Firat, 2019). The experts (adapters) are injected
in layers of both encoder and decoder of the pre-trained Mixed-nat. Then, for
each domain, its experts are fine-tuned with the in-domain dataset
independently.

Domain Control with Tag (DC-Tag) (Kobus et al., 2017). DC-Tag introduces
domain information as an additional token for each source sentence in both
training and inference.

Discriminative Mixing (DM), Adversarial Discriminative Mixing (ADM) and Target
Token Mixing (TTM). The three methods come from the same work (Britz et al.,
2017). DM and ADM minimize the joint training objective of domain
classification task and translation task by using different learning
strategies, while TTM adds domain tags in the front of target sentences to
make the model predict domain tag as a regular token.

Sparsely-gated mixture-of-experts (SG-MoE) (Shazeer et al., 2017) and Switch
transformer (Switch) (Fedus et al., 2021). SG-MoE introduces sparsely gated
MoE layers with the noisy top-k gating scheme, which activates $k>1$ experts
per time to obtain nonzero derivatives in back-propagation. It also introduces
auxiliary losses to deal with the expert load imbalance. In practice, we set
$k=2$ for SG-MoE. Switch is another MoE method that activates only one expert
per time. To avoid the expert load imbalance when training the gating network,
it introduces a capacity factor and an auxiliary load balancing loss. In our
implementations111The losses in our implementations are referred to that in
https://github.com/laekov/fastmoe ., different from the original works that
use token-level expert routing schemes, we adopt sentence-level routing scheme
and only introduce experts in the decoder layers to consist with ours.

Ours. In the discriminator training stage, we sample $50,000$ sentences from
the mixed NMT training set at random and use sentences of the Dev sets as the
domain anchors. In contrast, we also train the discriminator with only the
random sampled (RS) sentences to show the influence of the injected domain
information (DI). The models that the discriminator trained without and with
the domain anchors are referred to as Ours-RS and Ours-DI, respectively. We
choose Gaussian Mixture Model (GMM) as our clustering method. The expert
number $K$ is set to $12$ for all MoE models (SG-MoE, Switch and Ours), while
the inner dimension of the experts is set to $128$ for all MoE models and
ADPT. In the expert training stage, we set the hyper-parameters of the Gumbel-
Max routing scheme $k=4$ and $\tau=1.0$. The Gumbel-Max routing scheme is shut
down in inference with $k=1$.

##  6 Results and Discussion

###  6.1 Multi-domain machine translation

Most of the MDMT models rely on domain-annotated data in training, and some
even require domain labels in inference. We test the translation performance
of the MDMT models on the six domains (WMT, KOR, IT, MED, LAW, SUB) and report
their BLEU scores in Table 2. BLEU scores on the RND test set reflect the
translation performance under the label-free scenario. To show the differences
in model size, we also present the number of parameters ($Param.$) in Table 2.

As shown in Table 2, our models (Ours-RS and Ours-DI) outperform all other
MDMT models on overall multi-domain translation measurements ($AVG$ and RND).
Although FT shows a better average BLEU score, it requires an extremely large
model size in the meantime. In inference, label-required models (i.e. models
with flag $\diamond$) perform well with domain labels (see $AVG$). However,
when they are confronted with the scenario that domain labels are unavailable
(or probably not correct), their performance is significantly poorer than
others (see RND). In contrast, the outstanding performance of the MoE models
(i.e. SG-MoE, Switch, Ours-RS and Ours-DI) on the RND test set illustrates the
powerful ability of MoE on dealing with the MDMT problem. Given domain labels,
ADPT can be regarded as an MoE model with oracle discrimination ability. But
its expert number is limited by the known domain number, and the small domains
will not take benefits from the big generic dataset (i.e. the WMT training set
in our experiments). Results show that although it has oracle discrimination
ability, its model performance on some small domains is poorer than ours.

Comparing Ours-DI with Ours-RS, we find that Ours-DI shows steady improvement
on those small domains. The improvement is attributed to the improved domain
discrimination ability of the discriminator, which will be discussed in the
following part. It also indicates that the ability to flexibly inject domain
information is necessary for an MDMT model.

To evaluate the dataset robustness of our model, we further train our model
with only the WMT training set and on the English-to-German translation
direction, respectively. The experimental results are reported in Appendix
A.4.

###  6.2 Domain discrimination ability

Figure 4: Routing statistics of Ours-RS on the test sets. Figure 5: Routing
statistics of Ours-DI on the test sets.

To analyze the impact of the clustering results on domain discrimination
ability and ultimately on the translation performance, we conduct a set of
experiments in the discriminator training stage.

First, we count the routing results of Ours-RS and Ours-DI on the test sets
(Figure 4 and Figure 5). Based on the statistics, we roughly measure the
domain discrimination ability by two metrics. One is the category purity score
$PUR$,

$PUR=\frac{1}{U}\sum_{i=1}^{K}u_{i}^{max}$ (11)

where $U$ is the total number of the test cases, $K$ is the number of the
categories (NOT the number of the test domains), and $u_{i}^{max}$ is the
maximum number of $i$-th category. The other one is the normalized mutual
information $NMI$ (Danon et al., 2005) score between true domain labels and
the predicted category labels, as implemented in scikit-learn (Pedregosa et
al., 2011). The two metrics measure the mixing degree of different domains in
a category. The higher the $PUR$ and the $NMI$, the better the domain
discrimination ability.

Results in Table 3 show that the domain discrimination ability of our
discriminator is improved after injecting a few domain-annotated data. When we
investigate the routing statistical results of Ours-RS in Figure 4, we find
that although most sentences in the same domain are routed to the same
category, the category purity is relatively low. Take KOR as an example, most
of its sentences are routed to the $3$-rd category, but this category also
contains many sentences from WMT and SUB. In contrast, routing statistics of
Ours-DI show higher purity of clusters related to small domains.

| $PUR$ | $NMI$
---|---|---
Ours-RS | 0.8038 | 0.5976
Ours-DI | 0.8403 | 0.6455
Table 3: Measurements of the domain discrimination ability. | GMM | K-Means | Birch
---|---|---|---
WMT | 32.45 | 32.93 | 32.77
KOR | 22.27 | 22.09 | 22.35
IT | 46.29 | 46.23 | 46.08
MED | 53.54 | 53.78 | 53.58
LAW | 56.10 | 56.01 | 56.23
SUB | 31.72 | 31.45 | 31.17
$AVG$ | 40.40 | 40.42 | 40.36
RND | 42.22 | 42.12 | 42.00
Table 4: BLEU scores of different clustering methods.

We also try another two clustering methods (K-Means and Birch) to model the
domain differences in the discriminator training stage. The final BLEU scores
based on the three clustering methods are listed in Table 4. Results show that
the three clustering methods have their own advantages and disadvantages for
different domains. It is difficult to define which one is better.

Another interesting observation is that after distilling the clustering result
into the discriminator, the domain discrimination ability of the discriminator
is even better than that of the clustering model. Measurements of the domain
discrimination abilities are reported in Table 5. We find that there are
steady improvements on both $PUR$ and $NMI$ for all three clustering methods.
Previous works (Tian et al., 2017; Caron et al., 2018; Aharoni and Goldberg,
2020) also have reported similar observations on different tasks.

|  | $PUR$ | $NMI$
---|---|---|---
GMM | cluster | 0.8365 | 0.6377
distill | 0.8403 | 0.6402
K-Means | cluster | 0.8483 | 0.6216
distill | 0.8508 | 0.6242
Birch | cluster | 0.8068 | 0.5967
distill | 0.8133 | 0.6087
Table 5: Clustering models vs discriminators. ‘cluster’ means routing by
clustering models, while ‘distill’ means routing by discriminators (trained by
distilling knowledge from clustering models) Figure 6: Expert number analysis.
We set the expert number $K$ to $3$, $6$, $12$, $24$, and $48$ on each MoE
method to explore its impact.

###  6.3 Number of experts

Based on Ours-DI, we vary the expert number $K$ (i.e. the domain category
number) and explore its impact. Meanwhile, we also explore this impact on the
other two MoE methods (SG-MoE and Switch). Figure 6 shows our method can
improve the performance of the backbone model (Mixed-nat) even when $K$ is
smaller than the known domain number. With the increase of $K$, too many
experts will not bring significant improvement under our experimental
settings. Although SG-MoE and Switch introduce auxiliary losses to alleviate
expert load imbalance, the training instability problem still exists. When the
number of experts increases, their $AVG$ BLEU scores decrease instead.

###  6.4 Routing scheme

In the expert training stage, we adopt the Gumbel-Max routing scheme to
simulate the vague domain boundary. The two hyper-parameters, the category
candidate number $k$ and the temperature $\tau$, control the mixing degree
between different categories (w.r.t. Eq. 8). We experiment with adjusting them
in the expert training stage. Experimental results are reported in Table 6 and
Table 7.

| $AVG$ | RND
---|---|---
Mixed-nat | 38.82 | 40.66
$\tau\rightarrow 0.0$ | 40.10 | 41.96
$\tau=0.1$ | 40.31 | 42.12
$\tau=1.0$ | 40.40 | 42.22
$\tau=10.0$ | 40.11 | 42.17
Table 6: Hyper-parameter $\tau$ analysis. $\tau\rightarrow 0.0$ is equivalent to shutting down the Gumbel-Max routing scheme. | $AVG$ | RND
---|---|---
Mixed-nat | 38.82 | 40.66
$k=1$ | 40.10 | 41.96
$k=2$ | 40.39 | 42.11
$k=4$ | 40.40 | 42.22
$k=8$ | 40.31 | 42.06
$k=12$ | 40.17 | 41.98
Table 7: Hyper-parameter $k$ analysis. $k=1$ is equivalent to shutting down
the Gumbel-Max routing scheme.

In Table 6, we fix $k$ to $4$ and vary $\tau$ to analyze the difference.
Meanwhile, the experimental settings in Table 7 are that $\tau$ is fixed to
$1.0$ and $k$ is varied. Both $\tau\rightarrow 0.0$ and $k=1$ are equivalent
to shutting down the Gumbel-Max routing scheme. Results show that the moderate
values $\tau=1.0$ and $k=4$ perform better than the other settings. It
demonstrates that there is a balance in the domain knowledge of each expert in
specialization and generalization.

##  7 Conclusion

This paper proposes a label-free MDMT model, which requires only a few or no
domain-annotated data in training and no domain labels in inference. Inspired
by the concept of MoE, the model is composed of three parts and these parts
are trained sequentially via a stage-wise training strategy. Domain
differences are modeled explicitly with clustering and distilled into the
discriminator through a multi-classification task. In addition, the Gumbel-Max
sampling is adopted as the routing scheme in the expert training stage to
achieve a balance of each expert in specialization and generalization.
Experimental results show the model significantly improves BLEU scores on six
different domains and even outperforms most of the comparison models trained
with domain-annotated data.

## Limitations

In this work, in order to tackle the problem of domain-label annotation, we
use a clustering method to model domain differences with as few domain-
annotated sentences as possible. Empirical results show that although our
performance is one of the best among the comparison models, there is still a
small gap between ours and the fully fine-tuned model (FT). To further improve
our performance, it might be feasible to train a more powerful discriminator
with more domain-annotated data or to use experts that have larger capacities,
but that goes against our original intention. Exploring a better approach than
clustering to model domain differences by few to no domain-annotated sentences
should be an interesting direction. Additionally, the Gumbel-Max sampling
scheme helps us improve the model performance, but its two hyper-parameters
are fixed empirically in the current version. In future work, adjusting these
two hyper-parameters automatically according to the number of experts
(categories) and the characteristics of the training set may be better.

## References

  * Aharoni and Goldberg (2020) Roee Aharoni and Yoav Goldberg. 2020.  Unsupervised domain clusters in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7747–7763.
  * Axelrod et al. (2011) Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011.  Domain adaptation via pseudo in-domain data selection.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , pages 355–362.
  * Baldacchino et al. (2016) Tara Baldacchino, Elizabeth J Cross, Keith Worden, and Jennifer Rowson. 2016.  Variational bayesian mixture of experts models and sensitivity analysis for nonlinear dynamical systems.  _Mechanical Systems and Signal Processing_ , 66:178–200.
  * Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019.  Simple, scalable adaptation for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1538–1548.
  * Britz et al. (2017) Denny Britz, Quoc Le, and Reid Pryzant. 2017.  Effective domain mixing for neural machine translation.  In _Proceedings of the Second Conference on Machine Translation_ , pages 118–126.
  * Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 132–149.
  * Dabre et al. (2020) Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020.  A survey of multilingual neural machine translation.  _ACM Computing Surveys (CSUR)_ , 53(5):1–38.
  * Dai et al. (2022) Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Stablemoe: Stable routing strategy for mixture of experts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7085–7095.
  * Danon et al. (2005) Leon Danon, Albert Diaz-Guilera, Jordi Duch, and Alex Arenas. 2005.  Comparing community structure identification.  _Journal of statistical mechanics: Theory and experiment_ , 2005(09):P09008.
  * Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer. 2021.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv preprint arXiv:2101.03961_.
  * Gumbel (1954) Emil Julius Gumbel. 1954.  _Statistical theory of extreme values and some practical applications: a series of lectures_ , volume 33.  US Government Printing Office.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991\.  Adaptive mixtures of local experts.  _Neural computation_ , 3(1):79–87.
  * Jiang et al. (2020) Haoming Jiang, Chen Liang, Chong Wang, and Tuo Zhao. 2020.  Multi-domain neural machine translation with word-level adaptive layer-wise domain mixing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1823–1834.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kobus et al. (2017) Catherine Kobus, Josep M Crego, and Jean Senellart. 2017.  Domain control for neural machine translation.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 372–378.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions_ , pages 177–180. Association for Computational Linguistics.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_.
  * Maddison et al. (2014) Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014.  A* sampling.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2_ , pages 3086–3094.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830.
  * Pham et al. (2021) Minh Quang Pham, Josep M Crego, and François Yvon. 2021.  Revisiting multi-domain machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:17–35.
  * Pham et al. (2020) Minh Quang Pham, Josep-Maria Crego, François Yvon, and Jean Senellart. 2020\.  A study of residual adapters for multi-domain neural machine translation.  In _Conference on Machine Translation_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_.
  * Shen et al. (2019) Tianxiao Shen, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2019.  Mixture models for diverse machine translation: Tricks of the trade.  In _International conference on machine learning_ , pages 5719–5728. PMLR.
  * Su et al. (2019) Jinsong Su, Jiali Zeng, Jun Xie, Huating Wen, Yongjing Yin, and Yang Liu. 2019.  Exploring discriminative word-level domain contexts for multi-domain neural machine translation.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(5):1530–1545.
  * Tars and Fishel (2018) Sander Tars and Mark Fishel. 2018.  Multi-domain neural machine translation.  _arXiv preprint arXiv:1805.02282_.
  * Tian et al. (2017) Kai Tian, Shuigeng Zhou, and Jihong Guan. 2017.  Deepcluster: A general clustering framework based on deep learning.  In _Joint European conference on machine learning and knowledge discovery in databases_ , pages 809–825. Springer.
  * van der Wees et al. (2017) Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017.  Dynamic data selection for neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1400–1410.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Zeng et al. (2018) Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, and Jianqiang Zhao. 2018.  Multi-domain neural machine translation with word-level domain context discrimination.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 447–457.
  * Zhu et al. (2021) Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. 2021.  Counter-interference adapter for multilingual machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2812–2823.

##  Appendix A Appendices

params : Category scores $S$; Category candidate number $k$; Temperature
$\tau$; Activated expert indices $E$.

import $torch$;

import $torch.nn.functional$ as $F$;

if  _$k <=1$_ then

$E=torch.argmax(S)$;

return $E$;

end if

$topk\\_val,topk\\_idx=torch.topk(S,k=k,dim=1)$;

$topk\\_val\ /=\tau$;

$log\\_probs=torch.log(F.softmax(topk\\_val,dim=1))$;

$g=F.gumbel\\_softmax(log\\_probs,dim=1)$;

$sampled=torch.argmax(g,dim=1,keepdim=True)$;

$E=torch.gather(topk\\_idx,1,sampled).squeeze()$;

return $E$;

Algorithm 1 Gumbel-Max sampling of PyTorch version

###  A.1 Training details

In any training phase, we use the Adam optimizer (Kingma and Ba, 2014) with
$\beta_{1}=0.9$, $\beta_{2}=0.98$ and $\epsilon=10^{-9}$. For translation
optimization, we use the Noam decay as the learning rate scheduler with $4000$
warmup steps and a learning rate of $0.0007$. With a batch size of $8192$ in
the token level and the update frequency of $10$ on $2$ V100 GPUs, the maximum
update number of training is set to $100k$, while that of fine-tuning is set
to $10k$ with the early stopping strategy. The maximum update number of the
discriminator training stage is set to $6k$. In inference, the beam size is
set to $5$ for all models.

###  A.2 Clustering details

We choose Gaussian Mixture Model (GMM), K-Means and Birch implemented in
scikit-learn (Pedregosa et al., 2011) as our clustering approaches. The
convariance type of GMM is set to ‘full’, while all other settings are set by
default. Before clustering, we perform dimensionality reduction with Principal
Components Analysis (PCA) to reduce the vector dimension of the sentence
representations from $512$ to $64$. When testing the clustering performance on
the test set, we reuse both the clustering model and the dimensionality
reduction model generated by the training set.

###  A.3 Gumbel-Max sampling

We implement the Gumbel-Max sampling strategy with PyTorch (Paszke et al.,
2019) of version 1.10.1+cu102. Implementation details are shown in Algorithm
1. It is worth noting that the category scores $S$ and the activated expert
indices $E$ are at batch-level compared with that in subsection 4.3.

###  A.4 Dataset robustness

We conduct a set of experiments over different datasets to show the dataset
robustness of our model. First, we use only the WMT training set as the
translation training set and use only the random sampled sentences in the
discriminator training stage (Ours-RS). In this scenario, there is no concept
of the domain at all. Results in Table 8 show that without any in-domain data,
Ours-RS also shows steady improvement compared with the backbone model.

| Backbone | Ours-RS
---|---|---
WMT | 32.29 | 32.68
KOR | 13.24 | 13.41
IT | 35.91 | 36.04
MED | 38.29 | 37.87
LAW | 41.60 | 42.02
SUB | 23.76 | 24.07
$AVG$ | 30.85 | 31.02
RND | 31.92 | 32.13
Table 8: BLEU scores with only WMT training set.

Then, we test our model on the English-to-German translation direction.
Despite the translation direction, the model implementations and the training
settings are all the same with the German-to-English translation task. Results
in Table 9 further demonstrate that the conclusions about our model
performance in the main part can also be concluded on the English-to-German
translation direction.

| Mixed-nat | Ours-RS | Ours-DI
---|---|---|---
WMT | 27.06 | 27.23 | 27.42
KOR | 19.17 | 21.05 | 22.25
IT | 38.63 | 39.85 | 39.94
MED | 48.15 | 48.92 | 49.25
LAW | 48.97 | 50.10 | 50.39
SUB | 25.38 | 25.67 | 25.93
$AVG$ | 34.56 | 35.47 | 35.86
RND | 36.52 | 37.54 | 37.93
Table 9: BLEU scores on the English-to-German translation direction.

###  A.5 Routing statistics

Figure 7: Routing statistics of Ours-DI with K-Means clustering method on the
test sets. Figure 8: Routing statistics of Ours-DI with Birch clustering
method on the test sets.

We report the routing statistics of Ours-DI on the test sets with K-Means and
Birch clustering methods in Figure 7 and Figure 8, respectively.

###  A.6 Detailed BLEU scores

We report the detailed BLEU scores of the expert number analysis and the
hyper-parameter search experiments in Table 10 and Table 11, respectively.

|  | WMT | KOR | IT | MED | LAW | SUB | $AVG$ | RND
---|---|---|---|---|---|---|---|---|---
Mixed-nat | 32.08 | 19.65 | 44.79 | 51.47 | 54.34 | 30.60 | 38.82 | 40.66
Ours-DI | $K=3$ | 32.58 | 20.58 | 45.47 | 52.44 | 55.48 | 31.41 | 39.66 | 41.56
$K=6$ | 32.48 | 20.86 | 45.93 | 52.78 | 55.97 | 31.38 | 39.90 | 41.73
$K=12$ | 32.45 | 22.27 | 46.29 | 53.54 | 56.10 | 31.72 | 40.40 | 42.22
$K=24$ | 32.33 | 22.36 | 46.39 | 53.40 | 56.22 | 31.47 | 40.36 | 42.37
$K=48$ | 32.01 | 21.73 | 46.63 | 54.39 | 56.68 | 30.86 | 40.38 | 42.23
SG-MoE | $K=3$ | 32.19 | 20.86 | 45.30 | 53.47 | 56.31 | 31.96 | 40.02 | 41.83
$K=6$ | 32.33 | 21.02 | 45.45 | 53.59 | 56.77 | 31.44 | 40.10 | 41.97
$K=12$ | 31.83 | 21.04 | 45.45 | 53.48 | 56.79 | 31.30 | 39.98 | 41.74
$K=24$ | 31.57 | 20.53 | 44.66 | 51.91 | 55.56 | 31.43 | 39.28 | 41.38
$K=48$ | 31.53 | 20.33 | 44.51 | 52.05 | 55.41 | 30.34 | 39.03 | 40.76
Switch | $K=3$ | 31.95 | 21.02 | 45.95 | 52.72 | 56.53 | 31.40 | 39.93 | 42.00
$K=6$ | 32.12 | 20.66 | 45.29 | 52.18 | 55.98 | 30.63 | 39.48 | 41.17
$K=12$ | 32.13 | 20.55 | 45.53 | 52.57 | 55.72 | 30.97 | 39.58 | 41.46
$K=24$ | 31.85 | 20.59 | 45.54 | 53.40 | 56.40 | 31.04 | 39.80 | 41.81
$K=48$ | 31.49 | 20.84 | 45.64 | 52.09 | 56.27 | 31.10 | 39.57 | 41.49
Table 10: Detailed BLEU scores of the expert number analysis. The highest BLEU score of each test set in each method is shown in bold. |  | WMT | KOR | IT | MED | LAW | SUB | $AVG$ | RND
---|---|---|---|---|---|---|---|---|---
Mixed-nat | 32.08 | 19.65 | 44.79 | 51.47 | 54.34 | 30.60 | 38.82 | 40.66
Ours-DI | $\tau\rightarrow 0.0$ | 32.35 | 22.05 | 46.18 | 52.78 | 55.84 | 31.42 | 40.10 | 41.96
$\tau=0.1$ | 32.63 | 22.11 | 46.30 | 53.01 | 56.25 | 31.54 | 40.31 | 42.12
$\tau=1.0$ | 32.45 | 22.27 | 46.29 | 53.54 | 56.10 | 31.72 | 40.40 | 42.22
$\tau=10.0$ | 32.58 | 21.97 | 46.41 | 52.56 | 55.67 | 31.46 | 40.11 | 42.17
Ours-DI | $k=1$ | 32.35 | 22.05 | 46.18 | 52.78 | 55.84 | 31.42 | 40.10 | 41.96
$k=2$ | 32.63 | 22.68 | 46.33 | 53.50 | 55.93 | 31.24 | 40.38 | 42.11
$k=4$ | 32.45 | 22.27 | 46.29 | 53.54 | 56.10 | 31.72 | 40.40 | 42.22
$k=8$ | 32.58 | 22.52 | 46.09 | 53.12 | 56.01 | 31.55 | 40.31 | 42.06
$k=12$ | 32.68 | 21.92 | 46.14 | 53.34 | 55.71 | 31.22 | 40.17 | 41.98
Table 11: Detailed BLEU scores of the hyper-parameter search experiments. The
highest BLEU score of each test set in each hyper-parameter search is shown in
bold.


# The evolution of inner disk radius with orbital phase in Circinus X-1

Yanan Wang1,2 Guoqiang Ding1 Chunping Huang1,2 1Xinjiang Astronomical
Observatory, Chinese Academy of Sciences,
150, Science 1-Street, Urumqi, Xinjiang 830011, China
email<EMAIL_ADDRESS><EMAIL_ADDRESS>
2University of Chinese Academy of Sciences,
China

(2012)

###### Abstract

Using RXTE observations, we investigate the evolution of inner disk radius
($R_{\rm in}$) of Cir X-1 during two cycles and find obvious orbital
modulation. We argue that the modulation is attributed to its high orbital
eccentricity. The disk luminosity is inversely with the inner disk temperature
($kT_{\rm in}$), which is ascribed to the slow increase of $kT_{\rm in}$ and,
however, the rapid decrease of $R_{\rm in}$ during the passage for the neutron
star to depart from the companion star.

###### keywords:

Compact object, Neutron star, Accretion disk, Circinus X-1

††volume: 290††journal: Feeding compact objects: Accretion on all
scales††editors: C.M. Zhang, T. Belloni, M. Méndez & S.N. Zhang, eds.

##  1 Introduction

Circinus X-1 (Cir X-1), with an orbital period of 16.6 days ([Kaluzienski et
al. (1976), Kaluzienski et al. 1976]) and a distance of 5.5 kpc ([Case &
Bhattacharya (1998), Case & Bhattacharya 1998]), is a X-ray binary. Its
compact star has been considered as a neutron star (NS) since the discovery of
type-I X-ray bursts in this source ([Tennant et al. (1986), Tennant et al.
1986]). One of the special parameters of Cir X-1 is its high orbital
eccentricity ($\sim$ 0.7-0.9) ([Johnston et al. (1999), Johnston et al.
1999]), which makes it a peculiar source. [Shirey et al. (1996), Shirey et al.
(1996)] found the orbital modulation for the spectrum and quasi-periodic
oscillation (QPO) of Cir X-1 and [Ding et al. (2006b), Ding et al. (2006b)]
found that its power-law (PL) hard tail, evolving on its hardness-intensity
diagram (HID) ([Ding et al. (2003), Ding et al. 2003]), is modulated by
orbital phase too. At the periastron, a long-term dip was present ([Ding et
al. (2006a), Ding et al. 2006a]). For explaining the behaviors of observed
optical and infrared (IR) emission lines, [Johnston et al. (1999), Johnston et
al. (1999)] proposed that the accretion and accretion disk of this NS X-ray
binary (NSXB) could evolve with orbital phase.

##  2 Data Analysis

With software HEASOFT 6.11 and FTOOLS V.6.11, we choose the observations
during two orbital periods (1996 September 21–October 7, 1996 March 8–19) of
Cir X-1 to perform our analysis. Following [Stewart et al. (1991), Stewart et
al. (1991)], the time of zero phase is given by the ephemeris equation

$JD_{0}=2443076.87+(16.5768-0.0000353N)N.$ (1)

We produce the background-subtracted PCA spectra at different phases. [Shirey
et al. (1999), Shirey et al. (1999)] used several spectral models to fit the
PCA spectra of Cir X-1 and found that the best-fit model is the so-called
Eastern model, consisting of a blackbody (BB) and a multicolor disk blackbody
(MCD), which are interpreted as the emission from the NS surface and the
optically thick accretion disk, respectively. We adopt this model, use it to
fit the spectra of Cir X-1 during the two orbital periods, and then get the
inner disk temperature $(T_{\rm in})$, inner disk radius ($R_{\rm in}$), BB
temperature $(T_{\rm bb})$, and BB radius $(R_{\rm bb})$. The BB radii
multiplied by a coefficient are considered as the typical NS radius.

##  3 Result and Discussion

As shown in the right panel of Figure 1, the disk emission deviates from the
relation of $L\propto T^{4}$, which indicates that $R_{\rm in}$ is varied,
because of $L_{\rm disk}=4\pi R^{2}_{\rm in}\sigma T^{4}_{\rm in}$. In panel A
of Figure 1, one can see that at the periastron (phase 0-0.1) $R_{\rm in}$
increases abruptly, then from phase 0.1 to the apastron (phase 0.5) $R_{\rm
in}$ decreases rapidly, and, finally, from the apastron to phase 1 the $R_{\rm
in}$ roughly steadies. As suggested by [Johnston et al. (1999), Johnston et
al. (1999)], at the periastron the large tidal force could make the disk
unstable, resulting in large variation of $R_{\rm in}$; after the apastron
until the apastron the disk is formed gradually and meanwhile the NS departs
from the companion star, but the disk moves towards the NS due to decrease of
radiation pressure; after the apastron, steady accretion takes place on the
formed disk. It is obvious that during the passage for the NS to leave the
companion star the slow increase of $(T_{\rm in})$ and, however, the rapid
decrease of $R_{\rm in}$ contribute to the inverse correlation between the
disk luminosity and the inner disk temperature.

##  4 Acknowledgements

This work is supported by the National Basic Research Program of China (973
Program 2009CB824800) and the Natural Science Foundation of China under grant
no. 11143013.

## References

  * [Case & Bhattacharya (1998)] Case, G.L., & Bhattacharya, D. 1998, ApJ, 504, 761
  * [Ding et al. (2003)] Ding, G.Q., Qu, J.L., & Li, T. P. 2003, ApJ, 596, L219
  * [Ding et al. (2006a)] Ding, G.Q., Qu, J.L., & Li, T. P. 2006a, AJ, 131, 1693
  * [Ding et al. (2006b)] Ding, G.Q., Zhang, S.N., Li, T.P., & Qu, J. L. 2006b, ApJ, 645, 576
  * [Kaluzienski et al. (1976)] Kaluzienski, L.J., Holt, S.S., Boldt, E.A., & Serlemitsos, P.J. 1976, ApJ, 208, L71
  * [Johnston et al. (1999)] Johnston, H.M., Fender, R., & Wu, K. 1999, MNRAS, 308, 415
  * [Shirey et al. (1999)] Shirey, R.E., Bradt, H.V., & Levine, A.M. 1999, ApJ, 517, 472
  * [Shirey et al. (1996)] Shirey, R.E., Bradt, H.V., Levine, A.M., & Morgan, E.H. 1996, ApJ, 469, L21
  * [Stewart et al. (1991)] Stewart, R.T., Nelson, G.J., Penninx, W., Kitamoto, S., Miyamoto, S., & Nicolson, G.D. 1991, MNRAS, 253, 212
  * [Tennant et al. (1986)] Tennant, A.F., Fabian, A.C., & Shafer, R.A. 1986, MNRAS, 221, 27P

Figure 1: Left panel: the evolution of inner disk radius and the inferred NS
radius along orbital phase. Right panel: the luminosities of spectral
components (MCD/BB) vs. their characteristic temperatures; the dashed line and
dotted line correspond to $L=4\pi R^{2}\sigma T^{4}$, with $R=15\ {\rm km}$
and $R=3\ {\rm km}$, respectively; the solid line corresponds to $L\propto
T^{-2.6}$.