assignments under the FSM are generated using the open source R package FSM
available on CRAN. The total runtime of the FSM for the 800 simulated
experiments was about one and a half minutes on a Windows 64-bit computer with
an Intel(R) Core i7 processor. See Chattopadhyay et al. (2021) for detailed
step-by-step instructions and vignettes on the use of FSM package.

####  H.2 Balance

We evaluate balance on the main and transformed covariates. Figures A1(a) and
A1(b) show density plots of the Absolute Standardized Mean Differences (ASMD;
Rosenbaum and Rubin 1985, Stuart 2010) of the six main covariates and their
second-order transformations (including squares and pairwise products),
respectively. A smaller ASMD for a covariate indicates better mean-balance on
that covariate between the two treatment groups. Figure A1(a) indicates that
both rerandomization methods improve balance on the means of the original
covariates over CRD. As expected, the ASMD distribution under RR 0.001 is more
concentrated than that of RR 0.01, with 32% smaller mean ASMD than RR 0.01.
Both the FSM and RR 0.001 have similar distributions of the ASMD with FSM
having moderately (9%) smaller mean ASMD. See Table A5 for a comparison of the
average ASMD of each covariate.

(a) Main covariates

(b) Squares and pairwise products

(c) Frobenius norm

Figure A1: Panels (a) and (b) show distributions of absolute standardized mean
differences (ASMD) of the main covariates and all their second-order
transformations across 800 randomizations. For each plot, the legend presents
the average ASMD across simulations for the four methods. Panel (c) shows
distributions of discrepancies between the correlation matrices of the
covariates in the treatment and the control group (as measured by the
Frobenius norm, $||\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}||_{F}$). On
average the FSM achieves better covariate balance. In terms of the main
covariates, the FSM marginally outperforms RR 0.001. In terms of the second-
order transformations and correlation matrices, the FSM substantially
outperforms RR 0.001.

Figure A1(b) shows that the imbalances of covariate transformations are
substantially smaller with the FSM than with CRD, RR 0.01, and RR 0.001. In
fact, the FSM achieves a 70% reduction in the mean ASMD with respect to RR
0.001. Thus, although the FSM and RR 0.001 exhibit comparable balance in terms
of the main covariates, the FSM balances these transformations of the
covariates much better than RR 0.001. This highlights the improved robustness
of the FSM against model misspecification. Moreover, reducing the tuning
parameter of rerandomization from 0.01 to 0.001 yields only 2% improvement in
the mean ASMD.999In fact, for some covariate transformations, reducing this
tuning parameter exacerbates imbalance (see Table H.4). In Figure A1(b), both
RR 0.01 and RR 0.001 often produce ASMD larger than 0.1, and in some cases,
larger than 0.5, indicative of substantial imbalances on these covariate
transformations.

For each method, we also compare balance in the overall correlation structure
of the covariates. Figure A1(c) shows the boxplots of the distributions of
$||\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}||_{F}$. The FSM outperforms
the other three designs with at least 75% smaller average
$||\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}||_{F}$. In particular, among
the 800 randomizations, the highest value of
$||\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}||_{F}$ under FSM is smaller
than the corresponding lowest value under the other three designs, indicating
that in terms of the correlation structure (and hence the interactions) of the
covariates, the least balanced realization of the 800 FSMs exhibits better
balance than the best balanced realization of the 800 complete randomizations
and rerandomizations.

####  H.3 Efficiency

We now compare the efficiency of the methods under both model- and
randomization-based approaches to inference. Under the model-based approach,
we consider a potential outcome model where
$\mathbb{E}\\{Y_{i}(g)|\bm{X}_{i}\\}$ is linear in $\bm{X}_{i}$ (Model A1) and
another model where $\mathbb{E}\\{Y_{i}(g)|\bm{X}_{i}\\}$ is linear in
$\bm{X}_{i}$ and all its second-order transformations (Model A2). For each
potential outcome model, we fit the corresponding observed outcome model by
OLS and estimate $\text{PATE}_{2,1}$ using the regression imputation method
described in Section 5. Tables 3(b)(a) and 3(b)(b) show the average and
maximum model-based standard error (SE) of the regression imputation estimator
relative to the FSM across 800 randomizations under the two models.

Table A3: Average and maximum model-based standard errors relative to the FSM
across randomizations. Under Model A1 (linear model on the main covariates),
the FSM and RR exhibit similar performance, improving over CRD. Under Model A2
(linear model on the main covariates and their second-order transformations),
the FSM is considerably more efficient than both CRD and RR.

(a) Model A1

| Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
Average SE | 1.03 | 1.00 | 1.00 | 1.00
Maximum SE | 1.13 | 1.00 | 1.00 | 1.00

(b) Model A2

| Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
Average SE | 1.39 | 1.27 | 1.26 | 1.00
Maximum SE | 3.61 | 1.97 | 1.80 | 1.00

Under Model A1, since both rerandomization and the FSM are able to adequately
balance the means of the original covariates, they lead to lower SE (hence,
higher efficiency) than CRD. Across randomizations, the worst case SE under RR
0.01, RR 0.001, and the FSM are 13% smaller than under CRD. Under Model A1,
the FSM has similar model-based SE as the two rerandomization methods.
However, under Model A2, the FSM uniformly outperforms the other three
designs, with a 26% reduction in average SE and an 80% reduction in maximum SE
than RR 0.001. This improvement in efficiency can be attributed to the balance
achieved by the FSM on the main covariates and their squares and pairwise
products. In sum, when the model assumed at the design stage is correct and is
used at the analysis stage, the FSM is as efficient as the two
rerandomizations for estimating the treatment effect. However, when the model
assumed at the design stage is misspecified and later corrected by augmenting
transformations of the covariates (e.g., squares and pairwise products), the
FSM is considerably more efficient and robust than the other designs.

Under the randomization-based approach, we compare the standard errors of the
difference-in-means statistic under each design. Following Hainmueller (2012),
the potential outcomes are generated using the models:
$Y(1)=X_{1}+X_{2}+X_{3}-X_{4}+X_{5}+X_{6}+\eta$, $Y(2)=Y(1)$ (Model B1) and
$Y(1)=(X_{1}+X_{2}+X_{5})^{2}+\eta$, $Y(2)=Y(1)$ (Model B2), where
$\eta\sim\mathcal{N}(0,1)$. Both generative models satisfy the sharp-null
hypothesis of zero treatment effect for every unit and hence,
$\text{SATE}_{2,1}=0$. Conditional on these potential outcomes,
$\text{SATE}_{2,1}$ is estimated under each design using the standard
difference-in-means estimator. The corresponding randomization-based SE of
this estimator is obtained by generating 800 randomizations of the design and
computing the standard deviation of the difference-in-means estimator across
these 800 randomizations. Table 4(b) shows the randomization-based SE of the
difference-in-means statistic for $\text{SATE}_{2,1}$ under each model.

Table A4: Randomization-based standard errors relative to the FSM. The
standard error for the FSM is 0.2 under Model B1 (linear model on the main
covariates) and 0.43 under Model B2 (linear model on the main covariates and
their second-order transformations). Especially under Model B2, the FSM is
considerably more efficient than both CRD and RR.

(a) Model B1

| Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
SE | 2.72 | 1.26 | 1.08 | 1

(b) Model B2

| Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
SE | 5.69 | 4.56 | 4.47 | 1

Under Model B1, the potential outcomes depend linearly on the covariates and
therefore balancing the means of the covariates improves efficiency. This is
reflected in Table 4(b) as the FSM has the smallest SE, closely followed by RR
0.001. Under Model B2, the potential outcomes depend linearly on the squares
and pairwise products of the covariates. By better balancing these
transformations, the FSM yields a considerably smaller SE than the other
designs. In particular, under Model B2, the SE under the FSM is 67% smaller
than the SE under RR 0.001. Therefore, as in the model-based approach, in the
randomization-based approach the FSM exhibits comparable efficiency to
rerandomization under correct-specification of the outcome model and
considerable robustness under model misspecification.

####  H.4 Additional tables and figures from the simulation study

Table A5: Averages of the ASMD of the original covariates across 800
randomizations.

Covariates | Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
$X_{1}$ | 0.162 | 0.051 | 0.035 | 0.029
$X_{2}$ | 0.156 | 0.048 | 0.033 | 0.025
$X_{3}$ | 0.158 | 0.049 | 0.033 | 0.042
$X_{4}$ | 0.150 | 0.049 | 0.034 | 0.029
$X_{5}$ | 0.140 | 0.052 | 0.034 | 0.029
$X_{6}$ | 0.141 | 0.052 | 0.036 | 0.035
Mean | 0.151 | 0.050 | 0.034 | 0.032

Covariate  transformations | Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
$X_{1}X_{2}$ | 0.144 | 0.153 | 0.148 | 0.041
$X_{1}X_{3}$ | 0.144 | 0.140 | 0.137 | 0.041
$X_{1}X_{4}$ | 0.141 | 0.148 | 0.147 | 0.023
$X_{1}X_{5}$ | 0.150 | 0.135 | 0.134 | 0.035
$X_{1}X_{6}$ | 0.152 | 0.109 | 0.101 | 0.030
$X_{2}X_{3}$ | 0.147 | 0.147 | 0.146 | 0.051
$X_{2}X_{4}$ | 0.140 | 0.155 | 0.150 | 0.027
$X_{2}X_{5}$ | 0.147 | 0.143 | 0.136 | 0.030
$X_{2}X_{6}$ | 0.152 | 0.115 | 0.104 | 0.026
$X_{3}X_{4}$ | 0.141 | 0.143 | 0.152 | 0.032
$X_{3}X_{5}$ | 0.149 | 0.140 | 0.139 | 0.096
$X_{3}X_{6}$ | 0.148 | 0.099 | 0.091 | 0.035
$X_{4}X_{5}$ | 0.148 | 0.132 | 0.130 | 0.037
$X_{4}X_{6}$ | 0.152 | 0.100 | 0.095 | 0.027
$X_{5}X_{6}$ | 0.146 | 0.095 | 0.094 | 0.024
$X_{1}^{2}$ | 0.140 | 0.145 | 0.143 | 0.031
$X_{2}^{2}$ | 0.151 | 0.155 | 0.150 | 0.038
$X_{3}^{2}$ | 0.144 | 0.136 | 0.132 | 0.041
$X_{4}^{2}$ | 0.143 | 0.145 | 0.147 | 0.053
$X_{5}^{2}$ | 0.142 | 0.073 | 0.067 | 0.013
Mean | 0.146 | 0.130 | 0.127 | 0.037
$X_{5}^{1.5}$ | 0.141 | 0.060 | 0.048 | 0.018
$X_{2}^{3}$ | 0.155 | 0.090 | 0.081 | 0.071
$X_{4}^{4}$ | 0.140 | 0.143 | 0.147 | 0.072
$\frac{1}{4+X_{3}}$ | 0.157 | 0.073 | 0.064 | 0.050
Mean | 0.148 | 0.092 | 0.085 | 0.053

Table A6: Averages of the ASMD of squares, pairwise products, and other
transformations of the covariates across 800 randomizations.

Figure A2: Boxplot of the distribution of
$||\underline{\bm{S}}_{1}-\underline{\bm{S}}_{2}||_{F}$ across 800
randomizations, where $\underline{\bm{S}}_{g}$ is the sample covariance matrix
of the covariates in treatment group $g\in\\{1,2\\}$.

####  H.5 Additional tables and figures from the Health Insurance Experiment

Covariates | Designs
---|---
CRD | RR Wilks | RR Mahalanobis | FSM
$X_{1}:$ Log family size | 0.052 | 0.039 | 0.038 | 0.012
$X_{2}:$ Log family income | 0.052 | 0.040 | 0.043 | 0.010
$X_{3}:$ Max hourly wage | 0.051 | 0.042 | 0.047 | 0.017
$X_{4}:$ Adult med visits | 0.049 | 0.043 | 0.041 | 0.014
$X_{5}:$ Kid med visits | 0.048 | 0.039 | 0.040 | 0.010
$X_{6}:$ Female | 0.047 | 0.039 | 0.040 | 0.010
$X_{7}:$ Age 0 to 5 | 0.053 | 0.038 | 0.039 | 0.010
$X_{8}:$ Age 6 to 17 | 0.051 | 0.041 | 0.039 | 0.011
$X_{9}:$ Age 18 to 44 | 0.053 | 0.038 | 0.040 | 0.010
$X_{10}:$ Male HS Grad | 0.051 | 0.038 | 0.041 | 0.006
$X_{11}:$ Male more than HS | 0.048 | 0.037 | 0.041 | 0.006
$X_{12}:$ Insured | 0.049 | 0.040 | 0.038 | 0.010
$X_{13}:$ Excellent health | 0.052 | 0.040 | 0.037 | 0.009
$X_{14}:$ Good health | 0.053 | 0.038 | 0.037 | 0.010
$X_{15}:$ Family income mis | 0.052 | 0.038 | 0.041 | 0.011
$X_{16}:$ Max hourly wage mis | 0.051 | 0.038 | 0.041 | 0.013
$X_{17}:$ Adult med visits mis | 0.054 | 0.040 | 0.040 | 0.011
$X_{18}:$ Kid med visits mis | 0.057 | 0.041 | 0.039 | 0.011
$X_{19}:$ Education male mis | 0.048 | 0.038 | 0.041 | 0.008
$X_{20}:$ Insured mis | 0.048 | 0.039 | 0.038 | 0.011
Mean | 0.051 | 0.039 | 0.040 | 0.011

Table A7: Average ASMD of the main covariates between treatment groups $1$ and
$2$ across 400 randomizations.

Treatment group | Designs
---|---
CRD | RR Wilks | RR Mahalanobis | FSM
$1,2$ | 0.051 | 0.039 | 0.040 | 0.011
$1,3$ | 0.055 | 0.041 | 0.043 | 0.011
$1,4$ | 0.049 | 0.038 | 0.039 | 0.010
$2,3$ | 0.056 | 0.043 | 0.045 | 0.012
$2,4$ | 0.053 | 0.040 | 0.041 | 0.010
$3,4$ | 0.056 | 0.042 | 0.044 | 0.012
Mean | 0.053 | 0.040 | 0.042 | 0.011

Table A8: Averages of the ASMD between each pair of treatment groups across
the original covariates and across 400 randomizations.

Covariates | Designs
---|---
CRD | RR Wilks | RR Mahalanobis | FSM
$X_{1}X_{2}$ | 0.053 | 0.039 | 0.041 | 0.020
$X_{1}X_{3}$ | 0.053 | 0.047 | 0.046 | 0.027
$X_{1}X_{4}$ | 0.054 | 0.045 | 0.045 | 0.020
$X_{1}X_{5}$ | 0.049 | 0.040 | 0.041 | 0.013
$X_{2}X_{3}$ | 0.054 | 0.049 | 0.053 | 0.038
$X_{2}X_{4}$ | 0.050 | 0.045 | 0.048 | 0.017
$X_{2}X_{5}$ | 0.052 | 0.039 | 0.039 | 0.015
$X_{3}X_{4}$ | 0.054 | 0.043 | 0.045 | 0.022
$X_{3}X_{5}$ | 0.050 | 0.042 | 0.046 | 0.022
$X_{4}X_{5}$ | 0.054 | 0.044 | 0.045 | 0.015
$X^{2}_{1}$ | 0.053 | 0.041 | 0.040 | 0.026
$X^{2}_{2}$ | 0.051 | 0.041 | 0.042 | 0.015
$X^{2}_{3}$ | 0.057 | 0.055 | 0.058 | 0.026
$X^{2}_{4}$ | 0.053 | 0.053 | 0.053 | 0.012
$X^{2}_{5}$ | 0.051 | 0.043 | 0.044 | 0.004
Mean | 0.053 | 0.044 | 0.046 | 0.019

Table A9: Averages of the ASMD of the squares and pairwise products of the
(demeaned) covariates $X_{1}$,…, $X_{5}$ between treatment groups $1$ and $2$
across 400 randomizations.

Treatment group | Designs
---|---
CRD | RR 0.01 | RR 0.001 | FSM
$1,2$ | 0.053 | 0.044 | 0.046 | 0.019
$1,3$ | 0.056 | 0.046 | 0.048 | 0.020
$1,4$ | 0.051 | 0.043 | 0.044 | 0.017
$2,3$ | 0.058 | 0.049 | 0.049 | 0.021
$2,4$ | 0.054 | 0.046 | 0.046 | 0.018
$3,4$ | 0.058 | 0.048 | 0.049 | 0.023
Mean | 0.055 | 0.046 | 0.047 | 0.020

Table A10: Averages of the ASMD between each pair of treatment groups across
the squares and pairwise products of the (demeaned) covariates $X_{1}$,…,
$X_{5}$ and across 400 randomizations. Figure A3: Distributions of ASMD of all
cubes and three-way interactions of the non-binary covariates across
randomizations.

(a) $||\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}||_{F}$

(b) $||\underline{\bm{R}}_{3}-\underline{\bm{R}}_{4}||_{F}$

Figure A4: Distributions of discrepancies of the correlation matrices of the
covariates in the treatment groups of the HIE data across randomizations. The
discrepancies are measured by
$||\underline{\bm{R}}_{g}-\underline{\bm{R}}_{g^{\prime}}||_{F}$, where
$\underline{\bm{R}}_{g}$ is the sample correlation matrix of the covariates in
treatment group $g$ and $||\cdot||_{F}$ is the Frobenius norm. The FSM
systematically produces lower discrepancies than the other methods, exhibiting
substantially improved balance on the correlations of the covariates.

(a) $||\underline{\bm{S}}_{1}-\underline{\bm{S}}_{2}||_{F}$

(b) $||\underline{\bm{S}}_{3}-\underline{\bm{S}}_{4}||_{F}$

Figure A5: Boxplot of the distribution of
$||\underline{\bm{S}}_{g}-\underline{\bm{S}}_{g^{\prime}}||_{F}$ across 400
randomizations, where $\underline{\bm{S}}_{g}$ is the sample covariance matrix
of the covariates in treatment group $g\in\\{1,2,3,4\\}$.

We now compare the efficiency of the designs in the randomization-based
approach with four additional potential outcome models given below.

  * •

Model B3: $Y(3)=5-3X_{1}+X_{2}+X_{3}-0.2X_{4}+0.8X_{5}+\eta$, $Y(3)=Y(2)$.

  * •

Model B4:
$Y(3)=5-2X^{2}_{1}+0.5X^{2}_{3}+0.5X^{2}_{5}+5X_{1}X_{2}-0.8X_{3}X_{5}+\eta$,
$Y(3)=Y(2)$.

  * •

Model B5: $Y(3)=10+8X_{1}X_{2}+3X_{2}X_{5}-0.5X_{3}X_{5}+\eta$, $Y(3)=Y(2)$.

  * •

Model B6: $Y(3)=0.8X_{1}X_{2}-3X^{2}_{3}+\frac{1}{1+X_{4}}-4X^{3}_{1}+\eta$

For each model, the error term $\eta\sim\mathcal{N}(0,1.5^{2})$. Under each
design, $\text{SATE}_{3,2}$ is estimated using the standard difference-in-
means estimator and the corresponding randomization-based SE is obtained by
generating 800 randomizations and computing the standard deviation of the
estimator across these 800 randomizations. The results are summarized in Table
A11.

Table A11: Randomization-based standard errors relative to the FSM under Models B3, B4, B5, B6 | Designs
---|---
CRD | RR Wilks | RR Mahalanobis | FSM
Model B3 | 2.12 | 1.56 | 1.71 | 1
Model B4 | 2.14 | 1.75 | 1.81 | 1
Model B5 | 2.92 | 2.33 | 2.36 | 1
Model B6 | 1.51 | 1.37 | 1.33 | 1

We finish this section by evaluating and comparing the covariate balance on
the main covariates and the second order transformations, for CRD, RR, and the
FSM, where RR uses the Mahalanobis distance based on the main covariates only
and accepts $0.1\%$ of the assignments.

(a) Main covariates

(b) Squares and pairwise products

(c) Frobenius norm

Figure A6: Distributions of absolute standardized mean differences (ASMD) of
the main covariates (panel (a)) and the squares and pairwise products of the
scaled covariates (panel (b)) across randomizations. For each plot, the legend
presents the average ASMD across simulations for the four methods. Panel (c)
shows distributions of discrepancies between the correlation matrices of the
covariates in treatment groups 1 and 2 (as measured by the Frobenius norm,
$||\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}||_{F}$). In terms of the main
covariates, second-order transformations, and correlation matrices, the FSM
substantially outperforms CRD and RR.

####  H.6 Additional figures from the case studies

Figure A7: Distributions of the absolute standardized mean differences of the
main covariates and their squares and interactions, and the Frobenius norms of
$\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}$ under complete randomization,
rerandomization, and the FSM, for the five studies: (1) Angrist, (2) Blattman,
(3) Durocher, (4) Finkelstein, (5) Lalonde. Figure A8: Distributions of the
absolute standardized mean differences of the main covariates and their
squares and interactions, and the Frobenius norms of
$\underline{\bm{R}}_{1}-\underline{\bm{R}}_{2}$ under complete randomization,
rerandomization, and the FSM, for the five studies: (6) Ambler, (7) Crepon,
(8) Dupas, (9) Karlan, (10) Wantchekon.


# On the Copying Behaviors of Pre-Training
for Neural Machine Translation

Xuebo Liu1 , Longyue Wang2, Derek F. Wong1, Liang Ding3,
Lidia S. Chao1, Shuming Shi2 and Zhaopeng Tu2
1NLP2CT Lab, Department of Computer and Information Science, University of
Macau
2Tencent AI Lab 3The University of Sydney
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work was done when Xuebo Liu and Liang Ding were
interning at Tencent AI Lab.

###### Abstract

Previous studies have shown that initializing neural machine translation (NMT)
models with the pre-trained language models (LM) can speed up the model
training and boost the model performance. In this work, we identify a critical
side-effect of pre-training for NMT, which is due to the discrepancy between
the training objectives of LM-based pre-training and NMT.  Since the LM
objective learns to reconstruct a few source tokens and copy most of them, the
pre-training initialization would affect the copying behaviors of NMT models.
We provide a quantitative analysis of copying behaviors by introducing a
metric called copying ratio, which empirically shows that pre-training based
NMT models have a larger copying ratio than the standard one. In response to
this problem, we propose a simple and effective method named copying penalty
to control the copying behaviors in decoding. Extensive experiments on both
in-domain and out-of-domain benchmarks show that the copying penalty method
consistently improves translation performance by controlling copying behaviors
for pre-training based NMT models. Source code is freely available at
https://github.com/SunbowLiu/CopyingPenalty.

##  1 Introduction

Self-supervised pre-training Devlin et al. (2019); Song et al. (2019), which
acquires general knowledge from a large amount of unlabeled data to help
better and faster learning downstream tasks, has an intuitive appeal for
neural machine translation (Bahdanau et al., 2015; Vaswani et al., 2017,
NMT;). One direct way to utilize pre-trained knowledge is initializing the NMT
model with a pre-trained language model (LM) before training it on parallel
data Conneau and Lample (2019); Liu et al. (2020). As a range of surface,
syntactic and semantic information has been encoded in the initialized
parameters (Jawahar et al., 2019; Goldberg, 2019), they are expected to bring
benefits to NMT models and hence the translation quality.

LM Pre-Training: $\mathcal{L}_{\mathrm{PT}}=-\log
P(\bf{x}|\widetilde{\bf{x}})$
---
Source |  Military  Field Marshal Hussein  in attendance.
Target |  Military ruler Field Marshal Hussein Tantawi was in attendance.
NMT Training: $\mathcal{L}_{\mathrm{NMT}}=-\log P(\bf{y}|\bf{x})$
Source |  Military ruler Field Marshal Hussein Tantawi was in attendance.
Target |  Der Militärführer Feldmarschall Hussein Tantawi war anwesend.

Table 1: Training objective gap between Seq2Seq LM pre-training and NMT
training. LM learns to reconstruct a few source tokens and copy most of them,
while NMT learns more translation rather than copying. Underlines denote
artificial noises, and highlights indicate expected copying tokens.

However, there is a discrepancy between the training objective of sequence-to-
sequence LM pre-training and that of NMT training. As shown in Table 1, LM
learns to reconstruct all source tokens with some noises, while NMT learns to
translate most source tokens and copy few of them. Knowles and Koehn (2018)
and Liu et al. (2020) show that LM pre-training requires to copy $\sim$65% of
tokens, while NMT training only needs to copy $<$10%. We believe that
unexpected knowledge can be propagated to the NMT model via pre-training,
which may bias NMT models to mistakenly copy source tokens to the target side.
For example, the source word “Field Marshal” might be mistakenly copied to the
target side by pre-training based NMT models, since such copying behaviors can
be learned in the pre-training stage.

In this paper, we first validate the change of copying behaviors in NMT models
initialized with the pre-training weights. To this end, we propose a metric
named copying ratio to quantitatively measure the extent of copying behaviors
of NMT models. Experimental results on the WMT14 En-De data show that the NMT
model with pre-training improves translation performance at the cost of
introducing more copying predictions. Analyses on model training show that the
NMT model with pre-training attempts to forget the copying behaviors
transferred from pre-training, while the vanilla NMT model learns in the
opposite way.  Due to the dominating copying behaviors in the pre-training,
the copying ratio of pre-training based NMT model (i.e., 10.8%) is much higher
than that of the vanilla NMT model (i.e., 9.3%). Extensive analyses show that
higher copying ratios severely hurt sentence fluency and word accuracy in
translations, particularly for the translation of proper nouns,  establishing
the necessity for controlling the copying behaviors of NMT models.

To tackle this problem, we propose a simple and effective copying penalty to
control the copying behaviors in inference, which requires no modification to
model architectures and training algorithms. Specifically, we introduce a new
regularizing term to the prediction at each time step, which guides the model
to copy source tokens only when the model is highly confident. Experimental
results on the WMT14 English-German and the OPUS German-English benchmark
demonstrate that the proposed approach can significantly control copying
behaviors in NMT models, making the model more accurately generate copying
tokens.

Our contributions are summarized as follows:

  * •

We reveal a critical side-effect of pre-training for NMT, where pre-training
introduces more copying behaviors into NMT outputs.

  * •

We propose a simple and effective copying penalty to further improve the
performance of NMT models with pre-training by controlling copying behaviors
in generated translation.

  * •

We find that the domain containing a large number of copying tokens (e.g., the
IT) benefits more from the proposed copying penalty.

##  2 Observing Copying Behavior Changes

The fact is that some source words are excessively copied by NMT models from
the source to the target side instead of being translated, which leads to a
high copying ratio in NMT outputs. In this section, we first propose a metric
to measure the copying ratio of model predictions. Second, we quantitatively
investigate the effect of pre-training on NMT in the perspective of copying
behaviors. We expect to provide more evidence for controlling the copying
behaviors of NMT models.

###  2.1 Experimental Setup

#### Data

We conducted experiments on the widely-used WMT14 English-German benchmark. We
used the processed data provided by Vaswani et al. (2017), which consists of
4.5M sentence pairs.111https://drive.google.com/uc?id=0B_bZck-
ksdkpM25jRUN2X2UxMm8 We used all the training data for model training. The
validation set is newstest2013 of 3,000 examples and the test set is
newstest2014 of 3,003 examples.

#### Models and Settings

We implemented all the models by the open-sourced toolkit fairseq (Ott et al.,
2019).222https://github.com/pytorch/fairseq We used 8 V100 GPUs for the
experiments. We mainly compared two models: 1) Random, which is a vanilla NMT
model whose weights are randomly initialized without pre-training; and 2)
Pretrained, an NMT model using the weights of pre-trained
mBART.cc25333https://github.com/pytorch/fairseq/tree/master/examples/mbart for
parameter initialization, which has shown its usability and reliability for
translation tasks (Tran et al., 2020; Tang et al., 2020).

For the training of Random, we used the Transformer big setting of Ott et al.
(2018b) with a huge training batch size of 460K
tokens.444https://github.com/pytorch/fairseq/blob/master/examples/scaling_nmt/README.md#3-train-
a-model For Pretrained, we fine-tuned on the pre-trained mBART.cc25 with a
training batch size of 131K tokens. The hyperparameters keep the same with
Random except the 0.2 label smoothing, 2500 warm-up steps, and 1e-4 maximum
learning rate.

#### Evaluation

For each model, we selected the checkpoint with the lowest perplexity on the
validation set for testing. The beam size is 5 and the length penalty is 0.6.
In addition to reporting the commonly-used 4-gram BLEU score (Papineni et al.,
2002), we also report Translation Error Rate (TER) (Snover et al., 2006) to
better capture the translation performance of unigrams, which more directly
reflects the copying behaviors of NMT models. Both the scores are calculated
by sacrebleu (Post, 2018) with de-tokenized text and unmodified
references.555BLEU+c.mixed+#.1+s.exp+tok.13a+v.1.4.14,666TER+lang.en-
de+tok.tercom-nonorm-punct-noasian-uncased+version.1.4.14

Source |  Military ruler Field Marshal Hussein Tantawi was in attendance.
---|---
Target |  Der Militärführer Feldmarschall Hussein Tantawi war anwesend.
Random |  Anwesend war der Militärmachthaber Feldmarschall Hussein Tantawi.
Pretrained |  Militärischer Feldherr Marshal Hussein Tantawi war anwesend.

Table 2: Translation from English to German. The words in color denote the
copying tokens of which blue denotes right copies and red denotes copying
errors.

###  2.2 Copying Ratio

#### Ratio

To measure the extent of the copying behaviors in NMT models, we calculate the
ratio of copying tokens in translation outputs:

$\displaystyle\mathrm{Ratio}=\frac{\sum_{i=1}^{I}\mathrm{count(copying\
token)}}{\sum_{i=1}^{I}\mathrm{count(token)}}$ (1)

where $I$ denotes the total number of sentences in the test set. We count the
number of “copying token” by comparing each input and output sentence pair.
The denominator is the total number of tokens in output sentences. In general,
higher Ratio values indicate more copying behaviors produced by the NMT model,
and vice versa.

| Performance | Copying
---|---|---
BLEU | TER | Ratio | CER | #S
Oracle | - | - | 8.5% | - | 9
Random | 28.3 | 60.7 | 9.3% | 17.4 | 20
Pretrained | 29.4 | 59.4 | 10.8% | 27.6 | 50

Table 3: Results on the WMT14 En-De test set. Oracle denotes the statistics on
the reference. “#S” denotes the number of instances whose overlaps between the
source and target exceeding 50% (Ott et al., 2018a). Although Pretrained gains
better model performance than Random, it also excessively copies tokens from
the source. Figure 1: Copying ratio (left) and CER score (right) of Random and
Pretrained at different training steps. Reference lines report the values of
the final models gaining the lowest validation perplexities. Pretrained gains
89.4% copying ratio at 0.1K step, which is omitted for better display clarity.
Pretrained learns to forget its copying behaviors by reducing the copying
ratio from 89.4% to 10.8% and CER from 92.4 to 27.6, while Random learns
copying from scratch by increasing the copying ratio and CER from 0 to 9.3%
and 17.4, respectively.

#### Copying Error Rate (CER)

To further analyze the copying problem in NMT models, we propose to calculate
the rate of incorrect copying tokens among all copied ones:

$\displaystyle\mathrm{CER}=\frac{\sum_{i=1}^{I}\mathrm{count(copying\
error)}}{\sum_{i=1}^{I}\mathrm{count(copying\ token)}}$ (2)

where we count the number of “copying error” by checking whether the copying
tokens are included in its reference sentence. The CER is expected to be zero,
which indicates that all copying tokens are correct. Table 2 gives an example.
In experiments, Ratio and CER are computed based on words rather than sub-
words. We further filter all punctuations, which are similar across different
languages.

#### Model Performance

We compare the performance and copying behaviors of the final models in Table
3. The results show that although Pretrained improves the overall performance
in terms of BLEU and TER scores, it tends to generate more copying errors,
limiting its further improvement. In the following part, we probe into the
essence of the copying behaviors via carefully designed experiments.

###  2.3 Learning Curves of Copying Behaviors

We analyze copying behaviors in learning dynamics. Specifically, we translate
the test set using intermediate checkpoints at different training steps, and
then compute corresponding Ratio and CER values. We compare Random and
Pretrained, and plot their learning curves in Figure 1.

#### Ratio

Two models behave quite differently in the early stages of training. Taking
Step 100 for instance, Pretrained copies 89% tokens while Random does not
generate any copying tokens. This demonstrates that the copying habit in the
pre-trained model is transferred to NMT models. As training proceeds, the
copying behaviors of Pretrained are heavily suppressed, resulting in a rapid
drop in Ratio. On the contrary, Random is able to quickly learn copying from
scratch, leading to an upward trend. After learning curves become stable,
Pretrained performs more copying behaviors than Random (10.8% vs. 9.3% Ratio).

| PPL | Rand$>$Pre | Rand$<$Pre
---|---|---|---
Ratio | CER | Ratio | CER
Random | 60.8 | 10.0% | 18.5 | 8.7% | 17.4
Pretrained | 95.1 | 9.5% | 14.9 | 12.5% | 39.7

Table 4: Sentence perplexity on the test set. Pretrained’s translation of
worse perplexity (i.e., “Rand$<$Pre”) also contains the higher Ratio (12.5%)
and CER (39.7) scores.

#### CER

In general, the results of CER show similar trends to those observed in Ratio.
In the beginning, the CER of Pretrained is extremely high (i.e., 92.4),
revealing that most of the copying tokens are incorrect. The reason behind
this phenomenon is that pre-trained models are accustomed to copying source
words but the habit is overly transferred to the downstream translation
models. The interesting finding is that Random also makes more mistakes on
copying at the early training stage. Finally, the error rate of Pretrained is
much higher than that of Random (27.6 vs. 17.4), showing that pre-trained
models indeed expose harmful knowledge to NMT models.

Learning curves of two kinds of models perform in opposite ways: Random learns
copying from scratch while Pretrained tries to forget this behavior. As a
result, Pretrained copies more source tokens than Random and suffers severe
copying errors. This motivates us to further investigate the effects of
copying behaviors on NMT models in terms of translation quality.

###  2.4 Effect of Copying Ratio

#### Sentence Fluency

The copying tokens from the source usually contain some tokens that do not
belong to the target language, which might hurt the fluency of generated
translations. Starting from this intuition, we use an external language model
(Ng et al.,
2019)777https://dl.fbaipublicfiles.com/fairseq/models/lm/wmt19.de.tar.gz
trained on in-domain data to evaluate the fluency of translation outputs. As
shown in Table 4, Random achieves a much better perplexity than mBART (60.8
vs. 95.1 PPL), demonstrating that the NMT model with pre-training generates
less fluent sentences than that trained from scratch.

To take a closer look at the fluency gap, we divide outputs of each model into
two subsets: sentences with better or worse perplexity by comparing Random and
Pretrained. As seen, the fluency of translation is related to copying ratio
and errors. The sentences with higher Ratio and CER scores tend to be less
fluent. Taking Pretrained’s worse subset for example (Rand$<$Pre), it gains a
12.5% Ratio and 39.7 CER scores, confirming that excessive copying behaviors
lead to negative effects in terms of translation fluency.

| Total | PROPN | ADP | NUM | NOUN | Others
---|---|---|---|---|---|---
Ratio | CER∗ | Ratio | CER∗ | Ratio | CER∗ | Ratio | CER∗ | Ratio | CER∗ | Ratio | CER∗
Oracle | 8.5% | - | 5.7% | - | 1.1% | - | 0.8% | - | 0.5% | - | 0.3% | -
Random | 9.3% | 17.4 | 6.3% | 14.5 | 1.3% | 25.8 | 0.9% | 14.2 | 0.5% | 21.3 | 0.3% | 44.4
Pretrained | 10.8% | 27.6 | 7.5% | 27.3 | 1.3% | 24.9 | 0.9% | 14.0 | 0.6% | 21.0 | 0.5% | 63.8
$\Delta$ | +1.5% | +10.2 | +1.2% | +12.8 | 0% | -0.9 | 0% | -0.2 | +0.1% | +0.3 | +0.2% | +19.4

Table 5: Copying behaviors by part-of-speech (POS) bucket on the WMT14 En-De
task. “Oracle” denotes the statistics in the reference. “CER∗” denotes only
using the copying tokens belonging to each POS category for the CER
calculation. $\Delta$ denotes the changes from the Random to Pretrained, in
which the significant ones are bold. Most of the copying tokens are found in
translating proper nouns (PROPN) in Pretrained.

#### Word Accuracy

We also give a word-level analysis by bucketing copying tokens according to
part-of-speech (POS) tags and calculate Ratio and CER in each type. In
experiment, we employ Stanford POS tagger with the german-ud.tagger model to
automatically label output sentences Toutanova et al. (2003). Table 5 lists
the results. The “Oracle” (Row 1) denotes the statistics by comparing the
source input and its reference. As seen, most copying operations should occur
in the type of proper noun (PROPN). This type occupies 5.7% Ratios, followed
by adposition (ADP), numeral (NUM), noun (NOUN), and other types (Others).

Compared with Random, we observe that the increase of Ratio for Pretrained
mainly attributes to copying PROPN words (+1.2%). In addition, Pretrained
generates more copying errors (+10.2), especially on PROPN and Others types
(+12.8 and +19.4). These results reveal that it is necessary to pay more
attention to proper nouns on controlling copying behaviors for NMT.

##  3 Controlling Copying Behaviors

Based on the above experiments, we prove that pre-training indeed changes the
copying behaviors of NMT models, hurting the sentence fluency and word
accuracy of generated translations. To alleviate this issue, we propose a
simple and effective method copying penalty to make the copying behaviors in
NMT controllable.

###  3.1 Copying Penalty (CP)

To control copying behaviors in NMT, an intuitive way is generating copying
tokens only when the model is of high confidence. To this aim, we propose to
modify the probability distribution predicted by the NMT model, decreasing the
predicting probability of the tokens also occurred in the source (i.e.,
weakening the model confidence of making copying predictions). In this way,
for those predictions are wavering between copying and translating, the model
is more likely to translate them, and thus only those confident copying tokens
will be retained. Specifically, during inference, the predicting probability
of $t$-th time step is as follows:

$\displaystyle
P(y_{j}|y_{<j},\mathbf{x})\in\mathds{R}^{\mathcal{V}}=\mathrm{softmax}(\mathbf{y}_{t})$
(3)

where $P(y_{j}|y_{<j},\mathbf{x})$ denotes the probability over the whole
target vocabulary and $\mathbf{y}_{t}$ denotes the decoder output of $t$-th
time step. The search algorithm (e.g., beam search) will take this probability
distribution as a candidate to find the final translation of the source
sentence.

Copying penalty regularizes the prediction probability of each time step by
element-wisely multiplying a new constraint
$\mathrm{CP}\in\mathds{R}^{\mathcal{V}}$:

$\displaystyle\mathrm{CP}=\left\\{\begin{array}[]{lr}1,&y_{j}\notin\mathbf{x}/\mathcal{C}_{\mathrm{punc}}\\\
\alpha,&y_{j}\in\mathbf{x}/\mathcal{C}_{\mathrm{punc}}\end{array}\right.$ (6)

where $\alpha$ is a hyperparameter to control the penalty which can be tuned
on the development set, similar to length penalty (Wu et al., 2016).
$\mathbf{x}/\mathcal{C}_{\mathrm{punc}}$ denotes the set of sources tokens
excluding punctuation and eos, which means that the prediction probabilities
of punctuation and eos will not be penalized. For those predictions not
belonging to the source, their probabilities keep the same. But for those
predictions that are copied from the source, their probabilities will be
$\alpha$ times as large/small as before and the model will be more/less likely
to choose them as candidates for searching.

The proposed method is simple and effective: 1) It does not change the model
architecture and does not need any additional model training, thus no
parameters needed to be newly introduced; 2) Its implementation only requires
some low-cost matrix operations during model inference, slightly slowing the
decoding speed; and 3) It can significantly control the overall copying ratio
of the model predictions, making the model accurately generate copying tokens,
as shown in the following sections.

Figure 2: Copying ratios and CER scores by different copying penalties in
Pretrained. When setting CP smaller than 1 (i.e., penalizing copying), both
the copying ratios and CER scores decrease, and vice versa.

#### Effect of Copying Penalty

Figure 2 depicts the changes of the copying ratio and CER scores when setting
CP to different values on the test data. When setting CP smaller than 1 (i.e.,
punishing copying tokens), only those confident copying predictions will be
made, and thus reducing both the copying ratio and CER scores. Conversely,
setting CP larger than 1 makes the model generate more copying tokens even
some of them are of low confidence, thus both the copying ratios and CER
scores increase. Similar to length penalty (Wu et al., 2016), we also tuned CP
on the development data and found that setting CP to 0.7 wins the best BLEU
score. Therefore, we used this value for decoding test data in the following
experiments.

Empirical results also show that the copying penalty is very efficient. When
evaluated by a single 32GB V100 GPU card, the inference speed of Pretrained is
about 612 token/s, and that of the model with CP is 607 token/s. The extra
latency of the copying penalty is negligible.

| Performance | Copying
---|---|---
BLEU | TER | Ratio | CER | #S
All (3,003 sentences)
Oracle | - | - | 8.5% | - | 9
Random | 28.3 | 60.7 | 9.3% | 17.4 | 20
Pretrained | 29.4 | 59.4 | 10.8% | 27.6 | 50
+CP | 29.6 | 59.0 | 9.2% | 16.8 | 20
HasCopy (1,774 sentences)
Oracle | - | - | 12.5% | - | 9
Random | 29.7 | 59.4 | 13.4% | 13.8 | 15
Pretrained | 30.9 | 57.8 | 14.8% | 20.7 | 32
+CP | 31.0 | 57.5 | 13.0% | 13.2 | 17
NoCopy (1,229 sentences)
Oracle | - | - | 0% | - | 0
Random | 25.4 | 62.9 | 1.2% | - | 5
Pretrained | 26.2 | 62.7 | 2.7% | - | 18
+CP | 26.7 | 62.0 | 1.2% | - | 3

Table 6: Overall results on the WMT14 En-De test set. “Oracle” denotes the
statistics on the reference. “+CP” denotes using the proposed copying penalty
method for inference. “#S” denotes the number of instances whose overlaps
between the source and target exceeding 50%. “HasCopy” denotes evaluating on
the sampled test set containing copies between the source and target, while
“NoCopy” denotes evaluating on the remained set without any copying. CER score
is not applicable in NoCopy as all the copying tokens are copying errors.

###  3.2 Main Results

Table 6 lists the overall results of the model performance and copying
behavior. By looking at the part of evaluating the all test data of 3,003
sentences, the results first confirm the effectiveness of Pretrained that can
consistently improve the model performance in terms of BLEU and TER scores.
However, the introduction of pre-trained knowledge also brings more copying
properties to the model that increases the copying ratio, copying errors, and
the number of copying sentences at the same time. Thanks to the introduction
of the copying penalty, the model successfully alleviates the copying errors
(i.e., reducing the CER score from 27.6 to 16.8), making them be on par with
Random, and thus further improve the BLEU and TER scores over the strong
Pretrained.

| #Num | Total | PROPN
---|---|---|---
Ratio | CER | Ratio | CER∗
All | 3,003 | 10.8% | 27.6 | 7.5% | 27.3
+CP | 9.2% | 16.8 | 6.3% | 15.1
Tgt-Ori | 1,500 | 10.1% | 17.5 | 6.6% | 15.6
+CP | 9.4% | 12.6 | 6.1% | 10.1
Src-Ori | 1,503 | 11.4% | 34.4 | 8.3% | 34.8
+CP | 9.1% | 20.3 | 6.4% | 18.9

Table 7: Copying behaviors of the source original and target original text in
Pretrained. “#Num” denotes the total number of sentences in each test set. The
translation of source original text contains more copying tokens, and CP can
reduce the copying ratio.

To better understand how copying behaviors affect model performance, we split
the test data into two subsets: HasCopy and NoCopy. One intuitive assumption
is that copying errors would significantly hurt the performance of the NoCopy
data since every copying token in the translation is a copying error. The
results confirm our assumption that Pretrained can only improve limited model
performance on the NoCopy data (e.g., improving the TER scores from 62.9 to
62.7), but with the copying penalty, the copying errors less occur in
Pretrained (i.e., reducing the copying ratio from 2.7 to 1.2 and copying
sentences from 18 to 3) and thus, better model performance.

#### Sentence Fluency

The copying penalty improves sentence fluency. In §2.4, we show that the
perplexity of Pretrained (95.1) is worse than that of Random (60.9). However,
after introducing the copying penalty into Pretrained, the perplexity gets a
significant drop from 95.1 to 62.3, which can be on par with Random. This
confirms our hypothesis that more copying behaviors hurt NMT in terms of
translation fluency, and controlling copying behaviors can make the model
generate fluent outputs.

#### Word Accuracy

The copying penalty enhances the translation of PROPN. As shown in Table 7,
the copying penalty improves the translations of proper nouns, reducing the
copying ratio from 7.5% to 6.3% and the CER score from 27.3 to 15.1.

To make headway into the translation of proper nouns, we further investigate
the translations from various sources that usually show the large difference
in the number of proper nouns (Lembersky et al., 2011). Specifically, we
investigate the two kinds of sentence pairs in the WMT14 En-De test set: 1)
the source original text (Src-Ori) that originated in English and was human-
translated into German; 2) the target original text (Tgt-Ori) was translated
in the opposite direction, originating in German with manual translation into
English.

Zhang and Toral (2019) conclude that Tgt-Ori is artificially easier to
translate, resulted in inflated scores for NMT models. Our results of
Pretrained positively support this conclusion that translating Tgt-Ori makes
fewer copying errors and this might be a reason why it can win a better
translation performance. However, by looking at the last row, Src-Ori suffers
from serious copying errors especially in translating proper nouns, making it
harder to translate. Encouragingly, the copying penalty nicely reduces the
copying ratios and copying errors in translating both Src-Ori and Tgt-Ori.
These results further reveal the importance of controlling copying behaviors
in NMT models since translating source original text is the core task of most
NMT systems (Graham et al., 2020).

Figure 3: BLEU scores of different copying penalties in Pretrained. Penalizing
copying (i.e., $\alpha<1$ ) brings benefits to the translations of various
sources. Translating source original sentences is more sensitive to copying
behaviors, leading to a larger score degradation when encouraging copying
(i.e., $\alpha>1$ ).

The above results have shown that copying errors worsen the translation of
Src-Ori. To support the claim, we further investigate the effects of varying
degrees of copying errors in the translations of Src-Ori and Tgt-Ori. Figure 3
shows the change of BLEU scores with different copying penalties. Clearly, the
translation of Src-Ori is more sensitive to copying errors and thus the BLEU
scores get a sharp degradation when setting the copying penalty greater than
1, which verifies our claim.

###  3.3 Out-of-domain Robustness

Improving out-of-domain (OOD) robustness is one of the benefits of pre-
training for NLP tasks (Hendrycks et al., 2020; Tu et al., 2020), but the OOD
sentences usually contain some low-frequency proper nouns which are hard to
translate (Ding et al., 2021). In this part, we take the first step towards
understanding how pre-training affects the OOD robustness of NMT models.

#### Setup

We followed Müller et al. (2020) to preprocess all the used data
sets.888https://github.com/ZurichNLP/domain-
robustness/blob/master/scripts/preprocessing/preprocess_de_en.sh We served the
medical domain as the training domain (i.e., using the data from the medical
domain for model training and validation), which consists of 1.1M training
examples and 2,000 validation examples. The test set of the medical domain
contains 1,691 examples, while the test sets of the IT, Koran, law, and
subtitle domains are with 2,000 examples respectively. For training Random, we
used the Transformer base setting with 32K batch size. The model dropout is
set to 0.3, while the dropouts for attention and inner-FFN activation are set
to 0.2. For Pretrained, apart from using 32K batch size, the other
hyperparameters follow the training of the WMT14 English-German task. The beam
size is set to 5 and the length penalty is set to 1.4. We evaluated the model
performance on the OOD test sets from IT, Koran, Law, and Subtitles domains,
and the averaged BLEU scores can be seen as the OOD robustness of each NMT
model.

| InD | OutD
---|---|---
Med. | Avg. | IT | Kor. | Law | Sub.
Existing | 61.5 | 11.7 | 17.1 | 1.1 | 25.3 | 3.4
+Reg. | 60.8 | 13.1 | - | - | - | -
Random | 60.5 | 11.4 | 20.5 | 1.1 | 20.5 | 3.4
Pretrained | 63.1 | 17.6 | 29.8 | 2.4 | 31.0 | 7.3
+CP | 63.2 | 18.3 | 31.5 | 2.5 | 31.1 | 7.9

Table 8: BLEU scores on the OPUS De-En translation task trained on the in-
domain medical data. “Existing” and “+Reg.” denote the results of baseline and
regularization method from Müller et al. (2020). CP can significantly improve
the translation performance of the IT domain that needs to copy more tokens
from the source.

#### Results

Table 8 lists the results. Clearly, Pretrained substantially improves the
performances of in-domain translation and OOD robustness, increasing the in-
domain BLEU scores from 60.5 to 63.1 and the OOD BLEU scores from 11.4 to 17.6
respectively. The copying penalty can further improve the OOD robustness of
Pretrained that consistently improves the model performance of each OOD test
set. The copying penalty can even remarkably enhance Pretrained in translating
the sentences from the IT domain (when setting CP to 1.2). One possible reason
is that the IT domain needs to copy more tokens from the source sentence than
translating sentences from other domains, thus the copying penalty can play a
greater role and bring a significant performance boost. This also verifies the
effectiveness of the copying penalty.

##  4 Related Work

###  4.1 Pre-Training for NMT

Recently, pre-training has been shown useful for transferring general
knowledge to specific downstream tasks, including text classification,
question answering and natural language inference (Peters et al., 2018;
Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al.,
2019). Compared with training from scratch, fine-tuning a pre-trained model on
downstream datasets usually pushes state-of-the-art performances, while
reducing computational and labeling costs.

Previous studies mainly investigate the effect of pre-training on NMT from two
perspectives: 1) knowledge extraction, where a fixed pre-trained model is used
to encode input sequences into features which are then fed into NMT models;
and 2) parameter initialization, where part/all of the parameters of an NMT
model are initialized by a pre-trained model and then training the model on
downstream datasets (i.e., parallel corpus).

About knowledge extraction, Yang et al. (2020a) and Zhu et al. (2020) explore
enhancing encoder and decoder representations by leveraging pre-trained BERT
models (Devlin et al., 2019). In addition, Chen et al. (2020) distill the soft
labels from BERT to improve predictions for NMT. These methods are effective
but costly because the novel NMT architecture needed to be carefully designed
and the computation graph has to store the parameters of both the pre-trained
model and NMT model.

About parameter initialization, pre-trained models in different architectures
have been studied. For the pre-trained model whose architecture is similar to
Transformer encoder (e.g., BERT) or decoder (e.g., GPT (Radford et al.,
2018)), the parameters of encoder and decoder can be independently initialized
(Conneau and Lample, 2019; Rothe et al., 2020). For the pre-trained model
building upon the encoder-decoder architecture (Sutskever et al., 2014), all
the model parameters can be directly inherited by NMT, which is easy to use
and effective (Song et al., 2019; Lewis et al., 2020; Lin et al., 2020; Yang
et al., 2020b).

In general, most previous works focus on designing novel pre-training methods
and architectures to boost the model performance of NMT, but the understanding
of pre-training for NMT is still limited. This paper improves pre-training for
NMT by first understanding its weakness in copying behavior, revealing the
importance of further identifying the side-effect from pre-training.

###  4.2 Copying Behaviors of NMT

It is a common behavior in Seq2Seq models to copy source tokens to the target
sentences, especially in monolingual generation tasks. For example, Gu et al.
(2016) propose a copying mechanism to explicitly help model learn copying
predictions, showing its effectiveness in the tasks of dialogue and
summarization.

The copying behaviors also exist in NMT, particularly in languages that share
some alphabets (e.g., English and German). Koehn and Knowles (2017) observe
that subword-based NMT (Sennrich et al., 2016) outperforms statistical machine
translation when translating/copying unknown words. Knowles and Koehn (2018)
find that NMT is able to translate source words in specific contexts via
copying (e.g., personal names followed by “Mrs.”), and even these are unknown
words. However, too many copying signals (i.e., source and target sentences
are identical) in training data may lead to one potential threat: NMT models
prefer copying source tokens instead of translating them, resulting in
performance degradation (Ott et al., 2018a; Khayrallah and Koehn, 2018).

This paper broadens the understanding of copying behaviors in NMT models. We
observe that the translation of proper nouns in the source original text
contains more copying tokens, which sheds light upon future works.

##  5 Conclusion and Future Work

We find that NMT models with pre-training are prone to generate more copying
tokens. We introduce a copying ratio and a copying error rate to
quantitatively analyze copying behaviors in NMT evaluation. In addition, a
simple and effective copying penalty is proposed to enhance the copying
behaviors during model inference. Experimental results prove the effectiveness
of the copying penalty, which can effectively control copying behaviors and
improve the overall model performance, especially for the domains (e.g., the
IT) where much copying is needed. Extensive analyses reveal that translating
proper nouns in source original text generates more copying tokens, providing
a direction for the following works on controlling copying behaviors of NMT
models.

In the future, we would like to test the effectiveness of the copying penalty
on the NMT models with other powerful pre-trained models, and explore more
kinds of discrepancies between LM pre-training and NMT training which can be
investigated to improve the performance of NMT models. It is also worthwhile
to adapt it to other Seq2Seq tasks that need to make a large number of copying
predictions, e.g., text summarization and grammar error correction (Liu et
al., 2021).

## Acknowledgement

This work was supported in part by the Science and Technology Development
Fund, Macau SAR (Grant No. 0101/2019/A2), and the Multi-year Research Grant
from the University of Macau (Grant No. MYRG2020-00054-FST). We thank the
anonymous reviewers for their insightful comments.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Chen et al. (2020) Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020.  Distilling knowledge learned in BERT for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7893–7905, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 7057–7067.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2021) Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021.  Rejuvenating low-frequency words: Making the most of parallel data in non-autoregressive translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics_ , Online. Association for Computational Linguistics.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing bert’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_.
  * Graham et al. (2020) Yvette Graham, Barry Haddow, and Philipp Koehn. 2020.  Statistical power and translationese in machine translation evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 72–81, Online. Association for Computational Linguistics.
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016.  Incorporating copying mechanism in sequence-to-sequence learning.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1631–1640, Berlin, Germany. Association for Computational Linguistics.
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020.  Pretrained transformers improve out-of-distribution robustness.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2744–2751, Online. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018.  On the impact of various types of noise on neural machine translation.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 74–83, Melbourne, Australia. Association for Computational Linguistics.
  * Knowles and Koehn (2018) Rebecca Knowles and Philipp Koehn. 2018.  Context and copying in neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3034–3041, Brussels, Belgium. Association for Computational Linguistics.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Lembersky et al. (2011) Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011.  Language models for machine translation: Original vs. translated texts.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 363–374, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin et al. (2020) Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020.  Pre-training multilingual neural machine translation by leveraging alignment information.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2649–2663, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, and Zhaopeng Tu. 2021.  Understanding and improving encoder layer fusion in sequence-to-sequence learning.  In _International Conference on Learning Representations_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Müller et al. (2020) Mathias Müller, Annette Rios, and Rico Sennrich. 2020.  Domain robustness in neural machine translation.  In _Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 151–164, Virtual. Association for Machine Translation in the Americas.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019\.  Facebook FAIR’s WMT19 news translation task submission.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 314–319, Florence, Italy. Association for Computational Linguistics.
  * Ott et al. (2018a) Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018a.  Analyzing uncertainty in neural machine translation.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 3953–3962. PMLR.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ott et al. (2018b) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018b.  Scaling neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 1–9, Brussels, Belgium. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018.  Improving language understanding with unsupervised learning.  Technical report, OpenAI.
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Edinburgh neural machine translation systems for WMT 16.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 371–376, Berlin, Germany. Association for Computational Linguistics.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of association for machine translation in the Americas_ , volume 200. Citeseer.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  MASS: masked sequence to sequence pre-training for language generation.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 5926–5936. PMLR.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 3104–3112.
  * Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020.  Multilingual translation with extensible multilingual pretraining and finetuning.  _arXiv preprint arXiv:2008.00401_.
  * Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003.  Feature-rich part-of-speech tagging with a cyclic dependency network.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 252–259.
  * Tran et al. (2020) Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020.  Cross-lingual retrieval for iterative self-supervised training.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 2207–2219. Curran Associates, Inc.
  * Tu et al. (2020) Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020.  An empirical study on robustness to spurious correlations using pre-trained language models.  _Transactions of the Association for Computational Linguistics_ , 8:621–633.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Yang et al. (2020a) Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. 2020a.  Towards making the most of bert in neural machine translation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9378–9385.
  * Yang et al. (2020b) Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020b.  CSP:code-switching pre-training for neural machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2624–2636, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Zhang and Toral (2019) Mike Zhang and Antonio Toral. 2019.  The effect of translationese in machine translation test sets.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , pages 73–81, Florence, Italy. Association for Computational Linguistics.
  * Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020.  Incorporating BERT into neural machine translation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.


# The Popularity-Homophily Index: A new way to measure Homophily in Directed
Graphs

Shiva Oswal
UC Berkeley Extension, Berkeley, CA, USA

###### Abstract

In networks, the well-documented tendency for people with similar
characteristics to form connections is known as the principle of homophily.
Being able to quantify homophily into a number has a significant real-world
impact, ranging from government fund-allocation to finetuning the parameters
in a sociological model. This paper introduces the “Popularity-Homophily
Index” (PH Index) as a new metric to measure homophily in directed graphs. The
PH Index takes into account the “popularity” of each actor in the interaction
network, with popularity precalculated with an importance algorithm like
Google PageRank. The PH Index improves upon other existing measures by
weighting a homophilic edge leading to an important actor over a homophilic
edge leading to a less important actor. The PH Index can be calculated not
only for a single node in a directed graph but also for the entire graph. This
paper calculates the PH Index of two sample graphs, and concludes with an
overview of the strengths and weaknesses of the PH Index, and its potential
applications in the real world.

Keywords: measuring homophily; popularity-homophily index; social interaction
networks; directed graphs

##  1 Introduction

Homophily, meaning “love of the same,” captures the tendency of agents, most
commonly people, to connect with other agents that share sociodemographic,
behavioral, or intrapersonal characteristics (McPherson et al, 2001). In the
language of social interaction networks, homophily is the “tendency for
friendships and many other interpersonal relationships to occur between
similar people” (Thelwall, 2009). The social, economic, and political outcomes
of homophily are significant. In urban settings for instance, homophilic
tendencies in people tend to create exclaves of a highly concentrated race or
ethnicity (Xu et al, 2021). Chinatown in San Francisco and Little Italy in New
York, are prime examples of homophily in action. Race, socioeconomic factors,
and religion all play a prime role in determining the likelihood of two people
communicating with one another, or living nearby each other (Xu et al, 2021).

Identifying and measuring homophily can have significant real-world
consequences. Identifying homophily in housing can help authorities deal with
dogged pockets of segregation, while identifying homophily in social media can
reveal the importance of certain attributes and users, aiding social media
companies, advertisers, and others (Weng et al, 2010). Measuring homophily
provides numeric detail to the above situations, which in turn can influence
the amount of funding allocated, and resources spent in response. For these
reasons, an effective measure of homophily is needed. The most nuanced models
need to take the structure of the graph into consideration.

This paper presents a new way of measuring the homophily of an attribute in a
directed graph — the “Popularity-Homophily Index” or PH Index for short. The
paper then looks at two example datasets, calculates the PH Index in both
cases, and then compares their results.

The first dataset is a social network of Github developers. The second dataset
is an internet network of links between political blogs in the lead-up to the
2004 US presidential election. These two graphs are radically different in
their contents, but this paper finds that the PH Index has relevance in both.
For sufficiently complex directed graphs, the PH Index is significantly more
powerful than other related measures of homophily in common use, such as the
classic External-Internal Index (EI Index).

##  2 Related Work

The idea and motivation to measure homophily in directed graphs is not new.
For example, Twitter’s social network can be modelled as a directed graph,
with users as nodes, and follower relations as directed edges. Measuring
homophily was an important part of the development of TwitterRank, which
measured the influence of prominent Twitterers in a sample dataset (Weng et
al, 2010). Other papers have sought to determine the extent of the political
“echo chamber” on social media such as Twitter. After classifying users as
either Democratic or Republican based on the content of their tweets, one
paper then sought to measure the political homophily present in the resulting
graph (Colleoni et al, 2014). This measurement allowed them to quantify the
scale and impact of the political echo chamber on social media. Additionally,
“Degree Weighted Homophily” (DWH) takes the structure of the graph into
account, and provably gives a lower-bound for the convergence time of certain
simulations, where nodes represent “agents” that are either attracted to or
repelled by their neighbors (Golub and Jackson, 2012). These simulations are
often excellent models for the behavior of real-world people, such as the
tendency of people of the same race to cluster in the same neighborhoods.
However, DWH is nearly impossible to compute efficiently for real-world
graphs, since its basic formulation takes exponential time to compute.

A recently developed and widely used measure of homophily is called the
_Assortativity Coefficient_ $r$, which satisfies the formula
$r=\frac{\mathrm{tr}(E)-\mathrm{sum}(E^{2})}{1-\mathrm{sum}(E^{2})}$ (Newman,
2003). Easy and efficient to calculate, the assortativity coefficient has been
widely used in research into homophily (Chang et al, 2007). In the case of a
binary classification, $E$ is a 2x2 matrix containing the fraction of each
possible type of edge in the graph. For example, if every node has a color
attribute that is either “black” or “white”, the fraction of edges that start
at a “black” node and end at “white” node is one of the 4 possible kinds of
edges. Additionally, $tr$ is the trace of a matrix (the sum of its main
diagonal), and $sum$ is the sum of the elements of the entire matrix.

A significant issue with the assortativity coefficient is that it does not
take the overall structure of the graph into account since it reduces the
entire graph into a matrix, losing much of the intricate complexity of the
graph.

All of these measures of homophily have their advantages and drawbacks. The PH
Index is influenced by many of these measures, and is designed to be
versatile, easy, and efficient to calculate, while nuanced enough to take the
entire structure of the graph into account.

##  3 Background

Define an edge to be a _homophilic edge_ if it connects two nodes that share
some attribute, and define it be a _non-homophilic edge_ if otherwise. For the
purposes of this paper, assume that every node attribute is binary, and can be
represented as either 0 or 1. Many non-binary attributes can be converted into
binary attributes with a bit of extra work. For example, if every node is a
person with their own “height” attribute, we can create a binary attribute of
‘tallness’. Arbitrarily define a node to be ‘tall’ if its height attribute is
more than 6 feet, and ‘not tall’ if otherwise.

Define a pair of _similar nodes_ to be two nodes (not necessarily connected by
an edge) that have the same value for a given attribute, either both 0 or both
1. For a given node $n$, one of the oldest, simplest, and most powerful
measures of homophily is called the EI Index or alternatively the Coleman
Homophily Index since it was introduced in his landmark book _Introduction to
Mathematical Sociology_ (Coleman, 1964).

In the set of all node _n_ ’s out edges (in the case of a directed graph), let
$e_{n}$ represent the number of “external” non-homophilic edges, and $i_{n}$
represent the number of “internal” homophilic edges. Then the _EI Index_ is
defined by Equation 1.

$\mathrm{EI}=\frac{e_{n}-i_{n}}{e_{n}+i_{n}}$ (1)

Note that the EI Index is -1 for a completely homophilic node (all of its
edges are homophilic edges), and 1 for a completely heterophilic node
(Coleman, 1964). The EI Index for the entire graph is simply the average of
the EI Index for each individual node. Most random graphs that do not
demonstrate homophily tend to have a value of the EI Index of approximately 0.

For the purposes of the PH Index, using the _Weighted EI Index_ (WEI Index) is
more appropriate. The necessity of the WEI Index stems from the fact that in a
binary classification, the total number of nodes with one value of an
attribute may differ significantly from the total number of nodes with the
other value of that same attribute. For example, consider a social network of
200 individuals, with 180 men and 20 women. If Bob (a male) and Jennifer (a
female) both are connected to a single man and a single woman, then their EI
indices will both equal $0.0$. However, common sense suggests that Jennifer
exhibits a greater amount of homophily, since there are very few other women
she could be connected to.

Influenced by the above observation, define the weight $w$ of node _n_ to be
the ratio of the number of non-similar nodes in the entire graph, divided by
the number of similar nodes. Therefore, Bob’s weight would equal 0.11, and
Jennifer’s weight would equal 9.0. For node _n_ ¸ consider the same variables
as in the definition of the EI Index. The equation for the WEI Index is stated
in Equation 2.

$\mathrm{WEI}=\frac{\frac{e_{n}}{w}-i_{n}}{\frac{e_{n}}{w}+i_{n}}$ (2)

The Weighted EI Index essentially normalizes the EI Index, calculating it as
if there were an equal number of nodes with each possible value of the
attribute. In our example above, Bob’s WEI is $0.8$ and Jennifer’s WEI is
$-0.8$. As predicted, after weighting, Jennifer exhibits strong homophily.

##  4 Popularity

Define the _popularity_ of a node to informally represent its importance or
centrality in the graph. In the Twitter social network for example, both
Barack Obama and Kim Kardashian would have a high ‘popularity’. There are
multiple well-known methods to measure popularity in a directed graph. A few
of the most common types are listed below.

###  4.1 In-Degree Centrality

In-Degree Centrality is the simplest measure of popularity. It is also the
fastest, with a runtime of $\mathcal{O}(V+E)$ (Zhang and Luo, 2017).
Logically, it follows that a node with many incoming edges is important. Thus,
a node’s popularity is simply its in-degree, normalized by the total number of
nodes in the graph.

###  4.2 Betweenness Centrality

Betweenness Centrality defines a node’s popularity to be the fraction of
shortest paths that pass through the node. Logically, it follows that the more
central a node is in the graph, the higher the betweenness centrality, and the
higher the importance of that node. The major drawback of this algorithm is
its slowness: without optimization it runs in $\mathcal{O}\left(VE\right)$
(Zhang and Luo, 2017).

###  4.3 PageRank

The PageRank algorithm is a direct upgrade on the In-Degree Centrality
algorithm. It has a time complexity of $\mathcal{O}\left(k(V+E)\right)$, where
$k$ is the maximum number of iterations (the default in NetworkX is 100) to
run the algorithm for. The intuition behind the algorithm is that since not
all nodes are equally popular, they should not all be weighted the same.
Therefore, not all incoming edges are equal, and they should not be counted
equally. The PageRank algorithm was originally developed to order web pages
for the search engine Google (Page et al, 1999). The full details of the
PageRank algorithm are beyond the scope of this paper, although a short
overview is provided. By treating page-rank as a “fluid”, at every iteration
of the algorithm, each node equally distributes all of its “fluid” to its
neighbors. Generally, a node with many incoming edges tends to accumulate a
lot of page-rank, although each incoming edge provides a different amount of
“fluid” based on its node’s current amount of page-rank (Rogers, 2002). While
the argument to justify PageRank may appear circular, the algorithm provably
converges in most cases. Once convergence is reached, the amount of page-rank
a node has is equal to its popularity.

##  5 Popularity-Homophily Index (PH Index)

The Popularity-Homophily Index, presented and developed in this paper, is a
new method for measuring homophily in a directed graph. The PH Index for a
node is closely related to the EI Index, and more specifically to the WEI
Index. The major difference is that the PH index takes into account the
popularity of a node’s set of neighbors.

The first step of the algorithm is to calculate the popularity of every node
using one of the popularity algorithms above and store it in the array $P$.
Additionally, normalize the array so that $\sum P=1$.

The second step is to redefine the variables $e_{n}$ and $i_{n}$ that appeared
in the formula for the WEI Index, as shown in Equation 3. Essentially,
Equation 3 weights these variables by the popularity of the edges they
accumulate over.

$\displaystyle e_{n}$ $\displaystyle=\sum_{i\in\text{External\ Nodes}}P_{i},$
(3) $\displaystyle i_{n}$ $\displaystyle=\sum_{i\in\text{Internal\
Nodes}}P_{i}$

Once $e_{n}$ and $i_{n}$ have been re-calculated, the formula for the PH Index
is the same as the formula for the WEI Index.

_Why does taking popularity into account create a more robust index of
homophily?_ The reason is, in most real-world graphs, ties to influential
nodes are a greater indication of homophilic tendencies than ties to less
influential nodes. That is, not all out-edges are equally important. For
example, research has shown that people’s daily lives are influenced by the
YouTube vloggers that they follow (Ladhari et al, 2020). Additionally, the
vlogger’s popularity tends to be proportional to the amount of influence that
a vlogger has over a person, meaning that the most popular vloggers wield an
enormous amount of influence over legions of devoted viewers (Ladhari et al,
2020). Naturally, this phenomenon extends beyond just vloggers, and it is not
a stretch to say that an object or person’s “popularity” is fundamental to its
role in society.

Finally, the PH Index can easily be extended to measure the homophily of an
entire directed graph. First compute the PH Index for every node in
$\mathcal{O}(P+V+E)$ time. $\mathcal{O}(P)$ represents the time needed to
compute the popularity of every node, and it varies based on the algorithm
used. $\mathcal{O}(V+E)$ represents the time needed to actually compute the
index for each node. Let $\mathrm{PH}_{n}$ represent the value of the PH index
for node $n$. The PH Index for the entire graph is simply a weighted average
of the PH Index for every node. Once again, we need to weight by popularity,
since highly popular nodes have a greater influence on the overall homophily
of the graph than less popular nodes. Since we normalized the popularity
array, the formula is simply the one stated in Equation 4.

$\mathrm{PH}=\sum_{n\in\text{Nodes}}{P_{n}\times\mathrm{PH}_{n}}$ (4)

Like the classic EI Index, a PH Index value of -1 implies complete homophily,
while a value of 1 implies complete heterophily. Most non-homophilic graphs
will have a PH index of approximately 0.

##  6 Experimental Data

This paper demonstrates the use of the PH Index on two sample directed graphs.
The first graph is a network of software engineers on Github, a popular
website for storing and sharing code (Rozemberczki et al, 2021). Two engineers
are connected by a directed edge if one “follows” the other. Additionally, as
part of the data, each engineer job has a binary classification and is either
a web (0) or a machine learning (1) developer. This information was scraped
from the developer profiles by a machine learning algorithm (Rozemberczki et
al, 2021). The links in the graph themselves were scraped from the publicly
available Github API. The data was collected by researchers at the University
of Edinburgh led by Benedek Rozemberczki and accessed from the Stanford
Network Analysis Project (Leskovec and Krevl, 2014).

The second graph is a network of political blogs from the run up to the 2004
US Presidential election (Adamic and Glance, 2005). In this graph, a directed
edge represents a hyperlink from one network to the other. Each blog has a
binary classification as either left leaning (0) or right leaning (1), and the
creators of the second graph used a combination of automated and manual
labelling to create each label. The data was collected by Lada Adamic and
Natalie Glance, and accessed from the KONECT Network Collection (Kunegis,
2013).

For purposes of comparison, two additional attributes were added. For the
Github graph, each developer’s username was added. The “name length” attribute
was engineered from this information based on whether or not the developer’s
username was shorter than 7 characters in length, an arbitrary number. For the
political blogs graph, the second attribute was purely random, set to either 0
or 1 with equal probability.

Basic information about the two graphs is listed in Table 1. For both graphs,
both attributes are binary variables. Note that assortativity, the classic
measure of homophily, ranges from 1 (complete homophily), down to 0 (no
homophily), which is a different scale than the PH Index.

Table 1: Basic information about Github and Blogs graphs Graph |  Nodes ($V$) |  Edges ($E$) |  Attribute 1 |  Assortativity |  Attribute 2 |  Assortativity
---|---|---|---|---|---|---
Github |  37,700 |  2,89,003 |  Developer Type |  0.378 |  Name Length |  0.012
Blogs |  1,224 |  19,025 |  Political Lean |  0.823 |  Pure Random |  0.001

A visualization of both graphs is presented in Figures 1 and 2. Note how
homophily is clearly visible in both graphs, although in the Github graph, the
large number of nodes combined with the relatively small amount of machine
learning developers, make it harder to see. The two-dimensional visualizations
were made with Graphia (Freeman et al, 2020).

Figure 1: Political Lean in the Blog Graph—Blue represents left leaning blogs,
and Red represents right leaning blogs Figure 2: Job in the Github Graph—Blue
represents web developers, and Red represents machine learning developers

To store and manipulate the Github and Blog data, the python module NetworkX
(Hagberg et al, 2008) was used. Both graphs were stored as NetworkX DiGraphs.
All three popularity algorithms mentioned above are implemented into NetworkX,
shortening the code required to compute the PH Index significantly. Figure 3
contains the basic python source-code for computing the weighted EI Index for
a node. Figure 4 contains the basic python source-code for computing the
overall PH Index for a directed graph.

Figure 3: Find the weighted EI Index of node ”n” in digraph ”graph”, based on
attribute ”keyword”, with a precomputed weight ”w” Figure 4: Find the
Popularity Homophily Index for DiGraph ”graph” and attribute ”keyword”

##  7 Results

Using the code for the PH index, the homophily of each attribute in each graph
could be measured. The results are organized in Tables 2 and 3.

Table 2: Github Graph PH Index Popularity Type |  Attribute 1 (Web or ML Developer) |  Attribute 2 (Short or Long Name) |  Python Computation Time (seconds)
---|---|---|---
In-Degree Centrality |  -0.554 |  -0.067 |  0.14
Betweenness Centrality |  -0.537 |  -0.003 |  5077
PageRank |  -0.461 |  -0.113 |  5.22
Table 3: Blog Graph PH Index Popularity Type |  Attribute 1 (Left or Right Lean) |  Attribute 2 (Random Number Generator) |  Python Computation Time (seconds)
---|---|---|---
In-Degree Centrality |  -0.738 |  -0.030 |  0.14
Betweenness Centrality |  -0.813 |  -0.001 |  8.11
PageRank |  -0.709 |  -0.031 |  0.84

While weighting by popularity has a huge impact on the final answer, this
impact can be invisible if there are many popular nodes that are both highly
homophilic and highly heterophilic.

As an example, the popularity, calculated with PageRank of every node in the
Github dataset is plotted below in Figure 5, using the python package
Matplotlib (Hunter, 2007). Note that there are a handful of nodes with a huge
influence. Consider the most popular 150 developers, 74 of them have a WEI
Index below -0.461 (more homophilic than the PH Index), and 76 of them have a
WEI Index above -0.461 (less homophilic than the PH Index). Naturally, since
this split is so even, weighting by popularity does not have a huge impact on
the final answer.

However, we notice a much larger impact of popularity weighting when we
consider the second attribute of the Github graph. A developer is defined to
have a “short” username if his username is under 7 letters long. Approximately
17% of all developers have a “short” username. However, for some unknown
reason, the most popular developers seem to prefer short usernames, meaning
that this “random” attribute can and may actually hold significance.

Out of the top 150 developers, who together account for merely 0.4% of the
nodes but over 20% of the total popularity of the graph, an astounding 28%
have a short username, a statistically significant difference. Due to the
intrinsic nature of PageRank, popular nodes tend to have edges leading to
other popular nodes, meaning that the most popular nodes tend to be slightly
homophilic relative to name length. For this reason, the weighted PH Index
relative to name length is -0.113 as opposed to roughly 0, which clearly
indicates some amount of homophily.

Figure 5: Github Graph Popularity (PageRank)

##  8 Conclusion

The primary purpose of this paper is to introduce the Popularity-Homophily
(PH) Index as a method to compute homophily in directed graphs. The index was
tested on two real-world graphs, the Github developer dataset, and the
Political Blogs dataset. In both cases, homophily was detected, and as
expected from the assortativity coefficient, the magnitude of homophily in the
former graph was smaller than the magnitude of homophily in the latter graph.

The major advantage of the PH Index occurs in graphs that are dominated by a
limited number of main actors. These actors exert disproportional amounts of
influence on the state of the graph, and in the real-world, a handful of
homophilic connections to popular nodes can easily outweigh a multitude of
heterophilic connections to less popular nodes. The PH Index has some
versatility in that one of many possible algorithms can be employed in the
first step, to calculate the popularity of each node. In order to decide which
algorithm to use, the nature of the real-world data being analyzed must be
considered. For most social networks like Twitter or Facebook, the PageRank
algorithm is probably the most effective. However, for geographic data,
Betweenness Centrality may be more preferable due to the literal connection
between that algorithm and geography.

As a warning, the magnitude of the PH Index may be a red herring. The scale is
not really linear, as a PH Index of -0.7 demonstrates much stronger homophily
over -0.4 than what might be expected. Additionally, the magnitude of the PH
Index may have a slightly different meaning in one graph when compared to
another graph, since both graphs will have their own unique structure. For
this reason, the main conclusions one should draw from the PH Index are in
terms of relative magnitude.

##  9 Future Work

The full relevance and applications of the novel PH Index remain to be seen.
Below are a few real-world areas of research where the PH Index may be
applicable.

  * •

On social media, are people with one political lean more homophilic than
people with the opposite political lean?

  * •

In a particular school, is gender, race, income level, age, etc. the main
driver of homophilic relationships?

  * •

In a citation network, professors in which disciplines are the most likely to
cite papers written by professors in other disciplines?

## References

  * Adamic and Glance (2005) Lada A. Adamic and Natalie Glance.  The political blogosphere and the 2004 U.S. election: Divided they blog.  In _Proceedings of the 3rd International Workshop on Link Discovery_ , LinkKDD ’05, page 36–43, New York, NY, USA, 2005. Association for Computing Machinery.  ISBN 1595932151.  doi: 10.1145/1134271.1134277.
  * Chang et al. (2007) Hui Chang, Bei-Bei Su, Yue-Ping Zhou, and Da-Ren He.  Assortativity and act degree distribution of some collaboration networks.  _Physica A: Statistical Mechanics and its Applications_ , 383(2):687–702, 2007.  ISSN 0378-4371.  doi: 10.1016/j.physa.2007.04.045.
  * Coleman (1964) James S. Coleman.  _Introduction to mathematical sociology_.  Free Press of Glencoe, New York, 1964.
  * Colleoni et al. (2014) Elanor Colleoni, Alessandro Rozza, and Adam Arvidsson.  Echo chamber or public sphere? Predicting political orientation and measuring political homophily in Twitter using big data.  _Journal of Communication_ , 64(2):317–332, 03 2014.  ISSN 0021-9916.  doi: 10.1111/jcom.12084.
  * Freeman et al. (2020) Tom C. Freeman, Sebastian Horsewell, Anirudh Patir, Josh Harling-Lee, Tim Regan, Barbara B. Shih, James Prendergast, David A. Hume, and Tim Angus.  Graphia: A platform for the graph-based visualisation and analysis of complex data.  _bioRxiv_ , 2020.  doi: 10.1101/2020.09.02.279349.
  * Golub and Jackson (2012) Benjamin Golub and Matthew O. Jackson.  Network structure and the speed of learning measuring homophily based on its consequences.  _Annals of Economics and Statistics_ , 107/108:33–48, 2012\.  ISSN 21154430, 19683863.  URL http://www.jstor.org/stable/23646571.
  * Hagberg et al. (2008) Aric Hagberg, Pieter Swart, and Daniel Chult.  Exploring network structure, dynamics, and function using NetworkX.  Technical report, Los Alamos National Laboratory, 2008.
  * Hunter (2007) John D. Hunter.  Matplotlib: A 2D graphics environment.  _Computing in Science Engineering_ , 9(3):90–95, 2007.  doi: 10.1109/MCSE.2007.55.
  * Kunegis (2013) Jérôme Kunegis.  KONECT: The Koblenz network collection.  In _Proceedings of the 22nd International Conference on World Wide Web_ , WWW ’13 Companion, page 1343–1350, New York, NY, USA, 2013. Association for Computing Machinery.  ISBN 9781450320382.  doi: 10.1145/2487788.2488173.
  * Ladhari et al. (2020) Riadh Ladhari, Elodie Massa, and Hamida Skandrani.  YouTube vloggers’ popularity and influence: The roles of homophily, emotional attachment, and expertise.  _Journal of Retailing and Consumer Services_ , 54:102027, 2020.  ISSN 0969-6989.  doi: 10.1016/j.jretconser.2019.102027.
  * Leskovec and Krevl (2014) Jure Leskovec and Andrej Krevl.  SNAP Datasets: Stanford large network dataset collection.  http://snap.stanford.edu/data, June 2014.
  * McPherson et al. (2001) Miller McPherson, Lynn Smith-Lovin, and James M Cook.  Birds of a feather: Homophily in social networks.  _Annual Review of Sociology_ , 27(1):415–444, 2001.  doi: 10.1146/annurev.soc.27.1.415.
  * Newman (2003) M. E. J. Newman.  The structure and function of complex networks.  _SIAM Review_ , 45(2):167–256, 2003.  doi: 10.1137/S003614450342480.
  * Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd.  The PageRank citation ranking: Bringing order to the web.  Technical Report 1999-66, Stanford InfoLab, November 1999.  URL http://ilpubs.stanford.edu:8090/422/.
  * Rogers (2002) Ian Rogers.  The Google Pagerank algorithm and how it works, 2002.  URL http://www.iprcom.com/papers/pagerank.
  * Rozemberczki et al. (2021) Benedek Rozemberczki, Carl Allen, and Rik Sarkar.  Multi-Scale attributed node embedding.  _Journal of Complex Networks_ , 9(2), 05 2021.  ISSN 2051-1329.  doi: 10.1093/comnet/cnab014.
  * Thelwall (2009) Mike Thelwall.  Homophily in MySpace.  _Journal of the American Society for Information Science and Technology_ , 60(2):219–231, 2009.  doi: 10.1002/asi.20978.
  * Weng et al. (2010) Jianshu Weng, Ee-peng Lim, and Jing JIang.  TwitterRank: Finding topic-sensitive influential twitterers.  In _Proceedings of the third ACM international conference on Web search and data mining_ , pages 261–270, 2010.
  * Xu et al. (2019) Yang Xu, Alexander Belyi, Paolo Santi, and Carlo Ratti.  Quantifying segregation in an integrated urban physical-social space.  _J. Roy. Soc. Interface_ , 16:20190536, 2019.  doi: 10.1098/rsif.2019.0536.
  * Zhang and Luo (2017/03) Junlong Zhang and Yu Luo.  Degree centrality, betweenness centrality, and closeness centrality in social network.  In _Proceedings of the 2017 2nd International Conference on Modelling, Simulation and Applied Mathematics (MSAM2017)_ , pages 300–303. Atlantis Press, 2017/03.  ISBN 978-94-6252-324-1.  doi: 10.2991/msam-17.2017.68.