# Decoder-Only or Encoder-Decoder? Interpreting Language Model as a
Regularized Encoder-Decoder

Zihao Fu1, Wai Lam2, Qian Yu3, Anthony Man-Cho So2
Shengding Hu4, Zhiyuan Liu4, Nigel Collier1

( 1Language Technology Lab, University of Cambridge,
2The Chinese University of Hong Kong, 3JD.com
4Department of Computer Science and Technology, Tsinghua University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{hsd20<EMAIL_ADDRESS><EMAIL_ADDRESS>)

###### Abstract

The sequence-to-sequence (seq2seq) task aims at generating the target sequence
based on the given input source sequence. Traditionally, most of the seq2seq
task is resolved by the Encoder-Decoder framework which requires an encoder to
encode the source sequence and a decoder to generate the target text.
Recently, a bunch of new approaches has emerged that apply decoder-only
language models directly to the seq2seq task. Despite the significant
advancements in applying language models to the seq2seq task, there is still a
lack of thorough analysis on the effectiveness of the decoder-only language
model architecture. This paper aims to address this gap by conducting a
detailed comparison between the encoder-decoder architecture and the decoder-
only language model framework through the analysis of a regularized encoder-
decoder structure. This structure is designed to replicate all behaviors in
the classical decoder-only language model but has an encoder and a decoder
making it easier to be compared with the classical encoder-decoder structure.
Based on the analysis, we unveil the attention degeneration problem in the
language model, namely, as the generation step number grows, less and less
attention is focused on the source sequence. To give a quantitative
understanding of this problem, we conduct a theoretical sensitivity analysis
of the attention output with respect to the source input. Grounded on our
analysis, we propose a novel partial attention language model to solve the
attention degeneration problem. Experimental results on machine translation,
summarization, and data-to-text generation tasks support our analysis and
demonstrate the effectiveness of our proposed model.

##  1 Introduction

The sequence-to-sequence (seq2seq) task Sutskever et al. (2011, 2014); Cho et
al. (2014); Klein et al. (2017); Ott et al. (2019) sees rapid growth during
the past few years. It takes a source sequence as input and generates a
corresponding target sequence. Several natural language generation tasks, such
as translation, summarization, data-to-text generation, and story generation,
naturally fall under the category of seq2seq tasks. Moreover, recently,
several other non-generation tasks, including question answering,
classification, and etc. Raffel et al. (2020), have also been unified under
the seq2seq paradigm. Traditionally, most of the existing seq2seq frameworks
have employed the Encoder-Decoder (ED) architecture Cho et al. (2014);
Sutskever et al. (2014), where an encoder is responsible for encoding the
input data into a hidden space, while a decoder is used to generate the target
output text.

Figure 1: Encoder-Decoder (ED) framework and decoder-only Language Model (LM).

Recently, many promising large language models (GPT Radford et al. (2018),
GPT2 Radford et al. (2019), GPT3 Brown et al. (2020), InstructGPT/ChatGPT
Ouyang et al. (2022), Palm Chowdhery et al. (2022), OPT Zhang et al. (2022),
Bloom Scao et al. (2022), Galactica Taylor et al. (2022), Llama Touvron et al.
(2023)) have emerged that directly employ a decoder-only Language Model (LM)
Bengio et al. (2000); Mikolov et al. (2010, 2011); Mikolov and Zweig (2012);
Vig and Belinkov (2019) to solve the seq2seq tasks (Figure 1). It concatenates
each source sequence with the corresponding target sequence as a single one
and trains the LM on this new collection of sequences. The decoder-only
structure Liu et al. (2018); Radford et al. (2019); Dong et al. (2019); Brown
et al. (2020); Ghosh et al. (2017) has many obvious advantages over the
commonly used ED framework. First, it only has a decoder and thus reduces the
model size significantly. Second, LM can be pre-trained on unlabeled text data
which is much easier to obtain. Moreover, LM has many good properties
including parameter sharing, layer-wise coordination, etc.

Despite the remarkable achievements of recent large language models, it is
still unclear whether applying the LM in the seq2seq task is a promising
choice regarding the performance of this task. Liu et al. (2018) show that the
LM gets some gains over the ED structure in the summarization task while
Radford et al. (2019); Brown et al. (2020) show that the pre-trained LMs can
be applied in the unsupervised translation task. On the other hand, Raffel et
al. (2020) indicate that the LMs, when directly applied in the machine
translation tasks, perform worse than the classical ED structure. Deng et al.
(2023) show that ED structure still performs better than decoder-only LM
structure. Zhu et al. (2020) indicate that when applying the pre-trained LM to
a language domain beyond the corpus for training, no signiﬁcant improvement is
observed. Fu et al. (2021) show that LM suffers from the repetition generation
problem. Therefore, it motivates us to investigate the deployment and the
performance of LMs in seq2seq tasks.

We propose to study the characteristics of LM when applied in the seq2seq task
by conducting a detailed contrastive study between the LM and the ED
structure. Grounded with our analysis, we unveil the attention degeneration
problem and propose a novel partial attention language model to solve it.
Specifically, we first propose to analyze a variant of the traditional ED
structure named as Regularized Encoder-Decoder (RED) framework. It is designed
to replicate all behaviors in classical LM while structured with an encoder
and a decoder. This structure facilitates the comparison with the ED
structure. By comparing with the RED framework and the ED framework, we find
that some parts of the RED structure benefit the seq2seq task while some parts
do not. Moreover, we find the defects of the LM applied in a seq2seq task are
partially caused by the Attention Degeneration Problem (ADP) in its attention
component. We conduct a theoretical analysis of this problem by deriving an
upper bound of the attention sensitivity and we find that the sensitivity
decreases as the length of the generated sequence grows in the LM. Based on
this analysis, we propose a novel Partial Attention Language Model (PALM)
which takes advantage of some LM components while avoiding other unfavorable
ones. Our experimental results on machine translation, summarization, and
data-to-text generation datasets support the correctness of our analysis and
demonstrate the effectiveness of our proposed PALM structure.

Our contributions can be summarized as follows: (1) We conduct a detailed
analysis of applying the LM in the seq2seq task. (2) We conduct a theoretical
analysis of the attention degeneration problem and carry out extensive
experiments to support it. (3) Based on our analysis, we propose PALM to
alleviate the weakness of LM, from which the recent LM-based methods in
seq2seq tasks can benefit.

##  2 Related Works

Numerous works have recently proposed using LM in the seq2seq task. Liu et al.
(2018) suggest using decoder-only LM to generate summaries of Wikipedia
articles, whereas Raffel et al. (2020) demonstrate through extensive
experiments that LMs perform worse than ED frameworks in machine translation
tasks. On the other hand, several works propose using pre-training to enhance
performance. For instance, Radford et al. (2019); Brown et al. (2020) propose
pre-training the GPT LM and fine-tuning it on an unsupervised seq2seq task.
UniLM Dong et al. (2019) adopts the prefix LM Raffel et al. (2020), which uses
a fully visible mask of the source sequence instead of the traditional causal
mask. Additionally, Conneau and Lample (2019); Conneau et al. (2020) propose
the XLM model pre-trained on a monolingual dataset. However, Zhu et al. (2020)
show that when applying pre-trained LMs to a language domain beyond the
training corpus, there is no significant improvement. Nevertheless, these
results heavily rely on the pre-training process, and it remains unclear
whether the LM structure itself is suitable for handling the seq2seq task.

Compared to the ED structure, the LM has numerous advantageous features that
have been proven to enhance its performance. First, it utilizes the parameter
sharing technique Dehghani et al. (2019); Xia et al. (2019); Lan et al.
(2020); Conneau and Lample (2019) to share parameters for networks that handle
both the source and target sequences. This reduces the model size and improves
performance through parameter sharing. Additionally, the LM incorporates a
layer-wise coordination mechanism Belinkov et al. (2017); He et al. (2018);
Peters et al. (2018) that enables the decoder to attend to each corresponding
encoder layer’s output, allowing it to access various levels of source
information. Finally, Dong et al. (2021) demonstrate that the attention matrix
for the decoder-only is a full rank matrix since it is a triangular matrix.
This is superior to the encoder-decoder attention matrix, which may not be
full rank.

##  3 Contrastive Study of Language Model

In this section, we conduct a contrastive study of the Language Model (LM) and
the traditional Encoder-Decoder (ED) structure. We propose to analyze a
Regularized Encoder-Decoder (RED) framework which is designed to replicate all
behaviors in the classical LM but with a structure of an encoder and a
decoder. We give a detailed explanation of how each component in the RED
framework imitates the behaviors in the LM and how it is different from the ED
framework. We also illustrate how these components benefit or harm the overall
performance. Afterwards, we analyze the attention degeneration problem and
propose a theoretical analysis to unveil the cause.

###  3.1 Preliminary

Our analysis is based on the prevalent Transformer architecture Vaswani et al.
(2017); Ott et al. (2019). As illustrated in Figure 2, we show the main
components of the models and the detailed model structure can be found in Ott
et al. (2019). In a seq2seq task, we denote the input source sequence as
$s=[s_{1},s_{2},\cdots,s_{|s|}]$ and the target sequence as
$t=[t_{1},t_{2},\cdots,t_{|t|}]$, where $|\cdot|$ is the length of the
sequence and $s_{i}$ is the $i$th source word. The positional token for $s$
and $t$ is denoted as $p_{s}=[1,2,\cdots,|s|]$ and $p_{t}=[1,2,\cdots,|t|]$.
The attention layer is denoted as $\mathtt{ATT}(Q,K,V)$, and
$Z=\mathtt{ATT}(Q,K,V)=\mathtt{Softmax}(QW_{Q}W_{K}^{\top}K^{\top}/\sqrt{d})VW_{V}=\mathtt{Softmax}(QA^{\top}K^{\top}/\sqrt{d})VW_{V}$,
in which $A^{\top}=W_{Q}W_{K}^{\top}$ and
$W_{Q},W_{K},W_{V}\in\mathbb{R}^{d\times d}$ are trainable parameters that
transform input matrices into another space. $Q\in\mathbb{R}^{d_{Q}\times
d},K\in\mathbb{R}^{d_{K}\times d},V\in\mathbb{R}^{d_{V}\times d}$ stand for
the query, key and value matrices, $d_{Q}$, $d_{K}$, and $d_{V}$ are the
corresponding dimensions of the matrices while $Z\in\mathbb{R}^{d_{Q}\times
d}$ is the output of the attention layer. Here, the $\mathtt{Softmax}$
operation is applied to each row of the matrix concerned.

As shown on the left of Figure 2, ED’s encoder contains multiple Transformer
encoder blocks denoted with the shaded rectangle.
$G_{l}^{E}\in\mathbb{R}^{|s|\times d}$ is the input feature matrix for the
self attention layer $\mathtt{ATT}_{l}^{E}$ while
$H_{l}^{E}\in\mathbb{R}^{|s|\times d}$ is the output matrix of the $l$th
encoder block. $G_{l}^{E}$ equals to the sum of the word embedding and the
positional embedding for the first block ($l=1$) and $G_{l}^{E}$ equals to
$H_{l-1}^{E}$ when $l>1$. In ED’s decoder, we denote
$G_{l}^{D}\in\mathbb{R}^{|t|\times d}$ as the output of the self attention
layer $\mathtt{ATT}_{l}^{D}$. We feed $G_{l}^{D}$ into the encoder attention
$\mathtt{ATT}_{l}^{J}$ and denote the output as
$Q_{l}^{D}\in\mathbb{R}^{|t|\times d}$. $H_{l}^{D}\in\mathbb{R}^{|t|\times d}$
is the output matrix of the $l$th decoder block. We denote $\mathcal{L}^{D}$
as the negative log likelihood loss for the target sequence. Similarly, as
shown on the right of Figure 2, the RED structure has the same definition of
the matrices as that in ED. Moreover, We denote $\mathcal{L}^{E}$ as the
negative log likelihood loss for the source sequence and we denote the
unidirectional cross attention as $\mathtt{ATT}_{l}$. For more structural
details we refer the readers to Appendix A.3 and the original papers Vaswani
et al. (2017); Ott et al. (2019).

Figure 2: Encoder-Decoder framework (left) and Regularized Encoder-Decoder
framework (right).

###  3.2 Regularized Encoder-Decoder

Though the decoder-only Language Model (LM) is simply a decoder, it is still
difficult to be compared with an Encoder-Decoder (ED) structure because this
decoder handles both the source sequence and the target sequence together. To
facilitate the comparison between the ED and LM structure, we propose to
analyze a Regularized Encoder-Decoder (RED) framework as illustrated in Figure
2. It is a variant of the traditional ED framework while replicating the
behaviors of an LM. Compared with the traditional ED structure, the RED
framework mainly has the following different components: An unidirectional
cross attention attends to both the source matrix and the target matrix
simultaneously; a source auto-encoder recovers the input source; a parameter
sharing mechanism shares the parameters between the encoder and the decoder; a
layer-wise coordination component makes each decoder layer attending to the
corresponding encoder layer output; a consecutive positional encoding utilizes
a positional encoding starting from the length of the source tokens in the
decoder.

Unidirectional Cross Attention. The main difference between the ED framework
and the LM is how the input source information is merged into the decoder. As
illustrated in Figure 2, the ED framework first uses multiple Transformer
blocks to extract features $H_{-1}^{E}$ from the source sequence $s$.
Afterwards, it utilizes a self attention $\mathtt{ATT}_{l}^{D}$ to get the
feature matrix $G_{l}^{D}$. It then uses an encoder attention
$\mathtt{ATT}_{l}^{J}$ to take $G_{l}^{D}$ as query and uses the encoder’s
final output $H_{-1}^{E}$ as the key and value to calculate $Q_{l}^{D}$. On
the other hand, an LM uses an unidirectional attention to handle the
concatenated features. To simulate this mechanism in the LM, as illustrated in
Figure 2, the RED framework uses unidirectional cross attention
$\mathtt{ATT}_{l}$ which attends to both the source matrix $G_{l}^{E}$ and the
target matrix $G_{l}^{D}$ simultaneously. Since it attends to all features
with one attention, the output matrix $Q_{l}^{D}$ of the attention layer
becomes less sensitive to the input source matrix $G_{l}^{E}$ especially when
it has already generated many words and $G_{l}^{D}$ becomes relatively long.
We call this the attention degeneration problem and we will analyze it in
detail in Section 3.3.

Source Auto-Encoder. The traditional ED structure only predicts the
probability for the target sequence and just takes the source sequence as
features. On the contrary, an LM predicts the probability for the whole
concatenated sequence including the source sequence. Therefore, in the RED
framework, we adopt a Source Auto-Encoder (SAE) component to realize this
mechanism. As shown in Figure 2, the overall loss is composed of the decoder
loss $\mathcal{L}^{D}$ and the SAE loss $\mathcal{L}^{E}$. $\mathcal{L}^{D}$
is the same as that in the ED framework while $\mathcal{L}^{E}$ is actually a
regularizer that recovers the source sequence $s$ to itself. Therefore, it can
alleviate the overfitting problem in training and thus improve the
performance.

Parameter Sharing. In the traditional ED framework, the encoder and the
decoder have their own parameters. On the contrary, in an LM, the source and
target sequences are concatenated and passed through the same network with the
same parameters. To simulate this property, the RED framework shares the
parameters between the encoder and the decoder. Parameter sharing techniques
Dehghani et al. (2019); Xia et al. (2019); Lan et al. (2020); Conneau and
Lample (2019) can be recognized as another regularizer that prevents the model
from having too many parameters and thus alleviates the overfitting problem.

Layer-Wise Coordination. The traditional ED structure feeds the source
sequence into the encoder and get the final source hidden representation
matrix $H_{-1}^{E}$. Then, each decoder layer takes $H_{-1}^{E}$ as the input
matrix for the encoder attention layer $\mathtt{ATT}_{l}^{J}$. On the
contrary, if the source and target sequence are concatenated and feed into the
LM, the attention component in each layer will take the current layer’s hidden
representation instead of using the same representation matrix. To imitate
this feature, the RED framework adapts the layer-wise coordination component
which uses attention $\mathtt{ATT}_{l}$ in the decoder to attend to each
corresponding encoder layer feature $G_{l}^{E}$. This method enables the
decoder to access multiple levels of source information and thus improves the
performance Belinkov et al. (2017); He et al. (2018); Peters et al. (2018).

Consecutive Positional Encoding. In the traditional ED structure, the source
sequence $s$ and the target sequence $t$ have their own positional encoding
$p_{s}$ and $p_{t}$ starting both from 1 to the length of the source and
target sequences. However, if the source and target sequences are concatenated
and sent into the LM, the positional encoding for the target sequence will not
start from 1. To realize this mechanism, the RED framework uses a Consecutive
Positional Encoding (CPE) which encodes the position $p_{t}$ for the target
sequence $t$ starting from $|s|+1$ instead of restarting from 1. It should be
noted that this mechanism has been proved in many research works He et al.
(2018); Conneau and Lample (2019) to be less effective compared with its
counterpart called Separate Positional Encoding (SPE) which restarts the
positional embedding for the target sentence. This is because restarting the
positional embedding makes the model more aware of different languages.
Moreover, we observe that the LM always has strong attention on the first
element of the target sequence. This token may also be used by the model to
differentiate the input source and output target. Using a fixed starting
position for the target sequence makes it easier for the model to find it.

###  3.3 Attention Degeneration Problem

Since the unidirectional cross attention imposes attention on both the source
sequence and the target sequence simultaneously, less and less attention will
be focused on the source sequence as the target sequence length grows. This is
the attention degeneration problem. To quantitatively understand why the ED
structure does not have this problem and how the influence of the source
sequence decreases in LM, we propose a theoretical analysis on the sensitivity
of $\mathtt{ATT}_{l}$ and $\mathtt{ATT}_{l}^{J}$ in the RED framework and the
ED framework where the source information is merged into the decoder. The
sensitivity measures the influence of the source vectors to the attention
layer’s output vectors. To simplify the analysis, we only analyze one head of
the attention. Since we focus on analyzing the effect of concatenating the
target sequence, we assume $\mathtt{ATT}_{l}$ and $\mathtt{ATT}_{l}^{J}$ both
use the unidirectional encoding and have the same parameters to make them
easier to be compared with each other.

We denote the input source matrix for the attention layer as
$X=[x_{1},x_{2},\cdots,x_{N}]^{\top}\in\mathbb{R}^{N\times d}$, where
$x_{i}\in\mathbb{R}^{d}$ and $d$ is the vector dimension, $N$ is the length of
the source sequence. We denote the target matrix as
$Y=[y_{1},y_{2},\cdots,y_{i}]^{\top}\in\mathbb{R}^{i\times d}$, where $i$ is
the current step. We denote the encoder attention as
$Z^{E}=\mathtt{ATT}(Y,X,X)$ where $X$, $Y$, $Z^{E}$, and $\mathtt{ATT}$
correspond to $H_{-1}^{E}$, $G_{l}^{D}$, $Q_{l}^{D}$, and
$\mathtt{ATT}_{l}^{J}$ in the ED structure respectively (Figure 2). On the
other hand, the unidirectional cross attention can be denoted as
$Z^{C}=\mathtt{ATT}(Y,[X^{\top},Y^{\top}]^{\top},[X^{\top},Y^{\top}]^{\top})$,
where $X$, $Y$, $Z$, and $\mathtt{ATT}$ correspond to $G_{l}^{E}$,
$G_{l}^{D}$, $Q_{l}^{D}$ and $\mathtt{ATT}_{l}$ in the RED structure
respectively (Figure 2).

We begin our analysis by defining a notion of sensitivity. Intuitively, given
a function $y=f(x)$, the sensitivity of $x$ on $y$ can be described as how the
output vector $y$ changes ($\Delta y$) when imposing a pertubation $\Delta x$
on the input vector $x$. However, imposing different $\Delta x$ leads to
different $\Delta y$, we propose to study the upper bound on the ratio between
the magnitude of $x$ and $y$ based on the following proposition:

######  Proposition 3.1.

Given a function $y=f(x)$ with a Jacobian matrix $J_{f}$, if we have a
pertubation vector $\Delta x$ and $y+\Delta y=f(x+\Delta x)$, then

$\small\frac{\|\Delta y\|}{\|\Delta x\|}\leq\|J_{f}\|+o(1).$ (1)

The proof is provided in Appendix A.1. We can observe that the ratio is upper
bounded with the norm of the Jacobian matrix $J_{f}$ plus $o(1)$. Then, we
focus on $J_{f}$ and define the sensitivity of the attention component as the
norm of the Jacobian matrix of the $i$th row vector $z_{i}$ of the attention
output $Z$ with respect to the $j$th row vector $x_{j}$ in $X$.

######  Definition 3.2 (Sensitivity).

Let the vectors $x_{j}$ and $z_{i}$ be defined above. The sensitivity of
$z_{i}$ with respect to $x_{j}$ is defined as the norm of the Jacobian matrix:

$\small S_{ij}=\|J_{ij}\|=\|\frac{\partial z_{i}}{\partial x_{j}}\|.$ (2)

Intuitively, if the sensitivity is high, the output vector is closely related
to the input source vector. Otherwise, the output vector may not contain
enough information about the input source vector and the output is likely to
be generated by a simple LM. From the Section 3.1,
$Z=\mathtt{ATT}(Y,X,X)=\mathtt{Softmax}(YA^{\top}X^{\top}/\sqrt{d})XW_{V}=PXW_{V}$,
in which $P\in\mathbb{R}^{d_{Y}\times d_{X}}$ and $d_{X},d_{Y}$ are the first
dimensions of $X$ and $Y$ respectively. Following Kim et al. (2020), the
Softmax matrix $P$ is a stochastic matrix, namely, its entries are non-
negative and its rows sum to 1. For each element $p_{ij}$ in $P$,
$p_{ij}\in[0,1]$ and they have an equal chance of receiving attention.
Therefore, we have $\mathbb{E}(p_{ij})=\frac{1}{d_{X}}$. To compare the
sensitivity between the two kinds of attention, with the above assumption, we
have the following theorem:

######  Theorem 3.3.

For $Z^{E}=\mathtt{ATT}(Y,X,X)$, where $\|X\|,\|Y\|,\|A\|,\|W_{V}\|$ are
bounded, $\exists$ $C_{3}\geq 0,\delta\in(0,1)$, with probability at least
$1-\delta^{2}$,

$\small\|J_{ij}^{E}\|\leq C_{3}(\frac{1}{N}+\sqrt{\ln\frac{1}{\delta}}).$ (3)

For $Z^{C}=\mathtt{ATT}(Y,[X^{\top},Y^{\top}],[X^{\top},Y^{\top}])$, with
probability at least $1-\delta^{2}$,

$\small\|J_{ij}^{C}\|\leq C_{3}(\frac{1}{N+i}+\sqrt{\ln\frac{1}{\delta}}).$
(4)

The proof is provided in Appendix A.2. It can be directly observed that the
upper bound of the Jacobian matrix norm is negatively correlated with the
value sequence length. Therefore, $\|J_{ij}^{C}\|$ has a smaller upper bound
than $\|J_{ij}^{E}\|$, which implies that the encoder attention
($\mathtt{ATT}_{l}^{J}$) output is more likely to be sensitive to the input
source vectors. As a result, if the input vector changes, the output of the
encoder attention changes more significantly than that of the cross
unidirectional attention. Moreover, the difference between the two upper
bounds becomes even larger as the generating step number $i$ grows. It shows
that the sensitivity to the source input decreases in the unidirectional cross
attention as the step number $i$ grows. We will conduct extensive experiments
to verify these observations.

##  4 Partial Attention Language Model

Figure 3: PALM framework.

To overcome the defects in the LM discussed above, we propose a Partial
Attention Language Model (PALM) which is shown in Figure 3. We keep the
effective components and remove the unfavorable components as analyzed in the
RED framework. First, we propose a novel partial attention component to
alleviate the attention degeneration problem. Besides, we designate a few
adjustments according to the aforementioned analysis of defects in the LM.
Specifically, we propose to use a separate positional embedding to replace the
consecutive positional embedding and use bidirectional attention for the
source sequence. Moreover, we add a language encoding layer and keep other
components unchanged.

Partial Attention Component. To alleviate the attention degeneration problem,
we propose the Partial Attention (PA) component. Similar to the ED framework,
it adds a new attention layer $\mathtt{ATT}_{l}^{P}$ that only focuses on the
source part of the feature matrix $Q_{l}$ which is denoted as
$Q_{l[1:|s|]}\in\mathbb{R}^{|s|\times d}$ and ${[1:|s|]}$ stands for taking
the first to the $|s|$th row vectors of $Q_{l}$. Since $Q_{l[1:|s|]}$ is not
affected as the target sequence grows, the attention degeneration problem can
thus be alleviated. As shown in Figure 3, $Q_{l[1:|s|]}$ is passed through a
consecutive feedforward layer which is denoted as
$P_{l}=\mathcal{F}_{P,l}(Q_{l[1:|s|]})$. Afterwards, we use another attention
layer $\mathtt{ATT}_{l}^{P}$ to attend to each vector of $P_{l}$ by each
vector in $G_{l}$ as $R_{l}=\mathtt{ATT}_{l}^{P}(Q_{l},P_{l},P_{l})$. Then,
$R_{l}$ is passed through a feedforward layer to get the hidden representation
matrix $H_{l}$ for the next layer. Since $\mathtt{ATT}_{l}^{P}$ takes $P_{l}$
as the key and value matrix which has a fixed length and does not contain any
target part, it can be concluded from Theorem 3.3 that the sensitivity upper
bound will not change as the generation step grows. Therefore, it can
alleviate the attention degeneration problem.

Bidirectional Attention. In $\mathtt{ATT}_{l}$, we propose to set a
bidirectional attention mask Dong et al. (2019); Raffel et al. (2020) for
$G_{l}[1:|s|]$ making each position in the source sequence aware of the whole
source sequence.

Separate Positional Encoding. As discussed in Section 3.2, we adopt the
separate positional encoding in PALM to help the model differentiate between
the source and target parts.

Language Embedding. We further adopt the language embedding Conneau and Lample
(2019); Conneau et al. (2020) to help the model differentiate between the
source and target sequences.

##  5 Experiments

###  5.1 Experimental Settings

We conduct experiments on several tasks to show the effectiveness of our
proposed method. All the experiments are conducted with a single NVIDIA(R)
TITAN RTX graphics card with Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz CPU.
All the datasets used in this paper are very commonly used in corresponding
tasks. The detailed license and how the datasets are made can be found in the
corresponding instructions. To the best of our knowledge, no personally
identifiable information or offensive content have been reported in these
datasets.

Machine Translation task Cho et al. (2014); Bahdanau et al. (2015); Luong et
al. (2015); Wu et al. (2016); Gehring et al. (2017); Johnson et al. (2017);
Koehn and Knowles (2017); Vaswani et al. (2018) is the most popular NLP task
and we adopt several language pairs in the commonly used IWSLT’14 dataset
Cettolo et al. (2014) (Creative Commons License) including De-En, En-De, It-
En, En-It, En-Fr, Es-En, En-Es, Ru-En, En-Ru, He-En, En-He, Ro-En, En-Ro. Both
the source and target sentences are encoded by the byte-pair encoding Sennrich
et al. (2016) method with 10,000 subword units in a shared dictionary. In the
ED model, we adopt the traditional Transformer model Vaswani et al. (2017);
Ott et al. (2019) and keep all hyperparameters setting the same as the
“transformer_iwslt_de_en” architecture in fairseq Ott et al. (2019). To make a
fair comparison between the models, we also keep all the transformer layer
number to be 6 in all models. We use the BLEU score Papineni et al. (2002) as
the evaluation metric.

Data-to-Text task can also be accomplished with Transformer. We investigate
the performance on three datasets, namely the WebNLG Perez-Beltrachini et al.
(2016); Gardent et al. (2017a, b); Castro Ferreira et al. (2018); Shimorina et
al. (2019) dataset (CC BY-NC-SA 4.0 License), E2E Novikova et al. (2017);
Dušek et al. (2019) dataset, and the WITA Fu et al. (2020) dataset. The WebNLG
is a human-annotated dataset aiming at generating text describing given
knowledge base triples. The E2E is made with the Crowd Flower platform to
generate restaurant comments based on their properties. The WITA dataset is an
automatically generated dataset that generates Wikipedia article sentences
based on input knowledge triples. We use the unannotated target sentences as
the reference. We keep all the model hyperparameters the same as that in the
machine translation task and evaluate the overall metrics with several
evaluation metrics including BLEU Papineni et al. (2002), ROUGEL Lin (2004),
METEOR Banerjee and Lavie (2005), NIST Doddington (2002), and CIDEr Vedantam
et al. (2015).

Summarization task Rush et al. (2015); Chopra et al. (2016); Nallapati et al.
(2016); Cheng and Lapata (2016); See et al. (2017); Paulus et al. (2018);
Pilault et al. (2020) can also be implemented with a Transformer architecture.
We conduct the experiments on XSUM Narayan et al. (2018) dataset (MIT
License). We take the articles as the source sequence while using the
summarization as the target sequence. We keep all the model hyperparameters
the same as that in the machine translation task and report the ROUGE1,
ROUGE2, ROUGEL metrics.

###  5.2 Comparison Models

We compare the PALM structure against the following models.

ED is the traditional Encoder-Decoder framework built on Transformer Vaswani
et al. (2017) implemented with fairseq Ott et al. (2019).

LM is the traditional Language Model built with a Transformer decoder. It
concatenates the source sequence and the target sequence to train an LM. When
testing, it predicts the target sequence after inputting the source sequence
into the LM.

LM-SPE adopts the Separate Positional Encoding Conneau and Lample (2019); He
et al. (2018) in an LM as the positional encoding. The first position for the
target sequence is always set to 1.

LM-LE uses a Language Embedding Conneau and Lample (2019); Conneau et al.
(2020) to help LM differentiate source and target sequence.

LM-PA adopts our proposed Partial Attention component in an LM. Different from
PALM, it uses unidirectional attention and does not use the LE and SPE
components.

PreLM uses a Prefix Language Model Liu et al. (2018); Dong et al. (2019);
Raffel et al. (2020) to generate target sequences. Different from LM, it uses
a fully-visible masking of the input source instead of using a unidirectional
causal mask. The mask enables each source word depending on the whole input
sequence instead of the previous one.

###  5.3 Experimental Results

| De-En | En-De | It-En | En-It | En-Fr | Es-En | En-Es | Ru-En | En-Ru | He-En | En-He | Ro-En | En-Ro | Avg. | #Paras
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ED | 34.18 | 28.00 | 31.96 | 29.42 | 40.86 | 40.99 | 37.53 | 23.09 | 18.20 | 38.09 | 25.41 | 38.14 | 28.30 | 31.86 | 47.1M
LM | 33.19 | 26.43 | 30.92 | 28.64 | 39.16 | 39.33 | 36.67 | 22.25 | 17.53 | 34.81 | 24.35 | 35.51 | 27.02 | 30.45 | 29.3M
LM-SPE | 33.35 | 27.35 | 31.36 | 28.87 | 39.93 | 39.69 | 36.99 | 21.89 | 17.98 | 34.89 | 24.80 | 35.19 | 27.72 | 30.77 | 29.3M
LM-LE | 33.58 | 27.46 | 31.38 | 29.03 | 40.14 | 39.87 | 37.05 | 22.24 | 18.08 | 34.80 | 24.31 | 35.76 | 27.96 | 30.90 | 29.3M
LM-PA | 34.54 | 28.35 | 32.01 | 29.70 | 40.15 | 40.58 | 37.24 | 22.98 | 18.45 | 35.64 | 25.37 | 37.35 | 28.02 | 31.57 | 35.6M
PreLM | 33.90 | 27.88 | 31.77 | 29.35 | 40.53 | 40.41 | 37.46 | 22.80 | 18.11 | 35.96 | 24.78 | 37.55 | 28.11 | 31.43 | 29.3M
PALM | 34.73 | 28.59 | 32.22 | 29.83 | 40.31 | 40.91 | 37.61 | 23.27 | 18.74 | 37.00 | 25.29 | 37.59 | 28.75 | 31.91 | 35.6M
PALM w/o SAE | 34.03 | 28.30 | 31.83 | 29.42 | 39.87 | 40.65 | 37.25 | 22.21 | 17.92 | 37.06 | 24.89 | 37.09 | 28.40 | 31.46 | 35.6M

Table 1: BLEU scores for models on IWSLT’14 dataset. Avg. is the average BLEU
score for all language pairs while #Paras is the count of model parameters.

Figure 4: Numerical sensitivity analysis.

Figure 5: Stepwise hallucination analysis. Lower is better.

| Cs-En | En-Cs | Ro-En | En-Ro | Avg.
---|---|---|---|---|---
ED | 21.9 | 15.7 | 29.3 | 20.4 | 21.8
LM | 16.2 | 11.0 | 25.1 | 15.9 | 17.1
PALM | 21.1 | 15.1 | 28.3 | 19.3 | 21.0
PALM w/o SAE | 18.6 | 12.9 | 26.5 | 18.0 | 19.0
Table 2: BLEU score for WMT’16 dataset. | LM | LM-PA | PALM | ED | Gold | $\Delta L$
---|---|---|---|---|---|---
De-En | 21.00 | 21.37 | 21.30 | 21.08 | 22.25 | 0.30
En-De | 22.02 | 22.24 | 22.11 | 22.30 | 22.98 | 0.08
It-En | 22.35 | 22.40 | 22.37 | 22.84 | 23.88 | 0.02
En-It | 22.82 | 22.90 | 22.91 | 22.65 | 23.26 | 0.09
En-Fr | 23.70 | 24.29 | 24.23 | 24.00 | 24.01 | 0.53
Es-En | 21.16 | 21.46 | 21.35 | 21.88 | 22.48 | 0.19
En-Es | 21.66 | 21.97 | 21.96 | 22.06 | 22.03 | 0.30
Ru-En | 20.69 | 20.99 | 20.88 | 20.88 | 23.17 | 0.19
En-Ru | 22.21 | 22.71 | 22.72 | 22.87 | 22.96 | 0.50
He-En | 19.71 | 20.08 | 20.13 | 20.33 | 21.30 | 0.41
En-He | 19.61 | 20.16 | 20.03 | 20.31 | 20.66 | 0.42
Ro-En | 19.54 | 20.13 | 19.88 | 20.01 | 21.34 | 0.33
En-Ro | 22.38 | 22.72 | 22.61 | 22.82 | 23.03 | 0.24
Avg. | 21.45 | 21.80 | 21.73 | 21.85 | 22.57 | 0.28
Table 3: Early stop effect. $\Delta L$ is calculated by length of PALM minus length of LM.  | METEOR |  ROUGEL | CIDEr | NIST | BLEU
---|---|---|---|---|---
ED | 0.441 | 0.725 | 4.08 | 11.0 | 0.616
LM | 0.452 | 0.732 | 4.24 | 11.2 | 0.629
LM w/o SAE | 0.438 | 0.716 | 4.10 | 10.9 | 0.607
PALM | 0.457 | 0.736 | 4.27 | 11.3 | 0.632
Table 4: Results on WebNLG dataset. | METEOR |  ROUGEL | CIDEr | NIST | BLEU
---|---|---|---|---|---
ED | 0.396 | 0.630 | 1.62 | 8.07 | 0.597
LM | 0.423 | 0.661 | 2.00 | 7.93 | 0.604
LM w/o SAE | 0.405 | 0.646 | 1.92 | 7.93 | 0.602
PALM | 0.449 | 0.688 | 2.25 | 8.46 | 0.657
Table 5: Results on E2E dataset.

Main Results. The experimental results for IWSLT’14 are shown in Table 1. We
can draw the following conclusions. (1) Compared with the ED framework, the LM
has worse performance due to the defects we discussed in Section 3.3. (2)
Compared with the LM model, our proposed PALM structure achieved better
performance which indicates that the attention degeneration is alleviated. (3)
The PreLM improves the performance since the source text hidden representation
depends on the whole source sequence instead of the previous context. (4) The
LE component helps improve the performance as it provides extra language
information for the model. (5) the LM-SPE model performs better than the
traditional LM model using CPE. This observation verifies our analysis in
Section 3.2. (6) From the ablation study, the performance decreases without
the SAE component. It shows that the SAE component can alleviate the
overfitting problem and thus improves the performance. (7) Compared with the
ED model, LM, and our proposed PALM reduce the parameter number significantly.

| METEOR |  ROUGEL | CIDEr | NIST | BLEU
---|---|---|---|---|---
ED | 0.347 | 0.636 | 3.64 | 9.08 | 0.427
LM | 0.352 | 0.648 | 3.77 | 7.83 | 0.382
PALM | 0.364 | 0.662 | 4.03 | 8.9 | 0.437
Table 6: Results on WITA dataset.

|  ROUGE1 |  ROUGE2 |  ROUGEL
---|---|---|---
ED | 0.284 | 0.0877 | 0.245
LM | 0.279 | 0.0711 | 0.251
PALM | 0.317 | 0.0953 | 0.282

Table 7: Results on XSUM dataset.

Numerical Sensitivity Analysis. In Theorem 3.3, we prove that the upper bound
of the sensitivity in the unidirectional cross attention decreases as the
current step number increases. To experimentally show that the output vector
becomes less and less sensitive to the source sequence as the step number
grows, we conduct a numerical analysis by directly imposing a random small
perturbation vector $\Delta x_{j}$ on the source vector $x_{j}$ of the
attention layer. We take the norm of the output vector deviation $\|\Delta
z_{i}\|$ at the $i$th step divided by the input vector change norm $\|\Delta
x_{j}\|$. This quantity is upper bounded by the sensitivity as shown in
Proposition 3.1. The results are shown in Figure 5. It can be concluded from
the results that the sensitivity of the LM decreases as the step increases
which leads to the model ignoring the input sequence gradually. It can also
justify the conclusion drawn from Theorem 3.3 that the ED model does not have
this problem while our proposed PALM can also alleviate this problem.

Stepwise Hallucination Analysis. As implied by Theorem 3.3 and the above
experiments, the sensitivity decreases as the step number increases in LM. But
how does the sensitivity affect the generating performance? We conduct the
stepwise hallucination analysis to answer this question. Generation models
prone to generate unrelated content beyond the input data and this phenomenon
has been recognized as the hallucination problem Tian et al. (2019); Nie et
al. (2019); Liu et al. (2021); Rebuffel et al. (2021) in many recent research
works. Intuitively, the attention degeneration problem is likely to exacerbate
the hallucination problem when the generation step grows. This is because less
and less attention is focused on the input data and the models are more likely
to generate text groundlessly. As shown in Figure 5, we conduct an experiment
to analyze the hallucination problem at different generation steps. Similar to
Rebuffel et al. (2021), we use the Hallucination Ratio (HR) to measure the
hallucination problem. HR measures the proportion of the hallucination
content. The HR for the $i$th position is defined as
$H_{i}=1-\sum_{j}^{T}\max(\mathbbm{1}(t_{ji}\in
r_{j}),C[s_{j1},t_{ji}],\cdots,C[s_{j|s_{j}|},t_{ji}])/\sum_{j}^{T}\mathbbm{1}(|t_{j}|\geq
i),$ where $T$ is the target sentence count. $s_{ji}$ and $t_{ji}$ are the
$i$th word in the $j$th source and generated sentence respectively. $|t_{j}|$
is the length of the $j$th generated sentence. $r_{j}$ is the $j$th gold
standard sentence. $C[p,q]=\mathtt{Sigmoid}((\sum_{j}^{T}\mathbbm{1}(p\in
s_{j})\cdot\mathbbm{1}(q\in t_{j})-\alpha)/\beta)$ is the normalized alignment
score of the $p$th and $q$th words in the dictionary, where $\alpha$ and
$\beta$ are sigmoid shape parameters. High HR indicates the hallucination
problem is prone to occur. It can be observed that (1) HR changes moderately
between 5% and 7% as $i$ grows in all models. (2) LM has the highest (worst)
HR score especially when $i$ grows larger. When $i>50$, LM has a significant
high HR score than other models which implies that the hallucination problem
gets worse. (3) Simply adding the PA component on LM can alleviate the
hallucination problem and lower the HR score. It indicates that the attention
degeneration problem exacerbates the hallucination problem and it can be
alleviated with our proposed PA component. (4) Our proposed PALM achieves a
low HR score close to the ED model. It performs well even $i$ is large.

Early Stop Effect. To further study how the attention degeneration problem
affects the performance, we conduct a statistical analysis on the output
length of the generated sequence. The results are shown in Table 3. It can be
concluded from the results that (1) The LM usually generates shorter sentences
compared with the ED model and PALM structure while simply adding the PA
component in LM alleviates the early stop phenomenon. It shows that the
attention degeneration problem causes the early stop phenomenon and it can be
alleviated by our proposed PA component. Due to the attention degeneration
problem, the sensitivity on the input source vectors decreases as the step
number grows. The model becomes less and less sensitive to the source sequence
and it tends to stop generating because it gets less and less information
about what to say. (2) On the other hand, our proposed PALM structure
alleviates this problem. The average length of the PALM structure output is
longer than the LM model and close to the ED model showing that it alleviates
the attention degeneration problem and thus overcomes the early stop effect.

Data-to-Text Generation. To show that our analysis and the proposed method are
also applicable in other seq2seq tasks, we conduct experiments on the data-to-
text generation tasks. The results on WebNLG, E2E, and WITA datasets are shown
in Table 4, 5, and 5.3 respectively. It can be observed that (1) our proposed
PALM outperforms the LM model which shows the effectiveness of the partial
attention component. (2) The LM model outperforms the ED model with the help
of SAE component. Different from the translation task, the input sequence of
this task shares the same language as the target sequence. Therefore, LM’s SAE
component trains the model to recover source sequences and can thus alleviate
the overfitting problem. This can be directly verfied with the results of the
LM w/o SAE model. When the SAE component is removed, the performance is worse
than the LM model in all three datasets.

WMT’16 Results. We conduct our experiments on WMT’16 Bojar et al. (2016)
dataset which is larger than IWSLT’14. The performance is shown in Table 2. It
can be concluded from the results that (1) our proposed PALM also outperforms
LM’s results and is close the ED’s performance which shows PALM’s
effectiveness in alleviating the attention degeneration problem; (2)
interestingly, removing the SAE component improves the performance and closes
the gap to the performance of ED. This may be because the dataset is larger
and the overfitting problem is not serious.

Summarization. We conduct experiments on the summarization dataset XSUM to
further verify our analysis and the results are shown in Table 7. It can be
concluded from the results that (1) the ED model outperforms the LM model in
ROUGE1 and ROUGE2. This is because the input sequence is much longer than the
target sequence, the attention degeneration problem becomes the dominant
problem. This result is consistent with the analysis we discussed in Section
3.3; (2) our PALM structure still outperforms the LM model which is also
consistent with the previous analysis;

Stepwise Precision Analysis. As implied by Theorem 3.3 and the above
experiments, the sensitivity decreases as the step number increases in LM. We
further conduct the stepwise precision analysis to show that the sensitivity
directly affect the generating performance. We calculate the average word
precision for the $i$th position as $A_{i}=\sum_{j}^{S}\mathbbm{1}(s_{ji}\in
r_{j})/\sum_{j}^{S}\mathbbm{1}(|s_{j}|\geq i),$ where $s_{j}$ in the $j$th
generated sentence and $s_{ji}$ is the $i$th word in sentence $s_{j}$. $r_{j}$
is the $j$th reference sentence and there are $S$ sentences in total. The
results are shown in Figure 6. We have three observations. (1) $A_{i}$ changes
moderately between 0.64 and 0.67 as $i$ grows which is determined by the word
distribution. (2) $A_{i}$ for the LM model is slightly better than the ED
framework when the step number is small. This is because when the step number
is small, the attention degeneration problem is not the dominant factor for
the performance and the LE, SAE, and other components play as the key role to
make the performance better. (3) Contrastingly, $A_{i}$ for the LM model
becomes worse as the step size gets larger when compared with the ED model.
This observation shows that as the step number increases, the sensitivity
decreases, and thus the model is likely to generate wrong words. (4) Our
proposed PALM structure alleviates the attention degeneration problem and the
precision distribution is similar to the ED model.

Figure 6: Stepwise precision analysis. $x$-axis is the generation step number
and $y$-axis is the average word precision.

##  6 Conclusions

We propose a detailed comparison between the Encoder-Decoder (ED) framework
and the Language Model (LM) in the sequence-to-sequence (seq2seq) generation
task. We propose to analyze a Regularized Encoder-Decoder (RED) framework that
is equivalent to an LM but has an encoder and a decoder. We analyze the
attention degeneration problem and conduct a theoretical analysis. Based on
the analysis, we propose a novel partial attention language model. The
experimental results validate our analysis and demonstrate the effectiveness
of our proposed model.

##  7 Limitations

It should be noted that in Theorem 3.3, we utilize the PAC theory framework
which holds with high probability. Unfortunately, it cannot guarantee all the
cases satisfy the bound which means in some extreme cases, the bound will not
be valid. This is a common short back for the PAC analysis and we use
extensive experimental results to empirically show the effectiveness of the
analysis.

##  8 Broader Impact Statement

This paper focuses on comparing different architectures of the existing
language model which does not explicitly involve any enthetic concerns. We do
not make any new dataset and all used datasets are properly cited. This work
does not cause any kind of safety or security concerns while it also does not
raise any human rights concerns or environmental concerns. Therefore, there
will be no negative societal impact on our work.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Belinkov et al. (2017) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017\.  What do neural machine translation models learn about morphology?  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 861–872, Vancouver, Canada. Association for Computational Linguistics.
  * Bengio et al. (2000) Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000.  A neural probabilistic language model.  In _Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA_ , pages 932–938. MIT Press.
  * Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016.  Findings of the 2016 conference on machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 131–198, Berlin, Germany. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Castro Ferreira et al. (2018) Thiago Castro Ferreira, Diego Moussallem, Emiel Krahmer, and Sander Wubben. 2018\.  Enriching the WebNLG corpus.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 171–176, Tilburg University, The Netherlands. Association for Computational Linguistics.
  * Cettolo et al. (2014) Mauro Cettolo, J Niehues, S Stüker, Luisa Bentivogli, and Marcello Federico. 2014.  Report on the 11th iwslt evaluation campaign, iwslt 2014.  In _IWSLT-International Workshop on Spoken Language Processing_ , pages 2–17. Marcello Federico, Sebastian Stüker, François Yvon.
  * Cheng and Lapata (2016) Jianpeng Cheng and Mirella Lapata. 2016.  Neural summarization by extracting sentences and words.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 484–494, Berlin, Germany. Association for Computational Linguistics.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 93–98, San Diego, California. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 7057–7067.
  * Dehghani et al. (2019) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019.  Universal transformers.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Deng et al. (2023) Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. 2023.  Recent advances towards safe, responsible, and moral dialogue systems: A survey.  _arXiv preprint arXiv:2302.09270_.
  * Doddington (2002) George Doddington. 2002.  Automatic evaluation of machine translation quality using n-gram co-occurrence statistics.  In _Proceedings of the Second International Conference on Human Language Technology Research_ , pages 138–145.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13042–13054.
  * Dong et al. (2021) Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021.  Attention is not all you need: Pure attention loses rank doubly exponentially with depth.  In _International Conference on Machine Learning_ , pages 2793–2803. PMLR.
  * Dušek et al. (2019) Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019.  Semantic noise matters for neural natural language generation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 421–426, Tokyo, Japan. Association for Computational Linguistics.
  * Fu et al. (2021) Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021.  A theoretical analysis of the repetition problem in text generation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 12848–12856.
  * Fu et al. (2020) Zihao Fu, Bei Shi, Wai Lam, Lidong Bing, and Zhiyuan Liu. 2020.  Partially-aligned data-to-text generation with distant supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9183–9193, Online. Association for Computational Linguistics.
  * Gardent et al. (2017a) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a.  Creating training corpora for NLG micro-planners.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 179–188, Vancouver, Canada. Association for Computational Linguistics.
  * Gardent et al. (2017b) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017b.  The WebNLG challenge: Generating text from RDF data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017\.  Convolutional sequence to sequence learning.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1243–1252. PMLR.
  * Ghosh et al. (2017) Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017.  Affect-LM: A neural language model for customizable affective text generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 634–642, Vancouver, Canada. Association for Computational Linguistics.
  * He et al. (2018) Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018\.  Layer-wise coordination between encoder and decoder for neural machine translation.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 7955–7965.
  * Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _Transactions of the Association for Computational Linguistics_ , 5:339–351.
  * Kim et al. (2020) Hyunjik Kim, George Papamakarios, and Andriy Mnih. 2020.  The lipschitz constant of self-attention.  _arXiv preprint arXiv:2006.04710_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.  Generating wikipedia by summarizing long sequences.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Liu et al. (2021) Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021.  A token-level reference-free hallucination detection benchmark for free-form text generation.  _arXiv preprint arXiv:2104.08704_.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  * Mikolov et al. (2010) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010.  Recurrent neural network based language model.  In _Eleventh annual conference of the international speech communication association_.
  * Mikolov et al. (2011) Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2011.  Extensions of recurrent neural network language model.  In _2011 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5528–5531. IEEE.
  * Mikolov and Zweig (2012) Tomas Mikolov and Geoffrey Zweig. 2012.  Context dependent recurrent neural network language model.  In _2012 IEEE Spoken Language Technology Workshop (SLT)_ , pages 234–239.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Nie et al. (2019) Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin-Yew Lin. 2019.  A simple recipe towards reducing hallucination in neural surface realisation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2673–2679, Florence, Italy. Association for Computational Linguistics.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Perez-Beltrachini et al. (2016) Laura Perez-Beltrachini, Rania Sayed, and Claire Gardent. 2016.  Building RDF content for data-to-text generation.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 1493–1502, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Pilault et al. (2020) Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Christopher Pal. 2020.  On extractive and abstractive neural document summarization with transformer language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9308–9319.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rebuffel et al. (2021) Clément Rebuffel, Marco Roberti, Laure Soulier, Geoffrey Scoutheeten, Rossella Cancelliere, and Patrick Gallinari. 2021.  Controlling hallucinations at word level in data-to-text generation.  _arXiv preprint arXiv:2102.02810_.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Controlling politeness in neural machine translation via side constraints.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 35–40, San Diego, California. Association for Computational Linguistics.
  * Shimorina et al. (2019) Anastasia Shimorina, Elena Khasanova, and Claire Gardent. 2019.  Creating a corpus for Russian data-to-text generation using neural machine translation and post-editing.  In _Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing_ , pages 44–49, Florence, Italy. Association for Computational Linguistics.
  * Sutskever et al. (2011) Ilya Sutskever, James Martens, and Geoffrey E. Hinton. 2011.  Generating text with recurrent neural networks.  In _Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011_ , pages 1017–1024. Omnipress.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 3104–3112.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022\.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_.
  * Tian et al. (2019) Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019.  Sticking to the facts: Confident decoding for faithful data-to-text generation.  _arXiv preprint arXiv:1910.08684_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2018) Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018.  Tensor2Tensor for neural machine translation.  In _Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 193–199, Boston, MA. Association for Machine Translation in the Americas.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , pages 4566–4575. IEEE Computer Society.
  * Vig and Belinkov (2019) Jesse Vig and Yonatan Belinkov. 2019.  Analyzing the structure of attention in a transformer language model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 63–76, Florence, Italy. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Xia et al. (2019) Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. 2019.  Tied transformers: Neural machine translation with shared encoder and decoder.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 5466–5473.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhu et al. (2020) Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020.  Incorporating BERT into neural machine translation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

Appendix. Supplementary Material

##  Appendix A.1 Proof of Proposition 3.1

######  Proposition A.1.1.

3.1 Given a function $y=f(x)$ with a Jacobian matrix $J_{f}$, if we have a
pertubation vector $\Delta x$ and $y+\Delta y=f(x+\Delta x)$, then

$\small\frac{\|\Delta y\|}{\|\Delta x\|}\leq\|J_{f}\|+o(1).$ (5)

###### Proof.

Let $y=f(x)$ be given, in which $x,y\in\mathbb{R}^{d}$. We denote the output
of $f$ when $x$ is perturbed by a vector $\Delta x$ as $f(x+\Delta x)$. Since
for the Jacobian matrix, we have $f(x+\Delta x)-f(x)=J_{f}\Delta x+o(\|\Delta
x\|)$, it follows that:

$\displaystyle f(x+\Delta x)-f(x)$ $\displaystyle=J_{f}\Delta x+o(\|\Delta
x\|)$ $\displaystyle\|f(x+\Delta x)-f(x)\|$ $\displaystyle=\|J_{f}\Delta
x+o(\|\Delta x\|)\|$ $\displaystyle\|f(x+\Delta x)-f(x)\|$
$\displaystyle\leq\|J_{f}\Delta x\|+\|o(\|\Delta x\|)\|$
$\displaystyle\|f(x+\Delta x)-f(x)\|$ $\displaystyle\leq\|J_{f}\|\|\Delta
x\|+\|o(\|\Delta x\|)\|$ $\displaystyle\|\Delta y\|$
$\displaystyle\leq\|J_{f}\|\|\Delta x\|+\|o(\|\Delta x\|)\|$
$\displaystyle\frac{\|\Delta y\|}{\|\Delta x\|}$
$\displaystyle\leq\|J_{f}\|+\frac{o(\|\Delta x\|)}{\|\Delta x\|}$
$\displaystyle\frac{\|\Delta y\|}{\|\Delta x\|}$
$\displaystyle\leq\|J_{f}\|+o(1)$

∎

##  Appendix A.2 Proof of Theorem 3.3

######  Lemma A.2.1.

Let $Z=\mathtt{Softmax}(YA^{\top}X^{\top})XW$, $z_{i}^{\top}$ be the $i$th row
of $Z$ and $x_{j}^{\top}$ be the $j$th row of $X$. Then,

$\displaystyle J_{ij}=$ $\displaystyle\frac{\partial z_{i}}{\partial x_{j}}$
$\displaystyle=$ $\displaystyle
W^{\top}(X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})\cdot(e_{ji}YA^{\top})+Ip_{ij})$

in which $P=\mathtt{Softmax}(YA^{\top}X^{\top})$, $p_{i}^{\top}$ is the $i$th
row of $P$, $e_{ji}$ is a binary matrix with zeros exerywhere except the
$(i,j)$th entry, $N$ is the row number of $X$.

###### Proof.

$z_{i}=(PXW)_{i}^{\top}=\sum_{k=1}^{N}W^{\top}x_{k}p_{ik}$

$\displaystyle J_{ij}$
$\displaystyle=\frac{\partial\sum_{k=1}^{N}W^{\top}x_{k}p_{ik}}{\partial
x_{j}}$ $\displaystyle=\sum_{k=1}^{N}W^{\top}x_{k}\frac{\partial
p_{ik}}{\partial x_{j}}+\sum_{k=1}^{N}\frac{\partial W^{\top}x_{k}}{\partial
x_{j}}p_{ik}$
$\displaystyle=W^{\top}[x_{1},x_{2},\cdots,x_{N}]\begin{bmatrix}\frac{\partial
p_{i1}}{\partial z_{j}}\\\ \frac{\partial p_{i2}}{\partial z_{j}}\\\ \vdots\\\
\frac{\partial p_{iN}}{\partial z_{j}}\end{bmatrix}+\frac{\partial
W^{\top}x_{j}}{\partial x_{j}}p_{ij}$
$\displaystyle=W^{\top}X^{\top}\frac{\partial p_{i}}{\partial
x_{j}}+W^{\top}p_{ij}$
$\displaystyle=W^{\top}(X^{\top}\frac{\partial\mathtt{Softmax}((YA^{\top}X^{\top})_{i}^{\top})}{\partial
x_{j}}+Ip_{ij})$
$\displaystyle=W^{\top}(X^{\top}\mathtt{Softmax}^{\prime}((YA^{\top}X^{\top})_{i}^{\top}))\frac{\partial(XAy_{i})}{\partial
x_{j}}+Ip_{ij})$
$\displaystyle=W^{\top}(X^{\top}\mathtt{Softmax}^{\prime}((YA^{\top}X^{\top})_{i}^{\top}))\cdot$
$\displaystyle\quad\frac{\partial([x_{1},x_{2},\cdots,x_{N}]^{\top}Ay_{i})}{\partial
x_{j}}+Ip_{ij})$
$\displaystyle=W^{\top}(X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})\cdot$
$\displaystyle\quad([\frac{\partial x_{1}^{\top}Ay_{i}}{\partial
x_{j}},\frac{\partial x_{2}^{\top}Ay_{i}}{\partial
x_{j}},\cdots,\frac{\partial x_{N}^{\top}Ay_{i}}{\partial
x_{j}}]^{\top})+Ip_{ij})$
$\displaystyle=W^{\top}(X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})\cdot$
$\displaystyle\quad([0,\cdots,\underbrace{Ay_{i}}_{j\text{th
}col},\cdots,0]^{\top})+Ip_{ij})$
$\displaystyle=W^{\top}(X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})\cdot(e_{ji}YA^{\top})+Ip_{ij})$

∎

######  Lemma A.2.2.

Let $Z=\mathtt{Softmax}(QA^{\top}Q^{\top})QW$, $z_{i}^{\top}$ be the $i$th row
of $Z$ and $q_{j}^{\top}$ be the $j$th row of $Q$.

$\displaystyle J_{ij}=$ $\displaystyle\frac{\partial z_{i}}{\partial q_{j}}$
$\displaystyle=$ $\displaystyle
W^{\top}(Q^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})\cdot(e_{ji}QA^{\top}+QA\delta_{ij})+Ip_{ij}),$

in which $P=\mathtt{Softmax}(QA^{\top}Q^{\top})$, $\delta_{ij}$ is a scalar
equals to 1 if $i=j$ and equals to 0 otherwise.

The detailed proof can be found in Kim et al. (2020). To prove Theorem 3.3, we
follow the obsevation of Kim et al. (2020) that the Softmax matrix $P$ is a
stochastic matrix, namely, its entries are non-negative and its rows sum to 1.
For each element $p_{ij}$ in $P$, $p_{ij}\in[0,1]$ and they have an equal
chance of receiving attention. Therefore, we have
$\mathbb{E}(p_{ij})=\frac{1}{d_{X}}$.

######  Theorem A.2.3.

3.3 For $Z^{E}=\mathtt{ATT}(Y,X,X)$, where $\|X\|,\|Y\|,\|A\|,\|W_{V}\|$ are
bounded, $\exists$ $C_{3}\geq 0,\delta\in(0,1)$, with probability at least
$1-\delta^{2}$,

$\|J_{ij}^{E}\|\leq C_{3}(\frac{1}{N}+\sqrt{\ln\frac{1}{\delta}}).$

For $Z^{C}=\mathtt{ATT}(Y,[X^{\top},Y^{\top}],[X^{\top},Y^{\top}])$, with
probability at least $1-\delta^{2}$,

$\|J_{ij}^{C}\|\leq C_{3}(\frac{1}{N+i}+\sqrt{\ln\frac{1}{\delta}}).$

###### Proof.

We first prove the upper bound for $J_{ij}^{E}$. From Lemma A.2.1, we have:

$\displaystyle
J_{ij}^{E}=W^{\top}(X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})\cdot(e_{ji}YA^{\top})+Ip_{ij})$
$\displaystyle\|J_{ij}^{E}\|=\|W^{\top}(X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})e_{ji}YA^{\top}+Ip_{ij})\|$
$\displaystyle\leq\|W^{\top}\|\|X^{\top}(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})e_{ji}YA^{\top}\|+\|Ip_{ij}\|$
$\displaystyle\leq\|W^{\top}\|(\|X^{\top}\|\|(\operatorname{Diag}(p_{i})-p_{i}p_{i}^{\top})e_{ji}\|\cdot\|YA^{\top}\|+\|Ip_{ij}\|)$
$\displaystyle\leq(\|X^{\top}\|\|\begin{bmatrix}&0&\cdots&-p_{i1}p_{ij}&\cdots&0\\\
&0&\cdots&-p_{i2}p_{ij}&\cdots&0\\\ &\vdots&\vdots&\vdots&\vdots&\vdots\\\
&0&\cdots&p_{ij}-p_{ij}p_{ij}&\cdots&0\\\
&\vdots&\vdots&\vdots&\vdots&\vdots\\\
&0&\cdots&-p_{iN}p_{ij}&\cdots&0\end{bmatrix}\|\cdot\|YA^{\top}\|+\|IP_{ij}\|)\|W^{\top}\|$
$\displaystyle=(\|X^{\top}\|\|p_{ij}\|\|[p_{i1},p_{i2},\cdots,1-p_{ij},\cdots,p_{iN}]^{\top}\|\cdot\|YA^{\top}\|+\|Ip_{ij}\|)\|W^{\top}\|$
$\displaystyle\leq C_{1}p_{ij}\sqrt{p_{i1}^{2}+p_{i2}^{2}+\cdots
1+p_{ij}^{2}-2p_{ij}+\cdots p_{iN}^{2}}+C_{2}p_{ij}$ $\displaystyle\leq
C_{1}p_{ij}\sqrt{1+\|p_{i}\|^{2}}+C_{2}p_{ij}$ $\displaystyle\leq
C_{1}p_{ij}\sqrt{1+\|p_{i}\|^{2}_{1}}+C_{2}p_{ij}$
$\displaystyle=C_{1}\sqrt{2}p_{ij}+C_{2}p_{ij}$ $\displaystyle=C_{3}p_{ij},$

where $C_{1}\geq 0,C_{2}\geq 1,C_{3}\geq 1$. Since we assume $p_{ij}\in[0,1]$
and $\mathbb{E}(p_{ij})=\frac{1}{N}$, Therefore, with Hoeffding inequality,
for $p_{ij}$ we have:

$\displaystyle\operatorname{Pr}(p_{ij}\geq\frac{1}{N}+t)$ $\displaystyle\leq
e^{-2t^{2}}$
$\displaystyle\operatorname{Pr}(p_{ij}\geq\frac{1}{N}+\sqrt{\ln\frac{1}{\delta}})$
$\displaystyle\leq\delta^{2}$
$\displaystyle\operatorname{Pr}(p_{ij}\leq\frac{1}{N}+\sqrt{\ln\frac{1}{\delta}})$
$\displaystyle\leq 1-\delta^{2}.$

Therefore, with probability at least $1-\delta^{2}$

$\|J_{ij}\|\leq C_{3}(\frac{1}{N}+\sqrt{\ln\frac{1}{\delta}})$

Next, we prove the upper bound for $J_{ij}^{C}$. We denote $Q$ as
$Q=\begin{bmatrix}X\\\ Y\end{bmatrix}$. Then, $J_{ij}^{C}$ is the Jacobian
matrix for $\partial q_{N+i}/\partial q_{j}$. From Lemma A.2.2, we have:

$\displaystyle J_{ij}^{C}$
$\displaystyle=W^{\top}(Q^{\top}(\operatorname{Diag}(p_{N+i})-p_{N+i}p_{N+i}^{\top})\cdot$
$\displaystyle\ \ \ \ \ \
(e_{j,{N+i}}QA^{\top}+QA\delta_{{N+i},j})+Ip_{{N+i},j})$
$\displaystyle\|J_{ij}^{C}\|$
$\displaystyle=\|W^{\top}(Q^{\top}(\operatorname{Diag}(p_{N+i})-p_{N+i}p_{N+i}^{\top})\cdot$
$\displaystyle\ \ \ \ \ \
(e_{j,{N+i}}QA^{\top}+QA\delta_{{N+i},j})+Ip_{{N+i},j})\|$
$\displaystyle=\|W^{\top}(Q^{\top}(\operatorname{Diag}(p_{N+i})-p_{N+i}p_{N+i}^{\top})\cdot$
$\displaystyle\ \ \ \ \ \ (e_{j,{N+i}}QA^{\top})+Ip_{{N+i},j})\|$
$\displaystyle\leq C_{3}p_{{N+i},j},$

where $C_{1}\geq 0,C_{3}\geq 1$. Since we assume $p_{{N+i},j}\in[0,1]$ and
$\mathbb{E}(p_{{N+i},j})=\frac{1}{N+i}$, Therefore, with Hoeffding inequality,
for $p_{{N+i},j}$ we have:

$\displaystyle\operatorname{Pr}(p_{{N+i},j}\geq\frac{1}{N+i}+t)$
$\displaystyle\leq e^{-2t^{2}}$
$\displaystyle\operatorname{Pr}(p_{{N+i},j}\geq\frac{1}{N+i}+\sqrt{\ln\frac{1}{\delta}})$
$\displaystyle\leq\delta^{2}$
$\displaystyle\operatorname{Pr}(p_{{N+i},j}\leq\frac{1}{N+i}+\sqrt{\ln\frac{1}{\delta}})$
$\displaystyle\leq 1-\delta^{2}.$

Therefore, with probability at least $1-\delta^{2}$

$\|J_{ij}^{C}\|\leq C_{3}(\frac{1}{N+i}+\sqrt{\ln\frac{1}{\delta}})$

∎

##  Appendix A.3 Detailed Model Structure

In this section, we provide a more detailed description of each model
including LM (Table 8), ED (Table 10), RED (Table 9) and PALM (Table 11). We
denote $\mathcal{F}_{l}$ as the feedforward layer in the $l$th block. The
positional embedding is denoted as $E_{p}=\operatorname{Emb}_{p}(p_{s})$ while
the word embedding layer is denoted as $E_{w}=\operatorname{Emb}_{w}(s)$.
$E_{w},E_{p}\in\mathbb{R}^{|s|\times d}$ and $d$ is the dimension size.

Language Model
---
$a=[s_{1},s_{2},\cdots,s_{|s|},\Rightarrow,t_{1},t_{2},\cdots,t_{|t|}]$
$p_{a}=[1,2,\cdots,|s|+|t|]$
$E_{p}=\operatorname{Emb}_{p}(p_{a});E_{w}=\operatorname{Emb}_{w}(a)$
$G_{1}=E_{p}+E_{w}$
for $l$ in [1, $\cdots$, 6]:
$\quad Q_{l}=\mathtt{ATT}_{l}(G_{l},G_{l},G_{l})$
$\quad G_{l+1}=H_{l}=\mathcal{F}_{l}(Q_{l})$
$L=\mathcal{L}(H_{-1},a)$
Table 8: Language model framework. Regularized Encoder Decoder
---
Encoder | Decoder
$p_{s}=[1,2,\cdots,|s|]$ | $p_{t}=[|s|+1,\cdots,|s|+|t|+1]$
$E_{p}^{E}=\operatorname{Emb}_{p}(p_{s})$ | $E_{p}^{D}=\operatorname{Emb}_{p}(p_{t})$
$E_{w}^{E}=\operatorname{Emb}_{w}(s)$ | $E_{w}^{D}=\operatorname{Emb}_{w}(\text{`}\Rightarrow\text{'}+t)$
$G_{1}^{E}=E_{p}^{E}+E_{w}^{E}$ | $G_{1}^{D}=E_{p}^{D}+E_{w}^{D}$
for $l$ in [1, $\cdots$, 6]: |  for $l$ in [1, $\cdots$, 6]:
$Q_{l}^{E}=\mathtt{ATT}_{l}(G_{l}^{E},G_{l}^{E},G_{l}^{E})$ |  $Q_{l}^{D}=\mathtt{ATT}_{i}(G_{l}^{D},\begin{bmatrix}G_{l}^{E}\\\ G_{l}^{D}\end{bmatrix},\begin{bmatrix}G_{l}^{E}\\\ G_{l}^{D}\end{bmatrix}$)
$G_{l+1}^{E}=H_{l}^{E}=\mathcal{F}_{l}(Q_{l}^{E})$ | $G_{l+1}^{D}=H_{l}^{D}=\mathcal{F}_{l}(G_{l}^{D})$
$L=\mathcal{L}(H_{-1}^{E},s)+\mathcal{L}(H_{-1}^{D},t)$
Table 9: Regularized encoder decoder framework. Encoder-Decoder
---
Encoder | Decoder
$p_{s}=[1,2,\cdots,|s|]$ | $p_{t}=[1,2,\cdots,|t|]$
$E_{p}^{E}=\operatorname{Emb}_{p}^{E}(p_{s})$ | $E_{p}^{D}=\operatorname{Emb}_{p}^{D}(p_{t})$
$E_{w}^{E}=\operatorname{Emb}_{w}^{E}(s)$ | $E_{w}^{D}=\operatorname{Emb}_{w}^{D}(t)$
$G_{1}^{E}=E_{p}^{E}+E_{w}^{E}$ | $G_{1}^{\prime D}=E_{p}^{D}+E_{w}^{D}$
for $l$ in [1, $\cdots$, 6]: |  for $l$ in [1, $\cdots$, 6]:
$Q_{l}^{E}=\mathtt{ATT}_{l}^{E}(G_{l}^{E},G_{l}^{E},G_{l}^{E})$ | $G_{l}^{D}=\mathtt{ATT}_{l}^{D}(G_{l}^{\prime D},G_{l}^{\prime D},G_{l}^{\prime D})$
$G_{l+1}^{E}=H_{l}^{E}=\mathcal{F}_{l}^{E}(G_{l}^{E})$ | $Q_{l}^{D}=\mathtt{ATT}_{i}^{J}(G_{l}^{D},G_{l}^{E},G_{l}^{E})$
| $G_{l+1}^{D}=H_{l+1}^{D}=\mathcal{F}_{l}^{D}(Q_{l}^{D})$
$L=\mathcal{L}(H_{-1}^{D},t)$
Table 10: Encoder-decoder framework. Partial Attention Language Model
---
$a=[s_{1},s_{2},\cdots,s_{|s|},\Rightarrow,t_{1},t_{2},\cdots,t_{|t|}]$
$p_{a}=[1,2,\cdots,|s|,1,2,\cdots,|t|+1]$
$l_{a}=[0,0,\cdots,0,1,1,\cdots,1]$
$E_{p}=\operatorname{Emb}_{p}(p_{a});E_{w}=\operatorname{Emb}_{w}(a);E_{l}=\operatorname{Emb}_{l}(l_{a})$
$G_{1}=E_{p}+E_{w}+E_{l}$
for $l$ in [1, $\cdots$, 6]:
$\quad Q_{l}=\mathtt{ATT}_{l}(G_{l},G_{l},G_{l})$
$\quad
P_{l}=\mathcal{F}_{P,l}(Q_{l[1:|s|]});R_{l}=\mathtt{ATT}_{l}^{P}(Q_{l},P_{l},P_{l})$
$\quad G_{l+1}=H_{l}=\mathcal{F}_{i}(R_{l})$
$L=\mathcal{L}(H_{-1},a)$
Table 11: Partial attention language model framework.

Here, for the feedforward layer $\mathcal{F}_{P,l}$ in Table 11, it can be
interpreted as:

$\displaystyle
P_{l1}=\operatorname{Dropout}(\tanh(G_{l[1:|s|]}W_{P,l1}+\mathbb{1}b_{P,l1}^{\top}))$
$\displaystyle
P_{l2}=\operatorname{Dropout}(P_{l1}W_{P,l2}+\mathbb{1}b_{P,l2}^{\top})$
$\displaystyle P_{l}=P_{l2}+P_{l1},$

where $W_{P,l1},W_{P,l2}\in\mathbb{R}^{d\times
d},b_{P,l1},b_{P,l2}\in\mathbb{R}^{d}$ are trainable parameters,
$\mathbb{1}\in\mathbb{R}^{|s|\times 1}$ is a vector with all elements equal to
1.


# Signal Processing and Electronic Noise in LZ

D. Khaitana for the LZ Collaboration
aDepartment of Physics and Astronomy Corresponding author. University of
Rochester
Rochester  NY - 14627  USA
E-mail<EMAIL_ADDRESS>
###### Abstract

The electronics of the LUX-ZEPLIN (LZ) experiment, the 10-tonne dark matter
detector to be installed at the Sanford Underground Research Facility (SURF),
consists of low-noise dual-gain amplifiers and a 100-MHz, 14-bit data
acquisition system for the TPC PMTs. Pre-prototypes of the analog amplifiers
and the 32-channel digitizers were tested extensively with simulated pulses
that are similar to the prompt scintillation light and the electroluminescence
signals expected in LZ. These studies are used to characterize the noise and
to measure the linearity of the system. By increasing the amplitude of the
test signals, the effect of saturating the amplifier and the digitizers was
studied. The RMS ADC noise of the digitizer channels was measured to be 1.19
$\pm$ 0.01 ADCC. When a high-energy channel of the amplifier is connected to
the digitizer, the measured noise remained virtually unchanged, while the
noise added by a low-energy channel was estimated to be 0.38 $\pm$ 0.02 ADCC
(46 $\pm$ 2$\mu$V). A test facility is under construction to study saturation,
mitigate noise and measure the performance of the LZ electronics and data
acquisition chain.

###### keywords:

Data acquisition concepts; Digital signal processing (DSP); Front-end
electronics for detector readout

##  1 Introduction

The LUX-ZEPLIN (LZ) experiment is a second-generation dark-matter detector to
be installed in the Davis Cavern at the Sanford Underground Research Facility
(SURF) [1]. LZ contains 10 tonnes of liquid xenon with a 7-tonne active volume
in a two-phase time projection chamber (TPC). The detector will be housed in
the same water tank that is currently used for the Large Underground Xenon
(LUX) dark-matter detector to shield it from environmental radiation [2]. To
further reduce the background level in the detector, the TPC is surrounded by
a gadolinium-loaded liquid scintillator active veto. The scintillator and a
portion of the water tank are viewed by 120 PMTs. Another 180 PMTs, mounted
inside the cryovessel, monitor the thin layer of xenon between the cryostat
and walls of the TPC.

Figure 1: (Left) Overview of LZ installed in the Davis Cavern. (Right) Close-
up of the mezzanine level showing the breakout boxes and the cable conduits.
The electronics racks on the upper level are also indicated.

The TPC employs 488 Hamamtsu R11410 3-inch-diameter PMTs [1], distributed in
two arrays, one on the top with 247 PMTs in a hexagonal configuration with two
circular outer rows and one on the bottom with 241 PMTs distributed in a
close-packed hexagonal configuration. The high voltage (HV) and signal cables
for these PMTs are routed to a mezzanine level where two breakout boxes are
installed, as shown in Fig. 1. In this paper we describe tests carried out on
pre-prototype components of this electronics chain and discuss a setup that is
being assembled to test the full electronics and data acquisition chain.

##  2 Preliminary Tests of the LZ TPC Electronics Chain

The HVs for TPC PMTs are provided by Weiner MPOD EDS 20130x_504 modules [3].
Kerpen cables with 48-cores feed HV to the flanges on the breakout box from
the electronics rack, each of which provide HV for 32 PMTs [4]. Gore 3007
cables are used to supply HV from the breakout box to the base of the PMTs and
also to transport the signals from the PMT base to the breakout box [5]. These
cables have been chosen for their low heat load and low signal attenuation.
The area attentuation was measured with 45 and 90 feet of these cables with
2-ns risetime pulses and found to be 18% and 28%, respectively. The amplitude
attentuation was found to be 56% and 80%, respectively.

The signals of the TPC PMTs are amplified and shaped with dual-gain
amplifiers. The amplifier cards are located in cages on the breakout boxes
with each cage containing four amplifier cards with eight input channels each.
The high-energy (HE) output of the pre-prototype amplifier card has a full-
width at tenth maximum (FWTM) shaping time of 30 ns and an area gain of 0.5,
while the low-energy (LE) output has a FWTM shaping time of 60 ns and an area
gain of 20. The total front end LZ detector electronics noise requirement for
the amplifiers and digitizer is <0.5 mV.

Figure 2: Averaged noise spectrum for the pre-prototype amplifer and digitizer
chain. The trace of the noise spectrum for the DDC-32 is shown in black. The
noise spectra for a combinations of the DDC-32 and the LE and HE channels is
shown in light and dark grey, respectively.

The output signals from the amplifiers are routed to electronic racks located
directly above the breakout boxes on the upper level of the experimental hall,
using LMR-100-FR cables [6]. These cables are low smoke and zero halogen
(LSZH) cables and have a low signal area attenuation.

The output signals of the amplifier are digitized using DDC-32 digitizers
built by Skutek Instrumentation [7]. The digitizers have 32 input channels, 2V
dynamic range, 14 bit resolution and a sampling frequency of 100 MHz. The RMS
noise of the DDC-32 is 1.19 $\pm$ 0.02 ADCC, increasing to 1.22 $\pm$ 0.02
ADCC when a HE output channel of the amplifer is connected to it and 1.58
$\pm$ 0.02 ADCC in the case of the LE channel. The HE channel adds almosts no
noise to the inherent noise of the digitizer and the LE channel of the
amplifier adds 1.04 $\pm$ 0.02 ADCC of noise in quadrature.

The amplifier and digitizer chain was further studied to determine if the RMS
noise was dominated by noise at a particular frequency. For each of the three
combinations (DDC-32, HE+DDC-32 and LE+DDC-32), a thousand waveforms of 8,192
samples each were collected. The magnitude squared of the Fourier transforms
were averaged together and these spectra are shown in Fig. 2. All three
spectra fall as 1/$\sqrt{Hz}$ and there are no dominant noise contributing
frequencies. As was expected, the HE channel adds almost no noise to the noise
of the digitizer. The LE channel adds noise uniformly over a broad range of
frequencies.

Figure 3: Gain linearity plots generated with the pre-prototype amplifier. The
pre-prototype amplifiers have an area gain factor of 0.5 for the HE channel
(left) and an area gain factor of 20 for the LE channel (right).

The response of the pre-prototype amplifier and digitizer were tested for area
linearity with two types of pulses: a fast pulse with 10%-90% rise times in
the range 10-50 ns and fall times two and a half times the rise time and with
a slower pulse of Gaussian shape with rise times in the range of 150-1500 ns.
The input pulse amplitude to the amplifier was varied between 0.020 V and 0.50
V. For the results shown in Fig. 3, a fast pulse with a rise time of 20 ns was
used. A thousand waveforms were averaged to produce each point and the
standard deviation of the population is <1%. For all tested pulses, the
measured area gains match specifications. For slower pulses the saturation
points moved to larger input areas, thereby giving us a pulse width dependent
dynamic range for LZ [8].

##  3 Full Electronics Test Facility

Figure 4: Schematic overview of the setup for the electronics chain test. The
direction of signal flow is indicated as are the various sections of the
setup.

To test all the components of the LZ data acquisiton system, a electronics
chain test facility is being created at the University of Rochester. This
facility will include all the LZ components from PMT to disk, as shown in Fig.
4. The purpose of this facility is to provide a development system to
extensively test and measure the performance of hardware, software, and
firmware, before deployement in LZ. Having all the components from PMT to
digitizer will provide the opportunity to study the noise in the full
electronics chain.

This setup will include two Hamamatsu R11410 PMTs in a small vacuum vessel,
which includes a feed through for an LED signal. The same signal and HV
flanges to be used on the breakout boxes are used on the vessel. Only one
feed-through connector on the HV flange will be populated to bias the two
PMTs. The signal flange is mounted with an amplifier cage and two amplifier
cards. These extra channels from the amplifier will be digitized to measure
crosstalk and develop methods to mitigate common mode noise.

The signals from the amplifiers will be digitizied with two DDC-32s. Only when
a valid event is identified from the external trigger in the DAQ Master is
data offloaded from the DDC-32 to disk on the Data Collector via the Data
Extractor [9]. This entire digital chain will be monitored and controlled by
the Data Acquisition Master computer.

The Event Builder reads data from the disks on the Data Collectors and builds
event files. Run Control will initiate the start and stop of the data
acquisiton, as well as monitor its status and the status of the Event Builder.
Run Control will also be responsible for communicating the run parameters to
the Data Acquisition Master Computer and the Event Builder.

The use of an LED will allow a range of components in the chain to be tested.
Small, well timed pulses will be generated in the LED to study relative timing
difference between channels, while larges pulses will be used to study
saturation in the PMTs, the amplifiers, and the digitizers. Using a high pulse
rate provides the opportunity to stress test the data throughput and benchmark
the performance of the acquisition system. Currently, the vacuum vessel is
being assembled and firmware and software are being deployed.

##  4 Summary

Testing has been completed on pre-prototypes of the amplifier and digitizer
and they have met noise and area linearity requirements. The process of
constructing an electronics test facility to mimic the setup of LZ TPC PMTs
has begun. Using this setup we will study the effects of saturation as well as
possible time difference between channels on the PMT, amplifier, and digitizer
and develop techniques to mitigate noise. This setup will also be used to
optimize the preformance of the data acquisiton and event building hardware
and software.

###### Acknowledgements.

The work was partially supported by the U.S. Department of Energy (DOE) under
award numbers DE- SC0012704, DE-SC0010010, DE-AC02-05CH11231, DE-SC0012161,
DE-SC0014223, DE-FG02- 13ER42020, DE-FG02-91ER40674, DE-NA0000979, DE-
SC0011702, DE-SC0006572, DE-SC0012034, DE-SC0006605, and DE-FG02-10ER46709; by
the U.S. National Science Foundation (NSF) under award numbers NSF PHY-110447,
NSF PHY-1506068, NSF PHY-1312561, and NSF PHY-1406943; by the U.K. Science &
Technology Facilities Council under award numbers ST/K006428/1, ST/M003655/1,
ST/M003981/1, ST/M003744/1, ST/M003639/1, ST/M003604/1, and ST/M003469/1; and
by the Portuguese Foundation for Science and Technology (FCT) under award
numbers CERN/FP/123610/2011 and PTDC/FIS-NUC/1525/2014.

## References

  * [1] D.S. Akerib et al., _LUX-ZEPLIN (LZ) Conceptual Design Report_ , http://arxiv.org/abs/1509.02910 [arXiv:physics.ins-det/1509.02910].
  * [2] D.S. Akerib et al., _The Large Underground Xenon (LUX) experiment_ http://www.sciencedirect.com/science/article/pii/S0168900212014829http://www.sciencedirect.com/science/article/pii/S0168900212014829Nucl. Instrum. Meth. A 704 (2013) 111-126.
  * [3] Weiner HV System http://www.wiener-d.com/sc/power-supplies/mpod–lvhv/mpod-hv-module.html _http://www.wiener-d.com/sc/power-supplies/mpod–lvhv/mpod-hv-module.html_
  * [4] Kerpen Cable http://www.leoni-industrial-projects.com/uploads/tx_downloadleoni/en_Power_and_Control__Cables_web_01.pdf _http://www.leoni-industrial-projects.com/_
  * [5] Gore Cable http://www.gore.com/en_xx/products/cables/index.html _http://www.gore.com/_
  * [6] LMR Cables http://www.timesmicrowave.com/cms/products/cables/lmr/_http://www.timesmicrowave.com/cms/products/cables/lmr/_
  * [7] Skutek Instrumentation http://www.skutek.com/_http://www.skutek.com/_
  * [8] J. Yin _The Dynamic Range of LZ_ (JINST) 2015.
  * [9] E. Druszkiewics, _The Data Acquisition System for LZ_ , JINST (2015).