# Information-Weighted Neural Cache Language Models for ASR

###### Abstract

Neural cache language models (LMs) extend the idea of regular cache language
models by making the cache probability dependent on the similarity between the
current context and the context of the words in the cache. We make an
extensive comparison of ‘regular’ cache models with neural cache models, both
in terms of perplexity and WER after rescoring first-pass ASR results.
Furthermore, we propose two extensions to this neural cache model that make
use of the content value/information weight of the word: firstly, combining
the cache probability and LM probability with an information-weighted
interpolation and secondly, selectively adding only content words to the
cache. We obtain a 29.9%/32.1% (validation/test set) relative improvement in
perplexity with respect to a baseline LSTM LM on the WikiText-2 dataset,
outperforming previous work on neural cache LMs. Additionally, we observe
significant WER reductions with respect to the baseline model on the WSJ ASR
task.

Index Terms—  language modeling, LSTM, cache, speech recognition

##  1 Introduction

Language models (LMs) play a crucial role in many speech and language
processing tasks such as speech recognition and machine translation. The
current state of the art are recurrent neural network (RNN) based LMs [1], and
more specifically long short-term memory (LSTM) [2] LMs [3, 4].

This work focuses on extending LSTM LMs with a cache [5], which gives the LM
two main advantages: firstly, the cache is a simple and computationally cheap
manner to adapt the LM to the current topic since it increases the
probabilities of words recently seen. Secondly, Kuhn and De Mori [5] first
proposed a cache model as a means to overcome one of the limitations of n-gram
LMs, namely that they are not capable of modeling short-term patterns in word
use. One can argue that LSTMs are better at this because they should be able
to remember words for a longer time. However, the effect of a single word
might be obscured after a certain amount of time steps since with each time
step, a complete update of the cell state and hidden state is done. Explicitly
adding previously seen words to a cache and combining the cache and LSTM
probabilities can counteract this effect.

Grave et al. [6] recently proposed a continuous cache (henceforth referred to
as neural cache, as opposed to Kuhn and De Mori’s regular cache [5]), that
assigns cache probabilities based on the similarity between the current hidden
state and the hidden state associated with the word in the cache. As a result,
words previously seen in a similar context get a higher cache probability than
words seen in a different context. The advantages of the neural cache for
neural LMs have also been demonstrated by Merity et al. [7] and Khandelwal et
al. [8], but they use only a neural cache and do not compare with a regular
cache, and only evaluate on textual data.

The manner in which the cache probability is combined with the LSTM LM
probability is a crucial part in cache LMs. Grave et al. [6] mention linear
interpolation and another method called global normalization, but the latter
approach does not outperform linear interpolation. One can argue that linear
interpolation – assigning the same weight to the cache probability regardless
of the type of word – is suboptimal since the cache probability will be more
helpful for content words whereas the LM probability should receive more
weight for function words. Kuhn and De Mori [5] train separate interpolation
weights and use separate caches for each part-of-speech (POS), combining the
cache model with a class-based n-gram model in which the classes correspond to
POS classes. However, we believe that keeping a separate cache for certain
classes such as determiners and prepositions does not make much sense because
for those classes, the overall statistics captured by the LM should be much
more meaningful. For example, having seen the word in multiple times before
does not increase the probability of seeing it again. On the other hand,
certain constructions such as verbs requiring this preposition (e.g. believe,
specialize) or sentences in which adverbial phrases indicating time or space
are plausible (e.g. On the day of its release in Japan, Work began on the
Tower Building in 1840111Extracts from WikiText [9].), do increase the
probability of seeing in, and such dependencies should be captured by the LM.
Another disadvantage of Kuhn and De Mori’s approach is that it needs LM
training data that has been tagged for POS.

The goal of this paper is two-fold: 1) we propose to weigh the cache model
with a measure that is solely based on the frequency statistics of the
training data and which hence does not need training data augmented with POS:
the information weight (IW) of a word (see section 2.3 for a definition). This
measure can be automatically calculated, and we investigate whether it can
give further improvements for a cache model. The IW can be used in
information-weighted interpolation and/or for an information-weighted
selective cache, to which only words with an IW larger than a certain
threshold are added.

Secondly, as far as we know no comparison between regular and neural cache
models for automatic speech recognition (ASR), an application for which cache
models typically give good improvements, has been made. Thus, 2) the second
goal of this paper is to investigate what advantage a (information-weighted)
neural cache can have for ASR. Typically, neural LMs for ASR are used in a
multi-pass approach: in a first pass, the search space is narrowed down with
the help of a simple (usually n-gram or FST) LM, producing a set of hypotheses
in the format of a lattice or an N-best list. We evaluate our models by
rescoring N-best lists.

In the remainder of this paper, we will first define the different cache
models and different manners of interpolation (section 2) and validate the
proposed methods experimentally (section 3). We end with a conclusion and an
outlook to future work (section 4).

##  2 Cache language models

###  2.1 Regular cache

A regular cache model keeps track of the words previously seen and computes
the probability of the next word being $w_{t}$ as follows:

$P_{c}(w_{t}~{}|~{}w_{t-|C|-1}\ldots~{}w_{t-1})=\frac{\sum\limits_{j=t-|C|-1}^{t-1}I_{\\{w_{j}=w_{t}\\}}}{|C|}$
(1)

where $C$ is the cache and $|C|$ its size, $t$ is the current time step and
$I$ the indicator function, which is 1 if $w_{j}=w_{t}$ and 0 otherwise. An
exponential decay can be used to give more weight to words seen more recently:
$I_{\\{w_{j}=w_{t}\\}}e^{-\alpha~{}(t-j)}$, with $\alpha$ the decay rate.

###  2.2 Neural cache

The neural cache model proposed by Grave et al. [6] calculates the cache
probability as follows:

$\displaystyle\begin{split}P_{c}(w_{t}~{}&|~{}(w_{t-|C|-1},h_{t-|C|-2})\ldots~{}(w_{t-1},h_{t-2}))=\\\
&\frac{\sum\limits_{j=t-|C|-1}^{t-1}I_{\\{w_{j}=w_{t}\\}}e^{(\theta~{}h_{t}^{T}~{}h_{j-1})}}{\sum\limits_{w_{i}\in
C}~{}\sum\limits_{j=t-|C|-1}^{t-1}I_{\\{w_{j}=w_{i}\\}}e^{(\theta~{}h_{i}^{T}~{}h_{j-1})}}\end{split}$
(2)

In the equation above, $h_{j-1}$ is the hidden state of the LSTM cell, which
corresponds to the cell’s output that is used to predict the next word
$w_{j}$. In other words, the cache $h_{c}$ contains pairs of hidden states and
corresponding target words. $\theta$ is a hyperparameter that controls the
flatness of the distribution – the lower $\theta$, the flatter the
distribution of the cache probabilities. Given that $h_{t}^{T}~{}h_{j-1}$ is
in the range (-1,1), the minimum value of $e^{(\theta~{}h_{t}^{T}~{}h_{j-1})}$
will always be $<1$ and the maximum value $>1$ (for $\theta>0$), as opposed to
a regular cache, where the value in the denominator is always 1.

If a word is in the cache, its cache probability is calculated based on the
inner product between the current hidden state of the LSTM $h_{t}$ and the
hidden state stored in the cache $h_{j-1}$. Hence, the probability is not
solely based on whether the word has been seen before or not – as happens in
regular cache models –, but also based on the similarity between the current
context and the context in which the cache word occurred, since we assume that
the hidden state of the LSTM captures the previous context of the word. For
example, in a Wikipedia article about the Chinese poet Du Fu, we see that the
neural cache assigns a much higher probability (compared to the unigram
probability of the regular cache) to Fu if Du is the input word, since these
two words occur together very frequently.

###  2.3 Information-weighted interpolation of probabilities

Grave et al. [6] describe two ways of combining the cache probability with the
standard probabilities output by the network, of which linear interpolation
performs slightly better.

However, knowing that a word has been seen before might be more meaningful for
certain words than for others. Hence, we propose to exploit this fact by
assigning a larger interpolation weight to words with a large content value
than to words with a low content value, such as function words. The
information weight $\lambda_{i}$ of a word is defined as follows:

$\lambda_{i}=1+\frac{\sum_{j=1}^{N}~{}P(j|i)~{}\log(P(j|i))}{\log(N)}$ (3)

where $P(j|i)$ is the probability of document $j$ given that word $i$ occurs
in it, and is equal to the frequency of word $i$ in document $j$ divided by
the frequency of word $i$ in the whole corpus. $N$ is the number of documents
in the corpus; for our experiments we split the corpus in documents of an
equal amount of sentences. Equation 3 corresponds to 1 – normalized entropy,
hence giving higher importance to words that are sparsely distributed across
the corpus. This measure is among others used by Dumais [10] and Bellegarda
[11] as a global weighting for the Latent Semantic Analysis word-document
matrix. The values for $\lambda$ range between 0, for words that are uniformly
distributed among the corpus, and 1, for words that appear in only 1 document.
For example, bilingual has a weight of $\lambda=0.910$ whereas the has a
weight of $\lambda=0.019$ for the WikiText-2 corpus [9].

The final probability for word $i$ is obtained by information-weighted linear
interpolation:

$\begin{split}&P(w_{t}~{}|~{}history)=\\\
&\frac{(1-\gamma~{}\lambda_{t})P_{LM}(w_{t}~{}|~{}h_{t})+(\gamma~{}\lambda_{t})P_{c}(w_{t}~{}|~{}h_{c})}{\sum\limits_{w\in
V}(1-\gamma~{}\lambda_{t})P_{LM}(w_{t}~{}|~{}h_{t})+(\gamma~{}\lambda_{t})P_{c}(w_{t}~{}|~{}h_{c})}\end{split}$
(4)

where $\gamma$ is a hyperparameter that scales the information weights and
needs to be set empirically. We only consider values smaller than or equal to
0.5, such that the cache probability will never be assigned a higher weight
than the LM probability. Thus, the LM probabilities have more weight for
frequent words, while for topical words a combination of LM and cache
probabilities is used, which will lead to increased probabilities for topical
words in the cache.

###  2.4 Information-weighted selective cache

Another manner to use the information weight of a word is to use it as a hard
threshold: only words with an information weight greater than or equal to a
certain threshold $\phi$ are added to the cache. This approach has the
advantage that no space is ‘wasted’ on words with low content value. It
effectively enlarges the scope of the cache. A selective cache can be used in
combination both with a regular and neural cache model, and with linear and
information-weighted interpolation.

|  |  |  | Regular cache | Neural cache
---|---|---|---|---|---
Model | Size | Interp. | IW select. | Valid | Test | $\lambda~{}/~{}\gamma$ | $\phi$ | $\alpha$ | Valid | Test | $\lambda~{}/~{}\gamma$ | $\phi$ | $\theta$
Baseline [6] | - | - | - | - | 99.3 | - | - | - | - | 99.3 | - | - | -
Cache [6] | 100 | linear | - | - | - | - | - | - | - | 81.6 | - | - | -
Our baseline | - | - | - | 102.5 | 97.6 | - | - | - | 102.5 | 97.6 | - | - | -
Cache | 100 | linear | no | 92.6 | 87.9 | 0.10 | - | - | 85.6 | 81.1 | 0.10 | - | 0.3
Cache | 100 | IW | no | 91.9 | 87.1 | 0.20 | - | - | 86.1 | 81.9 | 0.25 | - | 0.3
Our baseline | - | - | - | 103.0 | 98.6 | - | - | - | 103.0 | 98.6 | - | - | -
Cache | 100 | linear | yes | 90.7 | 86.8 | 0.05 | 0.2 | - | 84.4 | 81.3 | 0.05 | 0.2 | 0.3
Cache | 100 | IW | yes | 86.3 | 82.6 | 0.25 | 0.4 | - | 79.6 | 76.7 | 0.35 | 0.2 | 0.3
Baseline [6] | - | - | - | - | 99.3 | - | - | - | - | 99.3 | - | - | -
Cache [6] | 2000 | linear | - | - | - | - | - | - | - | 68.9 | - | - | -
Our baseline | - | - | - | 103.4 | 98.1 | - | - | - | 103.4 | 98.1 | - | - | -
Cache | 2000 | linear | no | 88.6 | 83.8 | 0.10 | - | 0.005 | 73.4 | 69.6 | 0.15 | - | 0.3
Cache | 2000 | IW | no | 87.4 | 83.0 | 0.30 | - | 0.004 | 73.9 | 70.9 | 0.35 | - | 0.3
Our baseline | - | - | - | 103.3 | 97.5 | - | - | - | 103.3 | 97.5 | - | - | -
Cache | 2000 | linear | yes | 88.3 | 84.0 | 0.05 | 0.2 | 0.008 | 76.6 | 73.4 | 0.20 | 0.1 | 0.3
Cache | 2000 | IW | yes | 82.4 | 78.3 | 0.35 | 0.2 | 0.006 | 72.4 | 66.2 | 0.45 | 0.2 | 0.3
Table 1: Validation and test perplexities for LSTM LMs with a cache trained on
WikiText-2, along with their optimal hyperparameters (if applicable). The
column $\lambda~{}/~{}\gamma$ contains the optimal interpolation weight
$\lambda$ for models with linear interpolation and the optimal scale for the
IW $\gamma$ for models with IW interpolation.

##  3 Experiments

###  3.1 Setup

For perplexity experiments, we train our LMs on WikiText-2 [9]. This dataset
is also used by Grave et al. [6] and contains 2M words for training, 220k
words for validation, 250k words for testing and has a vocabulary of 33k
words.

All training and perplexity experiments are implemented with TensorFlow [12].
The LSTM LMs trained on WikiText have 1 layer of 512 LSTM cells, randomly
initialized with a uniform distribution between -0.05 and 0.05. We use mini-
batches of 20 samples and train the LSTMs with backpropagation through time
(35 steps) and 50% dropout [13]. The norm of the gradients is clipped at 5. We
train with stochastic gradient descent: during the first 6 epochs, the
learning rate is 1, after which we apply an exponential decay: the learning
rate for epoch $i$, $\eta_{i}$, is equal to $\alpha~{}\eta_{i-1}$, with
$\alpha$ the learning rate decay which is set to 0.8. After each epoch, we
check how often the validation perplexity has not improved with respect to the
previous epochs: if it has not improved for 3 times, we stop training. If the
validation perplexity keeps on improving, we stop training anyway after 39
epochs. We train on discourse level, which means that we do not reset the LSTM
state at sentence boundaries and predict the first word of the next sentence
from the end-of-sentence token (for more information, see Verwimp et al.
[14]). This model gives a baseline perplexity close the the one reported by
Grave et al. [6]. The hyperparameters specific to the cache model are
optimized on the validation set. We provide all values for the hyperparameters
together with the results, since it has been shown that these have an
important influence [15].

We use Kaldi [16] for the ASR experiments. We follow the standard nnet2 recipe
for Wall Street Journal (WSJ), generating 1000-best lists for rescoring. The
first pass of the recognition makes use of a 3-gram LM with a vocabulary of
120k words. The training data for the LSTM LMs consists of 37M words with a
vocabulary of 40k. It contains years 87–89 with non-verbalized punctuation.
The new LM probability is calculated as follows:

$\begin{split}P_{new}(W)&=(1-\lambda_{LSTM})*P_{ngram}(W)\\\
&+\lambda_{LSTM}*P_{LSTM+C}(W)\end{split}$ (5)

where $P_{ngram}$ is the first-pass LM probability and $P_{LSTM+C}$ is equal
to the probability in equation 4. A scaled version of $P_{new}(W)$ ($\beta$ =
scaling factor) is combined with the score of the acoustic model $P(O|W)$:

$\begin{split}\log P(W)=\log
P(O|W)+\beta~{}\log(P_{new}(W))+L~{}\omega\end{split}$ (6)

$\omega$ is the word insertion penalty and $L$ the number of words in the
hypothesis. For every model type, hyperparameters such as word insertion
penalty $\omega$, LM scale $\beta$, LSTM LM weight $\lambda_{LSTM}$ are
optimized on the validation set. The LSTM LMs used for rescoring are trained
with TensorFlow [12] with the same hyperparameters as the WikiText models,
except that we train on sentence level instead of discourse level.

We use the bootstrap method of Bisani and Ney [17] for significance analysis
of the WER results, which is implemented as Kaldi’s compute-wer-bootci script.
This method generates (by default 10k) random bootstrap samples of the data
and calculates the percentage of times the bootstrap sample from system B
improves over the bootstrap sample from system A: the result is a ‘probability
of improvement’ (poi) of system B over system A.

Since the length of the cache is usually longer than the length of the
hypotheses in a single N-best list, we would like to be able to transfer the
cache across lists. We keep track of the probabilities and caches for every
hypothesis in the list, and initialize the cache of the next N-best list with
the cache of the most likely hypothesis from the previous list.

###  3.2 Perplexity results

We report the perplexity of the LSTM LMs in table 1. Due to implementation
requirements, we have to train 4 different models: one with a non-selective
cache of size 100, one with a selective cache of size 100 and the same options
for a cache size of 2000. The baseline perplexities for LSTMs without cache of
these four models do not differ much, but we report them all for the sake of
completeness (‘Our baseline’ in the table). All perplexity results shown below
the same baseline can thus be attributed solely to the cache component since
they are obtained with the same trained model.

Firstly, we combine the LSTM with a regular cache model with linear
interpolation (left part of the table) and see that this simple combination
already gives us improvements: 11.4%/11.6% relatively on the validation/test
set for a cache size of 2000 (with exponential decay). This result confirms
our hypothesis, namely that an LSTM in itself is not sufficient in remembering
the relevant words in the previous context. Using information-weighted (IW)
interpolation or IW selective cache usually gives additional improvements. The
two IW-based techniques seem to give complimentary improvements, because the
combination yields the best perplexity result (relative improvement of
20.2%/19.7%) for the regular cache model.

The results for LSTM LMs with a neural cache are reported in the right part of
the table. We observe that, similarly to the regular cache, the combination of
both information-based extensions yields perplexity improvements, also with
respect to the results reported by Grave et al. [6]. Using only one of the two
IW-based techniques does not result in better perplexity results.
Additionally, we observe that the neural cache consistently outperforms the
regular cache. The best result is obtained with a neural selective cache of
size 2000 with information-weighted interpolation: a relative improvement of
29.9%/32.1% (validation/test) with respect to our own baseline.

###  3.3 Analysis

Taking a closer look at the optimal hyperparameters, we firstly observe that
$\gamma$ is larger for the neural cache than for the regular cache. For the
best model, $\gamma=0.45$, resulting in weights close to 0.5 for words with a
large IW $\lambda$, which indicates a large reliance on the cache for content
words.

Secondly, the optimal threshold $\phi$ for most IW selective cache models is
quite low, 0.2. This threshold essentially excludes only 222 word types from
the cache, that albeit form more than 50% of the test set (135k out of 240k).

In figure 1, we plot how often the information-thresholded cache assigns a
larger interpolated probability to the target word than the original LM
probability. Firstly, since $\phi$ is 0.2 for the selective model, very
frequent low-content words (with IW $<0.2$) will always have a lower
interpolated probability than the softmax probability because the cache
probability is 0. However, for IWs larger than 0.2, the selective model is
more likely to result in a better interpolated probability, and that
probability increases if the IW itself increases.

0-0.20.2-0.40.4-0.60.6-0.80.8-1.0$0$$10$$20$$30$$40$$50$information weight
range% interpolated prob $\geq$ softmax probIW interpolationIW interpolation +
IW selective

Fig. 1: The proportion of times that the interpolated probability is higher
than or equal to the baseline LM probability with respect to several ranges of
IWs, for the best neural cache models with IW interpolation and with the
combination of IW selective cache and IW interpolation.

As a result, if we look at the predictions over all IWs, the non-selective
model outperforms the LM probability in 31.9% of the cases and the selective
model only in 25.1% of the cases. But for IWs $>0.2$, the selective models
outperforms the baseline in 43.5% of the cases, and the non-selective one only
in 33.9% of the cases. When the interpolated probability is higher than the
softmax probability, the relative improvement of the interpolated probability
with respect to the softmax probability, averaged over the test set, is 55.4%
for the non-selective model and 54.3% for the selective model. For IWs $>0.2$,
the relative improvement is higher for the selective model (67.8%) than for
the non-selective one (65.8%), although the difference is small. Thus, the
cache model combining the IW interpolation and the IW selection gives worse
estimates for very frequent words than the baseline model and the cache with
only IW interpolation since they have a cache probability of 0, but better
estimates for topical words, resulting in an overall perplexity improvement.

###  3.4 Speech recognition results

In table 2, we report the perplexity results on the WSJ validation set and the
validation and test WER after rescoring with LSTM LMs. Similarly as for
WikiText, we report the results for two different baseline models, a selective
one and a non-selective one. Notice that all results within the upper part and
respectively the lower part of the table are obtained with the same trained
model. Thus, differences in performance can be attributed to the cache parts
solely and not to different initialization settings that might lead to better
convergence of the model during training. As a sanity check, we tested whether
the differences in WER between the two baseline models were significant:
according to the bootstrap method [17], the probability of the first baseline
improving over the second baseline is 39.97.5%/73.57% (validation/test set).
Thus, we conclude that the difference between the baselines is not
significant. We only test models with a cache size of 100, since preliminary
perplexity results with a larger cache size did not give substantial
improvements with respect to a cache of 100 for this dataset.

Firstly, we observe that the differences in perplexity are much smaller than
for WikiText: this shows that on the WSJ dataset, much less can be gained by
using a cache. This is not surprising given the fact that WikiText is
specifically designed to contain many long-term dependencies [9]. Secondly,
the neural cache is not consistently better than the regular cache, as opposed
to WikiText. The combination of a neural cache with linear interpolation gives
the best results, both for a non-selective and a selective model, but the
improvement is only 4.6%/4.1% (non-selective/selective) relative to the
baseline. Overall, the differences in perplexity between the cache models are
quite small, but notice that this dataset is quite small (8k words, compared
to 210k/240k words for the validation/test set of WikiText) and hence we
should be careful in drawing strong conclusions based on these results.

The cache models give WER improvements between 0.2 and 0.32 absolute for
rescoring the 1000-best lists of the validation set. Between brackets we
indicate the probability of improvement with respect to the baseline: all
results except the neural cache with linear interpolation are significant with
a confidence larger than 95%, but the best results are still quite far from
the oracle WER of 2.18. We observe that the best model in terms of perplexity
does not yield the best WER results, and that the WER differences between the
cache models are not significant.

Model | Valid PPL | Valid WER (poi) | Test WER (poi) |  $\lambda$ / $\gamma$ | $\phi$ | $\lambda_{LSTM}$ | $\beta$ | $\omega$
---|---|---|---|---|---|---|---|---
baseline | 176.9 | 5.61 | 2.55 | - | - | 0.5 | 18 | 0
regular linear | 170.5 |  5.36 (96.55) | 2.34 (93.95) | 0.05 | - | 0.5 | 15 | -0.5
regular IW | 171.2 | 5.41 (96.47) |  2.25 (99.34) | 0.4 | - | 0.5 | 13 | -0.5
neural linear | 168.8 | 5.38 (94.63) | 2.43 (80.66) | 0.05 | - | 0.5 | 15 | -0.5
neural IW | 170.3 | 5.38 (97.21) | 2.39 (89.87) | 0.7 | - | 0.4 | 13 | 0
baseline | 175.1 | 5.59 | 2.64 | - | - | 0.5 | 18 | 0
select. reg. linear | 170.2 | 5.32 (99.10) | 2.68 (33.68) | 0.05 | 0.05 | 0.3 | 13 | 0
select. reg. IW | 168.1 |  5.27 (99.81) |  2.61 (58.75) | 0.1 | 0.05 | 0.3 | 13 | 0
select. neur. linear | 167.9 | 5.32 (99.10) | 2.64 (46.33) | 0.05 | 0.05 | 0.3 | 13 | 0
select. neur. IW | 168.8 | 5.31 (99.57) | 2.63 (52.32) | 0.1 | 0.005 | 0.3 | 13 | 0
Table 2: Perplexities on the validation set of WSJ and WERs after rescoring
the 1000-best lists of the validation and test set. All cache models have a
cache size of 100. ‘poi’ = probability of improvement [17] of the cache model
with respect to its baseline. The 5 last columns contain the hyperparameter
values optimized on the validation set by rescoring, which are also used for
rescoring the test set. The column $\lambda~{}/~{}\gamma$ contains the optimal
$\lambda$ for models with linear interpolation and the optimal $\gamma$ for
models with IW interpolation. We do not include the decay values since
exponential decay did not improve the results. Oracle WER for the 1000-best
lists is 2.18 (validation) and 1.21 (test).

On the test set, the non-selective models achieve WER reductions between 0.12
and 0.3 absolute; notice that the largest possible reduction is 1.43 (oracle
WER 1.21). However, only the regular cache with IW interpolation achieves a
significant WER reduction with respect to the baseline. This model also
significantly improves the results for the neural models (both with a
confidence $>$ 98%), but not significantly the result of the regular cache
with linear interpolation (poi of 89.63%). The regular cache with linear
interpolation significantly improves its neural counterpart (poi 98.36%), but
not the neural cache with IW interpolation (poi 72.26%). Given that there are
no significant differences between the cache models on the validation set on
which the hyperparameters are optimized, this suggests that the regular cache
models are more robust against hyperparameter settings that are possibly not
optimal for the test set.

The selective models on the other hand, show no or only minor improvement with
respect to their baseline model on the test set. The best model only gives
0.03 absolute reduction in WER. We investigated several possible explanations
for this large discrepancy between the results of the selective models on the
validation and on the test set. Firstly, it is possible that the selective
models are much more sensitive to hyperparameter settings, but tuning the
hyperparameters on the test set as an oracle experiment does not improve the
results. A second possible explanation is that the selective models only work
well on data with a specific structure/specific patterns, and both WikiText
and the validation set have such structure while the test set does not.
However, if we calculate perplexity of the test set, the selective neural
cache with IW interpolation is among the best models (5.7% relative
improvement). On the other hand, we observe with this dataset that perplexity
results do not always extrapolate nicely to WER results. This might be because
the perplexity is calculated on the references only, while the WER is the
result of rescoring many hypotheses, of which some might be more probable than
the reference according to the LM. If the wrong hypothesis is selected, the
wrong cache is transferred across N-best lists. Since the selective models
have a larger scope because no space is wasted on function words, the wrong
words are effectively longer in the cache. The effect of wrong words in the
cache might be worse for certain datasets (e.g. the WSJ test set) than for
others (e.g. the WSJ validation set). A very crude measure that provides some
evidence for this explanation of the validation versus test set discrepancy,
is counting the number of times that a wrongly selected hypothesis is followed
by another wrongly selected hypothesis. We observe that for all non-selective
models, this number is lower for the cache models than for the baseline model,
while for all selective models, this number is higher for the cache models
than for the baseline model, providing plausible evidence for the fact that
errors are propagated longer for selective models. This is even the case for
the validation set, but on this dataset, the gains that the selective model
provides are larger than the losses, such that the overall WER is lower for
the cache models than for the baseline model.

##  4 Conclusion and future work

In this work, we present an extensive evaluation and comparison of regular and
neural cache LMs, in terms of perplexity and WER. Additionally, we propose two
extensions based on the information weight/content value of words that can be
applied to any cache model: IW interpolation of cache and LM probabilities and
an IW selective cache.

On the WikiText benchmark, we obtain large improvements in perplexity, the
largest one obtained by the combination of the two information-weighted
extensions proposed in this paper. Additionally, we observe that neural cache
models consistently outperform regular cache models on this dataset.

We also test the cache models on ASR by rescoring 1000-best lists of WSJ. We
observe significant improvements with respect to a baseline LM on the
validation set, while on the test set the regular cache with IW interpolation
was the only model giving a significant improvement. As opposed to WikiText,
neural cache models are not consistently better than regular cache models, and
the combination of the two information-weighted extensions is not the best
option. The neural cache seems to be more sensitive to hyperparameter
settings, since the regular cache models are significantly better than the
neural cache models on the test set, but not on the validation set. The
selective models provide significant gains on the validation set, but not on
the test set: we hypothesize that those models are more sensitive to the
selection of wrong hypotheses in previous segments, because the wrong words
stay longer in the cache.

In the future, we would like to test our models on ASR tasks that might give
more gains, and test a ‘smarter’ manner of transferring the cache across
N-best lists, e.g. a weighted combination of all hypotheses from the previous
list.

## References

  * [1] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur,  “Recurrent neural network based language model,”  in Proceedings of Interspeech, 2010, pp. 1045–1048.
  * [2] Sepp Hochreiter and Jürgen Schmidhuber,  “Long short-term memory,”  Neural Computation, vol. 9, no. 8, pp. 735–1780, 1997.
  * [3] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney,  “LSTM Neural Networks for Language Modeling,”  in Proceedings of Interspeech, 2012, pp. 1724–1734.
  * [4] Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever,  “An Empirical Exploration of Recurrent Network Architectures,”  in Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015, pp. 2342–2350.
  * [5] Roland Kuhn and Renato De Mori,  “A Cache-Based Natural Language Model for Speech Recognition,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 6, pp. 570–583, 1990.
  * [6] Edouard Grave, Armand Joulin, and Nicolas Usunier,  “Improving Neural Language Models with a Continuous Cache,”  in International Conference on Learning Representations (ICLR), 2017\.
  * [7] Stephen Merity, Nitish Shirish Keskar, and Richard Socher,  “Regularizing and Optimizing LSTM Language Models,”  in International Conference on Learning Representations (ICLR), 2018.
  * [8] Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky,  “Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context,”  in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) (Long Papers), 2018, pp. 284–294.
  * [9] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher,  “Pointer Sentinel Mixture Models,”  in International Conference on Learning Representations (ICLR), 2017\.
  * [10] Susan T. Dumais,  “Improving the retrieval of information from external sources,”  Behavior Research Methods, Instruments, & Computers, vol. 23, no. 2, pp. 229–236, 1991.
  * [11] Jerome R. Bellegarda,  “A Multispan Language Modeling Framework for Large Vocabulary Speech Recognition,”  IEEE Transactions on Speech and Audio Processing, vol. 6, no. 5, pp. 456–467, 1998.
  * [12] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng,  “Tensorflow: Large-scale machine learning on heterogeneous systems,” Software available from tensorflow.org, 2015.
  * [13] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov,  “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,”  Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014\.
  * [14] Lyan Verwimp, Hugo Van hamme, and Patrick Wambacq,  “TF-LM: TensorFlow-based Language Modeling Toolkit,”  in Proceedings of the Language Resources and Evaluation Conference (LREC), 2018.
  * [15] Gábor Melis, Chris Dyer, and Phil Blunsom,  “On the State of the Art of Evaluation in Neural Language Models,”  in International Conference on Learning Representations (ICLR), 2018\.
  * [16] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely,  “The Kaldi Speech Recognition Toolkit,”  in IEEE 2011 Workshop on Automatic Speech Recognition and Understanding (ASRU), 2011.
  * [17] Maximilian Bisani and Hermann Ney.,  “Bootstrap estimates for confidence intervals in ASR performance evaluation,”  in Proceedings of Acoustics, Speech, and Signal Processing (ICASSP), 2004, pp. 409–412.


# Bremsstrahlung photons from stopping in heavy-ion collisions

Sohyun Park<EMAIL_ADDRESS>Urs Achim Wiedemann<EMAIL_ADDRESS>Theoretical Physics Department, CERN, CH-1211 Genève 23, Switzerland

###### Abstract

We examine the spectrum of bremsstrahlung photons that results from the
stopping of the initial net charge distributions in ultrarelativistic nucleus-
nucleus collisions at the CERN Large Hadron Collier (LHC). This effect has
escaped detection so far since it becomes sizable only at very low transverse
momentum and at sufficiently forward rapidity. We argue that it may be within
reach of the next-generation LHC heavy-ion detector ALICE-3 that is currently
under study, and we comment on the physics motivation for measuring it.

††preprint: CERN-TH-2021-106

##  I Introduction

In ultrarelativistic nucleus-nucleus collisions, bremsstrahlung of soft
photons at forward rapidity traces the deceleration of incoming charges and is
insensitive to the subsequent dynamical evolution. This makes it suited for
constraining the initial conditions of the longitudinal net charge
distribution. The idea of testing stopping via bremsstrahlung is as old as
heavy-ion phenomenology Kapusta:1977zb ; Bjorken:1984sp ; Dumitru:1993ph . In
the late 1990s, calculations of classical electromagnetic bremsstrahlung
indicated that the expected effects are measurable in an experimentally
accessible kinematic regime and that they could allow one to distinguish
between different stopping scenarios Jeon:1998tq ; Kapusta:1999hb ;
Wong:2000hka . This prompted studies for a dedicated forward detector at the
BNL Relativistic Heavy Ion Collider (RHIC) Jeon:1998tq  which, however, was
not realized. As of today, forward bremsstrahlung from stopping of incoming
charges remains a generally expected physics effect that has never been
measured experimentally in heavy-ion collisions.

One decade into the heavy-ion program at the CERN Large Hadron Collider (LHC),
the ALICE Collaboration is currently investigating the physics opportunities
of a next-generation TeV-scale heavy-ion detector that is based on ultrathin
silicon technology Adamova:2019vkf . Besides high rate capabilities and
excellent particle identification, this detector concept promises experimental
access to observables at unprecedentedly low transverse momentum [$p_{T}\sim
O(10\,{\rm MeV})$] and up to very forward rapidity ($y=4$ or $y=5$), including
prospects for soft and ultrasoft photon measurements. The present paper aims
at initiating a discussion about the measurability of bremsstrahlung from
stopping with this future heavy-ion collision experiment. To this end, we
calculate the expected photon spectrum within the nominal acceptance of such a
future detector and we find that coverage in the range $10\,{\rm
MeV/c}<p_{T}<100\,{\rm MeV/c}$ should give access to a sizable yield. While a
full assessment of the experimental feasibility of such a bremsstrahlung
measurement lies outside the scope of the present study and will depend on
evolving detector studies, we shall find that the dominant photon background
from meson decays has characteristically different distributions in transverse
momentum and centrality. This should facilitate experimental strategies to
isolate the effect.

In the context of a next-generation heavy-ion experiment at the LHC
Adamova:2019vkf , the prospects for soft and ultrasoft photon measurements
have been discussed recently in the context of Low’s theorem Low:1958sn . This
theorem formally relates hadronic multiparticle production amplitudes without
photons to expectation values for soft photon production by dressing all
electrically charged in- and out-going lines of multiparticle production
amplitudes with soft photon emissions. Recent theoretical interest in these
soft theorems arises from relating them to symmetries that reflect the
infrared structure of gravity and gauge theory Lysov:2014csa . Low’s theorem
is a general quantum formulation of soft bremsstrahlung. On general grounds,
one expects that it interpolates as a function of resolution scale between the
incoherent and the totally coherent limits of multiphoton bremsstrahlung. A
classical formulation should apply for sufficiently long wavelength when the
entire system of charge $2\,Z$ acts coherently as a single emitter whose
internal structure is not resolved by the emitted photons. Here, we work
within the classical formalism of Jackson:1998nia  used previously in
Jeon:1998tq ; Kapusta:1999hb ; Wong:2000hka , and we check that this condition
is met. The characteristic $1/p_{T}$ divergence of photon bremsstrahlung is
captured by this classical formulation, and our calculation addresses the
question Adamova:2019vkf  at which $p_{T}$-scale it will become experimentally
accessible.

Figure 1: Toy models of the final rapidity distribution of net charge in a
nucleus-nucleus collision: i) if the collision were fully transparent, charges
in the outgoing state would stay at incoming projectile rapidities $y_{0}$ and
$-y_{0}$. ii) in a Bjorken boost-invariant scenario, the charge distribution
is flat iii) the case that charges are fully stopped in the center-of-mass
frame of the collision.

##  II Modelling the longitudinal charge distribution

Incoming nuclear projectiles of charge $Z$ represent charge currents
$J^{(in)}_{\pm}({\bf x},t)$ that propagate along the beam direction $z$ with
velocities $\pm v_{0}=\pm\tanh y_{0}$ set by the beam rapidity $y_{0}$ which
for all practical purposes can be identified with $\ln\left(\frac{\sqrt{s_{\rm
NN}}}{m_{N}}\right)$,

$J^{(in)}_{\pm}({\bf x},t)=\pm Z\,e\,v_{0}\,\rho_{\rm in}(r)\,\delta(z\mp
v_{0}t)\,\Theta(-t)\,.$ (1)

Here, ${\bf x}=({\bf r},z)$ and we denote by $\rho_{\rm in}(r)$ the incoming
charge density in the plane transverse to the beam. After the collision at
time $t=0$, charges are partially stopped, _i.e._ , they propagate with
velocities $v(y)$ satisfying $-v_{0}<v(y)<v_{0}$. The outgoing current takes
the form

$J^{(out)}({\bf x},t)=\Theta(t)\,\int_{-y_{0}}^{y_{0}}\rho({\bf
r},y,t)\,v(y)\,\delta\left(z-v(y)t\right)\,dy\,,$ (2)

where the charge density $\rho({\bf r},y,t)$ is differential with respect to
rapidity $y$. This density is normalized to the entire charge deposited in the
collision region, $\int d{\bf r}\,\int_{-y_{0}}^{y_{0}}dy\,\rho({\bf
r},y,t)=2\,Z\,e$. After the start of the collision at $t=0$, charges will be
decelerated rapidly and they will be stopped in their final rapidity window
after a very short time $t_{s}$, _i.e._ , $\rho_{\rm out}({\bf r},y)=\rho({\bf
r},y,t)|_{t>t_{s}}$. We work in natural units ($c=\hbar=1$).

The intensity and number of photons of energy $\omega$ radiated into the
angular opening $d\Omega$ can be obtained from the classical bremsstrahlung
formula Jackson:1998nia

$\frac{d^{2}I}{d\omega d\Omega}=\omega\frac{d^{2}N}{d\omega
d\Omega}=|\mathbf{A}|^{2}\,,$ (3)

where $\mathbf{A}$ is defined in terms of the current
$\mathbf{J}(\mathbf{x},t)$ and the direction ${\bf n}$ of the outgoing photon

$\mathbf{A}(\mathbf{n},\omega)=-\frac{i\omega}{2\pi}\int dt\int
d^{3}x\,\mathbf{n}\times(\mathbf{n}\times\mathbf{J}(\mathbf{x},t))e^{i\omega(t-\mathbf{n}\cdot\mathbf{x})}\,.$
(4)

The current sums over all in- and outgoing contributions

$\mathbf{J}=\mathbf{J}^{(in)}_{+}+\mathbf{J}^{(in)}_{-}+\mathbf{J}^{(out)}\,.$
(5)

For the problem under consideration, $\mathbf{J}$ points always along the beam
direction $\vec{e}_{z}$ and the direction of
$\mathbf{n}=\vec{e}_{\varphi}\sin\theta+\vec{e}_{z}\cos\theta$ with
$\vec{e}_{\varphi}\perp\vec{e}_{z}$ lying in the transverse plane. For the
discussion of experimental acceptances, it is useful to convert into
pseudorapidity

$\eta=-\ln\left[\tan\left(\tfrac{\theta}{2}\right)\right]\,.$ (6)

The current entering (4) depends on the outgoing charge distributions
$\rho({\bf r},y,t)$. It is in this way that classical bremsstrahlung becomes a
tool for constraining initial longitudinal conditions.

We consider first the simple scenarios depicted in Fig. 1:

  1. 1.

Full transparency: charges are not decelerated,

$\frac{d^{2}I}{d\omega d\Omega}=0\,.Eq.$ (7)

  2. 2.

Bjorken-stopping:

$\rho({\bf r},y,t)=\frac{Z\,e}{y_{0}}\,\rho_{\rm in}({\bf
r})\,\Theta(t)\,\Theta(y_{0}-|y|)\,.$ (8)

  3. 3.

Landau-stopping:

$\rho({\bf r},y,t)=2\,Z\,e\,\rho_{\rm in}({\bf r})\,\Theta(t)\,\delta(y)\,.$
(9)

To appreciate the usefulness of such simple models, let us consider briefly
the _hypothetical scenario_ of a bell-shaped charge rapidity distribution
$\rho_{\rm out}(y)$ which would amount to less (more) charge deceleration than
Eq. (9) [Eq. (8)], respectively. One therefore expects that the bremsstrahlung
spectrum of this bell-shaped distribution is bracketed by the cases of Eqs.
(8) and (9). A minimal prerequisite for being sensitive to bremsstrahlung from
stopping is then that the scenario of (9) can be distinguished from the null-
hypothesis (7), and a measurement that can disentangle the scenarios (8) and
(9) demonstrates sensitivity to distinguish between different conceivable
stopping scenarios.

In the simple scenarios of Eqs. (8) and (9), charges are assumed to be stopped
instantaneously and stopping is independent of radial position. In general,
charges will decelerate over a finite time $\Delta t_{f}$ and they may
decelerate differently at different radial positions. There is, however, a
simple parametric reason for why these details should have a negligible effect
on bremsstrahlung radiation: Pb ions at LHC have a gamma-factor $\gamma\approx
2700$ which makes them Lorentz-contracted pancakes of longitudinal thickness
$\approx 0.005$ fm in the rest-frame of the collision. Any stopping must be
completed before the charges have traversed the other nucleus, _i.e._ , it
must be completed within a time $\approx 0.005$ fm/c. To be sensitive to the
detailed time- and/or position dependence of stopping, _forward_
bremsstrahlung photons would have to resolve this Lorentz-contracted
thickness. However, this is not possible with the photon energies $\omega$
that we consider in the following and for which $1/\omega\gg 0.005$ fm/c.
Consistent with this simple parametric argument, we have found for $\omega<2$
GeV only very small ($<5\%$) numerical differences between the sophisticated
position- and time-dependent stopping scenario with Bjorken boost-invariant
final charge distribution considered in Ref. Kapusta:1999hb , and the
simplified model (8) considered here (data not shown).

We note that this argument applies only to photons at sufficiently forward
rapidity that would need to resolve the strongly Lorentz-contracted
_longitudinal_ structure of the nucleus. In contrast, if emitted at central
rapidity (e.g., emitted close to mid-rapidity $\theta=90^{\circ}$), a
$\omega=200$ MeV photon resolves $O(1)$ fm distances in the _transverse_
direction in which the nucleus is not Lorentz-contracted. Therefore, photon
emission around central mid-rapidity is expected to be sensitive to the
internal structure of the charge distribution, while soft photon emission at
forward rapidity is expected to be described by the classical formulation
recalled here.

##  III Numerical results

In the last section, we gave qualitative arguments for how the classical
bremsstrahlung (3) from the incoming and outgoing charge currents (1), (2)
provides insight into the longitudinal rapidity distribution of net charges in
the _initial stage_ of the collision. In this section, we discuss the
corresponding spectra and we provide numerical results.

Both the Bjorken and the Landau stopping scenarios in Eqs. (8) and (9) lead to
charge distributions for which the dependence on transverse radius and
rapidity factorizes, $\rho_{\rm out}({\bf r},y)=\rho_{\rm in}({\bf
r})\,\rho(y)$. Inserting this ansatz into (2), one finds

$\displaystyle\frac{d^{2}I}{d\omega d\Omega}$ $\displaystyle=$
$\displaystyle\frac{\alpha
Z^{2}}{4\pi^{2}}\sin^{2}\theta\left|F(\omega\sin\theta)\right|^{2}$ (10)
$\displaystyle\times\left|\left[\int
dy\frac{v(y)\rho(y)}{1-v(y)\cos\theta}-\frac{2v_{0}^{2}\cos\theta}{1-v_{0}^{2}\cos^{2}\theta}\right]\right|^{2},$

where $v(y)=\tanh(y)$ and where $F$ denotes the transverse nuclear form
factor,

$F(\omega\sin\theta)=\int d^{2}r_{\perp}\,\rho_{\rm
in}\left(r_{\perp}\right)e^{-i\omega{\bf n}\cdot{\bf r}_{\perp}}\,.$ (11)

In the product $\rho_{\rm out}({\bf r},y)=\rho_{\rm in}({\bf r})\,\rho(y)$, we
take the longitudinal rapidity distribution normalized to 2, so that the
scenarios discussed in Sec. II and sketched in Fig.1 correspond to

$\rho(y)=\left\\{\begin{array}[]{cc}\delta(y-y_{0})+\delta(y+y_{0})&\hbox{(transparent)}\\\
\tfrac{1}{y_{0}}\Theta(y_{0}-|y|)&\hbox{(Bjorken)}\\\
2\delta(y)&\hbox{(Landau)}\\\
\tfrac{2}{\sigma\sqrt{2\pi}}\exp\left[-\tfrac{y^{2}}{2\sigma^{2}}\right]&\hbox{(Gaussian)}\end{array}\right.$
(12)

For the simplifying assumption that charges in heavy nuclei are distributed
homogenously in a sphere of radius $R$, the form factor depends only on the
dimensionless scale $q\equiv\omega R\sin\theta$ Kapusta:1999hb

$\displaystyle F(q)$ $\displaystyle=$
$\displaystyle\frac{3}{q^{2}}\left(\frac{\sin q}{q}-\cos
q\right)\quad\hbox{[fixed sphere]}\,,$ $\displaystyle q\equiv\omega
R\sin\theta\,.$

Equation (11) could be evaluated easily for more refined distributions, such
as e.g. a Woods-Saxon-distribution. However, as argued above, mild differences
in transverse profiles should not affect longitudinal bremsstrahlung spectra
significantly and we therefore prefer to work with the simple analytic formula
(III).

Figure 2: The form factor (III) of the transverse nuclear charge distribution
as a function of $q\equiv\omega R\sin\theta$.

The form factors (11) satisfy by construction $\lim_{q\rightarrow 0}\,F(q)=1$.
For the case of a fixed sphere, Eq. (III), the corresponding form factor
$F(q)$ is plotted in Fig. 2. The characteristic fall-off properties of $F(q)$
(such as $F(q)\geq 0.9$ for $q<1$ and $F(q)<0.1$ for $q>4$) can be expected to
hold for a broad class of realistic charge distributions. In the following, we
are interested in forward bremsstrahlung. For a Pb nucleus with $R=6.8$ fm and
for pseudorapidity $\eta=3$ ($\eta=5$), the condition $q<1$ translates into
$\omega<1/(R\sin\theta)=1.5\tfrac{1}{\rm fm}\approx 300\,{\rm MeV}$
($\omega<2.2$ GeV), respectively. In this kinematic regime, on which our
discussion will focus, the squared form factor in (10) corresponds therefore
to a mild (20 % or less) deviation from unity.

|  |
---|---|---
Figure 3: The double-differential photon energy distribution $\tfrac{d^{2}I}{d\omega\,d\eta}$ for different photon energies $\omega$ as a function of pseudorapidity $\eta$. Results are shown for the three different stopping scenarios (12) in central PbPb collision at $\sqrt{s_{\rm NN}}=5.02$ TeV.  |  |
---|---|---
Figure 4: Same as Fig. 3, but for the stopping scenarios of Eq. (16).

###  III.1 The pseudorapidity distribution

We start by discussing the photon energy distribution

$\frac{d^{2}I}{d\omega d\eta}=\frac{d^{2}I}{d\omega\sin\theta d\theta
d\phi}\,2\pi\,\sin^{2}\theta$ (14)

which we obtain from Eq. (10) by integrating over the azimuth $d\phi$, using

$\frac{d\theta}{d\eta}=\frac{1}{\cosh\eta}=\sin\theta\,.$ (15)

Figure 3 plots Eq. (14) for different values of photon energy $\omega$ as a
function of pseudorapidity. We first discuss the kinematic region of very
forward pseudorapidity ($\eta>8$, say), before turning to the features seen at
smaller $\eta$.

In $\sqrt{s_{\rm NN}}=5.02$ TeV Pb-Pb collisions, the projectile rapidity is
$y_{0}=\ln\left(\frac{\sqrt{s_{\rm NN}}}{m_{N}}\right)=8.5$. As seen from Fig.
3, the bremsstrahlung energy distribution $\frac{d^{2}I}{d\omega d\eta}$
extends unattenuated up to comparable values of pseudorapidity $\eta$.
However, pseudorapidity
$\eta=-\ln\left[\tan\left(\tfrac{\theta}{2}\right)\right]$ measures a polar
angle, and a small amount of photon bremsstrahlung is emitted at any
arbitrarily small forward angle $\theta$. This is the reason for why there is
energy at $\eta>8.5$, though the energy decreases rather sharply with
increasing $\eta$.

At relatively high photon energy $\omega=1$ GeV and relatively low
pseudorapidity, Fig. (3) displays two peculiar dips in the energy distribution
around $\eta=2.2$ and $\eta=2.8$. As we discuss now, these are artifacts of
our simple fixed sphere model (III) for the charge distribution, and these
artifacts may help to illustrate the range of validity of our calculation. To
clarify this point, we recall our comments about Fig. 2: the simple form
factor $F(q)$ used in our calculation is expected to have a very small model-
dependence for $q<1$, but it will be completely model-dependent for $q>4$
where it shows peculiar zero-crossings. Indeed, for $\omega=1$ GeV, $R=6.8$ fm
the Pb radius used in our calculation and $q=4$, we find
$\sin\theta=\tfrac{q}{\omega R}=\tfrac{4}{5\,\times 6.8}=0.12$ which
corresponds exactly to $\eta=2.8$. The dips seen in Fig. 3 for $\omega=1$ GeV
are in one-to-one correspondence with the zero-crossings of $F(q)$ for $q>4$.
It also follows from $\sin\theta=\tfrac{q}{\omega R}$ that for $\omega=1$ GeV,
values $q<1$ correspond to $\eta>4.2$. This is the region in which we expect
our calculation to yield physical results.

For softer photon energies, $\omega=200$ MeV say, the same argument implies
that a dip should show up at $\eta=1.1$. This dip exists but it is not
displayed in Fig. 3, since we plot only for $\eta>2$. At this lower photon
energy, $q<1$ corresponds to $\eta>2.6$. In general, the softer the photon
energy, the less sensitive is our calculation to geometrical details and the
more it can be trusted over a wide range of pseudorapidity.

|  |
---|---|---
Figure 5: The $p_{T}$-dependence of the photon bremsstrahlung distribution
$\int_{\eta_{-}}^{\eta_{+}}\frac{d^{2}I}{dp_{T}d\eta}d\eta$, integrated over
different windows $\left[\eta_{-},\eta_{+}\right]$ of pseudorapidity. The
shape of the $p_{T}$-distribution remains almost unchanged, but the yield
increases steeply towards forward rapidity.

Within the range of forward pseudorapidity in which our calculation is
expected to be model-independent ($\eta>3$ or $\eta>4$, depending on photon
energy), the three models of charge stopping displayed in Fig. 3 lead to
energy distributions that are numerically different and that differ in their
rapidity dependence. To illustrate the physics behind these differences, it is
useful to introduce another class of stopping scenarios, in which all net
charges are shifted by the same fixed number of units $y_{\rm shift}$ in
rapidity,

$\rho(y)=\delta\left(y-(y_{0}-y_{\rm
shift})\right)+\delta\left(y+(y_{0}-y_{\rm shift})\right)\,.$ (16)

From the resulting energy distributions in Fig. 4, we conclude that for a
stopping scenario (16), the soft photon energy distribution at fixed $\omega$
forms a plateau within the pseudorapidity range $y_{0}-y_{\rm
shift}\lesssim\eta\lesssim y_{0}$. The more the net charge is stopped, the
more the radiation extends towards mid-rapidity. Any model of longitudinal
stopping that is described by a continuous function $\rho(y)$ may be viewed as
a linear superposition of distributions (16). This explains why for
sufficiently soft photons, the energy distributions in Fig. 3 is flat for the
Landau case, but rises with increasing $\eta$ for models with continuous final
longitudinal charge distribution $\rho(y)$. In this sense, the
$\eta$-distribution of bremsstrahlung photons monitors the rapidity-dependence
of stopped charges.

So far, we have discussed bremsstrahlung in terms of a double-differential
distribution in $\omega$ and $\eta$. To discuss issues of experimental
acceptance and measurability, it is preferable to switch to $p_{T}$ and
$\eta$,

$\frac{d^{2}I}{dp_{T}d\eta}=\frac{d^{2}I}{d\omega
d\Omega}\,2\pi\,\sin^{2}\theta\,\cosh\eta\,.$ (17)

Figure 5 shows the corresponding $p_{T}$-differential photon energy
distribution integrated over $\eta\in\left[\eta_{-};\eta_{+}\right]$,
$\int_{\eta_{-}}^{\eta_{+}}\frac{d^{2}I}{dp_{T}d\eta}d\eta$. This plot makes
it clear that experimental access to bremsstrahlung photons requires
acceptance for $p_{T}<100$ MeV. This is so irrespective of pseudorapidity.

Integrating the spectra in Fig. 5 over $p_{T}$, we find that for the Landau
stopping scenario, a total of 64 GeV (23.5 GeV, 8.5 GeV) energy is radiated
per central Pb-Pb collision into the phase space region $10\,{\rm
MeV}<p_{T}<100\,{\rm MeV}$ and $\eta\in\left[4,5\right]$ ($\left[3,4\right]$,
$\left[2,3\right]$), respectively. For the other stopping scenarios plotted in
Fig. 5, the total energy radiated into these three phase space regions is
accordingly smaller (47, 13 and 3 GeV for the Gaussian scenario, and 18, 4 and
0.8 GeV for the Bjorken scenario).

###  III.2 The photon number distribution

We finally translate the results shown above into the number of photons
radiated per unit phase space,

$\frac{d^{2}N}{dp_{T}d\eta}=\frac{1}{p_{T}\,\cosh\eta}\frac{d^{2}I}{dp_{T}d\eta}\,.$
(18)

Depending on the stopping scenario, we find between 5 and 20 photons per unit
pseudorapidity in the range $p_{T}\in\left[10\,{\rm MeV};20\,{\rm
MeV}\right]$, see Fig. 6. With increasing $p_{T}$, the number of
bremsstrahlung photons decreases, and in the $p_{T}$-bin $\left[50\,{\rm
MeV};60\,{\rm MeV}\right]$, we find between 0.5 and 5 bremsstrahlungs photons
per unit pseudorapidity and per event.

We find that for $p_{T}\lesssim 40$ MeV, the photon number spectrum in Fig. 6
follows the characteristic $dN/dp_{T}\propto 1/p_{T}$ dependence of soft
photon radiation while it decays somewhat more steeply at higher $p_{T}$.

|  |
---|---|---

|  |
---|---|---

Figure 6: Upper panel: the differential photon number spectrum (18) as a
function of transverse momentum for different pseudorapidity bins. Lower
panel: the corresponding number of photons in bins of 10 MeV in transverse
momentum.

###  III.3 “Background” photons

The question of whether and how bremsstrahlung photons can be disentangled
from other sources of photons requires detailed event generator studies that
lie outside the scope of the present exploratory calculations. Here, we
restrict ourselves to some qualitative considerations:

Photons from $\pi^{0}$-decays are expected to provide the most important
background. Also $\eta$-mesons have a branching ratio of $40\%$ into two
photons and need to be included in a realistic cocktail.111In addition to
resonance decays, forward photons are also radiated off electron-positron
pairs produced in the colliding Weizsäcker-Williams fields Hencken:1999xw .
The centrality dependence of this contribution is different from that of Eq.
(10). It is known that in PbPb collisions at the LHC, these mesons have the
same nuclear modification factor as charged pions in the range $1\,{\rm
GeV}<p_{T}<20\,{\rm GeV}$ and around mid-rapidity ALICE:2018mdl . For the
following simple estimates, we therefore assume that the $\pi^{0}$\- and the
$\pi^{\pm}$-distributions are the same for all rapidities and for all
transverse momenta.

At mid-rapidity, the $p_{T}$-differential charged pion spectrum
$dN^{\pi^{\pm}}/dp_{T}\,dy$ is approximately constant,
$dN^{\pi^{\pm}}/dp_{T}\,dy\simeq 2000/{\rm GeV}$ for $p_{T}<500$ MeV at $y=0$
(see e.g. Fig.21 of Ref. Nijs:2020roc  which replots data from ALICE:2014juv
). In central PbPb collisions at the LHC, $dN_{\rm ch}/d\eta$ decreases by
almost a factor two from $\eta=0$ to $\eta=5$ ALICE:2013jfw . We therefore
expect $dN^{\pi^{0}}/dp_{T}\,dy\simeq 500/{\rm GeV}$ for $p_{T}<500$ MeV and
$4<\eta<5$. This amounts to an approximately flat, $p_{T}$-independent
distribution of five $\pi^{0}$s per event and per 10 MeV-bin, to be compared
to a steeply falling $p_{T}$-distribution of a comparable number of
bremsstrahlung photons in the range $p_{T}<100$ MeV, see Fig. 6. Our simple
considerations thus indicate two characteristic differences between
bremsstrahlung photons and “background photons”

  1. 1.

_characteristically different $p_{T}$-dependence_
The $dN/dp_{T}$-spectrum of bremsstrahlung photons falls off $\propto 1/p_{T}$
or steeper in the range $p_{T}<100$ MeV. In contrast, light mesons that decay
into photons have an approximately flat, $p_{T}$-independent $dN/dp_{T}$
distribution in a wider $p_{T}$-range (up to $p_{T}<500$ MeV, say).
Bremsstrahlung photons should thus be visible as a characteristic low-$p_{T}$
enhancement above a smooth almost $p_{T}$-independent baseline.

  2. 2.

_characteristically different centrality-dependence_
The yield of bremsstrahlung photons increases with $Z^{2}$, see Eq. (10). For
non-central collisions, $Z$ should be regarded as the number of stopped
charges, _i.e._ , $Z^{2}\propto N^{2}_{\rm part}$. On the other hand, soft
hadron multiplicity is known to grow proportionally to $N_{\rm part}$. The
yield of bremsstrahlung photons therefore increases parametrically faster
towards mid-rapidity than the yield of soft hadrons that decay into
“background” photons.

While these qualitative considerations cannot replace a realistic modeling of
meson distributions and their photon decay kinematics, they suggest that there
are experimental handles to separate bremsstrahlung photons from other
“background” sources.

##  IV Conclusion

The ALICE collaboration plans to develop a new detector (ALICE-3) with
experimental acceptance in a previously uncharted, ultrasoft regime $10\,{\rm
MeV}<p_{T}<100\,{\rm MeV}$ and up to relatively forward pseudorapidity. As
demonstrated here (Fig. 6), photon bremsstrahlung due to stopping of the
incoming net charge distributions is an expected phenomenon that leaves
characteristic signatures in this newly accessible experimental regime. As
such, it is useful for illustrating the novel opportunities of a detector
design with ultrasoft acceptance. Its centrality- and $p_{T}$-dependence is
characteristically different from that of expected backgrounds.

Historically, the physics motivation for measuring bremsstrahlung photons is
to characterize the longitudinal dynamics of stopping. As demonstrated here
(Figs. 3 and 4), different stopping scenarios yield bremsstrahlung spectra
with characteristically different pseudorapidity distribution and yield. If
the pseudorapidity distribution of net charge became directly measurable with
a future detector, one would have access to two experimentally challenging but
complementary signatures of the same stopping phenomenon, thus allowing for
much-wanted cross checks.

Understanding bremsstrahlung photons is also of relevance in searches for
imprints of other conceivable phenomena that would show up in the ultrasoft
regime. For instance, unlike the situation in charged hadron distributions
where Coulomb repulsion counteracts Bose-Einstein enhancement, ultrasoft
$\pi^{0}$-yields seem ideally suited to test the quantum statistics of Cooper-
Frye freeze-out distributions Cooper:1974mv . While we are currently not in a
position to quantify these or other effects, we enjoy speculating about a
future in which photon bremsstrahlung from stopping is not only measured but
needs to be included in the baseline for searches of other conceivable
phenomena in the ultrasoft regime.

Acknowledgements. We thank Federico Antinori, Maximilian Attems, Peter Braun-
Munzinger, Jasmine Brewer, Adrian Dumitru, Stefan Floerchinger, Joe Kapusta,
Aleksas Mazeliauskas, Johanna Stachel and Wilke van der Schee for useful
questions and discussions during the preparation of this paper.

## References

  * (1) J. I. Kapusta, Phys. Rev. C 15 (1977), 1580-1582 doi:10.1103/PhysRevC.15.1580
  * (2) J. D. Bjorken and L. D. McLerran, Phys. Rev. D 31 (1985), 63 doi:10.1103/PhysRevD.31.63
  * (3) A. Dumitru, L. D. McLerran, H. Stoecker and W. Greiner, Phys. Lett. B 318 (1993), 583-586 doi:10.1016/0370-2693(93)90457-S
  * (4) S. Jeon, J. I. Kapusta, A. Chikanian and J. Sandweiss, Phys. Rev. C 58 (1998), 1666-1670 doi:10.1103/PhysRevC.58.1666 [arXiv:nucl-th/9806047 [nucl-th]].
  * (5) J. I. Kapusta and S. M. H. Wong, Phys. Rev. C 59 (1999), 3317 doi:10.1103/PhysRevC.59.3317 [arXiv:hep-ph/9903235 [hep-ph]].
  * (6) S. M. H. Wong, M. Belkacem, J. I. Kapusta, S. A. Bass, M. Bleicher and H. Stoecker, Phys. Rev. C 63 (2001), 014903 doi:10.1103/PhysRevC.63.014903 [arXiv:hep-ph/0008119 [hep-ph]].
  * (7) D. Adamová, G. Aglieri Rinella, M. Agnello, Z. Ahammed, D. Aleksandrov, A. Alici, A. Alkin, T. Alt, I. Altsybeev and D. Andreou, et al. [arXiv:1902.01211 [physics.ins-det]].
  * (8) F. E. Low, Phys. Rev. 110 (1958), 974-977 doi:10.1103/PhysRev.110.974
  * (9) V. Lysov, S. Pasterski and A. Strominger, Phys. Rev. Lett. 113 (2014) no.11, 111601 doi:10.1103/PhysRevLett.113.111601 [arXiv:1407.3814 [hep-th]].
  * (10) J. D. Jackson, Classical Electrodynamics (Wiley, New York, 1998).
  * (11) S. Acharya et al. [ALICE], Phys. Rev. C 98 (2018) no.4, 044901 doi:10.1103/PhysRevC.98.044901 [arXiv:1803.05490 [nucl-ex]].
  * (12) K. Hencken, D. Trautmann and G. Baur, Phys. Rev. C 60 (1999), 034901 doi:10.1103/PhysRevC.60.034901 [arXiv:nucl-th/9903019 [nucl-th]].
  * (13) E. Abbas et al. [ALICE], Phys. Lett. B 726 (2013), 610-622 doi:10.1016/j.physletb.2013.09.022 [arXiv:1304.0347 [nucl-ex]].
  * (14) G. Nijs, W. van der Schee, U. Gürsoy and R. Snellings, Phys. Rev. C 103 (2021) no.5, 054909 doi:10.1103/PhysRevC.103.054909 [arXiv:2010.15134 [nucl-th]].
  * (15) B. B. Abelev et al. [ALICE], Phys. Lett. B 736 (2014), 196-207 doi:10.1016/j.physletb.2014.07.011 [arXiv:1401.1250 [nucl-ex]].
  * (16) F. Cooper and G. Frye, Phys. Rev. D 10 (1974), 186 doi:10.1103/PhysRevD.10.186