# Mixture of A Million Experts

Xu Owen He<EMAIL_ADDRESS>
Google DeepMind

###### Abstract

The feedforward (FFW) layers in standard transformer architectures incur a
linear increase in computational costs and activation memory as the hidden
layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged
as a viable approach to address this issue by decoupling model size from
computational cost. The recent discovery of the fine-grained MoE scaling law
shows that higher granularity leads to better performance. However, existing
MoE models are limited to a small number of experts due to computational and
optimization challenges. This paper introduces PEER (parameter efficient
expert retrieval), a novel layer design that utilizes the product key
technique for sparse retrieval from a vast pool of tiny experts (over a
million). Experiments on language modeling tasks demonstrate that PEER layers
outperform dense FFWs and coarse-grained MoEs in terms of performance-compute
trade-off. By enabling efficient utilization of a massive number of experts,
PEER unlocks the potential for further scaling of transformer models while
maintaining computational efficiency.

(a) $6e18$ FLOPs

(b) $2e19$ FLOPs

Figure 1: Isoflop comparison on the C4 dataset between PEER and other
baselines with two different FLOP budgets ($6e18$ and $2e19$ FLOPs). The $x$
axis is in log scale.

##  1 Introduction

The past few years have seen the power of scaling (Kaplan et al., 2020;
Hoffmann et al., 2022): increasing the number of parameters, amount of
training data, or the computational budget has proven to be a reliable way to
improve model performance. Notably, feedforward (FFW) layers, responsible for
storing factual knowledge (Geva et al., 2021; Dai et al., 2022), account for
two-thirds of the total parameters in a transformer. However, one drawback of
these dense FFWs is that their computational footprint (FLOPs and device
memory consumption) is linearly proportional to their parameter count.

To break the coupling between computational cost and parameter count, many
recent works (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2022;
Zhou et al., 2022) have adopted the Mixture-of-Experts (MoE) architecture,
which uses a set of sparsely activated expert modules (often FFWs) in place of
a single dense FFW. Clark et al. (2022) studied the scaling law of MoE
language models and showed that increasing the number of experts is an
effective way to improve performance without increasing the inference cost.
However, their experiments showed that the efficiency gains provided by MoEs
plateau after a certain model size is reached. More recently, Krajewski et al.
(2024) discovered that this plateau was caused by using a fixed number of
training tokens. When the number of training tokens is compute-optimal, MoEs
consistently outperform dense models in terms of FLOP efficiency. Moreover,
they introduced granularity (the number of active experts) as a new scaling
axis and empirically showed that using higher granularity improves
performance. Extrapolating this fine-grained MoE scaling law suggests that
continued improvement of model capacity will ultimately lead to a large model
with high granularity, corresponding to an architecture of an immense number
of tiny experts.

Beyond efficient scaling, another reason to have a vast number of experts is
lifelong learning, where MoE has emerged as a promising approach (Aljundi et
al., 2017; Chen et al., 2023; Yu et al., 2024; Li et al., 2024). For instance,
Chen et al. (2023) showed that, by simply adding new experts and regularizing
them properly, MoE models can adapt to continuous data streams. Freezing old
experts and updating only new ones prevents catastrophic forgetting and
maintains plasticity by design. In lifelong learning settings, the data stream
can be indefinitely long or never-ending (Mitchell et al., 2018),
necessitating an expanding pool of experts.

Although both efficient scaling and lifelong learning require MoE designs
capable of handling a vast number of experts, to the best of our knowledge,
the only architecture supporting more than ten thousands of experts is the
Mixture of Word Experts (MoWE) (dos Santos et al., 2023). However, MoWE is
language-specific and uses a fixed routing scheme. Theoretical and empirical
evidence (Clark et al., 2022; Dikkala et al., 2023) highlights the advantages
of learned routers over non-trainable ones. Thus, an MoE design with a learned
router scalable to over a million experts remains an open area for
exploration.

This work introduces the Parameter Efficient Expert Retrieval (PEER)
architecture, leveraging product key retrieval (Lample et al., 2019) for
efficient routing to an extremely large number of experts, decoupling
computational cost from parameter count. This design demonstrates a superior
compute-performance trade-off in our experiments, positioning it as a
competitive alternative to dense FFW layers for scaling foundation models. The
main contributions of this work are:

  * •

Exploration of Extreme MoE Setting: Deviating from the focus on a small number
of large experts in previous MoE research, this work investigates the under-
explored case of numerous tiny experts.

  * •

Learned Index Structure for Routing: Demonstrating for the first time that a
learned index structure (Kraska et al., 2018) can efficiently route to over a
million experts.

  * •

New Layer Design: Combining product key routing with single-neuron experts, we
introduce the PEER layer that expands layer capacity without significant
computational overheads. Empirical results demonstrate its superior efficiency
compared to dense FFW, coarse-grained MoEs and Product Key Memory (PKM)
layers.

  * •

Comprehensive Ablation Studies: We investigate the impact of different design
choices of PEER such as number of experts, active parameters, number of heads
and query batch normalization on language modeling tasks.

Figure 2: Illustration of the PEER layer. A PEER layer can be inserted in the
middle of a transformer backbone or can be used to replace FFW layers. Given
the state vector $x$ from the previous layer, a query network $q$ maps it to a
query vector $q(x)$, which is then compared with the product keys to compute
the router scores and to retrieve the top $k$ experts $e_{1},...,e_{k}$. After
the retrieved experts make their predictions $e_{i}(x)$, their outputs are
linearly combined using the softmax-normalized router scores as weights.

##  2 Method

In this section, we introduce the Parameter Efficient Expert Retrieval (PEER)
layer, which is a Mixture of Experts architecture using product keys (Lample
et al., 2019) in the router and single-neuron MLPs as experts. Fig. 2
illustrates the computational process within a PEER layer.

#### PEER Overview

Formally, a PEER layer is a function $f:\mathbb{R}^{n}\to\mathbb{R}^{m}$ that
consists of three parts: a pool of $N$ experts
$\mathbb{E}:=\\{e_{i}\\}_{i=1}^{N}$, where each expert
$e_{i}:\mathbb{R}^{n}\to\mathbb{R}^{m}$ shares the same signature as $f$, a
corresponding set of $N$ product keys
$\mathbb{K}:=\\{k_{i}\\}_{i=1}^{N}\subset\mathbb{R}^{d}$, and a query network
$q:\mathbb{R}^{n}\to\mathbb{R}^{d}$ that maps the input vector
$x\in\mathbb{R}^{n}$ to a query vector $q(x)$. Let $\mathcal{T}_{k}$ denote
the top-k operator. Given an input $x$, we first retrieve a subset of $k$
experts whose corresponding product keys have the highest inner products with
the query $q(x)$.

$\displaystyle\mathbb{I}$
$\displaystyle=\mathcal{T}_{k}\left(\\{q(x)^{T}k_{i}\\}_{i=1}^{N}\right)$ #
Retrieve top $k$ experts (1)

Then we apply nonlinear activations (such as softmax or sigmoid) to the query-
key inner products of these top $k$ experts to obtain the router scores.

$\displaystyle g_{i}(x)$ $\displaystyle=s(q(x)^{T}k_{i})$ # Compute router
scores (2)

Finally, we compute the output by linearly combining the expert outputs
weighted by the router scores.

$\displaystyle f(x)$
$\displaystyle=\sum\nolimits_{i\in\mathbb{I}}g_{i}(x)e_{i}(x)$ # Aggregate
expert outputs (3)

#### Product Key Retrieval

Since we intend to use a very large number of experts ($N\geq 10^{6}$),
naively computing the top $k$ indices in Eq. 1 can be very expensive. Hence we
apply the product key retrieval technique here. Instead of using $N$
independent $d$-dimensional vectors as our keys $k_{i}$, we create them by
concatenating vectors from two independent sets of $\frac{d}{2}$-dimensional
sub-keys $\mathbb{C},\mathbb{C}^{\prime}\subset\mathbb{R}^{\frac{d}{2}}$:

$\mathbb{K}=\\{\begin{bmatrix}c\\\
c^{\prime}\end{bmatrix}|c\in\mathbb{C},c^{\prime}\in\mathbb{C}^{\prime}\\}$
(4)

Note that here $\mathbb{C},\mathbb{C}^{\prime}$ have cardinality $\sqrt{N}$
and $c,c^{\prime}$ have dimensionality $\frac{d}{2}$. So in practice, we
choose $N$ to be a perfect square and $d$ to be an even number.

This Cartesian product structure of $\mathbb{K}$ allows us to find the top $k$
experts efficiently. Instead of comparing $q(x)$ to all $N$ keys in
$\mathbb{K}$ and selecting the top k matches, we can split the query vector
$q(x)$ into two sub-queries $q_{1}$ and $q_{2}$ and apply the top k operations
to the inner products between the sub-queries and sub-keys respectively:

$\mathbb{I}_{\mathbb{C}}=\mathcal{T}_{k}\left((q_{1}^{T}c_{i})\right),\quad\quad\mathbb{I}_{\mathbb{C}^{\prime}}=\mathcal{T}_{k}\left((q_{2}^{T}c_{j}^{\prime})\right)\\\
$ (5)

This results in a set of $k^{2}$ candidate keys
$\mathbb{K}^{\prime}:=\\{\begin{bmatrix}c_{i}\\\
c_{j}\end{bmatrix}|i\in\mathbb{I}_{\mathbb{C}},j\in\mathbb{I}_{\mathbb{C}}^{\prime}\\}$,
and it is mathematically guaranteed that the $k$ most similar keys to $q(x)$
from $\mathbb{K}$ are in this candidate set. Moreover, the inner product
between the candidate key and $q(x)$ is simply the sum of inner products
between the sub-keys and sub-queries: $q(x)^{T}\begin{bmatrix}c_{i}\\\
c_{j}\end{bmatrix}=q_{1}^{T}c_{i}+q_{2}^{T}c_{j}$. Hence we can apply the
top-k operator again to these $k^{2}$ inner products to get the top k matching
keys from the original set of product keys $\mathbb{K}$. As explained in
Lample et al. (2019). This reduces the complexity of top k expert retrieval in
Eq. 1 from $O(Nd)$ as done naively by exhaustive search to
$O((\sqrt{N}+k^{2})d)$.

#### Parameter Efficient Experts and Multi-Head Retrieval

Unlike other MoE architectures, which often set the hidden layer of each
expert to the same size as other FFW layers, in PEER, every expert $e_{i}$ is
a singleton MLP, in other words, it has only one hidden layer with a single
neuron:

$e_{i}(x):=\sigma(u_{i}^{T}x)v_{i}$ (6)

where $v_{i},u_{i}$ are not matrices but vectors with the same dimension as
$x$, and $\sigma$ is a nonlinear activation function such as ReLU or GELU. We
omit bias terms here for brevity.

Instead of varying the size of individual experts, we adjust the
expressiveness of a PEER layer by using multi-head retrieval, similar to the
multi-head attention mechanism in transformers and the multi-head memory in
PKMs. In particular, we use $h$ independent query networks instead of one,
each computes its own query and retrieves a separate set of $k$ experts.
However, different heads share the same pool of experts with the same set of
product keys. The outputs of these $h$ heads are simply summed up:

$f(x):=\sum_{i=1}^{h}f^{i}(x)=\sum_{i=1}^{h}\sum_{j\in\mathbb{I}^{i}}g_{j}(x)e_{j}(x)$
(7)

One can verify that when only one expert is retrieved ($k=1$) per head, using
a PEER layer with $h$ heads is the same as using one expert with $h$ hidden
neurons:

$f(x)=\sum_{i=1}^{h}e^{i}(x)=\sum_{i=1}^{h}\sigma(u_{i}^{T}x)v_{i}=V\sigma(W^{T}x);$
(8)

where $W=[u_{1},\cdots,u_{h}],V=[v_{1},\cdots,v_{h}]$. In other words, PEER
dynamically assembles an MLP with $h$ neurons by aggregating $h$ singleton
MLPs retrieved from a shared repository. Compared to existing MoE approaches
that use MLPs with multiple hidden neurons as experts, this design allows
shared hidden neurons among experts, enhancing knowledge transfer and
parameter efficiency.

Algorithm 1 shows a simplified implementation of the PEER forward pass,
storing parameter-efficient expert weights in embedding layers and combining
them with einsum operations. This implementation can be easily extended to
experts of the GLU variants (Shazeer, 2020) by adding additional linear gating
weights. In practice, an efficient implementation may require specialized
hardware kernels to accelerate embedding lookup and fusion with the einsum
operations.

#### Why A Large Number of Small Experts?

Given an MoE layer, we can characterize it by three hyperparameters: the total
number of parameters $P$, the number of active parameters per token
$P_{\text{active}}$ and the size of a single expert $P_{\text{expert}}$.
Krajewski et al. (2024) showed that the scaling law of MoE models has the
following form:

$\mathcal{L}(P,D,G)=c+(\frac{g}{G^{\gamma}}+a)\frac{1}{P^{\alpha}}+\frac{b}{D^{\beta}},$
(9)

where $\mathcal{L}$ is the final test loss, $a,b,g,\gamma,\alpha,\beta$ are
constants, $D$ is the total number of training tokens and the granularity $G$
is the number of active experts:

$G:=\frac{P_{\text{active}}}{P_{\text{expert}}}$ (10)

In order to improve model performance, we need to scale up $P,D,G$. On the
other hand, it is essential to limit $P_{\text{active}}$ because the
computational and memory costs are primarily determined by the active
parameters during training and inference. Notably, the memory footprint
corresponding to $P_{\text{active}}$ has to be multiplied by the number of
tokens in a batch, while the memory cost of $P$ is independent of the batch
size and sequence length because only one copy of the model needs to be
stored.

As a result, we want to increase $P,G$ but not $P_{\text{active}}$. Since the
expert size $P_{\text{expert}}=P_{\text{active}}/G$ and the number of experts
$N=P/P_{\text{expert}}=P\cdot G/P_{\text{active}}$, this implies that we
should decrease the size of each expert, $P_{\text{expert}}$, and increase the
number of experts $N$. Hence we need a large number of small experts.

In general, for experts that are MLPs with a single hidden layer.
$P_{\text{expert}}=(2d_{\text{model}}+1)d_{\text{expert}}$ and
$P_{\text{active}}=(2d_{\text{model}}+1)d_{\text{active}}$, where
$d_{\text{model}}$, $d_{\text{expert}}$ and $d_{\text{active}}$ are the hidden
dimension of the transformer, the number of hidden neurons used in one expert
and the total number of hidden neurons activated per token, respectively.

In the case of PEER, we use the smallest expert size possible by setting
$d_{\text{expert}}=1$, and the number of activated neurons is the number of
retrieval heads multiplied by the number of experts retrieved per head:
$d_{\text{active}}=hk$. Consequently, the granularity of PEER is always
$G=P_{\text{active}}/P_{\text{expert}}=d_{\text{active}}/d_{\text{expert}}=hk$.

⬇

1def peer_forward(self, x):

2 # Embedding layers storing the down/up projection weights of all experts

3 self.w_down_embed = nn.Embed(num_embeddings=self.n_experts,
features=self.d_model)

4 self.w_up_embed = nn.Embed(num_embeddings=self.n_experts,
features=self.d_model)

5

6 # Retrieve the weights of the top matching experts using product keys

7 # indices and scores have the shape ’bthk’, where h is the number of heads

8 indices, scores = self.get_indices(self.query_proj(x), self.sub_keys,
top_k=self.k)

9 w_down = self.w_down_embed(indices)

10 w_up = self.w_up_embed(indices)

11

12 # Compute weighted average of expert outputs

13 x = jnp.einsum(’btd, bthkd->bthk’, x, w_down)

14 x = self.activation(x)

15 x = x * nn.softmax(scores)

16 x = jnp.einsum(’bthk, bthkd->btd’, x, w_up)

17 return x

Algorithm 1: Pseudo code implementation of a PEER layer forward pass. An
example implementation of the get_indices and query_proj functions in Pytorch
can be found in Lample et al. (2021)

##  3 Experiments

###  3.1 Pretraining isoFLOP Analysis

We compare PEER with various baselines using isoFLOP analysis (Borgeaud et
al., 2022b). We chose a fixed FLOP budget ($6e18$ and $2e19$) and jointly
varied the model size and the number of training tokens from the C4 dataset
(Raffel et al., 2020) to obtain isoFLOP curves. Each point on an isoFLOP curve
has the same computational cost, and we plot them in terms of their model size
and final validation perplexity on C4.

For the dense baselines, we varied their size by changing the number of
layers, attention heads and model dimensions. For MoE, PKM and PEER methods,
we took each of the dense models considered and replaced the FFW layer in the
middle block (e.g. in a 12 block transformer, we replace the FFN in block 6)
by a layer of MoE, PKM and PEER, respectively.

In MoE, we used the expert-choice (Zhou et al., 2022) routing algorithm, which
effectively addresses the expert load imbalance issue and generally
outperforms token-choice MoEs (see Section 4 for a review and comparison of
these approaches). Each expert has the same size as the original MLPs in the
corresponding dense model, and we use $128$ experts to cover the same range of
model sizes as our PEER models. This type of MoE represents standard coarse-
grained MoE approaches, which consist of a small number of large experts.

In PKM, we used $1024^{2}$ memories with $h=8$ heads and top $k=32$ memories
were selected per head. We also applied query batch normalization, as
recommended in the original PKM paper (Lample et al., 2019), to enhance memory
usage.

In PEER, we used $1024^{2}$ experts with $h=8$ heads and top $k=16$ experts
per head. By default, we also enabled query BatchNorm to increase expert
usage. Ablation studies in subsection 3.3 investigate the effect of these
hyperparameters. Unlike the expert-choice MoE baseline, PEER represents a
fine-grained approach where a large number of small experts are employed.

Across all model sizes and methods, we maintained a consistent batch size
(128) and sequence length (2048). We calculated the number of training steps
by dividing the total compute budget by the FLOPs per training step. Fig. 1
presents the isoFLOP profiles. Compared to the dense FFW baseline, the sparse
alternatives shift the isoFLOP curves downward and to right because they
introduce a larger number of total parameters $P$ but utilize a smaller or
equal number of active parameters $P_{\text{active}}$. Given the same compute
budget, a PEER model achieves the lowest compute-optimal perplexity.

###  3.2 Evaluation on Language Modeling Datasets

After determining the compute-optimal model for each method based on the
isoFLOP curves, we evaluated the performance of these pretrained models on
several popular language modeling datasets, including Curation Corpus
(Curation, 2020), Lambada (Paperno et al., 2016), the Pile (Gao et al., 2020),
Wikitext (Merity et al., 2016) and the pretraining dataset C4. Table 1
presents a summary of the evaluation results. We grouped the models based on
their FLOP budgets used during training.

Table 1: Perplexities of the compute-optimal models of each method on language modeling datasets.  Method | Curation | Lambada | Pile | Wikitext | C4
---|---|---|---|---|---
| Corpus |  |  |  |
Dense (6e18) | 23.26 | 21.95 | 24.55 | 29.14 | 23.84
MoE (6e18) | 20.98 | 19.09 | 23.26 | 26.10 | 21.41
PKM (6e18) | 21.80 | 19.39 | 20.49 | 27.09 | 21.92
PEER (6e18) | 20.68 | 17.65 | 19.01 | 25.48 | 20.63
Dense (2e19) | 17.70 | 12.28 | 18.19 | 21.21 | 18.31
MoE (2e19) | 16.88 | 12.97 | 17.41 | 20.28 | 17.12
PKM (2e19) | 17.03 | 11.18 | 16.34 | 20.26 | 17.36
PEER (2e19) | 16.34 | 10.33 | 14.99 | 19.09 | 16.45

###  3.3 Ablations

#### Varying the Number of Total Experts

The models in the isoFLOP plot depicted in Fig. 1 all have over a million
($1024^{2}$) experts. Here we conduct an ablation study on the effect of the
number of experts $N$, which determines the total parameter count $P$ in Eq.
9. We selected the model at the isoFLOP-optimal position and vary the number
of experts ($N=128^{2},256^{2},512^{2},1024^{2}$) in the PEER layer while
keeping the number of active experts constant ($h=8,k=16$). The results are
shown in Fig. 3 (a). As can be seen, the isoFLOP curve interpolates between
the PEER model with $1024^{2}$ experts and the corresponding dense backbone
without replacing the FFW layer in the middle block by a PEER layer. This
demonstrates that simply increasing the number experts can improve model
performance.

(a) Varying Total Expert Num

(b) Varying Active Expert Num

Figure 3: We conduct two ablation studies using the same PEER model
configuration. In (a), we vary the total number of experts $N$ while keeping
the same number of active experts $hk=128$. In (b), we vary the number of
active experts $G=hk$ by jointly changing $h$ and $k$ while keeping the total
number of experts at $N=1024^{2}$.

#### Varying the Number of Active Experts

We also conducted an ablation study on the effect of the number of active
experts $hk$, which equals the granularity $G$ in Eq. 9. We systematically
varied the number of active experts ($hk=32,64,128,256,512$) while keeping the
number of total experts constant ($N=1024^{2}$). Furthermore, for a given
$hk$, we jointly varied $h$ and $k$ to identify the optimal composition. The
resulting isoFLOP curves, plotted over the number of heads ($h$), are shown in
Fig. 3 (b).

The results indicate that, within the range of values considered, higher $hk$
generally leads to improved performance. Notably, the optimal $h$ increases as
$hk$ increases. However, the performance gradually saturates, and increasing
the number of active experts also increases device memory consumption and may
necessitate additional accelerator devices. Thus in practice, the appropriate
$hk$ values should be selected based on the trade-off between performance,
device number and computational resource requirements.

Table 2: KL and expert usage for different memory sizes, with and without query BN. Similar to the findings in PKM, using query BN results in a more balanced usage of the experts.  Expert num $N$ | 16k | 65k | 262k | 1M
---|---|---|---|---
BatchNorm | No | Yes | No | Yes | No | Yes | No | Yes
Perplexity | $23.4658$ | $23.4690$ | $22.6121$ | $22.5543$ | $21.5409$ | $21.4696$ | $20.7307$ | $20.6449$
Expert Usage (%) | $100$ | $100$ | $100$ | $100$ | $99.9996$ | $100$ | $99.7930$ | $99.9754$
Unevenness ($\downarrow$) | $0.4489$ | $0.2954$ | $0.6325$ | $0.4420$ | $0.9707$ | $0.6552$ | $1.5244$ | $1.0588$

#### Expert Usage and Query Batch Normalization

Given the presence of over a million experts in the PEER layer, it is natural
to inquire how many of these experts are actually selected during inference
and whether their usage is evenly distributed. To analyze this, we kept an
accumulated router score, denoted as $z^{\prime}_{i}=\sum_{x}g_{i}(x)$ for
each expert $e_{i}$ across all tokens $x$ within the C4 validation set. Here
$g_{i}(x)$ is the router score used to aggregate the expert output when token
$x$ is given as input, with $g_{i}(x)=0$ if expert $e_{i}$ is not selected.
From these accumulated router scores, we can obtain an empirical probability
distribution vector, denoted as $z=z^{\prime}/||z^{\prime}||_{1}$,
representing the distribution of all experts over the C4 validation set. Then
we computed the following metrics proposed by Lample et al. (2019) to assess
the usage and distribution of experts:

  * •

Expert Usage: the fraction of experts retrieved during inference:
$\\#\\{z_{i}\neq 0\\}$

  * •

Unevenness: KL divergence between $z$ and the uniform distribution:
$\log(N)+\sum_{i}z_{i}\log(z_{i})$

where $N$ is the number of total experts.

By default, we also added a batch normalization (BN) layer on top of the query
network, as proposed by Lample et al. (2019) to increase the expert usage
during training. Here we study the effect of adding this BN layer on the
above-mentioned metrics.

Table 2 presents the expert usage and unevenness for varying numbers of
experts, with and without BN. We can see that even for 1M experts, the expert
usage is close to $100\%$, and using BN can lead to more balanced utilization
of the experts and lower perplexities. These findings demonstrate the
effectiveness of the PEER model in utilizing a large number of experts.

Figure 4: Query BatchNorm Ablation. IsoFLOP curves of a PEER model with 1M
experts on the C4 dataset, with and without query BatchNorm.

We additionally compared isoFLOP curves with and without BN. Fig. 4 shows that
the PEER model with BN generally achieves lower perplexities. While the
difference is not significant, it is most pronounced around the isoFLOP-
optimal region.

##  4 Related Works

#### Mixture of Expert

Since Shazeer et al. (2017) demonstrated the effectiveness of sparsely-gated
Mixtures of Experts (MoEs) in efficiently increasing model capacity on GPU
clusters, MoEs have emerged as a popular technique for scaling large models
efficiently. Subsequent research (Fedus et al., 2022; Lepikhin et al., 2020;
Du et al., 2022) has proposed variations to address challenges such as load
balancing, communication overhead, and training instability. These methods
usually replace feedforward (FFW) layers in certain Transformer blocks with
sparsely-gated MoE layers, which consist of multiple FFW layers as experts.
Typically each expert matches the size of the regular dense FFW layer. Gating
scores are calculated for each expert and token, and only the top k experts
are activated for each token. These methods are known as token-choice methods.
More recently, Zhou et al. (2022) introduced the Expert Choice routing method,
where experts choose the top k tokens instead of tokens selecting experts.
However, both token-choice and expert-choice methods require the top-k
operator on a gating score matrix of size $N\times M$ ($N$: number of experts,
$M$: number of tokens), resulting in a routing cost of at least $O(N)$. This
limits their practical application to a small number of experts (typically
less than 128).

Instead of using the top-k operator, some works also proposed using
deterministic hash tables as routers (Roller et al., 2021; dos Santos et al.,
2023). With $O(1)$ average lookup complexity, these methods offer potential
scalability to a large number of experts. However, these routers are fixed and
not learned. Clark et al. (2022) showed that deterministic routing does not
scale as well as trainable routers. Furthermore, Dikkala et al. (2023) proved
theoretically that learned routers offer non-trivial advantages over their
fixed counterparts, such as removing spurious directions and identifying
latent clusters in data. In contrast to previous works, the proposed PEER
layer employs a learned router with sublinear ($O(\sqrt{N})$) complexity.

Since PEER uses lightweight experts, our work is also related to recent
studies on parameter-efficient MoEs (Wang et al., 2022; Zadouri et al., 2024).
These methods utilize parameter efficient fine-tuning (PEFT) adapters as
experts instead of full-sized FFWs. Their focus is on minimizing the number of
parameters updated during fine-tuning, allowing storage of only one copy of
the large backbone model. In PEER, parameter efficiency refers to the small
number of active parameters in the MoE layer, which directly affects FLOPs and
activation memory consumption during pre-training and inference. However, PEER
could potentially be adapted to retrieve a large number of PEFT adapters.

#### Retrieval-Augmented Models

Our proposed method, with its retrieval mechanism for a large number of
experts, aligns with the emerging field of retrieval-augmented models. These
models facilitate large model memorization by retrieving knowledge from
external databases, leading to improved accuracy and efficiency on knowledge-
intensive tasks. Some notable works in this domain include ones by Khandelwal
et al. (2019); Borgeaud et al. (2022a); Guu et al. (2020). While these methods
retrieve data in various formats, for instance, tokens (Khandelwal et al.,
2019), chunks (Borgeaud et al., 2022b) or knowledge graphs (Kang et al., 2023)
(see (Gao et al., 2023) for a comprehensive survey on this topic), they differ
from the proposed method in that they retrieve data rather than learned
functions (experts). This distinction sets our parameter-efficient expert
retrieval approach apart from existing retrieval-augmented models.

#### Efficient Feedforward Layers

Enhancing the efficiency of feedforward networks has been a long-standing area
of research. Similar to PEER, most approaches are based on the idea of
conditional computation (Bengio, 2013), where a gating mechanism is trained to
determine which subset of neurons to compute. For instance, Davis & Arel
(2013) utilized low-rank weight matrix approximation to estimate the sign of
pre-nonlinearity activations. Neurons with negative activations are omitted as
they will produce zeros after the nonlinearity. Bengio et al. (2015) explored
reinforcement learning to develop an activation-dependant policy for dropping
blocks of neurons. More recently, Belcak & Wattenhofer (2023) introduced the
Fast FeedForward (FFF) layer that employs a differentiable balanced binary
tree to select a neuron block for computation. During inference, only one leaf
(corresponding to one block) is selected, hence it has $O(\log(N))$
complexity, where $N$ is the total number of blocks in the tree. However,
during training, all leaves and intermediate nodes are activated for gradient
calculation, imposing a training complexity of $O(N)$ and limiting the total
number of blocks. The most relevant work to ours is the Product Key Memory
(PKM) (Lample et al., 2019), whose retrieval technique is utilized as the
router in the PEER layer. However, PKM retrieves memory vectors instead of
functions, thus their values cannot vary according to the inputs. As we show
in Section 3, by changing the memory vectors to input-dependent expert
networks, PEER can achieve significantly higher efficiency than PKM. Finally,
Csordás et al. (2023) presented a unified view encompassing FFW, MoE and PKM
and proposed to change the router normalization function in MoE and PKM from
softmax to sigmoid or ReLU.

##  5 Conclusion

This work introduces a fine-grained MoE architecture that decomposes an
extremely wide dense feedforward layer into a large number of small experts.
This design is supported by the recent discovery of the fine-grained MoE
scaling law. To overcome the computational overhead of routing to a large
number of experts, we apply the product keys to efficiently select a small
subset of hidden neurons within a wide MLP layer. Empirical analysis using
language modeling tasks demonstrate that given the same compute budget, PEER
significantly outperforms dense transformers, coarse-grained MoEs and product
key memory layers.

## Acknowledgments

The author would like to thank Adam Santoro, Arthur Guez, Arthur Szlam, Andrei
Rusu, Marc’aurelio Ranzato, Simon Schug, Utku Evci, Doina Precup and Razvan
Pascanu for their insightful discussions and invaluable advice. The author is
also grateful to Zhitao Gong, Daniel Toyama, Qixuan Feng and Jiajun Shen for
their technical assistance. Special thanks are due to Adam Santoro for sharing
the isoFLOP analysis scripts and to Andy Brock for building and maintaining
the internal codebase used to train the models.

## References

  * Aljundi et al. (2017) Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3366–3375, 2017.
  * Belcak & Wattenhofer (2023) Peter Belcak and Roger Wattenhofer.  Fast feedforward networks.  _arXiv preprint arXiv:2308.14711_ , 2023.
  * Bengio et al. (2015) Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup.  Conditional computation in neural networks for faster models.  _arXiv preprint arXiv:1511.06297_ , 2015.
  * Bengio (2013) Yoshua Bengio.  Deep learning of representations: Looking forward.  In _International conference on statistical language and speech processing_ , pp. 1–37. Springer, 2013.
  * Borgeaud et al. (2022a) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 2206–2240. PMLR, 17–23 Jul 2022a.  URL https://proceedings.mlr.press/v162/borgeaud22a.html.
  * Borgeaud et al. (2022b) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pp. 2206–2240. PMLR, 2022b.
  * Chen et al. (2023) Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui.  Lifelong language pretraining with distribution-specialized experts.  In _International Conference on Machine Learning_ , pp. 5383–5395. PMLR, 2023.
  * Clark et al. (2022) Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al.  Unified scaling laws for routed language models.  In _International Conference on Machine Learning_ , pp. 4057–4086. PMLR, 2022.
  * Csordás et al. (2023) Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber.  Approximating two-layer feedforward networks for efficient transformers.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * Curation (2020) Curation.  Curation corpus base, 2020.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei.  Knowledge neurons in pretrained transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8493–8502, 2022\.
  * Davis & Arel (2013) Andrew Davis and Itamar Arel.  Low-rank approximations for conditional feedforward computation in deep neural networks.  _arXiv preprint arXiv:1312.4461_ , 2013.
  * Dikkala et al. (2023) Nishanth Dikkala, Nikhil Ghosh, Raghu Meka, Rina Panigrahy, Nikhil Vyas, and Xin Wang.  On the benefits of learning to route in mixture-of-experts models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.  URL https://openreview.net/forum?id=QV79qiKAjD.
  * dos Santos et al. (2023) Cicero Nogueira dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, and David Uthus.  Memory augmented language models through mixture of word experts, 2023\.
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.  GLaM: Efficient scaling of language models with mixture-of-experts.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 5547–5569. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/du22c.html.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _The Journal of Machine Learning Research_ , 23(1):5232–5270, 2022.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_ , 2023.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.  Transformer feed-forward layers are key-value memories.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 5484–5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.446.  URL https://aclanthology.org/2021.emnlp-main.446.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pp. 3929–3938. PMLR, 2020.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Kang et al. (2023) Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang.  Knowledge graph-augmented language models for knowledge-grounded dialogue generation.  _arXiv preprint arXiv:2305.18846_ , 2023.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models.  In _International Conference on Learning Representations_ , 2019.
  * Krajewski et al. (2024) Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, et al.  Scaling laws for fine-grained mixture of experts.  _arXiv preprint arXiv:2402.07871_ , 2024.
  * Kraska et al. (2018) Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis.  The case for learned index structures.  In _Proceedings of the 2018 international conference on management of data_ , pp. 489–504, 2018.
  * Lample et al. (2019) Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou.  Large memory layers with product keys.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lample et al. (2021) Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou.  Minimalist implementation of a product-key memory layer.  https://github.com/facebookresearch/XLM/blob/main/PKM-layer.ipynb, 2021\.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_ , 2020.
  * Li et al. (2024) Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B. Shroff.  Theory on mixture-of-experts in continual learning, 2024.  URL https://arxiv.org/abs/2406.16437.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models, 2016.
  * Mitchell et al. (2018) Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al.  Never-ending learning.  _Communications of the ACM_ , 61(5):103–115, 2018\.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1144.  URL https://www.aclweb.org/anthology/P16-1144.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67, 2020.
  * Roller et al. (2021) Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al.  Hash layers for large sparse models.  _Advances in Neural Information Processing Systems_ , 34:17555–17566, 2021.
  * Shazeer (2020) Noam Shazeer.  Glu variants improve transformer, 2020.  URL https://arxiv.org/abs/2002.05202.
  * Shazeer et al. (2017) Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=B1ckMDqlg.
  * Wang et al. (2022) Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao.  AdaMix: Mixture-of-adaptations for parameter-efficient model tuning.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 5744–5760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.388.  URL https://aclanthology.org/2022.emnlp-main.388.
  * Yu et al. (2024) Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He.  Boosting continual learning of vision-language models via mixture-of-experts adapters.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 23219–23230, June 2024.
  * Zadouri et al. (2024) Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker.  Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=EvDeiLv7qc.
  * Zhou et al. (2022) Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al.  Mixture-of-experts with expert choice routing.  _Advances in Neural Information Processing Systems_ , 35:7103–7114, 2022.


[a]Yukiho Kobayashi

# Detection of the 2021 Outburst of RS Ophiuchi with the LST-1

Arnau Aguasca-Cabot  María Isabel Bernardos Martín  David Green  Rubén López-
Coto

###### Abstract

Novae are luminous explosions in close binaries which host a white dwarf and a
companion donor star. They are triggered by a thermonuclear runaway when the
white dwarf accretes a critical amount of matter from the secondary. Though
novae are established as high-energy gamma-ray emitters through observations
by the Fermi Large Area Telescope (LAT), the origin of the gamma-ray emission,
whether it is hadronic or leptonic, had been under intense debate until very
recently. RS Ophiuchi (RS Oph) is a well-known recurrent symbiotic nova with a
recurrence time scale of 15 years. The most recent outburst of RS Oph in 2021
brought the first detection of very-high-energy (VHE) gamma rays from a nova
ever. The first Large-Sized Telescope prototype (LST-1) of the Cherenkov
Telescope Array observed this historic event along with H.E.S.S. and MAGIC.
The LST-1 observations in the first days after the burst onset show a clear
VHE gamma-ray signal from RS Oph. The low energy threshold of LST-1 allows us
to reconstruct the RS Oph gamma-ray spectrum down to $\sim$30 GeV, providing
the best connection of the VHE gamma-ray data to the Fermi LAT energy range.
The results from the analysis of the LST-1 observations are consistent with
those obtained with H.E.S.S. and MAGIC, and also support a hadronic origin for
the observed gamma-ray fluxes. In this contribution, we will present the
analysis results of the LST-1 observations of the 2021 outburst of RS Oph.

##  1 Introduction

Novae are luminous eruptions observed in close binaries where a white dwarf
(WD) is interacting with its stellar companion. The explosion is powered by a
thermonuclear runaway ignited on the surface of the WD when it accretes a
critical amount of matter from the companion donor star. Novae have been
revealed to be high-energy gamma-ray emitters through observations by the
Fermi Large Area Telescope (LAT), starting with the detection of gamma-ray
emission from V407 Cyg in 2010 [1]. Though more than a dozen novae have been
detected by the Fermi LAT, the mechanism responsible for the production of
gamma-ray emission in novae had remained unclear until very recently. In
August 2021, RS Ophiuchi (RS Oph), a well-known recurrent nova, erupted after
an interval of 15 years since the previous outburst in 2006. Noteworthy, the
2021 outburst of RS Oph was detected in the very-high-energy (VHE, >100 GeV)
regime by Imaging Atmospheric Cherenkov Telescopes (IACTs) such as High Energy
Stereoscopic System (H.E.S.S.) and the Major Atmospheric Gamma Imaging
Cherenkov (MAGIC) telescopes, which marked the event as the first nova
explosion that was detected in VHE gamma rays in history [2, 3].

The Cherenkov Telescope Array (CTA) is the next-generation IACT array
consisting of three kinds of telescopes with different sizes. Among them, the
Large-Sized Telescopes (LSTs), equipped with a large mirror dish of 23 m in
diameter and a sensitive focal-plane camera with fast readout, dominate the
lower energy band of CTA down to $\sim$20 GeV. The LST prototype (LST-1) for
CTA, built on the CTA northern site, La Palma in the Canary Islands of Spain,
has been in its commissioning phase and performing astrophysical observations
since 2019 [4]. The LST-1 observed the historic nova outburst of RS Oph in
2021. In this contribution, we present the analysis results of the LST-1
observations.

##  2 Observations

RS Oph is a well-known recurrent nova with a relatively short recurrence time-
scale of $\sim$15 years. RS Oph is also known to be a symbiotic system, where
the WD is embedded in the outflow from the red giant companion. Thanks to the
short recurrence time-scale, outbursts of RS Oph have been recorded several
times in history [5]. The outburst in 2006 was especially well studied at
various wavelengths, such as optical, radio, and x-ray, but it was before the
launch of Fermi [6].

On August 8th 2021, RS Oph was reported to be in a new burst state from
observations in the optical and gamma-ray bands [7, 8, 9, 10]. In response to
these alerts, the LST-1 started observations of RS Oph on August 9th, just a
day after the burst onset. The observations were performed under proper
conditions, i.e., a clear and dark sky, until August 12th. These good-quality
data in the first nights amount to 6.4 hours of effective observational time.
From August 13th, however, bad weather and enhanced moonlight conditions
prevented further data taking. We resumed observations on August 29th and
continued until September 2nd under good observational conditions, amounting
to about four more hours. The zenith angle range of the observations was
between 35∘ and 64∘. The observations were performed in the so-called wobble
mode with a wobble offset of 0.4∘ [11]. The LST-1 observations of the 2021
outburst of RS Oph are summarized in Table 1.

Table 1: Summary of the LST-1 observations of RS Oph during its 2021 outburst date | observation | zenith range
---|---|---
| time [h] | [deg]
9 Aug. 2021 | 1.4 h | 35-42
10 Aug. 2021 | 2.7 h | 35-59
12 Aug. 2021 | 2.3 h | 35-55
13 Aug. 2021 | 1.3 h | 36-54
14 Aug. 2021 | 1.5 h | 35-46
15 Aug. 2021 | 1.3 h | 41-57
29 Aug. 2021 | 1.0 h | 46-58
30 Aug. 2021 | 1.5 h | 40-57
1 Sep. 2021 | 0.3 h | 56-64
2 Sep. 2021 | 1.3 h | 41-57

##  3 Analysis

The LST-1 observations are reduced following the standard LST analysis
procedure [12]. The analysis is performed with cta-
lstchain111https://github.com/cta-observatory/cta-lstchain, a dedicated LST
analysis software based on the CTA low-level analysis pipeline ctapipe [13,
14, 15, 16]. The cta-lstchain performs all the analysis steps from calibration
and signal extraction to the reconstruction of events. For the reconstruction
of the energy and direction of primary particles, random forest (RF)
algorithms are adopted. An RF is also used to reject background cosmic-ray
events by giving a score called gammaness to each event, which represents how
likely the primary particle is a gamma ray. The so-called source-independent
approach, a method to reconstruct events without an assumption on the source
position, is applied in this work. The LST-1 data are reduced to the DL3
format, which is subsequently fed to gammapy222https://gammapy.org, the
official science analysis tool for CTA, for computing the gamma-ray spectrum
of the source and the source flux light-curve [17, 18].

Monte Carlo (MC) simulations are prepared to train the RF algorithms and
evaluate the telescope’s instrumental response functions (IRFs). The MC
simulations that are used in this work are generated according to a standard
procedure for the LST analysis, but are tuned to the RS Oph observations [12].
For instance, the amount of night sky background contamination in the
simulations is adjusted to the observations at the camera image level.
Simulations are produced for different positions in the sky to take into
account the dependence of the telescope performance for different pointing
directions. As shown in Figure 1, the simulations to train the RF algorithms
are generated along a declination path close to that of RS Oph and those for
testing the telescope performance are produced in a grid of $\cos{\rm ZD}$ and
$\sin{\delta}$, where $\rm ZD$ is the zenith angle and $\delta$ is an angle
between the geomagnetic field and the pointing direction. The IRFs of the
LST-1 are evaluated in each testing MC node and the closest node to each
observational run is used to compute gamma-ray flux. For processing MC
simulations, lstmcpipe, a dedicated package for reduction of LST MC
simulations, is adopted [19, 20].

Figure 1: The pointing directions of the data taken on 9th, 10th, and 12th of
August 2021, the closest testing MC nodes to the data and the training MC
nodes on the plane of the zenith angle and the azimuth angle.

For the spectral analysis, the energy-dependent cuts on gammaness are defined
from the MC simulations so that gamma rays are kept with an efficiency of 50%.
Directional cuts are also applied energy-dependently with a 70% gamma-ray
efficiency. The energy threshold of the LST-1 observations for this work is
evaluated from the true energy distribution of the simulated gamma-ray events
surviving all the event selection criteria. Though the energy threshold
depends on the zenith angle, an average value over the LST-1 observations is
found to be $\sim$30 GeV, assuming that RS Oph has a power-law spectrum with
an index of $\sim$4, as indicated from the observations by H.E.S.S. and MAGIC.
This is the lowest energy threshold for the observations of VHE gamma rays
from RS Oph among the IACTs, and thus the LST-1 provides the best connection
of the VHE gamma-ray data to the Fermi LAT energy range. The gamma-ray flux of
RS Oph is reconstructed assuming a point-like source with a power-law spectral
model, $\Phi(E)=\Phi_{0}\left({E}/{E_{\rm ref}}\right)^{-p}$, where the
spectrum is normalized at $\Phi(E_{\rm ref})=\Phi_{0}$ with a reference energy
$E_{\rm ref}=130$ GeV.

##  4 Results

Figure 2 shows the distribution of the squared angular distance between the
reconstructed arrival direction of the gamma-ray candidate events obtained
from the LST-1 observations and the position of RS Oph. Events with
$\textit{gammaness}>0.6$ and reconstructed energies between 30 GeV and 1 TeV
are shown. From the observations between August 9th and August 12th, a gamma-
ray signal from the direction of RS Oph is clearly detected at a statistical
significance of 9.5 $\sigma$. The signal is also visible each night at a
significance above 5 $\sigma$. From the observations on and after 29th August,
on the other hand, no significant excess is found. The light curve during the
first nights after the burst onset is computed at energies above 100 GeV and
shown in Figure 3. The light curve that is reconstructed from the LST-1
observations is compatible with a constant flux during the first nights and it
is in good agreement with the MAGIC results.

Figure 2: Distribution of events recorded for RS Oph as a function of
$\theta^{2}$, the squared angular distance between the reconstructed arrival
direction of the gamma-ray candidate events and the position of RS Oph. Events
with $\textit{gammaness}>0.6$ and reconstructed energies between 30 GeV and 1
TeV are shown. Left: Observations between August 9th and August 12th of 2021.
Right: Observations between August 29th and September 2nd. Figure 3: Daily
gamma-ray flux of RS Oph during the first nights of its 2021 outburst
reconstructed with the LST-1 in comparison with the MAGIC results.

##  5 Conclusion

The LST-1 performed observations of RS Oph during its 2021 outburst and
acquired good-quality data during the first nights after the eruption. The
LST-1 data are analyzed with the standard LST analysis tools and dedicated
all-sky MC simulations tuned to the RS Oph observations. The gamma-ray
emission from RS Oph is firmly detected by the LST-1 during the first nights
after the burst onset at a statistical significance of 9.5 $\sigma$.
Noteworthy, the LST-1 achieves the lowest energy threshold for the VHE gamma-
ray observations of the 2021 outburst of RS Oph among the IACTs $\sim$30 GeV,
which gives the best connection of the VHE gamma-ray data to the Fermi LAT
energy band. A detailed interpretation of the LST-1 observations with
dedicated modeling is in progress and the results will be presented in a
forthcoming publication.

LST Acknowledgements

We gratefully acknowledge financial support from the following agencies and
organisations:

Ministry of Education, Youth and Sports, MEYS LM2015046, LM2018105, LTT17006,
EU/MEYS CZ.02.1.01/0.0/0.0/16_013/0001403, CZ.02.1.01/0.0/0.0/18_046/0016007
and CZ.02.1.01/0.0/0.0/16_019/0000754, Czech Republic; Max Planck Society,
German Bundesministerium für Bildung und Forschung (Verbundforschung / ErUM),
Deutsche Forschungsgemeinschaft (SFBs 876 and 1491), Germany; Istituto
Nazionale di Astrofisica (INAF), Istituto Nazionale di Fisica Nucleare (INFN),
Italian Ministry for University and Research (MUR); ICRR, University of Tokyo,
JSPS, MEXT, Japan; JST SPRING - JPMJSP2108; Narodowe Centrum Nauki, grant
number 2019/34/E/ST9/00224, Poland; The Spanish groups acknowledge the Spanish
Ministry of Science and Innovation and the Spanish Research State Agency (AEI)
through the government budget lines PGE2021/28.06.000X.411.01,
PGE2022/28.06.000X.411.01 and PGE2022/28.06.000X.711.04, and grants
PGC2018-095512-B-I00, PID2019-104114RB-C31, PID2019-107847RB-C44,
PID2019-104114RB-C32, PID2019-105510GB-C31, PID2019-104114RB-C33,
PID2019-107847RB-C41, PID2019-107847RB-C43, PID2019-107988GB-C22; the “Centro
de Excelencia Severo Ochoa” program through grants no. CEX2021-001131-S,
CEX2019-000920-S; the “Unidad de Excelencia María de Maeztu” program through
grants no. CEX2019-000918-M, CEX2020-001058-M; the “Juan de la Cierva-
Incorporación” program through grant no. IJC2019-040315-I. They also
acknowledge the “Programa Operativo” FEDER 2014-2020, Consejería de Economía y
Conocimiento de la Junta de Andalucía (Ref. 1257737), PAIDI 2020 (Ref.
P18-FR-1580) and Universidad de Jaén; “Programa Operativo de Crecimiento
Inteligente” FEDER 2014-2020 (Ref. ESFRI-2017-IAC-12), Ministerio de Ciencia e
Innovación, 15% co-financed by Consejería de Economía, Industria, Comercio y
Conocimiento del Gobierno de Canarias; the “CERCA” program of the Generalitat
de Catalunya; and the European Union’s “Horizon 2020” GA:824064 and
NextGenerationEU; We acknowledge the Ramon y Cajal program through grant
RYC-2020-028639-I and RYC-2017-22665; State Secretariat for Education,
Research and Innovation (SERI) and Swiss National Science Foundation (SNSF),
Switzerland; The research leading to these results has received funding from
the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant
agreements No 262053 and No 317446. This project is receiving funding from the
European Union’s Horizon 2020 research and innovation programs under agreement
No 676134. ESCAPE - The European Science Cluster of Astronomy & Particle
Physics ESFRI Research Infrastructures has received funding from the European
Union’s Horizon 2020 research and innovation programme under Grant Agreement
no. 824064.

## References

  * [1] The Fermi-LAT Collaboration, _Fermi establishes classical novae as a distinct class of gamma-ray sources_ , _Science_ 345 (2014) 554\.
  * [2] H.E.S.S. Collaboration, _Time-resolved hadronic particle acceleration in the recurrent nova rs ophiuchi_ , _Science_ 376 (2022) 77.
  * [3] MAGIC Collaboration, _Proton acceleration in thermonuclear nova explosions revealed by gamma rays_ , _Nature Astronomy_ 6 (2022) 689.
  * [4] CTA-LST Project, _Status and results of the prototype LST of CTA_ , in _37th International Cosmic Ray Conference_ , p. 872, Mar., 2022.
  * [5] B.E. Schaefer, _Comprehensive photometric histories of all known galactic recurrent novae_ , _The Astrophysical Journal Supplement Series_ 187 (2010) 275.
  * [6] V. Tatischeff and M. Hernanz, _Evidence for nonlinear diffusive shock acceleration of cosmic rays in the 2006 outburst of the recurrent nova rs ophiuchi_ , _The Astrophysical Journal_ 663 (2007) L101.
  * [7] S. Beck, “Aavso alert notice 752: Rare outburst of recurrent nova rs ophiuchi.” https://www.aavso.org/aavso-alert-notice-752.
  * [8] K. Geary, “Vsnet-alert notice, 26131.” http://ooruri.kusastro.kyoto-u.ac.jp/mailarchive/vsnet-alert/26131.
  * [9] C.C. Cheung, S. Ciprini and T.J. Johnson, _Fermi-LAT Gamma-ray Detection of the Recurrent Nova RS Oph_ , _The Astronomer’s Telegram_ 14834 (2021) 1.
  * [10] C. Cheung, T. Johnson, P. Jean, M. Kerr, K. Page, J. Osborne et al., _Fermi lat gamma-ray detection of the recurrent nova rs ophiuchi during its 2021 outburst_ , _The Astrophysical Journal_ 935 (2022) 44\.
  * [11] V. Fomin et al., _New methods of atmospheric cherenkov imaging for gamma-ray astronomy. i. the false source method_ , _Astroparticle Physics_ 2 (1994) 137.
  * [12] CTA-LST Project, _Observations of the Crab Nebula and Pulsar with the Large-Sized Telescope Prototype of the Cherenkov Telescope Array_ , _arXiv e-prints_ (2023) arXiv:2306.12960.
  * [13] R. Lopez-Coto et al., _cta-observatory/cta-lstchain: v0.9.4_ , Mar., 2022.  10.5281/zenodo.6344674.
  * [14] R. López-Coto et al., _Physics Performance of the Large Size Telescope prototype of the Cherenkov Telescope Array_ , in _Proceedings, 37th International Cosmic Ray Conference_ , vol. 395, p. 806, 2021.
  * [15] K. Kosack et al., _cta-observatory/ctapipe: v0.12.0_ , Nov., 2021.  10.5281/zenodo.5720333.
  * [16] M. Nöthe, K. Kosack, L. Nickel and M. Peresano, _Prototype open event reconstruction pipeline for the cherenkov telescope array_ , in _Proceedings, 37th International Cosmic Ray Conference_ , vol. 395, 2021.
  * [17] F. Acero et al., _Gammapy: Python toolbox for gamma-ray astronomy_ , Nov., 2022\.  10.5281/zenodo.7311399.
  * [18] C. Deil et al., _Gammapy - A prototype for the CTA science tools_ , in _35th International Cosmic Ray Conference (ICRC2017)_ , vol. 301 of _International Cosmic Ray Conference_ , p. 766, Jan., 2017.
  * [19] T. Vuillaume, E. Garcia and L. Nickel, _lstmcpipe_ , Oct., 2022.  10.5281/zenodo.7180216.
  * [20] E. Garcia, T. Vuillaume and L. Nickel, _The lstmcpipe library_ , 2022.

## Full Author List: CTA-LST Project

K. Abe1, S. Abe2, A. Aguasca-Cabot3, I. Agudo4, N. Alvarez Crespo5, L. A.
Antonelli6, C. Aramo7, A. Arbet-Engels8, C. Arcaro9, M. Artero10, K. Asano2,
P. Aubert11, A. Baktash12, A. Bamba13, A. Baquero Larriva5,14, L.
Baroncelli15, U. Barres de Almeida16, J. A. Barrio5, I. Batkovic9, J. Baxter2,
J. Becerra González17, E. Bernardini9, M. I. Bernardos4, J. Bernete Medrano18,
A. Berti8, P. Bhattacharjee11, N. Biederbeck19, C. Bigongiari6, E. Bissaldi20,
O. Blanch10, G. Bonnoli21, P. Bordas3, A. Bulgarelli15, I. Burelli22, L.
Burmistrov23, M. Buscemi24, M. Cardillo25, S. Caroff11, A. Carosi6, M. S.
Carrasco26, F. Cassol26, D. Cauz22, D. Cerasole27, G. Ceribella8, Y. Chai8, K.
Cheng2, A. Chiavassa28, M. Chikawa2, L. Chytka29, A. Cifuentes18, J. L.
Contreras5, J. Cortina18, H. Costantini26, M. Dalchenko23, F. Dazzi6, A. De
Angelis9, M. de Bony de Lavergne11, B. De Lotto22, M. De Lucia7, R. de
Menezes28, L. Del Peral30, G. Deleglise11, C. Delgado18, J. Delgado Mengual31,
D. della Volpe23, M. Dellaiera11, A. Di Piano15, F. Di Pierro28, A. Di
Pilato23, R. Di Tria27, L. Di Venere27, C. Díaz18, R. M. Dominik19, D. Dominis
Prester32, A. Donini6, D. Dorner33, M. Doro9, L. Eisenberger33, D. Elsässer19,
G. Emery26, J. Escudero4, V. Fallah Ramazani34, G. Ferrara24, F. Ferrarotto35,
A. Fiasson11,36, L. Foffano25, L. Freixas Coromina18, S. Fröse19, S. Fukami2,
Y. Fukazawa37, E. Garcia11, R. Garcia López17, C. Gasbarra38, D. Gasparrini38,
D. Geyer19, J. Giesbrecht Paiva16, N. Giglietto20, F. Giordano27, P. Gliwny39,
N. Godinovic40, R. Grau10, J. Green8, D. Green8, S. Gunji41, P. Günther33, J.
Hackfeld34, D. Hadasch2, A. Hahn8, K. Hashiyama2, T. Hassan18, K. Hayashi2, L.
Heckmann8, M. Heller23, J. Herrera Llorente17, K. Hirotani2, D. Hoffmann26, D.
Horns12, J. Houles26, M. Hrabovsky29, D. Hrupec42, D. Hui2, M. Hütten2, M.
Iarlori43, R. Imazawa37, T. Inada2, Y. Inome2, K. Ioka44, M. Iori35, K.
Ishio39, I. Jimenez Martinez18, J. Jurysek45, M. Kagaya2, V. Karas46, H.
Katagiri47, J. Kataoka48, D. Kerszberg10, Y. Kobayashi2, K. Kohri49, A. Kong2,
H. Kubo2, J. Kushida1, M. Lainez5, G. Lamanna11, A. Lamastra6, T. Le Flour11,
M. Linhoff19, F. Longo50, R. López-Coto4, A. López-Oramas17, S. Loporchio27,
A. Lorini51, J. Lozano Bahilo30, P. L. Luque-Escamilla52, P. Majumdar53,2, M.
Makariev54, D. Mandat45, M. Manganaro32, G. Manicò24, K. Mannheim33, M.
Mariotti9, P. Marquez10, G. Marsella24,55, J. Martí52, O. Martinez56, G.
Martínez18, M. Martínez10, A. Mas-Aguilar5, G. Maurin11, D. Mazin2,8, E.
Mestre Guillen52, S. Micanovic32, D. Miceli9, T. Miener5, J. M. Miranda56, R.
Mirzoyan8, T. Mizuno57, M. Molero Gonzalez17, E. Molina3, T. Montaruli23, I.
Monteiro11, A. Moralejo10, D. Morcuende5, A. Morselli38, V. Moya5, H.
Muraishi58, K. Murase2, S. Nagataki59, T. Nakamori41, A. Neronov60, L.
Nickel19, M. Nievas Rosillo17, K. Nishijima1, K. Noda2, D. Nosek61, S.
Nozaki8, M. Ohishi2, Y. Ohtani2, T. Oka62, A. Okumura63,64, R. Orito65, J.
Otero-Santos17, M. Palatiello22, D. Paneque8, F. R. Pantaleo20, R. Paoletti51,
J. M. Paredes3, M. Pech45,29, M. Pecimotika32, M. Peresano28, F. Pfeiffle33,
E. Pietropaolo66, G. Pirola8, C. Plard11, F. Podobnik51, V. Poireau11, M.
Polo18, E. Pons11, E. Prandini9, J. Prast11, G. Principe50, C. Priyadarshi10,
M. Prouza45, R. Rando9, W. Rhode19, M. Ribó3, C. Righi21, V. Rizi66, G.
Rodriguez Fernandez38, M. D. Rodríguez Frías30, T. Saito2, S. Sakurai2, D. A.
Sanchez11, T. Šarić40, Y. Sato67, F. G. Saturni6, V. Savchenko60, B.
Schleicher33, F. Schmuckermaier8, J. L. Schubert19, F. Schussler68, T.
Schweizer8, M. Seglar Arroyo11, T. Siegert33, R. Silvia27, J. Sitarek39, V.
Sliusar69, A. Spolon9, J. Strišković42, M. Strzys2, Y. Suda37, H. Tajima63, M.
Takahashi63, H. Takahashi37, J. Takata2, R. Takeishi2, P. H. T. Tam2, S. J.
Tanaka67, D. Tateishi70, P. Temnikov54, Y. Terada70, K. Terauchi62, T.
Terzic32, M. Teshima8,2, M. Tluczykont12, F. Tokanai41, D. F. Torres71, P.
Travnicek45, S. Truzzi51, A. Tutone6, M. Vacula29, P. Vallania28, J. van
Scherpenberg8, M. Vázquez Acosta17, I. Viale9, A. Vigliano22, C. F.
Vigorito28,72, V. Vitale38, G. Voutsinas23, I. Vovk2, T. Vuillaume11, R.
Walter69, Z. Wei71, M. Will8, T. Yamamoto73, R. Yamazaki67, T. Yoshida47, T.
Yoshikoshi2, N. Zywucka39

1Department of Physics, Tokai University. 2Institute for Cosmic Ray Research,
University of Tokyo. 3Departament de Física Quàntica i Astrofísica, Institut
de Ciències del Cosmos, Universitat de Barcelona, IEEC-UB. 4Instituto de
Astrofísica de Andalucía-CSIC. 5EMFTEL department and IPARCOS, Universidad
Complutense de Madrid. 6INAF - Osservatorio Astronomico di Roma. 7INFN Sezione
di Napoli. 8Max-Planck-Institut für Physik. 9INFN Sezione di Padova and
Università degli Studi di Padova. 10Institut de Fisica d’Altes Energies
(IFAE), The Barcelona Institute of Science and Technology. 11LAPP, Univ.
Grenoble Alpes, Univ. Savoie Mont Blanc, CNRS-IN2P3, Annecy. 12Universität
Hamburg, Institut für Experimentalphysik. 13Graduate School of Science,
University of Tokyo. 14Universidad del Azuay. 15INAF - Osservatorio di
Astrofisica e Scienza dello spazio di Bologna. 16Centro Brasileiro de
Pesquisas Físicas. 17Instituto de Astrofísica de Canarias and Departamento de
Astrofísica, Universidad de La Laguna. 18CIEMAT. 19Department of Physics, TU
Dortmund University. 20INFN Sezione di Bari and Politecnico di Bari. 21INAF -
Osservatorio Astronomico di Brera. 22INFN Sezione di Trieste and Università
degli Studi di Udine. 23University of Geneva - Département de physique
nucléaire et corpusculaire. 24INFN Sezione di Catania. 25INAF - Istituto di
Astrofisica e Planetologia Spaziali (IAPS). 26Aix Marseille Univ, CNRS/IN2P3,
CPPM. 27INFN Sezione di Bari and Università di Bari. 28INFN Sezione di Torino.
29Palacky University Olomouc, Faculty of Science. 30University of Alcalá UAH.
31Port d’Informació Científica. 32University of Rijeka, Department of Physics.
33Institute for Theoretical Physics and Astrophysics, Universität Würzburg.
34Institut für Theoretische Physik, Lehrstuhl IV: Plasma-Astroteilchenphysik,
Ruhr-Universität Bochum. 35INFN Sezione di Roma La Sapienza. 36ILANCE, CNRS .
37Physics Program, Graduate School of Advanced Science and Engineering,
Hiroshima University. 38INFN Sezione di Roma Tor Vergata. 39Faculty of Physics
and Applied Informatics, University of Lodz. 40University of Split, FESB.
41Department of Physics, Yamagata University. 42Josip Juraj Strossmayer
University of Osijek, Department of Physics. 43INFN Dipartimento di Scienze
Fisiche e Chimiche - Università degli Studi dell’Aquila and Gran Sasso Science
Institute. 44Yukawa Institute for Theoretical Physics, Kyoto University. 45FZU
- Institute of Physics of the Czech Academy of Sciences. 46Astronomical
Institute of the Czech Academy of Sciences. 47Faculty of Science, Ibaraki
University. 48Faculty of Science and Engineering, Waseda University.
49Institute of Particle and Nuclear Studies, KEK (High Energy Accelerator
Research Organization). 50INFN Sezione di Trieste and Università degli Studi
di Trieste. 51INFN and Università degli Studi di Siena, Dipartimento di
Scienze Fisiche, della Terra e dell’Ambiente (DSFTA). 52Escuela Politécnica
Superior de Jaén, Universidad de Jaén. 53Saha Institute of Nuclear Physics.
54Institute for Nuclear Research and Nuclear Energy, Bulgarian Academy of
Sciences. 55Dipartimento di Fisica e Chimica ’E. Segrè’ Università degli Studi
di Palermo. 56Grupo de Electronica, Universidad Complutense de Madrid.
57Hiroshima Astrophysical Science Center, Hiroshima University. 58School of
Allied Health Sciences, Kitasato University. 59RIKEN, Institute of Physical
and Chemical Research. 60Laboratory for High Energy Physics, École
Polytechnique Fédérale. 61Charles University, Institute of Particle and
Nuclear Physics. 62Division of Physics and Astronomy, Graduate School of
Science, Kyoto University. 63Institute for Space-Earth Environmental Research,
Nagoya University. 64Kobayashi-Maskawa Institute (KMI) for the Origin of
Particles and the Universe, Nagoya University. 65Graduate School of
Technology, Industrial and Social Sciences, Tokushima University. 66INFN
Dipartimento di Scienze Fisiche e Chimiche - Università degli Studi
dell’Aquila and Gran Sasso Science Institute. 67Department of Physical
Sciences, Aoyama Gakuin University. 68IRFU, CEA, Université Paris-Saclay.
69Department of Astronomy, University of Geneva. 70Graduate School of Science
and Engineering, Saitama University. 71Institute of Space Sciences (ICE-CSIC),
and Institut d’Estudis Espacials de Catalunya (IEEC), and Institució Catalana
de Recerca I Estudis Avançats (ICREA). 72Dipartimento di Fisica - Universitá
degli Studi di Torino. 73Department of Physics, Konan University.