# Diversifying the Mixture-of-Experts Representation for Language Models with
Orthogonal Optimizer

Boan Liu, Liang Ding, Li Shen, Keqin Peng, Yu Cao, Dazhao Cheng, Dacheng Tao
B. Liu and D. Cheng are with the School of Computer Science, Wuhan University,
Wuhan, China (e-mail<EMAIL_ADDRESS>dcheng@whu.edu.cn). L. Ding and D.
Tao are with The University of Sydney, Australia. (e-mail:
<EMAIL_ADDRESS>dacheng.tao@gmail.com). L. Shen and K. Peng are with
the JD Explore Academy, JD.com, Beijing, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Y. Cao is with the Tencent IEG, Shenzhen, China
(e-mail: rainyucao@tencent.com). This work was done during Boan’s internship
at JD Explore Academy. Our source code is publicly available at
https://github.com/XXX

###### Abstract

The Mixture of Experts (MoE) has emerged as a highly successful technique in
deep learning, based on the principle of divide-and-conquer to maximize model
capacity without significant additional computational cost. Even in the era of
large-scale language models (LLMs), MoE continues to play a crucial role, as
some researchers have indicated that GPT-4 adopts the MoE structure to ensure
diverse inference results. However, MoE is susceptible to performance
degeneracy, particularly evident in the issues of imbalance and homogeneous
representation among experts. While previous studies have extensively
addressed the problem of imbalance, the challenge of homogeneous
representation remains unresolved. In this study, we shed light on the
homogeneous representation problem, wherein experts in the MoE fail to
specialize and lack diversity, leading to frustratingly high similarities in
their representations (up to 99% in a well-performed MoE model). This problem
restricts the expressive power of the MoE and, we argue, contradicts its
original intention. To tackle this issue, we propose a straightforward yet
highly effective solution: OMoE, an orthogonal expert optimizer. Additionally,
we introduce an alternating training strategy that encourages each expert to
update in a direction orthogonal to the subspace spanned by other experts. Our
algorithm facilitates MoE training in two key ways: firstly, it explicitly
enhances representation diversity, and secondly, it implicitly fosters
interaction between experts during orthogonal weights computation. Through
extensive experiments, we demonstrate that our proposed optimization algorithm
significantly improves the performance of fine-tuning the MoE model on the
GLUE benchmark, SuperGLUE benchmark, question-answering task, and name entity
recognition tasks.

###### Index Terms:

Mixture of Expert, Orthogonal Optimization, Model Interpretability.

##  I Introduction

The Mixture of Experts (MoE) [1] is a widely adopted machine learning
technique that harnesses the collective intelligence of multiple models,
referred to as experts, to arrive at a final prediction. In essence, each
expert specializes in a specific area, and their combination is achieved
through a gating network, which selects the most suitable expert based on
input characteristics. The gating network does this by calculating a weighted
sum of the experts’ predictions and selecting the expert or few experts with
the most relevant expertise. Compared to relying on a single, general-purpose
model, this approach has demonstrated significant improvements in prediction
accuracy. Recently, MoE [2] has undergone significant advancements, resulting
in the development of MoE layers [3, 4, 5, 6]. These layers can be
incorporated into deep learning architectures and are capable of dynamically
routing inputs to the most appropriate expert. Consequently, the MoE has found
widespread application in various domains, including machine translation [7,
8, 9], sentiment analysis [10, 11], dialogue [12, 13], and natural language
generation [14]. By leveraging the strengths of multiple models, MoE has
become a go-to approach for numerous machine learning tasks for improving
performance.

Figure 1: The overview of OMoE optimizer. ① After being selected by the Gating
Function, the input is sent to different experts. ② Experts calculate their
own corresponding orthogonal projector based on its input. ③ Based on the
orthogonal projectors of the other experts (e.g. blue expert), the current
expert to be updated (e.g. red expert) calculates the average projector. The
average projector represents the orthogonal subspace of other experts. ④ Using
the projector calculated in the previous step, the parameters are updated in
the orthogonal direction of the other experts.

While the MoE has shown promising results in enhancing machine learning
models’ accuracy, it is prone to performance degeneracy [15], which refers to
a phenomenon where the MoE fails to outperform a single model with fewer
parameters. There are two forms of degeneracy, and the first form arises when
only one expert dominates, resulting in the imbalance problem where the rich
get richer and the mixture ceases to exist, undermining the collective nature
of the mixture. To address this issue, researchers have explored the use of
gating functions [16, 17, 18] and regularization techniques such as weight
decay on loss functions [18] to alleviate the imbalance. The second form of
degeneracy occurs when the experts fail to specialize and lack diversity,
leading to the homogeneous representation problem. Researchers [19] propose a
parameter-efficient MoE architecture that shares partial parameters among all
experts to reduce memory consumption while providing auxiliary parameters for
each expert to ensure diversity. This approach has been shown to improve the
accuracy of MoE models and help mitigate degeneracy issues, highlighting the
importance of diversity among experts for achieving higher performance. While
degeneracy is a challenge in MoE models, continued research and development
are helping to overcome these issues and further improve the accuracy of
machine learning models.

Although several studies have addressed the issue of imbalanced expert load in
MoE models, the second type of degeneracy caused by the homogeneous
representation problem remains unresolved [17, 20]. This degeneracy arises
when all experts learn the same distribution, with their parameters residing
in the same subspace, leading to the parameter redundancy and overfitting
issues during training.

To tackle the limited diversity among experts in MoE models, we propose a
novel optimizer, the OMoE Optimizer, rather than focusing solely on designing
a balanced routing strategy. The OMoE optimizer aims to enhance diversity
among experts and improve their overall performance. Our approach divides the
training process into two alternating phases, each employing a specific
optimizer. In the first accumulating phase, a standard optimizer such as SGD,
Adam [21], or AdamW [22] is used to update the model parameters, excluding the
experts in MoE. Concurrently, the OMoE optimizer collects input data for
subsequent utilization. In the second orthogonal phase, the OMoE optimizer
updates the orthogonal projector based on the accumulated inputs, as depicted
in Figure 1. For each expert requiring an update, the OMoE optimizer
calculates the average projector and employs gradient-based optimization
techniques, such as SGD, to refine the expert’s parameters using the updated
projector. To assess the efficacy of our proposed method, we conducted
comprehensive experiments involving fine-tuning pre-trained language models on
the GLUE benchmark, SuperGLUE benchmark, Question-Answering (QA) task, and
Name Entity Recognition (NER) task. The experimental results validate the
effectiveness of our approach in enhancing the diversity of experts in MoE
models, leading to superior performance when compared to existing
methodologies. In summary, our research introduces a pioneering approach to
address the issue of limited diversity among experts in MoE models. Through
the introduction of the OMoE optimizer, we furnish machine learning
practitioners with a novel tool to enhance prediction accuracy and drive
advancements in the field of MoE research.

Our contributions are summarized as follows:

  * •

We identify the crucial issue of degeneracy, that arises from the homogeneous
representation problem in MoE models. This issue, overlooked by existing
methodologies, results in a failure of experts to specialize. Alongside the
degeneracy caused by the imbalance problem that is well-explored in prior
works, both forms of degeneracy significantly deteriorate the performance of
MoE models.

  * •

Drawing inspiration from recent advancements in continual learning, we propose
the OMoE Optimizer along with an alternative training approach to effectively
enlarge the diversity among experts in MoE, which establishes a novel
connection between tasks in continual learning and mini-batches in MoE
training. Our approach involves two distinct phases: the accumulating phase
and the orthogonal phase, wherein two optimizers update the model
alternatively.

  * •

We evaluate the proposed OMoE upon several language models in fine-tuning the
GLUE benchmark, SuperGLUE benchmark, QA task, and NER task extensively,
demonstrating the superior performance of our method. In addition, we conduct
ablation studies on two key factors: The skipping step and the number of
experts. Finally, we discuss the relationship between diversity and
performance.

##  II Background

###  II-A Mixture of Expert

We first introduce the MoE architecture, which has been used to improve the
model capacity of transformer-based models. An MoE model contains a set of
expert network $E_{1},E_{2},......$ and a gating function $G$. Each expert
specializes in different inputs that are decided by the gating function:

$MoE(x)=\sum_{i=1}^{k}g_{i}(x)\cdot f_{i}(x),$ (1)

where $f_{i}(x)$ indicates the output produced by the expert $i$. The gating
function can be implemented as a random selector or a neural network. In the
context of attention-based models, the experts are represented by the
2-layers-FFN (feed-forward neural network) in transformer layers. When
processing an input token, only a few experts or a single expert will be
activated, indicating the relevance of that expert to the given token. This
selection process is determined by the gating function, which is responsible
for deciding which expert is best suited to handle the given input. Overall,
MoE allows attention-based models to efficiently process input sequences by
selectively activating a small subset of experts for each token, reducing the
computational complexity and improving the overall performance of the model.

TABLE I:  The variance of the parameters over experts in BERT-MoE.  Dataset |  AdamW |  OMoE
---|---|---
CoLA |  3.01E-06 |  3.07E-06
SST-2 |  1.63E-05 |  1.74E-05
MRPC |  5.12E-05 |  6.52E-05
STS-B |  1.76E-06 |  1.92E-06
QQP |  8.98E-05 |  3.11E-04
MNLI |  9.03E-05 |  2.08E-04
QNLI |  2.38E-05 |  2.95E-05
RTE |  7.23E-05 |  1.67E-04
TABLE II:  The variance of the parameters over experts in RoBERTa-MoE.  Dataset |  AdamW |  OMoE
---|---|---
CoLA |  9.30E-06 |  1.69E-05
SST-2 |  1.34E-05 |  2.58E-05
MRPC |  5.87E-05 |  7.32E-05
STS-B |  1.93E-06 |  2.54E-06
QQP |  8.17E-05 |  9.39E-05
MNLI |  3.95E-05 |  4.45E-05
QNLI |  5.47E-05 |  9.11E-05
RTE |  4.19E-05 |  1.52E-04

###  II-B Orthogonal Weights Modification (OWM)

OWM (Orthogonal Weight Modification) [23] is a learning algorithm that aims to
address the issue of catastrophic forgetting in continual learning.
Catastrophic forgetting occurs when a model trained on new tasks interferes
with previously learned knowledge, resulting in a significant decline in the
performance of the previously learned tasks. OWM was developed to ensure that
when training a network for new tasks, its weights can only be modified in a
direction that is orthogonal to the subspace spanned by all previously learned
inputs, which is called the input space. As a result, when combined with
gradient descent-based search, OWM enables the network to learn new tasks
while maintaining the performance of previously learned tasks. This algorithm
ensures that the updates in the weight space for a new task do not interfere
with the existing knowledge learned from previous tasks. OWM helps to improve
the generalization ability of the model by ensuring that the network captures
task-specific features while preserving previously learned knowledge. A
crucial step in protecting existing knowledge during network training is to
create an appropriate orthogonal projection that operates in a space
perpendicular to the one defined by previous inputs. This enables the network
to shield previously acquired knowledge effectively. OWM constructs the
orthogonal projector [24, 25, 26] as $A(A^{T}A)^{-1}A^{T}$, where $A$ contains
previous inputs as columns. To avoid the matrix invertibility problem, OWM
adds a small constant $\alpha$. So the projector will be $A(A^{T}A+\alpha
I)^{-1}A^{T}$. However, as $A$ includes all previously trained input vectors,
to update $P$, we need to recalculate it after incorporating the new input
into $A$. Therefore, OWM transform the projector in OWM to RLS projector. RLS
algorithm [26] is a recursive algorithm that solves the least squares problem.
At each iteration when new data sample is available, the weights of projector
are updated, leading to savings in computations. Specifically, in RLS
algorithm, projector $P^{\prime}$ will be:

$\displaystyle
P^{\prime}=\left(\sum_{i=1}^{n}\mathbf{x}(i)\mathbf{x}^{T}(i)+\alpha
I\right)^{-1}$ (2)

Equation 2 defines the projector used in RLS. To avoid computing the inverse
of $\sum_{i=1}^{n}\mathbf{x}(i)\mathbf{x}^{T}(i)+\alpha I$, we define the the
correlation matrix $\Phi$ of the input signals, which is the inversion of
$P^{\prime}$:

$\displaystyle\Phi(n)=\sum_{i=1}^{n}\gamma^{n-i}\mathbf{x}(i)\mathbf{x}(i)^{T}+\alpha\gamma^{n}I,$
(3)

where $\gamma$ is the forgetting factor and we assure $\gamma=1$. Projector in
OWM contains $A(n)=[x(1),x(2),...,x(n)]$, where $x(i)$ is the vector recording
the $i^{th}$ input. So $\Phi$ can also be written as:

$\displaystyle\Phi(n)=A(n)A^{T}(n)+\alpha I$ (4)

Based on Woodbury matrix identity, $P^{\prime}$ can be written as:

$\displaystyle P^{\prime}$ $\displaystyle={\Phi(n)}^{-1}$ (5)
$\displaystyle=\alpha^{-1}I-\alpha^{-1}A\left(I+\alpha^{-1}A^{T}A\right)^{-1}A^{T}\alpha^{-1}$
$\displaystyle=\alpha^{-1}\left[I-A\left(\alpha
I+A^{T}A\right)^{-1}A^{T}\right]$ $\displaystyle=P$

Therefore, projector in OWM is equivalent to the projector in RLS. We will use
recursive method in RLS to update $P$.

Specifically, we consider a feed-forward network of $L+1$ layers. The
orthogonal projector of layer $l$ is initialized as $\textbf{I}_{l}$, where
$\textbf{I}_{l}$ refers to a unit matrix and will be updated when one task is
finished. For the $i^{th}$ batch during training the $j^{th}$ task, the
orthogonal projector of layer $l$ after training $j-1$ tasks is represented
as:

$\displaystyle\mathbf{P}_{l}(i,j)=\mathbf{P}_{l}(i-1,j)$ (6)
$\displaystyle-\mathbf{k}_{l}(i,j)\overline{\mathbf{x}}_{l-1}(i,j)^{T}\mathbf{P}_{l}(i-1,j)$

$\displaystyle\mathbf{k}_{l}(i,j)=\mathbf{P}_{l}(i-1,j)\overline{\mathbf{x}}_{l-1}(i,j)/$
(7)
$\displaystyle\left[\alpha+\overline{\mathbf{x}}_{l-1}(i,j)^{T}\mathbf{P}_{l}(i-1,j)\overline{\mathbf{x}}_{l-1}(i,j)\right],$

where $\mathbf{x}_{l-1}$ is the output of the ${l-1}^{th}$ layer in response
to the mean of the inputs in the $i^{th}$ batch of $j^{th}$ task. $\alpha$
refers to decaying factor as $\alpha(i,j)=\alpha_{0}\lambda^{i/j}$ for the
$i^{th}$ batch of data in the $j^{th}$ task.

The orthogonal projector pushes the parameters updated to the orthogonal
direction of the previous tasks. By using gradient descent to search for a
suitable weight configuration, the OWM helps the network to learn new tasks
without compromising the performance of tasks it has already learned. Combined
with SGD, parameter updating will be:

$\displaystyle\mathbf{W}_{l}(i,j)$ $\displaystyle=\mathbf{W}_{l}(i-1,j)$ (8)
$\displaystyle+\kappa(i,j)\Delta\mathbf{W}_{l}^{BP}(i,j)\quad\text{ if }\quad
j=1$

$\displaystyle\mathbf{W}_{l}(i,j)$ $\displaystyle=\mathbf{W}_{l}(i-1,j)$ (9)
$\displaystyle+\kappa(i,j)\mathbf{P}_{l}(j-1)\Delta\mathbf{W}_{l}^{BP}(i,j)\quad$
$\displaystyle if\quad j=2,3,\cdots,$

where $\mathbf{W}_{l}(i,j)$ is the $l^{th}$ layer after training by $j-1$
tasks and $i$ batches in $j^{th}$ task, $\kappa(i,j)$ is the learning rate,
$\Delta\mathbf{W}_{l}^{BP}(i,j)$ is the gradient calculated by $i^{th}$ batch
in the $j^{th}$ task. In general, the procedure of OWM will be:

  1. 1.

Initialization of parameters: initialize $\mathbf{W}_{l}(0)$ and
$\mathbf{P}_{l}(0)$

  2. 2.

Forward propagate the inputs of $i^{th}$ batch in the $j^{th}$ task. then back
propagate the errors and calculate weight modification
$\Delta\mathbf{W}_{l}^{BP}(i,j)$

  3. 3.

Update the weight matrix in each layer by Equation 8 and Equation 9.

  4. 4.

Repeat steps 2) to 3) for the next batch.

  5. 5.

When $j^{th}$ task is finished, forward propagate the mean of the inputs for
each batch in the $j^{th}$ task successively. Then update $\mathbf{P}_{l}$ by
Equation 6 and Equation 7.

  6. 6.

Repeat steps 2) to 5) for the next task.

##  III Orthogonal Optimizer for MoE

###  III-A Degeneracy in MoE

Gao et al.[19] and Shen et al.[15] have identified several issues in MoE
architecture, including the degeneracy caused by imbalance problem, and
homogenous representation problem. While the imbalance problem has been
explored in previous research, we specifically focus on the homogenous
representation problem in this study. Our experiments also confirm the
existence of this issue, as illustrated in Table I and Table II. Table I
displays the similarity between the experts in the MoE architecture after
fine-tuning the BERT model on the GLUE dataset. The weights of the experts are
directly copied from the pre-trained model. The small variation in the
similarity scores indicates that the experts in the MoE architecture have not
learned diverse knowledge that is unique to specific inputs. We consider
parameters to be similar if their differences at the same position in
different experts are below a certain threshold ($threshold=1E-3$). Our
analysis reveals that the percentage of similar parameters in the fine-tuned
model exceeds 99%. Consequently, this often leads to MoE models
underperforming compared to the original model.

Figure 2: The full training process of OMoE. OMoE consists of two optimizers:
the base optimizer (the blue Optimizer in the figure) and the OWM optimizer
(the red OWM-Optimizer in the figure). The training process also consists of 2
kinds of alternative steps: R Step (correspondents to the accumulating phase)
and O Step (correspondents to the orthogonal phase). In R Step,
$\Delta\mathbf{W}_{l}^{BP}(i)$ directly guides the update of parameters
$\theta$ and $\phi$. In O Step, average orthogonal projector
$\mathbf{\overline{P}}^{m}_{l}(i)$ is calculated based on average input
$\overline{\mathbf{x}}_{l-1}(i)$ and then guide the gradient to be orthogonal.
The two optimizers share the states like momentum and hyperparameters.

###  III-B OWM for Experts

In this section, we present a comprehensive overview of the OMoE algorithm,
specifically designed to optimize the experts in MoE architecture. Taking
inspiration from the successful Orthogonal Weight Modification (OWM) technique
in multitask learning, we adapt and apply it to MoE models. In MoE, each
expert specializes in a specific subset of the dataset, akin to individual
tasks in multitask learning. Therefore, we introduce OMoE, an expert optimizer
for MoE, based on the OWM approach. In order to align with the concept of
tasks in multitask learning, we divide the training process into two distinct
phases: the accumulating phase and the orthogonal phase.

During the accumulating phase, the model undergoes regular updates, while the
optimizer simultaneously accumulates representations that capture the subspace
associated with each expert. These representations are learned by observing
the inputs of all experts within the same mini-batch. In essence, the
optimizer computes the input from all experts and utilizes it to represent the
subspace specific to each expert.

During the orthogonal phase, the parameters of each expert are updated in a
direction that is orthogonal to the subspace defined by the previously learned
inputs of other experts. This guarantees that each expert captures distinct
and non-overlapping facets of the data. To prevent updates in an orthogonal
direction that is unrelated to the representative subspace, experts first
acquire initial knowledge in the accumulating phase before entering the
orthogonal phase.

To summarize, our OMoE adopts an alternating training strategy of different
optimizers, which allows experts to update their parameters in orthogonal
directions from the subspaces of other experts, resulting in proper diversity
of experts and improved model performance. The accumulating phase corresponds
to the R(egular) Step in the updating process, and the orthogonal phase
corresponds to the O(WM) Step.

R Step: update all parameters by other optimizers like Adam:

$\displaystyle\mathbf{W}_{l}(i;\theta,\phi)$
$\displaystyle=\mathbf{W}_{l}(i-1;\theta,\phi)$ (10)
$\displaystyle+\kappa(i)\Delta\mathbf{W}_{l}^{BP}(i;\theta,\phi)$

In the R Step, all parameters, $\theta$ and $\phi$, will be updated. To be
more specific, parameters other than the experts (i.e., $\phi$) will be
updated using the standard optimizer, while all experts (i.e., $\theta$) will
accumulate input via the optimizer. The subspace spanned by these inputs will
guide the update of the projector during the O step, which in turn assists
with the updates to the experts.

O Step: For the $i^{th}$ update step in a accumulate phase, layer $l$ and an
expert $\theta^{m}$ belongs to all $M$ experts, we have:

$\displaystyle\mathbf{W}_{l}(i;\theta^{m})$
$\displaystyle=\mathbf{W}_{l}(i-1;\theta^{m})$ (11)
$\displaystyle+\kappa(i)\mathbf{\overline{P}}_{l}^{m}(i)\Delta\mathbf{W}_{l}^{BP}(i;\theta^{m})$
$\displaystyle\mathbf{\overline{P}}^{m}_{l}(i)=\frac{1}{M}\sum_{j=1}^{M}{\mathbf{P}^{j}_{l}(i)},j\neq
m$ (12)

where $\kappa(i)$ is the learning rate in $i^{th}$ update step.
${\overline{P}}^{m}_{l}(i)$ is calculated by the average orthogonal projector
of other experts except $m$. We can get an orthogonal projector of expert $m$
by:

$\displaystyle\mathbf{P}^{m}_{l}(i)$ $\displaystyle=\mathbf{P}^{m}_{l}(i-1)$
(13)
$\displaystyle-\mathbf{k}^{m}_{l}(i)\overline{\mathbf{x}}_{l-1}(\hat{i})^{T}\mathbf{P}^{m}_{l}(i-1)$

$\displaystyle\mathbf{k}^{m}_{l}(i)$
$\displaystyle=\mathbf{P}^{m}_{l}(i-1)\overline{\mathbf{x}}_{l-1}(\hat{i})/$
(14)
$\displaystyle\left[\alpha+\overline{\mathbf{x}}_{l-1}(\hat{i})^{T}\mathbf{P}^{m}_{l}(i-1)\overline{\mathbf{x}}_{l-1}(\hat{i})\right],$

where $\alpha$ decaying as $\alpha_{0}\lambda^{i/{n}}$ for $i$th mini-batch in
all $n$ batches. Noted that $i-s\leq\hat{i}\leq i$. The project must be
updated for all accumulating phase. In O Step, only $\theta$ will be updated.

Hidden states are shared between the ordinary optimizer and our OMoE
optimizer. We define skipping step $s$ in Algorithm 3. The algorithm makes O
updates every $s$ epoch while always taking R steps. The capacity of the
projector limits the skipping step to being too small. Let us consider a task
that has $U$ update steps. The capacity of the projector can be viewed as the
rank of $\mathbf{P}_{i}$, where i is the $0\leq i\leq U/s$. In O Step, we
update $\mathbf{P}_{i}$ by
$\mathbf{P}_{i+1}=\mathbf{P}_{i}-\Delta\mathbf{P}_{i+1}$. In ideal condition
that every phase is independent,
$\textit{range}(\mathbf{P}_{i})\bigcap\textit{range}(\Delta\mathbf{P}_{i})=\emptyset$.
Therefore
$\textit{rank}(\mathbf{P}_{i+1})=\textit{rank}(\mathbf{P}_{i})-\textit{rank}(\Delta\mathbf{P}_{i+1})$.
It is obvious that with $i$ increase, the $\mathbf{P}_{i}$ becomes close to 0
and if $\mathbf{P}_{i}$ is 0, the projector will not be able to represent the
subspace. So $s$ can not be set too small. In experiments, we set $s$ as 5.

Algorithm 1 The process of O Step for OMoE, with with optimizer $\mathcal{O}$

1:procedure OWM-O(X;$\mathcal{O}$)

2: $Z_{MoE}=\sum_{i=1}^{k}g_{i}(x)\cdot f_{i}(x;\theta)$

3: $Z=F(Z_{MoE};\phi)$

4: Forward propagation with $Z$ and calculate $L$.

5: Calculate the gradient $\Delta\mathbf{W}_{l}^{BP}(i;\theta,\phi)$ with back
propagation for layer $i$.

6: Update the projector by all accumulated inputs:
$\mathbf{P}^{m}_{l}(i)=\mathbf{P}_{l}(i-1)-\mathbf{k}^{m}_{l}(i)\overline{\mathbf{x}}_{l-1}(\hat{i})^{T}\mathbf{P}^{m}_{l}(i-1)$

7: Calculate the projector by averaging the projector of other experts:
$\mathbf{\overline{P}}^{m}_{l}(i)=\frac{1}{M}\sum_{j=1}^{M}{\mathbf{P}^{j}_{l}(i)},j\neq
m$

8:
$\theta\leftarrow\theta-\eta\cdot\mathbf{\overline{P}}^{m}_{l}(i)\cdot\Delta\mathbf{W}_{l}^{BP}(i;\theta)$
$\triangleright$ Update experts parameters

9:end procedure

Algorithm 2 The process of R Step for OMoE, with optimizer $\mathcal{R}$

1:procedure OWM-R(X;$\mathcal{R}$)

2: $Z_{MoE}=\sum_{i=1}^{k}g_{i}(x)\cdot f_{i}(x;\theta)$

3: $Z=F(Z_{MoE};\phi)$

4: Forward propagation with $Z$ and calculate $L$.

5: Calculate the gradient $\Delta\mathbf{W}_{l}^{BP}(i;\theta,\phi)$ with back
propagation for layer $l$.

6: $\theta\leftarrow\theta-\eta\cdot\Delta\mathbf{W}_{l}^{BP}(i;\theta,\phi)$

7: $\phi\leftarrow\phi-\eta\cdot\Delta\mathbf{W}_{l}^{BP}(i;\theta,\phi)$
$\triangleright$ Update all parameters

8:end procedure

Algorithm 3 Training process of OMoE, at mini-batch $e$, with data
$\mathcal{D}$, skipping step $s$, OMoE optimizer $\mathcal{O}$ and regular
optimizer $\mathcal{R}$

1:procedure Opti(X, $s$, $e$, $\mathcal{O}$, $\mathcal{R}$)

2: for X in $\mathcal{D}$ do

3: if $e$ mod $s$ = 0 then

4: OMoE-O(X) $\triangleright$ Do orthogonal update

5: else

6: OMoE-R(X)

7: $\mathcal{O}$.add(X) $\triangleright$ Accumulate Input in OMoE optimizer

8: end if

9: end for

10:end procedure

##  IV Experiment

In this section, we empirically evaluate OMoE by fine-tuning the pre-trained
language model on various benchmarks and tasks. Firstly, we introduce the
compared base models. Then, we evaluate our proposed method in the GLUE
benchmark [27], the SuperGLUE benchmark [28], the QA task [29], and the NER
task [30].

###  IV-A Compared Models

The proposed OMoE is evaluated on three language models:

  * •

BERT [31] is a transformer-based pre-trained model with 12 encoders with 12
bidirectional self-attention heads. The MoE version of the BERT base model
composes 4 experts for each layer.

  * •

RoBERTa [32] builds on BERT’s language masking strategy and modifies key
hyperparameters in BERT, including removing BERT’s next-sentence pre-training
objective and training with much larger mini-batches and learning rates.

  * •

ALBERT [33], which is known as A Lite BERT, employs a number of innovative
techniques, such as factorized embedding parameterization, cross-layer
parameter sharing, and parameter-sharing across layers, to reduce the number
of parameters required by the model and improve its efficiency, while
maintaining or even surpassing the performance of BERT.

Specifically, to evaluate the effectiveness of OMoE, we conduct experiments on
three popular transformer-based models: BERT-base, RoBERTa-base, and ALBERT-
base-v2. To transform these models into MoE models, we replace all layers with
MoE layers consisting of four experts. To initialize the experts, we follow
the common practice in previous works [34, 35, 19] by replicating the pre-
trained weights of the feed-forward network.

To facilitate effective collaboration among the experts, we introduce an
additional network that functions as a gating function. We adopt the token-
gate routing strategy, which assigns each token to a specific expert based on
the gating function’s output.

Our evaluation focuses on the fine-tuning performance of the models on several
benchmark datasets, including GLUE, SQuAD, SuperGLUE, and CoNLL-2003. For each
dataset, we train the model with a specific set of hyperparameters and
evaluate the performance on the test set. We compare the results of our
proposed OMoE method with the standard MoE. We use 8 Nvidia A100 GPUs and
HuggingFace Transformers library to fine-tune the model.

###  IV-B Fine-tune on GLUE

Dataset. GLUE (General Language Understanding Evaluation) [27] is a collection
of natural language processing tasks designed to evaluate the performance of
language models. It consists of diverse tasks that cover a wide range of
language understanding capabilities, including reading comprehension,
classification, and natural language inference.

TABLE III:  Summary of the GLUE dataset.  Task | Task Type | Examples
---|---|---
CoLA |  Single-sentence classification | 8,551 train / 1,043 test
SST-2 | 67,349 train / 872 test
MRPC |  Similarity and paraphrase detection | 3,668 train / 1,725 test
QQP | 363,849 train / 40,430 test
STS-B | 7,621 train / 1,431 test
MNLI |  Natural language inference | 393,462 train / 20,000 test
QNLI | 104,743 train / 5,463 test
RTE | 2,490 train / 2,568 test
WNLI | 634 train / 71 test

Result. The experimental results of our proposed OMoE optimizer on the GLUE
dataset are presented in Table IV, where we compare its performance with the
AdamW optimizer. The results show that our OMoE optimizer outperforms the
baseline on 7 out of 8 datasets, achieving an average score of 83.30 on BERT,
83.86 on RoBERTa, and 83.58 on ALBERT. In contrast to the AdamW optimizer, our
OMoE optimizer effectively enlarges the diversity among experts, leading to an
improvement of 0.62, 0.9, and 0.66 for the three models, respectively.
However, we surprisingly observe degradation in performance for some low-
resource tasks such as MPRC and CoLA. Upon analysis, we found that these tasks
are particularly challenging for the experts in MoE to extract useful
knowledge. Therefore, the OMoE optimizer cannot accurately identify the input
subspace and hence cannot guarantee improvement for these tasks. Overall, our
results demonstrate that our proposed OMoE optimizer is effective in improving
the performance of MoE models, but further investigation is needed for
improving performance on challenging low-resource tasks.

Similarity over Experts. As MoE-based Transformers are relatively new and
there are limited publicly available pre-trained models, there is currently no
standardized evaluation method for assessing the similarity over experts. In
this study, we propose two metrics to evaluate the diversity among experts in
the MoE model. Firstly, we calculate the percentage of different parameters
between two experts, which provides a quantitative measure of how distinct the
two experts are from each other. Specifically, we randomly select two experts
and compute the ratio of the different parameters to the total number of
parameters. A higher percentage indicates that the two experts are more
dissimilar. Secondly, we use variance as a metric to assess the diversity
among all the experts. We calculate the variance of the outputs of all experts
for a given input sample. A higher variance suggests that the experts are more
diverse in their predictions for the given input, which is desirable for the
MoE model. These two metrics provide complementary information on the
diversity among experts and can help to diagnose the degeneracy problems in
MoE models.

Figure. 3 shows the percentage of different parameters. We calculated the
difference over expert parameters in the model optimized by OMoE and AdamW at
the same position. Based on the difference, we recorded the proportion of
parameters that has a more significant difference compared with the baseline.
We define the proportion as diverse degree, indicating the gap between experts
compared with the baseline. We can see that in all tasks, the diverse degree
is improved compared to the baseline. Among them, the diverse degree in the
QQP and MNLI tasks is more considerable, showing the powerful role of the OMoE
algorithm in improving the difference for parameters. We also analyze the
variance of experts. Table I and Table II show the variance of all experts
relative to the mean value. Compared to the AdamW optimizer, our optimizer
effectively enhances the variance of the expert, thereby improving the
performance of the MoE architecture.

It is worth noting that the greater difference between experts does not
necessarily mean better model performance. For example, in the QQP and MNLI
tasks, although progress was made in terms of expert differences, the
performance improvement was less significant than in other tasks. One possible
explanation is that high-resource tasks can take advantage of a large number
of parameters in a sparse model, thereby improving model performance without
increasing expert differences. We will discuss the relationship between
similarity and performance in Section V-A.

Figure 3: The figure shows to what extent OMoE expands the difference between
experts. The first 3 columns are for experts in BERT while the other 3 columns
are for experts in RoBERTa. The depth of color in the figure represents the
percentage of parameters with a larger difference in OMoE compared to AdamW.
TABLE IV:  Results on GLUE. CoLA uses Matthews correlation coefficient (MCC)
as the metrics while others use Accuracy. Better results are in bold. We
select 3 models for evaluation: BERT, RoBERTa and ALBERT. Layers in all three
models are replaced with four-experts-MoE-layers. AdamW is used as the base
optimizer.

| MNLI | QNLI | QQP | RTE | SST-2 | MRPC | CoLA | STS-B | Avg
---|---|---|---|---|---|---|---|---|---
MoE for $BERT_{Base}$
BERT-MoEAdamW | 83.99 | 90.54 | 90.62 | 75.81 | 90.83 | 87.22 | 53.39 | 89.05 | 82.68
BERT-MoEOMoE | 84.31 | 90.46 | 90.98 | 77.62 | 91.51 | 87.47 | 54.69 | 89.39 | 83.30⇑+0.62
MoE for $RoBERTa_{Base}$
RoBERTa-MoEAdamW | 84.72 | 90.58 | 90.91 | 74.73 | 91.62 | 88.12 | 53.39 | 89.66 | 82.96
RoBERTa-MoEOMoE | 85.42 | 90.70 | 91.36 | 79.03 | 92.64 | 87.82 | 53.9 | 90.04 | 83.86⇑+0.9
MoE for $ALBERT_{Base}$
ALBERT-MoEAdamW | 84.81 | 91.37 | 88.73 | 74.25 | 90.93 | 88.01 | 54.95 | 90.32 | 82.92
ALBERT-MoEOMoE | 85.23 | 91.72 | 88.91 | 76.61 | 92.18 | 88.43 | 54.95 | 90.68 | 83.58⇑+0.66

TABLE V:  Performance on SuperGLUE dataset of different pretrained models,
e.g. BERT and RoBERTa.

| COPA | BoolQ | MultiRC | WiC | CB
---|---|---|---|---|---
MoE for $BERT_{Base}$
BERT-Dense | 70.0 | 68.9 | 59.6 | 63.2 | 64.3
BERT-MoEAdamW | 69.2 | 69.3 | 58.3 | 63.5 | 63.4
BERT-MoEOMoE | 70.9 | 70.0 | 59.8 | 63.5 | 63.8
MoE for $RoBERTa_{Base}$
RoBERTa-Dense | 46.4 | 77.6 | 61.2 | 65.4 | 67.1
RoBERTa-MoEAdamW | 46.3 | 75.1 | 63.2 | 65.7 | 66.4
RoBERTa-MoEOMoE | 48.2 | 76.4 | 63.9 | 66.5 | 67.6

###  IV-C Fine-tune on SuperGLUE

Dataset. SuperGLUE [28] (General Language Understanding Evaluation) is a
benchmark dataset for evaluating the performance of NLU models on a diverse
set of challenging language understanding tasks. It consists of eight tasks
that require reasoning beyond superficial patterns and evaluate a model’s
ability to perform a range of language understanding tasks, such as natural
language inference, question answering, and coreference resolution. We follow
[36] to select 5 tasks (COPA [37], BQ [38], MultiRC [39], WiC [40] and CB
[41]) in SuperGLUE dataset to evaluate OMoE.

Result. We evaluate OMoE based on BERT and RoBERTa on SuperGLUE benchmark. As
shown on Table V. The comparison of our results to AdamW suggests that our
approach represents a significant improvement over previous state-of-the-art
methods. For example, our OMoE optimizer achieves 1.7 increase in accuracy on
the COPA task. Our method also yields comparable results in other experiments.

###  IV-D Fine-tune on Question-Answering

Dataset. SQuAD1.1 [29]. Generated by crowd workers and designed to test the
ability of a QA system to understand and interpret natural language text,
SQuAD1.1 is a dataset for training and evaluating question-answering systems.
It is based on a set of over 100,000 questions from Wikipedia articles, along
with their corresponding answers. SQuAD1.1 is widely used in natural language
processing to train and evaluate QA models.

TABLE VI: F1 scores result on SQuAD1.1.  Model | AdamW | OMoE
---|---|---
BERT | 87.9 | /
RoBERTa | 89.6 | /
BERT-MoE | 88.4 | 88.8⇑+0.4
RoBERTa-MoE | 89.3 | 89.9⇑+0.6

Table VI presents the evaluation results of our OMoE algorithm on two popular
pre-trained language models, BERT and RoBERTa, with MoE layers. To evaluate
the performance, we fine-tuned the models on the SQuAD1.1 dataset, which is a
widely used benchmark for question-answering tasks. Our approach demonstrates
the effectiveness of our optimizer in improving the representation capability
of different experts. The results showed that our model achieved an exact
match improvement of over 0.4% on both BERT and RoBERTa models compared to the
baseline models with MoE layers. These improvements are significant and
demonstrate the effectiveness of our proposed OMoE algorithm in improving the
performance of MoE models.

###  IV-E Fine-tune on Named Entity Recognition

Dataset. CoNLL-2003 [30] is a shared task for named entity recognition and
multi-lingual named entity recognition. In this task, participants are
required to develop systems that can identify and classify named entities in
English and German language texts. Named entities include names of persons,
organizations, locations, and other miscellaneous entities like dates and
times. The goal is to automatically recognize these entities and their
categories in text. The task has become a benchmark for evaluating the
performance of named entity recognition systems, and the dataset is widely
used for research and development in this field.

Table VII presents the evaluation results of CoNLL-2003 dataset. Overall,
compared to AdamW optimizer, our OMoE achieves competitive performance for NER
tasks. By zooming in on specific model, OMoE for BERT-MoE achieves 0.2 gains
on Accuracy. Similarly, OMoE for RoBERTa-MoE and ALBERT-MoE performs better
than the state-of-art optimizer: AdamW. Results demonstrate the effectiveness
of OMoE in enlarging the diversity among expert and its performance on named
entity recognition tasks.

TABLE VII: Acc. Result on CoNLL-2003. Model | AdamW | OMoE
---|---|---
BERT | 98.4 | /
RoBERTa | 98.8 | /
ALBERTA | 98.5 | /
BERT-MoE | 98.4 | 98.6⇑+0.2
RoBERTa-MoE | 98.5 | 98.6⇑+0.1
ALBERT-MoE | 98.5 | 98.6⇑+0.1

##  V Analysis

In this section, we provide a detailed analysis of how the skipping step and
the number of experts affect the diversity and performance of models. We
explore the relationship between the diversity of experts and the performance
of MoE, and how our OMoE method addresses the issue of insufficient diversity
in MoE. Furthermore, we analyze the overhead of our proposed method, including
the additional storage space required for storing an extra optimizer, as well
as the computational complexity of our method. We compare these overheads with
the computational requirements of backpropagation and demonstrate that the
overhead introduced by OMoE is acceptable. Finally, we discuss the importance
of diversity in MoE models and how our method addresses this issue. We argue
that a lack of diversity can limit the performance of MoE, and we show how our
method increases the diversity of experts in MoE by allowing experts to update
their parameters in orthogonal directions from the subspaces of other experts.
By doing so, we ensure proper diversity of experts, leading to improved model
performance.

###  V-A Ablation Study: Effects of Skipping Step

Figure 4: The left figure shows the normalized variance of parameters in
experts with different skipping steps. The right figure shows the normalized
GLUE scores with different skipping steps.

We analyze how different skipping steps will affect the result, as shown in
Figure 4. As the skipping step increases, the variance shows a trend of
gradually decreasing, which also confirms our previous speculation: our method
can effectively increase the difference between experts. A smaller skipping
step indicates the OMoE optimizer is more frequently applied. By projecting
inputs from different training phases on the projector, the projector
preserves the subspace spanned by tokens processed by different experts,
effectively enhancing the representation ability of each expert. However, the
corresponding results show that increased differences between experts do not
necessarily mean better results. For example, in the CoLA task, the best
results appear when skipping step = 15. Some works [19] have confirmed the
role of similar parameters existing among different experts. Furthermore, the
additional parameters in the MoE layer possess a significant amount of
information capacity and have a great impact on model performance. The results
demonstrate this idea. Therefore, an alternating training strategy is
necessary.

###  V-B Ablation Study: Number of Experts

Figure 5: GLUE score improvement with different numbers of experts using OMoE
optimizer. In general, the benefit of orthogonal updates will become minor as
the number of experts increases,

The number of experts in the MoE architecture is a crucial hyperparameter that
affects the performance of the OMoE optimizer. We conducted experiments on
fine-tuning BERT by varying the number of experts and observed a highly
correlated relationship between the number of experts and the performance of
OMoE. As shown in Figure 5, we found that the network performance remains
acceptable until the number of experts is increased to a specific value,
beyond which the average subspaces become too complex to be orthogonal. This
suggests that the effectiveness of OMoE optimizer may saturate beyond a
certain number of experts, and adding more experts may not provide further
performance improvement.

Moreover, the increasing number of experts also means an increase in network
capacity, which can lead to overfitting. We noticed that the input sent to
each expert becomes fewer when there are more experts in the MoE architecture,
as the tokens within a mini-batch are fixed. This results in insufficient
input data being collected by the projector to represent the subspace
accurately. Hence, the OMoE optimizer may not work well when the number of
experts is too large, and the projector may fail to capture the necessary
information to ensure optimal performance.

###  V-C Ablation Study: Kind of Optimizer

Our proposed OMoE can improve the performance of various existing optimizers
on MoE networks. We selected AMSGrad and Adagrad optimizers and fine-tuned the
BERT model to demonstrate the generalizability of the OMoE optimizer. The
results in Table VIII show that the optimizers modified by OMoE perform better
than the unmodified ones. This indicates that our method is adaptable to
different optimizers. Additionally, OMoE can leverage the latest research
advances in optimizer techniques to continuously improve performance.

TABLE VIII: Different optimizers result of fine-tuning Bert on MRPC. Optimizer | Initial Version | OMoE Version
---|---|---
AdamW | 87.22 | 87.47
RMSProp | 87.15 | 87.31
Adagrad | 86.83 | 87.04

###  V-D Overhead Analysis

Our proposed OMoE optimizer requires additional storage space compared to
traditional optimizers, as well as additional computing overhead during
training. As shown in Table IX, the size of the OMoE optimizer and projector
is the same as that of all experts, leading to a modest increase in storage
requirements. However, this overhead is not significant (1.38$\times$) as the
state of the two optimizers can be shared, and only one projector needs to be
stored additionally.

In terms of computational overhead, the additional cost is 1.82GMacs (number
of multiply–accumulate operations) during the training process. This overhead
is acceptable compared to the high requirements of computational resources in
backpropagation. The computational complexity of our method is
$O(kN_{e}N^{2}_{w})$, where $k$ is the number of experts, $N_{e}$ is the
number of neurons per expert, and $N^{2}_{w}$ is the number of input weights
per neuron. Overall, the additional overhead introduced by our method is
affordable and does not significantly impact training efficiency.

TABLE IX: The overhead of our proposed method. T refers to the Transformer model. R refers to updating the model using R Step while O refers to updating the model using O Step. The FLOPs refers to additional FLOPs introduced by our method. Model | Optimizer | FLOPs | Memory
---|---|---|---
Model | Optimizer
T | R | 0 |  1$\times$ |  1$\times$
T-MoE | R | 0 |  2.55$\times$ |  1$\times$
T-MoE | R+O | 1.82GMacs |  2.55$\times$ |  1.38$\times$

###  V-E Discussion: Diversity and Performance

In the Motivation section, we pointed out that the unsatisfactory performance
of MoE is due to the excessive similarity over parameters of experts, which
contradicts the original intention of MoE. However, it does not necessarily
mean that the larger the difference between the parameters of the experts, the
better the performance of the MoE. There are several reasons for this.

Firstly, although the Gating Function assigns different tokens to different
experts, these tokens in the same dataset should follow the same distribution.
Therefore, the parameters of experts should be similar to a certain extent.
Moreover, overly different parameters can lead to instability and
inconsistency in the behavior of the MoE, which can further lead to poor
performance.

Secondly, in the OMoE algorithm, the O Step and R Step run alternately, and
experts do not always make orthogonal updates. This ensures that the
differences between experts should not expand infinitely, and prevents the
parameters of the experts from becoming too different. Furthermore, research
on the Gating Function has shown that the correctness of the router has a
significant impact on performance. If we make the parameters fully orthogonal
based on the possibly inaccurate routing result, it may violate the original
intention of MoE and lead to poor performance.

Overall, increasing the difference between the parameters of the experts
should not be the primary goal of MoE. Instead, the focus should be on finding
the optimal balance between the similarity and differences of the parameters
to achieve the best performance. The experiments have also confirmed this
view, showing that excessively different parameters can lead to performance
degradation.

##  VI Related Work

#### PLMs with MoE

MoE [1] was first proposed in 1991 and has been widely applied in various
fields [42, 43, 44, 45]. MoE Transformers represent a novel approach to
enhancing the performance of transformers. Unlike traditional feed forward
blocks, Sparse MoE blocks consist of two key components: a gating function [4]
and a collection of feed forward neural network experts [46]. The gating
function serves as the primary control mechanism for assigning tokens to
specific experts. It generates a sparse output that allocates each token to a
specific expert. This allocation is based on the token’s properties and the
expertise of the available experts. As a result, each expert specializes in a
particular type of token, which allows them to process those tokens more
effectively. After the advent of pre-trained language models [31, 32, 47, 48,
49, 50, 51, 52], the Gating-Expert architecture was quickly applied to PLMs
and has achieved many successes. Switch Transformer [18] uses a switch
mechanism to dynamically choose between different types of attention
mechanisms for improved performance on a variety of natural language
processing tasks. The 1 trillion parameter sparsely activated model shows
great performance on fine-tuning tasks. GLaM [53] is also a one-trillion-
number-of-trainable-parameters model, which is a family of decoder-only
language model. GLaM shows good performance on few-shot setting. GShard [54]
is also an MoE-based architecture that partitions the data and trains a
specialist model for each partition. BASE Layers [55] introduce a new balanced
assignment of experts layer for LLMs. The proposed sparse MoE layer guarantees
balanced compute loads and improves performance. These studies improve
training efficiency of PLMs with MoE by designing new routing strategies and
introducing more parameters. Our method is based on the PLMs with MoE and
improves the performance by enlarging the diversity of experts.

TABLE X: A sample of related PLMs with MoE. $n_{params}$ is the total number of trainable model parameters, $n_{act}$ is the number of activated model parameters per input token. Model Name | Model Type | $n_{params}$ | $n_{act}$
---|---|---|---
GShard-M4 | MoE Encoder-decoder | 600B | 1.5B
Switch-C | MoE Encoder-decoder | 1.5T | 1.5B
BASE Layers | MoE Decoder-only | 117B | 911M
GLaM (64B/64E) | MoE Decoder-only | 1.2T | 96.6B

#### Optimization for MoE

A more accurate gating mechanism is the key point to optimize MoE. Gating
which is learned via backpropagation [4, 54], perhaps with a regularizer to
encourage load balancing across experts, is the mainstream. Roller[16]
proposes a novel hash function-based gating layer, which requires no routing
parameters or extra terms in the objective function such as a load balancing
loss. Zhou [44] proposes a method that makes experts select the top-k tokens
instead of letting tokens select the top-k experts. As a result, each token
can be routed to a variable number of experts and each expert can have a fixed
bucket size. Fedus[18] simplifies the gating by selecting only the top expert
per token using a softmax over the hidden state and demonstrates better
scaling than previous work. Lample et.[56] introduce nearest neighbor search
in parameter selecting. Recently, THOR[17] finds that randomly-selected
experts also achieve satisfying results. THOR has demonstrated strong
performance on translation tasks. All works mentioned above tries to achieve
promising results. Another optimization for MoE is to make MoE faster. Recent
studies [57] apply knowledge distillation for faster inference. Switch
Transformer [18] tries to distill a fine-tuned sparse model into a dense
model. These kind of studies reduce the overall serving cost for inference. We
propose a novel optimizer for MoE to solve degeneracy instead of conventional
methods mentioned above. We encourage the orthogonal updates for experts to
enlarge the diversity.

#### Neural Network Optimizer

Optimizer is a method used to adjust the parameters of a neural network to
minimize the loss function. Different optimizers have a significant impact on
the performance of a neural network. Optimizers based on the stochastic
gradient descent method are currently mainstream optimizers. For example, the
Adam [21] that uses moving averages of the parameters to provide a running
estimate of the second raw moments of the gradients and RMSProp [58] that
divides the learning rate for weight by a running average of the magnitudes of
the recent gradients for that weight. Some optimizers like AdamW [22] use
regularization that adds a penalty to the loss function to prevent
overfitting. Also, other sharpness-aware optimizers [59, 60, 61] enhance the
model generalization by adding a perturbation to parameters. Recently, some
optimizers [62] use the orthogonal constraint to improve performance. To the
best of our knowledge, we are the first to introduce OWM to MoE.

#### Continual Learning

Continual learning enables AI models to adapt and evolve to new information,
without forgetting previously learned knowledge. Catastrophic forgetting [63]
refers to the phenomenon where learning a new task may result in modifications
to the existing network parameters that were learned for previous tasks. This
modification can lead to a degradation in the performance of the previous
tasks. One solution is to add a regularization term to the loss function [64,
65], which forces the model to memorize the previously learned tasks.
Additionally, previous training data can be included in the subsequent
training data to review previous knowledge (Replay) [66, 67, 68]. Another
approach is to partition the model’s parameters into isolated groups [69, 70],
with different groups responsible for different tasks. Besides splitting the
parameters w.r.t the tasks, the model memorizing the previous knowledge can be
fused with the new model that leaned the new information through model fusion
algorithms [71] Finally, the gradient projection approaches can be used to
make the gradients of different tasks be orthogonal [23], thus achieving
memory preservation. The OWM algorithm in this article is a gradient
projection approach.

##  VII Conclusion

In this paper, we propose an orthogonal optimizer (OMoE) for MoE-based
language models to address the issue of performance degeneracy and high expert
representation similarities in MoE. Specifically, we employ an alternating
training strategy consisting of different phases. Drawing inspiration from the
Orthogonal Weight Modification (OWM) technique in multi-task learning, we
enable experts to update their parameters in orthogonal directions with
respect to the subspaces defined by other experts. This fosters a diverse set
of experts and leads to improved model performance. To evaluate the
effectiveness of OMoE, we conduct extensive experiments on various benchmarks,
including the GLUE benchmark, the SuperGLUE benchmark, the QA task, and the
NER task. The experimental results demonstrate the significant performance
improvements achieved by OMoE compared to baseline methods. Notably, our work
represents the first application of OWM for optimizing experts in MoE models
with enhanced and appropriate diversity. In our future work, we plan to
explore the utilization of additional optimizers for MoE and investigate more
flexible methods for selecting hyperparameters in OMoE.

## References

  * [1] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” _Neural computation_ , 1991. [Online]. Available: https://doi.org/10.1162/neco.1991.3.1.79
  * [2] J. He, J. Qiu, A. Zeng, Z. Yang, J. Zhai, and J. Tang, “Fastmoe: A fast mixture-of-expert training system,” _arXiv preprint arXiv:2103.13262_ , 2021\. [Online]. Available: https://arxiv.org/abs/2103.13262
  * [3] D. Eigen, M. Ranzato, and I. Sutskever, “Learning factored representations in a deep mixture of experts,” in _ICLR_ , 2014. [Online]. Available: http://arxiv.org/abs/1312.4314
  * [4] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in _ICLR_ , 2017. [Online]. Available: https://openreview.net/forum?id=B1ckMDqlg
  * [5] Z. Zhang, D. Yang, Y. Xia, L. Ding, D. Tao, X. Zhou, and D. Cheng, “Mpipemoe: Memory efficient moe for pre-trained models with adaptive pipeline parallelism,” _2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_ , pp. 167–177, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:259979323
  * [6] S. He, L. Ding, D. Dong, B. Liu, F. Yu, and D. Tao, “Pad-net: An efficient framework for dynamic networks,” in _Annual Meeting of the Association for Computational Linguistics_ , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:258887710
  * [7] L. Ding and D. Tao, “The University of Sydney’s machine translation system for WMT19,” in _Proceedings of the Fourth Conference on Machine Translation_ , 2019. [Online]. Available: https://aclanthology.org/W19-5314
  * [8] L. Ding, L. Wang, X. Liu, D. F. Wong, D. Tao, and Z. Tu, “Understanding and improving lexical choice in non-autoregressive translation,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=ZTFeSBIX9C
  * [9] L. Ding, L. Wang, S. Shi, D. Tao, and Z. Tu, “Redistributing low-frequency words: Making the most of monolingual data in non-autoregressive translation,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , 2022, pp. 2417–2426.
  * [10] B. Wang, L. Ding, Q. Zhong, X. Li, and D. Tao, “A contrastive cross-channel data augmentation framework for aspect-based sentiment analysis,” in _Proceedings of the 29th International Conference on Computational Linguistics_ , 2022, pp. 6691–6704.
  * [11] Q. Zhong, L. Ding, J. Liu, B. Du, H. Jin, and D. Tao, “Knowledge graph augmented network towards multiview representation learning for aspect-based sentiment analysis,” _IEEE Transactions on Knowledge and Data Engineering_ , 2023.
  * [12] Y. Cao, L. Ding, Z. Tian, and M. Fang, “Towards efficiently diversifying dialogue generation via embedding augmentation,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7443–7447.
  * [13] Q. Wang, L. Ding, Y. Cao, Y. Zhan, Z. Lin, S. Wang, D. Tao, and L. Guo, “Divide, conquer, and combine: Mixture of semantic-independent experts for zero-shot dialogue state tracking,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2023. [Online]. Available: https://aclanthology.org/2023.acl-long.114
  * [14] Y. Chai, Q. Yin, and J. Zhang, “Improved training of mixture-of-experts language gans,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [15] T. Shen, M. Ott, M. Auli, and M. Ranzato, “Mixture models for diverse machine translation: Tricks of the trade,” in _ICML_ , 2019. [Online]. Available: http://proceedings.mlr.press/v97/shen19c.html
  * [16] S. Roller, S. Sukhbaatar, J. Weston _et al._ , “Hash layers for large sparse models,” _NIPS_ , 2021. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/92bf5e6240737e0326ea59846a83e076-Abstract.html
  * [17] S. Zuo, X. Liu, J. Jiao, Y. J. Kim, H. Hassan, R. Zhang, J. Gao, and T. Zhao, “Taming sparsely activated transformer with stochastic experts,” in _ICLR_ , 2022. [Online]. Available: https://openreview.net/forum?id=B72HXs80q4
  * [18] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” 2021. [Online]. Available: https://arxiv.org/abs/2101.03961
  * [19] Z. Gao, P. Liu, W. X. Zhao, Z. Lu, and J. Wen, “Parameter-efficient mixture-of-experts architecture for pre-trained language models,” in _COLING_ , 2022. [Online]. Available: https://aclanthology.org/2022.coling-1.288
  * [20] A. Yang, J. Lin, R. Men, C. Zhou, L. Jiang, X. Jia, A. Wang, J. Zhang, J. Wang, Y. Li _et al._ , “M6-t: Exploring sparse expert models and beyond,” _arXiv preprint arXiv:2105.15082_ , 2021. [Online]. Available: https://arxiv.org/abs/2105.15082
  * [21] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [22] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _ICLR_ , 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7
  * [23] G. Zeng, Y. Chen, B. Cui, and S. Yu, “Continual learning of context-dependent processing in neural networks,” _Nature Machine Intelligence_ , 2019. [Online]. Available: https://doi.org/10.1038/s42256-019-0080-x
  * [24] A. Ben-Israel and T. N. Greville, _Generalized inverses: theory and applications_. Springer Science & Business Media, 2003, vol. 15.
  * [25] H. Yanai, K. Takeuchi, and Y. Takane, _Projection matrices, generalized inverse matrices, and singular value decomposition_. Springer, 2011.
  * [26] S. Haykin, “Adaptive filter theory. pearson education india,” in _27th Annual International Conference of the Engineering in Medicine and Biology Society_. IEEE Press, 2008, pp. 1212–1215.
  * [27] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _ICLR_ , 2019. [Online]. Available: https://openreview.net/forum?id=rJ4km2R5t7
  * [28] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “Superglue: A stickier benchmark for general-purpose language understanding systems,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [29] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” _arXiv e-prints_ , p. arXiv:1606.05250, 2016.
  * [30] E. F. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” _arXiv preprint cs/0306050_ , 2003.
  * [31] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _NAACL_ , 2019. [Online]. Available: https://aclanthology.org/N19-1423
  * [32] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
  * [33] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” _arXiv preprint arXiv:1909.11942_ , 2019.
  * [34] S. Gupta, S. Mukherjee, K. Subudhi, E. Gonzalez, D. Jose, A. H. Awadallah, and J. Gao, “Sparsely activated mixture-of-experts are robust multi-task learners,” _arXiv preprint arXiv:2204.07689_ , 2022.
  * [35] Q. Ye, J. Zha, and X. Ren, “Eliciting transferability in multi-task learning with task-level mixture-of-experts,” _arXiv preprint arXiv:2205.12701_ , 2022\.
  * [36] Q. Zhong, L. Ding, J. Liu, X. Liu, M. Zhang, B. Du, and D. Tao, “Revisiting token dropping strategy in efficient bert pretraining,” in _Annual Meeting of the Association for Computational Linguistics_ , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258865591
  * [37] M. Roemmele, C. A. Bejan, and A. S. Gordon, “Choice of plausible alternatives: An evaluation of commonsense causal reasoning.” in _AAAI spring symposium: logical formalizations of commonsense reasoning_ , 2011, pp. 90–95.
  * [38] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” _arXiv preprint arXiv:1905.10044_ , 2019.
  * [39] D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, “Looking beyond the surface: A challenge set for reading comprehension over multiple sentences,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 2018, pp. 252–262.
  * [40] M. T. Pilehvar and J. Camacho-Collados, “Wic: 10,000 example pairs for evaluating context-sensitive representations,” _arXiv preprint arXiv:1808.09121_ , vol. 6, 2018.
  * [41] M.-C. De Marneffe, M. Simons, and J. Tonhauser, “The commitmentbank: Investigating projection in naturally occurring discourse,” in _proceedings of Sinn und Bedeutung_ , vol. 23, no. 2, 2019, pp. 107–124.
  * [42] R. Ebrahimpour, H. Nikoo, S. Masoudnia, M. R. Yousefi, and M. S. Ghaemi, “Mixture of mlp-experts for trend forecasting of time series: A case study of the tehran stock exchange,” _International Journal of Forecasting_ , vol. 27, no. 3, pp. 804–816, 2011.
  * [43] D. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei, “StableMoE: Stable routing strategy for mixture of experts,” in _ACL_ , 2022. [Online]. Available: https://aclanthology.org/2022.acl-long.489
  * [44] Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” _arXiv preprint arXiv:2202.09368_ , 2022. [Online]. Available: https://arxiv.org/abs/2202.09368
  * [45] S. He, R.-Z. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao, “Merging experts into one: Improving computational efficiency of mixture of experts,” _arXiv preprint_ , 2023.
  * [46] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 8583–8595, 2021.
  * [47] K. Clark, M. Luong, Q. V. Le, and C. D. Manning, “ELECTRA: pre-training text encoders as discriminators rather than generators,” in _ICLR_ , 2020. [Online]. Available: https://openreview.net/forum?id=r1xMH1BtvB
  * [48] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” in _ICLR_ , 2020. [Online]. Available: https://openreview.net/forum?id=H1eA7AEtvS
  * [49] Q. Zhong, L. Ding, Y. Zhan, Y. Qiao, Y. Wen, L. Shen, J. Liu, B. Yu, B. Du, Y. Chen, X. Gao, C. Miao, X. Tang, and D. Tao, “Toward efficient language model pretraining and downstream adaptation via self-evolution: A case study on superglue,” _ArXiv_ , vol. abs/2212.01853, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:254246784
  * [50] Q. Zhong, L. Ding, J. Liu, B. Du, and D. Tao, “E2s2: Encoding-enhanced sequence-to-sequence pretraining for language understanding and generation,” _arXiv preprint_ , 2022. [Online]. Available: https://arxiv.org/abs/2205.14912
  * [51] Q. Zhong, L. Ding, K. Peng, J. Liu, B. Du, L. Shen, Y. Zhan, and D. Tao, “Bag of tricks for effective language model pretraining and downstream adaptation: A case study on glue,” _ArXiv_ , vol. abs/2302.09268, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:257038296
  * [52] Q. Zhong, L. Ding, J. Liu, B. Du, and D. Tao, “Self-evolution learning for discriminative language model pretraining,” in _Annual Meeting of the Association for Computational Linguistics_ , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258865774
  * [53] N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat _et al._ , “Glam: Efficient scaling of language models with mixture-of-experts,” in _ICML_ , 2022. [Online]. Available: https://proceedings.mlr.press/v162/du22c.html
  * [54] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” in _ICLR_ , 2021. [Online]. Available: https://openreview.net/forum?id=qrwe7XHTmYb
  * [55] M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “Base layers: Simplifying training of large, sparse models,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 6265–6274.
  * [56] G. Lample, A. Sablayrolles, M. Ranzato, L. Denoyer, and H. Jégou, “Large memory layers with product keys,” _NIPS_ , 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/9d8df73a3cfbf3c5b47bc9b50f214aff-Abstract.html
  * [57] S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 18 332–18 346.
  * [58] T. Tieleman, G. Hinton _et al._ , “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” _COURSERA: Neural networks for machine learning_ , vol. 4, no. 2, pp. 26–31, 2012.
  * [59] Q. Zhong, L. Ding, L. Shen, P. Mi, J. Liu, B. Du, and D. Tao, “Improving sharpness-aware minimization with fisher mask for better generalization on language models,” _ArXiv_ , vol. abs/2210.05497, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252815713
  * [60] H. Sun, L. Shen, Q. Zhong, L. Ding, S.-Y. Chen, J. Sun, J. Li, G. Sun, and D. Tao, “Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks,” _ArXiv_ , vol. abs/2303.00565, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:257255005
  * [61] Y. Sun, L. Shen, S.-Y. Chen, L. Ding, and D. Tao, “Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape,” in _International Conference on Machine Learning_ , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258823234
  * [62] B. Zhang, W. Zheng, J. Zhou, and J. Lu, “Bort: Towards explainable neural networks with bounded orthogonal constraint,” _arXiv preprint arXiv:2212.09062_ , 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2212.09062
  * [63] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in _Psychology of learning and motivation_. Elsevier, 1989, vol. 24, pp. 109–165.
  * [64] A. Rannen, R. Aljundi, M. B. Blaschko, and T. Tuytelaars, “Encoder based lifelong learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1320–1328.
  * [65] H. Ahn, S. Cha, D. Lee, and T. Moon, “Uncertainty-based continual learning with adaptive regularization,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [66] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2001–2010.
  * [67] Z. Wang, S. V. Mehta, B. Póczos, and J. Carbonell, “Efficient meta lifelong-learning with limited memory,” _arXiv preprint arXiv:2010.02500_ , 2020.
  * [68] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato, “Continual learning with tiny episodic memories,” 2019.
  * [69] Z. Ke, B. Liu, N. Ma, H. Xu, and L. Shu, “Achieving forgetting prevention and knowledge transfer in continual learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 22 443–22 456, 2021.
  * [70] R. Aljundi, P. Chakravarty, and T. Tuytelaars, “Expert gate: Lifelong learning with a network of experts,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3366–3375.
  * [71] C. Zan, L. Ding, L. Shen, Y. Cao, W. Liu, and D. Tao, “On the complementarity between pre-training and random-initialization for resource-rich machine translation,” _ArXiv_ , vol. abs/2209.03316, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252110616