IEEEtran

# Enhancing Physical Layer Communication Security through Generative AI with
Mixture of Experts

Changyuan Zhao, Hongyang Du, Dusit Niyato, Fellow, IEEE, Jiawen Kang, Zehui
Xiong, Dong In Kim, Fellow, IEEE, Xuemin (Sherman) Shen, Fellow, IEEE, and
Khaled B. Letaief, Fellow, IEEE C. Zhao is with the College of Computing and
Data Science, Nanyang Technological University, Singapore, and CNRS@CREATE, 1
Create Way, 08-01 Create Tower, Singapore 138602 (e-mail:
zhao0441@e.ntu.edu.sg). H. Du, and D. Niyato are with the College of Computing
and Data Science, Nanyang Technological University, Singapore (e-mail:
<EMAIL_ADDRESS>dniyato@ntu.edu.sg).J. Kang is with the School of
Automation, Guangdong University of Technology, China. (e-mail:
kavinkang@gdut.edu.cn). Z. Xiong is with the Pillar of Information Systems
Technology and Design, Singapore University of Technology and Design,
Singapore (e-mail: zehui xiong@sutd.edu.sg).D. I. Kim is with the Department
of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419,
South Korea (email:dikim@skku.ac.kr).X. Shen is with the Department of
Electrical and Computer Engineering, University of Waterloo, Canada (e-mail:
sshen@uwaterloo.ca). Khaled B. Letaief is with the Department of Electrical
and Computer Engineering, Hong Kong University of Science and Technology, Hong
Kong (e-mail: eekhaled@ust.hk).

###### Abstract

AI technologies have become more widely adopted in wireless communications. As
an emerging type of AI technologies, the generative artificial intelligence
(GAI) gains lots of attention in communication security. Due to its powerful
learning ability, GAI models have demonstrated superiority over conventional
AI methods. However, GAI still has several limitations, including high
computational complexity and limited adaptability. Mixture of Experts (MoE),
which uses multiple expert models for prediction through a gate mechanism,
proposes possible solutions. Firstly, we review GAI model’s applications in
physical layer communication security, discuss limitations, and explore how
MoE can help GAI overcome these limitations. Furthermore, we propose an MoE-
enabled GAI framework for network optimization problems for communication
security. To demonstrate the framework’s effectiveness, we provide a case
study in a cooperative friendly jamming scenario. The experimental results
show that the MoE-enabled framework effectively assists the GAI algorithm,
solves its limitations, and enhances communication security.

###### Index Terms:

Physical Layer Communication Security, Generative AI, Mixture of Experts,
Network Optimization.

##  I Introduction

As communication devices become a part of people’s lives, carrying vast
amounts of crucial information, the security and privacy of these
communications face significant threats. Several threats exist in the physical
layer of communication systems. The physical layer is a fundamental layer
responsible for the raw transmission of bit streams across various
transmission media. Attacks on this layer aim to disrupt data transmission or
intercept data being transmitted. These vulnerabilities include attacks on
communication signals, equipment, and sensors, highlighting the wide range of
weaknesses that must be addressed for comprehensive communication network
security. Due to the foundational level of wireless communication systems,
ensuring the security of the physical layer is the first line of defense in
wireless communications.

With the rise of AI technology, there has been a significant shift towards
using advanced AI algorithms including machine learning and deep learning to
solve security concerns at the physical layer. For example, Convolutional
Neural Networks (CNNs) have been applied to facilitate multi-user
authentication [1], and Recurrent Neural Networks (RNNs) have been utilized to
design secure channel coding [2]. However, traditional AI methods fall short
in addressing communication security challenges due to their inability to
adapt to constantly changing wireless characteristics and sophisticated cyber
threats. These methods are usually trained on specific datasets, which limits
their effectiveness in unfamiliar conditions and unknown attacks.
Additionally, collecting enough labeled data for physical layer attacks is
difficult due to the complexity and variability of noise patterns, signal
interference, and channel conditions [3]. Therefore, advanced AI models are
needed to learn from and adapt to these environmental factors for robust
security.

Generative Artificial Intelligence (GAI) is an essential AI technology that
has proven to be remarkably effective in text, images, and audio generation.
Unlike traditional AI models, GAI operates as an unsupervised learning, which
can independently identify characteristics of data and generate new samples
that closely resemble the original dataset. For example, Stable Diffusion,
developed by Stability AI, is an advanced text-to-image generation model,
which can create highly detailed and accurate images based on textual
descriptions111https://stability.ai/news/stable-diffusion-3. This ability of
GAI to learn, replicate, and innovate is particularly useful in addressing
crucial challenges in communication security, such as handling incomplete
information and imbalanced data [4]. Furthermore, the integration of GAI
significantly strengthens the security framework, allowing for effective
learning of both emerging and complex threats [5]. Despite its potential, GAI
needs to be improved further due to high computational complexity and limited
adaptability. For instance, Stable Diffusion was trained using 256 Nvidia A100
GPUs on Amazon Web Services for a total of 150,000 GPU-
hours222https://en.wikipedia.org/wiki/Stable$\\_$Diffusion, which requires it
to take relatively more computing resources and computing speed in the
training and inference phases.

To address the aforementioned challenges effectively, the integration of the
Mixture of Experts (MoE) approach into GAI-based models such as
ChatGPT333https://medium.com/@seanbetts/peering-inside-gpt-4-understanding-
its-mixture-of-experts-moe-architecture-2a42eb8bdcb3 presents a promising
strategy. MoE is a machine learning ensemble technique that divides a complex
task into several subtasks, which are then solved by specialized models,
referred to as “experts.” The MoE approach uses a gating mechanism to decide
which experts to consult for a given input, effectively combining their
strengths to improve overall performance on diverse or large-scale problems.
In addition to language models, traditional AI models combined with MoE also
have several applications in wireless communication security. For instance,
ADMoE is an enhanced neural network-based anomaly detection algorithm by MoE
[6], which demonstrates superior performance in noisy label detection.

Therefore, deploying MoE-enabled GAI models in wireless communication systems
provides major advantages:

  * •

Flexibility and Specialization: By selecting different experts for different
attack types, the MoE framework ensures that the GAI model can provide more
targeted and effective defense against complex attackers. For example, TB-CBR
[7] selects different experts based on the feature of attacks to assign the
most appropriate techniques for identifying the type of attack.

  * •

Scalability and Efficiency: By strategically allocating computational
resources to the most pertinent experts for a specific task or learner, MoE-
based systems can simultaneously manage a larger number of experts, thereby
facilitating personalized learning at scale efficiently.

In conclusion, utilization of MoE in GAI models can enhance performance by
dynamically allocating computational resources to specialized sub-models,
thereby improving efficiency and adaptability in handling complex security
tasks. Based on these, we propose a framework to utilize MoE structure in this
article, enhancing the performance of GAI methods in communication security.
The contributions of this paper are summarized as follows.

  * •

We delve into the security considerations in the physical layer and provide a
detailed analysis of GAI structure in communication security. Then, we discuss
the benefits of implementing GAI models and explore the applications in
physical layer communication security.

  * •

We present a general MoE-enabled GAI framework to solve optimization problems
in communication security, which leverages the MoE’s structural
characteristics to enhance the protection performance of security protection
algorithms in various scenarios.

  * •

We consider friendly jamming strategies in multi-user scenarios as a case
study. We employ the MoE-enabled GAI reinforcement learning algorithm to solve
the optimization problem and analyze the algorithm’s efficacy improvement
under various indicators.

Figure 1: The overall of GAI for physical layer communication security. Part A
illustrates the structure of GAI models. Part B presents security issues faced
by physical layer communications. In Part C, we introduce three examples based
on GAI for communication security. These three models are based on VAE [8], DM
[9], and GAN [4], respectively. Part D demonstrates important physical layer
communication attack scenarios, including eavesdropping, jamming, and anomaly
detection.

##  II Generative AI and Mixture of Experts in Physical Layer Communication
Security

This section provides an overview of GAI models, and MoE concept, exploring
how GAI and MoE can be effectively employed in physical layer communication
security.

###  II-A Generative AI for Physical Layer Communication Security

GAI models, which are based on unsupervised learning, have an ability to
achieve self-learning from patterns and features in the data. They can emulate
the dataset as closely as possible without needing label information. This
capacity allows them to effectively address common data challenges in
communication security, such as missing datasets and imbalanced data (Figure
1). Currently, GAI models used in communication security include:

  * •

Autoencoder (AEs) and Variational Autoencoders (VAEs): AEs and VAEs compress
the input into a lower-dimensional code and reconstruct the output from this
representation as close as possible to the original input. Due to its encoding
and decoding structure, it can be used to ensure Confidentiality, including
wiretap code design, Joint Source Channel Coding (JSCC) [8]. Moreover,
according to the reconstruction error, they can keep signal Integrity, aiding
in anomaly detection.

  * •

Generative Adversarial Networks (GANs): GANs use adversarial learning to
simultaneously improve the generation ability of the generator and the
detection ability of the discriminator. Through adversarial learning, models
can generate data similar to the dataset and enhance overall performance,
ensuring data Availability and improving Resilience against various attacks,
including spoofing attacks [5].

  * •

Diffusion Models (DMs): DM is a novel generative model that employs a unique
technique for learning the underlying distribution of data. It first adds
noise to the data and then removes it, effectively generating new data that
follows the same distribution as the original. Due to its strong learning
ability to added noise, DM is able to effectively learn and purify attacked
data, preserving the Integrity of the data [9].

Even GAI models have different focuses and characteristics. It still faces
some problems in dealing with today’s more complex security issues.

  * •

High Complexity: GAI models are known for their sophisticated algorithms,
complex training processes, and long inference times, contributing to their
high complexity. When facing rapidly changing environments, they may struggle
to infer and protect against cyber threats with limited resources.

  * •

Low Adaptability: Due to its dependence on predefined datasets and algorithms,
the inherent inflexibility can challenge maintaining effectiveness in diverse
or novel contexts. As the model expands, retraining it from scratch becomes
increasingly challenging, rendering it less adept at responding to unexpected
events or data anomalies [10].

  * •

Limited Detection Performance: GAI models often focus on one or a few specific
security threats, and when faced with complex real-world tasks, they typically
demonstrate limited performance. This limitation is due to their high
complexity and low adaptability, which can hinder the AI’s ability to
accurately identify new, subtle, or sophisticated threats that differ from its
training sets.

Given the challenges faced by existing GAI models, new architectures have been
proposed to overcome these issues. MoE is an innovative approach aiming to
enhance GAI’s flexibility and effectiveness in dealing with a broader range of
security threats by combining the strengths of multiple specialized models,
i.e., experts.

Figure 2: The structure of MoE. In Part A, we illustrate the main components
and working mechanism of the MoE framework. Part B presents an example of
using the MoE framework to learn the different characteristic distributions of
multiple data. Part C demonstrates how to learn different features of data for
multiple tasks via MoE structure.

###  II-B Mixture of Experts

MoE improves model performance, adaptability, and scalability by assigning
tasks to specialized sub-models, or “experts,” for a tailored approach to
managing complex data [10]. By leveraging the unique strengths and expertise
of multiple sub-models, each specialized in particular data types or tasks,
MoE aims for higher accuracy and efficiency in computational problem-solving.
An MoE model includes the following important components:

  * •

Gating Mechanism: A gating mechanism evaluates the input and decides which
expert(s) should be activated for a given input, effectively routing the data
to the most relevant expert models.

  * •

Expert Models: Each expert model, specialized in different tasks, processes
the inputs that it receives. This specialization allows for tailored
processing and improved handling of complex patterns within the data.

  * •

Combining Outputs: The outputs of the expert models are then aggregated. The
method of aggregation can vary, including weighted sum, where the weights are
often determined by the gating mechanism’s output.

Figure 2 presents the main components of MoE and explain its working mechanism
through two examples.

Due to its dynamic allocation capabilities and efficiency in tackling complex
problems, the MoE framework has become widely used in Large Language Models
(LLMs). MoE enables these models to manage diverse and complex linguistic
tasks more effectively. For instance, Mixtral
8x7B444https://mistral.ai/news/mixtral-of-experts, a state-of-the-art
transformer model developed on the MoE framework, has 46.7B total parameters
but only uses 12.9B parameters per token. This efficient parameter usage
enables Mixtral to process inputs and generate outputs with the speed and cost
efficiency of a 12.9 billion parameter model. Notably, Mixtral achieves
competitive or superior performance compared to GPT-3.5 across a range of
benchmarks.

The above examples show that AI combined with MoE can bring a variety of
improvements. Especially, in the field of communication security, combining
MoE in GAI can bring many potential benefits:

  * •

Enhanced Detection: Learning the distribution of multiple hybrid attacks,
including various spoofing attacks and jamming attacks, is usually a
challenging task, which requires advanced structures for GAI models. By
utilizing the MoE structure that combines multiple expert models, a single
sub-model can focus on learning the distribution of certain attacks, thus
improving the fitting capability for the attacks with a relatively simple
model.

  * •

Resource Efficiency: In the inference phase, the GAI model needs to use all
parameters under its complex network structure, making it difficult to deploy
and perform real-time calculations on edge devices with limited resources. The
gate mechanism enables the MoE model to activate only the most suitable sub-
model for the specific task at hand, optimizing the use of available resources
and ensures a more effective response to cyber threats.

  * •

Adaptive Learning: In wireless communications, whenever the GAI model needs to
learn the distribution of a new attack, such as jamming attacks from a new
source, all parameters of the model need to be trained. The MoE uses a gate
mechanism to train the key parameters of GAI model, reducing the training cost
of the overall model, and improving the overall adaptability.

These features can address issues encountered by GAI models mentioned above,
providing a foundation for using the MoE structure to improve the performance
of GAI models in communication security.

##  III Applications of Generative AI and Mixture of Experts for Physical
Layer Security

GAI technology has numerous applications for wireless communications. In this
section, we present several GAI applications in physical layer communication
security and provide examples of combining GAI with MoE (Table I).

TABLE I: Summary of GAI and MoE for communication security.
Blue circles describe the methods; Green correct markers and Red cross markers represent pros and cons respectively. Security Consideration |  Reference |  Algorithm |  Pros & Cons
---|---|---|---
Confidentiality |  [8] |  VAE-based JSCC |  ⚫ A data-driven approach using VAE-based JSCC ✓ Conceal the confidential data different from the original signal ✗ Assume worse eavesdropping channel quality
Integrity |  [9] |  CDDM |  ⚫ A channel denoising diffusion models to remove noise in communications channels ✓ Remove the channel nosie under multiple fading channels ✗ Require relatively long sampling time
|  [11] |  MEx-CVAEC |  ⚫ An anomaly detection method combined with MoE framework ✓ Learn multiple latent space features instead of a single feature space ✗ Lack experiments on complex data sets
Availability |  [4] |  GAN |  ⚫ A GAN-based algorithm designed to uncover the data relationships and fill in the gaps in the data ✓ Recover spectrum data in multiple jamming conditions ✗ Lack real-world experiments

###  III-A GAI for Confidentiality

JSCC optimizes both source coding (compression) and channel coding (error
correction) in a unified approach to enhance efficiency and reliability of
data transmission while safeguarding data confidentiality against
eavesdroppers over a wiretap channel. GAI models, particularly AEs and VAEs,
are able to encode data into a latent space and then decode it back to the
original space through distribution decoding in the feature space, effectively
serving as a JSCC mechanism. For example, a VAE-based JSCC approach proposed
in [8] was designed to minimize information leakage and protect sensitive
information from being deciphered by unauthorized parties. Specifically, this
approach involves a transmitter (Alice) encoding source data and a receiver
(Bob) aiming to reconstruct the encoded data with high fidelity, while
ensuring that the sensitive information remains undetectable by any potential
eavesdroppers (Eves). This is achieved by maximizing the mutual information
between the user’s data and the noisy codewords observed by Bob, while
minimizing the distortion to improve the pixel-wise data reconstruction
quality. According to the experiments, the proposed method can achieve good
performance in both single channel and parallel channels scenarios. Especially
in the parallel Channels scenario, Eve’s classification accuracy is only 20%
on average [8].

###  III-B GAI for Integrity

During the transmission of data, it will be affected by factors such as noise
and attacks, which will affect the integrity and accuracy of the data received
by receivers, even with the presence of attacks to alter the data maliciously.
Due to its inherent ability to progressively remove noise, DMs have the
potential to aid the receivers in mitigating channel noise and purifying
received data. A representative example is Channel Denoising Diffusion Models
(CDDM) [9] designed to leverage the noise reduction properties of DMs. The
proposed model consists of three stages, where the first and last stages are
the JSCC encoder and decoder trained to minimize the reconstruction error,
respectively. The second stage utilizes a noise schedule that closely
simulates the distribution of channel noise, rendering the CDDM adaptable to a
variety of channel conditions. The efficacy of integrating CDDM into
communication systems is evident from the results, which show a consistent
performance improvement across all signal-to-noise ratio (SNR) levels.
Specifically, the inclusion of CDDM under an Additive White Gaussian Noise
(AWGN) channel and a Rayleigh fading channel at 20 dB SNR results in gains of
0.49 dB and 1.06 dB [9], respectively.

###  III-C GAI for Availability

The jamming attack aims to cause interference by introducing noise, thereby
disrupting legitimate communications at the physical layer. Jamming attacks
often result in incomplete data due to their attack characteristics, which
hinders the ability of anti-jamming strategies to accurately identify jamming
attacks. The authors in [4] have proposed an efficient algorithm that makes
use of a GAN to complete missing information in a spectrum waterfall. A
spectrum waterfall is a thermodynamic block diagram that defines the
environmental state. By automatically mining relationships within the data,
the algorithm can accurately complete any missing data. Unlike an original
GAN, where noise inputs were used, the authors utilized the spectrum waterfall
with the missing data as the generator input, consequently limiting the
generator’s artistry. According to the corresponding complement results, the
proposed algorithm outperforms the method without pre-classification since the
generator adding auxiliary information to the data is more targeted. The
accuracy of the proposed algorithm is over 95%, compared to the nearly 80%
accuracy of the method that does not use pre-classification [4].

###  III-D MoE-based GAI for Security

Typically, reconstruction-based GAI models for anomaly detection only consider
a single latent space. However, multiple latent spaces contain various
correlation discriminative features, each contributing differently to
efficient anomaly detection. MEx-CVAEC [11] is an MoE-based model, comprising
two convolutional VAEs and CNNs. Each component operates as an expert with an
identical procedural framework to efficiently process highly relevant
information. According to the latent space visualization comparison, a model
with multiple expert structures can better cluster normal and abnormal data
than a single expert. Compared with other unsupervised learning or GAI
baselines, the mean Area Under the Curve (AUC) of this model based on three
different datasets can achieve the best performance [11]. In conclusion, by
leveraging the potential of several latent spaces via the MoE structure, the
overall detection performance can be enhanced.

###  III-E Lessons Learned

From above applications, we can draw the following key observations:

  * •

GAI models are able to learn the distribution of the dataset and then
reconstruct the data from the distribution, which is suitable for secure
communications. For instance, VAE models can be utilized to improve the
encoding of physical layer data transmission [8].

  * •

Different from traditional AI models, GAI models can purify data by learning
the characteristics of attacks or noise. In the above example, if the data is
not purified, the accuracy of the received information will be seriously
affected and the integrity cannot be ensured.

  * •

By practicing adversarial learning, GAI has the ability to learn data features
more precisely, and also enhance the accuracy of the discriminator. This
approach resolves the issue of data incompleteness and imbalance caused by
intricate scenes such as jamming attacks.

  * •

The MoE structure allows AI detection models to utilize more complex data
including multi-label, multi-category, and multi-latent distributions,
improving detection accuracy compared to a single model.

However, most existing works only consider certain scenarios and lack real-
world testing, since they have difficulty to infer and adapt to additional
fast time-varying information in real-time [4]. Therefore, using the MoE
structure has become one of the feasible solutions to solve GAI’s issues in
communication security.

##  IV MoE-enabled Generative AI for Physical Layer Communication security

In this section, we discuss research challenges in communication security
optimizations. Then, we propose an MoE-enabled GAI framework focusing on
optimization problems in communication security.

###  IV-A Research Challenges

Communication security emphasizes the design of secure, safe, and reliable
information transmission methods [12]. From the perspective of system
optimization and design, ensuring communication security mainly includes the
following aspects:

  * •

Resource Allocation: Secure resource allocation aims to optimize limited
communication resources, such as bandwidth and energy, to maximize the
requirement of various performance metrics, including Secrecy Rate (SR),
Secrecy Outage Probability (SOP), power consumption, and Energy Efficiency
(EE) [12].

  * •

Signal Processing: Secure beamforming and precoding are advanced signal
processing techniques designed to enhance the confidentiality of wireless
communications. By intelligently shaping and directing transmitted signal
beams, these methods minimize potential eavesdropping and jamming, thus
optimizing the network’s SR and EE.

  * •

Antenna Selection and Cooperation: In multi-antenna wireless networks,
selection and cooperation of antennas can improve performance metrics such as
secrecy rate and outage probability [13]. These strategies can reduce power
consumption while boosting EE.

Reinforcement Learning (RL) algorithms based on DMs have recently achieved
success in solving such communication optimization problems, including UAV
path planning and resource allocation [13]. However, it usually focuses only
on a single optimization goal and cannot address joint considerations of
communication security [12]:

  * -

The transmission effectiveness is evaluated through the achievable SR.

  * -

Reliability is measured in terms of SOP.

  * -

Power cost refers to the minimum power consumption required to ensure secure
Quality of Service (QoS).

  * -

EE is concerned the quantity of secret bits transferred per unit of energy.

Figure 3: The workflow of the proposed framework. In Part A, the proposed
framework solves optimization problems through reinforcement learning
following 4 steps: 1). Task Setting; 2). MoE Learning; 3). Denoising
Diffusion; 4). Action Evaluation. Part B illustrates the cooperative friendly
jamming optimization problem considered in Section V. The figure introduces
the detail scenario and optimization problem settings including optimization
objectives, constraints, etc.

###  IV-B Framework Design

Based on the characteristics of MoE and GAI framework, in this part, we
propose the MoE-enabled GAI framework to solve communication security
optimization problems.

####  IV-B1 Architecture

Our framework follows the GAI-based RL model, GDM [13]. To consider multiple
performance metrics at the same time, we incorporate the MoE structure after
the state input layer. By combining the MoE structure, different experts can
focus on a few performance metrics with a shared-bottom model [14], rather
than optimizing all metrics at the same time. For instance, suppose two
performance metrics need to be optimized in a wireless communication system,
including secrecy rate and energy efficiency. In such a case, the GAI
framework with MoE structure can be trained by combining two experts. Each
expert can be automatically assigned to capture either shared performance
metrics information or a specific one, which is more conducive than one
network to finding the theoretically optimal solution for two metrics.

Specifically, regarding secure communication, the definitions of the state
space, action space, and reward are as follows:

  * •

State Space: The state space consists of the current information and
previously selected actions. The state is the channel information, which
includes characteristics such as channel quality, current interference levels,
signal fading, and other relevant environmental conditions affecting signal
transmission.

  * •

Action Space: The action space encompasses a range of adjustable parameters
that can significantly influence its performance and security. Examples of
these parameters include frequency, bandwidth, transmission power, and devices
distance.

  * •

Reward: The reward based on each action taken at the current state is
determined by security performance metrics [12]. The performance metrics are
different security metrics including SR, SOP, and EE, etc.

####  IV-B2 User Workflow

In the proposed framework, to solve an optimization problem about
communication security, one may follow these steps (Figure 3 Part A):

  * •

Task Setting: To optimize a secure transmission, users must first confirm
reasonable performance metrics that consider security performance. This is
usually achieved through a variety of metrics, which are then divided into
individual tasks. For example, when designing a system to prevent
eavesdropping, users can consider the hybrid metrics.

  * •

MoE Learning: By using the gate mechanism in MoE, the input states can be
divided into individual independent expert models. Then different experts can
automatically capture either shared and task-oriented information, allowing
them to obtain higher action reward. For instance, users can assign three or
more experts to optimize different performance metrics, through the learning
of gate network.

  * •

Denosing Diffusion: Based on the GDM structure [13], users predict actions in
the current state by using conditional denoising to filter out noise. The
conditions used in this process can correspond to various subtasks, such as
current state, previously selected action, or a combination of both. For
instance, when focusing on improving security performance, the users can
choose the current channel state and the previous transmission power as
conditions in the subtask of transmission power.

  * •

Action Evaluation: Instead of denoising towards the maximum value of the
probability distribution, the actor network is learned by denoising towards
the direction that maximizes reward, and then combined with the gate network
to obtain the finial decision.

Combining the powerful generation capability of GAI with learning for an
optimal action denoising can help to explore the optimal solution more
effectively [13]. By incorporating the MoE structure, the framework can focus
on multiple performance metrics simultaneously, achieving better results in
multi-objective optimization problems.

##  V Case study: Cooperative Friendly Jamming for Physical layer
Communication security

In this section, we present a case study that demonstrates the optimization
for cooperative friendly jamming and utilize the proposed framework to
demonstrate how MoE can improve the performance of GAI models.

###  V-A System Model

We consider a cooperative friendly jamming scenario, where Access Points (APs)
produce friendly jamming signals to further degrade eavesdropping [15]. This
scenario involves multiple APs, multiple users, and potential eavesdroppers.
When a user downloads data from a certain AP, the other APs will act as
jammers to prevent any eavesdroppers from intercepting the user’s traffic. In
this case study, we consider 3 APs, 3 users, and 2 eavesdroppers, where each
user communicates with one AP (Figure 3 Part B). For the received power
$p_{r}$ of the system, we consider large-scale fading based on distance and
small-scale fading based on the Rayleigh fading model. For a given user or
eavesdropper, the Shannon capacity is determined by the Signal to Interference
plus Noise Ratio (SINR), where SINR is calculated from the received APs power
and noise.

###  V-B Optimization Problem Formulation

We aim to jointly maximize SR and secure EE [12] to find the optimal power
allocation to the APs. The effectiveness of physical layer communication
security strategies can be measured using SR. Our assessment of SR involves
comparing the data rate of the legitimate channel to that of the wiretap
channel with the Gaussian codebook. For a given user, the SR $C_{s}$ is
calculated via the Shannon capacity minus the largest Shannon capacity of
eavesdroppers. Our goal at EE is to ensure that the transmission strategy
operates in a confidential and eco-friendly manner. To achieve this, we
utilize secure EE, which is the amount of secret bits transmitted with the
consumption of one unit of energy. For a given AP, its secure EE $S_{e}$ is
determined by the SR of the corresponding user to AP’s transmission power
$p_{t}$.

###  V-C Numerical Results

(a) Sum of SR and secure EE comparisons for three methods.

(b) Comparison of metrics relationship for three methods.

Figure 4: Performance comparison of three methods in cooperative friendly
jamming scenario.

####  V-C1 Experimental Setup

Our experiment is based on Python along with the PyTorch package, conducted on
a V100 Linux server. The main program is based on the GDM framework [13]. The
proposed MoE method includes 3 experts and selects the best expert each time.

####  V-C2 Performance Analysis

Figure 4 illustrates the comparison of various performance metrics between the
proposed MoE-enabled GAI method, GDM without MoE, and Deep Deterministic
Policy Gradient (DDPG) algorithm [13] in a given scenario. Among them, the
blue curve represents the MoE method, the orange curve represents GDM, the
green curve represents DDPG, and the reward represents the specific numerical
value of metrics. Based on the transmission power derived from two approaches,
Figure 4(a) illustrates the learning curve for the sum of SR and secure EE.
Clearly, the proposed method demonstrates the ability to converge to a reward
of approximately 40.56 and has the potential to explore and achieve a reward
as high as 47.19. In comparison, the reward obtained through GDM tends to
converge around 27.99, with the maximum reward during training reaching only
34.63. The DDPG method performs even worse, achieving only around 23.17 and
the maximum reward capped at 34.52. Figure 4(b) shows the numerical
relationship between the two indicators among the three methods. For a more
linear fitting trend line, as Secure EE decreases, both SR metrics show a
downward trend. However, the proposed method (organe curve) decreases slowly,
and higher Secure EE values can be explored compared to the other two methods
with the same SR value.

###  V-D Discussion

The experimental results above demonstrate that in a cooperative friendly
jamming scenario, the MoE-enabled GAI optimization algorithm can achieve
better exploration performance and convergence results in optimizing mixed
metrics, including SR and secure EE. Through the comparison results (Figure
4), the MoE can explore more suitable resource allocation strategies to
improve the performance of one task (secure EE) without compromising the
performance of another task (SR) significantly. Since the MoE only activates
one expert through the gate network for training and inference each time, the
overall computational complexity is similar to that of the original GDM and
does not increase significantly.

##  VI Future Directions

###  VI-A Zero-trust Physical Layer Communication

Zero-trust communication refers to a security strategy where every
communication attempt must be verified. The MoE-based GAI algorithm allows for
the simultaneous maximization of various security performance metrics during
the network design process, such as MoE-based DMs for graph generation.
Therefore, all network entities can be verified and trusted, minimizing the
risk of tampering or interception.

###  VI-B Real-time Anomaly Detection

The high communication speed has put forward requirements for real-time
anomaly detection. By leveraging MoE structure, only important network levels
will be activated according to the gate mechanism, reducing the inference
speed of GAI models. This approach could also significantly improve the
detection of sophisticated cyber-attacks that conventional systems might
overlook.

###  VI-C Enhanced Privacy Preservation

Data privacy protection is also an important part of physical layer security.
During transmission, MoE-enabled GAI model can process signals in a way that
maximizes data privacy. For instance, experts within the MoE model could
specialize in different privacy-preserving techniques such as data
anonymization, encryption, and differential privacy.

##  VII Conclusions

The paper demonstrated how GAI models can improve communication security and
explores their capabilities when combined with the MoE framework.
Specifically, we started by introducing the GAI technology and its application
in physical layer communication security. We emphasized superior capabilities
of GAI and compared them to conventional AI models in communication security
by providing examples. Subsequently, we highlighted the limitations of the GAI
model. By introducing the structure and characteristics of MoE, we explored
the possibility of the MoE framework to overcome limitations of GAI models.
Finally, we proposed an MoE-enabled GAI framework focusing on network
optimization problems in communication security. Through a case study in a
cooperative friendly jamming scenario, it illustrated how the MoE structure
improves GAI algorithms to ensure safety in wireless communication systems.

## References

  * [1] R. Liao, H. Wen, F. Pan, H. Song, A. Xu, and Y. Jiang, “A novel physical layer authentication method with convolutional neural network,” in _Proceedings of IEEE International Conference on Artificial Intelligence and Computer Applications_. IEEE, 2019, pp. 231–235.
  * [2] X. Xiao, B. Vasić, R. Tandon, and S. Lin, “Designing finite alphabet iterative decoders of ldpc codes via recurrent quantized neural networks,” _IEEE Transactions on Communications_ , vol. 68, no. 7, pp. 3963–3974, 2020.
  * [3] J. Wang, H. Du, D. Niyato, J. Kang, S. Cui, X. Shen, and P. Zhang, “Generative ai for integrated sensing and communication: Insights from the physical layer perspective,” _arXiv preprint arXiv:2310.01036_ , 2023.
  * [4] Y. Cai, F. Song, Y. Xu, X. Liu, X. Zhang, and H. Han, “Spectrum waterfall completion in jamming enviroment: A general adversarial networks method,” in _Proceedings of IEEE 9th Joint International Information Technology and Artificial Intelligence Conference_ , vol. 9. IEEE, 2020, pp. 1661–1665.
  * [5] Y. Shi, K. Davaslioglu, and Y. E. Sagduyu, “Generative adversarial network in the air: Deep adversarial learning for wireless signal spoofing,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 7, no. 1, pp. 294–303, 2020.
  * [6] Y. Zhao, G. Zheng, S. Mukherjee, R. McCann, and A. Awadallah, “Admoe: Anomaly detection with mixture-of-experts from noisy labels,” in _Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence_ , vol. 37, no. 4, 2023, pp. 4937–4945.
  * [7] C. I. Pinzón, J. F. De Paz, M. Navarro, J. Bajo, V. Julian, and J. M. Corchado, “Real-time cbr-agent with a mixture of experts in the reuse stage to classify and detect dos attacks,” _Applied Soft Computing_ , vol. 11, no. 7, pp. 4384–4398, 2011.
  * [8] E. Erdemir, P. L. Dragotti, and D. Gündüz, “Privacy-aware communication over a wiretap channel with generative networks,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2022, pp. 2989–2993.
  * [9] T. Wu, Z. Chen, D. He, L. Qian, Y. Xu, M. Tao, and W. Zhang, “CDDM: Channel denoising diffusion models for wireless communications,” in _Proceedings of IEEE Global Communications Conference_. IEEE, 2023, pp. 7429–7434.
  * [10] J. Wang, H. Du, D. Niyato, J. Kang, Z. Xiong, D. I. Kim, and K. B. Letaief, “Toward scalable generative ai via mixture of experts in mobile edge networks,” _arXiv preprint arXiv:2402.06942_ , 2024.
  * [11] Q. Yu, M. S. Kavitha, and T. Kurita, “Mixture of experts with convolutional and variational autoencoders for anomaly detection,” _Applied Intelligence_ , vol. 51, pp. 3241–3254, 2021.
  * [12] D. Wang, B. Bai, W. Zhao, and Z. Han, “A survey of optimization approaches for wireless physical layer security,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 2, pp. 1878–1911, 2018.
  * [13] H. Du, R. Zhang, Y. Liu, J. Wang, Y. Lin, Z. Li, D. Niyato, J. Kang, Z. Xiong, S. Cui _et al._ , “Beyond deep reinforcement learning: A tutorial on generative diffusion models in network optimization,” _arXiv preprint arXiv:2308.05384_ , 2023.
  * [14] J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” in _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 1930–1939.
  * [15] S. A. Hoseini, F. Bouhafs, N. Aboutorab, P. Sadeghi, and F. d. Hartog, “Cooperative jamming for physical layer security enhancement using deep reinforcement learning,” _arXiv preprint arXiv:2403.10342_ , 2024.


# An Effective Method using Phrase Mechanism in
Neural Machine Translation

Phuong Nguyen  Minh Le Nguyen
Japan Advanced Institute of Science and Technology, Japan
{phuongnm<EMAIL_ADDRESS>

###### Tóm tắt nội dung

Machine Translation is one of the essential tasks in Natural Language
Processing (NLP), which has massive applications in real life as well as
contributing to other tasks in the NLP research community. Recently,
Transformer -based methods have attracted numerous researchers in this domain
and achieved state-of-the-art results in most of the pair languages. In this
paper, we report an effective method using a phrase mechanism,
PhraseTransformer, to improve the strong baseline model Transformer, in
constructing a Neural Machine Translation (NMT) system for parallel corpora
Vietnamese-Chinese. Our experiments on the MT dataset of the VLSP 2022
competition achieved the BLEU score of 35.3 on Vietnamese to Chinese and 33.2
BLEU scores on Chinese to Vietnamese data. Our code is available at
https://github.com/phuongnm94/PhraseTransformer.

##  1 Introduction

In the NLP area, Machine Translation is the primary task that has a long time
history of development, especially in the approaches using Neural Networks
(Sutskever et al., 2014; Bahdanau et al., 2015). Besides, there are numerous
proposed architectures in this domain that have got a significant effect on
the NLP community in many domains (Sutskever et al., 2014; Bahdanau et al.,
2015; Vaswani et al., 2017; Devlin et al., 2019). Therefore, in this work, we
focus on Machine Translation task in The Ninth International Workshop on
Vietnamese Language and Speech Processing (VLSP
2022)111https://vlsp.org.vn/vlsp2022.

VLSP 2022 is a competition of one of the biggest NLP communities in Vietnam,
Association for Vietnamese Language and Speech Processing, hosted by the VNU
University of Science, Hanoi. This is a good chance for researchers in many
domains to introduce their awesome research results and catch the state-of-
the-art machine learning models in different domains.

This year, the machine translation task is designed to translate from
Vietnamese to Chinese and on the contrary direction. In this task, given the
input is a natural sentence in Chinese (or Vietnamese), the machine
translation system is required to generate a new sentence in Vietnamese (or
Chinese) that has the same meaning as the input sentence (Table 1).

Language | Content
---|---
Vietnamese |  Tôi vừa có kế hoạch thay họ biểu diễn ở bữa tiệc.
Chinese |  正打算让这群废物在派对上表演呢
(meaning) |  I just had the plan to perform on their behalf at the party.
Vietnamese |  Không! Cậu đã lấy đi tất cả những gì quý giá đối với tôi rời, cậu Toretto à!
Chinese |  我有价值的东西早被你榨取光了 托雷托先生
(meaning) |  No! You took everything that was precious to me, Mr. Toretto!
Table 1: Example of machine translation task in the parallel Vietnamese-
Chinese corpus of VLSP 2022. The (meaning) row is to support the readers of
this paper which are not provided in dataset.

Recently, the self-attention mechanism has occupied a lot of attention,
especially the Transformer architecture (Vaswani et al., 2017; Devlin et al.,
2019). This architecture leads many state-of-the-art results in various NLP
domains (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020). In
this work, we applied a new approach based on the Transformer architecture,
PhraseTransformer (Nguyen et al., 2023), which leverages a phrase attention
mechanism to solve the machine translation task. The main idea of this
approach is to enhance the word representation by its local contexts (or
phrases), and apply the self-attention mechanism to model the dependencies
between phrases in a sentences. The experimental results on the machine
translation dataset of the VLSP 2022 workshop show that our PhraseTransformer
clearly beat the original Transformer model in both two directions from
Vietnamese to Chinese and from Chinese to Vietnamese. Furthermore, the
PhraseTransformer does not require any external syntax tree information as the
previous works (Yang et al., 2018; Wang et al., 2019; Nguyen et al., 2020;
Bugliarello and Okazaki, 2020) and is more lightweight compared with other
phrase-level attention models (Xu et al., 2020). To this end, our
PhraseTransformer improved the original Transformer by 1.1 BLEU scores on the
setting from Vietnamese to Chinese and 1.3 BLEU scores on the setting from
Chinese to Vietnamese.

The remainder of this paper, we detail our work in four sections. Section 2,
we show the related works in this domain. And describe the details of our
PhraseTransformer architecture in section 3. Then, the experiments and result
analysis are shown in section 4. Finally, in section 5, we conclude this
paper.

##  2 Related works

In the success of Transformer Vaswani et al. (2017) in the machine translation
task, there are a numerous works proposed new directions to improve this
architecture. Among them, the approach considering of utilize the phrase
linguistic structure representation is promising. The works presented in Yang
et al. (2020); Shang et al. (2021); Nguyen et al. (2022) demonstrate that the
template prediction can guide an NMT system in the decoding process to improve
performance. The works (Wu et al., 2018; Nguyen et al., 2020; Wang et al.,
2019) indicate that the phrase information extracted from the syntax tree can
improve the meaning representation of sentence. However, the performance of
these systems is affected by the quality of the syntax tree extraction step
which is usually lower performance in low-resource languages. Close to our
PhraseTransformer, Xu et al. (2020) also presents a different method to model
the phrase in a source sentence, however, this architecture uses a huge number
of parameters to learn the attention scores between source phrases with target
words. Compared with this work, by using LSTM architecture (Hochreiter and
Schmidhuber, 1997) in the Multi-head layers to model various local context
information, our PhraseTransformer increases the model size with a small
margin but works effectively.

##  3 Model architecture

In this section, we describe our PhraseTransformer model (Nguyen et al.,
2023), which is used to submit the machine translation task in VLSP 2022
workshop (Figure 1). We define the input vector ($\mathbf{x}$) is synthesized
from the word embedding and positional encoding
$\mathbf{x}=[\mathbf{x}_{1},...,\mathbf{x}_{|S|}]$ ($|S|$ is sentence length)
similar to Vaswani et al. (2017). This vector is processed by a Dense layer to
get multi-views of input data corresponding to different heads.

$\displaystyle\mathbf{q}_{i},\mathbf{k}_{i},\mathbf{v}_{i}$
$\displaystyle=\mathbf{x}\mathbf{W}_{i}^{q},\mathbf{x}\mathbf{W}_{i}^{k},\mathbf{x}\mathbf{W}_{i}^{v}$
(1)

where $i$ is the head index, $1\leq i\leq{H}$ where $H$ is the number of heads
in Multi-Head layer. We define a $\operatorname{Phrase}$ function to extract
local context:

(2)

where $m_{t}$ is the hyper-parameter gram size, $\mathbf{s}_{i}$ is the
sequence of words vector, $n\mathrm{\\_gramLSTM}_{k}$ is the function which
capture local context using LSTM model:

$\displaystyle m_{t}\mathrm{\\_gram}_{k}(\mathbf{s}_{i})$
$\displaystyle=[\mathbf{H}_{k-n_{i}+1},\mathbf{H}_{k-n_{i}+2},...,\mathbf{H}_{k}]$
$\displaystyle n\mathrm{\\_gramLSTM}_{k}(\mathbf{s}_{i})$
$\displaystyle=\mathrm{LSTM}^{f}_{i}(m_{t}\mathrm{\\_gram}_{k}(\mathbf{s}_{i}))+$
$\displaystyle\quad\quad\mathrm{LSTM}^{b}_{i}(m_{t}\mathrm{\\_gram}_{k}(\mathbf{s}_{i}))$

Then, the phrase information is integrated to the word representations:

$\displaystyle\operatorname{zip}(A,B)$
$\displaystyle=[A^{\intercal};B^{\intercal}]^{\intercal}$ (3)
$\displaystyle\mathbf{ph}^{k}_{i,m_{t}},\mathbf{ph}^{q}_{i,m_{t}}$
$\displaystyle=\operatorname{unzip}(\operatorname{Phrase}(\operatorname{zip}(\mathbf{q}_{i},\mathbf{k}_{i}),m_{t}))$
(4) $\displaystyle\mathbf{ph}^{q}_{i}$
$\displaystyle=\operatorname{zip}(\\{\mathbf{ph}^{q}_{i,m_{t}}\mid
m_{t}\in\mathbf{m}\\})$ (5) $\displaystyle\mathbf{ph}^{k}_{i}$
$\displaystyle=\operatorname{zip}(\\{\mathbf{ph}^{k}_{i,m_{t}}\mid
m_{t}\in\mathbf{m}\\})$ (6)

After that, these vectors are fed to the Self-Attention layer to learn the
dependencies between words and phrases.

$\displaystyle\mathbf{head}_{i}$
$\displaystyle=\mathrm{softmax}(\frac{\mathbf{ph}^{q}_{i}\,\,{\mathbf{ph}^{k}_{i}}^{\intercal}}{\sqrt{d_{h}}})\,\mathbf{v}_{i}$
(7)

where $d_{h}$ is dimension of one head in Multi-Head layer. Finally, all head
vector representation is combined to get the final sentence vector
($\mathbf{h}_{out}$) following Vaswani et al. (2017):

$\displaystyle\mathbf{h}_{mh}$
$\displaystyle=[\mathbf{head}_{1};...;\mathbf{head}_{H}]\mathbf{W}^{o}$ (8)
$\displaystyle\mathbf{h}_{no}$
$\displaystyle=\mathrm{LayerNorm}(\mathbf{h}_{mh}+\mathbf{x})$ (9)
$\displaystyle\mathbf{h}_{out}$
$\displaystyle=\mathrm{LayerNorm}(\mathrm{FeedForward}(\mathbf{h}_{no})+\mathbf{h}_{no})$
(10)

This sentence vector is processed via a stack of $N$ Identical layers with the
same architecture and forwarded to the Transformer Decoder layer (Vaswani et
al., 2017) for decoding process.

Figure 1: Overview of PhraseTransformer (CrossH) using $n$_gram LSTM in
MultiHead layer. In this case, the phrase representations are built with
$\mathrm{gram\\_size}=\\{2,3\\}$, 2-gram, 3-gram models apply to all 8 heads.

### Training method

The training objective is to maximize the Log-Likelihood function of the
probabilities to generate the target sentence ($y$) given a source sentence
($x$) from an machine translation parallel dataset ($\mathcal{D}$):

$\displaystyle\mathrm{maximize}:\sum_{<x,y>\in\mathcal{D}}\log\,p_{\theta}\,(y\mid
x)$ (11)

##  4 Experiments and Results

In this section, we describe in detail about our experiments and the
corresponding results.

### Dataset.

We use the parallel corpora Chinese-Vietnamese provided by the VLSP 2022
workshop for the training and development process. For more understanding of
this dataset, we show the data analysis information on Table 2.

#examples | #vocab | Avg. length
---|---|---
Train | Dev | Test | Vi | Zh | Vi | Zh
300,347 | 999 | 999 | 4K | 16K | 18.99 | 22.85

Table 2:  Statistics information of parallel corpora Chinese-Vietnamese in
VLSP 2022 workshop. Vocabulary size and average length of source (Src) and
target (Tgt) side are computed on training set.

### Prepossessing.

We follow the previous work (Provilkov et al., 2020) for reprocessing step. To
deal with the out-of-vocab problem, we use BPE encode method (Sennrich et al.,
2016) to split words into sub-tokens with the number of operators is 4000 and
16000 operators in Vietnamese (Vi) and Chinese (Zh) languages, respectively.
In the Chinese language, because there is no space between characters, the BPE
segment module directly processes the whole raw sentence as a word segment.

### Evaluation metric.

We use the BLEU score to evaluate the quality of the translated sentence. For
a fair comparison with other works, we used SacreBleu (Post, 2018)
tool222https://github.com/mjpost/sacrebleu. To evaluate the system translated
to the Chinese language, we used a word tokenizer default provided by this
tool with setting \--tok zh. To this end, the output of our machine
translation system is compared with the raw sentence of the target language
provided by VLSP 2022 parallel corpora to get the BLEU score. The signature of
our evaluation setting on both translation directions are:

  * •

Vietnamese-Chinese: BLEU+case.mixed +lang.vi-zh +numrefs.1 +smooth.exp +tok.zh
+version.1.5.1

  * •

Chinese-Vietnamese: BLEU+case.mixed +lang.zh-vi +numrefs.1 +smooth.exp
+tok.13a +version.1.5.1

### Experimental setting.

For comparison between our PhraseTransformer and the original Transformer
Vaswani et al. (2017), we implemented to train and evaluate these models on
the parallel corpora Chinese-Vietnamese VLSP 2022 without any external data or
pre-trained model. We conducted all experiments with the same setting on
server NVIDIA A40. The hidden size is $512$; the learning rate is initialized
by $7e^{-4}$; the learning warm-up step is $6000$; both the transformer
encoder and decoder contain 6 layers, 4 heads. In the training process, the
batch size is 4096 tokens, and the maximum number of epochs is $100$.
Following Provilkov et al. (2020), we average 5 latest checkpoints333The
script is used to get average 5 latest checkpoints:
https://github.com/facebookresearch/fairseq/blob/main/scripts/average_checkpoints.py
to get a final model for the translation testing process.

### Results.

We report the the experimental results of machine translation systems from
Chinese to Vietnamese and from Vietnamese to Chinese on Table 3, and Table 4,
respectively444The underline setting in these tables is the model we submitted
at the time of the VLSP 2022 competition.. Based on these results, we found
that our PhraseTransformer can beat the original Transformer with almost
settings of gram sizes. These results proved the effectiveness of our phrase
modeling mechanism, which supports the translation system better in sentence
meaning representation. Besides, our model is work well without any external
syntax tree information that makes the PhraseTransformer architecture can
widely adaptable to many other languages as well as other NLP tasks.

Model | gram sizes ($\mathbf{m}$) | Dev | Test
---|---|---|---
Transformer | - | 30.4 | 31.9
PhraseTransformer | {3} | 31.0 | 33.0
PhraseTransformer | {4} | 30.9 | 32.7
PhraseTransformer | {2,3} | 30.8 | 33.2

Table 3: Experimental result using SacreBLEU of translation system from
Chinese to Vietnamese. The test set is the public test provided at the time of
the competition.

Model | gram sizes ($\mathbf{m}$) | Dev | Test
---|---|---|---
Transformer | - | 29.5 | 34.2
PhraseTransformer | {3} | 30.1 | 35.0
PhraseTransformer | {4} | 30.1 | 35.3
PhraseTransformer | {2,3} | 30.0 | 34.1

Table 4: The experimental result using SacreBLEU of translation system from
Vietnamese to Chinese.

### Improvement example.

For more understanding about the improvement of PhraseTransformer compared
with the original Transformer, we observe the difference between the outputs
of two models and show it in Table 5. In the first row, the original
Transformer has missed a part of the information in the translation process,
“các dịch xã hội cơ bản như thông tin” (basic social services such as
information), while our PhraseTransformer can recognize it. Besides, in the
second row, the output of the original Transformer is lack the smoothness of
the natural language “đặt mức độ suy giảm khủng hoảng trung bình” that makes
the translated sentence is misunderstood “4 loại dịch bệnh”. We argue that
because our PhraseTransformer captured the dependencies between phrases, that
make model is better in capture the sub-conditions of the long sentence. In
addition, our PhraseTransformer captured the local context in the phrases
which supports improving the meaning representation in the translation
process.

Input |  这旨在面向全面保障贫困人口的社会民生权利，提高贫困人口获得信息、就业、医疗、教育、住房、生活用水、卫生、环境等基本社会服务的机会。
---|---
Gold sentence |  Điều này hướng tới đảm bảo toàn diện quyền an sinh xã hội của người nghèo, cải thiện mức độ tiếp cận của người nghèo đối với các dịch vụ xã hội cơ bản như thông tin, việc làm, y tế, giáo dục, nhà ở, nước sinh hoạt và vệ sinh, môi trường.
Transformer |  Đây là cơ hội để đảm bảo toàn diện các quyền an sinh xã hội của người nghèo, nâng cao việc làm, y tế, nhà ở, nước sinh hoạt, vệ sinh, môi trường.
PhraseTransformer |  Đây là dịp để đảm bảo toàn diện các quyền an sinh xã hội cho người nghèo, nâng cao các dịch vụ xã hội cơ bản như thông tin, việc làm, y tế, giáo dục, nhà ở, nước sinh hoạt, vệ sinh, môi trường.
Input |  政府将设低危机、中等危机、高危机、最高危机4个疫情级别。
Gold sentence |  Chính phủ đã phân loại 4 cấp độ dịch: nguy cơ thấp (bình thường mới), nguy cơ trung bình, nguy cơ cao, nguy cơ rất cao.
Transformer |  Chính phủ sẽ đặt mức độ suy giảm khủng hoảng trung bình, khủng hoảng cao, nguy cơ cao nhất, 4 loại dịch bệnh.
PhraseTransformer |  Chính phủ sẽ có 4 cấp độ dịch bệnh thấp, nguy cơ trung bình, nguy cơ cao và khủng hoảng cao nhất.

Table 5: Improvement examples of the PhraseTransformer compared with
Transformer in the Chinese to Vietnamese test set. The red part is the main
difference between these models.

##  5 Conclusion

In this paper, we first applied our PhraseTransformer model to the machine
translation task for the pair of languages Vietnamese - Chinese. In this
architecture, the original Transformer Encoder is enhanced by incorporating
the phrase dependencies information into the Self-Attention mechanism. Our
experimental results showed that our PhaseTransformer beat the original
Transformer by a large margin in both translation directions. In future work,
we would like to apply our model architecture to other NLP tasks and explore
other effective phrase modeling methods to achieve better results.

## Tài liệu

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  3rd International Conference on Learning Representations, ICLR 2015 ; Conference date: 07-05-2015 Through 09-05-2015.
  * Bugliarello and Okazaki (2020) Emanuele Bugliarello and Naoaki Okazaki. 2020.  Enhancing machine translation with dependency-aware self-attention.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1618–1627, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Nguyen et al. (2022) Phuong Nguyen, Tung Le, Thanh-Le Ha, Thai Dang, Khanh Tran, Kim Anh Nguyen, and Nguyen Le Minh. 2022.  Improving neural machine translation by efficiently incorporating syntactic templates.  In _Advances and Trends in Artificial Intelligence. Theory and Practices in Artificial Intelligence_ , pages 303–314, Cham. Springer International Publishing.
  * Nguyen et al. (2023) Phuong Minh Nguyen, Tung Le, Huy Tien Nguyen, Vu Tran, and Minh Le Nguyen. 2023\.  Phrasetransformer: an incorporation of local context information into sequence-to-sequence semantic parsing.  _Applied Intelligence_ , 53(12):15889–15908.
  * Nguyen et al. (2020) Xuan-Phi Nguyen, Shafiq Joty, Steven Hoi, and Richard Socher. 2020.  Tree-structured attention with hierarchical accumulation.  In _International Conference on Learning Representations_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Provilkov et al. (2020) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020.  BPE-dropout: Simple and effective subword regularization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1882–1892, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Shang et al. (2021) Wei Shang, Chong Feng, Tianfu Zhang, and Da Xu. 2021.  Guiding neural machine translation with retrieved translation template.  _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–7.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Wang et al. (2019) Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019.  Tree transformer: Integrating tree structures into self-attention.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1061–1070, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2018) Wei Wu, Houfeng Wang, Tianyu Liu, and Shuming Ma. 2018.  Phrase-level self-attention networks for universal sentence encoding.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3729–3738, Brussels, Belgium. Association for Computational Linguistics.
  * Xu et al. (2020) Hongfei Xu, Josef van Genabith, Deyi Xiong, Qiuhui Liu, and Jingyi Zhang. 2020.  Learning source phrase representations for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 386–396, Online. Association for Computational Linguistics.
  * Yang et al. (2018) Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018.  Modeling localness for self-attention networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4449–4458, Brussels, Belgium. Association for Computational Linguistics.
  * Yang et al. (2020) Jian Yang, Shuming Ma, Dongdong Zhang, Zhoujun Li, and Ming Zhou. 2020.  Improving neural machine translation with soft template prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5979–5989, Online. Association for Computational Linguistics.