Deep-Q Learning with Hybrid Quantum Neural Network on Solving Maze
  Problems
Quantum computing holds great potential for advancing the limitations of
machine learning algorithms to handle higher dimensions of data and reduce
overall training parameters in deep learning (DL) models. This study uses a
trainable variational quantum circuit (VQC) on a gate-based quantum computing
model to investigate the potential for quantum benefit in a model-free
reinforcement learning problem. Through a comprehensive investigation and
evaluation of the current model and capabilities of quantum computers, we
designed and trained a novel hybrid quantum neural network based on the latest
Qiskit and PyTorch framework. We compared its performance with a full-classical
CNN with and without an incorporated VQC. Our research provides insights into
the potential of deep quantum learning to solve a maze problem and,
potentially, other reinforcement learning problems. We conclude that
reinforcement learning problems can be practical with reasonable training
epochs. Moreover, a comparative study of full-classical and hybrid quantum
neural networks is discussed to understand these two approaches' performance,
advantages, and disadvantages to deep-Q learning problems, especially on
larger-scale maze problems larger than 4x4.
Hao-Yuan Chen, Yen-Jui Chang, Shih-Wei Liao, Ching-Ray Chang
# Deep Reinforcement Learning Using Hybrid Quantum Neural Network

###### Abstract

Quantum computation has a strong implication for advancing the current limitation of machine learning algorithms to deal with higher data dimensions or reducing the overall training parameters for a deep neural network (DNN) model. Based on a gate-based quantum computer, a parameterized quantum circuit (PQC) was designed to solve a model-free reinforcement learning problem with the deep-Q learning method. This research has provided a comprehensive investigation and evaluation of its potential to provide quantum advantage on quantum computers' current model and capability. Therefore, a novel PQC based on the latest Qiskit and PyTorch framework was designed and trained to compare with a full-classical deep neural network (DNN) with and without integrated PQC. At the end of the research, the research draws its conclusion and prospects on developing deep quantum learning in solving a maze problem or other reinforcement learning problems.

 Quantum Machine Learning, Hybrid Neural Networks, Deep Reinforcement Learning

## I Introduction

Quantum machine learning is a novel and highly immature field of study. Its promising implication for combining two powerful areas in computer science and physics, quantum machine learning, has made an interesting topic for researchers to investigate to resolve various challenging problems. However, considering the limitations of Noisy Intermediate-Scale Quantum (NISQ) devices and machine learning algorithm designs, some experimental models haven't demonstrated strong effectiveness and eventually surpass the present performance of a full classical model. Therefore, this research would like to investigate the potential and possibility of integrating a parameterized quantum circuit with a deep neural network to solve a reinforcement learning problem. Ultimately, this research would like to provide some potential improvements that could be made within this hybrid deep learning model. Also, the study is based on the IBM Quantum Systems (IBM-Q) ecosystem to enable rapid testing and evaluation. The quantum computation experiments were tested and trained on the full-classical hardware with IBM Qiskit's Aer simulator. Current robotics devices could incorporate our quantum deep reinforcement learning model with this solution architecture.

### _Quantum Reinforcement Learning_

Based on the current quantum machine learning research, there are two approaches to the reinforcement learning problem; the first is to utilize a quantum walk algorithm and parameterized quantum circuit (PQC) to encode classical agents and environments into a quantum state of information. Unlike the previous method, the second approach incorporates the latest deep learning model and PQC to explore potential speed-up or model size reduction for the present deep learning models.

#### I-A1 Full-Quantum Reinforcement Learning

Quantum reinforcement learning using a full-quantum solution has been discussed before in 2022 [1]. In the paper [citation: QRL Maze], the research use a discrete-time quantum walk to map the state and action space into quantum states, resulting in a quantum maze problem. However, considering the constraints of NISQ devices, there is a limited practical application for this type of solution in the near term. Although the research contains limitations to the scalability, it has demonstrated substantial implications for the present full-quantum reinforcement learning algorithm that can provide speed-up for training under a small problem size, six by six maze size.

#### I-A2 Trainable VQC Model

Another approach to this problem is to build a deep neural network with a variational quantum circuit (VQC) [2]. The research, different from this paper, introduces a VQC as a trainable model without any conventional tensor layers for the frozen-lake environment, which is logically equivalent to the maze problem set used in this research. The research has demonstrated that a well-designed VQC is a trainable model for a reinforcement learning problem. Moreover, the model has shown a strong parameter reduction capability from \(O(n^{3})\) and \(O(n^{2})\) to \(O(n)\).

Considering the limitations of NISQ devices, near-term engineering applications on the first approach will be regarded as impractical. However, the performance of a trainable VQC would massively depend on the circuit design. Therefore, the research proposed a novel architecture from the previous study [1] with the latest IBM Qiskit Quantum Primitive to explore other design variants. Therefore, this research suggested a novel integration of quantum gate primitive with the Qiskit framework and conventional tensor layers as the third scheme to solve this problem class as a much generalized deep learning algorithm.

## II Reinforcement Learning

Reinforcement learning is the third paradigm of machine learning which is a type of machine learning that involves
an agent learning to make a sequence of decisions in an environment to maximize a cumulative reward signal. The agent interacts with the environment by taking action and receiving rewards. The goal is to learn a policy that maps states to actions to maximize the expected cumulative reward over time.

The fundamental mathematical reinforcement learning model can be formalized as a Markov decision-making process (MDP): At each time step t, the agent observes a state \(s_{t}\) of the environment. The agent selects an action \(a_{t}\) from a set of possible actions based on the observed state. The action is executed in the environment, and the agent receives a reward \(r_{t}\) and transitions to a new state \(s_{t+1}\).

The agent updates its policy based on the observed state, selected action, received a reward, and new state, using a learning algorithm. A performance metric guides the reward signal learning algorithm, which assigns a scalar value to each state-action pair based on its desirability. The agent aims to learn a policy that maximizes the expected cumulative reward over time, defined as the sum of the rewards received over a finite or infinite horizon, discounted by a factor gamma.

The mathematical model of RL can be represented as a Markov decision process (MDP), which is a tuple

\[(S,A,P,R,\gamma) \tag{1}\]

Where: S is a set of states where the environment can be. A is a set of actions that the agent can take. Given the current state and action, p is a probability distribution over the next state. R is a reward function that assigns a scalar reward to each state-action pair. Gamma is a discount factor that determines the importance of future rewards relative to immediate rewards. The goal of RL is to find a policy pi(s) that maps each state to a probability distribution over actions, such that the expected cumulative reward over time is maximized:

\[J(pi)=E[R_{0}+\gamma_{1}+\gamma^{2}R_{2}+...|s_{0},pi], \tag{2}\]

where\(R_{t}\) is the reward received at time step t, and\(s_{0}\)is the initial state of the computation.

RL algorithms can be categorized into model-based and model-free approaches. Model-based approaches learn a model of the environment, including the transition probabilities and reward function, and use this model to compute an optimal policy. Model-free methods, such as Q-learning and SARSA, directly estimate the value of the state-action pairs and use this estimate to update the policy. In this research, the researcher proposed integrating a novel parameterized quantum circuit and hybrid neural network to better approximate the relation of state-action based on the model-free approach framework.

### _Q-Learning_

Q-learning is a model-free reinforcement learning algorithm that learns an optimal policy for an agent in an environment with discrete states and actions. It is called "Q-learning" because it learns an estimate of the Q-value function, which is the expected cumulative reward for taking a step in a given state and following the optimal policy afterward.

The Q-value function is estimated using a table or function approximator. The Q-learning algorithm starts with an initial Q-value function, which is gradually updated as the agent interacts with the environment. The agent observes the current state of the environment, selects an action using an exploration-exploitation strategy (such as epsilon-greedy), and receives a reward from the environment. Based on this reward and the resulting state, the Q-value function is updated using the Bellman equation:

\[Q(s_{t},a_{t})\leftarrow\] \[Q(s_{t},a_{t})+\alpha\left[r_{t+1}+\gamma\max_{a}Q(s_{t+1},a)-Q(s _{t},a_{t})\right] \tag{3}\]

### _Deep-Q Learning_

Deep Q-learning is a reinforcement learning algorithm that uses deep neural networks to approximate the Q-function, which measures the expected cumulative reward for taking action in a particular state. The Q-function is learned through an iterative process where the agent takes actions in the environment and receives rewards. The agent uses the experiences gathered from these interactions to update its estimates of the Q-function using a technique called temporal-difference learning.

In deep Q-learning, a deep neural network approximates the Q-function. The neural network takes the state of the environment as input and outputs a vector of Q-values for each possible action. The agent then chooses the action with the highest Q-value and executes it in the environment.

The neural network training minimizes the difference between the predicted and target Q-values obtained using the Bellman equation. The Bellman equation expresses the expected cumulative reward for taking action in a particular state as the sum of the immediate reward and the discounted expected cumulative reward from the next state. The loss function used to train the model adopts the mean square error function to calculate the loss of the deep neural network's current parameter configuration.

\[L=\left(r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{target})-Q(s, a;\theta)\right)^{2} \tag{4}\]

The computational complexity of solving a maze problem using Q-learning and deep Q-learning depends on several factors such as the size of the maze, the complexity of the environment, the number of actions available to the agent, and the number of episodes or iterations required to converge to an optimal policy.

In Q-learning, the agent learns the optimal policy by iteratively updating a Q-value table, which stores the expected reward for each state-action pair. The algorithm requires a significant amount of memory to store the Q-value table, which grows linearly with the size of the maze and the number of possible actions. The time complexity of Q-learning is proportional to the number of iterations required to converge to an optimal policy, which can be large for complex environments.

On the other hand, deep Q-learning uses a deep neural network to approximate the Q-value function, reducing the
memory requirement and allowing for more efficient learning. However, the time complexity of deep Q-learning is proportional to the number of training iterations required to train the neural network, which can be large for large-scale environments. In addition, the computational complexity increases with the neural network architecture's complexity and the input data's size.

Solving a maze problem using Q-learning or deep Q-learning can be computationally expensive, especially for large-scale environments. However, advances in hardware and software technology have made it possible to implement these algorithms efficiently and effectively, making them useful for various applications in reinforcement learning and robotics.

As parameterized quantum circuits (PQCs) and deep neural networks (DNNs) share some similarities in their architecture and learning processes, which makes them helpful in solving similar types of problems, the research integrates both models using Qiskit's PyTorch Connector interface to facilitate a hybrid deep neural network. The study would like to investigate the potential complexity reduction for using PQC-DNN architecture to reduce the amount of energy and computational resources needed for the training.

Therefore, this research proposed a novel architecture to approach a deep-Q learning problem using a hybrid deep neural network in combination with a deep neural network and parameterized quantum circuit. The development process includes the following components, environment, and training agent.

### _Environment_

This is the system or simulation that the agent will interact with. The environment provides the agent with observations and receives actions from the agent. The environment can be anything from a simple game to a complex physical system.

### _Agent_

Firstly, state preparation is needed to encode the current state of the environment so that it can be fed into the agent's neural network. This can be done using a variety of techniques, such as raw pixels or more abstract features. Next, the agent uses its neural network to select an action based on the current state. The neural network takes the state as input and outputs a set of Q-values, which represent the expected future rewards for each possible action. The agent selects the action with the highest Q-value (or a random action with some probability, to encourage exploration). After that, the agent selects an action, the environment provides a reward signal based on the new state and the chosen action. This reward is used to update the neural network's Q-values. In addition, the agent could facilitate a replay-memory mechanism to store its experiences (state, action, reward, next state) in a replay buffer. The replay buffer is used to randomly sample past experiences, which are then used to update the neural network's Q-values. This helps to prevent the agent from over-fitting to recent experiences.

The most critical part of the agent's training process is the neural network used to estimate deep-Q value. The agent's neural network is a deep neural network that takes the current state as input and outputs a set of Q-values for each possible action. The network is trained using a combination of supervised learning (to minimize the difference between the predicted Q-values and the actual rewards) and reinforcement learning (to encourage the network to learn from its own experiences).

In the end, a complete training loop is needed to allow the agent interacts with the environment. The training loop is an iterative process to search for the optimal Q-value in this research. During the training loop, the agent interacts with the environment, stores its experiences in the replay buffer, samples experiences from the replay buffer to update the neural network, and updates the target network periodically. The training loop continues until the agent reaches a satisfactory level of performance.

## III Experiment Environment

Next, the experiment is set up via a classic maze problem. A maze problem refers to finding a path from a starting point to a goal point in a maze or labyrinth. A maze can be represented as a grid of cells, where each cell can be either a wall or a passage. The goal of the maze problem is to find a path from the starting cell to the exit cell while avoiding the walls. The maze problem can be modeled mathematically using a graph with an adjacent matrix, where each node in the graph represents a location in the maze, and each edge represents a possible path between two locations. We can represent the graph using an adjacency matrix \(A\), where \(A_{ij}\) is one if there is a path from node \(i\) to node \(j\), and 0 otherwise.

Maze problems are commonly used in computer science as a benchmark for path-finding algorithms. One common approach to solving a maze problem is to use graph-based algorithms, such as breadth-first search (BFS) or depth-first search (DFS), to explore the maze and find a path from the starting cell to the exit cell. However, a common problem of these algorithms is high computational complexity. However, Maze problems can also be solved using machine learning algorithms, such as reinforcement learning, which involves training an agent to find a path through the maze by rewarding it for reaching the goal and penalizing it for hitting walls or taking inefficient routes. Therefore, this research introduces a

Fig. 1: Overall research architecture, the environment, and the agent are mostly at classical states of information with determining using binary encoding, however, the parameterized quantum circuit (PQC) is using quantum state of information to help the deep neural network estimates the optimal Q-value of the deep reinforcement learning problems.
maze problem to understand the potential performance of our hybrid model suitable for this type of problem class. The maze size used in this research is 20 by 20 with a maze generator using depth-first search, which provides reproducible research results and performance evaluations.

## IV Hybrid Deep Neural Network

The hybrid neural network developed in this research contains two primary components, conventional neural layers with multiple tensors included and the parameterized quantum circuit integrated into the last layer of the neural network. Therefore, developing a hybrid network requires PQC and deep neural model design to form a generalized model for deep-Q learning problems.

### _Deep Neural Network Architecture_

Table 1 shows the overall architecture of the hybrid deep neural network in this research. The deep neural network (DNN) for the research defines a fully connected neural network (fc nn layer) as a subclass of the nn.Module class in the PyTorch library. The neural network has four fully connected layers (dense layer) (fc1, fc2, fc3, and fc4) with the ReLU activation function applied after the first two layers. The input arguments specify the sizes of these layers to acquire the size of the maze as the dimension of the input layer.

In addition, the neural network also has a Quantum Neural Network (QNN) layer, represented by the TorchConnector (QNN) interface. This object connects the QNN layer to the rest of the neural network and allows it to be trained along with the other layers. Additionally, the network has two linear layers (fc1 and fc5) that transform the output of the qnn layer into a final output. The fc1 layer has a single input and output, while the fc5 layer has a single input and four outputs.

The class's forward method defines the neural network's forward pass. It takes an input tensor x and performs the following operations: applies the ReLU activation function to the output of the first two fully connected layers, then uses the remaining layers (fc3, fc4, qnn, fc1, and fc5) sequentially. The forward method returns the output tensor of the fc5 layer, which has a shape of (batch size, 4). Moreover, the model summary is shown below:

### _Parameterized Quantum Circuit_

Next, a parameterized quantum circuit is designed with the IBM Qiskit framework. This code sets up a quantum neural network (QNN) using IBM's latest Estimator-QNN module. The QNN consists of a quantum circuit created using the feature map and Ansatz modules and is defined over two qubits.

#### Iv-B1 Quantum Feature Map

The feature map module, \(ZZFeatureMap\), creates two qubits quantum circuit that applies a parameterized sequence of gates to the input state. This feature map is designed to encode classical input data into a quantum state that the quantum circuit can process. The ZZ feature map is a quantum circuit used in quantum machine learning that can encode classical input data into a quantum state. It consists of a sequence of single-qubit rotations and controlled Z gates. The controlled-Z gates apply a phase shift to the quantum state depending on the product of the two

Fig. 3: An overall architecture of hybrid DNN.

Fig. 2: An untrained maze path configuration, whereas the black cells represent the block within the maze and the white cell, represents all of the possible routes within the maze. The arrows in the maze present the current agent travel policy based on the current experiences.
qubits' states, which is equivalent to using a phase shift of \(\exp(i*\theta)\) if the qubits are in the state --11?, where theta is a free parameter that can be learned during the training of the quantum circuit.

The feature map module, \(ZZFeatureMap\) is a second-order Pauli-Z evolution circuit is a quantum circuit that applies a series of gates to a set of qubits to implement a unitary transformation. The circuit is constructed using two-qubit gates that act on pairs of qubits and single-qubit gates that act on individual qubits.

Specifically, the second-order Pauli-Z evolution circuit is a sequence of gates that alternate between a controlled-Z gate (also known as a CZ gate) and a single qubit rotation gate around the Z-axis (also known as an RZ gate). The CZ gate acts on a pair of qubits and applies a conditional phase shift of -1 to the second qubit if the first qubit is in the state \(|1\!\!>\). The RZ gate applies a rotation around the Z-axis to a single qubit, with the angle of rotation determined by a parameter \(\theta\).

\[U=e^{-i\theta/2\sum_{j=1}^{n}Z_{j}}\cdot\prod_{j=1}^{n-1}\left(\prod_{k=j+1}^{ n}CZ_{jk}\right) \tag{5}\]

Equation 5 illustrates the mathematical model of the unitary operation of the quantum ZZ-Feature Map module. The circuit is called "second-order" because it is a generalization of the first-order Pauli-Z evolution circuit, which applies only CZ gates to implement a unitary transformation. By adding RZ gates to the circuit, the second-order Pauli-Z evolution circuit can implement a broader range of unitary transformations.

As the second-order Pauli-Z evolution circuit is helpful for various quantum computing applications, including quantum error correction, quantum state preparation, and quantum simulation, the research adopts this feature map module to prepare and load the classical data into a quantum state for the unitary operation within a trainable quantum circuit.

#### Iv-B2 Trainable Quantum Circuit

After the state preparation stage, the first layer consists of single-qubit rotations, where each qubit is rotated by an angle that depends on the input data. The second layer consists of controlled Z gates that apply a phase shift to the quantum state depending on the product of the two qubits' states. The number of controlled-Z gates in the circuit equals the number of unique pairs of qubits. Moreover, the \(R_{z}\) gate, in the trainable quantum circuit is a single-qubit gate used in quantum circuits to perform a rotation around the Z-axis of the Bloch sphere. It is represented by the following matrix:

\[Rz(\theta)=\begin{pmatrix}e^{-i\theta/2}&0\\ 0&e^{i\theta/2}\end{pmatrix} \tag{6}\]

As \(R_{z}\) gate rotates the quantum around the Z-axis of the Bloch sphere is particularly useful and common in quantum machine learning to approximate the real value of the complex linear system to understand the behavior of the model, for example, the Q-function in this research. Also, since the classical data is encoded by using ZZ-Feature Map it would be helpful for us to facilitate a trainable model around Z-axis rotation.

#### Iv-B3 Quantum States Measurement

The Ansatz module, Real-Amplitudes, is a template circuit that applies layers of parametrized rotations and entangling gates to the input state. It has two qubits and one repetition of the circuit layer. The Quantum-Circuit(2) function defines an open quantum circuit of two qubits.

Finally, the Estimator-QNN module defines a quantum neural network (QNN) that uses the quantum circuit as the model. The feature map and Ansatz parameters specify the circuit's input and weight parameters. The input gradient parameter is set to True, enabling the hybrid gradient backpropagation algorithm for training the QNN. This allows gradients to be backpropagated from the output of the QNN to the input parameters of the feature map module.

### _Training Algorithm_

The training algorithm for this hybrid deep neural network is a variant of the Q-learning algorithm, a widely used model-free reinforcement learning algorithm. The algorithm uses a deep neural network as a function approximator to estimate the optimal Q-value function, which maps state-action pairs to their expected future reward. Optimization minimizes the mean squared error between the predicted and target Q-values, computed using a Bellman equation.

The algorithm also employs experience replay, where past transitions are stored in a buffer and sampled randomly during training. This reduces the correlation between consecutive samples and stabilizes the training process. The agent updates the neural network weights using the Adam optimizer, which adapts the learning rate based on the gradient data.

The algorithm takes as input the size of the batch, the discount factor gamma, and the device for running the computations. The loss function is the Q-loss, which calculates the mean squared error between the predicted and target Q-values. The \(optimizer.zero_{g}rad()\) method clears the gradients of the optimizer, and the backward() method computes the gradients of the loss function to the neural network parameters. Finally, the optimizer.step() method updates the neural network weights using the calculated gradients.

## V Results

The research was trained and evaluated on the local machine with NVIDIA's CUDA acceleration framework on the RTX 3080 Ti GPU. In addition to the training infrastructure, the research uses PyTorch and IBM Qiskit to develop the PQC and DNN used in the study.

Fig. 4: An conceptual model of the parameterized quantum circuit for this research
### _Training Profile_

Firstly, the model training configuration is defined as an Epsilon profile. In reinforcement learning, an agent interacts with an environment and learns to take actions that maximize a cumulative reward signal. One common strategy for balancing the exploration of new actions with the exploitation of known good actions is the epsilon-greedy strategy. This strategy involves choosing a random action with probability epsilon and choosing the action with the highest expected reward with probability (1-epsilon).

The value of epsilon is typically set to decrease over time, resulting in an "epsilon profile" that describes how the agent's exploration behavior changes as it gains more experience. Initially, the agent may explore more (i.e., set epsilon to a higher value) to discover new actions that may lead to high rewards. As the agent learns which actions are most likely to lead to high rewards, it may reduce its exploration (i.e., set epsilon to a lower value) to focus more on exploiting known good actions.

In this context, the epsilon units are arbitrary and depend on the specific implementation of the reinforcement learning algorithm. The value of epsilon can range from 0 to 1, with a value of 0 meaning the agent always exploits known good actions and a value of 1 meaning the agent always explores new actions. The specific epsilon profile used will depend on the task being solved and the system's performance requirements.

### _Training Results_

Figure 5 shows the significant difference from the quantum deep neural network's training results. The figure on the right shows the initial environment setting without any training. However, the picture on the right demonstrates a path found in just 3000 epochs of training, combing both PQC and DNN.

Ultimately, our hybrid deep neural network demonstrates the capability and potential of near-term quantum deep learning solutions using a generalized PQC design. The findings suggest that the proposed hybrid network can perform highly in various applications. Using a combination of classical and quantum machine learning techniques, the researchers demonstrated the potential for achieving improved performance in challenging problems.

These results suggest hybrid deep learning approaches using generalized PQC designs could be vital in developing future quantum machine learning applications. Moreover, as this quantum-classical scheme, the solution can be trained on the GPU-accelerated hardware with IBM Aer and NVIDIA's cuQuantum library to explore future applications based on this model architecture.

## VI Discussion

The research provides the third architecture for approaching quantum reinforcement learning problems using a hybrid neural network. This section provides a comparison to two other architectures, full-quantum and trainable VQC models.

### _Differences from full-quantum model_

The fundamental difference between this research and the full-quantum model is the state of action and environment. The full-quantum model encodes all the state and action space into a quantum state using a discrete-time quantum walk with a hopping mechanism. The research demonstrates a strong performance and high level of parallelism possible for a full-quantum solution. The degree of complexity reduction is \(O(\sqrt{n})\). However, this solution is most sensitive to the noise within the quantum devices with the lowest scalability potential.

Fig. 5: Model’s Epsilon profile

Fig. 6: Left: Initial States at zero epoch, Right: Final State of Training at 3000 epochs
### _Differences from trainable VQC model_

Therefore, a trainable quantum circuit model is proposed to estimate the Q-value for the Bellman equation. The model also provides a potential speed-up and parameter reduction from \(O(n^{2})\) and \(O(n^{3})\) to \(O(n)\). Unlike our research, the VQC's approach doesn't include any classical tensor layers within the trainable model, which increases the difficulty for the model to generalize into other problem sets. However, this research has provided a theoretical and empirical foundation for the logical similarity between VQC and DNN.

### _Hybrid Neural Network_

As a result, a hybrid deep neural network incorporating the PQC model is introduced to mitigate the challenges from the previous two related research. With multiple classical neural layers, the model could resolve many generalized problems with various input dimensions. Also, as this neural network is full-trainable and functional on the classical device, a strong implication and potential for near-term applications emerged. However, some technical constraints are still involved within this solution architecture, like quantum-classical gradient descent, accelerated computing devices for this model class, etc.

### _Outlook_

The present challenges of this research are to explore an efficient training method on both classical simulators and real quantum hardware. However, in light of current constraints to hardware accessibility and software architecture, it is challenging to train a hybrid model on a real quantum computer. However, a well-architected training scheme can be designed on the GPU cluster with NVIDIA cutQuantum, IBM Qiskit, and PyTorch framework to reduce the amount of intercommunication between cores and processors, which results in the slow training for this model based on the various iteration of this experiments.

## VII Challenges

This research presents an introduction and technical demonstration of the quantum reinforcement learning algorithm with a hybrid neural network. The neural network comprises the latest circuit architecture with novel IBM Quantum Computing Primitive built-in. The results imply the future trajectory and application of parameterized quantum circuits (PQC) to accomplish an even more complex environment for deep reinforcement learning agent training. In addition, the research demonstrates how near-term quantum devices could provide potential speed-up or parameter reduction to the current deep learning model. In the end, the study would like to integrate a split-steps quantum walk circuit model with a deep neural network and PQC to load the classical state-space into a quantum state of information in which the current algorithm might provide a further advance in terms of speed and model size reduction for such applications.

## Acknowledgment

The authors would like to present immense gratitude to the Ph.D. candidates, Yen-Jui Chang and Wei-Ting Wang at National Taiwan University for their support and advisory on this project. Also, a big thanks to Prof. Shih-Wei Liao, Prof. Ching-Ray Chang, and Prof. Horng Wen-bing for providing professional guidance on research trajectories.
