Quantum Multi-Agent Actor-Critic Neural Networks for Internet-Connected
  Multi-Robot Coordination in Smart Factory Management
As one of the latest fields of interest in both academia and industry,
quantum computing has garnered significant attention. Among various topics in
quantum computing, variational quantum circuits (VQC) have been noticed for
their ability to carry out quantum deep reinforcement learning (QRL). This
paper verifies the potential of QRL, which will be further realized by
implementing quantum multi-agent reinforcement learning (QMARL) from QRL,
especially for Internet-connected autonomous multi-robot control and
coordination in smart factory applications. However, the extension is not
straightforward due to the non-stationarity of classical MARL. To cope with
this, the centralized training and decentralized execution (CTDE) QMARL
framework is proposed under the Internet connection. A smart factory
environment with the Internet of Things (IoT)-based multiple agents is used to
show the efficacy of the proposed algorithm. The simulation corroborates that
the proposed QMARL-based autonomous multi-robot control and coordination
performs better than the other frameworks.
Won Joon Yun, Jae Pyoung Kim, Soyi Jung, Jae-Hyun Kim, Joongheon Kim
Quantum Multi-Agent Actor-Critic Neural Networks for Internet-Connected Multi-Robot Coordination in Smart Factory Management

###### Abstract

As one of the latest fields of interest in both academia and industry, quantum computing has garnered significant attention. Among various topics in quantum computing, variational quantum circuits (VQC) have been noticed for their ability to carry out quantum deep reinforcement learning (QRL). This paper verifies the potential of QRL, which will be further realized by implementing quantum multi-agent reinforcement learning (QMARL) from QRL, especially for Internet-connected autonomous multi-robot control and coordination in smart factory applications. However, the extension is not straightforward due to the non-stationarity of classical MARL. To cope with this, the centralized training and decentralized execution (CTDE) QMARL framework is proposed under the Internet connection. A smart factory environment with the Internet of Things (IoT)-based multiple agents is used to show the efficacy of the proposed algorithm. The simulation corroborates that the proposed QMARL-based autonomous multi-robot control and coordination performs better than the other frameworks.

 Quantum deep learning, multi-agent reinforcement learning, quantum computing, robot control, smart factory

## I Introduction

In various Industry 4.0 scenarios, automated and autonomous management of smart factory systems are getting a lot of attention nowadays [2, 3, 4, 5, 6, 7, 8]. For the automation of factory management, the use of autonomous multiple mobile robots is widely studied [9, 10, 11]. According to the Verizon Report [12], _Industry 4.0 is squarely underway in manufacturing. The global market is expected to reach $219.8 billion by 2026, and autonomous mobile robots are becoming key workhorses in this transformation_. To realize the efficient and effective autonomous multi-robot control and coordination, multi-agent reinforcement learning (MARL)-based algorithms are essentially required [13, 14].

Recently, revolutionary innovations have been made in distributed learning and MARL due to the remarkable evolution in computing hardware and deep learning algorithms [14]. Moreover, the developments in quantum computing hardware and algorithms placed further emphasis on this trend [15], resulting in the incentivization of the research on quantum machine learning. Nowadays, quantum machine learning is at a newborn level compared to conventional machine learning. For instance, in the classification task of quantum machine learning, the performance of quantum machine learning is low given the MNIST dataset at 32.5% of top-1 accuracy [16] on quantum computers, and 74.2% [17] on ideal quantum machines. However, the theoretically discovered advantages (_i.e., quantum supremacy_) are being experimentally proven recently [18, 19, 20]. The potential of quantum algorithms is evident from their ability to downsize the model parameters while maintaining accuracy by exploiting quantum entanglements [21]. In addition, the empirical result of [22] shows that quantum machine learning outperforms the empirical result of classical machine learning. An outstanding example of this is the variational quantum circuit (VQC) architecture, also known as a quantum neural network (QNN) [23, 24]. QNN is a quantum circuit that reproduces the function of a classical deep neural network. By combining the QNN and classical deep learning models, hybrid quantum-classical models are built, which allow QRL to be carried out. Compared to RL, QRL uses lesser model parameters but significantly reduces the training and inference time [25, 26] while consuming lesser computing resources as well [27]. Thus, it is clear that quantum machine learning using quantum computing will become a big trend in the near future. This paper aims to combine VQC with the classical MARL to extend QRL to quantum MARL (QMARL).

The agents in the MARL environment interact with each other by either cooperating or competing. This interaction is realized based on Internet-of-Things (IoT)-based connectivity technologies. These interactions result in a non-stationary reward for each agent, which hinders the convergence of MARL training. The centralized training and decentralized execution (CTDE) method is used [28] to deal with the non-stationarity of the MARL model. In this scenario, the reward is distributed to all agents concurrently by concatenating their state-actions pairs. A naive implementation of a VQC version of CTDE is possible, as shown in [1]. However, such implementation causes the qubits to increase with the number of agents because when QRL is carried out via VQC, the state-action pairs are represented by qubits. Consequently, quantum errors will also increase with the qubits [29], significantly
affecting the MARL convergence and scalability. Furthermore, quantum error correction is not yet viable in the current noisy intermediate-scale quantum (NISQ) era.

This paper intends to improve on various pre-existing methods of implementing VQCs, agent policies, and state encoding [25, 30, 31]. Three significant differences exist for our proposed VQC compared to previous works, which are parameter sharing, non-random VQC design, and 2-variables dense encodings. Firstly, parameter sharing refers to sharing model parameter values between agents. All the agents in previous works had individual, distinct policies meaning more agents required more policies, resulting in excess computing power consumption in the process of formulating them. In this improved model, there will only be one policy that will be shared among the agents, increasing computing power efficiency. The second improvement is the non-random VQC design. The VQCs used in previous works are composed of randomly selected quantum gates. Although the performance is remarkable, it cannot be easily reproduced because of its randomness. The same model might not show the same performance in another iteration because of the random quantum gates. However, this is improved in this paper by designing a fixed model and removing the random nature of the previous VQC. This ensures the reproducibility and stability of the model. Finally, the proposed model in this paper utilizes the 2-variables dense encoding method instead of the 4-variables dense encoding method. The original encoding method is capable of reducing the dimensions of given data. Although this may be good for the NISQ-era quantum circuits, it inevitably causes a loss of information. The proposed 2-variables dense encoding method does not reduce data dimensions, but it is still compatible with NISQ-era quantum circuits. Thus, information loss is prevented, which will improve the performance of this model.

**Contributions.** The major contributions of this research are summarized as follows.

* This paper first provides a quantum-based MARL solution for autonomous multi-robot control and coordination in smart factory applications.
* An improved and novel CTDE QMARL framework which utilizes parameter sharing on policy, VQC design, and 2-variables dense encodings is additionally proposed.
* Lastly, via extensive experiments, the proposed QMARL framework is proven to be superior to the classical MARL model by carrying out simulations in smart factory scenarios. The results show that the proposed model produces higher performance than the others.

**Organization.** The rest of this paper is organized as follows. The preliminaries of this paper are described in Sec. II. Our considering autonomous mobile robots coordination for smart manufacturing is described in Sec. III. Sec. IV introduces our proposed algorithm; and the numerical results and demonstration of the proposed algorithm are shown in Sec. V. Sec. VI concludes this paper and presents future work. Note that the notations in this paper are listed in Table I. Most equations and notations used here are based on the _Dirac_ notations used in [32].

## II Preliminaries of Quantum Computing

**Single Qubit Quantum State.** QC utilizes a _qubit_ as the basic unit of computation. The qubit represents a quantum superposition state between two basis states, denoted as \(|0\rangle\) and \(|1\rangle\). There are two ways to describe a qubit state,

\[|\psi\rangle=\alpha|0\rangle+\beta|1\rangle, \tag{1}\]

where \(\|\alpha\|_{2}^{2}+\|\beta\|_{2}^{2}=1\), as well as,

\[|\psi\rangle=\cos\left(\frac{\delta}{2}\right)|0\rangle+e^{i\varphi}\sin\left( \frac{\delta}{2}\right)|1\rangle, \tag{2}\]

where \(\delta\in[-\pi,\pi]\) and \(\varphi\in[-\pi,\pi]\). The former is based on a normalized 2D complex vector, while the latter is based on polar coordinates \((\delta,\varphi)\) from a geometric viewpoint. The qubit state is mapped into the surface of a 3D unit sphere (_Bloch sphere_). In addition, a quantum gate is a unitary operator transforming a qubit state into another qubit state, which is represented as a \(2\times 2\) matrix with complex entries. The single-qubit Pauli gates \(X\), \(Y\), and \(Z\) are defined as follows,

\[X=\begin{bmatrix}0&1\\ 1&0\end{bmatrix},\quad Y=\begin{bmatrix}0&-i\\ i&0\end{bmatrix},\quad Z=\begin{bmatrix}1&0\\ 0&-1\end{bmatrix}. \tag{3}\]

There are additional quantum gates that are frequently used, i.e., \(R_{\text{s}}\), \(R_{\text{y}}\), and \(R_{\text{z}}\). These are rotation operator gates rotate a single qubit by \(\delta\) around their corresponding axes in the Bloch sphere and the single qubit operation can be expressed as the following equations,

\[R_{X}(\delta)=e^{-i\frac{\delta}{2}X},\quad R_{Y}(\delta)=e^{-i\frac{\delta}{2 }Y},\quad R_{Z}(\delta)=e^{-i\frac{\delta}{2}Z}, \tag{4}\]

where rotation angles are denoted as \(\delta\in\mathbb{R}[0,2\pi]\). These basic Pauli and rotation gates are unitary matrices, \(U^{\dagger}U=I\), where \(I\) denotes an identity matrix.

\begin{table}
\begin{tabular}{c|l} \hline \hline \multicolumn{2}{c}{Scenario Notations} \\ \hline \(N\) & The number of AMR agents \\ \(M\) & The number of sites/warehouses \\ \(T\) & An episode length \\ \(o^{n}\) & The observation of \(n\)-th AMR agent \\ \(a^{n}\) & The action of \(n\)-th AMR agent \\ \(\mathbf{a}\) & The actions set of AMR agents, i.e., \(\mathbf{a}=\{a_{n}\}_{n=1}^{N}\). \\ \(s\) & The ground truth state \\ \(q_{n,t}^{e}\) & The load status of \(m\)-th warehouse at time \(t\) \\ \(q_{n,t}^{e}\) & The load status of \(m\)-th AMR agent at time \(t\) \\ \(q_{\max}^{e}\) & A load capacity of warehouse \\ \(q_{\max}^{e}\) & A load capacity of AMR agent \\ \hline \multicolumn{2}{c}{Quantum Computing Notations} \\ \hline \(|\psi\rangle\) & Entangled quantum state \\ \(\langle O\rangle\) & Observable \\ \(\Gamma\) & Pauli-\(\Gamma\) gate, e.g., \(\Gamma\in\{X,Y,Z\}\) \\ \(R_{\Gamma}\) & Rotating \(\Gamma\) gate, e.g., \(\Gamma\in\{X,Y,Z\}\) \\ \((\cdot)^{\dagger}\) & Complex conjugate operator \\ \(\mathcal{M}\) & Measurement operator \\ \hline \hline \end{tabular}
\end{table} TABLE I: List of notations
**Multi-Qubit Quantum State.** Multi-qubit system enables super-fast quantum computing due to quantum superposition. The well-known quantum algorithms (e.g., Shor algorithm [33] and Grover search [34]) are based on the multi-qubit system. The quantum state with \(q\) qubits is denoted as \(|\mathbf{\psi}\rangle=|\psi_{1}\rangle\otimes|\psi_{2}\rangle\otimes\cdots\otimes| \psi_{q}\rangle=\sum_{n=0}^{2^{q-1}}\alpha_{n}|n\rangle\), where \(\otimes\), \(\alpha_{n}\) and \(|n\rangle\) stand for superposition operator (i.e., tensor-product), and \(n\)-th probability amplitude and \(n\)-th basis of \(q\)-qubits quantum state, respectively. Note that the sum of squared magnitude of probability amplitude equals 1, _i.e._, \(\sum_{n=0}^{2^{q}-1}|\alpha_{n}|^{2}=1\)[32]. To realize quantum superposition, there are quantum gates that operate on multiple qubits, called controlled rotation gates. They act on a qubit according to the signal of several control qubits, which generates quantum entanglement between those qubits. Among them, the _Controlled-\(X\)_ (or CNOT) gate \(\textit{CX}=\begin{bmatrix}I&0\\ 0&X\end{bmatrix}\) is one of the widely used control gates. These multi-qubit gates allow quantum algorithms to work with their features on VQC, which will eventually be utilized for QMARL.

## III Autonomous Mobile Robots Coordination for Smart Manufacturing

### _Design of an Autonomous Mobile Robot System_

An automated guided vehicle (AGV) is a portable robot that travels along lines or wires marked on the floor or navigates using radio waves, vision cameras, magnets, or lasers. AGVs are widely used in industrial applications to transport heavy materials around large industrial facilities such as factories and warehouses. Therefore, it is obvious that AGVs are essential to smart factory management. The autonomous mobile robot (AMR) differs from AGV because it has various sensors that enable autonomous location identification and search by detecting surrounding static and dynamic objects. Their paths are generated based on static and dynamic obstacles in real time so that AMR can travel freely without a predefined path. While the system is more flexible, real-time path generation poses additional challenges that fleet management systems (FMSs) must deal with such as, performing activities, e.g., shipping transport orders, routing vehicles, and scheduling task execution. Note that AMRs are tightly combined, which leads to high computational complexity. For example, the performance of AMRs suffer when considering all possible AMR paths, even though the numbers of AMRs and transfer orders are relatively small. As a result, centralized AMR fleet management and order execution optimization are often not performed in real time. Therefore, the use of MARL algorithms is widely considered and studied [35]. For further performance improvement, QMARL can be additionally utilized, as we discuss in this paper.

### _Automated LCD Smart Factory with Multiple AMRs_

Thanks to the properties of QC, QC has shown that QC could save many orders of magnitude in energy consumption compared to classical supercomputers [36]. Regarding QRL, recent studies show quantum supremacy [22]. In this paper, we consider a liquid crystal display (LCD) smart factory system which utilizes DC-based AMRs. As shown in Fig. 1, the color thin-film transistor (TFT) LCD panel consists of two glass substrates; a TFT array substrate and a color filter substrate. TFT LCD panels are fabricated by a combination of five processes; TFT array filter process, color filter process, repair process, cell fabrication process, and module assembly process. The first two processes (i.e., TFT array filter and color filter processes) are carried out at Site A, the two substrates are carried by AMR and the rest of the process is carried out at Site B. Each AMR has a role in the transition from providing services to flexible areas that require decisions to be made based on dividing the service area into several zones. In the process of manufacturing TFT LCDs, various defective LCDs can occur which should not be used. Therefore, to prevent the usage of such defective LCDs, the AMRs must identify the defective products and request a quality verification of the LCD. Techniques of detecting defects among LCDs have already been developed and implemented in smart factories [37, 38, 39]. By using the _precision_ parameter proposed by the works above, the AMRs will recognize defective LCDs and unload them in another collection point dedicated for defects.

In this paper, we assume that all AMRs have the optimal trajectory planners and charging schedulers such as [40, 41]. Thus, our proposed QMARL model must plot the trajectory of each AMR such that the defective LCDs are separated while the normal products are properly unloaded. For communication, the QDL server is wire-linked to every site, and each site is wirelessly connected with AMRs. Since the packet size is small, and the transmit power is sufficient in LCD smart factory, we assume that the packet loss is negligible. The QDL server receives observation from AMRs, reconfigures the state, and finally transmits action decisions to AMRs. For flexible manufacturing, AMRs should be properly planned to load goods, unload goods, and control the quality of LCD. Moreover, the decision-making process of scheduling and dispatching these resources is essential for optimal utilization and high AMR productivity performance.

**Problem Definition and Formulation.** In this situation, a quantum deep learning (QDL) server supports the decision-making process for efficiently scheduling material handling systems, under the fundamental concept of the CTDE-based QMARL framework. Specifically, the QDL server makes distributed and sequential decisions for each AMR to determine their goods (i.e., the number of goods to carry in each AMR and requesting quality control) for eliminating the overflow and underflow of delivering goods in each AMR.

## IV Quantum Multi-Agent Actor-Critic Network for Autonomous Multi-Robot Coordination

### _Fundamental MDP Formulation_

Our considering autonomous multi-robot coordination in a smart factory environment consists of \(M\) sites and \(N\) AMR agents. The smart factory environment is mathematically modeled with POMDP (referred to as Sec. II). Hereafter, we explain the description based on \(m\)-th site, \(n\)-th agent, and time step \(t\).
#### Iii-A1 Load Dynamics

Each site has a warehouse where the load capability is \(c_{\max}^{\text{W}}\). In addition, the load capacities of AMR agents is under the maximum capacities \(c_{\max}^{\text{A}}\). AMR agents receive goods (e.g., LCD panels or TFTs) from other AMRs. In this paper, we denote the load weights \(b_{t}^{\text{A},n}\). The load weights follow the uniform distribution \(\forall b_{t}^{\text{A},n}\sim\mathcal{U}(0,w_{\textit{load}}\cdot b_{\max})\). The warehouse and AMR agents have loading status \(c_{t}^{\text{W},m}\) and \(c_{t}^{\text{A},n}\) that are temporally loaded goods. All AMR agents carry their goods to warehouses. The dynamics are as follows,

\[c_{t+1}^{\xi,n}=\textit{clip}(c_{t}^{\xi,n}-a_{t}^{\xi,n}+b_{t}^{\xi,n},0,c_{ \max}^{\xi}), \tag{5}\]

where \(\xi\in\{\text{W},\text{A}\}\) identifies the warehouse and an AMR agent. The terms \(a_{t}^{\xi,k}\) and \(b_{t}^{\xi,n}\) imply the total delivered goods weights and the received goods weights of \(m\)-th warehouse or \(n\)-th AMR agent, respectively. Note that \(a_{t}^{A,n}\) is \(n\)-th AMR agent's action. In addition, a clipping function is defined as \(\textit{clip}(x,x_{\min},x_{\max})\triangleq\min(x_{\max},\max(x,x_{\min}))\).

#### Iii-A2 Quality Control

We assume that the loads have been classified by the previous defect detection process. In the defect detection process, four types are given to the loads (i.e., true positives, false positives, false negatives, and true negatives), which make the statistics (_i.e._, precision, recall, and F-score). This paper considers the load status as the quality statistics (_e.g._, precision). Among them, quality statistics are given to AMR agents. The AMR agents can make action decisions for re-requesting quality control of the load. If the loads are requested for quality control, the loads undergo quality verification by quality engineers. We assume that quality engineers can detect all defects on load perfectly. However, the quality re-assurance process via quality engineers additionally requires \(\tau_{\textit{p}ual}\) time delay.

#### Iii-A3 Utility Design

We design the utility for quality, time delay, and load balancing. First of all, AMR agents receive the goods with the type of true positives (_TP_) and false positives (_FP_). The metric, i.e., precision, can represent the ratio of positive predictive value, which is written as follows:

\[u_{t}^{q,n}=\frac{\textit{TP}_{t}^{n}}{\textit{TP}_{t}^{n}+\textit{FP}_{t}^{n }}, \tag{6}\]

where \(\textit{TP}_{t}^{n}=\sum_{\textit{load}\in\mathbb{I}_{t}^{n}}\mathbbm{1}( \textit{load}=\textit{TP})\) and \(\textit{FP}_{t}^{n}=\sum_{\textit{load}\in\mathbb{I}_{t}}\mathbbm{1}(\textit{ load}=\textit{FP})\) stand for the true positives and false positives of \(n\)-th AMR, respectively. Note that \(l_{t}^{n}\) denotes the whole load defect status of \(n\)-th AMR. Regarding the delay, we measure the processing time. Thus, the delay utility of \(n\)-th AMR agent is written as follows:

\[u_{t}^{d,n}=-\Big{(}1+\tau_{\textit{quad}}\cdot q_{t}^{n}\Big{)}, \tag{7}\]

where \(q_{t}^{n}\) denotes the quality control action. If \(q_{t}^{n}=1\), the loads are conveyed to quality engineers; otherwise, the loads are conveyed to other-site. Finally, load balancing is to minimize the total amount of overflowed load and the event where the load is empty. Thus, the utility for load balancing is written as follows:

\[u_{t}^{b,n}=\mathbbm{1}_{(c_{t+1}^{A,n}=0)}\cdot\tilde{c}_{t}^{A,n}+ \mathbbm{1}_{(c_{t+1}^{A,n}=c_{\max}^{A})}\cdot\tilde{c}_{t}^{W,n} \tag{8}\] \[u_{t}^{W,m}=\mathbbm{1}_{(c_{t+1}^{W,m}=0)}\cdot\tilde{c}_{t}^{W,m}+\mathbbm{1}_{(c_{t+1}^{W,m}=c_{\max}^{W,m})}\cdot\tilde{c}_{t}^{W,m} \tag{9}\]

where \(\tilde{c}_{t}^{W,m}=|c_{t}^{W,m}-a_{t}^{W,n}+b_{t}^{W,n}|\) and \(\tilde{c}_{t}^{W,n}=|c_{\max}-\tilde{c}_{t}^{W,n}|\). Note that \(r(s_{t},\textbf{a}_{t})\in[-\infty,0]\) (negative) because this paper considers the occurrence of abnormal loading status (e.g., load overflow or underflow) as a negative utility. The objective is to maximize the total precision and minimize the total delay and overflowed or underflowed event.

### _POMDP Setup_

This subsection introduces the formal definition of POMDP, i.e., a stochastic decision-making model under un

Fig. 1: System model: Liquid crystal display panel manufacturing using quantum multi-agent reinforcement learning
certainty among agents [42]; and our proposed QMARL is mathematically modeled with this fundamental concept of POMDP. Note that POMDP is defined as a tuple \(\langle\mathcal{N},\mathcal{S},\mathcal{A},P,r,\mathcal{Z},O,\rho,\gamma,T\rangle\). The sets of states and observations are represented as \(\mathcal{S}\) and \(\mathcal{Z}\), respectively. \(\mathcal{N}:=\{1,\cdots,N\}\) and \(s\in\mathcal{S}\) denote the set of \(N\) agents and the current state of the environment, respectively. The initial state \(s_{0}\sim\rho\) follows the distribution \(\rho\). The action of \(n\)-th agent \(a^{n}\in\mathcal{A}\) is discrete or continuous actions, and the joint action is denoted as \(\mathbf{a}:=\{a^{n}\}_{n=1}^{N}\). The transition is determined with probability function \(P(s^{\prime}|s,\mathbf{a}):\mathcal{S}\times\mathcal{A}\times\mathcal{S} \rightarrow\mathcal{S}\), where \(s^{\prime}\) denotes the next state. The shared reward \(r_{t}=r(s_{t},\mathbf{a}_{t}):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) is given to whole agents. In DecPOMDP, the true state \(s\) is not directly given to agents. Each agent \(n\in\mathcal{N}\) has observation \(z^{n}\in\mathcal{Z}\) from observation function \(O(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{Z}\). We consider that all agents have parameter-shared policy denoted as \(\pi_{\theta}\). Thus, the policy \(\pi_{\theta}\) takes the \(n\)-th agent's observation \(z_{t}^{n}\in\mathcal{Z}\) and decides \(n\)-th agent's action as \(\pi_{\theta}(a|z^{n}):\mathcal{Z}\times\mathcal{A}\rightarrow[0,1]\). The objective of POMDP is to obtain the optimal policy \(\pi_{\theta}^{*}=\arg\max_{\pi_{\theta}}\mathbb{E}_{\mathbf{a}\sim\pi_{\theta} }[\sum_{t=1}^{T}\gamma^{t-1}\cdot r_{t}]\), where \(\gamma\in\mathbb{R}[0,1]\) and \(T\) denote the discount factor and finite time, respectively. Based on this definition, we design the POMDP as follows:

#### Iii-B1 Observation

Each AMR agent partially obtains its observation. Because the parameter shared policy \(\pi_{\theta}\) is used, the observation contains the binary indicator vector \(\mathbf{n}_{2}\). In addition, \(n\)-th AMR agent makes its action decision with its loading status \(c_{t}^{A,n}\), and the current loading status of warehouse \(\{c_{t}^{W,m}\}_{m=1}^{M}\). In summary, \(n\)-th agent's observation is defined as \(z_{t}^{n}\triangleq\{\mathbf{n}_{2},c_{t}^{A,n}\}\cup\{c_{t}^{W,m}\}_{m=1}^{M}\).

#### Iii-B2 State

A state information variable containing information about all AMR agents' and warehouse's loading status is designed. The state variable at time \(t\) is as \(s_{t}=\{c_{t}^{A,n},u_{t}^{d,n}\}_{n=1}^{N}\cup\{c_{t}^{W,m}\}_{m=1}^{M}\). Note that the state information is utilized as the input of the quantum critic network.

#### Iii-B3 Action

It is considered that AMR agents can choose which warehouse to convey goods, where the destination space is defined as \(\mathcal{I}\triangleq\{1,\cdots,M\}\). In addition, AMR agents can determine conveying quantity to the warehouse. The conveying quantity and the quality control space is defined as \(\mathcal{P}\triangleq\{p_{\min},\cdots,p_{\max}\}\) and \(\mathcal{Q}\triangleq\{0,1\}\), respectively. Finally, \(n\)-th AMR agent's action and its action space are defined as \(a_{t}^{n}:=(i_{t}^{n},p_{t}^{n},q_{t}^{n})\in\mathcal{A}\equiv\mathcal{I} \times\mathcal{P}\times\mathcal{Q}\).

#### Iii-B4 Reward

The objective of POMDP is to minimize the total amount of overflowed load and the event where the load is empty. Thus, the reward \(r(s_{t},\mathbf{a_{t}})\) is defined as follows,

\[r(s_{t},\mathbf{a_{t}})=\sum_{n=1}^{N}(u_{t}^{q,t}+w_{d}\cdot u_{t}^{d,n}+w_{ b}\cdot u_{t}^{b,n})+w_{W}\cdot\sum_{m=1}^{M}u_{t}^{W,m}, \tag{10}\]

where \(w_{d}\), \(w_{b}\), and \(w_{W}\) stand for reward coefficients for time delay, and load-balancing of AMR agents and sites, respectively.

### _Quantum Multi-Agent Actor-Critic Network Design_

#### Iii-C1 State Encoding Circuit

The state encoding circuit is leveraged for feedforwarding a state input. Fig. 2 presents the two schemes of state encoder. Fig. 2(a)/(b) need a single gate or two gates per qubit, respectively. Despite the encoding system showing the best performance when the number of qubits is equal to the number of input variables, the number of input variables in RL (i.e., state) must be larger than the number of qubits [26]. Thus, this paper considers two state encoders under the consideration of the environment, as follows,

\[U_{\textit{enc}}^{a}(z) =\Big{[}\otimes_{k=1}^{K}\left(R_{Y}(x_{k}^{z})\right)\Big{]} \cdot|0\rangle^{\otimes_{\textbf{dec}}}, \tag{11}\] \[U_{\textit{enc}}^{c}(s) =\Big{[}\otimes_{k^{\prime}=1}^{K^{\prime}}\left(R_{Y}(x_{2k^{ \prime}}^{s})\cdot R_{X}(x_{2k^{\prime}-1}^{s})\right)\Big{]}\cdot|0\rangle^{ \otimes_{\textbf{dec}}}, \tag{12}\]

where \(x_{k}^{z}\) and \(x_{k^{\prime}}^{s}\) stand for \(k\)-th entry of observation \(z\) and \(k^{\prime}\)-th entry of state \(s\), respectively. Note that \(U_{\textit{enc}}^{a}(z)\) and \(U_{\textit{enc}}^{c}(s)\) denote an actor observation encoder and critic state encoder. The actor observation encoder and critic state encoder work in the \(K\) and \(K^{\prime}\) qubits system, respectively.

#### Iii-C2 Parameterized Circuit and Quantum Measurement

A parameterized circuit is a quantum circuit that performs numerical tasks such as estimation, optimization, approximation, and classification using learnable parameters. As shown in Fig. 3(a), The VQC block consists of rotating gates with different directions and _Controlled-Z_ gate, i.e., \(R_{X}\), \(R_{Y}\), \(R_{Z}\), and \(\textit{CZ}=\begin{bmatrix}I&0\\ 0&Z\end{bmatrix}\). Note that _CZ_ is used to entangle qubits. To improve the circuit's performance, this paper configures the parameterized circuit with multi-VQC blocks, which requires additional trainable parameters \(\theta\) as shown in Fig. 3. To obtain the desirable outputs, the measurement \(\mathcal{M}\) is leveraged, which calculates the expected value of superpositioned quantum states based on its computational basis. In summary, the observable (i.e., expected value) is written as follows:

\[\langle O\rangle_{x,\theta}=\prod_{M\in\mathcal{M}}\langle 0|U_{\textit{enc}}^{ \dagger}(x)U_{\textit{VQC}}^{\dagger}(\theta)MU_{\textit{VQC}}(\theta)U_{ \textit{enc}}(x)|0\rangle, \tag{13}\]

where \(\langle O\rangle_{x,\theta}\) is the output of VQC with inputs \(x\) and circuit parameter \(\theta\); \(\mathcal{M}\) is the set of quantum measurement bases in VQC with \(|\mathcal{M}|\leq n_{qubit}\).

Fig. 3: The illustration of the parameterized circuit.

Fig. 2: The illustration of the state encoder.
#### Iii-C3 Implementation on Quantum Actor-Critic

The proposed QMARL for a smart factory in this paper is decentralized for scalability. Every AMR agent in the QMARL has a VQC-based policy, i.e., agents do not require communication among agents. The observables of the actor/critic are as follows,

\[\langle O_{a}\rangle_{o,\theta} =\!\Big{\{}\!\langle 0|U_{\textit{mc}}^{a\dagger}(o)U_{\textit{VQC}}^{a \dagger}(\theta)MU_{\textit{VQC}}^{a}(\theta)U_{\textit{mc}}^{a}(o)|0\rangle \!\Big{\}}_{M\in\mathcal{M}_{a}}, \tag{14}\] \[\langle O_{c}\rangle_{s,\phi} =\!\Big{\{}\!\langle 0|U_{\textit{mc}}^{c\dagger}(s)U_{\textit{VQC}}^{c \dagger}(\phi)MU_{\textit{VQC}}^{c}(\phi)U_{\textit{mc}}^{c}(s)|0\rangle\! \Big{\}}_{M\in\mathcal{M}_{c}}. \tag{15}\]

**Quantum Actor.** For the quantum actor, the observable of (14) is used to calculate the probabilities of actions of each AMR agent. Then, the quantum policy is written via a softmax function of its observable,

\[\pi_{\theta}(a_{t}|z_{t})=\textit{softmax}(\beta_{a}\langle O_{a}\rangle_{z_{t }^{n},\theta}), \tag{16}\]

where

\[\textit{softmax}(\textbf{x})\triangleq\Bigg{[}\frac{e^{x_{1}}}{\sum_{i=1}^{ N}e^{x_{i}}};\cdots;\frac{e^{x_{N}}}{\sum_{i=1}^{N}e^{x_{i}}}\Bigg{]} \tag{17}\]

and \(\beta_{a}\) is the scaling factor for an actor observable, respectively. At the time \(t\), the actor policy of \(n\)-th agent makes an action-decision with the given observation \(o_{t}^{n}\), which is denoted as \(\pi_{\theta}(a_{t}^{n}|o_{t}^{n})\). Note that \(\theta\) denotes parameters of \(n\)-th actor. Then, the action \(a_{t}^{n}\) is computed as follows,

\[a_{t}^{n}=\arg\max_{a}\pi_{\theta}(a|o_{t}^{n}), \tag{18}\]

and note all agents use the same policy by parameter sharing.

**Quantum Centralized Critic.** The centralized critic is adopted for CTDE as a state-value function. At time \(t\), the parameterized critic estimates the discounted returns given \(a_{t}\) as follows:

\[V_{\phi}(s_{t})=\beta_{c}\langle O_{c}\rangle_{s_{t},\phi}\!\simeq\!\mathbb{E} [\!\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}\!\cdot\!r(s_{t^{\prime}}\!,\! \textbf{u}_{t^{\prime}})|s_{t}\!=\!s], \tag{19}\]

where \(\gamma\), \(T\), \(\textbf{a}_{t}\), \(\beta_{c}\), and \(r(s_{t^{\prime}},\textbf{a}_{t^{\prime}})\) stand for a discounted factor \(\gamma\in[0,1)\), an episode length, the actions of all agents, scaling factor for a critic observable and reward functions that the state \(s_{t^{\prime}}\) and action \(\textbf{a}_{t}^{\prime}\) are given, respectively. In addition, \(\phi\) presents the trainable parameters of a critic. Here, \(s_{t}\) is the ground truth state at \(t\).

### _Training Algorithm_

The objective of MARL agents is to maximize discounted returns. To derive the gradients for the maximization objective, we leverage the joint state-value function \(V_{\phi}\). To train \(V_{\phi}\), this paper leverages a multi-agent policy gradient (MAPG), which is formulated as follows,

\[\nabla_{\theta}\mathcal{L}_{\textit{actor}} =-\mathbb{E}_{\textbf{a}\sim\pi_{\theta}}\Bigg{[}\sum_{t=1}^{T} \sum_{n=1}^{N}y_{t}\nabla_{\theta}\!\log\pi_{\theta}(a_{t}^{n}|z_{t}^{n}) \Bigg{]}, \tag{20}\] \[\nabla_{\phi}\mathcal{L}_{\textit{critic}} =\nabla_{\phi}\!\sum_{t=1}^{T}\left\|y_{t}\right\|^{2}, \tag{21}\]

subject to

\[y_{t}=r(s_{t},\textbf{a}_{t})+\gamma V_{\phi^{\mathrm{T}}}(s_{t+1})-V_{\phi}( s_{t}), \tag{22}\]

where \(\phi^{\mathrm{T}}\) is the parameters of target critic network. Note that (20) and (21) are for following the parameter-shift rule [43], written as follows:

\[\frac{\partial\mathcal{L}_{\textit{actor}}}{\partial\theta_{i}} =\frac{\partial\mathcal{L}_{\textit{actor}}}{\partial\pi_{\theta}} \frac{\partial\pi_{\theta}}{\partial\langle O\rangle_{\theta,\theta}}\Big{[} \langle O\rangle_{o,\theta\frac{1}{2}\textbf{e}_{i}}\!-\!\langle O\rangle_{o, \theta\frac{-1}{2}\textbf{e}_{i}}\Big{]}, \tag{23}\] \[\frac{\partial\mathcal{L}_{\textit{critic}}}{\partial\phi_{j}} =\frac{\partial\mathcal{L}_{\textit{critic}}}{\partial V_{\phi}} \frac{\partial V_{\phi}}{\partial\langle O\rangle_{s,\phi}}\Big{[}\langle O \rangle_{s,\phi\frac{1}{2}\textbf{e}_{j}}\!-\!\langle O\rangle_{o,\phi-\frac {1}{2}\textbf{e}_{i}}\Big{]}, \tag{24}\]

where \(\textbf{e}_{i}\) and \(\textbf{e}_{j}\) stand the \(i\)- and \(j\)-th standard bases of parameterized vectors \(\theta\) and \(\phi\), respectively. Note that the two left partial derivatives are derived by classical computing, and the last term is obtained by quantum computing. The detailed training procedure is presented in **Algorithm 1**.

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Schemes** & **Computing method** & **\# of parameters** \\ \hline Proposed & Quantum & \(\approx 110\) \\ Comp1 & Quantum/Classical & \(\approx 110\) \\ Comp2 & Classical & \(\approx 110\) \\ Comp3 & Classical & \(\approx 40\)K \\ Comp4 & Random Walk & None \\ \hline \hline \end{tabular}
\end{table} TABLE II: The benchmark schemes.
## V Performance Evaluation

### _Experimental Setup_

To verify the effectiveness of the proposed QMARL framework for smart factory management (named, Proposed), the proposed QMARL-based algorithm is compared with four comparing methods as listed in Table II. The purpose of these numerical experiments is as follows,

* The comparative experiments of Proposed, Comp1, and Comp2 are conducted to corroborate the quantum advantages. The number of parameters is equally set for a fair comparison.
* This paper compares Proposed and Comp3 to verify that the proposed method can achieve better performance than the latest MARL technique.
* To verify the superiority of MARL, this paper compares MARL schemes to random walk schemes, i.e., Comp4.
* To investigate the robustness of quality control, we train benchmark schemes in various environments regarding precision. We validate the fact that the quality of the load is time-varying in the environment. We corroborate the robustness of the quality control in our proposed scheme.

The simulation parameter settings are listed in Table III. Because the number of qubits used in this paper is lower than \(110\), this paper assumes that quantum noise is negligible. Comp1 is a hybrid quantum classical method utilizing A2C critic structure which is proposed and developed in another work [45]. Moreover, Comp2 and Comp3 are based on CTDE structure. Specifically, the value decomposition network (VDN) [46]. For a fair comparison, we compose the neural network of linear operations and activation functions (i.e., linear or dense layer). The python software libraries (torchquantum and pytorch) are used for deploying VQCs and DL methods, which support GPU acceleration [16]. In addition, all experiments are conducted on a multi-GPU platform (equipped with 2 NVIDIA Titan XP GPUs using a 1405 MHz main clock and 12 GB memory) for training and inferencing/testing.

### _Performance of Training_

Fig. 4 presents the numerical results corresponding to the training metric. This paper adopts total reward, precision,

\begin{table}
\begin{tabular}{l|r} \hline \hline
**Parameters** & **Values** \\ \hline The number of sites (\(M\)) & \(2\) \\ The number of AMRs (\(N\)) & \(6\) \\ The load capacity of warehouse & \(2,000\,\)kg \\ The load capacity of AMR agent & \(500\,\)kg \\ Observation dimension & \(6\) \\ Precision (Reported [39]) & \(\{61.9,95.8,97.1\}\%\) \\ Weight of TFT-LCD (Reported [44]) & \(6\,\)kg \\ Action dimension & \(5\) \\ State dimension & \(8\) \\ Episode length & \(30\,\)timestep \\ Reward coefficient \((w_{d},w_{b},w_{W})\) & \((0.1,1,10)\) \\ Time delay by quality engineers (\(\tau_{qual}\)) & \(3\,\)timestep \\ \hline Actor observable hyperparameter \(\beta_{a}\) & \(3\) \\ Critic observable hyperparameter \(\beta_{c}\) & \(35\) \\ Optimizer & \(\text{Adam optimizer}\) \\ The number of gates in \(U_{p}^{a}\) and \(U_{p}^{c}\) & \(54\) \\ The number of qubits of actor & \(8\) \\ The number of qubits of critic & \(8\) \\ Learning rate of actor & \(1\times 10^{-2}\) \\ Learning rate of critic & \(1\times 10^{-3}\) \\ Weight decay & \(1\times 10^{-5}\) \\ \hline \hline \end{tabular}
\end{table} TABLE III: The experiment parameters.

Fig. 4: The experimental result of various metrics with comparing different MARL frameworks.
processing time, and loaded/overflowed/underflowed amount in AMR/server as training metrics. As shown in Fig. 4(a), all training benchmark schemes (_i.e.,_ Proposed, Comp1, Comp2 and Comp3) converge to the expected value for each different total reward. Proposed, which utilizes VQCs for both actor and critic network configuration, can observe the increased total reward from the beginning of learning to \(980\) epochs. Then, the total reward of Proposed achieves the final value of \(-37\). Comp1 and Comp2, which share a common state-value network composed of a small number of parameters, do not evaluate their values properly and cause the reward to exist between \(-205\) and \(-240\). This is lower than \(-200\), which is the expected value of the total reward when a random walk is performed. However, a classical actor and critic composed of a large number of parameters \((\approx 40\)K) show similar performance to the proposed scheme (_e.g.,_\(\pm 10\) performance difference in the total reward).

In Proposed and Comp3, policy evaluation and improvement are trained to increase the total reward. However, Comp1 and Comp2 with classical critic networks composed of small parameters are trained (i.e., actor loss and critic loss are reduced), but not in the direction of reward increasing. In other words, policy evaluation and improvement are not working correctly. The only difference between Proposed and Comp1 is whether the critic is a quantum-based or a classical critic, and there is a huge difference in training performance. In addition, compared with Comp3, the number of parameters is \(364\)x lower than that of Comp3, whereas the performances are almost equivalent to each other.

### _Feasibility Studies in the LCD Smart Factory Environment_

This section investigates the proposed model's performance in LCD smart factory environment. Fig. 4 shows the results of various metrics in the training process, and Table IV shows the performance after training is finished. The result of Table IV represents the average value of inference of \(100\) iterations. Fig. 4(b-c) represent the average quality and processing time, respectively. Fig. 4(d-i) show the amount of loaded, overflowed, and underflowed load amount generated in warehouse and AMR, respectively. For this simulation, the total amount of overflowed/underflowed loads achieve target performance if the corresponding values become 0. During the training process, it is shown that the values of the two metrics (i.e., the amount of overflowed/underflowed loads) of the Proposed and Comp3 are reduced. Therefore, it can be inferred that Proposed and Comp3 are trained in the correct direction. On the other hand, overflowed loads sparsely occurred in Comp1 and Comp2. However, the underflowed load amount is the highest, which proves that Comp1 and Comp2 are learning in the direction that satisfies only one of the two goals. Furthermore, this tendency also affects the average load status of the warehouse and AMR agents. In the proposed scheme of this paper, it is confirmed that all four values of the indicators continuously decrease until they reach the approximate value of 0. Hence, it is confirmed that the AMR agent has the average load of 3.6 kg in Fig. 4.

### _Impact on State Encoding Method_

According to [31, 47], state-encoding is crucial for the performance of QRL. Therefore, an experiment is designed to demonstrate the importance of state encoding. This experiment aims to transform four random bits into continuous scalar values. The output value is calculated as \(y=\sum_{i=1}^{4}x_{i}*2^{1-i}\). In this transformation process, \(\{4,2,1\}\) variables dense encoding is carried out to compare the dense encoding methods. Note that the number of parameters in VQCs is identical to 50. The result is shown in Fig. 6, and it is concluded that the performance of \(1,2\) variables dense encoding is high while the performance of \(4\) variables dense encoding is low. In other words, the \(2\)-variables encoding used in this paper has less performance degradation than the \(4\)-variables encoding technique.

### _Robustness of Quality Control_

We design the experiment to investigate the robustness of the proposed framework. To benchmark the robustness, we design the smart factory environment that is time-varying and configure the environment in four phases. In phase 1, the precision of load is randomly selected from \(\{61.9,95.8,97.1\}\%\), which is identical to the training environment. Note that the initial precision follows the uniform distribution \(\mathcal{U}[61.9,97.1]\%\). The quality of LCD load carried by each AMR varies with time (e.g., \(61.9\)%, \(95.8\)%, and \(97.1\)% for phase 2, phase 3, and phase 4, respectively). Then, the average precision is measured for 60 minutes to investigate the

\begin{table}
\begin{tabular}{c||c c c c c} \hline \hline \multicolumn{1}{c||}{**Metric**} & \multicolumn{5}{c}{**Benchmark Scheme**} \\ \multicolumn{1}{c||}{**[SI Unit of (a)-(f): kg]**} & Proposed & Comp1 & Comp2 & Comp3 & Comp4 \\ \hline (a) Avg. load status of AMR & 6.0 & 2.9 & 9.3 & 2.9 & 2.6 \\ (b) Avg. load status of server & 511 & 88 & 87 & 244 & 87 \\ (c) Avg. overflowed load in AMR & **81** & 101 & 224 & 131 & 103 \\ (d) Avg. overflowed load in server & **2.6** & 0 & 0 & 5.1 & 0 \\ (e) Avg. underflowed load in AMR & **77** & 106 & 227 & 136 & 100 \\ (f) Avg. underflowed load in server & **371** & 628 & 630 & 493 & 579 \\ \hline (g) Avg. precision of load [\%] & **92.1\%** & 90.8\% & 90.8\% & 92.2\% & 89.3\% \\ (h) Avg. processing time [Minute] & **292** & 294 & 255 & 371 & 253 \\ \hline \hline \end{tabular}
\end{table} TABLE IV: The numerical results of various metrics corresponding to the different benchmark schemes.
robustness of quality control. The result of Fig. 5 represents the average precision value of inference of \(100\) iterations. During phase 1, the precision records \(88.4\%\) on average. At \(t=30\), the quality of the input load decreases, i.e., the input load's precision equals \(61.9\%\). Thus, the precision is \(72.4\%\), the lowest precision during the episode. In response to this result, the AMR agents try to improve the precision value during phase 2. On the other hand, the AMR agents do not make actions on quality control in phases 3 and 4 since the quality of input load increases from \(61.9\%\) to \(97.1\%\). In summary, the robustness of quality control in our proposed scheme is corroborated by demonstrating the ability of our AMR agents to encounter and cope with the unpredictable quality of input load.

### _Discussion_

This section provides in-depth discussions to explain why the proposed scheme outperforms the other frameworks.

#### V-F1 Expressibility of Trainable Parameters

The authors of [48] have argued that the parameters of VQCs have more expressibility for quantum neural networks than classical neural networks. The small number of trainable parameters in the RL/MARL regime acts as a vulnerability for the classical neural network. In [30, 31], it is proven that QRL and QMARL can achieve similar performance to a classical RL/MARL. In the results of this paper, the classical neural network with fewer parameters yields lower performance due to two reasons; 1) index embedding on observation, and 2) parameter-shared policy. The index embedding on agents' observations and the parameter-shared policy method are utilized for faster convergence despite taking a loss in performance. Furthermore, the expressibility capacity of a neural network is also trusted to be sufficient, which is why the two methods introduced above are used [49]. Unfortunately, the degradation in performance is significant regardless of the expressibility capacity. On the other hand, the quantum circuit operates successfully even with a small number of parameters.

#### V-F2 Dimensional Reduction Corresponding to the State Encoding

Information loss occurs when the input variables are lost by dimensional reduction. In the experiments, the dimension of the input variable is set to four, and the output variable is set to four, two, and one for different schemes, respectively. In four variables dense encoding, the information loss is severe, because four independent variables are encoded using four rotation gates \(R_{Y}(x_{4})\), \(R_{Y}(x_{3})\), \(R_{Z}(x_{2})\), and \(R_{Z}(x_{1})\) through a single qubit. In the cases of two variables dense encoding and one variable dense encoding, the dimensional reduction does not occur. This is proven by showing the encoding processes on the Bloch sphere. For two and one variables encoding, the qubits are rotated twice in two orthogonal directions (e.g., \(y\)-axis and \(z\)-axis directions) and once in one direction, respectively. Consequently, the ranks of the resultant qubits are guaranteed. Therefore, the four variables dense encoding method has the lowest performance and is outperformed by the other aforementioned methods.

## VI Concluding Remarks

This work has investigated the design of QMARL agents based on VQCs for autonomous multi-robot control and coordination in smart factory management while taking POMDP into consideration. When utilizing AMRs as QMARL agents, the two variables dense encoding method is implemented to reduce the number of qubits in the proposed model. In addition, this paper adopts the parameter-shared policy with index embedding, which can reduce the number of trainable parameters. Using the abovementioned techniques, the quantum policy and state-value function are configured to quantum multi-agent actor-critic. The extensive numerical results show the superiority of the proposed QMARL-based AMR control in smart factory management. Finally, the proposed QMARL has an explicit performance gain when using the same number of parameters compared to the classical MARL algorithm and does not suffer from a severe dimensional reduction of data compared to other state-encoding methods.
