**2. Related Works**

In this section, we review the literature related to mobile crowdsensing problem, DRL approaches for multi-agent systems, and the joint studies of these two topics.

Threat to validity [28,29]: For this review, we have used multiple strings to search and identify relevant literature in recent decade, such as 'UAV swarm and mobile crowdsensing', 'multi-task allocation and mobile crowdsensing' and 'multi-agent deep reinforcement learning'. Google Scholar is used for forward searches and most of the related works are retrieved from five databases: IEEE Xplore, SpringerLink, Web of Science, ScienceDirect and Arxiv.

#### *2.1. Multi-Task Allocation for Mobile Crowdsensing*

MCS scenarios usually have multiple constraints and objectives. One of the key issues is how to perform task allocation, or how to choose appropriate action strategies for different tasks. The main tasks of SAG-MCS are data collection by covering PoIs and energy management by keeping batteries charged. UAVs need to automatically select action strategies to meet the data collection requirements under the energy-efficiency constraint. Solving such multi-agent task allocation is an NP-hard problem, and the related research is still in a relatively early stage. Feng et al. [30] utilized dynamic programming for path planning in UAV-aided MCS and used Gale-Shapley-based matching algorithm to allocate different tasks for agents. Wang et al. [31] modeled multi-task allocation as a dynamic matching problem, then proposed a multiple-waitlist based task assignment (MWTA) algorithm. In addition, several surveys of task allocation have demonstrated the effectiveness of heuristic algorithms. Hayat et al. [13] proposed a genetic algorithm approach to get the minimum task completion time for UAV path planning. Similarly, Xu et al. [32] formulated this problem as a specific mathematical model, and tried to minimize incentive cost under the constraint of sensing quality based on greedy algorithms and genetic algorithms.

#### *2.2. Deep Reinforcement Learning (DRL) for Multi-Agent Systems*

In multi-agent systems, Reinforcement Learning (RL) generally targets at problems of agents sequentially interacting with local environment. At timestep *t*, the environment is at state *st* and agent *i* obtains a observation *oi t*. Then, agent *i* selects and executes an action *ai t* based on *oi t*, and then gets a reward *ri t* from the environment. In POMDP, agents cannot directly perceive the underlying states and *oi t* is not equal to *st*. The objective of RL is to learn a policy *πi*(*ai* | *oi*) for agent *i*. The policy is expected to maximize the discounted reward E[*Rt*] = E ∑∞ *k*=0 *γkri k* , with a discounted factor *γ* ∈ [0, 1].

Currently, DRL methods have achieved state-of-the-art performance in various RL tasks [24,25], and can be categorized into value-based or policy-based ones. In this paper, we adopted the value-based method. Deep Q-learning (DQN [24]) is one of the most vital value-based DRL approaches. Based on Q-learning, DQN uses deep neural networks to learn a Q-value function *Q*(*o*, *a*), which could estimate the expected reward return E[*Rt*] and be recursively updated. DQN regards the action with biggest Q-value as the most optimal policy *π* (*s*) = arg max *Qπ*(*o*, *a*), and selects it to interact with the environment. In addition, DQN integrates fixed target network and experience replay methods to make the training process more efficient and stable [33]. Specifically, the Q-value function *Q*(*o*, *a*) is updated through minimizing the Q-loss function as:

$$Q\_{loss} = \left(r\_t + \max\_{a\_{t+1}} Q'(o\_{t+1}, a\_{t+1}) - Q(o\_t, a\_t)\right)^2. \tag{1}$$

where *Q* is the learned network and *Q* is the target network. Note that the policies learned by DQN are deterministic, therefore DQN should be trained with action policies such as − *greedy* to enhance exploration.

Compared with classical heuristic algorithms, agents can learn a strategy more efficiently and independently through DRL algorithms, so as to achieve multiple objectives in the sensing area simultaneously.

#### *2.3. DRL Methods for UAV Mobile Crowdsensing*

To date, several studies have investigated the application of DRL algorithms in the UAV Mobile Base Station (MBS) scenario, which is a sub-topic of MCS. In the UAV MBS scenario, a swarm of UAV serve as mobile base stations to provide long-term communication services for ground users. Liu et al. [15] proposed a DRL model based on Deep Deterministic Policy Gradient (DDPG [34]) to provide the long-term communications coverage in the MBS scenario. Further, Liu et al. [16] implemented DDPG in a fully distributed manner.

Different from policy gradient methods, Dai et al. [35] applied Graph Convolutional Reinforcement Learning (DGN [36]) in MBS. They modeled the UAV swarm as a graph, and used Graph Attention Network (GAT [26]) as a convolution kernel to extract adjacent information between neighboring UAVs. To further explore the potential of graph networks, Ye et al. [37] designed a FANET based on GAT, named GAT-FANET, allowing two adjacent UAV agents within the communication range to communicate and exchange information at low costs. This work also applied Gated Recurrent Unit (GRU) as a memory unit to record and process long-term temporal information from the graph network.

On the basis of MBS, Liu et al. [38,39] took practical factors such as obstacles and charging stations into consideration in the UAV MCS scenario. Based on the actor-critic network of DDPG, their DRL models used CNN to extract observed spatial information, and deployed a distributed experience replay buffer to store previous training information. Piao et al. [40], Dai et al. [41] and Liu et al. [38] utilized the concept of the Long Shortterm Memory (LSTM [42]) network to store sequential temporal information of previous interaction episodes. As a specific application of MCS, Dai et al. [41] designed an approach for mobile crowdsensing, where mobile agents are required to retrieve data and refresh the sensors distributed in the city, with limited storage capacities of the sensors. Wang et al. [43] proposed a more practical and challenging 3D MCS scene for disaster response simulation, where the UAVs' action space had been expanded to three dimensions.

Compared with the UAV MBS and MCS works mentioned above, this paper proposes a more complicated and promising SAG-MCS scenario, which incorporates global and local observations from space and air, respectively, and encourages UAVs to interact with charging stations as ground nodes. While [38–41] proposed multi-UAV MCS scenarios and used policy-based DRL methods as solutions which utilized LSTM to store temporal information of MCS systems, our approach selects the value-based method based on DQN and uses GRU as the memory unit, which performs similarly to LSTM but is more computationally efficient [44]. Furthermore, when most MADRL studies about MCS solved the problem with deterministic policies, our method learns a stochastic policy following Ye et al. [37] to improve robustness.

#### **3. System Model and Problem Statement**

In this section, we design a partially observable space-air-ground integrated MCS system, with space-based remote sensing satellites and an aerial UAV swarm jointly performing the MCS task. We define the problem and present the 2D simulation system model specifically. Then, we describe the design of evaluation metrics.

#### *3.1. System Model*

As illustrated in Figure 1, the SAG-MCS scenario is simplified to a 2-dimensional continuous square area with the size of *L* × *L* pixels. The simulation area has fixed borders and multiple obstacles that UAVs cannot fly over. We assume that there are a set K - {*k* | *k* = 1, 2, . . . , *K*} of PoIs, and each PoI is assigned a certain data amount *d*(*k*), ∀*k*. Note that PoIs are regarded as persistent information nodes and are not going to disappear after coverage. Additionally, we consider a set C - {*c* | *c* = 1, 2, . . . , *C*} of charging stations and a set B - {*b* | *b* = 1, 2, . . . , *B*} of round and rectangular obstacles. At the beginning of each simulation episode, the locations of all the PoIs, charging stations, and obstacles are randomly distributed in the 2D map. Each PoI's data amount *d*(*k*), ∀*k* is randomly assigned in a certain range as well, but the total data volume Σ*kd*(*k*) of different episodes remains consistent.

Let U - {*u* | *u* = 1, 2, . . . , *U*} be *U* UAV agents deployed in the simulation area, where the UAVs can perform continuous and horizontal flying movements at a fixed altitude. We define *Robs* as the observation range, and *Rcov* as the coverage range or sensing range of each UAV. Arbitrary UAV can observe the local map within the radius *Robs* in real-time and receive *Lsat* × *Lsat* fuzzy global map captured by satellites every some timesteps. Any PoI *k* within a UAV's *Rcov* is recognized as covered and all its data *d*(*k*) is collected once at each timestep *t*. Note that *Rcov* is smaller than *Robs*, as UAVs can only collect data when approaching to PoIs, but they can observe a wider range of area in general. Moreover, we consider the UAV swarm can autonomously form the ad-hoc network, and each pair of agents can be interconnected within communication range *Rcomm* and exchange observed information for joint decision making. Considering the delays and packet losses in realworld ad-hoc networking, we set a communication dropout probability *p* between adjacent UAV nodes in training and evaluation. As for the energy consumption, we set the onboard battery status *φ*(*u*) ∈ [0, 100%], ∀*u*.

For each simulation episode, the data collection task in SAG-MCS scenario will last for *T* timesteps in total. Each UAV's position is randomly assigned and their batteries are fully-charged in the beginning. At each timestep *t*, UAV *u* can obtain local observation from embedded sensors; while every few timesteps, it can obtain fuzzy global observation from the satellite. Using the multi-scale observations {**o***u t* }*u*∈U , UAV *u* performs an action {**a***u t* }*u*∈U . We set the battery *φ*(*u*) consumed at timestep *t* as {*eu t* }*u*∈U , which is determined by the current flying speed {*vu t* }*u*∈U and will be introduced in Section 3.4. When flying close to charging stations, their batteries will be fully charged in next timestep, simulating the real-world battery replacement process on the ground.

#### *3.2. Observation Space*

In SAG-MCS, each UAV agent *u* can obtain the multi-scale observation {**o***u t* }*u*∈U at timestep *t* from different sources, as introduced in Section 3.1. In Figure 2, we formulate the observation space with three elements: O - {**o***u t* = (O*u local* , O*u global* , O*u sel f*)}∀*u*∈U .

**Figure 2.** The observation space of UAV *u* in SAG-MCS.

(1) Local observation O*local* from embedded sensors: UAV can observe local information within a circle of radius *Robs* in real-time, centering on itself. Let O*local* - {**o***u l* = (O*u*,1 *local*, O*u*,2 *local*, O*u*,3 *local*)}∀*u*∈U denotes local observation space, which consists of three 2D vector channels. The first channel contains the data amounts and distribution of surrounding PoIs. We set the data value *d*(*k*) as the corresponding pixel value if it refers to PoI *k*, otherwise 0. The second channel contains the locations of obstacles relative to the UAV, where we set pixel value 1 for coordinates of obstacles, otherwise 0. The third channel includes the locations of other UAVs within *Robs*. In addition, we define pixel value 1 for coordinates of UAV agents as well, otherwise 0.

(2) Global observation O*u global* from satellites: Every *n* timesteps, satellites will capture fuzzy global observation and transmit the information to all UAVs. As shown in Figure 2, O*u global* consists of three 2D channels with reduced size of *Lsat* × *Lsat*(*Lsat* < *L*), which cannot provide precise locations of the environment elements globally. We define O*global* - {**o***u g* = (O*u*,1 *global*, O*u*,2 *global*, O*u*,3 *global*)}∀*u*∈U in absolute positioning coordinates. The encoding method for global observation is nearly the same as local observation, except in the third channel of UAV locations, we set −1 as the corresponding pixel value if it refers to the absolute location of UAV *u* in global map.

(3) Auxiliary observation O*u sel f* : Then we utilize information from onboard flight control computer to assist UAV to learn optimal policy. Specifically, we define O*sel f* - **o***u s* = concatenate *x*(*u*), *y*(*u*), *vx*(*u*), *vy*(*u*), *φ*(*u*), {Δ*x*(*c*), Δ*y*(*c*)}∀*c*∈C ∀*u*∈U . For UAV *u*, **o***u s* includes its absolute position, velocity and current remaining battery, and the relative locations of all charging stations towards UAV *u*.

#### *3.3. Action Space*

The rotor UAVs are capable of applying different *thrust* at all directions responsively. We choose to discretize the entire 2-dimensional continuous space into eight directions for simplicity, and UAV agents can apply *maximum-thrust* (denoted as 1.0 unit), *half-thrust* (0.5 unit), or *zero-thrust* (0 unit) at any direction. Note that zero-thrust represents hovering in place. Therefore, the action space in SAG-MCS is defined as:

$$\mathcal{A} \triangleq \left\{ \mathbf{a}\_t^u = (\theta\_t^u, f\_t^u) \mid \theta\_t^u \in \{\frac{k\pi}{4} \mid k = 0, 1, \dots, 7\}, f\_t^u \in \{0, 0.5, 1.0\} \right\}.\tag{2}$$

where *θu t* denotes the thrust angle and *f u t* is the thrust magnitude. The action space A consists of 17 actions in total. Since the timestep interval in the simulation is quite short, we assume the physical model is a uniform acceleration process. UAV can adjust the magnitude and direction of velocity using certain actions.


**Meet the editor**

The editor Dr. Paul Bracken is a Professor currently a Professor in the Mathematics Department at the University of Texas in Edinburg, TX. He obtained his BSc degree from the University of Toronto and PhD degree from the University of Waterloo in Canada. His research interests include mathematical problems in quantum mechanics and quantum field theory, differential geom-

etry and partial differential equations as well as gravity. He has published over one hundred papers in journals and books and as well presented about twenty talks at various meetings and conferences and have taught courses at all levels over the years.

Contents

**Preface IX**

**Section 1 The Classical-Quantum Correspondence 1**

**Between Conjugate Variables 3**

Gabino Torres-Vega

**Kepler Problem 23**

Francisco De Zela

**Section 2 The Schrödinger Equation 79**

**Decoherence 81**

Kazuyuki Fujii

Shimada

Sanchez

Chapter 1 **Classical and Quantum Conjugate Dynamics – The Interplay**

Keita Sumiya, Hisakazu Uchiyama, Kazuhiro Kubo and Tokuzo

**Possible Bridge Between Classical and Quantum Physics 41**

**Nonlinearity, Dynamical Scalar Product and some Ideas of**

Chapter 3 **Charathéodory's "Royal Road" to the Calculus of Variations: A**

Chapter 4 **The Improvement of the Heisenberg Uncertainty Principle 67** L. M. Arévalo Aguilar, C. P. García Quijas and Carlos Robledo-

Chapter 5 **Schrödinger Equation as a Hamiltonian System, Essential**

Chapter 6 **Schrödinger Equation and (Future) Quantum Physics 105** Miloš V. Lokajíček, Vojtěch Kundrát and Jiří Procházka

Jan J. Sławianowski and Vasyl Kovalchuk

Chapter 7 **Quantum Damped Harmonic Oscillator 133**

Chapter 2 **Classical and Quantum Correspondence in Anisotropic**


**6. Conclusion**

*4.2.6. Multimodal analgesia*

46 Pain Management in Special Circumstances

incidence and severity of CPSP.

severity of CPSP [122, 123].

*4.2.8. Glucocorticoids*

outcomes [82].

**5. Future directions**

*4.2.7. Non-steroidal anti-inflammatory drugs (NSAIDs)*

Multimodal analgesia has become the widely accepted modality of treatment for perioperative pain. It is utilising different regimens of different classes of medications according to the type of the surgical procedure and the institute where surgery is performed [115]. The main aim of multimodal analgesia is to target several peripheral and CNS mechanisms to maximise pain reduction, reduce opioids requirements and to decrease opioid-related side effects [103, 107]. A few studies have explored the effects of multimodal analgesia on CPSP prevention. The evidence indicated positive effects at 3 months [116] and at 1 year following surgery [117]. Additional studies are required to study the effects of multimodal analgesia on different types of surgical procedures and to find out whether its preventive effects do indeed reduce the

Prostaglandins are one of the inflammatory mediators activated during surgery which has a possible role in CPSP pathophysiology. NSAIDs are a group of medications which are widely used for their anti-inflammatory properties. They reduce the pain and inflammation through the inhibition of the synthesis of prostaglandins by inhibiting COX-1 and COX-2 receptors. NSAIDs can reduce secondary hyperalgesia and central sensitization [118, 119]. One study showed that Celecoxib (COX-2 inhibitor) had reduced post-operative pain, the need for post-operative opioid analgesia [120] and meanwhile did not inhibit bone healing following arthroplasty surgery [121]. In summary, the available clinical trials are heterogeneous and differ in the following: the type of drug used, follow-up time point and pain outcomes. None of these trials demonstrated a significant impact of NSAIDs on reduction in the incidence or

Glucocorticoids prevent pain by expressing anti-inflammatory properties and by preventing central sensitization [124]. Three trials studied the effects of perioperative corticosteroid on CPSP. The studies used different types of steroids Dexamethasone, Methylprednisolone and Hydrocortisone. A Cochrane review included these clinical trials. The results were inconclusive, and the heterogeneity precluded any possible meta-analysis. The heterogeneity was due to variations in drugs used, follow-up time intervals and the measured pain

CPSP remains as a challenging clinical problem. There are several areas which are promising and can be explored further in the future. Investigators could focus more on the better CPSP is a complex process which is not fully understood. When it occurs, it affects the patients' quality of life. Based on our current understanding of the pathophysiology, nerve injury and inflammation are the two main responsible mechanisms for the development of CPSP. Specific risk factors, which make some individuals at a higher risk than others for CPSP, have been identified. Timely identification of these individuals based on their risk profile allows us to develop appropriate interventions. Several modalities and interventions for CPSP prevention have been investigated. These include pre-emptive and perioperative interventions (**Table 1**). CPSP prevention is an important area for future research in view of the methodological problems with the majority of available studies.

Adapted with permission from reference [125].

**Table 1.** Evidence-based strategies to reduce the risk of chronic post-surgical pain (CPSP).


*2.4. The Concept of the Triple Bottom Line in Relation to SDGs*

A contribution to the growing literature on the measurement of infrastructure projects on sustainability is provided by Ding and Shen [30], who focus on the balance needed between benefits to society whilst protecting the environment and still achieving the economic benefits envisaged in the project business case. The linkage across the three areas in the construction industry is further defined by Kibert [25], who suggests that the interrelationship between a project's outputs and the society that is impacted is a central component of defining the sustainability success of an infrastructure project. This introduces the concept that project success definition needs to consider success against the triple bottom line (TBL) [7–9] of social, environmental (or ecological) and economic (or financial) effects, otherwise noted as the "three pillars" concept of "people, profit and the planet" [7–9]. However, the overemphasis on the last of the TBL criteria, namely finance, brings us to the root of the problem of measuring projects' SDG impact [16,17].

This is because the crux of the project reporting problem lies with the dominance of accounting tools, which have been the preeminent business method of reporting business success for over 500 years since Luca Paccioli first published his papers on double entry bookkeeping [31]. It has largely remained unchanged until the past 10 years. As evidence of this widening to cover the three pillars of TBL, there has been a proliferation of mechanisms and economic models to track different elements of TBL, for example, environmental, social and governance (ESG) [7], which introduces these three core areas into the business investments decisions that measure the ethical and sustainability impacts of a company. The contention of this current research study is that the proliferation of project success measurement theories, tools and concepts, which are mostly finance-driven, causes confusion and often leads to suboptimal action [32] and that a TBL perspective needs to be integrated from the start of any business case development (see later section on business cases). Considering the aforementioned literature, it is possible to synthesise the fourth proposition related to SDGs in the context of the concept of the triple bottom line in relation to SDGs as follows.

**Proposition 4 (P4).** *Measurement of SDG performance should accommodate the perspective of the triple bottom line (i.e., social, environmental and economic performance).*


**Denoising E**ff**ect of Jason-1 Altimeter Waveforms with Singular Spectrum Analysis: A Case Study of Modelling Mean Sea Surface Height over South China Sea**

#### **Jiajia Yuan, Jinyun Guo \*, Yupeng Niu, Chengcheng Zhu, Zhen Li and Xin Liu \***

College of Geodesy and Geomatics, Shandong University of Science and Technology, Qingdao 266590, China<EMAIL_ADDRESS>(J.Y<EMAIL_ADDRESS>(Y.N<EMAIL_ADDRESS>(C.Z<EMAIL_ADDRESS>(Z.L.)

**\*** Correspondence<EMAIL_ADDRESS>(J.G<EMAIL_ADDRESS>(X.L.)

Received: 11 May 2020; Accepted: 8 June 2020; Published: 10 June 2020

**Abstract:** Altimeter waveforms are usually contaminated due to nonmarine surfaces or inhomogeneous sea state conditions. The present work aimed to present how the singular spectrum analysis (SSA) can be used to reduce the noise level in Jason-1 altimeter waveforms to obtain SSA-denoised waveforms, improving the accuracy of a mean sea surface height (MSSH) model. Comparing the retracked sea surface heights (SSHs) by a 50% threshold retracker for the SSA-denoised waveforms with those for the raw waveforms, the results indicated that SSA allowed a noise reduction on Jason-1 waveforms, improving the accuracy of retracked SSHs. The MSSH model (called Model 1) over the South China Sea with a grid of 2 × 2 was established from the retracked SSHs of Jason-1 by the 50% threshold retracker for the SSA-denoised waveforms. Comparing Model 1 and Model 2 (established from the retracked SSHs by the 50% threshold retracker for the raw waveforms) with the CLS15 and DTU18 models in the South China Sea, it was found that the accuracy of Model 1 was higher than that of Model 2, which indicates that using SSA to reduce noise level in Jason-1 waveforms can effectively improve the accuracy of the MSSH model.

**Keywords:** singular spectrum analysis; altimeter waveform; Json-1; mean sea surface height; threshold retracker; waveform retracking