Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Architectures and Synchronization Techniques for Distributed Satellite Systems: A Survey
Liz Martinez Marrero1, Student Member, IEEE,
Juan C. Merlano Duncan2, Senior Member, IEEE,
Jorge Querol3, Member, IEEE,
Sumit Kumar4, Member, IEEE,
Jevgenij  Krivochiza5, Student Member, IEEE,
Shree Krishna Sharma6, Senior Member, IEEE,
Symeon Chatzinotas7, Senior Member, IEEE,
Adriano Camps8,9, Fellow Member, IEEE,
and Björn Ottersten10, Fellow Member, IEEE
[1-7,10]Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, 1855, Luxembourg City, Luxembourg
[8]Universitat Politècnica de Catalunya, Barcelona, Spain
[9]Institut d'Estudis Espacials de Catalunya, Barcelona, Spain

This research was funded in whole by the Luxembourg National Research Fund (FNR), under the CORE project COHESAT: Cognitive Cohesive Networks of Distributed Units for Active and Passive Space Applications, grant reference [FNR11689919]. For the purpose of open access, the author has applied a Creative Commons Attribution 4.0 International (CCBY4.0) license to any Author Accepted Manuscript version arising from this submission.

Marrero : Architectures and Synchronization Techniques for Distributed Satellite Systems: A Survey
Marrero : Architectures and Synchronization Techniques for Distributed Satellite Systems: A Survey

Corresponding author: Liz Martinez Marrero (e-mail: liz.martinez-marrero@ uni.lu).

CDSS is a key enabling technology for the future of remote sensing and communication missions. However, they have to meet strict synchronization requirements before their use is generalized. When clock or local oscillator signals are generated locally at each of the distributed nodes, achieving exact synchronization in absolute phase, frequency, and time is a complex problem. In addition, satellite systems have significant resource constraints, especially for small satellites, which are envisioned to be part of the future CDSS. Thus, the development of precise, robust, and resource-efficient synchronization techniques is essential for the advancement of future CDSS. In this context, this survey aims to summarize and categorize the most relevant results on synchronization techniques for DSS. First, some important architecture and system concepts are defined. Then, the synchronization methods reported in the literature are reviewed and categorized. This article also provides an extensive list of applications and examples of synchronization techniques for DSS in addition to the most significant advances in other operations closely related to synchronization, such as inter-satellite ranging and relative position. The survey also provides a discussion on emerging data-driven synchronization techniques based on ML. Finally, a compilation of current research activities and potential research topics is proposed, identifying problems and open challenges that can be useful for researchers in the field.

synchronization, distributed satellite systems, distributed beamforming, remote sensing, satellite communications.

§ INTRODUCTION

Distributed Satellite Systems is a very promising architecture for future remote sensing and communication missions [1][2]. The term DSS refers to satellite systems with two or more spacecraft communicating to accomplish the mission goal. Nowadays, most missions are DSS, in part influenced by the space industry's paradigm shift toward smaller and cheaper satellites [2]. Its potential increases with the use of signal coherent processing methods from different platforms, especially coherent transmission and reception, which implies that the system behaves as a single unit with collocated transceivers, also known as DBF. This entails the improvement of several DSS applications such as Earth observation, geolocation, navigation, imaging, communications, among others. However, it requires strict time, frequency, and phase synchronization among the distributed nodes.

For example, deep space communications could be accomplished by DSS applying distributed MIMO techniques [3] if the terminals at each end of the link are synchronized at symbol level. This implies that the stations need to maintain synchronized clocks with sub-nanosecond accuracy to guarantee a bandwidth of a few hundred MHz [4]. Another example can be Earth observation applications based on distributed SAR interferometry. In this case, the final resolution depends on the accuracy of the phase synchronization achieved between the satellites [5]. For some applications, the synchronization requirement is the critical limiting factor in making practical implementations possible [6]. In those cases, the level of effort, power consumption, and complexity required for the synchronization algorithm may be higher than those needed for the mission itself.

Achieving exact phase, frequency, and time synchronization is a complex problem when the clock reference signals are generated locally at each distributed node. It is significantly more challenging when the distance between the distributed nodes is much larger than the signal wavelength, and especially when this distance varies over time due to the relative movement of the nodes, as it typically happens in DSS [7]. Furthermore, the electrical distance, determined by the inter-satellite channels and the RF chains at each spacecraft, is not constant either. In these cases, it is not possible to rely on network backhaul links or to use the exact location of the collaborating nodes as can be done in some terrestrial networks [8][9].

The synchronization of distributed wireless systems is highly challenging and has received plenty of attention in the literature. For example, [10] summarizes some of the timing and carrier synchronization techniques proposed for wireless communication systems. Another survey on the synchronization in wireless systems is given in [11], where the authors summarize the clock synchronization protocols in WSN. However, both articles are limited to terrestrial systems, and they do not address synchronization for DSS.

On the other hand, the opportunities and challenges of DSS have been explored in many publications. For example, [12] summarizes the use of WSN for planetary exploration. In this article, the authors mentioned synchronization as one of the problems encountered in distributed systems design. Besides, [13] provided a comprehensive assessment of modern concepts and technologies of DSS and analyzed the technical barriers to DSS implementation. In addition, [14] stated the revolutionary strength of DSS such as satellite swarms and the limitations in its implementation due to synchronization requirements.

However, the synchronization of DSS is still an open and challenging research question that has attracted the attention of the scientific community in the last few years. This has resulted in a large body of work appearing in conferences and journals from different fields. To the best of authors' knowledge, a comprehensive survey document to cover topics on synchronization applied to DSS is still missing. This article intends to fill this gap. With this idea in mind, this paper summarizes and categorizes the most relevant publications on synchronization techniques for DSS. The result is a comprehensive survey that can be used as a guide for researchers and developers working in this field.

§.§ Contributions of This Paper

This survey describes the most relevant results on synchronization for DSS and the strict requirements and new technologies related to distributed satellite missions. For the sake of clarity, the main contributions of this paper can be summarized as:

* A brief survey of the DSS architectures is provided, classifying them into five general groups: Constellations, Clusters, Swarms, Fractionated and Federated spacecraft, which main features are identified.

* The distributed time, phase, and frequency wireless synchronization methods reported in the literature are summarized and compared, analyzing their feasibility for DSS.

* Other operations, closely related to synchronization in DSS such as inter-satellite ranging and relative positioning are also analyzed.

* It is offered an extensive compilation of the missions and PoC implementations reported up to the present.

* Some of the most relevant current research activities and potential research topics are presented, identifying problems and open challenges.

§.§ Paper Organization

The paper is organized as follows: After the Introduction, some important concepts on the architectures and system models for DSS are defined in Section <ref>. The synchronization methods reported in the literature are related and categorized in Section <ref>. Section <ref> deals with other operations closely related to the synchronization in DSS such as: inter-satellite ranging and relative position. Sections <ref> and <ref> comment examples of synchronization methods in DSS, whereas Section <ref> lists some ideas and new methods that have not been implemented yet. Section <ref> provides to the reader examples of PoC and launched missions of DSS with special emphasis on the synchronization methods implemented. Section <ref> presents the use of ML for synchronization purposes. The most relevant open questions and future research directions are highlighted in Section <ref>. Finally, the reader can find the general conclusions in Section <ref>. The list of acronyms is provided in Section <ref> to make the reading more accessible.

§.§ Acronyms

§ ARCHITECTURES AND SYSTEM MODEL FOR DISTRIBUTED SATELLITE SYSTEMS

§.§ Overview of Existing Architectures

DSS can be classified into five general groups: Constellations, Clusters, Swarms, Fractionated, and Federated spacecraft. Table <ref> summarizes the main characteristics of these groups.

* Constellations refer to a traditional approach where tens of medium to large satellites (over 500 kg each) are distributed around Earth to guarantee global or regional coverage of a service. Some of the most famous satellite constellations are: the GNSS constellations, such as GPS and Galileo and; the satellite communication constellations, such as Globalstar [15], Iridium [16], and OneWeb [17]. Inter-satellite communication in these networks is scarce or null, except in the Iridium constellation where each satellite can have four Ka-band ISL [16].

* A cluster is a group of at least two mini or micro satellites (from 10 to 500 kg each) deliberately positioned closely together to enhance or create new system capabilities. These DSS cover a smaller portion of the Earth and mainly require inter-satellite communications to keep a close flying formation. Some satellite clusters are GRACE [18], LISA [19], PRISMA [20], PROBA-3 [21], and TanDEM-X [22].

* Satellite swarms are similar to clusters, except they contain a significantly higher number of satellites, often smaller and less expensive (less than 10 kg each). They are envisioned to contain hundreds and even thousands of nanosatellites operating together in a loose flying formation. They will require inter-satellite communications, as each member determines and controls relative positions to the others. Unlike previous DSS that have several examples, satellite swarms are a new concept yet to be demonstrated. Examples of satellite swarms projects are QB-50 [23] and OLFAR [24].

* Fractionated spacecraft is a revolutionary satellite architecture that distributes the functions of a single large satellite into numerous modules that communicate by ISL in a highly dynamic topology [25]. Among the multiple advantages of this concept can be mentioned the flexibility,  robustness, and the significant decrease in the required time to launch and deploy a satellite mission. Fractionated satellite systems is a very recent concept that have not yet been implemented.

* Finally, the Federated Satellite Systems (FSS) paradigm visualizes opportunistic collaboration among fully independent and heterogeneous spacecraft [26]. This is one of the most recent DSS concepts and is inspired by the current peer-to-peer networks and cloud computing. The main idea is to benefit from the potential of under-utilized space commodities by trading and sharing previously unused resources available in space assets at any given time. It is worth noting that much of FSS's potential relies on the spacecraft capabilities to establish communications through ISL. The recently launched FSSCat mission [27] is an example of FSS.

Main characteristics of the DSS groups

Classification     Number of Satellites     Satellites Weight     Inter-satellite Comm     Examples

Constellation     tens     $> 500$ kg       scarce or null        GNSS, Globalstar, Iridium
Cluster     at least 2 satellites     10 to 500 kg       c]@c@required to keep a
close flying formation     c]@c@GRACE, LISA, PRISMA,
PROBA-3, TanDEM-X
Swarm     c]@c@hundreds in a loose
flying formation      $< 10$ kg       c]@c@required to determine and
control relative position     QB-50, OLFAR
spacecraft     at least 2 satellites     $< 500$ kg       c]@c@required to achieve
the mission objectives     not implemented yet
FSS     dynamic     any     c]@c@required to share
unused resources     FSSCat

In addition to this classification, CDSS can be categorized according to their synchronization scheme as centralized or distributed. The former refers to distributed systems where all the nodes adjust their carriers to follow one controller node, which has the most stable oscillator in the system. This synchronization scheme has a relatively simple implementation, but it may have robustness drawbacks. On the other hand, distributed synchronization satellite systems do not rely on a single node but try to find a common carrier considering all the oscillators in the system. This characteristic overcomes the robustness drawback of the former group but makes the synchronization algorithms more complex. The synchronization methods suitable for CDSS will be addressed in detail in the following sections.

Moreover, considering the communication links between the nodes of the DSS, the space segment can be classified as:

* Ring, in which each node connects to two other nodes, forming a single continuous path through all the elements of the DSS. This topology is not suitable for centralized synchronization algorithms.

* Star, in which each node connects to a central node that performs specific tasks ranging from communications with the ground segment to leading the synchronization. This topology generally uses centralized synchronization methods.

* Mesh, in which each node is connected to every other node in the distributed system. This topology can be implemented fully or partially depending on the complexity of the DSS, and it accepts the implementation of both distributed and centralized synchronization algorithms.

* Hybrid topologies combine two or more of the previous ones.

These topologies are shown in Fig. <ref>, where the arrows represent the ISL.

Classification of DSS considering the ISL

§.§ Generalized System Model

The general DSS considered in this article is a distributed array of autonomous nodes which collaborate to perform distributed beamforming towards an intended target node outside the array. From a general perspective, the nodes of the distributed arrays considered in this survey can be moving while the whole system tries to stay at a fixed position, or the entire array can be following a trajectory or orbit.

For both cases, the nodes require transmission and reception capabilities to synchronize the distributed system. For the general DSS, no specific geometric distribution of the nodes is assumed, none of the previously mentioned classes neither. However, each of them can be described by a state variable $\overrightarrow{x_n}  = (\overrightarrow{p_n},\overrightarrow{v_n})$ where $\overrightarrow{p_n}$ and $\overrightarrow{v_n} \triangleq \dv{\overrightarrow{p_n}}{t}$ represent position and speed of the node $n$, respectively.

Besides, it is assumed the existence of RF-ISL between all the elements of the DSS, which implies a Mesh configuration, even if all the links are not activated at the same time. The complex transfer function of these ISLs is formally represented as complex coefficients $h_{nm}$, where $n$ and $m$ are the node subscripts. The matrix $\mathbf{H}$, containing all the $h_{nm}$, can be used to describe the DSS. It determines the most suitable synchronization procedure. Fig. <ref> shows the general DSS considered in the next sections to analyze the synchronization techniques.

General scheme of the DSS performing the required synchronization for a DBF transmission

The quality of the ISL is considered variable as a function of the position and orientation of the nodes, given that omnidirectional antennas are not assumed. Besides, the frequency responses of the RF chains are considered different for each node.

For synchronization purposes, each of the distributed nodes generates its own initial reference signal, which stability depends on the available hardware at each satellite. The objective of the synchronization algorithm is to make all of those initial reference signals converge to a common reference with the best possible accuracy.

§.§ Summary and Lessons Learnt

* DSS can be classified as Constellations, Clusters, Satellite swarms, Fractionated spacecraft, and, FSS.

* Distributed synchronization algorithms are more robust than centralized synchronization algorithms, but more complex.

* According to the ISL, there are four types of DSS: Ring, Star, Mesh, or Hybrid topology.

* The general DSS model considered in this article is a distributed array of autonomous spacecraft which collaborate to perform distributed beamforming towards an intended target node outside the array.

* The general model assumes a Mesh configuration with the reference signals locally generated at each distributed satellite.

§ OVERVIEW OF SYNCHRONIZATION METHODS

A critical aspect of the synchronization of distributed radio systems, in general, is the clock or time synchronization in addition to the phase synchronization. This section summarizes the most significant synchronization methods reported in the state of art. These algorithms can be classified as Closed-loop or Open-loop methods based on the use of feedback from a node external to the DSS. The external node can be another satellite, an anchor point, or the intended communication target. The Closed-loop methods require a communication channel to transmit the feedback information between the external node and the DSS. Whereas in Open-loop, the synchronization is achieved without the participation of any node other than the distributed satellites.

Another way to classify the synchronization methods considers the communication between the elements of the DSS. In order to achieve synchronization, some algorithms require the exchange of information among the distributed satellites. This can be done as a two-way message exchange or closed loops, which requires a duplex channel between the nodes of the DSS, or as a broadcast or one-way communication. Another option is to synchronize without any communication among the elements of the DSS. In this case, it is possible to achieve coherence using the feedback from a node out of the DSS. Both classifications can be superimposed, as it is represented in Fig. <ref>. In this figure, some of the missions analyzed in the following sections were included as examples.

Classification of the synchronization methods

§.§ Time Synchronization

Time synchronization is critical in the application of many DSS. The coherent transmission in communication applications, as an instance, requires aligned signals at the symbol level to achieve the full potential of the beamforming gains. The tight clock and timing synchronization is achieved with different levels of accuracy in other areas, such as wired and optical networks and wireless sensor networks. For example, some distributed applications such as computer networking, distributed signal processing, instrumentation, and earth observation applications require accurate timing or clock synchronization. The purpose of this section is to explain those timing synchronization methods used in other areas and give advice on how to translate them into DSS, in particular for communications applications.

Previous studies have identified the time or clock synchronization challenge for clock frequencies in the order of tens to thousands of MHz and accuracies ranging between orders of one fractional digit relative to the clock frequency. Recently, it has been studied for multi-static and MIMO radar and distributed beamforming applications where the required precision is orders of magnitude more stringent [28]. Additionally, these systems require what is known as absolute time synchronization, which is different from relative time synchronization on which the timing of an impingent signal is tracked. The problem of absolute time synchronization was first formally defined by Poincaré and Einstein in 1898 [29], and 1905 [30] respectively. The formal definition of this problem and the TWTT concept was provided in the framework of relativistic-event-simultaneity.

Fig. <ref> depicts the basic idea of the TWTT concept. Here, an initiating or source node sends a signal (or packet) at time $T_1$. A slave (or follower node) receives the signal at time $T_2$ after a delay of $\Delta t_1 = T_2 - T_1$ and responds (or reflects) after a known delay at time $T_3$. The source node receives the response signal at time $T_4$. The time offset of the clocks is then $((T_2-T_1)-(T_4-T_3))/2 = (\Delta t_1- \Delta t_2)/2$, and the propagation delay is $((T_2-T_1)+(T_4-T_3))/2 = (\Delta t_1+\Delta t_2)/2$.  Therefore, proper knowledge of this propagation time offset will be used to achieve absolute synchronization.

Operating principle of the TWTT clock synchronization [28]

The TWTT concept created the foundations for clock synchronization of twenty-century networks and complex systems, such as satellites and the Internet [31]. This general concept can be applied to diverse kinds of systems and networks, including DSS. This section will emphasize the case of DSS. The following subsections describe timing synchronization methods that have been developed in different communications areas and could be extrapolated to DSS.
An example of the use of TWTT in DSS is presented in [32], where the authors compared the performance of three clocks offset prediction algorithms based on this method for a master-slave architecture. In addition, the authors in [7] used TWTT to synchronize four spacecraft in a distributed satellite formation flying, achieving time synchronization simulation errors smaller than $\pm$ 10 ns. The work in [33] analyzed the effect of the motion of the satellite on the two-way time synchronization accuracy. Another example based on TWTT calculates the time difference between two satellite clocks by measuring the phase difference of a pseudo-random noise code in a master-slave architecture [34]. The method is proposed to achieve on-orbit synchronization in a micro-satellite cluster [34]. Other authors combined RF carrier ranging methods with TWTT to obtain inter-satellite range and time synchronization simultaneously [35][36][37].

§.§.§ Time synchronization in wired networks

The TWTT concept is the basis of most synchronizations protocols in the literature. The NTP and PTP are extensively used in large-scale modern computer networks, and both operate with a TWTT approach. The network nodes exchange timestamps employing (UDP) packets to measure the round-trip propagation latency. NTP generates software timestamps with non-deterministic time offsets and achieves clock accuracies in the order of $\sim 10~\mu$s [38]. The PTP (IEEE-1588) is an evolution of the NTP, which generates hardware timestamps using the waveforms of the associated clock. This protocol achieves clock alignment by exchanging synchronization packets among the involved nodes. The IEEE-1588 standard limits the maximum rate for the timestamp counters to 125 MHz, providing accuracies of $\sim 10$ ns [39][40]. The performance of these systems, such as NTP and PTP, rely directly on the clock rates used, and the achieved accuracy is given by the granularity associated with an integer counter.

Further refinements of the time synchronization mechanism use the clock phase information. A salient example is the White Rabbit project of the European Organization for Nuclear Research (CERN). White Rabbit operates with Ethernet frames to detect the phase difference between the local clock and a clock extracted from the received Ethernet signal. Across large wired networks, White Rabbit operates under specially designed network switches to measure and compensate fractional clock phase differences achieving $\sim 10$ ps accuracy [41].

Even though these time synchronization mechanisms can not be used directly on DSS, the main concepts can be extrapolated to DSS scenarios. In the DSS, the coarse synchronization methods such as NTP and PTP can be used to eliminate the ambiguity in the total channel delay. Then, the time synchronization can be refined with differential time mechanisms at the waveform level, using White Rabbit or similar approaches.

§.§.§ Time synchronization in wireless sensor networks

In wireless communications, absolute time synchronization is also frequently desired, and the wireless channel is typically used to exchange the synchronization messages. The algorithms for absolute time synchronization in wireless networks can be classified as sender-to-receiver and receiver-to-receiver synchronization methods. The former is based on the TWTT between couples of nodes, whereas receiver-to-receiver methods use time readings of a standard signal broadcasted to a set of nodes from a common sender [42].

Some receiver-to-receiver solutions include RBS [43] and PBS [44]. The RBS protocol implements a TOA exchange between the distributed nodes disregarding the signal TOF over the physical medium [43]. PBS is a well-known timing synchronization scheme for WSN, which is based on sets of node-pairs for network-wide synchronization. PBS operates under the assumption that all the participating nodes will receive and detect the pairwise synchronization frames exchanged between two master/reference nodes. This approach assumes a hierarchically distributed structure [44] and assumes that distances between nodes and their associated delays (TOF) are identified in advance.

The most well-known sender-to-receiver synchronization methods are TPSN [45] and FTSP [46]. TPSN implements TWTT between pairs of nodes preceded by a discovery phase from where each node obtains a level. In FTSP and its variations [47], the distributed nodes synchronize to a signal broadcasted from the root node or a previously synchronized node.

For these protocols to work, synchronization must be performed several times. Additionally, the nature of the WSN, in which the network observes a physical phenomenon (temperature, pressure, etc.), determines synchronization requirements in the order of microseconds. However, this level of accuracy is inadequate to perform distributed coherent (beamforming) radio applications.

For example, the required time synchronization to achieve beamforming maintaining the performance at acceptable levels is around $\pm$ 7.5 percent of a symbol duration. For single-carrier communication baud rates of a hundred MHz, this represents a required accuracy of $\pm$ 0.75 ns. This accuracy could be achieved with a refinement of the methods mentioned above, such as the work in [48]. This article proposes a step forward into the timing accuracy increase by using frequency-modulated continuous-wave (FMCW) signals with relatively high bandwidth of 150-MHz. The method performs the synchronization between two stations using a TWTT approach (similar to the one proposed in [49]). It uses the aforementioned radar-like waveform to provide a joint carrier-phase and timing synchronization with an accuracy of 66 ps.

§.§.§ Ultrawideband Pulse Synchronization

Wireless synchronization approaches using UWB pulses instead of exchanges of a network packet have recently caught researchers' interest. To estimate the TOA at sub-nanosecond levels, UWB approaches take advantage of high-speed hardware, generally at sampling rates higher than 1 GHz.

Several applications have exploited UWB signaling using high speeds clocks and ADC. Some examples are the sets of multiple active receivers locked, and synchronous to a single transmitter [50], distributed consensus techniques [51], and distributed sensor positioning [52]. The works in [53] propose a propagation-aware TOF protocol and provide validation for the system using an atomic clock integrated on a chip and a 64 GHz hardware clock timestamp counter [54]. As a result, these experiments achieved a distributed timing accuracy of 5 ns between two sensors. It is essential to mention that the transmission of UWB pulses is not feasible for small satellites such as CubeSats due to power constraints. However, its advantages can be considered for DSS with less strict power consumption requirements.

Nevertheless, it is worth pointing out that increasing clock frequencies is not the only alternative to increase the timing accuracy in synchronization mechanisms. A common misconception in the literature regarding UWB synchronization systems is that the ADC frequency bounds the time resolution. As specified by the CRLB  [55], accurate TOA measurements within minuscule fractions of a sampling period is attainable, specially in LOS scenarios.  The measurement of these time offsets, fractional to the sampling time, can be achieved by time offset mechanisms, also known as timing-error-detectors, such as the Gardner method [56], and the Early-Late-Gate method [57] among others.

§.§ Frequency and Phase Synchronization

As previously stated, considering the use of feedback from an external node, frequency, and phase synchronization methods can be classified as:

* Closed-loop methods, where the feedback from the target nodes could be either a single bit or a few bits or could have the form of rich feedback with limited or full CSI [58].

* Open-loop methods that require either intra-node communication or blind beamforming, also known as 0F  [58].

In the following sections, some of the most notable examples from each category are discussed.

§.§.§ Closed-Loop Synchronization Methods

Fig. <ref> lists the closed-loop synchronization methods found in the literature. They are classified into different groups according to the feedback type for a better understanding, although more detailed information is provided in this section.

[edge from parent fork down, sibling distance=40mm, level distance=15mm]

[root] Closed-loop Methods

child node[level 2] (c1) Iterative Bit Feedback
child node[level 2] (c2) Rich Feedback;

[every node/.style=level 3]
[below = 0.25cm of c1, xshift=15pt] (c11) One/Two-bit feedback [59, 60];
[below = 0.25cm of c11] (c12) Cluster based [61, 62];
[below = 0.25cm of c12] (c13) Adjustable perturbation size [63, 64, 65];
[below = 0.25cm of c13] (c14) Deterministic perturbation [60, 66];

[below = 0.25cm of c2, xshift=15pt] (c21) Explicit Channel Feedback [67, 68, 69];
[below = 0.25cm of c21] (c22) Aggregate Rich Feedback [70, 71, 72, 73, 74];
[below = 0.25cm of c22] (c23) Reciprocity-based [75, 76, 77];

in 1,2,3,4
[->] (c1.180) |- (c1.west);
in 1,2,3
[->] (c2.180) |- (c2.west);

Classification of the Closed-loop synchronization methods

The group Iterative Bit Feedback includes the algorithms where the distributed nodes modify their signals according to one or more decision bits received from the target node. Among them, the most well-known algorithm is the classical 1BF, proposed in [59]. This method considers the beamforming nodes synchronized in time and frequency and achieves phase synchronization by applying independent random phase rotations in each beamforming node. At each time slot, transmitter nodes add a random perturbation to their signals. The target node measures the RSS and sends one bit indicating if the RSS is better than the previous value. Depending on this bit, the transmitters keep or update the phase rotation. The process is repeated during the next time slot until each node's phase has been adjusted to its optimal value. The primary constraint of this algorithm is its convergence time, which was improved in [60] using 2BF.

Other approaches improve convergence time by reducing the number of collaborative nodes. For example, [62] proposed to separate the distributed nodes in clusters and perform the phase synchronization in two stages. The outcome of this algorithm with respect to 1BF is more evident as the number of nodes increases, when, for $N \approx 100$ nodes, it can decrease to half the required iterations. However, comparing both techniques only by the number of iterations is not fair since the epoch in [62] is more complex and therefore takes a longer convergence time. Another iterative bit feedback method for phase synchronization named OCB was proposed in [61]. In this case, the destination node selects a subset of the distributed nodes whose transmitted signals produce the higher coherence gain at the destination.

Further enhancement of the 1BF is proposed in [64] by emulating the bacterial foraging techniques. In this algorithm which is called the BioRARSA, the “swim” mechanism, along with the step size adjustment, enables the beamforming nodes to decrease convergence time as well as improve robustness against variation of initial conditions. A comprehensive explanation of this method and its performance can b found in [63]. Another synchronization method that increases the convergence speed by adjusting the perturbation size was presented in [65], where the authors proposed to exploit the cumulative positive feedback information additionally.

The methods discussed above are based on random disturbances of the transmitted signal phase. However, D1BF [60], and its improved version, SDDB [66] proposed to limit the possible phase perturbations to a set of discrete values from where choose the one that allows achieving the maximum RSS possible. The number of elements in the perturbation set is proportional to the convergence time and determines the performance in terms of maximum achieved RSS. Even though SDDB shows steeper growth rates than D1BF, both deterministic algorithms have limited performance due to the digitization of the perturbation set. Some combinations of deterministic and random methods were considered in [60], where the hybrid methods obtained allow prioritizing the convergence time or the beamforming performance depending on the specific requirements of the network.

Iterative Bit Feedback algorithms are unsuited for DSS due to their slow convergence characteristic. Generally, the long distance between the DSS and the receiver implies a delay in the communication that, combined with the slow convergence of these methods, makes it not suitable to synchronize the system by Iterative Feedback algorithms.

The rich feedback methods use more information instead of just a few feedback bits to achieve synchronization. They can be classified in three categories according to the way the distributed nodes obtain the channel estimation:

* Explicit Channel Feedback method, where each distributed node transmits a known sequence of training symbols to estimate the channel response.

* Aggregate Rich Feedback methods, where the transmitters simultaneously send uncorrelated training sequences used to estimate each channel gain.

* Reciprocity-based methods, where the transmitters observe the uplink feedback signals sent by the target nodes and use reciprocity to estimate their downlink channel gains automatically.

These algorithms have proven to be more robust than the Iterative Bit Feedback methods at the price of considerable feedback overhead. For instance, the E1BF [68], which is an Explicit Channel Feedback algorithm, compensates for the effect of the average time-varying channel by combining explicit phase information with the 1BF method to provide better convergence and scalability as compared to 1BF. A comprehensive description of this algorithm can be found in [68]. Another Explicit Channel Feedback algorithm that outperforms 1BF for convergence time is the PA for DTB presented in [67], which is also more energetically efficient.

Generally, nullforming requires far tighter synchronization than beamforming. For that reason, it mainly uses rich feedback approaches. For example, in [69], a distributed gradient-descent algorithm is used to modify the signals' power and phase in consecutive time slots achieving beamforming and nullforming simultaneously. The method works in a time-scheduled way to estimate the CSI and perform the collaborative transmission [69].

In general, Aggregate Rich Feedback methods are more scalable than Explicit ones. For example, the algorithm proposed in [73] is an Aggregate Rich Feedback method based in 1BF that allows each collaborating node to estimate its channel response to the receiver. Similarly, in the scheme proposed in [71], all the distributed nodes simultaneously transmit, and the receiver sends a phase compensation vector to achieve distributed transmit nullforming. However, in this method, the transmitter nodes use Code Division Multiple Access (CDMA) to facilitate signal separation at the receiver. The receiver uses an EKF to generate state estimations for each transmitter, which implies some scalability limitations for this scheme. An improved version of the previous algorithm was proposed in [72].

The Reciprocity-based methods were first introduced as F-RT synchronization method in [75]. In this work, a distributed network of two nodes, each of them equipped with two PLL, achieves frequency and phase synchronization by continuously transmitting three unmodulated beacons in a round-trip way. This strategy is effective in highly dynamic networks. However, in typical multipath channels, the frequency division duplexing intrinsic of F-RT generates non-reciprocal phase shifts, which reduce the performance. To overcome the problem, in [76] it is proposed a T-RT algorithm, which is equivalent to the F-RT but using the same frequency for all beacons. Frequency interference is avoided by time division duplexing. Even when this method gives the advantage of simultaneous frequency and phase synchronization, F-RT and T-RT schemes have as a drawback the extreme power consumption due to the extensive signaling.

All the works listed in this section aim that all carriers arrive with the same initial phase at the destination based on perfect time alignment. Nevertheless, Mañosa-Caballu and Seco-Granados proposed a R-RT synchronization protocol to achieve frequency, phase, and timing synchronization [77]. This algorithm is based on the T-RT method mentioned above. The robustness of the protocol allows it to work in dynamic environments in the sense that nodes can disappear without severely affecting the system's performance. Other approaches [10] addressed joint time delay and CFO synchronization in closed-loop systems. For example, [74] covers maximum likelihood synchronization in multi-node decode-and-forward cooperative relaying networks considering time-varying channels. In  [78], the same goal was approached through a weighted consensus algorithm to reach synchronization in a dense wireless network.

§.§.§ Open-loop Synchronization Methods

whereas rich feedback may give faster and better convergence than bit feedback, the signaling overhead is substantially more significant. The overhead implies latency problems in real-time applications, which can represent a considerable challenge for its implementation in cases such as satellite communications systems. In such scenarios where quick and reliable feedback from the target nodes is not possible, Open-loop synchronization methods are the recommended schemes.

[edge from parent fork down, sibling distance=40mm, level distance=15mm]

[root] Open-loop Methods

child node[level 2] (c1) Intra-node communication
child node[level 2] (c2) Blind;

[every node/.style=level 3]
[below = 0.25cm of c1, xshift=15pt] (c11) Master-slaves architectures [6, 79, 80];
[below = 0.25cm of c11] (c12) DCA [81];
[below = 0.25cm of c12] (c13) 2WS [82, 83, 84];

[below = 0.25cm of c2, xshift=15pt] (c21) 0F [85, 86];

in 1,2,3
[->] (c1.180) |- (c1.west);
in 1
[->] (c2.180) |- (c2.west);

Classification of the Open-loop synchronization methods

One of the simplest ways to do open-loop synchronization is through Master-slaves architectures, where one primary node broadcasts a beacon, and the secondary nodes lock their oscillators to this reference [6]. This algorithm can be considered as a closed-loop method if the primary node is not part of the DSS [70]. For instance, in [79] all nodes acquire their relative locations from the beacon of a nearby reference point that doesn't have to be the destination node. This allows open-loop synchronization, but each node requires knowledge of its relative position from a predetermined reference point within the cluster. In [80], frequency and phase synchronization is achieved by locking the secondaries' oscillators to the reference beacon sent by the controller node after pre-compensating the phase mismatch and the propagation delay.

A solution more robust to link and node failures than Master-slave architectures is the DCA proposed in [81], where each node broadcasts its carrier signal to all of its neighbors. Thus, the total received signal at any node is the superposition of its neighbors' carrier signals distorted by the channel. The goal of the DCA is to use this received signal to adjust the instantaneous frequency at each distributed node in such a way that all the distributed nodes eventually become frequency locked to a common carrier [81].

Another Open-loop algorithm based in the retrodirective principle is 2WS, proposed in [82]. The main idea of this method is to implement a retrodirective beamforming between the base stations in a mobile network without the need of CSI or feedback from the mobiles. To this end, a sinusoidal beacon is transmitted in a forward and backward propagation through all the base stations allowing them to compute a common carrier frequency and phase used to perform beamforming. The most significant drawback of retrodirective beamforming methods is that generally, the channel and the RF chains at each transceiver are not reciprocal. A relative calibration method to compensate for this drawback is proposed in [83]. In  [84], it is proposed a faster version of 2WS by exploiting the broadcast nature of the wireless links. The difference with [82] is that in the Fast Open-loop protocol, each base station only considers the signal from its adjacent two neighbors and ignores any signal from other base stations. In this way, the algorithm minimizes the latency by reducing the required non-overlapping time slots.

Finally, the blind method or 0F is a synchronization algorithm that intends to synchronize the network without any feedback from the target or the rest of the distributed nodes. According to [85], the offsets of the oscillator drift cause intermittent coherent addition of signals at the receiver, which can result in significant beamforming gains. Even though the complexity of the 0F algorithm is simple, the data rate that can be achieved with this method is very low as the coherent beamforming is not stable. In addition, the statistical analysis reported in [86] shows that 0F only works efficiently for a small network.

Summary of synchronization methods

Classification     Method     Sync target      Proposed application     Feasibility of its use in DSS
50.1Closed-loop: Iterative Bit Feedback
1BF [59] / 2BF [60]     Phase     Terrestrial wireless networks
OCB [61]     Phase     WSN
[62]     Phase     WSN     Not suitable for synchronization of DSS due to their slow
BioRARSA [64]     Phase     Wireless sensor/relay network    convergence.
D1BF [60] / SDDB [66]     Phase     WSN
320.1Closed-loop: Rich Feedback
TPSN [45] and FTSP [46, 47]      Time     WSN     The basic ideas of these methods can be applied in DSS, specially for large distributed networks. However, some refinement should be considered in order to achieve the accuracy needed for communication applications.
E1BF [68]     Frequency and phase     Wireless networks     It could be used in some scenarios, depending on the distance between the DSS and the receiver. However, the convergence time, 50% smaller in this method than the original 1BF, can still be an problem. An advantage of this algorithm is that only one element of the DSS processes the feedback from the receiver.
[69]     Phase     Wireless networks     Even, when the possibility to jointly achieve null and beamforming is a very desirable characteristic in satellite systems, the slow convergence of this method may make it unsuitable for synchronization of DSS
PA [67]     Frequency and phase     Wireless communication networks     It can be used in a DSS as long as a feedback channel is guarantee. This method presents scalability limitations.
[73]     Phase     Wireless networks     This method allows simultaneously null and beamforming, which a desirable characteristic in satellite system. However, some limitations could be that (1) most of the calculation are performed by the distributed nodes and, (2) it can produce some latency for large DSS.
[71][72]     Frequency and phase     Wireless networks     Allows beam and null-forming considering significant feedback latency, which implies that this method could be used for long baseline communications. However, it can present scalability problems.
F-RT [75] and T-RT [76]     Frequency and phase     Wireless communication networks     These methods can be used to synchronize DSS as long as a feedback channel is guarantee. Another constraint related
R-RT [77]     Phase, frequency and time     WSN     to these methods is that all the satellites in the distributed system have to receive the reference signal broadcasted by the destination node.
[74]     Phase, frequency and time     Cooperative decode-and-forward communication system     To implement this method in a DSS, all the spacecraft have to simultaneously receive a common training sequence broadcasted by the destination node. In addition, some latency constraints must be taken into account.
120.1Open-loop: Intra-node communication
RBS [43] and PBS [44]     Time     WSN     The basic ideas of these methods can be applied in DSS, specially for large distributed networks. However, some refinement should be considered in order to achieve the accuracy needed for communication applications.
TWTT [7, 31, 32, 33, 34, 35, 36, 37]     Time     Satellite communications     It is used in DSS
Master-Slave [79, 80]     Frequency and phase     WSN     Suitable for DSS with the constraint that it requires accurate inter-satellite ranging.
DCA [81]     Frequency     Wireless networks     This method can be used to synchronize DSS as long as all the distributed nodes are inside the range of the rest.
2WS [82, 83, 84]      Frequency and phase     Mobile networks     Despite some scalability problems, it can be used to synchronize DSS.
[78]     Phase, frequency and time     Dense and compact wireless networks     This method is suitable for DSS where each node transmission can be received by almost all the satellites in the distributed system. However, algorithms with a central coordinator node acting as a reference for the delay compensation can be a simpler solution for less-dense networks.
Open-loop: Blind synchronization     0F [85, 86]     Frequency and phase     Wireless networks     Not recommended for DSS, since it is limited to small networks, and it allows very low data rate.

§.§ Summary and Lessons Learnt

* The synchronization algorithms can be classified as Closed-loop or Open-loop methods based on the use of feedback from a node external to the DSS.

* Another classification considers the communication between the elements of the DSS. It can be Closed-loop when the exchange of information among the distributed satellites is done as a two-way message exchange or; Open-loop when it is done as a broadcast or one-way communication.

* The time synchronization of DSS is mainly based on the TWTT algorithm. However, recent publications refer to the use of pseudo-random noise code and other techniques to achieve time synchronization and inter-satellite ranging simultaneously.

* The time synchronization methods used in WSN do not guarantee the accuracy required by communications applications. However, they could be applied to DSS after some refinement.

* Some of the methods used for time synchronization in terrestrial distributed wireless networks can be applied to synchronize DSS.

* Among the Closed-loop synchronization algorithms that use the feedback from a node external to the DSS, the rich feedback methods are more suitable to implement in DSS. Specifically, the PA algorithm for DTB and the reciprocity-based methods are the most recommended.

* Most Open-loop synchronization algorithms without any external aid, but with intra-node communication in the form of two-way message exchange are suitable for the synchronization of DSS.

* Higher synchronization accuracy could be achieved by combining the use of intra-node communication in the form of two-way message exchange and the feedback from a node external to the DSS. However, this configuration is not feasible for all applications.

§ RANGING AND RELATIVE POSITIONING IN DISTRIBUTED SATELLITE SYSTEMS

There are other operations closely related to the synchronization in DSS, such as the inter-node ranging and relative position. For these operations, the requirements and accuracy of the coherent operation depend on the performance of the ranging and relative positioning algorithms. Several synchronization methods are based on these measurements [87][88][89]. For example, in [87], a dynamic model describes the stochastic kinematics and the clock evolution of each distributed node relative to the frame of the receiver of a DTB system. Other examples are the remote sensing DSS missions, such as GRACE and PROBA-3, where knowing the relative position among the distributed nodes is fundamental to combining their measurements. For that reason, this section compiles recent advances in inter-node ranging and relative positioning for DSS.

§.§ Inter-Satellite Ranging

For many DSS, inter-satellite relative range measurement is a requirement for cooperative tasks. Generally, autonomous inter-satellite measurements and communications can reduce the dependence on ground stations, signal transmission delays, and improve the resilience and maneuverability of the DSS. To this end, the distributed nodes must have the capability of inter-satellite ranging.

The methods to do inter-satellite ranging can be classified into two main groups represented in Fig. <ref>: those based on RF and those based on optical signals. Measurements made using radio signals are the most mature technology, but optical measurements can achieve better ranging performance. However, the higher directivity of the laser beam, in comparison with the RF antenna's patterns, can represent a limitation for specific applications. Besides, the sunshine can blind optical sensors.

[edge from parent fork down, sibling distance=35mm, level distance=15mm,every node/.style = shape=rectangle, rounded corners,
draw, align=center]
Inter-satellite Ranging Methods
child  node Optical Inter-satellite
child  node Ranging based on
RF Signals
child  node Carrier-ranging
child  node pseudo-random code ranging ;

Classification of the inter-satellite ranging methods

§.§.§ Optical Inter-Satellite Ranging

The basic principle of optical inter-satellite ranging measurement, also known as transponder laser interferometry, is represented in Fig. <ref> as explained in [90]. The frequency-stabilized and power-stabilized master laser provides a highly coherent light source to meet interference requirements between the local and receiver laser lights. First, the master's laser light travels through the inter-satellite space and arrives at the slave satellite, where the OPLL locks the phase of the slave laser to that of the weak light received. Then, the compensated slave laser light points and propagates back to the master satellite. The precision phasemeter measures the phase difference between the master laser and the received light to calculate the inter-satellite distance.

Basic principle of optical inter-satellite ranging measurement

The authors in [90] discussed the state of the art of inter-satellite laser interferometer technologies for spaceborne gravitational-wave detection. However, inter-satellite interferometric ranging has been previously implemented in missions such as GRACE follow-on, where the ranging performance of the former mission was improved by a factor of 10 by including an interferometric laser ranging system [91]. This improvement is due to the laser wavelength being 10,000 times shorter than the microwave wavelength.

As a variant to the methods mentioned before, which required two phase-synchronized lasers on both satellites, a single-laser design was proposed in [92]. This solution included a Mach-Zehnder interferometer on one of the satellites and a secondary satellite equipped with a retro-reflector. The work in [93] overcame the need for highly stable lasers and complex phase-locking solutions of the previous ones. Though, it is required to maintain the coherence properties of the laser light in the reference arm, which is very technologically challenging for distances more significant than 500 km, i.e., twice the expected inter-satellite distance. However, simulations results in [92] showed similar ranging resolution to the two phase-synchronized lasers method.

Recent publications proposed the use of a single optical ISL for ranging and communications simultaneously [94][95]. For instance, [95] addressed the development of the optical transceivers to transmit ranging and frequency synchronization information through a coherent optical link between the spacecraft of the Kepler constellation. The Kepler system is based on a constellation of 24 Galileo-like MEO and six LEO satellites carrying stabilized lasers as optical frequency references and equipped with terminals for two-way optical links. This provides a means for broadcasting synchronization, and navigation messages across the DSS without any communication with the ground segment [95]. A ranging precision of $300~\mu$m and $10^{-15}$ s/s stability  (Allan deviation at 1 s) is achieved through an OPLL locked to a PRN sequence with a chip rate of 25.6 GChip/s. An additional data channel at 50 Mbit/s is multiplexed with the ranging signal to transmit timestamps in a known repetition rate. This allows time alignment of both satellites with the resolution of the FPGA' clock rate used at both ends.
Furthermore, the data link can be used for communication purposes by the constellation. Each MEO satellite has four bidirectional transceivers, two for connecting to neighboring satellites in the same orbit, and two for connecting to LEO satellites. The terminal aperture size is constrained to 75 mm due to the satellite's weight and size considered. However, to calculate the absolute ranging between terminals with an accuracy of 0.3 mm, the system requires aligning precision of $2.5\%$ the chip length or better [95].

Another example of simultaneous ranging and communications inter-satellite optical link is the coherent optical receiver implemented in [94]. In this article, the authors presented an FPGA-based feedback-homodyne scheme as an alternative to the OPLL to obtain a more flexible coherent optical receiver. Besides, a parallel FFT wavelength drift estimation algorithm was proposed, aiming to improve the speed and range of wavelength drift tracking simultaneously. Simulation results showed that the wavelength drift tracking performance depends on the number of FFT estimators used in parallel. However, using a real-time FPGA implementation, the authors demonstrated that the design meets the needs of phase offset compensation when three FFT estimators are used in parallel [94].

§.§.§ Inter-Satellite Ranging Based on RF Signals

The use of RF for inter-satellite ranging is not a new concept. Articles such as [96], published in 1985, already proposed a design to measure satellite-to-satellite range-rate with a precision smaller than $1~\mu$m/s in distances between 100 km and 300 km. However, further analyses included the frequency instability of the oscillators [97], the channel noise conditions [98], and the requirements of low-cost small satellites [99] among other specifications. Distances can be determined from either the signal's modulation (PRN codes) or the carrier phase.

One of the most popular carrier-phase based ranging methods is the DOWR [7][97][98]. By combining the one-way range measurements from two microwave-ranging devices, the method minimizes the effect of oscillator phase noise. Each satellite uses identical transmission and reception subsystems to send a carrier signal to the other. The recorded measurements are transmitted to a control segment for processing and calculating the inter-satellite range. DOWR method is represented in Fig. <ref>.

Basic principle of RF carrier-ranging by DOWR.

Some variations of this method have been proposed. For example, in [7] the authors addressed the use of DOWT synchronization and ranging (DOWT&DOWR) in DSS. Simulation results indicated less than 0.2 m and 0.5 ns precise ranging and time synchronization accuracy and error time synchronization smaller than $\pm10$ ns for inter-satellite distances up to 200 km in satellite formation flying [7]. Another example is the Random Access Inter-satellite Ranging (RAISSR) system proposed in [100]. In this system, each satellite transmits a signal to allow other satellites to perform a one-way distance measurement with respect to it. Even when the authors claimed no strict time synchronization, the method requires all the satellites to share a common time reference.

In [101], an approach to solving the problem of autonomous timing is proposed. This method aims to achieve time synchronization and high precision ranging between onboard oscillators and clocks on Earth, with only scarce information exchange. The main idea is to calculate the satellites CFO using a frequency tracking loop. Then, the inter-satellite range and the satellites' relative radial velocity are estimated on the ground [101]. The reported results showed a precision of 0.1 ns for inter-satellite baseline measurements and the accuracy of velocity measurement around $10^{-3}$ m/s, for approximately 0.01 Hz carrier frequency tracking error [101].

An essential aspect of DOWR that limits its performance is its synchronization requirements. As this method requires the transmitter and receiver to be synchronized, its accuracy is directly related to the performance of the synchronization algorithm. Clock drifts and offsets between nodes can introduce errors in the order of microseconds which can be too large for specific applications. On the other hand, the TWR method does not have this drawback. In TWR, introduced in [99], one satellite sends the ranging signal and activates a counter to calculate the elapsed time until the reception of the other satellite replies. In this way, there is no need to synchronize both satellites' clocks [99]. The evaluation of the algorithm on a SDR platform is reported in [99], showing an accuracy of a few centimeters.

Another hardware demonstration was presented in [88]. In this case, the authors developed a coherent distributed transmission system based on high-accuracy microwave ranging to perform DTB for mobile platforms. The Two-tone ranging method used for the indoor and outdoor experiments was introduced in [89].

The most well-known example of pseudo-random code ranging is the method applied by GNSS constellations. A variation of this method to measure the inter-satellite range in deep-space missions, where the navigation constellation is not available, was suggested in [102]. The design combines the typical GPS receiver with a RF transceiver to use when the satellite does not receive the GPS constellation' signals. The GPS module is a flight-qualified design that performs orbit determination using a EKF and the inter-satellite distance measurement is obtained by differencing both filtered absolute GPS solutions. The core of the whole system is a CPU core which controls the switch and coordination of the two ranging units [102]. Similarly, an ARNS for autonomous satellite formation flying in LEO is proposed in  [103]. The idea of this article is that inter-satellite ranging systems can provide additional observation information, which can be used to increase the GPS stand-alone observation dimension. Reported results indicated that the ARNS improved the relative positioning accuracy by one order of magnitude in comparison to the GPS stand-alone solution [103].

In addition, other authors addressed ranging methods based on PRN codes. For instance, in [104] it is presented an algorithm to determine the initial value of smooth pseudo-range for the smoothing method proposed by Hatch [105]. The algorithm uses the least-squares straight line fitting technique to decrease the convergence time and improve the ranging accuracy [104]. On the other hand, the authors of [106] analyzed how it is affected the accuracy of inter-satellite ranging in a DSSS by the used bandwidth and [107] introduced a new ranging scheme combining continuous phase modulation and a PRN ranging code. The chip pulse used in this method is based on a normally distributed signal, which improves the ranging accuracy and reliability of the system.

Another example of a RF ranging technique using the signal's modulation was proposed in [108]. This paper analyzed a hybrid OFDM communication-metrology system for a two satellites formation flying. The system uses OFDM signals for ISL communications and measurements functions. The inter-satellite distance is estimated using a training symbol, which is also used for time and frequency synchronizations and estimation of the channel impulse response. The accuracy of the technique depends mainly on the bandwidth of the transmitted signal [108].

Most recent works focus on carrier ranging methods assisted by pseudo-random code ranging, which allows higher precision range values. For example, [109] proposed a pseudo-code-assisted carrier ranging algorithm, which is the combination of pseudo-range and dual-frequency carrier phase ranging methods. In [110], the impact of the frequency selection on the ranging accuracy was analyzed. Too high frequency leads to problems in the resolution of integer ambiguity, but too low frequency impacts ranging accuracy. Both articles reported range errors on the order of 1 cm for inter-satellite distances around 100 km [109][110].

Even though the accuracy of carrier-ranging methods is higher than pseudo-random code ranging methods, the integer ambiguity is more difficult to resolve for the former ones. The solution of the phase integer ambiguity problem has been dealt in [103], [111] and [112]. For instance, [111] presents a robust integer-cycle ambiguity resolution method by modifying the well-known LAMBDA method [113]. The authors derived the validation threshold in closed-form as a function of the phase noise variance and the antenna baseline geometry. As a result, the success rate of the original LAMBDA is improved. However, the method requires that the spacecraft of which the LOS will be estimated is equipped with a body-fixed array of at least three antennas in different planes [111]. This could be a limitation for small satellites missions. Another procedure based on the LAMBDA method was proposed in [103]. This paper contains two approaches to combine the inter-satellite ranging measurements with GPS. Besides, it proposes a feedback scheme to convert the ambiguity float solutions obtained from the EKF into pseudo-perfect measurements through the LAMBDA method. After the possible integer solutions are verified, they are adopted as pseudo-perfect measurements to update the EKF for the next epoch.

Another approach to the phase integer ambiguity problem was proposed in [112]. In this case, the authors used the simplified time-differenced technique [114] to eliminate the influence of ambiguity in a GPS/INS integrated navigation algorithm [112]. The simplified time-differenced method consists of processing time differences of successive carrier phase measurements at a GPS base station, which increases the velocity and attitude accuracy. It reduces noise in the position information compared to the traditional tightly coupled systems. Unfortunately, the time-differenced carrier phase measurements do not fit easily in the framework of a KF measurement equation, which has adverse effects on the derivation of an appropriate measurement matrix [114].

§.§ Relative Positioning

Spacecraft relative positioning in formation flying is a crucial enabler of new space missions and of paramount importance for distributed satellites architectures. Clearly, two subcategories can be distinguished: in-orbit autonomous and assisted from the ground. Relative positioning can be seen as a coordinated operation of the ranging systems described in the section above, which can occur either in-orbit or on-ground.

§.§.§ In-Orbit Autonomous

The primary technology used for precise relative positioning of autonomous formation flying satellites is GNSS.

If the relative position between two GNSS receivers is required, the GNSS data differences from two receivers can be used. This method reduces common data errors, such as the GPS satellite clock offset. The difference is typically calculated using the pseudo-range estimation between the receivers and a GNSS satellite or more. Once resolved the code ambiguity, sub-metric precision is obtained by using this method [115].

In order to enable, cm to mm positioning accuracy levels, the handling of carrier differential GNSS (CDGNSS) estimations needs to happen [116]. Within the setting of the GRACE formation flying mission, the possibility of 1 mm level relative navigation over a 200 km separation has been illustrated by utilizing carrier GPS estimations from a high-grade dual-frequency BlackJack receiver [117].

A major problem in using CDGNSS is tackling for the unknown integer number of cycles (integer ambiguity) [118]. Especially when the resilience and precision of the integer solution are affected by variations in the number of common-in-view satellites, and ephemeris and ionospheric differential errors. This is enhanced when the inter-satellite distances are highly variable due to the relative orbital trajectory.

A few strategies based on arrays of GNSS sensors have been proposed to unravel the phase ambiguity problem. Since antennas are unbendingly mounted on the stage, the relative antenna position within the local body frame is known in advance. It can be used to improve the accuracy of the estimated integer ambiguity. In [119], the use of the Multivariate Constrained Least-squares AMBiguity Decorrelation Adjustment (MC-LAMBDA) strategy [120] successfully utilizing nonlinear geometrical constraints. By consolidating the known antenna geometry into its ambiguity objective function, this strategy has been appeared to illustrate reliable and immediate single-frequency integer ambiguity determination.

DSS operating above LEO loses coverage of the GNSS infrastructure and has to rely on other means for tracking and navigation. For those missions, a trilateration scheme is proposed in [121] that evaluates the 3D relative position between a reference spacecraft and a target spacecraft using raw-range measurements from a distance baseline of known locations, which is called “anchors”. The anchors can be antennas of a ground-based network or satellites of a space-based network (e.g., GPS). The method assumes the clocks of the anchors to be perfectly synchronized and requires some synchronization between the anchors and the reference spacecraft. However, the synchronization errors between the reference spacecraft and the rest of the satellites in the DSS is compensated by using an additional anchor. This method achieves sub-meter accuracy for a GEO DSS with two spacecraft, using a baseline network of three ground stations as anchors.

Another example of very accurate tracking and control of the relative position is the PROBA-3 mission. PROBA-3 is a European Space Agency (ESA) mission to obtain highly accurate formation flying. A couple of satellites, the Coronagraph Spacecraft (CSC) and the Occulter Spacecraft (OSC) will work together as an externally occulted solar coronagraph. The CSC hosts the optical assembly of the coronagraph as the primary payload, while the OSC carries the coronagraph external occulter disk. The mission requirements are a longitudinal accuracy better than 1 mm for the inter-satellite distance of 144 m [21]. To make this feasible, the formation flight system is distributed between both spacecraft: the CSC hosts the formation flying sensors, while the OSC performs the data processing [122]. The formation flight task requires the use of several metrology subsystems, from coarse accuracy and large scale range determination to very accurate and shorter scale range measurement and absolute positioning. All the data generated by these subsystems are processed in real-time by the Guidance and Navigation Control system, obtaining an unprecedented accuracy [123].

IRASSI is an interferometry-based mission concept composed of five free-flying telescopes orbiting the Sun-Earth/Moon second Lagrangian point, L2. It focuses on observing specific regions of the sky to study star formation, evolution processes, and early planetary origins. The study of these processes requires a telescope with angular resolution lower than 0.1 arcsec [124]. The interferometer relies on dynamically changing baselines obtained through the physical separation during scientific observations to achieve such strict resolution values. For example, the baseline vectors between the telescope reference points must be determined with an accuracy of $5~\mu$m to guarantee the precise correlation of the detected signals. The navigation concept capable of achieving these strict requirements consist of two components: the absolute position estimation concerning Earth and the relative position estimation, which determines the satellite's positions to each other [125]. The description of two autonomous relative positioning algorithms based on a geometric snapshot approach can be found in [126].

For formation flying missions, in addition to the inter-satellites position, it is imperative to estimate and control the satellites' attitudes accurately. In [127], the relationship between the precision estimation against the ranging accuracy, ranging distance, and satellite relative position were analyzed. As a result, it was concluded that obtaining accurate measurements required a balanced number of transmitting and receiving antennas with suitable configurations. It is a necessity to install more antennas to improve the precision estimation of the attitude angles [127].

Another critical requirement of formation flying is accurate relative navigation. Significant research results have been published about this topic. For example, [128] used the nonlinear dynamics describing the relative positioning of multiple spacecraft for formation flying trajectory tracking control. Using Lyapunov-based control design and stability analysis techniques, the authors developed a nonlinear adaptive higher-order sliding mode control commonly known as adaptive super twisting sliding mode control. On the other hand, [129] develops an efficient approach of autonomous relative orbit determination for satellite formation flying. The proposed solution uses the inter-satellite local measurements by the microwave radar and laser devices on board the satellites to perform the relative navigation. The design uses a decentralized Schmidt KF [129] to estimate the state of relative orbit between the satellites and proves that this approach is immune to the single satellite failure. Simulation results showed that the relative position estimation might achieve centimeter-level accuracy, and relative velocity estimation may achieve mm/s-level accuracy for a circular spatial formation consisting of three satellites where the chief satellite is at the center. The radius is about l km [129].
Similarly, in [130] a method for autonomous orbit determination combining X-ray pulsar measurements and inter-satellite ranging during Mars orbiting phase was presented. The method calculates an observability index reflecting the measurement information quality and optimizes the observable target selection and the observation scheduling. Then, the Unscented KF is used to estimate the autonomous pulsar assisted orbit determination [130].

§.§.§ Assisted from Ground

Determining the orbital position of satellites from the ground requires obtaining azimuth and elevation view angles of the satellite and the distance from the ground station to the satellite (range).

One of the most precise POD measurements assisted from the ground can be achieved by Radio Interferometry methods. Interferometry is a technique for passive tracking. A simple interferometer consists of two antennas. Using the phase difference between the signals received by the two antennas, the direction of the target can be determined. Connected-Element Interferometry (CEI) is a technique for determining the phase offset from the TOA difference of a downlink radio signal to two antennas on a short baseline. In [131], the authors used a small-scale CEI system of two orthogonal baselines (75 m $\times$ 35 m) to track a GEO satellite. Reported results showed accuracies smaller than 1 km in the radial and the cross-track directions, and 3D position accuracy in the order of 2 km. Another example of the positioning of GEO satellites by Radio Interferometry was proposed in [132]. In this case, the authors focused on determining the view angles of GEO satellites with an estimation accuracy of 0.001° in Ku band.

In addition, ISL can improve the accuracy of orbit determination for GNSS constellations. One example of this was presented in [133], where it was demonstrated that the POD of the Kepler system could be performed with just one ground station achieving orbit accuracies of 5 cm in 3D and 0.24 cm in radial direction [133]. On the other hand, in [134], it was proven that the use of ISL ranging measurements could reduce the first positioning time about six times in conventional navigation receivers.

§.§ Summary and Lessons Learnt

* In a DSS, the inter-node ranging and relative position are closely related to the synchronization algorithm. Generally, the requirements and accuracy of the coherent operation depend on the performance of the ranging and relative positioning algorithms as much as the synchronization itself.

* Optical inter-satellite ranging methods can achieve higher accuracy, but the higher directivity of the laser beam can represent a limitation for specific applications.

* The use of a single optical ISL to simultaneously perform ranging and communication is a trending topic in this field.

* Inter-satellite ranging based on RF can use measurements of the carrier's phase or use the signal modulation. Recent publications considered the combination of both: carrier ranging assisted by pseudo-random code ranging methods.

* Both optical and RF-based ranging algorithms have to deal with the phase integer ambiguity problem.

* Spacecraft relative positioning can be seen as a coordinated operation of the ranging systems described above, which can take place either in-orbit autonomous or assisted from the ground.

* The leading technology used for precise relative positioning of autonomous formation flying satellites is GNSS.

* DSS operating above LEO loses coverage of the GNSS infrastructure and has to rely on more complex schemes. Examples of these schemes were analyzed in this section.

* One of the most precise POD measurements assisted from the ground can be achieved by Radio Interferometry methods.

§ APPLICATIONS OF DISTRIBUTED SATELLITE SYSTEMS SYNCHRONIZATION

As stated above, carrier and time synchronization are critical and very challenging requirements for CDSS. Consequently, the research in the area is very prolific. In this section, the most significant articles about synchronization for DSS found in the literature are summarized. They are arranged in two groups attending to its application: Communications or Remote Sensing.

§.§ Communications

Small satellites, organized in distributed systems, are envisioned to be the future of space communications [14]. Forming a dynamic phased array in space with the nodes of a DSS can improve the communications capabilities between the network and the Earth. However, implementing such an array requires effective open-loop carrier synchronization [2]. In [8], the phase synchronization constraints of a DSS of very simple, resource-limited femto-satellites communicating with Earth were derived and discussed [8]. Fig. <ref> from [8] shows the normalized amplitude of the received signal as a function of the number of transmitters for different accuracies of phase synchronization. As can be appreciated in the figure for a high number of nodes, the effects of phase synchronization inaccuracies are more evident.

Normalized amplitude of the received signal vs. the number of satellites for different values of phase synchronization accuracy [8]

In [9], a time and phase synchronization solution to perform beamforming in a LEO-DSS is proposed. The mathematical framework and analysis of the synchronization scheme is developed, which relays in an external beacon transmitted from a GEO spacecraft above the intended ground receiver. This open-loop method achieves subcentimeter-level (subnanosecond-level) phase synchronization with localization accuracy on the order of meters [9]. However, these results are restrained to a particular array geometry, which is not always feasible.

In [7], the simulation and performance evaluation of the two-way range and time synchronization method for a DSS were presented. The DSS consisted of four satellites in a formation flying mission, with 50 km of inter-satellite distance and a carrier frequency of 400 MHz simulated in the Systems Tool Kit (STK) platform from Analytical Graphics, Inc. According to the simulation results, the error of the time synchronization was less than 10 ns [7].

§.§ Remote Sensing

Remote sensing and Earth observation from DSS have become attractive for the community in the last years. New concepts, like distributed synthetic aperture radiometers or fractionated radars, offer the potential to significantly reduce the costs of future multistatic SAR missions [135][136]. Besides, the low radar power budget and high spatial resolution requirements of future remote sensing satellite missions seem to be only possible to achieve through multistatic configurations [137]. However, the frequency, phase, and time synchronization is still the major challenge that slows down the launch of new missions. For that reason, new synchronization methods for Remote Sensing DSS are published daily.

For instance, an inter-satellite time synchronization algorithm for a micro-satellite cluster was proposed in [34]. The authors proposed a time control loop that dynamically adjusts each satellite reference frequency according to the time difference with the time benchmark of the cluster. Consequently, time synchronization is achieved by locking the clocks of all the satellites to a chosen one. Similarly, [138] presents a method to relatively synchronize the clocks in a DSS without the use of GNSS signals. A reference pulse transmitted by a master satellite is received and time-tamped by all the secondary satellites in its broadcast domain. The method estimates the clock offsets relative to the other secondary satellites by calculating the receive-time differences between two secondary satellites. Simulation results reported in [138] showed excellent convergence rates. However, this method requires that all the distributed nodes are in the same broadcast domain of a single master satellite, and the LOS from one satellite to all others must always be available. Besides, the algorithm is affected by the difference in propagation time between two receivers. This is not a problem for relatively small networks but, for DSS with longer inter-satellite distances, these propagation delays have to be compensated.

Other authors have studied the effects of time and frequency synchronization on DSS SAR [139][140]. In [139], a model considering the time and frequency synchronization errors in InSAR were presented. The performance of the range and azimuth compression with the synchronization errors was analyzed to obtain the time and frequency synchronization requirements for parasitic InSAR system design. On the other hand, [140] studied the performance degradation in bi-static and multi-static SAR due to the oscillators' phase noise. Using a dedicated synchronization link to quantify and compensate the oscillators phase noise was proposed considering three different synchronization schemes: continuous duplex, pulsed duplex and, pulsed alternated. The analysis included additional factors such as receiver noise and the Doppler effect and contributions known from sampling theory like aliasing and interpolation errors. According to the reported results, successful oscillator phase noise compensation is possible if the compensation algorithm and the signal timing are adapted to the link hardware and SAR parameters [140].

More advanced works have reached the implementation and test steps, which are fundamental to using a synchronization method in a mission. For example, [141] proposed a hardware-in-the-loop simulation and evaluation approach for DSS SAR. The proposal was used to model a typical bi-satellite formation spaceborne distributed SAR system in the X band. The SAR' central electronic equipment was implemented in hardware, whereas the echo generation and the processing and evaluation of the results were performed in software. Another example is the testbed for coherent distributed remote sensing systems proposed in [137]. This platform is composed of two satellites, the channel emulator, and two targets, all implemented in USRP. The authors proposed and validated a dual carrier point-to-point synchronization loop through the testbed. The synchronization algorithm is based on a Master-Slave architecture to autonomously synchronize both satellite clocks with a common reference using ISL [137]. Generally, the hardware simulations include features that can not be easily or accurately considered with computer-based simulations. First, there are coupling relationships among the channel mismatch, the time and phase synchronization errors, and other error sources. The single error analysis method used in the computer-based simulation can't meet this requirement. Besides, the hardware implementation of the synchronization algorithm gives a more accurate description of its performance. Computer-based simulations cannot precisely measure features like processing time and resources. On the other hand, the error source characteristic has to be known for data analysis and evaluation, which is more complicated for hardware simulations.

Several remote sensing distributed satellite missions have been proposed and deployed during the last years. Section <ref> will refer to already deployed missions, but the proposed synchronization algorithms for future missions are of interest to this section.

* ARGOS:  The Advanced Radar Geosynchronous Observation System (ARGOS) will be a MIMO SAR system hosted on a swarm of mini-satellites in quasi-geostationary orbits [142]. It consists of: a swarm of active and passive spacecraft in a zero inclination quasi-GEO orbit; a data-link to a telecommunication satellite that is used for both data download and synchronization; a ground segment and; a network of active calibrators for precise estimation of sensors clocks and orbits. According to [143], the link with the telecommunication satellite can be used to synchronize the DSS to a common clock and support the estimation of the precise orbits, in a similar way to a cloud PLL. Besides, the oscillators' phase noise can be compensated by exploiting a network of Compact Active Transponders or through an on-ground synchronization scheme capable of estimating the phase on very bright point targets.

* LuTan-1: LuTan-1 (LT-1), also known as TwinSAR-L mission, is an innovative spaceborne bistatic SAR mission based on the use of two radar satellites operating in the L-band with flexible formation flying, to generate the global digital terrain models in the bistatic interferometry mode [144]. Several articles about the LT-1 phase synchronization scheme have been published [144][145][146][147]. The algorithm, proposed in [145], considers the time multiplexing of the synchronization pulses with the radar pulse-repetition intervals avoiding any interference in the SAR data acquisition. Besides, [144] proposed a KF phase-error estimation and compensation method, which improves the synchronization accuracy for low SNR. On the other hand, [146] evaluated the performance of the LT-1's synchronization scheme for multipath effects, and [147] presents its experimental verification.

* ConGaLSAR: A Constellation of Geostationary and LEO SAR (ConGaLSAR) radars has been proposed in [136]. The phase and time synchronization of this constellation would be supported by a novel transponding mode known as MirrorSAR, where the LEO satellites work as relay nodes for the ground echoes to the illuminating GEO satellite. The key idea of MirrorSAR is to redirect the radar echoes to the spacecraft, where they can be coherently demodulated with the same clock references previously used to generate the radar pulses [135]. The authors introduce two alternatives for synchronization. In the most simple one, the LEO subsystem functionality is limited to amplify and forward the received RF signal. The second possible configuration, named Double Mirror Synchronization, requires the transmission of a very stable reference signal from the illuminator to the LEO-satellites by using a dedicated low-gain antenna. Even though this method increases the synchronization processing considerably and requires more complex LEO-satellites, it still assumes that the distance between the transmitter and receiver satellites is almost constant, and it can be accurately estimated [135].

§.§ Summary and Lessons Learnt

* Not many publications consider CDSS for communications applications. However, there are some simulations for LEO DSS, synchronized with the aid of a GEO spacecraft. In addition, the two-way time and range synchronization method has been considered for the synchronization of a DSS of four satellites in a formation flying mission.

* Remote sensing and Earth observation from DSS has become attractive for the community in the last years. However, the frequency, phase, and time synchronization is still the major challenge that slows down the launch of new missions. For that reason, new synchronization methods for Remote Sensing DSS are published constantly.

* Time and phase synchronization by master-slave architecture using intra-node communications are the preferred methods for remote sensing DSS. However, the complexity of the system increases with the number of distributed nodes to be synchronized due to the increment of ISL required.

* Information about the synchronization algorithms for future remote sensing distributed satellites missions was discussed in this section, specifically ARGOS, LuTan-1, and ConGaLSAR missions.

§ EXAMPLES OF SYNCHRONIZATION FOR DISTRIBUTED SATELLITE SYSTEMS

The development of new synchronization techniques in DSS as in any other wireless network requires three main steps: design, prototyping, and deployment. The new synchronization method is theoretically analyzed, tested, and improved using software simulators during the design step. However, no matter how thorough the design step, new algorithms should comply with the prototyping before deployment. Theoretical design - software simulation - prototyping is a closed-loop that must be executed repeatedly to improve a synchronization technique before its deployment in DSS. Theoretical design and simulation of synchronization algorithms suitable for DSS were already covered in previous sections. This section summarizes the most relevant hardware implementations and prototyping of such algorithms.

The most relevant and advanced examples of synchronization techniques for cohesive distributed systems are analyzed in the following. Unlike the previous section, where concepts and theoretical methods were discussed, this section presents synchronization algorithms that have been implemented and tested at least as a hardware PoC. Similar to the previous section, the examples are classified according to their application as Synchronization Examples in Distributed Communications Systems and Synchronization Examples in Remote Sensing DSS.

§.§ Synchronization Examples in Distributed Communications Systems

Hardware development for communication satellite systems has been pointing toward reconfigurable SDR System-on-a-Chip ground receivers, and to the ultimate extreme of Satellite-on-a-chip during the last years, [14]. In general, recent trends on prototyping and deployment of spacecraft are based on COTS [148], FPGA [149] and SDR [150] designs. Mostly due to its reconfigurable characteristics and because they allow less expensive and faster development processes. Several examples of the use of SDR for inter-satellite communication in small satellite systems can be found in [2].

However, there are not many publications about the hardware prototypes of synchronization algorithms, specifically in Communications CDSS. Mainly because the use of CDSS for communications is a very recent topic that does not have any launched missions yet. Nevertheless, it is helpful to analyze the PoC of the synchronization algorithms used in distributed terrestrial communications that DSS could apply. This can support the selection of the synchronization technique and the hardware platform during the design of a prototype for Communications CDSS. The examples are presented in order of increasing complexity, and some technical details are listed in Table <ref> at the end of the section.

§.§.§ Mobile Communications Networks

One example of distributed coherent radio system that shows some similarities with the DSS is the recently proposed CF massive MIMO concept. In CF operation, the network infrastructure performs beamforming towards the UE using multiple distributed radio nodes. The beamforming coordination relies on the TDD operation and uses a densely distributed network topology [151, 152]. Although there is a CPU in CF Massive MIMO system, the exchange of information happens only in the form of payload data while the power control coefficients change slowly. Neither is there a provision of sharing the instantaneous CSI among the AP or the CPU. Channel estimation is performed only at the AP using uplink pilot signals and further used for decoding uplink symbols and precoding the downlink symbols. [153].

A crucial requirement of the CF Massive MIMO is the global synchronization of RF carriers and information symbols across the geographically distributed service nodes network. In [154], CFO estimation for Distributed Massive MIMO and CF Massive MIMO is studied. Specifically, the article evaluates the performance of distributed large-scale MIMO systems with two collaborative (successive and joint), and one non-collaborative (independent) CFO estimation techniques [154]. According to [155], the global synchronization of RF carriers and information symbols across a distributed MIMO network can be done with a common 1 pps (pulse per second) GPS timing signal used to provide GPS disciplined oscillators [155]. This solution was demonstrated for two 50 MHz carriers tracked by frequency lock loops implemented on FPGA development platforms. Results showed a locking time of 500 s and a coherence time of 10 s, which is enough for the MIMO processing [155].

The TDD operation implemented in CF Massive MIMO is not very appealing for a DSS used for communication applications due to the long latency of the satellite orbits. However, an extrapolation of the method to FDD scenarios can be applied. In such a case, more robust synchronization in time and phase is required between the distributed satellite nodes. One example of FDD standardized technology in mobile networks is the CoMP transmission and/or reception used in Long Term Evolution Advanced (LTE-A) standard.

In CoMP operation, multiple transmission points coordinate to combine the user's information signal from neighboring evolved Node B (eNB) to improve the received signal quality. To this end, CoMP transmission requires sharing coordination information, usually consisting of user's feedback CSI, through the backhaul links [156]. It can be implemented in four different types depending on the degree of coordination among cells: (1) Coordinated Scheduling and Beamforming and, (2) Joint Processing, both in the downlink; (3) Coordinated Scheduling and, (4) Joint Reception and Processing, both in the uplink. Even though multiple types of CoMP can be used together [156]. For example, in [157] proposed to switch the CoMP transmission mode between Coordinated Beamforming and Joint Processing adaptively to maximize the average achievable rate, considering that the last one is more sensitive to phase synchronization errors than Coordinated Beamforming [157].

CoMP requires the transmission points to be tight time and frequency synchronized. This constraint mostly affects coordination between BS because some additional solution is needed to provide accurate phase synchronization. Large distances between the BS result in unavoidable differences in TOA between the users' signals which in turn lead to inter-symbol interference in OFDM systems if the cyclic prefix length is exceeded. Moreover, CFO which are caused by imperfect oscillators, lead to inter-carrier interference and are a problem, particularly in the downlink due to the feedback delay of the measured CSI. The most extended approach to synchronization in CoMP at outdoor BS is to obtain a precise clock reference from GNSS [158]. This could be extended to DSS in LEO MEO. However, indoor BS require to be synchronized over the backhaul network using standard protocols such as PTP or proprietary solutions like synchronous Ethernet [159] [160] [161].

In any typical cellular system, the users perform synchronization with a single BS where the BS transmits predefined signals to the UE to adjust their transmit timings so that signals received from many UEs to the BS are time-aligned. However, a concern in CoMP is that the UE needs to be synchronized to multiple BS simultaneously. Hence, the previously discussed technique is not going to work in the CoMP case. Due to this, the usage of CoMP becomes limited to deployments where the inter-site distances between BS are small [162].

Different approaches to the time synchronization problem stated before have been proposed. The most straightforward is to use a larger cyclic prefix, but it reduces the system's throughput. Another widely used approach is adjusting the time advance based on the nearest BS. However, such a solution is not capable of fully solving the problem since some of the users may not be able to use the CoMP. In [162], the authors propose a linear combination of samples corresponding to the two consecutively received OFDM symbols to identify
multiple transmitted signals. Such a scheme enables the mitigation of Multi-user Interference by applying Successive Interference Cancellation, as also proposed in [163] and [164]. Another approach considers the positions of the serving and interfering BS and the estimated location of the UE to timely relate the synchronization signals received from adjacent BS [165]. Several other approaches which consider asynchronous interference mitigation are discussed in [166] and [167]. All these methods could be extrapolated to time synchronization in DSS.

Joint processing of the carrier phase (along-with frequency) is one of the significant challenges in CoMP as in DSS. The time difference of arrivals imposes a phase offset on the transmit covariance matrix leading to improper precoding matrix index selection and therefore limiting the promised performance of CoMP [168]. It is a two-fold problem: (a) The RF-LO of the participating BS in CoMP are required to be synchronized among them, (b) the LO of the UE are also required to be synchronized with the corresponding BS. Since the phase noise of the oscillator evolves quickly and only with the help of backhaul information, it becomes challenging to establish synchronization between the participating BS. For achieving an acceptable level of synchronization, the phase noise needs to be very low in the LO, which makes such deployment cost-inefficient. In another approach, the UE can also keep tracking the phase noise. The effect of BS coordination and synchronization on the phase noise estimation at the user receiver was studied in [169], where it is derived the data-aided and non-data-aided CRLB for the phase noise estimation in a CoMP system. Another approach presented in [170] addresses the data-aided joint synchronization and channel estimation for the uplink of CoMP systems based on the MMSE criterion. This proposal considers an alternating minimization method to simplify the MMSE cost function such that the multiuser CFO, sampling timing offset, and channel frequency response can be estimated in an iterative approach more feasible for practical systems. Another method that addresses the estimation and compensation of time and frequency offsets in CoMP was proposed in [171]. In this case, the authors compare two solutions to the time-frequency offset estimation problem, one in which the UE is not aware of the transmitting point, and one in which the UE is provided with assistance information.

§.§.§ Implementation Examples of Closed-Loop Synchronization Methods

* One Bit Feedback: In [59], an FPGA (Spartan-3) based hardware prototype was developed to demonstrate 1BF. Phase synchronization is achieved OTA whereas the frequency synchronization is attained via RF cables, and a frequency distribution source Octoclock [172]. The implementation used three single transmitters (beam-formers) and a single antenna receiver. At a feedback rate of 300 ms, convergence within 90% of the theoretical limit is achieved after 60 iterations. The experiments were performed inside the laboratory, i.e., static environment. Hence, the performance under the realistic time-varying channels was not investigated. Besides, frequency synchronization was also required to be achieved OTA.

A hardware prototype of the 1BF method, using USRP SDR [173] was presented in [174]. This prototype functionality is "all-wireless", i.e., no wired connections are used to attain frequency and timing synchronization. This experimental setup used three single-antenna transmitters and one single antenna receiver. With a feedback interval of 50 ms, nine times beamforming gain was achieved. However, like the implementation in [59], the experiments were performed indoors. Then, the performance of the system against the time-varying channel was not investigated. Besides, only phase synchronization is achieved OTA whereas frequency synchronization is achieved through coaxial cables and frequency distribution source.

Another prototype described in [175] performs both frequency and phase synchronization using 1BF, which has been derived from [59]. Individual components of the hardware prototype have been developed by the authors themselves and the beamforming algorithms have been programmed on DSP cores. At a test frequency of 60 GHz, using three elements distributed phased array, 9.2 dB of distributed beamforming gain is achieved. In contrast to the methods discussed previously, i.e., [59] and [174], both frequency and phase synchronization are achieved OTA. However, the feedback interval of 0.5 ms is excessively high as compared to the other 1BF methods discussed ([59] and [174]). Besides, the prototype was tested against static channel environments; hence the performance of the system against the time-varying channel was not investigated.

* Full-feedback: Very few PoC demonstrators are available for Full-Feedback implementations. For example, in [176] a notable range extension by performing outdoor OTA tests was shown using COTS handheld radio (based on FPGA and DSP) equipment. Also, the KF is used to predict the offset in frequency and phase, which further assists in tackling the time variation of the channels. Thus making the system perform satisfactorily even when the nodes are mobile. It is shown that as compared to a COTS radio transmitter (approximate range 2 km), by performing distributed beamforming, the range can be extended up to 6 km using ten such radio nodes. Further extension of range up to 10 km can be achieved by using 30 such radio nodes. In [177], using the same experimental setup and beamforming method, 0.1 dB distributed beamforming gain was demonstrated using ten COTS radio nodes. OTA operation and application of KF for prediction makes this implementation suitable for DSS scenarios.

§.§.§ Implementation Examples of Open-Loop Synchronization Methods

Among the open-loop algorithms, in [178] an SDR implementation (USRP X310 [179]) using transmitter nodes was demonstrated. OTA tests confirmed reliable frequency lock even at very low power. In another open-loop SDR implementation (USRP N210 [180]), a solution for OFDM based frames is developed by designing specific preambles to estimate the frequency and timing offset. The system is capable to operate reliably under frequency selective and slow fading channels. OTA showed performance gains from 2.5 dB to 2.8 dB.

§.§.§ Implementation Examples of Master-Slave Synchronization Methods

In [181], an SDR implementation of Master-Slave closed-loop synchronization procedure named as NetBeam was demonstrated. NetBeam involves a network of radios that broadcast information locally and jointly transmit it to a remote receiver. This is equivalent to a Master-Slave closed-loop synchronization. The performance is further enhanced by the creation of a cluster of beamformers based on CSI. As for the hardware implementation: a dual antenna configuration of USRP X310 is used as a receiver, and a single antenna configuration of USRP B210 is used as transmitters. A significant aspect of this implementation is that the 3D distribution of radio nodes is considered, which is crucial for DSS (full-dimensional beamforming). Besides, a faster convergence is attained through machine learning techniques.

A SDR (USRP) implementation for parallel frequency and phase synchronization was demonstrated in [182]. In this case, the Master-Slave method was used for frequency synchronization whereas 1BF was used for phase synchronization. This is an all wireless implementation, where no wired medium is used for CSI feedback or clock/frequency distribution. Convergence time for the beamforming is of the order of several milliseconds. Another example of frequency and phase synchronization by the Master-Slave method is described in [183] using a SDR (USRP X310). In this case, coherent pulses are received with more than 90% of the ideal coherent energy. Similarly, [184] provided a comparison between wired and wireless clock distribution to achieve frequency and phase synchronization by Master-Slave. OTA tests showed a near-ideal 6 dB gain from a two-transmitter system at a distance of 85 m and coherent gain up to 90% of the ideal signal summation is achieved.

On the other hand, in [185] a SDR (USRP N210) implementation of a Master-Slave architecture for phase synchronization is presented. The external frequency source is used for stable clock distribution whereas phase synchronization is achieved through the Master-Slave method. Channel reciprocity is exploited to obtain CSI without the need for feedback. Besides, the nodes cooperate in disseminating the CSI. Beamforming gain as close as 90% to the ideal beamforming gain is achieved.

Besides, in [186] the authors analyze the wireless carrier frequency and sample timing synchronization by performing an exchange of RF signals between the master and slave nodes. Further, the proposed solution was implemented in GNU Radio, and wireless tests were performed using Ettus USRP N210 SDR. Besides, the estimation of the fractional clock phase was performed using matched filter bank consisting of sixteen fractionally delayed Zadoff-Chu sequences [187] which were capable of estimating residual timing offsets as small as 1/16 of the sample duration. However, the system did not address propagation delays, and also the system bandwidth was limited to only 1 MHz. The method yields a residual timing precision of approximately 500 ns which exceeds the expectation of CRLB.

In addition, authors in [188] used a SDR implementation (using USRP-E312) to perform synchronization, which did not require time-stamps. The observed accuracy of the method was less than 1 $\mu$s (0.8 $\mu$s) for 150 kHz sampling rate with precision limited to 1/10 of the sample rate. The authors reported the limited real-time processing capability of the USRP-E312 as the reason behind using such a low sampling rate. Three E310s were used in the over-the-wire setting: one as the master node, a second as the slave node, and a third as the measurement device for determining the resulting clock offset.

§.§.§ Implementation Examples of Consensus-Based Synchronization Methods

The PoC presented in [189] demonstrates consensus-based synchronization through SDR (USRP N210) implementation. GNU Radio was used for software-based signal processing. Both time delay and CFO estimation, as well as tracking, are addressed in the implementation. Convergence time is of the order of seconds when different nodes have different CFO. OTA tests showed carrier frequency offsets to be within 100 parts per billion (ppb) while timing offset were correctly estimated. Besides, the implementation also showed tracking capabilities.

Another notable consensus-based implementation can be found in [186]. For OFDM type frames, a specific frame structure was designed to facilitate frequency synchronization. A residual timing offset within 1/16 of symbol duration and a residual frequency offset of 5 Hz is achieved. With such residual frequency and timing offset, a near-optimal received signal power gain is shown when distributed beamforming is performed.

Summary of PoC hardware implementations of distributed synchronization techniques

1|c|      1c|Synch. Method     1c|c]@c@Frequency
Synch      1c|Phase Synch.      1c|c]@c@Feedback
rate (ms)      1c|c]@c@Beamforming Gain /
Convergence     1c|c]@c@Static /
Time-Varying     1|c|Platform

1|l|[59]       1l|Closed-Loop      1l|Wired        1l|Wireless     1l|300     1l|90% of theoretical limit            1l|Static           1l|FPGA
1|l|[174]         1l|Closed-Loop      1l|Wired        1l|Wireless     1l|50      1l|9x                                  1l|Static           1l|FPGA
1|l|[175]        1l|Closed-Loop      1l|Wireless     1l|Wireless     1l|0.5     1l|9.2 dB                               1l|Static           1l|DSP
1|l|[176]      1l|Closed-Loop      1l|Wireless     1l|Wireless     1l|NA      1l|0.1 dB                               1l|Time varying     1l|FPGA and DSP
1|l|[178]     1l|Open-Loop        1l|Wireless     1l|Wireless     1l|NA      1l|2.5 - 2.8 dB                           1l|Time Varying     1l|FPGA
1|l|[181]             1l|Master-Slave     1l|Wireless     1l|Wireless     1l|NA      1l|10x                                 1l|Time Varying     1l|FPGA
1|l|[182]           1l|Master-Slave     1l|Wireless     1l|Wireless     1l|NA      1l|NA                                  1l|Static           1l|FPGA
1|l|[184]           1l|Master-Slave     1l|Wireless     1l|NA                1l|NA      1l|90% of theoretical limit            1l|Time Varying     1l|COTS
1|l|[185]         1l|Master-Slave     1l|Wireless     1l|Wireless     1l|NA      1l|90% of theoretical limit            1l|Static           1l|FPGA
1|l|[183]          1l|Master-Slave     1l|NA                1l|Wireless     1l|NA      1l|90% of theoretical limit            1l|Time Varying     1l|FPGA
1|l|[186]          1l|Master-Slave     1l|Wireless                1l|Wireless     1l|NA      1l|NA            1l|Static     1l|USRP
1|l|[188]          1l|Master-Slave     1l|Wired                1l|Wired     1l|NA      1l|NA            1l|Static     1l|USRP
1|l|[189]          1l|Consensus-Based     1l|Wired                1l|Wireless     1l|NA      1l|NA            1l|Time Varying     1l|FPGA

§.§ Synchronization Examples in Remote Sensing Distributed Satellite Systems

This section analyzes the synchronization methods used by DSS missions to perform Earth observation and remote sensing tasks. At the end of the section, Table <ref> summarizes the technical details of these missions.

§.§.§ GRACE

The GRACE was a mission devoted to monitoring changes of the Earth's gravity field irregularities from its dispatch in March 2002 to the conclusion of its science mission in October 2017 [18]. GRACE comprised of two equal satellites in near‐circular orbits at  500 km elevation and 89.5$^\circ$ inclination, detached from each other by around 220 km along‐track, and connected by an exceedingly precise inter‐satellite, KBR system. The satellites were nominally held in a 3-axis stabilized, nearly Earth-pointed orientation, such that the KBR antennas were pointed accurately at each other. The KBR gives a micron-level precision (10 $\mu$m) using carrier phase estimations within the K (26 GHz) and Ka (32 GHz) frequencies [190]. A single horn serves as the K/Ka antenna for both transmitting and receiving the inter-satellite dual-band wave signals [191]. Each satellite transmits two sinusoidal signals (at K and Ka bands) with a frequency offset (nominally set to 0.5 MHz). The two 0.5 MHz down-converted RF signals are sampled at approximately 19 MHz and passed to the digital signal processing part of the receiver. Dedicated digital signal processing channels are used to digitally counter-rotate the phase of each down-converted signal, track phase with a digital phase-locked loop, and extract phase.

§.§.§ LISA

LISA stands for Laser Interferometer Space Antenna [192]. It is a distributed satellite mission formed of three spacecraft operating in formation flying at a 5,000,000 km distance, being the three peaks of an equilateral triangle. This arrangement composes a huge interferometer to monitor the gravitational waves coming from galactic and out of the galaxy sources. Each spacecraft contains two verification masses and laser bars that measure the separation between its masses and those from the other nodes with a required precision of 20 pm. This is the most extreme example of accurate ranging between flying payloads ever seen before. The spacecraft use inertial sensors and micro-newton thrusters to determine and control their orbits, whereas laser interferometry is used to measure the distance with the required accuracy.

§.§.§ OLFAR

The OLFAR is a space-based low-frequency radio telescope that investigates the universe's so-called dark ages, maps the interstellar medium, and finds planetary and sun-powered bursts in other solar systems. The telescope, which is composed of a swarm of fifty satellites, was sent to an area distant from Earth to maintain a strategic distance from the high Radio Frequency Interference found at frequencies underneath 30 MHz, coming from Earth [193]. The satellites can be maintained in a 3D configuration with a maximum diameter of 100 km [194] by using round-trip pulse-based synchronization.

§.§.§ TanDEM-X

TanDEM-X was a scientific mission that comprised of two X-band SAR satellites following an orbit in near arrangement, with variable separation between them between 500 and 1100 m [49]. The mission produces high accuracy SAR snapshots at X-band in both monostatic and bistatic setups. For the bistatic design, one of the two satellites works as a transmitter and the other one as a receiver, and the mission performs a closed-loop synchronization scheme to get the required coherence.

§.§.§ GRAIL

The GRAIL was a NASA mission to outline the gravity field of the Moon to a remarkable level of detail [195]. Twin shuttles were propelled on 10 September 2012 and were embedded into lunar orbit on 31 December 2011 and 01 January 2012 [196] correspondingly. The instrument for this mission was based on GRACE. Nevertheless, there were a few contrasts between both missions. The principal difference between the GRAIL and GRACE instruments emerged because GRAIL was not suffering the Earth's atmosphere drag, nor GPS navigation system was accessible. GRACE compensated for air disturbance basically by having an accelerometer to measure non-gravitational speed variations and by using two independent microwave ranging frequencies instead of only a single one.

GRAIL was streamlined by excluding the K-Band frequency (26 GHz) as well as the accelerometer (non-gravitational strengths are small enough to be modeled rather than measured). Furthermore, for GRACE, GPS was utilized to estimate the relative delay between the two spacecraft, calibrate on-board ultra-stable oscillators, and track the two satellite orbits. Without GPS accessible at the Moon, GRAIL included an extra S-Band (2 GHz) Time Transfer System to supplant the GPS timing measurements, and an extra X-Band (8 GHz) Radio Science Signal used for Doppler following of the shuttle and ultra-stable oscillator frequency calibrations through the Deep Space Network [195].

Summary of synchronization requirements for Remote Sensing DSS missions

1|c|Mission     1c|Application     1c|c]@c@Distance
nodes     1c|c]@c@Band
operation     1c|c]@c@Required
accuracy      1c|c]@c@Number
nodes     1c|c]@c@Synch.
signal     1c|c]@c@Phase (P)
Timing (T)
Ranging (R)

GRACE     Gravity     220 km     c]@c@K (26 GHz)
Ka (32 GHz)     10 $\mu$m     2     CW     R
LISA     c]@c@Gravitational
waves     $5\cdot10^6$ km     Optical     20 pm     3     CW     R
OLFAR     c]@c@Radio
telescope     100 km     TBD     $<$ 1 m     10 - 50     Pulsed     PT
TanDEM-X     SAR     0.5 - 1.1 km     X      $\pm 0.1^\circ$     2     c]@c@Pulsed
chirp     PT
GRAIL     Lunar gravity     50 - 225 km     c]@c@S (2 GHz)
X (8 GHz)
Ka (32 GHz)     1 $\mu$m     2     CW     PTR

§.§ Summary and Lessons Learnt

* Hardware development for communication satellite systems has been pointing toward re-configurable SDR System-on-a-Chip ground receivers and to the ultimate extreme of Satellite-on-a-chip during the last years. In general, recent trends for prototyping and deployment of spacecraft are based on COTS, FPGA and SDR designs.

* Synchronization algorithms from Mobile Networks performing cohesive distributed communications, such as CF Massive MIMO and CoMP, can be extrapolated to synchronize CDSS.

* Most prototypes of synchronization methods for distributed wireless communication networks implemented Master-Slave architecture and FPGA platforms.

* DSS missions performing Earth observation and remote sensing tasks were discussed in this section. Most of them do not include more than three distributed nodes, and the synchronization is achieved using RF signals. Missions included in the section are: GRACE, LISA, OLFAR, TanDEM-X and, GRAIL.

§ SYNCHRONIZATION USING MACHINE LEARNING

ML techniques can be significantly useful to address various synchronization problems associated with the end-to-end communication system. Some applications of ML in facilitating the synchronization process include frame synchronization, compensation of the errors caused by sampling frequency/time offsets, carrier synchronization, and characterization of phase noise. While modeling an end-to-end communication system, there may arise synchronization problems between transmitter and receiver due to various reasons including, sampling frequency offset, sampling timing error, and mismatch about the beginning of each frame, i.e., frame header, at the receiver. To address these problems, a CNN could be promising to build an additional synchronization model in order to achieve better frame synchronization and also to compensate for the impairments caused due to sampling timing error and sampling time offset. To this end, a CNN-based synchronization model with softmax activation function proposed in [197] has demonstrated 2 dB better detection than the direct correlation detection in terms of correctly detecting the actual position of a frame header. Furthermore, the traditional frame synchronization techniques based on maximum likelihood and correlation may fail due to hardware implementation constraints or frequency deviation problems. To address this, ML techniques such as multi-instance learning can be used to solve the frame synchronization problem under different frequency ranges without additional modifications [198].

Moreover, ML techniques can be used to accurately characterize amplitude and phase noises, which are essential parameters in the synchronization process of distributed satellite systems. In this regard, in [199] a Bayesian filtering-based framework was used in combination with the expectation-maximization to characterize the amplitude and phase noise characterization of the lasers. The carrier synchronization has been experimentally demonstrated using the proposed framework, and the Bayesian filtering has been shown as an efficient method to estimate laser phase noise even in the presence of low SNR.

Due to limited computing power, the complexity of the ML model, and the increased data volume, a single learning node/machine is generally not able to execute an ML model, and this will require the distributed implementation of ML models across several distributed nodes in a network. In distributed learning, a local node computes a data subset's local updates/models. It updates the local updates/model parameters to the centralized server, which then calculates the global parameters [200]. The computed global parameters are then distributed to the local nodes to all distributed nodes. This distributed approach significantly reduces the communication burden required to communicate all the local raw data to the centralized server and utilize the local computational power to generate local models.

However, the application of ML techniques in distributed systems may result in relatively low accuracies and poor convergence rate due to differences in the transmission delays and computational capabilities of distributed nodes/clusters. Therefore, it is crucial to have a proper synchronization among different nodes/clusters of distributed systems to enhance the accuracy of the ML training model and to accelerate the training time. The synchronization cost may result in a significant performance loss of a distributed ML model [201]. In general, to parallelize the data across distributed clusters, ML techniques utilize BSP strategy [202], in which all computational nodes need to commit and receive new global parameters before starting the next iteration, resulting in a load imbalance problem. This load imbalance problem can be addressed with an asynchronous strategy as it enables the distributed learning nodes to utilize local model parameters for the next iteration [203]. The main problem with this asynchronous method is that the model may not provide the accuracy guarantee since the model can be trapped in a local optimum without converging to a globally optimum solution.   Another approach is SSP strategy [204], in which the modes can utilize the stale global parameters to train the local model; however, this may not guarantee convergence due to the limitations of stale global parameters. A promising approach to address the drawbacks of the methods above is to dynamically adapt the communications method between the centralized/parameter server and the distributed nodes based on the performance of each node. In this regard, the ASP strategy proposed in [201] has been shown to achieve higher convergence speed and provide better accuracy than the SSP methods. However, the applicability of the ASP method in a large-size distributed ML framework remains an open problem.

The emerging cyber-physical systems requiring time synchronization are vulnerable to the threat of time synchronization attacks, which mainly focus on modifying the measurements' sampling time/time stamps without modifying the system measurements. Such time synchronization attacks may lead to serious consequences in cyber-physical systems, such as incorrect voltage stabilization in smart grid networks. The existing attack detection techniques such as residual-based bad data detection and the conventional supervised ML-based detectors may not be able to effectively detect such attacks, leading to the need for innovative ML-based solutions [205]. In this regard, "first difference aware" ML classifier proposed in [205] could be promising to detect two types of time synchronization attacks, namely, direct time synchronization, which only modifies some time stamps, and stealth time synchronization attack, which modifies all the timestamps at a certain time. The First Difference ML (FDML) techniques utilize the backward first difference of the time-series data in order to process the input data stream before employing an ML method.

For a satellite system with a DRT working in the FH-FDMA mode, it is common to use different hopping sequences for the uplink and downlink communications to avoid possible jamming or interference in both the links [206]. However, the synchronization between DRT and ground equipment with different hopping sequences becomes very complicated, and it is a crucial challenge to investigate an efficient synchronization method. In this regard, authors in [207] proposed an ML-based novel method to carry out synchronization with the Frequency Hopping signal for tactical SatCom system by utilizing serial search for coarse acquisition and LSTM network for fine acquisition. The main objective of the proposed work is to reduce the synchronization time. It has been shown that the proposed LSTM-based method enables the fast and quick fine acquisition in comparison to the existing methods in the literature and provides benefits in saving the overall synchronization time by allowing the fine acquisitions for both downlink and uplink and learning the temporal trend of the signal.

In the context of 5G integrated SatCom utilizing OFDM, authors in [208] proposed a Cyclic Prefix based multi-symbol merging blind timing algorithm to enhance the timing accuracy. Also, an improved synchronization method has been submitted to realize more accurate time-frequency error correction in the considered 5G integrated SatCom system. With the help of simulations results, it has been depicted that the proposed multi-symbol merging method provides better performance than the single-symbol method in low SNR conditions, and the proposed synchronization method can provide a good Bit Error Ratio (BER) performance.

Due to the availability of massive volumes of high-resolution videos and images taken by LEO satellites and UAV, many data sources have become available for SIN. However, the application of ML becomes challenging due to limited computing and communication resources and also the low frequency and orbit resources in SIN. Also, the unstable connections of  SIN further create the challenges of employing ML among a swarm of satellites and UAV. To address this, authors in [209] considered the application of distributed ML in SIN, called SpaceDML, to effectively reduce the communication overhead among the SIN devices. The proposed SpaceDML utilizes adaptive loss-aware quantization and partial weight averaging algorithms to compress models without sacrificing their quality and selectively average the active agents' partial model updates, respectively. The evaluation with public dataset and realistic model presented in  [209] has demonstrated that the proposed SpaceDML can enhance the model accuracy by about 2-3 % and reduce the communication burden among SIN devices can be up to 60 % as compared to the baseline algorithm.

§.§ Summary and Lessons Learnt

* ML can be considered as a promising technique to address several
synchronization problems related to frame synchronization, compensation of the errors caused due to sampling frequency/time offsets, carrier synchronization, and characterization of phase noise in communication systems, including a SatCom network.

* For distributed satellite systems, ML techniques can be used to accurately characterize amplitude and phase noises, which are essential parameters in the synchronization process of a distributed satellite system.

* Distributed learning techniques find significant importance in distributed satellite systems as they enable the distributed implementation of ML models across several distributed nodes in a network.

* Synchronization cost may result in a significant performance loss in a distributed ML implementation, and it is imperative to have a proper synchronization among different nodes/clusters of a distributed system to enhance the accuracy of the ML training model and to accelerate the training time.

* Time synchronization attack is one important problem to be considered in the emerging cyber-physical systems involving a distributed satellite system as it may lead to severe consequences in the overall system operation. The conventional supervised ML-based detectors may not be able to effectively detect such attacks, leading to the need for innovative ML-based solutions such as "first difference aware" ML classifier proposed in [205].

* Synchronization between DRT and ground equipment with different hopping sequences in an FH-FDMA-based satellite system becomes very complicated, and ML-based synchronization techniques such as LSTM seem to be promising as illustrated in [207].

* ML techniques could also be promising to enhance timing accuracy in 5G integrated satellite system [208] and to reduce the communication burden among different distributed SIN nodes in emerging SIN with the help of distributed ML [209].

§ RESEARCH CHALLENGES AND OPPORTUNITIES