# DCP-Net: A Distributed Collaborative Perception Network for Remote Sensing
Semantic Segmentation

Zhechao Wang, Peirui Cheng, Shujing Duan, Kaiqiang Chen, Zhirui Wang, ,
Xinming Li, Xian Sun This work was supported by the National Nature Science
Foundation of China under Grant 62171436, Grant 62076241, and Grant 62201550.
(Corresponding author: Zhirui Wang.)Zhechao Wang, Shujing Duan and Xian Sun
are with the Aerospace Information Research Institute, Chinese Academy of
Sciences, Beijing 100190, China, also with the Key Laboratory of Network
Information System Technology (NIST), Aerospace Information Research
Institute, Chinese Academy of Sciences, Beijing 100190, China, also with the
University of Chinese Academy of Sciences, Beijing 100190, China, and also
with the School of Electronic, Electrical and Communication Engineering,
University of Chinese Academy of Sciences, Beijing 100190, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>sunxian@aircas.ac.cn).Peirui Cheng, Kaiqiang Chen, Zhirui Wang and Xinming Li
are with the Aerospace Information Research Institute, Chinese Academy of
Sciences, Beijing 100094, China, and also with the Key Laboratory of Network
Information System Technology (NIST), Aerospace Information Research
Institute, Chinese Academy of Sciences, Beijing 100190, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>13911729321@139.com).

###### Abstract

Onboard intelligent processing is widely applied in emergency tasks in the
field of remote sensing. However, it is predominantly confined to an
individual platform with a limited observation range as well as susceptibility
to interference, resulting in limited accuracy. Considering the current state
of multi-platform collaborative observation, this article innovatively
presents a distributed collaborative perception network called DCP-Net.
Firstly, the proposed DCP-Net helps members to enhance perception performance
by integrating features from other platforms. Secondly, a self-mutual
information match module is proposed to identify collaboration opportunities
and select suitable partners, prioritizing critical collaborative features and
reducing redundant transmission cost. Thirdly, a related feature fusion module
is designed to address the misalignment between local and collaborative
features, improving the quality of fused features for the downstream task. We
conduct extensive experiments and visualization analyses using three semantic
segmentation datasets, including Potsdam, iSAID and DFC23. The results
demonstrate that DCP-Net outperforms the existing methods comprehensively,
improving mIoU by 2.61% $\sim$ 16.89% at the highest collaboration efficiency,
which promotes the performance to a state-of-the-art level.

###### Index Terms:

Collaborative perception, distributed neural network, semantic segmentation,
remote sensing.

##  I Introduction

Intelligent remote sensing platforms, such as satellites and drones, have
demonstrated exceptional performance in onboard real-time processing tasks,
which has been applied in disaster detection [1], maritime monitoring [2, 3],
water-body extraction [4] and urban mapping [5, 6]. With the widespread
deployment of intelligent remote sensing platforms, these platforms possess
the capability of collaborative observation [7, 8] and swarm internal
communication [9, 10], laying the foundation for collaborative perception.

Collaborative perception in remote sensing refers to the information sharing
among multiple intelligent platforms to enhance the perceptual ability, as
depicted in Fig.1. Multiple intelligent remote sensing platforms form a
collaborative group to jointly observe an identical scene from different
angles. Within this group, platforms can share their local information to
expand the scope of observation and leverage the collaborative information to
compensate for the limitations of individual platform perception. Given the
numerous benefits of collaborative perception, it is crucial to further
develop this technology in the field of remote sensing.

Figure 1: Illustration of multiple remote sensing platforms’ collaborative
perception. The members of the collaborative group can enhance their local
perception with the help of collaborative information from other platforms.

However, there exist several significant challenges that need to be addressed
while implementing collaborative perception in remote sensing. Firstly, high-
speed mobility [11, 12] and long-distance transmission [13] contribute to
limited bandwidth [14, 15] and weak communication [16] among platforms, posing
obstacles to the interaction in collaborative perception. Consequently, this
can lead to performance degradation and even complete interaction failure.
Although some collaborative methods alleviate this issue by reducing the
amount of transmitted data through the utilization of processed features
rather than raw data [17], [18], the frequent transmission of data streams and
the indiscriminative fully connected interaction mode remain insufficient for
the weak communication in remote sensing. Secondly, the multi-angle
characteristic of remote sensing introduces variations in observations across
different platforms for an identical scene, resulting in information
misalignment between platforms. Given that targets in remote sensing scenes
are typically small and densely distributed, simply adding or concatenating
features from different platforms without considering their potential
distinctions at the same region can interfere with original observations and
deteriorate the perception performance [19, 20].

As to the first issue, we recognize that not all collaboration is necessary,
and not all collaborative information is equally crucial. Therefore, we adopt
a dynamic on-demand manner rather than relying on frequent fully connected
interaction. This approach establishes collaboration only when necessary and
prioritizes the transmission of crucial collaboration information, reducing
the frequency and volume of transmission while enhancing interaction
efficiency during collaborative perception. As a result, it better adapts to
the prevalent weak communication environment commonly encountered in remote
sensing scenarios. Regarding the second issue, exploring the correspondence
between observations from different perspectives can effectively mitigate the
interference caused by feature misalignment on the original local features
during collaboration. Consequently, when filtering the collaborative features
among different platforms, we establish their correlation with the guidance of
the local features. This approach facilitates the extraction of relevant
information and ensures the alignment and integration of features, thus
guaranteeing the overall performance of collaborative perception.

To implement the solutions mentioned above, this paper introduces the
distributed collaborative perception network (DCP-Net) for multiple remote
sensing platforms. DCP-Net realizes dynamic interactions through the design of
a self-mutual information match (SMIM) module. Based on the local information,
inter-platform relevance, and designed strategy, the SMIM module evaluates the
necessity of collaboration and identifies suitable platforms for
collaboration, effectively avoiding redundant interactions and minimizing the
interference caused by low-correlation information. This process contributes
to relieving the transmission burden in weak communication while maintaining
the perception performance. Furthermore, this paper presents a related feature
fusion (RFF) module to overcome the challenge of information misalignment. The
RFF module models the relationship between local and collaborative features
and selectively integrates locally required information for feature fusion.
This approach effectively mitigates the interference caused by misaligned
features. With the implementation of these designed modules, DCP-Net
facilitates collaborative perception among remote sensing platforms with low
transmission amount and high-quality fusion.

In conclusion, the main contributions of this article can be summarized as
follows:

1) This paper presents an innovative approach called DCP-Net for collaborative
remote sensing semantic segmentation. Our DCP-Net enhances the overall
perception performance of platforms through collaboration and achieves state-
of-the-art results on Potsdam, iSAID and DFC23.

2) A SMIM module is designed to reduce redundant transmission and avoid
irrelevant interference, thereby improving adaptation to the weak
communication in remote sensing. This module determines the opportunities and
partners for collaboration, achieving a balance between perception enhancement
and communication overhead.

3) This paper presents an RFF module to address the issue of information
misalignment and ensure the quality of the fused feature representation. This
module is responsible for feature filtering and facilitates the effective
integration of both local and collaborative features.

4) To the best of our knowledge, this is the first paper to investigate
collaborative perception through feature interactions among multiple platforms
in the field of remote sensing.

##  II Related work

###  II-A Collaborative Perception

Collaborative perception is a burgeoning application for multi-agent systems,
where agents integrate local observations with those of neighboring agents in
a learnable manner to enhance accuracy in perception tasks. This field has
gained significant attention, and several works have been established to
support and advance research in this area.

Liu et al. [19] are the first to propose the concept of collaborative
perception and introduce a multi-stage handshake communication mechanism
called Who2com. This mechanism allows neural networks to learn how to compress
relevant information for each stage of communication. They also develop a
simulated dataset using the AirSim simulator, which is perceived by a group of
aerial robots. When2com [20], as an upgraded version of Who2com, reformulates
the communication framework by learning how to construct communication groups
and determine the optimal timing for communication. The generalizability of
When2com is demonstrated in tasks such as multi-agent 3D shape recognition and
collaborative semantic segmentation. Hu et al. [21] introduce Where2com, which
aims to optimize communication efficiency by conveying spatially sparse but
perceptually essential information. They demonstrate the effectiveness of
Where2com in the task of 3D vehicle detection with two modalities of camera
and LiDAR. Zhou et al. [22] adopt a general-purpose Graph Neural Network to
improve single-robot perception accuracy and resilience to sensor failures and
disturbances in multi-robot monocular depth estimation.

In some domains of cross-applications, collaborative perception is
increasingly embraced. A collaborative perception framework called Swarm-SLAM
[23] is proposed to tackle the challenge of sharing situational awareness
among multiple robots operating in GPS-absent conditions, simultaneously
achieving localization and mapping tasks. The primary objective of this
framework is to identify inter-robot map links and utilize them to merge the
individual local maps, ultimately creating a unified global understanding of
the environment. In the realm of few-shot learning, FS-MAP [24] is presented
as a metric-based framework for air-ground collaboration. The framework
incorporates multiple UAVs dedicated to collecting few-shot face samples,
along with a self-driving campus delivery vehicle for target queries. It
operates by first sending face queries from the vehicle to the UAVs for
matching. Next, the UAVs send back similarity scores to the delivery vehicle,
which then the vehicle sorts and filters the scores to determine the
geographic location of the target for subsequent path planning. In the field
of game AI agent learning, Nash et al. [25] introduce a novel paradigm called
Herd’s Eye View, inspired by the concept of the Bird’s Eye View in autonomous
vehicles. It employs cooperative perception to enhance the decision-making of
RL agents by providing them with global reasoning abilities. In contrast to
previous collaborative perception approaches, the method combines
reinforcement learning to address both low-level control tasks and high-level
planning challenges simultaneously in complex, procedurally generated
environments, demonstrating superior performance compared to traditional ego-
centric perception models. While there is limited research on collaborative
perception in remote sensing, Gao et al. [26] mention the potential of multi-
platform networking and collaboration in modern earth observation systems.
This collaboration can lead to acquiring earth observation data with higher
accuracy, advanced information dimensions, and higher space-time resolution
compared to current systems. With the proliferation of satellite
constellations and unmanned aerial vehicle swarms, numerous applications exist
for exploring collaborative perception.

###  II-B Remote Sensing Semantic Segmentation

Semantic segmentation in the field of natural scenes has advanced
significantly with deep neural networks. Long et al. [27] pioneer the
application of deep neural networks to semantic segmentation, replacing fully
connected layers with convolution layers and incorporating multi-scale feature
fusion techniques. U-Net [28] introduces a decoder that performs feature
upsampling and incorporates features from neighboring layers through
concatenation. This innovative design aims to effectively recover lost spatial
information during the process of semantic segmentation. Chen et al. [29]
propose the DeepLab series, incorporating dilated convolutions to capture
multi-scale contextual information. In addition to the CNN-based methods
mentioned above, there are also novel approaches that leverage transformer-
based architectures. Zheng et al.[30] propose SETR that replaces the
traditional CNN encoder with the vision transformer and employs a progressive
up-sampling and multi-level feature aggregation for relieving the noise during
the overall process. SegFormer [31] utilizes a hybrid architecture that
combines transformers with a lightweight multilayer perceptron decoder.

The methods employed for semantic segmentation in remote sensing are typically
derived from those utilized in natural scenes and subsequently tailored to
specific application scenarios. Mou et al. [32] apply relation-augmented
feature representations into FCNs in order to capture long-range spatial
relationships among entities in satellite images. Diakogiannis et al. [33]
introduce ResUNet, a novel approach that integrates the advantages of ResNet,
U-Net, and ASPP architecture to effectively handle objects with diverse sizes
in remote sensing imagery. Niu et al. [34] incorporate graph reasoning and
disentangled learning to the common decoder architectures to improve the
localization and precision of the segmentation results. DisOptNet [35] adopts
knowledge distillation between multi-source aerial images to enhance the
spatial consistency and contextual understanding of the segmentation results.
A novel attention-based framework, HMANet [36] captures global correlations
adaptively within the aspects of spaces, channels, and categories more
effectively and efficiently. ST-UNet [37] demonstrates the great potential of
the UNet-like transformer decoder in effectively modeling global information
for urban scene segmentation. These methods mentioned above aim to realize a
high-performance semantic segmentation of remote sensing imagery in a
centralized data center. For the sake of simplicity and practicality, DCP-Net
employs a lightweight FCN-based decoder to facilitate onboard intelligent
processing across multiple platforms.

##  III Method

###  III-A Overview

The proposed Distributed Collaborative Perception Network is depicted in
Fig.2, consisting of four modules, each with its specific role: feature
extraction, collaboration establishment, multi-platform feature fusion and
downstream prediction. The overall process can be described as follows,
divided into four steps:

Firstly, a lightweight backbone, ResNet-18 [38], is adopted as the feature
extractor. Each platform inputs observations from various angles of the
identical scene into the backbone, generating features with generally similar
content but localized differences in detail.

Secondly, leveraging the features generated by the fourth layer of the feature
extractor, the SMIM module calculates a self-information confidence score
locally to determine the collaboration request. Then, based on the feature
relationships between different platforms, it interactively generates mutual-
information match scores for the selection of collaborative platforms.

Thirdly, the RFF module captures the correlation between local features and
those from the selected collaborative platform. It filters the locally
required features and facilitates the fusion of misaligned features.

Finally, the fused features are fed into the downstream decoder, generating
predicted results based on collaborative perception.

Figure 2: The framework of Distributed Collaborative Perception Network.
Within this framework, the platform that requires collaboration is referred to
as a requester. The platforms involved in the collaborative network, excluding
the requester, are termed candidates. Among the candidates, the selected
platform that acts as a perception helper is referred to as the supporter. In
the given scenario, platform 1 acts as the requester, while the remaining
platforms serve as candidates for collaboration. Through the SMIM module,
platform 4 is chosen as the supporter, responsible for providing collaborative
features.

###  III-B Self-Mutual Information Match Module

A reliable information collaboration strategy should be able to ascertain the
necessity for request and select appropriate collaborative platforms,
precisely as the description in Fig.3. Additionally, the interaction cost must
be taken into account. To address these concerns, we design a SMIM module that
evaluates the need for collaboration based on the sufficiency of local
information and identifies suitable supporters based on the relevance of
supplementary features from the candidates. This guarantees the optimization
of perceptual gains while also reducing communication expenditures. The
components of the SMIM module are depicted in Fig.4.

Figure 3: The schematic diagram of the SMIM module’s function. Each remote
sensing platform in the collaborative network independently makes autonomous
decisions based on its local observations. For instance, platform 1 seeks
collaboration by sending requests to other platforms and selects a suitable
supporter based on the correlation received in response. Conversely, platforms
2, 3, and 4 carry out the perception task individually.  Figure 4: The process
framework of SMIM module. The module is divided into two stages: self-
information match, depicted in the left part, and mutual-information match,
depicted in the right part.

Initially, platform $i$ encodes local features $x_{i}$ into query $q_{i}$ and
key $k_{i}$ vector as the inputs of the self-information match stage:

$\displaystyle q_{i}=Q\left(x_{i};\theta_{q}\right)\in\mathbb{R}^{q};i\in
1,\ldots,N;$ (1) $\displaystyle
k_{i}=K\left(x_{i};\theta_{k}\right)\in\mathbb{R}^{k};i\in 1,\ldots,N,$

where $Q$ and $K$ refer to the query and key encoders, parameterized by
$\theta_{q}$ and $\theta_{k}$, respectively. Notably, the collaborative group
comprises a total of N platforms. Additionally, the vector space of query and
key is represented by $\mathbb{R}$.

Subsequently, the dot-product self-attention mechanism, denoted by $\varphi$,
is utilized to calculate the correlation $c_{i}$ between $q_{i}$ and $k_{i}$.
This correlation indicates the amount of local information for agent $i$.

Then, through the sigmoid activation, the correlation is compressed into the
interval $\left[0,1\right]$ to generate the self-information confidence score
$p_{i}$. This score represents the probability that the platform does not
require collaboration:

$\displaystyle\text{$p_{i}$}=\mathit{Sigmoid}\left(c_{i}\right)=\mathit{Sigmoid}\left(q_{i}^{T}k_{i}\right)$
(2) $\displaystyle=\frac{1}{1+e^{-q_{i}^{T}k_{i}}};i\in 1,\ldots,N.$

During the inference phase, a predefined request threshold is set to determine
whether collaboration is necessary. If $p_{i}$ exceeds the threshold, it
indicates that the local information is sufficient for the perceptual task
independently without collaboration. This approach conserves communication
resources by avoiding unnecessary requests for subsequent interactions.

$\displaystyle\text{$p_{i}$}=\mathit{Sigmoid}\left(c_{i}\right)=\mathit{Sigmoid}\left(q_{i}^{T}k_{i}\right)$
(3) $\displaystyle=\frac{1}{1+e^{-q_{i}^{T}k_{i}}};i\in 1,\ldots,N.$

When platform $i$ seeks assistance, it generates a compact request vector
$r_{i}$ by condensing the required features into a lower dimension $r$, which
is significantly smaller than $k$, the feature dimension of the key generated
in the self-information match stage:

$r_{i}=R\left(x_{i};\theta_{r}\right)\in\mathbb{R}^{r};i\in 1,\ldots,N,$ (4)

where $R$ denotes the request encoders, parameterized by $\theta_{r}$. This
compression results in a significant reduction in transmission cost. Then
platform i broadcasts its request to the candidates for collaboration.

In the mutual-information match stage, the SMIM module calculates the
relevance between each candidate and requester to select the appropriate
supporter. By re-utilizing the key $k_{j}$ generated in the self-information
match stage, candidate $j$ can directly respond to the request $r_{i}$ from
the requester, platform $i$, without incurring repeated feature encoding. To
handle inconsistent vector dimensions, the request $r_{i}$ is first projected
onto the feature space with the same dimension as the key $k_{j}$ using the
projection matrix $W_{\alpha}$. After the dot product operation, each
candidate then feeds back the relevance to the requester, which is
subsequently normalized to obtain the mutual-information match score $s_{ij}$.
Intuitively, a higher score $s_{ij}$ indicates that the candidate (platform
$j$) can provide more informative information to the requester (platform $i$).

$s_{ij}=\mathit{Softmax}\left(r_{i}^{T}W_{\alpha}k_{j}\right)=\frac{e^{r_{i}^{T}W_{\alpha}k_{j}}}{\sum_{j=1,j\neq
i}^{N}e^{r_{i}^{T}W_{\alpha}k_{j}}}$ (5)

In order to prioritize more helpful platforms for the requester and avoid low-
yield interactions, a collaboration threshold is set to $\frac{1}{N-1}$.
During the inference stage, the candidates whose mutual-information match
scores exceed the collaboration threshold are designated as supporters and
subsequently engaged in perceptive interaction with the requester.

###  III-C Related Feature Fusion Module

Due to discrepancies in observation scope and perspective among various
platforms, there exists the problem of inconsistent feature representation
within the same scene. The naive concatenation or addition of feature fusion
can contribute to feature misalignment. This misalignment disrupts the
original observation information and ultimately triggers a decrease in
perception accuracy. To this end, inspired by Non-local Neural Networks [39],
our paper proposes a RFF module that utilizes the requester’s features as a
reference to select the correlated portion of the collaborative feature for
fusion. By utilizing both the requester’s and supporter’s features, along with
the self-information confidence score and mutual-information match score as
inputs in this module, inter-dependencies between the elements of both feature
sequences are captured for more effective and accurate feature fusion. The
implementation details for the RRF module mentioned above are exhibited in
Fig.5.

The generic calculation of related features is listed as follows:

$\displaystyle F^{r}=\frac{1}{N(F)}\sum
h\left(F^{l},F^{c}\right)g\left(F^{c}\right)$ (6)
$\displaystyle=\frac{1}{N(F)}\theta\left(F^{l}\right)\varphi\left(F^{c}\right)^{T}g\left(F^{c}\right),$

where $F^{l}$, $F^{c}$ and $F^{r}$ represent local features, collaborative
features and related features, respectively. The pairwise function $h$
computes the relationship between $F^{l}$ and $F^{c}$, and the function $N(F)$
normalizes the relationship to obtain the affinity matrix. Ultimately, the
related feature $F^{r}$ is generated by the calculated affinity matrix and
collaborative feature $F^{c}$. As to the concrete operation, the
representation of input features is computed using linear embedding functions
$\theta$, $\varphi$, and $g$, implemented as $1\times 1$ convolution.
Moreover, $h$ is implemented as a dot product manner and softmax is selected
as the normalization function.

The specific steps of conducting the RFF module are demonstrated below.

Firstly, the local features $F^{l}$ and collaborative features $F^{c}$ are
embedded by their respective encoders $\theta$ and $\varphi$, projecting the
channel-wise dimension from $C$ to $C^{\prime}$, $C^{\prime}<<C$. This
reduction in dimension reduces the complexity of subsequent affinity matrix
computation. Additionally, the collaborative features are embedded by encoder
$g$ and fixed in the original dimension for subsequent feature selection.

$\displaystyle\theta\left(F^{l}\right),\varphi\left(F^{c}\right)\in\mathbb{R}^{H\times
W\times C^{\prime}},g\left(F^{c}\right)\in\mathbb{R}^{H\times W\times C}$ (7)

Secondly, a global cross-attention operation is performed on the flattened
embeddings $\theta(F^{l})$, $\varphi(F^{c})$, calculating the relationship
between each feature vector of the supporter and requester, and achieving
fine-grained feature match. The resulting relationship is then normalized
using softmax to obtain the affinity matrix $A\in\mathbb{R}^{HW\times HW}$,
which represents the relationship between the entire feature sequences.

$\displaystyle
A=\mathit{Softmax}\left(\theta\left(F^{l}\right)^{T}\varphi\left(F^{c}\right)\right)=\left[A_{i,j}\right]_{HW^{*}HW}$
(8) $\displaystyle
A_{i,j}=\frac{\theta\left(F^{l}\right)_{i}{}^{T}\varphi\left(F^{c}\right)_{j}}{\sum_{j\in
HW}\theta\left(F^{l}\right)_{i}^{T}\varphi\left(F^{c}\right)_{j}}\in\mathbb{R}^{1\times
HW}$

Thirdly, the related feature $F^{r}=A\times
g\left(F^{C}\right)\in\mathbb{R}^{H\times W\times C}$ is obtained with the
affinity matrix and embedded collaborative features.

Ultimately, the fused features $O^{f}$ are obtained by integrating local
features with related features, along with the self-information confidence
score and mutual-information match score determined by the SMIM module. This
fused feature is then inputted into the downstream decoder to make
predictions.

$O_{i}=p_{i}\cdot
F_{i}^{l}+\left(1-p_{i}\right)\cdot\mathit{request}_{i}\cdot\sum_{j=1,j\neq
i}^{n}s_{ij}\cdot F_{j}^{r}$ (9)

Figure 5: The exhibition of RFF module. The reshape operation converts the
image features into a sequence of features. In addition, the complementary
operation can be simply regarded as subtracting the input from 1.

###  III-D Training Strategy

Throughout the training process, it is necessary to instruct the platform on
identifying the collaboration opportunities and selecting the appropriate
supporter, which is a dynamic decision problem. Reinforcement learning is a
widely employed approach to tackle such dynamic strategies [40, 41]. However,
its effectiveness heavily depends on the reward mechanism design, presenting a
significant challenge in collaborative perception. In contrast to intricate
reinforcement learning methods, our proposed DCP-Net optimizes collaborative
strategies solely based on the supervision of the ground truth from the
downstream task.

To implement the collaboration strategy, DCP-Net utilizes centralized training
and distributed inference. During the training process, DCP-Net combines the
features from all platforms and quantitatively evaluates the effect of each
candidate’s features on perception enhancement. In the inference stage, the
principles designed in the SMIM module dynamically filter out unnecessary
collaboration opportunities and candidates. In this way, the requester learns
to make decisions that maximize downstream prediction improvements by
assessing local observations and interacting with the candidates.

Additionally, our experiments choose semantic segmentation as the downstream
task and utilize cross-entropy loss for optimization. Throughout the
iterations of loss minimization, DCP-Net continuously promotes its ability to
select information and fuse features, ultimately achieving peak performance in
collaborative perception. The overall loss function in our collaborative
perception network during the training process is:

$\displaystyle\mathit{Loss}$
$\displaystyle=L(y_{i},\hat{y_{i}})=L(\mathit{decoder}(O_{i}),\hat{y_{i}})$
(10) $\displaystyle=L(\mathit{decoder}(p_{i}\cdot
F_{i}^{l}+(1-p_{i})\sum_{j=1,j\neq i}^{n}{s_{ij}}\cdot
F_{j}^{r}),\hat{y_{i}}).$

The analysis below delves into the reasons for DCP-Net’s ability to achieve
interactive strategy training solely through downstream supervision. The
objective function optimization aims to minimize the loss in the downstream
task, which heavily relies on the quality of the fused feature provided to the
decoder. In other words, minimizing the loss corresponds to obtaining the
optimal fused features while enhancing the feature quality primarily origins
from integrating collaborative features during collaborative perception. The
ideal fused features should follow a strategy that promotes perception while
minimizing redundancy and interference. Consequently, in the SMIM module, when
additional information is required to enhance perception, the calculated self-
information confidence score is significantly lower, and the mutual-
information match score assigned to candidates with greater perception
improvement increases substantially. Additionally, the RFF module extracts
more relevant and helpful information from collaborative features to achieve
better feature fusion.

It is worth noting that the collaborative strategy supervised solely based on
the ground truth of the downstream task provides convenience in terms of
dataset availability. This approach eliminates the need for human intervention
in identifying optimal collaboration opportunities and partners, thus reducing
the challenge of sample annotation.

##  IV Experiments

This section is organized into four parts. Firstly, the methodology for
constructing datasets for multi-platform collaborative perception is
presented. Secondly, an overview of the experimental environment is provided.
Thirdly, the data evaluation method employed in this article is explained.
Lastly, the pertinent experimental details are outlined.

###  IV-A The Construction of Datasets

We evaluate DCP-Net through collaborative semantic segmentation tasks on
simulated multi-platform datasets for aerial and satellite remote sensing,
respectively, as well as a real joint observation dataset, DFC23 [42].

Due to the current scarcity of datasets for multi-platform collaborative
observations, we generate analogous multi-platform datasets by utilizing
existing datasets, including ISPRS Potsdam [43] and iSAID [44]. This process
is accomplished by following the steps outlined below:

Firstly, a full-size remote sensing image is cropped into multiple subsets of
size 1024$\times$1024 by sliding windows. Secondly, four observation views of
size 512$\times$512 are randomly cropped from each 1024$\times$1024 subset. By
implementing these steps, a collection of images with overlapped observation
ranges of identical scenes are obtained, forming the simulated datasets. The
Fig.6, 7 below visually illustrate the resulting simulated datasets.

Figure 6: Simulated multi-platform observation dataset of ISPRS Potsdam.
Figure 7: Simulated multi-platform observation dataset of iSAID. Figure 8:
Multi-platform joint observation dataset of DFC23.

The real multi-platform observation dataset is created with the help of DFC23,
jointly collected by the SuperView-1 and Gaofen-2 satellites. The dataset
primarily comprises urban building clusters for collaborative semantic
segmentation [45], aimed at assessing the practical effectiveness of various
methods. The Fig.9 depicts the full-scale observation captured by SuperView-1
and Gaofen-2. Notably, differences exist in observation perspective, imaging
characteristics, and resolution, making the dataset highly challenging and
reflective of real-world scenarios.

Figure 9: The images taken by SuperView-1 and Gaofen-2 are presented in the
row 1 and row 2, respectively.

As displayed in Fig.8, column 1 presents the image captured by SuperView-1,
while columns 2 to 4 showcase images captured by Gaofen-2. The image in column
2, taken at the same geographic coordinate as column 1, exhibits significant
differences in observation perspective, observation range and imaging payload.
The nearby captured images, listed in columns 3 and 4, serve as similar
interferences to verify DCP-Net’s ability to filter out unnecessary additional
information.

We have examined three experimental modes, categorized as homogeneous complete
information supplement (Homo-CIS), homogeneous partial information supplement
(Homo-PIS) and heterogeneous partial information supplement (Hetero-PIS).
Details for each mode are outlined below.

The Homo-CIS mode aims to verify the accuracy of collaboration establishment
and supporter selection. The term ”homogeneous” implies that all platforms
within the collaborative network possess the same imaging payload, while
”complete” means that the requester can obtain all the necessary information
comprehensively. In this mode, one platform suffers from imaging degradation,
and its original, noise-free observation randomly appears among the other
members within the group. Out of the four platforms, only one is selected as a
potential victim of degradation. We introduce noise into 50% of the images
captured by this platform and randomly replace another platform’s observations
with the non-degraded original images. As depicted in Fig.12, column 1
displays the degraded image captured by the selected platform. Columns 2 and 3
show the observation information from other platforms, while column 4 presents
the original, noise-free observation image. Furthermore, it’s worth noting
that only the segmentation mask of the potentially degraded platform is used
as supervision during the training process. An ideal goal of the Homo-CIS mode
is that the platform is capable of recognizing the degradation of local
observations and utilizing the original information provided by other
platforms to compensate for its perception.

In the Homo-PIS mode, we eliminate the assumption that there exists original,
noise-free observation of the degraded platform among the partners within the
group. Due to differences in observation range among various remote sensing
platforms, only the partial overlap of observations conforms to practical
scenarios. In this mode, we investigate the improvement in perception with the
help of supporters that only cover partially overlapped views. The degraded
platform can only collaborate with candidates whose mutual-information match
score exceeds the collaboration threshold. As depicted in Fig.12, the image in
column 1 is marred by imaging noise, thereby leading to a degraded
observation. In contrast, the images in the remaining columns offer clear and
detailed representations of the overlapped scene. The ultimate goal of the
Homo-PIS mode is to improve the perception of the degraded platform by
effectively utilizing the partial-overlapped collaborative features available
within the collaborative network.

Compared to the Homo-PIS mode, the Hetero-PIS mode is more complex, as it
takes into account inconsistent imaging payloads among different platforms.
The heterogeneity in payloads contributes to domain discrepancy, posing
challenges in integrating diverse sources of information. In this mode, we
utilize observations from multiple satellites, each equipped with unique
imaging capabilities, perspective biases, and range limitations. As shown in
Fig.12, the images in columns 2-4 sourced from Gaofen-2 are used to
collaborate with the degraded observations in column 1 sourced from
SuperView-1.

Figure 10: Visualization of the Homo-CIS mode.

Figure 11: Visualization of the Homo-PIS mode.

Figure 12: Visualization of the Hetero-PIS mode.

###  IV-B Implementation Details

In our experimental setup, ResNet-18 is employed as the feature backbone for
DCP-Net, alongside other baselines. For the segmentation task, FCN is served
as the decoder. Additionally, the training and validation subsets of the
datasets are divided as outlined in Table I. In terms of optimization, the
Adam optimizer is utilized with coefficient values of $\beta$ set to 0.9 and
0.999. The models undergo 50 epochs of training with a learning rate of
$5\times 10^{-5}$. All experiments are conducted on an RTX-3090 GPU with
PyTorch version 1.7.1.

dataset | classes | train | val
---|---|---|---
Potsdam | 6 | 7200 | 2800
iSAID | 16 | 19790 | 6289
DFC23 | 2 | 3688 | 1752
TABLE I: Split three datasets into the train set and validation set.

###  IV-C Evaluation Metrics

Our study conducts evaluations of all models in the collaborative segmentation
task, taking into account three key aspects: mean IoU, communication expense,
and collaboration efficiency.

Mean Intersection over Union (mIoU). It is a widely used evaluation metric in
image segmentation, quantifying the overlap between predicted and ground truth
targets. IoU is computed by dividing the intersection area of the predicted
and ground truth regions by their union, ranging between 0 and 1. MIoU
represents the average IoU across multiple targets and offers a concise and
comprehensive assessment of algorithm performance on diverse targets,
facilitating comparisons between different algorithms. A higher mIoU indicates
superior target localization and segmentation capabilities.

Communication Cost (Comm. Cost). This metric considers the average amount of
communication in the interactions of the collaborative perception task.
Additionally, we employ an indicator called MBpf (MegaBytes per frame) to
quantitatively measure the communication cost. The MBpf is determined by both
the frequency of collaboration establishment and the number of transmitted
features.

Collaboration Efficiency (CE). It is crucial to comprehensively consider the
metrics mentioned above to strike a balance between perception enhancement and
interaction cost in collaborative perception. Accordingly, the concept of CE
is introduced as a measure of this balance:

$CE=\frac{\Delta mIoU}{Comm.\ Cost}.$ (11)

CE is defined as the ratio of the improvement in accuracy achieved in the
collaborative semantic segmentation to the communication cost incurred during
the interaction. Consequently, a higher value of $\Delta$mIoU and a lower
Comm. Cost is desirable in collaborative perception, as it signifies a more
effective and efficient collaboration.

###  IV-D Collaborative Semantic Segmentation Experiments

We quantitatively compare the experimental results of our proposal DCP-Net
with several centralized and distributed baselines in three modes mentioned in
Section IV.A. The brief introduction of the baseline methods is as follows:

No-Interaction serves as a baseline for the independent execution of
downstream tasks without any information interaction.

Concat-All is a simple centralized model baseline that concatenates all
extracted image features from different platforms for downstream tasks.

Auxiliary-View Attention employs an attention mechanism to fuse observations
from all platforms with various weights.

Random-Selection randomly selects a platform to provide observation
information as a perception helper.

Who2com [19] utilizes an attention mechanism to select a perception helper
based on relevance each time.

When2com [20], an extension of Who2com, is able to determine the suitable
collaboration opportunity.

####  IV-D1 Homogeneous Complete Information Supplementation

In Homo-CIS mode, we investigate the improvement brought by the collaborative
perception in the background of both the same imaging payload and the
supplement of the corresponding complete original information. Table II shows
the performance of several baseline models and our proposed model. In both
simulated multi-platform Potsdam and iSAID datasets, compared to the No-
Interaction baseline, all centralized methods improved the predicted mIoU.
However, they require all observations as assistance each time, which leads to
high bandwidth consumption. In terms of the distributed manner, these methods,
except for Random-Selection, are able to approach, even surpass the
performance of centralized methods with only 1/6 to 1/3 of the communication
overhead of the centralized one. As to naive Random Selection, the platform
establishes collaboration without any strategy and filters the best perception
supporter, just like winning a lottery. Therefore, Random-Selection
contributes to a slight improvement of prediction in the simulated multi-
platform iSAID dataset but even behaves much worse than No-Interaction in the
simulated multi-platform Potsdam dataset.

Among the listed methods, our proposed DCP-Net achieves the optimal mask
prediction in the Homo-CIS mode. Our proposed DCP-Net improves the average
mIoU by 14.71% and 16.89% in the multi-platform Potsdam dataset and the multi-
platform iSAID dataset, respectively. Besides, DCP-Net consumes the least
Comm. Cost during the process of collaborative perception in this mode. In
other words, regarding the CE, DCP-Net behaves far ahead of other baselines,
realizing the ideal goal of the most perception enhancement and the least
transmission cost.

The primary objective of the Homo-CIS mode is to validate our model’s ability
to identify collaboration opportunities and perception supporters. The other
baselines, apart from Who2com and When2com, lack the capability of intelligent
dynamic interaction. Furthermore, When2com, an upgraded version of Who2com,
must establish continuous interaction among multiple platforms to determine
collaboration opportunities and supporters simultaneously. We compare our
model’s accuracy in assessing collaborative opportunities and selecting
supporters to When2com’s. Fig.13 illustrates that DCP-Net consistently
outperforms When2com in evaluating collaboration opportunities and selecting
supporters in the simulated multi-platform Potsdam and multi-platform iSAID
datasets, thereby confirming the effectiveness of the SIMM module. In contrast
to When2com, our DCP-Net assesses collaboration using a self-information
confidence score and then employs mutual-information match scores to select
supporters. This asynchronous operation sounds more reasonable and contributes
to better performance while minimizing redundant communication waste.

Figure 13: The gaps of accuracy in assessing degradation and selecting the supporter between DCP-Net and When2com in the Homo-CIS mode. Homo-CIS |  | Postdam |  | iSAID
---|---|---|---|---
|  |  | mIoU |  |  |  | mIoU |  |
Type | Method |  | Noisy | Normal | Avg. | Comm. Cost | CE |  | Noisy | Normal | Avg. | Comm. Cost | CE
individual | No-Interaction |  | 50.18 | 65.09 | 57.38 | - | - |  | 38.77 | 49.33 | 44.24 | - | -
| Concat-All |  | 55.48 | 65.92 | 60.60 | 1.500 | 2.15 |  | 44.57 | 52.20 | 48.47 | 1.500 | 2.82
centralized | Auxiliary-View Attention |  | 64.25 | 65.94 | 65.10 | 1.500 | 5.15 |  | 49.01 | 53.15 | 51.12 | 1.500 | 4.59
| Random-Selection |  | 49.58 | 60.67 | 54.98 | 0.500 | -4.80 |  | 40.03 | 51.06 | 45.73 | 0.500 | 2.98
| Who2com |  | 64.59 | 65.90 | 65.25 | 0.500 | 15.74 |  | 43.66 | 50.06 | 46.91 | 0.500 | 5.34
| When2com |  | 64.62 | 65.12 | 64.88 | 0.258 | 29.13 |  | 48.61 | 49.52 | 49.03 | 0.275 | 17.42
distributed | Ours |  | 65.14 | 66.49 | 65.82 | 0.255 | 33.10 |  | 51.45 | 52.13 | 51.71 | 0.250 | 29.88
TABLE II: Experimental results of baselines and DCP-Net in the Homo-CIS mode.
Since there is no transmission cost in No-Interaction, the value of Comm. Cost
and CE is represented as ’$-$’.

####  IV-D2 Homogeneous Partial Information Supplementation

The objective of the Homo-PIS mode is to verify the effectiveness of our
proposed method in dealing with the challenge of misaligned collaborative
features when there are only partially overlapped observations between
platforms. However, due to the complexity of the overlapped collaborative
features, accurately determining the optimal supporters becomes challenging.
Therefore, the assessment of supporter selection accuracy is not conducted in
this mode.

According to the results presented in Table III, the margin of performance
improvement for all methods is smaller with respect to No-Interaction,
compared to more significant improvement observed in the Homo-CIS mode. The
centralized methods outperform all other distributed methods, except for our
DCP-Net in the multi-platform Potsdam and multi-platform iSAID datasets. Among
the listed methods, our proposed DCP-Net achieves the optimal mask prediction
and attains the highest collaboration efficiency in both datasets. This
indicates that, unlike other distributed methods, DCP-Net still maintains a
more remarkable perception improvement while consuming less communication
cost, approximately 0.39MBpf. The experimental results quantitatively
demonstrate that in the Homo-PIS mode, DCP-Net improves the average mIoU by
6.18% and 7.34% in the multi-platform Potsdam dataset and the multi-platform
iSAID dataset, respectively.

It is worth noting that When2com incurs the lowest communication costs due to
its inability to handle the misaligned collaborative features. Additionally,
Table.III demonstrates a substantial decrease in collaboration frequency in
the multi-platform iSAID dataset and even negligible information exchange in
the multi-platform Potsdam dataset, which is a concession of maintaining the
original perception. Due to misalignment between collaborative features and
local features, a simple fusion can result in distorted observations. This
distortion makes collaboration susceptible to being treated as interference in
When2com. As to the other methods, they conduct collaborative perception in a
compulsory manner and have adapted to utilizing misaligned collaborative
features to enhance local perception. However, they are still inferior to our
proposed DCP-Net, which can be attributed to the effectiveness of our designed
RFF module.

Homo-PIS |  | Postdam |  | iSAID
---|---|---|---|---
|  |  | mIoU |  |  |  | mIoU |  |
Type | Method |  | Noisy | Normal | Avg. | Comm. Cost | CE |  | Noisy | Normal | Avg. | Comm. Cost | CE
individual | No-Interaction |  | 48.47 | 63.30 | 55.37 | - | - |  | 38.59 | 50.42 | 44.69 | - | -
| Concat-All |  | 50.56 | 64.18 | 56.92 | 1.500 | 1.03 |  | 41.83 | 51.61 | 46.89 | 1.500 | 1.47
centralized | Auxiliary-View Attention |  | 51.11 | 63.67 | 56.98 | 1.500 | 1.07 |  | 43.17 | 52.32 | 47.91 | 1.500 | 2.15
| Random-Selection |  | 48.81 | 63.60 | 55.66 | 0.500 | 0.58 |  | 39.85 | 48.84 | 44.46 | 0.500 | -0.46
| Who2com |  | 49.64 | 63.44 | 56.07 | 0.500 | 1.40 |  | 40.33 | 48.67 | 44.61 | 0.500 | -0.16
| When2com |  | 49.71 | 61.77 | 55.33 | 0.015 | -2.67 |  | 36.42 | 48.43 | 42.59 | 0.110 | -19.09
distributed | Ours |  | 53.39 | 64.84 | 58.79 | 0.388 | 8.81 |  | 45.56 | 50.59 | 47.97 | 0.385 | 8.52
TABLE III: Experimental results of baselines and DCP-Net in the Homo-PIS mode.

####  IV-D3 Heterogeneous Partial Information Supplement Mode

The objective of the Hetero-PIS mode is to test the performance of all methods
in intricate real-world scenarios. The platforms in the provided DFC23 dataset
exhibit variations in the observation angle, range, resolution, and imaging
payload, making it more challenging than the Homo-PIS mode. In this mode,
effective collaboration among heterogeneous data sources is crucial to
compensate for local ambiguous observations of building clusters and improve
prediction results.

As shown in Table IV, in comparison to the results of simulated experiments,
the perception enhancement achieved by all centralized and distributed methods
is constrained in such real-world situations. Due to the interference caused
by the similar appearance of buildings in DFC23, Auxiliary-View Attention
method, which is comparable to our proposed method in perceiving improvements
in the Homo-CIS and Homo-PIS modes, faces a dilemma in collaboration and
demonstrates inferior performance. Nevertheless, our proposed method
consistently outperforms other methods in the prediction accuracy with a
relatively low Comm. Cost of 0.355 MBpf and undoubtedly owns the highest CE.
These results demonstrate the generalizability and applicability of DCP-Net in
real-world settings.

Hetero-PIS | DFC23
---|---
Type | Method | mIoU | Comm. Cost | CE
Noisy | Normal | Avg.
individual | No-Interaction | 54.94 | 61.04 | 57.88 | - | -
centralized | Concat-All | 55.46 | 61.43 | 58.37 | 1.500 | 0.33
Auxiliary-View Attention | 55.50 | 62.22 | 58.85 | 1.500 | 0.65
distributed | Random-Selection | 55.82 | 61.58 | 58.62 | 0.500 | 1.48
Who2com | 54.28 | 61.63 | 57.91 | 0.500 | 0.06
When2com | 55.74 | 61.82 | 58.77 | 0.265 | 3.36
Ours | 56.03 | 62.81 | 59.39 | 0.355 | 4.25
TABLE IV: Experimental results of baselines and DCP-Net in the Hetero-PIS
mode.

###  IV-E Ablation Study

####  IV-E1 Designed Modules Ablation Analysis

Homo-CIS
---
Components | Postdam | iSAID
|  | mIoU |  |  | mIoU |  |
SMIM | RFF | Noisy | Normal | Avg. | Comm. Cost | CE | Noisy | Normal | Avg. | Comm. Cost | CE
|  | 50.18 | 65.09 | 57.38 | - | - | 38.77 | 49.33 | 44.24 | - | -
| $\sqrt{}$ | 65.49 | 66.36 | 65.92 | 1.500 | 5.69 | 51.65 | 52.96 | 52.24 | 1.500 | 5.33
$\sqrt{}$ |  | 64.74 | 65.87 | 65.31 | 0.255 | 31.10 | 50.11 | 51.37 | 50.69 | 0.248 | 25.98
$\sqrt{}$ | $\sqrt{}$ | 65.39 | 66.36 | 65.87 | 0.255 | 33.29 | 51.45 | 52.13 | 51.71 | 0.250 | 29.88
TABLE V: Ablation experiments on each designed module in the Homo-CIS mode.
Homo-PIS
---
Components | Postdam | iSAID
|  | mIoU |  |  | mIoU |  |
SMIM | RFF | Noisy | Normal | Average | Comm. Cost | CE | Noisy | Normal | Average | Comm. Cost | CE
|  | 48.47 | 63.30 | 55.37 | - | - | 38.59 | 50.42 | 44.69 | - | -
| $\sqrt{}$ | 56.19 | 64.54 | 60.11 | 1.500 | 3.16 | 47.20 | 50.82 | 48.95 | 1.500 | 2.84
$\sqrt{}$ |  | 49.98 | 63.48 | 56.19 | - | - | 39.93 | 49.67 | 45.03 | 0.315 | 1.08
$\sqrt{}$ | $\sqrt{}$ | 54.43 | 63.70 | 58.91 | 0.388 | 9.12 | 45.56 | 50.59 | 47.97 | 0.385 | 8.52
TABLE VI: Ablation experiments on each designed module in the Homo-PIS mode.
Hetero-PIS
---
Components | DFC23
|  | mIoU |  |
SMIM | RFF | Noisy | Normal | Average | Comm. Cost | CE
|  | 54.94 | 61.04 | 57.88 | - | -
| $\sqrt{}$ | 55.96 | 63.29 | 59.66 | 1.500 | 1.19
$\sqrt{}$ |  | 55.73 | 62.21 | 58.89 | 0.360 | 2.85
$\sqrt{}$ | $\sqrt{}$ | 56.03 | 62.81 | 59.39 | 0.355 | 4.25
TABLE VII: Ablation experiments on each designed module in the Hetero-PIS
mode.

Extensive ablation experiments are conducted on three datasets to evaluate the
effectiveness of our designed SMIM module and RFF module. As shown in Table V,
VI, VII, a consistent trend is observed across all three datasets. The DCP-Net
containing only the SMIM module or the RFF module both demonstrates improved
performance compared to the No-Interaction baseline. This provides preliminary
evidence for the effectiveness of the respective modules. For a single-module
DCP-Net, the model without the SMIM module losses the capability of dynamic
interactions and conducts centralized collaborative perception, demonstrating
the best performance. The centralized baselines perform much weaker than the
DCP-Net without the SMIM module in perception improvement. This phenomenon
reveals that our designed RFF module can leverage the collaborative features
more efficiently. The DCP-Net without the RFF module saves a significant
expense of communication overhead while still maintaining a relatively high
level of predicted accuracy. The result exhibits the talent of the SMIM module
in selecting the appropriate opportunity and perception supporter. However, in
some complicated conditions, such as the multi-platform Potsdam dataset of the
Homo-PIS mode, the DCP-Net without the RFF module exhibits weakness in
misaligned feature fusion and gives up the opportunity for collaborative
perception. The phenomenon also verifies the necessity of the RFF module.
Furthermore, the model with two modules working simultaneously obtains the
highest collaborative efficiency and achieves a win-win goal of improving
performance with less information exchange. This further validates the
effectiveness and excellent combined performance of the SMIM and RFF modules.

####  IV-E2 Hyperparameter Ablation Analysis

In the SMIM module, the request threshold is predefined as a hyperparameter
during the self-information match stage. The platform with a self-information
confidence score below the request threshold is prompted to send collaboration
requests to others. A proper threshold setting is crucial for establishing
collaborative perception. Therefore, an ablation experiment is performed to
evaluate the impact of various thresholds on the performance of the multi-
platform Potsdam dataset in the Homo-PIS mode, as shown in Fig.14. When the
request threshold is decreased from 1 to 0.9, there is a negligible decrease
in the average mIoU, indicating that the performance remains relatively
stable. However, there is a significant doubling in CE, suggesting that the
system becomes more efficient in terms of collaboration. Conversely, when the
request threshold is changed from 0.2 to 0, both the average mIoU and CE
experience a significant decline, which can be attributed to a sudden
reduction in collaboration frequency. Interestingly, a similar trend is
observed in the Homo-PIS mode, confirming the findings in the Homo-CIS mode.
To strike a balance between accuracy and efficiency, this paper selects a
collaboration threshold of 0.8.

Figure 14: Ablation experiments on the request threshold in the self-
information match stage of the SMIM module.

In the SMIM module, each mutual-information match score is calculated based on
the compressed request and its corresponding key. The scale of compression may
affect the later supporter selection. So it is essential to explore the
influence of the request vector size on collaborative perception prediction.
The ratio of compression can affect the subsequent supporter selection
process. Hence, it is crucial to explore the influence of the request vector
size on collaborative perception prediction. To explore this, an ablation
experiment is conducted using the multi-platform Potsdam dataset in the Homo-
PIS mode, with the size of the request vector varied from 2 to 1024. Fig.15
exhibits that when the request size is set to 32, there is a distinct turning
point in the curve for both metrics. Our proposed DCP-Net achieves the best
results in terms of semantic segmentation prediction and collaboration
efficiency while maintaining a sub-minimal communication overhead.
Consequently, the size of the compressed request is set at 32 in the SMIM
module.

Figure 15: Ablation experiments on the request size used in the mutual-
information match stage of the SMIM module.

Figure 16: The visualization of results predicted by various baselines and
DCP-Net in the Homo-CIS mode of the Potsdam dataset.

Figure 17: The visualization of results predicted by various baselines and
DCP-Net in the Homo-PIS mode of the Potsdam dataset.

Figure 18: The visualization of results predicted by various baselines and
DCP-Net in the Homo-CIS mode of the iSAID dataset.

Figure 19: The visualization of results predicted by various baselines and
DCP-Net in the Homo-PIS mode of the iSAID dataset.

Figure 20: The visualization of results predicted by various baselines and
DCP-Net in the Hetero-PIS mode of the DFC23 dataset.

###  IV-F Visualization

The improvement of downstream predictions through the collaborative perception
is presented in Fig.20,20,20,20,20 intuitively.

In Fig.20,20, for the multi-platform Potsdam dataset of both modes, No-
Interaction tends to classify uncertain objects resulting from image
degradation as clutter and often misidentified blurry buildings as impervious
surfaces. Additionally, it exhibits poor ability to distinguish between trees
and low vegetation. Compared with other baselines, our DCP-Net can provide
more accurate predictions for the foreground objects and boundary regions with
the help of proper selection and fusion of collaborative features during the
collaborative perception.

In the multi-platform iSAID dataset of both modes, it is more challenging to
discriminate 16 classes, particularly in a lower resolution, and we select
only six classes for concise visualization. Due to the incomplete
representation, No-Interaction mistakes some large vehicles for the small ones
in Fig.20 and directly ignores some tiny objects, such as the small vehicles
in Fig.20. In contrast with other methods, our DCP-Net is able to remedy the
terrible condition mentioned above better and provide a more accurate regional
prediction.

In the practical scene of DFC23, the buildings are densely clustered, and No-
Interaction often misclassifies certain ranges of degraded observations as
background indiscriminately. Furthermore, it is also tricky to improve the
degraded perception with the help of heterogeneous collaborative features.
Despite these challenging conditions, our DCP-Net still outperforms other
methods. In Fig.20, DCP-Net delivers a clearer demarcation between buildings
and compensates for the predictions in the right bottom and the top. This
visualization of predicted masks supports the effectiveness of DCP-Net in
practical applications.

##  V Conclusion

Motivated by the widespread deployment of intelligent remote sensing
platforms, this paper proposes DCP-Net, a novel collaborative perception
framework through feature interactions among multiple remote sensing
platforms. DCP-Net leverages the designed SIMM module to establish adaptive
collaboration with other platforms. In addition, the RFF module promotes the
integration of features across multiple platforms to facilitate superior
subsequent predictions. Importantly, the entire process can be completed
without the human intervention to determine the optimal collaboration
opportunities and partners. Extensive experiments and visualization analyses
are conducted on three datasets, namely Potsdam, iSAID and DFC23, which are
redesigned in three difficulty modes. Comparative analysis with existing
collaborative methods comprehensively demonstrates our proposed method’s
superiority.

## References

  * [1] C. Wang, D. Zhao, X. Qi, Z. Liu, and Z. Shi, “A hierarchical decoder architecture for multi-level fine-grained disaster detection,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  * [2] Q. Xu, Y. Li, and Z. Shi, “Lmo-yolo: A ship detection model for low-resolution optical satellite imagery,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 4117–4131, 2022.
  * [3] J. Chen, K. Chen, H. Chen, W. Li, Z. Zou, and Z. Shi, “Contrastive learning for fine-grained ship classification in remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022.
  * [4] J. Parajuli, R. Fernandez-Beltran, J. Kang, and F. Pla, “Attentional dense convolutional neural network for water body extraction from sentinel-2 images,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 6804–6816, 2022.
  * [5] Y. Gu, C. Wang, and X. Li, “An intensity-independent stereo registration method of push-broom hyperspectral scanner and lidar on uav platforms,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–14, 2022\.
  * [6] Y. Gu, Z. Xiao, and X. Li, “A spatial alignment method for uav lidar strip adjustment in non-urban scenes,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  * [7] Z. Qin and Y. Gu, “A new radiometric correction method for multiple uav multispectral images under varying illumination condition,” in IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium, pp. 7579–7582, IEEE, 2022.
  * [8] Z. Li, Y. Xie, W. Hou, Z. Liu, Z. Bai, J. Hong, Y. Ma, H. Huang, X. Lei, X. Sun, et al., “In-orbit test of the polarized scanning atmospheric corrector (psac) onboard chinese environmental protection and disaster monitoring satellite constellation hj-2 a/b,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–17, 2022.
  * [9] B. Zhang, Y. Wu, B. Zhao, J. Chanussot, D. Hong, J. Yao, and L. Gao, “Progress and challenges in intelligent remote sensing satellite systems,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022\.
  * [10] M. Campion, P. Ranganathan, and S. Faruque, “Uav swarm communication and control architectures: a review,” Journal of Unmanned Vehicle Systems, vol. 7, no. 2, pp. 93–106, 2018.
  * [11] W.-Q. Wang, “Large-area remote sensing in high-altitude high-speed platform using mimo sar,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 6, no. 5, pp. 2146–2158, 2013.
  * [12] Y. Liu, M. Xing, G. Sun, X. Lv, Z. Bao, W. Hong, and Y. Wu, “Echo model analyses and imaging algorithm for high-resolution sar on high-speed platform,” IEEE Transactions on Geoscience and remote sensing, vol. 50, no. 3, pp. 933–950, 2011.
  * [13] W. Li, Z. Nie, and X. Sun, “Wireless transmission of mwd and lwd signal based on guidance of metal pipes and relay of transceivers,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 8, pp. 4855–4866, 2016.
  * [14] J. Juan, M. Hernández-Pajares, J. Sanz, P. Ramos-Bosch, A. Aragon-Angel, R. Orus, W. Ochieng, S. Feng, M. Jofre, P. Coutinho, et al., “Enhanced precise point positioning for gnss users,” IEEE transactions on geoscience and remote sensing, vol. 50, no. 10, pp. 4213–4222, 2012.
  * [15] C. Ma, Y. Yang, F. Cao, X. Wang, X. Liu, C. Meng, J. Zhang, and S. Pan, “High-resolution microwave photonic radar with sparse stepped frequency chirp signals,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–10, 2022.
  * [16] K. F. Warnick, R. Maaskant, M. V. Ivashina, D. B. Davidson, and B. D. Jeffs, Phased arrays for radio astronomy, remote sensing, and satellite communications.  Cambridge University Press, 2018.
  * [17] T. Zhang, Z. Wang, P. Cheng, G. Xu, and X. Sun, “Dcnnet: A distributed convolutional neural network for remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–18, 2023\.
  * [18] Y. Wang, Z. Wang, P. Cheng, X. Zeng, H. Wang, X. Sun, and K. Fu, “Dcm: A distributed collaborative training method for the remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–17, 2023.
  * [19] Y.-C. Liu, J. Tian, C.-Y. Ma, N. Glaser, C.-W. Kuo, and Z. Kira, “Who2com: Collaborative perception via learnable handshake communication,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6876–6883, IEEE, 2020.
  * [20] Y.-C. Liu, J. Tian, N. Glaser, and Z. Kira, “When2com: Multi-agent perception via communication graph grouping,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 4106–4115, 2020.
  * [21] Y. Hu, S. Fang, Z. Lei, Y. Zhong, and S. Chen, “Where2comm: Communication-efficient collaborative perception via spatial confidence maps,” arXiv preprint arXiv:2209.12836, 2022.
  * [22] Y. Zhou, J. Xiao, Y. Zhou, and G. Loianno, “Multi-robot collaborative perception with graph neural networks,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2289–2296, 2022.
  * [23] P.-Y. Lajoie and G. Beltrame, “Swarm-slam: Sparse decentralized collaborative simultaneous localization and mapping framework for multi-robot systems,” arXiv preprint arXiv:2301.06230, 2023.
  * [24] C. Fan, J. Hu, and J. Huang, “Few-shot multi-agent perception with ranking-based feature learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [25] A. Nash, A. Vardy, and D. Churchill, “Herd’s eye view: Improving game ai agent learning with collaborative perception,” arXiv preprint arXiv:2306.06544, 2023.
  * [26] G. Gao, L. Yao, W. Li, L. Zhang, and M. Zhang, “Onboard information fusion for multisatellite collaborative observation: Summary, challenges, and perspectives,” IEEE Geoscience and Remote Sensing Magazine, vol. 11, no. 2, pp. 40–59, 2023.
  * [27] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015.
  * [28] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241, Springer, 2015.
  * [29] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, 2017.
  * [30] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881–6890, 2021.
  * [31] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in Neural Information Processing Systems, vol. 34, pp. 12077–12090, 2021.
  * [32] L. Mou, Y. Hua, and X. X. Zhu, “Relation matters: Relational context-aware fully convolutional network for semantic segmentation of high-resolution aerial images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 11, pp. 7557–7569, 2020.
  * [33] F. I. Diakogiannis, F. Waldner, P. Caccetta, and C. Wu, “Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 162, pp. 94–114, 2020\.
  * [34] R. Niu, X. Sun, Y. Tian, W. Diao, Y. Feng, and K. Fu, “Improving semantic segmentation in aerial imagery via graph reasoning and disentangled learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–18, 2021.
  * [35] J. Kang, Z. Wang, R. Zhu, J. Xia, X. Sun, R. Fernandez-Beltran, and A. Plaza, “Disoptnet: Distilling semantic knowledge from optical images for weather-independent building segmentation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
  * [36] R. Niu, X. Sun, Y. Tian, W. Diao, K. Chen, and K. Fu, “Hybrid multiple attention network for semantic segmentation in aerial images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–18, 2021.
  * [37] X. He, Y. Zhou, J. Zhao, D. Zhang, R. Yao, and Y. Xue, “Swin transformer embedding unet for remote sensing image semantic segmentation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [39] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794–7803, 2018.
  * [40] L. Mou, S. Saha, Y. Hua, F. Bovolo, L. Bruzzone, and X. X. Zhu, “Deep reinforcement learning for band selection in hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–14, 2021.
  * [41] J. Feng, D. Li, J. Gu, X. Cao, R. Shang, X. Zhang, and L. Jiao, “Deep reinforcement learning for semisupervised hyperspectral band selection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–19, 2021\.
  * [42] C. Persello et al., “2023 IEEE GRSS Data Fusion Contest: Large-scale fine-grained building classification for semantic urban reconstruction [technical committees],” IEEE Geosci. Remote Sens. Mag., vol. 11, no. 1, pp. 94–97, 2023.
  * [43] F. Rottensteiner, G. Sohn, M. Gerke, and J. D. Wegner, “Isprs semantic labeling contest,” ISPRS: Leopoldshöhe, Germany, vol. 1, no. 4, 2014\.
  * [44] S. Waqas Zamir, A. Arora, A. Gupta, S. Khan, G. Sun, F. Shahbaz Khan, F. Zhu, L. Shao, G.-S. Xia, and X. Bai, “isaid: A large-scale dataset for instance segmentation in aerial images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 28–37, 2019\.
  * [45] X. Huang et al., “Urban building classification (UBC) - a dataset for individual building detection and classification from satellite imagery,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, pp. 1413–1421, Jun. 2022.