# AOSLO-net: A deep learning-based method for automatic segmentation of
retinal microaneurysms from adaptive optics scanning laser ophthalmoscope
images

Qian Zhang1,†, Konstantina Sampani2,3,†, Mengjia Xu1,†, Shengze Cai1,
Yixiang Deng5, He Li5, Jennifer K. Sun2,4,∗, George Em Karniadakis1,5,∗

1Division of Applied Mathematics, Brown University, Providence, RI 02912, USA
2Beetham Eye Institute, Joslin Diabetes Center, Boston, MA 02215, USA
3Department of Medicine, Harvard Medical School, Boston, MA 02115, USA
4Department of Ophthalmology, Harvard Medical School, Boston, MA 02115, USA
5School of Engineering, Brown University, Providence, RI 02912, USA

† These authors contributed equally to this work.
∗ To whom correspondence should be addressed.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Microaneurysms (MAs) are one of the earliest signs of diabetic retinopathy
(DR), a frequent complication of diabetes that can lead to visual impairment
and blindness. Adaptive optics scanning laser ophthalmoscopy (AOSLO) provides
real-time retinal images with resolution down to $\sim$2 $\mu m$ and thus
allows detection of the morphologies of individual MAs, a potential marker
that might dictate MA pathology and affect the progression of DR. In contrast
to the numerous automatic models developed for assessing the number of MAs on
fundus photographs, currently there is no high throughput image protocol
available for automatic analysis of AOSLO photographs. To address this
urgency, we introduce AOSLO-net, a deep neural network framework with
customized training policies to automatically segment MAs from AOSLO images.
We evaluate the performance of AOSLO-net using 87 DR AOSLO images and our
results demonstrate that the proposed model outperforms the state-of-the-art
segmentation model both in accuracy and cost and enables correct MA
morphological classification.

## Introduction

Diabetic retinopathy (DR), a frequent complication of diabetes, remains the
leading cause of new cases of blindness, among working-age adults in the
United States[1], and it is expected to affect approximately 14.6 million
people in United States by the year 2050[2]. DR results in pathological
alterations in both neural and microvascular structures and can damage any
part of the central or peripheral retina [3]. Based on the Early Treatment
Diabetic Retinopathy Study (ETDRS) severity scale, eyes with DR progress
through a severity spectrum of DR from non-proliferative DR (NPDR)[4].
Classification of these severity stages is performed by evaluating certain
clinical features of the retina, such as microaneurysms (MAs), intraretinal
hemorrhages, hard exudates, cotton wool spots, intraretinal microvascular
abnormalities and venous beading[5]. Advanced and vision-threatening stages of
DR are characterized by neovascularization and by the presence of vessels that
may leak fluid and eventually cause diabetic macular edema (DME), which is
manifested as thickening of the central retinal[6].

Fundus photography, a conventional and non-invasive imaging modality, has been
widely used since the 1960’s to access the severity of DR on the basis of the
presence and severity of retinal vascular lesions[7, 8, 9]. In fundus images
(see Fig. 1(a)), MAs appear as red circular dots with sharp margins whereas
intraretinal hemorrhages, which also display as red dots or blots, may be
larger and more irregularly shaped than MAs[10]. MAs and intraretinal
hemorrhages are frequently not able to be distinguished on fundus images as
the standard fundus photographs do not provide microscopic details of these
lesions or allow determination of vascular perfusion. Indeed, the ETDRS
severity scale lumps the grading of these lesions into a single “H/MA”
category. In contrast, more advanced imaging modalities, such as optical
coherence tomography angiography (OCTA) [11, 12] and adaptive optics scanning
laser ophthalmoscopy (AOSLO) [13, 14], is capable of providing higher
resolution details of these abnormalities and address the presence or absence
of blood flow, thereby allowing differentiation hemorrhages from MAs [15].
AOSLO provides retinal imaging with the highest resolution of all the
available retinal imaging techniques on human retina (down to the cellular
level (2 $\mu m$)) and thus has been used to identify and quantify the
morphology of individual MA [13] (see Figs. 1(b-e)) and to measure blood flow
at the capillary level [16].

Recent OCTA and AOSLO-based studies have suggested a possible correlation
between the morphology of retinal MAs and their tendency to leak fluid,
rupture or form thrombus [17, 14, 18]. These findings imply that accurate
identification of the shapes of MAs might be useful in the future to improve
prediction of DR worsening or improvement. However, existing models for MA
segmentation and classification have been trained on standard fundus
photographs and therefore can only predict the number of MAs and their
locations [19, 20, 21, 22, 23, 24], because the resolution of standard fundus
photography is not sufficient to analyse the shape of individual MAs. In
contrast, the AOSLO imaging technique provides ultra-high resolution retinal
images that can be used to classify MA morphologies [13]. To date, AOSLO
retinal images have been analyzed manually by specially trained personnel [13,
25, 14, 17] as no model has been developed to automatically process these
images.

Emerging interest in automated analysis of retinal images has been sparked by
the rapidly increasing prevalence of diabetes worldwide and the consequent
need for scalable approaches to screen and triage patients at risk for vision
loss from DR [26, 27, 28]. With the recent advances in the computational power
of graphics processing units (GPUs), deep convolutional neural networks
(DCNNs) have become a widely used tool for efficient DR screening [29]. DCNNs
are more generic compared to conventional methods that rely on hand-crafted
features because the deep layers in the network act as a set of increased
levels of feature extractors that can learn directly from the input images
[30, 31, 32]. Moreover, DCNN models are highly discriminative in automated DR
severity grading and thus have achieved higher screening accuracy than
conventional methods [26]. In particular, UNet, combining an encoder and a
decoder to form a “U-shape” structure, is a specially designed CNN
architecture for biomedical image segmentation tasks. UNet is very effective
in few-shot prediction with only a few labeled images when combined with data
augmentation [33] and it has outperformed the plain DCNN in segmenting
biomedical images, particularly for those with complex subtleties [33, 34,
35]. Recent development of UNet has given rise to a number of variants, such
as deformable UNet [36], residual UNet [37], recurrent residual UNet [37] and
iterNet [38], which further improved the segmentation accuracy on fundus
images.

We have noted several features from the AOSLO image dataset that pose
challenges for the existing automatic segmentation models. As shown in Fig.
1(b), (i) the contrast between the MA body and background is low whereas the
level of the background noise is high; (ii) the boundary of the MA is not
clearly defined; (iii) a typical AOSLO image may contain one or multiple MAs
with different shapes and sizes; (iv) there are numerous background blood
vessels in the images, some of which are even at similar size as the MAs.
These background vessels may interfere with the segmentation of the feeding or
draining vessels of the MAs, which are crucial to determine MA morphology. In
this work, we design the first deep neural network model, or AOSLO-net (see
Fig. 2), to perform automatic segmentation of MAs from AOSLO images and
quantify their shape metrics that can be used for classification of MAs into
different types, such as focal bulging, saccular, fusiform, mixed
saccular/fusiform, pedunculated and irregular-shaped MAs [13]. This model is
trained and tested by using 87 AOSLO MA images with masks generated manually
by trained graders, the largest published AOSLO image dataset for this kind of
effort thus far. We evaluated the performance of this model by comparing the
model predictions with nnUNet [39], a state-of-the-art (SOTA) UNet model whose
superiority has been demonstrated for dozens of publicly available databases.

## Data

### Demographic Information

In this study, 87 MAs were imaged from 28 eyes of 20 subjects with varying
severity of DR (56% NPDR and 44% PDR). Sixteen (80%) subjects had type 1
diabetes, 7 (35%) of the subjects are female and average age is $41.9\pm 10.4$
years old, with mean diabetes duration 23.9±8.4 years, and mean HbA1c is
$8.2\pm 1.1$%. Informed written consent was obtained from each participant
prior to the performance of any study procedures at a single visit in Beetham
Eye Institute, a tertiary referral center for diabetes care. This study
adhered to the tenets of the Declaration of Helsinki and was approved by the
institutional review board of the Joslin Diabetes Center.

Figure 1: Figure 1. Multi-modality AOSLO images and multi-fidelity mask sets
are employed to train and test the AOSLO-net. (a) Five MAs imaged by AOSLO
(highlighted by boxes in different colors) superimposed on a digital fundus
photograph from an eye with diabetic retinopathy. (b-e) Examples of four sets
of AOSLO images with different modalities employed to train and test the
AOSLO-net: (b) raw images extracted from the AOSLO videos; (c) perfusion maps
(also see Fig. 1 in supporting material (SI)); (d) preprocessed AOSLO images
with enhancement; (e) two-modality images which are generated by concatenating
perfusion maps (c) and enhanced AOSLO images (d). Details of how multi-
modality AOSLO images are generated can be found in the Method section. These
images illustrate that a varied number of MAs can be detected in a single
AOSLO image. Images in row 1-3 contain multiple MAs, whereas images in row 4-5
contain a single MA with complicated background vessels whose size may be
comparable to the MA. (f-h) Three sets of masks are generated independently to
examine the robustness of the AOSLO-net to mask sets with different qualities.
Normal set (f): masks are created based on both the AOSLO videos and perfusion
maps to illustrate the body of MAs and their feeding and draining vessels.
Short set (g): masks are designed to show shorter feeding and draining vessels
of MAs compared to the normal set, while the thickness of the vessels remains
similar to the normal masks. Thick set (h): masks are designed to show thicker
feeding and draining vessels of MAs compared to the normal set, while the
length of the vessels remains similar to the normal masks.

### Data Set

87 MAs from the eyes of adult study participants with diabetes underwent AOSLO
imaging. All MAs were located within $\sim$$None$ of the foveal center. The
AOSLO system has been previously described in detail by Lu et al. [40]. This
system uses confocal and multiply scattered light (MSL) imaging modes, and
achieves a field size of $\sim$$None$ $\times$ $\sim$$None$ with lateral
resolution of $\sim$2.5 $\mu$m on the retina. Moreover, 75-frame videos of
each MA were aligned and averaged (MATLAB, The MathWorks, Inc., Natick, MA,
USA). The magnification factor on AOSLO images was determined by eye axial
length measurement or derived from the spherical equivalent of the eye. For
this exploratory study, we included MAs with high quality AOSLO images (see
Fig. 1(b)), where the 2D MSL images and corresponding perfusion maps (see Fig.
1(c)) provided sufficient detail to identify the full extent of MAs’ bodies
and their parent vessels’ boundaries.

The masks in our dataset, as illustrated in Fig. 1(f-h), are considered as the
ground truth for training the AOSLO-net and they are created manually by
ophthalmologists and skilled trainees using ImageJ [41]. Both the AOSLO images
(Fig. 1(b)) and the corresponding perfusion map (Fig. 1(c)) are referred when
masks are generated. Moreover, different groups of MA masks, as shown in Figs.
1(g-h), are created by varying the length and thickness of the parent vessels
to test the robustness of AOSLO-net to masks with different qualities. The
normal mask set (see Fig. 1(f)), which is generated by ophthalmologists,
intends to represent the true geometries of the parenting vessels illustrated
on the AOSLO images. The short mask set (see Fig. 1(g)) is designed to show
shorter feeding and draining vessels of MAs compared to the normal mask set,
while the thickness of the vessels remains similar to the normal masks. The
thick mask set (see Fig. 1(h)) is designed to show thicker feeding and
draining vessels of MAs compared to the normal set, while the length of the
vessels remains similar to the normal masks. This examination also
demonstrates the human ability of segmentation when comparing one kind of
masks with another, as they equivalently represent the MAs.

Figure 2: Figure 2. Overview of the architecture of AOSLO-net. The MA
segmentation is performed through the following steps: data pre-processing,
deep neural network training and inference, post-processing and output
ensembling. (a) Pre-processing AOSLO images for training the AOSLO-net. We
first created the perfusion map by computing the deviations from the AOSLO
video. Moreover, we created another set of enhanced AOSLO images from the
AOSLO video. These two sets of images are concatenated to generate a third set
of two-channel MA images. Data augmentation was performed on the two-channel
MA images to increase the diversity of the training set (see examples in Fig.
2 in SI). (b) Detailed modular components of the AOSLO-net, i.e., preprocessed
multi-modality images produced from (a) is fed into the AOSLO-net, which
consists of an EfficientNet-b3 encoder (c) and the regular UNet decoder. We
then perform post-processing and ensembling on the output images of the AOSLO-
net to generate the segmentation map. (c) A zoom-in view of the
EfficientNet-b3, which works as the encoder in the AOSLO-net.

## Results

First, we train the AOSLO-net and nnUNet using perfusion map (Fig. 1(c)),
enhanced AOSLO (Fig. 1(d)) and two-channel images (Fig. 1(e)), respectively,
to examine which image modality can optimize the model performance. The normal
mask set (Fig. 1(f)) is used as the target in these training processes. Our
results in Figs. 3(a-c,f-h) and Fig.3 in SI show that higher Dice and IoU
scores are achieved for both nnUNet and AOSLO-net models when two-channel
images are used, suggesting that two-channel images could provide more MA
geometrical information to the segmentation models than the other two image
modalities. Therefore, in the following section, we use the two-channel image
set as the input of segmentation models to assess the model performance on
different mask sets, including normal mask set, short mask set and thick mask
set. The performance of AOSLO-net and nnUNet on these three mask sets is
summarized in Fig. 3, which shows that based on Dice and IoU, AOSLO-net
achieves more high-quality predictions and fewer low-quality predictions for
these three groups of masks. Specifically, when models are trained with the
normal and short mask sets, as shown in Figs. 3(c,d), AOSLO-net achieves mean
Dice scores of 0.7816 and 0.8412, respectively, which are higher than 0.7686
and 0.8378 of nnUNet. Additionally, the performance of AOSLO-net appears to be
more stable than nnUNet, given smaller standard deviations in the results of
AOSLO-net. When these two models are trained with thick mask set (see Fig.
3(e)), although the mean Dice scores of AOSLO-net and nnUNet are the same,
AOSLO-net preserves a stable performance with a relatively smaller standard
deviation in Dice score. In addition to nnUNet, as shown in Fig.4 in SI, we
specifically compare the performance of AOSLO-net with other popular CNN-based
models, i.e., Deformable UNet[36], ResUNet[42, 43] and Deformable ResUNet[44,
42, 43], using the normal mask dataset. The statistics of the Dice scores
listed in Fig. 3(k) suggest that AOSLO-net outperforms these models
significantly in mean Dice score and standard deviation.

Figure 3: Figure 3. Performance metrics of AOSLO-net and nnUNet as well as
other deep-learning models on segmenting the AOSLO images. Histogram and
density distribution of Dice score (a-e) and IoU (f-j) obtained from AOSLO-net
and nnUNet when they are trained using (a,f) perfusion map and normal mask
set, (b,g) enhanced AOSLO images and normal mask set, (c,h) two-channel MA
images and normal mask set, (d,i) two-channel MA images and short mask set,
(e,j) two-channel MA images and thick mask set. The result of AOSLO-net is
denoted by cyan color whereas the result of nnUNet is denoted by red color.
The dash lines denote the mean values of the Dice or IoU distributions. (k)
The overall performance (mean value), performance stability (standard
deviation), the worst and best case performance (minimal and maximum value) of
Dice score of different AI models in segmenting AOSLO images. Figure 4: Figure
4. Typical examples of AOSLO-net outperforming nnUNet on detecting (a) the MA
body and (b) parenting vessels of MAs when they are trained using normal mask
set (n-Mask). First column: enhanced AOSLO images. Second column: normal masks
used to train the models. Third column: segmentation results of nnUNet. Fourth
column: segmentation results of AOSLO-net. Numbers in images are the Dice
scores. (a) For images with IDs 036, 043, 048, 062, and 092, the AOSLO-net is
able to detect MAs, which are missed by nnUNet (marked in red circles). For
images with IDs 013 and 048, the AOSLO-net can better reconstruct the full
shape of MAs compared to those of nnUNet (marked in yellow circles). We also
note that in cases of IDs 013, 043 and 092, the segmentation models can even
detect some potential MAs, which are not marked in the masks. (b) For images
with IDs 015, 055, 082, 088, 094 and 102, the AOSLO-net is capable of
detecting both feeding and draining vessels connected to the MAs, while the
nnUNet may miss one or both of the parenting vessels (marked in yellow
circles). These results show that the AOSLO-net is more reliable in detecting
MA parenting vessels from the input images, which are essential in the MA
classification — a downstream task for disease diagnosis.

### Morphology of MA body and its feeding and draining vessels

To further examine the capabilities of AOSLO-net and nnUNet in extracting the
detailed MA features, we perform image-wise analysis by comparing individual
pairs of masks and model predictions. We focus on the model performance on
detecting MA bodies and their feeding and draining vessels, respectively.
Typical examples of the comparisons between nnUNet and AOSLO-net on detecting
MA bodies are illustrated in Fig. 4(a). We note that AOSLO-net can identify
the targeted MAs and extract MA bodies that are comparable with the masks,
particularly for images containing multiple MAs, like IDs 036, 048, 062 and
092. In contrast, nnUNet fails to detect some of the MAs (highlighted in red
circle). As for the vessel detection, Fig. 4(b) shows that nnUNet may ignore
one or both of the parenting vessels of MAs (highlighted in yellow circle),
while AOSLO-net can segment out these missed vessels (from complicated
background with numerous vessels), which is essential for further
morphological analysis.

Next, we use another two sets of masks: one with shorter vessels compared to
the normal dataset and one with thicker vessels, as learning targets to train
AOSLO-net and nnUNet and compare their performance on detecting the feeding
and draining vessels of MAs. We first train the model with short vessel mask
dataset. The predictions from the two models in Fig. 5(a) show that the
performances of nnUNet and AOSLO-net on detecting feeding and draining vessels
are both compromised, as the shorter vessel masks provide less vessel end
information. However, the AOSLO-net still can detect correct vessels in these
four cases whereas nnUNet fails to predict some of the vessel ends
(highlighted in yellow circle). These comparisons indicate that AOSLO-net is
more robust against vessel length of the training masks.

When trained with thick vessel mask set, as shown in Fig. 5(b), AOSLO-net
outperforms the nnUNet although the predictions of nnUNet have improved in
detecting the feeding and draining vessels of MAs. Fig. 5(b) shows that while
nnUNet misses the MA bodies and vessel ends connected to MAs (highlighted in
yellow circle), AOSLO-net can detect these feeding and draining vessels. These
results again demonstrate the robustness of AOSLO-net when trained with
varying vessel thickness.

Figure 5: Figure 5. Typical examples of AOSLO-net outperforming nnUNet on
detecting the parenting vessels of MAs when they are trained using (a) short
mask set and (b) thick mask set. First column: enhanced AOSLO images. Second
column: normal masks used to train the models. Third column: segmentation
results of nnUNet. Fourth column: segmentation results of AOSLO-net. Numbers
in images are the Dice scores. (a) AOSLO-net are capable of detecting the
parenting vessel of MAs in images with ID 055, 082, 088 and 094, whereas
nnUNet misses one of the parenting vessels(marked in yellow circles).
Particularly, nnUNet again mistakenly predicts the vessel on the upper left
for the case of ID 055. (b) AOSLO-net are capable of detecting the parenting
vessel of MAs in images with ID 038, 046, 055 and 102, whereas nnUNet misses
one of the parenting vessels(marked in yellow circles). In case of ID 055,
implementation of thick mask set improves the segmentation results of AOSLO-
net, but not for nnUNet.

### Morphology Quantification for the MA segmentation maps

In order to better evaluate the MA segmentation performance of AOSLO-net and
nnUNet, we compute three important MA morphological indices (“largest caliber”
- $LC$, “narrowest caliber” - $NC$ and the “body-to-neck ratio” - $BNR$)
defined in [14] to quantify the specific morphological characteristics for the
segmented MAs. Specifically, we first compute the MA skeleton (or medial-axis)
for every single MA using the “Scikit-image” package [45], and then apply the
Euclidean distance transformation to compute the medial radius distances
$D=\\{d_{i}|i=1,2,...,N\\}$ ($N$ is the number of points on the MA skeleton)
from all points of the MA skeleton to the background pixels (i.e., pixels of
each MA contour). Consequently, the $LC$ value for each single MA corresponds
to twice of the largest distance value in the sorted distance list
$D_{sorted}$. Due to the varied vessel lengths of different MAs, we calculate
the NC value for each single MA by selecting the 10 smallest medial radius
distances from $D_{sorted}$, and double the average medial radius distances as
the final $NC$ value. Based on the $LC$ and $NC$ values, the BNR value for
each MA can be computed by using $LC/NC$. Fig. 6 gives some examples of the LC
and NC quantification results for the segmented MAs predicted by AOSLO-net and
nnUNet trained with three different MA masks (normal, short and thick mask
sets). From Fig. 6(a) and (c), we can find that the AOSLO-net trained with
thick MA ground truth masks attains the best MA segmentation performance; the
NC quantification results (red curve) as shown in the third row of Fig. 6(c)
are very close to the reference NC values (black dashed line) obtained from
the thick masks. More details about the corresponding examples of the enhanced
MA perfusion maps and the MA segmentation results using AOSLO-net and nnUNet
are, respectively, shown in columns 1-3 of Fig. 6(b). AOSLO-net can
effectively detect the important small vessels for the heterogeneous MAs (see
the 2nd column in Fig. 6(b)), which play a very important role in different
downstream tasks, e.g., MA morphological parameter quantification (NC, BNR,
convexity), MA severity stratification, and in hemodynamics simulations. The
three blue dots with very high NC values in different rows of Fig. 6(c) are
the same MAs for which nnUNet fails to detect the small vessels.
Quantification results for the BNR of MAs based on the analysis of NC and LC
is summarized in Fig.9 in SI.

Figure 6: Figure 6. Quantification results for the largest caliber and
narrowest caliber factors for the segmented MAs predicted by AOSLO-net and
nnUNet trained with three sets of MA masks (normal, short and thick masks).
(a) The largest caliber (LC) results for the segmented MAs. The black dashed
lines represent the reference MA LC values for different MAs, while the red
and blue curves are the MA LC quantification results based on the segmentation
maps from AOSLO-net and nnUNet using three different masks (from top to the
bottom row). (b) Examples for the original enhanced perfusion maps (1st
column), segmented MAs using AOSLO-net (2nd column) and nnUNet (3rd column)
trained with different MA masks. The MA segmentation results in yellow and
green bounding boxes correspond to the images in (c) marked with yellow and
green arrows. (c) Narrowest caliber (NC) quantification results for segmented
MAs using AOSLO-net and nnUNet trained with three different MA masks; the
black dashed lines represent the reference MA NC values computed using the
original 3 types of MA masks; the red and blue curves represent the NC
quantification results using segmentation maps obtained by AOSLO-net and
nnUNet using three different masks (from top to the bottom row).

## Discussion and Summary

Although 2D fundus photography has been primarily employed for DR screening
and severity grading, other advanced imaging modalities, such as AOSLO, which
is currently used for disease investigation and primarily for research
purposes, can provide additional information regarding the retinal
microvascular pathology, such as monitoring of variations in blood flow rates
[46], detection and identification of the MA morphologies [13], enhanced
visualization of MA thrombus status [14], etc. The information from this
imaging technique could be potentially used not only to improve the future
accuracy of DR screening, but also to better predict the rate of DR worsening,
with the goal of eventually providing individualized management and treatment
intervention plans [47]. Due to the rapidly rising global prevalence of
diabetes and shortage of skilled graders for retinal images, implementation of
automatic screening techniques is desirable to accommodate the corresponding
increasing need to screen and evaluate patients with diabetes ocular
complications [28, 27]. Development of an automated segmentation technique for
AOSLO images, which are at much smaller scale than fundus photographs,
represents a unique challenge and provides the opportunity to elucidate the
steps of in vivo worsening and regression of MAs, a key lesion in DR.

In the last decade, DCNN has become the state-of-the-art technique in
processing retinal images. Although existing DCNN models have achieved high
accuracy in analyzing fundus images, they have not been tested on retinal
images from other modalities. In this work, we design AOSLO-Net to
automatically segment MA bodies with their feeding and draining vessels from
AOSLO images. We use a very deep model to extract high-level features to
understand the MA morphology, which can vary substantially in shape and size
between individual MAs, and to create segmentation based on that information.
Our results show that the deep network structure can segment MAs from AOSLO
images with high accuracy for both metrics and morphology.

Our AOSLO dataset has a limited size, containing less than 100 videos, so the
deep structure of AOSLO-net may cause overfitting, and it can be hard to
train. Therefore, it is necessary to perform data augmentation according to
specific dataset. Moreover, the MAs are of different shapes and many of them
are not similar to others. This requires the AOSLO segmentation to be a
composition of many few-shot problems. Therefore, data augmentation is
necessary to train the deep network and fully utilize the data. To optimize
the data augmentation, we use horizontal and vertical flip, rotation in
uniformly distributed angles and scaling to achieve maximum space
configuration of MAs. As a result, the AOSLO-net can learn MA geometric
information very effectively from the limited dataset. In addition, transfer
learning is desirable for training large and deep networks. In the current
study, we use EfficientNet-b3 [48] as the feature extractor (encoder), and
pretrain AOSLO-net with ImageNet dataset [49], which is believed to accelerate
the training convergence and is widely deployed in transfer learning based
image analysis.

The relationship between model performance metrics and morphological features
of MAs is also investigated carefully in our work. Accurate detection of
feeding and draining vessels of MAs is critical to determine MA morphology,
which is important to predict blood flow characteristics and estimate the
likelihood of thrombosis within the MA [13]. Since the areas of the end of the
parenting vessels are much smaller compared to the areas of the whole MAs,
regional loss, like Dice and IoU, possibly ignore these vessel ends, and place
too much emphasis on the MA bodies. However, correct segmentation of MA bodies
alone is not sufficient for further classification. Thus, metrics like Dice
and IoU are important but should not be the only metrics to evaluate the
performance of MA segmentation. Our results demonstrate that AOSLO-net not
only achieves high scores on these two metrics, but also captures the detailed
features of the feeding and draining vessels of MAs, and therefore promises to
be an effective MA segmentation method in the clinic.

## Methods

### Image pre-processing

As shown in Fig. 1(b), the raw AOSLO images are featured with intense
background noise and low contrast. Thus, we perform imaging pre-processing on
these raw images and generate multi-modality images to improve the
effectiveness of the training. First, we generate a set of perfusion maps by
tracing the blood flow in the MAs and micro-vessels using the pixel-by-pixel
standard deviation method on different frames of the AOSLO video [14]. As
shown in Fig. 1(c), the vessels with blood motion appear bright in the
perfusion map, while static tissue shows up as dark background. To improve the
quality of the perfusion maps, we apply the following methods to denoise and
enhance the images: (1) we employ fast non-local means method to remove the
background noise; (2) use normalization to make the pixel value lie in the
interval $[0,1]$; (3) apply contrast limits adaptive histogram equalization
(CLAHE) to enhance image contrast without over stretching the contrast in
specific areas and balance the overall contrast; (4) apply Gamma Correction to
remove some bright stripes on the background surrounding tissues caused by
CLAHE. The impact of each of these four pre-processing steps on the raw
perfusion maps is illustrated in Fig. 7 in SI.

Perfusion maps may not be able to accurately illustrate the geometries for all
MAs as thrombosis may occur in some of the MAs, which leads to presence of
nonperfused areas [14, 18]. Therefore, we have developed enhanced AOSLO images
to provide more details on the boundaries of each MA, as shown in Fig. 1(d).
The procedure of creating enhanced AOSLO images follows three main steps: (i)
taking the average over all the frames in MA video; (ii) reversing image
colour; (iii) performing local mean filtering. We note that in some enhanced
AOSLO images, the boundaries of MAs and their parenting vessels are not
clearly illustrated due to the low quality of the AOSLO images. Therefore, we
further generate a two-channel image set by concatenating perfusion map and
enhanced AOSLO images (see Fig. 1(e)), which use the information of the blood
flow inside MAs to compensate the missing information of the MA boundaries.

Due to the limited size of the AOSLO data set, we employ data augmentation to
increase the number of images for training the AOSLO-net. We apply three types
of transformations, including flip, rotate and scaling, to the AOSLO images
(perfusion map, enhance AOSLO and two-channel images) and their corresponding
masks. The augmentation procedure follows three steps: (1) images are flipped
horizontally and vertically with probability of 0.5; (2) flipped images are
rotated with angles in the set

$\\{0,\frac{2\pi}{N},\frac{4\pi}{N},\ldots,\frac{2(N-1)\pi}{N}\\}$

where $N$ is selected to be 32. (3) The rotated images are scaled with a
factor randomly selected between 0.7 and 1.4 to improve the robustness of
AOSLO-net on segmenting MAs with varying sizes. Typical examples of augmented
images are shown in Fig. 8 in SI.

### Architecture of AOSLO-net and network training

Inspired by the popular UNet structure [33], AOSLO-net is composed of two key
parts: encoder and decoder. The function of the encoder is to extract the
features of MAs at different levels whereas the decoder integrates these
extracted features to compose the segmentation results. Since the role of the
encoder is critical to the performance of the segmentation model, we adopt the
current SOTA image classification network EfficientNet-b3 [48] with a depth of
5 as the encoder in the AOSLO-net. We also apply transfer learning in AOSLO-
net through pre-training the EfficientNet-b3 using ImageNet [49] to achieve
quick convergence during training. The evolution of the loss in the training
process can be found in Figs. 11 and 13 in SI.

The pre-processed AOSLO images are split into 5 folds. While one fold is
reserved as test data, the rest 4 folds are used to train and validate the
AOSLO-net. We perform a 10-fold cross-validation using these 4 folds of
images, meaning that these images are further separated into 10 folds, with 9
folds used for training after augmentation and one fold used for validation.
Overall, 1600 augmented images are used to train the AOSLO-net. 10 images are
used for validation and 17 images are used to test the model performance.

The loss function is a combination of binary cross-entropy and Dice loss,

$Loss=BCE+\alpha\times Dice,$

where BCE and Dice are defined as

$\displaystyle BCE$ $\displaystyle=\sum_{n}\left[y_{n}\cdot\ln
x_{n}+(1-y_{n})\cdot\ln(1-x_{n})\right]$ $\displaystyle Dice$
$\displaystyle=\frac{2|X\cap Y|}{|X|+|Y|}.$

Here, $|X|,\ |Y|$ are model prediction and target respectively, $x_{n},\
y_{n}$ are pixel values in $|X|$ and $|Y|$; $\alpha$ is set to 0.2 to equalize
the contribution of BCE and Dice to the loss. We also introduce Hausdorff
distance, which is commonly used to describe the contour difference of two
shape, into the loss function, but it does not improve the segmentation
results (see Fig.6 in SI).

We use the Adam optimizer with learning rate of 0.001 and weight decay of
$10^{-8}$. During training, a Plateau scheduler is specified such that once
the validation loss does not decrease for $5$ epochs, the learning rate is
decreased to $1/10$ of the current value to facilitate the convergence of the
AOSLO-net. The training is initially set to last 200 epochs, but it may end
earlier due to the implementation of Plateau scheduler. The batch size is set
to 16.

### Post-Processing and Ensembling

Binarization. The pixel value of the outputted images from AOSLO-net lies
within range [0, 1] resulting from employing the sigmoid activation function
in the last layer of AOSLO-net. To quantify the geometries of MAs, we convert
these pixel values into a binary form, meaning that the pixel value is either
1 (belongs to an MA), or 0 (not MA). A threshold of 0.5 is applied to binarize
the segmented images in the current study.
Clearing. We note that some segmented images contain small fragments that are
mistakenly predicted as MAs. Thus, we specify an area threshold of 1024
pixels, below which the fragments are removed from the segmented images.
Ensembling. Following the work of [39], we employ ensembling method by
selecting three best models, out of the 10 trained models, based on their
performance on the validation set and perform a union of their outputs to
improve the model performance. Some examples illustrating the effect of post-
processing are shown in Fig. 14 in SI.

### Performance metrics

We evaluate the segmentation performance of AOSLO-net and other segmentation
models using the Dice coefficient and intersection of union (IoU), defined as
follows,

$\displaystyle Dice$ $\displaystyle=\frac{2|X\cap Y|}{|X|+|Y|},$ (1)
$\displaystyle IoU$ $\displaystyle=\frac{|X\cap Y|}{|X\cup Y|},$

where $|X|$ and $|Y|$ denote the matrix representations of the target image
and the corresponding prediction image, respectively.

### Data availability

The dataset used in the current study is not currently available publicly, but
it will be released upon request.

### Code availability

The code is not currently available publicly but could be released upon
publication of this manuscript.

#### Duality of Interest.

No potential conflicts of interest relevant to this article were reported.

#### Acknowledgements.

Q.Z., M.X., S.C., Y.D., H.L. and G.E.K. acknowledge the support from R01
HL154150 and U01 HL142518. K.S. and J.K.S. acknowledge the support by NEI
5R01EY024702-04 as well as grants from Research to Prevent Blindness, JDRF
3-SRA-2014-264-M-R, and the Massachusetts Lions Eye Research Fund. High
Performance Computing resources were provided by the Center for Computation
and Visualization at Brown University and the Extreme Science and Engineering
Discovery Environment (XSEDE), which is supported by National Science
Foundation grant numbers ACI-1053575, TG-DMS140007 and TG-MCB190045.

#### Author Contribution.

Q.Z., K.S., M.X., S.C., Y.D., H.L., J.K.S. and G.E.K. contributed to the study
concept and design. Q.Z., K.S., S.C., Y.D. and H.L. contributed to
acquisition, analysis, or interpretation of data. Q.Z., K.S., M.X., S.C.,
Y.D., H.L., J.K.S. and G.E.K. contributed to drafting of the manuscript. Q.Z.,
M.X., S.C., Y.D., H.L. and G.E.K. contributed to algorithm development. Q.Z.
and M.X. performed hardware implementation. Q.Z. and M.X. contributed to
statistical analysis. J.K.S. and G.E.K supervised the project.

## References

  * [1] Centers for Disease Control and Prevention.  https://www.cdc.gov/diabetes/index.html.
  * [2] National Institutes of Health.  https://www.nei.nih.gov/learn-about-eye-health/resources-for-health-educators/eye-health-data-and-statistics/diabetic-retinopathy-data-and-statistics.
  * [3] David A Antonetti, Paolo S Silva, and Alan W Stitt.  Current understanding of the molecular and cellular pathology of diabetic retinopathy.  Nature Reviews Endocrinology, 17(4):195–206, 2021.
  * [4] Grading diabetic retinopathy from stereoscopic color fundus photographs—an extension of the modified airlie house classification: Etdrs report number 10\.  Ophthalmology, 98(5, Supplement):786–806, 1991.
  * [5] Charles P Wilkinson, Frederick L Ferris III, Ronald E Klein, Paul P Lee, Carl David Agardh, Matthew Davis, Diana Dills, Anselm Kampik, R Pararajasegaram, Juan T Verdaguer, et al.  Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales.  Ophthalmology, 110(9):1677–1682, 2003.
  * [6] Tien Y Wong, Chui Ming Gemmy Cheung, Michael Larsen, Sanjay Sharma, and Rafael Simó.  Diabetic retinopathy.  Nature Reviews Disease Primers, 2(1216), 2016.
  * [7] May Phu Paing, Somsak Choomchuay, and MD Rapeeporn Yodprom.  Detection of lesions and classification of diabetic retinopathy using fundus images.  In 2016 9th Biomedical Engineering International Conference (BMEiCON), pages 1–5. IEEE, 2016.
  * [8] Qi Yan, Daniel E Weeks, Hongyi Xin, Anand Swaroop, Emily Y Chew, Heng Huang, Ying Ding, and Wei Chen.  Deep-learning-based prediction of late age-related macular degeneration progression.  Nature Machine Intelligence, 2(2):141–150, 2020.
  * [9] Akinori Mitani, Abigail Huang, Subhashini Venugopalan, Greg S Corrado, Lily Peng, Dale R Webster, Naama Hammel, Yun Liu, and Avinash V Varadarajan.  Detection of anaemia from retinal fundus images via deep learning.  Nature Biomedical Engineering, 4(1):18–27, 2020.
  * [10] Changyow C Kwan and Amani A Fawzi.  Imaging and biomarkers in diabetic macular edema and diabetic retinopathy.  Current Diabetes Reports, 19(10):95, 2019.
  * [11] Enrico Borrelli, Riccardo Sacconi, Maria Brambati, Francesco Bandello, and Giuseppe Querques.  In vivo rotational three-dimensional octa analysis of microaneurysms in the human diabetic retina.  Scientific Reports, 9(1):1–8, 2019.
  * [12] Yoshihiro Kaizu, Shintaro Nakao, Iori Wada, Mitsuru Arima, Muneo Yamaguchi, Keijiro Ishikawa, Masato Akiyama, Junji Kishimoto, Toshio Hisatomi, and Koh-Hei Sonoda.  Microaneurysm imaging using multiple en face oct angiography image averaging: morphology and visualization.  Ophthalmology Retina, 4(2):175–186, 2020.
  * [13] Michael Dubow, Alexander Pinhas, Nishit Shah, Robert F Cooper, Alexander Gan, Ronald C Gentile, Vernon Hendrix, Yusufu N Sulai, Joseph Carroll, Toco YP Chui, et al.  Classification of human retinal microaneurysms using adaptive optics scanning light ophthalmoscope fluorescein angiography.  Investigative Ophthalmology & Visual Science, 55(3):1299–1309, 2014.
  * [14] Miguel O Bernabeu, Yang Lu, Omar Abu-Qamar, Lloyd Paul Aiello, and Jennifer K Sun.  Estimation of diabetic retinal microaneurysm perfusion parameters based on computational fluid dynamics modeling of adaptive optics scanning laser ophthalmoscopy.  Frontiers in Physiology, 9:989, 2018.
  * [15] Beau J Fenner, Raymond LM Wong, Wai-Ching Lam, Gavin SW Tan, and Gemmy CM Cheung.  Advances in retinal imaging and applications in diabetic retinopathy screening: a review.  Ophthalmology and Therapy, 7(2):333–346, 2018.
  * [16] Alberto de Castro, Gang Huang, Lucie Sawides, Ting Luo, and Stephen A Burns.  Rapid high resolution imaging with a dual-channel scanning technique.  Optics Letters, 41(8):1881–1884, 2016.
  * [17] Vivian Schreur, Artin Domanian, Bart Liefers, Freerk G Venhuizen, B Jeroen Klevering, Carel B Hoyng, Eiko K de Jong, and Thomas Theelen.  Morphological and topographical appearance of microaneurysms on optical coherence tomography angiography.  British Journal of Ophthalmology, 103(5):630–635, 2019.
  * [18] He Li, Konstantina Sampani, Xiaoning Zheng, Dimitrios P Papageorgiou, Alireza Yazdani, Miguel O Bernabeu, George E Karniadakis, and Jennifer K Sun.  Predictive modelling of thrombus formation in diabetic retinal microaneurysms.  Royal Society Open Science, 7(8):201102, 2020.
  * [19] Justis P Ehlers, Kevin Wang, Amit Vasanji, Ming Hu, and Sunil K Srivastava.  Automated quantitative characterisation of retinal vascular leakage and microaneurysms in ultra-widefield fluorescein angiography.  British Journal of Ophthalmology, 101(6):696–699, 2017.
  * [20] Syna Sreng, Noppadol Maneerat, and Kazuhiko Hamamoto.  Automated microaneurysms detection in fundus images using image segmentation.  In 2017 International Conference on Digital Arts, Media and Technology (ICDAMT), pages 19–23. IEEE, 2017.
  * [21] Meysam Tavakoli, Sina Jazani, and Mahdieh Nazar.  Automated detection of microaneurysms in color fundus images using deep learning with different preprocessing approaches.  In Medical Imaging 2020: Imaging Informatics for Healthcare, Research, and Applications, volume 11318, page 113180E. International Society for Optics and Photonics, 2020.
  * [22] Caixia Kou, Wei Li, Wei Liang, Zekuan Yu, and Jianchen Hao.  Microaneurysms segmentation with a U-Net based on recurrent residual convolutional neural network.  Journal of Medical Imaging, 6(2):025008, 2019.
  * [23] R Murugan, Anitha Juliette Albert, and Deepak Kumar Nayak.  An automatic localization of microaneurysms in retinal fundus images.  In 2019 International Conference on Smart Structures and Systems (ICSSS), pages 1–5. IEEE, 2019.
  * [24] Yutong Xie, Jianpeng Zhang, Hao Lu, Chunhua Shen, and Yong Xia.  Sesv: Accurate medical image segmentation by predicting and correcting errors.  IEEE Transactions on Medical Imaging, 40(1):286–296, 2020.
  * [25] Jan Lammer, Sonja G Karst, Michael M Lin, Michael Cheney, Paolo S Silva, Stephen A Burns, Lloyd Paul Aiello, and Jennifer K Sun.  Association of microaneurysms on adaptive optics scanning laser ophthalmoscopy with surrounding neuroretinal pathology and visual function in diabetes.  Investigative Ophthalmology & Visual Science, 59(13):5633–5640, 2018.
  * [26] Stela Vujosevic, Stephen J Aldington, Paolo Silva, Cristina Hernández, Peter Scanlon, Tunde Peto, and Rafael Simó.  Screening for diabetic retinopathy: new perspectives and challenges.  The Lancet Diabetes & Endocrinology, 8(4):337–347, 2020.
  * [27] Ramachandran Rajalakshmi.  The impact of artificial intelligence in screening for diabetic retinopathy in india.  Eye, 30(34):420–421, 2020.
  * [28] Jie He, Tingyi Cao, Feiping Xu, Shasha Wang, Haiqi Tao, Tao Wu, Liyan Sun, and Jili Chen.  Artificial intelligence-based screening for diabetic retinopathy at community hospital.  Eye, 34(3):572–576, 2020.
  * [29] Andrzej Grzybowski, Piotr Brona, Gilbert Lim, Paisan Ruamviboonsuk, Gavin SW Tan, Michael Abramoff, and Daniel SW Ting.  Artificial intelligence for diabetic retinopathy screening: a review.  Eye, pages 1–10, 2019.
  * [30] Harry Pratt, Frans Coenen, Deborah M Broadbent, Simon P Harding, and Yalin Zheng.  Convolutional neural networks for diabetic retinopathy.  Procedia Computer Science, 90:200–205, 2016.
  * [31] Brodie Fischbacher, Sarita Hedaya, Brigham J Hartley, Zhongwei Wang, Gregory Lallos, Dillion Hutson, Matthew Zimmer, Jacob Brammer, and Daniel Paull.  Modular deep learning enables automated identification of monoclonal cell lines.  Nature Machine Intelligence, pages 1–9, 2021.
  * [32] Paweł Liskowski and Krzysztof Krawiec.  Segmenting retinal blood vessels with deep neural networks.  IEEE Transactions on Medical Imaging, 35(11):2369–2380, 2016.
  * [33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-assisted Intervention, pages 234–241. Springer, 2015.
  * [34] Xiao Xiao, Shen Lian, Zhiming Luo, and Shaozi Li.  Weighted res-unet for high-quality retina vessel segmentation.  In 2018 9th International Conference on Information Technology in Medicine and Education (ITME), pages 327–331. IEEE, 2018.
  * [35] Yijun Bao, Somayyeh Soltanian-Zadeh, Sina Farsiu, and Yiyang Gong.  Segmentation of neurons from fluorescence calcium recordings beyond real time.  Nature Machine Intelligence, pages 1–11, 2021.
  * [36] Qiangguo Jin, Zhaopeng Meng, Tuan D Pham, Qi Chen, Leyi Wei, and Ran Su.  Dunet: A deformable network for retinal vessel segmentation.  Knowledge-Based Systems, 178:149–162, 2019.
  * [37] Md Zahangir Alom, Chris Yakopcic, Mahmudul Hasan, Tarek M Taha, and Vijayan K Asari.  Recurrent residual U-Net for medical image segmentation.  Journal of Medical Imaging, 6(1):014006, 2019.
  * [38] Liangzhi Li, Manisha Verma, Yuta Nakashima, Hajime Nagahara, and Ryo Kawasaki.  Iternet: Retinal image segmentation utilizing structural redundancy in vessel networks.  In The IEEE Winter Conference on Applications of Computer Vision, pages 3656–3665, 2020.
  * [39] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature Methods, 18(2):203–211, 2021.
  * [40] Yang Lu, Miguel O Bernabeu, Jan Lammer, Charles C Cai, Martin L Jones, Claudio A Franco, Lloyd Paul Aiello, and Jennifer K Sun.  Computational fluid dynamics assisted characterization of parafoveal hemodynamics in normal and diabetic eyes using adaptive optics scanning laser ophthalmoscopy.  Biomedical Optics Express, 7(12):4958–4973, 2016.
  * [41] Caroline A Schneider, Wayne S Rasband, and Kevin W Eliceiri.  NIH image to imagej: 25 years of image analysis.  Nature Methods, 9(7):671, 2012.
  * [42] Zhengxin Zhang and Qingjie Liu.  Road extraction by deep residual U-Net.  IEEE Geoscience and Remote Sensing Letters, PP, 11 2017.
  * [43] Pavel Yakubovskiy.  Segmentation models pytorch.  https://github.com/qubvel/segmentation_models.pytorch, 2020.
  * [44] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 764–773, 2017.
  * [45] Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu.  scikit-image: image processing in python.  PeerJ, 2:e453, 2014.
  * [46] Aude Couturier, Valérie Mané, Sophie Bonnin, Ali Erginay, Pascale Massin, Alain Gaudric, and Ramin Tadayoni.  Capillary plexus anomalies in diabetic retinopathy on optical coherence tomography angiography.  Retina, 35(11):2384–2391, 2015.
  * [47] James Kang Hao Goh, Carol Y Cheung, Shaun Sebastian Sim, Pok Chien Tan, Gavin Siew Wei Tan, and Tien Yin Wong.  Retinal imaging techniques for diabetic retinopathy screening.  Journal of Diabetes Science and Technology, 10(2):282–294, 2016\.
  * [48] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
  * [49] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern recognition, pages 248–255. IEEE, 2009.

##  Appendix A Data

###  A.1 Preprocessing on Perfusion Map

Figure 7: Figure 7. An example of the sequential image preprocessing steps for
perfusion map. (a) Raw perfusion map, which contains background noise and
suffers from low image contrast. (b) Perfusion map after applying the non-
local means filtering method, which creates a denoised image. (c) Perfusion
map after intensity normalization in the range of $[0,1]$, which enhances the
contrast. (d) Perfusion map after employing contrast limited adaptive
histogram equalization (CLAHE). (e) Perfusion map after employing Gamma
correction. We applied these sequential preprocessing steps for all the
perfusion maps in AOSLO-net training and testing.

###  A.2 Data augmentation

Figure 8: Figure 8. Examples of data augmentation. (a) Two-modality image
(using the processed perfusion map and the enhanced AOSLO image as two
channels in RGB space). (b)-(e) Augmented images from (a) by performing
different transformations: (b) flipping vertically, rotating with an angle
$\theta={\pi}/{6}$ and scaling with a factor $\lambda<1$; (c) flipping
horizontally, $\theta={\pi}/{3}$ and $\lambda>1$; (d) flipping both
horizentally and vertically, $\theta={\pi}/{4}$ and $\lambda<1$; (e) no
flipping, $\theta={3\pi}/{4}$ and $\lambda\sim 1$. The rotating angle was
uniformly distributed, while the scaling factor was randomly selected in an
appropriate range when we performed data augmentation on our AOSLO dataset.
Figure 9: Figure 9. Performance of AOSLO-net and nnU-Net with different input
modalities.  (a) Histogram and density distribution of Dice score obtained by
using the perfusion maps as input. The result of AOSLO-net is denoted by cyan
color whereas the result of nnU-Net is denoted as red color. The mean Dice
score (denoted by dash line) of AOSLO-net is slightly higher than that of nnU-
Net. (b) Histogram and density distribution of Dice score obtained by using
the enhanced images as input. In this case, the mean Dice score of AOSLO-net
is slightly lower than that of nnU-Net. Moreover, the AOSLO-net is more likely
to make low-quality predictions. (c) Histogram and density distribution of
Dice score obtained by using two-modality images as input, where the AOSLO-net
performs better than the nnU-Net. (d) The statistics of AOSLO-net based on
different input modalities. Using two-modality images as input results in the
best performance.

##  Appendix B AOSLO-net model

###  B.1 Network

Figure 10: Figure 10. A list of segmentation models that have been tested in
the current work. The arrow points to the model with a better performance in
terms of Dice and IoU score.

UNet. The vanilla version of UNet[33] was proposed in 2015. This network tends
to erroneously predict the background vessels as MAs, and overall the
predicted MA shapes are not satisfactory.
Deformable UNet. Deformable UNet[36] uses the deformable convolution layer
instead of the original convolution layer. Deformable UNet shows improved
performance on distinguishing the MAs from the vessels, but it fails to
capture the MA shape and vessels connected to MAs.
ResUNet. We implemented ResUNet with `Segmentation Models Pytorch`[43], a
python package. ResNet101 was used as backbone, which included more parameters
than the official implementation. As a result, this UNet is able to better
capture the MA shape and the inlet outlet vessels. However, the overall
performance of this UNet is not sufficient for MA shape quantification.
ResNet-based Deformable UNet. The main body of this kind of UNet is the same
as the ResNet-based UNet, but the convolution layers are replaced to be
deformable ones. We implemented it by substituting ordinary convolution kernel
in `Segmentation Models Pytorch` with a deformable convolution kernel[44]. But
these modifications do not improve model performance on segmenting AOSLO
images.
EfficientNet-based UNet. Using EfficientNet-b3[48], a SOTA classification
network, as encoder in AOSLO-net has show significant improvements over the
models listed above. We use the implementation from `Segmentation Models
Pytorch`.

###  B.2 Loss

Binary CrossEntropy (BCE) is the most common loss function for classification
and segmentation. However, in our study, there is also a strong imbalance
since the MA areas are relatively small compared with the whole image. Under
these circumstances, the BCE loss was overwhelmed and the neural network
predicted MAs as background. To overcome this imbalance, we use regional loss
based on Dice and IoU to maximize the intersection ratio of the prediction and
the ground truth.

Despite the advantage of regional loss like Dice and IoU, they may cause
oscillation of the training loss curve. Thus, we added BCE as a smoothing loss
to the regional loss. We used a coefficient $\alpha=0.2$ to equalize
contribution of BCE and regional loss as shown in Figure 11.

Figure 11: Figure 11. Training Losses and Dice score of AOSLO-net versus
training epochs. The original Dice loss (in green color), which is defined as
$1-\text{Dice}$, is much larger compared to the binary cross entropy (BCE)
loss (in orange). In order to balance the contribution of two terms in the
total loss function, a weighting coefficient $0.2$ was applied for the Dice
loss. The convergence of the rescaled Dice loss, $0.2\times(1-\text{Dice})$,
is shown in red color, while the total loss is shown in blue.

We also tested the Hausdorff distance, which is defined as

$d_{H}(X,Y)=\max\\{\sup_{x\in X}d(x,Y),\sup_{y\in Y}d(y,X)\\},$ (2)

where $X$ is prediction and $Y$ is target, $d$ is metric of the space, which
is the Eucleadian distance in our case. The Hausdorff distance is used to
describe the contour difference of two shape. However, as shown in Fig. 12,
the effect of using the Hausdorff distance as part of the loss was not notable
to our current method.

Figure 12: Figure 12. Performance of AOSLO-net and nnU-Net with and without
Hausdorff distance loss. (a) Histogram and density distribution of Dice score
when the models are trained with Hausdorff distance as part of the loss. (b)
Histogram and density distribution of Dice score when the models are trained
without using the Hausdorff distance. We observe that applying the Hausdorff
distance did not improve the performace of AOSLO-net. Instead, the mean value
of Dice score dropped from 0.7816 to 0.7774 for the AOSLO-net.

###  B.3 Optimizer

We first used the SGD as optimizer, the default for many deep learning based
computer vision problems and we found that loss descending rate is very slow.
Thus, we employ Adam optimizer and the training usually converged in 20
epochs. More importantly, after using Adam as optimizer, the segmentation
performance on validation and test set is also improved.
In order to prevent the overfitting, we used the Plateau learning rate
scheduler. When the validation loss does not decrease for $5$ epochs, the
scheduler decreases the learning rate to $1/10$ of the current one, so the
network converges before overfitting. The effect of using the scheduler is
illustrated in Figure 13, where the training finished in less than 15 epochs,
while nnU-Net will use 1000 epochs by default.

Figure 13: Figure 13. Loss and Dice score of AOSLO-net in training and
validation. The AOSLO-net undergoes a quick convergence and achieves a good
validation Dice score upon convergence.

##  Appendix C Post Processing

###  C.1 Binarization and Clearing

Figure 14: Figure 14. Two examples of using image binarization and clearing to
remove vessels that are not connected to MAs. (a) The raw output image of
AOSLO-net. (b) Output image after binarization. (c) Output images after
clearing.

###  C.2 Morphology quantification results for the BNR factor of MAs

Figure 15: Figure 15. Quantification results of the BNR (body-to-neck ratio)
factor for the segmented MAs using AOSLO-net and nnU-Net trained with normal,
short and thick masks. First row: Comparisons of BNR quantification results
based on AOSLO-net segmentation maps and the corresponding adopted three
different types of MA masks (normal, short and thick masks as shown in columns
1–3); Second row: Comparisons of BNR quantification results based on nnU-Net
segmentation maps and the corresponding adopted MA masks (normal, short and
thick masks as shown in columns 1–3). The dot size and color represent the
estimated BNR value difference between using the MA segmentation maps and the
original ground truth masks. AOSLO-net segmentation maps enable to generate
more accurate BNR estimation results (BNR difference values are relatively
smaller) in comparison to nnU-Net segmentation maps. Moreover, AOSLO-net
trained with thick masks can capture more accurate the MA geometry features
(especially for the narrowest calibers). The corresponding BNR estimation
results are closer to the BNR values estimated by using the normal masks.


# Low-Complexity Signal Detection for the Splitting Receiver Scheme

Yanyan Wang, _Member, IEEE_ , Qidi Li, Xiaohu Tang, _Senior Member, IEEE_

###### Abstract

†† The work of Y. Wang was supported by the National Natural Science
Foundation of China under Grant 62201476, and in part by the Natural Science
Foundation of Sichuan Province under Grant 2021YJ0549.(_Corresponding author:
Yanyan Wang_.) Y. Wang, Q. Li, and X. Tang are with School of Information
Science and Technology, Southwest Jiaotong University, Chengdu, China (email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>xhutang@swjtu.edu.cn).

This letter proposes a low-complexity signal detection method for the
splitting receiver scheme, which achieves an excellent symbol error rate (SER)
performance. Based on the three-dimensional (3D) received signal of the
splitting receiver, we derive an equivalent two-dimensional (2D) signal model
and develop a low-complexity signal detection method for the practical
modulation scheme. The computational complexity of the proposed signal
detector is reduced by at least 11.1 times as compared to the 3D signal
detection scheme shown in [1]. Simulation results demonstrate that the
approximation SER achieved by the low-complexity signal detection is very
close to that of the accurate SER at a certain power splitting ratio.

###### Index Terms:

Splitting receiver, signal splitting, maximum likelihood (ML) detector, low-
complexity signal detection.

##  I Introduction

Receiver design has received a amount of attention in wireless communication
systems. The existed receiver schemes are generally categorized into two
classes: coherent receiver [2] and non-coherent receiver [3]. The coherent
receiver is based on the coherent detection (CD), which has been extensively
utilized in cellular communications and local area networks because of the
better performance gain than those of the non-coherent receivers. Although the
power detection (PD) based non-coherent receiver suffers from performance loss
compared to the coherent receivers, it can provide the advantages of low power
consumption and low cost, which makes it attractive for the internet of things
(IoT), backscatter communications[4, 5], etc.

Recently, a new receiver architecture was proposed in [6, 1, 7, 8]. The
motivations of the works are to develop a practical radio-frequency (RF)
splitting receiver for providing better performance than the traditional non-
splitting receiver. For the splitting receiver architecture, the received
signal is split into two streams. Then, the two streams separately go through
the CD and non-coherent receivers and thus obtain the three-dimensional (3D)
signals. Ignoring the antenna noise, the work in [6] proposed the splitting
receiver architecture. Based on the received signal model, the logarithmic
likelihood ratio detection algorithm [9] was utilized to detect the
transmitted signals. The follow-up work in [1] formulated a more practical
system model, where both the antenna noise and post-processing noises were
considered. The maximum likelihood (ML) detection method was used in [1] for
recovering the transmitted signals based on the 3D received signal. Since the
likelihood function has no closed-form expression for the splitting receiver
scheme, the computational complexity of the optimal ML detector is very high.
To reduce the computational complexity, a relatively low-complexity signal
detection method was proposed in [1] through deriving the closed-form
likelihood function expression. However, the likelihood function expression is
still very complicated. Particularly, when the modulation order is large, the
signal detection complexity will further increase.

In this letter, we propose a low-complexity signal detection scheme for the
splitting receiver architecture and analyze the symbol error rate (SER)
performance. Specifically, based on the 3D splitting received signal, we
formulate an equivalent two-dimensional (2D) signal model. As a result, the
signal detection is transformed into the minimum distance detection by
changing the coordinate system. The complexity of the proposed signal
detection method is greatly reduced compared with the ML detection that based
on the 3D received signal. Precisely, the computational complexity of the
proposed signal detection method is reduced by about $11.1$ times compared to
that of the method in [1]. Notably, simulation results show that the SER
performance of the low-complexity detector is mostly indistinguishable from
that of the optimal detector at a certain power splitting ratio.

The remainder of the paper is organized as follows. In Section II, the
splitting receiver with considering practical noise model is illustrated. In
Section III, the low-complexity detector of the splitting receiver is
developed. Numerical results are provided in Section IV. Section V analyzes
the computational complexity of the proposed signal detection scheme. Finally,
Section VI concludes the letter.

Figure 1: The communication system model with splitting receiver architecture.

##  II System Model

Fig. 1 shows a single-antenna communication system with consideration of the
antenna noise $\tilde{W}^{{}^{\prime}}$, where a splitting receiver
architecture is utilized at the receiver side. The information bits are
modulated to generate complex transmitted signals ${{\tilde{X}}}$ with average
transmit power $P$ and then propagated through the wireless fading channel
$\tilde{h}$. Assuming that the channel gain is $|\tilde{h}|$ and phase shift
of the channel is $\phi$, the equivalent complex channel is written as
$\tilde{h}\triangleq|\tilde{h}|e^{j\phi}$. At the receiver side, the received
signal is split into two streams by a power splitter $\rho$. Then, the two
streams respectively go through the CD and PD-based non-coherent receivers.
Finally, the signals from the CD and PD receivers are jointly used for
information detection.

Based on the splitting receiver architecture, the received signals are
expressed as

${{{\tilde{Y}}}^{{}^{\prime}}_{1}}=\sqrt{\rho}(\sqrt{P}\tilde{h}{{\tilde{X}}}+{{\tilde{W}^{{}^{\prime}}}})+{{\tilde{Z}^{{}^{\prime}}}},$
(1)
${{{Y}}^{{}^{\prime}}_{2}}=\eta(1-\rho)|\sqrt{P}\tilde{h}{{\tilde{X}}}+{{\tilde{W}^{{}^{\prime}}}}|^{2}+{{N}}^{{}^{\prime}},$
(2)

where $\rho\in[0,1]$ and $\eta\in(0,1]$ respectively denote the power
splitting ratio and the conversion efficiency, which have been analyzed in
[1]. Specifically, when $\rho=0$ and $\rho=1$, the splitting receiver is
degraded to the PD receiver and the CD receiver, respectively.

For the ease of analysis, let
${{{\tilde{Y}}}_{1}}=e^{-j\phi}{{{\tilde{Y}}}^{{}^{\prime}}_{1}}$ and
${{{Y}}_{2}}={{{Y}}^{{}^{\prime}}_{2}}/\eta$. Then, we can obtain
$e^{-j\phi}{{{\tilde{Y}}}^{{}^{\prime}}_{1}}=e^{-j\phi}\sqrt{\rho}(\sqrt{P}|\tilde{h}|e^{j\phi}{{\tilde{X}}}+{{\tilde{W}}^{{}^{\prime}}})+e^{-j\phi}{{\tilde{Z}}^{{}^{\prime}}}$
and
${{{Y}}^{{}^{\prime}}_{2}}/\eta=|e^{-j\phi}|^{2}(1-\rho)\big{|}\sqrt{P}{|\tilde{h}|e^{j\phi}}{{\tilde{X}}}+{{\tilde{W}}^{{}^{\prime}}}\big{|}^{2}+{{N}}^{{}^{\prime}}/\eta$.
Hence, the received signals are rewritten as

${{{\tilde{Y}}}_{1}}=\sqrt{\rho}(\sqrt{P}|\tilde{h}|{{\tilde{X}}}+{{\tilde{W}}})+{{\tilde{Z}}},$
(3)
${{{Y}}_{2}}=(1-\rho)\big{|}\sqrt{P}{|\tilde{h}|}{{\tilde{X}}}+{{\tilde{W}}}\big{|}^{2}+{{N}},$
(4)

where ${\tilde{W}}\triangleq e^{-j\phi}{{\tilde{W}}^{{}^{\prime}}}$,
$\tilde{Z}\triangleq e^{-j\phi}{{\tilde{Z}^{{}^{\prime}}}}$ and
${{N}}\triangleq{{N}}^{{}^{\prime}}/\eta$. Notably, ${\tilde{W}}$, $\tilde{Z}$
and ${{N}}$ are called as the antenna noise, conversion noise, and rectifier
noise, respectively. These noises still meet zero-mean Gaussian distribution,
which follow $\tilde{W}\sim\mathcal{CN}(0,\sigma_{\textrm{A}}^{2})$,
$\tilde{Z}\sim\mathcal{CN}(0,\sigma_{\textrm{cov}}^{2})$, and
$N\sim\mathcal{N}(0,\sigma_{\textrm{rec}}^{2})$, respectively.

Considering the $M$-quadrature amplitude modulation (QAM) and $M$-phase shift
keying (PSK), the distributions of the transmitted and received signals from
(3) and (4) are illustrated in Fig. 2. It is observed that the constellation
points of the transmitted signals $\tilde{X}$ are distributed on the 2D I-Q
plane. At the receiver side, the received signals
$({{{\tilde{Y}}}_{1}},{{{Y}}_{2}})$ are distributed on the 3D paraboloid in
the I-Q-P space when the power splitting ratio $\rho\in(0,1)$. Specifically,
we observe that the received signal space of the traditional coherent
($\rho=1$) and PD-based non-coherent ($\rho=0$) receivers are respectively 2D
and 1D. Therefore, compared with the traditional receivers, the splitting
receiver architecture spreads the received signal space.

Figure 2: Distributions of the transmitted and received signals.

##  III  Proposed Low-complexity Signal Detection

In this section, we project the 3D received signal with high-dimensional space
into the 2D signal space and derive an equivalent 2D received signal. Based on
the simplified 2D signal model, a low-complexity signal detection is developed
for the practical modulation scheme. Note that when simplifying the signal
model, the $M$-QAM and $M$-PSK schemes are exploited in this letter, which
include the magnitude and phase information for the transmitted signals.

###  III-A Simplified 2D Signal Model

Without loss of generality, scaling (3) by $\sqrt{\rho}\sqrt{P}|\tilde{h}|$
and (4) by $(1-\rho)P|\tilde{h}|^{2}$, the output signal of the splitting
receiver is expressed as

${{{\tilde{Y}}}_{1}}={{\tilde{X}}}+\frac{{{\tilde{W}}}}{\sqrt{P}|\tilde{h}|}+\frac{{{\tilde{Z}}}}{\sqrt{\rho}\sqrt{P}|\tilde{h}|},$
(5)
${{{Y}}_{2}}=\bigg{|}{{\tilde{X}}}+\frac{{{\tilde{W}}}}{\sqrt{P}|\tilde{h}|}\bigg{|}^{2}+\frac{N}{(1-\rho)P|\tilde{h}|^{2}}.$
(6)

From (5) and (6), as the signal power $P$ increases, the 3D received signal
$({{{\tilde{Y}}}_{1}},{{{Y}}_{2}})$ can be expressed as the equivalent 2D
signal ${{\tilde{Y}}}$, where received signal ${{{Y}}_{2}}$ provides the
amplitude information, and the received signal ${{{\tilde{Y}}}_{1}}$ offers
the phase information. Defining
$\tilde{S}={{\tilde{X}}}+\frac{{{\tilde{W}}}}{\sqrt{P}|\tilde{h}|}$,
$\tilde{Z}_{s}=\frac{{{\tilde{Z}}}}{\sqrt{\rho}\sqrt{P}|\tilde{h}|}$ and
$N_{s}=\frac{N}{(1-\rho)P|\tilde{h}|^{2}}$, the simplified 2D signal is
rewritten as

$\displaystyle{\tilde{Y}}$
$\displaystyle\triangleq|\tilde{Y}|\textrm{e}^{j\angle{{{\tilde{Y}}}}}\triangleq\sqrt{Y_{2}}\textrm{e}^{j\angle{{{\tilde{Y}}}_{1}}}$
$\displaystyle=\sqrt{|\tilde{S}|^{2}+{{N}}_{s}}\textrm{e}^{j\angle(\tilde{S}+{{\tilde{Z}}}_{s})},$
(7)

where $|\tilde{Y}|\triangleq\sqrt{Y_{2}}$ and
$\textrm{e}^{j\angle{{{\tilde{Y}}}}}\triangleq\textrm{e}^{j\angle{{{\tilde{Y}}}_{1}}}$.
In the practical wireless communication system, the conversion noise
$\tilde{Z}$ is generally much stronger than the rectifier noise $N$ [7].
Therefore, ignoring the noise term ${{N}}_{s}$ in the high signal-to-noise
ratio (SNR) regime, (III-A) is approximated by

$\displaystyle{\tilde{Y}}$
$\displaystyle\approx|{{\tilde{S}}}|\textrm{e}^{j\frac{{{\tilde{S}}}+\tilde{Z}_{s}}{|{{\tilde{S}}}+\tilde{Z}_{s}|}}\approx|{{\tilde{S}}}|\textrm{e}^{j\angle{{\tilde{S}}}+j\frac{{\tilde{Z}}_{s}}{|{{\tilde{S}}}|}}$
$\displaystyle\approx|{{\tilde{S}}}|\textrm{e}^{j\angle{{\tilde{S}}}}\textrm{e}^{j\frac{{{{Z}}}_{p}}{|{{\tilde{S}}}|}}$
$\displaystyle={{\tilde{S}}}\left(\cos\left(\frac{{{{Z}}}_{p}}{|{{\tilde{S}}}|}\right)+j\sin\left(\frac{{{{Z}}}_{p}}{|{{\tilde{S}}}|}\right)\right),$
(8)

where ${{\tilde{S}}}=|{{\tilde{S}}}|\textrm{e}^{j\angle{{\tilde{S}}}}$ and
${{{Z}}}_{p}$ is the projection of ${\tilde{Z}_{s}}$ to ${{\tilde{S}}}$, which
follows ${{{Z}}}_{p}\sim\mathcal{N}(0,\frac{\sigma_{\textrm{cov}}^{2}}{2\rho
P|\tilde{h}|^{2}})$. By adopting the first-order Taylor-series approximation,
(III-A) is simplified as

$\displaystyle{\tilde{Y}}$
$\displaystyle\approx{{\tilde{S}}}\left(1+j\frac{{{{Z}}}_{p}}{|{{\tilde{S}}}|}\right)={{\tilde{S}}}+j{{{Z}}}_{p}\textrm{e}^{j\angle{{\tilde{S}}}}.$
(9)

As $P$ increases, it is observed that the argument of the complex number
${\tilde{S}}$ is almost the same as the ${\tilde{X}}$, i.e.,
$\textrm{e}^{j\angle{{\tilde{S}}}}\approx\textrm{e}^{j\angle{{\tilde{X}}}}$.
We define the argument of the complex number ${\tilde{X}}$ as $\theta$, then
(9) is approximated as

$\displaystyle{\tilde{Y}}$
$\displaystyle\approx{{\tilde{S}}}+\textrm{e}^{j\frac{\pi}{2}}{{{Z}}}_{p}\textrm{e}^{j\theta}={{\tilde{S}}}+{{{Z}}}_{p}\textrm{e}^{j(\frac{\pi}{2}+\theta)}$
$\displaystyle={{\tilde{X}}}+\frac{{{\tilde{W}}}}{\sqrt{P}|\tilde{h}|}+{{{Z}}}_{p}\textrm{e}^{j(\frac{\pi}{2}+\theta)}.$
(10)

By convenience, multiplying ${{{\tilde{Y}}}}$ by $\sqrt{P}|\tilde{h}|$ in
(III-A), the simplified 2D signal model can be rewritten as

${{{\tilde{Y}}}}=\sqrt{P}|\tilde{h}|{{\tilde{X}}}+{{\tilde{W}}}+e^{j(\theta+\frac{\pi}{2})}\frac{{{{Z_{r}}}}}{\sqrt{\rho}},$
(11)

where ${{{Z_{r}}}}=\sqrt{\rho P}|\tilde{h}|{{{Z}}}_{p}$, which denotes the
real AWGN with mean zero and variance $\frac{\sigma_{\textrm{cov}}^{2}}{2}$.
Since $e^{j(\theta+\frac{\pi}{2})}=-\sin\theta+j\cos\theta$, the 2D signal is
further given by

${{{\tilde{Y}}}}=\sqrt{P}|\tilde{h}|{{\tilde{X}}}+{{{W}}}_{r}-\sin\theta\frac{{{{Z_{r}}}}}{\sqrt{\rho}}+j({{{W}}}_{i}+\cos\theta\frac{{{{Z_{r}}}}}{\sqrt{\rho}}),$
(12)

where $\sin\theta=\frac{{{{X}}}_{i}}{|\tilde{X}|}$ and
$\cos\theta=\frac{{{{X}}}_{r}}{|\tilde{X}|}$. Assuming that ${Y}_{r}$ and
${Y}_{i}$ are respectively the real part and imaginary part of the output
signal $\tilde{Y}$. According to the definition of the joint PDF [10, (12)],
the joint PDF of ${Y}_{r}$ and ${Y}_{i}$ for given $\tilde{X}$ is represented
as

$\displaystyle f_{{Y}_{r},{Y}_{i}}$
$\displaystyle({y}_{r},{y}_{i}|\tilde{X})=\frac{1}{2\pi\sigma_{{Y}_{r}}\sigma_{{Y}_{i}}\sqrt{1-\xi^{2}}}\exp\biggl{(}-\frac{1}{2({1-\xi^{2}})}$
$\displaystyle\biggl{(}\frac{({y}_{r}-\sqrt{P}|\tilde{h}|{{{X}}}_{r})^{2}}{\sigma_{{Y}_{r}}^{2}}+\frac{({y}_{i}-\sqrt{P}|\tilde{h}|{{{X}}}_{i})^{2}}{\sigma_{{Y}_{i}}^{2}}$
$\displaystyle-\frac{2\xi({y}_{r}-\sqrt{P}|\tilde{h}|{{{X}}}_{r})({y}_{i}-\sqrt{P}|\tilde{h}|{{{X}}}_{i})}{\sigma_{{Y}_{r}}\sigma_{{Y}_{i}}}\biggr{)}\biggr{)},$
(13)

where
$\sigma_{{Y}_{r}}^{2}=\frac{\sigma_{\textrm{A}}^{2}}{2}+\frac{\sin^{2}\theta\sigma_{\textrm{cov}}^{2}}{2\rho}$
denots the variance of the received signal ${Y}_{r}$, and
$\sigma_{{Y}_{i}}^{2}=\frac{\sigma_{\textrm{A}}^{2}}{2}+\frac{\cos^{2}\theta\sigma_{\textrm{cov}}^{2}}{2\rho}$
represents the variance of the received signal ${Y}_{i}$. The means of
${Y}_{r}$ and ${Y}_{i}$ are $\sqrt{P}|\tilde{h}|{{{X}}}_{r}$ and
$\sqrt{P}|\tilde{h}|{{\tilde{X}}}_{i}$, respectively. Note that $\xi$ denotes
the correlation coefficient, which is calculated as
$\xi=-\frac{\sin\theta\cos\theta\sigma_{\textrm{cov}}^{2}}{2\rho\sigma_{{Y}_{r}}\sigma_{{Y}_{i}}}$.

###  III-B Low-complexity Signal Detector

As seen in (III-A), the variables ${Y}_{r}$ and ${Y}_{i}$ are correlated. To
remove the correlation of the received signal ${Y}_{r}$ and ${Y}_{i}$, we
express the joint PDF of the received signal in the new coordinate system.
Since the detection error probability remains invariant for any rotation of
the coordinate system, the PDF of the received signal can be represented as
another form by rotating the In-Phase and Quadrature axes [11]. In the new
coordinate system, the received signal $({Y}_{r},{Y}_{i})$ is changed as
$({Y}_{r}^{\prime},{Y}_{i}^{\prime})$, which is given by

$\displaystyle{Y}_{r}^{\prime}={Y}_{r}\cos\theta+{Y}_{i}\sin\theta-\sqrt{P}|\tilde{h}||\tilde{X}|,$
$\displaystyle{Y}_{i}^{\prime}={Y}_{i}\cos\theta-{Y}_{r}\sin\theta.$ (14)

Note that ${Y}_{r}^{\prime}$ and ${Y}_{i}^{\prime}$ for the In-Phase and
Quadrature branches are independent of each other. Then the PDF of the
received signal given $\tilde{X}$ in (III-A) becomes

$\displaystyle
f_{{Y}_{r}^{\prime},{Y}_{i}^{\prime}}({y}_{r}^{\prime},{y}_{i}^{\prime}|\tilde{X})=\frac{1}{2\pi\sigma_{{Y}_{r}^{\prime}}\sigma_{{Y}_{i}^{\prime}}}\exp\left(-\left(\frac{({y}_{r}^{\prime})^{2}}{2\sigma_{{Y}_{r}^{\prime}}^{2}}+\frac{({y}_{i}^{\prime})^{2}}{2\sigma_{{Y}_{i}^{\prime}}^{2}}\right)\right)$
$\displaystyle\\!\\!=\\!\\!\frac{1}{\pi\sqrt{\sigma_{\textrm{A}}^{2}(\sigma_{\textrm{A}}^{2}+\frac{\sigma_{\textrm{cov}}^{2}}{\rho})}}\exp\left(-\left(\frac{({y}_{r}^{\prime})^{2}}{\sigma_{\textrm{A}}^{2}}+\frac{({y}_{i}^{\prime})^{2}}{\sigma_{\textrm{A}}^{2}+\frac{\sigma_{\textrm{cov}}^{2}}{\rho}}\right)\right),$
(15)

where $\sigma_{{Y}_{r}^{\prime}}^{2}=\frac{\sigma_{\textrm{A}}^{2}}{2}$ and
$\sigma_{{Y}_{i}^{\prime}}^{2}=\frac{\sigma_{\textrm{A}}^{2}}{2}+\frac{\sigma_{\textrm{cov}}^{2}}{2\rho}$,
which means that the noise projections on the In-Phase axis and Quadrature
axis are $\frac{\sigma_{\textrm{A}}^{2}}{2}$ and
$\frac{\sigma_{\textrm{A}}^{2}}{2}+\frac{\sigma_{\textrm{cov}}^{2}}{2\rho}$,
respectively.

Based on the received signal PDF
$f_{Y^{\prime}_{r},Y^{\prime}_{i}}\big{(}y^{\prime}_{r},y^{\prime}_{i}|\tilde{X}\big{)}$
in (III-B), the low-complexity signal detection rule is expressed as

$\displaystyle\hat{X}=\mathop{\arg\max}\limits_{\tilde{X}\in\Omega_{\textrm{gen}}}f_{Y^{\prime}_{r},Y^{\prime}_{i}}\big{(}y^{\prime}_{r},y^{\prime}_{i}|\tilde{X}\big{)},$
(16)

where $\Omega_{\textrm{gen}}$ is the set of the transmitted constellation
points. To detect the transmitted signal $\tilde{X}_{k}$ for the $k$-th
symbol, we need to have
$f_{Y^{\prime}_{r},Y^{\prime}_{i}}\big{(}{y^{\prime}}_{r},y^{\prime}_{i}|\tilde{X}_{k}\big{)}>f_{Y^{\prime}_{r},Y^{\prime}_{i}}\big{(}{y^{\prime}}_{r},y^{\prime}_{i}|\tilde{X}_{j}\big{)}$
for different $k$ and $j$, i.e.

$\displaystyle\exp$
$\displaystyle\left(-\left(\frac{({y^{\prime}}_{r}^{(k)})^{2}}{\sigma_{\textrm{A}}^{2}}+\frac{({y^{\prime}}_{i}^{(k)})^{2}}{\sigma_{\textrm{A}}^{2}+\frac{\sigma_{\textrm{cov}}^{2}}{\rho}}\right)\right)>$
$\displaystyle\exp\left(-\left(\frac{({y^{\prime}}_{r}^{(j)})^{2}}{\sigma_{\textrm{A}}^{2}}+\frac{({y^{\prime}}_{i}^{(j)})^{2}}{\sigma_{\textrm{A}}^{2}+\frac{\sigma_{\textrm{cov}}^{2}}{\rho}}\right)\right),$
(17)

where ${y^{\prime}}_{r}^{(k)}$ and ${y^{\prime}}_{i}^{(k)}$ denote the
received signal when the signal $\tilde{X}_{k}$ transmitted.
${y^{\prime}}_{r}^{(j)}$ and ${y^{\prime}}_{i}^{(j)}$ denote the received
signal when the signal $\tilde{X}_{j}$ transmitted. Note that
$\tilde{X}_{j}\in\Omega_{\textrm{gen}}$ and $\tilde{X}_{j}\neq\tilde{X}_{k}$.
Thus, the signal detection is equivalent to the minimum distance detection,
which is given by

$\displaystyle\hat{X}$
$\displaystyle=\mathop{\arg\min}\limits_{\tilde{X}\in\Omega_{\textrm{gen}}}\left(\frac{({y}_{r}^{\prime})^{2}}{\sigma_{\textrm{A}}^{2}}+\frac{({y}_{i}^{\prime})^{2}}{\sigma_{\textrm{A}}^{2}+\frac{\sigma_{\textrm{cov}}^{2}}{\rho}}\right)$
$\displaystyle=\mathop{\arg\min}\limits_{\tilde{X}\in\Omega_{\textrm{gen}}}\Upsilon,$
(18)

where
${y}_{r}^{\prime}={y}_{r}\cos\theta+{y}_{i}\sin\theta-\sqrt{P}|\tilde{h}||\tilde{X}|$,
and ${y}_{i}^{\prime}={y}_{i}\cos\theta-{y}_{r}\sin\theta$. Taking
${y}_{r}^{\prime}$ and ${y}_{i}^{\prime}$ into (III-B), $\Upsilon$ is given by

$\displaystyle\Upsilon=$
$\displaystyle\frac{\left({y}_{r}\frac{{{{X}}}_{r}}{|\tilde{X}|}+{y}_{i}\frac{{{{X}}}_{i}}{|\tilde{X}|}-\sqrt{P}|\tilde{h}||\tilde{X}|\right)^{2}}{\sigma_{\textrm{A}}^{2}}$
$\displaystyle+\frac{\frac{1}{|\tilde{X}|^{2}}\left({y}_{i}{{{X}}}_{r}-{y}_{r}{{{X}}}_{i}\right)^{2}}{\sigma_{\textrm{A}}^{2}+\frac{\sigma_{\textrm{cov}}^{2}}{\rho}}.$
(19)

According to (III-B) and (III-B), for each candidate transmit symbol
$\tilde{X}$, the receiver can determine the detection output that minimizes
$\Upsilon$. The detailed steps of signal detector are summarized in Algorithm
1. Notably, this detection scheme is asymptotically optimal as the transmit
power $P$ increases. This is due to the fact that the derivation of the low-
complexity detector ignores one noise term which has vanishing importance as
the SNR increases [1].

Algorithm 1 The proposed low-complexity signal detector

1:Express the 2D approximated signal $\tilde{Y}$ based on
$({{{\tilde{Y}}}_{1}},{{{Y}}_{2}})$ according to (III-A).

2:Simplify the 2D signal $\tilde{Y}$ by the first-order Taylor-series
approximation method.

3:Obtain the 2D simplified signal
${{{\tilde{Y}}}}=\sqrt{P}|\tilde{h}|{{\tilde{X}}}+{{{W}}}_{r}-\sin\theta\frac{{{{Z_{r}}}}}{\sqrt{\rho}}+j({{{W}}}_{i}+\cos\theta\frac{{{{Z_{r}}}}}{\sqrt{\rho}})$.

4:Calculate the joint PDF $f_{{Y}_{r},{Y}_{i}}({y}_{r},{y}_{i}|\tilde{X})$
according to (III-A).

5:Rotate the coordinates system
${Y}_{r}^{\prime}={Y}_{r}\cos\theta+{Y}_{i}\sin\theta-\sqrt{P}|\tilde{h}||\tilde{X}|,{Y}_{i}^{\prime}={Y}_{i}\cos\theta-{Y}_{r}\sin\theta$.

6:Express the joint PDF
$f_{{Y}_{r}^{\prime},{Y}_{i}^{\prime}}({y}_{r}^{\prime},{y}_{i}^{\prime}|\tilde{X})$
given in (III-B) in the new coordinate system.

7:Design the low-complexity signal detector based on
$f_{{Y}_{r}^{\prime},{Y}_{i}^{\prime}}({y}_{r}^{\prime},{y}_{i}^{\prime}|\tilde{X})$.

8:Obtain the low-complexity signal detector shown in (III-B).

##  IV Numerical Results

In this section, we simulate the performance of the proposed signal detection
method with perfect channel state information. The antenna noise and post-
processing noises are both considered in the splitting receiver architecture.
Since it is difficult to define the SNR for the splitting receiver, the
transmit signal power $P$ can also be expressed as the SNR when the noise
variances are given, as did in [6].

Fig. 3 shows the SER versus the power splitting ratio $\rho$ for 64-QAM with
different signal powers. The variances of the antenna noise, conversion noise
and rectifier noise are respectively designed as
$\sigma_{\textrm{A}}^{2}=\sigma_{\textrm{cov}}^{2}=1$, and
$\sigma_{\textrm{rec}}^{2}=0.1$. As shown in Fig. 3, the approximation SER
achieved by the proposed low-complexity signal detection almost approaches the
accurate SER achieved by the 3D signal detection when $\rho\in(0,0.95]$.
Hence, the low-complexity signal detection method achieves asymptotically
optimal SER performance at a certain power splitting ratio.

Apart from 64-QAM, Fig. 4 shows the SER versus the power splitting ratio
$\rho$ for 32-APSK scheme. The 32-APSK has four rings with 6, 8, 8, and 10
points on each ring. In the simulation, the signal powers are respectively
designed as 200 and 300. The ring radii are assumed to be ordered and
uniformly spaced [12]. As indicated in Fig. 4, we see almost no difference
between the low-complexity signal detection and 3D signal detection schemes
when $\rho\in(0,0.9]$. This implies that the SER approximation achieved by the
proposed low-complexity detection gives nearly optimal performance at a
certain power splitting ratio.

Fig. 5 demonstrates the joint processing gain in terms of SER versus the
signal power $P$. The definition of joint processing gain $G$ is shown in [6],
which is represented as the maximum SER reduction compared to the traditional
coherent and non-coherent receivers. As shown in Fig. 5, the joint processing
gain $G\geq 1$ for different noise variances and signal powers, which means
that the splitting receiver scheme obtains an enhanced SER performance
compared to the traditional receivers.

Figure 3: SER versus $\rho$ for 64-QAM,
$\sigma_{\textrm{{A}}}^{2}=\sigma_{\textrm{{cov}}}^{2}=1,\sigma_{\textrm{{rec}}}^{2}=0.1$.
Figure 4: SER versus $\rho$ for 32-APSK,
$\sigma_{\textrm{{cov}}}^{2}=\sigma_{\textrm{{rec}}}^{2}=1$. Figure 5: Joint
processing gain versus $P$ for 64QAM.

##  V Computational Complexity Analysis

For the proposed low-complexity signal detection scheme, we transform the 3D
signal model into the 2D signal model. As a benefit, the signal detection is
simplified as the minimum distance detection, which calculates the distance
between two signal points. For the traditional CD receiver, the signal
detector can also be simplified as the minimum distance detection based on the
ML rule [13]. Thus, regarding the computational complexity, the proposed
signal detector used in the splitting receiver architecture is approximately
equal to that of the traditional CD receiver.

Furthermore, we compare the detection complexity with different signal
detectors for the splitting receiver architecture. For the optimal ML
detection with 3D received signal in [1, (19)], it needs to calculate double
integrals for achieving the likelihood function. Then the computational
complexity of 3D ML detection is extremely high. The low-complexity detection
rule with a closed-form likelihood function was also developed in [1], which
can be evaluated quickly compared with the 3D ML detection. However, the
obtained closed-form PDF expression in [1] is still very complicated,
resulting that the detection complexity in [1] is much higher than the
proposed signal detection method. Thus, the proposed signal detector greatly
reduces the computational complexity of signal detection at the receiver.

To further illustrate the advantage of our proposed signal detection scheme,
we quantitatively analyze the computational complexity of the proposed signal
detection and the method in [1]. Actually, the calculation of the PDF value in
[1, (26)] requires an exponential operation and about $300$ multiplication
operations. It is shown in [14] that the computational complexity of the
exponentiation $\exp(\cdot)$ is equivalent to $2\log_{2}(\cdot)$ numbers of
multiplication operations. Therefore, we need approximately
$2\log_{2}(\cdot)+300$ multiplications to compute one PDF in [1, (26)].
Whereas, the computation of (III-B) only requires about $27$ multiplications.
Thus, the computational complexity of the proposed scheme is at least reduced
by about $11.1$ times.

As for the hardware complexity, the splitting receiver is also easily
implemented by adding a PD receiver to the CD receiver as depicted in Fig. 1,
where the most commonly used PD circuits are passive [6, 8]. Although the
splitting receiver architecture is slightly more complex than that of the
traditional CD receiver, the splitting receiver ($0<\rho<1$) achieves lower
SER than the CD receiver ($\rho=1$), which has been shown in Section IV.

##  VI Conclusions

This paper has investigated the signal detection scheme for the splitting
receiver architecture with consideration of practical noise model. Based on
the derived two-dimensional signal model, a low-complexity signal detection
method has been proposed. We have shown that the computational complexity of
our proposed signal detection scheme can be reduced by about $11.1$ times
compared to the method in [1]. Simulation results have confirmed that the
approximation SER achieved by the proposed low-complexity signal detection
method approaches the accurate SER at a certain splitting ratio.

## References

  * [1] Y. Wang, W. Liu, X. Zhou, and G. Liu, “On the performance of splitting receiver with joint coherent and non-coherent processing,” _IEEE Trans. Signal Process._ , vol. 68, pp. 917–930, Jan. 2020.
  * [2] J. G. Proakis and M. Salehi, _Digital communications_. McGraw-hill New York, 2007, vol. 5.
  * [3] M. Chowdhury, A. Manolakos, and A. Goldsmith, “Scaling laws for noncoherent energy-based communications in the SIMO MAC,” _IEEE Trans. Inf. Theory_ , vol. 62, no. 4, pp. 1980–1992, Apr. 2016.
  * [4] X.-C. Gao, J.-K. Zhang, H. Chen, Z. Dong, and B. Vucetic, “Energy-efficient and low-latency massive SIMO using noncoherent ML detection for industrial IoT communications,” _IEEE Internet Things J._ , vol. 6, no. 4, pp. 6247–6261, Aug. 2019.
  * [5] W. Liu, S. Shen, D. H. K. Tsang, and R. Murch, “Enhancing ambient backscatter communication utilizing coherent and non-coherent space-time codes,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 10, pp. 6884–6897, May, 2021\.
  * [6] W. Liu, X. Zhou, S. Durrani, and P. Popovski, “A novel receiver design with joint coherent and non-coherent processing,” _IEEE Trans. Commun._ , vol. 65, no. 8, pp. 3479–3493, Aug. 2017.
  * [7] Y. Wang, W. Liu, and X. Zhou, “Splitting receiver with joint envelope and coherent detection,” _IEEE Commun. Lett._ , vol. 26, no. 6, pp. 1328–1332, Mar. 2022.
  * [8] Q. Wang, Q. Guan, J. Cheng, and F. Ji, “A splitting-detection joint-decision receiver for ultrasonic intra-body communications,” _IEEE Trans. Commun._ , vol. 69, no. 6, pp. 3586–3597, June, 2021.
  * [9] D. Kim, H.-M. Kim, and G.-H. Im, “Soft log likelihood ratio replacement for low-complexity maximum-likelihood detection,” _IEEE Commun. Lett._ , vol. 16, no. 3, pp. 296–299, Jan. 2012.
  * [10] A. van den Bos, “The multivariate complex normal distribution-a generalization,” _IEEE Trans. Inf. Theory_ , vol. 41, no. 2, pp. 537–539, Mar. 1995.
  * [11] H. Anton and C. Rorres, _Elementary linear algebra: applications version_. John Wiley & Sons, 2010.
  * [12] C. Hager, A. G. i Amat, A. Alvarado, and E. Agrell, “Design of APSK constellations for coherent optical channels with nonlinear phase noise,” _IEEE Trans. Commun._ , vol. 61, no. 8, pp. 3362–3373, Aug. 2013.
  * [13] D. Tse and P. Viswanath, _Fundamentals of wireless communication_. Cambridge Univ. Press, 2005.
  * [14] T. Hamano, N. Takagi, S. Yajima, and F. Preparata, “O(n)-depth modular exponentiation circuit algorithm,” _IEEE Trans. Comput._ , vol. 46, no. 6, pp. 701–704, June, 1997.