# Pruning Sparse Tensor Neural Networks Enables Deep Learning for 3D
Ultrasound Localization Microscopy

Brice Rauby, , Paul Xing, , Jonathan Porée, , Maxime Gasse, Jean Provost,
(Correspondingauthor:JeanProvost.) This work was supported in part by the
Institute for Data Valorization (IVADO), in part by the Canada Foundation for
Innovation under Grant 38095, in part by the Canadian Institutes of Health
Research (CIHR) under Grant 452530, and in part by a Natural Sciences and
Engineering Research Council of Canada (NSERC) discovery grant
(RGPIN-2020-06786). The work of Brice Rauby was supported in part by IVADO,
and in part by the TransMedTech Institute, and in part by the Fonds de
recherche du Québec—Nature et technologies. The work of Paul Xing was
supported by IVADO and in part by the TransMedTech Institute. The work of
Jonathan Porée was supported in part by IVADO, in part by the TransMedTech
Institute, and in part by the Canada First Research Excellence Fund
(Apogée/CFREF). This research was enabled in part by support provided by
Calcul Québec (calculquebec.ca) and the Digital Research Alliance of Canada
(alliancecan.ca)B. Rauby is the Department of Engineering Physics,
Polytechnique Montréal, Montréal, QC H3T 1J4, Canada, and also with Mila-
Quebec artificial intelligence institute, Montréal, QC H2S 3H1, Canada (email:
brice.rauby@polymtl.ca)P. Xing, and J. Porée are with the Department of
Engineering Physics, Polytechnique Montréal, Montréal, QC H3T 1J4, Canada
(e-mail: firstname.lastname@polymtl.ca)M. Gasse is with ServiceNow, Montréal,
QC H2S 3G9, Canada, also with the Department of Computer Engineering and
Software Engineering, Polytechnique Montréal, Montréal, QC H3T 1J4, Canada,
and also with Mila-Quebec artificial intelligence institute, Montréal, QC H2S
3H1, Canada (e-mail: maxime.gasse@servicenow.com)J. Provost is with the
Department of Engineering Physics, Polytechnique Montréal, Montréal, QC H3T
1J4, Canada, and also with the Montreal Heart Institute, Montréal, QC H1T 1C8,
Canada (email: jean.provost@polymtl.ca)Manuscript received XXXX XX, 202X;
revised XXXX XX, 202X.

###### Abstract

Ultrasound Localization Microscopy (ULM) is a non-invasive technique that
allows for the imaging of micro-vessels in vivo, at depth and with a
resolution on the order of ten microns. ULM is based on the sub-resolution
localization of individual microbubbles injected in the bloodstream. Mapping
the whole angioarchitecture requires the accumulation of microbubbles
trajectories from thousands of frames, typically acquired over a few minutes.
ULM acquisition times can be reduced by increasing the microbubble
concentration, but requires more advanced algorithms to detect them
individually. Several deep learning approaches have been proposed for this
task, but they remain limited to 2D imaging, in part due to the associated
large memory requirements. Herein, we propose to use sparse tensor neural
networks to reduce memory usage in 2D and to improve the scaling of the memory
requirement for the extension of deep learning architecture to 3D. We study
several approaches to efficiently convert ultrasound data into a sparse format
and study the impact of the associated loss of information. When applied in
2D, the sparse formulation reduces the memory requirements by a factor 2 at
the cost of a small reduction of performance when compared against dense
networks. In 3D, the proposed approach reduces memory requirements by two
order of magnitude while largely outperforming conventional ULM in high
concentration settings. We show that Sparse Tensor Neural Networks in 3D ULM
allow for the same benefits as dense deep learning based method in 2D ULM i.e.
the use of higher concentration in silico and reduced acquisition time.

###### Index Terms:

Deep Learning, 3D imaging, Ultrasound Localization Microscopy (ULM), Sparse
Tensor Neural Networks

##  I Introduction

Ultrasound Localization Microscopy (ULM) is an imaging method that non-
invasively maps the vascular tree and blood velocities at depth in vivo. By
localizing and tracking individual microbubbles injected into the blood flow
[1, 2], ULM achieves an imaging resolution approximately equal to one tenth of
the diffraction-limited resolution. More recently, Dynamic Ultrasound
Localization Microscopy (DULM) [3, 4] has extended the capabilities of ULM by
enabling the generation of retrospectively-gated, super-resolved movies of the
blood flow dynamics, with applications in pulsatility mapping [3], functional
imaging of the brain [5], and cardiac imaging [4]. ULM and DULM have been
extended to 3D imaging [6, 7, 8, 9] using fully addressed or multiplexed array
probes. Both localization and velocity estimation can be improved by rejecting
microbubbles that do not appear across several frames [1]. Such tracking can
be performed, e.g., using the Nearest Neighbor algorithm [3, 1] or the
Hungarian method [10, 11]. Some approaches have also incorporated Kalman
filtering to refine the position estimations of a track [6, 9, 12, 13]. The
acquisition time necessary to construct a complete vascular map is mainly
dependent on the required time to perfuse all vessels and, thus, on
microbubble concentration [14]. However, a trade-off exists between the
microbubble concentration and the localization precision and accuracy [15],
which can be partially lifteds using, e.g., methods based on the compressed
sensing theory [16], on the division of the k-space in several subregions
[17], or tracking the microbubble signals prior to sub-pixel localization
[18]. Deep learning-based methods have also investigated frame-by-frame,
spatial-only approaches [19, 20, 21], and, more recently, the spatio-temporal
context through convolution [22, 23] or sequential modeling [24].

Despite promising results with increased microbubble concentrations both in
silico [22] and in vivo [22, 23], deep-learning based approaches have been
limited to 2D imaging. Indeed, the addition of a third spatial dimension
considerably increases the size of intermediate feature maps and highly-
resolved outputs. For example, a straightforward implementation of Deep-stULM
[22] in 3D would require at least two orders of magnitude more memory than its
2D counterpart. Thus, the development of deep learning based models for 3D ULM
is conditioned on successfully addressing their memory complexity.

To improve the scaling of memory complexity of deep learning approaches in
ULM, we propose to leverage the recently introduced Sparse Tensor Neural
Networks [25]. Indeed, while ultrasound images are typically dense data that
cannot be stored directly as sparse tensors efficiently, filtered microbubbles
responses are sparsely distributed in space and time. One must thus design a
filter that extracts microbubble responses from ultrasound images and is
sufficiently restrictive that it enhances memory requirements without
discarding information enables the neural network to outperform conventional
approaches. Hereafter, filtering out most of the input signal from dense
tensors before conversion to sparse format while keeping the signal of
interest is designated as the dense-to-sparse operation (represented in dashed
green in Figure 2).

Our contributions can be summarized as follows:

  * •

A sparse formulation of Deep-stULM outperforming conventional ULM in 2D.

  * •

A comparative study in silico between ULM and the proposed approach under
varying concentrations in 3D.

  * •

A 2-D in silico study of performance and memory usage of dense-to-sparse
conversion strategies.

We show that Sparse Tensor Neural Networks reduce memory cost and scale better
with added input dimensions, which allows for the training of 3D ULM models
with a limited impact on the performance when compared to dense architectures.
We also provide the code and the dataset needed to reproduce the results at
https://github.com/provostultrasoundlab/SparseTensorULM.

Figure 1: The left column represents the filtered microbubble signal (i.e, the
input of the network), the center column represents the corresponding
microbubble tracks (i.e., the desired output of the network) and the right
column represent the final result after summation of all the predictions from
a dataset (i.e., the vascular structure imaged)  Figure 2: The top row shows a
dense representation of a trajectory in Deep-stULM as well as the intermediate
map dimension. The bottom row illustrates how sparse formulations could reduce
the memory cost: the green pixels represent the pixels of interest at each
resolution, and the gray pixels represent the pixels removed through pruning
based on intermediate prediction

##  II Theory

After image formation, ULM data is typically filtered to remove the signal
from the tissue while retaining microbubble responses. In this study, we focus
on the subsequent step, which is to find the sub-resolution positions of the
microbubbles based on their response. Several deep learning approaches [19,
20] use convolutional architectures where the microbubble position is
projected onto a grid with a finer resolution than the input signal. The
upscaling factor $r$ between the input dimension and the output dimension
typically ranges from $4$ to $8$. Therefore, the memory complexity for storing
the output grid of such architectures in dense format scales with :

$(r\times D)^{d}$

where $d$ is equal to $2$ (resp. $3$) for 2D (resp. $3$D) imaging and $D$ is a
typical dimension of the input in pixels. However, ULM is based on the
assumption of a sparse distribution of microbubbles. An upper bound on the
number of microbubbles that can be detected in a certain volume is given by :

$N<(\frac{\rho D}{\alpha})^{d}$

where $N$ is the number of microbubbles, $\rho$ is the size of a pixel in
wavelength, and $\alpha$ approximately describes the size of the point spread
function in wavelengths. In sparse format, the memory complexity of storing
the microbubble positions scales with $d\times N$ and thus it is upper-bounded
by:

$d\times(\frac{\rho D}{\alpha})^{d}$

Consequently, the ratio between the sparse and the dense representation of the
outputs of the networks scales with :

$d(\frac{\rho}{\alpha r})^{d}$ (1)

For example, with typical values $\alpha=3$ and $\rho=\frac{1}{2}$ , the ratio
between the sparse and the dense representation of an ideal outputs from such
network is expected to be multiplied by $\frac{3}{2}\frac{\rho}{\alpha
r}\simeq\frac{1}{32}=3.1\times 10^{-2}$ when extending from 2D to 3D. This
scaling law makes the sparse representation very attractive for the extension
in 3D.

However, practical factors such as the variability of the upscaling factor $r$
within the network architecture, the reduced sparsity of intermediate
representations, the non-uniform distribution of microbubbles in space, or
temporal context considerations, can alter this scaling law.

##  III Method

First, we detail the approach used to simulate the 2D and 3D datasets used for
training evaluation of the different method. Then we describe the model
architecture, training parameters and evaluation metrics. Additional studies
on the impact of the dense-to-sparse operations and on further architecture
modifications such as pruning [26] (represented in gold dotted line and
scissors in Figure 2) and deep-supervision [27] are presented.

###  III-A Simulations

####  III-A1 2D dataset

To compare Sparse Tensor Neural Networks with their dense counterpart, we
based our study on a previously introduced dense method [22] and used the same
2D dataset based on the previously published simulation pipeline. Microbubble
flow was simulated using a realistic model [15] based on ex vivo mice brains
obtained with two-photon imaging. Four portions of different mice brains were
used to generate the training set, one other portion was used for model
selection and validation and the last one was kept as a test set to assess the
performance. Since each region covers a volume of only $500\times 500\times
500\mu\text{m}^{3}$, we dilated the vascular network by a factor of 2 to fill
a $1000\times 1000\times 1000\mu\text{m}^{3}$ area, as done previously [22].
The ultrasound signal corresponding to the microbubble position was simulated
using an in-house GPU implementation of SIMUS [28] with parameters
corresponding to an L22-14 probe (Vermon, Tours). Three $15.625$-MHz, tilted
plane waves with angles of $-1\degree$, $0\degree$, and $1\degree$ were
simulated with a frame rate of $1$ kHz. The simulated signal was subsampled to
match the $100\%$ bandwidth IQ signal, mimicking the Verasonics Vantage
system. Finally, the IQ data were beamformed with a GPU implementation of the
delay and sum algorithm on a grid of $32\times 32$ pixels with a resolution of
$\frac{\lambda}{4}$ (i.e.,$25\mu\text{m}$), in groups of $512$ frames. The
point spread function (PSF) of the system was simulated at the center of the
grid and used to compute the local correlation between the beamformed IQ data
with the PSF. The obtained correlation maps were used as the input for the
different deep-learning models. In total, $2250$ movies were generated for
training, $250$ for validation, and $500$ for testing. The concentration of
microbubbles simulated for the training and validation sets was set to $5$
microbubbles per field of view (FOV), as done previously [22]. Several test
sets were simulated based on the trajectories from the test angiogram with
varying concentrations ($1$, $5$, $10$, and $20$ microbubbles per FOV) also
matching the test set from the previous study [22].

####  III-A2 3D Dataset

The $3$D dataset was obtained similarly but since they contain more
parameters, additional microbubble trajectories were included to reduce
overfitting. We divided the generated spatio-temporal samples in three groups:
$3500$ samples for training, $500$ for the validation, and $2000$ for testing.
We dilated the vascular network by a factor of $8$ to account for the larger
wavelength and the coarser beamforming grid ($\frac{\lambda}{2}$). We
simulated a $750$-fps imaging sequence containing 5 angles
($\\{-2\degree,0\degree\\},\ \\{2\degree,0\degree\\},\
\\{-1\degree,0\degree\\},\ \\{1\degree,0\degree\\},\ \\{0\degree,0\degree\\}$)
emitted with a 7.8125-MHz center frequency using a matrix array with
parameters matching a commercially available 8-MHz 2D matrix probe
(Verasonics, WA, USA). The concentration of microbubbles simulated for the
training, validation, and test sets was increased to 30 microbubbles (compared
to 5 for the 2D-case) per field of view (FOV) given the additional dimension.

###  III-B Model training and evaluation

####  III-B1 Sparse Tensor Neural Network and 4D convolutions

After the dense-to-sparse operation, the sparse tensor containing the low-
resolution signal was given as input to a Sparse Tensor Neural Network
implemented using the Python library MinkowskiEngine [25]. For each
intermediate layer, Sparse Tensor Neural Networks only apply their convolution
and activation on non-zero values, yielding another sparse tensor.
Conventional operations used in CNNs are implemented in MinkowskiEngine,
leading to a relatively straightforward translation of the model from dense to
sparse format. To assess the benefits of sparse formulation, we converted the
dense Deep-stULM architecture to a sparse formulation without additional
change, this approach is designated as Sparse Deep-stULM hereafter. However,
such dense architecture might not take most of the sparse tensor
implementation. Pruning or cascaded learning could further improve the memory
efficiency of the sparse formulation. Both of this additional modifications
require intermediate supervision and are detailed in the following sections.
The resulting models are also extended directly to 3D imaging with 4D
convolutions to handle 3D+T tensors. 4D convolutions are directly implemented
in the Python library MinkowskiEngine [25].

####  III-B2 Training procedure

For the 2D models based on Deep-stULM, the hyperparameters were set to the
same value as in the original study [22]: the optimizer used was Adam [29] and
the training was divided into two parts. During the first 150 epochs, the
ground truth microbubble trajectories were dilated with a radius of 2 to
stabilize the training. The initial learning rate was set to 0.1 and then
decayed by a factor of 10 at the epochs 15, 45, 75, and 100. During the last
150, the ground truths were no longer dilated, and the learning rate was set
to 0.001 at epoch 150 and then decayed by a factor of 10 at epochs 160, 200
and 250. We did not optimize the hyperparameters for the sparse formulation of
Deep-stULM and used the same as the original study. The batch size was set to
8 for all the runs in 2D. For the 3D models, we also used the Adam optimizer
with an initial learning set of 0.1. We trained the 3D networks for 20 epochs
in total, and the learning rate was decayed by a factor of 10 at epochs 15 and
17. For the first epoch, the batch size was set to 2 to allow every
configuration to fit in memory, then for the remaining epochs, the batch size
was increased to 4.

####  III-B3 Performance comparison with ULM and Deep-stULM

##### Evaluation metric

To compare our results with the previously established method [22] and
conventional ULM, we measure the overlap between the network prediction and
ground truth using the dice coefficient :

$\text{Dice}=\frac{2\times|\text{GT}\cup\text{Pred}|}{|\text{GT}|+|\text{Pred}|}$

with GT being the projection of all the trajectories from the ground truth to
the super-resolved grid and Pred the prediction of the network. Similarly to
the previous study [22], the dice values displayed use a dice computed between
the binarized angiograms (i.e., between the logical summation of all the
microfilms from the test set).

##### Conventional ULM

We also provide the results of a standard, non-deep-learning ULM method,
described in [3]. Briefly, Gaussian fitting was used to localize microbubbles
and the Hungarian method [30] was used for the tracking step. The number of
detections in each frame was set to the optimal value based on the number of
microbubbles simulated in the FOV (i.e., for the $5$MB/FOV concentration, the
number of detection would be set to $5$). Note that this setting is ideal and
may favor the conventional ULM. Indeed, in real applications, the exact number
of microbubbles in the FOV is unknown.

##### Memory monitoring

We monitored the memory usage of the training using CometML and took the
maximum value reached during the training of each method. As there is some
stochasticity involved both in training and in the measurement of the memory,
we used the average across 3 different runs and provided the standard
deviations between each run for deep learning approaches. For the 3D dense
formulation of Deep-stULM, it was not possible to train the model due to
practical memory limitation. Therefore, we provide only an estimate of the
memory usage. This estimate was based on scaling the 2D memory usage based on
the increase of memory for the intermediate maps due to the addition of a
spatial dimension.

###  III-C Additional studies

####  III-C1 Dense-to-sparse strategies

To assess the loss of information and its impact on the performance caused by
the dense-to-sparse filtering, we compared the performance of the sparse model
for two simple dense-to-sparse strategy referred as Top-k and thresholding
strategy with varying value for their respective parameters. To provide better
intuition on the performance that one can expect with more sophisticated
filtering, we developed a deep learning based solution. To compare between
each method, we computed the average number of non-zero pixels in the test set
movies to compare across methods and plotted it in Figure 6. In addition, to
differentiate between the performance loss induced by the dense-to-sparse
strategy from the effect of the sparse implementation, we also evaluated a
dense model with inputs filtered according to the dense-to-sparse strategy
(each model was retrained on the filtered data).

##### Top-k strategy

Typical ULM approaches use regional maxima for microbubble detection before
localizing them with high precision [10]. Based on the same underlying
assumption that the microbubble signals are located near local maxima, it is
reasonable to consider only the k-largest pixel of each input tensor. This
operation is designated as top-k operation later on. In practice, due to the
smoothness of the input, this approach is very similar to the use of local
maxima value while providing better control of the memory usage of the input
tensor. The explored values used for the top-k approaches range between $5000$
to $50000$ pixels.

##### Thresholding strategy

Previously published deep-learning methods [22] used a threshold based on the
value of the local correlation between the signal and the point spread
function of the imaging system to remove residual after clutter filtering in
vivo. We applied this same approach directly on our simulated datasets. For
the thresholding approaches, the threshold values were set to
$\\{0.01,0.05,0.10,0.25\\}$. As the thresholding strategy with a threshold set
at $0.10$ in 2D offered a good trade-off between performance and sparsity, we
used it for all the experiments where the dense-to-sparse strategy was not
specified. The threshold was heuristically set to $0.25$ for the 3D
experiments.

##### Deep learning based strategy

We trained a dense CNN to localize microbubbles at low resolution. To do so,
it is trained to predict the presence of microbubbles in every pixel of the
beamforming grid (low resolution). The dense network used to localize
microbubbles at low resolution is fully convolutional both in space and time
direction and takes as input a tensor of shape $2\times H\times W\times T$ in
$2D$. The inputs channels encode the real and imaginary parts of the input
signal. The input signal is composed of the local correlation of the
beamformed IQ data with the simulated PSF of the imaging system. The spatial
resolution was kept unchanged throughout the whole network. However, the
temporal dimension was reduced by a factor of 2. The output of this network
was then interpolated to match the correlation map dimension. The resulting
mask is used to convert the correlation map to a sparse format, where only the
pixel values with microbubbles are stored along with their coordinates. This
filtering network was trained using the dice loss between its predictions and
the ideal mask at low resolution obtained from the simulated microbubble
position.

####  III-C2 Architecture modifications

Herein, we describe further experiments to refine the sparsity using pruning
on the intermediate representation of the network along with deep-supervision
and cascaded learning.

##### Deep Supervision and pruning

Similarly to other architectures[21, 19, 20], Deep-stULM [22] uses up-sampling
layer that preserves the sparsity of the upsampled tensor. This is sub-optimal
as at finer resolution the sparsity of the trajectory is increased as depicted
in Figure 2. To mitigate these issues, we used the previously introduced
pruning operations [26], that aim to gradually remove the pixels where no
microbubbles are detected (green pixels versus gray pixels in Figure 2). The
feature maps are masked based on the output of an intermediate classifier.
Consequently, the removed pixels are no longer considered in the following
operation and their coordinates are not stored in memory. As depicted in
Figure 2, we implemented pruning at every resolution level based on
intermediate prediction. Since pruning requires the training of intermediate
classifiers at each level, we trained them in a supervised fashion. These
intermediate classifiers are trained using the same loss as the final loss and
consist of pointwise convolution directly applied to the intermediate
representation of the network. This form of supervision is similar to deep
supervision [27] and can also serve as regularization and improve the
performance of the network. These intermediate classifiers are required to
perform pruning, as they provide the mask used to remove the less relevant
pixels from the following operations.

##### Cascaded learning

In the case of super-resolution, deep supervision also makes possible a
certain form of cascaded learning inspired by [31]. Indeed, the intermediate
classifiers can be trained sequentially: during the first phase of the
training, only the first classifier is trained to predict the presence of
microbubbles on a grid at the input resolution. Then, during the following
phase, the intermediate classifiers corresponding to higher resolution levels
are sequentially added to the global loss. When applied, the cascaded learning
strategy used one epoch for each intermediate level, and the number of epochs
for the last resolution level was the same as in standard training.

##  IV Results

Figure 3: Evolution of the performance as a function of the concentration TABLE I: Comparison of the memory usage and performance Sparse and Dense formulation of DeepST-ULM. * Value estimated.  | 2D ($5$MB/FOV) | 3D ($30$MB/FOV)
---|---|---
|  Memory requirements (GB) | Dice (%) |  Memory requirements (GB) | Dice (%)
Sparse Deep-stULM | $6.8\pm 0.2$ | $71.66\pm 2.35$ | $9.9\pm 0.1$ | $49.97\pm 1.79$
Deep-stULM | $12.6$ | $80.38\pm 0.19$ |  $694$* | N.A
ULM | N.A | $59.48$ | N.A | $12.34$
1 MB5 MB10 MB20 MB Ground Truth Conventional ULM Dense Deep-stULM Sparse Deep-
stULM Concentration Figure 4: Comparison of performance under increasing
concentration for conventional ULM (center left column), Deep-stULM dense
formulation (center right column) and its sparse formulation (right column).
Ground truth is given for comparison (left column). Scale bar is 98µ and
corresponds to the wavelength of the simulated pulse. Concentration increases
from 1 (top row), 5, 10 and 20 (bottom row) microbubbles per field of view.

###  IV-A Memory reduction and performance comparison in 2D

The memory usage results and performance at 5 MB/FOV are reported in Table I
for Conventional ULM, Deep-stULM, and Sparse Deep-stULM. For the sparse
method, the threshold strategy was used to convert the input to sparse
formulation. Sparse formulation reduced the memory usage of Deep-stULM from
$12.6$ GB to $6.8$ GB during training, while the dice decreased from $80.4\%$
to $71.7\%$. In a similar setting, conventional ULM reached $59.5\%$ of dice.
The results of the different methods in 2D under varying concentrations are
shown in Figure 4 and the evolution of the dice value computed are reported in
Figure 3. Qualitatively, the standard ULM performances degraded as the
concentration increased with degradation in resolution starting from 5
microbubbles per field of view (MB/FOV). Quantitatively, this diminution of
performance was highlighted by a drop in dice coefficient from $71.9\%$ with 1
MB/FOV to $59.5\%$ for 5 MB/FOV. The performance continued to decrease with
the concentration until it reached a dice of $55.2\%$ at 20 MB/FOV with only
the biggest vessels being visible. In contrast, the dense Deep-stULM approach
exhibited a smaller degradation of performance from $84.7\%$ for 1 MB/FOV to
$75.2\%$ for 20 MB/FOV. The sparse formulation of Deep-stULM showed robustness
to increased concentration and reached performance levels at high
concentration (with a dice coefficient of $70.6\%$ (resp. $68.0\%$) for 10
(resp. 20) MB per FOV that were very close to its performance level at low
concentration ($73.0\%$ for 1MB per FOV). However, this performance at low
concentration (1MB per FOV) were lower than Deep-stULM ($84.7\%$) but were
comparable to conventional ULM ($71.9\%$).

###  IV-B 3D feasibility study

In Table I, we display the results and the memory usage for Sparse Deep-stULM
and conventional ULM as well as estimated memory usage in 3D for Deep-stULM.
The values for the dice in 3D were typically lower as there is more
possibility for a non-overlapping trajectory than in 2D. It is important to
note that just using the sparse formulation allowed us to train the network
with less than $11$GB of GPU memory while outperforming conventional ULM
($50.0\%$ versus $12.3\%$). For qualitative analysis, the reconstructed
angiograms from the test set are displayed in Figure 5 with concentration
increasing from 1 MB/FOV to 30 MB/FOV. At high concentration (10 MB/FOV and 30
MB/FOV), the sparse model accurately reconstructed the angiogram when
conventional ULM failed to do so. Indeed, the conventional ULM produced many
false detections that were not present in the sparse model reconstruction. At
low concentration (1 MB/FOV), both conventional ULM and sparse model
reconstructed the angiogram with fidelity.

1 MB10 MB30 MB Ground Truth Conventional ULM Sparse Deep-stULM Concentration
Figure 5: 3D Comparison of performance under increasing concentration for
conventional ULM (center column) and Deep-stULM sparse formulation (right
column). Ground truth is given for comparison (left column). Scale bar is 200
$\mu$m and corresponds to the wavelength of the simulated pulse. Concentration
increases from 1 (top row), 10 and 30 (bottom row) microbubbles per field of
view.

###  IV-C Additional studies

Figure 6: Evolution of performance as a function of the sparsity achieved for different dense-to-sparse and comparison with dense network with masked input as well as CNN-based dense to sparse operations. TABLE II: Comparison of the memory usage and performance of the different additions to Sparse DeepST-ULM architecture.  | Sparse input |  Intermediate loss | Pruning |  Cascaded learning | 2D ($5$MB/FOV) | 3D ($30$MB/FOV)
---|---|---|---|---|---|---
|  Memory requirements (GB) | Dice (%) |  Memory requirements (GB) | Dice (%)
Sparse Deep-stULM \+ Pruning \+ cascaded learning | ✓ | ✓ | ✓ | ✓ | $5.6\pm 0.2$ | $58.16\pm 0.77$ | $3.7\pm 0.4$ | $38.53\pm 0.27$
Sparse Deep-stULM \+ Pruning | ✓ | ✓ | ✓ |  | $5.7\pm 0.1$ | $62.85\pm 0.63$ | $3.6\pm 0.2$ | $38.65\pm 0.58$
Sparse Deep-stULM \+ int. loss | ✓ | ✓ |  |  | $6.9\pm 0.1$ | $71.33\pm 0.36$ | $10.6\pm 0.3$ | $43.51\pm 0.99$
Sparse Deep-stULM | ✓ |  |  |  | $6.8\pm 0.2$ | $71.66\pm 2.35$ | $9.9\pm 0.1$ | $49.97\pm 1.79$

####  IV-C1 Evaluation of strategies to convert to sparse formulation

In figure 6, it is observed that the thresholding and top-k dense-to-sparse
approaches reached a very similar trade-off between the sparsity obtained and
the level of performance independently of the formulation of the model (dense
or sparse). Indeed, for the sparse model with the thresholding dense-to-sparse
strategy the dice varies from $67.6\%$ with around $21000$ pixels to $75.3\%$
with $90000$ pixels while it varies from $65.4\%$ with around $5000$ pixels to
$73.9\%$ with $100000$ pixels for the top-k strategy. Similarly, for the dense
model with the thresholding dense-to-sparse strategy the dice varies from
$79.4\%$ with around $11000$ pixels to $79.3\%$ with $90000$ pixels while it
varies from $79.9\%$ with around $5000$ pixels to $79.6\%$ with $100000$
pixels for the top-k strategy. Finally, the CNN dense-to-sparse operation
yielded a better trade-off than all the other approaches. Indeed, the CNN
dense-to-sparse operation reached a dice of $67.8\%$ with only $1400$ pixels.

####  IV-C2 Impact of architecture modifications

In table II, we observed that using pruning jointly with sparse formulation
led to a decrease in memory requirements in 2D ($6.9$ GB to $5.6$ GB) but also
led to an important degradation of the performance ($8.8\%$ of dice). It
appeared that combining pruning and cascaded learning has a small impact on
the memory ($5.6$ GB versus $5.7$ GB) while degrading the performance (more
than $3\%$ of dice). The addition of intermediate loss for deep supervision
did not benefit the training performance and had a small memory cost. In 3D,
every variation of the model trained significantly outperformed the
conventional ULM (between $38\%$ and $50\%$ for the sparse models and $12.3\%$
for the conventional ULM). In contrast to 2D, the use of pruning reduced
memory usage by a factor of approximately 4. However, similarly to the 2D
case, pruning also degraded the performance ($50\%$ to $38.6.\%$).

##  V Discussion

In this work, we studied the impact of using a sparse formulation on the
memory usage and performance of Deep-stULM. We also investigated more multiple
methods to further increase sparsity and reduce the memory usage. Our results
suggest that solely using the sparse formulation allows for the extension of
existing deep learning architectures to 3D ULM, while preserving the
performance robustness at high concentration. However, more complex
modifications to the architecture had a less important impact on the memory
usage while degrading the performance.

###  V-A Reducing memory usage of existing methods in 2D

The sparse formulation offers a relatively simple approach to divide by nearly
two-fold the memory requirement of an existing architecture in 2D with little
impact on the performance. Indeed, the sparse formulation qualitatively
matches the performance on the concentration used during training (5 MB/FOV)
and for higher concentration (10 MB/FOV) and quantitatively outperforms
conventional ULM by an important margin. Dense formulation of Deep-stULM
performs better on very high concentration (20 MB/FOV) and isolated
microbubble (1 MB/FOV). As the training set only contains samples at a
concentration of 5 MB/FOV, the dense formulation seems to better generalize to
unseen data. This better generalization capability and the relatively small
size of the training set can also explain the performance gap between the
dense and sparse formulation on training concentration, as the dense
formulation would be able to better generalize to the test set. Increasing the
dataset size and its complexity could bridge the performance gap between the
dense and the sparse formulation, while still benefitting from the memory
improvements. Top-k and thresholding dense-to-sparse strategies appear to have
similar performance. This performance appear to be suboptimal when compared to
the performance reached with much fewer pixels by the CNN dense-to-sparse
strategy. Presumably, the sparse formulation might benefit from more
sophisticated dense-to-sparse strategy to further improve the proposed trade-
off between performance and memory requirements in training.

###  V-B Scaling to 3D imaging

In 3D, the sparse formulation allowed for training, with less than 11 GB of
memory, an architecture that would require close to $700$GB of memory in dense
formulation. As expected, the sparse formulation impact on memory was greater
in 3D than 2D. Experimentally, the ratio between the sparse and the dense
formulation was multiplied by $2.6\times 10^{-2}$ when extending from 2D to 3D
whereas the theoretical value from Eq. 1 was $3.1\times 10^{-2}$. The smaller
ratio could be explained by the aforementioned experimental factors such as
temporal context considerations, as well as the higher threshold for the
dense-to-sparse strategy in 3D. The performance gap between the sparse
formulation of Deep-stULM and the conventional ULM was larger in 3D,
suggesting an even higher potential for deep learning based approach in 3D
ULM. On the one hand, the drop of performance of conventional ULM might be
caused by the important side lobes of the simulated PSF in 3D. Indeed, these
sidelobes create false detection which are supposed to be filtered during the
tracking step based on the track length. However, in high concentration,
candidate detections in successive frame are multiple causing the filtering on
track length to be less effective and contribute to 3D conventional ULM map
with a high number of false positives. On the other hand, it is also possible
that the additional dimension increases the capability of the deep learning
approach to distinguish crossing trajectories, leading to better performance
relatively to conventional ULM. The results in $3$D shows that akin to the
dense approach in 2D, sparse models can accurately reconstruct angiograms at
concentration where conventional ULM is failing. When translated in vivo, such
performance in high concentration would allow for reduced acquisition time. In
3D, where dataset tends to be larger, reducing acquisition time is even more
crucial as it also reduces the storage needed and the associated transfer
time.

###  V-C Further reductions of memory and performance trade-off

##### Dense-to-sparse strategies

The parameter studies showed that simple strategies for dense-to-sparse
performed similarly and better trade-off could be achieved with deep-learning
based dense-to-sparse operations. However, the learned approach for dense-to-
sparse operation lacked flexibility on the trade-off between sparsity achieved
and performance. Increasing the number of pixels considered in the sparse
formulation reduced the performance gap between sparse and dense formulation
of Deep-stULM. This might suggest sub-optimal dense-to-sparse operations.

##### Architecture modifications

It appears that architecture modifications such as deep-supervision, cascaded
learning and pruning had an important impact on the performance with smaller
gain in memory than that then sole sparse formulation. Cascaded learning and
deep-supervision negative impact on the performance could be explained by the
fact that microbubble detection at coarse resolution is different from
localization at super-resolution and therefore intermediate losses enforce the
learning of less relevant latent representations. Considering the additional
complexity in training and architecture constraint going with this
modifications, it is less clear that they present an interesting trade-off
between memory usage and performance.

###  V-D Limitations and perspectives

The proposed method demonstrate sparse tensor neural networks can extend the
benefits of deep learning based approach in ULM from 2D to 3D imaging by
improving the scaling law of memory costs with dimensionality and resolution.
However, some limitations should be mentioned and could be addressed by future
studies.

####  V-D1 Problem formulation

This study is mostly focused on measuring the impact of using a sparse
formulation in deep learning for ULM. For this reason, it does not tackle some
key challenges in framing the learning problem. On the one hand, it is
important to mention that even though the dice loss has been proposed and
successfully applied in the previous study [22], it is unstable and lacks
smoothness when working with temporal projection of trajectories. In addition,
the final representation of the prediction, being a projection of the
microbubble trajectories on a grid, does not offer the same liberty for
downstream analysis as the individual detections provided by conventional ULM.
A formulation tackling these issues has been recently proposed [23] based on
the DECODE method [32] and could be worth investigating. On the other hand,
the encoding of the real and imaginary parts of the input signal as channels
lacks the proper arithmetic of complex numbers. Using a complex value neural
network [33] could allow a better representation of the signal with less
overfitting and better overall performance. Such networks have shown
interesting potential when dealing with ultrasound data [34, 35].

####  V-D2 Validity of the in silico model

The proposed approach reaches the level of performance of the state-of-the-art
dense model in silico under varying concentrations of microbubbles. It would
be interesting to evaluate the validity of the conclusions on real data as it
has not been tested in vivo. However, as the in vivo application of both dense
and proposed methods requires intensive parameter tuning, it is difficult to
ensure a fair comparison between the methods. Additionally, current training
datasets for ULM are limited in their diversity and realism. They often
require tuned pre-processing, and therefore it is not clear that better
learning ability on the simulated dataset would lead to improved performance
in vivo. Furthermore, as the main benefit of the proposed method is an
improvement of the scaling law of the memory complexity, it is reasonable to
assume that it is still valid in vivo.

##  VI Conclusion

In this study, we studied the potential of sparse formulation when applying
deep learning in 3D ULM. We also proposed further optimization of memory
efficiency through pruning of the 3D ULM deep learning model. The proposed
Sparse Deep-stULM method successfully improve the scaling of memory
requirements of deep learning based approaches, addressing the challenge of
their extension in 3D. While it comes at a small cost of performance in 2D,
the use of deep-learning in 3D ULM seems to be even more beneficial than in
2D. To the best of our knowledge, it is the first application of deep learning
for 3D ULM, and it could pave the way for further approaches both in silico
and in vivo. Further applications of such models could allow for improved
architectures capable of fitting to more diverse datasets, yielding better
results both in 2D and 3D ULM or DynULM.

## References

  * [1] Claudia Errico et al.  “Ultrafast Ultrasound Localization Microscopy for Deep Super-Resolution Vascular Imaging”  In _Nature_ 527.7579 Nature Publishing Group, 2015, pp. 499–502  DOI: 10.1038/nature16066
  * [2] K. Christensen-Jeffries et al.  “In Vivo Acoustic Super-Resolution and Super-Resolved Velocity Mapping Using Microbubbles”  In _IEEE Transactions on Medical Imaging_ 34.2, 2015, pp. 433–440  DOI: 10.1109/TMI.2014.2359650
  * [3] Chloé Bourquin, Jonathan Porée, Frédéric Lesage and Jean Provost  “In Vivo Pulsatility Measurement of Cerebral Microcirculation in Rodents Using Dynamic Ultrasound Localization Microscopy”  In _IEEE Transactions on Medical Imaging_ 41.4, 2022, pp. 782–792  DOI: 10.1109/TMI.2021.3123912
  * [4] Philippe Cormier, Jonathan Porée, Chloé Bourquin and Jean Provost  “Dynamic Myocardial Ultrasound Localization Angiography”  In _IEEE Transactions on Medical Imaging_ , 2021, pp. 1–1  DOI: 10.1109/TMI.2021.3086115
  * [5] Noémi Renaudin et al.  “Functional Ultrasound Localization Microscopy Reveals Brain-Wide Neurovascular Activity on a Microscopic Scale”  In _Nature Methods_ 19.8 Nature Publishing Group, 2022, pp. 1004–1012  DOI: 10.1038/s41592-022-01549-5
  * [6] Chloé Bourquin et al.  “Quantitative Pulsatility Measurements Using 3D Dynamic Ultrasound Localization Microscopy”  arXiv, 2023  DOI: 10.48550/arXiv.2303.14330
  * [7] Baptiste Heiles et al.  “Ultrafast 3D Ultrasound Localization Microscopy Using a 32×32 Matrix Array”  In _IEEE Transactions on Medical Imaging_ 38.9, 2019, pp. 2005–2015  DOI: 10.1109/TMI.2018.2890358
  * [8] Arthur Chavignon et al.  “3D Transcranial Ultrasound Localization Microscopy in the Rat Brain with a Multiplexed Matrix Probe”  In _IEEE transactions on bio-medical engineering_ PP, 2021  DOI: 10.1109/TBME.2021.3137265
  * [9] U-Wai Lok et al.  “Three-Dimensional Ultrasound Localization Microscopy with Bipartite Graph-Based Microbubble Pairing and Kalman-Filtering-Based Tracking on a 256-Channel Verasonics Ultrasound System with a 32×32 Matrix Array”  In _Journal of Medical and Biological Engineering_ 42.6, 2022, pp. 767–779  DOI: 10.1007/s40846-022-00755-y
  * [10] Baptiste Heiles et al.  “Performance Benchmarking of Microbubble-Localization Algorithms for Ultrasound Localization Microscopy”  In _Nature Biomedical Engineering_ 6.5 Nature Publishing Group, 2022, pp. 605–616  DOI: 10.1038/s41551-021-00824-8
  * [11] Pengfei Song et al.  “On the Effects of Spatial Sampling Quantization in Super-Resolution Ultrasound Microvessel Imaging”  In _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ 65.12, 2018, pp. 2264–2276  DOI: 10.1109/TUFFC.2018.2832600
  * [12] Iman Taghavi et al.  “Ultrasound Super-Resolution Imaging with a Hierarchical Kalman Tracker”  In _Ultrasonics_ 122, 2022, pp. 106695  DOI: 10.1016/j.ultras.2022.106695
  * [13] Shanshan Tang et al.  “Kalman Filter-Based Microbubble Tracking for Robust Super-Resolution Ultrasound Microvessel Imaging”  In _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ 67.9, 2020, pp. 1738–1751  DOI: 10.1109/TUFFC.2020.2984384
  * [14] Vincent Hingot et al.  “Microvascular Flow Dictates the Compromise between Spatial Resolution and Acquisition Time in Ultrasound Localization Microscopy”  In _Scientific Reports_ 9.1, 2019, pp. 2456  DOI: 10.1038/s41598-018-38349-x
  * [15] Hatim Belgharbi et al.  “An Anatomically Realistic Simulation Framework for 3D Ultrasound Localization Microscopy”  In _IEEE Open Journal of Ultrasonics, Ferroelectrics, and Frequency Control_ 3, 2023, pp. 1–13  DOI: 10.1109/OJUFFC.2023.3235766
  * [16] A. Bar-Zion et al.  “SUSHI: Sparsity-Based Ultrasound Super-Resolution Hemodynamic Imaging”  In _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ 65.12, 2018, pp. 2365–2380  DOI: 10.1109/TUFFC.2018.2873380
  * [17] Chengwu Huang et al.  “Short Acquisition Time Super-Resolution Ultrasound Microvessel Imaging via Microbubble Separation”  In _Scientific Reports_ 10.1 Nature Publishing Group, 2020, pp. 6007  DOI: 10.1038/s41598-020-62898-9
  * [18] Alexis Leconte et al.  “A Tracking Prior to Localization Workflow for Ultrasound Localization Microscopy”  arXiv, 2023  DOI: 10.48550/arXiv.2308.02724
  * [19] Xin Liu et al.  “Deep Learning for Ultrasound Localization Microscopy”  In _IEEE Transactions on Medical Imaging_ 39.10, 2020, pp. 3064–3078  DOI: 10.1109/TMI.2020.2986781
  * [20] Ruud J.. van Sloun et al.  “Super-Resolution Ultrasound Localization Microscopy Through Deep Learning”  In _IEEE Transactions on Medical Imaging_ 40.3, 2021, pp. 829–839  DOI: 10.1109/TMI.2020.3037790
  * [21] Xi Chen et al.  “Deep Learning-Based Microbubble Localization for Ultrasound Localization Microscopy”  In _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ 69.4, 2022, pp. 1312–1325  DOI: 10.1109/TUFFC.2022.3152225
  * [22] Léo Milecki et al.  “A Deep Learning Framework for Spatiotemporal Ultrasound Localization Microscopy”  In _IEEE Transactions on Medical Imaging_ 40.5, 2021, pp. 1428–1437  DOI: 10.1109/TMI.2021.3056951
  * [23] YiRang Shin et al.  “Context-Aware Deep Learning Enables High-Efficacy Localization of High Concentration Microbubbles for Super-Resolution Ultrasound Localization Microscopy”  bioRxiv, 2023, pp. 2023.04.21.536599  DOI: 10.1101/2023.04.21.536599
  * [24] Xi Chen et al.  “Localization Free Super-Resolution Microbubble Velocimetry Using a Long Short-Term Memory Neural Network”  In _IEEE Transactions on Medical Imaging_ , 2023, pp. 1–1  DOI: 10.1109/TMI.2023.3251197
  * [25] Christopher Choy, JunYoung Gwak and Silvio Savarese  “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks”  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ Long Beach, CA, USA: IEEE, 2019, pp. 3070–3079  DOI: 10.1109/CVPR.2019.00319
  * [26] JunYoung Gwak, Christopher Choy and Silvio Savarese  “Generative Sparse Detection Networks for 3D Single-Shot Object Detection”  In _Computer Vision – ECCV 2020_ 12349 Cham: Springer International Publishing, 2020, pp. 297–313  DOI: 10.1007/978-3-030-58548-8˙18
  * [27] Chen-Yu Lee et al.  “Deeply-Supervised Nets”  In _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics_ PMLR, 2015, pp. 562–570
  * [28] Damien Garcia  “SIMUS: An Open-Source Simulator for Medical Ultrasound Imaging. Part I: Theory & Examples”  In _Computer Methods and Programs in Biomedicine_ 218, 2022, pp. 106726  DOI: 10.1016/j.cmpb.2022.106726
  * [29] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”  arXiv, 2017  DOI: 10.48550/arXiv.1412.6980
  * [30] H.. Kuhn  “The Hungarian Method for the Assignment Problem”  In _Naval Research Logistics Quarterly_ 2.1-2, 1955, pp. 83–97  DOI: 10.1002/nav.3800020109
  * [31] Enrique S. Marquez, Jonathon S. Hare and Mahesan Niranjan  “Deep Cascade Learning”  In _IEEE Transactions on Neural Networks and Learning Systems_ 29.11, 2018, pp. 5475–5485  DOI: 10.1109/TNNLS.2018.2805098
  * [32] Artur Speiser et al.  “Deep Learning Enables Fast and Dense Single-Molecule Localization with High Accuracy”  In _Nature Methods_ 18.9 Nature Publishing Group, 2021, pp. 1082–1090  DOI: 10.1038/s41592-021-01236-x
  * [33] Chiheb Trabelsi et al.  “Deep Complex Networks”  In _International Conference on Learning Representations_ , 2018
  * [34] Jingfeng Lu et al.  “Complex Convolutional Neural Networks for Ultrafast Ultrasound Imaging Reconstruction From In-Phase/Quadrature Signal”  In _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ 69.2, 2022, pp. 592–603  DOI: 10.1109/TUFFC.2021.3127916
  * [35] Paul Xing et al.  “Phase Aberration Correction for in Vivo Ultrasound Localization Microscopy Using a Spatiotemporal Complex-Valued Neural Network”  In _IEEE Transactions on Medical Imaging_ , 2023, pp. 1–1  DOI: 10.1109/TMI.2023.3316995