# Medical needle tip tracking based on Optical Imaging and AI

Zhuoqi Cheng  Simon Lyck Bjært Sørensen  Mikkel Werge Olsen  René Lynge
Eriksen  Thiusius Rajeeth Savarimuthu

###### Abstract

Background and Objective:  Deep needle insertion to a target often poses a
huge challenge, requiring a combination of specialized skills, assistive
technology, and extensive training. One of the frequently encountered medical
scenarios demanding such expertise includes the needle insertion into a
femoral vessel in the groin. After the access to the femoral vessel, various
medical procedures, such as cardiac catheterization and extracorporeal
membrane oxygenation (ECMO) can be performed. However, even with the aid of
Ultrasound imaging, achieving successful insertion can necessitate multiple
attempts due to the complexities of anatomy and tissue deformation.
Methods:  To address this challenge, this paper presents an innovative
technology for needle tip real-time tracking, aiming for enhanced needle
insertion guidance. Specifically, our approach revolves around the creation of
scattering imaging using an optical fiber-equipped needle, and uses
Convolutional Neural Network (CNN) based algorithms to enable real-time
estimation of the needle tip’s position and orientation during insertion
procedures.
Results:  The efficacy of the proposed technology was rigorously evaluated
through three experiments. The first two experiments involved rubber and bacon
phantoms to simulate groin anatomy. The positional errors averaging
2.3$\pm$1.5 mm and 2.0$\pm$1.2 mm, and the orientation errors averaging
0.2$\pm$0.11 rad and 0.16$\pm$0.1 rad. Furthermore, the system’s capabilities
were validated through experiments conducted on fresh porcine phantom
mimicking more complex anatomical structures, yielding positional accuracy
results of 3.2$\pm$3.1 mm and orientational accuracy of 0.19$\pm$0.1 rad.
Conclusions: Given the average femoral arterial radius of 4 to 5 mm, the
proposed system is demonstrated with a great potential for precise needle
guidance in femoral artery insertion procedures. In addition, the findings
highlight the broader potential applications of the system in the medical
field.

###### keywords:

Needle tip tracking , Optical needle , Scattering imaging , Deep Learning

††journal: XXXXXX

cnn short = CNN, long = Convolutional Neural Network, tag = abbrev lstm short
= LSTM, long = Long Short-Term Memory, tag = abbrev rnn short = RNN, long =
Recurrent Neural Network, tag = abbrev std short = std. dev., long = standard
deviation, tag = abbrev gpu short = GPU, long = Graphics Processing Unit, tag
= abbrev cpu short = CPU, long = Central Processing Unit, tag = abbrev ram
short = RAM, long = Random-Access Memory, tag = abbrev vram short = VRAM, long
= Video Random-Access Memory, tag = abbrev fc short = FC, long = Fully
Connected, tag = abbrev lr short = lr, long = learning rate, tag = abbrev gt
short = GT, long = Ground Truth, tag = abbrev pp short = pp, long = percentage
points, tag = abbrev avg short = avg, long = average, tag = abbrev soa short =
SOA, long = State of the Art, tag = abbrev cm short = CM, long = Confusion
Matrix, tag = abbrev acc short = acc, long = accuracy, tag = abbrev f1ma short
= $F1_{ma}$, long = F1 macro average, tag = abbrev ai short = AI, long =
artificial intelligence, tag = abbrev dl short = DL, long = deep learning, tag
= abbrev ntt short = NTT, long = needle tip tracking, tag = abbrev relu short
= ReLU, long = Rectified Linear Unit, tag = abbrev mse short = MSE, long =
Mean Squared Error, tag = abbrev sgd short = SGD, long = Stochastic Gradient
Descent, tag = abbrev us short = US, long = Ultrasound, tag = abbrev

[inst1]organization=Maersk Mc-Kinney Møller Institute,addressline=University
of Southern Denmark, city=Odense, postcode=5230, country=Denmark

[inst2]organization=Mads Clausen Institute,addressline=University of Southern
Denmark, city=Odense, postcode=5230, country=Denmark

##  1 Introduction

Deep needle insertion is a medical procedure that allows medical professionals
inserting a needle from the skin and reach targets located beneath the
surface, such as tumors, nerves, or blood vessels. It plays a crucial role in
enabling precise and targeted medical interventions such as administering
medication, obtaining samples, or carrying out therapeutic procedures.
Therefore, deep needle insertion is widely utilized across a diverse range of
medical disciplines, including deep vessel interventions, biopsies,
brachytherapy, interventional radiology, anesthesia, and pain management.

Despite its wide applications, deep needle insertion presents inherent
challenges and requires both expertise and technological support [1]. One of
the main difficulties encountered in this procedure is the deformation of soft
tissue during insertion. As the needle penetrates the body, the surrounding
tissues have a tendency to move, making it challenging to precisely reach the
intended target. Moreover, once the needle tip is within the tissue, its
visibility becomes obscured, complicating the accurate tracking of its exact
location. This lack of visibility becomes particularly problematic when
encountering resistant tissues, as it frequently results in needle deflection,
further impeding the ability to precisely locate the needle.

Frequently, healthcare professionals rely on imaging techniques to guide
needle insertion, with ultrasound (US) and fluoroscopy being commonly used
modalities [2, 3, 4]. These imaging technologies provide real-time
visualization of the needle and surrounding anatomical structures, aiding in
the precise localization of the target and facilitating adjustments to the
insertion direction. However, the effectiveness of using this technique
heavily depends on the practitioner’s experience and necessitates extensive
training to ensure proficiency [5]. Advancements in robotics for needle deep
insertion have been explored in the literature [6]. A typical exploitation of
robotic and ai technology is through the extraction and detection of the
needle tip from US images [7]. Nevertheless, given the inherent noise in US
imaging, the utilization of complex image processing algorithms and
substantial computing power is required [8].

In addition to the previously described image-guided needle insertion method,
needle guidance technology can also incorporate embedded sensors, particularly
for the task of ntt. Previous studies have explored the use of concentric
electrode needle for electrical bioimpedance sensing [9]. But the
functionality of this technology is to detect the type of tissue in contact
with the needle, rather than to provide the spatial position of the needle tip
[10]. Alternatively, other studies have utilized electromagnetic (EM) sensors
embedded in needle tips to enable tracking within a generated magnetic field
[11]. Research indicates that in favorable environments, an average accuracy
of 1 mm can be achieved [12]. However, the use of EM sensors presents several
challenges. Firstly, calibration is required for seamless integration into the
workflow. Secondly, the tracking accuracy is contingent on the absence of
metal and magnetic fields in the environment. Lastly, the implementation of EM
sensors can be relatively costly.

Alternatively, the ntt can be achieved by leveraging the accurate model of the
needle deflection. Shape sensing techniques, such as strain gauges and Fiber
Bragg Gratings (FBG), can be attached along the needle shaft to retrieve
information on the needle’s deflection [13]. However, the FBG technology,
while offering accurate needle shape reconstruction in real-time, is sensitive
to temperature variations and can be relatively expensive to implement.

The objective of this study is to present a novel sensing method for tracking
the position of a needle tip accurately and in real time. As shown in Figure
1(A), the proposed method incorporates an optical fiber within the lumen of
the needle to emit light at the needle tip. By capturing the light scattering
imaging on the tissue surface, an ai-empowered image processing algorithm is
employed to analyze the images and estimates the spatial position of the
needle tip. The primary focus of this research is to utilize this technique
for guiding needle insertion into deep vessels in the groin region, with the
ultimate aim of enhancing patient care through improved effectiveness and
efficiency. To the best of the authors’ knowledge, the proposed ntt approach
presents a pioneering advancement, and holds significant potential for
extension to various other applications such as vessel puncture detection.

##  2 Material and methods

###  2.1 Design concept and requirements

Figure 1: (A) The proposed system configuration includes a camera, a PC, a
micro-controller, a light source and a needle with embedded optic fiber; (B)
The anatomy of groin area where a femoral artery is covered under skin and a
layer of fat; (C) The modeling of optic imaging generated by lighting at the
needle tip.

As described, one of the primary applications for the proposed technology is
to facilitate the guidance of needle insertion into the femoral artery in the
groin area. As illustrated in Figure 2(B), the artery is ensconced within
layers of adipose tissue and connective tissue beneath the skin. The distance
between the skin surface and the femoral artery in the groin area is subject
to variation, particularly in relation to the patient’s body mass index (BMI).
In individuals of normal weight for their height, the femoral artery usually
resides 2 to 3 cm below the skin. For those with obesity, the artery can be
positioned as deep as 4 cm [14, 15].

Figure 1(C) shows a simplified model that treats the skin and adipose tissues
together as a homogeneous material, assuming a uniform optical property for
both [16]. Given that the bevel tip of the needle consistently points upward
during insertion, once the needle tip penetrates the tissue, the light emitted
from it travels through the tissue and generates a scattering image on the
skin’s surface. Moreover, we assume that the tissue surface is flat. The
Bouguer-Beer-Lambert law, an exponential function, governs the intensity of
the light as it travels through the tissue, causing it to attenuate. We
further denote the light intensity emitting from the needle tip as $I_{0}$. In
this case, the intensity at point $P_{1}$ can be calculated as follows:

$I(P_{1})=(1-R_{F})I_{0}\exp{(-\mu_{t}d)}$ (1)

where $d$ is the distance between the light source $P_{0}$ and $P_{1}$,
$R_{F}$ and $\mu_{t}$ represent the diffuse Fresnel reflectance and the
attenuation coefficient of tissue respectively.

In addition, the current system set the distance between the tissue surface
and the camera’s imaging plane, $h$, as a fixed value. According to Equation
1, the generated scattering imaging on the tissue surface has an unique
mapping to the needle tip relative position inside the tissue. Since
establishing the correlation between the needle tip position and the produced
scattering image is complex, an ai based method is developed in this study.

###  2.2 System setup

Figure 2 depicts the system setup based on the proposed concept. The system
incorporates a custom-designed needle (shown in Figure 2(B)) which is created
by inserting an optic fiber (M137L02, Thorlabs Inc., US) into the lumen of a
22G needle and then securing the two components with glue. The Thorlabs 880 nm
LED is selected as the light source, because the near-infrared (NIR) spectrum
has relatively lower absorption rate in tissue compared to visible light,
enabling the light to penetrate thicker tissue [16]. Additionally, the
selected light source has a power output of about 0.6 mW, which provides
sufficient intensity while maintaining safety.

Figure 2: (A) The system setup for validating the proposed concept, (B) The EM
sensor is fixed to the needle hub for obtaining the ground-truth position &
orientation values, and (C) The workflow of the software system.

The control and data acquisition system is based on a portable PC (NUC10i7),
which manages a micro-controller (Arduino UNO) for toggling the light source
on and off. In addition, a camera (ASI178MM, ZWO Co. Ltd., China) is
positioned 100 mm above the tissue surface and interfaced with the PC to
capture the scattering image.

This study also employs a TrakSTAR electromagnet (EM) tracking system (NDI
Inc., Canada) to track the spatial position of the needle tip, serving as the
ground-truth values for supervised learning. The EM sensor system comprises
two main components: a coil and an EM sensor. The coil is securely fastened
onto the platform, while the EM sensor is affixed to the needle hub. Please
note that the EM sensor is only used for acquiring needle tip position to feed
the network training, and is not necessary for the actual deployment.

For data collection, the software system is constructed using the Robot
Operation System (ROS) framework, which operates on the NUC PC. Additionally,
a system with a Threadripper 5965WX CPU and an NVIDIA RTX 6000 Ada Generation
GPU is used for training the ai algorithm which is developed based on PyTorch
[17] with PyTorch Lightning [18]. After the ai model has been fully trained,
it is tested on the NUC PC to assess its efficacy and performance. We choose
to implement the final system on the NUC PC for consideration of system
portability.

###  2.3 Calibration

The calibration procedure comprises two essential steps.

Firstly, the intrinsic parameters of the camera should be acquired in order to
generate a distortion-free imaging. This is done by utilizing a chessboard
pattern to ensure accurate mapping between real-world coordinates and their
corresponding image coordinates. OpenCV lib is employed for calculating these
required camera parameters.

Secondly, a coordinate transformation is performed to establish a mapping
between the coordinate of the needle tip $\\{N\\}$ in relation to the global
coordinate on the platform $\\{G\\}$. This task is accomplished through the EM
sensor, whose relative position in relation to the coil
$\mathbf{T}^{Coil}_{EM}$ can be obtained directly. As shown in Figure 3, a
pivot calibration is done by manually controlling the needle tip to the origin
of $\\{G\\}$ multiple times and at varying orientations. By collecting a
series of data, we can retrieve the relation between the EM sensor and the
needle tip, namely $\mathbf{T}^{EM}_{N}$. Subsequently, the needle tip is
navigated to 9 predefined points on the platform , arranged in 3 rows and 3
columns and spaced 30 mm apart. The center of these 9 points is the origin of
$\\{G\\}$. The least square method is used to fit the collected points, and we
can calculate the transformation matrix between the coil frame and the global
frame as $\mathbf{T}_{Coil}^{G}$. The parameter fitting procedure also
calculate the residual which corresponds to the Euclidean distance
reprojection error of 0.64 mm. With the above transformation matrix obtained,
the position of the needle tip in the global coordinate system can be
calculated as follows:

$\mathbf{T}^{G}_{N}=\mathbf{T}_{Coil}^{G}\cdot\mathbf{T}_{EM}^{Coil}\cdot\mathbf{T}^{EM}_{N}$
(2)

Figure 3: Reference frames of the system: $\\{G\\}$ global frame, $\\{N\\}$
needle tip frame, $\\{EM\\}$ EM sensor frame, and $\\{Coil\\}$ coil frame.

###  2.4 Data acquisition

To acquire the required data for training and validating the proposed system,
a software system is designed following the workflow illustrated in Figure
2(C). This system runs in a loop, and controls the activation and deactivation
of the light source during data acquisition process. Images are taken with the
needle light on ($Img_{on}$) continuously only interrupted by an ambient light
image ($Img_{off}$) taken every 2 s with the needle light off. With the use of
the newest ambient light image, each illuminated image ambient light can be
removed with $Img_{res}$ = $Img_{on}$ \- $Img_{off}$ (see Figure 4). The
residual image $Img_{res}$ is then published to the $ROS$ network.

During the data acquisition, the user press a button on the keyboard and the
system automatically save the residual image (down-scaled to 400$\times$400)
and the current needle tip position with respect to the global coordinate.

(a) $Img_{on}$

(b) $Img_{off}$

(c) $Img_{res}$

Figure 4: Through activating and deactivating the light source, the influence
of ambient light can be removed, and the residual imaging $Img_{on}$ \-
$Img_{off}$ = $Img_{res}$ can be obtained.

###  2.5 Data normalization

In the context of neural networks receiving images as input, each image is
divided by the maximum pixel value to enhance convergence for the model. The
images are initially stored in a uint16 format, with a pixel value range
spanning from 0 to 65535. Consequently, the images are normalized to a range
of [0, 1].

Moreover, the 3D positional information corresponds to a spatial region within
the global coordinate system. To ensure consistent data representation, the
positional data in the X and Y axes are further normalized to the range of
[-1, 1], while the Z-axis data is normalized to the range of [0, 1].

###  2.6 Quaternion processing

Quaternion is used in this study to represent orientation of the needle tip.
Due to the unit norm constraint, the direct probabilistic modeling of
quaternion trajectories becomes intractable. For this, we use the method for
converting quaternion into Euclidean space presented in [19]. An auxiliary
quaternion $q_{0}$ is introduced, and a 4-D quaternion $q_{1}$ can be
transformed to a 3-D vector $\zeta$ by calculating the logarithm of
orientation difference as follows:

$\zeta(q_{1})=\log(q_{1}*q_{0})=\begin{cases}\arccos{v}\frac{\mathbf{u}}{||\mathbf{u}||}&\text{
if }\mathbf{u}\neq 0\\\ [0~{}0~{}0]^{T}&\text{ if }otherwise.\end{cases}$ (3)

where $v$ and $\mathbf{u}$ represent the real and imagine part of $q_{1}$
respectively.

To recover the quaternion given the 3-D Euclidean representation
$\zeta(q_{2})$, an exponential operation is conducted.

$q_{2}=\exp(\zeta(q_{2}))*q_{0}$ (4)

where

$\exp(\zeta)=\begin{cases}\cos{(||\zeta||)}+\sin{(||\zeta||)}\frac{\zeta}{||\zeta||}&\text{
if }\zeta\neq 0\\\ 1+[0~{}0~{}0]^{T}&\text{ if }otherwise.\end{cases}$ (5)

###  2.7 ntt algorithm architecture

The present study utilizes a cnn model for processing input images. The cnn
architecture chosen for this purpose consists of five convolutional blocks,
each employing a varying number of filters: 8, 16, 32, 64, and 128. Each block
comprises two convolutional layers that employ relu activation function.
Following the convolutional layers, a max pooling layer is employed to
downsample the feature map. As a result, the final feature map dimensions are
128$\times$4$\times$4\. To facilitate further processing, a fully connected
layer with 417 neurons, activated by relu, is incorporated. The output layer
of the model consists of two fully connected layers with linear activation,
which yield predictions for both needle position and orientation. A visual
representation of the model architecture can be seen in Figure 5.

Figure 5: The CNN Architecture for Needle Tip position & orientation
estimation with optical imaging as input.

##  3 Experiments

Two experiments were designed to evaluate the proposed ntt system and
algorithm. The first experiment was based on a rubber phantom which is made of
homogeneous material. Specifically, we were dedicated to test and compare the
system performances with different exposure times. Furthermore, the second
experiment was carried out on a more realistic porcine phantom. By employing
this sequential approach, the experimental design aims to validate the
proposed method in guiding needle insertion during the femoral artery
intravenous task. Also, we aims to evaluate the proposed method’s potential in
other medical applications that involve more intricate anatomical structures.

The evaluation metrics include positional accuracy, orientation accuracy and
processing time. The orientation accuracy is quantified by the absolute value
$||\zeta||$ using Equation 3 with $q_{0}$ as the ground-truth value and
$q_{1}$ as the predicted value.

###  3.1 Preliminary experiment on rubber phantom

The first experiment was conducted on a TruIV block phantom (Limbs & Things
Co., UK), which is made of homogeneous rubbery material (Figure 6(A)). During
the experiment, the optic needle was manually inserted into the phantom in
various depths and orientations for generating datasets. Specifically, the
data were collected within a box with a length of 60 mm, a width of 40 mm and
a depth of 45 mm. The insertion angle was kept with the needle bevel tip
facing up towards the camera. The pitch $\phi$ was between 30∘ and 70∘ with a
yaw $\theta$ from -45∘ to 45∘. The data collection process was conducted with
even sampling across the defined space.

In total, two datasets were generated, comprising images and the corresponding
spatial position of the needle tip in 6-DOFs. One dataset was captured with an
exposure time of 20 ms, while the other dataset utilized an exposure time of
100 ms with 511 samples each.

The dataset was divided into three subsets using a 60/20/20 split, where 60%
of the data was allocated for training, 20% for validation, and the remaining
20% for testing.

###  3.2 System evaluation based on porcine phantom

The second set of experiments involved the use of porcine tissue phantoms to
create a realistic evaluation environment. For this purpose, two types of
phantoms were obtained from the market: a piece of bacon and a piece of fresh
pork slab. The size of both phantoms were big enough for ensuring coverage of
the camera’s field of view. The bacon phantom featured a 2 mm thick layer of
skin and a substantial layer of fat, closely simulating the actual anatomical
characteristics of the human groin area. The total thickness of the bacon
phantom was measured to be 34.9$\pm$2.2 mm. As for the fresh pork phantom, a
more complicated anatomic structure was observed with intersecting layers of
fat and muscle beneath the skin surface. The overall thickness of the fresh
pork phantom was found to be 47.8$\pm$3.2 mm. This complex anatomical
structure allows us to assess the capabilities of the proposed technology in
handling more intricate medical applications.

During the experiment, one of the phantom was placed on the platform, and
then, the camera height was adjusted to maintain a distance of 100 mm above
the tissue surface. Following this setup, a data collection procedure similar
to the one employed in the first experiment was conducted, but only 100 ms
exposure time was used. This was because a far better performance was received
from the experiments on the rubber phantom (please refer to Section 4.1).
During the data collection, the needle was manually inserted into the tissue
at various depths and angles, encompassing a range identical to that of the
first experiment. Care was taken to ensure a straight needle shaft before
proceeding with data recording, including both the scattering image
$Img_{res}$ and the corresponding needle tip position.

In the second experiment, a dataset of 1367 samples on bacon and 1142 samples
on fresh was created. The data processing was the same as in the first
experiment. The data were divided into a 60/20/20 split for the training,
validation and test dataset. The collected data from the second experiment
were utilized to train the proposed CNN-based model. The previously identified
hyperparameters found effective for the phantom model were applied in this
training process.

Figure 6: The proposed system was evaluated on (A) a rubber phantom (TruIV
block), (B) a bacon phantom, and (C) a fresh pork phantom. The range of needle
insertion angle $\theta$ and $\phi$ are illustrated in (A).

###  3.3 Model training and hyperparameter tuning

In order to determine the optimal parameters for the model’s architecture and
training, a Bayesian hyperparameter tuning approach was employed with the
training and validation split from the rubber phantom. The ranges of the
explored parameter values can be found in Table 1. During the model’s training
process, a learning rate scheduler was employed if the hyperparameter was
true. This scheduler would reduce the current learning rate with a factor of
0.1 if the validation loss of the model did not demonstrate improvement within
a span of 30 epochs and terminates the training if no improvement is seen
within 100 epochs. The loss function used for the model training is a mse loss
with the position and orientation $Loss=MSE_{p}(p,\hat{p})+\alpha\cdot
MSE_{o}(o,\hat{o})$ where $p$ and $o$ is the ground-truth position and
orientation vectors $p=\begin{bmatrix}x&y&z\end{bmatrix}$,
$o=\begin{bmatrix}o_{x}&o_{y}&o_{z}\end{bmatrix}$, $\hat{p}$ and $\hat{o}$ the
model prediction for the position and orientation and $\alpha$ being the loss
weight for the orientation. The Bayesian hyperparameter tuning used the best
validation loss score for the parameter search.

Table 1: Parameter ranges for hyperparameter tuning Parameter | Range
---|---
Optimizer | Adam, AdamW, sgd
Batch Size | 4 | 64
Conv Blocks (CB) | 1 | 5
CB initial filter | 4 | 16
fc layers | 1 | 4
fc Layer Size | 10 | 512
Use lr schedule | Yes | No
Loss weight orientation | 0.0, | 1.0
Learning Rate | 1e-9 | 1e-1

The following parameter settings were selected from 668 runs: the AdamW
optimizer, with a batch size 32. The initial filter count for the
convolutional blocks was set to 8 with 5 blocks, while the fully connected
layer had a size of 517 neurons with one layer. The learning rate was set to
0.00013 with a learning rate scheduler. The orientation component of the loss
was weight with 1.174.

##  4 Results

###  4.1 Results based on the rubber phantom

Table 2 presents the computed position accuracy in the X, Y, and Z components,
as well as the $L2$ norm, based on the test set. For each component, the mean
error, standard deviation (STD) and 90 percentile (90%ile) are reported. To
mitigate the influence of random weight initialization, the models were
trained independently ten times, and the aggregated results were considered
for analysis.

Table 2: The mean error ($mean$), standard deviation ($STD$) and 90 percentile ($90\%ile$) in mm comparing the CNN output and ground-truth on the rubber phantom with 20 and 100 ms exposure time based on the test set. | mean | STD | 90th %ile
---|---|---|---
Exposure | 20 | 100 | 20 | 100 | 20 | 100
X | 1.7 | 1.1 | 2.6 | 1.0 | 3.5 | 2.7
Y | 3.5 | 1.2 | 2.8 | 1.3 | 7.4 | 2.3
Z | 1.5 | 1.1 | 1.9 | 1.0 | 3.2 | 2.4
L2 | 4.7 | 2.3 | 3.6 | 1.5 | 9.1 | 3.9

The results in Table 2 indicate that small differences of the position
accuracy in the X, Y, and Z component. Additionally, results using 100 ms
exposure time are found considerably better compared to the results using 20
ms exposure time.

Moreover, the $L2$ norm errors in different insertion depths $Z^{*}$ are
plotted in Figure 7. The insertion depth $Z^{*}$ is obtained by subtracting
the needle tip’s $Z$ value, relative to the global coordinate, from the height
of the phantom. Relatively higher errors are observed from the first 5 mm of
depth, while the results from 5 to 40 mm are found relatively consistent.
Also, higher positional accuracy of the system with 100 ms exposure time is
found compared to that with 20 ms exposure time.

Figure 7: Boxplot of the $L2$ norm error against different depths inside the
rubber phantom $Z^{*}$.

The orientation accuracy, presented as $||\zeta||$, are presented in table 3.
Results of different exposure time, namely 20 ms and 100 ms, are shown
together for comparison.

Table 3: The orientation error obtained from the rubber phantom experiment. | mean | STD | 90th %ile
---|---|---|---
Exposure | 20 | 100 | 20 | 100 | 20 | 100
$||\zeta||$ | 0.24 | 0.20 | 0.14 | 0.11 | 0.5 | 0.4

###  4.2 Results based on porcine phantoms

The positional tracking error based on the bacon and fresh porcine phantoms
are summarized in Table 4. Specifically, the errors in the X, Y, and Z
directions, as well as their corresponding $L2$ norm are presented. The
observed errors in the X, Y, and Z component exhibits similarities:
approximately 1.0 mm for the bacon phantom and 1.6 mm for the fresh pork
phantom. In fact, the tracking accuracy ($L2$ norm) is higher for the bacon
phantom compared to the fresh pork phantom, with values of 2.0$\pm$1.2 mm and
3.2$\pm$3.1 mm respectively.

Table 4: The mean error ($mean$), standard deviation ($STD$) and 90 percentile ($90\%ile$) in mm comparing the CNN output and ground-truth on both the bacon phantom ($Bacon$) and the fresh pork phantom ($Fresh$) based on the test set. | mean | STD | 90th %ile
---|---|---|---
Phantom | Bacon | Fresh | Bacon | Fresh | Bacon | Fresh
X | 0.9 | 1.5 | 0.8 | 2.0 | 2.0 | 3.1
Y | 1.0 | 1.6 | 0.8 | 2.4 | 2.0 | 3.2
Z | 1.2 | 1.6 | 1.0 | 1.4 | 2.5 | 3.4
L2 | 2.0 | 3.2 | 1.2 | 3.1 | 3.6 | 5.4

The $L2$ norm accuracy in different insertion depths $Z^{*}$ are shown in
Figure 8. The errors observed in the results obtained from the bacon phantom
are relatively smaller when compared to those from the fresh pork phantom.
This is coherent to the results reported in Table 4. In addition, the results
collected from the bacon phantom demonstrate consistency across different
insertion depths. In contrast, an increasing trend is evident in the results
acquired from the fresh pork phantom. In addition, the needle tip orientation
tracking errors are provided in Table 5. Slightly higher accuracy in tip
orientation estimation is found on the bacon phantom (0.16$\pm$0.1 rad)
compared to the fresh pork phantom (0.19$\pm$0.1 rad).

Figure 8: Boxplot of the $L2$ norm error against different insertion depths $Z^{*}$ on the bacon phantom ($Bacon$) and the fresh pork phantom ($Fresh$). Table 5: Orientation error of needle tip tracking results on the bacon phantom ($Bacon$) and the fresh pork phantom ($Fresh$). | mean | STD | 90th %ile
---|---|---|---
Phantom | Bacon | Fresh | Bacon | Fresh | Bacon | Fresh
$||\zeta||$ | 0.16 | 0.19 | 0.10 | 0.10 | 0.3 | 0.3

###  4.3 Processing time

The proposed system and the CNN based tracking algorithm was implemented and
tested on a portable PC, NUC (Intel i7 CPU). In addition to the camera’s frame
rate, the processing time was found to be 20.2$\pm$0.8 ms. Notably, the
processing time can be further reduced to sub-millisecond when the same
algorithm was tested on a computer with GPU. This observation suggests that
the proposed system is sufficiently fast for real-time tracking of the needle
tip, even when implemented on a portable setup.

##  5 Discussion

Through a comparison of the results obtained from the rubber phantom, it is
evident that the positional error with a 20 ms exposure time (4.7$\pm$3.6 mm)
is higher than that with a 100 ms exposure time (2.3$\pm$1.5 mm). These
findings highlight the significance of exposure time as a key factor in
generating adequate intensity on the tissue surface, particularly when the
needle is inserted to depths of up to 40 mm. Nevertheless, an alternative
approach to enhance lighting intensity can be through increasing the power of
the light source, a factor that will certainly be optimized in our future
study.

Based on the results obtained from the porcine experiment, the $L2$ norm
errors derived from the bacon phantom remained within 3.6 mm, whereas the
errors from the fresh pork phantom are generally higher, mostly within 5.4 mm.
In fact, the bacon phantom more closely resembles the anatomy of the human
groin area, where mostly consists of fat. Therefore, the achievement of a 3.6
mm error demonstrates the efficacy of the proposed technology, considering
that the radius of the femoral artery typically ranges from 4 to 5 mm. Also,
the proposed method is considered sufficient for this application since only
the tracking accuracy in $X$ and $Y$ direction are more critical to target the
needle to the artery.

The results obtained from the fresh pork phantom exhibit reasonable
performance up to an insertion depth of 30 mm. However, the errors noticeably
escalate from $Z^{*}$ = 30 mm to 40 mm, as shown in Figure 8. This occurrence
can be attributed to the significant reduction in light intensity captured by
the camera. In Figure 9, examples of imaging from two different depths (10 mm
and 30 mm) for both the bacon and fresh pork phantoms are provided. It is
evident that the lighting intensity for the fresh pork phantom at
$Z^{*}=30\,mm$ becomes hard to discern, whereas the lighting from the needle
tip at the same depth remains distinguishable for the bacon phantom. This can
be interpreted due to the higher concentration of muscle in the fresh pork
phantom, which absorbs a greater amount of light (near-infrared in this
study), affecting the intensity of the captured images. Given the tracking
error within 5.4 mm, the proposed system is still demonstrated to be able for
easy extension to various medical applications with more complex anatomical
features.

(a) Bacon at 10 mm

(b) Bacon at 30 mm

(c) Fresh at 10 mm

(d) Fresh at 30 mm

Figure 9: Examples of optical imaging with the bacon phantom ($Bacon$) and the
fresh pork phantom ($Fresh$) at two insertion depths 10 and 30 mm.

Furthermore, we can see that a sightly higher error is observed for
$Z^{*}<=5mm$ in the results obtained from the rubber phantom (Figure 7). This
discrepancy is suspected to be caused by the insufficient number of samples
collected near the surface of the phantom. To avoid this, an additional 100
samples were collected from the near-surface $Z^{*}<=10mm$ during the
experiments conducted on the bacon and the fresh pork phantoms. Consequently,
it is found that the error in the near-surface area is not higher than that in
the subsurface region, as shown in Figure 8.

It is also interesting to see that the darkened skin of the bacon phantom
surprisingly does not excessively hinder the passage of light. As a result,
the camera is still able to capture high-intensity images even at greater
insertion depths. This finding is encouraging as it suggests that the system
possesses robust potential for use across different skin tones. Our future
study will explore the incorporation of skin tone as an input to the ai model,
with the goal of enhancing the model’s generalization capabilities.

In this study, the accuracy of needle orientation tracking is measured by the
$||\zeta||$ value, which ranges from 0 to $\pi$, with smaller values
indicating higher accuracy. The results of this study demonstrate that the
orientation accuracy achieved is commendable. While it may not be critical for
this specific application, the ability to track needle orientation holds
significant importance in other medical applications, such as needle steering.

Lastly, this study has a few limitations that should be acknowledged. Firstly,
the power of the light source has not been optimized in this study. For safety
considerations, an ultra-low power light source (0.6 mW) is chosen. However,
it is possible to utilize a relatively higher power light source. In this
case, it would be feasible to reduce the exposure time of the camera while
maintaining a comparable level of imaging quality. Secondly, despite our
efforts to ensure a straight needle shaft during data collection, small manual
errors may still involve, affecting the consistency of the transformation
matrix between the needle tip and the EM sensor. While these errors are
challenging to completely eliminate, multiple measures were taken to minimize
their impact. Additionally, the above positional accuracy in each case must be
assessed in relation to the system’s calibration. Namely, the Euclidean
distance reprojection error, 0.64 mm, should be taken into account when
evaluating the system’s performance. Also, to use this method in practise,
extended reality technique can be integrated in the future for guiding the
whole insertion process [20].

##  6 Conclusion

This study presents a cutting-edge technology aiming to achieve precise and
efficient needle tip tracking during the insertion to a deep subsurface
location. The system is developed through combining optical imaging and AI-
based algorithms. Experimental evaluations conducted on rubber and porcine
tissue phantoms validate the system’s capability for accurate and real-time
tracking of the needle tip during the insertion procedure. In addition, the
results highlight the potential of the system to enhance the safety and
efficiency of femoral artery insertion procedures, and demonstrate its
versatility for different medical applications.

## Acknowledgement

We would like to thank Andrea Blasi Núñez, Yeongjun Hong and Beatriz Moreira
Santos for their techinical supports. This study is supported by the
Innovation Fund Denmark under the agreement 1061-00071A.

## References

  * [1] G. McLeod, Novel approaches to needle tracking and visualisation, Anaesthesia 76 (2021) 160–170.
  * [2] P. Beigi, S. E. Salcudean, G. C. Ng, R. Rohling, Enhancement of needle visualization and localization in ultrasound, International journal of computer assisted radiology and surgery 16 (2021) 169–178.
  * [3] M. G. Wagner, S. Periyasamy, M. A. Speidel, P. F. Laeseke, Feasibility of 3d motion-compensated needle guidance for tips procedures, in: Medical Imaging 2020: Image-Guided Procedures, Robotic Interventions, and Modeling, Vol. 11315, SPIE, 2020, pp. 30–36.
  * [4] J. Pan, D. Yu, R. Li, X. Huang, X. Wang, W. Zheng, B. Zhu, X. Liu, Multi-modality guidance based surgical navigation for percutaneous endoscopic transforaminal discectomy, Computer Methods and Programs in Biomedicine 212 (2021) 106460.
  * [5] M. Engberg, S. Mikkelsen, T. Hörer, H. Lindgren, E. Søvik, M. Frendø, M. B. Svendsen, L. Lönn, L. Konge, L. Russell, et al., Learning insertion of a resuscitative endovascular balloon occlusion of the aorta (reboa) catheter: Is clinical experience necessary? a prospective trial, Injury 54 (5) (2023) 1321–1329.
  * [6] M. Koskinopoulou, Z. Cheng, A. Acemoglu, D. G. Caldwell, L. S. Mattos, Robotic devices for assisted and autonomous intravenous access, IEEE Transactions on Medical Robotics and Bionics (2023).
  * [7] B. Konh, B. Padasdao, Z. Batsaikhan, S. Y. Ko, Integrating robot-assisted ultrasound tracking and 3d needle shape prediction for real-time tracking of the needle tip in needle steering procedures, The International Journal of Medical Robotics and Computer Assisted Surgery 17 (4) (2021) e2272.
  * [8] L. J. Brattain, T. T. Pierce, L. A. Gjesteby, M. R. Johnson, N. D. DeLosa, J. S. Werblin, J. F. Gupta, A. Ozturk, X. Wang, Q. Li, et al., Ai-enabled, ultrasound-guided handheld robotic device for femoral vascular access, Biosensors 11 (12) (2021) 522.
  * [9] Z. Cheng, B. L. Davies, D. G. Caldwell, L. S. Mattos, A hand-held robot for precise and safe pivc, IEEE Robotics and Automation Letters 4 (2) (2019) 655–661.
  * [10] Z. Cheng, A. L. C. Carobbio, L. Soggiu, M. Migliorini, L. Guastini, F. Mora, M. Fragale, A. Ascoli, S. Africano, D. G. Caldwell, et al., Smartprobe: a bioimpedance sensing system for head and neck cancer tissue detection, Physiological measurement 41 (5) (2020) 054003.
  * [11] J. C. Romero, T. M. Peters, E. C. Chen, Development of a mini stereotactic guidance system for percutaneous focal liver tumor ablation, in: Medical Imaging 2022: Image-Guided Procedures, Robotic Interventions, and Modeling, Vol. 12034, SPIE, 2022, pp. 568–576.
  * [12] A. M. Franz, T. Haidegger, W. Birkfellner, K. Cleary, T. M. Peters, L. Maier-Hein, Electromagnetic tracking in medicine—a review of technology, validation, and applications, IEEE transactions on medical imaging 33 (8) (2014) 1702–1725.
  * [13] P. Moreira, K. J. Boskma, S. Misra, Towards mri-guided flexible needle steering using fiber bragg grating-based tip tracking, in: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 4849–4854.
  * [14] E. Durand, M. Penso, T. Hemery, T. Levesque, G. Moles, C. Tron, N. Bouhzam, N. Bettinger, S. Wong, J.-N. Dacher, et al., Standardized measurement of femoral artery depth by computed tomography to predict vascular complications after transcatheter aortic valve implantation, The American Journal of Cardiology 145 (2021) 119–127.
  * [15] How deep is the femoral artery?, accessed: 2023-07-03.
URL https://www.healthtap.com/questions/1151063
-how-deep-is-the-femoral-artery/
  * [16] D. A. Boas, C. Pitris, N. Ramanujam, Handbook of biomedical optics, CRC press, 2016\.
  * [17] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, PyTorch: An Imperative Style, High-Performance Deep Learning Library, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035.
URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative
-style-high-performance-deep-learning-library.pdf
  * [18] W. Falcon, The PyTorch Lightning team, PyTorch Lightning (Mar. 2019).  doi:10.5281/zenodo.3828935.
URL https://github.com/Lightning-AI/lightning

  * [19] Y. Huang, F. J. Abu-Dakka, J. Silvério, D. G. Caldwell, Generalized orientation learning in robot task space, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 2531–2537.
  * [20] R. Moreta-Martínez, I. Rubio-Pérez, M. García-Sevilla, L. García-Elcano, J. Pascau, Evaluation of optical tracking and augmented reality for needle navigation in sacral nerve stimulation, Computer Methods and Programs in Biomedicine 224 (2022) 106991.