Accurate and Robust Pulmonary Nodule Detection by 3D Feature Pyramid Network with Self-supervised Feature Learning
Jingya Liu Liangliang Cao Oguz Akin Yingli Tian
eess.IV cs.CV

1
.001
Accurate and Robust Pulmonary Nodule Detection
CV Liu et al.

mode = title]Accurate and Robust Pulmonary Nodule Detection by 3D Feature Pyramid Network with Self-supervised Feature Learning 
1]Jingya Liu

2,3]Liangliang Cao


4]Oguz Akin

1]Yingli Tiancor1
[cor1]Corresponding author
ytian@ccny.cuny.edu

[1]The City College of New York, New York, NY 10031, USA
[2]UMass CICS, Amherst, MA 01002, USA
[3]Google AI, New York, NY 10011, USA
[4]Memorial Sloan Kettering Cancer Center, New York 10065, USA



Accurate detection of pulmonary nodules with high sensitivity and specificity is essential for automatic lung cancer diagnosis from CT scans. 
Although many deep learning-based algorithms make great progress for improving the accuracy of nodule detection, the high false positive rate is still a challenging problem which limits the automatic diagnosis in routine clinical practice. Moreover, the CT scans collected from multiple manufacturers may affect the robustness of Computer-aided diagnosis (CAD) due to the differences in intensity scales and machine noises. 
In this paper, we propose a novel self-supervised learning assisted pulmonary nodule detection framework based on a 3D Feature Pyramid Network (3DFPN) to improve the sensitivity of nodule detection by employing multi-scale features to increase the resolution of nodules, as well as a parallel top-down path to transit the high-level semantic features to complement low-level general features. 
Furthermore, a High Sensitivity and Specificity (HS^2) network is introduced to eliminate the false positive nodule candidates by tracking the appearance changes in continuous CT slices of each nodule candidate on  Location History Images (LHI). 
In addition, in order to improve the performance consistency of the proposed framework across data captured by different CT scanners without using additional annotations, an effective self-supervised learning schema is applied to learn spatiotemporal features of CT scans from large-scale unlabeled data. 
The performance and robustness of our method are evaluated on several publicly available datasets including LUNA16, SPIE-AAPM, LungTIME, and HMS. 
The proposed framework is able to accurately detect pulmonary nodules with high sensitivity and specificity and achieves 90.6% sensitivity with 1/8 false positive per scan which outperforms the state-of-the-art results 15.8% on LUNA16 dataset. The significant performance improvements on the three addition datasets (7.4% on SPIE-AAPM, 13.5% on LungTIME, and 8.9% on HMS respectively) compared with the results without using pre-trained model demonstrate the effectiveness of the self-supervised learning schema to handle data captured by different brands of CT scanners. 





 Lung Nodule DetectionFalse Positive ReductionSelf-supervised Learning 3D Feature Pyramid Network


[
    [
    July 24, 2019
=================





§ INTRODUCTION

Lung cancer is one of the leading cancer killers around the world which makes the study of lung cancer diagnosis eminently crucial <cit.>. Computer-aided diagnosis (CAD) systems provide assistance for radiologists to accelerate the diagnosing process <cit.>. Many efforts <cit.> have been made for lung nodule detection on CT scans by applying the recent powerful deep detection models in computer vision. These efforts made good progress to accurately detect pulmonary nodules in curated datasets, but underperform in different datasets and in real clinical settings. In addition, the false positive rate is still very high which limits the real application in routine clinical practice. For example, most of the previous work <cit.> obtained less than 75% sensitivity with 1/8 false positives per scan. To obtain sensitivity scores higher than 95%, these models would bear about eight false positives per scan. Furthermore, to maintain consistent performance across the data captured by different brands of CT scanners, there is a need to continuously adjust and re-train the networks which generally require a vast amount of training data to be labeled. However, manual labeling of large numbers of CT scans is tedious, attention-demanding, and very time-consuming. It also requires knowledge and skills of a particular subspecialty of radiology. 

There are three main challenges for accurate and robust lung nodule detection: 
1) The robustness of existing deep learning-based lung nodule detection methods is limited by the noise among CT scans from different equipment manufacturers. Various CT image texture reconstruction methods and image acquisition techniques are applied to CT scans from different manufacturers during the data collection process. For the data-dependent neural network based framework, artifacts such as intensity, machine setting, machine noise, and image protocol could cause systematical differences and affect the robustness of the nodule detection framework. 
2) Some normal tissues have similar morphological appearances as nodules in CT images which cause high false positives by wrongly detecting these tissues as nodules. The approach for differentiating between the tissue and nodule is very crucial to reduce false positives for automatic lung nodule detection scheme. 
3) The high discrepancy of the volume between nodules and the whole CT scan may cause missing detection of real nodules. For example, the volume of a nodule with 10 mm size in the diameter only occupied 0.059% of the volume of whole CT scan (in average 213×293 pixels with 281 slices). Furthermore, the size of pulmonary nodules can vary by as much as 10 times. For example, nodule sizes can range from 3 mm to 30 mm (in diameter) in LUNA16 dataset. 

Therefore, it is essential to design methods which can detect on CT scans with various machines settings and intensity scales, for the small volume nodules from large volume CT scans, as well as to differentiate nodules from tissues with similar appearances in CT images. 



To address the above three challenges, this paper attempts to integrate the most recent progress in computer vision as well as the domain expert knowledge in medical imaging. As shown in Figure <ref>, the proposed framework contains three main components. 
1) Inspired by the rotation ConvNets <cit.>, to improve the robustness of the proposed nodule detection framework across data captured by different CT scanners without using additional annotations, the self-supervised learning method with 3D CT scan rotation prediction network is proposed to obtain a pre-trained model to extract spatiotemporal features and leverage the differences among CT scans from various manufacturers to guarantee the robustness of the proposed detection framework. 
2) Motivated by the state-of-the-art image detector using 2D Feature Pyramid Network (FPN) <cit.>, a 3D high sensitivity feature pyramid network (3DFPN) is developed to detect nodules of different sizes by appending the low-level high-resolution features to the high-level strong semantic features and a multi-scale feature prediction to ensure the wide-scale nodule detection. 
3) A false positive reduction network is proposed to achieve high specificity while keeping high sensitivity. By carefully analyzing the difference between nodules and the tissues which are wrongly detected as nodule candidates, although they look similar on single CT slice, we observe that the distributions of their spatial variances among continuous CT slices are very different. This unique insight motivates us to design a novel refinement network based on the location history image in the continuous CT slices which significantly eliminates the false positive numbers. 

This paper presents an extension to <cit.>, the main contributions are described as follows:

    
  * We propose a novel and accurate two-stage framework 3DFPN-HS^2 for pulmonary nodule detection by integrating a nodule detection model with a false positive reduction scheme to achieve both high sensitivity and specificity.
    
  * A self-supervised learning-based pre-trained model is introduced to improve the robustness of the proposed 3DFPN-HS^2 framework across CT scans captured from multiple manufacturers without requiring additional labels. The pre-trained model could be further applied to other related tasks with lung CT scans, such as pulmonary nodule classification, and etc., which save the effort for collecting large labeled dataset and also brings great benefits to the study on the small datasets.
    
  * Our proposed framework achieves state-of-the-art performance on LUNA16 dataset at all the false positive levels with a significant improvement of the sensitivity, especially at the low false positive rates. It is robust to CT scans with multiple manufacturers on the different datasets, which makes the proposed work highly likely to be applied in clinical practice.


The remainder of this paper is organized as follows. Section 2 introduces the related work on self-supervised feature learning, object detection, and lung nodule detection from CT scans. Section 3 describes the proposed method. In Section 4, a variety of implementation details, experimental results, and discussions are presented. Finally, section 5 summarizes the remarks of this paper.



§ RELATED WORK

As ConvNets are data-driven computational mechanics, supervised frameworks generally require large-scale labeled data to achieve good performance and overcome overfitting. For lung nodule detection from CT scans, manual labeling of large numbers of CT scans is tedious, attention-demanding, and very time-consuming. It also requires knowledge and skills of a particular subspecialty of radiology. In addition, multiple expert radiologists are required to perform this task to address reader agreement and variability issues. Therefore, a self-supervised scheme may contribute to the applications without requiring additional data annotations. In computer vision research, several frameworks <cit.> have been proposed to learn image and video feature representation based on data transformation. <cit.> adopt the learning task by verifying the correct order of the shuffled sequences from continuous video clips. <cit.> propose a puzzle-solving method to obtain spatial context information. <cit.> learn image features by applying 2D ConvNets for image rotation prediction. However, previous methods are based on 2D which lack spatiotemporal information. Recently, <cit.> propose a self-supervised learning model 3DRotNet for 3D sequences as input adopting a rotation prediction task to learn the rich spatiotemporal features. In this paper, by treating each CT scan as a video, we follow the framework proposed in <cit.> to obtain a pre-trained model for 3D CT scan feature representation extraction to improve the robustness of the lung nodule detection framework across different datasets.

Several deep learning-based frameworks for object detection are proposed to handle small and multi-scale objects <cit.>. Single Shot multi-box Detector (SSD) <cit.> applies a pyramid feature hierarchy in the deep convolutional network to directly detect objects in multi-scales using feature maps from multiple layers by a single pass. However, SSD cannot reuse the low-level feature maps which result in missing small objects. In order to detect small objects, Scale Normalization for Image Pyramids (SNIP) <cit.> selectively back-propagate the gradient of objects in different scales as a function of image scale. Although the performance of small object detection has been significantly improved, by applying the multiple images as the input, the computation cost could be very high. To date, a 2D Feature Pyramid Network (FPN) <cit.> demonstrates the effectiveness for small object detection by extracting the multi-scale feature maps which contain general low-level features of objects at different scales. A top-down path is introduced to pass the global context information by lateral connections of the high-level features and low-level features. The computation cost of extracting features reduced by directly applying multi-scale feature maps. This FPN framework can be applied to lung nodule detection on each 2D CT slices. However, without 3D temporal information among CT slices, high false positives are produced.

Many efforts have been made for lung nodule detection from CT scans. Compared to the traditional method with intensity, shape, texture features, and context features <cit.>, the deep learning-based methods show significant improvements on the performance <cit.>. <cit.> apply a two-stage framework of 2D Faster R-CNN nodule detector with false positive reduction classifier and obtain 89.1% average sensitivity, while <cit.> could achieve an average sensitivity of 84.2% by a one-stage 3D Faster R-CNN detector to learn the rich nodule features combined with a dual-path network and a U-net like encoder-decoder structure without false positive reduction. Single-Shot Single-Scale Lung Nodule Detection (S4ND) <cit.> introduce 3D dense connection and investigated a down-sampling method for small nodule detection. These frameworks employ only a single scale feature map causing limitations in detecting nodules with a wide size range.
<cit.> propose a multisize nodule detection method. The method first segments the lung boundary delineation to obtain the lung region and then applies three sub-algorithms to detect the candidates in three nodule size intervals. Although nodules with different sizes are treated separately, the rule-based thresholding and morphological algorithms limit the sensitivity to 85.6% at 8 false positives per scan.
In addition, for false positive reduction, <cit.> adopt three different 3D ConvNet architectures trained for adapting the scales of nodules and manually setting the threshold to combine the weights. Multi-scale Gradual Integration Convolutional Neural Network (MGI-CNN) <cit.> use the image pyramid network with a multi-stream feature integration for small nodule detection and false positive reduction. Nevertheless, computation costs for these frameworks are expensive due to a large amount of time to extract feature maps for different size of images.
<cit.> apply a 2DFPN network for lung nodule detection, followed by a Conditional 3-Dimensional Non-Maximum Suppression (Conditional 3D-NMS) and Attention 3D CNN (Attention 3D-CNN) for false positive reduction. However, without the spatial features within continuous CT slices, the high false positive candidates are introduced leading a great effort on the reduction process. 



In this paper, we propose a rich spatiotemporal features extraction methodology, an accurate multi-scale nodule detection network, and an efficient false-positive reduction algorithm for accurate and robust lung nodule detection.



§ OUR METHOD

As shown in Figure <ref>, a pre-trained model is obtained from a backbone network of the nodule detection network by a self-supervised pretext task training process to extract rich spatiotemporal features to leverage the differences of CT scans captured by different brands of scanners. 
The pre-trained model is applied to the backbone network (ResNet-18) of the lung nodule detection framework 3DFPN. The input of 3DFPN is a whole 3D volume of CT scan and the output is the 3D locations of lung nodule candidates. After the nodule candidates are detected, a 3D cube centered with the candidate is cropped and fed to a High Sensitivity and Specificity (HS^2) network to further recognize whether the detected nodule candidates are real nodules or false positives. 



 §.§ Self-supervised Pre-trained Model

To allow the proposed nodule detection framework to learn rich 3D CT features from a large-scale dataset captured by CT scanners from multiple manufacturers without additional annotations, a pre-trained model is obtained from a self-supervised pretext task training by firstly performing a four-degree rotation transformation of 3D CT scan as four individual input classes and then followed with a network for rotation prediction task to maximize the probability of rotation angle classification. Thus, rich spatiotemporal CT scans features can be learned during the process.

As shown at Figure <ref>, inspired by <cit.>, rotation transformations are applied to 3D CT scans by rotating all images in CT scans on the image plane at following four angles θ respectively (0^∘,90^∘,180^∘, 270^∘). 
For an input CT scan c_i with dimensions {x,y,z}, we simply rotate it on y dimension and keep the {x,z} fixed. 
The backbone network of the proposed lung nodule detection framework (ResNet-18) is applied to classify the rotation categories of the input CT scans with the four angles. 
Two fully connected layers are followed for probability prediction and are not used for nodule detection. 
The cross entropy loss is applied for probability prediction of K rotation angles (here k=4) and rotation r as shown in Eq. (1):

    loss(c_i|θ)= - 1/K∑_r=1^Klog(F(G(c_i,r)|θ)),

where the classification network ResNet-18 is defined as F(·|θ) for spatiotemporal feature learning and the rotation transformation from the input 3D CT scans to the categories of rotation angle is represented as G(c|y).



 §.§ 3D Feature Pyramid Network for Nodule Detection

The recent progress in computer vision suggests feature pyramid networks (FPN) are good at detecting objects at different scales <cit.>. However, the original FPNs are designed for processing 2D images. Here, we propose a 3DFPN to detect 3D locations of lung nodules from 3D CT volumetric scans by using ResNet-18 as the backbone network which is also used in self-supervised learning to obtain the pre-trained model.
Different than <cit.> which only concatenates the upper-level features in feature pyramid, we use a dense pyramid network to integrate both the low-level high-resolution features as well as high-level high-semantic features, which enriches the location details and strong semantics for nodule detection. 
Table <ref> highlights the main differences between 2DFPN and our 3DFPN. 








As shown in Figure <ref>, the bottom-up network extracts features from the convolution layers 2–5, refer as C2, C3, C4, C5, followed by a convolution layer with kernel size 1×1×1 for converting the dimension of convolution layers to the same number. The feature pyramid network contains four layers P2, P3, P4, P5, which integrates the low-level features by a max-pooling layer and the downsampled high-level features by a 3D deconvolution layer. 3DFPN predicts a confidence score for each nodule candidate and the corresponding location with four parameters as [x,y,z,d], where [x,y] are the spatial coordinates at each CT slice, z is the CT slice index number, d as nodule diameter.




 §.§ HS^2 Network for False Positive Reduction


Due to the low resolution and the noise of CT images, as shown in Figure <ref>(a), the appearances of some tissues (orange boxes) are similar to real nodules (green boxes) which are very likely to be detected as nodule candidates by 3DFPN and leads to a large number of false positives. 
As shown in Table <ref>, we further analyze 300 false positives predicted by the 3DFPN and observe that 241 False Positives (FPs) are caused by the high appearance similarity of tissues (80.3%), 33 of them are caused by inaccurate size detection (11%), and 26 FPs are due to inaccurate location detection (8.7%).




Since the majority of the false positives are caused by tissue regions with similar appearance, it is crucial to design a method to distinguish them from nodules for false positive reduction. By treating each CT scan as a video, we discover that the orientation of the location changes could be tracked in certain patterns for tissues among continues slices, while the variance of true nodules tends to expand outside the contour or diminishing to the center at continuous CT slices as shown in  Figure <ref>(b), Figure <ref> and Figure <ref>.







Inspired by Motion History Image (MHI) <cit.>, we define the Location History Image (LHI) as f. By given any pixel location (x, y) on a CT slice s, f(x, y, s) represents the intensity value of LHI within (1, τ) slice. The LHI is fed to an HS^2 (high sensitivity and specificity) network which is a feed-forward neural network with 2 convolution layers followed by 3 fully connected layers. The outputs of the HS^2 network are refined predicted labels of true nodules and tissues. Sensitivity is defined as a ratio of true positives over the total number of true positives and false negatives. Specificity is the ratio of true negatives over the total number of true negatives and false positives.

The intensity of LHI is calculated as in Eq. (2):

    f(x,y,s)={[                      τ                            if  ψ(x, y, s) = 1; max(0, f(x, y, s-1)-1)                                     otherwise ].

where the update function ψ(x, y, s) given by the spatial differentiation of the pixel intensity of two continuous CT slices. 
The algorithm has the following steps. 1) If | I(x, y, s) - I(x, y, s-1)| is larger than a threshold, ψ(x, y, s) = 1, otherwise, ψ(x, y, s) = 0. 2) For the current slice, if ψ(x,y, s) = 1, f = τ. Otherwise, if f(x, y, s) is not zero, it is attenuated with a gradient of 1. If f(x, y, s) equals zero, then remains as zero. 3) Repeat steps 1) and 2) until all the slices are processed.
Therefore, the location variance among continues CT slices and their change patterns can be effectively represented by our proposed LHIs. 



§ EXPERIMENTAL RESULTS AND DISCUSSIONS



 §.§ Datasets


In this paper, we employ five publicly available datasets for training, testing, and performance evaluation across different datasets: NLST, LUNA16, SPIE-AAPM, LungTime, and HMS Lung Cancer datasets. Table <ref> summarizes the details of these datasets.




NLST Dataset: The national lung screening trial (NLST) <cit.> is a public dataset aimed to determine whether low-dose spiral CT screening for lung cancer can reduce lung cancer mortality in high-risk populations compared with chest screening. The data includes participant characteristics, screening test results, diagnostic procedures, lung cancer and mortality with more than 75,000 CT scans captured by four different manufacturers of CT scanners (i.e. GE, Philips, Siemens, and Toshiba.) Since this dataset is large and there are no annotations for nodule locations, the NLST dataset is used for rotation transformation based self-supervised feature learning. In a total of 13,762 CT scans are applied, where 12,000 CT scans are for training in this paper and 1,762 CT scans are for testing.

LUNA16 Dataset: LUNA16 challenge dataset <cit.> consists of 1,186 nodules in the size between 3-30 mm from 888 CT scans and agreed by at least 3 out of 4 radiologists. It is the most widely used dataset for performance evaluation and is divided into 10 subsets. It is used for fine-tuning the 3DFPN based on the self-supervised pre-trained model from NLST dataset and performance evaluation. To conduct a fair comparison with other methods for lung nodule detection, we follow the same protocol of cross-validation by applying 9 subsets as training and the remaining subset as testing and then report the average performance.

SPIE-AAPM Dataset: The SPIE-AAPM dataset is collected for a 'Grand Challenge' of the diagnostic classification of malignant and benign lung nodule by the international society for optics and photonics (SPIE) with the support of American Association of Physicists in Medicine (AAPM) and the National Cancer Institute (NCI) <cit.>. It contains 70 CT scans from 70 patients, with the annotation of nodule location as well as the nodule diagnosis categories of benign or malignant. It is employed in our paper for cross-dataset testing.

Lung TIME Dataset: The Lung Test Images from Motol Environment (Lung TIME) is publicly available and contains 157 CT scans with 394 nodules <cit.>. The nodules are in the range of 2-10 mm in diameter. The CT scans annotations of the nodule location are provided. It is employed in our paper for cross-dataset testing.

HMS Lung Cancer Dataset: HMS Lung Cancer dataset <cit.> contains the CT scans and lung tumor sections generated by clinical care professionals used in competition with 461 patients. HMS contains a total number of 229 CT scans and 254 nodules with the nodule location annotation. It is employed in our paper for cross-dataset testing.



 §.§ Data Preprocessing
  A preprocessing procedure is required to original CT scans for accurate nodule detection. First, the masks of the lung regions are extracted by lung region segmentation. By using the traditional method, the 2D single slice is processed first with a Gaussian filter to remove the fat, water and kidney background and followed by a 3D connection volume extraction to remove unrelated areas <cit.>. However, it takes 9 to 22 seconds to obtain the mask for each CT scan. In order to accelerate the processing speed for large datasets, we employ the LGAN method  <cit.> and train the network on 10,000 CT slices for lung mask extraction to speed up the process in an average of 5 seconds per scan. In addition, CT scans with an effective value of Hounsfield unit between [-1200, 600] are transformed into the gray value of [0, 255] by a linear mapping. The spacing (mm/pixel) of CT scans between different patients and machines are various and the re-sampling is applied to unify the spacing to 1 mm.




 §.§ Experimental Settings
 


  §.§.§ Self-supervised Pre-trained Model

The self-supervised pretext task takes the 3D CT images of four rotation angles at 0,90,180,270 degrees as input to learn the spatiotemporal features of CT scans. Followed by <cit.>, we adopt the flip and transpose for the rotation transformation. The backbone network of 3DFPN (ResNet-18) is adopted to predict the geometric transformation. The five ConvNets followed with two fully connected layers are trained to maximize the probability of input rotation angle by a four-way classification task. The pre-trained model without the 2 fully connected layers is then employed as the backbone network 3DFPN of for lung nodule detection.

During the training, the learning rate is set to 0.1, decreasing by 1/2 after 70 and 85 epochs, with the weight decay of 5e^-4. The total training includes 100 epochs and the batch size is set to 16.




  §.§.§ 3DFPN

The framework takes the whole CT scan as input, while volume at 96×96×96 pixels is selected by a sliding window method as the input of the 3DFPN network. This size is selected based on experiments to ensure it is big enough to contain the whole nodule even when it is with the largest size (about 30 mm). The anchor sizes employed in our 3DFPN to obtain the candidate regions from feature maps are [3^3, 5^3, 10^3, 15^3, 20^3, 25^3, 30^3] pixels. For all the anchors, the corresponding regions obtained from all the 3D feature map levels are gathered to predict the position of nodules.

In the training phase, the regions with an Intersection-over-Union (IoU) threshold to the ground-truth regions less than 0.02 are referred to negative samples and greater than 0.4 are positive samples. The regions with IoU value in between are ignored to avoid the similarity of the positive and negative samples. A classification layer is used to predict a confidence score for the candidate class and a region regression layer is applied to learn the offset between the position of region proposals and the ground-truth. We adopt Smooth L1 loss <cit.> and binary cross entropy loss (BCE-loss) for location regression and classification score respectively. 
In the testing, for each region proposal, a confidence score is calculated by the classification layer. The proposals with a probability larger than 0.1 are chosen as nodule candidates. Non-maximum suppression is further applied to eliminate the multiple predicted candidates for one nodule.



  §.§.§ HS^2 network
 
The two convolution layers of the HS^2 network are set to (1, 30), (30, 50) dimensions and followed by three fully connected layers with the channel sizes of (2048, 1024, 512). The cross entropy loss is applied for classification during the training. Image patches aligned with each predicted nodule candidate region but with twice size (in both x and y directions) are selected from 11 continuous CT slices (5 slices before and after the current slice of nodule candidate respectively). The LHIs of these patches are computed and resized to 48×48 pixels as the input of the HS^2 network. 

In the training, the learning rate starts from 0.01 and decreases to 1/10 for every 500 epochs. Total of 2,000 epochs is conducted for the framework. The average prediction time for a whole CT scan is about 0.53 min/scan on a server with one GeForce GTX 1080 GPU using Pytorch 2.7. 



 §.§ Evaluation Metrics

In order to conduct a fair comparison with other methods on LUNA16 dataset, we follow the same process to conduct cross validations by using 9 subsets for training and the remaining 1 subset for testing, then obtain the final results by averaging the 10 experiments. Data augment is applied by flipping and resizing the CT scans. 
Same as other methods, the Free-Response Receiver Operating Characteristic (FROC) analysis <cit.> and Competition Performance Metric (CPM) are employed to measure the performance. Follow the LUNA16 challenge evaluation method, the FROC curve plots detection sensitivity and the corresponding false positives where the points on the curve are obtained by the true positive rate (true positives over the sum of true positives and false negatives) while the false positive rate at 1/8, 1/4, 1/2, 1, 2, 4, 8 per scan respectively. The CPM score is calculated by the average of sensitivity for all the levels of false positives per scan.





 §.§ Experimental Results and Analysis



  §.§.§ Comparison with Other Methods
 Table <ref> shows the FROC evaluation results of our proposed method compared with state-of-the-art methods on LUNA16 at 1/8,1/4,1/2,1,2,4,8 false positive levels. The highlighted numbers in the table indicate the best performance within each column. Since the pre-trained model is not used in most state-of-the-art methods, for a fair comparison, all the methods are tested on LUNA16 dataset followed the same FROC evaluation without the pre-trained model. As shown in the table, our framework outperforms 5.5% average sensitivity than the best result of other methods. In addition, the proposed framework achieves the best performance at every FP level. As previously mentioned, the CAD system is not only required a high sensitivity, but also high specificity. Table <ref> demonstrates that the false positives are greatly reduced by the proposed HS^2 network. 3DFPN-HS^2 obtains a highest 97.14% sensitivity at 2 FPs per scan. In addition, for the FP of 1/8, 1/4, and 1/2 per scan, the proposed framework still remains a high sensitivity above 90%. The experimental results show that 3DFPN-HS^2 reaches a state-of-the-art performance for high sensitivity and specificity lung nodule detection. The illustration of nodule detection results is shown in Figure <ref>.










  §.§.§ Robustness with the Self-supervised Pre-trained Model

As shown in Table <ref>, two sets of experiments are conducted to evaluate the robustness of the proposed framework across different datasets without and with the self-supervised based pre-trained model respectively.  One set uses the 3DFPN-HS^2 model trained only on LUNA16 training data without using the self-supervised based pre-trained model from NLST dataset. Another set applies the pre-trained model and then fine-tuned on LUNA16 dataset which the results are highlighted in the table.
All the results are tested on LUNA16 testing data, SPIE-AAPM, LungTime, and HMS Lung Cancer datasets. 
The results with the pre-trained model on LUNA16 test set show slight improvements than the results without using the pre-trained model at all the false positive levels . 
For the testing on SPIE-AAPM, LungTime, and HMS Lung Cancer datasets which are never used in training and fine-tuning, the performance significantly decreases compared to the performance on LUNA16 especially at 1/8 false positive per scan without using the pre-trained model. This is because the training set of LUNA16 is relatively small to obtain a robust 3DFPN-HS^2 model. 
Compared with the model trained only on the LUNA16 data set, the framework with pre-trained model showed significant improvement in all false positive levels across all these datasets. In particular, on 1/8 false positive per scan, the sensitivity is boosted up to 7.4% on SPIE-AAPM, 13.5% on LungTIME, and 8.9% on HMS respectively. For the 8 false positives per scan, the accuracy is equivalent to that of LUNA16. 
The significant improvement in performance demonstrates the robustness of applying the self-supervised pre-trained model across different datasets without requiring additional annotations. 







  §.§.§ Effectiveness of HS^2 Network for FP Reduction
 Two experiments are conducted to demonstrate the advantages of HS^2 network on LUNA16 dataset. As shown in Figure <ref> (left), compared with 3DFPN without the HS^2 network, the result of 3DFPN-HS^2 with the false positive reduction is increased more than 5% at 1/8 FP level. In addition, the numbers of FPs with (blue bar) and without (orange bar) HS^2 network for all the predicted nodule candidates from a total of 88 CT scans (subset 9) are further compared in  Figure <ref> (right). By applying HS^2, the 3DFPN-HS^2 is able to distinguish the false detected tissues from true nodules, therefore significantly reduces FPs by 84.5%. It is worth noting that our proposed 3DFPN without HS^2 network still achieves 97% at 8 FPs per scan and 91.9% CPM, which surpasses other state-of-the-art methods (see Table <ref>.)



§ CONCLUSION

In this paper, we have proposed an effective and robust 3DFPN-HS^2 framework for lung nodule detection with a self-supervised feature learning schema. By employing a 3D feature pyramid network with local and global feature enrichment, different sizes of lung nodules can be detected. By introducing the HS^2 network and treating each CT scan as a video, false positives are significantly reduced based on the different patterns of location variance for nodules and tissues in continuous CT slices. By applying a self-supervised feature learning schema, spatiotemporal features of CT scans can be effectively learned from large-scale CT scans without using additional labels. The learned features can significantly improve the robustness of the proposed framework across different clinic datasets.  The proposed framework strikingly outperforms state-of-the-art methods, has achieved both high sensitivity and specificity and is robust to the data from multiple CT scanner manufacturers which has great potential in routine clinical practice. 



§ ACKNOWLEDGEMENTS

This material is based upon work supported by the National Science Foundation under award number IIS-1400802 and Memorial Sloan Kettering Cancer Center Support Grant/Core Grant P30 CA008748. Oguz Akin, MD serves as a scientific advisor for Ezra AI, Inc., which is unrelated to the research being reported.






