Annotation-Free Cardiac Vessel Segmentation via Knowledge Transfer from Retinal Images
Fei Yu Jie Zhao Yanjun Gong Zhi Wang Yuxi Li Fan Yang Bin Dong Quanzheng Li Li Zhang
eess.IV cs.CV




Annotation-Free Cardiac Vessel Segmentation














F. Yu et al.



Center for Data Science, Peking University, Beijing 100871, China; 

	zhangli_pku@pku.edu.cn Center for Data Science in Health and Medicine, Peking University,
 Beijing 100871, China; Department of Cardiology, Peking University First Hospital, Beijing 100034, China; Beijing International Center for Mathematical Research (BICMR), Peking University, Beijing 100871, China; MGH/BWH Center for Clinical Data Science, Boston, MA 02115, USA. 

Annotation-Free Cardiac Vessel Segmentation via Knowledge Transfer from Retinal Images
    Fei Yu1 Jie Zhao2 Yanjun Gong3 Zhi Wang3 Yuxi Li3 Fan Yang3 Bin Dong4,1,2 Quanzheng Li5,2 Li Zhang1,2()
    December 30, 2023
===========================================================================================================




Segmenting coronary arteries is challenging, as classic unsupervised methods fail to produce satisfactory results and modern supervised learning (deep learning) requires manual annotation which is often time-consuming and can some time be infeasible. To solve this problem, we propose a knowledge transfer based shape-consistent generative adversarial network (SC-GAN), which is an annotation-free approach that uses the knowledge from publicly available annotated fundus dataset to segment coronary arteries. The proposed network is trained in an end-to-end fashion, generating and segmenting synthetic images that maintain the background of coronary angiography and preserve the vascular structures of retinal vessels and coronary arteries. We train and evaluate the proposed model on a dataset of 1092 digital subtraction angiography images, and experiments demonstrate the supreme accuracy of the proposed method on coronary arteries segmentation.








§ INTRODUCTION

Quantitative measurement of coronary arteries in medical images is important for the diagnosis, prevention and therapeutic evaluation of related diseases including hypertension, myocardial infarction, and coronary atherosclerotic disease. In the diagnosis of coronary diseases, digital subtraction angiography (DSA) has been widely used and considered the "gold standard". To quantitatively segment blood vessels in DSA, researchers have developed automated methods including region growing, level sets, and Hessian analysis <cit.>. However, due to the complexity of the vascular morphology, these unsupervised methods are difficult to obtain a clinically satisfactory segmentation of coronary arteries. On the other hand, supervised learning such as deep neural networks (DNNs) can produce better segmentation results but relies heavily on pixel-level image annotation, which is often expensive, time-consuming, and even impossible to access especially for coronary artery segmentation.

To solve the problem of the lack of DSA vessel annotation, we aim to apply a strategy to transfer the knowledge of retinal vessel segmentation to the coronary artery segmentation. The researchers have established and validated a number of public retinal vessel segmentation datasets, including DRIVE<cit.>, STARE<cit.>, and RITE<cit.>. Due to the significant differences between their anatomical regions, traditional transfer methods are not suitable. Therefore, we present a novel knowledge transfer based adversarial model containing three parts called generator, discriminator, and segmentor. Training of the model contains three major steps: 1) Frangi vessel analysis <cit.> is used to segment the coronary artery in the DSA images roughly. 2) The adversarial training between generator and discriminator allows the model to fuse the fundus image and the DSA image. 3) A synthetic label is then created by computing the union of rough coronary artery segmentation and retinal vessel annotation. Moreover, a shape-consistent scheme is used to ensure the shape consistency of synthetic images and synthetic annotations. The fused image and corresponding synthetic label are used to train the segmentor. The supreme accuracy demonstrates the effectiveness of our methods, which improves the accuracy of coronary artery segmentation by using the knowledge of fundus segmentation without additional manual annotation of the DSA images. The ideas of knowledge transfer and data fusion in this paper have many other application scenarios, including cell segmentation, neural segmentation, and airway segmentation. As long as the object structures in the two datasets are similar, we can use the knowledge from the annotated dataset to guide the analysis of the unannotated one.

Our work relates closely to the recent rise of knowledge transfer techniques. In the field of natural image analysis, Domain-Adversarial Neural Network (DANN) transfers the feature distribution to solve the domain-shift problem <cit.>. Cycle-GAN introduces a cycle consistency loss and achieves unpaired image-to-image translation <cit.>. AdaptSegNet adopts adversarial learning in the output space and receives favorably accuracy and visual quality <cit.>. In the field of medical image analysis, many studies have been dedicated to exploring cross-modality translation with GAN <cit.>. Using synthetic data to overcome insufficient labeled data is also an active research area. For example, using synthetic data as augmented training data can help lesion segmentation <cit.> and cardiovascular volumes segmentation <cit.>. These methods show that knowledge transfer is effective for the same anatomical region, and our work contributes by accomplishing knowledge transfer between two different anatomical regions.



§ METHODS

In this section, we show two models based on knowledge transfer. The first one is the GAN model with a constraint of shape consistency (SC-GAN) proposed in this work. Then, to verify the necessity of fusion, we show another simpler model. It adopts from Mixup<cit.>, which is to train the U-Net model by computing an average of the fundus image and the DSA image (Add U-Net).



 §.§ SC-GAN


Figure <ref> shows the training and test processes of the proposed model. During the training process, the generator, discriminator, and segmentor are trained simultaneously. In the test process, only the trained segmentor is needed to segment coronary arteries, which requires much less memory and inference time than training. In the previously reported knowledge transfer between different modalities within the same anatomy, the foreground and the background can be reasonably registered. In our task, however, the foreground and background of the images are completely mismatched. So we have designed a shape-consistent scheme that allows the generator and discriminator to complete the knowledge transfer of foreground and background respectively. In the following subsections, we will discuss the model's architectures and the objective functions in detail.



  
Generator: As shown in Figure <ref>, the generator uses U-Net as its network backbone. The input of the generator is an average of the fundus image (Real A) and the DSA image (Real B). The output is a synthetic image (Fake B) with the same dimensions as the input. To ensure that the Fake B has both retinal vessels and coronary arteries, we extract the two vascular regions (Part A and Part B) in Fake B using manual annotation of retinal vessels and Frangi segmentation of DSA images. A shape-consistent loss (l1 loss) is then used to regularize the content of the vessel regions in Fake B to be consistent with the corresponding regions in the original images,

    ℒ_shape_A(G)=𝔼_A ∼ p_data(A), B ∼ p_data(B)[||label_A*(G(A+B)-A)||_1]


    ℒ_shape_B(G)=𝔼_A ∼ p_data(A), B ∼ p_data(B)[||label_B*(G(A+B)-B)||_1]

where A represents the fundus image, B represents the DSA image, and label_A and label_B are the retinal vessel annotation and the Frangi segmentation results, respectively.



  
Discriminator: We expect that the background of Fake B is sufficiently similar to the background of a DSA image, so we first use the annotations and Frangi analysis results to mask out the vascular regions and extract the background in the generated and real DSA images,

    Fake_bg=(label_A ∪ label_B)*G(A+B)


    Real_bg=(label_A ∪ label_B)*B^*

Where B^* (Real B^* in Figure <ref>) represents a randomly chosen DSA image before the injection of the contrast medium, which guarantees a vessel-free background. For the structure of the discriminator, we adopt PatchGAN <cit.>.  The adversarial loss between the generator and the discriminator can be expressed as,

    ℒ_GAN(G,D)   =𝔼_B^* ∼ p_data(B^*)[logD(Real_bg)]
       +𝔼_A ∼ p_data(A), B ∼ p_data(B)[log(1-D(Fake_bg))]




  
Segmentor: The main structure of the segmentor is also a U-Net. We use MultiLabelSoftMarginLoss <cit.> as the objective function of the segmentor:

    ℒ_seg(S)=-(y(i)log[exp(ŷ(i))/1+exp(ŷ(i))]+(1-y(i))log[1/1+exp(ŷ(i))])

where ŷ is the prediction and y the synthetic label.

Therefore, the final objective function of our proposed model is,

    ℒ(G,D,S)= ℒ_GAN(G,D) + ℒ_seg(S) + λℒ_shape_A(G) + μℒ_shape_B(G)

where λ and μ control the relative importance of the objectives. During training, we set λ=100,μ=50.

The model uses instance normalization <cit.> instead of batch normalization <cit.>, the generator uses ReLU and the discriminator uses LeakyReLU as activations.



 §.§ Add U-Net

Figure <ref> shows the overall structure of Add U-Net. Referring to Mixup, the model takes the average of the fundus image (Real A) and the DSA image (Real B). The manual annotation of retinal vessels and the Frangi analysis results of the DSA image are combined to obtain the label of the added image. We then train a U-Net with such  added images and annotations. An independent DSA dataset is used to evaluate the trained U-Net.




§ EXPERIMENTS

In this section, to evaluate the effectiveness of our proposed SC-GAN, we compare the segmentation of four methods: 1) Frangi Algorithm Multi-scale Frangi vessel analysis is used to segment coronary arteries. 2) Classic U-Net. A U-Net model is trained using Frangi Algorithm results as the learning targets. 3) Add U-Net. A U-Net model is trained using an average of fundus photographies and DSA images. 4) SC-GAN The proposed shape-consistent GAN. We also compare the synthetic results of SC-GAN and Cycle-GAN <cit.>. The data and experiment details are presented below.


  
Data:
We use the DRIVE <cit.> dataset as the source domain of knowledge transfer. The DRIVE dataset includes 40 fundus images with manually annotated retinal vessels. We also collect 1092 coronary angiographies (DSA) with no annotations as the target domain of knowledge transfer. Several preprocessing approaches are performed on the fundus images, including color to grayscale transform, median-filtering and contrast-limited adaptive histogram equalization <cit.>. Finally, we resize all images into the same size of 512×512 and randomly choose 256×256 patches as inputs of the models.


  
Experiment details:
In all experiments, 50% of DSA images are randomly selected as training set, 20% are validation set and 30% are test set. Meanwhile, we use the Adam solver <cit.> with a learning rate of 2e-4. After training for 50 epochs, we decrease the learning rate linearly for 50 epochs till 0.



§ RESULTS

In this section, we briefly report the evaluation results in two aspects: 1) Images synthesis and 2) Images segmentation.


  
Images synthesis:
Figure <ref> shows some examples of the fundus, DSA, and synthetic image patches. Compared to the results of Cycle-GAN <cit.>, the synthetic images from our proposed SC-GAN have more realistic DSA background and also preserve the vascular structures corresponding to the labels (see the columns (c) and (d) in Figure <ref>).



  
Images segmentation:
We annotate 30% of the DSA dataset (328 out of 1092 images) and evaluate our proposed model on it. Table <ref> compares the performance of different methods. As the baseline method of this article, the Frangi algorithm has a Dice score of 0.636±0.046. If the result of the Frangi algorithm is used as an annotation to train a U-Net (Classic U-Net), the Dice score reduces to 0.589±0.049. Both Add U-Net and SC-GAN have higher Dice scores (0.742±0.048 and 0.824±0.026). And SC-GAN also outperforms the other methods in terms of accuracy and recall. Figure <ref> shows some typical examples in the test set. Columns (d-e) show better results than columns (b-c), indicating that knowledge transfer effectively enhances the identification of small blood vessels. By comparing the results of Add U-Net and SC-GAN, we can also find that GAN is better than an average in terms of the quality of knowledge transfer.





§ DISCUSSION

In this paper, we proposed a shape-consistent GAN model (SC-GAN) for coronary artery segmentation, which was able to transfer the knowledge of the segmentation on public fundus dataset to an unlabeled DSA dataset. Experimental results demonstrated that SC-GAN obtained an obvious superior performance on coronary arteries segmentation. Despite the promising results, our method has several limitations and requires further investigation: 1. How well does the method perform on other datasets; 2. Although the segmentor is light-weighted in test, training SC-GAN is much more complex than in a classic supervised deep model. In future work, we will further test the proposed SC-GAN on other application scenarios and simplify the  training process.



  §.§.§ Acknowledgments.
 This work was supported by National Key R&D Program of China (No. 2018YFC0910700); The National Natural Science Foundation of China (NSFC) under Grants 81801778, 11831002; Beijing Natural Science Foundation (Z180001).


10
	
	
	
	
	adams1994seeded
	Adams, R., Bischof, L.: Seeded region growing. IEEE Transactions on pattern
	analysis and machine intelligence  16(6),  641–647 (1994)
	
	frangi1998multiscale
	Frangi, A.F., Niessen, W.J., Vincken, K.L., Viergever, M.A.: Multiscale vessel
	enhancement filtering. In: International conference on medical image
	computing and computer-assisted intervention. pp. 130–137. Springer (1998)
	
	ganin2016domain
	Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette,
	F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural
	networks. The Journal of Machine Learning Research  17(1),
	2096–2030 (2016)
	
	hoover2003locating
	Hoover, A., Goldbaum, M.: Locating the optic nerve in a retinal image using the
	fuzzy convergence of the blood vessels. IEEE transactions on medical imaging
	22(8),  951–958 (2003)
	
	hu2013automated
	Hu, Q., Abràmoff, M.D., Garvin, M.K.: Automated separation of binary
	overlapping trees in low-contrast color retinal images. In: International
	conference on medical image computing and computer-assisted intervention. pp.
	436–443. Springer (2013)
	
	ioffe2015batch
	Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training
	by reducing internal covariate shift. In: International Conference on Machine
	Learning. pp. 448–456 (2015)
	
	kamnitsas2017unsupervised
	Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A.,
	Menon, D., Nori, A., Criminisi, A., Rueckert, D., et al.: Unsupervised domain
	adaptation in brain lesion segmentation with adversarial networks. In:
	International conference on information processing in medical imaging. pp.
	597–609. Springer (2017)
	
	kingma2014adam
	Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv
	preprint arXiv:1412.6980  (2014)
	
	lapin2018analysis
	Lapin, M., Hein, M., Schiele, B.: Analysis and optimization of loss functions
	for multiclass, top-k, and multilabel classification. IEEE transactions on
	pattern analysis and machine intelligence  40(7),  1533–1554 (2018)
	
	nie2017medical
	Nie, D., Trullo, R., Lian, J., Petitjean, C., Ruan, S., Wang, Q., Shen, D.:
	Medical image synthesis with context-aware generative adversarial networks.
	In: International Conference on Medical Image Computing and Computer-Assisted
	Intervention. pp. 417–425. Springer (2017)
	
	osher1988fronts
	Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed:
	algorithms based on hamilton-jacobi formulations. Journal of computational
	physics  79(1),  12–49 (1988)
	
	staal2004ridge
	Staal, J., Abràmoff, M.D., Niemeijer, M., Viergever, M.A., Van Ginneken,
	B.: Ridge-based vessel segmentation in color images of the retina. IEEE
	transactions on medical imaging  23(4),  501–509 (2004)
	
	tsai2018learning
	Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.:
	Learning to adapt structured output space for semantic segmentation. In:
	Proceedings of the IEEE Conference on Computer Vision and Pattern
	Recognition. pp. 7472–7481 (2018)
	
	ulyanov2016instance
	Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing
	ingredient for fast stylization. arXiv preprint arXiv:1607.08022  (2016)
	
	zhang2017mixup
	Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical
	risk minimization. arXiv preprint arXiv:1710.09412  (2017)
	
	zhang2018translating
	Zhang, Z., Yang, L., Zheng, Y.: Translating and segmenting multimodal medical
	volumes with cycle-and shape-consistency generative adversarial network. In:
	Proceedings of the IEEE Conference on Computer Vision and Pattern
	Recognition. pp. 9242–9251 (2018)
	
	zhu2017unpaired
	Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image
	translation using cycle-consistent adversarial networks. In: Proceedings of
	the IEEE International Conference on Computer Vision. pp. 2223–2232 (2017)
	
	zuiderveld1994contrast
	Zuiderveld, K.: Contrast limited adaptive histogram equalization. In: Graphics
	gems IV. pp. 474–485. Academic Press Professional, Inc. (1994)
	











