MSTT-199: MRI Dataset for Musculoskeletal Soft Tissue Tumor Segmentation
Tahsin Reasat Stephen Chenard Akhil Rekulapelli Nicholas Chadwick Joanna Shechtel Katherine van Schaik David S. Smith Joshua Lawrenz

What is Normal? A Big Data Observational Science Model of Anonymized Internet Traffic

Research was sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Use of this work is controlled by the human-to-human license listed in Exhibit 3 of https://doi.org/10.48550/arXiv.2306.09267


    
Jeremy Kepner^1, Hayden Jananthan^1, Michael Jones^1,  William Arcand^1, David Bestor^1, William Bergeron^1, 
 Daniel Burrill^1,  Aydin Buluc^2, Chansup Byun^1, Timothy Davis^3, Vijay Gadepally^1, Daniel Grant^4, Michael Houle^1, 
 Matthew Hubbell^1,  Piotr Luszczek^1,5, Lauren Milechin^1, Chasen Milner^1, Guillermo Morales^1, Andrew Morris^4, 
 Julie Mullen^1, Ritesh Patel^1, Alex Pentland^1, Sandeep Pisharody^1, Andrew Prout^1,  Albert Reuther^1, Antonio Rosa^1, 
 Gabriel Wachman^1, Charles Yee^1, Peter Michaleas^1



^1MIT,  ^2LBNL, ^3Texas A&M, ^4GreyNoise, ^5University of Tennessee

    September 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



empty





§ INTRODUCTION


A musculoskeletal soft tissue tumor (MSTT) is an abnormal growth or mass that develops within the 
soft tissues of the body that support and connect the musculoskeletal system <cit.>. Soft tissues encompass a 
variety of structures, including muscles, tendons, ligaments, fat, blood 
vessels, nerves, and connective tissues. MSTTs can arise from any 
of these tissues and can be either benign (non-cancerous) or malignant 
(cancerous).
Benign MSTTs, such as lipomas or fibromas, typically grow slowly 
and do not invade surrounding tissues or metastasize to other parts of the body. 
Malignant MSTTs, on the other hand, can be aggressive and have the 
potential to spread to nearby organs or distant sites, posing a more significant 
health risk.
<cit.>

Tumor segmentation allows for precise delineation of the tumor boundaries, 
providing accurate measurements of size and shape. This information is 
essential for disease staging and determining the appropriate treatment 
strategy <cit.>. Moreover, precise segmentation facilitates the 
monitoring of tumor progression and response to therapy over time, enabling 
clinicians to make timely adjustments to the treatment plan 
<cit.>. Additionally, segmentation enables the identification of 
heterogeneous regions within the tumor allowing diagnosis of varying levels of 
malignancy <cit.>. 

A crucial step of building an automated model 
that identifies benign and malignant tumors is the manual segmentation of the 
tumor <cit.>. 
Segmentation is challenging, as tumor appearance can vary in shape, intensity, 
and tissue composition <cit.>. Additionally, the presence of 
artifacts such as noise, motion, and magnetic susceptibility can further obscure 
tumor boundaries, making segmentation challenging.  Moreover, the lack of 
standardized protocols for acquiring MRI data, leading to variations in image
contrast and quality across different institutions and scanners adds to the 
difficulty.
Even if the clinician has sufficient expertise, manual delineation of tumors in three dimensions is 
a time-consuming process, taking up to half an hour per 
MRI volume <cit.>.

In recent times, researchers have strived to automate the MSTT
segmentation process by employing various classical machine learning 
<cit.> and deep learning  <cit.> based methods. Researchers have explored models 
that take both single and multimodal images (MRI, PET, CT scans) as input and 
predict tumor segmentation. These models have been trained and evaluated using a 
small dataset of 51 patients presented in <cit.>.
The progress of automatic MSTT 
segmentation models has lagged due to the unavailability of large diversified datasets. 
To address this problem we have curated a dataset and trained a segmentation 
model using the data. The contributions of this paper are fourfold:

	
  * We created an MSTT segmentation dataset with 199 patients 
		and plan to make it publicly available for future research;
	
  * We described our process of selecting the patients, setting up the 
		labeling platform, the annotation protocol, and the curation method;
	
  * We created a segmentation model based on the curated data which 
		achieves state-of-the-art (SOTA) result on the only available public 
		dataset and analyzed the results; and
	
  * We identify easy versus hard-to-detect types of tumor and make 
		suggestions for future model development as well as data collection.
		


The paper is organized in the following sections. Section <ref> 
contains the detailed creation of the dataset. Section <ref> includes 
the architectural explanation of the segmentation models used in this work. 
Section <ref> contains the experimental setup and result analysis.
And finally the paper is concluded in Section <ref>.



§ MSTT-199: DATASET DESCRIPTION



In this section, we describe the 
process of patient selection,  data annotation, annotation protocol, and data 
curation method.



 §.§ Patient Selection


After receiving institutional review board approval, we queried our 
institution's orthopaedic oncology registry, which includes all patients treated 
for an MSTT at our institution since 1987. Using 
this registry, we initially identified 2,639 patients who underwent definitive 
oncological resection at Vanderbilt University Medical Center and had one of the 
following diagnoses on final pathological review: schwannoma (benign nerve 
tumor), MPNST (malignant peripheral nerve sheath tumor), well-differentiated liposarcoma (benign fat tumor), dedifferentiated liposarcoma (malignant fat tumor), desmoid fibromatosis (benign fibrous tumor), undifferentiated pleomorphic sarcoma 
(malignant fibrous tumor), hemangioma or arteriovenous malformation (benign 
vascular tumor), angiosarcoma (malignant vascular tumor), myxoma (benign myxoid 
tumor), or myxoid fibrosarcoma (malignant myxoid tumor). The tissue type of the tumors were divided in five broad categories Fibrous, Fat, Myxoid, Nerve, and Vascular.

We retrospectively 
reviewed the electronic health records of patients in reverse chronological 
order and sequentially included patients who had a tumor with largest dimension 
greater than 3 cm, and a pre-operative MRI that included both an axial T1 and an 
axial T2 fat-saturation sequence. Patients with MRI sequences that were deemed 
to be incomplete or of poor image quality as determined by a board-certified 
orthopedic oncologist or musculoskeletal radiologist were excluded. Records were 
reviewed until ∼40 eligible patients with each tumor tissue type (Fibrous, Fat, 
Myxoid, Nerve, Vascular) were included, or all available records were exhausted. 
This resulted in a collection of 199 patients with 199 MSTTs. 












 §.§ Labeling Platform


We selected LabelStudio <cit.> as our data annotation platform. 
LabelStudio provides a web browser-based labeling platform that has minimal user 
setup overhead at the annotator's end. Radiologists can log in through the 
provided URL, create an account, and start annotating the assigned image. 
LabelStudio provides a customizable user management system through which user 
activity and dataset progress can be tracked with ease. 

Both T1- and T2-weighted images were present for each tumor and registered 
using ANTs <cit.> algorithm on the 3D Slicer software 
<cit.>. The smaller image of the pair was also resampled to match 
the resolution of the largest image. Following registration, the axial slices of 
MRI images were exported to PNG files and listed on the project along with 
necessary metadata such as patient ID, modality, type of tumor, slice instance 
number, anatomy, etc. While logged in to the platform, the annotator could 
search images via patient ID and modality and sort by instance number. After 
selecting an image, the annotator drew segmentation masks on top of the tumor 
area using a brush tool. A screenshot of the annotation user interface is shown 
in Fig. <ref>. The size of the brush tool can be varied for fine 
or coarse masks. The choice of brush size creates a trade-off between accuracy 
and annotation time. 






 §.§ Annotation Protocol





Annotations were done primarily on the T2-weighted images except for the Fat tumors which were mostly done on the T1-weighted images due to better tumor visibility. The axial slices of the images were uploaded on a privately deployed annotation platform called LabelStudio <cit.>. The annotators logged into the platform to annotate and submit their annotations. 

The data was annotated in three stages. In the first stage, a radiologist (K) 
annotated the center slice of the tumor present in each MRI. In the second 
stage, three annotators (S, A, T

)
annotated the adjacent slices of the annotation following the guiding 
annotations done by K. The questions or confusion that arose during the second 
stage were all mitigated via discussion with the radiologist. 


In the final stage, the complete annotations were sent to the radiologists, J, 
K, and N 

for final review. The patients were randomly divided among the three 
radiologists and had one radiologist review per image. 

The radiologists identified tumor area regions by looking at T2 signal hyperintensity or edema on fluid-sensitive sequences and also using the contrast between normal muscle or tumors and surrounding fat planes on the conventional T1 sequences. While annotating most of the central mass is demarcated; however, some tumors may have narrow tail-like extension from the margins of the mass which might be excluded. Radiologists took image noise and artifact into consideration during the interpretation of the MR images and discounted them without difficulty.













 §.§ Dataset Statistics











The tissue type statistics and anatomy distribution in the dataset are shown in 
Table <ref>, and examples of tissue images are 
shown in Fig. <ref>. Although we assembled a balanced 
dataset for each tissue type, it was difficult to keep the anatomical location 
distribution balanced. MSTTs are most prevalent in the extremities 
with very few occurrences in the trunk or head and neck region 
<cit.>.



Fig. <ref> shows the tumor size variation for different 
tissue types. The Fat tumors are usually larger in size, and the nerve and 
vascular tumors are comparatively smaller.
Fig. <ref> shows the intensity distribution across the 
tissue types. For fibrous, nerve, and vascular tissue types there is an overlap 
in the distribution of the intensities from the two modalities. However, Fat 
tumors show higher intensity on T1 (Fig. <ref>),  whereas, the myxoid tumors show a higher intensity 
distribution on T2. This justifies our choice to annotate the fat 
tumors on T1 modality and the myxoid tumors on T2 modality.

















§ SEGMENTATION MODELS


In this section, we describe the segmentation model architectures and the loss function used to train the model.



 §.§ Architecture







We used the U-Net <cit.> architecture as our segmentation model.
The U-Net segmentation model is a convolutional neural network architecture 
initially designed for biomedical image segmentation tasks 
<cit.>. U-Net has two main components: an encoder and a decoder. 
The encoder downsamples the input image in steps and reduces it to a 
representation rich in information but with the loss of spatial resolution. The 
decoder takes this representation and gradually upsamples it to reconstruct a 
probability output having the spatial size of the image. The output map is 
thresholded to create a binary mask of the foreground object. There are shortcut 
connections (or skip connections) between different stages of the encoder and 
decoder that have similar feature dimensions which enables information flow 
between feature blocks. The encoder, decoder, and shortcut connections create a 
U-shaped visualization for the model (Fig. <ref>), hence the name U-Net.  



  §.§.§ Encoder

The encoder consists of a series of convolutional blocks. Each block typically 
consists of two consecutive convolutional layers followed by a rectified linear 
unit (ReLU) <cit.> activation function and a max-pooling 
layer. The purpose of this path is to capture context and spatial information 
from the input image while reducing its spatial dimensions. As the network 
progresses through the contracting path, the receptive field increases while the 
spatial resolution decreases.



  §.§.§ Decoder

The decoder is responsible for upsampling the feature maps to the original input 
resolution. Each block in this path consists of an upsampling operation (usually 
transposed convolution or interpolation), followed by concatenation with feature 
maps from the encoder path, and then a series of convolutional layers. The 
concatenation operation helps in preserving fine-grained details from the 
encoder path, facilitating precise segmentation.



  §.§.§ Shortcut Connections

One of the key features of U-Net is the skip connections that directly connect 
corresponding layers between the encoder and the decoder. These connections 
allow the network to bypass the loss of spatial information during downsampling 
and aid in the precise localization of objects in the segmentation masks. Skip 
connections provide a shortcut for gradient flow during training, helping to 
mitigate the vanishing gradient problem and enabling faster convergence.



  §.§.§ Final Layer

The final layer of the network consists of a 1× 1 convolutional layer 
followed by a sigmoid activation function. This layer produces the segmentation 
mask with the same spatial dimensions as the input image, where each pixel 
represents the predicted class or label. 



 §.§ Segment Anything Model (SAM)

The segment anything model (SAM) <cit.> is a foundational 
model proposed for image segmentation. It has been trained on a large dataset of 
diverse images and can be applied to a wider range of segmentation tasks. This 
model has been further fine tuned on a large-scale medical image segmentation 
dataset with 1,570,263 medical image-mask pairs, covering 10 imaging modalities, 
over 30 cancer types, and a multitude of imaging protocols to create the MedSAM 
model <cit.>.  The SAM model has three components, illustrated in 
Fig. <ref>: an image encoder, a flexible prompt encoder, and a fast mask 
decoder. These components are described at a high level here.



  §.§.§ Image Encoder

SAM utilizes a pre-trained Vision Transformer (ViT) <cit.> 
that is adapted to process high-resolution inputs. This encoder runs once per 
image and can efficiently process the image before any prompting.



  §.§.§ Prompt Encoder
 In the original SAM model, two types of 
prompts are considered: sparse (points, boxes, text) and dense (masks). These 
prompts are transformed into an embedding and combined element-wise with the 
image embedding. To make the model fully automated it can be supplied with the 
bounding box of the full image and the model predicts masks for all available 
foreground objects.



  §.§.§ Mask Decoder

The mask decoder takes the image embedding and the prompt embeddings to produce 
a mask. It employs a modified transformer decoder block which uses prompt 
self-attention and cross-attention to update all embeddings 
<cit.>. The output of these attention layers is upsampled, 
and a fully connected layer maps the output token to a dynamic linear 
classifier, which computes the mask foreground probability at each image 
location. 




 §.§ Loss

Segmentation models are typically trained using stochastic gradient descent which 
optimizes a loss function computed over the final layer.
We used binary cross entropy as our loss function:


    ℒ_bce =     y ln(p) + (1-y) ln(1-p), y∈{0, 1}.


 


§ EXPERIMENT AND RESULT ANALYSIS


In this section, we explain the details of the used datasets, along with the 
parameters for data preprocessing and model training. Additionally, we analyze 
the results produced from the experiments.



 §.§ External Dataset

Alongside evaluating the trained models on the test partition of our dataset, we 
performed out-of-domain evaluation on a publicly available soft tissue sarcoma 
dataset (STS)<cit.>, accessible via The Cancer Imaging 
Archive <cit.>. This dataset consisted of 51 patients with 
sarcomas located in the extremities, sourced from various sites and scanners, 
resulting in high heterogeneity. Each patient's data contained four different 
imaging modalities, including two paired MRI scans (T1 and T2) and a PET/CT 
scan. The MRI and PET/CT scans were conducted on different days, leading to 
variations in body positioning and anatomy. Tumor annotations are already 
provided in the dataset, delineated on the T2 scans. Additionally, we 
co-registered each T1 image onto the corresponding T2 image using the ANTs 
<cit.> algorithm in 3D Slicer. However, while processing 
the dataset we found two image pairs 

with excessive movement between the modalities that were excluded in our 
analysis. The processed STS data in the 3D NIfTI format is publicly available at <provide url>.



 §.§ Experimental Details


The 3D MRI images were resampled to have a voxel size of 1 mm × 1 mm 
× 1 mm. The intensity values of each image were clamped between 0.05% and 
99.95% of the distribution of that particular image intensities to exclude 
outliers. The images were normalized using the min-max normalization procedure 
where the minimum and maximum intensities were image dependent.  In the axial, 
sagittal, and coronal direction of the volume, 3 consecutive slices were grouped 
to create a 2.5D slice. Slices with more than 100 tumor voxels were used to 
train the model. We used 5-fold cross-validation over the 199 patients. 


The input size to the model was 256 × 256. Padding or cropping was done if 
needed. The augmentations used were random crop, horizontal, vertical flip, 
gamma, brightness, contrast, Gaussian blur, motion blur, and grid distortion. 





For the U-Net encoder, we used the pre-trained se_resnext50_32x4d 
<cit.> model and for the SAM image encoder we used the 
LiteMedSam model (a smaller version of the MedSAM model with a similar 
performance) weights which was provide in <cit.>. While training the 
segmentation models we fine tuned both the encoder and the decoder. The 
performance of the models was evaluated using Dice coefficient (Dice), 
i.e.,

    Dice = 2TP/2TP+FP+FN
.
Here, TP is true positive, FP is false positive, FN is false negative.  
The full details of the training parameters are listed in Appendix <ref>. During inference the images were resized to 256 × 256. We did test time augmentations during the inference (horizontal and vertical flip, rotated by 90) of each slice and the average output was taken. For each image volume, we conducted inference in the axial, coronal, and sagittal  direction and averaged the output.




 §.§ Result Analysis












In Table <ref>, we report the mean dice score achieved in the 
5-fold cross-validation experiment in the MSTT-199 dataset as well as the STS 
dataset and compare it with the existing model in the literature. For the STS 
dataset, there is no predefined test set, so we evaluate our model on the whole 
dataset. The existing Multi-Branch U-Net <cit.> model uses a 
5-fold cross-validation approach (which involves domain-specific training). 
Additionally, it uses multiple imaging modalities (MRI and PET) as input. 
Whereas, our simpler U-Net with no domain-specific training, outperforms the 
existing benchmark. This shows the diversity and usefulness of our dataset. 
Lower Dice scores on the MSTT-199 test domain show that our dataset has much harder 
samples compared to STS. Additionally, we observe that the LiteMedSAM model does 
not outperform the U-Net-based model. This is probably because the LiteMedSAM did 
not have an STT segmentation dataset in its large-scale pretraining phase and 
may not have learned features that are related to the STT segmentation task. Additionally, it's pretraining task was a semi-supervised prompting based approach where a tumor bounding box was provided alongside images. Without this additional information the model performance seems to suffer.

In Table <ref>, we report the average Dice score 
obtained across tissue types and show the distribution spread using boxplots in 
Fig. <ref>. The models perform best in segmenting myxoid 
and Fat tumors and perform worst in fibrous and vascular tissue types. Fat and 
myxoid tumors generally have a large homogeneous structure and are easy to 
differentiate from the background. Fibrous and vascular tissue often have a 
heterogeneous structure making it difficult to differentiate from the 
surrounding tissue. We visually confirm our observations in Fig. <ref>. A larger list of prediction failure montages 
can be found in Appendix <ref>.













 §.§ Effect of Volume


In general, larger tumor volumes are easier to segment for the model. Fig. <ref> shows the performance variation of the model across 
different tissue types. Especially for the fibrous tissue type,  there's a clear 
trend of performance increase with volume. 





 §.§ Effect of Anatomy


In Table <ref>, we report the average Dice scores 
across different tumor locations. As expected, the model does better in the 
extremities compared to the non-extremity locations due to having more sample 
representation in the training set. 





































 §.§ Effect of Tumor Intensity


There is a weak positive correlation between average tumor intensity in the T1 
image and dice scores for Fat tissue (Fig. <ref>). This 
is expected as lipid content is more distinguishable in the T1 image. However, 
for the rest of the tissue types, an increase in brightness has a no positive effect 
on tumor localization as those tissues are generally darker in the T1 image. 





 §.§ Suggestion on Future Data Collection


Although our dataset is large enough to train a state-of-the-art model, our result analysis 
suggests a dire need for a larger dataset to capture more diversity of 
tissue structures. Although we have set up a balanced dataset in terms of tissue 
types, there is an imbalance in terms of anatomy. In Table 
<ref>, we see fibrous and vascular tissue 
types have diverse anatomical representations. As observed in Table 
<ref> and Table <ref>, these 
tissue types have poorer Dice scores compared to myxoid and Fat, even in 
the most common anatomical locations such as the extremities. The lesser 
representation of these difficult tissue types adds up with the challenging 
visual characteristics (small size, unclear tumor boundaries) and worsens the 
learning capability of the model. Future iterations of this dataset should focus 
on collecting more of these less representative and diverse tissue types.





§ CONCLUSION



In this work, we have described the creation of an MSTT dataset. We 
have trained a segmentation model on this dataset and benchmarked its 
performance on a publicly available dataset which achieves state-of-the-art result on MSTT segmentation. Results show that the segmentation models 
work well for the Fat, myxoid, and nerve tumors but struggle to segment tumors 
on fibrous and Vvscular tumors. The segmentation model is sensitive to the 
volume of the tissue as well as the tumor location. Although this is the largest 
tumor segmentation dataset created, the size of the dataset needs to be 
increased further to make the segmentation models more robust. Special priority needs to 
be given to the tumors with fibrous and vascular tissue types as they have 
diverse anatomical locations and have challenging visual characteristics 
compared to Myxoid, Fat, and Nerve tissue.







§ CORRELATION OF TUMOR INTENSITY STATISTICS AND DICE SCORE

Standard Deviation of intensity Fig. <ref>, Fig. <ref>, 

Coefficient of Variation of intensity Fig. <ref> 
<ref>,

The tumor tissue intensity statistics are shown in the following images. I need 
some help in analyzing if these relations are meaningful. Or are there more 
suggestions for exploring different features?
















§ TRAINING PARAMETERS


List of augmentations used from the  package:

=-0.3em
	
  * ,
	
  * ,
	
  * ,
	
  * 
	
  * ,
	
  * ,
	
  * ,
	
  * ,
	
  * ,
	
  * ,
	
  * .


List of model parameters

=-0.3em
	
  * Learning rate: 1e-4
	
  * Batch size: 16
	
  * Epochs: 5





§ PREDICTION FAILURES



A list of images where the segmentation model has failed.















