Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Cord Paralysis
Yucong Zhang Xin Zou Jinshan Yang Wenjun Chen Faya Liang Ming Li




Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Cord Paralysis
    Yucong Zhang, Xin Zou, Jinshan Yang, Wenjun Chen, Faya Liang^∗ and Ming Li^∗
(The first two authors contributed equally to this work.)(Corresponding authors: Faya Liang, Ming Li.)
Ming li and Yucong Zhang are with the department of computer science, Wuhan University, Wuhan 430072, China and the Suzhou Municipal Key Laboratory of  Multimodal Intelligent Systems, Duke Kunshan University, Suzhou 215316, China (e-mail: yucong.zhang@whu.edu.cn, ming.li369@dukekunshan.edu.cn)
Faya Liang, Xin Zou, Jinshan Yang and Wenjun Chen are with the Sun Yat-sen Memorial Hospital of Sun Yat-sen University, Guangzhou 510000, China (e-mail: liangfy3@mail.sysu.edu.cn, zoux25@mail2.sysu.edu.cn, Lucky9812@163.com, 273147323@qq.com)

















    
==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
This paper presents the Multimodal Analyzing System for Laryngoscope (MASL), a system that combines audio and video data to automatically extract key segments and metrics from laryngeal videostroboscopic videos for clinical assessment. MASL integrates glottis detection with keyword spotting to analyze patient vocalizations and refine video highlights for better inspection of vocal cord movements. The system includes a strobing video extraction module that identifies frames by analyzing hue, saturation, and value fluctuations. MASL also provides effective metrics for vocal cord paralysis detection, employing a two-stage glottis segmentation process using U-Net followed by diffusion-based refinement to reduce false positives. Instead of glottal area waveforms, MASL estimates anterior glottic angle waveforms (AGAW) from glottis masks, evaluating both left and right vocal cords to detect unilateral vocal cord paralysis (UVFP). By comparing AGAW variances, MASL distinguishes between left and right paralysis. Ablation studies and experiments on public and real-world datasets validate MASL’s segmentation module and demonstrate its ability to provide reliable metrics for UVFP diagnosis.



Glottal Midline Estimition, Glottis Segmentation, Keyword Spotting, Laryngoscope, Vocal cord Paralysis





§ INTRODUCTION


vocal cord Paralysis (VP) is a condition where one of the vocal cords fails to move properly, leading to voice changes, difficulty swallowing, and potential breathing problems <cit.>, <cit.>. VP can result from nerve damage due to surgery, injury, infection, or tumors, significantly impacting a patient's quality of life <cit.>. Accurate diagnosis of VP is crucial, as it informs the appropriate medical or surgical intervention, which can restore vocal function, improve airway protection, and enhance overall patient outcomes. Clinicians often use laryngeal videostroboscopy to check the vocal cord vibration in details. Laryngeal videostroboscopy is a specialized diagnostic tool used to evaluate the function of the vocal cords and the larynx. The stroboscopic component is key because it allows for the visualization of vocal cord vibrations, which are often too rapid to be observed directly <cit.>. Stroboscopy uses a light source that flashes at a slightly different frequency than the vocal cord vibration. This creates a slow-motion effect, enabling the clinician to see the individual phases of vocal cord vibration in great detail. 

With the advent of the artificial intelligence, deep learning methods are developed to extract useful parameters <cit.> to assist clinicians, track the motion of vocal cords <cit.>, and even make predictions <cit.> or classifications <cit.> to help clinicians to conduct diagnosis. However, the diagnosis of VP requires to inspect the complete phonation cycles of patients. Previous works default to using video frames or images that contain the vocal cords, but in most of the time, the raw recordings from the endoscopic inspection carry useless information other than patient's phonation cycles. For instance, no phonation cycles in the beginning of the inspection, as the laryngoscope is not in its position, still finding the vocal cords. 


Although experienced experts can provide valuable insights by analyzing the video captured by the endoscope, this methodology heavily relies on personal diagnosis and lacks objectiveness, which on one hand reduces patient's confidence, on the other hand increases the risk of misdiagnosis. Thus, analysis tools <cit.> are developed to assist experts in making better diagnoses with the endoscopic video. <cit.> relies on physical features like color and shape to segment glottis area. <cit.> uses supervised method to keep track of the vocal cords' movement to extract the anterior glottic angle (AGAW). However, those three tools <cit.> focus solely on one single modality to assist the clinicians and cannot be used for left or right VP detection. Researchers in <cit.> extract more than one features from the laryngoscope, but those features all come from the AGA between two vocal cords, which cannot be used to detect LVP or RVP.  Moreover, current tools are restricted to pre-processed laryngoscopic video or short video segments without stroboscopic examinations, which cannot be directly used on raw and long laryngoscopic videos. As a result, labor works are still needed beforehand, which makes it inconvenient for the experts to use. 

In addition to video processing, audio-related methods have shown great potential to detect voice pathology <cit.> as well. Such method typically involves converting audio clips to spectrograms using short-time Fourier transform (STFT), which are then fed into various models for downstream tasks. For instance, deep neural networks (DNN) is utilized to predict vocal cord pathology <cit.>. Low et al. <cit.> builds machine learning model to detect anomalous patterns in the spectrogram. Compton et al. <cit.> uses an artificial neural network (ANN) to predict vocal cord pathology, claiming to outperform expert assessments. However, these approaches focus solely on the patient's audio data, neglecting the crucial visual information, which contains more explainable and comprehensive insights into vocal cord conditions.

In this article, we present the Multimodal Analyzing System for Laryngoscope (MASL), a novel system that leverages both the audio and video modalities to automatically process laryngeal endoscopic videos. The main contributions of our work are listed as follows. 

	
  * Our proposed system employs a multimodal approach, integrating glottis detection with keyword spotting (KWS) technique, to automatically identify and extract multiple video segments that include at least one complete phonation cycle of the patient. In addition, we design a strobing video extraction module by analyzing hue, saturation and value (HSV) of the video frames.
	
  * We propose a robust two-stage segmentation process: an initial U-Net model followed by a diffusion-based refinement that reduces false alarms and improve the segmentation precision.
	
  * We introduce a quadratic fitting method to get a better estimation of AGAWs for both left and right vocal cords and the glottal midline. To the best our knowledge, we are the first to extract AGAW for separate vocal cords. The experimental results show great potential on both LVP and RVP.
	

The organization of this paper is as follows: Section <ref> reviews some related works about U-Net segmentation and AGAW analysis. Section <ref> details our proposed method for key frames extraction, including KWS audio processing, glottis detection and HSV analysis. Section <ref> and Section <ref> focus on the analysis module. Section <ref> presents the two-stage glottis segmentation technique, including the initial U-Net segmentation and the diffusion refinement processes. Section <ref> describes our method to analyze VP, including AGA and glottal midline estimation by quadratic fitting and multimodal VP prediction using audio and AGAWs. Finally, section <ref> presents experimental results, and Section <ref> concludes the paper with a discussion of future work and potential applications.





§ RELATED WORK




 §.§ U-Net Based Glottis Segmentation

U-Net's encoder-decoder structure with skip connections has proved to be a robust framework for biomedical image segmentation <cit.>. Modifications to this architecture, such as the introduction of separable convolutions in S3AR U-Net <cit.> and the incorporation of attention mechanisms in Attention U-Net <cit.>, have demonstrated improved performance by capturing more salient features and reducing computational demands. The pursuit of efficiency and clinical applicability has led to the development of models like Efficient U-Net <cit.>, which offers a practical inference time while maintaining high segmentation quality. Yet, the trade-off between segmentation accuracy and computational cost is a recurring theme, with models like VGG19 U-Net <cit.> showing longer inference times. Despite the computational efficiency of U-Net variants, the need for substantial training data and the potential for overfitting remain concerns <cit.>. To counter this, weakly supervised learning approaches, like the one proposed by <cit.>, have gained traction, requiring only point annotations and demonstrating a remarkable balance between segmentation accuracy and convergence speed. However, these methods may still struggle with complex anatomical structures and the need for precise boundary localization.



While the field has made significant strides, challenges persist in achieving a harmonious balance between segmentation accuracy, and generalizability across varied clinical datasets. 

The limited size of the glottis segmentation mask often leads to a high incidence of false positives when using U-Net, resulting in segmentation outputs even in the absence of glottal regions. This can significantly impair the accuracy of subsequent analyses. To address this issue, we incorporate a diffusion-based refinement stage, which enhances the precision of the initial U-Net segmentation.



 §.§ AGAW Extraction and Glottal Midline Estimation

In the early years, researchers find out that by analyzing the maximum separation between vocal cords, one can effectively detect UVP <cit.>. Later on, researchers begin to use AGA to represent the separation between vocal cords, and AGAW analysis begin to play a critical role for VP detection. However, the existing methods typically assess AGAWs without differentiating between the left or right vocal cords <cit.>. To tackle this, our proposed approach introduces a technique that separately extract AGAWs for each vocal cord. By contrasting the AGAWs of the left vocal cord with the right one, we show that LVP and RVP are distinguishable. 

However, determining left or right AGAW requires identifying the glottal midline, since AGAW for each vocal cord is a reflection of the distance to the glottal midline. Previous methods rely on extensive labeled data <cit.> to form supervised learning, which is labor-intensive. Our method can automatically estimate the midline of the glottic area without using any annotated data.




§ MULTIMODAL KEY FRAMES EXTRACTION




 §.§ System Design




Our system aims to facilitate efficient clinical examinations by extracting key segments from laryngoscopic videos and providing objective indicators for specific laryngeal diseases. As Fig. <ref> shows, comprising two main modules – the voice module, the video module – our system ensures the accurate observation of vocalization cycles and the clear visualization of the glottal area.

 The voice module initially processes the audio extracted from the video using short-time Fourier transform to obtain spectrograms. Through keyword spotting (KWS) technique, each frame is analyzed to detect patient vocalizations. This enables the preliminary segmentation of vocalization segments within the video. 

Subsequently, the video module refines these vocalization segments to obtain key frames to form laryngoscopic highlights, ensuring the visibility of the vocal cords. Specifically, by utilizing the glottis detection model, MASL can identify regions containing the vocal cords and glottis in each frame. Moreover, given the importance of the stroboscopic portions in laryngoscopic videos for subjective analysis by physicians, we also include a stroboscopic video extraction method into our system. 







 §.§ Audio Processing Module

To get a better examination of the status of the vocal cords, clinicians often request the patients to pronounce "ee". This pronunciation is easy to be made and can largely enable the vibration of the vocal cords. Hence, by capturing the segments that pronounce "ee", phonation cycles might be captured. To accomplish this, we developed a KWS model that is used to detect keywords in a sentence, such as "Hey Siri", "Ok Google", and in our case, the word "ee".

The overall pipeline is illustrated in Fig. <ref>. Initially, the input audio is transformed into a spectrogram using the Short-Time Fourier Transform (STFT), converting the time-domain signal into a time-frequency representation for more effective analysis. The spectrogram is then segmented into chunks along the time axis, with each chunk containing a fixed number of frames. During the training phase, these chunks are randomly selected from each audio clip to ensure a diverse set of training samples. This method enhances the model's generalization capabilities by incorporating different spectrogram chunks from various audio clips into a robust batch of training data.

The spectrogram chunks are subsequently fed into the KWS model, the architecture of which is detailed in TABLE <ref>. The model comprises multiple convolutional blocks and residual blocks <cit.>, designed to efficiently extract relevant features from the spectrogram chunks. Within the model, we use max pooling to progressively reduce the spatial dimensions. In this way, the features are compressed and become more meaningful. In the end, we use an adaptive average pooling layer to aggregate the features before they pass through two fully connected layers, with the final layer producing the classification score output.

During inference, the input spectrogram is sliced into chunks using a sliding window. Each chunk is processed by the trained KWS model to generate decision results, as depicted in Fig. <ref>. This carefully constructed pipeline ensures that the KWS model reliably detects the "ee" vocalization, providing critical prior knowledge for subsequent analysis.



 §.§ Video Processing Module


Although the KWS model is employed to detect the vocalization video frames of the phonation of "ee", these detected frames may not always successfully capture the vocal cords and glottis. To address this issue, we train a vocal cord detection model, utilizing the famous object detection model structure YOLO-v5 <cit.>, to further refine the time masks.

Given the limitations of the open-source data for training a detection model, we utilize the public glottis segmentation dataset, BAGLS <cit.>, to construct our training dataset. To generate labels for vocal cord detection, we designed an automatic bounding box generator. As illustrated in Fig. <ref>, the process begins by obtaining the coordinates of the top, bottom, left, and right vertices from the glottis mask. These coordinates are denoted as U(x_1, y_1), D(x_2, y_2), L(x_3, y_3), and R(x_4, y_4), respectively. Next, these vertices are manually expanded by a fixed number of pixels to ensure adequate coverage around the vocal cords. Using the expanded points, a bounding box is computed, as shown on the right side of Fig. <ref>.

As depicted in Fig. <ref>, our pipeline integrates a strobing video extraction module alongside the detection modules. This module is utilized to isolate the strobing segments of laryngeal videos by analyzing  hue, saturation, and value (HSV) of video frames. The HSV parameters represent type, intensity, and brightness of colors, respectively. Transitions marked by empty frames typically occur at the beginning and end of strobing segments. We employ a unit step function to identify and mark all empty frames with a zero value in color (illustrated by the yellow line in Fig. <ref>). This technique segments the video into several continuous non-empty frame sequences. Within these non-empty segments, we examine the variations in HSV values. Due to the rapid color changes characteristic of strobing videos (as shown in Fig. <ref>), we are able to identify the strobing segments by calculating the frequency of HSV fluctuations within each non-empty segment, selecting the segment with the highest number of fluctuations.





§ GLOTTIS SEGMENTATION


In the evaluation of laryngeal function and pathology, accurate segmentation of the glottis is essential. This segmentation enables the extraction of objective metrics that clinicians can utilize for diagnosis and review. In our system, we 


implement a two-stage approach comprising a U-Net-based method followed by a diffusion-based refinement. The initial segmentation is performed using a naive U-Net model, which is simple yet effective in medical image segmentation tasks. This model provides a robust initial estimate of the glottis boundaries. 

However, the U-Net model tends to produce false positives, especially in cases where the glottal area is not visible (see Table <ref>). These incorrect segmentations can complicate subsequent analysis stages. To address this issue and enhance the precision of the segmentation results, we further refine the U-Net outputs using a diffusion model. This additional step helps to correct any inaccuracies and produce a more accurate and reliable glottis mask. The combination of these two methods ensures high-quality segmentation, which is crucial for subsequent analysis and metric computation.

The following subsections provide detailed descriptions of each component in our glottis segmentation pipeline, including the U-Net-based method and the diffusion-based refinement.





 §.§ U-Net-based Method


The U-Net model is a convolutional neural network specifically designed for biomedical image segmentation. Owing to its proven effectiveness and accuracy, we employ a U-Net as our segmentation model. As detailed in Table <ref>, the model comprises a series of convolutional blocks, referred to as ConvBlocks (see Table <ref>), which include convolution operations followed by batch normalization and ReLU activation functions. The U-Net architecture is symmetrical, featuring a contracting path that captures contextual information and an expansive path that facilitates precise localization through upsampling operations. The contracting path consists of ConvBlocks with progressively increasing numbers of channels (64, 128, 256, and 512), interspersed with MaxPool2D layers for downsampling. This is followed by a sequence of upsampling operations and ConvBlocks that reduce the feature map dimensions, effectively reconstructing the image to its original resolution. The final layer utilizes a ConvBlock with a single output channel to generate the segmentation mask. 




There are several advantages to using a U-Net model with this structure. First, the architecture captures both low-level and high-level features through its deep structure and skip connections, which concatenate features from the contracting path to the corresponding layers in the expansive path. This facilitates precise segmentation by preserving spatial information. Second, the use of ConvBlocks with batch normalization stabilizes and accelerates the training process, while ReLU activation functions introduce non-linearity, enabling the model to learn complex patterns. Additionally, the upsampling layers allow for finer segmentation outputs by gradually reconstructing the image details. 






 §.§ Diffusion-based Refinement


The U-Net model excels at extracting image masks. However, the generated masks may tend to yield false alarms (see Table <ref>). To further enhance the quality, we explore the integration of the diffusion-based model. By incorporating the diffusion model, we can inject more diversity into the masks extracted by U-Net, making the generated images richer and more diverse.

Diffusion models consist of two stages: forward diffusion and reverse diffusion. During the forward process, Gaussian noise is gradually added to the segmentation label x_0 over a series of steps T. Conversely, in the reverse process, a neural network is trained to recover the original data by reversing the noise addition, as represented by the following equation:

    p_θ(x_0: T-1| x_T)=∏_t=1^T p_θ(x_t-1| x_t),

where θ stands for the parameters for the reverse process.

Consistent with the conventional implementation of the Diffusion Probability Model (DPM) <cit.>, a U-Net model is employed for training. Following the idea of MedSegDiff <cit.>, we incorporate the original glottis images as priors for the step estimation function and employ dynamic conditional encoding to fuse the encoding outcomes from both the raw image and the segmentation mask at each step. Hence, for each step, the estimation function ϵ_θ is written as:

    ϵ_θ(x_t, I, t)=D((E_t^I+E_t^x, t), t),

where θ stands for the learning parameters, D is the decoder, I is the raw image prior, t is the current diffusion step. E_t^I and E_t^x are the embeddings encoded from the raw image and segmentation mask at step t respectively.

Different from the traditional training procedure, we do not start the diffusion process from a standard Gaussian noise, p_θ(x_T)=𝒩(x_T; 0, I_n× n) for an n× n image. Instead, by integrating the U-Net result introduced in Section <ref>, we start the diffusion process from a customized Gaussian noise, which is:
	
    p_θ(x_T)=𝒩(x_T; μ^', I_n× n),

	
    μ^' = (1 - (α·(1-m^')+(1-α)· m^'))× 10^-3,

where α∼U(0,0.3) is a random parameter sampled from a uniform distribution, m^' is the mask generated by the U-Net model in Section <ref>.

We determine the new mean for the Gaussian noise in the diffusion process by computing the weighted average between the glottis mask initially generated by U-Net and its complement (see Equation <ref>). This essentially guides the diffusion process to pay more attention to areas outside the glottal region, encouraging the model to refine the segmentation boundaries for improved accuracy.



§ MULTIMODAL VOCAL CORD PARALYSIS ANALYSIS




To detect VP, we adopt a two-stage process. First, we build a binary classification model to check if it belongs to VP. Then, with our proposed metrics, LVP and RVP can be identified by comparing the left and right vocal cord movement via the AGA.



 §.§ Anterior Glottic Angle Extraction with Quadratic Fitting

To diagnose the laryngeal paralysis in a more detailed way, instead of the whole glottal angle, we extract the glottal angles for left and right vocal cords separately. As a result, we can provide the contrastive diagnosis of laryngeal paralysis. The procedure for extracting glottal angles from a segmented glottis involves a systematic series of steps, detailed in Algorithm <ref>, with visual representations of the intermediate results provided in Fig. <ref>. 



Initially, the algorithm acquires the coordinates of the top, bottom, left, and right vertices from the glottis mask, computing their center point denoted as U(x_1, y_1), D(x_2, y_2), L(x_3, y_3), R(x_4, y_4), and C(x_c, y_c) respectively (see Fig. <ref>(a)). Subsequently, a line connecting points C and D is established, and the function f(x) passing through C and D is computed.

Equidistant points C_1, C_2, ..., C_N-1 are then positioned along the line segment intercepted by the glottal mask. For each C_k, orthogonal functions f^'_k(x) to f(x) are calculated, with their intersection points L_k and R_k with the glottis mask determined for k∈ [1, N-1] (see Fig. <ref>(b)).

To ensure uniformity, the coordinate system is rotated by an angle γ, aligning all intersection points along the y-axis after rotation, denoted as L_γ,k and R_γ,k. Leveraging these points, a quadratic curve q_γ(x) is approximated in the rotated coordinate system. The lowest point D_q of q_γ(x) is identified and mapped back to the original coordinate system (see Fig. <ref>(c)).

Subsequently, a calibrated middle line f^*(x) connecting points C and D_q is established, intersecting the glottis mask at point D^'. Finally, glottal angles are derived by connecting points {L_k}_k∈ [1, N-1] and {R_k}_k∈ [1, N-1] with D^', yielding {∠ L_kD^' C}_k∈ [1, N-1] and {∠ CD^' R_k}_k∈ [1, N-1] respectively.





 §.§ Multimodal vocal cord Paralysis Detection

We developed a model that integrates both audio and video modalities for enhanced diagnosis. As illustrated in Fig. <ref>, the model utilizes the audio spectrogram and the AGA movements as inputs. The audio spectrogram is encoded using EfficientNet-b0 <cit.>, a compact and efficient model renowned for its performance in image classification tasks. Given that multiple video highlights are extracted from a single laryngoscope video, with corresponding AGA movements for each, multiple AGA movement time series is generated. These time series are treated as multi-channel inputs to a ConvLSTM <cit.> model, comprising a convolutional layer followed by an LSTM layer. The ConvLSTM model is adept at handling multi-channel time series; the convolutional layer captures features across all channels, while the LSTM layer processes temporal information effectively.

To further differentiate between LVP and RVP, we analyze the variance of the AGA movements for the left and right vocal cords. Intuitively, the paralyzed side exhibits less activity during phonation, leading to a smoother AGA movement time sequence and consequently a lower variance. Therefore, by simply comparing the variance of the left and right AGA movements, we distinguish between LVP and RVP. 





§ EXPERIMENTS AND RESULTS




 §.§ BAGLS Dataset

We use the Benchmark for Automatic Glottis Segmentation (BAGLS) <cit.> as the dataset for glottis segmentation. It consists of 59,250 endoscopic glottis images acquired from hundreds of individuals at seven hospitals, which is split into 55,750 training images and 3,500 test images.



 §.§ SYSU Dataset

This is the dataset collected from real-world scenario by Sun Yat-sen Memorial Hospital of Sun Yat-sen University (SYSU). The dataset contains 520 video samples, including 106 normal samples and 414 paralysis samples (257 LVP and 157 RVP). All the video clips are recorded using a laryngeal videostroboscope, including one or two strobing video segments for each clip. 



 §.§ Keyword Spotting Model

The detailed architecture of the KWS model is presented in Table <ref>. For audio input, we utilize a Mel-spectrogram with 80 Mel filters, with the number of FFT points and hop length set to 1024 and 512, respectively. A sliding window of 400 samples and a hop length of 64 samples are employed to extract frame-level information. The model is trained for 100 epochs, optimized using the Adam optimizer with a learning rate of 0.005.

Since the KWS model outputs a posterior score rather than a direct decision value, a threshold is necessary to determine the final result. Table <ref> lists the thresholds applied in the system. The precision-recall (PR) curves and receiver operating characteristic (ROC) curves in Fig. <ref> illustrate the classification performance across different thresholds. Consequently, the threshold is selected based on the highest F1-score, a balanced metric that considers both precision and recall, providing a comprehensive measure of classification performance.

Table <ref> displays the results for both doctor and patient. The detection performance for the doctor's voice is inferior to that for the patient's voice due to the doctor's broader range of spoken words compared to the patient's singular utterance of "ee". Our system prioritizes frames where the patient is speaking "ee", resulting in a high F1-score on the test dataset, which underscores the efficacy of our KWS model.









 §.§ Glottis Segmentation


We train the U-Net baseline with the same strategy as <cit.>. For the diffusion model, it is built on the model of <cit.>, except that we slightly modify the noise to meet the prior knowledge of U-Net results. The diffusion step is set to 1,000. We train the diffusion model with 100,000 steps. The model is optimized by a AdamW optimizer with a learning rate of 0.0001.

We evaluate our segmentation model on the public glottis dataset BAGLS. We use intersection over union (IoU) as the metric, and the results are shown in Table. <ref>. From the table, we can see that the diffusion model can have a slightly better IoU performance than the traditional U-Net, By refining the U-Net results with the diffusion ones, the IoU performance can be further improved. 

Besides IoU, we  also compute the false alarm rate (FAR) when using different segmentation methods. The false alarm (FA) is defined as the situation where the glottis mask is generated when no glottis is seen. The FAR is calculated as the number of FA divided by the total number of images with no glottis to be seen. We care about FA as we need the segmentation masks for later analysis. FAs might produce wrong measurements and mislead the prediction results of VP. From Table <ref>, the traditional U-Net model shows a tendency to produce FA masks with a leading FAR of 15.8%. By contrast, diffusion-based segmentation model has a much lower FAR of 4.5%, which is more than two times decrease. The result may indicates that, in terms of the glottis segmentation, the diffusion model performs more robust when dealing with image with no targets. Hence, after we refine the U-Net result with the diffusion model, we can further achieve a result with a lower FAR but a better IoU.



 §.§ vocal cord Paralysis Detection

We manually divided the SYSMH dataset into training and testing subsets. The training set comprises 470 samples (81 normal, 389 paralysis), while the testing set includes 50 samples (25 normal, 25 paralysis). The split was performed randomly, ensuring an equal number of normal and paralysis samples in the test set to facilitate a balanced evaluation of our model's performance. We conduct the same experiment for 5 times with 5 different random seeds, so that the train-test-split operation is different for each run, resulting in performance variability. This approach allows for a comprehensive assessment of the model's overall performance, independent of the specific data subsets used in each iteration.

Ablation studies are conducted using different system configurations to show the effectiveness of our proposed system, as shown in Table <ref>. The naive system uses the images segmented by the U-Net baseline without quadratic fitting or diffusion refinement. Table <ref> summarizes the performance for vocal cord paralysis classification across various system configurations. Each configuration's precision, recall, and F1-score are reported for both normal and paralysis classes, as well as their weighted averages. In the naive settings, the system achieves a precision of 82.0% and recall of 69.0% for normal cases, and a precision of 73.2% and recall of 84.0% for paralysis, with a weighted average F1-score of 76.5%. 

Steady and accurate middle line prediction is critical for analyzing left and right vocal cord movement, since every time when measuring the AGA movement, distance between the sample points on the vocal cord and middle line is computed. However, the laryngeal camera is not always in fixed positions in real-world application. As a result, using the traditional way to estimate the middle line may introduce high variations in middle line positions, producing less robust results. It turns out that calibrating the middle line by fitting a quadratic function might be a good way to enhance the middle line prediction, resulting in a better classification performance. The results from Table <ref> shows that the proposed quadratic fitting method is able to improve the performance by a large margin, improving the balanced precision and recall to approximately 81.5% for both classes. 

After further incorporating the diffusion refinement to the glottis segmentation process, the classification results can be improved further, slightly increasing the weighted average F1-score to 82.5%. The diffusion refinement only involves the images where glottis cannot be detected by the YOLO-v5 detector (described in Section <ref>), since we want to use this method to reduce FAs (described in Section <ref>).

We also conduct experiments on the effect of multimodality. As a result,  by removing the AGA movement metrics, the classification performance drops dramatically, with the weighted average F1-score falling to 72.5%. This result demonstrate the critical role of integrating both audio and AGA movement modalities for robust performance.

We argue that the high standard deviation over all runs due to the fact that we randomly split the datasets for train and test for each run in the experiments. The SYSMH dataset is a challenging dataset with various kinds of recording situations, such as various types of laryngeal videostroboscopes, recording environment, etc., causing domain shifts.

In summary, the comparison shown in Table <ref> underscores the efficacy of quadratic fitting and diffusion refinement in enhancing classification accuracy and also show the critical role of integrating both audio and AGA movement modalities for better performance.





 §.§ Potential of Unilateral Vocal Cord Paralysis Detection


Fig. <ref> illustrates the glottal area and the AGA movement time series of a patient with RVP. These metrics, extracted by MASL, are derived from a single video clip in the SYSU dataset. As three video highlights are extracted from the clip, three distinct time series for each of the two metrics are presented. Fig. <ref>(a), (c), and (e) depict the variations of glottal area across time, indicating the phonation cycle through the opening and closing movements of the vocal cords. However, while the glottal area data reveals the vocal cord activity, it does not specify the movement of each vocal cord individually. By computing the AGA movement time series for both vocal cords, as shown in Fig. <ref>(b), (d), and (f), MASL provides clinicians with more detailed and actionable metrics for inspecting UVP.



From Table <ref>, by comparing the variance of left and right AGA metrics, we are able to distinguish the left or right paralysis for a UVP patient. We also conduct ablation studies to show effectiveness of different modules in our proposed system. Table <ref> illustrates that incorporating quadratic fitting improves the classification results. For both left and right vocal cord paralysis and the weighted average, all the metrics have improved by a large margin. Moreover, adding diffusion refinement can further improve all the metrics, with a weighted average of precision, recall and F1-score of 92%.
These improvements indicate that the quadratic fitting technique and the diffusion refinement method has the ability to enhance the system to model the vocal cord movement, providing a more robust diagnostic tool to the clinicians.



§ CONCLUSION


In this paper, we introduced the Multimodal Analyzing System for Laryngoscope (MASL), which integrates both the audio and video processing techniques to enhance the diagnosis of laryngeal disorders. MASL is able to produce high quality laryngoscopic video highlights for better diagnosis by clinicians. It incorporates KWS that ensures the complete phonation cycle of the patients, and video processing modules that ensure the occurrence of the vocal cords. Moreover, MASL is able to extract the stroboscopic slices from the raw laryngoscopic video, saving clinicians' time finding the stroboscopic slices.

To produce objective metrics for analysis, MASL employs a two-stage segmentation method with a U-Net followed by the diffusion-based refinement to accurately delineate the glottal area, significantly reducing false positives produced by U-Net. With the segmented masks, MASL adopts quadratic fitting and extracts AGA movements for both left and right vocal cords, producing a new modality for VP. With multi-modality, MASL improves the VP performance. Simply by comparing the variance between the AGA movements for left and right vocal cords, MASL yields 92% of F1-score on the real-world dataset. The Experimental results demonstrate MASL's robustness and effectiveness, offering clinicians a powerful tool to improve diagnostic accuracy and efficiency.



§ REFERENCES

IEEEtranbib






































