Unsupervised Welding Defect Detection Using Audio And Video
Georg Stemmer Jose A. Lopez Juan A. Del Hoyo Ontiveros Arvind Raju Tara Thimmanaik Sovan Biswas


Article Title]Unsupervised Welding Defect Detection Using Audio and Video













1]Georg Stemmer
[1]Jose A. Lopezjose.a.lopez@intel.com
1]Juan A. Del Hoyo Ontiveros
1]Arvind Raju
1]Tara Thimmanaik
1]Sovan Biswas





*[1]Intel Corp., 2200 Mission College Blvd., Santa Clara, 95054, CA, USA





In this work we explore the application of AI to robotic welding. Robotic welding is a widely used technology in many industries, but  robots currently do not have the capability to detect welding defects which get introduced due to various reasons in the welding process. We describe how deep-learning methods can be applied to detect weld defects in real-time by recording the welding process with microphones and a camera. Our findings are based on a large database with more than 4000 welding samples we collected which covers different weld types, materials and various defect categories. All deep learning models are trained in an unsupervised fashion because the space of possible defects is large and the defects in our data may contain biases. We demonstrate that a reliable real-time detection of most categories of weld defects is feasible both from audio and video, with improvements achieved by combining both modalities. Specifically, the multi-modal approach achieves an average Area-under-ROC-Curve (AUC) of 0.92 over all eleven defect types in our data. We conclude the paper with an analysis of the results by defect type and a discussion of future work.







[
    [
    
=====





§ INTRODUCTION

Robotic arc welding, i.e., the use of robots for automating the arc welding process, is a key manufacturing technology in many industries. As the quality of a weld depends on many factors, even a robot that repeats each step of the process perfectly will produce defective welds from time to time. The time at which a defect gets detected in the welding process has direct impact on the cost associated with correcting the problem. So to minimize the overall defects and reduce the correction cost there is a growing interest in detecting anomalies in real-time, i.e., during the welding process, rather than post weld defect detection. Ideally, a future intelligent manufacturing system would be able to adjust the welding robot’s parameters automatically even before the failure starts to occur in the welding process.

In this work, we investigate the use of deep learning models for automatic weld defect detection in real-time using a camera and microphones. Cameras can monitor the weld pool geometry and oscillations, which are known to be predictive of weld defects <cit.>. Cameras need to have line-of-sight to the weld pool, and this consequently limits the mounting options. Microphones, on the other hand, can capture audible disturbances in the welding process without line-of-sight. They have been shown to provide useful information about defects <cit.>.
 We investigate in the experimental section for several defect types whether they can be better detected visually or acoustically, and how the two modalities perform in combination.
 
Current deep learning models require large amounts of training data to estimate their parameters. We address this issue by collecting more than 4000 samples of good and defective welds, which is, to best of our knowledge, significantly more than what has been reported for similar datasets in the literature. Still, we consider the number of samples too small to train a classifier that can distinguish each relevant defect type reliably, given the variation of the input signals that can be expected in a real application. Therefore we decided to address the weld defect detection problem with an unsupervised anomaly detection approach. Anomaly detection models are trained on good welds only: all defects in our dataset occur solely during evaluation and are unseen in training. While this makes the defect specific performance more difficult to tune, we believe that the resulting performance is more representative of a real application.

Of course, we are not the first interested in bringing the benefits of AI to established spaces like manufacturing, and welding in particular. Companies have produced AI-based anomaly detection solutions in recent years <cit.>, and  numerous authors have explored the use of modern data-driven algorithms to improve weld defect detection <cit.>, which can be costly to remediate <cit.>. In <cit.>, the authors used sequence tagging and logistic regression to detect welding defects. Mohanasundari et al. <cit.>  used post-weld images to classify defects. Buongiorno et al. <cit.> leveraged thermographic image sequences from an infrared camera to detect defects.  
Our work is also related to predictive maintenance solutions, which tend to use vibration or acoustic-emission sensors. Acoustic emission sensors have even been used successfully to monitor weld quality <cit.> as well. There is good reason for using this type of sensors for machine condition monitoring, as 90% of rotating machinery uses rolling-element bearings, which tend to be points of failure <cit.>. On the other hand, welding is a much more complex process. Therefore we expect that microphones and cameras are better suited for detecting weld defects than acoustic emission or vibration sensors.

The main contributions of this work are the following: We collect a large multi-modal dataset of samples of robotic arc welding in a real industrial environment. It covers different weld types, various welding parameter configurations, and steel types. The size of the dataset allows us to train deep ML models, as opposed to shallow or analytical models often described in the literature (e.g., <cit.>). With these deep ML models we are able to demonstrate that camera and microphone are adequate sensors for real-time weld defect detection. Our unsupervised approach ensures that the model is not biased to defect characteristics which are specific for our data collection setup and allows us to compare results between different defect types and and modalities. Finally, we demonstrate that defect detection performance can be improved using a multi-modal combination of both sensor types.

In the next sections we provide a comprehensive description of how the dataset has been collected. We introduce our experimental setup, quality metrics and ML models. The performance of unsupervised algorithms trained in single and multi-modal fashion on this dataset is evaluated experimentally and we present our conclusions.
 
*



§ DATA SET


The goal of our data collection was to record enough samples for each of the most important weld categories to make statistically valid comparisons across modalities and weld defect types. We collaborated with a supplier that has access to automotive factories to conduct the data acquisition in a real factory environment. The welds were generated using a 6-axis arc welding robot using two steel types and thicknesses. The steel types were selected to be often-used varieties for automotive applications in India. The first type is known as “FE410” and the second “BSK46” type has higher carbon content and is used for higher-strength applications. The thicknesses used were 7mm for most of the samples and 3mm for specific defects that could not be efficiently induced using the 7mm steel, like burnthrough.



 §.§ Recording Setup, Data Collection Procedure, and Limitations


The data collection station comprised an AII-V6 6-axis arc welding robot <cit.>, an Intel i7-based workstation and camera <cit.>, two high-bandwidth microphones <cit.>, and an audio interface <cit.>. The microphones were selected to enable studies on the observability of weld defects at higher frequencies. Moreover, audio samples were recorded using a 192 KHz sampling rate and saved in lossless FLAC format <cit.>. The video samples were recorded at a nominal 30 FPS and saved in AVI format. Fig. <ref> shows the KML camera and computer. The camera was attached to the welder arm, about 200mm from the torch. The microphones were attached to the work table, about 300mm from the welder arm motion axis.







To generate weld defects, the supplier contracted a welding expert who supervised the initial configuration of the welding robot to produce the desired weld defects. The welding expert did not perform any post-weld validation or labeling. Thus, the dataset contains some amount of label noise. Moreover, all sensors were triggered to start recording at the same time, but there is some variation in the actual time it takes for the individual device to respond. To align the different modalities which is required for multi-modal experiments, the weld start and end times were identified for each modality by inspecting the audio and illumination changes at the start and end of welding. The audio data, which was collected through a dedicated audio interface with a low-latency driver <cit.> was taken as the ground truth source for determining the welding duration. In this way, it was determined that the actual recorded frames-per-second (FPS) of the video varies from the expected 30 FPS. It is worth mentioning that this post-collection alignment would not be needed if the robot command signals were readily accessible.




 §.§ Welding Sample Distribution

The dataset contains weld samples for the 12 weld categories shown in Tab. <ref>. Each weld category has been recorded for different weld types, and, where applicable, for different materials. Tab. <ref> contains a brief summary of the dataset by weld type. A complete breakdown of the dataset is included in  Appendix <ref>. Fig. <ref> in Appendix <ref> shows some examples of (post-weld) photos of welding samples.






§ EXPERIMENTAL SETUP

The experiments in this work focus on the following questions: (1) Can weld defects be reliably detected in real-time from audio and/or video recordings? (2) How does the detection accuracy depend on the defect type and modality? (3) What improvements can be expected from combining audio and video in a multi-modal system?

To answer those questions, we treat the weld defect problem as an anomaly detection problem, not a classification problem. That means, our models do not attempt to classify a sample into a weld category, but generate an anomaly score that increases with the likelihood of a defect. For a real use case a threshold has to be determined in advance. If the score of the model exceeds this threshold during welding, a defect will be detected and the supervisor of the robot will be notified. The choice of the threshold depends on the false positive and false negative rates of the model for the defect types that are relevant for a specific use case, the costs associated with a false detection of a defect, and the costs resulting from a missed defect. As we would like to evaluate the quality of our models independently of an application-specific threshold, we compare the models using the Area-under-ROC-Curve (AUC) metric which is scale- and threshold-independent <cit.>. AUC is defined as the area under the Receiver Operating Characteristic (ROC) curve for all possible values of the false positive rate (FPR):

    AUC = ∫_x=0^1 ROC(x) dx

where x is the FPR and ROC(x) is the true positive rate (TPR) <cit.>. 

When calculating FPR and TPR for a specific threshold on the validation and test sets, we have to take into account that we are targeting a real-time scenario, where the model produces an anomaly score at every time frame. On the other hand, each sample in our data set has just a single label denoting the weld category. For instance, a sample could be labeled as “porosity” but there is no indication at which time instance the defect occurs. Therefore, we aggregate all scores produced by the model for all frames of the sample by either taking their maximum or expected value. 


In this work we employ simple but effective convolutional neural networks (CNN) for weld defect detection. This allows us to focus more on gaining insights and less on conducting large hyper-parameter optimizations and architecture searches. For audio, we used a 1D CNN auto-encoder <cit.>. For video, we applied the relatively simple 3D CNN from <cit.> provided by the MMAction2 library <cit.>[The  “slowfast_r101_4x16x1_256e_kinetics400_rgb_20210218-d8b58813.pth” checkpoint from v0.15.0 was used <cit.>.]. All our work used the PyTorch tensor library for Python <cit.>.


For the experiments, the dataset is split into a training, validation, and test partition. The training data includes only good, i.e., normal-state, samples, while the defects are divided equally among the validation and test subsets. 

The validation partition of the dataset is used for hyperparameter tuning. The best hyperparameter configuration on the validation partition is used to generate the test results from the test partition. The data split is summarized in Tab. <ref>. The same partitions were used for all experiments with both modalities.







 §.§ Acoustic Anomaly Detection

For the audio experiments, the original sampling rate of 192 kHz was maintained and only one channel was used. The audio CNN auto-encoder architecture is shown in Table <ref>. A detailed description of the model's topology can be found in <cit.>: the key characteristic is that the bottleneck layer largely preserves the time dimension. This feature has been motivated by the work of Agrawal et al. <cit.>. The model in Tab. <ref> is much smaller than the one described in <cit.> because the convolutions are not gated.


 The model uses leaky-ReLU activations in all but the last activation before the output layer, which uses a PReLU activation. Overall, the model has 31,670,306 trainable parameters.

 The latency of this model equals the hop length used to train the model. For example, for a hop length of 8192 and 192 kHz audio, the latency is about 42.7 ms. The model has 5 encoding layers with kernel size 3. Each encoding layer decreases the time dimension by 2, therefore inputs must have more than 10 frames. This means the input buffer must satisfy Eq. <ref>, with the FFT window an integer multiple of the hop length. In the foregoing example, the input buffer needs to hold 12 × 8192 samples.

 
    buffer size = hop length×( 10 + FFT window/hop length)





 §.§ Visual Anomaly Detection


0 




Our approach to visual weld defect detection is based on a two-stage process. The first stage encodes each video frame into a fixed dimensional feature vector. For this, a window that is 64 frames long is shifted frame-by-frame over the whole video. 64 frames correspond to roughly 2 seconds at a frame rate of 30 FPS. As the window is centered around the frame of interest, the 64-frame window size leads to defect detection latency of around one second in a real-time scenario. This is a larger latency than for the acoustic anomaly detection, which is based on much smaller windows as will be described in Sec. <ref>. Still, we believe that it should be acceptable for many use cases. Each window is encoded into a 2304-dimensional feature vector using the pre-trained and fixed Slowfast <cit.> model. 
In the second stage, an auto-encoder model consisting of an encoder, a bottleneck, and a decoder is used for generating anomaly scores from the input feature vector. The encoder maps the 2304-dimensional input feature vector to 64-dimensional latent space by passing it through multiple linear layers along with ReLU activation and dropout in sequence. Each linear layer reduces the dimension by 1/2, thus creating a bottleneck with a 64-dimensional latent vector. Later, the decoder uses the latent space embedding to reconstruct the original feature of the frame. The decoder consists of multiple linear layers with ReLU activation and dropout as well. Each linear layer of the decoder scales the dimension by 2 until the original 2304 dimension is obtained. All dropout layers zero weights with probability 0.5.  The architecture details of the auto-encoder are provided in Tab. <ref>. 

The auto-encoder model is trained on welding videos minimizing the model's anomaly score which is defined as the mean-squared-error (MSE) between the input feature vector and the output of the decoder. Note, that the Slowfast model from the first stage is fixed, and no-back propagation is applied to the first stage during training.





 §.§ Multi-modal Anomaly Detection
 
We used a late-fusion approach to combine the anomaly scores of the two modalities. Since anomaly scores generated by different models generally have different scales, we first standardized the scores using the mean and standard deviation of the anomaly scores over the training set. Next, we identified the optimal convex combination of the audio and video anomaly scores by running grid search over convex combinations on the validation data. The best weighting is applied to compute the anomaly scores on the test set and to produce the final results.




§ RESULTS




 §.§ Acoustic Anomaly Detection

For hyperparameter tuning we trained separate auto-encoder models with different parameter settings on the training partition and evaluated their AUC on the validation set. More specifically, a grid search was performed over FFT window sizes 4096, 16384, 32768, and 65536 and bottleneck dimensions 16, 32, 48, and 64. The hop length was fixed at 50% of the FFT window size. We remind the reader that for a given analysis (time) window, the corresponding FFT window length is proportional to the sampling rate. Therefore, the FFT window sizes included in the grid search range from approximately 21 ms to 341 ms for the sampling rate in our data set, which is 192 kHz.  The latency of these models range from approximately 11ms to 171ms. All models were trained for 50 epochs using a one-cycle learning schedule with a peak learning rate of 1× 10^-4 for the Adam optimizer <cit.>. The MSE loss was minimized during training.

Tab. <ref> shows the FFT window search results. These experiments determine that an FFT window of 16384 with a bottleneck dimension of 48 produced the best performance on the validation set. To obtain the AUC scores we explored several frame-wise anomaly score aggregation methods: expected value, moving average (MA) smoothing, and taking the max. We found taking the average worked best, followed closely by MA smoothing, and taking taking the max was last. We only show the expected value scores for space considerations.











On the test set, the best validation model obtained an AUC of 0.8460. This is not much different from the AUC on the validation set which indicates that the model does not overfit.
To gain insights into the difficulty of detecting particular defects, we also determined the validation and test AUC by defect type – the results are shown in Tab. <ref> and Tab. <ref>, respectively. Again, the results indicate a high agreement between validation and test set AUCs. Note that these results are not directly actionable because in most use cases the defect type is not known a priori. Breaking-out performance by defect type is, however, useful for estimating performance gains that could be obtained when fusing audio with video information.

In cases where an error (or error cost) distribution is known a priori, one may select the optimal hyperparameters differently. Additionally, latency or memory requirements can affect the FFT window and hop length selection as well.



 §.§ Visual Anomaly Detection


For visual defect detection, we first generate feature vectors of dimension 2304 using the pre-trained Slowfast model for each sample in all data partitions. In a second step we train the auto-encoder model using the Adam optimizer with a learning rate of 0.0005 and MSE loss function for up to 1000 epochs on the feature vectors. For the model that performs best on the validation set, we compare different methods to aggregate the scores:

simply taking the maximum score (“Max w/o smoothing”), smoothing the scores by averaging them within a one-second window (“Max over 1s-MA”), or smoothing the scores by averaging them within the full two-second window (“Max over 2s-MA”). Table <ref> shows the AUC for different aggregation methods on the test set. The best validation model obtains an overall AUC of 0.9052 and 0.8977 on the validation and test data, respectively.







 §.§ Multi-modal Anomaly Detection

 The input scores for the late fusion combination of the video and audio modalities were created using the best performing models for each modality, i.e., the audio model using FFT window size 16384 and bottleneck dimension of 48, and the video model with 2s-MA smoothing. We determined the weighting for the modalities on the validation data using the grid search described in Sec. <ref> with step size 0.01.

This way, we found the best weighting to be 0.37 and 0.63 for the audio and video scores, respectively. These weightings make sense given the stronger performance of the video modality.

Using the optimal weighting we obtained our overall best test AUC of 0.9178. From Tab. <ref>, it can be seen that the overall audio validation and test AUCs improved by 9.8% and 8.5%, respectively, and the overall video validation and test AUCs improved by 2.5% and 2.2%, respectively. The defect-specific scores  improved as well. For audio, the AUC metric improved for 8 of 11 defect categories. For video, the AUC improved for 7 of 11 categories. It is worth mentioning that the weakest performing defect categories all improved. In particular, audio had four defect categories with AUCs in the 0.7s and all benefited from fusing. For video, two defect categories had AUCs in the 0.7s and both benefited from fusing.  See the entries of Tab. <ref> shown in bold.

Fig. <ref> shows the detection error curves for the multi-modal predictions, on the test data. As can be seen, the FPR and false-negative-rate (FNR) intersect at about 17%.










§ CONCLUSIONS AND FUTURE WORK

In this work, we explored unsupervised weld defect detection using audio, video, and their combination. We demonstrated that, using a deep-learning based approach, both modalities allow to reliably detect the most important defect types in real-time. The best audio model had a latency of 42.7ms and is best suited for detecting porosity, overlap, and crater cracks. It has the lowest AUC scores for burnthrough, excessive penetration, spatter, and warping. Video generally shows better AUC than audio, but this comes at the price of a larger overall model size and a higher latency. Using video, the best detection performance is achieved for porosity, excessive convexity, and spatter. We observe the lowest performance for lack of fusion, and warping. We demonstrated that a combined approach using late fusion of normalized scores for both modalities offers improvements. More specifically, AUC scores for all defect types average 0.92, with the lowest score approximately 0.80. The worst performing categories for both modalities all improved. 

Future work will investigate more elaborate ways to combine the two modalities, e.g., a joint model that directly incorporates input from all sensors. Furthermore, we believe that the biggest limitation of our dataset is that it has been collected in a supervised way, i.e., the robot has been configured by purpose to generate defects as we required a sufficient number of defect samples for our experiments. This approach creates a potential mismatch to a real use case where defects occur randomly and rarely. We expect that defects will show up  less pronounced than in our dataset. Therefore, we plan to record a real production process of a collaboration partner and to label the defects as they occur. This will allow us to validate our proposed methods in a more realistic setting.



§ COMPLIANCE WITH ETHICAL STANDARDS

This research was funded by the Intel Corporation. We are unaware of any conflicts of interest. The welding data used in this work does not contain information that could be used to identify an individual. All work was done in a manner consistent with expected ethical standards.





§ SAMPLE DISTRIBUTIONS


This appendix contains a complete dataset description by defect category: Tab. <ref> shows the distribution of the samples for the normal-state, i.e., good weld category, by weld type and material. Tab. <ref> to <ref> show the sample distribution for the remaining weld categories.




       






















§ SAMPLE IMAGES

This Appendix shows in Fig. <ref> examples of photos taken of samples after the welding has been completed.






























