Automated dermatoscopic pattern discovery by clustering neural network
  output for human-computer interaction
Background: As available medical image datasets increase in size, it becomes
infeasible for clinicians to review content manually for knowledge extraction.
The objective of this study was to create an automated clustering resulting in
human-interpretable pattern discovery.
  Methods: Images from the public HAM10000 dataset, including 7 common
pigmented skin lesion diagnoses, were tiled into 29420 tiles and clustered via
k-means using neural network-extracted image features. The final number of
clusters per diagnosis was chosen by either the elbow method or a compactness
metric balancing intra-lesion variance and cluster numbers. The amount of
resulting non-informative clusters, defined as those containing less than six
image tiles, was compared between the two methods.
  Results: Applying k-means, the optimal elbow cutoff resulted in a mean of
24.7 (95%-CI: 16.4-33) clusters for every included diagnosis, including 14.9%
(95% CI: 0.8-29.0) non-informative clusters. The optimal cutoff, as estimated
by the compactness metric, resulted in significantly fewer clusters (13.4;
95%-CI 11.8-15.1; p=0.03) and less non-informative ones (7.5%; 95% CI: 0-19.5;
p=0.017). The majority of clusters (93.6%) from the compactness metric could be
manually mapped to previously described dermatoscopic diagnostic patterns.
  Conclusions: Automatically constraining unsupervised clustering can produce
an automated extraction of diagnostically relevant and human-interpretable
clusters of visual patterns from a large image dataset.
Lidia Talavera-Martinez, Philipp Tschandl
Automated dermatoscopic pattern discovery by clustering neural network output for human-computer interaction

###### Abstract

Background: As available medical image datasets increase in size, it becomes infeasible for clinicians to review content manually for knowledge extraction. The objective of this study was to create an automated clustering resulting in human-interpretable pattern discovery.

Methods: Images from the public HAM10000 dataset, including 7 common pigmented skin lesion diagnoses, were tiled into 29420 tiles and clustered via k-means using neural network-extracted image features. The final number of clusters per diagnosis was chosen by either the elbow method or a compactness metric balancing intra-lesion variance and cluster numbers. The amount of resulting non-informative clusters, defined as those containing less than six image tiles, was compared between the two methods.

Results: Applying k-means, the optimal elbow cutoff resulted in a mean of 24.7 (95%-CI: 16.4-33) clusters for every included diagnosis, including 14.9% (95% CI: 0.8-29.0) non-informative clusters. The optimal cutoff, as estimated by the compactness metric, resulted in significantly fewer clusters (13.4; 95%-CI 11.8-15.1; p=0.03) and less non-informative ones (7.5%; 95% CI: 0-19.5; p=0.017). The majority of clusters (93.6%) from the compactness metric could be manually mapped to previously described dermatoscopic diagnostic patterns.

Conclusions: Automatically constraining unsupervised clustering can produce an automated extraction of diagnostically relevant and human-interpretable clusters of visual patterns from a large image dataset.

Pre-peer review version 1

Footnote 1: This is the pre-peer reviewed version of the following article: _Talavera-Martinez L, Tschandl P. Automated dermatoscopic pattern discovery by clustering neural network output for human-computer interaction. J Eur Acad Dermond Vennerol. 2023_, which has been published in final form at [https://doi.org/10.1111/jdv.19234](https://doi.org/10.1111/jdv.19234). This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions.

## I Introduction

In dermatology, but also other visual medical fields, the recognition and description of specific samples of diseases is important for a precise diagnosis and for the formulation of differential diagnoses. Apart from clinical dermatology, there have been a plethora of pattern-descriptions of symptoms of disease in dermatoscopy in recent decades [1], especially for the diagnosis of skin tumors and inflammatory diseases [2], which are used for teaching and diagnosis in daily practice. These descriptions were mostly based on mono- or multicentric case collections that were reviewed manually by a few authors and evaluated on possible repetitions of patterns [3, 4, 5]. As clinical image data collections are increasing in size [6, 7], entirely manual review for discovering diagnostic patterns is not a realistic scenario anymore. In addition to an insurmountable workload, interrater disagreement may be a hindering factor in identifying and describing objective, valid and teachable pattern groups [1].

Increasingly, neural networks - especially convolutional neural networks (CNN) - are described as an aid for diagnostic classification of medical images. In the field of dermatology, CNNs were described to have at least an equal accuracy as dermatologists in experimental settings for classifying clinical and dermatoscopic images, and shown to improve physicians' diagnostic accuracy when applied in diverse interactive settings [8, 9]. Such algorithms can not only classify images but also label anatomic areas [10], rate psoriasis [11], or retrieve similar images to a case, by implicitly analyzing patterns and pattern combinations after training to categorize images into distinct classes [12]. Therefore, we hypothesize that convolutional neural networks could be helpful in the extraction of diagnostically relevant patterns in medical image collections. Recent reports have shown unsupervised techniques when having limited label data [13].

The goal of this study was to create an automated workflow to extract diagnostic relevant pattern candidates for review by doctors and researchers, with dermatoscopic images of skin tumors as an example (Fig. 1). Eventually, from a big dataset with thousands of images, this should enable human-computer interaction and return an interpretable number of visually distinct patterns, by obtaining as few redundant or uninformative patterns as possible.

The approach we propose is a pipeline based on machine learning that consists of extracting deep features from CNNs, and applying an unsupervised clustering algorithm to these features. The clustering shall be constrained by a custom compactness metric that, in contrast to the well known elbow-metric, should better balance retrieval of all relevant patterns while at the same time keeping redundant information low.

## II Materials and Methods

### _Data and processing_

This non-interventional retrospective study was conducted on public image data only, specifically the HAM10000 dataset [14]. This dataset is composed of 10015 dermatoscopic images
of pigmented lesions with annotations on both the diagnosis and segmentation of the lesion area [8]. To focus on patterns rather than full images, analyses were performed on a tile-level. We extracted square subregions (tiles) of an image by a sliding window with a size of 128x128 pixels with 25% overlap, discarding tiles with \(<60\%\) lesion area. In sum, 29420 tiles were extracted, Suppl. Fig. S1 shows two example cases with resulting extracted tiles, and Suppl. Table S1 the number of tiles per diagnosis. To ensure approximately equal representation of diagnoses, included nevi were limited to a random subsample of 1100 cases, and resulting tiles limited to a maximum random subsample of 850 tiles. To reduce the influence of changes in illumination color, we applied color constancy correction [15] to all tiles (Suppl. Fig. S2).

### _Neural network and feature extraction_

A VGG16 [16] architecture, pretrained on ImageNet data, was fine-tuned to classify tiles into one of seven diagnoses included in the HAM10000 dataset. Training was performed with all 29420 tiles, using 70% for training and the remaining 30% for validation during a single training run, ensuring no overlap of tiles of the same image between sets. This training run was only performed as a means to parameterize the model, thus as knowledge discovery rather than classification accuracy was the goal, the complete training dataset was also used for cluster analyses downstream. Data augmentation steps were flips in both horizontal and vertical directions, random 90\({}^{\circ}\) rotations, and zooms. Training was performed with a batch size of 32, using the Adam [17] optimizer, a weighted categorical cross entropy loss, with an initial learning rate of 1-e5, and an early stopping policy based on validation loss. For extracting features from image tiles by the fine-tuned model, the numerical state of the layer before the classification layer was obtained, resulting in a 1280-length vector. Neural network experiments were conducted using tensorflow [18] and python 3.8. Experiments were repeated with EfficientNet-B0 [19] and a convolutional autoencoder, with results for those two models shown in the supplementary data. For the autoencoder, we trained the model from scratch with a mean-squared error loss, and extracted the features from the flattened embedding space.

### _Clustering_

Resulting extracted features are normalized and used as input to an unsupervised clustering algorithm, specifically k-means [20] with cosine distance as a distance metric. This calculation was performed using scikit-learn v1.1.2 [21] and scipy v1.9.0 [22]. To automatically obtain the optimal number of clusters without further intervention from a user, either the elbow method (optimal value as calculated by yellowbrick v1.5 [23]) or a custom compactness metric (W) was applied. The latter method is based on the assumption that each lesion, thus also the tiles that comprise it, on average only show one or two dermatoscopic patterns. Thus, the proposed metric measures both the similarity of the clusters to which tiles of the same image have been assigned, and the number of different clusters the tiles were assigned to in respect to the total number of clusters. The metric was implemented as follows:

\[I=\{img_{1},...,img_{q}\}\]

\[T_{q}=\{t_{1},...,t_{j}\}\]

\[C_{q}=\{c_{1},...,c_{i}\}\]

\[W=\frac{1}{M}\times\sum_{q=1}^{M}\left(\frac{K}{min(n_{clst},L)}\times\sum_{L }^{j=1}cosDst\left(\frac{1}{K}\sum_{i=1}^{K}(c_{i}),t_{j}\right)\right)\]

where \(M\) is the number of images \(I\) in the experiment, \(T_{q}\) are the \(L\) tiles of an image \(img_{q}\), \(K\) is the number of unique clusters to which \(T_{q}\) belong, \(n_{clst}\) are the total number of clusters used in the experiment. Reiterating, the first half of W for an image ensures tiles are spread to as few clusters as possible, and the second half of W ensures the distance of tiles to the common center of clusters, that tiles are assigned to, is low.

### _Classification ability_

To assess classification ability of the two clustering cutoff methods, clusters were created not only for each diagnosis separately, but also for the whole dataset spanning all diagnoses. The frequency of diagnoses contained in a resulting cluster was noted as a multi-class probability for a classification task. Test images from the ISIC 2018 challenge Task 3 [24, 25] were tiled and preprocessed as above (resulting in 10254 tiles from 1304 lesions with sufficient lesion area depicted), and probabilities of the closest cluster of each tile averaged. The top-1 class of the resulting probabilities was taken as a prediction, and accuracy as well as mean recall [24] calculated.

Fig. 1: Processing overview - Dermatoscopic images (upper left) are used as source data, and a neural network is trained for classification on tiled lesion-area tiles. Features are extracted from lesion tiles with this trained network, on which an unsupervised k-means clustering is applied to find pattern groups. Up to the closest 7 lesion-tiles within a cluster (examples shown for clusters within the BCC class) are stored as representatives of a pattern and presented to a human reader for qualitative interpretation.
### _Manual pattern descriptions_

Top-7 tiles of clusters, created for every diagnosis in the dataset separately with VGG16 feature vectors, k-means and the described compactness metric, were inspected by a dermatologist with substantial experience in dermatoscopy (PT). Patterns were scored for redundancy, i.e. showing the same pattern as another cluster of the diagnosis, informativeness, i.e. whether any reproducible pattern can be identified, number of patterns, and previous description, i.e. whether the pattern was already identified and described in the literature. A pattern was defined as a change in color and/or structure covering the majority of the image tile.

### _Statistics_

Differences of paired values were compared using a one-sample t-test after checking for normality assumptions. Statistical analyses were performed using R Statistics v4.1.0 [26], and plots created with ggplot2 [27]. A two sided p-value \(<.05\) was regarded as statistically significant, with a Bonferroni-Holm type correction applied.

## III Results

### _Pattern interpretability_

Applied on clusters of every diagnosis separately, the elbow method created a mean of 24.7 (95%-CI: 16.4-33) clusters for a diagnosis, whereas the compactness metric resulted in significantly less clusters (13.4; 95%-CI 11.8-15.1; p=0.03; Fig. 1(a)). The proportion of uninformative clusters was higher when using the elbow method (14.9%; 95% CI: 0.8-29.0) than using the compactness metric (7.5%, 95% CI: 0-19.5; p=0.017; Fig. 1(b)).

Qualitative interpretation of the clusters resulting from the compactness metric, for 93.6% (88 of 94) of diagnosis-specific clusters, at least one recognizable consistent pattern could be identified by a dermatologist. Identified patterns could be mapped to 53 unique known diagnostic descriptions from previous literature of at least 29 publications (Suppl. Table S3). Only 51 clusters could be described with one pattern alone, whereas 30 clusters encompassed two, and 7 clusters three recognizable patterns in combination. The proportion of redundant clusters within a diagnosis ranged from 0% (basal cell carcinoma and melanoma) to 27.3% (dermatofibroma and vascular lesions).

### _Retained classification performance_

Application of clustering on the whole dataset with all diagnoses included, using the elbow method resulted in a higher number of clusters than the compactness metric (42 vs. 7), as well as a higher mean recall (46.3 vs. 34.6) and accuracy (43.4%; 95%-CI 40.7-46.2 vs. 32.2%; 95%-CI 29.7-34.8) for predictions on the ISIC2018 test set. Clusters of the compactness metric were rarely able to predict actinic keratoses, and almost never dermatofibroma (Fig. 3).

## IV Discussion

With ever growing image datasets, human interpretation of available data becomes increasingly difficult, and herein we present an automated analysis pipeline which via automated image processing is supposed to aid human-computer interaction for diagnostic marker discovery. Providing only information on the diagnosis and image area, we were able to showcase that the presented workflow is able to reproduce a major fraction of diagnostic patterns in dermatoscopy described in the literature.

In contrast to other publications [6, 24, 28] trying to optimize for the best diagnostic accuracy of a neural network model, herein we propose a metric to constrain k-means clustering to optimize for human interpretability in a truly interactive human-computer interaction workflow. The proposed compactness metric reduces the information to a digestible amount, shown by the significant reduction in overall clusters (Fig 1(a)), alongside a reduction of non-informative information shown by the significant reduction of noninformative clusters (Fig 1(b)). These improvements, though come at a cost, namely a reduced diagnostic accuracy when applied in an automated classification setting. This underlines that the training proposed herein could be useful for human-computer interaction and interpretability, but not for safely predicting diagnoses as a standalone application. As biases from automated predictions of image data through neural networks is a significant problem [29], datasets should be inspected for potential biases. Although not explicitly shown in this pilot experiment, the proposed workflow may enable medical personnel and researchers to identify highly prevalent biases in a qualitative manner. It is certainly not a complete

Fig. 3: Confusion matrices showcasing performance of predictions via averaged nearest neighbor cluster-probabilities constrained by (a) compactness metric (7 clusters), or (b) the elbow method (42 clusters). Values within cells show proportions within one ground-truth class (=row).

Fig. 2: Number of overall (a) and uninformative (b) clusters per diagnosis when constraining cluster numbers via either the compactness or elbow method.
solution, as based on the failure to classify rare classes (Fig 3a) we hypothesize that biases on rare classes will equally not be detectable.

Through qualitative analysis of resulting clusters we found that for most it is not possible to find a single pattern to describe them, but the majority needed at least a combination of two patterns (Supp. Table S3). This finding may help in designing future annotation and pattern analysis studies, and we hypothesize that studies trying to annotate and analyze for a single structure may not be representing real patterns. Interestingly, this may be a missing link between descriptive and "metaphoric" language [1], as the former is more suitable for distinct and concise descriptions, but metaphoric language inherently tries to capture structure combinations. A further interesting insight was that the frequency of redundant clusters was not equally distributed, but higher in dermatofibroma and vascular lesions. This could be sourced by the fact that these diagnoses in general show less variability in their patterns, but also that the used dataset through small sample size for these diagnoses is not covering the real visual variability. Finally, it is also interesting to note that by qualitatively comparing different network architectures (Supp. Fig. S4 - S10), one can identify different utility of them for the purpose of pattern discovery. While an Autoencoder is mainly detecting color blobs, edges, corners and curves, it is focussing less on detailed structures. The top-7 tiles from clusters created by using EfficientNetB0, as a representative of a modern architecture with higher diagnostic accuracy than VGG16 [19], were less homogeneous and thus harder to interpret. Thus, despite being not ideal for classification, we hypothesize that VGG16, through its inner architecture, is a well fit for extracting features of interpretable mid-level patterns useful for human-computer interaction. Future studies should show feasibility of implementation of this workflow not only for present dermatoscopic datasets [30], but also other imaging modalities such as dermatopathology and clinical images.

### _Limitations_

This pilot study was supposed to showcase the general feasibility of the proposed process. Applicability to nonpigmented tumors, other localisations, inflammatory cases and darker skin types cannot be estimated as those were not included in the source datasets. The process at its core is analyzing substructures of dermatoscopic images, thus overall architecture is not integrated but could theoretically be overcome by changing the tile size and minimal lesion area. The latter is a relevant consideration when applying the workflow, as with the initially chosen tile size and lesion area constraints, some test cases with a very small lesion depicted did not produce any tile.

## Acknowledgements

Lidia Talavera-Martinez was a beneficiary of the scholarship BES-2017-081264 granted by the Ministry of Economy, Industry, and Competitiveness of Spain under a program co-financed by the European Social Fund. She is also part of the R&D&i Project PID2020-113870GB-I00, funded by MCIN/AEI/10.13039/50110 0011033/.

## Data availability

Used image data are openly available at [https://doi.org/10.7910/DVN/DBW86T](https://doi.org/10.7910/DVN/DBW86T) (Harvard Dataverse). Resulting Clusters and qualitative evaluations are available in the supplementary material of this article.
