Continuous variables were described as median and interquartile range (IQR). Categorical variables were presented as number (percentage). The diagnostic performance of the main RSSs was expressed through predictivity tests (i.e., sensitivity, specificity, positive (PPV) and negative (NPV) predictive value, and accuracy, with specific 95% confidence intervals), which were calculated according to Galen and Gambino [33], and the unnecessary FNA prevalence, defined as the number of benign nodules among the FNA-required nodules. Specifically, we employed assessments of malignant versus benign nodules in order to be able to report the estimates of accuracy on a lesion basis.

The interobserver agreement was evaluated by Cohen's kappa statistic, where the kappa value (*k*) denotes the strength of agreement and is interpreted as follows: 0–0.2, poor; 0.2–0.4, fair; 0.4–0.6, moderate; 0.6–0.8, good; 0.8–1.0 very good. Statistical significance was defined as a *p*-value < 0.05. Statistical analysis was performed by MedCalc software version 9 (Mariakerke, Belgium). The interobserver agreement was evaluated by Cohen's kappa statistic, where the – – – – –

#### **3. Results**

There were 81 thyroid nodules in 71 patients undergoing both nUS and FNA in the initial database. After applying our exclusion criteria, in our study, we finally included 41 nodules in 36 patients (Figure 1). Twenty-six patients were female (72.2%), and ten patients were male (27.8%). Median age was 15 years (11–17 years), with the final cohort including 12 prepubertal and 24 postpubertal patients. The nUS indication was the following: autoimmune chronic thyroiditis (±hypothyroidism) in 17 patients (47.2%); excluding thyroid disease in nine patients (25.0%); palpable thyroid nodules (±goiter) in six patients (16.7%); Graves' hyperthyroidism in four patients (11.1%). Among the 36 patients, 28 (77.8%) had a solitary thyroid nodule, and eight patients (22.2%) had multiple thyroid nodules. The median nodule' s maximal dimension was 13 mm (10–16 mm). Of the 41 thyroid nodules, 29 (70.7%) were benign (of which six (20.7%) underwent surgery) and 12 (29.3%) were malignant. Serum calcitonin was negative in all cases. – 7%); Graves' hyperthyroidism in four patients (11.1%). Among the 36 thyroid nodules. The median nodule' s maximal dimension was 13 mm (1 –

**Figure 1.** Flowchart of patient selection. nUS, neck ultrasound; FNA, fine-needle aspiration.

Most cancers were papillary carcinoma (10 (83.3%) of 12, nine conventional variants, including one multifocal and one follicular variant), followed by follicular carcinoma (two (16.7%) of 12). Median maximal dimension of malignant thyroid nodules was 10 mm (7–13). The characteristics of our patients are shown in Table 1.

**Table 1.** Main characteristics of our patients (*n* = 36).

IQR, interquartile range; nUS, neck ultrasound; mm, millimeter; PTC, papillary thyroid cancer; FTC, follicular thyroid cancer.

The distribution of thyroid nodules according to the ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS risk levels is summarized in Table 2. The highest number of nodules (16 of 41 nodules) fell into the intermediate-risk category (i.e., TR4, EU-TIRADS 4, K-TIRADS 4, intermediate suspicion). A 100% cancer prevalence was observed in the high-risk class (i.e., TR5, EU-TIRADS 5, K-TIRADS 5, high suspicion). While 6/12 (50%) of cancers were assessed by the highest-risk category (i.e., TR5, EU-TIRADS 5, K-TIRADS 5, high suspicion), the remaining half were classified as at low or intermediate risk in all US RSSs.

Table 3 shows the recommended management of nodules in this cohort based on the ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS criteria. Among 29 benign nodules, according to ACR TI-RADS and EU-TIRADS criteria, seven (24.1%) would have undergone FNA, while 22 (75.9%) would have been followed up without FNA. Instead, among 29 benign nodules, according to K-TIRADS and ATA US RSS criteria, 19 (65.5%) would have undergone FNA, while 10 (34.5%) would have been followed up without FNA. According to the ACR TI-RADS and EU-TIRADS criteria, five (41.7%) of the 12 malignant nodules would have undergone FNA, and seven (58.3%) would have been assigned follow-up without FNA. According to the K-TIRADS and ATA US RSS criteria, six (50%) of the 12 malignant nodules would have undergone FNA, and six (50%) would have been assigned follow-up without FNA. The unnecessary FNA prevalence for ACR TI-RADS and EU-TIRADS was 58.3%, while that for K-TIRADS and ATA US RSS was 76%.

Table 4 resumes the reliability of the four RRSs in correctly indicating FNA. Specifically, for ACR TI-RADS and EU-TIRADS, we found the following results: sensitivity 41.7%, specificity 75.9%, PPV 41.7%, NPV 75.9%, and accuracy 65.9%. Instead, for K-TIRADS and ATA US RSS, we found the following results: sensitivity 50%, specificity 34.5%, PPV 24%, NPV 62.5%, and accuracy 39%. The interobserver agreement in classifying nodules according to ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS was good with *k*-values of 0.7, 0.61, 0.66, and 0.62, respectively (*p* ≤ 0.002 in all cases).

**Table 2.** Distribution of 41 thyroid nodules according to the ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS risk levels in 36 patients in our cohort.

TIRADS, Thyroid Imaging Reporting and Data System; US RSS, ultrasound-based risk stratification system; ACR, American College of Radiology; EU, European; K, Korean; ATA, American Thyroid Association.

**Table 3.** Management of 41 thyroid nodules according to the the ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS criteria in 36 patients in our cohort.

TIRADS, Thyroid Imaging Reporting and Data System; US RSS, ultrasound-based risk stratification system; ACR, American College of Radiology; EU, European; K, Korean; ATA, American Thyroid Association; FNA, fine-needle aspiration. The unnecessary FNA prevalence for the diagnosis of thyroid cancer was defined as the number of benign nodules among the FNA-required nodules.

**Table 4.** Reliability of the ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS in correctly indicating FNA in 41 nodules of 36 patients in our cohort.

TIRADS, Thyroid Imaging Reporting and Data System; US RSS, ultrasound-based risk stratification system; ACR, American College of Radiology; EU, European; K, Korean; ATA, American Thyroid Association; FNA, fine-needle aspiration; PPV, positive predictive value; NPV, negative predictive value; CI, 95% confidence interval.

The clinical and US characteristics of malignant nodules that would not have undergone FNA according to the RRSs are presented in Table 5. Six nodules (ID 1, 2, 4, 5, 6, and 7) would not have been identified using the four RSSs at initial visit, while one nodule (ID 3) would have undergone FNA according to K-TIRADS and ATA US RSS but not according to ACR-TIRADS and EU-TIRADS criteria. The median maximal dimension of these seven malignant nodules was 10 mm (7–12 mm). Five of the seven nodules were solitary. These were papillary carcinoma in five cases (four with conventional variant, including one multifocal and one follicular variant) and follicular carcinoma in two cases.

**Table 5.**Clinical and US characteristics of proven malignancies not identified with the RSSs (ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS).

TIRADS, Thyroid Imaging Reporting and Data System; US RSS, ultrasound-based risk stratification system; ACR, American College of Radiology; EU, European; K, Korean; ATA, American Thyroid Association; FNA, fine-needle aspiration; mCPTC, multifocal conventional papillary thyroid cancer; CPTC, conventional papillary thyroid cancer; ACT, autoimmune chronic thyroiditis; follicular variant of papillary thyroid cancer; GD, Graves' disease; FTC, follicular thyroid cancer.

#### **4. Discussion**

#### *4.1. Principal Findings*

In our final pediatric cohort, we found a malignancy rate (nearly 30%) similar to that reported in previous studies of children and higher than that associated with thyroid nodules in adults [1,6,7]. The risk of malignancy was highest for the high-risk levels of all four RSSs (i.e., TR5, EU-TIRADS 5, K-TIRADS 5, high suspiscion) with 100% concordance between the US-based high-risk level and cancer. This finding was roughly in line with the adult-based estimated risk of malignancy reported in the four RSSs (i.e., >20% ACR-TIRADS, 26–87% EU-TIRADS, >60% K-TIRADS, and >20% ATA US RSS) [26–29]. Likewise, regarding the intermediate-risk level (i.e., TR4, EU-TIRADS 4, K-TIRADS 4, intermediate suspiscion), we found that the malignancy rate of 12.5% for all four RSSs was comparable to that reported in adults (i.e., 5–20% ACR-TIRADS, 6–17% EU-TIRADS, 15–50% K-TIRADS, and 10–20% ATA US RSS) [26–29]. Conversely, we found a risk of malignancy of about 30% associated with low-risk levels of all four RSSs (i.e., TR3, EU-TIRADS 3, K-TIRADS 3, low suspicion), which was relevant to and higher than that reported in adults (i.e., 5% ACR-TIRADS, 2–4% EU-TIRADS, 3–15% K-TIRADS, and 5–10% ATA US RSS) [26–29]. Moreover, a non-negligible risk of malignancy of 12.5–20% was associated with not suspicious/benign risk levels for ACR-TIRADS, EU-TIRADS, and K-TIRADS (i.e., TR2, EU-TIRADS 2, K-TIRADS 2), which was higher than the rates of adults (i.e., <2% ACR-TIRADS, 0% EU-TIRADS, and 1–3% K-TIRADS) [26–29]. All this means that, compared to thyroid nodules in adults, the probability of finding cancer in high- and intermediate-risk levels of the four RSSs (i.e., ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS) remains high and is not negligible for not suspicious/benign risk levels per ACR-TIRADS, EU-TIRADS, and K-TIRADS. These results are in line with what emerged in large studies by Richman et al. [34], Lee et al. [35], and Martinez-Rios et al. [36], where a significant number of malignant nodules fell in low-risk RSS categories.

While the majority of cancers (8/12, 66.7%) in our study fell within high- and intermediaterisk categories per all the four RSSs, as resumed in Table 5, six of the 12 cancers (50%) would not have undergone FNA at the initial visit according to all the four RSSs. One more cancer (ID 3) with a maximum dimension of 13 mm and intermediate-risk category would not have undergone FNA according to ACR-TIRADS and EU-TIRADS criteria. Two PTCs scored as high-risk lesions per all four RSSs would not have undergone FNA since they were 7 mm of maximum dimension. One PTC (ID 2) and the two FTCs (ID 5, ID 6) of the present cohort were scored as low-risk lesions, and, because of their size (<15 mm), FNA would have not been indicated per all four RSSs. The remaining PTC (ID 4) with maximum dimension <20 mm (i.e., 7 mm) fell within the not suspicious/benign risk categories per ACR-TIRADS, EU-TIRADS, and K-TIRADS (i.e., TR2, EU-TIRADS 2, K-TIRADS 2) and low-risk category per ATA US RSS; thus, it would not have undergone FNA.

Therefore, a high missed malignancy rate (~50%) was found in our study when using ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS. This result is conceptually comparable to what was reported by the largest study by Richman et al. [34], who found a 22.1% of missed malignancy rate applying ACR-TIRADS, and by Lee et al. [35], who found a 19.2% of missed malignancy rate applying K-TIRADS in the group without risk factors. This issue likely implies that the current RSSs (i.e., ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS) are likely inadequate for guiding FNA of thyroid nodules in patients younger than 19 years old. In this regard, as already shown for thyroid nodules in adults [19,20], we can hypothesize that the presence of two FTC cases in our pediatric cohort would also have increased the missed malignancy rate and, thus, decreased the overall ability of the four RSSs in detecting malignant nodules. Although a direct comparison with the adult population is somewhat difficult because the missed malignancy rate is largely influenced by the proportion of malignant nodules, we found the missed malignancy rate for the four RSSs to be significantly higher than that reported in the literature relative to adult patients (i.e., 2.2–9.5%) [24,37]. While this evidence may be acceptable for adult with thyroid nodules where US-based risk stratification systems are now mainly applied to

detect clinically important cancers and to avoid waste of resources (conservative approach), this may not be applied in children and adolescents where the first aim should consist of early detection of malignant nodules.

One other parameter underlying the diagnostic performance of RSSs is represented by the unnecessary FNA rate. For management of thyroid nodules in children and adolescents, this parameter could be less important to improve, as the primary objective is detecting malignancy. However, we found higher unnecessary FNA rates (i.e., almost 60% for ACR-TIRADS and EU-TIRADS, and almost 80% for K-TIRADS and ATA US RSS) than recently reported by Kim et al. [38] for adults (pooled unnecessary FNA rates of ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA were 25%, 38%, 55%, and 51%, respectively). The higher unnecessary FNA rates of K-TIRADS/ATA US RSS than ACR-TIRADS/EU-TIRADS could also be due to the lower cutoffs for FNA associated with intermediate- and lowrisk categories (i.e., 10 mm and 15 mm, respectively, compared to 15 and 20–25 mm of ACR-TIRADS and EU-TIRADS) [26–29].

In our cohort, the overall accuracy of the four RSSs in correctly indicating FNA was quite poor (i.e., ~66% for ACR-TIRADS and EU-TIRADS, ~40% for K-TIRADS and ATA US RSS). In particular, sensitivity values (i.e., 40–50%), although slightly higher for K-TIRADS and ATA US RSS, were inadequate to properly detect malignancy in this context, and they were significantly lower than that reported in adults (74% ACR-TIRADS, 54% EU-TIRADS, 86% K-TIRADS, and 87% ATA US RSS) [23].

All this suggests that, on the one hand, the four RSSs had an excellent yield in high-risk US nodules but, on the other hand, they should be appropriately modified to detect the best number of malignancies in children.

#### *4.2. Strengths and Weaknesses*

The strengths of our study are the following: (1) to our knowledge, this is the first comparative study regarding diagnostic performance of the four most used RSSs for detecting malignant thyroid lesions in pediatric patients; (2) in comparison with the largest study to date by Richman et al. [31], this study mainly provides data for the management of small thyroid nodules and cancers (the median nodule's maximal dimension was 13 mm, and the median maximal dimension of malignant thyroid nodules was 10 mm). The limitations of our study should also be discussed. This was a small and monocentric cohort. However, we strictly selected the cohort by excluding patients with apparent risk factors of malignancy, so that our results could be mainly applied to the majority of children and adolescents with sporadic thyroid cancer. Although we included patients with preexisting autoimmune thyroid disease, the putative role of the autoimmune background in the development of thyroid cancer in childhood is inconclusive to date [39,40]. This is a retrospective review of static US images which could result in inherent selection bias by the reviewers. However, interobserver agreement in scoring nodules according to all four RSSs was good. Patients with benign cytology could undergo surgical resection in the future, altering the current results of the current study. However, one-fifth of our benign cases received surgery and had histological confirmation. Since we did not have complete data on nUS relative to the vascularity of thyroid nodules, we could not assess this feature and score our nodules according to AACE/ACE/AME US RSS [41]. Our results mainly refer to PTC without apparent risk factors.

#### **5. Conclusions**

Our findings suggest that the four US-based RSSs (i.e., ACR-TIRADS, EU-TIRADS, K-TIRADS, and ATA US RSS) have suboptimal performance in managing pediatric patients with thyroid nodules, with one-half of cancers being without indication for FNA according to their recommendations. All thyroidologists, endocrinologists, and radiologists, as well as panelists of later TIRADSs, should be aware of these findings [42].

**Author Contributions:** Conceptualization, P.T. and L.S.; methodology, L.S. and P.T.; validation, L.S., M.L., C.L., A.P., P.T., G.B. and M.I.M.; formal analysis, L.S. and P.T.; investigation, L.S.; resources, S.I., L.S., G.B., A.G. and I.C.; data curation, L.S.; writing—original draft preparation, L.S., P.T., G.B. and M.I.M.; writing—review and editing, L.S., P.T., G.B. and M.I.M.; supervision, G.D., E.M.D.G. and K.E.; project administration, L.S.; funding acquisition, K.E. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board (or Ethics Committee) of University Hospital "L. Vanvitelli" (Naples, Italy) (ethic cade: 0028728/i approved date: 7 October 2021).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Validation of Four Thyroid Ultrasound Risk Stratification Systems in Patients with Hashimoto's Thyroiditis; Impact of Changes in the Threshold for Nodule's Shape Criterion**

**Dorota Słowi ´nska-Klencka 1, \* , Mariusz Klencki 1, \* , Martyna Wojtaszek-Nowicka 2 , Kamila Wysocka-Konieczna 1 , Ewa Wo ´zniak-Oseła 1 and Bozena Popowicz ˙ 1**

	-<EMAIL_ADDRESS>
**Simple Summary:** Thyroid Imaging Reporting and Data Systems (TIRADS) optimize the selection of thyroid nodules for cytological examination. There is a question: is the effectiveness of these systems affected by morphological changes to thyroid parenchyma that are visible in the course of Hashimoto's thyroiditis (HT)? This question is very important because of the increased risk of malignancy in thyroid nodules in patients with HT. We investigated widely accepted ultrasound malignancy risk features with a special consideration of the suspected nodule's shape in patients with and without HT. We also validated EU-TIRADS, K-TIRADS, ACR-TIRADS, and ATA guidelines in both groups and evaluated the impact of changes in the threshold for nodule's shape criterion on the diagnostic value of these TIRADS. The presence of Hashimoto's thyroiditis did not exert any significant adverse implications for the efficiency of examined TIRADS. The impact of changes in the threshold for nodule's shape criterion was the highest for EU-TIRADS.

**Abstract:** The aim of the study was to validate thyroid US malignancy features, especially the nodule's shape, and selected Thyroid Imaging Reporting and Data Systems (EU-TIRADS; K-TIRADS; ACR-TIRADS, ATA guidelines) in patients with or without Hashimoto's thyroiditis (HT and non-HT groups). The study included 1188 nodules (HT: 358, non-HT: 830) with known final diagnoses. We found that the strongest indications of nodule's malignancy were microcalcifications (OR: 22.7) in HT group and irregular margins (OR:13.8) in non-HT group. Solid echostructure and macrocalcifications were ineffective in patients with HT. The highest accuracy of nodule's shape criterion was noted on transverse section, with the cut-off value of anteroposterior to transverse dimension ratio (AP/T) close to 1.15 in both groups. When round nodules were regarded as suspicious in patients with HT (the cut-off value of AP/T set to ≥1), it led to a three-fold increase in sensitivity of this feature, with a disproportionally lower decrease in specificity and similar accuracy. Such a modification was effective also for cancers other than PTC. The diagnostic effectiveness of analyzed TIRADS in patients with HT and without HT was similar. Changes in the threshold for AP/T ratio influenced the number of nodules classified into the category of the highest risk, especially in the case of EU-TIRADS.

**Keywords:** thyroid; nodule; cancer; ultrasound; Thyroid Imaging Reporting and Data Systems (TIRADS)

#### **1. Introduction**

Preoperative diagnostics of thyroid nodules is based on two main pillars—ultrasound imaging (US) and fine needle aspiration biopsy (FNA). The ultrasonographic examination is mainly used for the assessment of ultrasound malignancy risk features (US malignancy features), and subsequent qualification of nodules into particular categories of sonographic

**Citation:** Słowi ´nska-Klencka, D.; Klencki, M.; Wojtaszek-Nowicka, M.; Wysocka-Konieczna, K.; Wo ´zniak-Oseła, E.; Popowicz, B. Validation of Four Thyroid Ultrasound Risk Stratification Systems in Patients with Hashimoto's Thyroiditis; Impact of Changes in the Threshold for Nodule's Shape Criterion. *Cancers* **2021**, *13*, 4900. https://doi.org/10.3390/ cancers13194900

Academic Editor: Pierpaolo Trimboli

Received: 17 August 2021 Accepted: 27 September 2021 Published: 29 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

risk stratification systems (SRSs). These systems are usually called TIRADS (Thyroid Imaging Reporting and Data Systems) and they enable a more efficient estimation of the risk of malignancy (RoM) in nodules than the evaluation of separate US malignancy features. The most popular SRSs include EU-TIRADS—recommended by European Thyroid Association (ETA), K-TIRADS—recommended by Korean Society of Thyroid Radiology (KSThR), ACR-TIRADS—created by American College of Radiology (ACR), and the system recommended by American Thyroid Association (ATA guidelines) [1–4]. Our analyses, as well as many reports from other centers, indicate that all these systems not only optimize the selection of nodules for cytological examination but also help to make clinical decisions in patients with an equivocal FNA outcome [5–9]. There is, however, some disagreement between TIRADS systems about the precise definition of particular US malignancy features and their optimal association. One of the areas of notable difference refers to the definition of the nodule's suspicious shape, usually described as 'more taller than wide'. Not all SRSs include precise instructions on how to categorize nodules with the anterior-posterior (AP) dimension equal to the transverse (T) dimension or which thyroid plane should be used for the shape evaluation. Remarkably, there are reports that suggest a rationale for adopting a larger than 1.0 threshold for the AP to T ratio [10], and even studies indicating that the optimal threshold should be <1 [11,12]. There is another important question: is the effectiveness of the suspected nodule's shape and other US malignancy features affected by morphological changes to thyroid parenchyma that are visible in the course of Hashimoto's thyroiditis (HT).

HT is the most common autoimmune endocrine disease, as well as the most common cause of hypothyroidism. This inflammation is characterized by a progressive loss of thyroid follicular cells and lymphocytic infiltration of the thyroid parenchyma associated with fibrosis [13,14]. It is usually accompanied by a decrease in the gland's volume and several characteristic changes visible on US imaging. The thyroid may be hypoechoic, with a coarse, heterogeneous parenchymal echotexture, or have the presence of the marginal abnormality, echogenic septations, multiple discrete hypoechoic micronodules or pseudonodular structures. These features may be present separately or in different sets and make it difficult to differentiate between thyroid nodules and pseudonodules, and in the former case—between cancers and benign lesions [15–17]. The latter problem is of particular importance considering the increased risk of papillary thyroid carcinoma (PTC) in the case of nodules coexisting with HT [18,19].

Thus, we decided to validate US malignancy features, with a special consideration for the nodule's shape, and selected TIRADS systems in patients with or without coexisting HT (HT and non-HT groups).

#### **2. Results**

#### *2.1. Effectiveness of the Assessment of Suspicious Nodule's Shape in Differentiation between Benign and Malignant Nodules in HT and non-HT Groups*

The usefulness of AP/T ratio assessment in the differentiation between benign nodules and cancers, as measured with area under the receiver operating characteristic curve (AUC), was similar in both groups (transverse plane, Z: −0.1893, *p* = 0.8498; longitudinal plane, Z: 0.2837, *p* = 0.7767) (Table 1). When indexes of diagnostic effectiveness were calculated for the threshold AP > T they were found to be nearly the same in the case of transverse plane. However, in the case of longitudinal plane the AP > T threshold was ineffective in patients with HT.

**Table 1.** Data on the diagnostic effectiveness of particular thresholds for AP/T ratio in examined groups of nodules (HT and non-HT). Evaluation of AP/T ratio on transverse and longitudinal planes.

ACC—accuracy, AP—anteroposterior diameter, AUC—area under the receiver operating characteristic curve, Ben.—benign lesion in histopathological outcome, CI—confidence intervals, HT—Hashimoto's thyroiditis, LR+—positive likelihood ratio, Mal.—thyroid malignancy in histopathological outcome, NPV—negative predictive value, OR—odds ratio, PPV—positive predictive value, RoM—risk of malignancy, SEN—sensitivity, SPC—specificity, T—transverse diameter.

> When AP > T threshold was replaced with AP ≥ T one, a significant increase in sensitivity (SEN) was observed in both groups, and that increase was higher in HT group than in non-HT group. More pronounced changes in HT group were a consequence of the higher incidence of round nodules with AP = T in that group in comparison with non-HT group (Table S1). On the transverse plane it was the case for both benign nodules and cancers. On the longitudinal plane the differences were smaller, insignificant and they were observed only for cancers. The higher incidence of round cancers on transverse plane was observed not only for PTC (HT: 27.6% vs. non-HT: 16.3%, *p* = 0.0408), but also for other malignant nodules, although insignificantly (HT: 27.3% vs. non-HT: 10.4%, *p* = 0.1408). Consequently, the threshold AP ≥ T on transverse plane was the only effective threshold in HT group for revealing cancers other than PTC, odds ratio (OR): 4.5, CI 95%: 1.3–15.6, *p* = 0.0160.

> The highest accuracy (ACC) values for the differentiation between benign and malignant nodules were noted in both groups in the case of transverse plane. Maximal accuracy was reached in HT group with the cut-off value of AP/T ratio set to 1.14, while in non-HT group—to 1.17 (Figure S1). With those thresholds changes in SEN and specificity (SPC) did not exceed 4% in both groups when compared with the threshold AP > T, while risk of malignancy (RoM) of nodules was about 20% higher, and positive likelihood ratio (LR+) and OR increased twofold (Table 1). There were no significant differences between examined groups in indexes of diagnostic effectiveness of the suspicious shape when thresholds optimized for ACC were used.

> When the longitudinal plane was used for measurements, the maximal ACC values were slightly lower than in the case of transverse plane and the optimal cut-off value of AP/T ratio was found to be AP ≥ T in both groups (Figure S2). Regardless of the adopted cut-off value, no improvement in ACC values was observed in any of the groups when the assessment of nodule's shape was performed on both planes (with positive nodules defined as those reaching the threshold on any plane) in comparison with the assessment on a single plane (Table S2).

There was no significant difference in AUC for nodules <1 cm and larger ones in either group (Z: 1.0524, *p* = 0.2926; non-HT: Z: −0.6656, *p* = 0.5056), but in HT group the assessment of suspicious shape feature was ineffective in nodules <1 cm (Table S3). In non-HT group significant differences in the frequency of nodules <1 cm with suspicious shape between cancers and benign nodules were observed only for the threshold AP ≥ T.

#### *2.2. Effectiveness of the Assessment of Other US Malignancy Features*

In non-HT group almost all other US malignancy features, except for pathological vascularization and rim calcifications, were observed significantly more often in cancers than in benign nodules (Tables 1 and 2). In the case of HT group, the list of insignificant features included also solid echostructure, more solid than cystic echostructure, and macrocalcifications. The logistic regression analysis showed that in HT group the presence of microcalcifications was the strongest indication of nodule's malignancy (OR: 22.7), and that the presence of suspicious margins or marked hypoechogenicity increased the risk of malignancy at least tenfold. In non-HT group, the strongest index of nodule's malignancy was the presence of irregular margins (OR: 13.8).

**Table 2.** Comparison of the incidence of sonographic features other than suspicious nodule's shape in HT and non-HT nodules in relation to the histopathological outcome: benign lesion vs. thyroid malignancy. Results of univariate logistic regression analysis in both groups.

\*—in the case of nodules with mixed echogenicity the presence of any hypoechoic tissue was considered; Ben.—benign lesion in histopathological outcome, CI—confidence intervals, OR—odds ratio, HT—Hashimoto's thyroiditis, Mal.—thyroid malignancy in histopathological outcome. Data on nodule's shape criterion are presented in Table 1.

> Microcalcifications, irregular margins, marked hypoechogenicity, suspicious shape and hypoechogenicity were independent features in the differentiation between benign and malignant nodules in both groups (Table S4). Macrocalcifications and solid echostructure were such features only in non-HT group.

> Benign nodules of HT group were solid significantly more often than those of non-HT group (91.1% vs. 70.1%, *p* < 0.0001) and more solid than cystic (97.4% vs. 85.7%,

*p* < 0.0001), but they showed pathological vascularization less frequently (13.7% vs. 19.1%, *p* < 0.0474). Malignant nodules of HT group contained microcalcifications more often than cancers of non-HT group (29.9% vs. 18.3%, *p* < 0.0273), while macrocalcifications were less frequent (5.8% vs. 19.2%, *p* < 0.0033). Spongiform echostructure was observed only in benign nodules of both groups, but it was less common in HT group than in non-HT group (5/1.9% vs. 46/7.4% respectively, *p* = 0.0018).

#### *2.3. Comparison of the Effectiveness of Analyzed SRSs*

Table 3 shows the distribution of benign and malignant nodules among particular categories of the examined SRSs for each of previously analyzed thresholds of AP/T ratio with measurements on transverse plane. In the majority of cases, calculated RoM for particular categories of analyzed TIRADSs corresponded to expected RoM or differed by less than 5%. Larger differences (up to 10%) were observed in the case of lower than expected RoM for category 5 of EU-TIRADS with the threshold AP ≥ T in both groups as well as higher than expected RoM for category 4 of K-TIRADS, ACR-TIRADS and ATA guidelines SRS in non-HT group.

All analyzed SRSs showed the highest ACC of distinguishing between benign and malignant nodules when category 5 was used as a cut-off level, irrespectively of the adopted threshold of AP/T ratio. Table 4 shows values of indexes describing the effectiveness of that distinction. System EU-TIRADS, with any of analyzed thresholds of AP/T ratio, showed the highest SEN and negative predictive value (NPV), but the lowest SPC and positive predictive value (PPV) (see Table 3). On the other hand, ACR-TIRADS system was characterized by the lowest SEN and the highest SPC. Generally, we did not find significant differences in AUC between analyzed SRSs at the same AP/T ratio thresholds. The only exception was observed in non-HT group, where AUC for ACR-TIRADS was significantly lower than for other SRSs when the threshold was AP/T ≥ 1.17. No significant differences were found in the effectiveness any of SRSs between HT and non-HT groups when the comparison was made at the same AP/T ratio threshold.

When AP > T threshold was replaced with AP ≥ T one, the numbers of nodules classified into category 5 increased in all SRSs. Accordingly, there was an increase in SEN of that category (when it was used as a threshold for malignancy) and a decrease in its SPC and RoM (Tables 3 and 4). In the case of EU-TIRADS that effect was stronger in HT group than non-HT group. In the former group the number of nodules in category 5 increased by 32.6%, SEN increased by 11.2%, SPC decreased by 11.2% and RoM decreased by 16.1%. In non-HT group analogous changes were 15.7%, 7.0%, 4.7% and 7.4%, respectively. In other SRSs the resultant changes in SEN, SPC, RoM and the percentage of nodules classified into category 5 were similar in both groups (Table S5).

When the AP/T ratio threshold was optimized to obtain the maximal ACC (HT group: AP/T ≥ 1.14; non-HT group: AP/T ≥ 1.17) the most distinct effects in comparison to AP > T threshold were observed again in the case of EU-TIRADS. They were especially visible in non-HT group, where RoM increased by 7.2% and SEN decreased by only 0.8%. AUC value in non-HT group for EU-TIRADS with the threshold AP/T ≥ 1.17 was significantly higher than for AP > T threshold. No significant difference in AUC was observed in any other SRSs when AP/T ratio threshold was changed in either group.

**Table 3.** Distribution of benign and malignant nodules between particular categories of TIRADS, the comparison of expected RoM with calculated RoM for each category (TIRADS categories corresponding to the lack of nodules have been omitted, nodule's shape evaluated on the transverse plane).

\*—included 71 non-hypoechoic nodules with high risk features (including 22 cancers, 8 in HT group and 14 in non-HT group). AP anteroposterior diameter, HT—Hashimoto's thyroiditis, PPV—positive predictive value, RoM—risk of malignancy, T—transverse diameter, TIRADS—Thyroid Imaging Reporting and Data Systems.

**Table 4.** Data on the diagnostic effectiveness of analyzed SRSs in HT and non-HT groups for the high risk category (nodule's shape evaluated on the transverse plane).

**Table 4.** *Cont.*

a—*p* < 0.01 vs. EU-TIRADS threshold AP > T. b—*p* < 0.05 vs. EU-TIRADS, K-TIRADS and ATA guidelines (threshold AP/T ≥ 1.17 all). c—*p* < 0.05 vs. EU-TIRADS and K-TIRADS (threshold AP > T all). ACC—accuracy, AP—anteroposterior diameter, AUC—area under the receiver operating characteristic curve, Ben.—benign lesion in histopathological outcome, CI—confidence intervals, HT—Hashimoto's thyroiditis, LR+—positive likelihood ratio, Mal.—thyroid malignancy in histopathological outcome, NPV—negative predictive value, SEN—sensitivity, SPC—specificity, T—transverse diameter, TIRADS—Thyroid Imaging Reporting and Data Systems.

#### **3. Discussion**

Hashimoto's thyroiditis is a common thyroid disease, especially in areas of high iodine supply. It is usually accompanied by significant changes in the morphology of the gland that impair the identification of thyroid nodules and the assessment of a nodule's US malignancy features. These difficulties are commonly aggravated by the small size of nodules. Despite these complications, our analysis shows that all four of the most recognized and strongest US malignancy features (i.e., marked hypoechogenicity, irregular margins, microcalcifications, and suspicious shape) are effective in distinguishing benign nodules from cancers in the case of coexisting HT. Other investigators also indicate that the majority of US malignancy features present similar efficiency in patients with or without HT [16,20–26]. However, there are some differences. In our study, microcalcifications were an almost three times stronger indication of malignancy in a nodule in patients with HT than in those without HT. We believe that this difference is a consequence of increased prevalence of PTC among cancers in patients with HT. In addition, it is PTC that microcalcifications are particularly characteristic of. However, there is no full agreement on the incidence of various types of calcifications in patients with HT. As with our observations, Baser et al. (2015) found that macrocalcifications were observed less often in HT than non-HT patients [22]. Durfee et al. (2014) and Gul et al. (2010) did not find any significant differences in calcification types between HT and non-HT groups [20,25]. In addition, Ohmori et al. (2007) even noted an increased incidence of dense calcifications and decreased incidence of psammoma bodies in thyroid cancer associated with HT compared to cancers without HT [27]. More concordant is the opinion that the assessment of solid structure of nodules is not useful in patients with HT [16,23,24,26], as can also be concluded from our study. That is a consequence of the fact that almost all nodules which accompany HT are solid, even the benign ones. This may result from their smaller sizes; an influence of morphological changes induced by HT cannot be excluded either. Observations regarding suspicious margins are less concordant [16,20,28]. In our study, their assessment was effective regardless of the presence of HT. Such results can be achieved only if the assessment is performed by an experienced ultrasonographer. Experience is necessary to avoid the interpretation of ill-defined pseudonodules as true nodules. Our team is fully aware of that danger due to our previous studies on the relation between FNA outcomes in patients

with HT and different ultrasonographic images of thyroid parenchyma, including variants with pseudonodules [29]. All systems analyzed in our study regard irregular margins as suspicious. The authors of K-TIRADS underline that ill-defined margins are visible not only in thyroiditis, but also in infiltrative malignant tumors, but in our opinion the latter almost always present irregularity in their margins. The problem arises because that irregularity is frequently very fine, which gives an impression of ill-defined margins.

The most complicated issue related to US malignancy features is the assessment of the suspicious shape of a nodule. We have shown that thyroid nodules in patients with HT are round (their AP = T) more often than nodules in patients without HT. When the transverse plane is used for measurements the above is equally true for benign nodules and cancers, and in the latter case, not only PTCs but other cancers too. When the longitudinal plane is used, only cancers are more often round. Consequently, in patients with HT the assessment of the suspicious shape of a nodule on the transverse plane is effective with both thresholds: AP > T and AP ≥ T, while on the longitudinal plane only AP ≥ T threshold is effective. Interestingly, when AP ≥ T threshold is used on the transverse plane the assessment of the suspicious shape becomes also effective in diagnosis of cancers other than PTC. Previously, other authors did not find the assessment of suspicious shape to be useful for diagnosing patients with FTC [30], so that issue should be investigated further. However, the key question is an equivocal definition of the suspicious shape of a nodule. It is usually described as 'more taller than wide', which in practice in the majority of studies translates into AP > T threshold, but in some of them into the AP ≥ T one [31]. In the guidelines and in many reports, there are no precise indications as how to categorize round nodules or such indications are contradictory. Among SRSs analyzed in our study, EU-TIRADS identifies the suspicious shape of a nodule as non-oval or round [1]. In contrast, K-TIRADS defines the suspicious shape as neither round nor oval [2]. Similarly, in ACR-TIRADS the threshold of AP/T ratio >1, and ATA guidelines use the description 'more taller than wide shape' [3,4]. There is also some controversy over the plane of measurements. K-TIRADS specifies that AP > T should be observed on a transverse or longitudinal plane, while ACR-TIRADS and ATA guidelines limit that condition to the transverse plane only. In addition, EU-TIRADS does not specify the plane of measurements in regard to non-oval shape, but indicates that the definition of round shape and oval shape demands the respective conditions (AP = T or AP < T) to be satisfied on both the transverse and longitudinal planes. One could argue that a nodule that is non-oval on any plane is suspicious according to EU-TIRADS. However, that way there would be two different regimens of measurements for non-oval and round shaped nodules, both types regarded as suspicious in EU-TIRADS.

Irrespectively of these differences, our analyzes, like reports from many other centers, indicate that the assessment of nodule's suspicious shape is characterized by a very high SPC [10,30,32,33], with lower SEN, especially on the longitudinal plane [30]. SEN of that feature on the longitudinal plane reaches values similar to those on the transverse plane when round nodules are regarded as suspicious too. The inclusion of round nodules into suspicious category improves SEN on the transverse plane by several times, especially in patients with HT. It is noteworthy that the effectiveness of the shape assessment is not improved when the conditions AP > T or AP ≥ T are evaluated on both planes (with positive feature defined as the condition satisfied on any plane) when compared with the measurements performed on a single plane. Both our study and other reports [34] indicate that the assessment on the transverse plane and the longitudinal plane have similar ACC. However, the nodule's suspicious shape is very rarely identified on the longitudinal plane only. Thus, like Kim et al. (2021) we believe that there is no need for the assessment of a nodule's shape on both planes and measurements on the transverse plane are satisfactory [32]. There are some earlier contradictory reports in this regard [35].

The highest ACC of the assessment could be obtained with measurements on the transverse plane when the threshold for AP/T ratio is close to 1.15 (1.14 in HT group and 1.17 in non-HT group). If such a threshold is used there are moderate changes in SEN and SPC, but a twofold increase in OR and LR+ is observed when compared to the threshold AP > T. A further increase of AP/T threshold to values proposed by Grani et al. (2020) (AP/T ratio = 1.2) in patients without HT would not improve OR (OR: 4.2 vs. 5.8), and in patients with HT there would even be a loss of discrimination power of the suspicious shape feature (incidence of benign nodules with AP/T ≥ 1.2 did not differ significantly from the incidence of such cancers: 1.1% vs. 3.5%, *p* = 0.1602) [10]. It is possible that these differences between our data and the study by Grani are a consequence of the lack of nodules with indeterminate cytology in the Italian study. In such nodules, US malignancy features generally have lower effectiveness, due to a lower percentage of PTC among cancers. On the other hand, Topaloglu et al. (2016) examined patients with nodules of category III of the Bethesda System for Reporting Thyroid Cytology (BSRTC) and proposed the threshold for AP/T ratio below one (0.81) [11]. A lower threshold was even proposed by Huang et al. (2018) while diagnosing papillary microcarcinomas (0.7) [12]. However, those thresholds were established in a different way. They were optimized to produce maximal SEN and SPC at the same time, but not ACC. When we followed the same priorities then the threshold (as determined with ROC curve analysis) was 0.92 in patients with HT (SEN: 43.7%, SPC: 80.1%, ACC: 71.2%, PPV: 41.3%, NPV: 81.6%, OR: 3.1) and 0.87 in patients without HT: 0.87 (SEN: 46.6%, SPC: 72.5%, ACC: 66.0, PPV: 36.2%, NPV: 80.2%, OR: 2.3). In HT group these values did not differ significantly from those obtained for the threshold AP ≥ T. However, in a non-HT group such a low threshold was unsatisfactory, leading to low SPC and PPV as almost 1/3 of nodules (32.3%) reached that threshold. This is an important weakness in the case of a US malignancy feature that is key in assigning nodules into the highest risk category of SRS. In the studies by Topaloglu and Huang reported SPC values for proposed thresholds were even lower (61.6% and 66.7%, respectively), and PPV did not exceed 30% (Topaloglu et al.: 29.1%, Huang et al.: missing data on PPV) [11,12].

We found that in patients with HT the assessment of nodule's suspicious shape has low effectiveness in the case of nodules <1 cm. Fukushima et al. (2021) did not find differences in the effectiveness of that feature between small and larger nodules [30]. Ren et al. (2015) found it to be even higher in nodules <1 cm, but they did not analyze the possible influence of HT on that effectiveness [33]. In our group of patients without HT the AUC value for nodules < 1 cm was also slightly higher than for larger nodules.

For obvious reasons, the use of a precise threshold for AP/T ratio in practice demands an additional effort from the ultrasonographer. The authors of some reports admitted that the suspicious shape was determined based on ultrasonographer's impression instead of the measurements of nodules' diameters. Such an assessment makes it difficult to detect fine abnormalities in a nodule's shape. As our data indicate, it is worthwhile to perform proper measurements because even a minor modification of the threshold for nodule's suspicious shape may lead to a several-fold change in SEN of that feature and marked changes in its SPC and RoM. The local experience of a center is especially valuable in this case, because the reproducibility of the assessment of nodule's suspicious shape, like other US malignancy features, is not high [36]. It results not only from technical differences in measurements (e.g., caused by different positions of a patient, or different pressure exerted by a probe on the examined area), but also from differences in the epidemiology of thyroid diseases in examined patients (different incidence of HT, other profile of cancers). Thus, in our opinion, it could be advisable to determine the optimal threshold for AP/T ratio in each center individually, to adjust it to its specific situation. It should be noted that while the discussed changes in the AP/T ratio threshold may lead to a several-fold increase in SEN, the resultant SEN is still far from satisfactory, similarly to other high risk features of malignancy.

The results of our study indicate that changes in the threshold of AP/T ratio have a different impact on the effectiveness of particular SRSs in classification of benign and malignant nodules. The impact is bigger for EU-TIRADS than for other SRSs. This is a result of the particular definition of high risk category in EU-TIRADS which is different from other systems based on similar rules: K-TIRADS and ATA guidelines (ACR-TIRADS

uses points for US malignancy features unlike all other SRSs). In EU-TIRADS a nodule with strong risk features does not need to be hypoechoic to be classified into category 5 [1]. Consequently, category 5 of that system directly reflects all effects of categorization of all round nodules as suspicious. In addition, these effects are larger in HT group than non-HT group. Similarly, the change of AP > T threshold to a threshold that gives maximal ACC of the nodule's shape assessment produces more pronounced effects on category 5 in EU-TIRADS than in other SRSs. This effect is advantageous as it leads to larger increase in RoM and SPC of that category. Therefore, in the case of EU-TIRADS there is the greatest possibility of selecting various thresholds of AP/T ratio in order to optimize SPC (the threshold close to 1.15) or SEN (AP ≥ T threshold) of that system.

Despite these differences it should be emphasized that all SRSs showed similar effectiveness in distinguishing benign nodules from cancers in patients with HT at the same thresholds of AP/T ratio. Additionally, there were no significant differences in the effectiveness of those systems in patients with or without HT. Likewise, Wang et al. (2015) did not find any significant influence of HT on the effectiveness of SRSs (they evaluated the system proposed by Kwak, ACR-TIRADS and ATA guidelines) [37]. We showed that EU-TIRADS had the highest SEN with the lowest SPC and ACR-TIRADS had the highest SPC with the lowest SEN irrespectively of the threshold used for AP/T ratio or the presence of HT, which is concordant with other reports [5,37–40]. In both HT and non-HT patients the calculated RoM of nodules in particular categories of SRSs was generally close to the expected one. Differences over 5% were observed for category 4 of K-TIRADS, ACR-TIRADS and ATA guidelines systems, where the risk in non-HT group was higher than expected. That could be the consequence of a notable fraction of cancers other than PTC in our material, amounting to 23.1% in non-HT group. These cancers were mainly FTC and HTC and were usually classified into category 4 of those SRSs (56.3% cancers other than PTC for K-TIRADS and ATA guidelines, 64.6% for ACR-TIRADS). The higher percentage of cancers other than PTC was also a probable cause of slightly lower AUC of those SRSs in non-HT group than in patients with HT. It should be kept in mind that US malignancy features and SRSs were established mainly on the basis of the ultrasound image of the most common PTCs [41]. Effectiveness of SRSs could potentially be improved by the inclusion of elastographic measurements [42,43]. Promising results in this respect were also obtained in patients with HT [21,44,45].

A limitation of our study is the difference in mean sizes of nodules in patients with and without HT. The size might affect the US characteristics of thyroid nodules, but small nodules are typical of HT. The advantage of our study is performing US malignancy feature evaluation directly prior to biopsy. Therefore, the result of FNA did not influence the evaluation. The majority of final diagnoses were based on postoperative histopathological examination. It is an advantage because of the certainty of the final diagnosis but on the other hand it could be a source of a bias in patient selection. In clinical practice, patients with HT are usually not referred to surgical treatment. Despite this way of including patients in the study the distribution of nodules among equivocal and unequivocal categories of BSRTC was similar in both groups. That is advantageous because there are significant differences in the usefulness of the assessment of US malignancy features in relation to nodule's category of FNA outcome [6]. The adopted way of confirmation of HT diagnosis may also be regarded as advantageous because of rather rigorous criteria. However, it may be seen as a limitation in relation to seronegative cases of HT. In our study, patients with a positive TPOab test dominated, and seronegative patients, in whom HT was confirmed by its characteristic features in the cytological examination, constituted a very small percentage of HT group.

#### **4. Materials and Methods**

#### *4.1. Examined Patients*

Ultrasound imaging and FNA examinations were performed in a single center, in the years 2012–2020, in patients referred by endocrinologists from outpatient clinics. The study

included all nodules classified into the categories II–VI of BSRTC with full ultrasound imaging data, a known result of the postoperative histopathological examination and a known status of HT presence. The exclusion criteria consisted of previous surgical or radioiodine treatment, as well as positive neck irradiation history. Because of relatively small number of nodules assigned to category II in BSRTC with a known result of histopathological examination in patients with HT, in that group we additionally included all nodules with full ultrasound imaging data and category II of BSRTC confirmed in at least three FNAs. The study included 1188 nodules (revealed in 1022 patients), i.e., 358 nodules in patients with HT (HT group) and 830 nodules in patients without HT (non-HT group) (Table 5). The differences in the incidence of particular categories of the Bethesda system between groups HT and non-HT did not exceed 5%. Frequencies of nodules with an equivocal (categories III–V) FNA outcome were similar in both groups (HT: 50.8% vs. non-HT: 54.1%, *p* = 0.3018).

**Table 5.** Data on the diagnostic effectiveness of analyzed SRSs in HT and non-HT groups for the high risk category (nodule's shape evaluated on the transverse plane).

\*—including 78 nodules, without the surgical treatment but after three FNAs with all outcomes classified into category II of BSRTC. #—both cancers and benign nodules <1 cm were more frequent in HT group than non-HT one (34.5% vs. 19.7%, *p* = 0.0068 and 10.3% vs. 5.8%, *p* = 0.0155, respectively). PTC—papillary thyroid carcinoma, MTC—medullary thyroid carcinoma, FTC follicular thyroid carcinoma, HTC—Hurthle cell thyroid carcinoma, PDTC—poorly differentiated thyroid carcinoma, AC—anaplastic carcinoma, ST—secondary tumor, ANG—angiosarcoma, FT-UMP—follicular tumor of uncertain malignant potential.

> In all patients, the clinical diagnosis of HT was established by an endocrinologist in the endocrine outpatient clinic on the basis of clinical symptoms, serological tests (measurement of anti-thyroid antibodies), assessment of serum concentrations of thyroid stimulating hormone and thyroid hormones, ultrasound examination, as well as cytological examinations. Because fewer than 30% of patients with HT present all the above-mentioned HT features [13,14,46], we decided to adopt minimal conditions for the diagnosis of HT. We assumed that all patients in HT group had to have a clinical diagnosis of HT confirmed with elevated levels of serum anti-thyroid peroxidase antibodies (TPOab) or characteristic features of HT in the microscopic examination. On the other hand, patients included in the non-HT group did not have any hormonal, morphological, or ultrasound features of HT and they had normal TPOab.

#### *4.2. Analysis of US Malignancy Features*

The analysis of US malignancy features was done prospectively. The presence of particular US malignancy features was assessed by experienced sonographers (four physicians with over ten years' experience) directly before FNA, according to a unified pattern that had been used at our department for many years. We used a computer system dedicated for

collecting detailed information on examined nodules in a database. The system was created by one of the authors of the study—MK. On the basis of those data, three diameters of biopsied nodules were determined as well as the presence of: (1) marked hypoechogenicity (compared to the echogenicity of the strap muscles); (2) hypoechogenicity (as compared to the normal thyroid); (3) solid echostructure (>90% solid) (4) more solid than cystic echostructure (>50% solid); (5) suspicious shape/orientation, assessed on the transverse and longitudinal planes, interpreted in two variants: AP ≥ T and AP > T; (6) suspicious margins—irregular (including microlobulated, spiculated, and suggesting extrathyroidal extension); (7) microcalcifications; (8) macrocalcifications; (9) rim calcifications; (10) pathological vascularization (marked intranodular vascular spots). The presence of spongiform echostructure (>50% of nodule, without obvious solid areas) was also assessed. The US examinations were performed with the use of the Aloka Prosound Alpha 7 ultrasound system, ALOKA co. Ltd., Tokyo, Japan with a 7.5–14 MHz linear transducer.

With the use of the set of features specified above, all thyroid nodules were classified into specific categories of four SRSs: EU-TIRADS [1], K-TIRADS [2], ACR-TIRADS [3], and the system recommended by ATA [4]. The differences in the interpretation of nodules with mixed echogenicity between particular SRSs were considered (the presence of any hypoechoic tissue assessed in EU-TIRADS or predominant echogenicity in other SRSs). In the case of the ATA guidelines, a modification was applied, because this system does not cover all nodule's ultrasound patterns; in particular it lacks patterns in which iso- or hyperechoic nodules show high malignancy risk features. In total, 71 (6.0%) nodules did not satisfy the criteria of ATA classification and those nodules corresponded to 22 cancers (8/9.2% in HT group and 14/6.7% in non-HT group, *p* = 0.4625). We decided to classify such nodules into the intermediate suspicion category. That allowed us to compare how all systems worked in the evaluation of the same set of nodules. We did not identify disrupted rim calcifications with small extrusive soft tissue component as a separate feature (which is included in ATA guidelines), but the nodules presenting such an image were treated as ones with irregular margins which resulted in the same output of the categorization. Two researchers (DSK and MK) independently assigned all the ultrasound features for TIRADS score calculation. In the case of discrepancy, the US report was jointly reevaluated and discussed to confirm its categorization.

#### *4.3. Analyses, Statistical Evaluation*

At the first step of the evaluation, receiver operating characteristic (ROC) curves were determined in both groups and both thyroid planes for AP/T ratio describing nodule's shape. The Z test was used to compare the area under the ROC (AUC) value between HT and non-HT groups. The cut-off values of AP/T ratio that showed the highest accuracy (ACC) in the classification of benign and malignant lesions were also identified. Odds ratios (OR) with relative 95% confidence intervals (95% CI) for the established cut-off values were assessed with the use of logistic regression analysis. The effectiveness of the determined thresholds as well as the 'standard' thresholds: AP > T and AP ≥ T was described with the use of sensitivity (SEN), specificity (SPC), ACC, positive likelihood ratio (LR+) and the percentage of nodules reaching given threshold. The RoM of those nodules (the proportion of cancers among all positive nodules, i.e., the positive predictive value—PPV) and the negative predictive value (NPV) were calculated. Additionally, the effectiveness of all examined thresholds of AP/T ratio was assessed for nodules <1 cm and larger as well as for cancers other than PTC.

Next, the incidence of other US malignancy features was assessed in the nodules classified into HT and non-HT groups in respect to the division of the nodules into benign lesions and cancers according to the final diagnosis. In the case of nodules of mixed echogenicity the presence of any hypoechoic tissue was regarded as hypoechogenicity. The associations between individual US malignancy features and malignancy were evaluated with the use of logistic regression analysis in both groups. The OR were calculated to determine the relevance of all potential predictors of the outcome. The incidence of all US malignancy features was also compared between benign nodules of HT and non-HT groups and between malignant nodules in both groups.

Then the distribution of benign and malignant nodules among particular categories of the examined SRSs was assessed. The efficiency of the systems was compared analyzing ROC curves and cut-off categories with the highest ACC were identified for each of the SRSs. Using those cut-off categories SEN, SPC ACC, RoM/PPV, NPV were calculated, as well as the percentage of positive nodules in both groups. Those analyzes were performed separately for three variants of AP/T ratio interpretation described above.

The statistical analysis was performed with the Dell Statistica (data analysis software system), version 13, Dell Inc. (2016), Round Rock, TX, USA. The comparison of frequency distributions was performed with chi2 test (with modifications appropriate for the number of analyzed cases). The Kruskal–Wallis test was used for comparing continuous variables between groups. The value of 0.05 was assumed as the level of significance.

#### *4.4. Microscopic Examination*

Biopsies were performed following regular procedures, on nodules with a diameter of at least 5 mm (and usually over 1 cm) and at least one malignancy risk factor (ultrasonographic or clinical). In most cases, two aspirations of a nodule were done. Smears were fixed with a 95% ethanol solution and stained with haematoxylin and eosin. The FNA outcome of each nodule was classified into one of six categories in the BSRTC—category I: non-diagnostic/unsatisfactory biopsies, category II: benign lesions (BL), category III: follicular lesion of undetermined significance (FLUS)/atypia of undetermined significance (AUS), category IV: suspicious for a follicular neoplasm (SFN), category V: suspicious for malignancy (SM) and category VI: malignant neoplasm (MN) [47,48]. Patients with a FNA outcome of category IV, V, or VI were routinely referred for thyroid surgery. In the case of a diagnosis of BL or FLUS/AUS, surgical treatment was performed based on the patient's preference or due to the large size of the goiter as well as the presence of other clinical, ultrasonographic or cytological risk features (especially in the case of AUS diagnosis).

#### **5. Conclusions**

The optimal usage of US malignancy features in patients with HT is very important because of the increased risk of malignancy in thyroid nodules in that group. To reach this goal, it may be helpful to adjust the threshold for AP/T ratio to the specific characteristics of nodules found in patients with HT. This specificity consists in more frequent occurrence of round nodules. In patients with HT, a slight modification of the threshold for AP/T ratio and regarding round nodules as suspicious leads to a three-fold increase in SEN of the suspected nodule's shape feature, with a disproportionally lower decrease in SPC and similar ACC. Importantly, such a modification is effective also for cancers other than PTC. In patients without HT there are analogical, yet less marked changes. Thus, the use of AP ≥ T threshold instead AP > T is justified, especially in patients with HT and in centers that intend to improve SEN. On the other hand, in centers where, due to the epidemiology of thyroid diseases, the priority is not an improvement of SEN but maximization of SPC and PPV, it is rational to use threshold for the AP/T ratio over 1, close to 1.15. It enables a twofold increase in OR of nodules that reach such a threshold in comparison to the classic AP > T threshold, with a very low loss of SEN and maximization of ACC. It also could lead to an even twofold reduction of the number of performed FNA. The assessment of a nodule's shape on the transverse plane is sufficient for the effective use of that feature with any of the analyzed thresholds. The diagnostic effectiveness of EU-TIRADS, K-TIRADS, ACR-TIRADS, and ATA guidelines in patients with HT and without HT is similar. Changes in the threshold for AP/T ratio modify that effectiveness, and their influence on the number of nodules classified into the category of the highest risk is the greatest in the case of EU-TIRADS.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/ 10.3390/cancers13194900/s1, Figure S1: ROC curve analysis of the evaluation of AP/T ratio on transverse plane in HT and non-HT groups; points of maximal ACC indicated on both curves, Figure S2: ROC curve analysis of the evaluation of AP/T ratio on longitudinal plane in HT and non-HT groups; points of maximal ACC indicated on both curves, Table S1: Incidence of round nodules (with anteroposterior diameter equal to transverse diameter) in HT and non-HT group, Table S2: Data on the diagnostic effectiveness of AP/T ratio evaluation in examined groups of nodules (HT and non-HT), with the threshold AP ≥ T or AP > T satisfied on any plane (transverse or longitudinal), Table S3: Data on the diagnostic effectiveness of AP/T ratio evaluation (on transverse plane) in examined groups of nodules (HT and non-HT), in relation to nodule's size (largest diameter <1 cm and ≥1 cm), Table S4: Results of multivariate logistic regression analysis in HT and non-HT groups, Table S5: Influence of modifications of the threshold for suspicious nodule's shape (from AP > T to AP ≥ T and AP/T ≥ 1.14 in HT group or AP/T ≥ 1.17 in non-HT group) on values of SEN, SPC, RoM and the percentage of nodules in the high risk category (changes expressed as percentage of values for AP > T threshold).

**Author Contributions:** Conceptualization, D.S.-K.; Methodology, D.S.-K., M.K., K.W.-K., M.W.-N. and B.P.; Project administration, D.S.-K., K.W.-K. and B.P.; Software: M.K.; Supervision, D.S.-K. and M.K.; Validation, D.S.-K. and M.K.; Formal analysis, M.K.; Investigation, D.S.-K., K.W.-K., M.W.-N., B.P., E.W.-O. and M.K.; Data curation, D.S.-K., K.W.-K., M.W.-N., E.W.-O. and M.K.; Writing—original draft preparation, D.S.-K. and M.K.; Writing—review and editing, M.K., M.W.-N. and D.S.-K. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Medical University of Lodz, grant number 503/1-153- 05/503-11-001.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Ethics Committee of Medical University of Lodz (protocol code RNN/151/17/KE approved on 16 May 2017).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding authors. The data are not publicly available due to patient privacy restrictions.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**

## *Review* **Artificial Intelligence in Thyroid Field—A Comprehensive Review**

#### **Fabiano Bini 1,\* , Andrada Pica 1 , Laura Azzimonti 2 , Alessandro Giusti 2 , Lorenzo Ruinelli 3,4 , Franco Marinozzi 1 and Pierpaolo Trimboli 5,6**

**Simple Summary:** The incidence of thyroid pathologies has been increasing worldwide. Historically, the detection of thyroid neoplasms relies on medical imaging analysis, depending mainly on the experience of clinicians. The advent of artificial intelligence (AI) techniques led to a remarkable progress in image-recognition tasks. AI represents a powerful tool that may facilitate understanding of thyroid pathologies, but actually, the diagnostic accuracy is uncertain. This article aims to provide an overview of the basic aspects, limitations and open issues of the AI methods applied to thyroid images. Medical experts should be familiar with the workflow of AI techniques in order to avoid misleading outcomes.

**Abstract:** Artificial intelligence (AI) uses mathematical algorithms to perform tasks that require human cognitive abilities. AI-based methodologies, e.g., machine learning and deep learning, as well as the recently developed research field of radiomics have noticeable potential to transform medical diagnostics. AI-based techniques applied to medical imaging allow to detect biological abnormalities, to diagnostic neoplasms or to predict the response to treatment. Nonetheless, the diagnostic accuracy of these methods is still a matter of debate. In this article, we first illustrate the key concepts and workflow characteristics of machine learning, deep learning and radiomics. We outline considerations regarding data input requirements, differences among these methodologies and their limitations. Subsequently, a concise overview is presented regarding the application of AI methods to the evaluation of thyroid images. We developed a critical discussion concerning limits and open challenges that should be addressed before the translation of AI techniques to the broad clinical use. Clarification of the pitfalls of AI-based techniques results crucial in order to ensure the optimal application for each patient.

**Keywords:** thyroid neoplasm; medical imaging; artificial intelligence; machine learning; deep learning; radiomics; prediction; diagnosis

#### **1. Introduction**

The role of medical imaging in the clinical workflow has noticeably increased from a mainly diagnostic tool up to a central contribution in early detection, diagnosis, treatment planning and monitoring of diseases [1–4]. Medical imaging provides information concerning the characteristics of human tissues in a non-invasive, repeatable manner and

**Citation:** Bini, F.; Pica, A.; Azzimonti, L.; Giusti, A.; Ruinelli, L.; Marinozzi, F.; Trimboli, P. Artificial Intelligence in Thyroid Field—A Comprehensive Review. *Cancers* **2021**, *13*, 4740. https://doi.org/10.3390/ cancers13194740

Academic Editor: Kennichi Kakudo

Received: 31 August 2021 Accepted: 20 September 2021 Published: 22 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

became a routine practice in clinical care [2]. In recent decades, the innovations in this field concerned both devices, i.e., hardware, and analysis tools used in medical imaging. In the clinical practice, the main use of medical images corresponds with qualitative assessment of the anatomical area. Images, in addition, are characterized also by a high quantity of numerical information and recently, a quantitative evaluation has been developed in order to identify possible correlations between the numerical data contained in the digital images and the pathophysiology of the tissue [3]. The quantitative analysis has the aim to achieve information from standard-of-care images, e.g., ultrasound imaging (US), computer tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography (PET), which are not easily quantifiable by means of naked-eye observations for clinical outcomes [5,6].

Analysis of image features in the context of medical imaging is an emerging field of study but extensive literature already exists [7–9]. In the majority of earlier works, the image features are analyzed with the aim of detection and diagnosis of abnormal regions within human tissues [10–12]. These applications are often referred as computer-aided detection (CADe) and computer-aided diagnosis (CADx) systems [3]. The output of the CAD analysis is used by the expert clinicians as a second opinion in detecting lesions or making diagnosis and aims at improving the accuracy of the diagnosis and reducing the time for image interpretation [6].

Recently, a further detailed extension associated with quantitative analysis of medical imagines has led to the emergence of radiomics as a new field of medical research [1,2]. Radiomics aims at extracting numerous quantitative descriptors with the purpose of achieving more useful information of tissue lesion and response of treatment in order to be used for personalized medicine [1,2,13]. It is worth noticing that standardization of the procedure is still under development, as thoroughly discussed in [14].

The above-mentioned approaches apply methodologies from the artificial intelligence (AI) field to achieve a partial or full automation of various steps of the process concerning the analysis of medical images [6]. Thorough understanding of their working principle is necessary in order to develop efficient predictive models and personalization treatment. This review article aims to highlight strengths and limitations of the different AI-based techniques applied for the evaluation of the pathophysiological state of the thyroid.

#### **2. Artificial Intelligence in Medical Imaging**

Artificial intelligence (AI) is a term coined by McCarthy and colleagues [15,16] in 1950s referring to a branch of computer sciences in which mathematical algorithms attempt to perform tasks that normally require human cognitive abilities [8]. Applications of AI have witnessed unprecedented growth in recent decades due to the enhancement of computational power and availability of large dataset. In the medical field, AI can use complex algorithms to develop models with the scope of improving diagnostic accuracy, prognosis, and medical image interpretation [17]. We discuss in the following two different machine learning (ML) methodologies adopted to perform medical imaging analysis.

#### *2.1. Machine Learning*

Machine learning (ML), a term first coined by Arthur Samuel [18], is a field of AI in which the computer is trained to perform tasks by learning from example data and make predictions based on its exposition to previous samples [4]. In medical imaging analysis, ML algorithms are crucial components of both CAD systems and radiomics studies.

ML algorithms are generally divided into supervised and unsupervised learning methods. Supervised learning requires a labelled dataset, i.e., a set of input data with their corresponding output (labels) that is used to identify a function linking inputs to outputs [19]. Unsupervised learning operates on an input dataset without the need of labels. This ML algorithm searches for patterns that can separate input data into subsets with similar characteristics [7]. In this review article, we focus on supervised learning since it is the most common approach applied to medical images analysis [20].

In medical applications, input data include medical images or clinical data, while the output label can be the differentiation of malignant from benign nodules, the classification of images into diagnostic categories or the response treatment, e.g., recurrence, survival. The output of the predictive model leads to a subsequent distinction of ML problems: classification and regression. In classification tasks, the model performs a decision among a small and discrete set of choices, i.e., binary classification, e.g., identifying a tumor as malignant or benign. Regression models refers to the estimation of continuous output variables, e.g., assessment of disease severity [20].

Historically, ML algorithms were applied in CAD systems for classifications purposes [20]. Subsequently, this method was used as a step of radiomics analysis. In this section we describe the workflow of the ML algorithm with classification task frequently encountered in the CAD framework [10,21,22] (Figure 1). A supervised ML model is composed of two phases, i.e., training and application phase (Figure 1a). In the training phase, a set of input images with their corresponding class labels are used to train the predicting model. From the input image, a region of interest (ROI) is delineated manually or semi-automatically by expert clinicians. Subsequently, a set of image features, e.g., morphological and grey level-based features, are extracted. Differently from other methods that will be discussed subsequently in this work, in ML algorithms of CAD systems, the extraction and selection of image features are performed manually by the expert. It represents a crucial step in order to identify the significant variables that can be correlated with the medical endpoint. In CAD applications, the features used in the analysis are those closely associated with what clinicians use in their diagnosis of the lesions [4]. Subsequently, the features are entered as input to the ML algorithm to train the model.

Examples of typical feature-based supervised learning algorithms are logistic regressions, support vector machine, random forests and neural networks [23]. As an example of these feature-based ML algorithms, we focus here on the support vector machine (SVM) method, which is commonly used in biomedical binary classification problems [17,24]. Overall, SVM (Figure 1b) is a binary classifier that aims to identify the decision boundary, or hyperplane, that maximizes the separating margin between two classes [4,25].

For instance, let consider N training samples {(*xi* , *yi*)} *N n*=1 of input features *x* and their corresponding class or label *yi* ∈ {−1;+1} where *yi* = −1 indicates the class with malignant samples and *yi* = 1 indicates the class with benign samples. In the simplest case, there exists a function *f*(*x*):

$$f(\mathbf{x}) = \boldsymbol{\beta} \cdot \mathbf{x} + \boldsymbol{\beta}\_{0} \tag{1}$$

with *β* and *β*0—decision boundary parameters such that *f*(*x*) ≥ 0 for *yi* = +1 and *f*(*x*) < 0 for *yi* = −1.

This means that the training samples from the two classes are separated by the hyperplane *f*(*x*) = *β*·*x* + *β*0 = 0. The margin m, i.e., the distance between a class and the decision boundary, is set to be inversely proportional to the decision boundary parameter, i.e., *m* = 1 k*β*k .

In order to identify the hyperplane that maximizes the separating margin between the two classes, SVM solves the following optimization problem that aims to minimize the cost function *J*(*β*, *ξ*) with respect to *β*, *ξ* [24,26]:

$$\min\_{\beta,\,\xi} J(\beta,\xi) = \min\_{\beta,\,\xi} \frac{1}{2} \left\| \beta \right\|^2 + \mathcal{C} \sum\_{i=1}^{N} \xi\_i \tag{2}$$

subject to the constraints *yi* (*β*·*x* + *β*0) ≥ 1 − *ξi* , *ξi* ≥ 0, *i* = 1, . . . , *N*. In Equation (2), *C* > 0 is a penalty parameter to control the tolerance error *ξi* allowed for each sample being on the wrong side of the margin.

From Equation (2), it can be noticed that the minimization of the parameter *β* increases the separation between the two classes and improves generalizability of the classifier, while minimization of second term of Equation (1) improves fitting accuracy [4].

Subsequently, in the testing phase, the trained classifier is used to characterize new input data with unknown label (test set).

It is worth pointing out that the decision function of the classifier is fully specified by the training set, while the test set is only used to evaluate the performance of the model. On one hand, to obtain a model that well-performs when applied to new data, the training dataset should be sufficiently large. On the other hand, to obtain robust and reliable evaluation of the performance of the model, the test set should be sufficiently large. Frequently, since this condition is difficult to achieve in the medical field by simply splitting the available data in training and test set, a k-fold cross-validation framework [7] is usually adopted. K-fold cross validation consists of partitioning the dataset into k subsets of equal size. The model is trained on (k − 1) datasets while one subset is retained for model test. The process is repeated k times with each subset used once as test dataset [20]. The overall performance of the model is then assessed for example as the average performance over the k repetitions.

**Figure 1.** (**a**) Schematic flowchart of the machine learning model implementation and application for medical images clas- **Figure 1.** (**a**) Schematic flowchart of the machine learning model implementation and application for medical images classification purposes. (**b**) Example of the support vector machine (SVM) classification with a hyperplane that maximizes the separating margin m between the two classes.

*β β* — ≥

−1

{(

∈ −1 −1

() = ∙ + 0

, )}=1

Feature-based ML algorithms are suitable for medical image analysis since predictive models can be developed from small datasets [7]. Moreover, these methods are usually interpretable and can provide insights on the reasons why a certain class is predicted. Nonetheless, some initial steps of the process, as the definition of the features to be extracted from images and the selection of the medical region of interest has to be performed by experts. In addition, it should be taken into account that all supervised ML methods could be affected by overfitting, i.e., the predicting model learns exactly the training set but fails to fit new data from the test set [20]. However, it is possible to mitigate this issue by adopting a cross-validation set-up and by reducing the number of features used by the model by means of feature selection methods.

#### *2.2. Deep Learning*

Deep learning (DL), a term coined in 1986 by Rina Dechter [27], is a new class of ML methods developed through the advancement of artificial neural networks which were considered as artificial representations of the human neural architecture [23]. DL relies on networks of computational units, i.e., neural units arranged in layers that gradually extract higher level features from input data, e.g., image. These structures learn discriminative features from data automatically, allowing to approximate complex nonlinear relationship with outstanding performance [27,28]. Differently from traditional feature-based ML approaches, DL is able to achieve diagnosis automation, avoiding human intervention [29]. In medical applications, DL algorithms are implemented for detection and characterization of tissue lesions as well as for the analysis of disease progression [27,28].

While several DL architectures have been developed, this article focuses on convolutional neural networks (CNNs), introduced by LeCun [30]. CNNs are typically applied for image recognition and computer vision applications because they preserve spatial relationships in 2D data, and therefore outperform other architectures on image pattern recognition. More specifically, the input of a CNN is arranged in a grid structure and processed through convolution and pooling layers that preserve these relationships. The final layers are typically fully connected and can be conceived as a multi-layer perceptron classifier on the features automatically extracted by the convolutional part. The network is trained to identify patterns in a set of labelled training data and the outputs are compared with the actual labels. During training the network parameters are tuned until the patterns identified by the network represent good predictions for training data. The network is then used to make predictions on new data in the test set [31].

Figure 2 shows a typical architecture of CNN developed to perform classification tasks. The input of the CNN algorithm is represented by numerical data of the selected ROI from the medical image. Firstly, a convolutional step is considered which contains a set of filters, e.g., k1 in Figure 2. Thus, a convolution is performed between each filter and the input of the layer, e.g., image data. A convolution is a space-invariant linear operation on 2D grids and is equivalent to applying a filter to an image. The filter slides over the input image, its values are multiplied with the image pixel values and then summed to determine the value in the corresponding position of the output feature map. An example of a convolution operation is reported in Figure 3a. The number and size of filters are CNN hyperparameters and are typically not optimized during training. More and larger filters lead to more powerful network with more parameters to optimize, which increases the risk of overfitting [32]. The convolutional process in every convolutional layer is expressed mathematically as follows:

$$X\_k^\ell = \sigma \left( W\_k^{\ell-1} \* X^{\ell-1} + b\_k^\ell \right) \tag{3}$$

where *X* ℓ *k* is the new feature map, σ(·) is an element-wise nonlinear activation function, *W* is the filter values, *b* ℓ *k* is a bias parameter and the symbol ∗ indicates a convolutional operator.

 ℓ is the new feature map, σ(∙) is an element

 ℓ ℓ = (

ℓ−1 ∗

ℓ−1 + ℓ )

**Figure 2.** Schematic flowchart of the deep learning model implementation and application for medical images classification purposes.

**Figure 3.** (**a**) Numerical example of the functions that compose the CNN architecture: (**a**) convolution, (**b**) max pooling, (**c**) flattening, (**d**) softmax function.

() = max (0, )

Subsequently, an activation function is applied element-by-element to the calculated output of the convolution prior to using the map as an input to the next layer of the network. Rectified linear unit (ReLU) is one of the most used activation functions, and has been empirically found to accelerate the convergence of the learning procedure [28]. It is linear for positive inputs, mapping them unchanged to the next layer, while it blocks negative values. Mathematically, ReLU is expressed as follows [28]:

$$f\left(\mathbf{x}\right) = \max(0, \mathbf{x})\tag{4}$$

where *x* is an activation value achieved from the previous layer.

Some CNN architectures also consider pooling operations, whose effect is to downsample the feature maps. This operation considers small regions of the input map and outputs a single number for each region, e.g., the maximum value as illustrated in Figure 3b. It reduces the dimensions of the feature map and decreases the number of pixels to be processed in the next layers of the network [33]. Conceptually, as we progress deeper in the network, neuron activation values represent progressively higher-level and larger-scale visual patterns in the input, and therefore require lower spatial resolution.

The final part of the CNN architecture is characterized by a fully connected layer, i.e., each neural unit of the actual layer is connected to every neural unit in the successive layer (Figure 2). Firstly, the feature map is flattened into a column vector (Figure 3c) and then connected to one or more fully connected layers. The output nodes of the last fully connected layer can be regarded as a vector of unnormalized probabilities [28].

The softmax function is a function applied to the last fully connected layer of the CNN in order to transform the k real values of the vector into values in the range (0;1) so that can be assumed as probabilities (Figure 3d). The relation is as follows [28,33]:

$$
\sigma(z)\_i = \frac{e^{z\_i}}{\sum\_{j=1}^K e^{z\_j}} \tag{5}
$$

where the *zi* values are the elements of the fully connected layer and the denominator represents the normalization term.

The output layer of the CNN considered is constituted by neural units which indicate the probabilities for each class.

The analysis of the available literature shows an increasing interest on applying DL architecture for medical image analysis. It is worth mentioning that for systems in which the set of visual features is well defined, simpler feature-based ML techniques, such as SVM algorithms, are easier, more interpretable and more effective [28].

The main limitation to the use of DL consists of the large datasets required to train the model [34]. Compared with publicly available datasets in other areas, the current availability of medical US datasets is still limited [34]. To face the data requirements, several studies [33,35] considered pre-trained CNN architectures developed with trainings on ImageNet, a large labelled collection of low-resolution color photographs. To date, DL architectures pre-trained on high resolution medical images are not available. Therefore, a large dataset of medical images is a mandatory step to enhance CNNs performance [34].

#### **3. Radiomics**

Radiomics is an emerging field that uses automated high-throughput extraction algorithms to achieve large amounts (200+) of quantitative features from medical images [1,2]. Radiomics is also indicated as quantitative imaging [36] which can be applied to any image generated in the clinical setting. It can be performed on subregions of a tumor, metastatic lesions and in normal tissues. The term feature represents a descriptor of an image, of tumor or healthy tissue, such as parameters derived from image grayscale intensity or shape [37].

Radiomics has its roots on computer-aided diagnosis systems [38], although methodological workflow and applications are distinct [2]. It concerns the extraction of quantitative

features from medical images that subsequently are related to biological endpoints and clinical outcomes [39]. Radiomics makes use of digital data stored in those images to develop diagnostic, predictive or prognostic models to support clinical decisions and optimize personalized treatment planning. The main difference with CAD systems consists of the relationship that radiomics has to identify between the current characteristics of the tissue lesion and its temporal evolution in the perspective of a personalization of the therapy [38].

Radiomics involves several processes, each with its own critical aspects that need to be taken into account. Two workflows can be implemented to perform radiomic studies in function of the AI technique adopted (Figure 4): (i) conventional or ML-based radiomics where the features to be extracted are predefined and (ii) DL-based radiomics where the features are not predefined but automatically extracted from the underlying data [6,7].

**Figure 4.** Schematic flowchart of radiomics approach. X represent the feature extracted from the image data. **Figure 4.** Schematic flowchart of radiomics approach. Xi represent the feature extracted from the image data.

– The main aspects of the conventional radiomics workflow concerns: image acquisition, data selection, feature extraction and selection and the development of predictive model [1,36]. From medical image such as US, CT, MR and/or PET images, the region of interest (ROI) is selected and subsequently the lesion is manually segmented, i.e., delineated with computer-assisted contouring, by an experienced clinician [7]. Subsequently, image data undergoes preprocessing operations, e.g., gray-level discretization, which enable a higher reproducibility of results [6]. The extraction of quantitative imaging features involves descriptors of spatial relationships between the various intensity level, heterogeneity patterns, shape and relations of the tissue lesion with surrounding tissues. A feature selection procedure is then performed to identify the most relevant predictive features [7,24]. The collection of features which hold prognostic or predictive value represent a feature signature, frequently indicated also as quantitative imaging biomarkers. The selected features are then analyzed to develop classified models to predict outcomes either alone or in combination with additional information, such as demographic, clinical, comorbidity or genomic data [1,3].

Segmentation represents a crucial subprocess of radiomics since many extracted features may depend on the segmented region. In several radiomics studies the ROI is manually delineated by experts [21,40–42]. A number of algorithms has been developed for semi-automatic segmentation [22]. Region growing-based algorithm and grey-scale threshold-based methods are frequent techniques applied for ROI definition. However, manual delineation by an expert is considered the gold standard though is subjected to interobserver variability and is a time-consuming task [37]. To avoid possible bias, evaluation by multiple clinicians or a combination of multiple algorithms could be considered [43].

Typically, radiomics features are divided into [2,6,44]:

Accurate definitions of radiomics features are provided in the image biomarker standardization initiative (IBSI) [14].

The radiomic features are subjected to a subsequent feature selection to prevent overfitting, improve learning accuracy and reduce computation time. The selection process should eliminate unreliable, not informative or redundant features. The selection methods can be divided into three classes: (i) filter methods which asses the usefulness of a given feature with various statistical tests for their correlation with the outcome variable [2,7]; (ii) wrapper method which uses an external classifier algorithm to score different subsets of features based on their classification performance; (iii) embedded method where the selection is intrinsic to the model training, i.e., features are selected to optimize the performance of the implemented learning algorithm. Filter methods are simple and computationally efficient, but consider features as independent and any interaction between them is ignored [24]. Wrapper methods reduce the risk of overfitting but are computationally intensive [7,24]. Embedded methods are computationally more efficient since the selection procedure is part of the training process [7,24]. A frequent embedded algorithm with good performance used in radiomics studies is the least absolute shrinkage and selection operator (LASSO) [7,24].

Subsequently, the selected features are used to implement a mathematical model in order to predict the established medical endpoints. Regarding the choice of modelling methodology, the identification of a suitable method depends on several factors as sample size or study endpoint [36]. It is advantageous to include in the model information beyond radiomics, e.g., clinical data and/or other "-omic" information, e.g., genomic data [45]. The integration of data from multiple sources, e.g., medical imaging, disease risk factors, therapy procedures and follow up data, in the mathematical model will facilitate the development of a personalized treatment.

As previously mentioned, the target of the radiomics studies can be either a present characteristic, e.g., tumor phenotype, or a future prediction, e.g., treatment response. Usually, radiomics studies make use of the feature-based ML algorithms that are also considered in CAD systems. By means of feature-based ML methods, the relationship

between input data, e.g., selected radiomics features and target outcome, is determined by means of training examples. SVM is one of the most successfully applied algorithms.

DL-based radiomics allows to automatically extract imaging features and achieve the predicted outcome. In fact, the different components of the DL architecture perform all the processing steps described in the ML-based model, including feature extraction, selection and predicting model implementation. CNNs is the most common architecture used in radiomics studies and its characteristics have been previously described in Section 2.2.

Validation is a crucial component of the workflow of both conventional and DL-based radiomics. Ideally, the trained model should be tested in cross-validation or on an external, independent dataset before being applied on the new dataset [38].

#### **4. AI and Radiomics in Thyroid Diseases**

Ultrasound imaging is the recommended method for early detection and diagnosis of thyroid lesions due to its economy, effectivity and absence of radiation [46–49]. It is widely accepted as the first imaging modality for thyroid disease, for instance by American and European associations of endocrinology [50]. AI applications in the medical field are of increasing interest since they represent a possible approach to reduce the number of invasive clinical procedures [36].

Mainly, AI algorithms have been implemented for the classification of thyroid nodules, i.e., differentiating among benign or malignant state [9,10,21,22,33,41,51–56]. The outcomes of these studies are compared with the diagnosis of radiologists with different levels of experience. Research comparing the diagnostic ability between feature-based ML and DL algorithms is limited in the literature, but interesting outcomes are provided in [22]. Overall, an improvement emerged in terms of both specificity and accuracy in DL studies [57,58] with respect to feature-based ML classical applications [22], mostly determined by the capacity of DL of capturing complex patterns. In some studies [57–59], DL algorithms show accuracy values in line with those of radiologists. In addition, Jin et al. [20] also pointed out that the use of AI algorithms was useful to junior radiologists allowing a noticeable improvement of their diagnostic performance, reaching values of accuracy similar to those of intermediate-level radiologists. Studies of interest concerning the application of featurebased ML methods and DL algorithms are described in Tables 1 and 2, respectively. Tables were organized according to the publication time, in a decreasing order.

**Table 1.** Machine learning (ML)-based studies.

Abbreviations: US—ultrasound; SVM—support vector machine; N/A—not available.

#### **Table 2.** Deep learning (DL) studies.

Abbreviations: US—ultrasound; CNN—convolutional neural network; N/A—not available.

Radiomics is considered a promising method to be encompassed in the pipeline of precision medicine on the basis of specific characteristics of the patient [2]. Whilst the first AI approach to the medical imaging, i.e., CAD system, is focused on the differentiation among benign and malignant thyroid lesions, radiomics extends the analysis to prognosis and response to treatment evaluation [1]. In fact, [42,60,61] implemented radiomics models that analyze the risk stratification and predict the aggressiveness of the thyroid carcinoma with high values of accuracy, i.e., roughly 85 percent. Radiomics analysis has the potential to determine tumor phenotypes or the presence of gene mutations [62,63]. Furthermore, several studies have investigated by means of radiomic features the occurrence of metastases [64] or disease-free survival [65]. It also emerged that radiomics studies aimed at performing classification tasks regarding the nature of thyroid nodules are characterized by minor accuracy with respect to classical ML approach [66]. It is worth pointing out that although radiomics has been applied for several anatomical areas, research concerning thyroid lesions is relatively limited. Studies of interest concerning radiomics applications for thyroid lesions are described in Table 3, organized according to the publication time, in a decreasing order.

#### **Table 3.**Radiomics studies.

Abbreviations: US—ultrasound; MRI—magnetic resonance imaging; CT—computer tomography; ML—machine learning; DL—deep learning; N/A—not available.

#### **5. Discussion**

Medical images provide a comprehensive view of the tumor and its environment, and they can be used to improve the diagnostic accuracy of early lesions, to classify benign from malignant tissues and to define risk and improve therapy [43,68]. Imaging is a non-invasive method and with no risk of the infections or the complications that accompany biopsies [2]. In recent decades, images have been converted into quantitative data and subsequently analyzed with AI tools.

Intratumoral heterogeneity and modifications over time are common features of neoplasms [43]. Samples of tumor acquired through biopsy may fail to represent the variations within the tumor. In addition, AI methods, analyzing the overall image of the lesion, have the potential to capture tumor heterogeneity and could represent an intermediate step between imaging and biopsy [28,36]. Nonetheless, it is worth pointing out that AI systems learn on a case-by-case basis. AI algorithms are implemented considering gold standards of pathological diagnosis that are hard to identify in every patient, due to inter-variability among subjects. Moreover, as it emerged from the overview of the AI methods, the predicting model is developed on the basis of a finite training dataset. Thus, since human tissues are characterized by high heterogeneity and variability inter- and intrasubjects, no finite training set can fully represent the variety of cases that might occur in the clinical practice. Extensive research is still required to improve the generalizability and accuracy of AI-based models. From this perspective, the standalone use of AI applications for diagnosis should be still avoided in the clinical practice. In fact, to this date, several studies [7,20,28,43] recommend that the lesion evaluation should be achieved from a combination between the clinician evaluation and ML or DL outcome. Moreover, it is worth noticing that most AI-based studies focused on thyroid pathologies are performed using retrospectively collected data [9,11,33,40,42,51,55,60–63,65–67]. Conversely, studies that prospectively evaluate AI predictive models concerning thyroid disease diagnosis are limited in the literature [22,41]. In retrospective studies, cohorts are selected among patients with definitive diagnosis achieved mainly through histopathological examination. As highlighted by Wu et al. [69], evaluations should include more prospective studies on medical AI models to reduce risk of overfitting and enhance accuracy of the clinical outcomes.

AI methods are based on the analysis of image features in order to develop predictive models. Differentiating benign and malignant thyroid nodule is mainly achieved from ML-based studies. The most used US features adopted by ML algorithms for thyroid investigations were size, shape, margin, composition echogenicity, as defined by the thyroid imaging reporting and data system (TI-RADS) classification [10,21,22,51]. According to an analysis of the available literature, the TI-RADS approach allows a good discrimination among benign and malignant thyroid nodules. However, the inclusion of additional features, e.g., calcifications, internal content, can represent a factor that improves accuracy [70].

Radiomics studies were applied also to other thyroid pathologies, e.g., extrathyroidal extension (ETE) in patients with papillary thyroid carcinoma (PTC) [42,61], thyroid cartilage invasion from laryngeal and hypopharyngeal squamous cell carcinoma [64]. In these studies, the extracted features derive from morphological, first order statistics, textural and higher order statistics groups. Wang and colleagues [42] highlighted that improvement of ETE diagnosis is achieved when features related to PTC heterogeneity are taken into account. Similarly, in [64] Guo et al. studied thyroid cartilage invasion from laryngeal and hypopharyngeal squamous cell carcinoma and showed that tumor invasiveness can be investigated considering features related to tumor heterogeneity. Furthermore, Kwon et al. [62] highlight that BRAF mutation may be investigated with histogram-based and textural features that reflect echogenicity and heterogeneity of the region of interest, respectively.

Several studies also performed comparison between the performance of AI-based models and that of expert clinicians. The available data in literature mostly report that the performance of DL algorithms is similar to that of healthcare professionals. As discussed by [20,67], AI applications may improve the accuracy of thyroid diagnosis diseases, especially for junior radiologists. In fact, interpretation of medical images highly depends on the experience level of clinicians. For instance, for junior radiologists the sensitivity is reported in a range between 40 percent and 100 percent while the specificity spans between 50 percent and 100 percent. It was observed that the use of AI algorithms to achieve a second opinion on the characterization of thyroid lesions can improve the accuracy of junior radiologists from roughly 82 percent to 87 percent [67]. Moreover, Peng and coworkers [67] highlighted that taking into account the outcomes of AI as a second opinion has reduced fine needle aspiration procedures by 27 percent and the number of missed malignancies of roughly 2 percent.

Furthermore, the experience level of the clinicians has an important impact also on the performance of the AI-based methods. The input data of the AI algorithms is the ROI selected by the expert. It is commonly accepted that image acquisition and segmentation are critical subprocesses due to inter-operator variability. Recent studies [8,28] suggest that semi- or fully automated methods could improve algorithm performance, but currently the manual segmentation performed by experts continues to be the main method adopted. For instance, most of the ML-based studies applied to the thyroid are performed considering a manual segmentation of the ROI [21,41]. In addition, the ML-based investigations reported in [10,22] have introduced a semi-automatic method that is characterized by an initial automatic selection of a box region and subsequently by a manual contouring performed by expert clinicians. Conversely, the studies that applied DL algorithms to thyroid imaging considered a manual selected box around the region under investigation [9,11,52,54]. Furthermore, it is worth pointing out that radiomics studies are based on a manual contouring along the borders of the thyroid tumor [60–62] or slightly within the borders of the tumor to avoid artifacts [64].

To date, most studies highlight that the main limitation of AI algorithms is the reduced dataset used for predictive model development and validation. Ideally, independent training and validation datasets, composed of data images achieved with different US equipment and from multiple centers, i.e., multicenter training cohorts, allow to optimally develop the predicting model, avoiding overfitting and enhancing generalizability and model performance [67].

For instance, in radiomics studies, Gilies and coworkers [43] provide an empirical rule concerning the size of the dataset in order to avoid overfitting. It is suggested that almost 10–15 patients are needed for each examined radiomic feature. Thus, also features selection represents a crucial step during the evaluation.

AI methods represent a powerful approach that in future may assist clinicians in diagnostic decisions [22,71], while combined with other "-omic" data as occur in radiomics analysis may improve the risk factor analysis for personalized estimation of disease-free survival. As mentioned, AI methods could be also applied to contribute to treatment planning. For instance, radiomics combined with other clinical parameters may help to predict which patients are likely to have a satisfactory response to emerging therapies as high-intensity focused ultrasound (HIFU), that allows the thermal tissue treatment and the consequent reduction in thyroid nodule volume by directing energy inside the target zone with non-invasive instruments [72–74].

Several efforts are performed to increase the availability of open access database of labeled medical images that will help to train the predictive models developed with AI techniques. However, pitfalls and limitations associated with the AI approach should be considered, especially related to the difficulty to achieve a generalizable model in order to ensure optimal application for each patient.

With regard to the application of the AI in the daily practice of the clinical medicine, beyond the hype around these technologies, the financial investment is pouring and

brand-new products started flowing into the market. As of early 2020, there were 64 FDAapproved AI-ML medical device and algorithms, many of which are already integrated into clinical care. Remarkably, 21 were related to Radiology [75]. Nonetheless, recent literature reviews report that the impact is still minimal as the majority of the AI-ML studies are retrospective in nature, deviate from existing reporting standards and often outline proof-of-concept approach [76].

From the pure clinical standpoint, all these findings should be interpreted according to the routine clinical practice. In fact, US is recognized as the most relevant imaging procedure for the assessment of thyroid nodule and almost all thyroid patients are managed according to US features of their thyroid gland. This worldwide diffused approach is based on the high sensitivity and specificity of US in discriminating malignant from benign thyroid lesions. Further improvement of US performance by AI remains however desirable [77,78]. In addition, a not negligible number of thyroid goiters are incidentally discovered during other imaging evaluations (i.e., CT, MR, PET/CT) of patients with non-thyroid indication [79]. While the performance of these imaging procedures is poor or suboptimal to identify malignant and benign nodules among adrenal thyroid incidentalomas, a significant effort should be made in the future to improve their capability to initially select patients requiring an urgent or not endocrinological evaluation combined with in-office US examination.

#### **6. Conclusions**

The evaluation of images has a central role in the clinical workflow. It is worth highlighting that image interpretation requires deductive reasoning, using knowledge of pathological processes, integration from prior examination and investigations and consultation with other physicians. To date, AI techniques can be an integral part of the procedure, but cannot emulate the overall process.

A further approach to improve the assessment of medical images can be represented by the integration of AI-based models with mixed reality tools. The authors retain that in-depth analysis should be performed to analyze the potential of mixed reality within the diagnostic workflow.

**Author Contributions:** Conceptualization, F.B. and P.T.; methodology, F.B., A.P., L.A., A.G., L.R., F.M. and P.T.; software, A.P.; validation, F.B., A.P., L.A., A.G., L.R., F.M. and P.T.; formal analysis, F.B. and A.P.; investigation, F.B., A.P., L.A., A.G., L.R. and P.T.; resources, P.T.; data curation, F.B. and A.P.; writing—original draft preparation, F.B. and A.P.; writing—review and editing, F.B., A.P., L.A., A.G., L.R., F.M. and P.T.; visualization, F.B. and A.P.; supervision, F.B., F.M. and P.T.; project administration, P.T.; funding acquisition, P.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Facing Thyroid Nodules in Paediatric Patients Previously Treated with Radiotherapy for Non-Thyroidal Cancers: Are Adult Ultrasound Risk Stratification Systems Reliable?**

**Arnoldo Piccardo 1 , Francesco Fiz 1, \*, Gianluca Bottoni 1 , Camilla De Luca 2 , Michela Massollo 1 , Ugo Catrambone 3 , Luca Foppiani 4 , Monica Muraca 5 , Alberto Garaventa 6 and Pierpaolo Trimboli 7,8**

**Simple Summary:** The risk of thyroid nodules harbouring cancer has been evaluated, in adults, using specific ultrasound criteria. However, it is unclear whether such evaluation can be translated in paediatric patients. In this study, we tested the effectiveness of three known risk evaluation systems in children with thyroid nodules and with a history of radiation exposure. We found that these systems are reliable in confirming or ruling out cancer in most cases, except when evaluating very small nodules (<1 cm). For these reasons, these risk criteria should be adopted to account for the reduced size of malignant lesions when evaluating paediatric subjects.

**Abstract:** Thyroid nodule ultrasound-based risk stratification systems (US-RSSs) have been successfully used in adults to predict the likelihood of malignancies. However, their applicability to the paediatric population is unclear, especially in children with a history of radiation exposure, who are at a higher cancer risk. We tested the efficacy of three US-RSSs in this setting by retrospectively applying three classification systems (ACR-TIRADS, ATA and EU-TIRADS) to all paediatric patients referred for thyroid nodules and with a radiation exposure history. We compared the results with a reference standard (pathology or 36-month follow-up); sensitivity, specificity, positive and negative predictive values (PPV and NPV) and accuracy were calculated. A total of 52 patients were included; fourteen of them (27%) had papillary thyroid cancer (PTC) at the final histology. No significant differences across the US-RSSs were detected; specificity (range 95–97%) and NPV (range 88–93%) were particularly elevated. However, ACR-TIRADS, ATA and EU-TIRADS did not indicate the need for a biopsy in six (42.8%), seven (50%) and eight (57%) cases of PTC; in five cases, this lack of indication was due to a small (<1 cm) nodule size. In conclusion, US-RSSs show a high NPV and specificity in paediatric patients, whereas the cytology indication could be improved by reconsidering the dimensional criterion.

**Keywords:** thyroid nodules; paediatrics; radiotherapy; risk assessment; ultrasonography; DTC

#### **1. Introduction**

Thyroid nodules are fairly uncommon among paediatric subjects [1]. However, whenever a thyroid lesion is identified in children and teenagers, it does bear a higher likelihood

**Citation:** Piccardo, A.; Fiz, F.; Bottoni, G.; De Luca, C.; Massollo, M.; Catrambone, U.; Foppiani, L.; Muraca, M.; Garaventa, A.; Trimboli, P. Facing Thyroid Nodules in Paediatric Patients Previously Treated with Radiotherapy for Non-Thyroidal Cancers: Are Adult Ultrasound Risk Stratification Systems Reliable?. *Cancers* **2021**, *13*, 4692. https:// doi.org/10.3390/cancers13184692

Academic Editor: Stefan Delorme

Received: 31 August 2021 Accepted: 16 September 2021 Published: 18 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

of malignancy, which can be as high as 20–25%, when compared with the adult counterpart (5%) [2,3]. Some risk factors may increase the probability of developing thyroid nodules in children, including iodine deficiency, prior radiation exposure and several genetic syndromes.

In particular, childhood cancer survivors who were treated for their non-thyroidal primary malignancy with radiation therapy (RT) represent a population at risk. This group includes survivors of Hodgkin lymphoma, leukaemia, neuroblastoma and central nervous system tumours [2,4,5]. In fact, the history of malignancy and the radiation exposure can represent synergic factors for the development of a second malignant neoplasm, particularly differentiated thyroid cancer (DTC) [6–8].

Neck ultrasonography is the first-line imaging procedure, which is able to identify and classify the risk of the thyroid nodules [9–11]. Adult-based neck US risk stratification systems (RSSs) have been developed in recent years to integrate the US features in an effort to improve diagnostic accuracy and as an aid in the stratification of the risk of malignancy [9,12,13].

However, few studies are available about the reliability of these systems in paediatric age, and conflicting results have been reported about the accuracy of the adult US-RSSs.

In particular, little can be said about the efficacy of US-RSSs when it comes to stratifying the risk of malignancy in patients with a history of previous radiotherapy for oncological reasons who need a strict follow-up after the identification of thyroid nodules [7,14]. This could bear particular relevance considering that an early identification of DTC could avoid a more advanced presentation that, in paediatric patients, can imply extrathyroidal extension and metastases.

The most recent guidelines seem to be concordant in considering those patients at high risk of developing DTCs with thyroid nodules and a previous history of irradiation regardless of the neck ultrasonography features and dimensions. Indeed, while fine-needle aspiration cytology (FNA) can be generally recommended in adults with nodules sized at least 1 cm, in the paediatric population, this procedure is indicated even for small nodules [9,15]. Current US-RSSs do not include young age and previous history of RT as risk factors [16]. Indeed, whether the RSS reliability is concordant with the one that is generally reported remains to be clarified [17].

The aim of our study was to: (1) evaluate the diagnostic performance of the principal neck US classification systems (ACR-TIRADS, ATA and EU-TIRADS) in a selected paediatric population of patients previously treated with radiotherapy, (2) test the malignancy prevalence of each category delineated by US-RSSs and (3) evaluate whether these neck US systems are able to correctly select nodules for FNA.

#### **2. Materials and Methods**

We retrospectively analysed all paediatric patients consecutively referred to our centre (Galliera Hospital) for FNA of a thyroid nodule between 1 January 2012 and 31 December 2017. Before FNA, all patients underwent thyroid US and were tested for TSH, free-T4, free-T3 and calcitonin. Additionally, thyroid scintigraphy (TS) was performed only in the case of suppressed TSH levels [2]. Patients were excluded only if US data had not been retrieved in the local picture archiving and communication system (PACS). Then, only patients with thyroid nodules and previously treated with radiotherapy (RT) for primary paediatric non-thyroidal tumours were included in our study. The institutional review board (Comitato Etico Regionale Liguria, Registration Number: 326/2020-DB id 10315) approved this retrospective study.

#### *2.1. Neck Ultrasonography*

Thyroid US was performed using a LOGIQ S8 (General Electric Medical Systems) with a 9 to 15 MHz linear probe. All imaging procedures were performed in combination with a clinical visit by 3 expert physicians (A.P., G.B., M.M.). For all patients, a greyscale and colour Doppler imaging data were acquired.

#### *2.2. Ultrasound Risk Stratification Systems*

All thyroid nodules were retrospectively risk stratified according to the principal US-RSSs (i.e., ACR-TIRADS, ATA and EU-TIRADS). Indications for FNA were ascertained depending on risk classes identified by each US-RSS. More in general, FNA could be indicated depending on US features and nodule dimensions.

#### *2.3. Imaging Review and Interpretation*

Neck US images we retrieved from the PACS and then visually analysed by 2 reviewers (AP, PT) unaware of patients' data and final outcome. The inter- and intra-reader variabilities in identifying the classes of each US-RSS were previously tested in a different set of 30 paediatric patients with thyroid nodules and showed excellent agreement (Cohen's κ, 0.82 [95% CI, 0.68–0.91]). In case of interpretation disagreement, the final diagnosis was achieved after a consensus meeting with a third expert (GB).

For statistical purposes, each thyroid nodule with US features corresponding to the last class of each US-RSS (i.e., EU-TIRADS 5, ACR-TIRAD5 and ATA High-risk) was regarded as positive. Prevalence of malignancy was calculated as the percent of nodules in each class that were confirmed as DTC at the final histology.

#### *2.4. Reference Standard*

Cytology according to the Italian consensus of cytopathology was adopted as the gold standard. The first edition, used until 2014, considered 5 classes, with a single indeterminate category, while the second edition included 6 classes, of which 2 were indeterminate [18,19]. However, for all these patients, a US follow-up of at least 36 months was available. In case of surgery, histopathology of the resected nodule represented the standard of trough.

#### *2.5. Statistical Analysis*

Sensitivity, specificity, positive and negative predictive values (PPV and NPV) and accuracy were calculated for each system. Differences in categorical variables between groups were analysed using the chi-square test or Fisher's exact test as appropriate.

The prevalence of malignancy was calculated as the ratio between the number of DTCs in each class and the total number of DTCs.

The proportion of FNA that would not have been indicated by the various systems in patients with a diagnosis of DTC was compared using the pairwise chi-square test.

#### **3. Results**

During the study period, we evaluated 259 paediatric patients with thyroid nodules who had undergone neck US at our department. Out of these patients, 52 were selected for the present study according to our inclusion criteria (Figure 1), and their principal characteristics are summarised in Table 1.

**Figure 1.** Flow chart illustrating the selection of the patients.

≥

≥

≥

**Table 1.** Characteristics of the patients.

**Table 1.** *Cont*.

Legend: \* According to the Italian Consensus Working Group [reference]. \*\* This feature included all histopathological findings and pre-surgical imaging. IQR: Interquartile range, RT: radiation treatment, DTC: differentiated thyroid carcinoma.

Among these 52 patients, 19 underwent surgery because of a symptomatic nodular goitre (n = 3), indeterminate cytology (n = 4) and cytology suspicious (n = 3) or consistent with DTC (n = 9). Finally, 14 papillary cancers (27%) were histologically confirmed (Table 1).

The diagnostic performances of each US-RSS in terms of sensitivity, specificity, positive and negative predictive values (PPV and NPV) and accuracy in identifying DTCs are summarised in Table 2. No significant differences across these systems were observed.

**Table 2.** Diagnostic performances of the US-RSSs.

Legend US-RSSs: Ultra-sound risk stratification systems, ACR: American college of radiology. TI-RADS: Thyroid imaging reporting and data system, EU: European thyroid association, ATA: American thyroid association, NPV: negative predictive value, PPV: positive predictive value.

> When benign nodules were evaluated according to the three US-RSSs, the principal classes in which they were included were EU-TIRADS 3 by EU-TIRADS, TR3 by ACR-TIRADS and "Low suspicion" by ATA.

> When PTCs were considered, the most represented classes were EU-TIRADS 5 by EU-TIRADS (71.4%), TR5 by ACR-TIRADS (71.4%) and "High suspicion" by ATA (64.2%).

> A statistical comparison showed no significant differences in the benign lesion and PTC distributions among the three systems (Figure 2).

> At evaluation of the risk of malignancy of each category of the various US-RSSs, we found that EU-TIRADS 5, TR5 and ATA High suspicion presented a DTC percentage of 71, 71 and 64, respectively (Table 3). Conversely, no DTCs were found in the lowest categories.

> When FNA indication in patients subsequently diagnosed with papillary thyroid cancer on histopathology was analysed according to the US-RSSs, EU-TIRADS missed eight cases (57%), ACR TIRADS six (42.8%) cases and ATA seven cases (50%) (Figure 3). The lack of indication for FNA was principally related (five patients) to the small dimensions (<1 cm) of the malignant thyroid nodules.

**Figure 2.** Distribution of benign and malignant thyroid nodules among the categories of the three US-RSSs.

**Table 3.** Frequency of malignancy according to the various US-RSSs.

#### **4. Discussion**

The pivotal role of neck ultrasonography in the identification and evaluation of the malignant potential of thyroid nodules, as well as in FNA guiding, has been recognised in the most recent paediatric guidelines [20,21]. Strikingly however, no specific US features to tell apart benign from malignant nodules have been identified, and no dedicated scoring system has been proposed [2]. Some papers have investigated the role of US-RSSs in the paediatric population with conflicting results [22,23]. To our knowledge, this is the first study testing US-RSSs in a selected population of paediatric patients with a history of neck radiation exposure.

In our population, a little less than one third of the radiotherapy-treated patients had developed a DTC; this result is well in line with the data reported in the existing literature [24,25].

Moreover, in this particular setting of patients in which FNA is "a priori" indicated due to the high prevalence of malignancy, we showed that the different US-RSSs can rule out DTCs, having a high NPV ranging from 89 (ATA) up to 91% (ACR TIRADS). Indeed, this finding, similar to that reported in a recent paper comparing the two American systems (i.e., ATA and ACR TIRADS) [23], may have a particular impact on the ability to monitor these subjects at increased risk with reliable, non-invasive procedures. In particular, by contributing to sparing futile and repeated invasive procedures, it can help in reducing stress and anxiety for patients who have been previously heavily pre-treated for non-thyroidal cancer.

We found that, by considering the highest category of each system as positive and the remaining ones as negative, the specificity is very high (from 95 to 97%), with a very low number of false positive US findings. Our data, in this regard, are concordant with a recent meta-analysis by Kim et al. showing that, by using the same interpretation, the pooled specificity of ACR TIRADS in paediatric patients is 97% [26]. On the contrary, the prevalence of DTC within the thyroid nodules classified in the highest categories is even higher than that reported by Kim and colleagues, ranging from 64 for ATA to 71% for ACR TIRADS, without significant differences among the US-RSSs. Generally, we found that all US stratification systems are reliable methods to identify DTCs, and that their diagnostic performances are adequate and higher than those reported by the meta-analysis by Kim et al. [26]. This discrepancy could be related to the higher prevalence of malignancy in the irradiated population and to the histopathology which is exclusively papillary thyroid cancer (PTC). Indeed, a large part of the studies considered in this meta-analysis [22,23,27] excluded patients with a history of radiation exposure and included patients with thyroid cancer other than PTC (10%) for which the US-RSSs are often not reliable enough [17,28].