of using opaque deep learning models at the cost of explainability are not
convincing and the community seems to acknowledge this in modalities such as
text and EHR where the majority of models are IIM or decision-based
classifiers like XGBoost. The approach taken, thus, is to not have to
compromise significantly between the trade-off but rather make inherently
interpretable models that can also be high-performing. It is difficult to beat
XGBoost and its cousins in tabular machine learning, even with deep learning
models like TabNet and NODE. This approach should be extended to other
modalities as well.

Some possible ideas for adoption of more rigorous testing of applied
explainability in clinical risk prediction modelling is to test the methods on
synthetic datasets with known underlying generative factors. There are already
some examples of such datasets extracted from existing and in-use clinical
datasets like the Clinical Practice Research Datalink (CPRD) containing EHRs
of millions of patients in the United Kingdom or UK Biobank which similarly
contains a large amount of relatively complex and diverse patient data [262].
Testing that the implemented XAI methods correctly identify most of the known
factors is an important reliability test before being proposed to be used in
clinical decision-making systems. It would add further trust when negotiating
with clinicians regarding uptake. In our review of EHR time-series
applications, we highlighted recent developments in implementing quantitative
evaluations using a combination of corruption and ranking solutions based on
adapted AUROC and F1 scores. More work still needs to be done on suggesting
further frameworks for robust evaluation and comparison of interpretability or
explainability more broadly in EHR and other clinical risk prediction domains.

While explainable AI holds great potential for revolutionizing clinical risk
prediction, it is essential to recognize that it is not a panacea. Human
expertise and domain knowledge remain indispensable in healthcare decision-
making. Explainable AI should be seen as a tool to assist and augment
healthcare professionals’ judgment, providing them with transparent insights
into the underlying factors contributing to predictions. If researchers seek
to use explainability to guarantee the trust and reliability of their models
for clinicians, patients, and other stakeholders, their implementations not
being transparent or easily accessible sends the wrong message and stifles
research growth. Since the papers that do have open access links on their
publications are from high-ranking journals that demand open access and code
sharing when submitting, a possible way to mitigate this is to have more
journals demand accessible code sharing resources from authors, especially in
cases of explainability research which, as we have seen, cannot be separated
from concepts around trustworthiness, the key to which is transparency and
reproducibility in the case of software.

We should not, however, be overly critical or pessimistic about the outcomes
of explainability research in clinical risk prediction. The medical field is
more than familiar with using black-box technology as many drugs’ mechanisms
have still not been elucidated for their health benefits. Paracetamol is a
commonly cited example of a popular over-the-counter drug whose mechanisms of
action have not been revealed for a much longer time than we have had deep
learning [263, 12]. Several methods for achieving interpretability in clinical
risk prediction were explored, including rule-based models, feature importance
techniques, and post-hoc methods but a key point has been made of
interpretability on its own being an insufficient attribute of a truly
explainable clinical risk prediction AI model. Each approach, thus, presented
its own advantages and limitations, emphasizing the need for a careful
selection and integration of multiple techniques to ensure a comprehensive
understanding of AI-driven predictions. The progress forward has to be focused
on an end-to-end approach to explainability in clinical risk prediction, no
longer being enough to simply apply an explainability method to a model, but
rather to clinically and quantitatively measure its success while also
including different stakeholders from clinicians, and patients, to developers
into the process. Clear regulations, guidelines, and research culture
practices will help make this transition smoother and for the benefit of a
larger group of stakeholders.

## References

  * [1] Fei-Yue Wang, Jun Jason Zhang, Xinhu Zheng, Xiao Wang, Yong Yuan, Xiaoxiao Dai, Jie Zhang, and Liuqing Yang.  Where does alphago go: From church-turing thesis to alphago thesis and beyond.  IEEE/CAA Journal of Automatica Sinica, 3(2):113–120, 2016.
  * [2] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609, 2020.
  * [3] Robert Dale.  Gpt-3: What’s it good for?  Natural Language Engineering, 27(1):113–118, 2021.
  * [4] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  Nature, 596(7873):583–589, 2021.
  * [5] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S Corrado, Ara Darzi, et al.  International evaluation of an ai system for breast cancer screening.  Nature, 577(7788):89–94, 2020.
  * [6] Christian Leibig, Moritz Brehmer, Stefan Bunk, Danalyn Byng, Katja Pinker, and Lale Umutlu.  Combining the strengths of radiologists and ai for breast cancer screening: a retrospective analysis.  The Lancet Digital Health, 4(7):e507–e519, 2022.
  * [7] Jianyuan Deng, Zhibo Yang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang.  Artificial intelligence in drug discovery: applications and techniques.  Briefings in Bioinformatics, 23(1), 2022.
  * [8] Guillaume Chassagnon, Maria Vakalopoulou, Enzo Battistella, Stergios Christodoulidis, Trieu-Nghi Hoang-Thi, Severine Dangeard, Eric Deutsch, Fabrice Andre, Enora Guillo, Nara Halm, et al.  Ai-driven quantification, staging and outcome prediction of covid-19 pneumonia.  Medical image analysis, 67:101860, 2021.
  * [9] Maarten Van Smeden, Georg Heinze, Ben Van Calster, Folkert W Asselbergs, Panos E Vardas, Nico Bruining, Peter De Jaegere, Jason H Moore, Spiros Denaxas, Anne Laure Boulesteix, et al.  Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease.  European Heart Journal, 43(31):2921–2930, 2022.
  * [10] Dawn Branley-Bell, Rebecca Whitworth, and Lynne Coventry.  User trust and understanding of explainable ai: exploring algorithm visualisations and user biases.  In International Conference on Human-Computer Interaction, pages 382–399. Springer, 2020.
  * [11] Sanjoy Dey, Prithwish Chakraborty, Bum Chul Kwon, Amit Dhurandhar, Mohamed Ghalwash, Fernando J Suarez Saiz, Kenney Ng, Daby Sow, Kush R Varshney, and Pablo Meyer.  Human-centered explainability for life sciences, healthcare, and medical informatics.  Patterns, 3(5):100493, 2022.
  * [12] Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L Beam.  The false hope of current approaches to explainable artificial intelligence in health care.  The Lancet Digital Health, 3(11):e745–e750, 2021.
  * [13] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information fusion, 58:82–115, 2020.
  * [14] Avleen Malhi, Samanta Knapic, and Kary Främling.  Explainable agents for less bias in human-agent decision making.  In International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, pages 129–146. Springer, 2020.
  * [15] Bryce Goodman and Seth Flaxman.  European union regulations on algorithmic decision-making and a “right to explanation”.  AI magazine, 38(3):50–57, 2017.
  * [16] Paul Voigt and Axel Von dem Bussche.  The eu general data protection regulation (gdpr).  A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):10–5555, 2017.
  * [17] Andrew Selbst and Julia Powles.  “meaningful information” and the right to explanation.  In Conference on Fairness, Accountability and Transparency, pages 48–48. PMLR, 2018.
  * [18] Sandra Wachter and Brent Mittelstadt.  A right to reasonable inferences: re-thinking data protection law in the age of big data and ai.  Colum. Bus. L. Rev., page 494, 2019.
  * [19] Hui Wen Loh, Chui Ping Ooi, Silvia Seoni, Prabal Datta Barua, Filippo Molinari, and U Rajendra Acharya.  Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011–2022).  Computer Methods and Programs in Biomedicine, page 107161, 2022\.
  * [20] Mobeen Nazar, Muhammad Mansoor Alam, Eiad Yafi, and Mazliham Mohd Su’ud.  A systematic review of human–computer interaction and explainable artificial intelligence in healthcare with artificial intelligence techniques.  IEEE Access, 9:153316–153348, 2021.
  * [21] Khansa Rasheed, Adnan Qayyum, Mohammed Ghaly, Ala Al-Fuqaha, Adeel Razi, and Junaid Qadir.  Explainable, trustworthy, and ethical machine learning for healthcare: A survey.  Computers in Biology and Medicine, page 106043, 2022.
  * [22] Felipe Giuste, Wenqi Shi, Yuanda Zhu, Tarun Naren, Monica Isgut, Ying Sha, Li Tong, Mitali Gupte, and May D Wang.  Explainable artificial intelligence methods in combating pandemics: A systematic review.  IEEE Reviews in Biomedical Engineering, 2022.
  * [23] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for interpreting and understanding deep neural networks.  Digital signal processing, 73:1–15, 2018.
  * [24] Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  Artificial intelligence, 267:1–38, 2019.
  * [25] Jinfeng Zhong and Elsa Negre.  Ai: To interpret or to explain?  In Congrès Inforsid ((INFormatique des ORganisations et Systèmes d’Information et de Décision) 2021, 2021.
  * [26] Maarten van Smeden, Johannes B Reitsma, Richard D Riley, Gary S Collins, and Karel GM Moons.  Clinical prediction models: diagnosis versus prognosis.  Journal of clinical epidemiology, 132:142–145, 2021.
  * [27] Irena Spasic, Goran Nenadic, et al.  Clinical text data in machine learning: systematic review.  JMIR medical informatics, 8(3):e17984, 2020.
  * [28] Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi.  Deep ehr: a survey of recent advances in deep learning techniques for electronic health record (ehr) analysis.  IEEE journal of biomedical and health informatics, 22(5):1589–1604, 2017.
  * [29] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.  Explaining explanations: An overview of interpretability of machine learning.  In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80–89. IEEE, 2018.
  * [30] Radwa ElShawi, Youssef Sherif, Mouaz Al-Mallah, and Sherif Sakr.  Interpretability in healthcare: A comparative study of local machine learning interpretability techniques.  Computational Intelligence, 37(4):1633–1650, 2021.
  * [31] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis.  Explainable ai: A review of machine learning interpretability methods.  Entropy, 23(1):18, 2020.
  * [32] Mildred K Cho.  Rising to the challenge of bias in health care ai.  Nature Medicine, 27(12):2079–2081, 2021.
  * [33] Salem Alelyani.  Detection and evaluation of machine learning bias.  Applied Sciences, 11(14):6271, 2021.
  * [34] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  ACM Computing Surveys (CSUR), 54(6):1–35, 2021.
  * [35] Yongsu Ahn and Yu-Ru Lin.  Fairsight: Visual analytics for fairness in decision making.  IEEE transactions on visualization and computer graphics, 26(1):1086–1095, 2019.
  * [36] Niels Bantilan.  Themis-ml: A fairness-aware machine learning interface for end-to-end discrimination discovery and mitigation.  Journal of Technology in Human Services, 36(1):15–30, 2018.
  * [37] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  Advances in neural information processing systems, 30, 2017.
  * [38] Arun Das and Paul Rad.  Opportunities and challenges in explainable artificial intelligence (xai): A survey.  arXiv preprint arXiv:2006.11371, 2020.
  * [39] Alun Preece, Dan Harborne, Dave Braines, Richard Tomsett, and Supriyo Chakraborty.  Stakeholders in explainable ai.  arXiv preprint arXiv:1810.00184, 2018.
  * [40] Haomin Chen, Catalina Gomez, Chien-Ming Huang, and Mathias Unberath.  Explainable medical imaging ai needs human-centered design: guidelines and evidence from a systematic review.  npj Digital Medicine, 5(1):1–15, 2022.
  * [41] Danton S Char, Michael D Abràmoff, and Chris Feudtner.  Identifying ethical considerations for machine learning healthcare applications.  The American Journal of Bioethics, 20(11):7–17, 2020.
  * [42] Chris Feudtner, Theodore Schall, Pamela Nathanson, and Jay Berry.  Ethical framework for risk stratification and mitigation programs for children with medical complexity.  Pediatrics, 141(Supplement_3):S250–S258, 2018.
  * [43] Anja Wölker and Thomas E Powell.  Algorithms in the newsroom? news readers’ perceived credibility and selection of automated journalism.  Journalism, 22(1):86–103, 2021.
  * [44] Donghee Shin, Bu Zhong, and Frank A Biocca.  Beyond user experience: What constitutes algorithmic experiences?  International Journal of Information Management, 52:102061, 2020\.
  * [45] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  " why should i trust you?" explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [46] Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  Queue, 16(3):31–57, 2018.
  * [47] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215, 2019.
  * [48] Christoph Molnar.  Interpretable machine learning.  Lulu. com, 2020.
  * [49] Nancy McCauley and Mohammad Ala.  The use of expert systems in the healthcare industry.  Information & Management, 22(4):227–235, 1992.
  * [50] Benjamin Letham, Cynthia Rudin, Tyler H McCormick, and David Madigan.  Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model.  The Annals of Applied Statistics, 9(3):1350–1371, 2015.
  * [51] Muhammad Rehman Zafar and Naimul Khan.  Deterministic local interpretable model-agnostic explanations for stable explainability.  Machine Learning and Knowledge Extraction, 3(3):525–541, 2021.
  * [52] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * [53] Cezary Z Janikow.  Fuzzy decision trees: issues and methods.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 28(1):1–14, 1998.
  * [54] Christophe Marsala.  Data mining with ensembles of fuzzy decision trees.  In 2009 IEEE Symposium on Computational Intelligence and Data Mining, pages 348–354. IEEE, 2009.
  * [55] Vitaly Levashenko, Elena Zaitseva, and Seppo Puuronen.  Fuzzy classifier based on fuzzy decision tree.  In EUROCON 2007-The International Conference on" Computer as a Tool", pages 823–827. IEEE, 2007.
  * [56] Chun-Fu Lin and Sheng-De Wang.  Fuzzy support vector machines.  IEEE transactions on neural networks, 13(2):464–471, 2002.
  * [57] Zekang Bian, Chi Man Vong, Pak Kin Wong, and Shitong Wang.  Fuzzy knn method with adaptive nearest neighbors.  IEEE Transactions on Cybernetics, 2020.
  * [58] Ravikiran Chimatapu, Hani Hagras, Andrew Starkey, and Gilbert Owusu.  Explainable ai and fuzzy logic systems.  In International Conference on Theory and Practice of Natural Computing, pages 3–20. Springer, 2018.
  * [59] Yue Deng, Zhiquan Ren, Youyong Kong, Feng Bao, and Qionghai Dai.  A hierarchical fused fuzzy deep neural network for data classification.  IEEE Transactions on Fuzzy Systems, 25(4):1006–1012, 2016.
  * [60] Seonyeong Park, Suk Jin Lee, Elisabeth Weiss, and Yuichi Motai.  Intra-and inter-fractional variation prediction of lung tumors using fuzzy deep learning.  IEEE journal of translational engineering in health and medicine, 4:1–12, 2016.
  * [61] Ravikiran Chimatapu, Hani Hagras, Andrew Starkey, and Gilbert Owusu.  Interval type-2 fuzzy logic based stacked autoencoder deep neural network for generating explainable ai models in workforce optimization.  In 2018 IEEE international conference on fuzzy systems (FUZZ-IEEE), pages 1–8. IEEE, 2018.
  * [62] Ravikiran Chimatapu, Hani Hagras, Mathias Kern, and Gilbert Owusu.  Hybrid deep learning type-2 fuzzy logic systems for explainable ai.  In 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pages 1–6. IEEE, 2020.
  * [63] Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec.  Interpretable decision sets: A joint framework for description and prediction.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1675–1684, 2016.
  * [64] Sanjeeb Dash, Oktay Gunluk, and Dennis Wei.  Boolean decision rules via column generation.  Advances in neural information processing systems, 31, 2018.
  * [65] Tong Wang and Cynthia Rudin.  Learning optimized or’s of and’s.  arXiv preprint arXiv:1511.02210, 2015.
  * [66] Connor Lawless and Oktay Gunluk.  Fair decision rules for binary classification.  arXiv preprint arXiv:2107.01325, 2021.
  * [67] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proceedings of the IEEE, 109(1):43–76, 2020.
  * [68] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  Advances in neural information processing systems, 27, 2014.
  * [69] Paul Micaelli and Amos J Storkey.  Zero-shot knowledge transfer via adversarial belief matching.  Advances in Neural Information Processing Systems, 32, 2019.
  * [70] Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, and Peder A Olsen.  Improving simple models with confidence profiles.  Advances in Neural Information Processing Systems, 31, 2018.
  * [71] Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach.  Manipulating and measuring model interpretability.  In Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1–52, 2021.
  * [72] David Alvarez Melis and Tommi Jaakkola.  Towards robust interpretability with self-explaining neural networks.  Advances in neural information processing systems, 31, 2018.
  * [73] Ruel Churchill and James Brown.  Ebook: Complex Variables and Applications.  McGraw Hill, 2014.
  * [74] Isha Puri, Amit Dhurandhar, Tejaswini Pedapati, Karthikeyan Shanmugam, Dennis Wei, and Kush R Varshney.  Cofrnets: interpretable neural architecture inspired by continued fractions.  Advances in Neural Information Processing Systems, 34:21668–21680, 2021.
  * [75] Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian.  A framework for learning ante-hoc explainable models via concepts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286–10295, 2022.
  * [76] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  Advances in neural information processing systems, 30, 2017.
  * [77] Lujain Ibrahim, Munib Mesinovic, Kai-Wen Yang, and Mohamad A Eid.  Explainable prediction of acute myocardial infarction using machine learning and shapley values.  IEEE Access, 8:210410–210417, 2020.
  * [78] Muhammad Rehman Zafar and Naimul Mefraz Khan.  Dlime: A deterministic local interpretable model-agnostic explanations approach for computer-aided diagnosis systems.  arXiv preprint arXiv:1906.10263, 2019.
  * [79] Mateusz Staniak and Przemyslaw Biecek.  Explanations of model predictions with live and breakdown packages.  arXiv preprint arXiv:1804.01955, 2018.
  * [80] Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das.  Explanations based on the missing: Towards contrastive explanations with pertinent negatives.  Advances in neural information processing systems, 31, 2018.
  * [81] Tim Miller.  Contrastive explanation: A structural-model approach.  The Knowledge Engineering Review, 36, 2021.
  * [82] Ronny Luss, Pin-Yu Chen, Amit Dhurandhar, Prasanna Sattigeri, Yunfeng Zhang, Karthikeyan Shanmugam, and Chun-Chen Tu.  Leveraging latent features for local explanations.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1139–1149, 2021.
  * [83] Guofan Shao, Lina Tang, and Hao Zhang.  Introducing image classification efficacies.  IEEE Access, 9:134809–134816, 2021.
  * [84] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo.  Examples are not enough, learn to criticize! criticism for interpretability.  Advances in neural information processing systems, 29, 2016.
  * [85] An-phi Nguyen and María Rodríguez Martínez.  On quantitative aspects of model interpretability.  arXiv preprint arXiv:2007.07584, 2020.
  * [86] Laurent Itti, Christof Koch, and Ernst Niebur.  A model of saliency-based visual attention for rapid scene analysis.  IEEE Transactions on pattern analysis and machine intelligence, 20(11):1254–1259, 1998.
  * [87] Yola Jones, Fani Deligianni, and Jeff Dalton.  Improving ecg classification interpretability using saliency maps.  In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 675–682. IEEE, 2020.
  * [88] Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al.  Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging.  Radiology: Artificial Intelligence, 3(6), 2021.
  * [89] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [90] Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi.  Improving deep learning interpretability by saliency guided training.  Advances in Neural Information Processing Systems, 34:26726–26739, 2021.
  * [91] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  Advances in neural information processing systems, 31, 2018.
  * [92] Jindong Gu and Volker Tresp.  Saliency methods for explaining adversarial attacks.  arXiv preprint arXiv:1908.08413, 2019.
  * [93] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [94] Ilaria Gandin, Arjuna Scagnetto, Simona Romani, and Giulia Barbati.  Interpretability of time-series deep learning models: A study in cardiovascular patients admitted to intensive care unit.  Journal of Biomedical Informatics, 121:103876, 2021.
  * [95] Sarthak Jain and Byron C Wallace.  Attention is not explanation.  arXiv preprint arXiv:1902.10186, 2019.
  * [96] Yuyang Xu, Haochao Ying, Siyi Qian, Fuzhen Zhuang, Xiao Zhang, Deqing Wang, Jian Wu, and Hui Xiong.  Time-aware context-gated graph attention network for clinical risk prediction.  IEEE Transactions on Knowledge and Data Engineering, 2022.
  * [97] Sundreen Asad Kamal, Changchang Yin, Buyue Qian, and Ping Zhang.  An interpretable risk prediction model for healthcare with pattern attention.  BMC Medical Informatics and Decision Making, 20(11):1–10, 2020\.
  * [98] Yue Zhang and Jie Li.  Application of heartbeat-attention mechanism for detection of myocardial infarction using 12-lead ecg records.  Applied Sciences, 9(16):3328, 2019.
  * [99] Uma Girkar, Ryo Uchimido, Li-wei H Lehman, Peter Szolovits, Leo Celi, and Wei-Hung Weng.  Predicting blood pressure response to fluid bolus therapy using neural networks with clinical interpretability.  Circulation Research, 125(Suppl_1):A448–A448, 2019.
  * [100] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 782–791, 2021.
  * [101] Domjan Barić, Petar Fumić, Davor Horvatić, and Tomislav Lipic.  Benchmarking attention-based interpretability of deep learning in multivariate time series predictions.  Entropy, 23(2):143, 2021.
  * [102] Radwa ElShawi, Youssef Sherif, Mouaz Al-Mallah, and Sherif Sakr.  Ilime: local and global interpretable model-agnostic explainer of black-box decision.  In European Conference on Advances in Databases and Information Systems, pages 53–68. Springer, 2019.
  * [103] Karthik S Gurumoorthy, Amit Dhurandhar, and Guillermo Cecchi.  Protodash: Fast interpretable prototype selection.  arXiv preprint arXiv:1707.01212, 2017.
  * [104] Gregory Plumb, Denali Molitor, and Ameet S Talwalkar.  Model agnostic supervised local explanations.  Advances in neural information processing systems, 31, 2018.
  * [105] Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  Harv. JL & Tech., 31:841, 2017.
  * [106] Thomas Bäck, Mike Preuss, André Deutz, Hao Wang, Carola Doerr, Michael Emmerich, and Heike Trautmann.  Parallel Problem Solving from Nature–PPSN XVI: 16th International Conference, PPSN 2020, Leiden, The Netherlands, September 5-9, 2020, Proceedings, Part II, volume 12270.  Springer Nature, 2020.
  * [107] Jooyoung Chang, Jinho Lee, Ahnul Ha, Young Soo Han, Eunoo Bak, Seulggie Choi, Jae Moon Yun, Uk Kang, Il Hyung Shin, Joo Young Shin, et al.  Explaining the rationale of deep learning glaucoma decisions with adversarial examples.  Ophthalmology, 128(1):78–88, 2021.
  * [108] Andreas Brandsæter and Ingrid K Glad.  Shapley values for cluster importance.  Data Mining and Knowledge Discovery, pages 1–32, 2022.
  * [109] Christoph Molnar, Gunnar König, Bernd Bischl, and Giuseppe Casalicchio.  Model-agnostic feature importance and effects with dependent features: a conditional subgroup approach.  Data Mining and Knowledge Discovery, pages 1–39, 2023.
  * [110] Jerome H Friedman.  Greedy function approximation: a gradient boosting machine.  Annals of statistics, pages 1189–1232, 2001.
  * [111] Namita Agarwal and Saikat Das.  Interpretable machine learning tools: A survey.  In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1528–1534. IEEE, 2020.
  * [112] Andrew Yeh and Anhthy Ngo.  Bringing a ruler into the black box: Uncovering feature impact from individual conditional expectation plots.  In Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2021, Virtual Event, September 13-17, 2021, Proceedings, Part I, pages 34–48. Springer, 2022\.
  * [113] Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin.  Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation.  journal of Computational and Graphical Statistics, 24(1):44–65, 2015.
  * [114] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  Model-agnostic interpretability of machine learning.  arXiv preprint arXiv:1606.05386, 2016.
  * [115] Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilović, et al.  One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques.  arXiv preprint arXiv:1909.03012, 2019.
  * [116] Chuizheng Meng, Loc Trinh, Nan Xu, and Yan Liu.  Mimic-if: Interpretability and fairness evaluation of deep learning models on mimic-iv dataset.  arXiv preprint arXiv:2102.06761, 2021.
  * [117] David Alvarez-Melis and Tommi S Jaakkola.  On the robustness of interpretability methods.  arXiv preprint arXiv:1806.08049, 2018.
  * [118] David Alvarez-Melis and Tommi S Jaakkola.  A causal framework for explaining the predictions of black-box sequence-to-sequence models.  arXiv preprint arXiv:1707.01943, 2017.
  * [119] Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols.  Perturbation-based methods for explaining deep neural networks: A survey.  Pattern Recognition Letters, 150:228–234, 2021.
  * [120] Zhongang Qi, Saeed Khorram, and Fuxin Li.  Visualizing deep networks by optimizing with integrated gradients.  In CVPR Workshops, volume 2, pages 1–4, 2019.
  * [121] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International conference on machine learning, pages 3145–3153. PMLR, 2017.
  * [122] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  arXiv preprint arXiv:1412.6806, 2014.
  * [123] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller.  Layer-wise relevance propagation: an overview.  Explainable AI: interpreting, explaining and visualizing deep learning, pages 193–209, 2019.
  * [124] John Grezmak, Jianjing Zhang, Peng Wang, Kenneth A Loparo, and Robert X Gao.  Interpretable convolutional neural network through layer-wise relevance propagation for machine fault diagnosis.  IEEE Sensors Journal, 20(6):3172–3181, 2019.
  * [125] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.
  * [126] Kira Vinogradova, Alexandr Dibrov, and Gene Myers.  Towards interpretable semantic segmentation via gradient-weighted class activation mapping (student abstract).  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13943–13944, 2020.
  * [127] Mohammed Bany Muhammad and Mohammed Yeasin.  Eigen-cam: Class activation map using principal components.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2020.
  * [128] Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath.  Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations.  In International Conference on Artificial Intelligence and Statistics, pages 1459–1467. PMLR, 2021.
  * [129] Jinsung Yoon, James Jordon, and Mihaela van der Schaar.  Invase: Instance-wise variable selection using neural networks.  In International Conference on Learning Representations, 2018.
  * [130] Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma.  Pcanet: A simple deep learning baseline for image classification?  IEEE transactions on image processing, 24(12):5017–5032, 2015.
  * [131] Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron Van Den Oord.  Unsupervised speech representation learning using wavenet autoencoders.  IEEE/ACM transactions on audio, speech, and language processing, 27(12):2041–2053, 2019.
  * [132] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: deep learning for interpretable image recognition.  Advances in neural information processing systems, 32, 2019.
  * [133] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang.  Concept bottleneck models.  In International Conference on Machine Learning, pages 5338–5348. PMLR, 2020.
  * [134] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim.  Explaining classifiers with causal concept effect (cace).  arXiv preprint arXiv:1907.07165, 2019.
  * [135] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim.  Towards automatic concept-based explanations.  Advances in Neural Information Processing Systems, 32, 2019.
  * [136] Vidhya Kamakshi, Uday Gupta, and Narayanan C Krishnan.  Pace: Posthoc architecture-agnostic concept extractor for explaining cnns.  In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2021.
  * [137] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar.  On completeness-aware concept-based explanations in deep neural networks.  Advances in Neural Information Processing Systems, 33:20554–20565, 2020.
  * [138] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg.  Null it out: Guarding protected attributes by iterative nullspace projection.  arXiv preprint arXiv:2004.07667, 2020.
  * [139] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu.  Metalearners for estimating heterogeneous treatment effects using machine learning.  Proceedings of the national academy of sciences, 116(10):4156–4165, 2019.
  * [140] Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart.  Causalm: Causal model explanation through counterfactual language models.  Computational Linguistics, 47(2):333–386, 2021.
  * [141] Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu.  Cebab: Estimating the causal effects of real-world concepts on nlp model behavior.  arXiv preprint arXiv:2205.14140, 2022.
  * [142] Vivek Kaul, Sarah Enslin, and Seth A Gross.  History of artificial intelligence in medicine.  Gastrointestinal endoscopy, 92(4):807–812, 2020.
  * [143] Irina M Cazacu, Anca Udristoiu, Lucian Gheorghe Gruionu, Andreea Iacob, Gabriel Gruionu, and Adrian Saftoiu.  Artificial intelligence in pancreatic cancer: Toward precision diagnosis.  Endoscopic ultrasound, 8(6):357, 2019.
  * [144] Félix Renard, Soulaimane Guedria, Noel De Palma, and Nicolas Vuillerme.  Variability and reproducibility in deep learning for medical image segmentation.  Scientific Reports, 10(1):1–16, 2020.
  * [145] Kaiyan Li, Weimin Zhou, Hua Li, and Mark A Anastasio.  Assessing the impact of deep neural network-based image denoising on binary signal detection tasks.  IEEE transactions on medical imaging, 40(9):2295–2305, 2021.
  * [146] Maram Mahmoud A Monshi, Josiah Poon, and Vera Chung.  Deep learning in generating radiology reports: A survey.  Artificial Intelligence in Medicine, 106:101878, 2020.
  * [147] Laith Alzubaidi, Muthana Al-Amidie, Ahmed Al-Asadi, Amjad J Humaidi, Omran Al-Shamma, Mohammed A Fadhel, Jinglan Zhang, J Santamaría, and Ye Duan.  Novel transfer learning approach for medical imaging with limited labeled data.  Cancers, 13(7):1590, 2021.
  * [148] Xiaoxuan Liu, Livia Faes, Aditya U Kale, Siegfried K Wagner, Dun Jack Fu, Alice Bruynseels, Thushika Mahendiran, Gabriella Moraes, Mohith Shamdas, Christoph Kern, et al.  A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.  The lancet digital health, 1(6):e271–e297, 2019.
  * [149] Hyunna Lee, Eun-Jae Lee, Sungwon Ham, Han-Bin Lee, Ji Sung Lee, Sun U Kwon, Jong S Kim, Namkug Kim, and Dong-Wha Kang.  Machine learning approach to identify stroke within 4.5 hours.  Stroke, 51(3):860–866, 2020.
  * [150] Phillip Chlap, Hang Min, Nym Vandenberg, Jason Dowling, Lois Holloway, and Annette Haworth.  A review of medical image data augmentation techniques for deep learning applications.  Journal of Medical Imaging and Radiation Oncology, 65(5):545–563, 2021.
  * [151] Alexander Selvikvåg Lundervold and Arvid Lundervold.  An overview of deep learning in medical imaging focusing on mri.  Zeitschrift für Medizinische Physik, 29(2):102–127, 2019.
  * [152] Martin J Willemink, Wojciech A Koszek, Cailin Hardell, Jie Wu, Dominik Fleischmann, Hugh Harvey, Les R Folio, Ronald M Summers, Daniel L Rubin, and Matthew P Lungren.  Preparing medical imaging data for machine learning.  Radiology, 295(1):4–15, 2020.
  * [153] Fatma Murat, Ozal Yildirim, Muhammed Talo, Ulas Baran Baloglu, Yakup Demir, and U Rajendra Acharya.  Application of deep learning techniques for heartbeats detection using ecg signals-analysis and review.  Computers in biology and medicine, 120:103726, 2020.
  * [154] Arlene Casey, Emma Davidson, Michael Poon, Hang Dong, Daniel Duma, Andreas Grivas, Claire Grover, Víctor Suárez-Paniagua, Richard Tobin, William Whiteley, et al.  A systematic review of natural language processing applied to radiology reports.  BMC medical informatics and decision making, 21(1):179, 2021.
  * [155] Ashwani Tanwar, Jingqing Zhang, Julia Ive, Vibhor Gupta, and Yike Guo.  Unsupervised numerical reasoning to extract phenotypes from clinical text by leveraging external knowledge.  In Multimodal AI in healthcare: A paradigm shift in health intelligence, pages 11–28. Springer, 2022.
  * [156] Kim Huat Goh, Le Wang, Adrian Yong Kwang Yeow, Hermione Poh, Ke Li, Joannas Jie Lin Yeow, and Gamaliel Yu Heng Tan.  Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare.  Nature communications, 12(1):711, 2021.
  * [157] Julia Lin, Todd Joseph, Joanna Jean Parga-Belinkie, Abigail Mandel, Ryan Schumacher, Karl Neumann, Laura Scalise, Jessica Gaulton, Lori Christ, Kirstin Leitner, et al.  Development of a practical training method for a healthcare artificial intelligence (ai) chatbot.  BMJ Innovations, 7(2), 2021.
  * [158] Mercedes Arguello Casteleiro, Julio Des Diz, Nava Maroto, Maria Jesus Fernandez Prieto, Simon Peters, Chris Wroe, Carlos Sevillano Torrado, Diego Maseda Fernandez, Robert Stevens, et al.  Semantic deep learning: Prior knowledge and a type of four-term embedding analogy to acquire treatments for well-known diseases.  JMIR medical informatics, 8(8):e16948, 2020.
  * [159] Akhil Vaid, Kipp W Johnson, Marcus A Badgeley, Sulaiman S Somani, Mesude Bicak, Isotta Landi, Adam Russak, Shan Zhao, Matthew A Levin, Robert S Freeman, et al.  Using deep-learning algorithms to simultaneously identify right and left ventricular dysfunction from the electrocardiogram.  Cardiovascular Imaging, 15(3):395–410, 2022.
  * [160] Miao Wu, Xianjin Du, Raymond Gu, and Jie Wei.  Artificial intelligence for clinical decision support in sepsis.  Frontiers in Medicine, 8:665464, 2021.
  * [161] Fei Li and Hong Yu.  Icd coding from clinical text using multi-filter residual convolutional neural network.  In proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8180–8187, 2020.
  * [162] Arjan Sammani, Ayoub Bagheri, Peter GM van der Heijden, Anneline SJM Te Riele, Annette F Baas, CAJ Oosters, Daniel Oberski, and Folkert W Asselbergs.  Automatic multilabel detection of icd10 codes in dutch cardiology discharge letters using neural networks.  NPJ digital medicine, 4(1):37, 2021.
  * [163] Kevin Lybarger, Mari Ostendorf, and Meliha Yetisgen.  Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction.  Journal of Biomedical Informatics, 113:103631, 2021.
  * [164] Xianghao Zhan, Marie Humbert-Droz, Pritam Mukherjee, and Olivier Gevaert.  Structuring clinical text with ai: Old versus new natural language processing techniques evaluated on eight common cardiovascular diseases.  Patterns, 2(7):100289, 2021.
  * [165] Martijn G Kersloot, Florentien JP van Putten, Ameen Abu-Hanna, Ronald Cornet, and Derk L Arts.  Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies.  Journal of biomedical semantics, 11:1–21, 2020.
  * [166] Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao.  Deep learning–based text classification: a comprehensive review.  ACM computing surveys (CSUR), 54(3):1–40, 2021.
  * [167] Diane M Korngiebel and Sean D Mooney.  Considering the possibilities and pitfalls of generative pre-trained transformer 3 (gpt-3) in healthcare delivery.  NPJ Digital Medicine, 4(1):93, 2021.
  * [168] Katherine Elkins and Jon Chun.  Can gpt-3 pass a writer’s turing test?  Journal of Cultural Analytics, 5(2), 2020.
  * [169] Wardah S Alharbi and Mamoon Rashid.  A review of deep learning applications in human genomics using next-generation sequencing data.  Human Genomics, 16(1):1–20, 2022.
  * [170] Sean K Wang, Surag Nair, Rui Li, Katerina Kraft, Anusri Pampari, Aman Patel, Joyce B Kang, Christy Luong, Anshul Kundaje, and Howard Y Chang.  Single-cell multiome of the human retina and deep learning nominate causal variants in complex eye diseases.  Cell Genomics, 2(8):100164, 2022.
  * [171] Taeho Jo, Kwangsik Nho, Paula Bice, Andrew J Saykin, Alzheimer’s Disease Neuroimaging Initiative, et al.  Deep learning-based identification of genetic variants: application to alzheimer’s disease classification.  Briefings in Bioinformatics, 23(2), 2022.
  * [172] Janani Venugopalan, Li Tong, Hamid Reza Hassanzadeh, and May D Wang.  Multimodal deep learning models for early detection of alzheimer’s disease stage.  Scientific reports, 11(1):1–13, 2021.
  * [173] Manojkumar Kumaran, Umadevi Subramanian, and Bharanidharan Devarajan.  Performance assessment of variant calling pipelines using human whole exome sequencing and simulated data.  BMC bioinformatics, 20(1):1–11, 2019.
  * [174] Gökcen Eraslan, Žiga Avsec, Julien Gagneur, and Fabian Theis.  Deep learning: new computational modelling techniques for genomics.  Nature Reviews Genetics, pages 1–13, 2019.
  * [175] Noura S Abul-Husn and Eimear E Kenny.  Personalized medicine and the power of electronic health records.  Cell, 177(1):58–69, 2019.
  * [176] OpenSAFELY Collaborative, Elizabeth Williamson, Alex J Walker, Krishnan Bhaskaran, Seb Bacon, Chris Bates, Caroline E Morton, Helen J Curtis, Amir Mehrkar, David Evans, et al.  Opensafely: factors associated with covid-19-related hospital death in the linked electronic health records of 17 million adult nhs patients.  MedRxiv, pages 2020–05, 2020.
  * [177] Sarah A Pendergrass and Dana C Crawford.  Using electronic health records to generate phenotypes for research.  Current protocols in human genetics, 100(1):e80, 2019.
  * [178] Elham Mahmoudi, Neil Kamdar, Noa Kim, Gabriella Gonzales, Karandeep Singh, and Akbar K Waljee.  Use of electronic medical records in development and validation of risk prediction models of hospital readmission: systematic review.  bmj, 369, 2020.
  * [179] Zahra Ebrahimi, Mohammad Loni, Masoud Daneshtalab, and Arash Gharehbaghi.  A review on deep learning methods for ecg arrhythmia classification.  Expert Systems with Applications: X, 7:100033, 2020.
  * [180] Ludivine Morvan, Cristina Nanni, Anne-Victoire Michaud, Bastien Jamet, Clément Bailly, Caroline Bodet-Milin, Stephane Chauvie, Cyrille Touzeau, Philippe Moreau, Elena Zamagni, et al.  Learned deep radiomics for survival analysis with attention.  In International Workshop on PRedictive Intelligence In MEdicine, pages 35–45. Springer, 2020.
  * [181] Esther Puyol-Antón, Chen Chen, James R Clough, Bram Ruijsink, Baldeep S Sidhu, Justin Gould, Bradley Porter, Marc Elliott, Vishal Mehta, Daniel Rueckert, et al.  Interpretable deep models for cardiac resynchronisation therapy response prediction.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 284–293. Springer, 2020.
  * [182] Patrik Sabol, Peter Sinčák, Pitoyo Hartono, Pavel Kočan, Zuzana Benetinová, Alžbeta Blichárová, L’udmila Verbóová, Erika Štammová, Antónia Sabolová-Fabianová, and Anna Jašková.  Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images.  Journal of biomedical informatics, 109:103523, 2020.
  * [183] Pavan Rajkumar Magesh, Richard Delwin Myloth, and Rijo Jackson Tom.  An explainable machine learning model for early detection of parkinson’s disease using lime on datscan imagery.  Computers in Biology and Medicine, 126:104041, 2020.
  * [184] Alexander Binder, Michael Bockmayr, Miriam Hägele, Stephan Wienert, Daniel Heim, Katharina Hellweg, Masaru Ishii, Albrecht Stenzinger, Andreas Hocke, Carsten Denkert, et al.  Morphological and molecular breast cancer profiling through explainable machine learning.  Nature Machine Intelligence, 3(4):355–366, 2021.
  * [185] Quan Zhang, Qian Du, and Guohua Liu.  A whole-process interpretable and multi-modal deep reinforcement learning for diagnosis and analysis of alzheimer’s disease.  Journal of Neural Engineering, 18(6):066032, 2021.
  * [186] Parnian Afshar, Farnoosh Naderkhani, Anastasia Oikonomou, Moezedin Javad Rafiee, Arash Mohammadi, and Konstantinos N Plataniotis.  Mixcaps: A capsule network-based mixture of experts for lung nodule malignancy prediction.  Pattern Recognition, 116:107942, 2021.
  * [187] Jun Wang, Chen Liu, Jingwen Li, Cheng Yuan, Lichi Zhang, Cheng Jin, Jianwei Xu, Yaqi Wang, Yaofeng Wen, Hongbing Lu, et al.  icovid: interpretable deep learning framework for early recovery-time prediction of covid-19 patients.  NPJ digital medicine, 4(1):124, 2021.
  * [188] Lijing Jia, Zijian Wei, Heng Zhang, Jiaming Wang, Ruiqi Jia, Manhong Zhou, Xueyan Li, Hankun Zhang, Xuedong Chen, Zheyuan Yu, et al.  An interpretable machine learning model based on a quick pre-screening system enables accurate deterioration risk prediction for covid-19.  Scientific Reports, 11(1):23127, 2021.
  * [189] Shu-Cheng Liu, Jesyin Lai, Jhao-Yu Huang, Chia-Fong Cho, Pei Hua Lee, Min-Hsuan Lu, Chun-Chieh Yeh, Jiaxin Yu, and Wei-Ching Lin.  Predicting microvascular invasion in hepatocellular carcinoma: a deep learning model validated across hospitals.  Cancer Imaging, 21:1–16, 2021.
  * [190] Xuejun Qian, Jing Pei, Hui Zheng, Xinxin Xie, Lin Yan, Hao Zhang, Chunguang Han, Xiang Gao, Hanqi Zhang, Weiwei Zheng, et al.  Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning.  Nature biomedical engineering, 5(6):522–532, 2021.
  * [191] Pedro L Ballester, Laura Tomaz Da Silva, Matheus Marcon, Nathalia Bianchini Esper, Benicio N Frey, Augusto Buchweitz, and Felipe Meneguzzi.  Predicting brain age at slice level: convolutional neural networks and consequences for interpretability.  Frontiers in Psychiatry, 12:598518, 2021.
  * [192] Emma Pierson, David M Cutler, Jure Leskovec, Sendhil Mullainathan, and Ziad Obermeyer.  An algorithmic approach to reducing unexplained pain disparities in underserved populations.  Nature Medicine, 27(1):136–140, 2021.
  * [193] Rikiya Yamashita, Jin Long, Teri Longacre, Lan Peng, Gerald Berry, Brock Martin, John Higgins, Daniel L Rubin, and Jeanne Shen.  Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study.  The Lancet Oncology, 22(1):132–141, 2021.
  * [194] Nguyen Quoc Khanh Le, Quang Hien Kha, Van Hiep Nguyen, Yung-Chieh Chen, Sho-Jen Cheng, and Cheng-Yu Chen.  Machine learning-based radiomics signatures for egfr and kras mutations prediction in non-small-cell lung cancer.  International journal of molecular sciences, 22(17):9254, 2021.
  * [195] Sarah Fremond, Sonali Andani, Jurriaan Barkey Wolf, Jouke Dijkstra, Sinéad Melsbach, Jan J Jobsen, Mariel Brinkhuis, Suzan Roothaan, Ina Jurgenliemk-Schulz, Ludy CHW Lutgens, et al.  Interpretable deep learning model to predict the molecular classification of endometrial cancer from haematoxylin and eosin-stained whole-slide images: a combined analysis of the portec randomised trials and clinical cohorts.  The Lancet Digital Health, 5(2):e71–e82, 2023.
  * [196] Ivan Girardi, Pengfei Ji, An-phi Nguyen, Nora Hollenstein, Adam Ivankay, Lorenz Kuhn, Chiara Marchiori, and Ce Zhang.  Patient risk assessment and warning symptom detection using deep attention-based neural networks.  arXiv preprint arXiv:1809.10804, 2018.
  * [197] Wutthipong Kongburan, Mark Chignell, Nipon Charoenkitkarn, and Jonathan H Chan.  Enhancing predictive power of cluster-boosted regression with text-based indexing.  IEEE Access, 7:43394–43405, 2019.
  * [198] Zfania Tom Korach, Kenrick D Cato, Sarah A Collins, Min Jeoung Kang, Christopher Knaplund, Patricia C Dykes, Liqin Wang, Kumiko O Schnock, Jose P Garcia, Haomiao Jia, et al.  Unsupervised machine learning of topics documented by nurses about hospitalized patients prior to a rapid-response event.  Applied Clinical Informatics, 10(05):952–963, 2019.
  * [199] Satish M Mahajan and Rayid Ghani.  Combining structured and unstructured data for predicting risk of readmission for heart failure patients.  In MedInfo, pages 238–242, 2019.
  * [200] Mengxin Sun, Jason Baron, Anand Dighe, Peter Szolovits, Richard G Wunderink, Tamara Isakova, and Yuan Luo.  Early prediction of acute kidney injury in critical care setting using clinical notes and structured multivariate physiological measurements.  MedInfo, 264:368–72, 2019.
  * [201] Xingyu Zhang, M Fernanda Bellolio, Pau Medrano-Gracia, Konrad Werys, Sheng Yang, and Prashant Mahajan.  Use of natural language processing to improve predictive models for imaging utilization in children presenting to the emergency department.  BMC Medical Informatics and Decision Making, 19(1):1–13, 2019.
  * [202] AA Danielsen, MHJ Fenger, SD Østergaard, KL Nielbo, and O Mors.  Predicting mechanical restraint of psychiatric inpatients by applying machine learning on electronic health data.  Acta Psychiatrica Scandinavica, 140(2):147–157, 2019.
  * [203] Zina Ben Miled, Kyle Haas, Christopher M Black, Rezaul Karim Khandker, Vasu Chandrasekaran, Richard Lipton, and Malaz A Boustani.  Predicting dementia with routine care emr data.  Artificial Intelligence in Medicine, 102:101771, 2020.
  * [204] J Gong, X Bai, D-a Li, J Zhao, and X Li.  Prognosis analysis of heart failure based on recurrent attention model.  IRBM, 41(2):71–79, 2020.
  * [205] Stephen Bacchi, Samuel Gluck, Yiran Tan, Ivana Chim, Joy Cheng, Toby Gilbert, David K Menon, Jim Jannes, Timothy Kleinig, and Simon Koblar.  Prediction of general medical admission length of stay with natural language processing and deep learning: a pilot study.  Internal and emergency medicine, 15:989–995, 2020.
  * [206] Weijia Chen, Zhijun Lu, Lijue You, Lingling Zhou, Jie Xu, Ken Chen, et al.  Artificial intelligence–based multimodal risk assessment model for surgical site infection (amrams): development and validation study.  JMIR medical informatics, 8(6):e18186, 2020.
  * [207] Chien-Hua Chen, Jer-Guang Hsieh, Shu-Ling Cheng, Yih-Lon Lin, Po-Hsiang Lin, and Jyh-Horng Jeng.  Early short-term prediction of emergency department length of stay using natural language processing for low-acuity outpatients.  The American journal of emergency medicine, 38(11):2368–2373, 2020\.
  * [208] Marta Fernandes, Ruben Mendes, Susana M Vieira, Francisca Leite, Carlos Palos, Alistair Johnson, Stan Finkelstein, Steven Horng, and Leo Anthony Celi.  Predicting intensive care unit admission among patients presenting to the emergency department using machine learning and natural language processing.  PloS one, 15(3):e0229331, 2020.
  * [209] Marta Fernandes, Rúben Mendes, Susana M Vieira, Francisca Leite, Carlos Palos, Alistair Johnson, Stan Finkelstein, Steven Horng, and Leo Anthony Celi.  Risk of mortality and cardiopulmonary arrest in critical patients presenting to the emergency department using machine learning and natural language processing.  PLoS One, 15(4):e0230876, 2020.
  * [210] Christopher A Hane, Vijay S Nori, William H Crown, Darshak M Sanghavi, and Paul Bleicher.  Predicting onset of dementia using clinical notes and machine learning: case-control study.  JMIR Medical Informatics, 8(6):e17819, 2020.
  * [211] Jose Luis Izquierdo, Julio Ancochea, Savana COVID-19 Research Group, and Joan B Soriano.  Clinical characteristics and prognostic factors for intensive care unit admission of patients with covid-19: retrospective study using machine learning and natural language processing.  Journal of medical Internet research, 22(10):e21801, 2020.
  * [212] Zfania Tom Korach, Jie Yang, Sarah Collins Rossetti, Kenrick D Cato, Min-Jeoung Kang, Christopher Knaplund, Kumiko O Schnock, Jose P Garcia, Haomiao Jia, Jessica M Schwartz, et al.  Mining clinical phrases from nursing notes to discover risk factors of patient deterioration.  International journal of medical informatics, 135:104053, 2020.
  * [213] Yue Li, Pratheeksha Nair, Xing Han Lu, Zhi Wen, Yuening Wang, Amir Ardalan Kalantari Dehaghi, Yan Miao, Weiqi Liu, Tamas Ordog, Joanna M Biernacka, et al.  Inferring multimodal latent topics from electronic health records.  Nature communications, 11(1):2536, 2020.
  * [214] Bruno P Roquette, Hitoshi Nagano, Ernesto C Marujo, and Alexandre C Maiorano.  Prediction of admission in pediatric emergency department with deep neural networks and triage textual data.  Neural Networks, 126:170–177, 2020.
  * [215] Lucas Sterckx, Gilles Vandewiele, Isabelle Dehaene, Olivier Janssens, Femke Ongenae, Femke De Backere, Filip De Turck, Kristien Roelens, Johan Decruyenaere, Sofie Van Hoecke, et al.  Clinical information extraction for preterm birth risk prediction.  Journal of Biomedical Informatics, 110:103544, 2020.
  * [216] Maxim Topaz, Kyungmi Woo, Miriam Ryvicker, Maryam Zolnoori, and Kenrick Cato.  Home health care clinical notes predict patient hospitalization and emergency department visits.  Nursing research, 69(6):448, 2020.
  * [217] Rebecka Weegar and Karin Sundström.  Using machine learning for predicting cervical cancer from swedish electronic health records by mining hierarchical representations.  PloS one, 15(8):e0237911, 2020.
  * [218] Tomasz Oliwa, Brian Furner, Jessica Schmitt, John Schneider, and Jessica P Ridgway.  Development of a predictive model for retention in hiv care using natural language processing of clinical notes.  Journal of the American Medical Informatics Association, 28(1):104–112, 2021.
  * [219] Hang Dong, Víctor Suárez-Paniagua, William Whiteley, and Honghan Wu.  Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation.  Journal of biomedical informatics, 116:103728, 2021.
  * [220] Shuyuan Hu, Fei Teng, Lufei Huang, Jun Yan, and Haibo Zhang.  An explainable cnn approach for medical codes prediction from clinical text.  BMC Medical Informatics and Decision Making, 21:1–12, 2021.
  * [221] Emma L Barber, Ravi Garg, Christianne Persenaire, and Melissa Simon.  Natural language processing with machine learning to predict outcomes after ovarian cancer surgery.  Gynecologic oncology, 160(1):182–186, 2021.
  * [222] Maxwell Levis, Christine Leonard Westgate, Jiang Gui, Bradley V Watts, and Brian Shiner.  Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models.  Psychological medicine, 51(8):1382–1391, 2021.
  * [223] Eyal Klang, Benjamin R Kummer, Neha S Dangayach, Amy Zhong, M Arash Kia, Prem Timsina, Ian Cossentino, Anthony B Costa, Matthew A Levin, and Eric K Oermann.  Predicting adult neuroscience intensive care unit admission from emergency department triage using a retrospective, tabular-free text machine learning approach.  Scientific reports, 11(1):1381, 2021.
  * [224] Haiyang Yang, Li Kuang, and FengQiang Xia.  Multimodal temporal-clinical note network for mortality prediction.  Journal of Biomedical Semantics, 12(1):1–14, 2021.
  * [225] Byung-Hak Kim, Zhongfen Deng, Philip S Yu, and Varun Ganapathi.  Can current explainability help provide references in clinical notes to support humans annotate medical codes?  arXiv preprint arXiv:2210.15882, 2022.
  * [226] Guillermo López-García, José M Jerez, Nuria Ribelles, Emilio Alba, and Francisco J Veredas.  Explainable clinical coding with in-domain adapted transformers.  Journal of Biomedical Informatics, 139:104323, 2023.
  * [227] Usman Ahmed, Gautam Srivastava, Unil Yun, and Jerry Chun-Wei Lin.  Eandc: An explainable attention network based deep adaptive clustering model for mental health treatment.  Future Generation Computer Systems, 130:106–113, 2022.
  * [228] Md Zia Uddin, Kim Kristoffer Dysthe, Asbjørn Følstad, and Petter Bae Brandtzaeg.  Deep learning for prediction of depressive symptoms in a large textual dataset.  Neural Computing and Applications, 34(1):721–744, 2022.
  * [229] Yanji Qu, Xinlei Deng, Shao Lin, Fengzhen Han, Howard H Chang, Yanqiu Ou, Zhiqiang Nie, Jinzhuang Mai, Ximeng Wang, Xiangmin Gao, et al.  Using innovative machine learning methods to screen and identify predictors of congenital heart diseases.  Frontiers in Cardiovascular Medicine, 8:2087, 2022.
  * [230] Erol S Kavvas, Laurence Yang, Jonathan M Monk, David Heckmann, and Bernhard O Palsson.  A biochemically-interpretable machine learning classifier for microbial gwas.  Nature communications, 11(1):2580, 2020.
  * [231] Tao Sun, Yue Wei, Wei Chen, and Ying Ding.  Genome-wide association study-based deep learning for survival prediction.  Statistics in medicine, 39(30):4605–4620, 2020.
  * [232] Sanjoy Dey, Aritra Bose, Prithwish Chakraborty, Mohamed Ghalwash, Aldo Guzman Saenz, Filippo Utro, Kenney Ng, Jianying Hu, Laxmi Parida, and Daby Sow.  Impact of clinical and genomic factors on sars-cov2 disease severity.  medRxiv, 2021.
  * [233] Arno van Hilten, Steven A Kushner, Manfred Kayser, M Arfan Ikram, Hieab HH Adams, Caroline CW Klaver, Wiro J Niessen, and Gennady V Roshchupkin.  Gennet framework: interpretable deep learning for predicting phenotypes from genetic data.  Communications biology, 4(1):1094, 2021.
  * [234] Qi Yan, Yale Jiang, Heng Huang, Anand Swaroop, Emily Y Chew, Daniel E Weeks, Wei Chen, and Ying Ding.  Genome-wide association studies-based machine learning for prediction of age-related macular degeneration risk.  Translational vision science & technology, 10(2):29–29, 2021.
  * [235] Lianhe Zhao, Qiongye Dong, Chunlong Luo, Yang Wu, Dechao Bu, Xiaoning Qi, Yufan Luo, and Yi Zhao.  Deepomix: A scalable and interpretable multi-omics deep learning framework and application in cancer survival analysis.  Computational and structural biotechnology journal, 19:2719–2725, 2021.
  * [236] Hugues Turbé, Mina Bjelogrlic, Christian Lovis, and Gianmarco Mengaldo.  Evaluation of post-hoc interpretability methods in time-series classification.  Nature Machine Intelligence, pages 1–11, 2023.
  * [237] Radwa Elshawi, Mouaz H Al-Mallah, and Sherif Sakr.  On the interpretability of machine learning-based model for predicting hypertension.  BMC medical informatics and decision making, 19(1):1–32, 2019.
  * [238] Xueqin Pang, Christopher B Forrest, Félice Lê-Scherban, and Aaron J Masino.  Understanding early childhood obesity via interpretation of machine learning model predictions.  In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1438–1443. IEEE, 2019.
  * [239] Jiayi Lu, Renchao Jin, Enmin Song, Mubarak Alrashoud, Khaled N Al-Mutib, and Mabrook S Al-Rakhami.  An explainable system for diagnosis and prognosis of covid-19.  IEEE Internet of Things Journal, 8(21):15839–15846, 2020.
  * [240] Theekshana Dissanayake, Tharindu Fernando, Simon Denman, Sridha Sridharan, Houman Ghaemmaghami, and Clinton Fookes.  A robust interpretable deep learning classifier for heart anomaly detection without segmentation.  IEEE Journal of Biomedical and Health Informatics, 25(6):2162–2171, 2020.
  * [241] Xing Song, Alan SL Yu, John A Kellum, Lemuel R Waitman, Michael E Matheny, Steven Q Simpson, Yong Hu, and Mei Liu.  Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction.  Nature communications, 11(1):1–12, 2020.
  * [242] Amie J Barda, Christopher M Horvat, and Harry Hochheiser.  A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare.  BMC medical informatics and decision making, 20(1):1–16, 2020.
  * [243] Sergio Penafiel, Nelson Baloian, Horacio Sanson, and Jose A Pino.  Predicting stroke risk with an interpretable classifier.  IEEE Access, 9:1154–1166, 2020.
  * [244] Julian Hatwell, Mohamed Medhat Gaber, and R Muhammad Atif Azad.  Ada-whips: explaining adaboost classification with applications in the health sciences.  BMC Medical Informatics and Decision Making, 20(1):1–25, 2020.
  * [245] Simon Meyer Lauritsen, Mads Kristensen, Mathias Vassard Olsen, Morten Skaarup Larsen, Katrine Meyer Lauritsen, Marianne Johansson Jørgensen, Jeppe Lange, and Bo Thiesson.  Explainable artificial intelligence model to predict acute critical illness from electronic health records.  Nature communications, 11(1):1–11, 2020.
  * [246] Nicasia Beebe-Wang, Alex Okeson, Tim Althoff, and Su-In Lee.  Efficient and explainable risk assessments for imminent dementia in an aging cohort study.  IEEE Journal of Biomedical and Health Informatics, 25(7):2409–2420, 2021.
  * [247] Seong-Hwan Kim, Eun-Tae Jeon, Sungwook Yu, Kyungmi Oh, Chi Kyung Kim, Tae-Jin Song, Yong-Jae Kim, Sung Hyuk Heo, Kwang-Yeol Park, Jeong-Min Kim, et al.  Interpretable machine learning for early neurological deterioration prediction in atrial fibrillation-related stroke.  Scientific reports, 11(1):1–9, 2021.
  * [248] Md Rashed-Al-Mahfuz, Abedul Haque, Akm Azad, Salem A Alyami, Julian MW Quinn, and Mohammad Ali Moni.  Clinically applicable machine learning approaches to identify attributes of chronic kidney disease (ckd) for use in low-cost diagnostic screening.  IEEE Journal of Translational Engineering in Health and Medicine, 9:1–11, 2021.
  * [249] Xian Zeng, Yaoqin Hu, Liqi Shu, Jianhua Li, Huilong Duan, Qiang Shu, and Haomin Li.  Explainable machine-learning predictions for complications after pediatric congenital heart surgery.  Scientific reports, 11(1):1–11, 2021.
  * [250] Yihan Zhang, Dong Yang, Zifeng Liu, Chaojin Chen, Mian Ge, Xiang Li, Tongsen Luo, Zhengdong Wu, Chenguang Shi, Bohan Wang, et al.  An explainable supervised machine learning predictor of acute kidney injury after adult deceased donor liver transplantation.  Journal of translational medicine, 19(1):1–15, 2021.
  * [251] Ankit Pal and Malaikannan Sankarasubbu.  Pay attention to the cough: Early diagnosis of covid-19 using interpretable symptoms embeddings with cough sound signal processing.  In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 620–628, 2021.
  * [252] Marcos Antonio Alves, Giulia Zanon Castro, Bruno Alberto Soares Oliveira, Leonardo Augusto Ferreira, Jaime Arturo Ramírez, Rodrigo Silva, and Frederico Gadelha Guimarães.  Explaining machine learning based diagnosis of covid-19 from routine blood tests with decision trees and criteria graphs.  Computers in Biology and Medicine, 132:104335, 2021.
  * [253] Arturo Moncada-Torres, Marissa C van Maaren, Mathijs P Hendriks, Sabine Siesling, and Gijs Geleijnse.  Explainable machine learning can outperform cox regression predictions and provide insights in breast cancer survival.  Scientific Reports, 11(1):1–13, 2021.
  * [254] Christopher Duckworth, Francis P Chmiel, Dan K Burns, Zlatko D Zlatev, Neil M White, Thomas WV Daniels, Michael Kiuber, and Michael J Boniface.  Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during covid-19.  Scientific reports, 11(1):1–10, 2021.
  * [255] Anna Markella Antoniadi, Miriam Galvin, Mark Heverin, Orla Hardiman, and Catherine Mooney.  Prediction of caregiver quality of life in amyotrophic lateral sclerosis using explainable machine learning.  Scientific Reports, 11(1):1–13, 2021.
  * [256] Isaac Ronald Ward, Ling Wang, Juan Lu, Mohammed Bennamoun, Girish Dwivedi, and Frank M Sanfilippo.  Explainable artificial intelligence for pharmacovigilance: What features are important when predicting adverse outcomes?  Computer Methods and Programs in Biomedicine, 212:106415, 2021.
  * [257] Oliver Zhang, Cheng Ding, Tania Pereira, Ran Xiao, Kais Gadhoumi, Karl Meisel, Randall J Lee, Yiran Chen, and Xiao Hu.  Explainability metrics of deep convolutional networks for photoplethysmography quality assessment.  IEEE Access, 9:29736–29745, 2021.
  * [258] Lucas M Thimoteo, Marley M Vellasco, Jorge Amaral, Karla Figueiredo, Cátia Lie Yokoyama, and Erito Marques.  Explainable artificial intelligence for covid-19 diagnosis through blood test variables.  Journal of Control, Automation and Electrical Systems, 33(2):625–644, 2022.
  * [259] Jie Hao, Sai Chandra Kosaraju, Nelson Zange Tsaku, Dae Hyun Song, and Mingon Kang.  Page-net: interpretable and integrative deep learning for survival analysis using histopathological images and genomic data.  In Pacific Symposium on Biocomputing 2020, pages 355–366. World Scientific, 2019.
  * [260] Hao Ren, Aslan B Wong, Wanmin Lian, Weibin Cheng, Ying Zhang, Jianwei He, Qingfeng Liu, Jiasheng Yang, Chen Jason Zhang, Kaishun Wu, et al.  Interpretable pneumonia detection by combining deep learning and explainable models with multisource data.  IEEE Access, 9:95872–95883, 2021.
  * [261] Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al.  Pan-cancer integrative histology-genomic analysis via multimodal deep learning.  Cancer Cell, 40(8):865–878, 2022.
  * [262] Emily Herrett, Arlene M Gallagher, Krishnan Bhaskaran, Harriet Forbes, Rohini Mathur, Tjeerd Van Staa, and Liam Smeeth.  Data resource profile: clinical practice research datalink (cprd).  International journal of epidemiology, 44(3):827–836, 2015.
  * [263] Peter Kirkpatrick.  New clues in the acetaminophen mystery.  Nature Reviews Drug Discovery, 4(11):883–883, 2005.


# Entanglement asymmetry as a probe of symmetry breaking

Filiberto Ares  SISSA and INFN, via Bonomea 265, 34136 Trieste, Italy  Sara
Murciano  SISSA and INFN, via Bonomea 265, 34136 Trieste, Italy  Pasquale
Calabrese  SISSA and INFN, via Bonomea 265, 34136 Trieste, Italy  The Abdus
Salam International Center for Theoretical Physics, Strada Costiera 11, 34151
Trieste, Italy

(30th August 2024)

###### Abstract

Symmetry and symmetry breaking are two pillars of modern quantum physics.
Still, quantifying how much a symmetry is broken is an issue that has received
little attention. In extended quantum systems, this problem is intrinsically
bound to the subsystem of interest. Hence, in this work, we borrow methods
from the theory of entanglement in many-body quantum systems to introduce a
subsystem measure of symmetry breaking that we dub entanglement asymmetry. As
a prototypical illustration, we study the entanglement asymmetry in a quantum
quench of a spin chain in which an initially broken global $U(1)$ symmetry is
restored dynamically. We adapt the quasiparticle picture for entanglement
evolution to the analytic determination of the entanglement asymmetry. We
find, expectedly, that larger is the subsystem, slower is the restoration, but
also the counterintuitive result that more the symmetry is initially broken,
faster it is restored, a sort of quantum Mpemba effect.

## Introduction —

Symmetries hold a special place in every branch of physics, from relativity to
quantum mechanics, passing through gauge/gravity duality and numerical
algorithms. It is difficult to identify who was the first in understanding
their relevance since the transversal development of the subject is a huge
puzzle where different scientists, from Galileo to Noether, gave their own
remarkable contributions. Sometimes it happens that, when a parameter reaches
a critical value, the lowest energy configuration respecting the symmetry of
the theory becomes unstable and new asymmetric lowest energy solutions can be
found. This phenomenon does not require an input, whence the name spontaneous
symmetry breaking. Other times a symmetry can be explicitly broken, in the
sense that the Hamiltonian describing the system contains terms that
manifestly break it. The present work fits in this framework: our main goal is
to find a tool that measures quantitatively how much a symmetry is broken.

To be more specific, the setup we are interested in is an extended quantum
system in a pure state $\ket{\Psi}$, which we divide into two spatial regions
$A$ and $B$. The state of $A$ is described by the reduced density matrix
$\rho_{A}=\mathrm{Tr}_{B}(\ket{\Psi}\bra{\Psi})$. We consider a charge
operator $Q$ that generates a global $U(1)$ symmetry group, hence satisfying
$Q=Q_{A}+Q_{B}$. If $\ket{\Psi}$ is an eigenstate of $Q$, then
$[\rho_{A},Q_{A}]=0$ and $\rho_{A}$ displays a block-diagonal structure, with
each block corresponding to a charge sector of $Q_{A}$. Thus the entanglement
entropy $S(\rho_{A})=-\Tr(\rho_{A}\log\rho_{A})$, which measures how entangled
$A$ and $B$ are, can be decomposed into the contributions of each charge
sector lr-14 ; goldstein ; xavier ; MDC-20 ; riccarda ; mcp-22  (known as
symmetry-resolved entanglement), recently accessed also experimentally fis ;
Azses ; Neven ; Vitale .

Here we consider the opposite situation: a state $\ket{\Psi}$ that breaks the
global $U(1)$ symmetry. Therefore, $[\rho_{A},Q_{A}]\neq 0$ and $\rho_{A}$ is
not block-diagonal in the eigenbasis of $Q_{A}$. The goal of this work is to
introduce a quantifier of the symmetry breaking at the level of the subsystem
$A$, which is the entanglement asymmetry defined as

$\Delta S_{A}=S(\rho_{A,Q})-S(\rho_{A}).$ (1)

Here $\rho_{A,Q}=\sum_{q\in\mathbb{Z}}\Pi_{q}\rho_{A}\Pi_{q}$, where $\Pi_{q}$
is the projector onto the eigenspace of $Q_{A}$ with charge $q\in\mathbb{Z}$.
Thus $\rho_{A,Q}$ is block-diagonal in the eigenbasis of $Q_{A}$. In Fig. 1,
we pictorially show how $\rho_{A,Q}$ is obtained from $\rho_{A}$. A similar
quantity, but for the full system, was introduced in Ref. ms-21  to study the
inseparability of mixed states.

Figure 1: The density matrices $\rho_{A}$ and $\rho_{A,Q}$ entering in the
definitions (1) and (2) of the entanglement asymmetries. In the eigenbasis of
the subsystem charge $Q_{A}$, $\rho_{A}$ generically displays off-diagonal
elements. Under a projective measurement of $Q_{A}$, we get $\rho_{A,Q}$,
where the off-diagonal blocks are annihilated. The difference $\Delta
S_{A}^{(n)}$ between the entanglement entropies of these matrices is our probe
of symmetry breaking.

The entanglement asymmetry (1) satisfies two natural properties to quantify
symmetry breaking: $(i)$ $\Delta S_{A}\geq 0$, because by definition it is
equal to the relative entropy between $\rho_{A}$ and $\rho_{A,Q}$, $\Delta
S_{A}=\Tr[\rho_{A}(\log\rho_{A}-\log\rho_{A,Q})]$, which is actually non-
negative nielsen ; $(ii)$ $\Delta S_{A}=0$ if and only if the state is
symmetric since, in this case, $\rho_{A}$ is block diagonal in the eigenbasis
of $Q_{A}$ and $\rho_{A,Q}=\rho_{A}$.

##  A replica construction —

The entanglement asymmetry can be computed from the moments of the density
matrices $\rho_{A}$ and $\rho_{A,Q}$ by exploiting the replica trick holzhey94
; cc-04 . Indeed, simply defining the Rényi entanglement asymmetry as

$\Delta
S_{A}^{(n)}=\frac{1}{1-n}\left[\log\Tr(\rho_{A,Q}^{n})-\log\Tr(\rho_{A}^{n})\right],$
(2)

one has that $\displaystyle{\lim_{n\to 1}}\Delta S_{A}^{(n)}=\Delta S_{A}$. As
usual, the advantage of this construction is that for integer $n$, $\Delta
S_{A}^{(n)}$ can be accessed from (charged) partition functions. Indeed, using
the Fourier representation of the projector $\Pi_{q}$, the post-measurement
density matrix $\rho_{A,Q}$ can be alternatively written in the form

$\rho_{A,Q}=\int_{-\pi}^{\pi}\frac{{\rm d}\alpha}{2\pi}e^{-i\alpha
Q_{A}}\rho_{A}e^{i\alpha Q_{A}},$ (3)

and its moments as

$\Tr(\rho_{A,Q}^{n})=\int_{-\pi}^{\pi}\frac{{\rm d}\alpha_{1}\dots\rm
d\alpha_{n}}{(2\pi)^{n}}Z_{n}(\boldsymbol{\alpha}),$ (4)

where $\boldsymbol{\alpha}=\\{\alpha_{1},\dots,\alpha_{n}\\}$ and

$Z_{n}(\boldsymbol{\alpha})=\Tr\left[\prod_{j=1}^{n}\rho_{A}e^{i\alpha_{j,j+1}Q_{A}}\right],$
(5)

with $\alpha_{ij}\equiv\alpha_{i}-\alpha_{j}$ and $\alpha_{n+1}=\alpha_{1}$.
Notice that, if $[\rho_{A},Q_{A}]=0$, then
$Z_{n}(\boldsymbol{\alpha})=Z_{n}(\boldsymbol{0})$, which implies
$\Tr(\rho_{A,Q}^{n})=\Tr(\rho_{A}^{n})$ and $\Delta S_{A}^{(n)}=0$.
Furthermore the order of terms in Eq. (5) matters because
$[\rho_{A},Q_{A}]\neq 0$. We will refer to $Z_{n}(\boldsymbol{\alpha})$ as
charged moments because they are a modification of the similar quantities
introduced for the symmetry resolution of entanglement goldstein .

##  Tilted Ferromagnet —

As warm up, we consider an infinite spin chain prepared in the tilted
ferromagnetic state

$|\theta;\nearrow\nearrow\cdots\rangle=e^{i\frac{\theta}{2}\sum_{j}\sigma_{j}^{x}}|\uparrow\uparrow\cdots\rangle.$
(6)

For $\theta\neq\pi m$, $m\in\mathbb{Z}$, this state breaks the $U(1)$ symmetry
associated to the conservation of the total transverse magnetization
$Q=\frac{1}{2}\sum_{j}\sigma_{j}^{z}$. When $\theta=\pi m$, it corresponds to
a fully polarized state in the $z$-direction, for which the transverse
magnetization is preserved. The angle $\theta$ controls how much the state
breaks this symmetry and, therefore, the state (6) is an ideal testbed for the
entanglement asymmetry, although it is a trivial product state. Let the
subsystem $A$ consist of $\ell$ contiguous sites of the chain; then $\Delta
S_{A}=0$ for $\theta=\pi m$ and $\Delta S_{A}>0$ otherwise. Since the state is
separable, $\Tr(\rho_{A}^{n})=1$, and $Z_{n}(\boldsymbol{\alpha})$ is
straightforwardly obtained as

$Z_{n}(\boldsymbol{\alpha})=\prod_{j=1}^{n}f(\cos(\theta),\alpha_{j,j+1})^{\ell}\\\
{\rm where}\quad
f(\lambda,\alpha)=i\lambda\sin\left(\frac{\alpha}{2}\right)+\cos\left(\frac{\alpha}{2}\right).$
(7)

Plugging Eq. (7) into the Fourier transform (4), we obtain

$\Delta
S_{A}^{(n)}=\frac{1}{1-n}\log\left[\cos^{2n\ell}\Big{(}\frac{\theta}{2}\Big{)}\sum_{p=0}^{\ell}\binom{\ell}{p}^{n}\tan^{2np}\Big{(}\frac{\theta}{2}\Big{)}\right].$
(8)

In Fig. 2, we plot this entanglement asymmetry as a function of
$\theta\in[0,\pi]$. As expected, it vanishes for $\theta=0,\pi$ while it takes
the maximum value at $\theta=\pi/2$, when all the spins point in the $x$
direction and the symmetry is maximally broken. Between these extremal points,
$\Delta S_{A}$ is a monotonic function of $\theta$ (but this is not true for
all $n$). For a large interval, it behaves as

$\Delta S_{A}^{(n)}=\frac{1}{2}\log\ell+\frac{1}{2}\log\frac{\pi
n^{\frac{1}{n-1}}\sin^{2}\theta}{2}+O(\ell^{-1}).$ (9)

The limit $\theta\to 0$ is not well defined in Eq. (9). Indeed, the limits
$\ell\to\infty$ and $\theta\to 0$ do not commute: to recover the symmetry, one
should take first $\theta\to 0$ in Eq. (7) and then consider the large
interval regime.

Figure 2: The Rényi entanglement asymmetry $\Delta S_{A}^{(n)}$ in Eq. (9) for
the tilted ferromagnetic state with $\ell=10$ as a function of the tilting
angle $\theta$ for different values of the replica index $n$.

##  Quench to the XX spin chain —

We now analyze the time evolution of the entanglement asymmetry after a
quantum quench. We prepare the infinite spin chain in the state

$\ket{\Psi(0)}=\frac{\ket{\theta;\nearrow\nearrow\cdots}-\ket{-\theta;\nearrow\nearrow\cdots}}{\sqrt{2}},$
(10)

which is the cat version of the symmetry-breaking state in Eq. (6). We then
let it evolve

$|\Psi(t)\rangle=e^{-itH}|\Psi(0)\rangle,$ (11)

with the symmetric XX Hamiltonian ($[H,Q]=0$)

$H=-\frac{1}{4}\sum_{j=-\infty}^{\infty}\left[\sigma_{j}^{x}\sigma_{j+1}^{x}+\sigma_{j}^{y}\sigma_{j+1}^{y}\right].$
(12)

This Hamiltonian is diagonalized via the Jordan-Wigner transformation to
fermionic operators $\boldsymbol{c}_{j}=(c_{j}^{\dagger},c_{j})$ followed by a
Fourier transform to momentum space lms-61 . The one-particle dispersion
relation is $\epsilon(k)=-\cos(k)$.

##  The analytic computation —

We now embark in the exact computation of the time evolution of the charged
moments (5). The reader uninterested in these details can skip directly to the
next paragraph in which we discuss the results for the entanglement asymmetry.

We choose as initial state the linear combination of Eq. (10), instead of Eq.
(6), because the corresponding reduced density matrix is a Gaussian operator
in terms of $\boldsymbol{c}_{j}$. We can then use Wick theorem to express
$\rho_{A}(t)$ in terms of the two-point correlation matrix

$\Gamma_{jj^{\prime}}(t)=2\Tr\left[\rho_{A}(t)\boldsymbol{c}_{j}^{\dagger}\boldsymbol{c}_{j^{\prime}}\right]-\delta_{jj^{\prime}},$
(13)

with $j,j^{\prime}\in A$ p-03 . If $A$ is a subsystem of length $\ell$, then
$\Gamma(t)$ has dimension $2\ell\times 2\ell$ and entries FC08

$\Gamma_{jj^{\prime}}(t)=\int_{0}^{2\pi}\frac{{\rm
d}k}{2\pi}\mathcal{G}(k,t)e^{-ik(j-j^{\prime})},$ (14)

with

$\mathcal{G}(k,t)=\left(\begin{array}[]{cc}\cos\Delta_{k}&-ie^{-2it\epsilon(k)}\sin\Delta_{k}\\\
ie^{2it\epsilon(k)}\sin\Delta_{k}&-\cos\Delta_{k}\end{array}\right),\\\
\cos\Delta_{k}=\frac{2\cos(\theta)-(1+\cos^{2}\theta)\cos(k)}{1+\cos^{2}\theta-2\cos(\theta)\cos(k)}.$
(15)

Under the Jordan-Wigner transformation, the transverse magnetization is mapped
to the fermion number operator and $e^{i\alpha Q_{A}}$ turns out to be
Gaussian, too. Therefore, $Z_{n}(\boldsymbol{\alpha})$ in Eq. (5) is the trace
of the product of Gaussian fermionic operators, $\rho_{A}$ and
$e^{i\alpha_{j,j+1}Q_{A}}$. Employing their composition properties bb-69 ;
FC10 , we express $Z_{n}(\boldsymbol{\alpha})$ as a determinant involving the
corresponding correlation matrices, finding

$Z_{n}(\boldsymbol{\alpha},t)=\sqrt{\det\left[\left(\frac{I-\Gamma(t)}{2}\right)^{n}\left(I+\prod_{j=1}^{n}W_{j}(t)\right)\right]},$
(16)

with $W_{j}(t)=(I+\Gamma(t))(I-\Gamma(t))^{-1}e^{i\alpha_{j,j+1}n_{A}}$ and
$n_{A}$ is a diagonal matrix with $(n_{A})_{2j,2j}=1$,
$(n_{A})_{2j-1,2j-1}=-1$, $j=1,\cdots,\ell$. We use Eq. (16) to numerically
compute the time evolution of the Rényi entanglement asymmetry $\Delta
S^{(n)}_{A}(t)$ and test the analytical predictions presented in what follows.

Figure 3: Time evolution of $Z_{n}(\boldsymbol{\alpha},t)$ after the quench
(11) for $n=2$ (upper panel) and $n=3$ (lower panel). We plot it as a function
of $t/\ell$ for several values of $\theta$, $\ell$ and $\alpha_{j,j+1}$. The
symbols were obtained numerically using Eq. (16) and the continuous lines
correspond to the analytic prediction (18).

At time $t=0$, for large $\ell$, $Z_{n}(\boldsymbol{\alpha},0)\sim
e^{A(\boldsymbol{\alpha})\ell}/2^{n-1}$ with

$A(\boldsymbol{\alpha})=\log\prod_{j=1}^{n}e^{i\sigma_{j}/2}f(\cos(\theta),\alpha_{j,j+1}-\sigma_{j}),$
(17)

where $\sigma_{j}=0$ if $|\alpha_{j,j+1}|\leq\pi/2$ and $\sigma_{j}=\pi$
otherwise ($f(\lambda,\alpha)$ is defined in Eq. (7)). Therefore, $\Delta
S_{A}^{(n)}(t=0)$ behaves asymptotically as Eq. (9). After the quench, the
natural ballistic regime is the scaling limit $t,\ell\to\infty$ with
$\zeta=t/\ell$ fixed FC08 ; cef-12I  in which we expect

$Z_{n}(\boldsymbol{\alpha},t)=Z_{n}(\boldsymbol{0},t)e^{\ell(A(\boldsymbol{\alpha})+B(\boldsymbol{\alpha},\zeta))}.$
(18)

For the function $B(\boldsymbol{\alpha},\zeta)$, we conjecture

$B(\boldsymbol{\alpha},\zeta)=-\int_{0}^{2\pi}\frac{\mathrm{d}k}{2\pi}\mathrm{min}(2\zeta|\epsilon^{\prime}(k)|,1)\\\
\times\log\prod_{j=1}^{n}f(e^{i\Delta_{k}},\alpha_{j,j+1}),$ (19)

and $\epsilon^{\prime}(k)={\rm d}\epsilon(k)/{\rm d}k$ (likely this expression
can be rigorously derived by properly adapting the calculations for the
symmetry resolved entanglement pbc-21-1 ; pbc-21 , but this is far beyond the
scope of this work). This cumbersome expression does not come out of a
magician hat, but from the quasiparticle picture cc-05 ; ac-17 ; ac-18 : the
time evolution of the entanglement is given by the pairs of entangled
excitations shared by $A$ and $B$ that are created after the quench and
propagate ballistically with momentum $\pm k$. Let us explain how to apply
this idea to deduce Eq. (19). According to Refs. FCEC14 ; pvc-16 , in the
quench protocol analyzed here, the $U(1)$ symmetry is restored in the large
time limit, i.e. $\Delta S_{A}^{(n)}(t)\to 0$. Therefore,
$Z_{n}(\boldsymbol{\alpha},t)$ has to tend to $Z_{n}(\boldsymbol{0},t)$, which
implies $B(\boldsymbol{\alpha},\zeta)\to-A(\boldsymbol{\alpha})$ as
$\zeta\to\infty$. Considering Eq. (17), we notice that, at $t=0$ and large
$\ell$, $Z_{n}(\boldsymbol{\alpha},0)$ factorizes into

$Z_{n}(\boldsymbol{\alpha},0)\sim
2\prod_{j=1}^{n}\frac{e^{i\sigma_{j}/2}}{2}\mathrm{Tr}(\rho_{A}(0)e^{i(\alpha_{j,j+1}-\sigma_{j})Q_{A}}).$
(20)

The expectation value $\mathrm{Tr}(\rho_{A}(0)e^{i\alpha Q_{A}})$ is the full
counting statistics (FCS) of the transverse magnetization in the subsystem
$A$. We can now take advantage of the fact that $\ket{\Psi(0)}$ is also the
ground state of a XY spin chain 111The corresponding parameters $h,\gamma$ of
the XY chain are given by $\gamma^{2}+h^{2}=1$ and
$\cos^{2}\theta=(1-\gamma)/(1+\gamma)$. to exploit the knowledge of the FCS in
that system cherng07 ; ia-13 ; stephan14 ; ARV21 ; groha18 . In particular,
employing the results of Ref. ARV21 , we can rewrite $A(\boldsymbol{\alpha})$
in Eq. (17) as an integral in momentum space

$A(\boldsymbol{\alpha})=-B(\boldsymbol{\alpha},\zeta\to\infty)\\\
=\displaystyle\int_{0}^{2\pi}\frac{\mathrm{d}k}{2\pi}\log\prod_{j=1}^{n}f(e^{i\Delta_{k}},\alpha_{j,j+1}).$
(21)

Now, using the quasiparticle picture, the integrand in Eq. (21) can be
interpreted as the contribution to $B(\boldsymbol{\alpha},\zeta)$ from each
entangled excitation of momentum $k$ created after the quench. Since they
propagate with velocity $|\epsilon^{\prime}(k)|$, the number of these pairs
shared between $A$ and its complement at time $t$ is determined by
$\mathrm{min}(2t|\epsilon^{\prime}(k)|,\ell)$. Combining these two
ingredients, we get Eq. (18). This approach makes also clear the crucial role
that entanglement plays in the restoration of the symmetry. In Fig. 3, we
check Eq. (18) against exact numerical computations performed using Eq. (16)
for different values of $n$, $\theta$, and $\boldsymbol{\alpha}$, finding a
remarkable agreement: note that Eq. (18) is exact for $\ell\to\infty$ and the
points are closer to the curves for larger $\ell$.

Figure 4:  Time evolution of the Rényi entanglement asymmetry $\Delta
S_{A}^{(n)}(t)$ after the quench (11). The symbols are the exact numerical
results for various values of $\ell$, $n$, and $\theta$. The continuous lines
are our prediction obtained by plugging Eq. (18) into (4) and (2). In the
inset, we check the asymptotic behavior (The entanglement asymmetry after the
quench—) (full lines) and (23) (dashed) of $\Delta S_{A}^{(n)}(t)$ for large
$t/\ell$.

##  The entanglement asymmetry after the quench—

The entanglement asymmetry $\Delta S^{(n)}_{A}(t)$ for any $\zeta=t/\ell$ in
the scaling limit $t,\ell\to\infty$ is obtained by plugging the charged
moments (18) into Eq. (4). The resulting curves are plotted in Fig. 4 as a
function of $\zeta$ for several values of $\theta$, finding a remarkable
agreement with the exact numerical values (symbols). We can also write a very
effective closed-form approximation of $\Delta S^{(n)}_{A}(t)$. Indeed when in
Eq. (18) $A(\boldsymbol{\alpha})+B(\boldsymbol{\alpha},\zeta)$ is close to
zero, the Fourier transform can be done analytically as

$\displaystyle\Delta S_{A}^{(n)}$
$\displaystyle(t)\simeq\frac{\pi^{2}b(\zeta)\ell}{24},$ $\displaystyle
b(\zeta)=\frac{\sin^{2}\theta}{2}-\int_{0}^{2\pi}\frac{\mathrm{d}k}{2\pi}\mathrm{min}(2\zeta|\epsilon^{\prime}(k)|,1)\sin^{2}\Delta_{k},$

which is independent of the replica index $n$. This approximation becomes
exact in the limit of large $\zeta$ and its effectiveness, also for not too
large $\zeta$, is proven by the inset of Fig. 4.

We now discuss some relevant features of the entanglement asymmetry and show
that it encodes a lot of new physics. First, as expected, $\Delta
S_{A}^{(n)}(t)$ tends to zero for large $\zeta$ (i.e. large $t$) and the
$U(1)$ symmetry, broken by the initial state, is restored. This is
analytically shown by Eq. (The entanglement asymmetry after the quench—) that
indeed at leading order in large $\zeta$ is

$\Delta
S_{A}^{(n)}(t)=\frac{\pi}{1152}\Big{(}1+8\frac{\cos^{2}\theta}{\sin^{4}\theta}\Big{)}\frac{\ell}{\zeta^{3}}\,,$
(23)

i.e. it vanishes for large times as $t^{-3}$ for any value of $\theta$.
Another characteristic, following from having a space-time scaling, is that
larger subsystems require more time to recover the symmetry, as it is clear
from Fig. 4 and Eq. (The entanglement asymmetry after the quench—): this
justifies the significance of the definition of $\Delta S_{A}^{(n)}$ in terms
of $\rho_{A}$ rather than the full state $\ket{\Psi}$. Finally, a very odd and
intriguing feature is that the more the symmetry is initially broken, i.e. the
larger $\theta$, the smaller the time to restore it. This is a quantum Mpemba
effect mpe : more the system is out of equilibrium, the faster it relaxes. At
a qualitative level this is a consequence of the fact that for larger symmetry
breaking there is a sharper drop of the (entanglement) asymmetry at short
time, see Fig. 4, before the truly asymptotic behavior takes place.
Furthermore, we can quantitatively understand the quantum Mpemba effect: From
Eq. (23) the prefactor of the $t^{-3}$ decay is a monotonously decreasing
function of $\theta$ in $[0,\pi/2]$. Thus the quantum Mpemba effect is not as
controversial as its classical version mpemba . To the best of our knowledge
this awkward effect was not known in the literature, showing the power of the
entanglement asymmetry to identify new physics.

##  Conclusions—

In this work, we introduced the entanglement asymmetry, a probe to study how
much a symmetry is broken at the level of subsystems of many-body systems. As
an application to show its potential, we have studied its dynamics after a
quench from an initial state breaking a $U(1)$ symmetry and evolving with a
Hamiltonian preserving it. We showed that the entanglement asymmetry detects
neatly all the physical relevant features of the dynamics and in particular
the restoration of the symmetry at late times. It also identifies the
appearance of an unexpected Mpemba effect, a phenomenon that very likely
happens in many settings that can be studied through the entanglement
asymmetry. It is then very important to study other quench protocols (e.g.
different initial state, interacting Hamiltonians, etc.) and understand how to
modify the quasiparticle description, following e.g. Ref. bkalc-22 , to
describe these more general situations.

We can easily imagine many other applications of the entanglement asymmetry.
The first one is in equilibrium situations that have been left out here. In
this respect, it would be useful to recast the charged moments (5) in terms of
twist fields cc-04 ; cdca-07  within the path-integral approach: this would
allow us to explore more complicated situations, e.g. the symmetry breaking
from $SU(2)$ to $U(1)$, which are also relevant in high-energy physics.
Similarly, our setup can be extended to non-Abelian symmetries cdm-21  to
explore, e.g., how the asymptotic behavior of $\Delta S_{A}$ with the
subsystem size of Eq. (9) depends on the symmetry group.

Finally, $\Delta S_{A}(t)$ can be experimentally accessible by developing a
protocol based on the random measurement toolbox Vermersch2019scrambling ;
RMtoolbox ; shadows . This would require the post-selection of data from an
experiment like the one in brydges-2019 , but with an initial state breaking
the $U(1)$ symmetry.

##  Acknowledgements—

We thank Jerome Dubail, Viktor Eisler, Maurizio Fagotti, Israel Klich, Lorenzo
Piroli, Eric Vernier, and Lenart Zadnik for useful discussions. The authors
acknowledge support from ERC under Consolidator grant number 771536 (NEMO).

F.A. and S.M. contributed equally to this manuscript.

## References

  * (1) N. Laflorencie and S. Rachel, Spin-resolved entanglement spectroscopy of critical spin chains and Luttinger liquids, J. Stat. Mech. (2014) P11013.
  * (2) M. Goldstein and E. Sela, Symmetry-Resolved Entanglement in Many-Body Systems, Phys. Rev. Lett. 120, 200602 (2018).
  * (3) J. C. Xavier, F. C. Alcaraz, and G. Sierra, Equipartition of the entanglement entropy, Phys. Rev. B 98, 041106 (2018).
  * (4) R. Bonsignori, P. Ruggiero, and P. Calabrese, Symmetry resolved entanglement in free fermionic systems, J. Phys. A 52, 475302 (2019).
  * (5) S. Murciano, G. Di Giulio, and P. Calabrese, Entanglement and symmetry resolution in two dimensional free quantum field theories, JHEP 08 (2020) 073.
  * (6) S. Murciano, P. Calabrese, and L. Piroli, Symmetry-resolved Page curves, arXiv.2206.0508.
  * (7) A. Lukin, M. Rispoli, R. Schittko, M. E. Tai, A. M. Kaufman, S. Choi, V. Khemani, J. Leonard, and M. Greiner, Probing entanglement in a many-body localized system, Science 364, 6437 (2019).
  * (8) D. Azses, R. Haenel, Y. Naveh, R. Raussendorf, E. Sela, and E. G. Dalla Torre, Identification of Symmetry-Protected Topological States on Noisy Quantum Computers, Phys. Rev. Lett. 125, 120502 (2020).
  * (9) A. Neven, J. Carrasco, V. Vitale, C. Kokail, A. Elben, M. Dalmonte, P. Calabrese, P. Zoller, B. Vermersch, R. Kueng, and B. Kraus, Symmetry-resolved entanglement detection using partial transpose moments, Npj Quantum Inf. 7, 152 (2021).
  * (10) V. Vitale, A. Elben, R. Kueng, A. Neven, J. Carrasco, B. Kraus, P. Zoller, P. Calabrese, B. Vermersch, and M. Dalmonte, Symmetry-resolved dynamical purification in synthetic quantum matter, SciPost Phys. 12, 106 (2022).
  * (11) Z. Ma, C. Han, Y. Meir, and E. Sela, Symmetric inseparability and number entanglement in charge conserving mixed states, Phys. Rev. A 105, 042416 (2022).
  * (12) M. A. Nielsen and I. L. Chuang, Quantum computation and quantum information, Cambridge University Press, Cambridge, UK, 10th anniversary ed. (2010).
  * (13) C. Holzhey, F. Larsen, and F. Wilczek, Geometric and renormalized entropy in conformal field theory, Nucl. Phys. B 424, 443 (1994).
  * (14) P. Calabrese and J. Cardy, Entanglement entropy and quantum field theory, J. Stat. Mech. (2004) P06002.
  * (15) E. Lieb, T. Schultz, and D. Mattis, Two soluble models of an antiferromagnetic chain, Ann. Phys. 16, 407 (1961).
  * (16) I. Peschel, Calculation of reduced density matrices from correlation functions, J. Phys. A 36, L205 (2003).
  * (17) M. Fagotti and P. Calabrese, Evolution of entanglement entropy following a quantum quench: Analytic results for the XY chain in a transverse magnetic field, Phys. Rev. A 78, 010306 (2008).
  * (18) R. Balian and E. Brezin, Nonunitary Bogoliubov transformations and extension of Wick’s theorem, Il Nuovo Cimento B 64, 37 (1969).
  * (19) M. Fagotti and P. Calabrese, Entanglement entropy of two disjoint blocks in XY chains, J. Stat. Mech. (2010) P04016.
  * (20) P. Calabrese, F. H. L. Essler, and M. Fagotti, Quantum Quench in the Transverse Field Ising chain I: Time evolution of order parameter correlators, J. Stat. Mech. (2012) P07016.
  * (21) G. Parez, R. Bonsignori, and P. Calabrese, Quasiparticle dynamics of symmetry resolved entanglement after a quench: the examples of conformal field theories and free fermions, Phys. Rev. B 103, L041104 (2021).
  * (22) G. Parez, R. Bonsignori, and P. Calabrese, Exact quench dynamics of symmetry resolved entanglement in a free fermion chain, J. Stat. Mech. 093102 (2021).
  * (23) P. Calabrese and J. Cardy, Evolution of Entanglement Entropy in One-Dimensional Systems, J. Stat. Mech. (2005) P04010.
  * (24) V. Alba and P. Calabrese, Entanglement and thermodynamics after a quantum quench in integrable systems, PNAS 114, 7947 (2017).
  * (25) V. Alba and P. Calabrese, Entanglement dynamics after quantum quenches in generic integrable systems, SciPost Phys. 4, 017 (2018).
  * (26) M. Fagotti, M. Collura, F. H. L. Essler, P. Calabrese, Relaxation after quantum quenches in the spin-1/2 Heisenberg XXZ chain, Phys. Rev. B 89, 125101 (2014).
  * (27) L. Piroli, E. Vernier, and P. Calabrese, Exact steady states for quantum quenches in integrable Heisenberg spin chains, Phys. Rev. B 94, 054313 (2016).
  * (28) R. W. Cherng and E. Demler, Quantum Noise Analysis of Spin Systems Realized with Cold Atoms, New J. Phys. 9, 7 (2007).
  * (29) D. A. Ivanov and A. G. Abanov, Characterizing correlations with full counting statistics: classical Ising and quantum XY spin chains, Phys. Rev. E 87, 022114 (2013).
  * (30) J.-M. Stéphan, Emptiness formation probability, Toeplitz determinants, and conformal field theory, J. Stat. Mech. (2014) P05010.
  * (31) S. Groha, F. H. L. Essler, and P. Calabrese, Full Counting Statistics in the Transverse Field Ising Chain, SciPost Phys. 4, 043 (2018).
  * (32) F. Ares, M. A. Rajabpour, and J. Viti, Exact full counting statistics for the staggered magnetization and the domain walls in the XY spin chain, Phys. Rev. E 103, 042107 (2021).
  * (33) E. B. Mpemba and D. G. Osborne, Cool?,  Phys. Educ. 4, 172 (1969).
  * (34) A. Kumar and J. Bechhoefer, Exponentially faster cooling in a colloidal system, Nature 584, 64 (2020).
  * (35) B. Bertini, K. Klobas, V. Alba, G. Lagnese, and P. Calabrese Growth of Rényi Entropies in Interacting Integrable Models and the Breakdown of the Quasiparticle Picture, arXiv:2203.17264.
  * (36) J. Cardy, B. Doyon, and O. A. Castro-Alvaredo, Form factors of branch-point twist fields in quantum integrable models and entanglement entropy, J. Stat. Phys. 130, 129 (2008).
  * (37) P. Calabrese, J. Dubail, and S. Murciano, Symmetry-resolved entanglement entropy in Wess-Zumino-Witten models, JHEP 10 (2021) 067.
  * (38) B. Vermersch, A. Elben, L. M. Sieberer, N. Y. Yao, and P. Zoller, Probing scrambling using statistical correlations between randomized measurements, Phys. Rev. X 9, 021061 (2019).
  * (39) A. Elben, S. T. Flammia, H.-Y. Huang, R. Kueng, J. Preskill, B. Vermersch, and P. Zoller, The randomized measurement toolbox, arXiv:2203.11374.
  * (40) H.-Y. Huang, R. Kueng, and J. Preskill, Predicting many properties of a quantum system from very few measurements, Nature Phys. 16, 1050 (2020).
  * (41) T. Brydges, A. Elben, P. Jurcevic, B. Vermersch, C. Maier, B. P. Lanyon, P. Zoller, R. Blatt, and C. F. Roos, Probing entanglement entropy via randomized measurements, Science 364, 260 (2019).