ATL-PHYS-PROC-2018-120 27th August 2024

Search for charged lepton-flavour violation in top-quark decays at the LHC
with the ATLAS detector

Carlo A. Gottardo111Supported by European Research Council grant ERC-
CoG-617185., on behalf of the ATLAS Collaboration11footnotetext: Copyright
2018 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0 license.

Physikalisches Institut
Universität Bonn

> A direct search for charged lepton-flavour violation in top-quark decays is
> presented. The data analysed correspond to $79.8\ \text{fb}^{-1}$ of
> proton–proton collisions at a centre-of-mass energy of $\sqrt{s}=13\
> \text{TeV}$ recorded by the ATLAS experiment at the LHC. The process studied
> is the production of top-quark pairs, where one top quark decays into a pair
> of opposite-sign different-flavour charged leptons and an up-type quark,
> while the other decays semileptonically according to the Standard Model. The
> signature of the signal is thus characterised by the presence of three
> charged leptons, a light jet and a $b$-jet. A multivariate discriminant is
> deployed and its distribution used as input to extract the signal strength.
> In the absence of a signal, an upper limit on the branching ratio of ${\cal
> B}(t\to\ell\ell^{\prime}q)<1.86\times 10^{-5}$ is set at the 95% confidence
> level.

> PRESENTED AT
>
>
>
>
> $11^{\mathrm{th}}$ International Workshop on Top Quark Physics
> Bad Neuenahr, Germany, September 16–21, 2018

##  1 Introduction

The observation of charged lepton-flavour violation (cLFV) would provide a
strong evidence for New Physics, beyond the simple inclusion of right-handed
neutrinos in the Standard Model (SM) [1]. Several extensions to the SM entail
cLFV [1], but a model-independent approach, based on effective field theory
(EFT), is adopted here. The set of operators reported in Equation 1,
describing two-lepton, two-quarks contact interactions, induces cLFV in
processes involving the top quark. The notation $e_{i}=\\{e,\mu,\tau\\}$ and
$u_{q}=\\{u,c\\}$ is used, and $H,H^{\prime},\bar{H}=L,R$ indicate the
chirality of the projectors $P$, with $H\neq\bar{H}$.

$\displaystyle\mathcal{O}^{AV}_{HH^{\prime}}=(\bar{e}_{i}\gamma^{\alpha}P_{H}e_{j})(\bar{u}_{q}\gamma_{\alpha}P_{H^{\prime}}t),$
(1)
$\displaystyle\mathcal{O}^{S+P}_{H}=(\bar{e}_{i}P_{H}e_{j})(\bar{u}_{q}P_{H}t),$
$\displaystyle\mathcal{O}^{S-P}_{H}=(\bar{e}_{i}P_{H}e_{j})(\bar{u}_{q}P_{\bar{H}}t),$
$\displaystyle\mathcal{O}^{LQ}_{H}=(\bar{u}_{q}P_{H}e_{j})(\bar{e}_{i}P_{H}t).$

The couplings of the operators are rather unconstrained, allowing for a
branching fraction of the decay $t\to e\mu q$, with $q=u,c$, of order
$10^{-3}$, well within the sensitivity of the Large Hadron Collider (LHC) and
not previously probed directly [2]. Taking advantage of the copious production
of top-quark pairs at the LHC, the present analysis investigates the decay
$t(\bar{t})\to\ell^{\pm}\ell^{{}^{\prime}\mp}q(\ell^{\pm}\ell^{{}^{\prime}\mp}\bar{q})$
with $\ell=\\{e,\mu,\tau\\}$ and $q=\\{u,c\\}$, in $t\bar{t}$ events, where
the other top quark of the pair decays semileptonically, according to the SM.
Signal events are generated at leading order precision in QCD with
MadGraph5_aMC@NLO interfaced with Pythia. The response of the detector is
simulated with GEANT4. The search is performed using the $pp$ collision data
collected by ATLAS [3] in 2015, 2016 and 2017, corresponding to an integrated
luminosity of $79.8\text{\,}\text{fb}^{-1}$. More details about the analysis
can be found in Reference [4].

##  2 Analysis strategy

Only events with three reconstructed charged light leptons (electrons or
muons, called leptons in the following) and at least two jets are selected.
Events having two opposite-sign same-flavour leptons whose invariant mass is
within $81.2$ and $101.2$ GeV, or less then $15$ GeV are vetoed. At most one
$b$-tagged jet is allowed. The signal region is defined by requiring the
presence of at least one electron and at least one muon. The signal kinematic
is reconstructed, and a multivariate discriminant, namely a boosted decision
tree (BDT), is trained in the signal region in order to separate the signal
from all backgrounds. Thirteen input variables are selected and provided to
the BDT, which is trained on simulated events. Several sources of background
populate the signal region: $t\bar{t}$, $Z+\text{jets}$, diboson production,
$t\bar{t}Z$, $t\bar{t}W$, $t\bar{t}H$, associated single-top production as
$tZ$, $tWZ$ and $tH$, and other minor processes. Despite the fact that three
isolated leptons are required, $t\bar{t}$ and $Z+\text{jets}$ constitute the
largest fraction of background (60% in the signal region), due to the
occurrence of additional non-prompt leptons originating from secondary
processes, such as the decay of a $b$\- or $c$-hadrons, photon conversion or
object mis-reconstruction. This so-called non-prompt lepton background is
related to the details in the particles’ interactions with the detector
material, therefore it has been modelled with a data-driven approach. The
technique used provides, through data events reweighting, both normalisation
and shape predictions for the non-prompt lepton background. The total expected
background is obtained by combining the data-driven non-prompt lepton
background prediction with simulated events containing three prompt leptons in
the region’s acceptance.

Systematic uncertainties, including both normalisation and shape components,
are derived and assigned to the data-driven non-prompt lepton background.
Modelling and instrumental uncertainties are assigned to the simulated event
samples. The most impacting uncertainties are those associated to the non-
prompt lepton background, which correspond to a 10% variation in the total
background yield and to a 9% variation in the observed limit.

##  3 Results

The expected and observed event yields in the signal region are summarized in
Table 1. The full BDT distribution, presented in Figure 1, is used as input
for a binned profile-likelihood fit, meant to test for the presence of signal
events. The data are found to be compatible with the absence of the signal and
an upper limit is set at the $95\text{\,}\mathrm{\char 37\relax}$ confidence
level, using the $CL_{s}$ method. The observed and expected limits on the cLFV
decay branching ratio are:

${\cal B}(t\to\ell\ell^{\prime}q)<1.36^{+0.61}_{-0.37}\times
10^{-5}\quad(\text{expected}),$ ${\cal B}(t\to\ell\ell^{\prime}q)<1.86\times
10^{-5}\quad(\text{observed}).$

The upper limit is recomputed removing all generated signal events where a
$\tau$ lepton is present in the cLFV decay vertex, resulting in:

${\cal B}(t\to e\mu q)<4.8^{+2.1}_{-1.4}\times 10^{-6}\quad(\text{no $\tau$ in
cLFV vertex, expected}),$ ${\cal B}(t\to e\mu q)<6.6\times
10^{-6}\quad(\text{no $\tau$ in cLFV vertex, observed}).$

Table 1: Background and data events expected in the signal region [4]. Non-prompt | $WZ$ | $ZZ$ | $t\bar{t}V$ | Other | Expected | Data
---|---|---|---|---|---|---
leptons |  |  |  | prompt SM | events |
1190(180) | $350(140)$ | $140(52)$ | $108(10)$ | $76(10)$ | $1860(230)$ | 1857
Figure 1: BDT discriminant distribution, with the signal including and
excluding $\tau$ leptons (Signal $\tau$-veto) in the cLFV vertex overlaid. The
signals are normalised to the branching fraction ${\cal
B}(t\to\ell^{\pm}\ell^{\prime\mp}q)=3\times 10^{-4}$ and ${\cal B}(t\to e\mu
q)=1\times 10^{-4}$, respectively. All sources of systematic uncertainty are
included. Data (black points) are compared to the sum of backgrounds in the
upper panel, while the ratio is shown in the lower panel [4].

## References

  * [1] L. Calibbi and G. Signorelli, Riv. Nuovo Cim. 41, 1 (2018) [arXiv:1709.00294].
  * [2] S. Davidson, M. L. Mangano, S. Perries and V. Sordini, Eur. Phys. J. C 75 (2015) 450 [arXiv:1507.07163].
  * [3] ATLAS Collaboration, JINST 3 (2008) S08003.

  * [4] ATLAS Collaboration, ATLAS-CONF-2018-044.
https://cds.cern.ch/record/2638305


* [243] L. Wu, F. Morstatter, K. M. Carley, and H. Liu, ‘Misinformation in Social Media: Definition, Manipulation, and Detection’, ACM SIGKDD Explorations Newsletter, vol. 21, no. 2, pp. 80–90, Nov. 2019.
  * [244] T. Rasool, W. H. Butt, A. Shaukat, and M. U. Akram, ‘Multi-Label Fake News Detection using Multi-layered Supervised Learning’, in Proceedings of the 2019 11th International Conference on Computer and Automation Engineering, 2019, pp. 73–77.
  * [245] H. Karimi, P. Roy, S. Saba-Sadiya, and J. Tang, ‘Multi-Source Multi-Class Fake News Detection’, in Proceedings of the 27th International Conference on Computational Linguistics, 2018, pp. 1546–1557.
  * [246] Z. Jin, J. Cao, Y.-G. Jiang, and Y. Zhang, ‘News Credibility Evaluation on Microblog with a Hierarchical Propagation Model’, in 2014 IEEE International Conference on Data Mining, 2014, pp. 230–239.
  * [247] Z. Jin, J. Cao, Y. Zhang, and J. Luo, ‘News Verification by Exploiting Conflicting Social Viewpoints in Microblogs’, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, Mar. 2016.
  * [248] S. Jindal, R. Sood, R. Singh, M. Vatsa, and T. Chakraborty, ‘NewsBag: A Multimodal Benchmark Dataset for Fake News Detection’.
  * [249] Z. Jin, J. Cao, Y. Zhang, J. Zhou, and Q. Tian, ‘Novel Visual and Statistical Image Features for Microblogs News Verification’, IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 598–608, Mar. 2017.
  * [250] M. D. Vicario, W. Quattrociocchi, A. Scala, and F. Zollo, ‘Polarization and Fake News: Early Warning of Potential Misinformation Targets’, ACM Transactions on the Web, vol. 13, no. 2, p. 10:1-10:22, Mar. 2019.
  * [251] C. Budak, D. Agrawal, and A. El Abbadi, ‘Limiting the spread of misinformation in social networks’, in Proceedings of the 20th international conference on World wide web, 2011, pp. 665–674.
  * [252] ‘Web-based statistical fact checking of textual documents |Proceedings of the 2nd international workshop on Search and mining user-generated contents.’
  * [253] S. Kwon, M. Cha, K. Jung, W. Chen, and Y. Wang, ‘Prominent Features of Rumor Propagation in Online Social Media’, in 2013 IEEE 13th International Conference on Data Mining, 2013, pp. 1103–1108.
  * [254] Q. Zhang, A. Lipani, S. Liang, and E. Yilmaz, ‘Reply-Aided Detection of Misinformation via Bayesian Deep Learning’, in The World Wide Web Conference, 2019, pp. 2333–2343.
  * [255] S. Kwon, M. Cha, and K. Jung, ‘Rumor Detection over Varying Time Windows’, PLOS ONE, vol. 12, no. 1, p. e0168344, Jan. 2017.
  * [256] H. Guo, J. Cao, Y. Zhang, J. Guo, and J. Li, ‘Rumor Detection with Hierarchical Social Attention Network’, in Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018, pp. 943–951.
  * [257] X. Lin, X. Liao, T. Xu, W. Pian, and K.-F. Wong, ‘Rumor Detection with Hierarchical Recurrent Convolutional Neural Network’, in Natural Language Processing and Chinese Computing, vol. 11839, J. Tang, M.-Y. Kan, D. Zhao, S. Li, and H. Zan, Eds. Cham: Springer International Publishing, 2019, pp. 338–348.
  * [258] L. Zeng, K. Starbird, and E. S. Spiro, ‘Rumors at the Speed of Light? Modeling the Rate of Rumor Transmission During Crisis’, in 2016 49th Hawaii International Conference on System Sciences (HICSS), 2016, pp. 1969–1978.
  * [259] P. Dewan and P. Kumaraguru, "Towards automatic real time identification of malicious posts on Facebook," 2015 13th Annual Conference on Privacy, Security and Trust (PST), Izmir, Turkey, 2015, pp. 85-92.
  * [260] ‘Rumors, False Flags, and Digital Vigilantes: Misinformation on Twitter after the 2013 Boston Marathon Bombing’, in iConference 2014 Proceedings, 2014.
  * [261] L. Cui, S. Wang, and D. Lee, ‘SAME: Sentiment-Aware Multi-Modal Embedding for Detecting Fake News’, in 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 2019, pp. 41–48.
  * [262] O. Ajao, D. Bhowmik, and S. Zargari, ‘Sentiment Aware Fake News Detection on Online Social Networks’, in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 2507–2511.
  * [263] L. Zhao, H. Cui, X. Qiu, X. Wang, and J. Wang, ‘SIR rumor spreading model in the new media age’, Physica A: Statistical Mechanics and its Applications, vol. 392, no. 4, pp. 995–1003, Feb. 2013.
  * [264] S. Singhal, R. R. Shah, T. Chakraborty, P. Kumaraguru, and S. Satoh, ‘SpotFake: A Multi-modal Framework for Fake News Detection’, in 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), 2019, pp. 39–47.
  * [265] A. Mukherjee, B. Liu, and N. Glance, ‘Spotting fake reviewer groups in consumer reviews’, in Proceedings of the 21st international conference on World Wide Web, 2012, pp. 191–200.
  * [266] D. Acemoglu, A. Ozdaglar, and A. ParandehGheibi, ‘Spread of (mis)information in social networks’, Games and Economic Behavior, vol. 70, no. 2, pp. 194–227, Nov. 2010.
  * [267] I. Y. R. Pratiwi, R. A. Asmara, and F. Rahutomo, ‘Study of hoax news detection using naïve bayes classifier in Indonesian language’, in 2017 11th International Conference on Information & Communication Technology and System (ICTS), 2017, pp. 73–78.
  * [268] K. Shu, H. R. Bernard, and H. Liu, ‘Studying Fake News via Network Analysis: Detection and Mitigation’. arXiv, Apr-2018.
  * [269] J. C. S. Reis, A. Correia, F. Murai, A. Veloso, and F. Benevenuto, ‘Supervised Learning for Fake News Detection’, IEEE Intelligent Systems, vol. 34, no. 2, pp. 76–81, Mar. 2019.
  * [270] J. C. S. Reis, A. Correia, F. Murai, A. Veloso, and F. Benevenuto, ‘Explainable Machine Learning for Fake News Detection,’ Web Science ’19, Boston, MA, USA, 2019, pp. 17-26.
  * [271] F. Torabi Asr and M. Taboada, ‘The Data Challenge in Misinformation Detection: Source Reputation vs. Content Veracity’, in Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), 2018, pp. 10–15.
  * [272] K. Shu, X. Zhou, S. Wang, R. Zafarani, and H. Liu, ‘The role of user profiles for fake news detection’, in Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 2020, pp. 436–439.
  * [273] C. Shao, G. L. Ciampaglia, O. Varol, K.-C. Yang, A. Flammini, and F. Menczer, ‘The spread of low-credibility content by social bots’, Nature Communications, vol. 9, no. 1, p. 4787, Nov. 2018.
  * [274] J. Li, M. Ott, C. Cardie, and E. Hovy, ‘Towards a General Rule for Identifying Deceptive Opinion Spam’, in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2014, pp. 1566–1576.
  * [275] S. Jain, V. Sharma, and R. Kaushal, ‘Towards automated real-time detection of misinformation on Twitter’, in 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2016, pp. 2015–2020.
  * [276] V. L. Rubin, N. J. Conroy, and Y. Chen, ‘Towards News Verification: Deception Detection Methods for News Discourse’.
  * [277] V. L. Rubin and T. Lukoianova, ‘Truth and deception at the rhetorical structure level: Truth and Deception at the Rhetorical Structure Level’, Journal of the Association for Information Science and Technology, vol. 66, no. 5, pp. 905–917, May 2015.
  * [278] A. Gupta, P. Kumaraguru, C. Castillo, and P. Meier, ‘TweetCred: Real-Time Credibility Assessment of Content on Twitter’. arXiv, Jan-2015. [258] Z. Yang, C. Wilson, and X. Wang, ‘Uncovering Social Network Sybils in the Wild’.
  * [279] K. Darwish, P. Stefanov, M. Aupetit, and P. Nakov, ‘Unsupervised User Stance Detection on Twitter’. arXiv, May-2020.
  * [280] B. Nyhan and J. Reifler, ‘When Corrections Fail: The Persistence of Political Misperceptions’, Political Behavior, vol. 32, no. 2, pp. 303–330, Jun. 2010.
  * [281] D. Katsaros, G. Stavropoulos, and D. Papakostas, ‘Which machine learning paradigm for fake news detection?’, in 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI), 2019, pp. 383–387.
  * [282] ‘Snopes’. www.snopes.com.
  * [283] ‘PolitiFact’. www.politifact.com.
  * [284] C. Silverman, C. Timberg, J. Kao, and J. B. Merrill, ‘Facebook groups topped 10,000 daily attacks on election before Jan. 6, analysis shows’, The Washington Post. WP Company, Jan-2022.
  * [285] M. Hindman and V. Barash, ‘Disinformation, ‘Fake News’ and Influence Campaigns on Twitter’, Knight Foundation. Knight Foundation, Oct-2018.
  * [286] S. Castelo et al., ‘A Topic-Agnostic Approach for Identifying Fake News Pages’, Companion Proceedings of The 2019 World Wide Web Conference. ACM, New York, NY, USA, pp. 975–980, 2019.
  * [287] H. Allcott and M. Gentzkow, ‘Social Media and Fake News in the 2016 Election’, Journal of Economic Perspectives, vol. 31, no. 2, pp. 211–236, May 2017.
  * [288] T. Vaidya, D. Votipka, M. L. Mazurek, and M. Sherr, ‘Does Being Verified Make You More Credible? Account Verification’s Effect on Tweet Credibility’, in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland Uk, 2019, pp. 1–13.
  * [289] Government News, ‘The Weibo rumor-refuting and co-governance platform is launched, giving government accounts the right to directly refute rumors’, Nov. 2018.
  * [290] I. Mehta and M. Singh, ‘Twitter to end free access to its API in Elon Musk’s latest monetization push’, TechCrunch. Feb-2023.
  * [291] J. Porter, ‘Twitter announces new API pricing, posing a challenge for small developers’, The Verge. The Verge, Mar-2023.
  * [292] ‘Developer policy – twitter developers | twitter developer platform’, Twitter. Twitter.
  * [293] S. McCarthy, ‘China’s promotion of Russian disinformation indicates where its loyalties lie’, CNN. Cable News Network, Mar-2022.
  * [294] Archive Team, ‘The Twitter Stream Grab’, Internet Archive. Internet Archive.
  * [295] ‘Media Bias/Fact Check News’, Media Bias/Fact Check. Jul-2021.
  * [296] ‘Fake news detection datasets’, ISOT research lab.
  * [297] F. K. Abu Salem, R. Al Feel, S. Elbassuoni, M. Jaber, and M. Farah, ‘FA-Kes: A fake news dataset around the Syrian War’, Zenodo. Jan-2019.
  * [298] K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu, ‘FakeNewsNet: A data repository with news content, social context and spatialtemporal information for studying fake news on social media’, arXiv. org. Mar-2019.
  * [299] F. Monti, F. Frasca, D. Eynard, D. Mannion, and M. M. Bronstein, ‘Fake News Detection on Social Media using Geometric Deep Learning’, arXiv [cs.SI]. 2019.
  * [300] R. Baly et al., ‘What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context’, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 3364–3374.
  * [301] R. Evangelista and F. Bruno, ‘WhatsApp and political instability in Brazil: Targeted messages and political radicalisation’, Internet Policy Review. Dec-2019.
  * [302] K. Garimella and D. Eckles, ‘Images and misinformation in political groups: Evidence from WhatsApp in India: HKS Misinformation Review’, Misinformation Review. Jul-2022.
  * [303] Slick, ‘Commit to transparency - sign up for the international fact-checking network’s code of Principles’, IFCN Code of Principles.
  * [304] S. Kapoor and A. Narayanan, ‘Leakage and the Reproducibility Crisis in ML-based Science’. arXiv, 2022.
  * [305] S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, ‘Leakage in Data Mining: Formulation, Detection, and Avoidance’, ACM Trans. Knowl. Discov. Data, vol. 6, no. 4, Dec. 2012.
  * [306] G. Miner, Handbook of Statistical Analysis and Data Mining Applications. Association for Computing Machinery, 2009.
  * [307] S. Feng, R. Banerjee, and Y. Choi, ‘Syntactic Stylometry for Deception Detection’, in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2012, pp. 171–175.
  * [308] R. Mihalcea and C. Strapparava, ‘The Lie Detector: Explorations in the Automatic Recognition of Deceptive Language’, in Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, 2009, pp. 309–312.
  * [309] F. Soldner, V. Pérez-Rosas, and R. Mihalcea, ‘Box of Lies: Multimodal Deception Detection in Dialogues’, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 1768–1777.
  * [310] J. B. Bak-Coleman et al., ‘Combining interventions to reduce the spread of viral misinformation’, Nature News. Nature Publishing Group, Jun-2022.
  * [311] C. Carrasco-Farré, ‘The fingerprints of misinformation: how deceptive content differs from reliable sources in terms of cognitive effort and appeal to emotions,’ Humanit Soc Sci Commun 9, 162, May-2022.
  * [312] S. van der Linden, ‘Misinformation: Susceptibility, spread, and interventions to immunize the public’, Nature News. Nature Publishing Group, Mar-2022.
  * [313] C. Lima, ‘A whistleblower’s power: Key Takeaways from the Facebook papers’, The Washington Post. WP Company, Mar-2022.
  * [314] ‘Content fact-checkers prioritize’, Transparency Center.
  * [315] D. Arp et al., ‘Dos and Don’ts of Machine Learning in Computer Security’, in 31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 3971–3988.
  * [316] F. Zollo and W. Quattrociocchi, ‘Misinformation Spreading on Facebook’, in Complex Spreading Phenomena in Social Systems: Influence and Contagion in Real-World Social Networks, S. Lehmann and Y.-Y. Ahn, Eds. Cham: Springer International Publishing, 2018, pp. 177–196.
  * [317] A. Bovet and H. A. Makse, ‘Influence of fake news in Twitter during the 2016 US presidential election’, Nature News. Nature Publishing Group, Jan-2019.
  * [318] S. O. Oyeyemi, E. Gabarron, and R. Wynn, ‘Ebola, Twitter, and misinformation: a dangerous combination?’, Bmj, vol. 349, 2014.
  * [319] W. Ahmed, J. Vidal-Alaball, J. Downing, F. L. Seguí, and Others, ‘COVID-19 and the 5G conspiracy theory: social network analysis of Twitter data’, Journal of medical internet research, vol. 22, no. 5, p. e19458, 2020.
  * [320] A. S. Jacobs, R. Beltiukov, W. Willinger, R. A. Ferreira, A. Gupta, and L. Z. Granville, ‘AI/ML and Network Security: The Emperor has no Clothes’, in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, Los Angeles, CA, USA, 2022.
  * [321] D. J. Shah, T. Schuster, and R. Barzilay, ‘Automatic Fact-guided Sentence Modification’, in Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
  * [322] R. Geirhos et al., ‘Shortcut learning in deep neural networks’, Nature Machine Intelligence, vol. 2, no. 11, pp. 665–673, Nov. 2020.
  * [323] Meta, ‘How Meta’s third-party fact-checking program works’, Meta Blog. Meta.
  * [324] A. Jamieson and O. Solon, ‘Facebook to begin flagging fake news in response to mounting criticism’, The Guardian. Guardian News and Media, Dec-2016.
  * [325] C. Crowell, ‘Our approach to bots and misinformation’, Twitter Blog. Twitter.
  * [326] N. Ibrahim, ‘Why facebook won’t be fact-checking Trump now that he’s announced candidacy’, Snopes. Snopes.com, Nov-2022.
  * [327] ‘GossipCop’. GossipCop.com.
  * [328] O. Darcy, ‘BuzzFeed News will shut down’, CNN Business. Cable News Network, Apr-2023.
  * [329] T. Hsu, ‘As COVID-19 continues to spread, so does misinformation about it’, The New York Times. The New York Times, Dec-2022.
  * [330] R. Woo and L. Gao, ‘China’s factory activity falls faster than expected as recovery stumbles’, Reuters. Thomson Reuters, May-2023.
  * [331] D. Snelling, ‘Xperia XZ4 release this month - five things every Sony fan should know’, Express. co. uk. Express.co.uk, Feb-2019.
  * [332] C. F. Bond and B. M. DePaulo, ‘Accuracy of deception judgments’, Sage Journals. Personality and Social Psychology Review, 2006.
  * [333] OpenAI, ‘ChatGPT’, chat.openai.com. OpenAI.
  * [334] J. Vincent, ‘OpenAI isn’t doing enough to make ChatGPT’s limitations clear’, The Verge. The Verge, May-2023.
  * [335] S. Ali, ‘Facebook’s formula prioritized anger and ended up spreading misinformation’, The Hill. The Hill, Oct-2021.
  * [336] W. S. J. Staff, ‘The Facebook Files’, The Wall Street Journal. Dow Jones & Company, Oct-2021.
  * [337] D. Coldewey, ‘Deconstructing “the Twitter files”’, TechCrunch. Jan-2023.
  * [338] S. Søe, ‘Algorithmic detection of misinformation and disinformation: Gricean perspectives’, Journal of Documentation. Dec-2017.
  * [339] I. Fried, ‘OpenAI touts GPT-4 for content moderation’, Axios. Aug-2023.
  * [340] M. R. DeVerna, H. Y. Yan, K. Yang, and F. Menczer, ‘Artificial intelligence is ineffective and potentially harmful for fact checking’, arXiv. Aug-2023.
  * [341] D. Andročec, ‘Machine learning methods for toxic comment classification: a systematic review’, Informatica. Jan-2021.
  * [342] H. Lee, T. Ermakov, V. Ververis, and B. Fabian, ‘Detecting child sexual abuse material: A comprehensive survey’, Forensic Science International: Digital Investigation. Sept-2020.


Mengzhe Geng1,4,∗, Xurong Xie2, Rongfeng Su3, Jianwei Yu4, Zengrui Jin1,
Tianzi Wang1,
Shujie Hu1,Zi Ye1, Helen Meng1, Xunying Liu1††thanks: * Part of this work was
done while the author was an intern at Tencent AI Lab. IEEEexample:BSTcontrol

# On-the-Fly Feature Based Rapid Speaker Adaptation
for Dysarthric and Elderly Speech Recognition

###### Abstract

Accurate recognition of dysarthric and elderly speech remain challenging tasks
to date. Speaker-level heterogeneity attributed to accent or gender, when
aggregated with age and speech impairment, create large diversity among these
speakers. Scarcity of speaker-level data limits the practical use of data-
intensive model based speaker adaptation methods. To this end, this paper
proposes two novel forms of data-efficient, feature-based on-the-fly speaker
adaptation methods: variance-regularized spectral basis embedding (SVR) and
spectral feature driven f-LHUC transforms. Experiments conducted on UASpeech
dysarthric and DementiaBank Pitt elderly speech corpora suggest the proposed
on-the-fly speaker adaptation approaches consistently outperform baseline
iVector adapted hybrid DNN/TDNN and E2E Conformer systems by statistically
significant WER reduction of 2.48%-2.85% absolute (7.92%-8.06% relative), and
offline model based LHUC adaptation by 1.82% absolute (5.63% relative)
respectively.

Index Terms: Speaker Adaptation, Rapid Adaptation, Dysarthric Speech, Elderly
Speech, Speech Recognition

##  1 Introduction

Despite the breakthroughs in automatic speech recognition (ASR) technologies
targeting normal speech, accurate recognition of dysarthric and elderly speech
remains highly challenging tasks to date [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20]. Speech impairments are commonly found
among dysarthric speakers and the elderly experiencing natural aging and
neurocognitive disorders [21]. ASR technologies tailored to their needs can
improve their quality of life.

Dysarthric and elderly speech presents a prominent challenge to current ASR
technologies primarily targeting normal speech in many aspects. Heterogeneity
commonly found in normal speech sourcing from accent or gender, when further
combined with that over age and speech impairment severity, create large
diversity among dysarthric and elderly speakers [22, 23]. Such diversity is
further aggregated when spectral or temporal perturbation based data
augmentation techniques [24, 25, 6] are used. To this end, speaker adaptation
techniques play a crucial role in the personalization of ASR systems for such
users.

Speaker adaptation techniques for normal speech have been widely studied in
three broad categories: 1) speaker-dependent (SD) auxiliary speaker embedding
features [26, 27, 28]; 2) feature transformations generating canonical
features at acoustic front-ends [29, 30]; 3) model based approaches using
specially designed SD DNN parameters [31, 32, 33].

In contrast, there are limited prior researches on dysarthric and elderly
speaker adaptation. Earlier works were mainly conducted on HMM based ASR
systems, including MLLR and MAP adaptation [34, 35, 1, 36] and their
combination with speaker adaptive training (SAT) [2], feature-space MLLR
(f-MLLR) based SAT [37] and regularized speaker adaptation via Kullback-
Leibler (KL) divergence [38]. More recent researches applied model based
adaptation to current DNN based ASR systems, including direct parameter fine-
tuning based adaptation in both hybrid TDNN [4, 39] and end-to-end RNN-T [40,
41] systems, LHUC [11, 8] and Bayesian speaker adaptation [42]. Spectro-
temporal basis embedding features (SBE) based offline adaptation via speaker-
level averaging was studied in [10, 16].

One major issue associated with the prior researches above is the lack of
suitable rapid, on-the-fly adaptation techniques targeting dysarthric and
elderly speech. Such methods serve as dual-purpose solutions to handle not
only the difficulty in collecting large quantities of data from such speakers
with mobility issues that are essential for model based adaptation but also
their latency issues due to multi-pass decoding and SD parameter estimation.
The Bayesian model based adaptation using very limited speaker data [11, 43,
42] only addressed the aforementioned data scarcity issue, but the latency
problem remains unvisited. Similarly, the spectro-temporal deep embedding
features [10] computed and averaged over all speaker-level data incurred
latency and precluded the use of on-the-fly adaptation.

In order to address this issue, two novel forms of feature-based on-the-fly
rapid speaker adaptation approaches are proposed in this paper. The first is
based on speaker-level variance-regularized spectral basis embedding (SVR)
features. An additional variance regularization term is included when training
spectral basis embedding DNNs [10, 16] to ensure speaker-level homogeneity of
the resulting embedding features and thus allow them to be applied on the fly
during test time adaptation. The second approach uses on-the-fly feature-based
LHUC (f-LHUC) transforms conditioned on spectral features. Specially designed
regression TDNN [44] predicting speaker-level LHUC transforms are used to
directly generate and apply such parameters during test time adaptation and
thus resolve the latency due to multi-pass decoding. Experiments were
conducted on the largest available and most widely used UASpeech [45]
dysarthric and DementiaBank Pitt [46] elderly speech datasets. Consistent
performance improvements were obtained by our proposed on-the-fly speaker
adaptation approaches using both hybrid DNN/TDNN and E2E Conformer [47]
systems.

The main contributions of the paper are summarized below:

1) This paper presents the first work of on-the-fly feature-based fast speaker
adaptation targeting dysarthric and elderly speech. In contrast, previous
works using feature-based dysarthric and elderly speaker adaptation uses all
speaker-level data and operate in batch mode [10, 16] while those using model
based adaptation not only require the usage of all speaker-level data, but
also additional multiple decoding passes and explicit parameter estimation in
test time [6, 8, 42, 11]. These prior works incur significant latency and are
not the on-the-fly, rapid speaker adaptation approaches considered in this
paper.

2) Our proposed SVR features can instantaneously extract homogeneous
dysarthric and elderly speaker characteristics on the fly. Experiments
conducted on benchmark UASpeech dysarthric and DementiaBank Pitt elderly
speech datasets suggest the proposed on-the-fly speaker adaptation approaches
consistently outperform the baseline iVector adapted hybrid DNN/TDNN and E2E
Conformer systems by statistically significant word error rate (WER) reduction
of 2.48%-2.85% absolute (7.92%-8.06% relative), and offline model based LHUC
adaptation by 1.82% absolute (5.63% relative) respectively.

##  2 Variance-Regularized Spectral Features

To model the latent diversity in dysarthric and elderly speech, singular value
decomposition (SVD) is performed on Mel-filterbank log amplitude spectrum
$\mathbf{S}$ [48], given as:

$\mathbf{S=U{\Sigma}V^{\mathrm{T}}}$ (1)

where the top-$d$ principal spectral bases are retrieved from the column
vectors of $\mathbf{U}$. Following [10, 16], further supervised learning is
performed via constructing DNN speech intelligibility or age classifier (the
upper part of Fig.1). The inputs are the selected spectral bases, and the
targets are speech intelligibility groups $+$ speaker IDs for the UASpeech
corpus and binary aged vs. non-aged annotations for the DementiaBank Pitt
corpus.

To ensure the speaker-level homogeneity of the embedding features, a pair of
such DNN classifiers (Fig.1) are constructed. The $25$-dim embedding features
taken from the bottleneck layer of the upper classifier in Fig. 1 are averaged
by speaker and then serve as the regression targets of the lower DNN
classifier (Fig.1 blue bold line) for variance regularization. A multitask
learning (MTL) cost function is used to train the lower classifier, using
interpolation between 1) the cross-entropy (CE) error computed over speech
intelligibility or age labels, optionally plus that computed over speaker IDs
and 2) the mean squared error (MSE) computed between the lower DNN bottleneck
features and the corresponding speaker-level averaged embedding features
produced by the upper DNN, which is given as:

$\mathcal{L}_{MTL}={\omega_{1}}{\cdot}\mathcal{L}_{MSE}+{\omega_{2}}{\cdot}\mathcal{L}_{CE_{group}}+{\omega_{3}}{\cdot}\mathcal{L}_{CE_{ID}}$
(2)

Figure 1: Our proposed DNN speech intelligibility or age classifier with a
bottleneck layer to extract variance-regularized spectral basis embedding
(SVR) features for speaker adaptation. Here ``intel.'' denotes speech
intelligibility.

The $25$-dim variance-regularized spectral basis embedding (SVR) features are
then taken from the bottleneck layer of the lower classifier (Fig.1 red bold
line), and appended to the acoustic features at the front-end of hybrid
DNN/TDNN (Fig.4) and E2E Conformer systems (Fig.5), to facilitate on-the-fly
test time adaptation. The aforementioned procedure is summarized in Fig. 2.
Directly using the top $d$ spectral bases or intermediate embedding features
are unsuitable for speaker adaptation [16].

Figure 2: Procedure of generating the SVR features.

##  3 On-the-Fly F-LHUC Transforms

In feature-based learning hidden unit contributions (f-LHUC) based adaptation
approaches [44], LHUC transforms are predicted from the acoustic features on
the fly. Supervised estimation of LHUC transforms on the training data is
first conducted via standard speaker adaptive training (SAT). Principal
component analysis (PCA) is further applied to produce compressed LHUC vectors
encoding the most distinctive speaker-level features. These serve as the
output targets for the TDNN based LHUC regression network (Fig.3), with a
specially designed online averaging layer [44] given as:

$\bm{m_{s}^{i}}=\frac{\bm{\sum_{t=1}^{T_{i}}\bm{h_{t}^{i}}+{\alpha}{\cdot}\bm{G_{s}^{i-1}}}}{T_{i}+{\alpha}{\cdot}N_{s}^{i-1}}$
(3)

Figure 3: TDNN based LHUC regression network.

where $G_{s}$, $N_{s}$ and $m_{s}$ denote the accumulated hidden vector, frame
count, and averaged hidden vector till the $i^{th}$ segment of speaker $s$.
The $i^{th}$ utterance contains $T_{i}$ frames, and the hidden vector of the
$t^{th}$ frame is $\bm{h_{t}^{i}}$. $\alpha\in[0,1]$ is the history
interpolation weight. Different from [44] where only Mel-filterbank (FBK)
features are considered, FBK $+$ variance-regularized spectral basis embedding
(SVR) features are used to train the LHUC regression network (Fig 3). An
additional affine transformation is further trained to map the predicted low-
dimensional LHUC features for training speakers to their corresponding LHUC
transforms. During test time on-the-fly adaptation, the regression network
(Fig. 3) and the affine transformation (Fig.4 circled in green) are applied in
turn to generate speaker-level LHUC transforms using FBK $+$ SVR features.

Figure 4: Incorporation of variance-regularized spectral basis embedding (SVR)
features at front-end of hybrid DNN ASR systems [11]. Selecting path (i) leads
to systems with auxiliary feature based adaptation only, while selecting (ii)
leads to systems with additional feature-based LHUC (f-LHUC) adaptation.
Figure 5: Incorporating SVR features into E2E Conformer.

##  4 Experiments and Results

###  4.1 Experiments on the UASpeech Dataset

Task Description: The UASpeech dataset is the largest publicly available and
widely used dysarthric speech dataset [45], which is an isolated word
recognition task containing $103$h speech from $16$ dysarthric and $13$
control speakers. It is split into three blocks B1, B2 and B3, each with the
same $155$ common words and a different set of $100$ uncommon words. The
training set includes B1 and B3 data of all $29$ speakers ($69.1$h), while the
test set includes B2 data of $16$ dysarthric speakers ($22.6$h, excluding
speech from control speakers). Silence stripping using an HTK [49] trained
GMM-HMM system [11] produces a $30.6$h training set ($99195$ utt.) and a $9$h
test set ($26520$ utt.). Data augmentation [6] produces a $130.1$h augmented
training set ($399110$ utt.). The average utterance length is $1.2$ seconds.
As E2E systems are sensitive to the training data coverage, B2 data of the
$13$ control speakers and their augmented version via speed perturbation are
also used for Conformer system training. This creates a $190$h training set
($538292$ utt.).

Experiment Setup: The $7$-layer hybrid DNN and E2E graphemic Conformer systems
were implemented using Kaldi [50] following [11] and ESPnet [51]222$12$
encoder layers + $12$ decoder layers, feed-forward dim = $2048$, $4$ attention
heads of $256$ dimensions, interpolated CTC+AED cost.. The inputs to both
systems were $80$-dim filter-bank (FBK) + $\Delta$ features plus $25$-dim
variance-regularized spectral basis embedding (SVR) features or $100$-dim
iVectors333Kaldi: egs/wsj/s5/local/nnet3/run_ivector_common.sh. Changing the
dimensionality of iVectors produces margin effect [16]. (Fig. 4\- 5). Top $2$
principal spectral bases [10, 16] were used to train the DNN speech
intelligibility classifier (Fig.1). The history interpolation weight $\alpha$
of the LHUC regression network was set to $0.9$ with the four context slicing
indices as $\\{-2,-1,0,1,2\\}$,$\\{-2,0,2\\}$,$\\{-3,0,3\\}$ and
$\\{-4,0,4\\}$. A uniform language model (LM) was used in decoding [1]. As an
ablation study, we used iVectors as the inputs to the DNN classifier (Fig.1)
for variance regularization and generate variance-regularized iVectors (iVRs).

Table 1: Performance of the proposed variance-regularized spectral basis
embedding (SVR) feature adaptation, iVector adaptation and LHUC adaptation on
the UASpeech test set of $16$ dysarthric speakers. ``SBE'' denotes spectral
basis embedding features. ``VL/L/M/H" refer to intelligibility very low, low,
mid and high. ``On Fly'' indicates using on-the-fly adaptation. † denotes a
statistically significant improvement ($\alpha=0.05$) is obtained over iVector
adapted systems (Sys.2,12,23).

Sys. |  Model (#Para.) |  Data Aug. | #Hrs |  Adapt. Feat. |  LHUC SAT | f-LHUC |  On Fly | WER%
---|---|---|---|---|---|---|---|---
VL | L | M | H | All
1 |  Hybrid DNN (6M) | ✗ | 30.6 | ✗ | ✗ | ✗ | - | 69.82 | 32.61 | 24.53 | 10.40 | 31.45
2 | iVector | ✓ | 69.46 | 33.78 | 22.58 | 10.45 | 31.33
3 |  SBE [10] | ✗ | 64.43 | 29.71 | 19.84 | 8.57 | 28.05
4 | iVR | ✓ | 68.66 | 33.72 | 22.84 | 9.83 | 30.99
5 | SVR | ✓ |  65.04† |  30.90† |  20.70† |  10.15† | 28.85†
6 | ✗ | ✓ | ✗ | ✗ | 64.39 | 29.88 | 20.27 | 8.95 | 28.29
7 |  SBE [10] | ✗ | 63.40 | 28.90 | 18.64 | 8.13 | 27.24
8 | ✗ | ✗ | (FBK) | ✓ |  66.47† |  29.55† |  21.00† |  8.99† |  28.80†
9 | iVector | (+iVector) | ✓ | 64.86 | 36.44 | 21.17 | 9.03 | 30.29
10 | SVR | (+SVR) | ✓ |  65.75† |  29.80† |  19.07† |  8.99† | 28.31†
5+10 | - | ✓ | 64.36† | 29.68† | 18.96† | 8.89† | 27.96†
11 |  Hybrid DNN (6M) | ✓ | 130.1 | ✗ | ✗ | ✗ | - | 66.45 | 28.95 | 20.37 | 9.62 | 28.73
12 | iVector | ✓ | 65.73 | 30.10 | 20.21 | 9.03 | 28.65
13 |  SBE [10] | ✗ | 61.55 | 27.52 | 17.31 | 8.22 | 26.26
14 | iVR | ✓ | 66.02 | 29.52 | 19.56 | 9.32 | 28.53
15 | SVR | ✓ |  62.54† | 30.22 |  18.54† |  8.59† | 27.54†
16 | ✗ | ✓ | ✗ | ✗ | 62.50 | 27.26 | 18.41 | 8.04 | 26.55
17 |  SBE [10] | ✗ | 59.83 | 27.16 | 16.80 | 7.91 | 25.60
18 | ✗ | ✗ | (FBK) | ✓ | 65.06 |  27.94† |  18.76† |  8.39† |  27.45†
19 | iVector | (+iVector) | ✓ | 63.63 | 32.56 | 18.52 | 8.31 | 28.28
20 | SVR | (+SVR) | ✓ |  61.56† | 28.81 |  18.39† |  8.50† | 26.90†
15+20 | - | ✓ | 60.80† | 28.19† | 17.72† | 8.23† | 26.36†
21 |  Conformer (52M) | ✓ | 130.1 | ✗ | ✗ | ✗ | - | 73.88 | 53.12 | 49.92 | 42.03 | 53.17
22 | 190 | ✗ | - | 65.70 | 40.63 | 33.39 | 9.53 | 34.07
23 | iVector | ✓ | 69.05 | 42.45 | 33.60 | 9.74 | 35.37
24 |  SBE [10] | ✗ | 65.18 | 34.90 | 24.21 | 5.00 | 29.19
25 | iVR | ✓ | 68.94 | 42.00 | 32.19 | 8.52 | 34.55
26 | SVR | ✓ |  67.52† | 38.85† | 28.60† | 7.88† | 32.52†

Result Analysis: Table 1 shows the performance comparison444A matched pairs
sentence-segment word error (MAPSSWE) based statistical signiﬁcance test [52]
was done at signiﬁcance level $\alpha=0.05$. between the proposed variance-
regularized spectral basis embedding (SVR) feature adaptation, spectral
feature driven f-LHUC adaptation, iVector adaptation and offline LHUC
adaptation on the UASpeech corpus. Several trends can be observed: 1) On-the-
fly SVR adaptation (Sys.5,15,26) consistently and statistically significantly
outperform iVector adaptation (Sys.2,12,23) with various amounts of training
data by up to 2.48% absolute (7.92% relative) overall WER reduction for hybrid
DNN (Sys.5 vs. Sys.2), and 2.85% absolute (8.06% relative) reduction (Sys.26
vs. Sys.23) for Conformer systems, respectively. 2) The improvements from
offline LHUC adaptation (Sys.6,16) over the SI systems (Sys.1,11) are largely
retained (by 82%) and comparable to those obtained using on-the-fly SVR
adaptation (Sys.5,15). 3) Compared with improvements over SI systems
(Sys.1,11) obtained by offline SBE adaptation [10] that requires expensive
speaker-level averaging (Sys.3,13), on-the-fly SVR adaptation (Sys.5,15)
produces comparable performance. 4) The spectral feature (FBK+SVR) driven
f-LHUC adapted systems (Sys.10,20) outperform both the FBK driven f-LHUC
adapted systems (Sys.8,18) and the SVR adaptation alone (Sys.5,15). 5) Frame-
level log-likelihood score combination between the on-the-fly SVR adaptation
and FBK+SVR driven f-LHUC adaptation leads to further improvements (Sys.5+10,
Sys.15+20). 6) The SVR on-the-fly adapted systems (Sys.5,15,26) consistently
outperform comparable variance-regularized iVector (iVR) adaptation
(Sys.4,14,25). 7) Our proposed FBK+SVR driven f-LHUC adapted systems
(Sys.10,20) consistently outperform the comparable FBK+iVector driven f-LHUC
adaptation (Sys.9,19). 8) A comparison between published systems on UASpeech
and ours is shown in Table 4. Our best-performing system (Table 1, Sys.15+20)
produced the lowest WERs among all systems using online speaker adaptation in
Table 4.

###  4.2 Experiments on the DementiaBank Pitt Dataset

Task Description: The DementiaBank Pitt [46] dataset contains $33$h speech
recorded over interviews between $292$ elderly participants and clinical
investigators. After split of the data and silence stripping [8], the training
set contains $15.7$h speech from $244$ elderly and $444$ investigators
($29682$ utt.) while the development and evaluation sets contain $2.5$h
($5103$ utt.) and $0.6$h ($928$ utt.) speech from $43$ elderly and $76$
investigators555The evaluation set is based on exactly the same $48$ speakers’
Cookie task recordings following [53] while the development set contains the
recordings of these speakers in other tasks if available.. Data augmentation
[8] produced an $58.9$h augmented training set ($112830$ utt.). The average
utterance length is $1.9$ seconds.

Experiment Setup: The inputs to the hybrid TDNN systems666$14$ context slicing
layers with a $3$-frame context [50]. and E2E graphemic Conformer
systems777$12$ encoder layers + $12$ decoder layers, feed-forward dim =
$2048$, $4$ attention heads of $256$ dimensions, interpolated CTC+AED cost.
were $40$-dim FBK + $25$-dim SVR features or $100$-dim iVectors. Top $3$
spectral bases [16] served as the inputs to the DNN age classifier (Fig.1). A
word $4$-gram LM [8] and a $3.8$k recognition vocabulary covering all words in
DementiaBank Pitt corpus was used.

Table 2: Performance of the proposed variance-regularized spectral basis
embedding (SVR) adaptation, iVector adaptation and LHUC adaptation on the
augmented DementiaBank Pitt corpus. ``INV'' and ``PAR'' denote investigator
and elderly. ${{\dagger}}$ denotes a statistically significant improvement
($\alpha=0.05$) over both the iVector adaptation (Sys.2) and offline LHUC
adaptation (Sys.6), while ${{\ddagger}}$ denotes a statistically significant
improvement ($\alpha=0.05$) over the iVector adaptation (Sys.12).

Sys. |  Model (#Para.) | #Hrs |  Adapt. Feat. |  LHUC SAT | f-LHUC |  On Fly | WER%
---|---|---|---|---|---|---|---
Dev | Eval | All
INV | PAR | INV | PAR
1 |  Hybrid TDNN (18M) | 58.9 | ✗ | ✗ | ✗ | - | 19.91 | 47.93 | 19.76 | 36.66 | 33.80
2 | iVector | ✓ | 19.97 | 46.76 | 18.20 | 37.01 | 33.37
3 |  SBE [16] | ✗ | 18.61 | 43.84 | 17.98 | 33.82 | 31.12
4 | iVR | ✓ | 19.19 | 47.64 | 18.65 | 35.80 | 33.26
5 | SVR | ✓ |  18.72† |  44.67† | 18.65 |  34.03† | 31.55†
6 | ✗ | ✓ | ✗ | ✗ | 19.26 | 45.49 | 18.42 | 35.44 | 32.33
7 |  SBE [16] | ✗ | 17.41 | 40.94 | 17.98 | 31.89 | 29.16
8 | ✗ | ✗ | (FBK) | ✓ | 19.61 | 45.40 | 18.87 | 34.77 | 32.33
9 | iVector | (+iVector) | ✓ | 18.75 | 47.07 | 17.98 | 36.11 | 32.85
10 | SVR | (+SVR) | ✓ |  17.87† |  43.83† |  16.87† |  34.56† | 30.91†
5+10 | - | ✓ |  17.66† | 43.48† |  16.09† | 33.68† | 30.51†
11 |  Conformer (52M) | 58.9 | ✗ | ✗ | ✗ | - | 20.97 | 48.71 | 19.42 | 36.93 | 34.57
12 | iVector | ✓ | 21.48 | 48.32 | 17.42 | 37.79 | 34.71
13 |  SBE [16] | ✗ | 20.44 | 47.70 | 17.31 | 36.11 | 33.76
14 | iVR | ✓ | 22.09 | 49.56 | 19.64 | 38.58 | 35.65
15 | SVR | ✓ | 20.83 | 47.39‡ | 17.64 | 36.34‡ | 33.84‡

Result Analysis: Table 2 shows the performance of the proposed on-the-fly SVR
adaptation, f-LHUC adaptation, iVector adaptation and LHUC adaptation on
DementiaBank Pitt. Trends similar to those on UASpeech in Table 1 are
observed: 1) On-the-fly SVR adaptation (Sys.5,15) statistically significantly
outperform iVector adaptation (Sys.2,12) on both TDNN and Conformer systems by
up to 1.82% absolute (5.45% relative) WER reduction (Sys.5 vs. Sys.2). 2) On-
the-fly SVR adaptation outperforms offline LHUC adaptation by 0.78% absolute
(2.41% relative) overall WER reduction (Sys.5 vs. Sys.6). 3) On-the-fly SVR
adaptation (Sys.5) largely retains (by 84%) the improvements obtained by
offline SBE adaptation [16] (Sys.3) over the SI system (Sys.1). 4) The
proposed FBK+SVR driven f-LHUC adapted system (Sys.10) outperforms offline
LHUC adaptation (Sys.6) while also outperforming FBK driven f-LHUC adaptation
(Sys.8) and SVR adaptation alone (Sys.5). 5) Frame-level score combination
between SVR and FBK+SVR driven f-LHUC on-the-fly adaptation leads to 1.82%
absolute (5.63% relative) WER reduction over the offline LHUC adaptation
(Sys.5+10 vs. Sys.6). 6) Our proposed on-the-fly SVR (Sys.5,15) and FBK+SVR
driven f-LHUC adaptation (Sys.10) consistently outperform the comparable iVR
(Sys.4,14) and FBK+iVector driven f-LHUC adaptation (Sys.9).

###  4.3 Further Ablation Studies

Figure 6: Performance (WER%) of offline LHUC [11], offline SBE adaptation [10,
16] and on-the-fly SVR adaptation of Sec.2 on varying percentage of test set
speaker-level adaptation data on UASpeech and DementiaBank Pitt.

As expected, the ablation study in Fig. 6 confirms on-the-fly SVR adaptation
is more robust to varying amounts of speaker-level data used in adaptation
than offline LHUC and SBE adaptation, and consistently outperforms both when
less than 40% of speaker-level data is used. A further ablation study in Table
3 suggests the performance of on-the-fly SVR adaptation is largely insensitive
to the length of analysis sliding windows (from 1 utt. down to 10ms).
Homogeneous dysarthric and elderly speaker characteristics can be
instantaneously extracted in the SVR features on the fly. The real-time (R.T.)
factor indicates the total delay of waiting for data and model processing.

Table 3: Ablation study on the augmented UASpeech (UA.) and DementiaBank Pitt
(DBK.) corpora with various sizes of sliding window (Slid. Wind.) for on-the-
fly SVR feature extraction. ``R.T.'' is short for real time. † denotes a
statistically significant improvement ($\alpha=0.05$) is obtained over the
comparable on-the-fly iVector adapted systems.

UA. Sys. |  Model (#Para.) | #Hrs |  Adapt. Feat. |  Slid. Wind. |  On Fly |  R.T. Factor | WER%
---|---|---|---|---|---|---|---
VL | L | M | H | All
1 |  Hybrid DNN (6M) | 130.1 | iVector | 100ms | ✓ | 0.10 | 65.73 | 30.10 | 20.21 | 9.03 | 28.65
2 | SVR | utt. | 1.02 |  62.54† | 30.22 |  18.54† |  8.59† |  27.54†
3 | 300ms | 0.27 |  63.74† |  29.01† |  19.56† | 9.09 |  27.84†
4 | 200ms | 0.18 |  63.81† |  29.15† | 20.19 | 9.31 |  28.08†
5 | 100ms | 0.10 |  64.49† |  28.36† |  19.47† | 9.27 |  27.87†
6 | 50ms | 0.06 |  63.99† |  28.68† |  19.17† | 8.99 |  27.70†
7 | 10ms | 0.03 | 64.77† | 29.03† | 19.21† | 9.09 | 28.00†
DBK. Sys. |  Model (#Para.) | #Hrs |  Adapt. Feat. |  Slid. Wind. |  On Fly |  R.T. Factor | WER%
Dev | Eval | All
INV | PAR | INV | PAR
1 |  Hybrid TDNN (18M) | 58.9 | iVector | 100ms | ✓ | 0.08 | 19.97 | 46.76 | 18.20 | 37.01 | 33.37
2 | SVR | utt. | 1.03 |  18.72† |  44.67† | 18.65 |  34.03† |  31.55†
3 | 300ms | 0.19 |  19.29† |  45.01† | 19.09 |  33.28† |  31.81†
4 | 200ms | 0.14 | 19.36 |  45.15† | 19.20 |  33.89† |  32.01†
5 | 100ms | 0.08 |  19.08† |  45.03† | 19.42 |  34.41† |  31.93†
6 | 50ms | 0.06 | 19.38 |  44.87† | 18.31 |  34.52† |  31.97†
7 | 10ms | 0.04 |  19.33† | 44.93† | 19.09 | 34.35† | 31.97†

Table 4: Performance comparison against recently published systems on
UASpeech. ``DA'' denotes data augmentation.

Sys. | Online | VL | All
---|---|---|---
Sheffield-2015 Speaker adaptive training [2] | ✗ | 70.78 | 34.85
Sheffield-2020 Fine-tuning CNN-TDNN speaker adaptation (15spk) [4] | ✓ | 68.24 | 30.76
CUHK-2020 DNN + DA + LHUC SAT [6] | ✗ | 62.44 | 26.37
CUHK-2021 LAS + CTC + Meta-learning + SAT [54] | ✗ | 68.70 | 35.00
CUHK-2021 QuartzNet + CTC + Meta-learning + SAT [54] | ✗ | 69.30 | 30.50
Sheffield-2022 DA + Source Filter Features + iVector adapt [17] | ✓ | - | 30.30
DA + SVR Adapt + f-LHUC system combination (Table 1, Sys.15+20) | ✓ | 60.80 | 26.36

##  5 Conclusions

This paper proposes two novel forms of feature-based on-the-fly speaker
adaptation approaches: speaker-level variance-regularized spectral basis
embedding (SVR) features adaptation and spectral feature driven f-LHUC
adaptation. Experiments conducted on benchmark UASpeech dysarthric and
DementiaBank Pitt Elderly datasets suggest both methods can efficiently encode
homogeneous dysarthric and elderly speaker specific characteristics and
outperform both online iVector and offline model based LHUC adaptation. Future
research will focus on rapid speaker adaptation of pre-trained ASR systems.

##  6 Acknowledgements

This research is supported by Hong Kong RGC GRF grant No. 14200220, 14200021,
TRS T45-407/19N, Innovation Technology Fund grant No. ITS/218/21, National
Natural Science Foundation of China (NSFC) Grant 62106255, and Youth
Innovation Promotion Association CAS Grant 2023119.

## References

  * [1] H. Christensen _et al._ , ``A comparative study of adaptive, automatic recognition of disordered speech,'' in _INTERSPEECH_ , 2012.
  * [2] S. Sehgal _et al._ , ``Model adaptation and adaptive training for the recognition of dysarthric speech,'' in _SLPAT_ , 2015.
  * [3] S. Hu _et al._ , ``The CUHK Dysarthric Speech Recognition Systems for English and Cantonese,'' in _INTERSPEECH_ , 2019.
  * [4] F. Xiong _et al._ , ``Source Domain Data Selection for Improved Transfer Learning Targeting Dysarthric Speech Recognition,'' in _ICASSP_ , 2020.
  * [5] S. Liu _et al._ , ``Exploiting cross-domain visual feature generation for disordered speech recognition,'' in _INTERSPEECH_ , 2020.
  * [6] M. Geng _et al._ , ``Investigation of Data Augmentation Techniques for Disordered Speech Recognition,'' in _INTERSPEECH_ , 2020.
  * [7] S. Hu _et al._ , ``Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition,'' _IEEE T AUDIO SPEECH_ , 2021.
  * [8] Z. Ye _et al._ , ``Development of the CUHK Elderly Speech Recognition System for Neurocognitive Disorder Detection Using the Dementiabank Corpus,'' in _ICASSP_ , 2021.
  * [9] J. Harvill _et al._ , ``Synthesis of new words for improved dysarthric speech recognition on an expanded vocabulary,'' in _ICASSP_ , 2021.
  * [10] M. Geng _et al._ , ``Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition,'' in _INTERSPEECH_ , 2021.
  * [11] S. Liu _et al._ , ``Recent Progress in the CUHK Dysarthric Speech Recognition System,'' _IEEE T AUDIO SPEECH_ , 2021.
  * [12] Z. Jin _et al._ , ``Adversarial Data Augmentation for Disordered Speech Recognition,'' in _INTERSPEECH_ , 2021.
  * [13] S. Hu _et al._ , ``Exploiting Cross Domain Acoustic-to-articulatory Inverted Features for Disordered Speech Recognition,'' in _ICASSP_ , 2022\.
  * [14] M. K. Baskar _et al._ , ``Speaker adaptation for Wav2vec2 based dysarthric ASR,'' in _INTERSPEECH_ , 2022.
  * [15] T. Wang _et al._ , ``Conformer based elderly speech recognition system for alzheimer's disease detection,'' in _INTERSPEECH_ , 2022.
  * [16] M. Geng _et al._ , ``Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition,'' _IEEE T AUDIO SPEECH_ , 2022.
  * [17] Z. Yue _et al._ , ``Acoustic modelling from raw source and filter components for dysarthric speech recognition,'' _IEEE T AUDIO SPEECH_ , 2022\.
  * [18] Z. Jin _et al._ , ``Adversarial data augmentation using vae-gan for disordered speech recognition,'' in _ICASSP_ , 2023.
  * [19] S. Hu _et al._ , ``Exploring self-supervised pre-trained asr models for dysarthric and elderly speech recognition,'' in _ICASSP_ , 2023.
  * [20] M. Geng _et al._ , ``Use of Speech Impairment Severity for Dysarthric Speech Recognition,'' in _INTERSPEECH_ , 2023.
  * [21] A. Association, ``2019 Alzheimer's disease facts and figures,'' _Alzheimer's & dementia_, 2019.
  * [22] I. Kodrasi _et al._ , ``Spectro-Temporal Sparsity Characterization for Dysarthric Speech Detection,'' _IEEE T AUDIO SPEECH_ , 2020.
  * [23] B. L. Smith _et al._ , ``Temporal characteristics of the speech of normal elderly adults,'' _J SPEECH LANG HEAR R_ , 1987.
  * [24] B. Vachhani _et al._ , ``Data Augmentation Using Healthy Speech for Dysarthric Speech Recognition,'' in _INTERSPEECH_ , 2018.
  * [25] F. Xiong _et al._ , ``Phonetic analysis of dysarthric speech tempo and applications to robust personalised dysarthric speech recognition,'' in _ICASSP_ , 2019.
  * [26] O. Abdel-Hamid _et al._ , ``Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code,'' in _ICASSP_ , 2013.
  * [27] G. Saon _et al._ , ``Speaker adaptation of neural network acoustic models using i-vectors,'' in _ASRU_ , 2013.
  * [28] H. Huang _et al._ , ``An investigation of augmenting speaker representations to improve speaker normalisation for dnn-based speech recognition,'' in _ICASSP_ , 2015.
  * [29] V. V. Digalakis _et al._ , ``Speaker adaptation using constrained estimation of Gaussian mixtures,'' _IEEE T SPEECH AUDI P_ , 1995.
  * [30] F. Seide _et al._ , ``Feature engineering in context-dependent deep neural networks for conversational speech transcription,'' in _ASRU_ , 2011.
  * [31] J. Neto _et al._ , ``Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system,'' in _EUROSPEECH_ , 1995.
  * [32] C. Zhang _et al._ , ``Parameterised sigmoid and ReLU hidden activation functions for DNN acoustic modelling,'' in _INTERSPEECH_ , 2015.
  * [33] P. Swietojanski _et al._ , ``Learning hidden unit contributions for unsupervised acoustic model adaptation,'' _IEEE T AUDIO SPEECH_ , 2016.
  * [34] A. BABA _et al._ , ``Elderly Acoustic Models for Large Vocabulary Continuous Speech Recognition,'' _IEICE T INF SYST_ , 2002.
  * [35] K. T. Mengistu _et al._ , ``Adapting acoustic and lexical models to dysarthric speech,'' in _ICASSP_ , 2011.
  * [36] M. J. Kim _et al._ , ``Dysarthric speech recognition using dysarthria-severity-dependent and speaker-adaptive models,'' in _INTERSPEECH_ , 2013.
  * [37] C. Bhat _et al._ , ``Recognition of Dysarthric Speech Using Voice Parameters for Speaker Adaptation and Multi-Taper Spectral Estimation,'' in _INTERSPEECH_ , 2016.
  * [38] M. Kim _et al._ , ``Regularized speaker adaptation of KL-HMM for dysarthric speech recognition,'' _IEEE T NEUR SYS REH_ , 2017.
  * [39] R. Takashima _et al._ , ``Two-step acoustic model adaptation for dysarthric speech recognition,'' in _ICASSP_ , 2020.
  * [40] J. Shor _et al._ , ``Personalizing ASR for Dysarthric and Accented Speech with Limited Data,'' in _INTERSPEECH_ , 2019.
  * [41] J. R. Green _et al._ , ``Automatic Speech Recognition of Disordered Speech: Personalized models outperforming human listeners on short phrases,'' in _INTERSPEECH_ , 2021.
  * [42] J. Deng _et al._ , ``Bayesian Parametric and Architectural Domain Adaptation of LF-MMI Trained TDNNs for Elderly and Dysarthric Speech Recognition,'' in _INTERSPEECH_ , 2021.
  * [43] X. Xie _et al._ , ``Bayesian learning for deep neural network adaptation,'' _IEEE T AUDIO SPEECH_ , 2021.
  * [44] X. Xie _et al._ , ``Fast DNN Acoustic Model Speaker Adaptation by Learning Hidden Unit Contribution Features,'' in _INTERSPEECH_ , 2019.
  * [45] H. Kim _et al._ , ``Dysarthric speech database for universal access research,'' in _INTERSPEECH_ , 2008.
  * [46] J. T. Becker _et al._ , ``The natural history of Alzheimer's disease: description of study cohort and accuracy of diagnosis,'' _ARCH NEUROL-CHICAGO_ , 1994.
  * [47] A. Gulati _et al._ , ``Conformer: Convolution-augmented Transformer for Speech Recognition,'' in _INTERSPEECH_ , 2020.
  * [48] A.-J. Van Der Veen _et al._ , ``Subspace-based signal analysis using singular value decomposition,'' _P IEEE_ , 1993.
  * [49] S. Young _et al._ , ``The HTK book,'' _Cambridge university engineering department_ , 2002.
  * [50] D. Povey _et al._ , ``The Kaldi speech recognition toolkit,'' in _ASRU_ , 2011.
  * [51] S. Watanabe _et al._ , ``ESPnet: End-to-End Speech Processing Toolkit,'' in _INTERPSEECH_ , 2018.
  * [52] D. S. Pallet _et al._ , ``Tools for the analysis of benchmark speech recognition tests,'' in _ICASSP_ , 1990.
  * [53] S. Luz _et al._ , ``Alzheimer’s Dementia Recognition through Spontaneous Speech: The ADReSS Challenge,'' in _INTERSPEECH_ , 2020.
  * [54] D. Wang _et al._ , ``Improved end-to-end dysarthric speech recognition via meta-learning based model re-initialization,'' in _ISCSLP_ , 2021.