# Network Error Logging: HTTP Archive Analysis

Kamil Jeřábek1 and Libor Polčák1
1Brno University of Technnology, Faculty of Information Technology,
Božetěchova 2, 612 66 Brno, Czech Republic
{ijerabek<EMAIL_ADDRESS>
https://orcid.org/0000-0002-5317-9222 https://orcid.org/0000-0001-9177-3073

###### Abstract

Network Error Logging helps web server operators detect operational problems
in real-time to provide fast and reliable services. HTTP Archive provides
detail information of historical data on HTTP requests. This paper leverages
the data and provides a long-term analysis of Network Error Logging
deployment. The deployment raised from 0 to 11.73 % (almost 2,250,000 unique
domains) since 2019\. Current deployment is dominated by Cloudflare. Although
we observed different policies, the default settings prevail. Third party
collectors emerge raising the diversity needed to gather sound data. Even so,
many service deploy self-hosted services. Moreover, we identify potentially
malicious adversaries deploy collectors on randomly-generated domains and
shortened URLs.

##  1 INTRODUCTION

Web server operators need to monitor their servers for availability, which is
a crucial success factor [Franke, 2012]. Network Error Logging (NEL) is a
recent World Wide Web Consortium (W3C) proposed standard [W3C, 2021] that
allows HTTP servers to employ browsers111At the time of the writing of this
paper, Chromium-based browsers like Google Chrome, Chrome for Android,
Microsoft Edge, Opera, and Opera GF support NEL. Brave is the only Chromium-
based browser that we discovered does not support NEL. to report failures in
the reachability of web servers, including failures during domain name
resolution [Burnett et al., 2020].

Current literature provides only a very shallow data on NEL deployment [Goenka
et al., 2022] even though related work identified security and privacy issues
in the current NEL implementation [Polčák and Jeřábek, 2023]. This paper
leverages HTTP archive and analyzes NEL deployment to shed more light on NEL
deployment and its trends. NEL deployment is rising, and its share is over 10
%.

This paper is the first to analyze the long-term deployment of NEL. Our
analysis shows that:

  * •

Current deployment is dominated by Cloudflare. Consequently, the NEL adoption
share strongly depends on future decision of Cloudflare. This paper highlights
Shopify, another NEL large provider, that stopped signalling NEL in 2021.

  * •

Even though the original NEL paper [Burnett et al., 2020] suggests that Google
was migrating to NEL, HTTP Archive data does not show any traces of Google
deploying NEL.

  * •

We provide data on the NEL policies deployed in the wild and study the NEL
ecosystem and the trends inside.

  * •

We identify NEL deployment on domains with strange, possibly randomly
generated, names that employ collectors hidden behind a shortened URL.

This paper is organized as follows. Section 2 explains how NEL works. Section
3 overviews related work. Section 4 describes the methodology and results of
the analyses of the HTTP Archive of the NEL deployment. Section 5 discusses
the results of this paper. Section 6 concludes this paper.

##  2 NEL BACKGROUND

NEL was introduced by researchers mostly affiliated with Google [Burnett et
al., 2020]. The World Wide Web Consortium (W3C) is in the process of
standardizing NEL [W3C, 2021]. NEL introduces NEL HTTP header sent by an HTTP
server that contains the NEL policy of the server. In addition, Report-To HTTP
header determines web servers that collect NEL reports for the visited domain.

The goal of NEL is to let web server operators instruct their visitors to
report failures in attempting to visit the service as well as successful
visits. A web server operator can control the fraction of failures
(failure_fraction) and success reports (success_fraction), and other
parameters in a NEL policy. The policy is valid for a limited time (max_age
parameter of a policy). By default, the policy applies to the visited domains,
but a server can also instruct the clients to apply the policy to subdomains
(include_subdomains parameter of a policy).

Figure 1 shows an example of a NEL report. The client reports the age of the
error (the browser often sends messages with a delay) and its type.
Additionally, the report contains other information about the event so that an
operator can react to the message.

    {
      "age": 0,
      "type": "network-error",
      "url": "https://www.example.com/",
      "body": {
        "sampling_fraction": 0.5,
        "referrer": "http://example.com/",
        "server_ip": "2001:DB8:0:0:0:0:0:42",
        "protocol": "h2",
        "method": "GET",
        "request_headers": {},
        "response_headers": {},
        "status_code": 200,
        "elapsed_time": 823,
        "phase": "application",
        "type": "http.protocol.error"
      }
    }

Figure 1: An example of a NEL report [W3C, 2021].

As browsers store only the last policy retrieved for each domain, the HTTP
server should install the same policy on all pages on each domain.

##  3 RELATED WORK

Goenka et al. [Goenka et al., 2022] focused on using dynamic NEL configuration
to show how NEL can be used for active measurements critical to CDN operations
such as alternate PoP measurements. Besides the main topic, Goenka et al. also
briefly analyzed the HTTP Archive. This paper differs as (1) they analyzed
only three samples in 2020 whereas this paper covers data between 2018 and
2023, (2) this paper provides deep analysis of the deployed policies not
available in their paper, (3) they did not describe the deployment strategies
and (4) did not foucus on the collector diversity; although they report a
rapid increase of reporting URLs, they do not investigate how many domains
operate these URLs.

Consequently, this paper provides much deeper analysis from the longitudinal
point of view as well as qualitative view.

Jeřábek and Polčák [Polčák and Jeřábek, 2023] studied NEL from the data
protection perspective raising serious questions about the legality of NEL
deployment in the European Union. Additionally, they explore security issues
concerning NEL. The data on the NEL deployment in this paper sheds more light
on the seriousness of the issues. To our best knowledge, all other papers
concerning NEL utilize NEL as a tool. This paper is the first that
systematically monitors the NEL deployment.

##  4 HTTP ARCHIVE ANALYSIS

This section analysis NEL deployment to learn whether NEL is actively
deployed.

###  4.1 Methodology

HTTP Archive periodically crawls millions of URLs at least once a month and
stores information about the fetched resources, including HTTP headers [HTTP
Archive, 2022]. The crawled data is publicly accessible for analysis via
Google Cloud Big Query. HTTP Archive has been crawling the web since November
2011. The data from each crawl is organized in a separate table with a date
label. Besides the original HAR logs222https://docs.fileformat.com/web/har/,
HTTP Archive provides summary tables with preprocessed records. A record is a
preprocessed HTTP request-response pair. Each page visit during a crawl may
include several HTTP requests and responses for additional resources such as
JavaScript, CSS, images, or other resources needed for the page load. Hence, a
single page visit is stored as multiple records.

We analyzed the summary tables since they contain all the necessary
information. We process all records of each page visit. However, we only
analyze the first record of each unique domain to remove duplicates since we
expect that operators deploy NEL policies consistently.

Moreover, HTTP Archive distinguishes mobile and desktop crawls. Some URLs
appear in both crawls. We investigated the overlapping domains and checked
that both crawls contain the same NEL policies. Hence, we merged the mobile
and desktop records and further analyzed overlapping domains once.

We base our analysis on data from HTTP Archive because it (1) is publicly
available, (2) includes HTTP headers, (3) contains historical records, and (4)
contains a vast amount of visited websites. To our best knowledge, we are not
aware of any better data source providing a comparable amount of data on NEL
deployment.

We analyze the data from the 1st February desktop and mobile crawl in each of
the last six years (February 2018 to February 2023). We selected the day
because a crawl labeled with that day occurs in every examined year for both
mobile and desktop. We extracted the data using SQL-like Big Query
jobs333https://github.com/kjerabek/nel-http-archive-analysis and further
processed in Python.

###  4.2 Results

In 2018, no service deployed NEL as the first implementation of NEL appeared
in Chromium at the end of 2018. The adoption raised in the following years.
However, the share dropped a little in 2022. Section 5 discusses a possible
explanation for the drop. Table 1 contains the number of all domains included
in the HTTP Archive, the number and the share of domains that deployed NEL and
responded with NEL policy.

Table 1: Unique domains queried within each crawl and those responding with valid NEL headers. Date | Domains | NEL | NEL [%]
---|---|---|---
Feb 2018 | $1\,022\,970$ | $0$ | $0$
Feb 2019 | $5\,707\,189$ | $355$ | $0.01$
Feb 2020 | $6\,636\,205$ | $109\,483$ | $1.65$
Feb 2021 | $10\,147\,089$ | $1\,004\,279$ | $9.90$
Feb 2022 | $10\,363\,447$ | $960\,033$ | $9.26$
Feb 2023 | $19\,159\,613$ | $2\,247\,233$ | $11.73$

The analysis below examines only the valid NEL responses. We only consider the
domains that deployed NEL. Other domains are not analyzed further. We do not
repeat the exact date of the crawls below and refer to each sample by its year
only.

Our analysis revealed several known companies and organizations using NEL.
Booking.com first deploys NEL in the 2020 records, and Cloudflare first
injected NEL headers in the 2021 records. Wikimedia monitors its services like
Wikipedia.org and first appears in the 2021 records. Reddit monitors
reddit.com and other language mutations in the 2022 and later records.
Microsoft monitors Office365 websites like sharepoint.com in the 2022 and 2023
records. Other companies experiment with NEL. For example, a bank site
Raiffeisen.ch signals NEL on domain boerse.Raiffeisen.ch, dell.com signals NEL
on dl.dell.com and downloads.dell.com, fastly-insights.com deploys NEL on some
subdomains only. Facebook uses NEL on www.beta.facebook.com and reports to
www.beta.facebook.com.

####  4.2.1 NEL Collectors

An operator of a web service may deploy their own NEL collector, or the
operator may employ a _NEL collector as a service_ (collector hosting)
[Burnett et al., 2020]. The collector-hosting provider would typically process
reports for many domains. Our analyses revealed both collector hosting and
private (self-hosted) collectors.

The original paper [Burnett et al., 2020] deems the collector diversity
crucial. It favors the services that send NEL reports to a completely
different domain to collect NEL reports in more failure scenarios. For
example, developer.valvesoftware.net collects NEL at reports.valve.net,
www.yandex.com, and various other Yandex domains report to dr.yandex.net and
dr2.yandex.net. In contrast, others signal to the same domain name, like
fel.cvut.cz and www.staremesto.sk.

In addition, we analyzed collector URLs and domain names and identified two
strategies of NEL collector naming to distinguish the origin. (1) NEL
collectors have different subdomain for each origin. An example of such a
collector is report-uri.com: domain www.expobeds.com reports to
expobeds.report-uri.com, whereas mattferderer.com reports to
mattferderer.report-uri.com. (2) All reporting domains share the same NEL
collector domain name, and the URL path or parameters encode an identifier to
distinguish the reporting domain. An example of such a service is
cloudflare.com, which uses 136-characters-long Base64 encoded string
identifiers in URL parameters.

To provide fair statistics that are not skewed by the two collector naming
strategies, we denote a _NEL collector provider_ as a set of NEL collectors
with the same second-level domain (SLD). At the beginning of 2019, only 11
(mostly self-hosted) collector providers appeared; the number raised to 147
collectors in 2023, see Table 2.

Table 2: The count of NEL collector providers, the top four NEL collector providers for each analyzed year, and their share over the analyzed period. Year | Count | Top 4 providers | Share [%]
---|---|---|---
2019 | 11 | report-uri.com | $87.32$
3gl.net | $5.63$
uriports.com | $2.54$
seloc.club | $1.41$
2020 | 36 | shopifycloud.com | $98.84$
report-uri.com | $0.56$
powerboutique.net | $0.23$
fastly-insights.com | $0.07$
2021 | 84 | cloudflare.com | $77.44$
shopifycloud.com | $22.26$
wikimedia.org | $0.12$
report-uri.com | $0.09$
2022 | 115 | cloudflare.com | $97.72$
cafe24.com | $1.79$
wikimedia.org | $0.13$
report-uri.com | $0.10$
2023 | 147 | cloudflare.com | $97.91$
cafe24.com | $0.94$
freshedge.net | $0.34$
office.net | $0.33$

In 2019, report-uri.com, a collector-hosting service, prevailed. In later
years, big CDN providers Shopify and Cloudflare appeared and deployed NEL on
the websites they hosted. Both utilize their own NEL collector. Cloudflare’s
collector has appeared in the reporting destinations of hundreds of thousands
of domains in the 2021 crawl and even increased to almost two million in 2023.

Some collector providers collect NEL reports from a single domain, whereas
others are shared between multiple domains, see Table 3. The number of
providers shared between many domains is rising. For example, nelreports.net
first appeared in 2020, serving 26 domains; it raised to 57 in 2021, 73 in
2022, and 1177 in 2023. Nevertheless, the number of providers employed by a
small number of domains also rises significantly.

Table 3: The number of NEL collector providers that are employed by the given number of domains. Domains | 2019 | 2020 | 2021 | 2022 | 2023
---|---|---|---|---|---
1 | 3 | 16 | 48 | 52 | 74
2 | 4 | 4 | 8 | 18 | 19
3–10 | 2 | 4 | 11 | 20 | 26
11–100 | 1 | 9 | 11 | 15 | 11
101–1K | 1 | 2 | 3 | 7 | 9
More | 0 | 1 | 3 | 3 | 8

####  4.2.2 NEL Policy Settings

We analyzed the NEL policies in HTTP Archive dataset and studied
failure_fraction, success_fraction, include_subdomains, and max_age that
appear in the wild.

In the beginning, several different values appeared in failure_fraction
policies, which reflects the early adoption and possibly experimentation with
the newly introduced technology. In 2020, the values were set to report mostly
a small share of errors (10 % in 99 % of cases). However, since 2021 almost
all operators have been interested in all errors, which is the default. Only
one domain in 2020 was not interested in failures at all. The desire to
collect failures is understandable, as failure detection is the main
motivation behind NEL. Even so, 21,606 domains collect only 10 % of the
failures, and 5,429 domains collect 5 % of the failures or less (in 2023).
Table 4 shows the settings observed each year.

Table 4: Observed failure fraction values. The symbol $\updownarrow$ denotes an open interval of values between range values on the row above and below. Value | 2019 | 2020 | 2021 | 2022 | 2023
---|---|---|---|---|---
0 | 0 | 1 | 0 | 0 | 0
$\updownarrow$ | 224 | 297 | 363 | 458 | 3761
0.01 | 1 | 108K | 78 | 107 | 137
$\updownarrow$ | 0 | 0 | 0 | 1 | 5
0.05 | 0 | 4 | 1269 | 1328 | 1526
$\updownarrow$ | 0 | 0 | 0 | 0 | 0
0.1 | 3 | 266 | 167 | 17K | 22K
$\updownarrow$ | 0 | 2 | 3 | 19 | 78
0.25 | 0 | 0 | 5 | 6 | 8
$\updownarrow$ | 0 | 0 | 0 | 0 | 2
0.5 | 2 | 3 | 3 | 9 | 72
$\updownarrow$ | 1 | 0 | 0 | 9 | 6
1 | 124 | 673 | 1.0M | 941K | 2.2M

Regarding success_fraction, we typically observed ranges from $0.0$ to $0.01$,
as shown in Table 5. The default value of $0.0$ dominates in all years except
for 2020 where the servers injected the policy of $0.001$ in most cases.
Although most servers limit the number of success reports, in the 2020 and
later crawls, at least two domains are interested in all success reports in
each analyzed year.

Table 5: Observed success fraction values. The symbol $\updownarrow$ denotes an open interval of values between range values on the row above and below. Value | 2019 | 2020 | 2021 | 2022 | 2023
---|---|---|---|---|---
0 | 350 | 1156 | 780K | 907K | 1.6M
$\updownarrow$ | 1 | 108K | 224K | 805 | 9267
0.01 | 2 | 1 | 19 | 51K | 620K
$\updownarrow$ | 0 | 0 | 0 | 0 | 3
0.05 | 0 | 14 | 7 | 21 | 41
$\updownarrow$ | 0 | 0 | 0 | 0 | 0
0.1 | 0 | 0 | 2 | 3 | 8
$\updownarrow$ | 0 | 0 | 0 | 9 | 509
0.25 | 2 | 2 | 3 | 0 | 0
$\updownarrow$ | 0 | 0 | 0 | 0 | 0
0.5 | 0 | 0 | 0 | 0 | 4
$\updownarrow$ | 0 | 0 | 0 | 0 | 0
1 | 0 | 2 | 6 | 4 | 19

As NEL policies are not applied to subdomains by default, operators need to
intentionally change the policy to include subdomains. The non-default setting
prevailed only in the first year of the NEL deployment. Later, the policies
apply to subdomains in about or less than $1\%$ of cases. Table 6 shows the
precise distribution in all years.

Table 6: The share of NEL-signalling servers with include_subdomains set to true. Year | true [%]
---|---
2019 | 98.03
2020 | 1.02
2021 | 0.14
2022 | 0.17
2023 | 0.63

The expiration policy, set by max_age, ranges from seconds to years since the
maximal value is not limited. Nevertheless, the most common values occurring
are hours, 1 day, 7 days, and 30 days. Hundreds of domains inject policies
valid for one year. The maximal value observed was two years, but it appeared
only in a few cases. Some servers removing NEL policies with max_age=0
appeared each year. Table 7 shows the distribution of max_age values in each
year.

Table 7: Observed max_age values. The symbol $\updownarrow$ denotes an open interval of values between range values on the row above and below; _ds_ stands for days. Value | 2019 | 2020 | 2021 | 2022 | 2023
---|---|---|---|---|---
0 | 1 | 5 | 9 | 6 | 7
$\updownarrow$ | 3 | 42 | 207 | 318 | 3645
1 hour | 228 | 302 | 466 | 1177 | 840
$\updownarrow$ | 0 | 333 | 0 | 78 | 7485
1 day | 5 | 50 | 1490 | 1562 | 2866
$\updownarrow$ | 1 | 0 | 0 | 7 | 54
7 ds | 3 | 62 | 778K | 939K | 2.2M
$\updownarrow$ | 2 | 68 | 76 | 94 | 87
30 ds | 29 | 108K | 224K | 18K | 30K
$\updownarrow$ | 4 | 10 | 63 | 21 | 32
365 ds | 79 | 234 | 486 | 489 | 881
$\updownarrow$ | 0 | 2 | 2 | 8 | 9
730 ds | 0 | 1 | 1 | 1 | 2

##  5 DISCUSSION

This section discusses the results and limitations of this paper.

###  5.1 Observations From the HTTP Archive Analysis

Since 2020 the default policy values have prevailed. However, non-default
values also appear in the data set. We observed values for each variable
across its validity range. The NEL deployment of Cloudflare causes the
uniformity of the observed values. We observed only two policies that send
data to the NEL collector of Cloudflare that differ in success_fraction only.

We suspect that Shopify stopped signaling NEL headers during 2021 and we
observed the drop in NEL share in 2022. In 2023, over 97 % of NEL deployment
is controlled by a single entity — Cloudflare. Hence, the observed 11.73 %
share of NEL deployment strongly depends on Cloudflare adoption. Should
Cloudflare remove NEL, NEL deployment share would dwindle.

We did not find any traces of Google’s NEL deployment. Even when we manually
visited Google domains, we did not see NEL headers. The original NEL paper
[Burnett et al., 2020] mentions two implementations. One is called _Domain
Reliability_ and was hard-coded in Chrome since 2014 (version 38). The other
is based on HTTP headers, as studied in this paper. The original NEL paper
[Burnett et al., 2020] claims that Google was migrating from Domain
Reliability to NEL. However, we have not observed Google sending NEL headers;
it looks like Google never deployed NEL based on HTTP headers. Another
explanation is that Google signals NEL headers only to a fraction of visitors,
and neither our manual attempts nor the crawls of HTTP Archive triggered the
injection.

In the last two years’ records, we have encountered a few cases of randomly
generated domains, like gkjw.org and embeumkm.com. These domains were
injecting NEL policies with the reporting destination set to shortened URLs
hosted by bit.ly. The policy setting used default values, except it included
subdomains and lasted one year. Such an approach demonstrates how malicious
actors can misuse NEL to monitor their deployed services while hiding their
NEL collectors, as it is impossible to retroactively learn the true identity
of these collectors from HTTP Archive.

###  5.2 Limitations of this study

During the HTTP Archive analyses, we focused on the first response of each
domain. We did not analyze if the NEL header setting is consistent within a
domain. As HTTP Archive focuses on main pages and pages linked from the main
page [HTTP Archive, 2022], only a minority of resources are crawled on most
domains. Hence, a sound analysis of the consistency of the NEL policies is not
possible based on HTTP Archive. For these reasons and because HTTP Archive
does not crawl data from various locations, we cannot confirm if NEL leaks
data to third parties in practice.

The original NEL paper [Burnett et al., 2020] recommends hosting collectors in
multiple physical and network (logical) locations. However, data concerning
physical and network locations are not available in the HTTP Archive.
Consequently, Section 4 does not study some characteristics of NEL deployment,
like collector diversity. Further work can amend Section 4 with a study of
additional aspects of NEL deployment.

A limitation of the study of NEL collectors (subsection 4.2.1) is that we
classify NEL collectors according to the second-level domain rather than the
effective TLD+1. Consequently, we aggregate all collectors in co.uk and co.th
domains (both domains have two collectors). Additionally, we aggregate tenants
of some cloud providers like Amazon AWS (amazonaws.com). We accepted the
limitation because (1) the number of collectors in the TLDs without the
possibility to register second level is tiny so the results are not
significantly affected and (2) it is not clear where to draw the line (should
the Amazon AWS tenants be counted separately or together; in the former case,
what other services should be treated similarly?).

##  6 CONCLUSION

Error monitoring is a crucial activity of web server operators. Even though
Chromium-based browsers have supported NEL for several years, NEL deployment
was not systematically studied. Our paper studies the current NEL deployment
and revealed that NEL is dominated by Cloudflare. Its customers employ one of
two policies. Nevertheless, we observe other important players that deployed
NEL or experiment with the deployment.

## ACKNOWLEDGMENTS

This work was supported by the Brno University of Technology grant Smart
information technology for a resilient society (FIT-S-23-8209).

## REFERENCES

  * [Burnett et al., 2020] Burnett, S., Chen, L., Creager, D. A., Efimov, M., Grigorik, I., Jones, B., Madhyastha, H. V., Papageorge, P., Rogan, B., Stahl, C., and Tuttle, J. (2020).  Network error logging: Client-side measurement of end-to-end web service reliability.  In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, pages 985–998. USENIX Association.
  * [Franke, 2012] Franke, U. (2012).  Optimal IT service availability: Shorter outages, or fewer?  IEEE Transactions on Network and Service Management, 9(1):22–33.
  * [Goenka et al., 2022] Goenka, P., Zarifis, K., Gupta, A., and Calder, M. (2022).  Towards client-side active measurements without application control.  SIGCOMM Comput. Commun. Rev., 52(1):20–27.
  * [HTTP Archive, 2022] HTTP Archive (2022).  Methodology — The Web Almanac by HTTP Archive.  https://almanac.httparchive.org/en/2022/methodology.
  * [Polčák and Jeřábek, 2023] Polčák, L. and Jeřábek, K. (2023).  Data protection and security issues with network error logging.
  * [W3C, 2021] W3C (2021).  Network error logging.  The World Wide Web Consortium (W3C) internal document, https://w3c.github.io/network-error-logging/, Editor’s Draft of 30th July 2021.


* [19] J. W. P. Hirschfeld, G. Korchmáros, and F. Torres.  Algebraic curves over a finite field.  Princeton Series in Applied Mathematics. Princeton University Press, Princeton, NJ, 2008.
  * [20] C.-P. Jeannerod, V. Neiger, E. Schost, and G. Villard.  Fast Computation of Minimal Interpolation Bases in Popov Form for Arbitrary Shifts.  In International Symposium on Symbolic and Algebraic Computation, ISSAC ’16, pages 295–302, New York, NY, USA, 2016. ACM.
  * [21] C.-P. Jeannerod, V. Neiger, E. Schost, and G. Villard.  Computing minimal interpolation bases.  Journal of Symbolic Computation, 83:272–314, Nov. 2017.
  * [22] T. Kailath.  Linear Systems.  Prentice-Hall, 1980.
  * [23] C. Kirfel and R. Pellikaan.  The minimum distance of codes in an array coming from telescopic semigroups.  volume 41, pages 1720–1732. 1995.  Special issue on algebraic geometry codes.
  * [24] K. Lee, M. Bras-Amoros, and M. O’Sullivan.  Unique Decoding of General AG Codes.  IEEE Transactions on Information Theory, 60(4):2038–2053, Apr. 2014\.
  * [25] K. Lee and M. E. O’Sullivan.  List Decoding of Reed–Solomon Codes from a Gröbner Basis Perspective.  Journal of Symbolic Computation, 43(9):645 – 658, 2008.
  * [26] K. Lee and M. E. O’Sullivan.  List decoding of Hermitian codes using Gröbner bases.  Journal of Symbolic Computation, 44(12):1662–1675, 2009.
  * [27] R. McEliece.  The Guruswami-Sudan Decoding Algorithm for Reed-Solomon Codes.  IPN progress report, pages 42–153, 2003.
  * [28] S. Miura and N. Kamiya.  Geometric-goppa codes on some maximal curves and their minimum distance.  Proceedings of 1993 IEEE Information Theory Workshop, pages 85–86, 06 1993.
  * [29] V. Neiger, J. Rosenkilde, and E. Schost.  Fast Computation of the Roots of Polynomials Over the Ring of Power Series.  In International Symposium on Symbolic and Algebraic Computation, July 2017.
  * [30] V. Neiger and T. X. Vu.  Computing Canonical Bases of Modules of Univariate Relations.  In International Symposium on Symbolic and Algebraic Computation, page 8, July 2017.
  * [31] H. Niederreiter and C. Xing.  Rational points on curves over finite fields: theory and applications, volume 285 of London Mathematical Society Lecture Note Series.  Cambridge University Press, Cambridge, 2001.
  * [32] J. Nielsen and P. Beelen.  Sub-Quadratic Decoding of One-Point Hermitian Codes.  IEEE Transactions on Information Theory, 61(6):3225–3240, June 2015\.
  * [33] R. R. Nielsen and T. Høholdt.  Decoding reed-solomon codes beyond half the minimum distance.  In Coding Theory, Cryptography and Related Areas, pages 221–236. Springer, 2000.
  * [34] H. O’Keeffe and P. Fitzpatrick.  Gröbner basis solutions of constrained interpolation problems.  Linear algebra and its applications, 351:533–551, 2002.
  * [35] J. Rosenkilde and A. Storjohann.  Algorithms for simultaneous hermite–padé approximations.  Journal of Symbolic Computation, 102:279 – 303, 2021.
  * [36] R. Roth and G. Ruckenstein.  Efficient Decoding of Reed–Solomon Codes Beyond Half the Minimum Distance.  IEEE Transactions on Information Theory, 46(1):246 –257, 2000.
  * [37] S. Sakata.  Extension of the Berlekamp-Massey algorithm to $N$ dimensions.  Information and Computation, 84(2):207–239, 1990.
  * [38] S. Sakata and M. Fujisawa.  Fast Decoding of Multipoint Codes from Algebraic Curves.  IEEE Transactions on Information Theory, 60(4):2054–2064, Apr. 2014\.
  * [39] S. Sakata, H. E. Jensen, and T. Høholdt.  Generalized Berlekamp-Massey Decoding of Algebraic-Geometric Codes up to Half the Feng–Rao Bound.  IEEE Transactions on Information Theory, 41(6):1762–1768, 1995\.
  * [40] J.-P. Serre.  Rational points on curves over finite fields, volume 18 of Documents Mathématiques (Paris).
  * [41] V. Shoup.  A fast deterministic algorithm for factoring polynomials over finite fields of small characteristic.  In Proceedings of the 1991 international symposium on Symbolic and algebraic computation, pages 14–21, 1991.
  * [42] G. Solomatov.  Computational aspects of Algebraic Geometry codes.  PhD thesis, Technical University of Denmark, 2021.
  * [43] H. Stichtenoth.  Algebraic Function Fields and Codes.  Springer, 2nd edition, 2009.
  * [44] Vincent Neiger.  Bases of relations in one or several variables: fast algorithms and applications.  PhD Thesis, ENS Lyon, Nov. 2016.
  * [45] J. von zur Gathen and J. Gerhard.  Modern Computer Algebra.  Cambridge University Press, 3rd edition, 2012.
  * [46] W. Zhou and G. Labahn.  Computing Column Bases of Polynomial Matrices.  In International Symposium on Symbolic and Algebraic Computation, ISSAC ’13, pages 379–386, New York, NY, USA, 2013. ACM.


# An Objective Evaluation Framework for Pathological Speech Synthesis

Bence Mark Halpern1,2,3, Julian Fritsch4,5, Enno Hermann4,5, Rob van Son1,3
Odette Scharenborg2, Mathew Magimai.-Doss4

###### Abstract

The development of pathological speech systems is currently hindered by the
lack of a standardised objective evaluation framework. In this work, (1) we
utilise existing detection and analysis techniques to propose a general
framework for the consistent evaluation of synthetic pathological speech. This
framework evaluates the voice quality and the intelligibility aspects of
speech and is shown to be complementary using our experiments. (2) Using our
proposed evaluation framework, we develop and test a dysarthric voice
conversion system (VC) using CycleGAN-VC and a PSOLA-based speech rate
modification technique. We show that the developed system is able to
synthesise dysarthric speech with different levels of speech intelligibility.

##  1 Introduction

In recent years, there has been a growing interest in pathological speech
processing [1]. It is a highly challenging area, as our understanding of
speech is largely limited to “typical" and unimpaired speech. The advances
made in detection and analysis of pathological speech are continuously
improving our understanding of it. These efforts could be further accelerated
with pathological speech synthesis - as speech processing has advanced through
the synergy between "analysis" and "synthesis". For instance, if one considers
the message component in the speech signal, automatic speech recognition (ASR)
can be regarded as analysis and text-to-speech synthesis (TTS) can be regarded
as synthesis. Such understandings have benefited both ASR and TTS and have
driven innovation, e.g., [2, 3, 4, 5].

Another motivation for the development of pathological speech synthesis is
that it could assist in informed decision making for the medical conditions at
the root of the pathological speech. For instance, oral cancer surgery results
in changes to a speaker’s voice. Availability of a synthesis model that can
generate how the voice could sound after surgery could help the patients and
clinicians to make informed decisions about the surgery and alleviate stress
of the patients [6].

Pathological speech synthesis is a non-trivial task due to two main
challenges. Firstly, state-of-the-art methods for TTS/voice conversion (VC)
often rely upon the availability of a large amount of speech data from many
different speakers and linguistic resources. In that sense, pathological
TTS/VC is inherently under-resourced, as the collection of pathological speech
is time-consuming and is governed by medical research ethical aspects.

Secondly, a crucial aspects in the development of TTS/VC systems is the
evaluation of the naturalness of the synthesised speech. This is typically
done by asking a group of naive human listeners to rate the synthesised speech
on its naturalness [7]. The most common example of such an evaluation by naive
listeners is the mean opinion score (MOS). MOS is a good evaluation measure
for TTS of typical speech where the only deterioration in the speech is due to
the decrease in naturalness. However, it is not appropriate for pathological
speech, for several reasons. Firstly, a pathological VC system that receives
better MOS scores than the reference pathological speech is likely not able to
capture the characteristics of the pathological speech. Secondly, on the
reverse, a VC system mimicking the pathology, albeit in an exaggerated manner,
would likely produce a MOS score that is a lot lower than that of the
reference. Thirdly, it is unknown whether non-expert listeners have sufficient
expertise to evaluate pathological speech. Fourthly, involving speech-language
pathologists for evaluation at every stage of development would make the
evaluation process not only costly but it is also not certain that they can
distinguish the unnaturalness of the synthesised speech from the
characteristics of the pathology.

To alleviate this problem, the present paper

  * (a)

proposes a new framework, based on the synthesis-analysis approach for ASR,
for the consistent evaluation of limited-resource, synthesised pathological
speech based on a combination of pathological speech detection and analysis
methods.

  * (b)

and, to that end, investigates a CycleGAN-based approach to convert healthy
control speech to pathological speech of different levels of intelligibility.

Specifically, in this paper, we focus on the synthesis of dysarthric speech
and its evaluation through pathological speech detection (SD) and automatic
intelligibility assessment. We demonstrate the viability of the proposed
framework through an investigation on the UASpeech corpus.

The paper is organised as follows: the UASpeech database used for training and
evaluation is described in Section 2. The employed CycleGAN-VC method is
described in Section 3. We present the proposed objective evaluation framework
in Section 4 and the evaluation results in Section 5. Speech samples and a
demo of the system is available
online.111https://karkirowle.github.io/publication/itg-2021-dysarthric-vc

##  2 Database

For all experiments, we use the UASpeech database [8], which contains 14
dysarthric speakers, 10 male and 4 female, and 455 unique isolated word
recordings for each speaker. The speakers have received an intelligibility
rating in the database (high, mid, low, very low) based on their average
scores in a word transcription task rated by five non-expert listeners
(subjective intelligibility score). In our experiments, low and very low
intelligibility speakers are grouped together to increase statistical power.
In addition, the database contains 9 male and 4 female unimpaired, control
speakers. All recordings were done with multiple microphones and repetitions,
however, only microphone 5 and the first repetition of each word was used in
our experiments. We refer to the healthy speakers’ utterances from the
database as ground truth (GT) healthy, and the dysarthric utterances as ground
truth dysarthric from now on.

Figure 1: Training (blue) and conversion (red) setup for dysarthric voice
conversion.

##  3 Proposed conversion system

We train a voice conversion model for each dysarthric speaker in the database,
using CM08 as a male source speaker and CF04 as a female source speaker. This
means that – contrary to traditional VC – we are not only converting speaker
traits, we are also aiming to change the intelligibility of the speech. The
overall experimental setup is illustrated in Figure 1. In the following
sections, we will explain the techniques used in detail.

###  3.1 Speech rate modification and analysis

VC systems (including CycleGAN-VC) do not change the speech rate, but for
pathological speech applications, it is essential that VC systems convert this
aspect of the speech: e.g., in the case of dysarthric speech, speech rate has
been shown to be a reliable predictor of speech severity [9]. Therefore, prior
to conversion, the tempo of the source speech is adjusted to the dysarthric
speaker’s tempo using the pitch synchronous overlap-add algorithm (PSOLA) [10]
from Praat [11]. We’ve run preliminary tempo modification experiments with
WSOLA (sox implementation) and phase vocoding (librosa [12] implementation)
techniques too, but we found that PSOLA results in the most natural sounding
speech. We would like to emphasise that, even though the target speaker’s
tempo is not generally known beforehand, for all practical purposes, the
expected speech rate can be estimated if the severity is known. Such
estimation has been already done in various works, such as [13, 9, 14].

Furthermore, a tempo-based data augmentation scheme is used to increase the
dataset size in the CycleGAN-VC training, which means that the same speech
signal is fed into the neural networks three times with three different speech
rates, modified using the pitch synchronous overlap-add algorithm from Praat.
The first version is the original unadjusted version of the speech signal,
spoken at the rate of the control speaker. The second version is adjusted to
the speech rate of the dysarthric speaker. The last version is adjusted to a
rate exactly halfway in between the original unadjusted and the dysarthric
adjusted. This tempo-based data augmentation scheme increases the size of the
training set three-fold. We refer to the dataset obtained after the
augmentation as the augmented database, and the speech adjusted to dysarthric
speech rate as adjusted speech. Subsequently, the augmented database is
preprocessed using the WORLD vocoder [15] to obtain the Mel-generalised
cepstrum (MCEP), the pitch ($F_{0}$) and the aperiodicities (AP).

###  3.2 CycleGAN-VC training and conversion

Previous studies have found that CycleGAN-VC-based systems can improve the
speech intelligibility of a dysarthric signal [16, 17]. For this reason, we
are interested if a CycleGAN-VC based system can also model the deterioration
of speech intelligibility too. An additional benefit of using the CycleGAN-VC
is that the system is known to work with small amount (approximately 10 mins)
of data, which can be non-parallel. This makes CycleGAN-VC lucrative for
pathological speech applications which are inherently low-resourced.

A CycleGAN-VC system is trained with the augmented database [18]222The
implementation used was https://github.com/leimao/Voice_Converter_CycleGAN
with the same hyperparameters and values (as most hyperparameters are directly
adapted from the original CycleGAN paper) for 1000 epochs [19]. During
conversion of the healthy speech to dysarthric speech, only the healthy tempo-
adjusted speech is used. The MCEPs are then converted using the trained model.
The APs are copied. During training, the mean and standard deviation of
$F_{0}$ values are calculated for both speakers (source and target). The
source $F_{0}$ is mean-std normalised to the source speaker’s $F_{0}$
distribution and mean-std unnormalised to the target speaker’s $F_{0}$
distribution.

##  4 Objective evaluation framework

Pathological speech deviates from healthy speech along several different
dimensions. Therefore, the proposed system has four parts, each of which
evaluates the pathological speech signal generated by the VC from a different
angle, together yielding a consistent evaluation measure: a pathological
speech detector (Section 4.1), a voice quality measure (Section 4.2), an
intelligibility assessment tool (Section 4.3) and an ASR system (Section 4.4).

Before the evaluation experiments, voice activity detection (VAD) is performed
using Praat to avoid analysis of non-speech cues [11]. This VAD is manually
checked for quality. Ideally, VAD would be done before training, however, our
experiments found that in this case the CycleGAN-VC does not converge well due
to the short duration of the utterances.

###  4.1 Voice quality measure: LTAS-LASSO-SD

Long time average spectrum (LTAS) has been successfully used as a voice
quality measure [20] and predictor for various speech pathologies [21, 22].
Thus, an LTAS based pathological speech detector (SD) will be trained on
detecting dysarthric speech using the voice quality cues provided by the LTAS-
feature, assessed for generalisation using the GT samples, and then used to
evaluate our converted (VC) samples.

The leave-one-out validation scheme used for the LTAS-LASSO-SD is illustrated
in Figure 2. The main factors considered during the design of this validation
scheme were that (1) the speaker who is evaluated should not be included in
the training data (2) each speaker should be mapped with a single control
speaker so that everyone is only left out during its own evaluation round and
that the number of control and pathological speakers remain changed. Two male
speakers had no control pairs, so we excluded one with very high
intelligbility (M10) and one with very low (M01). The weakness of this
experiment design is that male speakers are overrepresented in the dataset.
However, the alternative would be one single speaker per gender per severity,
which we found an inferior design.

Figure 2: Leave-one-out validation scheme for LTAS-LASSO-SD. Purple colour
shows the excluded speakers. The conversion pair (red) is excluded from the
evaluation, and in each fold a different speaker pair (yellow) is left out.
The green colour shows the speakers that we are training on. This is repeated
for the two genders separately.

First, an LTAS is extracted based on a 512-point FFT with Hann window and 128
sample frame shift with librosa [12] for the control, VC and original
dysarthric speech.

The chosen detector model was a LASSO due to its automatic feature selection
property. The sparsity penalty was $\alpha=10^{-5}$. The model was trained for
a maximum of 1000 iterations. The VC samples are then evaluated on these
detector models in terms of accuracy. Furthermore, we investigate to what
extent voice quality is influenced by the subjective intelligibility of
utterances (taken from the UASpeech database) by performing a Pearson’s
correlation ($r$) between the subjective intelligibility scores and mean
detection scores of the VC utterances.

###  4.2 Intelligibility decrease measure: LTAS-SKL measure

The idea of the LTAS-SKL measure is the following: each word of a reference
speaker is compared to the same word of all other speakers (control and
pathological; VC and GT). The comparison is done on the distribution of LTAS
energy bins using the symmetric Kullback-Leibler (SKL) divergence. The
speakers are then grouped based on intelligibility and the results visualised
using a box plot. This way we ensure that the difference in the utterances can
only be due to the individual differences in speakers (spread) and due to the
differences in intelligibility of the healthy ground-truth speech and the
generated pathological speech (median). Here, a normalised LTAS with a
1024-point FFT with Hann window and 256 sample frame shift is extracted. A
t-test is used to check the significance of the difference in intelligibility.

###  4.3 PPG-DTW-based utterance verification

The phonetic posteriorgram (PPG)-dynamic time warping (DTW) method [23]
evaluates pathological utterances by matching them to a healthy reference
utterance of the same word and aligning their PPG sequences with DTW. To
verify an utterance, the DTW matching score is converted into a probability
$P_{c}$, by plugging it into a logistic function, $c$ denoting the control
class; an utterance is hence verified correctly if $P_{c}\geq 0.5$. A
speaker’s intelligibility is then estimated in terms of the percentage
correctly verified utterances, just as in human listening tests.

For our experiments, we used an SKL local cost function in the DTW
implementation and posterior sequences of 45 context-independent phones (PPG).
Spearman’s $\rho$ and Pearson’s correlation $r$ analyses between the
intelligibility estimates of the dysarthric speech GT and the VC speech, show
that the method performs at $\rho=.965$, $r=.854$, which is comparable to
other state-of-the-art methods such as [24]. The evaluation is repeated on the
WORLD vocoder resynthesised GT dysarthric signals to see if vocoding affects
the intelligibility estimates.

###  4.4 ASR based intelligibility evaluation

As an additional objective intelligibility measure, we use the word error rate
(WER) of an ASR system trained only on unimpaired control speakers. We examine
whether the VC speech gives comparable results to the GT dysarthric data to
show that the VC system learns to model the lower intelligibility of
dysarthric speech.

We trained acoustic models with the Kaldi ASR toolkit [25] on block 2 and 3 of
the UASpeech corpus, which contain distinct sets of words from block 1 that
was used for VC. We trained the acoustic models on the data from all
microphones and all control speakers except the 2 source speakers for VC
(CM08, CF04), based on an existing Kaldi recipe [26] and decode with a unigram
grammar that includes all words in the UASpeech corpus. To verify that the
results are not specific to a single type of model, we compare both subspace
Gaussian mixture models (SGMMs) [27] and neural networks trained with the
sequence-discriminative lattice-free maximum mutual information (LF-MMI) loss
[28], the state-of-the-art methods in the GMM and hybrid hidden Markov
model/deep neural network ASR paradigms, respectively.

Figure 3: Result of the LTAS-SKL analysis. A grey interquartile range (IQR)
indicates original samples, a red IQR indicates VC samples. The center line
indicates the median. (***) $p<.001$ (**) $p<.01$ (*) $p<.05$, otherwise not
significant. Number of speakers are written in parentheses. The ground truth
control speakers of are plotted twice on each panel for easier comparison.

##  5 Results and discussion

###  5.1 Voice quality measure: LTAS-LASSO-SD

The LTAS-LASSO-SD detectors perform well on the held-out GT speakers, with the
best held-out speaker having an accuracy of $93.15\%\pm 9.75$ (M14), the worst
held-out speaker performing with a $90.84\%\pm 10.1$ (M05). This means that
the proposed evaluation framework is able to detect the voice quality of
dysarthric speech. On the VC samples the best held-out speaker accuracy is
$100\%\pm 0$ (multiple speakers, i.e., M16, F02, F03, F04, F05) and the worst
accuracy is $92.07\%\pm 6.05$ (M11). This means that the proposed VC system
successfully mimics the voice quality aspects of dysarthric speech.

There is a linear relationship between the male speakers’ detection scores and
the subjective intelligibility scores from UASpeech (male; $r=-0.85$,
$\text{p-value}=0.003$), which shows that the intelligibility aspects of the
synthesised dysarthric speech are closely related to the voice quality aspects
in case of the male speakers. This relationship does not exist in the case of
females (female; $r=-0.02$, $\text{p-value}=0.98$). This means, while the
voice quality aspects can be heavily influenced by the intelligibility of the
speakers in the case of male speakers, the proposed voice quality measure
should be used in complement to the other, intelligibility-based measures.

###  5.2 Intelligibility decrease measure: LTAS-SKL

In Figure 3, it can be seen that both for the GT (gray) and the VC (red)
speakers the median of the LTAS increases (SKL-LTAS dB) as the intelligibility
decreases. This increase in the median is significant, except for mid-low VC
male and mid-low GT female. This means that VC performs well overall, however
it is not that well able to generate dysarthric speech that differentiates
between mid and low intelligibility.

###  5.3 PPG-DTW-based utterance verification

Table 1 shows that the correlations between the subjective and objective
intelligibility scores are high, and are significant for all male speakers,
meaning that the proposed VC model successfully captures characteristics of
dysarthric speech that influence intelligibility for male speakers. The
results also show high correlation for female speakers, though not
significant, which might be due to the lower number of female speakers.
Additionally, it can be seen that overall there is a correlation gap between
the WORLD all (copy synthesis) and VC all (conversion). This indicates that
conversion performance is not limited by the vocoder, but rather by the used
model.

Data | $\rho$ | p-value | $r$ | p-value
---|---|---|---|---
WORLD all | .954 | *** | .836 | ***
VC all | .448 | * | .481 | *
WORLD male | .940 | *** | .807 | **
VC male | .735 | ** | .594 | *
WORLD female | 1.0 | *** | .957 | *
VC female | .774 | 0.229 | .720 | 0.279
Table 1: Pearson’s and Spearman’s correlation between subjective
intelligibility score (from UASpeech) and predicted intelligibility by PPG-
DTW. (*) $p<.1$ (**) $p<.05$. Figure 4: WER results grouped by intelligibility
of the target speakers, comparing GT and VC speech using original and adjusted
tempo (VC adj).

###  5.4 ASR-based intelligibility evaluation

Figure 4 shows the WER obtained with the two ASR systems for the low to highly
intelligible dysarthric and the control speakers. As expected, the GT speech
data (solid lines) exhibits a steep drop in error rates as intelligibility
increases. The VC speech mirrors this decrease. When the speech rate is
adjusted (dotted lines), the WERs resemble the ground truth more closely. This
is consistent with previous work that has shown slower speaking rates of
dysarthric speech to harm ASR performance [29]. However, the same pattern of
fewer errors when intelligibility is higher is still apparent without tempo
adjustment (dashed lines). This shows that the VC system learns to manipulate
the speech in such a way that makes it harder for an ASR system trained only
on control speech to recognise it correctly. Both when synthesising healty and
dysarthric speech, we expect the VC speech to perform slightly worse than the
GT because of the added noise introduced by the VC, which does not match the
acoustic conditions of the ASR training data. We observe the same broad
pattern for SGMM (green) and LF-MMI (orange) models, indicating that our
analysis generalises across different types of acoustic models.

##  6 Conclusion

In this paper, we proposed an objective evaluation framework for pathological
speech synthesis. The proposed framework (1) captures the voice quality
aspects of pathological speech, and consistently shows the decrease in
intelligibility of synthesised dysarthric speech compared to healthy speech
using the (2) LTAS-SKL (3) PPG-DTW (4) and an ASR-based methods.
Simultaneously, we have developed and evaluated a proof-of-concept CycleGAN-VC
system which (1) mimics the voice quality of dysarthric speech, (2,3,4) can
exhibit different levels of speech intelligibility (4) and shows that this
decrease in intelligibility is independent of the decrease in speech rate.

##  7 Acknowledgements

This project has received funding from the EU H2020 research and innovation
programme under MSCA grant agreement No 766287 (TAPAS). The Department of Head
and Neck Oncology and surgery of the Netherlands Cancer Institute receives a
research grant from Atos Medical (Hörby, Sweden), which contributes to the
existing infratructure for quality of life research.

## References

  * [1] R. Gupta, T. Chaspari, J. Kim, N. Kumar, D. Bone, and S. Narayanan, “Pathological speech processing: State-of-the-art, current challenges, and future directions,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6470–6474, IEEE, 2016.
  * [2] K. Tokuda, T. Masuko, J. Hiroi, T. Kobayashi, and T. Kitamura, “A very low bit rate speech coder using hmm-based speech recognition/synthesis techniques,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  * [3] K. Lee and R. V. Cox, “A very low bit rate speech coder based on a recognition/synthesis paradigm,” IEEE Trans. Speech Audio Process., vol. 9, no. 5, pp. 482–491, 2001.
  * [4] J. Dines, J. Yamagishi, and S. King, “Measuring the gap between hmm-based asr and tts,” IEEE Journal of Selected Topics in Signal Processing, vol. 4, no. 6, pp. 1046–1058, 2010.
  * [5] A. Tjandra, S. Sakti, and S. Nakamura, “Listening while speaking: Speech chain by deep learning,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 301–308, IEEE, 2017.
  * [6] J. B. Epstein, S. Emerton, D. A. Kolbinson, N. D. Le, N. Phillips, P. Stevenson-Moore, and D. Osoba, “Quality of life and oral function following radiotherapy for head and neck cancer.,” Head Neck, 1999.
  * [7] Z. Wu, Z. Xie, and S. King, “The blizzard challenge 2019,” in Proc. Blizzard Challenge workshop, vol. 2019, 2019.
  * [8] H. Kim, M. Hasegawa-Johnson, A. Perlman, J. Gunderson, T. S. Huang, K. Watkin, and S. Frame, “Dysarthric speech database for universal access research,” in Ninth Annual Conference of the International Speech Communication Association, 2008.
  * [9] B. Vachhani, C. Bhat, and S. K. Kopparapu, “Data augmentation using healthy speech for dysarthric speech recognition.,” in Interspeech, pp. 471–475, 2018.
  * [10] E. Moulines and F. Charpentier, “Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones,” Speech communication, vol. 9, no. 5-6, pp. 453–467, 1990.
  * [11] P. Boersma, “Praat: doing phonetics by computer [computer program],” http://www. praat. org/, 2011.
  * [12] B. McFee, V. Lostanlen, A. Metsai, M. McVicar, S. Balke, C. Thomé, C. Raffel, F. Zalkow, A. Malek, Dana, K. Lee, O. Nieto, J. Mason, D. Ellis, E. Battenberg, S. Seyfarth, R. Yamamoto, K. Choi, V. et. Reevich Morozov, J. Moore, R. Bittner, S. Hidaka, Z. Wei, nullmightybofo, D. Hereñú, F.-R. Stöter, P. Friesch, A. Weiss, M. Vollrath, and T. Kim, “librosa/librosa: 0.8.0,” July 2020.
  * [13] F. Xiong, J. Barker, and H. Christensen, “Phonetic analysis of dysarthric speech tempo and applications to robust personalised dysarthric speech recognition,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5836–5840, IEEE, 2019\.
  * [14] C. Bhat, B. Vachhani, and S. Kopparapu, “Improving recognition of dysarthric speech using severity based tempo adaptation,” in Speech and Computer (A. Ronzhin, R. Potapova, and G. Németh, eds.), (Cham), pp. 370–377, Springer International Publishing, 2016.
  * [15] M. Morise, F. Yokomori, and K. Ozawa, “World: a vocoder-based high-quality speech synthesis system for real-time applications,” IEICE TRANSACTIONS on Information and Systems, vol. 99, no. 7, pp. 1877–1884, 2016.
  * [16] S. H. Yang and M. Chung, “Improving dysarthric speech intelligibility using cycle-consistent adversarial training,” arXiv preprint arXiv:2001.04260, 2020.
  * [17] M. Purohit, M. Patel, H. Malaviya, A. Patil, M. Parmar, N. Shah, S. Doshi, and H. A. Patil, “Intelligibility improvement of dysarthric speech using mmse discogan,” in 2020 International Conference on Signal Processing and Communications (SPCOM), pp. 1–5, IEEE, 2020.
  * [18] T. Kaneko and H. Kameoka, “Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks,” 2018 26th European Signal Processing Conference (EUSIPCO), pp. 2100–2104, 2018.
  * [19] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
  * [20] S. Master, N. d. Biase, V. Pedrosa, and B. M. Chiari, “The long-term average spectrum in research and in the clinical practice of speech therapists,” Pró-Fono Revista de Atualização Científica, vol. 18, no. 1, pp. 111–120, 2006.
  * [21] B. M. Halpern, R. van Son, M. van den Brekel, and O. Scharenborg, “Detecting and Analysing Spontaneous Oral Cancer Speech in the Wild,” in Proc. Interspeech 2020, pp. 4826–4830, 2020.
  * [22] L. K. Smith and A. M. Goberman, “Long-time average spectrum in individuals with parkinson disease,” NeuroRehabilitation, vol. 35, no. 1, pp. 77–88, 2014.
  * [23] J. Fritsch, G. Quer, and M. Magimai.-Doss, “Probabilistic symbol sequence matching and its application to pathological speech intelligibility assessment,” Idiap-RR Idiap-RR-01-2021, Idiap, 1 2021.
  * [24] P. Janbakhshi, I. Kodrasi, and H. Bourlard, “Pathological speech intelligibility assessment based on the short-time objective intelligibility measure,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6405–6409, IEEE, 2019\.
  * [25] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlíček, Y. Qian, P. Schwarz, J. Silovský, G. Stemmer, and K. Veselý, “The Kaldi Speech Recognition Toolkit,” technical Report, 2011.
  * [26] Anonymous, “Handling Acoustic- and Pronunciation-Level Mismatch in Dysarthric Speech Recognition Systems through Model Combination,” In submission, 2020.
  * [27] D. Povey, L. Burget, M. Agarwal, P. Akyazi, F. Kai, A. Ghoshal, O. Glembek, N. Goel, M. Karafiát, A. Rastrow, R. C. Rose, P. Schwarz, and S. Thomas, “The subspace Gaussian mixture model - A structured model for speech recognition,” Computer Speech & Language, vol. 25, pp. 404–439, 2010\.
  * [28] D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI,” in Proc. Interspeech, pp. 2751–2755, 2016.
  * [29] F. Rudzicz, “Adjusting dysarthric speech signals to be more intelligible,” Computer Speech & Language, vol. 27, no. 6, pp. 1163–1177, 2013.