# Privacy-Preserving End-to-End Spoken Language Understanding

Yinggui Wang1∗ Wei Huang1 &Le Yang21Ant Group
2University of Canterbury<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>These authors contributed equally to this work

###### Abstract

Spoken language understanding (SLU), one of the key enabling technologies for
human-computer interaction in IoT devices, provides an easy-to-use user
interface. Human speech can contain a lot of user-sensitive information, such
as gender, identity, and sensitive content. New types of security and privacy
breaches have thus emerged. Users do not want to expose their personal
sensitive information to malicious attacks by untrusted third parties. Thus,
the SLU system needs to ensure that a potential malicious attacker cannot
deduce the sensitive attributes of the users, while it should avoid greatly
compromising the SLU accuracy. To address the above challenge, this paper
proposes a novel SLU multi-task privacy-preserving model to prevent both the
speech recognition (ASR) and identity recognition (IR) attacks. The model uses
the hidden layer separation technique so that SLU information is distributed
only in a specific portion of the hidden layer, and the other two types of
information are removed to obtain a privacy-secure hidden layer. In order to
achieve good balance between efficiency and privacy, we introduce a new
mechanism of model pre-training, namely joint adversarial training, to further
enhance the user privacy. Experiments over two SLU datasets show that the
proposed method can reduce the accuracy of both the ASR and IR attacks close
to that of a random guess, while leaving the SLU performance largely
unaffected.

##  1 Introduction

Voice-controlled IoT devices and smart home assistants are gaining popularity
in people’s lives, and spoken language understanding (SLU) has become one of
the main enabling technologies to achieve human-machine interaction between
users and devices Coucke et al. (2018). For example, SLU can be found in the
in-car voice system of Tesla electricity vehicles (EVs), the voice assistant
of Apple phones and Xiaomi’s smart home ecology. Figure 1(a) shows how these
systems work in general.

There are certain privacy risks associated with these IOT devices Atlam and
Wills (2020). Due to their limited storage space, some vendors choose to
deploy only the encoder of the model at the client side and transmit the
output of the encoder to the server side for performing tasks such as
prediction and classification, which gives opportunities for malicious
attackers to take advantage of. Recent end-to-end SLU systems predict user
intent directly from speech Serdyuk et al. (2018) (see also Figure 1 (b)).
This approach circumvents the conversion of speech to text, and to a certain
extent, it avoids privacy leakage caused by text theft. However, strictly
speaking, end-to-end SLU systems are still subject to certain risks of privacy
breach. As depicted in Figure 1(c), the recognition model may be vulnerable to
being stolen by an untrusted third party and used to deduce the sensitive
information about the user. In other words, to maintain sensitive user
information, the SLU systems need to be able to infer the user’s intent from
the masked voice only, while the specific voice content, identity information
and other user-sensitive attributes should be dispensed.

With the increasing public concern about the protection of personal privacy
information and higher requirements of national standards on the protection of
biological information, significant efforts have been invested to address the
problem of user privacy leakage. The utilization of cryptography, multi-party
secure computing, differential privacy, and federation learning has been
widely investigated Xu et al. (2021). However, these methods have their own
drawbacks. For example, using cryptography or differential privacy has a great
impact on the classification accuracy of the model. Besides, the voice data
being more complex leads to less efficient computation for multi-party secure
computation, while under the federated learning, the end-side computation
capability is limited. With the increasing refinement of deep learning
methods, various new schemes are developed with adversarial training and
disentangled representation being the more popular choices.

Figure 1: (a) Speech Control System, (b) End-to-End SLU, (c) Potential
Malicious Inference Attack.

The current privacy-preserving intention understanding algorithms normally
protect the voice print information of the speaker only, leaving the content
information unprotected. From the perspective of privacy protection, this is
not satisfactory. Intent understanding is usually used in the smart home and
human-computer interaction scenarios, both of which use voice as the control
signal. In those cases, wake-up words are sometimes used as a prompt for
issuing instructions. When judging whether the speaker’s voice contains wake-
up words, the system needs to record voice in real time, and then deduces the
intent when it finds that there are wake-up words in the recorded voice.
Nevertheless, the intent of the voice may also be recognized directly without
using wake-up words. In this case, the recording will be longer than usual. If
the speaker mentions something involving privacy during this time, the content
of the voice would involve privacy information. Therefore, the privacy-
preserving intention understanding algorithm needs to protect both the voice
print and content information at the same time.

Figure 2: Framework of SLU privacy protection.

Inspired by the adversarial training Timm Koppelmann et al. (2021) and
disentangled representations Aloufi et al. (2020b), this paper proposes a
novel model integrating hidden layer separation and adversarial training.
Figure 2 illustrates the developed privacy-preserving framework for SLU.
First, we note that under the multi-task training, the information of the
three tasks in consideration (i.e., SLU, automatic speech recognition (ASR)
and identity recognition (IR)) in the encoder and hidden layers of the model
would be entangled. This results in the attackers being potentially capable of
inferring sensitive information for ASR as well as IR from the hidden layer
output. We thus propose to partition the hidden layer of the model into
several parts. The SLU task uses only one part of the hidden layer during
training, while some other parts are reserved for ASR and IR processing. We
further notice that there is some information similarity across the three
considered tasks. The developed model is then augmented with the shared part
in the hidden layer for the three tasks and the cosine similarity is utilized
to impose isolation of the individual parts for the three tasks. With the
above hidden layer separation, the information for SLU, ASR and IR are no
longer mixed together. Instead, they occupy a separated part of the hidden
layer. Specifically, only a specific part of the hidden layer contains useful
information for SLU, but that part does not contain or contains only little
information valuable for ASR and IR. This achieves the purpose of protecting
privacy.

To further enhance the privacy preservation ability of the proposed model, two
approaches are employed in the inference of the model. We 1) use robust ASR
and IR attackers to reason about sensitive information, and we 2) train ASR
and IR attackers from scratch under the premise of freezing the privatizer
(encoder) and observe whether sensitive information can be recovered.

We verify the validity of the model framework using different datasets.
Experimental results show that our framework not only guarantees the accuracy
of the SLU task, but also reduces the success rate of attackers to close to
that of a random guess. It also has better privacy protection than the popular
adversarial training methods currently available.

The main contributions of this paper are:

  * •

We demonstrate the potential vulnerability of deep learning-based SLU models,
where a malicious attacker can accurately reason about sensitive user
attributes from a model without privacy protection.

  * •

To solve the above problem, this paper proposes a privacy-preserving framework
that combines hidden layer separation with adversarial training. It partitions
the hidden layer where multiple information is mixed together, so that the SLU
task is carried only in a specific portion of the hidden layer, and redundant
sensitive information is removed from that part for the purpose of privacy
preservation.

  * •

By verifying the security of the proposed framework using both pre-trained
robust attackers and re-trained test scenarios, experiments show that our
framework balances efficiency and privacy protection well.

##  2 Related Work

###  2.1 Spoken Language Understanding (SLU)

Early SLU tasks were carried out using two models. First, the speech sentences
are input to the ASR model to obtain the text output, and then the text is
input to the natural language unit (NLU) model for intent identification.
However, this method is complex and difficult to tune. Thanks to the powerful
abstraction capabilities of deep learning models, this solution is now
gradually being replaced by the end-to-end SLU systems, which predict intent
directly from the input speech fragments without going through generating
intermediate text Serdyuk et al. (2018); Haghani et al. (2018). In terms of
model architecture, the end-to-end SLU framework has evolved from the original
one that only uses classifiers to differentiate intent to one that uses ASR
models for parameter initialization, complex models as feature extractors, and
multi-task training with ASR to improve the classification accuracy Arora et
al. (2022).

###  2.2 Voice Privacy Protection

In this subsection, we briefly survey the techniques for voice privacy
protection, which can be broadly categorized into cryptography-based methods
and machine learning-based approaches without cryptography. Since the
cryptography-based methods are not only time consuming but also complex in
principle, machine learning-based approaches are more suitable for practical
applications. With the development of deep learning, it has made a big splash
in the field of voice privacy protection.

Aloufi et al. (2020b) uses disentangled representation learning explicitly and
extracts independent factors from the raw speech data, enabling different
branches of the model to learn information for different tasks. It applies the
variational auto-encoders (VAE) to reconstruct the raw data to ensure that
sensitive user information is removed before sharing the speech data with
cloud service providers. Aloufi et al. (2020a) later combines the model
pruning, mixed-accuracy training, and quantization to further improve the
utility of the model. Stoidis and Cavallaro (2021) also uses disentangled
learning to encode information about different attributes into separate
subspaces that can be decomposed independently, and applies the vector-
quantised VAE (VQ-VAE) to reconstruct the data to protect gender as well as
identity sensitive attributes. Aloufi et al. (2021) proposes the use of speech
separation plus discretization to avoid the speaker information leakage
problem due to overlapping speech. Noé et al. (2020) verifies that the
x-vector features extracted by speaker recognition systems are vulnerable to
privacy leakage by untrusted third parties, and its then combines adversarial
training and reconstruction algorithms to remove gender information from the
original features for privacy preservation purposes. Wu et al. (2021) uses a
self-encoding structure to decouple the speech signal into speaker-related
privacy factors and speaker-independent content factors, and then applies a
generative adversarial network (GAN) model to re-generate the privacy-
preserving speech data. Tran and Soleymani (2022) shows that the top k
positions of the generated features have higher privacy risk compared to other
positions after the speech passes the privacy risk saliency estimator. Thus,
Laplace noise is added to these positions to protect privacy. Timm Koppelmann
et al. (2021) exploits a bottleneck layer to downscale the original features
and combines it with the two-step adversarial training to remove the ASR
information in the model. According to our survey, it is found that many
approaches focus on ASR as the main task and remove the identity or gender
information, which cannot fully satisfy the needs of SLU privacy protection
algorithms.

##  3 Privacy-preserving SLU

The proposed SLU privacy-preserving model with hidden layer separation and
adversarial training, and the evaluation schemes are described in detail in
this section. To reflect the hierarchy of the experiment and verify the
validity of the proposed method, we present the experiments in the following
two parts: 1) Separate out only the SLU task in the hidden layer, 2) Separate
all three tasks in the hidden layer.

###  3.1 Separate Out the SLU Task Only

In this subsection, we evaluate the performance of the privacy-preserving SLU
framework with separate hidden layer for SLU (SH-PPSLU). Here, the SLU task
uses a specific part of the hidden layer, but the ASR as well as IR tasks
still utilize the whole hidden layer. Therefore, the SLU part of the hidden
layer will still contain incomplete ASR and IR information. The
desensitization capability is enhanced by incorporating adversarial training.
The detailed structure of the model and training process are given in the next
section (Section 4).

Figure 3 plots the diagram of the proposed SH-PPSLU model. The speech signals
for SLU, ASR and IR are denoted as $x_{s}$, $x_{a}$ and $x_{i}$. The joint
CTC/attention transformer model Deng et al. (2021), commonly adopted in speech
recognition, is chosen to be the encoder and ASR decoder. They will be denoted
as E and AD for simplicity. In the forward propagation of the SLU training,
$x_{s}$ is first passed through encoder E to compute the output of the hidden
layer,$E(x_{s})$. Then, we select the first n elements of $E(x_{s})$ as the
dimensionality of the SLU feature in the hidden layer, which is represented as
${E(x_{s})}_{n}$. Finally, ${E(x_{s})}_{n}$ is fed to SLU Classify, the SLU
classification module to find the output vector (see the top of Figure 3).
Different from Lugosch et al. (2019), the SLU classifier uses a structure with
linear layers plus a pooling layer only. We employ the cross entropy $L_{slu}$
as the training loss function for the SLU task.

Figure 3: Diagram of the SH-PPSLU model.

We pick the same amount of ASR training data as in the SLU task, in order to
ensure the data balance in multi-task training. We pre-train the encoder of
SH-PPSLU, E, and ASR decoder, AD, with large-scale data, which would render
the ASR attacker powerful enough and also enhance the SLU performance. Unlike
SLU training, the vector input to ASR decoder, AD, uses all the elements from
the hidden layer. The loss function for ASR training is $L_{asr}\ =\ \alpha
L_{att}+(1-\alpha)L_{CTC}$. It is a multi-task training loss function, and
$\alpha$ is an adjustable parameter Watanabe et al. (2017).

The IR model is chosen via referring to the Deepspeaker architecture Li et al.
(2017). To fit into the framework of multi-task training, the original
Deepspeaker model based on the CNN architecture is replaced using the one with
the transformer structure based on the self-attention mechanism. The IR task,
like the ASR task, also uses all the features from the hidden layer for
training. We shall use $L_{ir}$ to denote the Triplet loss function of IR.

In this paper, we use (1) as the loss function for SH-PPSLU training. The
weights for individual task can be adjusted according to the actual situation.
It can be expected that the information of three tasks will be mixed in the
hidden layer of SH-PPSLU after the multi-task training, which can be captured
by ASR and IR attackers to deduce the sensitive user information. On the other
hand, the hidden layer part for SLU will not only contain the SLU information,
but also will include a small amount of ASR and IR information. That is, the
SLU part of the hidden layer has a better privacy protection capability,
compared with the whole hidden layer.

$\displaystyle L_{all}\ =\
{\lambda_{1}L}_{slu}+{\lambda_{2}L}_{asr}+{\lambda_{3}L}_{ir}$ (1)

Since the privacy-preserving capability of the SH-PPSLU model needs to be
further enhanced, the adversarial training is incorporated into this model,
and the privacy-preserving SLU model with hidden layer separation for SLU and
adversarial training (SHA-PPSLU) is developed. The SHA-PPSLU model is obtained
by fine-tuning the obtained SH-PPSLU model through applying the adversarial
training loss shown in (2) below and ensuring that only the parameters of the
encoder E are updated during backpropagation. This can gradually remove the
ASR and IR information from the encoder E under the premise of having a robust
attacker.

$\displaystyle
L_{adv}={{min}_{\theta_{E}}(\lambda_{1}L}_{slu}-{\lambda_{2}L}_{asr}-{\lambda_{3}L}_{ir})$
(2)

In (2) $\theta_{E}$ represents the parameter vector of the encoder E in the
SHA-PPSLU model, and $\lambda$ is an adjustable hyperparameter.

###  3.2 Complete Separation of SLU, ASR and IR

Note that SH-PPSLU only reserves part of the hidden layer for SLU but this
part still retains certain sensitive user information. This could be mitigated
through employing the adversarial training to remove significant amount of
private information. In this section of experiments, we present the privacy-
preserving SLU with hidden layer separation (H-PPSLU), which completely
separates the three tasks in the hidden layer. With this model, the part of
hidden layer for SLU can have better privacy protection capability. The
training process of H-PPSLU is as follows.

Figure 4 below shows the overall architecture of the H-PPSLU model. It is
evident that H-PPSLU differs from SH-PPSLU in Figure 3 mainly in the design of
the hidden layer. In particular, we can see that the hidden layer is
partitioned into four parts, three of which are allocated to the three tasks
in consideration. They are denoted as ${E(x_{s})}_{m}$, ${E(x_{a})}_{k}$ and
${E(x_{i})}_{l}$, where $m$,$k$,$l$ denote the size of each part. The
remaining part is shared among the three tasks, SLU, ASR and IR, and it is
denoted as ${E(X)}_{c}$. This shared part is introduced because it is noted
that there would be some correlation between the three tasks and not including
a shared part in the hidden layer would result in a significant performance
degradation.

The dimensionality of the four parts is set to 64. Of course, different setups
can be adopted. In the training process, the ASR initialization is applied
first, and then the features from the individual and shared parts are input to
the corresponding classifier and decoder for training. Specifically SLU
Classify utilizes $concat({E(x_{s})}_{m},{E(X)}_{c})$, ASR decoder AD as well
as CTC loss accepts $concat({E(x_{a})}_{k},{E(X)}_{c})$, and IR Classify
exploits $concat({E(x_{i})}_{l},{E(X)}_{c})$.

Figure 4: Diagram of the H-PPSLU model.

We need to ensure that the correlation of the three tasks can be accounted for
in the shared part of the hidden layer and the individual part contains as
little information for other tasks as possible. For this purpose, we
incorporate the cosine similarity into the loss function to ensure that the
information of SLU is restrained in the individual part of the hidden layer
for SLU only. The cosine similarity quantifies the similarity of two vectors A
and B by calculating the cosine of their angles, as shown below in (3).

The training loss function of the H-PPSLU model is shown in (4). In (5)
${sim}_{s,i}$ represents the cosine similarity of the vectors ${E(x_{s})}_{m}$
and ${E(x_{i})}_{l}$, and the rest of the notations are defined in a similar
manner. With the above training method, SLU part of the hidden layer will have
less sensitive information than that in the SH-PPSLU model, i.e., we achieve
better privacy protection.

$\displaystyle sim=\frac{A\cdot B}{\|A\|\
\|B\|}=\frac{\sum^{n}_{i=1}{A_{i}\times
B_{i}}}{{(\sum^{n}_{i=1}{(A_{i})}^{2})}^{1/2}\times{(\sum^{n}_{i=1}{(B_{i})}^{2})}^{1/2}}$
(3) $\displaystyle
L_{all}={\lambda_{1}L}_{slu}+{\lambda_{2}L}_{asr}+{\lambda_{3}L}_{ir}+{\lambda_{4}sim}_{x,y}$
(4) $\displaystyle{sim}_{x,y}=\ {sim}_{s,i}\ +\ {sim}_{s,a}\ +\ {sim}_{i,a}$
(5)

For the sake of experimental comparability and to verify whether the
adversarial training can also continue to improve the privacy expectation
ability of H-PPSLU, the privacy-preserving SLU with hidden layer separation
and adversarial training (HA-PPSLU) is proposed. Its training process is the
same as before, and the adversarial loss function is in (2).

###  3.3 Test Program Description

From the descriptions in Sections 3.1 and 3.2, it is clear that the individual
part of the hidden layer for SLU guarantees the performance of SLU while
containing little sensitive information. Two test scenarios will be used to
verify the security of the proposed model. Figure 5 shows the specific process
of the two test scenarios. In the first test scenario, to ensure a strong
attacker, the encoder E, ASR decoder AD and IR Classify are tested using the
corresponding trained models for parameter initialization. For testing, the
test set of ASR or IR is passed through the encoder to produce the output of
the hidden layer, after which the individual part for SLU in the hidden layer
(i.e., the first $n$ elements in the hidden layer) is used as the input vector
of the ASR Decoder AD or IR Classify, and finally the result is obtained.

The second test scenario differs from the first one in that 1) the attackers
of ASR as well as IR are trained from scratch after random initialization of
parameters, and the parameters of the encoder E are guaranteed not to be
updated during the training process. 2) Considering the actual scenario, we
choose another batch of ASR as well as IR datasets for this test scenario.

Figure 5: Two testing/attacking scenarios.

##  4 Experiments

In this section, we describe the experimental datasets, model framework setup,
and experimental results. First, we expose the vulnerability of the SLU model.
Then, we follow the testing scheme to evaluate whether proposed model can
achieve a balance between maintaining SLU accuracy and privacy. We not only
evaluate the privacy-preserving ability of the model in terms of whether the
success rate of the attacker can be reduced to that of a random guess, but
also compare the adversarially trained models to show that the proposed
methods can further get rid of the privacy information.

###  4.1 Datasets

The five datasets were used to simulate realistic scenarios, namely SLURP
Bastianelli et al. (2020), Fluent Speech Commands (FSC) Lugosch et al. (2019),
LibriSpeech Panayotov et al. (2015),Voxceleb1 Nagrani et al. (2017) and TED-
Lium Rousseau et al. (2012).

#### LibriSpeech.

LibriSpeech is a large dataset consisting of approximately 1000 hours of
English reading. It is derived from the reading of audiobooks from the
LibriVox project. In this paper, we use train-clean360 and train-clean100 to
pre-training and training our ASR task.

#### VoxCeleb1.

The VoxCeleb1 dataset contains more than 100,000 discourses from 1,251
celebrities, extracted from videos uploaded to YouTube. These speakers have
different accents, occupations and ages.

#### FSC.

The FSC dataset consists of a single channel 16KHZ wav audio file, which
includes approximately 19 hours of voice duration. Each audio file contains
records of English voice command words that may be used in a smart home or
virtual assistant. The dataset contains a total of 31 intents.

#### SLURP.

SLURP is a publicly available multi-domain E2E-SLU dataset. It is much larger
and more diverse than existing SLU datasets. Its collection of about 72,000
audio recordings includes more than 18 different scenes with 46 defined
actions and 55 different entity types.

#### TED-LiUM.

TED-LIUM is a training corpus for automatic speech recognition from TED talks,
which containing in total about 118 hours of talks. Not only we use this
dataset for ASR testing, also adapt the dataset for IR testing, which contains
774 speakers and 13,296 data pairs.

###  4.2 Setup and Evaluation Metrics

Our experiments were all conducted on the open source tool, i.e., Espnet
Watanabe et al. (2018). First, the 80-dimensional Fbank features are extracted
as low-dimensional feature inputs, and its frame length is 25ms with 10ms
frame shift. In order to make the input size of the model match the selected
data volume, We set the number of layers for Encoder and Decoder to be 12 and
6 respectively, and the attention dimension, number of header as well as
droupout ratio to be 256, 4 and 0.1. Then, the individual part dimension of
the SLU of the SH-PPSLU model is set to be 128, and the individual and shared
part dimensions of the three tasks of the H-PPSLU model are both 64. Our model
is initialized with a 50-epoch trained ASR model weight. During the training
process, the algorithms are trained for 15 epochs using Adam optimizer with
the learning rate 0.001, and the coefficient of the ASR loss function 0.1. For
the adversarial training we train the model for 10 epochs.

For the evaluation of SLU and IR, the classification accuracy is calculated in
this paper and denoted as ACC-SLU and ACC-IR, respectively. For the ASR
evaluation, we compute the word error rate (WER) for a given test set. The
higher the value, the better the privacy-preserving ability for spoken
contents.

###  4.3 Experimental Results of Test Scenario I

In Table 1, ML-SAI is the base model without privacy protection for the SLU,
ASR, and IR multi-task training. SH-PPSLU, SHA-PPSLU, H-PPSLU-nocos, H-PPSLU,
and HA-PPSLU are the five privacy-preserving SLU models proposed in this
paper. H-PPSLU-nocos is the H-PPSLU model without the cosine similarity
constraint during the training process. AT-SAI,AT-SI, PP-WWV, VAE are four
comparison algorithms. Note that the last three methods only protect one type
of the private information.

From the experimental results, we can see that our proposed method of using
only a portion of the features of the hidden layer to train the back part of
the model does not cause a significant degradation in the performance of the
SLU task, which is basically maintained at the same level as that of the
baseline. It shows that the SLU classifier performs well in the test datasets
only using a specific dimension feature in the hidden layer. When the model
training is combined with adversarial training, it causes a small drop (two
points) in the SLU performance, which is acceptable. We can also find that the
SLU performance degradation is smaller using the FSC dataset than that using
the SLURP dataset. Overall, our proposed model can maintain the performance of
SLU well.

|  | SLURP |  |  | FSC |
---|---|---|---|---|---|---
Method |  ACC-SLU$\uparrow$ |  WER-ASR$\uparrow$ |  ACC-IR$\downarrow$ |  ACC-SLU$\uparrow$ |  WER-ASR$\uparrow$ |  ACC-IR$\downarrow$
ML-SAI (w/o privacy protection) | 74.1 | 12.6 | 82.8 | 99.4 | 12.0 | 83.1
AT-SAI | 72.8 | 69.1 | 54.3 | 98.2 | 77.1 | 55.0
SH-PPSLU | 73.8 | 49.7 | 69.8 | 99.3 | 65.3 | 77.2
SHA-PPSLU | 72.1 | 78.6 | 53.5 | 98.0 | 88.0 | 53.0
H-PPSLU-nocos | 73.9 | 75.3 | 69.0 | 99.2 | 78.5 | 69.5
H-PPSLU | 73.4 | 87.4 | 66.2 | 99.1 | 86.8 | 67.2
HA-PPSLU | 72.2 | 89.8 | 52.2 | 97.2 | 92.1 | 52.3
AT-SI Srivastava et al. (2019) | 72.9 | - | 53.8 | 98.2 | - | 54.0
PP-WWV Timm Koppelmann et al. (2021) | 71.4 | 81.9 | - | 97.1 | 85.4 | -
VAE Stoidis and Cavallaro (2021) | 72.7 | - | 54.4 | 98.3 | - | 55.7
Table 1: Comparisons of performance among algorithms using different datasets with and without privacy protection. We use metrics (WER-ASR and ACC-IR) to measure privacy protection capabilities. For our goal in this paper, $\uparrow$ represents the higher the value, the better the performance, $\downarrow$ denotes the opposite of the former. |  | SLURP |  |  | FSC |
---|---|---|---|---|---|---
Method |  ACC-SLU$\uparrow$ |  WER-ASR$\uparrow$ |  ACC-IR$\downarrow$ |  ACC-SLU$\uparrow$ |  WER-ASR$\uparrow$ |  ACC-IR$\downarrow$
ML-SAI (w/o privacy protection) | 74.1 | 12.6 | 82.8 | 99.4 | 12.0 | 83.1
H-PPSLU($c$=88) | 73.8 | 78.5 | 67.6 | 99.2 | 80.2 | 67.3
H-PPSLU($c$=76) | 73.8 | 84.0 | 67.1 | 99.0 | 85.9 | 67.3
H-PPSLU($c$=64) | 73.4 | 87.4 | 66.2 | 99.1 | 86.8 | 67.2
H-PPSLU($c$=52) | 73.2 | 89.0 | 65.3 | 99.0 | 90.1 | 66.8
H-PPSLU($c$=40) | 72.8 | 86.1 | 63.7 | 99.1 | 86.5 | 62.6
Table 2: Comparisons of the SLU, IR and ASR accuracy of H-PPSLU for different
shared dimensions on different datasets. $c$ denotes the dimension of the
shared part.

As seen in Table 1, the SLU model is a vulnerable deep learning model with a
potential hidden layer representation that can be easily stolen by attackers
and used to reason about the users’ sensitive information. We use metrics
(WER-ASR and ACC-IR) to measure privacy protection capability. The higher the
value of WER-ASR, the better the privacy protection ability for the speech
content. The smaller the value of ACC-IR, the better the voiceprint privacy
protection ability (for the 1:1 IR verification scenario, 50% is the lowest
performance value).

From Table 1, we can see that our proposed privacy-preserving scheme has
better privacy-preserving ability than others. From experiment results, the
ASR task relies more on all features of the hidden layer, while IR can achieve
better accuracy using only a portion of the features of the hidden layer. The
results from the 6$th$ row show that training the model combining with
adversarial training further improves the privacy protection capability. From
the results of H-PPSLU-nocos and SH-PPSLU, it can be seen that the specific
parts of the hidden layer for SLU, ASR, and IR respectively help improve the
privacy protection, especially for ASR. From the results in the 7$th$-9$th$
rows, we can see that adding cosine dissimilarity helps to reduce the
attacker’s recognition accuracy. The H-PPSLU model can get rid of most ASR
information, but there is still some room for improvement in protecting the IR
information. Finally, it can be seen that the model can better reduce the
success rate of IR attackers when adversarial training is added in the
training process.

Overall our proposed privacy-preserving model can well reduce the inference
success rate of malicious attackers to that of a random guess while ensuring
that the SLU performance is not affected (we consider when WER is
approximately close to 100% and ACC-IR close to 50% the outputs of the ASR and
IR task are approximately close to random guesses). Combining with adversarial
training is effective in improving the privacy preservation ability and
achieving consistent convergence using both of the SLU datasets.

Since the shared dimensions in the previous experiments were all 64, in order
to verify the effect of different shared dimensions on the privacy-preserving
ability of the H-PPSLU model. We adopt different values of the shared
dimensions $c$ (i.e., 88, 76, 64, 52 and 40, respectively) to carry out the
corresponding experiments. For above five experiments, the individual parts of
the three tasks have the same dimensionality.

From Table 2, we can see that the smaller $c$ is, the lower the accuracy of
SLU is, but the accuracy id changed within one percentage point. With the
value change of $c$, WER-ASR and ACC-IR also have relatively small
changes.From these observations, we find that the change of the shared
dimension $c$ has little influence on the effect of privacy protection.

###  4.4 Experimental Results of Test Scenario II

| SLURP |  | FSC |
---|---|---|---|---
Method |  WER-ASR$\uparrow$ |  ACC-IR$\downarrow$ |  WER-ASR$\uparrow$ |  ACC-IR$\downarrow$
ML-SAI | 22.1 | 90.5 | 23.3 | 90.2
AT-SAI | 82.6 | 71.8 | 83.0 | 77.8
H-PPSLU | 42.5 | 75.3 | 39.2 | 77.5
HA-PPSLU | 92.1 | 60.2 | 92.3 | 68.5
Table 3: Attack experiment results under the test scenario 2.

In the actual attack scenario, the attacker can use more powerful attack
means, that is, training the attack model from scratch, as shown in Scenario 2
in Figure 5. In addition, attackers generally do not know the data set used by
our training model. For this reason, we use different training data (ASR: TED-
LIUM, IR: TED-LIUM-I) to simulate the behavior of attackers. We use the
encoder model in the first test scenario for the test of the scenario 2. The
IR classifier as well as ASR decoder are trained with random initialization.
The test results can be seen in Table 3. Under this attack, the attacker is
able to obtain relatively high IR errors, but the accuracy of ASR is still
maintained at a low level. We also find that the adversarial training further
enhances the privacy-preserving ability of the hidden layer separation scheme.
In other words, the hidden layer separation method also improves that of the
adversarial learning. Overall, our model still has the better privacy
protection capability under this attack.

##  5 Conclusion

We propose a novel hybrid privacy-preserving algorithm that uses the hidden-
layer feature separation and adversarial training framework for spoken
language understanding. First, we divide the hidden layer into several parts
according to the number of tasks, each of which occupies a shared and an
individual part, and constrain the correlation between tasks by the cosine
similarity, so that the individual information of different tasks is
distributed in different parts of the hidden layer. Then, the adversarial
training is utilized to further enhance the ability of privacy protection. It
is shown that our method not only maintains the SLU accuracy without
significantly degradation, but also reduces the success rate of attackers
close to that of a random guess.

## References

  * Aloufi et al. [2020a] Ranya Aloufi, Hamed Haddadi, and David Boyle.  Paralinguistic privacy protection at the edge.  arXiv preprint arXiv:2011.02930, 2020.
  * Aloufi et al. [2020b] Ranya Aloufi, Hamed Haddadi, and David Boyle.  Privacy-preserving voice analysis via disentangled representations.  In Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop, pages 1–14, 2020.
  * Aloufi et al. [2021] Ranya Aloufi, Hamed Haddadi, and David Boyle.  Configurable privacy-preserving automatic speech recognition.  Interspeech, 2021.
  * Arora et al. [2022] Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang, Yushi Ueda, Yifan Peng, Yuekai Zhang, Sujay Kumar, Karthik Ganesan, Brian Yan, et al.  Espnet-slu: Advancing spoken language understanding through espnet.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7167–7171. IEEE, 2022.
  * Atlam and Wills [2020] Hany F Atlam and Gary B Wills.  Iot security, privacy, safety and ethics.  Digital twin technologies and smart cities, pages 123–149, 2020\.
  * Bastianelli et al. [2020] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser.  Slurp: A spoken language understanding resource package.  arXiv preprint arXiv:2011.13205, 2020.
  * Coucke et al. [2018] Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  arXiv preprint arXiv:1805.10190, 2018.
  * Deng et al. [2021] Keqi Deng, Songjun Cao, Yike Zhang, and Long Ma.  Improving hybrid ctc/attention end-to-end speech recognition with pretrained acoustic and language models.  In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 76–82. IEEE, 2021.
  * Haghani et al. [2018] Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters.  From audio to semantics: Approaches to end-to-end spoken language understanding.  In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 720–726. IEEE, 2018.
  * Li et al. [2017] Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu.  Deep speaker: an end-to-end neural speaker embedding system.  arXiv preprint arXiv:1705.02304, 2017.
  * Lugosch et al. [2019] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio.  Speech model pre-training for end-to-end spoken language understanding.  Interspeech, 2019.
  * Nagrani et al. [2017] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  Voxceleb: a large-scale speaker identification dataset.  arXiv preprint arXiv:1706.08612, 2017.
  * Noé et al. [2020] Paul-Gauthier Noé, Mohammad Mohammadamini, Driss Matrouf, Titouan Parcollet, Andreas Nautsch, and Jean-François Bonastre.  Adversarial disentanglement of speaker representation for attribute-driven privacy preservation.  Interspeech, 2020.
  * Panayotov et al. [2015] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015.
  * Rousseau et al. [2012] Anthony Rousseau, Paul Deléglise, and Yannick Esteve.  Ted-lium: an automatic speech recognition dedicated corpus.  In LREC, pages 125–129, 2012.
  * Serdyuk et al. [2018] Dmitriy Serdyuk, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu, and Yoshua Bengio.  Towards end-to-end spoken language understanding.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5754–5758. IEEE, 2018.
  * Srivastava et al. [2019] Brij Mohan Lal Srivastava, Aurélien Bellet, Marc Tommasi, and Emmanuel Vincent.  Privacy-preserving adversarial representation learning in asr: Reality or illusion?  arXiv preprint arXiv:1911.04913, 2019.
  * Stoidis and Cavallaro [2021] Dimitrios Stoidis and Andrea Cavallaro.  Protecting gender and identity with disentangled speech representations.  Interspeech, 2021.
  * Timm Koppelmann et al. [2021] Alexandru Nelus Timm Koppelmann, Lea Schönherr, Dorothea Kolossa, and Rainer Martin.  Privacy-preserving feature extraction for cloud-based wake word verification.  Interspeech, 2021.
  * Tran and Soleymani [2022] Minh Tran and Mohammad Soleymani.  Towards privacy-preserving speech representation for client-side data sharing.  arXiv preprint arXiv:2203.14171, 2022.
  * Watanabe et al. [2017] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi.  Hybrid ctc/attention architecture for end-to-end speech recognition.  IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253, 2017.
  * Watanabe et al. [2018] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al.  Espnet: End-to-end speech processing toolkit.  arXiv preprint arXiv:1804.00015, 2018.
  * Wu et al. [2021] Peter Wu, Paul Pu Liang, Jiatong Shi, Ruslan Salakhutdinov, Shinji Watanabe, and Louis-Philippe Morency.  Understanding the tradeoffs in client-side privacy for downstream speech tasks.  In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 841–848. IEEE, 2021\.
  * Xu et al. [2021] Runhua Xu, Nathalie Baracaldo, and James Joshi.  Privacy-preserving machine learning: Methods, challenges and directions.  arXiv preprint arXiv:2108.04417, 2021.


∎

eE-mail<EMAIL_ADDRESS>e1E-mail<EMAIL_ADDRESS>e2E-mail:
<EMAIL_ADDRESS>
11institutetext: Department of Physics, Bu-Ali Sina (Avicenna) University,
Hamedan 65178, 016016, Iran

# Intermediate Class of Warm Pseudoscalar Inflation

Saeid Ebrahimi e,addr  Vahid Kamalie1,addr  Asma Alaeie2,addr

(Received: date / Accepted: date)

###### Abstract

High dissipative regime of warm pseudoscalar inflation model Kamali (2019)
with an approximately constant value of dissipation parameter $Q$ is studied.
Intermediate solution of the scale-factor related to the accelerated expansion
of the Universe which is rolled out by observational data in the context of
standard (cold) model of inflation is used. There is a region of free
parameters phase-space of the model which is interestingly compatible with
recent observational data. It is discussed that the model is also compatible
with the swampland criteria in a broad range of parameters phase-space and TCC
in a limited area of parameters.

††journal: Eur. Phys. J. C

##  1 Introduction:

The inflation model is extensively studied as a standard paradigm of early
time cosmology Starobinsky (1979, 1980, 1987); Sato (1981); Guth (1981);
Albrecht and Steinhardt (1982). Outstanding concerns of the Hot Big Bang
model, e.g. horizon, flatness, and monopole, are resolved in this context. A
brilliant achievement of this model is presented in the context of
perturbation theory. The (scalar) field theory responsible for background
accelerated expansion provides a mechanism to explain the seeds of large-scale
structure (LSS) formation in terms of quantum fluctuations of the field around
the homogeneous and isotropic background Mukhanov and Chibisov (1981); Hawking
(1982); Guth and Pi (1982); Starobinsky (1982); Bardeen  _et al._ (1983). The
standard model of inflation is proposed by a single scalar field theory with a
nearly flat potential which is followed by a reheating epoch Kofman  _et al._
(1997); Allahverdi  _et al._ (2010); Shtanov  _et al._ (1995); Kofman  _et
al._ (1994); Traschen and Brandenberger (1990). The idea of early time
accelerated expansion, in term of the effective field theories that seem to be
in the string-landscape, is doubted in the literature Ooguri and Vafa (2007);
Obied  _et al._ (2018); Brennan  _et al._ (2017); Palti (2019). It was
speculated that the effective field theory related to standard single field
inflation is not in the landscape of string theory as a preferred scenario for
quantum gravity Obied  _et al._ (2018); Agrawal  _et al._ (2018). On the other
hand, the energy scale and duration of the standard inflation are constrained
by Trans-Planckian censorship conjecture TCC Bedroya and Vafa (2020) which
made the highest limitation on the tensor-to-scalar ratio far from the
accuracy of cosmological experiments. This conjecture is a cosmological
extension of cosmic censorship conjecture by Penrose Penrose (1969);
Brandenberger (2019). There are attempts in the literature to find the ways
that probably resolve these theoretical constraints Heisenberg  _et al._
(2018); Kamali and Brandenberger (2020); Das (2019); Motaharfar  _et al._
(2019); Berera and Calderón (2019); Kamali (2020); Laliberte and Brandenberger
(2020); Dhuria and Goswami (2019); Torabian (2020); Li  _et al._ (2020); Cai
and Piao (2020); Brahma (2020); Das (2020); Kamali  _et al._ (2020a); Lin and
Kinney (2020); Kadota  _et al._ (2020); Mizuno  _et al._ (2020); Kamali  _et
al._ (2020b); Berera  _et al._ (2020a, b). Warm inflation as an alternative
for the standard (cold) inflation is introduced as a more than one field model
of inflation Berera (1995). There is an interaction between the inflaton field
as quanta of the scalar field and light (radiation) fields during the slow-
roll epoch which smoothly connects the inflation era to radiation dominated
era. It has been discussed that some scenarios of high dissipative warm
inflation are in the landscape of string theory and compatible with
observational data Motaharfar  _et al._ (2019). Intermediate model of
inflation Barrow and Saich (1990); Barrow and Liddle (1993); Barrow  _et al._
(2006) is rolled out by observational data in the context of cold inflation
but agrees with latest data in the context of warm inflation in some cases del
Campo and Herrera (2009); Setare and Kamali (2012, 2013a, 2013b, 2014); Kamali
and Setare (2016); Kamali  _et al._ (2016). In this work, we will study a
model of warm inflation which introduces the standard Chern-Simons interaction
between pseudoscalar field as inflaton and SU(2) gauge fields as radiation
part Kamali (2019), by using an intermediate solution of Universe accelerated
expansion Barrow and Saich (1990); Barrow and Liddle (1993); Barrow  _et al._
(2006). It was discussed that the dissipation parameter of warm pseudoscalar
inflation (WPSI) is approximately constant Kamali (2019). We have found a
region of free parameters phase space of the intermediate WPSI compatible with
observational data as well as quantum gravity (swampland and TCC). We can
simply find a region of parameters for the model out of swampland. There is a
limited region of the parameters compatible with TCC and observation.

##  2 Intermediate warm inflation

The intermediate form of the scale factor Barrow and Saich (1990); Barrow and
Liddle (1993); Barrow  _et al._ (2006):

$\displaystyle a(t)=a_{0}\exp(At^{f})$ (1)

was introduced as a cosmological solution of an expanding Universe where the
expansion is faster than power-low inflation ($a(t)=a_{0}t^{A}$) and slower
than standard de Sitter expansion with the scale factor $a(t)=a_{0}\exp(Ht)$
($H$ is Hubble constant). This form of the scale factor is useful in studying
the models of inflation del Campo and Herrera (2009); Setare and Kamali (2012,
2013a, 2013b, 2014); Kamali and Setare (2016); Kamali  _et al._ (2016).
Generally, to study and parameterize a model of inflation, we have to use a
special form of the potential with some constant parameters that are supposed
to be constrained by observational data. On the other hand, there is another
nearly equivalent way where the form of scale factor, with some constant
parameters, is used, and its phase space of free parameters is constrained by
observational data Barrow and Saich (1990); Barrow and Liddle (1993); Barrow
_et al._ (2006). In this note, we study our model by using the scale factor
(1) which has two free parameters $(f,A)$. In the warm scenario of inflation
where the particle production is important during the slow-roll epoch, there
are some modifications in the evolution of the inflaton, background, and
perturbation parts, in a thermal bath with temperature $T$. At the background
level we can simply split the continuity equation into two parts with a
phenomenological form of interaction:

$\displaystyle\dot{\rho}_{\phi}+3H(\rho_{\phi}+P_{\phi})=-\Upsilon\dot{\phi}^{2}$
(2)
$\displaystyle\dot{\rho}_{\gamma}+3H(\rho_{\gamma}+P_{\gamma})=\Upsilon\dot{\phi}^{2}$

The first part provides a modified form of the inflaton evolution:

$\displaystyle\ddot{\phi}+3H(1+Q)\dot{\phi}+\frac{dV}{d\phi}=0$ (3)

where $Q=\frac{\Upsilon}{3H}$ is introduced as a new dissipation coefficient
beyond the Universe expansion dissipation $H$. $Q$ is an extra parameter in
the context of warm inflation that is supposed to be constrained by
observation data as well as quantum gravity. Therefore our model has a three-
dimensional phase space of free parameters $(f,A,Q)$ which will be studied in
the next section. In a high dissipative regime, $Q\gg 1$, dissipation term has
an important role in the evolution of the inflaton in the background and
linear perturbation parts which can be compared with observational data.
Dissipation parameter $Q$ is approximately constant in the context of WPSI
Kamali (2019), where the interaction term between axion inflaton field and
SU(2) gauge-field (as a radiation part) is presented by standard Chern-Simons
term111 We note that there are developments in this context where the gauge
fields are not just SU(2) and dissipation parameter is proportional to $T^{3}$
Berghaus  _et al._ (2020); Das  _et al._ (2020); Das and Ramos (2020). We have
also checked this form of dissipation using intermediate scale factor (1) but
the spectral index is out of the preferred region of CMB observation
($n_{s}-1=-\frac{1.5f+27}{4(f(N-1)+1})$) for all possible values of parameter
$0<f<1$.  :

$\displaystyle\mathcal{L}_{int}=\frac{\phi}{8M}F^{a}_{\mu\nu}\tilde{F}_{a}^{\mu\nu}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(4) $\displaystyle
F^{a}_{\mu\nu}=\frac{1}{ig}[D^{a}_{\mu},D^{a}_{\nu}]~{}~{}~{}~{}~{}\tilde{F}_{a}^{\mu\nu}=\frac{1}{2}\epsilon^{\mu\nu\rho\sigma}F_{a\mu\nu}$
$\displaystyle
D^{a}_{\mu}=\partial_{\mu}-igA_{\mu}J^{a}~{}~{}~{}~{}~{}~{}~{}~{}Tr[J_{a},J_{b}]=\frac{1}{2}\delta_{ab}$

Using interaction term (4), we can derive the dissipation parameter
$Q=\frac{g\psi^{2}}{2MH}$ (where $\psi$ relates to the gauge fields) for warm
inflation which is approximately constant Kamali (2019). In the linear
perturbation part the modification by interacting term (4) introduces a new
version of the scalar power-spectrum:

$\displaystyle\Delta_{R}^{2}=\frac{V(1+Q)}{24\pi^{2}M_{p}^{2}\epsilon_{\phi}}(1+2n+\frac{2\sqrt{3}Q}{\sqrt{3+4\pi
Q}}\frac{T}{H})$ (5)

which will be used in our study. In warm inflation there is non-negligible
radiation energy density which is subdominant during slow-roll inflation. The
first Friedmann equation is modified as:

$\displaystyle H^{2}=\frac{1}{3M_{p}^{2}}(\rho_{\phi}+\rho_{\gamma})$ (6)

where the evolution of $\rho_{\gamma}=C_{\gamma}T^{4}$ is presented by the
second equation of (2). Near the horizon crossing point, the evolution of the
universe is inflaton dominated $\rho_{\gamma}<\rho_{\phi}$ which is the
condition that we consider to obtain the background parameters of the model at
the first step and the perturbation part in the second step. Using Eqs.(1),
(2) and Eq.(6) we present the evolution of the scalar field during the cosmic
expansion Rendall (2005); del Campo and Herrera (2009); del Campo  _et al._
(2009); Herrera (2010); Herrera and San Martin (2011); Herrera  _et al._
(2013):

$\displaystyle\dot{\phi}^{2}=-\frac{2M_{p}^{2}\dot{H}}{1+Q}~{}~{}~{}~{}~{}~{}~{}~{}\phi=\phi_{0}+\sqrt{\frac{8M_{p}^{2}A(1-f)}{fQ}}t^{\frac{f}{2}}$
(7)

where $\phi_{0}$ is the constant of integration that can be neglected without
loos the generality. Cosmic time of the model is simply presented as a
function of the scalar field (7)

$\displaystyle
t=(\frac{fQ}{8M_{p}^{2}A(1-f)})^{\frac{1}{f}}\phi^{\frac{2}{f}}$ (8)

which is an important equation in our discussion. Using Eqs.(6) and (8), we
can find the potential in term of the scalar field:

$\displaystyle
V(\phi)=9M_{p}^{2}f^{2}A^{2}[\frac{fQ}{8M_{p}^{2}A(1-f)}]^{\frac{2(f-1)}{f}}\phi^{\frac{-4(1-f)}{f}}$
(9)

where $H=fAt^{f-1}$ (1). The slow-roll parameters are defined as (Eqs.(1) and
(8) have been used):

$\displaystyle\epsilon=-\frac{\dot{H}}{H^{2}}=\frac{8M_{p}^{2}(1-f)^{2}}{f^{2}Q}\frac{1}{\phi^{2}}$
(10)
$\displaystyle\eta=-\frac{\ddot{H}}{H\dot{H}}=\frac{8M_{p}^{2}(1-f)(2-f)}{f^{2}Q}\frac{1}{\phi^{2}}$

which have to be smaller than one during the slow-roll epoch. Number of
e-folding is another important parameter that will be used in perturbation
analysis:

$\displaystyle
N=\int_{t_{1}}^{t}Hdt=\frac{fQ}{8M_{p}^{2}(1-f)}(\phi^{2}_{2}-\phi^{2}_{1})$
(11)

where $\phi_{1}<\phi_{2}$ (Eq.(1) has been used.). Now we are ready to
introduce the perturbation parameters which are used to compare the model with
the results of observation. Scalar power spectrum which can be computed by
data analysis of the CMB at pivot scale is presented for our model in high
dissipative regime :

$\displaystyle\Delta_{R}^{2}=\frac{3\sqrt{3}Q^{\frac{5}{2}}}{8\pi^{\frac{3}{2}}M_{p}^{2}}[\frac{3M_{p}^{2}}{2C_{\gamma}}]^{\frac{1}{4}}\frac{(fA)^{\frac{9}{4}}}{(1-f)^{\frac{3}{4}}}[\frac{fQ}{24A(1-f)}]^{\frac{3(3f-2)}{4f}}\phi^{\frac{3(3f-2)}{2f}}.$
(12)

We have used Eqs.(5, 9,10) and the definition of thermal bath temperature
Herrera and San Martin (2011):

$\displaystyle
T=[-\frac{M_{p}^{2}\Upsilon\dot{H}}{2C_{\gamma}HQ}]^{\frac{1}{4}}.$ (13)

Tensor perturbation power-spectrum is not modified in the context of warm
inflation model. Using Eqs.(11,12) and the standard form of tensor power
spectrum:

$\displaystyle\Delta_{T}=\frac{2H^{2}}{\pi^{2}M_{p}^{2}},$ (14)

we can find the tensor-to-scalar ratio of our model:

$\displaystyle
r=\frac{16(1-f)^{\frac{3}{4}}}{3\sqrt{3\pi}Q^{\frac{5}{2}}}(\frac{2C_{\gamma}}{3M_{p}^{2}fA})^{\frac{1}{4}}[\frac{f(N-1)+1}{fA}]^{-\frac{f+2}{4f}}.$
(15)

On the other hands, the spectral index $n_{s}$ is presented by

$\displaystyle n_{s}-1=\frac{d\ln\Delta_{R}}{d\ln
k}=-\frac{3(2-3f)}{4[f(N-1)+1]}$ (16)

which leads to Harrison-Zeldovich spectrum with $f=\frac{2}{3}$ and red tilted
cases where $f<\frac{2}{3}$. These are two perturbation parameters (15,16)
which are used to compare the inflation model with the results of observation
Aghanim  _et al._ (2020). We will compare them with the CMB data in next
section.

Figure 1: In these two pictures, we compare the theoretical results of our
model with Planck observational data.

##  3 Constraints from quantum gravity, comparison with observation

In this section, we present a region of free parameters phase space of the
model which is compatible with observational data as well as quantum gravity.
It was speculated that a scalar field theory that is used to study the early
time accelerated expansion of the Universe is in the landscape if:

$\displaystyle\frac{\Delta\phi}{M_{p}}\leq c_{1},$ (17)

and

$\displaystyle\frac{V^{\prime}}{V}>\frac{c_{2}}{M_{p}}~{}~{}~{}~{}~{}or~{}~{}~{}~{}~{}\frac{V^{\prime\prime}}{V}<-\frac{c_{3}}{M_{p}},$
(18)

otherwise, it is in the swampland of string theory. $c_{i}$ is a constant of
order one. The first inequality is distance conjecture and the second one is
de Sitter conjecture Ooguri and Vafa (2007); Obied  _et al._ (2018); Brennan
_et al._ (2017); Agrawal  _et al._ (2018); Bedroya and Vafa (2020). It was
discussed in the literature that the high dissipative regime of warm inflation
is in the landscape because of the modification of slow-roll parameters and
tensor-to-scalar ratio in the context of warm inflation Das (2019); Motaharfar
_et al._ (2019); Kamali (2020). The main point is finding a region of free
parameters phase space where the model is in landscape (generally high
dissipative regime of warm inflation models) and compatible with observation
data which is the main goal of this section. Another important conjecture is
the Trans-Planckian censorship conjecture (TCC) (Bedroya and Vafa, 2020) which
forbids the horizon crossing of the sub-Planckian perturbation modes during
the accelerated expansion. It constrains the energy scale of inflation and
tensor-to-scalar ratio Bedroya  _et al._ (2020):

$\displaystyle V^{\frac{1}{4}}<10^{10}GeV$ (19)

$\displaystyle r<10^{-30}$ (20)

We have found a limited region of the model parameters in which the model is
compatible with limitations (19,20). In Fig. (1), we have shown $1\sigma$ and
$2\sigma$ confidence level of $r-n_{s}$ parameters which were borrowed from
Planck results. There are two points in these plots that are presented by our
theoretical results for the number of e-folds $N=60$ and $N=50$ which are in
$1\sigma$ confidence level of the $r-n_{s}$ Planck observational data.
Therefore, our theoretical results agree with the observational data as well
as swampland conjecture.

From Eqs.(10, 15,16), It is obvious that the large values of the dissipation
parameter $Q$ are useful. A limited region of constant $f$
($\frac{1}{4}<f<\frac{2}{3}$) is used to satisfy the observation as well as
swampland constraints (see Fig.(1)). On the other hand in Fig.(2,) we have
used larger (limited) values of $Q$ to find the results compatible with
observation, Swampland, and TCC. It seems one needs extremely huge values of
$Q$ to find the model compatible with TCC. It means a lot of energy transfers
to the gauge sector during inflation. In some cases, we can claim the model is
compatible with TCC and observation but not a physical model. Generally, it is
hard to find a physical model compatible with TCC in the context of
intermediate warm inflation.

Figure 2: Extremely Large values of the $Q$ parameter are used to find the
model compatible with observation, swampland, and TCC.  Figure 3: In this
figure, we fixed the number of e-folding and free parameter $A=13$. The
colorful area between two red and black-dashed trajectories is compatible with
just observation. The area upper than the black-dashed line is compatible with
swampland and observation but not TCC.

In Fig.(3), we show three distinguished area of free parameters $(Q,f)$ where
$A=13$ and $N=60$. The area between the $f$ axis and red trajectory is rolled
out by observation as well as quantum gravity. The colorful area between red
and black-dashed trajectories is a free parameters phase space area that the
model is compatible with observation but in the swampland. For the colorful
area of free parameters phase space, upper than black-dashed line, our model
is compatible with observation and in the landscape. In Fig.(4) we changed the
other free parameter $A$. The model is in landscape and compatible with the
observational data for the volume between two colorful surfaces where $Q>100$
Motaharfar  _et al._ (2019).

Figure 4: In this figure, we show that the model is compatible with
observation (the volume between two colorful surfaces) where $A$ is large
enough even with the smaller values of dissipative parameter $Q$. Two free
parameters $A$ and $Q$ have the same effects on the tensor-to-scalar ratio in
this case.

In Fig.(5), the area between black-dashed trajectory and red one introduces a
free parameters phase space of the model which is in the landscape and
compatible with observation but not matched with TCC condition (20). The model
with free parameters upper than red line trajectory is in the landscape,
compatible with observation and TCC condition (20). In Fig.(6), the volume
between two colorful surfaces presents a region of the model parameters in the
landscape which is compatible with observation and TCC.

Figure 5: There are two important regions in this plot. The first is between
black-dashed and red lines where the model is in landscape and compatible with
observation but not matched with TCC bound (20). The second region is upper
than the red line which is also compatible with TCC, observation, and out of
swampland.  Figure 6: There is a volume between two colorful surfaces which
shows the model in the landscape, compatible with observation and TCC.

##  4 Conclusion

In this paper, we studied the intermediate solution of cosmic expansion in the
context of warm pseudoscalar inflation. We have shown that the model is
compatible with observational data for a region of the free parameters phase
space. We also studied the constraints of quantum gravity on our model. It was
shown that the model is compatible with swampland criteria and observation for
a broad region of free parameters phase space. Our model is also compatible
with TCC and observation for a limited region of free parameters phase space.
Our analysis can be extended for the standard models of warm inflation
Bastero-Gil  _et al._ (2018); Arya and Rangarajan (2020). The important point
is that we need a high dissipative regime of warm inflation which is
compatible with observation Motaharfar  _et al._ (2019). It would be our
future works to find this regime for the standard warm inflation models
Bastero-Gil  _et al._ (2018); Arya and Rangarajan (2020).

###### Acknowledgements.

: We would like to thank Ehsan Barati and Reza Ghasemi for very useful
comments on the draft. In memory of John D. Barrow and Shahriar Bayegan.

####  4.0.1 Shortcomings of Hot Big Bang theory

###  4.1 Standard model of cosmology

Here we describe the dynamics of an expanding universe and also investigate
the situation of the cosmos in the presence of different components of
matterWald.

## References

  * Kamali (2019) V. Kamali, Phys. Rev. D 100, 043520 (2019), arXiv:1901.01897 [gr-qc] .
  * Starobinsky (1979) A. A. Starobinsky, JETP Lett. 30, 682 (1979).
  * Starobinsky (1980) A. A. Starobinsky, Phys. Lett. B 91, 99 (1980).
  * Starobinsky (1987) A. A. Starobinsky, Adv. Ser. Astrophys. Cosmol. 3, 130 (1987).
  * Sato (1981) K. Sato, Mon. Not. Roy. Astron. Soc. 195, 467 (1981).
  * Guth (1981) A. H. Guth, Phys. Rev. D23, 347 (1981).
  * Albrecht and Steinhardt (1982) A. Albrecht and P. J. Steinhardt, Phys. Rev. Lett. 48, 1220 (1982).
  * Mukhanov and Chibisov (1981) V. F. Mukhanov and G. V. Chibisov, JETP Lett. 33, 532 (1981).
  * Hawking (1982) S. W. Hawking, Phys. Lett. B 115, 295 (1982).
  * Guth and Pi (1982) A. H. Guth and S. Y. Pi, Phys. Rev. Lett. 49, 1110 (1982).
  * Starobinsky (1982) A. A. Starobinsky, Phys. Lett. B 117, 175 (1982).
  * Bardeen  _et al._ (1983) J. M. Bardeen, P. J. Steinhardt, and M. S. Turner, Phys. Rev. D 28, 679 (1983).
  * Kofman  _et al._ (1997) L. Kofman, A. D. Linde, and A. A. Starobinsky, Phys. Rev. D 56, 3258 (1997), arXiv:hep-ph/9704452 .
  * Allahverdi  _et al._ (2010) R. Allahverdi, R. Brandenberger, F.-Y. Cyr-Racine, and A. Mazumdar, Ann. Rev. Nucl. Part. Sci. 60, 27 (2010), arXiv:1001.2600 [hep-th] .
  * Shtanov  _et al._ (1995) Y. Shtanov, J. H. Traschen, and R. H. Brandenberger, Phys. Rev. D 51, 5438 (1995), arXiv:hep-ph/9407247 .
  * Kofman  _et al._ (1994) L. Kofman, A. D. Linde, and A. A. Starobinsky, Phys. Rev. Lett. 73, 3195 (1994), arXiv:hep-th/9405187 .
  * Traschen and Brandenberger (1990) J. H. Traschen and R. H. Brandenberger, Phys. Rev. D 42, 2491 (1990).
  * Ooguri and Vafa (2007) H. Ooguri and C. Vafa, Nucl. Phys. B 766, 21 (2007), arXiv:hep-th/0605264 .
  * Obied  _et al._ (2018) G. Obied, H. Ooguri, L. Spodyneiko, and C. Vafa, (2018), arXiv:1806.08362 [hep-th] .
  * Brennan  _et al._ (2017) T. D. Brennan, F. Carta, and C. Vafa, PoS TASI2017, 015 (2017), arXiv:1711.00864 [hep-th] .
  * Palti (2019) E. Palti, Fortsch. Phys. 67, 1900037 (2019), arXiv:1903.06239 [hep-th] .
  * Agrawal  _et al._ (2018) P. Agrawal, G. Obied, P. J. Steinhardt, and C. Vafa, Phys. Lett. B 784, 271 (2018), arXiv:1806.09718 [hep-th] .
  * Bedroya and Vafa (2020) A. Bedroya and C. Vafa, JHEP 09, 123 (2020), arXiv:1909.11063 [hep-th] .
  * Penrose (1969) R. Penrose, Riv. Nuovo Cim. 1, 252 (1969).
  * Brandenberger (2019) R. Brandenberger, in  _11th International Symposium on Quantum Theory and Symmetries_ (2019) arXiv:1911.06058 [hep-th] .
  * Heisenberg  _et al._ (2018) L. Heisenberg, M. Bartelmann, R. Brandenberger, and A. Refregier, Phys. Rev. D 98, 123502 (2018), arXiv:1808.02877 [astro-ph.CO] .
  * Kamali and Brandenberger (2020) V. Kamali and R. Brandenberger, Eur. Phys. J. C 80, 339 (2020), arXiv:2001.00040 [hep-th] .
  * Das (2019) S. Das, Phys. Rev. D 99, 063514 (2019), arXiv:1810.05038 [hep-th] .
  * Motaharfar  _et al._ (2019) M. Motaharfar, V. Kamali, and R. O. Ramos, Phys. Rev. D 99, 063513 (2019), arXiv:1810.02816 [astro-ph.CO] .
  * Berera and Calderón (2019) A. Berera and J. R. Calderón, Phys. Rev. D 100, 123530 (2019), arXiv:1910.10516 [hep-ph] .
  * Kamali (2020) V. Kamali, JHEP 01, 092 (2020), arXiv:1902.00701 [gr-qc] .
  * Laliberte and Brandenberger (2020) S. Laliberte and R. Brandenberger, JCAP 07, 046 (2020), arXiv:1911.00199 [hep-th] .
  * Dhuria and Goswami (2019) M. Dhuria and G. Goswami, Phys. Rev. D 100, 123518 (2019), arXiv:1910.06233 [astro-ph.CO] .
  * Torabian (2020) M. Torabian, Fortsch. Phys. 68, 1900092 (2020), arXiv:1910.06867 [hep-th] .
  * Li  _et al._ (2020) H.-H. Li, G. Ye, Y. Cai, and Y.-S. Piao, Phys. Rev. D 101, 063527 (2020), arXiv:1911.06148 [gr-qc] .
  * Cai and Piao (2020) Y. Cai and Y.-S. Piao, Sci. China Phys. Mech. Astron. 63, 110411 (2020), arXiv:1909.12719 [gr-qc] .
  * Brahma (2020) S. Brahma, Phys. Rev. D 101, 023526 (2020), arXiv:1910.04741 [hep-th] .
  * Das (2020) S. Das, Phys. Dark Univ. 27, 100432 (2020), arXiv:1910.02147 [hep-th] .
  * Kamali  _et al._ (2020a) V. Kamali, M. Motaharfar, and R. O. Ramos, Phys. Rev. D 101, 023535 (2020a), arXiv:1910.06796 [gr-qc] .
  * Lin and Kinney (2020) W.-C. Lin and W. H. Kinney, Phys. Rev. D 101, 123534 (2020), arXiv:1911.03736 [gr-qc] .
  * Kadota  _et al._ (2020) K. Kadota, C. S. Shin, T. Terada, and G. Tumurtushaa, JCAP 01, 008 (2020), arXiv:1910.09460 [hep-th] .
  * Mizuno  _et al._ (2020) S. Mizuno, S. Mukohyama, S. Pi, and Y.-L. Zhang, Phys. Rev. D 102, 021301 (2020), arXiv:1910.02979 [astro-ph.CO] .
  * Kamali  _et al._ (2020b) V. Kamali, M. Artymowski, and M. R. Setare, JCAP 07, 002 (2020b), arXiv:1905.04814 [gr-qc] .
  * Berera  _et al._ (2020a) A. Berera, R. Brandenberger, V. Kamali, and R. Ramos, (2020a), arXiv:2006.01902 [hep-th] .
  * Berera  _et al._ (2020b) A. Berera, S. Brahma, and J. R. Calderón, JHEP 08, 071 (2020b), arXiv:2003.07184 [hep-th] .
  * Berera (1995) A. Berera, Phys. Rev. Lett. 75, 3218 (1995), arXiv:astro-ph/9509049 .
  * Barrow and Saich (1990) J. D. Barrow and P. Saich, Phys. Lett. B 249, 406 (1990).
  * Barrow and Liddle (1993) J. D. Barrow and A. R. Liddle, Phys. Rev. D 47, 5219 (1993), arXiv:astro-ph/9303011 .
  * Barrow  _et al._ (2006) J. D. Barrow, A. R. Liddle, and C. Pahud, Phys. Rev. D 74, 127305 (2006), arXiv:astro-ph/0610807 .
  * del Campo and Herrera (2009) S. del Campo and R. Herrera, JCAP 04, 005 (2009), arXiv:0903.4214 [astro-ph.CO] .
  * Setare and Kamali (2012) M. Setare and V. Kamali, JCAP 08, 034 (2012), arXiv:1210.0742 [hep-th] .
  * Setare and Kamali (2013a) M. Setare and V. Kamali, JHEP 03, 066 (2013a), arXiv:1302.0493 [hep-th] .
  * Setare and Kamali (2013b) M. Setare and V. Kamali, Phys. Lett. B 726, 56 (2013b), arXiv:1309.2452 [gr-qc] .
  * Setare and Kamali (2014) M. Setare and V. Kamali, Gen. Rel. Grav. 46, 1698 (2014), arXiv:1403.0186 [gr-qc] .
  * Kamali and Setare (2016) V. Kamali and M. R. Setare, Adv. High Energy Phys. 2016, 9682398 (2016), arXiv:1508.05479 [gr-qc] .
  * Kamali  _et al._ (2016) V. Kamali, S. Basilakos, and A. Mehrabi, Eur. Phys. J. C 76, 525 (2016), arXiv:1604.05434 [gr-qc] .
  * Berghaus  _et al._ (2020) K. V. Berghaus, P. W. Graham, and D. E. Kaplan, JCAP 03, 034 (2020), arXiv:1910.07525 [hep-ph] .
  * Das  _et al._ (2020) S. Das, G. Goswami, and C. Krishnan, Phys. Rev. D 101, 103529 (2020), arXiv:1911.00323 [hep-th] .
  * Das and Ramos (2020) S. Das and R. O. Ramos, Phys. Rev. D 102, 103522 (2020), arXiv:2007.15268 [hep-th] .
  * Rendall (2005) A. D. Rendall, Class. Quant. Grav. 22, 1655 (2005), arXiv:gr-qc/0501072 .
  * del Campo  _et al._ (2009) S. del Campo, R. Herrera, and A. Toloza, Phys. Rev. D 79, 083507 (2009), arXiv:0904.1032 [astro-ph.CO] .
  * Herrera (2010) R. Herrera, Phys. Rev. D 81, 123511 (2010), arXiv:1006.1299 [astro-ph.CO] .
  * Herrera and San Martin (2011) R. Herrera and E. San Martin, Eur. Phys. J. C 71, 1701 (2011), arXiv:1108.1371 [gr-qc] .
  * Herrera  _et al._ (2013) R. Herrera, M. Olivares, and N. Videla, Phys. Rev. D 88, 063535 (2013), arXiv:1310.0780 [gr-qc] .
  * Aghanim  _et al._ (2020) N. Aghanim _et al._ (Planck), Astron. Astrophys. 641, A5 (2020), arXiv:1907.12875 [astro-ph.CO] .
  * Bedroya  _et al._ (2020) A. Bedroya, R. Brandenberger, M. Loverde, and C. Vafa, Phys. Rev. D 101, 103502 (2020), arXiv:1909.11106 [hep-th] .
  * Bastero-Gil  _et al._ (2018) M. Bastero-Gil, S. Bhattacharya, K. Dutta, and M. R. Gangopadhyay, JCAP 02, 054 (2018), arXiv:1710.10008 [astro-ph.CO] .
  * Arya and Rangarajan (2020) R. Arya and R. Rangarajan, Int. J. Mod. Phys. D 29, 2050055 (2020), arXiv:1812.03107 [astro-ph.CO] .


# Estimation of Quantile Functionals in Linear Model
Devoted to the Memory of Pranab Kumar Sen

Jana Jurečková, Jan Picek, Jan Kalina
The Czech Academy of Sciences, Institute of Information Theory and Automation,
Technical University in Liberec,
The Czech Academy of Sciences, Institute of Computer Science, Prague The
research of Jureckova and Picek was supported by the Grant GAČR 22-03636S

###### Abstract

Various indicators and measures of the real life procedures rise up as
functionals of the quantile process of a parent random variable $Z$. However,
$Z$ can be observed only through a response in a linear model whose covariates
are not under our control and the probability distribution of error terms is
generally unknown. The problem is that of nonparametric estimation or other
inference for such functionals. We propose an estimation procedure based on
the averaged two-step regression quantile, recently developed by the authors,
combined with an R-estimator of slopes of the linear model. AMS 2000 subject
classifications. Primary 62J05, 62G32, 62G35.
Key words and phrases: Linear regression model; linear functional of quantile
function; averaged regression quantile; two-step regression quantile;
R-estimators.

##  1 Introduction

Various indicators and measures of the real life procedures rise up as
functionals of the quantile process of a parent random variable $Z$. We assume
that $Z$ has an absolutely continuous distribution function $F$ and quantile
function $Q(\alpha)=F^{-1}(\alpha);0<\alpha<1,$ but their forms are generally
unknown. The functionals of $Q$ can be environmental, industrial, health,
physical, economic and other indicators. To study and predict their behavior,
we should monitor the values of $Z$ over time or space. However, the values of
$Z$ often cannot be directly observed; they can be hidden and not directly
physically measurable, or can be autocorrelated or regressed with covariates
which are not under our control.

We shall illustate this situation on estimation of a functional of the
quantile function $Q$ of parent random variable $Z,$ which is observable only
through a response $Y$ of the linear regression model, with regressors of
unknown intensities. Hence, instead of observations
$\mathbf{Z}_{n}=(Z_{1},\ldots,Z_{n})^{\top}$, we can observe only the
variables $\mathbf{Y}_{n}=(Y_{n1},...,Y_{nn})^{\top}$, what are the
$\mathbf{Z}_{n}$ affected by covariates $\mathbf{X}_{n}$ with unknown
intensities, measured by regression coefficients $\beta_{1},\ldots,\beta_{n}.$
Summarizing, we work with the linear regression model

${\bf
Y}_{n}=\beta_{0}\mathbf{1}_{n}+{\mathbf{X}}_{n}{\mbox{\boldmath$\beta$}}+{\mathbf{Z}}_{n}$
(1.1)

with observations ${\mathbf{Y}}_{n}=(Y_{n1},\ldots,Y_{nn})^{\top},$ unknown
parameters $\beta_{0}$ (intercept),
${\mbox{\boldmath$\beta$}}=(\beta_{1},\beta_{2},\ldots,\beta_{p})^{\top}$
(scales), $\mathbf{1}_{n}=(1,\ldots,1)^{\top}\in I\kern-3.69995ptR_{n}$ and
the $n\times p$ matrix ${\mathbf{X}}={\mathbf{X}}_{n}$ of covariates. It is a
known matrix with the rows
$\mathbf{x}_{i}^{\top}=(x_{i1},\ldots,x_{ip}),\;i=1,\ldots,n.$ Because
$Z_{1},\ldots,Z_{n}$ are not available, every inference on the functionals of
$Q$ is possible only by means of observations of $Y$. The variables
$Z_{1},\ldots,Z_{n}$ are assumed to be independent and identically distributed
(i.i.d.) and

$\int zdF(z)=0,\quad\int z^{2}dF(z)<\infty,$ (1.2)

but the methods can be extended to autocovariated and sequentially dependent
observations.

We are interested in estimating the value of the functional $\mathcal{S}(Q)$
of the quantile function $Q.$ That can be a risk measure, energy consumption
over a period, water flow over a period, etc. Our interest is also in the
confidence interval for $\mathcal{S}(Q),$ or in a comparison of two
functionals, representing two possible treatments. Our main tool for model
(1.1) are the two-step averaged regression quantiles, combined with the rank-
estimators (R-estimators) of the slope components $\mbox{\boldmath$\beta$}.$
These concepts and methods have been elaborated by the authors ([8], [10],
[11], [12]) , following the pioneering ideas of Koenker and Bassett (1978).
[16], [1] and [18] use the quantile regression for estimating the conditional
value at risk.

It is shown in [11] that the averaged two-step regression $\alpha$-quantile,
introduced in [8], approximates the quantile function $Q(\alpha)$ of $Z$,
asymptotically in probability for $n\rightarrow\infty$, up-to the regression
parameters.

In the present paper, we deal with estimating the linear functional of the
quantile function of the main variable in the linear regression model, in the
situation that this variable is unobservable, affected by the covariates. Our
estimator is based on the averaged two-step regression $\alpha$-quantile,
combined with R-estimator of the scale parameters of the model. As it is shown
by authors in [11], the averaged two-step regression $\alpha$-quantile,
introduced in [8], approximates the quantile function $Q(\alpha)$ of the
errors, asymptotically in probability for $n\rightarrow\infty$, up-to the
regression parameters. Because its number of breakpoints equals exactly to
$n$, while in the case of ordinary regression quantile it is much larger, and
it is nondecreasing in $\alpha\in(0,1),$ the averaged two-step regression
quantile also facilitates joint fitting several quantile functionals in the
regression model.

##  2 Regression quantile and its two-step version

###  2.1 Regression quantile

The model (1.1) can be rewritten as the model

$Y_{ni}=\beta_{0}+{\mathbf{x}}_{ni}^{\top}{\mbox{\boldmath$\beta$}}+Z_{ni},\;i=1,\ldots,n$
(2.1)

with covariates $\mathbf{x}_{n1},\ldots,\mathbf{x}_{nn},$ each element of
$I\kern-3.69995ptR_{p}.$ For the sake of brevity, we also use the notation
$\mathbf{x}^{*}_{ni}=(1,x_{i1},\ldots,x_{ip})^{\top},\;i=1,\ldots,n.$ Let
$\widehat{\mbox{\boldmath$\beta$}}_{n}(\alpha)\in R_{p+1}$ be the
$\alpha$-regression quantile of model (1.1), $0<\alpha<1$, i.e. the solution
of the minimization

$\sum_{i=1}^{n}\rho_{\alpha}(Y_{i}-b_{0}-\mathbf{x}_{i}^{\top}\mathbf{b})=\min,\;b_{0}\in
I\kern-3.69995ptR_{1},\;\mathbf{b}\in I\kern-3.69995ptR_{p}.$ (2.2)

If derivative $f$ of $F$ exists and is positive in a neighborhood of the
quantile $Q(\alpha)=F^{-1}(\alpha),$ and if the matrix

${\mathbf{D}}_{n}=n^{-1}\left(\mathbf{1}_{n},\mathbf{X}_{n}\right)^{\top}\left(\mathbf{1}_{n},\mathbf{X}_{n}\right)$

is positively definite starting with some $n$, then
$n^{\frac{1}{2}}(\widehat{\mbox{\boldmath$\beta$}}_{n}(\alpha)-\check{\mbox{\boldmath$\beta$}}(\alpha))$
admits the asymptotic representation (see e.g. [12])

$n^{\frac{1}{2}}(\widehat{\mbox{\boldmath$\beta$}}_{n}(\alpha)-\check{\mbox{\boldmath$\beta$}}(\alpha))=n^{-\frac{1}{2}}(f(F^{-1}(\alpha))^{-1}{\mathbf{Q}}_{n}^{-1}\sum_{i=1}^{n}{\mathbf{x}}_{i}^{*}(\alpha-I[Z_{i}<F^{-1}(\alpha)])+o_{p}(1)$
(2.3)

as $n\rightarrow\infty,$ where
$\check{\mbox{\boldmath$\beta$}}(\alpha)=(F^{-1}(\alpha)+\beta_{0},\beta_{1},\ldots,\beta_{p})^{\top}$
is the population counterpart of the regression quantile. The intercept part
of the representation (2.3) is rewritten as

$\displaystyle\hat{\beta}_{n0}(\alpha)-\beta_{0}-Q(\alpha)$ $\displaystyle=$
$\displaystyle(nf(Q(\alpha)))^{-1}\sum_{i=1}^{n}(\alpha-I[Z_{i}<F^{-1}(\alpha)])+o_{p}(n^{-\frac{1}{2}})$
(2.4) $\displaystyle=$ $\displaystyle
Z_{n:[n\alpha]}-Q(\alpha)+o_{p}(n^{-\frac{1}{2}})$

as $n\rightarrow\infty,$ where the first equality follows from (2.3), while
the second equality follows from the Bahadur representation of sample
quantile. $\hat{\beta}_{n1}(\alpha),\ldots,\hat{\beta}_{np}(\alpha)$ are
consistent estimates of the slope parameters $\beta_{1}\ldots,\beta_{p}.$ The
slope components of regression quantile are asymptotically independent of the
intercept component $\hat{\beta}_{n0}(\alpha).$

The solution of (2.2) minimizes the $(\alpha,1-\alpha)$ convex combination of
residuals $(Y_{i}-\mathbf{x}_{i}^{*\top}\mathbf{b})$ over
$\mathbf{b}\in\mathbb{R}^{p+1},$ where the choice of $\alpha$ depends on the
balance between underestimating and overestimating the respective losses
$Y_{i}.$ The increasing $\alpha\nearrow 1$ reflects a greater concern about
underestimating $Y,$ comparing to overestimating. A useful functional of the
regression quantile is the averaged regression $\alpha$-quantile, the weighted
mean of components of
$\widehat{\mbox{\boldmath$\beta$}}_{n}(\alpha),\;0\leq\alpha\leq 1$:

$\bar{B}_{n}(\alpha)=\overline{\mathbf{x}}_{n}^{*\top}\widehat{\mbox{\boldmath$\beta$}}_{n}(\alpha)=\widehat{\beta}_{n0}(\alpha)+\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{p}x_{ij}\widehat{\beta}_{j}(\alpha),\quad\overline{\mathbf{x}}_{n}^{*}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{x}_{i}^{*}$
(2.5)

As shown in [8], the
$\bar{B}_{n}(\alpha)-\beta_{0}-\bar{\mathbf{x}}_{n}^{\top}\mbox{\boldmath$\beta$}$
is asymptotically equivalent to the $[n\alpha]$-quantile $e_{n:[n\alpha]}$ of
the model errors, if they are identically distributed.

###  2.2 Two-step regression quantile

The two-step regression quantile was introduced in [10] and later studied e.g.
in [11], where it is shown that it is asymptotically equivalent to the
ordinary $\alpha$-regression quantile. The two-step regression
$\alpha$-quantile combines the rank-estimator (R-estimator)
$\widetilde{\mbox{\boldmath$\beta$}}_{nR}$ of the slope components $\beta$
with the $[n\alpha]$ order statistics of the residuals
$Y_{i}-\mathbf{x}_{i}^{\top}\widetilde{\mbox{\boldmath$\beta$}}_{nR},\;i=1,\ldots,n.$

The two-step regression quantile first estimates the slope components $\beta$
by means of an R-estimate
$\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)\in{I\kern-3.69995ptR}^{p},$
defined as a minimizer of the Jaeckel’s measure of the rank dispersion [7]
with a fixed $\lambda\in(0,1)$:

$\sum_{i=1}^{n}(Y_{i}-\mathbf{x}_{i}^{\top}\mathbf{b})[a_{i}(\lambda,\mathbf{b})-\bar{a}_{n}(\lambda)]=\min$
(2.6)

with respect to
$\mathbf{b}=(b_{1},b_{2},\ldots,b_{p})^{\top}\in{I\kern-3.69995ptR}^{p}.$ The
notation in (2.6) means:

$a_{i}(\lambda,\mathbf{b})=\left\\{\begin{array}[]{lll}0&\ldots&\quad
R_{ni}(Y_{i}-\mathbf{x}_{i}^{\top}\mathbf{b})<n\lambda\\\\[2.84526pt]
R_{i}-n\lambda&\ldots&\quad n\lambda\leq
R_{ni}(Y_{i}-\mathbf{x}_{i}^{\top}\mathbf{b})<n\lambda+1\\\\[2.84526pt]
1&\ldots&\quad n\lambda+1\leq
R_{ni}(Y_{i}-\mathbf{x}_{i}^{\top}\mathbf{b}),\\\ \end{array}\right..$

Here $R_{ni}(Y_{i}-\mathbf{x}_{i}^{\top}\mathbf{b}),i=1,\ldots,n$ are the
ranks of the residuals, and $a_{i}(\lambda,\mathbf{b})$ are known as Hájek’s
rank scores (see [6]). Note that
$\bar{a}_{n}(\lambda)=\frac{1}{n}\sum_{i=1}^{n}a_{i}(\lambda,\mathbf{b})$ is
constant in $\mathbf{b}$, as an average of the rank scores. The minimization
(2.6) can be rewritten as

$\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}_{n}-(\mathbf{x}_{i}-\bar{\mathbf{x}}_{n})^{\top}\mathbf{b}\right)a_{i}(\lambda,\mathbf{b})=\min.$
(2.7)

2 It implies that the solution of (2.6) is invariant to the intercept, which
is a nuisance component. The solution of (2.6) and (2.7) is the R-estimator
$\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)$ of
$\mbox{\boldmath$\beta$}=(\beta_{1},\ldots,\beta_{p})^{\top}$, generated by
the following score function $\varphi_{\lambda}:(0,1)\mapsto
I\kern-3.69995ptR^{1}$:

$\varphi_{\lambda}(u)+(1-\lambda)=\left\\{\begin{array}[]{lll}0&\ldots&0\leq
u<\lambda\\\ 1&\ldots&\lambda\leq u\leq 1.\end{array}\right..$ (2.8)

Generally, as the score function we can use another nondecreasing square
integrable function on $(0,1)$.

By [12], $\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)$ consistently
estimates $\beta$ under the following conditions on $F$ and on
$\mathbf{X}_{n}$:

(F1)

The distribution function $F$ has a continuous density $f$ with a positive an
finite Fisher information $\mathcal{I}(f)$.

(X1)

Assume that, as $n\rightarrow\infty$,

$\displaystyle n^{-1}\mathbf{V}_{n}=O_{p}(1)\quad\mbox{ where
}\;\mathbf{V}_{n}=\sum_{i=1}^{n}(\mathbf{x}_{ni}-\bar{\mathbf{x}}_{n})(\mathbf{x}_{ni}-\bar{\mathbf{x}}_{n})^{\top},$
(2.9) $\displaystyle\max_{1\leq i\leq
n}\|\mathbf{x}_{ni}-\bar{\mathbf{x}}_{n}\|=o(n^{1/4}),\quad\bar{\mathbf{x}}_{n}=n^{-1}\sum_{i=1}^{n}\mathbf{x}_{ni}.$

Moreover, we assume that $\mathbf{V}_{n}$ satisfies

$\lim_{n\rightarrow\infty}\max_{1\leq i\leq
n}(\mathbf{x}_{ni}-\bar{\mathbf{x}}_{n})^{\top}\mathbf{V}_{n}^{-1}(\mathbf{x}_{ni}-\bar{\mathbf{x}}_{n})=0.$
(2.10)

Under conditions (F1) and (X1), the R-estimator
$\widetilde{\mbox{\boldmath$\beta$}}_{nR}=\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)$
admits the following asymptotic representation, as $n\rightarrow\infty$ (see
e.g. [12] for the proof):

$\widetilde{\mbox{\boldmath$\beta$}}_{nR}-\mbox{\boldmath$\beta$}=(f(Q(\lambda))^{-1}\mathbf{V}_{n}^{-1}\sum_{i=1}^{n}(\mathbf{x}_{ni}-\bar{\mathbf{x}}_{n})\Big{(}I[Z_{ni}>Q(\lambda)]-(1-\lambda)\Big{)}+o_{p}(n^{-1/2}),$
(2.11)

hence
$\|n^{1/2}(\widetilde{\mbox{\boldmath$\beta$}}_{nR}-\mbox{\boldmath$\beta$})\|=O_{p}(1).$

The intercept component of the two-step regression
$\alpha$-quantile is defined as the $[n\alpha]$-quantile of the residuals
$Y_{i}-\mathbf{x}_{i}^{\top}\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda),\;i=1,\ldots,n.$
Denote it as $\widetilde{\beta}_{nR,0}(\alpha),$ hence

$\widetilde{\beta}_{nR,0}(\alpha)=\Big{(}Y_{i}-\mathbf{x}_{i}^{\top}\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)\Big{)}_{n:[n\alpha]}$

and we define the two-step $\alpha$-regression quantile as the vector in
$I\kern-3.69995ptR_{p+1}$

$\widetilde{\mbox{\boldmath$\beta$}}_{n}(\alpha)=\left(\widetilde{\beta}_{nR,0}(\alpha),(\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda))^{\top}\right)^{\top}.$
(2.12)

Hence, the averaged two-step regression $\alpha$-quantile equals to

$\widetilde{B}_{n\alpha}=\widetilde{\beta}_{nR,0}(\alpha)+\bar{\mathbf{x}}^{\top}_{n}\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)=\left(Y_{i}-(\mathbf{x}_{i}-\bar{\mathbf{x}}_{n})^{\top}\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)\right)_{n:[n\alpha]}$
(2.13)

The detailed account of regression quantiles and regression rank score
processes can be found in [5]. The averaged two-step regression
$\alpha$-quantile has been introduced in [11], where it is proven that

$\widetilde{B}_{n\alpha}-\beta_{0}-\bar{\mathbf{x}}_{n}^{\top}\mbox{\boldmath$\beta$}=Z_{n:[n\alpha]}+o_{p}(n^{-1/2})=Q(\alpha)+o_{p}(n^{-1/2})\;\mbox{
as }\;n\rightarrow\infty$ (2.14)

uniformly for $\alpha\in(\varepsilon,1-\varepsilon),\;0<\varepsilon\leq 1/2,$
and for any fixed $\lambda\in{0,1}$.

##  3 Estimation of the functional $\mathcal{S}(Q)$

If there are available independent observations $Z_{1},Z_{2},\ldots,Z_{n}$
then the linear functional $\mathcal{S}(Q)$ can be estimated with the aid of
their empirical quantile function, even if distribution function of $Z$ is
unknown. The corresponding estimate would be the corresponding functional of
the empirical quantile function.

If the observations $Z_{1},Z_{2},\ldots,Z_{n},$ are not at disposal, we can
profit from approximations (2.13) and (2.14) and estimate the functional by
means of the averaged two-step regression quantile
$\widetilde{B}_{n}(\alpha)$, which explicitly contains only the available
observations $Y_{1},\ldots,Y_{n}$.

The estimate of $\mathcal{S}_{Z}{Q}$ is determined up to the nuisance
$\beta_{0}+\bar{\mathbf{x}}_{n}^{\top}\mbox{\boldmath$\beta$}.$ The nuisance
parameters can be also approximated by means of $Y_{n1},\ldots,Y_{nn}$ under
the conditions (F1) and (X1), namely by
$\bar{\mathbf{x}}_{n}^{\top}\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)$
and by $\bar{Y}_{n}$. Indeed, by (2.1) and (1.2),

$\bar{Y}_{n}=\beta_{0}+\bar{\mathbf{x}}_{n}^{\top}{\mbox{\boldmath$\beta$}}+\bar{Z}_{n}=\beta_{0}+\bar{\mathbf{x}}_{n}^{\top}{\mbox{\boldmath$\beta$}}+o_{p}(n^{-1/2})$

as $n\rightarrow\infty.$

Summarizing, our estimate of the functional $\mathcal{S}_{Z}(Q)$ in the linear
model is described as follows: We assume that the distribution of $Z$
satisfies condition (F1), that $\int zdF(z)=0$, $\;0<\int z^{2}dF(z)<\infty,$
and that the matrix $\mathbf{X}_{n}$ satisfies (X1). Then, given a fixed
$\lambda\in(0,1)$,

$\widehat{\mathcal{S}}_{Z}(Q)=\mathcal{S}(\widetilde{B}_{n})+\mathcal{S}(\bar{Y}_{n})=\mathcal{S}_{Z}(Q)+o_{p}(n^{-1/2})\;\mbox{
as }\;n\rightarrow\infty.$ (3.1)

Moreover,

$\left(Y_{i}-(\mathbf{x}_{i}-\bar{\mathbf{x}}_{n})^{\top}\widetilde{\mbox{\boldmath$\beta$}}_{nR}(\lambda)\right)_{n:[n\alpha]}-\bar{Y}_{n}=Z_{n:[n\alpha]}+o_{p}(n^{-1/2})\;\mbox{
as }\;n\rightarrow\infty.$ (3.2)

##  4 Applications

The typical quantile functionals are measures of risk in various contexts, as
in finances, in health problems, environmental and technical risks. The
popular risk measure is the Conditional Value-at-Risk (or expected shortfall )
$\sf CVaR$ equal to

$\sf{CVaR}_{\alpha}(Z)=I\kern-3.69995ptE\\{Z|Z>Q(\alpha)\\}=(1-\alpha)^{-1}\int_{\alpha}^{1}Q(u)du=(1-\alpha)^{-1}\int_{Q(\alpha)}^{\infty}zdF(z).$
(4.1)

It has obtained applications in many areas immediately after its introduction;
let us mention the management of water supplies, risk management of the social
security fund, the cash flow risk measurement for non-life insurance industry,
the financial risk in the industrial areas, operational risk in the banks, and
others. There is also a rich bibliography on the subject, both from
theoretical and applications points. By (3.1), we get its estimation in the
form

$\widehat{\sf{CVaR}}_{n\alpha}(Z)={\left\lfloor
n(1-\alpha)\right\rfloor}^{-1}\sum_{i={\left\lfloor
n\alpha\right\rfloor}}^{n}\sum_{\alpha\leq\delta<1}\widetilde{B}_{n}(\delta)-\beta_{0}-\bar{\mathbf{x}}_{n}^{\top}\mbox{\boldmath$\beta$}.$
(4.2)

The estimation of $\sf{CVaR}_{\alpha}(Z)$ under various experimental
conditions is studied in [9]. A review of quantile functionals used in
finances is given in [15]. The closely related concepts are the mean excess
function and the Lorenz curve. The mean excess function is the mean excess
over a threshold $\gamma$, that is

$e(\gamma)=I\kern-3.69995ptE(Z-\gamma|Z\geq\gamma)=I\kern-3.69995ptE(Z|Z\geq\gamma)-\gamma.$

The Lorenz curve is used in economics to describe the distribution of income.
For a nonnegative $Z$ with the mean $\mu$, it is defined as

$L(\alpha)=\frac{1}{\mu}\int_{0}^{\alpha}Q(u)du,\;\;0<\alpha<1.$

[4] considers the ratio of the Lorenz curve at $\alpha$ to its value at
$1-\alpha$ as a measure of the fraction of income that the lowest
$100\alpha\%$ of the population have relative to the upper $100\alpha\%$, i.e.
the curve

$J(\alpha)=\frac{L(\alpha)}{1-L(1-\alpha)},\;0<\alpha<1.$

The area between $J(\alpha)$ and 1 can be regarded as a measure of inequality;
its size for $\alpha=0.5$ is of interest. [17] propose the symmetric ratio of
quantiles

$R(p)\equiv\frac{Q(\alpha/2)}{Q(1-(\alpha/2))}$

as an inequality measure of income. If $F(0)<1/2$, then $R(\alpha)$ is scale
invariant and nondecreasing in $\alpha\in(0,1)$. [4] and [17] contain a rich
additional bibliography on the matter.

The nuisance covariates of the linear model can in a combination affect the
final risk or other entity of the system, though they are not of a primary
inerest. They can include the weather conditions (temperature, humidity, wind
speed, precipitation, seasonal trends), energy price fluctuations, natural
events (power outages), or industrial activity. In finances they include
market conditions (interest rates, overall market sentiment), liquidity
shocks, etc. The nuisance effects were considered in [13] in testing the
predictive models or in [2] for combining two forecasts.

## Conclusion

We propose a consistent nonparametric estimates of the quantile functionals of
variable $Z$, whose observations are not available directly, while we observe
only the responses affected by covariates with unknown intensities. The
estimates are based on the averaged two-step regression $\alpha$-quantile of
the linear model, through an R-estimator of the slope components. The
functionals of interest can be the risk measures, inequality ratios of the
income, among many others.

## References

  * [1] Bassett, G.W., Koenker, R., Kordas, W. (2004). Pessimistic portfolio allocation and Choquet Expected Utility. Journal Financial Economics 2, 477–492.
  * [2] Dimitriadis T., Liu, X., and Schnaitmann, J. (2023). Encompassing tests for value at risk and expected shortfall multistep forecasts based on inference on the boundary. Journal of Financial Econometrics 21, 412–444.
  * [3] Fu, Y., Sun, Q., and Wennersten, R. (2020). Effectiveness of the CVaR method in risk management in an integrated energy system. Energy Reports 6, 1010–1015.
  * [4] Gastwirth, J. L. (2016). Measures of Economic Inequality Focusing on the Status of the Lower and Middle Income Groups.  Statistics and Public Policy 3/1, 1–9.
  * [5] Gutenbrunner, C. and Jurečková, J. (1992). Regression rank scores and regression quantiles. Ann. Statist. 20/1, 305-330.
  * [6] Hájek, J. (1965). Extension of the Kolmogorov-Smirnov test to regression alternatives. Proc. of Bernoulli-Bayes-Laplace Seminar (L. LeCam, ed.), pp. 45–60. Univ. of California Press.
  * [7] Jaeckel, L. A. (1972). Estimating regression coefficients by minimizing the dispersion of the residuals. Ann. Math. Statist. 43, 1449–1459.
  * [8] Jurečková, J. and Picek, J. (2014). Averaged regression quantiles. In: Contemporary Developments in Statistical Theory (Lahiri, S. et al., Editors), pp. 203–216 (Chapter 12). Springer, Switzerland.
  * [9] Jurečková, J., Kalina, J. and Večeř, J. (2022). Estimation of expected shortfall under various experimental conditions. arXiv:2212.12419v1
  * [10] Jurečková, J. and Picek, J. (2005). Two-step regression quantiles. Sankhya A 67, Part 2, 227–252.
  * [11] Jurečková, J., Picek, J. and Schindler, M. (2020). Empirical regression quantile processes. Aplications of Mathematics 65, 257—269.
  * [12] Jurečková, J., Sen, P.K. and Picek, J. (2013). Methodology in Robust and Nonparametric Statistics.  CRC Press, Boca Raton, FL.
  * [13] Khalaf, L., Leccadito, A., and Urga, G. (2022). Multilevel and tail risk management. Journal of Financial Econometrics 20, 839–874.
  * [14] Koenker, R. J. and Basset, G. (1978). Regression quantiles. Econometrica 46, 33–50.
  * [15] Peracchi, F. and Tanase, A. V. (2008). On Estimating the Conditional Expected Shortfall. Research Paper Series, CEIS Tor Vergata, Vol. 6, Issue 6, No. 122
  * [16] Pflug, G. C. (2000). Some Remarks on the Value-at-Risk and the Conditional Value-at-Risk. In: Probabilistic Constrained Optimization Methodology and Applications (Uryasev, S. P., Ed.). Springer, New York.
  * [17] Prendergast, L. A. & Staudte, R. G. (2018) A Simple and Effective Inequality Measure. The American Statistician 72/4, 328–343.
  * [18] Trindade, A. A., Uryasev, S. P., Shapiro, A. and Zrazhevsky, G. (2007). Financial prediction with constrained tail risk. Journal of Banking & Finance 31, 3524–3538.