# In Search for Linear Relations in Sentence Embedding Spaces

Petra Barančíková and Ondřej Bojar
Charles University
Faculty of Mathematics and Physics
Institute of Formal and Applied Linguistics
<EMAIL_ADDRESS>

###### Abstract

We present an introductory investigation into continuous-space vector
representations of sentences. We acquire pairs of very similar sentences
differing only by a small alterations (such as change of a noun, adding an
adjective, noun or punctuation) from datasets for natural language inference
using a simple pattern method. We look into how such a small change within the
sentence text affects its representation in the continuous space and how such
alterations are reflected by some of the popular sentence embedding models. We
found that vector differences of some embeddings actually reflect small
changes within a sentence.

##  1 Introduction

Continuous-space representations of sentences, so-called sentence embeddings,
are becoming an interesting object of study, consider e.g. the BlackBox
workshop.111https://blackboxnlp.github.io/ Representing sentences in a
continuous space, i.e. commonly with a long vector of real numbers, can be
useful in multiple ways, analogous to continuous word representations (word
embeddings). Word embeddings have provably made downstream processing robust
to unimportant input variations or minor errors (sometimes incl. typos), they
have greatly boosted the performance of many tasks in low data conditions and
can form the basis of empirically-driven lexicographic explanations of word
meanings.

One notable observation was made in Mikolov et al. (2013), showing that
several interesting relations between words have their immediate geometric
counterpart in the continuous vector space.

Our aim is to examine existing continuous representations of _whole sentences_
, looking for an analogous behaviour. The idea of what we are hoping for is
illustrated in Figure space-of-sentences. As with words, we would like to
learn if and to what extent some simple geometric operations in the continuous
space correspond to simple semantic operations on the sentence strings.
Similarly to Mikolov et al. (2013), we are deliberately _not including_ this
aspect in the training objective of the sentence presentations but instead
search for properties that are learned in an unsupervised way, as a side-
effect of the original training objective, data and setup. This approach has
the potential of _explaining_ the good or bad performance of the examined
types of representations in various tasks.

The paper is structured as follows: Section related reviews the closest
related work. Section examined-sentences,examined-embeddings, respectively,
describe the dataset of sentences and the sentence embeddings methods we use.
Section operations presents the selection of operations on the sentence
vectors. Section experiments provides the main experimental results of our
work. We conclude in Section conclusion.

Figure 1: An illustration of a continuous multi-dimensional vector space
representing individual sentences, a ‘space of sentences’ (upper plot) where
each sentence is represented as a dot. Pairs of related sentences are
connected with arrows; dashing indicates various relation types. The lower
plot illustrates a possible ‘space of operations’ (here vector difference, so
all arrows are simply moved to start at a common origin). The hope is that
similar operations (e.g. all vector transformations extracted from sentence
pairs differing in the speed of travel “running instead of walking”) would be
represented close to each other in the space of operations, i.e. form a more
or less compact _cluster_. Figure 2: Example of our pattern extraction method.
In the first step, the longest common subsequence of tokens (ear is playing a
guitar .) is found and replaced with the variable X. In the second step, with
a tattoo behind is substituted with the variable Y. As the variables are not
listed alphabetically in the premise, they are switched in the last step.

step | premise | hypothesis
---|---|---
1. | a man with a tattoo behind his ear is playing a guitar . | a woman with a tattoo behind her ear is playing a guitar .
2. | a man with a tattoo behind his X | a woman with a tattoo behind her X
3. | a man Y his X | a woman Y her X
4. | a man X his Y | a woman X her Y

Figure 3: Top 10 patterns extracted from sentence pairs labelled as
entailmens, contradictions and neutrals, respectively. Note the “X $\to$ X”
pattern indicating no change in the sentence string at all.

| entailments | contradictions | neutrals
---|---|---|---
| premise | hypothesis |  | premise | hypothesis |  | premise | hypothesis |
1. | X | X | 693 | X man Y | X woman Y | 413 | X Y | X sad Y | 701
2. | X man Y | X person Y | 224 | X woman Y | X man Y | 196 | X Y | X big Y | 119
3. | X . | X | 207 | X men | X women Y | 111 | X Y | X fat Y | 69
4. | X woman Y | X person Y | 118 | X boy Y | X girl Y | 109 | X young Y | X sad Y | 68
5. | X boy Y | X person Y | 65 | X dog Y | X cat Y | 98 | X people Y | X men Y | 60
6. | X Y | Y , X . | 61 | X girl Y | X boy Y | 97 | X | sad X | 51
7. | X men Y | X people Y | 56 | X women Y | X men Y | 64 | X | X | 41
8. | two X | X | 56 | X Y, | X not Y | 56 | X person Y | X man Y | 34
9. | X girl Y | X person Y | 55 | two X, | three X | 46 | X Y | X red Y | 30
10. | X , Y | Y X . | 53 | X child Y | X man Y | 44 | X Y | X busy Y | 28

##  2 Related Work

Series of tests to measure how well their word embeddings capture semantic and
syntactic information is defined in Mikolov et al. (2013). These tests include
for example declination of adjectives
(“easy”$\rightarrow$“easier”$\rightarrow$“easiest”), changing the tense of a
verb (“walking”$\rightarrow$“walk”) or getting the capital
(“Athens”$\rightarrow$“Greece”) or currency of a state
(“Angola”$\rightarrow$“kwanza”). Bojanowski et al. (2016) and Kocmi and Bojar
(2018) have further refined the support of sub-word units, leading to
considerable improvements in representing morpho-syntactic properties of
words. Vylomova et al. (2015) largely extended the set of considered semantic
relations of words.

Sentence embeddings are most commonly evaluated extrinsically in so called
‘transfer tasks’, i.e. comparing the evaluated representations based on their
performance in sentence sentiment analysis, question type prediction, natural
language inference and other assignments. Conneau et al. (2018) introduce
‘probing tasks’ for intrinsic evaluation of sentence embeddings. They measure
to what extent linguistic features like sentence length, word order, or the
depth of the syntactic tree are available in a sentence embedding. This work
was extended to SentEval Conneau and Kiela (2018), a toolkit for evaluating
the quality of sentence embedding both intrinsically and extrinsically. It
contains 17 transfer tasks and 10 probing tasks. SentEval is applied to many
recent sentence embedding techniques showing that no method had a consistently
good performance across all tasks Perone et al. (2018).

Voleti et al. (2018) examine how errors (such as incorrect word substitution
caused by automatic speech recognition) in a sentence affect its embedding.
The embeddings of corrupted sentences are then used in textual similarity
tasks and the performance is compared with original embedding. The results
suggest that pretrained neural sentence encoders are much more robust to
introduced errors contrary to bag-of-words embeddings.

##  3 Examined Sentences

Because manual creation of sentence variations is costly, we reuse existing
data from SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018).
Both these collections consist of pairs of sentences—a premise and a
hypothesis—and their relationship (entailment/contradiction/neutral). The two
datasets together contain 982k unique sentence pairs. All sentences were
lowercased and tokenized using NLTK (Loper and Bird, 2002).

From all the available sentence pairs, we select only a subset where the
difference between the sentences in the pair can be described with a simple
pattern. Our method goes as follows: given two sentences, a premise $p$ and
the corresponding hypothesis $h$, we find the longest common substring
consisting of whole words and replace it with a variable. This is repeated
once more, so our sentence patterns can have up to two variables. In the last
step, we make sure the pattern is in a canonical form by switching the
variables to ensure they are alphabetically sorted in $p$. The process is
illustrated in Figure tab:howisitdone.

Ten most common patterns for each NLI relation are shown in Figure patterns.
Many of the obtained patterns clearly match the sentence pair label. For
instance the pattern no. 2 (“X man Y $\to$ X person Y”) can be expected to
lead to a sentence pair illustrating entailment. If a man appears in a story,
we can infer that a person appeared in the story. The contradictions
illustrate typical oppositions like man–woman, dog–cat. Neutrals are various
refinements of the content described by the sentences, probably in part due to
the original instruction in SNLI that hypothesis “might be a true” given the
premise in neutral relation.

We kept only patterns appearing with at least 20 different sentence pairs in
order to have large and variable sets of sentence pairs in subsequent
experiments. We also ignored the overall most common pattern, namely the
identity, because it actually does not alter the sentence at all. Strangely
enough, identity was observed not just among entailment pairs (693 cases), but
also in neutral (41 cases) and contradiction (22) pairs.

Altogether, we collected 4,2k unique sentence pairs in 60 patterns. Only 10%
of this data comes from MultiNLI, the majority is from SNLI.

Table 1: This table presents the quality of pattern clustering in terms of the
three cluster evaluation measures in the space of operations. For all the
scores, the value of 1 represents a perfect assignment and 0 corresponds to
random label assignment. All the numbers were computed using the Scikit-learn
library (Pedregosa et al., 2011). Best operation according to each cluster
score across the various embeddings in bold.

|  | Adjusted Rank Index | V-measure | Adjusted Mutual Information
---|---|---|---|---
embedding | dim. | - | + | * | / | - | + | * | / | - | + | * | /
InferSent_1 | 4096 | 0.58 | 0.03 | 0.03 | 0.00 | 0.91 | 0.28 | 0.24 | 0.03 | 0.87 | 0.18 | 0.14 | 0.00
ELMo | 1024 | 0.55 | 0.03 | 0.02 | 0.00 | 0.85 | 0.28 | 0.23 | 0.03 | 0.82 | 0.18 | 0.13 | 0.00
LASER | 1024 | 0.48 | 0.02 | 0.01 | 0.00 | 0.79 | 0.19 | 0.15 | 0.03 | 0.76 | 0.09 | 0.04 | 0.00
USE_T | 512 | 0.25 | 0.04 | 0.08 | 0.00 | 0.73 | 0.25 | 0.30 | 0.03 | 0.69 | 0.14 | 0.20 | 0.00
InferSent_2 | 4096 | 0.31 | 0.04 | 0.04 | 0.01 | 0.69 | 0.28 | 0.28 | 0.10 | 0.65 | 0.19 | 0.19 | 0.03
BERT | 1024 | 0.33 | 0.02 | 0.01 | 0.00 | 0.66 | 0.22 | 0.16 | 0.03 | 0.62 | 0.12 | 0.06 | 0.00
USE_D | 512 | 0.21 | 0.05 | 0.08 | 0.00 | 0.65 | 0.27 | 0.33 | 0.03 | 0.58 | 0.17 | 0.23 | 0.00
average | 1775 | 0.39 | 0.03 | 0.04 | 0.00 | 0.75 | 0.25 | 0.24 | 0.04 | 0.71 | 0.15 | 0.14 | 0.00

##  4 Sentence Embeddings

We experiment with several popular pretrained sentence embeddings.

InferSent222https://github.com/facebookresearch/InferSent (Conneau et al.,
2017) is the first embedding model that used a supervised learning to compute
sentence representations. It was trained to predict inference labels on the
SNLI dataset. The authors tested 7 different architectures and BiLSTM encoder
with max pooling achieved the best results. InferSent comes in two versions:
InferSent_1 is trained with Glove embeddings (Pennington et al., 2014) and
InferSent_2 with fastText (Bojanowski et al., 2016). InferSent representations
are by far the largest, with the dimensionality of 4096 in both versions.

Similarly to InferSent, Universal Sentence Encoder Cer et al. (2018) uses
unsupervised learning augmented with training on supervised data from SNLI.
There are two models available. USE_T333https://tfhub.dev/google/universal-
sentence-encoder-large/3 is a transformer-network (Vaswani et al., 2017)
designed for higher accuracy at the cost of larger memory use and
computational time. USE_D444https://tfhub.dev/google/universal-sentence-
encoder/2 is a deep averaging network Iyyer et al. (2015), where words and bi-
grams embeddings are averaged and used as input to a deep neural network that
computes the final sentence embeddings. This second model is faster and more
efficient but its accuracy is lower. Both models output representation with
512 dimensions.

Unlike the previous models, BERT555https://github.com/google-research/bert
(Bidirectional Encoder Representations from Transformers) Devlin et al. (2018)
is a deep unsupervised language representation, pre-trained using only
unlabeled text. It has two self-supervised training objectives - masked
language modelling and next sentence classification. It is considered
bidirectional as the Transformer encoder reads the entire sequence of words at
once. We use a pre-trained BERT-Large model with Whole Word Masking. BERT
gives embeddings for every (sub)word unit, we take as a sentence embedding a
[CLS] token, which is inserted at the beginning of every sentence. BERT
embeddings have 1,024-dimensions.

ELMo666https://github.com/HIT-SCIR/ELMoForManyLangs (Embedding from Language
Models) Che et al. (2018) uses representations from a biLSTM that is trained
with the language model objective on a large text dataset. Its embeddings are
a function of the internal layers of the bi-directional Language Model (biLM),
which should capture not only semantics and syntax, but also different
meanings a word can represent in different contexts (polysemy). Similarly to
BERT, each token representation of ELMo is a function of the entire input
sentence - one word gets different embeddings in different contexts. ELMo
computes an embedding for every token and we compute the final sentence
embedding as the average over all tokens. It has dimensionality 1024.

LASER777https://github.com/facebookresearch/LASER (Language-Agnostic SEntence
Representations) Artetxe and Schwenk (2018) is a five-layer bi-directional
LSTM (BiLSTM) network. The 1,024-dimension vectors are obtained by max-pooling
over its last states. It was trained to translate from more than 90 languages
to English or Spanish at the same time, the source language was selected
randomly in each batch.

##  5 Choosing Vector Operations

Mikolov et al. (2013) used a simple vector difference as the operation that
relates two word embeddings. For sentences embeddings, we experiment a little
and consider four simple operations: addition, subtraction, multiplication and
division, all applied elementwise. More operations could be also considered as
long as they are reversible, so that we can isolate the vector change for a
particular sentence alternation and apply it to the embedding of any other
sentence. Hopefully, we would then land in the area where the correspondingly
altered sentence is embedded.

The underlying idea of our analysis was already sketched in Figure space-of-
sentences. From every sentence pair in our dataset, we extract the pattern,
i.e. the string edit of the sentences. The arithmetic operation needed to move
from the embedding of the first sentence to the embedding of the second
sentence (in the continuous space of sentences) can be represented as a point
in what we call the _space of operations_. Considering all sentence pairs that
share the same edit pattern, we obtain many points in the space of operations.
If the space of sentences reflects the particular edit pattern in an
accessible way, all the corresponding points in the space of operations will
be close together, forming a cluster.

To select which of the arithmetic operations best suits the data, we test
pattern clustering with three common clustering performance evaluation
methods:

  * •

Adjusted Rand index Hubert and Arabie (1985) is measure of the similarity
between two cluster assignments adjusted with chance normalization. The score
ranges from −1 to +1 with 1 being the perfect match score and values around 0
meaning random label assignment. Negative numbers show worse agreement than
what is expected from a random result.

  * •

V-measure Rosenberg and Hirschberg (2007) is harmonic mean of homogeneity
(each cluster should contain only members of one class) and completeness (all
members of one class should be assigned to the same cluster). The score ranges
from 0 (the worst situation) to 1 (perfect score).

  * •

Adjusted Mutual Information Strehl and Ghosh (2002) measures the agreement of
the two clusterings with the correction of agreement by chance. The random
label assignment gets a score close to 0, while two identical clusterings get
the score of 1.

As the detailed description of these measures is out of scope of this article,
we refer readers to related literature (e.g. Vinh and Epps (2009)). We use
these scores to compare patterns with labels predicted by k-Means (best result
of 100 random initialisations). The results are presented in Table
tab:vmeasure. It is apparent that the best distribution by far is achieved
using the most intuitive operation, vector subtraction.

There seems to be a weak correlation between the size of embeddings and the
scores. The smallest embeddings USE_D and USE_T are getting the worst scores,
while the largest embeddings InferSent_1 are the best scoring embeddings.
However, InferSent_2 with dimensionality 4096 is performing poorly. The fact
that several of the embeddings were trained on SNLI does not to seem benefit
those embeddings. Between the three top scored embeddings, only InferSent_1
was trained on the data that we use for evaluation of embeddings.

##  6 Experiments

For the following exploration of the continuous space of operations, we focus
only on the ELMo embeddings. They scored second best in all scores but unlike
the best scoring Infersent_1, ELMo was not trained on SNLI, which is the major
source of our sentence pairs.

The t-SNE (van der Maaten and Hinton, 2008) visualisation of subtractions of
ELMo vectors is presented in Figure tsne. The visualisation is constructed
automatically and, of course, _without_ the knowledge of the pattern label. It
shows that the patterns are generally grouped together into compact clusters
with the exception of a ‘chaos cloud’ in the middle and several outliers. Also
there are several patterns that seem inseparable, e.g. “two X $\to$ X” and
“three X $\to$ X”, or “X white Y $\to$ X Y” and “X black Y $\to$ X Y”.

We identified the patterns responsible for the noisy center and outliers by
computing weighted inertia for each pattern (the sum of squared distances of
samples to their cluster center divided by the size of sample). The clusters
with highest inertia consists of patterns representing a change of word order
and/or adding or removing punctuation. These patterns are:

X is Y . $\to$ Y is X | X Y . $\to$ Y X . | X $\to$ X .
---|---|---
X , Y . $\to$ Y X . | X , Y . $\to$ Y , X . |
X Y . $\to$ Y , X . | X . $\to$ X |
Figure 4: t-SNE representation of patterns. The points in the operation space
are obtained by subtracting the ELMo embedding of the hypothesis from the ELMo
embedding of the premise. Best viewed in color. Colors correspond to the
sentence patterns. Figure 5: t-SNE representation of patterns as in Figure
tsne with colors coding now fully automatic clusters. Each cluster is labelled
with the set of patterns extracted from sentence pairs assigned to the
cluster. The numbers in parentheses indicate how many sentence pairs belong to
the given pattern within this cluster and overall, resp. For instance the line
“two X $\rightarrow$ X (52/56)” says that of the 56 sentence pairs differing
in the prefix “two”, 52 were automatically clustered together based on the
subtraction of their ELMo embeddings.

To see if the space of operations can be interpreted also automatically, i.e.
if the sentence relations are generalizable, we remove the noisy patterns as
above and apply fully unsupervised clustering: we do not even disclose the
expected number of patterns, i.e. clusters. We try two metrics for finding the
optimal number of clusters: Davies-Bouldin’s index (Davies and Bouldin, 1979)
and Silhouette Coefficient (Rousseeuw, 1987). They are both designed to
measure compactness and separation of the clusters, i.e. they award dense
clusters that are far from each other. Both Davies-Bouldin index and
Silhouette Coefficient agree that the best separation is achieved at 9
clusters. Running k-Means with 9 clusters, we get the result as plotted in
Figure clusters.

Manually inspecting the contents of the automatically identified clusters, we
see that many clusters are meaningful in some way. For instance, Cluster 1
captures 90% (altogether 264 out of 292) sentence pairs exerting the pattern
of generalizing women, boys or girls to people. The counterpart for men
belonging to people is spread into Cluster 5 (218 out of 227 pairs) for the
singular case and not so clean Cluster 7 containing 57/57 of the plural pairs
“X men Y $\rightarrow$ X people Y” together with various oppositions. Cluster
2 covers all sentence pairs where a person is replaced with a dog. Cluster 3
is primarily connected with sentence pairs introducing bad mood. Cluster 4
unites patterns that represent omitting a numeral/group. Cluster 6 covers
gender oppositions in one direction and Cluster 9 adds the other direction
(with some noise for child/man and person/man and similar), etc.

##  7 Conclusion and Future Work

We examined vector spaces of sentence representations as inferred
automatically by sentence embedding methods such as InferSent or ELMo. Our
goal was to find out if some simple arithmetic operations in the vector space
correspond to meaningful edit operations on the sentence strings.

Our first explorations of 60 sentence edit patterns document that this is
indeed the case. Automatically identified frequent patterns with 20 or more
occurrences in the SNLI and MultiNLI datasets correspond to simple vector
differences. The ELMo space (and others such as Infersent_1, LASER and USE-T,
which are omitted due to paper length requirements) exerts this property very
well.

Unfortunately, choosing ELMo as example might not have been the best option –
we compute ELMo embeddings by averaging contextualized word embeddings and
majority of the patterns are just removing/adding/changing a single word.
Difference between two such sentence embeddings may be a simple difference
between the embeddings of the words substituted, depending on the effect of
the contextualization. Thus, the differences in vector space would show rather
the word embeddings than the sentence embeddings.

It should be noted that our search made use of only about 0.5% of the sentence
pairs available in SNLI and MultiNLI. The remaining sentence pairs differ
beyond what was extractable automatically using our simple pattern method. A
different approach for a fine-grained description of the semantic relation
between two sentences would have to be taken for a better exploitation of the
available data.

Our plans for the long term are to further verify these observations using a
more diverse set of vector operations and a larger set of sentence
alternations, primarily by extending the set of alternation _types_. We also
plan to examine the possibilities of _generating sentence strings_ back from
the sentence embedding space. If successful, our method could lead to
controlled paraphrasing via the continuous space: take an input sentence,
embed it, modify the embedding using a vector operation and generate the
target sentence in the standard textual from.

## Acknowledgment

This work has been supported by the grant No. 18-24210S of the Czech Science
Foundation. It has been using language resources and tools stored and
distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth
and Sports of the Czech Republic (project LM2015071).

## References

  * Artetxe and Schwenk (2018) Mikel Artetxe and Holger Schwenk. 2018.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _CoRR_ , abs/1812.10464.
  * Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016.  Enriching word vectors with subword information.  _CoRR_ , abs/1607.04606.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Universal sentence encoder.  _CoRR_ , abs/1803.11175.
  * Che et al. (2018) Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018.  Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation.  _CoRR_ , abs/1807.03121.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  Senteval: An evaluation toolkit for universal sentence representations.  _arXiv preprint arXiv:1803.05449_.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  _CoRR_ , abs/1705.02364.
  * Conneau et al. (2018) Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single vector: Probing sentence embeddings for linguistic properties.  _CoRR_ , abs/1805.01070.
  * Davies and Bouldin (1979) David L. Davies and Donald W. Bouldin. 1979.  A cluster separation measure.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 1(2):224–227.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Hubert and Arabie (1985) Lawrence Hubert and Phipps Arabie. 1985.  Comparing partitions.  _Journal of Classification_ , 2(1):193–218.
  * Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015.  Deep unordered composition rivals syntactic methods for text classification.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1681–1691, Beijing, China. Association for Computational Linguistics.
  * Kocmi and Bojar (2018) Tom Kocmi and Ondrej Bojar. 2018.  Subgram: Extending skip-gram word representation with substrings.  _CoRR_ , abs/1806.06571.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. Philadelphia: Association for Computational Linguistics_.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9:2579–2605.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg S. Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Perone et al. (2018) Christian S. Perone, Roberto Silveira, and Thomas S. Paula. 2018.  Evaluation of sentence embeddings in downstream and linguistic probing tasks.  _CoRR_ , abs/1806.06259.
  * Rosenberg and Hirschberg (2007) Andrew Rosenberg and Julia Hirschberg. 2007.  V-measure: A conditional entropy-based external cluster evaluation measure.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 410–420, Prague, Czech Republic. Association for Computational Linguistics.
  * Rousseeuw (1987) Peter Rousseeuw. 1987.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _J. Comput. Appl. Math._ , 20(1):53–65.
  * Strehl and Ghosh (2002) Alexander Strehl and Joydeep Ghosh. 2002.  Cluster ensembles: A knowledge reuse framework for combining partitionings.  In _Eighteenth National Conference on Artificial Intelligence_ , pages 93–98, Menlo Park, CA, USA. American Association for Artificial Intelligence.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_.
  * Vinh and Epps (2009) N. X. Vinh and J. Epps. 2009.  A novel approach for automatic number of clusters detection in microarray data based on consensus clustering.  In _2009 Ninth IEEE International Conference on Bioinformatics and BioEngineering_ , pages 84–91.
  * Voleti et al. (2018) Rohit Voleti, Julie M. Liss, and Visar Berisha. 2018.  Investigating the effects of word substitution errors on sentence embeddings.  _CoRR_ , abs/1811.07021.
  * Vylomova et al. (2015) Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2015.  Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning.  _CoRR_ , abs/1509.01692.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _NAACL-HLT_.


# Fairness-Aware Optimization of Vehicle-to-Vehicle Interaction for Smart EV
Charging Coordination ††thanks: †These authors contributed equally. ††thanks:
*Corresponding author: Hao Wang. ††thanks: This work has been supported in
part by the Australian Research Council (ARC) Discovery Early Career
Researcher Award (DECRA) under Grant DE230100046.

Aditya Khele† Faculty of Information Technology
Monash University
Melbourne, Australia
<EMAIL_ADDRESS>Canchen Jiang† Department of Data Science and AI
Monash University
Melbourne, Australia
<EMAIL_ADDRESS>Hao Wang∗ Department of Data Science and AI
and Monash Energy Institute
Monash University
Melbourne, Australia
<EMAIL_ADDRESS>
###### Abstract

As the number of electric vehicles (EVs) continues to grow, there is an
increasing need for smart charging strategies. This paper exploits the
vehicle-to-vehicle (V2V) concept to leverage EVs’ diverse charging patterns
and unlock the value of flexibility by enabling energy transfer among EVs. We
formulate a cost minimization problem for an EV charging station to optimize
the V2V schedule together with vehicle-to-grid (V2G), grid-to-vehicle (G2V)
charging, as well as the use of renewable energy. When EVs perform V2V to
transfer energy to charge EVs, the traditional cost-minimizing approach may
overuse some EVs with lower costs to perform V2V. We address the fairness
issue by developing fair V2V energy transfer strategies to avoid the excess
discharge from individual EVs. We introduced three kinds of fairness metrics
in the V2V optimization problem to demonstrate the fair energy transfers. In
addition, we formulate baseline optimization problems without V2G or V2V and
compare the results to prove the potency of V2V concept along with V2G method.
The simulation results demonstrate that V2V can significantly reduce EV
charging costs and highlight the trade-off between fairness enforcement and
charging cost minimization.

###### Index Terms:

Electric Vehicle, Energy Transfer, EV Charging, Fairness, Vehicle-to-Grid
(V2G), Vehicle-to-Vehicle (V2V)

##  I Introduction

The electric vehicle (EV) market is rapidly growing due to its potential as a
promising technical alternative to fossil-fuel-powered automobiles in the
fight against climate change [1]. The number of EVs on the road has increased
by 60% since 2018, and the reduced cost of high-performance battery technology
further boosts their development, leading to a considerable impact on the
electric grid [2]. Uncoordinated charging of a large number of EVs could
stress the grid, leading to voltage issues and congestion. Therefore, the
development of smart charging and vehicle-to-grid (V2G) technologies are
essential for the effective energy management of EVs, enabling the seamless
integration of large numbers of EVs without compromising grid stability and
reliability.

Coordinated EV charging is a fundamental aspect of ensuring a sustainable and
reliable power grid that can accommodate the growth of EVs and enable the
transition to a cleaner energy future. Recently, various EV charging
approaches using coordination, pricing, and incentives have been proposed,
e.g., in [3, 4, 5]. Organized discharging technologies, such as V2G, allow
bidirectional energy exchange between electric vehicles and the power grid. It
can aid the power grid in reducing the peak load [6, 7]. In V2G, a significant
number of EVs are utilized as both load and energy storage to support the grid
by storing energy during off-peak hours and selling it back to the grid during
peak hours [8]. However, the above studies only explored the interaction
between EVs and the grid but neglected the potential value brought by
interactions among EVs. _This research aims to unlock the value of EV
flexibility through vehicle-to-vehicle (V2V) energy transfer while taking EV
fair contribution into account._

###  I-A Existing Studies and Gap

Several studies have focused on EV energy management which can help reduce the
cost of charging EVs and reduce the energy load for the grid at peak hours. A
large-scale EV integration, also known as grid-to-vehicle (G2V), has been
studied to determine its effects on the power system. The study in [9]
demonstrated that controlled EV charging could lower the peak demand by
distributing the majority of the peak load. Moreover, [10] showed the
potential profitability from lowering energy peak load and customer costs
together. A priority-based V2G peak load optimization approach explained in
[11], using grid-connected EVs, reduces the disparity between peak demand and
off-peak load. As charging stations heavily rely on the grid power to charge
EVs, it is more environmentally friendly to integrate renewable energy sources
with V2G energy transfer [12]. In addition to drawing local renewable energy
to charge EVs, vehicle-to-vehicle (V2V), as an emerging technology,
demonstrates that EVs may transfer their energy via bidirectional chargers in
a charging station. V2V enables flexible power transfer between EVs, making EV
charging coordination smarter and more efficient [6].

Vehicle-to-vehicle charging is a technology that allows electric vehicles to
use their batteries to exchange energy with other vehicles. It is a relatively
new technology that emerged because EV charging patterns are diverse, and some
EVs connected to charging stations can share the energy to help each other and
reduce the peak load of the charging station and the grid. In the last few
years, a few aspects of V2V operations have been studied. A fuzzy energy
management algorithm was developed in [13] to reduce the overall daily cost of
charging the EVs and mitigate the impact of EV charging on the main grid. A
study in [14] proposed a matching-based method to assign a charging station or
an EV as a supplier to meet the charging demand of EVs, supported by V2V. The
study in [15] proposed a V2V charging strategy, which incorporates parking
duration to facilitate V2V charging under time restraints. This research
suggested that V2V can offer the charge services to grid-connected electric
vehicles to offload the power system’s power demands when needed. However,
discharging of the electric vehicles should be restricted and allow all EVs to
benefit from respective discharging.

EVs could perform a variety of tasks, such as V2G and V2V, all of which
generate value streams. Similar to photovoltaic (PV) energy exporting, EVs can
discharge their batteries to become energy suppliers. Energy customers used to
be solely consumers, but they have evolved into prosumers with the capacity to
generate surplus revenue. This changes the energy landscape and fair
allocation of benefits has emerged as a problem. If it is not addressed, it
may constitute a barrier to the adoption of EVs and PV systems. For example,
in distribution networks with PV generation, unfair distribution of energy is
a significant issue. Studies have addressed this unfair energy supply issue
for PV curtailment. The study in [16] presented a fairness-based energy
coordination strategy enabling customers to share the responsibility of
voltage regulation in distribution systems. Fair PV curtailment was
implemented in [17] utilizing various notions of fairness, resulting in varied
curtailment patterns. Similarly, EV discharging needs to be constrained to
benefit all participating EVs and prevent the over-discharge of one or a few
EVs, thus providing a fair chance to all EVs. Existing solutions to fair PV
curtailment could not be directly applied in the relation to V2V fairness. As
in the case of fairness in PV curtailment, energy flow is unidirectional
whereas fairness in V2V technology energy flow is bidirectional. In this
paper, along with different energy-exchanging strategies, we focus on reducing
the biased discharging of electric vehicles with the flexibility to charge and
discharge the vehicles over a period.

###  I-B Main Work and Contributions

In this paper, we study the fairness-aware optimization of V2V to facilitate
more flexible EV charging to reduce energy costs while explicitly ensuring
fair discharge of EVs. The EV charging is co-optimized with V2V, V2G, and
local PV energy supply. The V2V technology leverages the EV interactions and
benefits the system by transferring energy to EVs in need, thus reducing the
overall energy purchase from the grid. To avoid excess and biased discharging
from a particular group of EVs, we introduce the fairness concept to enforce
fair discharging of EVs participating in V2G and V2V activities. The key
contributions of this paper are summarized as follows.

  * •

Optimizing V2V for Smart Charging: We study smart charging techniques using
V2V to leverage the EV flexibility and benefit the EV users and the grid. V2V
leverages the diverse EV charging patterns and bidirectional chargers to fully
unlock the value of EV flexibility.

  * •

Fair Discharging and Impact: We design three different fairness constraints to
avoid excess discharging of individual EVs and maintain balanced energy
transfer contributions. The designed fairness constraints enable an explicit
analysis of the tradeoff between EV charging cost minimization and fairness
enforcement.

  * •

Value of V2V in Residential and Shopping Center EV Charging Scenarios: We
demonstrate the value of V2V in a smart charging environment to balance the
load and reduce the charging cost of EVs parked in residential and shopping
center EV charging stations.

The remainder of this paper is organized as follows. Section II presents the
system model. Section III presents three fairness constraints for EV
discharging. Section IV presents the joint optimization problem for EV
charging, V2G, and V2V. It also includes baseline optimization problems for
comparison. Section V evaluates our proposed fairness-aware V2V optimization.
Section VI concludes our work.

##  II System Model

###  II-A System Setup

We consider a system with a large parking space, such as a residential premise
and shopping center, having multiple EV chargers. As shown in Fig. 1, the
green dot-dashed lines represent the energy transfer between EVs performing
V2V, and the orange dotted lines signify V2G. The dashed blue lines denote the
EV charging. A controller is responsible for overall energy transactions with
the grid and coordinating EV charging and discharging operations in the
charging site. This problem aims to minimize the total cost for EVs and enable
fair participation of EVs in energy transfer. This section presents the system
model, including EV, energy supply, V2G, and V2V.

Figure 1: System Diagram of EV charging with V2G and V2V.

###  II-B System Model

We denote a set of EVs as $\mathcal{N}=\\{1,...,N\\}$, where $N$ is the total
number of EVs. We consider the operational horizon of a day, which is evenly
divided into $24$ time slots, such that each time slot denotes one hour. We
denote the operational horizon as $\mathcal{H}=\\{1,...,H\\}$, where $H=24$ is
the total number of time slots.

####  II-B1 EV battery model

For EV $i\in\mathcal{N}$, the parking period is denoted as
$[t^{i}_{a},t^{i}_{d}]$ $\subseteq$ $H$. At the arrival time $t^{i}_{a}$, EV
$i$ is connected to the charger. Similarly, $t^{i}_{d}$ represents the
departure time of EV $i$, when EV $i$ is plugged out. We denote
$P_{avail}^{i,t}$ as the available energy stored in EV $i$ at time $t$,
$P^{i}_{ch,t}$ as the charge energy, and $P^{i}_{dis,t}$ as the discharge
energy of EV $i$ in time slot $t$. We let $\lambda^{i}_{c}$ $\in$ [0,1] and
$\lambda^{i}_{d}$ $\in$ (0,1] denote charging and discharging efficiency of EV
$i$, respectively. The battery dynamics of EV $i$ in time slot $t$ is
presented as

$p_{avail}^{i,t}=p_{avail}^{i,t-1}+\lambda_{c}p_{ch}^{i,t}-\frac{1}{\lambda_{d}}p_{dis}^{i,t}.$
(1)

Since EVs cannot charge and discharge at the same time, we restrict the EV
operations by introducing a binary variable denoted as
$X_{t}^{i}\in\\{0,1\\}$, in which $X_{t}^{i}=0$ represents charging, and
$X_{t}^{i}=1$ represents discharging. The constraint for EV charging at time
$t\in[t^{i}_{a},t^{i}_{d}]$ is

$0\leq p_{ch}^{i,t}\leq\overline{P}_{ch}^{i,t}(1-X_{t}^{i}).$ (2)

Similarly, the constraint for EV discharging at time
$t\in[t^{i}_{a},t^{i}_{d}]$ is

$0\leq p_{dis}^{i,t}\leq\overline{P}_{dis}^{i,t}X_{t}^{i},$ (3)

where $\overline{P}_{ch}^{i,t}$ and $\overline{P}_{dis}^{i,t}$ denote the
upper bound for $P_{ch}^{i,t}$ and $P_{dis}^{i,t}$ in all time slots
$t\in[t^{i}_{a},t^{i}_{d}]$, respectively. When the value of $X_{t}^{i}$ is
$1$, the upper bound for $p_{ch,t}^{i}$ becomes $0$, allowing EV $i$ to
discharge. When the value of $X_{t}^{i}$ is $0$, the upper bound for
$p_{dis}^{i,t}$ becomes 0, allowing EV $i$ to charge. When EVs are not present
at the parking slot, the values of $p_{ch}^{i,t}$ and $p_{dis}^{i,t}$ are $0$.

The stored energy in EV $i$ is bounded by

$\underline{P}_{avail}^{i,t}\leq
p_{avail}^{i,t}\leq\overline{P}_{avail}^{i,t},$ (4)

where $\underline{P}_{avail}^{i,t}$ and $\overline{P}_{avail}^{i,t}$ are the
lower and upper bounds for $p_{avail}^{i,t}$ in all time slots
$t\in[t^{i}_{a},t^{i}_{d}]$.

We assume that each EV provides information at arrival time about the amount
of required energy to be charged by the departure time. We denote the required
charge by $P_{target}^{i,t_{d}^{i}}$ for EV $i$. The constraint for required
energy at departure time is

$p_{avail}^{i,t_{d}^{i}}=P_{target}^{i,t_{d}^{i}}.$ (5)

####  II-B2 Energy supply model

The energy supply can come from the grid $P_{grid}^{i,t}$ and the local
renewable generation $P_{Renew}^{i,t}$, e.g., photovoltaic (PV) energy, for EV
$i$ in time slot $t$. The constraints for energy supply are

$\displaystyle 0\leq\sum_{i\in\mathcal{N}}p_{grid}^{i,t}\leq G,$ (6)
$\displaystyle 0\leq\sum_{i\in\mathcal{N}}p_{Renew}^{i,t}\leq R_{Renew}^{t},$
(7)

where $G$ is the upper bound for grid power, and $R_{Renew}^{t}$ is the upper
bound for renewable supply per each EV at time $t$.

####  II-B3 V2V and V2G energy transactions

Energy transactions consist of V2G and V2V. For V2G, we denote $p_{V2G}^{i,t}$
as the amount of energy that EV $i$ performs V2G to sell to the grid. Through
V2G, EV batteries deliver services to the electrical grid, assisting in the
maintenance of high-quality and reliable electricity supply [18]. The
constraint for $p_{V2G}^{i,t}$ is

$0\leq p_{V2G}^{i,t}\leq\overline{P}_{V2G}^{i,t},$ (8)

where $\overline{P}_{V2G}^{i,t}$ is the upper bound for the amount of V2G for
EV $i$ at time $t$.

Another energy transaction performed among EVs is V2V, and we denote
$p_{V2V}^{i,j,t}$ as the energy transfer between EVs $i,j\in\mathcal{N}$ at
time $t$. The constraint for V2V is

$0\leq p_{V2V}^{i,j,t}\leq\overline{P}_{V2V}^{i,j,t},$ (9)

where $\overline{P}_{V2V}^{i,j,t}$ is the upper bound for V2V transfer at time
$t$.

Note that, when EV $i$ transfers its energy to EV $j\in\mathcal{N}\backslash
i$, we assume that the energy loss among EVs in the charging site is
negligible, and thus EV $j$ receives the same amount of energy, which can be
represented as

$p_{V2V}^{i,j,t}=-p_{V2V}^{j,i,t}.$ (10)

Here, $p_{V2V}^{i,j,t}$ denotes the charging of EV $i$ from EV $j$ and
$-p_{V2V}^{j,i,t}$ denotes discharging of EV $j$ to charge EV $i$ at time $t$.

The energy coming into the system and going out of the system should be
balanced out. The total energy charge consists of energy supply from the grid
and renewable and the energy received from V2V

$p_{ch}^{i,t}=p_{grid}^{i,t}+p_{Renew}^{i,t}+p_{V2V}^{j,i,t}.$ (11)

We consider the EV-optimized scheduling charging and discharging depending
upon the different values at different times. The discharging of the EV
satisfies the following constraint

$p_{dis}^{i,t}=p_{V2G}^{i,t}+p_{V2V}^{i,j,t}.$ (12)

##  III Fairness Metrics and Constraints

Fair discharging for V2G or V2V is necessary for participated EVs. For
example, the V2V technology enables energy transfer among connected EVs,
intending to offload significant charging load on the grid [19]. Similar to
demand response and PV curtailment, V2V interaction can lead to biased
discharging from specific EV groups and unfair contributions from EVs. Some
EVs excessively contribute to the V2V energy transfer, causing a burden, but
some do not get an equal opportunity to participate.

Our work focuses on the fair discharging of EVs participating in V2G and V2V
activities. We consider fairness constraints to prevent the over-discharging
of parked EVs and provide an opportunity for all EVs to benefit from V2G and
V2V technologies. In this paper, we consider and discuss two methods for
introducing fairness. In the first approach, we define a strictly fixed limit
for each EV participating in discharge activity. In this approach of fairness
constraints with fixed limits, we further define two fairness constraints with
fixed hard and soft limits separately. A more flexible EV discharging fairness
solution is discussed in the second approach. The detailed explanation of
these fairness constraints is as follows.

###  III-A Definition 1: Fairness constraints with fixed hard limit

Firstly, we consider stricter constraints where the amount of discharge is
restricted to the given limit for all EV $i$ $\in$ $\mathcal{N}$ over time
$t$. We define the upper boundary for each discharging EV $i$ for time $t$ by
$\overline{\mathcal{Z}}$. The upper limits will strictly restrict the excess
discharging of the EVs and hence, allowing EVs to discharge fairly. We define
this by

$p_{dis}^{i,t}\leq\overline{\mathcal{Z}},\>\>\forall\>t,$ (13)

This constraint ensures fairness in discharging EVs while providing a fair
chance to every EV at $t$ parked in the parking space to perform V2V and/or
V2G.

###  III-B Definition 2: Fairness constraints with fixed soft limit

The second considered fairness constraint with a fixed limit also restricts
the discharge of EVs but the applied limit is for cumulative discharge by each
EV over the horizon $\mathcal{H}$. Fairness constraints with a soft limit
allow EVs to discharge freely till the total cumulative amount of discharge
reach or is less than the defined limit. To define this, we consider
$\overline{\mathcal{Z}^{c}}$ to represent the threshold for the total amount
of discharged energy for each EV participating in the discharging activity
over the time period $\mathcal{H}$. This can be represented by

$\sum_{t\in\mathcal{H}}p_{dis}^{i,t}\leq\overline{\mathcal{Z}^{c}}.$ (14)

Note that the thresholds $\mathcal{Z}$ and $\overline{\mathcal{Z}^{c}}$ need
to be tuned for the problem, and we will present the effect of different
thresholds in Section V.

###  III-C Definition 3: Fairness constraints with budget

We consider static constants111There are different definitions for fairness,
such as pair-wise comparisons between EVs, leading to an exponential number of
constraints. In this work, we choose computationally efficient yet effective
constraints for fairness. $\theta$ and $\mathcal{D}_{i}^{c}$ for fairness in
the EV discharging activities. We define $\theta$ as the measure of a
threshold at time $t$ for each EV participating in the discharge activity. To
maintain the flexibility of the model, we allow vehicles to discharge over the
threshold $\theta$ at time $t$ and consider minimizing the excess discharging
by limiting the total excess discharge by $\mathcal{D}_{i}^{c}$. The excess
amount of discharge of energy can be denoted by $(p_{dis}^{i,t}-\theta)^{+}.$
The amount of excess discharge can only be a positive entity for any EV $i$ at
any time $t$. The fairness equation with budget $\mathcal{D}_{i}^{c}$
represented as

$\sum_{t\in\mathcal{H}}\mathcal{(}p_{dis}^{i,t}-\theta)^{+}\leq\mathcal{D}_{i}^{c}.$
(15)

The results from both approaches with different parameter settings in each of
case will be discussed in Section V.

##  IV Optimization Problem Formulation

After presenting the models for EV charging, V2G, V2V, and fairness, we
formulate an optimization problem to jointly schedule EV charging, V2G, and
V2V subject to fairness and other operational constraints for minimizing the
total cost. In the following, we will present the costs and the revenue of the
charging site.

###  IV-A Grid energy purchase cost

For the cost calculation of energy purchased from the grid, we consider a
time-of-use pricing scheme [4]. The most important characteristic of time-of-
use pricing is that it provides end-users with time-differentiated energy
prices. We consider three predefined price levels per day for peak, off-peak,
and shoulder periods. The energy purchase cost from the grid for EV $i$ is

$C_{grid}^{i}=\sum_{t\in\mathcal{H}}{\phi}_{G}^{t}p_{G}^{i,t},$

where ${\phi}_{G}^{t}$ is the predefined time-of-use price.

###  IV-B V2G revenue earned from the grid

We consider dynamic electricity prices from the Australian Energy Market
Operator (AEMO) for V2G energy transactions. AEMO is responsible for the day-
to-day operations of a variety of electricity and gas markets and information
services, as well as strategic forecasting and planning [20]. We are using
half-hourly prices retrieved from AEMO. Based on the retrieved prices, EVs can
sell energy back through an aggregator or retailer to the grid and generate
revenue. The revenue at time $t$ of EV $i$ is denoted as
${\phi}_{V2G}^{t}p_{V2G}^{i,t}$. Hence, the total revenue of V2G for EV $i$ is

$R_{V2G}^{i}=\sum_{t\in\mathcal{H}}\phi_{V2G}^{t}p_{V2G}^{i,t}.$

###  IV-C Battery degradation cost

The discharging of the EV batteries leads to a progressive alteration in the
physical structure of both the electrolyte and the electrode, resulting in
battery degradation [21]. We consider the battery degradation cost based on
the amount of EV energy discharging while formulating the optimization
problem. The cost of an EV battery degradation for EV $i$ is

$C_{battery}^{i}=\alpha^{i}\sum_{t\in\mathcal{H}}(p_{dis}^{i,t})^{2},$

where $\alpha^{i}$ is the cost coefficient of EV $i$’s battery.

###  IV-D Joint optimization problem for EV charging, V2G, and V2V

We formulate the optimization problem to jointly optimize the EV charging,
V2G, and V2V schedules in the parking site. We denote $c_{total}$ as the total
cost of charging all the EVs in the parking space till the specified
individual target. The objective is to minimize the $c_{total}$ by scheduling
charging and discharging of the EVs through variables including
$p_{grid}^{i,t}$, $p_{V2G}^{i,t}$ and $p_{V2V}^{j,i,t}$. Hence, the
optimization problem is formulated as

$\displaystyle\min\quad$ $\displaystyle
C_{total}=\sum_{i\in\mathcal{N}}(C_{grid}^{i}+C_{battery}^{i}-C_{V2G}^{i})$
(16) subject to
$\displaystyle\text{Constraints}~{}\eqref{cons:dynamics}-\eqref{cons:common_constraints}\text{
\& Fairness Constraints},$

where three types of fairness constraints are presented in Section III. We
include each type in the optimization problem to evaluate its effectiveness,
separately.

In this formulated joint optimization problem, fairness constraint can be
either one of the defined constraints (13), (14) or (15). In the case of
fairness constraint with a budget defined in (15), it’s equivalent to solve
the optimization problem (16) using the following constraints

$\displaystyle\mathcal{Z}_{i}^{t}\geq p_{dis}^{i,t}-\theta,$ (17)
$\displaystyle\mathcal{Z}_{i}^{t}\geq 0,$ (18)
$\displaystyle\sum_{t\in\mathcal{H}}\mathcal{Z}_{i}^{t}\leq\mathcal{D}_{i}^{c},$
(19)

where we denote excess amount of discharge for vehicle $i$ at time $t$ by
$\mathcal{Z}_{i}^{t}$ and $\mathcal{Z}_{i}^{t}$ is non-negative entity for all
$i$ at any time $t$.

The formulated optimization problem is a Mixed-Integer Quadratic programming
(MIQP) problem. Furthermore, baseline optimization problems (20), (21) are
presented in Section IV-E. To solve the optimization problem (16) and baseline
optimization problems (20) and (21), we used Gurobi Solver.

###  IV-E Baseline Optimization Problems

We construct two baseline optimization problems to compare them to the
proposed joint optimization problem and evaluate the benefit of V2V. First, we
consider a baseline problem, which only optimizes EV charging with no V2G and
V2V energy transactions. Second, we consider two baseline problems to jointly
optimize EV charging with V2G.

  * •

For the first baseline problem, we only consider EV charging. Hence, we set
$p_{V2G}^{i,t}=0$ and $p_{V2V}^{i,t}=0$, such that the EV charging uses grid
power and local renewable, i.e.,
$p_{ch}^{i,t}=p_{grid}^{i,t}+p_{Renew}^{i,t}$, and there is no discharge. The
objective function for baseline without considering V2G and V2V given by,

$\displaystyle\min\quad$ $\displaystyle
C_{total}=\sum_{i\in\mathcal{N}}(C_{grid}^{i}+C_{battery}^{i})$ (20) subject
to
$\displaystyle\text{Constraints}~{}\eqref{cons:dynamics}-\eqref{cons:common_constraints}.$

  * •

For the second baseline problem, we consider joint optimization of EV charging
and V2G. In addition to the EV charging similar to the first baseline, there
is EV discharge to perform V2G, i.e., $p_{dis}^{i,t}=p_{V2G}^{i,t}$. The
objective function for baseline only considering V2G given by,

$\displaystyle\min\quad$ $\displaystyle
C_{total}=\sum_{i\in\mathcal{N}}(C_{grid}^{i}+C_{battery}^{i}-C_{V2G}^{i})$
(21) subject to
$\displaystyle\text{Constraints}~{}\eqref{cons:dynamics}-\eqref{cons:common_constraints}.$

Comparing our proposed V2V to the above two baseline problems, we will compare
the value of V2V in EV charging with and without V2G.

##  V Experiments and Results

###  V-A Simulation Setup

We consider an EV parking space with chargers (e.g., an EV charging station),
where all parked EVs can perform bidirectional energy transactions. The
maximum charging rate and battery capacity of all EVs are set to be $7$kW and
$50$kWh, respectively. There are two different cases for EV parking space: a
residential charging station in Case 1 and a commercial charging station in
Case 2. Specifically, in Case 1, EVs are in a residential parking space with
an operational horizon of a day (24 hours), where every 30 minutes is a time
slot. For the simulation of case 1, we settled 50 EVs parked at the charging
station for 24 hours, indicating the EV owners are at home. Another 50 EVs
arrive and depart randomly according to a Gaussian distribution [22]. In Case
2, we consider a shopping center parking space with an operational horizon of
9 hours, from 9 AM to 6 PM as its business hour. In this case, 30 EVs are at a
parking space for the entire 9 hours to simulate staff parking. In addition,
we randomly set the arriving and departure times for another 70 EVs,
simulating customer parking. Fig. 2 presents the time-of-use (TOU) tariff for
EV charging, where the unit of the electricity price is the Australian dollar
($AUD/kWh), and the unit of cost is $AUD. The initial stored energy of EVs is
illustrated in Fig. 3 and desired energy at departure time is distributed from
70% to 100% of the battery capacity. In addition, inspired by [23], we
introduce Jain’s fairness index (JFI) to measure the fairness of energy
trading among EVs defined as

$JFI=\frac{(\sum_{i,j\in\mathcal{N}}\overline{P}_{V2V}^{i,j})^{2}}{R\sum_{i,j\in\mathcal{N}}(\overline{P}_{V2V}^{i,j})^{2}},$
(22)

where $\overline{P}_{V2V}^{i,j}$ represents the average amount of V2V energy
trading in 48 time slots for EV $i$ and $R$ is the number of EVs discharge to
perform V2V. Note that the higher the JFI value is, the fairer the EV
discharge is.

Figure 2: Time-of-Use Tariff in 48 time slots. Figure 3: Probability of EVs’
SOC at the arrival time [24].

###  V-B Simulation Results and Analysis

####  V-B1 Case 1: Simulation results for residential car parking space

In this case, we consider 24 hours horizon (48-time slots) in the simulation.
The EVs’ traffic flow and availability in the parking space are considered for
an entire day (24 hours). Fig. 4 illustrates the minimized energy transfer
cost for charging 100 vehicles in a residential parking space, where we
consider three different scenarios, namely V2V+V2G joint model, V2G, and EV
charging only, respectively. The vertical axis represents the total cost in
Australian dollars for charging all parked EVs to predefined target values
under three scenarios. The model operated on the V2G approach achieved nearly
1% cost reduction compared to the baseline with EV charging only. The joint
model (V2V + V2G) achieved the highest cost reduction of nearly 9.17% compared
to the baseline. This demonstrates the effectiveness of the joint model
(V2V+V2G) in achieving a higher cost reduction than V2G only.

Figure 4: Comparison of the cost of three scenarios in Case 1. Figure 5: Total
cost and Jain fairness index for V2V+V2G model based on fairness definite 1 in
Case 1. Figure 6: Total cost and Jain fairness index for V2V+V2G model based
on fairness definite 2 in Case 1. Figure 7: Total cost for V2V+V2G model based
on fairness definite 3 in Case 1. Figure 8: Jain fairness index for V2V+V2G
model based on fairness definite 3 in Case 1.

Then we evaluate the impact of fairness constraints and the tradeoff between
fairness and EV charging cost. Fig. 5 illustrates the change in the total cost
and Jain fairness index based on the joint model considering the fairness
constraints with fix limit. We can see that the value of threshold
$\overline{\mathcal{Z}}$ is in the range of 0.1 to 1.5. The total cost of the
joint model varies from 200.77 to 216.01 and the Jain fairness index from 0.81
to 0.83. That means the higher value of threshold $\overline{\mathcal{Z}}$ is,
the lower the total cost is. The results of the joint model based on the
second fairness constraint are presented in Fig. 6, which are similar to the
results compared to the first fairness constraint. The threshold
$\overline{\mathcal{Z}^{c}}$ ranges from 1 to 14 and we can see that the Jain
fairness index in the joint model is more sensitive, since the range of the
Jain fairness index in fairness two is from 0.81 to 0.98. Fig. 7 and Fig. 8
show the results considering the third fairness constraint with a budget. We
can see that both thresholds $\theta$ and $\mathcal{D}_{i}^{c}$ affect the
total cost and Jain fairness index of the joint model, where the higher
$\theta$ and $\mathcal{D}_{i}^{c}$ are, the lower total cost and the V2V
energy trading fairness become.

####  V-B2 Case 2: Simulation results for shopping center car parking space

Fig. 9 shows the minimized energy cost for charging 100 vehicles parked in the
shopping center parking space. Similar to the results of case 1, the joint
model achieves the highest cost reduction. It costs $AUD 289.01 to charge the
parked vehicles to their specified target values by only using energy from the
grid. While V2G achieves an insignificant cost reduction of 1%. The joint
model (V2G+V2V) is proven to be the most cost-effective, with a cost reduction
of 12.58% in the case of the shopping center parking.

Figure 9: Comparison of the cost of three scenarios in Case 2. Figure 10:
Total cost and Jain fairness index for V2V+V2G model based on fairness
definite 1 in Case 2. Figure 11: Total cost and Jain fairness index for
V2V+V2G model based on fairness definite 2 in Case 2. Figure 12: Jain fairness
index for V2V+V2G model 1 based on fairness definite 3 in Case 2. Figure 13:
Total cost for V2V+V2G model 1 based on fairness definite 3 in Case 2.

Fig. 10 presents the change of total cost and Jain fairness index based on the
V2V+V2G model considering the fairness constraints with fixed limit. We can
see that the higher value of threshold $\overline{\mathcal{Z}}$ is, the lower
the total cost is, where the total cost ranges from 252.65 to 285.01. However,
with the increase of fairness threshold $\overline{\mathcal{Z}}$, the value of
Jain fairness index fluctuates slightly from 0.73 to 0.74 when
$\overline{\mathcal{Z}}$ range from 0.1 to 0.7. The results based on the
second fairness constraint are presented in Fig. 11, which are similar to the
results compared to the first fairness constraint. With the increase of
threshold $\overline{\mathcal{Z}^{c}}$, the total cost of the joint model
increases from 252.65 to 276.02. Fig. 12 and Fig. 13 show the results
considering the third fairness constraint with a budget for the joint model.
Similarly, we can see both thresholds $\theta$ and $\mathcal{D}_{i}^{c}$
affect the total cost and Jain fairness index. The total cost and the EV
discharging fairness decrease with an increasing $\theta$ and
$\mathcal{D}_{i}^{c}$.

##  VI Conclusion

In this paper, we proposed fairness-aware optimization for the joint
coordination of EV charging, V2G, and V2V, while preventing unbiased
discharging of EVs. We considered two discharging options for EVs, which are
V2G and V2V. Furthermore, we defined fairness metrics for V2V activity to
avoid excessive discharge of EVs and included fairness constraints in the
optimization problem to maintain the fair contribution of EVs. We used the
time-of-use (TOU) scheme for energy purchase prices and wholesale market
prices for selling energy back to the grid. The resulting costs of charging
EVs to the desired levels by different methods were compared. The numerical
results showed the model with the combination of V2G and V2V systems achieves
better performance than V2G and only energy charging alone, with a cost
reduction of 9.17% in the case of residential car park whereas 12.58% in the
case of the shopping car park. In addition, the simulation results showed a
tradeoff between economic performance, charging cost, and fairness of
discharging among EVs.

## References

  * [1] A. Ajanovic, “The future of electric vehicles: prospects and impediments,” _Wiley Interdisciplinary Reviews: Energy and Environment_ , vol. 4, no. 6, pp. 521–536, 2015.
  * [2] U. Abronzini, C. Attaianese, M. D’Arpino, M. Di Monaco, and G. Tomasso, “Cost minimization energy control including battery aging for multi-source ev charging station,” _Electronics_ , vol. 8, no. 1, p. 31, 2019.
  * [3] Y. He, B. Venkatesh, and L. Guan, “Optimal scheduling for charging and discharging of electric vehicles,” _IEEE transactions on smart grid_ , vol. 3, no. 3, pp. 1095–1105, 2012.
  * [4] H. Suyono, M. T. Rahman, H. Mokhlis, M. Othman, H. A. Illias, and H. Mohamad, “Optimal scheduling of plug-in electric vehicle charging including time-of-use tariff to minimize cost and system stress,” _Energies_ , vol. 12, no. 8, p. 1500, 2019.
  * [5] R. K. Varma, G. Sanderson, and K. Walsh, “Global pv incentive policies and recommendations for utilities,” in _2011 24th Canadian Conference on Electrical and Computer Engineering (CCECE)_. IEEE, Conference Proceedings, pp. 001 158–001 163.
  * [6] C. Liu, K. Chau, D. Wu, and S. Gao, “Opportunities and challenges of vehicle-to-home, vehicle-to-vehicle, and vehicle-to-grid technologies,” _Proceedings of the IEEE_ , vol. 101, no. 11, pp. 2409–2427, 2013.
  * [7] K. M. Tan, V. K. Ramachandaramurthy, and J. Y. Yong, “Integration of electric vehicles in smart grid: A review on vehicle to grid technologies and optimization techniques,” _Renewable and Sustainable Energy Reviews_ , vol. 53, pp. 720–732, 2016.
  * [8] S. M. Shariff, D. Iqbal, M. S. Alam, and F. Ahmad, “A state of the art review of electric vehicle to grid (v2g) technology,” in _IOP Conference Series: Materials Science and Engineering_ , vol. 561, no. 1. IOP Publishing, 2019, p. 012103.
  * [9] Z. Yi, D. Scoffield, J. Smart, A. Meintz, M. Jun, M. Mohanpurkar, and A. Medam, “A highly efficient control framework for centralized residential charging coordination of large electric vehicle populations,” _International Journal of Electrical Power & Energy Systems_, vol. 117, p. 105661, 2020.
  * [10] M. R. Sarker, D. J. Olsen, and M. A. Ortega-Vazquez, “Co-optimization of distribution transformer aging and energy arbitrage using electric vehicles,” _IEEE Transactions on Smart Grid_ , vol. 8, no. 6, pp. 2712–2722, 2016.
  * [11] M. S. Hashim, J. Y. Yong, V. K. Ramachandaramurthy, K. M. Tan, M. Mansor, and M. Tariq, “Priority-based vehicle-to-grid scheduling for minimization of power grid load variance,” _Journal of Energy Storage_ , vol. 39, p. 102607, 2021.
  * [12] R. Iacobucci, B. McLellan, and T. Tezuka, “Optimization of shared autonomous electric vehicles operations with charge scheduling and vehicle-to-grid,” _Transportation Research Part C: Emerging Technologies_ , vol. 100, pp. 34–52, 2019.
  * [13] A. Mohamed, V. Salehi, T. Ma, and O. Mohammed, “Real-time energy management algorithm for plug-in hybrid electric vehicle charging parks involving sustainable energy,” _IEEE Transactions on Sustainable Energy_ , vol. 5, no. 2, pp. 577–586, 2013.
  * [14] O. T. T. Kim, N. H. Tran, V. Nguyen, S. M. Kang, and C. S. Hong, “Cooperative between V2C and V2V charging: Less range anxiety and more charged EVs,” _International Conference on Information Networking_ , vol. 2018-Janua, pp. 679–683, 2018.
  * [15] S. Liu, Q. Ni, Y. Cao, J. Cui, D. Tian, and Y. Zhuang, “A reservation-based vehicle-to-vehicle charging service under constraint of parking duration,” _IEEE Systems Journal_ , 2022.
  * [16] S. Poudel, M. Mukherjee, and A. P. Reiman, “A fairness-based distributed energy coordination for voltage regulation in distribution systems,” in _2022 IEEE Green Technologies Conference (GreenTech)_. IEEE, 2022, pp. 45–50.
  * [17] D. Gebbran, S. Mhanna, Y. Ma, A. C. Chapman, and G. Verbič, “Fair coordination of distributed energy resources with volt-var control and pv curtailment,” _Applied Energy_ , vol. 286, p. 116546, 2021.
  * [18] Y. Shirazi, E. Carr, and L. Knapp, “A cost-benefit analysis of alternatively fueled buses with special considerations for v2g technology,” _Energy Policy_ , vol. 87, pp. 591–603, 2015.
  * [19] M. Wang, M. Ismail, R. Zhang, X. S. Shen, E. Serpedin, and K. Qaraqe, “A semi-distributed v2v fast charging strategy based on price control,” in _2014 IEEE Global Communications Conference_. IEEE, 2014, pp. 4550–4555.
  * [20] “Australian energy market operator.” [Online]. Available: https://aemo.com.au/en
  * [21] K. Ginigeme and Z. Wang, “Distributed optimal vehicle-to-grid approaches with consideration of battery degradation cost under real-time pricing,” _IEEE Access_ , vol. 8, pp. 5225–5235, 2020.
  * [22] T.-K. Lee, Z. Bareket, T. Gordon, and Z. S. Filipi, “Stochastic modeling for studies of real-world phev usage: Driving schedule and daily temporal distributions,” _IEEE Transactions on vehicular technology_ , vol. 61, no. 4, pp. 1493–1502, 2011.
  * [23] B. L. Nguyen, D. T. Ngo, M. N. Dao, Q.-T. Duong, and M. Okada, “A joint scheduling and power control scheme for hybrid i2v/v2v networks,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 12, pp. 15 668–15 681, 2020\.
  * [24] R.-C. Leou, “Optimal charging/discharging control for electric vehicles considering power system constraints and operation costs,” _IEEE transactions on power systems_ , vol. 31, no. 3, pp. 1854–1860, 2015.


$\displaystyle\left[r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}\right]\mathrm{d}t+a_{i}(t)\sigma\mathrm{d}W(t)-\theta\mathrm{d}t$
$\displaystyle+\left[\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\right]\mathrm{d}t.$

Recall here $a(t)\sim\mathcal{N}(\psi_{1},\lambda\psi_{2})$, hence, for a
given $\theta,\boldsymbol{\psi}$,

$\displaystyle\mathbb{E}\left[\int_{0}^{T}\frac{\partial
V}{\partial\theta}\left(t,X(t);\theta\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right]$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[\int_{0}^{T}(T-t)\left(r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}-\theta+\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\right)\mathrm{d}t\right]$
$\displaystyle=$
$\displaystyle\int_{0}^{T}(T-t)\left(r+(\mu-r)\psi_{1}-\frac{\gamma}{2}\left(\psi_{1}^{2}+\lambda\psi_{2}\right)\sigma^{2}-\theta+\frac{\lambda}{2}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\right)\mathrm{d}t$ $\displaystyle=$
$\displaystyle\frac{T^{2}}{2}h_{\theta}(\psi_{1},\psi_{2},\theta),$

where

$h_{\theta}(\psi_{1},\psi_{2},\theta)\\\ =r-\theta+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}+(\mu-r)\psi_{1}-\frac{\gamma}{2}\left(\psi_{1}^{2}+\lambda\psi_{2}\right)\sigma^{2}+\frac{\lambda}{2}.$
(35) $\displaystyle\mathbb{E}\left[\int_{0}^{T}\frac{\partial
q}{\partial\psi_{1}}\left(a(t);\boldsymbol{\psi}\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right]$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[\int_{0}^{T}\frac{a(t)-\psi_{1}}{2\psi_{2}}\left(r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}-\theta+\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\right)\mathrm{d}t\right]$
$\displaystyle=$ $\displaystyle Th_{\psi,1}(\psi_{1},\psi_{2},\theta),$

where

$h_{\psi,1}(\psi_{1},\psi_{2},\theta)=\frac{\lambda\gamma\sigma^{2}}{2}\left(\frac{\mu-r}{\gamma\sigma^{2}}-\psi_{1}\right)=\frac{\lambda\gamma\sigma^{2}}{2}\left(\psi_{1}^{*}-\psi_{1}\right).$
(36) $\displaystyle-\mathbb{E}\left[\int_{0}^{T}\frac{\partial
q}{\partial\psi_{2}^{-1}}\left(t,R(t),a(t);\boldsymbol{\psi}\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right]$ $\displaystyle=$
$\displaystyle-\mathbb{E}\Bigg{[}\int_{0}^{T}\left(-\frac{\left(a(t)-\psi_{1}\right)^{2}}{2}+\frac{\lambda}{2}\psi_{2}\right)\bigg{(}r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}-\theta$
$\displaystyle+\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\bigg{)}\mathrm{d}t\Bigg{]}$
$\displaystyle=$ $\displaystyle Th_{\psi,2}(\psi_{1},\psi_{2},\theta),$

where

$h_{\psi,2}(\psi_{1},\psi_{2},\theta)\\\
=\frac{\lambda^{2}}{2}\left(\psi_{2}-\gamma\sigma^{2}\psi_{2}^{2}\right)=\frac{\lambda^{2}\gamma\sigma^{2}}{2}\psi_{2}(\frac{1}{\gamma\sigma^{2}}-\psi_{2})=\frac{\lambda^{2}\gamma\sigma^{2}}{2}\psi_{2}(\psi_{2}^{*}-\psi_{2}).$
(37)

Next, we estimate the variance of the increment. For a given
$\theta,\boldsymbol{\psi}$, by Cauchy-Schwarz inequality and Burkholder-Davis-
Gundy inequality, we have

$\displaystyle\operatorname{Var}\left[\int_{0}^{T}\frac{\partial
V}{\partial\theta}\left(t,X(t);\theta\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\left(\int_{0}^{T}\frac{\partial
V}{\partial\theta}\left(t,X(t);\theta\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right)^{2}\right]$
$\displaystyle\leq$ $\displaystyle
2\mathbb{E}\left[\int_{0}^{T}(T-t)^{2}\left(r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}-\theta+\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\right)^{2}\mathrm{d}t\right]$
$\displaystyle+2\mathbb{E}\left[\int_{0}^{T}(T-t)^{2}a(t)^{2}\sigma^{2}\mathrm{d}t\right]$
$\displaystyle\leq$ $\displaystyle
C(\mu,r,\sigma,T,\gamma)\left[1+\psi_{1}^{4}+\lambda^{2}\psi_{2}^{2}+\theta^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log\psi_{2})^{2}\right].$

Similarly,

$\displaystyle\operatorname{Var}\left[\int_{0}^{T}\frac{\partial
q}{\partial\psi_{1}}\left(a(t);\boldsymbol{\psi}\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\left(\int_{0}^{T}\frac{\partial
q}{\partial\psi_{1}}\left(a(t);\boldsymbol{\psi}\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right)^{2}\right]$
$\displaystyle\leq$ $\displaystyle
2\mathbb{E}\left[\int_{0}^{T}\left(\frac{a(t)-\psi_{1}}{2\psi_{2}}\right)^{2}\left(r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}-\theta+\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\right)^{2}\mathrm{d}t\right]$
$\displaystyle+2\mathbb{E}\left[\int_{0}^{T}\left(\frac{a(t)-\psi_{1}}{2\psi_{2}}\right)^{2}a(t)^{2}\sigma^{2}\mathrm{d}t\right]$
$\displaystyle\leq$ $\displaystyle\lambda
C(\mu,r,\sigma,T,\gamma)\left[1+\lambda^{2}+\lambda^{3}\psi_{2}+\psi_{2}^{-1}(1+\theta^{2}+\psi_{1}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log\psi_{2})^{2})\right].$
$\displaystyle\operatorname{Var}\left[\int_{0}^{T}\frac{\partial
q}{\partial\psi_{2}^{-1}}\left(t,R(t),a(t);\boldsymbol{\psi}\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\left(\int_{0}^{T}\frac{\partial
q}{\partial\psi_{2}^{-1}}\left(t,R(t),a(t);\boldsymbol{\psi}\right)\left[\mathrm{d}V\left(t,X(t);\theta\right)-q\left(a(t);\boldsymbol{\psi}\right)\mathrm{d}t+\frac{1-\gamma}{2}\mathrm{d}\langle
V\left(\cdot,X(\cdot);\theta\right)\rangle(t)\right]\right)^{2}\right]$
$\displaystyle\leq$ $\displaystyle
2\mathbb{E}\Bigg{[}\int_{0}^{T}\left(-\frac{\left(a(t)-\psi_{1}\right)^{2}}{2}+\frac{\lambda}{2}\psi_{2}\right)^{2}\bigg{(}r+(\mu-r)a(t)-\frac{\gamma}{2}a(t)^{2}\sigma^{2}-\theta$
$\displaystyle+\frac{\left(a(t)-\psi_{1}\right)^{2}}{2\psi_{2}}+\frac{\lambda}{2}\log
2\pi\lambda+\frac{\lambda}{2}\log\psi_{2}\bigg{)}^{2}\mathrm{d}t\Bigg{]}+2\mathbb{E}\left[\int_{0}^{T}\left(-\frac{\left(a(t)-\psi_{1}\right)^{2}}{2}+\frac{\lambda}{2}\psi_{2}\right)^{2}a(t)^{2}\sigma^{2}\mathrm{d}t\right]$
$\displaystyle\leq$
$\displaystyle\lambda^{2}C(\mu,r,\sigma,T,\gamma)\left[1+\lambda^{2}\psi_{2}^{2}\left(1+\theta^{2}+\psi_{1}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log\psi_{2})^{2}\right)+\lambda^{3}\psi_{2}^{3}\right].$

####  B.6.2 Stage 2: Recursive relations

From the above calculation, and ignoring the error of approximating the
integral (23), (24), and (25) using finite sums, the recursive relation of the
sequence in the learning procedure can be written as

$\displaystyle\theta_{i+1}=$
$\displaystyle\Pi_{[-c_{i+1},c_{i+1}]}\left(\theta_{i}+a_{\theta,i}\left(h_{\theta}(\psi_{1,i},\psi_{2,i},\theta_{i})+\zeta_{i+1}\right)\right),$
(38) $\displaystyle\psi_{1,i+1}=$
$\displaystyle\Pi_{[-c_{i+1},c_{i+1}]}\left(\psi_{1,i}+a_{\psi,i}\left(h_{\psi,1}(\psi_{1,i},\psi_{2,i},\theta_{i})+\xi_{1,i+1}\right)\right),$
$\displaystyle\psi_{2,i+1}=$
$\displaystyle\Pi_{[b_{i+1}^{-1},c_{i+1}]}\left(\psi_{2,i}+a_{\psi,i}\left(h_{\psi,2}(\psi_{1,i},\psi_{2,i},\theta_{i})+\xi_{2,i+1}\right)\right),$

where $\mathbb{E}[\zeta_{i+1}|\theta_{i},\boldsymbol{\psi}_{i}]=0$,
$\mathbb{E}[\xi_{1,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]=0$,
$\mathbb{E}[\xi_{2,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]=0$, and
$\operatorname{Var}[\zeta_{i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$,
$\operatorname{Var}[\xi_{1,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$, and
$\operatorname{Var}[\xi_{1,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$ satisfy the
upper bounds provided in Stage 1.

We first show the convergence of $\boldsymbol{\psi}_{i}$. Since
$\\{b_{i}\\}_{i\geq 0},\\{c_{i}\\}_{i\geq 0}$ are two increasing, divergent
sequence (condition $(i)$ in the statement), there exists an $I$ that depends
on $(\mu,r,\sigma,T,\gamma,\lambda)$, such that
$\psi_{1}^{*}\in[-c_{i},c_{i}]$ and $\psi_{2}^{*}\in[b_{i}^{-1},c_{i}]$ for
all $i\geq I$. In addition, by conditions $(iii)$, we know
$a_{\psi,i}c_{i}^{2}\to 0$, so does $a_{\psi,i}c_{i}\to 0$. Hence, without
loss of generality, we can take this $I(\mu,r,\sigma,T,\gamma,\lambda)$ large
enough such that $a_{\psi,i}b_{i}^{-1}\leq
a_{\psi,i}c_{i}\leq\frac{1}{\lambda^{2}\gamma\sigma^{2}}$ for all $i\geq I$.
In addition, by the conditions $(i),(ii),(iii)$ in the statement,
$b_{i}\uparrow\infty$, $\sum_{i=1}^{\infty}a_{\psi,i}b_{i}^{-1}=\infty$, and
$\sum_{i=1}^{\infty}a_{\psi,i}^{2}b_{j}^{2}<\infty$, we know
$\sum_{i=1}^{\infty}a_{\psi,i}=\infty$ and
$\sum_{i=1}^{\infty}a_{\psi,i}^{2}<\infty$.

For any $i\geq I$, by the property of the projection mapping as in the proof
of Andradóttir,  (1995, Theorem 2), we have

$\displaystyle\mathbb{E}\left[\left(\psi_{1,i+1}-\psi_{1}^{*}\right)^{2}|\theta_{i},\boldsymbol{\psi}_{i}\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\left(\psi_{1,i}+a_{\psi,i}\left(h_{\psi,1}(\psi_{1,i},\psi_{2,i},\theta_{i})+\xi_{1,i+1}\right)-\psi_{1}^{*}\right)^{2}|\theta_{i},\boldsymbol{\psi}_{i}\right]$
$\displaystyle\leq$
$\displaystyle(\psi_{1,i}-\psi_{1}^{*})^{2}+2a_{\psi,i}(\psi_{1,i}-\psi_{1}^{*})h_{\psi,1}(\psi_{1,i},\psi_{2,i},\theta_{i})+a_{\psi,i}^{2}h_{\psi,1}(\psi_{1,i},\psi_{2,i},\theta_{i})^{2}+a_{\psi,i}^{2}\operatorname{Var}[\xi_{1,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$
$\displaystyle=$
$\displaystyle(1-a_{\psi,i}\lambda\gamma\sigma^{2})(\psi_{1,i}-\psi_{1}^{*})^{2}+\frac{1}{4}a_{\psi,i}^{2}\lambda^{2}\gamma^{2}\sigma^{4}(\psi_{1,i}-\psi_{1}^{*})^{2}+a_{\psi,i}^{2}\operatorname{Var}[\xi_{1,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$
$\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}\lambda\gamma\sigma^{2})^{2}(\psi_{1,i}-\psi_{1}^{*})^{2}+a_{\psi,i}^{2}\lambda
C\left(1+\lambda^{2}+\lambda^{3}c_{i}+b_{i}(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
c_{i})^{2})\right).$

The almost sure convergence of $\\{(\psi_{1,i}-\psi_{1}^{*})^{2}\\}_{i\geq 0}$
follows from Robbins and Siegmund,  (1971, Theorem 1) and by the same argument
as in Andradóttir,  (1995, Theorem 2), we can show $\psi_{1,i}-\psi_{1}^{*}\to
0$ almost surely.

Taking the expectation on both sides yield the recursive relation between two
mean squared error:

$\displaystyle\epsilon_{1,i+1}$ (39) $\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}\lambda\gamma\sigma^{2})^{2}\epsilon_{1,i}+a_{\psi,i}^{2}\lambda
C\left(1+\lambda^{2}+\lambda^{3}c_{i}+b_{i}(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
c_{i})^{2})\right)$ $\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}\lambda\gamma\sigma^{2})\epsilon_{1,i}+a_{\psi,i}^{2}\lambda
C\left(1+\lambda^{2}+\lambda^{3}c_{i}+b_{i}(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
c_{i})^{2})\right),$

where $\epsilon_{1,i}=\mathbb{E}\left[(\psi_{1,i}-\psi_{1}^{*})^{2}\right]$.

By the similar calculation, we have

$\displaystyle\mathbb{E}\left[\left(\psi_{2,i+1}-\psi_{2}^{*}\right)^{2}|\theta_{i},\boldsymbol{\psi}_{i}\right]$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left[\left(\psi_{2,i}+a_{\psi,i}\left(h_{\psi,2}(\psi_{1,i},\psi_{2,i},\theta_{i})+\xi_{2,i+1}\right)-\psi_{2}^{*}\right)^{2}|\theta_{i},\boldsymbol{\psi}_{i}\right]$
$\displaystyle\leq$
$\displaystyle(\psi_{2,i}-\psi_{2}^{*})^{2}+2a_{\psi,i}(\psi_{2,i}-\psi_{2}^{*})h_{\psi,2}(\psi_{1,i},\psi_{2,i},\theta_{i})+a_{\psi,i}^{2}h_{\psi,2}(\psi_{1,i},\psi_{2,i},\theta_{i})^{2}+a_{\psi,i}^{2}\operatorname{Var}[\xi_{2,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$
$\displaystyle=$
$\displaystyle(1-a_{\psi,i}\lambda^{2}\gamma\sigma^{2}\psi_{2,i})(\psi_{2,i}-\psi_{2}^{*})^{2}+\frac{1}{4}a_{\psi,i}^{2}\lambda^{4}\gamma^{2}\sigma^{4}\psi_{2,i}^{2}(\psi_{2,i}-\psi_{2}^{*})^{2}+a_{\psi,i}^{2}\operatorname{Var}[\xi_{2,i+1}|\theta_{i},\boldsymbol{\psi}_{i}]$
$\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}\lambda^{2}\gamma\sigma^{2}\psi_{2,i})^{2}(\psi_{2,i}-\psi_{2}^{*})^{2}+a_{\psi,i}^{2}\lambda^{2}C\left(1+\lambda^{2}c_{i}^{2}\left(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
b_{i})^{2}\right)+\lambda^{3}c_{i}^{3}\right)$ $\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}\lambda^{2}\gamma\sigma^{2}b_{i}^{-1})^{2}(\psi_{2,i}-\psi_{2}^{*})^{2}+a_{\psi,i}^{2}\lambda^{2}C\left(1+\lambda^{2}c_{i}^{2}\left(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
b_{i})^{2}\right)+\lambda^{3}c_{i}^{3}\right),$

where the last inequality holds when $a_{\psi,i}b_{i}^{-1}\leq
a_{\psi,i}c_{i}\leq\frac{1}{\lambda^{2}\gamma\sigma^{2}}$.

The almost sure convergence of $\\{(\psi_{2,i}-\psi_{2}^{*})^{2}\\}_{i\geq 0}$
follows from Robbins and Siegmund,  (1971, Theorem 1) and by the same argument
as in Andradóttir,  (1995, Theorem 2), we can show $\psi_{2,i}-\psi_{2}^{*}\to
0$ almost surely.

Taking the expectation on both sides yield the recursive relation between two
mean squared error:

$\displaystyle\epsilon_{2,i+1}$ (40) $\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}b_{i}^{-1}\lambda^{2}\gamma\sigma^{2})^{2}\epsilon_{2,i}+a_{\psi,i}^{2}\lambda^{2}C\left(1+\lambda^{2}c_{i}^{2}\left(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
c_{i})^{2}\right)+\lambda^{3}c_{i}^{3}\right)$ $\displaystyle\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}b_{i}^{-1}\lambda^{2}\gamma\sigma^{2})\epsilon_{2,i}+a_{\psi,i}^{2}\lambda^{2}C\left(1+\lambda^{2}c_{i}^{2}\left(1+c_{i}^{2}+\lambda^{2}(\log\lambda)^{2}+\lambda^{2}+\lambda^{2}(\log
b_{i})^{2}\right)+\lambda^{3}c_{i}^{3}\right),$

where $\epsilon_{2,i}=\mathbb{E}\left[(\psi_{2,i}-\psi_{2}^{*})^{2}\right]$.

Finally, by the recursive relations (39) and (40), and Lemma 4, we obtain that
under suitable choices of sequences such that $a_{\psi,n}\sim\frac{1}{n}$,
$b_{n},c_{n}\sim\log n$, there are constants $C_{1},C_{2}$ such that

$\epsilon_{1,i+1}\leq C_{1}a_{\psi,i}b_{i}c_{i}^{2}=\tilde{O}(\frac{1}{n}),\
\epsilon_{2,i+1}\leq
C_{1}a_{\psi,i}b_{i}^{2}c_{i}^{4}=\tilde{O}(\frac{1}{n}).$

###  B.7 Proof of Theorem 5

From the definition of ERWL, we can solve

$\Delta(\psi_{1},\lambda\psi_{2})=1-\exp\left\\{-\left[J^{*}(x)-J(x,\psi_{1},\lambda\psi_{2})\right]\right\\}\leq
J^{*}(x)-J(x,\psi_{1},\lambda\psi_{2}),$

where the inequality is due to the fact that $e^{-x}\geq 1-x$. Moreover, from
the expression (27), we have

$J^{*}(x)-J(x,\psi_{1},\lambda\psi_{2})=\frac{\gamma\sigma^{2}T}{2}\left[(\psi_{1}-\psi_{1}^{*})^{2}+\lambda\psi_{2}\right].$

####  B.7.1 Executing deterministic policies

In this case, our choice of tuning sequence $\\{a_{\psi,i}\\}_{i\geq 0}$,
$\\{b_{i}\\}_{i\geq 0}$, $\\{c_{i}\\}_{i\geq 0}$, and $\lambda_{i}=\lambda$ is
identical to that in Theorem 4. Hence, we have

$\displaystyle\mathbb{E}\left[\sum_{i=1}^{N}\Delta(\psi_{1,i},0)\right]\leq$
$\displaystyle\mathbb{E}\left[\sum_{i=1}^{N}\left(J^{*}(x)-J(x,\psi_{1,i},0)\right)\right]$
$\displaystyle=$
$\displaystyle\frac{\gamma\sigma^{2}T}{2}\sum_{i=1}^{N}\mathbb{E}[(\psi_{1,i}-\psi_{1}^{*})^{2}]\leq\frac{C_{1}\gamma\sigma^{2}T}{2}\sum_{i=1}^{N}\frac{(\log
i)^{3}}{i}=O\left((\log N)^{4}\right).$

####  B.7.2 Executing stochastic policies

Suppose in the $i$-th episode, we use $\lambda_{i}$ in (22). The derivation
about the recursive relation in the proof of Theorem 4 is still applicable by
replacing constant $\lambda$ by $\lambda_{i}$. Hence, with the choice of
tuning sequences $a_{\psi,n}\sim\frac{1}{\sqrt{n}}$, $b_{n}=O(1)$,
$c_{n}\sim\log n$, $\lambda_{n}\sim\frac{1}{\sqrt{n}}$, (39) becomes

$\displaystyle\epsilon_{1,i+1}\leq$
$\displaystyle(1-\frac{1}{2}a_{\psi,i}\lambda_{i}\gamma\sigma^{2})\epsilon_{1,i}+a_{\psi,i}^{2}\lambda_{i}C\left(1+\lambda_{i}^{2}+\lambda_{i}^{3}c_{i}+b_{i}(1+c_{i}^{2}+\lambda_{i}^{2}(\log\lambda_{i})^{2}+\lambda_{i}^{2}+\lambda_{i}^{2}(\log
c_{i})^{2})\right)$ $\displaystyle=$
$\displaystyle(1-\frac{\gamma\sigma^{2}}{2}a_{\psi,i}\lambda_{i})\epsilon_{1,i}+a_{\psi,i}^{2}\lambda_{i}C(1+c_{i}^{2}).$

The convergence of $\epsilon_{1,i}$ is guaranteed since $\sum
a_{\psi,i}\lambda_{i}=\sum\frac{1}{i}=\infty$, and $\sum
a_{\psi,i}^{2}\lambda_{i}c_{i}^{2}=\sum\frac{\log i}{i^{3/2}}<\infty$.
Moreover, by Lemma 4, we obtain $\epsilon_{1,i+1}=O\left(\frac{(\log
i)^{2}}{\sqrt{i}}\right)$.

Hence,

$\displaystyle\mathbb{E}\left[\sum_{i=1}^{N}\Delta(\psi_{1,i},\lambda_{i}\psi_{2,i})\right]\leq$
$\displaystyle\mathbb{E}\left[\sum_{i=1}^{N}\left(J^{*}(x)-J(x,\psi_{1,i},\lambda_{i}\psi_{2,i})\right)\right]$
$\displaystyle\leq$
$\displaystyle\frac{\gamma\sigma^{2}T}{2}\sum_{i=1}^{N}\mathbb{E}[(\psi_{1,i}-\psi_{1}^{*})^{2}]+\frac{\gamma\sigma^{2}T}{2}\sum_{i=1}^{N}\lambda_{i}c_{i}$
$\displaystyle\leq$
$\displaystyle\frac{C\gamma\sigma^{2}T}{2}\sum_{i=1}^{N}\left(\frac{(\log
i)^{2}}{\sqrt{i}}+\frac{\log i}{\sqrt{i}}\right)=O(\sqrt{N}(\log N)^{2}).$

## References

  * Andersson et al., (2015) Andersson, O., Heintz, F., and Doherty, P. (2015).  Model-based reinforcement learning in continuous environments using real-time constrained optimization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29.
  * Andradóttir, (1995) Andradóttir, S. (1995).  A stochastic approximation algorithm with varying bounds.  Operations Research, 43(6):1037–1048.
  * Baird, (1994) Baird, L. C. (1994).  Reinforcement learning in continuous time: Advantage updating.  In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94), volume 4, pages 2448–2453. IEEE.
  * Bielecki and Pliska, (1999) Bielecki, T. R. and Pliska, S. R. (1999).  Risk-sensitive dynamic asset management.  Applied Mathematics and Optimization, 39:337–360.
  * Blanchet et al., (2022) Blanchet, J., Chen, L., and Zhou, X. Y. (2022).  Distributionally robust mean-variance portfolio selection with Wasserstein distances.  Management Science, 68(9):6382–6410.
  * Borkar, (2002) Borkar, V. S. (2002).  Q-learning for risk-sensitive control.  Mathematics of Operations Research, 27(2):294–311.
  * Broadie et al., (2011) Broadie, M., Cicek, D., and Zeevi, A. (2011).  General bounds and finite-time improvement for the Kiefer-Wolfowitz stochastic approximation algorithm.  Operations Research, 59(5):1211–1224.
  * Chow et al., (2015) Chow, Y., Tamar, A., Mannor, S., and Pavone, M. (2015).  Risk-sensitive and robust decision-making: A CVaR optimization approach.  Advances in Neural Information Processing Systems, 28.
  * (9) Dai, M., Dong, Y., and Jia, Y. (2023a).  Learning equilibrium mean-variance strategy.  Mathematical Finance, 33(4):1166–1212.
  * (10) Dai, M., Dong, Y., Jia, Y., and Zhou, X. Y. (2023b).  Learning Merton’s strategies in an incomplete market: Recursive entropy regularization and biased Gaussian exploration.  arXiv preprint arXiv:2312.11797.
  * Dai et al., (2021) Dai, M., Jin, H., Kou, S., and Xu, Y. (2021).  A dynamic mean-variance analysis for log returns.  Management Science, 67(2):1093–1108.
  * Davis and Lleo, (2014) Davis, M. H. and Lleo, S. (2014).  Risk-sensitive Investment Management, volume 19.  World Scientific, Singapore.
  * Donsker and Varadhan, (1983) Donsker, M. D. and Varadhan, S. S. (1983).  Asymptotic evaluation of certain Markov process expectations for large time. IV.  Communications on Pure and Applied Mathematics, 36(2):183–212.
  * Doya, (2000) Doya, K. (2000).  Reinforcement learning in continuous time and space.  Neural Computation, 12(1):219–245.
  * Duffie and Epstein, (1992) Duffie, D. and Epstein, L. G. (1992).  Stochastic differential utility.  Econometrica, pages 353–394.
  * Dupuis et al., (2000) Dupuis, P., James, M. R., and Petersen, I. (2000).  Robust properties of risk-sensitive control.  Mathematics of Control, Signals and Systems, 13:318–332.
  * Enders et al., (2024) Enders, T., Harrison, J., and Schiffer, M. (2024).  Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts.  arXiv preprint arXiv:2402.09992.
  * Epstein and Zin, (1989) Epstein, L. G. and Zin, S. E. (1989).  Substitution, risk aversion, and the temporal behavior of consumption.  Econometrica, 57(4):937–969.
  * Fei et al., (2021) Fei, Y., Yang, Z., Chen, Y., and Wang, Z. (2021).  Exponential Bellman equation and improved regret bounds for risk-sensitive reinforcement learning.  Advances in Neural Information Processing Systems, 34:20436–20446.
  * Fei et al., (2020) Fei, Y., Yang, Z., Chen, Y., Wang, Z., and Xie, Q. (2020).  Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret.  Advances in Neural Information Processing Systems, 33:22384–22395.
  * Fleming and McEneaney, (1995) Fleming, W. H. and McEneaney, W. M. (1995).  Risk-sensitive control on an infinite time horizon.  SIAM Journal on Control and Optimization, 33(6):1881–1915.
  * Fleming and Nisio, (1984) Fleming, W. H. and Nisio, M. (1984).  On stochastic relaxed control for partially observed diffusions.  Nagoya Mathematical Journal, 93:71–108.
  * Fleming and Sheu, (2002) Fleming, W. H. and Sheu, S. (2002).  Risk-sensitive control and an optimal investment model II.  The Annals of Applied Probability, 12(2):730–767.
  * Frikha et al., (2023) Frikha, N., Germain, M., Laurière, M., Pham, H., and Song, X. (2023).  Actor-critic learning for mean-field control in continuous time.  arXiv preprint arXiv:2303.06993.
  * Gilboa and Schmeidler, (1989) Gilboa, I. and Schmeidler, D. (1989).  Maxmin expected utility with non-unique prior.  Journal of Mathematical Economics, 18(2):141–153.
  * Glasserman and Xu, (2013) Glasserman, P. and Xu, X. (2013).  Robust portfolio control with stochastic factor dynamics.  Operations Research, 61(4):874–893.
  * Guo et al., (2022) Guo, X., Xu, R., and Zariphopoulou, T. (2022).  Entropy regularization for mean field games with learning.  Mathematics of Operations research, 47(4):3239–3260.
  * Hansen and Sargent, (2001) Hansen, L. P. and Sargent, T. J. (2001).  Robust control and model uncertainty.  American Economic Review, 91(2):60–66.
  * Hansen and Sargent, (2011) Hansen, L. P. and Sargent, T. J. (2011).  Robustness and ambiguity in continuous time.  Journal of Economic Theory, 146(3):1195–1223.
  * Jacobson, (1973) Jacobson, D. (1973).  Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games.  IEEE Transactions on Automatic control, 18(2):124–131.
  * (31) Jia, Y. and Zhou, X. Y. (2022a).  Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach.  Journal of Machine Learning Research, 23(1):6918–6972.
  * (32) Jia, Y. and Zhou, X. Y. (2022b).  Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms.  Journal of Machine Learning Research, 23(1):12603–12652.
  * Jia and Zhou, (2023) Jia, Y. and Zhou, X. Y. (2023).  q-Learning in continuous time.  Journal of Machine Learning Research, 24(161):1–61.
  * Jiang et al., (2022) Jiang, R., Saunders, D., and Weng, C. (2022).  The reinforcement learning Kelly strategy.  Quantitative Finance, 22(8):1445–1464.
  * Jin et al., (2018) Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018).  Is Q-learning provably efficient?  Advances in Neural Information Processing Systems, 31.
  * Kim et al., (2021) Kim, J., Shin, J., and Yang, I. (2021).  Hamilton-Jacobi deep Q-learning for deterministic continuous-time systems with Lipschitz continuous controls.  Journal of Machine Learning Research, 22(206):1–34.
  * Kober et al., (2013) Kober, J., Bagnell, J. A., and Peters, J. (2013).  Reinforcement learning in robotics: A survey.  The International Journal of Robotics Research, 32(11):1238–1274.
  * Kushner and Yin, (2003) Kushner, H. and Yin, G. (2003).  Stochastic Approximation and Recursive Algorithms, volume 35.  Springer-Verlag, New York, 2 edition.
  * Lai, (2003) Lai, T. L. (2003).  Stochastic approximation.  The Annals of Statistics, 31(2):391–406.
  * Lee and Sutton, (2021) Lee, J. and Sutton, R. S. (2021).  Policy iterations for reinforcement learning problems in continuous time and space—Fundamental theory and methods.  Automatica, 126:109421.
  * LeRoy and Singell Jr, (1987) LeRoy, S. F. and Singell Jr, L. D. (1987).  Knight on risk and uncertainty.  Journal of Political Economy, 95(2):394–406.
  * Maenhout, (2004) Maenhout, P. J. (2004).  Robust portfolio rules and asset pricing.  Review of Financial Studies, 17(4):951–983.
  * Menaldi and Robin, (2005) Menaldi, J.-L. and Robin, M. (2005).  Remarks on risk-sensitive control problems.  Applied Mathematics and Optimization, 52:297–310.
  * Merton, (1969) Merton, R. C. (1969).  Lifetime portfolio selection under uncertainty: The continuous-time case.  The Review of Economics and Statistics, pages 247–257.
  * Nagai, (1996) Nagai, H. (1996).  Bellman equations of risk-sensitive control.  SIAM Journal on Control and Optimization, 34(1):74–101.
  * Revuz and Yor, (2013) Revuz, D. and Yor, M. (2013).  Continuous Martingales and Brownian Motion, volume 293.  Springer Science & Business Media, Berlin.
  * Robbins and Monro, (1951) Robbins, H. and Monro, S. (1951).  A stochastic approximation method.  The Annals of Mathematical Statistics, pages 400–407.
  * Robbins and Siegmund, (1971) Robbins, H. and Siegmund, D. (1971).  A convergence theorem for non negative almost supermartingales and some applications.  Optimizing Methods in Statistics, pages 233–257.
  * Schulman et al., (2017) Schulman, J., Chen, X., and Abbeel, P. (2017).  Equivalence between policy gradients and soft Q-learning.  arXiv preprint arXiv:1704.06440.
  * Skiadas, (2003) Skiadas, C. (2003).  Robust control and recursive utility.  Finance and Stochastics, 7:475–489.
  * Sun, (2006) Sun, Y. (2006).  The exact law of large numbers via Fubini extension and characterization of insurable risks.  Journal of Economic Theory, 126(1):31–69.
  * Szpruch et al., (2024) Szpruch, L., Treetanthiploet, T., and Zhang, Y. (2024).  Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning.  SIAM Journal on Control and Optimization, 62(1):135–166.
  * Tallec et al., (2019) Tallec, C., Blier, L., and Ollivier, Y. (2019).  Making deep Q-learning methods robust to time discretization.  In International Conference on Machine Learning, pages 6096–6104. PMLR.
  * Tang et al., (2022) Tang, W., Zhang, P. Y., and Zhou, X. Y. (2022).  Exploratory HJB equations and their convergence.  SIAM Journal on Control and Optimization, 60(6):3191–3216.
  * (55) Wang, B., Gao, X., and Li, L. (2023a).  Reinforcement learning for continuous-time optimal execution: actor-critic algorithm and error analysis.  Available at SSRN 4378950.
  * Wang et al., (2020) Wang, H., Zariphopoulou, T., and Zhou, X. Y. (2020).  Reinforcement learning in continuous time and space: A stochastic control approach.  Journal of Machine Learning Research, 21(198):1–34.
  * Wang and Zhou, (2020) Wang, H. and Zhou, X. Y. (2020).  Continuous-time mean–variance portfolio selection: A reinforcement learning framework.  Mathematical Finance, 30(4):1273–1308.
  * (58) Wang, S., Si, N., Blanchet, J., and Zhou, Z. (2023b).  A finite sample complexity bound for distributionally robust Q-learning.  In International Conference on Artificial Intelligence and Statistics, pages 3370–3398. PMLR.
  * Wei and Yu, (2023) Wei, X. and Yu, X. (2023).  Continuous-time q-learning for McKean-Vlasov control problems.  arXiv preprint arXiv:2306.16208.
  * Wu and Xu, (2023) Wu, Z. and Xu, R. (2023).  Risk-sensitive Markov decision process and learning under general utility functions.  arXiv preprint arXiv:2311.13589.
  * Xu et al., (2023) Xu, W., Gao, X., and He, X. (2023).  Regret bounds for Markov decision processes with recursive optimized certainty equivalents.  In International Conference on Machine Learning, pages 38400–38427. PMLR.
  * Yong and Zhou, (1999) Yong, J. and Zhou, X. Y. (1999).  Stochastic Controls: Hamiltonian Systems and HJB Equations.  New York, NY: Spinger.
  * Zhou, (1992) Zhou, X. Y. (1992).  On the existence of optimal relaxed controls of stochastic partial differential equations.  SIAM Journal on Control and Optimization, 30(2):247–261.