11institutetext: Natural Language and Dialogue Systems Lab, University of
California, Santa Cruz 11email: aramir62, malsalih, kartik
<EMAIL_ADDRESS>
# Controlling Personality Style in Dialogue with Zero-Shot Prompt-Based
Learning

Angela Ramirez Mamon Alsalihy Kartik Aggarwal Cecilia Li Liren Wu and Marilyn
Walker

###### Abstract

Prompt-based or in-context learning has been shown to achieve high zero-shot
performance on many natural language generation (NLG) tasks. Here we explore
the performance of prompt-based learning for simultaneously controlling the
personality and the semantic accuracy of an NLG for task-oriented dialogue. We
experiment with prompt-based learning on the personage restaurant
recommendation corpus to generate semantically and stylistically-controlled
text for 5 different Big-5 personality types: agreeable, disagreeable,
conscientious, unconscientious, and extravert. We test two different classes
of discrete prompts to generate utterances for a particular personality style:
(1) prompts that demonstrate generating directly from a meaning representation
that includes a personality specification; and (2) prompts that rely on first
converting the meaning representation to a textual pseudo-reference, and then
using the pseudo-reference in a textual style transfer (TST) prompt. In each
case, we show that we can vastly improve performance by over-generating
outputs and ranking them, testing several ranking functions based on automatic
metrics for semantic accuracy, personality-match, and fluency. We also test
the effect of providing examples of multiple personalities, and of different
sampling strategies and numbers of examples, as well as testing whether NLG
personality demonstrations from the restaurant domain can be used with meaning
representations for the video game domain to generate personality stylized
utterances about video games. Our findings show that the TST prompts produces
the highest semantic accuracy (78.46% for restaurants and 87.6% for video
games) and personality accuracy (100% for restaurants and 97% for video
games). Our results on transferring personality style to video game utterances
are surprisingly good. To our knowledge, there is no previous work testing the
application of prompt-based learning to simultaneously controlling both style
and semantic accuracy in NLG.

###### Keywords:

p

ersonality, stylistic generation, task-oriented dialogue, natural language
generation, style transfer, prompt-based learning, evaluation

##  1 Introduction

Over the last few years, prompt-based or in-context learning has been shown to
achieve high performance on many natural language generation (NLG) tasks bach-
etal-2022-promptsource ; li-etal-2022-prompt-based ; liu2021pre . Here we
explore the performance of prompt-based learning for controlling both the
personality and the semantic accuracy in natural language generation for
dialogue. We experiment with prompt-based learning on the personage corpus, a
stylistic benchmark dataset for semantically-controlled NLG in the restaurant
domain, with reference utterances that vary stylistically according to
linguistic profiles of Big-5 personality types PennebakerKing99 ; Furnham90 ;
MairesseWalker11 ; harrison2019maximizing ; oraby2018neural ; orabyreed2018 .
The personality styles consist of 5 different Big-5 personality types:
agreeable, disagreeable, conscientious, unconscientious, and extravert.

We compare two different types of discrete prompts: (1) Data-to-text (D2T)
prompts that directly demonstrate generating from a meaning representation
that includes a personality specification; (2) prompts that are based on
textual style transfer (TST) instructions reif-etal-2022-recipe , that require
first converting the meaning representation to a a textual pseudo-reference.
The two types of prompts are illustrated in Table 1 for the agreeable Big-5
personality style. We also vary the number of demonstrations we provide, as
well as whether the examples illustrate one personality or multiple
personalities, and systematically examine the effect.

Figure 1: Two models for Semantically Controlled Generation with Style

The two methods are illustrated in Figure 1. Using both methods, we show that
we can vastly improve performance by over-generating multiple outputs for each
setting LangkildeKnight98 , and then ranking the outputs using a combination
of personality accuracy, semantic accuracy and fluency. For semantic accuracy,
we compare off-the-shelf semantic faithfulness metrics such as beyond-bleu and
bleurt to personage specific scripts for calculating slot error rate wieting-
etal-2019-beyond ; bleurt2020 ; reedetal20 . To measure personality accuracy,
we train a personality classifier.

Data To Text Prompt (D2T)
---
name = nameVariable $|$ eattype = restaurant $|$ food = chinese $|$ pricerange
= moderate $|$ area = riverside $|$ familyfriendly = yes $|$ near =
nearVariable $|$ personality = agreeable
Let’s see what we can find on nameVariable. oh right, it is an chinese
restaurant in riverside with a quite moderate rating and it is kid friendly,
also it is near nearVariable, you know, okay?

name = namevariable $|$ eattype = pub $|$ food = italian $|$ area = city
centre $|$ familyfriendly = no $|$ near = nearvariable $|$ personality =
agreeable
Textual Style Transfer Prompt (TST)
Here is some text: {namevariable restaurant chinese moderate riverside family
friendly nearvariable}. Here is a rewrite of the text which is agreeable :
{Let’s see what we can find on nameVariable. I see it is a Chinese restaurant
in riverside, also it is moderately priced and family friendly and near
nearVariable.}.

Here is some text: {nameVariable pub Italian city centre not family friendly
nearVariable }. Here is a rewrite of the text which is agreeable : {
Table 1: Example D2T and TST prompts for the Big-5 agreeable personality

Based on on our results for personage, we use our best performing experimental
setting on an out-of-domain dataset for Data-to-Text NLG for Video Games,
ViGGO. By doing so, we are testing whether personality examples from the
restaurant domain can be used on meaning representations for the video game
domain to generate personality stylized utterances about video games. The
ViGGO corpus comes with a script for calculating semantic accuracy that is
specific to this domain juraska2019viggo , so we are able to apply the same
ranking functions as we use for personage.

Our results show that prompting with a single personality performs better both
for achieving the target style and faithfully rendering the meaning. Our best
performing setting achieves personality accuracies of 100% and a best slot
error rate of 22%. To our knowledge, there is no previous work testing the
performance of prompt-based learning for simultaneously controlling both style
and semantic accuracy in NLG.

##  2 Related Work

Prompt-based learning has recently been applied to many different NLG tasks.
Previous work on semantically controlled NLG using prompt-based learning has
focused on semantic accuracy rather than attempting to simultaneously control
both semantics and style reed2021 ; li-liang-2021-prefix ; xiang2022asdot .
Previous work on controlling style using prompt-based learning has been framed
as a textual style transfer (TST) task, where the goal is to enhance the text
with stylistic features while preserving the overall semantics and fluency of
the text reif-etal-2022-recipe ; suzgun2022prompt ; jin-etal-2022-deep ; li-
etal-2022-learning-transfer . These measures strongly parallel the evaluation
measures that we use for stylistically and semantically controlled NLG. In
TST, stylistic correctness is typically measured with pre-trained style
classifiers, as we do here. However it is notoriously difficult to measure
semantic preservation in text-to-text tasks, where the definition of meaning
tends to be quite slippery. Much work still uses bleu even though for many
tasks it has been shown not to correlate with human judgements reif-
etal-2022-recipe ; Papineni02bleu:a ; jin-etal-2022-deep . Newer neural
measures such as beyond-bleu, bleurt and bertScore have also been used, with
some recent work showing that beyond-bleu produces good results when used
directly during fine-tuning liu-etal-2021-learning .

Earlier work on controlling both semantics and style was based on seq-to-seq
LSTM + attention models trained with thousands of examples oraby2018neural ;
oraby2019curate ; tsai-etal-2021-style ; reed2018can ; harrison2019maximizing
. We compare our results to previous seq-to-seq results on the personage
corpus and on the viggo corpus in Section 4.

One of the key elements of our novel approach is converting our data-to-text
problem to a text-to-text problem by generating pseudo-references directly
from our meaning representations. Work by Heidari et al. also experimented
with different ways to convert meaning representations to textual forms for
the purpose of fine-tuning heidari-etal-2021-getting . They then show that
they can use as few as 300 examples to fine-tune their NLG engine
heidari2021getting . Other work on data-to-text generation that has used
prompt-based learning has relied on models like GPT-3 to convert single KG
triples into texts, and then fused those texts into a paragraph xiang2022asdot
, but has not directly measured semantic accuracy or aimed to enforce a
specific style to be generated. However this work, as well as other research,
shows that meaning representations can be used directly in a prompt format to
generate sentences soltan2022alexatm ; liu2021pre ; reed2021 . Models are
clearly sensitive to the type of prompt provided webson-pavlick-2022-prompt ;
bach-etal-2022-promptsource , so we carefully compare classic data-to-text
prompts with prompts that convert data-to-text to a text-to-text problem.

##  3 Experimental Method

Here, we test two different prompt-based learning approaches for semantically
controlled stylistic generation, as illustrated in Figure 1. We aim to
understand which method best conditions the NLG outputs.

###  3.1 Personage and ViGGO Datasets

Our primary corpus is personage, 111nlds.soe.ucsc.edu/stylistic-nlg as
illustrated in Figure 2 oraby2018neural ; harrison2018neural . personage
contains $\sim$ 88,000 restaurant recommendations that vary along the Big Five
Personality traits: agreeable, disagreeable, conscientious, unconscientious,
and extrovert Furnham90 ; Mairesse07 ; mairesse-walker-2011-controlling .
Table 2 shows an example MR from Personage along with five surface
realizations: a pseudo-reference generated directly from the meaning
representation, as we describe in Section 3.2 juraska2022 ; a vanilla
utterance generated for the E2E generation challenge; examples of the
extravert, conscientious and agreeable personality types from the personage
corpus PennebakerKing99 ; Furnham90 ; MairesseWalker11 .

Meaning Representation
---
name = nameVariable $|$ eatType = pub $|$ food = English $|$ priceRange = high
$|$ area = city centre $|$ familyFriendly = no $|$ near = nearVariable
Pseudo Reference
nameVariable pub English city centre high price range kid friendly family
friendly.
E2E
nameVariable is a pub. It is an English place and it is in city centre. It has
a high price range and it is kid friendly and family friendly.
Personality: Extravert
nameVariable is a pub, it is an English place and it is in city centre, it has
a high price range friend and it is kid friendly and family friendly, you
know!
Personality: Conscientiousness
Let’s see what we can find out about nameVariable. Well, it seems it isn’t
family friendly and it has a mediocre rating, also it’s an English pub in city
centre near nearVariable, also it is expensive.
Personality: Agreeable
You want to know more about nameVariable Oh it is an English restaurant near
nearVariable and it isn’t kid friendly, also it’s rather expensive and a pub
in city centre, okay?
Figure 2: Sample meaning representation with a vanilla realization labelled
E2E and three personality-based stylistic realizations from the personage
Dataset.

Recent work on stylistic variation in NLG for task-oriented dialogue has
categorized stylistic variation into lexical, syntactic and semantic styles
tsai-etal-2021-style . Mairesse et al. mairesse2010towards  provides a
detailed summary of the psycholinguistic literature on how the Big 5
personality types are manifested in language, showing that personality affects
style at all three levels, e.g. an extraverted personality will use more
frequent words, will produce longer sentences with more aggregation
operations, and select more positive content, while introverts tend to use
rare words. The types of variation that are present in the Personage corpus
are both lexical and syntactic, and categorized into Aggregation operations
and Pragmatic operations. Aggregation operations modify syntactic dependency
trees to combine propositions into a single sentence: these syntactic
operations are aslo typically indicated by lexical cues such ”with”, ”and”,
and ”also”, as illustrated by the Extraversion personality in Table 2.
Pragmatic operations also often involve lexical cues, but their applications
typically requires knowledge of syntactic or semantic constraints. For
example, hedges such as “rather” can only be placed before scalar adjectives
such as “expensive”, as illustrated by the Agreeable example in Table 2, while
insertion of hedges such as “you know” as shown in the Extraversion example in
Table 2 is less constrained. Adding a tag question such as “okay?”, or “isn’t
it?” to the end of a sentencem as seen in the Agreeable example, may require
identifying the subject of the sentence in order to match the pronoun.

_give_opinion_(name [SpellForce 3], rating [poor], genres [real-time strategy,
role-playing], player_perspective [bird view])
---
I think that SpellForce 3 is one of the worst games I’ve ever played. Trying
to combine the real-time strategy and role-playing genres just doesn’t work,
and the bird’s eye view makes it near impossible to play.
_verify_attribute_(name [Little Big Adventure], rating [average],
has_multiplayer [no], platforms [PlayStation])
I recall that you were not that fond of Little Big Adventure. Does single-
player gaming on the PlayStation quickly get boring for you?
Table 2: Examples of MRs and corresponding reference utterances in the ViGGO
dataset. The DA of the MRs is indicated in italics, and the slots in small
caps. The slot mentions in the utterances are bolded.

To our knowledge, personage is the only corpus that provides reference
utterances for Big-5 personalities for a data-to-text generation task.
However, we hypothesized that we could achieve some style transfer to another
domain by prompting with personality demonstrations from personage, and
requesting an output using a meaning representations in another domain. Thus,
after experimenting with personage, we also test the ability to transfer
personality styles from personage to the ViGGO video games corpus. Several
examples of meaning representations and reference utterances from the original
ViGGO corpus are shown in Table 2.

###  3.2 Ranking using Automatic Metrics

The overgenerate-and-rank method for NLG for dialogue systems assumes that
overgeneration will produce multiple viable candidates, and that the best
candidate(s) can be identified through ranking, in real time. In this paper,
we leave aside the real-time requirement, and test whether ranking can improve
the fluency, semantic accuracy, and the manifestation of personality in the
selected output. More formally, a high-quality response generated from a model
based on the personality and the MR provided in the prompt should: (1)
strongly manifest the personality; (2) have no missing or incorrect mentions
of the attribute values; (3) produce no irrelevant attribute mentions i.e.
hallucinations; and (4) be fluent. The generated utterance $y$, conditioned on
a Personality $P$, and an MR $x$ with slot values $s$, can be formulated as
$y=f(P,s)$. The conditional likelihood of an utterance given P,MR can be
decomposed into the product of three probabilities:

$p(y|P,s)=p(P|y,s)*p(s|y)*p(y)$ (1)

The term $p(P|y,s)$ is the probability of a particular personality given the
generated utterance $y$ and the semantic attributes $s$. The term $p(s|y)$
represents the semantic accuracy. The term $p(y)$ is the unconditional
probability of the generated text. We calculate the Personality probability
with a personality classifier (Sec. 3.2), and test multiple ways of computing
semantic accuracy (Sec. 3.2). We automatically measure the fluency of the
generated text as sentence probability, as in previous work jin2022deep ;
suzgun2022prompt . We discuss how we can use these together to define several
different ranking functions in Sec. 3.2..

#### Measuring Semantic Accuracy

One advantage of semantically controlled natural language generation is that
the meaning representation, here represented as a dialogue act with its
attributes and values, provides an objective way to measure semantic
correctness. In contrast, work on machine translation, paraphrasing and
textual style transfer (TST) use more approximate measures such as semantic
similarity or bleu, despite acknowledging bleu’s limitations
chen2011collecting ; wieting-etal-2019-beyond .

Previous work on data-to-text NLG defined a metric called the Slot Error Rate
(SER) i.e. the percentage of slots $y$ that the NLG failed to realize in $x$
Wenetal15 ; juraska-walker-2021-attention . Work on the personage corpus
defines semantic error by adding the number of substitutions S, deletions D,
repeats R, and hallucinations H harrison2019maximizing ; juraska2018deep ;
Reedetal18 ; reedetal20 .222Hallucinations can only be recognized for known
attributes, but previous work has shown high correlations between human
judgements and the ViGGO and personage SER scripts. The SER formula is then:

$SER=\frac{S+D+R+H}{N}$ (2)

where N is the number of slots in the MR. Previous work on the ViGGO dataset
also provides scripts for calculating the SER juraska-walker-2021-attention .
We use these off-the-shelf SER scripts for both Personage and ViGGO here.
Ranking needs an accuracy measure rather than an error measure, so for both
Personage and ViGGO, we derive the SACC measure as:

$SACC=1-SER$ (3)

Measure | Personage | ViGGO
---|---|---
pBLEU | 0.11 | 0.29
pBeyond-BLEU | 0.29 | 0.48
pBLEURT | 0.39 | 0.50
pBERT precision | 0.31 | 0.47
pBERT recall | 0.11 | 0.34
pBERT F1 | 0.24 | 0.42
Table 3:  Pearson Correlation between saccand common Semantic Preservation
Measures when applied to Pseudo References. All correlations are statistically
significant at p $=$ 0.

However, because the sacc metric is specific to personage and ViGGO, we also
explore the use of common reference-based metrics for measuring semantic
preservation, namely bleu, beyond-bleu, bleurt and bertScore Papinenietal02 ;
liu-etal-2021-learning ; wieting-etal-2019-beyond ; bleurt2020 ;
zhang2019bertscore . These metrics are designed to be reference-based, i.e.
they compare a generated text with a reference text (or a set of them),
typically written by humans. However we apply a novel method to use reference-
based metrics with pseudo-references, where we compare the generated
utterances directly against the input MR that has been converted into a
textual representation juraska2022 ; heidari-etal-2021-getting . The MR is
linearized and the slot values are concatenated, except that boolean-valued
slots such as “family friendly” are represented by their slot names rather
than their values. A personage example is shown in the second row of Figure 2.
While these metrics may produce rather low scores on (pseudo-reference,
reference) text pairs, here we are interested in relative scores rather than
absolute values.

We assume that our domain specific sacc measures are the best possible measure
of semantic accuracy and compute the correlations between sacc and the
reference-based metrics for both personage and ViGGO as shown in Table 3. For
both personage and ViGGO, pbleu is the least correlated measure while beyond-
bleu, bleurt and bertScore Precision are the most highly correlated. We use
these correlations in defining the ranking functions in Section 3.2.

#### Personality Style Classifier

Personality |  F1 |  P |  R
---|---|---|---
extravert |  0.99 |  1.00 |  0.99
agreeable |  0.99 |  0.99 |  0.99
disagreeable |  0.99 |  0.99 |  0.99
conscientious |  0.99 |  0.99 |  0.99
unconscientious |  0.99 |  1.00 |  1.00
Table 4:  Precision, Recall and F1 for the Personality Classifier

We also need an automatic method for measuring stylistic strength or the
manifestation of personality. Style classifiers are usually used to measure
style strength in TST liu-etal-2021-learning ; jin2022deep , so we train a
multi-class personality style classifier on a balanced set of 4,000 samples of
reference utterances from the personage dataset, i.e. 800 per personality
reed2018can . The classifier is a 110 million parameter BERT model that was
fine-tuned with the following hyperparameters: learning rate: 3e-4, train
batch size: 128, evaluation batch size: 32, epochs: 3. Table 4. reports the
F1, Precision, and Recall for each personality type on the personage test set
references of 1390 examples. At inference time, each sample is a stylistic
personality realization generated by Jurassic. When the personality matches
the intended personality, the probabilities of classifier are used as a term
in the ranking functions detailed in Section 3.2 below.

#### Ranking Functions

RF1: SACC * PAC * P(S)
---
RF2: SACC * PAC *P(S) * pBLEU
RF3: pBBLEU * PAC *P(S)
RF4: pBLEURT * PAC * P(S)
RF5: pBERT * PAC * P(S)
Table 5: Ranking Functions

Over-generate and rank is an NLG paradigm that has been around since the
beginning of statistical NLG LangkildeKnight98 ; BangaloreRambow2000 , where
multiple candidates Si are first generated, and then ranked to select the best
candidate. Using a single ranking measure at a time can be misleading when the
aim is to optimize multiple aspects of the output krishna2020reformulating .
On the other hand, defining appropriate ranking functions is challenging,
since different aspects of the output can be weighted differently. Here we
treat all aspects as equally important by multiplying our measures, and
examine the effect on the output quality.

Automatic metrics that can be combined for ranking at run time includes
measures of semantic accuracy, the personality classifier probabilities (PAC),
and fluency, calculated as P(S), the probability of candidate S according to
an LLM. We experiment with the five ranking functions in Table 5. RF1 captures
all three components of the desired output, personality match (PAC), semantic
accuracy (SAC) and fluency P(S). Because pbleu is the least correlated metric
with sacc, we add it as an extra term to the ranking function in RF2 to see
whether enforcing lexical similarity improves results. The other three
measures all explore the effect of replacing sacc with a general measure for
semantic similarity. As shown in Table 3, beyond-bleu, bleurt and bertScore
Precision are the most highly correlated with sacc, so we replace sacc with
these three measures in RF3, RF4 and RF5. We calculate P(S) using GPT2-Large
radford2019language .

###  3.3 Prompt Formats, Prompt Sampling, and Prompt Selection

Our experiments test different prompt formats, number of examples, and
sampling methods. We test two discrete prompt formats, using the traditional
Data-to-Text representations (D2T) as well as representations similar to those
used for Textual Style Transfer (TST). The two formats were shown in Table 1,
namely D2T format demonstrates generating an utterance directly from the
meaning representation with a personality token, while the TST format provides
instructions to generate a particular personality from a textual pseudo-
reference of the meaning representation.

Prompt-based learning is restricted by the number of input examples provided:
for Jurassic the maximum is about 36 examples. In addition to varying the
prompt format, we also experiment with the number of examples, as previous
work has shown that also matters soltan-etal-2021-limitations . We test
between 1 and 36 examples.

We also experiment with whether we can create a “control knob” for personality
by presenting all 5 types of personalities in some prompts and only a single
personality in other prompts. Table 1 provides an example of one personality
being used for few shot learning. Experiments using only one personality at a
time used either 10 examples or 36 examples. Experiments using all
personalities used either 1 example per personality (5 total examples) or 6
examples per personality (30 total examples). In these experiments, examples
were randomly selected from the original personage train set where we select
examples given the criteria of the number of examples and the number of
personalities.

In addition, once we determine a good setting for type of prompt and number
and type of examples, we build on previous work using a diversity criterion
for selecting prompts for instruction tuning wang2022self . We hypothesized
that creating prompt examples using a diversity criteria might lead to better
performance. While the instruction tuning work used ROUGE score to select
diverse automatically generated prompts, we use bleurt, and select a set of
prompts that are the least similar according to bleurt. We start with a large
random sample from the training set, and randomly select our first example. We
then calculate the bleurt score between our first example and all the other
examples in our random sample. We greedily then select a second example with
the lowest bleurt score, i.e. with the lowest similarity to the example
already in the pool. We repeat this process, comparing each new candidate to
the examples already in the pool, and selecting the one with the lowest
average bleurt until we reach n number of examples, $T={e_{1},e_{2}...e_{n}}$.
Experiments using this selection process will be called diverse.

Finally, based on our findings on personage, we utilize the best experimental
settings to attempt personality style transfer in an out-of-domain dataset for
Video Games called ViGGO, by providing demonstrations of personalities in the
restaurant domain, with test items from ViGGO test set. juraska2019viggo .

##  4 Results

All experiments use the Jurassic-1 Jumbo 175B parameter PLM, a publicly
available 178B parameter autoregressive language model lieberetal21 ;
levine2020 . Based on tuning experiments with different settings, we set
temperature at .7 and top P at 1.

We first report results comparing the two prompt formats and types of samples
for personage, and then we report results for the five ranking functions.
Finally we report results for personality transfer on the ViGGO corpus, and
present a qualitative analysis of personality expression and diversity.

Prompt Style and Prompt Sampling. Here we compare the two prompt formats,
Data-to-Text (D2T) and Textual Style Transfer (TST) for personage, when
providing examples of either a single personality or examples of all 5
personalities, and varying the number of examples per prompt. Table 1 provided
examples of both the D2T and TST prompt styles and Table6 provides the
experimental results.

ID |  SACC BR |  SACC AR |  PAC BR |  PAC AR |  Perfect BR |  Perfect AR
---|---|---|---|---|---|---
D2T-10-specific |  66.08% |  65.77% |  97.61% |  100.00% |  6.80% |  13.45%
D2T-36-specific |  68.26% |  68.50% |  19.68% |  88.27% |  1.83% |  9.86%
D2T-1-all |  63.24% |  62.12% |  19.53% |  86.26% |  0.98% |  5.90%
D2T-6-all |  69.03% |  68.48% |  19.68% |  88.06% |  1.91% |  11.29%
TST-10-specific |  72.02% |  78.23% |  99.00% |  97.55% |  7.19% |  36.69%
TST-36-specific |  68.07% |  73.72% |  97.00% |  97.63% |  5.19% |  29.21%
TST-1-all |  66.36% |  70.30% |  38.00% |  97.63% |  4.32% |  25.76%
TST-6-all |  70.96% |  75.81% |  56.00% |  97.63% |  6.21% |  35.47%
TST-10-diverse |  67.96% |  78.46% |  98.00% |  100.00% |  9.47% |  38.99%
Table 6: Results comparing Data-to-Text prompts vs. Textual Style Transfer
prompts using RF2 for ranking. Rows indicate both Prompt type and number of
examples in prompt. BR (Before Ranking) metrics reports the average score over
the entire candidate pool of 13900 outputs. AR (After Ranking) reports
performance after selecting the best candidate according to RF2. sacc $=$
Semantic Accuracy. PAC $=$ Personality Accuracy. Perfect reports the
percentage of candidates that are correct for both semantic accuracy and
personality realization.

The top part of Table 6 shows that the D2T prompts consistently perform worse
in every experimental setting, independently of whether examples are provided
of multiple personalities (all) or single personalities (specific) or whether
fewer or more examples are provided. For example, comparing D2T-10-specific to
TST-10-specific, we can see that after ranking, the TST-10-specific setting
(10 examples of a specific personality) provides the best performance with a
semantic accuracy of 78.23% and a stylistic accuracy (PAC) of 99.00%. This
supports our hypothesis that textualizing the data-to-text representations to
make them look more like the natural text that LLMs are trained on would
result in better performance. We also achieve a significantly higher stylistic
accuracy over the whole candidate pool (99.00% PAC BR for TST-10-specific as
compared to 96.71% PAC BR for D2T-10-specific), by instructing the model to
realize the content as a particular personality type, rather than just
demonstrating. Also interestingly, the D2T performance for sacc is the same
before (BR) and after (AR) ranking, while the improvements in sacc after
ranking are large for the TST prompt style. This shows that the overall
quality of the pool of D2T candidates is lower.

The lower part of Table 6 focuses on the TST experiments. These results show
that it is more challenging to get the LLM to learn from diverse prompts how
to do more than one task at a time, i.e. performance is lower when we provide
examples of all personalities, such as TST-1-all (5 examples with one for each
personality) and TST-6-all (30 examples with 6 for each personality).
Moreover, interestingly, performance is better with only 10 examples of a
specific personality, rather than 36 examples. The LLM may find long contexts
such as would be provided with 36 examples more challenging than a shorter
context.

Finally, we compare our random sampling method for our best setting to our
diversity promoting sampling method described in Section 3.3 wang2022self .
The bottom row of Table 6 shows that we get a slight improvement in SACC to
78.46% by sampling more diverse prompts as well as an improvement from 97.55%
to 100% for personality accuracy after ranking (PAC AR). While the difference
in sacc is not significant (p$=$ .28), the difference in PAC AR is significant
(p$=$ 0). We therefore conclude that the diversity selection method is
beneficial.

However compared to SOTA of 99.00% SACC for fine-tuning, the overall
performance for sacc is low harrison2019maximizing . Semantically and
stylistically perfect outputs are also rare. Previous work comments that it is
difficult for a model to simultaneously achieve high stylistic accuracy and
high semantic accuracy. Here personality accuracy is high but there are few
semantically perfect outputs to choose from.

Ranking. We now explore how different ranking functions affect the results.
Table 7 provides the results for the best experimental setting for each
ranking function from Section 3.2. Here, we only experiment with the diverse
prompts, given the results in the last row of Table 6,

ID |  Formula |  ID |  SACC |  PAC |  BLEU
---|---|---|---|---|---
RF1 |  SACC * PAC * P(S) |  TST-10-diverse |  76.56% |  100.00% |  0.235
RF2 |  SACC * PAC *P(S) * pBLEU |  TST-10-diverse |  78.46% |  100.00% |  0.240
RF3 |  pBBLEU * PAC *P(S) |  TST-10-diverse |  65.87% |  100.00% |  0.224
RF4 |  pBLEURT * PAC * P(S) |  TST-10-diverse |  71.61% |  98.20% |  0.213
RF5 |  pBERT * PAC * P(S) |  TST-10-diverse |  63.10% |  100.00% |  0.219
Table 7: Results on personage using all Ranking functions for prompt examples
selected using a diversity criteria (TST-diverse), for the TST-10 best prompt
setting. sacc $=$ Semantic Accuracy. PAC $=$ Personality Accuracy. bleu is
Corpus pbleu.

Row RF2 of Table 7 shows that the addition of the pbleu term to RF1 achieves
higher semantic accuracy (p$=$ 0) and higher bleu, while maintaining the same
stylistic accuracy (PAC) of 100%. We speculate that the addition of the pbleu
term favors outputs whose lexical realizations more closely match the original
MR, enabling the SER script to more easily identify semantically correct
realizations.

Comparing RF3, RF4 and RF5 in the last three rows of Table 7 shows that the
best performing off-the-shelf semantic accuracy function is bleurt, with RF4
performing significantly better than both beyond-bleu and bertScore (p$=$ 0),
although with somewhat lower personality accuracy.

ID |  Formula |  ID |  SACC |  PAC |  BLEU
---|---|---|---|---|---
RF1 |  SACC* PAC * LMPROB |  TST-10-diverse |  86.02% |  96.44% |  0.139
RF2 |  SACC*PAC*LMPROB*PBLEU |  TST-10-diverse |  85.59% |  96.39% |  0.138
RF3 |  BBLEU*PAC*LMPROB |  TST-10-diverse |  86.15% |  96.61% |  0.139
RF4 |  BLEURT*PAC*LMPROB |  TST-10-diverse |  87.58% |  57.06% |  0.168
RF5 |  BERT*PAC*LMPROB |  TST-10-diverse |  85.59% |  96.61% |  0.138
RF1 |  SACC*PAC*LMPROB |  TST-10 |  80.75% |  96.44% |  0.095
RF2 |  SACC*PAC*LMPROB*PBLEU |  TST-10 |  85.44% |  96.00% |  0.113
RF3 |  BBLEU*PAC*LMPROB |  TST-10 |  84.40% |  94.44% |  0.107
RF4 |  BLEURT*PAC*LMPROB |  TST-10 |  95.75% |  55.94% |  0.398
RF5 |  BERT*PAC*LMPROB |  TST-10 |  80.00% |  96.61% |  0.063
Table 8: Results on ViGGO for all Ranking functions for prompt examples
selected using a diversity criteria (TST-diverse) vs. randomly selected
prompts for the TST-10 best prompt setting. sacc $=$ Semantic Accuracy. PAC
$=$ Personality Accuracy. bleu here is Corpus pseudo bleu.

Out of Domain Results for ViGGO. We now turn to our experiments using example
prompts from the personage corpus with meaning representations for the ViGGO
corpus. Our goal is to see whether we can transfer personality style across
domains. Table 2 provided examples of ViGGO MRs and vanilla outputs: there are
no reference utterances for ViGGO outputs with personality. We apply the best
prompt format and number of prompts combination (TST-10) from the personage
experiments on ViGGO, for both the randomly sampled and diverse prompt sets to
test how diverse examples affects generalization across domains.

Personality |  Generated Text |  Reference Text
---|---|---
Agree |  “Hey,” I say. “You want to know more about Marinello’s? Right, right, right,” I say. “It has like, an average rating and it is an Italian place in city centre, also it is a pub, you know, alright?” |  Let’s see what we can find on Marinello’s. Yeah it is an Italian restaurant, family friendly and expensive in the city centre, also it has an average rating, and it is a pub pal, okay? you know.
Disagree |  Oh God it’s an English pub with a relatively moderate price range and it isn’t kid friendly. it is near nearVariable. obviously. |  Damn moderately priced nameVariable is in city centre and it isn’t family friendly, also it’s an English place. It is near nearVariable. It is a pub.
Consc |  Right, let’s see what we can find on nameVariable. I see, it is a restaurant, you know and it isn’t kid friendly, also it is a Italian place, also it is in riverside, and it has a so-so rating. |  Let’s see, nameVariable… Well, I see nameVariable is moderately priced, an Italian restaurant near nearVariable and kid friendly in riverside with a mediocre rating, also it is a pub.
Unconsc |  Yeah, oh God I am not sure. Oh, i mean, oh God. Oh God. Anyway, nameVariable is a fast food restaurant, near nearVariable, also it is in city centre, and it isn’t family friendly. |  Oh gosh mmhm… I don’t know. I mean, nameVariable is a restaurant, also it is in city centre, it is expensive near nearVariable, also it isn’t kid friendly, and it’s a fast food place.
Extra |  There you are, now, let’s see what we can find on nameVariable. Well, nameVariable is a fast food restaurant in city centre, also it is a restaurant, it is moderately priced and it is family friendly. |  nameVariable is moderately priced near nearVariable and family friendly in city centre, it is a fast food place, you know buddy and it is a restaurant!
Table 9: Examples of Generated Text and References for Each Personality. Cues
indicative of each personality type are shown in bold.

Table 8 shows that the results for ViGGO are surprisingly good, and that there
is good personality transfer across domains, with personality accuracies of
97.00%. Interestingly, the upper part of Table 8 shows that the diverse
prompts yield higher overall sacc, suggesting better generalization via
diversity. Row 2 of Table 8 shows that RF3 with diverse prompts has the best
combined performance for sacc and personality accuracy. The RF4 rows in both
the parts of Table 8 show that RF4, using bleurt provides the highest sacc,
and the highest pbleu score, but unacceptably low PACs of 57.00% and 56.00%.
Perhaps the bleurt metric ranks candidates lower that manifest personality. A
paired t-test shows that RF3 performs significantly better than RF1 (p $=$ 0)
for sacc, even though bleurt was most highly correlated with sacc. Again, sacc
is low compared to the fine-tuning SOTA of 99.2% juraska-walker-2021-attention
.

Qualitative Analysis. Table 9 provides generation outputs for each personality
along with their reference texts for the restaurant domain, while Table 10
provides Viggo generation outputs when conditioned on personality utterances
from personage. In both tables we mark in bold the linguistics markers of
personality for each personality type mairesse2010towards . Table 9
illustrates how the LLM generalizes from the examples given for each
personality to produce similar markers that would not have been seen in the
demonstrations, for example the “Hey” formulation for agreeableness in the
first row is completely novel, as is the “There you are, …” formulation for
extraversion in the last row. Table 10 illustrates the differences between the
vanilla reference sentences for Viggo and the outputs that have been
personality conditioned, with many of the basic linguistic markers appearing
in the ViGGO outputs.

Personality |  Generated Text |  Reference Text
---|---|---
Agree |  I personally think that Call of Duty: Advanced Warfare 2014 is an average first person shooter, with pretty good graphics, alright? |  For a 2014 game, I think Call of Duty: Advanced Warfare wasn’t so fun to play from the first person perspective.
Disagree |  Oh God, I recommend Assetto Corsa, it’s basically a first person Steam game, damn near perfect. |  Assetto Corsa is a first person title on Steam. Are you familiar with it?
Consc |  Let’s see what we can find on Little Big Adventure. Well, it seems that Little Big Adventure is a game and has a mediocre rating, also it isn’t multiplayer, also it is for the PlayStation. |  I remember you saying you found Little Big Adventure to be average. Are you not usually that into single-player games on PlayStation?
Unconsc |  Mmhm… I know, er… F1 2014 is a driving/racing game, also it is a simulation, also it is a sport multiplayer, also… |  If you’re looking for a multiplayer sport racing sim, you can’t go wrong with F1 2014. Are you familiar with it?
Extra |  My opinion is that Guitar Hero: Smash Hits is a poor first person game, you know! |  I wanted to like Guitar Hero: Smash Hits but honestly, it was just an underwhelming first person game.
Table 10: ViGGO Personality Outputs with Reference utterances from ViGGO

##  5 Discussion and Conclusion

We tested two types of discrete prompts for stylistically and semantically
controlled NLG, and show treating data-to-text as a text-to-text task performs
better for both semantic and stylistic accuracy. To our knowledge these are
the first results testing prompt-based learning for simultaneously controlling
both semantics and style.

We varied the number of prompt examples, and the sampling of examples, to
either sample multiple personalities or single personalities, and comparing
random sampling to a sampling to encourage diversity. We found that examples
illustrating multiple personalities in the same prompt produces worse
performance rather than encouraging generalization. In addition, we find that
selecting sample demonstrations using a diversity criterion improves both
semantic and stylistic accuracy as well as stylistic transfer to the video
games domain.

We also surprisingly get lower overall semantic accuracies when prompting and
testing with restaurant examples than we do when prompting with restaurant
examples and testing with video game MRs. We speculate that this may be due to
the delexicalization of the restaurant name in the personage corpus. personage
is based on synthetic MRs created for the E2E generation challenge. This means
that the MRs do not describe real restaurants, while the ViGGO MRs do
correspond to real video games. Thus ViGGO realizations can benefit from the
knowledge that the LLM has about video games. In future work, we hope to test
whether using MRs that correspond to real restaurants improves semantic
accuracy.

This work has several limitations. One limitation is the overall semantic
accuracy performance for both restaurants and video games. These are lower
than fine-tuned models and thus, in a real setting, fine-tuned models would
still need to be used, as well as the fact that the Jurassic model cannot be
run in real time. Both of these limitations might be addressable by
instruction tuning a smaller model for data-to-text and stylistic control
tasks such as we report here ouyang-etal-2021-dialogue ; wang2022self .
Another limitations is that we only tested our approach on two domains, and
only on five personality styles.

## References

  * [1] S. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-david, C. Xu, G. Chhablani, H. Wang, J. Fries, M. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, D. Radev, M. T.-j. Jiang, and A. Rush.  PromptSource: An integrated development environment and repository for natural language prompts.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 93–104, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [2] S. Bangalore and O. Rambow.  Exploiting a probabilistic hierarchical model for generation.  In Proc. of the 18th Conference on Computational Linguistics, pages 42–48, 2000.
  * [3] D. Chen and W. B. Dolan.  Collecting highly parallel data for paraphrase evaluation.  In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190–200, 2011\.
  * [4] A. Furnham.  Language and personality.  In H. Giles and W. Robinson, editors, Handbook of Language and Social Psychology. Winley, 1990.
  * [5] V. Harrison, L. Reed, S. Oraby, and M. Walker.  Maximizing stylistic control and semantic accuracy in nlg: Personality variation and discourse contrast.  DSNNLG 2019, page 1, 2019.
  * [6] V. Harrison and M. Walker.  Neural generation of diverse questions using answer focus, contextual and linguistic features.  In Proceedings of the 11th International Conference on Natural Language Generation, pages 296–306, 2018.
  * [7] P. Heidari, A. Einolghozati, S. Jain, S. Batra, L. Callender, A. Arun, S. Mei, S. Gupta, P. Donmez, V. Bhardwaj, et al.  Getting to production with few-shot natural language generation models.  In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 66–76, 2021.
  * [8] P. Heidari, A. Einolghozati, S. Jain, S. Batra, L. Callender, A. Arun, S. Mei, S. Gupta, P. Donmez, V. Bhardwaj, A. Kumar, and M. White.  Getting to production with few-shot natural language generation models.  In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 66–76, Singapore and Online, July 2021. Association for Computational Linguistics.
  * [9] D. Jin, Z. Jin, Z. Hu, O. Vechtomova, and R. Mihalcea.  Deep learning for text style transfer: A survey.  Computational Linguistics, 48(1):155–205, Mar. 2022.
  * [10] D. Jin, Z. Jin, Z. Hu, O. Vechtomova, and R. Mihalcea.  Deep learning for text style transfer: A survey.  Computational Linguistics, 48(1):155–205, 2022.
  * [11] J. Juraska.  Diversifying language generated by deep learning models.
  * [12] J. Juraska, K. K. Bowden, and M. Walker.  ViGGO: A video game corpus for data-to-text generation in open-domain conversation.  In Proceedings of the 12th International Conference on Natural Language Generation, 2019.
  * [13] J. Juraska, P. Karagiannis, K. Bowden, and M. Walker.  A deep ensemble model with slot alignment for sequence-to-sequence natural language generation.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 152–162, 2018.
  * [14] J. Juraska and M. Walker.  Attention is indeed all you need: Semantically attention-guided decoding for data-to-text NLG.  In Proceedings of the 14th International Conference on Natural Language Generation, pages 416–431, Aberdeen, Scotland, UK, Aug. 2021. Association for Computational Linguistics.
  * [15] K. Krishna, J. Wieting, and M. Iyyer.  Reformulating unsupervised style transfer as paraphrase generation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, 2020.
  * [16] I. Langkilde and K. Knight.  Generation that exploits corpus-based statistical knowledge.  In Proc. of COLING-ACL, 1998.
  * [17] Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua.  Limits to depth efficiencies of self-attention.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22640–22651. Curran Associates, Inc., 2020.
  * [18] B. Li, Y. Weng, Q. Song, F. Ma, B. Sun, and S. Li.  Prompt-based pre-trained model for personality and interpersonal reactivity prediction.  In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 265–270, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [19] J. Li, T. Tang, J.-Y. Nie, J.-R. Wen, and X. Zhao.  Learning to transfer prompts for text generation.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3506–3518, Seattle, United States, July 2022. Association for Computational Linguistics.
  * [20] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online, Aug. 2021. Association for Computational Linguistics.
  * [21] O. Lieber, B. L. Sharir, and Y. Shoham.  Jurassic-1: Technical details and evaluation.  Technical report, AI21 Labs, 2021.
  * [22] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  2021\.
  * [23] Y. Liu, G. Neubig, and J. Wieting.  On learning text style transfer with direct rewards.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4262–4273, Online, June 2021. Association for Computational Linguistics.
  * [24] F. Mairesse and M. Walker.  Personage: Personality generation for dialogue.  In In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL, pages 496–503, 2007.
  * [25] F. Mairesse and M. A. Walker.  Towards personality-based user adaptation: psychologically informed stylistic language generation.  User Modeling and User-Adapted Interaction, 20:227–278, 2010.
  * [26] F. Mairesse and M. A. Walker.  Controlling user perceptions of linguistic style: Trainable generation of personality traits.  Computational Linguistics, 2011.
  * [27] F. Mairesse and M. A. Walker.  Controlling user perceptions of linguistic style: Trainable generation of personality traits.  Computational Linguistics, 37(3):455–488, Sept. 2011.
  * [28] S. Oraby, V. Harrison, A. Ebrahimi, and M. Walker.  Curate and generate: A corpus and method for joint control of semantics and style in neural nlg.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5938–5951, 2019.
  * [29] S. Oraby, L. Reed, T. Sharath, S. Tandon, and M. Walker.  Neural multivoice models for expressing novel personalities in dialog.  Proc. Interspeech 2018, pages 3057–3061, 2018.
  * [30] S. Oraby, L. Reed, S. Tandon, S. T. S., S. M. Lukin, and M. A. Walker.  Controlling personality-based stylistic variation with neural natural language generators.  CoRR, abs/1805.08352, 2018.
  * [31] S. Ouyang, Z. Zhang, and H. Zhao.  Dialogue graph modeling for conversational machine reading.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3158–3169, Online, Aug. 2021. Association for Computational Linguistics.
  * [32] K. Papineni, S. Roukos, T. Ward, and W. jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  pages 311–318, 2002.
  * [33] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
  * [34] J. W. Pennebaker and L. A. King.  Linguistic styles: Language use as an individual difference.  Journal of Personality and Social Psychology, 77:1296–1312, 1999\.
  * [35] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [36] L. Reed, V. Harrison, S. Oraby, D. Hakkani-Tur, and M. Walker.  Learning from mistakes: Combining ontologies via self-training for dialogue generation.  In Proceedings of the 21st Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL 2020), 2020.
  * [37] L. Reed, C. Li, A. Ramirez, L. Wu, and M. A. Walker.  Jurassic is (almost) all you need: Few-shot meaning-to-text generation for open-domain dialogue.  CoRR, abs/2110.08094, 2021.
  * [38] L. Reed, S. Oraby, and M. Walker.  Can neural generators for dialogue learn sentence planning and discourse structuring?  INLG 2018, page 284, 2018.
  * [39] L. Reed, S. Oraby, and M. Walker.  Can neural generators for dialogue learn sentence planning and discouse structuring?  In Proceedings of the 11th International Conference on Natural Language Generation, pages 284–295, 2018.
  * [40] E. Reif, D. Ippolito, A. Yuan, A. Coenen, C. Callison-Burch, and J. Wei.  A recipe for arbitrary text style transfer with large language models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 837–848, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [41] T. Sellam, D. Das, and A. P. Parikh.  BLEURT: learning robust metrics for text generation.  CoRR, abs/2004.04696, 2020.
  * [42] S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky, et al.  Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model.  arXiv preprint arXiv:2208.01448, 2022.
  * [43] S. Soltan, H. Khan, and W. Hamza.  Limitations of knowledge distillation for zero-shot transfer learning.  In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 22–31, Virtual, Nov. 2021. Association for Computational Linguistics.
  * [44] M. Suzgun, L. Melas-Kyriazi, and D. Jurafsky.  Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models.  arXiv preprint arXiv:2205.11503, 2022.
  * [45] A. Tsai, S. Oraby, V. Perera, J.-Y. Kao, Y. Du, A. Narayan-Chen, T. Chung, and D. Hakkani-Tur.  Style control for schema-guided natural language generation.  In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 228–242, Online, Nov. 2021. Association for Computational Linguistics.
  * [46] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi.  Self-instruct: Aligning language model with self generated instructions.  arXiv preprint arXiv:2212.10560, 2022.
  * [47] A. Webson and E. Pavlick.  Do prompt-based models really understand the meaning of their prompts?  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States, July 2022. Association for Computational Linguistics.
  * [48] T.-H. Wen, M. Gasic, N. Mrksic, P.-H. Su, D. Vandyke, and S. Young.  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.
  * [49] J. Wieting, T. Berg-Kirkpatrick, K. Gimpel, and G. Neubig.  Beyond BLEU:training neural machine translation with semantic similarity.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344–4355, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [50] J. Xiang, Z. Liu, Y. Zhou, E. P. Xing, and Z. Hu.  Asdot: Any-shot data-to-text generation with pretrained language models.  arXiv e-prints, pages arXiv–2210, 2022.
  * [51] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi.  Bertscore: Evaluating text generation with bert.  arXiv preprint arXiv:1904.09675, 2019.