# Sector Rotation by Factor Model and Fundamental Analysis

Runjia Yang  University of California, Davis  Beining Shi  University of
California, Davis

(Sept 2023)

## Abstract

This study presents an analytical approach to sector rotation, leveraging both
factor models and fundamental metrics. We initiate with a systematic
classification of sectors, followed by an empirical investigation into their
returns. Through factor analysis, the paper underscores the significance of
momentum and short-term reversion in dictating sectoral shifts. A subsequent
in-depth fundamental analysis evaluates metrics such as PE, PB, EV-to-EBITDA,
Dividend Yield, among others. Our primary contribution lies in developing a
predictive framework based on these fundamental indicators. The constructed
models, post rigorous training, exhibit noteworthy predictive capabilities.
The findings furnish a nuanced understanding of sector rotation strategies,
with implications for asset management and portfolio construction in the
financial domain.

Keywords: US Industrial Sectors, Factor Analysis, Fundamental Analysis,
Trading Strategy.

##  1 Introduction

Sector is composed by a basket of stocks that representing companies in
certain business class, which has unique features according to the business.
Under certain conditions, such as economic cycles, sectors may behave
accordingly due to the different characteristics of businesses. In this
report, we are exploring how to capture returns by finding the hidden features
behind different sectors and determining the leading sectors in some
particular market conditions or social environments. Generally, this report
covers a brief exploration of market and fundamental factors, explaining the
meaning of each factors and how they are related to some sectors.Then we
applied a neural network model to do a classification and prediction using the
fundamental factors as inputs. At the end of the report, we also covers how
sectors behaved under global events.

##  2 Sector Classification and Return Analysis

There are many different ways to divide sectors. For the purpose of common
acceptance and convenience for future data acquirement, we used the MSCI
Global Industry Classification Standard, which includes 11 level one sectors,
24 level two industry groups, 69 level three industries, and 158 sub-
industries. We use the 11 level one sectors as our main target. They are
Energy, Materials, Industrials, Consumer Discretionary, Consumer Staples,
Health Care, Financials, Information Technology, Communication Services,
Utilities, and Real Estate. In order to track the performance of each sector,
we use the S$\&$P500 GICS Indices which are constructed exactly as the MSCI
classification.Before working on any strategies further, we need to determine
if there are actually possible profits. In our case, we need to check how big
the differences between sectors’ returns are. For each observation time
period, Define: Return Difference =($\sum_{\text{top 3}}$ Sector Return -
$\sum_{\text{bottom 3}}$ Sector Return )/3 Based on a monthly frequency, we
calculate the return difference and get the following plot.

Also, by calculation, the quarterly return difference has a mean of 0.1306,
median of 0.1185, standard deviation of 0.0523. We can reach to a result that
half of the quarterly return difference is more than 11.85$\%$. It is easy to
see that there does exist potential investment opportunity by capturing the
return differences between sectors.

##  3 Factor Analysis

###  3.1 Momentum Factor Exploration

Momentum premium was first recognized by UCLA scholars Narasimhan Jegadeesh
and Sheridan Titman in 1993. The momentum premium is established on the
observation that assets that have performed well in the past have the trend to
persist good performance in the future. Though the momentum effect is
considered to be a market anomaly, it has been recognized widely among many
asset classes. We will explore the momentum effect based on the sector indices
introduced above.

####  3.1.1 Factor Construction

First of all, we need to construct the momentum factor. Typically, the
momentum factor is constructed by the past 6 or 12 months cumulative return
and excludes the most recent month’s return, considering that there are also
short-term reversion effects based on the mean-reversion effects. However,
without a clear idea of how the sector indices carry the momentum effect, we
need to explore through time intervals to find the best possible momentum
factor. Then we constructed 12 different momentum factors using the past 1 to
12 month’s return and excluding the most recent 0.1 portion trading days of
each time period to avoid short-term reversion. For each of the factors with
the period of n months

$MOM\\_{nM}=\sum_{\text{Past}n*21\text{ days}}R_{d}-\sum_{\text{Past
}0.1*n*21\text{ days}}R_{d}$

where $R_{d}$ is the daily return.

####  3.1.2 Calculate Factor Returns

For the 12 factors we got, we normalized them cross sections. Then we rank the
factor exposures for each sector and take long positions of sectors with the
highest two factor exposures, take short position of sectors with the lowest
two factor exposures. Then we trade our portfolio under a monthly frequency.
Here are the results from 2002 to 2022 February:

Factor | Factor Return | Sharpe Ratio
---|---|---
MOM_1M | $-0.0297$ | $-0.09$
MOM_2M | $-0.0583$ | $-0.19$
MOM_3M | $0.0082$ | $0.02$
MOM_4M | $-0.0245$ | $-0.08$
MOM_5M | $-0.0136$ | $-0.04$
MOM_6M | $0.0195$ | $0.06$
MOM_7M | $0.0535$ | $0.15$
MOM_8M | $0.0721$ | $0.21$
MOM_9M | $0.0413$ | $0.11$
MOM_10M | $-0.0470$ | $-0.13$
MOM_11M | $0.0371$ | $0.10$
MOM_12M | $0.0327$ | $0.01$

Since there are several market crashes where the momentum factor led to
negative returns, we also take a look at the most recent five years from 2017
to 2022 February:

Factor | Factor Return | Sharpe Ratio
---|---|---
MOM_1M | $0.0213$ | $0.06$
MOM_2M | $-0.0494$ | $-0.14$
MOM_3M | $0.1154$ | $0.29$
MOM_4M | $-0.1578$ | $0.41$
MOM_5M | $-0.1788$ | $0.47$
MOM_6M | $0.1163$ | $0.35$
MOM_7M | $0.2119$ | $0.62$
MOM_8M | $0.1869$ | $0.56$
MOM_9M | $0.1583$ | $0.43$
MOM_10M | $0.0504$ | $0.12$
MOM_11M | $0.1547$ | $0.40$
MOM_12M | $0.1289$ | $0.33$

From this table, we can tell that by using the MOM$\\_$7M factor, we can reach
a maximum annual return rate of 21.19$\%$ and a maximum Sharpe ratio of 0.62.
It is also interesting that we find the momentum factor with a short time
period, for example, MOM$\\_$1M and MOM$\\_$2M, have a very small even
negative return rate. However, it exactly conforms to the short term reversion
effect that the typical momentum factor would exclude.

###  3.2 Short Term Reversion Factor Exploration

Short term reversion factor follows the simple principle that asset’s price
will have the trend to stay on an average level. Since we can see from the
previous results of the momentum factor that there does exist short term
reversion effect, we can try different reversion factors and find out what
would be the best short term reversion observation period.

####  3.2.1 Factor Construction

Similarly, we can define several reversion factors with different time
periods. And we take the negative number of the past n days cumulative return
as the factor exposures.

$REV\\_{nD}=-\sum_{\text{Past }n\text{ days}}R_{d}$

For the purpose of exploring the optimal time period, we take 5-day time
interval and create 12 reversion factors from 5 trade days to 55 trade days.

####  3.2.2 Calculate Factor Returns

By using the same method, we compute the rank of each sector’s factor
exposure, and long the top two sectors, short the last two sectors on a
monthly observation frequency. Between 2002 and 2022 February, the results
are:

Factor | Factor Return | Sharpe Ratio
---|---|---
Rev_5D | $-0.0059$ | $-0.0597$
Rev_10D | $0.0294$ | $0.3320$
Rev_15D | $0.0127$ | $0.1338$
Rev_20D | $0.0072$ | $0.0715$
Rev_25D | $0.0690$ | $0.7768$
Rev_30D | $0.0877$ | $0.8735$
Rev_35D | $0.0135$ | $0.1288$
Rev_40D | $-0.0314$ | $-0.3422$
Rev_45D | $-0.0694$ | $-0.8417$
Rev_50D | $-0.0183$ | $-0.5919$
Rev_55D | $-0.0517$ | $-0.1863$

From this table, we can tell that for the time between 2002 to recent time,
the short term reversion effect is optimal for taking the past 30 days
cumulative return. It has an optimal annual return rate of 8.77$\%$ on average
and leads to a sharp ratio of 0.8735.

##  4 Fundamental Analysis

Fundamental Analysis are always a good aspect to look at for investing. We
collected quarterly data for all 11 indices from Bloomberg, including their
P/E ratio, EV/EBIT, Profit Margin, etc. Our fundamental analysis would start
from discovering features for each of the fundamental ratio,then we are trying
to predict the sector performance by constructing using some of the features
we found.

###  4.1 PE Ratio

The P/E is one of the most widely used tools to determine a stock’s relative
valuation. The purpose of analyzing the ratio is to show whether certain
sector is worth to be invested because P/E ratio can reflect the investment
risk in this sector. The figure below shows distribution of P/E ratios in
different sectors.

By comparing cross-sectional data, it is obvious that P/E of Real Estate
Sector and Consumer Discretionary Sector are higher than others. The reason is
that earning growth in the future is expected to grow fast in the two sectors
or these sectors have some special advantages that guarantee long-term
profitability with low risk. On the other hand, Financials Sector’s ratio is
relatively low compared with other sectors, which may result from its high
volatility so investors are reluctant to pay for it.
We also notice that the ratio in Energy Sector surged in 2015, which is
related to some changes in the sector. The end of the oil age and emergence of
alternative energy have reduced the earnings of the original sector. As a
result, its relative price becomes higher than before. The change in EV/EBIT
and EV/EBITDA is also due to this reason.

###  4.2 PB Ratio

The P/B ratio provides a valuable reality check for investors who are seeking
growth at a reasonable price. For those sectors with more assets, their book
value and market value are close, so P/B ratio is more useful when we analyze
Real Estate sector and Financials sector. The figure below shows distribution
of P/B ratio in different sectors.

As the picture shows, Consumer Discretionary sector and IT sector have higher
P/B ratio while Financials sector and Energy sector have relatively low
ratios. What’s more, Real Estate sector with high P/E ratio has relatively
lower P/B ratio.

###  4.3 EV/Sales

EV/Sales can help investors better understand cost relative to unit sales and
whether the company is overvalued or undervalued. If EV/Sales is relatively
high, the company or sector is less attractive to investors. The figure below
shows distribution of EV/Sales in different sectors.

The result shows that Real Estate sector’s ratio is higher than others’, which
means that index in this sector is overvalued. On the other hand, ratio in
Energy sector is low, which can attract more investors.

###  4.4 EV/EBIT $\&$ EV/EBITDA

EV/EBIT and EV/EBITDA are independent of the capital structure of the company,
whereas multiples like P/E ratio are impacted by financing decisions. Because
of this reason, the two are the most commonly relied-upon multiples in
relative valuation. However, one obvious distinction is that EV/EBIT considers
depreciation and amortization. In some capital-intensive industries which have
significant differences in D$\&$A, EV/EBIT may make it a more accurate measure
of value. But in our analysis, there is no such difference in the comparison
of these two ratios under different sectors.

The results can show that Real Estate sector has a higher ratio. The three
ratio EV/Sales, EV/EBIT and EV/EBITDA can give a consensus conclusion that
Real Estate sector is overvalued in the market.

###  4.5 Dividend Yield

Dividend Yield is used to measure the amount of cash flow investors are
getting back for each dollar. It is essentially the return on investment for a
stock without any capital gains. The figure below shows distribution of
Dividend yield in different sectors.

The ratio in Communication Services sector is higher before 2018 while Energy
sector’s ratio is higher after that time. This is because communication
services sector took place a reorganization of S$\&$P500 index in 2018. It now
includes at least eighteen companies from IT and Consumer Discretionary
sectors. Due to this reshuffling, dividend yield of this sector is impacted.

###  4.6 Gross Margin

Gross margin equals net sales less the cost of goods sold (COGS). Net sales
are equivalent to the total revenue from sales, and COGS is the direct cost
associated with producing goods. By calculating gross margin, we could measure
one company’s retain revenue after subtracting the production cost. The higher
the gross margin, the more capital a company retains, which it can then use to
pay other costs or satisfy debt obligations. Generally, companies with good
gross margins would have a relatively sustainable competitive advantage. By
analyzing gross margin data across sectors, we may observe some sectors that
have more stable development in the long run.

For our 11 sectors’ gross margin data, the line chart above shows significant
differences between the sectors. Overall, each industry index is relatively
flat on its own, and have gaps between each others. Utilities, Communication
Services and Information Technology(IT) have been among the top spears for
last 10 years, occupying the first, second and third positions respectively,
all above 40$\%$. On the contrary, the energy sector has been an under-
performer for the past decade, ranking at the bottom, with gross margins
consistently below 20$\%$. Gross margins in the rest industries are
concentrated in the 25$\%$-35$\%$ range and have not fluctuate much.
At the same time, by observing the comparison of fluctuations between
industries, it is not difficult to see that the gross margin fluctuations of
the energy industry and the utilities industry maybe relatively high in the
past decade, and their peaks correspond to each other. During 2016, the
utilities industry grew significantly, while energy declined comparatively.
The trend was even more pronounced in 2020, with utilities reaching its
highest level and the energy industry fell to the bottom.
Generally, the gross margin feature maybe a significant indicator for
Utilities, Communication Services and IT sectors. And our conjecture about the
correlation between utilities and energy sectors will need further observation
and verification.

###  4.7 Operating Margin $\&$ Profit Margin:

Operating margin equals operating income divided by revenue, it is a
profitability ratio measuring revenue after covering operating and non-
operating expenses of a business. And profit margin measures the profit ratio
after paying for variable costs of production. It is calculated by the
formula:

$\text{Profit Margin}=\frac{(\text{Total Revenue - Total Expenses
)}}{\text{Total Revenue}}$

Both operating margin and profit margin are used to gauge the degree of the
company’s activity makes money. Higher ratios are generally better,
illustrating the company is efficient in its operations and is good at turning
sales into profits. In our analysis, there is not a very big difference in the
comparison of these two ratios under different sectors, which is determined by
their definition.

For these two ratios, Real Estate sector, IT sector and Financial sector have
the top three high ratios.And Energy sector has the relatively lowest ratio.
Also, both operating margin and profit margin for almost all sectors have
similar trends in the last decade curves. This is attributed to the definition
difference between the two features, and that’s why the operating margin was
slightly higher than the profit margin.
Another thing that is worth to mentioning is that for Energy Sector, not just
operating margin and profit margin, but also the gross margin, it always has
the relatively lowest ratios and similar curve fluctuation, with sharp
declines in 2016 and 2020. The two time nodes may consistent with some big
revolution in the energy industry, which we will analyze later.

###  4.8 Return on Asset $\&$ Return on Equity

Return on equity (ROE) and return on assets (ROA) are two of the most
important measures for evaluating how effectively a company’s management team
is doing its job of managing the capital entrusted to it. ROE equals to
generally net income divided by equity, while Return on Assets (ROA) is net
income divided by average assets. So the primary differentiator between ROE
and ROA is financial leverage or debt. ROE measures profitability and ROA is
an efficiency measure of how well a company is using its assets. Investors may
prefer to observe ROE, since equity represents the owner’s interest in the
business. Compared to other sources of fund, equity capital tends to be the
most expensive source of funding and carries the largest risk premium of all
financing options. Therefore, in our analysis, ROE may be a better feature
that it could reflect the trend of market investment.

As shown in the picture, IT sector has the highest ROA, the Consumer Staples
sector and Consumer Discretionary sector also have a relatively higher ratio.
In contrast, Financial sector has a lower ROA. The past ten years, or even
twenty years, has been an era of rapid development of information technology.
And compared with traditional industry and commerce, information technology is
more flexible in the time and form of investment assets, that’s the reason why
IT will have the highest ratio. Also for the the Consumer Staples sector and
Consumer Discretionary sector,they are all industries with fast innovation and
short production cycle. Generally, these three will have constantly higher
ratio for the long run. Therefore, for these three industries, if the ROA
indicator fluctuates significantly, it may have an impact on the investment
trend.

For ROE ratio, similarly, IT, Consumer Staples stay high, and Consumer
Discretionary sectors is also at a slightly higher level, except that the IT
sector lost its prominence in ROA ratio. By comparing cross-sectional data,
the Consumer Discretionary Sector and Industrials Sector have similar patterns
in the last decade for both ROA and ROE ratios. They both have a low peak in
2020. It is conceivable that this is affected by the general environment of
the epidemic. And as we mentioned before, the ROE and ROA curves of the energy
sector still have a similar pattern, falling sharply in 2016 and 2020. In
2016, it was affected by changes in energy policy since 2015, reducing oil
production while encouraging the development of clean and new energy. For
2020, we attribute this decline to the outbreak of the COVID-19 pandemic.

##  5 Prediction by Fundamental Factors

###  5.1 Factor and Future Return

Having these fundamental data, next step is to find out what quantitative
relationships they have to futures sector returns. For fundamental factors,
they are usually exposed in the company report with annual, semi-annual, or
quarterly frequency. Our fundamental factors for each sector are reported
quarterly, leading to a problem that the sample size for each individual
sector is very small. To have a better performance of the prediction model, we
need to combine all the sectors together and make a uniformed and comparable
large sample. We neutralized each factor cross-sectional for the factor to
have a mean of 0 and standard deviation of 1. If $X_{i,t}$ denotes one
specific factor exposure for $i$-th sector at time $t$, in this case would be
at $t$-th quarter, then for each individual $t$ we have the neutralized
exposure to be:

$X_{i,t}\text{ -Neutral}=\frac{X_{i,t}-\text{Mean}(X_{i,t}\text{ ,}i\text{
from }1\text{ to }11)}{\text{Standard Deviation}(X_{i,t}\text{ ,}i\text{ from
}1\text{ to }11)}$

Then we used the next quarter’s cross-sectional normalized return as the
corresponding return. First, we want to have a general view of the relations.
The scatter plots between neutralized factors and future returns are as
following:

From the scatter plots, the relations between all factors and their future
returns cannot be well interpreted by simple linear models. However, it is
very common in the financial field that the sample will have a very low
signal-noise ratio.

###  5.2 Model Construction and Training

As we observed before, the relations between each factor and its future return
cannot be interpreted very well by linear models. Also, we have no idea what
model would exactly best fit the data. Therefore, converting prediction of
future returns to a classification problem and fitting the training sample
with a neural network model which has comparably good performance with non-
linear relations would be a great start point.

[height=10] [count=10, bias=false, title=Input

layer, text=$x_{\hiddenlayer}$[count=5, bias=false, title=Hidden

layer 1, text=$h^{(1)}_{\linklayers}$[count=5, bias=false, title=Hidden
layer 2, text=$h^{(2)}_{\linklayers}$[count=2, title=Output
layer, text=$\hat{y}_{\linklayers}$

Neural network takes a vector as the input, and goes to each of the neuron in
the first hidden layer and gains new activation vectors which act as the input
for next hidden layer. After the last hidden layer, neural network model would
pass out the probability for each of the prediction class and we choose the
one with the highest probability as the prediction. This process is called
front propagation. After comparing the prediction to the actual results, we
adjust the weights of the nodes by using back propagation for each training
pair in the training samples. Also, we use the rectified linear unit function
as the activation function for hidden layers and sigmoid function as the
activation function for final output. Since we only have a sample of size 200,
choosing quasi-Newton methods as the solver has better performance for small
sample training. Then we need to construct the training, validation, and test
sets. Since the fundamental factors are already neutralized (normalized)
within each sector, we divide the sample data to 60$\%$, 20$\%$, 20$\%$ by
convention. Without shuffling, we will have the historical data divided where
test set contains the most recent data. For the corresponding output value, we
assign 1 to samples with positive future return and 0 with negative returns.
The complexity of neural network directly related to the number and sizes of
hidden layers. For the purpose of avoiding overfitting or under-fitting, we
need to find proper hyper parameters for neural network model. We start from a
simple model with two layers. Let $N$ denote the number of nodes in each
hidden layer, alpha is the hyper parameter for L2 regularization penalty
function. With larger $N$, the model is more complex. If alpha increases, the
penalty for large weights increases, which makes the model tend to be more
simple. Considering our sample size is small, intuitively we need to focus
more on the overfitting problem. For a range of alpha and $N$, we train the
model using the training set data, and get the score for prediction on
validation set. The score represents the probability of making a right
prediction. Here are the results:

N$\backslash$alpha | 3 | 2.5 | 2 | 1.5 | 1 | 0.5 | 0.25 | 0.1 | 0.01 | 0.001
---|---|---|---|---|---|---|---|---|---|---
5 | 0.49 | 0.45 | 0.49 | 0.56 | 0.62 | 0.58 | 0.62 | 0.58 | 0.56 | 0.56
6 | 0.51 | 0.55 | 0.45 | 0.49 | 0.51 | 0.49 | 0.47 | 0.56 | 0.56 | 0.56
7 | 0.55 | 0.53 | 0.56 | 0.51 | 0.47 | 0.51 | 0.58 | 0.49 | 0.53 | 0.57
8 | 0.51 | 0.47 | 0.55 | 0.51 | 0.51 | 0.64 | 0.6 | 0.6 | 0.59 | 0.6
9 | 0.53 | 0.49 | 0.47 | 0.53 | 0.56 | 0.56 | 0.55 | 0.6 | 0.53 | 0.6
10 | 0.55 | 0.53 | 0.53 | 0.53 | 0.51 | 0.51 | 0.53 | 0.6 | 0.6 | 0.56
11 | 0.53 | 0.58 | 0.51 | 0.51 | 0.47 | 0.65 | 0.58 | 0.55 | 0.56 | 0.51
12 | 0.55 | 0.55 | 0.56 | 0.51 | 0.49 | 0.51 | 0.54 | 0.53 | 0.53 | 0.53
13 | 0.53 | 0.47 | 0.55 | 0.49 | 0.56 | 0.6 | 0.55 | 0.58 | 0.6 | 0.56
14 | 0.49 | 0.51 | 0.56 | 0.53 | 0.55 | 0.53 | 0.49 | 0.51 | 0.6 | 0.53
15 | 0.47 | 0.51 | 0.51 | 0.51 | 0.45 | 0.49 | 0.56 | 0.6 | 0.56 | 0.55

To better understand how the hyperparameters influence model performance, we
visualize the data by using $N$ and alpha as the bottom coordinates, and use
the corresponding probability as the height.

From the figure, we can tell that the model have several local optimal pairs.
And the scores at the optimal points with relative large N values are also
combined with small alpha values. For example, the combination of 14 nodes and
alpha equals 0.01 has a local optimal score of 0.6. Since we are training with
a small sample, using such a complex model with a high score is highly likely
overfitting. Therefore, we start from the simple model by looking at models
with 5 nodes model and check how the score varies with alpha. For model with 5
nodes, we see there are two local peaks with alpha equal to 1 and 0.25, then
we pick the middle value 0.5 as the value of alpha considering the trade-off
between variance and bias.

###  5.3 Model Performance

Constructed and trained the model, next we would test the model by feeding a
new data set to the model. On the test set, the score of the model is 0.64,
which means the model predicts 64$\%$ of the results correctly. More detailed
results are showed in the following table:

| Positive Return | Negative Return
---|---|---
Actual Sample | 27 | 28
Prediction | 37 | 18
Correct Prediction | 22 | 13
Winning Rate | 0.59 | 0.72

On the test set, we have a 0.59 winning rates on the positive predictions and
0.72 on the negative predictions, which gives an overall winning rate of 0.64.

###  5.4 Trade with Model Prediction

By using the predictions from the trained model, we used the data from
validation set to get trade signals. Instead of having signals of 1 or 0 as
the model’s output, we choose the probability of the prediction output being
1, which is given by the activation sigmoid function. Then we will have a time
series of the probability for each sector, and rank the probability from
highest to lowest where the highest probability will have a rank 1. For each
cross-sectional ranking, we equally-weighted long sectors with rank 1 to 3 and
short sectors with rank 9 to 11 to construct a dollar-neutral portfolio. On
the test set, which is from September in 2020 to September in 2021, we have a
Sharpe ratio of 2.21. The cumulative return plot is following:

###  5.5 Other Considerations

There are still issues that need to be considered carefully in the future.
First is the factor neutralization. In previous model, we neutralized the
factor exposure cross-sectionally, where the exposures reflect the relative
level of factor exposure for one sector compared to other sectors at a given
time. However, different sectors may have inner trends of higher exposures
than others for some factors, especially for fundamental factor. What’s more,
we only have quarterly fundamental data available from 2017 and it is hard to
implement time series normalization for each sector. Therefore, how to modify
the factor exposures to make them comparable is a difficult problem. Secondly,
as the sample size is small, the model might not be applicable on a wider
range of time since we only trained and tested on the most recent five years.
One possible way to improve this model is to use daily factors such as volume,
close price as input, and convert fundamental factors to daily frequency by
the corresponding quarter. Then we would have a sample size of approximately
1250 for each sector and over 13000 samples for training. However, the model
might depends more on the daily factors rather than fundamental factors since
their exposures would be the same value for each quarter.

##  6 References

1\. Returns to Buying Winners and Selling Losers: Implications for Stock
Market Efficiency Narasimhan Jegadeesh; Sheridan Titman The Journal of
Finance, Vol. 48, No. 1. (Mar., 1993), pp. 65-91.
2.The Global Industry Classification Standard, MSCI (1999)


# “Oh, Sorry, I Think I Interrupted You”: Designing Repair Strategies for
Robotic Longitudinal Well-being Coaching

Minja Axelsson<EMAIL_ADDRESS>University of CambridgeUnited
Kingdom ,  Micol Spitale<EMAIL_ADDRESS>Politecnico di MilanoItaly
and  Hatice Gunes<EMAIL_ADDRESS>University of CambridgeUnited
Kingdom

(2024)

###### Abstract.

Robotic well-being coaches have been shown to successfully promote people’s
mental well-being. To provide successful coaching, a robotic coach should have
the capability to repair the mistakes it makes. Past investigations of robot
mistakes are limited to game or task-based, one-off and in-lab studies. This
paper presents a 4-phase design process to design repair strategies for
robotic longitudinal well-being coaching with the involvement of real-world
stakeholders: 1) designing repair strategies with a professional well-being
coach; 2) a longitudinal study with the involvement of experienced users
(i.e., who had already interacted with a robotic coach) to investigate the
repair strategies defined in (1); 3) a design workshop with users from the
study in (2) to gather their perspectives on the robotic coach’s repair
strategies; 4) discussing the results obtained in (2) and (3) with the mental
well-being professional to reflect on how to design repair strategies for
robotic coaching. Our results show that users have different expectations for
a robotic coach than a human coach, which influences how repair strategies
should be designed. We show that different repair strategies (e.g.,
apologizing, explaining, or repairing empathically) are appropriate in
different scenarios, and that preferences for repair strategies change during
longitudinal interactions with the robotic coach.

socially assistive robotics, well-being, coaching, interaction ruptures, robot
mistakes, human-robot interaction, design research

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
2024 ACM/IEEE International Conference on Human-Robot Interaction; March
11–14, 2024; Boulder, CO, USA††booktitle: Proceedings of the 2024 ACM/IEEE
International Conference on Human-Robot Interaction (HRI ’24), March 11–14,
2024, Boulder, CO, USA††doi: 10.1145/3610977.3634948††isbn:
979-8-4007-0322-5/24/03††ccs: Human-centered computing User studies††ccs:
Human-centered computing HCI design and evaluation methods††ccs: Computer
systems organization Robotics

##  1\. Introduction

Robot mistakes and failures are a very well-known problem in the human-robot
interaction (HRI) research community (Esterwood and Robert, 2022). For
example, robots can hit objects, interrupt people speaking, misunderstand
human speech, not respond to people, and experience Wi-Fi malfunctions (Sebo
et al., 2019; Spitale et al., 2023c). Therefore, designing repair strategies
is extremely important for the success of the human-robot interaction. Very
recently, the HRI interest in applying and using robots as mental well-being
coaches has increased. Robotic coaches for well-being have been examined for
use at the workplace (Spitale et al., 2023a; Axelsson et al., 2023a), in
public environments (Axelsson et al., 2023b; Matheus et al., 2022), in a lab
context (Axelsson et al., 2022; Churamani et al., 2022; Bodala et al., 2021),
and for at-home use (Jeong et al., 2020, 2023b, 2023a). In these contexts,
addressing the problem of robot mistakes by designing repair strategies is
even more relevant for the success of the coaching practice and for making a
step forward towards the deployment of such robotic coaches in real-world
scenarios.

Past works have investigated the problem of repair strategies (Correia et al.,
2018; van Waveren et al., 2019; Salem et al., 2015; Kontogiorgos et al.,
2020a) but they have several limitations. First, most of them are limited to
game-based (e.g., (Sebo et al., 2019)) or task-based scenarios (e.g.,
(Kontogiorgos et al., 2020a)), without exploring the extent to which those
results may be applicable to other contexts, such as well-being coaching.
Second, in most of these studies, the users interacted with the robot in a
one-off interaction (Sebo et al., 2019; Ye et al., 2019). This constrained the
design of repair strategies to short-term interactions without considering
longitudinal effects (i.e., how the users’ perceptions change over time).
Third, most of the studies have been undertaken in the lab (Ye et al., 2019;
Kontogiorgos et al., 2020a), making it difficult to generalise these results
to real-world settings.

In this study, we sought to overcome those limitations by designing repair
strategies for robotic longitudinal well-being coaching with the involvement
of real-world stakeholders. To this end, we undertook a 4-phase design process
in which we involved a professional mental well-being coach, and users who had
already interacted with a robotic coach in their workplace. As a first step,
we designed a set of repair strategies together with the professional human
coach, utilizing their expertise and experience. In the second phase, we
deployed a robotic coach at a workplace and compared the two sets of repair
strategies over four weeks by involving 12 users who had already interacted
with a robotic coach before. After the study, we informed the users of the
study set-up, and asked for their feedback on the robot’s mistakes in a design
workshop. Finally, we reflected on the user study results and user insights
with the professional coach from the first phase, in order to further inform
the design of repair strategies for robotic well-being coaches.

This paper contributes to: 1) designing repair strategies appropriate for
robotic well-being coaching; 2) investigating the longitudinal effect of
repair strategies; and 3) collecting user perspectives on robot mistakes and
repair strategies in a real-world context.

Figure 1. A timeline of the four study phases with their goals, methodologies,
and outcomes that steered the next phase.

##  2\. Related Work

Well-being Coaching and Mistakes. While there is no single definition of well-
being (McNaught, 2011), we focus on the definition of well-being as positive
psychological functioning (Ryff, 1989). Mental well-being coaching aims to
support coachees to thrive in their life (Hart et al., 2001) and improve
positive psychological functioning (Van Dierendonck and Lam, 2023). Positive
psychology coaching in particular aims to help coachees to focus on positive
aspects of their life (Seligman, 2007), through exercises where the coachee
e.g. focuses on experiences where they felt grateful, in order to enhance
positive memories and goal attainment (Wood et al., 2010; Emmons and Mishra,
2011). During coaching, difficulties with the coach, such as the coach being
vague or not sensitive or supportive enough, not being flexible, and
struggling with the concepts of coaching, can disrupt the coach-coachee
relationship (Carter et al., 2017). Such issues can negatively impact the
alliance between the coach and coachee (Safran and Muran, 1996), and as such
disrupt the interaction (De Haan and Gannon, 2017). Resolving such mistakes is
important so that the coachee can gain the maximum positive impact from
coaching.

Robotic Coaching. In a workplace coaching context, robotic forms have been
compared, finding preference for a toy-like robot (Spitale et al., 2023a). In
public settings, robots have been used to conduct group Mindfulness sessions
(Axelsson et al., 2023b), finding that robots can be helpful but need to be
more responsive; and to conduct private deep-breathing sessions at a health
centre (Matheus et al., 2022), finding that participants successfully
completed deep breathing exercises with the robot. Finally, in home contexts,
a robotic coach has been deployed in several at-home studies (Jeong et al.,
2020, 2023b, 2023a), finding that users’ well-being improved, and that the
robot was perceived more positively when it related to the user as a companion
rather than a coach. While these studies all examined robotic coaching in
different contexts, none of them focused on the examination and repair of
robot mistakes during coaching. Spitale et al. examined how users expressed
their behaviour during the mistakes a robotic coach makes (Spitale et al.,
2023c), however they did not examine what repair strategies robotic coaches
should use when they make mistakes. In this study, we present the first steps
into examining what repairs are applicable to robotic well-being coach
mistakes.

Robot Mistakes and Repairs in HRI. Within the context of social interactions,
past works have focused on understanding robot failures to improve various
aspects of human perception towards robots, such as trust (Harrison et al.,
2023). Correia et al. explored how technical failures of an autonomous social
robot affects trust during a HRI collaborative scenario (Correia et al.,
2018). Their results showed that a faulty robot is perceived as significantly
less trustworthy. Analogously, van Wareven et al. investigated the effect of
robot failure severity on participants’ subjective rating of the robot in a
room-escape scenario (van Waveren et al., 2019) and found that the severity
affects the faith participants had on robots in future scenarios. Salem et al.
explored how robot mistakes affect trustworthiness and acceptance in human-
robot collaboration (Salem et al., 2015), and showed that subjective
perceptions of the robot and assessments of its reliability and
trustworthiness has been significantly affected by the robot’s performance.
Robot mistakes and repair strategies have been largely examined in the context
of task-based interactions. Kontogiorgos et al. examined a Furhat robot
simulating errors during Wizard-of-Oz cooking instructions (Kontogiorgos et
al., 2020a, b, 2021), finding that participants responded most positively
through non-verbal responses to a robot’s explanations. Esterwood and Robert
also found that explanations were the most effective strategy for trust repair
after repeated mistakes in a box sorting task (Esterwood and Robert, 2021,
2023). Additionally, Sebo et al. found that a robot’s apology was a more
effective trust repair strategy than denial during a game (Sebo et al., 2019).
However, these studies did not examine the longitudinal effects of different
repair strategies.

##  3\. Design Process Overview

This work aims to understand how a robotic longitudinal well-being coach could
repair mistakes during coaching. We have undertaken a design process that
included four phases depicted at glance in Fig. 1: (Phase 1) repair strategy
design, (Phase 2) user study, (Phase 3) design workshop with users, and (Phase
4) coach reflections on user insights. Each phase was informative for the
following one, and we set our goals every step of the way during the design
process.

Phase 1 aimed at designing two sets of repair strategies for robotic
longitudinal well-being coaches. To accomplish this, we first discussed with a
professional well-being coach the robot mistakes in well-being coaching
(Discussion 1), and we then defined with them potential repair strategies
(Discussion 2), as detailed in Section 4. The results of this design phase
were the formulation of empathic and non-empathic repair strategies. Building
upon these results, Phase 2 aimed at investigating the user perception towards
a robotic coach that used empathic and non-empathic repair strategies in a
user study, as described in Section 5. We found that users had various
opinions and preferences towards empathic and non-empathic conditions, opening
up further investigation. Hence, we designed Phase 3 as a design workshop with
users of the study to better understand the appropriateness of each repair
strategy in different scenarios, as detailed in Section 6. The workshop
results showed that different repair strategies may be preferred depending on
what errors the users experienced, and personal preferences. Finally, we
involved the mental well-being coach from Phase 1 in Phase 4 to gather their
feedback and reflect on the findings from Phase 2 and Phase 3, as reported in
Section 7.

##  4\. Phase 1: Repair Strategy Design

In Phase 1, we set out to understand how to design repair strategies for
robotic coaches, i.e., what a robot should say when it makes a mistake. To do
this, we had two discussions with a professional well-being coach, in order to
find out how they repair mistakes during a coaching session, and to design
appropriate repair strategies for a robotic well-being coach with the
professional coach.

DISCUSSION 1: Robot Mistakes

In this first 2-hour discussion with a professional well-being coach, we
followed a structure consisting of four parts: 1) show the coach videos of a
robotic coach interacting with participants, and instances of interaction
ruptures; 2) ask the coach what the robot did wrong; 3) ask the coach how the
robot should attempt to repair those situations; and 4) how these repair
strategies should best be deployed in a user study. Below, we detail each part
of the discussion.

1) In our previous study (Spitale et al., 2023a), we collected video
recordings of coachees interacting with a robotic coach in a workplace
setting, and subsequently analyzed interaction ruptures, i.e. instances where
the robotic coach made mistakes and/or where coachees felt awkward (Spitale et
al., 2023c). We used this video data, and selected short clips that contained
instances of interaction ruptures (as annotated in (Spitale et al., 2023c)).
We showed 6 clips in total, ranging from 22 seconds to 2 minutes and 20
seconds.

2) The coach noted that the robot’s main mistakes were interrupting the
coachee, and not providing a response for a long period of time. Additionally,
the robot did not respond appropriately to the content of the user’s speech.
In the videos from our previous study (Spitale et al., 2023c), the robot was
responding in a pre-scripted manner, which was a barrier to generating
appropriate responses on the go.

3) The coach mentioned that initially, the robot should apologize for the
mistake it has made, and then identify what mistake has been made (e.g., “Oh,
sorry, I think I interrupted you.”), as that is what they would do as a coach
in a session. Robot apologies have also been previously used to repair trust
in human-robot interactions (Pompe et al., 2022). The coach suggested that
after the apology, the robot should then acknowledge its limitations as a
robot: the robot should explain why a mistake happened, or give its best guess
of the technical fault if there is no definitive information (e.g., “The Wi-Fi
is not working well.”); and the robot should emphasize its intent to improve
(e.g., “I’m trying to improve.”), since self-awareness of robot limitations
could put people at ease.

4) In order to examine users’ perceptions of a robotic coach’s repair
strategies, we discussed with the coach whether the robot should make
intentional mistakes during a coaching session, and then deploy the repair
strategies defined above. We decided to examine the most common mistakes of
the robot, i.e., interrupting and not responding, which we identified from the
videos. In order to collect repair strategies with the structure defined
above, we scheduled a second discussion with the coach.

Outcome: Identifying common robot mistakes during coaching interactions
(interrupting and not responding), and creating the structure for repair
strategies (apology, explanation, and intent to improve).

DISCUSSION 2: Repair Strategies

In this 1.5-hour discussion, we used bodystorming (Oulasvirta et al., 2003),
i.e., we simulated an interaction with the robotic coach by asking the
professional coach to act as the robotic coach, and two researchers acting as
coachees. We did this in order to experience the coaching session, the
mistakes, and the perception of the planned repair strategies from the
perspective of the coachees. In preparation for the discussion, we adapted
four existing well-being exercises with the coach: (1) savouring, where the
coach asks the coachee to reflect on a positive memory in the recent past
((Smith and Hanni, 2019)); (2) gratitude, where the coach asks the coachee to
think of things they were grateful for ((Gregersen et al., 2014)); (3)
accomplishments, where coachees reflect on accomplishments ((Gregersen et al.,
2014)); and (4) one door closes one door opens, where coachees think of a time
when a door closed (i.e., they missed out on an opportunity), and what doors
opened as a result (i.e., new opportunities arose), in order to cultivate
optimism ((Kauffman, 2006)).

During the bodystorming, we asked the coach to act as a robot and
intentionally interrupt and not respond to the researchers, and then use the
repair strategy we designed (see Sec. 4) in Discussion 1 (i.e., apology,
explanation, and intent to improve). We provided the coach with a list of
causes for robot errors such as interrupting and not responding, to use in
their explanation: microphone fault, processing error, slow Wi-Fi, and error
in speech understanding.

We structured the 1.5-hour session as follows: 1) 30 minutes of coaching,
mistakes and repair strategies with 15 minutes per each researcher while the
other took notes; 2) 15 minutes of discussion; 3) 30 minutes of coaching,
mistakes and repair strategies with 15 minutes per researcher while the other
took notes; and 4) 15 minutes of discussion. We detail each part of the
discussion below.

1) The coach conducted each exercise with the two researchers, acting as the
robotic coach, and asked for 2 instances of positive experiences during each
exercise (e.g., 2 moments when the coachee felt grateful). During each
instance, the coach made a mistake (not responding or interrupting), and then
used a repair strategy (apology, explanation, and intent to improve). We
collected 8 different phrasings of repair strategies in this manner.

2) After the first half hour, both researchers and the coach had a discussion
on how the session went. Both researchers noted that despite the employed
repair strategies, they did not feel understood or listened to, and that the
coach did not understand how they felt. The coach also mentioned that it could
be helpful to ask the coachee how they felt after the coach made a mistake.
Both researchers noted that the coaching interaction appeared awkward. We
decided that to resolve these issues (which have also been previously reported
in robotic coaching (Axelsson et al., 2022; Spitale et al., 2023a; Axelsson et
al., 2023b)), the robotic coach should be empathic. In fact, empathic
communication in a therapeutic context can help a client feel listened to,
understood and accepted, and have their feelings validated—improving outcomes
for well-being (Jani et al., 2012; Watson et al., 2014). Empathy can also be
conducive to resolving affective ruptures in interpersonal interactions
(Galinsky et al., 2011). The coach suggested that we amend the repair strategy
structure to be empathic as follows (additions in bold): apology, ask for user
emotion, empathize with user emotion, reassure user, explanation, and intent
to improve.

3) For the second 30-minute coaching session, each researcher again did two
exercises with the coach. This time, the coach used the empathic repair
strategies as designed above. Again, the coach administered each of the four
exercises, with 2 instances of each, resulting in 8 different phrasings of
empathic repair strategies.

4) In the final discussion, we found that the coach felt the empathic
strategies better suited a robotic coach, and the researchers felt listened to
and heard, and that the coach understood how they felt. The researchers also
observed that the interaction appeared less awkward. We concluded that in
order to examine how empathy impacts users’ perceptions of repair strategies,
we should conduct a between-subjects study comparing the two different types
of repair strategies (empathic and non-empathic). We also discussed that in
order to investigate how users’ opinions of robotic coach repair strategies
evolve over time, the study should be longitudinal.

Outcome: Study design of comparing empathic and non-empathic repair
strategies, and 8 phrasings of each type of repair strategy.

##  5\. Phase 2: User Study

In order to examine the empathic and non-empathic repair strategies defined in
Phase 1 (Sec. 4), we conducted a longitudinal user study where the robot
executed those repair strategies (Phase 2 in Fig. 1). The study design, the
experiment protocol, and the consent forms were approved by the Ethics
Committee of the Department of Computer Science and Technology, University of
Cambridge.

###  5.1. Protocol & Questionnaires

We conducted the study over four weeks, with one well-being exercise (a
maximum of 10 minutes) administered by the robotic well-being coach per week.
Users interacted with the robot at their workplace, in a room reserved for the
study, with the robot standing on a table 1.5 meters away from the user, and
the user sat at a chair next to the table. After each interaction, users
filled in a PANAS questionnaire about their positive and negative emotions
(Watson et al., 1988), RoPE questionnaire about the user’s perception of the
robot’s empathy (Charrier et al., 2019), RoSAS about the user’s perception of
the robot’s social attributes (Carpinella et al., 2017), and the MDMT
questionnaire about trust in the robot (Malle and Ullman, 2021). At the end of
the study, users took part in a semi-structured interview about their overall
experience with the robot (15-20 minutes). This interview had two parts: one
with general questions about the robot, its mistakes and repair strategies;
and one after disclosing the study protocol and the pre-planned mistakes to
the user, followed by questions about the robot’s mistakes and repair
strategies. Interview questions are reported in the Supplementary Material
(Sec. 1).

###  5.2. Users

The users ($n=12$) were recruited from the host company called Cambridge
Consultants Inc., and had previously interacted with a robotic well-being
coach in a 4-week study (Spitale et al., 2023a), where they experienced
commonly known robot errors (such as interruptions and slow responses). We
selected these experienced users in order to to mitigate the novelty effect
(Belpaeme, 2020), so that we could engage users in a critical discussion about
the robot’s mistakes and repair strategies after the study, which would not be
influenced by the novelty of a robotic well-being coach. Users were also
screened for anxiety (GAD-7 questionnaire) (Williams, 2014) and depression
(PHQ-9 questionnaire) (Kroenke et al., 2001). We chose this screening in order
to not use a robotic coach with a clinical population, which we do not
consider ethical prior to thorough examination with a non-clinical population.
The users were split into two groups ($n=6$ per condition), one group
experiencing the empathic repair strategy condition, and the other the non-
empathic repair strategy condition. 1 user was aged 18–25, 4 were aged 26–35,
3 were aged 36–45, and 4 were aged 46–55. 3 users were female, 1 non-binary,
and 8 male. Minority genders (female and non-binary) were balanced across
conditions, with 2 in each condition. Users rated their previous experience
with social robots as $(M=3,SD=1.044)$ on a scale from 1 (lowest) to 5
(highest). All users were native or fluent speakers of English, and all had an
Undergraduate, Master’s, or PhD degree. The users’ demographics reflect the
demographics of the host company Cambridge Consultants Inc., and as such have
the distribution of the ages, genders, and degree statuses as described.

###  5.3. Robot Platform and Architecture

We used the QTrobot by LuxAI S.p.A.111https://luxai.com/—a 90 cm tall,
tabletop child-like robot with static legs, 4 degrees of freedom (DOF) arms, 2
DOF neck, and a screen face. We chose this robot as it has been previously
used successfully as a robotic coach (Spitale et al., 2023a; Axelsson et al.,
2023b). The fully autonomous robotic coach was implemented using our newly
developed VITA system (Spitale et al., 2023b), a multi-modal LLM-based system
for longitudinal and adaptive robotic mental well-being coaching, that is open
source leveraging on the HARMONI framework (Spitale et al., 2021).

###  5.4. Exercises

The robotic coach administered a different Positive Psychology exercise each
week, one per week. The robotic coach delivered the same four exercises used
during Discussion 2 described in Section 4. Each exercise consisted of the
robot asking for two different examples from the users, and two follow-up
questions per example. Follow-up questions were generated by sending the
user’s utterance in response to the robot’s questions to ChatGPT gpt-3.5-turbo
model via OpenAI APIs222https://platform.openai.com/. We used ChatGPT to
generate follow-up questions, in order to minimize the impact of a pre-
scripted robot not responding appropriately, which was a mistake the coach
identified from the robotic coach videos in Phase 1 (Sec. 4).

###  5.5. Administered Robot Mistakes and Repairs

Each week, the robot was pre-programmed to make two mistakes, each during one
of its utterance turns during the interaction. While the conversational flow
itself was automated (as described in Sec. 5.4), this conversational flow was
interrupted at pre-determined times to administer the mistake and the repair.
The timing and type of mistake was counterbalanced across the sessions to
avoid repetitiveness (each timing and type is listed in Tables 1, 2 and 3 in
the Supplementary Material). These mistakes were either interrupting the user
(3-7 seconds into the user’s speaking turn), or not responding to the user for
a longer period of time (12-18 seconds). These mistakes were chosen since they
were the most common mistakes in robotic well-being coaching, as identified
from watching the robotic coaching videos with the professional coach in Phase
1 (Sec. 4). The robot made these mistakes at different stages (either after
initially explaining the exercise, or when asking the follow-up questions).
The mistakes were distributed as follows: (Session 1) interrupting and not
responding, (Session 2) interrupting twice, (Session 3) not responding twice,
(Session 4) not responding and interrupting.

###  5.6. Study Conditions

Our two study conditions for the between-subjects study were the robot
administering either empathic repair strategies, or non-empathic repair
strategies. The repair strategies were deployed whenever the robot made a pre-
planned mistake, and were constructed together with the professional well-
being coach.

The basic structure of both conditions was defined together with the
professional well-being coach, to include an apology, a technical explanation
for why the error occurred (realistic explanations for why each type of
mistake typically occurs in HRI—e.g., microphone or Wi-Fi malfunction (Spitale
et al., 2023c)), and intent to improve (to reassure the user). In the empathic
condition, the robot would also ask the user about the emotion they were
experiencing due to the mistake, cognitively empathize with the emotion
(Birmingham et al., 2022) (i.e. repeating the emotion of the user to
acknowledge it), and affectively reassure them (Pincus et al., 2013) (aiming
to reduce worry and to reassure the user that the robot is attempting to
listen to them). To cognitively empathize and affectively reassure the user,
the user’s utterance in response to the robot’s question about how they were
feeling was sent to ChatGPT gpt-3.5-turbo model via OpenAI APIs, and ChatGPT
was prompted to return the utterance’s emotional valence (positive, neutral,
or negative), and the specific emotion (repeated back to the user by the robot
when ChatGPT returned the utterance’s emotion as negative, to acknowledge it).

####  5.6.1. Repair Strategy Construction

The repair strategies and examples of each were constructed (with differences
between conditions italicized) with the following structure. The specific
wording of each repair strategy was different, and was based on the
professional coach’s phrasing. A full list of all repair strategies is made
available in Supplementary Material (Sec. 2).

Non-empathic: Apology, explain technical error, intent to improve, ask for
repetition of user’s previous utterance

Example:  “Oh, sorry, I think I interrupted you. My microphone isn’t working
well today. I’m trying to do better. Could you repeat what you were saying
before I interrupted you?”

Empathic: Apology, ask for user emotion, cognitively empathize with user
emotion(Birmingham et al., 2022), affective reassurance(Pincus et al., 2013),
explain technical error, intent to improve, ask for repetition of user’s
previous utterance.

Example:  “Oh, sorry, I think I interrupted you. How did me interrupting you
make you feel?”

$\rightarrow$ Negative user feeling description:
“I’m sorry, I understand it can make you feel ¡feeling [e.g., awkward]¿ when I
make mistakes. My intention is to listen to what you are saying, but sometimes
I experience errors. My microphone isn’t working well today. I’m trying to do
better. Could you repeat what you were saying before I interrupted you?”

$\rightarrow$ Positive or neutral user feeling description:
“Thanks for being understanding. My intention is to listen to what you are
saying, but sometimes I experience errors. My microphone isn’t working well
today. I’m trying to do better. Could you repeat what you were saying before I
interrupted you?”

###  5.7. User Study Findings

####  5.7.1. Data Analysis

Due to the sample size for a between-subjects study, we use descriptive
statistics to describe the quantitative differences between user groups’
perceptions of the empathic and non-empathic repair strategies, as well as the
longitudinal perception of the repair strategies. For qualitative analysis, we
use Framework Analysis (Srivastava and Thomson, 2009), consisting of the steps
of: (1) familiarization with the data, (2) identifying a thematic framework,
(3) indexing, (4) charting the data, and (5) interpretation of the data.

####  5.7.2. Quantitative Results

We present these results to contextualize our qualitative results, as well as
the user design workshop (Sec. 6) and coach feedback (Sec. 7) findings. As
described in Sec. 5.1, we administered questionnaires after each session, and
after the four sessions of the study. We also measured users’ well-being with
Ryff’s well-being questionnaire (Ryff, 1989) pre- and post-study (min.: 18,
max.: 108). The median of well-being was Empathic: ($(pre=100,post=105)$);
Non-empathic: ($(pre=90.5,post=92.5)$). These results confirmed our
expectations of no significant impact on well-being, due the negative impact
of the planned mistakes on the coaching experience, as well as the screened
user group with high levels of well-being to begin with (see Sec. 5.2).

Questionnaire | Measure | Empathic | Non-Empathic | Average
---|---|---|---|---
WAI-SR (Munder et al., 2010) | Goal-subscale $\uparrow$ | M $=$ 7.500, SD $=$ 1.225 | $(M=5.667,SD=1.506)$ | $(M=6.583,SD=1.621)$
WAI-SR (Munder et al., 2010) | Task-subscale $\uparrow$ | M $=$ 14.000, SD $=$ 2.098 | $(M=8.167,SD=2.714)$ | $(M=11.083,SD=3.825)$
WAI-SR (Munder et al., 2010) | Bond-subscale $\uparrow$ | M $=$ 31.333, SD $=$ 4.761 | $(M=21.333,SD=7.340)$ | $(M=26.333,SD=7.878)$
WAI-SR (Munder et al., 2010) | Alliance total $\uparrow$ | M $=$ 31.333, SD $=$ 4.761 | $(M=21.333,SD=7.340)$ | $(M=26.333,SD=7.878)$
SUS (Grier et al., 2013) | Usability total $\uparrow$ | M $=$ 3.500, SD $=$ 1.517 | $(M=2.500,SD=0.837)$ | $(M=3.000,SD=1.280)$
(C) - Understand | What I said $\uparrow$ | M $=$ 3.500, SD $=$ 1.517 | $(M=2.500,SD=0.837)$ | $(M=3.000,SD=1.280)$
(C) - Understand | How I felt $\uparrow$ | M $=$ 2.500, SD $=$ 1.643 | $(M=1.500,SD=0.837)$ | $(M=2.000,SD=1.349)$
(C) - Understand | Adapted $\uparrow$ | M $=$ 3.500, SD $=$ 1.055 | $(M=3.000,SD=0.894)$ | $(M=3.25,SD=1.055)$
(C) - Mistakes | Made mistakes $\uparrow$ | M $=$ 4.8.333, SD $=$ 0.408 | $(M=4.500,SD=0.837)$ | $(M=4.667,SD=0.651)$
(C) - Mistakes | I understood why $\uparrow$ | M $=$ 4.167, SD $=$ 1.170 | $(M=3.667,SD=1.033)$ | $(M=3.917,SD=1.084)$
(C) - Mistakes | Irritation $\downarrow$ | $(M=3.167,SD=1.329)$ | M $=$ 4.167, SD $=$ 1.169 | $(M=3.667,SD=1.303)$
(C) - Mistakes | Disruption $\downarrow$ | $(M=3.667,SD=1.506)$ | M $=$ 4.500, SD $=$ 0.837 | $(M=4.083,SD=1.240)$
(C) - Mistakes | Repaired – | M $=$ 3.167, SD $=$ 1.412 | M $=$ 3.167, SD $=$ 0.983 | M $=$ 3.167, SD $=$ 1.193
(C) - Repairs | Appropriate $\uparrow$ | M $=$ 3.333, SD $=$ 1.211 | $(M=3.000,SD=1.095)$ | $(M=3.167,SD=1.115)$
(C) - Repairs | Appropriate amount $\uparrow$ | M $=$ 3.333, SD $=$ 1.033 | $(M=2.833,SD=0.753)$ | $(M=3.083,SD=0.900)$
(C) - Repairs | Right time $\uparrow$ | M $=$ 2.833, SD $=$ 1.170 | $(M=2.500,SD=1.049)$ | $(M=2.667,SD=1.073)$
(C) - Repairs | Empathic $\uparrow$ | M $=$ 3.667, SD $=$ 1.366 | $(M=2.833,SD=1.330)$ | $(M=3.250,SD=1.357)$

Table 1. Post-study quantitative measures (higher measures bolded). (C)
denotes a custom question. Arrows illustrate where positive changes occurred
in the empathic condition.

We present post-study quantitative measures in the Table 1. As post-study
measures, we administered the WAI-SR (Working Alliance Inventory Short)
questionnaire (Munder et al., 2010) to measure user alliance with the robotic
coach, and the SUS (System Usability Scale) questionnaire (Grier et al., 2013)
in order to examine users’ perceptions of the robotic coach between
conditions. Additionally, we administered custom questions on a Likert scale
from 1 (lowest) to 5 (highest), about the robot’s understanding of what the
user said, felt, and how it adapted to them; the robot’s mistakes in terms of
whether it made mistakes, whether the users understood why it made mistakes,
were irritated by the mistakes, the session was disrupted by the mistakes, and
whether the robot repaired the mistakes; and on the repairs in terms of how
appropriate they were, how appropriate the amount of repairs was, whether the
repairs were administered at the right time, and whether the repairs were
empathic. The data indicates that the robot was perceived slightly more
positively in the empathic condition for alliance, usability, understanding,
and success of repairs than in the non-empathic condition. For mistakes, the
robot was also perceived more positively, with users better understanding why
the robot made mistakes, and feeling less disrupted and irritated.

Figure 2. Longitudinal trends of quantitative measures in users’ experiences
of empathic and non-empathic repairs.

We find the longitudinal quantitative data from week-by-week measures to be a
useful indicator of users’ experience. We calculated the median for each
measure (reported in Sec. 5.1) for each week, within each condition group and
across all users. Our data show decreasing trends across all users and within
both conditions throughout the weeks (Fig. 2) for trust in the robot (MDMT),
robot’s perceived empathy (RoPE), and a similar trend for the empathic
condition for the robot’s social attributes (RoSAS). These data indicate that
over time, the users’ experience with the robot was decreasing in quality. The
qualitative data in the next section (Sec. 5.7.3) helps understand why this
may be the case. Mood (PANAS) measures remained relatively stable across the
weeks, with the non-empathic group experiencing lower mood throughout.
Together with the other measures, this may indicate a more negative experience
with the robot in the non-empathic condition.

####  5.7.3. Qualitative Results

We analysed our qualitative results with Framework Analysis (Srivastava and
Thomson, 2009), to examine users’ longitudinal experiences, and the two
condition groups (empathic and non-empathic). We present quotes related to
these themes in the Supplementary Material (Sec. 3). For instance, some users
called the empathic strategies of the robot “caring” (P20) and that it
“adapted to the sentiment” they were feeling (P01), but that the empathy
became less genuine over time (P01) and was “a bit over the top” (P05). Some
users stated a preference for not needing empathy in a robot: “it felt too
much like a machine and empathy doesn’t apply” (P02).

In terms of longitudinal experiences, users noted that in the initial weeks,
they were trusting of the robot recognizing its mistakes, and of its repairs.
However, this trust decreased throughout the weeks, as shown by quantitative
(Fig. 2). For instance, explanations were viewed as helpful to understand the
robot was not “broken” (P09), however users noted that the robot’s
explanations seemed less genuine towards the end of the study and felt like
“excuses” (P11). In fact, some users noted that repairs “made [the
interaction] worse” (P04), disrupting the interaction. The repetitiveness of
the repairs was also viewed as disruptive, and that they were not “meeting my
needs, as in not understanding” (P06).

The challenges for repair strategy design found by this analysis are 1) users
become less receptive to the repair strategies over time, 2) users’ personal
preferences for repairs (especially empathic vs non-empathic repairs), and 3)
repairs are sometimes helpful but sometimes disruptive. Users had extensive
feedback on how repair strategies could be improved, and when and how often to
apply them. To investigate this further, we organized a design workshop with
the users as the next phase of the study.

##  6\. Phase 3: Workshop with Users

We conducted a design workshop with users from the study in Phase 2 ($n=10$,
with P04 and P07 not available to attend), in order to gather their opinions
on how a robotic well-being coach should deploy repair strategies in different
scenarios during coaching. We discussed the challenges identified in Phase 2
(Sec. 5), namely the negative perception of repair strategies over time,
users’ personal preferences, and when repairs are helpful and when disruptive.
The workshop was conducted in an online video call, with a Miro board for
collaboration333https://miro.com/. The workshop was structured into five
sections: (1) showing users videos of both repair strategy conditions
(empathic and non-empathic) to inform them of the robot’s capabilities, (2)
warm-up exercise, (3) thinking through appropriate repairs for pre-defined
scenarios, and generating mistake scenarios that each user experienced, (4)
thinking through appropriate repairs for those user-generated scenarios, and
(5) distilling insights for robotic well-being coach repairs for different
contexts.

###  6.1. Repair Strategy Preferences for User-generated Mistake Scenarios

We have included the 12 user-generated mistake scenarios in the Supplementary
Material (Sec. 4). In addition to the robot’s pre-programmed mistakes of
interrupting and non-responding, users generated other scenarios (referred to
as “Sc.” throughout this paper) of robot mistakes, e.g. the robot
misunderstanding them or asking generic questions. Users could select multiple
options for repair strategies for their scenarios, from 5 categories
(including the categories “empathic” and “technical explanation” which they
experienced in the study, and “instructions on use”, “brief apology”, and “do
nothing”, which they suggested in the post-study interviews) and also add
another suggestion in the “other” section. Overall, for the 12 user-generated
scenarios, user responses from the categories tallied up as follows: do
nothing (32), empathic (15), brief apology (48), technical explanation (6),
and instructions on use (10).

Do nothing or apologize briefly – Users show an overall preference for the
robot either doing nothing or giving a brief apology. Reasons for these
included “For a small error just continue with the session” (P06, Sc. 4),
“Better to move on” (P01, Sc. 4), “Not worth dwelling over, but [use] quick
apology for throwing the user off.” (P08, Sc. 5). Users explained that longer
repairs may in some cases distract from the session, and the robot should
“continue momentum” (P01, Sc. 4) and “try and get [the] conversation back on
track” (P05, Sc. 5).

Empathic – There were no scenarios where most users wanted empathic repairs.
In scenarios where users did want them (2-3 users), the reasons were e.g.
“Being empathetic, efficient and give advice on improving user experience
seems useful.” (P11, Sc. 2), and “Any empathetic apology should be to validate
user feelings and be quick [..] Providing advice on how to avoid this error
would be useful again to continue conversation along.” (P11, Sc. 7). One user
noted that a repair “Should be an empathic apology but not necessarily asking
how user felt.” (P01, Sc.10), wanting an empathic repair but in a brief
format.

Technical explanation and Instructions on use – Users preferred instructions
over explanations. Reasons for instructions were e.g. “I would like to know
how to get robot to repeat question in the future if it is my fault.” (P11,
Sc. 9), “If the robot can’t adapt to the user can the users adapt to the
robot” (P06, Sc. 10). Users wanted instructions to interact with the robot
when they could be able to correct the error. In contrast, explanations should
be used e.g. “If the reason for not understanding is known, try and point it
out to the user so they can address it.” (P05, Sc. 10) and “Given it’s a
multiple repeat I’d want to know that it has realised it’s mistake and by
giving an explanation I would assume it can actually move on.” (P03, Sc. 11).
In these cases, the user wanted the robot to inform them of the cause of the
error to increase the transparency of whether the robot is aware of the error
and the reason for it, as well as whether the user should further address it
(outside of the session). Technical explanations and instructions could be
used in tandem, e.g., “Robot should explain the technical error, perhaps it
did not hear properly and microphone should be moved closer.” (P09, Sc. 7).

###  6.2. User Insights and Discussion

While the previous sections show that users had significant personal
preferences, we conducted a final discussion at the end of the design workshop
to shape some more general insights for robotic coach repair strategies. These
insights are detailed here.

To repair or not to repair? – In general, users wanted the robot to repair
when the mistake was more disruptive to the session than deploying a repair
would be. Users noted the robot should not repair minor mistakes, because
paying attention to the mistake could cause further disruption to the session.
Minor mistakes were e.g., “if it interrupts the user momentarily” (P11), “it
interrupts the user and the user keeps talking” (P08), “if it’s likely a one-
off issue that will not impact further” (P03). In these cases, the users
preferred the robot to either do nothing or give a brief apology. The robot
should also not repair if it disrupts the session: “if the user is in the flow
and it would be detrimental to the session to interrupt the user” (P10), and
“the repair would be more interrupting / distracting than what is trying to be
repaired”. Users noted that the repairs would need to be selected to be less
disruptive (i.e., in the case of a minor mistake, a more detailed repair than
a brief apology might be disruptive). Scenarios where the robot should
definitely repair were “when the user needs to participate in the fix” (P03),
or “if the robot’s response depends on the user” (P11). In terms of
longitudinally administering repairs, users noted that initially the robot
should focus on introducing its main functionalities. In the next few
sessions, it is important to administer repairs to introduce the repair
capability to the user and improve their experience (P01), and to give the
user any necessary instructions to resolve issues (P10). Users noted that over
time they would expect less repairs, since the robot is not a “stranger”
anymore (P01), and that it “can cut through and carry on” (P10).

When to use empathic repairs? – In general, users wanted empathic repairs when
they were likely to feel frustrated. This could be the case in “repeated
mistakes, especially when I have been talking for a while already” (P05),
“when it interrupts during a story/detailed explanation of an experience”
(P03), and “when the error has affected the session in a way that could have
negatively impacted the flow” (P10). However, one user noted that empathic
repairs could make the user “feel more frustrated” (P01). This indicates that
empathic repairs should be dynamically deployed based on the user’s response
to the repair itself.

When to give technical explanations? – In general, explanations were wanted to
increase transparency in the case of severe technical errors. For example,
“when the error has occurred for the first time overall for rare errors”
(P12), and “when an error has occurred that is outside the user’s control”
(P10). However, the explanation should be “something the user can understand”
(P03).

When to give instructions on use? – In general, users wanted instructions from
the robot when the mistake was something that the user can help correct,
during the first few interaction sessions with the robot. Examples were “when
the user can actually make a simple fix (like move the microphone closer)”
(P03) and “if the error has to do with the user’s way of communicating” (P11).

##  7\. Phase 4: Coach Reflections

To reflect on the insights provided by the robotic coach users, we took these
insights back to the professional well-being coach who had collaborated with
us to design the robotic coach and its repair strategies in Phase 1 (Sec. 4).
In this discussion (1 hour), we asked the coach to reflect on the results of
our user study and the collected user insights, and to give their opinions on
them, as follows.

Robotic coach repair strategy design can not be solely based on professional
coach repair strategies — We can not conclude simply that “empathic repair
strategies are better than non-empathic ones”, as was the original expectation
of the authors and the well-being coach. Instead, our findings indicate that a
robotic well-being coach is fundamentally different in its capabilities when
compared to a human coach, and as such users’ expectations are different. The
professional coach noted that repeated social mistakes (interrupting or non-
responding) in human coaching are rare, as humans have the capacity to better
intuit social signals and thus make fewer mistakes. As coachees do not expect
many mistakes in human coaching, empathic repairs may be helpful in the rare
situations where mistakes do occur. However, due to limited robot
capabilities, social mistakes occur often in robotic coaching (as shown by the
videos in Sec. 4 (Spitale et al., 2023c)). Due to repeated mistakes, while the
coach’s and researchers’ intuition was that empathic repairs would be
similarly applicable to a robotic coach, the user study (Sec. 5) and design
workshop with users (Sec. 6) contradicted this intuition. Some users enjoyed
empathic repairs in the first few interaction sessions, but became averse when
repeated in later interactions, viewing them as disingenuous. Other users
experienced the empathic repairs as disingenuous already from initial
interactions, due to viewing the robot as a tool that empathic expressions are
incongruous with. As such, repair design needs to acknowledge the limited
capabilities and higher error rate of robotic coaches.

Repair strategies are not always necessary – In some instances users perceived
the repair strategies as disruptive (see Sec. 5-6). The professional coach
agreed that repairs could be detrimental if they were “long-winded
explanations” that may be perceived as excuses rather than helpful. The coach
agreed that users may “get used to a robot and its errors”, and due to this,
repairs could be reduced over time in the case of repetitive errors. Future
research is needed in to distinguish whether and when a user no longer
requires repair for a specific error, e.g. by detecting a user’s behavioral
signals.

Repairs should be utilized according to user preference – There is no “one-
size-fits-all” with regards to non-/empathic (or other) repairs in robotic
well-being coaching. Some users appreciated the empathic repairs and viewed
them as increasing trust and improving the interaction, while some viewed them
as disingenuous and even deceptive due to the fundamentally non-empathic
nature of a robot as a machine. The professional coach noted that at times
they adjust their level of empathic expression to their client, by matching
their expression to that observed from the client. Previous research proposed
the adaptation of a robot’s empathic expressions to a user’s positive
emotional signals (Leite et al., 2012). Future research is needed to analyze
user feedback during the administration of the repair strategy itself (i.e.,
observing a user’s response to a repair via behavioral signals), and
accordingly adjusting future repairs.

Robot-specific repair strategies include technical explanations and
instructions on use – The professional coach noted that technical error
explanation is a robot specific behaviour, and there is no direct comparison
to human-to-human coaching, but that if they were making a repeated error,
they would want to give an explanation to their client. Future research is
needed to develop robot awareness of mistake occurrence, cause, and to
generate authentic technical explanations. In terms of instructions on use,
the coach compared this to the situation where their client may have hearing
loss, and the coach might in turn increase their speech volume. Future
research should focus on how robots may detect and adapt to such personal
requirements of each user, while applying appropriate personalised
instructions on use of the robot.

Repetitions of repair strategies are detrimental – Due to social interaction
timing issues (i.e., mistakes such as interruptions and non-responding) that
are inherent to social robots (Spitale et al., 2023c; Maitreyee and Lindgren,
2021; Honig and Oron-Gilad, 2018), robotic coaching can have an awkward
rhythm. When repair strategies are continuously administered to repair these
issues (even with different phrasings across sessions, as we have done in our
user study), the effect can be detrimental to the user. The coach compared
this to a “phone helpline”, where the phrase “your call is important to us” is
often repeated, but “becomes less true over time”. The repair strategies we
have proposed here assume the current level of disruption in state-of-the-art
HRI, and they may not hold as robot capabilities further develop. Future
research should aim to reduce such latency in robotic conversational
interaction, so that repair strategies will gradually become less needed.

##  8\. Conclusions With A Critical Look

This paper contributes insights for designing repair strategies for
longitudinal robotic well-being coaching, informed by real-world users’ and a
professional coach’s perspectives. We have shown how our 4-phase design
process and its outcomes can contribute toward the real-world deployment of
longitudinal robotic coaches that are capable of repairing their mistakes and
thus improving coaching interactions. As part of this process, we designed our
initial study to compare empathic and non-empathic repairs, based on two
discussions with a professional well-being coach (Sec. 4). However, in our
between-subjects study where a robotic coach administered these repair
strategies (Sec. 5), we found that users had detailed feedback beyond the
question of whether empathic or non-empathic repair strategies were better. We
then designed a workshop with users to give detailed feedback on repairs for a
robotic coach (Sec. 6), and reflecting on these user insights with the
professional coach from Phase 1 (Sec. 7). We encourage researchers to include
such retrospective discussions with users and stakeholders in their research,
especially when their intuitions are not confirmed, to better understand how
robots are experienced in the real world.

We would also like to direct a critical eye toward our own study. In this
paper, we conducted a user study in which the robot made intentional mistakes
and repairs. However, intentional mistakes rely on timing, which is difficult
for social robots. Additionally, the timing of a repair can have an impact on
its success (Robinette et al., 2015). Thoroughly investigating the timing of
the mistakes and repairs, and how this may have impacted user perceptions is
out of scope of this work. We will investigate this in future research, and
encourage the HRI field to reflect on how intentional social mistakes can be
investigated taking into account the challenge of timing in interactions.
Also, we used a LLM (ChatGPT) to generate the robot’s responses to users. In
some cases, the LLM spontaneously asked users for clarification when it did
not “understand” the user’s utterance. In this paper, we asked users to recall
the mistakes and repairs they experienced during the interaction (including
those spontaneously deployed by the LLM), however, further analysis on
spontaneous repairs and users’ perceptions of them is out of scope of this
work. We invite future research to investigate the impact of spontaneous LLM
repairs on users’ perceptions of robotic coaches. These repair strategies have
been designed specifically for well-being coaching. In previous HRI literature
on game- and task-based scenarios, repair strategies focus on a robot’s
mistakes where a right vs wrong condition is clear due to a set goal
(Kontogiorgos et al., 2020a, b, 2021; Sebo et al., 2019; Esterwood and Robert,
2021, 2023). In conversational contexts such as coaching, identifying the
mistake itself, and consequently the necessity for and appropriate type of
repair, is more complex due to factors such as user preferences. Despite this
complexity, we have attempted to distil relevant insights and reflections on
such repair strategies. Future research should further investigate
longitudinal coaching interactions, administering repair strategies according
to our insights, and further refine these in an iterative manner.

###### Acknowledgements.

We thank Cambridge Consultants Inc. and their employees for participating in
this study. Funding: M. Axelsson is funded by the Osk. Huttunen foundation and
the EPSRC under grant EP/T517847/1. M. Spitale and H. Gunes have been funded
by the EPSRC/UKRI under grant ref. EP/R030782/1 (ARoEQ). Open Access: For open
access purposes, the authors have applied a Creative Commons Attribution (CC
BY) licence to any Author Accepted Manuscript version arising. Data access:
Raw data related to this publication cannot be openly released due to
anonymity and privacy issues.

## References

  * (1)
  * Axelsson et al. (2022) Minja Axelsson, Nikhil Churamani, Atahan Caldir, and Hatice Gunes. 2022.  Participant Perceptions of a Robotic Coach Conducting Positive Psychology Exercises: A Systematic Analysis.  _arXiv preprint arXiv:2209.03827_ (2022).
  * Axelsson et al. (2023a) Minja Axelsson, Micol Spitale, and Hatice Gunes. 2023a.  Adaptive robotic mental well-being coaches. In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_. 733–735.
  * Axelsson et al. (2023b) Minja Axelsson, Micol Spitale, and Hatice Gunes. 2023b.  Robotic Coaches Delivering Group Mindfulness Practice at a Public Cafe. In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_. 86–90.
  * Belpaeme (2020) Tony Belpaeme. 2020.  Advice to new human-robot interaction researchers.  _Human-robot interaction: Evaluation methods and their standardization_ (2020), 355–369.
  * Birmingham et al. (2022) Christopher Birmingham, Ashley Perez, and Maja Matarić. 2022.  Perceptions of Cognitive and Affective Empathetic Statements by Socially Assistive Robots. In _2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 323–331.
  * Bodala et al. (2021) Indu P Bodala, Nikhil Churamani, and Hatice Gunes. 2021.  Teleoperated robot coaching for mindfulness training: A longitudinal study. In _2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)_. IEEE, 939–944.
  * Carpinella et al. (2017) Colleen M Carpinella, Alisa B Wyman, Michael A Perez, and Steven J Stroessner. 2017.  The robotic social attributes scale (RoSAS) development and validation. In _Proceedings of the 2017 ACM/IEEE International Conference on human-robot interaction_. 254–262.
  * Carter et al. (2017) Alison Carter, Anna Blackman, Ben Hicks, Matthew Williams, and Rachel Hay. 2017.  Perspectives on effective coaching by those who have been coached.  _International Journal of Training and Development_ 21, 2 (2017), 73–91.
  * Charrier et al. (2019) Laurianne Charrier, Alisa Rieger, Alexandre Galdeano, Amélie Cordier, Mathieu Lefort, and Salima Hassas. 2019.  The rope scale: a measure of how empathic a robot is perceived. In _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 656–657.
  * Churamani et al. (2022) Nikhil Churamani, Minja Axelsson, Atahan Caldır, and Hatice Gunes. 2022.  Continual learning for affective robotics: A proof of concept for wellbeing. In _2022 10th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)_. IEEE, 1–8.
  * Correia et al. (2018) Filipa Correia, Carla Guerra, Samuel Mascarenhas, Francisco S Melo, and Ana Paiva. 2018.  Exploring the impact of fault justification in human-robot trust. In _Proceedings of the 17th international conference on autonomous agents and multiagent systems_. 507–513.
  * De Haan and Gannon (2017) Erik De Haan and Judie Gannon. 2017.  The coaching relationship.  _The SAGE handbook of coaching_ (2017), 195–217.
  * Emmons and Mishra (2011) Robert A Emmons and Anjali Mishra. 2011.  Why gratitude enhances well-being: What we know, what we need to know.  _Designing positive psychology: Taking stock and moving forward_ 248 (2011), 262.
  * Esterwood and Robert (2021) Connor Esterwood and Lionel P Robert. 2021.  Do you still trust me? human-robot trust repair strategies. In _2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)_. IEEE, 183–188.
  * Esterwood and Robert (2022) Connor Esterwood and Lionel P Robert. 2022.  A literature review of trust repair in HRI. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 1641–1646.
  * Esterwood and Robert (2023) Connor Esterwood and Lionel P Robert. 2023.  Three Strikes and you are out!: The impacts of multiple human-robot trust violations and repairs on robot trustworthiness.  _Computers in Human Behavior_ (2023), 107658.
  * Galinsky et al. (2011) Adam D Galinsky, Debra Gilin, and William W Maddux. 2011.  _Using both your head and your heart: The role of perspective taking and empathy in resolving social conflict_. Vol. 13.  Psychology Press New York.
  * Gregersen et al. (2014) Tammy Gregersen, Peter D MacIntyre, Kate Hein Finegan, Kyle Talbot, and Shelby Claman. 2014.  Examining emotional intelligence within the context of positive psychology interventions.  (2014).
  * Grier et al. (2013) Rebecca A Grier, Aaron Bangor, Philip Kortum, and S Camille Peres. 2013.  The system usability scale: Beyond standard usability testing. In _Proceedings of the human factors and ergonomics society annual meeting_ , Vol. 57. SAGE Publications Sage CA: Los Angeles, CA, 187–191.
  * Harrison et al. (2023) Katherine Harrison, Giulia Perugia, Filipa Correia, Kavyaa Somasundaram, Sanne van Waveren, Ana Paiva, and Amy Loutfi. 2023.  The Imperfectly Relatable Robot: An Interdisciplinary Workshop on the Role of Failure in HRI. In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_. 917–919.
  * Hart et al. (2001) Vicki Hart, John Blattner, and Staci Leipsic. 2001.  Coaching versus therapy: A perspective.  _Consulting Psychology Journal: Practice and Research_ 53, 4 (2001), 229.
  * Honig and Oron-Gilad (2018) Shanee Honig and Tal Oron-Gilad. 2018.  Understanding and resolving failures in human-robot interaction: Literature review and model development.  _Frontiers in psychology_ 9 (2018), 861.
  * Jani et al. (2012) Bhautesh Dinesh Jani, David N Blane, and Stewart W Mercer. 2012.  The role of empathy in therapy and the physician-patient relationship.  _Forschende Komplementärmedizin/Research in Complementary Medicine_ 19, 5 (2012), 252–257.
  * Jeong et al. (2020) Sooyeon Jeong, Sharifa Alghowinem, Laura Aymerich-Franch, Kika Arias, Agata Lapedriza, Rosalind Picard, Hae Won Park, and Cynthia Breazeal. 2020.  A robotic positive psychology coach to improve college students’ wellbeing. In _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 187–194.
  * Jeong et al. (2023a) Sooyeon Jeong, Laura Aymerich-Franch, Sharifa Alghowinem, Rosalind W Picard, Cynthia L Breazeal, and Hae Won Park. 2023a.  A robotic companion for psychological well-being: A long-term investigation of companionship and therapeutic alliance. In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_. 485–494.
  * Jeong et al. (2023b) Sooyeon Jeong, Laura Aymerich-Franch, Kika Arias, Sharifa Alghowinem, Agata Lapedriza, Rosalind Picard, Hae Won Park, and Cynthia Breazeal. 2023b.  Deploying a robotic positive psychology coach to improve college students’ psychological well-being.  _User Modeling and User-Adapted Interaction_ 33, 2 (2023), 571–615.
  * Kauffman (2006) Carol Kauffman. 2006.  Positive psychology: The science at the heart of coaching.  _Evidence based coaching handbook: Putting best practices to work for your clients_ 219 (2006), 253.
  * Kontogiorgos et al. (2020a) Dimosthenis Kontogiorgos, Andre Pereira, Boran Sahindal, Sanne van Waveren, and Joakim Gustafson. 2020a.  Behavioural responses to robot conversational failures. In _Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_. 53–62.
  * Kontogiorgos et al. (2021) Dimosthenis Kontogiorgos, Minh Tran, Joakim Gustafson, and Mohammad Soleymani. 2021.  A systematic cross-corpus analysis of human reactions to robot conversational failures. In _Proceedings of the 2021 International Conference on Multimodal Interaction_. 112–120.
  * Kontogiorgos et al. (2020b) Dimosthenis Kontogiorgos, Sanne Van Waveren, Olle Wallberg, Andre Pereira, Iolanda Leite, and Joakim Gustafson. 2020b.  Embodiment effects in interactions with failing robots. In _Proceedings of the 2020 CHI conference on human factors in computing systems_. 1–14.
  * Kroenke et al. (2001) Kurt Kroenke, Robert L Spitzer, and Janet BW Williams. 2001.  The PHQ-9: validity of a brief depression severity measure.  _Journal of general internal medicine_ 16, 9 (2001), 606–613.
  * Leite et al. (2012) Iolanda Leite, Ginevra Castellano, André Pereira, Carlos Martinho, and Ana Paiva. 2012.  Modelling empathic behaviour in a robotic game companion for children: an ethnographic study in real-world settings. In _Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction_. 367–374.
  * Maitreyee and Lindgren (2021) Maitreyee and Helena Lindgren. 2021.  Younger and Older Adults’ Perceptions on Role, Behavior, Goal and Recovery Strategies for Managing Breakdown Situations in Human-Robot Dialogues. In _Proceedings of the 9th International Conference on Human-Agent Interaction_. 433–437.
  * Malle and Ullman (2021) Bertram F Malle and Daniel Ullman. 2021.  A multidimensional conception and measure of human-robot trust.  In _Trust in human-robot interaction_. Elsevier, 3–25.
  * Matheus et al. (2022) Kayla Matheus, Marnyel Vázquez, and Brian Scassellati. 2022.  A social robot for´ anxiety reduction via deep breathing. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 89–94.
  * McNaught (2011) Allan McNaught. 2011.  Defining wellbeing.  _Understanding wellbeing: An introduction for students and practitioners of health and social care_ (2011), 7–23.
  * Munder et al. (2010) Thomas Munder, Fabian Wilmers, Rainer Leonhart, Hans Wolfgang Linster, and Jürgen Barth. 2010.  Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients.  _Clinical Psychology & Psychotherapy: An International Journal of Theory & Practice_ 17, 3 (2010), 231–239.
  * Oulasvirta et al. (2003) Antti Oulasvirta, Esko Kurvinen, and Tomi Kankainen. 2003.  Understanding contexts by being there: case studies in bodystorming.  _Personal and ubiquitous computing_ 7 (2003), 125–134.
  * Pincus et al. (2013) Tamar Pincus, Nicola Holt, Steven Vogel, Martin Underwood, Richard Savage, David Andrew Walsh, and Stephanie Jane Caroline Taylor. 2013.  Cognitive and affective reassurance and patient outcomes in primary care: a systematic review.  _Pain®_ 154, 11 (2013), 2407–2416.
  * Pompe et al. (2022) Babiche L Pompe, Ella Velner, and Khiet P Truong. 2022.  The robot that showed remorse: Repairing trust with a genuine apology. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 260–265.
  * Robinette et al. (2015) Paul Robinette, Ayanna M Howard, and Alan R Wagner. 2015.  Timing is key for robot trust repair. In _Social Robotics: 7th International Conference, ICSR 2015, Paris, France, October 26-30, 2015, Proceedings 7_. Springer, 574–583.
  * Ryff (1989) Carol D Ryff. 1989.  Happiness is everything, or is it? Explorations on the meaning of psychological well-being.  _Journal of personality and social psychology_ 57, 6 (1989), 1069.
  * Safran and Muran (1996) Jeremy D Safran and J Christopher Muran. 1996.  The resolution of ruptures in the therapeutic alliance.  _Journal of consulting and clinical psychology_ 64, 3 (1996), 447.
  * Salem et al. (2015) Maha Salem, Gabriella Lakatos, Farshid Amirabdollahian, and Kerstin Dautenhahn. 2015.  Would you trust a (faulty) robot? Effects of error, task type and personality on human-robot cooperation and trust. In _Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction_. 141–148.
  * Sebo et al. (2019) Sarah Strohkorb Sebo, Priyanka Krishnamurthi, and Brian Scassellati. 2019.  “I don’t believe you”: Investigating the effects of robot trust violation and repair. In _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 57–65.
  * Seligman (2007) Martin EP Seligman. 2007.  Coaching and positive psychology.  _Australian Psychologist_ 42, 4 (2007), 266–267.
  * Smith and Hanni (2019) Jennifer L Smith and Agnieszka A Hanni. 2019.  Effects of a savoring intervention on resilience and well-being of older adults.  _Journal of Applied Gerontology_ 38, 1 (2019), 137–152.
  * Spitale et al. (2023a) Micol Spitale, Minja Axelsson, and Hatice Gunes. 2023a.  Robotic mental well-being coaches for the workplace: An in-the-wild study on form. In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_. 301–310.
  * Spitale et al. (2023b) Micol Spitale, Minja Axelsson, and Hatice Gunes. 2023b.  VITA: A Multi-modal LLM-based System for Longitudinal, Autonomous, and Adaptive Robotic Mental Well-being Coaching.  arXiv:2312.09740 [cs.RO]
  * Spitale et al. (2023c) Micol Spitale, Minja Axelsson, Neval Kara, and Hatice Gunes. 2023c.  Longitudinal Evolution of Coachees’ Behavioural Responses to Interaction Ruptures in Robotic Positive Psychology Coaching. In _2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE.
  * Spitale et al. (2021) Micol Spitale, Chris Birmingham, R Michael Swan, and Maja J Matarić. 2021.  Composing harmoni: An open-source tool for human and robot modular open interaction. In _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 3322–3329.
  * Srivastava and Thomson (2009) Aashish Srivastava and S Bruce Thomson. 2009.  Framework analysis: a qualitative methodology for applied policy research.  (2009).
  * Van Dierendonck and Lam (2023) Dirk Van Dierendonck and Hodar Lam. 2023.  Interventions to enhance eudaemonic psychological well-being: A meta-analytic review with Ryff’s Scales of Psychological Well-being.  _Applied Psychology: Health and Well-Being_ 15, 2 (2023), 594–610.
  * van Waveren et al. (2019) Sanne van Waveren, Elizabeth J Carter, and Iolanda Leite. 2019.  Take one for the team: The effects of error severity in collaborative tasks with social robots. In _Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents_. 151–158.
  * Watson et al. (1988) David Watson, Lee Anna Clark, and Auke Tellegen. 1988.  Development and validation of brief measures of positive and negative affect: the PANAS scales.  _Journal of personality and social psychology_ 54, 6 (1988), 1063.
  * Watson et al. (2014) Jeanne C Watson, Patricia L Steckley, and Evelyn J McMullen. 2014.  The role of empathy in promoting change.  _Psychotherapy Research_ 24, 3 (2014), 286–298.
  * Williams (2014) Nerys Williams. 2014.  The GAD-7 questionnaire.  _Occupational medicine_ 64, 3 (2014), 224–224.
  * Wood et al. (2010) Alex M Wood, Jeffrey J Froh, and Adam WA Geraghty. 2010.  Gratitude and well-being: A review and theoretical integration.  _Clinical psychology review_ 30, 7 (2010), 890–905.
  * Ye et al. (2019) Sean Ye, Glen Neville, Mariah Schrum, Matthew Gombolay, Sonia Chernova, and Ayanna Howard. 2019.  Human trust after robot mistakes: Study of the effects of different forms of robot communication. In _2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 1–7.