# Impact of Preference Noise on the Alignment Performance of Generative
Language Models

Yang Gao, Dana Alon & Donald Metzler
Google Research
{gaostayyang, danama<EMAIL_ADDRESS>

###### Abstract

A key requirement in developing Generative Language Models (GLMs) is to have
their values aligned with human values. _Preference-based alignment_ is a
widely used paradigm for this purpose, in which _preferences_ over generation
pairs are first elicited from human annotators or AI systems, and then fed
into some alignment techniques, e.g., Direct Preference Optimization. However,
a substantial percent (20 - 40%) of the preference pairs used in GLM alignment
are _noisy_ , and it remains unclear how the noise affects the alignment
performance and how to mitigate its negative impact. In this paper, we propose
a framework to inject desirable amounts and types of noise to the preferences,
and systematically study the impact of preference noise on the alignment
performance in two tasks (summarization and dialogue generation). We find that
the alignment performance can be highly sensitive to the noise rates in the
preference data: e.g., a 10 percentage points (pp) increase of the noise rate
can lead to 30 pp drop in the alignment performance (in win rate). To mitigate
the impact of noise, _confidence-based data filtering_ shows significant
benefit when certain types of noise are present. We hope our work can help the
community better understand and mitigate the impact of preference noise in GLM
alignment.

##  1 Introduction

As the capabilities of _Generative Language Models_ (GLMs) keep improving
through pre-training at a large scale, methods for _aligning GLMs with human
preferences_ , i.e., steering GLMs to follow user instructions effectively and
safely, have attracted increasing attention (Ji et al., 2023). A widely used
paradigm to align GLMs with human values is to first collect _binary
preferences on generation pairs_ , and then use techniques like _Proximal
Policy Optimization_ (a Reinforcement Learning algorithm, (Schulman et al.,
2017)), _Direct Preference Optimization_ (DPO, (Rafailov et al., 2023)), or
_Sequence Likelihood Calibration_ (SLiC, (Zhao et al., 2023)) to align the
GLMs with the collected preferences. The binary preferences can be provided by
human annotators, trained _Reward Models_ (RMs, (Ouyang et al., 2022)), or
_Constitutional AI agents_ (Bai et al., 2022b; Lee et al., 2024). Preference-
based GLM alignment has proven to be highly effective in improving the safety
and usability of GLMs, and hence has been used to develop both open-source
(Touvron et al., 2023; Gemma, 2024) and proprietary (Google, 2023; Achiam et
al., 2023) GLMs.

However, the binary preferences used in GLM alignment are often _noisy_ ,
i.e., containing preferences that disagree with the ground truth (e.g.,
preferences provided by domain experts). Zheng et al. (2023) report that
19-37% preferences provided by crowd workers are noisy. Similar noise rates
are also observed in preferences provided by RMs and Constitutional AI. Table
1 summarizes the noise rates of preferences used in recent GLM alignment
works. It is generally believed that the lower the noise rates in the
preferences, the better the final alignment performance (Lee et al., 2024),
but it remains unclear what is the _quantitative relation between noise rates
and alignment performance_ , and, furthermore, how to _mitigate the negative
impact of preference noise_ on alignment performance. In this paper, we answer
these questions with systematic empirical studies.

Oracle | Task | Noise% | Reference
---|---|---|---
Human | MTBench | 19-37 | (Zheng et al., 2023)
Constitutional AI | MTBench | 15-34 | (Zheng et al., 2023)
TL;DR | 22 | (Lee et al., 2024)
CBArena | 22-36 | (Zheng et al., 2023)
AntHH | 27.9-30.9 | (Lee et al., 2024)
MetaHS | 41.4-41.9 | (Touvron et al., 2023)
Reward Models | TL;DR | 21.3-27 | (Zhao et al., 2023; Munos et al., 2023)
SHP | 26.3 | (Cui et al., 2023)
MetaHS | 35.5-36.8 | (Touvron et al., 2023)
WebGPT | 34.8 | (Cui et al., 2023)
Table 1:  Preference noise are observed in a wide range of tasks, including
video game (Atari), QA (MTBench, StanfordHumanPreference), Summarization
(TL;DR), and Dialogue (WebGPT, ChatBotArena, AntropicHelpfulHarmless,
MetaHelpfulSafety).

There exist frameworks for studying the influence of preference noise on
alignment performance in Robotics, e.g., the _B-Pref_ framework by Lee et al.
(2021a). They assume a gold reward model is available, and design different
strategies to _corrupt_ the gold reward model to provide (simulated) noisy
preferences (see §2 for more details). Although they have been successfully
used to benchmark and compare different preference-based RL algorithms in
Robotics tasks, they are not applicable for GLM alignment, mainly for two
reasons: (i) some of their noise simulation strategies are unsuitable for NLP
tasks; and (ii) the alignment techniques used in Robotics (e.g., PREBBLE (Lee
et al., 2021b)) are different from those used in GLM (e.g., DPO and SLiC). To
alleviate these problems, we propose a new framework whose noise-simulation
strategies and alignment techniques are tailored for GLM alignment. Figure 1
illustrates the framework.

Figure 1: Our framework for evaluating the impact of preference noise on GLM
alignment.

With the proposed framework, we perform controlled experiments to study the
impact of preference noise on alignment performance on two tasks,
summarization (Stiennon et al., 2020) and dialogue generation (Bai et al.,
2022a). We find that even with high ($45\%$) noise rates, GLM alignment is
still beneficial (i.e., yielding 50%+ win rate). However, alignment
performance is also highly sensitive to noise rates: a 10 percentage points
(pp) increase of noise rates can lead to 30 pp drop in the alignment
performance (in terms of win rate). We also explore different strategies to
mitigate the negative impact of preference noise on alignment performance. We
find that some widely used _regularization_ methods fail to mitigate the
negative impact, but _confidence-based data selection_ can effectively improve
performance in realistic settings. We hope our findings can help GLM
developers better understand the impact of preference noise on alignment
performance, and that our framework can facilitate the exploration of more
effective and noise-robust alignment methods.

##  2 Related Works

#### Preference-based GLM alignment.

Pairwise preferences are widely used in AI alignment, because controlled
experiments have suggested that asking for preferences places a lower
cognitive burden on the human subjects than asking for absolute ratings or
categorized labels (Kendall, 1948; Thurstone, 2017; Kingsley & Brown, 2010).
In GLM alignment, a common practice is to first train an RM from the human
provided preferences, and then use the RM to provide reward signals in
Reinforcement Learning (RL) (Böhm et al., 2019; Gao et al., 2020; Stiennon et
al., 2020). Recent methods like DPO (Rafailov et al., 2023), SLiC (Zhao et
al., 2023), and Identify Preference Optimization (IPO, Azar et al. (2023)) go
one step further, by eliminating the RM training step and directly using
preference pairs to train the final GLM. Azar et al. (2023) has shown that
these methods essentially optimize the same learning objective, and they
differentiate in their regularization terms and function approximation
methods. Compared to the RM-RL two-stage paradigm, DPO and SLiC yield stronger
performance in multiple NLP applications with lower computational costs (Chen
et al., 2024; Yuan et al., 2024).

#### Learning from noisy data.

Data used in real-world machine learning applications are often noisy (c.f.,
Song et al. (2022)), and deep neural models are particularly sensitive to data
noise, as they are prone to overfit to noise patterns in the training data
(Han et al., 2020). Hence, multiple methods have been proposed to improve the
robustness of the neural models, mostly falling into four categories (Frénay &
Verleysen, 2014; Song et al., 2022): _robust neural architectures_ ,
_regularization methods_ , _robust loss functions_ , and _data filtering
methods_. In this paper, we stick to the well established architecture
(Transformers by Vaswani et al. (2017)) and loss functions (DPO), and explore
different regularization and data selection methods to mitigate the negative
impact of noisy preferences on the alignment performance.

The impact of noisy data on RL has also been studied. Lee et al. (2021a)
propose the _B-Pref_ benchmark, a platform to test the performance and
robustness of preference-based RL algorithms in the face of different types of
preference noise on various locomotion and robotic manipulation tasks. To
simulate realistic preference noise, they assume they have access to a gold-
standard reward model $r^{*}$, and design five strategies to derive noisy
preferences therefrom, including the _stochastic strategy_
($p(y_{0}>y_{1})=\sigma[r^{*}(y_{0})-r^{*}(y_{1})]$, where $\sigma$ is the
sigmoid function such that $\sigma(x)=(1+\exp(-x))^{-1}$), _myopic strategy_
(providing preferences only based on the last part of the presented
candidates), _skipping strategy_ (reject to provide preferences if both
candidates are low-quality), _equally-preferable strategy_ (when the quality
of the presented candidates are similar, mark them as a tie), and _random
mistake strategy_ (randomly flip the correct preference direction with a fixed
chance). We note that some of these strategies are unrealistic in GLM
alignment, e.g., the myopic strategy (users usually do not judge the quality
of texts based on their last parts), skipping strategy, and equally-preferable
strategy (ties are usually not allowed in annotating text qualities). Also,
they only consider noise from human annotators but ignore those from AI-based
annotators (e.g., RLAIF (Lee et al., 2024)). For these reasons, we propose a
new set of strategies for simulating preference noise in GLM alignment in §4.

##  3 Preliminaries

Let $\mathcal{X}$ be the set of all prompts, and $\mathcal{Y}_{\mathcal{X}}$
be the set of all possible continuations for all prompts in $\mathcal{X}$. We
assume there exists a gold reward model
$r^{*}:\mathcal{X}\times\mathcal{Y}_{\mathcal{X}}\to\mathbb{R}$, which
measures the quality of continuation $y\in\mathcal{Y}_{\mathcal{X}}$ for
prompt $x\in\mathcal{X}$ on some desired aspects (e.g., helpfulness,
informativeness, or harmlessness). A GLM can be defined as a policy $\pi$,
such that $\pi(y|x)$ is the probability of generating $y$ for the input prompt
$x$. The objective of _GLM alignment_ is to find the optimal policy that can
maximize the expected gold reward value while minimizing the divergence from a
reference policy:

$\displaystyle\max_{\pi}\sum_{x\in\mathcal{X},y\sim\pi(\cdot|x)}[r^{*}(y|x)]-\beta\mathbb{D}_{KL}[\pi_{\theta}(y|x)||\pi_{\text{sft}}(y|x)],$
(1)

where $\beta$ is a hyperparameter, $\mathbb{D}_{KL}$ is the Kullback–Leibler
divergence, and $\pi_{\text{sft}}$ is the reference policy.

In practice, we cannot optimize Eq. (1) directly, because the gold reward
$r^{*}$ is usually inaccessible, and the summation operation is prohibitively
expensive. Multiple algorithms have been proposed to obtain (approximate)
solutions for the objective (see §2). In this work, we use DPO (Rafailov et
al., 2023) because of its strong performance and lower computational cost
compared to other methods (e.g., PPO). The loss function in DPO is:

$\displaystyle\mathcal{L}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\\{\log\sigma[\beta\log(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{sft}}(y_{w}|x)})-\beta\log(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{sft}}(y_{l}|x)})]\\},$
(2)

where $\pi_{\theta}$ is the learnable policy parameterized by $\theta$,
$\sigma$ is the sigmoid function, and
$\mathcal{D}=\\{(x^{i},y^{i}_{w},y^{i}_{l})\\}_{i=1}^{n}$ is the training
dataset consisting of $n$ data entries. Each data entry in $\mathcal{D}$
consists of a prompt $x^{i}\in\mathcal{X}$ and two continuations
$y^{i}_{w},y^{i}_{l}\in\mathcal{Y}_{\mathcal{X}}$, such that $y^{i}_{w}$ is
preferred over $y^{i}_{l}$.

It has been proven that if $\mathcal{D}$ is sufficiently large and all pairs
in $\mathcal{D}$ are noise-free (i.e., $r^{*}(y^{i}_{w})>r^{*}(y^{i}_{l})$ for
$i=1,\cdots,n$), the policy learned by DPO is (near-)optimal with respect to
Eq. (1) (Rafailov et al., 2023; Azar et al., 2023). However, in practice, some
preferences in $\mathcal{D}$ can be _noisy_ , i.e., different from the
preference direction induced by the gold reward model. In this work, we remove
the (strong) noise-free assumption on $\mathcal{D}$, but instead introduce
different rates and types of noise to $\mathcal{D}$ (in §4) and empirically
study their impact on the quality of $\pi_{\theta}$ (in §6).

To measure the alignment performance (i.e., measure the performance of
$\pi_{\theta}$), we compute the _win rate_ between $\pi_{\theta}$ and
$\pi_{\text{sft}}$:

$\displaystyle
w=\frac{1}{|\mathcal{X}_{\text{test}}|}\sum_{x\in\mathcal{X}_{\text{test}}}\mathbbm{1}[r^{*}(x,y_{\pi_{\theta}})>r^{*}(x,y_{\pi_{\text{sft}}})],$
(3)

where $\mathcal{X}_{\text{test}}\subset\mathcal{X}$ is a held-out test prompt
set, and $y_{\pi_{\theta}}$ and $y_{\pi_{\text{sft}}}$ are generations sampled
from $\pi_{\theta}$ and $\pi_{\text{sft}}$, respectively.

##  4 Noisy Preferences

The preferences are often _noisy_ , i.e., disagree with the preference
directions induced by the gold reward model $r^{*}$. Inspired by past works
(Lee et al., 2021a) (see §2 for more discussions), we consider three _oracles_
to provide different types of noisy preferences.

  * •

Random Noise Oracle. When presented with a prompt $x\in\mathcal{X}$ and a pair
of responses $y_{w},y_{l}\in\mathcal{Y}_{\mathcal{X}}$, the oracle has
$(100-n)$% chance to return the correct preference (i.e.,
$r^{*}(y_{w}|x)>r^{*}(y_{l}|x)$), but has $n$% chance to return the
incorrect/flipped preferences. We can control the noise rate of this oracle by
adjusting the value of $n$.

  * •

Stochastic Noise Oracle. For a prompt $x$ and two responses $y_{w},y_{l}$,
Stochastic Noise Oracle prefers $y_{w}$ over $y_{l}$ with the probability
$\sigma[(r^{*}(y_{w})-r^{*}(y_{l}))/\gamma]$, where $\sigma$ is the sigmoid
function, and $\gamma\in\mathbb{R}^{+}$ is the temperature hyperparameter. We
can control the noise rate by tuning the $\gamma$ value: The higher the
$\gamma$ value, the more unpredictable the oracle is, and hence more noisy the
preferences will be.

  * •

Gaussian Noise Oracle. Stochastic Noise Oracle requires access to the gold
reward model, which is infeasible in practice. Gaussian Noise Oracle, instead,
only requires the access to an approximated reward model $r^{\prime}$, such
that $r^{\prime}(y|x)=r^{*}(y|x)+\epsilon$, where $\epsilon$ is the _noise
term_ drawn from a Gaussian distribution $\mathcal{N}(\mu,\delta^{2})$. The
preference directions are then derived from the approximated reward
$r^{\prime}$. With $r^{\prime}$, the probability of Gaussian Noise Oracle
prefers $y_{w}$ over $y_{l}$ is:

$\displaystyle p(y_{w}>y_{l}|x)=$
$\displaystyle\mathbbm{1}[r^{\prime}(y_{w}|x)-r^{\prime}(y_{l})]$
$\displaystyle=$
$\displaystyle\mathbbm{1}[(r^{*}(y_{w}|x)+\epsilon_{w})-(r^{*}(y_{l}|x)+\epsilon_{l})]$
$\displaystyle=$
$\displaystyle\mathbbm{1}[(r^{*}(y_{w}|x)-r^{*}(y_{l}|x))+(\epsilon_{w}-\epsilon_{l})].$

Since both $\epsilon_{w}$ and $\epsilon_{l}$ are drawn from the same Gaussian
distribution $\mathcal{N}(\mu,\delta^{2})$, $\epsilon_{w}-\epsilon_{l}$ is a
random variable drawn from $\mathcal{N}(0,2\delta^{2})$. Hence, the noise rate
of Gaussian Noise Oracle can be adjusted by tuning the value of $\delta$.

We believe the three strategies cover some widely observed noise types in
preferences. For example, Stochastic Noise Oracle is also known as _Boltzmann
rational_ (Ziebart et al., 2008; Jeon et al., 2020; Gao et al., 2020) and
widely used for simulating noise in human-provided preferences caused by
_aleatoric uncertainty_ (Hüllermeier & Waegeman, 2021). Gaussian Noise Oracle
simulates the noise caused by the _epistemic uncertainty_ , i.e., the RMs fail
to accurately approximate the human’s preferences. Random Noise Oracle
simulates the random mistakes observed in both human-provided (Lindner & El-
Assady, 2022) and heuristic-based preferences (Chen et al., 2024). We note
that in reality, multiple types of noise can co-exist in the preference pairs;
however, to ease analyses, in this paper, we assume there exist at most one
type of noise in the preferences. We leave mixed-type preference noise for
future work.

##  5 Experimental Setup

#### Tasks.

We consider preference-based GLM alignment on two tasks: Reddit TL;DR
(Stiennon et al., 2020) and Anthropic-Helpful (Bai et al., 2022a). Reddit
TL;DR has two subsets, a SFT set and a preference set. In its preference set,
each data entry consists of a document $x$ and two candidate summaries
$y_{w},y_{l}$ for $x$. It has 93k/53k/33k data entries in
train/validation/test splits. Anthropic-Helpful is a subset of the AnthropicHH
dataset, in which each data entry contains the dialogue history between a
human and an AI assistant ($x$), and two candidate responses ($y_{w},y_{l}$).
It has 161k/9k data entries in train/test splits, and we separate out 1k
randomly-sampled entries from the train set as the validation set.

#### Generative Language Model.

For each task, we fine-tune a T5-Large (770M parameters) model to obtain the
initial GLM $\pi_{\text{sft}}$. For TL;DR, we use the SFT subset in Reddit
TL;DR as the SFT training data, which has 117k/6k/6k examples in
train/validation/test splits. For Anthropic-Helpful, we use all the preferred
responses in the dataset as the SFT training data. All the hyperparameters
used in SFT are the same as in (Liu et al., 2024).

#### Gold Reward Model $r^{*}$.

For each task (TL;DR and Anthropic-Helpful), we train a T5-XXL (11B
parameters) (Raffel et al., 2020) model with the respective preference pairs
to build the gold reward model $r^{*}$. In line with (Zhao et al., 2023), we
format the input to the model with the prompt: [CONTEXT] {$x$} [RESPONSE]
{$y$}, and use the logit of the token 1 as a point-wise score for the reply.

#### Noisy Preference Oracles.

Based on the gold reward model $r^{*}$ described above, we build the noisy
preference oracles as described in §4. The noise rate of different noisy
preference oracles can be controlled by tuning their respective
hyperparameters: $n$ (noise rate) for Random Noise Oracle, $\gamma$
(temperature) for Stochastic Oracle, and $\delta$ (standard deviation) for
Gaussian Noise Oracle. To decide the exact values of the hyperparameters for a
target noise rate (e.g., 20%), we randomly sample 1k examples from each train
set, and increase the hyperparameters with a small step ($0.01$) until the
target noise rate is reached. The final hyperparameter values are presented in
Table 2.

|  | 5% | 10% | 15% | 20% | 25% | 30% | 35% | 40% | 45% | 50%
---|---|---|---|---|---|---|---|---|---|---|---
TL;DR | $\gamma$ | 0.2 | 0.4 | 0.65 | 0.9 | 1.25 | 1.75 | 2.50 | 3.90 | 8.0 | 100
$\delta$ | 0.34 | 0.70 | 1.09 | 1.55 | 2.15 | 2.90 | 4.05 | 6.50 | 13.0 | 100
Anthropic | $\gamma$ | 0.11 | 0.22 | 0.36 | 0.53 | 0.75 | 1.06 | 1.54 | 2.45 | 4.95 | 100
$\delta$ | 0.18 | 0.40 | 0.62 | 0.89 | 1.25 | 1.75 | 2.49 | 4.00 | 8.75 | 100
Table 2:  Hyperparameters for the Stochastic Noise ($\gamma$) and Gaussian
Noise Oracle ($\delta$) at each target noise rate (column). The hyperparameter
for Random Noise Oracle ($n$; see §4) is omitted, as it equals the
corresponding target noise rate.

#### Generation Pairs.

In line with Liu et al. (2024), we use the trained GLM $\pi_{\text{sft}}$ to
sample responses for each prompt, and pair up the sampled responses to build
the Generation Pairs dataset (see Fig. 1). For each prompt $x$, we sample
eight generations from $\pi_{\text{sft}}(\cdot|x)$ with temperature $0.7$, and
randomly group them into four pairs. The pairs are then presented to the noisy
preference oracles to build the Preference Pairs dataset (see Fig. 1).

#### Other Hyperparameters.

We choose the hyperparameters by following the choices made in (Rafailov et
al., 2023; Liu et al., 2024): in training the gold reward model and GLM
$\pi_{\text{sft}}$, we use batch size 32 and learning rate 1e-5 with Adafactor
optimizer (Shazeer & Stern, 2018); in the alignment training stage (Eq. (2)),
we use $\beta=0.5$, and dropout rate 0.1. Later in §7, we will explore
different values for the regularization weights ($\beta$ and dropout rate) to
study their effectiveness in mitigating the negative effect of noisy
preferences.

##  6 Impact of Noise Rates on Alignment Performance

Fig. 2 presents how the alignment performance changes with the growth of the
noise rates. We make the following observations.

Figure 2:  Influence of the noise rate (x-axis) on the alignment performance
(in terms of the win rate; y-axis). Error bars: 95% confidence intervals,
computed with double-tailed t-test on 5-10 repeated experiments with different
random seeds.

  * •

Alignment performance drops with more noise in preferences. This applies to
all types of noise and both tasks we have considered. Also, we note that the
alignment performance drops more quickly with the increase of the noise rates:
When the noise rates are below 0.3, an increase of 10 percentage points (pp)
in noise rate yields less than 10pp drop in the alignment performance;
however, when the noise rates are higher than 0.4, 10pp increase in noise can
yield 20-30 pp loss in performance.

  * •

Different types of noise cause similar harm. At the same noise rate, the
alignment performance of the three different noise types do not have
significant differences, suggesting that it is the noise rate rather than the
noise type that decides the alignment performance.

  * •

Alignment is beneficial even with highly noisy preferences. In both tasks, the
win rate is above $0.5$ even with noise rate at $0.45$. This observation
reaffirms the effectiveness of alignment training (Touvron et al., 2023), and
explains why even highly noisy preferences are used in alignment training in
practice (see Table 1). But it is also worth noting that when the preferences
are completely random (i.e., noise rate at 50%), the win rate drops below 0.5,
suggesting that alignment is detrimental with random preferences.

##  7 Mitigate the Negative Impact of Preference Noise

In this section, we explore two popular strategies to mitigate the negative
impact of preference noise: _regularization_ in §7.1 and _data filtering_ in
§7.2.

###  7.1 Regularization

We consider two methods to strengthen regularization: increasing the weight of
the KL divergence loss ($\beta$ in Eq. (2)), and increasing the dropout rate.

Fig. 3 presents the alignment performance with different strengths of KL
regularization. In general, we find that stronger KL regularization fails to
mitigate the negative impact of preference noise. In some cases (e.g., Random
and Stochastic Noise in TL;DR), higher strength of KL regularization even
hurts the performance. Our finding reaffirms the limitations of KL-based
regularization in DPO/SLiC (Azar et al., 2023).

Figure 3: Alignment performance with different KL regularization weights
$\beta$.

Fig. 4 presents the alignment performance with different dropout rates. We
find that higher dropout rates significantly harm the alignment performance.
To summarize, our findings suggest that high strength of regularization, in
general, fails to mitigate the negative impact of preference noise, and in
certain cases can even hurt the alignment performance.

Figure 4: Alignment performance with different dropout rates (dor).

###  7.2 Data Filtering

Another approach to fight against noise is to filter out noisy data in the
Preference Pairs dataset (see Fig. 1) and only use the remaining data to
perform alignment. We use the popular _confidence-based_ data filtering method
(Cheng et al., 2008). For a prompt $x$ and two candidate responses
$y_{w},y_{l}$, the _confidence of $y_{w}$ preferred over $y_{l}$_, denoted
$c(y_{w}>y_{l}|x)$, is a real value between 0 and 1. Ideally, the confidence
function $c$ should be _well-calibrated_ (Silva Filho et al., 2023), i.e.,
$c(y_{w}>y_{l}|x)$ is identical to the true probability of $y_{w}$ preferred
over $y_{l}$. In confidence-based data filtering, only pairs with the
confidence level larger than a pre-defined threshold $t$ are used in training;
hence, if threshold $t=0.5$, no filtering is performed; $t=1$ means all data
will be removed. We experiment with $t=0.5,0.6,\cdots,0.9$ to investigate the
effect of different levels of data filtering.

Based on the user studies made by Gao et al. (2020), we use the Bradley &
Terry (1952) model to estimate the confidence value:
$c(y_{w}>y_{l}|x)=\sigma[r^{*}(y_{w}|x)-r^{*}(y_{l}|x)]$, where $\sigma$ is
the sigmoid function. In practice, there are multiple methods for estimating
the confidence values, e.g., _conformal predictors_ (Shafer & Vovk, 2008;
Einbinder et al., 2022), _ensemble_ methods (Liang et al., 2022), and _bias
estimation_ methods (Chen et al., 2023).

Note that with higher filtering threshold $t$, the _quality_ of the remaining
data is improved at the cost of _quantity_ loss. We deliberately do not back-
fill the filtered data, because in practice it can be prohibitively expensive
to collect more preference pairs. This setup also allows us to study the
trade-off between data quality and data quantity in preference-based
alignment. Fig. 6 in Appendix A shows how the size of the remaining data
shrinks with higher confidence thresholds. We find that in both datasets, the
data size drops quite quickly with the growth of $t$ values: Almost 20% data
are filtered as $t$ increases by 0.1.

Fig. 5 presents the alignment performance with different strengths of data
filtering. We make the following observations.

  * •

Data filtering does not help to fight against Random Noise. This is because
Random Noise Oracle flips pairs _completely at random_ (Frénay & Verleysen,
2014), i.e., the flipping chance of each pair is uniformly at random, not
affected by any other factors (e.g., the prompt $x$ or the responses
$y_{w},y_{l}$). As a result, our confidence-based filtering cannot reduce the
number of the noisy pairs in the filtered data, and hence fails to improve the
performance.

  * •

Data filtering is effective to mitigate the harm from Stochastic and Gaussian
Noise. We find that with certain threshold (e.g., at $0.8$), data filtering
shows consistent and significant improvement across all noise rates, noise
types, and tasks. Considering that over 50% preference pairs have been removed
with data filtering at confidence threshold $0.8$, this result suggests that
data quality has a significant impact on the alignment performance.

  * •

Over-aggressive data filtering hurts the performance. With very high
confidence thresholds (e.g., $0.99$), the alignment performance is compromised
across all noise rates, noise types, and tasks, due to the severe loss of data
size. Hence, it is important to properly trade off between data quality and
data quantity, in order to yield the optimal alignment performance.

Figure 5: Alignment performance with confidence-based data filtering.

To better understand how data filtering improves the data quality, we
investigate the noise rate before and after data filtering, with different
noise types. Fig. 7 in Appendix A shows the noise rates in data filtered with
different confidence thresholds. We find that when the noise is from Random
Noise Oracle, the noise rate stays the same with all data filtering
thresholds; this explains why data filtering does not help improve its
performance. When the noise is from Stochastic or Gaussian Oracles, data
filtering can effectively reduce noise rate, explaining the performance boost
observed in Fig. 5.

##  8 Limitations & Future Work

#### Generalizability.

We apply a popular alignment algorithm (DPO) to T5-based language models in
our experiments. We believe our observations presented in §6 and §7 can be
generalized to other alignment methods (e.g., PPO, SLiC and IPO, because they
essentially optimize the same objective function; see §2) and other
Transformer (Vaswani et al., 2017) based language models, but this is yet to
be empirically verified. Considering the large number of possible combinations
between alignment techniques and language models, a systematic study is beyond
the scope of this paper, and we leave it for future work.

#### Mixture of Noise.

We consider three strategies to add noise to the preferences, but we only
allow one strategy to be used at one time (see §4). In practice, there may
exist other types of noise, and different types of noise can co-exist in
preferences. Our framework allows for creating new noise types by mixing
primitive noise types, but it remains unclear whether the current observations
can be applied to the new noise types or not. We hope our work can facilitate
and encourage more work on this direction.

##  9 Conclusion

Pairwise preferences are widely used for aligning Generative Language Models
(GLMs) with human values, but it remains unclear how the _noise in
preferences_ affect the alignment performance, and how to mitigate their
negative impact. To study these problems, we propose a framework in which the
types and rates of noise can be simulated and controlled, and we perform
systematic experiments on two generation tasks (summary and dialogue
generation) with this framework. Our findings suggest that alignment
performance can be highly sensitive to the increase of noise rates, and
appropriate data filtering is the most effective method to mitigate the
negative impact of noisy preferences. Our work builds the first quantitative
relation between noise rates and alignment performance across different noise
types. We hope our work can help the community better understand and mitigate
the impact of preference noise in GLM alignment.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A general theoretical paradigm to understand learning from human preferences.  _arXiv preprint arXiv:2310.12036_ , 2023.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Böhm et al. (2019) Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych.  Better rewards yield better summaries: Learning to summarise without references.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pp. 3108–3118. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/D19-1307.  URL https://doi.org/10.18653/v1/D19-1307.
  * Bradley & Terry (1952) Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Chen et al. (2023) Lin Chen, Michal Lukasik, Wittawat Jitkrittum, Chong You, and Sanjiv Kumar.  It’s an alignment, not a trade-off: Revisiting bias and variance in deep models.  _arXiv preprint arXiv:2310.09250_ , 2023.
  * Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  _arXiv preprint arXiv:2401.01335_ , 2024.
  * Cheng et al. (2008) Reynold Cheng, Jinchuan Chen, and Xike Xie.  Cleaning uncertain data with quality guarantees.  _Proceedings of the VLDB Endowment_ , 1(1):722–735, 2008.
  * Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback.  _arXiv preprint arXiv:2310.01377_ , 2023.
  * Einbinder et al. (2022) Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou.  Training uncertainty-aware classifiers with conformalized deep learning.  _Advances in Neural Information Processing Systems_ , 35:22380–22395, 2022.
  * Frénay & Verleysen (2014) Benoît Frénay and Michel Verleysen.  Classification in the presence of label noise: A survey.  _IEEE Trans. Neural Networks Learn. Syst._ , 25(5):845–869, 2014.  doi: 10.1109/TNNLS.2013.2292894.  URL https://doi.org/10.1109/TNNLS.2013.2292894.
  * Gao et al. (2020) Yang Gao, Christian M. Meyer, and Iryna Gurevych.  Preference-based interactive multi-document summarisation.  _Inf. Retr. J._ , 23(6):555–585, 2020.  doi: 10.1007/S10791-019-09367-8.  URL https://doi.org/10.1007/s10791-019-09367-8.
  * Gemma (2024) Gemma.  Gemma: Open models based on gemini research and technology.  Technical report, Google DeepMind, 2024.
  * Google (2023) Gemini Team Google.  Gemini: A family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Han et al. (2020) Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W. Tsang, James T. Kwok, and Masashi Sugiyama.  A survey of label-noise representation learning: Past, present and future.  _CoRR_ , abs/2011.04406, 2020.  URL https://arxiv.org/abs/2011.04406.
  * Hüllermeier & Waegeman (2021) Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 110:457–506, 2021.
  * Jeon et al. (2020) Hong Jun Jeon, Smitha Milli, and Anca Dragan.  Reward-rational (implicit) choice: A unifying formalism for reward learning.  _Advances in Neural Information Processing Systems_ , 33:4415–4426, 2020.
  * Ji et al. (2023) Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al.  Ai alignment: A comprehensive survey.  _arXiv preprint arXiv:2310.19852_ , 2023.
  * Kendall (1948) Maurice George Kendall.  Rank correlation methods.  1948\.
  * Kingsley & Brown (2010) David C Kingsley and Thomas C Brown.  Preference uncertainty, preference learning, and paired comparison experiments.  _Land Economics_ , 86(3):530–544, 2010.
  * Lee et al. (2024) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  In _International Conference on Learning Representations_ , 2024.
  * Lee et al. (2021a) K Lee, L Smith, A Dragan, and P Abbeel.  B-pref: Benchmarking preference-based reinforcement learning.  _Neural Information Processing Systems_ , 2021a.
  * Lee et al. (2021b) Kimin Lee, Laura Smith, and Pieter Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  _arXiv preprint arXiv:2106.05091_ , 2021b.
  * Liang et al. (2022) Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel.  Reward uncertainty for exploration in preference-based reinforcement learning.  _arXiv preprint arXiv:2205.12401_ , 2022.
  * Lindner & El-Assady (2022) David Lindner and Mennatallah El-Assady.  Humans are not boltzmann distributions: Challenges and opportunities for modelling human feedback and interaction in reinforcement learning.  _arXiv preprint arXiv:2206.13316_ , 2022.
  * Liu et al. (2024) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu.  Statistical rejection sampling improves preference optimization.  In _International Conference on Learning Representations_ , 2024.
  * Munos et al. (2023) Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al.  Nash learning from human feedback.  _arXiv preprint arXiv:2312.00886_ , 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  In _Thirty-seventh Annual Conference on Neural Information Processing Systems_ , 2023.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shafer & Vovk (2008) Glenn Shafer and Vladimir Vovk.  A tutorial on conformal prediction.  _Journal of Machine Learning Research_ , 9(3), 2008.
  * Shazeer & Stern (2018) Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pp. 4596–4604. PMLR, 2018.
  * Silva Filho et al. (2023) Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach.  Classifier calibration: a survey on how to assess and improve predicted class probabilities.  _Machine Learning_ , 112(9):3211–3260, 2023.
  * Song et al. (2022) Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee.  Learning from noisy labels with deep neural networks: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022\.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Thurstone (2017) Louis L Thurstone.  A law of comparative judgment.  In _Scaling_ , pp. 81–92. Routledge, 2017.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_ , 2024.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena.  In _Neural Information Processing Systems, Datasets and Benchmarks Track_ , 2023.
  * Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.  Maximum entropy inverse reinforcement learning.  In _Aaai_ , volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.

##  Appendix A Appendix

Figure 6: With higher data-selection threshold, fewer data will be remained
for training.

Figure 7: Noise rates of the filtered data, with different data filtering
threshold. Note that when the threshold is $0.5$, no data is filtered. Here
the original/unfiltered data has 10 - 40 % noise.


# User Training with Error Augmentation for Electromyogram-based Gesture
Classification

Yunus Bicer†, Niklas Smedemark-Margulies†, Basak Celik, Elifnur Sunger, Ryan
Orendorff, Stephanie Naufel, Tales Imbiriba, Deniz Erdoğmuş, Eugene Tunik,
Mathew Yarossi,
†Equal Contribution  This project was funded by Meta Reality Labs Research.
This work involved human subjects or animals in its research. Yunus Bicer,
Basak Celik, Elifnur Sunger, Tales Imbiriba, Deniz Erdogmus, and Mathew
Yarossi are with Department of Electrical and Computer Engineering,
Northeastern University, Boston, MA 02115, USA. Niklas Smedemark-Margulies is
with Khoury College of Computer Sciences, Northeastern University, Boston, MA
02115, USA. Eugene Tunik and Mathew Yarossi are with the Department of
Physical Therapy, Movement, and Rehabilitation Sciences, Northeastern
University, Boston, MA 02115, USA. Stephanie Naufel is with Meta Reality Labs
Research, Menlo Park, CA 94010, USA. (e-mail<EMAIL_ADDRESS>
###### Abstract

We designed and tested a system for real-time control of a user interface by
extracting surface electromyographic (sEMG) activity from eight electrodes in
a wrist-band configuration. sEMG data were streamed into a machine-learning
algorithm that classified hand gestures in real-time. After an initial model
calibration, participants were presented with one of three types of feedback
during a human-learning stage: veridical feedback, in which predicted
probabilities from the gesture classification algorithm were displayed without
alteration, modified feedback, in which we applied a hidden augmentation of
error to these probabilities, and no feedback. User performance was then
evaluated in a series of minigames, in which subjects were required to use
eight gestures to manipulate their game avatar to complete a task.
Experimental results indicated that, relative to baseline, the modified
feedback condition led to significantly improved accuracy and improved gesture
class separation. These findings suggest that real-time feedback in a gamified
user interface with manipulation of feedback may enable intuitive, rapid, and
accurate task acquisition for sEMG-based gesture recognition applications.

###### Index Terms:

Myoelectric control, Gesture recognition, Human-computer interaction, Error
augmentation, Co-adaptation, Surface Electromyography (sEMG),

##  I Introduction

Surface electromyography (EMG) provides a convenient sensor modality for
human-computer interaction (HCI) applications [1]. In the past two decades,
research efforts have sought to translate the electrical activity associated
with muscle contraction into control commands for general use computing,
prosthetic control, and motor rehabilitation [2, 3].

Traditional approaches to EMG-based gesture recognition assumed stationarity
of the muscle activation to gesture mapping, and did not consider the user’s
ability to adapt their behavior to feedback about the performance of the
algorithm used for gesture classification. The emergence of co-adaptive
learning algorithms in the past decade represented a marked shift,
acknowledging the human and machine learning as part of integrated system [4,
5, 6, 7, 8]. One key finding from these approaches is that when the human
receives continuous feedback about the mapping of muscle activation to
gesture, they can increase classification performance [9] through behavioral
adaptation to increase class separability [10] or increase movement
repeatability [11]. However, to-date, little attention has been paid to how
feedback about about classifier performance impacts behavioral adaptations of
human learner.

The ability to shape human behavioral adaptation and motor skill learning
through the use of augmented feedback is well established. Strategies such as
error augmentation [12, 13, 14] and reward manipulation [15, 16] have been
shown to affect the rate and retention of learning as well as the behavioral
variability. Yet, to our knowledge the use of augmented feedback has not been
tested for co-adaptation approaches to EMG-based gesture recognition.

In this study subjects were given real-time feedback on predicted gesture
probabilities from a machine learning model under three conditions: no
feedback, veridical feedback, and modified feedback via error augmentation.
Participants were asked to freely explore gesture positions while viewing
feedback in order to find optimal muscle activation patterns for producing a
desired gesture using a fixed model. Modified feedback was produced by a
hidden softening of the model probabilities toward a uniform distribution, to
guide participants to increase the class separability of their muscle
activation patterns. We hypothesized that users who explored hand positions
under the more challenging modified feedback condition would learn to create
more distinct gestures, thereby improving future classification performance.

##  II Experimental Design

All protocols were approved by the Northeastern University Institutional
Review Board (IRB number 15-10-22) in conformance with the declaration of
Helsinki.

###  II-A Subjects

Forty-four right-handed subjects (21 male/ 23 female, mean age $\pm$ 1
standard deviation: $20.9\pm 4.3$ years) participated after providing IRB-
approved written informed consent. Subjects were free of orthopedic or
neurological disease that could interfere with the task and had normal or
corrected-to-normal vision.

###  II-B Experimental Setup

Figure 1: Electrode Placement. sEMG data is collected using $8$ Delsys Trigno
sEMG sensors uniformly spaced around the right forearm.

Subjects viewed a computer display while seated at a table with their right
arm positioned comfortably in an armrest trough. Surface electromyography
(sEMG) (Trigno, Delsys Inc., sampling frequency: 1926 Hz) was collected from
the muscles of the right forearm. Eight sEMG electrodes were positioned at
equidistant positions around the circumference of the forearm, at a four
finger-width distance from the ulnar styloid (the subject’s left hand was
wrapped around the right forearm at the ulnar styloid to determine the sEMG
placement). The first electrode was placed mid-line on the dorsal aspect of
the forearm, and the other electrodes were then equally spaced (see Figure 1).

###  II-C Data Acquisition

Subjects were randomly assigned to one of three groups and performed a series
of tasks as described below. Subjects who were unable to complete all tasks
were excluded from further analysis. Each subject group was assigned a
different feedback condition: no feedback (“Control”, N=$14$), veridical
feedback (“Veridical”, N=$14$), or modified feedback (“Modified”, N=$16$) (see
Section II-C4 for details).

####  II-C1 Gesture Timing

Subjects performed a series of tasks composed of one or more gesture trials to
move an avatar dice (see details of user interface below). Prior to the start
of a trial, the subject’s forearm and wrist rested in a pronated position on
the trough with the wrist neutral. In each trial, subjects were required to
rest or to produce one of eight active gestures (label and action provided in
brackets): index-thumb pinch [“Pinch”, decrease number on avatar dice], index-
thumb key press [“Thumb”, increase number on avatar dice], closed fist
[“Fist”, decrease size of avatar dice], full finger extension [“Open”,
increase size of avatar dice], wrist extension [“Up”, move up], wrist flexion
[“Down”, move down], wrist ulnar deviation [“Left”, move left], radial wrist
deviation [“Right”, move right]. Each trial began with a ‘prompting’ epoch
($3$ sec) cued by a yellow bounding box the participant’s display and a
picture of the instructed gesture (Calibration and Instructed blocks only, see
below), a ‘gesture production’ epoch ($2$ sec) cued by a green bounding box,
and a ‘recovery’ epoch ($3$ sec) cued by a red bounding box. See Figure 2 for
example timing.

Figure 2: Gesture Trial Timing. In the yellow ‘prompting’ epoch, the subject
sees an instruction. In the green ‘gesture production’ epoch, the subject
performs the gesture. In the red ‘recovery’ epoch, the subject returns to the
rest position. Features for classification are extracted from the last $500$
ms of gesture production to help ensure that steady-state features are
collected.

Each session was divided into four blocks.

####  II-C2 Block One: Calibration

Subjects from all groups were instructed to perform five consecutive
repetitions of each active gesture and eight repetitions of a rest gesture
[“Rest”] in which they were asked to relax the hand. This consecutive
structure was chosen to help keep the task simple while the participant
initially learned the set of available gestures. A classification model was
trained on this small dataset before continuing to the next experimental
block.

####  II-C3 Block Two: Instructed Games

Subjects from all groups engaged in four practice mini-games. In each mini-
game, subjects were instructed to perform a sequence of six gestures to bring
an avatar that was shown on the computer screen from a starting position to a
desired goal state (e.g. see Figure 3). The trial timing epochs (prompting,
gesture production, and rest) were as described above. In this block, the
classifier model’s predicted probabilities were displayed as post-hoc feedback
to the user, but was not used to modify the avatar position or state; the
avatar always moved one step closer to the goal after each trial, so that each
game lasted exactly six moves. These games were structured so that the $24$
total gestures ($4$ games with $6$ moves each) were evenly distributed among
the $8$ active gestures. After this block, the classification model was
retrained from scratch using the labeled data from blocks one and two. This
training set comprised $8$ examples for each of the $9$ classes ($8$ active
gestures and “Rest”).

Figure 3: Example mini game. The blue player avatar must be moved to match the
gray target avatar. The minimal path includes moving right, down twice,
decreasing die number (using a pinch gesture) and reducing size (using a fist
gesture).

####  II-C4 Block Three: Live Feedback

Only subjects in the veridical feedback and modified feedback groups
participated in this block. Subjects engaged in mini-games while receiving
different types of real-time feedback ($30$ sec duration in the ‘gesture
production’ epoch to produce active gestures). Subjects were asked to freely
explore their hand posture in order to maximize the predicted probability of
the current gesture class, visible as real-time output of the trained model.
For the veridical feedback group, predicted class probabilities were displayed
without modification, whereas the modified feedback group was shown
probabilities that were softened. This softening procedure is described in
detail in Section III-C, and serves to adjust the model’s predicted
probabilities towards a uniform distribution, with the hope that this
encourages participants to compensate by performing more precise gestures.
Subjects in the modified feedback group were not informed about this softening
procedure.

####  II-C5 Block Four: Free Games

All subjects were instructed to perform a series of $12$ mini-games. The mini-
games had the same structure as in block two, with each game requiring a
minimum of six moves to bring the avatar from its starting position to a
desired goal state. However, unlike the practice mini-games of block two,
subjects were tasked with bringing the avatar to its goal state by planning
and producing a gesture sequence of their choice. Critically, the avatar only
changed its state when the classifier assigned one class a predicted
probability above a decision threshold of $0.5$. The experimenter manually
recorded each attempted gesture to serve as labels for subsequent analysis,
and the participant’s hand movements were also recorded on video to cross-
check these labels.

##  III Signal Modeling

###  III-A Feature Extraction

As described in Section II-C1, we extracted raw data for classification from
the final $500$ ms of the active gesture production period of each gesture
trial. From each of the $8$ sensor channels of raw sEMG, we computed the Root-
Mean-Square (RMS) and median frequency after Fourier transform, resulting in
$16$-dimension features. Given a data vector $x$, RMS is defined as:

$\displaystyle\textrm{RMS}(x)=\sqrt{\frac{1}{N}\sum_{i=1}^{N}x_{i}^{2}}~{}.$
(1)

The Median Power Frequency is defined as the frequency value
$f_{\textsc{med}}$ that divides the Power Spectral Density (PSD) into two
regions with equal power [17]:

$\displaystyle\int_{0}^{f_{\textsc{med}}}\\!\textrm{PSD}(f)df=\int_{f_{\textsc{med}}}^{\infty}\\!\textrm{PSD}(f)df=\frac{1}{2}\int_{0}^{\infty}\\!\textrm{PSD}(f)df~{}.$
(2)

###  III-B Classification Model

Given extracted features, we used a two-stage classification pipeline to
predict among $9$ possible gestures: Up, Thumb, Right, Pinch, Down, Fist,
Left, Open, Rest. The classification model consisted of an encoder formed from
Support Vector Machine (SVM) models that produced a latent representation, and
a logistic regression classifier that produced predicted class probabilities.
In the encoder portion of the model, we trained a one-vs-one (OVO) SVM
classifier [18] for each of the $\binom{9}{2}=36$ pairs of gestures. Each of
these OVO-SVM models produced a scalar output (representing the probability of
assigning to the first of its two classes); these $36$ scalars were stacked
into a latent vector and passed to the logistic regression model.

Given a supervised training dataset, we first fit the one-vs-one SVM models
using linear programming with the CVXPY Python library [19]. The linear
programming objective we used was based on the semi-supervised SVM formulation
of [20], to allow future semi-supervised extensions. Specifically, the SVM
parameters were trained according to the following optimization problem:

$\displaystyle\min_{w,b,\eta}C\sum_{i=1}^{l}\eta_{i}+\frac{1}{2}\|w\|^{2}$ (3)
s.t. $\displaystyle\;\;y_{i}(wx_{i}-b)+\eta_{i}\geq 1,\;\;\eta_{i}\geq
0,\;\;i=1,\ldots,l$

where $w,b$ were the parameters to be optimized, $\eta_{i}$ were slack
variables allowing misclassification of individual points, and $C>0$ is a
fixed penalty parameter controlling the margin’s strictness.

We implemented the logistic regression classifier with the PyTorch Python
library [21] using a single linear layer and a SoftMax function. After the SVM
encoder portion of the model was trained, it was held fixed while the logistic
regression classifier model was trained by stochastic gradient descent to
minimize the cross-entropy loss. We trained the classifier model for $1000$
epochs with a batch size of $20$ and AdamW [22] optimizer. See Algorithm 1 for
a summary of our classifier training procedure.

##### Smoothing

As noted, participants in the veridical feedback and modified feedback groups
were shown real-time output from the model. Due to the high sampling frequency
of the sEMG sensors used, and the relatively computationally simple prediction
model, the system was capable of making very fast adjustments to the predicted
output, which can result in unwanted jitter due to slight fluctuations in raw
signal or hand positioning. Therefore, we used an exponential moving average
(EMA) to smooth the model’s predictions in time. At time-step $t$, the model
produces a raw probability vector $\tilde{P}^{(t)}$, which is then mixed with
the previous probability vector using a momentum parameter $\lambda$ to
produce a smoothed vector $P_{\textsc{EMA}}^{(t)}$:

$\displaystyle P_{\textsc{EMA}}^{(t)}=\lambda
P_{\textsc{EMA}}^{(t-1)}+(1-\lambda)\tilde{P}^{(t)}.$ (4)

For values of $\lambda$ close to $1$, this causes the probability vector to
update more slowly.

Input : Features $X$, Labels $Y$

Output : OVO SVM parameters $w$, $b$, Classifier parameters $\theta$

1 Initialize $w,b,\theta$ randomly;

Train $w,b$ on $(X,Y)$ ;

// See Eq. 3

$S\leftarrow\textrm{OVO-SVM}(X)$ ;

// SVM scores

2 Train Classifier on $(S,Y)$ ;

3 return $w,b,\theta$ ;

Algorithm 1 Classifier Training Procedure

###  III-C Modified Feedback

As mentioned above, subjects in the modified feedback group were shown
modified real-time output from the trained classifier during Block Three of
the experiment. Specifically, the vector of predicted probabilities from the
model was modified according to the following formula:

$\displaystyle
P_{\textsc{modified}}=\frac{[P_{\textsc{EMA}}]^{m}}{\sum\limits_{c\in
C}[P_{\textsc{EMA}}]^{m}},$ (5)

where the modification exponent $m$ was set to $0.75$, and $C$ represents the
$8$ classes used.

###  III-D User Interface and Software Design

Figure 4 shows the user interface (UI) displayed to participants. All
components of the UI were implemented using PyQt [23] Python package. On the
top left, the UI displayed an instructed gesture via image and text during
blocks one and two (see Section II-C2 and II-C3). On the bottom left, the UI
showed post-hoc predicted probabilities for each gesture as a radial plot. The
length of each line was scaled according to the value; the outer circle
represented a value of $1$, and the inner circle represented a value of $0.5$
(i.e. the model’s decision threshold). The opacity of gesture images around
the radial plot were also scaled according to the value. The outer edge of the
UI was colored yellow, green, or red to indicate gesture timing epoch as
described in Section II-C1. On the right of the UI was the task window in
which the mini-games were played during blocks two and four (see Section II-C3
and II-C5). As described previously, participants used one of $8$ active
gestures to move their avatar (the blue die). The goal of each mini-game in
blocks two and four was to use these gestures to match the blue die to the
gray target die.

Figure 4: The participant User Interface. Top left: instructed gesture. Bottom
left: predicted gesture probabilities. Right: Task window including subject’s
avatar and target. Outer edge: gesture epoch indicator.

##### Error Augmentation in Live Feedback

During block three (see Section II-C4), participants who received real-time
feedback were presented with a different display, as shown in Figure 5. Here,
the probability of each class was displayed using a bar plot that was updated
in real-time. The participant’s goal during this block of the experiment was
to explore hand positions in order to maximize the predicted probability of
the current gesture class.

Figure 5: Top: Real-time probability feedback window. The horizontal line at
$0.5$ shows the decision threshold. Bottom: Example of probability values
without modification (“Veridical”) and with modification (“Modified”) as
described in Sec. III-C.

##### Data Streaming with LabGraph

Our experimental platform was built using the LabGraph[24] Python package. The
software was implemented as a collection of LabGraph nodes to separately
manage raw data collection, preprocessing, feature extraction, classification,
and UI. These nodes were connected using a declarative syntax in a directed
acyclic graph. At runtime, LabGraph spawned processes to run each node, and
automates both the delivery and logging of messages.

###  III-E Classifier Metrics

As mentioned in Section II-C5, the experimenter recorded each intended gesture
made by the participant, so that model accuracy could be evaluated after-the-
fact. Accuracy was defined as the fraction of correctly classified items. In
addition to the $8$ active gestures and the “rest” class, the decision
threshold of $0.5$ that was used resulted in another possible outcome for
gesture trials when no gesture rose above the decision threshold, which we
refer to as “NoClass”. Gesture trials in which the subject was not prepared to
make a gesture during the “gesture production” epoch were recorded as having a
true label of “rest”.

###  III-F Feature-Space Class Structure

To evaluate how feedback affects human learning, we analyzed the feature-space
distribution of trials from different gestures performed in block four of the
experiment. This feature-space representation does not depend on the model,
since these features are obtained using simple, deterministic transformations
of the raw data (RMS and median frequency after Fourier transform). The
differences in feature-space class structure across treatment groups can
therefore give information about human learning.

##### Kernel Similarities

We base our analysis of feature-space structure on a Radial Basis Function
(RBF) kernel similarity measure. The RBF kernel computes a similarity measure
which corresponds to an implicit infinite-dimensional vector space. For two
feature vectors $x,x^{\prime}$ belonging to a dataset $X$ and a length scale
parameter $\gamma\in\mathbb{R}$, the RBF kernel similarity is computed as:

$\displaystyle RBF(x,x^{\prime},\gamma)$
$\displaystyle=\exp\left(-\gamma\|x-x^{\prime}\|^{2}\right).$ (6)

The length scale $\gamma$ is an important hyperparameter which determines the
rate at which similarities decay as two points are moved farther apart. We
follow the so-called “median heuristic” [25], in which $\gamma$ is set based
on the median length scale of a dataset $X$:

$\displaystyle\gamma_{\textsc{med}}$
$\displaystyle=1/\textrm{med}(\|x-x^{\prime}\|^{2},\ \forall\
(x,x^{\prime})\in\\{X\times X\\}).$ (7)

We set $\gamma_{\textsc{med}}$ individually for each subject, based on all of
their pooled gesture trials.

##### Class Similarity Matrices

We use this notion of kernel similarity to construct a class similarity matrix
for each subject. For classes $C_{1},\ldots,C_{\mathscr{C}}$, we build a
square, symmetric matrix $D\in\mathbb{R}^{(\mathscr{C}\times\mathscr{C})}$
such that the entry at position $(i,j)$ describes the average RBF kernel
similarity between items in classes $C_{i}$ and $C_{j}$:

$\displaystyle D_{ij}$ $\displaystyle=\frac{1}{|C_{i}||C_{j}|}\sum_{x\in
C_{i}}\sum_{x^{\prime}\in C_{j}}RBF(x,x^{\prime},\gamma_{\textsc{med}}).$ (8)

After computing the entries in a similarity matrix, we normalize the entries
to the range $[0,1]$ so that these matrices may be easily compared across
subjects and groups.

Classes which are closer together in feature space will have a higher average
similarity and therefore a larger entry in this similarity matrix. A subject
whose gestures are easily classifiable may tend to have precise gestures that
are also well-separated from each other. This would result in having high
average similarity between trials in the same gesture class (diagonal entries
of the class similarity matrix) and low average similarity between trials of
different classes (off-diagonal entries). See Section IV-D for class
similarity matrices from each experimental group, and see Figure 6 for
didactic examples of similarity matrix $D$.

##### Scalar Class Separation Measure

In order to look for trends in the feature-space distribution over time and to
identify global trends across groups, we also summarize these normalized class
similarity matrices using a scalar class separation measure,
$d_{\textsc{sep}}$, which we define as the average within-class similarity
divided by the average between-class similarity. Given a normalized similarity
matrix $D$ as described above,

$\displaystyle d_{\textsc{sep}}$
$\displaystyle=\left(\frac{1}{N}\sum_{i=1}^{N}D_{ii}\right)/\left(\frac{2}{N(N-1)}\sum_{i=2}^{N}\sum_{j=1}^{i-1}D_{ij}\right).$
(9)

As indicated above, larger within-class similarities indicate that trials from
the same gesture are precise and repeated with high-fidelity, while smaller
between-class similarities indicate that trials from different gestures are
easily distinguished. Thus, a dataset with a larger value of
$d_{\textsc{sep}}$ may contain gestures that will be more easily classified.

In Figure 6, we show examples of class similarity matrix $D$ and scalar
similarity measure $d_{\textsc{sep}}$. To produce an example that can be
easily visualized, we select a subject from the “Modified” condition that
showed a large improvement in feature-space separation. For this subject, we
select three gestures (“Left”, “Down”, and “Right”) and three features (RMS
value from electrodes 1, 4, and 7). In the top row, we show metrics for this
subject’s data during the “Calibration” and “Instructed” blocks, and in the
bottom row, we show metrics from the “Free” block; recall that the subject
experiences live feedback training after the “Instructed” block. We observe
that the features of each class become more distinct after the user performs
live feedback training; this is captured as an increase in the similarities on
the diagonal of $D$ and a decrease in similarities off-diagonal. These changes
in $D$ are also summarized in $d_{\textsc{sep}}$, which increases from $2.8$
to $3.55$.

Figure 6: Didactic example for class similarity matrices $D$ and scalar class
separation measure $d_{\textsc{sep}}$. For a chosen subject from the Modified
condition, we analyze $3$ of the original $16$ features (RMS value from
electrodes 1, 4, and 7) and a subset of gestures (“Left”, “Down”, and
“Right”). Top row: features from calibration and instructed blocks. Bottom
row: features from free games. Left: Scatter plot of $3$-dimensional features,
and scalar class separation value. Right: The corresponding class separation
matrix.

###  III-G Within-Subject Normalization

The focus of this work is to measure the effect of the proposed veridical and
modified feedback strategies on subject performance. We note that overall
subject performance may be influenced by a relatively large number of factors
of variation, such as factors affecting dexterity and motor precision, subject
motor learning speed, and subject-intrinsic factors affecting raw sEMG signal-
to-noise ratio. Thus, a prohibitively large sample size may be required to
natively account for this variation.

We instead adopt a within-subject normalization strategy, obtaining baseline
statistics for each subject using only data measured before our interventions.

For each subject, we measure baseline accuracy by training a model from
scratch using that subject’s block one data (calibration, Section II-C2), and
testing this model’s classification accuracy on the subject’s block two data
(instructed games, Section II-C3).

We obtain baselines for class similarity matrices in the same manner. Within
each subject, we collect all gesture trials from the first two experimental
blocks, and compute a normalized class similarity matrix. This is subtracted
from the matrix computed using data from block four (free games, Section
II-C5) to visualize the difference in similarity for each class. Note that due
to the short experimental design, we have relatively few samples per class
with which to construct each matrix, and therefore this representation may be
somewhat noisy.

We transform the normalized similarity matrix describing blocks one and two
into the scalar class separation measure $d_{\textsc{sep}}$, and likewise
transform the similarity matrix describing block four. This results in a
baseline-subtracted class separation measure.

Overall, we measure changes from baseline as follows:

$\displaystyle\Delta\textrm{Acc}$
$\displaystyle=\textrm{Acc}_{\textsc{Free}}-\textrm{Acc}_{\textsc{Baseline}}$
(10) $\displaystyle\Delta D$
$\displaystyle=D_{\textsc{Free}}-D_{\textsc{Baseline}}$ $\displaystyle\Delta
d_{\textsc{sep}}$ $\displaystyle=d_{\textsc{sep, free}}-d_{\textsc{sep,
baseline}}$

###  III-H Statistical Analysis

We performed several pre-planned statistical analyses to determine the effect
of feedback on classification accuracy and feature space class separation.
Differences between Feedback Groups at baseline
($\textrm{Acc}_{\textsc{Baseline}}$, $\textrm{D}_{\textsc{Baseline}}$) were
analyzed using a one-way ANOVAs. Likewise, the effect of Feedback group on
change scores ($\Delta\textrm{Acc}$, $\Delta\textrm{D}$) were analzyed with
one-way ANOVAS ($\alpha=0.05$). Alpha level was set at 0.05. Significant
findings were further analyzed using post-hoc paired comparisons with
Bonferroni correction for multiple comparisons. One sided one sample t-tests
with Bonferroni correction for multiple comparisons ($\alpha=0.0167$) were
used on change scores to test whether each Feedback Group significantly
increased accuracy and distance.

##  IV Results

All participants were able to successfully complete the experiment, with no
reported adverse events.

###  IV-A Group Baselines

A one-way ANOVA indicated no significant differences in baseline accuracy
($F(2,43)=1.15$, $P=0.326$) or class separation ($F(2,43)=0.86$, $P=0.443$)
between Feedback Groups. Figure 7 shows a group-level summary of the baseline
accuracy and class separation measure. Though no significant differences were
found, mean baseline accuracy and class separation scores were greatest in the
Control Group and smallest in the Modified Group.

Figure 7: Baseline Performance. Left: Accuracy. Right: Scalar class separation
measure $d_{\textsc{sep}}$. Boxplots show median and quartiles; dotted lines
show mean. Note the relative difference in subject baseline task performance,
visible as a gap in baseline accuracy. This discrepancy (due to random group
assignment and low subject number) indicates the need for within-subject
normalization, as described in Section III-G. See Section IV-A for statistical
analysis.

###  IV-B Effects of Feedback

Individual one-sided one-sample t-test were used to test for significant
improvement in Free block performance from baseline (Bonferroni corrected for
3 comparisons, $\alpha=0.0167$). For accuracy, only the Modified Group showed
significant improvement ($t(13)=2.566$, $P=.012$). No group showed a
significant improvement in class separation. One-way ANOVAs indicated no
significant between-group differences in $\Delta\textrm{Acc}$
($F(2,43)=0.413$, $P=0.665$) or $\Delta\textrm{D}$ ($F(2,43)=1.309$,
$P=0.281$).

Figure 8 shows the average change from baseline performance in each
experimental group, as measured in accuracy of gesture classification (left
panel) and feature-space class separation measure (right panel). These data
demonstrate that, on average, the increase in performance over the course of
the experiment was greatest for subjects in the modified feedback group. Note
that the variation between subjects is relatively high, resulting in
overlapping estimates of mean performance.

Figure 8: Overall Changes from Baseline Performance. Left: Change in accuracy.
Right: Change in scalar class separation measure $d_{\textsc{sep}}$. Boxplots
show median and quartiles; dotted lines show mean. For each subject, we
perform baseline subtraction as described in Section III-G. Change in accuracy
for modified group was significantly greater than zero using; see Section IV-B
for statistical analysis.

###  IV-C Class Confusion

Figure 9: Confusion Matrices averaged across subjects and normalized within
each row. No within-subject correction is applied. Left: Control subject.
Middle: Veridical feedback. Right: Modified Feedback.

Figure 9 shows the group average confusion matrices of gesture trials during
block four (free games) for each group. Rows represent the classification of
the attempted gesture, normalized to $1$. There are notable similarities
across the groups, indicating several gestures that are intrinsically
difficult and gesture pairs that are inherently close. In particular, the
“thumb”, “pinch”, and “fist” gestures all have a large fraction (about $25\%$)
of gestures which fall below the decision threshold. Similarly, there was an
overall trend that these three gestures tended to be confused, resulting in
non-zero entries for the off-diagonal entries (fist, thumb), (fist, pinch),
(thumb, pinch), etc.

###  IV-D Class Feature Space Similarity

Figure 10: Normalized Class Similarity Matrices. Top row: Raw similarities
from block four (free games, see section II-C5). Class similarity matrix $D$
is computed for each subject, normalized to $[0,1]$, and then averaged across
subjects in a group. Large values on diagonal indicate tight clusters for each
class. Small values off-diagonal indicate well-separated clusters. Bottom row:
Change in similarity matrix from baseline $\Delta D$, as described in Equation
10. Positive values indicate pairs that became closer in feature space,
compared to baseline; subjects whose structure improved would show positive
values on diagonal and negative values off-diagonal. See Section III-F for
further details. Left: Control group. Middle: Veridical feedback. Right:
Modified feedback. Upper triangular omitted due to symmetry.

Figure 10 shows the average normalized class similarity matrix of each group.
As described previously, a “desirable” pattern for easy downstream
classification (in which the subject produced consistent and well-separated
gestures) would consist of larger entries on the diagonal and smaller entries
off-diagonal. Each group demonstrated a consistent pattern in which the
brightest entries were along the diagonal, indicating that the gestures were
generally separable, and a consistent pattern of bright off-diagonal cells,
indicating overlap between three specific gestures: “pinch”, “fist”, and
“thumb”.

##  V Discussion and Future Work

This study tested the potential of modified continuous feedback of model
performance in a gamified user interface for rapid user training on a sEMG-
based gesture recognition system for controlling actions on a computer
display.

We hypothesized that we could use manipulation of feedback about the gesture
class probabilities in a short (4 minute) online learning session to shape
user behavior in manner that would increase the separation between muscle
activation patterns of different gestures and increase the accuracy of model
performance on future attempts. Overall, our results demonstrate that a short
user training session using modified feedback has the potential to increase
post-calibration performance (accuracy and class separation relative) when
compared to veridical feedback and a no-feedback control.

###  V-A User Calibration

Despite the emergence of research into methods for co-adaptive learning for
sEMG-based gesture recognition, there have been few investigations
specifically testing the effect of user training as means of rapid
calibration. Numerous studies have shown that extended user training on a
sEMG-based controller results in significant gains in performance [26, 11,
10]. The majority of these studies have found increased model performance was
accompanied by changes in muscle activation patterns that are theoretically
favorable to better classification (i.e. increased class separability). In
contrast, a recent investigation found low correlation between improved real-
time performance and class separability, metrics of muscle activation patterns
in feature space and classification performance are unrelated [27]. Krasoulis
et. al. first demonstrated that short-term adaptation through biofeedback user
training could positively impact prosthetic finger control using sEMG-based
decoding [9]. Our results demonstrate that both performance and class
separability increase after live feedback training, compared to the no
feedback control, and that this increase is greatest when using error-
augmented feedback.

###  V-B Influence of Feedback Manipulation on User Behavior.

The Manipulated feedback group showed the largest change in classification
accuracy and class separability. Flattening of the class probabilities as was
done in this investigation can be considered a form of error augmentation as
subjects were lead to believe the separation between classes was smaller that
it actually was. As this made the task more difficult it is most closely
related to the feedback involving “error amplification” that has been studied
extensively. Feedback of performance outcomes that are worse than actual
performance, error amplification, has been previously found to expedite motor
adaptations to novel task constraints compared to accurate feedback [28, 29]
and amplification of task errors has shown promise as an approach to
facilitate motor recovery in patients with neurological disorders [30, 31].
Faster or more complete learning with error amplification has been attributed
to more brain processes associated with greater attention to execution of the
motor task [32, 33, 34] and reduction of sensorimotor noise [14]. We speculate
that increased improvement in classification accuracy with Modified feedback
in this study may be a product of similar mechanisms that have been previously
attributed to performance improvements with error amplification.

###  V-C Selected Gestures

We selected gestures that mimicked manipulation of commonplace items such as
remote controls and cellphones. No subject commented that the gestures were
unfamiliar or difficult to perform. Directional gestures using wrist movements
(“Up”, “Down”, “Left”, “Right”) were generally separable and yielded higher
classification accuracy compared to gestures using grasping movements
(“Pinch”, “Thumb”, “Open”, “Fist”). These gestures recruit similar extrinsic
hand muscle groups (where electrodes were placed), and creating separation in
the muscle activation patterns used to perform these gestures may not be
intuitive. Thus the feature-space similarity that we observed for these
gestures is somewhat expected. Importantly, the use of modified feedback
appeared to influence the class separation of these gestures specifically
compared to control and veridical feedback conditions.

###  V-D Limitations

There were several limitations of the current work that may have affected the
results and interpretations. Only a single classification model was used.
Several machine learning methods, including artificial neural networks (ANN),
linear discriminant analysis (LDA), support vector machines (SVM), and
Gaussian mixture models have been previously used for sEMG-based control. The
choice to use a model based on SVM and Logistic Regression was due its
simplicity and the popularity of SVM for this application. It is likely that
the choice of classifier model affects not only calibration accuracy, but also
the way user explores the mapping of muscle activation to gestures.
Nevertheless, the user training scheme employed here likely has general
benefit for use and understanding of human co-adaptive behavior.

There are a number of possible changes in the signal processing pipeline that
may yield improvements in overall model performance. The active window for
feature extraction may be tuned, and additional features such as time-
frequency domain or higher-dimensional feature vectors may be extracted. The
selected features (RMS, and median frequency) were chosen based on their
common use in for sEMG-based gesture classification and initial pilot testing.
Future work should evaluate how sEMG feature selection effects user training.

Only a single type of feedback manipulation was tested. We used a feedback
manipulation that essentially flattened probabilities across classes, making
it more difficult to achieve a correct classification. This approach was
selected as it was expected that participants would respond by increasing the
separation between muscle activation patterns for different gestures. While we
find this to be case, the manipulation was not directly optimized for this
purpose. Future research should explore the optimization of feedback
manipulation for shaping user behavior during co-adaptive sEMG-gesture
recognition. Adaptive feedback manipulation based on user and model
performance characteristics to target specific class confusions is a
attractive future direction.

## References

  * [1] Zhiwen Yang, Ying Sun Du Jiang, Bo Tao, Xiliang Tong, Guozhang Jiang, Manman Xu, Juntong Yun, Ying Liu, Baojia Chen, and Jianyi Kong.  Dynamic gesture recognition using surface emg signals based on multi-stream residual network.  Frontiers in Bioengineering and Biotechnology, 9, 2021.
  * [2] Dezhen Xiong, Daohui Zhang, Xingang Zhao, and Yiwen Zhao.  Deep learning for emg-based human-machine interaction: a review.  IEEE/CAA Journal of Automatica Sinica, 8(3):512–533, 2021.
  * [3] Jinxian Qi, Guozhang Jiang, Gongfa Li, Ying Sun, and Bo Tao.  Intelligent human-computer interaction based on surface emg gesture recognition.  IEEE Access, 7:61378–61387, 2019.
  * [4] Dalia De Santis.  A framework for optimizing co-adaptation in body-machine interfaces.  Frontiers in Neurorobotics, 15:40, 2021.
  * [5] Janne M Hahne, Sven Dähne, Han-Jeong Hwang, Klaus-Robert Müller, and Lucas C Parra.  Concurrent adaptation of human and machine improves simultaneous and proportional myoelectric control.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(4):618–627, 2015.
  * [6] Mathilde Couraud, Daniel Cattaert, Florent Paclet, Pierre-Yves Oudeyer, and Aymar De Rugy.  Model and experiments to optimize co-adaptation in a simplified myoelectric control system.  Journal of neural engineering, 15(2):026006, 2018.
  * [7] Dennis Yeung, Dario Farina, and Ivan Vujaklija.  Directional forgetting for stable co-adaptation in myoelectric control.  Sensors, 19(9):2203, 2019.
  * [8] Dennis Yeung, Irene Mendez Guerra, Ian Barner-Rasmussen, Emilia Siponen, Dario Farina, and Ivan Vujaklija.  Co-adaptive control of bionic limbs via unsupervised adaptation of muscle synergies.  IEEE Transactions on Biomedical Engineering, 2022.
  * [9] Agamemnon Krasoulis, Sethu Vijayakumar, and Kianoush Nazarpour.  Effect of user practice on prosthetic finger control with an intuitive myoelectric decoder.  Frontiers in neuroscience, page 891, 2019.
  * [10] Nathan E Bunderson and Todd A Kuiken.  Quantification of feature space changes with experience during electromyogram pattern recognition control.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 20(3):239–246, 2012.
  * [11] Michael A Powell, Rahul R Kaliki, and Nitish V Thakor.  User training for pattern recognition-based myoelectric prostheses: Improving phantom limb movement consistency and distinguishability.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(3):522–532, 2013.
  * [12] Yejun Wei, Preeti Bajaj, Robert Scheidt, and James Patton.  Visual error augmentation for enhancing motor learning and rehabilitative relearning.  In 9th International Conference on Rehabilitation Robotics, 2005\. ICORR 2005., pages 505–510. IEEE, 2005.
  * [13] Emanuel Todorov, Reza Shadmehr, and Emilio Bizzi.  Augmented feedback presented in a virtual environment accelerates learning of a difficult motor task.  Journal of motor behavior, 29(2):147–158, 1997.
  * [14] Christopher J Hasson, Zhaoran Zhang, Masaki O Abe, and Dagmar Sternad.  Neuromotor noise is malleable by amplifying perceived errors.  PLoS computational biology, 12(8):e1005044, 2016.
  * [15] Adrian M Haith, Thomas R Reppert, and Reza Shadmehr.  Evidence for hyperbolic temporal discounting of reward in control of movements.  Journal of neuroscience, 32(34):11727–11736, 2012.
  * [16] Meghan E Huber, Nikita Kuznetsov, and Dagmar Sternad.  Persistence of reduced neuromotor noise in long-term motor skill learning.  Journal of Neurophysiology, 116(6):2922–2935, 2016.
  * [17] Hermanus J Hermens, TAMv Bruggen, Christian TM Baten, WLC Rutten, and HBK Boom.  The median frequency of the surface emg power spectrum in relation to motor unit firing and action potential properties.  Journal of Electromyography and Kinesiology, 2(1):15–25, 1992.
  * [18] U Kreßel.  Pairwise classification and support vector machines.  In B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors, Advances in Kernel Methods — Support Vector Learning, pages 255–268, Cambridge, MA, 1999. MIT Press.
  * [19] Steven Diamond and Stephen Boyd.  CVXPY: A Python-embedded modeling language for convex optimization.  Journal of Machine Learning Research, 17(83):1–5, 2016.
  * [20] Kristin Bennett and Ayhan Demiriz.  Semi-supervised support vector machines.  Advances in Neural Information processing systems, 11, 1998.
  * [21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [22] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [23] PyQT.  Pyqt reference guide, 2012.
  * [24] Jimmy Feng, Pradeep Damodara, George Gensure, Ryan Catoen, and Allen Yin.  Labgraph.  https://github.com/facebookresearch/labgraph, 2021.
  * [25] Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa.  Large sample analysis of the median heuristic.  arXiv preprint arXiv:1707.07269, 2017.
  * [26] Jiayuan He, Dingguo Zhang, Ning Jiang, Xinjun Sheng, Dario Farina, and Xiangyang Zhu.  User adaptation in long-term, open-loop myoelectric training: implications for emg pattern recognition in prosthesis control.  Journal of neural engineering, 12(4):046005, 2015.
  * [27] Andreas W Franzke, Morten B Kristoffersen, Vinay Jayaram, Corry K van der Sluis, Alessio Murgia, and Raoul M Bongers.  Exploring the relationship between emg feature space characteristics and control performance in machine learning myoelectric control.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:21–30, 2020.
  * [28] Antoinette Domingo and Daniel P Ferris.  The effects of error augmentation on learning to walk on a narrow balance beam.  Experimental brain research, 206(4):359–370, 2010.
  * [29] James L Patton, Mary Ellen Stoykov, Mark Kovic, and Ferdinando A Mussa-Ivaldi.  Evaluation of robotic training forces that either enhance or reduce error in chronic hemiparetic stroke survivors.  Experimental brain research, 168(3):368–383, 2006.
  * [30] Sharon Israely and Eli Carmeli.  Error augmentation as a possible technique for improving upper extremity motor performance after a stroke–a systematic review.  Topics in stroke rehabilitation, 23(2):116–125, 2016.
  * [31] Sergei V Adamovich, Gerard G Fluet, Eugene Tunik, and Alma S Merians.  Sensorimotor training in virtual reality: a review.  NeuroRehabilitation, 25(1):29–44, 2009.
  * [32] Driss Boussaoud and Imane Kermadi.  The primate striatum: neuronal activity in relation to spatial attention versus motor preparation.  European Journal of Neuroscience, 9(10):2152–2168, 1997.
  * [33] Markus Jueptner and Cornelius Weiller.  A review of differences between basal ganglia and cerebellar control of movements as revealed by functional imaging studies.  Brain: a journal of neurology, 121(8):1437–1449, 1998.
  * [34] Navid Shirzad and HF Machiel Van der Loos.  Error amplification to promote motor learning and motivation in therapy robotics.  In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 3907–3910. IEEE, 2012.