§ INTRODUCTION

Using an appropriate position encoding and data formatting, we demonstrate that Transformers can generalize to 100-digit decimal addition tasks with more than 98% of accuracy when trained up to 40-digit addition, resulting in a length extension ratio of $2.5\times$,
which is much more than the ratio of Lee et al., 2023 ($1.0\times$), Kazemnejad et al., 2023 ($1.125\times$), Shen et al., 2023 ($1.1\times$), and Zhou et al., 2023 ($1.5\times$).
Unfilled markers (—${\color{white}\blacktriangledown}$  $\triangledown$ ) denote in-distribution test results, filled markers (—$\blacktriangledown$) denote out-of-distribution results.
In Zhou et al., 2023 and Our Work, each curve is the best out of 10 trials. For the other three methods, we report the value from their corresponding paper.

Transformer-based models have revolutionized natural language understanding and generation across diverse applications [OpenAI, 2023, Gemini et al., 2023]. Despite their impressive abilities in mathematical reasoning [Lewkowycz et al., 2022], code synthesis [Li et al., 2022], and theorem proving [Wu et al., 2022], Transformers often struggle with length generalization, an ability that requires the model to generalize to longer sequences than seen during training [Anil et al., 2022, Abbe et al., 2023, Zhou et al., 2023]. This limitation raises an essential question: do Transformers genuinely grasp the correct underlying algorithms for a given task, or are they merely resorting to superficial memorization or shortcuts that fail to scale to more complex problems [Liu et al., 2023]?

Recent work has scrutinized Transformers' shortcomings in length generalization across formal language learning [Deletang et al., 2023] and algorithmic reasoning tasks [Anil et al., 2022, Zhang et al., 2022, Veličković et al., 2022, Dziri et al., 2023]. These investigations consistently indicate a notable deficiency in length generalization capabilities. This recurring issue raises a crucial question: Is there an inherent limitation in Transformers' design preventing effective length generalization?

In this paper, we systematically examine the Transformer's capability of length generalization, specifically focusing on the $N$-digit decimal addition problem. We view the addition problem as a form of synthetic language learning, which despite its relative simplicity compared to natural language, provides valuable insights into the Transformer's ability to internalize fundamental algorithms. Notwithstanding its simplicity, recent work has demonstrated that Transformers exhibit limited length generalization in this task [Lee et al., 2023, Shen et al., 2023, Kazemnejad et al., 2023].

Previous attempts to improve Transformer's length generalization ability primarily focus on two areas: refining position encodings [Shen et al., 2023, Press et al., 2022] and optimizing data formats [Lee et al., 2023, Zhou et al., 2023]. Therefore, we perform an extensive empirical evaluation of combinations of widely used position encoding and various data formats, resulting in a recipe for successful length generalization. Our final recipe consists of: FIRE position encodings [Li et al., 2023], with randomized positions [Ruoss et al., 2023],
in reversed format, with index hints [Zhou et al., 2023].

As shown in <Ref>, when trained on only 40 digits, our model successfully extrapolates to sequences of up to 100 digits, exceeding the input length by $2.5\times$.
To the best of our knowledge, this is the strongest known generalization result for text-based Transformers on addition. Nevertheless, we observe that the robustness of this length generalization is fragile, significantly swayed by variables such as random initialization and the training data order.
The 2 (equally) important conclusions of this paper are: (1) Our approach achieves much better length generalization with XXX positional encoding and YYY data format; and (2) The generalization performance is sensitive to several factors. I personally think the 2nd point was not sufficiently discussed in prior work and is important to emphasize. However, the current introduction does not tell about what is the best combination, and there is very little discussion on the second point. I suggest to rewrite the bullet points and highlight these 2 conclusions, and discussion on redundant techniques is not necessary in the intro.

Our key contributions are summarized as follows:

* We demonstrate that the success in length generalization is markedly influenced by position encoding and data format. Through careful selection of these factors, we achieved extrapolation to lengths that are $2.5 \times$ longer than those seen during training.

* Our exploration of established data formatting and augmentation techniques indicates that their effectiveness in length generalization is primarily contingent on the choice of position encoding.

* Despite remarkable generalization to lengths $2.5 \times$ longer than training, we found this generalization to be fragile and heavily relying on factors like random weight initialization and training data order.

§ POSITION ENCODING AND DATA FORMATS

Recently proposed improvements in architectural design, notably in position encoding [Shen et al., 2023, Kazemnejad et al., 2023, Ruoss et al., 2023] and attention mechanisms [Dubois et al., 2019, Duan and Shi, 2023], aim to address the challenge of length generalization in arithmetic computations with Transformers. However, the effectiveness of such modifications is often constrained, either due to their overly ad-hoc nature or their poor performance on longer sequences. Although scaling the size of models and datasets has been recognized as a generally effective strategy to improve performance, prior research [Brown et al., 2020, Anil et al., 2022] suggests that relying solely on scale might not be sufficient for handling test sequences that are longer than training. Concurrently, with the rising focus on data-centric AI [Motamedi et al., 2021], recent work has investigated refining the data format to enhance the learning efficacy of existing Transformer models.
In this section, we review some of the most common position encodings (<Ref>) and relevant data formats (<Ref>)

§.§ Position Encoding for Length Generalization

The inability of transformers to extrapolate to longer sequences has been primarily attributed to position encoding <cit.>. In this section, we review existing positional encoding approaches with an emphasis on their length generalization abilities.

Absolute Positional Encoding (APE). APE enhances Transformer models with positional information by attaching a positional vector $\vp_i$ to each position $i$.
This is achieved through a predefined sinusoidal function [Vaswani et al., 2017] or a learnable approach [Devlin et al., 2018]. Then, the vector $\vp_i$ is combined with the token embedding $\ve_i$ before entering the transformer's first layer. Although straightforward, APE often struggles with generalizing to longer sequences, as observed in both NLP [Press et al., 2022] and algorithmic tasks [Kazemnejad et al., 2023].

Additive Relative Positional Encoding (RPE). Shaw et al., 2018 pioneered the additive RPEs, diverging from standard input-level integration by modifying keys and, optionally, values in each attention layer. This concept was advanced by T5, which employed scalar biases to directly affect pre-softmax attention logits, a method noted for its simplicity yet criticized for limited efficiency and positional differentiation in long sequences [Raffel et al., 2020, Press et al., 2022]. Later approaches such as Alibi [Press et al., 2022], Kerple [Chi et al., 2022] and FIRE [Li et al., 2023] build on the idea of learned additive bias, proposing different functions to model the scalar bias as a function of the key- and query-indices.
Most pre-softmax attention logits of
additive RPEs can be generally written as [Li et al., 2023]:

\begin{equation}
\label{eq:rpe-attn-mat}
\mA_{\mathrm{RPE}}(\mX) = \mX \mW_Q(\mX \mW_K)^{\top}+\mB,
\end{equation}

where $\mX$, $\mW_Q$, $\mW_K$ denote the input and weight matrices for queries and keys. The bias matrix $\mB\in\R^{n\times n}$ is induced by the position encoding function $b: \sN^{*2}\to\R$, with its $(i,j)$-th entry defined as $b(i,j)$. Instances of $b(i,j)$ include:

* T5 [Raffel et al., 2020]: $b(i,j) = r_{min}\{i-j,K\}$, where $K$ is a hyperparameter and $r_i$ are learned scalars.

* Alibi [Press et al., 2022]: $b(i,j) = -r\left|i-j \right|$, where $r>0$ is a hyperparameter.

* KerpleLog [Chi et al., 2022]: $b(i,j) = -r_1\log(1+r_2|i-j|)$, where $r_1, r_2>0$ are learnable scalars.

* FIRE [Li et al., 2023]: $b(i,j) = f_{\theta}\left(\frac{\psi(i-j)}{\psi(\max\{L, i\})}\right)$, where $f_{\theta}:\R\to\R$ is a learnable MLP parameterized by $\theta$, $\psi: \sN \to \sR_{+}$ is
and $c>0, L>0$ are learnable scalars.

Additional background on additive RPEs is provided in Appendix <ref>

Rotary Positional Encoding (RoPE).
RoPE [Su et al., 2024]
encodes position information in attention logits through rotational encoding of query and key vectors based on their relative positions. Despite being simple and effective, RoPE exhibits limited length generalization  [Press et al., 2022, Kazemnejad et al., 2023]. While extensions like Position Interpolation [Chen et al., 2023, Peng et al., 2023, Su, 2023] enhance RoPE's context length, they do not necessarily improve length generalization on algorithmic tasks where learning the underlying algorithm is crucial.

No Positional Encoding (NoPE).
While encoder-only Transformers (e.g., BERT [Devlin et al., 2018]) are permutation equivariant without positional encodings, decoder-only counterparts with causal attention, as shown by Haviv et al., 2022, acquire positional understanding autonomously, even without explicit PE. Interestingly, recent findings by Kazemnejad et al., 2023 further reveal that a model without PE outperforms those with specialized PEs on simple algorithmic tasks.

Randomized Position Encoding. Ruoss et al., 2023 introduced Randomized PE to enhance existing PEs by randomly sampling encodings from a range exceeding test-time lengths while preserving the order. Transformers trained this way adapt to larger positional encodings, effectively eliminating OOD position encodings during testing.

Comparative overview of PEs and data formats: While most related studies focus on APE or NoPE, our approach integrates FIRE [Li et al., 2023] and Randomized PE [Ruoss et al., 2023]. All studies utilize a reversed format. Shen et al., 2023 enhance this with random space augmentation, and both Zhou et al., 2023 and Our Work incorporate index hints.

§.§ Data Formats

Data format plays a pivotal role in enhancing Transformers' length generalization capabilities, primarily by transforming the data into a format that could be more easily learned.
We give an overview of the existing techniques below.

Reversed Format.
Computing addition in an algorithmic way (as taught in elementary school) requires starting with the least significant digit (LSD) and proceeds to the most significant digit (MSD). This sequence contrasts with the standard printed format ($A_3A_2A_1 + B_3B_2B_1 = C_3C_2C_1$, where $A_1$ and $B_1$ are the LSDs, which is not ideally suited for autoregressive models due to their outputting the MSD first. However, the reversed format ($A_1A_2A_3 + B_1B_2B_3 = C_1C_2C_3$) aligns better with these the natural order of computing the digits. It simplifies the learning task to a function that depends only on
the two corresponding operand digits and the carry from the previous step [Lee et al., 2023, Zhou et al., 2023, Shen et al., 2023].

Index Hints.
Zhou et al., 2023 introduced “index hints” in both the query and response of arithmetic tasks. For example, $42 + 39 = 81$ is represented as $a4b2 + a3b9 = a8b1$ during training and inference, enabling transformers to execute indexing via induction heads [Olsson et al., 2022].

Random Space Augmentation. Shen et al., 2023 explored the impact of random spacing between digits in addition, aiming to disrupt the model's reliance on absolute positional information. Their results show successful generalization from 10-digit to 11-digit addition, but falters with longer sequences.

<Ref> lists the position encodings and data formats used in some of the most related work to ours.

§ A RECIPE FOR LENGTH GENERALIZATION IN DECIMAL ADDITION

The task of decimal addition is composed of two critical subtasks: (a) the identification of the right operands to add;
and (b) the summation of these operands with the preceding carry.
While the summation step ((b)) is relatively easier because it has a finite set of possible inputs, the primary generalization challenge lies in the operand identification ((a)), where precise positional access is crucial.

Our best model, which leads to the results in <Ref>, uses the following combination:

* FIRE position encodings [Li et al., 2023]:
We believe that FIRE position encodings are helpful for length generalization because they are more expressive than other PEs, as shown by Li et al., 2023.

* Randomized position encodings [Ruoss et al., 2023]: We believe that randomized position encodings are crucial to avoid overfitting on the position indices and index differences that were seen during training.

* Reversed format: The reversed format makes it easier for the model to decompose the long computation to local, “markovian”, steps that depend only on the single previous step.

* Index hints [Zhou et al., 2023]: We believe that index hints are useful
because they ease the task of operand identification (discussed in (b)), of matching the right operands to add at a certain step.

We ablate each of these decisions and some other alternative choices in <Ref>.

§ EXPERIMENTS

§.§ Setup

Data. As shown in <Ref>, we adopt the reversed format with index hints as our default data format. During training, we randomly sample consecutive index hints from a pre-defined ordered set of hints with 102 symbols, thereby enhancing the learning of hint sequences and their order.
We generated a dataset comprising 30M examples on input lengths 1-40 for training and 1,000 examples per input length for testing.

Model. Our base model, following Zhou et al., 2023, is a 25M parameter Transformer featuring 6 blocks, a 512 hidden size, and a feedforward layer with a hidden dimension of 2048. We also adopt RMSNorm, integrating both PreNorm and PostNorm layers, following the Primer architecture [So et al., 2021].
We use the AdamW optimizer [Loshchilov and Hutter, 2017] to train the model with a weight decay value of 0.1 and no dropout, for 50,000 steps. The learning rate schedule incorporates an initial 500-step linear warm-up, followed by a cosine decay, starting at 3e-4. The hyperparameters are chosen based on <Ref>.

Randomized PE and Random Space Augmentation. As will be demonstrated in <Ref>, the success of these techniques is markedly PE-dependent. Hence, we tailor the default hyperparameter choice to best suit each PE.
Further, instead of using random spaces, we use another special token to prevent automatic merging by the tokenizer.

Due to the high variance (which we discuss in the next section), we repeat each experiment five times unless mentioned otherwise.
More implementation details are provided in <Ref>.

§.§ Results

FIRE enables significantly better length generalization.
<Ref> compares the length generalization capabilities of four positional encodings in the best of 10 trials (See <Ref> for all trials). Trained exclusively on sequences of lengths 1-40, the best trial of FIRE exhibit near-perfect generalization to sequences up to the length of 100. In contrast, other PEs show a visible degradation in generalization accuracy beyond the sequence length of 60. This finding counters the findings of Kazemnejad et al., 2023 that no positional encoding (NoPE) surpasses complex PE techniques for length generalization. Our findings suggest that a well-designed PE, such as FIRE, is essential for optimal length generalization.

Index hints are crucial. We compare models trained with and without index hints.
As shown in <Ref>, index hints significantly enhance length generalization across various PEs, corroborating the findings of Zhou et al., 2023. Notably, without index hints, NoPE and FIRE demonstrate poor in-distribution generalization for 40-digit additions, a marked deviation from their reasonable performance when trained on 10-digits, as shown in <Ref>.
<Ref> shows that this phenomenon occurs across all random seeds.
Conversely, RoPE and KerpleLog exhibit moderate in-distribution generalization but falter in out-of-distribution scenarios. <Ref> shows the training loss and test accuracy of these runs.

Analyzing errors in 11-digit additions from models trained on 10-digits revealed a common misalignment issue: the Transformer often adds operands adjacent to the correct ones. An attempt to rectify this by reformatting addition ($A_1B_1, A_2B_2, A_3B_3 = C_1C_2C_3$, with 1 as the least significant bit) failed to improve length generalization, merely shifting the error to adjacent output positions. This highlights the Transformer's inherent limitations in precise position identification.

Standard format vs reversed format. As shown in Figure <ref>, standard formatting shows limited length generalization in all PEs compared to the reversed format.
FIRE excels in length generalization even with the standard format, even matching RoPE in reverse format. However, FIRE's performance (with standard format) declines beyond 60-digit additions, likely due to increased carry propagation challenges exceeding the model's capacity.

Looking at the training loss and training next-token accuracy in both formats also shows interesting differences. As shown in <Ref>, the standard format training leads to gradual improvement, whereas reverse format yields a sharp performance transition. This transition,
which is a reminiscent of “grokking” phenomenon  [Power et al., 2022],
shows in this case the “Eureka moment” in which the Transformer learns the right addition algorithm.

Random space augmentation and randomized position encoding.
<Ref> reveals divergent impacts of random space augmentation on four PEs.
The augmentation's efficacy is notably contingent upon the chosen PE. While Random Spaces marginally enhances RoPE and KerpleLog's performance, it markedly deteriorates NoPE and FIRE. A similar PE-specific pattern is evident in Randomized PE, as <Ref> demonstrates. Randomized PE significantly degrades KerpleLog's effectiveness, yet it substantially boosts FIRE. See <Ref> for training loss and EM accuracy for all trials in each setting.

Length generalization is not robust to neither weight initialization nor training data order. <Ref> illustrates the varying performance of 10 FIRE trials using identical training data order but distinct weight initializations. Notably, while all trials achieve similar close-to-zero training losses after 10K training steps (<Ref>) and exhibit perfect in-distribution generalization, their out-of-distribution (OOD) length generalization shows significant variance.
Moreover, the length generalization performance fluctuates significantly across training steps (<Ref>). This observation contrasts with earlier studies suggesting in-distribution loss as a reliable OOD generalization predictor [Nagarajan et al., 2020].

We further examine 15 unique combinations, resulting from 3 weight initialization seeds and 5 data input orders. As shown in <Ref>, there is significant variance across training data orders
even when the weight initialization is constant.
Intriguingly, certain weight initializations demonstrate remarkable resilience to changes in data input order. This observation is reminiscent of the Lottery Ticket Hypothesis [Frankle and Carbin, 2018], which posits the existence of a sparse, equally effective sub-network within a larger neural network. Our findings suggest the presence of “fortunate” weight configurations that exhibit robust length generalization, akin to a “lucky weight ticket.”

While Anil et al., 2022 also noticed similar in-distribution accuracy but marked differences in OOD behavior on parity tasks, their OOD performance was quite poor across all runs. Moreover, contrary to the findings of Anil et al., 2022 on the impact of hyperparameter variation, our experiments reveal considerable performance fluctuations even with different random seeds. This inconsistency appears unrelated to position encoding (refer to <Ref> for different PEs), and is more likely due to variations in random weight initialization and data order.

§ ANALYSIS

Error analysis.
In examining Transformers' error characteristics, we classified erroneous predictions into two categories: those with and without carry. Figure <ref> shows no significant difference between these categories, thus carry propagation does not majorly impede length generalization.

Additionally, we analyzed the error distribution in 100-digit addition using FIRE, illustrated in <Ref>. As shown, <Ref> indicates an overall uniform error distribution across all indices, despite some individual model checkpoints showing errors at specific positions. Excluding two near-zero accuracy runs, over 90% of errors in incorrect examples are single-digit mistakes, following an exponential distribution. Additional results are shown in <Ref>.

Despite the imperfect calculation, the FIRE model does not show any systematic error.
Random errors may stem from phenomena such as attention glitches [Liu et al., 2023]. Conversely, other PEs systematically fail to identify the start or end of addition, leading to premature termination.

Performance evolution during training. <Ref> shows that while transformers achieve near-perfect in-distribution accuracy early in training, they explore different extrapolation strategies. This ability is remarkable considering the inherent unpredictability and architecture-dependent nature of OOD accuracy. Notably, transformers with FIRE exhibit a generally steady increase in OOD accuracy during training, suggesting that FIRE's inductive bias may be helpful in finding solutions that generalize to different lengths. In contrast, other PE methods display more volatile OOD performance. Interestingly, some methods exhibit a “grokking-like” phenomenon, where there is a sudden surge in the OOD accuracy despite no change in in-distribution accuracy.

Sequence length during training.
We trained separate models for addition involving up to 10, 20, 30, and 40 digits, and evaluated them on addition of up to 100 digits. As depicted in <Ref>, training length crucially improves performance in longer length generalizations across different PEs.
Notably, not only that models that were trained on 40 digits generalize better than models that were trained on shorter sequences, the generalization factor is also increasing: the model that was trained on 40 digits generalizes to 100 digits (2.5$\times$), while the model that was trained on up to 30 digits generalizes to 45 digits (1.5$\times$), the model that was trained on up to 20 digits generalizes to 25 digits (1.25$\times$), and the model that was trained on up to 10 digits does not generalize beyond training lengths (1.0$\times$).

Scaling model size.
The scaling of model size is crucial for improving large language models [Thoppilan et al., 2022, Chowdhery et al., 2023]. To assess its effect on length generalization, we contrasted models with 25M and 268M parameters. We find that model size variation has a minor effect on length generalization. <Ref> shows that larger models slightly improve generalization in short digit regimes (1 to 10 and 1 to 20 digit additions) but yield mixed results in longer regimes. While RoPE and KerpleLog show improvements, NoPE and FIRE experience performance degradation with a larger model, indicating model size may not be the primary factor in length generalization.

The efficacy of length generalization in the 25M model prompted us to explore the capabilities of smaller models.
Specifically, we trained models with 2M and 5M parameters. As <ref> illustrate, the 2M model's performance deteriorates with longer sequences, indicating limited model capacity as a potential performance bottleneck. Intriguingly, this model outperforms its larger counterparts (5M and 25M models) in tasks involving 1 to 10 digit addition. Furthermore, the 5M model remarkably achieves 80% accuracy in 100 digit addition, trained only on 1 to 40 digit tasks, surpassing the 268M model's performance.

Does stronger regularization reduce variance? To mitigate performance variance, we investigated standard regularization techniques, including weight decay and dropout. As depicted in <Ref>, higher weight decay values (e.g., 0.1, 0.3) slightly enhance the likelihood of achieving effective length generalization. Nonetheless, non-trivial length generalization remains attainable with either very low (e.g., 1e-6) or high (e.g., 1.0) weight decay values, evidenced by approximately 80% accuracy in 100 digit addition trained on 40-digit sequences. Conversely, <Ref> shows that substantial dropout values (e.g., 0.2) severely impair length generalization. Dropout rates of 0.0 or 0.1, however, do not show statistically significant improvements over their counterparts. Overall, while regularization can modestly decrease performance variability, it falls short in ensuring robust length generalization. The variance in performance is still significantly influenced by the randomness of weights initialization and the training data order (<Ref>).

§ RELATED WORK

Length generalization remains a significant challenge in neural networks, underscored by substantial research [Graves et al., 2016, Hupkes et al., 2020, Schwarzschild et al., 2021, Zhang et al., 2022, Deletang et al., 2023, Dziri et al., 2023]. Despite their advanced reasoning capabilities, Transformer-based large language models (LLMs) [Thoppilan et al., 2022, Chowdhery et al., 2023] struggle with processing sequences beyond their training scope [Anil et al., 2022]. Enhancements in length generalization, especially in the addition task, primarily focus on two areas: refining positional encoding and optimizing data format.

Position Encoding for Length Generalization
The inability of Transformers to extrapolate to longer sequences has been primarily attributed to Position Encoding (PE) [Shaw et al., 2018]. Various studies have suggested alternatives, such as relative positional encodings, which focus on the relative distances between tokens [Dai et al., 2019], the implementation of randomized position encoding [Ruoss et al., 2023], or the adoption of weighted attention mechanisms in place of position embeddings [Press et al., 2022, Raffel et al., 2020, Chi et al., 2022, Li et al., 2023]. These approaches have shown promise in natural language processing (NLP). However, Kazemnejad et al., 2023 found that omitting position encoding entirely yields better results for algorithmic tasks. In contrast, our experiments indicate that an effectively designed PE, such as the FIRE, is crucial for achieving optimal length generalization (<Ref>). Moreover, we show that a synergistic approach to consider both PE and data design markedly enhances length generalization capabilities.

Data format for Length Generalization
A range of heuristic-based data formatting methods have been introduced, particularly for pretrained LLMs. These methods, including the use of scratchpads and the chain of thoughts approach, aim to facilitate arithmetic learning either through in-context learning or fine-tuning [Anil et al., 2022, Zhou et al., 2022]. Conversely, there is a body of research focused on Transformers trained from scratch.
This research indicates that employing techniques such as reversed formatting and scratch pads can significantly boost length generalization performance [Shen et al., 2023, Lee et al., 2023].
Furthermore, it has been observed that both the data distribution and the sampling strategies can profoundly influence generalization [Lee et al., 2023].
Awasthi and Gupta, 2023 further demonstrates the benefits of incorporating a simpler auxiliary task (e.g., identifying the successor element) in supporting the primary task (e.g., sorting). In contrast, Jelassi et al., 2023 finds that train set priming enables length generalization for a encoder-only Transformer model.
In contrast, our good length generalization performance achieved with naive random sampling approach suggesting that sophisticated data sampling might be redundant.

§ CONCLUSION

Length generalization in Transformers has been a long-standing challenge.
We evaluate the ability of Transformers to generalize to longer test sequences using the decimal addition task.
Through extensive experiments, we find that there is no inherent limitation in Transformers’ design preventing effective
length generalization. Instead, the missing ingredient is the right combination of data format and position encoding.
We demonstrate that Transformers can achieve almost perfect generalization on sequences up to $2.5\times$ the training length, given appropriate data formatting and position encoding.

Our thorough empirical analysis of common length generalization techniques reveals a significant dependency between the type of position encoding and the data format. This underscores the importance of synergizing data format with model architecture for optimal generalization.
Despite these advancements, robust length generalization in Transformers remains elusive, even with meticulously finetuned regularization hyperparameters.

[Abbe et al., 2023]
E. Abbe, S. Bengio, A. Lotfi, and K. Rizk.
Generalization on the unseen, logic reasoning and degree curriculum.
arXiv preprint arXiv:2301.13105, 2023.

[Anil et al., 2022]
C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur.
Exploring length generalization in large language models.
Advances in Neural Information Processing Systems, 35:0 38546–38556, 2022.

[Awasthi and Gupta, 2023]
P. Awasthi and A. Gupta.
Improving length-generalization in transformers via task hinting.
arXiv preprint arXiv:2310.00726, 2023.

[Brown et al., 2020]
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.
Language models are few-shot learners.
Advances in neural information processing systems, 33:0 1877–1901, 2020.

[Chen et al., 2023]
S. Chen, S. Wong, L. Chen, and Y. Tian.
Extending context window of large language models via positional interpolation.
arXiv preprint arXiv:2306.15595, 2023.

[Chi et al., 2022]
T.-C. Chi, T.-H. Fan, P. J. Ramadge, and A. Rudnicky.
Kerple: Kernelized relative positional embedding for length extrapolation.
Advances in Neural Information Processing Systems, 35:0 8386–8399, 2022.

[Chowdhery et al., 2023]
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al.
Palm: Scaling language modeling with pathways.
Journal of Machine Learning Research, 240 (240):0 1–113, 2023.

[Dai et al., 2019]
Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov.
Transformer-xl: Attentive language models beyond a fixed-length context.
arXiv preprint arXiv:1901.02860, 2019.

[Deletang et al., 2023]
G. Deletang, A. Ruoss, J. Grau-Moya, T. Genewein, L. K. Wenliang, E. Catt, C. Cundy, M. Hutter, S. Legg, J. Veness, and P. A. Ortega.
Neural networks and the chomsky hierarchy.
In The Eleventh International Conference on Learning Representations, 2023.
URL <https://openreview.net/forum?id=WbxHAzkeQcn>.

[Devlin et al., 2018]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv preprint arXiv:1810.04805, 2018.

[Duan and Shi, 2023]
S. Duan and Y. Shi.
From interpolation to extrapolation: Complete length generalization for arithmetic transformers.
arXiv preprint arXiv:2310.11984, 2023.

[Dubois et al., 2019]
Y. Dubois, G. Dagan, D. Hupkes, and E. Bruni.
Location attention for extrapolation to longer sequences.
arXiv preprint arXiv:1911.03872, 2019.

[Dziri et al., 2023]
N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jian, B. Y. Lin, P. West, C. Bhagavatula, R. L. Bras, J. D. Hwang, et al.
Faith and fate: Limits of transformers on compositionality.
arXiv preprint arXiv:2305.18654, 2023.

[Frankle and Carbin, 2018]
J. Frankle and M. Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks.
arXiv preprint arXiv:1803.03635, 2018.

[Gemini et al., 2023]
Gemini, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.

[Graves et al., 2016]
A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. P. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis.
Hybrid computing using a neural network with dynamic external memory.
Nat., 5380 (7626):0 471–476, 2016.
URL <https://doi.org/10.1038/nature20101>.

[Haviv et al., 2022]
A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy.
Transformer language models without positional encodings still learn positional information.
arXiv preprint arXiv:2203.16634, 2022.

[Hupkes et al., 2020]
D. Hupkes, V. Dankers, M. Mul, and E. Bruni.
Compositionality decomposed: How do neural networks generalise?
Journal of Artificial Intelligence Research, 67:0 757–795, 2020.

[Jelassi et al., 2023]
S. Jelassi, S. d'Ascoli, C. Domingo-Enrich, Y. Wu, Y. Li, and F. Charton.
Length generalization in arithmetic transformers.
arXiv preprint arXiv:2306.15400, 2023.

[Kazemnejad et al., 2023]
A. Kazemnejad, I. Padhi, K. N. Ramamurthy, P. Das, and S. Reddy.
The impact of positional encoding on length generalization in transformers.
arXiv preprint arXiv:2305.19466, 2023.

[Lee et al., 2023]
N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos.
Teaching arithmetic to small transformers.
arXiv preprint arXiv:2307.03381, 2023.

[Lewkowycz et al., 2022]
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al.
Solving quantitative reasoning problems with language models.
Advances in Neural Information Processing Systems, 35:0 3843–3857, 2022.

[Li et al., 2023]
S. Li, C. You, G. Guruganesh, J. Ainslie, S. Ontanon, M. Zaheer, S. Sanghai, Y. Yang, S. Kumar, and S. Bhojanapalli.
Functional interpolation for relative positions improves long context transformers.
arXiv preprint arXiv:2310.04418, 2023.

[Li et al., 2022]
Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al.
Competition-level code generation with alphacode.
Science, 3780 (6624):0 1092–1097, 2022.

[Liu et al., 2023]
B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang.
Exposing attention glitches with flip-flop language modeling.
arXiv preprint arXiv:2306.00946, 2023a.

[Liu et al., 2023]
B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang.
Transformers learn shortcuts to automata.
In The Eleventh International Conference on Learning Representations, 2023b.
URL <https://openreview.net/forum?id=De4FYqjFueZ>.

[Loshchilov and Hutter, 2017]
I. Loshchilov and F. Hutter.
Decoupled weight decay regularization.
arXiv preprint arXiv:1711.05101, 2017.

[Motamedi et al., 2021]
M. Motamedi, N. Sakharnykh, and T. Kaldewey.
A data-centric approach for training deep neural networks with less data.
arXiv preprint arXiv:2110.03613, 2021.

[Nagarajan et al., 2020]
V. Nagarajan, A. Andreassen, and B. Neyshabur.
Understanding the failure modes of out-of-distribution generalization.
arXiv preprint arXiv:2010.15775, 2020.

[Olsson et al., 2022]
C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al.
In-context learning and induction heads.
arXiv preprint arXiv:2209.11895, 2022.

[OpenAI, 2023]
Gpt-4 technical report.
ArXiv, abs/2303.08774, 2023.
URL <https://api.semanticscholar.org/CorpusID:257532815>.

[Peng et al., 2023]
B. Peng, J. Quesnelle, H. Fan, and E. Shippole.
Yarn: Efficient context window extension of large language models.
arXiv preprint arXiv:2309.00071, 2023.

[Power et al., 2022]
A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra.
Grokking: Generalization beyond overfitting on small algorithmic datasets.
arXiv preprint arXiv:2201.02177, 2022.

[Press et al., 2022]
O. Press, N. Smith, and M. Lewis.
Train short, test long: Attention with linear biases enables input length extrapolation.
In International Conference on Learning Representations, 2022.
URL <https://openreview.net/forum?id=R8sQPpGCv0>.

[Raffel et al., 2020]
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.
Exploring the limits of transfer learning with a unified text-to-text transformer.
The Journal of Machine Learning Research, 210 (1):0 5485–5551, 2020.

[Ruoss et al., 2023]
A. Ruoss, G. Delétang, T. Genewein, J. Grau-Moya, R. Csordás, M. Bennani, S. Legg, and J. Veness.
Randomized positional encodings boost length generalization of transformers.
arXiv preprint arXiv:2305.16843, 2023.

[Schwarzschild et al., 2021]
A. Schwarzschild, E. Borgnia, A. Gupta, F. Huang, U. Vishkin, M. Goldblum, and T. Goldstein.
Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks.
Advances in Neural Information Processing Systems, 34:0 6695–6706, 2021.

[Shaw et al., 2018]
P. Shaw, J. Uszkoreit, and A. Vaswani.
Self-attention with relative position representations.
arXiv preprint arXiv:1803.02155, 2018.

[Shazeer, 2020]
N. Shazeer.
Glu variants improve transformer.
arXiv preprint arXiv:2002.05202, 2020.

[Shen et al., 2023]
R. Shen, S. Bubeck, R. Eldan, Y. T. Lee, Y. Li, and Y. Zhang.
Positional description matters for transformers arithmetic.
arXiv preprint arXiv:2311.14737, 2023.

[So et al., 2021]
D. R. So, W. Mańke, H. Liu, Z. Dai, N. Shazeer, and Q. V. Le.
Primer: Searching for efficient transformers for language modeling.
arXiv preprint arXiv:2109.08668, 2021.

[Su, 2023]
J. Su.
Rectified rotary position embeddings.
<https://github.com/bojone/rerope>, 2023.

[Su et al., 2024]
J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu.
Roformer: Enhanced transformer with rotary position embedding.
Neurocomputing, 568:0 127063, 2024.

[Thoppilan et al., 2022]
R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al.
Lamda: Language models for dialog applications.
arXiv preprint arXiv:2201.08239, 2022.

[Vaswani et al., 2017]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Veličković et al., 2022]
P. Veličković, A. P. Badia, D. Budden, R. Pascanu, A. Banino, M. Dashevskiy, R. Hadsell, and C. Blundell.
The clrs algorithmic reasoning benchmark.
In International Conference on Machine Learning, pages 22084–22102. PMLR, 2022.

[Wu et al., 2022]
Y. Wu, A. Q. Jiang, W. Li, M. Rabe, C. Staats, M. Jamnik, and C. Szegedy.
Autoformalization with large language models.
Advances in Neural Information Processing Systems, 35:0 32353–32368, 2022.

[Zhang et al., 2022]
Y. Zhang, A. Backurs, S. Bubeck, R. Eldan, S. Gunasekar, and T. Wagner.
Unveiling transformers with lego: a synthetic reasoning task.
arXiv preprint arXiv:2206.04301, 2022.

[Zhou et al., 2022]
H. Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi.
Teaching algorithmic reasoning via in-context learning.
arXiv preprint arXiv:2211.09066, 2022.

[Zhou et al., 2023]
H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. Susskind, S. Bengio, and P. Nakkiran.
What algorithms can transformers learn? a study in length generalization.
arXiv preprint arXiv:2310.16028, 2023.

§ POSITIONAL ENCODING

§.§ Additive Relative Positional Encoding (RPE)

Shaw et al., 2018 pioneered additive RPE by integrating position encodings into the attention layer's key, and optionally the value, rather than the input. This concept was further simplified in T5 [Raffel et al., 2020], where the vector representations of relative positions are simplified to scalar biases added to pre-softmax attention logits. Subsequent advancements in additive RPE, aimed at enhancing length generalization and computational efficiency, include notable methods like Alibi [Press et al., 2022], Kerple [Chi et al., 2022], and FIRE [Li et al., 2023]. A commonality among these methods is the unified computation formula for pre-softmax attention logits, as outlined by Li et al., 2023:

\begin{equation}
\label{eq:rpe-attn-mat}
\mA_{\mathrm{RPE}}(\mX) = \mX \mW_Q(\mX \mW_K)^{\top}+\mB,
\end{equation}

where the bias matrix $\mB\in\R^{n\times n}$ is induced by the position encoding function $b: \sN^{*2}\to\R$, has its $(i,j)$-th entry defined as $b(i,j)$.
Variations in $b$'s formulations and parameterizations give rise to diverse RPE variants.

* T5 [Raffel et al., 2020]: T5's RPE segments relative distances into distinct buckets with a logarithmic scale, each associated with a unique parameter. With $K+1$ buckets and a pre-defined distance $L_1$, the attention bias is calculated as (assuming $K+1$ is even)

\begin{equation}\label{eq:logbinning-t5-rpe}
r_{i-j} & 0\leq i-j <  \frac{K+1}{2} \\
r_{\frac{K+1}{2} +  \lfloor\frac{K+1}{2} \log\left(\frac{2(i-j)}{K+1}\right) / \log\left(\frac{2L_1}{K+1}\right) \rfloor}    & \frac{K+1}{2}\leq i-j< L_1 \\
r_K & i-j\geq L_1
\end{cases}.
\end{equation}

* Alibi [Press et al., 2022]: $b(i,j) = -r|i-j|$, where $r>0$ is a hyper-parameter.

* Kerple [Chi et al., 2022]: $b(i,j) = -r_1\log(1+r_2|i-j|)$ (logarithmic variant) or $-r_1|i-j|^{r_2}$ (power variant), where $r_1, r_2>0$ are learnable scalars.

* FIRE [Li et al., 2023]: $b(i,j) = f_{\theta}\left(\frac{\psi(i-j)}{\psi(\max\{L, i\})}\right)$, where $f_{\theta}:\R\to\R$ is a learnable MLP parameterized by $\theta$, $\psi: \sN \to \sR_{+}$ is monotonically increasing and $L>0$ is a learnable scalar.

§ IMPLEMENTATION DETAILS

§.§ Data Generation

As shown in <Ref>, we adopt the reversed format with index hints as our default data format. During training, we randomly sample a consecutive index hints from a pre-defined ordered index set with 102 distinct symbols, thereby enhancing the learning of hint sequences and their order. At inference, the same hint sampling strategy is applied to questions, prompting the model for answers.

To generate addition examples, we opt for a naive random sampling approach instead of structured data sampling [Lee et al., 2023], as our analysis indicates that carry operations are not a major hindrance to length generalization (See <Ref>). Our approach involves uniformly selecting the number's length from 1 to the maximum training length, followed by independent sampling of two operands based on this length, with an additional zero padding to accommodate potential carry-induced extra digits. For training, datasets comprising 30M, 40M, 60M, and 120M examples are generated for number lengths 1-40, 1-30, 1-20, and 1-10, respectively. In contrast, the test set consists of 1,000 examples per digit length.

§.§ Training Details

Our base model, following Zhou et al., 2023, is a 25M parameter Transformer featuring 6 blocks, a 512 hidden size, a feedforward layer with a hidden dimension of 2048 using GeGLU activation [Shazeer, 2020], and an 8-head attention mechanism. We also adopt RMSNorm, integrating both PreNorm and PostNorm layers, following the Primer architecture [So et al., 2021]. Additionally, our preliminary investigations underscore the significance of employing causal language modeling when applying the index hint technique. Conversely, attempts to leverage prefix language modeling paired with bidirectional attention in model inputs consistently falter in length generalization. Our three other model variants with size [2M, 5M, 268M] consist of [2, 4, 16] blocks, a [256, 256, 1024] hidden size, a feedforward layer with a hidden dimension of [1024, 1024, 4096], and a [4, 4, 16]-head attention mechanism, respectively.

In our implementation of FIRE [Li et al., 2023], we employ layerwise sharing of attention bias across all attention blocks to enhance training efficiency. The paraterization of FIRE consists of a 2-layer MLP with a 32-unit hidden layer, utilizing ReLU activation.

We use the AdamW optimizer [Loshchilov and Hutter, 2017] to train the model with a weight decay value of 0.1 and dropout rate of 0.0. The learning rate schedule incorporates an initial 500-step linear warm-up, followed by a cosine decay, starting at 3e-4. We train the model with sequence packing, a batch size of 128, and a sequence length of 2048, over 50,000 steps. We use greedy decoding to generate the model output during evaluation. We summarize the hyperparameters in <Ref>.

Hyperparameters Summary for Length Generalization

Hyperparameter     Value

Language Model Type     Causal

Activation Functions     GeGLU

Normalization Layer     RMSNorm

Normalization Type     PreNorm and PostNorm

Optimizer     AdamW

Training Steps     50,000

Batch size     128

Weight Decay     0.1

Dropout     0.0

Learning Rate (LR)     0.0003

LR Warmup Steps     500

LR Cooldown (Begin, End)     (500, 50,000)

Warmup Schedule     Linear (from 0 to LR)

Cooldown Schedule     Cosine Decay (from LR to 0.1LR)

Training Sequence Length     2048

Evaluation     Greedy

§ ADDITIONAL RESULTS

§.§ Training Loss and Sequence Exact Match Accuracy of Reverse Format with Index Hint trained up to 40

Exact match accuracy on 20 to 100 digit addition of all 10 trials trained on up to 40-digit addition with index hint and reverse format using four different position encodings.

Training loss over 10 trials in reverse formats. Despite similar nearly 0 log perplexity losses across runs after 10K training steps, different runs exhibit very different length generalization.

§.§ The evolution of EM Accuracy during training in reverse format using 4 PEs

Exact match accuracy on $[30, 50, 70, 90]$ digit addition of all 10 trials trained on up to 40-digit addition with index hint and reverse format using RoPE.

Exact match accuracy on $[30, 50, 70, 90]$ digit addition of all 10 trials trained on up to 40-digit addition with index hint and reverse format using KerpleLog.

Exact match accuracy on $[30, 50, 70, 90]$ digit addition of all 10 trials trained on up to 40-digit addition with index hint and reverse format using NoPE.

Exact match accuracy on $[30, 50, 70, 90]$ digit addition of all 10 trials trained on up to 40-digit addition with index hint and reverse format using FIRE.

§.§ Effect of Index Hint

Models trained up to 10-digit addition

Models trained up to 40-digit addition

Best sequence exact match accuracy over five trials without index hint, trained upto length 10. All position encoding methods fail to generalize beyond trivial lengths and struggle with in-distribution generalization, highlighting the crucial role of index hints in length generalization. See the performance of each run in <Ref> and <Ref> for trained up to 10-digit and 40-digit addition.

§.§ Source of Variance

Exact match accuracy on $[30, 50, 70, 100]$ digit addition of all 10 trials (2 weight seeds x 5 data seeds) trained on up to 40-digit addition with index hint and reverse format using FIRE.


*[enumerate,1]label=0) tee

# Haptic Feedback Improves Human-Robot Agreement and User Satisfaction in
Shared-Autonomy Teleoperation

Dawei Zhang1, Roberto Tron2, and Rebecca P. Khurshid2 1Dawei Zhang is with the
Department of Mechanical Engineering, Boston University, Boston, MA 02215, USA
<EMAIL_ADDRESS>Tron and Rebecca P. Khurshid are with the Department of
Mechanical Engineering 8 and the Division of Systems Engineering, Boston
University, Boston, MA 02215, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Shared autonomy teleoperation can guarantee safety, but does so by reducing
the human operator’s control authority, which can lead to reduced levels of
human-robot agreement and user satisfaction. This paper presents a novel
haptic shared autonomy teleoperation paradigm that uses haptic feedback to
inform the user about the inner state of a shared autonomy paradigm, while
still guaranteeing safety. This differs from haptic shared control, which uses
haptic feedback to inform the user’s actions, but gives the human operator
full control over the robot’s actions. We conducted a user study in which
twelve users flew a simulated UAV in a search-and-rescue task with no
assistance or assistance provided by haptic shared control, shared autonomy,
or haptic shared autonomy. All assistive teleoperation methods use control
barrier functions to find a control command that is both safe and as close as
possible to the human-generated control command. For assistive teleoperation
conditions with haptic feedback, we apply a force to the user that is
proportional to the difference between the human-generated control and the
safe control. We find that haptic shared autonomy improves the user’s task
performance and satisfaction. We also find that haptic feedback in assistive
teleoperation can improve the user’s situational awareness. Finally, results
show that adding haptic feedback to shared-autonomy teleoperation can improve
human-robot agreement.

##  I INTRODUCTION

Teleoperation plays an important role in the robotics field by allowing humans
to remotely work in hard-to-reach or hazardous environments. In traditional
teleoperation systems (see Figure 1a), the human operator has full control
over all of the robot’s actions, and no assistance (NA) is provided by the
teleoperation system [1]. Unfortunately, it can be very challenging for a
human operator to effectively use such a teleoperation system. For example,
when flying a UAV, the human operator typically receives a limited field of
view, which often leads to low levels of situational awareness, making it
difficult to safely and accurately control the UAV [2, 3]. Therefore,
substantial research has been dedicated to creating systems that help the
human operator complete their desired task with higher levels of efficiency
and safety.

(a) No Assistance (NA)  (b) Haptic Shared Control (HSC)  (c) Shared Autonomy
(SA)  (d) Haptic Shared Autonomy (HSA)

Figure 1: Paradigms of shared control and autonomy for teleoperation with and
without assistance.

Haptic shared control (HSC) has proven to be a promising approach to help the
human operator remotely control a robot by providing haptic cues to the
operator [4, 5]. As illustrated in Figure 1b, the human will feel these haptic
cues but will still maintain full control authority over the robot. For
example, in a driving task, a haptic shared controller might apply a torque to
a steering wheel to help keep that car in its current lane [6]. The human
operator can choose to follow the haptic suggestions, or they can overpower
their forces to override them. The fact that the human operator maintains full
control authority over the robot is both a strength and weakness of haptic
shared control systems, depending on the application. In the case of remote
control of agile UAVs, it has been repeatedly shown that although haptic
shared control can improve a user’s ability to fly the UAV, human operators
still tend to crash UAVs even when using haptic shared control [7, 3, 8].

Shared autonomy (SA) is an alternative approach to improve the robotic
teleoperation by providing some levels of autonomous assistance. As shown in
Fig. 1c, the direct link between the human’s control command and the robot’s
action is broken by the autonomous controller. Therefore, in shared autonomy
teleoperation, the human no longer maintains full control authority of the
robot’s immediate action. Therefore, it can be difficult for the human
operator to understand how and why their commands are modified by the
autonomous assistance, which can lead to low levels of inner compatibility
between the human and the shared autonomy system. The reduced transparency of
the internal control loop can reduce users’ satisfaction and their willingness
to use the robot [9, 10].To address the challenge of balancing the task
performance and safety, with system transparency and user satisfaction, we
propose a haptic shared autonomy (HSA) teleoperation paradigm. As shown in
Fig. 1d, haptic shared autonomy uses haptic feedback to help inform the user
about the inner state of a shared autonomy system. Specifically, we propose
that haptic feedback can better enable the human operator to understand how
their commands compare to the action ultimately taken by the robot. We propose
that this will lead to improved agreement between the human operator and the
autonomous assistance, and also improve operator satisfaction.

In synthesis, the main contribution of this paper lies in proposing and
evaluating a novel haptic shared autonomy teleoperation paradigm that takes
the advantage of both shared autonomy and the ability to communicate safe
actions via haptics feedback.

##  II RELATED WORK

###  II-A Haptic Shared Control

One example of haptic shared control is in the use of virtual fixtures, which
have been widely implemented to generate force feedback when the human
operator commands the robot to an area protected by the virtual fixture [11,
12]. Other haptic shared control systems provide the human with warnings of a
risk of collision between the robot and some protected region of space. For
example, Lam et al. proposed a parametric risk field (PRF) to calculate the
risk of a collision [7]. Brant and Colton set the magnitude of the haptic
feedback to be proportional to the time that it would take the UAV to collide
with an object in its environment [3]. Recently, our prior work introduced a
haptic shared control system that used control barrier functions to help guide
the human towards the input command that is closed to their current command
and deemed to be safe, rather than simply warning them about the risk of
collision [8]. Haptic shared control has also been implemented to help a human
operator perform grasping tasks with a robotic arm. Abi-Farraj et al. used
haptic feedback to help navigate a robotic arm to a predicted grasping pose
among multiple targets [13]. In a similar setup, Pocius et al. also proposed
to use forces to communicate the shared control’s prediction of the human’s
goal to the human operator [10].

###  II-B Shared Autonomy

Like haptic shared control, shared autonomy has been used to help improve
safety for a human-controlled robot. Because the shared autonomy will command
the robot’s final action, shared autonomy systems can guarantee safety.
Recently, Xu and Sreenath used Control Barrier Functions (CBFs) as a
supervisory controller that modifies the human operator’s control input and
guarantees safe teleoperation of UAVs [14]. Similar shared-autonomy paradigms
include outer-loop stabilization [15] and parallel autonomy [16]. For example,
Schwarting et al. applied Non-linear Model Predictive Control (NMPC) to
guarantee the safety of human-controlled automated vehicles [16]. Shared
autonomy has also been applied to assistive systems that predict the user’s
intent, then alter the user’s commands so that the robot makes better progress
towards achieving the predicted human’s goal [17, 9, 18].

###  II-C Haptic Shared Autonomy

Although haptic feedback has been widely used for haptic shared control,
haptic feedback has not been well-studied in the context of shared autonomy
teleoperation. Masone et al. implemented haptic feedback in a shared planning
method, in which the robot navigated a path whose parameters were jointly
controlled by the human operator and an autonomous algorithm that ensured the
path was safe and suitable for the task [19]. The team used haptic feedback to
reflect the difference between the planned and traveled path. A user study
found that the use of haptic feedback made it easier for the user to
manipulate the robot’s planned path. The effect of haptic feedback has also
been studied in a shared autonomy paradigm, where the autonomous assistance
fully takes over some aspect of the task and disregards the human’s input.
Griffin et al. implemented a shared autonomy system to help a human better
control a robot manipulator. In this system, the autonomous assistance would
regulate the manipulator’s grip force so as to not drop an object in the
robot’s gripper. The authors were found to reflect the operator’s commanded
grip, rather than the force measured at the robot’s fingertips [20]. To our
knowledge haptic feedback has not been used in shared autonomy teleoperation
when the human and autonomous assistance are working intimately together to
have an immediate effect on the same aspect of the robot’s action, for example
the robot’s position or velocity.

##  III METHODOLOGY

As discussed in Section I, there are many approaches that could be taken to
create assistive teleoperation paradigms. In this research study, we use
control barrier functions (CBFs) to find the control signal that is as close
as possible to the human commanded control input and would also guarantee that
the UAV does not collide with an obstacle. In this section, we first discuss
our model of the UAV’s dynamics. We then briefly introduce CBFs with respect
to our second order system. More details about CBFs can be found in [21, 22,
23].

###  III-A Quadrotor UAV Dynamics and Control

In this paper, we consider quadrotor UAVs that fly at relatively low speeds
without highly aggressive maneuvers, so that the roll and pitch angles of the
quadrotor will remain small. Under such conditions, the dynamics of the UAV
can be modeled by a double integrator, where the control input $u$ corresponds
to the acceleration command of the UAV. Let $x=(x_{1},x_{2})$ with $x_{1}$
being the position of the UAV, and $x_{2}=\dot{x}_{1}$ its velocity. The
dynamics of the system then become:

$\begin{bmatrix}\dot{x}_{1}\\\ \dot{x}_{2}\end{bmatrix}=\begin{bmatrix}0&1\\\
0&0\end{bmatrix}\begin{bmatrix}x_{1}\\\ x_{2}\end{bmatrix}+\begin{bmatrix}0\\\
1\end{bmatrix}u.$ (1)

We can write (1) in a matrix form as:

$\dot{x}=Ax+Bu,$ (2)

where $A\in\mathbb{R}^{n\times n}$, $B\in\mathbb{R}^{n\times m}$ and
$u\in\mathbb{R}^{m}$. We implement a rate-control scheme, in which the
position of the control interface, $p_{i}$, is scaled by a constant factor,
$K_{v}$, to generate a desired velocity command, $x_{2,d}=K_{v}p_{i}$ for the
UAV. We then define the user’s reference control input as:

$u_{ref}=\frac{1}{\Delta t}(x_{2d}-x_{2}),$ (3)

where $x_{2}$ is the velocity of the UAV and $\Delta t$ is the rate of the
control loop.

###  III-B Control Barrier Functions

####  III-B1 Notation

We denote the lie derivative of a function $h(x)$ along the dynamics (2) as
$L_{f}h(x):=\frac{\partial h(x(t))}{\partial x(t)}Ax$,
$L_{g}h(x):=\frac{\partial h(x(t))}{\partial x(t)}B$. We use $L_{f}^{b}h(x)$
to denote a Lie derivative of order $b$.

####  III-B2 Safety Set

A continuously differentiable function $h(x)$ can be define a safety set
$\mathcal{C}$, as follows:

$\mathcal{C}:=\left\\{x\in\mathbb{R}^{n}:h(x)\geq 0\right\\}.$ (4)

####  III-B3 CBFs for Second Order Systems

The goal of control barrier functions is to always produce a control signal so
that if $x_{0}\in\mathcal{C}$ implies $x(t)\in\mathcal{C},\forall
t\in\mathrm{I}\left(x_{0}\right)$, i.e. the system is forward invariant [24].
For a second order control system, a second-order exponential control barrier
function can be used to constrain $u$ to ensure safe control. That is that if
the state of system is in the safe set, $\mathcal{C}$, then any control signal
$u$ from the set:

$U_{CBF}=\\{u\in U:L_{f}^{2}h(x)+L_{g}L_{f}h(x)u\\\
+K\begin{bmatrix}h(x)&L_{f}h(x)\end{bmatrix}^{T}\geq 0\\},$ (5)

will result in the state of the system remaining in the safety set [22] when
$K$ is chosen such that the real parts of eigenvalues of matrix $A-BK$ are
negative [22].

###  III-C Assistive Teleoperation via CBFs

We can use a Quadratic Program with CBF constraints to find a control input
that is as close as possible to the human’s commanded control while also
guaranteeing safety. This formulation results in the following optimization
problem:

$\begin{array}[]{ ll
}{u_{CBF}=\underset{u\in\mathbb{R}^{m}}{\operatorname*{\arg\\!\min}}}{\frac{1}{2}\lVert
u-u_{ref}\rVert^{2}}\\\ \text{ s.t.
}L_{f}^{2}h(x)+L_{g}L_{f}h(x)u+K\begin{bmatrix}h(x)&L_{f}h(x)\end{bmatrix}^{T}\geq
0,\end{array}$ (6)

where $u_{ref}$ is the reference control input that is provided by the human
operator and $u_{CBF}$ is the safe control input. As summarized in Table I, we
can use $u_{ref}$ and $u_{CBF}$ to create a teleoperation condition that
offers no assistance (NA), as well as haptic shared control (HSC), shared
autonomy (SA), and haptic shared autonomy (HSA) teleoperation. In NA and HSC,
the humans control input $u_{ref}$ is sent to the UAV, giving the human full
control authority. In SA and HSA, the safe control command $u_{CBF}$ is used.
HSC and HSA both use force feedback to inform the operator about the
difference between their command and the closest safe control:

$F=K_{f}(u_{CBF}-u_{ref}),$ (7)

where $K_{f}$ is a constant parameter to adjust the magnitude of the haptic
feedback.

Table I: Teleoperation Conditions

Condition | Control Signal | Haptic Feedback
---|---|---
NA | $u_{ref}$ | None
HSC | $u_{ref}$ | $K_{f}(u_{CBF}-u_{ref})$
SA | $u_{CBF}$ | None
HSA | $u_{CBF}$ | $K_{f}(u_{CBF}-u_{ref})$

##  IV USER STUDY DESIGN

The goal of this paper is to assess whether haptic feedback can improve the
user experience in a shared autonomy setting. Therefore, we designed a user
study to test the following:

###### Hypothesis 1

HSA improves the user’s task performance (speed and safety) and satisfaction
in teleoperation tasks with respect to other paradigms shown in Table I.

During the user study, participants completed a simulated search-and-rescue
task by using a simulated UAV to locate targets of interest in a forest-like
environment under each of the four conditions (NA, HSC, SA, and HSA). We
designed the user study to also enable us to test the following specific
hypotheses, which together support our main hypothesis.

###### Hypothesis 2

Teleoperation paradigms with haptic feedback (HSC and HSA) will result in
better situational awareness compared with other paradigms (NA and SA).

###### Hypothesis 3

Teleoperation paradigms that include shared autonomy (SA and HSA) will result
in better task performance compared with other paradigms (NA and HSC).

###### Hypothesis 4

HSA will lead to better human-robot agreement as compared with SA.

The study was approved by the Boston University Institutional Review Board
under protocol number 5070E.

###  IV-A Experimental Setup

During the user study, each participant navigated a simulated quadrotor UAV in
a forest-like environment, as shown in Fig. 2. The UAV and the environment
were simulated using CoppeliaSim [25]. The UAV had a radius of $0.25$ m. The
UAV was equipped with a forward-facing camera (for navigation) and a bottom-
facing camera (for “inspection”). The view captured by the forward-facing
camera was displayed on a 24-inch computer monitor, while the view captured by
the bottom-facing camera was shown as a 5.5-inch insert at the top right of
the screen, as shown in Fig. 3.

Figure 2: Each participant used a haptic joystick to control the UAV in a
simulated environment.

The forest-like environment was a $25$ m by $15$ m rectangular area that
contained 45 cylindrical tree-like obstacles, each with a radius of $0.5$ m.
The environment also contained four targets. For each trial, the participant
was asked to fly the UAV through the simulated environment to “inspect” each
of the four targets.

For this environment, the safety set is the set of all states where the robot
is not in contact with an obstacle. Approximating the quadrotor as a
cylindrical disc, then the safety set can be described by using the following
$h(x)$ in (4):

$h(x)=\lVert x_{1}-x_{1,o}\rVert^{2}-r^{2},$ (8)

where $x_{1,o}$ denotes the position of the obstacle, $r$ denotes the sum of
the radius of the UAV and the obstacle.

Figure 3: A first-person view that was provided to a human subject of the
simulated environment.

A 3D Systems Touch Haptic Device was used as the interface to control the
motion of the UAV. The height of the robot above the floor was fixed and the
user only had control of the robot’s horizontal position and yaw angle. The
user controlled the robot from a first-person perspective so that moving the
stylus forward (i.e. away from the user) would result in a motion in the
direction of the UAV’s front-facing camera, moving the stylus to the left
would result in the UAV banking left, and so on. The yaw angle of the UAV was
controlled using the two buttons on the stylus of the haptic device, with one
button commanding a counterclockwise rotation and the other commanding a
clockwise rotation. The stylus command was mapped to the UAV’s commanded
velocity $x_{2d}$ through a constant of $2$ $\frac{m/s}{cm}$, with a dead-zone
of 1 cm to help the user hover the UAV in place. The rate of yaw rotation was
$\frac{\pi}{4}$ rad/s when a button was pressed. A constant $0.3$ N restoring
force was implemented to help the user return the haptic joystick to the
deadzone. We note that these restoring forces were always present, and were
simply added to any forces generated by the teleoperation conditions. The
parameters were set up through several pilot trials to make the user feel
comfortable while also keeping the implementation of our system feasible.

###  IV-B Experimental Procedure

Twelve subjects participated in this user study (aged $23$-$34$, two females,
one left-handed). Participants held the stylus of the haptic device with their
dominant hand, and the device was positioned to be in line with the subject’s
corresponding shoulder. Each subject completed four trials with each one of
the conditions shown in Fig. 1 (for a total of 16 trials).

To reduce learning, fatigue, and order effects, the presentation order of the
control methods was counterbalanced using a Latin Square. To prevent the
subject from memorizing the spatial layout of the task, the locations of the
tree-like obstacles and the targets were randomly assigned before each trial.
For all trials, the UAV started at the same location with the same camera
orientation.

After providing informed consent, the subject was shown how to control the
simulated UAV and was given the opportunity to practice flying in an
environment without any obstacles. The subject practiced flying without using
any assistance. Subjects were told that during some trials, they may feel
forced from the 3D Systems Touch. They were given no other information about
the four test conditions. Participants were told to fly the UAV in the
simulated environment and “inspect” each one of the four targets by hovering
the UAV over the target, and pressing a button on the keyboard to take a
picture with the bottom-facing camera. The participants were asked to complete
the task as quickly as possible without colliding with the simulated obstacles
while prioritizing safety over speed.

Each trial started when the participant issued the first non-zero velocity
command to the UAV. A successful trial ended when the participant inspected
all four targets. A failed trial ended when the participant crashed the UAV
into the obstacle in such a way that the simulated UAV lost flight.
Participants completed a block of four trials for each condition, then
provided subjective measures of their experience using the NASA Task Load
Index (NASA-TLX) [26] and a custom survey. After the subject completed all
four conditions, they completed a final survey where they rank-ordered the
four conditions according to several criteria. Details of the two custom
surveys are presented with the results.

##  V RESULTS

We evaluate the results of the user study using both objective and subjective
criteria.

##### Number of Failures

We recorded the number of successful trials and failures. The numbers of
failures are shown in Fig. 4. For total 48 trials of each condition, there
were 8 failed trials under HSC and 26 failed trials under NA. As expected, no
failures were resulted under HSA and SA, since they issued the safe control
signal to the UAV.

Figure 4: Results of the number of failures.

##### Task Performance Metrics

Figure 5: Results of the metrics of the successful trials. Black brackets
indicate significance between conditions at the p$<$0.05 level. A greater
value of $V_{avg}$ indicates a better performance. Lower values of
$T_{collision}$ and $Disagreement$ are preferable.

For each successful trial, we computed the following metrics when using each
of the four conditions:

  * •

$V_{avg}$: The average linear velocity of the virtual UAV.

  * •

$T_{collision}$: It was possible for the UAV to lightly contact the obstacle
and remain in stable flight. $T_{collision}$ is the total duration of time
that the UAV was in contact with the obstacles.

  * •

$Disagreement$: The norm of the difference between the human operator’s
command and the command returned by the shared autonomy divided by trial time.

Task metrics were analyzed using a repeated-measures analysis of the variance
(rANOVA) to determine whether the condition used to control the UAV had any
effect on task performance or user experience. When a significant difference
in subject performance was found, Tukey’s test was performed at a confidence
level of $\alpha=0.05$ to determine which methods led to significant
differences in the metric.

As shown in Fig. 5, there is a significant effect on the collision duration
(F(3,27) = 6.7912, p = 0.001474). HSA and SA have shorter collision duration
compared with HSC and NA. When considering the results of the disagreement,
the four conditions also have a significant effect (F(3,27) = 22.136, p =
1.9255e-07). HSC and NA caused less disagreement as compared with HSA and SA.
In addition, HSA resulted in significantly less disagreement as compared with
SA. The results did not show a significant difference among four tested
conditions, in terms of average velocity (F(3,27) = 0.94655, p = 0.43199) for
the successfully completed trials.

##### NASA-TLX

Figure 6: Results of the NASA-TLX survey. Black brackets indicate significance
between conditions at the p$<$0.05 level.

All users’ subjective ratings of the task performance and workload are shown
in Fig. 6. No significant differences were found when comparing conditions on
ratings of physical demand (F(3,33) = 2.1183, p = 0.11671) and temporal demand
(F(3,33)= 0.13142, p = 0.9407). Significant differences were found when
comparing the effect of each condition on mental demand (F(3,33)= 6.2525, p =
0.0018), performance (F(3,33) = 15.048, p = 2.4171e-06), effort (F(3,33) =
5.582, P = 0.0033) and frustration (F(3,33)= 9.3405, p = 0.00013). HSC and SA
have significantly lower mental demand compared with NA. Participants reported
a higher level of performance for HSA as compared with HSC and NA. Subjects
also thought they performed better with HSC and SA than with NA. In addition,
NA led to higher levels of effort as compared to SA and led to higher levels
of frustration to participants as compared with both HSA and SA.

##### Post-condition Surveys

Figure 7: Results of the post-condition surveys. Black brackets indicate
significance between conditions at the p$<$0.05 level.

After completing the task with each condition, participants rated the
condition on a scale that ranged from 0 to 100 using the following questions:

  1. Q1:

How easy was it to complete the task?

  2. Q2:

How easy was it to control the robot?

  3. Q3:

How confident were you in your ability to move the robot?

  4. Q4:

How well did the robot’s motion match your intended motion?

  5. Q5:

How would you rate your overall experience with this condition?

  6. Q6:

If you felt haptic feedback, how much did the feedback help you navigate the
robot?

The results of the surveys are shown in Fig. 7. Responses to Q1-Q5 were
analyzed with rANOVA with Tukey’s post-hoc test with $\alpha=0.05$. The
results showed that subjects generally rated the three assistive conditions
more positively than NA, with significance being found for Q1 (F(3,33) =
13.055, p = 0.0001198), Q2 (F(3,33) = 7.2053, p = 0.00075369), Q3 (F(3,33) =
10.618, p = 4.8663e-05), and Q5 (F(3,33) = 10.674, p = 4.6715e-05).
Furthermore, responses to Q6 indicate that subjects felt that the haptic
feedback helped them to navigate the robot in the HSA and HSC conditions.

##### Post-experiment Rankings

Figure 8: Results of the post-experiment ranking surveys with respect to five
questions.

After all four conditions were tested, each subject completed a final survey
in which they ranked the four conditions according to their favorite method,
the method that gave them the best awareness of the robot’s environment, the
method that made them feel most immersed in the robot’s environment, the
method they would choose to accomplish a task quickly, and the method they
would choose to accomplish a task safely.

We assigned a score of 4 to the best-ranked method and a score of 1 to the
worst-ranked method. We then found the average score of all the subjects’
ranking for each condition. Fig. 8 shows the average ranking for each
condition. HSA is the most preferred condition according to each of the
metrics. Overall, subjects indicated that SA was their second favorite method
overall and was their second preferred method to complete a task quickly and
safely. Subjects indicated that HSC resulted in the second highest levels of
awareness of and immersion in the robot’s environment.

##  VI DISCUSSION

Hypothesis 2 is directly supported by the results of the rankings that users
picked HSA and HSC as the conditions that made them most aware of and feel
most immersed in the robot’s environment. HSC teleoperation resulted in fewer
failed trials and lower user ratings of mental demand than NA, further
supporting 2.

Our results also support Hypothesis 3. Because the CBF-based shared autonomy
guarantees the safety of the UAV, no subject collided with the UAV when using
either HSA or SA. Consistent with this fact, subjects also indicated that HSA
and SA would be their top two choices to complete a task safely. No
significant differences were found in the average velocity of the UAV between
the conditions. However, subjects indicated that HSA and SA would be their top
two choices to complete a task quickly, so subjects may feel most confident
when flying the UAV with these conditions, which could result in faster task
performance.

During this task, the user’s issues commands were significantly closer to the
safe commands when using HSA as compared to SA, confirming Hypothesis 4. We
note that this metric is only calculated for successful trials. Therefore, it
makes sense that conditions without any shared autonomy (NA and HSC) resulted
in the lowest levels of human-robot disagreement because the UAV would crash
if the user’s control signal is too far from a safe control signal, resulting
in a failed trial. Both SA and HSA have significantly higher levels of human-
robot disagreement than NA and HSC. However, the results show that adding
haptic feedback to a shared autonomy paradigm can lead the human to issue
commands that are much closer to the commands issued by the autonomous
assistance.

Because Hypothesis 2-4 are all supported by the user study results, we note
the providing any form of assistance via HSC, SA, or HSA can improve the
user’s performance and satisfaction when flying a UAV. HSA consistently
outperforms the other assistive conditions, SA and HSC, as measured by user
performance and user ratings. Therefore, our findings confirm Hypothesis 1.

##  VII CONCLUSIONS AND FUTURE WORK

In this paper, we proposed a novel haptic shared autonomy teleoperation
paradigm that uses haptic feedback to inform the user about the inner state of
a shared autonomy paradigm. Shared autonomy teleoperation can guarantee
safety, but does so by reducing the human operator’s level of control
authority. This can lead to reduced levels of human-robot agreement and user
satisfaction. Importantly, we find that adding haptic feedback to shared-
autonomy teleoperation can improve both human-robot agreement and user
satisfaction in shared autonomy teleoperation, while still providing formal
guarantees to improve upon human-level performance.

Our immediate future work includes testing haptic shared autonomy on a real
UAV. In this study, the haptic shared autonomy system never presented any
stability issues. However, given that we are using force feedback, in the
future we will update our control methods to guarantee system stability.

## References

  * [1] G. Niemeyer, C. Preusche, and G. Hirzinger, “Telerobotics,” in _Springer handbook of robotics_. Springer, 2008, pp. 741–757.
  * [2] J. S. McCarley and C. D. Wickens, “Human factors implications of uavs in the national airspace,” Aviation Human Factors Division, Savoy, IL, Tech. Rep. AHFD-05-05/FAA-05-01, 2005.
  * [3] A. M. Brandt and M. B. Colton, “Haptic collision avoidance for a remotely operated quadrotor UAV in indoor environments,” in _Proc. International Conference on Systems Man and Cybernetics_. IEEE, 2010, pp. 2724–2731.
  * [4] A. M. Okamura, “Haptic feedback in robot-assisted minimally invasive surgery,” _Current opinion in urology_ , vol. 19, no. 1, p. 102, 2009.
  * [5] C. Pacchierotti, _Cutaneous haptic feedback in robotic teleoperation_. Springer, 2015.
  * [6] D. A. Abbink, M. Mulder, and E. R. Boer, “Haptic shared control: smoothly shifting control authority?” _Cognition, Technology & Work_, vol. 14, no. 1, pp. 19–28, 2012.
  * [7] T. M. Lam, H. W. Boschloo, M. Mulder, and M. M. Van Paassen, “Artificial force field for haptic feedback in UAV teleoperation,” _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , vol. 39, no. 6, pp. 1316–1330, 2009.
  * [8] D. Zhang, G. Yang, and R. P. Khurshid, “Haptic teleoperation of uavs through control barrier functions,” _IEEE Transactions on Haptics_ , vol. 13, no. 1, pp. 109–115, 2020.
  * [9] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell, “Shared autonomy via hindsight optimization for teleoperation and teaming,” _The International Journal of Robotics Research_ , vol. 37, no. 7, pp. 717–742, 2018.
  * [10] R. Pocius, N. Zamani, H. Culbertson, and S. Nikolaidis, “Communicating robot goals via haptic feedback in manipulation tasks,” in _Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_ , 2020, pp. 591–593.
  * [11] J. J. Abbott, P. Marayong, and A. M. Okamura, “Haptic virtual fixtures for robot-assisted manipulation,” in _Robotics research_. Springer, 2007, pp. 49–64.
  * [12] S. A. Bowyer, B. L. Davies, and F. R. y Baena, “Active constraints/virtual fixtures: A survey,” _IEEE Transactions on Robotics_ , vol. 30, no. 1, pp. 138–157, 2013.
  * [13] F. Abi-Farraj, C. Pacchierotti, O. Arenz, G. Neumann, and P. R. Giordano, “A haptic shared-control architecture for guided multi-target robotic grasping,” _IEEE transactions on haptics_ , 2019.
  * [14] B. Xu and K. Sreenath, “Safe teleoperation of dynamic uavs through control barrier functions,” in _Proc. 2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 7848–7855.
  * [15] A. Broad, T. Murphey, and B. Argall, “Learning models for shared control of human-machine systems with unknown dynamics,” _Robotics: Science and Systems Proceedings_ , 2018.
  * [16] W. Schwarting, J. Alonso-Mora, L. Pauli, S. Karaman, and D. Rus, “Parallel autonomy in automated vehicles: Safe motion generation with minimal intervention,” in _Proc. 2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 1928–1935.
  * [17] A. D. Dragan and S. S. Srinivasa, “A policy-blending formalism for shared control,” _The International Journal of Robotics Research_ , vol. 32, no. 7, pp. 790–805, 2013.
  * [18] S. Reddy, A. D. Dragan, and S. Levine, “Shared autonomy via deep reinforcement learning,” _arXiv preprint arXiv:1802.01744_ , 2018.
  * [19] C. Masone, M. Mohammadi, P. Robuffo Giordano, and A. Franchi, “Shared planning and control for mobile robots with integral haptic feedback,” _The International Journal of Robotics Research_ , vol. 37, no. 11, pp. 1395–1420, 2018\.
  * [20] W. B. Griffin, W. R. Provancher, and M. R. Cutkosky, “Feedback strategies for telemanipulation with shared control of object handling forces,” _Presence: Teleoperators & Virtual Environments_, vol. 14, no. 6, pp. 720–731, 2005.
  * [21] A. D. Ames, J. W. Grizzle, and P. Tabuada, “Control barrier function based quadratic programs with application to adaptive cruise control,” in _Proc. 53rd IEEE Conference on Decision and Control_. IEEE, 2014, pp. 6271–6278.
  * [22] Q. Nguyen and K. Sreenath, “Exponential control barrier functions for enforcing high relative-degree safety-critical constraints,” in _Proc. 2016 American Control Conference (ACC)_. IEEE, 2016, pp. 322–328.
  * [23] W. Xiao and C. Belta, “Control barrier functions for systems with high relative degree,” in _Proc. 58th IEEE Conference on Decision and Control_ , Nice, France, 2019, pp. 474–479.
  * [24] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” in _Proc. 2019 18th European Control Conference (ECC)_. IEEE, 2019, pp. 3420–3431.
  * [25] E. Rohmer, S. P. Singh, and M. Freese, “V-rep: A versatile and scalable robot simulation framework,” in _Proc. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 1321–1326.
  * [26] S. G. Hart and L. E. Staveland, “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research,” in _Advances in psychology_. Elsevier, 1988, vol. 52, pp. 139–183.