# AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for
Adapted Behavior Synthesis

Jieyeon Woo
ISIR - Sorbonne University
France
<EMAIL_ADDRESS>
Mireille Fares
ISIR - Sorbonne University
France
<EMAIL_ADDRESS>
Catherine Pelachaud
CNRS - ISIR - Sorbonne University
France
<EMAIL_ADDRESS>
Catherine Achard
ISIR - Sorbonne University
France
<EMAIL_ADDRESS>

###### Abstract

Socially Interactive Agents (SIAs) are physical or virtual embodied agents
that display similar behavior as human multimodal behavior. Modeling SIAs’
non-verbal behavior, such as speech and facial gestures, has always been a
challenging task, given that a SIA can take the role of a speaker or a
listener. A SIA must emit appropriate behavior adapted to its own speech, its
previous behaviors (intra-personal), and the User’s behaviors (inter-personal)
for both roles. We propose AMII, a novel approach to synthesize adaptive
facial gestures for SIAs while interacting with Users and acting
interchangeably as a speaker or as a listener. AMII is characterized by
modality memory encoding schema \- where modality corresponds to either speech
or facial gestures \- and makes use of attention mechanisms to capture the
intra-personal and inter-personal relationships. We validate our approach by
conducting objective evaluations and comparing it with the state-of-the-art
approaches.

_K_ eywords Adapted Behavior Synthesis $\cdot$ Reciprocal Adaptation $\cdot$
Conversational AI $\cdot$ Multimodal modeling

##  1 Introduction

Socially Interactive Agents (SIAs) are embodied agents emitting human-like
non-verbal behavior such as facial, body, and hand gesturing during speech
[1]. In many applications such as medical therapy [2] and educational
assistance [3], they can carry out interactive and natural conversations with
humans (Users) by sending and receiving multimodal non-verbal signals [4, 5]
in addition to the verbal message. To render SIA-User communication effective
and engaging, SIAs must play active roles of speakers and listeners. For
instance, in successful and engaging human-human conversations, speaker’s and
listener’s behaviors are constantly coordinated and adapted to each other [6],
through the perception and response of the multimodal emitted signals such as
linguistic [7], facial [5], and body [8] features. The behavior adaptation can
be: (1) intra-personal which is between the multimodal signals of a single
person or (2) inter-personal which is between listener’s and speaker’s signals
adapting to the other’s verbal and non-verbal behavior. This adaptation is
continuous, reciprocal and dynamical and is referred to as reciprocal
adaptation [9]. The display of such adaptation capability can enable SIAs to
be perceived as social and engaging [10].

Various works have generated SIAs’ non-verbal behavior focusing on only the
modeling of intra-personal relationship for a single person [11, 12, 13].
Other works looked into capturing multimodal inter-personal information in
dyadic settings. In the work of Feng et al. [14] and Dermouche et al. [15],
SIA facial gestures are synthesized based on past gestures of both SIA and
User without considering the existing relation with audio modality [16, 17],
and do not ensure the motion continuity. Grafsgaard et al. [18] synthesize
interlocutor’s gestures based on both the interlocutors’ audio and their
facial modalities. In the work of Jonell et al. [19], SIA’s facial gestures
are generated based on SIA’s speech and the User’s speech and facial gestures.
However, these models [18, 19] are prone to produce non-continuous gestures.
The works presented in Woo et al. [20, 21] and Ng et al. [22] ensure SIA’s
behavior continuity by employing autoregressive online inference while
modeling SIA’s and User’s multimodal features. Only the listener’s behavior is
modeled in [22]. [20, 21] and all the aforementioned models for dyadic setting
do not explicitly model the intra-personal relationship.

Our overall aim is to create a social and engaging SIA by modeling its
behavior adaptation. Our hypothesis is that behavior adaptation is captured by
intra-personal and inter-personal synchronizations. We propose Adaptive
Multimodal Inter-personal and Intra-personal (AMII) model, a novel method to
synthesize adaptive and engaging facial gesturing for SIAs. We explicitly
model the intra-personal relationship by encoding the prior emitted multimodal
signals (modality memory) while ensuring motion continuity. We model the
inter-personal relationship to generate SIA behavior for both roles of speaker
and listener. We explore the best way to capture the reciprocal adaptation to
generate adaptive non-verbal SIA behavior within a dyadic setting. Intra-
personal and inter-personal relations are learned through attention
mechanisms. We demonstrate that our AMII model outperforms the state-of-the-
art models, in terms of both behavior appropriateness and reciprocal
adaptation resemblance, by conducting objective evaluations. We perform
ablation studies and show the influence of AMII’s key encoders.

Our contributions are as follows:

  * •

We propose AMII, an approach to capture the reciprocal adaptation of SIA
behaviors.

  * •

We explicitly model the intra-personal and inter-personal relationships by
encoding prior emitted multimodal signals via our modality memory encoders.

The paper is organized as follows. Section 2 describe the proposed AMII model
architecture. The dataset is presented in Section 3 and we detail the training
regime in Section 4. Then, we report our experiments in Section 5 and discuss
our results in Section 6. We finally conclude and talk about our future works
in Section 7.

##  2 Adaptive Multimodal Inter-personal and Intra-personal (AMII) Model
Architecture

Figure 1: AMII (Adaptive Multimodal Intra-personal and Inter-personal) model
architecture. The intra-personal encoder ($E_{intra}$) takes the speech
$X_{speech}$ and the facial gestures $X_{face}$ of the previous 100 frames of
either the SIA ($A$) or the User ($U$) to encode the corresponding intra-
personal relationship $Z_{intra}$. The inter-personal encoder ($E_{inter}$)
learns from intra-personal relationships $Z_{intra}^{A}$ and $Z_{intra}^{U}$
to encode the inter-personal relationship between them $Z_{inter}$. The
behavior generator ($G_{face}$) takes $Z_{intra}^{A}$, $Z_{intra}^{U}$, and
$Z_{inter}$ to generate the sequence of facial gestures for the next frame t+1
$\widehat{Y}_{face}^{A}$ and $\widehat{Y}_{face}^{U}$. At training time, AMII
is trained with human-human ($U_{1}$-$U_{2}$) interactions ($U_{1}$ for $A$
and $U_{2}$ for $U$) and predicts both of humans’ facial gestures
($\widehat{Y}_{face}^{U_{1}}$ and $\widehat{Y}_{face}^{U_{2}}$). At inference
time, AMII renders the facial gestures of $A$ and $U$. To infer the next $A$’s
behavior, we feed back the predicted $A$’s behavior and the ground truth of
$U$.

We focus on generating adaptive non-verbal SIA behavior as both a speaker and
listener. For this purpose, we propose an approach trained on real human-human
interactions, to learn human-human inter-personal and intra-personal relations
for SIA and simulate our predictions on a SIA. We propose a new model
architecture, the Adaptive Multimodal Inter-personal and Intra-personal model
(AMII), to synthesize adaptive and engaging facial gestures for SIAs. It takes
as input speech and facial gestures of both SIA ($A$) and User ($U$),
corresponding to their past behavior, and predicts SIA’s and User’s facial
gestures at the next time step.

Facial gestures are represented by:

  * •

Gaze movements: represented by $G_{x}$ and $G_{y}$ which are the gaze angles
w.r.t. the $x$ and $y$ axis.

  * •

Head movements: represented by $R_{x}$, $R_{y}$, and $R_{z}$ which are the
Euler head rotations w.r.t. the $x$, $y$ and $z$ axis.

  * •

Facial movements: represented by facial Action Units (AUs) [23] which are
facial muscle movements defined by the Facial Action Coding Systems (FACS)
[24]. We use AU1 (inner brow raiser), AU2 (outer brow raiser), AU4 (brow
lowerer), AU6 (cheek raiser), and AU12 (lip corner puller for smile).

Speech and facial gestures are highly tied together [16, 17]. With this
relation, we decide to use speech information to drive SIA’s facial gestures.
Speech features are listed as follows:

  * •

Pitch: represented by the fundamental frequency $f_{0}$, which is the main
prosodic feature correlated with facial gestures [25].

  * •

Loudness: speech intensity from the auditory spectra.

  * •

Voicing probability:  speech presence probability expressed as a probability
score in the range of $0$ to $1$.

  * •

Mel-Frequency-Cepstral Coefficients (MFCC): represented by $13$ MFCC features
($0$-$12$).

AMII model operates as follows. It takes as input the $100$ past frames
($t-99:t$), where $t$ is the current frame, of the:

  1. 1.

Speech features of $A$ ($X_{speech}^{A}$) and those of $U$ ($X_{speech}^{U}$),

  2. 2.

Facial features of $A$ ($X_{face}^{A}$) and those of $U$ ($X_{face}^{U}$).

For each prediction of the next frame ($t+1$), the model predicts:

  1. 1.

$A$’s facial gestures ($\widehat{Y}_{face}^{A}$),

  2. 2.

$U$’s facial gestures ($\widehat{Y}_{face}^{U}$).

AMII consists of three main components, as illustrated in Figure 1. The first
component is the intra-personal encoder $E_{intra}$, which explicitly encodes
the intra-personal relations via a modality memory schema. This schema
consists of encoding each modality \- speech features and facial features \-
corresponding to the past 100 frames. The second component is the inter-
personal encoder $E_{inter}$, which encodes the inter-personal relations by
applying cross-attentions between A’s and U’s features’ embeddings. The last
component is the behavior generator which generates A’s and U’s facial
gestures of the next frame. These components are detailed in the following.

###  2.1 Intra-personal Encoder ($E_{intra}$)

As shown in Figure 1, $E_{intra}$ takes as input $X_{speech}$ and $X_{face}$
of either A or U and generates the corresponding intra-personal embedding
$Z_{intra}$. It consists of two sub-encoders. The first is the modality memory
encoder ($E_{speech}$ or $E_{face}$). The second is the dual-modality encoder
$E_{dual}$.

#### Modality Memory Encoder ($E_{speech}$ or $E_{face}$)

Both $E_{speech}$ and $E_{face}$ takes its corresponding modality \-
$X_{speech}$ or $X_{face}$ respectively - as input and renders the modality
memory embedding $Z_{speech}^{mem}$ and $Z_{face}^{mem}$ representing the past
$100$ frames. Each corresponding modality memory encoder firstly learns the
modality specific information by applying self-attention, with a $h=2$ (where
$h$ is the head size), preceded and followed by dense layers ($E_{d}$) with
$c=16$ (where $c$ is the cell size). Then, it embeds the memory sequence of
the chosen modality via a LSTM layer ($E_{m}$) with $c=16$, as depicted in
Figure 1. It takes $X_{mod}$ \- where $mod$ represents either speech or facial
gestures \- and outputs $Z_{mod}^{mem}$, and can be expressed as:

$Z_{mod}^{mem}=E_{m}\left(E_{d}\left(SA\left(E_{d}\left(X_{mod}\right)\right)\right)\right)$
(1)

where $SA$($\cdot$) denotes self-attention layer.

#### Dual-modality Encoder ($E_{dual}$)

$E_{dual}$ captures the relationship between the multimodal signals by
applying cross-attention mechanisms on the corresponding modalities:
$CA^{speech}$ and $CA^{face}$ with $h=2$ followed by $E_{d}$ with $c=16$, as
shown in Figure 1. $CA^{speech}$ has a query $Q$ equals to $Z_{speech}^{mem}$
with key $K$ and value $V$ equal to $Z_{face}^{mem}$. $CA^{face}$ has a query
$Q$ equals to $Z_{face}^{mem}$ with key $K$ and value $V$ equal to
$Z_{speech}^{mem}$. $E_{dual}$ takes $Z_{speech}^{mem}$ and $Z_{face}^{mem}$
as inputs and generates $Z_{intra}$. It can be written as:

$Z_{intra}=E_{d}\left(\left[\right.\right.CA_{speech}\left(Q_{speech},K_{face},V_{face}\right),\\\
CA_{face}\left(Q_{face},K_{speech},V_{speech}\right)\left.\left.\right]\right)$
(2)

where $CA$($Q,K,V$) denotes cross-attention layer and $[\cdot]$ denotes
concatenation layer.

###  2.2 Inter-personal Encoder ($E_{inter}$)

As illustrated in Figure 1, $E_{inter}$ takes as input $Z_{intra}^{A}$ and
$Z_{intra}^{U}$, which are the intra-personal representations of $A$ and $U$
respectively. It renders $Z_{inter}$, a representation of inter-personal
relation between $A$ and $U$. $E_{inter}$ applies cross-attention mechanisms
on the both intra-personal representations: $CA^{A}$ and $CA^{U}$ with $h=2$
followed by $E_{d}$ with $c=16$. $CA^{A}$ has a query $Q$ equals to
$Z_{intra}^{A}$ with key $K$ and value $V$ equal to $Z_{intra}^{U}$. $CA^{U}$
has a query $Q$ equals to $Z_{intra}^{U}$ with key $K$ and value $V$ equal to
$Z_{intra}^{A}$. It can be written as:

$Z_{inter}=E_{d}\left(\left[CA^{A}\left(Q^{A},K^{U},V^{U}\right),CA^{U}\left(Q^{U},K^{A},V^{A}\right)\right]\right)$
(3)

where $CA$($Q,K,V$) denotes cross-attention layer and $[\cdot]$ denotes
concatenation layer.

###  2.3 Behavior Generator ($G_{face}$)

$G_{face}$ takes as input the:

  1. 1.

$A$’s intra-personal representation ($Z_{intra}^{A}$),

  2. 2.

$U$’s intra-personal representation ($Z_{intra}^{U}$),

  3. 3.

Inter-personal representation of $A$ and $U$ ($Z_{inter}$).

It generates the corresponding facial gestures $\widehat{Y}_{face}^{A}$ and
$\widehat{Y}_{face}^{U}$ by decoding with a dense layer ($D_{d}$) with $c=20$,
as depicted in Figure 1. The final outputs $\widehat{Y}_{face}^{P}$ can be
written as:

$\widehat{Y}_{face}^{P}=D_{d}\left(Z_{intra}^{P},Z_{inter}\right)$ (4)

where $P$ represents either A or U. At training time, AMII synthesizes
$\widehat{Y}_{face}^{U}$ to better learn inter-personal and intra-personal
relations. $\widehat{Y}_{face}^{U}$ is disregarded at inference time since the
aim is to predict only $A$.

###  2.4 Training and Inference Modes

We train our model on real human-human ($U_{1}$-$U_{2}$) interactions. Our
model learns to synthesize adapted gestures of $U_{1}$ and $U_{2}$. During
inference, AMII synthesizes the behavior of $A$ and $U$.
$\widehat{Y}_{face}^{A}$ is inferred using the previous prediction of $A$ and
the ground truth of $U$. We apply adaptive online prediction to generate
continuous A’s behavior in an autoregressive fashion.

##  3 NoXi Dataset

For our experiments, we choose the French NoXi dataset [26] consisting of $21$
human-human dyadic interactions performed by $28$ participants with a total
duration of $7h22min$. We extract non-verbal behavior features for each time-
step: openSMILE [27] for speech and OpenFace [28] for facial gestures. Data
preprocessing - median filter and linear interpolation - is applied on both
extracted features and they are adjusted to $25fps$. We split our dataset into
$3$ sets: training ($70\%$), validation ($10\%$), and test ($20\%$). The test
set does not include data of speakers and listeners that are seen during
training. The aim is to test AMII’s capacity to extrapolate on new unseen
speakers and listeners and therefore its capability to generalize.

##  4 Training Regime

To train our model, we use the Mean Squared Error (MSE) as our loss function
and the Adam optimizer [29] with Cyclical Learning Rate (CLR) [30] (triangular
learning rate policy, base_lr of $1e-7$, max_lr of $1e-3$, and step size
factor of $10$). The training was done for $300$ epochs (with an average
runtime of $125h$) on a $2.2GHz$ Intel Xeon Linux server with NVIDIA GeForce
GTX TITAN X and $64GB$ RAM with a batch size of $32$. The best set of
hyperparameters is chosen after manual optimization based on the validation
set.

##  5 Experiments

To assess our model, we conduct objective evaluation to check its performance
against the state-of-the-art approaches, which we select as our baselines, and
to verify the effectiveness of each AMII’s key components through ablation
studies.

###  5.1 Objective Metrics

We want to assess whether the generated behavior is appropriate (intra-
personal) and reciprocally adaptive (inter-personal). To do so, we employ the
metrics used in previous works [14, 15, 18, 20, 21, 22]. We measure behavior
appropriateness of $A$’s predictions ($\widehat{A}$) against its ground truth
(GT) behavior ($A$). The metrics are as follows:

  * •

MAE and RMSE/L2: distance between the predictions and GT to measure the
generated error, which are used in [14, 15, 20, 21, 22].

  * •

Kolmogorov-Smirnov two-sample test (KS test) [31]: density probability
difference between A’s corresponding predictions and GT to check for the
distribution similarities between them, which is used in [21].

For reciprocal adaptation resemblance, we measure the resemblance between
$\widehat{A}$ and $U$’s GT data ($U$). We choose to assess only smile (AU12)
as it is a key socio-emotional signal [8]. The metrics are as follows:

  * •

Time lagged cross correlation coefficient (TLCC) [32]: linear relationship
invariant to speed, which is used to quantify global synchrony like PCC. TLCC
is computed in chunks of $8sec$ with a time lag of $2sec$ as in [22].

  * •

DTW [33]: proximity/resemblance check of $\widehat{A}\&U$ against that of the
GT interaction (between two humans) to evaluate reciprocal adaptation. DTW is
computed in chunks of $1min$ with a stride of $30sec$ as in [21].

  * •

Synchrony (Sync) and Entrainment Loop (EL) [9]: synchrony and entrainment loop
measures proposed in [9] to evaluate the reciprocal adaptation between
$\widehat{A}$ and U.

Lower values denote better performance for MAE, RMSE, and KS test. For the
resemblance metrics (PCC, TLCC, DTW, Sync, and EL), the closer the value of
the metric is to the GT, the better the model performs in generating adaptive
A’s behaviors.

###  5.2 Baseline Models and Ablation Studies

We consider the following baseline models (base):

  * •

IL-LSTM [15]: models only the inter-personal based on a single modality of
facial gestures of A and U,

  * •

Symmetrized IL-LSTM with online LSTM (sym-IL-LSTM) [20] and ASAP [21]: model
the inter-personal relation of A and U, multimodality of speech and facial
gestures, and assure motion continuity.

To check for the effectiveness and influence of each of AMII’s key encoders,
we conducted additional ablation studies. We performed the ablations of:

  * •

Modality Memory Encoder $E_{m}$ ($noE_{m}$),

  * •

Dual-modality Encoder $E_{dual}$ ($noE_{dual}$),

  * •

Inter-personal Encoder $E_{inter}$ ($noE_{inter}$).

##  6 Results and Discussion

Table 1: Objective evaluation of AMII against the baselines along with ablations using the selected metrics. GT denotes ground truth interaction. Best results are highlighted in bold. $\Delta_{base}$ represents the change in performance over the best performing baseline approach of each metric. $\Delta_{base}$ entries in green when AMII outperforms best baseline, in red when it is not the case. | MAE | RMSE | KS test | TLCC | DTW | Sync | EL
---|---|---|---|---|---|---|---
GT |  |  |  | 0.334 | 1317.5 | 132.4 | 1172.3
IL-LSTM [15] | 0.304 | 0.415 | 0.329 | 0.343 | 1216.2 | 45.3 | 323.3
sym-IL-LSTM [20] | 0.180 | 0.227 | 0.284 | 0.335 | 1281.3 | 33.3 | 232.3
ASAP [21] | 0.185 | 0.254 | 0.282 | 0.317 | 1399.3 | 142.0 | 1890.5
AMII-$noE_{m}$ (ours) | 0.099 | 0.132 | 0.515 | 0.271 | 1228.4 | 79.1 | 603.8
AMII-$noE_{dual}$ (ours) | 0.143 | 0.186 | 0.396 | 0.300 | 1352.6 | 262.3 | 2255.8
AMII-$noE_{inter}$ (ours) | 0.136 | 0.178 | 0.406 | 0.261 | 1127.8 | 82.3 | 586.3
AMII (ours) | 0.156 | 0.197 | 0.437 | 0.291 | 1319.6 | 137.4 | 989.0
$\Delta_{base}$ |  $\downarrow$ 0.024 |  $\downarrow$ 0.030 |  $\uparrow$ 0.155 |  $\uparrow$ 0.042 |  $\downarrow$ 79.7 |  $\downarrow$ 4.6 |  $\downarrow$ 534.9

The evaluation results are listed in Table 1. $\Delta_{base}$ represents the
change in performance over the best performing baseline approach for each
metric.

#### Comparing with Baselines

We remark that AMII outperforms the baselines in terms of behavior
appropriateness. This is reflected through low errors of MAE and RMSE
represented by $\Delta_{base}$ ($\downarrow$ $0.024$ and $\downarrow$ $0.020$
respectively). For the density distribution, via the KS test, we observe that
AMII performs comparatively less than the baselines indicating that AMII
possesses the least similar density distribution compared to that of the GT.
In detail, ASAP performs the best in terms of having the most similar density
w.r.t. GT ($0.282$) and AMII the worst ($0.437$) with $\Delta_{base}$ of
$\uparrow$ $0.155$. This low performance of AMII does not imply that it
generates wrong SIA behavior but that it has either a smaller or a wider range
of behavior variety than that of the GT. The focus of this study is not to
produce a variety of behaviors but to generate SIA behaviors that are adaptive
to its interlocutor. Thus, this weak performance of the KS test metric is not
critical for our aim. Moreover, AMII performs the best in terms of the
reciprocal adaptation resemblance metrics as seen in the Table 1. DTW,
synchrony, and entrainment loop of $\widehat{A}\&U$ show that AMII resembles
the GT the most with $\Delta_{base}$ of $\downarrow$ $79.7$, $\downarrow$
$4.6$, and $\downarrow$ $534.9$ respectively. With TLCC, we remark that the
sym-IL-LSTM is the closest to the GT while AMII is the farthest one with
$\Delta_{base}$ of $\uparrow$ $0.042$. As DTW considers the variation of
sequence length while being invariant to speed unlike TLCC, it represents
better the global correlation. Thus, for the interpretation, we can put more
emphasis on the DTW results compared to that of TLCC. This comparative study
shows that the inclusion of explicit modeling of intra-personal relation
leverages the quality of produced gestures in terms of both behavior
appropriateness and reciprocal adaptation resemblance.

#### Ablation studies

The ablation of each of AMII key encoder - $E_{m}$, $E_{dual}$, and
$E_{inter}$ \- results in the improvement of the reciprocal adaptation
resemblance. This was seen by an increase in DTW ($87.0$, $33.0$, and $187.6$
respectively), synchrony ($48.3$, $124.9$, and $45.1$ respectively), and
entrainment loop resemblance ($385.2$, $900.2$, and $402.7$ respectively).
TLCC shows that the insertion of $E_{m}$ improves the AMII by $0.020$ along
with $E_{inter}$ by $0.030$. However, $E_{dual}$ slightly deteriorates the
performance by $0.009$. As in the baseline comparison study, it is better to
concentrate on the other reciprocal adaptation resemblance metrics as DTW is a
more dynamic measure of synchrony than TLCC. However, this enhancement of
reciprocal adaptation resemblance is at the expense of lowering its behavior
appropriateness performance. This is observed via MAE ($0.057$, $0.013$,
$0.020$ respectively) and RMSE ($0.065$, $0.011$, $0.019$ respectively). The
same conclusion can be drawn by looking at KS test result. The fall of
performance is seen for the additions of $E_{dual}$ ($0.041$) and $E_{inter}$
($0.031$) while $E_{m}$ improves ($0.078$). This compromise of losing behavior
appropriateness to gain an adaptive one may be a good exchange. It is more
valuable to generate SIA behaviors with adaptation capacity than to reproduce
the same GT behavior. In fact, in a human-human interaction there could be
multiple possible behaviors and generation timings facing a same interacting
partner’s behavior. This might vary depending on the various factors such as
the context, situation, and interlocutor’s personality and mood.

We can conclude that it is important to model the intra-personal relation with
the encodings of $E_{m}$ and $E_{dual}$, and the inter-personal relation with
$E_{inter}$ to synthesize adaptive non-verbal facial gestures for both roles
as speaker and listener.

##  7 Conclusions and Future works

In this paper, we propose a new approach to generate adaptive SIA behavior as
both speaker and listener by encoding the multimodality and intra-personal and
inter-personal relationships. We conclude that AMII model achieves state-of-
the art performance notably in terms of reciprocal adaptation resemblance. Our
approach still has some limitations. We do not model the inter-personal
relation memory which could improve our results. As the next step, we plan to
add this component. We also plan to integrate and assess AMII within a real-
time SIA-User interaction. Our overall aim is to model SIA that can socially
engage users in an interaction through its behaviors that capture intra-
personal and inter-personal relations. In the next future, we will
subjectively evaluate the dimensions of social attitudes and engagement to
validate this hypothesis.

## References

  * [1] Justine Cassell, Tim Bickmore, Lee Campbell, Hannes Vilhjalmsson, and Hao Yan.  Human conversation as a system framework: Designing embodied conversational agents.  Embodied conversational agents, pages 29–63, 2000.
  * [2] Kazuhiro Shidara, Hiroki Tanaka, Hiroyoshi Adachi, Daisuke Kanayama, Yukako Sakagami, Takashi Kudo, and Satoshi Nakamura.  Automatic thoughts and facial expressions in cognitive restructuring with virtual agents.  Frontiers in Computer Science, 4:8, 2022.
  * [3] Yanghee Kim, Jeffrey Thayne, and Quan Wei.  An embodied agent helps anxious students in mathematics learning.  Educational Technology Research and Development, 65(1):219–235, 2017.
  * [4] Joseph N Cappella.  Mutual adaptation and relativity of measurement.  Studying interpersonal interaction, 1:103–117, 1991.
  * [5] Judee K Burgoon, Laura K Guerrero, and Valerie Manusov.  Nonverbal signals.  The SAGE handbook of interpersonal communication, pages 239–280, 2011.
  * [6] Judee K Burgoon, Lesa A Stern, and Leesa Dillman.  Interpersonal adaptation: Dyadic interaction patterns. Cambridge University Press, 1995.
  * [7] Martin J Pickering and Simon Garrod.  Toward a mechanistic psychology of dialogue.  Behavioral and brain sciences, 27(2):169–190, 2004.
  * [8] Mark L Knapp, Judith A Hall, and Terrence G Horgan.  Nonverbal communication in human interaction.  Cengage Learning, 2013.
  * [9] Jieyeon Woo, Catherine Pelachaud, and Catherine Achard.  Reciprocal adaptation measures for human-agent interaction evaluation.  In ICAART, 2023.
  * [10] Beatrice Biancardi, Soumia Dermouche, and Catherine Pelachaud.  Adaptation mechanisms in human–agent interaction: Effects on user’s impressions and engagement.  Frontiers in Computer Science, 3:696682, 2021.
  * [11] Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow.  Style-controllable speech-driven gesture synthesis using normalising flows.  In Computer Graphics Forum, volume 39, pages 487–496. Wiley Online Library, 2020.
  * [12] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents** this work has been supported in part by aro grants w911nf1910069 and w911nf1910315, and intel. code and additional materials available at: https://gamma. umd. edu/t2g.  In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 1–10. IEEE, 2021.
  * [13] Mireille Fares, Catherine Pelachaud, and Nicolas Obin.  Zero-shot style transfer for multimodal data-driven gesture synthesis.  In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–4. IEEE, 2023.
  * [14] Will Feng, Anitha Kannan, Georgia Gkioxari, and C Lawrence Zitnick.  Learn2smile: Learning non-verbal interaction through observation.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4131–4138. IEEE, 2017.
  * [15] Soumia Dermouche and Catherine Pelachaud.  Generative model of agent’s behaviors in human-agent interaction.  In 2019 International Conference on Multimodal Interaction, pages 375–384, 2019.
  * [16] Marc D Pell.  Prosody–face interactions in emotional processing as revealed by the facial affect decision task.  Journal of Nonverbal Behavior, 29:193–215, 2005.
  * [17] Hani C Yehia, Takaaki Kuratate, and Eric Vatikiotis-Bateson.  Linking facial animation, head motion and speech acoustics.  Journal of phonetics, 30(3):555–568, 2002.
  * [18] Joseph Grafsgaard, Nicholas Duran, Ashley Randall, Chun Tao, and Sidney D’Mello.  Generative multimodal models of nonverbal synchrony in close relationships.  In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 195–202. IEEE, 2018.
  * [19] Patrik Jonell, Taras Kucherenko, Gustav Eje Henter, and Jonas Beskow.  Let’s face it: Probabilistic multi-modal interlocutor-aware generation of facial gestures in dyadic settings.  In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pages 1–8, 2020.
  * [20] Jieyeon Woo, Catherine Pelachaud, and Catherine Achard.  Creating an interactive human/agent loop using multimodal recurrent neural networks.  In WACAI 2021, 2021.
  * [21] Jieyeon Woo, Catherine Pelachaud, and Catherine Achard.  Asap: Endowing adaptation capability to agent in human-agent interaction.  In 28th International Conference on Intelligent User Interfaces, 2023.
  * [22] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.  Learning to listen: Modeling non-deterministic dyadic facial motion.  arXiv preprint arXiv:2204.08451, 2022.
  * [23] Paul Ekman and Wallace V Friesen.  Measuring facial movement.  Environmental psychology and nonverbal behavior, 1(1):56–75, 1976\.
  * [24] Paul Ekman and Wallace V Friesen.  Facial action coding system.  Environmental Psychology & Nonverbal Behavior, 1978.
  * [25] Dwight L Bolinger.  A theory of pitch accent in english.  Word, 14(2-3):109–149, 1958.
  * [26] Angelo Cafaro, Johannes Wagner, Tobias Baur, Soumia Dermouche, Mercedes Torres Torres, Catherine Pelachaud, Elisabeth André, and Michel Valstar.  The noxi database: Multimodal recordings of mediated novice-expert interactions.  In Proceedings of the 19th ACM International Conference on Multimodal Interaction, page 350–359. Association for Computing Machinery, 2017\.
  * [27] Florian Eyben, Martin Wöllmer, and Björn Schuller.  Opensmile: the munich versatile and fast open-source audio feature extractor.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462, 2010.
  * [28] Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency.  Openface: an open source facial behavior analysis toolkit.  In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–10. IEEE, 2016.
  * [29] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [30] Leslie N Smith.  Cyclical learning rates for training neural networks.  In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464–472. IEEE, 2017.
  * [31] Frank J Massey Jr.  The kolmogorov-smirnov test for goodness of fit.  Journal of the American statistical Association, 46(253):68–78, 1951.
  * [32] Steven M Boker, Jennifer L Rotondo, Minquan Xu, and Kadijah King.  Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series.  Psychological methods, 7(3):338, 2002.
  * [33] Meinard Müller.  Dynamic time warping.  Information retrieval for music and motion, pages 69–84, 2007.


# Coordinated Pose Control of Mobile Manipulation with an Unstable Bikebot
Platform ††thanks: This work was partially supported by the US National
Science Foundation under award CNS-1932370.

Feng Han, Alborz Jelvani, Jingang Yi, and Tao Liu F. Han and J. Yi are with
the Department of Mechanical and Aerospace Engineering, Rutgers University,
Piscataway, NJ 08854 USA (e-mail<EMAIL_ADDRESS>jgyi@rutgers.edu).A. Jelvani is with the Department of Computer Science,
Rutgers University, Piscataway, NJ 08854 USA (e-mail:
aj654@scarletmail.rutgers.edu).T. Liu is with the State Key Lab of Fluid Power
and Mechatronic Systems and the School of Mechanical Engineering, Zhejiang
University, Hangzhou, Zhejiang 310027 China (email: liutao@zju.edu.cn).

###### Abstract

Bikebot manipulation has advantages of the single-track robot mobility and
manipulation dexterity. We present a coordinated pose control of mobile
manipulation with the stationary bikebot. The challenges of the bikebot
manipulation include the limited steering balance capability of the unstable
bikebot and kinematic redundancy of the manipulator. We first present the
steering balance model to analyze and explore the maximum steering capability
to balance the stationary platform. A balancing equilibrium manifold is then
proposed to describe the necessary condition to fulfill the simultaneous
platform balance and posture control of the end-effector. A coordinated
planning and control design is presented to determine the balance-prioritized
posture control under kinematic and dynamic constraints. Extensive experiments
are conducted to demonstrate the mechatronic design for autonomous plant
inspection in agricultural applications. The results confirm the feasibility
to use the bikebot manipulation for a plant inspection with end-effector
position and orientation errors about 5 mm and 0.3 degs, respectively.

###### Index Terms:

Underactuated robots, balance control, mobile manipulation, task priority
planning, bicycle control

##  I Introduction

Mobile manipulation integrates a mobile robot with an onboard multi-link
manipulator to expand workspace and improve capability for complex
manipulation tasks [1, 2, 3]. Mobile manipulation can be built on wheeled,
legged or aerial platforms and the applications include agriculture harvesting
[4], mobile cranes [5], underwater archaeology [6], and aerial manipulation
[7, 8], etc. The advantages of the mobile manipulation come at the cost of
coordinated planning and control [9]. The coupled dynamics of the mobile
platform and manipulator is one of the design challenges [10]. Unknown or
uncertain robot-environment interactions bring additional complexity for
control of mobile manipulation [5, 11]. For instance, ocean waves and tides
cause large dynamic disturbances to the ship-mounted manipulator [12] and a
wheeled/legged mobile robot would fall down when moving on a steep or rocky
field [13]. For aerial manipulation, interaction forces generates large
disturbances for robot motion and control due to small masses and limited
actuation of the quadrotors [7, 14].

Coordinated planning and control is critical when the mobile platform is
unstable or in complex, dynamic environments. Balance control of unstable
platform is among the highest priority tasks for mobile manipulation. In [15],
a model predictive control was presented for collaborative manipulation,
balancing and interaction of a ball-based three degree-of-freedom (DOF)
manipulator. In [16], an unstable mobile manipulation used a single spherical
wheel-based “ballbot” as the mobile platform and a balance motion control was
developed for the underactuated, nonholonomic robot. For kinematic redundant
manipulators, task-priority control takes advantages of design space in the
null space of the Jacobian matrix. Velocity control was designed through
optimization to satisfy the control tasks from the highest to lowest
priorities [17, 18, 19].

Figure 1: (a) The prototype of the two-wheel steered bikebot mobile
manipulation system. (b) Side-view configuration of the bikebot manipulation
system. (c) Fron-view schematic of the bikebot roll motion.

In this paper, we present a mobile manipulation system that is built on an
autonomous bikebot. A 6-DOF lightweight manipulator is mounted on the bikebot
and the system was developed for agricultural applications such as autonomous
plant inspection and scouting [20]. All existing agricultural robots are built
on double-track platform and their energy consumption is much higher than that
of single-track mobile robots such as bikebot [21]. It is challenging for
double-track robots to navigate in narrow, cluttered spaces and to actively
probe and flexibly inspect objects under the canopy of densely-grown, tall
plants. Light-weight bikebot provides additional advantages for small
footprints that potentially avoids potential severe soil compaction [22].
Steering and speed control of autonomous one-wheel steered bikebot has been
reported (e.g., [23, 24]) but balance control of two-wheel steered bikebot for
manipulation has not been studied. Because of the unstable platform and
limited actuation, assistive devices were used to generate additional balance
torque [25, 26, 27, 28]. However, the additional balance actuators increase
the systems complexity and operation cost. In this work, steering is used as
the only actuation for balancing the stationary platform.

We focus on stationary balance of the bikebot manipulation for several
reasons. First, it is more challenging to balance a stationary bikebot than a
moving platform. With front wheel steering actuation, the bikebot can be only
balanced within a small range of roll motion at stationary (i.e., 2-3 degs)
[29, 30]. It is desirable to design new mechanisms and control methods to
enlarge the controllable roll motion range for practical bikebot applications.
Second, many applications such as plant inspection require that the mobile
platform stays stationary, while the onboard manipulator conducts the visual
inspection or sample manipulation task. Therefore, robotic applications
require the stationary balance capability of single-track mobile robots.

This paper presents the coordinated control of the bikebot and the manipulator
to enhance the stationary balance and posture control task. The use of two-
wheel steering and onboard manipulator enhances the balance capability. We
first present a dynamic model of the system. A steering balance model is
presented to analyze the steering configuration and maximize the balance
capability. The balance condition is captured by an extended balance
equilibrium manifold (BEM) of the mobile manipulation system. A BEM-enabled
coordinated trajectory planning and control design is presented to achieve a
balance-prioritized posture control. We conduct extensive experiments to
validate and demonstrate the performance of the mechatronic and control
design. The main contributions of this work are twofold. First, the presented
two-wheel steering actuation analysis and model are innovative and provide a
guidance on how to use the two-wheel steering design to increase balance
capability of single-track mobile robots. It further explains the steering-
induced balance capability differences between the single-track robot such as
bicycles and other two-wheel, double-track Segway-like balance robots. Second,
the proposed coordinated motion control design integrates the dynamic balance
requirements with the task priority-based planning of a kinematic redundant
manipulator. The extended BEM provides a new control approach to integrate the
dynamic and kinematic constraints for mobile manipulation.

The remainder of this paper is organized as follows. Section II presents the
problem statement and the systems dynamics. In Section III, we analyze the
two-wheel steering mechanism and discuss the balance torque model. Section IV
presents the coordinated pose control of the mobile manipulation. Experimental
results are presented in Section V before we summarize the concluding remarks
in Section VI.

##  II Problem Statement and Systems Dynamics

In this section, we first present the problem statement and then the dynamic
models of the bikebot manipulation system.

###  II-A System Configuration and Problem Statement

Fig. 1 shows the prototype of the bikebot manipulation. Fig. 1 illustrates the
side-view schematic of the kinematic configuration and Fig. 1 for a front-view
of the system. An $n$-link lightweight manipulator with end-effector $E$ is
mounted on the bikebot body frame at point $S$. The two wheel/ground contact
points are denoted as $C_{1}$ and $C_{2}$ and the wheelbase is $l$. Two sets
of coordinate frames are introduced: an inertial frame $\mathcal{I}$ and body
frame $\mathcal{F}_{i}$ for the $i$th manipulator link, $i=1,\cdots,n$. Frame
$\mathcal{F}_{i}$ is constructed by following the DH parameter convention
[31]. $\mathcal{F}_{0}$ and $\mathcal{F}_{n}$ are for the base (platform) and
end-effector frames, respectively. The horizontal and vertical distances from
the bikebot’s mass center $G$ to $C_{1}$ are $l/2$ and $h_{G}$, respectively.
The front and rear steering mechanisms are symmetric with same caster angle
$\varepsilon$.

The bikebot’s steering and roll angles are denoted as $\phi$ and
$\varphi_{b}$, respectively. For both front and rear steering angles, the
positive direction is defined as the counterclockwise about the steering axis.
We define ${\bm{\Theta}}=[\theta_{1}\,\cdots\,\theta_{n}]^{T}$ as the
manipulator joint angles, $i=1,\cdots,n$. The generalized coordinates of the
system are denoted as
${\bm{q}}=[\varphi_{b}\;{\bm{\Theta}}^{T}]^{T}\in\mathcal{Q}\subset\mathbb{R}^{n+1}$,
where $\mathcal{Q}$ is admissible set for ${\bm{q}}$. We denote the pose
(i.e., position and orientation) of end-effector $E$ in $\mathcal{I}$ as
${\bm{\xi}}_{e}\in\mathbb{R}^{6}$.

Problem Statement: Given a set of ${N_{\xi}}$ desired poses
$\\{{\bm{\xi}}_{e}^{k}\\}_{k=1}^{N_{\xi}}$, ${N_{\xi}}\in\mathbb{N}$, the goal
is to design a planning and control method for the bikebot manipulation (i.e.,
steering and joint angles control) to let end-effector $E$ go through and hold
stationary for short time at each ${\bm{\xi}}_{e}^{k}$, $k=1,\cdots,N_{\xi}$.

###  II-B Systems Dynamics

We use DH parameters $(\theta_{i},d_{i},a_{i},\alpha_{i})$ for the $i$th link
of the manipulator, $i=1,\cdots,n$. The homogeneous transformation matrix from
$\mathcal{F}_{i}$ to $\mathcal{F}_{0}$ is written as [32]

$\mathcal{T}_{i}({\bm{q}})=\mathcal{A}^{0}_{1}\mathcal{A}^{1}_{2}\cdots\mathcal{A}^{i-1}_{i},$
(1)

where $\mathcal{A}^{i-1}_{i}$ denotes the transformation from
$\mathcal{F}_{i}$ to $\mathcal{F}_{i-1}$ as

${{\cal A}^{i-1}_{i}}=\begin{bmatrix}{\bm{R}}^{i-1}_{i}&{\bm{p}}_{i}\\\
{\bm{0}}_{1\times 3}&1\end{bmatrix},$ (2)

${\bm{R}}^{i-1}_{i}=\mathbf{R}_{z}(\theta_{i})\mathbf{R}_{x}(\alpha_{i})$,
$\mathbf{R}_{j}(\beta)\in\rm SO(3)$, $j=x,y,z$, denotes the rotational matrix
about $j$-axis with angle $\beta$,
${\bm{p}}_{i}=[a_{i}\cos{\theta_{i}}\;a_{i}\sin{\theta_{i}}\;d_{i}]^{T}$ is
the corresponding position vector in $\mathcal{F}_{i-1}$. With (1), we write
the pose of end-effector $E$ in $\mathcal{F}_{0}$ as
${\bm{\xi}}^{\mathcal{F}_{0}}_{e}={\bm{\xi}}_{e}(\mathcal{T}_{n}({\bm{q}}))$.

We denote the mass center position of the $i$th link in $\mathcal{F}_{0}$ as
${\bm{p}}_{i_{c}}^{0}$ and its position in $\mathcal{I}$ is
${\bm{p}}_{i_{c}}={\bm{R}}_{0}^{\mathcal{I}}({\bm{p}}_{0}+{\bm{p}}^{0}_{i_{c}})$,
where ${\bm{p}}_{0}$ is the position vector of point $S$ in $\mathcal{I}$. The
linear velocity ${\bm{v}}^{0}_{i_{c}}$ and angular velocity
${\bm{\omega}}^{0}_{i_{c}}$ of the $i$th link in $\mathcal{F}_{0}$ are
obtained as

$\dot{{\bm{\xi}}}_{i_{c}}^{\mathcal{F}_{0}}=\left[\left({\bm{v}}^{0}_{i_{c}}\right)^{T}\;\left({\bm{\omega}}^{0}_{i_{c}}\right)^{T}\right]^{T}={\bm{J}}_{i_{c}}\dot{{\bm{\Theta}}},$
(3)

where ${\bm{J}}_{i_{c}}\in\mathbb{R}^{6\times n}$ is the Jacobian from
$\mathcal{F}_{i}$ and $\mathcal{F}_{0}$. Therefore, the linear velocity
${\bm{v}}_{i_{c}}$ and angular velocity ${{\bm{\omega}}}_{i_{c}}$ in
$\mathcal{I}$ are

${\bm{v}}_{i_{c}}={\bm{\omega}}_{b}\times{\bm{R}}_{0}^{\mathcal{I}}({{\bm{p}}_{0}+{\bm{p}}^{0}_{i_{c}}})+{\bm{R}}_{0}^{\mathcal{I}}{\bm{v}}^{0}_{i_{c}},{{\bm{\omega}}}_{i_{c}}={\bm{\omega}}_{b}+{\bm{R}}_{0}^{\mathcal{I}}{\bm{\omega}}_{i_{c}}^{0},$
(4)

where ${\bm{\omega}}_{b}=[\dot{\varphi}_{b}\;0\;0]^{T}$ is the platform’s roll
velocity.

The dynamic model of the mobile manipulation system is obtained through
Lagrange’s equations. The system’s kinetic and potential energies are

$T=T_{b}+\sum\nolimits_{i=1}^{n}T_{i},\;U=U_{b}+\sum\nolimits_{i=1}^{n}{U}_{i},$
(5)

where
$T_{b}=\frac{1}{2}{\bm{\omega}}_{b}^{T}{\bm{I}}_{b}{\bm{\omega}}_{b}+\frac{1}{2}m_{b}{\bm{v}}_{G}^{T}{\bm{v}}_{G}$
is the kinetic energy for the bikebot and for the $i$th link of the
manipulator
$T_{i}=\frac{1}{2}m_{i}{\bm{v}}_{i_{c}}^{T}{\bm{v}}_{i_{c}}+\frac{1}{2}{\bm{\omega}}_{i_{c}}^{T}{\bm{R}}_{i}^{\mathcal{I}}{\bm{I}}_{i}({\bm{R}}_{i}^{\mathcal{I}})^{T}{\bm{\omega}}_{i_{c}}$,
$m_{b}$ and $m_{i}$ are respectively the masses for the bikebot and the $i$th
link, ${{\bm{v}}}_{G}$ is the velocity of the mass center $G$,
${\bm{I}}_{b}=\mathop{\rm diag}\nolimits(I_{b},0,0)$ and ${\bm{I}}_{i}$ are
the inertia matrices for the bikebot and the $i$th link about their mass
center, respectively. For potential energy terms in (5), for the bikebot,
$U_{b}={m_{b}}g\left({{\bm{p}}_{G}\cdot{{\bm{e}}_{z}}+\Delta h_{G}}\right)$
and for the $i$th link,
${U}_{i}={m_{i}}g{{\bm{p}}_{i_{c}}\cdot{{\bm{e}}_{z}}}$, where ${\bm{p}}_{G}$
is the position vector of $G$ in $\mathcal{I}$, $g=9.8$ m/s2 is the
gravitational constant, unit vector ${\bm{e}}_{z}=[0\;0\;1]^{T}$, and $\Delta
h_{G}$ is the height change of $G$ due to steering actuation [33].

The dynamic model is obtained by Lagrangian method as

${\bm{D}}\left({\bm{q}}\right)\ddot{{\bm{q}}}+{\bm{C}}\left({{\bm{q}},\dot{{\bm{q}}}}\right)\dot{{\bm{q}}}+{\bm{G}}\left({\bm{q}}\right)={\bm{\tau}},$

where ${\bm{D}}\left({\bm{q}}\right)\in\mathbb{R}^{(n+1)\times(n+1)}$,
${\bm{C}}\left({{\bm{q}},\dot{{\bm{q}}}}\right)\in\mathbb{R}^{(n+1)\times(n+1)}$,
and ${\bm{G}}\left({\bm{q}}\right)\in\mathbb{R}^{n+1}$ are the inertia,
Coriolis, gravitational matrices, respectively. We omit the details for these
lengthy matrices. The generalized force
${\bm{\tau}}=[\tau_{b}\;{\bm{\tau}}^{T}_{\theta}]^{T}\in\mathbb{R}^{n+1}$
includes the controlled steering-induced balance torque $\tau_{b}$ and joint
torque vector ${\bm{\tau}}_{\theta}\in\mathbb{R}^{n}$ for the manipulator. We
further write the above model into following block matrix form

$\begin{bmatrix}D_{bb}&{\bm{D}}_{b\theta}\\\ {\bm{D}}_{\theta
b}&{\bm{D}}_{\theta\theta}\end{bmatrix}\begin{bmatrix}\ddot{\varphi}_{b}\\\
\ddot{{\bm{\Theta}}}\end{bmatrix}+\begin{bmatrix}{\bm{C}}_{b}\\\
{\bm{C}}_{\theta}\end{bmatrix}\dot{{\bm{q}}}+\begin{bmatrix}G_{b}\\\
{\bm{G}}_{\theta}\end{bmatrix}=\begin{bmatrix}\tau_{b}\\\
{\bm{\tau}}_{\theta}\end{bmatrix},$ (6)

where the block matrices are in appropriate dimensions and their dependencies
on ${\bm{q}}$ and $\dot{{\bm{q}}}$ are dropped for presentation brevity. We
will derive the steering-induced torque model for $\tau_{b}$ in the next
section.

##  III Steering Balance Model

In this section, we analyze the steering mechanism and derive a model to
obtain the configuration that produces maximum steering-induced balance
torque. Fig. 2 illustrates the schematic of the steering effect. We denote the
wheel frame as $\mathcal{F}_{O}$ with wheel center $O$ and the $z_{O}$-axis is
along the steering axis and the $y_{O}$-axis is perpendicular to the wheel
plane. The projection of $O$ on the ground is denoted as $O_{g}$ and the
wheel/ground contact point as $C$. We consider quasi-static steering motion
such that steering angle $\phi$ is built on an initial steering angle
$\phi_{0}$ with a small increment $\delta$, namely, $\phi=\phi_{0}+\delta$.
Increment $\delta$ is small and the position change of point $G$ under
$\delta$ is negligible.

Figure 2: Illustration of the steering mechanism and analysis. (a) Wheel plane
geometry under steering angle increment $\delta$. Wheel contact point changes
from $C$ to $C^{\prime}$. (b) Wheel contact points $C_{1}C_{2}$ move to
$C^{\prime}_{1}C^{\prime}_{2}$ under a small steering angle increment $\delta$
and the geometric relationships between $G_{g}$ and wheelbase line
$C_{1}^{\prime}C_{2}^{\prime}$.

The orientation of the wheel plane with respect to frame $\mathcal{I}$ is
approximately obtained by three successive rotations: first $-\phi$ about
$z_{O}$-axis, then $-\varphi_{b}$ about $x_{O}$, and finally $-\varepsilon$
about $y_{O}$-axis. With this observation, we obtain the rotational
transformation from $\mathcal{F}_{O}$ to $\mathcal{I}$ as

${\bm{R}}_{\mathcal{F}_{O}}^{\mathcal{I}}=\mathbf{R}_{y}(-\varepsilon)\mathbf{R}_{x}(-\varphi_{b})\mathbf{R}_{z}(-\phi).$
(7)

We denote the angle between the wheel plane and the ground as $\gamma$ and it
is straightforward to obtain

$\cos\gamma={\bm{R}}_{\mathcal{F}_{O}}^{\mathcal{I}}{\bm{e}}_{y}\cdot{{\bm{e}}_{z}}=\sin\phi\sin\varepsilon-\cos\phi\cos\varepsilon\sin{\varphi_{b}},$
(8)

where unit vector ${\bm{e}}_{y}=[0\;1\;0]^{T}$.

To simplify the two-wheel steering design, both the front and rear steering
angles, denoted respectively by $\phi^{f}$ and $\phi^{r}$, are controlled and
kept at symmetric position (i.e., same amplitude but opposite directions) for
all time, namely, $\phi^{f}=-\phi^{r}=\phi$, with $\phi^{f}=\phi_{f0}+\delta$,
$\phi^{r}=\phi_{r0}-\delta$ and $\phi_{f0}=-\phi_{r0}$. For brevity, we only
use $\phi$ and $\delta$ in the following derivation. Under small $\delta$,
wheel/ground contact points $C_{1}$ and $C_{2}$ move to $C^{\prime}_{1}$ and
$C^{\prime}_{2}$, respectively. Fig. 2 illustrates the wheel/ground contact
points under $\delta$. With the above configuration, points $C_{1}$ and
$C^{\prime}_{1}$ ($C_{2}$ and $C^{\prime}_{2}$) are located on a circular arc
that is centered around $O^{f}_{g}$ ($O^{r}_{g}$), projected points $O^{f}$
($O^{r}$) on the ground. The radii of the circular arc
$\wideparen{C_{1}C^{\prime}_{1}}$ ($\wideparen{C_{2}C^{\prime}_{2}}$) and the
bikebot wheel are denoted as $r$ and $R$, respectively. From the geometric
relationship, we obtain

$r=R\cos\gamma=R\sin\phi\sin\varepsilon-R\cos\phi\cos\varepsilon\sin{\varphi_{b}}.$
(9)

As shown in Fig. 2, under the same front and rear steering angles, wheelbase
lines $C_{1}C_{2}$ and $C_{1}^{\prime}C_{2}^{\prime}$ are parallel. The
corresponding projected steering angles are $\phi_{g}=\angle AO_{g}^{r}x_{og}$
and $\phi^{\prime}_{g}=\angle AO_{g}^{r}x^{\prime}_{og}$ (for the rear wheel).
Let $L$ denote the distance from the $C_{1}C_{2}$ to $O_{g}^{f}O_{g}^{r}$ and
$L$ is obtained by the geometry relationship as

$L=r\cos\phi_{g}.$ (10)

The relationship between $\phi_{g}$ and $\phi$ is captured by [33]

$\phi_{g}=\arctan\left(\frac{\cos\varepsilon}{\cos{\varphi_{b}}}\tan\phi\right).$
(11)

Given a fixed $\phi_{0}$, from (9), $r=r_{\phi_{0}}$ is considered a constant
value for a small roll angle (e.g., $\varphi_{b}\approx 0$) and therefore,
plugging (9) and (11) into (10), we obtain

$L=r_{\phi_{0}}\frac{\cos\varphi_{b}}{\sqrt{\cos^{2}\varphi_{b}+\cos^{2}\varepsilon\tan^{2}\phi}}=L\left(\varphi_{b},\phi_{0},\delta\right),$

where $L$ is considered as a function of $\varphi_{b}$, $\phi_{0}$ and
$\delta$.

We approximate the gravity-induced balance torque by using the above
calculated $L$ as the distance between $G_{g}$ (the projected point of $G$ on
the ground) to $C^{\prime}_{1}C^{\prime}_{2}$ and obtain

$\tau_{b}\approx m_{b}gL=mgL\left(\varphi_{b},\phi_{0},\delta\right).$ (12)

It is helpful to find steering angle $\phi_{0}$ at which the increment
$\delta$ generates largest torque increase of $\tau_{b}$. Thus, we introduce
the steering torque sensitivity as the steering torque rate with respect to
$\delta$ at $\varphi_{b}=0$, namely,

$S_{\tau}(\phi_{0})=\left|\frac{\partial\tau_{b}}{\partial\delta}\right|_{\begin{subarray}{c}\delta=0\\\
\varphi_{b}=0\end{subarray}}=m_{b}gr_{\phi_{0}}\frac{\cos^{2}\varepsilon\tan\phi_{0}(\tan^{2}\phi_{0}+1)}{\left(\cos^{2}\varepsilon\tan^{2}\phi_{0}+1\right)^{3/2}}.$

From above equation, it is straightforward to see that at $\phi_{0}=0$,
$S_{\tau}(\phi_{0})=0$ and this implies that the commonly used zero steering
angle has the minimum steering torque sensitivity.

We further calculate that at $\phi^{*}_{0}=\frac{\pi}{2}$ rad,
$S_{\tau}(\phi_{0})$ reaches its maximum value as

$S_{\tau}(\phi_{0}^{*})=m_{b}gr_{\phi_{0}^{*}}\frac{1}{\sqrt{\cos^{2}\varepsilon}}=m_{b}gR\tan\varepsilon.$

Therefore, we focus on using $\phi_{0}^{*}=\frac{\pi}{2}$ rad for mobile
manipulation control since it generates the largest balance torque per unit of
steering angle. In this case, $\phi=\frac{\pi}{2}+\delta$ and
$\phi_{g}=\frac{\pi}{2}+\delta_{g}$, the steering torque is then calculated as

$\text{}\tau_{b90}=m_{b}gr_{90}\cos\phi_{g}=-m_{b}gR\sin\varepsilon\cos\delta\sin\left(\tfrac{\delta}{\cos\varepsilon}\right),$
(13)

where $r_{90}=R\sin\varepsilon\cos\delta$ is from (9) with $\varphi_{b}\approx
0$ and $\delta_{g}\cos\varepsilon\approx\delta$ is taken from (11). It is
clear that a large caster angle configuration helps increase the steering-
induced balance torque and therefore improve balance capability. For any other
initial steering angle $\phi_{0}$, the radius is calculated by (9) and the
steering torque $\tau_{b}$ is obtained by (12).

It is interesting to note that under $\phi_{0}=\frac{\pi}{2}$, the steering
configuration is different with commonly used zero initial steering angle
$\phi_{0}=0$. Indeed, the configuration is similar to double-track balance
robot such as Segway. This observation explains and implies that double-track
steering configuration such as Segway-like robots helps provide more steering-
induced balance torques than the single-track configuration such as bicycles.
We therefore use $\phi_{0}=\frac{\pi}{2}$ in the bikebot balance
implementation.

##  IV Coordinated Balance Control Design

In this section, we focus on the coordinated balance and pose control of the
bikebot manipulation. The controller design is built on the balancing
equilibrium manifold (BEM).

###  IV-A Balance Equilibrium Manifold

The kinematics redundancy of the multi-DOFs manipulator enables the end-
effector to reach the target poses with the balanced bikebot platform. If we
consider the manipulator moves quasi-statically (i.e., slowly), the balanced
bikebot roll angle $\varphi_{b}$ and manipulator joint angles ${\bm{\Theta}}$
should satisfy an intrinsic relationship that is captured by BEM. From (6),
the equation of motion of the bikebot is written as

$D_{bb}\ddot{\varphi}_{b}+{\bm{D}}_{b\theta}\ddot{{\bm{\Theta}}}+{\bm{C}}_{b}\dot{{\bm{q}}}+G_{b}({\bm{q}})=\tau_{b}$
(14)

where $G_{b}({\bm{q}})$ is the total gravitational torque from the bikebot and
the manipulator. Considering the quasi-static motion, namely,
$\ddot{{\bm{q}}}=\dot{{\bm{q}}}=0$, we define the BEM as

$\mathcal{E}=\\{{\bm{q}}_{e}=[\varphi^{e}_{b}\;{\bm{\Theta}}^{T}_{e}]^{T}:G_{b}({\bm{q}})=\tau_{b},\,{\bm{q}}\in\mathcal{Q}\\}.$
(15)

The BEM captures all configurations that satisfy the static equilibrium
constraint. Using BEM, we estimate the static maximum roll angle
$\varphi^{\max}_{b}$ under the maximum balance steering $\tau^{\max}_{b}$ with
possible ${\bm{\Theta}}$.

To move the end-effector from one pose to another, a trajectory should be
designed around the BEM at any time, namely, ${\bm{q}}\in\mathcal{E}$. A
velocity constraint should be enforced given the BEM and limited steering
actuation. Using (13), the steering torque is
$\tau_{b}=-MgR\sin\varepsilon\cos\delta\sin\delta_{g}$, where
$M=m_{b}+\sum_{i=1}^{n}m_{i}$ is the total mass of the entire system. Taking
derivative of BEM condition $G_{b}({\bm{q}})=\tau_{b}$, we obtain

$\dot{G}_{b}=\frac{{\partial{G_{b}}}}{{\partial{\bm{q}}}}\dot{{\bm{q}}}=-MgR\sin\varepsilon\frac{d}{d\delta}(\cos\delta\sin\delta_{g})\dot{\delta}=h(\delta)\dot{\delta},$

where
$h(\delta)=-MgR\sin\varepsilon\frac{d}{d\delta}(\cos\delta\sin\delta_{g})$.
Defining ${\bm{J}}_{G}=\frac{{\partial{G_{b}}}}{{\partial{\bm{q}}}}$ as a
Jacobian-like matrix, the above velocity constraint is specified as

$|{\bm{J}}_{G}\dot{{\bm{q}}}|\leq h_{\max}\dot{\delta}_{\max},$ (16)

where $h_{\max}=\sup_{\delta}|h(\delta)|$ and $\dot{\delta}_{\max}$ is the
maximum steering angular rate. Constraint (16) implies that when designing the
trajectory ${\bm{q}}(t)$, the allowed motion velocity is restricted by the
steering angular rate.

###  IV-B Balance-Prioritized Pose Trajectory Planning

The end-effector pose workspace in $\mathcal{I}$ is defined as

$\mathcal{X}({\bm{q}})=\\{{\bm{\xi}}_{e}:\,{\bm{\xi}}_{e}={\bm{\xi}}_{e}(\mathcal{T}_{n+1}({\bm{q}})),{\bm{q}}\in\mathcal{E},|\tau_{b}|\leq{\tau}_{b}^{\max}\\},$
(17)

where $\mathcal{T}_{n+1}({\bm{q}})$ is the homogeneous transformation from
$\mathcal{F}_{n}$ to $\mathcal{I}$. We further define the local end-effector
pose workspace
$\mathcal{X}_{\varphi_{b}^{0}}({\bm{\Theta}})\subseteq\mathcal{X}({\bm{q}})$
under roll angle $\varphi_{b}^{0}$,
${\bm{q}}^{0}=[\varphi_{b}^{0}\,{\bm{\Theta}}^{T}]^{T}$,

$\mathcal{X}_{\varphi_{b}^{0}}({\bm{\Theta}})=\left\\{{\bm{\xi}}_{e}:{\bm{\xi}}_{e}={\bm{\xi}}_{e}(\mathcal{T}_{n+1}({\bm{q}}^{0})),{\bm{q}}^{0}\in\mathcal{E}\right\\}.$

The rationale to introduce $\mathcal{X}_{\varphi_{b}^{0}}({\bm{\Theta}})$ is
to specify the bikebot roll angle $\varphi_{b}^{0}$ first for balance task and
then use the manipulator to achieve the pose control task. We consider the
task priority from high to low as follows: (1) bikebot platform balance; (2)
pose control of end-effector $E$; and (3) collision avoidance during arm
movement from one desired pose to another. Due to the redundant kinematics of
the manipulator, we use a task-priority optimization approach to plan the
trajectory.

We define the following balance-prioritized inverse kinematics (BPIK) problem.
Given a sequence of desired end-effector poses ${\bm{\xi}}_{e}^{k}$,
$k=1,\cdots,{N_{\xi}}$, the BPIK is to find optimal bikebot roll and
manipulator joint angles ${\bm{q}}_{k}^{*}$ as

$\displaystyle{\bm{q}}^{*}_{k}=$
$\displaystyle\arg\mathop{\min}\limits_{{\bm{q}}}{\lambda_{1}}\Gamma_{1}+\lambda_{2}\Gamma_{2}+{\lambda_{3}}\Gamma_{3}$
(18a) Subj. to
$\displaystyle\;\,\lambda_{4}|G_{b}|\leq{\tau}_{b}^{\max},\;{\bm{q}}\in\mathcal{E},$
(18b)

where
$\Gamma_{1}=\|{\bm{\xi}}_{e}^{k}-{\bm{\xi}}_{e}(\mathcal{T}_{n+1}({\bm{q}}))\|_{2}^{2}$,
$\Gamma_{2}=|G_{b}({\bm{q}}_{k})-G_{b}({\bm{q}}_{k-1}^{*})|^{2}$,
$\Gamma_{3}={\bm{e}}_{k-1}^{T}{\bm{P}}{\bm{e}}_{k-1}$,
${\bm{e}}_{k-1}={\bm{q}}-{\bm{q}}_{k-1}^{*}$,
$\lambda_{1},\lambda_{2},\lambda_{3}>0$, and $\lambda_{4}>1$ are weight
parameters, ${\bm{P}}>0$ is a symmetric positive definite matrix. We
initialize with ${\bm{q}}_{-1}^{*}={\bm{0}}$ and $G_{b}({\bm{q}}_{-1}^{*})=0$.

Inequality (18b) is set as a hard constraint such that the steering output is
always within the balance capability, while the pose regulation becomes a part
of the objective function (i.e., $\Gamma_{1}$). Therefore, balance serves as a
higher priority than pose regulation by the BPIK design. This is similar to
the approach in [3] by projecting the low level priority task into subspace of
solution of high level tasks. Terms $\Gamma_{2}$ and $\Gamma_{3}$ in (18a) try
to minimize the difference between configurations at the current and the
previous steps. We use the BPIK to obtain ${\bm{q}}^{*}_{k}$ from
${\bm{\xi}}_{e}^{k}$. We first search the solution in local workspace
$\mathcal{X}_{\varphi_{b}^{0}}({\bm{\Theta}})$ to avoid large bikebot
movement. If this is impossible, the BPIK then searches the solution in the
workspace $\mathcal{X}({\bm{q}})$. If the calculated feasible poses are
outside of $\mathcal{X}({\bm{q}})$, (18) will return the closest results. Once
obtaining the desired joint angles $\\{{\bm{q}}_{k}^{*}\\}_{k=1}^{N_{\xi}}$,
we need to design transition trajectory along $\mathcal{E}$ between each two
consecutive poses.

We consider a pair of desired consecutive configurations
$({\bm{q}}^{*}_{k-1},{\bm{q}}^{*}_{k})$ to position end-effector $E$. With
user-specified starting and ending times denoted respectively as $t_{0}$ and
$t_{f}$, we define ${\bm{q}}(t_{0})={\bm{q}}^{*}_{k-1}$ and
${\bm{q}}(t_{f})={\bm{q}}^{*}_{k}$. A motion trajectory ${\bm{q}}^{*}(t)$
needs to be designed from ${\bm{q}}(t_{0})$ and ${\bm{q}}(t_{f})$ along
$\mathcal{E}$. The trajectory planning is formulated as the following
optimization problem.

$\displaystyle\text{}\min_{{\bm{q}}(t)}\int_{{t_{0}}}^{{t_{f}}}$
$\displaystyle{\bm{e}}_{k-1}^{T}{\bm{W}}_{1}{\bm{e}}_{k-1}+{\dot{{\bm{q}}}^{T}}{\bm{W}}_{2}\dot{{\bm{q}}}+(\delta
G_{b,k})^{2}dt$ (19a) Subj. to
$\displaystyle\;\,\dot{{\bm{q}}}(t_{0})=\dot{{\bm{q}}}^{*}_{k-1},\dot{{\bm{q}}}(t_{f})=\dot{{\bm{q}}}^{*}_{k},$
$\displaystyle\ddot{{\bm{q}}}(t_{0})=\ddot{{\bm{q}}}^{*}_{k-1},\ddot{{\bm{q}}}(t_{f})=\ddot{{\bm{q}}}^{*}_{k},$
(19b) $\displaystyle{\bm{D}}_{\theta
b}\ddot{\varphi}_{b}+{\bm{D}}_{\theta\theta}\ddot{{\bm{\Theta}}}+{\bm{C}}_{\theta}\dot{{\bm{q}}}+{\bm{G}}_{\theta}={\bm{\tau}}_{\theta},$
(19c) $\displaystyle|{\bm{J}}_{G}\dot{{\bm{q}}}|\leq
h_{\max}\dot{\delta}_{\max},\lambda_{4}|G_{b}|\leq{\tau}_{b}^{\max},$ (19d)
$\displaystyle|\tau_{\theta,i}|\leq{\tau}_{\theta,i}^{\max},{\bm{q}}\in\mathcal{Q},|\dot{{\bm{q}}}|\leq\dot{q}_{\max},|\ddot{{\bm{q}}}|\leq\ddot{q}_{\max},$
(19e)

where $\delta G_{b,k}=G_{b}({\bm{q}})-G_{b}({\bm{q}}^{*}_{k-1})$,
${\bm{W}}_{1},{\bm{W}}_{2}\in\mathbb{R}^{n+1}$ are positive diagonal matrices,
and $\tau_{\theta,i}^{\max}$ is the maximum joint torque of the $i$th link,
$i=1,\cdots,n$. To consider the quasi-static motion, the angular velocity and
acceleration of the manipulator are bounded as in (19e). The constraint in
(19d) is similar to that in (18) along with joint torque limits.

To solve (19), we try to avoid integration of the differential constraint
(19c) and Bézier polynomials are used to specify the solution in each
dimension of ${\bm{q}}(t)$. We use Bézier polynomial because of its attractive
properties [34]. The solution ${\bm{q}}(t)$ is written in term of $N$th order
Bézier polynomials ($N\in\mathbb{N}$) as

$\varphi_{b}=b(s,{\bm{p}}_{b})=\sum\limits_{j=0}^{N}p_{b_{j}}b_{j}(s),\,\theta_{i}=b(s,{\bm{p}}_{\theta_{i}})=\sum\limits_{j=0}^{N}p_{\theta_{ij}}b_{j}(s)$

for $i=1,\cdots,n$, where $b_{j}(s)=\frac{N!}{(N-j)!j!}(1-s)^{N-j}s^{j}$,
parameters ${\bm{p}}_{b}=[p_{b_{0}}\,\cdots\,p_{b_{N}}]^{T}$ and
${\bm{p}}_{\theta_{i}}=[p_{\theta_{i0}}\,\cdots\,p_{\theta_{iN}}]^{T}$. The
normalized progress variable $s=\frac{t-t_{0}}{t_{f}-t_{0}}$ maps
$t\in[t_{0},t_{f}]$ to $s\in[0,1]$.

From above formulation, we obtain
${\bm{q}}(t_{0})=[b(0,{\bm{p}}_{b})\,b(0,{\bm{p}}_{\theta_{0}})\,\cdots\,b(0,{\bm{p}}_{\theta_{n}})]^{T}$,
${\bm{q}}(t_{f})=[b(1,{\bm{p}}_{b})\,b(1,{\bm{p}}_{\theta_{0}})\,\cdots\,b(1,{\bm{p}}_{\theta_{n}})]^{T}$
and ${\bm{q}}(t)$ is then written as polynomials of $s$ with parameters
${\bm{p}}=\\{{\bm{p}}_{b},{\bm{p}}_{\theta_{1}},\cdots,{\bm{p}}_{\theta_{n}}\\}$.
For $\dot{{\bm{q}}}(t)$, we obtain
$\dot{\varphi}_{b}=\sum\limits_{j=0}^{N}p_{b_{j}}b^{\prime}_{j}(s)\frac{ds}{dt}$
and
$\dot{\theta}_{i}=\sum\limits_{j=0}^{N}p_{\theta_{ij}}b^{\prime}_{j}(s)\frac{ds}{dt}$,
$i=1,\cdots,n$. Noting that $\frac{ds}{dt}=\frac{1}{t_{f}-t_{0}}$ is constant,
$\dot{{\bm{q}}}(t)$ and $\ddot{{\bm{q}}}(t)$ are written in term of
${\bm{p}}$. Therefore, the trajectory planning problem (19) is transformed
into the $s$-domain and the differential constraints are written as algebraic
formulation in polynomials of $s$ and ${\bm{p}}$.

To solve the optimization problem, we discretize $s\in[0,1]$ with $N_{s}$
sampling points and both the objective and constraint functions in (19) are
evaluated at these points. A sequential quadratic programming (SQP) algorithm
is then used to obtain the optimized trajectory ${\bm{q}}^{*}(t)$ [35]. In
each iteration, a total of $3N_{s}(n+1)$ evaluations of ${\bm{b}}(s;{\bm{p}})$
(for ${\bm{q}},\dot{{\bm{q}}},\ddot{{\bm{q}}}$) and $N_{s}(n+2)$ evaluations
of (19c) and (19d) are needed for $(n+1)(N-1)$ optimization variables.
Additionally, the SQP solver has complexity $O(N^{2}n^{2})$. Therefore, the
computational complexity for solving (19) by the proposed Bézier polynomial-
based approach is $O((N+N_{s})Nn^{2})$. As a comparison, a dynamic programming
(DP) method can be used to solve (19) with complexity $O(N^{2}_{s}n^{2})$.
Because of $N_{s}\gg N$, the proposed method is much faster than the DP
method.

Algorithm 1 summarizes the trajectory planning as described above.

Specify $\\{{\bm{\xi}}_{e}^{k}\\}_{k=1}^{N_{\xi}}$, $\lambda_{i},i=1,...,4$,
${\bm{P}}$, ${\bm{W}}_{1}$, ${\bm{W}}_{2}$, and $\epsilon>0$;

for  _$k\leq{N_{\xi}}$_ do

if  _$k=1$_ then

${\bm{q}}^{*}_{k-1}\leftarrow{\bm{0}}$, $G({\bm{q}}^{*}_{k-1})\leftarrow 0$;

Solve ${\bm{q}}^{*}_{k}\in\mathcal{X}({\bm{q}})$ by (18) with
${\bm{\xi}}_{e}^{k}$;

else

Solve ${\bm{q}}^{*}_{k}\in\mathcal{X}_{\varphi_{b,k-1}^{*}}({\bm{\Theta}})$ by
(18) with ${\bm{\xi}}_{e}^{k}$;

Calculate the pose ${\bm{\xi}}_{e}({\bm{q}}^{*}_{k})$;

if
_$\|{\bm{\xi}}_{e}^{k}-{\bm{\xi}}_{e}(\mathcal{T}_{n+1}({\bm{q}}^{*}_{k}))\|\geq\epsilon$_
then

Re-calculate ${\bm{q}}^{*}_{k}\in\mathcal{X}({\bm{q}})$ by (18);

$k\leftarrow k+1$

for  _$j\leq{N_{\xi}}$_ do

Specify $t_{0}$ and $t_{f}$;
${\bm{q}}\left({{t_{0}}}\right)\leftarrow{\bm{q}}^{*}_{j-1}$,
${\bm{q}}\left({{t_{f}}}\right)\leftarrow{\bm{q}}^{*}_{j}$;

Plan ${\bm{q}}^{*}(t)$, $t\in[t_{0},t_{f}]$ by (19) using Bézier polynomial
specification;

$j\leftarrow j+1$ ;

Algorithm 1 Trajectory planning for pose regulation

###  IV-C Bikebot Steering and Manipulator Control

Fig. 3 illustrates the balance-prioritized trajectory planning and control
design. The previous section discusses the trajectory planner to obtain
${\bm{q}}^{*}(t)$ for a given set of end-effector poses
$\\{{\bm{\xi}}_{e}^{k}\\}_{k=1}^{N_{\xi}}$. In this subsection, we present the
online control algorithms to follow the trajectory ${\bm{q}}^{*}(t)$.

Figure 3: The block diagram of the balance-prioritized trajectory planning and
control scheme.

We first present the steering control of the bikebot to follow
$\varphi_{b}^{*}$ and then discuss the manipulator controller to follow
${\bm{\Theta}}^{*}$. Using the dynamic model of the bikebot in (14), we define
the roll angle error $e_{b}=\varphi_{b}-\varphi_{b}^{*}$ and a feedback
linearization control is designed as

$\tau_{b}=D_{bb}\ddot{\varphi}^{*}_{b}+{\bm{D}}_{b\theta}\ddot{{\bm{\Theta}}}+{\bm{C}}_{b}\dot{{\bm{q}}}+G_{b}+k_{p}e_{b}+k_{d}\dot{e}_{b},$
(20)

where $k_{p},k_{d}>0$ are the feedback gains. From (20), we use the steering
balance model (13) to obtain the steering angle $\phi$.

For the manipulator, since the trajectory are designed in a quasi-static form,
we take the velocity control to follow the desired trajectory
${\bm{\Theta}}^{*}$ in $\mathcal{F}_{0}$. We recognize
$\ddot{{\bm{\Theta}}}\approx 0$ after the system compensating the gravity
($G_{\theta}$). We extend the velocity control in [10, 36] with additional
velocity correction under bikebot roll motion errors. Defining joint angle
error ${\bm{e}}_{\Theta}={\bm{\Theta}}-{\bm{\Theta}}^{*}\in\mathbb{R}^{n}$,
the velocity control in the joint workspace is given by

$\dot{{\bm{e}}}_{\Theta}=-{\bm{K}}_{p}{\bm{e}}_{\Theta}+I_{\Theta}\dot{{\bm{\delta}}}_{\Theta},$
(21)

where ${\bm{K}}_{p}=\mathop{\rm diag}\nolimits\\{K_{p1},\cdots,K_{pn}\\}$ with
$K_{pi}>0$, $i=1,\cdots,n$ and $I_{\Theta}=1$ if $|e_{b}|>\varepsilon_{b}$;
otherwise $I_{\Theta}=0$, with an error threshold $\varepsilon_{b}>0$. The
velocity correction $\dot{\delta}_{{\bm{\Theta}}}$ in (21) is designed as

$\dot{{\bm{\delta}}}_{\Theta}=-\kappa\tfrac{\partial\delta
G_{b}}{\partial{\bm{\Theta}}}=2\kappa\left[G_{b}({\bm{q}}^{*})-G_{b}({\bm{q}})\right]{\left(\tfrac{\partial{G_{b}}({\bm{q}})}{\partial{\bm{\Theta}}}\right)^{T}},$

where $\delta G_{b}=(G_{b}({\bm{q}}^{*})-G_{b}({\bm{q}}))^{2}$ denotes
deviation of actual motion from the BEM and $\kappa>0$ is a scalar.

Under (20), the closed-loop bikebot roll dynamics is

$\ddot{e}_{b}+k_{d}\dot{e}_{b}+k_{p}e_{b}=0$ (22)

and roll error $e_{b}(t)$ converges to zero exponentially. Without loss of
generality, let $k_{d}^{2}<4k_{p}$ and then from (22) we obtain

$|e_{b}(t)|\leq M_{b}e^{-\frac{k_{d}}{2}t},$ (23)

where $M_{b}>0$ is a finite constant that is related to $e_{b}(0)$ and
$\dot{e}_{b}(0)$. Therefore, for any $t\geq
t_{b}:=\frac{2}{k_{d}}\ln\left(\frac{M_{b}}{\varepsilon_{b}}\right)$,
$|e_{b}(t)|\leq\varepsilon_{b}$. To show the tracking error convergence for
${\bm{e}}_{\Theta}$, we consider a Lyapunov function candidate
$V(t)={\bm{e}}^{T}_{\Theta}{\bm{e}}_{\Theta}=\|{\bm{e}}_{\Theta}\|^{2}>0$ for
any non-zero error ${\bm{e}}_{\Theta}$. Let $l_{\Theta}$ denote the upper
bound for input $I_{\Theta}\dot{{\bm{\delta}}}_{\Theta}$ during
$t\in[0,t_{b}]$, namely, $l_{\Theta}:=\sup_{0\leq t\leq
t_{b}}\|I_{\Theta}\dot{{\bm{\delta}}}_{\Theta}\|$. Moreover, we have

$\dot{V}(t)=-2{\bm{e}}^{T}_{\Theta}{\bm{K}}_{p}{\bm{e}}_{\Theta}+2I_{\Theta}{\bm{e}}^{T}_{\Theta}\dot{{\bm{\delta}}}_{\Theta}\leq-2\lambda_{p}\|{\bm{e}}_{\Theta}\|^{2}+2l_{\Theta}\|{\bm{e}}_{\Theta}\|,$

where $\lambda_{p}=\min_{1\leq i\leq n}K_{pi}$. We introduce
$W(t)=\sqrt{V(t)}=\|{\bm{e}}_{\Theta}(t)\|$ and from above inequality, we
obtain $\dot{W}\leq-{\lambda_{p}}W+{l_{\Theta}}$. Thus, we have
$\frac{d}{dt}(We^{{\lambda_{p}}t})=\dot{W}e^{{\lambda_{p}}t}+W{\lambda_{p}}e^{{\lambda_{p}}t}\leq{l_{\Theta}}e^{{\lambda_{p}}t}$
and integrating from $0$ to $t$, we obtain

$W(t)e^{{\lambda_{p}}t}-W(0)\leq\tfrac{l_{\Theta}}{\lambda_{p}}(e^{{\lambda_{p}}t}-1).$

Noting $W(t)=\|{\bm{e}}_{\Theta}(t)\|$, the above inequality becomes

$\displaystyle\|{\bm{e}}_{\Theta}(t)\|$ $\displaystyle\leq$
$\displaystyle\|{\bm{e}}_{\Theta}(0)\|e^{-{\lambda_{p}}t}+\tfrac{l_{\Theta}}{\lambda_{p}}(1-e^{-{\lambda_{p}}t})$
(24) $\displaystyle\leq$
$\displaystyle\|{\bm{e}}_{\Theta}(0)\|e^{-{\lambda_{p}}t}+\tfrac{l_{\Theta}}{\lambda_{p}}$

From the above analysis, $\dot{V}$ is negative outside of the compact set
$\mathcal{S}=\\{{\bm{e}}_{\Theta}:\|{\bm{e}}_{\Theta}(t)\|\leq\frac{l_{\Theta}}{\lambda_{p}}\\}$
and ${\bm{e}}_{\Theta}(t)$ exponentially converges to $\mathcal{S}$. Note that
for $t\geq t_{b}$, $I_{\Theta}=0$, ${\bm{e}}_{\Theta}(t)$ converges to zeros
exponentially due to stable dynamics (21).

With error convergence in (23) and (24), we obtain the error bound for
${\bm{e}}_{q}={\bm{q}}-{\bm{q}}^{*}=[e_{b}\;{\bm{e}}^{T}_{\Theta}]^{T}$ as

$\displaystyle\|{\bm{e}}_{q}(t)\|$ $\displaystyle=$
$\displaystyle\sqrt{e^{2}_{b}(t)+\|{\bm{e}}_{\Theta}(t)\|^{2}}\leq|e_{b}(t)|+\|{\bm{e}}_{\Theta}(t)\|$
(25) $\displaystyle\leq$ $\displaystyle
M_{b}e^{-\frac{k_{d}}{2}t}+\|{\bm{e}}_{\Theta}(0)\|e^{-{\lambda_{p}}t}+\tfrac{l_{\Theta}}{\lambda_{p}}.$

Considering pose error
${\bm{e}}_{\xi}={\bm{\xi}}_{e}({\bm{q}})-{\bm{\xi}}_{e}({\bm{q}}^{*})$ and
${\bm{q}}={\bm{q}}^{*}+{\bm{e}}_{q}$, we have

${\bm{e}}_{\xi}={\bm{\xi}}_{e}({\bm{q}}^{*})+\frac{\partial{\bm{\xi}}}{\partial{\bm{q}}}\Big{|}_{{\bm{q}}^{*}}{\bm{e}}_{q}+{\bm{\Delta}}_{q}-{\bm{\xi}}_{e}({\bm{q}}^{*})={\bm{J}}_{e}({\bm{q}}^{*}){\bm{e}}_{q}+{\bm{\Delta}}_{q},$
(26)

where ${\bm{J}}_{e}({\bm{q}})\in\mathbb{R}^{6\times(n+1)}$ is the Jacobian
matrix from $\mathcal{F}_{n}$ (end-effector $E$) to inertial frame
$\mathcal{I}$ and ${\bm{\Delta}}_{q}\in\mathbb{R}^{6}$ is the higher order
term of error ${\bm{e}}_{q}$. From (26), it is straightforward to obtain that
$\|{\bm{e}}_{\xi}(t)\|\leq\|{\bm{J}}_{e}({\bm{q}}^{*})\|\|{\bm{e}}_{q}(t)\|+\|{\bm{\Delta}}_{q}\|$.
For the higher order term ${\bm{\Delta}}_{q}=O(\|{\bm{e}}_{q}\|^{2})$, there
exists a finite constant $M_{\delta}>0$ such that $\|{\bm{\Delta}}_{q}\|\leq
M_{\delta}\|{\bm{e}}_{q}(t)\|$ due to (25) and then $\|{\bm{e}}_{\xi}(t)\|\leq
M_{q}\|{\bm{e}}_{q}(t)\|$, where
$M_{p}=\sup_{{\bm{q}}^{*}}\|{\bm{J}}_{e}({\bm{q}}^{*})\|+M_{\delta}$.
Therefore, the pose error ${\bm{e}}_{\xi}(t)$ converges to a small ball near
zero exponentially and the robotic system is stable.

##  V Experiments

###  V-A Experiment Setup

Figure 6: Experimental results for the steering torque model. (a) Comparison
results of the model prediction with the experiments for radius
$r_{\phi_{0}}$. (b) Comparison of the steering sensitivity model prediction
with the experiments. (c) Comparison of the balance torque model prediction
with the experiments under steering angle increment $\delta$ and roll angle
$\varphi_{b}$ with $\phi_{0}=90$ degs.

Fig. 1 shows the prototype of the two-wheel steered bikebot with an onboard
6-DOF robotic manipulator (Jaco2 from Kinova Inc.). Fig. 4 illustrates the
interconnection schematic of the embedded systems and actuators and sensors.
Both the front and the rear wheels can be steered around $360$ degs by two
stepper motors. A real-time low-level embedded system (Teensy 4.0
microcontroller) is used for the steering motor control, while the robotic
manipulator is controlled by a powerful small-size computer (Intel NUC module)
with robot operating system (ROS). One inertial measurement unit (IMU) (model
800 from Motion Sense Inc.) is mounted at the upper frame of the bikebot to
measure the roll angle. The front and rear steering angles are measured by two
encoders and the manipulator joint angles are obtained by the embedded
encoders. The real-time bikebot steering control and data acquisition
frequency was implemented at 100 Hz and the low-level manipulator velocity
control was run at 1000 Hz.

Figure 4: The schematic of the interconnection among sensors, actuators and
embedded systems for the bikebot-manipulator system.

Table I lists the model parameter values for the bikebot. The values of the DH
parameters and mass moments of inertia of the manipulator links are listed in
Table II. The other physical parameters for each link can be found in [37]. To
validate the steering mechanism and models, we also built and conducted
experiments to measure the tire/ground contacts and movement. Fig. 5 shows the
experimental setup for the steering models validation. A motion capture system
(4 Vantage cameras from Vicon Ltd.) was used to measure the angle and contact
points between the wheel surface and the ground at different initial steering
angles $\phi_{0}$.

Figure 5: Experimental setup for validation of the steering mechanism and
balance torque model.

Figure 7: Bikebot balance control experimental results. (a) Bikebot roll angle $\varphi_{b}$. (b) Bikebot roll angle error $e_{b}$. Steering angle increments. The red markers “$\bullet$” in (a) and (b) indicate that the initial angles. TABLE I: Values for the model parameters of the bikebot platform $m_{b}$ (kg) |  $J_{b}$ (kgm2) |  $h_{G}$ (m) |  $l$ (m) |  $\varepsilon$ (deg) |  $R$ (m)
---|---|---|---|---|---
$46.9$ | $3.2$ | $0.53$ | $1.2$ | $20$ | $0.3$
TABLE II: DH parameter values and inertia parameters of the 6-DOF robotic manipulator Link |  $\alpha_{i}$ (deg) |  $a_{i}$ (m) |  $d_{i}$ (m) |  $m_{i}$ (kg) |  $[I_{xx},I_{yy},I_{zz}]$ (kgm2)
---|---|---|---|---|---
1 | $90$ | $0$ | $0.276$ | $1$ | $[0.0022,0.0006,0.0023]$
2 | $180$ | $0.41$ | 0 | $1.5$ | $[0.0041,0.0255,0.0217]$
3 | $90$ | $0$ | $-0.01$ | $0.8$ | $[0.0029,0.0027,0.0004]$
4 | $60$ | $0$ | $-0.25$ | $0.3$ | $[0.7085,0.7405,0.1782]$
5 | $60$ | $0$ | $-0.009$ | $0.3$ | $[0.8275,0.8520,0.1708]$
6 | $180$ | $0$ | $0.203$ | $0.6$ | $[0.0048,0.0048,0.0002]$

###  V-B Experimental Results

TABLE III: Collaborative end-effector pose control result. The unit for position is cm and for orientation is deg. The error mean and standard deviation values are calculated from $10$ s data of the pose holding phase. Pose | Desired ${\bm{\xi}}_{e}^{k}$ | BPIK-planned ${\bm{\xi}}_{e}(\mathcal{T}_{n+1}({\bm{q}}^{*}_{k}))$ | Actual ${\bm{\xi}}_{e}^{k}$ | Errors
---|---|---|---|---
Position | Orientation
1 | $[-13\;-55\;94\;-61\;52\;96]^{T}$ | $[-14\;-56\;95\;-64\;52\;98]^{T}$ | $[-14\;-56\;95\;-63\;51\;97]^{T}$ | $0.76\pm 0.02$ | $0.43\pm 0.005$
2 | $[-15\;-67\;109\;57\;31\;-57]^{T}$ | $[-15\;-65\;106\;58\;30\;-58]^{T}$ | $[-15\;-67\;109\;53\;30\;-58]^{T}$ | $0.34\pm 0.01$ | $0.16\pm 0.017$
3 | $[-17\;-69\;106\;38\;55\;-37]^{T}$ | $[-16\;-68\;107\;37\;55\;-37]^{T}$ | $[-16\;-68\;107\;37\;55\;-37]^{T}$ | $0.72\pm 0.13$ | $0.40\pm 0.08$
4 | $[-15\;-58\;114\;32\;48\;-21]^{T}$ | $[-15\;-61\;112\;39\;42\;-18]^{T}$ | $[-15\;-60\;112\;39\;42\;-18]^{T}$ | $0.34\pm 0.09$ | $0.15\pm 0.046$

We first present the validation of the steering balance models. Figs. 6 and 6
show the values of the turning radius $r_{\phi_{0}}$ and the steering torque
sensitivity $S_{\tau}$, respectively, as the static steering angle $\phi_{0}$
increases. The experiment data clearly confirm the model predictions. It is
clear that when $\phi_{0}=90$ degs, the projected radius $r_{90}$ reaches the
maximum value and the increasing trend is monotonic. The steering sensitivity
$S_{\tau}$ also reaches its maximum point around $\phi_{0}=90$ degs with
$S_{\tau}=0.87$ Nm/deg. At $\phi_{0}=0$, the projected radius $r_{0}$ and
steering torque sensitivity $S_{\tau}$ are near zero. From this observation,
an initial steering angle $\phi_{0}$ is chosen around $90$ degs for following
stationary balance experiments. At $\phi_{0}=90$ degs, multiple stationary
balancing experiments were conducted. Fig. 6 shows the steering-induced
balance torque $\tau_{b}$ at different roll angles $\varphi_{b}$ and
increments $\delta$. Multiple experimental trials are plotted together with
the steering torque model prediction from (13), i.e., the 3D surface as shown
in the figure. The experimental data are scattered around the torque model
prediction with small errors. These results validate the steering-induced
balance torque model.

Figure 8: Verification of the recoverable roll angle region. The markers
”$\times$”, ”$\circ$” and ”$\star$” represent the successful balance trial of
the bikebot by one-wheel steering, two-wheel steering, and collaboratively
two-wheel steering with manipulation balancing strategies, respectively. The
vertical lines indicate the estimated maximum angle boundaries
$\varphi_{b}=\pm 3.4$, $\pm 5.6$, and $\pm 11.6$ degs.

Figure 9: (a) Snapshots of the manipulator with an inspected plant at four
poses. (b) Pose transition and end-effector trajectory under the pose planning
and control. The arrows (i.e., unit vectors
$({\bm{e}}_{x},{\bm{e}}_{y},{\bm{e}}_{z})$) at each pose represent the actual
orientation of the end-effector in $\mathcal{I}$. (c) Pose error (mean with
one standard deviation) in transition process (from Poses 1 to 4 as indicated
by the vertical lines) from 15 experiment trials. Top: Position error; Bottom:
Orientation error. The video of the experiment can be fount at:
https://youtu.be/jHQRNrrnPMc.

Fig. 7 shows the bikebot balance control results. The manipulator was removed
from the bikebot in this experiment. The controller (20) was used with
feedback gains $k_{p}=8.5$ and $k_{d}=2$. The entire trial is divided into
three stages as separated by the vertical lines in Fig. 7. Fig. 7 shows the
roll angle tracking errors. Fig. 7 illustrates the front and rear wheel
steering angle increments. In the first stage, the initial roll angle was
about $4$ degs and it was then regulated around zero. A chattering phenomenon
was observed in the roll angle profile and this was due to the fact that the
IMU angular measurement resolution was around $0.1$ deg, that is, the IMU
measurement was discretized with a minimal resolution of $0.1$ deg. This
oscillation also caused a similar chattering behavior in steering angle
increments in Fig. 7 since the roll angle measurement was used in steering
control. In the second stage starting at around $t=60$ s, the bikebot was
commanded to move back and forth around the zero with the maximum roll angles
around $4.5$ degs. The change of the reference roll angle was slow to meet the
quasi-static movement. The tracking error approached to zero. In the third
stage starting around at $t=270$ s, multiple disturbances were applied by
manually pushing upper frame of the bikebot. The roll angle errors caused by
the disturbances reached $6$ degs and the steering actuation compensated for
the disturbances. These results demonstrate the steering balance control
performance. Since the design enforced symmetrical steering commands, the
actual front and rear steering angles’ responses showed highly similar
behaviors.

Using the BEM and model parameters, we estimate the maximum stationary balance
roll angles. With one-wheel steering control, the maximum balanced roll angle
is around $3.4$ degs; with two-wheel steering control, around $5.6$ degs; and
additionally, if the manipulator is used to help balance collaboratively, it
increases to $11.6$ degs. To validate these estimates, we conducted multiple
balance control experiments. One experimental trial was considered successful
if the system was kept balance for a time period over $50$ s. Fig. 8 shows the
successful trials in the $\dot{\varphi}_{b}$-$\varphi_{b}$ plane. Each marker
in the figure represents the state at which the bikebot successively started
to balance under steering control, which was confirmed by comparing the model
predictions from (13) and dynamics (14). The experiments, which are in
agreement with the model prediction, validate the model analysis and
demonstrate the balance capability under various bikebot balancing strategies.

Figure 10: Large roll angle balance control of the mobile manipulation system.
(a) Roll angle, (b) robotic arm joint angles, and (c) steering angle
increments. The vertical lines in (a) divides the entire process into four
phases for each pose. (d) Roll angle error. (e) Online robotic arm trajectory
correction in pose transition phase (the difference between off-line planing
results and the actual angle).

We now present a plant inspection example for the end-effector to continuously
go through and stop momentarily at four poses (i.e., $N_{\xi}=4$). This
represents the end-effector movement during a plant scanning and inspection
task [20]. Fig. 9 shows the snapshots of the end-effector at the four poses.
The major movement of the end-effector (with a mounted camera) was along the
$z$-axis in $\mathcal{I}$ and orientation always pointed towards to the stalk
of a fake corn plant. The end-effector moved from one pose to another in
sequence and stopped for about 15 s at each pose. Fig. 9 shows the 3D
trajectories with the four poses. The planning and control parameters used in
experiments include: $\lambda_{1}=10$, $\lambda_{2}=1$, $\lambda_{3}=5$,
$\lambda_{4}=1.5$, ${\bm{W}}_{1}=\mathop{\rm diag}\nolimits(10,5,5,5,1,1,1)$,
${\bm{W}}_{2}=\mathop{\rm diag}\nolimits(1,1,1,1,1,1)$, $\kappa=5$,
$\epsilon=0.1$, $\varepsilon_{b}=0.4$ degs, $\dot{q}_{\max}=36$ deg/s,
$\ddot{q}_{\max}=120$ deg/s2,
${\bm{\tau}}^{\max}_{\theta}=[10\;15\;10\;5\;5\;5]^{T}$ Nm, $\delta_{\max}=15$
degs, $\dot{\delta}_{\max}=20$ deg/s, and the degree of the Bézier polynomial
$N=7$. For off-line planning implementation, the number of data points was
chosen as $N_{s}=50$ in each dimension of $\bm{q}(t)$ and the SQP method (via
_fmincon_ function) in Matlab was used for solving (19). The obtained Bézier
polynomial trajectory was then sampled at 100 Hz for real time control. The
off-line planner computed the trajectory for the next pose transition when
conducting the real-time motion control of the current pose movement. By doing
so, the proposed planner was capable of obtaining the trajectory with fast
computational time. Table III lists the desired end-effector poses, the poses
planned by the BPIK and the actual poses under the controller. From the table,
the position errors are within 8 mm and orientation errors within $0.45$ degs
at these four poses.

Fig. 10 shows the detailed experimental results. Figs. 10 and 10 show the
bikebot roll angle $\varphi_{b}$ and the six joint angles ${\bm{\Theta}}$ of
the manipulator, respectively. Since $\varphi_{b}$ and the first three joint
angles ($\theta_{1}$-$\theta_{3}$) played a major role to balance the entire
system, their reference trajectories were designed to avoid large variations
as shown in the figures. Poses 2 and 3 were searched in the local workspace
$\mathcal{X}_{\varphi_{b1}^{0}}({\bm{\Theta}})$ of Pose 1, and Pose 4 is
searched in the workspace $\mathcal{X}({\bm{q}})$. The bikebot roll angle
change was approximately around $1.5$ degs. At $t=0$ s, the manipulator was at
the desired balance configuration as Pose 1. Around $t=30$ s, the manipulator
started moving to Pose 2. Small disturbances were introduced at around $t=40$
s, causing about a $0.4$-deg roll angle error. The velocity correction control
was applied to compensate for the roll angle error; see Fig. 10. No obvious
roll angle error was observed during the transition from Poses 2 to 3 (except
around $0.1$ degs oscillation). The bikebot platform was required to move in
the transition from Poses 3 to 4. Around $t=110$ s, a large roll angle change
was commanded by the steering actuation and the velocity correction control
was needed; see Fig. 10.

###  V-C Discussion

To further examine the performance, we conducted additional comparison
experiments. Fig. 10 shows the bikebot roll angle when the bikebot balance
priority is not considered in the trajectory planning. Clearly, the entire
system lost balance in the pose transition phase at $t=50s$. The result
confirms the challenge in manipulator pose movement control. We further
conducted and repeated the 4-pose control experiment 15 times. Fig. 9
summarizes the end-effector pose errors statistics (i.e., mean and one-
standard deviation) with and without online velocity correction under the
proposed trajectory planning method as modeling errors and system
uncertainties might exist. With the online modification the end-effector
position errors are less than $5$ mm and the orientation errors within $0.3$
degs. Relative large errors happened around $t=42$ and $120$ s with about $10$
mm and $0.7$ degs, respectively. This is consistent with the previous results.
The position errors are at the same level of the manipulator hardware
performance limits ($3.7$ mm) that are provided by the vendor and the
orientation errors are much less that level ($2.1$ degs). Without the velocity
correction, both the position and orientation tracking errors are larger than
these with correction. The results demonstrate the successful balance and pose
control performance by the design. We also conducted computational time
comparison between the proposed trajectory planning algorithm and the DP
method and the result is shown in Table IV. The numerical results confirmed
that the computational cost of the DP method was over 200 times higher than
that of the proposed algorithm to solve the optimization problem in (19).

TABLE IV: Computational cost comparison for the proposed and the DP algorithms $N_{s}$ | $50$ | $100$ | $200$ | $500$
---|---|---|---|---
Proposed algorithm (sec) | $9.69$ | $14.94$ | $31.58$ | $21.46$
DP method (hour) | $0.14$ | $1.04$ | $3.02$ | $\geq 12$

Although demonstrating successful results, the current work have several
limitations for further improvement. We only studied coordinated control of
the manipulator on stationary bikebot and it would be desirable to extend to
moving platform case. Second, the steering control in this work does not
include the dynamic effects of the steering mechanism. The control performance
might be improved with incorporating dynamic steering effect. The trajectory
of the bikebot roll angle and the manipulator joint angles was planned off-
line and online planning is desirable for applications with dynamic obstacle
avoidance. Finally, the proposed method is built on the accurate model of the
system and it is desirable to extend to handle model parameter uncertainties
in complex, dynamic environment. One possible approach is to use machine
learning-based methods. For example, as discussed in [38], the robot dynamics
might be approximated and estimated using a Gaussian process learning approach
and a learning-based motion control can be then designed.

##  VI Conclusion

We presented a coordinated balance and pose control for a stationary mobile
manipulation using a two-wheel steered bikebot. The mobile platform is
inherently unstable and the dynamics of the platform and the manipulator are
strongly coupled. We first presented a two-wheel steering model and identified
the use of $\phi_{0}=90$ degs as the most beneficial steering angle for
stationary balance. A balance equilibrium manifold was extended to the mobile
manipulation for coordinated motion control. Built on the BEM, a balance-
priority design was presented to solve the optimal joint angles for the
bikebot and the manipulator. Coordinated balance and pose control was achieved
by enforcing the entire system moving on the BEM with online manipulator
velocity correction control. We conducted extensive experiments and the
results demonstrated the performance of the balance and pose control for a
plant inspection application.

## References

  * [1] Z. Lu, A. Chauhan, F. Silva, and L. S. Lopes, “A brief survey of commercial robotic arms for research on manipulation,” in _Proc. 2012 IEEE Symp. Robot. Appli._ , Kuala Lumpur, Malaysia, 2012, pp. 986–991.
  * [2] O. Brock, J. Park, and M. Toussaint, “Mobility and Manipulation,” in _Springer Handbook of Robotics_ , B. Siciliano and O. Khatib, Eds. Berlin, Germany: Springer, 2016.
  * [3] E. Simetti, G. Casalino, F. Wanderlingh, and M. Aicardi, “A task priority approach to cooperative mobile manipulation: Theory and experiments,” _Rob. Auton. Syst._ , vol. 122, pp. 1–12, 2019.
  * [4] B. Arad, J. Balendonck, R. Barth, O. Ben-Shahar, Y. Edan, T. Hellström, J. Hemming, P. Kurtser, O. Ringdahl, T. Tielen, and B. van Tuijl, “Development of a sweet pepper harvesting robot,” _J. Field Robotics_ , vol. 37, no. 6, pp. 1027–1039, 2020.
  * [5] K.-D. Nguyen, “Stability and robustness of adaptive controllers for underactuated lagrangian systems and robotic networks,” Ph.D. dissertation, Dept. Mech. Sci. Eng., Univ. Illinois, Urbana-Champaign, IL, 2015.
  * [6] H. Stuart, S. Wang, O. Khatib, and M. R. Cutkosky, “The ocean one hands: An adaptive design for robust marine manipulation,” _Int. J. Robot. Res._ , vol. 36, no. 2, pp. 150–166, 2017.
  * [7] S. Kim, H. Seo, S. Choi, and H. J. Kim, “Vision-guided aerial manipulation using a multirotor with a robotic arm,” _IEEE/ASME Trans. Mechatronics_ , vol. 21, no. 4, pp. 1912–1923, 2016.
  * [8] H. Bonyan Khamseh, F. Janabi-Sharifi, and A. Abdessameud, “Aerial manipulation—a literature survey,” _Rob. Auton. Syst._ , vol. 107, pp. 221 – 235, 2018.
  * [9] P. Holmes, S. Kousik, B. Zhang, D. Raz, C. Barbalata, M. Johnson-Roberson, and R. Vasudevan, “Reachable sets for safe, real-time manipulator trajectory design,” in _Proc. Robotics: Sci. Syst._ , Oregon, USA, 2020, pp. 1–9.
  * [10] R. Raja, A. Dutta, and B. Dasgupta, “Learning framework for inverse kinematics of a highly redundant mobile manipulator,” _Rob. Auton. Syst._ , vol. 120, pp. 1–13, 2019.
  * [11] J. Woolfrey, “Control of manipulators on moving platforms under disturbance,” Ph.D. dissertation, Dept. Eng. and Info. Tech., Univ. Tech. Sydney, Sydney, Australia, 2019.
  * [12] P. J. From, V. Duindam, J. T. Gravdahl, and S. Sastry, “Modeling and motion planning for mechanisms on a non-inertial base,” in _Proc. IEEE Int. Conf. Robot. Autom._ , Kobe, Japan, 2009, pp. 3320–3326.
  * [13] M. Raibert, K. Blankespoor, G. Nelson, and R. Playter, “Bigdog, the rough-terrain quadruped robot,” in _Proc. IFAC World Congress_ , Seoul, Korea, 2008, pp. 10 822–10 825.
  * [14] S. Kim, H. Seo, J. Shin, and H. J. Kim, “Cooperative aerial manipulation using multirotors with multi-dof robotic arms,” _IEEE/ASME Trans. Mechatronics_ , vol. 23, no. 2, pp. 702–713, 2018.
  * [15] M. V. Minniti, F. Farshidian, R. Grandia, and M. Hutter, “Whole-body mpc for a dynamically stable mobile manipulator,” _IEEE Robot. Automat. Lett._ , vol. 4, no. 4, pp. 3687–3694, 2019.
  * [16] U. Nagarajan, G. Kantor, and R. L. Hollis, “Trajectory planning and control of an underactuated dynamically stable single spherical wheeled mobile robot,” in _Proc. IEEE Int. Conf. Robot. Autom._ , Kobe, Japan, 2009, pp. 3743–3748.
  * [17] S. Moe, G. Antonelli, A. R. Teel, K. Y. Pettersen, and J. Schrimpf, “Set-based tasks within the singularity-robust multiple task-priority inverse kinematics framework: General formulation, stability analysis, and experimental results,” _Front. Robot. AI_ , vol. 3, 2016, article 16.
  * [18] E. Simetti, G. Casalino, F. Wanderlingh, and M. Aicardi, “Task priority control of underwater intervention systems: Theory and applications,” _Ocean Eng._ , vol. 164, pp. 40–54, 2018.
  * [19] O. Kanoun, F. Lamiraux, and P.-B. Wieber, “Kinematic control of redundant manipulators: Generalizing the task-priority framework to inequality task,” _IEEE Trans. Robotics_ , vol. 27, no. 4, pp. 785–792, 2011.
  * [20] M. Edmonds, J. Yi, N. K. Singa, and L. Wang, “Generation of high-density hyperspectral point clouds of crops with robotic multi-camera planning,” in _Proc. IEEE Conf. Automat. Sci. Eng._ , Vancouver, Canada, 2019, pp. 1475–1480.
  * [21] A. Muetze and Y. C. Tan, “Electric bicycles - a performance evaluation,” _IEEE Industry Applications Magazine_ , vol. 13, no. 4, pp. 12–21, 2007.
  * [22] A. Bechar and C. Vigneault, “Agricultural robots for field operations: Concepts and components,” _Biosyst. Eng._ , vol. 149, pp. 94–111, 2016\.
  * [23] P. Wang, J. Yi, T. Liu, and Y. Zhang, “Trajectory tracking and balance control of an autonomous bikebot,” in _Proc. IEEE Int. Conf. Robot. Autom._ , Singapore, 2017, pp. 2414–2419.
  * [24] P. Wang, J. Yi, and T. Liu, “Stability and control of a rider–bicycle system: Analysis and experiments,” _IEEE Trans. Automat. Sci. Eng._ , vol. 17, no. 1, pp. 348–360, 2020.
  * [25] Y. Zhang, P. Wang, J. Yi, D. Song, and T. Liu, “Stationary balance control of a bikebot,” in _Proc. IEEE Int. Conf. Robot. Autom._ , Hong Kong, China, 2014, pp. 6706–6711.
  * [26] P. Seekhao, K. Tungpimolrut, and M. Parnichkun, “Development and control of a bicycle robot based on steering and pendulum balancing,” _Mechatronics_ , vol. 69, pp. 1–12, 2020.
  * [27] L. Keo and M. Yamakita, “Control of an autonomous electric bicycle with both steering and balancer controls,” _Adv. Robot._ , vol. 25, pp. 1–22, 2011\.
  * [28] N. Takasugi, K. Kojima, S. Nozawa, Y. Kakiuchi, K. Okada, and M. Inaba, “Nonlinear balance control of an unmanned bicycle: Design and experiments,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst._ , virtual conference, 2020, pp. 7279–7284.
  * [29] Y. Zhang, J. Li, J. Yi, and D. Song, “Balance control and analysis of stationary riderless motorcycles,” in _Proc. IEEE Int. Conf. Robot. Autom._ , Shanghai, China, 2011, pp. 3018–3023.
  * [30] S. Singhania, I. Kageyama, and V. M. Karanam, “Study on low-speed stability of a motorcycle,” _Appl. Sci._ , vol. 9, no. 11, pp. 1–15, 2019.
  * [31] P. Corke, _Robotics, Vision and Control: Fundamental Algorithms in MATLAB®_ , 2nd ed. New York, NY: Springer, 2017.
  * [32] R. M. Murray, Z. Li, and S. S. Sastry, _A Mathematical Introduction to Robotic Manipulation_. Boca Raton, FL: CRC Press, 1994.
  * [33] Y. Gong, K. Chen, J. Yi, and T. Liu, “Control of a two-wheel steering bikebot for agile maneuvers,” in _Proc. IEEE/ASME Int. Conf. Adv. Intelli. Mechatronics_ , Hong Kong, China, 2019, pp. 984–989.
  * [34] E. R. Westervelt, J. W. Grizzle, C. Chevallereau, J. H. Choi, and B. Morris, _Feedback Control of Dynamic Bipedal Robot Locomotion_. Boca Raton, FL: CRC Press, 2007.
  * [35] J. Nocedal and S. J. Wright, _Numerical Optimization_ , 2nd ed. New York, NY: Springer, 2006.
  * [36] G. Antonelli, “Stability analysis for prioritized closed-loop inverse kinematic algorithms for redundant robotic systems,” _IEEE Trans. Robotics_ , vol. 25, no. 5, pp. 985–994, 2009.
  * [37] Kinova Inc., _User Guide, KINOVA Gen2 Ultra lightweight robot_ , Boisbriand QC, Canada, 2019. [Online]. Available: https://www.kinovarobotics.com/en/resources/gen2-technical-resources
  * [38] F. Han and J. Yi, “Stable learning-based tracking control of underactuated balance robots,” _IEEE Robot. Automat. Lett._ , vol. 6, no. 2, pp. 1543–1550, 2021.