# Characterization of Asymmetric Gap-Engineered Josephson Junctions and 3D
Transmon Qubits

Zach Steffen  S. K. Dutta  Haozhi Wang  Kungang Li  Yizhou Huang  Yi-Hsiang
Huang  Advait Mathur  F. C. Wellstood B. S. Palmer Zach Steffen, Haozhi Wang,
Yizhou Huang, Yi-Hsiang Huang, Advait Mathur, and B. S. Palmer are with the
Quantum Materials Center at University of Maryland, College Park, MD 20742,
USA and also with Laboratory for Physical Sciences, College Park, MD 20740,
USA (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>bpalmer@umd.edu).Kungang Li, S. K. Dutta, and F. C.
Wellstood are with the Quantum Materials Center at University of Maryland,
College Park, MD 20742, USA (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>well@umd.edu).

###### Abstract

We have fabricated and characterized asymmetric gap-engineered junctions and
transmon devices. To create Josephson junctions with asymmetric gaps, Ti was
used to proximitize and lower the superconducting gap of the Al counter-
electrode. DC IV measurements of these small, proximitized Josephson junctions
show a reduced gap and larger excess current for voltage biases below the
superconducting gap when compared to standard Al/AlOx/Al junctions. The energy
relaxation time constant for an Al/AlOx/Al/Ti 3D transmon was $T_{1}=1\ \mu
s$, over two orders of magnitude shorter than the measured $T_{1}=134\ \mu s$
of a standard Al/AlOx/Al 3D transmon. Intentionally adding disorder between
the Al and Ti layers reduces the proximity effect and subgap current while
increasing the relaxation time to $T_{1}=32\ \mu s$.

###### Index Terms:

Transmon, qubit, gap-engineering, quantum computing.

††publicationid: pubid:

##  I Introduction

Qubits based on superconducting circuits are one of the leading platforms in
implementing a quantum computer [1, 2, 3, 4]. One limitation of
superconducting qubits is their coherence times which places a hard limit on
fidelity of operations and time of computation in the absence of error
correction. While state-of-the-art transmon qubits have energy relaxation
times reaching up to $T_{1}=500$ microseconds [5, 6], significant further
improvement is required for most quantum computation applications.

In a transmon qubit, intrinsic losses come predominantly from the coupling of
the qubit to two-level systems in lossy dielectrics [7, 8, 9] and non-
equilibrium quasiparticle loss channels [10, 11, 12, 13]. Quasiparticles in
superconducting qubits can tunnel across the Josephson junction, absorbing a
photon, and relaxing the qubit. The decay rate from the qubit’s excited state
$|e\rangle$ to the ground state $|g\rangle$ as a result of quasiparticles
tunneling across the qubit junction can be written as

$\displaystyle\Gamma_{e\rightarrow g}$
$\displaystyle=\frac{E_{C}}{hf_{ge}e}\Big{(}I_{\rightarrow}(f_{ge})+I_{\leftarrow}(f_{ge})\Big{)}.$
(1)

Here $E_{C}$ is the charging energy, $h$ is Planck’s constant, $e$ is the
electron charge, and $(I_{\rightarrow}(f_{ge})+I_{\leftarrow}(f_{ge}))$ is the
sum of the quasiparticle current in both directions across the junction at a
bias equivalent to the qubit frequency $f_{ge}$ [14].
$I_{\rightarrow}/I_{\leftarrow}$ is determined by the specifics of the
Josephson tunnel barrier and the density of quasiparticles $n_{qp}$ at the
junction. For a BCS superconductor in thermal equilibrium, $n_{qp}$ decreases
exponentially with decreasing temperature ($T$) [15] resulting in a reduced
quasiparticle current and therefore reduced $\Gamma_{e\rightarrow g}$. While
theoretically, Al below $T<50$ mK should have $n_{qp}<10^{-12}\mathrm{\mu
m}^{-3}$; experiments have reported nonequilibrium $n_{qp}\sim 0.1-1\
\mathrm{\mu m}^{-3}$ [16, 17, 18, 19, 20, 21].

Recently, $T_{1}$ measurements of 3D transmons have shown an increase in
$T_{1}$ as $T$ was increased from 20 mK to 150 mK. To explain this anomalous
temperature dependence, the superconducting gaps $\Delta_{1,2}$ for the two Al
layers forming the Al/AlOx/Al junction were modeled to be slightly different,
such that for low $T$, the density $n_{qp}$ at the junction increases in one
of the electrode layers, thus resulting in a decrease in $T_{1}$[22]. This
model, which is distinct from quasiparticle trapping models, suggests that
increasing the $|\Delta_{1}-\Delta_{2}|>hf_{ge}$ should increase $T_{1}$, even
in the presence of nonequilibrium quasiparticles. Recent results by oxygen-
doping Al have shown long lived transmons informed by these predictions [23].
Other work has also discussed the improvement in the performance of similar
devices by gap-engineering the junction [17, 24, 25].

With this motivation, we have fabricated and characterized junctions and
transmon qubits with asymmetric superconducting energy gaps across the
junction. In particular, the superconducting gap of the junction’s larger-
volume top electrode was lowered via the proximity effect by evaporating Ti on
top of Al. Because the superconducting energy gap of Ti is lower than that of
Al, the effective superconducting order parameter of an Al/Ti bilayer will
have a value less than that of an Al film.

In Section II, we describe the fabrication of the Al/Ti bilayer with and
without an additional thin AlOy disorder added in between these layers. The
theoretical critical temperature depends not only on the order parameters and
coherence lengths of the two films but also on the AlOy interface barrier
resistance [26, 27, 28, 29]. DC characterization of these junctions is
discussed in Section III. Junctions with the Ti capping layer directly on the
Al display a reduced energy gap but also a significant amount of tunneling
current in the subgap region. We model the IV’s, including some of subgap
features, using a multiple-Andreev reflection (MAR) theory and extract
junction parameters. Lifetime measurements of 3D transmons with these
electrode structures are discussed in Section IV. In particular, the Ti
capping layer displays a substantial shorter lifetime of $T_{1}=1\
\mu\mathrm{s}$ when compared to standard Al/AlOx/Al junctions while
introduction of AlOy disorder between the Al and Ti layers can partially
restore the qubit $T_{1}$.

##  II Experimental Setup

For this work, a variety of small area ($\sim$ 150 nm x 150 nm) low critical-
current junctions were fabricated either by the Dolan bridge [30] or Manhattan
technique [31]. To fabricate the junctions (see Fig. 1), a lift-off stencil
was created using an electron-beam lithography system in a bilayer electron-
beam resist with an undercut allowing metal to be deposited at an angle on the
substrate.

After developing the resist, multiple metal depositions and oxidations were
performed either in a Plassys MED 550 S electron-beam evaporator or using a
tungsten wound basket in a turbo-pumped thermal evaporator. For each junction,
30 nm of Al was first deposited, followed by a static oxidation with a fixed
dose of 0.8 - 3 mBar O2 for 10 minutes to form the insulating tunnel barrier.
Following the first oxidation, the chamber was evacuated and a secondary metal
counter-electrode was deposited at a different angle. The composition of this
top electrode was 50 nm Al for the standard Al/AlOx/Al junctions, 45 nm Al
followed by 5 nm of Ti for the Al/Ti bilayer, or 40 nm Al followed by an
additional oxidation and a deposition of 20 nm Ti for the disordered
Al/AlOy/Ti bilayer. The Al films were deposited at a rate of 0.5 nm/s; the Ti
films were deposited at 0.2 nm/s.

Next, the extraneous metal on the top of the resist was lifted off in a hot
bath of NMP. Table I has a summary of the devices that were fabricated in this
study. Because the contact pad areas were approximately $10^{7}$ times larger
than the junction areas, we argue each of these devices is effectively a
single Josephson junction with asymmetric electrodes.

Figure 1: Schematic of the Dolan bridge fabrication process for asymmetric junctions. We begin the process by spinning and baking MMA (c), and ZEP (or PMMA for the Manhattan junctions) (b), followed by depositing a 10 nm anti-charging Al layer (a) on a silicon or sapphire substrate (d). The desired pattern is defined via electron-beam lithography then developed. Within the evaporator, an Ar ion mill cleaning removes residual resist. Next, the junction is formed in standard double-angle evaporation where Al is deposited at the first angle; oxygen is let into the chamber to form the tunnel barrier; then Al is deposited at a second angle. To form the superconducting bilayer, we optionally perform another oxidation at a lower dose to create a layer of disorder. Finally, Ti is deposited at the same second angle, forming the bilayer directly on top of either the second Al or AlOy disordered layer Junction |  $O_{2}$ disorder | $\Delta_{2}$ | $R_{n}$
---|---|---|---
Composition | (mbar$\cdot$min) | $(\mu eV)$ | $(k\Omega)$
Al/AlOx/Al | - | 200 | 18.6
Al/AlOx/Al | - | 200 | 8.3
Al/AlOx/Al/Ti | - | 100 | 6.3
Al/AlOx/Al/Ti | - | 140 | 7.0
Al/AlOx/Al/Ti | - | 110 | 8.5
Al/AlOx/Al/AlOy/Ti | 0.3 | 120 | 2.6
Al/AlOx/Al/AlOy/Ti | 0.6 | 230 | 8.8
Al/AlOx/Al/AlOy/Ti | 0.6 | 220 | 8.5
Al/AlOx/Al/AlOy/Ti | 1.8 | 220 | 9.0
Al/AlOx/Al/AlOy/Ti | 2.8 | 220 | 8.5
TABLE I: Comparison of measured superconducting gap for the top counter-
electrode $\Delta_{2}$ and normal state resistances for different junction
compositions. The counter-electrode gap $\Delta_{2}$ is extracted from the
quasiparticle-quasiparticle tunneling peak assuming the bottom electrode
$\Delta_{1}~{}=~{}190~{}\mathrm{\mu eV}$. Adding Ti to the counter-electrode
reduces $\Delta_{2}$ while adding an AlOy barrier greater than 0.3
mbar$\cdot$min between the Al and Ti layers restores $\Delta_{2}$.

To perform low-temperature measurements, the devices were mounted and cooled
on a cryogen-free Leiden dilution refrigerator with a base temperature below
20 mK. To perform the junction DC measurements, the device was glued using GE
varnish to a copper package and then wire-bonded using Al wire to four
separate leads on a PCB with either a RC low-pass, LC low-pass filter, or no
filters. Each measurement line then went through a combined pi-powder low-pass
filter on the mixing chamber plate before going to room temperature using
thermal coax cables. Four-wire differential conductance measurements were
performed by applying a combined DC and low frequency 13 Hz AC voltage
excitation and measuring the AC current with a lock-in amplifier. Using the
additional two wires across the junction, the voltage was amplified with a
PARC pre-amplifier before being digitized or measured with an additional lock-
in amplifier. After the data were acquired, the dI/dV measurements were
numerically integrated resulting in the IV curves presented in the next
section.

To perform qubit energy relaxation measurements, a transmon qubit was
fabricated on a sapphire substrate and mounted in a 3D rectangular cavity
machined out of an aluminum-6063 alloy. The bare resonant frequency of the
cavity was approximately at 7.8 GHz and a loaded quality factor, limited by
the output coupler, of $Q_{L}\simeq 12,000$. The cavity was mounted in two
Amumetal 4K magnetic shields on the mixing chamber plate of the same Leiden
dilution refrigerator. To measure $T_{1}$, a $\pi$-pulse was used to initially
excite the qubit from the ground state, followed by a variable delay before
measuring the qubit state through dispersive cavity readout [32]. Measurements
were averaged 500-5000 times. The total acquisition time of a single decay
curve varied between one and ten minutes. $\overline{T_{1}}$ values are
averaged from 10’s of individual curves at base temperature.

##  III DC Tunneling Measurements

To quantify the superconducting gap electrodes and characterize the quality of
the junction, DC measurements were performed. Fig. 2 shows integrated IV
curves for three different junctions: with no Ti (dotted), Al/Ti bilayer
(solid), and with added AlOy disorder between the Al and Ti layers (dot-
dashed). These junctions were nominally identical to the measured transmon
junctions.

For the Al/AlOx/Al junctions, the current sharply increases at
$V=380~{}\mathrm{\mu V}$ by approximately 25 nA. This is associated with
quasiparticle-quasiparticle tunneling at a voltage bias
$V=(\Delta_{1}+\Delta_{2})/e$ corresponding to the sum of the two
superconducting gaps. Above this bias, the junction shows ohmic normal-state
tunneling.

In the subgap region below this voltage bias value, there are a number of
small subgap tunneling features revealing deviations from BCS tunneling
characteristics. At the lowest voltage biases, the measured current slowly
decreases. Dissipation from Josephson oscillations in a lossy radio-frequency
(RF) environment can give rise to subgap features when the junction sees
finite, real impedance at the Josephson frequency. This effect is likely the
cause of the anomalous tunneling observed at small biases corresponding to
Josephson oscillations from 2 - 20 GHz and could be reduced by measuring a
biased SQUID [33].

Subgap current steps at individual gap values can also appear in junctions
with finite transparency due to multiple Andreev reflections (MAR) [34, 35].
MAR theory expands quasiparticle tunneling to junctions with non-zero electron
transmission amplitude, $t$. In this case, a quasiparticle transmitted through
the tunnel barrier can “Andreev reflect” off the superconducting interface
repeatedly between the two superconducting electrodes. In tunneling
characteristics, these multiple reflections give rise to steps in the measured
current at biases equal to integer fractions of the individual electrode gaps,
$\Delta_{1,2}/n$. The height of these steps is proportional to the junction
transparency $D=|t|^{2}$. In principle, the MAR theory can be used to model
the IV data and extract $D$ and $\Delta_{1,2}$ by matching the features in the
theory to the measured IV’s. Small gap asymmetry is expected due to
differences in film thickness [36, 37], but self-heating, noise, and the
nature of some of these novel junctions limit the resolution of this
technique. At $V\approx 190\ \mathrm{\mu V}$, the curve in Fig. 2 displays an
approximate 1 nA step in the current, which we associate with the
superconducting gap of the Al electrodes.

Changing the counter-electrode to an Al/Ti bilayer modifies many aspects of
the IV characteristics (see the solid curve in Fig. 2). The onset of
quasiparticle-quasiparticle tunneling occurs at a reduced voltage bias such
that the sum of the superconducting gaps
$\Delta_{1}+\Delta_{2}=310~{}\mathrm{\mu eV}$. Because the fabrication of the
first layer for all of our devices is identical, the reduction in the sum of
the gaps is attributed to reduction of the gap in the top electrode (see Table
I), implying that the addition of a thin Ti layer reduces the gap of the
structure by nearly $100~{}\mathrm{\mu eV}$. Steps at $220~{}\mathrm{\mu eV}$
and $105~{}\mathrm{\mu eV}$ (see arrows in Fig. 2) match these proposed gap
values. Another striking feature of the IV data for the Al/Ti bilayer
junctions is the gradual increase in the current by about 15 nA in the subgap
region between $100~{}\mathrm{\mu eV}$ and $300~{}\mathrm{\mu eV}$, which is
significantly more than the other junctions.

From the MAR theory, the increased current suggests the junction barrier has
higher transparency which could lead to greater qubit depolarization rates.
Both Al and O can interdiffuse with Ti layers [38], potentially introducing
significant changes to the top electrode or the junction barrier [39].

Attempting to reduce this subgap current, new devices were fabricated with an
additional thin AlOy disordered layer between the Al and Ti layers. The oxygen
doses for this process were less than the dose to form the junction tunnel
barrier. For the lowest dose of 0.3 $\mathrm{mBar\cdot min}$, the IV
characteristics resemble Al/AlOx/Al/Ti junctions with no added disorder.
However, when the oxygen dose is increased to 0.6 $\mathrm{mBar\cdot min}$,
the sum of the gaps and low junction transparency of the initial Al/AlOx/Al
junctions are recovered (see green curve of Fig. 2). Further IV
characterization of junctions with disorder doses between 0.3
$\mathrm{mBar\cdot min}$ and 0.6 $\mathrm{mBar\cdot min}$ is still required to
determine if this bilayer technique can be used to fabricate junctions with
both a low transparency and the large desired gap asymmetry
($|\Delta_{1}-\Delta_{2}|>20~{}\mu\mathrm{eV}$). Table I shows the extracted
top electrode gaps for all of the measured junctions.

Figure 2: IV traces of three Josephson junctions at 20 mK with different
counter-electrode structures (bottom plot is the same data as the top plot but
with reduced current range). The dotted trace is a standard Al/AlOx/Al
transmon junction. The solid trace is data for an Al/AlOx/Al/Ti junction (note
reduction in $V$ to observe quasiparticle-quasiparticle tunneling due to the
Al/Ti bilayer counter-electrode). The dot-dashed curve is a measurement of a
Al/AlOx/Al/AlOy/Ti junction with an additional AlOy disorder layer between the
Al and Ti bilayer. Arrows mark visible steps at expected $\Delta_{1}$,
$\Delta_{2}$, or $\Delta_{1}+\Delta_{2}$ values.

##  IV Lifetime Measurements

We fabricated three separate 3D transmons to investigate how these different
junctions perform as qubits. The fabricated devices consisted of one
Al/AlOx/Al transmon with no Ti layer, one Al/AlOx/Al/Ti transmon with a
nominally clean interface between the Al and Ti, and one Al/AlOx/Al/AlOy/Ti
transmon with an oxygen dose of $0.6\ \mathrm{mbar}\cdot\mathrm{min}$ between
the Al and Ti depositions. The Al/AlOx/Al/Ti transmon was fabricated using the
Dolan bridge process and had frequency $f_{ge}=5.1~{}\mathrm{GHz}$ while the
Al/AlOx/Al and Al/AlOx/Al/AlOy/Ti transmons were fabricated using the
Manhattan process and had $f_{ge}=2.7~{}\mathrm{GHz},4.0~{}\mathrm{GHz}$
respectively.

The standard Al/AlOx/Al transmon with no Ti performed the best with a mean
energy relaxation time $\overline{T}_{1}=134~{}\mathrm{\mu s}$ at base
temperature. When the temperature of the fridge was increased to $T=140$ mK,
the relaxation time increased to $\overline{T}_{1}=210~{}\mathrm{\mu s}$
before decreasing at higher temperatures (see Fig. 3). This temperature
dependence of $T_{1}$ is consistent with previous results [22].

The Al/AlOx/Al/Ti transmon with no additional AlOy barrier had the worst
lifetime with $\overline{T}_{1}=1~{}\mathrm{\mu s}$. This device’s coherence
was significantly degraded from the standard Al/AlOx/Al devices and the DC
IV’s showed a large amount of subgap current. From these observations, the
junction barrier may be compromised by this specific fabrication procedure
potentially due to the ability of Al, Ti, and O to diffuse between layers
changing material properties of the electrode [38]. To attempt to remove this
issue, the next device implemented an AlOy barrier in between the Al and Ti
bilayer.

The Al/AlOx/Al/AlOy/Ti transmon with the additional barrier had a relaxation
time with $\overline{T}_{1}=32~{}\mathrm{\mu s}$. The addition of the
disordered layer not only reduced the current in the sub-gap region of the
IV’s (Fig. 2), but also increased the $T_{1}$ when compared to the Al/Ti
bilayer with no disorder. $T_{1}$ was still degraded compared to the
Al/AlOx/Al transmon, but further measurements are necessary to determine
device-to-device variability of the disorder bilayered transmons.

Figure 3: Repeated $T_{1}$ measurements of three different 3D transmon qubits
versus temperature. The squares show data of a standard Al/AlOx/Al qubit. The
triangle data had a bilayer of Al/Ti as the counter-electrode. The circle data
had an additional thin disordered AlOy ($0.6~{}\mathrm{mbar\cdot min}$ O2
dose) in between the Al and Ti counter-electrode. The fluctuations for each
qubit are approximately 10% of the respective average $T_{1}$ time.

Finally, to test the loss of the Ti film, a device with quarter-wave coplanar
waveguide resonators made from a 200 nm thick Ti film was fabricated. The low-
temperature single photon internal quality factors of these resonators were
measured to be $Q_{i}\simeq 3000$ resulting in an internal photon lifetime of
$\kappa^{-1}=100$ ns, a value even smaller than our transmon qubits.

Unlike Al, Ti does not form a self-limiting oxide [38, 40]. This could result
in a greater amount of dielectric loss or surface resistance loss and thus a
greater rate of energy decay as observed in the devices with Ti presented
here.

##  V Conclusion

By depositing a thin Ti film on top of Al counter-electrode, the Al is
proximitized and reduced the superconducting gap from approximately
$\Delta=200~{}\mathrm{\mu eV}$ to $100~{}\mu V$ for the Al/Ti bilayer. This
bilayer was used to create low critical current asymmetric superconducting gap
Josephson junctions. DC differential conductance measurements at 20 mK were
performed to compare these junctions to standard Al/AlOx/Al junctions as well
as devices with additional AlOy disorder placed in between the Al and Ti
layer. 3D transmon qubits with added Ti had reduced energy relaxation times
with the Al/Ti device having the smallest $T_{1}=1.0~{}\mathrm{\mu s}$. The
transmon fabricated with AlOy disorder between the Al and Ti showed
intermediate lifetimes up to $T_{1}=32~{}\mu$s. Future work could explore
burying the Ti under other metal layers to reduce oxidation, fabricating
asymmetric junctions with other superconducting materials, as well as
investigating the material composition used for these devices.

## Acknowledgements

We would like to thank Chris Lobb, Alan Kramer, Jon Cripe, Edwin Supple, and
Brian Gorman for useful discussions on the fabrication, measurement, and
interpretation of results.

## References

  * [1] M. H. Devoret and R. J. Schoelkopf, “Superconducting circuits for quantum information: An outlook,” _Science_ , vol. 339, no. 6124, pp. 1169–1174, 2013.
  * [2] M. Kjaergaard, M. E. Schwartz, J. Braumüller, P. Krantz, J. I.-J. Wang, S. Gustavsson, and W. D. Oliver, “Superconducting qubits: Current state of play,” _Annual Review of Condensed Matter Physics_ , vol. 11, no. 1, pp. 369–395, 2020.
  * [3] P. Krantz, M. Kjaergaard, F. Yan, T. P. Orlando, S. Gustavsson, and W. D. Oliver, “A quantum engineer’s guide to superconducting qubits,” _Applied Physics Reviews_ , vol. 6, no. 2, p. 021318, 2019.
  * [4] Krinner, S., Storz, S., Kurpiers, P., Magnard, P., Heinsoo, J., Keller, R., Lütolf, J., Eichler, C., and Wallraff, A., “Engineering cryogenic setups for 100-qubit scale superconducting circuit systems,” _EPJ Quantum Technol._ , vol. 6, no. 1, p. 2, 2019. [Online]. Available: https://doi.org/10.1140/epjqt/s40507-019-0072-0
  * [5] A. P. M. Place, L. V. H. Rodgers, P. Mundada, B. M. Smitham, M. Fitzpatrick, Z. Leng, A. Premkumar, J. Bryon, A. Vrajitoarea, S. Sussman, G. Cheng, T. Madhavan, H. K. Babla, X. H. Le, Y. Gang, B. Jäck, A. Gyenis, N. Yao, R. J. Cava, N. P. de Leon, and A. A. Houck, “New material platform for superconducting transmon qubits with coherence times exceeding 0.3 milliseconds,” _Nature Communications_ , vol. 12, no. 1, p. 1779, Mar 2021\.
  * [6] C. Wang, X. Li, H. Xu, Z. Li, J. Wang, Z. Yang, Z. Mi, X. Liang, T. Su, C. Yang, G. Wang, W. Wang, Y. Li, M. Chen, C. Li, K. Linghu, J. Han, Y. Zhang, Y. Feng, Y. Song, T. Ma, J. Zhang, R. Wang, P. Zhao, W. Liu, G. Xue, Y. Jin, and H. Yu, “Towards practical quantum computers: transmon qubit with a lifetime approaching 0.5 milliseconds,” _npj Quantum Information_ , vol. 8, no. 1, p. 3, Jan 2022.
  * [7] J. M. Martinis, K. B. Cooper, R. McDermott, M. Steffen, M. Ansmann, K. D. Osborn, K. Cicak, S. Oh, D. P. Pappas, R. W. Simmonds, and C. C. Yu, “Decoherence in josephson qubits from dielectric loss,” _Phys. Rev. Lett._ , vol. 95, p. 210503, Nov 2005.
  * [8] H. Paik and K. D. Osborn, “Reducing quantum-regime dielectric loss of silicon nitride for superconducting quantum circuits,” _Applied Physics Letters_ , vol. 96, no. 7, p. 072505, 2010.
  * [9] W. Woods, G. Calusine, A. Melville, A. Sevi, E. Golden, D. Kim, D. Rosenberg, J. Yoder, and W. Oliver, “Determining interface dielectric losses in superconducting coplanar-waveguide resonators,” _Phys. Rev. Applied_ , vol. 12, p. 014012, Jul 2019.
  * [10] J. M. Martinis, M. Ansmann, and J. Aumentado, “Energy decay in superconducting josephson-junction qubits from nonequilibrium quasiparticle excitations,” _Phys. Rev. Lett._ , vol. 103, p. 097002, Aug 2009.
  * [11] U. Vool, I. M. Pop, K. Sliwa, B. Abdo, C. Wang, T. Brecht, Y. Y. Gao, S. Shankar, M. Hatridge, G. Catelani, M. Mirrahimi, L. Frunzio, R. J. Schoelkopf, L. I. Glazman, and M. H. Devoret, “Non-poissonian quantum jumps of a fluxonium qubit due to quasiparticle excitations,” _Phys. Rev. Lett._ , vol. 113, p. 247001, Dec 2014. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.113.247001
  * [12] R. T. Gordon, C. E. Murray, C. Kurter, M. Sandberg, S. A. Hall, K. Balakrishnan, R. Shelby, B. Wacaser, A. A. Stabile, J. W. Sleight, M. Brink, M. B. Rothwell, K. P. Rodbell, O. Dial, and M. Steffen, “Environmental radiation impact on lifetimes and quasiparticle tunneling rates of fixed-frequency transmon qubits,” _Applied Physics Letters_ , vol. 120, no. 7, p. 074002, 2022. [Online]. Available: https://doi.org/10.1063/5.0078785
  * [13] C. D. Wilen, S. Abdullah, N. A. Kurinsky, C. Stanford, L. Cardani, G. D’Imperio, C. Tomei, L. Faoro, L. B. Ioffe, C. H. Liu, A. Opremcak, B. G. Christensen, J. L. DuBois, and R. McDermott, “Correlated charge noise and relaxation errors in superconducting qubits,” _Nature_ , vol. 594, no. 7863, pp. 369–373, Jun 2021. [Online]. Available: https://doi.org/10.1038/s41586-021-03557-5
  * [14] D. Rogovin and D. Scalapino, “Fluctuation phenomena in tunnel junctions,” _Annals of Physics_ , vol. 86, no. 1, pp. 1–90, 1974.
  * [15] M. Tinkham, _Introduction to Superconductivity_ , 2nd ed. Dover Publications, Jun. 2004. [Online]. Available: http://www.worldcat.org/isbn/0486435032
  * [16] P. J. de Visser, D. J. Goldie, P. Diener, S. Withington, J. J. A. Baselmans, and T. M. Klapwijk, “Evidence of a nonequilibrium distribution of quasiparticles in the microwave response of a superconducting aluminum resonator,” _Phys. Rev. Lett._ , vol. 112, p. 047004, Jan 2014. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.112.047004
  * [17] J. Aumentado, M. W. Keller, J. M. Martinis, and M. H. Devoret, “Nonequilibrium quasiparticles and $2e$ periodicity in single-cooper-pair transistors,” _Phys. Rev. Lett._ , vol. 92, p. 066802, Feb 2004.
  * [18] L. Sun, L. DiCarlo, M. D. Reed, G. Catelani, L. S. Bishop, D. I. Schuster, B. R. Johnson, G. A. Yang, L. Frunzio, L. Glazman, M. H. Devoret, and R. J. Schoelkopf, “Measurements of quasiparticle tunneling dynamics in a band-gap-engineered transmon qubit,” _Phys. Rev. Lett._ , vol. 108, p. 230509, Jun 2012. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.108.230509
  * [19] D. Ristè, C. C. Bultink, M. J. Tiggelman, R. N. Schouten, K. W. Lehnert, and L. DiCarlo, “Millisecond charge-parity fluctuations and induced decoherence in a superconducting transmon qubit,” _Nature Communications_ , vol. 4, no. 1, p. 1913, May 2013. [Online]. Available: https://doi.org/10.1038/ncomms2936
  * [20] K. Serniak, M. Hays, G. de Lange, S. Diamond, S. Shankar, L. D. Burkhart, L. Frunzio, M. Houzet, and M. H. Devoret, “Hot nonequilibrium quasiparticles in transmon qubits,” _Phys. Rev. Lett._ , vol. 121, p. 157701, Oct 2018. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.121.157701
  * [21] M. Bal, M. H. Ansari, J.-L. Orgiazzi, R. M. Lutchyn, and A. Lupascu, “Dynamics of parametric fluctuations induced by quasiparticle tunneling in superconducting flux qubits,” _Phys. Rev. B_ , vol. 91, p. 195434, May 2015\. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevB.91.195434
  * [22] R. Zhang, “Loss in superconducting quantum devices from non-equilibrium quasiparticles and inhomogeneity in energy gap,” Ph.D. dissertation, University of Maryland, 2020.
  * [23] K. Li, S. K. Dutta, Z. Steffen, D. Poppert, S. Keshvari, J. Bowser, B. S. Palmer, C. J. Lobb, and F. C. Wellstood, “Long-lived transmons with different electrode layouts,” _MRS Advances_ , vol. 7, no. 13, pp. 273–277, May 2022.
  * [24] T. Yamamoto, Y. Nakamura, Y. A. Pashkin, O. Astafiev, and J. S. Tsai, “Parity effect in superconducting aluminum single electron transistors with spatial gap profile controlled by film thickness,” _Applied Physics Letters_ , vol. 88, no. 21, p. 212509, 2006.
  * [25] G. Marchegiani, L. Amico, and G. Catelani, “Quasiparticles in superconducting qubits with asymmetric junctions,” 2022.
  * [26] G. Brammertz, A. Poelaert, A. A. Golubov, P. Verhoeve, A. Peacock, and H. Rogalla, “Generalized proximity effect model in superconducting bi- and trilayer films,” _Journal of Applied Physics_ , vol. 90, no. 1, pp. 355–364, 2001. [Online]. Available: https://doi.org/10.1063/1.1376411
  * [27] Y. V. Fominov and M. V. Feigel’man, “Superconductive properties of thin dirty superconductor–normal-metal bilayers,” _Phys. Rev. B_ , vol. 63, p. 094518, Feb 2001.
  * [28] S. Zhao, D. J. Goldie, C. N. Thomas, and S. Withington, “Calculation and measurement of critical temperature in thin superconducting multilayers,” _Superconductor Science and Technology_ , vol. 31, no. 10, p. 105004, aug 2018\.
  * [29] K. Serniak, “Nonequilibrium quasiparticles in superconducting qubits,” Ph.D. dissertation, Yale University, 2019.
  * [30] G. J. Dolan, “Offset masks for lift-off photoprocessing,” _Applied Physics Letters_ , vol. 31, no. 5, pp. 337–339, Sep 1977.
  * [31] M. V. Costache, G. Bridoux, I. Neumann, and S. O. Valenzuela, “Lateral metallic devices made by a multiangle shadow evaporation technique,” _Journal of Vacuum Science & Technology B_, vol. 30, no. 4, p. 04E105, 2012\.
  * [32] A. Wallraff, D. I. Schuster, A. Blais, L. Frunzio, R.-. S. Huang, J. Majer, S. Kumar, S. M. Girvin, and R. J. Schoelkopf, “Strong coupling of a single photon to a superconducting qubit using circuit quantum electrodynamics,” _Nature_ , vol. 431, no. 7005, pp. 162–167, Sep 2004.
  * [33] T. Stevenson, W.-T. Hsieh, M. Li, K. Rhee, R. Schoelkopf, C. Stahle, and J. Teufel, “Fabrication of tunnel junctions for direct detector arrays with single-electron transistor readout using electron-beam lithography,” _IEEE Transactions on Applied Superconductivity_ , vol. 13, no. 2, pp. 1139–1142, 2003.
  * [34] D. Averin and A. Bardas, “Ac josephson effect in a single quantum channel,” _Phys. Rev. Lett._ , vol. 75, pp. 1831–1834, Aug 1995.
  * [35] W.-T. Liao, “Investigation of tunneling in superconductors using a millikelvin scanning tunneling microscope,” Ph.D. dissertation, University of Maryland, 2019\.
  * [36] N. A. Court, A. J. Ferguson, and R. G. Clark, “Energy gap measurement of nanostructured aluminium thin films for single cooper-pair devices,” _Superconductor Science and Technology_ , vol. 21, no. 1, p. 015013, nov 2007\. [Online]. Available: https://dx.doi.org/10.1088/0953-2048/21/01/015013
  * [37] R. Meservey and P. M. Tedrow, “Properties of very thin aluminum films,” _Journal of Applied Physics_ , vol. 42, no. 1, pp. 51–53, 1971. [Online]. Available: https://doi.org/10.1063/1.1659648
  * [38] Z. Liu and G. Welsch, “Literature survey on diffusivities of oxygen, aluminum, and vanadium in alpha titanium, beta titanium, and in rutile,” _Metallurgical Transactions A_ , vol. 19, no. 4, pp. 1121–1125, Apr 1988\. [Online]. Available: https://doi.org/10.1007/BF02628396
  * [39] C. Verdozzi, P. Schultz, R. Wu, A. Edwards, and N. Kioussis, “Layer intermixing during metal/metal oxide adsorption: Ti/sapphire(0001),” _Phys. Rev. B_ , vol. 66, p. 125408, Sep 2002.
  * [40] C. Leyens and M. Peters, _Titanium and titanium alloys: Fundamentals and applications_. Wiley-VCH, 2005.


0 Research

# Leaning-Based Control of an Immersive-Telepresence Robot

Joona Halkola  Markku Suomalainen  Basak Sakcak  Katherine J. Mimnaugh
Juho Kalliokoski  Alexis P. Chambers  Timo Ojala and Steven M. LaValle
Center for Ubiquitous Computing, Faculty of Information Technology and
Electrical Engineering
University of Oulu, Oulu  Finland email<EMAIL_ADDRESS>
###### Abstract

In this paper, we present an implementation of a leaning-based control of a
differential drive telepresence robot and a user study in simulation, with the
goal of bringing the same functionality to a real telepresence robot. The
participants used a balance board to control the robot and viewed the virtual
environment through a head-mounted display. The main motivation for using a
balance board as the control device stems from Virtual Reality (VR) sickness;
even small movements of your own body matching the motions seen on the screen
decrease the sensory conflict between vision and vestibular organs, which lies
at the heart of most theories regarding the onset of VR sickness. To test the
hypothesis that the balance board as a control method would be less sickening
than using joysticks, we designed a user study (N=32, 15 women) in which the
participants drove a simulated differential drive robot in a virtual
environment with either a Nintendo Wii Balance Board or joysticks. However,
our pre-registered main hypotheses were not supported; the joystick did not
cause any more VR sickness on the participants than the balance board, and the
board proved to be statistically significantly more difficult to use, both
subjectively and objectively. Analyzing the open-ended questions revealed
these results to be likely connected, meaning that the difficulty of use
seemed to affect sickness; even unlimited training time before the test did
not make the use as easy as the familiar joystick. Thus, making the board
easier to use is a key to enable its potential; we present a few possibilities
towards this goal.

Telepresence robot VR sickness Balance board User comfort

##  1 Introduction

Immersive telepresence, the ability to be and feel present in a remote
location through technology, has the potential to enable hybrid meetings
(including both remote and local participants) where the remote attendees feel
like they really were there; it has been shown that with current regular
screen-based telepresence robots, the remote users operating the robot speak
less and consider group work more difficult [30]. A probable reason for this
type of (unnatural) behavior observed in remote participants is the lack of
presence [29], the feeling of ”being there”, which in virtual environments
(VEs) makes users behave more naturally [26]. It has been shown that regular,
non-immersive screens do not make users feel as present as an Head-Mounted
Display (HMD) [27]. Thus, further research on users embodying a physical robot
is warranted.

Besides a more immersive screen, another piece shown to increase the feeling
of presence is the ability to embody a robot that one can move, instead of a
stationary camera [24]. However, immersive telepresence needs more
contemplation than simply attaching a 360° camera to a mobile robot to drive
around. The user must be able to control the robot with little effort but
without succumbing to VR sickness [15]. Whereas there is a vast library of
research on locomotion in VR, the case of travelling aboard a moving ground
vehicle is often avoided; this kind of motion, especially rotations, has been
shown to significantly contribute to VR sickness [11, 12] and is often
replaced with teleportation in virtual environments. However, when controlling
physical systems, teleportation is not an option and more research is needed.

(a)

(b)

(c)

Figure 1: (a) Virtual environment used in the user study. (b) The model of the
telepresence robot’s base, into which the pole with a camera and other
equipment is attached. (c) An experimenter standing on a Wiiboard for leaning-
based control of a telepresence robot, he is slightly leaning forward.

In this paper we explore the use of a balance board, particularly the Nintendo
Wii Balance Board (Wiiboard), for leaning-based control of a telepresence
robot while being immersed in an HMD. The study is done as a simulation in a
Virtual Environment (VE) using an accurate virtual model of the mobile robot
at the research group’s disposal (Probot Dolly differential drive robot, see
Fig.1(b)). Even though we acknowledged that participants will be unfamiliar
with the Wiiboard and it might provide various physical challenges, we
hypothesized that the use of body to control the robot would make the
experience less sickening than the joystick due to the ability to mitigate
sensory mismatches. However, it turns out that we underestimated the
difficulty of using the balance board, partly based on earlier work which
reports using it as easy [7]. In our study, the joystick was found
statistically significantly easier to use than the balance board, both
subjectively and objectively. Furthermore, we found that the leaning-based
control method using Wiiboard induced more VR sickness; open-ended answers
reveal that difficulty of the Wiiboard may have also played a role in the
sickness results. We also discuss the findings about presence across control
methods.

The novelty in this paper is the development of a control mapping to drive a
telepresence robot using a balance board, and a rigorous user study done in a
VE using a virtual model of a real telepresence robot. Even though the results
were not in favor of using this particular board in the way we imagined, the
user study results show the potential of the method with better hardware and
more training time than what was allowed in this study.

##  2 Related work

There is a rising interest in HMD-based immersive telepresence, due to its
potential in enabling hybrid meetings. The motivation for HMD-based
telepresence instead of using current commercial telepresence robots, such as
the Double and Gobe, stems from the observation that participants through such
telepresence robots are much less engaged in, for example, group work [30].
This is likely related to presence, a well researched concept in VR showing
that the immersion of an HMD gives you a better sense of ”being there” than a
regular screen [27]. Thus, there is increasing interest in immersive
telepresence: recent use cases include, for example, education [5], earthquake
rescue [21] and social interaction [8].

There is a wide variety of different locomotion methods for virtual
environments, with well established advantages and disadvantages; for a
comprehensive survey see, e.g., [1]. With a scalable method also usable in
smaller spaces in mind when considering choices for HMD-based control of a
telepresence robot, we decided not to consider walking (redirected or regular)
but instead opted for leaning-based methods, keeping in mind the popular
Segway robots, which the leaning-based methods mimic (even though the
experience without actual motions is quite different). There are various
methods for implementing leaning-based navigation for VR; by harnessing the
user, methods such as the human joystick [19] can allow leaning that properly
resembles falling. Another popular method is simply tracking the HMD to detect
leaning, or stepping, with different tactile feedback from the ground to help
the user [6, 22], which has been shown to increase presence and decrease VR
sickness over joystick; similar methods have also been shown to work while
sitting [10].

Of the works using Wiiboard, de Haan et al. [7] described the technical
methods and positive initial user feedback for using Wiiboard for locomotion
in VR. Valkoc et al. [32] then used Wiiboard as a component in their
navigation system, however, they did not compare Wiiboard to another control
method. Thus, especially with the symmetries between Wiiboard, Segway, and
Differential Drive Robot (DDR) (rotation is performed with different speeds on
each wheel, which can be controlled with the pressure on each foot), we
believe it is necessary to properly test a balance board in this use;
comparing with the stepping motion, we hypothesize that small adjustments to
the robot’s direction would be easier to do with just weight adjustment on the
board when compared to trying to make the same adjustment in the head-tracking
with the whole body, as in [6, 22]. However, there is little previous work on
VR sickness using a balance board, for either virtual environments or
immersive telepresence; regardless of the vast literature on VR sickness in
virtual environments (see, e.g., [25]), in such ground-based motions VR
sickness is often mitigated by teleportation, which is infeasible for
telepresence. For autonomous motions of a telepresence robot with an HMD,
several considerations were given to VR sickness in [2, 31]; however, there
was no consideration on the actual control of the robot, since users often
want to control the robot also manually, especially for short distances.

In using a balance board for controlling a robot, there are both potential
mitigating and escalating factors for VR sickness. From one point of view,
having to move yourself may mitigate sensory mismatch between the visual
system and the vestibular organs, which should prevent VR sickness. However,
another issue to consider is the postural stability, deterioration of which
has been shown to have the potential to predict the onset of VR sickness [17],
even with Wiiboard as the measurement device when watching a movie through the
HMD [4]. However, the order of cause and consequence is still complex [17];
thus, it is difficult to predict whether having to balance on purpose can have
a causal effect on VR sickness, and whether this has a stronger effect on VR
sickness than the decrease of the sensory mismatch.

##  3 System

The system comprised a VE (see Sec. 5.2 for a more detailed description)
including a simulated robot (Fig.1(b)) moving within the environment A virtual
360° camera is attached 1.5 meters above the robot base, from which the user
sees the virtual world through an HMD; this is a height suggested by [14] for
360° videos. The position of the camera is fixed with respect to the robot
while allowing 3 Degrees of Freedom (DOF) in rotation as determined by the
headset tracking. The robot can translate and rotate in a two-dimensional
plane determined by the user input. In total, the robotic system has 5DOF
since the camera height is fixed. The robot motion was controlled directly by
the users through the inputs given using a Wiiboard or the joysticks on the
Oculus Quest 2 controllers. The user input was mapped to reference angular
velocities for the robot wheels. The maximum attainable wheel angular velocity
was set to approximately $573^{\circ}$ per second to limit the robot forward
speed to a maximum of $0.75m/s$. The reason for this choice was motivated by
[20] presenting that $1m/s$ is a suitable speed; however, we used a slightly
lower value compared to precedents to increase comfort in controlling the
robot using a device (Wiiboard) that most participants would be unfamiliar
with.

###  3.1 Wiiboard control method

The Wiiboard has four pressure sensors, one at each corner of the board, and
communicates with a computer via Bluetooth (see Fig. 2). We used the WiiBuddy
Unity Asset of BitLegit [3] to establish the connection between the Wiiboard
and the PC together with [18] to stabilize the connection.

Forward movement with Wiiboard was achieved by leaning forward, and turning
while moving forward by leaning towards either of the front corners. The
initial implementation followed the guidelines of [7]. However, due to the
observed difficulty during initial tests of backward motion (by putting
pressure on both heels) and rotation in place (by putting pressure on the heel
with one foot and toe on the other), we decided for a simpler implementation,
with further motivation being the low probability of needing backward motion
for a telepresence robot. Thus, we implemented rotation in place by putting
weight towards one of the back corners, which was reasonable to achieve by
turning the upper body towards the desired corner, similarly to reaching for
an object behind you.

Figure 2: The Nintendo Wii Balance Board with the four force sensors at the
corners marked.

The sensor readings, corresponding to the applied pressure on each corner,
were used to calculate the two components that constitute the total motion: go
forward ($fwd$) and turn ($turn$). These were calculated according to the
equations, in which $F_{\cdot}$ is the force measured by the sensor at the
respective corner, (see Fig. 2 for the meaning of each index term)

$\displaystyle fwd$
$\displaystyle=\max\left(\frac{(F_{TL}+F_{TR})-(F_{BL}+F_{BR})}{(F_{TL}+F_{TR})+(F_{BL}+F_{BR})},0\right)$
(1) $\displaystyle turn$ $\displaystyle=\frac{F_{TR}-F_{TL}}{F_{TR}+F_{TL}}.$

Note that $fwd$ and $turn$ lie within the ranges $[0,1]$ and $[-1,1]$,
respectively. In case of a collision with the environment, the forward motion
was disabled and the user was expected to rotate in place until the robot was
free to move forward. In that case, the $turn$ component was calculated as

$turn=\frac{F_{BR}-F_{BL}}{F_{BR}+F_{BL}}$ (2)

and $fwd$ was set to $0$. Respective reference wheel rotational velocities
were calculated as

$\omega_{L}=c_{f}fwd+c_{t}\,turn,\;\omega_{R}=c_{f}fwd-c_{t}\,turn$ (3)

in which $\omega_{L}$ and $\omega_{R}$ are the reference velocities for the
left and right wheels, and $c_{f}$ and $c_{t}$ are the respective weights. The
weights were selected considering limits on forward and turning speeds and
ensuring that $\omega_{L},\omega_{R}\leq\omega_{max}$. In case of rotate in
place motion, used for getting out of a collision state,
$\omega_{max}=75deg/s$ was used to avoid uncomfortably fast rotations. This
value was selected among three candidates ($132deg/s$, $103deg/s$, $75deg/s$)
by testing it on six people.

For each user, the Wiiboard was calibrated to account for varying weight
distribution patterns during standing still and leaning. This way, the mapping
from sensor readings to robot controls was made to adjust Wiiboard for each
user individually. In the calibration process, values from the sensors are
measured for six different cases: standing still, leaning forward, and leaning
towards each of the four corners. Measurements corresponding to standing still
are used as offsets to ensure that the robot does not move at this posture
since $fwd$ and $turn$ are zero only when the weight is equally distributed
(see Eqs (1) and (2)). The rest of the cases were used to find maximum values
achieved for $fwd$ and $turn$ when the participants leaned _as much as they
felt comfortable_. These values are then mapped to the $[-1,1]$ interval so
that when the participant was at an extreme posture the robot speed was the
highest. Since most users leaned too much when calibrating (even when told to
“lean as much as you feel comfortable”), sensor readings corresponding to
maximum speeds were calculated by taking 60% of the calibration values.

###  3.2 Joystick control method

Two joysticks on the Oculus Quest 2 controllers were used as an alternative to
contrast the Wiiboard control method. Whereas one joystick was used to control
the forward speed ($fwd$), the other one was used to control the turning speed
and direction ($turn$). The joystick position on the horizontal axis (moving
the joystick left-right) was mapped to intervals $[0,1]$ and $[-1,1]$
corresponding to $fwd$ and $turn$, respectively. Robot motion is achieved by
tracking the reference wheel rotational speeds, calculated in terms of $fwd$
and $turn$ using Eq.(3). Participants were standing still when using the
joystick in the study.

##  4 Hypotheses

We pre-registered the following three hypotheses, together with the procedure
and analyses to be used in the study, in Open Science Foundation (OSF)111
https://osf.io/6rxby/?view_only=9990df8e0c104ac9b4bbc03ac72515a8. From now on
we will refer to the  Wii Balance Board control method (WB) condition as WB
and  Joystick control method (JS) condition as JS.

  1. H1:

Less VR sickness in WB condition as indicated by lower total weighted
Simulator Sickness Questionnaire (SSQ) score.

  2. H2:

No difference in how far the participants get during the four minutes driving
time and no difference in the number of crash events.

  3. H3:

No difference in cognitive load as indicated by no difference in NASA-TLX
scores in any dimension except the physical load.

We hypothesized H1 based on the evidence that incorporating some form of
vestibular stimulation can potentially mitigate sensory conflict in VR, as
described in Section 2. Even though there is a slight mismatch between the
vestibular and visual stimuli (such as forward-leaning being circular and not
linear motion, and the rotations providing only an impulse of the motion that
then further carries on visually in VR while the vestibular stimulation from
the bodily motion ends), we believe that some amount of vestibular stimuli
should ameliorate VR sickness as even random stimulation of the vestibular
organs has been shown to help [33]. H2 stems from the work of de Haan el al.
[7], who reported subjects found the Wiiboard to be easy to use; however, due
to our expectation that our sample would have some familiarity with using a
joystick, we predicted that there would not be any difference in performance.
We postulated H3 for similar reasons; the Wiiboard clearly needs more physical
effort, but otherwise, based on the ease of use reported in [7], we expected
there would be no difference in cognitive load.

##  5 User Study

###  5.1 Procedure

Two conditions with two different starting positions (either ends of the path
shown in Fig. 3) were tried by the participants in a counterbalanced order
such that each pair of starting position and control method was experienced as
first and second equal number of times. The participant was welcomed by a
researcher upon arrival and asked to sign a consent form. To pre-screen
already sick-feeling people, the participants were asked if they felt nauseous
or had a headache, and asked to reschedule if they did. After this, they were
instructed to take their shoes off, stand on the Wiiboard, and place their
feet in the middle of the textured areas of the Wiiboard (see Fig. 2).

Next, the researcher gave instructions about the practice session to the
participant; if the control method was the Wiiboard, the participant was first
shown an instruction video on how to calibrate the balance board, after which
the researcher instructed the participant how to put on the HMD and told the
participant to complete the calibration. Next, the participant was told to
take off the HMD and a second instruction video was shown to the participant
which explained how to use the Wiiboard; this was done because the calibration
was better to do with the HMD on to get realistic values to use with the HMD,
but practicing with the HMD on would likely have caused too much VR sickness
on the participants. If the control method was the joysticks, the researcher
read out instructions on how to use the joysticks to move the robot. Finally,
the researcher read out the instructions for the practice session, which were
identical for both control methods: the participant was told to complete the
practice session without the HMD and follow the robot movement from a monitor
in front of them. The participant was specifically instructed to crash into a
wall to test out the turn-in-place feature and learn how to recover from a
collision. After the practice, the participant was asked how confident they
felt using the control method on a scale from 1 to 7 and the researcher marked
down the answer. Then, the participant was told to put back on the HMD, follow
the direction of the white line on the ground and count the sailboat paintings
on the walls.

After the participant completed the run, they were told to take off the HMD
and controllers and fill out questionnaires on a laptop, after which the same
procedure was repeated for the second control method. At the end, the
participant was rewarded with a 12€ gift voucher to Amazon and given a short
debrief about the study.

###  5.2 Virtual environment and tasks

VE for the study was loosely based on parts of the local university; the VE
and the path in the environment that the participants were asked to follow can
be seen in Fig. 3. Each participant ran into opposite direction on the second
attempt (combinations of directions and conditions were also counterbalanced).
The environment had five differently colored lines on the floor as shown in
Fig. 1(b), from which the participants were told to follow the white line
during the test. This served as a sidetask to test whether the participants
could still pay attention to the environment while commanding the robot.
Another sidetask was the counting of six sailboat paintings scattered around
the VE. The training area was an empty hall with a line drawn on the floor to
work as reference.

Figure 3: Birds-eye view of the path (white line) the participants should
follow in the main task. The other lines have been removed for clarity.

A basic collision-avoidance method is implemented that stops the robot if it
is too close to an obstacle (a wall or an object). In case of such an event,
the forward motion was disabled and only turn-in-place motion was allowed,
until the robot was heading sufficiently away from the obstacle. The system
measures the total amount of time spent being stuck in collision states.

###  5.3 Participants

Sample size for the study was 32 participants and we aimed at having an equal
split of male and female participants. However, due to difficulties of getting
enough female participants we ended up with 17 male and 15 female
participants. All participants reported having normal or corrected-to-normal
vision and none of the participants were colorblind.

VR system usage reported by the participants was: $12.5\%$ never, $40.6\%$
once or just a couple of times, $25.0\%$ once or twice a year, $15.6\%$ once
or twice a month, $3.1\%$ once or twice a week, and $3.1\%$ every day.
Responses to how often they play computer games were: $12.5\%$ never, $15.6\%$
once or just a couple of times, $9.4\%$ once or twice a year, $18.8\%$ once or
twice a month, $21.9\%$ once or twice a week, $12.5\%$ several times a week,
and $9.4\%$ every day.

Initially, we pre-registered that we would exclude all participants who got
stuck against the walls for more than 30 seconds; the time limit was based on
the same initial test that we used to choose the rotation speed. However, the
number of participants who exceeded this limit were higher than what we
expected. Due to the difficulties in recruiting participants, we decided to
relax this limit and used also the data corresponding to the participants who
were stuck for more than $30$ seconds but otherwise completed the required
tasks (4 males, 3 females among 32 participants). The pre-registered
hypotheses were run both with all participants ($N=32$) and the subset who did
not get stuck for more than $30s$ ($N=25$). On average, participants were
stuck for $21.05s$ in WB condition and $0.87s$ in JS condition for N=32;
averages were $7.39s$ in WB condition and $0.32s$ in JS condition for N=25.
Each session lasted exactly $4$ minutes.

###  5.4 Measures

Two sets of questionnaires were presented to the participant after finishing
each of the sessions. The latter part of the second questionnaire contained
post-experiment questions. Both questionnaires started with SSQ [13] and SSQ
total score was used to measure sickness. SSQ is an established questionnaire
for measuring sickness in VR by presenting 16 possible sickness symptoms,
which the participants gauge on a scale none (0) to severe (3). The SSQ total
score is calculated by weighting the answers for a maximum score of 236.
Higher scores indicate greater levels of sickness experienced. Consequently,
NASA Task Load Index (NASA-TLX) questionnaire [9] was administered, which is
used to measure six dimensions of workload (mental demand, physical demand,
temporal demand, performance, effort, and frustration) of the task. Each
dimension was rated on a scale of 1 to 20 and higher values indicate that the
task is more demanding in that aspect. Additionally, we used 7-point Likert-
scale questions, forced-choice questions comparing the two methods, and open-
ended questions about reasons for some choices and demographic questions.

To measure the participants’ confidence in using each control method we asked
how confident the participant felt (on a 7-point Likert-scale) after each
training session about the condition that they would try next. Their answer
was then registered by the experimenter.

For each session, the path executed by the robot was recorded in world
coordinates and how far along the reference path (see Fig. 3 for the reference
path) the participant got was calculated projecting the final robot position
onto the reference path. We also recorded the number of times the robot
collided with the obstacles (including walls) and the total length of time
spent in a collision-state.

##  6 Results

All statistical tests were run in SPSS with significance levels set to $0.05$
and with a $95\%$ confidence interval.

###  6.1 Confirmatory results

All confirmatory analyses were performed first using the whole dataset
$(N=32)$ and then using only the data corresponding to the participants who
were not in a collision-state for longer than $30s$.

Less VR sickness in JS condition (H1 rejected) A Wilcoxon Signed-Ranks test
(two sided) was performed to compare the differences between the total
weighted SSQ scores for JS (${Mean}=32.02$) and WB (${Mean}=40.44$) conditions
$(N=32)$. The test indicated that JS elicited significantly lower SSQ scores
compared to WB, $Z=-2.32$, $p=.02$, $r=0.41$. We also checked for potential
effect of getting stuck for more than $30s$ on sickness and run the test again
using only the participants who did not get stuck or got stuck less than $30s$
$(N=25)$. We did not observe any significant difference between the total
weighted SSQ scores for JS $(Mean=29.92)$ and WB $(Mean=37.84)$ conditions
when participants who got stuck were excluded from the analysis, as indicated
by a Wilcoxon Signed-Ranks test (two sided), $Z=-1.84$, $p=.066$, $r=0.369$.

Participants reached farther along the path in JS condition (H2 rejected) A
paired t-test was run to determine whether in one condition the participants
reached farther along the path $(N=32)$. The total path length was $186.6$
meters and none of the participants reached the end (we ensured this by
selecting the path length and the time limit on each session so that all
participants have equal exposure to the VE). The distances in both conditions
were normally distributed, as indicated by a Shapiro-Wilk test,
$W(32)=0.956,p=.214$ for JS, $W(32)=0.966$, $p=.398$ for WB, and there were no
outliers in the data, as assessed by inspection of the boxplots. The mean
distance was higher in JS condition $(162.39\pm 13.52)$ compared to in WB
condition $(124.86\pm 26.53)$; a statistically significant increase of $37.54$
($95\%$ CI, $30.30$ to $44.78$), $t(31)=10.58$, $p=.00$. We removed the data
corresponding to participants who were stuck, and ran the same test again. The
distances conformed to a normal distribution in both conditions as indicated
by a Shapiro-Wilk test ($W(25)=0.944$, $p=.182$ for JS and $W(25)=0.954$,
$p=.301$ for WB) and there were no outliers. A paired samples t-test indicated
that there was a statistically significant increase in the distances that the
participants reached within a given time frame in JS condition from $132.83\pm
23.45$ to $165.17\pm 11.93$; an increase of $32.33$ ($95\%$ CI, $24.73$ to
$39.94$), $t(24)=8.776$, $p=.00$.

Less collisions in JS condition (H2 rejected) The number of crash events in JS
condition $(m=0.4)$ were lower than the number of crash events in WB condition
$(m=1.4)$ for $N=32$, this decrease was statistically significant as indicated
by a Wilcoxon Signed-Ranks test (two sided), $Z=-3.477$, $p=.001$, $r=0.615$.
This tendency in having less collisions persisted also when the data
corresponding to the people who got stuck in a collision-state for longer than
$30s$ was removed $N=25$. There were significantly less crash events in JS
condition $(m=0.4)$ as opposed to in WB condition $(m=1.2)$, as indicated by a
Wilcoxon Signed-Ranks test (two sided), $Z=-2.862$, $p=.004$, $r=0.572$.

Higher perceived workload in WB condition (H3 rejected) For each subscale of
NASA-TLX, a Wilcoxon Signed-Ranks test (two sided) was performed to compare
the ratings for JS and WB conditions considering both the data with and
without exclusions (see Table 1 for respective means and the corresponding
standardized test statistics, significance values, and effect sizes). With
$N=32$ there was a statistically significant increase in the TLX scores in all
dimensions other than performance. When the same test was performed with
$N=25$, we observed that TLX scores were higher in all dimensions except
performance and temporal demand.

TLX subscale | Means (N=32) | Test summary (N=32) | Means (N=25) | Test summary (N=25)
---|---|---|---|---
JS | WB | JS | WB
Mental demand | 7.88 | 11.22 |  $Z=-3.73$, $p=.00$, $r=0.659$ | 7.28 | 10.56 |  $Z=-3.47$, $p=.001$, $r=0.694$
Physical demand | 4.035 | 11.09 |  $Z=-4.63$, $p=.00$, $r=0.819$ | 4.96 | 6.48 |  $Z=-4.02$, $p=.00$, $r=0.804$
Temporal demand | 4.5 | 6.81 |  $Z=-2.64$, $p=.008$, $r=0.467$ | 4.08 | 10.28 |  $Z=-1.82$, $p=.069$, $r=0.364$
Performance | 6.31 | 7.81 |  $Z=-1.40$, $p=.16$, $r=0.248$ | 6.44 | 7.60 |  $Z=-1.02$, $p=.307$, $r=0.204$
Effort | 5.9 | 11.62 |  $Z=-4.55$, $p=.00$, $r=0.804$ | 5.6 | 11.12 |  $Z=-3.96$, $p=.00$, $r=0.792$
Frustration | 3.78 | 8.22 |  $Z=-4.38$, $p=.00$, $r=0.774$ | 3.48 | 7.88 |  $Z=-3.93$, $p=.00$, $r=0.786$
Table 1: Means and the respective results of a Wilcoxon Signed-Ranks test
(two-sided) for the NASA-TLX subscales corresponding to $N=32$ and $N=25$
datasets.

###  6.2 Exploratory results

We performed an exploratory analysis of all $32$ participants to get a deeper
insight into our results.

####  6.2.1 Quantitative Data

Participants felt more confident in using the JS condition To measure
participants’ confidence in using either control method, we asked them to rank
their confidence in Likert-scale $(1-7)$ after each training session. People
felt statistically significantly more confident in using JS $(Mean=6.25)$ as
opposed to WB condition, as indicated by a Wilcoxon Signed-Ranks test (two-
sided), $Z=4.478$, $p=.00$, $r=0.79$.

JS condition was found easier to use We measured the relative ease-of-use by
explicitly asking: Which control method was easier to use? $30$ out of $32$
participants $(94\%)$ found JS condition easier to use. This bias towards JS
condition was statistically significant in an exact binomial test with exact
Clopper-Pearson $95\%$ CI and had a $95\%$ CI of $79.2\%$ to $99.2\%$, $p=.00$
(two-sided). In addition to the forced-choice question we also asked 7-point
Likert-scale questions to measure ease-of-use. Comparing the ratings in JS
condition $(Mean=6.03)$ with the ones in WB condition $(Mean=3.59)$, we found
that JS elicited a statistically significant increase in the comfort rankings,
as indicated by a Wilcoxon Signed-Ranks test (two sided), $Z=4.783$, $p=.00$,
$r=0.846$.

JS condition was preferred When asked explicitly which condition did the
participants prefer, $26$ out of $32$ participants $(81\%)$ picked JS
condition. An exact binomial test with exact Clopper-Pearson $95\%$ CI
indicated that this bias towards JS condition was statistically significant
and had a $95\%$ CI of $63.6\%$ to $92.8\%$, $p=.001$ (two-sided).

WB condition did not increase presence When asked “Thinking back to both of
the experiences, which one gave a better sense of being in the robot’s
location?” $19$ participants $(59\%)$ picked the WB condition, though this
bias towards WB condition was not statistically significant in an exact
binomial test (two-sided), $p=.377$.

(a) Ease-of-use

(b) Preference

Figure 4: Frequent codes found in open-ended questions regarding ease-of-use
and preference together with their occurrences.

####  6.2.2 Qualitative data

The open-ended data was analyzed using the thematic analysis method with
inductive approach [23]. Fig. 4(a) presents the frequent codes found in the
data related to ease-of-use, divided by which condition was found easier to
use. These codes are then grouped into three themes: expectations and previous
experience, less (physical) effort, and control. The first theme refers to the
previous experience of people in terms of moving in VR or in the real world
and their expectations based on that. It encompasses the codes _realistic_ ,
_learning curve_ , and _familiarity_ containing 11 comments in total. Codes
_less (physical) effort_ ($12$ occurrences) and _control_ ($7$ occurrences)
are themes of their own referring to less required physical effort to use a
control method and the sense of control over the robot, respectively. The
majority of people who found JS condition easier thought so because it
required _less physical effort_ (“It only required the movements of my thumbs,
no more was required.”). This is followed by reasons related to their
expectations and previous experience with VR. In particular, they felt JS was
familiar and that it was easier to learn (coded as _learning curve_). Finally,
some people thought JS was easier to use because they had more _control_ over
the robot with this method (“You feel more in control with this method”).
Responses given by the two participants who found WB easier stemmed from their
expectations and previous experience; they thought it was more _realistic_ ,
that is, similar to moving in the real world (“Personal choice of use by feet
than hand, more realistic”).

Codes and their frequencies related to the data on reasons for participants’
preferences can be found in Fig. 4(b). The codes are grouped under the themes:
bodily feelings encompassing _less physical demand_ and _less sickening_
containing 11 comments, expectations and previous experience encompassing
_realistic_ and _familiarity_ containing 7 comments, emotions encompassing
_fear_ and _exciting_ containing 9 comments, and finally _easier_ being an
important theme of its own. The most popular reason given by the participants
who preferred JS condition was because it was _easier_ to use ($19$
occurrences in JS condition, for example, “I didn’t have to focus as much on
how to use the control method on the first one…”). This was followed by the
reasoning that fell under the theme bodily feelings, meaning that it induced
less negative physiological response or that it was physically less demanding
(“…The second one caused more motion sickness-like symptoms.”). Some people
who preferred JS thought that it made them feel better emotionally (falling
under the theme _emotions_), compared to WB, especially since they did not
experience a fear of falling down (“I didn’t feel like falling”). People who
preferred the WB condition named the biggest factor as it being _realistic_
based on their expectations and previous experience (for example, ”Despite
being more physically challenging it gives more movement and feels almost like
real movement.”). Other reasons stated fell under having better _emotions_
(“It was cool and made me feel more in the place.”) and having less bad
_bodily feelings_ (“And I also felt really nauseous doing the task with the
controllers but not nauseous at all doing it with the balance board”) .

##  7 Discussion

###  7.1 Original hypotheses

On average, JS condition induced less VR sickness measured using the SSQ total
score. Thus, refusing H1. The answers to open-ended questions reveal
interesting insights and possible reasons for this result on VR sickness.
Moreover, in the open-ended questions the _less sickening_ code was used as a
reason by 5 participants who preferred the JS and only once by who preferred
the WB, hinting towards participants suffering more from VR sickness while
using the board than with the joystick. Similarly, H2 and H3 were rejected
too; participants reached farther along the path while using the joystick as
opposed to the Wiiboard, and the NASA-TLX scores showed that the participants
perceived controlling the robot using joysticks less demanding.

Further examination of the answers to the open-ended questions reveals a
potential relation between perceived difficulty and VR sickness. Two of the
participants who preferred the JS condition because it induced less sickness
stated that “Not having to look at floor while leaning prevents nausea” and “I
didn’t have to focus as much on how to use the control method on the first
one. The second one caused more motion sickness-like symptoms.” Since staring
at the floor would result in more optical flow, and thus likely induce more VR
sickness, if participants looked at the floor more with the Wiiboard, it may
likely have increased their VR sickness levels.

A potential relation between difficulty and VR sickness was further indicated
by looking at the confidence scores of the participants who commented their
respective method to be less sickening. The participant who found the Wiiboard
less sickening gave a confidence score of 7 (on a 7-point Likert-scale) for
using the WB, whereas the average of the confidence scores given by the 5
people who found WB more sickening was $5.2$. Similarly, one participant who
mentioned that the WB condition was less sickening rated its ease-of-use as
$5$ (out of 7), whereas the average of the ease-of-use ratings given by the 5
participants who found the board more sickening was $2.6$. Despite being based
on a small sample size, these insights further strengthen the implication of
difficulty causing more VR sickness. The other possible cause for increased VR
sickness with the board is the postural instability caused by the control
motions, even though, as mentioned, the order of cause and consequence is
unclear [17] and not researched when leaning is used for control.

Interestingly, earlier papers [7] described the board feeling “easy” when used
among the researchers and colleagues. Similarly, all our pre-pilot users did
prefer the board and effectively used it to control the robot. However, during
the study, regardless of the unlimited training time, the participants still
perceived using board as more difficult ($Mean=3.59$ on a 7-point Likert-
scale). We expected that participants would find using the board more
difficult compared to joystick; this was in part due to the well known result
in 3D user interfaces such that that using a high DOF input to control a lower
DOF system is typically challenging [16], and in part due to the familiarity
of the general population with the joystick. However, we did not expect to see
a difference in the perceived difficulty of using the Wiiboard to the extent
that we observed (statistically significant difference in favor of JS in
almost all dimensions of the NASA-TLX). Besides VR sickness, the difficulty
had a major impact on preference: 20 out of 32 participants, stated ”easiness”
as their reason for preferring a particular method, among which 1 person
stated it in preference of the Wiiboard and 19 in favor of the joystick.
Additionally, when asked why one method felt easier than the other, three
people specifically noted the steep learning curve for the Wiiboard (“The
second method is a bit harder to get used to”) and six people mentioned that
their familiarity with joysticks may have helped (“Joysticks are a familiar
method and very intuitive too. Balancing takes more effort”).

###  7.2 Exploratory data

Chen et al. [6] found, in an old study, that stepping-based (head-tracked,
with steps taken at each direction to move towards that direction) navigation
caused less VR sickness and induced a (moderately) greater feeling of presence
than a joystick. However, there are several differences between their study
and ours. First, Chen et al. considered 6D-navigation without physical limits.
Moreover, all of their 20 participants were experienced with VR, and it was
not explained how much prior experience they had with the system; also, they
used a different kind of joysticks. Thus, even though there is evidence that
self-rotation, easily achieved by the stepping method, reduces VR sickness and
increases the feeling of presence and spatial awareness [22], it is not clear
which method in general would be more useful for telepresence robot
navigation. A method combining leaning and self-rotations could probably
provide best of both worlds.

We decided to make participants undergo a training session without the HMD. We
wanted to give the participants as much time as they needed and did not want
to limit the training duration because we expected that they would not be
familiar with the board and that using the board would employ a steeper
learning curve. Our expectations were supported by the participants’ comments;
for example, “More exciting and would “become” easier after many practices.”,
“I’m used to moving in VR with joysticks, the balance board was a bit harder
to get used to especially because the HMD made it a bit harder to balance
myself.”. However, a longer training with the HMD on could have caused
excessive sickness and potential differences in training times could have
corrupted the SSQ data. Despite mitigating these potential issues, practicing
without the HMD caused a shortcoming in our approach, mentioned also in the
comment above, that is, using WB with the HMD was slightly different than
using it without. It would be interesting to perform a smaller-sample
qualitative long-term study with the board, where we give participants ample
time to practice on the board on the scale of days or weeks, and then compare
the differences and SSQ scores; we suspect that this kind of study would
better bring out the strengths of the leaning-based control using a balance
board.

We also wondered whether the body-based locomotion would make participants
feel more present; whereas it is well established that having an actual
virtual body that tracks your motions increases presence [26], there is, to
the knowledge of the authors, no clear evidence whether having “a bit more
realistic” body-based control should increase the feeling of presence (in [28]
simply moving the body increased presence, but the comparison was standing
still, not moving via a joystick or similar). There were hints towards the
board making participants feel more present; four participants preferred the
Wiiboard because it felt more _realistic_ (”It was cool and made me feel more
in the place. Although it was easier with the first method and I think that I
also accomplished the task better with the first method (like no bumping into
the wall).”), and some participants who preferred joystick at the end still
stated the positives of the Wiiboard towards that direction (“Because I could
accomplish the task with the first one and with the second I got stuck.
However, the second one was more fun and I felt more engaged.”). However, the
forced-choice question about presence was not significant (even if leaning
towards that direction), and the SUS showed no significant difference either.
Nonetheless, there seems potential for more studies on presence with body-
based locomotion.

###  7.3 Limitations and future work

A clear limitation of the study was the hardware: the Wiiboard is an old
technology and has limited sensors, which are not extremely accurate. This
could have been a reason for nine people mentioning better control of the
robot with the joystick as a reason for finding it easier. Also, altogether 5
participants reported fear of falling off the board as a reason for preferring
the joystick; it seems that a thin mattress, or carpet, with sensors would be
better to alleviate this fear. Additionally, such a system would allow
rotating the robot also via rotating the whole body, which has been shown
useful in reducing VR sickness and enhancing spatial awareness [22]. Such a
combination of the strengths of the methods could provide interesting results.

These results could also be used as a stepping stone to find out the
relationship between postural instability and VR sickness; does active
destabilization of the posture, in the form of leaning, contribute to VR
sickness? With a more accurate pressure mat, we may be able to differentiate
between wanted and unwanted postural sway, and correlate them with the
(perhaps continuously measured) VR sickness and HMD tracking data;
additionally, if the pressure sensor was used to measure the postural
instability also when controlling via the joystick, this comparison could
reveal interesting facets about wanted and unwanted postural instability.

##  8 Conclusion

The study presented in this paper reveals that leaning-based methods on a
balance board have potential, but the unfamiliarity and difficulty of the
Wiiboard prevented the possible positive effects of reducing VR sickness. We
identified several reasons making the board so difficult, such as not enough
responsiveness, steep learning curve requiring more training than a few
minutes before the actual study, training with the headset on even with the
chance of inducing VR sickness during training, and the height of the board
causing a fear of falling. A more responsive and larger or flat board, which
could also allow self-rotations, could be tested for chances of improvement.

###### Acknowledgements.

This work was supported by a European Research Council Advanced Grant (ERC
AdG, ILLUSIVE: Foundations of Perception Engineering, 101020977), Academy of
Finland (projects PERCEPT 322637, CHiMP 342556), and Business Finland (project
HUMOR 3656/31/2019).

## References

  * [1] M. Al Zayer, P. MacNeilage, and E. Folmer.  Virtual locomotion: a survey.  IEEE transactions on visualization and computer graphics, 26(6):2315–2334, 2018.
  * [2] I. Becerra, M. Suomalainen, E. Lozano, K. J. Mimnaugh, R. Murrieta-Cid, and S. M. LaValle.  Human perception-optimized planning for comfortable vr-based telepresence.  IEEE Robotics and Automation Letters, 5(4):6489–6496, 2020.
  * [3] BitLegit.  WiiBuddy API.  [Accessed Dec. 3, 2019].
  * [4] J. E. Bos, W. D. Ledegang, A. J. Lubeck, and J. F. Stins.  Cinerama sickness and postural instability.  Ergonomics, 56(9):1430–1436, 2013.
  * [5] J. Botev and F. J. Rodríguez Lera.  Immersive robotic telepresence for remote educational scenarios.  Sustainability, 13(9):4717, 2021.
  * [6] W. Chen, A. Plancoulaine, N. Férey, D. Touraine, J. Nelson, and P. Bourdot.  6dof navigation in virtual worlds: comparison of joystick-based and head-controlled paradigms.  In Proceedings of the 19th ACM Symposium on Virtual Reality Software and Technology, pp. 111–114, 2013.
  * [7] G. de Haan, E. J. Griffith, and F. H. Post.  Using the wii balance board™ as a low-cost vr interaction device.  In Proceedings of the 2008 ACM symposium on Virtual reality software and technology, pp. 289–290, 2008.
  * [8] J. Du, H. M. Do, and W. Sheng.  Human–robot collaborative control in a virtual-reality-based telepresence system.  International Journal of Social Robotics, pp. 1–12, 2020.
  * [9] S. G. Hart.  Nasa-task load index (nasa-tlx); 20 years later.  Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9):904–908, 2006. doi: 10 . 1177/154193120605000909
  * [10] A. M. Hashemian, A. Adhikari, E. Kruijff, M. Von der Heyde, and B. E. Riecke.  Leaning-based interfaces improve ground-based vr locomotion in reach-the-target, follow-the-path, and racing tasks.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [11] S. Hu, K. A. McChesney, K. A. Player, A. M. Bahl, J. B. Buchanan, and J. E. Scozzafava.  Systematic investigation of physiological correlates of motion sickness induced by viewing an optokinetic rotating drum.  Aviation, space, and environmental medicine, 1999.
  * [12] A. Kemeny, P. George, F. Mérienne, and F. Colombet.  New VR navigation techniques to reduce cybersickness.  Electronic Imaging, 2017(3):48–53, 2017.
  * [13] R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal.  Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness.  The international journal of aviation psychology, 3(3):203–220, 1993.
  * [14] T. Keskinen, V. Mäkelä, P. Kallioniemi, J. Hakulinen, J. Karhu, K. Ronkainen, J. Mäkelä, and M. Turunen.  The effect of camera height, actor behavior, and viewer position on the user experience of 360 videos.  In 2019 IEEE conference on virtual reality and 3D user interfaces (VR), pp. 423–430. IEEE, 2019.
  * [15] J. J. LaViola Jr.  A discussion of cybersickness in virtual environments.  ACM Sigchi Bulletin, 32(1):47–56, 2000.
  * [16] J. J. LaViola Jr, E. Kruijff, R. P. McMahan, D. Bowman, and I. P. Poupyrev.  3D user interfaces: theory and practice.  Addison-Wesley Professional, 2017.
  * [17] S. Litleskare.  The relationship between postural stability and cybersickness: It’s complicated–an experimental trial assessing practical implications of cybersickness etiology.  Physiology & Behavior, 236:113422, 2021.
  * [18] R. L. Lynch.  Permanently Pair Wiimote/Wii Balance Board.  [Accessed Dec. 16, 2019].
  * [19] M. Marchal, J. Pettré, and A. Lécuyer.  Joyman: A human-scale joystick for navigating in virtual worlds.  In 2011 IEEE Symposium on 3D User Interfaces (3DUI), pp. 19–26. IEEE, 2011.
  * [20] K. J. Mimnaugh, M. Suomalainen, I. Becerra, E. Lozano, R. Murrieta-Cid, and S. M. LaValle.  Analysis of user preferences for robot motions in immersive telepresence.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4252–4259. IEEE.
  * [21] F. Negrello, A. Settimi, D. Caporale, G. Lentini, M. Poggiani, D. Kanoulas, L. Muratore, E. Luberto, G. Santaera, L. Ciarleglio, et al.  Humanoids at work: The walk-man robot in a postearthquake scenario.  IEEE Robotics & Automation Magazine, 25(3):8–22, 2018.
  * [22] T. Nguyen-Vo, B. E. Riecke, W. Stuerzlinger, D.-M. Pham, and E. Kruijff.  Naviboard and navichair: Limited translation combined with full rotation for efficient virtual locomotion.  IEEE transactions on visualization and computer graphics, 27(1):165–177, 2019.
  * [23] M. Q. Patton.  Qualitative research.  Encyclopedia of statistics in behavioral science, 2005.
  * [24] I. Rae, B. Mutlu, and L. Takayama.  Bodies in motion: mobility, presence, and task awareness in telepresence.  In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, pp. 2153–2162. ACM, 2014.
  * [25] L. Rebenitsch and C. Owen.  Review on cybersickness in applications and visual displays.  Virtual Reality, 20(2):101–125, 2016.
  * [26] M. Slater.  Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments.  Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535):3549–3557, Dec. 2009. doi: 10 . 1098/rstb . 2009 . 0138
  * [27] M. Slater.  Immersion and the illusion of presence in virtual reality.  British Journal of Psychology, 109(3):431–433, 2018.
  * [28] M. Slater, A. Steed, J. McCarthy, and F. Maringelli.  The influence of body movement on subjective presence in virtual environments.  Human factors, 40(3):469–477, 1998.
  * [29] M. Slater and S. Wilbur.  A framework for immersive virtual environments (five): Speculations on the role of presence in virtual environments.  Presence: Teleoperators & Virtual Environments, 6(6):603–616, 1997\.
  * [30] B. Stoll, S. Reig, L. He, I. Kaplan, M. F. Jung, and S. R. Fussell.  Wait, can you move the robot? examining telepresence robot use in collaborative teams.  In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 14–22, 2018.
  * [31] M. Suomalainen, B. Sakcak, A. Widagdo, J. Kalliokoski, K. J. Mimnaugh, A. P. Chambers, T. Ojala, and S. M. LaValle.  Unwinding rotations improves user comfort with immersive telepresence robots.  In 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2022.
  * [32] D. Valkov, F. Steinicke, G. Bruder, and K. H. Hinrichs.  Traveling in 3d virtual environments with foot gestures and a multi-touch enabled wim.  In Proceedings of virtual reality international conference (VRIC 2010), pp. 171–180, 2010.
  * [33] S. Weech, J. Moon, and N. F. Troje.  Influence of bone-conducted vibration on simulator sickness in virtual reality.  PloS one, 13(3):e0194137, 2018.

  *[HMD]: Head-Mounted Display
  *[Wiiboard]: Wii Balance Board
  *[VE]: Virtual Environment
  *[DDR]: Differential Drive Robot
  *[DOF]: Degrees of Freedom
  *[OSF]: Open Science Foundation
  *[WB]: Wii Balance Board control method
  *[JS]: Joystick control method
  *[SSQ]: Simulator Sickness Questionnaire
  *[NASA-TLX]: NASA Task Load Index