# Towards Sim2Real Transfer of Autonomy Algorithms using AutoDRIVE Ecosystem

Chinmay Samak  Tanmay Samak  Venkat Krovi  Department of Automotive
Engineering, Clemson University International Center for Automotive Research,
Greenville, SC 29607, USA. {csamak, tsamak<EMAIL_ADDRESS>
###### Abstract

The engineering community currently encounters significant challenges in the
development of intelligent transportation algorithms that can be transferred
from simulation to reality with minimal effort. This can be achieved by
robustifying the algorithms using domain adaptation methods and/or by adopting
cutting-edge tools that help support this objective seamlessly. This work
presents AutoDRIVE, an openly accessible digital twin ecosystem designed to
facilitate synergistic development, simulation and deployment of cyber-
physical solutions pertaining to autonomous driving technology; and focuses on
bridging the autonomy-oriented simulation-to-reality (sim2real) gap using the
proposed ecosystem. In this paper, we extensively explore the modeling and
simulation aspects of the ecosystem and substantiate its efficacy by
demonstrating the successful transition of two candidate autonomy algorithms
from simulation to reality to help support our claims: (i) autonomous parking
using probabilistic robotics approach; (ii) behavioral cloning using deep
imitation learning. The outcomes of these case studies further strengthen the
credibility of AutoDRIVE as an invaluable tool for advancing the state-of-the-
art in autonomous driving technology.

###### keywords:

Autonomous Vehicles; Mobile Robots; Digital Twins; Sim2Real; Real2Sim

##  1 Introduction

The progression of connected autonomous vehicles (CAVs) necessitates a dual
approach of cutting-edge research and comprehensive education. Research
endeavors propel immediate advancements in the field, while education plays a
pivotal role in equipping the next generation with the necessary skills and
knowledge to propel the field even further in the long term. Recently,
majority of researchers, educators and students rely on simulation tools
and/or a scaled testbeds to to alleviate the monetary, spatial, temporal and
safety constraints associated with rapid-prototyping of CAV solutions. In
research settings, these platforms accelerate the process of designing
experiments, recording datasets as well as re-iteratively prototyping and
validating autonomy solutions. In educational contexts, such platforms
facilitate the creation of interactive demonstrations, hands-on assignments,
projects, and competitions centered around CAV technology.

Figure 1: AutoDRIVE Ecosystem offers of a tightly integrated trio for
designing, simulating and deploying autonomy solutions using a unified
workflow.

However, existing platforms catering to the development and validation of
connected autonomy solutions have been observed to impose limitations on
throughput. Firstly, a significant portion of these platforms lacks the
essential integrity necessary to foster hardware-software co-development. Some
platforms solely offer software simulators, while others merely provide scaled
vehicles for testing autonomy algorithms. Such isolated platforms not only
impede the prototyping phase due to compatibility issues but also hinder the
validation phase involving the transition from simulation to real-world.
Secondly, a majority of these platforms concentrate exclusively on vehicles,
neglecting the holistic integration of an intelligent transportation ecosystem
encompassing infrastructure, traffic elements, and peer agents. Consequently,
their applications remain limited in scope. Thirdly, certain platforms are
confined to specific domains or applications, featuring restricted sensing
modalities, stringent design requirements, and fixed development frameworks.
Such constraints further inhibit the versatility and adaptability of these
platforms for broader use cases in connected autonomy research and education.

This research presents AutoDRIVE111https://autodrive-ecosystem.github.io Samak
et al. (2023), a publicly accessible ecosystem specifically designed to
facilitate the integrated development, simulation, and deployment of cyber-
physical solutions pertaining to autonomous driving technology. This seamless
workflow is made possible by a tightly integrated trio, consisting of an
algorithm development kit for designing autonomy solutions, a software
simulator for virtual prototyping and testing them under a variety of
conditions and edge-cases, and a hardware testbed for physical deployment and
validation (refer Fig. 1). The synergy among these three sub-systems not only
enhances the hardware-software co-development of autonomy solutions but also
effectively bridges the gap between simulation and reality. This work places
particular emphasis on the challenges associated with bridging the sim2real
gap for autonomy-oriented applications using the proposed ecosystem. In this
context, we delve into the percepto-dynamics modeling and simulation aspects
of a scaled vehicle and infrastructure using the proposed ecosystem.
Furthermore, we substantiate our claims by showcasing the successful
transition of two candidate autonomy algorithms from simulation to reality to
help support our claims: (i) autonomous parking using probabilistic robotics
approach for mapping, localization, path planning and control; (ii) behavioral
cloning using computer vision and end-to-end deep imitation learning.

##  2 State of the Art

###  2.1 Software Simulators

Over the years, open-source community has contributed several simulators for
autonomous driving applications. Gazebo Koenig and Howard (2004), natively
integrated with Robot Operating System (ROS) Quigley et al. (2009), is
commonly used for scaled autonomous robots. TORCS Wymann et al. (2021) has
been an early focus in the self-driving community, particularly for manual and
autonomous racing. Other notable examples include CARLA Dosovitskiy et al.
(2017), AirSim Shah et al. (2018), and Deepdrive Voyage (2021), developed
using the Unreal game engine, as well as Apollo GameSim Baidu Inc. (2021),
LGSVL Simulator Rong et al. (2020) and AutoRACE Simulator Samak et al.
(2021a), created using the Unity game engine.

However, these simulators pose 3 major limitations:

  * •

Firstly, some simulation tools prioritize photorealism over physical accuracy,
while others prioritize physical accuracy over graphics quality. In contrast,
AutoDRIVE Simulator strikes a balance between physics and graphics, providing
a range of configurations to accommodate varying compute capabilities.

  * •

Secondly, the dynamics and perception systems of scaled vehicles and
environments differ significantly from their full-scale counterparts. Existing
simulation tools do not adequately support scaled ecosystems to their full
capacity. Consequently, transitioning from full-scale simulation to scaled
real-world deployment necessitates substantial additional effort to re-tune
the autonomy algorithms.

  * •

Thirdly, the existing simulators may lack precise real-world counterparts,
rendering them unsuitable for “digital twin” applications, involving synthetic
data generation, variability testing, reinforcement learning, real2sim and
sim2real transfer, among others.

###  2.2 Hardware Testbeds

In recent times, numerous educational institutions have embarked on the
development of scaled autonomous vehicles. Popular examples include the MIT
Racecar Karaman et al. (2017), F1TENTH O’Kelly et al. (2019), and AutoRally
Goldfain et al. (2019). Additionally, community-driven platforms like HyphaROS
RaceCar HyphaROS Workshop (2021) and Donkey Car Donkey Community (2021) have
emerged, tailored to specific applications like map-based navigation and
vision-aided imitation learning, respectively. Duckietown Paull et al. (2017)
is another platform, which employs differential-drive robots instead of
kinodynamically constrained car-like vehicles, thereby falling short of the
community’s requirements. Nevertheless, it remains a popular platform for
teaching autonomy fundamentals, much like TurtleBot3 Robotis Inc. (2021).

However, these platforms pose 3 major limitations:

  * •

Firstly, some of these platforms lack diverse sensing modalities, sufficient
computational power, constrained actuation mechanisms, and active or passive
infrastructural elements.

  * •

Secondly, most platforms utilize commercial radio-controlled (RC) cars as
their base-chassis, which are expensive, may not be readily available
worldwide, and limit exploration in the realm of mechatronics engineering.
Additionally, many platforms are confined to specific software frameworks like
ROS, creating a skill-set dependency for end-users.

  * •

Thirdly, some platforms lack any form of simulation support, some support
simulation using RViz Hershberger et al. (2021) or Gazebo, while others
provide task-specific Gym Brockman et al. (2016) environments for machine
learning; none of which is ideal.

##  3 AutoDRIVE Ecosystem

This section primarily focuses on digital-twin capabilities of AutoDRIVE
Ecosystem, highlighting the strong resemblance between AutoDRIVE Simulator
Samak et al. (2021c); Samak and Samak (2022b) and AutoDRIVE Testbed for
seamless sim2real transfer of autonomy algorithms developed using AutoDRIVE
Devkit.

###  3.1 Physical Vehicle

Figure 2: Native vehicle (Nigel) of AutoDRIVE Ecosystem with its components
and sub-systems highlighted.

AutoDRIVE’s native vehicle, named Nigel (refer Fig. 2), offers realistic
driving and steering actuation, comprehensive sensor suite, high-performance
computational resources, and a standard vehicular lighting system.

####  3.1.1 Chassis

Nigel is a 1:14 scale vehicle, which adopts front/rear/all-wheel-drive
Ackermann-steered mechanism, thereby resembling car-like kinodynamic
constraints.

####  3.1.2 Power Electronics

An 11.1 V 5200 mAh lithium-polymer (LiPo) battery acts as the power-source for
the vehicle. Other components such as the buck converter, motor driver, switch
and voltage checker help route the raw power to appropriate sub-systems of the
vehicle.

####  3.1.3 Sensor Suite

Nigel hosts a comprehensive sensor suite comprising throttle and steering
feedbacks, 1920 CPR incremental encoders, 3-axis IPS, 9-axis IMU, two 62.2∘
FOV cameras with 3.04 mm focal length and a 7-10 Hz, 360∘ FOV LIDAR with
0.15-12 m range and 1∘ resolution.

####  3.1.4 Computation, Communication and Software

Nigel relies on Jetson Nano Developer Kit - B01 for high-level computation
(autonomy algorithms), communication (V2V and V2I), and software installation
(JetPack SDK and AutoDRIVE Devkit). It also utilizes Arduino Nano (running the
vehicle firmware) for sensor data acquisition and filtering, and
actuators/lights control.

####  3.1.5 Actuators

Nigel is equipped with two 6V 160 RPM rated DC geared motors to drive its rear
wheels, and a 9.4 kgf.cm servo motor (saturated at $\pm$ 30∘ w.r.t. zero-steer
value) to steer its front wheels. All the actuators are operated at 5V,
providing a top speed of $\sim$0.26 m/s for driving and $\sim$0.42 rad/s for
steering.

####  3.1.6 Lights and Indicators

Nigel’s lighting system comprises of dual-mode headlights, triple-mode turning
indicators, and automated taillights and reverse indicators.

###  3.2 Virtual Vehicle

Figure 3: Simulation of vehicle dynamics, sensors and actuators. The left
inset depicts actuator dynamics model and the right inset depicts tire
dynamics model.

The vehicle is jointly modelled as a rigid-body and a collection of sprung
masses with inherent damping (refer Fig. 3). The “sprung-mass” representation
computes the suspension forces, which, aggregated with the tire forces, are
applied to the “rigid-body” representation that exactly mimics the mass,
center of mass and moment of inertia of the “sprung-mass” representation.
Needless to say, the two representations are related through the rigid-body
center of mass equation:
$X_{COM}=\frac{\sum{{{}^{i}M}*{{}^{i}X}}}{\sum{{}^{i}M}}$; where, $X_{COM}$ is
the rigid-body center of mass offset, ${}^{i}M$ are the sprung masses such
that $M=\sum{{}^{i}M}$ is the total mass of rigid-body, and ${}^{i}X$ are the
sprung mass coordinates w.r.t. ${}^{i}M$.

####  3.2.1 Suspension Dynamics

The vehicle is modeled with a rather stiff suspension system to simulate the
selective passive compliance between wheel mounts and vehicle chassis to
account for losses at these interfaces due to vibration, friction, wear,
damping, loosening, deformation, fatigue and fretting. The suspension force
acting on each of the sprung masses can be then computed using a second-order
dynamic model:
${{}^{i}M}*{{}^{i}{\ddot{Z}}}+{{}^{i}B}*({{}^{i}{\dot{Z}}}-{{}^{i}{\dot{z}}})+{{}^{i}K}*({{}^{i}{Z}}-{{}^{i}{z}})$;
where, ${}^{i}B$ and ${}^{i}K$ are the damping and spring coefficients of
$i$-th suspension, respectively. The computed suspension forces jointly affect
the rigid-body dynamics of the vehicle as well as the tire forces being
computed at that time instant (since it affects the load bearing down on the
tires).

####  3.2.2 Wheel Dynamics

The wheels of the vehicle are also modelled as rigid bodies of mass $m$, acted
upon by gravitational and suspension forces:
${{}^{i}m}*{{}^{i}{\ddot{z}}}+{{}^{i}B}*({{}^{i}{\dot{z}}}-{{}^{i}{\dot{Z}}})+{{}^{i}K}*({{}^{i}{z}}-{{}^{i}{Z}})$;
and the wheel rotations are damped to mimic rotational losses due to rolling
resistance.

####  3.2.3 Tire Dynamics

The tire forces are broken down into longitudinal $F_{t_{x}}$ and lateral
$F_{t_{y}}$ components, and are computed based on the respective friction
curve for each tire: $\left\\{\begin{matrix}{{}^{i}F_{t_{x}}}=F(^{i}S_{x})\\\
{{}^{i}F_{t_{y}}}=F(^{i}S_{y})\\\ \end{matrix}\right.$; where, ${}^{i}S_{x}$
and ${}^{i}S_{y}$ are the longitudinal and lateral slips of $i$-th tire,
respectively. Here, the friction curve is approximated as a two-piece spline
$F(S)$; one from zero $(S_{0},F_{0})$ to extremum point $(S_{e},F_{e})$, and
other from extremum point $(S_{e},F_{e})$ to asymptote point $(S_{a},F_{a})$
(refer right inset in Fig. 3):
$F(S)=\left\\{\begin{matrix}f_{0}(S);\;\;S_{0}\leq S<S_{e}\\\
f_{1}(S);\;\;S_{e}\leq S<S_{a}\\\ \end{matrix}\right.$; where,
$f_{k}(S)=a_{k}*S^{3}+b_{k}*S^{2}+c_{k}*S+d_{k}$ is a cubic polynomial
function, and $F(S)$ is saturated after the asymptote point $(S_{a},F_{a})$.

Now, slip is in-turn dependent on the various factors like tire stiffness,
steering angle, wheel speeds, suspension forces and rigid-body momentum.
Longitudinal slip $S_{x}$ is determined based on the difference between the
longitudinal components of surface velocity of the wheel compared to the
angular velocity of the wheel:
${{}^{i}S_{x}}=\frac{{{}^{i}r}*{{}^{i}\omega}-v_{x}}{v_{x}}$; where, $v_{x}$
is the longitudinal linear velocity of the vehicle (i.e., surface velocity of
the wheel), ${}^{i}r$ is the radius of $i$-th wheel, and ${}^{i}\omega$ is the
angular velocity of $i$-th wheel. Lateral slip $S_{y}$ is determined by the
angle (commonly denoted as $\alpha$) between the direction the tire is moving
in and the direction the tire is pointing in:
${{}^{i}S_{y}}=\frac{v_{y}}{\left|v_{x}\right|}$; where, $v_{x}$ is the
longitudinal linear velocity of the vehicle, and $v_{y}$ is the lateral linear
velocity of the vehicle (a.k.a. sideslip velocity).

####  3.2.4 Sensor Simulation

As described earlier, the vehicle is provided with an abundance of sensing
modalities, all of which are modeled and simulated close to their real-world
counterparts.

  * •

Throttle Feedback: Instantaneous feedback of throttle command $[-1,1]$, where
positive values indicate driving forward and negative values indicate driving
reverse.

  * •

Steering Feedback: Instantaneous feedback of steering command $[-1,1]$, where
positive values indicate left turns and negative values indicate right turns.

  * •

Incremental Encoders: These are simulated by measuring the rotation of each of
the rear wheels (based on their rigid-body transform update) and factoring in
the resolution of the encoders.

  * •

IPS: Position of the vehicle is measured based on its rigid-body transform
update. The values are converted from Unity’s left-handed coordinate system to
the right-handed coordinate system widely adopted for robotics applications.
This mimics the AprilTag-based fiducial localization system on the physical
vehicle.

  * •

IMU: Orientation of the vehicle is measured based on its rigid-body transform
update. Additionally, the linear acceleration and angular velocity of the
vehicle are measured based on temporally-coherent rigid-body transformations,
using rigid-body equations of motion. This mimics the MPU-9250 on the physical
vehicle.

  * •

LIDAR: Planar laser scan is recorded by 360∘ iterative ray-casting at 1∘
resolution and 7 Hz update rate. The raycast hits are recorded between the
minimum (0.15 m) and maximum (12.0 m) ranges of the LIDAR, respectively. This
mimics the RPLIDAR A1 on the physical vehicle.

  * •

Cameras: Physical cameras are simulated based on their focal length (3.04 mm),
field of view (62.2∘), sensor size (4.6 mm) and target resolution (720p).
Additionally, lens and film effects are simulated by post-processing the raw
frames. This mimics the two PiCamera V2.1 modules on the physical vehicle.

####  3.2.5 Actuator Simulation

As described earlier, the vehicle has driving and steering actuators, the
response delays and saturation limits of which are matched with their real-
world counterparts by tuning their torque profiles and actuation limits,
respectively (refer left inset in Fig. 3).

  * •

Driving Actuators: Each of the rear wheels is driven using a rotary motor,
which applies a torque to it:
${{}^{i}\tau_{drive}}={{}^{i}I_{w}}*{{}^{i}\dot{\omega}_{w}}$; where,
${{}^{i}I_{w}}=\frac{1}{2}*{{}^{i}m_{w}}*{{}^{i}{r_{w}}^{2}}$ is the moment of
inertia of $i$-th wheel, and ${}^{i}\dot{\omega}_{w}$ is the angular
acceleration of $i$-th wheel. Additionally, the holding torque of the driving
actuators is simulated by applying an idle motor torque equivalent to the
braking torque: ${{}^{i}\tau_{idle}}={{}^{i}\tau_{brake}}$.

  * •

Steering Actuators: The front wheels are steered using a steering actuator
coupled to the steering mechanism of the vehicle. The steering actuator also
produces a torque proportional to the moment of inertia of the steering
mechanism: $\tau_{steer}=I_{steer}*\dot{\omega}_{steer}$. The individual
turning angles, $\delta_{l}$ and $\delta_{r}$, for left and right wheels,
respectively, are calculated based on the commanded steering angle $\delta$,
using the Ackermann steering geometry defined by wheelbase $l$ and track width
$w$, as follows:
$\left\\{\begin{matrix}\delta_{l}=\textup{tan}^{-1}\left(\frac{2*l*\textup{tan}(\delta)}{2*l+w*\textup{tan}(\delta)}\right)\\\
\delta_{r}=\textup{tan}^{-1}\left(\frac{2*l*\textup{tan}(\delta)}{2*l-w*\textup{tan}(\delta)}\right)\\\
\end{matrix}\right.$

###  3.3 Physical Infrastructure

AutoDRIVE provides a modular and reconfigurable infrastructure development
kit, enabling swift design and construction of customized driving scenarios.

(a) Parking School in simulation.

(b) Parking School in reality.

(c) Driving School in simulation.

(d) Driving School in reality.

Figure 4: Infrastructure setup in simulation and reality. Note the degree of
dimensional and visual similarity between real and virtual worlds.

####  3.3.1 Environment Modules

Environment modules include static terrain and road layouts as well as
obstruction objects for rapidly designing custom scenarios. Apart from these,
experts may also choose to design scaled real-world or imaginary scenarios
using third-party tools, and import them into AutoDRIVE Ecosystem.

####  3.3.2 Traffic Elements

Traffic signs and lights define traffic laws within a particular driving
scenario, thereby governing the traffic flow. These modules support IoT and
V2I communication technologies, and can be therefore integrated with AutoDRIVE
Smart City Manager (SCM).

####  3.3.3 Surveillance Elements

AutoDRIVE features a surveillance element called AutoDRIVE Eye to view the
scene from a bird’s eye view. The said element is also integrated with
AutoDRIVE SCM, and is capable of estimating vehicle’s 2D pose within the map
by detecting and tracking the AprilTag markers attached to each of them.

####  3.3.4 Preconfigured Maps

This work focuses on two of the several preconfigured maps offered by
AutoDRIVE Ecosystem. Parking School (refer Fig. 4(a), 4(b)) is designed
specifically for autonomous parking applications, wherein construction boxes
define static obstacles and all the available free-space is drivable. On the
other hand, Driving School (refer Fig. 4(c), 4(d)) covers driving over
straight roads, curved roads and crossing an intersection.

###  3.4 Virtual Infrastructure

At every time step, the simulator performs mesh-mesh collision/interference
detection and accordingly computes the contact forces, frictional forces and
momentum transfer, along with linear and angular drag acting over each of the
rigid bodies (vehicle/infrastructure) present in the scene. This closely
emulates the interactions among vehicle(s), infrastructure elements and the
environment.

##  4 Case Studies

This work showcases sim2real capability of AutoDRIVE Ecosystem through two
different case-studies. Although this paper cannot furnish exhaustive details
pertaining to either case study, we recommend interested readers to peruse
this technical report Samak and Samak (2022a).

(a) Autonomous parking.

(b) Behavioral cloning.

Figure 5: Architectures of the two presented case-studies.

###  4.1 Autonomous Parking

This case study was implemented using the probabilistic robotics approach
comprising 5 different stages. Firstly, the vehicle mapped its surroundings
using the Hector SLAM algorithm Kohlbrecher et al. (2011). It then localized
itself against this known static map using range-flow-based odometry Jaimez et
al. (2016) and an adaptive particle filter algorithm Fox (2001). For
autonomous navigation, the vehicle planned a feasible global path from its
current pose to the parking pose using the A* algorithm Hart et al. (1968).
Simultaneously, it re-planned its local trajectory for dynamic collision
avoidance, leveraging the timed-elastic-band approach Rösmann et al. (2017). A
proportional controller generated driving (throttle/brake) and steering
commands to enable the vehicle to follow the local trajectory accurately.

(a) Virtual-world deployment.

(b) Real-world deployment.

Figure 6: Sim2real transition of autonomous parking algorithm. Video:
https://youtu.be/piCyvTM2dek

During simulation-based testing (refer Fig. 6(a)), parameter variations such
as infusion of Gaussian noise in LIDAR measurements [$n_{lidar}\sim
N\left(0,0.025\right)$ m] and actuator commands [$n_{drive}\sim
N\left(0,0.013\right)$ m/s and $n_{steer}\sim N\left(0,0.018\right)$ rad/s].
Furthermore, perturbation of wall modules’ poses [$n_{x,y}\sim
N\left(0,0.01\right)$ m and $n_{\theta}\sim N\left(0,0.087\right)$ rad] as
well as introduction of unmapped obstacles were performed. Going forward, the
same pipeline was deployed on the real vehicle (refer Fig. 6(b)) using
AutoDRIVE Testbed to validate seamless sim2real transfer. The pipeline
performed flawlessly owing to realistic LIDAR and vehicle dynamics models as
well as variability analysis during simulation.

###  4.2 Behavioral Cloning

This case study was based on Samak et al. (2021b), wherein the objective was
to utilize a convolutional neural network (CNN) for cloning the end-to-end
driving behavior of a human (refer to Fig. 5(b)). To achieve this, AutoDRIVE
Simulator was employed to record 5-laps of temporally-coherent labeled manual
driving data. This data was then balanced, augmented, and pre-processed using
standard computer vision techniques to train a 6-layer deep CNN model for 4
epochs with a learning rate of 1e-3 using the Adam optimizer Kingma and Ba
(2014). Following training, the model was deployed back into AutoDRIVE
Simulator to evaluate its performance (refer to Fig. 7(a)).

(a) Virtual-world deployment.

(b) Real-world deployment.

Figure 7: Sim2real transition of behavioral cloning algorithm. Video:
https://youtu.be/rejpoogaXOE

Variability testing concerning light intensity and direction as well as
vehicle’s initial conditions and velocity limit was performed to ensure
algorithm robustness. Further, the same CNN model was deployed onto AutoDRIVE
Testbed for validating the sim2real transition of this vision-based algorithm
(refer Fig. 7(b)). Despite subtle variations in the environmental conditions,
the model generalized well and the vehicle was able to complete several laps
along the track without major deviation and/or collision.

##  5 Conclusion

In this paper, we presented AutoDRIVE, a publicly accessible digital twin
ecosystem for CAVs developed with an aim of tightly integrating reality and
simulation into a unified toolchain, without compromising on the
comprehensiveness, flexibility and accessibility required for prototyping and
validating autonomy solutions. This work focused on bridging the autonomy-
oriented sim2real gap using the proposed ecosystem. Furthermore, we
extensively discussed the modeling and simulation aspects of the ecosystem and
substantiated its efficacy by demonstrating the successful transition of two
candidate autonomy algorithms including autonomous parking and behavioral
cloning from simulation to reality to help support our claims. Further
research will delve into the investigation of handling real and virtual world
uncertainties, formulating qualitative/quantitative evaluation metrics and
benchmarks, as well as improving the robustness and generalization of sim2real
frameworks for autonomous vehicles.

## References

  * Baidu Inc. (2021) Baidu Inc. (2021).  Apollo Game Engine Based Simulator.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016).  OpenAI Gym.
  * Donkey Community (2021) Donkey Community (2021).  An Open-Source DIY Self-Driving Platform for Small-Scale Cars.
  * Dosovitskiy et al. (2017) Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017).  CARLA: An Open Urban Driving Simulator.  In S. Levine, V. Vanhoucke, and K. Goldberg (eds.), _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , 1–16. PMLR.
  * Fox (2001) Fox, D. (2001).  KLD-Sampling: Adaptive Particle Filters.  In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), _Advances in Neural Information Processing Systems_ , volume 14. MIT Press.
  * Goldfain et al. (2019) Goldfain, B., Drews, P., You, C., Barulic, M., Velev, O., Tsiotras, P., and Rehg, J.M. (2019).  AutoRally: An Open Platform for Aggressive Autonomous Driving.  _IEEE Control Systems Magazine_ , 39(1), 26–55.  10.1109/MCS.2018.2876958.
  * Hart et al. (1968) Hart, P.E., Nilsson, N.J., and Raphael, B. (1968).  A Formal Basis for the Heuristic Determination of Minimum Cost Paths.  _IEEE Transactions on Systems Science and Cybernetics_ , 4(2), 100–107.  10.1109/TSSC.1968.300136.
  * Hershberger et al. (2021) Hershberger, D., Gossow, D., and Faust, J. (2021).  RViz: 3D Visualization Tool for ROS.
  * HyphaROS Workshop (2021) HyphaROS Workshop (2021).  HyphaROS Racecar.
  * Jaimez et al. (2016) Jaimez, M., Monroy, J.G., and Gonzalez-Jimenez, J. (2016).  Planar Odometry from a Radial Laser Scanner. A Range Flow-based Approach.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 4479–4485.  10.1109/ICRA.2016.7487647.
  * Karaman et al. (2017) Karaman, S., Anders, A., Boulet, M., Connor, J., Gregson, K., Guerra, W., Guldner, O., Mohamoud, M., Plancher, B., Shin, R., and Vivilecchia, J. (2017).  Project-based, collaborative, algorithmic robotics for high school students: Programming self-driving race cars at MIT.  In _2017 IEEE Integrated STEM Education Conference (ISEC)_ , 195–203.  10.1109/ISECon.2017.7910242.
  * Kingma and Ba (2014) Kingma, D.P. and Ba, J. (2014).  Adam: A Method for Stochastic Optimization.  10.48550/ARXIV.1412.6980.
  * Koenig and Howard (2004) Koenig, N.P. and Howard, A. (2004).  Design and use paradigms for Gazebo, an open-source multi-robot simulator.  In _2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566)_ , volume 3, 2149–2154.  10.1109/IROS.2004.1389727.
  * Kohlbrecher et al. (2011) Kohlbrecher, S., von Stryk, O., Meyer, J., and Klingauf, U. (2011).  A Flexible and Scalable SLAM System with Full 3D Motion Estimation.  In _2011 IEEE International Symposium on Safety, Security, and Rescue Robotics_ , 155–160.  10.1109/SSRR.2011.6106777.
  * O’Kelly et al. (2019) O’Kelly, M., Sukhil, V., Abbas, H., Harkins, J., Kao, C., Pant, Y.V., Mangharam, R., Agarwal, D., Behl, M., Burgio, P., and Bertogna, M. (2019).  F1/10: An Open-Source Autonomous Cyber-Physical Platform.
  * Paull et al. (2017) Paull, L., Tani, J., Ahn, H., Alonso-Mora, J., Carlone, L., Cap, M., Chen, Y.F., Choi, C., Dusek, J., Fang, Y., Hoehener, D., Liu, S.Y., Novitzky, M., Okuyama, I.F., Pazis, J., Rosman, G., Varricchio, V., Wang, H.C., Yershov, D., Zhao, H., Benjamin, M., Carr, C., Zuber, M., Karaman, S., Frazzoli, E., Del Vecchio, D., Rus, D., How, J., Leonard, J., and Censi, A. (2017).  Duckietown: An Open, Inexpensive and Flexible Platform for Autonomy Education and Research.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 1497–1504.  10.1109/ICRA.2017.7989179.
  * Quigley et al. (2009) Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A. (2009).  ROS: an open-source Robot Operating System.  In _ICRA Workshop on Open Source Software_ , volume 3.
  * Robotis Inc. (2021) Robotis Inc. (2021).  TurtleBot3.
  * Rong et al. (2020) Rong, G., Shin, B.H., Tabatabaee, H., Lu, Q., Lemke, S., Možeiko, M., Boise, E., Uhm, G., Gerow, M., Mehta, S., Agafonov, E., Kim, T.H., Sterner, E., Ushiroda, K., Reyes, M., Zelenkovsky, D., and Kim, S. (2020).  LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving.  In _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , 1–6.  10.1109/ITSC45102.2020.9294422.
  * Rösmann et al. (2017) Rösmann, C., Hoffmann, F., and Bertram, T. (2017).  Kinodynamic Trajectory Optimization and Control for Car-Like Robots.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 5681–5686.  10.1109/IROS.2017.8206458.
  * Samak et al. (2021a) Samak, C.V., Samak, T.V., and Kandhasamy, S. (2021a).  Autonomous Racing using a Hybrid Imitation-Reinforcement Learning Architecture.  10.48550/ARXIV.2110.05437.
  * Samak et al. (2023) Samak, T., Samak, C., Kandhasamy, S., Krovi, V., and Xie, M. (2023).  AutoDRIVE: A Comprehensive, Flexible and Integrated Digital Twin Ecosystem for Autonomous Driving Research & Education.  _Robotics_ , 12(3).  10.3390/robotics12030077.
  * Samak and Samak (2022a) Samak, T.V. and Samak, C.V. (2022a).  AutoDRIVE - Technical Report.  10.48550/ARXIV.2211.08475.
  * Samak and Samak (2022b) Samak, T.V. and Samak, C.V. (2022b).  AutoDRIVE Simulator - Technical Report.  10.48550/ARXIV.2211.07022.
  * Samak et al. (2021b) Samak, T.V., Samak, C.V., and Kandhasamy, S. (2021b).  Robust Behavioral Cloning for Autonomous Vehicles Using End-to-End Imitation Learning.  _SAE International Journal of Connected and Automated Vehicles_ , 4(3), 279–295.  https://doi.org/10.4271/12-04-03-0023.
  * Samak et al. (2021c) Samak, T.V., Samak, C.V., and Xie, M. (2021c).  AutoDRIVE Simulator: A Simulator for Scaled Autonomous Vehicle Research and Education.  In _2021 2nd International Conference on Control, Robotics and Intelligent System_ , CCRIS’21, 1–5. Association for Computing Machinery, New York, NY, USA.  10.1145/3483845.3483846.
  * Shah et al. (2018) Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2018).  AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles.  In M. Hutter and R. Siegwart (eds.), _Field and Service Robotics_ , 621–635. Springer International Publishing, Cham.
  * Voyage (2021) Voyage (2021).  Deepdrive.
  * Wymann et al. (2021) Wymann, B., Espié, E., Guionneau, C., Dimitrakakis, C., Coulom, R., and Sumner, A. (2021).  TORCS, The Open Racing Car Simulator.


# Enhancing Human-robot Collaboration by Exploring Intuitive Augmented Reality
Design Representations

Chrisantus Eze  0000-0001-5440-4316 Oklahoma State UniversityStillwaterOKUSA
and  Christopher Crick  0000-0002-1635-823X Oklahoma State
UniversityStillwaterOKUSA

(2023)

###### Abstract.

As the use of Augmented Reality (AR) to enhance interactions between human
agents and robotic systems in a work environment continues to grow, robots
must communicate their intents in informative yet straightforward ways. This
improves the human agent’s feeling of trust and safety in the work environment
while also reducing task completion time. To this end, we discuss a set of
guidelines for the systematic design of AR interfaces for Human-Robot
Interaction (HRI) systems. Furthermore, we develop design frameworks that
would ride on these guidelines and serve as a base for researchers seeking to
explore this direction further. We develop a series of designs for visually
representing the robot’s planned path and reactions, which we evaluate by
conducting a user survey involving 14 participants. Subjects were given
different design representations to review and rate based on their
intuitiveness and informativeness. The collated results showed that our design
representations significantly improved the participants’ ease of understanding
the robot’s intents over the baselines for the robot’s proposed navigation
path, planned arm trajectory, and reactions.

augmented reality, user interface design

††journalyear: 2023††copyright: acmlicensed††conference: Companion of the 2023
ACM/IEEE International Conference on Human-Robot Interaction; March 13–16,
2023; Stockholm, Sweden††booktitle: Companion of the 2023 ACM/IEEE
International Conference on Human-Robot Interaction (HRI ’23 Companion), March
13–16, 2023, Stockholm, Sweden††price: 15.00††doi:
10.1145/3568294.3580089††isbn: 978-1-4503-9970-8/23/03††ccs: Human-centered
computing Mixed / augmented reality††ccs: Human-centered computing User
interface design

##  1\. Introduction

There have been a burst of augmented reality (AR) applications to help improve
HRI in different fields, mainly in the industrial setting. As we transition
into an era of highly automated manufacturing, more emphasis will be given to
improving how human agents interact and collaborate with robots in a work
environment. These interactions have yielded remarkable results so far, but
many HRI challenges remain. One such challenge is reducing the cognitive
workload of a human agent when interacting with a robot via an AR interface,
especially one unfamiliar with the system. Poor communication of robot
intention and planned movements can cause hazards in the work environment that
might jeopardize safety, reduce task efficiency, and lead to flawed
perceptions of robot usability (Walker et al., 2018). However, active research
is currently being carried out to address these issues. AR has had a
significant impact on enhancing human-robot collaboration in the workspace in
different ways. It has been used for robot programming and motion planning to
instruct robots on task execution (Tsamis et al., 2021), (Hernández et al.,
2020). This could be by issuing either low-level (Tsamis et al., 2021) or
high-level goals (Hernández et al., 2020) where the robot figures out the best
way to execute the request. AR has also been used to communicate the robot’s
intentions, such as warning alerts, proposed path, arm motion, and trajectory
to the agent.

In our work, we outline and discuss a set of design guidelines that should be
followed when designing AR interfaces for HRI systems. These guidelines not
only ensure that interfaces are designed as intuitively as possible for a
novel user, but also provides systematic best practices for an HRI researcher
when developing HRI interfaces. In order to illustrate the usage of the
guidelines, we have developed sets of interface designs for various types of
robot feedback during the course of a robot’s interactions with a human agent
in a work environment. These interfaces include the proposed robot navigation
path, proposed robotic arm end-effector trajectory, and alert and warning
displays. Safety and trust of both the robot and the human agent(s) are two
important factors to be considered when designing HRI systems, and in our
work, we have explored the AR design space in order to develop interfaces that
would enhance these factors whilst also ensuring a reduced task completion
time. To evaluate our proposed design frameworks, we created a survey where
the participants gave different ratings for the various interface designs.

##  2\. Related Work

State-of-the-art manufacturing systems (Hermann et al., 2016) has created the
need to reduce the complexity that arises from a hybrid work environment
(ElMaraghy, 2005). As noted in (Lotsaris et al., 2021), there should be tools
that can conceal these complexities from human agents and help them
communicate and interact with the rest of the system. Furthermore, (Lotsaris
et al., 2021) noted that these tools must offer easy-to-use interfaces with an
intuitive mechanism that should not distract the workers from their tasks.
Here, they designed an application that doesn’t require a human agent to have
specialized knowledge or training. However, the interface was cluttered with
many buttons, which could make the human agent have difficulties figuring out
the right button to press. In our work, we tried to avoid this clutter by
limiting the number of view items that we overlay on the agent’s screen.

Georgios et al. (Tsamis et al., 2021) proposed a system that allows for
bidirectional communication between a robot and multiple users in a work
environment in real-time. This work may not have represented the design of
their AR application in a way that would be intuitive for the agents. For
instance, in the paper, the end-effector’s planned trajectory was represented
using a 3D sphere and the arm movement radius as a red transparent sphere.
However, using red for this purpose could be misinterpreted to mean danger or
a problem. Our system explores different design representations for capturing
the robot’s planned trajectory, making it easier for a human agent unfamiliar
with the system to grasp it quickly.

Lotsaris et al. (Michalos et al., 2016) used an AR application to provide
functionalities such as robot motion and workspace visualization, visual and
audio alerts, and production data. Here, operators can visualize all the
components involved in the assembly sequence intuitively using 3D models.
Their proposed approach, despite its innovative audio alerts, is less
informative than it could be. The text on the visual alert is blurry and the
3D model lacks sufficient fidelity and might be challenging for a novel
operator to understand and navigate. In contrast, in our proposed framework,
we have ensured that the design of the interfaces is very simple and easy even
for an untrained user. We also ensured that our 3D models have high fidelity.

Robot trajectory planning is one of the most important aspects of a human-
robot interaction system and research have been carried out to improve and
achieve better results. Fang et al. (Fang et al., 2012) designed a high-
fidelity AR-based virtual manipulator which facilitates robot programming and
trajectory planning. Despite the high fidelity of the interface and the
accuracy achieved by the simulated motion planner, the presented system would
pose a challenge to a less technical user given the level of technical
sophistication and low intuition provided by the system.

##  3\. AR Design Best Practices and Guidelines

When designing AR interfaces for HRI purposes, it is important to follow
specific laid-out guidelines. These rules not only ensure that interfaces are
designed appropriately for new and experienced users, but they also provide a
systematic approach for the researcher towards designing interfaces for HRI
systems. Radkowski et al. (Radkowski et al., 2015) proposed that the level of
difficulty of carrying out a task during a manual assembly process should
determine the complexity of the user interface (UI). The factors to be
considered include the number of moving directions of a part, the number of
active surfaces the operator needs to keep aligned during a joining procedure,
the number of parts and operations in the assembly task, the hierarchy of
parts and operations in the assembly task, the visibility of the part to
assemble, and lastly, the posture of the operator. However, the authors noted
that these parameters can be difficult to quantify, but with formal user
studies using interface prototyping, the assessment can be facilitated.
Furthermore, Danielsson (Danielsson, 2016) discussed a concept called ”design
science” which aims at developing ways to achieve human goals through the
creation of artifacts that are evaluated to ensure they meet the design
requirements. Wilkinson et al. (Wilkinson et al., 2021) also proposed some
guidelines which should guide the design of interfaces for HRI systems. The
authors discussed the implementation of these guidelines using two UIs they
developed. We have also applied these guidelines to our work. Below is a
description of the guidelines and how we adhered to them.

####  3.0.1. Understandable

(Wilkinson et al., 2021) stated that since HRI systems should be simple even
for an untrained user, the UIs must offer enough feedback and intuition to
allow for ease of understanding and use. Our proposed the framework used basic
visual features which most people are familiar with, such as arrows and cones,
thereby offering good intuition to the users and reducing task completion
time.

####  3.0.2. Reliable

Since the real world cannot be completely predictable, there are chances that
the user might make errors. A well-designed HRI system should have a robust
error-handling capability (Wilkinson et al., 2021). We adhered to this
guideline by designing some sets of visual alerts which give a user a proper
visualization of the status of the robot, enabling the user to avoid cases
that could cause harm to them or the robot.

####  3.0.3. Accessible

The HRI system should be able to be used by various people with different
physical and cognitive abilities. In our system, we ensured that visual
features such as labels, arrows, and cones are not only accessible but also
very legible and informative to all users. Furthermore, (Dünser et al., 2007)
discusses some Human-Computer Interaction (HCI) principles could serve as
guidelines for designing AR systems. We have extended and incorporated this
set of guidelines in our work. Below we discuss the guidelines most relevant
to AR applications for HRI purposes.

####  3.0.4. Low Cognitive Overhead

An AR interface should be designed in a way that would enable the user to
focus on the actual task and not get distracted by unimportant information,
helping reduce the agent’s cognitive overhead. According to (Wilkinson et al.,
2021), registration errors in AR systems could reduce the performance of users
when the virtual elements from the AR interface are not well aligned with the
physical objects. The result of this could lead to a reduction in task
performance since associating the virtual elements with the physical
environment requires cognitive effort from the user (Dünser et al., 2007).

####  3.0.5. Learnable and Consistent

A good user interface should enable users quickly to learn how to navigate an
AR application. Maintaining a consistent interface, in both appearance and
functionality, is essential for this purpose. In our work, we have ensured
that we maintained a consistent look and feel of the user interface. We used
consistent color schemes, object shapes, and dimensions to ensure uniformity
and coherence.

####  3.0.6. Responsive

It is essential that AR applications are designed with responsiveness in mind.
It is almost impossible to rule out the occurrence of lags in an application.
Therefore, an AR application should be designed to provide feedback to the
user whenever such occurs. This helps keep the user informed and less anxious.

The proposed design frameworks adhered to the above guidelines which ensures a
systematic approach to designing AR interfaces for HRI systems in order to
achieve informative and intuitive interfaces.

Figure 1. Different robot feedback warning alert designs.

Different designs for representing robot reactions

##  4\. Technical Description

Our work uses a 3D/AR Unity application to explain and recommend the different
ways AR interfaces should be designed to enhance Human-Robot collaboration in
a work environment. We undertook an iterative process to explore the design
space of AR-HRI systems by providing intuitive and concise feedback on robot
motion intent and obstructions identified by the robot. We evaluated several
past AR-HRI systems across different application scenarios and came up with a
base for a high-level framework on how to design and represent robot feedback
in an AR-HRI system for enhanced human-robot interactions.

We implemented our proposed HRI system using the open-source simulated Niryo
One robotic manipulator, a collaborative 6-axis robot in the Unity 3D
environment. It was provided by (Niryo, 2022) in the form of a Unified Robot
Description File (URDF) (Unity-Technologies, 2022), which is similar to what
was used in (Xue Er Shamaine et al., 2020).

Figure 2. Different proposed robot navigation path designs

Different designs for representing robot navigation paths

##  5\. Experimental Evaluation

In this section, we will discuss the problems identified in the design of
interfaces for AR-HRI systems and evaluate our proposed solutions. Our
proposed solution tries to classify the interfaces into three broad
categories: visual alerts, robot navigation paths and workspace visualization,
and robotic arm end-effector manipulation and object pickup. These designs
were evaluated by recruiting 14 participants who were presented with the
different design representations of our system in an online survey. The
designs were presented in the order: path-arm-alert and they were asked to
express their thoughts about the designs, how quickly they were able to figure
out what the designs were about, and how intuitive the designs were. These
responses in addition to the Likert scale ratings were collated from the
participants.

###  5.1. Visual Alerts

Visual alerts could be general messages designated by the process planner
(Makris et al., 2016) or more specific alerts like the path of the robot. Due
to the impact of these alerts in determining both user and robot safety in the
work environment, it is crucial to ensure that the the information they convey
is intuitive and easily understood by human operators without requiring much
cognitive effort. The alert interfaces in previous AR-HRI systems are often
either illegible due to poor choice of color and font or are cluttered with so
many views items that the messages are challenging for the human operator to
understand. This can be seen in (Makris et al., 2016) and (Lotsaris et al.,
2021).

To address this challenge, we proposed four different visual alert design
representations (see Figure 1) which were evaluated by the survey
participants.

Figure 3. Different planned robot arm trajectory designs

Different designs for representing robot arm trajectory and motion

Figure 4. User ratings for the robot proposed navigation path, planned arm
trajectory, and alert designs

This shows the result from the online survey carried out

###  5.2. Robot Navigation Path and Workspace Visualisation

Robot navigation and path planning are essential aspects of an HRI system. A
poorly designed visual representation of the robot’s planned the path is
likely to lead to confusion in the work environment among the human operators,
which not only has the potential to harm the operators but also increases the
production time. In (Tsamis et al., 2021), the planned robotic path was
represented using a set of 3D green spheres and has no similarity with real-
world path indicators, introducing the potential of being misunderstood by an
operator unfamiliar with the system and environment.

We have addressed this challenge by applying familiar real-world design
representations and concepts such as traffic cones and road barriers to
developing three different design representations of robot navigation paths
(see Figure 2) that have been evaluated and rated by the survey participants.

###  5.3. Robotic Arm End-Effector Manipulation and Object Pickup

Augmented reality has often been used to plan the path and orientation of the
end-effector of a robot. However, adequate measures should be taken to ensure
that these paths are represented in the design space intuitively, avoiding
complexity whenever possible. Just like their proposed robot navigation path,
(Tsamis et al., 2021) represented the proposed path of their end-effector
using a set of green 3D spheres that made an oval shape. This design concept
could potentially confuse the operators, especially those unfamiliar with the
system because they failed to use familiar and conventional view elements to
represent paths and trajectories in the physical world. To address this, we
have evaluated several design approaches to represent the planned paths and
trajectories of end-effectors (see Figure 3).

##  6\. Results & Discussions

This section summarises our findings based on the survey involving 14
participants. The designs were reviewed based on their intuitiveness,
informativeness, and proximity to natural and familiar representations. The
participants were asked to rate the proposed methods and the baseline methods
((Tsamis et al., 2021), (Michalos et al., 2016), and (Fang et al., 2012)) on a
5-point Likert scale (see Figure 4).

###  6.1. Proposed Navigation Path

The first subfigure shows the survey results for the different robot
navigation path designs. Most of the participants had difficulty understanding
the robot’s intent when they were presented with the baseline’s
representations. However, our approach significantly improved the
participants’ understanding of the robot’s intent. The Design 1 representation
was the most informative and imposed the least cognitive overload to the
participants.

###  6.2. Planned Arm Trajectory

Results for the proposed robotic arm trajectory designs are shown in the
second subfigure. In this category, the baseline approaches were also rated as
less informative by most of the participants compared to our proposed
approaches. However, most participants rated their understanding of the
proposed designs as either neutral or informative. Thus, the proposed
approaches did not significantly improve the users’ understanding of the
robot’s intent.

###  6.3. Alert

The third subfigure shows the survey results for the various robot reactions
designs, such as warning alerts. The participants rated our baseline
approaches as informative. In contrast, our proposed designs provided a more
explicit representation of the robot’s intent to the participants and most
participants rated them as either informative or very informative.

We found a significant effect of design on perceived communication clarity,
which translates to the user’s perception of the robot as a good work partner.
Aside from the ratings of the presented designs, some participants left
important feedback regarding how they felt during their evaluations of the
different designs.

  * •

Participant X [Arm Trajectory]: ”At first, I thought the robot was shooting
cannon balls at the object.”

  * •

Participant Y [Arm Trajectory]: ”Is this trying to represent projectile motion
or something of that sort?”

  * •

Participant Z [Navigation Path]: ”I initially didn’t understand what the
dotted colored lines on the floor meant. It took me a while to figure it out.”

Interestingly, some participants found some of the design representations from
the baseline methods to be more informative than some of our design
representations in some categories. However, the same participants rated at
least one of our proposed designs higher than all the baseline approaches,
even in those cases.

##  7\. Conclusions & Recommendations

This research explored the design space of human interactions with a robot
through AR to convey the robot’s intentions. We conducted a survey involving
14 participants, where each participant was presented with different design
representations which they rated against the baseline approaches from previous
work. We found that they were able to easily and more quickly understand the
intents of the robot at the different stages of the human-robot collaboration
cycle. From the results, we note that our proposed representations
significantly improved their understanding of the robot’s intents over the
baseline approaches.

The aim of this work was to introduce a systematic approach to designing AR
interfaces for HRI systems to ensure that they adhere to the best practice of
providing a very limited cognitive workload to the users. From the statistics
of our collated results, we could see the impact our proposed design
representations made on the participants in terms of the ease of understanding
the robot’s intents, hence we believe this aim was achieved. Therefore, we
recommend our approach as a base for researchers exploring ways to improve the
usability of their HRI systems, especially when designing their robot’s visual
feedback systems for conveying robot intentions and messages to human agents.

## References

  * (1)
  * Danielsson (2016) Oscar Danielsson. 2016\.  Designing Augmented Reality Interfaces for Human-Robot Collaboration in Engine Assembly: Research Proposal.
  * Dünser et al. (2007) Andreas Dünser, Raphaël Grasset, Hartmut Seichter, and Mark Billinghurst. 2007.  Applying HCI principles to AR systems design.
  * ElMaraghy (2005) Hoda A ElMaraghy. 2005\.  Flexible and reconfigurable manufacturing systems paradigms.  _International journal of flexible manufacturing systems_ 17, 4 (2005), 261–276.
  * Fang et al. (2012) HC Fang, SK Ong, and AYC Nee. 2012.  Interactive robot trajectory planning and simulation using augmented reality.  _Robotics and Computer-Integrated Manufacturing_ 28, 2 (2012), 227–237.
  * Hermann et al. (2016) Mario Hermann, Tobias Pentek, and Boris Otto. 2016\.  Design Principles for Industrie 4.0 Scenarios. In _2016 49th Hawaii International Conference on System Sciences (HICSS)_. 3928–3937.
  * Hernández et al. (2020) Juan David Hernández, Shlok Sobti, Anthony Sciola, Mark Moll, and Lydia E. Kavraki. 2020\.  Increasing Robot Autonomy via Motion Planning and an Augmented Reality Interface.  _IEEE Robotics and Automation Letters_ 5, 2 (2020), 1017–1023.
  * Lotsaris et al. (2021) Konstantinos Lotsaris, Nikos Fousekis, Spyridon Koukas, Sotiris Aivaliotis, Niki Kousi, George Michalos, and Sotiris Makris. 2021\.  Augmented reality (ar) based framework for supporting human workers in flexible manufacturing.  _Procedia CIRP_ 96 (2021), 301–306.
  * Makris et al. (2016) Sotiris Makris, Panagiotis Karagiannis, Spyridon Koukas, and Aleksandros-Stereos Matthaiakis. 2016\.  Augmented reality system for operator support in human–robot collaborative assembly.  _CIRP Annals_ 65, 1 (2016), 61–64.
  * Michalos et al. (2016) George Michalos, Panagiotis Karagiannis, Sotiris Makris, Önder Tokçalar, and George Chryssolouris. 2016.  Augmented reality (AR) applications for supporting human-robot interactive cooperation.  _Procedia CIRP_ 41 (2016), 370–375.
  * Niryo (2022) Niryo. 2022.  _Niryo One, a collaborative and open source 6-axis robot_.  https://niryo.com/fr/product/niryo-one/ [Online; 08. March 2022].
  * Radkowski et al. (2015) Rafael Radkowski, Jordan Herrema, and James H. Oliver. 2015\.  Augmented Reality-Based Manual Assembly Support With Visual Features for Different Degrees of Difficulty.  _International Journal of Human-Computer Interaction_ 31 (2015), 337 – 349.
  * Tsamis et al. (2021) Georgios Tsamis, Georgios Chantziaras, Dimitrios Giakoumis, Ioannis Kostavelis, Andreas Kargakos, Athanasios Tsakiris, and Dimitrios Tzovaras. 2021\.  Intuitive and Safe Interaction in Multi-User Human Robot Collaboration Environments through Augmented Reality Displays. In _2021 30th IEEE International Conference on Robot Human Interactive Communication (RO-MAN)_. 520–526.
  * Unity-Technologies (2022) Unity-Technologies. 2022\.  _Pick-and-Place Tutorial_.  https://github.com/Unity-Technologies/Unity-Robotics-Hub/tree/main/tutorials/pick_and_place [Online; 03. March 2022].
  * Walker et al. (2018) Michael Walker, Hooman Hedayati, Jennifer Lee, and Daniel Szafir. 2018.  Communicating Robot Motion Intent with Augmented Reality. In _2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. 316–324.
  * Wilkinson et al. (2021) Alexander Wilkinson, Michael Gonzales, Patrick Hoey, David Kontak, Dian Wang, Noah Torname, Sam Laderoute, Zhao Han, Jordan Allspaw, Robert Platt, and Holly Yanco. 2021\.  Design guidelines for human–robot interaction with assistive robot manipulation systems.  _Paladyn, Journal of Behavioral Robotics_ 12, 1 (2021), 392–401.
  * Xue Er Shamaine et al. (2020) Chung Xue Er Shamaine, Yuansong Qiao, John Henry, Ken McNevin, and Niall Murray. 2020\.  RoSTAR: ROS-based Telerobotic Control via Augmented Reality. In _2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)_. 1–6.