# Test Framework for a Virtual Competition Testbed

Liam Wellacott1, Emilyann Nault2, Ioannis Skottis3 ***Alexandre Colle4,
Shreyank N Gowda5,
Pierre Nicolay6, and Emily Rolley-Parnell7 ***The following authors are
ordered alphebetically
Code: https://github.com/LiamWellacott/CDT2019-ERL
Video: https://www.youtube.com/watch?v=RoxGla2P-uY&ab_channel=ColleAlexandre
Heriot-Watt University & University of Edinburgh
Edinburgh, United Kingdom
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Virtual environments have been utilised in robotics research as a tool to
assess systems before deploying them in the field. The COVID-19 pandemic has
brought about additional motivation for the development of virtual benchmarks
in order to aid in safe and productive development. In-person robotics
competitions have also halted, thus limiting the scope of opportunities for
students and researchers. We implemented the structure of a service robotics
competition into an extendable and adaptable virtual scoring environment. The
competition challenges the state of the art in home service robotics by
presenting realistic household tasks for robots to complete. The virtual
environment provides a foundation for competition teams to assess their
systems when accessing the physical environment is not possible. We believe
that utilising virtual environments as a means of assessment will lead to
other benefits such as increased access and generalisation.

###### Index Terms:

Robotic Competitions, Benchmarking, Simulation, Human-Robot Interaction

##  I Introduction

Competitions are an important tool for pushing the state of the art in
robotics applications. They also provide new researchers with practical skills
and knowledge used throughout their career. Hosting physical competitions has
not been possible during the COVID-19 pandemic. This has highlighted the need
for tools which allow teams to continue their work regardless of access to the
physical competition environment.

In this work we target the European Robotics League Consumer Service Robots
competition (ERL Consumer for short). Teams are challenged with performing
realistic household tasks for an older adult, Granny Annie. The competition is
normally hosted at locations across Europe with accredited testbeds designed
to appear like a typical small apartment. The competition organisers provide a
handbook which describes the tasks and scoring criteria. Additionally,
functional benchmarks provide a method to assess a team’s solution in a single
capability (e.g. natural language understanding). Both the tasks and
functional benchmarks are performed across a three-day period. They are
partially automated, but require human intervention for purposes such as
setting up objects in the environment.

To our knowledge, there is no automated virtual environment where teams can
test their system against the ERL competition tasks and functional benchmarks.
We identified this gap first-hand when attempting to produce our own
competition entry while working under the restrictions of the COVID-19
lockdown in the UK. In this paper we present our virtual benchmark for the ERL
Consumer competition.

##  II Background

Robotics competitions have made a shift to becoming more virtual, which has
only increased in prominence since the start of the COVID-19 pandemic. Some
competitions such as the SpaceRobotsChallenge have opted to use simulation
since sending competition robots to space would be infeasible/unreasonable. In
recognition of the disruptive nature of the COVID-19 pandemic, other
competitions have shifted towards virtual platforms such the VirtualRobotX
competition. More recently other competitions have announced virtual
competitions entries such as 2021 RoboCup@Home and 2021 RoboCup Rescue.

In the context of service robotics, OpenRobotics, in collaboration with
Hitatchi, created a virtual robotics competition called ServiceSim[1]. The
competition’s focus is HRI in an office environment. In an effort to
facilitate research in the field of social robotics and specifically the
learning of social norms (e.g. proxemics), Pimentel and Aquino-Junior [2] have
developed a virtual environment with a scoring mechanism using ROS, Gazebo and
OpenAI. Most recently, SEAN by Nathan Tsoi et al. [3] developed a virtual
world with the focus on the development and evaluation of algorithms for
social navigation (dynamic conditions such as simulated pedestrians, cars,
etc.).

Robotics competitions are not only important for robotics research, but also
are used to educate and engage students. A good example of this is the
competition RoboCupJunior [4]. Virtual environments can grant students access
to state of the art equipment in cases where it otherwise would not have been
possible.

##  III Virtual Benchmark

The virtual benchmark is a system for testing ROS-based ERL Consumer
competition solutions in an automated virtual environment. When the user
starts the benchmark, the simulation environment, which has been configured
for a given scenario, is launched. The user’s competition solution performs
the task and notifies the virtual referee upon completion. The referee
evaluates the task performance and returns a score.

The simulation environment is a Gazebo world designed with the same furniture
and overall dimensions as the physical Heriot-Watt University Robotic Assisted
Living Testbed (RALT) (Fig. 1). To configure the world for a particular
scenario, we utilise the large number of Gazebo-compatible object models
available online (e.g. boxes of food, standing human). Each scenario is built
using a separate roslaunch file which describes the positions of the objects
as well as the starting position of the robot. As in the physical competition,
the team has time to ensure the robot is fully initialised before the scenario
begins.

Figure 1: The virtual RALT Gazebo world

To start the scenario, the user launches the virtual referee, which is based
on the Referee, Scoring and Benchmarking Box (RSBB) used in the ERL Consumer
competition. In addition to its role in scoring and scenario monitoring, the
RSBB provides a single interface for smart home features through which the
robot can interact with devices in the home. Importantly, the virtual referee
employs the same ROS topics used in the RSBB in the physical competition,
meaning teams don’t have to change their solution to interact with our
referee. Once the user launches the referee, a start signal is sent to the
robot which contains the scenario context. The scenario context could be:

  * •

The smart doorbell has been activated and the robot must greet the visitor.

  * •

Granny Annie has used her smart tablet to “summon” the robot to her.

  * •

Granny Annie makes a natural language request.

The virtual referee then waits for the robot to signal it has completed the
task. Once the signal is received, the state of the environment (using gazebo
model states) is examined to provide a score for the run.

The virtual benchmark contains the following scenario, a simplified version of
the ERL Consumer competition task “catering for Granny Annie’s comfort”.
Granny Annie summons the robot through the smart tablet and requests the
cracker box from the kitchen. The robot must interpret this command and
retrieve the box. Once the task is completed, the robot notifies the referee
and receives a score of two for (1) removing the box from the kitchen island
and (2) placing the box next to Granny Annie.

The virtual benchmark is designed to be extended with additional test
scenarios, which requires two steps to set up. Firstly, the user must
configure the world state by producing a new roslaunch file to describe the
objects and starting location for the robot. Secondly, the user must add the
scenario logic to the virtual referee. This can be accomplished by extending
the existing referee with the start signal and end logic, reusing the topics
and services already in place.

The virtual benchmark does not rely on a particular robot or software solution
to be compatible. The only requirement is the ROS interface, which is also
true for the physical competition. Currently the virtual benchmark is
configured for the RALT environment and Tiago robot. However, the roslaunch
file can be reconfigured to use another environment/robot and reuse the
virtual referee scenario logic freely. This means that it is possible to test
your solution across multiple competition testbeds to assess generalisation
capability[5].

##  IV Conclusion and Further Work

Implementing automatic scoring benchmarks provides a potential tool for
holding robotics competitions virtually. They can also be applied to other
similar competitions such as RoboCup@Home. The tasks and functional benchmarks
provide obvious candidates for developing further scenarios, but we can also
create new situations to bridge the gap in complexity between benchmark and
task. Finally, incorporating elements of noise (e.g. human models active in
the environment) and uncertainty (e.g. sensor readings) can help improve
generalisability through making the virtual testbed more similar to a real
world environment.

Looking beyond COVID-19, virtual benchmarks provide milestones and test cases
for team’s systems, thus providing a means for developers to test their system
without physical access. They can help improve the generalisation of a system
by making it easy to test across many virtual benchmarks. Finally, we believe
automation of the environment set up will allow teams to spend more time
focusing on their strategy, ultimately leading to more innovative systems.

## References

  * [1] “Service robot simulator,” https://www.openrobotics.org/blog/2018/5/22/service-robot-simulator, accessed: 2021-04-07.
  * [2] F. Pimentel and P. Aquino Junior, “Simulation system for learning of social rules using gazebo and openai gym,” in _III Brazilian Humanoid Robot Workshop and IV Brazilian Workshop on Service Robotics_ , 01 2020, pp. 79–83.
  * [3] N. Tsoi, M. Hussein, J. Espinoza, X. Ruiz, and M. Vázquez, “Sean: Social environment for autonomous navigation,” in _Proceedings of the 8th International Conference on Human-Agent Interaction_ , 2020, pp. 281–283.
  * [4] A. Eguchi, “Robocupjunior for promoting stem education, 21st century skills, and technological advancement through robotics competition,” _Robotics and Autonomous Systems_ , vol. 75, pp. 692–699, 2016.
  * [5] “Isr testbed simulated environment,” https://github.com/socrob/mbot_simulation_environments, accessed: 2021-04-07.