# Simulators for Mobile Social Robots: State-of-the-Art and Challenges

Prabhjot Kaur1, Zichuan Liu2, and Weisong Shi3  Department of Computer
Science, Wayne State University
Detroit, Michigan 48202
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The future robots are expected to work in a shared physical space with humans
[1], however, the presence of humans leads to a dynamic environment that is
challenging for mobile robots to navigate. The path planning algorithms
designed to navigate a collision free path in complex human environments are
often tested in real environments due to the lack of simulation frameworks.
This paper identifies key requirements for an ideal simulator for this task,
evaluates existing simulation frameworks and most importantly, it identifies
the challenges and limitations of the existing simulation techniques.

First and foremost, we recognize that the simulators needed for the purpose of
testing mobile robots designed for human environments are unique as they must
model realistic pedestrian behavior in addition to the modelling of mobile
robots. Our study finds that Pedsim_ros [2] and a more recent SocNavBench
framework [3] are the only two 3D simulation frameworks that meet most of the
key requirements defined in our paper. In summary, we identify the need for
developing more simulators that offer an ability to create realistic 3D
pedestrian rich virtual environments along with the flexibility of designing
complex robots and their sensor models from scratch.

###### Index Terms:

Simulation and Animation, Human-Aware Motion Planning, Modeling and Simulating
Humans, Collision Avoidance, Human-Centered Robotics.

##  I Introduction

The social Autonomous Mobile Robots (AMRs) include robots that can perceive
their environment, plan and execute a collision free trajectory to reach their
destination autonomously in complex human environments [4]. The current
forecast by the International Federation of Robotics (IFR) estimates the
market of mobile robots to grow on an average by 97% annually. Their presence
has already increased in response to COVID-19, with public safety and clinical
care being the two dominant applications in 2020 [5]. The other active
applications include usage in hotels [6], [7], factories [8], construction [9]
and hospitals [10]. Further, there has been an increased interest in using
robots to provide quality care for the elderly, either in their own homes [11]
or in the nursing homes [12] to meet the staggering shortage of care givers
[13]. With the current state of the technology, the AMRs are able to achieve
some level of autonomy in controlled environments, however, the real world is
dynamic and unpredictable which makes it challenging to scale up their
deployment in human environments.

This is where strategic testing plays an important role. Since the real world
testing is costly and unsafe, it can be complemented with simulation testing
which is cost effective, safer and reproducible. The simulation testing in the
field of mobile social robots is currently inadequate because of the
challenges in recreating realistic 3D scenarios and accurate modelling of
human behavior in simulation [3]. Considering the diverse applications of
robotics mentioned before, one would expect a simulator to be able to model
both indoor environments (such as hospitals, nursing homes, etc.) and outdoor
environments. Further, it must model different types of pedestrian behavior
including people on wheel chairs, with canes, crowds, etc. to make the virtual
environment representative of a real world. Luckily, the study of social
behavior of humans is a well-researched topic in psychology [14], which has
lead to the development of various mathematical models for pedestrian behavior
[15], [16]. These models generalize human behavior and they aim to predict
pedestrians’ next move. Some of these pedestrian models have made their way
into robotics where they are integrated with the path planning algorithms to
help the AMRs execute a path without colliding with humans. Our survey shows
that there is a lack of open-source simulators suitable for testing such
algorithms [15], [17].

So far, there are two 3D frameworks namely, Pedsim_ros and SocNavBench that
have been released as open source to fill the gap. However, these frameworks
have their own limitations which are discussed below. In this paper, we first
identify key requirements for building an effective simulator for testing AMRs
in pedestrian environments. Secondly, we study and evaluate existing
simulation techniques from the perspective of simulating robots and
pedestrians in a shared environment. Thirdly, we provide our analysis of the
current simulation frameworks and discuss the challenges with using existing
techniques.

This paper is organized into various sections. Section II provides background
on why a simulator for testing AMRs in a human rich environment is distinct
from a standalone robotic simulator or a pedestrian crowd simulator. Section
III identifies requirements for building a holistic simulator. Section IV
reviews and summarizes the current state-of-the art simulators. Section V
provides a thorough evaluation of these simulators through various experiments
that we conducted. Section VI discusses challenges that remain to be addressed
when it comes to building a holistic simulator for AMR testing in a human rich
environment. Section VII reviews related works. Finally, this paper concludes
in Section VIII.

##  II Motivation and Background

There are many robotic and pedestrian crowd simulators available today that
are intended for testing robotic systems and pedestrian/human models,
respectively. However, to test novel navigation algorithms for AMRs in
pedestrian rich environments, there is a need for a holistic simulator that
combines the capabilities of these two types of simulators. Currently, to test
these algorithms, the researchers rely on either testing directly in real
environments as in [18] and [19] or spend significant time in building their
own simulation environments as in [20] and [21]. Therefore, there is a clear
need to have more open source simulators that are available for researchers to
use for testing mobile robots in human rich environments. As mentioned before,
the simulator for such a testing environment is unique because one must model
not just the 3D virtual environment and a robot, but also the models for
pedestrians that are behaving realistically. In the next section, we define
key requirements that an ideal simulator for this purpose must meet.

##  III Requirements

While testing the AMR, there are many aspects of its performance that must be
tested, such as its perception functionality, its interaction with humans, its
path planning and control [3]. However, before we can even consider evaluating
these capabilities in simulation, one must construct a realistic 3D virtual
environment that includes a scene, a model of a robot including its sensors
and models of humans/pedestrians. Therefore, we start with identifying key
requirements that are essential to construct such an environment. These
requirements can then be used as a metric to compare which simulators are best
suited for the task at hand.

  1. 1.

3D virtual environment modelling: The simulator must support the creation of
any 3D virtual scene (both indoor and outdoor) such as a nursing home,
hospital, trade show set up, stadium, school campus, etc. in which to test and
deploy the mobile robot.

  2. 2.

Mobile robot modelling: The simulator must provide the flexibility of
modelling a mobile robot from scratch so that every aspect of the robot can be
tested thoroughly, such as its on-board sensors, sensor placement, and its
physical properties.

  3. 3.

Sensor modelling: The sensor suit for any mobile robot is highly dependent on
the navigation algorithm being used to guide a robot’s path. Some of the
common navigation algorithms use visual, ultrasonic, and laser sensors [22].
Since each type of physical sensor has its own intrinsic properties such as
its sensing range, noise and latency, the simulator must allow one to model a
realistic sensor for testing the social mobile robot.

  4. 4.

Human behavior modelling: Most importantly, the simulator must realistically
model virtual humans in the 3D environment. Therefore, the simulator must have
built-in models for modelling human behavior. This requirement is what makes
such simulators unique and challenging to construct.

  5. 5.

Integration with a robotic middleware: The simulator must support easy
integration with a robotic middleware. A middleware is a software framework
that resides between the operating system of the host machine and the
application layer. It handles lower level essential tasks such as cross
communication between various components of a complex system. Since robotic
systems are inherently complex, relying on a middleware to handle low level
tasks helps. As an example, the underlying processes involved with AMRs are
sensing, motion planning and control. These processes need to cross
communicate to achieve the goal of collision avoidance. The number of such
processes grows exponentially with the complexity of the task and the number
of robots in the scene.

Besides the requirements identified above, it is important that the simulator
is open source, scalable, modular and portable. As part of our review, various
simulators were studied while keeping the above requirements in mind. Our
survey shows that most traditional simulators are either specialized robot
simulators or are specialized pedestrian/crowd simulators [17]. However, there
are very few holistic simulators that are suitable for testing mobile social
robots. The details of these simulation frameworks are discussed in the
following section.

##  IV Simulators

This section is divided into four subsections. Each subsection reviews
simulation frameworks specializing in a certain task as identified in the key
requirements in Section III.

###  IV-A Simulators for 3D environment, robot and sensor modelling

The first and foremost requirement to test mobile robots is having a virtual
3D environment and a 3D model of the robot including its on-board sensors.
Some of the key characteristics for such simulators are that they must support
automatic 3D scene generation and must be open-source with wide community
support. Additionally, these simulators must be easy to integrate with robotic
middleware systems. This subsection describes various 3D simulators that
enable creation of virtual 3D environment, robot model and its sensors..

####  IV-A1 Gazebo/Ignition

It is one of the most popular multi-robot 3D simulator [23]. It enables the
users to generate both indoor and outdoor environments, along with the support
for modelling almost all kinds of robots such as humanoid [24], mobile,
underwater, aerial, etc.

The 3D scene in Gazebo environment is called a world which consists of several
static and dynamic objects called models. The world is defined by the user in
a Simulation Description File (SDF) format [25]. Further, the users are
empowered to customize every detail of the model such as its inertial, visual
or collision properties. All aspects of a 3D scene can be controlled through
plugins. [26] showcases the advanced capabilities of Gazebo where the user
defines a prius model of a car moving in a simulated M-City. These realistic
simulations are possible in Gazebo because of the three main libraries namely,
physics, rendering, and communication libraries [27]. Gazebo is the most
popular simulator in the robotic community because it seamlessly integrates
with Robotic Operating System (ROS) [28], [29]. This allows the users to
benefit from Gazebo’s 3D visualizations, along with the middleware support
from ROS.

Despite the numerous beneficial features of Gazebo, it is not the most
convenient tool to use. The construction of a 3D scene is a manual process.
One must collect models of various objects and include them in a world file
one by one. The time and effort spent on creating a 3D world grows with the
complexity of the 3D scene to be reconstructed and number of models in it. A
few researchers have attempted to automate the process. These approaches are
discussed below.

####  IV-A2 Height-map based automatic 3D world construction in Gazebo

[30] presents a method to automate the process of constructing a 3D virtual
environment (world) in Gazebo from a 2D map that is built using real sensor
data. The map is represented in the form of an occupancy grid, where a white
pixel represents a free space, black an occupied space and gray represents a
space that is not clearly defined. Once the map is obtained, it is processed
through two intermediate steps before a world file is generated for Gazebo.

In the first step, the occupancy grid map is filtered using a modified non-
linear filter to remove noise. Then, the filtered map is converted into a
grayscale image to prepare for the second step. In the second step, the
authors use heightmap element of the SDF [25], where the size and height for
the 2D space defined in an occupancy map is defined. The heightmap is simply a
2D array where each element of an array represents a height of a certain
region in a 2D plane. The resulting virtual environment has texture, height
and supports collision. Finally, this world file is launched in Gazebo
simulation. The authors validated the efficacy of the method by testing
various robot models in virtual environments that were constructed using the
defined method.

####  IV-A3 LIRS World Construction Tool (LIRS-WCT)

In [31], authors present a tool called LIRS-World Construction Tool (LIRS-WCT)
that automates the process of generating a world file for Gazebo from 2D
images or a 2D laser range data.

The conversion process occurs in three steps. The first step converts a grey
scale image to a 3D model. This is done by first transforming a 2D image into
a heightmap and then converting the heightmap into a COLLADA [53] format,
which is one of the mesh formats supported by Gazebo. Then in the second step,
texture is applied to the mesh where it gets the visual features. Finally,
during the third step of the procedure, a world file is generated which is
used to launch the model in Gazebo that was created in steps 1 and 2.

Abbyasov et al. [31] validated the efficacy of the tool by using it to create
virtual environments in Gazebo and using them to test various navigation
algorithms. The authors mention that the LIRS-WCT is more efficient than the
approach in [30] because of its added features such as scalability and it’s
relatively higher performance measured in terms of the real time factor in
Gazebo.

####  IV-A4 map2gazebo

The map2gazebo is an open source ROS package [32] that creates a Gazebo world
from a 2D map. A ROS node called map2gazebo subscribes to a topic where the
static map is being published and it creates a mesh from a static 2D map. The
3D mesh generated from a 2D map is made by extruding occupied pixels in the
occupancy grid map. Thus, the 3D Gazebo world generated is basically a
skeleton world with boxes without any texture. This may be sufficient to test
basic object avoidance models but if a more complicated environment is
desired, then the user must re-work the created world by adding other objects
to it.

####  IV-A5 USARSim

The Urban Search and Rescue Simulation (USARSim) [33] is an open source,
extensible, multi-robot, and flexible simulator which was originally developed
for urban and rescue type of robotic applications. It builds on an Unreal
gaming engine and supports all major operating systems. It is widely used
during the RoboCup competitions. There is also an interface that connects
USARSim with ROS called USARSimROS [34], however, it does not have a strong
community support like Gazebo and is not as well maintained as Gazebo.

###  IV-B Middleware for Robotics Development

The robotic middleware facilitates faster, collaborative, and efficient
development of robotic systems. The simulators discussed in the previous
section are often used in conjunction with the middleware frameworks. The
Robotic Operating System (ROS) is one the most popular middleware, however,
few other software frameworks that are similar to ROS also exist. We briefly
introduce them in this section.

####  IV-B1 Robotic Operating System (ROS)

It is an open source, robotic middleware that supports multiple languages such
as Python, Octave, C++ and LISP [35], [36]. It’s modular design approach makes
it suitable for use in peer to peer systems where multiple processes
implemented on heterogeneous systems are able to communicate with each other.
There are four basic elements in ROS architecture, namely the nodes, topics,
messages and services. The node is essentially a software process that is
designed to do a specific task. A ROS based system usually has many
specialized nodes that work together. Each node is capable of publishing and
subscribing data to and from other nodes through channels called topics. These
topics contain messages. The design choice of dividing the work amongst
various specialized nodes makes it easier to debug and it enables a
collaborative development.

ROS also offers logging and playback capability, thus making it easier to run
offline simulations with the real data. Additionally, it supports
visualization through a program called rviz. However, it is often used in
conjunction with other 3D simulators such as Gazebo for enhanced
visualizations [28], [29]. Finally, like Gazebo, ROS also has a huge community
that actively supports and contributes to the open source middleware. The ROS
community has released ROS2 recently.

####  IV-B2 LCM: Lightweight Communications and Marshalling

LCM [38] is a library designed to facilitate communication between various
robotic processes in real-time systems. LCM supports message definition that
is language agnostic. The messages are passed based on the publish/subscribe
mechanism which is based on the UDP multicast. There is no central master in
LCM unlike in ROS. [35] describes in detail how message mapping is achieved in
the absence of a central hub. The notable feature of LCM is that it also has a
tool to debug and inspect messages without any significant toll on the
efficiency and performance of the system.

It is important to note that LCM is not a full-fledged middleware like ROS. It
is mainly an enhanced message passing library that can be integrated with the
existing software framework.

####  IV-B3 ZeroMQ

This is yet another open source and popular library for communication purposes
[37]. It supports different types of message models for message delivery such
as a centralized approach where there is a central hub responsible for
coordinating message flow between various constituent processes, a peer to
peer coupling, a method where a central hub is used as a directory server and
also a distributed system that can have one or many central hubs.

###  IV-C Simulators for human behavior modelling

The simulators discussed in this section are called pedestrian simulators.
Their aim is to facilitate research in studying crowd behavior such as
passenger flow at the airports, railway stations, or for urban planning. There
are several commercial and few open source pedestrian simulators available in
the market. The basic building blocks of these simulators are the mathematical
pedestrian models, such as the social force model [16], [38], which simulate
how pedestrians generally move in a shared environment. While these simulators
are quite mature and are widely used in testing new pedestrian models, most of
them do not have a support for testing mobile robots in the same simulation
environment. This section reviews most popular pedestrian simulators.

####  IV-C1 Menge

It is an open source, cross platform and extensible simulator designed to
facilitate research in pedestrian crowd dynamics [39]. It is based on
microscopic models of pedestrians where each of the human models is considered
as an independent agent moving in a shared space. The problem of simulating
crowd dynamics is divided into several sub-problems, which are then
individually addressed. This makes Menge a modular and flexible framework.

The pipeline of various sub-problems include goal selection, path computation
and path adaptation. The plan adaptation step uses various pedestrian models
to compute a more realistic velocity called the feasible velocity which aims
to prevent collision with the fellow pedestrians in the environment. Menge
comes pre-loaded with various pedestrian path prediction models built into it.
These include PedVO [40], helbing [38], karamouzas [41], johansson [42]. It
also includes ORCA [40] as the collision avoidance model. Besides the elements
that are already part of the Menge library, it allows users to create their
own algorithms and plug into Menge’s architecture for the purpose of
evaluating their algorithms.

Menge is a versatile simulator for studying crowd dynamics, however, it is not
suited for testing mobile robots without the additional work.

####  IV-C2 PEDSIM

PEDSIM [45] is an open source simulator that is based on the social force
model [38] and the extended social force model. This is mainly used to study
crowd behavior of pedestrians in both indoor and outdoor environments. It is
based on a standalone C++ library called libpedsim, which is easy to extend.
Typically, the pedestrian trajectories calculated by PEDSIM are exported and
visualized in any rendering engine of user’s choice. It is capable of running
on both Linux and Windows operating systems. The key feature of PEDSIM is that
is easy to use and can be extended with less effort.

There are several other pedestrian simulators, such Mass Motion, Pedestrian
Dynamics, Legion, Simtread, Simwalk, Anylogic, Viswalk, Simped, and Alpsim.
These are reveiwed in [16] and [43].

###  IV-D Simulators for combined robot and human modelling

The simulators for testing mobile social robots are unique as they must
support all of the key requirements mentioned in Section III. This section
elaborates on the current state-of-the-art simulators that aim at a
facilitating simulation testing for social mobile robots in pedestrian rich
environments.

####  IV-D1 MengeROS

[17] extends Menge’s functionality by integrating Menge[39] with ROS[35] in
order to provide a simulation tool for testing mobile robots in pedestrian
rich environments. MengeROS is an open source ROS package. It is implemented
as a single ROS node where the pedestrians and mobile robots are both present.
This node takes in velocity as an input that moves the robot and provides
laser readings of the environment as an output. The robot avoids collisions
with pedestrians that are moving per Menge’s specifications after processing
the laser sensor readings.

It is important to note that MengeROS is a 2D simulator where the pedestrians
and mobile robots are represented as circles. This type of simulation
framework may suffice for testing general robot path planning purposes. If a
researcher is interested in gesture recognition, face detection or other
distinctive features of pedestrians, MengeROS is not enough.

####  IV-D2 menge_core

[44] is a work that is in experimental stages. It provides an open source
library that integrates Menge [39] with Gazebo [23] simulator. This library
serves as a crowd simulator that supports mobile robots. The author introduces
the concept of an external agent, which separates it from rest of the
simulation moving objects such as pedestrians. One of the important features
of this library is that it can represent robots and pedestrians as 3D objects.
This library is in developmental stages at the time of this writing.

####  IV-D3 Pedsim_ros

Pedsim_ros [2] is a ROS package that adapts PEDSIM simulator to be used in
conjunction with ROS [35]. It offers all the benefits of PEDSIM simulator
while also supporting robot integration with ROS. It also supports realistic
visualization of human models in Gazebo [23]. Pedsim_ros is used in [45] to
evaluate a navigation model for sidewalk robot navigation. Additionally, [46]
uses Pedsim_ros to model human behavior for the proposed SOCIALGYM 2D
simulation framework.

While Pedsim_ros certainly meets all the key requirements for a pedestrian
simulator mentioned in Section III, it is important to note that in order to
use this package, the user must create a 3D virtual environment and human
models in Gazebo from scratch. A second drawback is that Pedsim_ros supports a
single mathematical model for pedestrians unlike Menge [39] and MengeROS [17].

####  IV-D4 SocNavBench

SocNavBnech framework [3] is the most recent advancement in this area and as
such it takes a very unique approach to modelling pedestrians in simulation.
It uses real world pedestrian data to simulate human behavior as opposed to
relying on mathematical models, which are the backbone of all the simulators
(Sections IV-C and IV-D) built for this task so far. Additionally, the
simulation framework comes with a default mobile robot model. Furthermore, the
authors re-create virtual environments that correspond to the actual physical
locations where pedestrian data was originally recorded. The simulator has two
modes of operation, schematic and full render. The full render mode provides
photogenic/3D view of the physical environment and pedestrians. This type of
framework allows one to have a more realistic representation of a physical
environment to test navigation algorithms for mobile robots. As the pedestrian
activities are rooted in the real world data, it naturally contains the
diversity of activities that pedestrians are known to perform in public, such
as walking, running, crossing, grouping, etc. We believe SocNavBench is a
significant advancement in this area. However, it has few limitations that we
discuss in Sections V and VI.

In Section IV-D, we described the current state-of-the-art simulation
frameworks for testing mobile robots in pedestrian rich environments. We want
to highlight that besides these simulation frameworks, many researchers have
leveraged Gazebo simulator to create their own simulation environments. Some
of the notable works include [47] which introduces a method to integrate human
animation models developed using the Editor for Manual Work (EMA) tool into
Gazebo simulation. This framework extends the capabilities of Gazebo, while
retaining the other key features such as its ability to be integrated with
ROS. Additionally, [48] releases an open source plugin for simulating humans
in Gazebo environment. This plugin models human behavior as per the social
force model [38]. Furthermore, [49] also uses Gazebo to construct a simulator
to test robot navigation in dense pedestrian crowds.

##  V Evaluation

The simulators in Section IV-A specialize in simulating 3D virtual
environments, robot models and sensor models. Further, Section IV-C describes
the simulators that specialize in modelling pedestrian behavior. However, the
simulators for testing navigation algorithms for mobile robots in pedestrian
rich environments must have the combined capabilities of these two types of
simulators. Section IV-D sheds light on such simulators. We experimented with
some of the open source and the commonly used simulators mentioned above. In
this section, we provide our analysis as to which simulation frameworks are
best for creating 3D environments, robot and human models individually. Then,
we discuss which of them are suitable for testing AMRs in pedestrian rich
environments.

  1. 1.

For creating 3D virtual environments and various kinds of robot models, Gazebo
is the most preferred robotic simulator [24] due to its flexibility and easy
integration with ROS. The creation of 3D environment is a manual process in
Gazebo where each element of the environment is hand crafted. This allows one
to simulate different types of road terrains such as the ones with different
slopes, indoor versus outdoor environments, etc. While this offers a lot of
flexibility to control every aspect of the simulation, creation of such an
environment is a time consuming task. Fig. 1 shows an example of a 3D
environment of a museum constructed in Gazebo.

  2. 2.

To reduce the time consumption involved in creating 3D environments in Gazebo,
two open source packages, namely, LIRS-WCT [31] and map2gazebo [32] have been
released recently. These packages aim to automate the creation of 3D
environments in Gazebo. We tested both of these frameworks by re-creating a 3D
scene shown in Fig. 1. The results after automatic creation of this scene
using LIRS-WCT and map2gazebo are shown in Fig. 2 and 3, respectively. We
believe that map2gazebo package is easier to use and it also gives better
results. While these two packages do ease the work load for creating a 3D
scene, they require one to have a 2D map of the environment that needs to be
fed as an input. So, if one does not have a 2D map of the environment, then
these packages do not help.

  3. 3.

For robotic simulation, middleware plays a key role in facilitating lower
level communication between various processes. ROS is the preferred middleware
in robotics. It is well maintained and can handle complex robotic systems. ROS
does come with its own visualization tool called rviz, however, it is often
used in conjunction with Gazebo simulator to provide more realistic 3D
simulations.

  4. 4.

Finally, for testing AMRs in pedestrian rich environments, some of the
traditional pedestrian simulators described in Section IV-C have been extended
by the robotics community for further use. MengeROS [17] and Pedsim_ros are
two such simulators. These have been released as open source ROS packages to
test AMRs in pedestrian rich environments. MengeROS and Pedsim_ros use
traditional pedestrian simulators Menge and PEDSIM, respectively, to model
pedestrian behavior, while relying on Gazebo and ROS for modelling robots and
their sensors in a 3D virtual environment. Additionally, SocNavBench is a
recent add to the community. We attempted to test both MengeROS (a 2D
simulator) and Pedsim_ros (a 3D simulator). While MengeROS was not testable as
it is way behind the current ROS release. Additionally, this package does not
appear it is being actively maintained. However, we were able to successfully
test the performance of Pedsim_ros. Next, we provide a summary of its
capabilities and performance.

As the name suggests, Pedsim_ros integrates PEDSIM simulator with ROS.
Additionally, it also has a plugin to connect with Gazebo. This gives user a
powerful combination that meets all of the key requirements mentioned in
Section III above. Fig. 4 shows an example of an airport environment in rviz
where pedestrians are moving in crowds. The pedestrian behavior is as per the
mathematical model called the social force model [38], that is pre-programmed
in the PEDSIM simulator. Therefore, if researchers are interested in designing
and testing a mobile robot in such an environment, they can focus on a
specific task such as designing a robot that maintains a collision free path
without having to model the pedestrian behavior themselves manually. Further,
to evaluate the Gazebo plugin, we can see the comparison of the 3D scene with
pedestrians in rviz and Gazebo simultaneously in Fig. 5. (Note: The tall boxes
in the simulation correspond to human models.) Finally, in Fig. 6, we show a
3D model of a robot along with the pedestrian models (tall boxes) in Gazebo.
The robot model shown is constructed in Gazebo from scratch. We believe that
Pedsim_ros is a great start for testing AMRs in pedestrian environments as it
allows users to focus on modeling robot behavior while it takes care of the
pedestrian modelling itself. This is especially true if the robot path
planning and control algorithms solely rely on predicted pedestrian
trajectories. However, if a navigation algorithm relies other types of human
robot interactions such as human body orientation, skeleton tracking, gestures
or gaze estimation [50], then Pedsim_ros is not an ideal choice.

To overcome the shortcomings of existing simulators, SocNavBench [3], a very
recent add to the simulation frameworks in this area, offers added features
such as 3D rendering of human figures. The human meshes used in the simulation
represent humans of different sizes and shapes, performing diverse activities.
Additionally, the human trajectories are derived from the real world data,
which makes the scene more representative of a real environment. The current
shortcomings of SocNavBench are that it is not integrated with ROS and does
not support creation of robot models and sensor models from scratch as in
Gazebo. We believe that ability to integrate with ROS is critical as ROS is
the most widely used middleware in the robotics community. Additionally,
creation of robot and sensor models from scratch is also essential as robots
come in different forms, equipped with wide variety of sensors. It is
pertinent to be able to test the kinematics and sensor performance for a
specific robot type in a simulated environment.

See Table 1 for comparison of various simulators reviewed as part of our
survey.

Figure 1: A 3D scene of a museum in Gazebo. Figure 2: A 3D model of the scene in Fig.3 reconstructed using LIRS-WCT. Figure 3: A 3D model of the scene in Fig.3 reconstructed using map2gazebo. Figure 4: 3D simulation of pedestrians moving in an airport scenario using the pedsim_ros package. Figure 5: 3D simulation of pedestrians in rviz and Gazebo using the pedsim_ros package. The tall boxes in the image represent human models. Figure 6: 3D simulation of pedestrians (tall boxes) along with a custom designed 3D model of a robot in Gazebo. TABLE I: A Comparison of simulation frameworks Simulator |  3D environment modelling |  Robot modelling |  Sensor modelling |  Integration with robotic middleware |  Human modelling
---|---|---|---|---|---
Gazebo/ Ignition |  Y |  Y |  Y |  Y, with ROS |  Y, limited ability
USARSim |  Y |  Y |  N |  Y, with ROS |  N
Menge |  N |  N |  N |  N |  Y, limited to 2D
PEDSIM |  N |  N |  N |  N |  Y
MengeROS |  N |  Y, limited to 2D |  N |  Y, with ROS |  Y,limited to 2D
Pedsim_ros |  Y, with Gazebo interface |  Y, with Gazebo interface |  Y, with Gazebo interface |  Y, with ROS |  Y, supports 3D
SocNavBench |  Y |  N, supports a default model |  N |  N |  Y
aY=Yes, N=No. |  |

##  VI Challenges

This section discusses challenges and limitations of the existing simulators.

  1. 1.

Inability to dynamically switch between pedestrian models: All pedestrian
simulators except the SocNavBench framework rely on fixed mathematical models
(either, microscopic or macroscopic) to simulate pedestrian behavior. This
implies that the simulators model pedestrian dynamics either at the crowd
level or at the individual level. However, the amount of information required
to asses a potential human-robot collision at any time is highly dependent on
the distance between them at that time. Therefore, it is necessary to model
pedestrians using both, the macroscopic and macroscopic methods, depending on
the immediate situation around the robot. As an example, when the human crowd
is at a farther distance from the robot, it is sufficient to use macroscopic
mathematical models which define the properties of the crowd such as its
velocity and position. This would give enough information to guide robot’s
next steps as shown in [51]. However, as the crowd gets closer to the robot in
terms of its distance, the microscopic pedestrian models are more suitable to
provide the details of each pedestrian’s dynamics. Currently, no simulator
supports dynamic switching between macroscopic and microscopic pedestrian
models as a response to the changing environment around the robot.
SocialNavBench framework has an added advantage here as it uses real
pedestrian data to model pedestrian trajectories. Threfore, the different
types of pedestrian dynamics, such as the corwd behavior versus an individual
behavior are intrinsically built in the simulated environment.

  2. 2.

Absence of modelling of human behavior in the presence of a robot: All
simulators in Section IV-C and IV-D have a common limitation. While these
models are capable of modelling human interactions at the individual level and
at the group level, they do not incorporate human behavior in the presence of
a robot. The human behavior changes during the presence of a robot due to its
novelty [52], [53]. No simulator currently models such interactions. This
limits robot’s ability to plan its path in a real environment where people are
expected to form crowds around a robot.

  3. 3.

Lack of diverse pedestrian models: All simulators except SocNavBench suffer
from homogeneity as they deploy one fixed mathematical model for all
pedestrians in the scene. Since the behavior of humans differ depending upon
their age, demographics, gender and culture [54], [55], [56], a single
pedestrian model applied to all human models is not a realistic representation
of human population, especially in the public places like hospitals, airports,
hotels, etc. SocNavBench is an exception as it is rooted in the real
pedestrian data for simulating pedestrian behavior.

  4. 4.

Absence of modelling of robot to robot interactions: It is expected for public
places such as hospitals, nursing homes, airports, and construction sites to
have multiple robots from different manufacturers working together. In order
for them to work efficiently, they must be able to communicate with each
other. An example of issues where robots from different vendors came to a halt
because they did not know how to respond to each other is recently reported
from the Changi General Hospital in Singapore, which has about 50 robots.
Therefore, testing a stand alone AMR in an environment is not sufficient,
especially if it is intended for a site that is likely to have robots from
other manufacturers in its vicinity.

  5. 5.

Absence of various visual cues for navigation such as human gaze/head/body
orientation:  Some robotic algorithms rely on visual cues [57] such as human
gaze or head and body orientation to do the path planning [58], [59]. The
current simulators do not support testing such algorithms because they either
model pedestrians as 2D, or 3D human models with minimal to no facial
features. The importance of such navigation cues is evident from the the live
images of pedestrians shown in Fig. 7 and Fig. 8. These images are from Times
Square in New York as captured by the EarthCam network. After careful
observation, a human can easily distinguish various different social groups as
well as the individuals that do not belong to any group. Such distinction is
possible by examining the type of interaction occurring between individuals
and other cues such as the orientation of their head/body. This type of
granularity is currently lacking in the simulators we have today.

  6. 6.

Absence of ambient sound modelling in the virtual scene: Additionally, some
robots use environmental sound or human speech for navigation [60], [61],
[62], and [63]. Currently, there is no simulator for testing sound based
navigation algorithms..

  7. 7.

Lack of realistic noise modelling in simulation: Although some of the
simulators such as Gazebo allow one to model sensor noise for various sensors
such as for lidar and camera [64], [30], there are many types of noises that
currently cannot be modelled in simulation. This includes ambient noise for
the acoustic input based navigation algorithms [65] or a noise originating
from the robot itself through its motors and fans [66]. A proper amount of
noise modelling in the simulation environment will help fill the current gap
between real and the simulated environments [49].

  8. 8.

Lack of automatic 3D scene generation:  [67] presents ten challenges of using
simulators that roboticists currently face based on the survey that involved
82 roboticists from various backgrounds. The data shows that since the
construction of a simulation environment with the current methods is a tedious
and time-consuming task, the researchers hesitate using simulators all
together. In order to enhance simulation testing and make it more efficient,
there is a need for more open source models of various 3D environments. This
will help the entire robotic community to design complex scenarios without re-
inventing the wheel every time.

Figure 7: Live footage of pedestrians at Times Square, NYC as captured by
EarthCam. The annotated rectangular boxes and ovals represent groups of
pedestrians and individuals that do not belong to any group, respectively.

Figure 8: Live footage of pedestrians at Times Square, NYC as captured by
EarthCam. The annotated rectangular boxes and ovals represent groups of
pedestrians and individuals that do not belong to any group, respectively.

##  VII Related Works

There are few other 3D simulators like Gazebo that are presented in [23],
[68], [69] and [70]. [71] also provides a very detailed comparison of Gazebo
with other alike simulators such as V-REP and ARGoS. Similarly, a
comprehensive comparison of various robotic middleware is provided in [72],
[35], and [73]. [72] compares RT-Middleware, Robotic Operating System (ROS),
Open Software Platform for Robotics Services (OPRoS), and Orocos based on
their capabilities such as the support for various operating systems, real
time operability, programming languages supported and their respective
simulation environments. A comparison of Player, ROS, Microsoft Robotics
Studio (MSRS), Webots, Robot Construction Kit (ROCK), iRobot Aware, Python
Robotics (Pyro), Carmen (Robot Navigation Toolkit) can be found in [73].
Further, [16] and [43] provide a survey on various pedestrian simulators.
Finally, [74] introduces a new framework called SEAN-EP which is an
experimental platform that allows human in the loop robot navigation.

##  VIII Conclusion

This paper reviewed available simulation tools to test social mobile robots in
pedestrian rich environments. It is concluded that there exist various
simulators such as Gazebo that specialize in testing robot path planning
algorithms in static environments. Further, there also exist many pedestrian
simulators that specialize in testing pedestrian behavior models. However,
there are very few simulation frameworks for testing mobile robots in human
environments. According to our study, SocNavBench and Pedsim_ros are the only
3D frameworks currently suitable for testing social AMRs. Further, the paper
identifies several challenges and limitations that the current simulation
techniques have. There is a need for more open source simulators that will
enable effective testing of AMRs. This can help accelerate the deployment of
mobile robots in complex human environments.

## References

  * [1] L. Major and J. Shah, _What To Expect When You’re Expecting Robots: The Future of Human-Robot Collaboration_ , ser. The future of human-robot collaboration. New York: Basic Books, 2020\.
  * [2] “Pedsim_ros,” https://github.com/srl-freiburg/pedsim_ros, Online; accessed: 11-October-2020.
  * [3] A. Biswas, A. Wang, G. Silvera, A. Steinfeld, and H. Admoni, “Socnavbench: A grounded simulation testing framework for evaluating social navigation,” _ACM Transactions on Human-Robot Interaction (THRI)_ , August 2021.
  * [4] T. P. Fries, _Evolutionary Path Planning for Robot Navigation Under Varying Terrain Conditions_ , ser. In Mobile Robots: The Evolutionary Approach, Springer Berlin Heidelberg. Springer, Berlin, Heidelberg, 2007.
  * [5] R. R. Murphy, V. B. M. Gandudi, and J. Adams, “Applications of robots for covid-19 response,” _arXiv preprint arXiv:2008.06976_ , 2020.
  * [6] Y. Choi, M. Choi, M. Oh, and S. Kim, “Service robots in hotels: understanding the service quality perceptions of human-robot interaction,” _Journal of Hospitality Marketing & Management_, vol. 29, no. 6, pp. 613–635, 2020.
  * [7] S. S. Kim, J. Kim, F. Badu-Baiden, M. Giroux, and Y. Choi, “Preference for robot service or human service in hotels? impacts of the covid-19 pandemic,” _International Journal of Hospitality Management_ , vol. 93, p. 102795, 2021\.
  * [8] M. Schneier, M. Schneier, and R. Bostelman, _Literature review of mobile robots for manufacturing_. US Department of Commerce, National Institute of Standards and Technology …, 2015\.
  * [9] H. Ardiny, S. Witwicki, and F. Mondada, “Are autonomous mobile robots able to take over construction? a review,” 2015.
  * [10] H. S. Ahn, M. H. Lee, and B. A. MacDonald, “Healthcare robot systems for a hospital environment: Carebot and receptionbot,” in _2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)_ , 2015, pp. 571–576.
  * [11] H. Robinson, B. MacDonald, and E. Broadbent, “The role of healthcare robots for older people at home: A review,” _International Journal of Social Robotics_ , vol. 6, no. 4, pp. 575–591, 2014.
  * [12] H. Aloulou, M. Mokhtari, T. Tiberghien, J. Biswas, C. Phua, J. H. K. Lin, and P. Yap, “Deployment of assistive living technology in a nursing home environment: methods and lessons learned,” _BMC medical informatics and decision making_ , vol. 13, no. 1, pp. 1–17, 2013.
  * [13] M. Miller, “The future of us caregiving: High demand scare workers,” _The Thomson Reuters_ , 2017.
  * [14] P. Wang, “Understanding social-force model in psychological principles of collective behavior,” in _arXiv preprint arXiv: abs/1605.05146, 2016_ , 2016\.
  * [15] F. Camara, N. Bellotto, S. Cosar, F. Weber, D. Nathanael, M. Althoff, J. Wu, J. Ruenz, A. Dietrich, G. Markkula, A. Schieben, F. Tango, N. Merat, and C. Fox, “Pedestrian models for autonomous driving part ii: High-level models of human behavior,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–20, 2020.
  * [16] C. CARAMUTA, G. COLLODEL, C. GIACOMINI, C. Gruden, G. LONGO, and P. Piccolotto, “Survey of detection techniques, mathematical models and simulation software in pedestrian dynamics,” 2017.
  * [17] A. Aroor, S. L. Epstein, and R. Korpan, “Mengeros: A crowd simulation tool for autonomous robot navigation,” _arXiv preprint arXiv:1801.08823_ , 2018.
  * [18] C. Shi, M. Shiomi, C. Smith, T. Kanda, and H. Ishiguro, “A model of distributional handing interaction for a mobile robot.” in _Robotics: science and systems_ , 2013, pp. 24–28.
  * [19] M. Shiomi, F. Zanlungo, K. Hayashi, and T. Kanda, “Towards a socially acceptable collision avoidance for a mobile robot navigating among pedestrians using a pedestrian model,” _International Journal of Social Robotics_ , vol. 6, no. 3, pp. 443–455, 2014.
  * [20] W. Zhi, T. Lai, L. Ott, and F. Ramos, “Anticipatory navigation in crowds by probabilistic prediction of pedestrian future movements,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 8459–8464.
  * [21] E. Chebotareva, K.-H. Hsia, K. Yakovlev, and E. Magid, “Laser rangefinder and monocular camera data fusion for human-following algorithm by pmb-2 mobile robot in simulated gazebo environment,” in _Proceedings of 15th International Conference on Electromechanics and Robotics” Zavalishin’s Readings”_. Springer, 2021, pp. 357–369.
  * [22] R. Siegwart, I. R. Nourbakhsh, and D. Scaramuzza, _Introduction to autonomous mobile robots_. MIT press, 2011\.
  * [23] N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in _2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566)_ , vol. 3. IEEE, 2004, pp. 2149–2154.
  * [24] S. Ivaldi, J. Peters, V. Padois, and F. Nori, “Tools for simulating humanoid robot dynamics: a survey based on user feedback,” in _2014 IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2014, pp. 842–849.
  * [25] “Simulation description format,” http://sdformat.org/spec, Online; accessed: 01-November-2020.
  * [26] “Demo of prius in ros/gazebo,” https://github.com/osrf/car_demo, Online; accessed: 01-November-2020.
  * [27] P. Kaur, S. Taghavi, Z. Tian, and W. Shi, “A survey on simulators for testing self-driving cars,” _arXiv preprint arXiv:2101.05337_ , 2021.
  * [28] K. Takaya, T. Asai, V. Kroumov, and F. Smarandache, “Simulation environment for mobile robots testing using ros and gazebo,” in _2016 20th International Conference on System Theory, Control and Computing (ICSTCC)_. IEEE, 2016, pp. 96–101.
  * [29] W. Yao, W. Dai, J. Xiao, H. Lu, and Z. Zheng, “A simulation system based on ros and gazebo for robocup middle size league,” in _2015 IEEE international conference on robotics and biomimetics (ROBIO)_. IEEE, 2015, pp. 54–59.
  * [30] R. Lavrenov and A. Zakiev, “Tool for 3d gazebo map construction from arbitrary images and laser scans,” in _2017 10th International Conference on Developments in eSystems Engineering (DeSE)_. IEEE, 2017, pp. 256–261.
  * [31] B. Abbyasov, R. Lavrenov, A. Zakiev, K. Yakovlev, M. Svinin, and E. Magid, “Automatic tool for gazebo world construction: from a grayscale image to a 3d solid model,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 7226–7232.
  * [32] S. Curtis, “Ros package for creating gazebo environments from 2d maps,” https://github.com/shilohc/map2gazebo, Online; accessed: 17-October-2020.
  * [33] S. Carpin, M. Lewis, J. Wang, S. Balakirsky, and C. Scrapper, “Usarsim: a robot simulator for research and education,” in _Proceedings 2007 IEEE International Conference on Robotics and Automation_. IEEE, 2007, pp. 1400–1405.
  * [34] S. Balakirsky and Z. Kootbally, “Usarsim/ros: A combined framework for robotic control and simulation,” in _International Symposium on Flexible Automation_ , vol. 45110. American Society of Mechanical Engineers, 2012, pp. 101–108.
  * [35] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “Ros: an open-source robot operating system,” in _ICRA workshop on open source software_ , vol. 3, no. 3.2. Kobe, Japan, 2009, p. 5.
  * [36] L. Joseph, _Robot Operating System (ROS) for Absolute Beginners_. Springer, 2018.
  * [37] M. Sústrik _et al._ , “Zeromq,” _Introduction Amy Brown and Greg Wilson_ , 2015.
  * [38] D. Helbing and P. Molnar, “Social force model for pedestrian dynamics,” _Physical review E_ , vol. 51, no. 5, p. 4282, 1995.
  * [39] S. Curtis, A. Best, and D. Manocha, “Menge: A modular framework for simulating crowd movement,” _Collective Dynamics_ , vol. 1, pp. 1–40, 2016.
  * [40] J. Van Den Berg, S. J. Guy, M. Lin, and D. Manocha, “Reciprocal n-body collision avoidance,” in _Robotics research_. Springer, 2011, pp. 3–19.
  * [41] I. Karamouzas, P. Heil, P. Van Beek, and M. H. Overmars, “A predictive collision avoidance model for pedestrian simulation,” in _International workshop on motion in games_. Springer, 2009, pp. 41–52.
  * [42] D. Helbing, A. Johansson, and H. Z. Al-Abideen, “Dynamics of crowd disasters: An empirical study,” _Physical review E_ , vol. 75, no. 4, p. 046109, 2007\.
  * [43] P. M. Kielar, D. H. Biedermann, and A. Borrmann, “Momentumv2: A modular, extensible, and generic agent-based pedestrian behavior simulation framework,” 2016.
  * [44] “Menge library for large scale crowd simulation,” https://github.com/osrf/menge_core, Online; accessed: 11-October-2020.
  * [45] Y. Du, N. J. Hetherington, C. L. Oon, W. P. Chan, C. P. Quintero, E. Croft, and H. M. Van der Loos, “Group surfing: A pedestrian-based approach to sidewalk robot navigation,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6518–6524.
  * [46] J. Holtz and J. Biswas, “Socialgym: A framework for benchmarking social robot navigation,” _arXiv preprint arXiv:2109.11011_ , 2021.
  * [47] L. He, P. Glogowski, K. Lemmerz, B. Kuhlenkötter, and W. Zhang, “Method to integrate human simulation into gazebo for human-robot collaboration,” in _IOP Conference Series: Materials Science and Engineering_ , vol. 825, no. 1. IOP Publishing, 2020, p. 012006\.
  * [48] L. Tai, J. Zhang, M. Liu, and W. Burgard, “Socially compliant navigation through raw depth inputs with generative adversarial imitation learning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1111–1117.
  * [49] J. Liang, U. Patel, A. J. Sathyamoorthy, and D. Manocha, “Crowd-steer: Realtime smooth and collision-free robot navigation in densely crowded scenarios trained using high-fidelity simulation,” in _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2021, pp. 4221–4228.
  * [50] Y. Mohamed and S. Lemaignan, “Ros for human-robot interaction,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 3020–3027.
  * [51] A. Wang, C. Mavrogiannis, and A. Steinfeld, “Group-based motion prediction for navigation in crowded environments,” in _Conference on Robot Learning_. PMLR, 2022, pp. 871–882.
  * [52] H. Kidokoro, T. Kanda, D. Brščić, and M. Shiomi, “Simulation-based behavior planning to prevent congestion of pedestrians around a robot,” _IEEE Transactions on Robotics_ , vol. 31, no. 6, pp. 1419–1431, 2015.
  * [53] C. Chen, Y. Liu, S. Kreiss, and A. Alahi, “Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6015–6022.
  * [54] J. V. Hurtado, L. Londoño, and A. Valada, “From learning to relearning: A framework for diminishing bias in social robot navigation,” _Frontiers in Robotics and AI_ , vol. 8, p. 69, 2021.
  * [55] Y. Gao and C.-M. Huang, “Evaluation of socially-aware robot navigation,” _Frontiers in Robotics and AI_ , p. 420, 2022.
  * [56] C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh, “Core challenges of social robot navigation: A survey,” _arXiv preprint arXiv:2103.05668_ , 2021.
  * [57] V. Tolani, S. Bansal, A. Faust, and C. Tomlin, “Visual navigation among humans with optimal control as a supervisor,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2288–2295, 2021.
  * [58] Y. Chen, C. Liu, B. E. Shi, and M. Liu, “Robot navigation in crowds by graph convolutional networks with attention learned from human gaze,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2754–2761, 2020.
  * [59] R. Singh, T. Miller, J. Newn, L. Sonenberg, E. Velloso, and F. Vetere, “Combining planning with gaze for online human intention recognition,” in _Proceedings of the 17th international conference on autonomous agents and multiagent systems_ , 2018, pp. 488–496.
  * [60] K. Nakadai, T. Lourens, H. G. Okuno, and H. Kitano, “Active audition for humanoid,” in _AAAI/IAAI_ , 2000, pp. 832–839.
  * [61] H. Liu and M. Shen, “Continuous sound source localization based on microphone array for mobile robots,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 4332–4339.
  * [62] C. Rascon and I. Meza, “Localization of sound sources in robotics: A review,” _Robotics and Autonomous Systems_ , vol. 96, pp. 184–210, 2017.
  * [63] A. Schmidt, H. W. Löllmann, and W. Kellermann, “Acoustic self-awareness of autonomous systems in a world of sounds,” _Proceedings of the IEEE_ , vol. 108, no. 7, pp. 1127–1149, 2020.
  * [64] F. Furrer, M. Burri, M. Achtelik, and R. Siegwart, “Rotors—a modular gazebo mav simulator framework,” in _Robot operating system (ROS)_. Springer, 2016, pp. 595–625.
  * [65] E. Martinson and R. C. Arkin, “Noise maps for acoustically sensitive navigation,” in _Mobile Robots XVII_ , vol. 5609. International Society for Optics and Photonics, 2004, pp. 50–60.
  * [66] E. Martinson, “Hiding the acoustic signature of a mobile robot,” in _2007 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2007, pp. 985–990.
  * [67] A. Afzal, D. S. Katz, C. L. Goues, and C. S. Timperley, “A study on the challenges of using robotics simulators for testing,” _arXiv preprint arXiv:2004.07368_ , 2020.
  * [68] A. Staranowicz and G. L. Mariottini, “A survey and comparison of commercial and open-source robotic simulator software,” in _Proceedings of the 4th International Conference on PErvasive Technologies Related to Assistive Environments_ , 2011, pp. 1–8.
  * [69] M. Torres-Torriti, T. Arredondo, and P. Castillo-Pizarro, “Survey and comparative study of free simulation software for mobile robots,” _Robotica_ , vol. 34, no. 4, p. 791, 2016.
  * [70] P. Castillo-Pizarro, T. V. Arredondo, and M. Torres-Torriti, “Introductory survey to open-source mobile robot simulation software,” in _2010 Latin American Robotics Symposium and Intelligent Robotics Meeting_. IEEE, 2010, pp. 150–155.
  * [71] L. Pitonakova, M. Giuliani, A. Pipe, and A. Winfield, “Feature and performance comparison of the v-rep, gazebo and argos robot simulators,” in _Annual Conference Towards Autonomous Robotic Systems_. Springer, 2018, pp. 357–368.
  * [72] G. Magyar, P. Sinčák, and Z. Krizsán, “Comparison study of robotic middleware for robotic applications,” in _Emergent Trends in Robotics and Intelligent Systems_. Springer, 2015, pp. 121–128.
  * [73] E. Tsardoulias and P. Mitkas, “Robotic frameworks, architectures and middleware comparison,” _arXiv preprint arXiv:1711.06842_ , 2017.
  * [74] N. Tsoi, M. Hussein, O. Fugikawa, J. Zhao, and M. Vázquez, “Sean-ep: A platform for collecting human feedback for social robot navigation at scale,” _arXiv preprint arXiv:2012.12336_ , 2020.


11institutetext: University of Minnesota – Twin Cities
11email: {helle246, rickm014, cjweight<EMAIL_ADDRESS>
# The Role of Publicly Available Data
in MICCAI Papers from 2014 to 2018

Nicholas Heller  Jack Rickman  Christopher Weight
Nikolaos Papanikolopoulos

###### Abstract

Widely-used public benchmarks are of huge importance to computer vision and
machine learning research, especially with the computational resources
required to reproduce state of the art results quickly becoming untenable. In
medical image computing, the wide variety of image modalities and problem
formulations yields a huge task-space for benchmarks to cover, and thus the
widespread adoption of standard benchmarks has been slow, and barriers to
releasing medical data exacerbate this issue. In this paper, we examine the
role that publicly available data has played in MICCAI papers from the past
five years. We find that more than half of these papers are based on private
data alone, although this proportion seems to be decreasing over time.
Additionally, we observed that after controlling for open access publication
and the release of code, papers based on public data were cited over 60% more
per year than their private-data counterparts. Further, we found that more
than 20% of papers using public data did not provide a citation to the dataset
or associated manuscript, highlighting the ”second-rate” status that data
contributions often take compared to theoretical ones. We conclude by making
recommendations for MICCAI policies which could help to better incentivise
data sharing and move the field toward more efficient and reproducible
science.

##  1 Introduction

With the proliferation of Deep Learning (DL) methods in medical image
computing, a large proportion of papers presented at the International
Conference on Medical Image Computing and Computer Assisted Interventions
(MICCAI) are now based on large-scale medical imaging datasets which are often
expensive and time-consuming to collect. In the broader computer vision
community, there is a trend toward the use of standardized benchmarks such as
CIFAR [11], ImageNet [4], and MSCOCO [12] which allows for researchers to
objectively compare their methods to the state of the art without having to
repeat the experiments of others–a time-consuming and expensive endeavor on
its own. At MICCAI, this has seen only modest adoption, possibly due to the
exceptional diversity of imaging modalities and target variables [8], and the
corresponding dearth of publicly available benchmarks.

The practice of publicly releasing research data, especially in a recognized
archive accompanied by a detailed data descriptor, is a promising avenue for
expanding the number and variety of medical imaging benchmarks. Medical data
inherently has more barriers to publication (e.g. ethics standards in human
subjects research, risks of leaking protected health information) than other
scientific data, but these are typically not insurmountable, especially with
organizations such as The Cancer Imaging Archive [2] now offering support in
this area. Public datasets free machine learning researchers to focus their
attention and resources on methods, and it frees their peers from having to
replicate both the dataset and analysis when conducting reproducibility
studies or building on their work.

In this paper, we explore the evolution of this practice at each MICCAI
conferences of the past five years. In particular, we report the prevalence of
papers based on public data vs. those based on only private data. We found
that more than half of all MICCAI papers about machine learning for computer
vision are based on private data alone, which is anomalously high for ML/CV
literature. We also use each paper’s citation count per year elapsed as a
surrogate for its impact on the field and show that papers using public data
are cited roughly 60% more than their private data counterparts, even when
controlling for open access publication and release of code. We conclude with
recommendations for how these data can inform policies to increase the impact
of the MICCAI conference and the field of medical image computing as a whole.

##  2 Related Work

There is a large and diverse body of literature that shows the benefit of
sharing research data. In a 2013 article, Piwowar and Vision [13] succinctly
articulated the many benefits to publishing your data in the biomedical field:

  * … sharing data encourages multiple perspectives, helps to identify errors, discourages fraud, is useful for training new researchers, and increases efficient use of funding and patient population resources by avoiding duplicate data collection.

They then went on to provide the results of a thorough and well-controlled
experiment which showed that papers that released their associated gene
expression microarray data were cited nearly 10% more than their counterparts
that kept the same data private.

In 2016, Drachen et al [6] conducted a similar bibliometric analysis for three
astrophysics journals between 2000 and 2014. They found that papers which
linked to a dataset were cited 25% more than those that did not, and
interestingly, when they restricted their analysis to only papers from 2009 to
2014, the effect size increased to 40%, which suggests that this issue is
becoming more pronounced over time.

Very recently, Colavizza et al [3] conducted a text mining and citation
analysis of more than half a million papers published by PLOS and BMC that
were also part of the PubMed Open Access Collection. These journals are
interesting cases because they each recently enacted policies requiring
authors to include a Data Availability Statement (DAS) belonging to one of
three categories: (1) ”data is available on request”, (2) ”data is contained
within the article or supplementary material”, and (3) ”data is in a public
repository and here is the link”. Their results showed that papers with a
category 3 DAS were cited over 25% more than those with categories 1, 2 or no
DAS at all. Their work also showed that such DAS statements can very quickly
be made commonplace by a change in journal policy.

These works paint a clear picture that data publication is strongly associated
with receiving a higher number of citations, but what about simply using
public data? To our knowledge, there are no published studies looking at the
citation advantage to using publicly available data rather than private data.

##  3 Methods

###  3.1 Data Collection

For each of the five MICCAI events from 2014 to 2018, we randomly ordered all
accepted papers, manually iterated through them and selected all papers that
made use of machine learning for a computer vision task until we had accrued
100 papers from each year. We henceforth refer to such papers as ”MICCAI CV/ML
papers”. We then manually collected the following information about each
paper:

  * •

Citation count according to Google Scholar

  * •

Whether they used public data, and if so, which public dataset(s) they used

  * •

How they referenced the public data they used, if applicable (e.g. citation,
footnote URL, or just a name in the text)

  * •

Whether they released the private data they used, if applicable

  * •

Whether they released their source code

  * •

Whether the paper is available open access (e.g. through a preprinting server,
through a funding agency, or on an author’s homepage)

The release of code [17] and making the paper open access [9] are known to
independently associate with high citation counts, so we collected these
attributes to control for potential confounding effects.

At the start of this study, we originally did not collect the manner in which
data was referenced. It was only after we noticed a surprising number of non-
citation references that we decided to go back and record this for our sample.

###  3.2 Statistical Analysis

In order to make our study of citation counts robust to the very few papers
with an exceptionally high number of citations, we used Winsorization [5]. In
particular, we trimmed each paper’s citation rate to 50 per year. Two papers
were affected by this [1, 14] and the Winsorization affected neither the
direction nor the significance of the results.

We are interested in estimating the mean citation advantage to using public
data vs using only private data, after controlling for release of code and
open access publication. Regression is not suitable in this case since the
prerequisite of normal residuals fails badly. Prior works [16] have used OLS
to predict the log of citations/year plus one, but this dramatically
underestimates effect sizes when there is a high prevalence of low-citation
papers, as there is in our case. Luckily, since we are controlling only for
two binary variables, we can stratify papers into four groups without loss of
precision: (1) no code release and no open access, (2) code release but not
open access, (3) no code release but open access, and (4) code release and
open access. Within each stratum, we compute a ratio of the mean citations per
year of papers using public data to that of papers that used only private
data. We then aggregate these ratios with a weighted sum according to their
prevalence. In order to estimate a confidence interval for this ratio, we use
the bootstrap [7].

##  4 Results and Discussion

###  4.1 More than half of papers used only private data

Of the 500 papers we reviewed, 271 (54.2%) used only privately available data.
It does appear, however, that even within our short study period, this
practice has become less common, down from 64.0% in 2014 to 44% in 2018. See
Fig. 1 for a depiction of this trend.

Figure 1: The various modes of data used by MICCAI CV/ML papers from 2014
through 2018. The lightest region (top) represents papers that used at least
one existing public dataset, the middle region represents papers that used
their own data but publicly released it with their paper, and the darkest
region (bottom) represents papers using only private data that was not
released with publication.

We have no explanation for the anomalously high proportion of papers using
private data at MICCAI 2017. We surmise that this is a much higher proportion
than at other computer vision conferences such as CVPR and ICCV (although we
have not collected the data to confirm this), which highlights the unique
position of the medical image computing field where, in addition to the steep
barriers to data release, the ratio of effort required in data collection to
that in method development often seems much higher than in other applications
of computer vision.

###  4.2 Few papers released their data or code

Of the 500 papers we reviewed, only 36 (7.2%) released their code. While this
is a discouragingly small proportion, it appears that steady progress has been
made from 6% in 2014 to 9% in 2018. See Fig. 2 for a depiction of this trend.
It’s important to note, however, that with such small proportions each year,
we are unable to reject the null hypothesis that this is just random variation
over time.

Figure 2: The prevalence of MICCAI CV/ML papers releasing code over time. The
lighter region (top) represents papers that did release their code with
publications, and the darker region represents papers than did not (bottom).

Even rarer was the practice of releasing one’s data. Of the 309 papers that
used their own data, only 15 (4.9%) released this data by the time of
publication. We believe that this illustrates the high barriers and perceived
low incentives to releasing medical imaging data.

###  4.3 Papers using public data were cited more than 60% more

Among the four strata corresponding to open access or not and code release or
not, the use of a public dataset (i.e. existing public or released with
publication) was associated with 60.8% more citations per year than their
private-data-only counterparts (95% CI: 28.1% – 110.2%). Such a large effect
is highly surprising, considering that prior studies in other fields [13, 6,
3] have found that releasing one’s data is associated with only a 10%-30%
increase in citations, where our study found a much larger effect from simply
using public data. In our view, this illustrates the outsized importance of
data in machine learning research, and suggests that the medical image
computing field is highly catalyzed by the public release of imaging data. Of
important note, however, is that due to limitations with our experimental
design, we were unable to control for author reputation, which is also known
to associate with citations [15]. We leave this to future work, and we thus
stop short of making a causal claim about the association between public data
use and citation count. However, we still believe that this association is of
interest and warrants future study, especially due to its size.

###  4.4 More than quarter of data references were not citations

It follows from sections 4.1-4.3 that the medical image computing community
stands to benefit considerably from widespread data release practices.
Unfortunately, as seen in section 4.2, this has yet to take hold. One
potential contributing factor to this is that many seem to believe that data
is not a standalone scientific contribution. In our sample, of the 218 papers
that used at least one existing public dataset, 47 (21.6%) referenced a
dataset in some way other than a citation (e.g. with a footnote or simply a
mention of its name). This is not always the fault of the MICCAI authors,
since in 11 instances (5.0%), the datasets did not have an indexed entity
available to cite! If the medical image computing community is to effectively
capitalize on the many benefits of public data, the following two things must
happen:

  1. 1.

Dataset creators must do better at archiving research data in such a way that
they can receive due academic credit. Publicly funded archives such as The
Cancer Imaging Archive [2] and PhysioNet [10] are ideal for indexing and
serving this data. Additionally, peer reviewed journals for data description
manuscripts such as Nature’s Scientific Data and MPDI’s Data are a great
avenue for incentivizing high-quality, detailed descriptions of data, in
addition to enforcing that the data be added to a suitable archive. These are
all in line with the wider initiative to make more research data FAIR
(Findable, Accessible, Interoperable, and Re-usable) [18].

  2. 2.

Data users must do better to properly reference datasets and/or data
description manuscripts such that the creators do receive academic credit.
Concretely, naming a benchmark should be accompanied by a formal reference,
and footnotes should be accompanied by citations whenever possible. To ensure
this, reviewers and editors must be vigilant for–and stringent
about–inadequate data references.

We return now to Colavizza et al and their study of Data Availability
Statements. In PLOS One, where most of the articles in their study came from,
a policy was enacted in 2014 to require a DAS from each accepted paper, even
if it was simply category 1 (”data is available on request”). In the course of
two years, PLOS One articles went from virtually no articles with a DAS (2013)
to more than half of articles (2015) having a DAS of category 2 or 3, (those
are ”data within the article” and ”data in a repository” respectively). We
posit that the MICCAI Proceedings would benefit from a similar policy,
possibly not as dramatically nor as quickly due to the unique barriers we
face, but it is likely to accelerate our progress toward more efficient and
reproducible science.

##  5 Conclusion

In our study of a sample of MICCAI papers that used machine learning for
computer vision from 2014 through 2018, we found that a large proportion of
papers (54.2%) made use of privately available data alone. In addition, we
showed that even after controlling for release of code and open access
publishing, the use of publicly available data was associated with receiving
more than 60% more citations per year than the use of private data alone. We
noted also that a surprising proportion (21.6%) of papers using public data
referenced that data in some way other than a citation, for instance a
footnote with a URL, or just a name. We noticed that this was due in part to
the fact that in several instances (5.0%), no entity for the dataset in
question was available to cite.

Based on these findings, we recommend that measures be taken to encourage the
sharing of data and to ensure that the adequate credit is awarded to those who
release data that is then reused. In particular, we recommend that reviewers
be instructed to inspect data references and call out instances where the
reference is inadequate. We also recommend that MICCAI enact a policy
requiring authors to make a short statement about the availability of their
data (DAS), even if that statement is ”our data cannot be made available due
to [legitimate reason]”.

The code and data for this study has been made available at
https://github.com/neheller/labels19.

## Acknowledgements

Research reported in this publication was supported by the National Cancer
Institute of the National Institutes of Health under Award Number R01CA225435.
The content is solely the responsibility of the authors and does not
necessarily represent the official views of the National Institutes of Health.

## References

  * [1] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
  * [2] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al.: The cancer imaging archive (tcia): maintaining and operating a public information repository. Journal of digital imaging 26(6), 1045–1057 (2013)
  * [3] Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., McGillivray, B.: The citation advantage of linking publications to research data. arXiv preprint arXiv:1907.02565 (2019)
  * [4] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [5] Dixon, W.J., Yuen, K.K.: Trimming and winsorization: A review. Statistische Hefte 15(2-3), 157–170 (1974)
  * [6] Drachen, T., Ellegaard, O., Larsen, A., Dorch, S.: Sharing data increases citations. Liber Quarterly 26(2) (2016)
  * [7] Efron, B., Tibshirani, R.J.: An introduction to the bootstrap. CRC press (1994)
  * [8] Erickson, B.J., Korfiatis, P., Akkus, Z., Kline, T.L.: Machine learning for medical imaging. Radiographics 37(2), 505–515 (2017)
  * [9] Eysenbach, G.: Citation advantage of open access articles. PLoS biology 4(5), e157 (2006)
  * [10] Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)
  * [11] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Tech. rep., Citeseer (2009)
  * [12] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [13] Piwowar, H.A., Vision, T.J.: Data reuse and the open data citation advantage. PeerJ 1, e175 (2013)
  * [14] Roth, H.R., Lu, L., Farag, A., Shin, H.C., Liu, J., Turkbey, E.B., Summers, R.M.: Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation. In: International conference on medical image computing and computer-assisted intervention. pp. 556–564. Springer (2015)
  * [15] Sekara, V., Deville, P., Ahnert, S.E., Barabási, A.L., Sinatra, R., Lehmann, S.: The chaperone effect in scientific publishing. Proceedings of the National Academy of Sciences 115(50), 12603–12607 (2018)
  * [16] Thelwall, M., Wilson, P.: Regression for citation data: An evaluation of different methods. Journal of Informetrics 8(4), 963–971 (2014)
  * [17] Vandewalle, P.: Code sharing is associated with research impact in image processing. Computing in Science & Engineering 14(4), 42–47 (2012)
  * [18] Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., et al.: The fair guiding principles for scientific data management and stewardship. Scientific data 3 (2016)