# A Survey on End-User Robot Programming

Gopika Ajaykumar<EMAIL_ADDRESS>,  Maureen Steele<EMAIL_ADDRESS>and
Chien-Ming Huang<EMAIL_ADDRESS>0000-0002-6838-3701 Department of
Computer Science, Johns Hopkins University3400 N. Charles
St.BaltimoreMaryland21218

###### Abstract.

As robots interact with a broader range of end-users, end-user robot
programming has helped democratize robot programming by empowering end-users
who may not have experience in robot programming to customize robots to meet
their individual contextual needs. This article surveys work on end-user robot
programming, with a focus on end-user program specification. It describes the
primary domains, programming phases, and design choices represented by the
end-user robot programming literature. The survey concludes by highlighting
open directions for further investigation to enhance and widen the reach of
end-user robot programming systems.

End-User Robot Programming, Human-Robot Interaction, End-User Programming

This work was supported in part by the National Science Foundation Graduate
Research Fellowship Program under Grant No. DGE-1746891 and the
Nursing/Engineering joint fellowship from the Johns Hopkins University.

††copyright: rightsretained††journal: CSUR††journalyear: 2021††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††doi:
10.1145/3466819††ccs: Computer systems organization External interfaces for
robotics††ccs: Human-centered computing User studies††ccs: Software and its
engineering Development frameworks and environments

##  1\. Introduction

As computing technologies have been adopted in a variety of domains, the needs
of their end-users in quickly customizing the operation of these technologies
to meet domain-specific goals has resulted in increasing development of _end-
user programming_ systems for users who are not professional software
developers, such as children, accountants, and scientists. End-user
programming technologies, such as spreadsheets, have reached such
pervasiveness that it is estimated that end-user programmers substantially
outnumber professional programmers today (Scaffidi et al., 2005). In this way,
end-user programming has helped make computer programming widely accessible
for those who do not practice software development as a career and has
fostered the financial growth of individuals (Scaffidi, 2016) and the
productivity of organizations (Wulf and Jarke, 2004). Analogous to how end-
user programming has increased the accessibility of computer programming for
end-users of computing technologies without extensive training in software
development, _end-user robot programming_ is an emerging research area that
seeks to enable end-users of general-purpose robotic technologies who are not
robotics engineers to re-task and customize robots according to their needs.

End-users of robots are working closely with robots in manufacturing plants,
warehouses, offices, shops, homes, and even on the road to a growing degree
(Christensen et al., 2009). As of 2020, there is a record of 2.7 million
industrial robots being used in factories, with a growing share of these
industrial robots being collaborative robots that work closely with end-users
(of Robotics, 2020). The variety of domains in which robots are adopted
involve different user needs, environments, and task requirements. Rather than
requiring robots to be pre-programmed by robotics engineers for specific
application domains, which reduces flexibility, or for all potential task
scenarios, end-user robot programming allows the end-users of robots to modify
the operation of robots to work within their contexts. However, enabling end-
user robot programming is a challenging problem. Robot programming is
characterized by the embodied nature of the technology being programmed and
the situated nature of its programs, introducing unique challenges such as the
need for programs to reference and interact with the surrounding environment,
which may potentially include task objects, obstacles, and other agents,
including people. Effective robot programming not only requires mastery of the
technical concepts involved in computer programming in general, such as data
structures and algorithms, but also of a variety of multidisciplinary topics,
from control engineering to mechatronics. Robotics software engineers must
also have an in-depth knowledge of advanced topics related to robotics
including path planning, localization, mapping, kinematics, control theory,
computer vision, and machine learning. Furthermore, unlike the average
robotics engineer, end-users vary widely in their backgrounds and
technological literacy and may not have the time or capacity to learn how to
program robotic technologies. The primary goal of end-user robot programming
is to distill the complexity of robot programming into programming methods
that are approachable for users without extensive experience in robotics. The
challenge of enabling end-users to effectively develop robot programs that can
safely be run in real-world environments without requiring expertise in
software development practices and in the underlying hardware, mechanics, and
control of robots has inspired a variety of methods for easily re-tasking
robots from both industry and academia.

Methods to enable end-users to re-task robots and author robot behaviors
largely involve two different approaches— _demonstration of skills_ and
_specification of programs_. Robot _Learning from Demonstration (LfD)_ , known
variously as _Programming by Demonstration (PbD)_ and _imitation learning_ ,
seeks to eliminate the complexities involved in having end-users manually
specify robot programs of their desired robot skills by instead having users
_demonstrate_ how to perform the desired robot skills. Under the LfD paradigm,
robots develop new skills not by executing pre-specified programs, but by
learning how to generalize multiple user-provided demonstrations of a task
into a robot skill that can be applied in a variety of environmental and task
variations. While LfD seeks to simplify end-user customization of robot
behavior by eliminating the act of program specification completely, an
alternative approach is to simplify robot programming enough such that end-
users are capable of program specification. In this article, we survey
existing work targeted towards this approach, which, unlike LfD, seeks to
enable end-user programming of the structure, logic, and characteristics of
the desired robot behavior rather than having end-users demonstrate several
instantiations of it. Our survey of end-user robot programming may subsume
works on end-user programming using demonstrations, but we focus on
demonstrations as a tool for program specification rather than on the
techniques to enable robot learning from end-user demonstrations, which have
been covered by numerous surveys (e.g., (Argall et al., 2009; Billard et al.,
2008; Schaal et al., 2003; Breazeal and Scassellati, 2002; Zhu and Hu, 2018;
Calinon, 2009; Hussein et al., 2017; Chernova and Thomaz, 2014; Liu et al.,
2020)). Furthermore, we only survey papers on end-user program specification,
as opposed to papers on direct control or instruction of robots (e.g., (Han et
al., 2020)).

We organize our paper around the domains, processes, and designs involved in
end-user robot programming. In Section 2, we give an overview of how end-user
robot programming fits into the larger context of end-user programming and
robot programming. Section 3 details the methodology we used to collect papers
for the survey. In Section 4, we begin our survey, focusing specifically on
the domains covered by end-user robot programming systems. Section 5 surveys
techniques used by end-user robot programming systems to enable end-users to
participate in different phases of the robot programming process. In Section
6, we review trends and goals in the design of end-user robot programming
systems. Section 7 provides an overview of user evaluation methods in end-user
robot programming. Finally, we discuss open challenges and opportunities for
future research in Section 8 and provide concluding remarks in Section 9.

##  2\. Background

Although end-user robot programming may differ from other forms of programming
in terms of its goals and constraints, it draws parallels and inspiration from
various related fields. We describe two related fields, end-user programming
and robot programming.

###  2.1. End-User Programming

End-user robot programming is just one direction within the overall area of
end-user programming, which seeks to empower users to program their computing
technologies themselves even without experience in programming. In addition to
robotics, end-user programming has found use in a variety of application
domains, including animation, e-mail, and gaming. End-user programming
technologies have reached widespread use, with some estimates indicating that
there were more than 55 million end-user programmers in the United States by
2012 (Scaffidi et al., 2005). End-user programming has also seen increasing
research interest in the last decade, especially within the computer science
and human-computer interaction communities, with several surveys being
published on the topic over the past decade (e.g., (Barricelli et al., 2019;
Paternò, 2013; Spahn et al., 2008; Maceli, 2017)). Common end-user programming
approaches include spreadsheets, natural language programming, rule-based
programming, visual programming, wizard-based programming, and programming by
demonstration or example, many of which are used to enable end-user robot
programming as well (Section 5.2.3).

Using end-user programming tools, users may perform operations such as
automating processes or manipulating data objects without extensive experience
in using lower-level programming languages. Unlike traditional programming,
end-user programming is generally intended for personal, rather than public,
use (Ko et al., 2011). Although end-users may program to meet different goals
compared to professional developers, they face similar software engineering
challenges, such as the need to test, debug, and secure their software (Ko et
al., 2011), which has inspired focus on enabling not just end-user
programming, but _end-user software engineering_. A primary challenge in end-
user software engineering has been to support the development of high-quality
software in the face of extensive errors in software developed by end-users
(Burnett et al., 2006) and despite the tendency of end-users who are novice
programmers to prioritize efficiency and ease over maintainability and
robustness during programming (Brandt et al., 2008). End-user software
engineering research in the past decade has focused on discovery of new
programming methods and platforms, software engineering concepts, and
application domains (Ko et al., 2011), a trend seen in end-user robot
programming research as well.

Rooted in the increase in the number and diversity of computer users, end-user
programming has helped transform programming by enabling shorter development
cycles and empowering end-users to realize their real-time contextual needs
through computing. Similarly, as the user base of robotic technologies grows
in size and variety, end-user robot programming has emerged as a research area
in its own right. Although it shares similarities and builds off of end-user
programming approaches, end-user robot programming researchers must also
contend with challenges and user needs specific to robot programming and
human-robot interaction (HRI), motivating the need to consider end-user robot
programming not only as a subcategory within end-user programming, but as an
independent field involving unique user needs, goals, and constraints.

###  2.2. Robot Programming

A central force driving end-user adoption of robotic technologies lies in
their programmability, which enables them to be used for a variety of tasks
without requiring substantial modification of their hardware or control. The
availability of adequate robot programming tools to leverage robots’
programmability is a key factor in determining the tasks a robot can be used
for, as well as the total cost of applications involving robots (Lozano-Perez,
1983). Thus, robot programming systems have played a central role in
determining the adoption and evolution of robotics. Some of the earliest
instances of robot programming took the form of robot-specific languages,
which enabled programmers to specify robot motions and sensing operations for
particular robots (Biggs and MacDonald, 2003). These languages, which are
commonly used to program industrial robots (e.g., Universal Robots’ URScript
and KUKA’s KUKA Robot Language) and are still popular today, provide
programmers with application programming interfaces and scripting languages
for programming a specific robot or multiple robots from the same company.
Although robot-specific languages tend to involve relatively simple syntax and
programming commands, they still require programmers to have experience in
programming languages and robotics. Furthermore, programmers need to learn a
new robot-specific language every time they program a different robot due to
the specificity of robot-specific languages. The limitation of robot-specific
languages has resulted in a drive towards more abstracted robot programming
systems that can be used to program different robots (Biggs and MacDonald,
2003). Among these more abstracted programming methods, Robot Operating System
(ROS) (Quigley et al., 2009) has emerged as the foremost framework for
programming a variety of robotic technologies. As one of the most widely used
tools for robot programming today, especially in robotics research labs, ROS
has helped standardize the procedure for programming different types of robots
and provides numerous libraries and simulation tools to simplify specification
of robot operations such as navigation and path planning. While ROS provides
some GUI-based tools to specify aspects of robot behavior, such as motion,
without the need to program using textual program languages, it is primarily
intended for use by expert robot programmers, which is seen through its
largely command line-based interaction style. At a bare minimum, programmers
need experience with a textual programming language, such as C++ or Python,
and an understanding of concepts such as reference frames, kinematics, and
motion planning to effectively program robots using ROS. Therefore, while ROS
and similar frameworks have helped address the overspecificity of robot-
specific languages through their applicability to many different robotic
technologies and by driving robot programming towards standardization, they
require a similarly high level of programming and robotics expertise from
robot programmers, limiting their accessibility for end-users who may need to
customize robots but are not robotics experts, such as workers on factory
floors. The technical barriers preventing end-users of robots from using both
robot-specific and more abstracted programming languages and frameworks have
motivated the need for end-user robot programming systems.

The confluence of trends in end-user programming and robot programming has
helped propel research on end-user robot programming. Several surveys have
reviewed end-user robot programming research literature in the context of
specific methods and application domains (e.g., (Coronado et al., 2020;
Villani et al., 2018)) or in the context of end-user programming (e.g.,
(Paternò and Santoro, 2019)) or robot programming (e.g., (Biggs and MacDonald,
2003)) in general. In this article, we aim to provide an updated and more
comprehensive survey of research on enabling end-user specification of robot
programs.

##  3\. Methodology

We conducted a multi-phased collection process to obtain relevant papers for
our survey. We focused our search on the databases of four publishers that
publish works from major conferences and journals on human-computer
interaction, human-robot interaction, and robotics: ACM Digital Library (ACM),
IEEE Xplore (IEEE), ScienceDirect (Elsevier), and SpringerLink (Springer).

###  3.1. Phase 1: Initial Paper Collection

During Phase 1 of our collection process, we used the search string end-user
AND (robot OR robots OR robotic OR robotics) AND (program OR programming OR
development OR instruction) to obtain an initial set of papers from each of
the four databases. To determine whether these initial papers were relevant to
our survey, we read each paper’s title and abstract to determine whether the
paper referenced end-user robot programming and then applied the following
exclusion criteria:

  * •

The paper is written in a language other than English.

  * •

The paper is about end-user robot programming for children or students, with a
focus on outreach or learning outcomes rather than on program specification
methods and techniques (see (Bravo et al., 2017) for a review on end-user
robot programming in the context of non-technical education).

  * •

The paper was about robot learning rather than end-user robot program
specification. Several existing surveys review this topic (e.g., (Argall et
al., 2009; Billard et al., 2008)).

  * •

The paper focused on intuitive or user-friendly robot programming for
programming or robotics experts, rather than end-users without significant
experience in programming and robotics (e.g., (Diprose et al., 2017; Neto and
Mendes, 2013)).

If the title and abstract referenced end-user robot programming and did not
meet the exclusion criteria, the paper was collected. In cases where it was
ambiguous from the title and abstract whether the paper was related to end-
user robot programming, we skimmed through the paper to determine whether it
should be collected, with a special focus on whether the paper discussed end-
users, experts, novices, experience, and user evaluations. Throughout this
process, we only collected peer-reviewed papers from symposiums, conferences,
and journals.

Since three of the four databases yielded from 3,000 to around 33,000 results
using our search string, we set an upper limit to the collection for these
databases, which we chose empirically for each database based on the relevancy
of the search results. Some papers were listed on both the ACM Digital Library
and on IEEE Xplore. We collected these papers from ACM Digital Library but not
from IEEE Xplore to avoid duplicates.

  * •

ACM Digital Library: Yielded 32,548 results; read through the first 400 (after
which papers were less relevant); collected 45 papers

  * •

IEEE Xplore: Yielded 258 results; collected 18 papers

  * •

ScienceDirect: Yielded 33,497 results; read through the first 375 (after which
papers were less relevant); collected 19 papers

  * •

SpringerLink: Yielded 3,140 conference papers and 2,654 journal papers; read
through the first 200 conference papers and the first 50 journal papers (after
which papers were less relevant); collected 16 papers

###  3.2. Phase 2: Additional Paper Collection

After collecting papers from the four databases, we performed a search on
Google Scholar using the same search string as before, which yielded 128,000
results. We looked through the first 100 items in the search results to
discover any peer-reviewed papers that we failed to collect in Phase 1.
Through this process, we added nine additional papers to our initial paper
collection, resulting in a total of 107 papers. Six of the papers collected
from Google Scholar were published by _IEEE_ , and three of the papers were
published by independent publishers, such as the _Association for the
Advancement of Artificial Intelligence (AAAI)_. Our collection process,
including Phases 1 and 2, was concluded on June 2020. While we limited our
paper collection to papers published by June 2020, we did not include any
bounds on how early the papers we collected could be published.

###  3.3. Phase 3: Additional Paper Elimination

Finally, we applied the additional elimination criteria listed below to the
107 papers that we collected:

  * •

The paper describes designs for end-user robot programming systems but does
not describe a system implementation realizing the proposed designs (e.g.,
(Fernaeus and Jacobsson, 2009)).

  * •

The paper is a short paper consisting of four pages or less (e.g., (Oishi et
al., 2017)).

  * •

The paper does not describe the results from any form of user evaluation of
the end-user robot programming system, including case studies (e.g.,
(Gorostiza and Salichs, 2010)). We included this elimination criterion because
of the focus of our paper on user experiences in end-user program
specification, rather than on isolated programming methods or system
performance metrics.

The application of the additional criteria resulted in the elimination of 62
papers (two of which described no system implementation, 28 of which were
short papers, and 32 of which did not include results from a user evaluation).
This left a total of 45 pages for our survey, the majority of which were from
the United States ($n=18$).

##  4\. Domains of End-User Robot Programming

End-user robot programming systems are designed for the common goal of making
robot programming more accessible for a wider range of people than only
robotics experts. However, they may differ in terms of the categories of users
and robots that they are meant for. Below, we describe the diversity of end-
users and robots covered by the end-user robot programming research
literature.

###  4.1. Types of End-Users

End-user robot programming systems cover a spectrum of end-user expertise
levels and backgrounds (Table LABEL:tab:table1). While the majority of the
systems described in the surveyed papers are geared for general users from any
background or expertise level, some systems are targeted towards specific
groups of users. These include domain specialists such as healthcare workers
(e.g., (Kubota et al., 2020)) and industrial workers (e.g., (Ong et al.,
2020)). These domain specialists may have experience in Science, Technology,
Engineeering, and Mathematics (STEM) (Barakova et al., 2013; Paxton et al.,
2018) but tend to have little to no background in robotics or computer
programming and no prior experience with robot programming. On the other hand,
some systems are meant for everyday groups of users who may not be specialized
in a particular field, including children (e.g., (Barišić et al., 2018; Ryokai
et al., 2009; Sapounidis and Demetriadis, 2013)), elderly adults (e.g., (Datta
et al., 2011)), caregivers (e.g., (Barakova et al., 2013; Datta et al.,
2011)), store workers (e.g., (Liang et al., 2018)), and non-programmer adults
(e.g., (Ramoğlu et al., 2017; Weintrop et al., 2018)).

Due to the diversity of expertise levels that users have, most end-user robot
programming systems are designed to support programming for users who have no
experience at all in STEM, programming, and robotics. However, some systems
may include more advanced programming concepts, such as parallelization (e.g.,
(Barakova et al., 2013; Gorostiza and Salichs, 2011; Guerin et al., 2015;
Kapinus et al., 2019; Leonardi et al., 2019)), procedural abstraction (e.g.,
(Alexandrova et al., 2015)), and functions (e.g., (Huang et al., 2020)).
Furthermore, systems may support a variety of expertise levels, such as by
including natural language-based high-level programming features for novice
programmers and more expressive, block-based programming features for more
advanced users (e.g., (Beschi et al., 2019)). Such systems can support novice
robot programmers in gradually learning to leverage the full expressivity of
an end-user robot programming system and allow both inexperienced and
experienced users to make use of the system for their programming needs.

###  4.2. Types of Robots

As robots vary greatly in terms of form factors, capabilities, and use cases,
end-user robot programming systems often focus on enabling intuitive
customization of a specific type or make of robot (Table LABEL:tab:table1).
The types of robots used in end-user robot programming systems today can be
broadly categorized into two categories: robots primarily intended for
manipulation tasks and robots primarily intended for non-manipulation tasks.
We describe the robots that are commonly programmed by end-users within these
two classes below.

####  4.2.1. Robots Primarily Intended for Manipulation Tasks

The most common type of robot used for end-user robot programming in the
surveyed literature is the industrial robot manipulator (arm), which is
traditionally used for manufacturing tasks. These robots, which have six to
seven movable joints, can be programmed to perform various manipulation tasks,
ranging from simple pick-and-place tasks to high precision assembly, depending
on the individual robot’s capabilities and range of motion. Another common
type of robot used in end-user robot programming systems exposes even more
capabilities for programming through the use of two manipulators, paving the
way for complex manipulation tasks that may require the simultaneous use of
multiple forces, such as unscrewing the lid of a jar. Within the surveyed
literature, three collaborative robots of this type are used in end-user robot
programming systems: the PR2 robot, the Baxter robot, and the YuMi robot.
Systems using the PR2 robot from Willow Garage may expose its navigation,
manipulation, and sensing and perception capabilites for end-user programming,
whereas systems meant for the Baxter robot from Rethink Robotics may expose
its customizable facial appearance, its sensing and manipulation capabilities,
and, depending on whether it is mounted on a stationary or mobile pedestal,
its navigation capabilities for programming. Both types of robots have two
7-degrees-of-freedom arms and a 2-degrees-of-freedom head. On the other hand,
the YuMi robot from ABB Robotics includes two 7-degrees-of-freedom arms but no
manueverable head or base, which limits it to end-user robot programming of
manipulation tasks. While the PR2 robot is a service robot meant for everyday
human environments and daily living tasks that require dexterity, the Baxter
robot is an industrial robot intended for simple industrial tasks, with a
focus on pick-and-place operations, and the YuMi robot is an industrial robot
designed for manufacturing environments and assembly tasks.

####  4.2.2. Robots Primarily Intended for Non-Manipulation Tasks

In addition to robots intended for manipulation tasks, social robots, mobile
service robots, home robots, and educational robots are also types of robots
that are commonly programmed by end-users. Social robots include
anthropomorphic and zoomorphic robots. The anthropomorphic robots used in end-
user robot programming systems include the NAO (e.g., (Barakova et al., 2013;
Buchina et al., 2016; Buchina et al., 2019; Manohar and Crandall, 2014; Erich
et al., 2017)) and the Pepper (e.g., (Leonardi et al., 2019)) robots from
SoftBank Robotics, the Kaspar robot from the University of Hertfordshire
(e.g., (Moros et al., 2019)), the Maggie robot from the University Carlos III
of Madrid (e.g., (Gorostiza and Salichs, 2011)), and the Kuri robot from
Mayfield Robotics (e.g., (Kubota et al., 2020)). The Pepper, Maggie, and Kuri
robots are mobile, allowing them to be programmed for navigation tasks. The
zoomorphic robots used in end-user robot programming systems include stuffed
animal robots (e.g., (Young et al., 2014)) and the Pleo robot from Innvo Labs
(e.g., (Manohar and Crandall, 2014; Ryokai et al., 2009)). End-user robot
programming systems that use social robots allow users to program both low-
level actions, such as sensing and motion, as well as high-level interactive
behaviors, such as emotion expression.

Other robots used in end-user robot programming systems include mobile service
robots (e.g., (Datta et al., 2011; Huang et al., 2016)), such as the Savioke
Relay robot. End-user robot programming systems that work with these types of
robots allow users to program various types of commands, from navigation to
sound and visual display. In addition to robots found in service sectors, some
end-user robot programming systems focus on programming of robots found in
everyday environments. These include home robots such as the Roomba robot
(e.g., (Young et al., 2014)), robotic toys such as robotic spheres from Sphero
(e.g., (Ramoğlu et al., 2017)), Arduino cars (e.g., (Barišić et al., 2018)),
and LEGO robots (e.g., (Sapounidis and Demetriadis, 2013)). Programming
systems that work with lower-cost consumer robots often center on the
programming of robot movement and navigation. While end-user robot programming
systems tend to focus on the programming of one specific robot, some systems
provide users with the option to program multiple robots simultaneously (e.g.,
(Barakova et al., 2013; Riedl and Henrich, 2019)), including robots of
different makes (e.g., (Riedl and Henrich, 2019)). Recent systems further
extend end-user programming by enabling the simultaneous programming of robots
together with the Internet of Things that is present in the robot’s
environment (e.g., (Huang et al., 2020; Leonardi et al., 2019)).

##  5\. Phases of End-User Robot Programming

The process of developing a program using an end-user robot programming system
involves various phases, such as setting up the system, program, or task
environment; specifying program logic; fixing and checking the authored
program for correctness; and executing the program. We describe the various
ways in which programming systems enable users to participate in each of these
phases.

###  5.1. Initialization/Setup

Similar to how computer programming may require initialization of variables or
setup of a program editor, end-user robot programming may require initial
setup before program authoring can begin. While a setup or initialization
phase is not supported or required in all end-user robot programming systems,
some systems may offer system-level setup features, such as hardware testing
or calibration (e.g., (Cakmak and Takayama, 2014)) or specification of user
information (e.g., (Gorostiza and Salichs, 2011)), and program-level setup
features, such as specifying task objects (e.g., (Huang and Cakmak, 2017)) or
the workspace (e.g., (Schou et al., 2018)) that must be represented in the
programming system. Since robot programming systems, unlike most computer
programming systems, often work with programs involving interaction with other
objects and entities in the environment, the end-user must also set up the
task environment to include necessary components, such as task objects. Setup
may also involve preparing necessary hardware for the programming system
itself, such as microphones (e.g., (Alexandrova et al., 2014; Cakmak and
Takayama, 2014; Ryokai et al., 2009)), wearable devices (e.g., (Bambušsek et
al., 2019; Quintero et al., 2018)), or projectors (e.g., (Schou et al., 2018;
Sefidgar et al., 2018; Gao and Huang, 2019)). To minimize the overhead
required on the part of the user, some end-user robot programming systems
automate parts of the setup process, such as mapping of the programming
workspace and detection of programmable devices and robots (e.g., (Huang et
al., 2020)). On the other hand, supporting users in appropriately initializing
programs, which is challenging for novice programmers (Franklin et al., 2016),
has been less explored in the literature, though prior work has suggested its
potential usefulness in preventing program errors by end-users (e.g.,
(Weintrop et al., 2017)).

###  5.2. Authoring

The literature on end-user robot programming systems includes a diversity of
focal points, methods, and approaches towards enabling end-user authoring of
robot programs. We highlight the variety of authoring techniques that end-
users can leverage for robot programming.

####  5.2.1. Programmable Robot Capabilities

End-user robot programming systems limit programmable robot capabilities to
specific actions that can interlock with their respective robots’ capabilities
and meet the needs of their intended users and domains (Table
LABEL:tab:table1). Programs relating to manipulation are commonly supported by
end-user robot programming systems. Some end-user robot programming systems
focus on lower-level programming of manipulation and motion paths through
which users may specify robot end-effector poses (e.g., (Cakmak and Takayama,
2014; Stenmark et al., 2017; Gadre et al., 2019)) or continuous trajectories
(e.g., (Jha et al., 2015)). Others focus on higher-level programming of
manipulation through specification of the objects and locations to be used for
pick-and-place actions (e.g., (Gao and Huang, 2019; Sefidgar et al., 2017; Ong
et al., 2020)), or they may provide additional user flexibility by allowing
both low-level and high-level programming of manipulation (e.g., (Quintero et
al., 2018)).

Table 1. Domains, Programming Methods, and Programmable Capabilities in End-User Robot Programming. _Programming Method (Section 5.2.3): V = Visual, XR = Augmented/Mixed Reality, D = Demonstration, NL = Natural Language, T = Tangible Programmable Capabilities (Section 5.2.1): Interactive Behavior (e.g., emotion, dialog, interaction style), Manipulation (e.g., any motion involving manipulation of objects via gripper), Motion (e.g., motion paths, gestures), Navigation (e.g., motion of mobile robots to specified locations in environment), Sensing (e.g., active perception, touch sensor readings), Audiovisual (e.g., audio output such as robot speech and visual output such as turning on an LED)_ |  | Programming
---
Method
| Programmable
---
Capabilities
Basics | End-User | Robot | V | XR | D | NL | T | Interac. | Manip. | Motion | Nav. | Sensing | Aud./Vis.
| Alexandrova et al.
---
(2014) (Alexandrova et al., 2014)
General | PR2 | ✓ |  | ✓ | ✓ |  |  | ✓ | ✓ |  |  |
| Alexandrova et al.
---
(2015) (Alexandrova et al., 2015)
General | PR2 | ✓ |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |
| Bambušek et al.
---
(2019) (Bambušsek et al., 2019)
General | PR2 |  | ✓ |  |  |  |  | ✓ | ✓ |  |  |
| Barakova et al.
---
(2013) (Barakova et al., 2013)
| Healthcare
---
workers
Social | ✓ |  |  | ✓ |  |  |  | ✓ |  | ✓ | ✓
| Barišić et al.
---
(2018) (Barišić et al., 2018)
Children |  | Arduino
---
car
✓ |  |  |  |  |  |  | ✓ |  | ✓ |
| Beschi et al.
---
(2019) (Beschi et al., 2019)
| Industrial
---
workers
Arm | ✓ |  |  | ✓ |  |  | ✓ | ✓ |  | ✓ |
| Buchina et al.
---
(2016) (Buchina et al., 2016)
| Healthcare
---
workers
Social | ✓ |  |  | ✓ |  |  |  | ✓ |  | ✓ | ✓
| Buchina et al.
---
(2019) (Buchina et al., 2019)
| Healthcare
---
workers
Social | ✓ |  |  | ✓ |  |  |  | ✓ |  | ✓ | ✓
| Cakmak and
---
Takayama
(2014) (Cakmak and Takayama, 2014)
General | PR2 |  |  | ✓ | ✓ |  |  | ✓ | ✓ |  |  |
| Datta et al.
---
(2011) (Datta et al., 2011)
| Healthcare
---
workers,
Elderly
adults,
Caregivers
| Mobile
---
service
✓ |  |  |  |  | ✓ |  |  |  |  | ✓
| Erich et al.
---
(2017) (Erich et al., 2017)
| Healthcare
---
workers
Social | ✓ |  |  |  |  |  |  |  |  | ✓ | ✓
| Forbes et al.
---
(2014) (Forbes et al., 2014)
General | PR2 | ✓ |  | ✓ |  |  |  | ✓ | ✓ |  |  |
| Gadre et al.
---
(2019) (Gadre et al., 2019)
General | Baxter |  | ✓ |  |  |  |  | ✓ | ✓ |  |  |
| Gao and Huang
---
(2019) (Gao and Huang, 2019)
General | Arm |  | ✓ |  |  |  |  | ✓ |  |  |  |
| Gorostiza and
---
Salichs (2011) (Gorostiza and Salichs, 2011)
General | Social |  |  |  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ | ✓
| Guerin et al.
---
(2015) (Guerin et al., 2015)
| Industrial
---
workers
Arm | ✓ |  |  |  |  |  | ✓ | ✓ |  | ✓ |
| Huang et al.
---
(2016) (Huang et al., 2016)
General |  | Mobile
---
service
✓ |  |  |  |  | ✓ |  | ✓ | ✓ | ✓ | ✓
| Huang and
---
Cakmak (2017) (Huang and Cakmak, 2017)
General | PR2 | ✓ |  | ✓ | ✓ |  | ✓ | ✓ | ✓ |  | ✓ | ✓
| Huang et al.
---
(2020) (Huang et al., 2020)
| Industrial
---
workers
Mobile | ✓ |  |  |  |  |  | ✓ | ✓ | ✓ |  |
| Jha et al.
---
(2015) (Jha et al., 2015)
General | Arm |  |  | ✓ |  |  |  |  | ✓ |  |  |
| Kapinus et al.
---
(2019) (Kapinus et al., 2019)
| Industrial
---
workers
PR2 |  | ✓ |  |  |  |  | ✓ |  |  |  |
| Kubota et al.
---
(2020) (Kubota et al., 2020)
| Healthcare
---
workers
Social |  |  |  |  | ✓ | ✓ |  |  |  | ✓ | ✓
| Leonardi et al.
---
(2019) (Leonardi et al., 2019)
General | Social | ✓ |  |  |  |  |  |  | ✓ |  | ✓ | ✓
| Liang et al.
---
(2017) (Liang et al., 2017)
General | Baxter |  |  | ✓ |  |  |  | ✓ |  |  |  |
| Liang et al.
---
(2018) (Liang et al., 2018)
| Store
---
workers
| Mobile
---
arm
✓ |  | ✓ |  |  |  | ✓ |  |  |  |
| Liang et al.
---
(2019) (Liang et al., 2019)
General | Baxter | ✓ |  | ✓ |  |  |  | ✓ |  |  |  |
| Manohar and
---
Crandall (2014) (Manohar and Crandall, 2014)
General | Social |  |  | ✓ |  |  | ✓ |  |  |  |  |
| Matthaiakis et al.
---
(2017) (Matthaiakis et al., 2017)
| Industrial
---
workers
Arm | ✓ |  |  |  |  |  | ✓ |  |  |  |
| Moros et al.
---
(2019) (Moros et al., 2019)
General | Social | ✓ |  |  |  |  | ✓ |  | ✓ |  | ✓ | ✓
| Ong et al.
---
(2020) (Ong et al., 2020)
| Industrial
---
workers
Arm |  | ✓ |  |  |  |  | ✓ | ✓ |  |  |
| Paxton et al.
---
(2018) (Paxton et al., 2018)
| Industrial
---
workers
Arm | ✓ |  |  |  |  |  | ✓ | ✓ |  | ✓ |
| Pedersen and
---
Krüger (2015) (Pedersen and Krüger, 2015)
| Industrial
---
workers
| Mobile
---
arm
✓ |  | ✓ |  |  |  | ✓ |  |  |  |
| Quintero et al.
---
(2018) (Quintero et al., 2018)
| Industrial
---
workers
Arm |  | ✓ |  | ✓ |  |  | ✓ | ✓ |  |  |
| Racca et al.
---
(2020) (Racca et al., 2020)
General | Arm | ✓ |  |  |  |  |  | ✓ | ✓ |  | ✓ |
| Ramoğlu et al.
---
(2017) (Ramoğlu et al., 2017)
Adults | Toy | ✓ |  |  |  |  |  |  | ✓ |  | ✓ | ✓
| Riedl and
---
Henrich (2019) (Riedl and Henrich, 2019)
| Industrial
---
workers
Arm | ✓ |  | ✓ |  |  |  | ✓ | ✓ |  |  |
| Ryokai et al.
---
(2009) (Ryokai et al., 2009)
Children | Social | ✓ |  |  |  | ✓ | ✓ |  | ✓ |  | ✓ | ✓
| Sapounidis and
---
Demetriadis
(2013) (Sapounidis and Demetriadis, 2013)
Children | LEGO | ✓ |  |  |  | ✓ |  |  | ✓ |  | ✓ | ✓
| Schou et al.
---
(2018) (Schou et al., 2018)
| Industrial
---
workers
Arm | ✓ |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |
| Sefidgar et al.
---
(2017) (Sefidgar et al., 2017)
General | PR2 |  |  |  |  | ✓ |  | ✓ |  |  |  |
| Sefidgar et al.
---
(2018) (Sefidgar et al., 2018)
General | PR2 |  |  |  |  | ✓ |  | ✓ |  |  |  |
| Stenmark et al.
---
(2017) (Stenmark et al., 2017)
General | YuMi | ✓ |  | ✓ |  |  |  | ✓ | ✓ |  | ✓ |
| Weintrop et al.
---
(2017) (Weintrop et al., 2017)
| Industrial
---
workers
Arm | ✓ |  |  |  |  |  | ✓ | ✓ |  |  |
| Weintrop et al.
---
(2018) (Weintrop et al., 2018)
Adults | Arm | ✓ |  |  |  |  |  | ✓ | ✓ |  |  |
| Young et al.
---
(2014) (Young et al., 2014)
General |  | Roomba,
---
Toy
|  | ✓ |  |  | ✓ |  | ✓ |  |  |

For mobile robots, end-user robot programming systems allow users to author
programs related to robot navigation (e.g., (Alexandrova et al., 2015; Huang
et al., 2016; Huang et al., 2020)). When authoring navigation-related
programs, end-users may program the robot to navigate to a known location,
which may either be preprogrammed by the system developer, automatically
detected by the system (e.g., (Huang et al., 2016; Huang et al., 2020)), or
specified by the end-user (e.g., (Pedersen and Krüger, 2015)). Beyond task-
related movements, end-users may also specify motion for robot gestures (e.g.,
(Barakova et al., 2013; Buchina et al., 2019)).

In addition to motion-related programs, end-user robot programming systems may
allow user authoring of robot sensing using sensors such as cameras, motion
sensors, and touch sensors to enable the robot to react appropriately to its
surroundings (e.g., (Barakova et al., 2013; Buchina et al., 2019; Guerin et
al., 2015; Ramoğlu et al., 2017; Kubota et al., 2020)). Including sensing
capabilities can extend the flexibility of end-user robot programming systems
by allowing programming in more dynamic environments where task objects or
obstacles change locations frequently. Sensor-related actions are often used
for event-driven programming or conditional constructs in programming (e.g.,
(Kubota et al., 2020)). For robots with first-person cameras, end-user robot
programming systems may allow user authoring of programs related to active
perception (e.g., (Alexandrova et al., 2014)). Active perception enables the
robot to maneuver itself or its sensors to obtain information about its
environment, which can further minimize the responsibility of the user in
specifying object locations during robot programming. In addition to
supporting general robot actions related to sensing and audiovisual
capabilities such as speech, sound, and lighting (e.g., (Barakova et al.,
2013; Buchina et al., 2019)), end-user robot programming systems may also
support programming of higher-level actions, such as human-robot interaction
and collaboration (e.g., (Alexandrova et al., 2015; Guerin et al., 2015; Huang
et al., 2016; Ryokai et al., 2009; Young et al., 2014)), or task-level
commands, such as common therapeutic exercise commands (e.g., (Kubota et al.,
2020)) or object assembly commands (e.g., (Schou et al., 2018)).

####  5.2.2. Authoring Scope

End-user robot programming systems vary in their authoring scope. Many systems
focus on enabling users to specify the structure and logic of a program using
pre-specified robot primitives, such as grasps or spinning, that are often
developed by expert robot programmers (e.g., (Barišić et al., 2018; Huang et
al., 2016; Huang et al., 2020; Kubota et al., 2020; Leonardi et al., 2019;
Ryokai et al., 2009; Sapounidis and Demetriadis, 2013; Stenmark et al., 2017;
Weintrop et al., 2018; Barakova et al., 2013; Gorostiza and Salichs, 2011;
Weintrop et al., 2017; Alexandrova et al., 2015; Buchina et al., 2016; Buchina
et al., 2019; Guerin et al., 2015; Kapinus et al., 2019; Beschi et al., 2019;
Moros et al., 2019; Pedersen and Krüger, 2015; Ramoğlu et al., 2017)). Some
end-user robot programs give the user more granular access by allowing end-
users to specify the primitives used for subsequent programming (e.g., (Huang
et al., 2016; Liang et al., 2017; Schou et al., 2018; Erich et al., 2017;
Liang et al., 2019; Stenmark et al., 2017)) or by taking an end-to-end
approach in which end-users are fully responsible for specifying every aspect
of the program, as in imitation learning (e.g., (Jha et al., 2015; Manohar and
Crandall, 2014)). On the other hand, other end-user robot programming systems
handle the bulk of the program specification using techniques such as
automated planning (e.g., (Liang et al., 2017, 2019)). In such systems, end-
users may only need to specify high-level aspects of the system, such as end
goals or style, or make minor refinements to the system-generated program
(e.g., (Sefidgar et al., 2018; Bambušsek et al., 2019; Liang et al., 2018)).
Similarly, some systems may focus on enabling rapid prototyping of robot
programs rather than longer development cycles (e.g., (Huang et al., 2016;
Huang and Cakmak, 2017)). In addition to enabling specification of the overall
logic and flow of a program, some systems may enable end-users to specify the
parameters for commands or functions (e.g., (Huang et al., 2020; Paxton et
al., 2018; Weintrop et al., 2017)). In order to ease the process of specifying
continuous parameters or parameters in 3-D space, systems may provide
intuitive methods for parametrization, such as directional (e.g., (Racca et
al., 2020)) or gesture-based specification (e.g., (Pedersen and Krüger,
2015)).

####  5.2.3. Programming Methods

End-user robot programming systems offer a variety of methods for users to
program robot actions and behaviors (Table LABEL:tab:table1), each of which
has its own advantages and limitations. We describe the most common methods
used to date (see Fig. 1 for examples).

Figure 1. Examples of End-User Robot Programming Methods.
_(Top)_ Visual, Tangible
_(Bottom)_ Demonstration, Augmented Reality

_Visual._ Visual programming is a common end-user programming method that is
widely recognized for its success in making programming more accessible for
users without technical experience (Cypher and Smith, 1995; Resnick et al.,
2009). Using visual programming interfaces, end-users are able to author robot
programs by manipulating a graphical representation of the program. In the
end-user robot programming literature, visual programming interfaces commonly
use flow diagrams (e.g., (Barakova et al., 2013; Erich et al., 2017)),
behavior trees (e.g., (Barišić et al., 2018; Guerin et al., 2015; Paxton et
al., 2018)), blocks (e.g., (Beschi et al., 2019; Huang et al., 2016; Huang and
Cakmak, 2017; Moros et al., 2019; Ramoğlu et al., 2017; Weintrop et al., 2017,
2018)), and icons (e.g., (Ryokai et al., 2009; Sapounidis and Demetriadis,
2013; Stenmark et al., 2017)) as their graphical representations. A primary
weakness of the visual programming method is its limited representational
power, since it can be intractable to represent every possible robot command
visually. Thus, visual programming is often combined with alternate methods to
enhance its expressivity, such as textual programming (e.g., (Barakova et al.,
2013)) or kinesthetic demonstrations (e.g., (Huang and Cakmak, 2017; Stenmark
et al., 2017)). Visual programming requires users to program within a 2-D
interface where the programming commands are disconnected from the task
environment. This limitation has led to new approaches to end-user robot
programming that combine visual programming concepts with spatial context
(i.e., spatial-visual programming (Huang et al., 2020)).

For some visual programming systems (e.g., (Datta et al., 2011; Leonardi et
al., 2019; Liang et al., 2018; Liang et al., 2019; Matthaiakis et al., 2017;
Pedersen and Krüger, 2015; Racca et al., 2020; Schou et al., 2018)), end-users
specify robot programs using Graphical User Interfaces (GUIs) that makes use
of common interactive visual components for authoring programs, such as
windows, buttons, input forms, menus, and sliders. GUIs generally provide pre-
specified programming options rather than allowing open-ended user input. Some
systems also include simulations of the robot in their GUI for visualization
and programming purposes (e.g., (Matthaiakis et al., 2017; Liang et al.,
2019)). GUI-based end-user robot programming emphasizes familiar, intuitive
interaction styles and visualization of program information, but it may still
result in high cognitive burden for end-users because of the difficulty of
programming robot capabilities meant for 3-D task environments using 2-D
screens (Gadre et al., 2019; Huang et al., 2020; Ramoğlu et al., 2017;
Weintrop et al., 2017, 2018). This drawback of GUIs for end-user robot
programming has contributed to increased interest in augmented and mixed
reality-based robot programming in recent years.

_Augmented/Mixed Reality._ Augmented Reality (AR), in which the real-world
environment is supplemented with virtual information, and Mixed Reality (MR),
in which physical and virtual objects can interact, are increasingly leveraged
as mediums for end-user robot programming. AR- and MR-based robotics
applications have been commonly used in industrial environments (Gattullo et
al., 2019; Fang et al., 2014), where the need for safety awareness may limit
the use of potentially distracting interfaces that are disjoint from the
working environment, and most AR- and MR-based programming systems are
intended for programming industrial robots and manufacturing tasks (e.g., (Gao
and Huang, 2019; Kapinus et al., 2019; Ong et al., 2020; Quintero et al.,
2018)). Augmented and mixed reality interfaces for end-user robot programming
enhance the physical environment with virtual overlays through the use of
head-mounted displays (e.g., (Bambušsek et al., 2019; Gadre et al., 2019; Ong
et al., 2020; Quintero et al., 2018)), projectors (e.g., (Bambušsek et al.,
2019; Gao and Huang, 2019)), or mobile devices (e.g., (Kapinus et al., 2019)).
End-users are able to interact with virtual items in the augmented environment
using gestures (e.g., (Bambušsek et al., 2019; Gadre et al., 2019; Gao and
Huang, 2019; Quintero et al., 2018)), touch (e.g., (Bambušsek et al., 2019)),
speech (e.g., (Quintero et al., 2018)), direct interaction (e.g., (Bambušsek
et al., 2019; Gadre et al., 2019; Kapinus et al., 2019)), or using devices,
such as pointers (e.g., (Ong et al., 2020)) or gesture control bands (e.g.,
(Quintero et al., 2018)). End-user program specification is performed by
indicating task objects and locations for manipulation (e.g., (Bambušsek et
al., 2019; Gao and Huang, 2019)), motion trajectories (e.g., (Gadre et al.,
2019; Ong et al., 2020; Quintero et al., 2018)) or the high-level, situated
flow of the program (e.g., (Kapinus et al., 2019)). AR- and MR-based end-user
robot programming overcomes the limitations of other programming methods that
may require end-users to be collocated with the robots being programmed, to
translate 3-D environments onto 2-D screens, and to perform context switches
between the robot and the programming interface. Furthermore, they may also
better enable non-anthropomorphic robots to convey intent during interactive
programming (Hedayati et al., 2018; Coovert et al., 2014). However, augmented
reality interfaces may be susceptible to perceptual issues during end-user
robot programming because of issues such as occluded or constrained viewpoints
and difficult depth perception (Kruijff et al., 2010). Designers of AR-based
programming systems may need to consider these issues carefully, as perceptual
issues caused by hardware limitations or overdraw during object rendering may
result in higher user cognitive load (Bambušsek et al., 2019).

_Demonstration._ A common and intuitive technique end-users use to program
robots is demonstration of the capabilities to be programmed (e.g., (Jha et
al., 2015; Young et al., 2014)). A common form of demonstration used for
program specification is kinesthetic demonstration, which is provided by end-
users using _kinesthetic teaching_ , also known as lead-through programming.
During kinesthetic teaching, users physically maneuver the robot through the
desired robot motion trajectory. Kinesthetic teaching requires low cognitive
load for users (Bambušsek et al., 2019; Quintero et al., 2018); allows
programmers to have better control of a robot’s movements compared to
alternative methods, such as direct manipulation of a virtual image of the
robot (Manohar and Crandall, 2014); and can be faster and more intuitive for
end-users because it works within the robot’s configuration space (Ong et al.,
2020), making it the predominant end-user robot programming method in robotics
(Weintrop et al., 2017) and a popular programming method in scientific
literature (e.g., (Cakmak and Takayama, 2014; Liang et al., 2019; Riedl and
Henrich, 2019; Schou et al., 2018; Stenmark et al., 2017)). Despite its
advantages and widespread use, kinesthetic teaching has limitations, including
the requirement for the robot to be physically present during programming
(Bambušsek et al., 2019; Manohar and Crandall, 2014), which may lessen
workplace productivity (Ong et al., 2020); the physical demands placed on the
user in physically guiding the robot (Bambušsek et al., 2019; Quintero et al.,
2018; Ajaykumar and Huang, 2020), which may make it infeasible to program
heavy industrial robots using this method (Quintero et al., 2018); and the
need for the robot being programmed to be equipped with sensors and motors
that enable kinesthetic demonstrations (Manohar and Crandall, 2014).

_Natural Language._ Natural language is commonly employed for end-user robot
programming in the form of speech (e.g., (Alexandrova et al., 2014; Beschi et
al., 2019; Cakmak and Takayama, 2014; Gorostiza and Salichs, 2011; Huang and
Cakmak, 2017; Quintero et al., 2018)) and text (e.g., (Beschi et al., 2019;
Buchina et al., 2016; Buchina et al., 2019)). Speech-based programming in
particular has the advantage of easy accessibility for most users because of
its basis in innate human communication, including for users with physical
limitations or who have difficulties spelling (Kubota et al., 2020). However,
speech-based programming is subject to vulnerabilities, such as speech
misrecognition in noisy environments, which may be common in industrial
workplaces where end-user robot programming systems may be deployed (Beschi et
al., 2019), and is constrained to work with programming commands that are easy
to describe verbally (Gadre et al., 2019). Perhaps due to the potentially
unreliable nature of natural language recognition, end-user robot programming
systems that use natural language also make use of additional programming
methods, such as kinesthetic teaching (e.g., (Alexandrova et al., 2014; Cakmak
and Takayama, 2014; Huang and Cakmak, 2017)), GUI visualizations (e.g.,
(Alexandrova et al., 2014)), augmented reality (e.g., (Quintero et al.,
2018)), and visual programming (e.g., (Beschi et al., 2019; Buchina et al.,
2016; Buchina et al., 2019; Huang and Cakmak, 2017)), or additional input
modalities, such as gestures (e.g., (Gorostiza and Salichs, 2011; Quintero et
al., 2018)).

_Tangible._ Tangible programming is a programming method where end-users can
author a program by indicating a program’s structure and instructions using
physical objects situated in the real-world environment. Common objects used
for tangible programming include cards (e.g., (Kubota et al., 2020; Ryokai et
al., 2009)) and blocks (e.g., (Sapounidis and Demetriadis, 2013; Sefidgar et
al., 2017; Sefidgar et al., 2018)). Tangible programming often involves high-
level commands, which can include task-level concepts (e.g., (Kubota et al.,
2020)), robot behaviors (e.g., (Ryokai et al., 2009)), or directional or goal-
based aspects of motions (e.g., (Sefidgar et al., 2017; Sefidgar et al., 2018;
Sapounidis and Demetriadis, 2013)). Due to its simplicity and ease of use,
tangible programming systems are often designed for use by children (e.g.,
(Ryokai et al., 2009; Sapounidis and Demetriadis, 2013)), though recent
systems have expanded their use to general end-users (e.g., (Sefidgar et al.,
2017; Sefidgar et al., 2018)) and domain specialists (e.g., (Kubota et al.,
2020)). Unlike other programming systems that combine multiple programming
methods, tangible robot programming systems generally rely on interaction with
physical objects alone for program specification, making it difficult for
tangible robot programming to be used in specifying and modifying longer and
more complex programs.

####  5.2.4. Programming Features

End-user robot programming systems include various programming features common
to computer programming, such as conditional instructions and loops. Since
robotics often involves the simultaneous processing of multiple sensor streams
and actuation of various motors, end-user robot programming systems may also
allow end-users to specify elements of parallelization (e.g., (Barakova et
al., 2013; Gorostiza and Salichs, 2011; Guerin et al., 2015; Kapinus et al.,
2019; Leonardi et al., 2019)) and event-driven programming (e.g., (Beschi et
al., 2019; Guerin et al., 2015)). Depending on the background of the intended
end-users, designers may also choose to add more advanced programming features
into end-user robot programming systems, such as nesting (e.g., (Alexandrova
et al., 2015)) or functions and recursion (e.g., (Huang et al., 2020)). To
reduce the complexity of working with programming features for users without
coding experience, some systems embed concepts such as conditions (e.g.,
(Alexandrova et al., 2015; Sefidgar et al., 2017; Sefidgar et al., 2018)) and
looping (e.g., (Pedersen and Krüger, 2015; Sefidgar et al., 2017; Sefidgar et
al., 2018)) within commands so that users do not explicitly have to work with
statements like if/else or while.

End-user robot programming systems often involve online (e.g., (Alexandrova et
al., 2014; Matthaiakis et al., 2017; Quintero et al., 2018)) or offline (e.g.,
(Alexandrova et al., 2014; Beschi et al., 2019; Manohar and Crandall, 2014;
Matthaiakis et al., 2017; Weintrop et al., 2017, 2018)) programming features.
Online programming features, such as those that use kinesthetic teaching,
require the use of a real, physical robot. In contrast, offline programming
features enable users to program using a virtual simulation of a robot. While
interacting with a physical robot may give end-users a better idea of the
robot’s capabilities and may be less computationally intensive than
simulation-based programming, offline programming features can help reduce
user effort and program suboptimalities (Ong et al., 2020), cost (Pan et al.,
2012), difficulties in modifying and developing programs (Pan et al., 2012),
and idle time in waiting for a robot to become available (Pan et al., 2012).
To leverage the advantages of both online and offline features, end-user robot
programming systems may provide a combination of both, such as by decoupling
online program specification from offline parameter specification (e.g.,
(Schou et al., 2018)). Alternatively, some end-user robot programming systems
may offer programming capabilities that require neither physical nor virtual
robot presence (e.g., (Bambušsek et al., 2019)).

###  5.3. Editing and Debugging

End-user robot programming systems typically provide program editing tools to
allow users to iteratively refine the program and remove any errors or
suboptimalities. At the simplest level, end-users can often delete, move,
copy, or clear individual commands or the entire program (e.g., (Cakmak and
Takayama, 2014; Gao and Huang, 2019; Alexandrova et al., 2015; Guerin et al.,
2015; Ryokai et al., 2009; Stenmark et al., 2017)) or modify parameters for
saved programs or templates (e.g., (Bambušsek et al., 2019; Datta et al.,
2011; Gadre et al., 2019; Liang et al., 2018; Liang et al., 2019)). In
addition, systems focused on editing may provide additional editing
functionality, including modification of the configuration or reference frame
of individual points in manipulation programs (e.g., (Alexandrova et al.,
2014; Forbes et al., 2014; Gadre et al., 2019)). End-users may also use
editing tools to correct system inferences for systems using techniques such
as intent recognition or automated planning to automatically specify portions
of the program (e.g., (Liang et al., 2017; Liang et al., 2018; Liang et al.,
2019)). Program modification may also be achieved using more indirect
feedback, such as via simple reinforcement learning (e.g., (Young et al.,
2014)).

While most systems support pre- and post-execution editing, some systems
support real-time editing while the program is running, such as by dynamically
changing a robot motion path while it is being previewed in simulation (e.g.,
(Quintero et al., 2018)) or executed (e.g., (Matthaiakis et al., 2017; Young
et al., 2014)). To support debugging, end-user robot programming systems may
include features such as stepwise execution and program reversion (e.g.,
(Racca et al., 2020; Stenmark et al., 2017; Pedersen and Krüger, 2015)). These
systems may enable editing of visualizations of the robot and the program via
GUIs (e.g., (Alexandrova et al., 2014; Forbes et al., 2014; Gadre et al.,
2019; Riedl and Henrich, 2019)) or augmented reality (e.g., (Liang et al.,
2019)), which allows end-users to avoid maneuvering the physical robot to
modify portions of the program. To make program editing more intuitive, end-
user robot programming systems may use common concepts from other domains,
such as filmstrip views or playback from video editing (e.g., (Riedl and
Henrich, 2019)). For systems focusing on editing to help end-users in
providing more diverse, optimal demonstrations for subsequent robot learning,
editing may be accomplished through crowdsourcing (e.g., (Forbes et al.,
2014)).

###  5.4. Verification

Verifying that a program will work correctly when executed on the robot is an
important part of the programming process, especially for safety-critical
environments like clinical settings where users may need to confirm that the
program is safe to run around patients (Buchina et al., 2016). A program may
fail to run correctly because of errors introduced during program authoring or
because of errors due to hardware limitations or probabilistic system
components, such as robot perception (e.g., (Sefidgar et al., 2018)).
Verification of program correctness is often the responsibility of the user,
even when the program is automatically generated by the system (e.g., (Liang
et al., 2017, 2019)), but systems may offer error feedback (e.g., (Weintrop et
al., 2017; Sefidgar et al., 2018)) and prevention mechanisms (e.g., (Barišić
et al., 2018; Beschi et al., 2019; Weintrop et al., 2017)) to ease or automate
verification. Some systems check for errors during program authoring (e.g.,
(Huang et al., 2020; Leonardi et al., 2019)), while others also perform error
checking during program execution (e.g., (Huang et al., 2020; Sefidgar et al.,
2018; Alexandrova et al., 2014)). Furthermore, systems using techniques like
control synthesis may be able to guarantee users that their program is
automatically correct-by-construction, elimating the need for manual
verification (e.g., (Kubota et al., 2020)). End-user robot programming systems
may also include a compilation phase prior to execution (e.g., (Sapounidis and
Demetriadis, 2013; Sefidgar et al., 2017; Huang and Cakmak, 2017; Huang et
al., 2020)), similar to computer programming systems.

##  6\. Design of End-User Robot Programming Systems

The design of end-user robot programming systems is often inspired by observed
user difficulties and needs in robot programming, and designers focus on
common design aspects to meet these user needs and make robot programming more
accessible. We summarize common design inspirations and aspects and how they
are implemented in end-user robot programming systems.

###  6.1. Design Inspirations

End-user robot programming systems are often developed with user needs and
domain requirements in mind. For example, end-user robot programming systems
are commonly developed using a Human-Centered Design (HCD) process, where the
system design is iteratively updated based on user evaluations and feedback.
The HCD design process often involves observation of or interviews with
potential end-users at the domain the system is intended for (e.g., (Huang et
al., 2016; Huang et al., 2020)), which may help designers understand design
requirements and goals for their system (Huang et al., 2020; Stenmark et al.,
2017). Furthermore, designers may follow a bottom-up design approach where
user characteristics and needs are used to inspire design (e.g., (Beschi et
al., 2019)). The HCD process may be applied for both the design of the
programming system as well as of any supplementary instructional materials
that will accompany it (Cakmak and Takayama, 2014). End-user robot programming
systems may also take inspiration from interfaces from other fields and
domains, such as projected interfaces in human-computer interaction (e.g.,
(Gao and Huang, 2019)), programming environments in education (e.g., (Weintrop
et al., 2017)), or editors in multimedia (e.g., (Riedl and Henrich, 2019)).

###  6.2. Design Aspects

End-user robot programming systems prioritize some common design aspects to
help provide a more flexible, robust, and easy robot programming experience
for users. We detail the aspects commonly considered in the design of end-user
robot programming systems.

####  6.2.1. Reusability

End-user robot programming systems commonly enable end-users to reuse
elements, or the entirety, of their programs. Reusability can be enabled at
multiple levels within an end-user robot programming scenario, from reusing
robot primitives or modules of a program to reusing a program in a different
domain (Barakova et al., 2013). Furthermore, program parameters, such as
locations, may also be reused (e.g., (Weintrop et al., 2018)). Reusability can
be a valuable design aspect for end-user robot programming systems in
improving programming efficiency and simplicity (Stenmark et al., 2017).

To enhance reusability, end-user robot programming systems may follow a
modular design where the program is broken into independent modules that can
be reused within or shared among programs (e.g., (Barakova et al., 2013; Huang
et al., 2016)). In many cases, end-users may develop their own custom modules
that group together specific portions of the program that they would like to
reuse (e.g., (Barišić et al., 2018; Gadre et al., 2019; Guerin et al., 2015;
Stenmark et al., 2017)). End-user robot programming systems may also enable
reusability using traditional modular programming constructs like functions
and recursion (e.g., (Huang et al., 2020)). Program templates are another
popular mechanism to support reusability in end-user robot programming.
Program templates enable end-users to program robot actions by quickly and
easily parametrizing prespecified program sequences. Program templates can be
created by end-users (Bambušsek et al., 2019; Gadre et al., 2019), or they may
be predefined in the programming system for common tasks, such as screwing
(e.g., (Matthaiakis et al., 2017)) or pick-and-place (e.g., (Weintrop et al.,
2017)). Although end-user robot programming systems for program specification
do not focus on robot learning, they may still include components in their
systems to help increase the generalizability, and thus reusability, of user
programs to new scenarios without relying on the robot to learn how to adapt
the program. Program generalizability is often emphasized in robot
manipulation programming systems, as manipulation programs are often specified
in terms of specific objects and positions instead of higher-level concepts
that can apply to different situations.

For robot manipulation programs, generalizability is often achieved by
representing the waypoints in a motion trajectory relative to reference
frames, often in reference to a landmark in the environment (e.g.,
(Alexandrova et al., 2014; Alexandrova et al., 2015; Paxton et al., 2018;
Stenmark et al., 2017; Huang and Cakmak, 2017)), rather than as an absolute
point. Using this approach, program generalizability may be the responsibility
of the user, as users can generalize a program to new environments by manually
editing the reference frames for waypoints as needed when a program needs to
be run for a new task or in a new setting (e.g., (Alexandrova et al., 2014;
Stenmark et al., 2017)). Alternatively, the programming system may be
responsible for helping make the manipulation program generalizable by
automatically adapting the program to work with task changes, such as the
number, types, configurations, or environments of task objects (e.g.,
(Alexandrova et al., 2015; Liang et al., 2019; Paxton et al., 2018;
Matthaiakis et al., 2017)), allowing flexibility in how and when programmed
actions can be performed (Liang et al., 2019). Another common approach to make
end-user robot programming systems and programs generalizable to different
domains, tasks, and environments is to include parametric programming
components that can be easily reparametrized according to the requirements of
a new scenario (e.g., (Beschi et al., 2019; Guerin et al., 2015; Liang et al.,
2018; Liang et al., 2019; Pedersen and Krüger, 2015; Stenmark et al., 2017)).
To further enhance program generalizability, end-user robot programming
systems may support parameter ranges, rather than single values, for
parametric components so that the system can adapt its parameters’ values to
new scenarios (e.g., (Racca et al., 2020)). As a whole, more generalizable
end-user robot programming helps drive robot programs towards being more
reusable for a diversity of tasks (Liang et al., 2018; Liang et al., 2019).

####  6.2.2. Context

As robot programs are situated within the real, physical environment, end-user
robot programming systems need to represent the task environment throughout
the programming process. While some end-user robot programming systems are
inherently situated in the task environment through the use of online, AR,
natural language, or tangible programming features, other robot programming
systems may need to include additional components to help users contextualize
their programs. In addition to displaying a representation of the program
structure, these systems may include visualizations of the robot (e.g.,
(Alexandrova et al., 2014; Weintrop et al., 2017)), task objects (e.g., (Huang
and Cakmak, 2017; Liang et al., 2019)), and surfaces in the environment (e.g.,
(Alexandrova et al., 2014)). For manipulation programs, programmed motion
paths and robot poses (e.g., (Ong et al., 2020; Alexandrova et al., 2014;
Forbes et al., 2014; Quintero et al., 2018; Gadre et al., 2019; Liang et al.,
2019; Riedl and Henrich, 2019)) or the high-level program sequence (e.g.,
(Kapinus et al., 2019)) may also be visualized within the environment. For
navigation programs, where a 2-D representation is sufficient for conveying
spatial information, visualizations of robots and relevant objects may be
represented at a lower fidelity (e.g., (Huang et al., 2020)). Some systems
assume that the task environment will remain static during the programming
process, with objects and landmarks being predefined (e.g., (Liang et al.,
2019)), while others assume dynamic, flexible task environments (e.g., (Gao
and Huang, 2019; Matthaiakis et al., 2017)), where real-time changes to the
environment are supported (e.g., (Alexandrova et al., 2015)). For programs
involving interactive robot behaviors, end-users may benefit from social or
interaction context. Social context is especially critical when end-users
demonstrate programs by acting out robot behaviors, as contextual cues can
support effective acting (Schmidt et al., 2002). Social context can take the
form of nonverbal or verbal behaviors (e.g., (Manohar and Crandall, 2014)) or
collaborative scenarios (e.g., (Young et al., 2014)).

####  6.2.3. Transparency

End-user robot programming systems help increase the transparency of robot
programming by helping end-users in developing accurate mental models of
robots and programming systems. Accurate mental models are essential in
helping users effectively learn to operate devices (Kieras and Bovair, 1984)
and use programming commands properly (Sefidgar et al., 2017), helping novice
programmers avoid programming errors (Huang et al., 2016) and better align
their perceived and actual task performance (Weintrop et al., 2018), and
helping users better understand the robot behavior resulting from complex
programs (Leonardi et al., 2019) and make use of advanced robotic capabilities
(Paxton et al., 2018). Supporting accurate mental models may be especially
important for systems that include probabilistic components, such as speech
recognition, that could lead to unintended changes to the program that the
user may fail to notice (Huang and Cakmak, 2017).

To help users form clearer mental models during the programming process, end-
user robot programming systems may contextualize the program within the
physical environment so that end-users can understand what is visible to the
robot and visualize where commands will occur in the real world (e.g.,
(Alexandrova et al., 2014; Sefidgar et al., 2018)). In addition, end-user
robot programming systems may use direct mapping techniques to make the
process of robot programming easier and draw off of end-users’ existing mental
models, such as by directly mapping the programming environment to the spatial
task environment (e.g., (Huang et al., 2020)), directly mapping program
primitives to human actions (e.g., (Pedersen and Krüger, 2015)), or directly
mapping program building to jigsaw puzzle building to help users understand
what commands can be connected sequentially (e.g., (Huang and Cakmak, 2017)).
Furthermore, systems may display the real-time status of the system and the
robot as it is programmed so that users can better understand the effects of
their programming actions as well as causes of errors or failures (e.g.,
(Huang et al., 2020; Sefidgar et al., 2018)). Besides displaying the state and
status of robot programs, systems may better enhance transparency of robot
programming by displaying the “thought process” of the robot as it executes
programmed operations, such as by displaying its current state in terms of
inputs, outputs, and commands (e.g., (Ryokai et al., 2009)) or its
interpretation of the current programming command (e.g., (Sefidgar et al.,
2018; Liang et al., 2019)), to help users understand their programs’ effects
in real-time.

Effective robot programming also requires end-users to have a good
understanding of what robots are and are not capable of. End-users who do not
have prior experience with robots may be susceptible to misperceptions and
incorrect mental models of robot capabilities (e.g., (Liang et al., 2017)) due
to robots’ physical (Fussell et al., 2008; Powers and Kiesler, 2006), speech
(Cha et al., 2014), movement (Cha et al., 2015), and behavioral (Lee et al.,
2005) characteristics, as well as due to overblown representations of robots
in the media (Sandoval et al., 2014). This can lead to novice robot
programmers having incorrect expectations on what a robot and programming
system can accomplish during robot programming. Thus, end-user robot
programming systems often include elements in the system to better convey
robot and system capabilities. Robot capabilities may be communicated by
representing the reachability of different points in space by the robot for
manipulation tasks (e.g., (Ong et al., 2020)), providing all programmable
robot capabilities as programming commands (e.g., (Kubota et al., 2020)) and
providing a status bar indicating their real-time feasibility (e.g., (Huang et
al., 2020; Sefidgar et al., 2018)), specifying which task and programming
objects are detected by the robot (e.g., (Sefidgar et al., 2018)), and
indicating the workspace in which the robot can operate (e.g., (Sefidgar et
al., 2018)). For end-user robot programming methods that may appear to have an
unlimited space of programming options, such as natural language-based
programming, it is especially important for systems to communicate the
constrained set of programming commands that the robot is capable of executing
(Buchina et al., 2016).

####  6.2.4. Abstraction

Abstraction is a common design aspect of end-users robot programming systems
that helps simplify the complexity involved in robot programming. Abstraction
is inherently part of end-user robot programming systems, which abstract away
processes like motion planning or localization. However, some programming
systems may include more abstraction than others, depending on whether they
choose to abstract away computer programming concepts such as conditions
(e.g., (Alexandrova et al., 2015; Sefidgar et al., 2017)) or loops (e.g.,
(Pedersen and Krüger, 2015; Sefidgar et al., 2017)), represent programming
commands at a primitive or semantic level (e.g., (Huang et al., 2020; Kubota
et al., 2020; Paxton et al., 2018; Schou et al., 2018)), or enable the
creation of reusable modules (e.g., (Barakova et al., 2013; Liang et al.,
2019)). Though abstraction is a critical aspect of end-user robot programming
systems, it may be difficult for users to create and use abstractions
themselves (Green and Blackwell, 1998). Thus, end-user robot programming
systems employing abstractions such as templates or functions may need to
either pre-define the abstractions for end-users or provide assistance in
their development, or they could risk reducing the accessibility of the system
for some users (Green and Blackwell, 1998). One approach to enhance user
control and flexibility in working with abstractions is to provide users with
access to multiple levels of abstraction during robot programming (e.g.,
(Buchina et al., 2016; Leonardi et al., 2019; Ong et al., 2020)), which can
allow them to use or ignore abstractions as desired.

####  6.2.5. Assistance

End-user robot programming systems may offer different levels of assistance to
further decrease the complexity of robot programming. At the simplest level,
end-user robot programming systems may help simplify the process of manual
specification of programs. Error prevention, detection, and handling
mechanisms are a common example of this type of assistance (e.g., (Alexandrova
et al., 2015; Huang et al., 2016)). At a more complex level, programming
systems may use interactive, data-driven approaches to automate aspects of
program specification. For example, the system may use mixed-initiative
features to actively prompt the end-user to provide information that may
improve the quality of their program, rather than relying on the user to
optimize the program themselves (e.g., (Gorostiza and Salichs, 2011; Racca et
al., 2020)). In a similar vein, programming systems may use automated planning
techniques to assist the user by automatically generating programs (e.g.,
(Liang et al., 2017, 2019)) and control synthesis to automatically generate
control code for specified high-level robot behaviors (e.g., (Kubota et al.,
2020)). Recent works have explored the use of more data-driven and
probabilistic approaches to providing assistance, allowing the use of
techniques like intent recognition to infer task goals (e.g., (Liang et al.,
2018)) or active learning to help end-users narrow down the range of feasible
values for continuous program parameters (e.g., (Racca et al., 2020)).

##  7\. Evaluation of End-User Robot Programming Systems

In this section, we summarize common components and approaches in the design
of user evaluations for end-user robot programming.

###  7.1. Methods

User evaluations of end-user robot programming systems often seek to evaluate
system use for program authoring, with users being instructed to program a
specific or open-ended task. In some cases, it may be desirable to evaluate
different or more nuanced aspects of the system use. For this reason, in
addition to tasks related to program authoring, users may be assigned
different tasks testing their comprehension of existing programs or their
ability to debug erroneous programs (e.g., (Alexandrova et al., 2015; Erich et
al., 2017; Huang et al., 2020; Sefidgar et al., 2017)). User evaluations are
often conducted using controlled experiments, but may also take form in case
(e.g., (Huang et al., 2016; Huang and Cakmak, 2017; Stenmark et al., 2017)),
field (e.g., (Datta et al., 2011)), and observational studies (e.g., (Huang
and Cakmak, 2017)). Observation tends to be favored over controlled
experiments when the goal is to evaluate the expressivity of an end-user robot
programming system for expert users (e.g., (Guerin et al., 2015; Liang et al.,
2019)). Evaluations may be conducted not only to understand user experiences
in using an end-user robot programming system, but also in the programming
process as a whole. For example, user evaluations may seek to understand the
effectiveness of supplementary materials or instructions for end-user robot
programming systems in addition to the effectiveness of the programming
systems (e.g., (Cakmak and Takayama, 2014; Ramoğlu et al., 2017)). User
evaluations generally involve one participant at a time, but user evaluations
involving children often allow pairs of users to program at a time (e.g.,
(Moros et al., 2019; Ryokai et al., 2009; Sapounidis and Demetriadis, 2013)).

End-user robot programming systems may be evaluated alone or against
comparative examples or baseline conditions. Comparative examples and
evaluation baselines may involve robot programming using existing commercial
(e.g., (Barišić et al., 2018; Buchina et al., 2016; Erich et al., 2017;
Manohar and Crandall, 2014; Ong et al., 2020; Stenmark et al., 2017; Weintrop
et al., 2018)) or academic (e.g., (Barišić et al., 2018; Gao and Huang, 2019))
programming systems. For systems that are intended for use by expert
programmers as well as novices, traditional textual programming languages may
serve as the baseline programming method (e.g., (Stenmark et al., 2017)). For
user studies seeking to evaluate system learnability, the baseline condition
can involve the use of the same version of the system without instructions
(e.g., (Sefidgar et al., 2017)) to gauge the immediate understanding of users
who do not have any training on how to use the system.

###  7.2. Data

Most evaluations collect data about users’ demographics and backgrounds, which
often include questions about their attitudes towards and prior experience
with technology (e.g., (Cakmak and Takayama, 2014)), programming (e.g.,
(Leonardi et al., 2019)), and robots (e.g., (Stenmark et al., 2017)), to
determine whether there is any correlation between user characteristics and
various study measures (e.g., (Leonardi et al., 2019; Stenmark et al., 2017)).
We describe common categories of study measures used in user evaluations for
end-user robot programming.

####  7.2.1. Success and Quality of User-Authored Programs

Many user evaluations for end-user robot programming systems evaluate the
success and quality of programs developed using the end-user robot programming
system. We summarize common measures related to task and program success and
quality.

_Task Success._ A common task measure used to evaluate whether a programming
system enables effective robot programming is task success (e.g., (Alexandrova
et al., 2014; Cakmak and Takayama, 2014; Gadre et al., 2019; Stenmark et al.,
2017)). Task success is measured by checking whether a user-authored program
results in successful execution (e.g., (Huang and Cakmak, 2017; Quintero et
al., 2018)) or meets pre-defined standards of correctness (e.g., (Huang et
al., 2020; Liang et al., 2018; Weintrop et al., 2018)).

_Subjective Evaluation of Program Quality._ When the programming task relates
to developing stylistic or social robot behaviors, task success may be based
on more subjective measures related to the user-authored program, such as
discernability (e.g., (Manohar and Crandall, 2014)). In this case, the user-
authored program may need to undergo evaluation by other study participants to
determine its quality (e.g., (Manohar and Crandall, 2014; Young et al.,
2014)).

_Similarity._ In some cases, a program’s quality may be determined by its
similarity to other programs, such as programs from previously related work
(e.g., (Manohar and Crandall, 2014)) or experts’ programs (e.g., (Racca et
al., 2020)). Similarity-related measures may also be used to evaluate the
operation of the system. For instance, for systems that automatically generate
a program by imitating a user demonstration, the evaluation may measure the
dissimilarity of the generated program from the user’s original demonstration
to understand how faithfully the system managed to recreate the user’s
demonstration as a robot manipulation program (e.g., (Jha et al., 2015)).

_Generalizability._ Generalizability tests are often performed for programs
created by experts, most frequently by the system developers themselves (e.g.,
(Liang et al., 2019)). However, the generalizability of programs created by
study participants may also be evaluated. This is commonly accomplished by
having the participant develop the program and then having the experimenter
test its generalizability in a new scenario after the study is over (e.g.,
(Alexandrova et al., 2014; Forbes et al., 2014)).

_Complexity._ User-authored programs may be evaluated for their complexity.
Depending on the goal of the end-user robot programming system, complexity may
be either a desired or undesired quality of a program. For example, for
systems meant for the development of complex programs, program complexity may
be evaluated to indicate whether users are able to master the use of the
system to create more involved programs (e.g., (Buchina et al., 2016)).

####  7.2.2. Programming Time and Progress

Time and progress are often the focus of user evaluations for end-user robot
programming. Around 64% of the surveyed papers included user evaluations that
measure task and programming time, which can be used to get an idea of how
efficient end-user robot programming is using the evaluated system. Another
task metric used to understand user efficiency while programming an end-user
robot programming system is task progress, or the number of tasks that a user
managed to complete during the study (e.g., (Buchina et al., 2016; Gadre et
al., 2019; Huang and Cakmak, 2017; Weintrop et al., 2018)). While metrics such
as time and progress may provide an idea of the programming efficiency enabled
by an end-user robot programming system, these measures are not infallible
representations of efficiency since they may be affected by other factors. For
instance, studies have suggested that expert programmers may have higher task
times because they spend more time exploring a system beyond the requirements
of a task (Liang et al., 2019) and that some programming methods are
inherently faster than others, which may affect task progress measures in
studies comparing different programming methods (e.g., (Quintero et al.,
2018)). Furthermore, high task times may be desirable in some circumstances,
as spending more time on earlier tasks may be an effective strategy for
programming by helping users master programming features (Alexandrova et al.,
2014; Liang et al., 2019).

In addition to efficiency, time metrics may highlight the user’s experience
and understanding of the system. For example, measuring the amount of time the
user spends practicing before they begin programming using a system may
suggest the immediate learnability of the system (e.g., (Gao and Huang,
2019)). While longer interaction times are generally considered negative when
considering aspects such as system efficiency and learnability, they may be
desirable when the end-user robot programming system is meant for edutainment.
For example, interaction time may be measured as part of the user evaluation
to assess how engaged a user is when programming using the system (e.g.,
(Gorostiza and Salichs, 2011)).

####  7.2.3. User Perceptions

User evaluations often seek to understand user perceptions of the end-user
robot programming process, including their perceptions of the system, the
robot, and the programming tasks. User perceptions of the system may be
measured by having users give subjective evaluations of the system and its
features in terms of their ease to learn (e.g., (Weintrop et al., 2018)) and
use (e.g., (Huang et al., 2016; Sapounidis and Demetriadis, 2013; Weintrop et
al., 2018; Moros et al., 2019)), functionality (e.g., (Pedersen and Krüger,
2015)), usefulness (e.g., (Alexandrova et al., 2014; Liang et al., 2018; Moros
et al., 2019; Sefidgar et al., 2018)), effectiveness (e.g., (Barišić et al.,
2018)), naturalness (e.g., (Gadre et al., 2019)), and intuitiveness (e.g.,
(Pedersen and Krüger, 2015)). The perceived usability of an end-user robot
programming system is widely measured in user evaluations. The most common
tool for measuring system usability in user evaluations for end-user robot
programming systems is the System Usability Scale (SUS) (e.g., (Bambušsek et
al., 2019; Beschi et al., 2019; Gadre et al., 2019; Huang and Cakmak, 2017;
Kapinus et al., 2019; Kubota et al., 2020; Leonardi et al., 2019; Liang et
al., 2019; Paxton et al., 2018; Racca et al., 2020; Ramoğlu et al., 2017)), an
established 10-item questionnaire that produces a system usability score
(Brooke, 1996). User evaluations may also use variations of the Cognitive
Dimensions Framework (Green and Petre, 1996) to evaluate the usability of the
system along various cognitive dimensions (e.g., (Barakova et al., 2013;
Buchina et al., 2016; Huang et al., 2020)). Similarly, users may evaluate the
user experience of the system using established questionnaires such as the
User Experience Questionnaire (UEQ) (Laugwitz et al., 2008) (e.g., (Bambušsek
et al., 2019; Kapinus et al., 2019)). In addition, users’ perceptions of the
programming system may also be better understood by measuring their
satisfaction with the system (e.g., (Barišić et al., 2018; Weintrop et al.,
2018)), as well as by asking open-ended questions about their experience,
opinions, and suggestions related to the system and its features. Open-ended
comments may be subsequently grouped to represent common user perceptions
using a Grounded Theory approach (Strauss and Corbin, 1994) (e.g., (Kubota et
al., 2020; Weintrop et al., 2018; Young et al., 2014)). Furthermore, users may
be asked about their willingness to use a system again (e.g., (Moros et al.,
2019)) and about their preference for the system relative to alternate
programming systems (e.g., (Sapounidis and Demetriadis, 2013; Sefidgar et al.,
2018)).

In addition to users’ perceptions and subjective evaluation of the end-user
robot programming system, user evaluations may investigate users’ perceptions
of the robot being programmed to understand how using an end-user robot
programming system affects a user’s mental model of and relationship with the
robot. User studies may include questions about the user’s overall perception
of the robot (e.g., (Barakova et al., 2013)) or their evaluation of how the
robot performed (e.g., (Alexandrova et al., 2014)). Users’ perceptions of the
robot being programmed may also be observed indirectly. For instance, the
number of interactions with the robot may be measured to understand how
engaging a user perceives the robot, and the programming system as a whole, to
be (e.g., (Datta et al., 2011)).

User evaluations often aim to understand user perceptions of the programming
tasks, since these tasks are representative of real-world programming
scenarios the system is intended for. To understand how easy robot programming
is using a system, user evaluations may include questions about the users’
perceived difficulty (e.g., (Alexandrova et al., 2014; Cakmak and Takayama,
2014; Ramoğlu et al., 2017)) or success (e.g., (Cakmak and Takayama, 2014))
for the programming tasks used in the study. For programming systems meant for
edutainment, user evaluations may also ask users how fun or enjoyable they
found the programming tasks to be (e.g., (Moros et al., 2019; Sapounidis and
Demetriadis, 2013)).

####  7.2.4. User Effort

User evaluations may include measures related to user effort to understand the
extent to which an end-user robot programming system alleviates an end-user’s
burden during robot programming. User effort may be measured through the
number of programming attempts the user makes to achieve a task (e.g., (Young
et al., 2014)). The most common tool for measuring user workload during end-
user robot programming is the NASA Task Load Index (NASA-TLX) (e.g.,
(Alexandrova et al., 2014; Bambušsek et al., 2019; Cakmak and Takayama, 2014;
Erich et al., 2017; Forbes et al., 2014; Gadre et al., 2019; Gao and Huang,
2019; Kapinus et al., 2019; Quintero et al., 2018)), a subjective tool for
assessing a user’s workload while working with a system in terms of mental,
physical and temporal demand; performance; effort; and frustration (Hart and
Staveland, 1988).

####  7.2.5. User Understanding

User evaluations for end-user robot programming systems may seek to measure
the level of user understanding on how to effectively program using the
system. Gaps in user understanding may be observed by measuring the number,
rate, and type of errors users make while using the system (e.g., (Alexandrova
et al., 2015; Gao and Huang, 2019; Weintrop et al., 2018)) and, in cases where
the experimenter is available for assistance, the number of help requests they
make to the researcher running the evaluation study (e.g., (Cakmak and
Takayama, 2014; Huang and Cakmak, 2017; Schou et al., 2018)). Lack of user
understanding may also be identified by observing the challenges or
difficulties encountered by users during programming (e.g., (Beschi et al.,
2019; Kubota et al., 2020; Sefidgar et al., 2018; Stenmark et al., 2017)).
Users may also be directly tested on their understanding of different system
features (e.g., (Sefidgar et al., 2017)), or they may be tested on their
understanding of how to use the system before beginning the study to discover
correlations between pre-study understanding with final task performances
(e.g., (Liang et al., 2019)).

####  7.2.6. User Behavior

In addition to the data collected from questionnaires and interviews, data
from observations of users may provide insight into user behaviors while using
an end-user robot programming system. Data from observing user behavior may
provide information on how frequently different system features are used
(Alexandrova et al., 2014; Huang and Cakmak, 2017; Liang et al., 2019; Ryokai
et al., 2009) and how different user groups approach programming tasks using
the system (Alexandrova et al., 2015; Pedersen and Krüger, 2015; Sapounidis
and Demetriadis, 2013; Young et al., 2014). It may also help showcase
unexpected participant behaviors that the system may not be designed for
(Bambušsek et al., 2019). Evaluators may use techniques such as encouraging
users to think aloud while using the system to understand common user
behaviors and thought processes during programming (e.g., (Barakova et al.,
2013; Ramoğlu et al., 2017; Young et al., 2014)).

###  7.3. Participants

If possible, user evaluations will include participants from the intended user
base of the programming system being evaluated. These may include specific
populations, such as domain specialists (e.g., (Barakova et al., 2013; Buchina
et al., 2016; Kubota et al., 2020; Riedl and Henrich, 2019)) or children
(e.g., (Moros et al., 2019; Ryokai et al., 2009; Sapounidis and Demetriadis,
2013)) with or without programming experience. Because end-user robot
programming is intended to be approachable for those without technical
backgrounds, user evaluations may focus on the recruitment of participants
without experience in robotics, in the programming method used by the system,
and in programming in general. However, many evaluations also invite
participation from experts in these technical areas. Conducting user
evaluations with expert users, in addition to novice robot programmers, may be
beneficial for gaining insights into how to improve a programming system
(Alexandrova et al., 2014; Young et al., 2014) and serve a broader range of
users who could benefit from the system (Young et al., 2014) and into how well
the system closes the performance gap between novice robot programmers and
experts (Schou et al., 2018).

###  7.4. Instruction and Training

A common practice in user evaluations for end-user robot programming systems
is to provide some form of instruction and training on how to use the systems
to users before they begin performing the study tasks. This training may seek
to expose users to not just the procedure for programming using the system,
but also to special hardware needed to use the system (e.g., (Bambušsek et
al., 2019)). Common instruction materials include one-page reference sheets
(e.g., (Alexandrova et al., 2014; Weintrop et al., 2018)) and video (e.g.,
(Forbes et al., 2014; Huang et al., 2020)) or written (e.g., (Cakmak and
Takayama, 2014; Gao and Huang, 2019)) tutorials. The experimenter may also
provide a demonstration (e.g., (Alexandrova et al., 2014; Alexandrova et al.,
2015)) or verbal explanation of how to use the programming system.
Occasionally, tutorials and program examples may be embedded in the system
interface itself (e.g., (Ramoğlu et al., 2017)). Users may be given an
opportunity to perform practice tasks (e.g., (Bambušsek et al., 2019; Huang et
al., 2016; Kapinus et al., 2019; Liang et al., 2019; Paxton et al., 2018)) and
to seek clarifications for any confusion they may still have (e.g., (Paxton et
al., 2018)) before performing the experimental tasks. Furthermore,
experimenters may choose not to provide any instruction or training on how to
program using the system when they want to observe the immediate understanding
of the users without any priming (e.g., (Sefidgar et al., 2017)).

###  7.5. Tasks

The study tasks used in user evaluations for end-user robot programming
systems may represent common robot actions, domain-specific activities, or
specific phases of end-user robot programming. Study tasks involving common
robot actions include pick-and-place tasks (e.g., (Gao and Huang, 2019; Paxton
et al., 2018)) and human-robot or robot-robot interaction tasks, such as
authoring scenarios commonly encountered in robot-facilitated therapy (e.g.,
(Buchina et al., 2016)) or robot emotions and roles for multi-robot
interaction (e.g., (Young et al., 2014)). Study tasks representing domain-
specific activities cover daily living tasks such as cleaning, retrieving and
storing household items, and folding towels (e.g., (Alexandrova et al., 2014;
Alexandrova et al., 2015; Cakmak and Takayama, 2014; Huang and Cakmak, 2017;
Quintero et al., 2018; Stenmark et al., 2017)); manufacturing tasks such as
machine tending, circuit board testing, and welding (e.g., (Guerin et al.,
2015; Kapinus et al., 2019; Ong et al., 2020; Pedersen and Krüger, 2015;
Quintero et al., 2018; Riedl and Henrich, 2019; Schou et al., 2018)); and
store shelving tasks (e.g., (Liang et al., 2018)). While most study tasks
focus on testing program authoring, they may also test other phases of end-
user robot programming, such as parametrization (e.g., (Bambušsek et al.,
2019)), editing (e.g., (Bambušsek et al., 2019; Gadre et al., 2019; Liang et
al., 2017)), and debugging (e.g., (Kapinus et al., 2019)). Users may also go
through an open-ended task where they freely interact with the programming
system to develop their desired program (e.g., (Kubota et al., 2020; Leonardi
et al., 2019; Young et al., 2014)).

User evaluations for end-user robot programming systems generally involve
multiple study tasks. Often, the number and type of tasks are chosen such that
they fit within a prescribed time limit. Study tasks may be given to users in
order of increasing complexity (e.g., (Beschi et al., 2019; Cakmak and
Takayama, 2014)) or with decreasing instruction (e.g., (Barakova et al.,
2013)) to challenge users as they gain experience in using a system.
Furthermore, different study tasks may be given to users of different levels
of expertise (e.g., (Huang and Cakmak, 2017)).

##  8\. Discussion

In this survey, we presented an overview of design and evaluation approaches
used for end-user robot programming systems to provide users with the ability
to program, edit and debug, and execute programs for various robots without
requiring expertise in robot programming. While prior research has helped
drive the movement towards connecting a wider range of users with the power of
robot customization, there still remain unexplored directions in the field
that could further highlight techniques for better understanding and meeting
user needs in robot programming. Below, we discuss open challenges and areas
of research in end-user robot programming.

###  8.1. Supporting Representative User Evaluations

User evaluations are a key component in the design of end-user robot
programming systems, motivating the need to develop robust and representative
evaluation methods. In this section, we describe open directions for improving
evaluations of robot programming systems and to help researchers and
developers further understand end-user experiences in robot programming.

####  8.1.1. Longitudinal Studies

User evaluations in the end-user robot programming literature are largely
short in scope, with studies often taking place within an hour and users
performing pre-defined, timed programming tasks. Although user evaluations of
this form may be useful in highlighting various aspects of system use,
especially in terms of immediate user understanding and first-sight
preferences, they may fail to capture the complete user experience in end-user
robot programming. This can be observed in the prevalence of the learning
effect in user evaluations for end-user robot programming systems (e.g.,
(Alexandrova et al., 2014; Huang and Cakmak, 2017; Huang et al., 2020; Liang
et al., 2019; Racca et al., 2020; Ramoğlu et al., 2017; Stenmark et al., 2017;
Young et al., 2014)), where users’ task performances improve as they progress
with study tasks. With users often devoting substantial study time to
understanding how to use a programming system, evaluations may miss out on
user experiences outside of the learning phase and in programming using more
complex features that require more time to learn than a short-term study can
provide. Longitudinal studies may help researchers and developers obtain a
better idea of how users would use systems in the long term, as well as help
them understand the dynamics of how users’ system usage may change over time
as they become more familiar with an end-user robot programming system.
Further work may be needed to enable system developers to easily track how
long-term usage and user assessments evolve over time (Barišić et al., 2018).

####  8.1.2. Benchmark Tasks and Metrics

A primary challenge found in user evaluations for end-user robot programming
systems is the difficulty of preventing irrelevant factors from influencing
measures in user evaluations. For example, programming tasks requiring more
physical input from users (i.e., kinesthetic teaching) take inherently longer
to complete, which may make it difficult to fairly compare the task times for
programming tasks involving different levels of physical involvement to
measure relative efficiency (Alexandrova et al., 2015). Furthermore,
subjective measures, such as system usability, may be susceptible to influence
from factors unrelated to the programming methods and techniques being
evaluated, such as hardware unreliability (Bambušsek et al., 2019; Pedersen
and Krüger, 2015) or users’ perceptions of the robot being programmed (Ramoğlu
et al., 2017). Additionally, the novelty effect users may experience from
programming or interacting with a robot for the first time may result in
overly positive or negative evaluations of the system, especially for users
with unrealistic expectations about a robot’s abilities. End-users of robots
are far from a homogenous group and vary greatly in their backgrounds and
demographics, which can in turn result in a substantial variance in how
different users program (Alexandrova et al., 2015; Pedersen and Krüger, 2015;
Sapounidis and Demetriadis, 2013; Young et al., 2014). This variance can be
seen in the prevalence of unexpected user programming behaviors that emerge
during user evaluations of end-user robot programming systems (e.g.,
(Bambušsek et al., 2019)). While following a strict experimental protocol,
including a variety of study measures, and collecting covariate data may
offset some of the variability introduced by the influence of peripheral
factors on study measures, it may be infeasible for experimenters to control
for every factor in an evaluation. Moving towards deriving an established
benchmark set of evaluation tasks and metrics, an approach common in fields
such as computer vision and natural language processing, may highlight one way
to make systematic progress in end-user robot programming by standardizing the
evaluation process and simplifying the process of understanding and comparing
different robot programming works.

####  8.1.3. Evaluations in Real-World Settings

Recent years have seen the publication of several works in end-user robot
programming enabled through collaboration between academics and companies
(e.g., (Barišić et al., 2018; Huang et al., 2016)), highlighting increasing
commercial interests in making end-user robot programming accessible through
real-world products and applications. However, academic work on end-user robot
programming may present a limited view of user experiences in end-user robot
programming due to a lack of deployments and user evaluations in non-
laboratory settings. Laboratory settings are generally highly controlled and
are set up to prioritize participant safety, which may cause participants to
program less cautiously than they would in a less controlled setting (e.g.,
(Racca et al., 2020)). While some systems have been deployed in real-world
settings, such as on production floors (e.g., (Guerin et al., 2015; Pedersen
and Krüger, 2015; Schou et al., 2018)), these deployments generally focus on
evaluating the technical capabilities of programming systems in enabling
programming of complex tasks rather than on how end-users use the systems as
part of their typical workflows. Further exploration is therefore needed to
understand the applicability of end-user robot programming systems in the
research literature to real-world domains, which may have requirements and
constraints that laboratory settings may fail to represent. Conducting more
observational and field studies may further help in gaining insight into how
users program for real-world use cases in uncontrolled lab environments, while
including open-ended programming tasks where end-users freely interact with a
system in user evaluations may reveal user programming behaviors outside of a
timed task context. Overall, further work is needed in end-user robot
programming to create experimental approaches and data measures and analyses
that faithfully represent realistic end-user programming experiences.

###  8.2. Supporting a Wider Range of Programming Capabilities

Although end-user robot programming has made many aspects of robot programming
accessible for end-users, there still remain key use cases of robots and
aspects of robot behavior that should be available for end-user customization
to further the applicability of end-user robot programming.

####  8.2.1. Programming Collaborative Tasks

While there have been several proposed approaches for end-user programming of
human-robot interactions, end-user robot programming techniques for
programming human-robot collaboration are less represented in literature. Most
end-user robot programming systems today skew towards programming autonomous
tasks, with any programmed human-robot interactions being simple or highly
structured. Unlike social human-robot interactions, which lend themselves to
highly structured programming since they can be composed of established social
norms (e.g., (Porfirio et al., 2018)), human-robot collaboration may not
always take the form of rigid turn-taking interactions. It instead can involve
open-ended structures that the sequential and templated constructs of
conventional end-user robot programming systems may not be able to handle.
Human-robot collaboration can involve an unbounded space of inputs from a
human collaborative partner and abstract concepts such as intent recognition
that may be difficult for end-users to program. Furthermore, human-robot
collaboration is highly dependent on external events performed by
collaborative agents, making it a highly dynamic process, with a pace and form
that may change over time.

Because of the idiosyncrasies involved in human-robot collaboration compared
to autonomous and structured robot actions, end-user robot programming systems
for programming human-robot collaboration must provide new types of
programming capabilities for end-users. For example, systems may need to move
towards enabling users to specify not only _what_ a robot will do (i.e., task
goal) but _how_ it will do it (i.e., style or collaborative role) (e.g.,
(Young et al., 2014)). Since stylistic aspects and robot roles may need to
change in real-time according to the collaborative context at any given
moment, which end-users may not be able to anticipate during programming, end-
user robot programming systems should also enable _metaprogramming_ whereby
end-user programs can generate robot behavior in real-time as the need arises
rather than requiring users to pre-author the robot behavior. End-user robot
programming systems should also support concepts such as parallel programming
and synchronization, which are typically difficult even for advanced
programmers (McKenney, 2017), in an intuitive manner for end-users, as
collaborative techniques such as action understanding and intent recognition
may need to run as a background thread while a collaborative program sequence
unfolds and inputs from human collaborators may arrive simultaneously, such as
co-temporal speech and gestures. In cases of collocated collaboration, end-
user robot programming systems may need to prioritize meeting strict safety
constraints, possibly by performing automatic verification and optimization
checks on user-created programs similarly to how social human-robot
interaction programs have been verified to meet social norms (Porfirio et al.,
2018).

Bridging the areas of end-user robot programming and human-robot collaboration
may be beneficial, as end-users know their collaborative preferences and
abilities best and may want to personalize the nature and timing of their
collaboration with a robot to better align with these preferences and goals
(e.g., (Wang et al., 2020)). Similar to how robotics as a field has shifted
from largely autonomous machines to interactive, collaborative technologies,
end-user robot programming will need to move from enabling highly structured
autonomous programming capabilities to robot capabilities that will interact
with and support human actions in new task contexts such as collaborative
manufacturing or co-design.

####  8.2.2. Programming Multiple Robots

The applicability of end-user robot programming systems to real-world
scenarios may be enhanced by enabling the simultaneous programming of multiple
types of robots. Due to variance in the programmable capabilities of different
robots, end-user robot programming systems are often designed to work with a
particular robot. Although the availability of resources such as ROS, which
has helped standardize the programming process for different robot
technologies, may help developers create open-source end-user robot
programming systems that can be easily modified to work with different robots,
most systems do not enable users to program different robots simultaneously.
This may limit the use of end-user robot programming systems, as some tasks
may be too complex or infeasible to perform without simultaneously programming
multiple robots, such as when tasks are not collocated but require
synchronization, as in a missile launch (Dudek et al., 1996). In these task
scenarios, multi-robot systems (MRSs), homogeneous or heterogeneous groups of
robots, can be required. MRSs are increasingly adopted in real-world domains
due to their benefits in terms of task efficiency (Burgard et al., 2000) and
system reliability and scalability (Prorok et al., 2012). Enabling intuitive
robot programming of MRSs while handling the increased user workload and
diminished situation awareness associated with attending to multiple robots
(Wong and Seet, 2017) is an open challenge that must be addressed to combine
the benefits of end-user robot programming and MRSs.

####  8.2.3. Programming Social Human-Robot Interaction

As robots make their way into everyday environments, there is expected to be
an increasing amount of social encounters between humans and robots. While
there have been tools designed for easing the process of generating social
behaviors for robots (e.g., (Huang and Mutlu, 2012)) and of authoring programs
for social human-robot interaction (e.g., (Porfirio et al., 2019b, 2018, a,
2020)), these tools are generally intended for expert human-robot interaction
designers rather than for the end-users interacting with the robot.
Introducing more programming tools for social HRI may be beneficial in
allowing human interactants to customize interactions according to their
preferences by, for example, specifying the robot’s personality and modifying
the robot’s interactive behaviors to meet social norms and work appropriately
in the current interaction context. End-user robot programming for social HRI
may be especially beneficial in keeping end-users engaged with robots in long-
term deployments by enabling them to customize robot operation to produce new
behaviors over time, which may reduce habituation effects (Koay et al., 2007;
Leite et al., 2013). End-user robot programming may also serve as a tool for
helping populations such as older adults or people with disabilities retain
their sense of independence when using the services of socially assistive
robots.

Work on end-user robot programming for social human-robot interaction remains
largely limited to programming of robots intended for activities such as
education and therapy, where interactions involve a limited set of robot
responses and are not necessarily representative of social human-robot
interaction in general. Enabling end-user robot programming of a wider space
of social HRI requires further investigation to determine the appropriate
methods and interfaces for enabling real-time end-user customization during
interaction scenarios. By investigating these open areas of research, end-user
robot programming tools can be put to use to better empower end-users during
social human-robot interactions and potentially reduce the possible negative
consequences of social HRI, such as emotional deception, by exposing the
programmable nature of robots to end-users.

###  8.3. Supporting Various Degrees of Programmability

End-user robot programming systems that emphasize expressivity are critical in
making programming of complex programs accessible for a wider range of users,
especially in domains such as manufacturing that may require programming of
more intricate procedures such as welding. However, citizen-level programming
systems that emphasize a constrained programming space may serve as an equally
valid approach towards developing end-user robot programming systems for
everyday environments, such as homes. For day-to-day tasks, citizen developers
may not have the need or the desire to engage in the end-to-end programming
processes typical of most end-user robot programming systems. They may instead
want to engage in quick specification of aspects of the programs that are
relevant to the need at hand, such as setting up the area to be cleaned for a
vacuum cleaner robot. User interactions with everyday technologies can
exemplify how robot programming should be structured for everyday needs. For
example, moving towards standard interaction conventions, such as the drag-
and-drop and swipe interactions end-users use to operate touchscreen
interfaces, for instructing robots may help make end-user robot programming
accessible when the goal is quick and easy modification and personalization of
robot behaviors, rather than detailed specification of a task procedure, and
may also minimize user difficulties in learning new programming conventions
and methods for every new robot that they encounter (e.g., (Han et al.,
2020)).

The varying needs of different users and domains with respect to the
expressivity of end-user robot programs centers around the tradeoff between
expressivity and simplicity. End-user robot programming systems often seek to
strike a balance between these concepts, including enough features to enable
programming of non-trivial tasks while constraining system features to
simplify programming for untrained users and to minimize the chance of user
errors (e.g., (Alexandrova et al., 2015)). End-user robot programming systems
that prioritize expressivity may be able to better handle a variety of use
cases but may be less accessible for users without programming experience
(Huang et al., 2016; Huang et al., 2020), who may favor a limited set of
programming capabilities (Leonardi et al., 2019). On the other hand, end-user
robot programming systems that oversimplify programming may be less favored by
more advanced users, who may desire a similar level of expressivity to the
amount found in traditional textual languages (Sefidgar et al., 2018).
Discovering the ideal balance between simplicity and expressivity and the
limits to which end-users can effectively make use of programming expressivity
is an open challenge that may determine how end-user robot programming systems
will take form in the future.

###  8.4. Supporting End-Users

End-user robot programming centralizes around the goal of making end-user
robot customization attainable for those without robotics experience. We
highlight open research directions for further supporting end-users in meeting
their programming goals without undue difficulties.

####  8.4.1. Collaborative Programming

Work on end-user robot programming has largely focused on supporting singular
users in easily programming robots. While some end-user robot programming
systems support robot programming by multiple users with different roles, such
as clinicians and caregivers, (e.g., (Datta et al., 2011)), the programming
interactions are generally limited to a single user at a time. However, in
real world domains, there are situations where collaborative or multi-user
programming may be desired. For example, clinicians may want to
collaboratively develop robot programs for interactive therapies with patients
to provide patients with flexibility in choosing robot actions that meet their
goals (Kubota et al., 2020). Furthermore, multi-user tandem programming can
help ease the workload of users when developing programs involving multiple
robots (Young et al., 2014). Further investigation is needed to determine
methods to simultaneously support multiple end-users during robot programming
while avoiding interference between users. Prior work on virtual pair
programming and multi-user visual programming may highlight design and
implementation tools that can be used to make end-user robot programming more
collaborative and social (e.g., (Campbell, 2002; Repenning et al., 2011)).
Special considerations should be taken for end-user robot programming systems
meant for children, where it may be necessary to provide equal opportunities
and control for each user to help reduce antagonism between children in
collaborative robot programming scenarios (Sapounidis and Demetriadis, 2013).

####  8.4.2. Assisted End-User Robot Programming

End-user robot programming has helped transform robot programming from an
occupation reserved for robotics engineers to an act that everyday users and
domain specialists can take to meet their personal needs. While the increased
accessibility of robot programming for users without robotics experience is
beneficial in empowering more users to customize robots to meet their needs,
it is not without its risks, as software developed by end-users without
programming experience tend to contain substantial amount of errors (Burnett
et al., 2006). Without the training undergone by developers, end-users may be
more likely to include workarounds and shortcuts in their programs and to
misjudge their own programming abilities (Harrison, 2004). Errors in robotics
software developed by end-users can be especially dangerous since robot
programs often include interactions with the physical world and people,
including vulnerable populations such as children and patients. Therefore,
there is a need for end-user robot programming systems to _assist_ users in
identifying errors and developing more optimal, correct programs.

Assistance is commonly found in computer programming, with integrated
development environments increasingly providing syntax-level assistance, such
as autocorrect and autocomplete, and more personalized, intelligent
assistance, such as code completion and smart suggestion. Similarly, end-user
robot programming systems should offer both syntax-level and personalized
assistance to users. While end-user robot programming systems in the research
literature have included assistance at the syntax-level (Section 6.2.5),
research on personalized assistance in end-user robot programming is still
incipient. To meet the needs of diverse groups of end-users while handling the
idiosyncrasies of individuals’ programming approaches and behaviors, end-user
robot programming systems should incorporate personalized data-driven
assistance as a tool, introducing intelligent tools that can identify an
individual’s programming intent and challenges in real-time to provide just-
in-time assistance that can help end-users avoid introducing errors and
suboptimalities into their programs. For example, assistance may be introduced
to kinesthetic teaching to reduce the tradeoff between users’ mental and
physical workloads. Programming by kinesthetic demonstration may be favored by
end-users for the low cognitive effort involved in kinesthetic teaching, but
the method may also be tiring and difficult for users because of the high
physical workload required to move robot manipulators (Quintero et al., 2018).
Introducing assistance into the kinesthetic programming workflow, such as by
offering support positioning the robot when the end-user is experiencing
difficulties in maneuvering a robot to a specific configuration or when they
are approaching a joint limit, may reduce the physical difficulties associated
with kinesthetic teaching and in turn reduce suboptimal and redundant
movements in the resulting robot program. Alternatively, assistance can be
provided by predicting the user’s intent during programming and proactively
completing the user’s program (e.g., (Liang et al., 2018)) or suggesting
program commands that can be used to meet the user’s programming goal.
Different programming methods may warrant different types of assistance, such
as assistance in perceiving relevant aspects of a program within a limited
field of view for AR-based programming.

Including assistance in end-user robot programming systems will involve
shifting more of the control over the programming process from the user to the
system. Providing less control to the user during robot programming can be
beneficial in reducing user workload in dealing with tedious aspects of robot
programming unrelated to the programming task such as motion planning (Ong et
al., 2020; Weintrop et al., 2017), in reducing imperfections in user programs
such as noise in kinesthetic demonstrations due to natural hand tremors
(Antonelli et al., 2013), and in increasing the flexibility of programs in
working in new contexts (Young et al., 2014). However, although prior work has
suggested that participants like automatic assistance provided by end-user
robot programming systems (Liang et al., 2019), end-users may be unwilling to
accept assistance if it means relinquishing control to automated assistance in
specifying programs (Racca et al., 2020) or if it means interference to their
programming workflow (Robertson et al., 2004). Furthermore, reduced user
control could lead to disinterest in programming for younger populations
(Sapounidis and Demetriadis, 2013). Therefore, although assistance has
potential to further reduce barriers that prevent everyday users from
effectively programming robots, more work is needed to understand the level of
assistance end-users are willing to accept and to develop assistance that
supports users in programming while still empowering them in leveraging robots
according to their individual needs.

##  9\. Conclusion

Similarly to how the pervasiveness of computers have necessitated methods for
end-users to program computing technologies, the increasing number of robots
in a diversity of domains, from the home to factory floors, has warranted
tools to achieve end-user robot programming. The challenges associated with
end-user robot programming in terms of the prerequisite programming,
engineering, and robotics expertise required to effectively program robots
have inspired many different techniques and methods to lower the barriers
involved in customizing robot behavior. The present work presents a survey of
the research literature covering end-user program specification, with a
structured review of the application domains, programming phases, and design
techniques and aspects encompassed by the end-user robot programming space. We
highlight both past trends and future directions in better supporting users,
conducting more representative user evaluations, and providing wider
programming spaces and ranges of programmability. Our survey not only provides
a comprehensive review of the end-user robot programming space, but also seeks
to highlight the need for continuous improvement on end-user robot programming
systems to truly democratize the customization of robotic technologies.

## References

  * (1)
  * Ajaykumar and Huang (2020) Gopika Ajaykumar and Chien-Ming Huang. 2020.  User needs and design opportunities in end-user robot programming. In _Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_. 93–95.
  * Alexandrova et al. (2014) Sonya Alexandrova, Maya Cakmak, Kaijen Hsiao, and Leila Takayama. 2014.  Robot programming by demonstration with interactive action visualizations.. In _Robotics: science and systems_. Citeseer.
  * Alexandrova et al. (2015) Sonya Alexandrova, Zachary Tatlock, and Maya Cakmak. 2015\.  RoboFlow: A flow-based visual programming language for mobile manipulation tasks. In _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 5537–5544.
  * Antonelli et al. (2013) Dario Antonelli, S Astanin, M Galetto, and L Mastrogiacomo. 2013.  Training by demonstration for welding robots by optical trajectory tracking.  _Procedia Cirp_ 12 (2013), 145–150.
  * Argall et al. (2009) Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009.  A survey of robot learning from demonstration.  _Robotics and autonomous systems_ 57, 5 (2009), 469–483.
  * Bambušsek et al. (2019) Daniel Bambušsek, Zdeněk Materna, Michal Kapinus, Vítězslav Beran, and Pavel Smrž. 2019.  Combining Interactive Spatial Augmented Reality with Head-Mounted Display for End-User Collaborative Robot Programming. In _2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 1–8.
  * Barakova et al. (2013) Emilia I Barakova, Jan CC Gillesen, Bibi EBM Huskens, and Tino Lourens. 2013.  End-user programming architecture facilitates the uptake of robots in social therapies.  _Robotics and Autonomous Systems_ 61, 7 (2013), 704–713.
  * Barišić et al. (2018) Ankica Barišić, João Cambeiro, Vasco Amaral, Miguel Goulão, and Tarquínio Mota. 2018.  Leveraging teenagers feedback in the development of a domain-specific language: the case of programming low-cost robots. In _Proceedings of the 33rd Annual ACM Symposium on Applied Computing_. 1221–1229.