# DeFINE: Delayed Feedback based Immersive Navigation Environment for Studying
Goal-Directed Human Navigation

###### Abstract

With the advent of consumer-grade products for presenting an immersive virtual
environment (VE), there is a growing interest in utilizing VEs for testing
human navigation behavior. However, preparing a VE still requires a high level
of technical expertise in computer graphics and virtual reality, posing a
significant hurdle to embracing the emerging technology. To address this
issue, this paper presents Delayed Feedback based Immersive Navigation
Environment (DeFINE), a framework that allows for easy creation and
administration of navigation tasks within customizable VEs via intuitive
graphical user interfaces and simple settings files. Importantly, DeFINE has a
built-in capability to provide performance feedback to participants during an
experiment, a feature that is critically missing in other similar frameworks.
To show the usability of DeFINE from both experimentalists’ and participants’
perspectives, a demonstration was made in which participants navigated to a
hidden goal location with feedback that differentially weighted speed and
accuracy of their responses. In addition, the participants evaluated DeFINE in
terms of its ease of use, required workload, and proneness to induce
cybersickness. The demonstration exemplified typical experimental
manipulations DeFINE accommodates and what types of data it can collect for
characterizing participants’ task performance. With its out-of-the-box
functionality and potential customizability due to open-source licensing,
DeFINE makes VEs more accessible to many researchers.

###### keywords:

Virtual reality, Software, Closed-loop, Locomotion, Gamification, Unity

Kshitij Tiwari* and Ville KyrkiAllen CheungNaohide Yamamoto* Department of
Electrical Engineering and Automation, Aalto UniversityQueensland Brain
Institute, The University of QueenslandSchool of Psychology and Counselling,
Queensland University of Technology (QUT)

Orcid IDs: Kshitij Tiwari https://orcid.org/0000-0003-1789-7961

Ville Kyrki https://orcid.org/0000-0002-5230-5549

Allen Cheung https://orcid.org/0000-0001-9770-217X

Naohide Yamamoto https://orcid.org/0000-0001-9734-7470

The authors would like to thank Onur Sari and Ville Sinkkonen for their
contributions to the development of the framework presented in this article.

*Corresponding authors—KT: Center of Ubiquitous Computing, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland (email: kshitij.tiwari@oulu.fi); NY: School of Psychology and Counselling, Queensland University of Technology, Brisbane, Australia (email: naohide.yamamoto@qut.edu.au).

Behavioral researchers are increasingly becoming interested in understanding
the underlying mechanisms for goal-directed navigation in humans (Spiers
Maguire, 2006; Cornwell ., 2008; Pezzulo ., 2014). Whilst it would be ideal to
gather data on navigation in a well-controlled real-world setting, not every
researcher has access to such a setup. However, with the advances in virtual
reality (VR), it is becoming easier to set up economical game-like
environments to test hypotheses about human navigational behaviors.

Although VR technologies are becoming more accessible, they are primarily for
playing computer games and not for carrying out behavioral experiments. Thus,
to use them for investigating human navigation, researchers need to have
working knowledge of graphics design and game engines like Unity
(https://unity.com/) and Panda3D (https://www.panda3d.org/), which may not be
the case even when they are proficient in general scientific programming. To
address this issue, attempts have been made at developing easy-to-use VR
frameworks for researchers who are novice at computer graphics and game
creation. Notable examples include Python-based Experiment Programming Library
(PyEPL; Geller ., 2007), Maze Suite (Ayaz ., 2008, 2011), PandaEPL (Solway .,
2013), Experiments in Virtual Environments (EVE; Grübel ., 2017), Virtual
Reality Experiments (VREX; Vasser ., 2017), VRmaze (Machado ., 2019), Unity
Experiment Framework (UXF; Brookes ., 2020), Route Learning and Navigation
Test Battery (Wiener ., 2020), NavWell (Commins ., 2020), BiomotionLab Toolkit
for Unity Experiments (bmlTUX; Bebko Troje, 2020), and Landmarks (Starrett .,
2020).

These existing frameworks are similar in that they offer some or all of the
following functions for conducting behavioral experiments in VR: modeling and
rendering virtual environments (VEs), designing the structure of an experiment
(e.g., setting the number and order of trials), executing the trials,
recording data (e.g., participants’ navigation trajectories), and performing
preset analyses of the recorded data. A major difference between the
frameworks is in the extent to which they are designed for specific purposes.
For example, Maze Suite is specialized for creating and running experiments in
standard mazes (i.e., mazes created by dividing an enclosed space into
connected paths by walls). By limiting its scope, Maze Suite achieves a high
level of ease of use—that is, everything can be done in graphical user
interfaces (GUIs) and no coding is necessary on the part of end-users. At the
other end of the spectrum, PyEPL and PandaEPL are Python libraries for
programming behavioral experiments in general (PyEPL) and spatial navigation
experiments in particular (PandaEPL). Because they are not compiled software
packages but programming libraries, researchers can create any experiments
using PyEPL and PandaEPL, as far as their programming proficiency goes. The
other frameworks lie in between, balancing ease-of-use and flexibility to
varying degrees by preparing some ready-made functionality while providing
users with the way of doing their own coding for fine customization. Some of
them are still relatively specific to certain types of experiments (e.g., VREX
focuses on indoor environments that consist of connected rooms). The others
offer generic modules for experimental design and data recording, leaving
high-level features of experiments including stimulus presentation up to
users’ programming (e.g., UXF).

Despite the different purposes these frameworks are designed to achieve, one
aspect shared by them is that they focus on the stimulus–response relationship
in examining navigational behavior. That is, participants are presented with a
stimulus with which they carry out a navigation response (e.g., walking to a
goal location indicated by visual cues), and during and at the completion of
the response, they do not receive any feedback on their performance. Although
such a research design is appropriate and even required for investigating
certain aspects of navigational behavior (Philbeck Loomis, 1997; Loomis Knapp,
2003), it makes it impossible to examine how participants modulate their
subsequent response to the stimulus using feedback (Brand, 2008).

Importantly, when goal-directed navigation is performed in real-world
settings, navigators often receive feedback with which they can adjust their
behavior. For example, when they walk in the dark to reach a door and fail to
touch its knob, the lack of tactile sensation serves as feedback and informs
them that they still have a few more steps to go. In this instance, the
navigational behavior should be characterized as consisting of a closed
stimulus–response–feedback loop, instead of an open stimulus–response loop as
in the foregoing frameworks. Indeed, studies have been conducted to capture
human navigation by the stimulus–response–feedback loop. For instance, Carton
. (2016) have found evidence that humans tend to adapt their trajectory
planning horizon (i.e., how far in the future they plan their locomotion
trajectory) when they detect potential collision situations. In this case,
impending collisions constitute a stimulus that triggers a response, which is
witnessed in the form of a shortened planning horizon. The successful
avoidance of collision or a failure thereof functions as feedback that can
subsequently be used to tune the length of planning horizons. In this manner,
experiments on human navigation come in a variety of designs, utilizing both
stimulus–response and stimulus–response–feedback loops. To fully accommodate
this diversity of the experiments, there is a need for a new VR framework that
allows for incorporation of feedback into the experimental design. To this
end, we developed the Delayed Feedback based Immersive Navigation Environment
(DeFINE).

##  1 DeFINE: Delayed Feedback based Immersive Navigation Environment

DeFINE is freely available to download via GitHub at
https://github.com/ktiwari9/define-VR. DeFINE is based on the Unity game
engine, and hence, relies heavily on C# as a programming language. All the
low-level implementation is already taken care of to minimize the workload of
end-users who will use DeFINE in its default settings or with minimal
customization as required by their experimental design. DeFINE aims
specifically to provide an easy-to-use experimental framework that is based on
the stimulus–response–feedback architecture, which can be used to study goal-
directed spatial navigation in an immersive three-dimensional space. In order
to reduce the burden of researchers when setting up an experiment, DeFINE
allows them to make simple alterations of experimental parameters through GUIs
and by changing settings files that are in a straightforward JSON format. We
provide a short video that succinctly explains how to use the basic
functionality of DeFINE “out-of-the-box” (https://youtu.be/OVYiSHygye0). To
allow for further customization of the experiment and the framework itself,
DeFINE is being released open-source under the MIT license. Thus, interested
researchers can modify its code directly. To facilitate this, DeFINE’s
codebase is made modular, making it possible to alter a particular pre-
programmed functionality by adjusting the code in a specific module only, and
also to incorporate a new functionality into the framework by creating a new
module and placing it in an appropriate folder. The file hierarchy is kept
intuitive for this purpose. Another video we provide offers a quick tutorial
of how to change various elements of DeFINE using the Unity software
(https://youtu.be/smIp5n9kyAM). A detailed user manual is also available
(https://github.com/ktiwari9/define-VR/blob/master/user_manual.pdf).

Currently, DeFINE can be integrated into Unity tasks built for Windows
personal computers. It is assumed that DeFINE will be used with a head-mounted
display (HMD) such as HTC Vive and Oculus Rift. For example, DeFINE is
designed to utilize the HMD’s motion tracking sensors for implementing various
methods of participants’ locomotion within VEs (see the locomotion methods
section below for details). In addition to the HMD worn by the participants,
DeFINE simultaneously presents a VE to a desktop display so that
experimentalists can monitor the progress of an experiment. Further details
about hardware and software requirements for DeFINE are available in the user
manual.

The main capabilities and options of DeFINE are detailed below in the
following order: (1) the generic experimental structure, (2) time- and
accuracy-based feedback, (3) the GUI, (4) a diverse suite of locomotion
methods, (5) static and dynamic goals, (6) performance leader-board, and (7)
intra-VR surveys.

###  1.1 Experiment Structure

Human behavioral experiments are often defined by a trial–block–session
architecture which allows experimentalists to repeat a task multiple times to
acquire requisite data (Figure 1). Just like the UXF (Brookes ., 2020), DeFINE
adopts this architecture. A trial is an instance of the experiment where
participants are presented with a stimulus and their response is recorded. At
the end of the trial, the participants receive feedback. To clarify that the
feedback is given after the response is made, as opposed to during the trial
as the response unfolds, this feedback is referred to as “delayed” in DeFINE.
Trials are often repeated multiple times for various purposes (e.g., to
measure variability of responses to the same stimulus, to decipher the
learning effects over trials, or to train the participants on the task),
constituting a block. At the end of the block, the participants are assumed to
be familiar with the environment and to have formulated a behavior of choice
for the task at hand. In order to evaluate the quality of this behavior, the
experimentalists may choose to make some modifications to the environment
before proceeding with the next block. The experiment can consist of a single
or multiple blocks, and when there are multiple blocks, a single iteration of
the task over the blocks is called a session.

Figure 1: Structure of typical human spatial navigation experiments.
Participants are presented with a stimulus, to which they provide a response
and are given feedback after the trial has been fully executed. This feedback
is then used to modify performance in the next trial. Usually, multiple trials
are carried out under the same condition, which constitute a block. Once the
participants are finished with the block, they progress to the next block
which may involve modifications of the condition, and their navigation
performance is evaluated again over multiple trials. A single iteration of the
task is called a session. This trial–block–session architecture as well as the
diagram itself are modeled after those presented by Brookes . (2020). Figure
2: The architecture of the Delayed Feedback based Immersive Navigation
Environment (DeFINE). The left panel shows the low-level functionality
provided by DeFINE while the high-level experimentalist-defined
functionalities are shown on the right. The surveys/questionnaires are
optional, and hence, shown using dashed lines. Modules shown in purple are
parsed during a trial, those in red during a block and those in green during a
session. The logo of DeFINE shown at the top was inspired by that of UXF
(Brookes ., 2020), symbolizing that DeFINE is built as a navigation-focused
extension of UXF.

DeFINE has primarily two levels of abstraction as shown by the two columns
(demarcated by orange dotted lines) in Figure 2. All modules mentioned on the
left are those implemented at the low-level abstraction that come pre-
programmed with DeFINE, while those on the right are implemented at a higher
level for easy and quick modifications. Such an arrangement streamlines
customization of an experiment to suit different research studies because the
modules at the low level of abstraction are common across many experiments and
thus, the experimentalists can rely on DeFINE’s built-in functionality, which
in turn allows them to focus their effort on customizing high-level modules
that tend to be unique to each experiment. All modules shown by dotted black
lines represent optional modules that can be utilized as seen fit by the
experimentalists. In keeping with the trial–block–session architecture as
described in Figure 1, all modules clustered under the green polygon (“Begin
Session”) represent a session, those under the red polygon (“Begin Block”)
represent blocks, and those under the purple polygon (“Begin Trial”) represent
trials. All other modules (under either column of abstraction) are preset
before the session starts.

The flow of an experiment starts from the top of Figure 2. DeFINE takes care
of initializing and rendering an immersive environment and setting up a
session, blocks, and trials, but the experimentalists are required to
configure the VE (e.g., setting its size) and specify the experimental design
(e.g., the number of trials per block) by modifying the settings files in
order to make DeFINE behave in the way they need for their experiment. Details
of the settings files are described later. If the experimentalists want
questionnaires to be filled within the VE, these questionnaires need to be
created online, and the respective links to the questionnaires need to be
added to the environment settings. Setting up the questionnaires is followed
by inputting a participant’s information and selecting preset settings that
are appropriate for the experiment. Once the participant information has been
entered, the experimentalists can start the experimental session that DeFINE
will generate. If there are block-specific settings, those are applied by
DeFINE before starting the first trial. The trials follow the
stimulus–response–feedback structure, by showing the participant a stimulus in
the environment, to which the participant responds, and after the
experimentalist-defined end condition has been met, the trial ends and
feedback is shown to the participant. A new trial is started automatically
after the feedback has been given, until the program reaches the end of the
block. At the end of the block, the participant will be shown the
questionnaires, if any were specified during the initial setup. During the
trials the experimentalists can take down notes, or manually mark the session
as bad to indicate that it should not be taken into account for data analysis.
At any point of the session the experimentalists can abort the session, which
will also mark the session as bad in the stored session notes. If there are
more blocks remaining in the session, the next block is started after DeFINE
has applied its specific settings to the environment. After the final block of
trials has been completed, and the questionnaires following it have been
answered, the session ends and the participant and experimentalists are
returned to the startup screen. For each of the trials executed with DeFINE,
the participant’s trajectory during the trial, the status and changes of
environment variables, the feedback score obtained at the end of the trial
(see the next section for details), trial start time, trial end time, total
time taken for the trial, and straight-line distance to the goal when the
participant ends the response are logged.

###  1.2 Feedback

As opposed to other closed-loop systems where real-time feedback may be made
available to participants as they carry out a response, DeFINE provides
delayed feedback at the end of each trial. This is because continuous feedback
will most likely go against the purpose of typical goal-directed navigation
experiments. Usually, these experiments are to test participants’ ability to
estimate their location relative to a goal by using sensory and other spatial
cues in an environment (Loomis Philbeck, 2008). If the participants were given
external feedback about whether they were moving in the right direction for
every step they took, this feedback would essentially be a non-spatial cue
that would directly aid them in their location estimation. In extreme cases,
with continuous feedback, the participants could perform the task by moving in
an arbitrary direction without processing the sensory and spatial cues and
seeing if that would result in positive feedback. Such a strategy would lead
them to take myopic unstructured paths to the goal, causing non-optimal
navigation performance. Thus, by default, DeFINE is designed to give
performance feedback only after the trial is completed. However, it is
possible for experimentalists to modify DeFINE’s source code in Unity and have
it provide real-time feedback, if they so choose.

Feedback on a goal-directed navigation behavior can be given in a number of
different forms, but DeFINE adopts a reward/cost function that evaluates
participants’ performance and provides feedback as gains and losses of scores.
It has been shown that feedback of this type is very effective in affecting
participants’ behavior and decision making under a variety of conditions
(Brand, 2008; Hossain List, 2012; Yechiam Hochman, 2014). The reward and cost
in the context of navigation can also be defined in various ways, and it is up
to experimentalists’ discretion how they formulate the reward/cost function in
DeFINE, but one straightforward method would be to define them by using speed
and accuracy of navigation. That is, the quicker the participants are in
performing a trial, the greater the reward (or the smaller the cost); and the
more accurate they are in reaching a goal, the greater the reward (or the
smaller the cost). By default, DeFINE implements a reward/cost function of
this form. Specifically:

$R=\beta_{1}\exp(-\alpha_{1}t)+\beta_{2}\exp(-\alpha_{2}d)$ (1)

In Equation (1), $t$ refers to the time taken for navigating towards a goal in
a trial, and $d$ refers to the residual straight-line distance to the goal
from the location at which participants end the trial. $\beta_{1}\text{ and
}\beta_{2}$ are weights used for combining the time and distance into the
decaying reward function, which penalizes both the time taken and the residual
distance to the goal (i.e., rewarding shorter times and smaller distances with
higher scores). $\alpha_{1}\text{ and }\alpha_{2}$ are factors for scaling the
effects of time and distance. If experimentalists choose to use this function
in their experiments, they can assign values of their choice to these
parameters simply by specifying them in a settings file (details shown later).
If they are to calculate the reward/cost scores using their own equation, they
can do so by modifying a relevant section of DeFINE’s codes in Unity. It
should further be noted that, by changing the relevant codes and implementing
their own equation, the experimentalists can use any kinds of feedback that do
not take the form of a cost/reward function. For example, it is possible to
simply present how far away participants are from the goal at the end of each
trial.

###  1.3 Graphical User Interface (GUI)

Utilizing the GUI of UXF (Brookes ., 2020), DeFINE allows experimentalists to
log participant information including, but not limited to, name (or
participant identification), age, gender, and educational qualification
(Figure 3(a)). Should other personal particulars be required, they can be
easily added to the framework by modifying relevant settings files. As an
extension to UXF’s original GUI, DeFINE allows the experimentalists to quickly
set up the environment of choice with a desired locomotion method (see the
next section for details about the locomotion methods). This unique feature
can also be scaled and automated to handle multiple combinations of
environments and locomotion methods via DeFINE’s auto-pilot mode. In this
mode, the experimentalists can provide DeFINE with preset instructions so that
it loads specific combinations of the environment and the locomotion method in
a specified order. This way, a sequence of participants can be tested
automatically, doing away with the need to individually set up an appropriate
combination of the environment and the locomotion method for every
participant. For example, if an experimental design requires that each
participant is shown a different environment, a sequence of environments can
be explicitly listed in settings files which will then be autonomously parsed
when executing the auto-pilot mode. If each participant is to do trials with a
different locomotion method, explicit participant-locomotion method
combinations can be listed in the settings files in a similar fashion.

As a significant extension to the predecessor, UXF, DeFINE also provides
functionalities to study the role of lighting conditions and auditory cues in
spatial navigation. At any point during a trial, experimentalists can toggle
the lights and sound of a VE on or off by clicking on the dedicated buttons on
the user interface, shown in Figure 3(b). The change of the status of these
environmental variables are logged along with the information about
participants’ performance in a navigation task (e.g., their position within
the environment at a given time point).

(a) Drop-down menus to specify experimental parameters (adapted from UXF;
Brookes ., 2020).

(b) The buttons for toggling sound and lights.

Figure 3: DeFINE’s intuitive GUI for experimentalists to set up an experiment.

###  1.4 Locomotion Methods

In order to provide a locomotion suite for participants to perform goal-
directed navigation in VR, DeFINE comes equipped with a variety of locomotion
methods.

####  1.4.1 Teleoperated locomotion

In order to allow teleoperated locomotion, DeFINE is compatible with both a
keyboard based and a VR controller based teleoperated methods. A typical use
case may involve head direction sensors in an HMD being used to update
participants’ headings in a VE while the keyboard or the VR controller being
used to linearly traverse at preset velocity (which is to be specified in a
settings file by experimentalists). The necessary key-bindings and further
details are available in the user manual (https://github.com/ktiwari9/define-
VR/blob/master/user_manual.pdf).

####  1.4.2 Arm-swing locomotion

Arm-swing locomotion is an implementation of walking-in-place locomotion. In
this method, the participants walk in place, including swinging their arms in
a manner consistent with their pace of walking. It has been shown that such
arm swings are effective in having the participants experience a naturalistic
sense of locomotion without actually moving in real space (Kunz ., 2009;
Yamamoto ., 2018). This locomotion method uses the physical movement of the VR
controller(s), held by the participants, to determine forward speed in the VE.
The movement speed is calculated from the positional difference of the tracked
controller(s) between two consecutive frames. This calculation can be done
either by requiring movement of two controllers (typically, one in each hand)
or by using one controller (or either of the two controllers) that moves more
than a given threshold amount between the frames. When the use of the two
controllers is required, the forward speed in the environment is set to be
zero, unless both of the controllers exceed the threshold value.

####  1.4.3 Head-bob locomotion

Head-bob locomotion is another implementation of walking-in-place locomotion.
In order to move forward in the VE, the participants need to walk in place,
and as they do, their head, and in particular, the HMD, bobs slightly
vertically. This locomotion method uses this vertical bobbing to determine the
forward velocity. DeFINE tracks the vertical direction of the bobbing and its
starting position. Once the direction changes, the participants are considered
to have stepped, if the vertical height difference between two successive
flexion points exceeds a threshold value specified in the settings of the
locomotion method. The detected physical step is then translated into a step
in the VE so that the participants walk forward in the VE at preset velocity.
Due to the fact that the HMD is in front of the participants’ face, turning
their head up or down causes the HMD to move vertically. In order to avoid
reading these vertical movements as steps of the participants, DeFINE also
tracks the participants’ rotational head movements about the pitch axis and
ignores any “bobs” that are accompanied with the rotational head movements
that exceed a specified threshold value.

####  1.4.4 Physical walking

Physical walking is the only locomotion method in which the participants
physically move around in the real world. The movement of the participants is
tracked by using the HMD’s motion tracking sensors and the participants’
position in the VE is updated accordingly. Owing to the limited size of a
physical area in which the participants’ movement can be tracked (which is
typically around $10\times 10$ m), the size of the VE is going to be limited.
To alleviate this limitation, sometimes modified physical walking methods such
as redirected walking are adapted (Paludan ., 2016; Nilsson ., 2018). In these
methods, the rotations and translations of the participants are slightly
altered between physical and virtual worlds in order to steer the participants
away from the edges of the available physical area. However, DeFINE does not
utilize these methods because they can induce disruption to mental and neural
spatial representations as well as to navigational behavior by causing a
mismatch between intended (and physically carried out) movements and
consequent virtual movements (Du ., 2020; Tcheang ., 2011).

In DeFINE, a visible grid barrier, shown in Figure 4, is displayed in the HMD
when the participants approach the limits of a configured area in which they
can safely move around. The grid barrier serves two purposes. First and
foremost, it prevents the participants from going out of the physical safe
area, ensuring their safety. It is advisable that a navigation task in DeFINE
be well confined to an area smaller than the safe area so that the
participants will never encounter the barriers in the first place. If they do
view the barrier, it essentially functions as an extra landmark that informs
about the boundary of an environment, which can induce significant bias in
their navigational behavior (Cheung, 2014; Mou Zhou, 2013; Bird ., 2010).

Second, the barrier makes it possible to extend a navigable virtual space
beyond the physical safe area, in case it is necessary for an experiment. To
do this, the participants hold a trigger button in the VR controller, which
locks the VE in place. While the VE is locked, the participants’ physical
rotation in real space is not reflected in their virtual heading. Thus, the
participants appear to keep facing the same direction in the VE, despite
physically turning to face away from the edge of the safe area. The grid
barrier remains visible and in correct orientation in respect to the physical
safe area, allowing the participants to reorient themselves before continuing.
In order to minimize the motion sickness caused by the VE remaining still
during the participants’ physical rotation, DeFINE blurs the VE during the
rotation. Unlike similar approaches used in the literature (Williams ., 2007),
DeFINE does not require the participants to rotate a fixed amount as long as
they steer clear of the physical boundary.

Although this method of extending the virtual space can be practical, it must
be used with caution because by physically rotating in the locked VE, the
participants will most likely be forced to go through a mental process of
dissociating real and virtual spaces once and realigning them after the
physical rotation. It is very probable that this process will have significant
impact on the participants’ mental and neural spatial representations, and in
turn, on their subsequent navigational behavior.

Figure 4: The visible blue grid marking the edge of a configured safe area.

####  1.4.5 Teleportation

Teleportation locomotion differs from all of the other locomotion methods in
that the participants never move through space, but instead teleport directly
to their desired location some distance away. Before and after teleporting,
the direction of the participants’ body and head remain unchanged relative to
the environment. In DeFINE, the participants teleport by holding the trigger
of a VR controller, which brings up the teleportation marker, as seen in
Figure 5. Then the participants place the marker on the desired teleportation
target location by aiming the controller at the location. Once the
participants release the trigger, they are teleported to the marked location,
given that the location is on the horizontal X-Z plane and clear of all
collision regions around the objects in the environment. A valid target
location is indicated by the blue color of the marker, whereas invalid
locations turn the marker red. Although teleportation is not a naturalistic
method of locomotion, its use is increasingly common in VEs including those
for spatial navigation research (Cherep ., 2020; Kelly ., 2020).

Figure 5: Teleportation locomotion with the teleportation marker visible.

###  1.5 Goal Demarcation

To accommodate a variety of experiments, DeFINE offers two possible ways of
demarcating the goal location for a navigation task: firstly, presenting
static objects at goal locations (e.g., arrows, exclamation marks, or other
similar objects) and secondly, showing dynamic objects like a buzzing fly that
can give imprecise indication of the goal location. An example of the dynamic
goal markers is available in the demonstration section below.

###  1.6 Leader-board

Gamification of learning has shown increased participant engagement be it for
online programs (Looyestyn ., 2017) or education (Barata ., 2013). Thus, as an
optional feature, DeFINE is equipped with a leader-board which provides a
ranking based on scores obtained using Equation (1) or other equivalent
equations implemented by experimentalists (Figure 6). DeFINE keeps track of
the scores and displays ten best scores in the leader-board. A new high-score
is indicated with red font in the leader-board, while a score that was not
high enough to get to the leader-board is shown at the bottom to illustrate
the difference between the latest score and pre-existing scores. If
participants are to carry out some practice or training trials first, it may
not be appropriate to compare their scores against the pre-existing scores
before they become fully familiarized with an experimental task. In that case,
it is possible to show a provisional ranking which is not integrated with the
leader-board. For clarity, this is labeled with a red Practice tag in the
board. Once the practice phase is finished, the actual scores of the
participants are integrated into the leader-board that includes their own
previous high scores.

(a) A new high-score is shown in red in the leader-board.

(b) The latest score, which does not make it to the top ten, is shown at the
bottom.

Figure 6: Leader-boards inform participants about their performance in a given
trial relative to their own and other participants’ previous trials.

While having a leader-board can motivate participants, it can also cause the
conditions of an experiment to be different between the participants. As
earlier participants obtain their place in the leader-board, they keep
replacing lower scores on it. As such, it gets systematically more difficult
for later participants to score high enough to make it to the top-ten scores
of all time. Having a leader-board that is seemingly unreachable might provide
a different motivation to the later participants than having an easily
reachable one would. In order to ensure that each participant can have an
equal experimental condition, DeFINE offers two options. First, because the
leader-board is an optional feature, experimentalists can choose to remove it
entirely. Second, they can use a fake leader-board that behaves like a normal
leader-board during the session of one participant, except that the changes to
the board are not in fact stored to a log file. Once the next participant
begins a session, the board reverts to its original condition, giving
subsequent participants the same competitive challenge.

###  1.7 Surveys

Often in behavioral studies, experimentalists would like participants to fill
surveys for quality assurance or other purposes. Some of the most commonly
used surveys for such studies using VR include the simulator sickness
questionnaire (SSQ; Kennedy ., 1993) and the NASA task load index (NASA TLX;
Hart, 2006). The SSQ studies the onset of simulator sickness symptoms like
nausea or headache owing to being immersed in VR. It contains 27 questions and
the participants answer each of them using a scale ranging from none (0) to
severe (3). The NASA TLX is a survey for evaluating the workload of a given
task utilizing six questions. Administering these and any other surveys has
been made conveniently possible in DeFINE (Figure 7). The surveys are visible
in an HMD to the participants and also on a desktop display to the
experimentalists. While questions that have preset choices can be answered
directly by the participants using the VR controller(s), questions that
require free-form responses are to be typed in by the experimentalists on
behalf of the participants.

While DeFINE’s survey system allows the experimentalists to administer surveys
while keeping the participants immersed in VR, other systems typically require
them to take off an HMD to answer the surveys. Thus, if an experiment involves
multiple sessions or blocks, each of which contains surveys, the participants
need to be re-immersed in VR every time they remove the HMD and put it back on
(Schatz ., 2017). This can be very cumbersome and make the participants feel
uncomfortable, possibly inducing cybersickness. An alternative could be that
the experimentalists orally ask questions and fill in the surveys on behalf of
the participants, but this can feel intrusive to the participants and reduce
the sense of immersion in VR because the participants have to directly
communicate with the experimentalists who do not belong to the virtual world
(Bowman ., 2002). DeFINE remedies these issues by displaying the surveys in
the HMD. To our knowledge, only Grübel . (2017) and Regal . (2018) implemented
a similar system previously.

Figure 7: Filling in surveys whilst being immersed in VR using DeFINE.

##  2 Demonstration

To demonstrate various built-in functionalities of DeFINE and its overall
usability for human navigation experiments, we asked actual participants to
perform a simple goal-directed navigation task in DeFINE. In each trial, the
participants navigated to a hidden goal that was indicated by a dynamic visual
cue and received a feedback score on the leader-board. The participants’
navigation response was characterized by the total duration of the response,
the residual distance to the goal at the end of the trial, the feedback score,
and detailed time-course plots of their locomotion trajectory, exemplifying
the types of data DeFINE can record. To implement the trial–block–session
architecture, two blocks of trials were used, between which the environment
and the dynamic cue were modified. Two ways of deriving the feedback scores,
which differentially weighted the response duration and the residual distance,
were defined through DeFINE’s settings files. The participants also answered
the SSQ and NASA TLX. This was to give a demonstration of DeFINE’s intra-VR
survey feature, and also to examine whether the participants tolerated the use
of DeFINE in the current experiment, which was of typical scale as a
behavioral experiment.

Figure 8: The default experimentalist view of an environment containing walls
and floor with non-repeating textures. A countdown timer and score window
helps participants keep track of their performance. Light and sound can be
toggled by experimentalists. The firefly serves as a noisy visual cue to guide
the participants to a goal. The light/audio toggle switches (top right) and
experimentalist button descriptors (top left) are only visible to the
experimentalists on a desktop running DeFINE, and are not made visible to the
participants in an HMD. Except for this difference, the participant and
experimentalist views are the same. The arrows specifying X, Y, and Z axes in
the lower-left corner are shown in this figure only; they appear neither in
the participant nor experimentalist view. See text for further details.

###  2.1 Method

####  2.1.1 Participants

Twenty-four participants (15 males, 8 females, and 1 other) took part in this
experiment. Twenty-three of them were students of Aalto University, and one
was from the vocational university Metropolia, Finland. The mean age of the
participants was $24.8\pm 2.9$ years. The participants’ educational background
ranged from having graduated from high school to having a master’s degree. All
participants gave written informed consent to participate in the experiment
and received a movie ticket in return for their participation. The protocol of
the experiment was approved by the Aalto University Research Ethics Committee.

####  2.1.2 Design and materials

The participants were asked to navigate from start to goal positions within a
virtual room of $10\leavevmode\nobreak\ \times 10$ m with non-repeating
textures as shown in Figure 8. The walls were $4$-m tall. The participants
navigated using the controller teleoperation method. Specifically, they turned
the head to face the direction they wanted to go and moved in that direction
by pressing a button of a VR controller. The participants’ eye height in the
virtual room was set at $1.36$ m, which approximately corresponded to their
actual eye height while seated in a real room. The participants first went
through a block containing $15$ trials in which the room walls were visible.
Subsequently, they performed the same navigation task in the second block in
which the walls were removed and the floor texture was extended to the
horizon. This block also consisted of $15$ trials. The starting and goal
positions were fixed across the trials as well as across the blocks. Relative
to the center of the room, in a left-handed coordinate system, the starting
position was at ($4.5$ m, $4.5$ m) and the goal position was at ($-3$ m, $-1$
m). At the starting position, the participants directly faced the hidden goal
position with an orientation of $-135^{\circ}$ about the vertical Y-axis. As a
dynamic goal marker, a firefly buzzed around the goal position in such a way
that its randomly fluctuating flying trajectory had its center directly over
the goal position. Specifically, in each frame, the fly’s position along the
horizontal X-Z plane was randomly sampled within the radii of 0.75 and 1.5 m
from the goal in the first and second blocks, respectively. The height of the
fly along the Y-axis was randomly sampled in the range of
$0.75$–$1.25\leavevmode\nobreak\ $m. To make the fly move smoothly, its
position was incremented with a step size of $5$ mm. For a graphical
presentation of X, Y, and Z axes, see Figure 8.

In this manner, the fly represented a noisy visual cue to guide the
participants to the goal position. That is, the exact goal position was never
revealed to the participants, and instead they were told that the goal was
somewhere inside the area delimited by the fly’s trajectory. Hence, the goal
position was provided imprecisely to the participants via the noisy visual
cue, and also the feedback score.

The participants were assigned to one of two groups, both of which received
scores for their performance that depended on both navigation speed and
accuracy, but with different weights. Time group received feedback that put
more importance on speed while the feedback in accuracy group was weighted in
favor of being as close as possible to the goal. The feedback provided to the
participants was computed using Equation (1) with the constants shown in Table
1. The score was presented to the participants at the end of each trial.

Table 1: Constants of the reward function for two participant groups. Constant | Time group value | Accuracy group value
---|---|---
$\alpha_{1}$ | $-0.05$ | 0.2
$\alpha_{2}$ | 0.2 | 1
$\beta_{1}$ | $-2$ | 0.5
$\beta_{2}$ | 6.2 | 3.4

The participants were informed that they would be graded according to the time
elapsed and residual distance to the actual goal. However, they were not told
about the existence of the two groups or which group they belonged to.
Instead, the participants of both groups were told that the scores obtained
for the trials would be based on both speed and accuracy. In order to make the
scores easier for the participants to follow, they were scaled up by a factor
of 300 and their minimum value was set at 0 (i.e., no negative scores).

DeFINE was run on a high-performance Windows 10 personal computer with an
Intel Core i5 processor, 32 gigabytes of random-access memory, and an Nvidia
GTX 1070 graphics card. The HTC Vive Pro with a wireless extension was used
for a VR HMD. DeFINE was used for presenting the VE and recording data for
each frame (approximately 90 frames per second). Specifically, the following
log files were created for each participant per session:

##### Environment settings

This file specified the size of the virtual room, the fly’s trajectory
(buzzing speed, minimum and maximum heights of flight, and a buzzing radius),
and whether to remove or retain the bounding walls during the second block.
Additionally, the links to survey forms were added here.

##### Locomotion settings

This log file specified the locomotion method used and its presets like
traversal gain and rotation speed.

##### Scenario settings

This log file specified how many trials were to be presented in first and
second blocks, the longest possible duration of a trial (with the maximum of
120 s), and start and goal locations.

##### Participant particulars

The participant information as collected via the GUI (i.e., identification,
age, gender, and highest qualification achieved) was recorded.

##### Movement logs

This log file recorded each participant’s X and Z positions and rotation about
the Y axis with time stamps. Owing to flexibility provided by DeFINE to toggle
lights and sounds even during a trial, the status of these parameters was also
logged every frame in these logs. A new log was created per trial along with
trial numbers.

##### Trial results

This log file assimilated component-wise and cumulative rewards per
participant along with distance covered and time elapsed during a trial.

##### Notes

The experimentalists’ notes during the experiment were recorded in this file.
For instance, if some participants felt dizzy and opted for early termination,
the particular session can be marked as bad and further details can be stored
as notes for later use.

####  2.1.3 Procedure

The participants sat in the middle of a room that was clear of any obstacles.
At the outset of the experiment, the participants were asked to fill the SSQ
to log their state of health before being immersed in VR. Their age, gender,
and the highest qualification achieved were also recorded using DeFINE’s GUI
(Figure 3(a)). An experimentalist then put an HMD on the participants’ head
(over the spectacles, as and when necessary) and handed them hand-held VR
controllers. The participants were run individually.

As soon as the participants had verbally confirmed to be ready, the first
block was started by the experimentalist. The participants began a trial by
leaving the start position, using the controller teleoperation method to
navigate, and ended their locomotion by pressing a key on a VR controller when
they thought they had reached the goal position. The goal was positioned
diagonally across the other side of the room and remained unchanged across
trials. The participants then received a score from the trial in a leader-
board (Figure 6). The fake leader-board feature was used so that all
participants performed the navigation task with the same competitive
challenge. The board was filled with made-up scores at the beginning, which
were set low to give everyone a reasonable chance of making it to the top ten.
Pilot testing was conducted to empirically determine how low should be low
enough for this purpose. The leader-board was displayed for 10 s (or until the
participants pressed the “End Trial” key on the VR controller), and the room
was automatically shown from the start position again for the next trial
thereby resetting the scene to the exact same configuration for each trial.
The participants completed the trials at their own pace, until reaching the
end of the block. The participants were allowed to have a short break between
trials. When necessary, the participants were able to skip a trial by pressing
a controller key. At the end of the first block, the participants filled the
NASA TLX in the DeFINE’s form system (i.e., without taking off the HMD) using
a 7-point Likert scale. Upon having filled the form, the participants started
the second block at their own input, prompted on the HMD. Once again the
participants performed the trials at their own pace, until filling the NASA
TLX one more time at the end of the second block. Filling the form completed
the VR part of the experiment.

After taking off the HMD and the controllers, the participants filled the SSQ
again to evaluate their simulation sickness after the exposure to the
immersive VR. In addition, the participants were invited to provide feedback
about the experiment and DeFINE by indicating the degree of agreement with
each of the following five statements in a 5-point Likert scale: “Instructions
were easy to understand”; “I understood what the score depended on”; “moving
in the VE was easy”; “the walls in the practice phase were helpful”; and
“filling a form in the VE was easy”.

###  2.2 Results

Two participants from each group misunderstood task instructions and simply
chased the fly rather than navigating towards the goal it indicated. This was
determined through real-time observation of their locomotion patterns on the
experimentalist’s desktop (Figure 8). Although it was not intended, this gave
another demonstration of the utility of the experimentalist view that allowed
the experimentalist to observe participants’ responses as they took place
during trials. Due to this behavior, data from the four participants were
excluded from analysis. The data presented in this section represent the
results of the remaining $10$ participants per group, accounting for $20$
participants in total. In addition, in $0.7\%$ of all trials, the participants
accidentally pressed the button to end the trial immediately after it had
begun. These trials were also discarded for the analysis presented herewith.

For each trial, the total elapsed time, the residual distance to the goal, and
the score were derived as dependent measures from log files. The entire
trajectory of a participant’s locomotion was also reconstructed from recorded
position data. For each dependent measure, data points that were more than
three standard deviations away from each participant’s mean of each block were
defined as outliers and removed from analysis. This resulted in removal of
1.17% of trials on average.

Table 2: Means and standard deviations of the total elapsed time, the residual distance to the goal, and the score as a function of participant group and block. | Time group |  | Accuracy group
---|---|---|---
| First block | Second block |  | First block | Second block
Time (s) | 11.13 (5.93) | 7.81 (4.44) |  | 10.97 (5.25) | 13.33 (10.77)
Distance (m) | 0.43 (0.18) | 0.67 (0.32) |  | 0.63 (0.34) | 0.80 (0.36)
Score | 664.97 (239.02) | 718.87 (255.85) |  | 624.77 (141.36) | 542.41 (160.26)
Note. Standard deviations are shown in parentheses.

####  2.2.1 Time, distance, and score

Table 2 shows descriptive statistics of the dependent measures as a function
of participant groups and blocks. Overall, participants in the time group
performed trials more quickly in the second block than in the first block, and
those in the accuracy group showed the opposite pattern. It is likely that
this resulted from the types of feedback the participants received in each
group—that is, the swiftness of a response was more heavily rewarded in the
time group, whereas the closeness to the goal was given more emphasis in the
accuracy group. In terms of the residual distance to the goal, both groups
performed worse in the second block, reflecting the fact that the navigation
task was more challenging in the second block because of the lack of walls and
decreased precision of the dynamic goal marker. Capturing these outcomes, the
scores increased in the time group and decreased in the accuracy group between
the first and second blocks.

Figure 9: Mean times participants spent for performing each trial (top row),
mean residual distances to the goal (middle row), and mean feedback scores the
participants received (bottom row) as a function of participant group, block,
and trial number. Error bars represent $\pm 1$ standard error of the mean.

Figure 9 shows the elapsed time, residual distance, and score in each trial,
providing a more detailed picture of the participants’ performance. In terms
of speed, the two groups performed similarly in the first block, but they
differed in the second block. Specifically, the time group maintained
approximately the same speed throughout the block, performing the trials
consistently quicker than the accuracy group. This pattern suggests that the
feedback scores affected the participants’ navigation differently in the two
groups. On the other hand, with the reward parameters used in this experiment
(Table 1), the effects of the scores were less clear on the accuracy of
performance. The participants in the accuracy group showed no visible
improvement of accuracy in later trials, even though they received scores that
rewarded accurate performance. This might indicate that the parameters did not
favor accuracy enough to elicit observable change in behavior within a block.
Between the blocks, it is also possible that the increase of task difficulty
overrode the effects of the scores on navigation performance, as suggested by
the overall lower scores in the second block.

To statistically examine the time, distance, and score data, they were
analyzed separately by mixed analyses of variance (ANOVAs) in which block
(first and second) was a within-participant factor and group (time and
accuracy) was a between-participant factor. Because the data plotted in Figure
9 suggest that the first trial of the first block yielded rather different
results (particularly in time and score), the ANOVAs were run with and without
this trial. The two sets of ANOVAs showed the same outcomes, and thus those
including the first trial are reported here. The main effect of block on
distance was significant, $F(1,18)=10.52,p=0.005,{\eta_{G}}^{2}=0.11$,
reflecting the overall decrease of accuracy in the second block. All the other
main effects and interactions were not significant,
$F\text{s}(1,18)<3.09,p\text{s}>0.096,{\eta_{G}}^{2}\text{s}<0.075$.

####  2.2.2 Trajectory

Figure 10: The residual distance to the goal as a function of participant
group and time in a trial (first 30 s only). Panels A and B show sample
participants’ 30 trials (black lines), one participant from each group. The
red lines represent the mean time courses of the residual distance derived
from the 30 trials. To calculate the means, trials that lasted shorter than 30
s were extended by using the end values of each such trial. The insets display
the corresponding trajectories of each participant from the start (cyan
circle) to the goal (red cross). Panels C and D show the mean time courses of
each participant (black lines) separately for the two groups. The red lines
indicate group means. In calculating them, short time series were dealt with
in the same way as in Panels A and B.

In addition to the time, distance, and score measures that aggregate
participants’ performance in a trial, DeFINE can provide dynamic information
about their movement within the trial. To illustrate this feature, the time
courses of the residual distance to the goal as well as sample participants’
trajectories are plotted in Figure 10. By visualizing the trajectories in this
manner, researchers can gain additional insights into the way participants
navigate in their experiments. For example, by comparing panels A and B of
Figure 10, it can be seen that the trajectories of the sample participant from
the time group were more dispersed than those of the sample participant from
the accuracy group. Such an observation is easily possible in this
visualization, but it is not readily available from the aggregate measures.

####  2.2.3 Simulation sickness questionnaire (SSQ)

Responses to the SSQ are summarized in Table 3. As shown in the table, the
participants scored very low not only before but also after exposure to the
VE. Because the scores were very low overall, we used total raw scores for
analysis, instead of deriving weighted scores for each sub-scale of the SSQ
(Kennedy ., 1993). The total raw SSQ scores were analyzed by a mixed ANOVA
with exposure (before and after) as a within-participant factor and group
(time and accuracy) as a between-participant factor. This ANOVA yielded no
significant outcomes,
$F\text{s}(1,18)<0.22,p\text{s}>0.64,{{\eta_{G}}^{2}}\text{s}<0.010$,
suggesting that the SSQ scores did not differ between pre- and post-exposure
to the VE as well as between the time and accuracy groups. These results
indicate that the use of DeFINE did not induce any major symptoms of
cybersickness.

Because non-significant results in an ANOVA do not necessarily constitute
positive evidence for null hypotheses, we also conducted a Bayes factor
analysis to gauge the extent to which the data actually supported the claim
that neither exposure nor group had an effect on the SSQ scores (Rouder .,
2012). When the null model was compared against the full model that included
the main effects of exposure and group as well as the interaction between the
two, it yielded a Bayes factor of 13.9. This constitutes positive evidence for
the null hypothesis (Kass Raftery, 1995), supporting the conclusion that doing
the navigation task in DeFINE (and which group each participant was in) did
not cause cybersickness above and beyond what the participants had prior to
navigating in the VE.

Table 3: Means and standard deviations of raw scores of the simulator sickness
questionnaire (SSQ).

| Pre-exposure |  | Post-exposure
---|---|---|---
| Time group | Accuracy group |  | Time group | Accuracy group
General discomfort | 0.1 (0.32) | 0.2 (0.63) |  | 0.2 (0.42) | 0.3 (0.48)
Fatigue | 0.4 (0.52) | 0.7 (0.95) |  | 0.2 (0.42) | 0.6 (0.70)
Boredom | 0.3 (0.67) | 0.1 (0.32) |  | 0.3 (0.67) | 0.1 (0.32)
Drowsiness | 0.5 (0.71) | 0.2 (0.42) |  | 0 (0) | 0.2 (0.63)
Headache | 0.1 (0.32) | 0 (0) |  | 0.1 (0.32) | 0.2 (0.42)
Eyestrain | 0.5 (0.53) | 0.5 (0.97) |  | 0.7 (0.67) | 0.5 (0.97)
Difficulty focusing | 0.2 (0.42) | 0.4 (0.70) |  | 0.1 (0.32) | 0.3 (0.48)
Salivation increase/decrease | 0.1 (0.32) | 0.2 (0.42) |  | 0.1 (0.32) | 0 (0)
Sweating | 0 (0) | 0.1 (0.32) |  | 0.2 (0.42) | 0.2 (0.63)
Nausea | 0 (0) | 0 (0) |  | 0.2 (0.42) | 0.1 (0.32)
Difficulty concentrating | 0.3 (0.48) | 0.3 (0.67) |  | 0.1 (0.32) | 0.3 (0.48)
Mental depression | 0.2 (0.42) | 0.2 (0.42) |  | 0.2 (0.42) | 0.2 (0.42)
Fullness of the head | 0.3 (0.48) | 0.4 (0.52) |  | 0.3 (0.48) | 0.2 (0.42)
Blurred vision | 0.1 (0.32) | 0.3 (0.48) |  | 0.2 (0.42) | 0.4 (0.52)
Dizziness with eyes open/closed | 0.1 (0.32) | 0.2 (0.63) |  | 0.2 (0.42) | 0.3 (0.48)
Vertigo | 0 (0) | 0 (0) |  | 0.1 (0.32) | 0.1 (0.32)
Visual flashbacks | 0 (0) | 0 (0) |  | 0.3 (0.67) | 0.1 (0.33)
Faintness | 0.1 (0.32) | 0 (0) |  | 0 (0) | 0.1 (0.32)
Breathing awareness | 0.4 (0.52) | 0 (0) |  | 0.3 (0.48) | 0.1 (0.32)
Stomach awareness | 0 (0) | 0.1 (0.32) |  | 0.1 (0.32) | 0.1 (0.32)
Loss of appetite | 0 (0) | 0 (0) |  | 0 (0) | 0.1 (0.32)
Increase of appetite | 0.1 (0.32) | 0.3 (0.48) |  | 0.1 (0.32) | 0.4 (0.52)
Desire to move bowels | 0.1 (0.32) | 0 (0) |  | 0.1 (0.32) | 0 (0)
Confusion | 0 (0) | 0.4 (0.70) |  | 0.1 (0.32) | 0.1 (0.32)
Burping | 0 (0) | 0 (0) |  | 0 (0) | 0 (0)
Vomiting | 0 (0) | 0 (0) |  | 0 (0) | 0.1 (0.32)
Others | 0 (0) | 0 (0) |  | 0 (0) | 0 (0)
Total | 3.9 (3.51) | 4.6 (5.66) |  | 4.2 (3.74) | 5.1 (4.12)
Note. Standard deviations are shown in parentheses. The possible range of the
total score was from 0 to 81.

Table 4: Means and standard deviations of scores of the NASA task load index (NASA TLX). | Time group |  | Accuracy group
---|---|---|---
| First block | Second block |  | First block | Second block
Mental demand | 3.0 (1.33) | 3.7 (1.64) |  | 2.3 (1.25) | 3.0 (1.41)
Physical demand | 1.7 (0.82) | 2.2 (1.14) |  | 1.7 (1.25) | 2.7 (1.77)
Temporal demand | 3.5 (1.35) | 4.1 (1.79) |  | 2.9 (1.60) | 2.9 (1.85)
Effort | 3.4 (1.58) | 3.6 (1.71) |  | 2.8 (1.14) | 3.2 (1.32)
Performance | 3.5 (1.43) | 2.6 (1.35) |  | 3.9 (1.60) | 4.0 (1.70)
Frustration level | 2.6 (1.71) | 3.2 (1.62) |  | 1.9 (1.20) | 2.5 (1.43)
Total | 17.7 (6.38) | 19.4 (7.29) |  | 15.5 (5.15) | 18.3 (7.89)
Note. Standard deviations are shown in parentheses. The possible range of the
total score was from 6 to 42.

####  2.2.4 NASA task load index (NASA TLX)

Responses to each item of the NASA TLX ranged from one to seven, with smaller
scores indicating lower task load. As shown in Table 4, participants generally
indicated that doing the navigation task in DeFINE required medium workload.
There was some variation of the scores between groups, blocks, and questions.
For example, the scores of the temporal demand question suggest that the time
group felt stronger time pressure than the accuracy group, which is consistent
with the feedback function that put emphasis on speedy response in the time
group. In addition, scores in the second block tended to be higher than those
in the first block, which corresponds to the fact that the task was made more
difficult in the second block. In line with these observations, a mixed ANOVA
with block (first and second) and question (six questions of the NASA TLX) as
within-participant factors and group (time and accuracy) as a between-
participant factor yielded a significant interaction between question and
group, $F(5,90)=2.96,p=0.035,{\eta_{G}}^{2}=0.049,\epsilon=0.66$ (this ANOVA
was corrected for non-sphericity with the Greenhouse-Geisser method when
appropriate). The interaction between question and block as well as the main
effect of question were also significant,
$F(5,90)=3.31,p=0.008,{\eta_{G}}^{2}=0.019$ and
$F(5,90)=7.14,p<0.001,{\eta_{G}}^{2}=0.11,\epsilon=0.66$, respectively. The
main effect of block was not significant,
$F(1,18)=3.58,p=0.075,{\eta_{G}}^{2}=0.018$. The interaction between block and
group and the main effect of group were virtually non-existent,
$F\text{s}(1,18)<0.36,p\text{s}>0.55,{{\eta_{G}}^{2}}\text{s}<0.010$,
suggesting that overall, the two groups tolerated the workload of using DeFINE
in a similar way.

####  2.2.5 Participant feedback on the experiment and DeFINE

Scores of the participant feedback survey at the end of the experiment are
summarized in Table 5. Larger scores denote stronger agreement with the
statements. Overall, participants gave high scores, indicating that DeFINE
provided an easy-to-use interface for doing the navigation experiment. The
scores were analyzed by a mixed ANOVA in which statement (five statements in
the survey) was a within-participant factor and group (time and accuracy) was
a between-participant factor. The main effect of statement was significant,
$F(4,72)=6.51,p<0.001,{\eta_{G}}^{2}=0.20$, which suggests that scores were
reliably lower in the statement about the usefulness of walls than in the
other statements. The interaction between statement and group and the main
effect of group were not significant,
$F(4,72)=1.73,p=0.15,{\eta_{G}}^{2}=0.062$ and
$F(1,18)=0.18,p=0.67,{\eta_{G}}^{2}=0.003$, respectively, suggesting that
there was no overall difference between the groups in the way they responded
to the feedback survey.

Table 5: Means and standard deviations of scores of the participant feedback survey. | Time group | Accuracy group
---|---|---
Clarity of instructions | 4.2 (0.63) | 4.1 (0.99)
Score interpretation | 4.5 (0.71) | 4.0 (1.05)
Ease of movement | 4.3 (0.67) | 4.2 (0.79)
Usefulness of walls | 2.9 (1.37) | 3.3 (0.82)
Ease of filling forms in DeFINE | 3.4 (0.97) | 4.2 (1.03)
Total | 19.3 (2.63) | 19.8 (2.62)
Note. Standard deviations are shown in parentheses. The possible range of the
total score was from 5 to 25.

###  2.3 Discussion

The purpose of this demonstration was to showcase the key functionalities of
DeFINE—namely, its ability to set up an experiment in a trial–block–session
structure, run goal-directed navigation trials in a stimulus–response–feedback
loop, and collect both moment-by-moment and aggregate measures of
participants’ task performance. In this experiment, the measures included the
duration of a trial, the residual distance to the goal, the feedback score
given to the participants, and the time course of the participants’ positions
within the trial. It may be worth noting that among these measures, those that
capture temporal information about the participants’ locomotion (i.e., the
trial duration and dynamic position data) can be of particular use in future
studies because past studies of goal-directed navigation in small-scale space
tended to put emphasis on accuracy or precision of responses with little
regard for how quickly they were carried out (e.g., Chen ., 2017; Chrastil
Warren, 2014; Harris Wolbers, 2012; Yamamoto, Meléndez Menzies, 2014;
Yamamoto, Philbeck ., 2014). However, it is important to consider the speed of
the responses in evaluating their accuracy because there can be a trade-off
relationship between them (Bogacz ., 2010). DeFINE allows researchers to
examine the speed and accuracy of navigation either in conjunction as in the
current experiment or in isolation by setting the parameters of the reward
function accordingly (e.g., $\beta_{1}=0$ makes the reward function
exclusively focused on the accuracy).

A defining feature of DeFINE is that with its default settings, it provides
feedback on participants’ performance in each trial. Results from this
experiment suggested that by using differential weights on speed and accuracy
of navigation in calculating feedback scores, DeFINE has potential for
eliciting different responses from the participants. Although the behavioral
measures of navigation were largely non-significant in the statistical
analyses, the effects of the feedback scores, particularly those on navigation
speed, were implied in Figure 9. The time group improved the speed of
responses in early trials and kept the same speed during the rest of the
experiment because this helped increase and then maintain the feedback scores.
On the other hand, the accuracy group appeared to care less for making a
speedy response toward the end of the experiment, as the slower speed had
little influence on the feedback scores in this group. It is likely that the
feedback scores were more effective in affecting the speed because of the
specific way in which the current experiment was designed—that is, the
participants were self-aware of the speed of their response, but the accuracy
was never explicitly revealed to them, making it harder for the participants
to improve the accuracy. Importantly, this pattern is a result of one
particular installation of DeFINE, and its architecture flexibly enables
researchers to set up a suitable balance between the speed and accuracy
according to the objectives of their studies. For example, by giving heavier
weights to accuracy in the feedback function, researchers can make feedback
scores more directly informative about how well participants are reaching a
goal. Similarly, by demarcating the goal location more specifically by using
different goal markers (e.g., a static marker or a dynamic marker with less
variability) and environmental features (e.g., walls that provide spatial
cues), researchers can run experiments in which focus is entirely on speed
(i.e., accuracy is a given) or subtle changes in accuracy are scrutinized.

This experiment also examined the participants’ experience in using DeFINE.
Results from the SSQ indicated that DeFINE caused no major symptoms of
cybersickness. Considering that the participants repeatedly experienced
sensory conflicts between their vision and body-based (i.e., vestibular and
proprioceptive) senses due to the use of the teleoperation method, the absence
of cybersickness is notable (Bos ., 2008). The NASA TLX showed that the
participants found doing the navigation task in DeFINE moderately challenging
but not unreasonably taxing. In the feedback survey, the participants gave a
positive evaluation to DeFINE itself and the design of the experiment.
Generally, these results did not differ between the two groups of the
participants, suggesting that DeFINE provided a versatile platform that
accommodates different types of experiments.

In sum, this demonstration showed that by using DeFINE in its default
settings, we were able to run a human navigation experiment of typical scale,
involving 24 participants and consisting of a single continuous session with
two blocks of 15 trials each. It exemplified an assortment of data types that
DeFINE can collect, which allow for detailed characterization of navigational
behavior. Both objective and subjective measures of the participants’
experience indicated that they found DeFINE easy to use and the navigation
task it implemented well tolerable in terms of cybersickness and task
workload, irrespective of the ways in which they carried out the navigation
task (i.e., whether they were implicitly driven to perform it more quickly or
accurately via feedback scores). Together, these results validated DeFINE’s
capability and potential as a tool for investigating goal-directed navigation
in humans under a variety of conditions.

##  3 Conclusions

This paper presented the Delayed Feedback based Immersive Navigation
Environment (DeFINE) for studying goal-directed navigation in humans using VR.
Although similar frameworks have already been developed (Brookes ., 2020;
Vasser ., 2017; Commins ., 2020; Machado ., 2019; Wiener ., 2020; Geller .,
2007; Ayaz ., 2008, 2011; Solway ., 2013; Grübel ., 2017; Bebko Troje, 2020;
Starrett ., 2020), they are based on an open-loop stimulus–response
architecture that omits performance feedback to participants. DeFINE
distinguishes itself from the previous frameworks by implementing the closed-
loop stimulus–response–feedback architecture as its core element (Figures 1
and 2). The feedback is delayed by default in order to suit the needs of
typical navigation experiments, but if necessary, it can be made real-time
through relatively simple changes in DeFINE’s code so that the
stimulus–response–feedback loop is even more tightly closed.

As discussed in the introduction, the VR frameworks for navigation research
mainly differ in whether they are geared toward ease of use by limiting their
scope or wide applicability by providing general-purpose tools that demand
technical skills of end-users. In this spectrum, DeFINE aims to position
itself toward the ease-of-use end by focusing primarily on goal-directed
navigation tasks and also by making it possible to set up an experiment mostly
through GUIs and simple settings files (demonstrated in the video clips
available online). However, this does not mean that coding is absolutely
unnecessary or impossible in DeFINE. Indeed, some customization that goes
beyond the GUIs and settings files is expected, and for this purpose the
software is made open-source and its codebase is modularized.

The demonstration experiment showed the utility of DeFINE as a platform for
navigation research and its general friendliness to participants in VR
experiments. Additionally, this experiment demonstrated DeFINE’s potential as
a tool for testing hypotheses about the temporal aspects of navigational
behavior. The optional feature of seamlessly administering surveys within an
HMD enhances the immersion of the participants in VR, thereby improving the
quality of data collected via DeFINE. The optional leader-board enables
investigation of the effect of gamification on spatial navigation. Previous
studies have shown its impact in other domains of learning (Barata ., 2013;
Looyestyn ., 2017), but it is yet to be thoroughly explored for navigation-
related applications (Coutrot ., 2018; Coughlan ., 2019). These out-of-the-box
features of DeFINE, together with its customizability via the Unity software,
open up many new possibilities for human navigation research.

##  4 Open Practices Statements

The software used in the experiment reported in this article—the Delayed
Feedback based Immersive Navigation Environment (DeFINE)—is available at
https://github.com/ktiwari9/define-VR. The data and other materials for the
experiment are available upon request. The experiment was not preregistered.

## References

  * Ayaz . (2008) ayaz2008Ayaz, H., Allen, SL., Platek, SM. Onaral, B.  2008\.  Maze Suite 1.0: A complete set of tools to prepare, present, and analyze navigational and spatial cognitive neuroscience experiments Maze Suite 1.0: A complete set of tools to prepare, present, and analyze navigational and spatial cognitive neuroscience experiments.  Behavior Research Methods40353–359.  10.3758/BRM.40.1.353
  * Ayaz . (2011) ayaz2011Ayaz, H., Shewokis, PA., Curtin, A., Izzetoglu, M., Izzetoglu, K. Onaral, B.  2011\.  Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation.  Journal of Visualized Experiments56e3443.  10.3791/3443
  * Barata . (2013) barata2013improvingBarata, G., Gama, S., Jorge, J. Gonçalves, D.  2013\.  Improving participation and learning with gamification Improving participation and learning with gamification.  LE. Nacke, K. Harrigan N. Randall (), Gamification 2013: Proceedings of the First International Conference on Gameful Design, Research, and Applications Gamification 2013: Proceedings of the first International Conference on Gameful Design, Research, and Applications ( 10–17).  New York, NYAssociation for Computing Machinery.  10.1145/2583008.2583010
  * Bebko Troje (2020) bebko2020Bebko, AO. Troje, NF.  2020\.  bmlTUX: Design and control of experiments in virtual reality and beyond bmlTUX: Design and control of experiments in virtual reality and beyond.  i-Perception1141–12.  10.1177/2041669520938400
  * Bird . (2010) bird_establishing_2010Bird, CM., Capponi, C., King, JA., Doeller, CF. Burgess, N.  2010\.  Establishing the Boundaries: The Hippocampal Contribution to Imagining Scenes Establishing the boundaries: The hippocampal contribution to imagining scenes.  Journal of Neuroscience3011688–11695.  10.1523/JNEUROSCI.0723-10.2010
  * Bogacz . (2010) bogacz2010humansBogacz, R., Hu, PT., Holmes, PJ. Cohen, JD.  2010\.  Do humans produce the speed–accuracy trade-off that maximizes reward rate? Do humans produce the speed–accuracy trade-off that maximizes reward rate?  The Quarterly Journal of Experimental Psychology63863–891.  10.1080/17470210903091643
  * Bos . (2008) bos2008Bos, JE., Bles, W. Groen, EL.  2008\.  A theory on visually induced motion sickness A theory on visually induced motion sickness.  Displays2947–57.  10.1016/j.displa.2007.09.002
  * Bowman . (2002) bowman2002surveyBowman, DA., Gabbard, JL. Hix, D.  2002\.  A survey of usability evaluation in virtual environments: Classification and comparison of methods A survey of usability evaluation in virtual environments: Classification and comparison of methods.  Presence: Teleoperators & Virtual Environments11404–424.  10.1162/105474602760204309
  * Brand (2008) brand2008doesBrand, M.  2008\.  Does the feedback from previous trials influence current decisions? A study on the role of feedback processing in making decisions under explicit risk conditions Does the feedback from previous trials influence current decisions? A study on the role of feedback processing in making decisions under explicit risk conditions.  Journal of Neuropsychology2431–443.  10.1348/174866407x220607
  * Brookes . (2020) Brookes2019Brookes, J., Warburton, M., Alghadier, M., Mon-Williams, M. Mushtaq, F.  2020\.  Studying human behavior with virtual reality: The Unity Experiment Framework Studying human behavior with virtual reality: The Unity Experiment Framework.  Behavior Research Methods52455–463.  10.3758/s13428-019-01242-0
  * Carton . (2016) carton2016towardsCarton, D., Nitsch, V., Meinzer, D. Wollherr, D.  2016\.  Towards assessing the human trajectory planning horizon Towards assessing the human trajectory planning horizon.  PLoS ONE11e0167021.  10.1371/journal.pone.0167021
  * Chen . (2017) chen_cue_2017Chen, X., McNamara, TP., Kelly, JW. Wolbers, T.  2017\.  Cue combination in human spatial navigation Cue combination in human spatial navigation.  Cognitive Psychology95105–144.  10.1016/j.cogpsych.2017.04.003
  * Cherep . (2020) cherep_spatial_2020Cherep, LA., Lim, AF., Kelly, JW., Acharya, D., Velasco, A., Bustamante, E.Gilbert, SB.  2020\.  Spatial cognitive implications of teleporting through virtual environments Spatial cognitive implications of teleporting through virtual environments.  Journal of Experimental Psychology: Applied26480–492.  10.1037/xap0000263
  * Cheung (2014) cheung_estimating_2014Cheung, A.  2014\.  Estimating Location without External Cues Estimating location without external cues.  PLOS Computational Biology10e1003927.  10.1371/journal.pcbi.1003927
  * Chrastil Warren (2014) chrastil_does_2014Chrastil, ER. Warren, WH.  2014\.  Does the human odometer use an extrinsic or intrinsic metric? Does the human odometer use an extrinsic or intrinsic metric?  Attention, Perception, & Psychophysics76230–246.  10.3758/s13414-013-0549-3
  * Commins . (2020) commins_2019Commins, S., Duffin, J., Chaves, K., Leahy, D., Corcoran, K., Caffrey, M.Thornberry, C.  2020\.  NavWell: A simplified virtual-reality platform for spatial navigation and memory experiments Navwell: A simplified virtual-reality platform for spatial navigation and memory experiments.  Behavior Research Methods521189–1207.  10.3758/s13428-019-01310-5
  * Cornwell . (2008) cornwell2008humanCornwell, BR., Johnson, LL., Holroyd, T., Carver, FW. Grillon, C.  2008\.  Human hippocampal and parahippocampal theta during goal-directed spatial navigation predicts performance on a virtual Morris water maze Human hippocampal and parahippocampal theta during goal-directed spatial navigation predicts performance on a virtual morris water maze.  Journal of Neuroscience285983–5990.  10.1523/JNEUROSCI.5001-07.2008
  * Coughlan . (2019) coughlan_toward_2019Coughlan, G., Coutrot, A., Khondoker, M., Minihane, AM., Spiers, H. Hornberger, M.  2019\.  Toward personalized cognitive diagnostics of at-genetic-risk Alzheimer’s disease Toward personalized cognitive diagnostics of at-genetic-risk Alzheimer’s disease.  Proceedings of the National Academy of Sciences of the United States of America1169285–9292.  10.1073/pnas.1901600116
  * Coutrot . (2018) coutrot_global_2018Coutrot, A., Silva, R., Manley, E., de Cothi, W., Sami, S., Bohbot, VD.Spiers, HJ.  2018\.  Global Determinants of Navigation Ability Global determinants of navigation ability.  Current Biology282861–2866.  10.1016/j.cub.2018.06.009
  * Du . (2020) du_unidirectional_2018Du, Y., Mou, W. Zhang, L.  2020\.  Unidirectional influence of vision on locomotion in multimodal spatial representations acquired from navigation Unidirectional influence of vision on locomotion in multimodal spatial representations acquired from navigation.  Psychological Research841284–1303.  10.1007/s00426-018-1131-3
  * Geller . (2007) geller2007Geller, AS., Schleifer, IK., Sederberg, PB., Jacobs, J. Kahana, MJ.  2007\.  PyEPL: A cross-platform experiment-programming library PyEPL: A cross-platform experiment-programming library.  Behavior Research Methods39950–958.  10.3758/BF03192990
  * Grübel . (2017) grubel2017Grübel, J., Weibel, R., Jiang, MH., Hölscher, C., Hackman, DA. Schinazi, VR.  2017\.  EVE: A framework for experiments in virtual environments EVE: A framework for experiments in virtual environments.  T. Barkowsky, H. Burte, C. Hölscher H. Schultheis (), Lecture Notes in Computer Science: Vol. 10523. Spatial Cognition X Lecture Notes in Computer Science: Vol. 10523\. Spatial Cognition X ( 159–176).  Cham, SwitzerlandSpringer International Publishing.  10.1007/978-3-319-68189-4_10
  * Harris Wolbers (2012) harris_ageing_2012Harris, MA. Wolbers, T.  2012\.  Ageing effects on path integration and landmark navigation Ageing effects on path integration and landmark navigation.  Hippocampus221770–1780.  10.1002/hipo.22011
  * Hart (2006) hart2006nasaHart, SG.  2006\.  NASA-task load index (NASA-TLX); 20 years later NASA-task load index (NASA-TLX); 20 years later.  Proceedings of the Human Factors and Ergonomics Society Annual Meeting50904–908.  10.1177/154193120605000909
  * Hossain List (2012) hossain_behavioralist_2012Hossain, T. List, JA.  2012\.  The Behavioralist Visits the Factory: Increasing Productivity Using Simple Framing Manipulations The behavioralist visits the factory: Increasing productivity using simple framing manipulations.  Management Science582151–2167.  10.1287/mnsc.1120.1544
  * Kass Raftery (1995) kass_bayes_1995Kass, RE. Raftery, AE.  1995\.  Bayes factors Bayes factors.  Journal of the American Statistical Association90773–795.  10.1080/01621459.1995.10476572
  * Kelly . (2020) kelly_teleporting_2020Kelly, JW., Ostrander, AG., Lim, AF., Cherep, LA. Gilbert, SB.  2020\.  Teleporting through virtual environments: Effects of path scale and environment scale on spatial updating Teleporting through virtual environments: Effects of path scale and environment scale on spatial updating.  IEEE Transactions on Visualization and Computer Graphics261841–1850.  10.1109/TVCG.2020.2973051
  * Kennedy . (1993) kennedy_simulator_1993Kennedy, RS., Lane, NE., Berbaum, KS. Lilienthal, MG.  1993\.  Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness.  The International Journal of Aviation Psychology3203–220.  10.1207/s15327108ijap0303_3
  * Kunz . (2009) kunz_evidence_2009Kunz, BR., Creem-Regehr, SH. Thompson, WB.  2009\.  Evidence for motor simulation in imagined locomotion Evidence for motor simulation in imagined locomotion.  Journal of Experimental Psychology: Human Perception and Performance351458–1471.  10.1037/a0015786
  * Loomis Knapp (2003) loomis_visual_2003Loomis, JM. Knapp, J.  2003\.  Visual perception of egocentric distance in real and virtual environments Visual perception of egocentric distance in real and virtual environments.  LJ. Hettinger MW. Haas (), Virtual and adaptive environments: Applications, implications, and human performance issues Virtual and adaptive environments: Applications, implications, and human performance issues ( 21–46).  Mahwah, NJLawrence Erlbaum Associates.
  * Loomis Philbeck (2008) loomis_measuring_2008Loomis, JM. Philbeck, JW.  2008\.  Measuring spatial perception with spatial updating and action Measuring spatial perception with spatial updating and action.  RL. Klatzky, B. MacWhinney M. Behrmann (), Embodiment, ego-space, and action Embodiment, ego-space, and action ( 1–43).  New York, NYPsychology Press.
  * Looyestyn . (2017) looyestyn2017doesLooyestyn, J., Kernot, J., Boshoff, K., Ryan, J., Edney, S. Maher, C.  2017\.  Does gamification increase engagement with online programs? A systematic review Does gamification increase engagement with online programs? A systematic review.  PLoS ONE12e0173403.  10.1371/journal.pone.0173403
  * Machado . (2019) machado2019newMachado, M., Lefèvre, N., Philoxene, B., Le Gall, A., Madeleine, S., Fleury, P.Besnard, S.  2019\.  New Software dedicated to Virtual Mazes for Human NeuroCognitive investigations New software dedicated to virtual mazes for human neurocognitive investigations.  Journal of Neuroscience Methods327108388.  10.1016/j.jneumeth.2019.108388
  * Mou Zhou (2013) mou_defining_2013Mou, W. Zhou, R.  2013\.  Defining a boundary in goal localization: Infinite number of points or extended surfaces Defining a boundary in goal localization: Infinite number of points or extended surfaces.  Journal of Experimental Psychology: Learning, Memory, and Cognition391115–1127.  10.1037/a0030535
  * Nilsson . (2018) nilsson201815Nilsson, NC., Peck, T., Bruder, G., Hodgson, E., Serafin, S., Whitton, M.Rosenberg, ES.  2018\.  15 years of research on redirected walking in immersive virtual environments 15 years of research on redirected walking in immersive virtual environments.  IEEE Computer Graphics and Applications38244–56.  10.1109/MCG.2018.111125628
  * Paludan . (2016) paludan2016disguisingPaludan, A., Elbaek, J., Mortensen, M., Zobbe, M., Nilsson, NC., Nordahl, R.Serafin, S.  2016\.  Disguising rotational gain for redirected walking in virtual reality: Effect of visual density Disguising rotational gain for redirected walking in virtual reality: Effect of visual density.  T. Höllerer, V. Interrante, A. Lécuyer E. Suma (), 2016 IEEE Virtual Reality (VR) 2016 IEEE virtual reality (VR) ( 259–260).  Piscataway, NJInstitute of Electrical and Electronics Engineers.  10.1109/VR.2016.7504752
  * Pezzulo . (2014) pezzulo2014internallyPezzulo, G., van der Meer, MA., Lansink, CS. Pennartz, CM.  2014\.  Internally generated sequences in learning and executing goal-directed behavior Internally generated sequences in learning and executing goal-directed behavior.  Trends in Cognitive Sciences18647–657.  10.1016/j.tics.2014.06.011
  * Philbeck Loomis (1997) philbeck_comparison_1997Philbeck, JW. Loomis, JM.  1997\.  Comparison of two indicators of perceived egocentric distance under full-cue and reduced-cue conditions Comparison of two indicators of perceived egocentric distance under full-cue and reduced-cue conditions.  Journal of Experimental Psychology: Human Perception and Performance2372–85.  10.1037/0096-1523.23.1.72
  * Regal . (2018) regal2018vrateRegal, G., Schatz, R., Schrammel, J. Suette, S.  2018\.  VRate: A Unity3D asset for integrating subjective assessment questionnaires in virtual environments VRate: A Unity3D asset for integrating subjective assessment questionnaires in virtual environments.  2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX)1–3.  10.1109/QoMEX.2018.8463296
  * Rouder . (2012) rouder_default_2012Rouder, JN., Morey, RD., Speckman, PL. Province, JM.  2012\.  Default Bayes factors for ANOVA designs Default Bayes factors for ANOVA designs.  Journal of Mathematical Psychology56356–374.  10.1016/j.jmp.2012.08.001
  * Schatz . (2017) schatz2017towardsSchatz, R., Sackl, A., Timmerer, C. Gardlo, B.  2017\.  Towards subjective quality of experience assessment for omnidirectional video streaming Towards subjective quality of experience assessment for omnidirectional video streaming.  2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX)1–6.  10.1109/QoMEX.2017.7965657
  * Solway . (2013) solway2013Solway, A., Miller, JF. Kahana, MJ.  2013\.  PandaEPL: A library for programming spatial navigation experiments PandaEPL: A library for programming spatial navigation experiments.  Behavior Research Methods451293–1312.  10.3758/s13428-013-0322-5
  * Spiers Maguire (2006) spiers2006thoughtsSpiers, HJ. Maguire, EA.  2006\.  Thoughts, behaviour, and brain dynamics during navigation in the real world Thoughts, behaviour, and brain dynamics during navigation in the real world.  NeuroImage311826–1840.  10.1016/j.neuroimage.2006.01.037
  * Starrett . (2020) starrett2020Starrett, MJ., McAvan, AS., Huffman, DJ., Stokes, JD., Kyle, CT., Smuda, DN.Ekstrom, AD.  2020\.  Landmarks: A solution for spatial navigation and memory experiments in virtual reality Landmarks: A solution for spatial navigation and memory experiments in virtual reality.  Behavior Research MethodsAdvance online publication.  10.3758/s13428-020-01481-6
  * Tcheang . (2011) tcheang_visual_2011Tcheang, L., Bülthoff, HH. Burgess, N.  2011\.  Visual influence on path integration in darkness indicates a multimodal representation of large-scale space Visual influence on path integration in darkness indicates a multimodal representation of large-scale space.  Proceedings of the National Academy of Sciences of the United States of America1081152–1157.  10.1073/pnas.1011843108
  * Vasser . (2017) vasser2017vrexVasser, M., Kängsepp, M., Magomedkerimov, M., Kilvits, K., Stafinjak, V., Kivisik, T.Aru, J.  2017\.  VREX: An open-source toolbox for creating 3D virtual reality experiments VREX: An open-source toolbox for creating 3D virtual reality experiments.  BMC Psychology54.  10.1186/s40359-017-0173-4
  * Wiener . (2020) wiener2019Wiener, JM., Carroll, D., Moeller, S., Bibi, I., Ivanova, D., Allen, P. Wolbers, T.  2020\.  A novel virtual-reality-based route-learning test suite: Assessing the effects of cognitive aging on navigation A novel virtual-reality-based route-learning test suite: Assessing the effects of cognitive aging on navigation.  Behavior Research Methods52630–640.  10.3758/s13428-019-01264-8
  * Williams . (2007) williams2007exploringWilliams, B., Narasimham, G., Rump, B., McNamara, TP., Carr, TH., Rieser, J. Bodenheimer, B.  2007\.  Exploring large virtual environments with an HMD when physical space is limited Exploring large virtual environments with an HMD when physical space is limited.  APGV07: Proceedings of the 4th Symposium on Applied Perception in Graphics and Visualization41–48.  10.1145/1272582.1272590
  * Yamamoto . (2018) yamamoto_imagined2018Yamamoto, N., Mach, DE., Philbeck, JW. Van Pelt, J.  2018\.  Why is the Duration of Imagined Walking Underproduced? A dual-representation view on the mental imagery of locomotion Why is the duration of imagined walking underproduced? A dual-representation view on the mental imagery of locomotion.  PsyArXiv.  10.31234/osf.io/rm298
  * Yamamoto, Meléndez Menzies (2014) yamamoto_homing_2014Yamamoto, N., Meléndez, JA. Menzies, DT.  2014\.  Homing by path integration when a locomotion trajectory crosses itself Homing by path integration when a locomotion trajectory crosses itself.  Perception431049–1060.  10.1068/p7624
  * Yamamoto, Philbeck . (2014) yamamoto_medial_2014Yamamoto, N., Philbeck, JW., Woods, AJ., Gajewski, DA., Arthur, JC., Potolicchio, SJ., Jr.Caputy, AJ.  2014\.  Medial temporal lobe roles in human path integration Medial temporal lobe roles in human path integration.  PLoS ONE9e96583.  10.1371/journal.pone.0096583
  * Yechiam Hochman (2014) yechiam_loss_2014Yechiam, E. Hochman, G.  2014\.  Loss Attention in a Dual-Task Setting Loss attention in a dual-task setting.  Psychological Science25494–502.  10.1177/0956797613510725


* [138] D. Verma, A. Jana and K. Ramamritham. Transfer learning approach to map urban slums using high and medium resolution satellite imagery. Habitat International, vol. 88, pp.101981, 2019.
  * [139] T. Fisher, H. Gibson, Y Liu, M. Abdar, M. Posa, G. Salimi-Khorshidi, A. Hassaine, Y. Cai, K. Rahimi and M. Mamouei. Uncertainty-aware interpretable deep learning for slum mapping and monitoring. Remote Sensing, vol. 14, no. 13, pp.3072, 2022.
  * [140] Q. Cheng, M. Zaber, A. M. Rahman, H. Zhang, Z. Guo, A. Okabe and R. Shibasaki. Understanding the urban environment from satellite images with new classification Method—Focusing on formality and informality. Sustainability, vol. 14, no. 7, pp.4336, 2022.
  * [141] Y. Li, X. Huang and H. Liu. Unsupervised deep feature learning for urban village detection from high-resolution remote sensing images. Photogrammetric Engineering & Remote Sensing, vol. 83, no. 8, pp.567-579, 2017.
  * [142] S. Hafner, Y. Ban and A. Nascetti. Unsupervised domain adaptation for global urban extraction using Sentinel-1 SAR and Sentinel-2 MSI data. Remote Sensing of Environment, vol. 280, pp.113192, 2022.
  * [143] R. Fan, J. Li, W. Song, W. Han, J. Yan and L. Wang. Urban informal settlements classification via a transformer-based spatial-temporal fusion network using multimodal remote sensing and time-series human activity data. International Journal of Applied Earth Observation and Geoinformation, vol. 111, pp.102831, 2022.
  * [144] E. Luo, M. Kuffer and J. Wang. Urban poverty maps-From characterising deprivation using geo-spatial data to capturing deprivation from space. Sustainable Cities and Society, vol. 84, p.104033, 2022.
  * [145] Z. Pan, J. Xu, Y. Guo, Y. Hu and G. Wang. Deep learning segmentation and classification for urban village using a worldview satellite image based on U-Net. Remote Sensing, vol. 12, no. 10, pp.1574, 2022.
  * [146] S. Georganos, S. Hafner, M. Kuffer, C. Linard and Y. Ban. A census from heaven: Unraveling the potential of deep learning and Earth Observation for intra-urban population mapping in data scarce environments. International Journal of Applied Earth Observation and Geoinformation, vol. 114, p.103013, 2022.
  * [147] O. Hall, M. Ohlsson and T. Rögnvaldsson. A review of explainable AI in the satellite data, deep machine learning, and human poverty domain. Patterns, vol. 3, no. 10, 2022.
  * [148] M. Kochupillai, M. Kahl, M. Schmitt, H. Taubenböck and X.X. Zhu. Earth observation and artificial intelligence: Understanding emerging ethical issues and opportunities. IEEE Geoscience and Remote Sensing Magazine, vol. 10, no. 4, pp.90-124, 2022.
  * [149] M. Owusu, M. Kuffer, M. Belgiu, T. Grippa, M. Lennert, S. Georganos and S. Vanhuysse. Geo-ethics in slum mapping. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 5700-5703, 2021.
  * [150] T. Stark, M. Wurm, X.X. Zhu and H. Taubenbock. Quantifying Uncertainty in Slum Detection: Advancing Transfer-Learning with Limited Data in Noisy Urban Environments. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024.
  * [151] T. Lobo. The Ethics of Mapping Slums—And How AI Complicates the Picture. Philosophy of the City Journal, vol. 1, no. 1, pp.127-134, 2023.
  * [152] D.R. Thomson, M. Kuffer,G. Boo, B. Hati, T. Grippa, H. Elsey, C. Linard, R. Mahabir, C. Kyobutungi, J. Maviti and D. Mwaniki. Need for an integrated deprived area “slum” mapping system (IDEAMAPS) in low-and middle-income countries (LMICs). Social Sciences, vol. 9, no. 5, p.80, 2020.
  * [153] G. Yeboah, J. Porto de Albuquerque, R. Troilo, G. Tregonning, S. Perera, S.A. Ahmed, M. Ajisola, O. Alam, N. Aujla, S.I. Azam and K. Azeem. Analysis of openstreetmap data quality at different stages of a participatory mapping process: Evidence from slums in Africa and Asia. ISPRS International Journal of Geo-Information, vol. 10, no. 4, pp.265, 2021.
  * [154] F.E. Fassnacht, H. Latifi and F. Hartig. Using synthetic data to evaluate the benefits of large field plots for forest biomass estimation with LiDAR. Remote sensing of environment, vol. 213, pp.115-128, 2018.
  * [155] V.A. Le, V. Reddy, Z. Chen, M. Li, X. Tang, A. Ortiz, S.F. Nsutezo, and C. Robinson. Mask Conditional Synthetic Satellite Imagery. arXiv preprint arXiv:2302.04305, 2023.
  * [156] L. Wang, S. Fang, X. Meng and R. Li, Building Extraction With Vision Transformer. IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-11, 2022.
  * [157] J. Yao, B. Zhang, C. Li, D. Hong and J. Chanussot, Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework. IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-15, 2023.
  * [158] A. Jamali, M. Mahdianpari, F. Mohammadimanesh and S. Homayouni. A deep learning framework based on generative adversarial networks and vision transformer for complex wetland classification using limited training samples. International Journal of Applied Earth Observation and Geoinformation, vol. 115, pp.103095, 2022.
  * [159] P. Song, J. Li, Z. An, H. Fan and L. Fan. CTMFNet: CNN and Transformer Multiscale Fusion Network of Remote Sensing Urban Scene Imagery. IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-14, 2023.
  * [160] U.N. Habitat. ”The challenge of slums: global report on human settlements”, United Nations Human Settlement Program, 2003.
  * [161] C.M. Gevaert, D. Kohli and M. Kuffer. Challenges of mapping the missing spaces. 2019 Joint Urban Remote Sensing Event (JURSE), pp. 1-4, 2019.
  * [162] R.J. Schmitt, E. Morgenroth and T.A. Larsen. Robust planning of sanitation services in urban informal settlements: An analytical framework. Water research, vol. 110, pp.297-312, 2017.
  * [163] K.Klemmer, G. Yeboah, J.P. de Albuquerque and S.A. Jarvis. Population mapping in informal settlements with high-resolution satellite imagery and equitable ground-truth. arXiv preprint arXiv:2009.08410, 2020.