This work has been submitted to Elsevier for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible.

# Vehicle Teleoperation: Performance Assessment of SRPT Approach Under State
Estimation Errors

Jai Prakash, Michele Vignati, and Edoardo Sabbioni The authors belong to the
Department of Mechanical Engineering, Politecnico Di Milano, Italy (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Vehicle teleoperation has numerous potential applications, including serving
as a backup solution for autonomous vehicles, facilitating remote delivery
services, and enabling hazardous remote operations. However, complex urban
scenarios, limited situational awareness, and network delay increase the
cognitive workload of human operators and degrade teleoperation performance.
To address this, the successive reference pose tracking (SRPT) approach was
introduced in earlier work, which transmits successive reference poses to the
remote vehicle instead of steering commands. The operator generates reference
poses online with the help of a joystick steering and an augmented display,
potentially mitigating the detrimental effects of delays. However, it is not
clear which minimal set of sensors is essential for the SRPT vehicle
teleoperation control loop.

This paper tests the robustness of the SRPT approach in the presence of state
estimation inaccuracies, environmental disturbances, and measurement noises.
The simulation environment, implemented in Simulink, features a 14-dof vehicle
model and incorporates difficult maneuvers such as tight corners, double-lane
changes, and slalom. Environmental disturbances include low adhesion track
regions and strong cross-wind gusts. The results demonstrate that the SRPT
approach, using either estimated or actual states, performs similarly under
various worst-case scenarios, even without a position sensor requirement.
Additionally, the designed state estimator ensures sufficient performance with
just an inertial measurement unit, wheel speed encoder, and steer encoder,
constituting a minimal set of essential sensors for the SRPT vehicle
teleoperation control loop.

###### Index Terms:

Vehicle teleoperation, remote driving, network delay, SRPT, NMPC, state
estimation, measurement noises, simulink.

##  I Introduction

Figure 1: Graphical depiction of the SRPT approach utilized for direct vehicle
teleoperation. The remote vehicle is provided with successive reference poses
as it progresses forward. The control-loop includes a state-estimation block.

(a)

(b)

Figure 2: SRPT block diagrams. (a) Without state estimation, its efficacy is
discussed in [1]. (b) With state estimation, robustness assessment in presence
of environmental disturbances and measurement noises is the prime focus of
this work.

Teleoperation is the remote control of a device or a vehicle from a distance.
This can be done using either wired communication or wireless communication.
Here the vehicle is a mobile robot that can be controlled remotely, typically
wirelessly.

Automated vehicles (AVs) are a promising technology for mobility in the
future. Since their development has been a focus of academic and industrial
study for many years, their introduction to the market is anticipated.
Driverless cars with safety drivers can already be seen on public roads in
some parts of the world. The utilization potential of AVs is, however,
seriously threatened by lingering difficulties, required sophistication, or
even as of yet undiscovered edge cases. A teleoperation system can apparently
be considered a backup plan for AVs. The goal of teleoperation technology is
to offer a secure and effective method to get over these restrictions anytime
an autonomous driving (AD) function hits the limits of its Operational Design
Domain (ODD) [2].

Ongoing shortcomings in vehicle teleoperation are:

  1. !

Human-machine interaction: User experience of teleoperation systems,
particularly in the areas of input modality (e.g. touch, gesture, voice),
feedback (e.g. haptic, visual, auditory), and cognitive load [3].

  2. !

Network latency: It introduces variable-delays and degradation in the
performance of teleoperation, particularly in low-bandwidth or high-latency
environments.

  3. !

Safety and reliability: Robust fault detection, recovery mechanisms, and
techniques for handling communication failures or malfunctions.

While these issues continue to pose significant challenges, the objective of
the approach discussed in this paper is to mitigate the impact of network
latency in vehicle teleoperation. By addressing this specific challenge, we
aim to improve the safety and effectiveness of teleoperation systems and move
closer towards achieving reliable and efficient teleoperation of vehicles.

Time delay reduces the speed and accuracy with which human operators can
accomplish a teleoperation task [4, 5]. When delays are considerable, human
operators tend to overcorrect steer, causing oscillations that degrade
teleoperation performance and possibly destabilize the control-loop [6, 7].
Over the years, different teleoperation concepts for road vehicles have been
developed and researched. Domagoj Majstorovic et al. [2] suggest a taxonomy
for vehicle teleoperation approaches:

  1. $\blacksquare$

Direct control: The human operator sees the sensor data and transmits control
signals, such as steer, throttle, etc. It suffers from reduced situational
awareness and transmission latency [8, 9, 10, 11, 12, 13, 14, 15, 16].

  2. $\blacksquare$

Shared control: A shared controller inside the vehicle assesses the operator
commands as per current surrounding and manipulates them to avoid an imminent
collision [17, 18, 19, 20, 21, 22]. It improves safety but still suffers from
latency.

  3. $\blacksquare$

Trajectory or waypoints guidance: The vehicle tracks the path and speed
profile generated by the operator. Network latency doesn’t impact the
performance, but real-time generation of path and speed profile is difficult
[23, 24, 25, 26, 27].

  4. $\blacksquare$

Interactive path planning: The perception module of vehicle computes optimal
paths and the operator confirms one of them to follow. It circumvents network
latency but requires a functional set of AD modules such as perception,
trajectory planning, etc. [28, 29]

  5. $\blacksquare$

Perception modification: The human operator supports the AD perception module
by identifying false-positive obstacles. It majorly depends upon the AD
modules [30].

###  I-A Previous Work

Previously, we proposed [31] and evaluated [1] the novel SRPT approach for
vehicle teleoperation. It combines the benefits of direct control and waypoint
guidance concepts. The human operator online generates waypoints using
joystick steering by manipulating an augmented lookahead point on the display.
Eventually, the vehicle controller drives the vehicle. In this earlier work, a
simulation environment is used, and so the vehicle controller is aware of the
(perfect) actual vehicle states. But in the real world, all the states are not
readily available due to obvious measurement limitations. Robustness of the
novel SRPT approach for vehicle teleoperation has to be assessed in presence
of state estimation accuracies. Furthermore, a minimal but sufficient set of
sensors has to be identified for the state estimation requirement of SRPT.

###  I-B Related work - Vehicle state estimation

Bersani Mattia et al. [32] formulated an unscented Kalman filter (UKF) with
kinematic single-track vehicle model to estimate vehicle pose and velocities.
Its plant model input are IMU measurements and not the steering angle. The
measurement model consists of GPS position, GPS speed and speed measured by
wheel encoders. It doesn’t estimate tire shear forces. Doumiati Moustapha et
al. [33] formulated an extended Kalman filter (EKF) with a four-wheel vehicle
model to estimate the side-slip angle and lateral tire forces. Its plant model
input is the steering angle. Its measurement model consists of yaw rate,
vehicle velocity, longitudinal, and lateral accelerations. In our paper, we
use the single-track version of this EKF formulation.

###  I-C Contribution of Paper

  * 1

Performance assessment of the SRPT vehicle teleoperation approach in the
presence of state estimation inaccuracies.

  * 2

Identification of the minimum set of sensors required for state estimation in
the SRPT vehicle teleoperation control loop, even under challenging scenarios
involving difficult maneuvers, adverse environmental conditions, and various
types of measurement noises.

  * 3

Highlights that this approach does not require a GPS sensor, despite relying
on sending reference poses to the remote vehicle.

###  I-D Outline of Paper

The paper is organized as follows. Section II presents the skeleton of the EKF
state-estimator. Section II-C reiterates the role of human operator in SRPT
approach. Section III provides an overview of the simulation platform. Section
IV discusses the experimental structure. Section V presents and discusses the
results. Section VI concludes with the work summary.

##  II Method

The SRPT approach proposed in [1], requires vehicle states. The block diagram
of SRPT approach is shown in figure 2a. Considering the single-track vehicle
model for state estimation, the states are side-slip angle ($\beta$), vehicle
yaw-rate ($\dot{\psi}$), vehicle heading ($\psi$), lateral force at front axle
($F_{y,F}$), lateral force at rear axle ($F_{y,R}$), vehicle longitudinal
speed ($V_{x}$), position of vehicle CG ($[x;y]$), and equivalent steer angle
at front axle ($\delta$):

$X=\left[\>\beta\>,\>\dot{\psi}\>,\>\psi\>,\>F_{y,F}\>,\>F_{y,R}\>,\>V_{x},\>x\>,\>y\>,\>\delta\>\right]^{T}\;\;\;\;.$
(1)

The NMPC block optimizes for vehicle steer-speed commands, to keep the vehicle
motion in sync with the received successive reference poses [1]. It also
accounts for input constraints such as steer-rate constraint (actuator
limitation) and acceleration constraint (for passenger comfort):

$\displaystyle\left[\dot{\delta}_{min}=-20^{\circ}/s\right]$
$\displaystyle\leq\dot{\delta}\leq\left[\dot{\delta}_{max}=+20^{\circ}/s\right]$
(2) $\displaystyle\>\>\>\>\left[a_{min}=-3m/s^{2}\right]$ $\displaystyle\leq
a\leq\left[a_{max}=1m/s^{2}\right]$ (3)

Its outputs are, the steer angular velocity ($\dot{\delta}$), and vehicle
acceleration ($a$).

Figure 2b shows the incorporation of a state-estimator in the form of EKF. Its
prediction model consists of the single-track version of the four-wheel
vehicle model given in [33] and it is given by

$\dot{X}=\left[\begin{array}[]{c}\frac{1}{mV_{x}}\left(F_{y,F}\cos\delta+F_{x,F}\sin\delta+F_{y,R}\right)-\frac{\beta\cdot
a}{V_{x}}-\dot{\psi}\\\
\frac{1}{I_{Z}}\left[\left(F_{y,F}\cos\delta+F_{x,F}\sin\delta\right)l_{F}-F_{y,R}\>l_{R}\right]\\\
\dot{\psi}\\\
\frac{V_{x}}{\lambda}\left[C_{\sigma,F}\>\sigma_{F}-F_{y,F}\right]\\\
\frac{V_{x}}{\lambda}\left[C_{\sigma,R}\>\sigma_{R}-F_{y,R}\right]\\\
\dot{V_{x}}\\\ V_{x}\left(\cos\psi-\sin\psi\;\tan\beta\right)\\\
V_{x}\left(\sin\psi+\cos\psi\;\tan\beta\right)\\\
\dot{\delta}\end{array}\right].$ (4)

$[C_{\sigma,F},C_{\sigma,R}]$ are the lumped cornering stiffness of front and
rear axles respectively. [$m_{f},m_{R}$] are distribution of vehicle mass on
front and rear axle based on [$l_{F},l_{R}$] respective distances of axles
from CG. To make it applicable for zero vehicle speed, wherever the $V_{x}$ is
in the denominator in (4), it is substituted by $max(0.01,V_{x})$.

For simplicity, relaxation length phenomenon, rolling resistance and
aerodynamic drag are ignored in longitudinal force development and thus given
by

$\displaystyle F_{x,F}=\begin{cases}\>\>\>\>\>\>\>\>m\,a,&\text{if }a\geq 0\\\
\>\>\>\>\gamma\,m\,a&\text{otherwise},\end{cases}$ (5)

where $\gamma$ is the braking bias.

$[\sigma_{F},\sigma_{R}]$ are the tires slips given by

$\begin{array}[]{l}\sigma_{F}\simeq\tan\delta-\beta-\dot{\psi}\frac{l_{F}}{V_{x}}\\\
\sigma_{R}\simeq\>\>\>\>\>\>\>\>\>\>\>-\beta+\dot{\psi}\frac{l_{R}}{V_{x}}\end{array}$
(6)

Figure 3: Single-track vehicle model. Reprinted with permission from Ref. [31]

Parameters used in the state estimation correspond to a FWD typical passenger
vehicle and mentioned in Table I.

TABLE I: Vehicle parameters for the single-track model. Parameter | Value
---|---
$m$ | 1681 kg
$I_{z}$ | 2600 kg s2
$[m_{F};\>m_{R}]$ | [871.6; 809.4] kg
$[l_{F};\>l_{R}]$ | [1.3; 1.4] m
$[C_{\sigma,F};\>C_{\sigma,R}]$ | [1.057e+05; 1.050e+05] N
$\lambda$ (Relaxation length) | 0.3 m
$\gamma$ (Braking bias) | 0.6

Discretization steps are the usual first-order approximation with a time step
of $0.001s$, as also presented in the appendix of [34]. Tuning of prediction
noise covariance, $Q_{k}\in\mathbb{R}^{9\times 9}$, is discussed in the
section below.

Figure 4: Block diagram used to optimally tune the prediction covariance
$Q_{k}$, by minimizing prediction error of relative pose at $\hat{B}$ wrt pose
at $\hat{A}$. The trajectory is traversed at $V_{Ref}=22km/h$.

###  II-A Prediction noise, $Q_{k}$

$Q_{k}$ is assumed to be a diagonal matrix with respective variances at the
diagonal. To tune $Q_{k}$, the vehicle performs one lap over the whole
trajectory given in figure 8. Actual vehicle states and (zero mean Gaussian)
noisy measurements are recorded. Referring to figure 4, to obtain relative
poses for a time window of $0.3s$ (max round-trip delay), both estimated pose
and actual pose are delayed by $0.3s$. Comparison is performed between
estimated relative poses and actual relative poses. $Q_{k}$ is optimally tuned
offline, aiming minimum prediction error in relative poses, the cost function
is given by (7). The aim is to tune the variances corresponding to just $6$
out of $9$ states, i.e.,
$\left[\>\beta\>,\>\dot{\psi}\>,\>F_{y,F}\>,\>F_{y,R}\>,\>\delta\>,\>V_{x}\right]$.
This is because other states are unobservable with the measurement model given
in (10). The nonlinear multivariable solver of Matlab, fmincon, is used to
optimize iteratively the variances corresponding to these $6$ states.

$J=W_{Position}RMS\left(\left\|\vec{\hat{B}}_{\hat{A}}-\vec{B}_{A}\right\|_{2}\right)+\\\
W_{Heading}RMS\left(\left\|\psi_{\hat{A}}^{\hat{B}}-\psi_{A}^{B}\right\|_{2}\right)\;\;\;\;.$
(7)

$\vec{\hat{B}}_{\hat{A}}$ \- Predicted relative position of $\hat{B}$ in
$\hat{A}$ frame (a vector of 2 elements $[\hat{x};\hat{y}]$)
$\vec{B}_{A}$ \- Actual relative position of ${B}$ in ${A}$ frame (a vector of
2 elements $[x;y]$)
$\psi_{\hat{A}}^{\hat{B}}$ \- Predicted relative heading of $\hat{B}$ in
$\hat{A}$ frame (a scalar)
$\psi_{A}^{B}$ \- Actual relative heading of ${B}$ in ${A}$ frame (a scalar)
$W_{Position}=1$ \- Weight for the position error
$W_{Heading}=1e^{-2}$ \- Weight for the heading error
$RMS()$ is performed over the whole trajectory. After the solver optimization,
the tuned process covariance ($Q_{k}$) is given by

$\begin{split}Q_{k}=\frac{\left(diag\left[\begin{aligned}
&\>0.0125\>,\>0.0011\>,\>1\>,\>0.3162\>,\\\
&\>0.3415\>,\>0.0008\>,\>1\>,\>1\>,\>2e-5\>\\\
\end{aligned}\right]\right)^{2}}{10}\end{split}$ (8)

E.g. process noise for state $\beta$ is
$\mathcal{N}\left(0,\,0.0125^{2}/10\right)$ and so on for other states. It is
presented as a factor of $10$, so that $std$ of the vehicle states which are
as it is in measurement vector, can be readily compared, as there are $10$
prediction steps in between two measurements. Process noise corresponding to
unobservable states, $[x,y,\psi]$, are arbitrarily assigned
$\mathcal{N}\left(0,\,1^{2}/10\right)$, as estimation of their evolution is
unimportant.

###  II-B Measurement model

A minimal set of sensors, comprising a virtual IMU, a speed encoder and a
steer encoder is considered. Intending to measure only lateral acceleration at
CG ($a_{y,meas}$), angular velocity at CG ($\dot{\psi}_{meas}$), longitudinal
speed ($V_{x,meas}$) and steer angular position ($\delta_{meas}$). Hence the
measurement vector is given as

$Z_{k}=\left[\>a_{y,meas}\>,\>\dot{\psi}_{meas}\>,\>V_{x,meas}\>,\>\delta_{meas}\right]^{T}\;\;\;\;,$
(9)

and the measurement model is given as

$h\left(X\right)=\left[\begin{array}[]{c}\frac{1}{m}\left(F_{y,F}\cos\delta+F_{y,R}\right)\\\
\dot{\psi}\\\ V_{x}\\\ \delta\end{array}\right]\;\;\;\;.$ (10)

The update steps are incapable to correct the vehicle pose ($x,y,\psi$)
because these states are unobservable with the measurement model given in
(10). Consequently, these states have the tendency to diverge with time.
Besides of this divergence, robustness of SRPT approach majorly depends upon
the accuracy of (relative) pose estimation for a time window equivalent to the
round trip network delay, further discussed in section V-A.

Sampling time-step is $0.010s$. Taking advantage of using a simulation
framework, zero-mean Gaussian noise is added to the measurements. Measurement
covariance respective to measurement vector (9) is given by

$R_{k}=diag\left(\left[\>0.112^{2}\>,0.005^{2}\>,\>0.083^{2}\>,\>0.003^{2}\right]\right)\;\;\;\;.$
(11)

Standard deviations corresponding to $a_{y,meas}$ and $\dot{\psi}_{meas}$ are
taken from a static experiments performed with Bosch 5-axis IMU. Standard
deviations corresponding to $V_{x,meas}$ and $\delta_{meas}$ are taken from
speed and steer encoders.

###  II-C Human model

In contrast to conventional vehicle teleoperation, where human operator
transmits steer-throttle commands to the remote vehicle, in SRPT approach,
human operator transmits reference poses to be followed by the remote vehicle.
These reference poses are generated and transmitted in such a way that, when
the vehicle receives them, they resemble those ahead of it. Look-ahead time of
$[1+\tau_{1}+\tau_{2}]s$ is considered for reference pose generation. As a
result, the vehicle receives reference-poses that are approximately $1s$ ahead
of it. The $1s$ horizon is chosen arbitrary, taking into account that, in
general, a driver steers a vehicle based on an upcoming vehicle position.
Eventually, the same time horizon of $\Delta t_{Horizon}=1s$ is considered for
the NMPC block to optimize for vehicle steer-speed commands.

Figure 5: Working principle of the human model block for the case of perfectly
known states (Figure 2a). Its task is to choose the future reference pose
based on the received vehicle pose and look-ahead distance.

The task of the human model block is to transmit information that informs the
vehicle about its aiming direction. Referring to figure 5, human model block
receives delayed vehicle states, $X(t)e^{-\tau_{2}s}$, which consists of
vehicle pose, ${P}^{C^{\prime}}_{O}$. It is the delayed vehicle pose in global
reference frame, $O$. Being aware of the whole trajectory, the human model
block first finds the closest point $C$ on the reference trajectory. Then it
finds the point $D$, which is $L_{ind}$ distance ahead of point $C$. The
$L_{ind}$ is the look-ahead distance govern by below relation:

$L_{ind}=V_{x}\cdot\tau+\max(V_{x}\cdot\Delta t_{Horizon},\;\;l_{F})\;\;\;\;.$
(12)

It is lower bounded by $l_{F}$, the front axle distance from CG. It is
linearly proportional to the round trip delay ($\tau$) and to the vehicle
speed ($V_{x}$). The first term tries to compensate for round-trip delay, and
the second term aims to generate the terminal condition for the NMPC horizon.

Figure 6: SRPT block diagram with state estimator and human model block. To
compensate for the absence of visual feedback to human model block, it has
additional feedback (highlighted in green) of delayed but actual vehicle
state, $X(t)$.

There are two ways in which the aiming direction can be transmitted to the
vehicle:

  1. 1.

Transmit the relative reference pose, ${P}^{D}_{C^{\prime}}$. It is relative
position and heading of pose-D with respect to pose-C′, it acts as a
correction term, which tries to bring the vehicle close to the desired
trajectory. Upon receive of this relative-pose, the vehicle first estimates
how much it has already traveled during the round-trip delay and how much more
it has to travel. This estimation is possible, as messages are timestamped.

  2. 2.

Transmit the global reference pose. Transmit the reference pose, $X_{Ref}$, in
global reference frame.

$X_{Ref}={P}^{C^{\prime}}_{O}+{P}^{D}_{C^{\prime}}={P}^{D}_{O}\;\;\;\;.$ (13)

For this paper, the second way is adopted. It eliminates the vehicle task of
estimating how much it has traveled during the round-trip delay.

In case when the human block receives predicted states
$\left(\hat{X}\left(t\right)e^{-\tau_{2}s}\right)$, two further cases has to
be discussed.

  1. (i)

Human in the loop - The human block is a human operator. The operator receives
the delayed predicted vehicle state as well as he/she sees the delayed visual
environment. With these modality the correction term, ${P}^{D}_{C^{\prime}}$,
can be decided by the operator. In our previous work [1], this correction term
is getting generated online using steering joystick, briefly represented by
below relation

$\hat{X}_{Ref}=\hat{P}^{C^{\prime}}_{O}+\Delta P_{Joystick}\;\;\;\;.$ (14)

$\hat{P}^{C^{\prime}}_{O}$ \- It denotes the predicted vehicle pose, readily
available in $\hat{X}\left(t\right)e^{-\tau_{2}s}$
$\Delta P_{Joystick}$ \- It is the correction term generated by the augmented
steer indicator marker on the visual interface with help of joystick steering
[1].

  2. (ii)

Human model in the loop - Human is substituted by a human model that only
receives delayed predicted vehicle states. To compensate for the absence of
visual environment, this block has to be additionally fed with delayed but
actual vehicle states, as shown in green in figure 6. This is because the
predicted vehicle poses diverge from the actual vehicle poses (due to
measurement model limitations discussed section II-B ). Diverging vehicle
states can’t be compared with the desired trajectory. To get the closest point
$C$, actual vehicle states are required. Passing actual states makes the human
model block perfect, which supports the main focus of this paper, which is,
the performance assessment of SRPT approach in vehicle teleoperation in
presence of state estimation error. The teleoperation loop contains only state
prediction error, no human error.

The reference pose to be transmitted by the human model block is given by:

$\hat{X}_{Ref}=\hat{P}^{C^{\prime}}_{O}+{P}^{D}_{C^{\prime}}\;\;\;\;.$ (15)

${P}^{D}_{C^{\prime}}$ \- It is the relative pose of $D$ on the trajectory
with respect to the delayed actual vehicle pose ($C^{\prime}$). The correction
term is same as in (13). This is available, as the human model block is
additionally fed with delayed actual vehicle states.

##  III Simulation Platform

Figure 7: Simplified block diagram of the simulation platforms set up on Simulink. Unity has no role in simulation, it is just to display the manoeuvres. TABLE II: Description of the blocks used in the simulation platform. Block | Description | Rate
---|---|---
Vehicle 14dof | A 14dof vehicle model to simulate a real vehicle | 1000 Hz
Sensors | It is a channel extracting IMU and encoders observations | 100 Hz
EKF | It is a Simulink Extented Kalman filter block | 1000/100 Hz
$e^{-\tau_{2}s}$ | Variable network downlink delay | 30 Hz
Human model | Ref Pose decider logic | 30 Hz
$e^{-\tau_{1}s}$ | Constant network uplink delay, $\tau_{1}=0.060s$ | 30 Hz
NMPC | Non-linear model predictive controller, Acados toolkit | 50 Hz
Unity | An external block, to visualize the real vehicle maneuvers | 100 Hz

A faster than real-time, vehicle teleoperation simulation test platform is
developed using Simulink + Unity3D to emulate the network delayed vehicle
teleoperation system, as shown in figure 7. The Simulink output is connected
to Unity only to provide visuals of the real vehicle manoeuvres.

Further descriptions of the blocks are mentioned in table II. The table also
mentions the working rate of each block, different working rate of each block
aligns with reality, where each module has its own working capabilities. The
EKF block has prediction step of $dt=0.001s$ and measurement update at each
$0.010s$. The blocks $e^{-\tau_{2}s}$, Human model, and $e^{-\tau_{1}s}$ work
synchronously with each other at $30Hz$, to simulate the usual discrete nature
of video streaming to the control station. The downlink delay ($\tau_{2}$) is
considered variable delay to simulate usual network delays but the uplink
delay ($\tau_{1}$) is considered a constant of $0.060s$ due its lower
magnitude and lower variability. For downlink delay, generalized extreme value
distribution, $GEV(\xi=0.29,\mu=0.200,\sigma=0.009)$ is used [35, 16]. Here,
$\xi$ is the shape parameter, $\mu$ is the location parameter and $\sigma>0$
is the scale parameter. Positive $\xi$ means that the distribution has a lower
bound $(\mu-\frac{\sigma}{\xi})\approx 0.169$ s$\>(>0)$ and a continuous right
tail based on the extreme value theory. This probability distribution keeps
variable downlink delay in range of $0.169s-0.300s$.

TABLE III: Sources of measurement noise Noise sources | Description
---|---
Gaussian noise | AWGN; $R_{k}=diag\left(\left[\>0.112^{2}\>,0.005^{2}\>,\>0.083^{2}\>,\>0.003^{2}\right]\right)$
$gainV=1.05$ |  | Vehicle speed = rolling speed $\times$ rolling radius.
---
Inaccuracy of rolling radius act as a gain form of disturbance.
$bias\delta=0.5^{\circ}$ | Offset mounting of steer position encoder.
$3^{\circ}tiltedImu$ |  |  $5^{\circ}$ tilted mounting of IMU in clockwise roll direction;
---
$a_{y,noisy}=a_{y}cos(3^{\circ})-9.8sin(3^{\circ})\;\;\;\;\;\;\;;\;\;\;\;\;\;\;\;\dot{\psi}_{noisy}=\dot{\psi}cos(3^{\circ})$
$6^{\circ}tiltedImu$ |  |  $6^{\circ}$ tilted mounting of IMU in clockwise roll direction;
---
$a_{y,noisy}=a_{y}cos(6^{\circ})-9.8sin(6^{\circ})\;\;\;\;\;\;\;;\;\;\;\;\;\;\;\;\dot{\psi}_{noisy}=\dot{\psi}cos(6^{\circ})$
| $C_{F,over}=1.2C_{F}$
---
$C_{R,over}=1.2C_{R}$
Use of overly estimated cornering stiffness for state estimation.
Figure 8: Track contains various sections A-H of difficult manoeuvres and
worst-case environmental conditions.

(a)

(b)

(c)

Figure 9: A visual depiction of noise sets. (a) Set 1. (b) Set 2. (c) Set 4.
Actual values are in dark shades and respective noise values are in light
shades.

##  IV Experimental Setup

Figure 8 depicts a $438m$ long test track with eight regions labeled A-H.
These regions attempt to simulate increasingly difficult maneuvers and worst
environmental conditions. A is cornering with radius of $15$m ($R15$), B is
cornering ($R8$) on a surface with road adherence coeffcient, $\mu=0.7$. C is
double lane change, D is cornering with $\mu=0.5$, E-F are strong lateral wind
with Chinese hat profile [36, 37], G is a U-turn with $\mu=0.33$, and H is a
slalom. The task is to follow the track center line as closely as possible,
with an upper cap on vehicle speed, $V_{Ref}=22$ km/h. It is expected that
during difficult maneuvers the NMPC block modulates the vehicle speed
($V_{opt}$), to reduce the cross-track error, which is a desirable behaviour.

In order to test the limits of SRPT approach, in addition to the environmental
disturbances depicted in the trajectory above, some measurement noises are
also considered. They are mentioned in Table III.

For the experiments, these noise sources are grouped in the following sets:

  1. (i)

Actual states, no EKF

  2. (ii)

Noise set 1 : EKF + Gaussian noises

  3. (iii)

Noise set 2 : EKF + Gaussian noises + gainV + $bias\delta$ \+
$3^{\circ}tiltedImu$

  4. (iv)

Noise set 3 : EKF + Gaussian noises + gainV + $bias\delta$ \+
$3^{\circ}tiltedImu+C_{F,over}+C_{R,over}$

  5. (v)

Noise set 4 : EKF + Gaussian noises + gainV + $bias\delta$ \+
$6^{\circ}tiltedImu$

  6. (vi)

Noise set 5 : EKF + Gaussian noises + gainV + $bias\delta$ \+
$6^{\circ}tiltedImu+C_{F,over}+C_{R,over}$

To have a visual apprehension of these noise sets, [ii-iii-v] sets are
depicted in figure 9. These sets are recorded while the vehicle is travelling
through region-B of the trajectory at $22km/h$ in the simulation framework. To
further illustrate, in figure 9b-9c, evident difference can be observed in
noisy $a_{y}$, $\dot{\psi}$, and $V_{x}$ measurements fed to the EKF vs actual
conditions.

(a)

(b)

Figure 10: (a) Look-ahead driver model control. (b) Tuning of $k1$ for the
look-ahead driver model.

###  IV-A Look-ahead driver model

Performance of SRPT approach where subsequent reference poses are transmitted
to vehicle is better than general vehicle teleoperation where steer commands
are transmitted [1]. This is because subsequent reference poses are advance in
time, this advancement in time compensates for the network delays.

For the sake of completeness of this paper, besides considering above
mentioned noise cases, a separate vehicle teleoperation mode of a look-ahead
driver (figure 10a) is also assessed. This mode represents the general driving
case, in which the human operator steers the vehicle to try to align a look-
ahead point with the reference trajectory. This mode is independent of the
sensor noises, as in this mode the vehicle pose is not inferred by the state
estimator but rather by the human perception of the environment.

Figure 11: Region-wise Max and RMS cross-track error with various
teleoperation modes at $vRef=22km/h$. (a) No network delays. (b) Under network
delays (200-300ms). Consistent performances of SRPT modes in both no-delay and
delay cases. Figure 12: Trajectory traversed with various modes under network
delays (200-300ms) at $vRef=22km/h$. Apart from the simple look-ahead driver
mode, every other mode matches the reference trajectory

Look-ahead driver model based on cross-track error at the look-ahead point
(motivated by [38]) is given by

$\displaystyle\delta$ $\displaystyle=-k_{1}\cdot\Delta Y$ (16) $\displaystyle
lookaheadDistance$ $\displaystyle=k_{2}\cdot V_{x}\;\;\;\;.$ (17)

$\delta$ : Steer angle (rad)
$k_{1}$ : Gain term, constant for a given vehicle longitudinal speed.
$\Delta Y$ : Cross-track error of the look-ahead point from the reference
trajectory.
$k_{2}=0.90$ : look-ahead time (s).

$k_{1}$ and $k_{2}$ are considered unchanged for both no-delay and delayed
teleoperation cases. Although in presence of delays a human operator can adapt
his actions, but keeping $[k_{1};\;k_{2}]$ unchanged ensures no adaptability
and highlights performance deterioration due to delays.

$k_{1}=0.213$ is tuned to have less $\Delta Y_{RMS}$ and less $\Delta
Y_{Max}$, while driving across A-B-C regions of the trajectory. Tuning is
performed by iterating over a range of $k_{1}=[0.17:0.002:0.30]$ and observing
the cross-track error (observations are shown in figure 10b).

##  V Results and discussion

Figure 12 , qualitatively shows better performance of SRPT approach even in
the presence of all the disturbances and measurement noises. The red
trajectory of look-ahead driver model resulted in big oscillations due to
network delay and due to steer-rate saturation. These oscillations are not
present in SRPT approach because NMPC block accounts the steer-rate limitation
and consequently decelerates vehicle to allow more time to steer.

Figure 13: Vehicle speed, steer and lateral acceleration with various modes
under network delays (200-300ms) at $vRef=22km/h$. Data legends are identical
as shown in figure 11. Time delay induced oscillation of vehicle steer is
significantly reduced with SRPT approach even in the presence of sensor
noises.

Figure 14: EKF state estimation assessment at $vRef=22km/h$. Data legends are
identical as shown in figure 11. Comparison of estimation of important states
($\beta,\dot{\psi},F_{y,F},F_{y,R}$) with its respective actual values.

Figure 11 , quantitatively compares performances in terms of observed cross-
track error region-wise respectively. Larger and lighter bars are
representative of the maximum values while thinner and darker bars within
larger ones are representative of the rms values.

  1. 1.

Observation I: Insignificant effect of noise-sets. Across all the regions
whether in no-delay or in delay cases, SRPT approach resulted consistent
cross-track performance, whether true states are considered or EKF estimations
are considered.

  2. 2.

Observation II: Significant path-tracking performance improvement. In network
delay case, for region C and region H, where maneuvers are difficult due to
aggressive steer requirement, SRPT approach resulted in significant reduction
in cross-track error.

  3. 3.

Observation III: Relative to look-ahead driver mode, SRPT mode resulted in
slightly large cross-track error in region-G (very low ground adhesion,
$\mu=0.33$). This is because, the NMPC block is unaware of this big change in
plant model for long time. Still, SRPT mode results similar performance in
delay case as in no-delay case. E.g., Obs III rectangles highlight almost same
cross track error in delay case as in no-delay case.

(a)

(b)

Figure 15: (a) EKF estimation of diverging states ($x,y,\psi$) at
$vRef=22km/h$ for a time window of $\Delta=300$ms. Data legends are identical
as in figure 11. (b) EKF estimation error of diverging states ($x,y,\psi$) for
the time window of $\Delta=300$ms.

In figure 14, modulation of vehicle speed ($V_{x}$) at various regions
(especially in region H) of the track, results in better path tracking
performance and ensures stability of the operation. With modulated vehicle
speed, more time is available to steer, which eliminates oscillations in
steering ($\delta$). Eventually, lateral acceleration ($a_{y}$) is also
constrained, enhancing comfort inside the vehicle.

The NMPC block in figure 7 requires all the states mentioned in eq 1. Out of
these states, $[x,y,\psi]$ states diverge because of measurement model
limitation. Figure 14 assess EKF state estimation accuracy by comparing
estimated values of important non-diverging states
$[\beta,\dot{\psi},F_{y,F},F_{y,R}]$ with its respective actual values. For
all noise-sets states $[\beta,\dot{\psi}]$ matches quite accurately for all
regions, except E and F. These are high speed wind gust regions. Even in such
an extreme disturbances the estimation error is less, e.g., in region-F max
error in $\beta$ estimation is around $1.5^{\circ}$. The estimation error of
axle lateral forces is significant; because of the IMU tilt, some of the
gravity contribution is getting added to the lateral acceleration measurement.
But, overall path tracking performance with SRPT approach in vehicle
teleoperation is found to be insignificantly affected by the estimation error
of lateral axle forces.

###  V-A How SRPT is unaffected from diverging state estimation?

Estimation of states $[x,y,\psi]$ diverge with time, as it is unobservable by
the measurement model. Even then the SRPT approach improves vehicle
teleoperation performance.

This is due to the fact that human block in figure 2b-figure 5 or human model
block in figure 6-figure 7, adds a relative pose to the received pose. The
received pose is in global coordinates and the relative pose is relative to
the received pose. This means, if the received pose is diverged (due to EKF
state estimation), the reference-pose generated by the human block is also
diverged. The NMPC block at the remote vehicle, compares the received
reference-pose with the current estimation of vehicle pose, to extract the
relative reference-pose. Based on that, the NMPC block optimizes for steer-
speed commands. Needless to mention that, due to network delay, by the time
vehicle receives a reference-pose, it has moved forward a little. What ensures
the overall performance is the accuracy of vehicle pose estimation by the EKF
block for a time window equal to the round-trip network delay ($200-300$ms).

To analyze the divergence in estimations of the diverging states $[x,y,\psi]$,
a moving time window of $\Delta=300$ms (maximum of round-trip delay) is
considered. Figure 15a, compares the estimated state evolution of states
$[\Delta x,\Delta y,\Delta\psi]$ with the actual evolution of respective
states for the moving time window of $300$ms (for all the noise-sets). The
estimated state evolution matches the actual state evolution for all the
noise-sets and in all the regions of the track.

As in this figure, errors seem unnoticeable. Figure 15b, highlights the error
by presenting the difference between the estimated state evolution in the time
window and the actual state evolution in the time window. The positive error
in $x$-direction is due to the positively biased speed measurement. In the
H-region the max error is $-0.2$m, this is the region where the NMPC
controller decided to decelerate vehicle speed, even at this large estimation
error, the SRPT approach performed better. In $y$ and $\psi$ direction, the
largest error is present at the F-region, which is a region of strong cross-
wind of $80$km/h. Even then, estimation error is of the order of $8$cm in
lateral direction and $-0.4^{\circ}$ in heading direction. Again, the SRPT
approach performed better in F-region too.

##  VI Conclusion

This article formulates an EKF-based state-estimator, that employs a minimal
set of sensors, to provide estimations of vehicle state to SRPT vehicle
teleoperation (previously conceptualized and assessed in [31, 1]). To test the
performance of state-estimation and SRPT with state-estimation, some worst-
case conditions are considered. The worst-case conditions consist of
aggressive maneuvers, strong environmental disturbances, and various
measurement noises. EKF state estimations are found to match actual states.
Vehicle teleoperation performances of ’SRPT with actual states’ and ’SRPT with
estimated states’ are found to be close to each other in all worst-case
conditions. For sake of completeness, performance of a simple look-ahead
driver is also incorporated into the analysis. SRPT teleoperation with or
without EKF, performs better than the look-ahead driver model in aggressive
manoeuvres, such as double-lane change and slalom. In general, SRPT modes make
vehicle teleoperation unsusceptible to network delay and its variability.

Due to measurement model limitations, states that correspond to vehicle pose
$[x,y,\psi]$ diverge with time, as they are unobservable. Despite this fact,
SRPT with EKF performs well. This is due to the fact that the divergence of
these states is negligible for a time window equal to the round-trip network
delay (of $300ms$). Hence, no exteroceptive sensor (such as GPS) is required
to explicitly measure the vehicle’s global pose. Being independent of GPS
sensors, SRPT teleoperation becomes unaffected to problems such as unreliable
GPS signals in presence of trees, tall buildings, ionospheric storms, tunnel,
etc.

## References

  * [1] J. Prakash, M. Vignati, E. Sabbioni, and F. Cheli, “Vehicle teleoperation: Human in the loop performance comparison of smith predictor with novel successive reference-pose tracking approach,” _Sensors_ , vol. 22, no. 23, 2022.
  * [2] D. Majstorovic, S. Hoffmann, F. Pfab, A. Schimpe, M.-M. Wolf, and F. Diermeyer, “Survey on teleoperation concepts for automated vehicles,” 2022. [Online]. Available: https://arxiv.org/abs/2208.08876
  * [3] J. Y. C. Chen, E. C. Haas, and M. J. Barnes, “Human performance issues and user interface design for teleoperated robots,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , vol. 37, no. 6, pp. 1231–1245, 2007.
  * [4] J. Storms and D. Tilbury, “Equating user performance among communication latency distributions and simulation fidelities for a teleoperated mobile robot,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 4440–4445.
  * [5] J. P. Luck, P. L. McDermott, L. Allender, and D. C. Russell, “An investigation of real world control of robotic assets under communication latency,” in _Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction_ , ser. HRI ’06. New York, NY, USA: Association for Computing Machinery, 2006, p. 202–209.
  * [6] T. B. Sheridan, “Space Teleoperation Through Time Delay: Review and Prognosis,” _IEEE Transactions on Robotics and Automation_ , 1993.
  * [7] D. J. Gorsich, P. Jayakumar, M. P. Cole, C. M. Crean, A. Jain, and T. Ersal, “Evaluating mobility performance of unmanned ground vehicles,” US ARMY TARDEC WARREN United States, Tech. Rep., 2018.
  * [8] J.-M. Georg, J. Feiler, S. Hoffmann, and F. Diermeyer, “Sensor and actuator latency during teleoperation of automated vehicles,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 2020, pp. 760–766.
  * [9] C. Mutzenich, S. Durant, S. Helman, and P. Dalton, “Updating our understanding of situation awareness in relation to remote operators of autonomous vehicles,” _Cognitive Research: Principles and Implications_ , vol. 6, no. 1, Feb. 2021. [Online]. Available: https://doi.org/10.1186/s41235-021-00271-8
  * [10] S. Hoffmann and F. Diermeyer, “Systems-theoretic safety assessment of teleoperated road vehicles,” in _Proceedings of the 7th International Conference on Vehicle Technology and Intelligent Transport Systems - Volume 1: VEHITS,_ , INSTICC. SciTePress, 2021, pp. 446–456.
  * [11] F. E. Chucholowski, “Evaluation of display methods for teleoperation of road vehicles,” _Journal of Unmanned System Technology_ , vol. 3, no. 3, pp. 80–85, 2016.
  * [12] T. Tang, P. Vetter, S. Finkl, K. Figel, and M. Lienkamp, “Teleoperated road vehicles – the ”free corridor” as a safety strategy approach,” _Applied Mechanics and Materials_ , vol. 490-491, pp. 1399–1409, Jan. 2014\. [Online]. Available: https://doi.org/10.4028/www.scientific.net/amm.490-491.1399
  * [13] G. Graf, Y. Abdelrahman, H. Xu, Y. Abdrabou, D. Schitz, H. Hußmann, and F. Alt, “The Predictive Corridor: A Virtual Augmented Driving Assistance System for Teleoperated Autonomous Vehicles,” in _ICAT-EGVE 2020 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments_ , F. Argelaguet, R. McMahan, and M. Sugimoto, Eds. The Eurographics Association, 2020.
  * [14] J.-M. Georg and F. Diermeyer, “An adaptable and immersive real time interface for resolving system limitations of automated vehicles with teleoperation,” in _2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)_. IEEE Press, 2019, p. 2659–2664. [Online]. Available: https://doi.org/10.1109/SMC.2019.8914306
  * [15] J.-M. Georg, E. Putz, and F. Diermeyer, “Longtime effects of videoquality, videocanvases and displays on situation awareness during teleoperation of automated vehicles¡sup¿*¡/sup¿,” in _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE Press, 2020, p. 248–255. [Online]. Available: https://doi.org/10.1109/SMC42975.2020.9283364
  * [16] J. Prakash, M. Vignati, E. Sabbioni, and F. Cheli, “Predictive display with perspective projection of surroundings in vehicle teleoperation to account time-delays,” _IEEE Transactions on Intelligent Transportation Systems_ , 2023.
  * [17] S. Saparia, A. Schimpe, and L. Ferranti, “Active safety system for semi-autonomous teleoperated vehicles,” in _2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops)_ , 2021, pp. 141–147.
  * [18] S. J. Anderson, S. B. Karumanchi, K. Iagnemma, and J. M. Walker, “The intelligent copilot: A constraint-based approach to shared-adaptive control of ground vehicles,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 5, no. 2, pp. 45–54, 2013.
  * [19] A. Schimpe and F. Diermeyer, “Steer with me: A predictive, potential field-based control approach for semi-autonomous, teleoperated road vehicles,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_ , 2020, pp. 1–6.
  * [20] B. Qiao, H. Li, and X. Wu, “Intelligent-assist algorithm for remote shared-control driving based on game theory,” _Journal of Shanghai Jiaotong University (Science)_ , vol. 26, no. 5, pp. 615–625, Oct. 2021. [Online]. Available: https://doi.org/10.1007/s12204-021-2351-z
  * [21] D. Schitz, G. Graf, D. Rieth, and H. Aschemann, “Model-predictive cruise control for direct teleoperated driving tasks,” in _2021 European Control Conference (ECC)_ , 2021, pp. 1808–1813.
  * [22] J. Storms, K. Chen, and D. Tilbury, “A shared control method for obstacle avoidance with mobile robots and its interaction with communication delay,” _The International Journal of Robotics Research_ , vol. 36, no. 5-7, pp. 820–839, 2017. [Online]. Available: https://doi.org/10.1177/0278364917693690
  * [23] S. Gnatzig, F. Schuller, and M. Lienkamp, “Human-machine interaction as key technology for driverless driving - a trajectory-based shared autonomy control approach,” in _2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication_ , 2012, pp. 913–918.
  * [24] S. Hoffmann, D. Majstorović, and F. Diermeyer, “Safe corridor: A trajectory-based safety concept for teleoperated road vehicles,” in _2022 International Conference on Connected Vehicle and Expo (ICCVE)_ , 2022, pp. 1–6.
  * [25] J. S. Kay and C. E. Thorpe, “Operator interface design issues in a low-bandwidth and high-latency vehicle teleoperation system,” in _SAE Technical Paper Series_. SAE International, Jul. 1995. [Online]. Available: https://doi.org/10.4271/951485
  * [26] A. Björnberg, “Shared control for vehicle teleoperation with a virtual environment interface,” Master’s thesis, KTH, School of Electrical Engineering and Computer Science (EECS), 2020.
  * [27] D. Schitz, G. Graf, D. Rieth, and H. Aschemann, “Interactive corridor-based path planning for teleoperated driving,” in _2021 7th International Conference on Mechatronics and Robotics Engineering (ICMRE)_. IEEE, Feb. 2021. [Online]. Available: https://doi.org/10.1109/icmre51691.2021.9384848
  * [28] A. Hosseini, T. Wiedemann, and M. Lienkamp, “Interactive path planning for teleoperated road vehicles in urban environments,” in _17th International IEEE Conference on Intelligent Transportation Systems (ITSC)_ , 2014, pp. 400–405.
  * [29] D. Schitz, S. Bao, D. Rieth, and H. Aschemann, “Shared autonomy for teleoperated driving: A real-time interactive path planning approach,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 999–1004.
  * [30] J. Feiler and F. Diermeyer, “The perception modification concept to free the path of an automated vehicle remotely,” in _Proceedings of the 7th International Conference on Vehicle Technology and Intelligent Transport Systems_. SCITEPRESS - Science and Technology Publications, 2021. [Online]. Available: https://doi.org/10.5220/0010433304050412
  * [31] J. Prakash, M. Vignati, E. Sabbioni, and F. Cheli, “Vehicle teleoperation: Successive reference-pose tracking to improve path tracking and to reduce time-delay induced instability,” in _2022 IEEE Vehicle Power and Propulsion Conference (VPPC)_ , 2022, pp. 1–8.
  * [32] M. Bersani, S. Mentasti, P. Dahal, S. Arrigoni, M. Vignati, F. Cheli, and M. Matteucci, “An integrated algorithm for ego-vehicle and obstacles state estimation for autonomous driving,” _Robotics and Autonomous Systems_ , vol. 139, p. 103662, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0921889020305029
  * [33] M. Doumiati, A. C. Victorino, A. Charara, and D. Lechner, “Onboard real-time estimation of vehicle lateral tire–road forces and sideslip angle,” _IEEE/ASME Transactions on Mechatronics_ , vol. 16, no. 4, pp. 601–614, 2011\.
  * [34] S. Cheng, L. Li, B. Yan, C. Liu, X. Wang, and J. Fang, “Simultaneous estimation of tire side-slip angle and lateral tire force for vehicle lateral stability control,” _Mechanical Systems and Signal Processing_ , vol. 132, pp. 168–182, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0888327019304054
  * [35] Y. Zheng, M. J. Brudnak, P. Jayakumar, J. L. Stein, and T. Ersal, “Evaluation of a predictor-based framework in high-speed teleoperated military ugvs,” _IEEE Transactions on Human-Machine Systems_ , vol. 50, no. 6, pp. 561–572, 2020.
  * [36] C. Baker, “Unsteady wind loading on a wall,” _Wind and Structures_ , vol. 4, pp. 413–440, 10 2001.
  * [37] Chris Baker. Modelling of extreme wind gusts. [Online]. Available: https://profchrisbaker.com/2020/09/05/modelling-of-extreme-wind-gusts/
  * [38] J. Park and P. E. Nikravesh, “A look-ahead driver model for autonomous cruising on highways,” _SAE Transactions_ , vol. 105, pp. 1569–1577, 1996\.


# Synthetic Test Collections for Retrieval Evaluation

Hossein A. Rahmani  0000-0002-2779-4942 University College LondonLondonUK
<EMAIL_ADDRESS>,  Nick Craswell  0000-0002-9351-8137
MicrosoftBellevueUS<EMAIL_ADDRESS>,  Emine Yilmaz  0000-0003-4734-4532
University College London & AmazonLondonUK<EMAIL_ADDRESS>,  Bhaskar
Mitra  0000-0002-5270-5550 MicrosoftMontréalCanada<EMAIL_ADDRESS>0000-0002-5138-8426 and  Daniel Campos  SnowflakeNew YorkUS
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Test collections play a vital role in evaluation of information retrieval (IR)
systems. Obtaining a diverse set of user queries for test collection
construction can be challenging, and acquiring relevance judgments, which
indicate the appropriateness of retrieved documents to a query, is often
costly and resource-intensive. Generating synthetic datasets using Large
Language Models (LLMs) has recently gained significant attention in various
applications. In IR, while previous work exploited the capabilities of LLMs to
generate synthetic queries or documents to augment training data and improve
the performance of ranking models, using LLMs for constructing synthetic test
collections is relatively unexplored. Previous studies demonstrate that LLMs
have the potential to generate synthetic relevance judgments for use in the
evaluation of IR systems. In this paper, we comprehensively investigate
whether it is possible to use LLMs to construct fully synthetic test
collections by generating not only synthetic judgments but also synthetic
queries. In particular, we analyse whether it is possible to construct
reliable synthetic test collections and the potential risks of bias such test
collections may exhibit towards LLM-based models. Our experiments indicate
that using LLMs it is possible to construct synthetic test collections that
can reliably be used for retrieval evaluation.

github https://github.com/rahmanidashti/SyntheticTestCollections

Synthetic Data Generation, Large Language Model, Test Collection

††journalyear: 2024††copyright: rightsretained††doi:
10.1145/3626772.3657942††conference: Proceedings of the 47th International ACM
SIGIR Conference on Research and Development in Information Retrieval; July
14–18, 2024; Washington, DC, USA††booktitle: Proceedings of the 47th
International ACM SIGIR Conference on Research and Development in Information
Retrieval (SIGIR ’24), July 14–18, 2024, Washington, DC, USA††isbn:
979-8-4007-0431-4/24/07††ccs: Information systems Information retrieval

##  1\. Introduction

Test collection construction is a pivotal process to evaluate the
effectiveness of information retrieval (IR) systems. The most widely used
approach for constructing test collections is based on the Cranfield paradigm
(Cleverdon, 1967; Aslam et al., 2006), which involves creating a collection
comprising queries and associated relevance judgments. Queries used in test
collection construction are expected to come from real usage logs,
representing real information needs. However, it is very difficult to get
access to such logs outside of search engine companies. Hence, lots of
existing test collections used in IR are based on manually created queries
(Craswell et al., 2021; Yilmaz et al., 2008). This process demands time and
expertise, making it costly in terms of labor and resources; furthermore,
there are no guarantees that queries generated at the end of this process are
representative of real information needs. Similarly, obtaining relevance
judgments for a query is an expensive procedure requiring significant human
effort. The highly demanding nature of the test construction process is a
major bottleneck in constructing large test collections; hence, most existing
publicly available test collections tend to consist of a small number of
queries, which could degrade the reliability of these collections.

Recently, Large Language Models (LLMs) have demonstrated remarkable
performance on unseen tasks by only considering the instructions (so-called
‘prompts’) provided to them (Chiang and Lee, 2023; Jiang et al., 2022).
Synthetic datasets generated using LLMs have recently gained attention across
a range of diverse tasks (Li et al., 2023; Zhang et al., 2018; Bao et al.,
2023). In IR, previous studies use LLMs to generate synthetic training data
for augmentation to boost the quality of retrievers (Askari et al., 2023;
Bonifacio et al., 2022). LLMs have also been used to generate relevance labels
(Thomas et al., 2023; Faggioli et al., 2023), as well as to generate query
variants for evaluation and training of IR systems (Rajapakse and de Rijke,
2023; Alaofi et al., 2023). However, to the best of our knowledge, no prior
study has thoroughly explored the potential of LLMs to construct fully
synthetic test collections where both queries and associated relevance
judgments are automatically generated using LLMs.

Given the aforementioned challenges in constructing large-scale test
collections, our goal in this paper is to investigate whether it is possible
to create reliable synthetic test collections so that there is (i) no need for
real usage query logs or manual creation of queries, and (ii) no need to
obtain manual relevance judgments. We investigate different approaches to
construct synthetic test collections using LLMs and show using synthetic test
collections it is possible to obtain evaluation results that are similar to
results obtained using real test collections.

Figure 1. Scatter plot of the effectiveness (i.e., NDCG@10) of TREC DL 2023
runs according to the real queries and synthetic queries with human judgments.
A point represents a single run averaged over all queries.

##  2\. Synthetic Test Collection

Table 1. Statistics of queries per query type

All Real T5 Generated GPT-4 Generated No. of Queries 82 51 13 18 Avg. Query
Length 6.84 5.76 5.69 10.72 Min Query Length 2 2 4 6 Max Query Length 15 14 8
15

###  2.1. Synthetic Query Generation

The reliability of a test collection highly depends on the queries included in
the collection. Since obtaining real user queries may not always be possible,
we analyzed the feasibility of using synthetic queries in test collection
construction, focusing on constructing test collections for passage retrieval
task.

Our goal is to create meaningful queries, which can be characterized as a
collection of terms that can be used to reach a document and make sense alone
without any further context from the document from which the query is derived.
The approach we followed is based on starting with a passage and generating a
query to which this passage would be relevant.

To generate the synthetic queries, we first sampled $1000$ passages at random
from the MS MARCO v2 passage
corpus111https://msmarco.blob.core.windows.net/msmarcoranking/msmarco_v2_passage.tar
as anchor documents. The selected passages are then go through a filtering
step whose goal is to get rid of “bad” passages, which can lead to low quality
queries being generated. Examples of bad passages include passages that talk
about a person without saying the person’s name (“she then completed her first
novel…”) or passages that use terms that might not be clear once the passage
is removed from its document context (“we eventually chose the second
solution, since it gave a good balance of accuracy and efficiency”). To
identify passages that might not be good stand-alone search results, we ran
each of the $1000$ passages through a GPT-4 prompt, which generates a query
independent passage quality score. Since this is a rough one-off filter, we
did not repeat it in the case of some error or malformed output, which
eliminated $8.9\%$ of passages. We filtered low-quality passages with passage
quality scores less than $50$, removing $14.6\%$ passages. The remaining
passages were seeds for generating synthetic queries.

We then generated queries using two methods: The first method uses a small
pre-trained model based on T5 (Raffel et al., 2020), and the second method is
based on zero-shot query generation using GPT-4 (Achiam et al., 2023). For
query generation using T5, we applied the BeIR query generation
model222https://huggingface.co/BeIR/query-gen-msmarco-t5-large-v1 that uses a
T5-based model pre-trained on the MS MARCO Passage dataset (Thakur et al.,
2021). We then generated one query per passage using both T5 and GPT-4
approaches. The details of the approaches can be found in our GitHub.

Table 2. Average number of documents per query for each relevance grade for
different query types.

Relevance Grade All Real T5 Generated GPT-4 Generated Nonrelevant (0) 169.09
159.31 213.30 164.88 Related (1) 53.31 64.15 31.23 38.55 Highly relevant (2)
27.54 31.60 19.46 21.88 Perfectly relevant (3) 22.31 24.35 21.07 17.44

Then, we asked experts who are professional assessors with very good
experience in relevance annotation to remove the queries that did not look
reasonable, that contained too few or too many relevant documents as these
queries tend to be noisy or not very informative for evaluation purposes.
Amongst $48$ T5-generated queries, $13$ of them were selected and amongst $49$
queries generated using GPT-4, $18$ of them were selected to be included in
the test collection. Queries generated as a result of this process were
included in the test collections created as part of the TREC Deep Learning
Track (TREC DL) 2023 (Craswell et al., 2024), which consist of $51$ real
queries in addition to the synthetic queries. Systems submitted to the DL
Track were run on both synthetic and real queries in the test collection. For
all query types, depth-10 pooling was used to select the documents to be
judged by the expert assessors who labelled the documents based on four
relevance grades (irrelevant, related, highly relevant and perfectly
relevant).

Table 1 shows the total number of queries included in the TREC DL 2023 test
collection for each query type, together with the average, min and max query
length for each category. We also included statistics about real queries for a
better comparison of synthetic queries with human queries. It can be seen that
queries generated using GPT-4 tend to be much longer than the queries
generated using T5 and the real queries and in general, queries generated
using T5 tend to be shorter than the other query types. For each query type,
Table 2 shows the average number of documents per query for each relevance
grade. It can be seen that synthetic queries contain much fewer relevant
documents (documents with a relevance grade greater than zero) compared to the
real queries ($120.1$ documents of relevance grade greater than zero for real
queries vs. $71.76$ documents for queries generated by T5 and $77.87$
documents for queries generated by GPT-4.) Also, pools constructed using
queries generated by T5 tend to contain significantly more non-relevant
documents compared to the rest of the query types, suggesting that these
queries may be more difficult than the other queries.

To compute the reliability of the synthetic queries, we compared the
performance of systems that were submitted to TREC DL 2023 on real queries
with the performance of systems solely on synthetic queries. For this purpose,
we evaluate the quality of the $31$ systems submitted to the full ranking task
of the track using official judgments of the track obtained from expert human
assessors from NIST, and compare the ranking of these systems on real queries
and synthetic queries. Figure 1 shows how the performance of systems using
synthetically generated queries (both T5 and GPT-4 based) compare with system
performance on real queries. The figure also includes the line $y=x$ for easy
comparison. It can be seen that synthetic queries and real queries show
similar patterns in terms of evaluation results and system ranking, with a
system ordering agreement of Kendall’s $\tau=0.8151$.

###  2.2. Synthetic Relevance Judgment Generation

Once a sample of queries to be included in a test collection has been
identified, the second step is to identify documents relevant to these
queries, a process that is usually done by collecting manual annotations.
While in the previous section we assumed that manual annotations are available
for all query types (including synthetic queries), obtaining manual
annotations is a very expensive process and it may not always be possible to
get these many judgments from expert annotators. Hence, we next investigate
the possibility of generating synthetic relevance judgments for test
collections constructed using synthetic queries.

As the simplest method, one can assume that the passages that we used to
generate the synthetic queries are the only passages that are relevant to
those queries, referred to as sparse judgments. Figure 2a shows the system
ranking and Kendall’s $\tau$ correlation on real queries with human judgments
and synthetic queries with sparse judgments. Compared to synthetic queries
with human judgments ($\tau=0.8151$), evaluation using synthetic queries with
sparse judgments does not give similar results to human queries, with a very
low system ordering agreement of $\tau=0.157$.

(a)  (b)

Figure 2. Scatter plots of the effectiveness (i.e., NDCG@10) of TREC DL 2023
run according to the real queries with human judgments and our synthetic
queries with (a) sparse judgments and (b) synthetic judgments. Table 3.
Judgment agreement on TREC DL 2023 between TREC assessors and the LLM (i.e.,
GPT-4) on real queries and synthetic queries.

GPT-4 TREC DL 2023 Assessments Cohen’s $\kappa$ Prediction Perfect rel. High.
rel. Related Irrelevant Real Perfect. rel. 597 469 496 324 0.24 High. rel. 322
473 648 501 Related 298 548 1358 2736 Irrelevant 25 122 770 4564 Synthetic
Perfect. rel. 166 92 40 150 0.26 High. rel. 227 188 197 305 Related 170 301
695 1871 Irrelevant 25 66 168 3415

Recent work (Thomas et al., 2023) showed that it is possible to get very high-
quality relevance judgements using LLMs. Thus, the question we investigate
further is can we generate a fully synthetic test collection by not only
generating synthetic queries but also by generating synthetic relevance
judgements? To answer this question, we used the GPT-4 language model as
accessed through Open AI’s API in order to automatically label the documents
(that were originally annotated using human annotators) for the synthetic
queries to generate synthetic relevance judgements. For this purpose, we used
the prompt template introduced in the recent study (Thomas et al., 2023) that
shows the possibility of getting high-quality relevance judgements using LLMs.
In our experiments the temperature was set at zero, so the model would select
the single most likely output and other parameters of the model were top $p$ =
1, frequency penalty 0.5, and presence penalty 0.

(a) Real vs. Synthetic queries (b) Real vs. GPT-4 queries (c) Real vs. T5
queries

Figure 3. Scatter plots of the effectiveness of TREC DL 2023 runs based on
synthetic vs. real test collections to analyse the bias towards systems using
the same language model as the one used in synthetic test collection
construction.

Table 3 shows how the judgments generated using GPT-4 compare with manual
judgments. The table reports the “agreement” on the full 4-point relevance
scale on real and synthetic queries, respectively. For both real and synthetic
queries, we observe a fair level of agreement between synthetic judgements
generated using GPT-4 and manual judgments: The Cohen’s $\kappa$ on real
queries is $0.24$ and on synthetic queries is $0.26$. In earlier studies,
Faggioli et al. (Faggioli et al., 2023) report a Cohen’s $\kappa$ of $0.26$
between TREC DL 2021 assessors and GPT-3.5 with two types of human judgments,
and Thomas et al. (Thomas et al., 2023) show that Cohen’s $\kappa$ varies from
$0.20$ to $0.64$ on two-level scale of TREC Robust data based on different
versions of prompts.

The generated judgments on synthetic queries show that GPT-4 consistently
underestimates ‘Perfectly relevant’ and ‘Highly relevant’ labels. On real
queries, GPT-4 labels on ‘Perfectly relevant’ are more likely to be labelled
as the same but on synthetic queries, according to GPT-4, a ‘Perfectly
relevant’ document is more likely to be rated as a ‘Highly relevant’ document.
For documents judged as ‘Perfectly relevant’ by human assessors, GPT-4
generates the same judgment in 28% of the cases. The results indicate that
human assessors may use more detailed information to distinguish between
‘Perfectly relevant’ from ‘Highly relevant’ documents that are not fully
captured by an LLM. We notice a similar pattern when we compare the ‘Highly
relevant’ and ‘Related’ documents on both real and synthetic queries.

One of the primary uses of test collections is for system evaluation. Even
though the synthetic judgments generated could be slightly different than the
actual judgments, they may still evaluate systems in a similar way. Figure 2b
shows how the ordering of system on real queries compare with system ordering
on our fully synthetic test collection (synthetic queries + synthetic
judgments). It can be seen that evaluation on the fully synthetic test
collection results in similar results to human queries with human judgments in
terms of system ordering, with a Kendall’s $\tau$ value of $0.8568$. For
comparison, Faggioli et al. (Faggioli et al., 2023) report Kendall’s
$\tau=0.86$ for NDCG@10 on a similar experiment where only judgments were
synthetically generated using GPT-3.5. Interestingly, compared with the system
rankings using the human judgments on synthetic queries (see Figure 1), using
the synthetic judgments generated with LLMs on synthetic queries result in
higher correlations and Kendall’s $\tau$ values as shown in Figure 2b.

##  3\. Analysis of Bias in System Evaluation

One potential issue with using synthetic queries and judgements in test
collection construction is the possible bias these collections may exhibit
towards systems that are based on a similar approach (similar language model)
to the one that was used in the synthetic test collection construction process
(e.g., synthetic test collections constructed using T5 might favour systems
that are based on T5).

In order to analyse the possible bias, we categorised the runs submitted to
TREC DL 2023 based on the approach they use333To this end, we carefully
analysed the metadata file of submissions. GPT models are GPT-4 or GPT-3.5 and
T5 models include MonoT5, FlanT5, and RankT5. (i.e., language models used in
their ranking or retrieval pipeline), resulting in four different system
categories: systems based on GPT ($\times$), T5 ($|$), GPT + T5 (+) (i.e., a
combination of GPT and T5), and others ($\medbullet$) (i.e., traditional
methods such as BM25, or any model that does not use either GPT or T5). Figure
3a shows that the synthetic test collection we have constructed that contains
synthetic queries generated by LLMs (T5 and GPT-4) exhibits little to no bias
towards LLM-based systems.

To further analyse possible bias that might arise from a system using a
similar language model as the one used in test collection construction, Figure
3b shows how system performance computed on synthetic test collections
generated using queries generated by GPT-4 compare with system performance on
real test collections. It can be seen that synthetic test collections based on
GPT-4 do not systematically overestimate the performance of systems based on
GPT. Similarly, as can be seen in Figure 3c, synthetic test collections with
queries generated using T5 exhibit almost no bias towards systems based on T5.
Similar results were obtained on other evaluation metrics such as average
precision and NDCG@100, results for which are omitted due to space
limitations. However, all three plots in the figure show that for all system
types, systems consistently achieve higher performance on synthetic test
collections when compared to real queries, suggesting that synthetic test
collections tend to be easier than real queries and they tend to overestimate
system performance across all system types.

Although previous studies (Liu et al., 2023) on LLM evaluation discussed the
potential bias towards LLM-generated text when we use LLMs for evaluation, in
our experiments we did not observe a very clear evidence of systematic bias,
where runs using GPT-4 were favored when evaluated using synthetic GPT-4
queries, or where runs using T5 were favored when evaluated on synthetic T5
queries. While our results look encouraging, we would like to emphasize that
our results are based on one test collection we have constructed and further
experiments are needed to analyse potential biases that might arise from using
a fully synthetic test collection.

##  4\. Conclusion and Future Work

In this paper, we explored the construction of fully synthetic test
collections using LLMs. Overall, our analysis suggests that by using fully
synthetic test collections consisting of synthetically generated queries and
judgments, it is possible to obtain evaluation results that are similar to
evaluation results obtained using the traditional test collection approach.
Future studies can examine more advanced prompting methods and different LLMs
to compare with the test collection we have created.

Exploring the potential biases that may arise from generating a fully
synthetic test collection is crucial for ensuring the quality, fairness, and
reliability of the test collections. While our preliminary analysis showed
that the synthetic test collections we have generated using a particular LLM
do exhibit little to no bias towards systems based on the same LLM, further
research is needed to provide a deeper understanding of the potential biases
and to develop appropriate mitigation strategies.

###### Acknowledgements.

This research is supported by the Engineering and Physical Sciences Research
Council [EP/S021566/1] and the EPSRC Fellowship titled “Task Based Information
Retrieval” [EP/P024289/1].

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Alaofi et al. (2023) Marwah Alaofi, Luke Gallagher, Mark Sanderson, Falk Scholer, and Paul Thomas. 2023.  Can generative llms create query variants for test collections? an exploratory study. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1869–1873.
  * Askari et al. (2023) Arian Askari, Mohammad Aliannejadi, Chuan Meng, Evangelos Kanoulas, and Suzan Verberne. 2023.  Expand, Highlight, Generate: RL-driven Document Generation for Passage Reranking. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_. 10087–10099.
  * Aslam et al. (2006) Javed A Aslam, Virgil Pavlu, and Emine Yilmaz. 2006.  A statistical method for system evaluation using incomplete judgments. In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_. 541–548.
  * Bao et al. (2023) Jianzhu Bao, Rui Wang, Yasheng Wang, Aixin Sun, Yitong Li, Fei Mi, and Ruifeng Xu. 2023.  A synthetic data generation framework for grounded dialogues. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 10866–10882.
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  Inpars: Unsupervised dataset generation for information retrieval. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2387–2392.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can Large Language Models Be an Alternative to Human Evaluations?  _arXiv preprint arXiv:2305.01937_ (2023).
  * Cleverdon (1967) Cyril Cleverdon. 1967.  The Cranfield tests on index language devices. In _Aslib proceedings_ , Vol. 19. MCB UP Ltd, 173–194.
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M Voorhees, and Ian Soboroff. 2021.  TREC deep learning track: Reusable test collections in the large data regime. In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_. 2369–2375.
  * Craswell et al. (2024) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2024.  Overview of the TREC 2023 Deep Learning Track. In _Text REtrieval Conference (TREC)_. NIST, TREC.  https://www.microsoft.com/en-us/research/publication/overview-of-the-trec-2023-deep-learning-track/
  * Faggioli et al. (2023) Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023\.  Perspectives on large language models for relevance judgment. In _Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval_. 39–50.
  * Jiang et al. (2022) Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022.  Promptmaker: Prompt-based prototyping with large language models. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. 1–8.
  * Li et al. (2023) Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023.  Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations.  _arXiv preprint arXiv:2310.07849_ (2023).
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_ (2023).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ 21, 140 (2020), 1–67.
  * Rajapakse and de Rijke (2023) Thilina C Rajapakse and Maarten de Rijke. 2023.  Improving the Generalizability of the Dense Passage Retriever Using Generated Datasets. In _European Conference on Information Retrieval_. Springer, 94–109.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _arXiv preprint arXiv:2104.08663_ (2021).
  * Thomas et al. (2023) Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023.  Large language models can accurately predict searcher preferences.  _arXiv preprint arXiv:2309.10621_ (2023).
  * Yilmaz et al. (2008) Emine Yilmaz, Evangelos Kanoulas, and Javed A Aslam. 2008.  A simple and efficient sampling method for estimating AP and NDCG. In _Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval_. 603–610.
  * Zhang et al. (2018) Lichao Zhang, Abel Gonzalez-Garcia, Joost Van De Weijer, Martin Danelljan, and Fahad Shahbaz Khan. 2018.  Synthetic data generation for end-to-end thermal infrared tracking.  _IEEE Transactions on Image Processing_ 28, 4 (2018), 1837–1850.