# An implementation of ROS Autonomous Navigation on Parallax Eddie platform

1st Hafiq Anas  School of Digital Science
Universiti Brunei Darussalam
Jalan Tungku Link, Brunei
<EMAIL_ADDRESS>2nd Ong Wee Hong  School of Digital Science
Universiti Brunei Darussalam
Jalan Tungku Link, Brunei
<EMAIL_ADDRESS>
###### Abstract

This paper presents an implementation of autonomous navigation functionality
based on Robot Operating System (ROS) on a wheeled differential drive mobile
platform called Eddie robot. ROS is a framework that contains many reusable
software stacks as well as visualization and debugging tools that provides an
ideal environment for any robotic project development. The main contribution
of this paper is the description of the customized hardware and software
system setup of Eddie robot to work with an autonomous navigation system in
ROS called Navigation Stack and to implement one application use case for
autonomous navigation. For this paper, photo taking is chosen to demonstrate a
use case of the mobile robot.

###### Index Terms:

Autonomous Mobile Robot; ROS; Navigation; Odometry; Kinect RGB-D

##  I Introduction

The implementation of autonomous navigation based on Robot Operating System
(ROS) on a differential drive mobile platform requires solving problems in
robot localization, mapping, and path planning. The usage of ROS in robotic
development is becoming increasingly popular due to the robustness and
flexibility of the framework. ROS helps to ease the development of any robotic
project due to the powerful features provided such as hardware abstraction,
message-passing between processes, package management, debugging tools and the
availability of packages for various functions of a robotic system [1]. While
the desired functionality of autonomous navigation is provided by ROS
Navigation stack, successful implementation on an unsupported mobile platform
would require proper setup of the stack’s requirements and understanding of
its fundamental concepts [1][2].

A requirement to achieve autonomous navigation is to solve the problem of
Simultaneous Localization And Mapping (SLAM) [3]. SLAM is concerned with
building a map based on what the robot perceives and localizes itself within
that built map [4]. Gmapping algorithm is used for the ROS Navigation Stack’s
SLAM functionality [5]. This is a good option since a good map is able to be
generated from SLAM without using too much computational resources.

Generating a suitable traverse path for a mobile robot requires a map of the
environment the robot is working in and the generated path should work in such
a way that the robot is able to go from one arbitrary position to another
without colliding with any obstacles [6]. A* algorithm is used for ROS
Navigation Stack’s Global planner functionality. This approach should enable
the Navigation Stack to create an optimal path trajectory for the robot to
reach its target location in a shorter period of time.

Odometry is one of the most common localization solutions for Wheeled Mobile
Robots (WMRs). The encoders mounted on the wheels can determine a robot’s
movement in an environment based on the measured amount and direction of wheel
rotation. Eddie robot is equipped with two-wheel encoders [7]. The
implementation of the odometry system is based on interpreting the information
from the wheel encoders of the robot to give the robot’s position and
orientation information for localization.

SLAM algorithms perform well when a robotic platform is equipped with a high
resolution scanning laser rangefinder. However, the cost of such a sensor may
not be accessible when the budget is limited. For example, the price of a
Hokuyo Laser Sensor and LIDAR sensor costs about $2,000 and $75,000
respectively [8]. An alternative solution is to use an RGB-D sensor like
Microsoft Kinect [8] that costs around $100 which can emulate a laser scan
sensor by using some packages provided in ROS [9]. Due to its accessibility,
Kinect is one of the most commonly used RGB-D sensors for robotic platforms
and produces an acceptable map with ROS packages.

In our hardware system setup, we have replaced the original Eddie Control
Board with a Raspberry Pi 3, added a Microsoft Kinect RGB-D sensor, a pair of
Parallax HB-25 motor controllers, two LM2596 step-down modules, and three
14.4v Lithium Polymer (LiPo) batteries. Further details including the circuit
diagram and hardware system diagram will be provided in the next section.

For our software system setup, we have implemented various custom ROS packages
to complete the requirements of the Navigation Stack, and this included nodes
to process encoder information for odometry localization and nodes to
interface between the robot’s motor hardware and Navigation Stack. In
addition, we have implemented a node that does face detection with an existing
Haar Cascade classifier and a node that handles the behavior of the robot
which is to detect a person using the aforementioned face detection node and
subsequently traverse toward that person to take a photo.

##  II System overview

###  II-A Hardware framework

The implementation is carried out on a differential drive wheeled mobile robot
platform made by Parallax company, called Eddie robot as shown in Figure 1. A
Raspberry Pi 3 Model B computer and a notebook computer is installed on the
robot to run the software system. Additionally, a remote computer is used as
the user interface to send instructions to the robot. Table I shows the full
hardware list of the modified Eddie robot.

Figure 1: Eddie: A differential drive mobile robot TABLE I: Eddie hardware specification Hardware | Details
---|---
Computer 1 | Raspberry Pi 3 Model B (Added)
Computer 2 | Lenovo ThinkPad X230 (Added)
Remote computer | Acer Aspire E5 (Added)
Motor controller | 2 * Parallax HB-25 (Added)
Wheel motor | 2 * Parallax motor #27971
Wheel encoder | 2 * Parallax encoder #29321
RGB-D sensor | Microsoft Kinect v1 (Added)
Step down module | 2 * LM2596 module (5V, 12V) (Added)
Power supply | 3 * 14.4v 5000mAH LiPo (Added)

####  II-A1 Eddie robot system hardware setup

Figure 2 shows the block diagram of the hardware setup of the robot. Several
modifications have been done on the Eddie robot to implement ROS Navigation
Stack. The existing control board has been replaced by a Raspberry Pi 3 Model
B computer which is used to interface with the sensors and motors. An aluminum
frame is installed on the platform with a Microsoft Kinect v1 fitted on top
and connected to a notebook computer through a USB port. Due to the low
computational power of Raspberry Pi, a notebook computer is used to run the
more computationally demanding functionality such as SLAM and navigation.
Additionally, this notebook computer communicates with the Raspberry Pi
through a direct LAN link. A remote computer is connected to the robot through
SSH over WLAN and is used to run a visualization software called RVIZ to
monitor operations and send motor commands. The hardware specifications of all
the computers in this setup is shown in Table II.

Figure 2: Hardware system of Eddie robot

The wiring diagram of the hardware setup is shown in Figure 3. The Raspberry
Pi is connected to all GPIO supported hardware modules (i.e wheel encoders and
motor controllers), and the DC Power ports (i.e 3V, 5V) are used to supply
voltage for these modules. Since the Raspberry Pi and Kinect require different
voltages, two 14.4V Lithium Polymer (LiPo) batteries are connected in series
and a step-down module is used to lower down the combined voltage of 28.8V to
5.1V and 12V for the two devices respectively. In addition, another identical
LiPo battery is used to supply voltage for the wheel motors. Different
batteries are used to supply the electronic circuits and the motors to avoid
the running motors to interfere with the voltage of the electronic circuits.
The Kinect is connected to the Notebook instead of the Raspberry Pi since the
Notebook has a higher computational capability and a LAN cable is connected to
allow communications between the two computers in our software system setup.

TABLE II: Computer hardware specifications Computer | Computers
---|---
Parts | Raspberry Pi | Notebook | Remote
CPU | Cortex-A53 | i5-3320M | i5-5200U
GPU | VideoCore IV | Intel HD | Nvidia 840M
RAM | 1GB | 4GB | 8GB
STORAGE | 16GB | 500GB | 500GB
OS | Raspbian Stretch | Ubuntu 18.04 | Ubuntu 18.04
ROS | Kinetic | Melodic | Melodic
Figure 3: Wiring diagram of Eddie robot

###  II-B Software framework

The implementation of autonomous navigation on Eddie robot uses ROS as a
framework to provide a network environment where information from the
Raspberry Pi and Notebook can be transmitted from one to another. The software
framework is mainly divided into two sections which are: navigation system and
photo-taking behavior. Several ROS nodes have been implemented specifically
for the Eddie robot’s hardware listed in Table I to enable the Navigation
stack to work with the current hardware setup. Additionally, the parameters of
the Navigation stack have been optimized through trial and error. A photo-
taking behavior has been implemented to demonstrate a use case for the
navigation system. A detailed explanation regarding the implementation of the
navigation system will be given in section III.

####  II-B1 Software structure of navigation system

The flowchart of the navigation system is shown in Figure 4. Initially, a user
sends a goal location containing position and orientation information to the
navigation system through the interface of the ROS visualization package,
RVIZ. Then, the system starts to generate a suitable path of trajectory and
sends motor commands (i.e linear and angular velocity) to the two-wheel motor
drivers. While the robot is moving, the odometry is continuously updated and
used by SLAM for localization and mapping. Finally, the map which was
generated by SLAM and robot location (pose) can be visualized on RVIZ.
Additionally, teleoperation allows users to assume manual remote control of
the robot from a remote computer and this was used frequently during the
experimentation phase. Detailed information on the implementation of ROS is
given in section III.

Figure 4: Navigation system software flowchart

####  II-B2 Software structure of photo taking behavior

The flowchart of the photo-taking system is shown in Figure 5. Firstly, the
live raw image data from the Kinect is used by the face detector to detect
faces. Then, the detected face contains depth information which is converted
to point cloud data and this is used to determine the position of the human.
Finally, the human location will be used as a goal position and sent to the
navigation system where the robot will start its autonomous navigation toward
the human.

Figure 5: Photo taking behavior flow flowchart

After successful navigation, the OpenCV library is used to save and show the
taken picture. A more detailed discussion is provided in section IV.

##  III Implementation of navigation system

The implementation process of autonomous navigation will be based on
satisfying the requirements to run ROS Navigation stack [2]. Figure 6 shows
the packages that are required for the Navigation stack to run. Navigation
stack, SLAM, Sensor sources, and Explorer are packages obtained from the ROS
library while the remaining are packages implemented in this paper.

The odometry sources package provides odometry information computed from the
two wheel encoders. The Sensor sources package provides Laser scans
information from the Kinect sensor. Sensor Transforms package provides the
necessary physical position and orientation relationship between different
robot components such as robot base, wheel position, and Kinect sensor.
Altogether, these packages are used by SLAM to properly localize the robot
based on wheel encoder and Kinect sensor information. The Teleoperation
package gets keyboard inputs from the remote computer to manually drive the
robot. The Base controller package accepts velocity commands from either the
Navigation stack or the Teleoperation package that interprets the velocity
commands and sends appropriate signals to the robot motors to drive the robot.
The goal controller package accepts goal location input from RVIZ and finally,
the Explorer package sends the goal location to the Navigation stack based on
the unexplored regions of the map generated by SLAM that allows the robot to
explore its surroundings.

Figure 6: ROS Navigation stack setup

The overall overview and details of the Navigation Stack packages’
implementation will be explained in subsection III-D. Meanwhile, the following
sections will be about the implementations of the robot’s motor control, wheel
encoder odometry, base controller, base teleoperator, goal controller, and
photo-taking behavior.

The Teleoperation, Goal Controller, and Base Controller packages shown in
Figure 6 require implementing a robot motor control (subsection III-A) which
converts velocity commands into motor commands to drive the robot. While the
Odometry Sources package requires implementing a robot encoder reader
(subsection III-B) and odometry model (subsection III-C) for robot
localization using encoder hardware.

###  III-A Motor control model

PIGPIO library [10] is used to control the wheel motor controllers with Pulse
Width Modulation (PWM). A relationship between velocity and Pulse Width (PW)
of the left and right wheel motors is defined. The relationship between the
PWs of the motor controllers and velocity is assumed to be proportional [11].

Firstly, the PWs of the motor controllers are set up in such a way that the
two wheels turn at maximum speed and a tachometer is used to measure the
Revolution Per Minute (RPM) of the two wheels at that speed. The linear
equation to establish the relationship between RPM and PW (time in
milliseconds) is given below:

$RPM=K\cdot PW$ (1)

Where $K$ is the proportionality constant between $RPM$ and $PW$. In Equation
1, $K$ can be obtained since $PW$ and $RPM$ are known. By using the robot’s
wheel radius $r$, an equation between $PW$ and linear velocity $V$ is derived
as below:

$V=\frac{2{\pi}r}{60}\cdot{RPM}$ (2)

$V=\frac{2{\pi}r}{60}\cdot{K}\cdot{PW}$ (3)

Based on Equation 3, a program is implemented to convert velocity commands
into motor commands to control the speed of the wheel motors.

###  III-B Encoder reading model

RPI.GPIO library [12] is used to detect generated pulses from the encoders for
localization purpose. Two wheel encoders are used to deduce the speed and
travel direction by measuring the number of pulses registered from each wheel.
Each wheel encoder has two sensors and is capable of registering a distance
resolution of 1/36th of the robot’s wheel circumference. Figure 7 [13] shows
the signals generated by the encoders.

Figure 7: Visualizaion of signals generated by the wheel encoders

Knowing that the sensor has 1/36th resolution of wheel circumference, the
sensor produces 36 pulses for every complete revolution of the wheel. Based on
this, the distance traveled in the duration of one pulse is given below:

$d=\frac{2{\pi}r}{36}$ (4)

Where $d$ is the distance travelled in meters and $r$ is the radius of the
wheel in meters.

Speed information is computed in Equation 5.

$s=\frac{d}{t}$ (5)

Where $t$ is the Pulse Width (PW) which is the time that the encoder’s signal
stays high (i.e A-C) as shown in Figure 7.

###  III-C Odometry model

This model is used to describe the robot’s position and orientation as a
function of the movement of its wheels but ignores any physics involved in
making that motion. So, it is used to estimate a robot’s position using the
sensor information from wheel encoders and this is known as dead reckoning
[14].

In this case, the robot has two independent controllable wheels which are
mounted on a common axis and the control of its movement direction is by
manipulating the difference in velocities of its two wheels. Under the
assumption that the wheels are fixed on the robot, the two wheels must
describe arcs on the plane such that the robot rotates around a point known as
ICC (Instantaneous Center of Curvature) which lies on the wheels’ common axis
(Figure 8) [14]. The X-Y axes refer to the global reference frame in the
environment.

Let us consider the contribution of each wheel’s linear velocity to the
movement direction of the robot.

  * •

When $V_{l}$ and $V_{r}$ are equal, the robot moves in a straight line.

  * •

When $V_{l}$ and $V_{r}$ are not equal, the robot moves in the direction of
wheel with lower linear velocity.

  * •

When $V_{l}$ and $V_{r}$ are equal and in opposite direction, the robot spins
at stationary position. If the left wheel is in forward direction, the robot
spins clockwise; and vice versa.

Figure 8: Robot Cartesian coordinate system and variable definitions

If the linear velocity of the left and right wheels are ${V}_{l}$ and
${V}_{r}$ respectively, and the right and left wheels are separated by
distance $D$ in meters, then linear velocities can be computed as follows:

$V_{l}=(R-\frac{D}{2})\cdot{\omega}$ (6) $V_{r}=(R+\frac{D}{2})\cdot{\omega}$
(7)

Where, $R$ is the distance between the center of the robot to the ICC in
meters and $\omega$ is the rate of rotation about the ICC in radian per
second. The following equations (Equation 8 and Equation 9) are derived by
rearranging Equation 6 and Equation 7 to determine $R$ and $\omega$ at any
instance of time.

${\omega}=\frac{({V}_{r}-{V}_{l})}{D}$ (8)
$R=\frac{D}{2}\cdot\frac{{{V}_{r}+{V}_{l}}}{{V}_{l}-{V}_{r}}$ (9)

Thus, the instantaneous velocity at the center point of the robot is given by
Equation 10 and can be derived as Equation 11 from Equation 8 and Equation 9.

${V}_{s}=\frac{V_{l}+V_{r}}{2}$ (10)

${V}_{s}={\omega}{R}$ (11)

Using the center point between the wheels as the origin of the robot, and
writing $\theta$ as the orientation of the robot with respect to the x-axis of
a global Cartesian coordinate system shown in Figure 8, a set of motion
equations for the robot can be generated as shown below in Equation 12 and
Equation 13 where ${V}_{x}$ and ${V}_{y}$ are the velocity in x and y
direction respectively. These equations can be rearranged to obtain Equation
14 and Equation 15.

$\cos\theta=\frac{V_{x}}{V_{s}}$ (12) $\sin\theta=\frac{V_{y}}{V_{s}}$ (13)
$V_{x}=V_{s}\cdot\cos\theta$ (14) $V_{y}=V_{s}\cdot\sin\theta$ (15)

The movement in x and y direction as well as the angular movement are obtained
by taking the integration of the respective direction components at small time
steps, we can simplify the computation of change in $x(t)$, $y(t)$ and
$\theta(t)$ as below:

$\delta{x(t)}={\int_{0}^{t}V_{s}(t)\cdot\cos[\theta(t)]\cdot\delta{t}}$ (16)
$\delta{y(t)}={\int_{0}^{t}V_{s}(t)\cdot\sin[\theta(t)]\cdot\delta{t}}$ (17)
$\delta{\theta(t)}={\int_{0}^{t}\omega(t)\cdot\delta{t}}$ (18)

Where $\delta{t}$ is time step where change of position in $x$, $y$ and
orientation $\theta$ is computed. In this case, the time step can be based on
the Pulse Width (PW) from the wheel encoders’ readings that is the period when
the encoder signal is detected as high (see subsection III-B).

Suppose that the robot is positioned at ($x$, $y$, $\theta$) at time $t$ in
the coordinate system (Figure 8), then the next position ($x^{\prime}$,
$y^{\prime}$, $\phi^{\prime}$) can be calculated using the equations below:

$x^{\prime}=x+\delta{x(t)}$ (19) $y^{\prime}=y+\delta{y(t)}$ (20)
$\theta^{\prime}=\theta+\delta{\theta(t)}$ (21)
$x^{\prime}=x+{\int_{0}^{t}V_{s}(t)\cdot\cos[\theta(t)]\cdot\delta{t}}$ (22)
$y^{\prime}=y+{\int_{0}^{t}V_{s}(t)\cdot\sin[\theta(t)]\cdot\delta{t}}$ (23)
$\theta{{}^{\prime}}=\theta+{\int_{0}^{t}\omega(t)\cdot\delta{t}}$ (24)

Where, [$x$, $y$, $\theta$] is the current position and orientation of the
robot and [$\delta{x(t)}$, $\delta{y(t)}$, $\delta{\theta(t)}$] is the change
in position and orientation at time step $t$. So, this Odometry model is used
to estimate the position and orientation of the robot for localization.

###  III-D Navigation stack setup

Figure 6 shows the complete system for the Navigation stack [1][2] and the
packages that are required to run the stack. Table III shows the list of
packages that were implemented or obtained from the ROS library. This section
will mainly discuss the tuning of the Navigation stack and some of its
packages, and the implementation of custom packages. The implementation of
Face Detector and Photo Taking Behavior packages will be discussed in section
IV

TABLE III: ROS Navigation stack packages Packages | Status
---|---
ODOMETRY SOURCES | Implemented
GOAL CONTROLLER | Implemented
BASE CONTROLLER | Implemented
TELEOPERATION | Implemented
SENSOR TRANSFORMS | Implemented
SLAM | ROS library
SENSOR SOURCES | ROS library
EXPLORER | ROS library

For reference, the SLAM and EXPLORER packages used are called ”gmapping” [5]
and ”explorer_lite” [15] respectively. Meanwhile, there are two packages used
in SENSOR SOURCES which are ”openni_launch” [16] and ”depthimage_to_laserscan”
[9] respectively.

ODOMETRY SOURCES package is used to provide localization information based on
the robot’s wheel encoders, GOAL CONTROLLER package is used to send a goal
location to the Navigation Stack for the robot to move to and is used in the
implementation of photo-taking behavior. BASE CONTROLLER package is used to
drive the robot after receiving drive commands from the Navigation Stack.
TELEOPERATION package is used to remotely control the robot through keyboard
inputs for debugging purposes. SLAM package is used to create a map based on
the robot’s sensor readings from the ODOMETRY SOURCES and SENSOR SOURCES
packages. SENSOR SOURCES is used to send the robot’s Laser Scan information
from its Kinect sensor. Finally, EXPLORER package is used for the robot to
explore the environment by sending goal locations to the Navigation Stack
based on unexplored regions of the environment.

####  III-D1 Sensor Transforms package

A package called robot_transforms has been created which maintains a
Transforms (TF) tree for the robot. Within the package, a node called
transforms_publisher publishes messages on a topic named “/tf” with message
type “tf/tfMessage” as shown in Figure 9.

Figure 9: robot_transforms package node

A TF tree is required to maintain the position and orientation relationship
between different sensors attached to a robot platform. In this case, the
reference point of Odometry is at the center of the robot base. So, a TF
between the Odometry (Odom), robot base (base_link), and Kinect sensor
(camera_link) is established using TF library in ROS. Table IV shows the 3D
measurements of the robot’s TF tree. Note that the Kinect sensor (camera_link)
is installed at the height of 1 meter above the center of the robot base
(base_link).

TABLE IV: Robot Transforms tree measurements TF | x, y, z (in m)
---|---
camera_link | 0.0, 0.0, 1.0
base_link | 0.0, 0.0, 0.0
odom | 0.0, 0.0, 0.0

####  III-D2 Odometry Source package

An Odometry source is required to provide position($x$, $y$) and orientation
($\theta$) information to the Navigation stack. The odometry information is
referenced from the center of the robot base and computed based on the signal
detected from the wheel encoders. A package called robot_odometry has been
created which performs robot localization and the implementation is based on
Encoder Reading model and Odometry model (subsection III-B and subsection
III-C respectively). Within the package, a node called encoder_tick_publisher
publishes messages on a topic named “/odom” with type “nav_msgs/Odometry” as
shown in Figure 10, which encapsulate the pose (x, y) and orientation
($\theta$) of the robot.

Figure 10: robot_odometry package node

####  III-D3 Base Controller package

A package called robot_controller has been created which controls the robot
and the implementation is based on the Motor Control layer (subsection III-A).
Within the package, a node called robot_controller subscribes to a topic named
“/cmd_vel” which publishes messages with type “geometry_msgs/Twist” as shown
in Figure 11. The message type geometry_msgs/Twist stores velocity commands in
the form of linear and angular velocities.

Figure 11: robot_controller package node

Since path planning is done within the Navigation stack (Figure 4), the
Navigation stack is responsible for publishing velocity commands on /cmd_vel
topic and this package converts velocity commands into motor control commands
that drive the robot.

####  III-D4 Teleoperation package

Within the robot_controller package, two nodes called keyboard_driver (Figure
12) and keys_to_twist (Figure 13) have been created which detect keyboard key
presses on a remote computer and convert the key presses into velocity
commands respectively. “keyboard_driver” publishes messages on a topic named
“/keys” with type “std_msgs/String” while “keys_to_twist” subscribes to the
topic named “/keys” and publishes messages with type “geometry_msgs/Twist”.

Figure 12: Teleopration package node Figure 13: Teleoperation package node

The two nodes mentioned above is used to publish velocity commands for the
robot_controller (Figure 11) to manually move the robot using key presses from
a remote computer. The use of this functionality was mainly for testing and
debugging of the Odometry system as well as for potential remote control
applications. So, the robot_controller node can subscribe to the topic
“/cmd_vel” published by the Navigation stack or Teleoperation node.

####  III-D5 Goal controller package

A package called robot_send_goals has been created which sends goal commands
to the Navigation stack without the RVIZ graphical interface. Within the
package, a node called move_base_to_goal shown in Figure 14 sends a goal
message with a message type of “geometry_msgs/PoseStamped” that contains the
position ($x$, $y$) and orientation($\theta$) of the goal location to the
Navigation stack. A goal location input in the form of x, y and orientation
$\theta$ can be passed from a function that intends to move the robot to a
specific location.

Figure 14: robot_send_goals package node

Besides being used by the Navigation stack, this package is also used in the
implementation of photo-taking behavior (section IV) where the position and
orientation of a human face are set as a goal position where the robot will go
to for photo taking.

###  III-E Navigation stack parameter tuning

The default parameters in the configuration files of the ROS Navigation stack
did not work well for the robot as it moved erratically with highly
fluctuating ground wheel speed and did not manage to stop at the designated
goal position. So, custom parameters built for the robot hardware
specification are required to run the Navigation stack successfully. Through
visual inspection, these parameters were found to provide the best balance
between navigation performance and computational cost. Table V, Table VI,
Table VII and Table VIII show the tuned parameters. The tuned parameters and
detailed explanation can be found in [17].

TABLE V: Base local planner parameters Parameters | Values
---|---
controller_frequency | 15.0
holonomic_robot | false
yaw_goal_tolerance | 0.2
xy_goal_tolerance | 0.3
sim_time | 4.0
vtheta_samples | 40
TABLE VI: Costmap common parameters Parameters | Values
---|---
footprint | [[-0.1, -0.1], [-0.1, 0.1], [0.1, 0.1], [0.1,-0.1]]
transform_tolerance | 0.3
publish_frequency | 10.0
inflation_radius | 1.75
cost_scaling_factor | 2.58
TABLE VII: Global costmap parameters Parameters | Values
---|---
update_frequency | 10.0
static_map | true
TABLE VIII: Local costmap parameters Parameters | Values
---|---
update_frequency | 10.0
publish_frequency | 10.0
static_map | false
rolling_window | true
width | 6.0
height | 6.0
resolution | 0.05

##  IV Implementation of Photo taking behavior

The photo-taking behavior is implemented based on the Goal Controller node
(Figure 14). Instead of a user-defined goal position, the position of the
detected face is used as a goal position for the robot to traverse to.

In this section, the implementation of two additional packages will be
discussed: 1. Face Detector and 2. Robot Behavior.

###  IV-A Face Detector package

A package called robot_face_detector has been created which performs face
detection on Kinect sensor RGB images and outputs the detected face’s goal
location based on the Depth image. Within the package, a node called
face_detector which subscribes to the topics “/camera/rgb/image_raw” and
“/camera/depth/image_raw” that contain messages published by openni_launch
node. As the names imply, these messages contains the RGB and Depth images
information from the Kinect sensor. The face_detector node is shown in Figure
15.

Figure 15: robot_face_detector package node

OpenCV Haar Cascade classifier [18] has been used to detect human faces on the
RGB image and the 2D location of the detected face is obtained from the face
point cloud information derived from the depth image [19]. The OpenCV Bridge
[16] was used to convert the raw image message data to the cv2 image.

###  IV-B Robot behavior configuration

A package called robot_behavior has been created which contains the behavior
of the robot during exploration. Within the package, a node called
robot_behavior shown in Figure 16 subscribes to the following four topics:

  * •

“/odom” which publishes messages of type “nav_msgs/Odometry”

  * •

“/camera/rgb/image_color” which publishes messages of type “sensor_msgs/Image”

  * •

“/face_detector/is_there_face” which publishes messages of type
“std_msgs/Bool”

  * •

“/face_detector/goal_position” which publishes messages of type
“sensor_msgs/Point”

Figure 16: robot_behavior package node

The “/odom” topic is to obtain the robot’s current location so that when a
face is detected, the position and orientation information that is needed to
get the robot to the goal location (i.e face) can be computed relative to the
current location of the robot. Then, the “/face_detector/is_there_face” topic
and “/face_detector/goal_position” topics are used in the behavior program for
sending the location of the detected face to the Navigation stack using a ROS
package called Actionlib [20]. Finally, the “/camera/rgb/image_color” topic is
used to obtain the detected face images for saving.

By default, the behavior of the robot is to explore and roam its environment,
and this functionality is provided by the Explorer package [15] mentioned
earlier in subsection III-D. When the face_detector (subsection IV-A) package
has detected a face, the robot will start navigating toward the face location
to initiate taking a picture by using OpenCV’s save() and show() function
[18].

##  V Experiment setup

The evaluation of the implemented functionalities (i.e navigation system and
photo-taking behavior) was carried out in an indoor environment shown in
Figure 17 and Figure 18. The confirmation of results was through visual
inspections due to the lack of a mocap system.

Figure 17: Outside environment of working lab (angle A) Figure 18: Outside
environment of working lab (angle B)

The evaluation was performed on the following functions:

  1. 1.

Wheel encoder odometry: a test to determine correct implementation of encoder-
based odometry.

  2. 2.

Mapping and localization: a test to determine correct implementation of
encoder-based odometry with SLAM.

  3. 3.

Autonomous navigation: capability to reach a goal location while avoiding
obstacle.

  4. 4.

Face detection: capability to detect face and compute its location in 2D map.

  5. 5.

Photo taking behavior: capability to use the implemented navigation system to
traverse to a human reliably and take photo.

Table IX shows the set travel speed and acceleration of the robot during the
experiment. These are also parameters of the Navigation stack similar to Table
V, Table VI, Table VII and Table VIII. During the experiment, visual
inspection in addition to RVIZ were used to monitor the robot. At the
beginning of each experiment, the robot was moved to its starting position,
and the necessary commands to run the functions for evaluation listed above
were sent through a remote computer.

TABLE IX: Robot travel speed and acceleration setup Parameters | Values
---|---
max_vel_x | 0.175 m/s
min_vel_x | 0.175 m/s
max_vel_theta | 1.0 m/s
min_vel_theta | -1.0 m/s
min_in_place_velocity_theta | 1.0 m/s
escape_vel | -0.175 m/s
acc_lim_theta | 0.5 m/s
acc_lim_x | 0.05 m/s
acc_lim_y | 0.05 m/s

The maximum and minimum velocity in x direction (max_vel_x, min_vel_x) were
set at 0.175 m/s for fixed speed navigation. Similarly, the maximum and
minimum rotational velocity (max_vel_theta, min_vel_theta) were set at 1.0 m/s
and -1.0 m/s respectively for fixed speed rotation during navigation. The
minimum in-place rotational velocity (min_in_place_vel_theta) was set at 1.0
m/s. The escape velocity (escape_vel) which is the reverse speed was set at
-0.175 m/s. The acceleration of velocity in x and y direction (acc_lim_x,
acc_lim_y) was set at 0.05 m/s2 to ensure smooth navigation and no jerkiness.
Similarly, the maximum acceleration of rotational velocity (acc_lim_theta) was
set at 0.5 m/s2 for smooth rotation during navigation.

##  VI Results

###  VI-A Wheel encoder odometry

In this experiment, the Base Controller and Teleoperation package (subsection
III-D) were used to manually control the robot from a remote computer. In
addition, Odometry source (subsection III-D) was used to compute the robot’s
encoder Odometry. We remotely controlled the robot and through visual
inspection, the robot’s actual location correlated with the estimated Odometry
readings.

###  VI-B Mapping and localization

SLAM package (subsection III-D) was used to build a map of the perceived
environment. We remotely controlled the robot and through the RVIZ interface,
the generated map correlated with the actual test environment shown in Figure
16 and Figure 17. Additionally, the robot’s position throughout the test
correlated with the estimated Odometry readings.

Figure 19: Generated map from slam_mapping node

###  VI-C Autonomous navigation

In this experiment, all packages that were shown in Table IV were run and an
obstacle in the form of cardboard boxes was laid in the middle of the test
environment. Through the RVIZ interface, a goal location was sent and the
robot started its navigation and stopped at the set goal location without
collisions. Moreover, the robot traversed according to the trajectory
generated by the Navigation stack and moved at a smooth and steady pace. A
drawback with using the Navigation stack is that it requires fine-tuning in
the settings in order for it to work properly with different robots and this
can be a time-consuming trial and error process. Additionally, the Navigation
stack will not run smoothly on an embedded system like the Raspberry Pi and
would require setting up an additional computer with more computational power
to run the autonomous navigation.

Figure 20: Environment with added obstacle Figure 21: Global path trajectory
of robot goal position

###  VI-D Face detection

In this experiment, a human subject sat in front of the robot at varying
distances and the robot can reliably detect human faces at a distance of up to
3.0 meters. The face point cloud that was computed correlated to the actual
position of the detected face in the test environment. However, the OpenCV
Haar cascade [18] face detection implementation used in this paper was prone
to false positives and sometimes detect faces in a location where there are
were no faces. The face detector is capable of detecting frontal faces.
However, humans may not always be in a frontal position or within a short
distance of 3.0 meters (subsection VI-D). So, human detection can be
implemented as a prior before initiating face detection for the robot’s photo-
taking behavior.

Figure 22: Detected human face at 1.0 meter away Figure 23: Detected human
face at 3.0 meter away

###  VI-E Photo taking behavior

During the experiment, the robot managed to navigate to the human subject
using the implemented navigation system for photo taking, subsequently
rotating and continue exploration until another human face is detected. There
were instances when the face detector failed to correctly detect faces and
this resulted in the robot navigating to a location and a taking a picture
with no humans.

##  VII Conclusion

In this paper, we have described the implementation of ROS Navigation stack on
a Parallax Eddie mobile robot platform which was customized and fitted with
different hardware components. We have given detailed information of the robot
hardware specifications which included the wiring diagram and a list of
hardware used for the robot. In terms of software contributions, we have
created several custom ROS packages for the Navigation stack such as robot
odometry, goal controller, base controller, teleoperation and sensor
transforms. Additionally, we have shown a detailed implementation of the
robot’s motor control with Pulse Width Modulation (PWM) and wheel encoder-
based odometry localization. Furthermore, the parameters of the Navigation
stack have been tuned according to the robot’s hardware specifications.
Finally, two more ROS packages were implemented which are face detector and
photo-taking behavior to demonstrate a use case for autonomous navigation.

An important component that needs to be improved is that the face detector
package used in the experiments cannot reliably detect human faces in non-
frontal positions and from a distance of more than 3.0 meters. So, perhaps a
more robust face detection feature that incorporates human detection as a
prior before face detection can be implemented using Kinect sensor’s
capability to detect skeletal joints of human bodies. This will allow the
robot to know the orientation of the human and allow it to traverse to the
front side of the human and subsequently allow the face detector to perform
optimally since the robot will be facing the front side of a human and be
within the range of 3.0 meters.

A GitLab repository containing details of the experimental setup as well as
source codes used for the robot system has been made public in [21].

## References

  * [1] Morgan Quigley et al.  “ROS: an open-source Robot Operating System”  In _ICRA workshop on open source software_ 3.3.2, 2009, pp. 5  Kobe, Japan
  * [2] Eitan Marder-Eppstein  “Navigation” Accessed: 2020-03-25, http://wiki.ros.org/navigation
  * [3] Hugh Durrant-Whyte and Tim Bailey  “Simultaneous localization and mapping: part I”  In _IEEE robotics & automation magazine_ 13.2 IEEE, 2006, pp. 99–110
  * [4] Sebastian Thrun, Wolfram Burgard and Dieter Fox  “Probabilistic robotics”  MIT press Cambridge, 2000
  * [5] Brian Gerkey  “Gmapping” Accessed: 2020-03-25, http://wiki.ros.org/gmapping
  * [6] Vladimir J Lumelsky and Alexander A Stepanov  “Path-planning strategies for a point mobile automaton moving amidst unknown obstacles of arbitrary shape”  In _Algorithmica_ 2.1-4 Springer, 1987, pp. 403–430
  * [7] Ioannis Rekleitis  “Cooperative localization and multi-robot exploration”  McGill University, 2003
  * [8] Laurent Kneip, Fabien Tâche, Gilles Caprari and Roland Siegwart  “Characterization of the compact Hokuyo URG-04LX 2D laser range scanner”  In _2009 IEEE International Conference on Robotics and Automation_ , 2009, pp. 1447–1454  IEEE
  * [9] Chad Rocket  “Depth image to laserscan” Accessed: 2020-03-25, http://wiki.ros.org/depthimage_to_laserscan
  * [10] Croston  “raspberry-gpio-python” Accessed: 2020-03-25, https://sourceforge.net/projects/raspberry-gpio-python/
  * [11] Atul Kumar Dewangan, Nibbedita Chakraborty, Sashi Shukla and Vinod Yadu  “PWM based automatic closed loop speed control of DC motor”  In _International journal of engineering trends and technology_ 3.2, 2012, pp. 110–112
  * [12] Fivdi  “Pigpio” Accessed: 2020-03-25, https://github.com/fivdi/pigpio
  * [13] “Motor Mount and Wheel Kit Product Guide” Accessed: 2020-03-25, https://www.parallax.com/downloads/motor-mount-wheel-kit-product-guide
  * [14] Gregory Dudek and Michael Jenkin  “Inertial Sensors, GPS, and Odometry”, 2008, pp. 477–490  DOI: 10.1007/978-3-540-30301-5˙21
  * [15] Jim Horner  “Explore lite” Accessed: 2020-03-25, http://wiki.ros.org/explore_lite
  * [16] James Bowman Patrick Mihelich  “CV Bridge” Accessed: 2020-03-25, http://wiki.ros.org/cv_bridge
  * [17] Kaiyu Zheng  “Ros navigation tuning guide”  In _arXiv preprint arXiv:1706.09068_ , 2017
  * [18] G. Bradski  “The OpenCV Library”  In _Dr. Dobb’s Journal of Software Tools_ , 2000
  * [19] Kamarulzaman Kamarudin et al.  “Method to convert Kinect’s 3D depth data to a 2D map for indoor SLAM”, 2013, pp. 247–251  DOI: 10.1109/CSPA.2013.6530050
  * [20] Michael Caroll  “Actionlib” Accessed: 2020-03-25, http://wiki.ros.org/actionlib
  * [21] AI Lab  “ROS Eddie Navigation Robot”  In _GitLab repository_ GitLab, https://gitlab.com/ailab.space/ros-eddie-navigation-robot, 2021


# Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient
Crossmodal Learning

Xiaobao Guo1 Equal contribution Nithish Muthuchamy Selvaraj111footnotemark: 1
Zitong Yu1 Corresponding author Adams Kong1 Bingquan Shen2
1Nanyang Technological University, Singapore
Alex Kot1
2DSO National Laboratories, Singapore

###### Abstract

Deception detection in conversations is a challenging yet important task,
having pivotal applications in many fields such as credibility assessment in
business, multimedia anti-frauds, and custom security. Despite this, deception
detection research is hindered by the lack of high-quality deception datasets,
as well as the difficulties of learning multimodal features effectively. To
address this issue, we introduce DOLOS111The name ‘DOLOS’ comes from Greek
mythology., the largest gameshow deception detection dataset with rich
deceptive conversations. DOLOS includes 1,675 video clips featuring 213
subjects, and it has been labeled with audio-visual feature annotations. We
provide train-test, duration, and gender protocols to investigate the impact
of different factors. We benchmark our dataset on previously proposed
deception detection approaches. To further improve the performance by fine-
tuning fewer parameters, we propose Parameter-Efficient Crossmodal Learning
(PECL), where a Uniform Temporal Adapter (UT-Adapter) explores temporal
attention in transformer-based architectures, and a crossmodal fusion module,
Plug-in Audio-Visual Fusion (PAVF), combines crossmodal information from
audio-visual features. Based on the rich fine-grained audio-visual annotations
on DOLOS, we also exploit multi-task learning to enhance performance by
concurrently predicting deception and audio-visual features. Experimental
results demonstrate the desired quality of the DOLOS dataset and the
effectiveness of the PECL. The DOLOS dataset and the source codes will be
publicly available soon.

##  1 Introduction

Deception is a pervasive and complex phenomenon that occurs in all areas of
life, and understanding its nature and impact is crucial in preventing
negative consequences. Effective deception detection has crucial implications
in the area of border security, anti-fraud, business negotiations and etc.
[14, 12, 1, 13, 9, 43]. Deep learning algorithms have achieved comparable or
even better performance than human in many complex tasks [22, 40, 39, 7, 37].
One may expect AI models can also bring significant breakthroughs in deception
detection. Albeit the fruitful progress in computer vision [47, 10, 30, 45]
and audio representation learning [5, 18, 27], it remains a significant
challenge to efficiently explore AI ability to process multimodal information
in perceiving and predicting human deceptive behaviors.

The performance of AI models in deception detection is heavily reliant on the
availability of authentic and effective deception samples from the real world.
The deceptive subjects must be spontaneous and motivated [8, 48] such that
certain behavioral cues (e.g., vocal pitch and chin raise) could be more
pronounced. Public benchmark datasets collected from court trials [35], game
shows [41], and lab-based scenarios [15], have contributed to spurring
interest and progress in deception detection research. Deception can be
affected by a range of factors in diverse real-world situations. Thus, it is
necessary to create deception datasets from different scenarios. However,
current datasets are still insufficient to drive further progress and inspire
novel ideas due to their limitations on both quantity and quality. These
limitations include (1) the small number of deceptive samples and subjects,
(2) the lack of rich annotated visual and speech attributes, and (3) the
variety of protocols. It is imperative to build a larger and richer deception
detection dataset. In particular, more deceptive samples and subjects, better
annotations for facial movements, gestures, and audio attributes, and more
types of protocols for investigating factors affecting deception detection.

In addition to a high-quality dataset, effective methods are equally important
to deception detection. A variety of works have been done towards using visual
and acoustic information in videos for deception detection [41, 15, 35].
Current methods fall into two categories: unimodal learning and multimodal
fusion. However, both approaches have limitations as they do not fully exploit
unimodal features or integrate complementary information from multiple
modalities. To make better use of available information, fine-tuning large
pre-trained models has shown promising results [38, 26, 20, 46]. However,
fully fine-tuning pre-trained models, such as W2V2 [5] and ViT [10], can be
inefficient and lead to overfitting, especially if the downstream dataset is
limited. Therefore, it is important to consider parameter efficiency when
fine-tuning pre-trained models. Adapters [17, 21] offer an efficient approach
for fine-tuning models. While originally proposed for language [17] or vision
[21] tasks, adapters have not yet been applied to multiple modalities for
temporal feature extraction.

Contributions. In this paper, we establish a new deception detection dataset
and propose parameter-efficient crossmodal learning for audio-visual deception
detection.

As our first contribution, we introduce DOLOS, a new gameshow dataset for
audio-visual deception detection. DOLOS has several merits compared with
current public datasets. First, the gameshow is a reliable source for
collecting deception detection data because all the participants are motivated
to cheat and the ground truths are available. Second, the proposed dataset is
in a conversational setup, where the deception behaviors are more naturally
presented. Third, our dataset is the largest in terms of the number of
subjects and also the largest non-lab based dataset in terms of the number of
video clips. The dataset has been labeled with fine-grained audio-visual
annotations. We also benchmark our dataset on previous deception detection
approaches that involve both unimodal and multimodal features. In addition, we
offer three distinct protocols, namely train-test, duration, and gender, to
explore various factors that may impact deception detection.

Our second contribution is Parameter-Efficient Crossmodal Learning (PECL), a
method for deception detection that achieves high performance by fine-tuning a
small number of extra learnable weights. Specifically, we introduce a Uniform
Temporal Adapter (UT-Adapter) that explores the temporal attention between
input embeddings for both visual and audio modalities without the need for
delicate modifications. For multimodal fusion, we propose Plug-in Audio-Visual
Fusion (PAVF), which utilizes the complementary information between visual and
audio features to enhance the overall performance. PECL is designed to be
parameter efficient, with only the UT-Adapter, PAVF modules, and the
classifier being trainable. Furthermore, we explore the benefits of multi-task
learning, a proven method that enhances performance in audio and visual tasks
[53, 44]. By simultaneously predicting deception, facial movements, and
phonetic features, our proposed method can be further improved.

To compare and show the advantages of our dataset and the proposed method, we
conducted extensive experiments. In the cross-testing with the current
gameshow benchmark dataset [41], DOLOS performs better on several test sets in
different scenarios. The experimental results on DOLOS also showed that PECL
yields superior performance on different protocols. Through our experiments,
we uncover valuable insights into multimodal deception detection. We believe
that DOLOS, the proposed method, and the benchmarking experiments provide
valuable resources to other researchers in advancing research in this area.

##  2 Related Work

###  2.1 Deception Detection Datasets

Early works in deception detection research investigated the psychological
premises of deception [8, 16, 49, 29, 48] and identified potential cues, such
as physiological, visual, vocal, verbal and behavioral cues. The pioneering
work on multimodal deception detection emerged with the introduction of the
Real-Life Trials (RLT) dataset [35] collected from courtroom trials. The
courtroom setting provides a valuable and authentic scenario of people lying,
but RLT is small and its performance bottleneck was quickly reached. Pérez-
Rosas  _et al._ [36] and Kamboj  _et al._ [23] later proposed another two new
datasets collected from street interviews and political speeches. However, it
is difficult to verify the veracity of the responses of the subjects in these
datasets. Several lab-based multimodal deception detection datasets were
proposed to overcome these shortcomings. The settings of these datasets were
typically in situations that speakers were allowed to lie, such as describing
an object [15], a personality [31], and face-to-face interviews with the
subjects [42]. However, the incentive to deceive may be low, leading to fewer
deceptive cues captured.

Reality TV shows based on truth or lie games provide an avenue to collect
examples of conversational deception. They also offer a sweet spot among
incentive, veracity, and sample size requirements. Often, the participants lie
to each other to win the game, and the truth is revealed at the end of the
round. Large amounts of gameshow footage are available online. The first
gameshow-based deception detection dataset was proposed in [41] and it
primarily focused on two modalities: text and manually annotated visual
features. However, directly applying this dataset for multimodal deception
detection by using audio and visual modalities faces several challenges, which
we will describe in Sec. 3.

Figure 1:  Gameshow format. An example of a deceptive round.

###  2.2 Multimodal Deception Detection

Previous works on multimodal deception detection include using unimodal and
multimodal features and exploring effective fusion mechanism [32, 24, 25, 4,
9, 50, 13]. Gogate _et al._ [13] proposed a deep model that incorporated the
audio cues with visual and text modalities to improve the accuracy of
deception prediction. Karimi _et al._ [24] explored deceptive cues from RGB
images and raw audio in an end-to-end manner. Wu _et al._ [50] utilized
several types of features, including micro-expression and IDT (Improved Dense
Trajectory) features from RGB images, MFCC (Mel-frequency Cepstral
Coefficients) features from the audio, and transcripts. Avola _et al._ [4]
addressed the deception detection issue by directly using AUs (facial action
units) from video frames and classifying them through an SVM. Ding _et al._
[9] tried to combine facial expressions and body movements to extract more
visual deception cues. They also tried to improve the performance by involving
audio features and transcripts. Mathur and Matarić [32] focused on facial
expressions for deception detection and studied interpretable features from
visual, vocal, and verbal modalities. Karnati _et al._ [25] proposed deep
networks that learn audio, visual, and EEG representations for deception
detection.

Most of the previous works tried to enhance the performance by involving more
modalities, more features, and using better feature extraction methods.
However, they heavily rely on spatial features rather than temporal
information within and between the modalities, which may lose potentially
important cues and lead to sub-optimal results.

In terms of fusion methods, the feature-level fusion which concatenates
multimodal embeddings and uses the linear layers to extract crossmodal
information [24, 4, 9, 50, 13] was the most popular approach. Other works
implement decision-level fusion, such as score-level fusion [25, 13]. In all
cases, performance can be boosted by a reliable and efficient fusion method.

Figure 2: DOLOS dataset for multimodal deception detection.

##  3 The DOLOS Dataset

###  3.1 Gameshow Format

We collect data from a British reality comedy gameshow available on YouTube
222Videos are collected from YouTube under the fair use policy, where six
participants compete as two teams. In each taking turn, one of their members
read out a statement about their personal life from a card. This statement
could be either true or false, and only the speaker knows which one it is. The
opposing team then asks the speaker a series of questions, and the speaker
must provide answers in defense of their statement. The goal of the speaker is
to convince the opposing team that their statement is true, while the opposing
team tries to determine whether the statement is actually true or false based
solely on the speaker’s answers. At the end of each round, the veracity of the
statement is revealed, and if the opposing team correctly predicts whether it
was true or false, they win the round; otherwise, the speaker’s team wins. The
game continues for several rounds until all participants have played. Notably,
if the statement is true, then the speaker’s answers will be considered as
truthful video clips and vice versa. The gameshow format is given in Fig. 1.

###  3.2 Data Collection and Annotation Procedure

For each episode of the gameshow videos, the video clips that satisfy the
following requirements are extracted:

  * •

The participant speaks only the relevant content (_i.e._ , telling the truth
or lies) in a clear voice and without strong background noise.

  * •

The face of the participant is clearly visible without occlusion.

Dataset Hand annotated features #Subjects Samples Scenario Total Deceptive
Truthful Deception/Truth Ratio Real Life Trials[35] ✓ 56 121 61 60 1.02 Court-
room Real Life Deception[36] - 56 31 25 1.24 Street interviews Political
Deception[23] 88 180 - - - Political speech Bag of Lies[15] 35 325 162 163
0.99 Lab MU3D[31] 80 320 160 160 1 Lab Deception Detection and remote PPG[42]
70 1680 630 1050 0.6 Lab Box of Lies[41] ✓ 26 1049 862 187 4.61 Gameshows
Gainsboro!60 DOLOS (Ours) ✓ 213 1675 899 776 1.16 Gameshows

Table 1: Comparison of multimodal deception detection datasets

From a total of 84 episodes, we extracted 1675 video clips from 213 (141 Male
and 72 Female) participants. The duration of the video clips ranges from 2-19
seconds. The extracted video clips are manually annotated for non-verbal
deceptive cues using the popular MUMIN coding scheme [2], where we
particularly focus on the visual (25 facial) and vocal (5 speech) features.
The non-verbal features in the MUMIN coding scheme and the distributions of
video clips are shown in Fig. 2.

To eliminate bias among the human annotators, all six human annotators
underwent calibration by annotating the same subset of the dataset, and their
performance was evaluated using Cohen’s Kappa scores [6]. If significant
discrepancies are found in visual and audio annotations between annotators,
they reconciled their differences through discussion and then re-annotated the
subset. This iterative process was repeated until inter-annotator agreement
was achieved. Before calibration, the average Cohen’s Kappa scores were 0.5
(0.53 for audio and 0.47 for visual annotations). After calibration, the
average scores increased to 0.65 (0.67 for audio and 0.63 for visual
annotations).

Based on the statistics in Fig. 2, we organize DOLOS dataset into three
different protocols. First, we provide a train-test protocol with a 3-fold
split to evaluate the performance of the deception detector for DOLOS dataset.
Second, to reflect the variability in speaking duration in real-world
scenarios, we provide a duration protocol with short clips (2-4s) and long
clips (5-10s) based on the statistics from our dataset. We also provide a
gender (male and female) protocol to investigate different factors in the
deception detection task.

###  3.3 Comparison with Box of Lies and Other Datasets

We compare our DOLOS with Box of Lies (BOL) [41], which was also collected
from gameshow videos. In terms of quantity, BOL has fewer video clips and
fewer subjects. BOL contains only 26 subjects, which is 17.9% of DOLOS. In
terms of quality, among 1,049 video clips in BOL, 537 video clips missed the
speaker’s face. In the other video clips, the speaker and the audio of some
video clips do not match. In BOL, the host of the gameshow has more video
clips than other subjects, which introduces a large bias. In comparison, the
data collection requirements mentioned in Sec. 3.2 inherently prevent these
problems. Therefore, DOLOS is more suitable for audio-visual deception
detection research (see Sec. 5.3) than BOL.

We also compare DOLOS with other publicly available datasets. A summary of the
comparisons is shown in Table 1. Briefly, DOLOS offers the following
advantages:

  * •

DOLOS is the largest in terms of number of subjects. It is also the largest in
terms of video clips in the non-lab scenario.

  * •

Compared to the Box of Lies gameshow dataset, Dolo contains a larger number of
video clips and is more well-balanced in terms of the proportion of deceptive
and truthful samples.

  * •

DOLOS provides manually annotated MUMIN features for all video clips. These
annotations can be directly used for deception detection task. They can also
be applied for other related tasks (_e.g._ , facial expression prediction) or
multi-task learning.

Figure 3: Parameter-Efficient Crossmodal Learning (PECL) Framework. (a) is the
main architecture. (b) is the encoder layer $A_{i}$ for audio features. (c) is
the UT-Adapter with a shared structure between audio and visual features. (d)
is the encoder layer $V_{i}$ for visual features. (e) is the PAVF module
$F_{i}$. The trainable modules and layers are marked by “fire”, and the frozen
layers are marked by “snowflakes” .

##  4 Methodology

To efficiently leverage the prior knowledge from large-scale generic audio and
visual tasks, we propose Parameter-Efficient Crossmodal Learning (PECL) for
audio-visual deception detection. PECL’s overall framework is illustrated in
Fig. 3 with the main network architecture (Fig. 3 (a)) and several components
with detailed structures (Fig. 3 (b)-(e)).

Specifically, in Fig. 3 (a), visual inputs are tokenized by a 2D-CNN module
and audio inputs are tokenized by a 1D-CNN module (See Sec. 5.1) with the same
dimensions, which are denoted as $\boldsymbol{X_{v}}\in\mathbb{R}^{L\times D}$
and $\boldsymbol{X_{a}}\in\mathbb{R}^{L\times D}$, where $L$ is the length of
sequences and $D$ is the dimension of each sequence embedding. $V_{i}$ and
$A_{i}$ represent transformer-based visual and audio modality encoder layers,
where we adopt W2V2 [5] as the backbone network for audio modality and ViT
[10] for visual modality. $F_{i}$ are fusion modules between each pair of
unimodal encoder layers and $i={1,2,3,4}$. $C_{1}$, $C_{2}$, and $C_{3}$ are
classifiers for audio, visual, and fusion networks, respectively. $L_{A}$,
$L_{V}$, and $L_{F}$ are cross-entropy losses for audio, visual, and fused
modalities, respectively.

###  4.1 Uniform Temporal Adapter (UT-Adapter)

To improve the parameter efficiency and alleviate the overfitting issue, we
insert learnable adapter layers [17, 21] in the pre-trained visual and audio
models. During training, all the weights, except those in the adapter layers
and the classification layers are frozen. Unlike NLP-Adapter for language
tasks [17] and Conv-Adapter for vision tasks [21], our adapter layer, named
Uniform Temporal Adapter (UT-Adapter), is to explore temporal attention for
both visual and audio modalities.

Let $F_{V}$ and $G_{A}$ represent the visual and audio encoder layers in Fig.
3. Their outputs are denoted as

$\small\boldsymbol{X^{{}^{\prime}}_{v}}=F_{V}(\boldsymbol{X_{v}})\ and\
\boldsymbol{X^{{}^{\prime}}_{a}}=G_{A}(\boldsymbol{X_{a}}),$ (1)

where $\boldsymbol{X^{{}^{\prime}}_{v}}\in\mathbb{R}^{L\times D}$ and
$\boldsymbol{X^{{}^{\prime}}_{a}}\in\mathbb{R}^{L\times D}$. $F_{V}$ and
$G_{A}$ are formed by a Multi-Head Self-Attention (MHSA) module, UT-Adapters,
a Multi-Layer Perception ($MLP$) module, and LayerNorm ($LN$). To simplify the
notations, let $\boldsymbol{X}$ be one of the input features, _i.e_ ,
$\boldsymbol{X_{v}}$ and $\boldsymbol{X_{a}}$ (in Fig. 3 (b) and (d)).
$\boldsymbol{X}$ is projected to a query $\boldsymbol{Q}=\boldsymbol{XW_{Q}}$,
a key $\boldsymbol{K}=\boldsymbol{XW_{K}}$, and a value
$\boldsymbol{V}=\boldsymbol{XW_{V}}$, where $\boldsymbol{W_{Q}}$,
$\boldsymbol{W_{K}}$, and $\boldsymbol{W_{V}}\in\mathbb{R}^{D\times D}$ are
pre-trained projection weights. For a single head
$H_{j},j\in\\{1,2,\cdots,N\\}$, the self-attention operation is formulated as

$\small
H_{j}(\boldsymbol{X})=Softmax\left(\cfrac{\boldsymbol{Q_{j}}\boldsymbol{K_{j}}^{\intercal}}{\sqrt{D}}\right)\boldsymbol{V_{j}},$
(2)

and the outputs of all the $H_{j},j\in\\{1,2,\cdots,N\\}$ are then
concatenated and projected to a single output $H(\boldsymbol{X})$.

UT-Adapters are placed in parallel with MHSA and MLP modules. In particular, a
UT-Adapter is a stack of linear and 1D-convolutional layers. As shown in Fig.
3 (c), the output of a UT-Adapter is operated as

$U(\boldsymbol{X})=L_{2}(P(C(P(L_{1}(\boldsymbol{X};\boldsymbol{W_{1}}));\boldsymbol{W_{C}}));\boldsymbol{W_{2}}),$
(3)

where $L_{1}$ and $L_{2}$ are _Linear 1_ and _Linear 2_ layers in Fig. 3 (c)
and $P$ and $C$ are _Permutation_ and _Conv-1D_ , respectively.
$\boldsymbol{W_{1}}\in\mathbb{R}^{D\times 128}$ and
$\boldsymbol{W_{2}}\in\mathbb{R}^{128\times D}$ are trainable weights of
$L_{1}$ and $L_{2}$. $\boldsymbol{W_{C}}$ is the 1D-convolutional weight with
the kernel size of 3. $L_{1}$ projects the input
$\boldsymbol{X}\in\mathbb{R}^{L\times D}$ to
$\boldsymbol{X}\in\mathbb{R}^{L\times 128}$, Permutation layer $P$ shifts
$\boldsymbol{X}\in\mathbb{R}^{L\times 128}$ to
$\boldsymbol{X}\in\mathbb{R}^{128\times L}$ and $C$ is applied along the
temporal dimension to capture the temporal dynamics. $P$ layer and $L_{2}$
then project $\boldsymbol{X}\in\mathbb{R}^{128\times L}$ to
$\boldsymbol{X}\in\mathbb{R}^{L\times D}$.

To be specific, for $F_{V}$, the output from MHSA and a UT-adapter can be
represented as:

$\small\boldsymbol{X^{{}^{\prime\prime}}_{v}}=AN(\boldsymbol{X_{v}}+H(LN(\boldsymbol{X_{v}}))+U(\boldsymbol{X_{v}})),$
(4)

and $\boldsymbol{X^{{}^{\prime}}_{v}}$ in Eq. 1 is obtained by:

$\small\boldsymbol{X^{{}^{\prime}}_{v}}=AN(\boldsymbol{X^{{}^{\prime\prime}}_{v}}+MLP(LN(\boldsymbol{X^{{}^{\prime\prime}}_{v}}))+U(\boldsymbol{X^{{}^{\prime\prime}}_{v}})),$
(5)

where $AN$ is a normalization layer introduced after UT-Adapter, which is
trainable.

$G_{A}$ shares a similar structure with $F_{V}$. The output
$\boldsymbol{X_{a}}$ is computed by

$\small\begin{split}\boldsymbol{X^{{}^{\prime\prime}}_{a}}=AN(\boldsymbol{X_{a}}&+LN(H(\boldsymbol{X_{a}}))+U(\boldsymbol{X_{a}})),\\\
\boldsymbol{X^{{}^{\prime}}_{a}}=AN(\boldsymbol{X^{{}^{\prime\prime}}_{a}}&+LN(MLP(\boldsymbol{X^{{}^{\prime\prime}}_{a}}))+U(\boldsymbol{X^{{}^{\prime\prime}}_{a}})).\end{split}$
(6)

The input $\boldsymbol{X_{v}}$ and $\boldsymbol{X_{a}}$ are sequence
embeddings from vision and audio. The MHSA module in the transformer encoder
facilitates global attention, allowing the model to effectively learn both
spatial and temporal attention from the input. However, this may not be
optimal for learning local temporal attention and local spatial information.
Thus, UT-Adapter is proposed to capture the local temporal attention in
parallel with MHSA and MLP modules. The 2D-CNN and 1D-CNN capture the local
spatial information. We empirically show that this adapter-transformer
architecture is parameter-efficient tuning and delivers better performance.

###  4.2 Plug-in Audio-Visual Fusion (PAVF)

To explore the audio-visual interactions, we propose a Plug-in Audio-Visual
Fusion (PAVF) module that learns crossmodal attention for fusion.

We denote the output features from $V_{i}$ and $A_{i}$ as
$\boldsymbol{X_{v}^{i}}\in\mathbb{R}^{L\times D}$ and
$\boldsymbol{X_{a}^{i}}\in\mathbb{R}^{L\times D}$ and propose to learn
crossmodal attention in an embedding space with lower dimension, which is more
efficient and reduces computational costs. $\boldsymbol{X_{v}^{i}}$ and
$\boldsymbol{X_{a}^{i}}$ are first projected to
$\boldsymbol{X_{v}^{i}}\in\mathbb{R}^{L\times D^{{}^{\prime}}}$ and
$\boldsymbol{X_{a}^{i}}\in\mathbb{R}^{L\times D^{{}^{\prime}}}$,
$D^{{}^{\prime}}<D$, by linear projection layers. Crossmodal correlation
$\boldsymbol{P^{i}}\in\mathbb{R}^{L\times L}$ is learned to indicate the
importance between visual sequences and audio sequences. A specific visual
sequence may have a higher correlation to certain audio sequences in a video
clip, which offers important cues to the deception detection task. To
calculate $\boldsymbol{P^{i}}$, we introduce a trainable crossmodal
correlation weight matrix
$\boldsymbol{W_{P}^{i}\in\mathbb{R}^{D^{{}^{\prime}}\times D^{{}^{\prime}}}}$
and the calculation of $\boldsymbol{P^{i}}$ is

$\boldsymbol{P^{i}}=\boldsymbol{X_{v}^{i}}^{\intercal}\boldsymbol{W_{P}^{i}}\boldsymbol{X_{a}^{i}}.$
(7)

We further use the learned crossmodal correlation $\boldsymbol{P^{i}}$, to
perform $\boldsymbol{X_{v}^{i}}\rightarrow\boldsymbol{X_{a}^{i}}$ and
$\boldsymbol{X_{a}^{i}}\rightarrow\boldsymbol{X_{v}^{i}}$ attentions. The
attended features are

$\small\begin{split}\boldsymbol{\tilde{X}_{v}^{i}}=Softmax(\boldsymbol{P^{i}})\boldsymbol{X_{v}^{i}}+\boldsymbol{X_{v}^{i}},\\\
\boldsymbol{\tilde{X}_{a}^{i}}=Softmax(\boldsymbol{P^{i}}^{\intercal})\boldsymbol{X_{a}^{i}}+\boldsymbol{X_{a}^{i}},\end{split}$
(8)

where $\boldsymbol{P^{i}}^{\intercal}$ is the transpose of
$\boldsymbol{P^{i}}$, $\boldsymbol{\tilde{X}_{v}^{i}}\in\mathbb{R}^{L\times
D^{{}^{\prime}}}$, and $\boldsymbol{\tilde{X}_{a}^{i}}\in\mathbb{R}^{L\times
D^{{}^{\prime}}}$. Softmax is conducted column-wise. The attended features are
concatenated and inputted into a fusion head. The fusion head consists of a
linear projection $L_{p}$ that further reduces the embedding dimension to
$D^{\prime\prime}$, a normalization layer $LN$, and a non-linear activation
$ReLU$. It can be denoted as

$\small\boldsymbol{\tilde{X}_{va}^{i}}=ReLU(LN(L_{p}(\boldsymbol{\tilde{X}_{v}^{i}}\oplus\boldsymbol{\tilde{X}_{a}^{i}}))),$
(9)

where $\boldsymbol{\tilde{X}_{va}^{i}}\in\mathbb{R}^{L\times
D^{{}^{\prime\prime}}}$ and $\oplus$ is a concatenation operation.

PAVF modules are inserted into each pair of $V_{i}$ and $A_{i}$ to perform
fusion, in which richer fused information can be learned. The outputs
$\boldsymbol{\tilde{X}_{va}^{i}}$ from each PAVF module are then concatenated
and used for prediction.

###  4.3 Multi-task Learning

Multi-task learning has demonstrated promising performance in various
audiovisual tasks [19, 44], effectively enhancing the performance of multiple
tasks through mutual learning. With DOLOS’ MUMIN features (Fig. 2 (b)), we
enhance performance through multi-task learning, simultaneously conducting two
prediction tasks using fused multimodal features. More clearly, for $K$ MUMIN
features labels and one deception-truth label, we predict $K+1$ labels. Note
that all the labels are binary. We denote the fused audio-visual embedding as
$\boldsymbol{\tilde{X}_{va}}$. The classification layer takes
$\boldsymbol{\tilde{X}_{va}}$ as input to produce prediction scores
$S\in\mathbb{R}^{1\times(K+1)}$. Given the ground-truth label
$Y\in\mathbb{R}^{1\times(K+1)}$, we use cross-entropy loss to perform multi-
task learning. It can be formulated as

$\small\mathcal{L}_{M}=-\sum_{k=1}^{K+1}\left(Y_{k}\log(S_{k})+(1-Y_{k})\log(1-S_{k})\right),$
(10)

where $\mathcal{L}_{M}$ is the sum of the cross-entropy losses for each
element in $S$ and $Y$.

Modalities | 3-Fold Average | Duration Protocol | Gender Protocol
---|---|---|---
ACC | F1 | AUC | ACC | F1 | AUC | ACC | F1 | AUC
Visual | 61.44 | 69.42 | 58.89 | 61.03 | 72.01 | 56.51 | 59.37 | 64.19 | 54.94
Audio | 59.19 | 73.46 | 52.54 | 58.24 | 71.83 | 52.38 | 52.62 | 63.22 | 51.08
Concatenation | 61.62 | 70.2 | 60.5 | 60.8 | 69.63 | 57.79 | 58.0 | 66.22 | 53.75
Gainsboro!60 Fusion (PAVF) | 64.75 | 71.2 | 62.71 | 62.43 | 70.04 | 59.92 | 58.28 | 65.41 | 53.31
Gainsboro!60 PAVF + Multi-task | 66.84 | 73.35 | 64.58 | 64.48 | 71.09 | 62.44 | 59.04 | 66.84 | 55.1

Table 2: Multimodal deception detection on DOLOS dataset. The metrics are ACC
(%), F1 (%), and AUC(%).

Method Train Data - DOLOS Train Data - Box of Lies Test BgOL Test RLT Test
BgOL Test RLT V RN18+LSTM 55.08 (+4.62) 56.62 (+3.85) 50.46 52.77 3D RN 55.69
(+3.38) 57.87 (+5.53) 52.31 52.34 Ours (Vision) 56.92 (+4.30) 55.32 (+2.55)
52.62 52.77 A RN18 ${\dagger}$ 55.84 (+4.91) 53.53 (+1.64) 50.93 51.89 MLP
(MFCC) 52.31 (+2.46) 52.34 (+0.42) 49.85 51.92 Ours (Audio) ${\ddagger}$ 55.39
(+4.00) 59.15 (+4.68) 51.39 54.47 V+A 3D RN+RN18 50.77 (+1.23) 54.47 (+1.70)
49.54 52.77 Ours (PAVF) 57.54 (+4.31) 56.17 (+2.98) 53.23 53.19

Table 3: Cross-testing on Bag-of-Lies (BgOL) and Real-Life-trials (RLT)
datasets. The metric is ACC (%). RN stands for ResNet. All the visual features
are from face frames. ${\dagger}$ is trained on mel-spectrogram and
${\ddagger}$ is trained on raw audio.

##  5 Experimental Results

###  5.1 Implementation Details

Data pre-processing. For each video clip, we evenly sampled $L=64$ images and
cropped the face areas using MTCNN face detector [52]. These images were
normalized and resized to $160\times 160$ pixels. The raw speech audio was
resampled such that the W2V2 feature extractor outputs $L=64$ tokens. No data
augmentation was applied to both modalities.

Model details. We used ImageNet pre-trained ViT as the backbone network for
visual modality. We tokenized face images with a 2D-CNN module, which resulted
in a feature with a dimension of $64\times 256$. For the audio modality, we
adopted the pre-trained W2V2 model. The raw audio was tokenized by the 1D-CNN
module and the feature size was $64\times 512$ for each audio sample. Using a
linear projection layer, visual and audio tokens were projected to $64\times
768$ dimensions. We empirically utilized the first 4 transformer encoder
layers from ViT and W2V2 models to extract features because deception
detection can benefit from the low-level features [43, 28, 13]. The UT-Adapter
and PAVF modules were inserted in these four encoder layers (in Fig. 3 (a)).
The one-dimensional convolution layer (in Fig 3 (c)) in the UT-Adapter for
both visual and audio encoders had a kernel size of 3 and a stride of 1.

Training and evaluation. The models were trained with cross-entropy loss. We
used Adam optimizer and trained the models for 20 epochs with a learning rate
of 3e-4 and a batch size of 16. We evaluated our method with accuracy (ACC),
F1 score (F1), and Area Under Curve (AUC).

###  5.2 Audio-visual Deception Detection in DOLOS

The performance of PECL model on DOLOS is presented in Table 2. The average
results were calculated from the 3 folds defined in the train-test protocol.
Duration and gender results were calculated based on long-short and male-
female protocols. For unimodal results, the visual modality showed slightly
better performance than the audio across all the protocols. To study the
performance of a simple feature-level fusion, we concatenated the last output
embeddings from visual and audio encoders to perform prediction. The
concatenation usually draws the average of unimodal predictors. To overcome
this issue, the proposed PAVF module learned the correlation between visual
and audio features from different encoder layers and improved the performance.
The multi-task learning further boosted the accuracy across all the protocols.
However, by comparing the overall performances, we observed that the duration
and gender factors showed an adverse effect on performance. This experimental
comparison pointed out the importance of duration and gender factors in
deception detection.

Features Method ACC (%) V Open Face [32, 28] LSTM 56.81 Action Units (AU) [32,
4] LSTM 57.47 Facial Affect [32, 24] LSTM 57.67 MUMIN Features [35, 36] MLP
58.84 Face [25, 9, 50, 13, 28, 24] RN18+LSTM 59.55 Gainsboro!60 Face
Gainsboro!60 Ours (Vision) Gainsboro!60 61.44 A MFCC [50] MLP 56.86 Open SMILE
[32, 13, 51] MLP 57.49 Gainsboro!60 Speech Audio [25, 9, 28, 24]
Gainsboro!60Ours (Audio) Gainsboro!60 59.19 Open Face + Open SMILE [32] Score-
Fusion 59.75 V+A Face + Open SMILE [13, 28, 51] Score-Fusion 60.00
Gainsboro!60 Face + Speech [25, 24, 9] Gainsboro!60Ours(Concat) Gainsboro!60
61.62 Gainsboro!60 Face + Speech Gainsboro!60 Ours(PAVF) Gainsboro!60 64.75

Table 4: Benchmarking DOLOS dataset on visual and audio features. The weighted
average is used for score fusion. RN18 stands for ResNet-18.

###  5.3 Comparison with Box of Lies

To compare DOLOS with the current gameshow dataset BOL, we conducted cross-
testing, where DOLOS and BOL were used for training and other deception
detection datasets, _i.e._ , Bag of Lies and Real Life Trials, for testing. It
was challenging because these datasets were collected from different deceptive
scenarios. As shown in Table 3, we compared several methods for audio, visual,
and fusion and found that models trained on DOLOS achieved higher accuracies
overall. PECL model trained on DOLOS also showed better test accuracy than
that trained on BOL, (_e.g._ , Fusion (PAVF) improved 4.31% on BgOL and 2.98%
on RLT). These results suggest that models trained on DOLOS can be more
effective in identifying deceptive content in other scenarios than those
trained on BOL.

###  5.4 Benchmarking DOLOS

We benchmark DOLOS in Table 4, following the deception detection literature.
It is worth noting that the existing literature on deception detection is not
standardized with protocols, data processing, and feature extraction
methodologies. To highlight and cover the most representative works in
deception detection literature, we benchmarked DOLOS on the popular visual and
audio features used before for deception detection. The Open Face features (2D
facial landmarks, eye gaze, and head pose) and Action Units (AU) features were
extracted using the OpenFace toolkit [3]. The facial affect (emotion) features
were extracted using the Affectnet [33] model. For audio, the Mel Cepstral
Frequency Coefficients (MFCC) and open SMILE features were extracted by using
the openSMILE toolkit [11].

Performance. For visual modality, the RGB face images gave the best
performance, since they contained more deceptive cues. This was followed by
manually annotated MUMIN features, affect features, and AU features, which all
represented high-level human expressions. The low-level facial landmark
features from Open Face did not deliver good performance. For audio modality,
raw speech audio with rich deceptive cues performed the best. The openSMILE
features performed better than MFCC, since it extracted nearly three times the
audio features compared with MFCC. We also performed simple score-level fusion
for the best performing features in the two modalities. We can observe that
the proposed transformer-based deception detector achieved the highest
performance in both unimodal learning and multimodal setting.

###  5.5 Ablation Study

Here we conducted detailed ablation studies on different components of PECL
model. The analysis was based on the train-test protocol.

Figure 4:  Ablation on (a) encoder depth and (b) type of adapter.

Method Trainable Total Trainable/Total Adapter PAVF Total Fine-Tune $-$ 1.982
70.180 70.180 100% NLP [17] 5.345 1.982 7.327 73.347 9.99% UT-A 3.077 1.982
5.059 71.079 7.12%

Table 5: Comparison of the number of trainable parameters and total parameters
(Millions).

Position $\shortparallel$ MHSA $\shortparallel$ FFN $\shortparallel$ MHSA
$\shortparallel$ FFN MHSA $\triangle$ FFN ACC (%) 64.75 63.88 63.37 64.39

Table 6: Ablation study on UT-Adapter positions. $\shortparallel$ indicates
“in parallel with” and $\triangle$ indicates “between”.

Impact of encoder depth and adapter types. Fig. 4 (a) indicated that the
proposed PECL performed best with four encoder layers, with performance
decreasing when additional encoder layers were added, which may due to
overfitting. Fig. 4 (b) showed that UT-Adapter outperformed the NLP adapter
[17] and full fine-tuning.

Comparison of parameter amount. Table 5 demonstrated that our proposed method
achieved the best performance while utilizing the least amount of trainable
parameters when compared to both the NLP adapter and full fine-tuning
approaches.

Impact of adapter position. The ablation study for UT-Adapter positions inside
the transformer encoders is presented in Table 6. The results showed that the
UT-Adapter worked the best when parallel to both MHSA and FFN layers [21], as
they captured the temporal attention for both MHSA and MLP. The performance
dropped when parallel to only either layer. The performance was also close for
UT-Adapter placed inbetween MHSA and FFN layers.

Figure 5: Ablation on numbers of PAVF modules. Figure 6: Results of multi-task
learning with different features.

Impact of PAVF modules. Fig. 5 shows the ablation results of the number of
PAVF modules used. For example, two PAVF modules correspond to the last two
audio-visual encoders (encoders 3 and 4), while the first two encoders operate
without them. The advantage of fusion in earlier stages [34] can be clearly
seen with a substantial increase in performance for four PAVF modules, where
the low-level crossmodal interactions were captured. The results revealed that
four PAVF modules are particularly helpful in capturing audio-visual deceptive
cues.

Impact of multi-task learning. In Fig. 6, we illustrated the ablation study of
multi-task learning described in Sec 4.3, for different audio-visual features
(Refer Fig 2 (b)). The results showed that using 20 visual features (20V)
achieved better results than using 5 audio features (5A) and using 25 audio-
visual features further boosted the deception detection accuracy. The results
demonstrated that the manually annotated features in DOLOS are beneficial for
deception detection by multi-task learning.

##  6 Conclusion

In this paper, we introduced DOLOS, the largest gameshow deception dataset
with fine-grained audio-visual feature annotations. We also proposed
Parameter-Efficient Crossmodal Learning, where the UT-Adapter learns temporal
attention for both visual and audio modalities, and the PAVF module captures
correlation information between audio and visual modalities. We benchmarked
our dataset on the existing deception detection approaches. Extensive
experiments showed that DOLOS has better quality than the Box of Lies and PECL
model achieved better results on DOLOS and can be applied to other deceptive
content. In the future, we will also investigate language modality and other
open issues such as domain generalization in deception detection. We hope our
work can inspire the community and attract more attention to the deception
detection task.

## References

  * [1] Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo.  Detecting deceptive behavior via integration of discriminative features from multiple modalities.  IEEE Transactions on Information Forensics and Security, 12(5):1042–1055, 2016.
  * [2] Jens Allwood, Loredana Cerrato, Laila Dybkjaer, Kristiina Jokinen, Costanza Navarretta, and Patrizia Paggio.  The mumin multimodal coding scheme.  NorFA yearbook, 2005:129–157, 2005.
  * [3] Brandon Amos, Bartosz Ludwiczuk, Mahadev Satyanarayanan, et al.  Openface: A general-purpose face recognition library with mobile applications.  CMU School of Computer Science, 6(2):20, 2016.
  * [4] Danilo Avola, Luigi Cinque, Gian Luca Foresti, and Daniele Pannone.  Automatic deception detection in rgb videos using facial action units.  In Proceedings of the 13th International Conference on Distributed Smart Cameras, pages 1–6, 2019.
  * [5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  Advances in Neural Information Processing Systems, 33:12449–12460, 2020.
  * [6] Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, and Debajyoti Sinha.  Beyond kappa: A review of interrater agreement measures.  Canadian journal of statistics, 27(1):3–23, 1999.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [8] Bella M DePaulo, James J Lindsay, Brian E Malone, Laura Muhlenbruck, Kelly Charlton, and Harris Cooper.  Cues to deception.  Psychological bulletin, 129(1):74, 2003.
  * [9] Mingyu Ding, An Zhao, Zhiwu Lu, Tao Xiang, and Ji-Rong Wen.  Face-focused cross-stream network for deception detection in videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7802–7811, 2019.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  International Conference on Learning Representations, pages 0–7, 2021.
  * [11] Florian Eyben, Martin Wöllmer, and Björn Schuller.  Opensmile: the munich versatile and fast open-source audio feature extractor.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462, 2010.
  * [12] Tommaso Fornaciari and Massimo Poesio.  Automatic deception detection in italian court cases.  Artificial intelligence and law, 21(3):303–340, 2013.
  * [13] Mandar Gogate, Ahsan Adeel, and Amir Hussain.  Deep learning driven multimodal fusion for automated deception detection.  In 2017 IEEE symposium series on computational intelligence (SSCI), pages 1–6. IEEE, 2017.
  * [14] Martin Graciarena, Elizabeth Shriberg, Andreas Stolcke, Frank Enos, Julia Hirschberg, and Sachin Kajarekar.  Combining prosodic lexical and cepstral systems for deceptive speech detection.  In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages 0–7. IEEE, 2006.
  * [15] Viresh Gupta, Mohit Agarwal, Manik Arora, Tanmoy Chakraborty, Richa Singh, and Mayank Vatsa.  Bag-of-lies: A multimodal dataset for deception detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–7, 2019.
  * [16] Julia Bell Hirschberg, Stefan Benus, Jason M Brenier, Frank Enos, Sarah Friedman, Sarah Gilman, Cynthia Girand, Martin Graciarena, Andreas Kathol, Laura Michaelis, et al.  Distinguishing deceptive from non-deceptive speech.  2005\.
  * [17] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [18] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * [19] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439–1449, 2021.
  * [20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [21] Shibo Jie and Zhi-Hong Deng.  Convolutional bypasses are better vision transformer adapters.  arXiv preprint arXiv:2207.07039, 2022.
  * [22] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  Nature, 596(7873):583–589, 2021.
  * [23] Manvi Kamboj, Christian Hessler, Priyanka Asnani, Kais Riani, and Mohamed Abouelenien.  Multimodal political deception detection.  IEEE MultiMedia, 28(1):94–102, 2020.
  * [24] Hamid Karimi, Jiliang Tang, and Yanen Li.  Toward end-to-end deception detection in videos.  In 2018 IEEE International Conference on Big Data (Big Data), pages 1278–1283. IEEE, 2018.
  * [25] Mohan Karnati, Ayan Seal, Anis Yazidi, and Ondrej Krejcar.  Lienet: a deep convolution neural networks framework for detecting deception.  IEEE Transactions on Cognitive and Developmental Systems, 2021.
  * [26] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 491–507. Springer, 2020.
  * [27] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020.
  * [28] Gangeshwar Krishnamurthy, Navonil Majumder, Soujanya Poria, and Erik Cambria.  A deep learning approach for multimodal deception detection.  arXiv preprint arXiv:1803.00344, 2018.
  * [29] Timothy R Levine and Steven A McCornack.  Theorizing about deception.  Journal of Language and Social Psychology, 33(4):431–440, 2014\.
  * [30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [31] E Paige Lloyd, Jason C Deska, Kurt Hugenberg, Allen R McConnell, Brandon T Humphrey, and Jonathan W Kunstman.  Miami university deception detection database.  Behavior research methods, 51(1):429–439, 2019.
  * [32] Leena Mathur and Maja J Matarić.  Introducing representations of facial affect in automated multimodal deception detection.  In Proceedings of the 2020 International Conference on Multimodal Interaction, pages 305–314, 2020.
  * [33] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 10(1):18–31, 2017.
  * [34] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  Advances in Neural Information Processing Systems, 34:14200–14213, 2021.
  * [35] Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo.  Deception detection using real-life trial data.  In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pages 59–66, 2015.
  * [36] Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, Yao Xiao, CJ Linton, and Mihai Burzo.  Verbal and nonverbal clues for real-life deception detection.  In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2336–2346, 2015.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [38] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf, 2018\.
  * [39] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [40] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [41] Felix Soldner, Verónica Pérez-Rosas, and Rada Mihalcea.  Box of lies: Multimodal deception detection in dialogues.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1768–1777, 2019.
  * [42] Jeremy Speth, Nathan Vance, Adam Czajka, Kevin W Bowyer, Diane Wright, and Patrick Flynn.  Deception detection and remote physiological monitoring: A dataset and baseline experimental results.  In 2021 IEEE International Joint Conference on Biometrics (IJCB), pages 1–8. IEEE, 2021.
  * [43] Anastasis Stathopoulos, Ligong Han, Norah Dunbar, Judee K Burgoon, and Dimitris Metaxas.  Deception detection in videos using robust facial features.  In Proceedings of the Future Technologies Conference, pages 668–682. Springer, 2020.
  * [44] Fei Tao and Carlos Busso.  End-to-end audiovisual speech recognition system with multitask learning.  IEEE Transactions on Multimedia, 23:1–11, 2020.
  * [45] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.  Mlp-mixer: An all-mlp architecture for vision.  Advances in Neural Information Processing Systems, 34:24261–24272, 2021.
  * [46] Nik Vaessen and David A Van Leeuwen.  Fine-tuning wav2vec2 for speaker recognition.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7967–7971. IEEE, 2022.
  * [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [48] Aldert Vrij and Pär Anders Granhag.  Eliciting cues to deception and truth: What matters are the questions asked.  Journal of Applied Research in Memory and Cognition, 1(2):110–117, 2012.
  * [49] Gemma Warren, Elizabeth Schertler, and Peter Bull.  Detecting deception from emotional and unemotional cues.  Journal of Nonverbal Behavior, 33(1):59–69, 2009.
  * [50] Zhe Wu, Bharat Singh, Larry Davis, and V Subrahmanian.  Deception detection in videos.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [51] Jun-Teng Yang, Guei-Ming Liu, and Scott C-H Huang.  Multimodal deception detection in videos via analyzing emotional state-based feature.  arXiv preprint arXiv:2104.08373, 2021.
  * [52] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE signal processing letters, 23(10):1499–1503, 2016.
  * [53] Zixing Zhang, Bingwen Wu, and Björn Schuller.  Attention-augmented end-to-end multi-task learning for emotion prediction from speech.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6705–6709. IEEE, 2019.