11institutetext: Helsinki Institute for Information Technology HIIT,
Department of Computer Science, Aalto University 22institutetext: Department
of Computer Science, University of Manchester 33institutetext: Bitville Oy,
Espoo, Finland

# Behaviour-conditioned policies for cooperative reinforcement learning tasks

Antti Keurulainen 1133 Isak Westerlund 33 Ariel Kwiatkowski 33
Samuel Kaski 1122 Alexander Ilin 11

###### Abstract

The cooperation among AI systems, and between AI systems and humans is
becoming increasingly important. In various real-world tasks, an agent needs
to cooperate with unknown partner agent types. This requires the agent to
assess the behaviour of the partner agent during a cooperative task and to
adjust its own policy to support the cooperation. Deep reinforcement learning
models can be trained to deliver the required functionality but are known to
suffer from sample inefficiency and slow learning. However, adapting to a
partner agent behaviour during the ongoing task requires ability to assess the
partner agent type quickly. We suggest a method, where we synthetically
produce populations of agents with different behavioural patterns together
with ground truth data of their behaviour, and use this data for training a
meta-learner. We additionally suggest an agent architecture, which can
efficiently use the generated data and gain the meta-learning capability. When
an agent is equipped with such a meta-learner, it is capable of quickly
adapting to cooperation with unknown partner agent types in new situations.
This method can be used to automatically form a task distribution for meta-
training from emerging behaviours that arise, for example, through self-play.

###### Keywords:

Agent behaviour Cooperative AI Deep Reinforcement Learning

##  1 Introduction

In many real-world applications, the ability of AI agents to cooperate with
each other and humans is of crucial importance. It is especially important to
have the capability to cooperate with different types of agents with different
characteristics and behavioural patterns. In practice, this calls for
functionalities to infer the behaviour of the partner agent and then adapting
to the inferred agent type for common good. Furthermore, it is useful to infer
the behaviour of an unknown partner agent as fast as possible during the
ongoing task. Such situations arise, for example, when self-driving cars try
to adapt to the various driving styles of the surrounding (human) drivers, or
when an AI teacher tries to adapt to the skill level of the human student.

Human cognition studies suggest that predictions on other peoples’ behaviour
are based on high-level models, such as mental states and abstractions [1, 3,
14]. Such high-level abstractions may be, for example, intentions, goals,
desires or some other factors describing the mental state. Taking inspiration
from human cognition, our goal is to enable an agent to infer the latent
variable that explains the behaviour of the partner agent, and then make use
of the inferred information during the ongoing task. In practice, the latent
variable needs to be low-dimensional since it needs to be inferred from short-
term behaviour. For example, it might be known that the behaviour of the
partner agent depends heavily on the skill level of the partner agent, which
may be unknown and therefore needs to be inferred.

In our method, we synthetically produce partner agent populations that are
used for training a meta-learner, which can be integrated as part of the agent
policy. As a result, an agent gains the capability of ”learning to learn” and
can quickly adapt to unknown agent types in new situations. Self-play has been
shown to create an automatic curriculum for learning and to produce emerging
behaviours [27, 28], thus it is a good candidate to synthetically produce
populations with different behavioural patterns. Since the agents are
synthetically generated, the process also produces appropriate ground truth
data of the partner agent behaviour that can be used for training the meta-
learner.

We frame this setting as a deep meta-reinforcement learning problem [8, 12,
26], where the partner agent behaviour is embedded in the transition dynamics
of the Markov Decision Process (MDP). During the training phase of the meta-
learning process, an agent is partnered with different partner agent types in
different instances of the environment. A detailed description of the meta-
learning approach is described in section 3.

Our method is also based on the assumption that the two tasks, of inferring
the partner agent behaviour, and of conducting the actual task, have different
requirements and hence need different solutions. In particular, the former
requires observing the agent for some time period and hence requires memory in
its policy. A common way for introducing memory in neural networks is by
allowing recurrent connections, which brings challenges such as vanishing and
exploding gradients and credit assignment [4]. On the other hand, many
cooperative tasks can be executed without memory, once the policy network is
conditioned on the partner agent behaviour. As an example, if the partner
agent is closer to a specific item to be collected in a gridworld, it is
useful to let the partner agent collect the item, if it is known to have the
skills to do it.

In our approach, which we call the behaviour-conditioned policy, we suggest a
separate dedicated network with memory that infers the agent behaviour quickly
during the first steps of the task execution and a separate policy network
without memory, which is conditioned on the inferred behaviour. As a result,
the actual policy is easier to train as it can be implemented by a simpler
feedforward network without recurrent connections. Both networks can be
trained separately by using the synthetically produced agent populations
together with the ground truth data of their behaviour. During the execution,
the ground truth data is replaced by the predictions of the partner agent
behaviour, resulting in a policy that is not dependent on the ground truth
information that is used for training.

The contributions of this paper can be summarized as:

  * •

A method to automatically generate a task distribution and associated ground
truth data from scratch by self-play for training a meta-learner.

  * •

An architecture and training method that can efficiently use the generated
task distribution and the ground truth data.

We demonstrate the capabilities of our approach in two different types of
experiments. In the first experiment, we compare the performance against the
$RL^{2}$ meta-reinforcement learning architecture [8] in a simple matrix game.
In the second, more complex environment, an agent solves a travelling salesman
gridworld (TSG) task in cooperation with a partner agent of an unknown type,
and we show higher performance when compared to an end-to-end solution with an
LSTM architecture.

##  2 Related work

There is a long history of research in opponent modelling [22]. For example,
[25] present an idea, where one agent predicts the behaviour of the other
agent by putting itself in a similar situation and [30] present a method of
breaking the symmetries in the underlying task and thus improving cooperation.
The former is an example where the partner agent is considered to be similar
to oneself, and the latter assumes that the partner agent is optimal but might
have converged to a different convention than oneself. In our case, we make
weaker assumptions as the partner agent is not assumed to be optimal. In [16]
and [24], methods to shape the learning of the other agent are presented. In
our work, we do not try to affect the learning of the partner agent, but
rather adapt to the behaviour of the agent, which is embedded in the
transition dynamics.

Applying centralized training with decentralized execution (CTDE) [6, 9] is an
active research area and it is an open problem how to best benefit from the
centralized training when private information is not available during
execution. For example, MADPPG [19] uses a centralized critic with
decentralized actors and [23] suggests a method based on using a centralised
critic and a specific version of the baseline in an actor-critic algorithm to
tackle the multi-agent credit assignment challenge. Since centralized critics
appear to be beneficial, we make use of this concept in our baselines and thus
reduce variance in the policy gradient estimates. Our method suggests a
natural extension of CTDE to the transition dynamics of the MDP.

Self-play has been shown to be a successful method for producing an automated
curriculum for learning in multi-agent reinforcement learning settings [11,
27, 28]. When training is conducted with a copy of itself, the cooperability
with other agent types with different behaviour might be limited. One possible
solution to improve generalization and robustness in self-play is to
interleave old versions of the policy in the training procedure. In [11], a
method to sample old opponents, which were stored during the self-play
training, was used to prevent overfitting to the latest policy and for
stabilizing the self-play training procedure. Also, [15] show that sampling
opponents from a pool of past versions of the policy improves the performance
and robustness of training agents in adversarial settings. Our method also
benefits from the sampling of old opponents, but instead of adversarial
training we use self-play to produce a suitable task distribution for meta-
learning.

One typical way to encourage neural networks to learn more useful
representations for a particular task, is to add auxiliary tasks to the
learning process [2]. For example, auxiliary tasks can be used to improve
representations in classification tasks [7, 13], or in the context of deep
reinforcement learning [10, 18, 29]. Our mechanism differs from the typical
way of using auxiliary heads, as instead of adding an extra auxiliary head to
the common representation, we train a distinct prediction network with a
supervised loss, and condition the policy network with this prediction.

##  3 Setting

We formalize our problem as a meta-reinforcement learning setting [8, 12, 26].
We synthetically construct a set of MDPs by training populations of agents
with self-play to obtain a distribution of agents with different behaviours.
We sample MDPs from a family $\mathcal{M}$ according to a meta learning
distribution $M_{i}\sim p(M)$. From the point of view of one agent, every MDP
has a different transition dynamics dictated by the behaviour of the
partnering agent. To be more specific, we define the MDP by the tuple
$M=\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma,\rho_{0},H\rangle$,
where $\mathcal{S}$ is the state space and $\mathcal{A}$ is the action space,
$\gamma$ is the discount factor, $\rho_{0}:\mathcal{S}\rightarrow[0,1]$ is the
initial state distribution and $H$ is the horizon. The transition function
$\mathcal{P}$ is defined as a function of the action space and the state space
as $\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$
and $\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow{\rm I\\!R}$ is the
reward function experienced by the agent.

Under this setting, the objective is to maximize the expected sum of
discounted rewards over the episodes $J(\pi_{\theta})=\mathbb{E}_{\tau\sim
p_{\pi}(\tau)}[\sum_{t=0}^{H}\gamma^{t}\mathcal{R}(s_{t},a_{t})]$, where
$\tau=(s_{0},a_{0},...)$ is the trajectory of states $s_{t}$ and actions
$a_{t}$ of the agent at time step $t$ for an episode of length $H$. The
initial state is sampled from the distribution $\rho_{0}$ and the agent
samples the action $a_{t}$ from its policy function $\pi(a_{t}|s_{t})$. The
next state is sampled from the transition dynamics function
$s_{t+1}\sim\mathcal{P}(s_{t+1}|s_{t},a_{t})$. The meta-MDP objective is to
find the policy that maximizes the expected returns over the task
distribution:
$\text{arg}\,\max\limits_{\theta}\mathbb{E}_{p(M)}[J(\pi_{\theta})]$.

##  4 Behaviour-conditioned policy

The two distinct structures of the task can be efficiently exploited by
constructing separate networks for both task structures, which we name the
task prediction and the policy. We synthetically produce populations of agents
together with ground truth information about their behaviour. Under the meta-
learning framework, each sampled MDP embeds the behaviour of the partner
agent, and the task prediction refers to predicting the behaviour of the
partner agent. Following the centralised training and decentralised execution
(CTDE) paradigm, we use this ground truth information during training, but not
during decentralised execution. The architectures for the training and
execution phases are shown in Fig 1.

The task prediction network is stateful and thus able to preserve the
representation of the trajectories $\tau_{i}=\\{s_{t}\\}_{t=0}^{l}$, where $i$
is the task index and $l$ is the length of the episode. The task prediction is
performed by a neural network $f_{\phi}$ as $\hat{T}_{i}=f_{\phi}(\tau_{i})$,
where $\hat{T_{i}}$ is the estimation of the task $i$. More specifically, the
task prediction network uses a standard LSTM architecture [5]. The task
prediction network is trained by minimizing the loss under the task
distribution $\mathcal{L}=\mathbb{E}_{p(M)}[L_{M}(f_{\phi})]$ by using the
ground truth task labels ${T}_{i}$.

During the training, the policy network is a feedforward network
$\pi=g_{\theta}(s_{i},T_{i})$. During the execution, the ground truth task
label ${T}_{i}$ is replaced by the prediction $\hat{T}_{i}$, produced by the
task prediction network.

Figure 1: Illustration of the training and execution architectures. Two
separate networks are trained with ground truth data to predict the task and
to run the policy. The execution is run by using the task prediction to allow
decentralised execution

##  5 Experiments

In this section, we present the experiments and their results. In these
experiments one agent (the main agent) is equipped with the behaviour-
conditioned policy, and adapts to the behaviour of the other agent (the
partner agent). The first experiment is a simple matrix game, where the task
is to adapt to the partner policy. The purpose of this experiment is to
investigate how well the suggested architecture can benefit from a
synthetically generated task distribution compared to a strong meta-learning
baseline. In the second experiment, we combine the behaviour conditioned
policy architecture with the creation of a meta-learning task distribution
with self-play. In essence, the task is a travelling salesman-type problem for
which self-play produces a distribution of agents of different skill levels.

###  5.1 Matrix game experimental setup

In this matrix game two agents perform actions sampled from their action
distributions. For each time step, the shared reward of their joint action is
defined by the payoff matrix, shown in Table 1.

For the partner agent, the action distributions are generated by drawing them
from a symmetric Dirichlet distribution $Dir(\alpha)$ with concentration
parameter $\alpha$. The main agent is equipped with the behaviour-conditioned
policy and it is thus able to learn during the episode the behaviour of the
unknown partner agent, and adjust its own policy accordingly.

One episode is fixed to ten steps, and the partner agent uses the same sampled
action distribution throughout the episode, and for each episode a new action
distribution is sampled. In one training iteration, 50 partner action
distributions are drawn and thus one iteration consists of 50 different tasks
in the meta-training scheme. During the training, the task prediction network
of the main agent learns to predict the task (the action distribution of the
partner agent) by using ground truth information, and the policy network
learns to conduct the task by using ground truth information of the action
distribution. During execution, the ground truth information is replaced by
the predictions from the task prediction network.

The difficulty of the task can be adjusted by altering the concentration
parameter $\alpha$. Small values of $\alpha$ create a high probability of one
single action and make the prediction of the partner agent behaviour easy.
Higher values of $\alpha$ create action distributions that are more even, in
which case predicting the partner agent action distribution is more difficult.

The payoff matrix is designed in such a way, that the better the
interpretation of the other agent behaviour, the higher the payoff. As an
example, predicting the partnering agent to take an action $p_{0}$ results in
high reward if predicted correctly, and a high negative reward if predicted
wrongly. If the prediction of the partner agent behaviour is uncertain, the
main agent is tempted to select the action $m_{4}$, a low risk but low reward
option, to avoid high negative rewards.

| $m_{0}$ | $m_{1}$ | $m_{2}$ | $m_{3}$ | $m_{4}$
---|---|---|---|---|---
$p_{0}$ | 1.0 | -0.7 | -0.4 | -0.1 | 0.0
$p_{1}$ | -1.0 | 0.8 | -0.4 | -0.1 | 0.0
$p_{2}$ | -1.0 | -0.7 | 0.6 | -0.1 | 0.0
$p_{3}$ | -1.0 | -0.7 | -0.4 | 0.4 | 0.0
$p_{4}$ | -1.0 | -0.7 | -0.4 | -0.1 | 0.2
Table 1: The payoff matrix for the matrix game. The partnering agent actions
represent the rows $p_{0}-p_{4}$, and the main agent selects the columns
$m_{0}-m_{4}$.

The prediction network uses the MSE loss function
$\mathcal{L}=\mathbb{E}_{p(M)}\|T-\hat{T}\|_{2}^{2}$ and the policy network is
trained using PPO loss [21], with separate networks for policy and value
predictions. In this experiment the input is augmented by the time step index,
in order to allow the network to perform value prediction. More details of the
network implementation is available in the appendix.

###  5.2 Matrix game results

We compare our algorithm against the $RL^{2}$ meta-reinforcement learning
algorithm [8], which is capable of performing end-to-end training in the
matrix game task. The expressive capacity of the behaviour conditioned policy
was designed to be similar to the baseline $RL^{2}$ implementation, see
details about the specific architectures in the appendix. Both methods receive
the ground truth information of the partner agent behaviour during training by
the use of a centralized critic, in order to reduce the variance of the policy
gradient estimates.

In order to compare the performance for various levels of task difficulty, the
experiment is run with six different values of the concentration parameter
$\alpha$. When $\alpha=0.01$ or $\alpha=0.03$, the task is easy and can be
solved by both methods, but as the task is made more difficult by increasing
the value of $\alpha$, the behaviour-conditioned policy outperforms the end-
to-end alternative (Fig. 2). The results show that if the factor that explains
the behaviour is known, a separate network can infer the partner agent
behaviour fast enough with sufficient accuracy during an episode, and that
conditioning a simple feedforward policy with the inferred behaviour reaches
higher mean reward and more stable training compared to the baseline, as is
shown in Fig 2.

Figure 2: Results of the matrix game experiment. The behaviour-conditioned
policy is able to learn to predict the partner agent behaviour during an
episode, and use that information in its policy function efficiently. The
results show that the baseline $RL^{2}$ method does not reach as high mean
reward, and the variance across the seeds is considerably higher. Lines are
mean values over five random seeds and error bars indicate the standard
deviation. The rewards are measured at the last time step of the episodes.

###  5.3 Travelling Salesman Gridworld (TSG) experimental setup

The TSG experiment is designed to encourage cooperation and quick task
inference during the execution. In this task, two agents collect subgoals
together in a gridworld (Fig. 3), and once all subgoals have been collected,
either one of the agents needs to collect the final goal.

Figure 3: Illustration of the 11x11 gridworld. Red and blue squares are the
two agents, black circle is the final goal, green circles are the subgoals.

Both agents have four possible actions: move up, down, left or right. The
actions of both agents are fed to the environment simultaneously at each time
step. Each step causes a small negative reward (-0.01), and collecting goals
results in a positive reward (0.05). An episode ends when the final goal is
collected or if the maximum amount of steps have been taken. As our setting is
fully cooperative, both agents share the same reward.

For this task, several populations of agents with various skill levels are
trained using self-play and by collecting agents from different phases of the
training. Our solution tackles the non-stationarity problem [17, 31] by
running the training in two phases. During the first phase (clone training)
the self-play training is conducted with the most recent versions of the
policy. During the second phase, the partner is sampled from the older
versions of the policy, but only the parameters of the latest version of the
policy are updated. Thus, during the second phase, which we call co-op
training, only one agent keeps on learning, thus avoiding the non-stationarity
problem. We produce populations with three different skill levels: fully
trained (skilled), untrained (novice) and a skill level that reaches a
specified intermediate performance (intermediate).

The state $s_{t}$ from the environment consist of the gridworld coordinates of
all objects and flags indicating whether certain subgoals are already
collected. We use a modified relation network [20] to produce improved
representations of the states in all networks during the training and
execution. In this experiment, the different tasks refer to cooperating with
partner agents with various skill levels.

The skill prediction network uses a standard LSTM implementation. We minimize
the negative log likelihood under the task distribution
$\mathcal{L}^{XE}=\mathbb{E}_{p(M)}[-log(\hat{T_{i}})]$, where $\hat{T_{i}}$
is the prediction of the correct task. The policy network is a feedforward
network with PPO loss and with separate policy and value networks. More
details of the implementation is available in the appendix.

###  5.4 Travelling Salesman Gridworld (TSG) results

In the TSG experiment, we compare the behaviour-conditioned policy against a
baseline LSTM policy with a similar capacity which can learn the required
functionality in an end-to-end manner, see the appendix for details. As with
the matrix game experiment, the baseline LSTM policy and the behaviour
conditioned policy receive the ground truth information about the partner
agent behaviour and use it in the form of a centralized critic. All
populations that are trained with five random seeds, are partnered during
evaluation with agents from an additional reference population that is not
seen during the training in order to penalize overfitting to the behaviours
present within the training population.

The results in Table 2 show that both the behaviour conditioned policy and
baseline reach strong absolute performance with all partner types when
comparing to the optimal policy. Furthermore, the performance of both methods
consistently improves as the partner agent becomes more skilled. This shows
that the meta-learning distribution generated through self-play not only
produces strong co-operative agents, but also allows for gaining the ability
to quickly adapt based on the behaviour of the partner agent in novel
situations. The results also show that the behaviour conditioned policy
outperforms the baseline for all partner types. This indicates that the
inductive bias present for exploiting the two-fold structure of the task is
useful for fast adaptation, and that the labels produced during self-play can
be used as an additional training signal in parallel to their typical use for
a centralized critic.

Method | partner type | Mean episode length | Mean return
---|---|---|---
Behaviour-conditioned policy | skilled | 15.9 $\pm 0.1$ | 0.080 $\pm 0.001$
Behaviour-conditioned policy | intermediate | 22.9 $\pm 0.1$ | 0.011 $\pm 0.001$
Behaviour-conditioned policy | novice | 28.0 $\pm 0.4$ | -0.040 $\pm 0.004$
LSTM policy | skilled | 17.8 $\pm 0.2$ | 0.062 $\pm 0.002$
LSTM policy | intermediate | 25.4 $\pm 0.8$ | -0.015 $\pm 0.009$
LSTM policy | novice | 31.8 $\pm 0.6$ | -0.084 $\pm 0.007$
Optimal | skilled | 12.9 | 0.121
Optimal | novice | $26.2^{*}$ | $-0.012^{*}$
Table 2: Main results for simulations in 11 x 11 gridworld, two agents, four
subgoals and one final goal. The results show mean values and standard
deviations over five random seeds. *This is optimal when one agent collects
everything. In our simulations, the other novice agent can accidentally pick
up subgoals or the final goal, meaning that the simulation can exceed this
value.

##  6 Discussion

In this paper, we introduced ways to improve cooperation with an unknown
partner agent type by synthetically generating population of agents and
training a meta-learner by using these populations. Furthermore, we suggested
an architecture that exploits efficiently the two-fold structure that is
common in many real world scenarios.

The results showed that self-play can be used to construct useful task
distributions for meta-learning the ability to quickly adapt to different
partner types in novel cooperative situations. The results additionally
indicate that utilizing data produced by self-play together with suitable
inductive biases further improves the performance.

Our experiments covered cases where the latent variable that explained the
behaviour of the partner agent was the skill level of the partner agent.
Depending on the tasks, many other latent variables could be identified. An
interesting research direction would be to explore methods for incorporating
the identification of important factors as a part of the autocurriculum.

##  7 Acknowledgements

This work was supported by the Academy of Finland (Flagship programme: Finnish
Center for Artificial Intelligence FCAI, grants 319264, 292334).

## References

  * [1] David Premack and Guy Woodruff  “Does the chimpanzee have a theory of mind?”  In _Behavioral and brain sciences_ 1.4 Cambridge University Press, 1978, pp. 515–526
  * [2] Steven C Suddarth and YL Kergosien  “Rule-injection hints as a means of improving network performance and learning time”  In _European Association for Signal Processing Workshop_ , 1990, pp. 120–129  Springer
  * [3] Alison Gopnik and Henry M Wellman  “Why the child’s theory of mind really is a theory”  In _Mind & Language_ Blackwell Publishing Ltd, 1992
  * [4] Yoshua Bengio, Patrice Simard and Paolo Frasconi  “Learning long-term dependencies with gradient descent is difficult”  In _IEEE transactions on neural networks_ IEEE, 1994
  * [5] Sepp Hochreiter and Jürgen Schmidhuber  “Long short-term memory”  In _Neural computation_ 9.8 MIT Press, 1997
  * [6] Frans A Oliehoek  “Decentralized pomdps”  In _Reinforcement Learning_ Springer, 2012
  * [7] Antti Rasmus et al.  “Semi-supervised learning with ladder networks”  In _Advances in Neural Information Processing Systems_ , 2015
  * [8] Yan Duan et al.  “${RL}^{2}$: Fast reinforcement learning via slow reinforcement learning”  In _arXiv preprint arXiv:1611.02779_ , 2016
  * [9] Landon Kraemer and Bikramjit Banerjee  “Multi-agent reinforcement learning as a rehearsal for decentralized planning”  In _Neurocomputing_ Elsevier, 2016
  * [10] Piotr Mirowski et al.  “Learning to navigate in complex environments”  In _arXiv preprint arXiv:1611.03673_ , 2016
  * [11] David Silver et al.  “Mastering the game of Go with deep neural networks and tree search”  In _nature_ Nature Publishing Group, 2016
  * [12] Jane X Wang et al.  “Learning to reinforcement learn”  In _arXiv preprint arXiv:1611.05763_ , 2016
  * [13] Yuting Zhang, Kibok Lee and Honglak Lee  “Augmenting supervised neural networks with unsupervised objectives for large-scale image classification”  In _International conference on machine learning_ , 2016  ICML
  * [14] Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe and Joshua B Tenenbaum  “Rational quantitative attribution of beliefs, desires and percepts in human mentalizing”  In _Nature Human Behaviour_ Nature Publishing Group, 2017
  * [15] Trapit Bansal et al.  “Emergent complexity via multi-agent competition”  In _arXiv preprint arXiv:1710.03748_ , 2017
  * [16] Jakob N Foerster et al.  “Learning with opponent-learning awareness”  In _arXiv preprint arXiv:1709.04326_ , 2017
  * [17] Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag and Enrique Munoz Cote  “A survey of learning in multiagent environments: Dealing with non-stationarity”  In _arXiv preprint arXiv:1707.09183_ , 2017
  * [18] Max Jaderberg et al.  “Reinforcement learning with unsupervised auxiliary tasks”  In _5th International Conference on Learning Representations_ , 2017
  * [19] Ryan Lowe et al.  “Multi-agent actor-critic for mixed cooperative-competitive environments”  In _Advances in Neural Information Processing Systems_ , 2017
  * [20] Adam Santoro et al.  “A simple neural network module for relational reasoning”  In _Advances in Neural Information Processing Systems_ , 2017
  * [21] John Schulman et al.  “Proximal policy optimization algorithms”  In _arXiv preprint arXiv:1707.06347_ , 2017
  * [22] Stefano V Albrecht and Peter Stone  “Autonomous agents modelling other agents: A comprehensive survey and open problems”  In _Artificial Intelligence_ Elsevier, 2018
  * [23] Jakob Foerster et al.  “Counterfactual multi-agent policy gradients”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32.1, 2018
  * [24] Alistair Letcher et al.  “Stable opponent shaping in differentiable games”  In _arXiv preprint arXiv:1811.08469_ , 2018
  * [25] Roberta Raileanu, Emily Denton, Arthur Szlam and Rob Fergus  “Modeling others using oneself in multi-agent reinforcement learning”  In _International conference on machine learning_ , 2018
  * [26] Bradly C Stadie et al.  “Some considerations on learning to explore via meta-reinforcement learning”  In _Advances in Neural Information Processing Systems_ , 2018
  * [27] Bowen Baker et al.  “Emergent tool use from multi-agent autocurricula”  In _arXiv preprint arXiv:1909.07528_ , 2019
  * [28] Joel Z Leibo, Edward Hughes, Marc Lanctot and Thore Graepel  “Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research”  In _arXiv preprint arXiv:1903.00742_ , 2019
  * [29] Hengyuan Hu and Jakob N Foerster  “Simplified action decoder for deep multi-agent reinforcement learning”  In _ICLR_ , 2020
  * [30] Hengyuan Hu, Adam Lerer, Alex Peysakhovich and Jakob Foerster  ““Other-Play” for Zero-Shot Coordination”  In _International Conference on Machine Learning_ , 2020
  * [31] Thanh Thi Nguyen, Ngoc Duy Nguyen and Saeid Nahavandi  “Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications”  In _IEEE transactions on cybernetics_ IEEE, 2020

##  8 Appendix: Details of the neural network implementations

###  8.1 Matrix game experiment

The baseline RL2 implementation has an LSTM layer with 32 units, followed by a
MLP layer with 16 units and a linear layer with softmax. The value network
does not share any weights with the policy network, and it has an LSTM layer
with 16 units, followed by a MLP layer of 16 units and a linear layer.

The behaviour-conditioned policy network has two MLP layers with 6 and 16
units followed by a linear layer with softmax. The value network does not
share any weights with the policy network, and it has similar structure as the
policy network, except there is no softmax.

The task prediction network has an LSTM layer with 32 units, followed by one
MLP layer with 16 units and a linear layer with softmax.

###  8.2 Travelling salesman gridworld experiment

All networks use modified version of relation net [20], where the pairwise
relations are processed only between the agents and other objects, instead of
relations between all possible objects. The relation nets have 7 MLP layers
with the hidden size of 128 before the summation operation, and 2 MLP layers
of 64 hidden units after the summation. All LSTM layers have 64 hidden units.

The LSTM policy network (baseline) implementation has an LSTM layer on top of
the relation net and a softmax layer to produce action distribution. The value
network is a separate network with 2 MLP layers on top of the relation net.
The baseline was trained 3000 iterations of clone training, and another 1000
of co-op training, with a dataset batch size of 4000 time steps.

The skill prediction network has an LSTM layer on top of the relation net and
2 MLP layers with softmax layer. It was trained with 5000 training iteration
with a batch size of 4000 time steps.

The behaviour-conditioned policy network has 2 MLP layers and a softmax on top
of the relation net. It was trained 3000 iterations of clone training, and
another 1000 of co-op training, with a dataset batch size of 4000 time steps.

The optimizer is Adam with initial learning rate 5e-4 and scheduled learning
rate decrease.


scaletikzpicturetowidth[1]

# Automatic Classification of Subjective Time Perception Using Multi-modal
Physiological Data of Air Traffic Controllers

Till Aust1, Eirini Balta2, Argiro Vatakis2, and Heiko Hamann1 1Till Aust and
Heiko Hamann are with the Department of Computer and Information Science,
University of Konstanz, Konstanz, Germany<EMAIL_ADDRESS>Balta and Argiro Vatakis are with the Department of Psychology, Panteion
University of Social and Political Sciences, Athens, Greece.
<EMAIL_ADDRESS>
###### Abstract

One indicator of well-being can be the person’s subjective time perception. In
our project ChronoPilot, we aim to develop a device that modulates human
subjective time perception. In this study, we present a method to
automatically assess the subjective time perception of air traffic
controllers, a group often faced with demanding conditions, using their
physiological data and eleven state-of-the-art machine learning classifiers.
The physiological data consist of photoplethysmogram, electrodermal activity,
and temperature data. We find that the support vector classifier works best
with an accuracy of 79 % and electrodermal activity provides the most
descriptive biomarker. These findings are an important step towards closing
the feedback loop of our ChronoPilot-device to automatically modulate the
user’s subjective time perception. This technological advancement may promise
improvements in task management, stress reduction, and overall productivity in
high-stakes professions.

##  I Introduction

In a rapidly evolving world, prioritizing mental health becomes more crucial
especially among the working population [1]. Work can be a significant
stressor, which may reduce well-being, lower productivity, or even lead to
burn out. One indicator can be the subjective perceived passage of time, which
then passes slower [2]. When working individuals are positively engaged they
perceive time faster.

In our EU-funded project ChronoPilot [3], we are developing a device that
modulates human time perception to increase performance and well-being of
human workers. For this so-called ChronoPilot-device, we propose a control
loop (see Fig. 1), which interacts with the human using haptic [4], visual
[5], auditory [6], or situational stimuli [7], and gathers non-invasive
physiological data. Our objective in this paper is to present a methodology to
extract automated feedback on the user’s subjective perceived passage of time
out of the physiological data for the ChronoPilot-device. To do so, we derive
meaningful features from the physiological data, also called biomarkers, for
classification, similar to related fields of automated detection of stress [8]
or emotion [9].

A common approach to obtain automated feedback from humans is to collect their
physiological data using wearable sensing devices [10, 11, 12]. Montesinos et
al. [10] used wearable devices to implement multi-modal acute stress
recognition. We follow a similar pipeline as proposed by them, consisting of
signal acquisition, data processing, and model learning and evaluation. Unlike
other works in this domain, they tested on unseen data to validate the
generalizability of their trained classifiers. We propose to use leave-one-
subject-out cross-validation (validating our classifiers on data not used for
training) to prove generalizability.

ChronoPilot DeviceHumanControllerStimuliTP Feeback$-$TP inputOutput Figure 1:
Fundamental control loop of the envisioned ChronoPilot-device. TP Input is the
user’s desired time perception for maximal productivity and well-being. The
Controller produces Stimuli and applies them to the user. During the
stimulation, physiological data are collected, which are used as feedback for
the ChronoPilot-device to adjust the stimuli accordingly. Our objective is to
develop machine learning methods, which are able to provide this feedback by
classifying the perceived passage of time based on physiological data.

Masinelli et al. [13] use wearable sensors in combination with machine
learning methods to monitor cognitive load of individual users. We advance
this study by classifying for subjective perceived passage of time.

The only study we are aware of classifying subjective perceived passage of
time based on physiological signals was done by Orlandic et al. [11]. They
stimulated users with different video clips selected from the Emotional Movie
[14] and FilmStim [15] databases, which lead to different emotional states
(e.g., neutral, fear, sadness) along with cognitive tasks, such as
mathematical tasks or the Stroop Color Test (varies cognitive load through
congruent and incongruent stimuli). They collected skin temperature,
electrodermal activity, respiration, electrocardiogram, and photoplethysmogram
data. The feature extraction was done on non-overlapping 45-second slices,
which were normalized by subtracting the mean and dividing by the standard
deviation of the training dataset. They then applied a set of machine learning
classification algorithms. We go beyond their approach as we use data of a
more complex real-world scenario, namely flight control, instead of an in-lab
experimental setup. We assess the perceived passage of time using a
questionnaire instead of the relative estimation error. Our two main
contributions are: (1) an automatic multi-modal subjective perceived passage
of time perception estimation technique, using photoplethysmogram,
electrodermal activity, and temperature data measured by of-the-shelf wearable
sensors and (2) a comparison between eleven state-of-the-art machine learning
classifiers for subjective perceived passage of time classification in
combination with feature selection methods and feature importance analysis
using the SHapley Additive exPlanations (SHAP) [16] framework. We show
generalizability by validating our methods using leave-one-subject-out cross
validation (LOSOCV). We obtain accuracies of up to 79 % on real-world
physiological data obtained during flight controller helicopter training
sequences.

##  II Scenario and Methodology

We first describe the air traffic control scenario under various workload
manipulations in real-world conditions [17]. We present our overall
methodology consisting of data acquisition, preprocessing, and classification.
Preprocessing includes manual cleaning, filtering, segmentation, background
subtraction, and feature extraction. The classification consists of machine
learning classifiers and feature selection methods.

###  II-A Scenario

Twelve air traffic controllers (4 females; mean age = 36.9, SD = 6.6),
participated in the experiment. The participants were all professionals,
certified by the Civil Aviation Authority of the Ministry of Infrastructure
and Transport (Greece), and working at a Greek military airport. Their native
language was Greek, they all had an excellent knowledge of English as a second
language, and they had undergone the necessary training in air traffic
management terminology. In all experimental conditions, the air-traffic
controllers were wearing the EmotiBit device111https://www.emotibit.com and
completed a flying control session, consisting of a scheduled training flight
for pilots of military helicopters, where the pilots had to take-off, perform
a full circle over the airport, and land. During the cruising phase the
controllers only maintained visual contact with the helicopter, while during
the landing phase the controllers were actively engaged in the process. At the
end of each experimental session, each controller made a verbal time
estimation of the total session duration. Additionally, the controllers gave a
rating on how slow or fast they felt the time had passed during the session
(i.e., passage of time judgments) on an integer scale from 1 to 5. The level
of workload was varied by the number of helicopters controlled by the air
traffic controller, either 1 or 2, and the language of communication between
the controller and the pilot of the helicopter, either Greek or English. The
experiment took place on two consecutive days, between the hours of 08:00 am
and 02:00 pm with similar weather conditions. Each air traffic controller
participated in two sessions per day, for a total of four sessions. The order
of the sessions was randomized.

###  II-B Data acquisition

The experiment was conducted in the air traffic control premises located on a
military airport. The main equipment used was the communication device, and
the digital recorder of the control tower. The communication device was an
Integrated Communications (ICOM) VHF air-band base station radio, equipped
with a hand-held microphone that the controllers used to communicate with the
pilots. The EmotiBit open-source wearable device was used to measure the
cardiac and electrodermal activity throughout experimentation. The
3-wavelength photoplethysmogram (PPG) sensor was used to monitor blood volume
changes at a sampling rate of 25 Hz, and the electrodermal activity (EDA)
sensor was used to monitor skin conductance at a sampling rate of 15 Hz. The
physiological data were recorded on a secure digital card (SD card) for
offline processing. Hence, we have a total of 48 sequences with an average
length of 182 s. For preprocessing, we sort the sequences per participant and
increasing mental load, that is, from one helicopter communicating in Greek to
two helicopters communicating in English (non-native language).

###  II-C Data processing

We apply similar preprocessing steps to all obtained physiological
signals.222Code available at:
https://github.com/tilly111/chronopilot_feature_extraction First, we manually
clean the signals to remove artifacts caused, for example, by sensor movement.
Second, we filter the physiological signals to remove measurement noise and
split them into the desired segments for background subtraction. Finally, we
extract psychologically meaningful features from the segments, later used for
classification.

####  II-C1 Photoplethysmogram (PPG) data

Initially, we apply manual cleaning, which consists of removing the last eight
and 13 values of sequence 3 from participant 12 and sequence one of
participant five, which peaked due to artifacts from motion of the measuring
device. After the manual cleaning, we use the cleaning method from HeartPy
[18] with cutoff frequencies 0.7 Hz and 3.5 Hz allowing for heart rates from
42 bpm to 210 bpm in the bandpass filter of order three. The original sampling
rate is approximately 25 Hz. For a more stable feature calculation, we up-
sample the signal to approximately 100 Hz using the Fourier
method.333https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.resample.html
Thereafter, we extract 13 physiological features using the HeartPy toolkit for
PPG data.

####  II-C2 Electrodermal activity (EDA) data

Similar to the PPG data, we apply manual cleaning to remove peaks at the end
of the same sequences. The original sampling rate is approximately 15 Hz. For
a more stable feature calculation, we up-sampled the signal to approximately
$10^{4}$ Hz using the same Fourier method as for the PPG data. From the up-
sampled signal, we extract 6 physiological features using the NeuroKit2
toolkit [19] for EDA data. The toolkit automatically cleans the data before
extracting the features. We extend the signal used for background subtraction
of participant 6 in sequence 1 by one and a half seconds to have 10 s, which
is the minimum required length for processing the signal.

####  II-C3 Temperature (Thermopile)

The temperature of the thermopile and one reference surrounding temperature
measurement is sampled with a frequency of 7.5 Hz. We calculate $\delta$ the
mean difference between the reference and the thermopile temperature during
the whole experiment and the individual means. Further, we calculate $\delta$
the gradient of the difference and the total power of its power spectral
density (PSD). Hence, we extract a total of five features from the temperature
data. These are similar features to Orlandic et al. [11] with the difference
that they measure skin temperature, while we measure the body temperature.

####  II-C4 Normalization

We employed background subtraction to reduce inter-participant variances. For
the baseline interval, we use the time span between the start of the
experiment and the take-off of the helicopter(s). We extract the features from
the baseline interval to subtract them from the features extracted from the
whole experiment. On the background subtracted features, we performed min-max-
scaling. That is, we scaled the data based on the training data between 0 and
1 to eliminate the learning of spurious relations between feature scale and
setting. This normalization yielded better numerical stability during
training. We compare the min-max-scaling with z-score normalization and none
normalization as a baseline. The unseen test data are scaled using the
parameters derived from the training data.

####  II-C5 Labels

1 H, GR1 H, EN2 H, GR2 H, ENvery slowslowmediumfastvery fastslowfast Figure 2:
Violin plots of the perceived passage of time labels of the different settings
of the flight controller experiment (H: helicopter number, GR: Greek, EN:
English; $n=12$). The left side of the violin plot (orange) displays the
scaled label distribution from the questionnaire (scale is the left axis). The
right side (blue) shows the label distribution after thresholding (dashed
line) in the slow and fast class. The dots represent individual samples. We
observe that depending on the number of helicopters the majority of the
controllers perceived the passage of time as slow when there is a one
helicopter and fast when there are two helicopters.

For classification, we use the perceived passage of time estimation from the
controllers’ questionnaires obtained at the end of the experiment. To allow
for better comparability between questionnaire answers, we min-max-scale each
controller’s answers from “very slow” to “very fast”, see Fig. 2, left half
(orange). We sort the settings by speaking in Greek and one helicopter to
speaking in English and two helicopters as we suspect an increase in mental
load for speaking in a non-native language and an increased number of to-be-
controlled objects. This sorting suggest a binary classification problem,
namely, “slow” for one helicopter settings and “fast” for two helicopters
settings. By thresholding the scaled labels at medium ($>3$ is labeled as
fast), we obtain a binary classification problem with 26 sequences of class
slow and 22 sequences of class fast, see Fig. 2, right half (blue).

We train our models using LOSOCV, which means we train with all but one
participant and use this participant for validation (validating on unseen
data). Hence, we train each classifier in each setting 12 times and average
the results. We chose this validation strategy given the small dataset and
accurately estimating model performance is more important than computational
cost [20].

###  II-D Classification

####  II-D1 Classifiers

For classification, we follow a similar approach as Schmidt et al. [12]
proposed by combining the extracted features to feature vectors, which we feed
into our machine learning classifiers. Due to the small dataset, we avoid deep
learning classifiers as they are known to require immense amounts of data
[21]. That is why we use a similar set of state-of-the-art machine learning
(ML) classifiers as presented by Orlandic et al. [11], namely Support Vector
Classifier (SVC), Decision-tree Classifier (DTC), k-Nearest Neighbor
Classifier (KNN), Logistic Regression (LR), Gaussian Naive Bayes (GNB), Linear
Discriminant Analysis (LDA), Random Forest Classifier (RF), and eXtreme
Gradient Boosting (XGB). In addition, we test Quadratic Discriminant Analysis
(QDA), Ada Boost (AB), and Gradient Boosting (GB).

For most ML classifiers we use the scikit-learn library [22] implementation.
The eXtreme Gradient Boosting is implemented using the xgboost444eXtreme
Gradient Boosting classifier https://github.com/dmlc/xgboost package. The
hyperparameter tuning was done manually and the values were chosen
individually such that the performance for each classifier was
maximized.555Code available at:
https://github.com/tilly111/chronopilot_classification

####  II-D2 Feature selecting algorithms

We train our classifiers with all features and subsets of features (split by
physiological signals). Additionally, we test two feature selection
algorithms: sequential feature selection (SFS) and recursive feature
elimination with cross-validation (RFECV). These methods shrink the input
space and, thus, should simplify the classification problem and make the
classifiers smaller, which improves performance, reduces overfitting, and
enhances the interpretability of the model.

SFS adds (forward selection) or removes (backward selection) features from a
feature subset in a greedy fashion. The criteria at each stage is that the
estimator chooses the best feature to add or remove based on the cross-
validation score of the given ML classifier.

RFECV removes the least important feature at each stage by evaluating the
model’s performance using cross-validation. As the classifier iterates in each
stage through all possible features it is not greedy but should yield the best
possible feature set. Both feature selection algorithms are implemented using
the scikit-learn library [22].

##  III Results

In this paper we work towards providing automated feedback to the envisioned
ChronoPilot-device based on the physiological data of the human user. The
device would trigger the corresponding stimuli to modulate the user’s time
perception. The device would operate online, continuously, and individually
for each user in real-world settings. Hence, we have relaxed requirements for
classification accuracy; we could incorporate temporal preceding
classification results to boost certainty exploiting the temporal dependency
of the problem. If the system operates over longer periods of time, we can use
online learning techniques to optimize and adapt for the individual user to
improve performance even more. Despite the ambitious objective of our project,
here we only aim to show general feasibility. We evaluate a group of users
without fine-tuning for individuals. Hence, we are satisfied by performance
that is better than a randomized approach or choosing always the majority
class (here that constitutes a baseline accuracy of 54 %).

We test the eleven classifiers using different approaches of feature
selection: no/manual feature selection, SFS, and RFECV. In manual selection,
we select subsets of features split by the physiological signals they were
derived from. For each feature selection setting, we test none, min-max, and
z-score normalization. We found min-max normalization works best with respect
to our dataset and classifiers. Hence, for the analysis we focus only on min-
max normalization. We train and validate each classifier twelve times to
obtain a mean accuracy over all participants, see Tab. I. Due to limited
space, we only show results for subsets of PPG features, and PPG and EDA
features because they are best in accuracy. We employ SHAP [16] to find the
most descriptive features of the classifiers.

TABLE I: Overview of the mean accuracies over all participants obtained by our selected state-of-the-art classifiers using various feature selection algorithm and min-max normalization for all participants. The “*” marked classifiers can have varying results due to their implementation. Classifier | None | SFS | RFECV | PPG | PPG+EDA
---|---|---|---|---|---
SVC | 0.7917 | 0.7708 | 0.7083 | 0.6875 | 0.7917
DTC* | 0.5417 | 0.5625 | 0.5625 | 0.5833 | 0.5625
KNN | 0.75 | 0.6458 | N.A. | 0.7292 | 0.6458
GNB | 0.6042 | 0.5833 | N.A. | 0.6667 | 0.6042
LR | 0.5625 | 0.6875 | 0.6667 | 0.6875 | 0.7083
LDA | 0.7708 | 0.6458 | 0.75 | 0.7708 | 0.7917
QDA | 0.625 | 0.5625 | N.A. | 0.6042 | 0.625
RF* | 0.6875 | 0.5208 | 0.7083 | 0.6875 | 0.625
GB* | 0.6458 | 0.5417 | 0.625 | 0.6042 | 0.5833
AB | 0.6458 | 0.5 | 0.5833 | 0.6458 | 0.6458
XGB | 0.5625 | 0.4167 | 0.6458 | 0.6667 | 0.5833

###  III-A None and manual feature selection

The SVC reached the highest mean accuracy (79 %) for all features and together
with the LDA classifier a similar accuracy with features derived from PPG and
EDA. The worst accuracy is achieved using the decision tree classifier (54 %)
and XGB (56 %). When inspecting the individual accuracies, we find that
participant 4 behaves dissimilar to the rest of the participants. While the
SVC obtains 75 % to 100 % accuracy on all other participants, it obtains 0 %
for participant 4. This pattern, of performing much worse than all the other
participants, holds for all classifiers. Applying the SHAP analysis we see
that variability of tonic EDA is a meaningful feature for SVC, KNN, and LDA
(the three best performing classifiers). We observe that reducing the number
of features, for example, by only using PPG and EDA features, can increase the
classifiers performance.

###  III-B Sequential feature selection (SFS)

We allow the algorithm to select 12 out of the 24 features (default setting).
The mean accuracy is slightly lower than with out the feature selection
algorithm and the support vector classifier is performing best with a mean
accuracy of 77 %. Using the SHAP analysis on the support vector classifier
combined with SFS, we find that breathing rate, SCR Peaks Amplitude Mean, and
EDA Sympathetic are selected most often as a feature (11 out of 12 times) with
breathing rate having the highest mean($|$SHAP value$|$) (for simplicity, SHAP
value). For the LR classifier (second best performing classifier in this
setting) area of ellipse of sd1 and sd2 (s), interbeat interval (ibi), portion
of pairs of successive NNs that differ by more than 20 ms (pNN20), and EDA
Sympathetic are chosen most often (10 out of 12 times), with s having the
highest SHAP value. The feature that was used most often with all classifiers
is pNN20 with (avg. 9.09 out of 12 times). The least often picked feature is
number of occurrences of SCR (3.45 out of 12 times). The most descriptive
feature over all classifiers is s with a SHAP value of 0.5859. The least
descriptive feature is PSD power of the temperature data SHAP value = 0.030).
Overall, there is no feature that is dominant throughout all classifiers and
participants.

###  III-C Recursive feature elimination with cross-validation (RFECV)

This algorithm chooses the number of used features by itself using five-fold-
cross-validation (default setting). The accuracies are similar to the SFS
algorithm and slightly lower than not using a feature selection algorithm. The
LDA classifier is performing best with a mean accuracy of 75 %. The worst
performing classifier is the decision tree classifier (56 %). The k-nearest
neighbor classifier, Gaussian Naive Bayes, and quadratic discriminant analysis
cannot be used in combination with RFECV because they do not implement any
feature importance measure. Hence, we do not consider them for the RFECV
setting.

As the algorithm determines itself how many features to select, we observe
that s is the most often selected feature (avg. 9.75 out of 12 times) followed
by heart beats per minute (bpm) (avg. 9.375 out of 12 times), and standard
deviation of NN intervals (sdNN) (avg. 8.5 out of 12 times). The least often
selected feature is portion of pairs of successive NNs that differ by more
than 50 ms (avg. 3.5 out of 12 times) followed by root mean squared difference
of RR intervals (avg. 3.875 out of 12 times) (rmssd), and standard deviation
of differences of RR intervals (avg. 4 out of 12 times). The most descriptive
feature using the SHAP analysis is rmssd (outlier in LR classifier participant
6) followed by sd1, and bpm. The least descriptive feature is the breathing
rate followed by temperature gradient, and SCR Peaks Amplitude Mean. Similar
to the observation for the SFS algorithm, we do not observe any dominant
features throughout all classifiers and participants.

##  IV Discussion

We extracted 24 features from PPG, EDA, and temperature data collected during
air traffic controller training sequences. Based on these features we
estimated the air traffic controllers’ perceived passage of time under various
workload conditions using state-of-the-art machine learning classifiers. We
extend our analysis by two feature selection algorithms, SFS and RFECV,
because manually selecting feature subsets showed that reducing the used
features can simplify the problem and increase performance. Last, we applied
SHAP analysis to measure feature importance.

Our results show that eight out of eleven classifiers are able to classify the
perceived passage of time sufficiently enough (over 60 % accuracy) for
previously unseen participants. However, we noticed that not all participants
provide qualitatively equivalent feedback. For example, participant 4
consistently achieves low classification accuracies across all classifiers,
that is, participant 4 has a different physiological response to the applied
stimulus compared to all other participants. Considering SVC, LDA, and KNN for
participant 4, the features derived from the thermopile (temperature gradient
and PSD Power) have a larger influence than the features derived from EDA
(variability of tonic EDA), which are meaningful features for all other
participants. This implies that we can increase the classifier accuracy by
fine-tuning our classifiers to individual users, which could be done by means
of online learning, for example, during “adaptive configuration” of the
ChronoPilot-device.

####  IV-1 None and manual feature selection

The decision tree classifier has the worst performance because it uses only a
small subset of all possible features to build its decision tree. Even though
we use the best splitting strategy the dataset is too small so not all
features are required to optimally split the training dataset. This explains
the varying results between runs (also holds for the random forest and GB
classifier, which are ensembles of decision or regression trees). The SVC,
LDA, and kNN classifier performed best. For all of these classifiers
variability of tonic skin conductance level is a meaningful feature. This
finding can be explained by the fact that EDA is a proxy for quantifying
cognitive load [23] and the perceived passage of time labels correlate with
the experimental setting, that is, with the induced mental load. Other worse
performing classifiers rely less on EDA derived features causing less
accuracy. The other classifiers show no patterns in their use of biomarkers
based on the SHAP analysis. Interestingly, all types of physiological data are
incorporated for classifying. However, not all classifiers perform best when
provided with all available information, indicating that feature selection is
indeed a useful tool in particular combinations of scenarios and classifiers.

####  IV-2 Sequential feature selection

Using the SFS, half the features are chosen in a greedy approach. Accuracies
drop for all but the decision tree classifier and logistic regression
classifier. We achieve a higher accuracy using the DTC because SFS forced it
to use 12 features, which was not the case using no feature selection
algorithm (previously 6.4 features selected in average). The SFS caused to
disregard the second most descriptive feature sdNN of the LR classifier and
replace it with the previously hardly considered feature ibi, which may
explain its increased performance. Although the mean accuracy of SVC over all
participants slightly decreased from 79 % to 77 %, the SFS was able to
increase the minimal accuracy to 50 % (reached by participant 7) from 0 %
(participant 4) using no feature selection algorithm. For participant 4 it
achieves an accuracy of 75 %. These results suggest that the SFS algorithm can
increase robustness for training our models on diverse participants with the
trade off of reduced maximum accuracy.

####  IV-3 Recursive feature selection with cross-validation

Using the RFECV algorithm cross-validation is used to find the least important
feature, which can be removed. The best performing classifier LDA used in
average 17.6 features over all participants. The selected features show
comparable SHAP values compared to no feature selection algorithm. DTC and RF
are able to slightly increase their accuracy compared to not applying a
feature selection algorithm. The increase of mean accuracy in the decision
tree classifier could be due to favored random feature selection. Due to the
RFECV the random forest classifier slightly increased the importance of EDA
based features, which could explain the better performance. The same holds for
the slight performance gains of the XGB classifier.

The largest increase in mean accuracy is obtained using the logistic
regression classifier (approximately 10 %). Based on the SHAP values, we can
argue that again the EDA based features are valued higher alongside the
temperature values.

####  IV-4 Limitations

Considering the future application scenario of providing feedback to a
ChronoPilot-device, we encounter several practical limitations for data
collection. Depending on the actual use case, users may be required to be
mobile, see for example, the industrial production scenario [3]. Motion
artifacts may corrupt the physiological data, such as PPG. Hence, we may limit
ourselves to physiological data that can be collected even if users are mobile
and in motion. One possible PPG replacement is measuring electrocardiography
(ECG), which directly measures the heart electrical activity. ECG allows for
extracting the same features as PPG. In addition, we could derive respiration
features from ECG (also called EDR), which is another often used non-
invasively measurable physiological signal. Using a thermopile as sensor also
highly depends on the later use case, for example, considering again the
industrial production scenario the user has to stand or walk around. One
possible replacement could be measuring the skin and surrounding temperature.
Skin temperature can often be measured together with EDA using the same
device. Our results indicate that EDA is a strong descriptive feature and
should definitely be used in further studies. Another interesting information
source could be eye tracking, as it provides information about the user’s
focus [24] and it could be integrated in a wearable ChronoPilot-device, such
as VR/AR headsets.

##  V Conclusion

In the present study, we were able to show that it is possible to estimate the
perceived passage of time of air traffic controllers based on physiological
data, such as PPG, EDA, and temperature measurements of a thermopile during
helicopter training sequences of varying workload conditions with high
accuracy up to 79 %. EDA seems to provide the most descriptive features based
on SHAP analysis. This finding reinforces our assumption that we can influence
the perceived passage of time by varying mental workload given that EDA is
considered a biomarker for mental workload and stress. In terms of our
objective of building a ChronoPilot-device, we have shown that we can use
physiological data as feedback to close the control loop. For our future work,
larger datasets may improve the likelihood of generalizability. This can be
done in two ways: increasing the number of participants or the amount of data
per participant. In addition, larger datasets would allow us to explore deep
learning methods. The literature on exploiting physiological data to
automatically detect a person’s subjective time perception is scarce. To date,
a closed-loop approach for guided manipulation of time perception has not yet
been attempted. Our results reported here encourage us in our effort of
creating an automatic closed-loop approach for modulating subjective time
perception in real-world scenarios as envisioned in the ChronoPilot project.
By exploiting the plasticity of subjective time perception, we can develop
future technology that enhances productivity and well-being.

## ACKNOWLEDGMENT

This work was supported by European Union’s Horizon 2020 FET research program
under grant agreement No. 964464 (ChronoPilot). We thank Andreas Psarrakis for
assistance in the data collection process of this study.

## References

  * [1] J. J. McGrath _et al._ , “Age of onset and cumulative risk of mental disorders: a cross-national analysis of population surveys from 29 countries,” _The Lancet Psychiatry_ , vol. 10, no. 9, pp. 668–681, 2023\.
  * [2] R. S. Ogden, “The passage of time during the uk covid-19 lockdown,” _Plos one_ , vol. 15, no. 7, p. e0235871, 2020.
  * [3] J. Botev, K. Drewing, H. Hamann, Y. Khaluf, P. Simoens, and A. Vatakis, “ChronoPilot — modulating time perception,” in _2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)_ , 2021, pp. 215–218.
  * [4] M. Cavdan, B. Celebi, and K. Drewing, “Simultaneous emotional stimuli prolong the timing of vibrotactile events,” _IEEE Transactions on Haptics_ , vol. 16, no. 4, pp. 622–627, 2023.
  * [5] C. Schatzschneider, G. Bruder, and F. Steinicke, “Who turned the clock? effects of manipulated Zeitgebers, cognitive load and immersion on time estimation,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 22, no. 4, pp. 1387–1395, 2016.
  * [6] S. Picard and J. Botev, “Rhythmic stimuli effects on subjective time perception in immersive virtual environments,” in _Proc. of the 14th International Workshop on MMEV_ , 2022, pp. 5–11.
  * [7] J. Kaduk, M. Cavdan, K. Drewing, A. Vatakis, and H. Hamann, “Effects of human-swarm interaction on subjective time perception: Swarm size and speed,” in _Proc. of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 456–465.
  * [8] S. Gedam and S. Paul, “Automatic stress detection using wearable sensors and machine learning: A review,” in _2020 11th International Conference on ICCCNT_ , 2020, pp. 1–7.
  * [9] A. Kumar, K. Sharma, and A. Sharma, “Memor: A multimodal emotion recognition using affective biomarkers for smart prediction of emotional health for people analytics in smart industries,” _Image and Vision Computing_ , vol. 123, p. 104483, 2022.
  * [10] V. Montesinos, F. Dell’Agnola, A. Arza, A. Aminifar, and D. Atienza, “Multi-modal acute stress recognition using off-the-shelf wearable devices,” in _41st Annual International Conference of the IEEE EMBC_ , 2019, pp. 2196–2201.
  * [11] L. Orlandic, A. A. Valdes, and D. Atienza, “Wearable and continuous prediction of passage of time perception for monitoring mental health,” in _2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS)_ , 2021, pp. 444–449.
  * [12] P. Schmidt, A. Reiss, R. Dürichen, and K. Van Laerhoven, “Wearable-based affect recognition—A review,” _Sensors_ , vol. 19, no. 19, p. 4079, 2019\.
  * [13] G. Masinelli, F. Forooghifar, A. Arza, D. Atienza, and A. Aminifar, “Self-aware machine learning for multimodal workload monitoring during manual labor on edge wearable sensors,” _IEEE Design & Test_, vol. 37, no. 5, pp. 58–66, 2020.
  * [14] S. Carvalho, J. Leite, S. Galdo-Álvarez, and O. F. Gonçalves, “The emotional movie database (emdb): A self-report and psychophysiological study,” _Applied psychophysiology and biofeedback_ , vol. 37, pp. 279–294, 2012.
  * [15] A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, “Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers,” _Cognition and emotion_ , vol. 24, no. 7, pp. 1153–1172, 2010.
  * [16] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in _Advances in Neural Information Processing Systems 30_. Curran Associates, Inc., 2017, pp. 4765–4774.
  * [17] E. Balta, A. Psarrakis, and A. Vatakis, “The effects of increased mental workload of air traffic controllers on time perception: Behavioral and physiological evidence,” _Applied Ergonomics_ , vol. 115, 2024.
  * [18] P. van Gent, H. Farah, N. Nes, and B. van Arem, “Heart rate analysis for human factors: Development and validation of an open source toolkit for noisy naturalistic heart rate data,” in _Proc. of the 6th HUMANIST publications_ , 2018.
  * [19] D. Makowski, T. Pham, Z. J. Lau, J. C. Brammer, F. Lespinasse, H. Pham, C. Schölzel, and S. H. A. Chen, “NeuroKit2: A python toolbox for neurophysiological signal processing,” _Behavior Research Methods_ , vol. 53, no. 4, pp. 1689–1696, 2021.
  * [20] A. Luntz, “On estimation of characters obtained in statistical procedure of recognition,” _Technicheskaya Kibernetica_ , 1969.
  * [21] M. Anthony, P. L. Bartlett, P. L. Bartlett _et al._ , _Neural network learning: Theoretical foundations_. Cambridge University Press, 1999.
  * [22] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [23] P. Vanneste, A. Raes, J. Morton, K. Bombeke, B. B. Van Acker, C. Larmuseau, F. Depaepe, and W. Van den Noortgate, “Towards measuring cognitive load through multimodal physiological data,” _Cognition, Technology & Work_, vol. 23, pp. 567–585, 2021.
  * [24] S. Balzarotti, F. Cavaletti, A. D’Aloia, B. Colombo, E. Cardani, M. R. Ciceri, A. Antonietti, and R. Eugeni, “The editing density of moving images influences viewers’ time perception: The mediating role of eye movements,” _Cognitive Science_ , vol. 45, no. 4, p. e12969, 2021.